* Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget The whole-input budget for the elements of optimized UBJSON arrays of type Z, T and F (1048576 plus 4 per byte read) rejects files written by 3.12.0 that have long arrays of these types, such as 1100 rows of 1000 nulls with a `$Z` annotation, and there was no way to read them. The per-byte rate is now the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE. Semantics: - default 4, so the default build is unchanged (the generated code of from_ubjson is identical at -O2 up to source line numbers); - the value must be a positive integer: 0, negative, non-integral, bool, character and wider-than-uintmax_t values fail to compile with a static_assert that names the macro; - a value that does not fit into std::size_t is the largest size_t, so one spelling works for 32- and 64-bit targets; - the limit base + rate * bytes_read saturates at the largest size_t instead of wrapping around, for any rate and input size; a rate of 131072 (2^20 / 8) or more cannot be exceeded by any input (the shortest header of an array of more than 32767 elements is 8 bytes), which switches the whole-input budget off; - the limit of one array (1048576) and the base (1048576) stay fixed constants, so the writer fallback still follows the base only and a reader with any rate reads everything the writer produces; - the macro is not part of the ABI tag and must have the same value in every translation unit; it is documented that way. The memory the arrays can take in a DOM is about 20 bytes per rate and byte of input (measured, macOS, containers of 9 * rate elements: 25 bytes per input byte at rate 1, 81 at 4, 312 at 16, 1226 at 64, 5493 at 256) plus about 33 MB for the base. A 3.12.0 row of N values has a header of b bytes (5 up to 255, 6 up to 32767, 8 above), so a table is read completely if N <= rate * b, and otherwise up to (1048576 + h * rate, h = outer header: 6, 7 or 9 bytes) / (N - rate * b) rows; 1100 rows of 1000 nulls need a rate of 8. The new page of the macro documents the formulas, the measurements, and two examples that are run by the documentation checks. The tests of the budget move to tests/src/unit-ubjson_valueless.cpp, which derives every boundary from the macro and is built with the rates 1, 1000000 and 0xFFFFFFFFFFFFFFFF in addition to the default (the last two check that nothing is refused and that the limit does not wrap around); they fail with the unchanged header for rate 1, 1000000 and the maximum. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Refuse char8_t for JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE and clarify its documentation Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
7.9 KiB
7.9 KiB
nlohmann::basic_json::from_ubjson
// (1)
template<typename InputType>
static basic_json from_ubjson(InputType&& i,
const bool strict = true,
const bool allow_exceptions = true,
const error_handler_t error_handler = error_handler_t::keep);
// (2)
template<typename IteratorType, typename SentinelType = IteratorType>
static basic_json from_ubjson(IteratorType first, SentinelType last,
const bool strict = true,
const bool allow_exceptions = true,
const error_handler_t error_handler = error_handler_t::keep);
Deserializes a given input to a JSON value using the UBJSON (Universal Binary JSON) serialization format.
- Reads from a compatible input.
- Reads from an iterator range, or an iterator and a sentinel of a different type (C++20 ranges support).
The exact mapping and its limitations are described on a dedicated page.
Template parameters
InputType- A compatible input, for instance:
- an
std::istreamobject - a
FILEpointer - a C-style array of characters
- a pointer to a null-terminated string of single byte characters
- a container
objfor whichbegin(obj)andend(obj)produce a valid pair of iterators (as found via ADL or member functions, with semantics compatible tostd::beginandstd::end)
- an
IteratorType- a compatible iterator type
SentinelType- defaults to
IteratorType; may be a different type comparable toIteratorTypeviaoperator!=, for instance.- a custom sentinel type for C++20 ranges
std::default_sentinel_t, whenIteratorTypeisstd::counted_iterator
Parameters
i(in)- an input in UBJSON format convertible to an input adapter
first(in)- iterator to the start of the input
last(in)- iterator to the end of the input, or a sentinel value that compares equal to the end iterator with
operator!= strict(in)- whether to expect the input to be consumed until EOF (
#!cpp trueby default) allow_exceptions(in)- whether to throw exceptions in case of a parse error (optional,
#!cpp trueby default) error_handler(in)- how to treat a string value or object key that is not valid UTF-8; see
error_handler_t. UBJSON does not require a decoder to reject ill-formed UTF-8, so checking is opt-in: the default,keep, does not check at all, as every binary reader did before this parameter was added;strictchecks and throws;replace/ignoresanitize the string the same waydumpwould
Return value
deserialized JSON value; in case of a parse error and allow_exceptions set to #!cpp false, the return value will be
value_t::discarded. The latter can be checked with is_discarded.
Exception safety
Strong guarantee: if an exception is thrown, there are no changes in the JSON value.
Exceptions
- Throws parse_error.110 if the given input ends prematurely or
the end of the file was not reached when
strictwas set to true - Throws parse_error.112 if an invalid type marker is read
(including a type marker after
$of an optimized container, even if its count is 0), or if$is not followed by# - Throws parse_error.113 if a string could not be parsed
successfully (negative or non-integer length,
Cfollowed by a byte above 0x7F), if the count of an optimized container is negative, or if a string value or object key is not valid UTF-8 anderror_handlerisstrict - Throws parse_error.115 if the text of a high-precision number
(
H) is not a valid JSON number - Throws out_of_range.406 if a high-precision number, or a
float64 when
number_float_tisfloat, does not fit intonumber_float_t - Throws out_of_range.408 if the count of an optimized
container exceeds the capacity of the container (or of
std::size_t; an array of typeNis empty, so only the latter applies to it), if an optimized array ofZ,T, orFhas more than 1,048,576 elements, or if the optimized arrays ofZ,T, andFin the input together have more than 1,048,576 elements plusJSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE(4 by default) for every byte read so far
Complexity
Linear in the size of the input.
Examples
??? example
The example shows the deserialization of a byte vector in UBJSON format to a JSON value.
```cpp
--8<-- "examples/from_ubjson.cpp"
```
Output:
```json
--8<-- "examples/from_ubjson.output"
```
See also
- UBJSON - the format, its byte layout and how values are mapped
- Binary formats - overview and comparison of the supported binary formats
- Parsing untrusted input - limits and safe handling of hostile data
- Binary values - how binary values and subtypes are handled
to_ubjson- create a UBJSON serialization of a JSON valueJSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE- set how many elements of arrays ofnull,true, orfalsethe input may announce per bytefrom_cbor- create a JSON value from an input in CBOR formatfrom_msgpack- create a JSON value from an input in MessagePack formatfrom_bson- create a JSON value from an input in BSON formatfrom_bjdata- create a JSON value from an input in BJData formatfrom_bon8- create a JSON value from an input in BON8 format
Version history
- Added in version 3.1.0.
- Added
allow_exceptionsparameter in version 3.2.0. - Extended container support (1) to include types with lvalue-only ADL
begin/end(matchingstd::begin/std::endsemantics) in version 3.13.0. - Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
- Added
error_handlerparameter in version 3.13.0. - Report
start_array(0)to a SAX handler for an optimized array of typeN([$N#n), whatever its count, in version 3.13.0. - Limited the total number of elements of optimized arrays of
Z,T, andFin one input in version 3.13.0; the number of elements per byte can be set withJSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE, for example to read files written by version 3.12.0 and earlier that exceed the default.
!!! warning "Deprecation"
- Overload (2) replaces calls to `from_ubjson` with a pointer and a length as first two parameters, which has been
deprecated in version 3.8.0. In version 4.0.0, this overload
will be deleted (`= delete`) rather than removed. Please replace all calls like
`#!cpp from_ubjson(ptr, len, ...);` with `#!cpp from_ubjson(ptr, ptr+len, ...);`.
- Overload (2) replaces calls to `from_ubjson` with a pair of iterators as their first parameter, which has been
deprecated in version 3.8.0. This overload will be removed in version 4.0.0. Please replace all calls like
`#!cpp from_ubjson({ptr, ptr+len}, ...);` with `#!cpp from_ubjson(ptr, ptr+len, ...);`.
You should be warned by your compiler with a `-Wdeprecated-declarations` warning if you are using a deprecated
function.
See the [migration guide](../../integration/migration_guide.md#parsing) for how to update existing code.