Files
json/docs/mkdocs/docs/api/basic_json/from_ubjson.md
T
Niels Lohmann 73aea7ce54 Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget (#6083)
* Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget

The whole-input budget for the elements of optimized UBJSON arrays of
type Z, T and F (1048576 plus 4 per byte read) rejects files written by
3.12.0 that have long arrays of these types, such as 1100 rows of 1000
nulls with a `$Z` annotation, and there was no way to read them. The
per-byte rate is now the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE.

Semantics:
- default 4, so the default build is unchanged (the generated code of
  from_ubjson is identical at -O2 up to source line numbers);
- the value must be a positive integer: 0, negative, non-integral, bool,
  character and wider-than-uintmax_t values fail to compile with a
  static_assert that names the macro;
- a value that does not fit into std::size_t is the largest size_t, so
  one spelling works for 32- and 64-bit targets;
- the limit base + rate * bytes_read saturates at the largest size_t
  instead of wrapping around, for any rate and input size; a rate of
  131072 (2^20 / 8) or more cannot be exceeded by any input (the
  shortest header of an array of more than 32767 elements is 8 bytes),
  which switches the whole-input budget off;
- the limit of one array (1048576) and the base (1048576) stay fixed
  constants, so the writer fallback still follows the base only and a
  reader with any rate reads everything the writer produces;
- the macro is not part of the ABI tag and must have the same value in
  every translation unit; it is documented that way.

The memory the arrays can take in a DOM is about 20 bytes per rate and
byte of input (measured, macOS, containers of 9 * rate elements:
25 bytes per input byte at rate 1, 81 at 4, 312 at 16, 1226 at 64, 5493
at 256) plus about 33 MB for the base. A 3.12.0 row of N values has a
header of b bytes (5 up to 255, 6 up to 32767, 8 above), so a table is
read completely if N <= rate * b, and otherwise up to
(1048576 + h * rate, h = outer header: 6, 7 or 9 bytes) / (N - rate * b) rows; 1100 rows of 1000 nulls need
a rate of 8. The new page of the macro documents the formulas, the
measurements, and two examples that are run by the documentation checks.

The tests of the budget move to tests/src/unit-ubjson_valueless.cpp, which
derives every boundary from the macro and is built with the rates 1,
1000000 and 0xFFFFFFFFFFFFFFFF in addition to the default (the last two
check that nothing is refused and that the limit does not wrap around);
they fail with the unchanged header for rate 1, 1000000 and the maximum.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Refuse char8_t for JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE and clarify its documentation

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-11-11 22:37:35 +01:00

7.9 KiB

nlohmann::basic_json::from_ubjson

// (1)
template<typename InputType>
static basic_json from_ubjson(InputType&& i,
                              const bool strict = true,
                              const bool allow_exceptions = true,
                              const error_handler_t error_handler = error_handler_t::keep);
// (2)
template<typename IteratorType, typename SentinelType = IteratorType>
static basic_json from_ubjson(IteratorType first, SentinelType last,
                              const bool strict = true,
                              const bool allow_exceptions = true,
                              const error_handler_t error_handler = error_handler_t::keep);

Deserializes a given input to a JSON value using the UBJSON (Universal Binary JSON) serialization format.

  1. Reads from a compatible input.
  2. Reads from an iterator range, or an iterator and a sentinel of a different type (C++20 ranges support).

The exact mapping and its limitations are described on a dedicated page.

Template parameters

InputType
A compatible input, for instance:
  • an std::istream object
  • a FILE pointer
  • a C-style array of characters
  • a pointer to a null-terminated string of single byte characters
  • a container obj for which begin(obj) and end(obj) produce a valid pair of iterators (as found via ADL or member functions, with semantics compatible to std::begin and std::end)
IteratorType
a compatible iterator type
SentinelType
defaults to IteratorType; may be a different type comparable to IteratorType via operator!=, for instance.
  • a custom sentinel type for C++20 ranges
  • std::default_sentinel_t, when IteratorType is std::counted_iterator

Parameters

i (in)
an input in UBJSON format convertible to an input adapter
first (in)
iterator to the start of the input
last (in)
iterator to the end of the input, or a sentinel value that compares equal to the end iterator with operator!=
strict (in)
whether to expect the input to be consumed until EOF (#!cpp true by default)
allow_exceptions (in)
whether to throw exceptions in case of a parse error (optional, #!cpp true by default)
error_handler (in)
how to treat a string value or object key that is not valid UTF-8; see error_handler_t. UBJSON does not require a decoder to reject ill-formed UTF-8, so checking is opt-in: the default, keep, does not check at all, as every binary reader did before this parameter was added; strict checks and throws; replace/ignore sanitize the string the same way dump would

Return value

deserialized JSON value; in case of a parse error and allow_exceptions set to #!cpp false, the return value will be value_t::discarded. The latter can be checked with is_discarded.

Exception safety

Strong guarantee: if an exception is thrown, there are no changes in the JSON value.

Exceptions

  • Throws parse_error.110 if the given input ends prematurely or the end of the file was not reached when strict was set to true
  • Throws parse_error.112 if an invalid type marker is read (including a type marker after $ of an optimized container, even if its count is 0), or if $ is not followed by #
  • Throws parse_error.113 if a string could not be parsed successfully (negative or non-integer length, C followed by a byte above 0x7F), if the count of an optimized container is negative, or if a string value or object key is not valid UTF-8 and error_handler is strict
  • Throws parse_error.115 if the text of a high-precision number (H) is not a valid JSON number
  • Throws out_of_range.406 if a high-precision number, or a float64 when number_float_t is float, does not fit into number_float_t
  • Throws out_of_range.408 if the count of an optimized container exceeds the capacity of the container (or of std::size_t; an array of type N is empty, so only the latter applies to it), if an optimized array of Z, T, or F has more than 1,048,576 elements, or if the optimized arrays of Z, T, and F in the input together have more than 1,048,576 elements plus JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE (4 by default) for every byte read so far

Complexity

Linear in the size of the input.

Examples

??? example

The example shows the deserialization of a byte vector in UBJSON format to a JSON value.
 
```cpp
--8<-- "examples/from_ubjson.cpp"
```

Output:

```json
--8<-- "examples/from_ubjson.output"
```

See also

  • UBJSON - the format, its byte layout and how values are mapped
  • Binary formats - overview and comparison of the supported binary formats
  • Parsing untrusted input - limits and safe handling of hostile data
  • Binary values - how binary values and subtypes are handled
  • to_ubjson - create a UBJSON serialization of a JSON value
  • JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE - set how many elements of arrays of null, true, or false the input may announce per byte
  • from_cbor - create a JSON value from an input in CBOR format
  • from_msgpack - create a JSON value from an input in MessagePack format
  • from_bson - create a JSON value from an input in BSON format
  • from_bjdata - create a JSON value from an input in BJData format
  • from_bon8 - create a JSON value from an input in BON8 format

Version history

  • Added in version 3.1.0.
  • Added allow_exceptions parameter in version 3.2.0.
  • Extended container support (1) to include types with lvalue-only ADL begin/end (matching std::begin/std::end semantics) in version 3.13.0.
  • Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
  • Added error_handler parameter in version 3.13.0.
  • Report start_array(0) to a SAX handler for an optimized array of type N ([$N#n), whatever its count, in version 3.13.0.
  • Limited the total number of elements of optimized arrays of Z, T, and F in one input in version 3.13.0; the number of elements per byte can be set with JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE, for example to read files written by version 3.12.0 and earlier that exceed the default.

!!! warning "Deprecation"

- Overload (2) replaces calls to `from_ubjson` with a pointer and a length as first two parameters, which has been
  deprecated in version 3.8.0. In version 4.0.0, this overload
  will be deleted (`= delete`) rather than removed. Please replace all calls like
  `#!cpp from_ubjson(ptr, len, ...);` with `#!cpp from_ubjson(ptr, ptr+len, ...);`.
- Overload (2) replaces calls to `from_ubjson` with a pair of iterators as their first parameter, which has been
  deprecated in version 3.8.0. This overload will be removed in version 4.0.0. Please replace all calls like
  `#!cpp from_ubjson({ptr, ptr+len}, ...);` with `#!cpp from_ubjson(ptr, ptr+len, ...);`.

You should be warned by your compiler with a `-Wdeprecated-declarations` warning if you are using a deprecated
function.

See the [migration guide](../../integration/migration_guide.md#parsing) for how to update existing code.