Files
json/docs/mkdocs/docs/features/parsing/untrusted_input.md
T
Niels Lohmann 253e4c3926 Bound what the DOM parsers reserve for arrays that announce a size (#6082)
* Bound the total number of valueless UBJSON elements per input

An optimized UBJSON array of type Z, T or F has no payload, so its
declared count is not bounded by the input. The limit of 1048576 applied
to each array on its own, so 9 bytes (`[$Z#l` and a count) produced
1048576 values and every further 9 bytes another 1048576: 362 bytes
allocated 661 MB in a DOM, and a SAX handler received 1M events per 9
bytes.

The reader now adds the counts of all such arrays of one input and
rejects the input with out_of_range.408 when the sum exceeds 1048576
plus 4 for every byte read so far. The base is the per-array limit, so
an array that was accepted on its own before is still accepted at the
start of an input. These arrays add about 80 bytes of memory per input
byte (measured 77; one `Z` per byte needs 32), plus the base; the memory
of a document as a whole is not bounded by this. The check runs once per
valueless array, before its start event, with plain size_t arithmetic
that cannot overflow for a 32-bit size_t. Containers with a payload type
and objects (every member needs a key) are bounded by the input already
and are unchanged. BJData rejects these types.

to_ubjson counts the elements of the typed arrays it writes in one call
and writes an array in the unoptimized form (`[#n Z Z ...`) once it would
bring the total above 1048576, which a reader accepts at any input size.
The check is made only for a typed array of a valueless type, so BJData
and other arrays do not pay for it. Everything to_ubjson writes in
version 3.13.0 can therefore be read back.

Two consequences:
- to_ubjson output changes for values with more than 1048576 elements in
  arrays of type Z, T or F: such an array is written at one byte per
  element, e.g. 270000 rows of 100 nulls take 26.8 MB instead of 1.35 MB.
- Output of version 3.12.0 and earlier (typed arrays without such a
  total) is rejected by from_ubjson when the arrays hold more than
  1048576 elements plus 4 per byte of input. 3.12.0 writes a row of up to 255 nulls in 5 bytes (6 above
  255), so a table is rejected from 1075 rows of 1000 nulls or 209723
  rows of 25 (accepted: 1000 rows of 1000, 209000 rows of 25); rows of
  20 or fewer are never rejected.

Measured on the test data: no file of json_test_data (converted with
to_ubjson in all three modes, 846 round trips) holds an array of this
kind with more than 0.17 values per byte. All of them are read back.

Reproducer (K arrays of 1048576 nulls), Apple clang -O2:
  K=40 (362 bytes): before 41943040 values, 664 MB; after rejected, 33 MB
  K=2 (20 bytes):   before 2097152 values, 53 MB;   after rejected, 33 MB

The new tests count events with a SAX handler where they can: building
and comparing million-element values made the first version take 176 s
under Valgrind, the current one about 22 s.

Behaviour change: an input that holds more than 1048576 elements in
arrays of type Z, T or F at more than 4 per byte is now rejected.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Report an empty array for UBJSON `[$N#<n>` instead of n elements

A no-op (N) is not a value, so an optimized array of type N holds no
element whatever count it declares. The reader skipped the elements but
still passed the declared count to the SAX parser: a handler was told
start_array(n) and no element followed, and the DOM parser reserved
min(n, 16384) values (256 KB) for an array that stays empty. Nine bytes
(`[$N#l 0x7FFFFFFF`) therefore committed 256 KB: 2000 of them in one
input, 18 KB, committed 512 MB of heap (capacity 163,840,000 values for
the 10000 arrays of the original probe: 2.6 GB for 70 KB of input).
Untouched pages keep the resident size small, but the commit charge and
a 32-bit address space are consumed.

The reader now announces 0 elements for an optimized array of type N.
The loop that skips the elements is unchanged (it never ran). A count
that does not fit into std::size_t is still rejected by the count
reader; a count that is too large for the array type of the DOM (before:
out_of_range.408 "excessive array size" from the DOM handler) is
accepted now, because the array is empty.

Behaviour change: sax_parse reports start_array(0) for `[$N#n` (before:
n); from_ubjson no longer throws out_of_range.408 for an N-typed array
whose count exceeds the maximum size of the array type.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Bound what the DOM parsers reserve for arrays that announce a size

Cause: #5476 made the DOM parsers reserve storage (up to 16384 elements)
for a counted array of CBOR, MessagePack, UBJSON and BJData when it
opens, before any input backs the count. The arrays that are open at the
same time keep their storage until the parse ends, so N nested headers
that announce 16384 elements and are never completed hold
N * 16384 * sizeof(json): 100,000 nested CBOR headers `99 40 00`
(300,000 bytes) peaked at 1.65 GiB, 3.12.0 at 12 MiB; the same for
MessagePack `DC 40 00`, UBJSON `[#I 16384` and BJData.

Fix: json_sax_dom_error_state, the base of the three DOM parsers, keeps a
credit of elements that may still be reserved: 65,536 at the start plus
2 for every byte the reader has read. An array that announces no more
than the credit left is reserved as before (one comparison and one
subtraction per counted array); a larger one is reserved what is left
and grows as its elements arrive. binary_reader tells a DOM parser (a
handler whose pointer converts to json_sax_dom_error_state*; a handler
that derives from one privately or protectedly, or any other SAX handler,
is not told anything and reserves as in the base of this change) where to
find its count of characters read for the duration of sax_parse(). An
RAII guard puts back the previous binding on every exit, also by
exception, so the parser never keeps a pointer to a reader that is gone
(unbound, only the per-array cap of 16384 applies); the public SAX
interface is unchanged, and start_array() still reports the announced number. A
bound per array by the remaining input would not help, since the same
bytes would back every open level; the bound has to be over all arrays.

The elements of an array take at least one byte of input each, so the
credit is more than honest input needs: none of the 19 inputs measured
(canada, citm_catalog, twitter, mesh, update-center, random, numbers,
animation, ik_rig, instruments, geo_response, jeopardy and 7 synthetic
ones, each as CBOR, MessagePack, UBJSON and BJData, with and without
size/type optimization) was given less than it announced. A value that
nests five or more arrays of 16384 elements, each with an array as its
first element, is the exception: the later ones grow as before.

Measurements (clang -O2, macOS arm64, peak memory footprint of
`/usr/bin/time -l`, reproducer: n nested headers):

              n = 20,000                n = 100,000
              develop   fix   3.12.0   develop    fix   3.12.0
  CBOR        338.7    5.9    8.2      1687.7    23.3   11.8   MiB
  MessagePack 338.9    6.0    7.3      1687.9    23.8   11.9   MiB
  UBJSON      339.0    7.6    8.0      1687.9    30.1   11.6   MiB
  BJData      339.0    7.3    7.9      1688.0    30.8   11.6   MiB

That is at most (65,536 + 2 per input byte) values: 1 MiB plus 32 bytes
per input byte with 16-byte values (a value is 24 bytes with
JSON_DIAGNOSTICS and up to 40 with JSON_DIAGNOSTIC_POSITIONS), as
documented in untrusted_input.md. Lower fixed caps do not get there: a
cap of 64 elements still peaks at 113 MiB (n = 100,000) and costs 4.7 %
on arrays of 300 elements (mat300.cbor) and 5.0 % on arrays of 10,000
(arrays_of_big.cbor).

Reader instructions retired (min of 7, 108 rows: 18 inputs in the 6 forms
CBOR, MessagePack, UBJSON and BJData plain and optimized; jeopardy, 46 MB,
is within 0.1 %), compared with the base of this change: -0.19 % to +0.37 % in the formats that announce
sizes (canada.cbor 81.9 M -> 82.1 M, many_small.cbor 333.5 M -> 334.2 M);
the benefit of #5476 is kept (without any reserve canada.cbor takes
116.8 M, +42.6 %, many_small.cbor +71.5 %, mat300.cbor +81.6 %). Rows of
plain UBJSON and BJData, whose arrays announce no size, move by -2.7 % to
+2.6 % without any code of theirs changing: the base alternates
between two instruction counts per process (452 M and 463 M for
many_pairs.ubjson, equal in cycles), and a struct with three unused
members added to the same class reproduces the 474 M mode.

Tests (CBOR, MessagePack, UBJSON, BJData): 200 nested headers that
announce 16384 elements peak at 52,433,600 bytes of DOM allocations on
the base and below 128 Ki values with the fix (scaled by sizeof(value),
so that it holds with the diagnostic macros) (counting allocator, no
process memory measured); an array that completes is still reserved in
one allocation; CBOR also checks that a DOM parser is not bound to a finished
parse (ASan: stack-use-after-scope without the guard), that wrappers
derived publicly (also with a parse inside the parse), privately and
protectedly compile and are bound or not as described, that an array that got less than it
announced decodes and that a SAX handler is told the announced size.
The new tests take 4.6 s under Valgrind at -O0 (gcc:12, all four files).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Clarify the reservation note for announced array counts

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Report an empty array for UBJSON [$N#<n>] instead of n elements (#6079)

* Report an empty array for UBJSON `[$N#<n>` instead of n elements

A no-op (N) is not a value, so an optimized array of type N holds no
element whatever count it declares. The reader skipped the elements but
still passed the declared count to the SAX parser: a handler was told
start_array(n) and no element followed, and the DOM parser reserved
min(n, 16384) values (256 KB) for an array that stays empty. Nine bytes
(`[$N#l 0x7FFFFFFF`) therefore committed 256 KB: 2000 of them in one
input, 18 KB, committed 512 MB of heap (capacity 163,840,000 values for
the 10000 arrays of the original probe: 2.6 GB for 70 KB of input).
Untouched pages keep the resident size small, but the commit charge and
a 32-bit address space are consumed.

The reader now announces 0 elements for an optimized array of type N.
The loop that skips the elements is unchanged (it never ran). A count
that does not fit into std::size_t is still rejected by the count
reader; a count that is too large for the array type of the DOM (before:
out_of_range.408 "excessive array size" from the DOM handler) is
accepted now, because the array is empty.

Behaviour change: sax_parse reports start_array(0) for `[$N#n` (before:
n); from_ubjson no longer throws out_of_range.408 for an N-typed array
whose count exceeds the maximum size of the array type.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Clarify the reservation note for announced array counts

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget (#6083)

* Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget

The whole-input budget for the elements of optimized UBJSON arrays of
type Z, T and F (1048576 plus 4 per byte read) rejects files written by
3.12.0 that have long arrays of these types, such as 1100 rows of 1000
nulls with a `$Z` annotation, and there was no way to read them. The
per-byte rate is now the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE.

Semantics:
- default 4, so the default build is unchanged (the generated code of
  from_ubjson is identical at -O2 up to source line numbers);
- the value must be a positive integer: 0, negative, non-integral, bool,
  character and wider-than-uintmax_t values fail to compile with a
  static_assert that names the macro;
- a value that does not fit into std::size_t is the largest size_t, so
  one spelling works for 32- and 64-bit targets;
- the limit base + rate * bytes_read saturates at the largest size_t
  instead of wrapping around, for any rate and input size; a rate of
  131072 (2^20 / 8) or more cannot be exceeded by any input (the
  shortest header of an array of more than 32767 elements is 8 bytes),
  which switches the whole-input budget off;
- the limit of one array (1048576) and the base (1048576) stay fixed
  constants, so the writer fallback still follows the base only and a
  reader with any rate reads everything the writer produces;
- the macro is not part of the ABI tag and must have the same value in
  every translation unit; it is documented that way.

The memory the arrays can take in a DOM is about 20 bytes per rate and
byte of input (measured, macOS, containers of 9 * rate elements:
25 bytes per input byte at rate 1, 81 at 4, 312 at 16, 1226 at 64, 5493
at 256) plus about 33 MB for the base. A 3.12.0 row of N values has a
header of b bytes (5 up to 255, 6 up to 32767, 8 above), so a table is
read completely if N <= rate * b, and otherwise up to
(1048576 + h * rate, h = outer header: 6, 7 or 9 bytes) / (N - rate * b) rows; 1100 rows of 1000 nulls need
a rate of 8. The new page of the macro documents the formulas, the
measurements, and two examples that are run by the documentation checks.

The tests of the budget move to tests/src/unit-ubjson_valueless.cpp, which
derives every boundary from the macro and is built with the rates 1,
1000000 and 0xFFFFFFFFFFFFFFFF in addition to the default (the last two
check that nothing is refused and that the limit does not wrap around);
they fail with the unchanged header for rate 1, 1000000 and the maximum.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Refuse char8_t for JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE and clarify its documentation

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Use fseek instead of rewind in the UBJSON FILE* test

clang-tidy (bugprone-unsafe-functions) rejects rewind() because it has no
error detection. Check the result of fseek() instead.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Charge an announced array only what is reserved for it

An array that announced more than reserve_cap() elements but no more than
the credit left was charged the announced count, while reserve_array()
reserved at most reserve_cap() of them. An outer array of 20000 elements
therefore left a third nested array of 16384 with 12796 instead of the
full reservation. reserve_count() now caps the count first and charges
that, and the DOM parsers reserve it through reserve_array_capped(),
which does not cap again, so the fast path costs the same as before
(instructions retired within +-0.06 % on canada, citm_catalog and twitter
in CBOR, MessagePack and UBJSON).

Also re-wrap the comment of valueless_elements_base.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-11-15 17:02:05 +01:00

21 KiB

Parsing Untrusted Input

This page is for applications that parse JSON — or one of the supported binary formats (BJData, BON8, BSON, CBOR, MessagePack, UBJSON) — from a source they do not fully control, such as a network connection, an uploaded file, or another process. It summarizes what the library already does for such input and what remains the caller's responsibility, linking to the pages that cover each aspect in detail rather than repeating them.

For the project's threat model and the countermeasures behind these behaviors, see the assurance case; to report a vulnerability, see the security policy.

Errors without exceptions

By default, parse() throws a parse_error (for instance parse_error.101 for a syntax error) when the input is not valid. If your environment cannot use exceptions for untrusted input, the library offers several alternatives; see Parsing and exceptions for the full comparison:

  • Pass #!cpp false as the third argument to parse() to get a discarded value (checked with is_discarded()) instead of a thrown exception, with no diagnostic information.
  • Use accept() to only check whether the input is valid JSON, without building a value.
  • Implement the SAX interface and override parse_error() to react to an error yourself, with the byte position and the exception that would otherwise have been thrown; see the example that overrides it to print instead of throw.

If exceptions are unavailable entirely (-fno-exceptions, or JSON_NOEXCEPTION defined), every #!cpp throw in the library becomes a call to std::abort(). The alternatives above do not throw, so parse() with allow_exceptions set to #!cpp false (and likewise accept() and the from_* functions with allow_exceptions set to #!cpp false) can still reject invalid input in that configuration; a plain parse() call aborts on invalid input. See Working without exceptions and Switch off exceptions for the details and for overriding the abort with JSON_THROW_USER.

Nesting depth

The JSON parser and the binary readers are iterative: they keep the containers they are currently inside of on a heap-allocated stack instead of calling themselves once per nesting level, so the native call stack does not grow with the nesting depth of the input. A deeply nested document is therefore bounded by available memory, not by the call stack, however deeply it is nested.

!!! warning "No built-in depth limit while parsing"

Neither the parser nor the binary readers impose a limit on how deep the input may nest. An attacker can still
exhaust memory (though not the call stack) with a sufficiently deep document. If you need to reject over-deep
untrusted input outright, track the depth yourself, either with a
[parser callback](parser_callbacks.md#recipe-max-nesting-depth-via-a-callback) for the JSON parser, or by counting
`start_object`/`start_array` and `end_object`/`end_array` calls in a
[SAX handler](sax_interface.md) (for the JSON parser or a binary format alike) and throwing once your limit is
exceeded.

Once a value has been parsed, operations that walk it recursively — serializing it with dump, hashing it, copying it, comparing two values with #!cpp ==, #!cpp <, or (in C++20) #!cpp <=>, merging with update, applying a merge_patch, creating a patch with diff or merge_diff, converting it to another basic_json type, destroying it, and the binary writers (to_cbor, to_msgpack, ...) — descend at most JSON_RECURSION_DEPTH_LIMIT levels on the call stack, 128 by default, and continue below that with an explicit stack on the heap. So the call stack does not grow with the nesting depth of the value, however deeply it is nested. (flatten, unflatten, and to_bson never use the call stack for the nesting.)

The call stack still needs room for the levels that are recursed into, and the default limit assumes a stack of ordinary size. The stack that an operation needs is roughly proportional to the limit; with the default of 128 levels, the most demanding operations (copying a value and converting it to another basic_json type, in an unoptimized build) needed up to 224 KiB, and an optimized build needed 48 KiB or less, in measurements on macOS with arm64. The most demanding operation needed this much stack at the limits 0, 32, 128, and 254:

Stack needed, in KiB limit 0 limit 32 limit 128 (default) limit 254
unoptimized build (-O0) 16 or less 64 224 448
optimized build (-O2) 16 or less 16 or less 48 112

A thread with a smaller stack (a task of a real-time operating system, a fiber, or a thread of the musl C library, whose default stack is 128 KiB) can still overflow its stack on deeply nested input, which is why the limit can be lowered: define JSON_RECURSION_DEPTH_LIMIT to a smaller value, down to 0, before including the library. Such a limit changes only the stack that is needed and the speed (values nested more deeply than the limit are walked more slowly), never a result. See the documentation of the macro for the complete tables, the speed, and the rules for choosing a value. Destroying a value (its destructor) recurses over the same number of levels and then continues without the call stack, and needs 32 KiB or less at every limit.

!!! warning "Indented output and flattened keys grow with the depth"

Some results are larger than the value by a factor of its nesting depth.
[`dump`](../../api/basic_json/dump.md) with an `indent` of `n` (and likewise `operator<<` with `std::setw(n)` and
[`basic_json_view::dump`](../../api/basic_json_view/dump.md)) writes up to `n` characters per nesting level in
front of every line, and [`flatten`](../../api/basic_json/flatten.md) repeats the path of every value in its key.
For the 80,001-byte array `[1,[1,...[1,1]...]]` with 20,000 levels, `dump(1)` returns 600 MB, `dump(4)` 2.4 GB,
and the keys of `flatten()` hold 400 MB. A compact `dump()` without indentation stays the size of the value. If you
pretty-print, log, or flatten untrusted input, limit its nesting depth while parsing, for example with a
[parser callback](parser_callbacks.md#recipe-max-nesting-depth-via-a-callback): with a limit of `d`, every line has
at most `n * d` indentation characters, and every flattened key has at most `d` reference tokens.
[`JSON_RECURSION_DEPTH_LIMIT`](../../api/macros/json_recursion_depth_limit.md) does not help here: it changes only
how much of the call stack a walk uses, never its result, and it does not limit the depth that `parse()` accepts.

Input size

The library does not limit the overall size of a JSON text; a value nested or wide enough will use memory proportional to the input. If you parse untrusted input of unbounded size, check the size of the file or stream yourself before — or while — handing it to parse().

For the binary formats, an announced size is never trusted outright:

  • Reading a string or binary value copies the input in bounded 4096-byte chunks and grows the result as bytes are actually consumed, rather than allocating the announced length up front — a truncated input runs out of bytes (reported as a parse error) instead of triggering an oversized allocation.
  • When an array announces its number of elements and the array container supports reserve() (as #!cpp std::vector, the default, does), the DOM parser reserves storage for them before the elements are read, but for at most 16384 elements per array, regardless of how large the announced count is; further elements still grow the container normally as they are read. The arrays that are open at the same time hold their storage at the same time, and an announcement is not backed by input when it is read, so the elements reserved for all arrays of one input together may not exceed 65,536 plus 2 for every byte of input read so far; an array that announces more gets what is left of that and grows as its elements arrive. An element takes at least one byte of input (except in the UBJSON arrays of type Z, T or F described below), so the elements of the arrays that were read use up what was reserved for them, and only arrays that are nested deeply, with an array as their first element, can announce more than is left. For a truncated or hostile input, the storage that announcements can hold is therefore at most (65,536 + 2 per byte of input) values, that is about 1 MiB plus 32 bytes per byte of input with 16-byte values (a value is larger with JSON_DIAGNOSTICS or JSON_DIAGNOSTIC_POSITIONS, 24 to 40 bytes, and so is the bound). For example, 100,000 nested CBOR array headers 99 40 00 (300,000 bytes) that announce 16384 elements each peaked at 23 MiB, against 1.65 GiB if every array reserved what it announced and 12 MiB with version 3.12.0, which reserved nothing. A SAX handler is not affected: it is told the announced number of elements.
  • An announced array or object size that exceeds what the target container could ever hold (its max_size()) is rejected immediately as out_of_range.408, without attempting to allocate anything.
  • A UBJSON array of the no-op type ([$N#<n>) is empty whatever count it declares: a SAX handler is told start_array(0) and the DOM parser reserves nothing.
  • A UBJSON array of type Z, T or F has no payload, so its announced count is not bounded by the input. One array may announce at most 1,048,576 elements, and all such arrays of one input together may contain at most 1,048,576 elements plus K for every byte read so far (checked when each array's header has been read), where K is the value of the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE, 4 by default; beyond that the input is rejected as out_of_range.408. For a DOM that bounds the memory these arrays can need to about 20 K bytes per byte of input (81 at the default, measured; 312 for K = 16) plus about 33 MB for the first 1,048,576 elements; a SAX handler receives at most that many events. Every other optimized array element, and every UBJSON object member (it has a key), takes at least one byte of input. Leave the macro at the default for input you do not control; raise it only to read trusted files that version 3.12.0 or earlier wrote with long arrays of these types, and see its page for how a value is chosen and what it costs. The limit of one array does not depend on the macro.

Strings

Invalid UTF-8 is rejected while parsing JSON text, not just while serializing:

  • In JSON text, an ill-formed UTF-8 byte in a string is a parse_error.101 ("invalid string: ill-formed UTF-8 byte").
  • In UTF-16 or UTF-32 input, an ill-formed sequence is a parse_error.101 as well: a lone surrogate, a high surrogate that is not followed by its low surrogate, and a code point above U+10FFFF are all rejected.

In a binary format, a string that is not valid UTF-8 is accepted unchanged by default (none of the formats except BON8 requires a decoder to reject it). Pass error_handler_t::strict to the from_* function to get a parse_error.113 instead. BON8 always validates (parse_error.112 for an ill-formed sequence).

A #!cpp '\0' (NUL) byte inside a quoted JSON string is always rejected (it must be escaped as \u0000). A NUL byte outside of a string is different: by default it is silently treated as the end of the input, so trailing bytes after it — including further, otherwise well-formed JSON — are silently ignored rather than rejected. Since untrusted input that happens to embed a NUL is a way to make part of it disappear without a parse error, see the FAQ entry and consider defining JSON_STRICT_NUL_HANDLING to 1 to reject a NUL byte like any other unexpected byte instead.

Parsing is not the only place invalid UTF-8 matters: a string that reached a #!cpp json value some other way (for example, constructed by application code, or read from a binary format with the default error_handler_t::keep) still has to round-trip back to JSON text. By default, dump() throws type_error.316 if the string is not valid UTF-8; passing error_handler_t::replace or error_handler_t::ignore avoids the exception instead of crashing an application that forgot to catch it. See Handling invalid UTF-8 for the options and an example.

Duplicate object keys

The JSON specification leaves the handling of repeated keys in an object up to the implementation, and this library does too: as described in object_t, it is unspecified which of the values for a repeated key ends up in the parsed object. If your application must reject duplicate keys instead of silently resolving them one way or another, see the parser callback recipe for rejecting duplicate keys.

Objects with many keys in ordered_json

ordered_json keeps the members of an object in a vector in insertion order (ordered_map) and finds a key by searching all members. Parsing JSON text into an ordered_json avoids that search for objects with many keys and takes linear time. The binary readers (from_cbor, from_msgpack, from_ubjson, from_bjdata, from_bson, and from_bon8) do not yet: reading an object of n keys into an ordered_json costs time proportional to n². A flat object of 40,000 keys (about 300 KB of CBOR) takes about 1.3 s, against a few milliseconds as JSON text or into a json (#6104). json is not affected. If you read untrusted binary input into ordered_json, bound the input size more tightly, read it into a json, or reject objects with many keys in a SAX handler.

Numbers

A number whose value cannot be represented — for instance 1E1000, which overflows double — is rejected while parsing as out_of_range.406 rather than silently becoming infinity. An integer that is syntactically valid but does not fit the 64-bit integer types is not rejected; it is instead stored as a double, which may lose precision for very large values. See number limits for the exact ranges and an example.

Comments and trailing commas

Both comments and trailing commas are rejected by default, matching the JSON specification; they must be explicitly enabled per call with the ignore_comments and ignore_trailing_commas parameters of parse() or accept(). Do not enable either for input whose conformance you cannot otherwise control, since interoperability with strictly conforming JSON consumers is exactly what the default rejects.

JSON Pointers and flattened keys

A JSON Pointer can come from untrusted input as well, for instance as a path in a request or as a key of a flattened object. The non-const operator[] with a JSON Pointer and unflatten create missing values, and when an array index lies beyond the end of an array, they fill the array with #!json null values up to that index. The memory used therefore grows with the index, not with the size of the pointer: on a 64-bit platform, the index 10000000 allocates about 160 MB, the index 100000000 about 1.6 GB. The library does not limit the index. For pointers from untrusted input, check the array indices (or the pointer as a whole) before using them, or use at or contains, which never add values.

Checklist

  • Wrap parsing in a #!cpp try/#!cpp catch block, or use allow_exceptions=false/accept() if your environment cannot use exceptions; never let JSON_NOEXCEPTION's abort() be the first time you think about error handling.
  • If the input's nesting depth matters to you (for example because you pretty-print, log, or flatten it), enforce your own limit with a parser callback or a SAX handler; the library bounds the call stack but not memory use.
  • If you read untrusted binary input into ordered_json, mind that objects with many keys are slow to read.
  • Bound the size of the input itself before parsing, independent of the library's own bounded allocations for binary format lengths.
  • Decide up front how you want a string with invalid UTF-8 — from any source, not only parsing — to be serialized (strict, replace, or ignore), rather than discovering it from an uncaught type_error.316.
  • For untrusted input in a binary format, pass error_handler_t::strict to the from_* function if the strings must be valid UTF-8; by default they are not checked (BON8 always checks).
  • If a stray NUL byte silently truncating trailing input is a problem for your input format, define JSON_STRICT_NUL_HANDLING.
  • Decide whether duplicate object keys should be an error for your application, and add a callback if so.
  • Do not enable ignore_comments or ignore_trailing_commas for input that must be strictly conforming JSON.
  • Check the array indices of untrusted JSON Pointers and flattened keys before passing them to the non-const operator[] or unflatten(), which fill arrays with #!json null values up to the index.

For the broader design rationale and how it is tested (fuzzing, sanitizers, static analysis), see the assurance case and quality assurance. To report a security issue in the library itself, follow the security policy.

See also