wt/json
1
0 Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-11-12 16:50:22 +00:00
Code Issues Packages Projects Releases Wiki Activity

Bound what the DOM parsers reserve for arrays that announce a size

Browse Source 
Cause: #5476 made the DOM parsers reserve storage (up to 16384 elements)
for a counted array of CBOR, MessagePack, UBJSON and BJData when it
opens, before any input backs the count. The arrays that are open at the
same time keep their storage until the parse ends, so N nested headers
that announce 16384 elements and are never completed hold
N * 16384 * sizeof(json): 100,000 nested CBOR headers `99 40 00`
(300,000 bytes) peaked at 1.65 GiB, 3.12.0 at 12 MiB; the same for
MessagePack `DC 40 00`, UBJSON `[#I 16384` and BJData.

Fix: json_sax_dom_error_state, the base of the three DOM parsers, keeps a
credit of elements that may still be reserved: 65,536 at the start plus
2 for every byte the reader has read. An array that announces no more
than the credit left is reserved as before (one comparison and one
subtraction per counted array); a larger one is reserved what is left
and grows as its elements arrive. binary_reader tells a DOM parser (a
handler whose pointer converts to json_sax_dom_error_state*; a handler
that derives from one privately or protectedly, or any other SAX handler,
is not told anything and reserves as in the base of this change) where to
find its count of characters read for the duration of sax_parse(). An
RAII guard puts back the previous binding on every exit, also by
exception, so the parser never keeps a pointer to a reader that is gone
(unbound, only the per-array cap of 16384 applies); the public SAX
interface is unchanged, and start_array() still reports the announced number. A
bound per array by the remaining input would not help, since the same
bytes would back every open level; the bound has to be over all arrays.

The elements of an array take at least one byte of input each, so the
credit is more than honest input needs: none of the 19 inputs measured
(canada, citm_catalog, twitter, mesh, update-center, random, numbers,
animation, ik_rig, instruments, geo_response, jeopardy and 7 synthetic
ones, each as CBOR, MessagePack, UBJSON and BJData, with and without
size/type optimization) was given less than it announced. A value that
nests five or more arrays of 16384 elements, each with an array as its
first element, is the exception: the later ones grow as before.

Measurements (clang -O2, macOS arm64, peak memory footprint of
`/usr/bin/time -l`, reproducer: n nested headers):

              n = 20,000                n = 100,000
              develop   fix   3.12.0   develop    fix   3.12.0
  CBOR        338.7    5.9    8.2      1687.7    23.3   11.8   MiB
  MessagePack 338.9    6.0    7.3      1687.9    23.8   11.9   MiB
  UBJSON      339.0    7.6    8.0      1687.9    30.1   11.6   MiB
  BJData      339.0    7.3    7.9      1688.0    30.8   11.6   MiB

That is at most (65,536 + 2 per input byte) values: 1 MiB plus 32 bytes
per input byte with 16-byte values (a value is 24 bytes with
JSON_DIAGNOSTICS and up to 40 with JSON_DIAGNOSTIC_POSITIONS), as
documented in untrusted_input.md. Lower fixed caps do not get there: a
cap of 64 elements still peaks at 113 MiB (n = 100,000) and costs 4.7 %
on arrays of 300 elements (mat300.cbor) and 5.0 % on arrays of 10,000
(arrays_of_big.cbor).

Reader instructions retired (min of 7, 108 rows: 18 inputs in the 6 forms
CBOR, MessagePack, UBJSON and BJData plain and optimized; jeopardy, 46 MB,
is within 0.1 %), compared with the base of this change: -0.19 % to +0.37 % in the formats that announce
sizes (canada.cbor 81.9 M -> 82.1 M, many_small.cbor 333.5 M -> 334.2 M);
the benefit of #5476 is kept (without any reserve canada.cbor takes
116.8 M, +42.6 %, many_small.cbor +71.5 %, mat300.cbor +81.6 %). Rows of
plain UBJSON and BJData, whose arrays announce no size, move by -2.7 % to
+2.6 % without any code of theirs changing: the base alternates
between two instruction counts per process (452 M and 463 M for
many_pairs.ubjson, equal in cycles), and a struct with three unused
members added to the same class reproduces the 474 M mode.

Tests (CBOR, MessagePack, UBJSON, BJData): 200 nested headers that
announce 16384 elements peak at 52,433,600 bytes of DOM allocations on
the base and below 128 Ki values with the fix (scaled by sizeof(value),
so that it holds with the diagnostic macros) (counting allocator, no
process memory measured); an array that completes is still reserved in
one allocation; CBOR also checks that a DOM parser is not bound to a finished
parse (ASan: stack-use-after-scope without the guard), that wrappers
derived publicly (also with a parse inside the parse), privately and
protectedly compile and are bound or not as described, that an array that got less than it
announced decodes and that a SAX handler is told the announced size.
The new tests take 4.6 s under Valgrind at -O0 (gcc:12, all four files).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann committed 2026-11-11 17:10:37 +01:00
1 parent 44b1e7b2b9
commit 4c75628374
10 files changed
+936 -17
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

single_include/nlohmann/json.hpp
Generated
+199 -5
View File
@@ -8866,6 +8866,7 @@ NLOHMANN_JSON_NAMESPACE_END
#include <iterator> // back_inserter
#include <limits> // numeric_limits
#include <string> // char_traits, string
#include <type_traits> // is_convertible, true_type, false_type
#include <utility> // make_pair, move
#include <vector> // vector
#ifdef __cpp_lib_byteswap
@@ -16056,6 +16057,40 @@ constexpr std::size_t unknown_size()
return (std::numeric_limits<std::size_t>::max)();
}
/// the largest number of elements the DOM parsers reserve for one array upfront
constexpr std::size_t reserve_cap()
{
return 16384;
}
/*!
@brief number of array elements a DOM parser may reserve upfront on top of
@ref reserved_elements_per_byte
An array that announces its size is given storage for that many elements when
it opens (at most @ref reserve_cap of them), before any input backs the
announcement, and the arrays that are open at the same time keep their storage
until the input is complete: 100000 nested headers of three bytes each that
announce 16384 elements would otherwise need 1.7 GB. The elements reserved for
all arrays of an input are therefore added up, and the sum must stay below
this base plus @ref reserved_elements_per_byte for every byte read so far; an
array that would go beyond reserves what is left, and grows as its elements
arrive. The base is four arrays of the maximum size, so the first nested
arrays of an input are not affected.
*/
JSON_INLINE_VARIABLE constexpr std::size_t reserved_elements_base = 4 * reserve_cap();
/*!
@brief number of array elements a DOM parser may reserve upfront per byte of
input, in addition to @ref reserved_elements_base
An element of an array takes at least one byte of input, so an array that
completes has used what was reserved for it. At a rate of 2 elements per
byte, an array that announces more than the input provides keeps at most
32 bytes of memory per byte of input, plus the base, until the parse fails.
*/
JSON_INLINE_VARIABLE constexpr std::size_t reserved_elements_per_byte = 2;
/*!
@brief reserve capacity for @a len elements in array @a arr
@@ -16071,8 +16106,7 @@ template<typename ArrayType>
auto reserve_array(ArrayType& arr, std::size_t len, priority_tag<1> /*unused*/)
-> decltype(arr.reserve(len), void())
{
constexpr std::size_t reserve_cap = 16384;
arr.reserve((std::min)(len, reserve_cap));
arr.reserve((std::min)(len, reserve_cap()));
}
template<typename ArrayType>
@@ -16239,6 +16273,13 @@ struct diagnostic_positions
Holds whether a parse error occurred and whether errors are thrown, and
implements the parse_error() and is_errored() members of the SAX interface for
json_sax_dom_parser and json_sax_dom_callback_parser.
It also bounds what the parsers reserve for arrays that announce their size
(reserve_count()): the announced size is not backed by input yet when the
array opens, and the arrays that are open at the same time each keep what was
reserved for them, so the binary_reader tells the parser where to find the
number of characters it has read (set_input_position()), which the bound
is derived from.
*/
class json_sax_dom_error_state
{
@@ -16272,7 +16313,103 @@ class json_sax_dom_error_state
return errored;
}
/// bind the number of characters the input has provided so far; see
/// @ref reserve_count
/// (nullptr: not known, which bounds only the size of each array)
void set_input_position(const std::size_t* position) noexcept
{
input_position = position;
credit_position = 0;
reserve_credit = position != nullptr ? reserved_elements_base : (std::numeric_limits<std::size_t>::max)();
}
/// what set_input_position() changes, so that a binding can be undone
struct reserve_state
{
const std::size_t* position;
std::size_t position_at_refill;
std::size_t credit;
};
reserve_state get_reserve_state() const noexcept
{
return {input_position, credit_position, reserve_credit};
}
void set_reserve_state(const reserve_state& state) noexcept
{
input_position = state.position;
credit_position = state.position_at_refill;
reserve_credit = state.credit;
}
protected:
/*!
@brief the number of elements to reserve for an array that announces @a len
The announced number is not backed by input when the array opens, and the
arrays that are open at the same time each keep what was reserved for
them, so what is reserved for all arrays together is bounded by the
input read so far: @ref reserved_elements_base plus
@ref reserved_elements_per_byte for every character. An array that
announces more than what is left of that is given what is left, and
grows as its elements arrive. Without the position of the input (the
parser is not driven by a binary_reader that is running) nothing is
bounded but the cap of reserve_array().
The common case, an array that announces no more than the credit left,
costs a comparison and a subtraction. It is charged what it announces, which
for an array of more than reserve_cap() elements is more than reserve_array()
reserves for it, which its own elements refill as they are read.
*/
std::size_t reserve_count(const std::size_t len) noexcept
{
if (JSON_HEDLEY_LIKELY(len <= reserve_credit))
{
reserve_credit -= len;
return len;
}
return reserve_count_slowly(len);
}
private:
JSON_HEDLEY_NEVER_INLINE
std::size_t reserve_count_slowly(const std::size_t len) noexcept
{
constexpr std::size_t max_size = (std::numeric_limits<std::size_t>::max)();
if (input_position == nullptr)
{
reserve_credit = max_size;
return len;
}
// refill the credit with 'reserved_elements_per_byte' for every
// character read since the last refill; it saturates instead of
// overflowing when size_t is 32 bits wide and a very large input has
// been read (the position goes back by up to two in BON8, which has
// no sizes)
const std::size_t position = *input_position;
if (position > credit_position)
{
const std::size_t bytes = position - credit_position;
reserve_credit = bytes <= (max_size - reserve_credit) / reserved_elements_per_byte
? reserve_credit + (reserved_elements_per_byte * bytes)
: max_size;
credit_position = position;
}
std::size_t count = (std::min)(len, reserve_cap());
count = (std::min)(count, reserve_credit);
reserve_credit -= count;
return count;
}
/// the number of characters the input has provided so far, if known
const std::size_t* input_position = nullptr;
/// the number of characters the input had provided when the credit was refilled
std::size_t credit_position = 0;
/// the number of elements that may still be reserved
std::size_t reserve_credit = reserved_elements_base;
/// whether a syntax error occurred
bool errored = false;
/// whether to throw exceptions in case of errors
@@ -17107,7 +17244,7 @@ class json_sax_dom_parser : public json_sax_dom_error_state
if (len != detail::unknown_size())
{
reserve_array(*ref->m_data.m_value.array, len, priority_tag<1> {});
reserve_array(*ref->m_data.m_value.array, reserve_count(len), priority_tag<1> {});
}
}
@@ -17455,7 +17592,7 @@ class json_sax_dom_binary_parser : public json_sax_dom_error_state
if (len != detail::unknown_size())
{
reserve_array(*ref_stack.back()->m_data.m_value.array, len, priority_tag<1> {});
reserve_array(*ref_stack.back()->m_data.m_value.array, reserve_count(len), priority_tag<1> {});
}
return true;
@@ -17747,7 +17884,7 @@ class json_sax_dom_callback_parser : public json_sax_dom_error_state
if (len != detail::unknown_size())
{
reserve_array(*ref_stack.back()->m_data.m_value.array, len, priority_tag<1> {});
reserve_array(*ref_stack.back()->m_data.m_value.array, reserve_count(len), priority_tag<1> {});
}
}
@@ -18459,6 +18596,8 @@ class binary_reader
sax = sax_;
container_stack.clear();
bon8_pushback_size = 0;
// the position is only known to the DOM parser while this reader lives
const position_binding binding(bound_parser(sax, std::is_convertible<SAX*, json_sax_dom_error_state*> {}), &chars_read);
bool result = false;
switch (input_format)
@@ -18516,6 +18655,61 @@ class binary_reader
}
private:
/// the DOM parser of @a s whose reserved storage is bounded by the input
/// read so far; nullptr for any other SAX parser, which does not reserve,
/// and for one that derives from a DOM parser privately or protectedly
/// (or ambiguously), which the reader cannot reach
static json_sax_dom_error_state* bound_parser(SAX* s, std::true_type /*is a DOM parser*/) noexcept
{
return s;
}
static json_sax_dom_error_state* bound_parser(SAX* /*s*/, std::false_type /*is a DOM parser*/) noexcept
{
return nullptr;
}
/*!
@brief tells a DOM parser where to find the number of characters read
while sax_parse() runs, and puts back what it knew before when
sax_parse() returns or throws
The reader is a local of the caller and is gone then, so the parser must
not keep its position. A parser that was bound by a reader that is still
running (a parse inside a parse) gets that binding back; that reader is
still alive, as it encloses this one.
*/
class position_binding
{
public:
position_binding(json_sax_dom_error_state* parser_, const std::size_t* position) noexcept
: parser(parser_), saved()
{
if (parser != nullptr)
{
saved = parser->get_reserve_state();
parser->set_input_position(position);
}
}
position_binding(const position_binding&) = delete;
position_binding(position_binding&&) = delete;
position_binding& operator=(const position_binding&) = delete;
position_binding& operator=(position_binding&&) = delete;
~position_binding()
{
if (parser != nullptr)
{
parser->set_reserve_state(saved);
}
}
private:
json_sax_dom_error_state* parser;
json_sax_dom_error_state::reserve_state saved;
};
////////////////////////
// nested containers //
////////////////////////