Bound what the DOM parsers reserve for arrays that announce a size (#6082)
* Bound the total number of valueless UBJSON elements per input An optimized UBJSON array of type Z, T or F has no payload, so its declared count is not bounded by the input. The limit of 1048576 applied to each array on its own, so 9 bytes (`[$Z#l` and a count) produced 1048576 values and every further 9 bytes another 1048576: 362 bytes allocated 661 MB in a DOM, and a SAX handler received 1M events per 9 bytes. The reader now adds the counts of all such arrays of one input and rejects the input with out_of_range.408 when the sum exceeds 1048576 plus 4 for every byte read so far. The base is the per-array limit, so an array that was accepted on its own before is still accepted at the start of an input. These arrays add about 80 bytes of memory per input byte (measured 77; one `Z` per byte needs 32), plus the base; the memory of a document as a whole is not bounded by this. The check runs once per valueless array, before its start event, with plain size_t arithmetic that cannot overflow for a 32-bit size_t. Containers with a payload type and objects (every member needs a key) are bounded by the input already and are unchanged. BJData rejects these types. to_ubjson counts the elements of the typed arrays it writes in one call and writes an array in the unoptimized form (`[#n Z Z ...`) once it would bring the total above 1048576, which a reader accepts at any input size. The check is made only for a typed array of a valueless type, so BJData and other arrays do not pay for it. Everything to_ubjson writes in version 3.13.0 can therefore be read back. Two consequences: - to_ubjson output changes for values with more than 1048576 elements in arrays of type Z, T or F: such an array is written at one byte per element, e.g. 270000 rows of 100 nulls take 26.8 MB instead of 1.35 MB. - Output of version 3.12.0 and earlier (typed arrays without such a total) is rejected by from_ubjson when the arrays hold more than 1048576 elements plus 4 per byte of input. 3.12.0 writes a row of up to 255 nulls in 5 bytes (6 above 255), so a table is rejected from 1075 rows of 1000 nulls or 209723 rows of 25 (accepted: 1000 rows of 1000, 209000 rows of 25); rows of 20 or fewer are never rejected. Measured on the test data: no file of json_test_data (converted with to_ubjson in all three modes, 846 round trips) holds an array of this kind with more than 0.17 values per byte. All of them are read back. Reproducer (K arrays of 1048576 nulls), Apple clang -O2: K=40 (362 bytes): before 41943040 values, 664 MB; after rejected, 33 MB K=2 (20 bytes): before 2097152 values, 53 MB; after rejected, 33 MB The new tests count events with a SAX handler where they can: building and comparing million-element values made the first version take 176 s under Valgrind, the current one about 22 s. Behaviour change: an input that holds more than 1048576 elements in arrays of type Z, T or F at more than 4 per byte is now rejected. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Report an empty array for UBJSON `[$N#<n>` instead of n elements A no-op (N) is not a value, so an optimized array of type N holds no element whatever count it declares. The reader skipped the elements but still passed the declared count to the SAX parser: a handler was told start_array(n) and no element followed, and the DOM parser reserved min(n, 16384) values (256 KB) for an array that stays empty. Nine bytes (`[$N#l 0x7FFFFFFF`) therefore committed 256 KB: 2000 of them in one input, 18 KB, committed 512 MB of heap (capacity 163,840,000 values for the 10000 arrays of the original probe: 2.6 GB for 70 KB of input). Untouched pages keep the resident size small, but the commit charge and a 32-bit address space are consumed. The reader now announces 0 elements for an optimized array of type N. The loop that skips the elements is unchanged (it never ran). A count that does not fit into std::size_t is still rejected by the count reader; a count that is too large for the array type of the DOM (before: out_of_range.408 "excessive array size" from the DOM handler) is accepted now, because the array is empty. Behaviour change: sax_parse reports start_array(0) for `[$N#n` (before: n); from_ubjson no longer throws out_of_range.408 for an N-typed array whose count exceeds the maximum size of the array type. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Bound what the DOM parsers reserve for arrays that announce a size Cause: #5476 made the DOM parsers reserve storage (up to 16384 elements) for a counted array of CBOR, MessagePack, UBJSON and BJData when it opens, before any input backs the count. The arrays that are open at the same time keep their storage until the parse ends, so N nested headers that announce 16384 elements and are never completed hold N * 16384 * sizeof(json): 100,000 nested CBOR headers `99 40 00` (300,000 bytes) peaked at 1.65 GiB, 3.12.0 at 12 MiB; the same for MessagePack `DC 40 00`, UBJSON `[#I 16384` and BJData. Fix: json_sax_dom_error_state, the base of the three DOM parsers, keeps a credit of elements that may still be reserved: 65,536 at the start plus 2 for every byte the reader has read. An array that announces no more than the credit left is reserved as before (one comparison and one subtraction per counted array); a larger one is reserved what is left and grows as its elements arrive. binary_reader tells a DOM parser (a handler whose pointer converts to json_sax_dom_error_state*; a handler that derives from one privately or protectedly, or any other SAX handler, is not told anything and reserves as in the base of this change) where to find its count of characters read for the duration of sax_parse(). An RAII guard puts back the previous binding on every exit, also by exception, so the parser never keeps a pointer to a reader that is gone (unbound, only the per-array cap of 16384 applies); the public SAX interface is unchanged, and start_array() still reports the announced number. A bound per array by the remaining input would not help, since the same bytes would back every open level; the bound has to be over all arrays. The elements of an array take at least one byte of input each, so the credit is more than honest input needs: none of the 19 inputs measured (canada, citm_catalog, twitter, mesh, update-center, random, numbers, animation, ik_rig, instruments, geo_response, jeopardy and 7 synthetic ones, each as CBOR, MessagePack, UBJSON and BJData, with and without size/type optimization) was given less than it announced. A value that nests five or more arrays of 16384 elements, each with an array as its first element, is the exception: the later ones grow as before. Measurements (clang -O2, macOS arm64, peak memory footprint of `/usr/bin/time -l`, reproducer: n nested headers): n = 20,000 n = 100,000 develop fix 3.12.0 develop fix 3.12.0 CBOR 338.7 5.9 8.2 1687.7 23.3 11.8 MiB MessagePack 338.9 6.0 7.3 1687.9 23.8 11.9 MiB UBJSON 339.0 7.6 8.0 1687.9 30.1 11.6 MiB BJData 339.0 7.3 7.9 1688.0 30.8 11.6 MiB That is at most (65,536 + 2 per input byte) values: 1 MiB plus 32 bytes per input byte with 16-byte values (a value is 24 bytes with JSON_DIAGNOSTICS and up to 40 with JSON_DIAGNOSTIC_POSITIONS), as documented in untrusted_input.md. Lower fixed caps do not get there: a cap of 64 elements still peaks at 113 MiB (n = 100,000) and costs 4.7 % on arrays of 300 elements (mat300.cbor) and 5.0 % on arrays of 10,000 (arrays_of_big.cbor). Reader instructions retired (min of 7, 108 rows: 18 inputs in the 6 forms CBOR, MessagePack, UBJSON and BJData plain and optimized; jeopardy, 46 MB, is within 0.1 %), compared with the base of this change: -0.19 % to +0.37 % in the formats that announce sizes (canada.cbor 81.9 M -> 82.1 M, many_small.cbor 333.5 M -> 334.2 M); the benefit of #5476 is kept (without any reserve canada.cbor takes 116.8 M, +42.6 %, many_small.cbor +71.5 %, mat300.cbor +81.6 %). Rows of plain UBJSON and BJData, whose arrays announce no size, move by -2.7 % to +2.6 % without any code of theirs changing: the base alternates between two instruction counts per process (452 M and 463 M for many_pairs.ubjson, equal in cycles), and a struct with three unused members added to the same class reproduces the 474 M mode. Tests (CBOR, MessagePack, UBJSON, BJData): 200 nested headers that announce 16384 elements peak at 52,433,600 bytes of DOM allocations on the base and below 128 Ki values with the fix (scaled by sizeof(value), so that it holds with the diagnostic macros) (counting allocator, no process memory measured); an array that completes is still reserved in one allocation; CBOR also checks that a DOM parser is not bound to a finished parse (ASan: stack-use-after-scope without the guard), that wrappers derived publicly (also with a parse inside the parse), privately and protectedly compile and are bound or not as described, that an array that got less than it announced decodes and that a SAX handler is told the announced size. The new tests take 4.6 s under Valgrind at -O0 (gcc:12, all four files). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Clarify the reservation note for announced array counts Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Report an empty array for UBJSON [$N#<n>] instead of n elements (#6079) * Report an empty array for UBJSON `[$N#<n>` instead of n elements A no-op (N) is not a value, so an optimized array of type N holds no element whatever count it declares. The reader skipped the elements but still passed the declared count to the SAX parser: a handler was told start_array(n) and no element followed, and the DOM parser reserved min(n, 16384) values (256 KB) for an array that stays empty. Nine bytes (`[$N#l 0x7FFFFFFF`) therefore committed 256 KB: 2000 of them in one input, 18 KB, committed 512 MB of heap (capacity 163,840,000 values for the 10000 arrays of the original probe: 2.6 GB for 70 KB of input). Untouched pages keep the resident size small, but the commit charge and a 32-bit address space are consumed. The reader now announces 0 elements for an optimized array of type N. The loop that skips the elements is unchanged (it never ran). A count that does not fit into std::size_t is still rejected by the count reader; a count that is too large for the array type of the DOM (before: out_of_range.408 "excessive array size" from the DOM handler) is accepted now, because the array is empty. Behaviour change: sax_parse reports start_array(0) for `[$N#n` (before: n); from_ubjson no longer throws out_of_range.408 for an N-typed array whose count exceeds the maximum size of the array type. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Clarify the reservation note for announced array counts Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget (#6083) * Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget The whole-input budget for the elements of optimized UBJSON arrays of type Z, T and F (1048576 plus 4 per byte read) rejects files written by 3.12.0 that have long arrays of these types, such as 1100 rows of 1000 nulls with a `$Z` annotation, and there was no way to read them. The per-byte rate is now the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE. Semantics: - default 4, so the default build is unchanged (the generated code of from_ubjson is identical at -O2 up to source line numbers); - the value must be a positive integer: 0, negative, non-integral, bool, character and wider-than-uintmax_t values fail to compile with a static_assert that names the macro; - a value that does not fit into std::size_t is the largest size_t, so one spelling works for 32- and 64-bit targets; - the limit base + rate * bytes_read saturates at the largest size_t instead of wrapping around, for any rate and input size; a rate of 131072 (2^20 / 8) or more cannot be exceeded by any input (the shortest header of an array of more than 32767 elements is 8 bytes), which switches the whole-input budget off; - the limit of one array (1048576) and the base (1048576) stay fixed constants, so the writer fallback still follows the base only and a reader with any rate reads everything the writer produces; - the macro is not part of the ABI tag and must have the same value in every translation unit; it is documented that way. The memory the arrays can take in a DOM is about 20 bytes per rate and byte of input (measured, macOS, containers of 9 * rate elements: 25 bytes per input byte at rate 1, 81 at 4, 312 at 16, 1226 at 64, 5493 at 256) plus about 33 MB for the base. A 3.12.0 row of N values has a header of b bytes (5 up to 255, 6 up to 32767, 8 above), so a table is read completely if N <= rate * b, and otherwise up to (1048576 + h * rate, h = outer header: 6, 7 or 9 bytes) / (N - rate * b) rows; 1100 rows of 1000 nulls need a rate of 8. The new page of the macro documents the formulas, the measurements, and two examples that are run by the documentation checks. The tests of the budget move to tests/src/unit-ubjson_valueless.cpp, which derives every boundary from the macro and is built with the rates 1, 1000000 and 0xFFFFFFFFFFFFFFFF in addition to the default (the last two check that nothing is refused and that the limit does not wrap around); they fail with the unchanged header for rate 1, 1000000 and the maximum. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Refuse char8_t for JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE and clarify its documentation Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use fseek instead of rewind in the UBJSON FILE* test clang-tidy (bugprone-unsafe-functions) rejects rewind() because it has no error detection. Check the result of fseek() instead. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Charge an announced array only what is reserved for it An array that announced more than reserve_cap() elements but no more than the credit left was charged the announced count, while reserve_array() reserved at most reserve_cap() of them. An outer array of 20000 elements therefore left a third nested array of 16384 with 12796 instead of the full reservation. reserve_count() now caps the count first and charges that, and the DOM parsers reserve it through reserve_array_capped(), which does not cap again, so the fast path costs the same as before (instructions retired within +-0.06 % on canada, citm_catalog and twitter in CBOR, MessagePack and UBJSON). Also re-wrap the comment of valueless_elements_base. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann authored and GitHub committed
1 parent
73ec296320
commit
253e4c3926
28 files changed
+2068
-295
No files matched your search
Generated
+343
-32
@@ -3307,6 +3307,17 @@ JSON_HEDLEY_DIAGNOSTIC_POP
|
||||
#define JSON_DISABLE_TUPLE_REFERENCE_CONVERSION 0
|
||||
#endif
|
||||
|
||||
// JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE: how many elements of optimized
|
||||
// UBJSON containers of type Z, T or F (which have no payload) a reader accepts
|
||||
// per byte of input, on top of a fixed base. The value is checked and converted
|
||||
// in binary_reader.hpp; the macro is documented in
|
||||
// docs/mkdocs/docs/api/macros/json_ubjson_valueless_elements_per_byte.md.
|
||||
// It is not part of the ABI tag: it must have the same value in every
|
||||
// translation unit of a program.
|
||||
#ifndef JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE
|
||||
#define JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE 4
|
||||
#endif
|
||||
|
||||
#if JSON_HAS_THREE_WAY_COMPARISON
|
||||
#include <compare> // partial_ordering
|
||||
#endif
|
||||
@@ -8980,6 +8991,7 @@ NLOHMANN_JSON_NAMESPACE_END
|
||||
#include <iterator> // back_inserter
|
||||
#include <limits> // numeric_limits
|
||||
#include <string> // char_traits, string
|
||||
#include <type_traits> // false_type, integral_constant, is_convertible, is_integral, is_same, remove_cv, remove_reference, true_type
|
||||
#include <utility> // make_pair, move
|
||||
#include <vector> // vector
|
||||
#ifdef __cpp_lib_byteswap
|
||||
@@ -16178,6 +16190,40 @@ constexpr std::size_t unknown_size()
|
||||
return (std::numeric_limits<std::size_t>::max)();
|
||||
}
|
||||
|
||||
/// the largest number of elements the DOM parsers reserve for one array upfront
|
||||
constexpr std::size_t reserve_cap()
|
||||
{
|
||||
return 16384;
|
||||
}
|
||||
|
||||
/*!
|
||||
@brief number of array elements a DOM parser may reserve upfront on top of
|
||||
@ref reserved_elements_per_byte
|
||||
|
||||
An array that announces its size is given storage for that many elements when
|
||||
it opens (at most @ref reserve_cap of them), before any input backs the
|
||||
announcement, and the arrays that are open at the same time keep their storage
|
||||
until the input is complete: 100000 nested headers of three bytes each that
|
||||
announce 16384 elements would otherwise need 1.7 GB. The elements reserved for
|
||||
all arrays of an input are therefore added up, and the sum must stay below
|
||||
this base plus @ref reserved_elements_per_byte for every byte read so far; an
|
||||
array that would go beyond reserves what is left, and grows as its elements
|
||||
arrive. The base is four arrays of the maximum size, so the first nested
|
||||
arrays of an input are not affected.
|
||||
*/
|
||||
JSON_INLINE_VARIABLE constexpr std::size_t reserved_elements_base = 4 * reserve_cap();
|
||||
|
||||
/*!
|
||||
@brief number of array elements a DOM parser may reserve upfront per byte of
|
||||
input, in addition to @ref reserved_elements_base
|
||||
|
||||
An element of an array takes at least one byte of input, so an array that
|
||||
completes has used what was reserved for it. At a rate of 2 elements per
|
||||
byte, an array that announces more than the input provides keeps at most
|
||||
32 bytes of memory per byte of input, plus the base, until the parse fails.
|
||||
*/
|
||||
JSON_INLINE_VARIABLE constexpr std::size_t reserved_elements_per_byte = 2;
|
||||
|
||||
/*!
|
||||
@brief reserve capacity for @a len elements in array @a arr
|
||||
|
||||
@@ -16193,14 +16239,26 @@ template<typename ArrayType>
|
||||
auto reserve_array(ArrayType& arr, std::size_t len, priority_tag<1> /*unused*/)
|
||||
-> decltype(arr.reserve(len), void())
|
||||
{
|
||||
constexpr std::size_t reserve_cap = 16384;
|
||||
arr.reserve((std::min)(len, reserve_cap));
|
||||
arr.reserve((std::min)(len, reserve_cap()));
|
||||
}
|
||||
|
||||
template<typename ArrayType>
|
||||
inline void reserve_array(ArrayType& /*arr*/, std::size_t /*len*/, priority_tag<0> /*unused*/)
|
||||
{}
|
||||
|
||||
/// reserve capacity for @a count elements in array @a arr, where @a count is
|
||||
/// at most reserve_cap() already (see json_sax_dom_error_state::reserve_count)
|
||||
template<typename ArrayType>
|
||||
auto reserve_array_capped(ArrayType& arr, std::size_t count, priority_tag<1> /*unused*/)
|
||||
-> decltype(arr.reserve(count), void())
|
||||
{
|
||||
arr.reserve(count);
|
||||
}
|
||||
|
||||
template<typename ArrayType>
|
||||
inline void reserve_array_capped(ArrayType& /*arr*/, std::size_t /*count*/, priority_tag<0> /*unused*/)
|
||||
{}
|
||||
|
||||
/*!
|
||||
@brief reports an object or array whose announced size exceeds max_size()
|
||||
|
||||
@@ -16361,6 +16419,13 @@ struct diagnostic_positions
|
||||
Holds whether a parse error occurred and whether errors are thrown, and
|
||||
implements the parse_error() and is_errored() members of the SAX interface for
|
||||
json_sax_dom_parser and json_sax_dom_callback_parser.
|
||||
|
||||
It also bounds what the parsers reserve for arrays that announce their size
|
||||
(reserve_count()): the announced size is not backed by input yet when the
|
||||
array opens, and the arrays that are open at the same time each keep what was
|
||||
reserved for them, so the binary_reader tells the parser where to find the
|
||||
number of characters it has read (set_input_position()), which the bound
|
||||
is derived from.
|
||||
*/
|
||||
class json_sax_dom_error_state
|
||||
{
|
||||
@@ -16394,7 +16459,105 @@ class json_sax_dom_error_state
|
||||
return errored;
|
||||
}
|
||||
|
||||
/// bind the number of characters the input has provided so far; see
|
||||
/// @ref reserve_count
|
||||
/// (nullptr: not known, which bounds only the size of each array)
|
||||
void set_input_position(const std::size_t* position) noexcept
|
||||
{
|
||||
input_position = position;
|
||||
credit_position = 0;
|
||||
reserve_credit = position != nullptr ? reserved_elements_base : (std::numeric_limits<std::size_t>::max)();
|
||||
}
|
||||
|
||||
/// what set_input_position() changes, so that a binding can be undone
|
||||
struct reserve_state
|
||||
{
|
||||
const std::size_t* position;
|
||||
std::size_t position_at_refill;
|
||||
std::size_t credit;
|
||||
};
|
||||
|
||||
reserve_state get_reserve_state() const noexcept
|
||||
{
|
||||
return {input_position, credit_position, reserve_credit};
|
||||
}
|
||||
|
||||
void set_reserve_state(const reserve_state& state) noexcept
|
||||
{
|
||||
input_position = state.position;
|
||||
credit_position = state.position_at_refill;
|
||||
reserve_credit = state.credit;
|
||||
}
|
||||
|
||||
protected:
|
||||
/*!
|
||||
@brief the number of elements to reserve for an array that announces @a len
|
||||
|
||||
The announced number is not backed by input when the array opens, and the
|
||||
arrays that are open at the same time each keep what was reserved for
|
||||
them, so what is reserved for all arrays together is bounded by the
|
||||
input read so far: @ref reserved_elements_base plus
|
||||
@ref reserved_elements_per_byte for every character. An array that
|
||||
announces more than what is left of that is given what is left, and
|
||||
grows as its elements arrive. Without the position of the input (the
|
||||
parser is not driven by a binary_reader that is running) nothing is
|
||||
bounded but reserve_cap() for each array.
|
||||
|
||||
The common case, an array whose reservation is no more than the credit
|
||||
left, costs a comparison and a subtraction. An array is charged what is
|
||||
reserved for it, at most reserve_cap() elements, not what it announces, so
|
||||
an array of more elements does not take credit from the arrays nested in
|
||||
it.
|
||||
*/
|
||||
std::size_t reserve_count(const std::size_t len) noexcept
|
||||
{
|
||||
const std::size_t count = (std::min)(len, reserve_cap());
|
||||
if (JSON_HEDLEY_LIKELY(count <= reserve_credit))
|
||||
{
|
||||
reserve_credit -= count;
|
||||
return count;
|
||||
}
|
||||
return reserve_count_slowly(count);
|
||||
}
|
||||
|
||||
private:
|
||||
/// @a count is at most reserve_cap()
|
||||
JSON_HEDLEY_NEVER_INLINE
|
||||
std::size_t reserve_count_slowly(std::size_t count) noexcept
|
||||
{
|
||||
constexpr std::size_t max_size = (std::numeric_limits<std::size_t>::max)();
|
||||
if (input_position == nullptr)
|
||||
{
|
||||
reserve_credit = max_size;
|
||||
return count;
|
||||
}
|
||||
|
||||
// refill the credit with 'reserved_elements_per_byte' for every
|
||||
// character read since the last refill; it saturates instead of
|
||||
// overflowing when size_t is 32 bits wide and a very large input has
|
||||
// been read (the position goes back by up to two in BON8, which has
|
||||
// no sizes)
|
||||
const std::size_t position = *input_position;
|
||||
if (position > credit_position)
|
||||
{
|
||||
const std::size_t bytes = position - credit_position;
|
||||
reserve_credit = bytes <= (max_size - reserve_credit) / reserved_elements_per_byte
|
||||
? reserve_credit + (reserved_elements_per_byte * bytes)
|
||||
: max_size;
|
||||
credit_position = position;
|
||||
}
|
||||
|
||||
count = (std::min)(count, reserve_credit);
|
||||
reserve_credit -= count;
|
||||
return count;
|
||||
}
|
||||
|
||||
/// the number of characters the input has provided so far, if known
|
||||
const std::size_t* input_position = nullptr;
|
||||
/// the number of characters the input had provided when the credit was refilled
|
||||
std::size_t credit_position = 0;
|
||||
/// the number of elements that may still be reserved
|
||||
std::size_t reserve_credit = reserved_elements_base;
|
||||
/// whether a syntax error occurred
|
||||
bool errored = false;
|
||||
/// whether to throw exceptions in case of errors
|
||||
@@ -17229,7 +17392,7 @@ class json_sax_dom_parser : public json_sax_dom_error_state
|
||||
|
||||
if (len != detail::unknown_size())
|
||||
{
|
||||
reserve_array(*ref->m_data.m_value.array, len, priority_tag<1> {});
|
||||
reserve_array_capped(*ref->m_data.m_value.array, reserve_count(len), priority_tag<1> {});
|
||||
}
|
||||
}
|
||||
|
||||
@@ -17577,7 +17740,7 @@ class json_sax_dom_binary_parser : public json_sax_dom_error_state
|
||||
|
||||
if (len != detail::unknown_size())
|
||||
{
|
||||
reserve_array(*ref_stack.back()->m_data.m_value.array, len, priority_tag<1> {});
|
||||
reserve_array_capped(*ref_stack.back()->m_data.m_value.array, reserve_count(len), priority_tag<1> {});
|
||||
}
|
||||
|
||||
return true;
|
||||
@@ -17869,7 +18032,7 @@ class json_sax_dom_callback_parser : public json_sax_dom_error_state
|
||||
|
||||
if (len != detail::unknown_size())
|
||||
{
|
||||
reserve_array(*ref_stack.back()->m_data.m_value.array, len, priority_tag<1> {});
|
||||
reserve_array_capped(*ref_stack.back()->m_data.m_value.array, reserve_count(len), priority_tag<1> {});
|
||||
}
|
||||
}
|
||||
|
||||
@@ -18488,32 +18651,119 @@ JSON_INLINE_VARIABLE constexpr std::size_t max_valueless_container_size = 1 << 2
|
||||
|
||||
The limit @ref max_valueless_container_size applies to each container on its
|
||||
own, so on its own it lets 9 bytes announce 1048576 values, and every further
|
||||
9 bytes another 1048576. The elements of all containers of a valueless type
|
||||
in one input are therefore added up, and the sum must stay below this base
|
||||
8 bytes (a row of an optimized array of arrays) another 1048576. The elements
|
||||
of all containers of a valueless type in one input are therefore added up, and
|
||||
the sum must stay below this base
|
||||
plus @ref valueless_elements_per_byte for every byte read so far. The base is
|
||||
the per-container limit, so a container that was accepted on its own before
|
||||
the sum was introduced is still accepted at the start of an input.
|
||||
*/
|
||||
JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_base = max_valueless_container_size;
|
||||
|
||||
/// the value of JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE as a std::size_t:
|
||||
/// saturated at the largest one, not truncated, for a value that is wider
|
||||
template<typename T>
|
||||
constexpr std::size_t valueless_elements_rate(const T value) noexcept
|
||||
{
|
||||
return static_cast<std::uintmax_t>(value) > (std::numeric_limits<std::size_t>::max)()
|
||||
? (std::numeric_limits<std::size_t>::max)()
|
||||
: static_cast<std::size_t>(value);
|
||||
}
|
||||
|
||||
/*!
|
||||
@brief number of elements of valueless UBJSON containers accepted per byte of
|
||||
input, in addition to @ref valueless_elements_base
|
||||
|
||||
An element of a value that has a payload costs at least one byte of input.
|
||||
Valueless elements are accepted at a rate of 4 per byte. A value takes 16
|
||||
bytes of memory in a DOM, so these arrays add about 80 bytes of memory per
|
||||
byte of input (measured: 77; one `Z` per byte needs about 32), beyond what the
|
||||
same bytes cost anyway, plus the base. This bounds the arrays of valueless
|
||||
type, not the memory of a whole document. Containers written by
|
||||
@ref binary_writer never need more than the base, because it writes an array
|
||||
in the unoptimized form once the typed arrays written so far and this one add
|
||||
up to more than that. The rate matters for input written by other encoders and
|
||||
by versions before 3.13.0 only; in the data this was measured on, no input has
|
||||
more than 0.17 elements per byte.
|
||||
*/
|
||||
JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_per_byte = 4;
|
||||
This is the value of the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE (4 by
|
||||
default), see docs/mkdocs/docs/api/macros/json_ubjson_valueless_elements_per_byte.md.
|
||||
|
||||
An element of a value that has a payload costs at least one byte of input, but
|
||||
an element of a valueless type costs none, so these are the only elements whose
|
||||
number the input does not bound. They are accepted at this rate per byte read,
|
||||
which bounds the memory they take in a DOM: a value takes 16 bytes there, so
|
||||
the arrays at the limit take about 20 times this rate bytes of memory per byte
|
||||
of input (16 for the value, the rest is allocator rounding and the arrays
|
||||
themselves; measured on macOS, DOM, with containers of 9 times the rate
|
||||
elements: 25 bytes per byte at a rate of 1, 81 at 4, 312 at 16, 1226 at 64 and
|
||||
5493 at 256), plus about 33 MB for the first elements of the base (16 bytes for
|
||||
each of the 1048576 elements, and a vector that grows by reallocation, as only
|
||||
16384 elements are reserved upfront, needs the old and the new storage at once).
|
||||
|
||||
The bound is meant for input written by other encoders and by versions before
|
||||
3.13.0, which wrote long arrays of this type optimized without a limit: input
|
||||
written by @ref binary_writer never needs more than the base, because the
|
||||
writer falls back to the unoptimized form once the arrays written so far and
|
||||
the next one add up to more than that, whatever the rate is.
|
||||
|
||||
Range: a positive integer. A value that is not one is a compile-time error (a
|
||||
reader with a rate of 0 would accept nothing beyond the base, and a negative or
|
||||
fractional rate has no meaning). A value that does not fit into std::size_t is
|
||||
used as the largest std::size_t: the rate is so high then that the bound is
|
||||
out of reach anyway, and a project can use one spelling (for example
|
||||
`0xFFFFFFFFFFFFFFFFu`) for 32 and 64 bit targets. The type of the value must
|
||||
be an integer type that is not wider than std::uintmax_t and not bool or a
|
||||
character type. A rate of at least 2^20 / 8 = 131072 effectively switches the
|
||||
whole-input bound off: an array of more than 32767 elements needs a count of
|
||||
type `l`, so its shortest header is `$Z#l` and four bytes (8 bytes) inside an
|
||||
optimized array of arrays (9 bytes, `[$Z#l`, at the top level), and a shorter
|
||||
header (`$Z#I` and two bytes) announces at most 32767. Every container thus has
|
||||
at most (rate * header bytes) elements, which never exceeds the limit. With a
|
||||
rate of 131071 an input that is refused exists (rows of 1048576 elements gain
|
||||
8 elements each); the per-container limit still applies.
|
||||
|
||||
The value must be the same in every translation unit of a program (it is
|
||||
compiled into member functions of binary_reader, which are inline), and it is
|
||||
not part of the ABI tag of the namespace.
|
||||
*/
|
||||
JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_per_byte =
|
||||
// a value that does not fit into std::size_t is the largest one (see above);
|
||||
// the comparison is made in the widest unsigned type, so that nothing is
|
||||
// truncated before it is made. A function template, so that the casts are
|
||||
// not "useless" (-Wuseless-cast) where the macro already has the type
|
||||
valueless_elements_rate((JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE));
|
||||
|
||||
/// the type of the macro's value is char8_t (C++20)
|
||||
template<typename T>
|
||||
#if defined(__cpp_char8_t)
|
||||
struct valueless_elements_rate_is_char8 : std::is_same<T, char8_t> {};
|
||||
#else
|
||||
struct valueless_elements_rate_is_char8 : std::false_type {};
|
||||
#endif
|
||||
|
||||
/// the type of the macro's value is a character or boolean type
|
||||
template<typename T>
|
||||
struct valueless_elements_rate_is_char_or_bool
|
||||
: disjunction<std::is_same<T, bool>, std::is_same<T, char>, std::is_same<T, signed char>,
|
||||
std::is_same<T, unsigned char>, std::is_same<T, wchar_t>, std::is_same<T, char16_t>,
|
||||
std::is_same<T, char32_t>, valueless_elements_rate_is_char8<T>>
|
||||
{};
|
||||
|
||||
// these messages are what a user sees for a wrong value of the macro. The
|
||||
// integer test comes first because a fractional value would also fail the
|
||||
// positivity test; a value wider than std::uintmax_t would be truncated by
|
||||
// the conversion above, so it is rejected
|
||||
static_assert(std::is_integral<typename std::remove_reference<decltype((JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE))>::type>::value
|
||||
&& !valueless_elements_rate_is_char_or_bool<typename std::remove_cv<typename std::remove_reference<decltype((JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE))>::type>::type>::value,
|
||||
"JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE must be an integer (not bool or a character type)");
|
||||
static_assert(sizeof(decltype((JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE))) <= sizeof(std::uintmax_t),
|
||||
"JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE must not be wider than std::uintmax_t");
|
||||
static_assert((JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE) > 0,
|
||||
"JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE must be a positive integer");
|
||||
|
||||
/*!
|
||||
@brief the number of bytes read up to which
|
||||
@ref valueless_elements_base plus @ref valueless_elements_per_byte times
|
||||
that number can be computed in std::size_t
|
||||
|
||||
For more bytes than this the limit of the valueless elements is the largest
|
||||
std::size_t, which is saturation: the sum is not computed, because it would wrap
|
||||
around, which happens for a rate that is large (the largest std::size_t as the
|
||||
rate wraps already at the first byte) or an input that is large (more than
|
||||
about a billion bytes for the default rate with a 32-bit std::size_t). It is a
|
||||
constant of the build, so no division is made while reading.
|
||||
*/
|
||||
JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_saturation_chars =
|
||||
((std::numeric_limits<std::size_t>::max)() - valueless_elements_base) / (valueless_elements_per_byte == 0 ? 1 : valueless_elements_per_byte);
|
||||
///////////////////
|
||||
// binary reader //
|
||||
///////////////////
|
||||
@@ -18581,6 +18831,8 @@ class binary_reader
|
||||
sax = sax_;
|
||||
container_stack.clear();
|
||||
bon8_pushback_size = 0;
|
||||
// the position is only known to the DOM parser while this reader lives
|
||||
const position_binding binding(bound_parser(sax, std::is_convertible<SAX*, json_sax_dom_error_state*> {}), &chars_read);
|
||||
bool result = false;
|
||||
|
||||
switch (input_format)
|
||||
@@ -18649,6 +18901,61 @@ class binary_reader
|
||||
}
|
||||
|
||||
private:
|
||||
/// the DOM parser of @a s whose reserved storage is bounded by the input
|
||||
/// read so far; nullptr for any other SAX parser, which does not reserve,
|
||||
/// and for one that derives from a DOM parser privately or protectedly
|
||||
/// (or ambiguously), which the reader cannot reach
|
||||
static json_sax_dom_error_state* bound_parser(SAX* s, std::true_type /*is a DOM parser*/) noexcept
|
||||
{
|
||||
return s;
|
||||
}
|
||||
|
||||
static json_sax_dom_error_state* bound_parser(SAX* /*s*/, std::false_type /*is a DOM parser*/) noexcept
|
||||
{
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
/*!
|
||||
@brief tells a DOM parser where to find the number of characters read
|
||||
while sax_parse() runs, and puts back what it knew before when
|
||||
sax_parse() returns or throws
|
||||
|
||||
The reader is a local of the caller and is gone then, so the parser must
|
||||
not keep its position. A parser that was bound by a reader that is still
|
||||
running (a parse inside a parse) gets that binding back; that reader is
|
||||
still alive, as it encloses this one.
|
||||
*/
|
||||
class position_binding
|
||||
{
|
||||
public:
|
||||
position_binding(json_sax_dom_error_state* parser_, const std::size_t* position) noexcept
|
||||
: parser(parser_), saved()
|
||||
{
|
||||
if (parser != nullptr)
|
||||
{
|
||||
saved = parser->get_reserve_state();
|
||||
parser->set_input_position(position);
|
||||
}
|
||||
}
|
||||
|
||||
position_binding(const position_binding&) = delete;
|
||||
position_binding(position_binding&&) = delete;
|
||||
position_binding& operator=(const position_binding&) = delete;
|
||||
position_binding& operator=(position_binding&&) = delete;
|
||||
|
||||
~position_binding()
|
||||
{
|
||||
if (parser != nullptr)
|
||||
{
|
||||
parser->set_reserve_state(saved);
|
||||
}
|
||||
}
|
||||
|
||||
private:
|
||||
json_sax_dom_error_state* parser;
|
||||
json_sax_dom_error_state::reserve_state saved;
|
||||
};
|
||||
|
||||
////////////////////////
|
||||
// nested containers //
|
||||
////////////////////////
|
||||
@@ -22161,13 +22468,14 @@ class binary_reader
|
||||
*/
|
||||
bool charge_valueless_elements(const std::size_t count) noexcept
|
||||
{
|
||||
// the limit saturates instead of overflowing when size_t is 32 bits
|
||||
// wide and a very large input has been read; valueless_elements
|
||||
// never exceeds the limit, so the difference cannot underflow
|
||||
constexpr std::size_t max_size = (std::numeric_limits<std::size_t>::max)();
|
||||
const std::size_t limit = chars_read <= (max_size - valueless_elements_base) / valueless_elements_per_byte
|
||||
// limit = base + rate * chars_read, saturating at the largest size_t
|
||||
// instead of wrapping around, which would make a huge rate or input
|
||||
// reject everything. valueless_elements never exceeds the limit of an
|
||||
// earlier call, and the limit does not decrease, so the difference
|
||||
// below cannot underflow.
|
||||
const std::size_t limit = chars_read <= valueless_elements_saturation_chars
|
||||
? valueless_elements_base + (valueless_elements_per_byte * chars_read)
|
||||
: max_size;
|
||||
: (std::numeric_limits<std::size_t>::max)();
|
||||
if (JSON_HEDLEY_UNLIKELY(count > limit - valueless_elements))
|
||||
{
|
||||
return false;
|
||||
@@ -30793,12 +31101,14 @@ class binary_writer
|
||||
// reader has nothing but the declared count to bound the allocation
|
||||
// by. It refuses one array of more than max_valueless_container_size
|
||||
// elements, and all such arrays of one input together when they add
|
||||
// up to more than valueless_elements_base plus a number per byte
|
||||
// read. Write the unoptimized form, at one byte per element, for
|
||||
// an array that would exceed the base with the arrays written so
|
||||
// far, so that the result can be read back: the limit of a reader
|
||||
// is never below the base. Objects are not affected: every element
|
||||
// is preceded by its key.
|
||||
// up to more than valueless_elements_base plus
|
||||
// JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE per byte read. Write the
|
||||
// unoptimized form, at one byte per element, for an array that
|
||||
// would exceed the base with the arrays written so far, so that the
|
||||
// result can be read back: the limit of a reader is never below the
|
||||
// base, whatever the macro is set to, so the writer does not use
|
||||
// it (and a reader with a larger rate reads this output as well).
|
||||
// Objects are not affected: every element is preceded by its key.
|
||||
if (same_prefix && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
|
||||
{
|
||||
const bool valueless_type = IsArray && (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
|
||||
@@ -48046,6 +48356,7 @@ struct formatter<nlohmann::NLOHMANN_BASIC_JSON_TPL, char> // NOLINT(cert-dcl58-c
|
||||
#undef JSON_USE_OBJECTS_FOR_ENUM_KEYED_MAPS
|
||||
#undef JSON_DISABLE_STREAM_BULK_SCAN
|
||||
#undef JSON_DISABLE_UNLOCKED_FILE_IO
|
||||
#undef JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE
|
||||
#endif
|
||||
|
||||
// #include <nlohmann/thirdparty/hedley/hedley_undef.hpp>
|
||||
|
||||
Reference in new issue
Block a user