Bound what the DOM parsers reserve for arrays that announce a size (#6082)

* Bound the total number of valueless UBJSON elements per input

An optimized UBJSON array of type Z, T or F has no payload, so its
declared count is not bounded by the input. The limit of 1048576 applied
to each array on its own, so 9 bytes (`[$Z#l` and a count) produced
1048576 values and every further 9 bytes another 1048576: 362 bytes
allocated 661 MB in a DOM, and a SAX handler received 1M events per 9
bytes.

The reader now adds the counts of all such arrays of one input and
rejects the input with out_of_range.408 when the sum exceeds 1048576
plus 4 for every byte read so far. The base is the per-array limit, so
an array that was accepted on its own before is still accepted at the
start of an input. These arrays add about 80 bytes of memory per input
byte (measured 77; one `Z` per byte needs 32), plus the base; the memory
of a document as a whole is not bounded by this. The check runs once per
valueless array, before its start event, with plain size_t arithmetic
that cannot overflow for a 32-bit size_t. Containers with a payload type
and objects (every member needs a key) are bounded by the input already
and are unchanged. BJData rejects these types.

to_ubjson counts the elements of the typed arrays it writes in one call
and writes an array in the unoptimized form (`[#n Z Z ...`) once it would
bring the total above 1048576, which a reader accepts at any input size.
The check is made only for a typed array of a valueless type, so BJData
and other arrays do not pay for it. Everything to_ubjson writes in
version 3.13.0 can therefore be read back.

Two consequences:
- to_ubjson output changes for values with more than 1048576 elements in
  arrays of type Z, T or F: such an array is written at one byte per
  element, e.g. 270000 rows of 100 nulls take 26.8 MB instead of 1.35 MB.
- Output of version 3.12.0 and earlier (typed arrays without such a
  total) is rejected by from_ubjson when the arrays hold more than
  1048576 elements plus 4 per byte of input. 3.12.0 writes a row of up to 255 nulls in 5 bytes (6 above
  255), so a table is rejected from 1075 rows of 1000 nulls or 209723
  rows of 25 (accepted: 1000 rows of 1000, 209000 rows of 25); rows of
  20 or fewer are never rejected.

Measured on the test data: no file of json_test_data (converted with
to_ubjson in all three modes, 846 round trips) holds an array of this
kind with more than 0.17 values per byte. All of them are read back.

Reproducer (K arrays of 1048576 nulls), Apple clang -O2:
  K=40 (362 bytes): before 41943040 values, 664 MB; after rejected, 33 MB
  K=2 (20 bytes):   before 2097152 values, 53 MB;   after rejected, 33 MB

The new tests count events with a SAX handler where they can: building
and comparing million-element values made the first version take 176 s
under Valgrind, the current one about 22 s.

Behaviour change: an input that holds more than 1048576 elements in
arrays of type Z, T or F at more than 4 per byte is now rejected.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Report an empty array for UBJSON `[$N#<n>` instead of n elements

A no-op (N) is not a value, so an optimized array of type N holds no
element whatever count it declares. The reader skipped the elements but
still passed the declared count to the SAX parser: a handler was told
start_array(n) and no element followed, and the DOM parser reserved
min(n, 16384) values (256 KB) for an array that stays empty. Nine bytes
(`[$N#l 0x7FFFFFFF`) therefore committed 256 KB: 2000 of them in one
input, 18 KB, committed 512 MB of heap (capacity 163,840,000 values for
the 10000 arrays of the original probe: 2.6 GB for 70 KB of input).
Untouched pages keep the resident size small, but the commit charge and
a 32-bit address space are consumed.

The reader now announces 0 elements for an optimized array of type N.
The loop that skips the elements is unchanged (it never ran). A count
that does not fit into std::size_t is still rejected by the count
reader; a count that is too large for the array type of the DOM (before:
out_of_range.408 "excessive array size" from the DOM handler) is
accepted now, because the array is empty.

Behaviour change: sax_parse reports start_array(0) for `[$N#n` (before:
n); from_ubjson no longer throws out_of_range.408 for an N-typed array
whose count exceeds the maximum size of the array type.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Bound what the DOM parsers reserve for arrays that announce a size

Cause: #5476 made the DOM parsers reserve storage (up to 16384 elements)
for a counted array of CBOR, MessagePack, UBJSON and BJData when it
opens, before any input backs the count. The arrays that are open at the
same time keep their storage until the parse ends, so N nested headers
that announce 16384 elements and are never completed hold
N * 16384 * sizeof(json): 100,000 nested CBOR headers `99 40 00`
(300,000 bytes) peaked at 1.65 GiB, 3.12.0 at 12 MiB; the same for
MessagePack `DC 40 00`, UBJSON `[#I 16384` and BJData.

Fix: json_sax_dom_error_state, the base of the three DOM parsers, keeps a
credit of elements that may still be reserved: 65,536 at the start plus
2 for every byte the reader has read. An array that announces no more
than the credit left is reserved as before (one comparison and one
subtraction per counted array); a larger one is reserved what is left
and grows as its elements arrive. binary_reader tells a DOM parser (a
handler whose pointer converts to json_sax_dom_error_state*; a handler
that derives from one privately or protectedly, or any other SAX handler,
is not told anything and reserves as in the base of this change) where to
find its count of characters read for the duration of sax_parse(). An
RAII guard puts back the previous binding on every exit, also by
exception, so the parser never keeps a pointer to a reader that is gone
(unbound, only the per-array cap of 16384 applies); the public SAX
interface is unchanged, and start_array() still reports the announced number. A
bound per array by the remaining input would not help, since the same
bytes would back every open level; the bound has to be over all arrays.

The elements of an array take at least one byte of input each, so the
credit is more than honest input needs: none of the 19 inputs measured
(canada, citm_catalog, twitter, mesh, update-center, random, numbers,
animation, ik_rig, instruments, geo_response, jeopardy and 7 synthetic
ones, each as CBOR, MessagePack, UBJSON and BJData, with and without
size/type optimization) was given less than it announced. A value that
nests five or more arrays of 16384 elements, each with an array as its
first element, is the exception: the later ones grow as before.

Measurements (clang -O2, macOS arm64, peak memory footprint of
`/usr/bin/time -l`, reproducer: n nested headers):

              n = 20,000                n = 100,000
              develop   fix   3.12.0   develop    fix   3.12.0
  CBOR        338.7    5.9    8.2      1687.7    23.3   11.8   MiB
  MessagePack 338.9    6.0    7.3      1687.9    23.8   11.9   MiB
  UBJSON      339.0    7.6    8.0      1687.9    30.1   11.6   MiB
  BJData      339.0    7.3    7.9      1688.0    30.8   11.6   MiB

That is at most (65,536 + 2 per input byte) values: 1 MiB plus 32 bytes
per input byte with 16-byte values (a value is 24 bytes with
JSON_DIAGNOSTICS and up to 40 with JSON_DIAGNOSTIC_POSITIONS), as
documented in untrusted_input.md. Lower fixed caps do not get there: a
cap of 64 elements still peaks at 113 MiB (n = 100,000) and costs 4.7 %
on arrays of 300 elements (mat300.cbor) and 5.0 % on arrays of 10,000
(arrays_of_big.cbor).

Reader instructions retired (min of 7, 108 rows: 18 inputs in the 6 forms
CBOR, MessagePack, UBJSON and BJData plain and optimized; jeopardy, 46 MB,
is within 0.1 %), compared with the base of this change: -0.19 % to +0.37 % in the formats that announce
sizes (canada.cbor 81.9 M -> 82.1 M, many_small.cbor 333.5 M -> 334.2 M);
the benefit of #5476 is kept (without any reserve canada.cbor takes
116.8 M, +42.6 %, many_small.cbor +71.5 %, mat300.cbor +81.6 %). Rows of
plain UBJSON and BJData, whose arrays announce no size, move by -2.7 % to
+2.6 % without any code of theirs changing: the base alternates
between two instruction counts per process (452 M and 463 M for
many_pairs.ubjson, equal in cycles), and a struct with three unused
members added to the same class reproduces the 474 M mode.

Tests (CBOR, MessagePack, UBJSON, BJData): 200 nested headers that
announce 16384 elements peak at 52,433,600 bytes of DOM allocations on
the base and below 128 Ki values with the fix (scaled by sizeof(value),
so that it holds with the diagnostic macros) (counting allocator, no
process memory measured); an array that completes is still reserved in
one allocation; CBOR also checks that a DOM parser is not bound to a finished
parse (ASan: stack-use-after-scope without the guard), that wrappers
derived publicly (also with a parse inside the parse), privately and
protectedly compile and are bound or not as described, that an array that got less than it
announced decodes and that a SAX handler is told the announced size.
The new tests take 4.6 s under Valgrind at -O0 (gcc:12, all four files).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Clarify the reservation note for announced array counts

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Report an empty array for UBJSON [$N#<n>] instead of n elements (#6079)

* Report an empty array for UBJSON `[$N#<n>` instead of n elements

A no-op (N) is not a value, so an optimized array of type N holds no
element whatever count it declares. The reader skipped the elements but
still passed the declared count to the SAX parser: a handler was told
start_array(n) and no element followed, and the DOM parser reserved
min(n, 16384) values (256 KB) for an array that stays empty. Nine bytes
(`[$N#l 0x7FFFFFFF`) therefore committed 256 KB: 2000 of them in one
input, 18 KB, committed 512 MB of heap (capacity 163,840,000 values for
the 10000 arrays of the original probe: 2.6 GB for 70 KB of input).
Untouched pages keep the resident size small, but the commit charge and
a 32-bit address space are consumed.

The reader now announces 0 elements for an optimized array of type N.
The loop that skips the elements is unchanged (it never ran). A count
that does not fit into std::size_t is still rejected by the count
reader; a count that is too large for the array type of the DOM (before:
out_of_range.408 "excessive array size" from the DOM handler) is
accepted now, because the array is empty.

Behaviour change: sax_parse reports start_array(0) for `[$N#n` (before:
n); from_ubjson no longer throws out_of_range.408 for an N-typed array
whose count exceeds the maximum size of the array type.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Clarify the reservation note for announced array counts

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget (#6083)

* Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget

The whole-input budget for the elements of optimized UBJSON arrays of
type Z, T and F (1048576 plus 4 per byte read) rejects files written by
3.12.0 that have long arrays of these types, such as 1100 rows of 1000
nulls with a `$Z` annotation, and there was no way to read them. The
per-byte rate is now the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE.

Semantics:
- default 4, so the default build is unchanged (the generated code of
  from_ubjson is identical at -O2 up to source line numbers);
- the value must be a positive integer: 0, negative, non-integral, bool,
  character and wider-than-uintmax_t values fail to compile with a
  static_assert that names the macro;
- a value that does not fit into std::size_t is the largest size_t, so
  one spelling works for 32- and 64-bit targets;
- the limit base + rate * bytes_read saturates at the largest size_t
  instead of wrapping around, for any rate and input size; a rate of
  131072 (2^20 / 8) or more cannot be exceeded by any input (the
  shortest header of an array of more than 32767 elements is 8 bytes),
  which switches the whole-input budget off;
- the limit of one array (1048576) and the base (1048576) stay fixed
  constants, so the writer fallback still follows the base only and a
  reader with any rate reads everything the writer produces;
- the macro is not part of the ABI tag and must have the same value in
  every translation unit; it is documented that way.

The memory the arrays can take in a DOM is about 20 bytes per rate and
byte of input (measured, macOS, containers of 9 * rate elements:
25 bytes per input byte at rate 1, 81 at 4, 312 at 16, 1226 at 64, 5493
at 256) plus about 33 MB for the base. A 3.12.0 row of N values has a
header of b bytes (5 up to 255, 6 up to 32767, 8 above), so a table is
read completely if N <= rate * b, and otherwise up to
(1048576 + h * rate, h = outer header: 6, 7 or 9 bytes) / (N - rate * b) rows; 1100 rows of 1000 nulls need
a rate of 8. The new page of the macro documents the formulas, the
measurements, and two examples that are run by the documentation checks.

The tests of the budget move to tests/src/unit-ubjson_valueless.cpp, which
derives every boundary from the macro and is built with the rates 1,
1000000 and 0xFFFFFFFFFFFFFFFF in addition to the default (the last two
check that nothing is refused and that the limit does not wrap around);
they fail with the unchanged header for rate 1, 1000000 and the maximum.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Refuse char8_t for JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE and clarify its documentation

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Use fseek instead of rewind in the UBJSON FILE* test

clang-tidy (bugprone-unsafe-functions) rejects rewind() because it has no
error detection. Check the result of fseek() instead.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Charge an announced array only what is reserved for it

An array that announced more than reserve_cap() elements but no more than
the credit left was charged the announced count, while reserve_array()
reserved at most reserve_cap() of them. An outer array of 20000 elements
therefore left a third nested array of 16384 with 12796 instead of the
full reservation. reserve_count() now caps the count first and charges
that, and the DOM parsers reserve it through reserve_array_capped(),
which does not cap again, so the fast path costs the same as before
(instructions retired within +-0.06 % on canada, citm_catalog and twitter
in CBOR, MessagePack and UBJSON).

Also re-wrap the comment of valueless_elements_base.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

tests/CMakeLists.txt

@@ -434,6 +434,34 @@ json_test_add_test_for(src/unit-stream-bulk-scan.cpp
     MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force}
 )
 
+# test the budget for the elements of UBJSON containers of a valueless type again
+# with other values of JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE: 1 (the smallest),
+# 1000000 (so large that the budget cannot be reached), and the largest value of
+# a 64-bit integer (which saturates the limit already at the first byte, and is
+# clamped to the largest std::size_t where that has 32 bits). The file does not
+# mention JSON_HAS_CPP_<n>, so only C++11 is built unless the standards are forced
+json_test_set_test_options(test-ubjson_valueless_rate_1
+    COMPILE_DEFINITIONS JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE=1
+)
+json_test_add_test_for(src/unit-ubjson_valueless.cpp
+    NAME test-ubjson_valueless_rate_1
+    MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force}
+)
+json_test_set_test_options(test-ubjson_valueless_rate_1000000
+    COMPILE_DEFINITIONS JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE=1000000
+)
+json_test_add_test_for(src/unit-ubjson_valueless.cpp
+    NAME test-ubjson_valueless_rate_1000000
+    MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force}
+)
+json_test_set_test_options(test-ubjson_valueless_rate_max
+    COMPILE_DEFINITIONS JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE=0xFFFFFFFFFFFFFFFFu
+)
+json_test_add_test_for(src/unit-ubjson_valueless.cpp
+    NAME test-ubjson_valueless_rate_max
+    MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force}
+)
+
 # test reading from a FILE* again with the locking stdio functions
 json_test_set_test_options(test-file_input_unlocked_io_disabled
     COMPILE_DEFINITIONS JSON_DISABLE_UNLOCKED_FILE_IO=1

tests/src/allocation_counter.hpp

@@ -0,0 +1,86 @@
+//     __ _____ _____ _____
+//  __|  |   __|     |   | |  JSON for Modern C++ (supporting code)
+// |  |  |__   |  |  | | | |  version 3.12.0
+// |_____|_____|_____|_|___|  https://github.com/nlohmann/json
+//
+// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
+// SPDX-License-Identifier: MIT
+
+#pragma once
+
+#include <algorithm> // max
+#include <cstddef> // size_t
+#include <cstdint> // uint8_t
+#include <memory> // allocator
+#include <vector> // vector
+
+#include <nlohmann/json.hpp>
+
+namespace utils
+{
+/// the allocations of the values of a JSON type that uses counting_allocator
+struct allocation_counts
+{
+    /// bytes allocated and not yet deallocated
+    std::size_t current = 0;
+    /// the largest value current had
+    std::size_t peak = 0;
+    /// number of allocations
+    std::size_t allocations = 0;
+};
+
+/// the counts of all counting_allocators; the tests are single-threaded
+inline allocation_counts& counts()
+{
+    static allocation_counts c; // NOLINT(misc-const-correctness)
+    return c;
+}
+
+/// an allocator that counts the bytes and the number of allocations of a
+/// JSON type, so that a test can check how much memory a parse holds at any
+/// time without measuring the memory of the process
+template<class T>
+struct counting_allocator : std::allocator<T>
+{
+    using std::allocator<T>::allocator;
+
+    counting_allocator() = default;
+    template<class U> counting_allocator(const counting_allocator<U>& /*unused*/) noexcept {} // NOLINT(google-explicit-constructor,hicpp-explicit-conversions)
+
+    template <class U>
+    struct rebind
+    {
+        using other = counting_allocator<U>;
+    };
+
+    T* allocate(const std::size_t n)
+    {
+        T* const result = std::allocator<T>::allocate(n);
+        allocation_counts& c = counts();
+        c.current += n * sizeof(T);
+        c.peak = (std::max)(c.peak, c.current);
+        ++c.allocations;
+        return result;
+    }
+
+    void deallocate(T* p, const std::size_t n)
+    {
+        counts().current -= n * sizeof(T);
+        std::allocator<T>::deallocate(p, n);
+    }
+};
+
+using counted_json = nlohmann::json::with_allocator_t<counting_allocator>;
+
+/// @a count times the bytes of @a unit
+inline std::vector<std::uint8_t> repeated(const std::vector<std::uint8_t>& unit, const std::size_t count)
+{
+    std::vector<std::uint8_t> result;
+    result.reserve(unit.size() * count);
+    for (std::size_t i = 0; i < count; ++i)
+    {
+        result.insert(result.end(), unit.begin(), unit.end());
+    }
+    return result;
+}
+} // namespace utils

tests/src/ubjson_valueless.hpp

@@ -0,0 +1,125 @@
+//     __ _____ _____ _____
+//  __|  |   __|     |   | |  JSON for Modern C++ (supporting code)
+// |  |  |__   |  |  | | | |  version 3.12.0
+// |_____|_____|_____|_|___|  https://github.com/nlohmann/json
+//
+// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
+// SPDX-License-Identifier: MIT
+
+#pragma once
+
+#include <cstddef> // size_t
+#include <cstdint> // uint8_t, uint32_t
+#include <limits> // numeric_limits
+#include <string> // string
+#include <vector> // vector
+
+#include <nlohmann/json.hpp>
+
+// helpers of the tests of the budget for the elements of UBJSON containers of
+// a valueless type (unit-ubjson.cpp and unit-ubjson_valueless.cpp)
+#if defined(__clang__)
+    #pragma clang diagnostic push
+    #pragma clang diagnostic ignored "-Wweak-vtables"
+#endif
+
+namespace utils
+{
+/// SAX consumer that counts the values and keeps the id of a parse error
+struct ValuelessEventCounter : public nlohmann::json_sax<nlohmann::json>
+{
+    bool null() override
+    {
+        return ++values < stop_after;
+    }
+    bool boolean(bool /*val*/) override
+    {
+        return ++values < stop_after;
+    }
+    bool number_integer(nlohmann::json::number_integer_t /*val*/) override
+    {
+        ++values;
+        return true;
+    }
+    bool number_unsigned(nlohmann::json::number_unsigned_t /*val*/) override
+    {
+        ++values;
+        return true;
+    }
+    bool number_float(nlohmann::json::number_float_t /*val*/, const std::string& /*s*/) override
+    {
+        ++values;
+        return true;
+    }
+    bool string(std::string& /*val*/) override
+    {
+        ++values;
+        return true;
+    }
+    bool binary(nlohmann::json::binary_t& /*val*/) override
+    {
+        ++values;
+        return true;
+    }
+    bool start_object(std::size_t /*elements*/) override
+    {
+        return true;
+    }
+    bool key(std::string& /*val*/) override
+    {
+        return true;
+    }
+    bool end_object() override
+    {
+        return true;
+    }
+    bool start_array(std::size_t /*elements*/) override
+    {
+        return true;
+    }
+    bool end_array() override
+    {
+        return true;
+    }
+    bool parse_error(std::size_t /*position*/, const std::string& /*last_token*/, const nlohmann::json::exception& ex) override
+    {
+        error_id = ex.id;
+        return false;
+    }
+
+    /// the handler refuses the value that makes this many
+    std::size_t stop_after = (std::numeric_limits<std::size_t>::max)();
+    std::size_t values = 0;
+    int error_id = 0;
+};
+
+/// appends `[$<marker>#l<count>` (nine bytes) to @a v
+inline void append_valueless_array(std::vector<std::uint8_t>& v, const char marker, const std::uint32_t count)
+{
+    v.push_back('[');
+    v.push_back('$');
+    v.push_back(static_cast<std::uint8_t>(marker));
+    v.push_back('#');
+    v.push_back('l');
+    v.push_back(static_cast<std::uint8_t>(count >> 24));
+    v.push_back(static_cast<std::uint8_t>(count >> 16));
+    v.push_back(static_cast<std::uint8_t>(count >> 8));
+    v.push_back(static_cast<std::uint8_t>(count));
+}
+
+/// `[` followed by one `[$<marker>#l<count>` per entry of @a counts and `]`
+inline std::vector<std::uint8_t> valueless_arrays(const char marker, const std::vector<std::uint32_t>& counts)
+{
+    std::vector<std::uint8_t> v = {'['};
+    for (const auto count : counts)
+    {
+        append_valueless_array(v, marker, count);
+    }
+    v.push_back(']');
+    return v;
+}
+} // namespace utils
+
+#if defined(__clang__)
+    #pragma clang diagnostic pop
+#endif

tests/src/unit-bjdata.cpp

@@ -31,6 +31,7 @@ using nlohmann::ordered_json;
 #include "make_test_data_available.hpp"
 #include "test_utils.hpp"
 #include "sax_countdown.hpp"
+#include "allocation_counter.hpp"
 #include "sax_event_loggers.hpp"
 using utils::SaxCountdown;
 using utils::SaxEventLogger;
@@ -4030,6 +4031,60 @@ TEST_CASE("issue #5405 - array reserve for definite-length BJData arrays")
     }
 }
 
+TEST_CASE("BJData - open arrays reserve only what the input has backed")
+{
+    // an array that announces its size gets storage for (up to 16384 of) its
+    // elements when it opens; the arrays that are open at the same time must
+    // not hold more than the input read so far can back
+    SECTION("nested headers that announce 16384 elements and are never completed")
+    {
+        // 200 levels would hold 200 * 16384 * sizeof(json),
+        // if each of them reserved what it announces
+        const auto input = utils::repeated({'[', '#', 'I', 0x00, 0x40}, 200);
+
+        utils::counts() = utils::allocation_counts{};
+        const auto j = utils::counted_json::from_bjdata(input, true, false);
+        CHECK(j.is_discarded());
+        CAPTURE(utils::counts().peak)
+        // the credit is 65,536 values; the base peaks at more than 3 million
+        CHECK(utils::counts().peak < sizeof(utils::counted_json) * 128 * 1024);
+    }
+
+    SECTION("a completed array is still reserved in one allocation")
+    {
+        json array = json::array();
+        for (int i = 0; i < 10000; ++i)
+        {
+            array.push_back(i % 100);
+        }
+
+        // an array without a size is not reserved, so only these two
+        {
+            const auto packed = json::to_bjdata(array, true);
+
+            utils::counts() = utils::allocation_counts{};
+            const auto j = utils::counted_json::from_bjdata(packed);
+            CHECK(j.size() == 10000);
+            // the array and its storage; growing the storage as the elements
+            // arrive would take more than a dozen allocations
+            CAPTURE(utils::counts().allocations)
+            CHECK(utils::counts().allocations <= 4);
+        }
+
+        {
+            const auto packed = json::to_bjdata(array, true, true);
+
+            utils::counts() = utils::allocation_counts{};
+            const auto j = utils::counted_json::from_bjdata(packed);
+            CHECK(j.size() == 10000);
+            // the array and its storage; growing the storage as the elements
+            // arrive would take more than a dozen allocations
+            CAPTURE(utils::counts().allocations)
+            CHECK(utils::counts().allocations <= 4);
+        }
+    }
+}
+
 TEST_CASE("Universal Binary JSON Specification Examples 1")
 {
     SECTION("Null Value")

tests/src/unit-cbor.cpp

@@ -29,6 +29,7 @@ using nlohmann::json;
 #include "round_trip_corpus.hpp"
 #include "test_utils.hpp"
 #include "sax_countdown.hpp"
+#include "allocation_counter.hpp"
 #include "sax_event_loggers.hpp"
 using utils::SaxCountdown;
 using utils::SaxEventLogger;
@@ -2709,6 +2710,321 @@ TEST_CASE("issue #5405 - array reserve for definite-length CBOR arrays")
     }
 }
 
+namespace
+{
+// NOLINTBEGIN(bugprone-derived-method-shadowing-base-method)
+using dom_parser = nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::input_stream_adapter>;
+
+// handlers that wrap a DOM parser, as users of the SAX interface do
+struct public_wrapper : dom_parser
+{
+    explicit public_wrapper(json& result) : dom_parser(result, false)
+    {}
+};
+
+/// parses another input with itself while the first parse is still running
+struct reentrant_wrapper : dom_parser
+{
+    explicit reentrant_wrapper(json& result) : dom_parser(result, false)
+    {}
+
+    bool start_array(std::size_t len)
+    {
+        if (!inner_done)
+        {
+            inner_done = true;
+            const std::vector<std::uint8_t> input = {0x80};
+            json::sax_parse(input, this, json::input_format_t::cbor);
+        }
+        return dom_parser::start_array(len);
+    }
+
+    bool inner_done = false;
+};
+
+// ICC does not detect SAX members that a using-declaration exposes from a private or protected base
+#if !defined(__INTEL_COMPILER)
+class private_wrapper : private dom_parser
+{
+  public:
+    explicit private_wrapper(json& result) : dom_parser(result, false)
+    {}
+
+    using dom_parser::null;
+    using dom_parser::boolean;
+    using dom_parser::number_integer;
+    using dom_parser::number_unsigned;
+    using dom_parser::number_float;
+    using dom_parser::string;
+    using dom_parser::binary;
+    using dom_parser::start_array;
+    using dom_parser::end_array;
+    using dom_parser::parse_error;
+
+    // the DOM parser has private overloads of these, which a using
+    // declaration cannot expose; the input of the test has no objects
+    static bool start_object(std::size_t /*len*/)
+    {
+        return false;
+    }
+
+    static bool key(std::string& /*val*/)
+    {
+        return false;
+    }
+
+    static bool end_object()
+    {
+        return false;
+    }
+};
+
+class protected_wrapper : protected dom_parser
+{
+  public:
+    explicit protected_wrapper(json& result) : dom_parser(result, false)
+    {}
+
+    using dom_parser::null;
+    using dom_parser::boolean;
+    using dom_parser::number_integer;
+    using dom_parser::number_unsigned;
+    using dom_parser::number_float;
+    using dom_parser::string;
+    using dom_parser::binary;
+    using dom_parser::start_array;
+    using dom_parser::end_array;
+    using dom_parser::parse_error;
+
+    // the DOM parser has private overloads of these, which a using
+    // declaration cannot expose; the input of the test has no objects
+    static bool start_object(std::size_t /*len*/)
+    {
+        return false;
+    }
+
+    static bool key(std::string& /*val*/)
+    {
+        return false;
+    }
+
+    static bool end_object()
+    {
+        return false;
+    }
+};
+#endif
+
+// NOLINTEND(bugprone-derived-method-shadowing-base-method)
+
+/// the capacity of the fifth array of five that are open
+std::size_t innermost_capacity(const json& result)
+{
+    const json* node = &result;
+    for (int level = 0; level < 4; ++level)
+    {
+        node = &node->front();
+    }
+    return node->get_ref<const json::array_t&>().capacity();
+}
+} // namespace
+
+TEST_CASE("CBOR - open arrays reserve only what the input has backed")
+{
+    // an array that announces its size gets storage for (up to 16384 of) its
+    // elements when it opens; the arrays that are open at the same time must
+    // not hold more than the input read so far can back
+    SECTION("nested headers that announce 16384 elements and are never completed")
+    {
+        // 200 levels would hold 200 * 16384 * sizeof(json),
+        // if each of them reserved what it announces
+        const auto input = utils::repeated({0x99, 0x40, 0x00}, 200);
+
+        utils::counts() = utils::allocation_counts{};
+        const auto j = utils::counted_json::from_cbor(input, true, false);
+        CHECK(j.is_discarded());
+        CAPTURE(utils::counts().peak)
+        // the credit is 65,536 values; the base peaks at more than 3 million
+        CHECK(utils::counts().peak < sizeof(utils::counted_json) * 128 * 1024);
+    }
+
+    SECTION("a completed array is still reserved in one allocation")
+    {
+        json array = json::array();
+        for (int i = 0; i < 10000; ++i)
+        {
+            array.push_back(i % 100);
+        }
+
+        {
+            const auto packed = json::to_cbor(array);
+
+            utils::counts() = utils::allocation_counts{};
+            const auto j = utils::counted_json::from_cbor(packed);
+            CHECK(j.size() == 10000);
+            // the array and its storage; growing the storage as the elements
+            // arrive would take more than a dozen allocations
+            CAPTURE(utils::counts().allocations)
+            CHECK(utils::counts().allocations <= 4);
+        }
+    }
+
+    SECTION("an array that is given less than it announces is still decoded")
+    {
+        // five levels that announce 16384 elements each, the first of which
+        // is the next level: they are more than the input could back when
+        // they open
+        constexpr std::size_t levels = 5;
+        std::vector<std::uint8_t> input;
+        for (std::size_t level = 0; level < levels; ++level)
+        {
+            input.insert(input.end(), {0x99, 0x40, 0x00});
+        }
+        input.insert(input.end(), 16384, 0x01);
+        for (std::size_t level = 1; level < levels; ++level)
+        {
+            input.insert(input.end(), 16383, 0x02);
+        }
+
+        const auto j = json::from_cbor(input);
+        const json* node = &j;
+        for (std::size_t level = 0; level < levels; ++level)
+        {
+            REQUIRE(node->is_array());
+            CHECK(node->size() == 16384);
+            node = &node->front();
+        }
+        CHECK(*node == 1);
+        CHECK(j.back() == 2);
+    }
+
+    SECTION("an array that announces more than 16384 elements is charged what it is reserved")
+    {
+        // an array of 20000 elements (header 9A 00 00 4E 20) and three nested
+        // arrays of 16384: the four reservations of 16384 add up to the credit
+        // of 65,536. Charged the 20000 it announces, the outer array would
+        // leave the innermost one less than it announces
+        std::vector<std::uint8_t> input = {0x9A, 0x00, 0x00, 0x4E, 0x20};
+        const auto nested = utils::repeated({0x99, 0x40, 0x00}, 3);
+        input.insert(input.end(), nested.begin(), nested.end());
+
+        json result;
+        dom_parser parser(result, false);
+        CHECK_FALSE(json::sax_parse(input, &parser, json::input_format_t::cbor));
+        const json* node = &result;
+        for (int level = 0; level < 4; ++level)
+        {
+            CAPTURE(level)
+            REQUIRE(node->is_array());
+            CHECK(node->get_ref<const json::array_t&>().capacity() == 16384);
+            if (level < 3)
+            {
+                REQUIRE(node->size() == 1);
+                node = &node->front();
+            }
+        }
+    }
+
+    SECTION("a DOM parser that a parse has finished with is not bounded by the input of that parse")
+    {
+        // the DOM parser learns where the reader counts its characters, and
+        // must forget it when the parse is over: a handler may be driven by
+        // other code afterwards, while the reader no longer exists
+        // five levels announce more than the credit of one input, which a
+        // parse of one byte has hardly refilled
+        const auto check_unbounded = [](dom_parser & parser, json & result)
+        {
+            for (int level = 0; level < 5; ++level)
+            {
+                CHECK(parser.start_array(16384));
+            }
+
+            const json* node = &result;
+            for (int level = 0; level < 4; ++level)
+            {
+                REQUIRE(node->is_array());
+                REQUIRE(node->size() == 1);
+                node = &node->front();
+            }
+            REQUIRE(node->is_array());
+            CHECK(node->get_ref<const json::array_t&>().capacity() == 16384);
+        };
+
+        SECTION("after a parse that succeeded")
+        {
+            json result;
+            dom_parser parser(result, false);
+            const std::vector<std::uint8_t> input = {0x80};
+            CHECK(json::sax_parse(input, &parser, json::input_format_t::cbor));
+            check_unbounded(parser, result);
+        }
+
+#if !defined(JSON_NOEXCEPTION)
+        SECTION("after a parse that threw")
+        {
+            json result;
+            dom_parser parser(result, true);
+            const std::vector<std::uint8_t> input = {0x1C}; // not a CBOR value
+            CHECK_THROWS_AS(json::sax_parse(input, &parser, json::input_format_t::cbor), json::parse_error&);
+            check_unbounded(parser, result);
+        }
+#endif
+    }
+
+    SECTION("a handler that wraps a DOM parser is bound if the reader can reach the parser")
+    {
+        // five arrays announce 16384 elements: the credit of one input covers
+        // four of them, and a parse of 15 bytes has hardly refilled it
+        const auto input = utils::repeated({0x99, 0x40, 0x00}, 5);
+
+        SECTION("public base")
+        {
+            json result;
+            public_wrapper handler(result);
+            CHECK_FALSE(json::sax_parse(input, &handler, json::input_format_t::cbor));
+            CHECK(innermost_capacity(result) < 16384);
+        }
+
+        SECTION("public base, a parse inside the parse")
+        {
+            json result;
+            reentrant_wrapper handler(result);
+            CHECK_FALSE(json::sax_parse(input, &handler, json::input_format_t::cbor));
+            CHECK(handler.inner_done);
+            CHECK(innermost_capacity(result) < 16384);
+        }
+
+#if !defined(__INTEL_COMPILER)
+        // the reader cannot convert these to the DOM parser: they are not bound
+        SECTION("private base")
+        {
+            json result;
+            private_wrapper handler(result);
+            CHECK_FALSE(json::sax_parse(input, &handler, json::input_format_t::cbor));
+            CHECK(innermost_capacity(result) == 16384);
+        }
+
+        SECTION("protected base")
+        {
+            json result;
+            protected_wrapper handler(result);
+            CHECK_FALSE(json::sax_parse(input, &handler, json::input_format_t::cbor));
+            CHECK(innermost_capacity(result) == 16384);
+        }
+#endif
+    }
+
+    SECTION("SAX events report the announced size")
+    {
+        SaxEventLogger logger;
+        const std::vector<std::uint8_t> input = {0x99, 0x40, 0x00};
+        CHECK_FALSE(json::sax_parse(input, &logger, json::input_format_t::cbor));
+        REQUIRE(logger.events.size() == 2);
+        CHECK(logger.events[0] == "start_array(16384)");
+        CHECK(logger.events[1] == "parse_error(4)");
+    }
+}
+
 #if !defined(JSON_NOEXCEPTION) // corpus values that do not survive the round trip are skipped by catching the exception
 TEST_CASE("CBOR round-trip invariants")
 {

tests/src/unit-msgpack.cpp

@@ -33,6 +33,7 @@ using nlohmann::json;
 #include "round_trip_corpus.hpp"
 #include "test_utils.hpp"
 #include "sax_countdown.hpp"
+#include "allocation_counter.hpp"
 #include "sax_event_loggers.hpp"
 using utils::SaxCountdown;
 using utils::SaxEventLogger;
@@ -1759,6 +1760,47 @@ TEST_CASE("issue #5405 - array reserve for definite-length MessagePack arrays")
     }
 }
 
+TEST_CASE("MessagePack - open arrays reserve only what the input has backed")
+{
+    // an array that announces its size gets storage for (up to 16384 of) its
+    // elements when it opens; the arrays that are open at the same time must
+    // not hold more than the input read so far can back
+    SECTION("nested headers that announce 16384 elements and are never completed")
+    {
+        // 200 levels would hold 200 * 16384 * sizeof(json),
+        // if each of them reserved what it announces
+        const auto input = utils::repeated({0xDC, 0x40, 0x00}, 200);
+
+        utils::counts() = utils::allocation_counts{};
+        const auto j = utils::counted_json::from_msgpack(input, true, false);
+        CHECK(j.is_discarded());
+        CAPTURE(utils::counts().peak)
+        // the credit is 65,536 values; the base peaks at more than 3 million
+        CHECK(utils::counts().peak < sizeof(utils::counted_json) * 128 * 1024);
+    }
+
+    SECTION("a completed array is still reserved in one allocation")
+    {
+        json array = json::array();
+        for (int i = 0; i < 10000; ++i)
+        {
+            array.push_back(i % 100);
+        }
+
+        {
+            const auto packed = json::to_msgpack(array);
+
+            utils::counts() = utils::allocation_counts{};
+            const auto j = utils::counted_json::from_msgpack(packed);
+            CHECK(j.size() == 10000);
+            // the array and its storage; growing the storage as the elements
+            // arrive would take more than a dozen allocations
+            CAPTURE(utils::counts().allocations)
+            CHECK(utils::counts().allocations <= 4);
+        }
+    }
+}
+
 TEST_CASE("regression test - MessagePack ext type rejects a subtype that doesn't fit a single byte")
 {
     // subtype 0-255 must still round-trip correctly (regression guard, pre-existing behavior)

tests/src/unit-ubjson.cpp

@@ -19,19 +19,17 @@
 #include <nlohmann/json.hpp>
 using nlohmann::json;
 
-#include <cstdio>
 #include <iostream>
-#include <sstream>
-#include <utility>
 #include <fstream>
-#include <limits>
 #include <list>
 #include <set>
 #include "make_test_data_available.hpp"
 #include "round_trip_corpus.hpp"
 #include "test_utils.hpp"
 #include "sax_countdown.hpp"
+#include "allocation_counter.hpp"
 #include "sax_event_loggers.hpp"
+#include "ubjson_valueless.hpp"
 using utils::SaxCountdown;
 using utils::SaxEventLogger;
 
@@ -2440,208 +2438,22 @@ TEST_CASE("UBJSON optimized arrays of a valueless type are bounded")
     }
 }
 
-namespace
+TEST_CASE("UBJSON the writer keeps the valueless elements of an input within the base")
 {
-/// SAX consumer that counts the values and keeps the id of a parse error
-struct ValuelessEventCounter : public nlohmann::json_sax<json>
-{
-    bool null() override
-    {
-        return ++values < stop_after;
-    }
-    bool boolean(bool /*val*/) override
-    {
-        return ++values < stop_after;
-    }
-    bool number_integer(json::number_integer_t /*val*/) override
-    {
-        ++values;
-        return true;
-    }
-    bool number_unsigned(json::number_unsigned_t /*val*/) override
-    {
-        ++values;
-        return true;
-    }
-    bool number_float(json::number_float_t /*val*/, const std::string& /*s*/) override
-    {
-        ++values;
-        return true;
-    }
-    bool string(std::string& /*val*/) override
-    {
-        ++values;
-        return true;
-    }
-    bool binary(json::binary_t& /*val*/) override
-    {
-        ++values;
-        return true;
-    }
-    bool start_object(std::size_t /*elements*/) override
-    {
-        return true;
-    }
-    bool key(std::string& /*val*/) override
-    {
-        return true;
-    }
-    bool end_object() override
-    {
-        return true;
-    }
-    bool start_array(std::size_t /*elements*/) override
-    {
-        return true;
-    }
-    bool end_array() override
-    {
-        return true;
-    }
-    bool parse_error(std::size_t /*position*/, const std::string& /*last_token*/, const json::exception& ex) override
-    {
-        error_id = ex.id;
-        return false;
-    }
-
-    /// the handler refuses the value that makes this many
-    std::size_t stop_after = (std::numeric_limits<std::size_t>::max)();
-    std::size_t values = 0;
-    int error_id = 0;
-};
-
-/// appends `[$<marker>#l<count>` (nine bytes) to @a v
-void append_valueless_array(std::vector<std::uint8_t>& v, const char marker, const std::uint32_t count)
-{
-    v.push_back('[');
-    v.push_back('$');
-    v.push_back(static_cast<std::uint8_t>(marker));
-    v.push_back('#');
-    v.push_back('l');
-    v.push_back(static_cast<std::uint8_t>(count >> 24));
-    v.push_back(static_cast<std::uint8_t>(count >> 16));
-    v.push_back(static_cast<std::uint8_t>(count >> 8));
-    v.push_back(static_cast<std::uint8_t>(count));
-}
-
-/// `[` followed by one `[$<marker>#l<count>` per entry of @a counts and `]`
-std::vector<std::uint8_t> valueless_arrays(const char marker, const std::vector<std::uint32_t>& counts)
-{
-    std::vector<std::uint8_t> v = {'['};
-    for (const auto count : counts)
-    {
-        append_valueless_array(v, marker, count);
-    }
-    v.push_back(']');
-    return v;
-}
-} // namespace
-
-TEST_CASE("UBJSON the valueless elements of a whole input are bounded")
-{
-    // The per-container limit alone let nine bytes (`[$Z#l` and a count of
-    // 1048576) announce a million values, and every further nine bytes another
-    // million. The elements of all containers of type 'Z', 'T' or 'F' in one
-    // input are now added up: they may not exceed 1048576 plus 4 for every
-    // byte read when the container's header has been read.
-    //
-    // The tests count events with a SAX handler where they can: building and
-    // comparing values of a million elements is what makes them slow under
-    // Valgrind, so there is one rejection and one round trip with a DOM.
+    // The budget of the reader is 1048576 plus a rate per byte read, see
+    // unit-ubjson_valueless.cpp, which tests it for several values of the rate.
+    // The writer does not use the rate and stays within the base
     json _;
-    const char* const budget_message = "[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive number of elements without payload";
-
-    // two arrays that exceed the limit: the second one, with its header read
-    // at byte 19, may hold 4 * 19 = 76 values on top of the base of 1048576
-    const std::uint32_t over_first = 524288;
-    const std::uint32_t over_second = 1048576 + 76 - over_first + 1;
-
-    SECTION("the input of the original report is rejected")
-    {
-        // 362 bytes (40 arrays of 1048576 values) used to allocate 661 MB; the
-        // second array is the one that is refused
-        const auto input = valueless_arrays('Z', {over_first, over_second});
-        CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), budget_message, json::out_of_range&);
-    }
-
-    SECTION("every input adapter applies the budget")
-    {
-        const auto input = valueless_arrays('Z', {over_first, over_second});
-
-        // stream; not strict, which does not matter
-        std::stringstream stream;
-        stream.write(reinterpret_cast<const char*>(input.data()), static_cast<std::streamsize>(input.size()));
-        ValuelessEventCounter from_stream;
-        CHECK(!json::sax_parse(stream, &from_stream, json::input_format_t::ubjson, false));
-        CHECK(from_stream.error_id == 408);
-
-        // a container that is not contiguous; the counts of arrays of
-        // different types are added up
-        std::vector<std::uint8_t> mixed = {'['};
-        append_valueless_array(mixed, 'T', over_first);
-        append_valueless_array(mixed, 'F', over_second);
-        mixed.push_back(']');
-        const std::list<std::uint8_t> as_list(mixed.begin(), mixed.end());
-        ValuelessEventCounter from_list;
-        CHECK(!json::sax_parse(as_list.begin(), as_list.end(), &from_list, json::input_format_t::ubjson));
-        CHECK(from_list.error_id == 408);
-        CHECK(from_list.values == over_first);
-
-        // FILE*
-        FILE* const file = std::tmpfile(); // NOLINT(cppcoreguidelines-owning-memory)
-        REQUIRE(file != nullptr);
-        CHECK(std::fwrite(input.data(), 1, input.size(), file) == input.size());
-        CHECK(std::fseek(file, 0, SEEK_SET) == 0);
-        ValuelessEventCounter from_file;
-        CHECK(!json::sax_parse(file, &from_file, json::input_format_t::ubjson));
-        CHECK(from_file.error_id == 408);
-        std::fclose(file); // NOLINT(cert-err33-c,cppcoreguidelines-owning-memory)
-    }
-
-    SECTION("the limit is exact and the bytes read raise it")
-    {
-        // [ [$Z#l 524288] [$U#l 4096 + 4096 bytes] [$Z#l count] ]: chars_read
-        // is 1 + 9 + (9 + 4096) + 9 = 4124 when the last header has been read,
-        // so the last array may hold 4 * 4124 = 16496 values on top of the
-        // base of 1048576 less the 524288 of the first array
-        const std::uint32_t last_allowed = 1048576 + 16496 - 524288;
-        for (const std::uint32_t count :
-                {
-                    last_allowed, last_allowed + 1
-                })
-        {
-            CAPTURE(count)
-            std::vector<std::uint8_t> input = {'['};
-            append_valueless_array(input, 'Z', 524288);
-            input.insert(input.end(), {'[', '$', 'U', '#', 'l', 0x00, 0x00, 0x10, 0x00});
-            input.insert(input.end(), 4096, 0x01);
-            append_valueless_array(input, 'Z', count);
-            input.push_back(']');
-            ValuelessEventCounter sax;
-            const bool accepted = count == last_allowed;
-            CHECK(json::sax_parse(input, &sax, json::input_format_t::ubjson) == accepted);
-            CHECK(sax.error_id == (accepted ? 0 : 408));
-            CHECK(sax.values == 524288u + 4096u + (accepted ? count : 0u));
-        }
-
-        // an array of 1048576 values on its own is accepted as before
-        const std::vector<std::uint8_t> single = {'[', '$', 'Z', '#', 'l', 0x00, 0x10, 0x00, 0x00};
-        // (the handler ends the parse at the first value, so that the million
-        // values are not read)
-        ValuelessEventCounter sax;
-        sax.stop_after = 1;
-        CHECK(!json::sax_parse(single, &sax, json::input_format_t::ubjson));
-        CHECK(sax.error_id == 0);
-        CHECK(sax.values == 1);
-    }
 
     SECTION("to_ubjson writes the typed form up to the base only")
     {
         // 1048 rows of 1000 values fit under the base of 1048576, the 1049th
         // would not: its elements are written one by one. Written all typed,
-        // 1100 rows of 6 bytes would be refused by a reader (1075 rows are the
-        // most it accepts). The rows hold null, true or false, and the
-        // fallback does not depend on which
+        // 1100 rows of 6 bytes would be refused by a reader with the default
+        // rate of the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE (1074 rows
+        // are the most it accepts); the writer does not follow the macro, since
+        // the limit of a reader is never below the base. The rows hold null,
+        // true or false, and the fallback does not depend on which
         std::vector<std::uint8_t> encoded;
         {
             json table = json::array();
@@ -2657,7 +2469,7 @@ TEST_CASE("UBJSON the valueless elements of a whole input are bounded")
         CHECK(encoded.size() == 7 + (1048 * 6) + (52 * (4 + 1000)));
 
         // the output is read back
-        ValuelessEventCounter sax;
+        utils::ValuelessEventCounter sax;
         CHECK(json::sax_parse(encoded, &sax, json::input_format_t::ubjson));
         CHECK(sax.error_id == 0);
         CHECK(sax.values == 1100u * 1000u);
@@ -2684,7 +2496,7 @@ TEST_CASE("UBJSON the valueless elements of a whole input are bounded")
         // an object member costs a key, so the count is bounded by the input
         // and is not charged to the budget
         const std::vector<std::uint8_t> input = {'{', '$', 'Z', '#', 'l', 0x00, 0x10, 0x00, 0x00};
-        ValuelessEventCounter sax;
+        utils::ValuelessEventCounter sax;
         CHECK(!json::sax_parse(input, &sax, json::input_format_t::ubjson));
         CHECK(sax.error_id == 110);
         CHECK(sax.values == 0);
@@ -2880,6 +2692,60 @@ TEST_CASE("issue #5405 - array reserve for definite-length UBJSON arrays")
     }
 }
 
+TEST_CASE("UBJSON - open arrays reserve only what the input has backed")
+{
+    // an array that announces its size gets storage for (up to 16384 of) its
+    // elements when it opens; the arrays that are open at the same time must
+    // not hold more than the input read so far can back
+    SECTION("nested headers that announce 16384 elements and are never completed")
+    {
+        // 200 levels would hold 200 * 16384 * sizeof(json),
+        // if each of them reserved what it announces
+        const auto input = utils::repeated({'[', '#', 'I', 0x40, 0x00}, 200);
+
+        utils::counts() = utils::allocation_counts{};
+        const auto j = utils::counted_json::from_ubjson(input, true, false);
+        CHECK(j.is_discarded());
+        CAPTURE(utils::counts().peak)
+        // the credit is 65,536 values; the base peaks at more than 3 million
+        CHECK(utils::counts().peak < sizeof(utils::counted_json) * 128 * 1024);
+    }
+
+    SECTION("a completed array is still reserved in one allocation")
+    {
+        json array = json::array();
+        for (int i = 0; i < 10000; ++i)
+        {
+            array.push_back(i % 100);
+        }
+
+        // an array without a size is not reserved, so only these two
+        {
+            const auto packed = json::to_ubjson(array, true);
+
+            utils::counts() = utils::allocation_counts{};
+            const auto j = utils::counted_json::from_ubjson(packed);
+            CHECK(j.size() == 10000);
+            // the array and its storage; growing the storage as the elements
+            // arrive would take more than a dozen allocations
+            CAPTURE(utils::counts().allocations)
+            CHECK(utils::counts().allocations <= 4);
+        }
+
+        {
+            const auto packed = json::to_ubjson(array, true, true);
+
+            utils::counts() = utils::allocation_counts{};
+            const auto j = utils::counted_json::from_ubjson(packed);
+            CHECK(j.size() == 10000);
+            // the array and its storage; growing the storage as the elements
+            // arrive would take more than a dozen allocations
+            CAPTURE(utils::counts().allocations)
+            CHECK(utils::counts().allocations <= 4);
+        }
+    }
+}
+
 
 TEST_CASE("Universal Binary JSON Specification Examples 1")
 {

tests/src/unit-ubjson_valueless.cpp

@@ -0,0 +1,324 @@
+//     __ _____ _____ _____
+//  __|  |   __|     |   | |  JSON for Modern C++ (supporting code)
+// |  |  |__   |  |  | | | |  version 3.12.0
+// |_____|_____|_____|_|___|  https://github.com/nlohmann/json
+//
+// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
+// SPDX-License-Identifier: MIT
+
+// The budget for the elements of the UBJSON containers of a valueless type
+// ('Z', 'T', 'F') of a whole input: 1048576 plus
+// JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE for every byte read (see
+// docs/mkdocs/docs/api/macros/json_ubjson_valueless_elements_per_byte.md).
+//
+// The file is built several times, with the macro defined to other values (see
+// tests/CMakeLists.txt), and every boundary below is computed from the macro.
+// What it can test depends on the value: with a rate of at most 10000 the
+// budget can be reached with the few bytes of a test, so the tests exceed it
+// at the exact byte; with a rate of 131072 or more the budget cannot be reached
+// by an input of any size (see the macro's page: 2^20 / 8 = 131072 elements per
+// byte is what the shortest header of a container of the largest size allows), and
+// the tests check that nothing is refused, in particular that the limit does
+// not wrap around for a rate near the largest integer. The input is read with
+// a SAX handler that counts the values, because a DOM of a million values is
+// what makes such a test slow under Valgrind.
+
+#include "doctest_compatibility.h"
+
+#include <nlohmann/json.hpp>
+using nlohmann::json;
+
+#include <cstddef> // size_t
+#include <cstdio> // FILE, tmpfile, fwrite, fseek, fclose
+#include <cstdint> // uint8_t, uint32_t
+#include <list>
+#include <sstream>
+#include <vector>
+
+#include "ubjson_valueless.hpp"
+using utils::ValuelessEventCounter;
+using utils::valueless_arrays;
+
+#ifndef JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE
+    #error "this test needs the macro: compile it with JSON_TEST_KEEP_MACROS (see tests/CMakeLists.txt)"
+#endif
+
+namespace
+{
+/// the base of the budget and the limit of a single container: both 2^20
+constexpr std::uint32_t budget_base = 1048576;
+
+#if JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE <= 10000 || JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE >= 131072
+/// what version 3.12.0 wrote for a table of @a rows (at most 32767) rows of
+/// @a width nulls with to_ubjson(table, true, true): `[$[#I` and the count of
+/// the rows (7 bytes), then a row as `$Z#` and its count (`U` and 1 byte up to
+/// 255, `I` and 2 bytes up to 32767, `l` and 4 bytes above): 5, 6 or 8 bytes
+std::vector<std::uint8_t> rows_3_12_0(const std::uint32_t rows, const std::uint32_t width)
+{
+    std::vector<std::uint8_t> v = {'[', '$', '[', '#', 'I', static_cast<std::uint8_t>(rows >> 8), static_cast<std::uint8_t>(rows)};
+    for (std::uint32_t row = 0; row < rows; ++row)
+    {
+        v.push_back('$');
+        v.push_back('Z');
+        v.push_back('#');
+        if (width <= 255)
+        {
+            v.push_back('U');
+        }
+        else if (width <= 32767)
+        {
+            v.push_back('I');
+            v.push_back(static_cast<std::uint8_t>(width >> 8));
+        }
+        else
+        {
+            v.push_back('l');
+            v.push_back(static_cast<std::uint8_t>(width >> 24));
+            v.push_back(static_cast<std::uint8_t>(width >> 16));
+            v.push_back(static_cast<std::uint8_t>(width >> 8));
+        }
+        v.push_back(static_cast<std::uint8_t>(width));
+    }
+    return v;
+}
+#endif
+} // namespace
+
+// the macro must not be a character or bool: every character type is refused
+static_assert(nlohmann::detail::valueless_elements_rate_is_char_or_bool<bool>::value, "bool is refused");
+static_assert(nlohmann::detail::valueless_elements_rate_is_char_or_bool<char>::value, "char is refused");
+static_assert(nlohmann::detail::valueless_elements_rate_is_char_or_bool<signed char>::value, "signed char is refused");
+static_assert(nlohmann::detail::valueless_elements_rate_is_char_or_bool<unsigned char>::value, "unsigned char is refused");
+static_assert(nlohmann::detail::valueless_elements_rate_is_char_or_bool<wchar_t>::value, "wchar_t is refused");
+static_assert(nlohmann::detail::valueless_elements_rate_is_char_or_bool<char16_t>::value, "char16_t is refused");
+static_assert(nlohmann::detail::valueless_elements_rate_is_char_or_bool<char32_t>::value, "char32_t is refused");
+#if defined(__cpp_char8_t)
+    static_assert(nlohmann::detail::valueless_elements_rate_is_char_or_bool<char8_t>::value, "char8_t is refused");
+#endif
+static_assert(!nlohmann::detail::valueless_elements_rate_is_char_or_bool<int>::value, "int is accepted");
+static_assert(!nlohmann::detail::valueless_elements_rate_is_char_or_bool<unsigned long long>::value, "unsigned long long is accepted");
+
+TEST_CASE("UBJSON the limit of a single container does not depend on the macro")
+{
+#if !defined(JSON_NOEXCEPTION)
+    json _;
+
+    SECTION("a container of more than 1048576 elements is refused")
+    {
+        // the first check of every container; its message differs from that of
+        // the budget, so that it is clear which limit was exceeded
+        for (const char marker :
+                {'Z', 'T', 'F'
+                })
+        {
+            const auto input = valueless_arrays(marker, {budget_base + 1});
+            CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), "[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive array size", json::out_of_range&);
+        }
+    }
+#endif
+
+    SECTION("a container of 1048576 elements is accepted at the start of an input")
+    {
+        // whatever the rate: the base is the per-container limit. The handler
+        // ends the parse at the first value, so that the million values are
+        // not read
+        const std::vector<std::uint8_t> single = {'[', '$', 'Z', '#', 'l', 0x00, 0x10, 0x00, 0x00};
+        ValuelessEventCounter sax;
+        sax.stop_after = 1;
+        CHECK(!json::sax_parse(single, &sax, json::input_format_t::ubjson));
+        CHECK(sax.error_id == 0);
+        CHECK(sax.values == 1);
+    }
+}
+
+#if JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE <= 10000
+
+namespace
+{
+/// the rate of the build; at most 10000 here, which the boundaries rely on
+constexpr std::uint32_t rate = JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE;
+
+/// elements of the first of two containers; the second one is then refused at
+/// its header, which is read at byte 1 + 9 + 9 = 19
+constexpr std::uint32_t first_count = 524288;
+
+/// the largest count the second container may have at byte 19: base plus rate
+/// for each of the 19 bytes, less what the first one used. For a rate of at
+/// most 27594, the largest that keeps this within the limit of one container
+constexpr std::uint32_t second_allowed = budget_base - first_count + (19 * rate);
+} // namespace
+
+TEST_CASE("UBJSON the valueless elements of a whole input are bounded")
+{
+    // The per-container limit alone let nine bytes (`[$Z#l` and a count of
+    // 1048576) announce a million values, and every further nine bytes another
+    // million. The elements of all containers of type 'Z', 'T' or 'F' in one
+    // input are added up: they may not exceed 1048576 plus
+    // JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE for every byte read when the
+    // container's header has been read.
+#if !defined(JSON_NOEXCEPTION)
+    json _;
+    const char* const budget_message = "[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive number of elements without payload";
+#endif
+
+    SECTION("the input of the original report is rejected")
+    {
+        // the second array is the first one that is refused, by one element
+#if !defined(JSON_NOEXCEPTION)
+        CHECK_THROWS_WITH_AS(_ = json::from_ubjson(valueless_arrays('Z', {first_count, second_allowed + 1})), budget_message, json::out_of_range&);
+#endif
+
+        // and is accepted with one element less
+        ValuelessEventCounter sax;
+        CHECK(json::sax_parse(valueless_arrays('Z', {first_count, second_allowed}), &sax, json::input_format_t::ubjson));
+        CHECK(sax.values == first_count + second_allowed);
+    }
+
+    SECTION("every input adapter applies the budget")
+    {
+        const auto input = valueless_arrays('Z', {first_count, second_allowed + 1});
+
+        // stream; not strict, which does not matter
+        std::stringstream stream;
+        stream.write(reinterpret_cast<const char*>(input.data()), static_cast<std::streamsize>(input.size()));
+        ValuelessEventCounter from_stream;
+        CHECK(!json::sax_parse(stream, &from_stream, json::input_format_t::ubjson, false));
+        CHECK(from_stream.error_id == 408);
+
+        // a container that is not contiguous; the counts of arrays of
+        // different types are added up
+        std::vector<std::uint8_t> mixed = {'['};
+        utils::append_valueless_array(mixed, 'T', first_count);
+        utils::append_valueless_array(mixed, 'F', second_allowed + 1);
+        mixed.push_back(']');
+        const std::list<std::uint8_t> as_list(mixed.begin(), mixed.end());
+        ValuelessEventCounter from_list;
+        CHECK(!json::sax_parse(as_list.begin(), as_list.end(), &from_list, json::input_format_t::ubjson));
+        CHECK(from_list.error_id == 408);
+        CHECK(from_list.values == first_count);
+
+        // FILE*
+        FILE* const file = std::tmpfile(); // NOLINT(cppcoreguidelines-owning-memory)
+        REQUIRE(file != nullptr);
+        CHECK(std::fwrite(input.data(), 1, input.size(), file) == input.size());
+        CHECK(std::fseek(file, 0, SEEK_SET) == 0);
+        ValuelessEventCounter from_file;
+        CHECK(!json::sax_parse(file, &from_file, json::input_format_t::ubjson));
+        CHECK(from_file.error_id == 408);
+        std::fclose(file); // NOLINT(cert-err33-c,cppcoreguidelines-owning-memory)
+    }
+
+    SECTION("the limit is exact and the bytes read raise it")
+    {
+        // [ [$Z#l first_count] [$U#l payload + payload bytes] [$Z#l count] ]:
+        // chars_read is 1 + 9 + (9 + payload) + 9 = 28 + payload when the last
+        // header has been read, so the last array may hold
+        // rate * (28 + payload) values on top of the base less the
+        // first_count of the first array. The payload is as large as the
+        // limit of one container allows, but at most 4096
+        constexpr std::uint32_t payload_limit = (524288 / rate) - 40;
+        constexpr std::uint32_t payload = payload_limit < 4096 ? payload_limit : 4096;
+        constexpr std::uint32_t last_allowed = budget_base - first_count + (rate * (28 + payload));
+        static_assert(last_allowed < budget_base, "the count must be accepted by the limit of one container");
+
+        for (const std::uint32_t count :
+                {
+                    last_allowed, last_allowed + 1
+                })
+        {
+            CAPTURE(count)
+            std::vector<std::uint8_t> input = {'['};
+            utils::append_valueless_array(input, 'Z', first_count);
+            input.insert(input.end(), {'[', '$', 'U', '#', 'l',
+                                       static_cast<std::uint8_t>(payload >> 24), static_cast<std::uint8_t>(payload >> 16),
+                                       static_cast<std::uint8_t>(payload >> 8), static_cast<std::uint8_t>(payload)
+                                      });
+            input.insert(input.end(), payload, 0x01);
+            utils::append_valueless_array(input, 'Z', count);
+            input.push_back(']');
+            ValuelessEventCounter sax;
+            const bool accepted = count == last_allowed;
+            CHECK(json::sax_parse(input, &sax, json::input_format_t::ubjson) == accepted);
+            CHECK(sax.error_id == (accepted ? 0 : 408));
+            CHECK(sax.values == first_count + payload + (accepted ? count : 0u));
+        }
+    }
+
+    SECTION("a table written by 3.12.0")
+    {
+        // Version 3.12.0 wrote a table of rows of nulls with one `$Z#` header
+        // per row (see the macro's page). A row of `width` elements has a
+        // header of 6 bytes up to 32767 elements (5 up to 255), so the budget
+        // grows by 6 * rate per row, and the row brings `width`. The 7 bytes
+        // of the outer header are read first. Row r is accepted while
+        //     r * width <= 1048576 + rate * (7 + 6 * r),
+        // which is r <= (1048576 + 7 * rate) / (width - 6 * rate) for a width
+        // above 6 * rate, and always true otherwise.
+        constexpr std::uint32_t width = 1000;
+        constexpr std::uint32_t rows = 1200;
+        constexpr std::uint32_t credit = 6 * rate;
+        constexpr std::uint32_t accepted_rows = width > credit ? (budget_base + (7 * rate)) / (width - credit) : rows;
+        constexpr std::uint32_t read_rows = accepted_rows < rows ? accepted_rows : rows;
+
+        ValuelessEventCounter sax;
+        CHECK(json::sax_parse(rows_3_12_0(rows, width), &sax, json::input_format_t::ubjson) == (accepted_rows >= rows));
+        CHECK(sax.error_id == (accepted_rows >= rows ? 0 : 408));
+        CHECK(sax.values == static_cast<std::size_t>(read_rows) * width);
+    }
+}
+
+#elif JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE >= 131072
+
+TEST_CASE("UBJSON a rate that cannot be reached switches the budget off")
+{
+    // The rate is so large (2^20 / 8 = 131072 elements per byte or more)
+    // that no input can exceed the budget: the shortest header of a container
+    // of the largest size is 8 bytes (`$Z#l` and a count of 4 bytes, in an
+    // array of arrays). The limit is base + rate * bytes read, which wraps
+    // around for a rate near the largest integer in 64 and in 32
+    // bits already at the first byte, unless it saturates: a wrapped limit
+    // would refuse the second container below (at byte 19 or at byte 10, where
+    // a limit of 2^20 + 10 * rate wraps for a 32-bit rate of 0x7FFFFFFF).
+#if !defined(JSON_NOEXCEPTION)
+    json _;
+#endif
+
+    SECTION("containers of the largest size are accepted one after the other")
+    {
+        // the first container is accepted by the base alone, the second one is
+        // not: it needs the rate. The handler ends the parse at its first
+        // value, so that the second million values are not read
+        const auto input = valueless_arrays('Z', {budget_base, budget_base});
+        ValuelessEventCounter sax;
+        sax.stop_after = static_cast<std::size_t>(budget_base) + 1;
+        CHECK(!json::sax_parse(input, &sax, json::input_format_t::ubjson));
+        CHECK(sax.error_id == 0);
+        CHECK(sax.values == static_cast<std::size_t>(budget_base) + 1);
+    }
+
+    SECTION("the limit of a single container still applies")
+    {
+        const auto input = valueless_arrays('T', {budget_base, budget_base + 1});
+        ValuelessEventCounter sax;
+        CHECK(!json::sax_parse(input, &sax, json::input_format_t::ubjson));
+        CHECK(sax.error_id == 408);
+        CHECK(sax.values == budget_base);
+#if !defined(JSON_NOEXCEPTION)
+        CHECK_THROWS_WITH_AS(_ = json::from_ubjson(valueless_arrays('F', {budget_base + 1})), "[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive array size", json::out_of_range&);
+#endif
+    }
+
+    SECTION("a table written by 3.12.0 is read completely")
+    {
+        // 1200 rows of 1000 values are refused after 1074 rows at the default
+        // rate and are accepted with every rate of 167 and more
+        ValuelessEventCounter sax;
+        CHECK(json::sax_parse(rows_3_12_0(1200, 1000), &sax, json::input_format_t::ubjson));
+        CHECK(sax.error_id == 0);
+        CHECK(sax.values == 1200u * 1000u);
+    }
+}
+
+// (rates from 10001 to 131071 can be reached, but not with the few bytes of a
+// test: there is nothing to test for them here)
+#endif