Bound what the DOM parsers reserve for arrays that announce a size

Cause: #5476 made the DOM parsers reserve storage (up to 16384 elements)
for a counted array of CBOR, MessagePack, UBJSON and BJData when it
opens, before any input backs the count. The arrays that are open at the
same time keep their storage until the parse ends, so N nested headers
that announce 16384 elements and are never completed hold
N * 16384 * sizeof(json): 100,000 nested CBOR headers `99 40 00`
(300,000 bytes) peaked at 1.65 GiB, 3.12.0 at 12 MiB; the same for
MessagePack `DC 40 00`, UBJSON `[#I 16384` and BJData.

Fix: json_sax_dom_error_state, the base of the three DOM parsers, keeps a
credit of elements that may still be reserved: 65,536 at the start plus
2 for every byte the reader has read. An array that announces no more
than the credit left is reserved as before (one comparison and one
subtraction per counted array); a larger one is reserved what is left
and grows as its elements arrive. binary_reader tells a DOM parser (a
handler whose pointer converts to json_sax_dom_error_state*; a handler
that derives from one privately or protectedly, or any other SAX handler,
is not told anything and reserves as in the base of this change) where to
find its count of characters read for the duration of sax_parse(). An
RAII guard puts back the previous binding on every exit, also by
exception, so the parser never keeps a pointer to a reader that is gone
(unbound, only the per-array cap of 16384 applies); the public SAX
interface is unchanged, and start_array() still reports the announced number. A
bound per array by the remaining input would not help, since the same
bytes would back every open level; the bound has to be over all arrays.

The elements of an array take at least one byte of input each, so the
credit is more than honest input needs: none of the 19 inputs measured
(canada, citm_catalog, twitter, mesh, update-center, random, numbers,
animation, ik_rig, instruments, geo_response, jeopardy and 7 synthetic
ones, each as CBOR, MessagePack, UBJSON and BJData, with and without
size/type optimization) was given less than it announced. A value that
nests five or more arrays of 16384 elements, each with an array as its
first element, is the exception: the later ones grow as before.

Measurements (clang -O2, macOS arm64, peak memory footprint of
`/usr/bin/time -l`, reproducer: n nested headers):

              n = 20,000                n = 100,000
              develop   fix   3.12.0   develop    fix   3.12.0
  CBOR        338.7    5.9    8.2      1687.7    23.3   11.8   MiB
  MessagePack 338.9    6.0    7.3      1687.9    23.8   11.9   MiB
  UBJSON      339.0    7.6    8.0      1687.9    30.1   11.6   MiB
  BJData      339.0    7.3    7.9      1688.0    30.8   11.6   MiB

That is at most (65,536 + 2 per input byte) values: 1 MiB plus 32 bytes
per input byte with 16-byte values (a value is 24 bytes with
JSON_DIAGNOSTICS and up to 40 with JSON_DIAGNOSTIC_POSITIONS), as
documented in untrusted_input.md. Lower fixed caps do not get there: a
cap of 64 elements still peaks at 113 MiB (n = 100,000) and costs 4.7 %
on arrays of 300 elements (mat300.cbor) and 5.0 % on arrays of 10,000
(arrays_of_big.cbor).

Reader instructions retired (min of 7, 108 rows: 18 inputs in the 6 forms
CBOR, MessagePack, UBJSON and BJData plain and optimized; jeopardy, 46 MB,
is within 0.1 %), compared with the base of this change: -0.19 % to +0.37 % in the formats that announce
sizes (canada.cbor 81.9 M -> 82.1 M, many_small.cbor 333.5 M -> 334.2 M);
the benefit of #5476 is kept (without any reserve canada.cbor takes
116.8 M, +42.6 %, many_small.cbor +71.5 %, mat300.cbor +81.6 %). Rows of
plain UBJSON and BJData, whose arrays announce no size, move by -2.7 % to
+2.6 % without any code of theirs changing: the base alternates
between two instruction counts per process (452 M and 463 M for
many_pairs.ubjson, equal in cycles), and a struct with three unused
members added to the same class reproduces the 474 M mode.

Tests (CBOR, MessagePack, UBJSON, BJData): 200 nested headers that
announce 16384 elements peak at 52,433,600 bytes of DOM allocations on
the base and below 128 Ki values with the fix (scaled by sizeof(value),
so that it holds with the diagnostic macros) (counting allocator, no
process memory measured); an array that completes is still reserved in
one allocation; CBOR also checks that a DOM parser is not bound to a finished
parse (ASan: stack-use-after-scope without the guard), that wrappers
derived publicly (also with a parse inside the parse), privately and
protectedly compile and are bound or not as described, that an array that got less than it
announced decodes and that a SAX handler is told the announced size.
The new tests take 4.6 s under Valgrind at -O0 (gcc:12, all four files).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

tests/src/allocation_counter.hpp

@@ -0,0 +1,86 @@
+//     __ _____ _____ _____
+//  __|  |   __|     |   | |  JSON for Modern C++ (supporting code)
+// |  |  |__   |  |  | | | |  version 3.12.0
+// |_____|_____|_____|_|___|  https://github.com/nlohmann/json
+//
+// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
+// SPDX-License-Identifier: MIT
+
+#pragma once
+
+#include <algorithm> // max
+#include <cstddef> // size_t
+#include <cstdint> // uint8_t
+#include <memory> // allocator
+#include <vector> // vector
+
+#include <nlohmann/json.hpp>
+
+namespace utils
+{
+/// the allocations of the values of a JSON type that uses counting_allocator
+struct allocation_counts
+{
+    /// bytes allocated and not yet deallocated
+    std::size_t current = 0;
+    /// the largest value current had
+    std::size_t peak = 0;
+    /// number of allocations
+    std::size_t allocations = 0;
+};
+
+/// the counts of all counting_allocators; the tests are single-threaded
+inline allocation_counts& counts()
+{
+    static allocation_counts c; // NOLINT(misc-const-correctness)
+    return c;
+}
+
+/// an allocator that counts the bytes and the number of allocations of a
+/// JSON type, so that a test can check how much memory a parse holds at any
+/// time without measuring the memory of the process
+template<class T>
+struct counting_allocator : std::allocator<T>
+{
+    using std::allocator<T>::allocator;
+
+    counting_allocator() = default;
+    template<class U> counting_allocator(const counting_allocator<U>& /*unused*/) noexcept {} // NOLINT(google-explicit-constructor,hicpp-explicit-conversions)
+
+    template <class U>
+    struct rebind
+    {
+        using other = counting_allocator<U>;
+    };
+
+    T* allocate(const std::size_t n)
+    {
+        T* const result = std::allocator<T>::allocate(n);
+        allocation_counts& c = counts();
+        c.current += n * sizeof(T);
+        c.peak = (std::max)(c.peak, c.current);
+        ++c.allocations;
+        return result;
+    }
+
+    void deallocate(T* p, const std::size_t n)
+    {
+        counts().current -= n * sizeof(T);
+        std::allocator<T>::deallocate(p, n);
+    }
+};
+
+using counted_json = nlohmann::json::with_allocator_t<counting_allocator>;
+
+/// @a count times the bytes of @a unit
+inline std::vector<std::uint8_t> repeated(const std::vector<std::uint8_t>& unit, const std::size_t count)
+{
+    std::vector<std::uint8_t> result;
+    result.reserve(unit.size() * count);
+    for (std::size_t i = 0; i < count; ++i)
+    {
+        result.insert(result.end(), unit.begin(), unit.end());
+    }
+    return result;
+}
+} // namespace utils

tests/src/unit-bjdata.cpp

@@ -31,6 +31,7 @@ using nlohmann::ordered_json;
 #include "make_test_data_available.hpp"
 #include "test_utils.hpp"
 #include "sax_countdown.hpp"
+#include "allocation_counter.hpp"
 #include "sax_event_loggers.hpp"
 using utils::SaxCountdown;
 using utils::SaxEventLogger;
@@ -4030,6 +4031,60 @@ TEST_CASE("issue #5405 - array reserve for definite-length BJData arrays")
     }
 }
 
+TEST_CASE("BJData - open arrays reserve only what the input has backed")
+{
+    // an array that announces its size gets storage for (up to 16384 of) its
+    // elements when it opens; the arrays that are open at the same time must
+    // not hold more than the input read so far can back
+    SECTION("nested headers that announce 16384 elements and are never completed")
+    {
+        // 200 levels would hold 200 * 16384 * sizeof(json),
+        // if each of them reserved what it announces
+        const auto input = utils::repeated({'[', '#', 'I', 0x00, 0x40}, 200);
+
+        utils::counts() = utils::allocation_counts{};
+        const auto j = utils::counted_json::from_bjdata(input, true, false);
+        CHECK(j.is_discarded());
+        CAPTURE(utils::counts().peak)
+        // the credit is 65,536 values; the base peaks at more than 3 million
+        CHECK(utils::counts().peak < sizeof(utils::counted_json) * 128 * 1024);
+    }
+
+    SECTION("a completed array is still reserved in one allocation")
+    {
+        json array = json::array();
+        for (int i = 0; i < 10000; ++i)
+        {
+            array.push_back(i % 100);
+        }
+
+        // an array without a size is not reserved, so only these two
+        {
+            const auto packed = json::to_bjdata(array, true);
+
+            utils::counts() = utils::allocation_counts{};
+            const auto j = utils::counted_json::from_bjdata(packed);
+            CHECK(j.size() == 10000);
+            // the array and its storage; growing the storage as the elements
+            // arrive would take more than a dozen allocations
+            CAPTURE(utils::counts().allocations)
+            CHECK(utils::counts().allocations <= 4);
+        }
+
+        {
+            const auto packed = json::to_bjdata(array, true, true);
+
+            utils::counts() = utils::allocation_counts{};
+            const auto j = utils::counted_json::from_bjdata(packed);
+            CHECK(j.size() == 10000);
+            // the array and its storage; growing the storage as the elements
+            // arrive would take more than a dozen allocations
+            CAPTURE(utils::counts().allocations)
+            CHECK(utils::counts().allocations <= 4);
+        }
+    }
+}
+
 TEST_CASE("Universal Binary JSON Specification Examples 1")
 {
     SECTION("Null Value")

tests/src/unit-cbor.cpp

@@ -29,6 +29,7 @@ using nlohmann::json;
 #include "round_trip_corpus.hpp"
 #include "test_utils.hpp"
 #include "sax_countdown.hpp"
+#include "allocation_counter.hpp"
 #include "sax_event_loggers.hpp"
 using utils::SaxCountdown;
 using utils::SaxEventLogger;
@@ -2709,6 +2710,289 @@ TEST_CASE("issue #5405 - array reserve for definite-length CBOR arrays")
     }
 }
 
+namespace
+{
+// NOLINTBEGIN(bugprone-derived-method-shadowing-base-method)
+using dom_parser = nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::input_stream_adapter>;
+
+// handlers that wrap a DOM parser, as users of the SAX interface do
+struct public_wrapper : dom_parser
+{
+    explicit public_wrapper(json& result) : dom_parser(result, false)
+    {}
+};
+
+/// parses another input with itself while the first parse is still running
+struct reentrant_wrapper : dom_parser
+{
+    explicit reentrant_wrapper(json& result) : dom_parser(result, false)
+    {}
+
+    bool start_array(std::size_t len)
+    {
+        if (!inner_done)
+        {
+            inner_done = true;
+            const std::vector<std::uint8_t> input = {0x80};
+            json::sax_parse(input, this, json::input_format_t::cbor);
+        }
+        return dom_parser::start_array(len);
+    }
+
+    bool inner_done = false;
+};
+
+class private_wrapper : private dom_parser
+{
+  public:
+    explicit private_wrapper(json& result) : dom_parser(result, false)
+    {}
+
+    using dom_parser::null;
+    using dom_parser::boolean;
+    using dom_parser::number_integer;
+    using dom_parser::number_unsigned;
+    using dom_parser::number_float;
+    using dom_parser::string;
+    using dom_parser::binary;
+    using dom_parser::start_array;
+    using dom_parser::end_array;
+    using dom_parser::parse_error;
+
+    // the DOM parser has private overloads of these, which a using
+    // declaration cannot expose; the input of the test has no objects
+    static bool start_object(std::size_t /*len*/)
+    {
+        return false;
+    }
+
+    static bool key(std::string& /*val*/)
+    {
+        return false;
+    }
+
+    static bool end_object()
+    {
+        return false;
+    }
+};
+
+class protected_wrapper : protected dom_parser
+{
+  public:
+    explicit protected_wrapper(json& result) : dom_parser(result, false)
+    {}
+
+    using dom_parser::null;
+    using dom_parser::boolean;
+    using dom_parser::number_integer;
+    using dom_parser::number_unsigned;
+    using dom_parser::number_float;
+    using dom_parser::string;
+    using dom_parser::binary;
+    using dom_parser::start_array;
+    using dom_parser::end_array;
+    using dom_parser::parse_error;
+
+    // the DOM parser has private overloads of these, which a using
+    // declaration cannot expose; the input of the test has no objects
+    static bool start_object(std::size_t /*len*/)
+    {
+        return false;
+    }
+
+    static bool key(std::string& /*val*/)
+    {
+        return false;
+    }
+
+    static bool end_object()
+    {
+        return false;
+    }
+};
+
+// NOLINTEND(bugprone-derived-method-shadowing-base-method)
+
+/// the capacity of the fifth array of five that are open
+std::size_t innermost_capacity(const json& result)
+{
+    const json* node = &result;
+    for (int level = 0; level < 4; ++level)
+    {
+        node = &node->front();
+    }
+    return node->get_ref<const json::array_t&>().capacity();
+}
+} // namespace
+
+TEST_CASE("CBOR - open arrays reserve only what the input has backed")
+{
+    // an array that announces its size gets storage for (up to 16384 of) its
+    // elements when it opens; the arrays that are open at the same time must
+    // not hold more than the input read so far can back
+    SECTION("nested headers that announce 16384 elements and are never completed")
+    {
+        // 200 levels would hold 200 * 16384 * sizeof(json),
+        // if each of them reserved what it announces
+        const auto input = utils::repeated({0x99, 0x40, 0x00}, 200);
+
+        utils::counts() = utils::allocation_counts{};
+        const auto j = utils::counted_json::from_cbor(input, true, false);
+        CHECK(j.is_discarded());
+        CAPTURE(utils::counts().peak)
+        // the credit is 65,536 values; the base peaks at more than 3 million
+        CHECK(utils::counts().peak < sizeof(utils::counted_json) * 128 * 1024);
+    }
+
+    SECTION("a completed array is still reserved in one allocation")
+    {
+        json array = json::array();
+        for (int i = 0; i < 10000; ++i)
+        {
+            array.push_back(i % 100);
+        }
+
+        {
+            const auto packed = json::to_cbor(array);
+
+            utils::counts() = utils::allocation_counts{};
+            const auto j = utils::counted_json::from_cbor(packed);
+            CHECK(j.size() == 10000);
+            // the array and its storage; growing the storage as the elements
+            // arrive would take more than a dozen allocations
+            CAPTURE(utils::counts().allocations)
+            CHECK(utils::counts().allocations <= 4);
+        }
+    }
+
+    SECTION("an array that is given less than it announces is still decoded")
+    {
+        // five levels that announce 16384 elements each, the first of which
+        // is the next level: they are more than the input could back when
+        // they open
+        constexpr std::size_t levels = 5;
+        std::vector<std::uint8_t> input;
+        for (std::size_t level = 0; level < levels; ++level)
+        {
+            input.insert(input.end(), {0x99, 0x40, 0x00});
+        }
+        input.insert(input.end(), 16384, 0x01);
+        for (std::size_t level = 1; level < levels; ++level)
+        {
+            input.insert(input.end(), 16383, 0x02);
+        }
+
+        const auto j = json::from_cbor(input);
+        const json* node = &j;
+        for (std::size_t level = 0; level < levels; ++level)
+        {
+            REQUIRE(node->is_array());
+            CHECK(node->size() == 16384);
+            node = &node->front();
+        }
+        CHECK(*node == 1);
+        CHECK(j.back() == 2);
+    }
+
+    SECTION("a DOM parser that a parse has finished with is not bounded by the input of that parse")
+    {
+        // the DOM parser learns where the reader counts its characters, and
+        // must forget it when the parse is over: a handler may be driven by
+        // other code afterwards, while the reader no longer exists
+        // five levels announce more than the credit of one input, which a
+        // parse of one byte has hardly refilled
+        const auto check_unbounded = [](dom_parser & parser, json & result)
+        {
+            for (int level = 0; level < 5; ++level)
+            {
+                CHECK(parser.start_array(16384));
+            }
+
+            const json* node = &result;
+            for (int level = 0; level < 4; ++level)
+            {
+                REQUIRE(node->is_array());
+                REQUIRE(node->size() == 1);
+                node = &node->front();
+            }
+            REQUIRE(node->is_array());
+            CHECK(node->get_ref<const json::array_t&>().capacity() == 16384);
+        };
+
+        SECTION("after a parse that succeeded")
+        {
+            json result;
+            dom_parser parser(result, false);
+            const std::vector<std::uint8_t> input = {0x80};
+            CHECK(json::sax_parse(input, &parser, json::input_format_t::cbor));
+            check_unbounded(parser, result);
+        }
+
+#if !defined(JSON_NOEXCEPTION)
+        SECTION("after a parse that threw")
+        {
+            json result;
+            dom_parser parser(result, true);
+            const std::vector<std::uint8_t> input = {0x1C}; // not a CBOR value
+            CHECK_THROWS_AS(json::sax_parse(input, &parser, json::input_format_t::cbor), json::parse_error&);
+            check_unbounded(parser, result);
+        }
+#endif
+    }
+
+    SECTION("a handler that wraps a DOM parser is bound if the reader can reach the parser")
+    {
+        // five arrays announce 16384 elements: the credit of one input covers
+        // four of them, and a parse of 15 bytes has hardly refilled it
+        const auto input = utils::repeated({0x99, 0x40, 0x00}, 5);
+
+        SECTION("public base")
+        {
+            json result;
+            public_wrapper handler(result);
+            CHECK_FALSE(json::sax_parse(input, &handler, json::input_format_t::cbor));
+            CHECK(innermost_capacity(result) < 16384);
+        }
+
+        SECTION("public base, a parse inside the parse")
+        {
+            json result;
+            reentrant_wrapper handler(result);
+            CHECK_FALSE(json::sax_parse(input, &handler, json::input_format_t::cbor));
+            CHECK(handler.inner_done);
+            CHECK(innermost_capacity(result) < 16384);
+        }
+
+        // the reader cannot convert these to the DOM parser: they are not bound
+        SECTION("private base")
+        {
+            json result;
+            private_wrapper handler(result);
+            CHECK_FALSE(json::sax_parse(input, &handler, json::input_format_t::cbor));
+            CHECK(innermost_capacity(result) == 16384);
+        }
+
+        SECTION("protected base")
+        {
+            json result;
+            protected_wrapper handler(result);
+            CHECK_FALSE(json::sax_parse(input, &handler, json::input_format_t::cbor));
+            CHECK(innermost_capacity(result) == 16384);
+        }
+    }
+
+    SECTION("SAX events report the announced size")
+    {
+        SaxEventLogger logger;
+        const std::vector<std::uint8_t> input = {0x99, 0x40, 0x00};
+        CHECK_FALSE(json::sax_parse(input, &logger, json::input_format_t::cbor));
+        REQUIRE(logger.events.size() == 2);
+        CHECK(logger.events[0] == "start_array(16384)");
+        CHECK(logger.events[1] == "parse_error(4)");
+    }
+}
+
 #if !defined(JSON_NOEXCEPTION) // corpus values that do not survive the round trip are skipped by catching the exception
 TEST_CASE("CBOR round-trip invariants")
 {

tests/src/unit-msgpack.cpp

@@ -33,6 +33,7 @@ using nlohmann::json;
 #include "round_trip_corpus.hpp"
 #include "test_utils.hpp"
 #include "sax_countdown.hpp"
+#include "allocation_counter.hpp"
 #include "sax_event_loggers.hpp"
 using utils::SaxCountdown;
 using utils::SaxEventLogger;
@@ -1759,6 +1760,47 @@ TEST_CASE("issue #5405 - array reserve for definite-length MessagePack arrays")
     }
 }
 
+TEST_CASE("MessagePack - open arrays reserve only what the input has backed")
+{
+    // an array that announces its size gets storage for (up to 16384 of) its
+    // elements when it opens; the arrays that are open at the same time must
+    // not hold more than the input read so far can back
+    SECTION("nested headers that announce 16384 elements and are never completed")
+    {
+        // 200 levels would hold 200 * 16384 * sizeof(json),
+        // if each of them reserved what it announces
+        const auto input = utils::repeated({0xDC, 0x40, 0x00}, 200);
+
+        utils::counts() = utils::allocation_counts{};
+        const auto j = utils::counted_json::from_msgpack(input, true, false);
+        CHECK(j.is_discarded());
+        CAPTURE(utils::counts().peak)
+        // the credit is 65,536 values; the base peaks at more than 3 million
+        CHECK(utils::counts().peak < sizeof(utils::counted_json) * 128 * 1024);
+    }
+
+    SECTION("a completed array is still reserved in one allocation")
+    {
+        json array = json::array();
+        for (int i = 0; i < 10000; ++i)
+        {
+            array.push_back(i % 100);
+        }
+
+        {
+            const auto packed = json::to_msgpack(array);
+
+            utils::counts() = utils::allocation_counts{};
+            const auto j = utils::counted_json::from_msgpack(packed);
+            CHECK(j.size() == 10000);
+            // the array and its storage; growing the storage as the elements
+            // arrive would take more than a dozen allocations
+            CAPTURE(utils::counts().allocations)
+            CHECK(utils::counts().allocations <= 4);
+        }
+    }
+}
+
 TEST_CASE("regression test - MessagePack ext type rejects a subtype that doesn't fit a single byte")
 {
     // subtype 0-255 must still round-trip correctly (regression guard, pre-existing behavior)

tests/src/unit-ubjson.cpp

@@ -31,6 +31,7 @@ using nlohmann::json;
 #include "round_trip_corpus.hpp"
 #include "test_utils.hpp"
 #include "sax_countdown.hpp"
+#include "allocation_counter.hpp"
 #include "sax_event_loggers.hpp"
 using utils::SaxCountdown;
 using utils::SaxEventLogger;
@@ -2880,6 +2881,60 @@ TEST_CASE("issue #5405 - array reserve for definite-length UBJSON arrays")
     }
 }
 
+TEST_CASE("UBJSON - open arrays reserve only what the input has backed")
+{
+    // an array that announces its size gets storage for (up to 16384 of) its
+    // elements when it opens; the arrays that are open at the same time must
+    // not hold more than the input read so far can back
+    SECTION("nested headers that announce 16384 elements and are never completed")
+    {
+        // 200 levels would hold 200 * 16384 * sizeof(json),
+        // if each of them reserved what it announces
+        const auto input = utils::repeated({'[', '#', 'I', 0x40, 0x00}, 200);
+
+        utils::counts() = utils::allocation_counts{};
+        const auto j = utils::counted_json::from_ubjson(input, true, false);
+        CHECK(j.is_discarded());
+        CAPTURE(utils::counts().peak)
+        // the credit is 65,536 values; the base peaks at more than 3 million
+        CHECK(utils::counts().peak < sizeof(utils::counted_json) * 128 * 1024);
+    }
+
+    SECTION("a completed array is still reserved in one allocation")
+    {
+        json array = json::array();
+        for (int i = 0; i < 10000; ++i)
+        {
+            array.push_back(i % 100);
+        }
+
+        // an array without a size is not reserved, so only these two
+        {
+            const auto packed = json::to_ubjson(array, true);
+
+            utils::counts() = utils::allocation_counts{};
+            const auto j = utils::counted_json::from_ubjson(packed);
+            CHECK(j.size() == 10000);
+            // the array and its storage; growing the storage as the elements
+            // arrive would take more than a dozen allocations
+            CAPTURE(utils::counts().allocations)
+            CHECK(utils::counts().allocations <= 4);
+        }
+
+        {
+            const auto packed = json::to_ubjson(array, true, true);
+
+            utils::counts() = utils::allocation_counts{};
+            const auto j = utils::counted_json::from_ubjson(packed);
+            CHECK(j.size() == 10000);
+            // the array and its storage; growing the storage as the elements
+            // arrive would take more than a dozen allocations
+            CAPTURE(utils::counts().allocations)
+            CHECK(utils::counts().allocations <= 4);
+        }
+    }
+}
+
 
 TEST_CASE("Universal Binary JSON Specification Examples 1")
 {