Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget (#6083)

* Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget

The whole-input budget for the elements of optimized UBJSON arrays of
type Z, T and F (1048576 plus 4 per byte read) rejects files written by
3.12.0 that have long arrays of these types, such as 1100 rows of 1000
nulls with a `$Z` annotation, and there was no way to read them. The
per-byte rate is now the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE.

Semantics:
- default 4, so the default build is unchanged (the generated code of
  from_ubjson is identical at -O2 up to source line numbers);
- the value must be a positive integer: 0, negative, non-integral, bool,
  character and wider-than-uintmax_t values fail to compile with a
  static_assert that names the macro;
- a value that does not fit into std::size_t is the largest size_t, so
  one spelling works for 32- and 64-bit targets;
- the limit base + rate * bytes_read saturates at the largest size_t
  instead of wrapping around, for any rate and input size; a rate of
  131072 (2^20 / 8) or more cannot be exceeded by any input (the
  shortest header of an array of more than 32767 elements is 8 bytes),
  which switches the whole-input budget off;
- the limit of one array (1048576) and the base (1048576) stay fixed
  constants, so the writer fallback still follows the base only and a
  reader with any rate reads everything the writer produces;
- the macro is not part of the ABI tag and must have the same value in
  every translation unit; it is documented that way.

The memory the arrays can take in a DOM is about 20 bytes per rate and
byte of input (measured, macOS, containers of 9 * rate elements:
25 bytes per input byte at rate 1, 81 at 4, 312 at 16, 1226 at 64, 5493
at 256) plus about 33 MB for the base. A 3.12.0 row of N values has a
header of b bytes (5 up to 255, 6 up to 32767, 8 above), so a table is
read completely if N <= rate * b, and otherwise up to
(1048576 + h * rate, h = outer header: 6, 7 or 9 bytes) / (N - rate * b) rows; 1100 rows of 1000 nulls need
a rate of 8. The new page of the macro documents the formulas, the
measurements, and two examples that are run by the documentation checks.

The tests of the budget move to tests/src/unit-ubjson_valueless.cpp, which
derives every boundary from the macro and is built with the rates 1,
1000000 and 0xFFFFFFFFFFFFFFFF in addition to the default (the last two
check that nothing is refused and that the limit does not wrap around);
they fail with the unchanged header for rate 1, 1000000 and the maximum.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Refuse char8_t for JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE and clarify its documentation

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/from_ubjson.md

@@ -91,8 +91,9 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 - Throws [out_of_range.408](../../home/exceptions.md#jsonexceptionout_of_range408) if the count of an optimized
   container exceeds the capacity of the container (or of `std::size_t`; an array of type `N` is empty, so only the
   latter applies to it), if an optimized array of `Z`, `T`, or `F` has more than 1,048,576 elements, or if the
-  optimized arrays of `Z`, `T`, and `F` in the input together have more than 1,048,576 elements plus 4 for every byte
-  read so far
+  optimized arrays of `Z`, `T`, and `F` in the input together have more than 1,048,576 elements plus
+  [`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../macros/json_ubjson_valueless_elements_per_byte.md) (4 by default) for
+  every byte read so far
 
 ## Complexity
 
@@ -121,6 +122,8 @@ Linear in the size of the input.
 - [Parsing untrusted input](../../features/parsing/untrusted_input.md) - limits and safe handling of hostile data
 - [Binary values](../../features/binary_values.md) - how binary values and subtypes are handled
 - [`to_ubjson`](to_ubjson.md) - create a UBJSON serialization of a JSON value
+- [`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../macros/json_ubjson_valueless_elements_per_byte.md) -
+  set how many elements of arrays of `null`, `true`, or `false` the input may announce per byte
 - [`from_cbor`](from_cbor.md) - create a JSON value from an input in CBOR format
 - [`from_msgpack`](from_msgpack.md) - create a JSON value from an input in MessagePack format
 - [`from_bson`](from_bson.md) - create a JSON value from an input in BSON format
@@ -136,7 +139,10 @@ Linear in the size of the input.
 - Added `error_handler` parameter in version 3.13.0.
 - Report `start_array(0)` to a SAX handler for an optimized array of type `N` (`[$N#n`), whatever its count, in version
   3.13.0.
-- Limited the total number of elements of optimized arrays of `Z`, `T`, and `F` in one input in version 3.13.0.
+- Limited the total number of elements of optimized arrays of `Z`, `T`, and `F` in one input in version 3.13.0; the
+  number of elements per byte can be set with
+  [`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../macros/json_ubjson_valueless_elements_per_byte.md), for example to read
+  files written by version 3.12.0 and earlier that exceed the default.
 
 !!! warning "Deprecation"
 

docs/mkdocs/docs/api/basic_json/to_ubjson.md

@@ -105,6 +105,8 @@ Linear in the size of the JSON value `j`.
 - [Binary formats](../../features/binary_formats/index.md) - overview and comparison of the supported binary formats
 - [Binary values](../../features/binary_values.md) - how binary values and subtypes are handled
 - [`from_ubjson`](from_ubjson.md) - create a JSON value from an input in UBJSON format
+- [`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../macros/json_ubjson_valueless_elements_per_byte.md) -
+  set how many elements of arrays of `null`, `true`, or `false` a reader accepts per byte of input
 - [`to_cbor`](to_cbor.md) - create a CBOR serialization of a JSON value
 - [`to_msgpack`](to_msgpack.md) - create a MessagePack serialization of a JSON value
 - [`to_bson`](to_bson.md) - create a BSON serialization of a JSON value
@@ -119,6 +121,8 @@ Linear in the size of the JSON value `j`.
   [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled) throws `type_error.316`.
 - Writes arrays of `Z`, `T`, or `F` without the `$` type annotation (`[#n Z Z ...`) once their elements in one call
   add up to more than 1,048,576, in version 3.13.0, so that `from_ubjson` can read the output back. Such output is
-  larger: 270,000 rows of 100 `null` values take 26.8 MB instead of 1.35 MB.
+  larger: 270,000 rows of 100 `null` values take 26.8 MB instead of 1.35 MB. The limit does not depend on
+  [`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../macros/json_ubjson_valueless_elements_per_byte.md): a reader accepts at
+  least 1,048,576 elements whatever it is set to, so the writer follows that base only.
 - Throws `type_error.321` for a discarded value since version 3.13.0; previously, a discarded value nested in an
   array or object was silently skipped, producing invalid UBJSON.

docs/mkdocs/docs/api/macros/index.md

@@ -26,6 +26,8 @@ header. See also the [macro overview page](../../features/macros.md).
   UBJSON, BJData, and BSON writers
 - [**JSON_STRICT_NUL_HANDLING**](json_strict_nul_handling.md) - opt in to rejecting a NUL byte in the input instead of
   treating it as end of input
+- [**JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE**](json_ubjson_valueless_elements_per_byte.md) - set how many elements of
+  UBJSON arrays of `null`, `true`, or `false` an input may announce per byte it has
 
 ## Language support
 

docs/mkdocs/docs/api/macros/json_ubjson_valueless_elements_per_byte.md

@@ -0,0 +1,174 @@
+# JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE
+
+```cpp
+#define JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE /* value */
+```
+
+[`from_ubjson`](../basic_json/from_ubjson.md) and [`sax_parse`](../basic_json/sax_parse.md) refuse an input in which the
+optimized arrays of type `Z` (null), `T` (true), and `F` (false) together have more than 1,048,576 elements plus
+`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE` for every byte read so far, with
+[`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408). The macro sets that number. It is the
+elements per byte that the input is allowed to announce without paying for them: an element of such an array is a
+single marker that is implied by the array's `$Z#n` header, so a few bytes can announce a million values (see
+[Parsing Untrusted Input](../../features/parsing/untrusted_input.md)).
+
+The macro only matters for input that was not written by this library 3.13.0 or later:
+[`to_ubjson`](../basic_json/to_ubjson.md) never writes more than the base of 1,048,576 elements in typed arrays of
+these types, and a reader accepts at least that many whatever the macro is. Its main use is reading files written by
+version 3.12.0 and earlier, which did not limit them (see
+[below](#reading-files-written-by-version-3120-and-earlier)).
+
+## Default definition
+
+The default value is `4`.
+
+```cpp
+#define JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE 4
+```
+
+## Notes
+
+!!! info "Allowed values"
+
+    The value must be a **positive integer** (a value of `0` would not allow any element beyond the base, and a negative
+    or fractional rate has no meaning). Any other value is a compile-time error with a message that names the macro:
+
+    ```
+    error: static assertion failed: JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE must be a positive integer
+    error: static assertion failed: JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE must be an integer (not bool or a character type)
+    error: static assertion failed: JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE must not be wider than std::uintmax_t
+    ```
+
+    The first is reported for `0` and negative values, the second for values such as `4.5`, `1e3`, `true`, or a
+    character of any character type (`'a'`, `L'a'`, `u8'a'`, ...), and the third for an integer type that is wider than
+    `#!cpp std::uintmax_t` (for example `unsigned __int128`). A value that is larger than the largest `#!cpp std::size_t` is treated as that largest value, so that a project can use one
+    spelling, for example `0xFFFFFFFFFFFFFFFFu`, for 32-bit and 64-bit targets.
+
+!!! info "Large values: the limit is switched off"
+
+    The limit is the base plus the rate times the bytes read. It saturates at the largest `#!cpp std::size_t` instead of
+    overflowing, for any rate and any input size (also with a 32-bit `#!cpp std::size_t`), so a large value never makes
+    the limit smaller. A rate of **at least 131,072** (`1048576 / 8`) effectively switches the limit for the whole
+    input off: the shortest header of an array of more than 32,767 elements takes 8 bytes (`$Z#l` and four bytes of the
+    count, inside an optimized array of arrays; 9 bytes, `[$Z#l`, at the top level), and a shorter header (`$Z#I` and
+    two bytes) can announce at most 32,767. No array can then announce more than the rate times the bytes of its own
+    header. With a rate of 131,071 that is not so: rows of 1,048,576 elements in 8 bytes each gain 8 elements per row,
+    and a table of 278,527 such rows (2.2 MB) is the shortest input that is refused.
+
+!!! info "What the macro does not change"
+
+    - The limit of **one array** stays at 1,048,576 elements
+      ([`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408), "excessive array size"), whatever
+      the macro is. A value that switches the whole-input limit off does not allow a single array of 1,048,577
+      elements.
+    - The **base** of 1,048,576 elements is a fixed constant, the same as the limit of one array. An array that was
+      accepted on its own before the whole-input limit existed is therefore still accepted at the start of an input.
+    - The **writer** is not affected. [`to_ubjson`](../basic_json/to_ubjson.md) follows the base only: it writes a typed
+      array of `Z`, `T`, or `F` only while the arrays written so far and this one stay within 1,048,576 elements, and
+      writes the elements one by one otherwise. That is within the limit of every reader, because the limit is never
+      below the base.
+    - **Objects** and arrays of other types are not counted: every member of an object has a key, and every element of
+      another type takes at least a byte of input.
+    - [`from_bjdata`](../basic_json/from_bjdata.md) is not affected: BJData has no optimized arrays of these types.
+
+!!! info "Memory"
+
+    A value takes 16 bytes in a DOM, so every element that the limit lets pass costs memory. Beyond the first 1,048,576
+    elements (about 33 MB: 16 bytes per element, and a vector that grows by reallocation needs the old and the new
+    storage at once), a malicious input of `n` bytes can take **about `20 * K * n` bytes** of memory in a DOM, where `K`
+    is the value of the macro. The measurements below read inputs that are at the limit, with containers of `9 * K`
+    elements (`[$Z#l` and a count, 9 bytes each), with `from_ubjson` on macOS, arm64, compiled with `-O2`. The numbers
+    on other systems differ by the allocator; the 16 bytes per element are the lower bound.
+
+    | `K`   | bytes of memory per byte of input | memory for 1 MB of input |
+    |------:|----------------------------------:|-------------------------:|
+    |     1 |                                25 |                    25 MB |
+    |     4 |                                81 |                    81 MB |
+    |    16 |                               312 |                   312 MB |
+    |    64 |                             1,226 |                   1.2 GB |
+    |   256 |                             5,493 |                   5.5 GB |
+
+    A SAX handler receives the same number of events, and no memory beyond what it keeps. Choose the smallest value that
+    accepts your files, not a large one, if the input comes from an untrusted source: a value of 131,072 or more
+    switches the limit of the whole input off, so that every 8 bytes of input can announce another 1,048,576 values,
+    the limit of one array.
+
+!!! warning "Define it before the include, in every translation unit"
+
+    This macro must be defined **before** including `<nlohmann/json.hpp>`. Defining it after the include has no effect.
+
+    The value is compiled into the member functions of the binary reader, which are inline, but it is **not**
+    encoded in the [namespace](../../features/namespace.md) of the library (there is no ABI tag for it). It must
+    have the same value in every translation unit of a program (for example by defining it on the command line or in
+    a common header); translation units with different values violate the One Definition Rule, which is undefined
+    behavior (no diagnostic is required). In practice, which value a read uses depends on which functions the compiler
+    inlined and which out-of-line copy of each remaining function the linker kept, so a function compiled with one
+    value can call a function compiled with the other, and a single parse can mix both values.
+
+### Reading files written by version 3.12.0 and earlier
+
+Version 3.12.0 wrote an array of `Z`, `T`, or `F` of any length in the optimized form, and a table of equal rows as an
+optimized array of optimized arrays: a header of `[$[#` and the number of rows (`i` or `U` and 1 byte, `I` and 2 bytes, or
+`l` and 4 bytes: `h` = 6, 7, or 9 bytes in all for up to 255, 32,767, or more rows), then for every row `$Z#` and its count:
+5 bytes for up to 255 elements (`U`), 6 bytes for up to 32,767 (`I`), and 8 bytes above (`l`). Every row announces `N`
+elements for the `b` bytes of its header, so the limit grows by `K * b` and the number of elements by `N` per row.
+
+- A table is never refused if its rows have at most `K * b` elements each: at most 5 `K` for rows of up to 255
+  elements (20 at the default of `K = 4`), at most 6 `K` for rows of 256 to 32,767 (rows of 1,000 need `K >= 167`),
+  and at most 8 `K` above.
+- Otherwise the first `(1,048,576 + h * K) / (N - K * b)` rows are accepted, rounded down (`h` as above; 7 for the
+  numbers here). For the default, a table of rows of 1,000 `null` values is read up to row 1,074 (the input of the first
+  example below has 1,100); for `K = 1`, up to row 1,054; for `K = 16`, up to row 1,160.
+- In general, the smallest value that accepts an input is the largest of
+  `(the elements of the arrays so far - 1,048,576) / (the bytes read when the header of the last of them has been
+  read)` over all arrays, rounded up. For 1,100 rows of 1,000 elements that is
+  `(1,100,000 - 1,048,576) / 6,607 = 7.8`, so `K = 8`.
+
+To read such a file, define the macro, before the include (or with `-DJSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE=8`), to
+the smallest value that accepts the file, and see the table above for what that value allows an untrusted input to
+cost. A file can be converted once: read it with the larger value and write it again with `to_ubjson`, which writes no
+typed array that a reader with the default value would refuse, so that the result does not need the macro.
+
+## Examples
+
+??? example "Example: default limit"
+
+    The table of 1,100 rows of 1,000 `null` values (6,607 bytes) that version 3.12.0 wrote announces 1,100,000 values,
+    which is more than 1,048,576 plus 4 for every byte read.
+
+    ```cpp
+    --8<-- "examples/ubjson_valueless_elements_per_byte.cpp"
+    ```
+
+    Output:
+
+    ```
+    --8<-- "examples/ubjson_valueless_elements_per_byte.output"
+    ```
+
+??? example "Example: raised limit"
+
+    With 16 elements per byte the same table is read. The limit of one array still applies.
+
+    ```cpp
+    --8<-- "examples/ubjson_valueless_elements_per_byte_raised.cpp"
+    ```
+
+    Output:
+
+    ```
+    --8<-- "examples/ubjson_valueless_elements_per_byte_raised.output"
+    ```
+
+## See also
+
+- [`from_ubjson`](../basic_json/from_ubjson.md) - create a JSON value from a UBJSON serialization
+- [`to_ubjson`](../basic_json/to_ubjson.md) - create a UBJSON serialization of a JSON value
+- [`sax_parse`](../basic_json/sax_parse.md) - generate SAX events
+- [UBJSON](../../features/binary_formats/ubjson.md) - the format and its limits in the library
+- [Parsing Untrusted Input](../../features/parsing/untrusted_input.md) - limits and checks for input you do not control
+- [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408) - the exception that reports the limit
+
+## Version history
+
+- Added in version 3.13.0.

docs/mkdocs/docs/examples/ubjson_valueless_elements_per_byte.cpp

@@ -0,0 +1,34 @@
+#include <cstdint>
+#include <iostream>
+#include <vector>
+#include <nlohmann/json.hpp>
+
+using json = nlohmann::json;
+
+// the bytes version 3.12.0 wrote for a table of 1100 rows of 1000 null values
+// with to_ubjson(table, true, true): `[$[#I` and the number of rows, then
+// `$Z#I` and the number of elements for every row
+std::vector<std::uint8_t> table_written_by_3_12_0()
+{
+    std::vector<std::uint8_t> bytes = {'[', '$', '[', '#', 'I', 1100 >> 8, 1100 & 0xFF};
+    for (int row = 0; row < 1100; ++row)
+    {
+        bytes.insert(bytes.end(), {'$', 'Z', '#', 'I', 1000 >> 8, 1000 & 0xFF});
+    }
+    return bytes;
+}
+
+int main()
+{
+    try
+    {
+        // 6607 bytes announce 1,100,000 values, more than 1,048,576 plus 4 for
+        // every byte read (the limit is exceeded in row 1075)
+        json table = json::from_ubjson(table_written_by_3_12_0());
+        std::cout << table.size() << " rows\n";
+    }
+    catch (const json::out_of_range& e)
+    {
+        std::cout << e.what() << '\n';
+    }
+}

docs/mkdocs/docs/examples/ubjson_valueless_elements_per_byte.output

@@ -0,0 +1 @@
+[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive number of elements without payload

docs/mkdocs/docs/examples/ubjson_valueless_elements_per_byte_raised.cpp

@@ -0,0 +1,42 @@
+#include <cstdint>
+#include <iostream>
+#include <vector>
+
+// accept 16 values per byte of input instead of 4; define the macro before the
+// first inclusion of the header, and to the same value in every translation unit
+#define JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE 16
+#include <nlohmann/json.hpp>
+
+using json = nlohmann::json;
+
+// the bytes version 3.12.0 wrote for a table of 1100 rows of 1000 null values
+// with to_ubjson(table, true, true): `[$[#I` and the number of rows, then
+// `$Z#I` and the number of elements for every row
+std::vector<std::uint8_t> table_written_by_3_12_0()
+{
+    std::vector<std::uint8_t> bytes = {'[', '$', '[', '#', 'I', 1100 >> 8, 1100 & 0xFF};
+    for (int row = 0; row < 1100; ++row)
+    {
+        bytes.insert(bytes.end(), {'$', 'Z', '#', 'I', 1000 >> 8, 1000 & 0xFF});
+    }
+    return bytes;
+}
+
+int main()
+{
+    // 6607 bytes announce 1,100,000 values, which is below 1,048,576 plus 16
+    // for every byte read
+    json table = json::from_ubjson(table_written_by_3_12_0());
+    std::cout << table.size() << " rows of " << table[0].size() << " values\n";
+
+    // a single container is still limited to 1,048,576 elements
+    const std::vector<std::uint8_t> too_long = {'[', '$', 'Z', '#', 'l', 0x00, 0x10, 0x00, 0x01};
+    try
+    {
+        json j = json::from_ubjson(too_long);
+    }
+    catch (const json::out_of_range& e)
+    {
+        std::cout << e.what() << '\n';
+    }
+}

docs/mkdocs/docs/examples/ubjson_valueless_elements_per_byte_raised.output

@@ -0,0 +1,2 @@
+1100 rows of 1000 values
+[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive array size

docs/mkdocs/docs/features/binary_formats/ubjson.md

@@ -119,17 +119,23 @@ The library uses the following mapping from JSON values types to UBJSON types ac
     receiving side is immediately informed on the number of elements of the container.
 
     An array whose type marker is `Z` (null), `T` (true) or `F` (false) stores no payload at all, because the marker
-    already is the value. Its declared count is therefore the only thing that decides how much memory the receiving side
-    allocates, and a handful of bytes can describe billions of elements. `from_ubjson` rejects such an array with
+    already is the value. Its declared count is therefore the only thing that decides how much memory the receiving
+    side allocates, and a handful of bytes can describe billions of elements. `from_ubjson` rejects such an array with
     [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408) when the count exceeds 1,048,576
-    (`1 << 20`), or when the counts of all such arrays in the input add up to more than 1,048,576 plus 4 for every byte
-    read so far. `to_ubjson` therefore writes an array of these types in the counted but untyped form when it is longer
-    than that, or when it would bring the total of the typed arrays written so far above 1,048,576: the `$` type marker
-    is dropped, the `#` count is kept, and every element is written with its own marker (`[#l 1048577 Z Z ...`), so
-    everything it produces can be read back. Only input from another encoder, or written by version 3.12.0 and earlier of
-    this library, can exceed the limit: for example, 1,100 rows of 1,000 `null` values or 210,000 rows of 25, each with
-    a `$Z` annotation. Version 3.12.0 writes such a row in 5 bytes (6 above 255 elements), so rows of 20 or fewer such
-    values are never rejected. Objects and untyped counted arrays are not subject to this limit.
+    (`1 << 20`), or when the counts of all such arrays in the input add up to more than 1,048,576 plus _K_ for every byte
+    read so far, where _K_ is the value of the macro
+    [`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../../api/macros/json_ubjson_valueless_elements_per_byte.md) (4 by
+    default). `to_ubjson` therefore writes an array of these types in the counted but untyped form when it is longer
+    than that, or when it would bring the total of the typed arrays written so far above 1,048,576: the `$` type
+    marker is dropped, the `#` count is kept, and every element is written with its own marker 
+    (`[#l 1048577 Z Z ...`), so everything it produces can be read back. Only input from another encoder, or written by version 3.12.0
+    and earlier of this library, can exceed the limit: for example, 1,100 rows of 1,000 `null` values or 210,000 rows
+    of 25, each with a `$Z` annotation. Version 3.12.0 writes a row in 5 bytes (6 above 255 elements, 8 above 32,767),
+    which raises the limit by 5 _K_ (6 _K_, 8 _K_), so rows of up to 5 _K_ values (20 at the default) are never
+    rejected. To read such a file, define the macro to a larger value (`8` is enough for the 1,100 rows of 1,000
+    `null` values; memory for the arrays then rises to about 20 _K_ bytes per byte of input in a DOM); the macro's
+    page explains how to choose a value and what it costs. The limit of one array of 1,048,576 elements does not
+    depend on the macro. Objects and untyped counted arrays are not subject to this limit.
 
     An optimized array of type `N` (no-op) is empty whatever its count is, because a no-op is not a value. Its count is
     not passed on: `sax_parse` reports `start_array(0)` (versions before 3.13.0 reported the declared count), and the DOM
@@ -189,7 +195,9 @@ The library maps UBJSON types to JSON value types as follows:
 
     Every well-formed UBJSON value can be converted to a JSON value, with these exceptions: a high-precision number
     outside the range of `number_float_t` ([`out_of_range.406`](../../home/exceptions.md#jsonexceptionout_of_range406)),
-    optimized arrays of `Z`, `T`, or `F` with more elements than the limit allows (1,048,576 plus 4 per byte read),
+    optimized arrays of `Z`, `T`, or `F` with more elements than the limit allows (1,048,576 plus
+    [`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../../api/macros/json_ubjson_valueless_elements_per_byte.md), 4 by default,
+    per byte read),
     and a count that exceeds the capacity of the container
     ([`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408)). High-precision numbers that do not
     fit an integer type are stored as floating-point numbers and may lose precision. Ill-formed input is rejected with a

docs/mkdocs/docs/features/macros.md

@@ -251,6 +251,16 @@ This macro overrides `#!cpp try` calls inside the library.
 
 See [full documentation of `JSON_TRY_USER`](../api/macros/json_throw_user.md).
 
+## `JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`
+
+Sets how many elements of optimized UBJSON arrays of `Z` (null), `T` (true), and `F` (false) a reader accepts for
+every byte of input it has read, on top of a base of 1,048,576 elements. These arrays have no payload, so a few bytes
+can announce a million values. The value must be a positive integer; the default is `4`, and a value of 131,072 or
+more switches the limit of the whole input off. The limit of 1,048,576 elements for a single array always applies. The
+macro is not part of the [ABI tag](namespace.md): use the same value in every translation unit.
+
+See [full documentation of `JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../api/macros/json_ubjson_valueless_elements_per_byte.md).
+
 ## `JSON_USE_IMPLICIT_CONVERSIONS`
 
 When defined to `0`, implicit conversions are switched off. By default, implicit conversions are switched on.

docs/mkdocs/docs/features/parsing/untrusted_input.md

@@ -93,11 +93,17 @@ For the binary formats, an announced size is never trusted outright:
   `start_array(0)` and the DOM parser reserves nothing.
 - A UBJSON array of type `Z`, `T` or `F` has no payload, so its announced count is not bounded by the input. One array
   may announce at most 1,048,576 elements, and all such arrays of one input together may contain at most 1,048,576
-  elements plus 4 for every byte read so far (checked when each array's header has been read); beyond that the input is
-  rejected as [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408). For a DOM that bounds the memory these
-  arrays can need to about 80 bytes per byte of input plus about 33 MB for the first 1,048,576 elements; a SAX handler
-  receives at most that many events. Every other optimized array element, and every UBJSON object member (it has a
-  key), takes at least one byte of input.
+  elements plus _K_ for every byte read so far (checked when each array's header has been read), where _K_ is the
+  value of the macro
+  [`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../../api/macros/json_ubjson_valueless_elements_per_byte.md), 4 by
+  default; beyond that the input is rejected as
+  [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408). For a DOM that bounds the memory these
+  arrays can need to about 20 _K_ bytes per byte of input (81 at the default, measured; 312 for _K_ = 16) plus about
+  33 MB for the first 1,048,576 elements; a SAX handler receives at most that many events. Every other optimized array
+  element, and every UBJSON object member (it has a key), takes at least one byte of input. Leave the macro at the
+  default for input you do not control; raise it only to read trusted files that version 3.12.0 or earlier wrote with
+  long arrays of these types, and see its page for how a value is chosen and what it costs. The limit of one array
+  does not depend on the macro.
 
 ## Strings
 

docs/mkdocs/docs/home/exceptions.md

@@ -1054,13 +1054,17 @@ The exception is also thrown for [UBJSON](../features/binary_formats/ubjson.md)
 its marker alone (`Z`, `T` or `F`). Such an array has no payload, so its count alone decides how much memory is
 allocated, and a handful of bytes would otherwise describe billions of values. It is thrown if the declared count of one
 array exceeds 1,048,576 (`1 << 20`), and if the counts of all such arrays read so far in one input add up to more than
-1,048,576 plus 4 for every byte of the input read when the header of the last array has been read.
+1,048,576 plus 4 for every byte of the input read when the header of the last array has been read. The 4 is the
+default of the macro
+[`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../api/macros/json_ubjson_valueless_elements_per_byte.md), which raises it
+(for the limit of the whole input; the limit of one array stays).
 [`to_ubjson`](../api/basic_json/to_ubjson.md) writes an array of these types without the `$` type annotation (the `#`
 count and one marker per element are kept) when it is longer, or when it would bring the total of the typed arrays
 written so far above 1,048,576, so everything it produces can be read back. Only input written by another encoder, or
 by version 3.12.0 and earlier of this library, can exceed the limit: for example, 1,100 rows of 1,000 `null` values or
-210,000 rows of 25, each with a `$Z` annotation. Version 3.12.0 writes such a row in 5 bytes (6 above 255 elements),
-so rows of 20 or fewer such values are never rejected.
+210,000 rows of 25, each with a `$Z` annotation. Version 3.12.0 writes such a row in 5 bytes (6 above 255 elements, 8
+above 32,767), so rows of 20 or fewer such values are never rejected at the default; define the macro to a larger value
+to read them.
 
 !!! failure "Example messages"
 

docs/mkdocs/mkdocs.yml

@@ -458,6 +458,7 @@ nav:
           - 'JSON_SKIP_UNSUPPORTED_COMPILER_CHECK': api/macros/json_skip_unsupported_compiler_check.md
           - 'JSON_STRICT_BINARY_UTF8': api/macros/json_strict_binary_utf8.md
           - 'JSON_STRICT_NUL_HANDLING': api/macros/json_strict_nul_handling.md
+          - 'JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE': api/macros/json_ubjson_valueless_elements_per_byte.md
           - 'JSON_USE_GLOBAL_UDLS': api/macros/json_use_global_udls.md
           - 'JSON_USE_IMPLICIT_CONVERSIONS': api/macros/json_use_implicit_conversions.md
           - 'JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON': api/macros/json_use_legacy_discarded_value_comparison.md

include/nlohmann/detail/input/binary_reader.hpp

@@ -16,7 +16,7 @@
 #include <iterator> // back_inserter
 #include <limits> // numeric_limits
 #include <string> // char_traits, string
-#include <type_traits> // is_convertible, true_type, false_type
+#include <type_traits> // false_type, integral_constant, is_convertible, is_integral, is_same, remove_cv, remove_reference, true_type
 #include <utility> // make_pair, move
 #include <vector> // vector
 #ifdef __cpp_lib_byteswap
@@ -88,7 +88,7 @@ JSON_INLINE_VARIABLE constexpr std::size_t max_valueless_container_size = 1 << 2
 
 The limit @ref max_valueless_container_size applies to each container on its
 own, so on its own it lets 9 bytes announce 1048576 values, and every further
-9 bytes another 1048576. The elements of all containers of a valueless type
+8 bytes (a row of an optimized array of arrays) another 1048576. The elements of all containers of a valueless type
 in one input are therefore added up, and the sum must stay below this base
 plus @ref valueless_elements_per_byte for every byte read so far. The base is
 the per-container limit, so a container that was accepted on its own before
@@ -96,24 +96,110 @@ the sum was introduced is still accepted at the start of an input.
 */
 JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_base = max_valueless_container_size;
 
+/// the value of JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE as a std::size_t:
+/// saturated at the largest one, not truncated, for a value that is wider
+template<typename T>
+constexpr std::size_t valueless_elements_rate(const T value) noexcept
+{
+    return static_cast<std::uintmax_t>(value) > (std::numeric_limits<std::size_t>::max)()
+           ? (std::numeric_limits<std::size_t>::max)()
+           : static_cast<std::size_t>(value);
+}
+
 /*!
 @brief number of elements of valueless UBJSON containers accepted per byte of
        input, in addition to @ref valueless_elements_base
 
-An element of a value that has a payload costs at least one byte of input.
-Valueless elements are accepted at a rate of 4 per byte. A value takes 16
-bytes of memory in a DOM, so these arrays add about 80 bytes of memory per
-byte of input (measured: 77; one `Z` per byte needs about 32), beyond what the
-same bytes cost anyway, plus the base. This bounds the arrays of valueless
-type, not the memory of a whole document. Containers written by
-@ref binary_writer never need more than the base, because it writes an array
-in the unoptimized form once the typed arrays written so far and this one add
-up to more than that. The rate matters for input written by other encoders and
-by versions before 3.13.0 only; in the data this was measured on, no input has
-more than 0.17 elements per byte.
-*/
-JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_per_byte = 4;
+This is the value of the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE (4 by
+default), see docs/mkdocs/docs/api/macros/json_ubjson_valueless_elements_per_byte.md.
 
+An element of a value that has a payload costs at least one byte of input, but
+an element of a valueless type costs none, so these are the only elements whose
+number the input does not bound. They are accepted at this rate per byte read,
+which bounds the memory they take in a DOM: a value takes 16 bytes there, so
+the arrays at the limit take about 20 times this rate bytes of memory per byte
+of input (16 for the value, the rest is allocator rounding and the arrays
+themselves; measured on macOS, DOM, with containers of 9 times the rate
+elements: 25 bytes per byte at a rate of 1, 81 at 4, 312 at 16, 1226 at 64 and
+5493 at 256), plus about 33 MB for the first elements of the base (16 bytes for
+each of the 1048576 elements, and a vector that grows by reallocation, as only
+16384 elements are reserved upfront, needs the old and the new storage at once).
+
+The bound is meant for input written by other encoders and by versions before
+3.13.0, which wrote long arrays of this type optimized without a limit: input
+written by @ref binary_writer never needs more than the base, because the
+writer falls back to the unoptimized form once the arrays written so far and
+the next one add up to more than that, whatever the rate is.
+
+Range: a positive integer. A value that is not one is a compile-time error (a
+reader with a rate of 0 would accept nothing beyond the base, and a negative or
+fractional rate has no meaning). A value that does not fit into std::size_t is
+used as the largest std::size_t: the rate is so high then that the bound is
+out of reach anyway, and a project can use one spelling (for example
+`0xFFFFFFFFFFFFFFFFu`) for 32 and 64 bit targets. The type of the value must
+be an integer type that is not wider than std::uintmax_t and not bool or a
+character type. A rate of at least 2^20 / 8 = 131072 effectively switches the
+whole-input bound off: an array of more than 32767 elements needs a count of
+type `l`, so its shortest header is `$Z#l` and four bytes (8 bytes) inside an
+optimized array of arrays (9 bytes, `[$Z#l`, at the top level), and a shorter
+header (`$Z#I` and two bytes) announces at most 32767. Every container thus has
+at most (rate * header bytes) elements, which never exceeds the limit. With a
+rate of 131071 an input that is refused exists (rows of 1048576 elements gain
+8 elements each); the per-container limit still applies.
+
+The value must be the same in every translation unit of a program (it is
+compiled into member functions of binary_reader, which are inline), and it is
+not part of the ABI tag of the namespace.
+*/
+JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_per_byte =
+    // a value that does not fit into std::size_t is the largest one (see above);
+    // the comparison is made in the widest unsigned type, so that nothing is
+    // truncated before it is made. A function template, so that the casts are
+    // not "useless" (-Wuseless-cast) where the macro already has the type
+    valueless_elements_rate((JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE));
+
+/// the type of the macro's value is char8_t (C++20)
+template<typename T>
+#if defined(__cpp_char8_t)
+struct valueless_elements_rate_is_char8 : std::is_same<T, char8_t> {};
+#else
+struct valueless_elements_rate_is_char8 : std::false_type {};
+#endif
+
+/// the type of the macro's value is a character or boolean type
+template<typename T>
+struct valueless_elements_rate_is_char_or_bool
+    : disjunction<std::is_same<T, bool>, std::is_same<T, char>, std::is_same<T, signed char>,
+      std::is_same<T, unsigned char>, std::is_same<T, wchar_t>, std::is_same<T, char16_t>,
+      std::is_same<T, char32_t>, valueless_elements_rate_is_char8<T>>
+{};
+
+// these messages are what a user sees for a wrong value of the macro. The
+// integer test comes first because a fractional value would also fail the
+// positivity test; a value wider than std::uintmax_t would be truncated by
+// the conversion above, so it is rejected
+static_assert(std::is_integral<typename std::remove_reference<decltype((JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE))>::type>::value
+              && !valueless_elements_rate_is_char_or_bool<typename std::remove_cv<typename std::remove_reference<decltype((JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE))>::type>::type>::value,
+              "JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE must be an integer (not bool or a character type)");
+static_assert(sizeof(decltype((JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE))) <= sizeof(std::uintmax_t),
+              "JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE must not be wider than std::uintmax_t");
+static_assert((JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE) > 0,
+              "JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE must be a positive integer");
+
+/*!
+@brief the number of bytes read up to which
+       @ref valueless_elements_base plus @ref valueless_elements_per_byte times
+       that number can be computed in std::size_t
+
+For more bytes than this the limit of the valueless elements is the largest
+std::size_t, which is saturation: the sum is not computed, because it would wrap
+around, which happens for a rate that is large (the largest std::size_t as the
+rate wraps already at the first byte) or an input that is large (more than
+about a billion bytes for the default rate with a 32-bit std::size_t). It is a
+constant of the build, so no division is made while reading.
+*/
+JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_saturation_chars =
+    ((std::numeric_limits<std::size_t>::max)() - valueless_elements_base) / (valueless_elements_per_byte == 0 ? 1 : valueless_elements_per_byte);
 ///////////////////
 // binary reader //
 ///////////////////
@@ -3807,13 +3893,14 @@ class binary_reader
     */
     bool charge_valueless_elements(const std::size_t count) noexcept
     {
-        // the limit saturates instead of overflowing when size_t is 32 bits
-        // wide and a very large input has been read; valueless_elements
-        // never exceeds the limit, so the difference cannot underflow
-        constexpr std::size_t max_size = (std::numeric_limits<std::size_t>::max)();
-        const std::size_t limit = chars_read <= (max_size - valueless_elements_base) / valueless_elements_per_byte
+        // limit = base + rate * chars_read, saturating at the largest size_t
+        // instead of wrapping around, which would make a huge rate or input
+        // reject everything. valueless_elements never exceeds the limit of an
+        // earlier call, and the limit does not decrease, so the difference
+        // below cannot underflow.
+        const std::size_t limit = chars_read <= valueless_elements_saturation_chars
                                   ? valueless_elements_base + (valueless_elements_per_byte * chars_read)
-                                  : max_size;
+                                  : (std::numeric_limits<std::size_t>::max)();
         if (JSON_HEDLEY_UNLIKELY(count > limit - valueless_elements))
         {
             return false;

include/nlohmann/detail/macro_scope.hpp

@@ -917,3 +917,14 @@
 #ifndef JSON_DISABLE_TUPLE_REFERENCE_CONVERSION
     #define JSON_DISABLE_TUPLE_REFERENCE_CONVERSION 0
 #endif
+
+// JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE: how many elements of optimized
+// UBJSON containers of type Z, T or F (which have no payload) a reader accepts
+// per byte of input, on top of a fixed base. The value is checked and converted
+// in binary_reader.hpp; the macro is documented in
+// docs/mkdocs/docs/api/macros/json_ubjson_valueless_elements_per_byte.md.
+// It is not part of the ABI tag: it must have the same value in every
+// translation unit of a program.
+#ifndef JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE
+    #define JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE 4
+#endif

include/nlohmann/detail/macro_unscope.hpp

@@ -60,6 +60,7 @@
     #undef JSON_USE_OBJECTS_FOR_ENUM_KEYED_MAPS
     #undef JSON_DISABLE_STREAM_BULK_SCAN
     #undef JSON_DISABLE_UNLOCKED_FILE_IO
+    #undef JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE
 #endif
 
 #include <nlohmann/thirdparty/hedley/hedley_undef.hpp>

include/nlohmann/detail/output/binary_writer.hpp

@@ -1206,12 +1206,14 @@ class binary_writer
             // reader has nothing but the declared count to bound the allocation
             // by. It refuses one array of more than max_valueless_container_size
             // elements, and all such arrays of one input together when they add
-            // up to more than valueless_elements_base plus a number per byte
-            // read. Write the unoptimized form, at one byte per element, for
-            // an array that would exceed the base with the arrays written so
-            // far, so that the result can be read back: the limit of a reader
-            // is never below the base. Objects are not affected: every element
-            // is preceded by its key.
+            // up to more than valueless_elements_base plus
+            // JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE per byte read. Write the
+            // unoptimized form, at one byte per element, for an array that
+            // would exceed the base with the arrays written so far, so that the
+            // result can be read back: the limit of a reader is never below the
+            // base, whatever the macro is set to, so the writer does not use
+            // it (and a reader with a larger rate reads this output as well).
+            // Objects are not affected: every element is preceded by its key.
             if (same_prefix && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
             {
                 const bool valueless_type = IsArray && (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');

single_include/nlohmann/json.hpp

@@ -3291,6 +3291,17 @@ JSON_HEDLEY_DIAGNOSTIC_POP
     #define JSON_DISABLE_TUPLE_REFERENCE_CONVERSION 0
 #endif
 
+// JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE: how many elements of optimized
+// UBJSON containers of type Z, T or F (which have no payload) a reader accepts
+// per byte of input, on top of a fixed base. The value is checked and converted
+// in binary_reader.hpp; the macro is documented in
+// docs/mkdocs/docs/api/macros/json_ubjson_valueless_elements_per_byte.md.
+// It is not part of the ABI tag: it must have the same value in every
+// translation unit of a program.
+#ifndef JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE
+    #define JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE 4
+#endif
+
 #if JSON_HAS_THREE_WAY_COMPARISON
     #include <compare> // partial_ordering
 #endif
@@ -8866,7 +8877,7 @@ NLOHMANN_JSON_NAMESPACE_END
 #include <iterator> // back_inserter
 #include <limits> // numeric_limits
 #include <string> // char_traits, string
-#include <type_traits> // is_convertible, true_type, false_type
+#include <type_traits> // false_type, integral_constant, is_convertible, is_integral, is_same, remove_cv, remove_reference, true_type
 #include <utility> // make_pair, move
 #include <vector> // vector
 #ifdef __cpp_lib_byteswap
@@ -18503,7 +18514,7 @@ JSON_INLINE_VARIABLE constexpr std::size_t max_valueless_container_size = 1 << 2
 
 The limit @ref max_valueless_container_size applies to each container on its
 own, so on its own it lets 9 bytes announce 1048576 values, and every further
-9 bytes another 1048576. The elements of all containers of a valueless type
+8 bytes (a row of an optimized array of arrays) another 1048576. The elements of all containers of a valueless type
 in one input are therefore added up, and the sum must stay below this base
 plus @ref valueless_elements_per_byte for every byte read so far. The base is
 the per-container limit, so a container that was accepted on its own before
@@ -18511,24 +18522,110 @@ the sum was introduced is still accepted at the start of an input.
 */
 JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_base = max_valueless_container_size;
 
+/// the value of JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE as a std::size_t:
+/// saturated at the largest one, not truncated, for a value that is wider
+template<typename T>
+constexpr std::size_t valueless_elements_rate(const T value) noexcept
+{
+    return static_cast<std::uintmax_t>(value) > (std::numeric_limits<std::size_t>::max)()
+           ? (std::numeric_limits<std::size_t>::max)()
+           : static_cast<std::size_t>(value);
+}
+
 /*!
 @brief number of elements of valueless UBJSON containers accepted per byte of
        input, in addition to @ref valueless_elements_base
 
-An element of a value that has a payload costs at least one byte of input.
-Valueless elements are accepted at a rate of 4 per byte. A value takes 16
-bytes of memory in a DOM, so these arrays add about 80 bytes of memory per
-byte of input (measured: 77; one `Z` per byte needs about 32), beyond what the
-same bytes cost anyway, plus the base. This bounds the arrays of valueless
-type, not the memory of a whole document. Containers written by
-@ref binary_writer never need more than the base, because it writes an array
-in the unoptimized form once the typed arrays written so far and this one add
-up to more than that. The rate matters for input written by other encoders and
-by versions before 3.13.0 only; in the data this was measured on, no input has
-more than 0.17 elements per byte.
-*/
-JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_per_byte = 4;
+This is the value of the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE (4 by
+default), see docs/mkdocs/docs/api/macros/json_ubjson_valueless_elements_per_byte.md.
 
+An element of a value that has a payload costs at least one byte of input, but
+an element of a valueless type costs none, so these are the only elements whose
+number the input does not bound. They are accepted at this rate per byte read,
+which bounds the memory they take in a DOM: a value takes 16 bytes there, so
+the arrays at the limit take about 20 times this rate bytes of memory per byte
+of input (16 for the value, the rest is allocator rounding and the arrays
+themselves; measured on macOS, DOM, with containers of 9 times the rate
+elements: 25 bytes per byte at a rate of 1, 81 at 4, 312 at 16, 1226 at 64 and
+5493 at 256), plus about 33 MB for the first elements of the base (16 bytes for
+each of the 1048576 elements, and a vector that grows by reallocation, as only
+16384 elements are reserved upfront, needs the old and the new storage at once).
+
+The bound is meant for input written by other encoders and by versions before
+3.13.0, which wrote long arrays of this type optimized without a limit: input
+written by @ref binary_writer never needs more than the base, because the
+writer falls back to the unoptimized form once the arrays written so far and
+the next one add up to more than that, whatever the rate is.
+
+Range: a positive integer. A value that is not one is a compile-time error (a
+reader with a rate of 0 would accept nothing beyond the base, and a negative or
+fractional rate has no meaning). A value that does not fit into std::size_t is
+used as the largest std::size_t: the rate is so high then that the bound is
+out of reach anyway, and a project can use one spelling (for example
+`0xFFFFFFFFFFFFFFFFu`) for 32 and 64 bit targets. The type of the value must
+be an integer type that is not wider than std::uintmax_t and not bool or a
+character type. A rate of at least 2^20 / 8 = 131072 effectively switches the
+whole-input bound off: an array of more than 32767 elements needs a count of
+type `l`, so its shortest header is `$Z#l` and four bytes (8 bytes) inside an
+optimized array of arrays (9 bytes, `[$Z#l`, at the top level), and a shorter
+header (`$Z#I` and two bytes) announces at most 32767. Every container thus has
+at most (rate * header bytes) elements, which never exceeds the limit. With a
+rate of 131071 an input that is refused exists (rows of 1048576 elements gain
+8 elements each); the per-container limit still applies.
+
+The value must be the same in every translation unit of a program (it is
+compiled into member functions of binary_reader, which are inline), and it is
+not part of the ABI tag of the namespace.
+*/
+JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_per_byte =
+    // a value that does not fit into std::size_t is the largest one (see above);
+    // the comparison is made in the widest unsigned type, so that nothing is
+    // truncated before it is made. A function template, so that the casts are
+    // not "useless" (-Wuseless-cast) where the macro already has the type
+    valueless_elements_rate((JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE));
+
+/// the type of the macro's value is char8_t (C++20)
+template<typename T>
+#if defined(__cpp_char8_t)
+struct valueless_elements_rate_is_char8 : std::is_same<T, char8_t> {};
+#else
+struct valueless_elements_rate_is_char8 : std::false_type {};
+#endif
+
+/// the type of the macro's value is a character or boolean type
+template<typename T>
+struct valueless_elements_rate_is_char_or_bool
+    : disjunction<std::is_same<T, bool>, std::is_same<T, char>, std::is_same<T, signed char>,
+      std::is_same<T, unsigned char>, std::is_same<T, wchar_t>, std::is_same<T, char16_t>,
+      std::is_same<T, char32_t>, valueless_elements_rate_is_char8<T>>
+{};
+
+// these messages are what a user sees for a wrong value of the macro. The
+// integer test comes first because a fractional value would also fail the
+// positivity test; a value wider than std::uintmax_t would be truncated by
+// the conversion above, so it is rejected
+static_assert(std::is_integral<typename std::remove_reference<decltype((JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE))>::type>::value
+              && !valueless_elements_rate_is_char_or_bool<typename std::remove_cv<typename std::remove_reference<decltype((JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE))>::type>::type>::value,
+              "JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE must be an integer (not bool or a character type)");
+static_assert(sizeof(decltype((JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE))) <= sizeof(std::uintmax_t),
+              "JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE must not be wider than std::uintmax_t");
+static_assert((JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE) > 0,
+              "JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE must be a positive integer");
+
+/*!
+@brief the number of bytes read up to which
+       @ref valueless_elements_base plus @ref valueless_elements_per_byte times
+       that number can be computed in std::size_t
+
+For more bytes than this the limit of the valueless elements is the largest
+std::size_t, which is saturation: the sum is not computed, because it would wrap
+around, which happens for a rate that is large (the largest std::size_t as the
+rate wraps already at the first byte) or an input that is large (more than
+about a billion bytes for the default rate with a 32-bit std::size_t). It is a
+constant of the build, so no division is made while reading.
+*/
+JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_saturation_chars =
+    ((std::numeric_limits<std::size_t>::max)() - valueless_elements_base) / (valueless_elements_per_byte == 0 ? 1 : valueless_elements_per_byte);
 ///////////////////
 // binary reader //
 ///////////////////
@@ -22222,13 +22319,14 @@ class binary_reader
     */
     bool charge_valueless_elements(const std::size_t count) noexcept
     {
-        // the limit saturates instead of overflowing when size_t is 32 bits
-        // wide and a very large input has been read; valueless_elements
-        // never exceeds the limit, so the difference cannot underflow
-        constexpr std::size_t max_size = (std::numeric_limits<std::size_t>::max)();
-        const std::size_t limit = chars_read <= (max_size - valueless_elements_base) / valueless_elements_per_byte
+        // limit = base + rate * chars_read, saturating at the largest size_t
+        // instead of wrapping around, which would make a huge rate or input
+        // reject everything. valueless_elements never exceeds the limit of an
+        // earlier call, and the limit does not decrease, so the difference
+        // below cannot underflow.
+        const std::size_t limit = chars_read <= valueless_elements_saturation_chars
                                   ? valueless_elements_base + (valueless_elements_per_byte * chars_read)
-                                  : max_size;
+                                  : (std::numeric_limits<std::size_t>::max)();
         if (JSON_HEDLEY_UNLIKELY(count > limit - valueless_elements))
         {
             return false;
@@ -30813,12 +30911,14 @@ class binary_writer
             // reader has nothing but the declared count to bound the allocation
             // by. It refuses one array of more than max_valueless_container_size
             // elements, and all such arrays of one input together when they add
-            // up to more than valueless_elements_base plus a number per byte
-            // read. Write the unoptimized form, at one byte per element, for
-            // an array that would exceed the base with the arrays written so
-            // far, so that the result can be read back: the limit of a reader
-            // is never below the base. Objects are not affected: every element
-            // is preceded by its key.
+            // up to more than valueless_elements_base plus
+            // JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE per byte read. Write the
+            // unoptimized form, at one byte per element, for an array that
+            // would exceed the base with the arrays written so far, so that the
+            // result can be read back: the limit of a reader is never below the
+            // base, whatever the macro is set to, so the writer does not use
+            // it (and a reader with a larger rate reads this output as well).
+            // Objects are not affected: every element is preceded by its key.
             if (same_prefix && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
             {
                 const bool valueless_type = IsArray && (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
@@ -48042,6 +48142,7 @@ struct formatter<nlohmann::NLOHMANN_BASIC_JSON_TPL, char> // NOLINT(cert-dcl58-c
     #undef JSON_USE_OBJECTS_FOR_ENUM_KEYED_MAPS
     #undef JSON_DISABLE_STREAM_BULK_SCAN
     #undef JSON_DISABLE_UNLOCKED_FILE_IO
+    #undef JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE
 #endif
 
 // #include <nlohmann/thirdparty/hedley/hedley_undef.hpp>

tests/CMakeLists.txt

@@ -410,6 +410,34 @@ json_test_add_test_for(src/unit-stream-bulk-scan.cpp
     MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force}
 )
 
+# test the budget for the elements of UBJSON containers of a valueless type again
+# with other values of JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE: 1 (the smallest),
+# 1000000 (so large that the budget cannot be reached), and the largest value of
+# a 64-bit integer (which saturates the limit already at the first byte, and is
+# clamped to the largest std::size_t where that has 32 bits). The file does not
+# mention JSON_HAS_CPP_<n>, so only C++11 is built unless the standards are forced
+json_test_set_test_options(test-ubjson_valueless_rate_1
+    COMPILE_DEFINITIONS JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE=1
+)
+json_test_add_test_for(src/unit-ubjson_valueless.cpp
+    NAME test-ubjson_valueless_rate_1
+    MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force}
+)
+json_test_set_test_options(test-ubjson_valueless_rate_1000000
+    COMPILE_DEFINITIONS JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE=1000000
+)
+json_test_add_test_for(src/unit-ubjson_valueless.cpp
+    NAME test-ubjson_valueless_rate_1000000
+    MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force}
+)
+json_test_set_test_options(test-ubjson_valueless_rate_max
+    COMPILE_DEFINITIONS JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE=0xFFFFFFFFFFFFFFFFu
+)
+json_test_add_test_for(src/unit-ubjson_valueless.cpp
+    NAME test-ubjson_valueless_rate_max
+    MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force}
+)
+
 # test reading from a FILE* again with the locking stdio functions
 json_test_set_test_options(test-file_input_unlocked_io_disabled
     COMPILE_DEFINITIONS JSON_DISABLE_UNLOCKED_FILE_IO=1

tests/src/ubjson_valueless.hpp

@@ -0,0 +1,125 @@
+//     __ _____ _____ _____
+//  __|  |   __|     |   | |  JSON for Modern C++ (supporting code)
+// |  |  |__   |  |  | | | |  version 3.12.0
+// |_____|_____|_____|_|___|  https://github.com/nlohmann/json
+//
+// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
+// SPDX-License-Identifier: MIT
+
+#pragma once
+
+#include <cstddef> // size_t
+#include <cstdint> // uint8_t, uint32_t
+#include <limits> // numeric_limits
+#include <string> // string
+#include <vector> // vector
+
+#include <nlohmann/json.hpp>
+
+// helpers of the tests of the budget for the elements of UBJSON containers of
+// a valueless type (unit-ubjson.cpp and unit-ubjson_valueless.cpp)
+#if defined(__clang__)
+    #pragma clang diagnostic push
+    #pragma clang diagnostic ignored "-Wweak-vtables"
+#endif
+
+namespace utils
+{
+/// SAX consumer that counts the values and keeps the id of a parse error
+struct ValuelessEventCounter : public nlohmann::json_sax<nlohmann::json>
+{
+    bool null() override
+    {
+        return ++values < stop_after;
+    }
+    bool boolean(bool /*val*/) override
+    {
+        return ++values < stop_after;
+    }
+    bool number_integer(nlohmann::json::number_integer_t /*val*/) override
+    {
+        ++values;
+        return true;
+    }
+    bool number_unsigned(nlohmann::json::number_unsigned_t /*val*/) override
+    {
+        ++values;
+        return true;
+    }
+    bool number_float(nlohmann::json::number_float_t /*val*/, const std::string& /*s*/) override
+    {
+        ++values;
+        return true;
+    }
+    bool string(std::string& /*val*/) override
+    {
+        ++values;
+        return true;
+    }
+    bool binary(nlohmann::json::binary_t& /*val*/) override
+    {
+        ++values;
+        return true;
+    }
+    bool start_object(std::size_t /*elements*/) override
+    {
+        return true;
+    }
+    bool key(std::string& /*val*/) override
+    {
+        return true;
+    }
+    bool end_object() override
+    {
+        return true;
+    }
+    bool start_array(std::size_t /*elements*/) override
+    {
+        return true;
+    }
+    bool end_array() override
+    {
+        return true;
+    }
+    bool parse_error(std::size_t /*position*/, const std::string& /*last_token*/, const nlohmann::json::exception& ex) override
+    {
+        error_id = ex.id;
+        return false;
+    }
+
+    /// the handler refuses the value that makes this many
+    std::size_t stop_after = (std::numeric_limits<std::size_t>::max)();
+    std::size_t values = 0;
+    int error_id = 0;
+};
+
+/// appends `[$<marker>#l<count>` (nine bytes) to @a v
+inline void append_valueless_array(std::vector<std::uint8_t>& v, const char marker, const std::uint32_t count)
+{
+    v.push_back('[');
+    v.push_back('$');
+    v.push_back(static_cast<std::uint8_t>(marker));
+    v.push_back('#');
+    v.push_back('l');
+    v.push_back(static_cast<std::uint8_t>(count >> 24));
+    v.push_back(static_cast<std::uint8_t>(count >> 16));
+    v.push_back(static_cast<std::uint8_t>(count >> 8));
+    v.push_back(static_cast<std::uint8_t>(count));
+}
+
+/// `[` followed by one `[$<marker>#l<count>` per entry of @a counts and `]`
+inline std::vector<std::uint8_t> valueless_arrays(const char marker, const std::vector<std::uint32_t>& counts)
+{
+    std::vector<std::uint8_t> v = {'['};
+    for (const auto count : counts)
+    {
+        append_valueless_array(v, marker, count);
+    }
+    v.push_back(']');
+    return v;
+}
+} // namespace utils
+
+#if defined(__clang__)
+    #pragma clang diagnostic pop
+#endif

tests/src/unit-ubjson.cpp

@@ -19,12 +19,8 @@
 #include <nlohmann/json.hpp>
 using nlohmann::json;
 
-#include <cstdio>
 #include <iostream>
-#include <sstream>
-#include <utility>
 #include <fstream>
-#include <limits>
 #include <list>
 #include <set>
 #include "make_test_data_available.hpp"
@@ -33,6 +29,7 @@ using nlohmann::json;
 #include "sax_countdown.hpp"
 #include "allocation_counter.hpp"
 #include "sax_event_loggers.hpp"
+#include "ubjson_valueless.hpp"
 using utils::SaxCountdown;
 using utils::SaxEventLogger;
 
@@ -2441,208 +2438,22 @@ TEST_CASE("UBJSON optimized arrays of a valueless type are bounded")
     }
 }
 
-namespace
+TEST_CASE("UBJSON the writer keeps the valueless elements of an input within the base")
 {
-/// SAX consumer that counts the values and keeps the id of a parse error
-struct ValuelessEventCounter : public nlohmann::json_sax<json>
-{
-    bool null() override
-    {
-        return ++values < stop_after;
-    }
-    bool boolean(bool /*val*/) override
-    {
-        return ++values < stop_after;
-    }
-    bool number_integer(json::number_integer_t /*val*/) override
-    {
-        ++values;
-        return true;
-    }
-    bool number_unsigned(json::number_unsigned_t /*val*/) override
-    {
-        ++values;
-        return true;
-    }
-    bool number_float(json::number_float_t /*val*/, const std::string& /*s*/) override
-    {
-        ++values;
-        return true;
-    }
-    bool string(std::string& /*val*/) override
-    {
-        ++values;
-        return true;
-    }
-    bool binary(json::binary_t& /*val*/) override
-    {
-        ++values;
-        return true;
-    }
-    bool start_object(std::size_t /*elements*/) override
-    {
-        return true;
-    }
-    bool key(std::string& /*val*/) override
-    {
-        return true;
-    }
-    bool end_object() override
-    {
-        return true;
-    }
-    bool start_array(std::size_t /*elements*/) override
-    {
-        return true;
-    }
-    bool end_array() override
-    {
-        return true;
-    }
-    bool parse_error(std::size_t /*position*/, const std::string& /*last_token*/, const json::exception& ex) override
-    {
-        error_id = ex.id;
-        return false;
-    }
-
-    /// the handler refuses the value that makes this many
-    std::size_t stop_after = (std::numeric_limits<std::size_t>::max)();
-    std::size_t values = 0;
-    int error_id = 0;
-};
-
-/// appends `[$<marker>#l<count>` (nine bytes) to @a v
-void append_valueless_array(std::vector<std::uint8_t>& v, const char marker, const std::uint32_t count)
-{
-    v.push_back('[');
-    v.push_back('$');
-    v.push_back(static_cast<std::uint8_t>(marker));
-    v.push_back('#');
-    v.push_back('l');
-    v.push_back(static_cast<std::uint8_t>(count >> 24));
-    v.push_back(static_cast<std::uint8_t>(count >> 16));
-    v.push_back(static_cast<std::uint8_t>(count >> 8));
-    v.push_back(static_cast<std::uint8_t>(count));
-}
-
-/// `[` followed by one `[$<marker>#l<count>` per entry of @a counts and `]`
-std::vector<std::uint8_t> valueless_arrays(const char marker, const std::vector<std::uint32_t>& counts)
-{
-    std::vector<std::uint8_t> v = {'['};
-    for (const auto count : counts)
-    {
-        append_valueless_array(v, marker, count);
-    }
-    v.push_back(']');
-    return v;
-}
-} // namespace
-
-TEST_CASE("UBJSON the valueless elements of a whole input are bounded")
-{
-    // The per-container limit alone let nine bytes (`[$Z#l` and a count of
-    // 1048576) announce a million values, and every further nine bytes another
-    // million. The elements of all containers of type 'Z', 'T' or 'F' in one
-    // input are now added up: they may not exceed 1048576 plus 4 for every
-    // byte read when the container's header has been read.
-    //
-    // The tests count events with a SAX handler where they can: building and
-    // comparing values of a million elements is what makes them slow under
-    // Valgrind, so there is one rejection and one round trip with a DOM.
+    // The budget of the reader is 1048576 plus a rate per byte read, see
+    // unit-ubjson_valueless.cpp, which tests it for several values of the rate.
+    // The writer does not use the rate and stays within the base
     json _;
-    const char* const budget_message = "[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive number of elements without payload";
-
-    // two arrays that exceed the limit: the second one, with its header read
-    // at byte 19, may hold 4 * 19 = 76 values on top of the base of 1048576
-    const std::uint32_t over_first = 524288;
-    const std::uint32_t over_second = 1048576 + 76 - over_first + 1;
-
-    SECTION("the input of the original report is rejected")
-    {
-        // 362 bytes (40 arrays of 1048576 values) used to allocate 661 MB; the
-        // second array is the one that is refused
-        const auto input = valueless_arrays('Z', {over_first, over_second});
-        CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), budget_message, json::out_of_range&);
-    }
-
-    SECTION("every input adapter applies the budget")
-    {
-        const auto input = valueless_arrays('Z', {over_first, over_second});
-
-        // stream; not strict, which does not matter
-        std::stringstream stream;
-        stream.write(reinterpret_cast<const char*>(input.data()), static_cast<std::streamsize>(input.size()));
-        ValuelessEventCounter from_stream;
-        CHECK(!json::sax_parse(stream, &from_stream, json::input_format_t::ubjson, false));
-        CHECK(from_stream.error_id == 408);
-
-        // a container that is not contiguous; the counts of arrays of
-        // different types are added up
-        std::vector<std::uint8_t> mixed = {'['};
-        append_valueless_array(mixed, 'T', over_first);
-        append_valueless_array(mixed, 'F', over_second);
-        mixed.push_back(']');
-        const std::list<std::uint8_t> as_list(mixed.begin(), mixed.end());
-        ValuelessEventCounter from_list;
-        CHECK(!json::sax_parse(as_list.begin(), as_list.end(), &from_list, json::input_format_t::ubjson));
-        CHECK(from_list.error_id == 408);
-        CHECK(from_list.values == over_first);
-
-        // FILE*
-        FILE* const file = std::tmpfile(); // NOLINT(cppcoreguidelines-owning-memory)
-        REQUIRE(file != nullptr);
-        CHECK(std::fwrite(input.data(), 1, input.size(), file) == input.size());
-        std::rewind(file);
-        ValuelessEventCounter from_file;
-        CHECK(!json::sax_parse(file, &from_file, json::input_format_t::ubjson));
-        CHECK(from_file.error_id == 408);
-        std::fclose(file); // NOLINT(cert-err33-c,cppcoreguidelines-owning-memory)
-    }
-
-    SECTION("the limit is exact and the bytes read raise it")
-    {
-        // [ [$Z#l 524288] [$U#l 4096 + 4096 bytes] [$Z#l count] ]: chars_read
-        // is 1 + 9 + (9 + 4096) + 9 = 4124 when the last header has been read,
-        // so the last array may hold 4 * 4124 = 16496 values on top of the
-        // base of 1048576 less the 524288 of the first array
-        const std::uint32_t last_allowed = 1048576 + 16496 - 524288;
-        for (const std::uint32_t count :
-                {
-                    last_allowed, last_allowed + 1
-                })
-        {
-            CAPTURE(count)
-            std::vector<std::uint8_t> input = {'['};
-            append_valueless_array(input, 'Z', 524288);
-            input.insert(input.end(), {'[', '$', 'U', '#', 'l', 0x00, 0x00, 0x10, 0x00});
-            input.insert(input.end(), 4096, 0x01);
-            append_valueless_array(input, 'Z', count);
-            input.push_back(']');
-            ValuelessEventCounter sax;
-            const bool accepted = count == last_allowed;
-            CHECK(json::sax_parse(input, &sax, json::input_format_t::ubjson) == accepted);
-            CHECK(sax.error_id == (accepted ? 0 : 408));
-            CHECK(sax.values == 524288u + 4096u + (accepted ? count : 0u));
-        }
-
-        // an array of 1048576 values on its own is accepted as before
-        const std::vector<std::uint8_t> single = {'[', '$', 'Z', '#', 'l', 0x00, 0x10, 0x00, 0x00};
-        // (the handler ends the parse at the first value, so that the million
-        // values are not read)
-        ValuelessEventCounter sax;
-        sax.stop_after = 1;
-        CHECK(!json::sax_parse(single, &sax, json::input_format_t::ubjson));
-        CHECK(sax.error_id == 0);
-        CHECK(sax.values == 1);
-    }
 
     SECTION("to_ubjson writes the typed form up to the base only")
     {
         // 1048 rows of 1000 values fit under the base of 1048576, the 1049th
         // would not: its elements are written one by one. Written all typed,
-        // 1100 rows of 6 bytes would be refused by a reader (1075 rows are the
-        // most it accepts). The rows hold null, true or false, and the
-        // fallback does not depend on which
+        // 1100 rows of 6 bytes would be refused by a reader with the default
+        // rate of the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE (1074 rows
+        // are the most it accepts); the writer does not follow the macro, since
+        // the limit of a reader is never below the base. The rows hold null,
+        // true or false, and the fallback does not depend on which
         std::vector<std::uint8_t> encoded;
         {
             json table = json::array();
@@ -2658,7 +2469,7 @@ TEST_CASE("UBJSON the valueless elements of a whole input are bounded")
         CHECK(encoded.size() == 7 + (1048 * 6) + (52 * (4 + 1000)));
 
         // the output is read back
-        ValuelessEventCounter sax;
+        utils::ValuelessEventCounter sax;
         CHECK(json::sax_parse(encoded, &sax, json::input_format_t::ubjson));
         CHECK(sax.error_id == 0);
         CHECK(sax.values == 1100u * 1000u);
@@ -2685,7 +2496,7 @@ TEST_CASE("UBJSON the valueless elements of a whole input are bounded")
         // an object member costs a key, so the count is bounded by the input
         // and is not charged to the budget
         const std::vector<std::uint8_t> input = {'{', '$', 'Z', '#', 'l', 0x00, 0x10, 0x00, 0x00};
-        ValuelessEventCounter sax;
+        utils::ValuelessEventCounter sax;
         CHECK(!json::sax_parse(input, &sax, json::input_format_t::ubjson));
         CHECK(sax.error_id == 110);
         CHECK(sax.values == 0);

tests/src/unit-ubjson_valueless.cpp

@@ -0,0 +1,324 @@
+//     __ _____ _____ _____
+//  __|  |   __|     |   | |  JSON for Modern C++ (supporting code)
+// |  |  |__   |  |  | | | |  version 3.12.0
+// |_____|_____|_____|_|___|  https://github.com/nlohmann/json
+//
+// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
+// SPDX-License-Identifier: MIT
+
+// The budget for the elements of the UBJSON containers of a valueless type
+// ('Z', 'T', 'F') of a whole input: 1048576 plus
+// JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE for every byte read (see
+// docs/mkdocs/docs/api/macros/json_ubjson_valueless_elements_per_byte.md).
+//
+// The file is built several times, with the macro defined to other values (see
+// tests/CMakeLists.txt), and every boundary below is computed from the macro.
+// What it can test depends on the value: with a rate of at most 10000 the
+// budget can be reached with the few bytes of a test, so the tests exceed it
+// at the exact byte; with a rate of 131072 or more the budget cannot be reached
+// by an input of any size (see the macro's page: 2^20 / 8 = 131072 elements per
+// byte is what the shortest header of a container of the largest size allows), and
+// the tests check that nothing is refused, in particular that the limit does
+// not wrap around for a rate near the largest integer. The input is read with
+// a SAX handler that counts the values, because a DOM of a million values is
+// what makes such a test slow under Valgrind.
+
+#include "doctest_compatibility.h"
+
+#include <nlohmann/json.hpp>
+using nlohmann::json;
+
+#include <cstddef> // size_t
+#include <cstdio> // FILE, tmpfile, fwrite, fseek, fclose
+#include <cstdint> // uint8_t, uint32_t
+#include <list>
+#include <sstream>
+#include <vector>
+
+#include "ubjson_valueless.hpp"
+using utils::ValuelessEventCounter;
+using utils::valueless_arrays;
+
+#ifndef JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE
+    #error "this test needs the macro: compile it with JSON_TEST_KEEP_MACROS (see tests/CMakeLists.txt)"
+#endif
+
+namespace
+{
+/// the base of the budget and the limit of a single container: both 2^20
+constexpr std::uint32_t budget_base = 1048576;
+
+#if JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE <= 10000 || JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE >= 131072
+/// what version 3.12.0 wrote for a table of @a rows (at most 32767) rows of
+/// @a width nulls with to_ubjson(table, true, true): `[$[#I` and the count of
+/// the rows (7 bytes), then a row as `$Z#` and its count (`U` and 1 byte up to
+/// 255, `I` and 2 bytes up to 32767, `l` and 4 bytes above): 5, 6 or 8 bytes
+std::vector<std::uint8_t> rows_3_12_0(const std::uint32_t rows, const std::uint32_t width)
+{
+    std::vector<std::uint8_t> v = {'[', '$', '[', '#', 'I', static_cast<std::uint8_t>(rows >> 8), static_cast<std::uint8_t>(rows)};
+    for (std::uint32_t row = 0; row < rows; ++row)
+    {
+        v.push_back('$');
+        v.push_back('Z');
+        v.push_back('#');
+        if (width <= 255)
+        {
+            v.push_back('U');
+        }
+        else if (width <= 32767)
+        {
+            v.push_back('I');
+            v.push_back(static_cast<std::uint8_t>(width >> 8));
+        }
+        else
+        {
+            v.push_back('l');
+            v.push_back(static_cast<std::uint8_t>(width >> 24));
+            v.push_back(static_cast<std::uint8_t>(width >> 16));
+            v.push_back(static_cast<std::uint8_t>(width >> 8));
+        }
+        v.push_back(static_cast<std::uint8_t>(width));
+    }
+    return v;
+}
+#endif
+} // namespace
+
+// the macro must not be a character or bool: every character type is refused
+static_assert(nlohmann::detail::valueless_elements_rate_is_char_or_bool<bool>::value, "bool is refused");
+static_assert(nlohmann::detail::valueless_elements_rate_is_char_or_bool<char>::value, "char is refused");
+static_assert(nlohmann::detail::valueless_elements_rate_is_char_or_bool<signed char>::value, "signed char is refused");
+static_assert(nlohmann::detail::valueless_elements_rate_is_char_or_bool<unsigned char>::value, "unsigned char is refused");
+static_assert(nlohmann::detail::valueless_elements_rate_is_char_or_bool<wchar_t>::value, "wchar_t is refused");
+static_assert(nlohmann::detail::valueless_elements_rate_is_char_or_bool<char16_t>::value, "char16_t is refused");
+static_assert(nlohmann::detail::valueless_elements_rate_is_char_or_bool<char32_t>::value, "char32_t is refused");
+#if defined(__cpp_char8_t)
+    static_assert(nlohmann::detail::valueless_elements_rate_is_char_or_bool<char8_t>::value, "char8_t is refused");
+#endif
+static_assert(!nlohmann::detail::valueless_elements_rate_is_char_or_bool<int>::value, "int is accepted");
+static_assert(!nlohmann::detail::valueless_elements_rate_is_char_or_bool<unsigned long long>::value, "unsigned long long is accepted");
+
+TEST_CASE("UBJSON the limit of a single container does not depend on the macro")
+{
+#if !defined(JSON_NOEXCEPTION)
+    json _;
+
+    SECTION("a container of more than 1048576 elements is refused")
+    {
+        // the first check of every container; its message differs from that of
+        // the budget, so that it is clear which limit was exceeded
+        for (const char marker :
+                {'Z', 'T', 'F'
+                })
+        {
+            const auto input = valueless_arrays(marker, {budget_base + 1});
+            CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), "[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive array size", json::out_of_range&);
+        }
+    }
+#endif
+
+    SECTION("a container of 1048576 elements is accepted at the start of an input")
+    {
+        // whatever the rate: the base is the per-container limit. The handler
+        // ends the parse at the first value, so that the million values are
+        // not read
+        const std::vector<std::uint8_t> single = {'[', '$', 'Z', '#', 'l', 0x00, 0x10, 0x00, 0x00};
+        ValuelessEventCounter sax;
+        sax.stop_after = 1;
+        CHECK(!json::sax_parse(single, &sax, json::input_format_t::ubjson));
+        CHECK(sax.error_id == 0);
+        CHECK(sax.values == 1);
+    }
+}
+
+#if JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE <= 10000
+
+namespace
+{
+/// the rate of the build; at most 10000 here, which the boundaries rely on
+constexpr std::uint32_t rate = JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE;
+
+/// elements of the first of two containers; the second one is then refused at
+/// its header, which is read at byte 1 + 9 + 9 = 19
+constexpr std::uint32_t first_count = 524288;
+
+/// the largest count the second container may have at byte 19: base plus rate
+/// for each of the 19 bytes, less what the first one used. For a rate of at
+/// most 27594, the largest that keeps this within the limit of one container
+constexpr std::uint32_t second_allowed = budget_base - first_count + (19 * rate);
+} // namespace
+
+TEST_CASE("UBJSON the valueless elements of a whole input are bounded")
+{
+    // The per-container limit alone let nine bytes (`[$Z#l` and a count of
+    // 1048576) announce a million values, and every further nine bytes another
+    // million. The elements of all containers of type 'Z', 'T' or 'F' in one
+    // input are added up: they may not exceed 1048576 plus
+    // JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE for every byte read when the
+    // container's header has been read.
+#if !defined(JSON_NOEXCEPTION)
+    json _;
+    const char* const budget_message = "[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive number of elements without payload";
+#endif
+
+    SECTION("the input of the original report is rejected")
+    {
+        // the second array is the first one that is refused, by one element
+#if !defined(JSON_NOEXCEPTION)
+        CHECK_THROWS_WITH_AS(_ = json::from_ubjson(valueless_arrays('Z', {first_count, second_allowed + 1})), budget_message, json::out_of_range&);
+#endif
+
+        // and is accepted with one element less
+        ValuelessEventCounter sax;
+        CHECK(json::sax_parse(valueless_arrays('Z', {first_count, second_allowed}), &sax, json::input_format_t::ubjson));
+        CHECK(sax.values == first_count + second_allowed);
+    }
+
+    SECTION("every input adapter applies the budget")
+    {
+        const auto input = valueless_arrays('Z', {first_count, second_allowed + 1});
+
+        // stream; not strict, which does not matter
+        std::stringstream stream;
+        stream.write(reinterpret_cast<const char*>(input.data()), static_cast<std::streamsize>(input.size()));
+        ValuelessEventCounter from_stream;
+        CHECK(!json::sax_parse(stream, &from_stream, json::input_format_t::ubjson, false));
+        CHECK(from_stream.error_id == 408);
+
+        // a container that is not contiguous; the counts of arrays of
+        // different types are added up
+        std::vector<std::uint8_t> mixed = {'['};
+        utils::append_valueless_array(mixed, 'T', first_count);
+        utils::append_valueless_array(mixed, 'F', second_allowed + 1);
+        mixed.push_back(']');
+        const std::list<std::uint8_t> as_list(mixed.begin(), mixed.end());
+        ValuelessEventCounter from_list;
+        CHECK(!json::sax_parse(as_list.begin(), as_list.end(), &from_list, json::input_format_t::ubjson));
+        CHECK(from_list.error_id == 408);
+        CHECK(from_list.values == first_count);
+
+        // FILE*
+        FILE* const file = std::tmpfile(); // NOLINT(cppcoreguidelines-owning-memory)
+        REQUIRE(file != nullptr);
+        CHECK(std::fwrite(input.data(), 1, input.size(), file) == input.size());
+        CHECK(std::fseek(file, 0, SEEK_SET) == 0);
+        ValuelessEventCounter from_file;
+        CHECK(!json::sax_parse(file, &from_file, json::input_format_t::ubjson));
+        CHECK(from_file.error_id == 408);
+        std::fclose(file); // NOLINT(cert-err33-c,cppcoreguidelines-owning-memory)
+    }
+
+    SECTION("the limit is exact and the bytes read raise it")
+    {
+        // [ [$Z#l first_count] [$U#l payload + payload bytes] [$Z#l count] ]:
+        // chars_read is 1 + 9 + (9 + payload) + 9 = 28 + payload when the last
+        // header has been read, so the last array may hold
+        // rate * (28 + payload) values on top of the base less the
+        // first_count of the first array. The payload is as large as the
+        // limit of one container allows, but at most 4096
+        constexpr std::uint32_t payload_limit = (524288 / rate) - 40;
+        constexpr std::uint32_t payload = payload_limit < 4096 ? payload_limit : 4096;
+        constexpr std::uint32_t last_allowed = budget_base - first_count + (rate * (28 + payload));
+        static_assert(last_allowed < budget_base, "the count must be accepted by the limit of one container");
+
+        for (const std::uint32_t count :
+                {
+                    last_allowed, last_allowed + 1
+                })
+        {
+            CAPTURE(count)
+            std::vector<std::uint8_t> input = {'['};
+            utils::append_valueless_array(input, 'Z', first_count);
+            input.insert(input.end(), {'[', '$', 'U', '#', 'l',
+                                       static_cast<std::uint8_t>(payload >> 24), static_cast<std::uint8_t>(payload >> 16),
+                                       static_cast<std::uint8_t>(payload >> 8), static_cast<std::uint8_t>(payload)
+                                      });
+            input.insert(input.end(), payload, 0x01);
+            utils::append_valueless_array(input, 'Z', count);
+            input.push_back(']');
+            ValuelessEventCounter sax;
+            const bool accepted = count == last_allowed;
+            CHECK(json::sax_parse(input, &sax, json::input_format_t::ubjson) == accepted);
+            CHECK(sax.error_id == (accepted ? 0 : 408));
+            CHECK(sax.values == first_count + payload + (accepted ? count : 0u));
+        }
+    }
+
+    SECTION("a table written by 3.12.0")
+    {
+        // Version 3.12.0 wrote a table of rows of nulls with one `$Z#` header
+        // per row (see the macro's page). A row of `width` elements has a
+        // header of 6 bytes up to 32767 elements (5 up to 255), so the budget
+        // grows by 6 * rate per row, and the row brings `width`. The 7 bytes
+        // of the outer header are read first. Row r is accepted while
+        //     r * width <= 1048576 + rate * (7 + 6 * r),
+        // which is r <= (1048576 + 7 * rate) / (width - 6 * rate) for a width
+        // above 6 * rate, and always true otherwise.
+        constexpr std::uint32_t width = 1000;
+        constexpr std::uint32_t rows = 1200;
+        constexpr std::uint32_t credit = 6 * rate;
+        constexpr std::uint32_t accepted_rows = width > credit ? (budget_base + (7 * rate)) / (width - credit) : rows;
+        constexpr std::uint32_t read_rows = accepted_rows < rows ? accepted_rows : rows;
+
+        ValuelessEventCounter sax;
+        CHECK(json::sax_parse(rows_3_12_0(rows, width), &sax, json::input_format_t::ubjson) == (accepted_rows >= rows));
+        CHECK(sax.error_id == (accepted_rows >= rows ? 0 : 408));
+        CHECK(sax.values == static_cast<std::size_t>(read_rows) * width);
+    }
+}
+
+#elif JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE >= 131072
+
+TEST_CASE("UBJSON a rate that cannot be reached switches the budget off")
+{
+    // The rate is so large (2^20 / 8 = 131072 elements per byte or more)
+    // that no input can exceed the budget: the shortest header of a container
+    // of the largest size is 8 bytes (`$Z#l` and a count of 4 bytes, in an
+    // array of arrays). The limit is base + rate * bytes read, which wraps
+    // around for a rate near the largest integer in 64 and in 32
+    // bits already at the first byte, unless it saturates: a wrapped limit
+    // would refuse the second container below (at byte 19 or at byte 10, where
+    // a limit of 2^20 + 10 * rate wraps for a 32-bit rate of 0x7FFFFFFF).
+#if !defined(JSON_NOEXCEPTION)
+    json _;
+#endif
+
+    SECTION("containers of the largest size are accepted one after the other")
+    {
+        // the first container is accepted by the base alone, the second one is
+        // not: it needs the rate. The handler ends the parse at its first
+        // value, so that the second million values are not read
+        const auto input = valueless_arrays('Z', {budget_base, budget_base});
+        ValuelessEventCounter sax;
+        sax.stop_after = static_cast<std::size_t>(budget_base) + 1;
+        CHECK(!json::sax_parse(input, &sax, json::input_format_t::ubjson));
+        CHECK(sax.error_id == 0);
+        CHECK(sax.values == static_cast<std::size_t>(budget_base) + 1);
+    }
+
+    SECTION("the limit of a single container still applies")
+    {
+        const auto input = valueless_arrays('T', {budget_base, budget_base + 1});
+        ValuelessEventCounter sax;
+        CHECK(!json::sax_parse(input, &sax, json::input_format_t::ubjson));
+        CHECK(sax.error_id == 408);
+        CHECK(sax.values == budget_base);
+#if !defined(JSON_NOEXCEPTION)
+        CHECK_THROWS_WITH_AS(_ = json::from_ubjson(valueless_arrays('F', {budget_base + 1})), "[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive array size", json::out_of_range&);
+#endif
+    }
+
+    SECTION("a table written by 3.12.0 is read completely")
+    {
+        // 1200 rows of 1000 values are refused after 1074 rows at the default
+        // rate and are accepted with every rate of 167 and more
+        ValuelessEventCounter sax;
+        CHECK(json::sax_parse(rows_3_12_0(1200, 1000), &sax, json::input_format_t::ubjson));
+        CHECK(sax.error_id == 0);
+        CHECK(sax.values == 1200u * 1000u);
+    }
+}
+
+// (rates from 10001 to 131071 can be reached, but not with the few bytes of a
+// test: there is nothing to test for them here)
+#endif