Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget (#6083)

* Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget

The whole-input budget for the elements of optimized UBJSON arrays of
type Z, T and F (1048576 plus 4 per byte read) rejects files written by
3.12.0 that have long arrays of these types, such as 1100 rows of 1000
nulls with a `$Z` annotation, and there was no way to read them. The
per-byte rate is now the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE.

Semantics:
- default 4, so the default build is unchanged (the generated code of
  from_ubjson is identical at -O2 up to source line numbers);
- the value must be a positive integer: 0, negative, non-integral, bool,
  character and wider-than-uintmax_t values fail to compile with a
  static_assert that names the macro;
- a value that does not fit into std::size_t is the largest size_t, so
  one spelling works for 32- and 64-bit targets;
- the limit base + rate * bytes_read saturates at the largest size_t
  instead of wrapping around, for any rate and input size; a rate of
  131072 (2^20 / 8) or more cannot be exceeded by any input (the
  shortest header of an array of more than 32767 elements is 8 bytes),
  which switches the whole-input budget off;
- the limit of one array (1048576) and the base (1048576) stay fixed
  constants, so the writer fallback still follows the base only and a
  reader with any rate reads everything the writer produces;
- the macro is not part of the ABI tag and must have the same value in
  every translation unit; it is documented that way.

The memory the arrays can take in a DOM is about 20 bytes per rate and
byte of input (measured, macOS, containers of 9 * rate elements:
25 bytes per input byte at rate 1, 81 at 4, 312 at 16, 1226 at 64, 5493
at 256) plus about 33 MB for the base. A 3.12.0 row of N values has a
header of b bytes (5 up to 255, 6 up to 32767, 8 above), so a table is
read completely if N <= rate * b, and otherwise up to
(1048576 + h * rate, h = outer header: 6, 7 or 9 bytes) / (N - rate * b) rows; 1100 rows of 1000 nulls need
a rate of 8. The new page of the macro documents the formulas, the
measurements, and two examples that are run by the documentation checks.

The tests of the budget move to tests/src/unit-ubjson_valueless.cpp, which
derives every boundary from the macro and is built with the rates 1,
1000000 and 0xFFFFFFFFFFFFFFFF in addition to the default (the last two
check that nothing is refused and that the limit does not wrap around);
they fail with the unchanged header for rate 1, 1000000 and the maximum.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Refuse char8_t for JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE and clarify its documentation

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

include/nlohmann/detail/output/binary_writer.hpp

@@ -1206,12 +1206,14 @@ class binary_writer
             // reader has nothing but the declared count to bound the allocation
             // by. It refuses one array of more than max_valueless_container_size
             // elements, and all such arrays of one input together when they add
-            // up to more than valueless_elements_base plus a number per byte
-            // read. Write the unoptimized form, at one byte per element, for
-            // an array that would exceed the base with the arrays written so
-            // far, so that the result can be read back: the limit of a reader
-            // is never below the base. Objects are not affected: every element
-            // is preceded by its key.
+            // up to more than valueless_elements_base plus
+            // JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE per byte read. Write the
+            // unoptimized form, at one byte per element, for an array that
+            // would exceed the base with the arrays written so far, so that the
+            // result can be read back: the limit of a reader is never below the
+            // base, whatever the macro is set to, so the writer does not use
+            // it (and a reader with a larger rate reads this output as well).
+            // Objects are not affected: every element is preceded by its key.
             if (same_prefix && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
             {
                 const bool valueless_type = IsArray && (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');