Bound the total number of valueless UBJSON elements per input

An optimized UBJSON array of type Z, T or F has no payload, so its
declared count is not bounded by the input. The limit of 1048576 applied
to each array on its own, so 9 bytes (`[$Z#l` and a count) produced
1048576 values and every further 9 bytes another 1048576: 362 bytes
allocated 661 MB in a DOM, and a SAX handler received 1M events per 9
bytes.

The reader now adds the counts of all such arrays of one input and
rejects the input with out_of_range.408 when the sum exceeds 1048576
plus 4 for every byte read so far. The base is the per-array limit, so
an array that was accepted on its own before is still accepted at the
start of an input. These arrays add about 80 bytes of memory per input
byte (measured 77; one `Z` per byte needs 32), plus the base; the memory
of a document as a whole is not bounded by this. The check runs once per
valueless array, before its start event, with plain size_t arithmetic
that cannot overflow for a 32-bit size_t. Containers with a payload type
and objects (every member needs a key) are bounded by the input already
and are unchanged. BJData rejects these types.

to_ubjson counts the elements of the typed arrays it writes in one call
and writes an array in the unoptimized form (`[#n Z Z ...`) once it would
bring the total above 1048576, which a reader accepts at any input size.
The check is made only for a typed array of a valueless type, so BJData
and other arrays do not pay for it. Everything to_ubjson writes in
version 3.13.0 can therefore be read back.

Two consequences:
- to_ubjson output changes for values with more than 1048576 elements in
  arrays of type Z, T or F: such an array is written at one byte per
  element, e.g. 270000 rows of 100 nulls take 26.8 MB instead of 1.35 MB.
- Output of version 3.12.0 and earlier (typed arrays without such a
  total) is rejected by from_ubjson when the arrays hold more than
  1048576 elements plus 4 per byte of input. 3.12.0 writes a row of up to 255 nulls in 5 bytes (6 above
  255), so a table is rejected from 1075 rows of 1000 nulls or 209723
  rows of 25 (accepted: 1000 rows of 1000, 209000 rows of 25); rows of
  20 or fewer are never rejected.

Measured on the test data: no file of json_test_data (converted with
to_ubjson in all three modes, 846 round trips) holds an array of this
kind with more than 0.17 values per byte. All of them are read back.

Reproducer (K arrays of 1048576 nulls), Apple clang -O2:
  K=40 (362 bytes): before 41943040 values, 664 MB; after rejected, 33 MB
  K=2 (20 bytes):   before 2097152 values, 53 MB;   after rejected, 33 MB

The new tests count events with a SAX handler where they can: building
and comparing million-element values made the first version take 176 s
under Valgrind, the current one about 22 s.

Behaviour change: an input that holds more than 1048576 elements in
arrays of type Z, T or F at more than 4 per byte is now rejected.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

include/nlohmann/detail/output/binary_writer.hpp

@@ -1204,20 +1204,27 @@ class binary_writer
 
             // an optimized array of a valueless type carries no payload, so a
             // reader has nothing but the declared count to bound the allocation
-            // by and refuses an excessive one. Write the unoptimized form for
-            // those, at one byte per element, so the result can be read back.
-            // Objects are not affected: every element is preceded by its key.
-            const bool valueless_type = (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
-            const bool excessive_valueless = IsArray
-                                             && valueless_type
-                                             && size > detail::max_valueless_container_size;
-
-            if (same_prefix && !excessive_valueless
-                    && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
+            // by. It refuses one array of more than max_valueless_container_size
+            // elements, and all such arrays of one input together when they add
+            // up to more than valueless_elements_base plus a number per byte
+            // read. Write the unoptimized form, at one byte per element, for
+            // an array that would exceed the base with the arrays written so
+            // far, so that the result can be read back: the limit of a reader
+            // is never below the base. Objects are not affected: every element
+            // is preceded by its key.
+            if (same_prefix && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
             {
-                prefix_required = false;
-                oa.write_character(to_char_type('$'));
-                oa.write_character(first_prefix);
+                const bool valueless_type = IsArray && (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
+                if (!valueless_type || size <= detail::valueless_elements_base - valueless_elements_written)
+                {
+                    prefix_required = false;
+                    oa.write_character(to_char_type('$'));
+                    oa.write_character(first_prefix);
+                    if (valueless_type)
+                    {
+                        valueless_elements_written += size;
+                    }
+                }
             }
         }
 
@@ -5441,6 +5448,10 @@ class binary_writer
     /// how to treat a string value or object key that is not valid UTF-8
     /// (CBOR, MessagePack, UBJSON, BJData, and BSON; not BON8)
     const error_handler_t error_handler = binary_writer_default_error_handler();
+
+    /// UBJSON: the number of elements written so far in optimized arrays of a
+    /// valueless type, which a reader adds up and bounds
+    std::size_t valueless_elements_written = 0;
 };
 
 }  // namespace detail