Keep only the beginning and the end of a long token for error messages

For "last read" in parse errors, the lexer keeps the text of the token it
reads. The token start moves only at strings and numbers, so everything
after a string or number belongs to it until the next one starts. Input
that cannot be read again copies that text while parsing (the istream and
FILE* adapters in their carry buffer, the lexer in token_string for
iterators), so valid input of 1 followed by 100,000,000 newlines peaked
at about 225 MB, and a long string was kept a second time.

Replace the carry buffer and token_string by token_text_buffer, which
keeps the first 4096 and at least the last 4112 characters of a token and
drops the characters in between where it would otherwise grow, so
appending stays as cheap as before. get_token_string() shows a token of
more than 8192 characters as its first and last 4096 characters around
"...", the same for every kind of input; contiguous input copies only
these parts.

The example input now peaks at 1.8 MB. Instruction counts of parsing
from std::string, std::istream and FILE* are unchanged (clang 21 and
GCC 15); parsing from std::list<char> needs 1 to 17 % fewer.

Fixes #6120.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/dump.md

@@ -58,9 +58,7 @@ is not UTF-8 encoded and `error_handler` is set to `strict`
 
 ## Complexity
 
-Linear in the size of the result. With an `indent`, every line starts with up to `indent` characters per nesting level,
+Linear.
-so the result can be larger than the value by a factor of its nesting depth (see
-[Parsing untrusted input](../../features/parsing/untrusted_input.md#nesting-depth)).
 
 ## Notes
 

docs/mkdocs/docs/api/basic_json/flatten.md

@@ -18,8 +18,7 @@ Strong exception safety: if an exception occurs, the original value stays intact
 
 ## Complexity
 
-Linear in the size of the result. Every key repeats the path of its value, so the keys can be larger than the value by a
+Linear in the size of the JSON value.
-factor of its nesting depth (see [Parsing untrusted input](../../features/parsing/untrusted_input.md#nesting-depth)).
 
 ## Notes
 

docs/mkdocs/docs/api/basic_json/from_bon8.md

@@ -69,10 +69,8 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 
 ## Exceptions
 
-- Throws [parse_error.110](../../home/exceptions.md#jsonexceptionparse_error110) if the given input ends prematurely,
+- Throws [parse_error.110](../../home/exceptions.md#jsonexceptionparse_error110) if the given input ends prematurely or
-  if the end of the file was not reached when `strict` was set to true, or if a string at the end of the value is not
+  the end of the file was not reached when `strict` was set to true
-  terminated by 0xFF although a byte follows it (also when `strict` is false: the byte that ends the string is
-  taken from the input, so the next value would be lost)
 - Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if a parse error occurs, for instance
   an invalid byte, a string that is not valid UTF-8, or an object key that is not a string
 

docs/mkdocs/docs/api/macros/json_recursion_depth_limit.md

@@ -27,7 +27,7 @@ comparison of two [`json_view`](../basic_json_view/index.md) documents has a fix
 and does not depend on it either.
 
 The call stack that these operations need grows with the limit: roughly *a few kilobytes plus a cost per level times the
-limit*, where the cost per level is up to about 1.8 KiB in an unoptimized and up to about 0.5 KiB in an optimized build
+limit*, where the cost per level is up to about 2.3 KiB in an unoptimized and up to about 0.5 KiB in an optimized build
 (see [Stack needed](#stack-needed)). The default of `128` levels is tuned for threads with ordinary stacks of a few
 hundred kilobytes or more. Set a lower limit if your threads have small stacks (RTOS tasks, fibers, threads of the musl
 C
@@ -67,7 +67,7 @@ When the macro is not defined, the library defines it to its default value.
       the levels that are descended into are counted, so the count never exceeds the limit, not even when user code (a
       copy constructor of a custom base class, say) copies a value while a copy is going on. A byte holds `255` at
       most, and the limit stays below that, so that the count cannot wrap around. (More levels would hardly be
-      useful: `254` levels need up to 448 KiB of stack in an unoptimized build already, twice as much as the
+      useful: `254` levels need up to 576 KiB of stack in an unoptimized build already, twice as much as the
       default.)
 
 ### Stack needed
@@ -83,7 +83,7 @@ Unoptimized (`-O0`):
 
 | Operation | limit 16 | limit 32 | limit 64 | limit 128 | limit 254 |
 |---|---:|---:|---:|---:|---:|
-| `dump()` and `dump(indent)`, `operator<<` | 16 / 16 | 16 / 16 | 32 / 32 | 48 / 48 | 96 / 96 |
+| `dump()` and `dump(indent)`, `operator<<` | 48 / 32 | 80 / 64 | 144 / 128 | 288 / 256 | 576 / 512 |
 | copy construction and assignment | 32 / 16 | 48 / 32 | 112 / 80 | 224 / 144 | 432 / 304 |
 | `==` | 16 / 16 | 16 / 16 | 16 / 16 | 48 / 32 | 96 / 48 |
 | `<=>` (C++20) | 16 / 16 | 16 / 16 | 32 / 32 | 64 / 64 | 144 / 128 |
@@ -123,24 +123,23 @@ The rule of thumb is
 
 > stack needed ≈ a few KiB + cost per level × limit
 
-with a cost per level of up to 1.8 KiB (conversion to another `basic_json` type at `-O0` with Clang; 1.3 KiB with
+with a cost per level of up to 2.3 KiB (`dump` at `-O0` with Clang; 2.1 KiB with GCC), 1.7 KiB for copying and merging
-GCC), 1.7 KiB for copying and merging (1.3 KiB with GCC), 0.4 KiB for `dump`, and 0.1 to 0.5 KiB for most other
+(1.3 KiB with GCC), and 0.1 to 0.5 KiB for most other operations at `-O0`; and 0.5 KiB or less at `-O2`. The numbers
-operations at `-O0`; and 0.5 KiB or less at `-O2`. The numbers come from the difference between the limits `128` and
+come from the difference between the limits `128` and `254` in the tables, so they are rough: other compilers,
-`254` in the tables, so they are rough: other compilers, other targets (the frames of x86-64 code are about as large),
+other targets (the frames of x86-64 code are about as large), other standard libraries, `JSON_DIAGNOSTICS`, and
-other standard libraries, `JSON_DIAGNOSTICS`, and sanitizers change them. Measure with your own tool chain (see the
+sanitizers change them. Measure with your own tool chain (see the example below) rather than relying on the tables.
-example below) rather than relying on the tables. The stack that your own code and the operating system need comes on
+The stack that your own code and the operating system need comes on top of these numbers.
-top of these numbers.
 
 Which limit to choose, for the stack that you can spare for the library (the largest limit of those measured
 for which the slowest operation needs at most that much):
 
 | Stack for the library | optimized (`-O2`) | unoptimized (`-O0`) |
 |---|---:|---:|
-| 32 KiB | 64 | 16 |
+| 32 KiB | 64 | 0 |
-| 64 KiB | 128 | 32 |
+| 64 KiB | 128 | 16 |
-| 128 KiB (the default stack of threads on musl) | 254 | 64 |
+| 128 KiB (the default stack of threads on musl) | 254 | 32 |
-| 256 KiB | 254 | 128 |
+| 256 KiB | 254 | 64 |
-| 512 KiB | 254 | 254 |
+| 512 KiB | 254 | 128 |
 | 1 MiB or more | 254 | 254 |
 
 ### Speed

docs/mkdocs/docs/api/ordered_map.md

@@ -89,21 +89,19 @@ This differs from `#!cpp std::map`, where the same operations are O(log n).
 !!! warning "Quadratic cost of building large objects"
 
     Because every insertion scans all elements inserted so far, building an object of `n` distinct keys costs
-    **O(n²)** in total. This applies to filling an [`ordered_json`](ordered_json.md) object key by key (with
+    **O(n²)** in total. This applies to filling an [`ordered_json`](ordered_json.md) object key by key as well as to
-    `operator[]`, `emplace`, or `insert`) and, for now, to reading one with a binary reader such as
+    parsing one, since the parser inserts each key as it is read.
-    [`from_cbor`](basic_json/from_cbor.md) ([#6104](https://github.com/nlohmann/json/issues/6104)). Parsing JSON text
-    is not affected: the parser finds the keys of an object with many members in a hash table and takes linear time.
 
     The cost is negligible for the object sizes typically found in configuration files or API payloads, but it grows
-    steeply for machine-generated objects with many thousands of keys. Measured with Apple clang 21, `-O2 -DNDEBUG`,
+    steeply for machine-generated objects with many thousands of keys. Measured with `-O2 -DNDEBUG` for parsing a flat
-    arm64, for a flat object of `n` keys (best of seven runs):
+    object of `n` keys, relative to `#!cpp nlohmann::json` (which uses `#!cpp std::map`):
 
-    | `n`    | `json::parse` | `ordered_json::parse` | `ordered_json`, key by key | `ordered_json::from_cbor` |
+    | `n`    | `json` | `ordered_json` | factor |
-    |--------|---------------|-----------------------|----------------------------|---------------------------|
+    |--------|--------|----------------|--------|
-    | 2000   | 0.3 ms        | 0.1 ms                | 4.5 ms                     | 4.4 ms                    |
+    | 2000   | 0.7 ms | 3.6 ms         | 5×     |
-    | 4000   | 0.5 ms        | 0.2 ms                | 17.1 ms                    | 16.8 ms                   |
+    | 4000   | 0.8 ms | 14.0 ms        | 19×    |
-    | 8000   | 0.9 ms        | 0.3 ms                | 67.0 ms                    | 66.8 ms                   |
+    | 8000   | 1.6 ms | 67.8 ms        | 43×    |
-    | 16 000 | 2.0 ms        | 0.7 ms                | 282.4 ms                   | 282.4 ms                  |
+    | 16 000 | 3.3 ms | 181.6 ms       | 54×    |
 
     If key order matters for objects of that size, consider a container with a lookup index, such as
     [`nlohmann::fifo_map`](https://github.com/nlohmann/fifo_map)

docs/mkdocs/docs/features/binary_formats/bon8.md

@@ -181,10 +181,7 @@ of the message are reported as [parse_error.110](../../home/exceptions.md#jsonex
     arrays and objects with up to four elements that are terminated by 0xFE, unsorted object keys, or a 0xFF after a
     string that would also end without it, are accepted. A second 0xFF is not a terminator but an empty string.
 
-    Strings must be valid UTF-8, and a string at the very end of a message must be terminated by 0xFF. This also
+    Strings must be valid UTF-8, and a string at the very end of a message must be terminated by 0xFF.
-    holds when `strict` is false and more data follows the value: the byte that would end the string belongs to the
-    next value, and reading it would remove it from the input, so the missing 0xFF is reported as
-    [parse_error.110](../../home/exceptions.md#jsonexceptionparse_error110).
 
     Duplicate object keys are accepted; the last value wins (`ordered_json` keeps the position of the first occurrence).
     The specification does not say how to treat them; the HikoGUI reference decoder keeps the first value.

docs/mkdocs/docs/features/parsing/untrusted_input.md

@@ -64,13 +64,13 @@ stack for the nesting.)
 
 The call stack still needs room for the levels that are recursed into, and the default limit assumes a stack of ordinary
 size. The stack that an operation needs is roughly proportional to the limit; with the default of 128 levels, the most
-demanding operations (copying a value and converting it to another `basic_json` type, in an unoptimized build) needed
+demanding operation (`dump` in an unoptimized build) needed up to 288 KiB, and an optimized build needed 48 KiB or less,
-up to 224 KiB, and an optimized build needed 48 KiB or less, in measurements on macOS with arm64. The most demanding
+in measurements on macOS with arm64. The most demanding operation needed this much stack at the limits `0`, `32`,
-operation needed this much stack at the limits `0`, `32`, `128`, and `254`:
+`128`, and `254`:
 
 | Stack needed, in KiB | limit 0 | limit 32 | limit 128 (default) | limit 254 |
 |---|---:|---:|---:|---:|
-| unoptimized build (`-O0`) | 16 or less | 64 | 224 | 448 |
+| unoptimized build (`-O0`) | 16 or less | 80 | 288 | 576 |
 | optimized build (`-O2`) | 16 or less | 16 or less | 48 | 112 |
 
 A thread with a smaller stack (a task of a real-time operating system, a fiber, or a thread of the musl C library, whose
@@ -81,20 +81,6 @@ result. See the [documentation of the macro](../../api/macros/json_recursion_dep
 the speed, and the rules for choosing a value. Destroying a value (its destructor) recurses over the same number of
 levels and then continues without the call stack, and needs 32 KiB or less at every limit.
 
-!!! warning "Indented output and flattened keys grow with the depth"
-
-    Some results are larger than the value by a factor of its nesting depth.
-    [`dump`](../../api/basic_json/dump.md) with an `indent` of `n` (and likewise `operator<<` with `std::setw(n)` and
-    [`basic_json_view::dump`](../../api/basic_json_view/dump.md)) writes up to `n` characters per nesting level in
-    front of every line, and [`flatten`](../../api/basic_json/flatten.md) repeats the path of every value in its key.
-    For the 80,001-byte array `[1,[1,...[1,1]...]]` with 20,000 levels, `dump(1)` returns 600 MB, `dump(4)` 2.4 GB,
-    and the keys of `flatten()` hold 400 MB. A compact `dump()` without indentation stays the size of the value. If you
-    pretty-print, log, or flatten untrusted input, limit its nesting depth while parsing, for example with a
-    [parser callback](parser_callbacks.md#recipe-max-nesting-depth-via-a-callback): with a limit of `d`, every line has
-    at most `n * d` indentation characters, and every flattened key has at most `d` reference tokens.
-    [`JSON_RECURSION_DEPTH_LIMIT`](../../api/macros/json_recursion_depth_limit.md) does not help here: it changes only
-    how much of the call stack a walk uses, never its result, and it does not limit the depth that `parse()` accepts.
-
 ## Input size
 
 The library does not limit the overall size of a JSON text; a value nested or wide enough will use memory
@@ -154,20 +140,6 @@ values for a repeated key ends up in the parsed object. If your application must
 silently resolving them one way or another, see the
 [parser callback recipe for rejecting duplicate keys](parser_callbacks.md#recipe-rejecting-duplicate-object-keys).
 
-## Objects with many keys in `ordered_json`
-
-[`ordered_json`](../../api/ordered_json.md) keeps the members of an object in a vector in insertion order
-([`ordered_map`](../../api/ordered_map.md)) and finds a key by searching all members. Parsing JSON text into an
-`ordered_json` avoids that search for objects with many keys and takes linear time. The binary readers
-([`from_cbor`](../../api/basic_json/from_cbor.md), [`from_msgpack`](../../api/basic_json/from_msgpack.md),
-[`from_ubjson`](../../api/basic_json/from_ubjson.md), [`from_bjdata`](../../api/basic_json/from_bjdata.md),
-[`from_bson`](../../api/basic_json/from_bson.md), and [`from_bon8`](../../api/basic_json/from_bon8.md)) do not yet:
-reading an object of `n` keys into an `ordered_json` costs time proportional to `n²`. A flat object of 40,000 keys
-(about 300 KB of CBOR) takes about 1.3 s, against a few milliseconds as JSON text or into a `json`
-([#6104](https://github.com/nlohmann/json/issues/6104)). `json` is not affected. If you read untrusted binary input
-into `ordered_json`, bound the input size more tightly, read it into a `json`, or reject objects with many keys in a
-[SAX handler](sax_interface.md).
-
 ## Numbers
 
 A number whose value cannot be represented — for instance `1E1000`, which overflows `double` — is rejected while
@@ -199,11 +171,9 @@ indices (or the pointer as a whole) before using them, or use [`at`](../../api/b
 
 - Wrap parsing in a `#!cpp try`/`#!cpp catch` block, or use `allow_exceptions=false`/`accept()` if your environment
   cannot use exceptions; never let `JSON_NOEXCEPTION`'s `abort()` be the first time you think about error handling.
-- If the input's nesting depth matters to you (for example because you pretty-print, log, or flatten it), enforce your
+- If the input's nesting depth matters to you, enforce your own limit with a
-  own limit with a [parser callback](parser_callbacks.md#recipe-max-nesting-depth-via-a-callback) or a
+  [parser callback](parser_callbacks.md#recipe-max-nesting-depth-via-a-callback) or a
   [SAX handler](sax_interface.md); the library bounds the call stack but not memory use.
-- If you read untrusted binary input into `ordered_json`, mind that objects with many keys are
-  [slow to read](#objects-with-many-keys-in-ordered_json).
 - Bound the size of the input itself before parsing, independent of the library's own bounded allocations for binary
   format lengths.
 - Decide up front how you want a string with invalid UTF-8 — from any source, not only parsing — to be serialized

docs/mkdocs/docs/home/exceptions.md

@@ -140,6 +140,8 @@ Exceptions have ids 1xx.
 
 This error indicates a syntax error while deserializing a JSON text. The error message describes that an unexpected token (character) was encountered, and the member `byte` indicates the error position.
 
+The message ends with the last read token (`last read: '...'`), with control characters written as `<U+XXXX>`. A token of more than 8192 characters is shown by its first and its last 4096 characters, separated by `...`.
+
 !!! failure "Example message"
 
     Input ended prematurely:

include/nlohmann/detail/input/binary_reader.hpp

@@ -177,17 +177,6 @@ class binary_reader
                 JSON_ASSERT(false); // NOLINT(cert-dcl03-c,hicpp-static-assert,misc-static-assert) LCOV_EXCL_LINE
         }
 
-        // BON8: a string ends at the first byte that cannot continue it. If
-        // that byte follows the value, the reader has already taken it from
-        // the input and holds it in bon8_pushback. Without the end-of-input
-        // check below it would be dropped, and the next read would start
-        // behind it, so report the missing 0xFF instead.
-        if (result && !strict && input_format == input_format_t::bon8 && bon8_pushback_size != 0)
-        {
-            get_bon8();
-            return last_byte_error(exception_id::unexpected_end_of_input, "string at the end of the value is not terminated by 0xFF", "value", "");
-        }
-
         // strict mode: next byte must be EOF
         if (result && strict)
         {

include/nlohmann/detail/input/input_adapters.hpp

@@ -110,87 +110,164 @@ inline const char* empty_input_message() noexcept
     return "attempting to parse an empty input; check that your input string or stream contains the expected JSON";
 }
 
-////////////////////
-// input adapters //
-////////////////////
-
-#ifndef JSON_NO_IO
 /*!
+@brief the characters of the last read token, kept for error messages
+
 A growable array of characters for the part of a token that an input adapter
-copied out of its window: unlike std::vector<char>::push_back(), appending a
+copied out of its window, or that the lexer copies while it scans input it
-character takes no reference to a local, and is a few instructions when there
+cannot read again: unlike std::vector::push_back(), appending a character
-is room.
+takes no reference to a local, and is a few instructions when there is room.
+
+The token is only needed for error messages, which show its first head()
+characters and its last head() characters (see lexer::get_token_string). The
+buffer therefore keeps the first head() and at least the last tail()
+characters: when it is full and holds more than limit() characters, it drops
+the characters in between instead of growing. That happens where it would
+otherwise reallocate, so appending stays as cheap as before, and a long string
+or a long run of whitespace read from a stream is not kept a second time.
 */
-class token_carry_buffer
+template<typename CharType>
+class token_text_buffer
 {
   public:
-    token_carry_buffer() = default;
+    static constexpr std::size_t head() noexcept
-    token_carry_buffer(const token_carry_buffer&) = delete;
+    {
-    token_carry_buffer& operator=(const token_carry_buffer&) = delete;
+        return 4096;
-    token_carry_buffer(token_carry_buffer&& rhs) noexcept
+    }
+    /// a little more than head(), as the last character may be handed back
+    static constexpr std::size_t tail() noexcept
+    {
+        return head() + 16;
+    }
+    static constexpr std::size_t limit() noexcept
+    {
+        return head() + (2 * tail());
+    }
+
+    token_text_buffer() = default;
+    token_text_buffer(const token_text_buffer&) = delete;
+    token_text_buffer& operator=(const token_text_buffer&) = delete;
+    token_text_buffer(token_text_buffer&& rhs) noexcept
         : m_data(std::move(rhs.m_data)), m_end(rhs.m_end), m_capacity_end(rhs.m_capacity_end)
     {
         rhs.m_end = rhs.m_capacity_end = nullptr;
     }
-    token_carry_buffer& operator=(token_carry_buffer&&) = delete;
+    token_text_buffer& operator=(token_text_buffer&& rhs) noexcept
-    ~token_carry_buffer() = default;
+    {
+        m_data = std::move(rhs.m_data);
+        m_end = rhs.m_end;
+        m_capacity_end = rhs.m_capacity_end;
+        rhs.m_end = rhs.m_capacity_end = nullptr;
+        return *this;
+    }
+    ~token_text_buffer() = default;
 
     void clear() noexcept
     {
         m_end = m_data.get();
     }
 
-    void push_back(char c)
+    bool empty() const noexcept
+    {
+        return m_end == m_data.get();
+    }
+
+    void push_back(CharType c)
     {
         if (JSON_HEDLEY_UNLIKELY(m_end == m_capacity_end))
         {
-            grow(1);
+            make_room(1);
         }
         *m_end++ = c;
     }
 
-    void append(const char* first, const char* last)
+    /// drop the last character
+    void pop_back() noexcept
+    {
+        JSON_ASSERT(!empty());
+        --m_end;
+    }
+
+    void append(const CharType* first, const CharType* last)
     {
         const auto n = static_cast<std::size_t>(last - first);
         const auto room = static_cast<std::size_t>(m_capacity_end - m_end);
-        if (n > room)
+        if (JSON_HEDLEY_UNLIKELY(n > room))
         {
-            grow(n);
+            append_long(first, n);
+            return;
         }
         if (n != 0)
         {
-            std::memcpy(m_end, first, n);
+            std::memcpy(m_end, first, n * sizeof(CharType));
         }
         m_end += n;
     }
 
     /// replace the contents of @a out
-    void copy_to(std::vector<char>& out) const
+    void copy_to(std::vector<CharType>& out) const
     {
         out.assign(m_data.get(), m_end);
     }
 
   private:
-    /// make room for @a extra more characters
+    /// append @a n characters that do not fit, in pieces of at most tail()
-    JSON_HEDLEY_NEVER_INLINE void grow(std::size_t extra)
+    /// characters, so that dropping the middle keeps the last tail() of them
+    JSON_HEDLEY_NEVER_INLINE void append_long(const CharType* first, std::size_t n)
     {
-        const auto size = static_cast<std::size_t>(m_end - m_data.get());
+        while (n != 0)
-        const auto capacity = static_cast<std::size_t>(m_capacity_end - m_data.get());
+        {
-        const std::size_t new_capacity = (std::max)(capacity < 32 ? static_cast<std::size_t>(32) : 2 * capacity, size + extra);
+            const std::size_t piece = n < tail() ? n : tail();
-        std::unique_ptr<char[]> data(new char[new_capacity]); // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+            make_room(piece);
+            std::memcpy(m_end, first, piece * sizeof(CharType));
+            m_end += piece;
+            first += piece;
+            n -= piece;
+        }
+    }
+
+    /// make room for @a extra (at most tail()) more characters: drop the
+    /// middle of a long token instead of growing beyond limit()
+    JSON_HEDLEY_NEVER_INLINE void make_room(std::size_t extra)
+    {
+        // nothing is allocated before the first character
+        std::size_t size = m_data ? static_cast<std::size_t>(m_end - m_data.get()) : 0;
+        if (size + extra > limit() && size > head() + tail())
+        {
+            std::memmove(m_data.get() + head(), m_end - tail(), tail() * sizeof(CharType));
+            m_end = m_data.get() + head() + tail();
+            size = head() + tail();
+        }
+        const std::size_t capacity = m_data ? static_cast<std::size_t>(m_capacity_end - m_data.get()) : 0;
+        if (size + extra > capacity)
+        {
+            // never more than limit(): from there on, the middle is dropped
+            const std::size_t doubled = capacity < 32 ? static_cast<std::size_t>(32) : 2 * capacity;
+            const std::size_t new_capacity = (std::max)(doubled < limit() ? doubled : limit(), size + extra);
+            std::unique_ptr<CharType[]> data(new CharType[new_capacity]); // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
             if (size != 0)
             {
-            std::memcpy(data.get(), m_data.get(), size);
+                std::memcpy(data.get(), m_data.get(), size * sizeof(CharType));
             }
             m_data = std::move(data);
             m_end = m_data.get() + size;
             m_capacity_end = m_data.get() + new_capacity;
         }
+    }
 
-    std::unique_ptr<char[]> m_data{}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays,readability-redundant-member-init)
+    std::unique_ptr<CharType[]> m_data{}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays,readability-redundant-member-init)
-    char* m_end = nullptr;
+    CharType* m_end = nullptr;
-    char* m_capacity_end = nullptr;
+    CharType* m_capacity_end = nullptr;
 };
+
+/// the part of a token that an input adapter copied out of its window
+using token_carry_buffer = token_text_buffer<char>;
+
+////////////////////
+// input adapters //
+////////////////////
+
+#ifndef JSON_NO_IO
 /*!
 Input adapter for stdio file access. The file is locked for the lifetime of the
 adapter and read with the unlocked stdio functions (see

include/nlohmann/detail/input/lexer.hpp

@@ -494,7 +494,10 @@ class lexer : public lexer_base<BasicJsonType>
     /// streaming adapter: capture the characters
     void bulk_track_token(const char* data, std::size_t n, std::false_type /*lazy*/)
     {
-        token_string.insert(token_string.end(), data, data + n);
+        for (std::size_t i = 0; i < n; ++i)
+        {
+            token_string.push_back(static_cast<char_type>(data[i]));
+        }
     }
 
     /// streaming input: no bulk fast path
@@ -2236,16 +2239,29 @@ scan_number_done:
         const bool pending_real_unget = next_unget && current != char_traits<char_type>::eof();
         const std::size_t stop = ia.get_consumed_count() - (pending_real_unget ? 1u : 0u);
         if (JSON_HEDLEY_LIKELY(stop >= token_string_start))
+        {
+            // only the beginning and the end of a long token are shown (see
+            // get_token_string), so copy only these
+            constexpr std::size_t head = token_text_buffer<char_type>::head();
+            constexpr std::size_t tail = token_text_buffer<char_type>::tail();
+            if (stop - token_string_start > head + tail)
+            {
+                ia.copy_consumed_range(token_string_start, token_string_start + head, out);
+                ia.copy_consumed_range(stop - tail, stop, out);
+            }
+            else
             {
                 ia.copy_consumed_range(token_string_start, stop, out);
             }
+        }
         return out;
     }
 
     /// streaming adapter: the token was copied eagerly while scanning
-    const std::vector<char_type>& collect_token_chars(std::vector<char_type>& /*out*/, std::false_type /*lazy*/) const
+    const std::vector<char_type>& collect_token_chars(std::vector<char_type>& out, std::false_type /*lazy*/) const
     {
-        return token_string;
+        token_string.copy_to(out);
+        return out;
     }
 
     /// return the last read token (for errors only).  Will never contain EOF
@@ -2256,10 +2272,32 @@ scan_number_done:
         std::vector<char_type> reconstructed;
         const std::vector<char_type>& chars = collect_token_chars(reconstructed, std::integral_constant<bool, lazy_token_string> {});
 
-        // escape control characters
+        // a long token is shown by its first and its last head() characters;
+        // the buffers that keep it while it is read keep no more than these
+        // and a few more (see token_text_buffer)
+        constexpr std::size_t shown = token_text_buffer<char_type>::head();
         std::string result;
-        for (const auto c : chars)
+        if (chars.size() > 2 * shown)
         {
+            append_escaped(result, chars.begin(), chars.begin() + static_cast<std::ptrdiff_t>(shown));
+            result += "...";
+            append_escaped(result, chars.end() - static_cast<std::ptrdiff_t>(shown), chars.end());
+        }
+        else
+        {
+            append_escaped(result, chars.begin(), chars.end());
+        }
+        return result;
+    }
+
+  private:
+    /// append the characters [first, last) to @a result, escaping control characters
+    template<typename Iterator>
+    static void append_escaped(std::string& result, Iterator first, Iterator last)
+    {
+        for (; first != last; ++first)
+        {
+            const auto c = *first;
             if (static_cast<unsigned char>(c) <= '\x1F')
             {
                 // escape control characters
@@ -2273,10 +2311,10 @@ scan_number_done:
                 result.push_back(static_cast<std::string::value_type>(c));
             }
         }
-
-        return result;
     }
 
+  public:
+
     /// return syntax error message
     JSON_HEDLEY_RETURNS_NON_NULL
     constexpr const char* get_error_message() const noexcept
@@ -2525,7 +2563,7 @@ scan_number_done:
 
     /// raw input token string for error messages; only populated for streaming
     /// adapters (seekable adapters reconstruct it lazily via token_string_start)
-    std::vector<char_type> token_string {};
+    token_text_buffer<char_type> token_string {};
 
     /// start offset of the current token within the input, used to reconstruct
     /// the last read token on error for seekable adapters (see collect_token_chars)

include/nlohmann/detail/macro_scope.hpp

@@ -230,16 +230,6 @@
 // detail/recursion_depth_limit.hpp. Like JSON_NO_THREAD_LOCAL it is neither part
 // of the ABI tag nor undefined at the end of json.hpp.
 
-// forced inlining of the serializer's helpers only in optimized builds: at -O0,
-// GCC and Clang give every inlined copy its own stack slots, which made the
-// recursive frame of serializer::dump_container() about 2.3 KB (MSVC does not
-// expand __forceinline at /Ob0, so it keeps it)
-#if defined(__OPTIMIZE__) || defined(_MSC_VER)
-    #define JSON_INTERNAL_SERIALIZER_INLINE JSON_HEDLEY_ALWAYS_INLINE
-#else
-    #define JSON_INTERNAL_SERIALIZER_INLINE inline
-#endif
-
 // allow disabling exceptions
 #if (defined(__cpp_exceptions) || defined(__EXCEPTIONS) || defined(_CPPUNWIND)) && !defined(JSON_NOEXCEPTION)
     #define JSON_THROW(exception) throw exception

include/nlohmann/detail/macro_unscope.hpp

@@ -11,7 +11,6 @@
 // clean up
 #undef JSON_ASSERT
 #undef JSON_INTERNAL_CATCH
-#undef JSON_INTERNAL_SERIALIZER_INLINE
 #undef JSON_THROW
 #undef JSON_PRIVATE_UNLESS_TESTED
 #undef NLOHMANN_BASIC_JSON_TPL_DECLARATION

include/nlohmann/detail/output/serializer.hpp

@@ -265,7 +265,7 @@ class serializer
     for a container; writing them from here spares each the entry and exit of
     that function.
     */
-    JSON_INTERNAL_SERIALIZER_INLINE void dump_element(const BasicJsonType& val,
+    JSON_HEDLEY_ALWAYS_INLINE void dump_element(const BasicJsonType& val,
             const std::size_t current_indent,
             const std::size_t depth)
     {
@@ -971,7 +971,7 @@ class serializer
     }
 
     /// write out the bytes pending in @ref string_buffer and reset @a bytes
-    JSON_INTERNAL_SERIALIZER_INLINE void emit_pending(std::size_t& bytes)
+    JSON_HEDLEY_ALWAYS_INLINE void emit_pending(std::size_t& bytes)
     {
         if (bytes != 0)
         {
@@ -983,7 +983,7 @@ class serializer
     /// write buffer and reset index if fewer than 13 bytes are left, as this
     /// is the maximal number of bytes to be written ("\uxxxx\uxxxx\0") for one
     /// code point
-    JSON_INTERNAL_SERIALIZER_INLINE void reserve_string_buffer(std::size_t& bytes)
+    JSON_HEDLEY_ALWAYS_INLINE void reserve_string_buffer(std::size_t& bytes)
     {
         if (string_buffer.size() - bytes < 13)
         {
@@ -1344,7 +1344,7 @@ class serializer
           inlining both into the pretty-printing loops once @ref dump_escaped
           has grown, which made indented output up to 13% slower
     */
-    JSON_INTERNAL_SERIALIZER_INLINE void put_indent(std::size_t indent)
+    JSON_HEDLEY_ALWAYS_INLINE void put_indent(std::size_t indent)
     {
         // closing braces at the outermost level ask for no indentation at all
         if (indent == 0)
@@ -1398,7 +1398,7 @@ class serializer
     its room and four stores of the indentation, see @ref put_short_literal_indent.
     */
     template<std::size_t N>
-    JSON_INTERNAL_SERIALIZER_INLINE void put_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+    JSON_HEDLEY_ALWAYS_INLINE void put_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
     {
         if (JSON_HEDLEY_LIKELY(put_short_literal_indent(s, indent)))
         {
@@ -1418,7 +1418,7 @@ class serializer
     @a indent is wider or the write buffer lacks the room.
     */
     template<std::size_t N>
-    JSON_INTERNAL_SERIALIZER_INLINE bool put_short_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+    JSON_HEDLEY_ALWAYS_INLINE bool put_short_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
     {
         constexpr std::size_t length = N - 1;
         if (indent > 32 || write_buffer_pos + length + 32 > write_buffer.size())
@@ -1445,7 +1445,7 @@ class serializer
     write-through path for oversized runs.
     */
     template<std::size_t N>
-    JSON_INTERNAL_SERIALIZER_INLINE void put_literal(const char (&s)[N]) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+    JSON_HEDLEY_ALWAYS_INLINE void put_literal(const char (&s)[N]) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
     {
         static_assert(N >= 2, "put_literal expects a non-empty string literal");
         // the array bound counts the terminating NUL, which is not written

single_include/nlohmann/json.hpp

@@ -2603,16 +2603,6 @@ JSON_HEDLEY_DIAGNOSTIC_POP
 // detail/recursion_depth_limit.hpp. Like JSON_NO_THREAD_LOCAL it is neither part
 // of the ABI tag nor undefined at the end of json.hpp.
 
-// forced inlining of the serializer's helpers only in optimized builds: at -O0,
-// GCC and Clang give every inlined copy its own stack slots, which made the
-// recursive frame of serializer::dump_container() about 2.3 KB (MSVC does not
-// expand __forceinline at /Ob0, so it keeps it)
-#if defined(__OPTIMIZE__) || defined(_MSC_VER)
-    #define JSON_INTERNAL_SERIALIZER_INLINE JSON_HEDLEY_ALWAYS_INLINE
-#else
-    #define JSON_INTERNAL_SERIALIZER_INLINE inline
-#endif
-
 // allow disabling exceptions
 #if (defined(__cpp_exceptions) || defined(__EXCEPTIONS) || defined(_CPPUNWIND)) && !defined(JSON_NOEXCEPTION)
     #define JSON_THROW(exception) throw exception
@@ -9333,87 +9323,164 @@ inline const char* empty_input_message() noexcept
     return "attempting to parse an empty input; check that your input string or stream contains the expected JSON";
 }
 
-////////////////////
-// input adapters //
-////////////////////
-
-#ifndef JSON_NO_IO
 /*!
+@brief the characters of the last read token, kept for error messages
+
 A growable array of characters for the part of a token that an input adapter
-copied out of its window: unlike std::vector<char>::push_back(), appending a
+copied out of its window, or that the lexer copies while it scans input it
-character takes no reference to a local, and is a few instructions when there
+cannot read again: unlike std::vector::push_back(), appending a character
-is room.
+takes no reference to a local, and is a few instructions when there is room.
+
+The token is only needed for error messages, which show its first head()
+characters and its last head() characters (see lexer::get_token_string). The
+buffer therefore keeps the first head() and at least the last tail()
+characters: when it is full and holds more than limit() characters, it drops
+the characters in between instead of growing. That happens where it would
+otherwise reallocate, so appending stays as cheap as before, and a long string
+or a long run of whitespace read from a stream is not kept a second time.
 */
-class token_carry_buffer
+template<typename CharType>
+class token_text_buffer
 {
   public:
-    token_carry_buffer() = default;
+    static constexpr std::size_t head() noexcept
-    token_carry_buffer(const token_carry_buffer&) = delete;
+    {
-    token_carry_buffer& operator=(const token_carry_buffer&) = delete;
+        return 4096;
-    token_carry_buffer(token_carry_buffer&& rhs) noexcept
+    }
+    /// a little more than head(), as the last character may be handed back
+    static constexpr std::size_t tail() noexcept
+    {
+        return head() + 16;
+    }
+    static constexpr std::size_t limit() noexcept
+    {
+        return head() + (2 * tail());
+    }
+
+    token_text_buffer() = default;
+    token_text_buffer(const token_text_buffer&) = delete;
+    token_text_buffer& operator=(const token_text_buffer&) = delete;
+    token_text_buffer(token_text_buffer&& rhs) noexcept
         : m_data(std::move(rhs.m_data)), m_end(rhs.m_end), m_capacity_end(rhs.m_capacity_end)
     {
         rhs.m_end = rhs.m_capacity_end = nullptr;
     }
-    token_carry_buffer& operator=(token_carry_buffer&&) = delete;
+    token_text_buffer& operator=(token_text_buffer&& rhs) noexcept
-    ~token_carry_buffer() = default;
+    {
+        m_data = std::move(rhs.m_data);
+        m_end = rhs.m_end;
+        m_capacity_end = rhs.m_capacity_end;
+        rhs.m_end = rhs.m_capacity_end = nullptr;
+        return *this;
+    }
+    ~token_text_buffer() = default;
 
     void clear() noexcept
     {
         m_end = m_data.get();
     }
 
-    void push_back(char c)
+    bool empty() const noexcept
+    {
+        return m_end == m_data.get();
+    }
+
+    void push_back(CharType c)
     {
         if (JSON_HEDLEY_UNLIKELY(m_end == m_capacity_end))
         {
-            grow(1);
+            make_room(1);
         }
         *m_end++ = c;
     }
 
-    void append(const char* first, const char* last)
+    /// drop the last character
+    void pop_back() noexcept
+    {
+        JSON_ASSERT(!empty());
+        --m_end;
+    }
+
+    void append(const CharType* first, const CharType* last)
     {
         const auto n = static_cast<std::size_t>(last - first);
         const auto room = static_cast<std::size_t>(m_capacity_end - m_end);
-        if (n > room)
+        if (JSON_HEDLEY_UNLIKELY(n > room))
         {
-            grow(n);
+            append_long(first, n);
+            return;
         }
         if (n != 0)
         {
-            std::memcpy(m_end, first, n);
+            std::memcpy(m_end, first, n * sizeof(CharType));
         }
         m_end += n;
     }
 
     /// replace the contents of @a out
-    void copy_to(std::vector<char>& out) const
+    void copy_to(std::vector<CharType>& out) const
     {
         out.assign(m_data.get(), m_end);
     }
 
   private:
-    /// make room for @a extra more characters
+    /// append @a n characters that do not fit, in pieces of at most tail()
-    JSON_HEDLEY_NEVER_INLINE void grow(std::size_t extra)
+    /// characters, so that dropping the middle keeps the last tail() of them
+    JSON_HEDLEY_NEVER_INLINE void append_long(const CharType* first, std::size_t n)
     {
-        const auto size = static_cast<std::size_t>(m_end - m_data.get());
+        while (n != 0)
-        const auto capacity = static_cast<std::size_t>(m_capacity_end - m_data.get());
+        {
-        const std::size_t new_capacity = (std::max)(capacity < 32 ? static_cast<std::size_t>(32) : 2 * capacity, size + extra);
+            const std::size_t piece = n < tail() ? n : tail();
-        std::unique_ptr<char[]> data(new char[new_capacity]); // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+            make_room(piece);
+            std::memcpy(m_end, first, piece * sizeof(CharType));
+            m_end += piece;
+            first += piece;
+            n -= piece;
+        }
+    }
+
+    /// make room for @a extra (at most tail()) more characters: drop the
+    /// middle of a long token instead of growing beyond limit()
+    JSON_HEDLEY_NEVER_INLINE void make_room(std::size_t extra)
+    {
+        // nothing is allocated before the first character
+        std::size_t size = m_data ? static_cast<std::size_t>(m_end - m_data.get()) : 0;
+        if (size + extra > limit() && size > head() + tail())
+        {
+            std::memmove(m_data.get() + head(), m_end - tail(), tail() * sizeof(CharType));
+            m_end = m_data.get() + head() + tail();
+            size = head() + tail();
+        }
+        const std::size_t capacity = m_data ? static_cast<std::size_t>(m_capacity_end - m_data.get()) : 0;
+        if (size + extra > capacity)
+        {
+            // never more than limit(): from there on, the middle is dropped
+            const std::size_t doubled = capacity < 32 ? static_cast<std::size_t>(32) : 2 * capacity;
+            const std::size_t new_capacity = (std::max)(doubled < limit() ? doubled : limit(), size + extra);
+            std::unique_ptr<CharType[]> data(new CharType[new_capacity]); // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
             if (size != 0)
             {
-            std::memcpy(data.get(), m_data.get(), size);
+                std::memcpy(data.get(), m_data.get(), size * sizeof(CharType));
             }
             m_data = std::move(data);
             m_end = m_data.get() + size;
             m_capacity_end = m_data.get() + new_capacity;
         }
+    }
 
-    std::unique_ptr<char[]> m_data{}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays,readability-redundant-member-init)
+    std::unique_ptr<CharType[]> m_data{}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays,readability-redundant-member-init)
-    char* m_end = nullptr;
+    CharType* m_end = nullptr;
-    char* m_capacity_end = nullptr;
+    CharType* m_capacity_end = nullptr;
 };
+
+/// the part of a token that an input adapter copied out of its window
+using token_carry_buffer = token_text_buffer<char>;
+
+////////////////////
+// input adapters //
+////////////////////
+
+#ifndef JSON_NO_IO
 /*!
 Input adapter for stdio file access. The file is locked for the lifetime of the
 adapter and read with the unlocked stdio functions (see
@@ -13908,7 +13975,10 @@ class lexer : public lexer_base<BasicJsonType>
     /// streaming adapter: capture the characters
     void bulk_track_token(const char* data, std::size_t n, std::false_type /*lazy*/)
     {
-        token_string.insert(token_string.end(), data, data + n);
+        for (std::size_t i = 0; i < n; ++i)
+        {
+            token_string.push_back(static_cast<char_type>(data[i]));
+        }
     }
 
     /// streaming input: no bulk fast path
@@ -15650,16 +15720,29 @@ scan_number_done:
         const bool pending_real_unget = next_unget && current != char_traits<char_type>::eof();
         const std::size_t stop = ia.get_consumed_count() - (pending_real_unget ? 1u : 0u);
         if (JSON_HEDLEY_LIKELY(stop >= token_string_start))
+        {
+            // only the beginning and the end of a long token are shown (see
+            // get_token_string), so copy only these
+            constexpr std::size_t head = token_text_buffer<char_type>::head();
+            constexpr std::size_t tail = token_text_buffer<char_type>::tail();
+            if (stop - token_string_start > head + tail)
+            {
+                ia.copy_consumed_range(token_string_start, token_string_start + head, out);
+                ia.copy_consumed_range(stop - tail, stop, out);
+            }
+            else
             {
                 ia.copy_consumed_range(token_string_start, stop, out);
             }
+        }
         return out;
     }
 
     /// streaming adapter: the token was copied eagerly while scanning
-    const std::vector<char_type>& collect_token_chars(std::vector<char_type>& /*out*/, std::false_type /*lazy*/) const
+    const std::vector<char_type>& collect_token_chars(std::vector<char_type>& out, std::false_type /*lazy*/) const
     {
-        return token_string;
+        token_string.copy_to(out);
+        return out;
     }
 
     /// return the last read token (for errors only).  Will never contain EOF
@@ -15670,10 +15753,32 @@ scan_number_done:
         std::vector<char_type> reconstructed;
         const std::vector<char_type>& chars = collect_token_chars(reconstructed, std::integral_constant<bool, lazy_token_string> {});
 
-        // escape control characters
+        // a long token is shown by its first and its last head() characters;
+        // the buffers that keep it while it is read keep no more than these
+        // and a few more (see token_text_buffer)
+        constexpr std::size_t shown = token_text_buffer<char_type>::head();
         std::string result;
-        for (const auto c : chars)
+        if (chars.size() > 2 * shown)
         {
+            append_escaped(result, chars.begin(), chars.begin() + static_cast<std::ptrdiff_t>(shown));
+            result += "...";
+            append_escaped(result, chars.end() - static_cast<std::ptrdiff_t>(shown), chars.end());
+        }
+        else
+        {
+            append_escaped(result, chars.begin(), chars.end());
+        }
+        return result;
+    }
+
+  private:
+    /// append the characters [first, last) to @a result, escaping control characters
+    template<typename Iterator>
+    static void append_escaped(std::string& result, Iterator first, Iterator last)
+    {
+        for (; first != last; ++first)
+        {
+            const auto c = *first;
             if (static_cast<unsigned char>(c) <= '\x1F')
             {
                 // escape control characters
@@ -15687,10 +15792,10 @@ scan_number_done:
                 result.push_back(static_cast<std::string::value_type>(c));
             }
         }
-
-        return result;
     }
 
+  public:
+
     /// return syntax error message
     JSON_HEDLEY_RETURNS_NON_NULL
     constexpr const char* get_error_message() const noexcept
@@ -15939,7 +16044,7 @@ scan_number_done:
 
     /// raw input token string for error messages; only populated for streaming
     /// adapters (seekable adapters reconstruct it lazily via token_string_start)
-    std::vector<char_type> token_string {};
+    token_text_buffer<char_type> token_string {};
 
     /// start offset of the current token within the input, used to reconstruct
     /// the last read token on error for seekable adapters (see collect_token_chars)
@@ -18578,17 +18683,6 @@ class binary_reader
                 JSON_ASSERT(false); // NOLINT(cert-dcl03-c,hicpp-static-assert,misc-static-assert) LCOV_EXCL_LINE
         }
 
-        // BON8: a string ends at the first byte that cannot continue it. If
-        // that byte follows the value, the reader has already taken it from
-        // the input and holds it in bon8_pushback. Without the end-of-input
-        // check below it would be dropped, and the next read would start
-        // behind it, so report the missing 0xFF instead.
-        if (result && !strict && input_format == input_format_t::bon8 && bon8_pushback_size != 0)
-        {
-            get_bon8();
-            return last_byte_error(exception_id::unexpected_end_of_input, "string at the end of the value is not terminated by 0xFF", "value", "");
-        }
-
         // strict mode: next byte must be EOF
         if (result && strict)
         {
@@ -37006,7 +37100,7 @@ class serializer
     for a container; writing them from here spares each the entry and exit of
     that function.
     */
-    JSON_INTERNAL_SERIALIZER_INLINE void dump_element(const BasicJsonType& val,
+    JSON_HEDLEY_ALWAYS_INLINE void dump_element(const BasicJsonType& val,
             const std::size_t current_indent,
             const std::size_t depth)
     {
@@ -37712,7 +37806,7 @@ class serializer
     }
 
     /// write out the bytes pending in @ref string_buffer and reset @a bytes
-    JSON_INTERNAL_SERIALIZER_INLINE void emit_pending(std::size_t& bytes)
+    JSON_HEDLEY_ALWAYS_INLINE void emit_pending(std::size_t& bytes)
     {
         if (bytes != 0)
         {
@@ -37724,7 +37818,7 @@ class serializer
     /// write buffer and reset index if fewer than 13 bytes are left, as this
     /// is the maximal number of bytes to be written ("\uxxxx\uxxxx\0") for one
     /// code point
-    JSON_INTERNAL_SERIALIZER_INLINE void reserve_string_buffer(std::size_t& bytes)
+    JSON_HEDLEY_ALWAYS_INLINE void reserve_string_buffer(std::size_t& bytes)
     {
         if (string_buffer.size() - bytes < 13)
         {
@@ -38085,7 +38179,7 @@ class serializer
           inlining both into the pretty-printing loops once @ref dump_escaped
           has grown, which made indented output up to 13% slower
     */
-    JSON_INTERNAL_SERIALIZER_INLINE void put_indent(std::size_t indent)
+    JSON_HEDLEY_ALWAYS_INLINE void put_indent(std::size_t indent)
     {
         // closing braces at the outermost level ask for no indentation at all
         if (indent == 0)
@@ -38139,7 +38233,7 @@ class serializer
     its room and four stores of the indentation, see @ref put_short_literal_indent.
     */
     template<std::size_t N>
-    JSON_INTERNAL_SERIALIZER_INLINE void put_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+    JSON_HEDLEY_ALWAYS_INLINE void put_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
     {
         if (JSON_HEDLEY_LIKELY(put_short_literal_indent(s, indent)))
         {
@@ -38159,7 +38253,7 @@ class serializer
     @a indent is wider or the write buffer lacks the room.
     */
     template<std::size_t N>
-    JSON_INTERNAL_SERIALIZER_INLINE bool put_short_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+    JSON_HEDLEY_ALWAYS_INLINE bool put_short_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
     {
         constexpr std::size_t length = N - 1;
         if (indent > 32 || write_buffer_pos + length + 32 > write_buffer.size())
@@ -38186,7 +38280,7 @@ class serializer
     write-through path for oversized runs.
     */
     template<std::size_t N>
-    JSON_INTERNAL_SERIALIZER_INLINE void put_literal(const char (&s)[N]) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+    JSON_HEDLEY_ALWAYS_INLINE void put_literal(const char (&s)[N]) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
     {
         static_assert(N >= 2, "put_literal expects a non-empty string literal");
         // the array bound counts the terminating NUL, which is not written
@@ -47906,7 +48000,6 @@ struct formatter<nlohmann::NLOHMANN_BASIC_JSON_TPL, char> // NOLINT(cert-dcl58-c
 // clean up
 #undef JSON_ASSERT
 #undef JSON_INTERNAL_CATCH
-#undef JSON_INTERNAL_SERIALIZER_INLINE
 #undef JSON_THROW
 #undef JSON_PRIVATE_UNLESS_TESTED
 #undef NLOHMANN_BASIC_JSON_TPL_DECLARATION

tests/src/unit-bon8.cpp

@@ -608,26 +608,14 @@ TEST_CASE("BON8")
             CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing BON8 value: expected end of input; last byte: 0x90", json::parse_error&);
             CHECK(json::from_bon8(vec, true, false).is_discarded());
 
-            // the byte after a string that ends a container: without strict, the
+            // the byte after a string that ends a container
-            // missing 0xFF is reported, as the byte has been taken from the input
             const bytes vec2 = {0x81, 'a', 0x90};
-            CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec2, false), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing BON8 value: string at the end of the value is not terminated by 0xFF; last byte: 0x90", json::parse_error&);
+            CHECK(json::from_bon8(vec2, false) == json({"a"}));
-            CHECK(json::from_bon8(vec2, false, false).is_discarded());
             CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec2), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing BON8 value: expected end of input; last byte: 0x90", json::parse_error&);
 
             const bytes vec3 = {0x81, 'a', 0xC2, 0x00};
-            CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec3, false), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing BON8 value: string at the end of the value is not terminated by 0xFF; last byte: 0xC2", json::parse_error&);
+            CHECK(json::from_bon8(vec3, false) == json({"a"}));
             CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec3), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing BON8 value: expected end of input; last byte: 0xC2", json::parse_error&);
-
-            // with a terminated string, strict = false leaves the next value in the stream
-            std::istringstream terminated(std::string("\x61\xFF\x90", 3));
-            CHECK(json::from_bon8(terminated, false) == "a");
-            CHECK(terminated.tellg() == 2);
-            CHECK(json::from_bon8(terminated, false) == 0);
-
-            // without the terminator, the next value would be lost
-            std::istringstream unterminated(std::string("\x61\x90", 2));
-            CHECK_THROWS_WITH_AS(_ = json::from_bon8(unterminated, false), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing BON8 value: string at the end of the value is not terminated by 0xFF; last byte: 0x90", json::parse_error&);
         }
     }
 

tests/src/unit-class_lexer.cpp

@@ -12,6 +12,7 @@
 #include <nlohmann/json.hpp>
 using nlohmann::json;
 
+#include <algorithm> // min
 #include <array> // array
 #include <cctype> // toupper
 #include <cstdint> // uint32_t, uint64_t
@@ -2233,3 +2234,123 @@ TEST_CASE("serialization of short strings")
     CHECK(json("caf\xC3\xA9").dump() == "\"caf\xC3\xA9\"");
     CHECK(json("caf\xC3\xA9").dump(-1, ' ', true) == "\"caf\\u00e9\"");
 }
+
+TEST_CASE("token_text_buffer keeps the beginning and the end of a long token")
+{
+    using buffer_t = nlohmann::detail::token_text_buffer<char>;
+    constexpr std::size_t head = buffer_t::head();
+    constexpr std::size_t tail = buffer_t::tail();
+
+    // the characters 0, 1, ..., 250, 0, 1, ...
+    const auto at = [](std::size_t i)
+    {
+        return static_cast<char>(i % 251);
+    };
+    // whether out holds the first head and the last tail characters of a
+    // token of n characters (and maybe more characters of its middle)
+    const auto keeps_ends = [&](const std::vector<char>& out, std::size_t n)
+    {
+        if (out.size() < head + tail || out.size() > buffer_t::limit())
+        {
+            return false;
+        }
+        for (std::size_t i = 0; i < head; ++i)
+        {
+            if (out[i] != at(i))
+            {
+                return false;
+            }
+        }
+        for (std::size_t i = 0; i < tail; ++i)
+        {
+            if (out[out.size() - tail + i] != at(n - tail + i))
+            {
+                return false;
+            }
+        }
+        return true;
+    };
+
+    SECTION("a short token is kept as it is")
+    {
+        buffer_t buffer;
+        CHECK(buffer.empty());
+        for (std::size_t i = 0; i < head + tail; ++i)
+        {
+            buffer.push_back(at(i));
+        }
+        std::vector<char> out;
+        buffer.copy_to(out);
+        REQUIRE(out.size() == head + tail);
+        bool same = true;
+        for (std::size_t i = 0; i < out.size(); ++i)
+        {
+            same = same && out[i] == at(i);
+        }
+        CHECK(same);
+    }
+
+    SECTION("characters added one by one")
+    {
+        for (const std::size_t n :
+                {
+                    head + tail + 1, buffer_t::limit(), buffer_t::limit() + 1, static_cast<std::size_t>(100000)
+                })
+        {
+            CAPTURE(n)
+            buffer_t buffer;
+            for (std::size_t i = 0; i < n; ++i)
+            {
+                buffer.push_back(at(i));
+            }
+            std::vector<char> out;
+            buffer.copy_to(out);
+            CHECK(keeps_ends(out, n));
+
+            // the last character can be handed back
+            buffer.pop_back();
+            buffer.copy_to(out);
+            CHECK(keeps_ends(out, n - 1));
+        }
+    }
+
+    SECTION("characters added in pieces of different sizes")
+    {
+        std::vector<char> input(200000);
+        for (std::size_t i = 0; i < input.size(); ++i)
+        {
+            input[i] = at(i);
+        }
+        for (const std::size_t piece :
+                {
+                    static_cast<std::size_t>(1), static_cast<std::size_t>(100), tail - 1, tail, tail + 1, static_cast<std::size_t>(65536), input.size()
+                })
+        {
+            CAPTURE(piece)
+            buffer_t buffer;
+            for (std::size_t pos = 0; pos < input.size(); pos += piece)
+            {
+                const std::size_t n = (std::min)(piece, input.size() - pos);
+                buffer.append(input.data() + pos, input.data() + pos + n);
+            }
+            std::vector<char> out;
+            buffer.copy_to(out);
+            CHECK(keeps_ends(out, input.size()));
+        }
+    }
+
+    SECTION("clear starts a new token")
+    {
+        buffer_t buffer;
+        for (std::size_t i = 0; i < 100000; ++i)
+        {
+            buffer.push_back('x');
+        }
+        buffer.clear();
+        CHECK(buffer.empty());
+        buffer.push_back('a');
+        std::vector<char> out;
+        buffer.copy_to(out);
+        CHECK(out == std::vector<char> {'a'});
+    }
+}

tests/src/unit-class_parser.cpp

@@ -3269,3 +3269,116 @@ TEST_CASE("a plain function is accepted as parser callback for containers")
     CHECK((nlohmann::detail::can_compare_ne<std::string::const_iterator, std::string::const_iterator>::value));
     CHECK((nlohmann::detail::can_compare_ne<const char*, const char*>::value));
 }
+
+#if !defined(JSON_NOEXCEPTION)
+TEST_CASE("parse error message of a long token")
+{
+    // the message of a parse error shows the last read token; a long token is
+    // shown by its first and its last 4096 characters, and the message is the
+    // same for every kind of input
+    const auto message = [](const std::string & text, int kind) -> std::string
+    {
+        json _;
+        try
+        {
+            if (kind == 0)
+            {
+                _ = json::parse(text);
+            }
+            else if (kind == 1)
+            {
+                std::istringstream stream(text);
+                _ = json::parse(stream);
+            }
+            else if (kind == 2)
+            {
+                FILE* const file = std::tmpfile(); // NOLINT(cppcoreguidelines-owning-memory)
+                REQUIRE(file != nullptr);
+                CHECK(std::fwrite(text.data(), 1, text.size(), file) == text.size());
+                CHECK(std::fseek(file, 0, SEEK_SET) == 0);
+                try
+                {
+                    _ = json::parse(file);
+                }
+                catch (...)
+                {
+                    std::fclose(file); // NOLINT(cert-err33-c,cppcoreguidelines-owning-memory)
+                    throw;
+                }
+                std::fclose(file); // NOLINT(cert-err33-c,cppcoreguidelines-owning-memory)
+            }
+            else
+            {
+                const std::list<char> chars(text.begin(), text.end());
+                _ = json::parse(chars.begin(), chars.end());
+            }
+        }
+        catch (const json::parse_error& e)
+        {
+            return e.what();
+        }
+        return "no error";
+    };
+
+    SECTION("a short token is shown completely")
+    {
+        const std::string text = "[\"" + std::string(8000, 'a') + "\x01\"]";
+        const std::string expected = "last read: '\"" + std::string(8000, 'a') + "<U+0001>'";
+        for (int kind = 0; kind < 4; ++kind)
+        {
+            CAPTURE(kind)
+            const std::string m = message(text, kind);
+            CHECK(m.size() > expected.size());
+            CHECK(m.compare(m.size() - expected.size(), expected.size(), expected) == 0);
+        }
+    }
+
+    SECTION("a long string")
+    {
+        const std::string text = "[\"" + std::string(100000, 'a') + "b\x01\"]";
+        // the token is '"', 100,000 times 'a', 'b', and the control character
+        const std::string expected = "last read: '\"" + std::string(4095, 'a') + "..." + std::string(4094, 'a') + "b<U+0001>'";
+        for (int kind = 0; kind < 4; ++kind)
+        {
+            CAPTURE(kind)
+            const std::string m = message(text, kind);
+            CHECK(m.size() > expected.size());
+            CHECK(m.compare(m.size() - expected.size(), expected.size(), expected) == 0);
+        }
+    }
+
+    SECTION("a long run of whitespace after a number")
+    {
+        // the token begins with the number and ends with the invalid character
+        const std::string text = "[1," + std::string(100000, '\n') + "?]";
+        std::string expected = "last read: '1,";
+        for (int i = 0; i < 4094; ++i)
+        {
+            expected += "<U+000A>";
+        }
+        expected += "...";
+        for (int i = 0; i < 4095; ++i)
+        {
+            expected += "<U+000A>";
+        }
+        expected += "?'";
+        for (int kind = 0; kind < 4; ++kind)
+        {
+            CAPTURE(kind)
+            const std::string m = message(text, kind);
+            CHECK(m.size() > expected.size());
+            CHECK(m.compare(m.size() - expected.size(), expected.size(), expected) == 0);
+        }
+    }
+
+    SECTION("valid input with a long run of whitespace")
+    {
+        const std::string text = "1" + std::string(100000, ' ');
+        for (int kind = 0; kind < 4; ++kind)
+        {
+            CAPTURE(kind)
+            CHECK(message(text, kind) == "no error");
+        }
+    }
+}
+#endif