Merge branch 'develop' into claude/fix-issue-3989-db7e45

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

.github/CONTRIBUTING.md

@@ -111,6 +111,8 @@ context which existing file needs to be extended, and only very few cases requir
 When fixing a bug, edit `unit-regression3.cpp` and add a section referencing the fixed issue.
 `unit-regression2.cpp` holds the older tests; the two files exist because a single one grew large enough for the
 MinGW linker to fail relocating it, so please keep adding to the smaller file rather than growing the larger one.
+Tests that call `sax_parse` go into `unit-sax_parse.cpp` rather than into a large test file: every call instantiates
+the parser and the binary reader that recover from errors, which grows a test file considerably.
 
 #### Exceptions
 

README.md

@@ -495,7 +495,7 @@ bool key(string_t& val);
 bool parse_error(std::size_t position, const std::string& last_token, const detail::exception& ex);
 ```
 
-The return value of each function determines whether parsing should proceed.
+The return value of each function determines whether parsing should proceed. For `parse_error`, returning `true` [recovers from the error](https://json.nlohmann.me/features/parsing/error_recovery/): the parser repairs the input and continues.
 
 To implement your own SAX handler, proceed as follows:
 
@@ -503,7 +503,7 @@ To implement your own SAX handler, proceed as follows:
 2. Create an object of your SAX interface class, e.g. `my_sax`.
 3. Call `bool json::sax_parse(input, &my_sax)`; where the first parameter can be any input like a string or an input stream and the second parameter is a pointer to your SAX interface.
 
-Note the `sax_parse` function only returns a `bool` indicating the result of the last executed SAX event. It does not return a  `json` value - it is up to you to decide what to do with the SAX events. Furthermore, no exceptions are thrown in case of a parse error -- it is up to you what to do with the exception object passed to your `parse_error` implementation. Internally, the SAX interface is used for the DOM parser (class `json_sax_dom_parser`) as well as the acceptor (`json_sax_acceptor`), see file [`json_sax.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/json_sax.hpp).
+Note the `sax_parse` function only returns a `bool` indicating whether the input was parsed without errors and no SAX event returned `false`. It does not return a  `json` value - it is up to you to decide what to do with the SAX events. Furthermore, no exceptions are thrown in case of a parse error -- it is up to you what to do with the exception object passed to your `parse_error` implementation. Internally, the SAX interface is used for the DOM parser (class `json_sax_dom_parser`) as well as the acceptor (`json_sax_acceptor`), see file [`json_sax.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/json_sax.hpp).
 
 ### STL-like access
 

docs/mkdocs/docs/api/basic_json/sax_parse.md

@@ -96,7 +96,9 @@ The SAX event lister must follow the interface of [`json_sax`](../json_sax/index
 
 ## Return value
 
-return value of the last processed SAX event
+`#!cpp true` if the input was parsed without errors and no SAX event returned `#!cpp false`; `#!cpp false` otherwise.
+In particular, the result is `#!cpp false` for input with errors, even if the SAX parser recovered from all of them
+(see [error recovery](../../features/parsing/error_recovery.md)).
 
 ## Exception safety
 
@@ -153,6 +155,7 @@ use the same file with stdio functions, but it must not wait for another thread
 - Added `ignore_trailing_commas` in version 3.13.0.
 - Added `tag_handler` in version 3.13.0.
 - Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
+- Recovering from parse errors (see [`parse_error`](../json_sax/parse_error.md)) added in version 3.13.0.
 - Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
 - `JSON_PRECISE_STREAM_POSITION` added in version 3.13.0 to optionally leave a `#!cpp std::istream` positioned right
   after the parsed value when `strict` is `#!cpp false`.

docs/mkdocs/docs/api/basic_json_view/at.md

@@ -80,7 +80,8 @@ None of these exceptions carry a [`JSON_DIAGNOSTICS`](../macros/json_diagnostics
    Objects with 128 or more members get a hash index of their keys while parsing, so that a lookup in them takes
    constant time on average.
 2. Linear in `idx`: elements are skipped one at a time from the first one, since they are not a fixed size in the
-   index (unlike `BasicJsonType`'s array, which is random-access).
+   index (unlike `BasicJsonType`'s array, which is random-access). Constant if no element of the array is an array
+   or object, since the elements are then adjacent in the index.
 3. Linear in the number of reference tokens of `ptr` and, for each token, in the number of members of the object at
    that level (as 1.) or the index into the array (as 2.).
 

docs/mkdocs/docs/api/basic_json_view/back.md

@@ -28,7 +28,8 @@ This exception does not carry a [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.
 
 Linear in the size of the array or object: unlike [`front()`](front.md), which only ever looks at the first
 element, `back()` has to walk every element to find where they end, since elements are not a fixed size in the
-index.
+index. Constant if no element of the array is an array or object, since the elements are then adjacent
+in the index.
 
 ## Notes
 

docs/mkdocs/docs/api/basic_json_view/operator[].md

@@ -86,7 +86,8 @@ None of these exceptions carry a [`JSON_DIAGNOSTICS`](../macros/json_diagnostics
    Objects with 128 or more members get a hash index of their keys while parsing, so that a lookup in them takes
    constant time on average.
 2. Linear in `idx`: elements are skipped one at a time from the first one, since they are not a fixed size in the
-   index (unlike `BasicJsonType`'s array, which is random-access).
+   index (unlike `BasicJsonType`'s array, which is random-access). Constant if no element of the array is an array
+   or object, since the elements are then adjacent in the index.
 3. Linear in the number of reference tokens of `ptr` and, for each token, in the number of members of the object at
    that level (as 1.) or the index into the array (as 2.).
 

docs/mkdocs/docs/api/json_sax/index.md

@@ -7,7 +7,8 @@ struct json_sax;
 
 This class describes the SAX interface used by [sax_parse](../basic_json/sax_parse.md). Each function is called in
 different situations while the input is parsed. The boolean return value informs the parser whether to continue
-processing the input.
+processing the input; for [`parse_error`](parse_error.md), it decides whether to
+[recover from the error](../../features/parsing/error_recovery.md).
 
 For instance, parsing the JSON text `{"a": [1, true]}` triggers the following callbacks, in order:
 

docs/mkdocs/docs/api/json_sax/parse_error.md

@@ -21,11 +21,18 @@ A parse error occurred.
 
 ## Return value
 
-Whether parsing should proceed (**must return `#!cpp false`**).
+Whether to recover from the error:
+
+- `#!cpp false` stops parsing.
+- `#!cpp true` recovers from the error: the error is repaired and parsing continues. If that is not possible, which
+  happens in the binary formats when the end of the item with the error is unknown, the value read so far is completed
+  and parsing stops. See [error recovery](../../features/parsing/error_recovery.md) for how errors are repaired.
+
+Either way, [`sax_parse`](../basic_json/sax_parse.md) returns `#!cpp false`.
 
 ## Examples
 
-??? example
+??? example "Example: (1) the SAX interface"
 
     The example below shows how the SAX interface is used.
 
@@ -39,12 +46,29 @@ Whether parsing should proceed (**must return `#!cpp false`**).
     --8<-- "examples/sax_parse.output"
     ```
 
+??? example "Example: (2) recovering from errors"
+
+    The example below shows how a SAX parser recovers from errors.
+
+    ```cpp
+    --8<-- "examples/sax_parse__error_recovery.cpp"
+    ```
+    
+    Output:
+    
+    ```
+    --8<-- "examples/sax_parse__error_recovery.output"
+    ```
+
 ## See also
 
 - [sax_parse](../basic_json/sax_parse.md) - SAX parser
 - [Parsing and Exceptions](../../features/parsing/parse_exceptions.md) - the article on handling parse errors without
   exceptions
+- [Error Recovery](../../features/parsing/error_recovery.md) - the article on recovering from parse errors
 
 ## Version history
 
 - Added in version 3.2.0.
+- Returning `#!cpp true` recovers from the error since version 3.13.0; before, parsing stopped, but the result of
+  [`sax_parse`](../basic_json/sax_parse.md) could be wrong.

docs/mkdocs/docs/examples/sax_parse__error_recovery.cpp

@@ -0,0 +1,43 @@
+#include <iostream>
+#include <iomanip>
+#include <nlohmann/json.hpp>
+
+using json = nlohmann::json;
+
+// a SAX parser that creates a JSON value like json::parse does, but that
+// recovers from parse errors instead of stopping at the first one
+class recovering_parser : public nlohmann::detail::json_sax_dom_parser<json>
+{
+  public:
+    explicit recovering_parser(json& result)
+        : nlohmann::detail::json_sax_dom_parser<json>(result, false)
+    {}
+
+    bool parse_error(std::size_t position,
+                     const std::string& /*last_token*/,
+                     const json::exception& ex)
+    {
+        std::cout << "byte " << position << ": " << ex.what() << '\n';
+
+        // repair the input and continue
+        return true;
+    }
+};
+
+int main()
+{
+    // JSON text with several mistakes that ends too early
+    const std::string text = R"({
+    "name": "Hello World",
+    "tags": ["a" "b",],
+    "valid": tru,
+    "size": 1.,
+    "nested": {"x": 1)";
+
+    json result;
+    recovering_parser sax(result);
+    const bool valid = json::sax_parse(text, &sax);
+
+    std::cout << "\nvalid JSON: " << std::boolalpha << valid << '\n'
+              << std::setw(4) << result << std::endl;
+}

docs/mkdocs/docs/examples/sax_parse__error_recovery.output

@@ -0,0 +1,19 @@
+byte 49: [json.exception.parse_error.101] parse error at line 3, column 20: syntax error while parsing array - unexpected string literal; expected ']'
+byte 51: [json.exception.parse_error.101] parse error at line 3, column 22: syntax error while parsing value - unexpected ']'; expected '[', '{', or a literal
+byte 70: [json.exception.parse_error.101] parse error at line 4, column 17: syntax error while parsing value - invalid literal; last read: '"valid": tru,'
+byte 86: [json.exception.parse_error.101] parse error at line 5, column 15: syntax error while parsing value - invalid number; expected digit after '.'; last read: '1.,'
+byte 109: [json.exception.parse_error.101] parse error at line 6, column 22: syntax error while parsing object - unexpected end of input; expected '}'
+
+valid JSON: false
+{
+    "name": "Hello World",
+    "nested": {
+        "x": 1
+    },
+    "size": 1,
+    "tags": [
+        "a",
+        "b"
+    ],
+    "valid": null
+}

docs/mkdocs/docs/features/parsing/error_recovery.md

@@ -0,0 +1,122 @@
+# Error Recovery
+
+By default, parsing stops at the first error. With the [SAX interface](sax_interface.md), you can instead ask the
+parser to *recover*: to repair the error and continue, so that you get as much as possible out of malformed input, for
+instance a file that was cut off, JSON edited by hand, or the output of a language model.
+
+## Recovering from errors
+
+The SAX parser's [`parse_error`](../../api/json_sax/parse_error.md) function is called for every error. Its return value
+decides what happens next:
+
+- `#!cpp false` stops parsing. This is what the SAX parsers of the library do, so [`parse`](../../api/basic_json/parse.md)
+  and [`accept`](../../api/basic_json/accept.md) never recover.
+- `#!cpp true` repairs the error and continues parsing.
+
+When recovering, the SAX parser still receives well-formed events: every `start_object` or `start_array` is followed by
+the matching `end_object` or `end_array`, and every `key` is followed by exactly one value. A SAX parser that creates a
+JSON value, such as the one in the example below, therefore gets a complete value. Parsing always ends, and
+[`sax_parse`](../../api/basic_json/sax_parse.md) returns `#!cpp false` for input that is not valid JSON, even if every
+error was repaired. Each token is reported at most once, and the SAX parser can stop at any error by returning
+`#!cpp false`.
+
+!!! example
+
+    The example below derives a SAX parser from the library's parser for `json` values (`json_sax_dom_parser`),
+    and recovers from all errors.
+
+    ```cpp
+    --8<-- "examples/sax_parse__error_recovery.cpp"
+    ```
+    
+    Output:
+    
+    ```
+    --8<-- "examples/sax_parse__error_recovery.output"
+    ```
+
+## How errors are repaired
+
+Each error is repaired with the smallest local edit: a missing separator is inserted, a stray token is removed, what can
+be read of a broken string or number is kept, and a value that cannot be read at all becomes `#!json null`.
+
+| Mistake                   | Repair                                                                         | Example                                  | Result                     |
+|---------------------------|--------------------------------------------------------------------------------|------------------------------------------|----------------------------|
+| missing `,` or `:`        | inserted                                                                       | `#!json [1 2]`, `#!json {"a" 1}`         | `[1,2]`, `{"a":1}`         |
+| missing value             | `#!json null` for an object key or between commas in an array                  | `#!json {"a":}`, `#!json [1,,2]`         | `{"a":null}`, `[1,null,2]` |
+| trailing comma            | removed                                                                        | `#!json [1,2,]`                          | `[1,2]`                    |
+| broken string             | invalid escapes and bytes are replaced (see below); a line break ends the string | `#!json ["a\qb"]`                        | `["aqb"]`                  |
+| broken number             | the longest valid beginning is kept                                            | `#!json [1., 2e+]`                       | `[1,2]`                    |
+| unreadable value          | `#!json null`                                                                  | `#!json [1, NaN, tru]`                   | `[1,null,null]`            |
+| number too large          | passed as infinity, together with its text                                     | `#!json [1e999]`                         | infinity (see below)       |
+| stray `:`                 | removed                                                                        | `#!json ["a":1]`                         | `["a",1]`                  |
+| member without a key      | skipped up to the next `,` or `}`                                              | `#!json {1:2, "b":3}`                    | `{"b":3}`                  |
+| wrong closing bracket     | closes the innermost array or object                                           | `#!json {"a":[1,2}, "b":3}`              | `{"a":[1,2],"b":3}`        |
+| input ends too early      | all open arrays and objects are closed                                         | `#!json {"a":[1,2`                       | `{"a":[1,2]}`              |
+| text before the value     | skipped                                                                        | `#!json )]}'{"a":1}`                     | `{"a":1}`                  |
+
+In a string, an unknown escape like `\q` stands for the escaped character (`q`), as in JavaScript. An invalid `\u`
+escape, a lone surrogate, and ill-formed UTF-8 are each replaced by U+FFFD (REPLACEMENT CHARACTER), and control
+characters are kept. A string without its closing quote ends at the next line break or at the end of the input.
+
+The input after the top-level value is not repaired: as without recovery, it is reported as an error, and parsing stops.
+
+## Binary formats
+
+The binary formats ([BJData](../binary_formats/bjdata.md), [BON8](../binary_formats/bon8.md),
+[BSON](../binary_formats/bson.md), [CBOR](../binary_formats/cbor.md), [MessagePack](../binary_formats/messagepack.md),
+and [UBJSON](../binary_formats/ubjson.md)) have no delimiters to find the next value by. So what can be repaired depends
+on whether the end of the item with the error is known, a distinction that
+[RFC 8949, Section 5.3](https://www.rfc-editor.org/rfc/rfc8949.html#section-5.3) makes for CBOR, too.
+
+If the item is complete, but cannot be passed on as it is, it is replaced, and parsing continues after it:
+
+| Mistake                                                             | Formats                 | Repair                                                                  |
+|---------------------------------------------------------------------|-------------------------|-------------------------------------------------------------------------|
+| tag, if `tag_handler` is `cbor_tag_handler_t::error` (the default)  | CBOR                    | ignored                                                                 |
+| simple value other than `false`, `true`, and `null`, like undefined | CBOR                    | `#!json null`                                                           |
+| number too large for a custom `number_float_t`, like `float`        | all                     | infinity                                                                |
+| character (`C`) that is not ASCII                                   | BJData, UBJSON          | U+FFFD                                                                  |
+| invalid high-precision number (`H`)                                 | BJData, UBJSON          | the longest valid beginning is kept, as for JSON text, or `#!json null` |
+| high-precision number too large                                     | BJData, UBJSON          | passed as infinity, together with its text                              |
+| object key that is not a string                                     | BON8, CBOR, MessagePack | the member is skipped                                                   |
+| element of a type the library does not read, like ObjectId or date  | BSON                    | `#!json null`                                                           |
+| string without its terminator                                       | BSON                    | kept                                                                    |
+| document whose size does not match its content                      | BSON                    | kept                                                                    |
+
+CBOR tags and simple values are repaired as [RFC 8949, Section 6.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-6.1)
+suggests for converting CBOR to JSON. By default, [`sax_parse`](../../api/basic_json/sax_parse.md) reports every tag as
+an error; when recovering, tags are then ignored like with
+[`cbor_tag_handler_t::ignore`](../../api/basic_json/cbor_tag_handler_t.md). Strings that are not valid UTF-8 are no
+error: like [`from_cbor`](../../api/basic_json/from_cbor.md) and the other functions by default, `sax_parse` passes
+them on as they are.
+
+After any other error, the end of the item is unknown: the input ended, a byte is not a valid type marker, or a size
+cannot be right. Parsing then stops, and the value read so far is completed: a key that waits for its value gets
+`#!json null`, and all open arrays and objects are closed. This keeps everything before the error of an input that was
+cut off. The exception is BSON, which stores the size of every document: an element whose end is unknown gets
+`#!json null`, the rest of its document is skipped, and parsing continues after the document.
+
+## Limitations
+
+- A repair is a guess. For example, `#!json {"a" "b": 1}` could be meant as `#!json {"a": "b"}` or as
+  `#!json {"a": null, "b": 1}`; it is repaired to the former. Treat recovered values as a best effort, and check the
+  reported errors.
+- A closing bracket always closes the innermost array or object. If a bracket is missing rather than wrong, the
+  repair differs from the intention: `#!json {"a": {"b": [1, 2}, "c": 3}` is repaired to
+  `#!json {"a": {"b": [1, 2], "c": 3}}`, although `#!json {"a": {"b": [1, 2]}, "c": 3}` may have been meant.
+- Keys without quotes, and strings in single quotes, are not supported; such members are skipped.
+- In the binary formats, a member that is skipped because its key is not a string is lost, and so are the elements of a
+  BSON document after one whose end is unknown.
+- A number that is too large for `number_float_t` is passed as positive or negative infinity. The SAX parser's
+  `number_float` also gets the number's text, but a JSON value cannot store it, and
+  [`dump`](../../api/basic_json/dump.md) serializes infinity as `#!json null`.
+- When parsing is not strict (see [`sax_parse`](../../api/basic_json/sax_parse.md)), a repair may read parts of the
+  input after the value, for instance of the next value in a stream of concatenated values.
+
+## See also
+
+- [SAX interface](sax_interface.md) - implement a custom SAX handler
+- [`parse_error`](../../api/json_sax/parse_error.md) - the SAX event for parse errors
+- [`sax_parse`](../../api/basic_json/sax_parse.md) - generate SAX events
+- [parsing and exceptions](parse_exceptions.md) - control error handling

docs/mkdocs/docs/features/parsing/index.md

@@ -103,7 +103,7 @@ You can influence a DOM parse without switching to the SAX interface by passing
 When the input is not valid JSON, the `parse` function throws an exception by default. If exceptions are undesired or
 unavailable, the parser can instead return a discarded value, or [`accept`](../../api/basic_json/accept.md) can be used
 to only check whether an input is valid JSON. See [parsing and exceptions](parse_exceptions.md) for the available
-options.
+options. To get as much as possible out of malformed input, a SAX parser can [recover from errors](error_recovery.md).
 
 ## See also
 
@@ -114,4 +114,5 @@ options.
 - [parser callbacks](parser_callbacks.md) - influence the parsing by a callback function
 - [SAX interface](sax_interface.md) - implement a custom SAX handler
 - [parsing and exceptions](parse_exceptions.md) - control error handling
+- [error recovery](error_recovery.md) - get as much as possible out of malformed input
 - [parsing untrusted input](untrusted_input.md) - what to consider when parsing input from untrusted sources

docs/mkdocs/docs/features/parsing/parse_exceptions.md

@@ -64,7 +64,8 @@ bool parse_error(std::size_t position,
                  const json::exception& ex);
 ```
 
-The return value indicates whether the parsing should continue, so the function should usually return `#!cpp false`.
+The return value decides whether to stop parsing (`#!cpp false`) or to repair the error and continue
+(`#!cpp true`); see [error recovery](error_recovery.md) for the latter.
 
 ??? example "Example: report parse errors without exceptions"
 

docs/mkdocs/docs/features/parsing/sax_interface.md

@@ -60,7 +60,8 @@ bool key(string_t& val);
 bool parse_error(std::size_t position, const std::string& last_token, const json::exception& ex);
 ```
 
-The return value of each function determines whether parsing should proceed.
+The return value of each function determines whether parsing should proceed. For `parse_error`, returning
+`#!cpp true` [recovers from the error](error_recovery.md).
 
 To implement your own SAX handler, proceed as follows:
 
@@ -68,7 +69,7 @@ To implement your own SAX handler, proceed as follows:
 2. Create an object of your SAX interface class, e.g. `my_sax`.
 3. Call `#!cpp bool json::sax_parse(input, &my_sax);` where the first parameter can be any input like a string or an input stream and the second parameter is a pointer to your SAX interface.
 
-Note the `sax_parse` function only returns a `#!cpp bool` indicating the result of the last executed SAX event. It does not return `json` value - it is up to you to decide what to do with the SAX events. Furthermore, no exceptions are thrown in case of a parse error - it is up to you what to do with the exception object passed to your `parse_error` implementation. Internally, the SAX interface is used for the DOM parser (class `json_sax_dom_parser`) as well as the acceptor (`json_sax_acceptor`), see file `json_sax.hpp`.
+Note the `sax_parse` function only returns a `#!cpp bool` indicating whether the input was parsed without errors and no SAX event returned `#!cpp false`. It does not return `json` value - it is up to you to decide what to do with the SAX events. Furthermore, no exceptions are thrown in case of a parse error - it is up to you what to do with the exception object passed to your `parse_error` implementation. Internally, the SAX interface is used for the DOM parser (class `json_sax_dom_parser`) as well as the acceptor (`json_sax_acceptor`), see file `json_sax.hpp`.
 
 ## See also
 

docs/mkdocs/mkdocs.yml

@@ -88,6 +88,7 @@ nav:
       - features/performance.md
       - Parsing:
           - features/parsing/index.md
+          - features/parsing/error_recovery.md
           - features/parsing/json_lines.md
           - features/parsing/parse_exceptions.md
           - features/parsing/parser_callbacks.md

docs/mkdocs/scripts/mermaid/package-lock.json

@@ -1105,9 +1105,9 @@
       }
     },
     "node_modules/dompurify": {
-      "version": "3.4.16",
-      "resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.4.16.tgz",
-      "integrity": "sha512-sqo+pNp3qRhCIpbgRi1y8Tgk27Bo2Ry7w0dC1NBeNTdZChWjz9Xb/KOoZbRP/R6pQZ80Qw8YhXw13hWWBbMRnQ==",
+      "version": "3.4.17",
+      "resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.4.17.tgz",
+      "integrity": "sha512-B/pSlYVHQ58kqISD/yDjuJ1yfzI34m1gF4lAod2bwSRhiN4K5Cl3H8Af7hqB2PmTL911tdEudnnXn5FnhRoz0w==",
       "license": "(MPL-2.0 OR Apache-2.0)",
       "optionalDependencies": {
         "@types/trusted-types": "^2.0.7"

include/nlohmann/detail/conversions/from_json.hpp

@@ -69,10 +69,12 @@ void from_json(const BasicJsonType& j, std::optional<T>& opt)
 }
 #endif // JSON_HAS_CPP_17
 
-// read a number value of any of the three number types into val;
-// returns false (leaving val untouched) if j is not a number
-template<typename BasicJsonType, typename ArithmeticType>
-bool get_number_value(const BasicJsonType& j, ArithmeticType& val)
+// overloads for basic_json template parameters
+template < typename BasicJsonType, typename ArithmeticType,
+           enable_if_t < std::is_arithmetic<ArithmeticType>::value&&
+                         !std::is_same<ArithmeticType, typename BasicJsonType::boolean_t>::value,
+                         int > = 0 >
+void get_arithmetic_value(const BasicJsonType& j, ArithmeticType& val)
 {
     switch (static_cast<value_t>(j))
     {
@@ -83,17 +85,17 @@ bool get_number_value(const BasicJsonType& j, ArithmeticType& val)
             const auto* ptr = j.template get_ptr<const typename BasicJsonType::number_unsigned_t*>();
             JSON_ASSERT(ptr != nullptr);
             val = static_cast<ArithmeticType>(*ptr);
-            return true;
+            break;
         }
         case value_t::number_integer:
         {
             val = static_cast<ArithmeticType>(*j.template get_ptr<const typename BasicJsonType::number_integer_t*>());
-            return true;
+            break;
         }
         case value_t::number_float:
         {
             val = static_cast<ArithmeticType>(*j.template get_ptr<const typename BasicJsonType::number_float_t*>());
-            return true;
+            break;
         }
 
         case value_t::null:
@@ -104,20 +106,7 @@ bool get_number_value(const BasicJsonType& j, ArithmeticType& val)
         case value_t::binary:
         case value_t::discarded:
         default:
-            return false;
-    }
-}
-
-// overloads for basic_json template parameters
-template < typename BasicJsonType, typename ArithmeticType,
-           enable_if_t < std::is_arithmetic<ArithmeticType>::value&&
-                         !std::is_same<ArithmeticType, typename BasicJsonType::boolean_t>::value,
-                         int > = 0 >
-void get_arithmetic_value(const BasicJsonType& j, ArithmeticType& val)
-{
-    if (JSON_HEDLEY_UNLIKELY(!get_number_value(j, val)))
-    {
-        throw_type_must_be("number", j);
+            throw_type_must_be("number", j);
     }
 }
 
@@ -267,18 +256,18 @@ auto from_json_array_impl(const BasicJsonType& j, std::array<T, N>& arr,
     }
 }
 
-// reserve() is called through this pair (used for arrays and objects alike)
-// so from_json_array_impl and the object from_json have a single body for both
-// containers that support reserve() and those that don't.
-template<typename ConstructibleContainerType>
-auto reserve_if_possible(ConstructibleContainerType& c, typename ConstructibleContainerType::size_type size, priority_tag<1> /*unused*/)
--> decltype(c.reserve(size), void())
+// reserve() is called through this pair (modeled on from_json_object_reserve)
+// so from_json_array_impl below has a single body for both ConstructibleArrayType
+// that support reserve() and those that don't.
+template<typename ConstructibleArrayType>
+auto from_json_array_reserve(ConstructibleArrayType& arr, typename ConstructibleArrayType::size_type size, priority_tag<1> /*unused*/)
+-> decltype(arr.reserve(size), void())
 {
-    c.reserve(size);
+    arr.reserve(size);
 }
 
-template<typename ConstructibleContainerType>
-void reserve_if_possible(ConstructibleContainerType& /*c*/, std::size_t /*size*/, priority_tag<0> /*unused*/)
+template<typename ConstructibleArrayType>
+void from_json_array_reserve(ConstructibleArrayType& /*arr*/, std::size_t /*size*/, priority_tag<0> /*unused*/)
 {}
 
 // append through push_back() where available; containers without it (such as
@@ -308,7 +297,7 @@ auto from_json_array_impl(const BasicJsonType& j, ConstructibleArrayType& arr, p
 {
     ConstructibleArrayType ret;
     const auto* inner_array = j.template get_ptr<const typename BasicJsonType::array_t*>();
-    reserve_if_possible(ret, inner_array->size(), priority_tag<1> {});
+    from_json_array_reserve(ret, inner_array->size(), priority_tag<1> {});
     for (const auto& elem : *inner_array)
     {
         // get<BasicJsonType>() returns *this, this won't call a from_json
@@ -389,6 +378,17 @@ inline void from_json(const BasicJsonType& j, CompatibleArrayType& bin)
     }
 }
 
+template<typename ConstructibleObjectType>
+auto from_json_object_reserve(ConstructibleObjectType& obj, typename ConstructibleObjectType::size_type size, priority_tag<1> /*unused*/)
+-> decltype(obj.reserve(size), void())
+{
+    obj.reserve(size);
+}
+
+template<typename ConstructibleObjectType>
+inline void from_json_object_reserve(ConstructibleObjectType& /*obj*/, std::size_t /*size*/, priority_tag<0> /*unused*/)
+{}
+
 // the keys of the source are sorted for the usual object types, so hinting the
 // end makes every insertion constant-time (for containers that order their keys
 // the same way); containers without emplace_hint() (such as ordered_map) fall
@@ -417,7 +417,7 @@ inline void from_json(const BasicJsonType& j, ConstructibleObjectType& obj)
 
     ConstructibleObjectType ret;
     const auto* inner_object = j.template get_ptr<const typename BasicJsonType::object_t*>();
-    reserve_if_possible(ret, inner_object->size(), priority_tag<1> {});
+    from_json_object_reserve(ret, inner_object->size(), priority_tag<1> {});
     for (const auto& p : *inner_object)
     {
         from_json_object_emplace(ret, p.first, p.second.template get<typename ConstructibleObjectType::mapped_type>(), priority_tag<1> {});
@@ -437,10 +437,6 @@ template < typename BasicJsonType, typename ArithmeticType,
                int > = 0 >
 inline void from_json(const BasicJsonType& j, ArithmeticType& val)
 {
-    // the number cases repeat get_number_value() on purpose: calling it here
-    // changes the inlining of get<T>() in the std::array conversion, which then
-    // runs 2.4x slower (GetArray benchmark, #5891); unlike get_arithmetic_value(),
-    // a boolean is accepted
     switch (static_cast<value_t>(j))
     {
         case value_t::number_unsigned:

include/nlohmann/detail/exceptions.hpp

@@ -66,7 +66,6 @@ enum class exception_id : int
     invalid_string_or_size = 113, ///< a binary input contains an invalid string or size specification
     bson_unsupported_type = 114, ///< a BSON record type is not supported
     invalid_high_precision_number = 115, ///< a UBJSON/BJData high-precision number cannot be parsed
-    invalid_document_image = 116, ///< json_document::load() rejects an image
     // invalid_iterator
     iterators_incompatible = 201, ///< the iterators of a range belong to different values
     iterator_from_other_value = 202, ///< an iterator does not belong to the value it is used with
@@ -101,8 +100,6 @@ enum class exception_id : int
     invalid_utf8 = 316, ///< dump() finds a string that is not valid UTF-8
     type_not_serializable = 317, ///< a value cannot be serialized to the requested binary format
     enum_key_duplicate = 318, ///< an enum-keyed map has two keys that serialize to the same string
-    document_binary_value = 319, ///< a binary value is stored in a json_document
-    document_image_unsupported = 320, ///< a json_document image cannot be saved or loaded (discarded document, big-endian target)
     discarded_value_used = 321, ///< a discarded value is used
     // out_of_range
     array_index_out_of_range = 401, ///< an array index is out of range
@@ -120,7 +117,6 @@ enum class exception_id : int
     patch_remove_parent_not_container = 413, ///< the parent of a JSON Patch 'remove' target is not a container
     patch_move_into_child = 414, ///< a JSON Patch 'move' moves a value into one of its children
     subtype_out_of_range = 415, ///< a binary subtype does not fit into one byte
-    document_too_large = 416, ///< a json_document input, its edit storage, or an image reaches the 32-bit (4 GiB) limit
     // other_error
     internal_error = 500, ///< unreachable code was reached
     patch_test_failed = 501, ///< a JSON Patch 'test' operation failed
@@ -150,13 +146,6 @@ class exception : public std::exception
         return concat("[json.exception.", ename, '.', std::to_string(id_), "] ");
     }
 
-    /// build the message "[json.exception.<ename>.<id>] <diagnostics><what_arg>"
-    template<typename BasicJsonContext>
-    static std::string what_impl(const std::string& ename, int id_, const std::string& what_arg, BasicJsonContext context)
-    {
-        return concat(name(ename, id_), diagnostics(context), what_arg);
-    }
-
     static std::string diagnostics(std::nullptr_t /*leaf_element*/)
     {
         return "";
@@ -169,33 +158,48 @@ class exception : public std::exception
         std::vector<std::string> tokens;
         for (const auto* current = leaf_element; current != nullptr && current->m_parent != nullptr; current = current->m_parent)
         {
-            const auto* parent = current->m_parent;
-            if (parent->is_array())
+            switch (current->m_parent->type())
             {
-                for (std::size_t i = 0; i < parent->m_data.m_value.array->size(); ++i)
+                case value_t::array:
                 {
-                    if (&parent->m_data.m_value.array->operator[](i) == current)
+                    for (std::size_t i = 0; i < current->m_parent->m_data.m_value.array->size(); ++i)
                     {
-                        tokens.emplace_back(std::to_string(i));
-                        break;
+                        if (&current->m_parent->m_data.m_value.array->operator[](i) == current)
+                        {
+                            tokens.emplace_back(std::to_string(i));
+                            break;
+                        }
                     }
+                    break;
                 }
-            }
-            else if (parent->is_object())
-            {
-                for (const auto& element : *parent->m_data.m_value.object)
+
+                case value_t::object:
                 {
-                    if (&element.second == current)
+                    for (const auto& element : *current->m_parent->m_data.m_value.object)
                     {
-                        // data() is null-terminated, so a key containing
-                        // a null byte is cut short here rather than
-                        // truncating the whole message at what()
-                        tokens.emplace_back(element.first.data());
-                        break;
+                        if (&element.second == current)
+                        {
+                            // data() is null-terminated, so a key containing
+                            // a null byte is cut short here rather than
+                            // truncating the whole message at what()
+                            tokens.emplace_back(element.first.data());
+                            break;
+                        }
                     }
+                    break;
                 }
+
+                case value_t::null: // LCOV_EXCL_LINE
+                case value_t::string: // LCOV_EXCL_LINE
+                case value_t::boolean: // LCOV_EXCL_LINE
+                case value_t::number_integer: // LCOV_EXCL_LINE
+                case value_t::number_unsigned: // LCOV_EXCL_LINE
+                case value_t::number_float: // LCOV_EXCL_LINE
+                case value_t::binary: // LCOV_EXCL_LINE
+                case value_t::discarded: // LCOV_EXCL_LINE
+                default:   // LCOV_EXCL_LINE
+                    break; // LCOV_EXCL_LINE
             }
-            // all other parent types (primitives) contribute no token
         }
 
         if (tokens.empty())
@@ -311,7 +315,7 @@ class invalid_iterator : public exception
     template<typename BasicJsonContext, enable_if_t<is_basic_json_context<BasicJsonContext>::value, int> = 0>
     static invalid_iterator create(int id_, const std::string& what_arg, BasicJsonContext context)
     {
-        const std::string w = exception::what_impl("invalid_iterator", id_, what_arg, context);
+        const std::string w = concat(exception::name("invalid_iterator", id_), exception::diagnostics(context), what_arg);
         return {id_, w.c_str()};
     }
 
@@ -335,7 +339,7 @@ class type_error : public exception
     template<typename BasicJsonContext, enable_if_t<is_basic_json_context<BasicJsonContext>::value, int> = 0>
     static type_error create(int id_, const std::string& what_arg, BasicJsonContext context)
     {
-        const std::string w = exception::what_impl("type_error", id_, what_arg, context);
+        const std::string w = concat(exception::name("type_error", id_), exception::diagnostics(context), what_arg);
         return {id_, w.c_str()};
     }
 
@@ -358,7 +362,7 @@ class out_of_range : public exception
     template<typename BasicJsonContext, enable_if_t<is_basic_json_context<BasicJsonContext>::value, int> = 0>
     static out_of_range create(int id_, const std::string& what_arg, BasicJsonContext context)
     {
-        const std::string w = exception::what_impl("out_of_range", id_, what_arg, context);
+        const std::string w = concat(exception::name("out_of_range", id_), exception::diagnostics(context), what_arg);
         return {id_, w.c_str()};
     }
 
@@ -381,7 +385,7 @@ class other_error : public exception
     template<typename BasicJsonContext, enable_if_t<is_basic_json_context<BasicJsonContext>::value, int> = 0>
     static other_error create(int id_, const std::string& what_arg, BasicJsonContext context)
     {
-        const std::string w = exception::what_impl("other_error", id_, what_arg, context);
+        const std::string w = concat(exception::name("other_error", id_), exception::diagnostics(context), what_arg);
         return {id_, w.c_str()};
     }
 

include/nlohmann/detail/input/input_adapters.hpp

@@ -67,36 +67,6 @@ namespace detail
 /// the supported input formats
 enum class input_format_t { json, cbor, msgpack, ubjson, bson, bjdata, bon8 };
 
-/// @return the name of a binary input format, as used in exception messages
-inline const char* input_format_name(const input_format_t format) noexcept
-{
-    switch (format)
-    {
-        case input_format_t::cbor:
-            return "CBOR";
-        case input_format_t::msgpack:
-            return "MessagePack";
-        case input_format_t::ubjson:
-            return "UBJSON";
-        case input_format_t::bson:
-            return "BSON";
-        case input_format_t::bjdata:
-            return "BJData";
-        case input_format_t::bon8:
-            return "BON8";
-        case input_format_t::json: // LCOV_EXCL_LINE
-        default:            // LCOV_EXCL_LINE
-            JSON_ASSERT(false); // NOLINT(cert-dcl03-c,hicpp-static-assert,misc-static-assert) LCOV_EXCL_LINE
-            return ""; // LCOV_EXCL_LINE
-    }
-}
-
-/// @return the message of the parse_error thrown or reported for an empty input
-inline const char* empty_input_message() noexcept
-{
-    return "attempting to parse an empty input; check that your input string or stream contains the expected JSON";
-}
-
 ////////////////////
 // input adapters //
 ////////////////////
@@ -796,17 +766,22 @@ class iterator_input_adapter
         return get_consumed_count() + eof_reads;
     }
 
-    // number of newlines in the consumed range [0, last_index); @a line_start
-    // is set to the index after the last of them (0 if there is none); only
-    // valid when supports_seek
-    std::size_t count_consumed_lines(std::size_t last_index, std::size_t& line_start) const
+    // number of newlines in the consumed range [first_index, last_index); if
+    // there is one, @a line_start is set to the index after the last of them,
+    // otherwise it is left unchanged; only valid when supports_seek
+    std::size_t count_consumed_lines(std::size_t first_index, std::size_t last_index, std::size_t& line_start) const
     {
-        const auto to = std::next(begin, static_cast<typename std::iterator_traits<IteratorType>::difference_type>(last_index));
+        using difference_type = typename std::iterator_traits<IteratorType>::difference_type;
+        const auto from = std::next(begin, static_cast<difference_type>(first_index));
+        const auto to = std::next(from, static_cast<difference_type>(last_index - first_index));
         const auto newline = char_traits<char_type>::to_char_type('\n');
         using reverse_it = std::reverse_iterator<IteratorType>;
-        const auto last_newline = std::find(reverse_it(to), reverse_it(begin), newline);
-        line_start = static_cast<std::size_t>(std::distance(begin, last_newline.base()));
-        return static_cast<std::size_t>(std::count(begin, last_newline.base(), newline));
+        const auto last_newline = std::find(reverse_it(to), reverse_it(from), newline);
+        if (last_newline.base() != from)
+        {
+            line_start = static_cast<std::size_t>(std::distance(begin, last_newline.base()));
+        }
+        return static_cast<std::size_t>(std::count(from, last_newline.base(), newline));
     }
 
     // append the already-consumed characters in the half-open range
@@ -1298,7 +1273,7 @@ inline file_input_adapter input_adapter(std::FILE* file)
 {
     if (file == nullptr)
     {
-        JSON_THROW(parse_error::create(exception_id::syntax_error, 0, empty_input_message(), nullptr));
+        JSON_THROW(parse_error::create(exception_id::syntax_error, 0, "attempting to parse an empty input; check that your input string or stream contains the expected JSON", nullptr));
     }
     return file_input_adapter(file);
 }
@@ -1307,7 +1282,7 @@ inline input_stream_adapter input_adapter(std::istream& stream)
 {
     if (stream.rdbuf() == nullptr)
     {
-        JSON_THROW(parse_error::create(exception_id::syntax_error, 0, empty_input_message(), nullptr));
+        JSON_THROW(parse_error::create(exception_id::syntax_error, 0, "attempting to parse an empty input; check that your input string or stream contains the expected JSON", nullptr));
     }
     return input_stream_adapter(stream);
 }
@@ -1332,7 +1307,7 @@ contiguous_bytes_input_adapter input_adapter(CharT b)
 {
     if (b == nullptr)
     {
-        JSON_THROW(parse_error::create(exception_id::syntax_error, 0, empty_input_message(), nullptr));
+        JSON_THROW(parse_error::create(exception_id::syntax_error, 0, "attempting to parse an empty input; check that your input string or stream contains the expected JSON", nullptr));
     }
     auto length = std::strlen(reinterpret_cast<const char*>(b));
     const auto* ptr = reinterpret_cast<const char*>(b);

include/nlohmann/detail/input/parser.hpp

@@ -56,8 +56,8 @@ using parser_callback_t =
 template<typename SAX>
 struct sax_ignores_number_text : std::false_type {};
 
-template<typename BasicJsonType, typename InputAdapterType>
-struct sax_ignores_number_text<json_sax_dom_parser<BasicJsonType, InputAdapterType>> : std::true_type {};
+template<typename BasicJsonType, typename InputAdapterType, bool Collect>
+struct sax_ignores_number_text<json_sax_dom_parser<BasicJsonType, InputAdapterType, Collect>> : std::true_type {};
 
 template<typename BasicJsonType, typename InputAdapterType>
 struct sax_ignores_number_text<json_sax_dom_callback_parser<BasicJsonType, InputAdapterType>> : std::true_type {};
@@ -131,7 +131,7 @@ class parser
         }
         else
         {
-            json_sax_dom_parser<BasicJsonType, InputAdapterType> sdp(result, allow_exceptions, &m_lexer);
+            json_sax_dom_parser<BasicJsonType, InputAdapterType, true> sdp(result, allow_exceptions, &m_lexer);
 
             // in case of an error, return a discarded value
             if (!parse_dom(sdp, strict))
@@ -153,25 +153,72 @@ class parser
     bool accept(const bool strict = true)
     {
         json_sax_acceptor<BasicJsonType> sax_acceptor;
-        return sax_parse(&sax_acceptor, strict);
+        return sax_parse_impl<false>(&sax_acceptor, strict);
     }
 
+    /*!
+    @brief public SAX interface
+
+    If the SAX parser's parse_error() returns true, the parser recovers from
+    the error: it repairs the input and continues (see #3989).
+
+    @param[in] sax     the SAX parser
+    @param[in] strict  whether to expect the last token to be EOF
+    @return whether the input was parsed without errors and no SAX event
+            returned false
+    */
     template<typename SAX>
     JSON_HEDLEY_NON_NULL(2)
     bool sax_parse(SAX* sax, const bool strict = true)
     {
-        (void)detail::is_sax_static_asserts<SAX, BasicJsonType> {};
-        const bool result = sax_parse_internal(sax);
-
-        if (result)
-        {
-            return finish_input(*sax, strict);
-        }
-
-        return false;
+        return sax_parse_impl<true>(sax, strict);
     }
 
   private:
+    /// what sax_parse_internal() does after an object key was expected
+    enum class next_step : std::uint8_t
+    {
+        /// stop parsing
+        stop,
+        /// parse a value that begins with last_token
+        parse_value,
+        /// evaluate the state of the innermost container, which reads
+        /// last_token again
+        evaluate_state
+    };
+
+    template<bool AllowRecovery, typename SAX>
+    JSON_HEDLEY_NON_NULL(2)
+    bool sax_parse_impl(SAX* sax, const bool strict)
+    {
+        (void)detail::is_sax_static_asserts<SAX, BasicJsonType> {};
+        const bool result = sax_parse_internal<AllowRecovery>(sax);
+
+        if (result)
+        {
+            if (strict)
+            {
+                // strict mode: next byte must be EOF; after recovering from an
+                // error, the end of the input may already have been read
+                if (last_token != token_type::end_of_input && get_token() != token_type::end_of_input)
+                {
+                    // the value is complete, so there is nothing to recover
+                    static_cast<void>(report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(), exception_message(token_type::end_of_input, "value"), nullptr),
+                                                   std::integral_constant<bool, AllowRecovery> {}));
+                    return false;
+                }
+            }
+            else
+            {
+                // the caller keeps using the input: position it right after
+                // the value by leaving the character that terminated it
+                m_lexer.release_lookahead();
+            }
+        }
+
+        return result && !error_reported;
+    }
+
     /// the text of the current float token for @a sax (see sax_ignores_number_text)
     template<typename SAX>
     const string_t& number_float_text(SAX* /*sax*/)
@@ -189,39 +236,6 @@ class parser
         return m_lexer.get_string();
     }
 
-    /*!
-    @brief apply the end-of-input handling after a value was parsed
-
-    In strict mode, the next token must be the end of the input; otherwise
-    parse_error.101 is reported through @a sax. In non-strict mode, the
-    lookahead is released so the caller can keep reading the input right after
-    the parsed value.
-
-    @param[in,out] sax  the SAX parser to report an error to
-    @param[in] strict   whether to expect the last token to be EOF
-    @return true, or what @a sax returned for the error
-    */
-    template<typename SAX>
-    bool finish_input(SAX& sax, const bool strict)
-    {
-        if (strict)
-        {
-            // strict mode: next byte must be EOF
-            if (get_token() != token_type::end_of_input)
-            {
-                return syntax_error(sax, exception_message(token_type::end_of_input, "value"));
-            }
-        }
-        else
-        {
-            // the caller keeps using the input: position it right after
-            // the value by leaving the character that terminated it
-            m_lexer.release_lookahead();
-        }
-
-        return true;
-    }
-
     /*!
     @brief run a DOM SAX parser to completion and position the lexer
 
@@ -239,23 +253,52 @@ class parser
     template<typename DomSax>
     bool parse_dom(DomSax& sdp, const bool strict)
     {
-        sax_parse_internal(&sdp);
+        sax_parse_internal<false>(&sdp);
 
-        static_cast<void>(finish_input(sdp, strict));
+        if (strict)
+        {
+            // in strict mode, input must be completely read
+            if (get_token() != token_type::end_of_input)
+            {
+                syntax_error(sdp, exception_message(token_type::end_of_input, "value"));
+            }
+        }
+        else
+        {
+            // the caller keeps using the input: position it right after
+            // the value by leaving the character that terminated it
+            m_lexer.release_lookahead();
+        }
 
         return !sdp.is_errored();
     }
 
-    template<typename SAX>
+    /*!
+    @brief parse a JSON value and pass it to a SAX parser
+
+    @tparam AllowRecovery  whether to recover from an error if the SAX parser's
+                           parse_error() returns true; false for the SAX parsers
+                           of parse() and accept(), which never do, so that no
+                           code for recovering is generated for them
+    */
+    template<bool AllowRecovery, typename SAX>
     JSON_HEDLEY_NON_NULL(2)
     bool sax_parse_internal(SAX* sax)
     {
+        const std::integral_constant<bool, AllowRecovery> allow_recovery{};
+        // value to avoid a goto (see comment where set to true)
+        bool skip_to_state_evaluation = false;
+
         // stack to remember the hierarchy of structured values we are parsing
         // true = array; false = object (bytes, not std::vector<bool>, whose
         // push_back() is a call with GCC)
         std::vector<std::uint8_t> states;
-        // value to avoid a goto (see comment where set to true)
-        bool skip_to_state_evaluation = false;
+        // a document that starts with a container pushes onto the stack at
+        // once: allocate it in one step instead of growing it from 1 byte
+        if (last_token == token_type::begin_object || last_token == token_type::begin_array)
+        {
+            states.reserve(32);
+        }
 
         while (true)
         {
@@ -281,10 +324,18 @@ class parser
                             break;
                         }
 
-                        // parse key
+                        // remember we are now inside an object
+                        states.push_back(0);
+
+                        // parse key (the steps of parse_key(), which are
+                        // repeated here and below for speed)
                         if (JSON_HEDLEY_UNLIKELY(last_token != token_type::value_string))
                         {
-                            return syntax_error(*sax, exception_message(token_type::value_string, "object key"));
+                            if (!continue_after(key_error(sax, allow_recovery, false), skip_to_state_evaluation))
+                            {
+                                return false;
+                            }
+                            continue;
                         }
                         if (JSON_HEDLEY_UNLIKELY(!sax->key(m_lexer.get_string_token())))
                         {
@@ -294,12 +345,13 @@ class parser
                         // parse separator (:)
                         if (JSON_HEDLEY_UNLIKELY(!get_token_expecting(token_type::name_separator)))
                         {
-                            return syntax_error(*sax, exception_message(token_type::name_separator, "object separator"));
+                            if (!continue_after(key_error(sax, allow_recovery, true), skip_to_state_evaluation))
+                            {
+                                return false;
+                            }
+                            continue;
                         }
 
-                        // remember we are now inside an object
-                        states.push_back(0);
-
                         // parse values
                         get_token();
                         continue;
@@ -335,9 +387,11 @@ class parser
 
                         if (JSON_HEDLEY_UNLIKELY(!std::isfinite(res)))
                         {
-                            return sax->parse_error(m_lexer.get_position(),
-                                                    m_lexer.get_token_string(),
-                                                    out_of_range::create(exception_id::number_overflow, concat("number overflow parsing '", m_lexer.get_token_string(), '\''), nullptr));
+                            if (!overflow_error(sax, res, allow_recovery))
+                            {
+                                return false;
+                            }
+                            break;
                         }
 
                         if (JSON_HEDLEY_UNLIKELY(!sax->number_float(res, number_float_text(sax))))
@@ -405,16 +459,63 @@ class parser
                     case token_type::parse_error:
                     {
                         // using "uninitialized" to avoid an "expected" message
-                        return syntax_error(*sax, exception_message(token_type::uninitialized, "value"));
+                        if (!report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(), exception_message(token_type::uninitialized, "value"), nullptr), allow_recovery))
+                        {
+                            return false;
+                        }
+
+                        // recover: keep what can be read of the token
+                        recover_token(allow_recovery);
+                        if (last_token != token_type::uninitialized)
+                        {
+                            // a string or a number
+                            continue;
+                        }
+                        if (states.empty())
+                        {
+                            // look for the value after the garbage
+                            if (!skip_to_value(allow_recovery))
+                            {
+                                return false;
+                            }
+                            continue;
+                        }
+                        // nothing could be read
+                        if (JSON_HEDLEY_UNLIKELY(!sax->null()))
+                        {
+                            return false;
+                        }
+                        break;
                     }
                     case token_type::end_of_input:
                     {
                         if (JSON_HEDLEY_UNLIKELY(m_lexer.get_chars_read_total() == 1))
                         {
-                            return syntax_error(*sax, empty_input_message());
+                            // there is nothing to recover
+                            static_cast<void>(report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(),
+                                                           "attempting to parse an empty input; check that your input string or stream contains the expected JSON", nullptr), allow_recovery));
+                            return false;
                         }
 
-                        return syntax_error(*sax, exception_message(token_type::literal_or_value, "value"));
+                        if (!report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(), exception_message(token_type::literal_or_value, "value"), nullptr), allow_recovery))
+                        {
+                            return false;
+                        }
+
+                        // recover: the input ends where a value is missing
+                        if (states.empty())
+                        {
+                            // there is no value
+                            return false;
+                        }
+                        if (!recover_missing_value(sax, states, allow_recovery))
+                        {
+                            return false;
+                        }
+                        // the state evaluation reads the token again
+                        unget_token(allow_recovery);
+                        skip_to_state_evaluation = true;
+                        continue;
                     }
                     case token_type::uninitialized:
                     case token_type::end_array:
@@ -424,7 +525,35 @@ class parser
                     case token_type::literal_or_value:
                     default: // the last token was unexpected
                     {
-                        return syntax_error(*sax, exception_message(token_type::literal_or_value, "value"));
+                        if (!report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(), exception_message(token_type::literal_or_value, "value"), nullptr), allow_recovery))
+                        {
+                            return false;
+                        }
+
+                        // recover
+                        if (states.empty())
+                        {
+                            // look for the value after the garbage
+                            if (!skip_to_value(allow_recovery))
+                            {
+                                return false;
+                            }
+                            continue;
+                        }
+                        if (last_token == token_type::name_separator)
+                        {
+                            // a stray ':'; the value may follow
+                            get_token();
+                            continue;
+                        }
+                        if (!recover_missing_value(sax, states, allow_recovery))
+                        {
+                            return false;
+                        }
+                        // the state evaluation reads the token again
+                        unget_token(allow_recovery);
+                        skip_to_state_evaluation = true;
+                        continue;
                     }
                 }
             }
@@ -475,7 +604,30 @@ class parser
                     continue;
                 }
 
-                return syntax_error(*sax, exception_message(token_type::end_array, "array"));
+                if (!report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(), exception_message(token_type::end_array, "array"), nullptr), allow_recovery))
+                {
+                    return false;
+                }
+
+                // recover
+                if (last_token == token_type::end_of_input)
+                {
+                    // the input ends inside the array
+                    return close_containers(sax, states, allow_recovery);
+                }
+                if (last_token == token_type::end_object)
+                {
+                    // a wrong closing bracket closes the innermost container
+                    if (JSON_HEDLEY_UNLIKELY(!sax->end_array()))
+                    {
+                        return false;
+                    }
+                    states.pop_back();
+                    skip_to_state_evaluation = true;
+                }
+                // otherwise, a missing ',' (or a stray ':', which value
+                // parsing drops): the next value begins here
+                continue;
             }
 
             // states.back() is false -> object
@@ -492,9 +644,12 @@ class parser
                     // parse key
                     if (JSON_HEDLEY_UNLIKELY(last_token != token_type::value_string))
                     {
-                        return syntax_error(*sax, exception_message(token_type::value_string, "object key"));
+                        if (!continue_after(key_error(sax, allow_recovery, false), skip_to_state_evaluation))
+                        {
+                            return false;
+                        }
+                        continue;
                     }
-
                     if (JSON_HEDLEY_UNLIKELY(!sax->key(m_lexer.get_string_token())))
                     {
                         return false;
@@ -503,7 +658,11 @@ class parser
                     // parse separator (:)
                     if (JSON_HEDLEY_UNLIKELY(!get_token_expecting(token_type::name_separator)))
                     {
-                        return syntax_error(*sax, exception_message(token_type::name_separator, "object separator"));
+                        if (!continue_after(key_error(sax, allow_recovery, true), skip_to_state_evaluation))
+                        {
+                            return false;
+                        }
+                        continue;
                     }
 
                     // parse values
@@ -531,10 +690,546 @@ class parser
                 continue;
             }
 
-            return syntax_error(*sax, exception_message(token_type::end_object, "object"));
+            if (!report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(), exception_message(token_type::end_object, "object"), nullptr), allow_recovery))
+            {
+                return false;
+            }
+
+            // recover
+            if (last_token == token_type::end_of_input)
+            {
+                // the input ends inside the object
+                return close_containers(sax, states, allow_recovery);
+            }
+            if (last_token == token_type::end_array)
+            {
+                // a wrong closing bracket closes the innermost container
+                if (JSON_HEDLEY_UNLIKELY(!sax->end_object()))
+                {
+                    return false;
+                }
+                states.pop_back();
+                skip_to_state_evaluation = true;
+                continue;
+            }
+            if (!continue_after(recover_member(sax, allow_recovery), skip_to_state_evaluation))
+            {
+                return false;
+            }
         }
     }
 
+    /*!
+    @brief continue sax_parse_internal() after a recovery
+    @return whether to continue parsing
+    */
+    bool continue_after(const next_step step, bool& skip_to_state_evaluation)
+    {
+        if (step == next_step::evaluate_state)
+        {
+            // the state evaluation reads the token again
+            m_lexer.unget_token();
+            skip_to_state_evaluation = true;
+        }
+        return step != next_step::stop;
+    }
+
+    /// the parser for parse() and accept() never recovers: stop parsing
+    JSON_INTERNAL_ALWAYS_INLINE
+    static std::false_type continue_after(std::false_type /*step*/, bool& /*skip_to_state_evaluation*/) noexcept
+    {
+        return {};
+    }
+
+    /*!
+    @brief parse an object key and the name separator (:) after it
+
+    last_token is the token where the key is expected. sax_parse_internal()
+    repeats these steps rather than calling this function, which is used
+    when recovering from an error.
+
+    @return next_step::parse_value if the value follows, with last_token its
+            first token; next_step::evaluate_state if the object's state is
+            to be evaluated after recovering from an error; next_step::stop
+            to stop parsing
+    */
+    template<typename SAX>
+    next_step parse_key(SAX* sax)
+    {
+        const std::true_type allow_recovery{};
+
+        if (JSON_HEDLEY_UNLIKELY(last_token != token_type::value_string))
+        {
+            return key_error(sax, allow_recovery, false);
+        }
+
+        if (JSON_HEDLEY_UNLIKELY(!sax->key(m_lexer.get_string_token())))
+        {
+            return next_step::stop;
+        }
+
+        // parse separator (:)
+        if (JSON_HEDLEY_UNLIKELY(get_token() != token_type::name_separator))
+        {
+            return key_error(sax, allow_recovery, true);
+        }
+
+        // the value begins with the next token
+        get_token();
+        return next_step::parse_value;
+    }
+
+    /*!
+    @brief report a number that is too large for number_float_t, and recover
+           from the error by passing the value on; the SAX parser gets the
+           number's text as well
+
+    This is a separate function, as reading other numbers is measurably
+    slower if the error is handled where they are read.
+
+    @param[in] sax    the SAX parser
+    @param[in] value  the value that is not finite
+    @return whether to continue parsing
+    */
+    template<typename SAX>
+    bool overflow_error(SAX* sax, const number_float_t value, std::true_type allow_recovery)
+    {
+        if (!report_error(sax, out_of_range::create(exception_id::number_overflow, concat("number overflow parsing '", m_lexer.get_token_string(), '\''), nullptr), allow_recovery))
+        {
+            return false;
+        }
+        return sax->number_float(value, number_float_text(sax));
+    }
+
+    /// @copydoc overflow_error
+    template<typename SAX>
+    JSON_INTERNAL_ALWAYS_INLINE
+    std::false_type overflow_error(SAX* sax, const number_float_t /*value*/, std::false_type allow_recovery)
+    {
+        return report_error(sax, out_of_range::create(exception_id::number_overflow, concat("number overflow parsing '", m_lexer.get_token_string(), '\''), nullptr), allow_recovery);
+    }
+
+    /*!
+    @brief report a missing key, or a missing name separator (:) after the
+           key; the parser for parse() and accept() never recovers
+
+    @param[in] key_read  whether the key was read, so that the name separator
+                         is missing
+    @return std::false_type, see report_error()
+    */
+    template<typename SAX>
+    JSON_INTERNAL_ALWAYS_INLINE
+    std::false_type key_error(SAX* sax, std::false_type allow_recovery, const bool key_read)
+    {
+        return report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(), key_read
+                            ? exception_message(token_type::name_separator, "object separator")
+                            : exception_message(token_type::value_string, "object key"), nullptr), allow_recovery);
+    }
+
+    /*!
+    @brief report a missing key, or a missing name separator (:) after the
+           key, and recover from it
+
+    @param[in] key_read  whether the key was read, so that the name separator
+                         is missing
+    */
+    template<typename SAX>
+    next_step key_error(SAX* sax, std::true_type allow_recovery, const bool key_read)
+    {
+        if (!key_read)
+        {
+            if (!report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(), exception_message(token_type::value_string, "object key"), nullptr), allow_recovery))
+            {
+                return next_step::stop;
+            }
+            return recover_key(sax);
+        }
+
+        if (!report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(), exception_message(token_type::name_separator, "object separator"), nullptr), allow_recovery))
+        {
+            return next_step::stop;
+        }
+        return recover_name_separator(sax);
+    }
+
+    /////////////////////
+    // error recovery
+    /////////////////////
+
+    /*
+    The functions below repair an error after the SAX parser's parse_error()
+    returned true (see #3989). Each mistake is repaired by the smallest local
+    edit: a missing ',' or ':' is inserted, a stray token is removed, what can
+    be read of an invalid string or number is kept (see
+    lexer::recover_token()), a missing value becomes null, a wrong closing
+    bracket closes the innermost container, and the end of the input closes
+    all of them. The events stay balanced, and every key() is followed by
+    exactly one value.
+
+    A repair hands a token to the state evaluation, by returning it to the
+    lexer (lexer::unget_token()) so that the state evaluation reads it again,
+    only if it is ',', ']', '}', or the end of the input. The state evaluation
+    hands a token to value or key parsing only if it is none of them, so a
+    token is never handed back and forth. Every other step reads a token or
+    closes a container, so parsing always ends.
+    */
+
+    /*!
+    @brief report an error to the SAX parser; the parser for parse() and
+           accept() never recovers
+
+    @return std::false_type rather than false: its value is known where the
+            function is called even if the call is not inlined, so the code
+            for recovering is not generated
+    */
+    template<typename SAX, typename Exception>
+    JSON_INTERNAL_ALWAYS_INLINE
+    std::false_type report_error(SAX* sax, const Exception& ex, std::false_type /*allow_recovery*/)
+    {
+        static_cast<void>(sax); // MSVC 2015 does not count calling a static parse_error() as using it
+        error_reported = true;
+        static_cast<void>(sax->parse_error(m_lexer.get_position(), m_lexer.get_token_string(), ex));
+        return {};
+    }
+
+    /*!
+    @brief report an error to the SAX parser
+    @return whether to recover from the error
+    */
+    template<typename SAX, typename Exception>
+    bool report_error(SAX* sax, const Exception& ex, std::true_type /*allow_recovery*/)
+    {
+        static_cast<void>(sax); // MSVC 2015 does not count calling a static parse_error() as using it
+        const std::size_t position = m_lexer.get_position().chars_read_total;
+        if (error_reported && position == last_error_position && last_token == last_error_token)
+        {
+            // a repair handed on the token of the error it repaired; the
+            // token was reported already, and the SAX parser asked to recover
+            return true;
+        }
+
+        error_reported = true;
+        last_error_position = position;
+        last_error_token = last_token;
+
+        if (!sax->parse_error(m_lexer.get_position(), m_lexer.get_token_string(), ex))
+        {
+            return false;
+        }
+
+        // the token string of the next error begins here
+        m_lexer.restart_token_string();
+        return true;
+    }
+
+    /*!
+    @brief keep what can be read of the token that the lexer rejected
+
+    The error was reported for the rejected token, so it is not reported again
+    for the token it is repaired to (see lexer::recover_token()).
+    */
+    token_type recover_token()
+    {
+        last_token = m_lexer.recover_token();
+        last_error_position = m_lexer.get_position().chars_read_total;
+        last_error_token = last_token;
+        return last_token;
+    }
+
+    // The functions below are called from sax_parse_internal() after an error
+    // was reported. The parser for parse() and accept() stopped there, so for
+    // it they are stubs: the code for recovering is then not even referenced,
+    // and unoptimized builds do not emit it either.
+
+    /// @copydoc recover_token()
+    JSON_INTERNAL_ALWAYS_INLINE
+    token_type recover_token(std::true_type /*allow_recovery*/)
+    {
+        return recover_token();
+    }
+
+    /// @copydoc recover_token()
+    JSON_INTERNAL_ALWAYS_INLINE
+    static std::false_type recover_token(std::false_type /*allow_recovery*/) noexcept
+    {
+        return {};
+    }
+
+    /// return the token that was read last to the lexer (see lexer::unget_token())
+    JSON_INTERNAL_ALWAYS_INLINE
+    void unget_token(std::true_type /*allow_recovery*/)
+    {
+        m_lexer.unget_token();
+    }
+
+    /// @copydoc unget_token(std::true_type)
+    JSON_INTERNAL_ALWAYS_INLINE
+    static void unget_token(std::false_type /*allow_recovery*/) noexcept {}
+
+    /// @copydoc skip_to_value
+    JSON_INTERNAL_ALWAYS_INLINE
+    static std::false_type skip_to_value(std::false_type /*allow_recovery*/) noexcept
+    {
+        return {};
+    }
+
+    /// @copydoc recover_missing_value
+    template<typename SAX>
+    JSON_INTERNAL_ALWAYS_INLINE
+    static std::false_type recover_missing_value(SAX* /*sax*/, const std::vector<std::uint8_t>& /*states*/, std::false_type /*allow_recovery*/) noexcept
+    {
+        return {};
+    }
+
+    /// pass the end events of all open containers
+    template<typename SAX>
+    bool close_containers(SAX* sax, std::vector<std::uint8_t>& states, std::true_type /*allow_recovery*/)
+    {
+        while (!states.empty())
+        {
+            const bool is_array = states.back() != 0;
+            states.pop_back();
+            if (JSON_HEDLEY_UNLIKELY(is_array ? !sax->end_array() : !sax->end_object()))
+            {
+                return false;
+            }
+        }
+        return true;
+    }
+
+    /// @copydoc close_containers
+    template<typename SAX>
+    JSON_INTERNAL_ALWAYS_INLINE
+    static std::false_type close_containers(SAX* /*sax*/, std::vector<std::uint8_t>& /*states*/, std::false_type /*allow_recovery*/) noexcept
+    {
+        return {};
+    }
+
+    /*!
+    @brief read tokens until one begins a value, skipping everything before
+           the top-level value
+    @return whether a value begins with last_token
+    */
+    bool skip_to_value(std::true_type /*allow_recovery*/)
+    {
+        while (true)
+        {
+            switch (get_token())
+            {
+                case token_type::begin_array:
+                case token_type::begin_object:
+                case token_type::literal_false:
+                case token_type::literal_null:
+                case token_type::literal_true:
+                case token_type::value_float:
+                case token_type::value_integer:
+                case token_type::value_string:
+                case token_type::value_unsigned:
+                    return true;
+
+                case token_type::end_of_input:
+                    return false;
+
+                case token_type::parse_error:
+                    recover_token();
+                    if (last_token != token_type::uninitialized)
+                    {
+                        return true;
+                    }
+                    break;
+
+                case token_type::uninitialized:
+                case token_type::end_array:
+                case token_type::end_object:
+                case token_type::name_separator:
+                case token_type::value_separator:
+                case token_type::literal_or_value:
+                default:
+                    break;
+            }
+        }
+    }
+
+    /*!
+    @brief skip the rest of an object member that cannot be read
+
+    Reads tokens, beginning with last_token, until a ',', '}', or ']' that is
+    not inside a container that begins in the skipped tokens, or the end of
+    the input.
+    */
+    void skip_member()
+    {
+        std::size_t depth = 0;
+        while (true)
+        {
+            switch (last_token)
+            {
+                case token_type::begin_array:
+                case token_type::begin_object:
+                    ++depth;
+                    break;
+
+                case token_type::end_array:
+                case token_type::end_object:
+                    if (depth == 0)
+                    {
+                        return;
+                    }
+                    --depth;
+                    break;
+
+                case token_type::value_separator:
+                    if (depth == 0)
+                    {
+                        return;
+                    }
+                    break;
+
+                case token_type::end_of_input:
+                    return;
+
+                case token_type::parse_error:
+                    recover_token();
+                    break;
+
+                case token_type::uninitialized:
+                case token_type::literal_true:
+                case token_type::literal_false:
+                case token_type::literal_null:
+                case token_type::value_string:
+                case token_type::value_unsigned:
+                case token_type::value_integer:
+                case token_type::value_float:
+                case token_type::name_separator:
+                case token_type::literal_or_value:
+                default:
+                    break;
+            }
+            get_token();
+        }
+    }
+
+    /*!
+    @brief pass a value where it is missing
+
+    last_token is ',', ']', '}', or the end of the input, where a value was
+    expected. In an object, the key gets null; in an array, a ',' where a
+    value is missing stands for null (as in JavaScript), while an array that
+    ends there just ends.
+    */
+    template<typename SAX>
+    bool recover_missing_value(SAX* sax, const std::vector<std::uint8_t>& states, std::true_type /*allow_recovery*/)
+    {
+        JSON_ASSERT(!states.empty());
+        if (states.back() == 0 || last_token == token_type::value_separator)
+        {
+            return sax->null();
+        }
+        return true;
+    }
+
+    /// recover from a missing key; last_token is where it was expected
+    template<typename SAX>
+    next_step recover_key(SAX* sax)
+    {
+        switch (last_token)
+        {
+            case token_type::value_separator:
+            case token_type::end_object:
+            case token_type::end_array:
+            case token_type::end_of_input:
+                // no member: the object's state handles the token
+                return next_step::evaluate_state;
+
+            case token_type::parse_error:
+                recover_token();
+                if (last_token == token_type::value_string)
+                {
+                    // a key that could be repaired
+                    return parse_key(sax);
+                }
+                skip_member();
+                return next_step::evaluate_state;
+
+            case token_type::uninitialized:
+            case token_type::literal_true:
+            case token_type::literal_false:
+            case token_type::literal_null:
+            case token_type::value_string:
+            case token_type::value_unsigned:
+            case token_type::value_integer:
+            case token_type::value_float:
+            case token_type::begin_array:
+            case token_type::begin_object:
+            case token_type::name_separator:
+            case token_type::literal_or_value:
+            default:
+                // a member without a key
+                skip_member();
+                return next_step::evaluate_state;
+        }
+    }
+
+    /// recover from a missing name separator (:) after the key; last_token
+    /// is where it was expected
+    template<typename SAX>
+    next_step recover_name_separator(SAX* sax)
+    {
+        switch (last_token)
+        {
+            case token_type::value_separator:
+            case token_type::end_object:
+            case token_type::end_array:
+            case token_type::end_of_input:
+                // the value is missing as well
+                return sax->null() ? next_step::evaluate_state : next_step::stop;
+
+            case token_type::uninitialized:
+            case token_type::literal_true:
+            case token_type::literal_false:
+            case token_type::literal_null:
+            case token_type::value_string:
+            case token_type::value_unsigned:
+            case token_type::value_integer:
+            case token_type::value_float:
+            case token_type::begin_array:
+            case token_type::begin_object:
+            case token_type::name_separator:
+            case token_type::parse_error:
+            case token_type::literal_or_value:
+            default:
+                // a missing ':'; the value begins here
+                return next_step::parse_value;
+        }
+    }
+
+    /// recover from a token after an object member that is neither ',' nor
+    /// '}' (nor ']' or the end of the input, which the caller handles)
+    template<typename SAX>
+    next_step recover_member(SAX* sax, std::true_type /*allow_recovery*/)
+    {
+        if (last_token == token_type::parse_error)
+        {
+            recover_token();
+        }
+        if (last_token == token_type::value_string)
+        {
+            // a missing ','; the next key begins here
+            return parse_key(sax);
+        }
+        skip_member();
+        return next_step::evaluate_state;
+    }
+
+    /// the parser for parse() and accept() never recovers (and does not come
+    /// here, as report_error() returned false)
+    template<typename SAX>
+    JSON_INTERNAL_ALWAYS_INLINE
+    std::false_type recover_member(SAX* /*sax*/, std::false_type /*allow_recovery*/) const noexcept
+    {
+        return {};
+    }
+
     /// get next token from lexer
     token_type get_token()
     {
@@ -605,6 +1300,12 @@ class parser
     const bool allow_exceptions = true;
     /// whether trailing commas in objects and arrays should be ignored (true) or signaled as errors (false)
     const bool ignore_trailing_commas = false;
+    /// whether an error was reported to the SAX parser
+    bool error_reported = false;
+    /// the position of the last reported error
+    std::size_t last_error_position = 0;
+    /// the token of the last reported error
+    token_type last_error_token = token_type::uninitialized;
 };
 
 }  // namespace detail

include/nlohmann/detail/input/string_scan.hpp

@@ -96,7 +96,7 @@ JSON_HEDLEY_ALWAYS_INLINE std::size_t find_string_special(const unsigned char* d
 // printable ASCII except '"' and '\\' - is emitted unchanged. Note this differs
 // from is_string_special() only in that 0x7F is also a stop (it is escaped as
 // \u007f under ensure_ascii).
-constexpr bool is_ascii_copyable(unsigned char c) noexcept
+inline bool is_ascii_copyable(unsigned char c) noexcept
 {
     return c >= 0x20u && c < 0x7Fu && c != '"' && c != '\\';
 }

include/nlohmann/detail/iterators/iter_impl.hpp

@@ -108,18 +108,33 @@ class iter_impl // NOLINT(cppcoreguidelines-special-member-functions,hicpp-speci
     {
         JSON_ASSERT(m_object != nullptr);
 
-        if (m_object->is_object())
+        switch (m_object->m_data.m_type)
         {
-            m_it.object_iterator = typename object_t::iterator();
-        }
-        else if (m_object->is_array())
-        {
-            m_it.array_iterator = typename array_t::iterator();
-        }
-        else
-        {
-            // primitive (is_primitive(), including discarded)
-            m_it.primitive_iterator = primitive_iterator_t();
+            case value_t::object:
+            {
+                m_it.object_iterator = typename object_t::iterator();
+                break;
+            }
+
+            case value_t::array:
+            {
+                m_it.array_iterator = typename array_t::iterator();
+                break;
+            }
+
+            case value_t::null:
+            case value_t::string:
+            case value_t::boolean:
+            case value_t::number_integer:
+            case value_t::number_unsigned:
+            case value_t::number_float:
+            case value_t::binary:
+            case value_t::discarded:
+            default:
+            {
+                m_it.primitive_iterator = primitive_iterator_t();
+                break;
+            }
         }
     }
 
@@ -234,28 +249,33 @@ class iter_impl // NOLINT(cppcoreguidelines-special-member-functions,hicpp-speci
     {
         JSON_ASSERT(m_object != nullptr);
 
-        if (m_object->is_object())
+        switch (m_object->m_data.m_type)
         {
-            m_it.object_iterator = m_object->m_data.m_value.object->end();
-        }
-        else if (m_object->is_array())
-        {
-            m_it.array_iterator = m_object->m_data.m_value.array->end();
-        }
-        else
-        {
-            // primitive (is_primitive(), including discarded)
-            m_it.primitive_iterator.set_end();
-        }
-    }
+            case value_t::object:
+            {
+                m_it.object_iterator = m_object->m_data.m_value.object->end();
+                break;
+            }
 
-    /// @throw invalid_iterator.212 if @a other points to another container
-    template<typename IterImpl>
-    JSON_HEDLEY_ALWAYS_INLINE void check_same_container(const IterImpl& other) const
-    {
-        if (JSON_HEDLEY_UNLIKELY(m_object != other.m_object))
-        {
-            throw_literal<invalid_iterator>(exception_id::iterators_compare_different_values, "cannot compare iterators of different containers", m_object);
+            case value_t::array:
+            {
+                m_it.array_iterator = m_object->m_data.m_value.array->end();
+                break;
+            }
+
+            case value_t::null:
+            case value_t::string:
+            case value_t::boolean:
+            case value_t::number_integer:
+            case value_t::number_unsigned:
+            case value_t::number_float:
+            case value_t::binary:
+            case value_t::discarded:
+            default:
+            {
+                m_it.primitive_iterator.set_end();
+                break;
+            }
         }
     }
 
@@ -404,7 +424,10 @@ class iter_impl // NOLINT(cppcoreguidelines-special-member-functions,hicpp-speci
     bool operator==(const IterImpl& other) const
     {
         // if objects are not the same, the comparison is undefined
-        check_same_container(other);
+        if (JSON_HEDLEY_UNLIKELY(m_object != other.m_object))
+        {
+            throw_literal<invalid_iterator>(exception_id::iterators_compare_different_values, "cannot compare iterators of different containers", m_object);
+        }
 
         // value-initialized forward iterators can be compared, and must compare equal to other value-initialized iterators of the same type #4493
         if (m_object == nullptr)
@@ -443,7 +466,10 @@ class iter_impl // NOLINT(cppcoreguidelines-special-member-functions,hicpp-speci
     bool operator<(const iter_impl& other) const
     {
         // if objects are not the same, the comparison is undefined
-        check_same_container(other);
+        if (JSON_HEDLEY_UNLIKELY(m_object != other.m_object))
+        {
+            throw_literal<invalid_iterator>(exception_id::iterators_compare_different_values, "cannot compare iterators of different containers", m_object);
+        }
 
         // value-initialized forward iterators can be compared, and must compare equal to other value-initialized iterators of the same type #4493
         if (m_object == nullptr)
@@ -452,18 +478,25 @@ class iter_impl // NOLINT(cppcoreguidelines-special-member-functions,hicpp-speci
             return false;
         }
 
-        if (m_object->is_object())
+        switch (m_object->m_data.m_type)
         {
-            throw_literal<invalid_iterator>(exception_id::iterator_order_on_object, "cannot compare order of object iterators", m_object);
-        }
+            case value_t::object:
+                throw_literal<invalid_iterator>(exception_id::iterator_order_on_object, "cannot compare order of object iterators", m_object);
 
-        if (m_object->is_array())
-        {
-            return (m_it.array_iterator < other.m_it.array_iterator);
-        }
+            case value_t::array:
+                return (m_it.array_iterator < other.m_it.array_iterator);
 
-        // primitive (is_primitive(), including discarded)
-        return (m_it.primitive_iterator < other.m_it.primitive_iterator);
+            case value_t::null:
+            case value_t::string:
+            case value_t::boolean:
+            case value_t::number_integer:
+            case value_t::number_unsigned:
+            case value_t::number_float:
+            case value_t::binary:
+            case value_t::discarded:
+            default:
+                return (m_it.primitive_iterator < other.m_it.primitive_iterator);
+        }
     }
 
     /*!
@@ -501,19 +534,30 @@ class iter_impl // NOLINT(cppcoreguidelines-special-member-functions,hicpp-speci
     {
         JSON_ASSERT(m_object != nullptr);
 
-        if (m_object->is_object())
+        switch (m_object->m_data.m_type)
         {
-            throw_literal<invalid_iterator>(exception_id::iterator_arithmetic_on_object, "cannot use offsets with object iterators", m_object);
-        }
+            case value_t::object:
+                throw_literal<invalid_iterator>(exception_id::iterator_arithmetic_on_object, "cannot use offsets with object iterators", m_object);
 
-        if (m_object->is_array())
-        {
-            std::advance(m_it.array_iterator, i);
-        }
-        else
-        {
-            // primitive (is_primitive(), including discarded)
-            m_it.primitive_iterator += i;
+            case value_t::array:
+            {
+                std::advance(m_it.array_iterator, i);
+                break;
+            }
+
+            case value_t::null:
+            case value_t::string:
+            case value_t::boolean:
+            case value_t::number_integer:
+            case value_t::number_unsigned:
+            case value_t::number_float:
+            case value_t::binary:
+            case value_t::discarded:
+            default:
+            {
+                m_it.primitive_iterator += i;
+                break;
+            }
         }
 
         return *this;
@@ -569,18 +613,25 @@ class iter_impl // NOLINT(cppcoreguidelines-special-member-functions,hicpp-speci
     {
         JSON_ASSERT(m_object != nullptr);
 
-        if (m_object->is_object())
+        switch (m_object->m_data.m_type)
         {
-            throw_literal<invalid_iterator>(exception_id::iterator_arithmetic_on_object, "cannot use offsets with object iterators", m_object);
-        }
+            case value_t::object:
+                throw_literal<invalid_iterator>(exception_id::iterator_arithmetic_on_object, "cannot use offsets with object iterators", m_object);
 
-        if (m_object->is_array())
-        {
-            return m_it.array_iterator - other.m_it.array_iterator;
-        }
+            case value_t::array:
+                return m_it.array_iterator - other.m_it.array_iterator;
 
-        // primitive (is_primitive(), including discarded)
-        return m_it.primitive_iterator - other.m_it.primitive_iterator;
+            case value_t::null:
+            case value_t::string:
+            case value_t::boolean:
+            case value_t::number_integer:
+            case value_t::number_unsigned:
+            case value_t::number_float:
+            case value_t::binary:
+            case value_t::discarded:
+            default:
+                return m_it.primitive_iterator - other.m_it.primitive_iterator;
+        }
     }
 
     /*!

include/nlohmann/detail/iterators/iteration_proxy.hpp

@@ -109,25 +109,35 @@ template<typename IteratorType> class iteration_proxy_value
     {
         JSON_ASSERT(anchor.m_object != nullptr);
 
-        if (anchor.m_object->is_array())
+        switch (anchor.m_object->type())
         {
             // use integer array index as key
-            if (array_index != array_index_last)
+            case value_t::array:
             {
-                int_to_string( array_index_str, array_index );
-                array_index_last = array_index;
+                if (array_index != array_index_last)
+                {
+                    int_to_string( array_index_str, array_index );
+                    array_index_last = array_index;
+                }
+                return array_index_str;
             }
-            return array_index_str;
-        }
 
-        if (anchor.m_object->is_object())
-        {
             // use key from the object
-            return anchor.key();
-        }
+            case value_t::object:
+                return anchor.key();
 
-        // use an empty key for all primitive types
-        return empty_str;
+            // use an empty key for all primitive types
+            case value_t::null:
+            case value_t::string:
+            case value_t::boolean:
+            case value_t::number_integer:
+            case value_t::number_unsigned:
+            case value_t::number_float:
+            case value_t::binary:
+            case value_t::discarded:
+            default:
+                return empty_str;
+        }
     }
 
     /// return value of the iterator

include/nlohmann/detail/json_pointer.hpp

@@ -38,102 +38,6 @@ NLOHMANN_JSON_NAMESPACE_BEGIN
 namespace detail
 {
 struct json_pointer_access;
-
-/*!
-@brief result of parse_array_index()
-
-json_pointer::array_index() maps each value to the corresponding
-parse_error/out_of_range exception; json_pointer::contains() and
-get_checked_or_null(), which must not throw for an out-of-range or
-unrepresentable index, and the json_view pointer resolution switch on it
-directly instead.
-*/
-enum class array_index_status
-{
-    ok,                ///< @a s is a valid, representable array index
-    leading_zero,      ///< @a s begins with '0' but has more than one character
-    not_a_number,      ///< @a s is neither empty nor "-" and does not begin with a digit
-    unresolved,        ///< @a s could not be converted to an integer
-    exceeds_size_type  ///< @a s converts to an integer that exceeds SizeType
-};
-
-/*!
-@brief convert a JSON Pointer reference token into an array index (RFC 6901, Sect. 4)
-
-@param[in] s    reference token to be converted into an array index; its data()
-                must be NUL-terminated
-@param[out] idx the integer representation of @a s if array_index_status::ok
-                is returned; left unchanged otherwise
-
-@return whether @a s is a valid array index, and if not, why
-
-@note this function never throws; json_pointer::array_index(), the callers that
-      must not throw, and the json_view pointer resolution build on it instead
-      of each re-implementing the RFC 6901 digit rules and the range check
-@note force-inlined: as a free function template it is no longer inlined into
-      the pointer resolution, which makes JsonPointerAt up to 17% slower (#5891)
-*/
-template<typename SizeType, typename StringType>
-JSON_HEDLEY_ALWAYS_INLINE array_index_status parse_array_index(const StringType& s, SizeType& idx) noexcept
-{
-    // error condition (cf. RFC 6901, Sect. 4)
-    if (JSON_HEDLEY_UNLIKELY(s.size() > 1 && s[0] == '0'))
-    {
-        return array_index_status::leading_zero;
-    }
-
-    // error condition (cf. RFC 6901, Sect. 4); this also covers single-
-    // character tokens, so "/x" and "/xy" fail alike; "-" and the empty
-    // token are left to the conversion below and are reported as unresolved
-    if (JSON_HEDLEY_UNLIKELY(!s.empty() && !(s.size() == 1 && s[0] == '-') && !(s[0] >= '0' && s[0] <= '9')))
-    {
-        return array_index_status::not_a_number;
-    }
-
-    // fast path: up to 18 decimal digits cannot overflow unsigned long long or
-    // SizeType (9 digits for a 32-bit SizeType), so strtoull and errno are not needed
-    constexpr std::size_t max_fast_digits = sizeof(SizeType) >= 8 ? 18 : 9;
-    if (!s.empty() && s.size() <= max_fast_digits)
-    {
-        SizeType acc = 0;
-        for (std::size_t i = 0; i < s.size(); ++i)
-        {
-            const char c = s[i];
-            if (JSON_HEDLEY_UNLIKELY(c < '0' || c > '9'))
-            {
-                break; // not a plain number: let strtoull decide
-            }
-            acc = static_cast<SizeType>((acc * 10) + static_cast<SizeType>(c - '0'));
-            if (JSON_HEDLEY_UNLIKELY(i == s.size() - 1))
-            {
-                idx = acc;
-                return array_index_status::ok;
-            }
-        }
-    }
-
-    const char* p = s.data();
-    char* p_end = nullptr; // NOLINT(misc-const-correctness)
-    errno = 0; // strtoull doesn't reset errno
-    const unsigned long long res = std::strtoull(p, &p_end, 10); // NOLINT(runtime/int)
-    if (p == p_end // invalid input or empty string
-            || errno == ERANGE // out of range
-            || JSON_HEDLEY_UNLIKELY(static_cast<std::size_t>(p_end - p) != s.size())) // incomplete read
-    {
-        return array_index_status::unresolved;
-    }
-
-    // the index does not fit into SizeType; on 64-bit platforms this is
-    // only SIZE_MAX itself (see #2203 and #5395)
-    if (res >= static_cast<unsigned long long>((std::numeric_limits<SizeType>::max)()))  // NOLINT(runtime/int)
-    {
-        return array_index_status::exceeds_size_type;
-    }
-
-    idx = static_cast<SizeType>(res);
-    return array_index_status::ok;
-}
-
 }  // namespace detail
 
 /// @brief JSON Pointer defines a string syntax for identifying a specific value within a JSON document
@@ -398,28 +302,6 @@ class json_pointer
         JSON_THROW(detail::out_of_range::create(detail::exception_id::patch_on_root, "JSON pointer has no parent", nullptr));
     }
 
-    /// whether the '~' at @a pos of @a s is not followed by '0' or '1'
-    JSON_HEDLEY_ALWAYS_INLINE static bool is_invalid_escape(const string_t& s, std::size_t pos)
-    {
-        return pos == s.size() - 1 || (s[pos + 1] != '0' && s[pos + 1] != '1');
-    }
-
-    /// @throw parse_error.107 always
-    JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE static void throw_missing_slash(const string_t& pointer)
-    {
-        static_cast<void>(pointer); // unused when JSON_NOEXCEPTION is defined
-        JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_missing_slash, 1, detail::concat("JSON pointer must be empty or begin with '/' - was: '", pointer, "'"), nullptr));
-    }
-
-    /// @throw parse_error.107 if a nonempty @a pointer does not begin with '/'
-    JSON_HEDLEY_ALWAYS_INLINE static void check_begins_with_slash(const string_t& pointer)
-    {
-        if (JSON_HEDLEY_UNLIKELY(pointer[0] != '/'))
-        {
-            throw_missing_slash(pointer);
-        }
-    }
-
     /// @throw out_of_range.404 always
     template<typename BasicJsonContext>
     JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE static void throw_unresolved(const string_t& reference_token, BasicJsonContext context)
@@ -429,7 +311,21 @@ class json_pointer
         JSON_THROW(detail::out_of_range::create(detail::exception_id::pointer_unresolved, detail::concat("unresolved reference token '", reference_token, "'"), context));
     }
 
-    using array_index_status = detail::array_index_status;
+    /*!
+    @brief result of @ref parse_array_index
+
+    @ref array_index maps each value to the corresponding parse_error/out_of_range
+    exception; @ref contains and @ref get_checked_or_null, which must not throw for
+    an out-of-range or unrepresentable index, switch on it directly instead.
+    */
+    enum class array_index_status
+    {
+        ok,                ///< @a s is a valid, representable array index
+        leading_zero,      ///< @a s begins with '0' but has more than one character
+        not_a_number,      ///< @a s is neither empty nor "-" and does not begin with a digit
+        unresolved,        ///< @a s could not be converted to an integer
+        exceeds_size_type  ///< @a s converts to an integer that exceeds size_type
+    };
 
     /// whether a reference token is "-" (a plain comparison with a string literal is not always inlined)
     static bool is_dash(const string_t& s) noexcept
@@ -437,11 +333,78 @@ class json_pointer
         return s.size() == 1 && s[0] == '-';
     }
 
-    /// detail::parse_array_index() for the size_type of @a BasicJsonType
+    /*!
+    @param[in] s    reference token to be converted into an array index
+    @param[out] idx the integer representation of @a s if @ref array_index_status::ok
+                     is returned; left unchanged otherwise
+
+    @return whether @a s is a valid array index, and if not, why
+
+    @note this function never throws; @ref array_index and the callers that must not
+          throw (@ref contains, @ref get_checked_or_null) build on it instead of each
+          re-implementing the RFC 6901 digit rules and the @a size_type range check
+    */
     template<typename BasicJsonType>
     static array_index_status parse_array_index(const string_t& s, typename BasicJsonType::size_type& idx) noexcept
     {
-        return detail::parse_array_index(s, idx);
+        using size_type = typename BasicJsonType::size_type;
+
+        // error condition (cf. RFC 6901, Sect. 4)
+        if (JSON_HEDLEY_UNLIKELY(s.size() > 1 && s[0] == '0'))
+        {
+            return array_index_status::leading_zero;
+        }
+
+        // error condition (cf. RFC 6901, Sect. 4); this also covers single-
+        // character tokens, so "/x" and "/xy" fail alike; "-" and the empty
+        // token are left to the conversion below and are reported as unresolved
+        if (JSON_HEDLEY_UNLIKELY(!s.empty() && !is_dash(s) && !(s[0] >= '0' && s[0] <= '9')))
+        {
+            return array_index_status::not_a_number;
+        }
+
+        // fast path: up to 18 decimal digits cannot overflow unsigned long long or
+        // size_type (9 digits for a 32-bit size_type), so strtoull and errno are not needed
+        constexpr std::size_t max_fast_digits = sizeof(size_type) >= 8 ? 18 : 9;
+        if (!s.empty() && s.size() <= max_fast_digits)
+        {
+            size_type acc = 0;
+            for (std::size_t i = 0; i < s.size(); ++i)
+            {
+                const char c = s[i];
+                if (JSON_HEDLEY_UNLIKELY(c < '0' || c > '9'))
+                {
+                    break; // not a plain number: let strtoull decide
+                }
+                acc = static_cast<size_type>((acc * 10) + static_cast<size_type>(c - '0'));
+                if (JSON_HEDLEY_UNLIKELY(i == s.size() - 1))
+                {
+                    idx = acc;
+                    return array_index_status::ok;
+                }
+            }
+        }
+
+        const char* p = s.data();
+        char* p_end = nullptr; // NOLINT(misc-const-correctness)
+        errno = 0; // strtoull doesn't reset errno
+        const unsigned long long res = std::strtoull(p, &p_end, 10); // NOLINT(runtime/int)
+        if (p == p_end // invalid input or empty string
+                || errno == ERANGE // out of range
+                || JSON_HEDLEY_UNLIKELY(static_cast<std::size_t>(p_end - p) != s.size())) // incomplete read
+        {
+            return array_index_status::unresolved;
+        }
+
+        // the index does not fit into size_type; on 64-bit platforms this is
+        // only SIZE_MAX itself (see #2203 and #5395)
+        if (res >= static_cast<unsigned long long>((std::numeric_limits<size_type>::max)()))  // NOLINT(runtime/int)
+        {
+            return array_index_status::exceeds_size_type;
+        }
+
+        idx = static_cast<size_type>(res);
+        return array_index_status::ok;
     }
 
     /*!
@@ -491,7 +454,10 @@ class json_pointer
             return reference_tokens.empty();
         }
 
-        check_begins_with_slash(reference_string);
+        if (JSON_HEDLEY_UNLIKELY(reference_string[0] != '/'))
+        {
+            JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_missing_slash, 1, detail::concat("JSON pointer must be empty or begin with '/' - was: '", reference_string, "'"), nullptr));
+        }
 
         // the whole string is validated, whether or not it matches
         const std::size_t size = reference_string.size();
@@ -499,7 +465,7 @@ class json_pointer
                 pos != string_t::npos;
                 pos = reference_string.find_first_of('~', pos + 1))
         {
-            if (JSON_HEDLEY_UNLIKELY(is_invalid_escape(reference_string, pos)))
+            if (JSON_HEDLEY_UNLIKELY(pos == size - 1 || (reference_string[pos + 1] != '0' && reference_string[pos + 1] != '1')))
             {
                 JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_invalid_escape, 0, "escape character '~' must be followed with '0' or '1'", nullptr));
             }
@@ -777,7 +743,10 @@ class json_pointer
         }
 
         const char* const data = key.data();
-        check_begins_with_slash(key);
+        if (JSON_HEDLEY_UNLIKELY(data[0] != '/'))
+        {
+            JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_missing_slash, 1, detail::concat("JSON pointer must be empty or begin with '/' - was: '", key, "'"), nullptr));
+        }
 
         // The beginning of the key that it has in common with the previous
         // one, which was checked already, is skipped: the scan starts with
@@ -810,7 +779,7 @@ class json_pointer
             else if (data[i] == '~')
             {
                 // ~ must be followed by 0 or 1
-                if (JSON_HEDLEY_UNLIKELY(is_invalid_escape(key, i)))
+                if (JSON_HEDLEY_UNLIKELY(i + 1 == size || (data[i + 1] != '0' && data[i + 1] != '1')))
                 {
                     JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_invalid_escape, 0, "escape character '~' must be followed with '0' or '1'", nullptr));
                 }
@@ -909,46 +878,58 @@ class json_pointer
                 ++end;
             }
 
-            if (ptr->is_object())
+            switch (ptr->type())
             {
-                token.clear();
-                token.append(begin, static_cast<std::size_t>(end - begin));
-                const auto it = ptr->m_data.m_value.object->find(token);
-                if (JSON_HEDLEY_UNLIKELY(it == ptr->m_data.m_value.object->end()))
+                case detail::value_t::object:
                 {
-                    return nullptr;
-                }
-                ptr = &it->second;
-            }
-            else if (ptr->is_array())
-            {
-                // an index of up to 18 digits without a leading zero fits into size_type
-                // (9 digits for a 32-bit size_type); everything else is left to json_pointer
-                constexpr std::size_t max_digits = sizeof(typename BasicJsonType::size_type) >= 8 ? 18 : 9;
-                const auto digits = static_cast<std::size_t>(end - begin);
-                if (JSON_HEDLEY_UNLIKELY(digits == 0 || digits > max_digits || (digits > 1 && *begin == '0')))
-                {
-                    return nullptr;
-                }
-                typename BasicJsonType::size_type idx = 0;
-                for (const char* c = begin; c != end; ++c)
-                {
-                    if (JSON_HEDLEY_UNLIKELY(*c < '0' || *c > '9'))
+                    token.clear();
+                    token.append(begin, static_cast<std::size_t>(end - begin));
+                    const auto it = ptr->m_data.m_value.object->find(token);
+                    if (JSON_HEDLEY_UNLIKELY(it == ptr->m_data.m_value.object->end()))
                     {
                         return nullptr;
                     }
-                    idx = static_cast<typename BasicJsonType::size_type>((idx * 10) + static_cast<typename BasicJsonType::size_type>(*c - '0'));
+                    ptr = &it->second;
+                    break;
                 }
-                if (JSON_HEDLEY_UNLIKELY(idx >= ptr->m_data.m_value.array->size()))
+
+                case detail::value_t::array:
                 {
-                    return nullptr;
+                    // an index of up to 18 digits without a leading zero fits into size_type
+                    // (9 digits for a 32-bit size_type); everything else is left to json_pointer
+                    constexpr std::size_t max_digits = sizeof(typename BasicJsonType::size_type) >= 8 ? 18 : 9;
+                    const auto digits = static_cast<std::size_t>(end - begin);
+                    if (JSON_HEDLEY_UNLIKELY(digits == 0 || digits > max_digits || (digits > 1 && *begin == '0')))
+                    {
+                        return nullptr;
+                    }
+                    typename BasicJsonType::size_type idx = 0;
+                    for (const char* c = begin; c != end; ++c)
+                    {
+                        if (JSON_HEDLEY_UNLIKELY(*c < '0' || *c > '9'))
+                        {
+                            return nullptr;
+                        }
+                        idx = static_cast<typename BasicJsonType::size_type>((idx * 10) + static_cast<typename BasicJsonType::size_type>(*c - '0'));
+                    }
+                    if (JSON_HEDLEY_UNLIKELY(idx >= ptr->m_data.m_value.array->size()))
+                    {
+                        return nullptr;
+                    }
+                    ptr = &(*ptr->m_data.m_value.array)[idx];
+                    break;
                 }
-                ptr = &(*ptr->m_data.m_value.array)[idx];
-            }
-            else
-            {
-                // primitive value with a reference token left
-                return nullptr;
+
+                case detail::value_t::null:
+                case detail::value_t::string:
+                case detail::value_t::boolean:
+                case detail::value_t::number_integer:
+                case detail::value_t::number_unsigned:
+                case detail::value_t::number_float:
+                case detail::value_t::binary:
+                case detail::value_t::discarded:
+                default:
+                    return nullptr;
             }
             pos = static_cast<std::size_t>(end - path.data());
             steps.emplace_back(pos, ptr);
@@ -998,28 +979,40 @@ class json_pointer
                        : detail::value_t::object;
             }
 
-            if (ptr->is_object())
+            switch (ptr->type())
             {
-                // use unchecked object access
-                ptr = &object_member(*ptr, reference_token);
-            }
-            else if (ptr->is_array())
-            {
-                if (reference_token == "-")
+                case detail::value_t::object:
                 {
-                    // explicitly treat "-" as index beyond the end
-                    ptr = &ptr->operator[](ptr->m_data.m_value.array->size());
+                    // use unchecked object access
+                    ptr = &object_member(*ptr, reference_token);
+                    break;
                 }
-                else
+
+                case detail::value_t::array:
                 {
-                    // convert array index to number; unchecked access
-                    ptr = &ptr->operator[](array_index<BasicJsonType>(reference_token));
+                    if (reference_token == "-")
+                    {
+                        // explicitly treat "-" as index beyond the end
+                        ptr = &ptr->operator[](ptr->m_data.m_value.array->size());
+                    }
+                    else
+                    {
+                        // convert array index to number; unchecked access
+                        ptr = &ptr->operator[](array_index<BasicJsonType>(reference_token));
+                    }
+                    break;
                 }
-            }
-            else
-            {
-                // primitive value with a reference token left
-                throw_unresolved(reference_token, ptr);
+
+                case detail::value_t::null:
+                case detail::value_t::string:
+                case detail::value_t::boolean:
+                case detail::value_t::number_integer:
+                case detail::value_t::number_unsigned:
+                case detail::value_t::number_float:
+                case detail::value_t::binary:
+                case detail::value_t::discarded:
+                default:
+                    throw_unresolved(reference_token, ptr);
             }
         }
 
@@ -1037,39 +1030,51 @@ class json_pointer
     {
         for (const auto& reference_token : reference_tokens)
         {
-            if (ptr->is_object())
+            switch (ptr->type())
             {
-                const auto it = ptr->m_data.m_value.object->find(reference_token);
-                if (JSON_HEDLEY_UNLIKELY(it == ptr->m_data.m_value.object->end()))
+                case detail::value_t::object:
                 {
-                    // at() throws out_of_range.403 with the usual message
-                    ptr->at(reference_token);
-                }
-                ptr = &it->second;
-            }
-            else if (ptr->is_array())
-            {
-                if (JSON_HEDLEY_UNLIKELY(is_dash(reference_token)))
-                {
-                    // "-" always fails the range check
-                    JSON_THROW(detail::out_of_range::create(detail::exception_id::pointer_past_the_end_index, detail::concat(
-                            "array index '-' (", std::to_string(ptr->m_data.m_value.array->size()),
-                            ") is out of range"), ptr));
+                    const auto it = ptr->m_data.m_value.object->find(reference_token);
+                    if (JSON_HEDLEY_UNLIKELY(it == ptr->m_data.m_value.object->end()))
+                    {
+                        // at() throws out_of_range.403 with the usual message
+                        ptr->at(reference_token);
+                    }
+                    ptr = &it->second;
+                    break;
                 }
 
-                const auto idx = array_index<BasicJsonType>(reference_token);
-                // Bounds check before access to avoid exception with JSON_NOEXCEPTION
-                if (JSON_HEDLEY_UNLIKELY(idx >= ptr->m_data.m_value.array->size()))
+                case detail::value_t::array:
                 {
-                    JSON_THROW(detail::out_of_range::create(detail::exception_id::array_index_out_of_range, detail::concat(
-                            "array index ", std::to_string(idx), " is out of range"), ptr));
+                    if (JSON_HEDLEY_UNLIKELY(is_dash(reference_token)))
+                    {
+                        // "-" always fails the range check
+                        JSON_THROW(detail::out_of_range::create(detail::exception_id::pointer_past_the_end_index, detail::concat(
+                                "array index '-' (", std::to_string(ptr->m_data.m_value.array->size()),
+                                ") is out of range"), ptr));
+                    }
+
+                    const auto idx = array_index<BasicJsonType>(reference_token);
+                    // Bounds check before access to avoid exception with JSON_NOEXCEPTION
+                    if (JSON_HEDLEY_UNLIKELY(idx >= ptr->m_data.m_value.array->size()))
+                    {
+                        JSON_THROW(detail::out_of_range::create(detail::exception_id::array_index_out_of_range, detail::concat(
+                                "array index ", std::to_string(idx), " is out of range"), ptr));
+                    }
+                    ptr = &(*ptr->m_data.m_value.array)[idx];
+                    break;
                 }
-                ptr = &(*ptr->m_data.m_value.array)[idx];
-            }
-            else
-            {
-                // primitive value with a reference token left
-                throw_unresolved(reference_token, ptr);
+
+                case detail::value_t::null:
+                case detail::value_t::string:
+                case detail::value_t::boolean:
+                case detail::value_t::number_integer:
+                case detail::value_t::number_unsigned:
+                case detail::value_t::number_float:
+                case detail::value_t::binary:
+                case detail::value_t::discarded:
+                default:
+                    throw_unresolved(reference_token, ptr);
             }
         }
 
@@ -1098,32 +1103,44 @@ class json_pointer
     {
         for (const auto& reference_token : reference_tokens)
         {
-            if (ptr->is_object())
+            switch (ptr->type())
             {
-                // use unchecked object access; the const operator[]
-                // asserts that the key exists
-                const auto it = ptr->m_data.m_value.object->find(reference_token);
-                JSON_ASSERT(it != ptr->m_data.m_value.object->end());
-                ptr = &it->second;
-            }
-            else if (ptr->is_array())
-            {
-                if (JSON_HEDLEY_UNLIKELY(is_dash(reference_token)))
+                case detail::value_t::object:
                 {
-                    // "-" cannot be used for const access
-                    JSON_THROW(detail::out_of_range::create(detail::exception_id::pointer_past_the_end_index, detail::concat("array index '-' (", std::to_string(ptr->m_data.m_value.array->size()), ") is out of range"), ptr));
+                    // use unchecked object access; the const operator[]
+                    // asserts that the key exists
+                    const auto it = ptr->m_data.m_value.object->find(reference_token);
+                    JSON_ASSERT(it != ptr->m_data.m_value.object->end());
+                    ptr = &it->second;
+                    break;
                 }
 
-                // use unchecked array access; the const operator[]
-                // asserts that the index exists
-                const auto idx = array_index<BasicJsonType>(reference_token);
-                JSON_ASSERT(idx < ptr->m_data.m_value.array->size());
-                ptr = &(*ptr->m_data.m_value.array)[idx];
-            }
-            else
-            {
-                // primitive value with a reference token left
-                throw_unresolved(reference_token, ptr);
+                case detail::value_t::array:
+                {
+                    if (JSON_HEDLEY_UNLIKELY(is_dash(reference_token)))
+                    {
+                        // "-" cannot be used for const access
+                        JSON_THROW(detail::out_of_range::create(detail::exception_id::pointer_past_the_end_index, detail::concat("array index '-' (", std::to_string(ptr->m_data.m_value.array->size()), ") is out of range"), ptr));
+                    }
+
+                    // use unchecked array access; the const operator[]
+                    // asserts that the index exists
+                    const auto idx = array_index<BasicJsonType>(reference_token);
+                    JSON_ASSERT(idx < ptr->m_data.m_value.array->size());
+                    ptr = &(*ptr->m_data.m_value.array)[idx];
+                    break;
+                }
+
+                case detail::value_t::null:
+                case detail::value_t::string:
+                case detail::value_t::boolean:
+                case detail::value_t::number_integer:
+                case detail::value_t::number_unsigned:
+                case detail::value_t::number_float:
+                case detail::value_t::binary:
+                case detail::value_t::discarded:
+                default:
+                    throw_unresolved(reference_token, ptr);
             }
         }
 
@@ -1147,54 +1164,66 @@ class json_pointer
     {
         for (const auto& reference_token : reference_tokens)
         {
-            if (ptr->is_object())
+            switch (ptr->type())
             {
-                const auto it = ptr->m_data.m_value.object->find(reference_token);
-                if (JSON_HEDLEY_UNLIKELY(it == ptr->m_data.m_value.object->end()))
+                case detail::value_t::object:
                 {
-                    return nullptr;
-                }
-                ptr = &it->second;
-            }
-            else if (ptr->is_array())
-            {
-                if (JSON_HEDLEY_UNLIKELY(is_dash(reference_token)))
-                {
-                    // "-" always fails the range check
-                    return nullptr;
-                }
-
-                // a malformed index throws parse_error.106/109; an
-                // index that is syntactically valid but cannot be
-                // represented (out_of_range.404/410) is treated like an
-                // out-of-range index below
-                typename BasicJsonType::size_type idx{};
-                switch (parse_array_index<BasicJsonType>(reference_token, idx))
-                {
-                    // the branches differ in their messages, not after JSON_THROW's expansion
-                    // NOLINTNEXTLINE(bugprone-branch-clone)
-                    case array_index_status::leading_zero:
-                        JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_index_leading_zero, 0, detail::concat("array index '", reference_token, "' must not begin with '0'"), nullptr));
-                    case array_index_status::not_a_number:
-                        JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_index_not_a_number, 0, detail::concat("array index '", reference_token, "' is not a number"), nullptr));
-                    case array_index_status::unresolved:
-                    case array_index_status::exceeds_size_type:
+                    const auto it = ptr->m_data.m_value.object->find(reference_token);
+                    if (JSON_HEDLEY_UNLIKELY(it == ptr->m_data.m_value.object->end()))
+                    {
                         return nullptr;
-                    case array_index_status::ok:
-                    default:
-                        break;
+                    }
+                    ptr = &it->second;
+                    break;
                 }
 
-                if (JSON_HEDLEY_UNLIKELY(idx >= ptr->m_data.m_value.array->size()))
+                case detail::value_t::array:
                 {
-                    return nullptr;
+                    if (JSON_HEDLEY_UNLIKELY(is_dash(reference_token)))
+                    {
+                        // "-" always fails the range check
+                        return nullptr;
+                    }
+
+                    // a malformed index throws parse_error.106/109; an
+                    // index that is syntactically valid but cannot be
+                    // represented (out_of_range.404/410) is treated like an
+                    // out-of-range index below
+                    typename BasicJsonType::size_type idx{};
+                    switch (parse_array_index<BasicJsonType>(reference_token, idx))
+                    {
+                        // the branches differ in their messages, not after JSON_THROW's expansion
+                        // NOLINTNEXTLINE(bugprone-branch-clone)
+                        case array_index_status::leading_zero:
+                            JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_index_leading_zero, 0, detail::concat("array index '", reference_token, "' must not begin with '0'"), nullptr));
+                        case array_index_status::not_a_number:
+                            JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_index_not_a_number, 0, detail::concat("array index '", reference_token, "' is not a number"), nullptr));
+                        case array_index_status::unresolved:
+                        case array_index_status::exceeds_size_type:
+                            return nullptr;
+                        case array_index_status::ok:
+                        default:
+                            break;
+                    }
+
+                    if (JSON_HEDLEY_UNLIKELY(idx >= ptr->m_data.m_value.array->size()))
+                    {
+                        return nullptr;
+                    }
+                    ptr = &(*ptr->m_data.m_value.array)[idx];
+                    break;
                 }
-                ptr = &(*ptr->m_data.m_value.array)[idx];
-            }
-            else
-            {
-                // primitive value with a reference token left
-                return nullptr;
+
+                case detail::value_t::null:
+                case detail::value_t::string:
+                case detail::value_t::boolean:
+                case detail::value_t::number_integer:
+                case detail::value_t::number_unsigned:
+                case detail::value_t::number_float:
+                case detail::value_t::binary:
+                case detail::value_t::discarded:
+                default:
+                    return nullptr;
             }
         }
 
@@ -1210,48 +1239,63 @@ class json_pointer
     {
         for (const auto& reference_token : reference_tokens)
         {
-            if (ptr->is_object())
+            switch (ptr->type())
             {
-                const auto it = ptr->m_data.m_value.object->find(reference_token);
-                if (it == ptr->m_data.m_value.object->end())
+                case detail::value_t::object:
                 {
-                    // we did not find the key in the object
-                    return false;
+                    const auto it = ptr->m_data.m_value.object->find(reference_token);
+                    if (it == ptr->m_data.m_value.object->end())
+                    {
+                        // we did not find the key in the object
+                        return false;
+                    }
+
+                    ptr = &it->second;
+                    break;
                 }
 
-                ptr = &it->second;
-            }
-            else if (ptr->is_array())
-            {
-                if (JSON_HEDLEY_UNLIKELY(is_dash(reference_token)))
+                case detail::value_t::array:
                 {
-                    // "-" always fails the range check
-                    return false;
+                    if (JSON_HEDLEY_UNLIKELY(is_dash(reference_token)))
+                    {
+                        // "-" always fails the range check
+                        return false;
+                    }
+
+                    // any parse failure (malformed index, or one that is syntactically
+                    // valid but not representable as size_type) means the reference
+                    // token cannot denote an existing array element -- contains() must
+                    // not throw (see #5395), so it is treated as "not found"
+                    typename BasicJsonType::size_type idx{};
+                    if (JSON_HEDLEY_UNLIKELY(parse_array_index<BasicJsonType>(reference_token, idx) != array_index_status::ok))
+                    {
+                        return false;
+                    }
+
+                    if (idx >= ptr->m_data.m_value.array->size())
+                    {
+                        // index out of range
+                        return false;
+                    }
+
+                    ptr = &(*ptr->m_data.m_value.array)[idx];
+                    break;
                 }
 
-                // any parse failure (malformed index, or one that is syntactically
-                // valid but not representable as size_type) means the reference
-                // token cannot denote an existing array element -- contains() must
-                // not throw (see #5395), so it is treated as "not found"
-                typename BasicJsonType::size_type idx{};
-                if (JSON_HEDLEY_UNLIKELY(parse_array_index<BasicJsonType>(reference_token, idx) != array_index_status::ok))
+                case detail::value_t::null:
+                case detail::value_t::string:
+                case detail::value_t::boolean:
+                case detail::value_t::number_integer:
+                case detail::value_t::number_unsigned:
+                case detail::value_t::number_float:
+                case detail::value_t::binary:
+                case detail::value_t::discarded:
+                default:
                 {
+                    // we do not expect primitive values if there is still a
+                    // reference token to process
                     return false;
                 }
-
-                if (idx >= ptr->m_data.m_value.array->size())
-                {
-                    // index out of range
-                    return false;
-                }
-
-                ptr = &(*ptr->m_data.m_value.array)[idx];
-            }
-            else
-            {
-                // we do not expect primitive values if there is still a
-                // reference token to process
-                return false;
             }
         }
 
@@ -1279,7 +1323,10 @@ class json_pointer
         }
 
         // check if a nonempty reference string begins with slash
-        check_begins_with_slash(reference_string);
+        if (JSON_HEDLEY_UNLIKELY(reference_string[0] != '/'))
+        {
+            JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_missing_slash, 1, detail::concat("JSON pointer must be empty or begin with '/' - was: '", reference_string, "'"), nullptr));
+        }
 
         // every slash starts a reference token
         const std::size_t size = reference_string.size();
@@ -1320,7 +1367,9 @@ class json_pointer
                     JSON_ASSERT(reference_token[pos] == '~');
 
                     // ~ must be followed by 0 or 1
-                    if (JSON_HEDLEY_UNLIKELY(is_invalid_escape(reference_token, pos)))
+                    if (JSON_HEDLEY_UNLIKELY(pos == reference_token.size() - 1 ||
+                                             (reference_token[pos + 1] != '0' &&
+                                              reference_token[pos + 1] != '1')))
                     {
                         JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_invalid_escape, 0, "escape character '~' must be followed with '0' or '1'", nullptr));
                     }
@@ -1477,34 +1526,50 @@ class json_pointer
         auto hint = object.end();
         const auto enter = [&stack, &path, &object, &hint](const BasicJsonType & v)
         {
-            if (v.is_array())
+            switch (v.type())
             {
-                if (v.m_data.m_value.array->empty())
+                case detail::value_t::array:
                 {
-                    // flatten empty array as null
-                    add_flattened(object, hint, path, nullptr, detail::priority_tag<2> {});
+                    if (v.m_data.m_value.array->empty())
+                    {
+                        // flatten empty array as null
+                        add_flattened(object, hint, path, nullptr, detail::priority_tag<2> {});
+                    }
+                    else
+                    {
+                        stack.emplace_back(&v, object_const_iterator(), path.size());
+                    }
+                    return;
                 }
-                else
+
+                case detail::value_t::object:
                 {
-                    stack.emplace_back(&v, object_const_iterator(), path.size());
+                    if (v.m_data.m_value.object->empty())
+                    {
+                        // flatten empty object as null
+                        add_flattened(object, hint, path, nullptr, detail::priority_tag<2> {});
+                    }
+                    else
+                    {
+                        stack.emplace_back(&v, v.m_data.m_value.object->begin(), path.size());
+                    }
+                    return;
                 }
-            }
-            else if (v.is_object())
-            {
-                if (v.m_data.m_value.object->empty())
+
+                case detail::value_t::null:
+                case detail::value_t::string:
+                case detail::value_t::boolean:
+                case detail::value_t::number_integer:
+                case detail::value_t::number_unsigned:
+                case detail::value_t::number_float:
+                case detail::value_t::binary:
+                case detail::value_t::discarded:
+                default:
                 {
-                    // flatten empty object as null
-                    add_flattened(object, hint, path, nullptr, detail::priority_tag<2> {});
+                    // add a primitive value with its reference string
+                    add_flattened(object, hint, path, v, detail::priority_tag<2> {});
+                    return;
                 }
-                else
-                {
-                    stack.emplace_back(&v, v.m_data.m_value.object->begin(), path.size());
-                }
-            }
-            else
-            {
-                // add a primitive value with its reference string
-                add_flattened(object, hint, path, v, detail::priority_tag<2> {});
             }
         };
 

include/nlohmann/detail/macro_scope.hpp

@@ -215,6 +215,17 @@
     #define JSON_NO_UNIQUE_ADDRESS
 #endif
 
+// Inlines small functions even in unoptimized builds, so that they are not
+// emitted. The parsers for parse(), accept(), and from_*() use it for the
+// functions that stand in for the code recovering from errors (see #3989).
+// MSVC is left to decide, as it warns (C4714) where it does not inline a
+// __forceinline function.
+#if defined(_MSC_VER) && !defined(__clang__)
+    #define JSON_INTERNAL_ALWAYS_INLINE
+#else
+    #define JSON_INTERNAL_ALWAYS_INLINE JSON_HEDLEY_ALWAYS_INLINE
+#endif
+
 // Clang targeting MinGW does not survive the thread_local storage the copy
 // constructor uses to bound its descent: every test that copies a value
 // segfaults with clang 11.0.1 and clang 18.1.8, while the same tests pass with

include/nlohmann/detail/macro_unscope.hpp

@@ -19,6 +19,7 @@
 #undef NLOHMANN_CAN_CALL_STD_FUNC_IMPL
 #undef JSON_INLINE_VARIABLE
 #undef JSON_NO_UNIQUE_ADDRESS
+#undef JSON_INTERNAL_ALWAYS_INLINE
 #undef JSON_DISABLE_ENUM_SERIALIZATION
 #undef JSON_DISABLE_TUPLE_REFERENCE_CONVERSION
 #undef JSON_UNLOCKED_FILE_IO

include/nlohmann/detail/output/binary_writer.hpp

@@ -21,7 +21,6 @@
 
 #include <nlohmann/detail/bit_ops.hpp>
 #include <nlohmann/detail/input/binary_reader.hpp>
-#include <nlohmann/detail/input/input_adapters.hpp>
 #include <nlohmann/detail/input/string_scan.hpp>
 #include <nlohmann/detail/macro_scope.hpp>
 #include <nlohmann/detail/output/error_handler.hpp>
@@ -312,7 +311,9 @@ class binary_writer
                 write_cbor_head(0x40, N);
 
                 // step 2: write each element
-                write_bytes(j.m_data.m_value.binary->data(), N);
+                oa.write_characters(
+                      reinterpret_cast<const CharType*>(j.m_data.m_value.binary->data()),
+                      N);
 
                 break;
             }
@@ -325,7 +326,7 @@ class binary_writer
 
             case value_t::discarded:
             default:
-                throw_on_discarded(j, input_format_t::cbor);
+                throw_on_discarded(j, "CBOR");
         }
     }
 
@@ -525,7 +526,9 @@ class binary_writer
                 }
 
                 // step 2: write the byte string
-                write_bytes(j.m_data.m_value.binary->data(), N);
+                oa.write_characters(
+                      reinterpret_cast<const CharType*>(j.m_data.m_value.binary->data()),
+                      N);
 
                 break;
             }
@@ -539,7 +542,7 @@ class binary_writer
 
             case value_t::discarded:
             default:
-                throw_on_discarded(j, input_format_t::msgpack);
+                throw_on_discarded(j, "MessagePack");
         }
     }
 
@@ -623,7 +626,7 @@ class binary_writer
             case value_t::array:
             {
                 bool prefix_required = true;
-                const bool write_closer = write_ubjson_start_container<true>(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
+                const bool write_closer = write_ubjson_start_array(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
 
                 {
                     const auto& array = *j.m_data.m_value.array;
@@ -673,7 +676,9 @@ class binary_writer
 
                 if (use_type)
                 {
-                    write_bytes(j.m_data.m_value.binary->data(), j.m_data.m_value.binary->size());
+                    oa.write_characters(
+                          reinterpret_cast<const CharType*>(j.m_data.m_value.binary->data()),
+                          j.m_data.m_value.binary->size());
                 }
                 else
                 {
@@ -699,7 +704,7 @@ class binary_writer
                 }
 
                 bool prefix_required = true;
-                const bool write_closer = write_ubjson_start_container<false>(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
+                const bool write_closer = write_ubjson_start_object(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
 
                 write_ubjson_members(j, use_count, use_type, prefix_required, use_bjdata, bjdata_version, depth);
 
@@ -713,7 +718,7 @@ class binary_writer
 
             case value_t::discarded:
             default:
-                throw_on_discarded(j, use_bjdata ? input_format_t::bjdata : input_format_t::ubjson);
+                throw_on_discarded(j, use_bjdata ? "BJData" : "UBJSON");
         }
     }
 
@@ -739,11 +744,11 @@ class binary_writer
     @brief throws because @a j is discarded and cannot be serialized
     @throw type_error.321 always
     */
-    JSON_HEDLEY_NO_RETURN static void throw_on_discarded(const BasicJsonType& j, const input_format_t format)
+    JSON_HEDLEY_NO_RETURN static void throw_on_discarded(const BasicJsonType& j, const char* format_name)
     {
         static_cast<void>(j); // unused when JSON_NOEXCEPTION is defined
-        static_cast<void>(format);
-        JSON_THROW(type_error::create(exception_id::discarded_value_used, concat("cannot serialize discarded value to ", input_format_name(format)), &j));
+        static_cast<void>(format_name);
+        JSON_THROW(type_error::create(exception_id::discarded_value_used, concat("cannot serialize discarded value to ", format_name), &j));
     }
 
     /// how the range of integer type @a NumberType relates to that of @a TargetType
@@ -863,11 +868,11 @@ class binary_writer
     }
 
     /*!
-    @brief throws because the integer @a j is too large for @a format
+    @brief throws because the integer @a j is too large for @a format_name
     @throw out_of_range.407 always
     */
     JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE
-    static void throw_integer_out_of_range(const BasicJsonType& j, const input_format_t format, const char* range)
+    static void throw_integer_out_of_range(const BasicJsonType& j, const char* format_name, const char* range)
     {
         // neither std::to_string(), which has no overload for integer types
         // wider than 64 bits, nor dump(), which would also instantiate the
@@ -878,9 +883,9 @@ class binary_writer
                                    ? integer_to_string(j.m_data.m_value.number_unsigned, std::integral_constant<bool, unsigned_limits::is_signed>())
                                    : integer_to_string(j.m_data.m_value.number_integer, std::integral_constant<bool, integer_limits::is_signed>());
         static_cast<void>(number); // unused when JSON_NOEXCEPTION is defined
-        static_cast<void>(format);
+        static_cast<void>(format_name);
         static_cast<void>(range);
-        JSON_THROW(out_of_range::create(exception_id::integer_too_large, concat("integer number ", number, " cannot be represented by ", input_format_name(format), " as it does not fit ", range), &j));
+        JSON_THROW(out_of_range::create(exception_id::integer_too_large, concat("integer number ", number, " cannot be represented by ", format_name, " as it does not fit ", range), &j));
     }
 
     // The exceptions below are thrown from functions of their own, out of line:
@@ -923,56 +928,53 @@ class binary_writer
         JSON_THROW(other_error::create(exception_id::size_marker_required, "use_type requires use_size = true", &j));
     }
 
-    /// @brief write @a size bytes starting at @a data to the output adapter
-    JSON_HEDLEY_ALWAYS_INLINE
-    void write_bytes(const void* data, const std::size_t size)
-    {
-        oa.write_characters(static_cast<const CharType*>(data), size);
-    }
-
-    /// @brief write the MessagePack prefix of an array or object with @a N
-    ///        elements: the fix marker @a fix_marker | n, or @a marker_16 or
-    ///        @a marker_32 followed by the length
-    JSON_HEDLEY_ALWAYS_INLINE
-    void write_msgpack_container_prefix(const std::size_t N, const BasicJsonType& j,
-                                        const std::uint8_t fix_marker, const std::uint8_t marker_16, const std::uint8_t marker_32)
+    void write_msgpack_array_prefix(const std::size_t N, const BasicJsonType& j)
     {
         const auto n = to_msgpack_length(N, j);
         if (n <= 15)
         {
-            // fixarray, fixmap
-            oa.write_character(to_char_type(static_cast<std::uint8_t>(fix_marker | n)));
+            // fixarray
+            oa.write_character(to_char_type(static_cast<std::uint8_t>(0x90 | n)));
         }
         else if (n <= (std::numeric_limits<std::uint16_t>::max)())
         {
-            // array 16, map 16
-            oa.write_character(to_char_type(marker_16));
+            // array 16
+            oa.write_character(to_char_type(0xDC));
             write_number(static_cast<std::uint16_t>(n));
         }
         else
         {
-            // array 32, map 32
-            oa.write_character(to_char_type(marker_32));
+            // array 32
+            oa.write_character(to_char_type(0xDD));
             write_number(static_cast<std::uint32_t>(n));
         }
     }
 
-    JSON_HEDLEY_ALWAYS_INLINE
-    void write_msgpack_array_prefix(const std::size_t N, const BasicJsonType& j)
-    {
-        write_msgpack_container_prefix(N, j, 0x90, 0xDC, 0xDD);
-    }
-
-    JSON_HEDLEY_ALWAYS_INLINE
     void write_msgpack_object_prefix(const std::size_t N, const BasicJsonType& j)
     {
-        write_msgpack_container_prefix(N, j, 0x80, 0xDE, 0xDF);
+        const auto n = to_msgpack_length(N, j);
+        if (n <= 15)
+        {
+            // fixmap
+            oa.write_character(to_char_type(static_cast<std::uint8_t>(0x80 | (n & 0xF))));
+        }
+        else if (n <= (std::numeric_limits<std::uint16_t>::max)())
+        {
+            // map 16
+            oa.write_character(to_char_type(0xDE));
+            write_number(static_cast<std::uint16_t>(n));
+        }
+        else
+        {
+            // map 32
+            oa.write_character(to_char_type(0xDF));
+            write_number(static_cast<std::uint32_t>(n));
+        }
     }
 
-    /// @brief a CBOR, MessagePack, BON8, or (as the base of @ref ubjson_frame)
-    ///        UBJSON array or object whose elements @ref write_container_iterative,
-    ///        @ref write_bon8_iterative, or @ref write_ubjson_iterative is still
-    ///        writing
+    /// @brief a CBOR, MessagePack, or BON8 array or object whose elements
+    ///        @ref write_cbor_iterative, @ref write_msgpack_iterative, or
+    ///        @ref write_bon8_iterative is still writing
     struct binary_container_frame
     {
         explicit binary_container_frame(const BasicJsonType* value_) noexcept
@@ -1008,82 +1010,19 @@ class binary_writer
         typename BasicJsonType::array_t::const_iterator array_it{};
     };
 
-    /// @brief the CBOR specifics of @ref write_container_iterative
-    struct cbor_walk
-    {
-        static void array_head(binary_writer& w, const BasicJsonType& j)
-        {
-            w.write_cbor_head(0x80, j.m_data.m_value.array->size());
-        }
-
-        static void object_head(binary_writer& w, const BasicJsonType& j)
-        {
-            w.write_cbor_head(0xA0, j.m_data.m_value.object->size());
-        }
-
-        static void scalar(binary_writer& w, const BasicJsonType& j)
-        {
-            w.write_cbor(j);
-        }
-
-        static void string(binary_writer& w, const string_t& key, const BasicJsonType& context)
-        {
-            w.write_cbor_string(key, context);
-        }
-
-        template<typename KeyType>
-        static void key(binary_writer& w, const KeyType& k, const BasicJsonType& object)
-        {
-            w.write_cbor_key(k, object);
-        }
-    };
-
-    /// @brief the MessagePack specifics of @ref write_container_iterative
-    struct msgpack_walk
-    {
-        static void array_head(binary_writer& w, const BasicJsonType& j)
-        {
-            w.write_msgpack_array_prefix(j.m_data.m_value.array->size(), j);
-        }
-
-        static void object_head(binary_writer& w, const BasicJsonType& j)
-        {
-            w.write_msgpack_object_prefix(j.m_data.m_value.object->size(), j);
-        }
-
-        static void scalar(binary_writer& w, const BasicJsonType& j)
-        {
-            w.write_msgpack(j);
-        }
-
-        static void string(binary_writer& w, const string_t& key, const BasicJsonType& context)
-        {
-            w.write_msgpack_string(key, context);
-        }
-
-        template<typename KeyType>
-        static void key(binary_writer& w, const KeyType& k, const BasicJsonType& object)
-        {
-            w.write_msgpack_key(k, object);
-        }
-    };
-
     /*!
-    @brief write @a j with @a Walk::scalar, or write its header and push a
-           frame for @ref write_container_iterative to continue with its
-           elements
+    @brief write @a j with @ref write_cbor, or write its header and push a
+           frame for @ref write_cbor_iterative to continue with its elements
 
     A scalar, and an empty array or object, are written out in full: there is
-    nothing below them for @ref write_container_iterative to come back to, so
+    nothing below them for @ref write_cbor_iterative to come back to, so
     nothing is pushed for them.
     */
-    template<typename Walk>
-    JSON_HEDLEY_ALWAYS_INLINE
-    void write_container_value_or_push(const BasicJsonType& j, std::vector<binary_container_frame>& stack)
+    void write_cbor_value_or_push(const BasicJsonType& j, std::vector<binary_container_frame>& stack)
     {
         if (j.is_array())
         {
-            Walk::array_head(*this, j);
+            write_cbor_head(0x80, j.m_data.m_value.array->size());
             if (!j.m_data.m_value.array->empty())
             {
                 stack.emplace_back(&j);
@@ -1093,7 +1032,7 @@ class binary_writer
 
         if (j.is_object())
         {
-            Walk::object_head(*this, j);
+            write_cbor_head(0xA0, j.m_data.m_value.object->size());
             if (!j.m_data.m_value.object->empty())
             {
                 stack.emplace_back(&j);
@@ -1101,26 +1040,22 @@ class binary_writer
             return;
         }
 
-        Walk::scalar(*this, j);
+        write_cbor(j);
     }
 
     /*!
     @brief write out @a root and everything below it without the call stack
 
-    Emits the same bytes as the recursive writer, keeping the containers it has
+    Emits the same bytes as @ref write_cbor, keeping the containers it has
     entered on an explicit stack instead of descending into them. Only reached
     for values nested deeper than @ref recursion_depth_limit, which is why it
-    is not written for speed. Shared by CBOR and MessagePack, which differ only
-    in the @a Walk policy (@ref cbor_walk, @ref msgpack_walk); BON8 has its own
-    walker, as it tracks an open string run and closes a container with a
-    marker.
+    is not written for speed.
     */
-    template<typename Walk>
-    void write_container_iterative(const BasicJsonType& root)
+    void write_cbor_iterative(const BasicJsonType& root)
     {
-        // only a container with elements is ever pushed; see write_container_value_or_push
+        // only a container with elements is ever pushed; see write_cbor_value_or_push
         std::vector<binary_container_frame> stack;
-        write_container_value_or_push<Walk>(root, stack);
+        write_cbor_value_or_push(root, stack);
 
         while (!stack.empty())
         {
@@ -1138,7 +1073,7 @@ class binary_writer
                 // read the child before pushing: entering it can move every frame
                 const BasicJsonType* child = &(*current.array_it);
                 ++stack.back().array_it;
-                write_container_value_or_push<Walk>(*child, stack);
+                write_cbor_value_or_push(*child, stack);
             }
             else
             {
@@ -1150,53 +1085,110 @@ class binary_writer
                 }
 
                 // the key is written directly (not via a temporary basic_json),
-                // with the object as diagnostics context, as in the recursive writer
-                Walk::key(*this, current.object_it->first, *current.value);
+                // with the object as diagnostics context, as in write_cbor
+                write_cbor_key(current.object_it->first, *current.value);
                 const BasicJsonType* child = &(current.object_it->second);
                 ++stack.back().object_it;
-                write_container_value_or_push<Walk>(*child, stack);
+                write_cbor_value_or_push(*child, stack);
             }
         }
     }
 
-    void write_cbor_iterative(const BasicJsonType& root)
-    {
-        write_container_iterative<cbor_walk>(root);
-    }
+    /*!
+    @brief write @a j with @ref write_msgpack, or write its header and push a
+           frame for @ref write_msgpack_iterative to continue with its elements
 
-    void write_msgpack_iterative(const BasicJsonType& root)
+    @sa @ref write_cbor_value_or_push
+    */
+    void write_msgpack_value_or_push(const BasicJsonType& j, std::vector<binary_container_frame>& stack)
     {
-        write_container_iterative<msgpack_walk>(root);
+        if (j.is_array())
+        {
+            write_msgpack_array_prefix(j.m_data.m_value.array->size(), j);
+            if (!j.m_data.m_value.array->empty())
+            {
+                stack.emplace_back(&j);
+            }
+            return;
+        }
+
+        if (j.is_object())
+        {
+            write_msgpack_object_prefix(j.m_data.m_value.object->size(), j);
+            if (!j.m_data.m_value.object->empty())
+            {
+                stack.emplace_back(&j);
+            }
+            return;
+        }
+
+        write_msgpack(j);
     }
 
     /*!
-    @brief write the start of an array (@a IsArray) or object: its opening
-           marker, the optimized type, and the count
+    @brief write out @a root and everything below it without the call stack
 
-    @return true when a closing ']' or '}' still has to be written after the
-            elements
+    @sa @ref write_cbor_iterative
     */
-    template<bool IsArray>
-    bool write_ubjson_start_container(const BasicJsonType& j, const bool use_count, const bool use_type,
-                                      const bool add_prefix, const bool use_bjdata, bool& prefix_required)
+    void write_msgpack_iterative(const BasicJsonType& root)
+    {
+        std::vector<binary_container_frame> stack;
+        write_msgpack_value_or_push(root, stack);
+
+        while (!stack.empty())
+        {
+            const binary_container_frame current = stack.back();
+
+            if (current.value->is_array())
+            {
+                const auto& array = *current.value->m_data.m_value.array;
+                if (current.array_it == array.cend())
+                {
+                    stack.pop_back();
+                    continue;
+                }
+
+                const BasicJsonType* child = &(*current.array_it);
+                ++stack.back().array_it;
+                write_msgpack_value_or_push(*child, stack);
+            }
+            else
+            {
+                const auto& object = *current.value->m_data.m_value.object;
+                if (current.object_it == object.cend())
+                {
+                    stack.pop_back();
+                    continue;
+                }
+
+                // as in write_cbor_iterative, the key is written directly with
+                // the object as diagnostics context
+                write_msgpack_key(current.object_it->first, *current.value);
+                const BasicJsonType* child = &(current.object_it->second);
+                ++stack.back().object_it;
+                write_msgpack_value_or_push(*child, stack);
+            }
+        }
+    }
+
+    /// @return true when a closing ']' still has to be written after the elements
+    bool write_ubjson_start_array(const BasicJsonType& j, const bool use_count, const bool use_type,
+                                  const bool add_prefix, const bool use_bjdata, bool& prefix_required)
     {
         prefix_required = true;
         if (add_prefix)
         {
-            oa.write_character(to_char_type(IsArray ? '[' : '{'));
+            oa.write_character(to_char_type('['));
         }
 
-        const std::size_t size = IsArray ? j.m_data.m_value.array->size() : j.m_data.m_value.object->size();
-
-        if (use_type && size != 0)
+        if (use_type && !j.m_data.m_value.array->empty())
         {
             if (!use_count)
             {
                 throw_size_marker_required(j);
             }
             const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata);
-            // the first element of an array has been compared with itself
-            const bool same_prefix = std::all_of(IsArray ? j.begin() + 1 : j.begin(), j.end(),
+            const bool same_prefix = std::all_of(j.begin() + 1, j.end(),
                                                  [this, first_prefix, use_bjdata](const BasicJsonType & v)
             {
                 return ubjson_prefix(v, use_bjdata) == first_prefix;
@@ -1208,9 +1200,8 @@ class binary_writer
             // those, at one byte per element, so the result can be read back.
             // Objects are not affected: every element is preceded by its key.
             const bool valueless_type = (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
-            const bool excessive_valueless = IsArray
-                                             && valueless_type
-                                             && size > detail::max_valueless_container_size;
+            const bool excessive_valueless = valueless_type
+                                             && j.m_data.m_value.array->size() > detail::max_valueless_container_size;
 
             if (same_prefix && !excessive_valueless
                     && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
@@ -1223,7 +1214,46 @@ class binary_writer
 
         if (use_count)
         {
-            write_ubjson_count(size, use_bjdata);
+            write_ubjson_count(j.m_data.m_value.array->size(), use_bjdata);
+        }
+
+        return !use_count;
+    }
+
+    /// @return true when a closing '}' still has to be written after the elements
+    bool write_ubjson_start_object(const BasicJsonType& j, const bool use_count, const bool use_type,
+                                   const bool add_prefix, const bool use_bjdata, bool& prefix_required)
+    {
+        prefix_required = true;
+        if (add_prefix)
+        {
+            oa.write_character(to_char_type('{'));
+        }
+
+        if (use_type && !j.m_data.m_value.object->empty())
+        {
+            if (!use_count)
+            {
+                throw_size_marker_required(j);
+            }
+            const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata);
+            const bool same_prefix = std::all_of(j.begin(), j.end(),
+                                                 [this, first_prefix, use_bjdata](const BasicJsonType & v)
+            {
+                return ubjson_prefix(v, use_bjdata) == first_prefix;
+            });
+
+            if (same_prefix && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
+            {
+                prefix_required = false;
+                oa.write_character(to_char_type('$'));
+                oa.write_character(first_prefix);
+            }
+        }
+
+        if (use_count)
+        {
+            write_ubjson_count(j.m_data.m_value.object->size(), use_bjdata);
         }
 
         return !use_count;
@@ -1251,12 +1281,21 @@ class binary_writer
 
     /// @brief an object or array @ref write_ubjson_iterative is still writing
     ///        the elements of
-    struct ubjson_frame : binary_container_frame
+    struct ubjson_frame
     {
         ubjson_frame(const BasicJsonType* value_, const bool prefix_required_) noexcept
-            : binary_container_frame(value_)
+            : value(value_)
             , prefix_required(prefix_required_)
-        {}
+        {
+            if (value->is_object())
+            {
+                object_it = value->m_data.m_value.object->cbegin();
+            }
+            else
+            {
+                array_it = value->m_data.m_value.array->cbegin();
+            }
+        }
 
         // declared for GCC's -Weffc++, which asks for them in a class with
         // pointer members and a non-trivial destructor; the exception
@@ -1268,9 +1307,13 @@ class binary_writer
         ubjson_frame& operator=(ubjson_frame&&) = default;
         ~ubjson_frame() = default;
 
+        /// the array or object being written
+        const BasicJsonType* value;
         /// whether value's elements each carry their own type marker; an
         /// optimized ($type) container writes it once for all of them instead
         bool prefix_required;
+        typename BasicJsonType::object_t::const_iterator object_it{};
+        typename BasicJsonType::array_t::const_iterator array_it{};
     };
 
     /*!
@@ -1281,7 +1324,7 @@ class binary_writer
                elements of an optimized container, and everything below the
                top level, never repeat it)
 
-    @sa @ref write_container_value_or_push
+    @sa @ref write_cbor_value_or_push
     */
     void write_ubjson_value_or_push(const BasicJsonType& j, const bool add_prefix, const bool use_count,
                                     const bool use_type, const bool use_bjdata, const bjdata_version_t bjdata_version,
@@ -1304,11 +1347,11 @@ class binary_writer
         bool prefix_required = true;
         if (is_array)
         {
-            write_ubjson_start_container<true>(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
+            write_ubjson_start_array(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
         }
         else
         {
-            write_ubjson_start_container<false>(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
+            write_ubjson_start_object(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
         }
 
         const bool empty = is_array ? j.m_data.m_value.array->empty() : j.m_data.m_value.object->empty();
@@ -1318,7 +1361,7 @@ class binary_writer
             return;
         }
 
-        // write_ubjson_start_container returns !use_count, i.e. whether a
+        // write_ubjson_start_array/_object return !use_count, i.e. whether a
         // closer still has to be written; use_count is constant for the whole
         // document, so that is recomputed here instead of being carried along
         if (!use_count)
@@ -1507,15 +1550,14 @@ class binary_writer
     {
         oa.write_character(to_char_type(element_type));
 
-        if (JSON_HEDLEY_LIKELY(error_handler == error_handler_t::keep || error_handler == error_handler_t::strict))
+        if (error_handler == error_handler_t::keep || error_handler == error_handler_t::strict || is_valid_utf8(name))
         {
-            write_bytes(name.data(), name.size());
+            oa.write_characters(reinterpret_cast<const CharType*>(name.data()), name.size());
         }
         else
         {
-            string_t storage;
-            const string_t& written = sanitize_utf8_for_bson_write(name, storage);
-            write_bytes(written.data(), written.size());
+            const string_t sanitized = sanitize_utf8(name, error_handler);
+            oa.write_characters(reinterpret_cast<const CharType*>(sanitized.data()), sanitized.size());
         }
 
         // the terminating null byte is written explicitly rather than taken
@@ -1591,11 +1633,16 @@ class binary_writer
     {
         write_bson_entry_header(name, 0x02);
 
-        string_t storage;
-        const string_t& written = sanitize_utf8_for_bson_write(value, storage);
+        const bool sanitize = error_handler != error_handler_t::keep
+                              && error_handler != error_handler_t::strict
+                              && !is_valid_utf8(value);
+        const string_t sanitized = sanitize ? sanitize_utf8(value, error_handler) : string_t{};
+        const string_t& written = sanitize ? sanitized : value;
 
         write_number<std::int32_t>(to_bson_length(written.size() + 1ul), true);
-        write_bytes(written.data(), written.size());
+        oa.write_characters(
+              reinterpret_cast<const CharType*>(written.data()),
+              written.size());
         // the terminating null byte is written explicitly rather than taken
         // from the buffer, so that string_t::data() need not be null-terminated
         oa.write_character(to_char_type(0x00));
@@ -1701,7 +1748,7 @@ class binary_writer
 
         write_number(value.has_subtype() ? static_cast<std::uint8_t>(value.subtype()) : static_cast<std::uint8_t>(0x00));
 
-        write_bytes(value.data(), value.size());
+        oa.write_characters(reinterpret_cast<const CharType*>(value.data()), value.size());
     }
 
     /*!
@@ -1731,14 +1778,14 @@ class binary_writer
             case value_t::number_integer:
                 if (JSON_HEDLEY_UNLIKELY(!integer_fits<std::int64_t>(j.m_data.m_value.number_integer)))
                 {
-                    throw_integer_out_of_range(j, input_format_t::bson, "int64");
+                    throw_integer_out_of_range(j, "BSON", "int64");
                 }
                 return calc_bson_integer_size(static_cast<std::int64_t>(j.m_data.m_value.number_integer));
 
             case value_t::number_unsigned:
                 if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_unsigned)))
                 {
-                    throw_integer_out_of_range(j, input_format_t::bson, "uint64");
+                    throw_integer_out_of_range(j, "BSON", "uint64");
                 }
                 return calc_bson_unsigned_size(conditional_static_cast<std::uint64_t>(j.m_data.m_value.number_unsigned));
 
@@ -1749,7 +1796,7 @@ class binary_writer
                 return 0ul;
 
             case value_t::discarded:
-                throw_on_discarded(j, input_format_t::bson);
+                throw_on_discarded(j, "BSON");
 
             // LCOV_EXCL_START
             case value_t::object:
@@ -1913,7 +1960,7 @@ class binary_writer
             // the terminating null byte is written explicitly rather than
             // taken from the buffer, so that string_t::data() need not be
             // null-terminated
-            write_bytes(value.data(), value.size());
+            oa.write_characters(reinterpret_cast<const CharType*>(value.data()), value.size());
             oa.write_character(to_char_type(0x00));
             return true;
         }
@@ -1930,7 +1977,7 @@ class binary_writer
             p += sizeof(std::int32_t);
             *p++ = value.has_subtype() ? to_char_type(static_cast<std::uint8_t>(value.subtype())) : to_char_type(0x00);
             oa.commit(static_cast<std::size_t>(p - out));
-            write_bytes(value.data(), value.size());
+            oa.write_characters(reinterpret_cast<const CharType*>(value.data()), value.size());
             return true;
         }
 
@@ -1978,7 +2025,7 @@ class binary_writer
                 return write_bson_null(name);
 
             case value_t::discarded:
-                throw_on_discarded(j, input_format_t::bson);
+                throw_on_discarded(j, "BSON");
 
             // LCOV_EXCL_START
             case value_t::object:
@@ -2759,7 +2806,7 @@ class binary_writer
         {
             if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_unsigned)))
             {
-                throw_integer_out_of_range(j, input_format_t::cbor, "uint64");
+                throw_integer_out_of_range(j, "CBOR", "uint64");
             }
             write_cbor_head(0x00, conditional_static_cast<std::uint64_t>(j.m_data.m_value.number_unsigned));
             return;
@@ -2769,7 +2816,7 @@ class binary_writer
         {
             if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_integer)))
             {
-                throw_integer_out_of_range(j, input_format_t::cbor, "[-2^64, 2^64-1]");
+                throw_integer_out_of_range(j, "CBOR", "[-2^64, 2^64-1]");
             }
             // CBOR does not differentiate between positive signed
             // integers and unsigned integers
@@ -2780,7 +2827,7 @@ class binary_writer
             // a negative integer n is encoded as -1 - n
             if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(-1 - j.m_data.m_value.number_integer)))
             {
-                throw_integer_out_of_range(j, input_format_t::cbor, "[-2^64, 2^64-1]");
+                throw_integer_out_of_range(j, "CBOR", "[-2^64, 2^64-1]");
             }
             write_cbor_head(0x20, static_cast<std::uint64_t>(-1 - j.m_data.m_value.number_integer));
         }
@@ -2856,7 +2903,7 @@ class binary_writer
         {
             if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_unsigned)))
             {
-                throw_integer_out_of_range(j, input_format_t::msgpack, "uint64");
+                throw_integer_out_of_range(j, "MessagePack", "uint64");
             }
             write_msgpack_unsigned(static_cast<std::uint64_t>(j.m_data.m_value.number_unsigned));
             return;
@@ -2866,7 +2913,7 @@ class binary_writer
         {
             if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_integer)))
             {
-                throw_integer_out_of_range(j, input_format_t::msgpack, "[-2^63, 2^64-1]");
+                throw_integer_out_of_range(j, "MessagePack", "[-2^63, 2^64-1]");
             }
             // MessagePack does not differentiate between positive
             // signed integers and unsigned integers.
@@ -2876,7 +2923,7 @@ class binary_writer
         {
             if (JSON_HEDLEY_UNLIKELY(!integer_fits_min<std::int64_t>(j.m_data.m_value.number_integer)))
             {
-                throw_integer_out_of_range(j, input_format_t::msgpack, "[-2^63, 2^64-1]");
+                throw_integer_out_of_range(j, "MessagePack", "[-2^63, 2^64-1]");
             }
             if (j.m_data.m_value.number_integer >= -32)
             {
@@ -3010,37 +3057,40 @@ class binary_writer
     }
 
     /*!
-    @brief write an object key as a CBOR or MessagePack string (@a Walk)
+    @brief write an object key as a CBOR text string
 
     A key convertible to string_t is written directly. Other key types (only
     an explicit conversion, or only a to_json overload) go through a temporary
     basic_json, as in version 3.12.0; the temporary is then the diagnostics
     context for strict UTF-8 checks.
     */
-    template<typename Walk, typename Key = typename BasicJsonType::object_t::key_type,
+    template<typename Key = typename BasicJsonType::object_t::key_type,
              enable_if_t<std::is_convertible<Key, string_t>::value, int> = 0>
-    void write_object_key(const typename BasicJsonType::object_t::key_type& key, const BasicJsonType& context)
-    {
-        Walk::string(*this, key, context);
-    }
-
-    template<typename Walk, typename Key = typename BasicJsonType::object_t::key_type,
-             enable_if_t<negation<std::is_convertible<Key, string_t>>::value, int> = 0>
-    void write_object_key(const typename BasicJsonType::object_t::key_type& key, const BasicJsonType& /*context*/)
-    {
-        Walk::scalar(*this, BasicJsonType(key));
-    }
-
-    /// @brief write an object key as a CBOR text string
     void write_cbor_key(const typename BasicJsonType::object_t::key_type& key, const BasicJsonType& context)
     {
-        write_object_key<cbor_walk>(key, context);
+        write_cbor_string(key, context);
     }
 
-    /// @brief write an object key as a MessagePack str
+    template < typename Key = typename BasicJsonType::object_t::key_type,
+               enable_if_t < !std::is_convertible<Key, string_t>::value, int > = 0 >
+    void write_cbor_key(const typename BasicJsonType::object_t::key_type& key, const BasicJsonType& /*context*/)
+    {
+        write_cbor(BasicJsonType(key));
+    }
+
+    /// @brief write an object key as a MessagePack str, as in @ref write_cbor_key
+    template<typename Key = typename BasicJsonType::object_t::key_type,
+             enable_if_t<std::is_convertible<Key, string_t>::value, int> = 0>
     void write_msgpack_key(const typename BasicJsonType::object_t::key_type& key, const BasicJsonType& context)
     {
-        write_object_key<msgpack_walk>(key, context);
+        write_msgpack_string(key, context);
+    }
+
+    template < typename Key = typename BasicJsonType::object_t::key_type,
+               enable_if_t < !std::is_convertible<Key, string_t>::value, int > = 0 >
+    void write_msgpack_key(const typename BasicJsonType::object_t::key_type& key, const BasicJsonType& /*context*/)
+    {
+        write_msgpack(BasicJsonType(key));
     }
 
     /// the longest string that is written, with its head of one or two
@@ -3115,7 +3165,9 @@ class binary_writer
         write_cbor_head(0x60, sanitized.size());
 
         // step 2: write the string
-        write_bytes(sanitized.data(), sanitized.size());
+        oa.write_characters(
+              reinterpret_cast<const CharType*>(sanitized.data()),
+              sanitized.size());
     }
 
     /////////////
@@ -3196,7 +3248,9 @@ class binary_writer
         }
 
         // step 2: write the string
-        write_bytes(sanitized.data(), sanitized.size());
+        oa.write_characters(
+              reinterpret_cast<const CharType*>(sanitized.data()),
+              sanitized.size());
     }
 
     ////////////
@@ -3479,7 +3533,9 @@ class binary_writer
             oa.write_character(to_char_type('S'));
         }
         write_number_with_ubjson_prefix(sanitized.size(), true, use_bjdata);
-        write_bytes(sanitized.data(), sanitized.size());
+        oa.write_characters(
+              reinterpret_cast<const CharType*>(sanitized.data()),
+              sanitized.size());
     }
 
     /*!
@@ -4279,7 +4335,7 @@ class binary_writer
             {
                 if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::int64_t>(j.m_data.m_value.number_unsigned)))
                 {
-                    throw_integer_out_of_range(j, input_format_t::bon8, "int64");
+                    throw_integer_out_of_range(j, "BON8", "int64");
                 }
                 write_bon8_integer(static_cast<std::int64_t>(j.m_data.m_value.number_unsigned));
                 string_open = false;
@@ -4290,7 +4346,7 @@ class binary_writer
             {
                 if (JSON_HEDLEY_UNLIKELY(!integer_fits<std::int64_t>(j.m_data.m_value.number_integer)))
                 {
-                    throw_integer_out_of_range(j, input_format_t::bon8, "int64");
+                    throw_integer_out_of_range(j, "BON8", "int64");
                 }
                 write_bon8_integer(static_cast<std::int64_t>(j.m_data.m_value.number_integer));
                 string_open = false;
@@ -4358,7 +4414,7 @@ class binary_writer
 
             case value_t::discarded:
             default:
-                throw_on_discarded(j, input_format_t::bon8);
+                throw_on_discarded(j, "BON8");
         }
     }
 
@@ -4369,7 +4425,7 @@ class binary_writer
     A scalar, and an empty array or object, are written out in full and not
     pushed.
 
-    @sa @ref write_container_value_or_push
+    @sa @ref write_cbor_value_or_push
     */
     void write_bon8_value_or_push(const BasicJsonType& j, bool& string_open, std::vector<binary_container_frame>& stack)
     {
@@ -4755,7 +4811,7 @@ class binary_writer
         }
         else
         {
-            write_bytes(s.data(), s.size());
+            oa.write_characters(reinterpret_cast<const CharType*>(s.data()), s.size());
             string_open = true;
         }
     }
@@ -4846,31 +4902,6 @@ class binary_writer
         }
     }
 
-    /*!
-    @brief the write-pass counterpart of @ref sanitize_utf8_for_write for BSON
-
-    The size pass (@ref calc_bson_entry_header_size, @ref
-    calc_bson_string_size) has already validated @a s, and thrown for @ref
-    error_handler_t::strict if it was ill-formed. Here, @ref
-    error_handler_t::strict therefore behaves like @ref error_handler_t::keep,
-    and @ref error_handler_t::replace / @ref error_handler_t::ignore produce
-    exactly the bytes the size pass measured. Hence no context is needed.
-
-    @param[in] s        the string (value or object key) to write
-    @param[out] storage  backing storage for a sanitized copy
-
-    @return a reference to @a s, or to @a storage once it holds a sanitized copy
-    */
-    const string_t& sanitize_utf8_for_bson_write(const string_t& s, string_t& storage) const
-    {
-        if (error_handler == error_handler_t::keep || error_handler == error_handler_t::strict || is_valid_utf8(s))
-        {
-            return s; // NOLINT(bugprone-return-const-ref-from-parameter): callers pass lvalues that outlive the call
-        }
-        storage = sanitize_utf8(s, error_handler);
-        return storage;
-    }
-
     /// deleted: anything but a string_t would bind a temporary that dies before the returned reference is used
     template < typename T, enable_if_t < !std::is_same<T, string_t>::value, int > = 0 >
     const string_t& sanitize_utf8_for_write(const T& /*s*/, const BasicJsonType& /*context*/, string_t& /*storage*/) const = delete; // NOLINT(hicpp-use-equals-delete,modernize-use-equals-delete): a private helper's guard, not part of the interface

include/nlohmann/detail/output/serializer.hpp

@@ -966,28 +966,6 @@ class serializer
         }
     }
 
-    /// write out the bytes pending in @ref string_buffer and reset @a bytes
-    JSON_HEDLEY_ALWAYS_INLINE void emit_pending(std::size_t& bytes)
-    {
-        if (bytes != 0)
-        {
-            put_buffer(string_buffer, bytes);
-            bytes = 0;
-        }
-    }
-
-    /// write buffer and reset index if fewer than 13 bytes are left, as this
-    /// is the maximal number of bytes to be written ("\uxxxx\uxxxx\0") for one
-    /// code point
-    JSON_HEDLEY_ALWAYS_INLINE void reserve_string_buffer(std::size_t& bytes)
-    {
-        if (string_buffer.size() - bytes < 13)
-        {
-            put_buffer(string_buffer, bytes);
-            bytes = 0;
-        }
-    }
-
     /*!
     @brief worker for @ref dump_escaped
 
@@ -1046,7 +1024,11 @@ class serializer
                 {
                     // emit any bytes still pending in string_buffer first to
                     // preserve output order, then write the run directly
-                    emit_pending(bytes);
+                    if (bytes != 0)
+                    {
+                        put_buffer(string_buffer, bytes);
+                        bytes = 0;
+                    }
                     put_run(s, i, run);
                     bytes_after_last_accept = 0;
                     undumped_chars = 0;
@@ -1067,10 +1049,11 @@ class serializer
                 {
                     // emit any bytes still pending in string_buffer first to
                     // preserve output order
-                    emit_pending(bytes);
-                    // deliberately not put_buffer(): a fixed 8-byte memcpy
-                    // has no asserts and no write-through branch, which
-                    // matters on the hot path
+                    if (bytes != 0)
+                    {
+                        put_buffer(string_buffer, bytes);
+                        bytes = 0;
+                    }
                     const auto& escape = ascii_escapes()[data[i]];
                     if (JSON_HEDLEY_UNLIKELY(write_buffer.size() - write_buffer_pos < escape.size()))
                     {
@@ -1145,7 +1128,14 @@ class serializer
                         string_buffer[bytes++] = s[i];
                     }
 
-                    reserve_string_buffer(bytes);
+                    // write buffer and reset index; there must be 13 bytes
+                    // left, as this is the maximal number of bytes to be
+                    // written ("\uxxxx\uxxxx\0") for one code point
+                    if (string_buffer.size() - bytes < 13)
+                    {
+                        put_buffer(string_buffer, bytes);
+                        bytes = 0;
+                    }
 
                     // remember the byte position of this accept
                     bytes_after_last_accept = bytes;
@@ -1197,7 +1187,14 @@ class serializer
                                     string_buffer[bytes++] = '\xBD';
                                 }
 
-                                reserve_string_buffer(bytes);
+                                // write buffer and reset index; there must be 13 bytes
+                                // left, as this is the maximal number of bytes to be
+                                // written ("\uxxxx\uxxxx\0") for one code point
+                                if (string_buffer.size() - bytes < 13)
+                                {
+                                    put_buffer(string_buffer, bytes);
+                                    bytes = 0;
+                                }
 
                                 bytes_after_last_accept = bytes;
                             }
@@ -1230,7 +1227,14 @@ class serializer
                                 string_buffer[bytes++] = s[i];
                             }
 
-                            reserve_string_buffer(bytes);
+                            // write buffer and reset index; there must be 13 bytes
+                            // left, as this is the maximal number of bytes to be
+                            // written ("\uxxxx\uxxxx\0") for one code point
+                            if (string_buffer.size() - bytes < 13)
+                            {
+                                put_buffer(string_buffer, bytes);
+                                bytes = 0;
+                            }
 
                             bytes_after_last_accept = bytes;
                             undumped_chars = 0;
@@ -1596,17 +1600,30 @@ class serializer
     */
     static const std::array<std::uint8_t, 256>& ascii_copyable() noexcept
     {
-        // constant-initialized from the predicate: no runtime loop, no guard
-        static constexpr std::array<std::uint8_t, 256> copyable = make_ascii_copyable(make_index_sequence<256> {});
+        static const std::array<std::uint8_t, 256> copyable =
+        {
+            {
+                0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0x00
+                0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0x10
+                1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 0x20
+                1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 0x30
+                1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 0x40
+                1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, // 0x50
+                1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 0x60
+                1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, // 0x70
+                0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0x80
+                0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0x90
+                0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0xA0
+                0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0xB0
+                0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0xC0
+                0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0xD0
+                0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0xE0
+                0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 // 0xF0
+            }
+        };
         return copyable;
     }
 
-    template<std::size_t... Is>
-    static constexpr std::array<std::uint8_t, 256> make_ascii_copyable(index_sequence<Is...> /*unused*/) noexcept
-    {
-        return {{ static_cast<std::uint8_t>(is_ascii_copyable(static_cast<unsigned char>(Is)) ? 1 : 0)... }};
-    }
-
     /*!
     @brief the JSON form of each character below 0x60
 
@@ -1735,6 +1752,7 @@ class serializer
     */
     JSON_HEDLEY_NEVER_INLINE std::size_t write_u_escapes(const unsigned char* data, std::size_t n)
     {
+        constexpr const char* nibble_to_hex = "0123456789abcdef";
         std::size_t i = 0;
         while (i < n && data[i] >= 0x80)
         {
@@ -1771,21 +1789,26 @@ class serializer
             char* d = write_buffer.data() + write_buffer_pos;
             if (codepoint > 0xFFFFu)
             {
-                d = put_u_escape(d, static_cast<std::uint16_t>(0xD7C0u + (codepoint >> 10u)));
+                d = put_u_escape(d, static_cast<std::uint16_t>(0xD7C0u + (codepoint >> 10u)), nibble_to_hex);
                 codepoint = 0xDC00u + (codepoint & 0x3FFu);
             }
-            d = put_u_escape(d, static_cast<std::uint16_t>(codepoint));
+            d = put_u_escape(d, static_cast<std::uint16_t>(codepoint), nibble_to_hex);
             write_buffer_pos = static_cast<std::size_t>(d - write_buffer.data());
             i += length;
         }
         return i;
     }
 
-    static char* put_u_escape(char* d, const std::uint16_t codeunit) noexcept
+    static char* put_u_escape(char* d, const std::uint16_t codeunit, const char* nibble_to_hex) noexcept
     {
+        const auto unit = static_cast<std::uint32_t>(codeunit);
         d[0] = '\\';
         d[1] = 'u';
-        return hex4_lower(codeunit, d + 2);
+        d[2] = nibble_to_hex[(unit >> 12u) & 0x0Fu];
+        d[3] = nibble_to_hex[(unit >> 8u) & 0x0Fu];
+        d[4] = nibble_to_hex[(unit >> 4u) & 0x0Fu];
+        d[5] = nibble_to_hex[unit & 0x0Fu];
+        return d + 6;
     }
 
     /*!
@@ -1794,7 +1817,7 @@ class serializer
      * Branch-free replacement for `snprintf(buf, 7, "\\u%04x", codeunit)` in the
      * string escaping hot path. It writes exactly six characters ('\\', 'u' and
      * four hex digits) at position @a pos of @a string_buffer via a nibble
-     * lookup table (@ref hex4_lower), avoiding the format-string parsing and locale machinery of
+     * lookup table, avoiding the format-string parsing and locale machinery of
      * `snprintf`. Advances @a pos by the number of bytes written (6).
      *
      * @param[in] pos       position in @a string_buffer to write at; there must
@@ -1804,9 +1827,13 @@ class serializer
     void write_u_escape(std::size_t& pos, std::uint16_t codeunit) noexcept
     {
         JSON_ASSERT(string_buffer.size() - pos >= 6);
+        constexpr const char* nibble_to_hex = "0123456789abcdef";
         string_buffer[pos + 0] = '\\';
         string_buffer[pos + 1] = 'u';
-        hex4_lower(codeunit, &string_buffer[pos + 2]);
+        string_buffer[pos + 2] = nibble_to_hex[(codeunit >> 12u) & 0x0Fu];
+        string_buffer[pos + 3] = nibble_to_hex[(codeunit >> 8u) & 0x0Fu];
+        string_buffer[pos + 4] = nibble_to_hex[(codeunit >> 4u) & 0x0Fu];
+        string_buffer[pos + 5] = nibble_to_hex[codeunit & 0x0Fu];
         pos += 6;
     }
 

include/nlohmann/detail/string_utils.hpp

@@ -13,6 +13,7 @@
 #include <cstddef> // size_t
 #include <cstdint> // uint8_t, uint32_t
 #include <string> // string, to_string
+#include <utility> // move
 
 #include <nlohmann/detail/abi_macros.hpp>
 #include <nlohmann/detail/macro_scope.hpp>
@@ -48,24 +49,6 @@ inline std::string hex_byte(const std::uint8_t byte)
     return result;
 }
 
-/*!
-@brief write a UTF-16 code unit as four lowercase hexadecimal digits
-
-The lowercase counterpart of hex_byte(), used for the `\\uXXXX` escapes of the
-serializers, whose output is lowercase. Writes exactly four characters to
-@a out and returns the pointer past them.
-*/
-JSON_HEDLEY_ALWAYS_INLINE char* hex4_lower(const std::uint16_t codeunit, char* out) noexcept
-{
-    constexpr const char* nibble_to_hex = "0123456789abcdef";
-    const unsigned int unit = codeunit;
-    out[0] = nibble_to_hex[(unit >> 12u) & 0x0Fu];
-    out[1] = nibble_to_hex[(unit >> 8u) & 0x0Fu];
-    out[2] = nibble_to_hex[(unit >> 4u) & 0x0Fu];
-    out[3] = nibble_to_hex[unit & 0x0Fu];
-    return out + 4;
-}
-
 ///////////////////
 // UTF-8 encoding //
 ///////////////////
@@ -121,66 +104,6 @@ void encode_utf8(std::uint32_t cp, const Out& out)
     }
 }
 
-/*!
-@brief the shape of a well-formed UTF-8 sequence, by its lead byte
-
-@a length is 1 for ASCII, 2 to 4 for a valid lead byte, and 0 for a byte that
-cannot start a well-formed sequence (a continuation byte, 0xC0, 0xC1, or 0xF5
-to 0xFF). For a multibyte sequence, the second byte must lie in
-[@a second_lo, @a second_hi], and every later byte in [0x80, 0xBF]
-(RFC 3629, section 4; Unicode Table 3-7). The narrower second-byte ranges
-exclude overlong forms (0xE0, 0xF0), surrogates (0xED), and code points
-beyond U+10FFFF (0xF4).
-*/
-struct utf8_sequence_shape
-{
-    std::uint8_t length;
-    std::uint8_t second_lo;
-    std::uint8_t second_hi;
-};
-
-/// @return the shape of a UTF-8 sequence starting with @a lead, see utf8_sequence_shape
-inline utf8_sequence_shape utf8_shape(const std::uint8_t lead) noexcept
-{
-    if (lead < 0x80u)
-    {
-        return {1, 0, 0};
-    }
-    if (lead < 0xC2u)
-    {
-        return {0, 0, 0};
-    }
-    if (lead <= 0xDFu)
-    {
-        return {2, 0x80u, 0xBFu};
-    }
-    if (lead == 0xE0u)
-    {
-        return {3, 0xA0u, 0xBFu};
-    }
-    if (lead == 0xEDu)
-    {
-        return {3, 0x80u, 0x9Fu};
-    }
-    if (lead <= 0xEFu)
-    {
-        return {3, 0x80u, 0xBFu};
-    }
-    if (lead == 0xF0u)
-    {
-        return {4, 0x90u, 0xBFu};
-    }
-    if (lead <= 0xF3u)
-    {
-        return {4, 0x80u, 0xBFu};
-    }
-    if (lead == 0xF4u)
-    {
-        return {4, 0x80u, 0x8Fu};
-    }
-    return {0, 0, 0};
-}
-
 ///////////////////
 // UTF-8 decoding //
 ///////////////////
@@ -196,7 +119,7 @@ This is a single-byte step of a "shift-based" UTF-8 decoder originally
 written by Björn Hoehrmann. See
 http://bjoern.hoehrmann.de/utf-8/decoder/dfa/ for details.
 
-The library checks UTF-8 well-formedness (RFC 3629, section 4) in five
+The library checks UTF-8 well-formedness (RFC 3629, section 4) in three
 places, which differ in speed, diagnostics, and how they read the input:
 
 - decode() below: the serializer, to escape and, in strict mode, reject
@@ -210,15 +133,10 @@ places, which differ in speed, diagnostics, and how they read the input:
   bulk string scan, the bulk path of the BON8 reader, and the BON8 writer.
   They must accept exactly what the lexer's switch accepts.
 - the byte path of binary_reader::get_bon8_string(): BON8 input without bulk
-  access, and the bytes the bulk path leaves to it. It keeps its own ranges:
-  taking them from utf8_shape() made BON8 decoding 2-3% slower (#5891).
-- check_utf8() in detail/view/edit.hpp: strings stored in an editable
-  json_document, with the index of the offending byte. It also uses
-  utf8_shape().
+  access, and the bytes the bulk path leaves to it.
 
-All five must accept the same set of sequences, which the test "UTF-8
-validators accept the same sequences" in unit-class_lexer.cpp checks
-exhaustively against decode() and utf8_shape().
+All four must accept the same set of sequences, so a change to one needs a
+matching change to the others.
 
 @param[in,out] state  the current decoder state
 @param[in,out] codep  codepoint (valid only if resulting state is UTF8_ACCEPT)

include/nlohmann/detail/value_t.hpp

@@ -177,36 +177,6 @@ inline bool owns_memory(const value_t t) noexcept
     }
 }
 
-/*!
-@brief whether a value of the given type is neither an object nor an array
-
-Unlike basic_json::is_primitive(), this includes value_t::discarded: it names
-the types that iterators walk with a primitive_iterator_t (a single value, or no
-value for null and discarded) and that JSON Pointer resolution and the
-serializer treat as leaves. Every enumerator is listed so that -Wswitch-enum
-flags this function when a new type is added.
-*/
-inline bool is_primitive(const value_t t) noexcept
-{
-    switch (t)
-    {
-        case value_t::object:
-        case value_t::array:
-            return false;
-
-        case value_t::null:
-        case value_t::string:
-        case value_t::boolean:
-        case value_t::number_integer:
-        case value_t::number_unsigned:
-        case value_t::number_float:
-        case value_t::binary:
-        case value_t::discarded:
-        default:
-            return true;
-    }
-}
-
 
 /*!
 @brief compare an integer with a floating point number without precision loss

include/nlohmann/detail/view/builder.hpp

@@ -275,11 +275,34 @@ class builder
                 0, 0, '\b', 0, 0, 0, '\f', 0, 0, 0, 0, 0, 0, 0, '\n', 0, 0, 0, '\r', 0, '\t', 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
             }
         };
+        // The write position is kept in locals: the stores of characters into
+        // the arena may alias the members of the builder, which would then be
+        // reloaded after every one. It is stored back where the arena grows,
+        // and by `write_back` on every return.
+        const unsigned char* const end = m_end;
         const std::size_t start = arena_used();
-        arena_run(s, static_cast<std::size_t>(p - s));
+        char* w = m_arena_write;
+        const char* w_end = m_arena_end;
+        const arena_write_back write_back(*this, w);
+        {
+            const auto n = static_cast<std::size_t>(p - s);
+            if (n <= 16 && end - s >= 16)
+            {
+                arena_ensure(w, w_end, 16);
+                std::memcpy(w, s, 16);
+            }
+            else if (n != 0)
+            {
+                // (an empty run leaves an empty arena unallocated, and memcpy()
+                // must not be passed its null pointer even for zero bytes)
+                arena_ensure(w, w_end, n);
+                std::memcpy(w, s, n);
+            }
+            w += n;
+        }
         for (;;)
         {
-            if (p == m_end)
+            if (p == end)
             {
                 return failed(error_code::string_missing_quote, p);
             }
@@ -287,7 +310,9 @@ class builder
             if (c == '"')
             {
                 ++p;
-                return decoded{p, start, arena_used() - start};
+                // (from w: `write_back` stores it only after the return value
+                // is made; the arena is allocated, since an escape was written)
+                return decoded{p, start, static_cast<std::size_t>(w - m_doc.arena.data()) - start};
             }
             if (c != '\\')
             {
@@ -297,7 +322,7 @@ class builder
                 return failed(c < 0x20 ? error_code::string_control_character : error_code::string_utf8, p);
             }
             ++p;
-            if (p == m_end)
+            if (p == end)
             {
                 return failed(error_code::string_missing_quote, p);
             }
@@ -315,7 +340,7 @@ class builder
                     {
                         return failed(error_code::string_surrogate_low, p);
                     }
-                    if (m_end - p < 2 || p[0] != '\\' || p[1] != 'u')
+                    if (end - p < 2 || p[0] != '\\' || p[1] != 'u')
                     {
                         return failed(error_code::string_surrogate_high, p);
                     }
@@ -331,26 +356,106 @@ class builder
                     }
                     cp = 0x10000 + ((cp - 0xD800) << 10) + (lo - 0xDC00);
                 }
-                arena_ensure(4);
-                m_arena_write = put_utf8(m_arena_write, cp);
+                arena_ensure(w, w_end, 4);
+                w = put_utf8(w, cp);
             }
             else if (NLOHMANN_VIEW_LIKELY(d < 128 && simple_escape[d] != 0))
             {
-                arena_ensure(1);
-                *m_arena_write++ = simple_escape[d];
+                arena_ensure(w, w_end, 1);
+                *w++ = simple_escape[d];
             }
             else
             {
                 --p;
                 return failed(error_code::string_escape, p);
             }
-            if (p != m_end && *p == '\\')
+#if NLOHMANN_VIEW_NEON
+            if (p != end && *p == '\\')
             {
                 continue; // consecutive escapes ("\u00e4\u00f6"): no run in between
             }
+            // The first 16 bytes of the run are checked one by one, in predicted branches, and copied
+            // as one block: where the runs between escapes are short, the vector step below would put
+            // its longer dependency chain (load, compare, mask, count, add) on p for each of them,
+            // and be slower than these steps (the scanner of the keys works the same way)
+            {
+                if (end - p >= 16)
+                {
+                    const std::uint8_t* const plain = string_plain();
+                    std::size_t steps = 16;
+#define NLOHMANN_VIEW_STEP(i) if (NLOHMANN_VIEW_LIKELY(plain[p[i]] != 0)) {} else { steps = (i); goto run_found; }
+                    NLOHMANN_VIEW_REPEAT16(NLOHMANN_VIEW_STEP)
+#undef NLOHMANN_VIEW_STEP
+run_found:
+                    arena_ensure(w, w_end, 16);
+                    std::memcpy(w, p, 16); // (only the first `steps` bytes are kept)
+                    w += steps;
+                    p += steps;
+                    if (steps < 16)
+                    {
+                        if (*p >= 0x80)
+                        {
+                            // non-ASCII: the UTF-8 check, and the rest of the run
+                            const unsigned char* const r = p;
+                            p = scan_string_run(p, end);
+                            const auto n = static_cast<std::size_t>(p - r);
+                            arena_ensure(w, w_end, n);
+                            std::memcpy(w, r, n);
+                            w += n;
+                        }
+                        continue;
+                    }
+                }
+            }
+#else
+            if (p != end && (*p == '\\' || *p == '"'))
+            {
+                continue; // consecutive escapes ("\u00e4\u00f6"), or the end of the string: no run in between
+            }
+#endif
+#if NLOHMANN_VIEW_VECTOR
+            // the plain bytes up to the next quote, backslash, or control
+            // character are copied as they are scanned, 16 bytes at a time
+            for (;;)
+            {
+                if (NLOHMANN_VIEW_UNLIKELY(end - p < 16))
+                {
+                    const unsigned char* const r = p;
+                    p = scan_string_run(p, end);
+                    const auto n = static_cast<std::size_t>(p - r);
+                    arena_ensure(w, w_end, n);
+                    std::memcpy(w, r, n);
+                    w += n;
+                    break;
+                }
+                arena_ensure(w, w_end, 16);
+                const unsigned k = copy_plain_run16(p, w);
+                w += k;
+                p += k;
+                if (k == 16)
+                {
+                    continue;
+                }
+                if (NLOHMANN_VIEW_UNLIKELY(*p >= 0x80))
+                {
+                    // non-ASCII: the UTF-8 check, and the rest of the run
+                    const unsigned char* const r = p;
+                    p = scan_string_run(p, end);
+                    const auto n = static_cast<std::size_t>(p - r);
+                    arena_ensure(w, w_end, n);
+                    std::memcpy(w, r, n);
+                    w += n;
+                }
+                break;
+            }
+#else
             const unsigned char* const r = p;
-            p = scan_string_run(p, m_end);
-            arena_run(r, static_cast<std::size_t>(p - r));
+            p = scan_string_run(p, end);
+            const auto n = static_cast<std::size_t>(p - r);
+            arena_ensure(w, w_end, n);
+            std::memcpy(w, r, n);
+            w += n;
+#endif
         }
     }
 
@@ -392,19 +497,49 @@ class builder
     char* m_arena_write = nullptr;
     char* m_arena_end = nullptr;
 
+    /// room for n more bytes at w, the write position kept in a local, with
+    /// w_end the end of the arena. (A lambda instead is not always inlined,
+    /// and then w would be kept in memory, as m_arena_write is.)
+    NLOHMANN_VIEW_ALWAYS_INLINE void arena_ensure(char*& w, const char*& w_end, std::size_t n)
+    {
+        if (NLOHMANN_VIEW_UNLIKELY(static_cast<std::size_t>(w_end - w) < n))
+        {
+            m_arena_write = w;
+            arena_grow(n);
+            w = m_arena_write;
+            w_end = m_arena_end;
+        }
+    }
+
+    /// stores a write position kept in a local back into m_arena_write when
+    /// the function that keeps it returns, on every path
+    class arena_write_back
+    {
+      public:
+        arena_write_back(builder& b, char*& w) noexcept
+            : m_builder(b)
+            , m_w(w)
+        {}
+        ~arena_write_back()
+        {
+            m_builder.m_arena_write = m_w;
+        }
+
+        arena_write_back(const arena_write_back&) = delete;
+        arena_write_back& operator=(const arena_write_back&) = delete;
+        arena_write_back(arena_write_back&&) = delete;
+        arena_write_back& operator=(arena_write_back&&) = delete;
+
+      private:
+        builder& m_builder;
+        char*& m_w;
+    };
+
     NLOHMANN_VIEW_ALWAYS_INLINE std::size_t arena_used() const noexcept
     {
         return m_arena_write != nullptr ? static_cast<std::size_t>(m_arena_write - m_doc.arena.data()) : 0;
     }
 
-    NLOHMANN_VIEW_ALWAYS_INLINE void arena_ensure(std::size_t n)
-    {
-        if (NLOHMANN_VIEW_UNLIKELY(static_cast<std::size_t>(m_arena_end - m_arena_write) < n))
-        {
-            arena_grow(n);
-        }
-    }
-
     NLOHMANN_VIEW_NOINLINE void arena_grow(std::size_t n)
     {
         const std::size_t used = arena_used();
@@ -413,25 +548,6 @@ class builder
         m_arena_end = m_doc.arena.data() + m_doc.arena.size();
     }
 
-    /// append the run [r, r + n) to the arena; short runs as one fixed-size
-    /// 16-byte move when both sides have the room (no library call)
-    NLOHMANN_VIEW_ALWAYS_INLINE void arena_run(const unsigned char* r, std::size_t n)
-    {
-        if (n <= 16 && m_end - r >= 16)
-        {
-            arena_ensure(16);
-            std::memcpy(m_arena_write, r, 16);
-        }
-        else if (n != 0)
-        {
-            // (an empty run leaves an empty arena unallocated, and memcpy()
-            // must not be passed its null pointer even for zero bytes)
-            arena_ensure(n);
-            std::memcpy(m_arena_write, r, n);
-        }
-        m_arena_write += n;
-    }
-
     /// UTF-8 encoding of cp at w (room for 4 bytes); returns the position after
     /// it. (A returned pointer stays in a register; writing through a
     /// reference to m_arena_write would store and reload it around every byte.)
@@ -954,11 +1070,11 @@ indent_done:
                 ++p;
             }
             const unsigned char* const int_start = p;
-            if (p != e && *p == '0')
+            if (cur() == '0')
             {
                 ++p;
             }
-            else if (NLOHMANN_VIEW_LIKELY(p != e && *p >= '1' && *p <= '9'))
+            else if (NLOHMANN_VIEW_LIKELY(cur() >= '1' && cur() <= '9'))
             {
                 p = skip_digits(p + 1, e);
             }
@@ -969,7 +1085,7 @@ indent_done:
             const auto int_digits = static_cast<std::size_t>(p - int_start);
             std::size_t frac_digits = 0;
             bool is_float = false;
-            if (p != e && *p == '.')
+            if (cur() == '.')
             {
                 ++p;
                 const unsigned char* const f0 = p;
@@ -982,20 +1098,20 @@ indent_done:
                 is_float = true;
             }
             std::int64_t exponent = 0;
-            if (p != e && (*p | 0x20) == 'e')
+            if ((cur() | 0x20) == 'e')
             {
                 ++p;
                 bool exp_negative = false;
-                if (p != e && (*p == '+' || *p == '-'))
+                if (cur() == '+' || cur() == '-')
                 {
                     exp_negative = *p == '-';
                     ++p;
                 }
-                if (NLOHMANN_VIEW_UNLIKELY(p == e || !is_digit(*p)))
+                if (NLOHMANN_VIEW_UNLIKELY(!is_digit(cur())))
                 {
                     return fail(error_code::number_after_exponent);
                 }
-                while (p != e && is_digit(*p))
+                while (is_digit(cur()))
                 {
                     if (exponent < 100000)
                     {
@@ -1057,7 +1173,7 @@ indent_done:
             ++p; // opening quote
             const unsigned char* const s = p;
             p = scan_string_run<Value>(p, e);
-            if (NLOHMANN_VIEW_LIKELY(p != e && *p == '"'))
+            if (NLOHMANN_VIEW_LIKELY(cur() == '"'))
             {
                 push_node(value_t::string, 0, 0, static_cast<std::size_t>(s - b), static_cast<std::uint64_t>(p - s));
                 ++p;

include/nlohmann/detail/view/edit.hpp

@@ -51,6 +51,11 @@ struct first_true<false, Conditions...> : std::integral_constant < int, 1 + firs
 inline void check_utf8(const char* s, std::size_t n)
 {
     const auto* const p = reinterpret_cast<const unsigned char*>(s); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
+    const auto hex = [](unsigned char c)
+    {
+        constexpr const char* digits = "0123456789ABCDEF";
+        return std::string{digits[c >> 4u], digits[c & 0xFu]};
+    };
     for (std::size_t i = 0; i < n;)
     {
         const unsigned char c = p[i];
@@ -59,22 +64,39 @@ inline void check_utf8(const char* s, std::size_t n)
             ++i;
             continue;
         }
-        const utf8_sequence_shape shape = utf8_shape(c);
-        const std::size_t len = shape.length;
-        if (len == 0)
+        std::size_t len = 0;
+        unsigned char lo = 0x80;
+        unsigned char hi = 0xBF;
+        if (c >= 0xC2 && c <= 0xDF)
         {
-            throw_type_error(exception_id::invalid_utf8, concat("invalid UTF-8 byte at index ", std::to_string(i), ": 0x", hex_byte(c)));
+            len = 2;
+        }
+        else if (c >= 0xE0 && c <= 0xEF)
+        {
+            len = 3;
+            lo = c == 0xE0 ? 0xA0 : 0x80;
+            hi = c == 0xED ? 0x9F : 0xBF;
+        }
+        else if (c >= 0xF0 && c <= 0xF4)
+        {
+            len = 4;
+            lo = c == 0xF0 ? 0x90 : 0x80;
+            hi = c == 0xF4 ? 0x8F : 0xBF;
+        }
+        else
+        {
+            throw_type_error(316, concat("invalid UTF-8 byte at index ", std::to_string(i), ": 0x", hex(c)));
         }
         for (std::size_t k = 1; k < len; ++k)
         {
             if (i + k == n)
             {
-                throw_type_error(exception_id::invalid_utf8, concat("incomplete UTF-8 string; last byte: 0x", hex_byte(p[n - 1])));
+                throw_type_error(316, concat("incomplete UTF-8 string; last byte: 0x", hex(p[n - 1])));
             }
             const unsigned char b = p[i + k];
-            if (b < (k == 1 ? shape.second_lo : 0x80) || b > (k == 1 ? shape.second_hi : 0xBF))
+            if (b < (k == 1 ? lo : 0x80) || b > (k == 1 ? hi : 0xBF))
             {
-                throw_type_error(exception_id::invalid_utf8, concat("invalid UTF-8 byte at index ", std::to_string(i + k), ": 0x", hex_byte(b)));
+                throw_type_error(316, concat("invalid UTF-8 byte at index ", std::to_string(i + k), ": 0x", hex(b)));
             }
         }
         i += len;
@@ -122,7 +144,7 @@ class editor
         node* const o = own(object);
         if (o->kind != static_cast<std::uint8_t>(value_t::object) && o->kind != static_cast<std::uint8_t>(value_t::null))
         {
-            throw_type_error(exception_id::subscript_wrong_type, "cannot use operator[] with a string argument with ", object.type_name());
+            throw_type_error(305, "cannot use operator[] with a string argument with ", object.type_name());
         }
         check_utf8(key.data(), key.size());
         const encoded e = encode(std::forward<V>(value));
@@ -173,7 +195,7 @@ class editor
         node* const a = own(array);
         if (a->kind != static_cast<std::uint8_t>(value_t::array))
         {
-            throw_type_error(exception_id::subscript_wrong_type, "cannot use operator[] with a numeric argument with ", array.type_name());
+            throw_type_error(305, "cannot use operator[] with a numeric argument with ", array.type_name());
         }
         check_index(idx, a->len);
         const encoded e = encode(std::forward<V>(value));
@@ -190,7 +212,7 @@ class editor
         node* const a = own(array);
         if (a->kind != static_cast<std::uint8_t>(value_t::array) && a->kind != static_cast<std::uint8_t>(value_t::null))
         {
-            throw_type_error(exception_id::push_back_wrong_type, "cannot use push_back() with ", array.type_name());
+            throw_type_error(308, "cannot use push_back() with ", array.type_name());
         }
         const encoded e = encode(std::forward<V>(value));
         if (a->kind == static_cast<std::uint8_t>(value_t::null))
@@ -214,7 +236,7 @@ class editor
         node* const a = own(array);
         if (a->kind != static_cast<std::uint8_t>(value_t::array))
         {
-            throw_type_error(exception_id::insert_wrong_type, "cannot use insert() with ", array.type_name());
+            throw_type_error(309, "cannot use insert() with ", array.type_name());
         }
         check_index(idx, a->len + 1);
         const encoded e = encode(std::forward<V>(value));
@@ -234,7 +256,7 @@ class editor
         node* const o = own(object);
         if (o->kind != static_cast<std::uint8_t>(value_t::object))
         {
-            throw_type_error(exception_id::erase_wrong_type, "cannot use erase() with ", object.type_name());
+            throw_type_error(307, "cannot use erase() with ", object.type_name());
         }
         for (const node* k = nav::first(m_doc, o), *end = nav::end(m_doc, o); k != end; k = document_data::after(k + 1))
         {
@@ -252,7 +274,7 @@ class editor
         node* const a = own(array);
         if (a->kind != static_cast<std::uint8_t>(value_t::array))
         {
-            throw_type_error(exception_id::erase_wrong_type, "cannot use erase() with ", array.type_name());
+            throw_type_error(307, "cannot use erase() with ", array.type_name());
         }
         check_index(idx, a->len);
         node* const h = block_of(m_doc, a, 0);
@@ -275,7 +297,7 @@ class editor
     {
         if (NLOHMANN_VIEW_UNLIKELY(v.m_doc != &m_doc || v.m_node == nullptr))
         {
-            throw_invalid_iterator(exception_id::iterator_from_other_value, "view does not belong to this document");
+            throw_invalid_iterator(202, "view does not belong to this document");
         }
         edit_state_of(m_doc);
         return const_cast<node*>(v.m_node); // NOLINT(cppcoreguidelines-pro-type-const-cast): the nodes belong to this document
@@ -285,7 +307,7 @@ class editor
     {
         if (idx >= limit)
         {
-            throw_array_index_out_of_range(idx);
+            throw_out_of_range(401, concat("array index ", std::to_string(idx), " is out of range"));
         }
     }
 
@@ -426,7 +448,7 @@ class editor
     {
         if (NLOHMANN_VIEW_UNLIKELY(v.m_node == nullptr))
         {
-            throw_type_error(exception_id::type_mismatch, "type must be a value, but is ", "discarded");
+            throw_type_error(302, "type must be a value, but is ", "discarded");
         }
         encoded r;
         if (!is_container(*v.m_node))
@@ -576,7 +598,7 @@ class editor
     {
         if (NLOHMANN_VIEW_UNLIKELY(len >= 0xFFFFFFFFu))
         {
-            throw_out_of_range(exception_id::document_too_large, "strings of 4 GiB or more are not supported by json_document"); // LCOV_EXCL_LINE
+            throw_out_of_range(416, "strings of 4 GiB or more are not supported by json_document"); // LCOV_EXCL_LINE
         }
         node n = plain_node(value_t::string);
         n.flags = static_cast<std::uint8_t>(n.flags | node_flags::edited);
@@ -651,12 +673,12 @@ class editor
                 return string_node(s.data(), s.size());
             }
             case value_t::binary:
-                throw_type_error(exception_id::document_binary_value, "cannot store a binary value in a json_document", "");
+                throw_type_error(319, "cannot store a binary value in a json_document", "");
             case value_t::discarded:
             case value_t::object:
             case value_t::array:
             default:
-                throw_type_error(exception_id::type_mismatch, "type must be a value, but is ", "discarded");
+                throw_type_error(302, "type must be a value, but is ", "discarded");
         }
     }
 

include/nlohmann/detail/view/edit_storage.hpp

@@ -84,7 +84,7 @@ inline std::size_t text_capacity(std::size_t cap, std::size_t used, std::size_t
     constexpr std::size_t limit = 0xFFFFFFFFu;
     if (NLOHMANN_VIEW_UNLIKELY(used > limit || n > limit - used))
     {
-        throw_out_of_range(exception_id::document_too_large, "edits of 4 GiB or more are not supported by json_document");
+        throw_out_of_range(416, "edits of 4 GiB or more are not supported by json_document");
     }
     const std::size_t needed = used + n;
     const std::size_t wanted = needed + (std::min)(limit - needed, std::size_t{256});
@@ -135,7 +135,7 @@ inline void reserve_moved(document_data& d)
     document_data::edit_state& e = edit_state_of(d);
     if (e.moved.size() >= 0xFFFFFFFFu)
     {
-        throw_out_of_range(exception_id::document_too_large, "more than 4294967295 edited arrays and objects are not supported by json_document"); // LCOV_EXCL_LINE
+        throw_out_of_range(416, "more than 4294967295 edited arrays and objects are not supported by json_document"); // LCOV_EXCL_LINE
     }
     if (e.moved.size() == e.moved.capacity() || e.moved_cap.size() == e.moved_cap.capacity())
     {

include/nlohmann/detail/view/errors.hpp

@@ -25,52 +25,32 @@ namespace view
 // Exceptions are thrown out of line, so that the accessors that may throw stay
 // small enough to be inlined.
 
-[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_type_error(exception_id id, const char* prefix, const char* type)
+[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_type_error(int id, const char* prefix, const char* type)
 {
     NLOHMANN_VIEW_THROW(type_error::create(id, concat(prefix, type), nullptr));
 }
 
-[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_type_error(exception_id id, const std::string& msg)
+[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_type_error(int id, const std::string& msg)
 {
     NLOHMANN_VIEW_THROW(type_error::create(id, msg, nullptr));
 }
 
-[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_out_of_range(exception_id id, const std::string& msg)
+[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_out_of_range(int id, const std::string& msg)
 {
     NLOHMANN_VIEW_THROW(out_of_range::create(id, msg, nullptr));
 }
 
-[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_invalid_iterator(exception_id id, const char* msg)
+[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_invalid_iterator(int id, const char* msg)
 {
     NLOHMANN_VIEW_THROW(invalid_iterator::create(id, msg, nullptr));
 }
 
 /// a parse error without a position (as those of json_pointer)
-[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_parse_error(exception_id id, const std::string& msg)
+[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_parse_error(int id, const std::string& msg)
 {
     NLOHMANN_VIEW_THROW(parse_error::create(id, 0, msg, nullptr));
 }
 
-/// out_of_range.401: the array index @a idx is not below the size
-template<typename Index>
-[[noreturn]] NLOHMANN_VIEW_NOINLINE void throw_array_index_out_of_range(const Index idx)
-{
-    throw_out_of_range(exception_id::array_index_out_of_range, concat("array index ", std::to_string(idx), " is out of range"));
-}
-
-/// out_of_range.402: the JSON Pointer token '-' of an array of @a size elements
-[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_array_index_past_end(std::size_t size)
-{
-    throw_out_of_range(exception_id::pointer_past_the_end_index, concat("array index '-' (", std::to_string(size), ") is out of range"));
-}
-
-/// out_of_range.403: the object has no member @a key
-template<typename Key>
-[[noreturn]] NLOHMANN_VIEW_NOINLINE void throw_key_not_found(const Key& key)
-{
-    throw_out_of_range(exception_id::key_not_found, concat("key '", key, "' not found"));
-}
-
 /// SAX handler of the error replay: accepts every event without storing it,
 /// and throws the exception the parser reports
 template<typename BasicJsonType>
@@ -144,11 +124,21 @@ struct throwing_sax
 
     /// (a template, as in json_sax_dom_parser: the parser passes the concrete
     /// exception type, which is thrown as is)
+    ///
+    /// It is not [[noreturn]]: sax_parse() recovers from an error if
+    /// parse_error() returns true, and MSVC reports that code as unreachable
+    /// (C4702) for a handler that never returns. The flag hides that from it.
     template<class Exception>
-    [[noreturn]] bool parse_error(std::size_t /*unused*/, const std::string& /*unused*/, const Exception& ex)
+    bool parse_error(std::size_t /*unused*/, const std::string& /*unused*/, const Exception& ex)
     {
-        NLOHMANN_VIEW_THROW(ex);
+        if (throwing)
+        {
+            NLOHMANN_VIEW_THROW(ex);
+        }
+        return false; // LCOV_EXCL_LINE
     }
+
+    bool throwing = true;
 };
 
 /*!
@@ -169,7 +159,7 @@ template<typename BasicJsonType>
     if (f.code == error_code::input_too_large)
     {
         // (the limit is detail::view::max_input_size(): 4 GiB minus 16 bytes)
-        NLOHMANN_VIEW_THROW(out_of_range::create(exception_id::document_too_large, "input of 4294967280 bytes or more is not supported by json_document", nullptr));
+        NLOHMANN_VIEW_THROW(out_of_range::create(416, "input of 4294967280 bytes or more is not supported by json_document", nullptr));
     }
     throwing_sax<BasicJsonType> sax;
     const bool accepted = BasicJsonType::sax_parse(src, src + size, &sax, input_format_t::json, true, ignore_comments, ignore_trailing_commas);
@@ -189,7 +179,7 @@ template<typename BasicJsonType>
         }
     }
     pos.chars_read_current_line = off + 1 - line_start;
-    NLOHMANN_VIEW_THROW(parse_error::create(exception_id::syntax_error, pos, "syntax error while parsing value", nullptr));
+    NLOHMANN_VIEW_THROW(parse_error::create(101, pos, "syntax error while parsing value", nullptr));
     // LCOV_EXCL_STOP
 }
 

include/nlohmann/detail/view/image.hpp

@@ -186,7 +186,7 @@ inline void compact_nodes(const document_data& d, std::size_t arena_size, std::v
 inline std::vector<std::uint8_t> save_image(const document_data& d)
 {
 #if !NLOHMANN_VIEW_LITTLE_ENDIAN
-    throw_type_error(exception_id::document_image_unsupported, "json_document images need a little-endian target"); // LCOV_EXCL_LINE
+    throw_type_error(320, "json_document images need a little-endian target"); // LCOV_EXCL_LINE
 #endif
     const std::size_t arena_size = d.arena_size;
     std::vector<node> compacted;
@@ -203,7 +203,7 @@ inline std::vector<std::uint8_t> save_image(const document_data& d)
     if (NLOHMANN_VIEW_UNLIKELY(text_size >= image_limit || total_arena >= image_limit || count >= image_limit))
     {
         // LCOV_EXCL_START (4 GiB)
-        throw_out_of_range(exception_id::document_too_large, "images of 4 GiB or more are not supported by json_document");
+        throw_out_of_range(416, "images of 4 GiB or more are not supported by json_document");
         // LCOV_EXCL_STOP
     }
     image_header h{};
@@ -660,7 +660,7 @@ inline bool check_image(const node* nodes, std::size_t count, const unsigned cha
 
 [[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_invalid_image(const char* what)
 {
-    throw_parse_error(exception_id::invalid_document_image, concat("invalid json_document image: ", what));
+    throw_parse_error(116, concat("invalid json_document image: ", what));
 }
 
 /// Read an image into d. The text and the decoded strings stay in the image;
@@ -668,7 +668,7 @@ inline bool check_image(const node* nodes, std::size_t count, const unsigned cha
 inline void load_image(document_data& d, const std::uint8_t* image, std::size_t size, image_check check)
 {
 #if !NLOHMANN_VIEW_LITTLE_ENDIAN
-    throw_type_error(exception_id::document_image_unsupported, "json_document images need a little-endian target"); // LCOV_EXCL_LINE
+    throw_type_error(320, "json_document images need a little-endian target"); // LCOV_EXCL_LINE
 #endif
     if (image == nullptr || size < sizeof(image_header))
     {

include/nlohmann/detail/view/input.hpp

@@ -10,6 +10,9 @@
 
 #include <algorithm> // max
 #include <cstddef> // size_t
+#ifndef JSON_NO_IO
+    #include <cstdio> // FILE, fread
+#endif
 #include <istream> // istream, streambuf
 #include <string> // basic_string, char_traits, string
 #include <type_traits> // decay, integral_constant, is_array, is_const, is_integral, is_lvalue_reference, is_pointer, is_same, remove_cv, remove_extent, remove_pointer, remove_reference
@@ -154,6 +157,37 @@ inline void append_stream(std::string& buf, std::istream& is)
     is.clear(is.rdstate() | std::ios::eofbit);
 }
 
+/// a pointer to a C stream (never with JSON_NO_IO, which leaves out <cstdio>)
+template<typename T>
+struct is_c_stream
+#ifndef JSON_NO_IO
+    : std::is_same<T, std::FILE*>
+#else
+    : std::false_type
+#endif
+{};
+
+#ifndef JSON_NO_IO
+/// append the rest of a C stream: in blocks with fread, not character by
+/// character (the library's file adapter reads with fgetc);
+/// f is not null
+inline void append_file(std::string& buf, std::FILE* f)
+{
+    for (;;)
+    {
+        const std::size_t used = buf.size();
+        const std::size_t room = (std::max)(buf.capacity() - used, std::size_t{4096});
+        buf.resize(used + room);
+        const std::size_t got = std::fread(&buf[used], 1, room, f);
+        buf.resize(used + got);
+        if (got < room)
+        {
+            break; // end of file, or an error (as fgetc, which returns EOF for both)
+        }
+    }
+}
+#endif
+
 }  // namespace view
 }  // namespace detail
 NLOHMANN_JSON_NAMESPACE_END

include/nlohmann/detail/view/iterator.hpp

@@ -110,7 +110,7 @@ class view_iterator
     {
         if (NLOHMANN_VIEW_UNLIKELY(m_value_offset == 0))
         {
-            throw_invalid_iterator(exception_id::iterator_key_not_object, "cannot use key() for non-object iterators");
+            throw_invalid_iterator(207, "cannot use key() for non-object iterators");
         }
         return string_view_t(m_doc->str(*m_pos), m_pos->len);
     }
@@ -147,20 +147,21 @@ class view_item
 
     view_item(const iterator& it, std::size_t index)
         : m_it(it)
-    {
-        if (!it.is_object_iterator())
-        {
-            m_index = std::to_string(index);
-        }
-    }
+        , m_position(index)
+    {}
 
-    /// the member key, or the element index for arrays
+    /// the member key, or the element index for arrays (as text made on the
+    /// first call, so that iterating over the values pays nothing for it)
     string_view_t key() const
     {
         if (m_it.is_object_iterator())
         {
             return m_it.key();
         }
+        if (m_index.empty())
+        {
+            m_index = std::to_string(m_position);
+        }
         return string_view_t(m_index.data(), m_index.size());
     }
 
@@ -183,7 +184,8 @@ class view_item
 
   private:
     iterator m_it;
-    std::string m_index{}; // NOLINT(readability-redundant-member-init)
+    std::size_t m_position; ///< the index of an array element
+    mutable std::string m_index{}; ///< the text of m_position, once key() was called // NOLINT(readability-redundant-member-init)
 };
 
 /// the range returned by basic_json_view::items()

include/nlohmann/detail/view/lookup.hpp

@@ -166,6 +166,10 @@ const node* element_at(const document_data& d, const node* array, std::size_t id
     {
         return e + idx; // a growable block: one link per element
     }
+    if (array->next == array->len + 1u && (!nav::editable() || (array->flags & node_flags::moved) == 0))
+    {
+        return e + idx; // no element has a subtree: the elements are the nodes after the array
+    }
     for (std::size_t i = 0; i < idx; ++i)
     {
         e = document_data::after(e);
@@ -181,6 +185,10 @@ const node* last_child(const document_data& d, const node* container) noexcept
     const std::size_t value_offset = container->kind == static_cast<std::uint8_t>(value_t::object) ? 1 : 0;
     const node* const end = navigation<Editable>::end(d, container);
     const node* last = navigation<Editable>::first(d, container);
+    if (value_offset == 0 && container->next == container->len + 1u && (!navigation<Editable>::editable() || (container->flags & node_flags::moved) == 0))
+    {
+        return last + (container->len - 1u); // an array without subtrees: the elements are the nodes after it
+    }
     for (const node* c = document_data::after(last + value_offset); c != end; c = document_data::after(c + value_offset))
     {
         last = c;

include/nlohmann/detail/view/materialize.hpp

@@ -8,7 +8,7 @@
 
 #pragma once
 
-#include <algorithm> // max
+#include <array> // array
 #include <cstddef> // size_t
 #include <cstdint> // int64_t, uint8_t
 #include <string> // string
@@ -40,12 +40,13 @@ template<typename BasicJsonType, bool Editable>
 BasicJsonType materialize(const document_data& d, const node* n)
 {
     using string_t = typename BasicJsonType::string_t;
-    using sax_t = json_sax_dom_parser<BasicJsonType, iterator_input_adapter<const char*>>;
+    using sax_t = json_sax_dom_parser<BasicJsonType, iterator_input_adapter<const char*>, true>;
     using nav = navigation<Editable>;
 
     // the open containers: their next element (or key of the next member),
-    // their end, and whether they are objects; sized once from the
-    // document's deepest nesting (for a subtree an upper bound)
+    // their end, and whether they are objects; for the first 16 levels on the
+    // stack, else sized once from the document's deepest nesting (for a
+    // subtree an upper bound)
     struct frame
     {
         const node* pos;
@@ -53,10 +54,43 @@ BasicJsonType materialize(const document_data& d, const node* n)
         bool object;
     };
 
+    // a scalar needs neither the handler nor the stack of open containers
+    switch (static_cast<value_t>(n->kind))
+    {
+        case value_t::string:
+            return BasicJsonType(string_t(d.str(*n), n->len));
+        case value_t::number_integer:
+            return BasicJsonType(static_cast<typename BasicJsonType::number_integer_t>(static_cast<std::int64_t>(integer_bits(*n))));
+        case value_t::number_unsigned:
+            return BasicJsonType(static_cast<typename BasicJsonType::number_unsigned_t>(integer_bits(*n)));
+        case value_t::number_float:
+            return BasicJsonType(float_value<typename BasicJsonType::number_float_t>(d, *n));
+        case value_t::boolean:
+            return BasicJsonType((n->flags & node_flags::is_true) != 0);
+        case value_t::null:
+            return BasicJsonType(nullptr);
+        case value_t::object:
+        case value_t::array:
+        case value_t::binary:
+        case value_t::discarded:
+        default:
+            break;
+    }
+
     BasicJsonType result;
     sax_t sax(result, true);
     const string_t no_token{};
-    std::vector<frame> open((std::max)(d.max_depth, std::size_t{1}));
+    // (shallow subtrees on the stack: a call for a small value then allocates only for the value)
+    std::array<frame, 16> shallow; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): filled as containers open
+    std::vector<frame> deep{};
+    frame* open = shallow.data();
+    std::size_t capacity = shallow.size();
+    if (d.max_depth > capacity)
+    {
+        deep.resize(d.max_depth);
+        open = deep.data();
+        capacity = deep.size();
+    }
     std::size_t depth = 0;
     for (;;)
     {
@@ -76,9 +110,13 @@ BasicJsonType materialize(const document_data& d, const node* n)
                 {
                     sax.start_array(n->len);
                 }
-                if (NLOHMANN_VIEW_UNLIKELY(depth == open.size()))
+                if (NLOHMANN_VIEW_UNLIKELY(depth == capacity))
                 {
-                    open.resize(depth * 2); // (deeper than the recorded nesting, e.g. after edits)
+                    // (deeper than the recorded nesting, e.g. after edits)
+                    deep.assign(open, open + depth);
+                    deep.resize(depth * 2);
+                    open = deep.data();
+                    capacity = deep.size();
                 }
                 open[depth++] = frame{nav::first(d, n), nav::end(d, n), object};
                 break;

include/nlohmann/detail/view/object_index.hpp

@@ -47,27 +47,96 @@ constexpr std::size_t index_max_displacement = 64;
 static_assert(document_data::index_min_members >= 2 * index_max_displacement,
               "a table must be much larger than the probe window of a lookup");
 
-/// hash of a key: its bytes, eight at a time, in a fixed byte order
+/// hash of a key: its bytes, eight at a time (in a fixed byte order), where the
+/// last word may overlap the one before (so that no loop over single bytes is
+/// needed); a key of fewer than eight bytes is read as two overlapping words of
+/// four or two bytes (in the native byte order: the hash is not stored, and the
+/// same on both sides of a lookup)
 inline std::uint64_t key_hash(const char* s, std::size_t n) noexcept
 {
     std::uint64_t h = 0x9E3779B97F4A7C15u * (n + 1);
     const auto* p = reinterpret_cast<const unsigned char*>(s); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
-    while (n >= 8)
-    {
-        h = (h ^ read_eight_bytes(p)) * 0xBF58476D1CE4E5B9u;
-        h ^= h >> 29u;
-        p += 8;
-        n -= 8;
-    }
     std::uint64_t w = 0;
-    for (std::size_t i = 0; i < n; ++i)
+    if (n >= 8)
     {
-        w |= static_cast<std::uint64_t>(p[i]) << (8u * i);
+        const unsigned char* const last = p + n - 8;
+        while (p < last)
+        {
+            h = (h ^ read_eight_bytes(p)) * 0xBF58476D1CE4E5B9u;
+            h ^= h >> 29u;
+            p += 8;
+        }
+        w = read_eight_bytes(last);
     }
-    h = (h ^ w) * 0x94D049BB133111EBu;
+    else if (n >= 4)
+    {
+        std::uint32_t head = 0;
+        std::uint32_t tail = 0;
+        std::memcpy(&head, p, 4);
+        std::memcpy(&tail, p + n - 4, 4);
+        w = static_cast<std::uint64_t>(head) | (static_cast<std::uint64_t>(tail) << 32u);
+    }
+    else if (n >= 2)
+    {
+        std::uint16_t head = 0;
+        std::uint16_t tail = 0;
+        std::memcpy(&head, p, 2);
+        std::memcpy(&tail, p + n - 2, 2);
+        w = static_cast<std::uint64_t>(head) | (static_cast<std::uint64_t>(tail) << 16u);
+    }
+    else if (n == 1)
+    {
+        w = p[0];
+    }
+    // (a product only carries bits upwards, and the slot is taken from the low
+    // bits: the high bits of the last word are folded down before the multiplication)
+    h = (h ^ w ^ (w >> 29u)) * 0x94D049BB133111EBu;
     return h ^ (h >> 31u);
 }
 
+/// whether the n <= 16 bytes at a and b are equal, as two overlapping words
+/// (the first and the last 8, 4, or 2 bytes) and without a call
+NLOHMANN_VIEW_ALWAYS_INLINE bool same_short_bytes(const unsigned char* a, const unsigned char* b, std::size_t n) noexcept
+{
+    if (n >= 8)
+    {
+        std::uint64_t x0 = 0;
+        std::uint64_t x1 = 0;
+        std::uint64_t y0 = 0;
+        std::uint64_t y1 = 0;
+        std::memcpy(&x0, a, 8);
+        std::memcpy(&x1, a + n - 8, 8);
+        std::memcpy(&y0, b, 8);
+        std::memcpy(&y1, b + n - 8, 8);
+        return x0 == y0 && x1 == y1;
+    }
+    if (n >= 4)
+    {
+        std::uint32_t x0 = 0;
+        std::uint32_t x1 = 0;
+        std::uint32_t y0 = 0;
+        std::uint32_t y1 = 0;
+        std::memcpy(&x0, a, 4);
+        std::memcpy(&x1, a + n - 4, 4);
+        std::memcpy(&y0, b, 4);
+        std::memcpy(&y1, b + n - 4, 4);
+        return x0 == y0 && x1 == y1;
+    }
+    if (n >= 2)
+    {
+        std::uint16_t x0 = 0;
+        std::uint16_t x1 = 0;
+        std::uint16_t y0 = 0;
+        std::uint16_t y1 = 0;
+        std::memcpy(&x0, a, 2);
+        std::memcpy(&x1, a + n - 2, 2);
+        std::memcpy(&y0, b, 2);
+        std::memcpy(&y1, b + n - 2, 2);
+        return x0 == y0 && x1 == y1;
+    }
+    return n == 0 || *a == *b;
+}
+
 /// build the table of a large object
 inline void build_object_index(document_data& d, node* obj)
 {
@@ -132,6 +201,27 @@ inline const node* find_indexed(const document_data& d, const node* obj, const c
     const document_data::object_index& ix = d.indexes[obj->extra - 1u];
     const std::uint32_t* const table = d.index_slots.data() + ix.start;
     std::size_t slot = conditional_static_cast<std::size_t>(key_hash(key, n)) & ix.mask;
+    const auto* const k_bytes = reinterpret_cast<const unsigned char*>(key); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
+    // (the loop for longer keys comes first, as the code is laid out; it is not the more common case)
+    if (NLOHMANN_VIEW_UNLIKELY(n <= 16))
+    {
+        // (up to 16 bytes, the bytes are compared as two words, without a call)
+        for (std::size_t distance = 0; distance <= index_max_displacement; ++distance)
+        {
+            const std::uint32_t s = table[slot];
+            if (s == 0)
+            {
+                return nullptr;
+            }
+            const node* const k = obj + s;
+            if (k->len == n && same_short_bytes(reinterpret_cast<const unsigned char*>(d.str(*k)), k_bytes, n)) // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
+            {
+                return k;
+            }
+            slot = (slot + 1) & ix.mask;
+        }
+        return nullptr;
+    }
     for (std::size_t distance = 0; distance <= index_max_displacement; ++distance)
     {
         const std::uint32_t s = table[slot];

include/nlohmann/detail/view/pointer.hpp

@@ -32,15 +32,51 @@ enum class pointer_mode
     contains,  ///< as basic_json::contains(): no exceptions at all
 };
 
-/// the outcome of reading an array index: detail::parse_array_index(), shared with json_pointer
-using index_status = detail::array_index_status;
+/// the outcome of reading an array index from a reference token
+enum class index_status
+{
+    ok,
+    leading_zero, ///< parse_error.106
+    not_number,   ///< parse_error.109
+    unresolved,   ///< out_of_range.404
+    too_large,    ///< out_of_range.410
+};
 
 /// reads an array index like json_pointer::array_index (RFC 6901, Sect. 4),
 /// but reports errors instead of throwing them
 template<typename StringType>
 index_status array_index(const StringType& s, std::size_t& idx) noexcept
 {
-    return detail::parse_array_index(s, idx);
+    // as basic_json: the empty token and "-" (past the last element) are
+    // unresolved rather than not a number
+    if (s.empty() || (s.size() == 1 && s[0] == '-'))
+    {
+        return index_status::unresolved;
+    }
+    if (s[0] < '0' || s[0] > '9')
+    {
+        return index_status::not_number;
+    }
+    if (s.size() > 1 && s[0] == '0')
+    {
+        return index_status::leading_zero;
+    }
+    std::uint64_t v = 0;
+    for (const auto c : s)
+    {
+        const auto d = static_cast<unsigned>(static_cast<unsigned char>(c)) - '0';
+        if (d > 9 || v > ((std::numeric_limits<std::uint64_t>::max)() - d) / 10)
+        {
+            return index_status::unresolved; // not a number, or beyond unsigned long long
+        }
+        v = (v * 10) + d;
+    }
+    if (v >= (std::numeric_limits<std::size_t>::max)()) // (std::size_t converts to std::uint64_t implicitly)
+    {
+        return index_status::too_large;
+    }
+    idx = static_cast<std::size_t>(v);
+    return index_status::ok;
 }
 
 /// throws the exception json_pointer::array_index throws for this status
@@ -50,15 +86,15 @@ template<typename StringType>
     switch (status)
     {
         case index_status::leading_zero:
-            throw_parse_error(exception_id::pointer_index_leading_zero, concat("array index '", s, "' must not begin with '0'"));
-        case index_status::not_a_number:
-            throw_parse_error(exception_id::pointer_index_not_a_number, concat("array index '", s, "' is not a number"));
-        case index_status::exceeds_size_type:
-            throw_out_of_range(exception_id::value_out_of_range, concat("array index ", s, " exceeds size_type")); // LCOV_EXCL_LINE
+            throw_parse_error(106, concat("array index '", s, "' must not begin with '0'"));
+        case index_status::not_number:
+            throw_parse_error(109, concat("array index '", s, "' is not a number"));
+        case index_status::too_large:
+            throw_out_of_range(410, concat("array index ", s, " exceeds size_type")); // LCOV_EXCL_LINE
         case index_status::unresolved:
         case index_status::ok:
         default:
-            throw_out_of_range(exception_id::pointer_unresolved, concat("unresolved reference token '", s, "'"));
+            throw_out_of_range(404, concat("unresolved reference token '", s, "'"));
     }
 }
 
@@ -83,7 +119,7 @@ View resolve_pointer(View cur, const Tokens& tokens, pointer_mode mode)
             {
                 if (mode == pointer_mode::checked)
                 {
-                    throw_key_not_found(token);
+                    throw_out_of_range(403, concat("key '", token, "' not found"));
                 }
                 return View();
             }
@@ -95,7 +131,7 @@ View resolve_pointer(View cur, const Tokens& tokens, pointer_mode mode)
             {
                 if (throwing)
                 {
-                    throw_array_index_past_end(cur.size());
+                    throw_out_of_range(402, concat("array index '-' (", std::to_string(cur.size()), ") is out of range"));
                 }
                 return View();
             }
@@ -103,7 +139,7 @@ View resolve_pointer(View cur, const Tokens& tokens, pointer_mode mode)
             const index_status status = array_index(token, idx);
             if (status != index_status::ok)
             {
-                const bool parse_error = status == index_status::leading_zero || status == index_status::not_a_number;
+                const bool parse_error = status == index_status::leading_zero || status == index_status::not_number;
                 if (throwing || (mode == pointer_mode::value && parse_error))
                 {
                     throw_array_index_error(status, token);
@@ -114,7 +150,7 @@ View resolve_pointer(View cur, const Tokens& tokens, pointer_mode mode)
             {
                 if (mode == pointer_mode::checked)
                 {
-                    throw_array_index_out_of_range(idx);
+                    throw_out_of_range(401, concat("array index ", std::to_string(idx), " is out of range"));
                 }
                 return View();
             }
@@ -124,7 +160,7 @@ View resolve_pointer(View cur, const Tokens& tokens, pointer_mode mode)
         {
             if (throwing)
             {
-                throw_out_of_range(exception_id::pointer_unresolved, concat("unresolved reference token '", token, "'"));
+                throw_out_of_range(404, concat("unresolved reference token '", token, "'"));
             }
             return View();
         }

include/nlohmann/detail/view/serializer.hpp

@@ -23,6 +23,7 @@
 #include <nlohmann/detail/view/macro_scope.hpp>
 #include <nlohmann/detail/view/node.hpp>
 #include <nlohmann/detail/view/number.hpp>
+#include <nlohmann/detail/view/simd.hpp>
 
 NLOHMANN_JSON_NAMESPACE_BEGIN
 namespace detail
@@ -80,13 +81,6 @@ class output_buffer
         put(s, N - 1);
     }
 
-    void put_repeated(char c, std::size_t n)
-    {
-        reserve(n);
-        std::memset(m_pos, c, n);
-        m_pos += n;
-    }
-
     /// the write position and the end of the writable space, for a writer
     /// that keeps the position in a local variable (set_cursor() hands it back)
     char* cursor() const noexcept
@@ -143,9 +137,27 @@ class output_buffer
 /// image_check::bounds can have others, which are then written unchanged).
 inline std::size_t plain_output_run(const unsigned char* s, std::size_t n) noexcept
 {
+    std::size_t i = 0;
+#if NLOHMANN_VIEW_SSE2
+    // (16 bytes per step; with NEON, the mask of a step costs more than it saves on the
+    // short runs that are typical here)
+    // NOLINTBEGIN(portability-simd-intrinsics)
+    for (; i + 16 <= n; i += 16)
+    {
+        const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(s + i))); // NOLINT(bugprone-casting-through-void)
+        // (a byte is below 0x20 if it is its own minimum with 0x1F)
+        const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
+                                             _mm_cmpeq_epi8(_mm_min_epu8(in, _mm_set1_epi8(0x1F)), in));
+        const auto mask = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
+        if (mask != 0)
+        {
+            return i + static_cast<std::size_t>(count_trailing_zeros(mask));
+        }
+    }
+    // NOLINTEND(portability-simd-intrinsics)
+#endif
     constexpr std::uint64_t ones = 0x0101010101010101ull;
     constexpr std::uint64_t high = 0x8080808080808080ull;
-    std::size_t i = 0;
     for (; i + 8 <= n; i += 8)
     {
         const std::uint64_t v = read_eight_bytes(s + i);
@@ -196,6 +208,70 @@ inline bool is_ascii_without_delete(const char* s, std::size_t n) noexcept
     return true;
 }
 
+#if NLOHMANN_VIEW_VECTOR
+/// the bytes of the 16 at p that ensure_ascii cannot copy (a quote, a
+/// backslash, a control character, DEL, or not ASCII; a signed compare with
+/// 0x20 finds control characters and non-ASCII bytes at once), one bit per
+/// byte (x86) or one nibble per byte (NEON)
+NLOHMANN_VIEW_ALWAYS_INLINE std::uint64_t ascii_stop_mask(const unsigned char* p) noexcept
+{
+#if NLOHMANN_VIEW_NEON
+    const uint8x16_t in = vld1q_u8(p);
+    const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))),
+                                        vorrq_u8(vceqq_u8(in, vdupq_n_u8(0x7F)), vcltq_s8(vreinterpretq_s8_u8(in), vdupq_n_s8(0x20))));
+    return vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0);
+#else
+    const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
+    const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
+                                         _mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8(0x7F)), _mm_cmplt_epi8(in, _mm_set1_epi8(0x20))));
+    return static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
+#endif
+}
+
+/// the width in bits of a byte in the mask of ascii_stop_mask()
+constexpr unsigned ascii_stop_bits() noexcept
+{
+    return NLOHMANN_VIEW_NEON ? 4u : 1u;
+}
+#endif
+
+/// As find_ascii_copyable_run(), 16 bytes per step. end is the end of the
+/// buffer that holds the n bytes at p (a short run is read as 16 bytes if
+/// they are in the buffer).
+inline std::size_t ascii_copyable_run(const unsigned char* p, std::size_t n, const unsigned char* end) noexcept
+{
+#if NLOHMANN_VIEW_VECTOR
+    if (n >= 16)
+    {
+        std::size_t i = 0;
+        for (; i + 16 <= n; i += 16)
+        {
+            const std::uint64_t mask = ascii_stop_mask(p + i);
+            if (mask != 0)
+            {
+                return i + (static_cast<std::size_t>(count_trailing_zeros(mask)) / ascii_stop_bits());
+            }
+        }
+        if (i == n)
+        {
+            return n;
+        }
+        // the last 16 bytes: those before i are known to be copyable
+        const std::uint64_t mask = ascii_stop_mask(p + n - 16);
+        return mask != 0 ? n - 16 + (static_cast<std::size_t>(count_trailing_zeros(mask)) / ascii_stop_bits()) : n;
+    }
+    if (end - p >= 16)
+    {
+        // (the bytes after the run are masked out)
+        const std::uint64_t mask = ascii_stop_mask(p) & ((std::uint64_t{1} << (n * ascii_stop_bits())) - 1u);
+        return mask != 0 ? static_cast<std::size_t>(count_trailing_zeros(mask)) / ascii_stop_bits() : n;
+    }
+#else
+    static_cast<void>(end);
+#endif
+    return find_ascii_copyable_run(p, n);
+}
+
 /// A stack that starts in a buffer of the caller (a local array) and moves to
 /// the heap (a vector of the caller) only when that is full, so that dumps of
 /// shallow documents need no allocation. The top is a pointer, as in
@@ -275,6 +351,39 @@ NLOHMANN_VIEW_ALWAYS_INLINE char* put_literal(char* w, const char (&s)[N]) noexc
     return w + N - 1;
 }
 
+/// the depth of the innermost open container, counted only for indented output
+template<bool Counted>
+struct depth_counter
+{
+    std::size_t value = 0;
+    void enter() noexcept
+    {
+        ++value;
+    }
+    std::size_t leave() noexcept
+    {
+        return --value;
+    }
+    std::size_t get() const noexcept
+    {
+        return value;
+    }
+};
+
+template<>
+struct depth_counter<false>
+{
+    void enter() noexcept {}
+    std::size_t leave() noexcept // NOLINT(readability-convert-member-functions-to-static): same interface as depth_counter<true>
+    {
+        return 0;
+    }
+    std::size_t get() const noexcept // NOLINT(readability-convert-member-functions-to-static): same interface as depth_counter<true>
+    {
+        return 0;
+    }
+};
+
 /// how the view's dump() writes a value
 struct dump_style
 {
@@ -311,139 +420,95 @@ class view_serializer
 
     void dump(const node* root)
     {
-        if (!m_style.pretty)
+        if (m_style.ensure_ascii)
         {
-            if (m_style.ensure_ascii)
-            {
-                dump_compact_ascii(root);
-            }
-            else if (m_style.source_numbers)
-            {
-                dump_compact<true, false>(root);
-            }
-            else
-            {
-                dump_compact<false, false>(root);
-            }
-            return;
+            dump_ascii(root);
         }
-        struct frame
+        else if (m_style.pretty)
         {
-            const node* pos; ///< next element, or key of the next member
-            const node* end;
-            bool object;
-            bool first;  ///< nothing written yet
-        };
-        std::array<frame, 32> buffer; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
-        std::vector<frame> heap;
-        small_stack<frame> stack(buffer.data(), buffer.size(), heap);
-        const node* n = root;
-        for (;;)
+            dump_pretty(root);
+        }
+        else if (m_style.source_numbers)
         {
-            // write the value at n
-            if (is_container(*n))
-            {
-                const bool object = n->kind == static_cast<std::uint8_t>(value_t::object);
-                if (n->len == 0)
-                {
-                    m_out.put(object ? "{}" : "[]", 2);
-                }
-                else
-                {
-                    m_out.put(object ? '{' : '[');
-                    stack.push_back(frame{nav::first(m_doc, n), nav::end(m_doc, n), object, true});
-                }
-            }
-            else
-            {
-                write_scalar(*n);
-            }
-
-            // go to the next value: close finished containers, then separate
-            for (;;)
-            {
-                if (stack.empty())
-                {
-                    m_out.finish();
-                    return;
-                }
-                if (stack.back().pos == stack.back().end)
-                {
-                    const bool object = stack.back().object;
-                    stack.pop_back();
-                    newline(stack.size());
-                    m_out.put(object ? '}' : ']');
-                    continue;
-                }
-                frame& f = stack.back();
-                if (!f.first)
-                {
-                    m_out.put(',');
-                }
-                f.first = false;
-                newline(stack.size());
-                if (f.object)
-                {
-                    write_string(*f.pos);
-                    if (m_style.pretty)
-                    {
-                        m_out.put(": ");
-                    }
-                    else
-                    {
-                        m_out.put(':');
-                    }
-                    n = nav::value(f.pos + 1);
-                    f.pos = document_data::after(f.pos + 1);
-                }
-                else
-                {
-                    n = nav::value(f.pos);
-                    f.pos = document_data::after(f.pos);
-                }
-                break;
-            }
+            dump_text<true, false, false>(root);
+        }
+        else
+        {
+            dump_text<false, false, false>(root);
         }
     }
 
   private:
+    /// dump_text() with ensure_ascii, out of line: the walk without it is
+    /// faster when it is not merged with this one into dump()
+    NLOHMANN_VIEW_NOINLINE void dump_ascii(const node* root)
+    {
+        if (m_style.pretty)
+        {
+            if (m_style.source_numbers)
+            {
+                dump_text<true, true, true>(root);
+            }
+            else
+            {
+                dump_text<false, true, true>(root);
+            }
+        }
+        else if (m_style.source_numbers)
+        {
+            dump_text<true, false, true>(root);
+        }
+        else
+        {
+            dump_text<false, false, true>(root);
+        }
+    }
+
+    /// dump_text() with indentation, out of line: the compact walk is faster
+    /// when it is not merged with the indented ones into dump()
+    NLOHMANN_VIEW_NOINLINE void dump_pretty(const node* root)
+    {
+        if (m_style.source_numbers)
+        {
+            dump_text<true, true, false>(root);
+        }
+        else
+        {
+            dump_text<false, true, false>(root);
+        }
+    }
+
     /// a string that is not a plain string of the source (decoded, or written
     /// by an edit), without ensure_ascii: runs without characters to escape
-    /// are copied
-    NLOHMANN_VIEW_NOINLINE void write_decoded(const node& n)
+    /// are copied. If validate is set, the string must be valid UTF-8: the
+    /// runs are found by the lexer's scan, which validates the sequences, and
+    /// invalid UTF-8 throws what basic_json::dump() throws. (The compact output
+    /// writes the bytes of a damaged image unchanged, and scans without
+    /// validating them.)
+    NLOHMANN_VIEW_NOINLINE void write_decoded(const node& n, bool validate)
     {
         const auto* const s = reinterpret_cast<const unsigned char*>(m_doc.str(n)); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
         m_out.put('"');
         for (std::size_t i = 0; i < n.len;)
         {
-            const std::size_t run = plain_output_run(s + i, n.len - i);
+            const std::size_t run = validate ? string_bulk_run(s + i, n.len - i) : plain_output_run(s + i, n.len - i);
             if (run != 0)
             {
                 m_out.put(reinterpret_cast<const char*>(s + i), run); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
                 i += run;
                 continue;
             }
+            if (validate && s[i] >= 0x80)
+            {
+                invalid_utf8(s, n.len); // (the scan stopped at ill-formed UTF-8)
+            }
             write_codepoint<false>(s[i], s + i, 1); // a quote, a backslash, or a control character
             ++i;
         }
         m_out.put('"');
     }
 
-    /// dump_compact() with ensure_ascii, out of line: the walk without it is
-    /// faster when it is not merged with this one into dump()
-    NLOHMANN_VIEW_NOINLINE void dump_compact_ascii(const node* root)
-    {
-        if (m_style.source_numbers)
-        {
-            dump_compact<true, true>(root);
-        }
-        else
-        {
-            dump_compact<false, true>(root);
-        }
-    }
-
-    /// whether dump_compact() copies a string or key from the source: one of
+    /// whether dump_text() copies a string or key from the source: one of
     /// the source, and with ensure_ascii one without bytes >= 0x7F (an overload
     /// each, so that the walk without ensure_ascii is compiled exactly as
     /// before: with GCC, a condition that only folds away later changed its
@@ -458,13 +523,155 @@ class view_serializer
         return (n.flags & node_flags::storage) == 0 && is_ascii_without_delete(src + n.off, n.len);
     }
 
-    /// a string with ensure_ascii (dump_compact()), out of line
-    NLOHMANN_VIEW_NOINLINE void write_ascii_string(const node& n)
+    /*!
+    @brief a string with ensure_ascii: the characters that are not printable ASCII as \\uXXXX
+    (a character beyond U+FFFF as a surrogate pair); strings of a document
+    are valid UTF-8, except in a damaged image loaded with image_check::bounds,
+    for which this throws what basic_json::dump() throws for the string
+    */
+    NLOHMANN_VIEW_NOINLINE void write_ascii(const node& n)
     {
-        write_string(n);
+        const auto* const s = reinterpret_cast<const unsigned char*>(m_doc.str(n)); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
+        const std::size_t size = n.len;
+        char* w = m_out.cursor();
+        char* lim = m_out.limit();
+        // room for k bytes
+        const auto room = [&](std::size_t k)
+        {
+            const auto left = static_cast<std::size_t>(lim - w);
+            if (NLOHMANN_VIEW_UNLIKELY(left < k))
+            {
+                m_out.set_cursor(w);
+                m_out.reserve(k);
+                w = m_out.cursor();
+                lim = m_out.limit();
+            }
+        };
+        static constexpr const char* hex = "0123456789abcdef";
+        // \\uXXXX
+        const auto u_escape = [&](std::uint32_t u)
+        {
+            w[0] = '\\';
+            w[1] = 'u';
+            w[2] = hex[(u >> 12u) & 0xFu];
+            w[3] = hex[(u >> 8u) & 0xFu];
+            w[4] = hex[(u >> 4u) & 0xFu];
+            w[5] = hex[u & 0xFu];
+            w += 6;
+        };
+        room(1);
+        *w++ = '"';
+        std::size_t i = 0;
+        // (a string of the source starts with a run that has nothing to escape, 16 bytes per step)
+        const std::size_t prefix = (n.flags & node_flags::storage) == 0 ? ascii_copyable_run(s, size, reinterpret_cast<const unsigned char*>(m_doc.src + m_doc.size)) : 0; // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
+        if (prefix != 0)
+        {
+            room(prefix);
+            std::memcpy(w, s, prefix);
+            w += prefix;
+            i = prefix;
+        }
+        while (i < size)
+        {
+            // (a character that needs an escape sequence is not worth a call)
+            const std::size_t run = is_ascii_copyable(s[i]) ? find_ascii_copyable_run(s + i, size - i) : 0;
+            if (run != 0)
+            {
+                room(run);
+                std::memcpy(w, s + i, run);
+                w += run;
+                i += run;
+                continue;
+            }
+            room(12);
+            if (s[i] >= 0x80)
+            {
+                const unsigned char lead = s[i];
+                // (the two-byte sequences of Latin text and the three-byte sequences of most other
+                // text, with continuation bytes of any value: the ranges that are restricted (E0,
+                // ED, and the sequences of four bytes) go through validate_one_utf8())
+                if (lead >= 0xE1 && lead <= 0xEF && lead != 0xED && size - i >= 3 && (s[i + 1] & 0xC0u) == 0x80 && (s[i + 2] & 0xC0u) == 0x80)
+                {
+                    u_escape(((lead & 0x0Fu) << 12u) | ((s[i + 1] & 0x3Fu) << 6u) | (s[i + 2] & 0x3Fu));
+                    i += 3;
+                    continue;
+                }
+                if (lead >= 0xC2 && lead <= 0xDF && size - i >= 2 && (s[i + 1] & 0xC0u) == 0x80)
+                {
+                    u_escape(((lead & 0x1Fu) << 6u) | (s[i + 1] & 0x3Fu));
+                    i += 2;
+                    continue;
+                }
+                const std::size_t len = validate_one_utf8(s + i, size - i);
+                if (NLOHMANN_VIEW_UNLIKELY(len == 0))
+                {
+                    m_out.set_cursor(w);
+                    invalid_utf8(s, size);
+                    return; // LCOV_EXCL_LINE (invalid_utf8 throws)
+                }
+                std::uint32_t codepoint = s[i] & (0xFFu >> (len + 1));
+                for (std::size_t k = 1; k < len; ++k)
+                {
+                    codepoint = (codepoint << 6u) | (s[i + k] & 0x3Fu);
+                }
+                i += len;
+                if (codepoint <= 0xFFFF)
+                {
+                    u_escape(codepoint);
+                }
+                else
+                {
+                    u_escape(0xD7C0u + (codepoint >> 10u));
+                    u_escape(0xDC00u + (codepoint & 0x3FFu));
+                }
+                continue;
+            }
+            // a quote, a backslash, or a control character (or DEL)
+            const unsigned char c = s[i++];
+            char escape = 0;
+            switch (c)
+            {
+                case 0x08:
+                    escape = 'b';
+                    break;
+                case 0x09:
+                    escape = 't';
+                    break;
+                case 0x0A:
+                    escape = 'n';
+                    break;
+                case 0x0C:
+                    escape = 'f';
+                    break;
+                case 0x0D:
+                    escape = 'r';
+                    break;
+                case 0x22:
+                    escape = '"';
+                    break;
+                case 0x5C:
+                    escape = '\\';
+                    break;
+                default:
+                    break;
+            }
+            if (escape != 0)
+            {
+                w[0] = '\\';
+                w[1] = escape;
+                w += 2;
+            }
+            else
+            {
+                u_escape(c);
+            }
+        }
+        room(1);
+        *w++ = '"';
+        m_out.set_cursor(w);
     }
 
-    /// the copies of dump_compact() that are not fixed-size moves (long
+    /// the copies of dump_text() that are not fixed-size moves (long
     /// strings, or near the end of the source); out of line, so that the
     /// compiler does not merge the fixed-size moves into this call
     NLOHMANN_VIEW_NOINLINE static void copy_long(char* to, const char* from, std::size_t n) noexcept
@@ -481,7 +688,7 @@ class view_serializer
     }
 
     /*!
-    @brief the compact output (the default dump(), or with ensure_ascii)
+    @brief the output, compact (the default dump()) or indented, with or without ensure_ascii
 
     The same walk as dump(), with the write position in a local variable
     (stores through char pointers would otherwise force a reload of the
@@ -489,11 +696,11 @@ class view_serializer
     the source copied by fixed-size moves of 32 bytes where the source has
     that many bytes left, instead of a library call per token. The buffer
     keeps 64 bytes of slack for the overshoot. With ensure_ascii (Ascii), a
-    string of the source is copied only if it has no byte >= 0x7F (see
-    is_ascii_without_delete()); any other string is written by write_string().
+    string of the source is copied only if it has nothing to escape (see
+    copied()); any other string is written by write_ascii().
     */
-    template<bool SourceNumbers, bool Ascii>
-    void dump_compact(const node* root)
+    template<bool SourceNumbers, bool Pretty, bool Ascii>
+    void dump_text(const node* root)
     {
         struct frame
         {
@@ -508,6 +715,16 @@ class view_serializer
         const char* const src_end = src + m_doc.size;
         char* w = m_out.cursor();
         char* lim = m_out.limit();
+        // (indented output) a line break and 63 indentation characters, from
+        // which the line break of a level of up to 63 characters is a copy
+        std::array<char, 64> ruler; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): filled below if used
+        if (Pretty)
+        {
+            ruler.fill(m_style.indent_char);
+            ruler[0] = '\n';
+        }
+        const std::size_t indent = m_style.indent;
+        depth_counter<Pretty> level; // depth of the innermost open container (indented output)
         // room for n bytes and the slack
         const auto room = [&](std::size_t n)
         {
@@ -539,6 +756,22 @@ class view_serializer
             }
             w += n;
         };
+        // (indented output) the line break and the indentation of a level
+        const auto newline = [&](std::size_t depth)
+        {
+            const std::size_t count = 1 + (depth * indent);
+            room(count);
+            if (count <= ruler.size())
+            {
+                std::memcpy(w, ruler.data(), ruler.size());
+            }
+            else
+            {
+                *w = '\n';
+                std::memset(w + 1, m_style.indent_char, count - 1);
+            }
+            w += count;
+        };
         // a string that is not a plain string of the source (out of line, so
         // that the cursor stays in a register here)
         const auto escaped = [&](const node & n)
@@ -546,11 +779,11 @@ class view_serializer
             m_out.set_cursor(w);
             if (enabled(Ascii))
             {
-                write_ascii_string(n);
+                write_ascii(n);
             }
             else
             {
-                write_decoded(n);
+                write_decoded(n, Pretty);
             }
             w = m_out.cursor();
             lim = m_out.limit();
@@ -651,6 +884,7 @@ class view_serializer
                     {
                         *w++ = object ? '{' : '[';
                         stack.push_back(cur);
+                        level.enter();
                         if (Editable)
                         {
                             cur = frame{nav::first(m_doc, n), nav::end(m_doc, n), object};
@@ -693,6 +927,10 @@ class view_serializer
                     {
                         break;
                     }
+                    if (Pretty)
+                    {
+                        newline(level.leave());
+                    }
                     room(1);
                     *w++ = cur.object ? '}' : ']';
                     cur = stack.back();
@@ -701,13 +939,17 @@ class view_serializer
                 room(1);
                 *w++ = ',';
             }
+            if (Pretty)
+            {
+                newline(level.get());
+            }
             const node* const at = Editable ? cur.pos : n;
             if (cur.object)
             {
                 const node& key = *at;
                 if (copied(key, src, std::integral_constant<bool, Ascii>()))
                 {
-                    room(key.len + 3);
+                    room(key.len + (Pretty ? 4 : 3));
                     *w++ = '"';
                     copy(src + key.off, key.len);
                     w[0] = '"';
@@ -717,9 +959,13 @@ class view_serializer
                 else
                 {
                     escaped(key);
-                    room(1);
+                    room(Pretty ? 2 : 1);
                     *w++ = ':';
                 }
+                if (Pretty)
+                {
+                    *w++ = ' ';
+                }
                 if (Editable)
                 {
                     n = nav::value(at + 1);
@@ -738,69 +984,6 @@ class view_serializer
         }
     }
 
-    void newline(std::size_t level)
-    {
-        if (m_style.pretty)
-        {
-            m_out.put('\n');
-            m_out.put_repeated(m_style.indent_char, level * m_style.indent);
-        }
-    }
-
-    void write_scalar(const node& n)
-    {
-        switch (static_cast<value_t>(n.kind))
-        {
-            case value_t::null:
-                m_out.put("null");
-                break;
-            case value_t::boolean:
-                if ((n.flags & node_flags::is_true) != 0)
-                {
-                    m_out.put("true");
-                }
-                else
-                {
-                    m_out.put("false");
-                }
-                break;
-            case value_t::string:
-                write_string(n);
-                break;
-            case value_t::number_integer:
-            case value_t::number_unsigned:
-            {
-                const char* const token = m_doc.str(n);
-                const std::uint32_t len = number_length(n);
-                if (!m_style.source_numbers && len == 2 && token[0] == '-' && token[1] == '0')
-                {
-                    m_out.put('0'); // parse() reads -0 as the integer 0
-                }
-                else
-                {
-                    m_out.put(token, len);
-                }
-                break;
-            }
-            case value_t::number_float:
-                if (m_style.source_numbers && (n.flags & node_flags::storage) != node_flags::edited)
-                {
-                    m_out.put(m_doc.str(n), n.len); // (a float set by an edit is written as with shortest)
-                }
-                else
-                {
-                    write_float_node(n);
-                }
-                break;
-            case value_t::object:    // LCOV_EXCL_LINE (containers are written by dump())
-            case value_t::array:     // LCOV_EXCL_LINE
-            case value_t::binary:    // LCOV_EXCL_LINE (not in a document)
-            case value_t::discarded: // LCOV_EXCL_LINE
-            default:                 // LCOV_EXCL_LINE
-                break;               // LCOV_EXCL_LINE
-        }
-    }
-
     /// a float node as dump() writes it
     void write_float_node(const node& n)
     {
@@ -890,89 +1073,27 @@ class view_serializer
             m_out.put("null");
             return;
         }
-        write_float(x, std::integral_constant<bool, ::nlohmann::detail::has_native_float_format<number_float_t>::value>());
+        static constexpr bool is_ieee_single_or_double
+            = std::numeric_limits<number_float_t>::is_iec559
+              && ((std::numeric_limits<number_float_t>::digits == 24 && std::numeric_limits<number_float_t>::max_exponent == 128)
+                  || (std::numeric_limits<number_float_t>::digits == 53 && std::numeric_limits<number_float_t>::max_exponent == 1024));
+        write_float(x, std::integral_constant<bool, is_ieee_single_or_double>());
     }
 
-    void write_float(number_float_t x, std::true_type /*has_native_float_format*/)
+    void write_float(number_float_t x, std::true_type /*is_ieee_single_or_double*/)
     {
         std::array<char, 64> buf{};
         const char* const end = ::nlohmann::detail::to_chars(buf.data(), buf.data() + buf.size(), x);
         m_out.put(buf.data(), static_cast<std::size_t>(end - buf.data()));
     }
 
-    void write_float(number_float_t x, std::false_type /*has_native_float_format*/)
+    void write_float(number_float_t x, std::false_type /*is_ieee_single_or_double*/)
     {
         // other types (e.g. long double) are rare: the library writes them
         const string_t s = BasicJsonType(x).dump();
         m_out.put(s.data(), s.size());
     }
 
-    void write_string(const node& n)
-    {
-        const char* const s = m_doc.str(n);
-        m_out.put('"');
-        if ((n.flags & node_flags::storage) == 0 && (!m_style.ensure_ascii || is_ascii_without_delete(s, n.len)))
-        {
-            // a string of the source without escape sequences has nothing to
-            // escape (with ensure_ascii, if it is printable ASCII)
-            m_out.put(s, n.len);
-        }
-        else if (m_style.ensure_ascii)
-        {
-            write_escaped<true>(reinterpret_cast<const unsigned char*>(s), n.len); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
-        }
-        else
-        {
-            write_escaped<false>(reinterpret_cast<const unsigned char*>(s), n.len); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
-        }
-        m_out.put('"');
-    }
-
-    /// as serializer::dump_escaped(); strings of a document are valid UTF-8,
-    /// except in a damaged image loaded with image_check::bounds, for which
-    /// this throws what basic_json::dump() throws for the string
-    template<bool EnsureAscii>
-    void write_escaped(const unsigned char* s, std::size_t n)
-    {
-        std::size_t i = 0;
-        while (i < n)
-        {
-            std::size_t run = 0;
-            if (!EnsureAscii)
-            {
-                run = string_bulk_run(s + i, n - i);
-            }
-            else if (is_ascii_copyable(s[i]))
-            {
-                run = find_ascii_copyable_run(s + i, n - i);
-            }
-            if (run != 0)
-            {
-                m_out.put(reinterpret_cast<const char*>(s + i), run); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
-                i += run;
-                continue;
-            }
-            std::uint32_t codepoint = s[i];
-            std::size_t len = 1;
-            if (codepoint >= 0x80)
-            {
-                len = validate_one_utf8(s + i, n - i);
-                if (NLOHMANN_VIEW_UNLIKELY(len == 0))
-                {
-                    invalid_utf8(s, n);
-                    return;
-                }
-                codepoint &= 0xFFu >> (len + 1);
-                for (std::size_t k = 1; k < len; ++k)
-                {
-                    codepoint = (codepoint << 6u) | (s[i + k] & 0x3Fu);
-                }
-            }
-            write_codepoint<EnsureAscii>(codepoint, s + i, len);
-            i += len;
-        }
-    }
-
     /// throw what basic_json::dump() throws for a string that is not valid UTF-8
     NLOHMANN_VIEW_NOINLINE static void invalid_utf8(const unsigned char* s, std::size_t n)
     {
@@ -1027,8 +1148,8 @@ class view_serializer
 
     void write_u_escape(std::uint32_t u)
     {
-        std::array<char, 6> e = {{'\\', 'u', 0, 0, 0, 0}};
-        hex4_lower(static_cast<std::uint16_t>(u), &e[2]);
+        static constexpr const char* hex = "0123456789abcdef";
+        const std::array<char, 6> e = {{'\\', 'u', hex[(u >> 12u) & 0xFu], hex[(u >> 8u) & 0xFu], hex[(u >> 4u) & 0xFu], hex[u & 0xFu]}};
         m_out.put(e.data(), e.size());
     }
 

include/nlohmann/detail/view/simd.hpp

@@ -116,6 +116,34 @@ NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned
     }
     return p;
 }
+
+/*!
+@brief copy 16 bytes from p to w, and find the first quote, backslash,
+control character, or non-ASCII byte among them
+
+The bytes are copied whether or not they are all plain (the caller has room
+for 16 bytes and keeps only the plain ones).
+
+@return the offset of that byte, or 16 if there is none
+*/
+NLOHMANN_VIEW_ALWAYS_INLINE unsigned copy_plain_run16(const unsigned char* p, char* w) noexcept
+{
+#if NLOHMANN_VIEW_NEON
+    const uint8x16_t in = vld1q_u8(p);
+    vst1q_u8(reinterpret_cast<std::uint8_t*>(w), in); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
+    const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))),
+                                        vcltq_s8(vreinterpretq_s8_u8(in), vdupq_n_s8(0x20)));
+    const std::uint64_t bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0);
+    return bits != 0 ? static_cast<unsigned>(count_trailing_zeros(bits) >> 2u) : 16u;
+#else
+    const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
+    _mm_storeu_si128(static_cast<__m128i*>(static_cast<void*>(w)), in); // NOLINT(bugprone-casting-through-void)
+    const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
+                                         _mm_cmplt_epi8(in, _mm_set1_epi8(0x20)));
+    const auto bits = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
+    return bits != 0 ? static_cast<unsigned>(count_trailing_zeros(bits)) : 16u;
+#endif
+}
 #endif
 
 #if NLOHMANN_VIEW_SSSE3_DISPATCH

include/nlohmann/detail/view/value.hpp

@@ -60,7 +60,7 @@ NLOHMANN_VIEW_ALWAYS_INLINE T arithmetic_value(const document_data& d, const nod
         case value_t::binary:
         case value_t::discarded:
         default:
-            throw_type_error(exception_id::type_mismatch, "type must be number, but is ", value_type_name(static_cast<value_t>(n.kind)));
+            throw_type_error(302, "type must be number, but is ", value_type_name(static_cast<value_t>(n.kind)));
     }
 }
 
@@ -70,7 +70,7 @@ std::vector<U, A> vector_value(const View& v)
 {
     if (NLOHMANN_VIEW_UNLIKELY(!v.is_array()))
     {
-        throw_type_error(exception_id::type_mismatch, "type must be array, but is ", v.type_name());
+        throw_type_error(302, "type must be array, but is ", v.type_name());
     }
     std::vector<U, A> r;
     r.reserve(v.size());
@@ -88,7 +88,7 @@ Map map_value(const View& v)
 {
     if (NLOHMANN_VIEW_UNLIKELY(!v.is_object()))
     {
-        throw_type_error(exception_id::type_mismatch, "type must be object, but is ", v.type_name());
+        throw_type_error(302, "type must be object, but is ", v.type_name());
     }
     Map r;
     for (auto it = v.begin(); it != v.end(); ++it)

include/nlohmann/json_view.hpp

@@ -30,6 +30,7 @@
 #include <array> // array
 #include <cstddef> // nullptr_t, size_t // IWYU pragma: keep
 #include <cstdint> // uint8_t, uint32_t
+#include <cstdio> // FILE
 #include <cstring> // memcpy, strlen
 #include <istream> // istream // IWYU pragma: keep
 #include <iterator> // distance, input_iterator_tag, iterator_traits
@@ -267,7 +268,7 @@ class basic_json_view
             {
                 return {};
             }
-            detail::view::throw_type_error(detail::exception_id::subscript_wrong_type, "cannot use operator[] with a string argument with ", type_name());
+            detail::view::throw_type_error(305, "cannot use operator[] with a string argument with ", type_name());
         }
         return lookup(key);
     }
@@ -293,9 +294,9 @@ class basic_json_view
             {
                 return {};
             }
-            detail::view::throw_type_error(detail::exception_id::subscript_wrong_type, "cannot use operator[] with a numeric argument with ", type_name());
+            detail::view::throw_type_error(305, "cannot use operator[] with a numeric argument with ", type_name());
         }
-        return idx < m_node->len ? basic_json_view(m_doc, navigation::value(detail::view::element_at<Editable>(*m_doc, m_node, idx))) : basic_json_view();
+        return idx < m_node->len ? element(idx) : basic_json_view();
     }
 
     /// any other integer type (int, unsigned, long, std::int64_t, ...; a
@@ -326,12 +327,12 @@ class basic_json_view
     {
         if (NLOHMANN_VIEW_UNLIKELY(!is_object()))
         {
-            detail::view::throw_type_error(detail::exception_id::at_wrong_type, "cannot use at() with ", type_name());
+            detail::view::throw_type_error(304, "cannot use at() with ", type_name());
         }
         const basic_json_view r = lookup(key);
         if (NLOHMANN_VIEW_UNLIKELY(r.is_discarded()))
         {
-            detail::view::throw_key_not_found(std::string(key.data(), key.size()));
+            detail::view::throw_out_of_range(403, detail::concat("key '", std::string(key.data(), key.size()), "' not found"));
         }
         return r;
     }
@@ -352,13 +353,13 @@ class basic_json_view
     {
         if (NLOHMANN_VIEW_UNLIKELY(!is_array()))
         {
-            detail::view::throw_type_error(detail::exception_id::at_wrong_type, "cannot use at() with ", type_name());
+            detail::view::throw_type_error(304, "cannot use at() with ", type_name());
         }
         if (NLOHMANN_VIEW_UNLIKELY(idx >= m_node->len))
         {
-            detail::view::throw_array_index_out_of_range(idx);
+            detail::view::throw_out_of_range(401, detail::concat("array index ", std::to_string(idx), " is out of range"));
         }
-        return basic_json_view(m_doc, navigation::value(detail::view::element_at<Editable>(*m_doc, m_node, idx)));
+        return element(idx);
     }
 
     /// any other integer type, see operator[]; negative values are out of
@@ -384,7 +385,7 @@ class basic_json_view
     {
         if (NLOHMANN_VIEW_UNLIKELY(!is_object()))
         {
-            detail::view::throw_type_error(detail::exception_id::value_wrong_type, "cannot use value() with ", type_name());
+            detail::view::throw_type_error(306, "cannot use value() with ", type_name());
         }
         const basic_json_view r = lookup(key);
         return r.is_discarded() ? default_value : r.template get<T>();
@@ -392,7 +393,13 @@ class basic_json_view
 
     string_t value(string_view_t key, const char* default_value) const
     {
-        return value(key, string_t(default_value));
+        if (NLOHMANN_VIEW_UNLIKELY(!is_object()))
+        {
+            detail::view::throw_type_error(306, "cannot use value() with ", type_name());
+        }
+        // (the default is made only if there is no such member)
+        const basic_json_view r = lookup(key);
+        return r.is_discarded() ? string_t(default_value) : r.template get<string_t>();
     }
 
     /// the value a JSON pointer refers to converted to T, or the default
@@ -403,7 +410,7 @@ class basic_json_view
     {
         if (NLOHMANN_VIEW_UNLIKELY(!is_structured()))
         {
-            detail::view::throw_type_error(detail::exception_id::value_wrong_type, "cannot use value() with ", type_name());
+            detail::view::throw_type_error(306, "cannot use value() with ", type_name());
         }
         const basic_json_view r = detail::view::resolve_pointer(*this, detail::json_pointer_access::reference_tokens(ptr), detail::view::pointer_mode::value);
         return r.is_discarded() ? default_value : r.template get<T>();
@@ -411,7 +418,12 @@ class basic_json_view
 
     string_t value(const json_pointer& ptr, const char* default_value) const
     {
-        return value(ptr, string_t(default_value));
+        if (NLOHMANN_VIEW_UNLIKELY(!is_structured()))
+        {
+            detail::view::throw_type_error(306, "cannot use value() with ", type_name());
+        }
+        const basic_json_view r = detail::view::resolve_pointer(*this, detail::json_pointer_access::reference_tokens(ptr), detail::view::pointer_mode::value);
+        return r.is_discarded() ? string_t(default_value) : r.template get<string_t>();
     }
 
     /// the first element or member value; a primitive value itself. Throws
@@ -421,7 +433,7 @@ class basic_json_view
         const iterator it = begin();
         if (NLOHMANN_VIEW_UNLIKELY(it == end()))
         {
-            detail::view::throw_invalid_iterator(detail::exception_id::iterator_value_unavailable, "cannot get value");
+            detail::view::throw_invalid_iterator(214, "cannot get value");
         }
         return *it;
     }
@@ -433,7 +445,8 @@ class basic_json_view
     {
         if (is_structured() && m_node->len != 0)
         {
-            return basic_json_view(m_doc, navigation::value(detail::view::last_child<Editable>(*m_doc, m_node) + (is_object() ? 1 : 0)));
+            return unedited() ? basic_json_view(m_doc, detail::view::last_child<false>(*m_doc, m_node) + (is_object() ? 1 : 0))
+                   : basic_json_view(m_doc, navigation::value(detail::view::last_child<Editable>(*m_doc, m_node) + (is_object() ? 1 : 0)));
         }
         return front();
     }
@@ -450,7 +463,7 @@ class basic_json_view
         {
             return end();
         }
-        const node* const k = detail::view::find_member<Editable>(*m_doc, m_node, key.data(), key.size());
+        const node* const k = find_key(key);
         return k != nullptr ? iterator(m_doc, k, true) : end();
     }
 
@@ -467,7 +480,7 @@ class basic_json_view
     /// whether this is an object with a member with this key
     bool contains(string_view_t key) const
     {
-        return is_object() && detail::view::find_member<Editable>(*m_doc, m_node, key.data(), key.size()) != nullptr;
+        return is_object() && find_key(key) != nullptr;
     }
 
     bool contains(const char* key) const
@@ -573,7 +586,7 @@ class basic_json_view
     {
         if (NLOHMANN_VIEW_UNLIKELY(!is_string()))
         {
-            detail::view::throw_type_error(detail::exception_id::type_mismatch, "type must be string, but is ", type_name());
+            detail::view::throw_type_error(302, "type must be string, but is ", type_name());
         }
         return {m_doc->str(*m_node), m_node->len};
     }
@@ -585,7 +598,7 @@ class basic_json_view
     {
         if (NLOHMANN_VIEW_UNLIKELY(!is_number()))
         {
-            detail::view::throw_type_error(detail::exception_id::type_mismatch, "type must be number, but is ", type_name());
+            detail::view::throw_type_error(302, "type must be number, but is ", type_name());
         }
         return {m_doc->str(*m_node), detail::view::number_length(*m_node)};
     }
@@ -627,7 +640,14 @@ class basic_json_view
         // the compact writer keeps 64 bytes of slack, so that it does not grow
         // the buffer just before the end
         const std::size_t estimate = source_extent() + (style.pretty ? source_extent() / 2 : 0) + 160;
-        detail::view::view_serializer<BasicJsonType, Editable>(*m_doc, out, estimate, style).dump(m_node);
+        if (unedited())
+        {
+            detail::view::view_serializer<BasicJsonType, false>(*m_doc, out, estimate, style).dump(m_node);
+        }
+        else
+        {
+            detail::view::view_serializer<BasicJsonType, Editable>(*m_doc, out, estimate, style).dump(m_node);
+        }
         return out;
     }
 
@@ -707,7 +727,7 @@ class basic_json_view
         {
             return BasicJsonType(value_t::discarded);
         }
-        return detail::view::materialize<BasicJsonType, Editable>(*m_doc, m_node);
+        return unedited() ? detail::view::materialize<BasicJsonType, false>(*m_doc, m_node) : detail::view::materialize<BasicJsonType, Editable>(*m_doc, m_node);
     }
 
     /// byte offset of this value in the source text (for strings: of the
@@ -785,17 +805,57 @@ class basic_json_view
     /// (object required)
     NLOHMANN_VIEW_ALWAYS_INLINE basic_json_view lookup(string_view_t key) const noexcept
     {
+        if (unedited())
+        {
+            const node* const k = detail::view::find_member<false>(*m_doc, m_node, key.data(), key.size());
+            return k != nullptr ? basic_json_view(m_doc, k + 1) : basic_json_view();
+        }
         const node* const k = detail::view::find_member<Editable>(*m_doc, m_node, key.data(), key.size());
         return k != nullptr ? basic_json_view(m_doc, navigation::value(k + 1)) : basic_json_view();
     }
 
+    /// whether the document has not been edited yet: until the first edit, its index is walked like that of a
+    /// read-only document (nothing is moved or linked), which is cheaper
+    NLOHMANN_VIEW_ALWAYS_INLINE bool unedited() const noexcept
+    {
+        return !navigation::editable() || m_doc->edits == nullptr;
+    }
+
+    /// the member of this object with @a key (the last one if the key occurs more than once), or nullptr
+    NLOHMANN_VIEW_ALWAYS_INLINE const node* find_key(string_view_t key) const
+    {
+        return find_key(key, std::integral_constant<bool, Editable> {});
+    }
+
+    NLOHMANN_VIEW_ALWAYS_INLINE const node* find_key(string_view_t key, std::true_type /*editable*/) const
+    {
+        // (an editable document without edits is looked up as one that cannot be edited)
+        return unedited() ? detail::view::find_member<false>(*m_doc, m_node, key.data(), key.size())
+               : detail::view::find_member<true>(*m_doc, m_node, key.data(), key.size());
+    }
+
+    NLOHMANN_VIEW_ALWAYS_INLINE const node* find_key(string_view_t key, std::false_type /*editable*/) const
+    {
+        return detail::view::find_member<false>(*m_doc, m_node, key.data(), key.size());
+    }
+
+    /// the element of an array at an index below its size
+    NLOHMANN_VIEW_ALWAYS_INLINE basic_json_view element(size_type idx) const
+    {
+        if (unedited())
+        {
+            return basic_json_view(m_doc, detail::view::element_at<false>(*m_doc, m_node, idx));
+        }
+        return basic_json_view(m_doc, navigation::value(detail::view::element_at<Editable>(*m_doc, m_node, idx)));
+    }
+
     // --- get() dispatch ---
 
     bool get_impl(detail::view::value_tag<bool> /*unused*/, detail::priority_tag<2> /*unused*/) const
     {
         if (NLOHMANN_VIEW_UNLIKELY(!is_boolean()))
         {
-            detail::view::throw_type_error(detail::exception_id::type_mismatch, "type must be boolean, but is ", type_name());
+            detail::view::throw_type_error(302, "type must be boolean, but is ", type_name());
         }
         return (m_node->flags & detail::view::node_flags::is_true) != 0;
     }
@@ -805,7 +865,7 @@ class basic_json_view
     {
         if (NLOHMANN_VIEW_UNLIKELY(m_node == nullptr))
         {
-            detail::view::throw_type_error(detail::exception_id::type_mismatch, "type must be number, but is ", type_name());
+            detail::view::throw_type_error(302, "type must be number, but is ", type_name());
         }
         return detail::view::arithmetic_value<T, BasicJsonType>(*m_doc, *m_node);
     }
@@ -814,7 +874,7 @@ class basic_json_view
     {
         if (NLOHMANN_VIEW_UNLIKELY(!is_null()))
         {
-            detail::view::throw_type_error(detail::exception_id::type_mismatch, "type must be null, but is ", type_name());
+            detail::view::throw_type_error(302, "type must be null, but is ", type_name());
         }
         return nullptr;
     }
@@ -1123,7 +1183,7 @@ class basic_json_document
     {
         if (NLOHMANN_VIEW_UNLIKELY(!m_data || m_data->discarded))
         {
-            detail::view::throw_type_error(detail::exception_id::document_image_unsupported, "cannot save a discarded json_document");
+            detail::view::throw_type_error(320, "cannot save a discarded json_document");
         }
         return detail::view::save_image(*m_data);
     }
@@ -1224,7 +1284,7 @@ class basic_json_document
                 const std::size_t idx = pointer_index(token);
                 if (idx >= 0xFFFFFFFFu)
                 {
-                    detail::view::throw_array_index_out_of_range(idx);
+                    detail::view::throw_out_of_range(401, detail::concat("array index ", std::to_string(idx), " is out of range"));
                 }
                 for (std::size_t i = 0; i < idx; ++i)
                 {
@@ -1280,7 +1340,7 @@ class basic_json_document
     {
         if (ptr.empty())
         {
-            detail::view::throw_out_of_range(detail::exception_id::patch_on_root, "JSON pointer has no parent");
+            detail::view::throw_out_of_range(405, "JSON pointer has no parent");
         }
         const view_type parent = root().at(ptr.parent_pointer());
         const auto& token = ptr.back();
@@ -1300,7 +1360,7 @@ class basic_json_document
         static_assert(Editable, "only an editable document can be edited: use basic_json_document<BasicJsonType, true> (json_editable_document)");
         if (NLOHMANN_VIEW_UNLIKELY(!m_data || m_data->discarded))
         {
-            detail::view::throw_invalid_iterator(detail::exception_id::iterator_from_other_value, "view does not belong to this document");
+            detail::view::throw_invalid_iterator(202, "view does not belong to this document");
         }
         return detail::view::editor<BasicJsonType, view_type>(*m_data);
     }
@@ -1317,7 +1377,7 @@ class basic_json_document
     {
         if (idx < 0)
         {
-            detail::view::throw_array_index_out_of_range(idx);
+            detail::view::throw_out_of_range(401, detail::concat("array index ", std::to_string(idx), " is out of range"));
         }
         return static_cast<std::size_t>(idx);
     }
@@ -1528,7 +1588,8 @@ class basic_json_document
         // NOLINTBEGIN(readability-avoid-nested-conditional-operator): a constant expression of C++11
         append_input_impl(buf, std::forward<T>(input), std::integral_constant < int, detail::is_contiguous_byte_container<D>::value ? 0
                           : std::is_base_of<std::istream, D>::value ? 1
-                          : detail::view::is_adapter_input<D>::value ? 2 : 3 > {});
+                          : detail::view::is_adapter_input<D>::value ? 2
+                          : detail::view::is_c_stream<D>::value ? 4 : 3 > {});
         // NOLINTEND(readability-avoid-nested-conditional-operator)
     }
 
@@ -1550,6 +1611,19 @@ class basic_json_document
         detail::view::append_adapter(buf, std::move(input.ia));
     }
 
+#ifndef JSON_NO_IO
+    static void append_input_impl(std::string& buf, std::FILE* file, std::integral_constant<int, 4> /*C stream*/)
+    {
+        if (file == nullptr)
+        {
+            // (as before: the library's adapter decides what a null stream yields)
+            detail::view::append_adapter(buf, detail::input_adapter(file));
+            return;
+        }
+        detail::view::append_file(buf, file);
+    }
+#endif
+
     template<typename T>
     static void append_input_impl(std::string& buf, T&& input, std::integral_constant<int, 3> /*other*/)
     {

include/nlohmann/ordered_map.hpp

@@ -125,19 +125,6 @@ private:
         Container::pop_back();
     }
 
-    /// @brief remove the entry with the given key; returns the number of removed entries
-    template<class K>
-    size_type erase_impl(const K& key)
-    {
-        const auto it = find_impl(*this, key);
-        if (it != this->end())
-        {
-            erase_at(it);
-            return 1;
-        }
-        return 0;
-    }
-
 public:
     template<class V, detail::enable_if_t<
                  detail::is_constructible<T, V>::value, int> = 0>
@@ -227,14 +214,26 @@ public:
 
     size_type erase(const key_type& key)
     {
-        return erase_impl(key);
+        const auto it = find_impl(*this, key);
+        if (it != this->end())
+        {
+            erase_at(it);
+            return 1;
+        }
+        return 0;
     }
 
     template<class KeyType, detail::enable_if_t<
                  detail::is_usable_as_key_type<key_compare, key_type, KeyType>::value, int> = 0>
     size_type erase(KeyType && key) // NOLINT(cppcoreguidelines-missing-std-forward)
     {
-        return erase_impl(key);
+        const auto it = find_impl(*this, key);
+        if (it != this->end())
+        {
+            erase_at(it);
+            return 1;
+        }
+        return 0;
     }
 
     iterator erase(iterator pos)

src/modules/json.cppm

@@ -60,6 +60,7 @@ inline namespace json_literals
 namespace detail
 {
     using NLOHMANN_JSON_NAMESPACE::detail::json_sax_dom_callback_parser;
+    using NLOHMANN_JSON_NAMESPACE::detail::json_sax_dom_parser;
     using NLOHMANN_JSON_NAMESPACE::detail::unknown_size;
 } // namespace detail
 

tests/src/fuzzer-parse_bjdata.cpp

@@ -47,6 +47,10 @@ dumps is stable under exactly the same values that break operator==.
 The unit tests run the same checks on a fixed corpus (see the "BJData round-trip
 invariants" test case), so keep both in sync.
 
+Furthermore, it reads data with a SAX parser that recovers from every error
+and checks that the events are balanced, that reading ends, and that it
+reports an error exactly when from_bjdata() fails (see #3989).
+
 Finally, it reads the data from a stream, a non-contiguous iterator range, and
 a FILE*, which read byte by byte, and checks that each gives the same value or
 error (id, message, byte position) as the pointer range, which takes the bulk
@@ -57,6 +61,7 @@ drivers.
 */
 
 #include "fuzzer_common.hpp"
+#include "fuzzer-recovering_checker.hpp"
 
 namespace
 {
@@ -81,6 +86,9 @@ static bool is_value_stable(const json& lhs, const json& rhs)
 // see http://llvm.org/docs/LibFuzzer.html
 extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
 {
+    // recover from all errors, reading from memory and from a stream
+    const bool recovered_without_errors = check_recovering_parse(data, size, json::input_format_t::bjdata).errors == 0;
+
     std::vector<uint8_t> const vec1(data, data + size);
 
     // step 0: parse input without exceptions
@@ -99,6 +107,9 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
         // without exceptions, the same input must give the same value
         assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
 
+        // the recovering parser must not have reported an error either
+        assert(recovered_without_errors);
+
         try
         {
             // step 2.1: round trip without adding size annotations to container types
@@ -132,6 +143,7 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
     {
         // parse errors are ok, because input may be random bytes
         assert(parsed || noexcept_threw || j_noexcept.is_discarded());
+        assert(parsed || !recovered_without_errors);
     }
     catch (const json::type_error&)
     {
@@ -142,6 +154,7 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
     {
         // out of range errors may happen if provided sizes are excessive
         assert(parsed || noexcept_threw || j_noexcept.is_discarded());
+        assert(parsed || !recovered_without_errors);
     }
 
     // step 3: the other input adapters must read the data alike

tests/src/fuzzer-parse_bon8.cpp

@@ -17,6 +17,10 @@ array data, it performs the following steps:
 - j2 = from_bon8(vec)
 - assert(to_bon8(j2) == vec)
 
+Furthermore, it reads data with a SAX parser that recovers from every error
+and checks that the events are balanced, that reading ends, and that it
+reports an error exactly when from_bon8() fails (see #3989).
+
 Finally, it reads the data from a stream, a non-contiguous iterator range, and
 a FILE*, which read byte by byte, and checks that each gives the same value or
 error (id, message, byte position) as the pointer range, which takes the bulk
@@ -27,6 +31,7 @@ drivers.
 */
 
 #include "fuzzer_common.hpp"
+#include "fuzzer-recovering_checker.hpp"
 
 namespace
 {
@@ -44,6 +49,9 @@ struct read_bon8
 // see http://llvm.org/docs/LibFuzzer.html
 extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
 {
+    // recover from all errors, reading from memory and from a stream
+    const bool recovered_without_errors = check_recovering_parse(data, size, json::input_format_t::bon8).errors == 0;
+
     std::vector<uint8_t> const vec1(data, data + size);
 
     // step 0: parse input without exceptions
@@ -62,6 +70,9 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
         // without exceptions, the same input must give the same value
         assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
 
+        // the recovering parser must not have reported an error either
+        assert(recovered_without_errors);
+
         try
         {
             // step 2: round trip
@@ -83,6 +94,7 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
     {
         // parse errors are ok, because input may be random bytes
         assert(parsed || noexcept_threw || j_noexcept.is_discarded());
+        assert(parsed || !recovered_without_errors);
     }
     catch (const json::type_error&)
     {
@@ -93,6 +105,7 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
     {
         // out of range errors may happen if provided sizes are excessive
         assert(parsed || noexcept_threw || j_noexcept.is_discarded());
+        assert(parsed || !recovered_without_errors);
     }
 
     // step 3: the other input adapters must read the data alike

tests/src/fuzzer-parse_bson.cpp

@@ -17,6 +17,10 @@ array data, it performs the following steps:
 - j2 = from_bson(vec)
 - assert(to_bson(j2) == vec)
 
+Furthermore, it reads data with a SAX parser that recovers from every error
+and checks that the events are balanced, that reading ends, and that it
+reports an error exactly when from_bson() fails (see #3989).
+
 Finally, it reads the data from a stream, a non-contiguous iterator range, and
 a FILE*, which read byte by byte, and checks that each gives the same value or
 error (id, message, byte position) as the pointer range, which takes the bulk
@@ -27,6 +31,7 @@ drivers.
 */
 
 #include "fuzzer_common.hpp"
+#include "fuzzer-recovering_checker.hpp"
 
 namespace
 {
@@ -44,6 +49,9 @@ struct read_bson
 // see http://llvm.org/docs/LibFuzzer.html
 extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
 {
+    // recover from all errors, reading from memory and from a stream
+    const bool recovered_without_errors = check_recovering_parse(data, size, json::input_format_t::bson).errors == 0;
+
     std::vector<uint8_t> const vec1(data, data + size);
 
     // step 0: parse input without exceptions
@@ -62,6 +70,9 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
         // without exceptions, the same input must give the same value
         assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
 
+        // the recovering parser must not have reported an error either
+        assert(recovered_without_errors);
+
         try
         {
             // step 2: round trip
@@ -83,6 +94,7 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
     {
         // parse errors are ok, because input may be random bytes
         assert(parsed || noexcept_threw || j_noexcept.is_discarded());
+        assert(parsed || !recovered_without_errors);
     }
     catch (const json::type_error&)
     {
@@ -93,6 +105,7 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
     {
         // out of range errors can occur during parsing, too
         assert(parsed || noexcept_threw || j_noexcept.is_discarded());
+        assert(parsed || !recovered_without_errors);
     }
 
     // step 3: the other input adapters must read the data alike

tests/src/fuzzer-parse_cbor.cpp

@@ -17,6 +17,10 @@ array data, it performs the following steps:
 - j2 = from_cbor(vec)
 - assert(to_cbor(j2) == vec)
 
+Furthermore, it reads data with a SAX parser that recovers from every error
+and checks that the events are balanced, that reading ends, and that it
+reports an error exactly when from_cbor() fails (see #3989).
+
 Finally, it reads the data from a stream, a non-contiguous iterator range, and
 a FILE*, which read byte by byte, and checks that each gives the same value or
 error (id, message, byte position) as the pointer range, which takes the bulk
@@ -27,6 +31,7 @@ drivers.
 */
 
 #include "fuzzer_common.hpp"
+#include "fuzzer-recovering_checker.hpp"
 
 namespace
 {
@@ -44,6 +49,9 @@ struct read_cbor
 // see http://llvm.org/docs/LibFuzzer.html
 extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
 {
+    // recover from all errors, reading from memory and from a stream
+    const bool recovered_without_errors = check_recovering_parse(data, size, json::input_format_t::cbor).errors == 0;
+
     std::vector<uint8_t> const vec1(data, data + size);
 
     // step 0: parse input without exceptions
@@ -62,6 +70,9 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
         // without exceptions, the same input must give the same value
         assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
 
+        // the recovering parser must not have reported an error either
+        assert(recovered_without_errors);
+
         try
         {
             // step 2: round trip
@@ -83,6 +94,7 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
     {
         // parse errors are ok, because input may be random bytes
         assert(parsed || noexcept_threw || j_noexcept.is_discarded());
+        assert(parsed || !recovered_without_errors);
     }
     catch (const json::type_error&)
     {
@@ -93,6 +105,7 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
     {
         // out of range errors can occur during parsing, too
         assert(parsed || noexcept_threw || j_noexcept.is_discarded());
+        assert(parsed || !recovered_without_errors);
     }
 
     // step 3: the other input adapters must read the data alike

tests/src/fuzzer-parse_json.cpp

@@ -24,11 +24,16 @@ array data, it performs the following steps:
 The pointer range takes the bulk fast paths, the other input adapters read byte
 by byte; see check_input_adapters().
 
+Furthermore, it parses data with a SAX parser that recovers from every error
+and checks that the events are balanced, that parsing ends, and that valid
+input is parsed without errors (see #3989).
+
 The provided function `LLVMFuzzerTestOneInput` can be used in different fuzzer
 drivers.
 */
 
 #include "fuzzer_common.hpp"
+#include "fuzzer-recovering_checker.hpp"
 
 namespace
 {
@@ -59,6 +64,13 @@ struct accept_json
 // see http://llvm.org/docs/LibFuzzer.html
 extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
 {
+    // recover from all errors, reading from memory and from a stream
+    {
+        const auto checker = check_recovering_parse(data, size, json::input_format_t::json);
+        assert(checker.events <= (4 * size) + 4);
+        assert((checker.errors == 0) == json::accept(data, data + size));
+    }
+
     // step 0: parse input without exceptions
     bool noexcept_threw = false;
     json const j_noexcept = parse_without_exceptions([&] { return json::parse(data, data + size, nullptr, false); }, noexcept_threw);

tests/src/fuzzer-parse_msgpack.cpp

@@ -17,6 +17,10 @@ array data, it performs the following steps:
 - j2 = from_msgpack(vec)
 - assert(to_msgpack(j2) == vec)
 
+Furthermore, it reads data with a SAX parser that recovers from every error
+and checks that the events are balanced, that reading ends, and that it
+reports an error exactly when from_msgpack() fails (see #3989).
+
 Finally, it reads the data from a stream, a non-contiguous iterator range, and
 a FILE*, which read byte by byte, and checks that each gives the same value or
 error (id, message, byte position) as the pointer range, which takes the bulk
@@ -27,6 +31,7 @@ drivers.
 */
 
 #include "fuzzer_common.hpp"
+#include "fuzzer-recovering_checker.hpp"
 
 namespace
 {
@@ -44,6 +49,9 @@ struct read_msgpack
 // see http://llvm.org/docs/LibFuzzer.html
 extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
 {
+    // recover from all errors, reading from memory and from a stream
+    const bool recovered_without_errors = check_recovering_parse(data, size, json::input_format_t::msgpack).errors == 0;
+
     std::vector<uint8_t> const vec1(data, data + size);
 
     // step 0: parse input without exceptions
@@ -62,6 +70,9 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
         // without exceptions, the same input must give the same value
         assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
 
+        // the recovering parser must not have reported an error either
+        assert(recovered_without_errors);
+
         try
         {
             // step 2: round trip
@@ -83,6 +94,7 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
     {
         // parse errors are ok, because input may be random bytes
         assert(parsed || noexcept_threw || j_noexcept.is_discarded());
+        assert(parsed || !recovered_without_errors);
     }
     catch (const json::type_error&)
     {
@@ -93,6 +105,7 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
     {
         // out of range errors may happen if provided sizes are excessive
         assert(parsed || noexcept_threw || j_noexcept.is_discarded());
+        assert(parsed || !recovered_without_errors);
     }
 
     // step 3: the other input adapters must read the data alike

tests/src/fuzzer-parse_ubjson.cpp

@@ -26,6 +26,10 @@ array data, it performs the following steps:
 The unit tests run the same checks on a fixed corpus (see the "UBJSON round-trip
 invariants" test case), so keep both in sync.
 
+Furthermore, it reads data with a SAX parser that recovers from every error
+and checks that the events are balanced, that reading ends, and that it
+reports an error exactly when from_ubjson() fails (see #3989).
+
 Finally, it reads the data from a stream, a non-contiguous iterator range, and
 a FILE*, which read byte by byte, and checks that each gives the same value or
 error (id, message, byte position) as the pointer range, which takes the bulk
@@ -36,6 +40,7 @@ drivers.
 */
 
 #include "fuzzer_common.hpp"
+#include "fuzzer-recovering_checker.hpp"
 
 namespace
 {
@@ -53,6 +58,9 @@ struct read_ubjson
 // see http://llvm.org/docs/LibFuzzer.html
 extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
 {
+    // recover from all errors, reading from memory and from a stream
+    const bool recovered_without_errors = check_recovering_parse(data, size, json::input_format_t::ubjson).errors == 0;
+
     std::vector<uint8_t> const vec1(data, data + size);
 
     // step 0: parse input without exceptions
@@ -71,6 +79,9 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
         // without exceptions, the same input must give the same value
         assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
 
+        // the recovering parser must not have reported an error either
+        assert(recovered_without_errors);
+
         try
         {
             // step 2.1: round trip without adding size annotations to container types
@@ -102,6 +113,7 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
     {
         // parse errors are ok, because input may be random bytes
         assert(parsed || noexcept_threw || j_noexcept.is_discarded());
+        assert(parsed || !recovered_without_errors);
     }
     catch (const json::type_error&)
     {
@@ -112,6 +124,7 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
     {
         // out of range errors may happen if provided sizes are excessive
         assert(parsed || noexcept_threw || j_noexcept.is_discarded());
+        assert(parsed || !recovered_without_errors);
     }
 
     // step 3: the other input adapters must read the data alike

tests/src/fuzzer-recovering_checker.hpp

@@ -0,0 +1,154 @@
+//     __ _____ _____ _____
+//  __|  |   __|     |   | |  JSON for Modern C++ (supporting code)
+// |  |  |__   |  |  | | | |  version 3.12.0
+// |_____|_____|_____|_|___|  https://github.com/nlohmann/json
+//
+// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
+// SPDX-License-Identifier: MIT
+
+#pragma once
+
+#include <cassert>
+#include <cstddef>
+#include <cstdint>
+#include <sstream>
+#include <string>
+#include <vector>
+#include <nlohmann/json.hpp>
+
+namespace
+{
+// a SAX parser that recovers from every error and checks that the events are
+// balanced and that every key is followed by exactly one value
+class recovering_checker : public nlohmann::json_sax<nlohmann::json>
+{
+  public:
+    bool null() override
+    {
+        return value();
+    }
+
+    bool boolean(bool /*val*/) override
+    {
+        return value();
+    }
+
+    bool number_integer(number_integer_t /*val*/) override
+    {
+        return value();
+    }
+
+    bool number_unsigned(number_unsigned_t /*val*/) override
+    {
+        return value();
+    }
+
+    bool number_float(number_float_t /*val*/, const string_t& /*s*/) override
+    {
+        return value();
+    }
+
+    bool string(string_t& /*val*/) override
+    {
+        return value();
+    }
+
+    bool binary(binary_t& /*val*/) override
+    {
+        return value();
+    }
+
+    bool start_object(std::size_t /*elements*/) override
+    {
+        value();
+        stack.push_back('o');
+        return true;
+    }
+
+    bool key(string_t& /*val*/) override
+    {
+        ++events;
+        assert(!stack.empty() && stack.back() == 'o');
+        stack.back() = 'v';
+        return true;
+    }
+
+    bool end_object() override
+    {
+        ++events;
+        assert(!stack.empty() && stack.back() == 'o');
+        stack.pop_back();
+        return true;
+    }
+
+    bool start_array(std::size_t /*elements*/) override
+    {
+        value();
+        stack.push_back('a');
+        return true;
+    }
+
+    bool end_array() override
+    {
+        ++events;
+        assert(!stack.empty() && stack.back() == 'a');
+        stack.pop_back();
+        return true;
+    }
+
+    bool parse_error(std::size_t /*position*/, const std::string& /*last_token*/, const nlohmann::detail::exception& /*ex*/) override
+    {
+        ++errors;
+        return true;
+    }
+
+    bool complete() const
+    {
+        return stack.empty();
+    }
+
+    std::size_t events = 0;
+    std::size_t errors = 0;
+
+  private:
+    bool value()
+    {
+        ++events;
+        if (!stack.empty())
+        {
+            // an array element, or the value of a key
+            assert(stack.back() != 'o');
+            if (stack.back() == 'v')
+            {
+                stack.back() = 'o';
+            }
+        }
+        return true;
+    }
+
+    // 'a' for an array, 'o' for an object that expects a key, 'v' for an
+    // object that expects the value of a key
+    std::vector<char> stack {}; // NOLINT(readability-redundant-member-init)
+};
+
+/// parses @a data with a recovering_checker from memory and from a stream,
+/// checks that both see the same, that the events are balanced, and that the
+/// number of errors is bounded, and returns the checker (see #3989)
+inline recovering_checker check_recovering_parse(const std::uint8_t* data, const std::size_t size, const nlohmann::json::input_format_t format)
+{
+    recovering_checker checker;
+    const bool ok = nlohmann::json::sax_parse(data, data + size, &checker, format);
+    assert(checker.complete());
+    assert(checker.errors <= size + 1);
+    assert(ok == (checker.errors == 0));
+
+    std::istringstream stream(std::string(reinterpret_cast<const char*>(data), size));
+    recovering_checker stream_checker;
+    assert(nlohmann::json::sax_parse(stream, &stream_checker, format) == ok);
+    assert(stream_checker.complete());
+    assert(stream_checker.events == checker.events);
+    assert(stream_checker.errors == checker.errors);
+
+    return checker;
+}
+} // namespace

tests/src/sax_event_loggers.hpp

@@ -131,11 +131,13 @@ struct SaxEventLogger : public nlohmann::json_sax<nlohmann::json>
     {
         errored = true;
         events.push_back("parse_error(" + std::to_string(position) + ")");
-        return false;
+        return recover;
     }
 
     std::vector<std::string> events {}; // NOLINT(readability-redundant-member-init)
     bool errored = false;
+    /// whether parse_error() asks the parser to recover from the error (see #3989)
+    bool recover = false;
 };
 
 struct SaxEventLoggerExitAfterStartObject : public SaxEventLogger

tests/src/unit-alt-string.cpp

@@ -415,6 +415,46 @@ TEST_CASE("alternative string type")
         CHECK(j2.dump() == R"({"/foo/0":"bar","/foo/1":"baz"})");
     }
 
+    SECTION("error recovery")
+    {
+        // a SAX parser that recovers from every error (see #3989)
+        struct recovering_parser : nlohmann::detail::json_sax_dom_parser<alt_json>
+        {
+            explicit recovering_parser(alt_json& j)
+                : nlohmann::detail::json_sax_dom_parser<alt_json>(j, false)
+            {}
+
+            // sax_parse() calls the SAX parser's own parse_error(), so hiding
+            // the one of the base class is what recovering takes
+            // NOLINTNEXTLINE(bugprone-derived-method-shadowing-base-method)
+            bool parse_error(std::size_t /*unused*/, const std::string& /*unused*/, const nlohmann::detail::exception& /*unused*/)
+            {
+                ++errors;
+                return true;
+            }
+
+            std::size_t errors = 0;
+        };
+
+        alt_json j;
+        recovering_parser sax(j);
+        // not inside CHECK(): MSVC reads the escape in a stringized raw string
+        const std::string input = R"([1., "a\qb", tru, {"k" 2}])";
+        CHECK(!alt_json::sax_parse(input, &sax));
+        CHECK(sax.errors == 4);
+        CHECK(j.dump() == R"([1,"aqb",null,{"k":2}])");
+
+        // a UBJSON high-precision number, a CBOR key that is not a string
+        alt_json u;
+        recovering_parser ubjson_sax(u);
+        CHECK(!alt_json::sax_parse(std::vector<std::uint8_t> {'[', 'H', 'i', 2, '1', '.', ']'}, &ubjson_sax, alt_json::input_format_t::ubjson));
+        CHECK(u.dump() == "[1]");
+        alt_json c;
+        recovering_parser cbor_sax(c);
+        CHECK(!alt_json::sax_parse(std::vector<std::uint8_t> {0xA2, 0x01, 0x02, 0x61, 'a', 0x03}, &cbor_sax, alt_json::input_format_t::cbor));
+        CHECK(c.dump() == R"({"a":3})");
+    }
+
     SECTION("conversion between basic_json specializations (#2649)")
     {
         // explicit conversions are always possible

tests/src/unit-class_lexer.cpp

@@ -2042,118 +2042,3 @@ TEST_CASE("string scanning kernels")
     CHECK(nlohmann::detail::read_eight_bytes(bytes.data() + 8) == 0xFF0F0E0D0C0B0A09u);
     CHECK(nlohmann::detail::read_eight_bytes(reinterpret_cast<const char*>(bytes.data()) + 3) == 0x0B0A090807060504u);
 }
-
-TEST_CASE("find_ascii_copyable_run agrees with is_ascii_copyable for every byte")
-{
-    // the SWAR mask in find_ascii_copyable_run() encodes the predicate a
-    // second time; check every byte value at every position of a 24-byte buffer
-    // (three words) and for several lengths, with the byte at the end or inside
-    for (unsigned int value = 0; value < 256; ++value)
-    {
-        for (std::size_t position = 0; position < 24; ++position)
-        {
-            std::array<unsigned char, 24> buffer{};
-            buffer.fill(static_cast<unsigned char>('a'));
-            buffer[position] = static_cast<unsigned char>(value);
-            const bool copyable = nlohmann::detail::is_ascii_copyable(static_cast<unsigned char>(value));
-            for (const std::size_t n :
-                    {
-                        position, position + 1, std::size_t{24}
-                    })
-            {
-                if (n > 24)
-                {
-                    continue;
-                }
-                const std::size_t expected = (copyable || n <= position) ? n : position;
-                CAPTURE(value)
-                CAPTURE(position)
-                CAPTURE(n)
-                CHECK(nlohmann::detail::find_ascii_copyable_run(buffer.data(), n) == expected);
-            }
-        }
-    }
-}
-
-TEST_CASE("UTF-8 validators accept the same sequences")
-{
-    // The library checks UTF-8 in several places: detail::decode (serializer),
-    // the per-lead-byte switch of the lexer (json::accept), validate_one_utf8
-    // (bulk string scan), utf8_shape, and the byte path of the BON8 reader.
-    // They must accept exactly the same sequences (RFC 3629, section 4).
-    using nlohmann::json;
-
-    // bytes for the third and fourth position: the first two pairs are checked
-    // against all validators, the others only against the cheap ones
-    const std::array<std::array<unsigned char, 2>, 6> tails =
-    {
-        {
-            {{0x80, 0x80}}, {{0xBF, 0xBF}}, {{0x7F, 0x80}}, {{0xC0, 0x80}}, {{0x80, 0x7F}}, {{0x80, 0xC0}}
-        }
-    };
-
-    for (unsigned lead = 0x80; lead <= 0xFF; ++lead)
-    {
-        const auto shape = nlohmann::detail::utf8_shape(static_cast<std::uint8_t>(lead));
-        for (unsigned second = 0; second <= 0xFF; ++second)
-        {
-            for (std::size_t t = 0; t < tails.size(); ++t)
-            {
-                const std::array<unsigned char, 4> bytes = {{static_cast<unsigned char>(lead), static_cast<unsigned char>(second), tails[t][0], tails[t][1]}};
-
-                // expectation from the shape: the sequence is bytes[0..length)
-                const std::size_t length = shape.length;
-                bool expected = length >= 2 && second >= shape.second_lo && second <= shape.second_hi;
-                for (std::size_t i = 2; expected && i < length; ++i)
-                {
-                    expected = bytes[i] >= 0x80 && bytes[i] <= 0xBF;
-                }
-
-                // the DFA of the serializer
-                bool dfa = length >= 2;
-                if (dfa)
-                {
-                    std::uint8_t state = nlohmann::detail::UTF8_ACCEPT;
-                    std::uint32_t codep = 0;
-                    for (std::size_t i = 0; i < length; ++i)
-                    {
-                        nlohmann::detail::decode(state, codep, bytes[i]);
-                    }
-                    dfa = state == nlohmann::detail::UTF8_ACCEPT;
-                }
-
-                // the bulk path: exactly the sequence, and one byte short
-                const std::size_t bulk = nlohmann::detail::validate_one_utf8(bytes.data(), length);
-                const std::size_t bulk_short = length > 0 ? nlohmann::detail::validate_one_utf8(bytes.data(), length - 1) : 0;
-
-                CAPTURE(lead)
-                CAPTURE(second)
-                CAPTURE(t)
-                CHECK((dfa == expected && bulk == (expected ? length : 0) && bulk_short == 0));
-
-                if (t < 2 && length >= 2)
-                {
-                    // the lexer's byte path
-                    const std::string sequence(reinterpret_cast<const char*>(bytes.data()), length); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
-                    CHECK(json::accept("\"" + sequence + "\"") == expected);
-
-                    // the byte path of the BON8 reader (a string ends at 0xFF)
-                    if (lead >= 0xC2 && lead <= 0xF7)
-                    {
-                        std::vector<std::uint8_t> packed(bytes.begin(), bytes.begin() + static_cast<std::ptrdiff_t>(length));
-                        packed.push_back(0xFF);
-                        const json j = json::from_bon8(packed, true, false);
-                        if (expected)
-                        {
-                            CHECK(j == json(sequence));
-                        }
-                        else
-                        {
-                            CHECK_FALSE((j.is_string() && j.get<std::string>() == sequence));
-                        }
-                    }
-                }
-            }
-        }
-    }
-}

tests/src/unit-class_parser.cpp

@@ -2952,3 +2952,726 @@ TEST_CASE("diagnostic positions: value lifetime, input adapters, and SAX")
     }
 }
 #endif
+
+namespace
+{
+/// builds a value like json::parse(), but asks the parser to recover from
+/// errors (see #3989), and checks that the events it receives are balanced
+class RecoveringDomParser
+{
+  public:
+    explicit RecoveringDomParser(json& j, std::size_t max_errors_ = static_cast<std::size_t>(-1))
+        : dom(j, false)
+        , max_errors(max_errors_)
+    {}
+
+    bool null()
+    {
+        value();
+        return dom.null();
+    }
+
+    bool boolean(bool val)
+    {
+        value();
+        return dom.boolean(val);
+    }
+
+    bool number_integer(json::number_integer_t val)
+    {
+        value();
+        return dom.number_integer(val);
+    }
+
+    bool number_unsigned(json::number_unsigned_t val)
+    {
+        value();
+        return dom.number_unsigned(val);
+    }
+
+    bool number_float(json::number_float_t val, const std::string& s)
+    {
+        value();
+        return dom.number_float(val, s);
+    }
+
+    bool string(std::string& val)
+    {
+        value();
+        return dom.string(val);
+    }
+
+    bool binary(json::binary_t& val)
+    {
+        value();
+        return dom.binary(val);
+    }
+
+    bool start_object(std::size_t elements)
+    {
+        value();
+        stack.push_back('o');
+        return dom.start_object(elements);
+    }
+
+    bool key(std::string& val)
+    {
+        ++events;
+        if (stack.empty() || stack.back() != 'o')
+        {
+            well_formed = false;
+            return false;
+        }
+        stack.back() = 'v';
+        return dom.key(val);
+    }
+
+    bool end_object()
+    {
+        ++events;
+        if (stack.empty() || stack.back() != 'o')
+        {
+            well_formed = false;
+            return false;
+        }
+        stack.pop_back();
+        return dom.end_object();
+    }
+
+    bool start_array(std::size_t elements)
+    {
+        value();
+        stack.push_back('a');
+        return dom.start_array(elements);
+    }
+
+    bool end_array()
+    {
+        ++events;
+        if (stack.empty() || stack.back() != 'a')
+        {
+            well_formed = false;
+            return false;
+        }
+        stack.pop_back();
+        return dom.end_array();
+    }
+
+    bool parse_error(std::size_t /*unused*/, const std::string& /*unused*/, const json::exception& ex)
+    {
+        errors.emplace_back(ex.what());
+        return errors.size() < max_errors;
+    }
+
+    /// whether the events were balanced and every key was followed by a value
+    bool balanced() const
+    {
+        return well_formed && stack.empty();
+    }
+
+    /// builds the value
+    nlohmann::detail::json_sax_dom_parser<json> dom;
+    std::vector<std::string> errors {}; // NOLINT(readability-redundant-member-init)
+    std::size_t events = 0;
+    /// the open containers: 'a' for an array, 'o' for an object that expects
+    /// a key, 'v' for an object that expects the value of a key
+    std::vector<char> stack {}; // NOLINT(readability-redundant-member-init)
+    bool well_formed = true;
+    std::size_t max_errors;
+
+  private:
+    /// a value is passed: it is an array element, or the value of a key
+    void value()
+    {
+        ++events;
+        if (!stack.empty())
+        {
+            if (stack.back() == 'v')
+            {
+                stack.back() = 'o';
+            }
+            else if (stack.back() == 'o')
+            {
+                // a value without a key
+                well_formed = false;
+            }
+        }
+    }
+
+};
+
+struct RecoveryResult
+{
+    json value;
+    std::vector<std::string> errors;
+    std::size_t events;
+    bool ok;
+    bool balanced;
+};
+
+template<typename InputType>
+RecoveryResult parse_recovering(InputType&& input, const bool strict = true,
+                                const bool ignore_comments = false, const bool ignore_trailing_commas = false)
+{
+    json j;
+    RecoveringDomParser sax(j);
+    const bool ok = json::sax_parse(std::forward<InputType>(input), &sax, json::input_format_t::json,
+                                    strict, ignore_comments, ignore_trailing_commas);
+    return {j, sax.errors, sax.events, ok, sax.balanced()};
+}
+
+/// stops after a number of events, but recovers from errors
+class RecoveringCountdown
+{
+  public:
+    explicit RecoveringCountdown(const int count)
+        : countdown(count)
+    {}
+
+    bool null()
+    {
+        return countdown.null();
+    }
+
+    bool boolean(bool val)
+    {
+        return countdown.boolean(val);
+    }
+
+    bool number_integer(json::number_integer_t val)
+    {
+        return countdown.number_integer(val);
+    }
+
+    bool number_unsigned(json::number_unsigned_t val)
+    {
+        return countdown.number_unsigned(val);
+    }
+
+    bool number_float(json::number_float_t val, const std::string& s)
+    {
+        return countdown.number_float(val, s);
+    }
+
+    bool string(std::string& val)
+    {
+        return countdown.string(val);
+    }
+
+    bool binary(json::binary_t& val)
+    {
+        return countdown.binary(val);
+    }
+
+    bool start_object(std::size_t elements)
+    {
+        return countdown.start_object(elements);
+    }
+
+    bool key(std::string& val)
+    {
+        return countdown.key(val);
+    }
+
+    bool end_object()
+    {
+        return countdown.end_object();
+    }
+
+    bool start_array(std::size_t elements)
+    {
+        return countdown.start_array(elements);
+    }
+
+    bool end_array()
+    {
+        return countdown.end_array();
+    }
+
+    bool parse_error(std::size_t /*position*/, const std::string& /*last_token*/, const json::exception& /*ex*/) // NOLINT(readability-convert-member-functions-to-static)
+    {
+        return true;
+    }
+
+  private:
+    SaxCountdown countdown;
+};
+
+/// a repaired input: the value it is repaired to, and the number of errors
+struct Repair
+{
+    const char* input;
+    const char* expected;
+    std::size_t errors;
+};
+} // namespace
+
+TEST_CASE("parser error recovery (#3989)")
+{
+    SECTION("repairs")
+    {
+        const std::vector<Repair> repairs =
+        {
+            // a missing separator is inserted
+            {"[1 2]", "[1,2]", 1},
+            {R"({"a":1 "b":2})", R"({"a":1,"b":2})", 1},
+            {R"({"a" 1})", R"({"a":1})", 1},
+            {"[1 tru 2]", "[1,null,2]", 2},
+            {R"({"a" "b": 1})", R"({"a":"b"})", 2},
+
+            // a missing value is null in an object; in an array, a ',' stands
+            // for null, while an array that ends there just ends
+            {R"({"a":})", R"({"a":null})", 1},
+            {R"({"a"})", R"({"a":null})", 1},
+            {R"({"a","b":1})", R"({"a":null,"b":1})", 1},
+            {"[1,,2]", "[1,null,2]", 1},
+            {"[,1]", "[null,1]", 1},
+            {"[1,]", "[1]", 1},
+            {"[1,2,3,]", "[1,2,3]", 1},
+            {R"({"a":1,})", R"({"a":1})", 1},
+
+            // a broken string keeps what can be read
+            {R"(["a\qb"])", R"(["aqb"])", 1},
+            {R"({"na\me":1})", R"({"name":1})", 1},
+            {"[\"\xFF\"]", R"(["\uFFFD"])", 1},
+            {"[\"a\xC3(\"]", R"(["a\uFFFD("])", 1},
+            {"[\"\xE2\x82\"]", R"(["\uFFFD"])", 1},
+            {"[\"\xC3\\\\\", 1]", R"(["\uFFFD\\",1])", 1},
+            {R"(["\u12"])", R"(["\uFFFD"])", 1},
+            {R"(["\u12G4"])", R"(["\uFFFDG4"])", 1},
+            {R"(["\uDC00x"])", R"(["\uFFFDx"])", 1},
+            {R"(["\uD800x"])", R"(["\uFFFDx"])", 1},
+            {R"(["\uD800\u0041"])", R"(["\uFFFDA"])", 1},
+            {R"(["\uD800\uD800\uDC00"])", R"(["\uFFFD\uD800\uDC00"])", 1},
+            {R"(["\uD800\uD800\uD800x"])", R"(["\uFFFD\uFFFD\uFFFDx"])", 1},
+            {
+                R"(["\uD800\"x", 1])", R"(["\uFFFD\"x",1])", 1
+            },
+            {R"(["\uD800\q"])", R"(["\uFFFDq"])", 1},
+            {"[\"a\tb\"]", R"(["a\tb"])", 1},
+            {R"(["a\qb\u0041\x"])", R"(["aqbAx"])", 1},
+
+            // a broken number keeps its longest valid prefix
+            {"[1.]", "[1]", 1},
+            {"[-2.]", "[-2]", 1},
+            {"[1.5e]", "[1.5]", 1},
+            {"[1e+]", "[1]", 1},
+            {"[1.x2, 3]", "[1,3]", 1},
+
+            // what cannot be read at all is null
+            {"[1,NaN,3]", "[1,null,3]", 1},
+            {"[tru]", "[null]", 1},
+            {"[-]", "[null]", 1},
+            {R"({"a":Infinity})", R"({"a":null})", 1},
+
+            // a stray token is dropped
+            {"[:1]", "[1]", 1},
+            {R"(["a":1])", R"(["a",1])", 1},
+            {R"({"a"::1})", R"({"a":1})", 1},
+
+            // a member that cannot be read is skipped
+            {R"({1:2,"b":3})", R"({"b":3})", 1},
+            {R"({"a":1 2})", R"({"a":1})", 1},
+            {R"({,"a":1})", R"({"a":1})", 1},
+            {R"({"a":1,,"b":2})", R"({"a":1,"b":2})", 1},
+            {"{a:1}", "{}", 1},
+            {R"({"a":1 [1,{"b":2}], "c":3})", R"({"a":1,"c":3})", 1},
+            {R"([{1}, "a"])", R"([{},"a"])", 1},
+
+            // a wrong closing bracket closes the innermost container
+            {R"({"a":[1,2}, "b":3})", R"({"a":[1,2],"b":3})", 1},
+            {R"([{"a":1], 2])", R"([{"a":1},2])", 1},
+            {"{]", "{}", 1},
+            {"[}", "[]", 1},
+
+            // the end of the input closes all containers
+            {R"({"a":[1,2)", R"({"a":[1,2]})", 1},
+            {"[", "[]", 1},
+            {"{", "{}", 1},
+            {R"({"a")", R"({"a":null})", 1},
+            {R"({"a":)", R"({"a":null})", 1},
+            {"[1,", "[1]", 1},
+            {"[[[1", "[[[1]]]", 1},
+            {
+                R"(["abc)", R"(["abc"])", 2
+            },
+            {"[1,tr", "[1,null]", 2},
+            {"\"abc", "\"abc\"", 1},
+            {"[\"ab\ncd\"]", R"(["ab",null,"]"])", 4},
+
+            // what comes before the top-level value is skipped
+            {")]}'\n{\"a\":1}", R"({"a":1})", 1},
+            {R"(data: {"a":1})", R"({"a":1})", 1},
+            {"\xEF\xBB[1]", "[1]", 1},
+
+            // what comes after it is an error that ends parsing
+            {R"({"a":1}})", R"({"a":1})", 1},
+            {"[1}]", "[1]", 2},
+            {"[1] [2]", "[1]", 1},
+        };
+
+        for (const auto& repair : repairs)
+        {
+            CAPTURE(repair.input)
+            const auto result = parse_recovering(std::string(repair.input));
+            CHECK(!result.ok);
+            CHECK(result.balanced);
+            CHECK(result.value == json::parse(repair.expected));
+            CHECK(result.errors.size() == repair.errors);
+        }
+    }
+
+    SECTION("number overflow")
+    {
+        const auto result = parse_recovering(std::string("[1e999,-1e999]"));
+        CHECK(!result.ok);
+        CHECK(result.balanced);
+        CHECK(result.errors.size() == 2);
+        CHECK(result.errors[0] == "[json.exception.out_of_range.406] number overflow parsing '1e999'");
+        REQUIRE(result.value.size() == 2);
+        CHECK(result.value[0].is_number_float());
+        CHECK(result.value[0].get<double>() == std::numeric_limits<double>::infinity());
+        CHECK(result.value[1].get<double>() == -std::numeric_limits<double>::infinity());
+
+        // the SAX parser gets the number's text
+        SaxEventLogger logger;
+        logger.recover = true;
+        CHECK(!json::sax_parse("1e999", &logger));
+        CHECK(logger.events == std::vector<std::string>({"parse_error(5)", "number_float(1e999)"}));
+    }
+
+    SECTION("nothing to recover")
+    {
+        for (const std::string s :
+                {
+                    "", "   ", "]", "tru", "NaN", ",:", "/* comment"
+                })
+        {
+            CAPTURE(s)
+            const auto result = parse_recovering(s, true, true);
+            CHECK(!result.ok);
+            CHECK(result.balanced);
+            CHECK(result.events == 0);
+            CHECK(result.value == nullptr);
+            CHECK(result.errors.size() == 1);
+        }
+    }
+
+    SECTION("error messages")
+    {
+        // the first error is reported as without recovery
+        for (const std::string s :
+                {
+                    "[1 2]", R"({"a":1 "b":2})", R"({"a" 1})", R"({"a":})", "[1,]", "[1.]",
+                    R"(["a\qb"])", "[1e999]", "{1:2}", R"({"a":[1,2}})", "[1,", "[1] [2]", "{a:1}"
+                })
+        {
+            CAPTURE(s)
+            const auto result = parse_recovering(s);
+            REQUIRE(!result.errors.empty());
+            json _;
+            CHECK_THROWS_WITH_STD_STR(_ = json::parse(s), result.errors.front());
+        }
+
+        // the token of an error begins where the previous error was
+        const auto result = parse_recovering(std::string("[tru, fals, nul]"));
+        CHECK(result.errors == std::vector<std::string>(
+        {
+            "[json.exception.parse_error.101] parse error at line 1, column 5: syntax error while parsing value - invalid literal; last read: '[tru,'",
+            "[json.exception.parse_error.101] parse error at line 1, column 11: syntax error while parsing value - invalid literal; last read: ', fals,'",
+            "[json.exception.parse_error.101] parse error at line 1, column 16: syntax error while parsing value - invalid literal; last read: ', nul]'"
+        }));
+        CHECK(result.value == json::parse("[null,null,null]"));
+    }
+
+    SECTION("events")
+    {
+        // see #4522
+        SaxEventLogger logger;
+        logger.recover = true;
+        CHECK(!json::sax_parse(R"([{1}, "a"])", &logger));
+        CHECK(logger.events == std::vector<std::string>(
+        {
+            "start_array()", "start_object()", "parse_error(3)", "end_object()", "string(a)", "end_array()"
+        }));
+    }
+
+    SECTION("options")
+    {
+        SECTION("strict")
+        {
+            const auto result = parse_recovering(std::string("[1 2] [3]"), false);
+            CHECK(!result.ok);
+            CHECK(result.value == json::parse("[1,2]"));
+            CHECK(result.errors.size() == 1);
+        }
+
+        SECTION("ignore_trailing_commas")
+        {
+            for (const std::string s :
+                    {
+                        "[1,]", R"({"a":1,})", "[[1,],]"
+                    })
+            {
+                CAPTURE(s)
+                const auto result = parse_recovering(s, true, false, true);
+                CHECK(result.ok);
+                CHECK(result.errors.empty());
+            }
+
+            auto result = parse_recovering(std::string("[1,,]"), true, false, true);
+            CHECK(result.value == json::parse("[1,null]"));
+            CHECK(result.errors.size() == 1);
+
+            result = parse_recovering(std::string(R"({"a":1,,})"), true, false, true);
+            CHECK(result.value == json::parse(R"({"a":1})"));
+            CHECK(result.errors.size() == 1);
+        }
+
+        SECTION("ignore_comments")
+        {
+            auto result = parse_recovering(std::string("[1 /* one */ 2]"), true, true);
+            CHECK(result.value == json::parse("[1,2]"));
+            CHECK(result.errors.size() == 1);
+
+            // a comment that is not closed runs to the end of the input, which
+            // is not reported again
+            result = parse_recovering(std::string("[1, 2 /* unterminated"), true, true);
+            CHECK(result.balanced);
+            CHECK(result.value == json::parse("[1,2]"));
+            CHECK(result.errors.size() == 1);
+
+            // a '/' that does not begin a comment is garbage
+            result = parse_recovering(std::string("[1, /x, 2]"), true, true);
+            CHECK(result.balanced);
+            CHECK(result.value == json::parse("[1,null,2]"));
+            CHECK(result.errors.size() == 1);
+        }
+    }
+
+    SECTION("null bytes")
+    {
+        // a null byte ends the input, unless JSON_STRICT_NUL_HANDLING is set
+        const auto result = parse_recovering(std::string("[1,\0x", 5));
+        CHECK(result.balanced);
+        CHECK(!result.ok);
+#ifdef JSON_TEST_STRICT_NUL_HANDLING_ENABLED
+        CHECK(result.value == json::parse("[1,null]"));
+#else
+        CHECK(result.value == json::parse("[1]"));
+        CHECK(result.errors.size() == 1);
+#endif
+
+        const auto in_string = parse_recovering(std::string("[\"a\0b\"]", 7));
+        CHECK(in_string.balanced);
+#ifdef JSON_TEST_STRICT_NUL_HANDLING_ENABLED
+        CHECK(in_string.value == json::array({std::string("a\0b", 3)}));
+#else
+        CHECK(in_string.value == json::parse(R"(["a"])"));
+#endif
+    }
+
+    SECTION("the SAX parser stops recovering")
+    {
+        json j;
+        RecoveringDomParser sax(j, 2);
+        CHECK(!json::sax_parse("[1 2 3 4 5]", &sax));
+        CHECK(sax.errors.size() == 2);
+
+        // an error at a delimiter that an invalid token consumed is reported
+        // to the SAX parser, too
+        json j2;
+        RecoveringDomParser sax2(j2, 2);
+        CHECK(!json::sax_parse("[tru}, 1]", &sax2));
+        CHECK(sax2.errors.size() == 2);
+    }
+
+    SECTION("an event stops parsing during a repair")
+    {
+        // start_object() and key() are passed, then null() for the missing
+        // value returns false
+        RecoveringCountdown countdown(2);
+        CHECK(!json::sax_parse(R"({"a":})", &countdown));
+
+        // the end of the input: end_array() for the second array returns false
+        RecoveringCountdown countdown2(4);
+        CHECK(!json::sax_parse("[[1", &countdown2));
+    }
+
+    SECTION("input adapters")
+    {
+        // the lexer reads contiguous and streaming input differently, and it
+        // puts back a character that ended an invalid token
+        for (const std::string s :
+                {
+                    "[1 2]", "[tru}, 1]", R"({"a" "b\q", "c":[1.x, 2}})", "[\"\xFF\xC3(\", -, 1e+]", "{a:1,\"b\":2", ")]}' [1]"
+                })
+        {
+            CAPTURE(s)
+            const auto reference = parse_recovering(s);
+            CHECK(reference.balanced);
+
+            const auto from_c_string = parse_recovering(s.c_str());
+            CHECK(from_c_string.value == reference.value);
+            CHECK(from_c_string.errors == reference.errors);
+
+            const std::list<char> l(s.begin(), s.end());
+            json j;
+            RecoveringDomParser sax(j);
+            CHECK(!json::sax_parse(l.begin(), l.end(), &sax));
+            CHECK(j == reference.value);
+            CHECK(sax.errors == reference.errors);
+
+            std::istringstream ss(s);
+            const auto from_stream = parse_recovering(ss);
+            CHECK(from_stream.value == reference.value);
+            CHECK(from_stream.errors == reference.errors);
+        }
+    }
+
+    SECTION("long runs of errors")
+    {
+        // no error may copy all the input read before it
+        const auto closing = parse_recovering("[" + std::string(100000, '}'));
+        CHECK(closing.balanced);
+        CHECK(closing.value == json::array());
+
+        const auto garbage = parse_recovering("[" + std::string(100000, 'x') + "]");
+        CHECK(garbage.balanced);
+        CHECK(garbage.errors.size() == 1);
+
+        const auto commas = parse_recovering("{" + std::string(100000, ',') + "}");
+        CHECK(commas.balanced);
+        CHECK(commas.value == json::object());
+    }
+
+    SECTION("mutations of valid input")
+    {
+        // whatever the input, the events are balanced, every error is reported
+        // at most once, and valid input is parsed as usual
+        const std::vector<std::string> documents =
+        {
+            R"({"name": "value", "list": [1, -2.5, true, null, {"x": [[]]}], "e": "\u00e9"})",
+            R"([{"a": [1, 2, {"b": "c"}]}, [], {}, "\ud83d\ude00", 1e10])",
+            "{\"\xC3\xA9\": \"\xF0\x9F\x98\x80\"}",
+            R"(  {"k" : [ "v" , 0 ] }  )",
+        };
+        // each character that can be inserted, including a null byte
+        const std::string insertions("[]{},:\"x\\\0\xFF", 11);
+
+        std::vector<std::string> inputs;
+        for (const auto& doc : documents)
+        {
+            for (std::size_t i = 0; i <= doc.size(); ++i)
+            {
+                inputs.push_back(doc.substr(0, i));
+                if (i < doc.size())
+                {
+                    inputs.push_back(doc.substr(0, i) + doc.substr(i + 1));
+                }
+                for (const char c : insertions)
+                {
+                    inputs.push_back(doc.substr(0, i) + c + doc.substr(i));
+                }
+            }
+        }
+
+        for (const auto& s : inputs)
+        {
+            CAPTURE(s)
+            const auto result = parse_recovering(s);
+            CHECK(result.balanced);
+            CHECK(result.errors.size() <= s.size() + 1);
+            CHECK(result.events <= (4 * s.size()) + 4);
+            if (json::accept(s))
+            {
+                CHECK(result.ok);
+                CHECK(result.errors.empty());
+                CHECK(result.value == json::parse(s));
+            }
+            else
+            {
+                CHECK(!result.ok);
+                CHECK(!result.errors.empty());
+            }
+        }
+    }
+}
+
+TEST_CASE("accept() agrees with parse() on floats around the overflow limit")
+{
+    // accept() does not convert a float that cannot overflow; the tokens
+    // around the limit (and beyond it) must still be rejected exactly as
+    // parse() rejects them
+    const std::vector<const char*> numbers =
+    {
+        "1e0", "123.456e-5", "1e-400", "-1e-400", "1e307", "1e308", "1e309", "-1e309",
+        "10e307", "100e307", "0.1e309", "0.1e310", "1E+308", "1E+309",
+        "1.7976931348623157e308", "1.7976931348623159e308", "1.8e308", "-1.7976931348623157e308",
+        "12345678901234567e291", "12345678901234567e292", "123456789012345678e290", "123456789012345678e291",
+        "1234567890123456789e289", "1234567890123456789e290", "9999999999999999999e289", "9999999999999999999e290",
+        "0.00000000000000000001e328", "0.00000000000000000001e329"
+    };
+
+    for (const char* number : numbers)
+    {
+        for (const std::string& text :
+                {
+                    std::string(number), "[" + std::string(number) + "]", "{\"a\":" + std::string(number) + "}"
+                })
+        {
+            INFO(text);
+            const bool parses = !json::parse(text, nullptr, false).is_discarded();
+            CHECK(json::accept(text) == parses);
+        }
+    }
+}
+
+namespace
+{
+// a DOM parser that reports parse errors without throwing, as in the sax_no_exception example
+template<typename BasicJsonType>
+class quiet_dom_parser : public nlohmann::detail::json_sax_dom_parser<BasicJsonType, nlohmann::detail::string_input_adapter_type>
+{
+  public:
+    explicit quiet_dom_parser(BasicJsonType& j)
+        : nlohmann::detail::json_sax_dom_parser<BasicJsonType, nlohmann::detail::string_input_adapter_type>(j, false)
+    {}
+
+    bool parse_error(std::size_t /*unused*/, const std::string& /*unused*/, const nlohmann::detail::exception& /*unused*/)
+    {
+        return false;
+    }
+};
+
+template<typename BasicJsonType>
+std::string read_before_error(const std::string& text)
+{
+    BasicJsonType j;
+    quiet_dom_parser<BasicJsonType> sax(j);
+    CHECK(!BasicJsonType::sax_parse(text, &sax));
+    return j.dump();
+}
+} // namespace
+
+TEST_CASE("a json_sax_dom_parser keeps the values read before a parse error")
+{
+    // a json_sax_dom_parser that a user constructs stores every value in place, so a
+    // handler that does not throw on a parse error sees what was read before it
+    const std::vector<std::pair<std::string, std::string>> cases =
+    {
+        {"[1,2,3,]", "[1,2,3]"},
+        {"[[1,[2,3],4],[5,", "[[1,[2,3],4],[5]]"},
+        {R"({"a":1,"b":[1,2,{"c":)", R"({"a":1,"b":[1,2,{"c":null}]})"},
+        {R"({"a":1,"a":)", R"({"a":1})"},
+        {"[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,]", "[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]"},
+        {R"({"k0":0,"k1":1,"k2":{"x":[true,)", R"({"k0":0,"k1":1,"k2":{"x":[true]}})"},
+        {R"([{},[],{"a":[]},x)", R"([{},[],{"a":[]}])"},
+    };
+    for (const auto& c : cases)
+    {
+        CAPTURE(c.first)
+        CHECK(read_before_error<json>(c.first) == c.second);
+        CHECK(read_before_error<nlohmann::ordered_json>(c.first) == c.second);
+    }
+}

tests/src/unit-json_view.cpp

@@ -185,6 +185,33 @@ TEST_CASE("json_view")
         // duplicate keys: the last value, at the position of the first key
         CHECK(materialized<json_document>(R"({"a":1,"b":2,"a":3})") == json::parse(R"({"a":1,"b":2,"a":3})"));
         CHECK(materialized<ordered_json_document>(R"({"a":1,"b":2,"a":3})").dump() == R"({"a":3,"b":2})");
+        // scalars (made without the handler), and nesting around the depth up to which the stack of open containers is local
+        for (const char* scalar :
+                {"null", "true", "false", "0", "-7", "18446744073709551615", "1.5e300", "-0.0", "\"\"", "\"short\"", "\"a string that is longer than fifteen bytes\"", R"("\u00e4\n")"
+                })
+        {
+            CAPTURE(scalar)
+            CHECK(materialized<json_document>(scalar) == json::parse(scalar));
+            const std::string in_array = std::string("[1,") + scalar + "]";
+            const json_document d = json_document::parse(in_array);
+            CHECK(d.root()[1].materialize() == json::parse(scalar));
+            CHECK(d.root()[1].get<json>() == json::parse(scalar));
+        }
+        for (const std::size_t levels :
+                {
+                    std::size_t{1}, std::size_t{15}, std::size_t{16}, std::size_t{17}, std::size_t{32}, std::size_t{33}
+                })
+        {
+            CAPTURE(levels)
+            const std::string nested = std::string(levels, '[') + R"({"k":[1,{"j":2}]})" + std::string(levels, ']');
+            CHECK(materialized<json_document>(nested) == json::parse(nested));
+            std::string wide = R"({"a":)";
+            wide += nested;
+            wide += R"(,"b":)";
+            wide += nested;
+            wide += "}";
+            CHECK(materialized<json_document>(wide) == json::parse(wide));
+        }
         // very deep nesting (iterative, as parse())
         const std::string deep = std::string(100000, '[') + std::string(100000, ']');
         // (one check per step, so that a failure shows which step it is)
@@ -410,6 +437,31 @@ TEST_CASE("json_view")
         std::istringstream rvalue_source("[1]");
         CHECK(json_document::parse_copy(std::move(rvalue_source)).owns_source());
 
+        // a C stream is read in blocks as well (more than one block, and a
+        // text whose length is a multiple of the block size)
+        for (const std::size_t extra :
+                {
+                    std::size_t{0}, std::size_t{1}
+                })
+        {
+            std::FILE* long_file = std::tmpfile(); // NOLINT(cppcoreguidelines-owning-memory)
+            if (long_file != nullptr)
+            {
+                std::string padded = text;
+                padded.append(((4096 - (padded.size() % 4096)) % 4096) + extra, ' ');
+                CHECK(std::fwrite(padded.data(), 1, padded.size(), long_file) == padded.size());
+                CHECK(std::fseek(long_file, 0, SEEK_SET) == 0);
+                const json_document from_long_file = json_document::parse(long_file);
+                CHECK(from_long_file.source() == padded);
+                CHECK(from_long_file.root().materialize() == json::parse(text));
+                CHECK(std::fclose(long_file) == 0); // NOLINT(cppcoreguidelines-owning-memory)
+            }
+        }
+        {
+            json_document from_null;
+            CHECK_THROWS_AS(from_null = json_document::parse(static_cast<std::FILE*>(nullptr)), json::parse_error&);
+        }
+
         // a stream without a buffer is an empty input
         std::istream no_buffer(nullptr);
         CHECK(json_document::parse(no_buffer, false).is_discarded());
@@ -1006,6 +1058,49 @@ TEST_CASE("json_view element access and iteration")
 #endif
     }
 
+    SECTION("arrays whose elements have no subtrees: index and back()")
+    {
+        // (the elements are adjacent in the index; an array, an object, or an
+        // empty container anywhere makes the lookup skip element by element)
+        const std::vector<std::string> texts =
+        {
+            "[]", "[1]", "[1,2,3]", "[1,\"two\",3.5,true,null,-7,18446744073709551615]", "[[]]", "[{}]", "[1,[2],3]",
+            "[1,2,{\"a\":3}]", "[[1,2],3,4]", "[1,2,3,[4,5,6,7,8],9,10,11,12]", "[{\"a\":[1]},2]", R"(["a\u00e4","b"])"
+        };
+        for (const std::string& text : texts)
+        {
+            CAPTURE(text)
+            const json_document d = json_document::parse(text);
+            const json_view v = d.root();
+            const json expected = json::parse(text);
+            for (std::size_t i = 0; i < expected.size(); ++i)
+            {
+                CHECK(v[i].materialize() == expected[i]);
+                CHECK(v.at(i).materialize() == expected[i]);
+            }
+            CHECK(v[expected.size()].is_discarded());
+            if (!expected.empty())
+            {
+                CHECK(v.back().materialize() == expected.back());
+            }
+        }
+        // a long array (the direct lookup is not limited to short ones), also with a container at its end
+        std::string text = "[0";
+        for (int i = 1; i < 1000; ++i)
+        {
+            text += "," + std::to_string(i);
+        }
+        const json_document scalars = json_document::parse(text + "]");
+        CHECK(scalars.root()[999].get<int>() == 999);
+        CHECK(scalars.root()[500].get<int>() == 500);
+        CHECK(scalars.root().at(1).get<int>() == 1);
+        CHECK(scalars.root().back().get<int>() == 999);
+        const json_document container_last = json_document::parse(text + ",[1000]]");
+        CHECK(container_last.root()[999].get<int>() == 999);
+        CHECK(container_last.root()[1000][0].get<int>() == 1000);
+        CHECK(container_last.root().back()[0].get<int>() == 1000);
+    }
+
     SECTION("iterators")
     {
         const json_document d = json_document::parse(R"({"x":[1,{"y":2}],"z":null})");
@@ -1045,6 +1140,28 @@ TEST_CASE("json_view element access and iteration")
         CHECK((*first++).key() == "0");
         CHECK((*first).key() == "1");
         CHECK(++first == items.end());
+
+        // the keys of array elements are their indexes, whether or not they are asked for
+        std::string numbers = "[0";
+        for (int i = 1; i < 1234; ++i)
+        {
+            numbers += "," + std::to_string(i);
+        }
+        const json_document array = json_document::parse(numbers + "]");
+        std::size_t position = 0;
+        for (const auto& item : array.root().items())
+        {
+            CHECK(item.value().get<std::size_t>() == position);
+            if (position % 3 != 0) // (not for every item: values alone do not make the key)
+            {
+                CHECK(item.key() == std::to_string(position));
+                CHECK(item.key().data() == item.key().data()); // (the same text every time)
+                const json_view::item copy = item;
+                CHECK(copy.key() == std::to_string(position));
+            }
+            ++position;
+        }
+        CHECK(position == 1234);
 #ifdef JSON_HAS_CPP_17
         std::string pairs;
         for (const auto [key, value] : d.root().items())
@@ -1383,7 +1500,17 @@ TEST_CASE("json_view values")
         CHECK(exception_of([&] { static_cast<void>(v.value("s", 0)); }) == without_path(exception_of([&] { static_cast<void>(j.value("s", 0)); })));
         CHECK(exception_of([&] { static_cast<void>(v["n"].value("x", 0)); }) == without_path(exception_of([&] { static_cast<void>(j["n"].value("x", 0)); })));
         CHECK(exception_of([&] { static_cast<void>(v["n"].value(json::json_pointer("/x"), 0)); }) == without_path(exception_of([&] { static_cast<void>(j["n"].value(json::json_pointer("/x"), 0)); })));
+        // string defaults: the member is not a string, or the value is neither an object nor an array
+        CHECK(exception_of([&] { static_cast<void>(v.value("n", "default")); }) == without_path(exception_of([&] { static_cast<void>(j.value("n", "default")); })));
+        CHECK(exception_of([&] { static_cast<void>(v["n"].value("x", "default")); }) == without_path(exception_of([&] { static_cast<void>(j["n"].value("x", "default")); })));
+        CHECK(exception_of([&] { static_cast<void>(v.value(json::json_pointer("/n"), "default")); }) == without_path(exception_of([&] { static_cast<void>(j.value(json::json_pointer("/n"), "default")); })));
+        CHECK(exception_of([&] { static_cast<void>(v["s"].value(json::json_pointer("/x"), "default")); }) == without_path(exception_of([&] { static_cast<void>(j["s"].value(json::json_pointer("/x"), "default")); })));
 #endif
+        // (defaults and members longer than the short string optimization)
+        CHECK(v.value("s", "a default longer than fifteen characters") == "text");
+        CHECK(v.value("missing", "a default longer than fifteen characters") == "a default longer than fifteen characters");
+        CHECK(v.value(json::json_pointer("/s"), "a default longer than fifteen characters") == "text");
+        CHECK(v.value(json::json_pointer("/zz"), "a default longer than fifteen characters") == "a default longer than fifteen characters");
     }
 }
 

tests/src/unit-json_view_builder.cpp

@@ -523,6 +523,58 @@ TEST_CASE("json_view builder: strings across vector blocks")
     }
 }
 
+TEST_CASE("json_view builder: escapes between runs of text")
+{
+    // The text between escapes is copied 16 bytes at a time where there are
+    // that many bytes left, and with the UTF-8 check where there is
+    // non-ASCII text. Runs of every length from 0 to 40 between valid and
+    // invalid escapes, UTF-8 sequences, and control characters, in strings
+    // that are cut at all places near the end of the text.
+    const std::array<const char*, 9> valid = {{"\\n", "\\\"", "\\\\", "\\/", "\\t", "\\u00e4", "\\u20ac", "\\ud83d\\ude00", "\\u0000"}};
+    const std::array<const char*, 9> invalid = {{"\\x", "\\ud800", "\\udc00", "\\u12G4", "\\u12", "\\ud83dx", "\\ud83d\\u0041", "\\", "\x01"}};
+    const std::array<const char*, 6> utf8 = {{"\xc3\xa4", "\xe2\x82\xac", "\xf0\x9f\x98\x80", "\xc3", "\xff", "\x80"}};
+    std::mt19937 rng(2026); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed): reproducible
+    for (std::size_t run = 0; run <= 40; ++run)
+    {
+        for (int i = 0; i < 60; ++i)
+        {
+            std::string content = std::string(rng() % 20, 'a');
+            const auto pieces = 1 + (rng() % 6);
+            for (unsigned k = 0; k < pieces; ++k)
+            {
+                content += valid[rng() % valid.size()];
+                content += std::string(k == 0 ? run : rng() % 41, static_cast<char>('b' + (rng() % 20)));
+                switch (rng() % 8)
+                {
+                    case 0:
+                        content += utf8[rng() % 3];
+                        break;
+                    case 1:
+                        content += utf8[rng() % utf8.size()];
+                        break;
+                    case 2:
+                        content += invalid[rng() % invalid.size()];
+                        break;
+                    default:
+                        break;
+                }
+            }
+            check_string(content);
+            check_string(content.substr(0, content.size() - (rng() % 4)));
+        }
+    }
+    // the text before the first escape, and runs and escapes at the end of the text
+    for (std::size_t before = 0; before <= 40; ++before)
+    {
+        for (const char* escape : valid)
+        {
+            check_string(std::string(before, 'a') + escape);
+            check_string(std::string(before, 'a') + escape + std::string(before % 19, 'z'));
+            check_string(std::string(before, 'a') + escape + escape + std::string(before % 19, 'z'));
+        }
+    }
+}
+
 TEST_CASE("json_view node integer bits")
 {
     using nlohmann::detail::view::integer_bits;

tests/src/unit-json_view_dump.cpp

@@ -21,6 +21,7 @@ using nlohmann::ordered_json_view;
 using json_view_test::generator;
 using json_view_test::has_duplicate_keys;
 
+#include <algorithm>
 #include <array>
 #include <cmath>
 #include <cstddef>
@@ -29,6 +30,7 @@ using json_view_test::has_duplicate_keys;
 #include <iomanip>
 #include <map>
 #include <random>
+#include <set>
 #include <sstream>
 #include <string>
 #include <vector>
@@ -186,6 +188,133 @@ TEST_CASE("json_view dump")
         CHECK(deep_document.root().dump() == deep);
     }
 
+    SECTION("ensure_ascii: a character that needs an escape at each position of a string")
+    {
+        // string lengths around the width of the vector steps, in a string value, a key, and a value near the end of the text
+        for (const char* const special :
+                {
+                    "\xC3\xA9", "\xE4\xB8\xAD", "\xF0\x9F\x98\x80", "\x7F", "\\n", "\\u0001", "\\\"", "\\\\"
+                })
+        {
+            for (std::size_t length = 0; length <= 40; ++length)
+            {
+                for (std::size_t position = 0; position <= length; ++position)
+                {
+                    std::string content(length, 'a');
+                    content.insert(position, special);
+                    const std::array<std::string, 3> texts =
+                    {
+                        {
+                            "\"" + content + "\"",
+                            "[\"" + content + "\"]",
+                            "{\"" + content + "\":1}",
+                        }
+                    };
+                    for (const std::string& text : texts)
+                    {
+                        CAPTURE(text)
+                        const ordered_json_document d = ordered_json_document::parse(text);
+                        const ordered_json j = ordered_json::parse(text);
+                        CHECK(d.root().dump(-1, ' ', true) == j.dump(-1, ' ', true));
+                        CHECK(d.root().dump(2, ' ', true) == j.dump(2, ' ', true));
+                        CHECK(d.root().dump() == j.dump());
+                    }
+                }
+            }
+        }
+    }
+
+    SECTION("ensure_ascii: every character of the Basic Multilingual Plane and some beyond")
+    {
+        // (the two- and three-byte sequences, including the ranges that need a check: E0, ED, EF BF BF)
+        std::string text = "[";
+        std::size_t in_chunk = 0;
+        const auto append = [&](std::uint32_t cp)
+        {
+            if (cp < 0x80)
+            {
+                if (cp < 0x20 || cp == '"' || cp == '\\')
+                {
+                    return;
+                }
+                text += static_cast<char>(cp);
+            }
+            else if (cp < 0x800)
+            {
+                text += static_cast<char>(0xC0 | (cp >> 6));
+                text += static_cast<char>(0x80 | (cp & 0x3F));
+            }
+            else if (cp < 0x10000)
+            {
+                text += static_cast<char>(0xE0 | (cp >> 12));
+                text += static_cast<char>(0x80 | ((cp >> 6) & 0x3F));
+                text += static_cast<char>(0x80 | (cp & 0x3F));
+            }
+            else
+            {
+                text += static_cast<char>(0xF0 | (cp >> 18));
+                text += static_cast<char>(0x80 | ((cp >> 12) & 0x3F));
+                text += static_cast<char>(0x80 | ((cp >> 6) & 0x3F));
+                text += static_cast<char>(0x80 | (cp & 0x3F));
+            }
+        };
+        for (std::uint32_t cp = 0x20; cp < 0x10400; ++cp)
+        {
+            if (cp >= 0xD800 && cp < 0xE000)
+            {
+                continue;
+            }
+            if (in_chunk == 0)
+            {
+                text += text.size() > 1 ? ",\"" : "\"";
+            }
+            append(cp);
+            if (++in_chunk == 37)
+            {
+                text += "\"";
+                in_chunk = 0;
+            }
+        }
+        text += in_chunk != 0 ? "\"]" : "]";
+        const ordered_json_document d = ordered_json_document::parse(text);
+        const ordered_json j = ordered_json::parse(text);
+        CHECK(d.root().dump(-1, ' ', true) == j.dump(-1, ' ', true));
+        CHECK(d.root().dump(2, ' ', true) == j.dump(2, ' ', true));
+        CHECK(d.root().dump() == j.dump());
+        CHECK(d.root().dump(2) == j.dump(2));
+    }
+
+    SECTION("deep nesting, indented")
+    {
+        // indentations of up to and beyond the width of the writer's fixed-size copy
+        for (const int depth :
+                {
+                    1, 5, 20, 70, 300
+                })
+        {
+            std::string text;
+            for (int i = 0; i < depth; ++i)
+            {
+                text += i % 2 == 0 ? "[1," : R"({"k":2,"\u00e9":)";
+            }
+            text += "[]";
+            for (int i = depth - 1; i >= 0; --i)
+            {
+                text += i % 2 == 0 ? "]" : "}";
+            }
+            const ordered_json_document d = ordered_json_document::parse(text);
+            const ordered_json j = ordered_json::parse(text);
+            for (const int indent :
+                    {
+                        0, 1, 2, 3, 8
+                    })
+            {
+                CHECK(d.root().dump(indent) == j.dump(indent));
+                CHECK(d.root().dump(indent, '\t') == j.dump(indent, '\t'));
+            }
+        }
+    }
+
     SECTION("the output buffer of a small value is small")
     {
         // an escaped key after the value: its node lies in the arena, so the
@@ -598,6 +727,152 @@ TEST_CASE("json_view large objects")
         CHECK(without_collisions.memory_usage() >= with_collisions.memory_usage() + (table_size * sizeof(std::uint32_t)));
     }
 
+    SECTION("keys that differ in their last bytes are spread over the table")
+    {
+        // The hash must spread the usual families of keys (numbered names, hex
+        // strings, dates, short names): a key farther than index_max_displacement
+        // from its home slot would leave the object without a table. The keys
+        // are placed with linear probing, as the table does.
+        const auto max_displacement = [](const std::vector<std::string>& keys)
+        {
+            std::size_t slots = 16;
+            while (slots < 2 * keys.size())
+            {
+                slots *= 2;
+            }
+            std::vector<bool> used(slots, false);
+            std::size_t farthest = 0;
+            for (const std::string& key : keys)
+            {
+                std::size_t i = nlohmann::detail::conditional_static_cast<std::size_t>(nlohmann::detail::view::key_hash(key.data(), key.size())) & (slots - 1);
+                std::size_t distance = 0;
+                while (used[i])
+                {
+                    i = (i + 1) & (slots - 1);
+                    ++distance;
+                }
+                used[i] = true;
+                farthest = (std::max)(farthest, distance);
+            }
+            return farthest;
+        };
+        // the keys of a printf-like pattern with one %d, %0Nd or %0Nx, formatted with a stream
+        // (a format string that is not a literal is a Flawfinder error)
+        const auto numbered = [](const char* format, int count, int step)
+        {
+            const std::string pattern(format);
+            const std::size_t percent = pattern.find('%');
+            std::size_t conversion = percent + 1;
+            std::size_t width = 0;
+            while (pattern[conversion] >= '0' && pattern[conversion] <= '9')
+            {
+                width = (width * 10) + static_cast<std::size_t>(pattern[conversion] - '0');
+                ++conversion;
+            }
+            const bool hex = pattern[conversion] == 'x';
+            std::vector<std::string> keys;
+            for (int i = 0; i < count; ++i)
+            {
+                std::ostringstream key;
+                key << pattern.substr(0, percent) << std::setfill('0') << std::setw(static_cast<int>(width))
+                    << (hex ? std::hex : std::dec) << (i * step) << pattern.substr(conversion + 1);
+                keys.push_back(key.str());
+            }
+            return keys;
+        };
+        for (const char* format :
+                {"key%d", "%d", "item_%04d", "a%05d", "%03dxyz", "id%d", "user_%d", "%d_suffix", "0x%04x", "A%07d", "12345678%d", "1234567%d", "123456789%d",
+                 "configuration_parameter_value_%04d_extra", "2026-01-%02dT00:00:00Z"
+                })
+        {
+            CAPTURE(format)
+            CHECK(max_displacement(numbered(format, std::string(format).find("%02d") != std::string::npos ? 60 : 1000, 1)) < nlohmann::detail::view::index_max_displacement); // NOLINT(abseil-string-find-str-contains)
+            CHECK(max_displacement(numbered(format, 600, 7)) < nlohmann::detail::view::index_max_displacement);
+        }
+        std::mt19937_64 rng(7); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed)
+        for (const std::size_t length :
+                {
+                    std::size_t{1}, std::size_t{2}, std::size_t{3}, std::size_t{5}, std::size_t{7}, std::size_t{8}, std::size_t{9}, std::size_t{12}, std::size_t{16}, std::size_t{17}, std::size_t{33}
+                })
+        {
+            CAPTURE(length)
+            std::set<std::string> distinct;
+            // (there are 26 and 676 keys of one and two letters)
+            std::size_t wanted = 700;
+            if (length == 1)
+            {
+                wanted = 26;
+            }
+            else if (length == 2)
+            {
+                wanted = 500;
+            }
+            while (distinct.size() < wanted)
+            {
+                std::string key(length, 'a');
+                for (char& c : key)
+                {
+                    c = static_cast<char>('a' + (rng() % 26));
+                }
+                distinct.insert(key);
+            }
+            CHECK(max_displacement(std::vector<std::string>(distinct.begin(), distinct.end())) < nlohmann::detail::view::index_max_displacement);
+        }
+    }
+
+    SECTION("keys of every length that differ in one byte")
+    {
+        // In a large object, a key is found by comparing its bytes with those of the member the
+        // table points to: up to 16 bytes as two overlapping words, longer ones with memcmp.
+        // Keys of every length from 0 to 40, and keys that differ from them in one byte.
+        std::mt19937_64 rng(2026); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed)
+        std::vector<std::string> keys;
+        std::set<std::string> distinct;
+        for (std::size_t length = 0; length <= 40; ++length)
+        {
+            for (int copy = 0; copy < 5; ++copy)
+            {
+                std::string key(length, 'a');
+                for (char& c : key)
+                {
+                    c = static_cast<char>('a' + (rng() % 26));
+                }
+                if (distinct.insert(key).second)
+                {
+                    keys.push_back(key);
+                }
+            }
+        }
+        std::string text = "{";
+        for (std::size_t i = 0; i < keys.size(); ++i)
+        {
+            text += (i != 0 ? ",\"" : "\"") + keys[i] + "\":" + std::to_string(i);
+        }
+        text += "}";
+        REQUIRE(keys.size() >= 128); // (a table is built for 128 members)
+        const json_document d = json_document::parse(text);
+        const json_view v = d.root();
+        for (std::size_t i = 0; i < keys.size(); ++i)
+        {
+            CAPTURE(keys[i])
+            CHECK(v[keys[i]].get<std::size_t>() == i);
+            CHECK(v.at(keys[i]).get<std::size_t>() == i);
+            CHECK(v.contains(keys[i]));
+            for (std::size_t at = 0; at < keys[i].size(); ++at)
+            {
+                std::string other = keys[i];
+                other[at] = other[at] == 'z' ? 'y' : 'z';
+                CAPTURE(other)
+                CHECK(v.contains(other) == (distinct.count(other) != 0));
+            }
+            CHECK(v.contains(keys[i] + "a") == (distinct.count(keys[i] + "a") != 0));
+            if (!keys[i].empty())
+            {
+                CHECK(v.contains(keys[i].substr(1)) == (distinct.count(keys[i].substr(1)) != 0));
+            }
+        }
+    }
+
     SECTION("shrink_to_fit releases the tables' spare capacity")
     {
         const auto make_text = [](int objects, int members)

tests/src/unit-json_view_edit.cpp

@@ -661,6 +661,115 @@ TEST_CASE("json_view edits: replacing arrays and objects by scalars")
     CHECK(d.root().materialize() == expected);
 }
 
+TEST_CASE("json_view edits: arrays whose elements have no subtrees")
+{
+    // such an array is indexed directly while it is in its parsed layout; edits
+    // that replace elements by scalars keep it, others move it
+    json_editable_document d = json_editable_document::parse("[10,20,30,40]");
+    json expected = json::array({10, 20, 30, 40});
+    const auto check = [&]
+    {
+        CHECK(d.root().size() == expected.size());
+        for (std::size_t i = 0; i < expected.size(); ++i)
+        {
+            CHECK(d.root()[i].materialize() == expected[i]);
+            CHECK(d.root().at(i).materialize() == expected[i]);
+        }
+        CHECK(d.root()[expected.size()].is_discarded());
+        CHECK(d.root().back().materialize() == expected.back());
+        CHECK(d.root().dump() == expected.dump());
+    };
+    check();
+    d.set(d.root()[1], "x");
+    expected[1] = "x";
+    check();
+    d.set(d.root()[3], 2.5);
+    expected[3] = 2.5;
+    check();
+    d.set(d.root()[2], json::array({1, 2}));
+    expected[2] = json::array({1, 2});
+    check();
+    d.push_back(d.root(), 50);
+    expected.push_back(50);
+    check();
+    d.insert(d.root(), 0, true);
+    expected.insert(expected.begin(), true);
+    check();
+    d.erase(d.root(), 1);
+    expected.erase(expected.begin() + 1);
+    check();
+    d.set(d.root()[2], nullptr);
+    expected[2] = nullptr;
+    check();
+}
+
+TEST_CASE("json_view edits: materializing values nested deeper than the document was parsed")
+{
+    // the stack of open containers of materialize() is sized from the nesting of the parsed text;
+    // a value added by an edit can be deeper than that (and than the stack that is local to the call)
+    json_editable_document d = json_editable_document::parse("[1,[2]]");
+    json expected = json::array({1, json::array({2})});
+    json nested = 3;
+    for (int level = 0; level < 40; ++level)
+    {
+        nested = json::array({nested, json{{"k", level}}});
+    }
+    d.push_back(d.root(), nested);
+    expected.push_back(nested);
+    CHECK(d.root().materialize() == expected);
+    CHECK(d.root()[2].materialize() == nested);
+    d.set(d.root()[1][0], nested);
+    expected[1][0] = nested;
+    CHECK(d.root().materialize() == expected);
+    CHECK(d.root().get<json>() == expected);
+}
+
+TEST_CASE("json_view edits: an editable document before and after its first edit")
+{
+    // until the first edit, an editable document is read like a read-only one; views made before the edit
+    // keep working after it
+    const std::string text = R"({"a":[1,2,{"x":"y"},4],"b":{"c":true,"d":null},"e":"text","f":[]})";
+    const json_document read_only = json_document::parse(text);
+    json_editable_document d = json_editable_document::parse(text);
+    const auto root = d.root();
+    const auto same_as_read_only = [&]
+    {
+        CHECK(root.dump() == read_only.root().dump());
+        CHECK(root.dump(2) == read_only.root().dump(2));
+        CHECK(root.materialize() == read_only.root().materialize());
+        CHECK(root["a"][2]["x"].get_string() == "y");
+        CHECK(root["a"].at(3).get<int>() == 4);
+        CHECK(root["a"].back().get<int>() == 4);
+        CHECK(root.contains("b"));
+        CHECK(!root.contains("zz"));
+        CHECK(root.find("e")->get_string() == "text");
+        CHECK(root.find("zz") == root.end());
+        CHECK(root["b"]["c"].get<bool>());
+        CHECK(root.value("e", "none") == "text");
+        std::size_t count = 0;
+        for (const auto m : root["a"])
+        {
+            CHECK(m.materialize() == read_only.root()["a"][count].materialize());
+            ++count;
+        }
+        CHECK(count == 4);
+    };
+    same_as_read_only();
+    // an edit that changes a scalar in place, then one that moves an array
+    d.set(root["e"], "other");
+    CHECK(root["e"].get_string() == "other");
+    CHECK(root["a"][2]["x"].get_string() == "y");
+    CHECK(root["a"].back().get<int>() == 4);
+    d.push_back(root["f"], 7);
+    CHECK(root["f"][0].get<int>() == 7);
+    CHECK(root["f"].back().get<int>() == 7);
+    CHECK(root.dump() == R"({"a":[1,2,{"x":"y"},4],"b":{"c":true,"d":null},"e":"other","f":[7]})");
+    d.set(root["a"], 2, json::array({1, 2}));
+    CHECK(root["a"][2][1].get<int>() == 2);
+    CHECK(root["a"].dump() == "[1,2,[1,2],4]");
+    CHECK(root.materialize() == json::parse(R"({"a":[1,2,[1,2],4],"b":{"c":true,"d":null},"e":"other","f":[7]})"));
+}
+
 namespace
 {
 // reads of "a" in an object that repeats it: the first member wins, however the

tests/src/unit-json_view_image.cpp

@@ -682,6 +682,26 @@ TEST_CASE("json_view images: ownership")
 #if !defined(JSON_NOEXCEPTION)
 TEST_CASE("json_view images: errors")
 {
+    SECTION("invalid UTF-8 of a decoded string: the compact dump writes it, the others throw")
+    {
+        const json_document d = json_document::parse(R"({"k\n":"\u00e9x"})");
+        std::vector<std::uint8_t> image = d.save();
+        const std::size_t arena = text_at(image) + header_size_field(image, 16) + 1;
+        image[arena + 3] = 0x28; // the key is "k\n"; the value starts at 3: 0xC3 0x28 is invalid
+        const json_document loaded = json_document::load(image, image_check::bounds);
+        CHECK(loaded.root().dump().size() == 13);
+        for (const int indent :
+                {
+                    0, 2
+                })
+        {
+            CHECK_THROWS_AS(loaded.root().dump(indent), json::type_error);
+            CHECK_THROWS_AS(loaded.root().dump(indent, ' ', true), json::type_error);
+        }
+        CHECK_THROWS_AS(loaded.root().dump(-1, ' ', true), json::type_error);
+        CHECK_THROWS_WITH_AS(loaded.root().dump(2), "[json.exception.type_error.316] invalid UTF-8 byte at index 1: 0x28", json::type_error&);
+    }
+
     SECTION("a literal as the root: dump() after loading")
     {
         for (const char* text :

tests/src/unit-msgpack.cpp

@@ -31,8 +31,6 @@ using nlohmann::json;
 #include "make_test_data_available.hpp"
 #include "round_trip_corpus.hpp"
 #include "test_utils.hpp"
-#include "sax_countdown.hpp"
-using utils::SaxCountdown;
 
 
 TEST_CASE("MessagePack")
@@ -1645,30 +1643,6 @@ TEST_CASE("MessagePack")
             }
         }
     }
-
-    SECTION("SAX aborts")
-    {
-        SECTION("start_array(len)")
-        {
-            std::vector<uint8_t> const v = {0x93, 0x01, 0x02, 0x03};
-            SaxCountdown scp(0);
-            CHECK(!json::sax_parse(v, &scp, json::input_format_t::msgpack));
-        }
-
-        SECTION("start_object(len)")
-        {
-            std::vector<uint8_t> const v = {0x81, 0xa3, 0x66, 0x6F, 0x6F, 0xc2};
-            SaxCountdown scp(0);
-            CHECK(!json::sax_parse(v, &scp, json::input_format_t::msgpack));
-        }
-
-        SECTION("key()")
-        {
-            std::vector<uint8_t> const v = {0x81, 0xa3, 0x66, 0x6F, 0x6F, 0xc2};
-            SaxCountdown scp(1);
-            CHECK(!json::sax_parse(v, &scp, json::input_format_t::msgpack));
-        }
-    }
 }
 
 TEST_CASE("issue #5405 - array reserve for definite-length MessagePack arrays")
@@ -1739,21 +1713,6 @@ TEST_CASE("issue #5405 - array reserve for definite-length MessagePack arrays")
             CHECK(json::from_msgpack(packed) == j);
         }
     }
-
-    SECTION("a user-defined SAX consumer is unaffected by the internal DOM reserve optimization")
-    {
-        // the reserve() call is local to json_sax_dom_parser / json_sax_dom_callback_parser;
-        // a custom SAX consumer that does not touch a DOM array sees identical events
-        json j = json::array();
-        for (int i = 0; i < 100; ++i)
-        {
-            j.push_back(i);
-        }
-        const auto packed = json::to_msgpack(j);
-
-        SaxCountdown scp(1000000); // large enough to never trigger an abort
-        CHECK(json::sax_parse(packed, &scp, json::input_format_t::msgpack));
-    }
 }
 
 TEST_CASE("regression test - MessagePack ext type rejects a subtype that doesn't fit a single byte")
@@ -1792,15 +1751,6 @@ TEST_CASE("MessagePack nesting does not consume the call stack")
         CHECK(json::from_msgpack(input, true, false).is_discarded());
     }
 
-    SECTION("a well-formed deep value is read through the SAX interface")
-    {
-        std::vector<uint8_t> input(300000, 0x91);
-        input.push_back(0x01); // innermost value
-
-        SaxCountdown accept_all(600001);
-        CHECK(json::sax_parse(input, &accept_all, json::input_format_t::msgpack));
-    }
-
     SECTION("a well-formed deep value is read into a value")
     {
         const std::size_t depth = 10000;
@@ -1848,31 +1798,6 @@ TEST_CASE("MessagePack input that cannot be read is discarded by every overload"
 #endif
 }
 
-TEST_CASE("MessagePack SAX parsing stops at every event")
-{
-    // Containers are opened and closed by the loop that reads them; a SAX
-    // handler that rejects any event - including the end of a nested
-    // container - must stop the parse right there.
-    const auto count_events = [](const std::vector<std::uint8_t>& input)
-    {
-        int events = 0;
-        while (true)
-        {
-            SaxCountdown scp(events);
-            if (json::sax_parse(input, &scp, json::input_format_t::msgpack))
-            {
-                return events;
-            }
-            ++events;
-            REQUIRE(events < 1000);
-        }
-    };
-
-    // 20 events: every container kind closes inside another one
-    const json j = json::parse(R"({"a": [1, {"b": []}], "c": {"d": [[2]]}})");
-    CHECK(count_events(json::to_msgpack(j)) == 20);
-}
-
 TEST_CASE("single MessagePack roundtrip")
 {
     SECTION("sample.json")

tests/src/unit-ordered_json.cpp

@@ -12,6 +12,11 @@
 using nlohmann::json;
 using nlohmann::ordered_json;
 
+#include <array>
+#include <cstddef>
+#include <cstdint>
+#include <string>
+
 TEST_CASE("ordered_json")
 {
     json j;
@@ -237,3 +242,276 @@ TEST_CASE("regression test for issue #5673 - ordered_json::emplace with a non-rv
         CHECK(oj.dump() == "{\"a\":1}");
     }
 }
+
+namespace
+{
+// the sizes around which the DOM parser changes how it stores the elements of
+// an array (16) and the members of an object in a vector (32, and 64 for the
+// key index)
+const std::array<std::size_t, 17> container_sizes = {{0, 1, 2, 3, 15, 16, 17, 18, 31, 32, 33, 34, 63, 64, 65, 66, 200}};
+
+template<typename Json>
+std::string array_text(const std::size_t n, const std::size_t depth)
+{
+    std::string text = "[";
+    for (std::size_t i = 0; i < n; ++i)
+    {
+        text += (i == 0 ? "" : ",");
+        // alternate scalars, empty containers and nested containers
+        switch (i % 4)
+        {
+            case 0:
+                text += std::to_string(i);
+                break;
+            case 1:
+                text += "\"s" + std::to_string(i) + "\"";
+                break;
+            case 2:
+                text += depth == 0 ? "[]" : array_text<Json>(n / 2, depth - 1);
+                break;
+            default:
+                text += depth == 0 ? "{}" : "{\"a\":" + array_text<Json>(n / 3, depth - 1) + ",\"b\":null}";
+                break;
+        }
+    }
+    return text + "]";
+}
+
+template<typename Json>
+std::string object_text(const std::size_t n, const std::size_t depth, const bool duplicates)
+{
+    std::string text = "{";
+    for (std::size_t i = 0; i < n; ++i)
+    {
+        text += (i == 0 ? "" : ",");
+        text += "\"key" + std::to_string(i) + "\":";
+        switch (i % 4)
+        {
+            case 0:
+                text += std::to_string(i);
+                break;
+            case 1:
+                text += depth == 0 ? "[]" : array_text<Json>(n / 2, depth - 1);
+                break;
+            case 2:
+                text += depth == 0 ? "{}" : object_text<Json>(n / 2, depth - 1, duplicates);
+                break;
+            default:
+                text += "\"value\"";
+                break;
+        }
+    }
+    if (duplicates)
+    {
+        // keys that are there already, the first and the last one, with other values
+        text += n == 0 ? "" : ",";
+        text += R"("key0":[1,2],"key)";
+        text += std::to_string(n - 1);
+        text += R"(":{"again":true})";
+    }
+    return text + "}";
+}
+
+template<typename Json>
+void check_same_dump(const std::string& text)
+{
+    // the text is what dump() writes for the parsed value, and the value
+    // survives a second round trip: arrays keep their elements in order, an
+    // ordered_json keeps its members in order of appearance
+    const Json parsed = Json::parse(text);
+    const Json again = Json::parse(parsed.dump());
+    CHECK(parsed == again);
+    CHECK(parsed.dump() == again.dump());
+    CHECK(Json::accept(text));
+}
+} // namespace
+
+TEST_CASE("parsing containers of sizes around the limits of the DOM parser")
+{
+    SECTION("arrays")
+    {
+        for (const std::size_t n : container_sizes)
+        {
+            const std::string text = array_text<json>(n, 2);
+            INFO("size ", n);
+
+            const json parsed = json::parse(text);
+            CHECK(parsed.is_array());
+            CHECK(parsed.size() == n);
+            for (std::size_t i = 0; i < n; i += 4)
+            {
+                CHECK(parsed[i] == i);
+            }
+            check_same_dump<json>(text);
+            check_same_dump<ordered_json>(text);
+            CHECK(ordered_json::parse(text).dump() == parsed.dump());
+        }
+    }
+
+    SECTION("objects")
+    {
+        for (const bool duplicates :
+                {
+                    false, true
+                })
+        {
+            for (const std::size_t n : container_sizes)
+            {
+                const std::string text = object_text<json>(n, 2, duplicates);
+                INFO("size ", n, " duplicates ", duplicates);
+
+                const json parsed = json::parse(text);
+                const ordered_json ordered = ordered_json::parse(text);
+                CHECK(parsed.is_object());
+                CHECK(ordered.is_object());
+                CHECK(parsed.size() == ordered.size());
+                // the same members, in the order of their first appearance
+                std::size_t i = 0;
+                for (auto it = ordered.begin(); it != ordered.end(); ++it, ++i)
+                {
+                    if (i < n)
+                    {
+                        CHECK(it.key() == "key" + std::to_string(i));
+                    }
+                    CHECK(parsed.at(it.key()) == json(it.value()));
+                }
+                if (duplicates && n > 1)
+                {
+                    // the later value replaced the earlier one in its place
+                    CHECK(ordered.at("key0") == ordered_json::parse("[1,2]"));
+                    CHECK(ordered.at("key" + std::to_string(n - 1)) == ordered_json::parse("{\"again\":true}"));
+                    CHECK(ordered.begin().key() == "key0");
+                    CHECK(ordered.size() == n);
+                }
+                check_same_dump<json>(text);
+                check_same_dump<ordered_json>(text);
+            }
+        }
+    }
+
+    SECTION("objects with many different and repeated keys")
+    {
+        // 300 members of which a third repeat an earlier key
+        std::string text = "{";
+        for (std::size_t i = 0; i < 300; ++i)
+        {
+            text += (i == 0 ? "" : ",");
+            text += "\"k" + std::to_string(i % 200) + "\":" + std::to_string(i);
+        }
+        text += '}';
+
+        const ordered_json ordered = ordered_json::parse(text);
+        const json parsed = json::parse(text);
+        CHECK(ordered.size() == 200);
+        CHECK(parsed.size() == 200);
+        std::size_t i = 0;
+        for (auto it = ordered.begin(); it != ordered.end(); ++it, ++i)
+        {
+            CHECK(it.key() == "k" + std::to_string(i));
+            // the last value written
+            CHECK(it.value() == (i < 100 ? i + 200 : i));
+            CHECK(parsed.at(it.key()) == json(it.value()));
+        }
+    }
+}
+
+namespace
+{
+// a deterministic source of structure for parsing tests: builds a JSON text
+// and, at the same time, the value it has to parse to
+class tree_generator
+{
+  public:
+    explicit tree_generator(std::uint32_t seed) : state(seed) {}
+
+    template<typename Json>
+    Json make(std::string& text, const std::size_t depth)
+    {
+        switch (next(depth == 0 ? 5 : 9))
+        {
+            case 0:
+                text += "null";
+                return Json(nullptr);
+            case 1:
+                text += "true";
+                return Json(true);
+            case 2:
+            {
+                const int n = static_cast<int>(next(2000)) - 1000;
+                text += std::to_string(n);
+                return Json(n);
+            }
+            case 3:
+                text += "\"text\"";
+                return Json("text");
+            case 4:
+                text += "2.5";
+                return Json(2.5);
+            case 5:
+            case 6:
+            case 7:
+            {
+                Json array = Json::array();
+                text += '[';
+                const std::size_t n = size();
+                for (std::size_t i = 0; i < n; ++i)
+                {
+                    text += (i == 0 ? "" : ",");
+                    array.push_back(make<Json>(text, depth - 1));
+                }
+                text += ']';
+                return array;
+            }
+            default:
+            {
+                Json object = Json::object();
+                text += '{';
+                const std::size_t n = size();
+                for (std::size_t i = 0; i < n; ++i)
+                {
+                    const std::string key = "k" + std::to_string(i);
+                    text += (i == 0 ? "\"" : ",\"") + key + "\":";
+                    object[key] = make<Json>(text, depth - 1);
+                }
+                text += '}';
+                return object;
+            }
+        }
+    }
+
+  private:
+    std::uint32_t next(const std::uint32_t bound)
+    {
+        state = (state * 1664525U) + 1013904223U;
+        return (state >> 8) % bound;
+    }
+
+    std::size_t size()
+    {
+        return container_sizes[next(static_cast<std::uint32_t>(container_sizes.size()))];
+    }
+
+    std::uint32_t state;
+};
+} // namespace
+
+TEST_CASE("parsing random trees of arrays and objects")
+{
+    for (std::uint32_t seed = 1; seed <= 150; ++seed)
+    {
+        INFO("seed ", seed);
+
+        std::string text;
+        std::string ordered_text;
+        const json expected = tree_generator(seed).make<json>(text, 3);
+        const auto ordered_expected = tree_generator(seed).make<ordered_json>(ordered_text, 3);
+        CHECK(text == ordered_text);
+
+        CHECK(json::parse(text) == expected);
+        CHECK(json::accept(text));
+        const ordered_json ordered = ordered_json::parse(text);
+        CHECK(ordered == ordered_expected);
+        CHECK(ordered.dump() == ordered_expected.dump());
+        CHECK(json::parse(text).dump() == expected.dump());
+    }
+}

tests/src/unit-regression3.cpp

@@ -22,14 +22,6 @@
 // scoped enum, so get<std::byte>() (needed below to get<std::vector<std::byte>>()
 // from a plain JSON array, not just from an already-binary value) relies on
 // enum serialization being enabled
-// capture whether JSON_DELETE_DEPRECATED_FUNCTIONS was enabled on the command
-// line *before* including json.hpp, since the library #undefs it once the header
-// has been fully processed (see include/nlohmann/detail/macro_unscope.hpp); the
-// tests of deprecated functions are skipped if these functions are deleted
-#if defined(JSON_DELETE_DEPRECATED_FUNCTIONS) && (JSON_DELETE_DEPRECATED_FUNCTIONS == 1)
-    #define JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
-#endif
-
 #if defined(JSON_DISABLE_ENUM_SERIALIZATION) && (JSON_DISABLE_ENUM_SERIALIZATION == 1)
     #define SKIP_TESTS_FOR_ENUM_SERIALIZATION
 #endif
@@ -810,50 +802,6 @@ TEST_CASE("issue #5338 - truncated CBOR tagged binary subtype is rejected")
     }
 }
 
-TEST_CASE("issue #5676 - SAX parsing of CBOR tags")
-{
-    const json expected = json::binary({1, 2, 3}, 42);
-    const auto cbor = json::to_cbor(expected);
-
-    nlohmann::detail::json_sax_acceptor<json> acceptor;
-    CHECK_FALSE(json::sax_parse(cbor, &acceptor, json::input_format_t::cbor));
-    CHECK_FALSE(json::sax_parse(cbor, &acceptor, json::input_format_t::cbor,
-                                true, false, false, json::cbor_tag_handler_t::error));
-
-    CHECK(json::sax_parse(cbor, &acceptor, json::input_format_t::cbor,
-                          true, false, false, json::cbor_tag_handler_t::ignore));
-
-    json parsed;
-    nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::string_input_adapter_type> sax(parsed);
-    CHECK(json::sax_parse(cbor, &sax, json::input_format_t::cbor,
-                          true, false, false, json::cbor_tag_handler_t::store));
-    CHECK(parsed == expected);
-
-    json iterator_parsed;
-    nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::string_input_adapter_type> iterator_sax(iterator_parsed);
-    CHECK(json::sax_parse(cbor.begin(), cbor.end(), &iterator_sax, json::input_format_t::cbor,
-                          true, false, false, json::cbor_tag_handler_t::store));
-    CHECK(iterator_parsed == expected);
-
-#ifndef JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
-    json span_parsed;
-    nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::string_input_adapter_type> span_sax(span_parsed);
-    CHECK(json::sax_parse(nlohmann::detail::span_input_adapter(cbor.data(), cbor.size()), &span_sax,
-                          json::input_format_t::cbor, true, false, false, json::cbor_tag_handler_t::store));
-    CHECK(span_parsed == expected);
-#endif
-
-    const std::string text = "null";
-    CHECK(json::sax_parse(text, &acceptor, json::input_format_t::json,
-                          true, false, false, json::cbor_tag_handler_t::store));
-    CHECK(json::sax_parse(text.begin(), text.end(), &acceptor, json::input_format_t::json,
-                          true, false, false, json::cbor_tag_handler_t::store));
-#ifndef JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
-    CHECK(json::sax_parse(nlohmann::detail::span_input_adapter(text.data(), text.size()), &acceptor,
-                          json::input_format_t::json, true, false, false, json::cbor_tag_handler_t::store));
-#endif
-}
-
 TEST_CASE("issue #5402 - update(merge_objects=true) overwrites a primitive with an object")
 {
     json t = {{"k", 1}};

tests/src/unit-sax_parse.cpp

@@ -0,0 +1,768 @@
+//     __ _____ _____ _____
+//  __|  |   __|     |   | |  JSON for Modern C++ (supporting code)
+// |  |  |__   |  |  | | | |  version 3.12.0
+// |_____|_____|_____|_|___|  https://github.com/nlohmann/json
+//
+// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
+// SPDX-License-Identifier: MIT
+
+/////////////////////////////////////////////////////////////////////
+// Tests that call basic_json::sax_parse have a file of their own: every
+// sax_parse call instantiates the parser and binary reader that recover from
+// errors (see #3989), and in unit-regression2.cpp, unit-regression3.cpp, and
+// unit-msgpack.cpp this made the objects too large for the MinGW linker to
+// relocate (see #5511).
+/////////////////////////////////////////////////////////////////////
+
+#include "doctest_compatibility.h"
+
+// capture whether JSON_DELETE_DEPRECATED_FUNCTIONS was enabled on the command
+// line *before* including json.hpp, since the library #undefs it once the header
+// has been fully processed (see include/nlohmann/detail/macro_unscope.hpp); the
+// tests of deprecated functions are skipped if these functions are deleted
+#if defined(JSON_DELETE_DEPRECATED_FUNCTIONS) && (JSON_DELETE_DEPRECATED_FUNCTIONS == 1)
+    #define JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
+#endif
+
+#include <nlohmann/json.hpp>
+using json = nlohmann::json;
+
+#include <cmath>
+#include <cstddef>
+#include <cstdint>
+#include <initializer_list>
+#include <map>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "sax_countdown.hpp"
+using utils::SaxCountdown;
+
+// a narrow number_float_t, so that a double read from binary input can
+// overflow it
+using float_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t, float>;
+
+DOCTEST_CLANG_SUPPRESS_WARNING_PUSH
+DOCTEST_CLANG_SUPPRESS_WARNING("-Wexit-time-destructors")
+
+namespace
+{
+/// builds a value from SAX events, asks the parser to recover from its first
+/// 100 errors, and checks that the events are balanced (see #3989)
+template<typename BasicJsonType>
+class BasicRecoveringParser
+{
+  public:
+    explicit BasicRecoveringParser(BasicJsonType& j)
+        : dom(j, false)
+    {}
+
+    bool null()
+    {
+        value();
+        return dom.null();
+    }
+
+    bool boolean(bool val)
+    {
+        value();
+        return dom.boolean(val);
+    }
+
+    bool number_integer(typename BasicJsonType::number_integer_t val)
+    {
+        value();
+        return dom.number_integer(val);
+    }
+
+    bool number_unsigned(typename BasicJsonType::number_unsigned_t val)
+    {
+        value();
+        return dom.number_unsigned(val);
+    }
+
+    bool number_float(typename BasicJsonType::number_float_t val, const std::string& s)
+    {
+        value();
+        return dom.number_float(val, s);
+    }
+
+    bool string(std::string& val)
+    {
+        value();
+        return dom.string(val);
+    }
+
+    bool binary(typename BasicJsonType::binary_t& val)
+    {
+        value();
+        return dom.binary(val);
+    }
+
+    bool start_object(std::size_t elements)
+    {
+        value();
+        stack.push_back('o');
+        return dom.start_object(elements);
+    }
+
+    bool key(std::string& val)
+    {
+        if (stack.empty() || stack.back() != 'o')
+        {
+            well_formed = false;
+            return false;
+        }
+        stack.back() = 'v';
+        return dom.key(val);
+    }
+
+    bool end_object()
+    {
+        if (stack.empty() || stack.back() != 'o')
+        {
+            well_formed = false;
+            return false;
+        }
+        stack.pop_back();
+        return dom.end_object();
+    }
+
+    bool start_array(std::size_t elements)
+    {
+        value();
+        stack.push_back('a');
+        return dom.start_array(elements);
+    }
+
+    bool end_array()
+    {
+        if (stack.empty() || stack.back() != 'a')
+        {
+            well_formed = false;
+            return false;
+        }
+        stack.pop_back();
+        return dom.end_array();
+    }
+
+    bool parse_error(std::size_t /*unused*/, const std::string& /*unused*/, const json::exception& ex)
+    {
+        messages.emplace_back(ex.what());
+        // a limit, so that a reader that does not stop fails the test
+        // instead of making it hang
+        return ++errors < 100;
+    }
+
+    /// whether the events were balanced and every key was followed by a value
+    bool balanced() const
+    {
+        return well_formed && stack.empty();
+    }
+
+    /// builds the value
+    nlohmann::detail::json_sax_dom_parser<BasicJsonType> dom;
+    std::size_t errors = 0;
+    std::vector<std::string> messages {}; // NOLINT(readability-redundant-member-init)
+    std::vector<char> stack {}; // NOLINT(readability-redundant-member-init)
+    bool well_formed = true;
+
+  private:
+    void value()
+    {
+        if (!stack.empty())
+        {
+            if (stack.back() == 'v')
+            {
+                stack.back() = 'o';
+            }
+            else if (stack.back() == 'o')
+            {
+                well_formed = false;
+            }
+        }
+    }
+
+};
+
+using RecoveringParser = BasicRecoveringParser<json>;
+
+struct BinaryParseResult
+{
+    json value;
+    std::size_t errors;
+    std::vector<std::string> messages;
+    bool ok;
+    bool balanced;
+};
+
+BinaryParseResult parse_binary_recovering(const std::vector<std::uint8_t>& input, const json::input_format_t format)
+{
+    json j;
+    RecoveringParser sax(j);
+    const bool ok = json::sax_parse(input, &sax, format);
+    return {j, sax.errors, sax.messages, ok, sax.balanced()};
+}
+
+#if !defined(JSON_NOEXCEPTION)
+/// the message of the exception that reading @a input into a JSON value
+/// throws, or an empty string if reading succeeds
+std::string binary_error_message(const std::vector<std::uint8_t>& input, const json::input_format_t format)
+{
+    try
+    {
+        json _;
+        switch (format)
+        {
+            case json::input_format_t::cbor:
+                _ = json::from_cbor(input);
+                break;
+            case json::input_format_t::msgpack:
+                _ = json::from_msgpack(input);
+                break;
+            case json::input_format_t::ubjson:
+                _ = json::from_ubjson(input);
+                break;
+            case json::input_format_t::bjdata:
+                _ = json::from_bjdata(input);
+                break;
+            case json::input_format_t::bson:
+                _ = json::from_bson(input);
+                break;
+            case json::input_format_t::bon8:
+                _ = json::from_bon8(input);
+                break;
+            case json::input_format_t::json:
+            default:
+                break;
+        }
+    }
+    catch (const json::exception& e)
+    {
+        return e.what();
+    }
+    return "";
+}
+#endif
+
+/// a BSON element: its type, its name, and its value
+std::vector<std::uint8_t> bson_element(const std::uint8_t type, const std::string& name, const std::vector<std::uint8_t>& value)
+{
+    std::vector<std::uint8_t> result = {type};
+    result.insert(result.end(), name.begin(), name.end());
+    result.push_back(0x00);
+    result.insert(result.end(), value.begin(), value.end());
+    return result;
+}
+
+/// a BSON document of the given elements; @a size_offset is added to the
+/// size it declares
+std::vector<std::uint8_t> bson_document(const std::vector<std::vector<std::uint8_t>>& elements, const int size_offset = 0)
+{
+    std::vector<std::uint8_t> body;
+    for (const auto& element : elements)
+    {
+        body.insert(body.end(), element.begin(), element.end());
+    }
+    const auto size = static_cast<std::uint32_t>(static_cast<int>(body.size()) + 5 + size_offset);
+    std::vector<std::uint8_t> result = {static_cast<std::uint8_t>(size & 0xFFu), static_cast<std::uint8_t>((size >> 8u) & 0xFFu),
+                                        static_cast<std::uint8_t>((size >> 16u) & 0xFFu), static_cast<std::uint8_t>((size >> 24u) & 0xFFu)
+                                       };
+    result.insert(result.end(), body.begin(), body.end());
+    result.push_back(0x00);
+    return result;
+}
+
+/// a BSON int32 value
+std::vector<std::uint8_t> bson_int32(const std::int32_t value)
+{
+    const auto u = static_cast<std::uint32_t>(value);
+    return {static_cast<std::uint8_t>(u & 0xFFu), static_cast<std::uint8_t>((u >> 8u) & 0xFFu),
+            static_cast<std::uint8_t>((u >> 16u) & 0xFFu), static_cast<std::uint8_t>((u >> 24u) & 0xFFu)};
+}
+
+/// a BSON string value, whose length is @a length_offset off
+std::vector<std::uint8_t> bson_string(const std::string& value, const std::int32_t length_offset = 0)
+{
+    auto result = bson_int32(static_cast<std::int32_t>(value.size() + 1) + length_offset);
+    result.insert(result.end(), value.begin(), value.end());
+    result.push_back(0x00);
+    return result;
+}
+
+/// @a count bytes of value 0xAB
+std::vector<std::uint8_t> bytes(const std::size_t count)
+{
+    return std::vector<std::uint8_t>(count, 0xAB);
+}
+
+/// the bytes of @a parts, one after the other
+std::vector<std::uint8_t> concatenated(std::initializer_list<std::vector<std::uint8_t>> parts)
+{
+    std::vector<std::uint8_t> result;
+    for (const auto& part : parts)
+    {
+        result.insert(result.end(), part.begin(), part.end());
+    }
+    return result;
+}
+
+/// U+FFFD REPLACEMENT CHARACTER
+std::string replacement_character()
+{
+    return "\xEF\xBF\xBD";
+}
+}  // namespace
+
+TEST_CASE("regression test - #3989 SAX parse_error() returning true")
+{
+    SECTION("binary formats complete what was read before the input ends")
+    {
+        const json j = {{"a", {1, -2, {{"b", "c"}}, json::array()}}, {"d", {{"e", nullptr}, {"f", true}}}, {"g", 1.5}, {"h", json::binary({1, 2, 3})}};
+
+        const std::vector<std::pair<json::input_format_t, std::vector<std::uint8_t>>> encodings =
+        {
+            {json::input_format_t::cbor, json::to_cbor(j)},
+            {json::input_format_t::msgpack, json::to_msgpack(j)},
+            {json::input_format_t::ubjson, json::to_ubjson(j)},
+            {json::input_format_t::ubjson, json::to_ubjson(j, true, true)},
+            {json::input_format_t::bjdata, json::to_bjdata(j)},
+            {json::input_format_t::bjdata, json::to_bjdata(j, true, true)},
+            {json::input_format_t::bson, json::to_bson(j)},
+            {json::input_format_t::bon8, json::to_bon8(j)},
+        };
+
+        for (const auto& encoding : encodings)
+        {
+            const auto format = encoding.first;
+            const auto& bytes = encoding.second;
+            CAPTURE(format)
+
+            // every prefix is truncated input
+            for (std::size_t length = 0; length < bytes.size(); ++length)
+            {
+                CAPTURE(length)
+                const auto result = parse_binary_recovering(std::vector<std::uint8_t>(bytes.begin(), bytes.begin() + static_cast<std::ptrdiff_t>(length)), format);
+                CHECK(!result.ok);
+                CHECK(result.errors == 1);
+                CHECK(result.balanced);
+            }
+
+            // the complete input is read as usual (binary values do not
+            // round-trip through every format, so compare with a plain parse)
+            json expected;
+            nlohmann::detail::json_sax_dom_parser<json> dom(expected);
+            CHECK(json::sax_parse(bytes, &dom, format));
+            const auto complete = parse_binary_recovering(bytes, format);
+            CHECK(complete.ok);
+            CHECK(complete.errors == 0);
+            CHECK(complete.value == expected);
+
+            // a byte after the value
+            auto trailing_bytes = bytes;
+            trailing_bytes.push_back(0x01);
+            const auto trailing = parse_binary_recovering(trailing_bytes, format);
+            CHECK(!trailing.ok);
+            CHECK(trailing.errors == 1);
+            CHECK(trailing.value == expected);
+        }
+    }
+
+    SECTION("containers without an end")
+    {
+        // these made the readers loop, or read on, after the error
+        const auto cbor_array = parse_binary_recovering({0x9F}, json::input_format_t::cbor);
+        CHECK(cbor_array.errors == 1);
+        CHECK(cbor_array.value == json::array());
+
+        const auto cbor_map = parse_binary_recovering({0xBF, 0x61, 'a'}, json::input_format_t::cbor);
+        CHECK(cbor_map.errors == 1);
+        CHECK(cbor_map.value == json({{"a", nullptr}}));
+
+        const auto msgpack_array = parse_binary_recovering({0xDD, 0xFF, 0xFF, 0xFF, 0xFF}, json::input_format_t::msgpack);
+        CHECK(msgpack_array.errors == 1);
+        CHECK(msgpack_array.value == json::array());
+
+        const auto msgpack_map = parse_binary_recovering({0x81, 0xA1, 'a', 0x92, 0x01}, json::input_format_t::msgpack);
+        CHECK(msgpack_map.errors == 1);
+        CHECK(msgpack_map.value == json({{"a", {1}}}));
+    }
+
+    SECTION("BJData ndarray")
+    {
+        // a 2x3 int8 array with two of its six elements; the annotated array
+        // format opens an object and two arrays of its own
+        const auto result = parse_binary_recovering({'[', '$', 'i', '#', '[', '$', 'i', '#', 'i', 2, 2, 3, 1, 2}, json::input_format_t::bjdata);
+        CHECK(result.errors == 1);
+        CHECK(result.balanced);
+        CHECK(result.value == json({{"_ArrayType_", "int8"}, {"_ArraySize_", {2, 3}}, {"_ArrayData_", {1, 2}}}));
+    }
+
+    SECTION("binary formats repair items whose end is known")
+    {
+        struct Repair
+        {
+            json::input_format_t format;
+            std::vector<std::uint8_t> input;
+            json expected;
+            std::size_t errors;
+        };
+
+        const std::vector<Repair> repairs =
+        {
+            // CBOR: tags are ignored (here tag 1 and the self-describe tag 55799)
+            {json::input_format_t::cbor, {0x82, 0xC1, 0x05, 0xD9, 0xD9, 0xF7, 0x06}, {5, 6}, 2},
+            // CBOR: undefined and other simple values become null
+            {json::input_format_t::cbor, {0x84, 0xF7, 0xE0, 0xF8, 0x20, 0x01}, {nullptr, nullptr, nullptr, 1}, 3},
+            // CBOR: members whose key is not a string are skipped, whatever their key and value
+            {json::input_format_t::cbor, {0xA4, 0x01, 0x02, 0x82, 0x01, 0x02, 0xA1, 0x61, 'x', 0x9F, 0xFF, 0xC1, 0x01, 0x5F, 0x41, 0x00, 0xFF, 0x61, 'a', 0x03}, {{"a", 3}}, 3},
+            {json::input_format_t::cbor, {0xBF, 0xF5, 0xBF, 0x61, 'x', 0x7F, 0x61, 'y', 0xFF, 0xFF, 0x61, 'a', 0x03, 0xFF}, {{"a", 3}}, 1},
+            // MessagePack: members whose key is not a string are skipped
+            {json::input_format_t::msgpack, {0x84, 0x01, 0x02, 0x81, 0xA1, 'x', 0x01, 0x92, 0x01, 0x02, 0xD4, 0x01, 0x02, 0xC0, 0xA1, 'a', 0x04}, {{"a", 4}}, 3},
+            // UBJSON: a char that is not ASCII becomes U+FFFD
+            {json::input_format_t::ubjson, {'[', 'C', 0x80, 'C', 'A', ']'}, {replacement_character(), "A"}, 1},
+            // UBJSON: the longest beginning of a high-precision number is kept
+            {json::input_format_t::ubjson, {'[', 'H', 'i', 5, '1', '2', 'a', 'b', 'c', 'H', 'i', 2, '1', '.', 'H', 'i', 3, 'a', 'b', 'c', 'H', 'i', 3, '4', '.', '5', ']'}, {12, 1, nullptr, 4.5}, 3},
+            // BJData, too
+            {json::input_format_t::bjdata, {'[', 'C', 0xFF, 'H', 'i', 2, '-', '1', 'H', 'i', 2, '-', 'x', ']'}, {replacement_character(), -1, nullptr}, 2},
+            // a NUL ends a high-precision number, as it ends JSON text
+            {json::input_format_t::ubjson, {'[', 'H', 'i', 4, '1', '2', 0, '9', 'H', 'i', 2, 0, '1', 'i', 3, ']'}, {12, nullptr, 3}, 2},
+            // BON8: members whose key is not a string are skipped
+            {json::input_format_t::bon8, {0x89, 0x91, 0x92, 0xC9, 0x40, 0x82, 0x91, 0x92, 0x61, 0x93}, {{"a", 3}}, 2},
+            {json::input_format_t::bon8, {0x8B, 0x91, 0x85, 0x91, 0xFE, 0xFA, 0x8B, 'x', 0x91, 0xFE, 0x61, 0x93, 0xFE}, {{"a", 3}}, 2},
+            // BSON: elements of types the library does not read become null
+            {
+                json::input_format_t::bson, bson_document(
+                {
+                    bson_element(0x07, "_id", bytes(12)),                         // ObjectId
+                    bson_element(0x09, "date", bytes(8)),                         // UTC datetime
+                    bson_element(0x13, "decimal", bytes(16)),                     // 128-bit decimal
+                    bson_element(0x0B, "regex", {'a', '+', 0, 'i', 0}),           // regular expression
+                    bson_element(0x0D, "code", bson_string("f()")),               // JavaScript code
+                    bson_element(0x0E, "symbol", bson_string("s")),               // symbol
+                    bson_element(0x0C, "pointer", concatenated({bson_string("c"), bytes(12)})), // DBPointer
+                    bson_element(0x0F, "scope", concatenated({bson_int32(15), bson_string("g"), bson_document({})})), // code with scope
+                    bson_element(0x06, "undefined", {}),                          // undefined
+                    bson_element(0xFF, "min", {}),                                // min key
+                    bson_element(0x7F, "max", {}),                                // max key
+                    bson_element(0x10, "z", bson_int32(7)),
+                }),
+                {{"_id", nullptr}, {"date", nullptr}, {"decimal", nullptr}, {"regex", nullptr}, {"code", nullptr}, {"symbol", nullptr}, {"pointer", nullptr}, {"scope", nullptr}, {"undefined", nullptr}, {"min", nullptr}, {"max", nullptr}, {"z", 7}},
+                11
+            },
+            // BSON: an element of an unknown type becomes null, and the rest of its document is skipped
+            {
+                json::input_format_t::bson, bson_document(
+                {
+                    bson_element(0x03, "inner", bson_document({bson_element(0x10, "a", bson_int32(1)), bson_element(0x42, "x", bytes(3)), bson_element(0x10, "b", bson_int32(2))})),
+                    bson_element(0x04, "array", bson_document({bson_element(0x10, "0", bson_int32(1)), bson_element(0x42, "1", bytes(3))})),
+                    bson_element(0x10, "after", bson_int32(3)),
+                }),
+                {{"inner", {{"a", 1}, {"x", nullptr}}}, {"array", {1, nullptr}}, {"after", 3}},
+                2
+            },
+            // BSON: so does a string or byte array whose length cannot be right
+            {
+                json::input_format_t::bson, bson_document(
+                {
+                    bson_element(0x03, "inner", bson_document({bson_element(0x02, "s", bson_string("abc", -10)), bson_element(0x10, "b", bson_int32(2))})),
+                    bson_element(0x03, "bin", bson_document({bson_element(0x05, "b", concatenated({bson_int32(-1), bytes(1)})), bson_element(0x10, "b", bson_int32(2))})),
+                    bson_element(0x10, "after", bson_int32(3)),
+                }),
+                {{"inner", {{"s", nullptr}}}, {"bin", {{"b", nullptr}}}, {"after", 3}},
+                2
+            },
+            // BSON: a string without its terminator, and a document whose size does not match, are kept
+            {
+                json::input_format_t::bson, bson_document(
+                {
+                    bson_element(0x02, "s", {2, 0, 0, 0, 'a', 'X'}),
+                    bson_element(0x03, "inner", bson_document({bson_element(0x10, "a", bson_int32(1))}, 1)),
+                }),
+                {{"s", "a"}, {"inner", {{"a", 1}}}},
+                2
+            },
+        };
+
+        for (const auto& repair : repairs)
+        {
+            CAPTURE(repair.format)
+            CAPTURE(repair.input)
+            const auto result = parse_binary_recovering(repair.input, repair.format);
+            CHECK(!result.ok);
+            CHECK(result.balanced);
+            CHECK(result.errors == repair.errors);
+            CHECK(result.value == repair.expected);
+            REQUIRE(!result.messages.empty());
+#if !defined(JSON_NOEXCEPTION)
+            // the first error is the one reported without recovering; under
+            // JSON_NOEXCEPTION, reading without recovering aborts instead of
+            // throwing, so there is no message to compare with
+            CHECK(result.messages.front() == binary_error_message(repair.input, repair.format));
+#endif
+        }
+    }
+
+    SECTION("binary formats repair numbers that are out of range")
+    {
+        // CBOR: a double too large for a float number_float_t
+        float_json cbor;
+        BasicRecoveringParser<float_json> sax(cbor);
+        const std::vector<std::uint8_t> cbor_input = {0x82, 0xFB, 0x7E, 0x37, 0xE4, 0x3C, 0x88, 0x00, 0x75, 0x9C, 0x01}; // [1e300, 1]
+        CHECK(!float_json::sax_parse(cbor_input, &sax, float_json::input_format_t::cbor));
+        CHECK(sax.errors == 1);
+        CHECK(sax.messages.front() == "[json.exception.out_of_range.406] syntax error while parsing CBOR value: number overflow");
+        REQUIRE(cbor.size() == 2);
+        CHECK(std::isinf(cbor[0].get<float>()));
+        CHECK(cbor[1] == 1);
+
+        // UBJSON: a high-precision number too large for number_float_t
+        const auto ubjson = parse_binary_recovering({'H', 'i', 5, '1', 'e', '9', '9', '9'}, json::input_format_t::ubjson);
+        CHECK(ubjson.errors == 1);
+        CHECK(ubjson.value.is_number_float());
+        CHECK(std::isinf(ubjson.value.get<double>()));
+    }
+
+    SECTION("binary formats stop where the end of an item is not known")
+    {
+        // a byte that begins no item
+        const auto cbor = parse_binary_recovering({0x82, 0x01, 0x1C, 0x02}, json::input_format_t::cbor);
+        CHECK(cbor.errors == 1);
+        CHECK(cbor.value == json({1}));
+
+        // a key that is no item: the unused MessagePack byte, a CBOR break
+        // in a map of known size, and the end of a BON8 container
+        const auto msgpack = parse_binary_recovering({0x82, 0xA1, 'a', 0x01, 0xC1, 0x02}, json::input_format_t::msgpack);
+        CHECK(msgpack.errors == 1);
+        CHECK(msgpack.value == json({{"a", 1}}));
+        const auto cbor_break = parse_binary_recovering({0xA2, 0x61, 'a', 0x01, 0xFF, 0x02}, json::input_format_t::cbor);
+        CHECK(cbor_break.errors == 1);
+        CHECK(cbor_break.value == json({{"a", 1}}));
+        const auto bon8 = parse_binary_recovering({0x88, 0x61, 0x91, 0xFE}, json::input_format_t::bon8);
+        CHECK(bon8.errors == 1);
+        CHECK(bon8.value == json({{"a", 1}}));
+
+        // an indefinite-length string inside an indefinite-length string
+        const auto nested = parse_binary_recovering({0x82, 0x01, 0x7F, 0x7F, 0x61, 'a', 0xFF, 0xFF}, json::input_format_t::cbor);
+        CHECK(nested.errors == 1);
+        CHECK(nested.value == json({1}));
+
+        // a BJData ndarray whose element type has no name: the object that
+        // holds the ndarray was already begun
+        const auto ndarray = parse_binary_recovering({'[', '$', 0x01, '#', '[', '$', 'i', '#', 'i', 2, 2, 3}, json::input_format_t::bjdata);
+        CHECK(ndarray.errors == 1);
+        CHECK(ndarray.balanced);
+        CHECK(ndarray.value == json::object());
+
+        // a skipped member that the input ends in
+        const auto truncated = parse_binary_recovering({0xA2, 0x01, 0x82, 0x01}, json::input_format_t::cbor);
+        CHECK(truncated.errors == 2);
+        CHECK(truncated.balanced);
+        CHECK(truncated.value == json::object());
+
+        // a BSON element of an unknown type in a document whose size cannot be right
+        const auto bson = parse_binary_recovering(bson_document({bson_element(0x10, "a", bson_int32(1)), bson_element(0x42, "x", bytes(3))}, -10), json::input_format_t::bson);
+        CHECK(bson.errors == 1);
+        CHECK(bson.value == json({{"a", 1}, {"x", nullptr}}));
+    }
+
+    SECTION("changed bytes in binary input")
+    {
+        const json j = {{"a", {1, -2, {{"b", "c"}}, json::array()}}, {"d", {{"e", nullptr}, {"f", true}}}, {"g", 1.5}, {"h", json::binary({1, 2, 3})}, {"i", "\xC3\xA4"}};
+
+        const std::vector<std::pair<json::input_format_t, std::vector<std::uint8_t>>> encodings =
+        {
+            {json::input_format_t::cbor, json::to_cbor(j)},
+            {json::input_format_t::msgpack, json::to_msgpack(j)},
+            {json::input_format_t::ubjson, json::to_ubjson(j)},
+            {json::input_format_t::ubjson, json::to_ubjson(j, true, true)},
+            {json::input_format_t::bjdata, json::to_bjdata(j)},
+            {json::input_format_t::bjdata, json::to_bjdata(j, true, true)},
+            {json::input_format_t::bson, json::to_bson(j)},
+            {json::input_format_t::bon8, json::to_bon8(j)},
+        };
+        const std::vector<std::uint8_t> replacements = {0x00, 0x01, 0x7F, 0x80, 0xC1, 0xD9, 0xE0, 0xF7, 0xFE, 0xFF};
+
+        for (const auto& encoding : encodings)
+        {
+            const auto format = encoding.first;
+            const auto& original = encoding.second;
+            CAPTURE(format)
+
+            std::vector<std::vector<std::uint8_t>> inputs;
+            for (std::size_t position = 0; position < original.size(); ++position)
+            {
+                for (const auto replacement : replacements)
+                {
+                    auto changed = original;
+                    changed[position] = replacement;
+                    inputs.push_back(changed);
+                }
+                auto removed = original;
+                removed.erase(removed.begin() + static_cast<std::ptrdiff_t>(position));
+                inputs.push_back(removed);
+            }
+
+            for (const auto& input : inputs)
+            {
+                CAPTURE(input)
+                const auto result = parse_binary_recovering(input, format);
+                CHECK(result.balanced);
+                CHECK(result.errors <= input.size() + 1);
+#if !defined(JSON_NOEXCEPTION)
+                // an error is reported exactly if reading into a JSON value
+                // fails, and the first one is the same (under JSON_NOEXCEPTION,
+                // that reading aborts instead of throwing)
+                const auto message = binary_error_message(input, format);
+                CHECK(result.ok == message.empty());
+                if (!result.ok && result.errors < 100)
+                {
+                    CHECK(result.messages.front() == message);
+                }
+#endif
+            }
+        }
+    }
+
+    SECTION("JSON text")
+    {
+        // the parser stopped, but reported success
+        json j;
+        RecoveringParser sax(j);
+        CHECK(!json::sax_parse("[1,2,3,]", &sax));
+        CHECK(sax.errors == 1);
+        CHECK(j == json({1, 2, 3}));
+    }
+
+    SECTION("the SAX parsers of the library stop")
+    {
+        json _;
+        CHECK(json::from_cbor(std::vector<std::uint8_t> {0x9F}, true, false).is_discarded());
+        CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<std::uint8_t> {0x9F}), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing CBOR value: unexpected end of input", json::parse_error&);
+        CHECK(json::parse("[1,2,3,]", nullptr, false).is_discarded());
+        CHECK(!json::accept("[1,2,3,]"));
+    }
+}
+
+
+TEST_CASE("issue #5676 - SAX parsing of CBOR tags")
+{
+    const json expected = json::binary({1, 2, 3}, 42);
+    const auto cbor = json::to_cbor(expected);
+
+    nlohmann::detail::json_sax_acceptor<json> acceptor;
+    CHECK_FALSE(json::sax_parse(cbor, &acceptor, json::input_format_t::cbor));
+    CHECK_FALSE(json::sax_parse(cbor, &acceptor, json::input_format_t::cbor,
+                                true, false, false, json::cbor_tag_handler_t::error));
+
+    CHECK(json::sax_parse(cbor, &acceptor, json::input_format_t::cbor,
+                          true, false, false, json::cbor_tag_handler_t::ignore));
+
+    json parsed;
+    nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::string_input_adapter_type> sax(parsed);
+    CHECK(json::sax_parse(cbor, &sax, json::input_format_t::cbor,
+                          true, false, false, json::cbor_tag_handler_t::store));
+    CHECK(parsed == expected);
+
+    json iterator_parsed;
+    nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::string_input_adapter_type> iterator_sax(iterator_parsed);
+    CHECK(json::sax_parse(cbor.begin(), cbor.end(), &iterator_sax, json::input_format_t::cbor,
+                          true, false, false, json::cbor_tag_handler_t::store));
+    CHECK(iterator_parsed == expected);
+
+#ifndef JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
+    json span_parsed;
+    nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::string_input_adapter_type> span_sax(span_parsed);
+    CHECK(json::sax_parse(nlohmann::detail::span_input_adapter(cbor.data(), cbor.size()), &span_sax,
+                          json::input_format_t::cbor, true, false, false, json::cbor_tag_handler_t::store));
+    CHECK(span_parsed == expected);
+#endif
+
+    const std::string text = "null";
+    CHECK(json::sax_parse(text, &acceptor, json::input_format_t::json,
+                          true, false, false, json::cbor_tag_handler_t::store));
+    CHECK(json::sax_parse(text.begin(), text.end(), &acceptor, json::input_format_t::json,
+                          true, false, false, json::cbor_tag_handler_t::store));
+#ifndef JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
+    CHECK(json::sax_parse(nlohmann::detail::span_input_adapter(text.data(), text.size()), &acceptor,
+                          json::input_format_t::json, true, false, false, json::cbor_tag_handler_t::store));
+#endif
+}
+
+
+TEST_CASE("MessagePack SAX aborts")
+{
+    SECTION("start_array(len)")
+    {
+        std::vector<uint8_t> const v = {0x93, 0x01, 0x02, 0x03};
+        SaxCountdown scp(0);
+        CHECK(!json::sax_parse(v, &scp, json::input_format_t::msgpack));
+    }
+
+    SECTION("start_object(len)")
+    {
+        std::vector<uint8_t> const v = {0x81, 0xa3, 0x66, 0x6F, 0x6F, 0xc2};
+        SaxCountdown scp(0);
+        CHECK(!json::sax_parse(v, &scp, json::input_format_t::msgpack));
+    }
+
+    SECTION("key()")
+    {
+        std::vector<uint8_t> const v = {0x81, 0xa3, 0x66, 0x6F, 0x6F, 0xc2};
+        SaxCountdown scp(1);
+        CHECK(!json::sax_parse(v, &scp, json::input_format_t::msgpack));
+    }
+}
+
+TEST_CASE("issue #5405 - a user-defined SAX consumer is unaffected by the internal DOM reserve optimization")
+{
+    // the reserve() call is local to json_sax_dom_parser / json_sax_dom_callback_parser;
+    // a custom SAX consumer that does not touch a DOM array sees identical events
+    json j = json::array();
+    for (int i = 0; i < 100; ++i)
+    {
+        j.push_back(i);
+    }
+    const auto packed = json::to_msgpack(j);
+
+    SaxCountdown scp(1000000); // large enough to never trigger an abort
+    CHECK(json::sax_parse(packed, &scp, json::input_format_t::msgpack));
+}
+
+TEST_CASE("MessagePack nesting does not consume the call stack - SAX interface")
+{
+    // see the test case of the same name in unit-msgpack.cpp (#5104)
+    std::vector<uint8_t> input(300000, 0x91);
+    input.push_back(0x01); // innermost value
+
+    SaxCountdown accept_all(600001);
+    CHECK(json::sax_parse(input, &accept_all, json::input_format_t::msgpack));
+}
+
+TEST_CASE("MessagePack SAX parsing stops at every event")
+{
+    // Containers are opened and closed by the loop that reads them; a SAX
+    // handler that rejects any event - including the end of a nested
+    // container - must stop the parse right there.
+    const auto count_events = [](const std::vector<std::uint8_t>& input)
+    {
+        int events = 0;
+        while (true)
+        {
+            SaxCountdown scp(events);
+            if (json::sax_parse(input, &scp, json::input_format_t::msgpack))
+            {
+                return events;
+            }
+            ++events;
+            REQUIRE(events < 1000);
+        }
+    };
+
+    // 20 events: every container kind closes inside another one
+    const json j = json::parse(R"({"a": [1, {"b": []}], "c": {"d": [[2]]}})");
+    CHECK(count_events(json::to_msgpack(j)) == 20);
+}
+
+DOCTEST_CLANG_SUPPRESS_WARNING_POP