deploy: cdfb130fa6

features/parsing/untrusted_input/index.md

@@ -22,7 +22,16 @@ No built-in depth limit while parsing
 
 Neither the parser nor the binary readers impose a limit on how deep the input may nest. An attacker can still exhaust memory (though not the call stack) with a sufficiently deep document. If you need to reject over-deep untrusted input outright, track the depth yourself, either with a [parser callback](https://json.nlohmann.me/features/parsing/parser_callbacks/#recipe-max-nesting-depth-via-a-callback) for the JSON parser, or by counting `start_object`/`start_array` and `end_object`/`end_array` calls in a [SAX handler](https://json.nlohmann.me/features/parsing/sax_interface/index.md) (for the JSON parser or a binary format alike) and throwing once your limit is exceeded.
 
-Once a value has been parsed, operations that walk it recursively — serializing it with [`dump`](https://json.nlohmann.me/api/basic_json/dump/index.md), hashing it, copying it, comparing two values with `==`, `<`, or (in C++20) `<=>`, merging with [`update`](https://json.nlohmann.me/api/basic_json/update/index.md), applying a [`merge_patch`](https://json.nlohmann.me/api/basic_json/merge_patch/index.md), creating a patch with [`diff`](https://json.nlohmann.me/api/basic_json/diff/index.md) or [`merge_diff`](https://json.nlohmann.me/api/basic_json/merge_diff/index.md), [`flatten`](https://json.nlohmann.me/api/basic_json/flatten/index.md), and the binary writers (`to_cbor`, `to_msgpack`, ...) — descend at most 128 levels on the call stack and continue below that with an explicit stack instead, so none of them can exhaust the stack either, however deeply the value is nested. Destroying a value (its destructor) never recurses at all, regardless of nesting depth, for the same reason.
+Once a value has been parsed, operations that walk it recursively — serializing it with [`dump`](https://json.nlohmann.me/api/basic_json/dump/index.md), hashing it, copying it, comparing two values with `==`, `<`, or (in C++20) `<=>`, merging with [`update`](https://json.nlohmann.me/api/basic_json/update/index.md), applying a [`merge_patch`](https://json.nlohmann.me/api/basic_json/merge_patch/index.md), creating a patch with [`diff`](https://json.nlohmann.me/api/basic_json/diff/index.md) or [`merge_diff`](https://json.nlohmann.me/api/basic_json/merge_diff/index.md), converting it to another `basic_json` type, destroying it, and the binary writers (`to_cbor`, `to_msgpack`, ...) — descend at most [`JSON_RECURSION_DEPTH_LIMIT`](https://json.nlohmann.me/api/macros/json_recursion_depth_limit/index.md) levels on the call stack, 128 by default, and continue below that with an explicit stack on the heap. So the call stack does not grow with the nesting depth of the value, however deeply it is nested. ([`flatten`](https://json.nlohmann.me/api/basic_json/flatten/index.md), [`unflatten`](https://json.nlohmann.me/api/basic_json/unflatten/index.md), and [`to_bson`](https://json.nlohmann.me/api/basic_json/to_bson/index.md) never use the call stack for the nesting.)
+
+The call stack still needs room for the levels that are recursed into, and the default limit assumes a stack of ordinary size. The stack that an operation needs is roughly proportional to the limit; with the default of 128 levels, the most demanding operation (`dump` in an unoptimized build) needed up to 288 KiB, and an optimized build needed 48 KiB or less, in measurements on macOS with arm64. The most demanding operation needed this much stack at the limits `0`, `32`, `128`, and `254`:
+
+| Stack needed, in KiB      | limit 0    | limit 32   | limit 128 (default) | limit 254 |
+| ------------------------- | ---------- | ---------- | ------------------- | --------- |
+| unoptimized build (`-O0`) | 16 or less | 80         | 288                 | 576       |
+| optimized build (`-O2`)   | 16 or less | 16 or less | 48                  | 112       |
+
+A thread with a smaller stack (a task of a real-time operating system, a fiber, or a thread of the musl C library, whose default stack is 128 KiB) can still overflow its stack on deeply nested input, which is why the limit can be lowered: define `JSON_RECURSION_DEPTH_LIMIT` to a smaller value, down to `0`, before including the library. Such a limit changes only the stack that is needed and the speed (values nested more deeply than the limit are walked more slowly), never a result. See the [documentation of the macro](https://json.nlohmann.me/api/macros/json_recursion_depth_limit/index.md) for the complete tables, the speed, and the rules for choosing a value. Destroying a value (its destructor) recurses over the same number of levels and then continues without the call stack, and needs 32 KiB or less at every limit.
 
 ## Input size
 
@@ -36,15 +45,16 @@ For the binary formats, an announced size is never trusted outright:
 
 ## Strings
 
-Invalid UTF-8 is rejected while parsing, not just while serializing:
+Invalid UTF-8 is rejected while parsing JSON text, not just while serializing:
 
 - In JSON text, an ill-formed UTF-8 byte in a string is a [`parse_error.101`](https://json.nlohmann.me/home/exceptions/#jsonexceptionparse_error101) ("invalid string: ill-formed UTF-8 byte").
 - In UTF-16 or UTF-32 input, an ill-formed sequence is a [`parse_error.101`](https://json.nlohmann.me/home/exceptions/#jsonexceptionparse_error101) as well: a lone surrogate, a high surrogate that is not followed by its low surrogate, and a code point above `U+10FFFF` are all rejected.
-- In a binary format, a string that is not valid UTF-8 is a [`parse_error.113`](https://json.nlohmann.me/home/exceptions/#jsonexceptionparse_error113).
+
+In a binary format, a string that is not valid UTF-8 is accepted unchanged by default (none of the formats except BON8 requires a decoder to reject it). Pass [`error_handler_t::strict`](https://json.nlohmann.me/api/basic_json/error_handler_t/index.md) to the `from_*` function to get a [`parse_error.113`](https://json.nlohmann.me/home/exceptions/#jsonexceptionparse_error113) instead. BON8 always validates ([`parse_error.112`](https://json.nlohmann.me/home/exceptions/#jsonexceptionparse_error112) for an ill-formed sequence).
 
 A `'\0'` (NUL) byte *inside* a quoted JSON string is always rejected (it must be escaped as `\u0000`). A NUL byte *outside* of a string is different: by default it is silently treated as the end of the input, so trailing bytes after it — including further, otherwise well-formed JSON — are silently ignored rather than rejected. Since untrusted input that happens to embed a NUL is a way to make part of it disappear without a parse error, see the [FAQ entry](https://json.nlohmann.me/home/faq/#nul-bytes-in-the-input) and consider defining [`JSON_STRICT_NUL_HANDLING`](https://json.nlohmann.me/api/macros/json_strict_nul_handling/index.md) to `1` to reject a NUL byte like any other unexpected byte instead.
 
-Parsing is not the only place invalid UTF-8 matters: a string that reached a `json` value some other way (for example, constructed by application code, or, before JSON_STRICT_NUL_HANDLING existed, read from a binary format that does not validate strings) still has to round-trip back to JSON text. By default, [`dump()`](https://json.nlohmann.me/api/basic_json/dump/index.md) throws [`type_error.316`](https://json.nlohmann.me/home/exceptions/#jsonexceptiontype_error316) if the string is not valid UTF-8; passing [`error_handler_t::replace`](https://json.nlohmann.me/api/basic_json/error_handler_t/index.md) or `error_handler_t::ignore` avoids the exception instead of crashing an application that forgot to catch it. See [Handling invalid UTF-8](https://json.nlohmann.me/features/serialization/#handling-invalid-utf-8) for the options and an example.
+Parsing is not the only place invalid UTF-8 matters: a string that reached a `json` value some other way (for example, constructed by application code, or read from a binary format with the default `error_handler_t::keep`) still has to round-trip back to JSON text. By default, [`dump()`](https://json.nlohmann.me/api/basic_json/dump/index.md) throws [`type_error.316`](https://json.nlohmann.me/home/exceptions/#jsonexceptiontype_error316) if the string is not valid UTF-8; passing [`error_handler_t::replace`](https://json.nlohmann.me/api/basic_json/error_handler_t/index.md) or `error_handler_t::ignore` avoids the exception instead of crashing an application that forgot to catch it. See [Handling invalid UTF-8](https://json.nlohmann.me/features/serialization/#handling-invalid-utf-8) for the options and an example.
 
 ## Duplicate object keys
 
@@ -68,6 +78,7 @@ A [JSON Pointer](https://json.nlohmann.me/features/json_pointer/index.md) can co
 - If the input's nesting depth matters to you, enforce your own limit with a [parser callback](https://json.nlohmann.me/features/parsing/parser_callbacks/#recipe-max-nesting-depth-via-a-callback) or a [SAX handler](https://json.nlohmann.me/features/parsing/sax_interface/index.md); the library bounds the call stack but not memory use.
 - Bound the size of the input itself before parsing, independent of the library's own bounded allocations for binary format lengths.
 - Decide up front how you want a string with invalid UTF-8 — from any source, not only parsing — to be serialized (`strict`, `replace`, or `ignore`), rather than discovering it from an uncaught `type_error.316`.
+- For untrusted input in a binary format, pass `error_handler_t::strict` to the `from_*` function if the strings must be valid UTF-8; by default they are not checked (BON8 always checks).
 - If a stray NUL byte silently truncating trailing input is a problem for your input format, define `JSON_STRICT_NUL_HANDLING`.
 - Decide whether duplicate object keys should be an error for your application, and add a callback if so.
 - Do not enable `ignore_comments` or `ignore_trailing_commas` for input that must be strictly conforming JSON.