Make parsing from std::istream and FILE* fast

Squashed onto develop cd6b428ef (CI fixes from #5860). Original commits:

- Let the lexer's bulk fast paths run on a window of the input
- Scan the get area of an input stream in bulk
- Lock a FILE once per parse instead of once per character
- Add benchmarks for parsing from a FILE* and for many values from a stream
- Avoid implicit unsigned char to char conversion in stream tests
- Make file_input_adapter::unlock() safe to call on a moved-from adapter

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann committed 2026-10-17 08:29:45 +02:00
1 parent 0e95fe8200
commit 1d2526b1d8
22 files changed
+2107 -67

No files matched your search

@@ -132,6 +132,10 @@ outside of a string, invalid) byte; see the [FAQ entry](../../home/faq.md#nul-by
- [operator>>](../operator_gtgt.md) - deserialize from stream
- [`JSON_STRICT_NUL_HANDLING`](../macros/json_strict_nul_handling.md) - opt in to rejecting a NUL byte in the input
instead of treating it as end of input
- [`JSON_DISABLE_STREAM_BULK_SCAN`](../macros/json_disable_stream_bulk_scan.md) - read a `std::istream` one character
at a time
- [`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md) - read a `FILE*` with the locking stdio
functions
## Version history
@@ -143,6 +147,10 @@ outside of a string, invalid) byte; see the [FAQ entry](../../home/faq.md#nul-by
- Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
- `JSON_STRICT_NUL_HANDLING` added in version 3.13.0 to optionally reject a NUL byte in the input instead of treating
it as end of input; planned to become the default in version 4.0.0.
- Parsing from an `std::istream` scans the characters in its stream buffer in bulk in version 3.13.0
([`JSON_DISABLE_STREAM_BULK_SCAN`](../macros/json_disable_stream_bulk_scan.md)).
- Parsing from a `FILE*` locks the file once instead of once per character in version 3.13.0
([`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md)).
!!! warning "Deprecation"
+19
View File
@@ -112,6 +112,16 @@ By default, a `'\0'` (NUL) byte anywhere in the input is treated as end of input
outside of a string, invalid) byte; see the [FAQ entry](../../home/faq.md#nul-bytes-in-the-input) for details and the
[`JSON_STRICT_NUL_HANDLING`](../macros/json_strict_nul_handling.md) macro to opt into rejecting it instead.
!!! tip "Performance"
Contiguous inputs (strings, string views, vectors, arrays, pointer ranges) are the fastest to parse. A
`#!cpp std::istream` comes close: the parser scans the characters its stream buffer holds in bulk, without reading
ahead, so the stream is left exactly where it would be after reading one character at a time. A `#!cpp FILE*` is
read one character at a time, but the file is locked only once, so other threads that use the same `FILE` object
wait until parsing is done. See [input sources and performance](../../features/parsing/index.md#input-sources-and-performance),
[`JSON_DISABLE_STREAM_BULK_SCAN`](../macros/json_disable_stream_bulk_scan.md), and
[`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md).
## Examples
??? example "Example: (1) parse from a character array"
@@ -247,6 +257,10 @@ outside of a string, invalid) byte; see the [FAQ entry](../../home/faq.md#nul-by
- [operator>>](../operator_gtgt.md) - deserialize from stream
- [`JSON_STRICT_NUL_HANDLING`](../macros/json_strict_nul_handling.md) - opt in to rejecting a NUL byte in the input
instead of treating it as end of input
- [`JSON_DISABLE_STREAM_BULK_SCAN`](../macros/json_disable_stream_bulk_scan.md) - read a `std::istream` one character
at a time
- [`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md) - read a `FILE*` with the locking stdio
functions
## Version history
@@ -261,6 +275,11 @@ outside of a string, invalid) byte; see the [FAQ entry](../../home/faq.md#nul-by
it as end of input; planned to become the default in version 4.0.0.
- Extended empty-input detection to also cover an `std::istream` without a stream buffer, and fixed a crash
(`std::terminate`) when parsing from an `std::istream` with `eofbit` in its exception mask, in version 3.13.0.
- Parsing from an `std::istream` scans the characters in its stream buffer in bulk in version 3.13.0
([`JSON_DISABLE_STREAM_BULK_SCAN`](../macros/json_disable_stream_bulk_scan.md) restores reading one character at a
time).
- Parsing from a `FILE*` locks the file once instead of once per character in version 3.13.0
([`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md) restores `std::fgetc`).
!!! warning "Deprecation"
@@ -116,6 +116,10 @@ consumer `sax` has a super-linear complexity.
A UTF-8 byte order mark is silently ignored.
When reading from a `#!c FILE*`, the file is locked while it is parsed. The lock is recursive, so the SAX handler may
use the same file with stdio functions, but it must not wait for another thread that uses the same `FILE` object; see
[`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md).
## Examples
??? example
@@ -137,6 +141,10 @@ A UTF-8 byte order mark is silently ignored.
- [parse](parse.md) - deserialize from a compatible input
- [accept](accept.md) - check if the input is valid JSON
- [`JSON_DISABLE_STREAM_BULK_SCAN`](../macros/json_disable_stream_bulk_scan.md) - read a `std::istream` one character
at a time
- [`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md) - read a `FILE*` with the locking stdio
functions
## Version history
@@ -148,6 +156,10 @@ A UTF-8 byte order mark is silently ignored.
- Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
- `JSON_PRECISE_STREAM_POSITION` added in version 3.13.0 to optionally leave a `#!cpp std::istream` positioned right
after the parsed value when `strict` is `#!cpp false`.
- Parsing from an `std::istream` scans the characters in its stream buffer in bulk in version 3.13.0
([`JSON_DISABLE_STREAM_BULK_SCAN`](../macros/json_disable_stream_bulk_scan.md)).
- Parsing from a `FILE*` locks the file once instead of once per character in version 3.13.0
([`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md)).
!!! warning "Deprecation"