Docs: fill documentation gaps found in user questions (#5907)
* Docs: fill documentation gaps found in user questions Add pages on class-hierarchy/smart-pointer/variant/chrono recipes, working without exceptions, and large, streamed and partial input. Extend the FAQ (23 entries, Known bugs first), parse error documentation (empty-input checklist, line/column, diagnostic positions), element access, conversions, creating values, ADL troubleshooting, memory, and build/distribution notes. Add 14 examples with generated output. Correct the statement that parse errors cannot be recovered from with -fno-exceptions. The topics come from recurring questions on Stack Overflow, Reddit, Hacker News, YouTube, and the project's GitHub discussions and issues. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: declare range-for lists in variables to keep astyle quiet astyle reformats a braced init list placed directly in a range-for header; declare the list first so check_amalgamation passes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
1 parent
cc917d7123
commit
ffd4242dc2
45 files changed
+1670
-8
No files matched your search
@@ -36,7 +36,9 @@ json data = json::parse(f);
|
||||
The input must be encoded in UTF-8; other encodings are not supported. The exception is wide-character input, which
|
||||
is read as UTF-16 or UTF-32 -- depending on the character type -- and converted to UTF-8; see the
|
||||
[`parse`](../../api/basic_json/parse.md) page for an example. A single input may contain only one JSON value.
|
||||
Inputs consisting of multiple values separated by newlines are handled by the [JSON Lines](json_lines.md) format.
|
||||
Inputs consisting of multiple values separated by newlines are handled by the [JSON Lines](json_lines.md) format; input
|
||||
that arrives in chunks or contains several values is covered in
|
||||
[Large, streamed, and partial input](large_and_partial_input.md).
|
||||
|
||||
By default, the library rejects comments and trailing commas. Both can be enabled with parameters of the `parse`
|
||||
function — see [comments](../comments.md) and [trailing commas](../trailing_commas.md).
|
||||
@@ -92,7 +94,7 @@ again without changing any result.
|
||||
|
||||
## SAX vs. DOM parsing
|
||||
|
||||
The library offers two parsing models:
|
||||
The library offers two parsing models, and a third, read-only alternative:
|
||||
|
||||
- **DOM parsing** (the default): the complete input is read and stored as an in-memory `basic_json` value that can be
|
||||
traversed and modified freely. This is what [`parse`](../../api/basic_json/parse.md) does, and it is the right choice
|
||||
@@ -102,6 +104,13 @@ The library offers two parsing models:
|
||||
inputs or when you only need to extract parts of the input. See the [SAX interface](sax_interface.md) for details and
|
||||
[`sax_parse`](../../api/basic_json/sax_parse.md) for the API.
|
||||
|
||||
- **Document and views** (`#!cpp <nlohmann/json_view.hpp>`, added in version 3.13.0): [`json_document::parse`](../../api/basic_json_document/parse.md)
|
||||
reads the same JSON grammar but builds a flat, read-only index of the values instead of a `basic_json` tree. Strings and
|
||||
floating-point numbers are read from the text on demand, and only the parts you
|
||||
[`materialize()`](../../api/basic_json_view/materialize.md) become `basic_json` values. This avoids most of the work when you
|
||||
need only a few values from a large or frequently parsed text; see [Zero-copy JSON views](../json_view.md). It is read-only, so use
|
||||
`parse` when you need to modify the value.
|
||||
|
||||
You can influence a DOM parse without switching to the SAX interface by passing a
|
||||
[parser callback](parser_callbacks.md), which is called during parsing and can, for example, discard parts of the input.
|
||||
|
||||
@@ -139,8 +148,10 @@ options.
|
||||
- [`accept`](../../api/basic_json/accept.md) - check if the input is valid JSON
|
||||
- [`sax_parse`](../../api/basic_json/sax_parse.md) - generate SAX events
|
||||
- [JSON Lines](json_lines.md) - parse newline-delimited JSON
|
||||
- [Large, streamed, and partial input](large_and_partial_input.md) - chunked, concatenated, and very large input
|
||||
- [parser callbacks](parser_callbacks.md) - influence the parsing by a callback function
|
||||
- [SAX interface](sax_interface.md) - implement a custom SAX handler
|
||||
- [Zero-copy JSON views](../json_view.md) - parse into a read-only document and read it through views
|
||||
- [parsing and exceptions](parse_exceptions.md) - control error handling
|
||||
- [parsing untrusted input](untrusted_input.md) - what to consider when parsing input from untrusted sources
|
||||
- [Zero-copy JSON views](../json_view.md) - parse into a document and read it through a view
|
||||
@@ -0,0 +1,154 @@
|
||||
# Large, Streamed, and Partial Input
|
||||
|
||||
[`parse`](../../api/basic_json/parse.md) reads exactly one complete JSON value and builds it in memory. It cannot be
|
||||
paused and resumed, and it does not hand back a half-parsed value when the input ends early. This page shows what to do
|
||||
when the input does not fit that model: it arrives in pieces, it contains several values, or it is too large to hold
|
||||
as a `json` value.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["my input is..."] --> B{"several JSON values in one stream?"}
|
||||
B -->|"yes"| C["read them one at a time"]
|
||||
A --> D{"arriving in chunks (socket, pipe)?"}
|
||||
D -->|"yes"| E["frame the messages, or buffer until complete"]
|
||||
A --> F{"too large to keep in memory?"}
|
||||
F -->|"yes"| G["discard parts with a callback, or use SAX"]
|
||||
A --> H{"only a small part is needed?"}
|
||||
H -->|"yes"| I["stop early with SAX, or read through a view"]
|
||||
```
|
||||
|
||||
## Several values in one stream
|
||||
|
||||
For newline-delimited input, read line by line and parse each line, as described in [JSON Lines](json_lines.md). For
|
||||
values that are simply concatenated, [`operator>>`](../../api/operator_gtgt.md) reads one value and leaves the stream
|
||||
positioned after it. Check for the end of the stream before each read, because a read at the end throws:
|
||||
|
||||
??? example "Example: read concatenated values"
|
||||
|
||||
```cpp
|
||||
--8<-- "examples/partial_input_concatenated.cpp"
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```
|
||||
--8<-- "examples/partial_input_concatenated.output"
|
||||
```
|
||||
|
||||
The same works for binary formats. [`from_cbor`](../../api/basic_json/from_cbor.md) and its relatives accept
|
||||
`#!cpp strict = false` to stop after the first value and leave the rest of the stream untouched; `tellg()` then reports
|
||||
how many bytes the value took:
|
||||
|
||||
??? example "Example: read one CBOR value and find out how many bytes it used"
|
||||
|
||||
```cpp
|
||||
--8<-- "examples/partial_input_cbor.cpp"
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```
|
||||
--8<-- "examples/partial_input_cbor.output"
|
||||
```
|
||||
|
||||
## Input arriving in chunks
|
||||
|
||||
A socket, pipe, or HTTP body hands over data in pieces of arbitrary size, and a piece can end in the middle of a
|
||||
number, a string, or an object. The parser does not keep state between calls, so there are two ways to deal with this.
|
||||
|
||||
**Frame the messages.** The most robust approach is a protocol that says where a value ends: one value per line
|
||||
([JSON Lines](json_lines.md)), a length prefix, or a delimiter. Collect the bytes of one message and parse them when
|
||||
the frame is complete. Prefer this whenever you control the protocol.
|
||||
|
||||
**Buffer until the value is complete.** If the data has no framing, append every chunk to a buffer and try to parse
|
||||
it. A parse error at the very end of the buffer means that the input ended prematurely, while an error anywhere else is
|
||||
a real syntax error. The member `byte` of the [`parse_error`](../../home/exceptions.md#jsonexceptionparse_error101)
|
||||
exception is one past the buffer size in the first case:
|
||||
|
||||
??? example "Example: buffer chunks until the value is complete"
|
||||
|
||||
```cpp
|
||||
--8<-- "examples/partial_input_buffering.cpp"
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```
|
||||
--8<-- "examples/partial_input_buffering.output"
|
||||
```
|
||||
|
||||
The example feeds a document in five chunks. The first four are reported as incomplete, and a syntax error is reported as
|
||||
such, not as missing data.
|
||||
|
||||
!!! warning "Limits of buffering"
|
||||
|
||||
- The buffer must contain exactly one value. Data that belongs to the next value is a syntax error. Use framing if
|
||||
values follow each other without a separator.
|
||||
- A number at the end of the buffer is a complete value, even if more digits are still to come (`12` may be the
|
||||
start of `123`). Buffering cannot detect this; use framing for top-level numbers.
|
||||
- Every attempt parses the buffer from the beginning, so the total work grows quadratically with the size of the
|
||||
value. Limit the size of the buffer for input you do not trust (see [untrusted input](untrusted_input.md)).
|
||||
|
||||
## Input too large for memory
|
||||
|
||||
The input does not have to become one huge `json` value. The two tools below process it piece by piece.
|
||||
|
||||
### Handle and discard elements with a callback
|
||||
|
||||
A [parser callback](parser_callbacks.md) is called for every finished value. If it returns `#!cpp false`, the value is
|
||||
discarded and its memory is released at once. For a file that consists of one huge array (or one huge object) of
|
||||
records, handle each record at depth 1 and discard it:
|
||||
|
||||
??? example "Example: handle and discard each record of a large array"
|
||||
|
||||
```cpp
|
||||
--8<-- "examples/partial_input_discard_records.cpp"
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```
|
||||
--8<-- "examples/partial_input_discard_records.output"
|
||||
```
|
||||
|
||||
Only one record is in memory at any time. The same recipe, with more detail, is in the
|
||||
[parser callbacks](parser_callbacks.md#recipe-streaming-a-large-homogeneous-array) page. To skip entire members you do not
|
||||
need, discard them when their key is read; see the [example](parser_callbacks.md) for `parse_event_t::key`.
|
||||
|
||||
### Extract a few values and stop with SAX
|
||||
|
||||
With the [SAX interface](sax_interface.md), no value is built at all, and **returning `#!cpp false` from any handler
|
||||
function stops the parser**. [`sax_parse`](../../api/basic_json/sax_parse.md) then returns `#!cpp false`, and the rest
|
||||
of the input is not read. The example finds the first `id` in a document without parsing the remainder:
|
||||
|
||||
??? example "Example: stop parsing as soon as a value is found"
|
||||
|
||||
```cpp
|
||||
--8<-- "examples/partial_input_sax_stop.cpp"
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```
|
||||
--8<-- "examples/partial_input_sax_stop.output"
|
||||
```
|
||||
|
||||
Note that `sax_parse` returns `#!cpp false` both when a handler stopped the parse and when the input is invalid, so
|
||||
record the reason in the handler (here: `id`) if you need to tell the two apart.
|
||||
|
||||
### Read parts of a large document with a view
|
||||
|
||||
If the whole text is available and you only need to read some of it, a [`json_document`](../json_view.md) from
|
||||
`<nlohmann/json_view.hpp>` (added in version 3.13.0) avoids the `basic_json` tree altogether: it keeps the text, builds a flat index of the values,
|
||||
and gives read-only [views](../../api/basic_json_view/index.md) of the parts you look at. Only what you
|
||||
[`materialize()`](../../api/basic_json_view/materialize.md) becomes a `basic_json` value. See
|
||||
[Zero-copy JSON views](../json_view.md) for details and for the lifetime rules.
|
||||
|
||||
## See also
|
||||
|
||||
- [Parsing](index.md) - overview of the parsing functions
|
||||
- [JSON Lines](json_lines.md) - newline-delimited JSON
|
||||
- [Parser callbacks](parser_callbacks.md) - discard values while parsing
|
||||
- [SAX interface](sax_interface.md) - event-based parsing
|
||||
- [Zero-copy JSON views](../json_view.md) - read parts of a large document without building a tree
|
||||
- [Untrusted input](untrusted_input.md) - limits for input you do not control
|
||||
@@ -21,6 +21,10 @@ catch (json::parse_error& ex)
|
||||
In case exceptions are undesired or not supported by the environment, there are different ways to proceed:
|
||||
|
||||
|
||||
!!! tip
|
||||
|
||||
[Working without exceptions](../no_exceptions.md) also covers accessing values and converting to your own types.
|
||||
|
||||
## Switch off exceptions
|
||||
|
||||
The [`parse()`](../../api/basic_json/parse.md) function accepts a `#!cpp bool` parameter `allow_exceptions` which
|
||||
@@ -81,3 +85,61 @@ The return value indicates whether the parsing should continue, so the function
|
||||
```
|
||||
--8<-- "examples/sax_no_exception.output"
|
||||
```
|
||||
|
||||
## Where did the error occur?
|
||||
|
||||
The message of a [`parse_error`](../../home/exceptions.md#jsonexceptionparse_error101) for JSON text contains the line
|
||||
and column of the error:
|
||||
|
||||
```
|
||||
[json.exception.parse_error.101] parse error at line 3, column 8: syntax error while parsing value - unexpected ']'; expected '[', '{', or a literal
|
||||
```
|
||||
|
||||
The exception object itself only stores the **byte index** of the last read character in the member `byte` (the first
|
||||
byte of the input has index 1). The same index is passed to `parse_error` in the [SAX interface](sax_interface.md). If
|
||||
you need the line and column as numbers, for example to show them in an editor, compute them from the input:
|
||||
|
||||
??? example "Example: compute line and column of a parse error"
|
||||
|
||||
```cpp
|
||||
--8<-- "examples/parse_error_position.cpp"
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```
|
||||
--8<-- "examples/parse_error_position.output"
|
||||
```
|
||||
|
||||
Columns count bytes, so a line with multi-byte UTF-8 characters before the error may differ from the column your editor
|
||||
shows.
|
||||
|
||||
### Where is the value that caused a later error?
|
||||
|
||||
A `parse_error` is raised while the text is read. Many mistakes show up later: the text is valid JSON, but a value has
|
||||
the wrong type when you convert it (`type_error.302`), or a key is missing (`out_of_range.403`). The exception then says
|
||||
what is wrong, but not where in the input the value was. Define
|
||||
[`JSON_DIAGNOSTIC_POSITIONS`](../../api/macros/json_diagnostic_positions.md) before including the library to add the
|
||||
byte range of the offending value to such messages (since version 3.12.0), and to query the range of any value with
|
||||
[`start_pos()`](../../api/basic_json/start_pos.md) and [`end_pos()`](../../api/basic_json/end_pos.md):
|
||||
|
||||
??? example "Example: locate a value with diagnostic positions"
|
||||
|
||||
```cpp
|
||||
--8<-- "examples/parse_error_position_diagnostic.cpp"
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```
|
||||
--8<-- "examples/parse_error_position_diagnostic.output"
|
||||
```
|
||||
|
||||
Positions are only available for values created by [`parse`](../../api/basic_json/parse.md), they are not updated when a
|
||||
value is changed, and the macro makes every value 16 bytes larger on 64-bit platforms, so use it where the cost is
|
||||
acceptable, for example for configuration files. Note that `start_pos()` is a 0-based index into the input, whereas
|
||||
`byte` of a `parse_error` starts at 1.
|
||||
|
||||
For binary formats such as CBOR, there are no lines; `byte` is the position in the byte sequence. To find out how many
|
||||
bytes a successfully parsed value used, see
|
||||
[Several values in one stream](large_and_partial_input.md#several-values-in-one-stream).
|
||||
@@ -26,9 +26,11 @@ several alternatives; see [Parsing and exceptions](parse_exceptions.md) for the
|
||||
[example](parse_exceptions.md#user-defined-sax-interface) that overrides it to print instead of throw.
|
||||
|
||||
If exceptions are unavailable entirely (`-fno-exceptions`, or [`JSON_NOEXCEPTION`](../../api/macros/json_noexception.md)
|
||||
defined), every `#!cpp throw` in the library becomes a call to `std::abort()` -- there is no way to recover from a
|
||||
parse error of untrusted input in that configuration; see
|
||||
[Switch off exceptions](../../home/exceptions.md#switch-off-exceptions) for the details and for overriding this with
|
||||
defined), every `#!cpp throw` in the library becomes a call to `std::abort()`. The alternatives above do not throw, so
|
||||
`parse()` with `allow_exceptions` set to `#!cpp false` (and likewise `accept()` and the `from_*` functions with
|
||||
`allow_exceptions` set to `#!cpp false`) can still reject invalid input in that configuration; a plain `parse()` call
|
||||
aborts on invalid input. See [Working without exceptions](../no_exceptions.md) and
|
||||
[Switch off exceptions](../../home/exceptions.md#switch-off-exceptions) for the details and for overriding the abort with
|
||||
`JSON_THROW_USER`.
|
||||
|
||||
## Nesting depth
|
||||
|
||||
Reference in new issue
Block a user