Parsing from a std::istream read one character at a time through the streambuf, so none of the parser's fast paths applied: it was 1.8 to 2.5 times slower than reading the file into a string and parsing that. input_stream_adapter now exposes the streambuf's get area (gptr() to egptr()) to the lexer as a window. The protected get-area members are reached through member pointers formed in a class derived from std::streambuf, which is ordinary, standard-conforming member access. The lexer scans the window with the same bulk paths as a string, and asks the streambuf for the next character as before when the window is used up. Nothing is read ahead, copied, or put back, so the stream is left exactly where it was left before, with and without JSON_PRECISE_STREAM_POSITION, and a streambuf without a get area is still read one character at a time. The adapter also keeps the current token for error messages as a pointer into the get area, and copies it out only before the get area is refilled, instead of the lexer copying every character. On an Apple M1 Max (Apple Clang, -O3), parsing from an std::ifstream is 1.6 to 2.1 times as fast (twitter.json 4.08 -> 2.30 ms, citm_catalog.json 8.75 -> 4.08 ms), and reading 100,000 small values with operator>> 1.4 (objects) to 4.0 (integers) times as fast. With GCC 13 and libstdc++ on x86-64 Linux, std::ifstream is 1.5 to 2.2 times as fast. Parsing from a stream is now within 11 to 18 % of reading the file into a string first. JSON_DISABLE_STREAM_BULK_SCAN restores reading one character at a time, for streambufs that do not follow the usual get-area contract. The new tests compare every observable result (values, exception messages, stream state, remaining characters) for get areas of 1 to 4096 bytes against a streambuf without a get area, and run again with the macro and with JSON_PRECISE_STREAM_POSITION. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
6.7 KiB
Parsing
This library can create a JSON value from a wide range of inputs. This page gives an overview of the available parsing functions and how they behave; the linked pages go into more detail.
flowchart LR
I["JSON input"] --> P["parse()"]
I --> S["sax_parse()"]
I --> A["accept()"]
P -->|"optional parser callback filters values"| D["basic_json value (DOM)"]
S --> H["events delivered to a user SAX handler"]
A --> V["bool: is the input valid JSON?"]
Input
The parse function reads a JSON value from an input. The input can be
- a string (
#!cpp std::string, C string, or string literal), - a
#!cpp std::istream(e.g., an#!cpp std::ifstreamreading from a file), - a
#!cpp FILE*pointer, - a pair of iterators over a contiguous range (e.g., a
#!cpp std::vector<std::uint8_t>), or - a contiguous container.
// parse from a string
json j = json::parse(R"({"happy": true, "pi": 3.141})");
// parse from a file
std::ifstream f("example.json");
json data = json::parse(f);
The input must be encoded in UTF-8; other encodings are not supported. A single input may contain only one JSON value. Inputs consisting of multiple values separated by newlines are handled by the JSON Lines format.
By default, the library rejects comments and trailing commas. Both can be enabled with parameters of the parse
function — see comments and trailing commas.
Strictness and trailing data
parse reads a single JSON value and requires the whole input to be consumed: any
non-whitespace data after the value is reported as a parse error. Use it when you want to guarantee that an input is
exactly one complete JSON document.
operator>> follows relaxed #!cpp std::istream semantics instead: it parses one JSON
value and leaves the stream positioned right after it, without requiring the rest of the stream to be consumed. This is
what makes it possible to read several concatenated values from the same stream, but it also means that "a valid
document followed by trailing bytes" is accepted rather than rejected. If you are validating conformance, or need to
reject any input that is not exactly one JSON document, prefer parse.
When using operator>> to read several concatenated values this way, a value that is a number must be followed by
whitespace, because operator>> consumes the character that terminates a number, unless
JSON_PRECISE_STREAM_POSITION is defined to 1 — see the
operator>> notes for details and examples.
Input sources and performance
All inputs produce the same values and the same exceptions, but they are read in different ways, which makes some of them faster than others:
| input | how it is read |
|---|---|
| strings, string views, string literals, contiguous containers, pointer or iterator ranges over contiguous memory | the whole input is scanned in bulk: runs of string characters, numbers, and indentation are recognized several bytes at a time |
#!cpp std::istream (#!cpp std::ifstream, #!cpp std::istringstream, ...) |
the characters the stream buffer holds are scanned in bulk like a string; the stream buffer is asked for more when they are used up |
#!cpp FILE* |
one character at a time |
other iterator ranges (e.g., #!cpp std::list<char>, #!cpp std::istreambuf_iterator<char>), wide strings |
one character at a time |
Parsing a file from a #!cpp std::ifstream is therefore only about 10 to 20 % slower than reading the whole file into a
#!cpp std::string first and parsing that, and it needs no memory for a copy of the file. The stream is never read
ahead: after parsing, it is positioned exactly as if every character had been read one at a time, so
operator>> can still read several values from one stream (see also
JSON Lines).
The macro JSON_DISABLE_STREAM_BULK_SCAN switches the bulk
scanning of streams off without changing any result, for stream buffers that do not follow the usual std::streambuf
contract.
SAX vs. DOM parsing
The library offers two parsing models:
- DOM parsing (the default): the complete input is read and stored as an in-memory
basic_jsonvalue that can be traversed and modified freely. This is whatparsedoes, and it is the right choice for most use cases. - SAX parsing: instead of building a value, the parser reports events (such as "a string was read" or "an object
started") to a handler that you implement. This avoids building the full value in memory and is useful for very large
inputs or when you only need to extract parts of the input. See the SAX interface for details and
sax_parsefor the API.
You can influence a DOM parse without switching to the SAX interface by passing a parser callback, which is called during parsing and can, for example, discard parts of the input.
Exceptions
When the input is not valid JSON, the parse function throws an exception by default. If exceptions are undesired or
unavailable, the parser can instead return a discarded value, or accept can be used
to only check whether an input is valid JSON. See parsing and exceptions for the available
options.
See also
parse- deserialize from a compatible inputaccept- check if the input is valid JSONsax_parse- generate SAX events- JSON Lines - parse newline-delimited JSON
- parser callbacks - influence the parsing by a callback function
- SAX interface - implement a custom SAX handler
- parsing and exceptions - control error handling
- parsing untrusted input - what to consider when parsing input from untrusted sources