wt/json
1
0 Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-10-24 11:54:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: wt/json:claude/implement-5891-deff16
wt/json:develop wt/json:perf/view-compare wt/json:claude/fix-issue-3989-db7e45 wt/json:claude/from-json-arith-array wt/json:claude/implement-5891-deff16 wt/json:docs/cross-reference-json-view-3-13-0 wt/json:pin-ci-dependencies wt/json:gh-pages wt/json:claude/test-compilation-speed-993125 wt/json:claude/todo-191-plan-508110 wt/json:fix/json_pointer_create_object_5357 wt/json:master
wt/json:v3.12.0 wt/json:v3.11.3 wt/json:v3.11.2 wt/json:v3.11.1 wt/json:v3.11.0 wt/json:v3.10.5 wt/json:v3.10.4 wt/json:v3.10.3 wt/json:v3.10.2 wt/json:v3.10.1 wt/json:v3.10.0 wt/json:v3.9.1 wt/json:v3.9.0 wt/json:v3.8.0 wt/json:v3.7.3 wt/json:v3.7.2 wt/json:v3.7.1 wt/json:v3.7.0 wt/json:v3.6.1 wt/json:3.6.1 wt/json:v3.6.0 wt/json:v3.5.0 wt/json:v3.4.0 wt/json:v3.3.0 wt/json:3.3.0 wt/json:v3.2.0 wt/json:3.2.0 wt/json:v3.1.2 wt/json:3.1.2 wt/json:v3.1.1 wt/json:v3.1.0 wt/json:3.1.0 wt/json:v3.0.1 wt/json:3.0.1 wt/json:v3.0.0 wt/json:3.0.0 wt/json:v2.1.1 wt/json:2.1.1 wt/json:v2.1.0 wt/json:2.1.0 wt/json:v2.0.10 wt/json:2.0.10 wt/json:v2.0.9 wt/json:2.0.9 wt/json:v2.0.8 wt/json:2.0.8 wt/json:v2.0.7 wt/json:v2.0.6 wt/json:v2.0.5 wt/json:v2.0.4 wt/json:v2.0.3 wt/json:v2.0.2 wt/json:v2.0.1 wt/json:v2.0.0 wt/json:v1.1.0 wt/json:v1.0.0 wt/json:v1.0.0-rc1
..
compare: wt/json:perf/view-compare
wt/json:perf/view-compare wt/json:claude/fix-issue-3989-db7e45 wt/json:develop wt/json:claude/from-json-arith-array wt/json:claude/implement-5891-deff16 wt/json:docs/cross-reference-json-view-3-13-0 wt/json:pin-ci-dependencies wt/json:gh-pages wt/json:claude/test-compilation-speed-993125 wt/json:claude/todo-191-plan-508110 wt/json:fix/json_pointer_create_object_5357 wt/json:master
wt/json:v3.12.0 wt/json:v3.11.3 wt/json:v3.11.2 wt/json:v3.11.1 wt/json:v3.11.0 wt/json:v3.10.5 wt/json:v3.10.4 wt/json:v3.10.3 wt/json:v3.10.2 wt/json:v3.10.1 wt/json:v3.10.0 wt/json:v3.9.1 wt/json:v3.9.0 wt/json:v3.8.0 wt/json:v3.7.3 wt/json:v3.7.2 wt/json:v3.7.1 wt/json:v3.7.0 wt/json:v3.6.1 wt/json:3.6.1 wt/json:v3.6.0 wt/json:v3.5.0 wt/json:v3.4.0 wt/json:v3.3.0 wt/json:3.3.0 wt/json:v3.2.0 wt/json:3.2.0 wt/json:v3.1.2 wt/json:3.1.2 wt/json:v3.1.1 wt/json:v3.1.0 wt/json:3.1.0 wt/json:v3.0.1 wt/json:3.0.1 wt/json:v3.0.0 wt/json:3.0.0 wt/json:v2.1.1 wt/json:2.1.1 wt/json:v2.1.0 wt/json:2.1.0 wt/json:v2.0.10 wt/json:2.0.10 wt/json:v2.0.9 wt/json:2.0.9 wt/json:v2.0.8 wt/json:2.0.8 wt/json:v2.0.7 wt/json:v2.0.6 wt/json:v2.0.5 wt/json:v2.0.4 wt/json:v2.0.3 wt/json:v2.0.2 wt/json:v2.0.1 wt/json:v2.0.0 wt/json:v1.1.0 wt/json:v1.0.0 wt/json:v1.0.0-rc1
12 Commits
claude/implement-5891-deff16 .. perf/view-compare
Author SHA1 Message Date
Niels Lohmann 6775994761 Merge develop into perf/view-compare 
Resolves the conflict in unit-json_view_image.cpp, where develop (#5889)
and this branch each added a section to "json_view images: errors": both
are kept. single_include is a fresh amalgamation.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-24 07:48:54 +02:00
Niels Lohmann 9c50bdd140 Make the DOM parser cheaper per container and per member (#5882) 
* Reserve the parser's nesting stack once

The stack of open arrays and objects was a std::vector<std::uint8_t> that
allocated at the first push_back and then reallocated at 1, 2, 4, 8 bytes,
so a document nested four levels deep made four allocations before it
parsed a value. A document that starts with a container now reserves 32
levels in one allocation; deeper documents grow as before. The loop that
reads the tokens is the code it was, so large documents are not affected
(a stack object with its own storage, and a reserve at every container,
both cost the token loop instructions or cycles).

Fixed cost of parsing a 124-byte object (macOS instructions): accept
-7.6 %, DOM -2.5 %; no change for a single number.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Build arrays of the DOM parser in one exact-size allocation

Reading an array of unknown length appended its elements to a std::vector
one at a time, so a small array reallocated (and moved its elements) at
sizes 1, 2, 4, ... The DOM SAX parser now collects the elements of arrays
whose length is unknown on one stack shared by all open arrays and moves
them into an array of exactly the right size when the closing bracket
arrives. An array that grows past 16 elements switches to appending to its
own vector, so large arrays pay no second copy. Binary formats, which
announce the length, and builds with JSON_DIAGNOSTICS or
JSON_DIAGNOSTIC_POSITIONS (whose values need their final place from the
start) keep storing into the array directly.

The stacks of open containers are reserved on first use instead of growing
from one element.

The binary readers get a DOM parser of their own, json_sax_dom_binary_parser:
it is the plain parser that stores every value directly in its container
(the code the readers used before). Their containers announce their size, so
the bookkeeping of the collecting parser (frame entries, the destination of a
finished container) was pure overhead for them: with the collecting parser
FromBinaryShape nested was 3-12 % and containers up to 5 % more instructions
(GCC) than before. With the plain parser Linux Ir of FromBinaryShape and
FromBinaryBuffer is within 0.8 % of develop (Clang: identical).

canada.json has 56 000 two-element arrays that took three allocations each
and now take two. DOM parse Ir on Linux against develop (GCC / Clang): canada
-7.0 % / -7.7 %, citm -0.8 % / -1.6 %, 124-byte object -5.4 % / -5.5 %;
accept and SAX are untouched.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Reuse one value for the key events of the callback parser

parse() with a callback built a new string value for every key it passed to
the callback (a heap allocation and release per key), and copied the key a
second time into the object. The callback parser now keeps one string value
for key events, which it assigns, and moves the key into the object.

Callback parse, Linux Ir (GCC / Clang): twitter -13.3 % / -11.7 %, citm
-9.5 % / -7.6 %. (Earlier macOS measurement: twitter -19.5 % instructions.)

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Remember where the DOM parser stores the next value

The DOM SAX parser looked up the innermost open container for every value:
the end of its frame stack, then the object or array pointer in the frame.
Compilers have to reload all of that after each value is stored, since a
store of a value may alias the frame. The parser now keeps the innermost
container's kind and, for an array that stores directly, the array itself
in members, updated when a container opens or closes.

Flat arrays of numbers are the case that mattered: parsing floats.json
(1 000 000 doubles) into a DOM took 2 % more cycles than before the
previous commit on macOS and now takes 0.4 % fewer than the code before it
(signed_ints.json -0.8 %). Linux Ir is unchanged (canada -6.1 % GCC /
-6.9 % Clang, citm -0.3 % / -0.8 %, twitter +0.2 % / 0.0 %).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Look up the keys of big ordered_json objects in a hash table

ordered_map keeps its elements in a vector and searches it linearly for
every key it adds, so parsing an object with n keys into ordered_json took
O(n^2) key comparisons: 33 billion instructions for one object with 30 000
keys, where json needs 245 million. From 64 keys on, the DOM parser now
keeps a hash table of the keys of the open object (hashes and positions
only; the elements stay in the vector) and finds duplicate keys there. The
result is the same, including the position of a key that occurs twice.
Smaller objects, std::map based objects and key types without std::hash
are untouched.

ordered_json parse, macOS instructions: object with 1 000 keys 64.8 M ->
14.6 M, 10 000 keys 5 104 M -> 38.7 M, 30 000 keys 33 358 M -> 89 M.
Linux Ir of an object with 2 000 keys: 43.0 M -> 1.9 M (GCC), 37.9 M ->
1.8 M (Clang).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Trim the work the DOM parser does per container and per member

Opening and closing an array or object cost the DOM parser more than before
the exact-size array change, which showed on documents of many small
objects (jeopardy.json: +1.4 % Ir with GCC). This takes it back:

- the stack of open containers keeps its first eight entries in the parser
  object instead of in a vector, so it needs no capacity check beyond the
  push itself and no allocation for typical documents;
- a container is created in the place it ends up in, instead of as a
  temporary that is then moved there;
- a new object member, which is null, is constructed over the null instead
  of being swapped with it (only with the default base class of basic_json);
- the key index that big ordered_json objects use is only a part of the
  stack entries of objects that can have one;
- an array that outgrows the stack reserves twice what it has so far, as a
  vector that grew by itself would have.

DOM parse Ir of the series up to here against develop, Linux GCC / Clang:
canada -6.6 % / -7.3 %, citm +1.2 % / +0.2 %, jeopardy +0.4 % / -0.1 %,
124-byte object -6.5 % / -6.2 %. (twitter measures +6.8 % / +6.6 % at this
point in steady state, almost all of it in glibc's malloc/consolidate for the
new allocation sizes; with the last commit of the series it is -0.1 % / 0.0 %.)

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Test parsing containers of the sizes where the DOM parser changes storage

The DOM parser collects the elements of an array on a stack (up to 16) and
the members of an ordered_json object (up to 32) before it moves them into
the container, and uses a hash table for keys from 64 members on. Parse
arrays and objects of the sizes around these limits, nested into each other,
with repeated keys, and as random trees that are built next to their text,
and compare json and ordered_json against the value built through the API.

The tests pass on the code from before these changes as well. The random
trees are what finds the mistakes that are easy to make here: while the
collecting change was being written, a collected array that outgrew the
stack at the moment its parent had just reached the limit left its elements
in the parent, and only these tests (142 failed assertions) showed it.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Do not convert floats that cannot overflow in accept()

accept() only has to find out whether a float token is finite: json_sax_acceptor
ignores the value, and the parser rejects only a float that overflows. A
float whose digits fit into 19 digits and whose decimal exponent is at most
max_exponent10 - 19 is below 10^max_exponent10 and cannot overflow, so the
lexer, told by accept() that the value is not needed, skips the conversion
for those. A token that may overflow, with more digits, or in another float
format is converted as before, so 1e309 and its neighbours are rejected as
they were (the new test compares accept() with parse() on tokens around the
limit).

json::accept(): canada.json Ir -8.6 % (GCC) / -10.9 % (Clang), 124-byte
object -3.2 %; cycles on the M1 (release benchmarks, LTO): canada -7.8 %,
floats.json -9.0 %. The check tests the accept() flag first, so the parser
that needs the value pays one load and branch: SaxParse canada +0.35 % Ir
(GCC and Clang), DOM canada +0.4 %.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Collect the members of ordered_json objects on a stack until they are known

An ordered_json object keeps its members in a vector and appends every new
member to it, so reading an object of unknown size reallocated the vector at
1, 2, 4, 8, ... members, and ordered_map copies all keys on every
reallocation. The DOM SAX parser now collects the members of such objects on
one stack shared by all open objects, as it does the elements of arrays, and
moves them into a vector of exactly the right size when the closing brace
arrives. An object that gets past 32 members switches to appending to its
own vector (and, from 64, to the key index of the earlier commit). The
members are the same, in the same order, and a key that occurs twice keeps
the place of its first occurrence. std::map based objects, objects whose
size the input announces (binary formats), and builds with JSON_DIAGNOSTICS
or JSON_DIAGNOSTIC_POSITIONS are untouched.

What only ordered objects need (the collected members, the key indexes) is
a member of the parser only for them, and the code that stores a value tries
the array that takes its elements directly first. An array that outgrows
the stack now reserves 32 elements, the capacity a vector that grew by
itself has at that point; with the larger reserve before, parsing
floats.json (1 000 000 numbers) was 2 % slower on Apple M1 than before this
series.

ordered_json parse, Linux Ir against develop (GCC / Clang): citm -11.6 % /
-11.7 %, twitter -8.4 % / -8.4 %, jeopardy (12 000 objects of 7 keys)
-10.5 % / -8.0 %, canada -3.9 % / -5.2 %, 124-byte object -20.9 % / -16.5 %.
macOS time (M1, earlier base): citm -19 %, twitter -13 %, jeopardy -13 %,
canada -14 %, 124-byte object -26 %. json parse: steady-state twitter -0.1 %
(see the previous commits), citm -1.2 %.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Rename the key index's match callback so Flawfinder doesn't read it as std::equal

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Bind the parser test's strings by reference in the range-for loop

GCC and Clang 14 (-Wrange-loop-construct) reported the copy.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Collect container values only in basic_json's own parse functions

A json_sax_dom_parser that a user constructs (as in the sax_no_exception
example) must hold the values read before a parse error, as on develop:
its value is used after the error. The values of containers of unknown
size are now collected on a stack only with the new template parameter
Collect (default false), which json::parse() and the json_view
materialization set; without it, every value is stored in place as it
arrives. A test checks the values kept after an error for json and
ordered_json.

Also fixes the clang-tidy, cpplint and Infer findings: the frame stack
keeps its first frames in a std::array (left uninitialized) and the
others in a std::unique_ptr, auto for create<>(), NOLINT for member
initializers -Weffc++ needs, and an @infer-ignore for the nullptr that
place_container() does not use.

json::parse() and ordered_json::parse() instructions are unchanged
(Apple clang; twitter, citm, canada, jeopardy: -0.03 % to +0.25 %).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Clarify the DOM parser's collection stacks and rename reenter_enclosing_container()

Address review comments: explain why a closed container erases only the top
of the shared stack, set the payload before the type, rename the helper that
makes the enclosing container current again, and declare the parser's state
stack right before reserving it.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Export json_sax_dom_parser from the C++20 module for MSVC

MSVC needs the detail classes that basic_json befriends to be visible in
the module (#3970, as json_sax_dom_callback_parser); with its new
template parameter, json_sax_dom_parser failed in ci_module_cpp20
(C2039, not a member of nlohmann::detail).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Keep the overflow-limit numbers of the parser test in a std::vector

clang-tidy (cppcoreguidelines-avoid-c-arrays) rejected the C array.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix the clang-tidy findings of the ordered_json parser tests

The container sizes are a std::array, the expected text uses raw string
literals, the generator's arithmetic is in parentheses, auto for the
result of make<>(), and single characters are appended as characters.
clang-tidy now reports nothing on the lines the branch adds to
unit-ordered_json.cpp and unit-class_parser.cpp (apart from checks CI's
clang-tidy does not have).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-24 07:41:21 +02:00
Niels Lohmann dd7686a269 json_view: faster reading, indexing and C-stream parsing (#5888) 
* Read C streams into a json_document in blocks

json_document::parse(FILE*) (and read(), accept(), parse_copy()) went through the
library's file input adapter and appended the text character by character
with fgetc. Read it with fread in blocks instead, like the std::istream
overload already does.

Parsing citm_catalog.json (1.7 MB) from a FILE*:
  Apple M1: 80.6 M -> 8.4 M instructions, 4.8 ms -> 0.55 ms (parse of the same text from a std::string: 8.7 M)
  Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 47.0 M -> 13.7 M / 45.8 M -> 14.2 M
The parsed documents and the error cases are identical (checked over the test corpus
and 30 mutants of each file). A null FILE* is handled as before (by the library's adapter).

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Index arrays without subtrees directly

operator[](idx), at(idx) and back() skipped the elements of an array one
after the other, whatever they were. In an array none of whose elements is
an array or object (numbers, strings, booleans, null) the elements are
adjacent nodes, which the array node's own sizes tell: its subtree has
exactly one node more than it has elements. Such an element is now the
node idx after the array node; any other array is walked as before.

Reading r[i] for every i of an array of 3000 integers:
  Apple M1: 28.8 M instructions, 3.1 ms -> 0.1 M instructions, 4 us
  Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 84.1 M -> 3.1 M / 75.0 M -> 3.0 M
Arrays that contain arrays are unchanged.

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Skip the bounds checks of number and string tokens in NUL-terminated input

For a NUL-terminated input (std::string, C string, string literal), the
parser reads the terminator instead of comparing p with the end, via
cursor::cur(). Number and string scanning still tested `p != e` before each
look at the next byte (after the integer digits for '.', for 'e', after
the exponent sign and digits, and after a string for its closing quote),
although the terminator can never be one of those characters. They now use
cur(), which is the plain load for NUL-terminated input and the bounds check
for ptr+len input, as before.

Parsing, Apple M1 instructions per parse: twitter 3.126 M -> 3.089 M (-1.2 %),
citm_catalog 8.70 M -> 8.58 M (-1.3 %), jeopardy 306.1 M -> 302.5 M (-1.2 %), signed
integers 281 M -> 266 M (-5.4 %), canada and floats -0.1 % / -0.9 %.
Linux x86-64 Ir, GCC 13 / Clang 18: twitter -0.5 % / -0.0 %, signed integers +0.2 % / -0.7 %.

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Make the index text of items() on arrays lazy

basic_json_view::items() on an array formatted every element's index with
std::to_string when the item was made, whether or not the loop asks for the
key. The item now keeps the position and makes the text on the first key()
call. Objects are not affected.

Iterating over the items of an array of 3000 numbers and reading only the values:
  Apple M1: 25.5 us -> 4.6 us (0.45 M -> ~0 net instructions)
  Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 14.9 M -> 4.1 M / 15.3 M -> 4.6 M
Reading key() costs what it did (24.5 us).

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Make the default of value() with a C string only if the member is missing

basic_json_view::value(key, "text") and value(pointer, "text") built a
std::string from the default first and passed it on, so a call allocated
for the default even when the member exists, and copied it again when it
does not. They now look the member up and make the result directly: from the
member if there is one, else from the C string. Behavior, including the
exceptions, is the same.

Two calls (member found, member missing) with a 40 character default, Apple M1:
  1.6 K -> 0.7 K instructions
  Linux x86-64 Ir (500 rounds), GCC 13 / Clang 18: 2.46 M -> 2.22 M / 2.40 M -> 2.15 M

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Materialize scalars directly and keep the stack of small subtrees local

basic_json_view::materialize() (and get<BasicJsonType>()) always set up the
SAX handler that parse() uses and a std::vector for the open containers,
sized from the document's deepest nesting. For a scalar, which is most of what
is materialized one value at a time, neither is needed: the value is made
directly (the same constructors the handler ends up calling). For arrays and
objects the stack of open containers is now a local array for the first 16
levels, and the vector is only used for deeper documents.

Materializing five scalars of a small document, Apple M1: 3.6 K -> 0.9 K instructions
  Linux x86-64 Ir (500 rounds), GCC 13 / Clang 18: 2.96 M -> 2.37 M / 2.90 M -> 2.29 M

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Copy the text between escapes while scanning it

A string that contains an escape is decoded by builder::slow_string(). After
each escape it scanned the next run of plain bytes (8 or 16 byte-by-byte
steps, then a vector loop), and then copied that run to the decode arena in a
second pass. The run is now copied 16 bytes at a time while it is scanned
(one vector load, a store of the same 16 bytes, and the mask of the bytes
that end the run), and the write position lives in a local variable, not in
the builder, since a store of a character may alias the builder's members and
made the compiler reload it after every escape. A run that ends at a quote
needs no scan, and non-ASCII text goes through the UTF-8 check as before;
the portable code is unchanged.

Parsing, Apple M1 instructions: jeopardy.json 304.0 M -> 293.8 M (-3.4 %), 3000 strings with
escapes 1.72 M -> 1.54 M (-10 %), twitter -0.6 %.
Linux x86-64 Ir, GCC 13 / Clang 18: escaped strings -7.6 % / -11.4 %, twitter 0.0 % / 0.0 %.
Documents and error messages are identical on 900 k random strings of escapes, UTF-8 and control
characters (also under ASan), and on mutants of the test corpus.

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Hash the last bytes of a key of a large object without a loop

key_hash() mixed a key eight bytes at a time and then packed the remaining
0-7 bytes one by one, in a loop. The last word of a key of eight or more
bytes is now read as the last eight bytes of the key (overlapping the word
before), and a key of 2-7 bytes as two overlapping words of four or two
bytes. A product only moves bits upwards, and the slot is taken from the low
bits: the high bits of the last word are folded down before the
multiplication, since keys that differ only in their last characters ("user_17",
"user_18") would otherwise share slots. The hash is not stored, so its values
may change.

Looking up every key of an object of 1000 random keys, Apple M1 instructions per 1000 lookups:
  keys of 5 bytes 175 K -> 93 K, hex strings of 8 bytes 177 K -> 168 K, keys of 12 bytes 196 K -> 186 K
Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 5-byte keys -4.7 % / -8.8 %, 12-byte keys -3.3 % / -7.4 %.
Building the tables while parsing such objects is cheaper as well (5-byte keys: 13.3 us -> 10.1 us).

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Compare short keys of a large object without a call of memcmp

find_indexed() compared the key of a member the table points to with
memcmp, a library call for any length. Up to 16 bytes, the bytes are now
compared as two overlapping words (the first and the last 8, 4, or 2 bytes),
as find_member() already does for the objects without a table; longer keys
still use memcmp.

Looking up every key of an object of 1000 random keys, Apple M1 instructions per 1000 lookups
(with the previous commit): keys of 5 bytes 93 K -> 72 K, hex strings of 8 bytes 168 K -> 148 K,
keys of 12 bytes 186 K -> 170 K, keys of 24 bytes 198 K -> 193 K (unchanged, memcmp).
Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 5-byte keys -7.2 % / -7.3 %, 12-byte keys -5.9 % / -6.6 %.

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Read an unedited editable document like a read-only one

Views of a json_editable_document followed edits at every step: the
lookup, the element access, back(), dump() and materialize() were built
for nodes that can be moved or linked, though no edit changes the index until
the first one (every edit creates the edit state before it touches a node).
Until then these now take the read-only code: one check of the document's
edit state per call, not per member or per node. After the first edit
nothing changes. Views that were made earlier keep working, as the check is
made when they are used.

Unedited editable document, Apple M1: dump() of twitter.json 1.40 M -> 1.22 M instructions (-12.6 %);
lookups unchanged. Linux x86-64 Ir, GCC 13 / Clang 18: dump -1.7 % / -1.0 %, lookups -0.1 % / 0.0 %.
Walking with iterators is unchanged, and so is dump() after an edit.

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Format the numbered test keys of json_view's index with a stream

The keys were made with snprintf() and a format from a list, which
Flawfinder reports as an error (CWE-134, a format string that is not a
literal). A stream now formats them; the 30000 keys of the test are the
same as before.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Put the key number of the json_view index test in parentheses

astyle formatted i * step after << as a pointer declaration (i* step),
which failed the amalgamation check.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix the compiler and lint findings of json_view's read side and tests

- find() and contains() dispatch on Editable for the member lookup; for
  documents that cannot be edited both branches were the same call
  (GCC -Wduplicated-branches)
- json_view.hpp includes <cstdio> for the FILE overload (cpplint)
- the arena check macro of the builder is on one line (cpplint read it
  as an if body without braces)
- tests: no nested conditional (also MSVC C4018), no useless cast,
  parentheses around mixed operators, raw string literals, += instead
  of temporary concatenations, and NOLINT for the fixed seeds and a
  find() != npos, as elsewhere in the tests

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Grow the decode arena through a function and store its position back on every return

slow_string() kept its write position in locals and grew the arena through
the macro NLOHMANN_VIEW_ENSURE. The macro is now the always-inline member
arena_ensure() (a lambda is not inlined at every call site by clang, which
then keeps the position in memory: +11 % instructions on escape-heavy
strings), and a guard stores the position back into m_arena_write on every
return instead of by hand on the successful one. Instruction counts are
unchanged (clang) or 1 % lower (GCC).

Also word the complexity of back() as that of at().

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-24 07:41:06 +02:00
Niels Lohmann e9c4fe3cdf json_view: indented and ensure_ascii dumps through the fast writer (#5889) 
* Write the indented output of json_view's dump() with the fast writer

The compact writer of dump() (cursor in a register, fixed-size copies of
tokens) served dump(-1) only; every dump(indent), and everything with
ensure_ascii that was indented, went through a generic walk that calls put()
for each token and put_repeated() (a memset call) for each line break. The
fast writer now writes indented output, too, with or without ensure_ascii, and
the generic walk is gone. A line break is a copy of a 64-byte ruler (a line
break and 63 indentation characters) when the line is at most 64 bytes, and a
store plus memset beyond that. The depth is counted only in the indented
instantiations (counting it in the compact one cost an editable document 5 %
with Clang). The indented walks are out of line (dump_pretty()): inlined
into dump() next to the compact ones, they made compact dumps of integer
arrays 7 % slower.

A decoded string (one with escape sequences) of a damaged image loaded with
image_check::bounds can contain invalid UTF-8. As before, the compact dump
writes such bytes unchanged, and the indented dump throws type_error.316; its
runs are found by the lexer's validating scan (a scalar validation of every
sequence made dump(2) of decoded CJK text twice as slow as before).

dump(2) instructions per call, macOS arm64 (before -> after):

               twitter         citm            canada          jeopardy(4k)
  json         3.75M -> 1.85M  8.28M -> 3.93M  67.1M -> 56.0M  9.56M -> 4.81M
  editable     3.82M -> 1.84M  8.48M -> 4.02M  68.1M -> 56.2M  9.66M -> 4.75M

Linux instructions (GCC 13 / Clang 18), dump(2) and dump(2, ' ', true):

  twitter      -25 % / -30 %      -11 % / -11 %
  citm         -29 % / -28 %      -24 % / -24 %
  jeopardy(4k) -25 % / -32 %      -20 % / -24 %
  CJK text with \u escapes  -6 % / -11 %   -2 % / -2 %

The compact dump is unchanged (within +0.4 % on Linux, within noise on the Mac).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Scan the strings of json_view's dump() with ensure_ascii 16 bytes at a time

With ensure_ascii, a string of the source that has a byte >= 0x7F was written
by a generic path: put() per piece and a validate-and-decode step for every
character beyond ASCII. It now goes through a writer with the cursor in a
local variable. The part of the string that has nothing to escape is found
16 bytes at a time (SSE2 or NEON; the existing 8-byte scan without vector
support) and copied; strings that are ASCII throughout are still decided by
the 8-byte check of dump() with ensure_ascii (a 16-byte scan for them was
2-5 % slower for short strings). The two-byte sequences (lead C2..DF) and
the three-byte sequences whose continuation bytes may have any value (lead
E1..EF except ED) are decoded with a few compares; E0, ED, four-byte
sequences, and damaged input go through validate_one_utf8() as before, so
damaged images still throw type_error.316.

dump(-1, ' ', true) instructions per call, macOS arm64 (develop -> now);
the compact dump without ensure_ascii and dumps of ASCII text are unchanged:

  twitter 5.42M -> 3.57M    citm 3.28M -> 3.24M     jeopardy(4k) 5.78M -> 5.65M
  CJK text, raw 66.5M -> 35.3M    escaped \u 32.7M -> 17.0M
  Latin text with accents 11.5M -> 9.4M    ASCII text 4.40M -> 4.40M

Linux instructions (GCC 13 / Clang 18), dump(-1, ' ', true):

  twitter -9 % / -33 %    CJK text -14 % / -47 %    Latin text -8 % / -18 %
  citm -0.2 % / -0.4 %    ASCII text -0.1 % / -0.3 %

The compact dump without ensure_ascii is unchanged (canada 0.0 %).

The unit test covers every character of the Basic Multilingual Plane, and a
character that needs an escape at each position of strings of 0..40 bytes.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Find the end of the plain run of a decoded string 16 bytes at a time with SSE2

The strings of a document that had escape sequences are written in runs
without a quote, backslash, or control character; the compact dump found the
end of a run 8 bytes at a time with a SWAR test. With SSE2 (x86-64) it now
compares 16 bytes per step. NEON keeps the SWAR code: there the mask of a
step costs more than it saves on the short runs that are typical here.

Instructions of dump(), Linux (GCC 13 / Clang 18):

  twitter.json                  1.99M -> 1.88M (-5.2 %) / 1.97M -> 1.92M (-3.0 %)
  CJK text with \u escapes      5.98M -> 4.52M (-24.5 %) / 5.30M -> 4.25M (-19.9 %)
  ASCII text with a few escapes 4.28M -> 4.09M (-4.4 %) / 4.24M -> 4.11M (-3.2 %)
  jeopardy (4,000 questions)    5.08M -> 4.78M (-5.8 %) / 4.97M -> 4.80M (-3.4 %)

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Rename depth_counter's open() and close() to enter() and leave()

Flawfinder took them for the file functions open() and close().

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix the clang-tidy findings in json_view's dump and its tests

- the SSE2 loop of the string scan is within NOLINTBEGIN/END
  (portability-simd-intrinsics), as in to_chars.hpp
- depth_counter<false>'s leave() and get() keep the interface of
  depth_counter<true> instead of being static
- write_ascii() compares the room left as a size_t variable
- the test's texts are a std::array, and a JSON fragment with escapes
  is a raw string literal

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Move dump_ascii() and dump_pretty() next to dump()

Address a review comment: the two dispatchers that dump() calls now follow
it directly.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-24 07:38:17 +02:00
dependabot[bot] 1e9222f955 Bump dompurify from 3.4.16 to 3.4.17 in /docs/mkdocs/scripts/mermaid (#5904) 
Bumps [dompurify](https://github.com/cure53/DOMPurify) from 3.4.16 to 3.4.17.
- [Release notes](https://github.com/cure53/DOMPurify/releases)
- [Commits](https://github.com/cure53/DOMPurify/compare/3.4.16...3.4.17)

---
updated-dependencies:
- dependency-name: dompurify
  dependency-version: 3.4.17
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-10-24 07:37:42 +02:00
Niels Lohmann 70d1baa57a Remove a nested duplicate guard from the json_view image test 
clang-tidy reported the inner #if !defined(JSON_NOEXCEPTION) as
redundant (the test case is in the same #if) and a find() != npos, which
gets the NOLINT the other tests use.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-23 18:53:34 +02:00
Niels Lohmann 536672e927 Dispatch on the compile-time conditions of the json_view comparison 
MSVC (/W4 /WX) reported C4127 (conditional expression is constant) for
the template parameter EA and the object type's ordering in an if
condition; both are now tag-dispatched.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-23 09:44:39 +02:00
Niels Lohmann 55cdc93ff8 Keep the byte comparison helpers in compare.hpp 
Address review comments: bytes_equal() and bytes_less() move from node.hpp
to compare.hpp, their only user; key_less() no longer splits the comment of
resolve_indices() from its function; and the loop counters of bytes_equal()
and all_digits() are scoped to their loops.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-23 06:47:22 +02:00
Niels Lohmann db5bd7fbcc Fix the compiler and lint failures of the json_view comparison 
- all_digits(): the lambda declares its constant itself; MSVC does not
  let a lambda without captures read a constexpr local of the function
- equal(): the frame's "in document order" index is a namespace-scope
  constant (document_order); the local constexpr npos in the default
  member initializers of the local struct crashed GCC 9 and MinGW GCC
  and was not found by MSVC
- keys_differ_early() keeps the first keys in std::arrays (cpplint:
  variable-length arrays; clang-tidy: C arrays), and the basic_json
  side's first_keys() is not noexcept (key() may throw)
- the numbers of one type are compared in an if chain instead of a
  nested conditional operator
- the dump test avoids C arrays and temporary string concatenations

View comparison instructions are unchanged (Apple clang, twitter, citm,
canada: -0.1 % to +0.1 %).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-22 20:13:22 +02:00
Niels Lohmann 89bd5d63e1 Rename names that Flawfinder takes for the file function open() 
The comparison cursors' open() is now enter(), and a test's open and
close strings are head and tail.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-22 13:27:54 +02:00
Niels Lohmann 3fb7d9c710 Check the float tokens of a json_document image without a sort and a scan of every character 
load(image_check::full) checked every float token of an image by sorting
the token ranges, scanning the token character by character, and comparing
the digit layout with the node's. Now:

- the ranges are sorted only if they are not sorted already (save() writes
  them in the order of the nodes, so they are);
- a token of the shape [-]digits.digits whose digit counts are the ones the
  node records (nearly all of them) is accepted by checking its digits eight
  at a time, without the general scan;
- the general scan skips runs of digits eight at a time.

Damaged tokens are still rejected as before: the check accepts exactly the
tokens that the general scan accepts.

Instructions of load(image_check::full) (develop -> now):

  macOS arm64        canada 44.6M -> 30.9M (-31 %), jeopardy 11.3M -> 10.7M,
                     citm 10.4M -> 10.0M, twitter 5.06M -> 4.94M
  Linux GCC 13       canada -50 %, jeopardy -43 %, twitter -31 %, citm -28 %
  Linux Clang 18     canada -42 %, jeopardy -48 %, twitter -35 %, citm -32 %

image_check::bounds is unchanged (within 0.4 %).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-22 09:30:44 +02:00
Niels Lohmann fa8134ae85 Compare json_view values without a vector per container and a basic_json per number 
operator== of views (view/view, view/json, and editable documents) built two
vectors of members for every object it entered, sorted them with stable_sort
for every object, and converted each number to a basic_json value. It now:

- keeps the members of all open objects in one vector (reserved with
  geometric growth, so that nested objects do not reallocate), and orders
  them by insertion sort for up to 16 members, or not at all when the keys
  ascend already;
- compares numbers of one type directly, and builds a basic_json only for
  numbers of different types;
- compares short texts (keys, numbers, most strings) with two overlapping
  loads (bytes_equal(), in node.hpp);
- first walks both documents in lockstep, in the order of their text, which
  finds most pairs of equal values without the work of the comparison. The
  walk only enters an object if the comparison would visit its members in the
  same order (the object type keeps the order, the keys ascend, or all values
  are scalars), so a difference is never found later than before, and it
  records the containers it found different, so that the comparison does not
  walk them again;
- rejects values of different types and arrays of different sizes before any
  of this, and, for an object type that keeps the order, objects whose first
  (up to four) members have different keys, which ends a comparison that
  differs near the start without preparing the members.

Results are unchanged: numbers compare by value across types, duplicate keys
resolve as parse() resolves them, and objects compare in document order for
ordered_json and by key otherwise.

Instructions of one comparison of a document with an equal copy, macOS arm64
(develop -> now), json_document / ordered_json_document / editable:

  twitter    16.1M -> 7.0M   17.7M -> 1.5M    16.6M -> 7.3M
  citm       45.0M -> 3.4M   47.0M -> 3.2M    45.7M -> 4.3M
  canada    111.4M -> 13.7M  111.4M -> 13.7M  113.3M -> 15.0M
  jeopardy   26.2M -> 3.7M   27.3M -> 3.7M    26.6M -> 12.1M

and of a view with a basic_json value:

  twitter    11.4M -> 5.9M   11.2M -> 7.3M    11.6M -> 5.9M
  citm       33.9M -> 9.7M   32.6M -> 9.1M    34.6M -> 9.8M
  canada     92.9M -> 32.1M  92.9M -> 32.1M   93.9M -> 32.8M

and Linux instruction counts (GCC 13 / Clang 18) of the same comparisons:

  document with an equal copy   json_document     ordered_json_document   editable
    twitter                     -49 % / -52 %     -89 % / -89 %           -48 % / -50 %
    citm                        -90 % / -89 %     -91 % / -90 %           -88 % / -87 %
    jeopardy (4,000 questions)  -85 % / -85 %     -86 % / -86 %           -46 % / -49 %
  view with a basic_json value
    twitter                     -41 % / -45 %     -28 % / -35 %           -45 % / -44 %
    citm                        -63 % / -65 %     -63 % / -66 %           -65 % / -64 %
    jeopardy                    -39 % / -43 %     -14 % / -24 %           -44 % / -43 %

A sweep that places one difference at about 2,100 positions of eight
documents (a token at an even spacing, the first and last tokens, the first,
middle and last member of the root removed, one appended, the document
wrapped) in five kinds of comparison (view/view of read-only and of editable
documents, view/json, json_document, editable/view) finds the same results as
before and no placement slower on the Mac (Clang); on Linux no placement is
slower than 3 % (Clang and GCC).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-22 09:30:42 +02:00
48 changed files with 10942 additions and 4454 deletions
Expand all files Collapse all files

No files matched your search

docs/mkdocs/docs/api/basic_json_view/at.md
+2 -1
View File
@@ -80,7 +80,8 @@ None of these exceptions carry a [`JSON_DIAGNOSTICS`](../macros/json_diagnostics
Objects with 128 or more members get a hash index of their keys while parsing, so that a lookup in them takes Objects with 128 or more members get a hash index of their keys while parsing, so that a lookup in them takes
constant time on average. constant time on average.
2. Linear in `idx`: elements are skipped one at a time from the first one, since they are not a fixed size in the 2. Linear in `idx`: elements are skipped one at a time from the first one, since they are not a fixed size in the
index (unlike `BasicJsonType`'s array, which is random-access). index (unlike `BasicJsonType`'s array, which is random-access). Constant if no element of the array is an array
or object, since the elements are then adjacent in the index.
3. Linear in the number of reference tokens of `ptr` and, for each token, in the number of members of the object at 3. Linear in the number of reference tokens of `ptr` and, for each token, in the number of members of the object at
that level (as 1.) or the index into the array (as 2.). that level (as 1.) or the index into the array (as 2.).
docs/mkdocs/docs/api/basic_json_view/back.md
+2 -1
View File
@@ -28,7 +28,8 @@ This exception does not carry a [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.
Linear in the size of the array or object: unlike [`front()`](front.md), which only ever looks at the first Linear in the size of the array or object: unlike [`front()`](front.md), which only ever looks at the first
element, `back()` has to walk every element to find where they end, since elements are not a fixed size in the element, `back()` has to walk every element to find where they end, since elements are not a fixed size in the
index. index. Constant if no element of the array is an array or object, since the elements are then adjacent
in the index.
## Notes ## Notes
docs/mkdocs/docs/api/basic_json_view/operator[].md
+2 -1
View File
@@ -86,7 +86,8 @@ None of these exceptions carry a [`JSON_DIAGNOSTICS`](../macros/json_diagnostics
Objects with 128 or more members get a hash index of their keys while parsing, so that a lookup in them takes Objects with 128 or more members get a hash index of their keys while parsing, so that a lookup in them takes
constant time on average. constant time on average.
2. Linear in `idx`: elements are skipped one at a time from the first one, since they are not a fixed size in the 2. Linear in `idx`: elements are skipped one at a time from the first one, since they are not a fixed size in the
index (unlike `BasicJsonType`'s array, which is random-access). index (unlike `BasicJsonType`'s array, which is random-access). Constant if no element of the array is an array
or object, since the elements are then adjacent in the index.
3. Linear in the number of reference tokens of `ptr` and, for each token, in the number of members of the object at 3. Linear in the number of reference tokens of `ptr` and, for each token, in the number of members of the object at
that level (as 1.) or the index into the array (as 2.). that level (as 1.) or the index into the array (as 2.).
docs/mkdocs/scripts/mermaid/package-lock.json
Generated
+3 -3
View File
@@ -1105,9 +1105,9 @@
} }
}, },
"node_modules/dompurify": { "node_modules/dompurify": {
"version": "3.4.16", "version": "3.4.17",
"resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.4.16.tgz", "resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.4.17.tgz",
"integrity": "sha512-sqo+pNp3qRhCIpbgRi1y8Tgk27Bo2Ry7w0dC1NBeNTdZChWjz9Xb/KOoZbRP/R6pQZ80Qw8YhXw13hWWBbMRnQ==", "integrity": "sha512-B/pSlYVHQ58kqISD/yDjuJ1yfzI34m1gF4lAod2bwSRhiN4K5Cl3H8Af7hqB2PmTL911tdEudnnXn5FnhRoz0w==",
"license": "(MPL-2.0 OR Apache-2.0)", "license": "(MPL-2.0 OR Apache-2.0)",
"optionalDependencies": { "optionalDependencies": {
"@types/trusted-types": "^2.0.7" "@types/trusted-types": "^2.0.7"
include/nlohmann/detail/conversions/from_json.hpp
+32 -36
View File
@@ -69,10 +69,12 @@ void from_json(const BasicJsonType& j, std::optional<T>& opt)
} }
#endif // JSON_HAS_CPP_17 #endif // JSON_HAS_CPP_17
// read a number value of any of the three number types into val; // overloads for basic_json template parameters
// returns false (leaving val untouched) if j is not a number template < typename BasicJsonType, typename ArithmeticType,
template<typename BasicJsonType, typename ArithmeticType> enable_if_t < std::is_arithmetic<ArithmeticType>::value&&
bool get_number_value(const BasicJsonType& j, ArithmeticType& val) !std::is_same<ArithmeticType, typename BasicJsonType::boolean_t>::value,
int > = 0 >
void get_arithmetic_value(const BasicJsonType& j, ArithmeticType& val)
{ {
switch (static_cast<value_t>(j)) switch (static_cast<value_t>(j))
{ {
@@ -83,17 +85,17 @@ bool get_number_value(const BasicJsonType& j, ArithmeticType& val)
const auto* ptr = j.template get_ptr<const typename BasicJsonType::number_unsigned_t*>(); const auto* ptr = j.template get_ptr<const typename BasicJsonType::number_unsigned_t*>();
JSON_ASSERT(ptr != nullptr); JSON_ASSERT(ptr != nullptr);
val = static_cast<ArithmeticType>(*ptr); val = static_cast<ArithmeticType>(*ptr);
return true; break;
} }
case value_t::number_integer: case value_t::number_integer:
{ {
val = static_cast<ArithmeticType>(*j.template get_ptr<const typename BasicJsonType::number_integer_t*>()); val = static_cast<ArithmeticType>(*j.template get_ptr<const typename BasicJsonType::number_integer_t*>());
return true; break;
} }
case value_t::number_float: case value_t::number_float:
{ {
val = static_cast<ArithmeticType>(*j.template get_ptr<const typename BasicJsonType::number_float_t*>()); val = static_cast<ArithmeticType>(*j.template get_ptr<const typename BasicJsonType::number_float_t*>());
return true; break;
} }
case value_t::null: case value_t::null:
@@ -104,20 +106,7 @@ bool get_number_value(const BasicJsonType& j, ArithmeticType& val)
case value_t::binary: case value_t::binary:
case value_t::discarded: case value_t::discarded:
default: default:
return false; throw_type_must_be("number", j);
}
}
// overloads for basic_json template parameters
template < typename BasicJsonType, typename ArithmeticType,
enable_if_t < std::is_arithmetic<ArithmeticType>::value&&
!std::is_same<ArithmeticType, typename BasicJsonType::boolean_t>::value,
int > = 0 >
void get_arithmetic_value(const BasicJsonType& j, ArithmeticType& val)
{
if (JSON_HEDLEY_UNLIKELY(!get_number_value(j, val)))
{
throw_type_must_be("number", j);
} }
} }
@@ -267,18 +256,18 @@ auto from_json_array_impl(const BasicJsonType& j, std::array<T, N>& arr,
} }
} }
// reserve() is called through this pair (used for arrays and objects alike) // reserve() is called through this pair (modeled on from_json_object_reserve)
// so from_json_array_impl and the object from_json have a single body for both // so from_json_array_impl below has a single body for both ConstructibleArrayType
// containers that support reserve() and those that don't. // that support reserve() and those that don't.
template<typename ConstructibleContainerType> template<typename ConstructibleArrayType>
auto reserve_if_possible(ConstructibleContainerType& c, typename ConstructibleContainerType::size_type size, priority_tag<1> /*unused*/) auto from_json_array_reserve(ConstructibleArrayType& arr, typename ConstructibleArrayType::size_type size, priority_tag<1> /*unused*/)
-> decltype(c.reserve(size), void()) -> decltype(arr.reserve(size), void())
{ {
c.reserve(size); arr.reserve(size);
} }
template<typename ConstructibleContainerType> template<typename ConstructibleArrayType>
void reserve_if_possible(ConstructibleContainerType& /*c*/, std::size_t /*size*/, priority_tag<0> /*unused*/) void from_json_array_reserve(ConstructibleArrayType& /*arr*/, std::size_t /*size*/, priority_tag<0> /*unused*/)
{} {}
// append through push_back() where available; containers without it (such as // append through push_back() where available; containers without it (such as
@@ -308,7 +297,7 @@ auto from_json_array_impl(const BasicJsonType& j, ConstructibleArrayType& arr, p
{ {
ConstructibleArrayType ret; ConstructibleArrayType ret;
const auto* inner_array = j.template get_ptr<const typename BasicJsonType::array_t*>(); const auto* inner_array = j.template get_ptr<const typename BasicJsonType::array_t*>();
reserve_if_possible(ret, inner_array->size(), priority_tag<1> {}); from_json_array_reserve(ret, inner_array->size(), priority_tag<1> {});
for (const auto& elem : *inner_array) for (const auto& elem : *inner_array)
{ {
// get<BasicJsonType>() returns *this, this won't call a from_json // get<BasicJsonType>() returns *this, this won't call a from_json
@@ -389,6 +378,17 @@ inline void from_json(const BasicJsonType& j, CompatibleArrayType& bin)
} }
} }
template<typename ConstructibleObjectType>
auto from_json_object_reserve(ConstructibleObjectType& obj, typename ConstructibleObjectType::size_type size, priority_tag<1> /*unused*/)
-> decltype(obj.reserve(size), void())
{
obj.reserve(size);
}
template<typename ConstructibleObjectType>
inline void from_json_object_reserve(ConstructibleObjectType& /*obj*/, std::size_t /*size*/, priority_tag<0> /*unused*/)
{}
// the keys of the source are sorted for the usual object types, so hinting the // the keys of the source are sorted for the usual object types, so hinting the
// end makes every insertion constant-time (for containers that order their keys // end makes every insertion constant-time (for containers that order their keys
// the same way); containers without emplace_hint() (such as ordered_map) fall // the same way); containers without emplace_hint() (such as ordered_map) fall
@@ -417,7 +417,7 @@ inline void from_json(const BasicJsonType& j, ConstructibleObjectType& obj)
ConstructibleObjectType ret; ConstructibleObjectType ret;
const auto* inner_object = j.template get_ptr<const typename BasicJsonType::object_t*>(); const auto* inner_object = j.template get_ptr<const typename BasicJsonType::object_t*>();
reserve_if_possible(ret, inner_object->size(), priority_tag<1> {}); from_json_object_reserve(ret, inner_object->size(), priority_tag<1> {});
for (const auto& p : *inner_object) for (const auto& p : *inner_object)
{ {
from_json_object_emplace(ret, p.first, p.second.template get<typename ConstructibleObjectType::mapped_type>(), priority_tag<1> {}); from_json_object_emplace(ret, p.first, p.second.template get<typename ConstructibleObjectType::mapped_type>(), priority_tag<1> {});
@@ -437,10 +437,6 @@ template < typename BasicJsonType, typename ArithmeticType,
int > = 0 > int > = 0 >
inline void from_json(const BasicJsonType& j, ArithmeticType& val) inline void from_json(const BasicJsonType& j, ArithmeticType& val)
{ {
// the number cases repeat get_number_value() on purpose: calling it here
// changes the inlining of get<T>() in the std::array conversion, which then
// runs 2.4x slower (GetArray benchmark, #5891); unlike get_arithmetic_value(),
// a boolean is accepted
switch (static_cast<value_t>(j)) switch (static_cast<value_t>(j))
{ {
case value_t::number_unsigned: case value_t::number_unsigned:
include/nlohmann/detail/exceptions.hpp
+36 -32
View File
@@ -66,7 +66,6 @@ enum class exception_id : int
invalid_string_or_size = 113, ///< a binary input contains an invalid string or size specification invalid_string_or_size = 113, ///< a binary input contains an invalid string or size specification
bson_unsupported_type = 114, ///< a BSON record type is not supported bson_unsupported_type = 114, ///< a BSON record type is not supported
invalid_high_precision_number = 115, ///< a UBJSON/BJData high-precision number cannot be parsed invalid_high_precision_number = 115, ///< a UBJSON/BJData high-precision number cannot be parsed
invalid_document_image = 116, ///< json_document::load() rejects an image
// invalid_iterator // invalid_iterator
iterators_incompatible = 201, ///< the iterators of a range belong to different values iterators_incompatible = 201, ///< the iterators of a range belong to different values
iterator_from_other_value = 202, ///< an iterator does not belong to the value it is used with iterator_from_other_value = 202, ///< an iterator does not belong to the value it is used with
@@ -101,8 +100,6 @@ enum class exception_id : int
invalid_utf8 = 316, ///< dump() finds a string that is not valid UTF-8 invalid_utf8 = 316, ///< dump() finds a string that is not valid UTF-8
type_not_serializable = 317, ///< a value cannot be serialized to the requested binary format type_not_serializable = 317, ///< a value cannot be serialized to the requested binary format
enum_key_duplicate = 318, ///< an enum-keyed map has two keys that serialize to the same string enum_key_duplicate = 318, ///< an enum-keyed map has two keys that serialize to the same string
document_binary_value = 319, ///< a binary value is stored in a json_document
document_image_unsupported = 320, ///< a json_document image cannot be saved or loaded (discarded document, big-endian target)
discarded_value_used = 321, ///< a discarded value is used discarded_value_used = 321, ///< a discarded value is used
// out_of_range // out_of_range
array_index_out_of_range = 401, ///< an array index is out of range array_index_out_of_range = 401, ///< an array index is out of range
@@ -120,7 +117,6 @@ enum class exception_id : int
patch_remove_parent_not_container = 413, ///< the parent of a JSON Patch 'remove' target is not a container patch_remove_parent_not_container = 413, ///< the parent of a JSON Patch 'remove' target is not a container
patch_move_into_child = 414, ///< a JSON Patch 'move' moves a value into one of its children patch_move_into_child = 414, ///< a JSON Patch 'move' moves a value into one of its children
subtype_out_of_range = 415, ///< a binary subtype does not fit into one byte subtype_out_of_range = 415, ///< a binary subtype does not fit into one byte
document_too_large = 416, ///< a json_document input, its edit storage, or an image reaches the 32-bit (4 GiB) limit
// other_error // other_error
internal_error = 500, ///< unreachable code was reached internal_error = 500, ///< unreachable code was reached
patch_test_failed = 501, ///< a JSON Patch 'test' operation failed patch_test_failed = 501, ///< a JSON Patch 'test' operation failed
@@ -150,13 +146,6 @@ class exception : public std::exception
return concat("[json.exception.", ename, '.', std::to_string(id_), "] "); return concat("[json.exception.", ename, '.', std::to_string(id_), "] ");
} }
/// build the message "[json.exception.<ename>.<id>] <diagnostics><what_arg>"
template<typename BasicJsonContext>
static std::string what_impl(const std::string& ename, int id_, const std::string& what_arg, BasicJsonContext context)
{
return concat(name(ename, id_), diagnostics(context), what_arg);
}
static std::string diagnostics(std::nullptr_t /*leaf_element*/) static std::string diagnostics(std::nullptr_t /*leaf_element*/)
{ {
return ""; return "";
@@ -169,33 +158,48 @@ class exception : public std::exception
std::vector<std::string> tokens; std::vector<std::string> tokens;
for (const auto* current = leaf_element; current != nullptr && current->m_parent != nullptr; current = current->m_parent) for (const auto* current = leaf_element; current != nullptr && current->m_parent != nullptr; current = current->m_parent)
{ {
const auto* parent = current->m_parent; switch (current->m_parent->type())
if (parent->is_array())
{ {
for (std::size_t i = 0; i < parent->m_data.m_value.array->size(); ++i) case value_t::array:
{ {
if (&parent->m_data.m_value.array->operator[](i) == current) for (std::size_t i = 0; i < current->m_parent->m_data.m_value.array->size(); ++i)
{ {
tokens.emplace_back(std::to_string(i)); if (&current->m_parent->m_data.m_value.array->operator[](i) == current)
break; {
tokens.emplace_back(std::to_string(i));
break;
}
} }
break;
} }
}
else if (parent->is_object()) case value_t::object:
{
for (const auto& element : *parent->m_data.m_value.object)
{ {
if (&element.second == current) for (const auto& element : *current->m_parent->m_data.m_value.object)
{ {
// data() is null-terminated, so a key containing if (&element.second == current)
// a null byte is cut short here rather than {
// truncating the whole message at what() // data() is null-terminated, so a key containing
tokens.emplace_back(element.first.data()); // a null byte is cut short here rather than
break; // truncating the whole message at what()
tokens.emplace_back(element.first.data());
break;
}
} }
break;
} }
case value_t::null: // LCOV_EXCL_LINE
case value_t::string: // LCOV_EXCL_LINE
case value_t::boolean: // LCOV_EXCL_LINE
case value_t::number_integer: // LCOV_EXCL_LINE
case value_t::number_unsigned: // LCOV_EXCL_LINE
case value_t::number_float: // LCOV_EXCL_LINE
case value_t::binary: // LCOV_EXCL_LINE
case value_t::discarded: // LCOV_EXCL_LINE
default: // LCOV_EXCL_LINE
break; // LCOV_EXCL_LINE
} }
// all other parent types (primitives) contribute no token
} }
if (tokens.empty()) if (tokens.empty())
@@ -311,7 +315,7 @@ class invalid_iterator : public exception
template<typename BasicJsonContext, enable_if_t<is_basic_json_context<BasicJsonContext>::value, int> = 0> template<typename BasicJsonContext, enable_if_t<is_basic_json_context<BasicJsonContext>::value, int> = 0>
static invalid_iterator create(int id_, const std::string& what_arg, BasicJsonContext context) static invalid_iterator create(int id_, const std::string& what_arg, BasicJsonContext context)
{ {
const std::string w = exception::what_impl("invalid_iterator", id_, what_arg, context); const std::string w = concat(exception::name("invalid_iterator", id_), exception::diagnostics(context), what_arg);
return {id_, w.c_str()}; return {id_, w.c_str()};
} }
@@ -335,7 +339,7 @@ class type_error : public exception
template<typename BasicJsonContext, enable_if_t<is_basic_json_context<BasicJsonContext>::value, int> = 0> template<typename BasicJsonContext, enable_if_t<is_basic_json_context<BasicJsonContext>::value, int> = 0>
static type_error create(int id_, const std::string& what_arg, BasicJsonContext context) static type_error create(int id_, const std::string& what_arg, BasicJsonContext context)
{ {
const std::string w = exception::what_impl("type_error", id_, what_arg, context); const std::string w = concat(exception::name("type_error", id_), exception::diagnostics(context), what_arg);
return {id_, w.c_str()}; return {id_, w.c_str()};
} }
@@ -358,7 +362,7 @@ class out_of_range : public exception
template<typename BasicJsonContext, enable_if_t<is_basic_json_context<BasicJsonContext>::value, int> = 0> template<typename BasicJsonContext, enable_if_t<is_basic_json_context<BasicJsonContext>::value, int> = 0>
static out_of_range create(int id_, const std::string& what_arg, BasicJsonContext context) static out_of_range create(int id_, const std::string& what_arg, BasicJsonContext context)
{ {
const std::string w = exception::what_impl("out_of_range", id_, what_arg, context); const std::string w = concat(exception::name("out_of_range", id_), exception::diagnostics(context), what_arg);
return {id_, w.c_str()}; return {id_, w.c_str()};
} }
@@ -381,7 +385,7 @@ class other_error : public exception
template<typename BasicJsonContext, enable_if_t<is_basic_json_context<BasicJsonContext>::value, int> = 0> template<typename BasicJsonContext, enable_if_t<is_basic_json_context<BasicJsonContext>::value, int> = 0>
static other_error create(int id_, const std::string& what_arg, BasicJsonContext context) static other_error create(int id_, const std::string& what_arg, BasicJsonContext context)
{ {
const std::string w = exception::what_impl("other_error", id_, what_arg, context); const std::string w = concat(exception::name("other_error", id_), exception::diagnostics(context), what_arg);
return {id_, w.c_str()}; return {id_, w.c_str()};
} }
include/nlohmann/detail/input/binary_reader.hpp
+83 -109
View File
@@ -86,7 +86,7 @@ JSON_INLINE_VARIABLE constexpr std::size_t max_valueless_container_size = 1 << 2
/*! /*!
@brief deserialization of BJData, BON8, BSON, CBOR, MessagePack, and UBJSON values @brief deserialization of BJData, BON8, BSON, CBOR, MessagePack, and UBJSON values
*/ */
template<typename BasicJsonType, typename InputAdapterType, typename SAX = json_sax_dom_parser<BasicJsonType, InputAdapterType>> template<typename BasicJsonType, typename InputAdapterType, typename SAX = json_sax_dom_binary_parser<BasicJsonType, InputAdapterType>>
class binary_reader class binary_reader
{ {
using number_integer_t = typename BasicJsonType::number_integer_t; using number_integer_t = typename BasicJsonType::number_integer_t;
@@ -293,40 +293,6 @@ class binary_reader
return is_object ? sax->end_object() : sax->end_array(); return is_object ? sax->end_object() : sax->end_array();
} }
/*!
@brief close the innermost open container and report whether that
completed the value being read
The value begun by the driver is complete once its container is not
inside another one.
@param[out] value_complete whether no container is left open
@return whether the SAX parser accepted the end event
*/
bool leave_container_and_check(bool& value_complete)
{
if (JSON_HEDLEY_UNLIKELY(!leave_container()))
{
return false;
}
value_complete = container_stack.empty();
return true;
}
/*!
@brief read an object key and pass it to the SAX parser
@tparam GetKey member function that reads the key into a string
@param[in,out] key string the key is read into; cleared first
@return whether reading the key and the SAX event succeeded
*/
template<bool (binary_reader::*GetKey)(string_t&)>
bool read_object_key(string_t& key)
{
key.clear();
return (this->*GetKey)(key) && sax->key(key);
}
////////// //////////
// BSON // // BSON //
////////// //////////
@@ -1098,34 +1064,6 @@ class binary_reader
return last_byte_error(exception_id::invalid_string_or_size, concat("indefinite-length ", type_name, " is not allowed inside indefinite-length ", type_name, "; last byte: 0x", get_token_string()), context); return last_byte_error(exception_id::invalid_string_or_size, concat("indefinite-length ", type_name, " is not allowed inside indefinite-length ", type_name, "; last byte: 0x", get_token_string()), context);
} }
/*!
@brief report an object key that is not a string
@param[in] found description of what was found instead of a string key
@return false
*/
bool unsupported_key_type(const char* found) const
{
return last_byte_error(exception_id::invalid_string_or_size, "only string keys are supported, but found ", "object key", found);
}
/*!
@brief read the length of a CBOR string or byte array and then its bytes
@tparam NumberType the type of the length field
@tparam ContainerType the destination container (string_t or binary_t)
@param[in] context further context information (for diagnostics)
@param[out] result container the bytes are appended to
@return whether reading completed
*/
template<typename NumberType, typename ContainerType>
bool get_cbor_sized_bytes(const char* context, ContainerType& result)
{
NumberType len{};
return get_number(len) && get_bytes(len, context, result);
}
/*! /*!
@brief reads a definite-length CBOR string @brief reads a definite-length CBOR string
@@ -1176,22 +1114,26 @@ class binary_reader
case 0x78: // UTF-8 string (one-byte uint8_t for n follows) case 0x78: // UTF-8 string (one-byte uint8_t for n follows)
{ {
return get_cbor_sized_bytes<std::uint8_t>("string", result); std::uint8_t len{};
return get_number(len) && get_string(len, result);
} }
case 0x79: // UTF-8 string (two-byte uint16_t for n follow) case 0x79: // UTF-8 string (two-byte uint16_t for n follow)
{ {
return get_cbor_sized_bytes<std::uint16_t>("string", result); std::uint16_t len{};
return get_number(len) && get_string(len, result);
} }
case 0x7A: // UTF-8 string (four-byte uint32_t for n follow) case 0x7A: // UTF-8 string (four-byte uint32_t for n follow)
{ {
return get_cbor_sized_bytes<std::uint32_t>("string", result); std::uint32_t len{};
return get_number(len) && get_string(len, result);
} }
case 0x7B: // UTF-8 string (eight-byte uint64_t for n follow) case 0x7B: // UTF-8 string (eight-byte uint64_t for n follow)
{ {
return get_cbor_sized_bytes<std::uint64_t>("string", result); std::uint64_t len{};
return get_number(len) && get_string(len, result);
} }
default: default:
@@ -1326,7 +1268,7 @@ class binary_reader
break; break;
} }
return unsupported_key_type(found); return last_byte_error(exception_id::invalid_string_or_size, "only string keys are supported, but found ", "object key", found);
} }
/*! /*!
@@ -1379,22 +1321,30 @@ class binary_reader
case 0x58: // Binary data (one-byte uint8_t for n follows) case 0x58: // Binary data (one-byte uint8_t for n follows)
{ {
return get_cbor_sized_bytes<std::uint8_t>("binary", result); std::uint8_t len{};
return get_number(len) &&
get_binary(len, result);
} }
case 0x59: // Binary data (two-byte uint16_t for n follow) case 0x59: // Binary data (two-byte uint16_t for n follow)
{ {
return get_cbor_sized_bytes<std::uint16_t>("binary", result); std::uint16_t len{};
return get_number(len) &&
get_binary(len, result);
} }
case 0x5A: // Binary data (four-byte uint32_t for n follow) case 0x5A: // Binary data (four-byte uint32_t for n follow)
{ {
return get_cbor_sized_bytes<std::uint32_t>("binary", result); std::uint32_t len{};
return get_number(len) &&
get_binary(len, result);
} }
case 0x5B: // Binary data (eight-byte uint64_t for n follow) case 0x5B: // Binary data (eight-byte uint64_t for n follow)
{ {
return get_cbor_sized_bytes<std::uint64_t>("binary", result); std::uint64_t len{};
return get_number(len) &&
get_binary(len, result);
} }
default: default:
@@ -1609,12 +1559,12 @@ class binary_reader
if (at_end) if (at_end)
{ {
bool value_complete = false; if (JSON_HEDLEY_UNLIKELY(!leave_container()))
if (JSON_HEDLEY_UNLIKELY(!leave_container_and_check(value_complete)))
{ {
return false; return false;
} }
if (value_complete) // the value begun here is complete once its container is
if (container_stack.empty())
{ {
return true; return true;
} }
@@ -1623,7 +1573,8 @@ class binary_reader
if (top.is_object) if (top.is_object)
{ {
if (JSON_HEDLEY_UNLIKELY(!read_object_key<&binary_reader::get_cbor_object_key>(key))) key.clear();
if (JSON_HEDLEY_UNLIKELY(!get_cbor_object_key(key) || !sax->key(key)))
{ {
return false; return false;
} }
@@ -2094,17 +2045,20 @@ class binary_reader
case 0xD9: // str 8 case 0xD9: // str 8
{ {
return get_string_with_length<std::uint8_t>(result, context); std::uint8_t len{};
return get_number(len) && get_string(len, result) && check_string_utf8(result, context);
} }
case 0xDA: // str 16 case 0xDA: // str 16
{ {
return get_string_with_length<std::uint16_t>(result, context); std::uint16_t len{};
return get_number(len) && get_string(len, result) && check_string_utf8(result, context);
} }
case 0xDB: // str 32 case 0xDB: // str 32
{ {
return get_string_with_length<std::uint32_t>(result, context); std::uint32_t len{};
return get_number(len) && get_string(len, result) && check_string_utf8(result, context);
} }
default: default:
@@ -2201,7 +2155,7 @@ class binary_reader
break; break;
} }
return unsupported_key_type(found); return last_byte_error(exception_id::invalid_string_or_size, "only string keys are supported, but found ", "object key", found);
} }
/*! /*!
@@ -2346,12 +2300,12 @@ class binary_reader
if (container_stack.back().remaining == 0) if (container_stack.back().remaining == 0)
{ {
bool value_complete = false; if (JSON_HEDLEY_UNLIKELY(!leave_container()))
if (JSON_HEDLEY_UNLIKELY(!leave_container_and_check(value_complete)))
{ {
return false; return false;
} }
if (value_complete) // the value begun here is complete once its container is
if (container_stack.empty())
{ {
return true; return true;
} }
@@ -2364,7 +2318,8 @@ class binary_reader
if (is_object) if (is_object)
{ {
get(); get();
if (JSON_HEDLEY_UNLIKELY(!read_object_key<&binary_reader::get_msgpack_object_key>(key))) key.clear();
if (JSON_HEDLEY_UNLIKELY(!get_msgpack_object_key(key) || !sax->key(key)))
{ {
return false; return false;
} }
@@ -2548,7 +2503,8 @@ class binary_reader
{ {
case 'U': case 'U':
{ {
return get_string_with_length<std::uint8_t>(result, context); std::uint8_t len{};
return get_number(len) && get_string(len, result) && check_string_utf8(result, context);
} }
case 'i': case 'i':
@@ -2581,7 +2537,8 @@ class binary_reader
{ {
break; break;
} }
return get_string_with_length<std::uint16_t>(result, context); std::uint16_t len{};
return get_number(len) && get_string(len, result) && check_string_utf8(result, context);
} }
case 'm': case 'm':
@@ -2590,7 +2547,8 @@ class binary_reader
{ {
break; break;
} }
return get_string_with_length<std::uint32_t>(result, context); std::uint32_t len{};
return get_number(len) && get_string(len, result) && check_string_utf8(result, context);
} }
case 'M': case 'M':
@@ -2599,7 +2557,8 @@ class binary_reader
{ {
break; break;
} }
return get_string_with_length<std::uint64_t>(result, context); std::uint64_t len{};
return get_number(len) && get_string(len, result) && check_string_utf8(result, context);
} }
default: default:
@@ -3415,12 +3374,12 @@ class binary_reader
if (at_end) if (at_end)
{ {
bool value_complete = false; if (JSON_HEDLEY_UNLIKELY(!leave_container()))
if (JSON_HEDLEY_UNLIKELY(!leave_container_and_check(value_complete)))
{ {
return false; return false;
} }
if (value_complete) // the value begun here is complete once its container is
if (container_stack.empty())
{ {
return true; return true;
} }
@@ -3429,7 +3388,8 @@ class binary_reader
if (top.is_object) if (top.is_object)
{ {
if (JSON_HEDLEY_UNLIKELY(!read_object_key<&binary_reader::get_bon8_key>(key))) key.clear();
if (JSON_HEDLEY_UNLIKELY(!get_bon8_key(key) || !sax->key(key)))
{ {
return false; return false;
} }
@@ -4169,22 +4129,6 @@ class binary_reader
return true; return true;
} }
/*!
@brief read a length of type @a NumberType, then a string of that many bytes
@tparam NumberType the type of the length field
@param[out] result string the bytes are appended to
@param[in] context further context information (for diagnostics)
@return whether reading and validating the string succeeded
*/
template<typename NumberType>
bool get_string_with_length(string_t& result, const char* context)
{
NumberType len{};
return get_number(len) && get_string(len, result) && check_string_utf8(result, context);
}
/*! /*!
@brief create a byte array by reading bytes from the input @brief create a byte array by reading bytes from the input
@@ -4394,7 +4338,37 @@ class binary_reader
const std::string& context) const const std::string& context) const
{ {
std::string error_msg = "syntax error while parsing "; std::string error_msg = "syntax error while parsing ";
error_msg += input_format_name(input_format);
switch (input_format)
{
case input_format_t::cbor:
error_msg += "CBOR";
break;
case input_format_t::msgpack:
error_msg += "MessagePack";
break;
case input_format_t::ubjson:
error_msg += "UBJSON";
break;
case input_format_t::bson:
error_msg += "BSON";
break;
case input_format_t::bjdata:
error_msg += "BJData";
break;
case input_format_t::bon8:
error_msg += "BON8";
break;
case input_format_t::json: // LCOV_EXCL_LINE
default: // LCOV_EXCL_LINE
JSON_ASSERT(false); // NOLINT(cert-dcl03-c,hicpp-static-assert,misc-static-assert) LCOV_EXCL_LINE
}
return concat(error_msg, ' ', context, ": ", detail); return concat(error_msg, ' ', context, ": ", detail);
} }
include/nlohmann/detail/input/input_adapters.hpp
+3 -33
View File
@@ -67,36 +67,6 @@ namespace detail
/// the supported input formats /// the supported input formats
enum class input_format_t { json, cbor, msgpack, ubjson, bson, bjdata, bon8 }; enum class input_format_t { json, cbor, msgpack, ubjson, bson, bjdata, bon8 };
/// @return the name of a binary input format, as used in exception messages
inline const char* input_format_name(const input_format_t format) noexcept
{
switch (format)
{
case input_format_t::cbor:
return "CBOR";
case input_format_t::msgpack:
return "MessagePack";
case input_format_t::ubjson:
return "UBJSON";
case input_format_t::bson:
return "BSON";
case input_format_t::bjdata:
return "BJData";
case input_format_t::bon8:
return "BON8";
case input_format_t::json: // LCOV_EXCL_LINE
default: // LCOV_EXCL_LINE
JSON_ASSERT(false); // NOLINT(cert-dcl03-c,hicpp-static-assert,misc-static-assert) LCOV_EXCL_LINE
return ""; // LCOV_EXCL_LINE
}
}
/// @return the message of the parse_error thrown or reported for an empty input
inline const char* empty_input_message() noexcept
{
return "attempting to parse an empty input; check that your input string or stream contains the expected JSON";
}
//////////////////// ////////////////////
// input adapters // // input adapters //
//////////////////// ////////////////////
@@ -1298,7 +1268,7 @@ inline file_input_adapter input_adapter(std::FILE* file)
{ {
if (file == nullptr) if (file == nullptr)
{ {
JSON_THROW(parse_error::create(exception_id::syntax_error, 0, empty_input_message(), nullptr)); JSON_THROW(parse_error::create(exception_id::syntax_error, 0, "attempting to parse an empty input; check that your input string or stream contains the expected JSON", nullptr));
} }
return file_input_adapter(file); return file_input_adapter(file);
} }
@@ -1307,7 +1277,7 @@ inline input_stream_adapter input_adapter(std::istream& stream)
{ {
if (stream.rdbuf() == nullptr) if (stream.rdbuf() == nullptr)
{ {
JSON_THROW(parse_error::create(exception_id::syntax_error, 0, empty_input_message(), nullptr)); JSON_THROW(parse_error::create(exception_id::syntax_error, 0, "attempting to parse an empty input; check that your input string or stream contains the expected JSON", nullptr));
} }
return input_stream_adapter(stream); return input_stream_adapter(stream);
} }
@@ -1332,7 +1302,7 @@ contiguous_bytes_input_adapter input_adapter(CharT b)
{ {
if (b == nullptr) if (b == nullptr)
{ {
JSON_THROW(parse_error::create(exception_id::syntax_error, 0, empty_input_message(), nullptr)); JSON_THROW(parse_error::create(exception_id::syntax_error, 0, "attempting to parse an empty input; check that your input string or stream contains the expected JSON", nullptr));
} }
auto length = std::strlen(reinterpret_cast<const char*>(b)); auto length = std::strlen(reinterpret_cast<const char*>(b));
const auto* ptr = reinterpret_cast<const char*>(b); const auto* ptr = reinterpret_cast<const char*>(b);
include/nlohmann/detail/input/json_sax.hpp
+1090 -44
View File
File diff suppressed because it is too large. Load diff
include/nlohmann/detail/input/lexer.hpp
+166 -36
View File
@@ -14,6 +14,7 @@
#include <cstdio> // snprintf #include <cstdio> // snprintf
#include <cstring> // memcmp #include <cstring> // memcmp
#include <initializer_list> // initializer_list #include <initializer_list> // initializer_list
#include <limits> // numeric_limits
#include <string> // char_traits, string #include <string> // char_traits, string
#include <utility> // move #include <utility> // move
#include <vector> // vector #include <vector> // vector
@@ -656,74 +657,194 @@ class lexer : public lexer_base<BasicJsonType>
// invalid control characters // invalid control characters
case 0x00: case 0x00:
{
error_message = "invalid string: control character U+0000 (NUL) must be escaped to \\u0000";
return token_type::parse_error;
}
case 0x01: case 0x01:
{
error_message = "invalid string: control character U+0001 (SOH) must be escaped to \\u0001";
return token_type::parse_error;
}
case 0x02: case 0x02:
{
error_message = "invalid string: control character U+0002 (STX) must be escaped to \\u0002";
return token_type::parse_error;
}
case 0x03: case 0x03:
{
error_message = "invalid string: control character U+0003 (ETX) must be escaped to \\u0003";
return token_type::parse_error;
}
case 0x04: case 0x04:
{
error_message = "invalid string: control character U+0004 (EOT) must be escaped to \\u0004";
return token_type::parse_error;
}
case 0x05: case 0x05:
{
error_message = "invalid string: control character U+0005 (ENQ) must be escaped to \\u0005";
return token_type::parse_error;
}
case 0x06: case 0x06:
{
error_message = "invalid string: control character U+0006 (ACK) must be escaped to \\u0006";
return token_type::parse_error;
}
case 0x07: case 0x07:
{
error_message = "invalid string: control character U+0007 (BEL) must be escaped to \\u0007";
return token_type::parse_error;
}
case 0x08: case 0x08:
{
error_message = "invalid string: control character U+0008 (BS) must be escaped to \\u0008 or \\b";
return token_type::parse_error;
}
case 0x09: case 0x09:
{
error_message = "invalid string: control character U+0009 (HT) must be escaped to \\u0009 or \\t";
return token_type::parse_error;
}
case 0x0A: case 0x0A:
{
error_message = "invalid string: control character U+000A (LF) must be escaped to \\u000A or \\n";
return token_type::parse_error;
}
case 0x0B: case 0x0B:
{
error_message = "invalid string: control character U+000B (VT) must be escaped to \\u000B";
return token_type::parse_error;
}
case 0x0C: case 0x0C:
{
error_message = "invalid string: control character U+000C (FF) must be escaped to \\u000C or \\f";
return token_type::parse_error;
}
case 0x0D: case 0x0D:
{
error_message = "invalid string: control character U+000D (CR) must be escaped to \\u000D or \\r";
return token_type::parse_error;
}
case 0x0E: case 0x0E:
{
error_message = "invalid string: control character U+000E (SO) must be escaped to \\u000E";
return token_type::parse_error;
}
case 0x0F: case 0x0F:
{
error_message = "invalid string: control character U+000F (SI) must be escaped to \\u000F";
return token_type::parse_error;
}
case 0x10: case 0x10:
{
error_message = "invalid string: control character U+0010 (DLE) must be escaped to \\u0010";
return token_type::parse_error;
}
case 0x11: case 0x11:
{
error_message = "invalid string: control character U+0011 (DC1) must be escaped to \\u0011";
return token_type::parse_error;
}
case 0x12: case 0x12:
{
error_message = "invalid string: control character U+0012 (DC2) must be escaped to \\u0012";
return token_type::parse_error;
}
case 0x13: case 0x13:
{
error_message = "invalid string: control character U+0013 (DC3) must be escaped to \\u0013";
return token_type::parse_error;
}
case 0x14: case 0x14:
{
error_message = "invalid string: control character U+0014 (DC4) must be escaped to \\u0014";
return token_type::parse_error;
}
case 0x15: case 0x15:
{
error_message = "invalid string: control character U+0015 (NAK) must be escaped to \\u0015";
return token_type::parse_error;
}
case 0x16: case 0x16:
{
error_message = "invalid string: control character U+0016 (SYN) must be escaped to \\u0016";
return token_type::parse_error;
}
case 0x17: case 0x17:
{
error_message = "invalid string: control character U+0017 (ETB) must be escaped to \\u0017";
return token_type::parse_error;
}
case 0x18: case 0x18:
{
error_message = "invalid string: control character U+0018 (CAN) must be escaped to \\u0018";
return token_type::parse_error;
}
case 0x19: case 0x19:
{
error_message = "invalid string: control character U+0019 (EM) must be escaped to \\u0019";
return token_type::parse_error;
}
case 0x1A: case 0x1A:
{
error_message = "invalid string: control character U+001A (SUB) must be escaped to \\u001A";
return token_type::parse_error;
}
case 0x1B: case 0x1B:
{
error_message = "invalid string: control character U+001B (ESC) must be escaped to \\u001B";
return token_type::parse_error;
}
case 0x1C: case 0x1C:
{
error_message = "invalid string: control character U+001C (FS) must be escaped to \\u001C";
return token_type::parse_error;
}
case 0x1D: case 0x1D:
{
error_message = "invalid string: control character U+001D (GS) must be escaped to \\u001D";
return token_type::parse_error;
}
case 0x1E: case 0x1E:
{
error_message = "invalid string: control character U+001E (RS) must be escaped to \\u001E";
return token_type::parse_error;
}
case 0x1F: case 0x1F:
{ {
static const std::array<const char*, 32> control_character_messages = {{ error_message = "invalid string: control character U+001F (US) must be escaped to \\u001F";
"invalid string: control character U+0000 (NUL) must be escaped to \\u0000",
"invalid string: control character U+0001 (SOH) must be escaped to \\u0001",
"invalid string: control character U+0002 (STX) must be escaped to \\u0002",
"invalid string: control character U+0003 (ETX) must be escaped to \\u0003",
"invalid string: control character U+0004 (EOT) must be escaped to \\u0004",
"invalid string: control character U+0005 (ENQ) must be escaped to \\u0005",
"invalid string: control character U+0006 (ACK) must be escaped to \\u0006",
"invalid string: control character U+0007 (BEL) must be escaped to \\u0007",
"invalid string: control character U+0008 (BS) must be escaped to \\u0008 or \\b",
"invalid string: control character U+0009 (HT) must be escaped to \\u0009 or \\t",
"invalid string: control character U+000A (LF) must be escaped to \\u000A or \\n",
"invalid string: control character U+000B (VT) must be escaped to \\u000B",
"invalid string: control character U+000C (FF) must be escaped to \\u000C or \\f",
"invalid string: control character U+000D (CR) must be escaped to \\u000D or \\r",
"invalid string: control character U+000E (SO) must be escaped to \\u000E",
"invalid string: control character U+000F (SI) must be escaped to \\u000F",
"invalid string: control character U+0010 (DLE) must be escaped to \\u0010",
"invalid string: control character U+0011 (DC1) must be escaped to \\u0011",
"invalid string: control character U+0012 (DC2) must be escaped to \\u0012",
"invalid string: control character U+0013 (DC3) must be escaped to \\u0013",
"invalid string: control character U+0014 (DC4) must be escaped to \\u0014",
"invalid string: control character U+0015 (NAK) must be escaped to \\u0015",
"invalid string: control character U+0016 (SYN) must be escaped to \\u0016",
"invalid string: control character U+0017 (ETB) must be escaped to \\u0017",
"invalid string: control character U+0018 (CAN) must be escaped to \\u0018",
"invalid string: control character U+0019 (EM) must be escaped to \\u0019",
"invalid string: control character U+001A (SUB) must be escaped to \\u001A",
"invalid string: control character U+001B (ESC) must be escaped to \\u001B",
"invalid string: control character U+001C (FS) must be escaped to \\u001C",
"invalid string: control character U+001D (GS) must be escaped to \\u001D",
"invalid string: control character U+001E (RS) must be escaped to \\u001E",
"invalid string: control character U+001F (US) must be escaped to \\u001F"
}
};
error_message = control_character_messages[static_cast<std::size_t>(current)];
return token_type::parse_error; return token_type::parse_error;
} }
@@ -1696,6 +1817,15 @@ scan_number_done:
/// significand (see scan_number_bulk_contiguous()) /// significand (see scan_number_bulk_contiguous())
bool convert_accumulated_float(const float_significand& s, std::true_type /*native*/) noexcept bool convert_accumulated_float(const float_significand& s, std::true_type /*native*/) noexcept
{ {
// accept() only needs to know that the value is finite (see the parser):
// w < 10^19, so a value below 10^max_exponent10 cannot overflow, and a
// value that underflows is a finite number as well. Only a token that
// may overflow has to be converted to find out.
if (JSON_HEDLEY_UNLIKELY(discard_number_values && s.exponent + 19 <= std::numeric_limits<number_float_t>::max_exponent10))
{
value_float = 0;
return true;
}
value_float = decimal_to_float<number_float_t>(s); value_float = decimal_to_float<number_float_t>(s);
return true; return true;
} }
include/nlohmann/detail/input/parser.hpp
+42 -42
View File
@@ -56,8 +56,8 @@ using parser_callback_t =
template<typename SAX> template<typename SAX>
struct sax_ignores_number_text : std::false_type {}; struct sax_ignores_number_text : std::false_type {};
template<typename BasicJsonType, typename InputAdapterType> template<typename BasicJsonType, typename InputAdapterType, bool Collect>
struct sax_ignores_number_text<json_sax_dom_parser<BasicJsonType, InputAdapterType>> : std::true_type {}; struct sax_ignores_number_text<json_sax_dom_parser<BasicJsonType, InputAdapterType, Collect>> : std::true_type {};
template<typename BasicJsonType, typename InputAdapterType> template<typename BasicJsonType, typename InputAdapterType>
struct sax_ignores_number_text<json_sax_dom_callback_parser<BasicJsonType, InputAdapterType>> : std::true_type {}; struct sax_ignores_number_text<json_sax_dom_callback_parser<BasicJsonType, InputAdapterType>> : std::true_type {};
@@ -131,7 +131,7 @@ class parser
} }
else else
{ {
json_sax_dom_parser<BasicJsonType, InputAdapterType> sdp(result, allow_exceptions, &m_lexer); json_sax_dom_parser<BasicJsonType, InputAdapterType, true> sdp(result, allow_exceptions, &m_lexer);
// in case of an error, return a discarded value // in case of an error, return a discarded value
if (!parse_dom(sdp, strict)) if (!parse_dom(sdp, strict))
@@ -165,10 +165,23 @@ class parser
if (result) if (result)
{ {
return finish_input(*sax, strict); if (strict)
{
// strict mode: next byte must be EOF
if (get_token() != token_type::end_of_input)
{
return syntax_error(*sax, exception_message(token_type::end_of_input, "value"));
}
}
else
{
// the caller keeps using the input: position it right after
// the value by leaving the character that terminated it
m_lexer.release_lookahead();
}
} }
return false; return result;
} }
private: private:
@@ -189,39 +202,6 @@ class parser
return m_lexer.get_string(); return m_lexer.get_string();
} }
/*!
@brief apply the end-of-input handling after a value was parsed
In strict mode, the next token must be the end of the input; otherwise
parse_error.101 is reported through @a sax. In non-strict mode, the
lookahead is released so the caller can keep reading the input right after
the parsed value.
@param[in,out] sax the SAX parser to report an error to
@param[in] strict whether to expect the last token to be EOF
@return true, or what @a sax returned for the error
*/
template<typename SAX>
bool finish_input(SAX& sax, const bool strict)
{
if (strict)
{
// strict mode: next byte must be EOF
if (get_token() != token_type::end_of_input)
{
return syntax_error(sax, exception_message(token_type::end_of_input, "value"));
}
}
else
{
// the caller keeps using the input: position it right after
// the value by leaving the character that terminated it
m_lexer.release_lookahead();
}
return true;
}
/*! /*!
@brief run a DOM SAX parser to completion and position the lexer @brief run a DOM SAX parser to completion and position the lexer
@@ -241,7 +221,20 @@ class parser
{ {
sax_parse_internal(&sdp); sax_parse_internal(&sdp);
static_cast<void>(finish_input(sdp, strict)); if (strict)
{
// in strict mode, input must be completely read
if (get_token() != token_type::end_of_input)
{
syntax_error(sdp, exception_message(token_type::end_of_input, "value"));
}
}
else
{
// the caller keeps using the input: position it right after
// the value by leaving the character that terminated it
m_lexer.release_lookahead();
}
return !sdp.is_errored(); return !sdp.is_errored();
} }
@@ -250,12 +243,19 @@ class parser
JSON_HEDLEY_NON_NULL(2) JSON_HEDLEY_NON_NULL(2)
bool sax_parse_internal(SAX* sax) bool sax_parse_internal(SAX* sax)
{ {
// value to avoid a goto (see comment where set to true)
bool skip_to_state_evaluation = false;
// stack to remember the hierarchy of structured values we are parsing // stack to remember the hierarchy of structured values we are parsing
// true = array; false = object (bytes, not std::vector<bool>, whose // true = array; false = object (bytes, not std::vector<bool>, whose
// push_back() is a call with GCC) // push_back() is a call with GCC)
std::vector<std::uint8_t> states; std::vector<std::uint8_t> states;
// value to avoid a goto (see comment where set to true) // a document that starts with a container pushes onto the stack at
bool skip_to_state_evaluation = false; // once: allocate it in one step instead of growing it from 1 byte
if (last_token == token_type::begin_object || last_token == token_type::begin_array)
{
states.reserve(32);
}
while (true) while (true)
{ {
@@ -411,7 +411,7 @@ class parser
{ {
if (JSON_HEDLEY_UNLIKELY(m_lexer.get_chars_read_total() == 1)) if (JSON_HEDLEY_UNLIKELY(m_lexer.get_chars_read_total() == 1))
{ {
return syntax_error(*sax, empty_input_message()); return syntax_error(*sax, "attempting to parse an empty input; check that your input string or stream contains the expected JSON");
} }
return syntax_error(*sax, exception_message(token_type::literal_or_value, "value")); return syntax_error(*sax, exception_message(token_type::literal_or_value, "value"));
include/nlohmann/detail/input/string_scan.hpp
+1 -1
View File
@@ -96,7 +96,7 @@ JSON_HEDLEY_ALWAYS_INLINE std::size_t find_string_special(const unsigned char* d
// printable ASCII except '"' and '\\' - is emitted unchanged. Note this differs // printable ASCII except '"' and '\\' - is emitted unchanged. Note this differs
// from is_string_special() only in that 0x7F is also a stop (it is escaped as // from is_string_special() only in that 0x7F is also a stop (it is escaped as
// \u007f under ensure_ascii). // \u007f under ensure_ascii).
constexpr bool is_ascii_copyable(unsigned char c) noexcept inline bool is_ascii_copyable(unsigned char c) noexcept
{ {
return c >= 0x20u && c < 0x7Fu && c != '"' && c != '\\'; return c >= 0x20u && c < 0x7Fu && c != '"' && c != '\\';
} }
include/nlohmann/detail/iterators/iter_impl.hpp
+113 -62
View File
@@ -108,18 +108,33 @@ class iter_impl // NOLINT(cppcoreguidelines-special-member-functions,hicpp-speci
{ {
JSON_ASSERT(m_object != nullptr); JSON_ASSERT(m_object != nullptr);
if (m_object->is_object()) switch (m_object->m_data.m_type)
{ {
m_it.object_iterator = typename object_t::iterator(); case value_t::object:
} {
else if (m_object->is_array()) m_it.object_iterator = typename object_t::iterator();
{ break;
m_it.array_iterator = typename array_t::iterator(); }
}
else case value_t::array:
{ {
// primitive (is_primitive(), including discarded) m_it.array_iterator = typename array_t::iterator();
m_it.primitive_iterator = primitive_iterator_t(); break;
}
case value_t::null:
case value_t::string:
case value_t::boolean:
case value_t::number_integer:
case value_t::number_unsigned:
case value_t::number_float:
case value_t::binary:
case value_t::discarded:
default:
{
m_it.primitive_iterator = primitive_iterator_t();
break;
}
} }
} }
@@ -234,28 +249,33 @@ class iter_impl // NOLINT(cppcoreguidelines-special-member-functions,hicpp-speci
{ {
JSON_ASSERT(m_object != nullptr); JSON_ASSERT(m_object != nullptr);
if (m_object->is_object()) switch (m_object->m_data.m_type)
{ {
m_it.object_iterator = m_object->m_data.m_value.object->end(); case value_t::object:
} {
else if (m_object->is_array()) m_it.object_iterator = m_object->m_data.m_value.object->end();
{ break;
m_it.array_iterator = m_object->m_data.m_value.array->end(); }
}
else
{
// primitive (is_primitive(), including discarded)
m_it.primitive_iterator.set_end();
}
}
/// @throw invalid_iterator.212 if @a other points to another container case value_t::array:
template<typename IterImpl> {
JSON_HEDLEY_ALWAYS_INLINE void check_same_container(const IterImpl& other) const m_it.array_iterator = m_object->m_data.m_value.array->end();
{ break;
if (JSON_HEDLEY_UNLIKELY(m_object != other.m_object)) }
{
throw_literal<invalid_iterator>(exception_id::iterators_compare_different_values, "cannot compare iterators of different containers", m_object); case value_t::null:
case value_t::string:
case value_t::boolean:
case value_t::number_integer:
case value_t::number_unsigned:
case value_t::number_float:
case value_t::binary:
case value_t::discarded:
default:
{
m_it.primitive_iterator.set_end();
break;
}
} }
} }
@@ -404,7 +424,10 @@ class iter_impl // NOLINT(cppcoreguidelines-special-member-functions,hicpp-speci
bool operator==(const IterImpl& other) const bool operator==(const IterImpl& other) const
{ {
// if objects are not the same, the comparison is undefined // if objects are not the same, the comparison is undefined
check_same_container(other); if (JSON_HEDLEY_UNLIKELY(m_object != other.m_object))
{
throw_literal<invalid_iterator>(exception_id::iterators_compare_different_values, "cannot compare iterators of different containers", m_object);
}
// value-initialized forward iterators can be compared, and must compare equal to other value-initialized iterators of the same type #4493 // value-initialized forward iterators can be compared, and must compare equal to other value-initialized iterators of the same type #4493
if (m_object == nullptr) if (m_object == nullptr)
@@ -443,7 +466,10 @@ class iter_impl // NOLINT(cppcoreguidelines-special-member-functions,hicpp-speci
bool operator<(const iter_impl& other) const bool operator<(const iter_impl& other) const
{ {
// if objects are not the same, the comparison is undefined // if objects are not the same, the comparison is undefined
check_same_container(other); if (JSON_HEDLEY_UNLIKELY(m_object != other.m_object))
{
throw_literal<invalid_iterator>(exception_id::iterators_compare_different_values, "cannot compare iterators of different containers", m_object);
}
// value-initialized forward iterators can be compared, and must compare equal to other value-initialized iterators of the same type #4493 // value-initialized forward iterators can be compared, and must compare equal to other value-initialized iterators of the same type #4493
if (m_object == nullptr) if (m_object == nullptr)
@@ -452,18 +478,25 @@ class iter_impl // NOLINT(cppcoreguidelines-special-member-functions,hicpp-speci
return false; return false;
} }
if (m_object->is_object()) switch (m_object->m_data.m_type)
{ {
throw_literal<invalid_iterator>(exception_id::iterator_order_on_object, "cannot compare order of object iterators", m_object); case value_t::object:
} throw_literal<invalid_iterator>(exception_id::iterator_order_on_object, "cannot compare order of object iterators", m_object);
if (m_object->is_array()) case value_t::array:
{ return (m_it.array_iterator < other.m_it.array_iterator);
return (m_it.array_iterator < other.m_it.array_iterator);
}
// primitive (is_primitive(), including discarded) case value_t::null:
return (m_it.primitive_iterator < other.m_it.primitive_iterator); case value_t::string:
case value_t::boolean:
case value_t::number_integer:
case value_t::number_unsigned:
case value_t::number_float:
case value_t::binary:
case value_t::discarded:
default:
return (m_it.primitive_iterator < other.m_it.primitive_iterator);
}
} }
/*! /*!
@@ -501,19 +534,30 @@ class iter_impl // NOLINT(cppcoreguidelines-special-member-functions,hicpp-speci
{ {
JSON_ASSERT(m_object != nullptr); JSON_ASSERT(m_object != nullptr);
if (m_object->is_object()) switch (m_object->m_data.m_type)
{ {
throw_literal<invalid_iterator>(exception_id::iterator_arithmetic_on_object, "cannot use offsets with object iterators", m_object); case value_t::object:
} throw_literal<invalid_iterator>(exception_id::iterator_arithmetic_on_object, "cannot use offsets with object iterators", m_object);
if (m_object->is_array()) case value_t::array:
{ {
std::advance(m_it.array_iterator, i); std::advance(m_it.array_iterator, i);
} break;
else }
{
// primitive (is_primitive(), including discarded) case value_t::null:
m_it.primitive_iterator += i; case value_t::string:
case value_t::boolean:
case value_t::number_integer:
case value_t::number_unsigned:
case value_t::number_float:
case value_t::binary:
case value_t::discarded:
default:
{
m_it.primitive_iterator += i;
break;
}
} }
return *this; return *this;
@@ -569,18 +613,25 @@ class iter_impl // NOLINT(cppcoreguidelines-special-member-functions,hicpp-speci
{ {
JSON_ASSERT(m_object != nullptr); JSON_ASSERT(m_object != nullptr);
if (m_object->is_object()) switch (m_object->m_data.m_type)
{ {
throw_literal<invalid_iterator>(exception_id::iterator_arithmetic_on_object, "cannot use offsets with object iterators", m_object); case value_t::object:
} throw_literal<invalid_iterator>(exception_id::iterator_arithmetic_on_object, "cannot use offsets with object iterators", m_object);
if (m_object->is_array()) case value_t::array:
{ return m_it.array_iterator - other.m_it.array_iterator;
return m_it.array_iterator - other.m_it.array_iterator;
}
// primitive (is_primitive(), including discarded) case value_t::null:
return m_it.primitive_iterator - other.m_it.primitive_iterator; case value_t::string:
case value_t::boolean:
case value_t::number_integer:
case value_t::number_unsigned:
case value_t::number_float:
case value_t::binary:
case value_t::discarded:
default:
return m_it.primitive_iterator - other.m_it.primitive_iterator;
}
} }
/*! /*!
include/nlohmann/detail/iterators/iteration_proxy.hpp
+22 -12
View File
@@ -109,25 +109,35 @@ template<typename IteratorType> class iteration_proxy_value
{ {
JSON_ASSERT(anchor.m_object != nullptr); JSON_ASSERT(anchor.m_object != nullptr);
if (anchor.m_object->is_array()) switch (anchor.m_object->type())
{ {
// use integer array index as key // use integer array index as key
if (array_index != array_index_last) case value_t::array:
{ {
int_to_string( array_index_str, array_index ); if (array_index != array_index_last)
array_index_last = array_index; {
int_to_string( array_index_str, array_index );
array_index_last = array_index;
}
return array_index_str;
} }
return array_index_str;
}
if (anchor.m_object->is_object())
{
// use key from the object // use key from the object
return anchor.key(); case value_t::object:
} return anchor.key();
// use an empty key for all primitive types // use an empty key for all primitive types
return empty_str; case value_t::null:
case value_t::string:
case value_t::boolean:
case value_t::number_integer:
case value_t::number_unsigned:
case value_t::number_float:
case value_t::binary:
case value_t::discarded:
default:
return empty_str;
}
} }
/// return value of the iterator /// return value of the iterator
include/nlohmann/detail/json_pointer.hpp
+384 -319
View File
@@ -38,102 +38,6 @@ NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail namespace detail
{ {
struct json_pointer_access; struct json_pointer_access;
/*!
@brief result of parse_array_index()
json_pointer::array_index() maps each value to the corresponding
parse_error/out_of_range exception; json_pointer::contains() and
get_checked_or_null(), which must not throw for an out-of-range or
unrepresentable index, and the json_view pointer resolution switch on it
directly instead.
*/
enum class array_index_status
{
ok, ///< @a s is a valid, representable array index
leading_zero, ///< @a s begins with '0' but has more than one character
not_a_number, ///< @a s is neither empty nor "-" and does not begin with a digit
unresolved, ///< @a s could not be converted to an integer
exceeds_size_type ///< @a s converts to an integer that exceeds SizeType
};
/*!
@brief convert a JSON Pointer reference token into an array index (RFC 6901, Sect. 4)
@param[in] s reference token to be converted into an array index; its data()
must be NUL-terminated
@param[out] idx the integer representation of @a s if array_index_status::ok
is returned; left unchanged otherwise
@return whether @a s is a valid array index, and if not, why
@note this function never throws; json_pointer::array_index(), the callers that
must not throw, and the json_view pointer resolution build on it instead
of each re-implementing the RFC 6901 digit rules and the range check
@note force-inlined: as a free function template it is no longer inlined into
the pointer resolution, which makes JsonPointerAt up to 17% slower (#5891)
*/
template<typename SizeType, typename StringType>
JSON_HEDLEY_ALWAYS_INLINE array_index_status parse_array_index(const StringType& s, SizeType& idx) noexcept
{
// error condition (cf. RFC 6901, Sect. 4)
if (JSON_HEDLEY_UNLIKELY(s.size() > 1 && s[0] == '0'))
{
return array_index_status::leading_zero;
}
// error condition (cf. RFC 6901, Sect. 4); this also covers single-
// character tokens, so "/x" and "/xy" fail alike; "-" and the empty
// token are left to the conversion below and are reported as unresolved
if (JSON_HEDLEY_UNLIKELY(!s.empty() && !(s.size() == 1 && s[0] == '-') && !(s[0] >= '0' && s[0] <= '9')))
{
return array_index_status::not_a_number;
}
// fast path: up to 18 decimal digits cannot overflow unsigned long long or
// SizeType (9 digits for a 32-bit SizeType), so strtoull and errno are not needed
constexpr std::size_t max_fast_digits = sizeof(SizeType) >= 8 ? 18 : 9;
if (!s.empty() && s.size() <= max_fast_digits)
{
SizeType acc = 0;
for (std::size_t i = 0; i < s.size(); ++i)
{
const char c = s[i];
if (JSON_HEDLEY_UNLIKELY(c < '0' || c > '9'))
{
break; // not a plain number: let strtoull decide
}
acc = static_cast<SizeType>((acc * 10) + static_cast<SizeType>(c - '0'));
if (JSON_HEDLEY_UNLIKELY(i == s.size() - 1))
{
idx = acc;
return array_index_status::ok;
}
}
}
const char* p = s.data();
char* p_end = nullptr; // NOLINT(misc-const-correctness)
errno = 0; // strtoull doesn't reset errno
const unsigned long long res = std::strtoull(p, &p_end, 10); // NOLINT(runtime/int)
if (p == p_end // invalid input or empty string
|| errno == ERANGE // out of range
|| JSON_HEDLEY_UNLIKELY(static_cast<std::size_t>(p_end - p) != s.size())) // incomplete read
{
return array_index_status::unresolved;
}
// the index does not fit into SizeType; on 64-bit platforms this is
// only SIZE_MAX itself (see #2203 and #5395)
if (res >= static_cast<unsigned long long>((std::numeric_limits<SizeType>::max)())) // NOLINT(runtime/int)
{
return array_index_status::exceeds_size_type;
}
idx = static_cast<SizeType>(res);
return array_index_status::ok;
}
} // namespace detail } // namespace detail
/// @brief JSON Pointer defines a string syntax for identifying a specific value within a JSON document /// @brief JSON Pointer defines a string syntax for identifying a specific value within a JSON document
@@ -398,28 +302,6 @@ class json_pointer
JSON_THROW(detail::out_of_range::create(detail::exception_id::patch_on_root, "JSON pointer has no parent", nullptr)); JSON_THROW(detail::out_of_range::create(detail::exception_id::patch_on_root, "JSON pointer has no parent", nullptr));
} }
/// whether the '~' at @a pos of @a s is not followed by '0' or '1'
JSON_HEDLEY_ALWAYS_INLINE static bool is_invalid_escape(const string_t& s, std::size_t pos)
{
return pos == s.size() - 1 || (s[pos + 1] != '0' && s[pos + 1] != '1');
}
/// @throw parse_error.107 always
JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE static void throw_missing_slash(const string_t& pointer)
{
static_cast<void>(pointer); // unused when JSON_NOEXCEPTION is defined
JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_missing_slash, 1, detail::concat("JSON pointer must be empty or begin with '/' - was: '", pointer, "'"), nullptr));
}
/// @throw parse_error.107 if a nonempty @a pointer does not begin with '/'
JSON_HEDLEY_ALWAYS_INLINE static void check_begins_with_slash(const string_t& pointer)
{
if (JSON_HEDLEY_UNLIKELY(pointer[0] != '/'))
{
throw_missing_slash(pointer);
}
}
/// @throw out_of_range.404 always /// @throw out_of_range.404 always
template<typename BasicJsonContext> template<typename BasicJsonContext>
JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE static void throw_unresolved(const string_t& reference_token, BasicJsonContext context) JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE static void throw_unresolved(const string_t& reference_token, BasicJsonContext context)
@@ -429,7 +311,21 @@ class json_pointer
JSON_THROW(detail::out_of_range::create(detail::exception_id::pointer_unresolved, detail::concat("unresolved reference token '", reference_token, "'"), context)); JSON_THROW(detail::out_of_range::create(detail::exception_id::pointer_unresolved, detail::concat("unresolved reference token '", reference_token, "'"), context));
} }
using array_index_status = detail::array_index_status; /*!
@brief result of @ref parse_array_index
@ref array_index maps each value to the corresponding parse_error/out_of_range
exception; @ref contains and @ref get_checked_or_null, which must not throw for
an out-of-range or unrepresentable index, switch on it directly instead.
*/
enum class array_index_status
{
ok, ///< @a s is a valid, representable array index
leading_zero, ///< @a s begins with '0' but has more than one character
not_a_number, ///< @a s is neither empty nor "-" and does not begin with a digit
unresolved, ///< @a s could not be converted to an integer
exceeds_size_type ///< @a s converts to an integer that exceeds size_type
};
/// whether a reference token is "-" (a plain comparison with a string literal is not always inlined) /// whether a reference token is "-" (a plain comparison with a string literal is not always inlined)
static bool is_dash(const string_t& s) noexcept static bool is_dash(const string_t& s) noexcept
@@ -437,11 +333,78 @@ class json_pointer
return s.size() == 1 && s[0] == '-'; return s.size() == 1 && s[0] == '-';
} }
/// detail::parse_array_index() for the size_type of @a BasicJsonType /*!
@param[in] s reference token to be converted into an array index
@param[out] idx the integer representation of @a s if @ref array_index_status::ok
is returned; left unchanged otherwise
@return whether @a s is a valid array index, and if not, why
@note this function never throws; @ref array_index and the callers that must not
throw (@ref contains, @ref get_checked_or_null) build on it instead of each
re-implementing the RFC 6901 digit rules and the @a size_type range check
*/
template<typename BasicJsonType> template<typename BasicJsonType>
static array_index_status parse_array_index(const string_t& s, typename BasicJsonType::size_type& idx) noexcept static array_index_status parse_array_index(const string_t& s, typename BasicJsonType::size_type& idx) noexcept
{ {
return detail::parse_array_index(s, idx); using size_type = typename BasicJsonType::size_type;
// error condition (cf. RFC 6901, Sect. 4)
if (JSON_HEDLEY_UNLIKELY(s.size() > 1 && s[0] == '0'))
{
return array_index_status::leading_zero;
}
// error condition (cf. RFC 6901, Sect. 4); this also covers single-
// character tokens, so "/x" and "/xy" fail alike; "-" and the empty
// token are left to the conversion below and are reported as unresolved
if (JSON_HEDLEY_UNLIKELY(!s.empty() && !is_dash(s) && !(s[0] >= '0' && s[0] <= '9')))
{
return array_index_status::not_a_number;
}
// fast path: up to 18 decimal digits cannot overflow unsigned long long or
// size_type (9 digits for a 32-bit size_type), so strtoull and errno are not needed
constexpr std::size_t max_fast_digits = sizeof(size_type) >= 8 ? 18 : 9;
if (!s.empty() && s.size() <= max_fast_digits)
{
size_type acc = 0;
for (std::size_t i = 0; i < s.size(); ++i)
{
const char c = s[i];
if (JSON_HEDLEY_UNLIKELY(c < '0' || c > '9'))
{
break; // not a plain number: let strtoull decide
}
acc = static_cast<size_type>((acc * 10) + static_cast<size_type>(c - '0'));
if (JSON_HEDLEY_UNLIKELY(i == s.size() - 1))
{
idx = acc;
return array_index_status::ok;
}
}
}
const char* p = s.data();
char* p_end = nullptr; // NOLINT(misc-const-correctness)
errno = 0; // strtoull doesn't reset errno
const unsigned long long res = std::strtoull(p, &p_end, 10); // NOLINT(runtime/int)
if (p == p_end // invalid input or empty string
|| errno == ERANGE // out of range
|| JSON_HEDLEY_UNLIKELY(static_cast<std::size_t>(p_end - p) != s.size())) // incomplete read
{
return array_index_status::unresolved;
}
// the index does not fit into size_type; on 64-bit platforms this is
// only SIZE_MAX itself (see #2203 and #5395)
if (res >= static_cast<unsigned long long>((std::numeric_limits<size_type>::max)())) // NOLINT(runtime/int)
{
return array_index_status::exceeds_size_type;
}
idx = static_cast<size_type>(res);
return array_index_status::ok;
} }
/*! /*!
@@ -491,7 +454,10 @@ class json_pointer
return reference_tokens.empty(); return reference_tokens.empty();
} }
check_begins_with_slash(reference_string); if (JSON_HEDLEY_UNLIKELY(reference_string[0] != '/'))
{
JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_missing_slash, 1, detail::concat("JSON pointer must be empty or begin with '/' - was: '", reference_string, "'"), nullptr));
}
// the whole string is validated, whether or not it matches // the whole string is validated, whether or not it matches
const std::size_t size = reference_string.size(); const std::size_t size = reference_string.size();
@@ -499,7 +465,7 @@ class json_pointer
pos != string_t::npos; pos != string_t::npos;
pos = reference_string.find_first_of('~', pos + 1)) pos = reference_string.find_first_of('~', pos + 1))
{ {
if (JSON_HEDLEY_UNLIKELY(is_invalid_escape(reference_string, pos))) if (JSON_HEDLEY_UNLIKELY(pos == size - 1 || (reference_string[pos + 1] != '0' && reference_string[pos + 1] != '1')))
{ {
JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_invalid_escape, 0, "escape character '~' must be followed with '0' or '1'", nullptr)); JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_invalid_escape, 0, "escape character '~' must be followed with '0' or '1'", nullptr));
} }
@@ -777,7 +743,10 @@ class json_pointer
} }
const char* const data = key.data(); const char* const data = key.data();
check_begins_with_slash(key); if (JSON_HEDLEY_UNLIKELY(data[0] != '/'))
{
JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_missing_slash, 1, detail::concat("JSON pointer must be empty or begin with '/' - was: '", key, "'"), nullptr));
}
// The beginning of the key that it has in common with the previous // The beginning of the key that it has in common with the previous
// one, which was checked already, is skipped: the scan starts with // one, which was checked already, is skipped: the scan starts with
@@ -810,7 +779,7 @@ class json_pointer
else if (data[i] == '~') else if (data[i] == '~')
{ {
// ~ must be followed by 0 or 1 // ~ must be followed by 0 or 1
if (JSON_HEDLEY_UNLIKELY(is_invalid_escape(key, i))) if (JSON_HEDLEY_UNLIKELY(i + 1 == size || (data[i + 1] != '0' && data[i + 1] != '1')))
{ {
JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_invalid_escape, 0, "escape character '~' must be followed with '0' or '1'", nullptr)); JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_invalid_escape, 0, "escape character '~' must be followed with '0' or '1'", nullptr));
} }
@@ -909,46 +878,58 @@ class json_pointer
++end; ++end;
} }
if (ptr->is_object()) switch (ptr->type())
{ {
token.clear(); case detail::value_t::object:
token.append(begin, static_cast<std::size_t>(end - begin));
const auto it = ptr->m_data.m_value.object->find(token);
if (JSON_HEDLEY_UNLIKELY(it == ptr->m_data.m_value.object->end()))
{ {
return nullptr; token.clear();
} token.append(begin, static_cast<std::size_t>(end - begin));
ptr = &it->second; const auto it = ptr->m_data.m_value.object->find(token);
} if (JSON_HEDLEY_UNLIKELY(it == ptr->m_data.m_value.object->end()))
else if (ptr->is_array())
{
// an index of up to 18 digits without a leading zero fits into size_type
// (9 digits for a 32-bit size_type); everything else is left to json_pointer
constexpr std::size_t max_digits = sizeof(typename BasicJsonType::size_type) >= 8 ? 18 : 9;
const auto digits = static_cast<std::size_t>(end - begin);
if (JSON_HEDLEY_UNLIKELY(digits == 0 || digits > max_digits || (digits > 1 && *begin == '0')))
{
return nullptr;
}
typename BasicJsonType::size_type idx = 0;
for (const char* c = begin; c != end; ++c)
{
if (JSON_HEDLEY_UNLIKELY(*c < '0' || *c > '9'))
{ {
return nullptr; return nullptr;
} }
idx = static_cast<typename BasicJsonType::size_type>((idx * 10) + static_cast<typename BasicJsonType::size_type>(*c - '0')); ptr = &it->second;
break;
} }
if (JSON_HEDLEY_UNLIKELY(idx >= ptr->m_data.m_value.array->size()))
case detail::value_t::array:
{ {
return nullptr; // an index of up to 18 digits without a leading zero fits into size_type
// (9 digits for a 32-bit size_type); everything else is left to json_pointer
constexpr std::size_t max_digits = sizeof(typename BasicJsonType::size_type) >= 8 ? 18 : 9;
const auto digits = static_cast<std::size_t>(end - begin);
if (JSON_HEDLEY_UNLIKELY(digits == 0 || digits > max_digits || (digits > 1 && *begin == '0')))
{
return nullptr;
}
typename BasicJsonType::size_type idx = 0;
for (const char* c = begin; c != end; ++c)
{
if (JSON_HEDLEY_UNLIKELY(*c < '0' || *c > '9'))
{
return nullptr;
}
idx = static_cast<typename BasicJsonType::size_type>((idx * 10) + static_cast<typename BasicJsonType::size_type>(*c - '0'));
}
if (JSON_HEDLEY_UNLIKELY(idx >= ptr->m_data.m_value.array->size()))
{
return nullptr;
}
ptr = &(*ptr->m_data.m_value.array)[idx];
break;
} }
ptr = &(*ptr->m_data.m_value.array)[idx];
} case detail::value_t::null:
else case detail::value_t::string:
{ case detail::value_t::boolean:
// primitive value with a reference token left case detail::value_t::number_integer:
return nullptr; case detail::value_t::number_unsigned:
case detail::value_t::number_float:
case detail::value_t::binary:
case detail::value_t::discarded:
default:
return nullptr;
} }
pos = static_cast<std::size_t>(end - path.data()); pos = static_cast<std::size_t>(end - path.data());
steps.emplace_back(pos, ptr); steps.emplace_back(pos, ptr);
@@ -998,28 +979,40 @@ class json_pointer
: detail::value_t::object; : detail::value_t::object;
} }
if (ptr->is_object()) switch (ptr->type())
{ {
// use unchecked object access case detail::value_t::object:
ptr = &object_member(*ptr, reference_token);
}
else if (ptr->is_array())
{
if (reference_token == "-")
{ {
// explicitly treat "-" as index beyond the end // use unchecked object access
ptr = &ptr->operator[](ptr->m_data.m_value.array->size()); ptr = &object_member(*ptr, reference_token);
break;
} }
else
case detail::value_t::array:
{ {
// convert array index to number; unchecked access if (reference_token == "-")
ptr = &ptr->operator[](array_index<BasicJsonType>(reference_token)); {
// explicitly treat "-" as index beyond the end
ptr = &ptr->operator[](ptr->m_data.m_value.array->size());
}
else
{
// convert array index to number; unchecked access
ptr = &ptr->operator[](array_index<BasicJsonType>(reference_token));
}
break;
} }
}
else case detail::value_t::null:
{ case detail::value_t::string:
// primitive value with a reference token left case detail::value_t::boolean:
throw_unresolved(reference_token, ptr); case detail::value_t::number_integer:
case detail::value_t::number_unsigned:
case detail::value_t::number_float:
case detail::value_t::binary:
case detail::value_t::discarded:
default:
throw_unresolved(reference_token, ptr);
} }
} }
@@ -1037,39 +1030,51 @@ class json_pointer
{ {
for (const auto& reference_token : reference_tokens) for (const auto& reference_token : reference_tokens)
{ {
if (ptr->is_object()) switch (ptr->type())
{ {
const auto it = ptr->m_data.m_value.object->find(reference_token); case detail::value_t::object:
if (JSON_HEDLEY_UNLIKELY(it == ptr->m_data.m_value.object->end()))
{ {
// at() throws out_of_range.403 with the usual message const auto it = ptr->m_data.m_value.object->find(reference_token);
ptr->at(reference_token); if (JSON_HEDLEY_UNLIKELY(it == ptr->m_data.m_value.object->end()))
} {
ptr = &it->second; // at() throws out_of_range.403 with the usual message
} ptr->at(reference_token);
else if (ptr->is_array()) }
{ ptr = &it->second;
if (JSON_HEDLEY_UNLIKELY(is_dash(reference_token))) break;
{
// "-" always fails the range check
JSON_THROW(detail::out_of_range::create(detail::exception_id::pointer_past_the_end_index, detail::concat(
"array index '-' (", std::to_string(ptr->m_data.m_value.array->size()),
") is out of range"), ptr));
} }
const auto idx = array_index<BasicJsonType>(reference_token); case detail::value_t::array:
// Bounds check before access to avoid exception with JSON_NOEXCEPTION
if (JSON_HEDLEY_UNLIKELY(idx >= ptr->m_data.m_value.array->size()))
{ {
JSON_THROW(detail::out_of_range::create(detail::exception_id::array_index_out_of_range, detail::concat( if (JSON_HEDLEY_UNLIKELY(is_dash(reference_token)))
"array index ", std::to_string(idx), " is out of range"), ptr)); {
// "-" always fails the range check
JSON_THROW(detail::out_of_range::create(detail::exception_id::pointer_past_the_end_index, detail::concat(
"array index '-' (", std::to_string(ptr->m_data.m_value.array->size()),
") is out of range"), ptr));
}
const auto idx = array_index<BasicJsonType>(reference_token);
// Bounds check before access to avoid exception with JSON_NOEXCEPTION
if (JSON_HEDLEY_UNLIKELY(idx >= ptr->m_data.m_value.array->size()))
{
JSON_THROW(detail::out_of_range::create(detail::exception_id::array_index_out_of_range, detail::concat(
"array index ", std::to_string(idx), " is out of range"), ptr));
}
ptr = &(*ptr->m_data.m_value.array)[idx];
break;
} }
ptr = &(*ptr->m_data.m_value.array)[idx];
} case detail::value_t::null:
else case detail::value_t::string:
{ case detail::value_t::boolean:
// primitive value with a reference token left case detail::value_t::number_integer:
throw_unresolved(reference_token, ptr); case detail::value_t::number_unsigned:
case detail::value_t::number_float:
case detail::value_t::binary:
case detail::value_t::discarded:
default:
throw_unresolved(reference_token, ptr);
} }
} }
@@ -1098,32 +1103,44 @@ class json_pointer
{ {
for (const auto& reference_token : reference_tokens) for (const auto& reference_token : reference_tokens)
{ {
if (ptr->is_object()) switch (ptr->type())
{ {
// use unchecked object access; the const operator[] case detail::value_t::object:
// asserts that the key exists
const auto it = ptr->m_data.m_value.object->find(reference_token);
JSON_ASSERT(it != ptr->m_data.m_value.object->end());
ptr = &it->second;
}
else if (ptr->is_array())
{
if (JSON_HEDLEY_UNLIKELY(is_dash(reference_token)))
{ {
// "-" cannot be used for const access // use unchecked object access; the const operator[]
JSON_THROW(detail::out_of_range::create(detail::exception_id::pointer_past_the_end_index, detail::concat("array index '-' (", std::to_string(ptr->m_data.m_value.array->size()), ") is out of range"), ptr)); // asserts that the key exists
const auto it = ptr->m_data.m_value.object->find(reference_token);
JSON_ASSERT(it != ptr->m_data.m_value.object->end());
ptr = &it->second;
break;
} }
// use unchecked array access; the const operator[] case detail::value_t::array:
// asserts that the index exists {
const auto idx = array_index<BasicJsonType>(reference_token); if (JSON_HEDLEY_UNLIKELY(is_dash(reference_token)))
JSON_ASSERT(idx < ptr->m_data.m_value.array->size()); {
ptr = &(*ptr->m_data.m_value.array)[idx]; // "-" cannot be used for const access
} JSON_THROW(detail::out_of_range::create(detail::exception_id::pointer_past_the_end_index, detail::concat("array index '-' (", std::to_string(ptr->m_data.m_value.array->size()), ") is out of range"), ptr));
else }
{
// primitive value with a reference token left // use unchecked array access; the const operator[]
throw_unresolved(reference_token, ptr); // asserts that the index exists
const auto idx = array_index<BasicJsonType>(reference_token);
JSON_ASSERT(idx < ptr->m_data.m_value.array->size());
ptr = &(*ptr->m_data.m_value.array)[idx];
break;
}
case detail::value_t::null:
case detail::value_t::string:
case detail::value_t::boolean:
case detail::value_t::number_integer:
case detail::value_t::number_unsigned:
case detail::value_t::number_float:
case detail::value_t::binary:
case detail::value_t::discarded:
default:
throw_unresolved(reference_token, ptr);
} }
} }
@@ -1147,54 +1164,66 @@ class json_pointer
{ {
for (const auto& reference_token : reference_tokens) for (const auto& reference_token : reference_tokens)
{ {
if (ptr->is_object()) switch (ptr->type())
{ {
const auto it = ptr->m_data.m_value.object->find(reference_token); case detail::value_t::object:
if (JSON_HEDLEY_UNLIKELY(it == ptr->m_data.m_value.object->end()))
{ {
return nullptr; const auto it = ptr->m_data.m_value.object->find(reference_token);
} if (JSON_HEDLEY_UNLIKELY(it == ptr->m_data.m_value.object->end()))
ptr = &it->second; {
}
else if (ptr->is_array())
{
if (JSON_HEDLEY_UNLIKELY(is_dash(reference_token)))
{
// "-" always fails the range check
return nullptr;
}
// a malformed index throws parse_error.106/109; an
// index that is syntactically valid but cannot be
// represented (out_of_range.404/410) is treated like an
// out-of-range index below
typename BasicJsonType::size_type idx{};
switch (parse_array_index<BasicJsonType>(reference_token, idx))
{
// the branches differ in their messages, not after JSON_THROW's expansion
// NOLINTNEXTLINE(bugprone-branch-clone)
case array_index_status::leading_zero:
JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_index_leading_zero, 0, detail::concat("array index '", reference_token, "' must not begin with '0'"), nullptr));
case array_index_status::not_a_number:
JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_index_not_a_number, 0, detail::concat("array index '", reference_token, "' is not a number"), nullptr));
case array_index_status::unresolved:
case array_index_status::exceeds_size_type:
return nullptr; return nullptr;
case array_index_status::ok: }
default: ptr = &it->second;
break; break;
} }
if (JSON_HEDLEY_UNLIKELY(idx >= ptr->m_data.m_value.array->size())) case detail::value_t::array:
{ {
return nullptr; if (JSON_HEDLEY_UNLIKELY(is_dash(reference_token)))
{
// "-" always fails the range check
return nullptr;
}
// a malformed index throws parse_error.106/109; an
// index that is syntactically valid but cannot be
// represented (out_of_range.404/410) is treated like an
// out-of-range index below
typename BasicJsonType::size_type idx{};
switch (parse_array_index<BasicJsonType>(reference_token, idx))
{
// the branches differ in their messages, not after JSON_THROW's expansion
// NOLINTNEXTLINE(bugprone-branch-clone)
case array_index_status::leading_zero:
JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_index_leading_zero, 0, detail::concat("array index '", reference_token, "' must not begin with '0'"), nullptr));
case array_index_status::not_a_number:
JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_index_not_a_number, 0, detail::concat("array index '", reference_token, "' is not a number"), nullptr));
case array_index_status::unresolved:
case array_index_status::exceeds_size_type:
return nullptr;
case array_index_status::ok:
default:
break;
}
if (JSON_HEDLEY_UNLIKELY(idx >= ptr->m_data.m_value.array->size()))
{
return nullptr;
}
ptr = &(*ptr->m_data.m_value.array)[idx];
break;
} }
ptr = &(*ptr->m_data.m_value.array)[idx];
} case detail::value_t::null:
else case detail::value_t::string:
{ case detail::value_t::boolean:
// primitive value with a reference token left case detail::value_t::number_integer:
return nullptr; case detail::value_t::number_unsigned:
case detail::value_t::number_float:
case detail::value_t::binary:
case detail::value_t::discarded:
default:
return nullptr;
} }
} }
@@ -1210,48 +1239,63 @@ class json_pointer
{ {
for (const auto& reference_token : reference_tokens) for (const auto& reference_token : reference_tokens)
{ {
if (ptr->is_object()) switch (ptr->type())
{ {
const auto it = ptr->m_data.m_value.object->find(reference_token); case detail::value_t::object:
if (it == ptr->m_data.m_value.object->end())
{ {
// we did not find the key in the object const auto it = ptr->m_data.m_value.object->find(reference_token);
return false; if (it == ptr->m_data.m_value.object->end())
{
// we did not find the key in the object
return false;
}
ptr = &it->second;
break;
} }
ptr = &it->second; case detail::value_t::array:
}
else if (ptr->is_array())
{
if (JSON_HEDLEY_UNLIKELY(is_dash(reference_token)))
{ {
// "-" always fails the range check if (JSON_HEDLEY_UNLIKELY(is_dash(reference_token)))
return false; {
// "-" always fails the range check
return false;
}
// any parse failure (malformed index, or one that is syntactically
// valid but not representable as size_type) means the reference
// token cannot denote an existing array element -- contains() must
// not throw (see #5395), so it is treated as "not found"
typename BasicJsonType::size_type idx{};
if (JSON_HEDLEY_UNLIKELY(parse_array_index<BasicJsonType>(reference_token, idx) != array_index_status::ok))
{
return false;
}
if (idx >= ptr->m_data.m_value.array->size())
{
// index out of range
return false;
}
ptr = &(*ptr->m_data.m_value.array)[idx];
break;
} }
// any parse failure (malformed index, or one that is syntactically case detail::value_t::null:
// valid but not representable as size_type) means the reference case detail::value_t::string:
// token cannot denote an existing array element -- contains() must case detail::value_t::boolean:
// not throw (see #5395), so it is treated as "not found" case detail::value_t::number_integer:
typename BasicJsonType::size_type idx{}; case detail::value_t::number_unsigned:
if (JSON_HEDLEY_UNLIKELY(parse_array_index<BasicJsonType>(reference_token, idx) != array_index_status::ok)) case detail::value_t::number_float:
case detail::value_t::binary:
case detail::value_t::discarded:
default:
{ {
// we do not expect primitive values if there is still a
// reference token to process
return false; return false;
} }
if (idx >= ptr->m_data.m_value.array->size())
{
// index out of range
return false;
}
ptr = &(*ptr->m_data.m_value.array)[idx];
}
else
{
// we do not expect primitive values if there is still a
// reference token to process
return false;
} }
} }
@@ -1279,7 +1323,10 @@ class json_pointer
} }
// check if a nonempty reference string begins with slash // check if a nonempty reference string begins with slash
check_begins_with_slash(reference_string); if (JSON_HEDLEY_UNLIKELY(reference_string[0] != '/'))
{
JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_missing_slash, 1, detail::concat("JSON pointer must be empty or begin with '/' - was: '", reference_string, "'"), nullptr));
}
// every slash starts a reference token // every slash starts a reference token
const std::size_t size = reference_string.size(); const std::size_t size = reference_string.size();
@@ -1320,7 +1367,9 @@ class json_pointer
JSON_ASSERT(reference_token[pos] == '~'); JSON_ASSERT(reference_token[pos] == '~');
// ~ must be followed by 0 or 1 // ~ must be followed by 0 or 1
if (JSON_HEDLEY_UNLIKELY(is_invalid_escape(reference_token, pos))) if (JSON_HEDLEY_UNLIKELY(pos == reference_token.size() - 1 ||
(reference_token[pos + 1] != '0' &&
reference_token[pos + 1] != '1')))
{ {
JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_invalid_escape, 0, "escape character '~' must be followed with '0' or '1'", nullptr)); JSON_THROW(detail::parse_error::create(detail::exception_id::pointer_invalid_escape, 0, "escape character '~' must be followed with '0' or '1'", nullptr));
} }
@@ -1477,34 +1526,50 @@ class json_pointer
auto hint = object.end(); auto hint = object.end();
const auto enter = [&stack, &path, &object, &hint](const BasicJsonType & v) const auto enter = [&stack, &path, &object, &hint](const BasicJsonType & v)
{ {
if (v.is_array()) switch (v.type())
{ {
if (v.m_data.m_value.array->empty()) case detail::value_t::array:
{ {
// flatten empty array as null if (v.m_data.m_value.array->empty())
add_flattened(object, hint, path, nullptr, detail::priority_tag<2> {}); {
// flatten empty array as null
add_flattened(object, hint, path, nullptr, detail::priority_tag<2> {});
}
else
{
stack.emplace_back(&v, object_const_iterator(), path.size());
}
return;
} }
else
case detail::value_t::object:
{ {
stack.emplace_back(&v, object_const_iterator(), path.size()); if (v.m_data.m_value.object->empty())
{
// flatten empty object as null
add_flattened(object, hint, path, nullptr, detail::priority_tag<2> {});
}
else
{
stack.emplace_back(&v, v.m_data.m_value.object->begin(), path.size());
}
return;
} }
}
else if (v.is_object()) case detail::value_t::null:
{ case detail::value_t::string:
if (v.m_data.m_value.object->empty()) case detail::value_t::boolean:
case detail::value_t::number_integer:
case detail::value_t::number_unsigned:
case detail::value_t::number_float:
case detail::value_t::binary:
case detail::value_t::discarded:
default:
{ {
// flatten empty object as null // add a primitive value with its reference string
add_flattened(object, hint, path, nullptr, detail::priority_tag<2> {}); add_flattened(object, hint, path, v, detail::priority_tag<2> {});
return;
} }
else
{
stack.emplace_back(&v, v.m_data.m_value.object->begin(), path.size());
}
}
else
{
// add a primitive value with its reference string
add_flattened(object, hint, path, v, detail::priority_tag<2> {});
} }
}; };
include/nlohmann/detail/output/binary_writer.hpp
+263 -232
View File
@@ -21,7 +21,6 @@
#include <nlohmann/detail/bit_ops.hpp> #include <nlohmann/detail/bit_ops.hpp>
#include <nlohmann/detail/input/binary_reader.hpp> #include <nlohmann/detail/input/binary_reader.hpp>
#include <nlohmann/detail/input/input_adapters.hpp>
#include <nlohmann/detail/input/string_scan.hpp> #include <nlohmann/detail/input/string_scan.hpp>
#include <nlohmann/detail/macro_scope.hpp> #include <nlohmann/detail/macro_scope.hpp>
#include <nlohmann/detail/output/error_handler.hpp> #include <nlohmann/detail/output/error_handler.hpp>
@@ -312,7 +311,9 @@ class binary_writer
write_cbor_head(0x40, N); write_cbor_head(0x40, N);
// step 2: write each element // step 2: write each element
write_bytes(j.m_data.m_value.binary->data(), N); oa.write_characters(
reinterpret_cast<const CharType*>(j.m_data.m_value.binary->data()),
N);
break; break;
} }
@@ -325,7 +326,7 @@ class binary_writer
case value_t::discarded: case value_t::discarded:
default: default:
throw_on_discarded(j, input_format_t::cbor); throw_on_discarded(j, "CBOR");
} }
} }
@@ -525,7 +526,9 @@ class binary_writer
} }
// step 2: write the byte string // step 2: write the byte string
write_bytes(j.m_data.m_value.binary->data(), N); oa.write_characters(
reinterpret_cast<const CharType*>(j.m_data.m_value.binary->data()),
N);
break; break;
} }
@@ -539,7 +542,7 @@ class binary_writer
case value_t::discarded: case value_t::discarded:
default: default:
throw_on_discarded(j, input_format_t::msgpack); throw_on_discarded(j, "MessagePack");
} }
} }
@@ -623,7 +626,7 @@ class binary_writer
case value_t::array: case value_t::array:
{ {
bool prefix_required = true; bool prefix_required = true;
const bool write_closer = write_ubjson_start_container<true>(j, use_count, use_type, add_prefix, use_bjdata, prefix_required); const bool write_closer = write_ubjson_start_array(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
{ {
const auto& array = *j.m_data.m_value.array; const auto& array = *j.m_data.m_value.array;
@@ -673,7 +676,9 @@ class binary_writer
if (use_type) if (use_type)
{ {
write_bytes(j.m_data.m_value.binary->data(), j.m_data.m_value.binary->size()); oa.write_characters(
reinterpret_cast<const CharType*>(j.m_data.m_value.binary->data()),
j.m_data.m_value.binary->size());
} }
else else
{ {
@@ -699,7 +704,7 @@ class binary_writer
} }
bool prefix_required = true; bool prefix_required = true;
const bool write_closer = write_ubjson_start_container<false>(j, use_count, use_type, add_prefix, use_bjdata, prefix_required); const bool write_closer = write_ubjson_start_object(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
write_ubjson_members(j, use_count, use_type, prefix_required, use_bjdata, bjdata_version, depth); write_ubjson_members(j, use_count, use_type, prefix_required, use_bjdata, bjdata_version, depth);
@@ -713,7 +718,7 @@ class binary_writer
case value_t::discarded: case value_t::discarded:
default: default:
throw_on_discarded(j, use_bjdata ? input_format_t::bjdata : input_format_t::ubjson); throw_on_discarded(j, use_bjdata ? "BJData" : "UBJSON");
} }
} }
@@ -739,11 +744,11 @@ class binary_writer
@brief throws because @a j is discarded and cannot be serialized @brief throws because @a j is discarded and cannot be serialized
@throw type_error.321 always @throw type_error.321 always
*/ */
JSON_HEDLEY_NO_RETURN static void throw_on_discarded(const BasicJsonType& j, const input_format_t format) JSON_HEDLEY_NO_RETURN static void throw_on_discarded(const BasicJsonType& j, const char* format_name)
{ {
static_cast<void>(j); // unused when JSON_NOEXCEPTION is defined static_cast<void>(j); // unused when JSON_NOEXCEPTION is defined
static_cast<void>(format); static_cast<void>(format_name);
JSON_THROW(type_error::create(exception_id::discarded_value_used, concat("cannot serialize discarded value to ", input_format_name(format)), &j)); JSON_THROW(type_error::create(exception_id::discarded_value_used, concat("cannot serialize discarded value to ", format_name), &j));
} }
/// how the range of integer type @a NumberType relates to that of @a TargetType /// how the range of integer type @a NumberType relates to that of @a TargetType
@@ -863,11 +868,11 @@ class binary_writer
} }
/*! /*!
@brief throws because the integer @a j is too large for @a format @brief throws because the integer @a j is too large for @a format_name
@throw out_of_range.407 always @throw out_of_range.407 always
*/ */
JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE
static void throw_integer_out_of_range(const BasicJsonType& j, const input_format_t format, const char* range) static void throw_integer_out_of_range(const BasicJsonType& j, const char* format_name, const char* range)
{ {
// neither std::to_string(), which has no overload for integer types // neither std::to_string(), which has no overload for integer types
// wider than 64 bits, nor dump(), which would also instantiate the // wider than 64 bits, nor dump(), which would also instantiate the
@@ -878,9 +883,9 @@ class binary_writer
? integer_to_string(j.m_data.m_value.number_unsigned, std::integral_constant<bool, unsigned_limits::is_signed>()) ? integer_to_string(j.m_data.m_value.number_unsigned, std::integral_constant<bool, unsigned_limits::is_signed>())
: integer_to_string(j.m_data.m_value.number_integer, std::integral_constant<bool, integer_limits::is_signed>()); : integer_to_string(j.m_data.m_value.number_integer, std::integral_constant<bool, integer_limits::is_signed>());
static_cast<void>(number); // unused when JSON_NOEXCEPTION is defined static_cast<void>(number); // unused when JSON_NOEXCEPTION is defined
static_cast<void>(format); static_cast<void>(format_name);
static_cast<void>(range); static_cast<void>(range);
JSON_THROW(out_of_range::create(exception_id::integer_too_large, concat("integer number ", number, " cannot be represented by ", input_format_name(format), " as it does not fit ", range), &j)); JSON_THROW(out_of_range::create(exception_id::integer_too_large, concat("integer number ", number, " cannot be represented by ", format_name, " as it does not fit ", range), &j));
} }
// The exceptions below are thrown from functions of their own, out of line: // The exceptions below are thrown from functions of their own, out of line:
@@ -923,56 +928,53 @@ class binary_writer
JSON_THROW(other_error::create(exception_id::size_marker_required, "use_type requires use_size = true", &j)); JSON_THROW(other_error::create(exception_id::size_marker_required, "use_type requires use_size = true", &j));
} }
/// @brief write @a size bytes starting at @a data to the output adapter void write_msgpack_array_prefix(const std::size_t N, const BasicJsonType& j)
JSON_HEDLEY_ALWAYS_INLINE
void write_bytes(const void* data, const std::size_t size)
{
oa.write_characters(static_cast<const CharType*>(data), size);
}
/// @brief write the MessagePack prefix of an array or object with @a N
/// elements: the fix marker @a fix_marker | n, or @a marker_16 or
/// @a marker_32 followed by the length
JSON_HEDLEY_ALWAYS_INLINE
void write_msgpack_container_prefix(const std::size_t N, const BasicJsonType& j,
const std::uint8_t fix_marker, const std::uint8_t marker_16, const std::uint8_t marker_32)
{ {
const auto n = to_msgpack_length(N, j); const auto n = to_msgpack_length(N, j);
if (n <= 15) if (n <= 15)
{ {
// fixarray, fixmap // fixarray
oa.write_character(to_char_type(static_cast<std::uint8_t>(fix_marker | n))); oa.write_character(to_char_type(static_cast<std::uint8_t>(0x90 | n)));
} }
else if (n <= (std::numeric_limits<std::uint16_t>::max)()) else if (n <= (std::numeric_limits<std::uint16_t>::max)())
{ {
// array 16, map 16 // array 16
oa.write_character(to_char_type(marker_16)); oa.write_character(to_char_type(0xDC));
write_number(static_cast<std::uint16_t>(n)); write_number(static_cast<std::uint16_t>(n));
} }
else else
{ {
// array 32, map 32 // array 32
oa.write_character(to_char_type(marker_32)); oa.write_character(to_char_type(0xDD));
write_number(static_cast<std::uint32_t>(n)); write_number(static_cast<std::uint32_t>(n));
} }
} }
JSON_HEDLEY_ALWAYS_INLINE
void write_msgpack_array_prefix(const std::size_t N, const BasicJsonType& j)
{
write_msgpack_container_prefix(N, j, 0x90, 0xDC, 0xDD);
}
JSON_HEDLEY_ALWAYS_INLINE
void write_msgpack_object_prefix(const std::size_t N, const BasicJsonType& j) void write_msgpack_object_prefix(const std::size_t N, const BasicJsonType& j)
{ {
write_msgpack_container_prefix(N, j, 0x80, 0xDE, 0xDF); const auto n = to_msgpack_length(N, j);
if (n <= 15)
{
// fixmap
oa.write_character(to_char_type(static_cast<std::uint8_t>(0x80 | (n & 0xF))));
}
else if (n <= (std::numeric_limits<std::uint16_t>::max)())
{
// map 16
oa.write_character(to_char_type(0xDE));
write_number(static_cast<std::uint16_t>(n));
}
else
{
// map 32
oa.write_character(to_char_type(0xDF));
write_number(static_cast<std::uint32_t>(n));
}
} }
/// @brief a CBOR, MessagePack, BON8, or (as the base of @ref ubjson_frame) /// @brief a CBOR, MessagePack, or BON8 array or object whose elements
/// UBJSON array or object whose elements @ref write_container_iterative, /// @ref write_cbor_iterative, @ref write_msgpack_iterative, or
/// @ref write_bon8_iterative, or @ref write_ubjson_iterative is still /// @ref write_bon8_iterative is still writing
/// writing
struct binary_container_frame struct binary_container_frame
{ {
explicit binary_container_frame(const BasicJsonType* value_) noexcept explicit binary_container_frame(const BasicJsonType* value_) noexcept
@@ -1008,82 +1010,19 @@ class binary_writer
typename BasicJsonType::array_t::const_iterator array_it{}; typename BasicJsonType::array_t::const_iterator array_it{};
}; };
/// @brief the CBOR specifics of @ref write_container_iterative
struct cbor_walk
{
static void array_head(binary_writer& w, const BasicJsonType& j)
{
w.write_cbor_head(0x80, j.m_data.m_value.array->size());
}
static void object_head(binary_writer& w, const BasicJsonType& j)
{
w.write_cbor_head(0xA0, j.m_data.m_value.object->size());
}
static void scalar(binary_writer& w, const BasicJsonType& j)
{
w.write_cbor(j);
}
static void string(binary_writer& w, const string_t& key, const BasicJsonType& context)
{
w.write_cbor_string(key, context);
}
template<typename KeyType>
static void key(binary_writer& w, const KeyType& k, const BasicJsonType& object)
{
w.write_cbor_key(k, object);
}
};
/// @brief the MessagePack specifics of @ref write_container_iterative
struct msgpack_walk
{
static void array_head(binary_writer& w, const BasicJsonType& j)
{
w.write_msgpack_array_prefix(j.m_data.m_value.array->size(), j);
}
static void object_head(binary_writer& w, const BasicJsonType& j)
{
w.write_msgpack_object_prefix(j.m_data.m_value.object->size(), j);
}
static void scalar(binary_writer& w, const BasicJsonType& j)
{
w.write_msgpack(j);
}
static void string(binary_writer& w, const string_t& key, const BasicJsonType& context)
{
w.write_msgpack_string(key, context);
}
template<typename KeyType>
static void key(binary_writer& w, const KeyType& k, const BasicJsonType& object)
{
w.write_msgpack_key(k, object);
}
};
/*! /*!
@brief write @a j with @a Walk::scalar, or write its header and push a @brief write @a j with @ref write_cbor, or write its header and push a
frame for @ref write_container_iterative to continue with its frame for @ref write_cbor_iterative to continue with its elements
elements
A scalar, and an empty array or object, are written out in full: there is A scalar, and an empty array or object, are written out in full: there is
nothing below them for @ref write_container_iterative to come back to, so nothing below them for @ref write_cbor_iterative to come back to, so
nothing is pushed for them. nothing is pushed for them.
*/ */
template<typename Walk> void write_cbor_value_or_push(const BasicJsonType& j, std::vector<binary_container_frame>& stack)
JSON_HEDLEY_ALWAYS_INLINE
void write_container_value_or_push(const BasicJsonType& j, std::vector<binary_container_frame>& stack)
{ {
if (j.is_array()) if (j.is_array())
{ {
Walk::array_head(*this, j); write_cbor_head(0x80, j.m_data.m_value.array->size());
if (!j.m_data.m_value.array->empty()) if (!j.m_data.m_value.array->empty())
{ {
stack.emplace_back(&j); stack.emplace_back(&j);
@@ -1093,7 +1032,7 @@ class binary_writer
if (j.is_object()) if (j.is_object())
{ {
Walk::object_head(*this, j); write_cbor_head(0xA0, j.m_data.m_value.object->size());
if (!j.m_data.m_value.object->empty()) if (!j.m_data.m_value.object->empty())
{ {
stack.emplace_back(&j); stack.emplace_back(&j);
@@ -1101,26 +1040,22 @@ class binary_writer
return; return;
} }
Walk::scalar(*this, j); write_cbor(j);
} }
/*! /*!
@brief write out @a root and everything below it without the call stack @brief write out @a root and everything below it without the call stack
Emits the same bytes as the recursive writer, keeping the containers it has Emits the same bytes as @ref write_cbor, keeping the containers it has
entered on an explicit stack instead of descending into them. Only reached entered on an explicit stack instead of descending into them. Only reached
for values nested deeper than @ref recursion_depth_limit, which is why it for values nested deeper than @ref recursion_depth_limit, which is why it
is not written for speed. Shared by CBOR and MessagePack, which differ only is not written for speed.
in the @a Walk policy (@ref cbor_walk, @ref msgpack_walk); BON8 has its own
walker, as it tracks an open string run and closes a container with a
marker.
*/ */
template<typename Walk> void write_cbor_iterative(const BasicJsonType& root)
void write_container_iterative(const BasicJsonType& root)
{ {
// only a container with elements is ever pushed; see write_container_value_or_push // only a container with elements is ever pushed; see write_cbor_value_or_push
std::vector<binary_container_frame> stack; std::vector<binary_container_frame> stack;
write_container_value_or_push<Walk>(root, stack); write_cbor_value_or_push(root, stack);
while (!stack.empty()) while (!stack.empty())
{ {
@@ -1138,7 +1073,7 @@ class binary_writer
// read the child before pushing: entering it can move every frame // read the child before pushing: entering it can move every frame
const BasicJsonType* child = &(*current.array_it); const BasicJsonType* child = &(*current.array_it);
++stack.back().array_it; ++stack.back().array_it;
write_container_value_or_push<Walk>(*child, stack); write_cbor_value_or_push(*child, stack);
} }
else else
{ {
@@ -1150,53 +1085,110 @@ class binary_writer
} }
// the key is written directly (not via a temporary basic_json), // the key is written directly (not via a temporary basic_json),
// with the object as diagnostics context, as in the recursive writer // with the object as diagnostics context, as in write_cbor
Walk::key(*this, current.object_it->first, *current.value); write_cbor_key(current.object_it->first, *current.value);
const BasicJsonType* child = &(current.object_it->second); const BasicJsonType* child = &(current.object_it->second);
++stack.back().object_it; ++stack.back().object_it;
write_container_value_or_push<Walk>(*child, stack); write_cbor_value_or_push(*child, stack);
} }
} }
} }
void write_cbor_iterative(const BasicJsonType& root) /*!
{ @brief write @a j with @ref write_msgpack, or write its header and push a
write_container_iterative<cbor_walk>(root); frame for @ref write_msgpack_iterative to continue with its elements
}
void write_msgpack_iterative(const BasicJsonType& root) @sa @ref write_cbor_value_or_push
*/
void write_msgpack_value_or_push(const BasicJsonType& j, std::vector<binary_container_frame>& stack)
{ {
write_container_iterative<msgpack_walk>(root); if (j.is_array())
{
write_msgpack_array_prefix(j.m_data.m_value.array->size(), j);
if (!j.m_data.m_value.array->empty())
{
stack.emplace_back(&j);
}
return;
}
if (j.is_object())
{
write_msgpack_object_prefix(j.m_data.m_value.object->size(), j);
if (!j.m_data.m_value.object->empty())
{
stack.emplace_back(&j);
}
return;
}
write_msgpack(j);
} }
/*! /*!
@brief write the start of an array (@a IsArray) or object: its opening @brief write out @a root and everything below it without the call stack
marker, the optimized type, and the count
@return true when a closing ']' or '}' still has to be written after the @sa @ref write_cbor_iterative
elements
*/ */
template<bool IsArray> void write_msgpack_iterative(const BasicJsonType& root)
bool write_ubjson_start_container(const BasicJsonType& j, const bool use_count, const bool use_type, {
const bool add_prefix, const bool use_bjdata, bool& prefix_required) std::vector<binary_container_frame> stack;
write_msgpack_value_or_push(root, stack);
while (!stack.empty())
{
const binary_container_frame current = stack.back();
if (current.value->is_array())
{
const auto& array = *current.value->m_data.m_value.array;
if (current.array_it == array.cend())
{
stack.pop_back();
continue;
}
const BasicJsonType* child = &(*current.array_it);
++stack.back().array_it;
write_msgpack_value_or_push(*child, stack);
}
else
{
const auto& object = *current.value->m_data.m_value.object;
if (current.object_it == object.cend())
{
stack.pop_back();
continue;
}
// as in write_cbor_iterative, the key is written directly with
// the object as diagnostics context
write_msgpack_key(current.object_it->first, *current.value);
const BasicJsonType* child = &(current.object_it->second);
++stack.back().object_it;
write_msgpack_value_or_push(*child, stack);
}
}
}
/// @return true when a closing ']' still has to be written after the elements
bool write_ubjson_start_array(const BasicJsonType& j, const bool use_count, const bool use_type,
const bool add_prefix, const bool use_bjdata, bool& prefix_required)
{ {
prefix_required = true; prefix_required = true;
if (add_prefix) if (add_prefix)
{ {
oa.write_character(to_char_type(IsArray ? '[' : '{')); oa.write_character(to_char_type('['));
} }
const std::size_t size = IsArray ? j.m_data.m_value.array->size() : j.m_data.m_value.object->size(); if (use_type && !j.m_data.m_value.array->empty())
if (use_type && size != 0)
{ {
if (!use_count) if (!use_count)
{ {
throw_size_marker_required(j); throw_size_marker_required(j);
} }
const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata); const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata);
// the first element of an array has been compared with itself const bool same_prefix = std::all_of(j.begin() + 1, j.end(),
const bool same_prefix = std::all_of(IsArray ? j.begin() + 1 : j.begin(), j.end(),
[this, first_prefix, use_bjdata](const BasicJsonType & v) [this, first_prefix, use_bjdata](const BasicJsonType & v)
{ {
return ubjson_prefix(v, use_bjdata) == first_prefix; return ubjson_prefix(v, use_bjdata) == first_prefix;
@@ -1208,9 +1200,8 @@ class binary_writer
// those, at one byte per element, so the result can be read back. // those, at one byte per element, so the result can be read back.
// Objects are not affected: every element is preceded by its key. // Objects are not affected: every element is preceded by its key.
const bool valueless_type = (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F'); const bool valueless_type = (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
const bool excessive_valueless = IsArray const bool excessive_valueless = valueless_type
&& valueless_type && j.m_data.m_value.array->size() > detail::max_valueless_container_size;
&& size > detail::max_valueless_container_size;
if (same_prefix && !excessive_valueless if (same_prefix && !excessive_valueless
&& !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix))) && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
@@ -1223,7 +1214,46 @@ class binary_writer
if (use_count) if (use_count)
{ {
write_ubjson_count(size, use_bjdata); write_ubjson_count(j.m_data.m_value.array->size(), use_bjdata);
}
return !use_count;
}
/// @return true when a closing '}' still has to be written after the elements
bool write_ubjson_start_object(const BasicJsonType& j, const bool use_count, const bool use_type,
const bool add_prefix, const bool use_bjdata, bool& prefix_required)
{
prefix_required = true;
if (add_prefix)
{
oa.write_character(to_char_type('{'));
}
if (use_type && !j.m_data.m_value.object->empty())
{
if (!use_count)
{
throw_size_marker_required(j);
}
const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata);
const bool same_prefix = std::all_of(j.begin(), j.end(),
[this, first_prefix, use_bjdata](const BasicJsonType & v)
{
return ubjson_prefix(v, use_bjdata) == first_prefix;
});
if (same_prefix && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
{
prefix_required = false;
oa.write_character(to_char_type('$'));
oa.write_character(first_prefix);
}
}
if (use_count)
{
write_ubjson_count(j.m_data.m_value.object->size(), use_bjdata);
} }
return !use_count; return !use_count;
@@ -1251,12 +1281,21 @@ class binary_writer
/// @brief an object or array @ref write_ubjson_iterative is still writing /// @brief an object or array @ref write_ubjson_iterative is still writing
/// the elements of /// the elements of
struct ubjson_frame : binary_container_frame struct ubjson_frame
{ {
ubjson_frame(const BasicJsonType* value_, const bool prefix_required_) noexcept ubjson_frame(const BasicJsonType* value_, const bool prefix_required_) noexcept
: binary_container_frame(value_) : value(value_)
, prefix_required(prefix_required_) , prefix_required(prefix_required_)
{} {
if (value->is_object())
{
object_it = value->m_data.m_value.object->cbegin();
}
else
{
array_it = value->m_data.m_value.array->cbegin();
}
}
// declared for GCC's -Weffc++, which asks for them in a class with // declared for GCC's -Weffc++, which asks for them in a class with
// pointer members and a non-trivial destructor; the exception // pointer members and a non-trivial destructor; the exception
@@ -1268,9 +1307,13 @@ class binary_writer
ubjson_frame& operator=(ubjson_frame&&) = default; ubjson_frame& operator=(ubjson_frame&&) = default;
~ubjson_frame() = default; ~ubjson_frame() = default;
/// the array or object being written
const BasicJsonType* value;
/// whether value's elements each carry their own type marker; an /// whether value's elements each carry their own type marker; an
/// optimized ($type) container writes it once for all of them instead /// optimized ($type) container writes it once for all of them instead
bool prefix_required; bool prefix_required;
typename BasicJsonType::object_t::const_iterator object_it{};
typename BasicJsonType::array_t::const_iterator array_it{};
}; };
/*! /*!
@@ -1281,7 +1324,7 @@ class binary_writer
elements of an optimized container, and everything below the elements of an optimized container, and everything below the
top level, never repeat it) top level, never repeat it)
@sa @ref write_container_value_or_push @sa @ref write_cbor_value_or_push
*/ */
void write_ubjson_value_or_push(const BasicJsonType& j, const bool add_prefix, const bool use_count, void write_ubjson_value_or_push(const BasicJsonType& j, const bool add_prefix, const bool use_count,
const bool use_type, const bool use_bjdata, const bjdata_version_t bjdata_version, const bool use_type, const bool use_bjdata, const bjdata_version_t bjdata_version,
@@ -1304,11 +1347,11 @@ class binary_writer
bool prefix_required = true; bool prefix_required = true;
if (is_array) if (is_array)
{ {
write_ubjson_start_container<true>(j, use_count, use_type, add_prefix, use_bjdata, prefix_required); write_ubjson_start_array(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
} }
else else
{ {
write_ubjson_start_container<false>(j, use_count, use_type, add_prefix, use_bjdata, prefix_required); write_ubjson_start_object(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
} }
const bool empty = is_array ? j.m_data.m_value.array->empty() : j.m_data.m_value.object->empty(); const bool empty = is_array ? j.m_data.m_value.array->empty() : j.m_data.m_value.object->empty();
@@ -1318,7 +1361,7 @@ class binary_writer
return; return;
} }
// write_ubjson_start_container returns !use_count, i.e. whether a // write_ubjson_start_array/_object return !use_count, i.e. whether a
// closer still has to be written; use_count is constant for the whole // closer still has to be written; use_count is constant for the whole
// document, so that is recomputed here instead of being carried along // document, so that is recomputed here instead of being carried along
if (!use_count) if (!use_count)
@@ -1507,15 +1550,14 @@ class binary_writer
{ {
oa.write_character(to_char_type(element_type)); oa.write_character(to_char_type(element_type));
if (JSON_HEDLEY_LIKELY(error_handler == error_handler_t::keep || error_handler == error_handler_t::strict)) if (error_handler == error_handler_t::keep || error_handler == error_handler_t::strict || is_valid_utf8(name))
{ {
write_bytes(name.data(), name.size()); oa.write_characters(reinterpret_cast<const CharType*>(name.data()), name.size());
} }
else else
{ {
string_t storage; const string_t sanitized = sanitize_utf8(name, error_handler);
const string_t& written = sanitize_utf8_for_bson_write(name, storage); oa.write_characters(reinterpret_cast<const CharType*>(sanitized.data()), sanitized.size());
write_bytes(written.data(), written.size());
} }
// the terminating null byte is written explicitly rather than taken // the terminating null byte is written explicitly rather than taken
@@ -1591,11 +1633,16 @@ class binary_writer
{ {
write_bson_entry_header(name, 0x02); write_bson_entry_header(name, 0x02);
string_t storage; const bool sanitize = error_handler != error_handler_t::keep
const string_t& written = sanitize_utf8_for_bson_write(value, storage); && error_handler != error_handler_t::strict
&& !is_valid_utf8(value);
const string_t sanitized = sanitize ? sanitize_utf8(value, error_handler) : string_t{};
const string_t& written = sanitize ? sanitized : value;
write_number<std::int32_t>(to_bson_length(written.size() + 1ul), true); write_number<std::int32_t>(to_bson_length(written.size() + 1ul), true);
write_bytes(written.data(), written.size()); oa.write_characters(
reinterpret_cast<const CharType*>(written.data()),
written.size());
// the terminating null byte is written explicitly rather than taken // the terminating null byte is written explicitly rather than taken
// from the buffer, so that string_t::data() need not be null-terminated // from the buffer, so that string_t::data() need not be null-terminated
oa.write_character(to_char_type(0x00)); oa.write_character(to_char_type(0x00));
@@ -1701,7 +1748,7 @@ class binary_writer
write_number(value.has_subtype() ? static_cast<std::uint8_t>(value.subtype()) : static_cast<std::uint8_t>(0x00)); write_number(value.has_subtype() ? static_cast<std::uint8_t>(value.subtype()) : static_cast<std::uint8_t>(0x00));
write_bytes(value.data(), value.size()); oa.write_characters(reinterpret_cast<const CharType*>(value.data()), value.size());
} }
/*! /*!
@@ -1731,14 +1778,14 @@ class binary_writer
case value_t::number_integer: case value_t::number_integer:
if (JSON_HEDLEY_UNLIKELY(!integer_fits<std::int64_t>(j.m_data.m_value.number_integer))) if (JSON_HEDLEY_UNLIKELY(!integer_fits<std::int64_t>(j.m_data.m_value.number_integer)))
{ {
throw_integer_out_of_range(j, input_format_t::bson, "int64"); throw_integer_out_of_range(j, "BSON", "int64");
} }
return calc_bson_integer_size(static_cast<std::int64_t>(j.m_data.m_value.number_integer)); return calc_bson_integer_size(static_cast<std::int64_t>(j.m_data.m_value.number_integer));
case value_t::number_unsigned: case value_t::number_unsigned:
if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_unsigned))) if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_unsigned)))
{ {
throw_integer_out_of_range(j, input_format_t::bson, "uint64"); throw_integer_out_of_range(j, "BSON", "uint64");
} }
return calc_bson_unsigned_size(conditional_static_cast<std::uint64_t>(j.m_data.m_value.number_unsigned)); return calc_bson_unsigned_size(conditional_static_cast<std::uint64_t>(j.m_data.m_value.number_unsigned));
@@ -1749,7 +1796,7 @@ class binary_writer
return 0ul; return 0ul;
case value_t::discarded: case value_t::discarded:
throw_on_discarded(j, input_format_t::bson); throw_on_discarded(j, "BSON");
// LCOV_EXCL_START // LCOV_EXCL_START
case value_t::object: case value_t::object:
@@ -1913,7 +1960,7 @@ class binary_writer
// the terminating null byte is written explicitly rather than // the terminating null byte is written explicitly rather than
// taken from the buffer, so that string_t::data() need not be // taken from the buffer, so that string_t::data() need not be
// null-terminated // null-terminated
write_bytes(value.data(), value.size()); oa.write_characters(reinterpret_cast<const CharType*>(value.data()), value.size());
oa.write_character(to_char_type(0x00)); oa.write_character(to_char_type(0x00));
return true; return true;
} }
@@ -1930,7 +1977,7 @@ class binary_writer
p += sizeof(std::int32_t); p += sizeof(std::int32_t);
*p++ = value.has_subtype() ? to_char_type(static_cast<std::uint8_t>(value.subtype())) : to_char_type(0x00); *p++ = value.has_subtype() ? to_char_type(static_cast<std::uint8_t>(value.subtype())) : to_char_type(0x00);
oa.commit(static_cast<std::size_t>(p - out)); oa.commit(static_cast<std::size_t>(p - out));
write_bytes(value.data(), value.size()); oa.write_characters(reinterpret_cast<const CharType*>(value.data()), value.size());
return true; return true;
} }
@@ -1978,7 +2025,7 @@ class binary_writer
return write_bson_null(name); return write_bson_null(name);
case value_t::discarded: case value_t::discarded:
throw_on_discarded(j, input_format_t::bson); throw_on_discarded(j, "BSON");
// LCOV_EXCL_START // LCOV_EXCL_START
case value_t::object: case value_t::object:
@@ -2759,7 +2806,7 @@ class binary_writer
{ {
if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_unsigned))) if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_unsigned)))
{ {
throw_integer_out_of_range(j, input_format_t::cbor, "uint64"); throw_integer_out_of_range(j, "CBOR", "uint64");
} }
write_cbor_head(0x00, conditional_static_cast<std::uint64_t>(j.m_data.m_value.number_unsigned)); write_cbor_head(0x00, conditional_static_cast<std::uint64_t>(j.m_data.m_value.number_unsigned));
return; return;
@@ -2769,7 +2816,7 @@ class binary_writer
{ {
if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_integer))) if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_integer)))
{ {
throw_integer_out_of_range(j, input_format_t::cbor, "[-2^64, 2^64-1]"); throw_integer_out_of_range(j, "CBOR", "[-2^64, 2^64-1]");
} }
// CBOR does not differentiate between positive signed // CBOR does not differentiate between positive signed
// integers and unsigned integers // integers and unsigned integers
@@ -2780,7 +2827,7 @@ class binary_writer
// a negative integer n is encoded as -1 - n // a negative integer n is encoded as -1 - n
if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(-1 - j.m_data.m_value.number_integer))) if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(-1 - j.m_data.m_value.number_integer)))
{ {
throw_integer_out_of_range(j, input_format_t::cbor, "[-2^64, 2^64-1]"); throw_integer_out_of_range(j, "CBOR", "[-2^64, 2^64-1]");
} }
write_cbor_head(0x20, static_cast<std::uint64_t>(-1 - j.m_data.m_value.number_integer)); write_cbor_head(0x20, static_cast<std::uint64_t>(-1 - j.m_data.m_value.number_integer));
} }
@@ -2856,7 +2903,7 @@ class binary_writer
{ {
if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_unsigned))) if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_unsigned)))
{ {
throw_integer_out_of_range(j, input_format_t::msgpack, "uint64"); throw_integer_out_of_range(j, "MessagePack", "uint64");
} }
write_msgpack_unsigned(static_cast<std::uint64_t>(j.m_data.m_value.number_unsigned)); write_msgpack_unsigned(static_cast<std::uint64_t>(j.m_data.m_value.number_unsigned));
return; return;
@@ -2866,7 +2913,7 @@ class binary_writer
{ {
if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_integer))) if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::uint64_t>(j.m_data.m_value.number_integer)))
{ {
throw_integer_out_of_range(j, input_format_t::msgpack, "[-2^63, 2^64-1]"); throw_integer_out_of_range(j, "MessagePack", "[-2^63, 2^64-1]");
} }
// MessagePack does not differentiate between positive // MessagePack does not differentiate between positive
// signed integers and unsigned integers. // signed integers and unsigned integers.
@@ -2876,7 +2923,7 @@ class binary_writer
{ {
if (JSON_HEDLEY_UNLIKELY(!integer_fits_min<std::int64_t>(j.m_data.m_value.number_integer))) if (JSON_HEDLEY_UNLIKELY(!integer_fits_min<std::int64_t>(j.m_data.m_value.number_integer)))
{ {
throw_integer_out_of_range(j, input_format_t::msgpack, "[-2^63, 2^64-1]"); throw_integer_out_of_range(j, "MessagePack", "[-2^63, 2^64-1]");
} }
if (j.m_data.m_value.number_integer >= -32) if (j.m_data.m_value.number_integer >= -32)
{ {
@@ -3010,37 +3057,40 @@ class binary_writer
} }
/*! /*!
@brief write an object key as a CBOR or MessagePack string (@a Walk) @brief write an object key as a CBOR text string
A key convertible to string_t is written directly. Other key types (only A key convertible to string_t is written directly. Other key types (only
an explicit conversion, or only a to_json overload) go through a temporary an explicit conversion, or only a to_json overload) go through a temporary
basic_json, as in version 3.12.0; the temporary is then the diagnostics basic_json, as in version 3.12.0; the temporary is then the diagnostics
context for strict UTF-8 checks. context for strict UTF-8 checks.
*/ */
template<typename Walk, typename Key = typename BasicJsonType::object_t::key_type, template<typename Key = typename BasicJsonType::object_t::key_type,
enable_if_t<std::is_convertible<Key, string_t>::value, int> = 0> enable_if_t<std::is_convertible<Key, string_t>::value, int> = 0>
void write_object_key(const typename BasicJsonType::object_t::key_type& key, const BasicJsonType& context)
{
Walk::string(*this, key, context);
}
template<typename Walk, typename Key = typename BasicJsonType::object_t::key_type,
enable_if_t<negation<std::is_convertible<Key, string_t>>::value, int> = 0>
void write_object_key(const typename BasicJsonType::object_t::key_type& key, const BasicJsonType& /*context*/)
{
Walk::scalar(*this, BasicJsonType(key));
}
/// @brief write an object key as a CBOR text string
void write_cbor_key(const typename BasicJsonType::object_t::key_type& key, const BasicJsonType& context) void write_cbor_key(const typename BasicJsonType::object_t::key_type& key, const BasicJsonType& context)
{ {
write_object_key<cbor_walk>(key, context); write_cbor_string(key, context);
} }
/// @brief write an object key as a MessagePack str template < typename Key = typename BasicJsonType::object_t::key_type,
enable_if_t < !std::is_convertible<Key, string_t>::value, int > = 0 >
void write_cbor_key(const typename BasicJsonType::object_t::key_type& key, const BasicJsonType& /*context*/)
{
write_cbor(BasicJsonType(key));
}
/// @brief write an object key as a MessagePack str, as in @ref write_cbor_key
template<typename Key = typename BasicJsonType::object_t::key_type,
enable_if_t<std::is_convertible<Key, string_t>::value, int> = 0>
void write_msgpack_key(const typename BasicJsonType::object_t::key_type& key, const BasicJsonType& context) void write_msgpack_key(const typename BasicJsonType::object_t::key_type& key, const BasicJsonType& context)
{ {
write_object_key<msgpack_walk>(key, context); write_msgpack_string(key, context);
}
template < typename Key = typename BasicJsonType::object_t::key_type,
enable_if_t < !std::is_convertible<Key, string_t>::value, int > = 0 >
void write_msgpack_key(const typename BasicJsonType::object_t::key_type& key, const BasicJsonType& /*context*/)
{
write_msgpack(BasicJsonType(key));
} }
/// the longest string that is written, with its head of one or two /// the longest string that is written, with its head of one or two
@@ -3115,7 +3165,9 @@ class binary_writer
write_cbor_head(0x60, sanitized.size()); write_cbor_head(0x60, sanitized.size());
// step 2: write the string // step 2: write the string
write_bytes(sanitized.data(), sanitized.size()); oa.write_characters(
reinterpret_cast<const CharType*>(sanitized.data()),
sanitized.size());
} }
///////////// /////////////
@@ -3196,7 +3248,9 @@ class binary_writer
} }
// step 2: write the string // step 2: write the string
write_bytes(sanitized.data(), sanitized.size()); oa.write_characters(
reinterpret_cast<const CharType*>(sanitized.data()),
sanitized.size());
} }
//////////// ////////////
@@ -3479,7 +3533,9 @@ class binary_writer
oa.write_character(to_char_type('S')); oa.write_character(to_char_type('S'));
} }
write_number_with_ubjson_prefix(sanitized.size(), true, use_bjdata); write_number_with_ubjson_prefix(sanitized.size(), true, use_bjdata);
write_bytes(sanitized.data(), sanitized.size()); oa.write_characters(
reinterpret_cast<const CharType*>(sanitized.data()),
sanitized.size());
} }
/*! /*!
@@ -4279,7 +4335,7 @@ class binary_writer
{ {
if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::int64_t>(j.m_data.m_value.number_unsigned))) if (JSON_HEDLEY_UNLIKELY(!integer_fits_max<std::int64_t>(j.m_data.m_value.number_unsigned)))
{ {
throw_integer_out_of_range(j, input_format_t::bon8, "int64"); throw_integer_out_of_range(j, "BON8", "int64");
} }
write_bon8_integer(static_cast<std::int64_t>(j.m_data.m_value.number_unsigned)); write_bon8_integer(static_cast<std::int64_t>(j.m_data.m_value.number_unsigned));
string_open = false; string_open = false;
@@ -4290,7 +4346,7 @@ class binary_writer
{ {
if (JSON_HEDLEY_UNLIKELY(!integer_fits<std::int64_t>(j.m_data.m_value.number_integer))) if (JSON_HEDLEY_UNLIKELY(!integer_fits<std::int64_t>(j.m_data.m_value.number_integer)))
{ {
throw_integer_out_of_range(j, input_format_t::bon8, "int64"); throw_integer_out_of_range(j, "BON8", "int64");
} }
write_bon8_integer(static_cast<std::int64_t>(j.m_data.m_value.number_integer)); write_bon8_integer(static_cast<std::int64_t>(j.m_data.m_value.number_integer));
string_open = false; string_open = false;
@@ -4358,7 +4414,7 @@ class binary_writer
case value_t::discarded: case value_t::discarded:
default: default:
throw_on_discarded(j, input_format_t::bon8); throw_on_discarded(j, "BON8");
} }
} }
@@ -4369,7 +4425,7 @@ class binary_writer
A scalar, and an empty array or object, are written out in full and not A scalar, and an empty array or object, are written out in full and not
pushed. pushed.
@sa @ref write_container_value_or_push @sa @ref write_cbor_value_or_push
*/ */
void write_bon8_value_or_push(const BasicJsonType& j, bool& string_open, std::vector<binary_container_frame>& stack) void write_bon8_value_or_push(const BasicJsonType& j, bool& string_open, std::vector<binary_container_frame>& stack)
{ {
@@ -4755,7 +4811,7 @@ class binary_writer
} }
else else
{ {
write_bytes(s.data(), s.size()); oa.write_characters(reinterpret_cast<const CharType*>(s.data()), s.size());
string_open = true; string_open = true;
} }
} }
@@ -4846,31 +4902,6 @@ class binary_writer
} }
} }
/*!
@brief the write-pass counterpart of @ref sanitize_utf8_for_write for BSON
The size pass (@ref calc_bson_entry_header_size, @ref
calc_bson_string_size) has already validated @a s, and thrown for @ref
error_handler_t::strict if it was ill-formed. Here, @ref
error_handler_t::strict therefore behaves like @ref error_handler_t::keep,
and @ref error_handler_t::replace / @ref error_handler_t::ignore produce
exactly the bytes the size pass measured. Hence no context is needed.
@param[in] s the string (value or object key) to write
@param[out] storage backing storage for a sanitized copy
@return a reference to @a s, or to @a storage once it holds a sanitized copy
*/
const string_t& sanitize_utf8_for_bson_write(const string_t& s, string_t& storage) const
{
if (error_handler == error_handler_t::keep || error_handler == error_handler_t::strict || is_valid_utf8(s))
{
return s; // NOLINT(bugprone-return-const-ref-from-parameter): callers pass lvalues that outlive the call
}
storage = sanitize_utf8(s, error_handler);
return storage;
}
/// deleted: anything but a string_t would bind a temporary that dies before the returned reference is used /// deleted: anything but a string_t would bind a temporary that dies before the returned reference is used
template < typename T, enable_if_t < !std::is_same<T, string_t>::value, int > = 0 > template < typename T, enable_if_t < !std::is_same<T, string_t>::value, int > = 0 >
const string_t& sanitize_utf8_for_write(const T& /*s*/, const BasicJsonType& /*context*/, string_t& /*storage*/) const = delete; // NOLINT(hicpp-use-equals-delete,modernize-use-equals-delete): a private helper's guard, not part of the interface const string_t& sanitize_utf8_for_write(const T& /*s*/, const BasicJsonType& /*context*/, string_t& /*storage*/) const = delete; // NOLINT(hicpp-use-equals-delete,modernize-use-equals-delete): a private helper's guard, not part of the interface
include/nlohmann/detail/output/serializer.hpp
+71 -44
View File
@@ -966,28 +966,6 @@ class serializer
} }
} }
/// write out the bytes pending in @ref string_buffer and reset @a bytes
JSON_HEDLEY_ALWAYS_INLINE void emit_pending(std::size_t& bytes)
{
if (bytes != 0)
{
put_buffer(string_buffer, bytes);
bytes = 0;
}
}
/// write buffer and reset index if fewer than 13 bytes are left, as this
/// is the maximal number of bytes to be written ("\uxxxx\uxxxx\0") for one
/// code point
JSON_HEDLEY_ALWAYS_INLINE void reserve_string_buffer(std::size_t& bytes)
{
if (string_buffer.size() - bytes < 13)
{
put_buffer(string_buffer, bytes);
bytes = 0;
}
}
/*! /*!
@brief worker for @ref dump_escaped @brief worker for @ref dump_escaped
@@ -1046,7 +1024,11 @@ class serializer
{ {
// emit any bytes still pending in string_buffer first to // emit any bytes still pending in string_buffer first to
// preserve output order, then write the run directly // preserve output order, then write the run directly
emit_pending(bytes); if (bytes != 0)
{
put_buffer(string_buffer, bytes);
bytes = 0;
}
put_run(s, i, run); put_run(s, i, run);
bytes_after_last_accept = 0; bytes_after_last_accept = 0;
undumped_chars = 0; undumped_chars = 0;
@@ -1067,10 +1049,11 @@ class serializer
{ {
// emit any bytes still pending in string_buffer first to // emit any bytes still pending in string_buffer first to
// preserve output order // preserve output order
emit_pending(bytes); if (bytes != 0)
// deliberately not put_buffer(): a fixed 8-byte memcpy {
// has no asserts and no write-through branch, which put_buffer(string_buffer, bytes);
// matters on the hot path bytes = 0;
}
const auto& escape = ascii_escapes()[data[i]]; const auto& escape = ascii_escapes()[data[i]];
if (JSON_HEDLEY_UNLIKELY(write_buffer.size() - write_buffer_pos < escape.size())) if (JSON_HEDLEY_UNLIKELY(write_buffer.size() - write_buffer_pos < escape.size()))
{ {
@@ -1145,7 +1128,14 @@ class serializer
string_buffer[bytes++] = s[i]; string_buffer[bytes++] = s[i];
} }
reserve_string_buffer(bytes); // write buffer and reset index; there must be 13 bytes
// left, as this is the maximal number of bytes to be
// written ("\uxxxx\uxxxx\0") for one code point
if (string_buffer.size() - bytes < 13)
{
put_buffer(string_buffer, bytes);
bytes = 0;
}
// remember the byte position of this accept // remember the byte position of this accept
bytes_after_last_accept = bytes; bytes_after_last_accept = bytes;
@@ -1197,7 +1187,14 @@ class serializer
string_buffer[bytes++] = '\xBD'; string_buffer[bytes++] = '\xBD';
} }
reserve_string_buffer(bytes); // write buffer and reset index; there must be 13 bytes
// left, as this is the maximal number of bytes to be
// written ("\uxxxx\uxxxx\0") for one code point
if (string_buffer.size() - bytes < 13)
{
put_buffer(string_buffer, bytes);
bytes = 0;
}
bytes_after_last_accept = bytes; bytes_after_last_accept = bytes;
} }
@@ -1230,7 +1227,14 @@ class serializer
string_buffer[bytes++] = s[i]; string_buffer[bytes++] = s[i];
} }
reserve_string_buffer(bytes); // write buffer and reset index; there must be 13 bytes
// left, as this is the maximal number of bytes to be
// written ("\uxxxx\uxxxx\0") for one code point
if (string_buffer.size() - bytes < 13)
{
put_buffer(string_buffer, bytes);
bytes = 0;
}
bytes_after_last_accept = bytes; bytes_after_last_accept = bytes;
undumped_chars = 0; undumped_chars = 0;
@@ -1596,17 +1600,30 @@ class serializer
*/ */
static const std::array<std::uint8_t, 256>& ascii_copyable() noexcept static const std::array<std::uint8_t, 256>& ascii_copyable() noexcept
{ {
// constant-initialized from the predicate: no runtime loop, no guard static const std::array<std::uint8_t, 256> copyable =
static constexpr std::array<std::uint8_t, 256> copyable = make_ascii_copyable(make_index_sequence<256> {}); {
{
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0x00
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0x10
1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 0x20
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 0x30
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 0x40
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, // 0x50
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, // 0x60
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, // 0x70
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0x80
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0x90
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0xA0
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0xB0
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0xC0
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0xD0
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 0xE0
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 // 0xF0
}
};
return copyable; return copyable;
} }
template<std::size_t... Is>
static constexpr std::array<std::uint8_t, 256> make_ascii_copyable(index_sequence<Is...> /*unused*/) noexcept
{
return {{ static_cast<std::uint8_t>(is_ascii_copyable(static_cast<unsigned char>(Is)) ? 1 : 0)... }};
}
/*! /*!
@brief the JSON form of each character below 0x60 @brief the JSON form of each character below 0x60
@@ -1735,6 +1752,7 @@ class serializer
*/ */
JSON_HEDLEY_NEVER_INLINE std::size_t write_u_escapes(const unsigned char* data, std::size_t n) JSON_HEDLEY_NEVER_INLINE std::size_t write_u_escapes(const unsigned char* data, std::size_t n)
{ {
constexpr const char* nibble_to_hex = "0123456789abcdef";
std::size_t i = 0; std::size_t i = 0;
while (i < n && data[i] >= 0x80) while (i < n && data[i] >= 0x80)
{ {
@@ -1771,21 +1789,26 @@ class serializer
char* d = write_buffer.data() + write_buffer_pos; char* d = write_buffer.data() + write_buffer_pos;
if (codepoint > 0xFFFFu) if (codepoint > 0xFFFFu)
{ {
d = put_u_escape(d, static_cast<std::uint16_t>(0xD7C0u + (codepoint >> 10u))); d = put_u_escape(d, static_cast<std::uint16_t>(0xD7C0u + (codepoint >> 10u)), nibble_to_hex);
codepoint = 0xDC00u + (codepoint & 0x3FFu); codepoint = 0xDC00u + (codepoint & 0x3FFu);
} }
d = put_u_escape(d, static_cast<std::uint16_t>(codepoint)); d = put_u_escape(d, static_cast<std::uint16_t>(codepoint), nibble_to_hex);
write_buffer_pos = static_cast<std::size_t>(d - write_buffer.data()); write_buffer_pos = static_cast<std::size_t>(d - write_buffer.data());
i += length; i += length;
} }
return i; return i;
} }
static char* put_u_escape(char* d, const std::uint16_t codeunit) noexcept static char* put_u_escape(char* d, const std::uint16_t codeunit, const char* nibble_to_hex) noexcept
{ {
const auto unit = static_cast<std::uint32_t>(codeunit);
d[0] = '\\'; d[0] = '\\';
d[1] = 'u'; d[1] = 'u';
return hex4_lower(codeunit, d + 2); d[2] = nibble_to_hex[(unit >> 12u) & 0x0Fu];
d[3] = nibble_to_hex[(unit >> 8u) & 0x0Fu];
d[4] = nibble_to_hex[(unit >> 4u) & 0x0Fu];
d[5] = nibble_to_hex[unit & 0x0Fu];
return d + 6;
} }
/*! /*!
@@ -1794,7 +1817,7 @@ class serializer
* Branch-free replacement for `snprintf(buf, 7, "\\u%04x", codeunit)` in the * Branch-free replacement for `snprintf(buf, 7, "\\u%04x", codeunit)` in the
* string escaping hot path. It writes exactly six characters ('\\', 'u' and * string escaping hot path. It writes exactly six characters ('\\', 'u' and
* four hex digits) at position @a pos of @a string_buffer via a nibble * four hex digits) at position @a pos of @a string_buffer via a nibble
* lookup table (@ref hex4_lower), avoiding the format-string parsing and locale machinery of * lookup table, avoiding the format-string parsing and locale machinery of
* `snprintf`. Advances @a pos by the number of bytes written (6). * `snprintf`. Advances @a pos by the number of bytes written (6).
* *
* @param[in] pos position in @a string_buffer to write at; there must * @param[in] pos position in @a string_buffer to write at; there must
@@ -1804,9 +1827,13 @@ class serializer
void write_u_escape(std::size_t& pos, std::uint16_t codeunit) noexcept void write_u_escape(std::size_t& pos, std::uint16_t codeunit) noexcept
{ {
JSON_ASSERT(string_buffer.size() - pos >= 6); JSON_ASSERT(string_buffer.size() - pos >= 6);
constexpr const char* nibble_to_hex = "0123456789abcdef";
string_buffer[pos + 0] = '\\'; string_buffer[pos + 0] = '\\';
string_buffer[pos + 1] = 'u'; string_buffer[pos + 1] = 'u';
hex4_lower(codeunit, &string_buffer[pos + 2]); string_buffer[pos + 2] = nibble_to_hex[(codeunit >> 12u) & 0x0Fu];
string_buffer[pos + 3] = nibble_to_hex[(codeunit >> 8u) & 0x0Fu];
string_buffer[pos + 4] = nibble_to_hex[(codeunit >> 4u) & 0x0Fu];
string_buffer[pos + 5] = nibble_to_hex[codeunit & 0x0Fu];
pos += 6; pos += 6;
} }
include/nlohmann/detail/string_utils.hpp
+4 -87
View File
@@ -48,24 +48,6 @@ inline std::string hex_byte(const std::uint8_t byte)
return result; return result;
} }
/*!
@brief write a UTF-16 code unit as four lowercase hexadecimal digits
The lowercase counterpart of hex_byte(), used for the `\\uXXXX` escapes of the
serializers, whose output is lowercase. Writes exactly four characters to
@a out and returns the pointer past them.
*/
JSON_HEDLEY_ALWAYS_INLINE char* hex4_lower(const std::uint16_t codeunit, char* out) noexcept
{
constexpr const char* nibble_to_hex = "0123456789abcdef";
const unsigned int unit = codeunit;
out[0] = nibble_to_hex[(unit >> 12u) & 0x0Fu];
out[1] = nibble_to_hex[(unit >> 8u) & 0x0Fu];
out[2] = nibble_to_hex[(unit >> 4u) & 0x0Fu];
out[3] = nibble_to_hex[unit & 0x0Fu];
return out + 4;
}
/////////////////// ///////////////////
// UTF-8 encoding // // UTF-8 encoding //
/////////////////// ///////////////////
@@ -121,66 +103,6 @@ void encode_utf8(std::uint32_t cp, const Out& out)
} }
} }
/*!
@brief the shape of a well-formed UTF-8 sequence, by its lead byte
@a length is 1 for ASCII, 2 to 4 for a valid lead byte, and 0 for a byte that
cannot start a well-formed sequence (a continuation byte, 0xC0, 0xC1, or 0xF5
to 0xFF). For a multibyte sequence, the second byte must lie in
[@a second_lo, @a second_hi], and every later byte in [0x80, 0xBF]
(RFC 3629, section 4; Unicode Table 3-7). The narrower second-byte ranges
exclude overlong forms (0xE0, 0xF0), surrogates (0xED), and code points
beyond U+10FFFF (0xF4).
*/
struct utf8_sequence_shape
{
std::uint8_t length;
std::uint8_t second_lo;
std::uint8_t second_hi;
};
/// @return the shape of a UTF-8 sequence starting with @a lead, see utf8_sequence_shape
inline utf8_sequence_shape utf8_shape(const std::uint8_t lead) noexcept
{
if (lead < 0x80u)
{
return {1, 0, 0};
}
if (lead < 0xC2u)
{
return {0, 0, 0};
}
if (lead <= 0xDFu)
{
return {2, 0x80u, 0xBFu};
}
if (lead == 0xE0u)
{
return {3, 0xA0u, 0xBFu};
}
if (lead == 0xEDu)
{
return {3, 0x80u, 0x9Fu};
}
if (lead <= 0xEFu)
{
return {3, 0x80u, 0xBFu};
}
if (lead == 0xF0u)
{
return {4, 0x90u, 0xBFu};
}
if (lead <= 0xF3u)
{
return {4, 0x80u, 0xBFu};
}
if (lead == 0xF4u)
{
return {4, 0x80u, 0x8Fu};
}
return {0, 0, 0};
}
/////////////////// ///////////////////
// UTF-8 decoding // // UTF-8 decoding //
/////////////////// ///////////////////
@@ -196,7 +118,7 @@ This is a single-byte step of a "shift-based" UTF-8 decoder originally
written by Björn Hoehrmann. See written by Björn Hoehrmann. See
http://bjoern.hoehrmann.de/utf-8/decoder/dfa/ for details. http://bjoern.hoehrmann.de/utf-8/decoder/dfa/ for details.
The library checks UTF-8 well-formedness (RFC 3629, section 4) in five The library checks UTF-8 well-formedness (RFC 3629, section 4) in three
places, which differ in speed, diagnostics, and how they read the input: places, which differ in speed, diagnostics, and how they read the input:
- decode() below: the serializer, to escape and, in strict mode, reject - decode() below: the serializer, to escape and, in strict mode, reject
@@ -210,15 +132,10 @@ places, which differ in speed, diagnostics, and how they read the input:
bulk string scan, the bulk path of the BON8 reader, and the BON8 writer. bulk string scan, the bulk path of the BON8 reader, and the BON8 writer.
They must accept exactly what the lexer's switch accepts. They must accept exactly what the lexer's switch accepts.
- the byte path of binary_reader::get_bon8_string(): BON8 input without bulk - the byte path of binary_reader::get_bon8_string(): BON8 input without bulk
access, and the bytes the bulk path leaves to it. It keeps its own ranges: access, and the bytes the bulk path leaves to it.
taking them from utf8_shape() made BON8 decoding 2-3% slower (#5891).
- check_utf8() in detail/view/edit.hpp: strings stored in an editable
json_document, with the index of the offending byte. It also uses
utf8_shape().
All five must accept the same set of sequences, which the test "UTF-8 All four must accept the same set of sequences, so a change to one needs a
validators accept the same sequences" in unit-class_lexer.cpp checks matching change to the others.
exhaustively against decode() and utf8_shape().
@param[in,out] state the current decoder state @param[in,out] state the current decoder state
@param[in,out] codep codepoint (valid only if resulting state is UTF8_ACCEPT) @param[in,out] codep codepoint (valid only if resulting state is UTF8_ACCEPT)
include/nlohmann/detail/value_t.hpp
-30
View File
@@ -177,36 +177,6 @@ inline bool owns_memory(const value_t t) noexcept
} }
} }
/*!
@brief whether a value of the given type is neither an object nor an array
Unlike basic_json::is_primitive(), this includes value_t::discarded: it names
the types that iterators walk with a primitive_iterator_t (a single value, or no
value for null and discarded) and that JSON Pointer resolution and the
serializer treat as leaves. Every enumerator is listed so that -Wswitch-enum
flags this function when a new type is added.
*/
inline bool is_primitive(const value_t t) noexcept
{
switch (t)
{
case value_t::object:
case value_t::array:
return false;
case value_t::null:
case value_t::string:
case value_t::boolean:
case value_t::number_integer:
case value_t::number_unsigned:
case value_t::number_float:
case value_t::binary:
case value_t::discarded:
default:
return true;
}
}
/*! /*!
@brief compare an integer with a floating point number without precision loss @brief compare an integer with a floating point number without precision loss
include/nlohmann/detail/view/builder.hpp
+163 -47
View File
@@ -275,11 +275,34 @@ class builder
0, 0, '\b', 0, 0, 0, '\f', 0, 0, 0, 0, 0, 0, 0, '\n', 0, 0, 0, '\r', 0, '\t', 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 0, 0, '\b', 0, 0, 0, '\f', 0, 0, 0, 0, 0, 0, 0, '\n', 0, 0, 0, '\r', 0, '\t', 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
} }
}; };
// The write position is kept in locals: the stores of characters into
// the arena may alias the members of the builder, which would then be
// reloaded after every one. It is stored back where the arena grows,
// and by `write_back` on every return.
const unsigned char* const end = m_end;
const std::size_t start = arena_used(); const std::size_t start = arena_used();
arena_run(s, static_cast<std::size_t>(p - s)); char* w = m_arena_write;
const char* w_end = m_arena_end;
const arena_write_back write_back(*this, w);
{
const auto n = static_cast<std::size_t>(p - s);
if (n <= 16 && end - s >= 16)
{
arena_ensure(w, w_end, 16);
std::memcpy(w, s, 16);
}
else if (n != 0)
{
// (an empty run leaves an empty arena unallocated, and memcpy()
// must not be passed its null pointer even for zero bytes)
arena_ensure(w, w_end, n);
std::memcpy(w, s, n);
}
w += n;
}
for (;;) for (;;)
{ {
if (p == m_end) if (p == end)
{ {
return failed(error_code::string_missing_quote, p); return failed(error_code::string_missing_quote, p);
} }
@@ -287,7 +310,9 @@ class builder
if (c == '"') if (c == '"')
{ {
++p; ++p;
return decoded{p, start, arena_used() - start}; // (from w: `write_back` stores it only after the return value
// is made; the arena is allocated, since an escape was written)
return decoded{p, start, static_cast<std::size_t>(w - m_doc.arena.data()) - start};
} }
if (c != '\\') if (c != '\\')
{ {
@@ -297,7 +322,7 @@ class builder
return failed(c < 0x20 ? error_code::string_control_character : error_code::string_utf8, p); return failed(c < 0x20 ? error_code::string_control_character : error_code::string_utf8, p);
} }
++p; ++p;
if (p == m_end) if (p == end)
{ {
return failed(error_code::string_missing_quote, p); return failed(error_code::string_missing_quote, p);
} }
@@ -315,7 +340,7 @@ class builder
{ {
return failed(error_code::string_surrogate_low, p); return failed(error_code::string_surrogate_low, p);
} }
if (m_end - p < 2 || p[0] != '\\' || p[1] != 'u') if (end - p < 2 || p[0] != '\\' || p[1] != 'u')
{ {
return failed(error_code::string_surrogate_high, p); return failed(error_code::string_surrogate_high, p);
} }
@@ -331,26 +356,106 @@ class builder
} }
cp = 0x10000 + ((cp - 0xD800) << 10) + (lo - 0xDC00); cp = 0x10000 + ((cp - 0xD800) << 10) + (lo - 0xDC00);
} }
arena_ensure(4); arena_ensure(w, w_end, 4);
m_arena_write = put_utf8(m_arena_write, cp); w = put_utf8(w, cp);
} }
else if (NLOHMANN_VIEW_LIKELY(d < 128 && simple_escape[d] != 0)) else if (NLOHMANN_VIEW_LIKELY(d < 128 && simple_escape[d] != 0))
{ {
arena_ensure(1); arena_ensure(w, w_end, 1);
*m_arena_write++ = simple_escape[d]; *w++ = simple_escape[d];
} }
else else
{ {
--p; --p;
return failed(error_code::string_escape, p); return failed(error_code::string_escape, p);
} }
if (p != m_end && *p == '\\') #if NLOHMANN_VIEW_NEON
if (p != end && *p == '\\')
{ {
continue; // consecutive escapes ("\u00e4\u00f6"): no run in between continue; // consecutive escapes ("\u00e4\u00f6"): no run in between
} }
// The first 16 bytes of the run are checked one by one, in predicted branches, and copied
// as one block: where the runs between escapes are short, the vector step below would put
// its longer dependency chain (load, compare, mask, count, add) on p for each of them,
// and be slower than these steps (the scanner of the keys works the same way)
{
if (end - p >= 16)
{
const std::uint8_t* const plain = string_plain();
std::size_t steps = 16;
#define NLOHMANN_VIEW_STEP(i) if (NLOHMANN_VIEW_LIKELY(plain[p[i]] != 0)) {} else { steps = (i); goto run_found; }
NLOHMANN_VIEW_REPEAT16(NLOHMANN_VIEW_STEP)
#undef NLOHMANN_VIEW_STEP
run_found:
arena_ensure(w, w_end, 16);
std::memcpy(w, p, 16); // (only the first `steps` bytes are kept)
w += steps;
p += steps;
if (steps < 16)
{
if (*p >= 0x80)
{
// non-ASCII: the UTF-8 check, and the rest of the run
const unsigned char* const r = p;
p = scan_string_run(p, end);
const auto n = static_cast<std::size_t>(p - r);
arena_ensure(w, w_end, n);
std::memcpy(w, r, n);
w += n;
}
continue;
}
}
}
#else
if (p != end && (*p == '\\' || *p == '"'))
{
continue; // consecutive escapes ("\u00e4\u00f6"), or the end of the string: no run in between
}
#endif
#if NLOHMANN_VIEW_VECTOR
// the plain bytes up to the next quote, backslash, or control
// character are copied as they are scanned, 16 bytes at a time
for (;;)
{
if (NLOHMANN_VIEW_UNLIKELY(end - p < 16))
{
const unsigned char* const r = p;
p = scan_string_run(p, end);
const auto n = static_cast<std::size_t>(p - r);
arena_ensure(w, w_end, n);
std::memcpy(w, r, n);
w += n;
break;
}
arena_ensure(w, w_end, 16);
const unsigned k = copy_plain_run16(p, w);
w += k;
p += k;
if (k == 16)
{
continue;
}
if (NLOHMANN_VIEW_UNLIKELY(*p >= 0x80))
{
// non-ASCII: the UTF-8 check, and the rest of the run
const unsigned char* const r = p;
p = scan_string_run(p, end);
const auto n = static_cast<std::size_t>(p - r);
arena_ensure(w, w_end, n);
std::memcpy(w, r, n);
w += n;
}
break;
}
#else
const unsigned char* const r = p; const unsigned char* const r = p;
p = scan_string_run(p, m_end); p = scan_string_run(p, end);
arena_run(r, static_cast<std::size_t>(p - r)); const auto n = static_cast<std::size_t>(p - r);
arena_ensure(w, w_end, n);
std::memcpy(w, r, n);
w += n;
#endif
} }
} }
@@ -392,19 +497,49 @@ class builder
char* m_arena_write = nullptr; char* m_arena_write = nullptr;
char* m_arena_end = nullptr; char* m_arena_end = nullptr;
/// room for n more bytes at w, the write position kept in a local, with
/// w_end the end of the arena. (A lambda instead is not always inlined,
/// and then w would be kept in memory, as m_arena_write is.)
NLOHMANN_VIEW_ALWAYS_INLINE void arena_ensure(char*& w, const char*& w_end, std::size_t n)
{
if (NLOHMANN_VIEW_UNLIKELY(static_cast<std::size_t>(w_end - w) < n))
{
m_arena_write = w;
arena_grow(n);
w = m_arena_write;
w_end = m_arena_end;
}
}
/// stores a write position kept in a local back into m_arena_write when
/// the function that keeps it returns, on every path
class arena_write_back
{
public:
arena_write_back(builder& b, char*& w) noexcept
: m_builder(b)
, m_w(w)
{}
~arena_write_back()
{
m_builder.m_arena_write = m_w;
}
arena_write_back(const arena_write_back&) = delete;
arena_write_back& operator=(const arena_write_back&) = delete;
arena_write_back(arena_write_back&&) = delete;
arena_write_back& operator=(arena_write_back&&) = delete;
private:
builder& m_builder;
char*& m_w;
};
NLOHMANN_VIEW_ALWAYS_INLINE std::size_t arena_used() const noexcept NLOHMANN_VIEW_ALWAYS_INLINE std::size_t arena_used() const noexcept
{ {
return m_arena_write != nullptr ? static_cast<std::size_t>(m_arena_write - m_doc.arena.data()) : 0; return m_arena_write != nullptr ? static_cast<std::size_t>(m_arena_write - m_doc.arena.data()) : 0;
} }
NLOHMANN_VIEW_ALWAYS_INLINE void arena_ensure(std::size_t n)
{
if (NLOHMANN_VIEW_UNLIKELY(static_cast<std::size_t>(m_arena_end - m_arena_write) < n))
{
arena_grow(n);
}
}
NLOHMANN_VIEW_NOINLINE void arena_grow(std::size_t n) NLOHMANN_VIEW_NOINLINE void arena_grow(std::size_t n)
{ {
const std::size_t used = arena_used(); const std::size_t used = arena_used();
@@ -413,25 +548,6 @@ class builder
m_arena_end = m_doc.arena.data() + m_doc.arena.size(); m_arena_end = m_doc.arena.data() + m_doc.arena.size();
} }
/// append the run [r, r + n) to the arena; short runs as one fixed-size
/// 16-byte move when both sides have the room (no library call)
NLOHMANN_VIEW_ALWAYS_INLINE void arena_run(const unsigned char* r, std::size_t n)
{
if (n <= 16 && m_end - r >= 16)
{
arena_ensure(16);
std::memcpy(m_arena_write, r, 16);
}
else if (n != 0)
{
// (an empty run leaves an empty arena unallocated, and memcpy()
// must not be passed its null pointer even for zero bytes)
arena_ensure(n);
std::memcpy(m_arena_write, r, n);
}
m_arena_write += n;
}
/// UTF-8 encoding of cp at w (room for 4 bytes); returns the position after /// UTF-8 encoding of cp at w (room for 4 bytes); returns the position after
/// it. (A returned pointer stays in a register; writing through a /// it. (A returned pointer stays in a register; writing through a
/// reference to m_arena_write would store and reload it around every byte.) /// reference to m_arena_write would store and reload it around every byte.)
@@ -954,11 +1070,11 @@ indent_done:
++p; ++p;
} }
const unsigned char* const int_start = p; const unsigned char* const int_start = p;
if (p != e && *p == '0') if (cur() == '0')
{ {
++p; ++p;
} }
else if (NLOHMANN_VIEW_LIKELY(p != e && *p >= '1' && *p <= '9')) else if (NLOHMANN_VIEW_LIKELY(cur() >= '1' && cur() <= '9'))
{ {
p = skip_digits(p + 1, e); p = skip_digits(p + 1, e);
} }
@@ -969,7 +1085,7 @@ indent_done:
const auto int_digits = static_cast<std::size_t>(p - int_start); const auto int_digits = static_cast<std::size_t>(p - int_start);
std::size_t frac_digits = 0; std::size_t frac_digits = 0;
bool is_float = false; bool is_float = false;
if (p != e && *p == '.') if (cur() == '.')
{ {
++p; ++p;
const unsigned char* const f0 = p; const unsigned char* const f0 = p;
@@ -982,20 +1098,20 @@ indent_done:
is_float = true; is_float = true;
} }
std::int64_t exponent = 0; std::int64_t exponent = 0;
if (p != e && (*p | 0x20) == 'e') if ((cur() | 0x20) == 'e')
{ {
++p; ++p;
bool exp_negative = false; bool exp_negative = false;
if (p != e && (*p == '+' || *p == '-')) if (cur() == '+' || cur() == '-')
{ {
exp_negative = *p == '-'; exp_negative = *p == '-';
++p; ++p;
} }
if (NLOHMANN_VIEW_UNLIKELY(p == e || !is_digit(*p))) if (NLOHMANN_VIEW_UNLIKELY(!is_digit(cur())))
{ {
return fail(error_code::number_after_exponent); return fail(error_code::number_after_exponent);
} }
while (p != e && is_digit(*p)) while (is_digit(cur()))
{ {
if (exponent < 100000) if (exponent < 100000)
{ {
@@ -1057,7 +1173,7 @@ indent_done:
++p; // opening quote ++p; // opening quote
const unsigned char* const s = p; const unsigned char* const s = p;
p = scan_string_run<Value>(p, e); p = scan_string_run<Value>(p, e);
if (NLOHMANN_VIEW_LIKELY(p != e && *p == '"')) if (NLOHMANN_VIEW_LIKELY(cur() == '"'))
{ {
push_node(value_t::string, 0, 0, static_cast<std::size_t>(s - b), static_cast<std::uint64_t>(p - s)); push_node(value_t::string, 0, 0, static_cast<std::size_t>(s - b), static_cast<std::uint64_t>(p - s));
++p; ++p;
include/nlohmann/detail/view/compare.hpp
+913 -80
View File
File diff suppressed because it is too large. Load diff
include/nlohmann/detail/view/edit.hpp
+41 -19
View File
@@ -51,6 +51,11 @@ struct first_true<false, Conditions...> : std::integral_constant < int, 1 + firs
inline void check_utf8(const char* s, std::size_t n) inline void check_utf8(const char* s, std::size_t n)
{ {
const auto* const p = reinterpret_cast<const unsigned char*>(s); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast) const auto* const p = reinterpret_cast<const unsigned char*>(s); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
const auto hex = [](unsigned char c)
{
constexpr const char* digits = "0123456789ABCDEF";
return std::string{digits[c >> 4u], digits[c & 0xFu]};
};
for (std::size_t i = 0; i < n;) for (std::size_t i = 0; i < n;)
{ {
const unsigned char c = p[i]; const unsigned char c = p[i];
@@ -59,22 +64,39 @@ inline void check_utf8(const char* s, std::size_t n)
++i; ++i;
continue; continue;
} }
const utf8_sequence_shape shape = utf8_shape(c); std::size_t len = 0;
const std::size_t len = shape.length; unsigned char lo = 0x80;
if (len == 0) unsigned char hi = 0xBF;
if (c >= 0xC2 && c <= 0xDF)
{ {
throw_type_error(exception_id::invalid_utf8, concat("invalid UTF-8 byte at index ", std::to_string(i), ": 0x", hex_byte(c))); len = 2;
}
else if (c >= 0xE0 && c <= 0xEF)
{
len = 3;
lo = c == 0xE0 ? 0xA0 : 0x80;
hi = c == 0xED ? 0x9F : 0xBF;
}
else if (c >= 0xF0 && c <= 0xF4)
{
len = 4;
lo = c == 0xF0 ? 0x90 : 0x80;
hi = c == 0xF4 ? 0x8F : 0xBF;
}
else
{
throw_type_error(316, concat("invalid UTF-8 byte at index ", std::to_string(i), ": 0x", hex(c)));
} }
for (std::size_t k = 1; k < len; ++k) for (std::size_t k = 1; k < len; ++k)
{ {
if (i + k == n) if (i + k == n)
{ {
throw_type_error(exception_id::invalid_utf8, concat("incomplete UTF-8 string; last byte: 0x", hex_byte(p[n - 1]))); throw_type_error(316, concat("incomplete UTF-8 string; last byte: 0x", hex(p[n - 1])));
} }
const unsigned char b = p[i + k]; const unsigned char b = p[i + k];
if (b < (k == 1 ? shape.second_lo : 0x80) || b > (k == 1 ? shape.second_hi : 0xBF)) if (b < (k == 1 ? lo : 0x80) || b > (k == 1 ? hi : 0xBF))
{ {
throw_type_error(exception_id::invalid_utf8, concat("invalid UTF-8 byte at index ", std::to_string(i + k), ": 0x", hex_byte(b))); throw_type_error(316, concat("invalid UTF-8 byte at index ", std::to_string(i + k), ": 0x", hex(b)));
} }
} }
i += len; i += len;
@@ -122,7 +144,7 @@ class editor
node* const o = own(object); node* const o = own(object);
if (o->kind != static_cast<std::uint8_t>(value_t::object) && o->kind != static_cast<std::uint8_t>(value_t::null)) if (o->kind != static_cast<std::uint8_t>(value_t::object) && o->kind != static_cast<std::uint8_t>(value_t::null))
{ {
throw_type_error(exception_id::subscript_wrong_type, "cannot use operator[] with a string argument with ", object.type_name()); throw_type_error(305, "cannot use operator[] with a string argument with ", object.type_name());
} }
check_utf8(key.data(), key.size()); check_utf8(key.data(), key.size());
const encoded e = encode(std::forward<V>(value)); const encoded e = encode(std::forward<V>(value));
@@ -173,7 +195,7 @@ class editor
node* const a = own(array); node* const a = own(array);
if (a->kind != static_cast<std::uint8_t>(value_t::array)) if (a->kind != static_cast<std::uint8_t>(value_t::array))
{ {
throw_type_error(exception_id::subscript_wrong_type, "cannot use operator[] with a numeric argument with ", array.type_name()); throw_type_error(305, "cannot use operator[] with a numeric argument with ", array.type_name());
} }
check_index(idx, a->len); check_index(idx, a->len);
const encoded e = encode(std::forward<V>(value)); const encoded e = encode(std::forward<V>(value));
@@ -190,7 +212,7 @@ class editor
node* const a = own(array); node* const a = own(array);
if (a->kind != static_cast<std::uint8_t>(value_t::array) && a->kind != static_cast<std::uint8_t>(value_t::null)) if (a->kind != static_cast<std::uint8_t>(value_t::array) && a->kind != static_cast<std::uint8_t>(value_t::null))
{ {
throw_type_error(exception_id::push_back_wrong_type, "cannot use push_back() with ", array.type_name()); throw_type_error(308, "cannot use push_back() with ", array.type_name());
} }
const encoded e = encode(std::forward<V>(value)); const encoded e = encode(std::forward<V>(value));
if (a->kind == static_cast<std::uint8_t>(value_t::null)) if (a->kind == static_cast<std::uint8_t>(value_t::null))
@@ -214,7 +236,7 @@ class editor
node* const a = own(array); node* const a = own(array);
if (a->kind != static_cast<std::uint8_t>(value_t::array)) if (a->kind != static_cast<std::uint8_t>(value_t::array))
{ {
throw_type_error(exception_id::insert_wrong_type, "cannot use insert() with ", array.type_name()); throw_type_error(309, "cannot use insert() with ", array.type_name());
} }
check_index(idx, a->len + 1); check_index(idx, a->len + 1);
const encoded e = encode(std::forward<V>(value)); const encoded e = encode(std::forward<V>(value));
@@ -234,7 +256,7 @@ class editor
node* const o = own(object); node* const o = own(object);
if (o->kind != static_cast<std::uint8_t>(value_t::object)) if (o->kind != static_cast<std::uint8_t>(value_t::object))
{ {
throw_type_error(exception_id::erase_wrong_type, "cannot use erase() with ", object.type_name()); throw_type_error(307, "cannot use erase() with ", object.type_name());
} }
for (const node* k = nav::first(m_doc, o), *end = nav::end(m_doc, o); k != end; k = document_data::after(k + 1)) for (const node* k = nav::first(m_doc, o), *end = nav::end(m_doc, o); k != end; k = document_data::after(k + 1))
{ {
@@ -252,7 +274,7 @@ class editor
node* const a = own(array); node* const a = own(array);
if (a->kind != static_cast<std::uint8_t>(value_t::array)) if (a->kind != static_cast<std::uint8_t>(value_t::array))
{ {
throw_type_error(exception_id::erase_wrong_type, "cannot use erase() with ", array.type_name()); throw_type_error(307, "cannot use erase() with ", array.type_name());
} }
check_index(idx, a->len); check_index(idx, a->len);
node* const h = block_of(m_doc, a, 0); node* const h = block_of(m_doc, a, 0);
@@ -275,7 +297,7 @@ class editor
{ {
if (NLOHMANN_VIEW_UNLIKELY(v.m_doc != &m_doc || v.m_node == nullptr)) if (NLOHMANN_VIEW_UNLIKELY(v.m_doc != &m_doc || v.m_node == nullptr))
{ {
throw_invalid_iterator(exception_id::iterator_from_other_value, "view does not belong to this document"); throw_invalid_iterator(202, "view does not belong to this document");
} }
edit_state_of(m_doc); edit_state_of(m_doc);
return const_cast<node*>(v.m_node); // NOLINT(cppcoreguidelines-pro-type-const-cast): the nodes belong to this document return const_cast<node*>(v.m_node); // NOLINT(cppcoreguidelines-pro-type-const-cast): the nodes belong to this document
@@ -285,7 +307,7 @@ class editor
{ {
if (idx >= limit) if (idx >= limit)
{ {
throw_array_index_out_of_range(idx); throw_out_of_range(401, concat("array index ", std::to_string(idx), " is out of range"));
} }
} }
@@ -426,7 +448,7 @@ class editor
{ {
if (NLOHMANN_VIEW_UNLIKELY(v.m_node == nullptr)) if (NLOHMANN_VIEW_UNLIKELY(v.m_node == nullptr))
{ {
throw_type_error(exception_id::type_mismatch, "type must be a value, but is ", "discarded"); throw_type_error(302, "type must be a value, but is ", "discarded");
} }
encoded r; encoded r;
if (!is_container(*v.m_node)) if (!is_container(*v.m_node))
@@ -576,7 +598,7 @@ class editor
{ {
if (NLOHMANN_VIEW_UNLIKELY(len >= 0xFFFFFFFFu)) if (NLOHMANN_VIEW_UNLIKELY(len >= 0xFFFFFFFFu))
{ {
throw_out_of_range(exception_id::document_too_large, "strings of 4 GiB or more are not supported by json_document"); // LCOV_EXCL_LINE throw_out_of_range(416, "strings of 4 GiB or more are not supported by json_document"); // LCOV_EXCL_LINE
} }
node n = plain_node(value_t::string); node n = plain_node(value_t::string);
n.flags = static_cast<std::uint8_t>(n.flags | node_flags::edited); n.flags = static_cast<std::uint8_t>(n.flags | node_flags::edited);
@@ -651,12 +673,12 @@ class editor
return string_node(s.data(), s.size()); return string_node(s.data(), s.size());
} }
case value_t::binary: case value_t::binary:
throw_type_error(exception_id::document_binary_value, "cannot store a binary value in a json_document", ""); throw_type_error(319, "cannot store a binary value in a json_document", "");
case value_t::discarded: case value_t::discarded:
case value_t::object: case value_t::object:
case value_t::array: case value_t::array:
default: default:
throw_type_error(exception_id::type_mismatch, "type must be a value, but is ", "discarded"); throw_type_error(302, "type must be a value, but is ", "discarded");
} }
} }
include/nlohmann/detail/view/edit_storage.hpp
+2 -2
View File
@@ -84,7 +84,7 @@ inline std::size_t text_capacity(std::size_t cap, std::size_t used, std::size_t
constexpr std::size_t limit = 0xFFFFFFFFu; constexpr std::size_t limit = 0xFFFFFFFFu;
if (NLOHMANN_VIEW_UNLIKELY(used > limit || n > limit - used)) if (NLOHMANN_VIEW_UNLIKELY(used > limit || n > limit - used))
{ {
throw_out_of_range(exception_id::document_too_large, "edits of 4 GiB or more are not supported by json_document"); throw_out_of_range(416, "edits of 4 GiB or more are not supported by json_document");
} }
const std::size_t needed = used + n; const std::size_t needed = used + n;
const std::size_t wanted = needed + (std::min)(limit - needed, std::size_t{256}); const std::size_t wanted = needed + (std::min)(limit - needed, std::size_t{256});
@@ -135,7 +135,7 @@ inline void reserve_moved(document_data& d)
document_data::edit_state& e = edit_state_of(d); document_data::edit_state& e = edit_state_of(d);
if (e.moved.size() >= 0xFFFFFFFFu) if (e.moved.size() >= 0xFFFFFFFFu)
{ {
throw_out_of_range(exception_id::document_too_large, "more than 4294967295 edited arrays and objects are not supported by json_document"); // LCOV_EXCL_LINE throw_out_of_range(416, "more than 4294967295 edited arrays and objects are not supported by json_document"); // LCOV_EXCL_LINE
} }
if (e.moved.size() == e.moved.capacity() || e.moved_cap.size() == e.moved_cap.capacity()) if (e.moved.size() == e.moved.capacity() || e.moved_cap.size() == e.moved_cap.capacity())
{ {
include/nlohmann/detail/view/errors.hpp
+7 -27
View File
@@ -25,52 +25,32 @@ namespace view
// Exceptions are thrown out of line, so that the accessors that may throw stay // Exceptions are thrown out of line, so that the accessors that may throw stay
// small enough to be inlined. // small enough to be inlined.
[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_type_error(exception_id id, const char* prefix, const char* type) [[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_type_error(int id, const char* prefix, const char* type)
{ {
NLOHMANN_VIEW_THROW(type_error::create(id, concat(prefix, type), nullptr)); NLOHMANN_VIEW_THROW(type_error::create(id, concat(prefix, type), nullptr));
} }
[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_type_error(exception_id id, const std::string& msg) [[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_type_error(int id, const std::string& msg)
{ {
NLOHMANN_VIEW_THROW(type_error::create(id, msg, nullptr)); NLOHMANN_VIEW_THROW(type_error::create(id, msg, nullptr));
} }
[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_out_of_range(exception_id id, const std::string& msg) [[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_out_of_range(int id, const std::string& msg)
{ {
NLOHMANN_VIEW_THROW(out_of_range::create(id, msg, nullptr)); NLOHMANN_VIEW_THROW(out_of_range::create(id, msg, nullptr));
} }
[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_invalid_iterator(exception_id id, const char* msg) [[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_invalid_iterator(int id, const char* msg)
{ {
NLOHMANN_VIEW_THROW(invalid_iterator::create(id, msg, nullptr)); NLOHMANN_VIEW_THROW(invalid_iterator::create(id, msg, nullptr));
} }
/// a parse error without a position (as those of json_pointer) /// a parse error without a position (as those of json_pointer)
[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_parse_error(exception_id id, const std::string& msg) [[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_parse_error(int id, const std::string& msg)
{ {
NLOHMANN_VIEW_THROW(parse_error::create(id, 0, msg, nullptr)); NLOHMANN_VIEW_THROW(parse_error::create(id, 0, msg, nullptr));
} }
/// out_of_range.401: the array index @a idx is not below the size
template<typename Index>
[[noreturn]] NLOHMANN_VIEW_NOINLINE void throw_array_index_out_of_range(const Index idx)
{
throw_out_of_range(exception_id::array_index_out_of_range, concat("array index ", std::to_string(idx), " is out of range"));
}
/// out_of_range.402: the JSON Pointer token '-' of an array of @a size elements
[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_array_index_past_end(std::size_t size)
{
throw_out_of_range(exception_id::pointer_past_the_end_index, concat("array index '-' (", std::to_string(size), ") is out of range"));
}
/// out_of_range.403: the object has no member @a key
template<typename Key>
[[noreturn]] NLOHMANN_VIEW_NOINLINE void throw_key_not_found(const Key& key)
{
throw_out_of_range(exception_id::key_not_found, concat("key '", key, "' not found"));
}
/// SAX handler of the error replay: accepts every event without storing it, /// SAX handler of the error replay: accepts every event without storing it,
/// and throws the exception the parser reports /// and throws the exception the parser reports
template<typename BasicJsonType> template<typename BasicJsonType>
@@ -169,7 +149,7 @@ template<typename BasicJsonType>
if (f.code == error_code::input_too_large) if (f.code == error_code::input_too_large)
{ {
// (the limit is detail::view::max_input_size(): 4 GiB minus 16 bytes) // (the limit is detail::view::max_input_size(): 4 GiB minus 16 bytes)
NLOHMANN_VIEW_THROW(out_of_range::create(exception_id::document_too_large, "input of 4294967280 bytes or more is not supported by json_document", nullptr)); NLOHMANN_VIEW_THROW(out_of_range::create(416, "input of 4294967280 bytes or more is not supported by json_document", nullptr));
} }
throwing_sax<BasicJsonType> sax; throwing_sax<BasicJsonType> sax;
const bool accepted = BasicJsonType::sax_parse(src, src + size, &sax, input_format_t::json, true, ignore_comments, ignore_trailing_commas); const bool accepted = BasicJsonType::sax_parse(src, src + size, &sax, input_format_t::json, true, ignore_comments, ignore_trailing_commas);
@@ -189,7 +169,7 @@ template<typename BasicJsonType>
} }
} }
pos.chars_read_current_line = off + 1 - line_start; pos.chars_read_current_line = off + 1 - line_start;
NLOHMANN_VIEW_THROW(parse_error::create(exception_id::syntax_error, pos, "syntax error while parsing value", nullptr)); NLOHMANN_VIEW_THROW(parse_error::create(101, pos, "syntax error while parsing value", nullptr));
// LCOV_EXCL_STOP // LCOV_EXCL_STOP
} }
include/nlohmann/detail/view/image.hpp
+104 -22
View File
@@ -186,7 +186,7 @@ inline void compact_nodes(const document_data& d, std::size_t arena_size, std::v
inline std::vector<std::uint8_t> save_image(const document_data& d) inline std::vector<std::uint8_t> save_image(const document_data& d)
{ {
#if !NLOHMANN_VIEW_LITTLE_ENDIAN #if !NLOHMANN_VIEW_LITTLE_ENDIAN
throw_type_error(exception_id::document_image_unsupported, "json_document images need a little-endian target"); // LCOV_EXCL_LINE throw_type_error(320, "json_document images need a little-endian target"); // LCOV_EXCL_LINE
#endif #endif
const std::size_t arena_size = d.arena_size; const std::size_t arena_size = d.arena_size;
std::vector<node> compacted; std::vector<node> compacted;
@@ -203,7 +203,7 @@ inline std::vector<std::uint8_t> save_image(const document_data& d)
if (NLOHMANN_VIEW_UNLIKELY(text_size >= image_limit || total_arena >= image_limit || count >= image_limit)) if (NLOHMANN_VIEW_UNLIKELY(text_size >= image_limit || total_arena >= image_limit || count >= image_limit))
{ {
// LCOV_EXCL_START (4 GiB) // LCOV_EXCL_START (4 GiB)
throw_out_of_range(exception_id::document_too_large, "images of 4 GiB or more are not supported by json_document"); throw_out_of_range(416, "images of 4 GiB or more are not supported by json_document");
// LCOV_EXCL_STOP // LCOV_EXCL_STOP
} }
image_header h{}; image_header h{};
@@ -255,6 +255,81 @@ struct number_token
bool is_float; ///< a fraction or an exponent bool is_float; ///< a fraction or an exponent
}; };
/// advance over ASCII digits, eight bytes at a time
inline const unsigned char* digit_run(const unsigned char* p, const unsigned char* e) noexcept
{
constexpr std::uint64_t ones = 0x0101010101010101ull;
while (e - p >= 8)
{
// (the digits become 0..9; a byte above 9, or with bit 7 set, is no digit. A carry
// out of a byte with bit 7 set only affects the bytes above, after the first one
// that count_trailing_zeros finds.)
const std::uint64_t x = read_eight_bytes(p) ^ (ones * 0x30u);
const std::uint64_t not_digit = ((x + (ones * 0x76u)) | x) & (ones * 0x80u);
if (not_digit != 0)
{
return p + (static_cast<std::size_t>(count_trailing_zeros(not_digit)) / 8);
}
p += 8;
}
while (p != e && is_digit(*p))
{
++p;
}
return p;
}
/// whether the n bytes at p are ASCII digits (n > 0)
inline bool all_digits(const unsigned char* p, std::size_t n) noexcept
{
const auto digits = [](const unsigned char* q)
{
constexpr std::uint64_t ones = 0x0101010101010101ull;
const std::uint64_t x = read_eight_bytes(q) ^ (ones * 0x30u);
return (((x + (ones * 0x76u)) | x) & (ones * 0x80u)) == 0;
};
if (n >= 8)
{
for (std::size_t i = 0; i + 8 < n; i += 8)
{
if (!digits(p + i))
{
return false;
}
}
return digits(p + n - 8); // (overlapping the bytes before: they are digits)
}
for (std::size_t i = 0; i < n; ++i)
{
if (!is_digit(p[i]))
{
return false;
}
}
return true;
}
/// Whether a float token has the shape that the parser records for most of
/// them: [-]digits.digits, with the numbers of digits of the node's layout
/// (extra). Then the token is a number of that layout, and the general scan
/// that follows is not needed.
inline bool plain_float_token(const unsigned char* s, std::size_t len, std::uint16_t extra) noexcept
{
const std::size_t int_digits = extra & 0xFFu;
const std::size_t frac_digits = extra >> 8u;
if (int_digits == 0 || int_digits >= 255 || frac_digits == 0 || frac_digits >= 255 || len == 0)
{
return false;
}
const std::size_t negative = *s == '-' ? 1 : 0;
if (len != negative + int_digits + 1 + frac_digits)
{
return false;
}
const unsigned char* const p = s + negative;
return !(int_digits > 1 && *p == '0') && p[int_digits] == '.' && all_digits(p, int_digits) && all_digits(p + int_digits + 1, frac_digits);
}
/// whether [s, s + len) is a JSON number (the grammar the parser accepts) /// whether [s, s + len) is a JSON number (the grammar the parser accepts)
inline bool scan_number_token(const unsigned char* s, std::size_t len, number_token& t) inline bool scan_number_token(const unsigned char* s, std::size_t len, number_token& t)
{ {
@@ -273,11 +348,7 @@ inline bool scan_number_token(const unsigned char* s, std::size_t len, number_to
} }
else if (*p >= '1' && *p <= '9') else if (*p >= '1' && *p <= '9')
{ {
do p = digit_run(p + 1, e);
{
++p;
}
while (p != e && is_digit(*p));
} }
else else
{ {
@@ -289,10 +360,7 @@ inline bool scan_number_token(const unsigned char* s, std::size_t len, number_to
if (p != e && *p == '.') if (p != e && *p == '.')
{ {
const unsigned char* const f0 = ++p; const unsigned char* const f0 = ++p;
while (p != e && is_digit(*p)) p = digit_run(p, e);
{
++p;
}
if (p == f0) if (p == f0)
{ {
return false; return false;
@@ -403,10 +471,15 @@ struct float_range
template<typename Check> template<typename Check>
bool check_distinct_ranges(std::vector<byte_range>& ranges, Check check) bool check_distinct_ranges(std::vector<byte_range>& ranges, Check check)
{ {
std::sort(ranges.begin(), ranges.end(), [](const byte_range & a, const byte_range & b) // (save() writes the ranges in the order of the nodes, which is sorted already)
const auto before = [](const byte_range & a, const byte_range & b)
{ {
return a.off != b.off ? a.off < b.off : a.len < b.len; return a.off != b.off ? a.off < b.off : a.len < b.len;
}); };
if (!std::is_sorted(ranges.begin(), ranges.end(), before))
{
std::sort(ranges.begin(), ranges.end(), before);
}
std::size_t end = 0; std::size_t end = 0;
for (std::size_t i = 0; i < ranges.size(); ++i) for (std::size_t i = 0; i < ranges.size(); ++i)
{ {
@@ -428,7 +501,8 @@ bool check_distinct_ranges(std::vector<byte_range>& ranges, Check check)
/// record layouts that match it), tokens must not overlap. /// record layouts that match it), tokens must not overlap.
inline bool check_float_ranges(std::vector<float_range>& ranges, const unsigned char* text) inline bool check_float_ranges(std::vector<float_range>& ranges, const unsigned char* text)
{ {
std::sort(ranges.begin(), ranges.end(), [](const float_range & a, const float_range & b) // (save() writes the tokens in the order of the nodes, which is sorted already)
const auto before = [](const float_range & a, const float_range & b)
{ {
if (a.off != b.off) if (a.off != b.off)
{ {
@@ -439,7 +513,11 @@ inline bool check_float_ranges(std::vector<float_range>& ranges, const unsigned
return a.len < b.len; return a.len < b.len;
} }
return a.extra < b.extra; return a.extra < b.extra;
}); };
if (!std::is_sorted(ranges.begin(), ranges.end(), before))
{
std::sort(ranges.begin(), ranges.end(), before);
}
std::size_t end = 0; std::size_t end = 0;
std::size_t i = 0; std::size_t i = 0;
while (i < ranges.size()) while (i < ranges.size())
@@ -449,14 +527,18 @@ inline bool check_float_ranges(std::vector<float_range>& ranges, const unsigned
{ {
return false; return false;
} }
number_token t{};
const unsigned char* const s = text + r.off; const unsigned char* const s = text + r.off;
if (!scan_number_token(s, r.len, t) || !float_token_finite(s, r.len, t)) std::uint16_t layout = r.extra;
if (!plain_float_token(s, r.len, r.extra))
{ {
return false; number_token t{};
if (!scan_number_token(s, r.len, t) || !float_token_finite(s, r.len, t))
{
return false;
}
// the digit layout the parser records (or "many", as compaction writes it)
layout = float_layout(t);
} }
// the digit layout the parser records (or "many", as compaction writes it)
const std::uint16_t layout = float_layout(t);
for (; i < ranges.size() && ranges[i].off == r.off && ranges[i].len == r.len; ++i) for (; i < ranges.size() && ranges[i].off == r.off && ranges[i].len == r.len; ++i)
{ {
if (ranges[i].extra != layout && ranges[i].extra != 0xFFFFu) if (ranges[i].extra != layout && ranges[i].extra != 0xFFFFu)
@@ -660,7 +742,7 @@ inline bool check_image(const node* nodes, std::size_t count, const unsigned cha
[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_invalid_image(const char* what) [[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_invalid_image(const char* what)
{ {
throw_parse_error(exception_id::invalid_document_image, concat("invalid json_document image: ", what)); throw_parse_error(116, concat("invalid json_document image: ", what));
} }
/// Read an image into d. The text and the decoded strings stay in the image; /// Read an image into d. The text and the decoded strings stay in the image;
@@ -668,7 +750,7 @@ inline bool check_image(const node* nodes, std::size_t count, const unsigned cha
inline void load_image(document_data& d, const std::uint8_t* image, std::size_t size, image_check check) inline void load_image(document_data& d, const std::uint8_t* image, std::size_t size, image_check check)
{ {
#if !NLOHMANN_VIEW_LITTLE_ENDIAN #if !NLOHMANN_VIEW_LITTLE_ENDIAN
throw_type_error(exception_id::document_image_unsupported, "json_document images need a little-endian target"); // LCOV_EXCL_LINE throw_type_error(320, "json_document images need a little-endian target"); // LCOV_EXCL_LINE
#endif #endif
if (image == nullptr || size < sizeof(image_header)) if (image == nullptr || size < sizeof(image_header))
{ {
include/nlohmann/detail/view/input.hpp
+34
View File
@@ -10,6 +10,9 @@
#include <algorithm> // max #include <algorithm> // max
#include <cstddef> // size_t #include <cstddef> // size_t
#ifndef JSON_NO_IO
#include <cstdio> // FILE, fread
#endif
#include <istream> // istream, streambuf #include <istream> // istream, streambuf
#include <string> // basic_string, char_traits, string #include <string> // basic_string, char_traits, string
#include <type_traits> // decay, integral_constant, is_array, is_const, is_integral, is_lvalue_reference, is_pointer, is_same, remove_cv, remove_extent, remove_pointer, remove_reference #include <type_traits> // decay, integral_constant, is_array, is_const, is_integral, is_lvalue_reference, is_pointer, is_same, remove_cv, remove_extent, remove_pointer, remove_reference
@@ -154,6 +157,37 @@ inline void append_stream(std::string& buf, std::istream& is)
is.clear(is.rdstate() | std::ios::eofbit); is.clear(is.rdstate() | std::ios::eofbit);
} }
/// a pointer to a C stream (never with JSON_NO_IO, which leaves out <cstdio>)
template<typename T>
struct is_c_stream
#ifndef JSON_NO_IO
: std::is_same<T, std::FILE*>
#else
: std::false_type
#endif
{};
#ifndef JSON_NO_IO
/// append the rest of a C stream: in blocks with fread, not character by
/// character (the library's file adapter reads with fgetc);
/// f is not null
inline void append_file(std::string& buf, std::FILE* f)
{
for (;;)
{
const std::size_t used = buf.size();
const std::size_t room = (std::max)(buf.capacity() - used, std::size_t{4096});
buf.resize(used + room);
const std::size_t got = std::fread(&buf[used], 1, room, f);
buf.resize(used + got);
if (got < room)
{
break; // end of file, or an error (as fgetc, which returns EOF for both)
}
}
}
#endif
} // namespace view } // namespace view
} // namespace detail } // namespace detail
NLOHMANN_JSON_NAMESPACE_END NLOHMANN_JSON_NAMESPACE_END
include/nlohmann/detail/view/iterator.hpp
+11 -9
View File
@@ -110,7 +110,7 @@ class view_iterator
{ {
if (NLOHMANN_VIEW_UNLIKELY(m_value_offset == 0)) if (NLOHMANN_VIEW_UNLIKELY(m_value_offset == 0))
{ {
throw_invalid_iterator(exception_id::iterator_key_not_object, "cannot use key() for non-object iterators"); throw_invalid_iterator(207, "cannot use key() for non-object iterators");
} }
return string_view_t(m_doc->str(*m_pos), m_pos->len); return string_view_t(m_doc->str(*m_pos), m_pos->len);
} }
@@ -147,20 +147,21 @@ class view_item
view_item(const iterator& it, std::size_t index) view_item(const iterator& it, std::size_t index)
: m_it(it) : m_it(it)
{ , m_position(index)
if (!it.is_object_iterator()) {}
{
m_index = std::to_string(index);
}
}
/// the member key, or the element index for arrays /// the member key, or the element index for arrays (as text made on the
/// first call, so that iterating over the values pays nothing for it)
string_view_t key() const string_view_t key() const
{ {
if (m_it.is_object_iterator()) if (m_it.is_object_iterator())
{ {
return m_it.key(); return m_it.key();
} }
if (m_index.empty())
{
m_index = std::to_string(m_position);
}
return string_view_t(m_index.data(), m_index.size()); return string_view_t(m_index.data(), m_index.size());
} }
@@ -183,7 +184,8 @@ class view_item
private: private:
iterator m_it; iterator m_it;
std::string m_index{}; // NOLINT(readability-redundant-member-init) std::size_t m_position; ///< the index of an array element
mutable std::string m_index{}; ///< the text of m_position, once key() was called // NOLINT(readability-redundant-member-init)
}; };
/// the range returned by basic_json_view::items() /// the range returned by basic_json_view::items()
include/nlohmann/detail/view/lookup.hpp
+8
View File
@@ -166,6 +166,10 @@ const node* element_at(const document_data& d, const node* array, std::size_t id
{ {
return e + idx; // a growable block: one link per element return e + idx; // a growable block: one link per element
} }
if (array->next == array->len + 1u && (!nav::editable() || (array->flags & node_flags::moved) == 0))
{
return e + idx; // no element has a subtree: the elements are the nodes after the array
}
for (std::size_t i = 0; i < idx; ++i) for (std::size_t i = 0; i < idx; ++i)
{ {
e = document_data::after(e); e = document_data::after(e);
@@ -181,6 +185,10 @@ const node* last_child(const document_data& d, const node* container) noexcept
const std::size_t value_offset = container->kind == static_cast<std::uint8_t>(value_t::object) ? 1 : 0; const std::size_t value_offset = container->kind == static_cast<std::uint8_t>(value_t::object) ? 1 : 0;
const node* const end = navigation<Editable>::end(d, container); const node* const end = navigation<Editable>::end(d, container);
const node* last = navigation<Editable>::first(d, container); const node* last = navigation<Editable>::first(d, container);
if (value_offset == 0 && container->next == container->len + 1u && (!navigation<Editable>::editable() || (container->flags & node_flags::moved) == 0))
{
return last + (container->len - 1u); // an array without subtrees: the elements are the nodes after it
}
for (const node* c = document_data::after(last + value_offset); c != end; c = document_data::after(c + value_offset)) for (const node* c = document_data::after(last + value_offset); c != end; c = document_data::after(c + value_offset))
{ {
last = c; last = c;
include/nlohmann/detail/view/materialize.hpp
+45 -7
View File
@@ -8,7 +8,7 @@
#pragma once #pragma once
#include <algorithm> // max #include <array> // array
#include <cstddef> // size_t #include <cstddef> // size_t
#include <cstdint> // int64_t, uint8_t #include <cstdint> // int64_t, uint8_t
#include <string> // string #include <string> // string
@@ -40,12 +40,13 @@ template<typename BasicJsonType, bool Editable>
BasicJsonType materialize(const document_data& d, const node* n) BasicJsonType materialize(const document_data& d, const node* n)
{ {
using string_t = typename BasicJsonType::string_t; using string_t = typename BasicJsonType::string_t;
using sax_t = json_sax_dom_parser<BasicJsonType, iterator_input_adapter<const char*>>; using sax_t = json_sax_dom_parser<BasicJsonType, iterator_input_adapter<const char*>, true>;
using nav = navigation<Editable>; using nav = navigation<Editable>;
// the open containers: their next element (or key of the next member), // the open containers: their next element (or key of the next member),
// their end, and whether they are objects; sized once from the // their end, and whether they are objects; for the first 16 levels on the
// document's deepest nesting (for a subtree an upper bound) // stack, else sized once from the document's deepest nesting (for a
// subtree an upper bound)
struct frame struct frame
{ {
const node* pos; const node* pos;
@@ -53,10 +54,43 @@ BasicJsonType materialize(const document_data& d, const node* n)
bool object; bool object;
}; };
// a scalar needs neither the handler nor the stack of open containers
switch (static_cast<value_t>(n->kind))
{
case value_t::string:
return BasicJsonType(string_t(d.str(*n), n->len));
case value_t::number_integer:
return BasicJsonType(static_cast<typename BasicJsonType::number_integer_t>(static_cast<std::int64_t>(integer_bits(*n))));
case value_t::number_unsigned:
return BasicJsonType(static_cast<typename BasicJsonType::number_unsigned_t>(integer_bits(*n)));
case value_t::number_float:
return BasicJsonType(float_value<typename BasicJsonType::number_float_t>(d, *n));
case value_t::boolean:
return BasicJsonType((n->flags & node_flags::is_true) != 0);
case value_t::null:
return BasicJsonType(nullptr);
case value_t::object:
case value_t::array:
case value_t::binary:
case value_t::discarded:
default:
break;
}
BasicJsonType result; BasicJsonType result;
sax_t sax(result, true); sax_t sax(result, true);
const string_t no_token{}; const string_t no_token{};
std::vector<frame> open((std::max)(d.max_depth, std::size_t{1})); // (shallow subtrees on the stack: a call for a small value then allocates only for the value)
std::array<frame, 16> shallow; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): filled as containers open
std::vector<frame> deep{};
frame* open = shallow.data();
std::size_t capacity = shallow.size();
if (d.max_depth > capacity)
{
deep.resize(d.max_depth);
open = deep.data();
capacity = deep.size();
}
std::size_t depth = 0; std::size_t depth = 0;
for (;;) for (;;)
{ {
@@ -76,9 +110,13 @@ BasicJsonType materialize(const document_data& d, const node* n)
{ {
sax.start_array(n->len); sax.start_array(n->len);
} }
if (NLOHMANN_VIEW_UNLIKELY(depth == open.size())) if (NLOHMANN_VIEW_UNLIKELY(depth == capacity))
{ {
open.resize(depth * 2); // (deeper than the recorded nesting, e.g. after edits) // (deeper than the recorded nesting, e.g. after edits)
deep.assign(open, open + depth);
deep.resize(depth * 2);
open = deep.data();
capacity = deep.size();
} }
open[depth++] = frame{nav::first(d, n), nav::end(d, n), object}; open[depth++] = frame{nav::first(d, n), nav::end(d, n), object};
break; break;
include/nlohmann/detail/view/object_index.hpp
+101 -11
View File
@@ -47,27 +47,96 @@ constexpr std::size_t index_max_displacement = 64;
static_assert(document_data::index_min_members >= 2 * index_max_displacement, static_assert(document_data::index_min_members >= 2 * index_max_displacement,
"a table must be much larger than the probe window of a lookup"); "a table must be much larger than the probe window of a lookup");
/// hash of a key: its bytes, eight at a time, in a fixed byte order /// hash of a key: its bytes, eight at a time (in a fixed byte order), where the
/// last word may overlap the one before (so that no loop over single bytes is
/// needed); a key of fewer than eight bytes is read as two overlapping words of
/// four or two bytes (in the native byte order: the hash is not stored, and the
/// same on both sides of a lookup)
inline std::uint64_t key_hash(const char* s, std::size_t n) noexcept inline std::uint64_t key_hash(const char* s, std::size_t n) noexcept
{ {
std::uint64_t h = 0x9E3779B97F4A7C15u * (n + 1); std::uint64_t h = 0x9E3779B97F4A7C15u * (n + 1);
const auto* p = reinterpret_cast<const unsigned char*>(s); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast) const auto* p = reinterpret_cast<const unsigned char*>(s); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
while (n >= 8)
{
h = (h ^ read_eight_bytes(p)) * 0xBF58476D1CE4E5B9u;
h ^= h >> 29u;
p += 8;
n -= 8;
}
std::uint64_t w = 0; std::uint64_t w = 0;
for (std::size_t i = 0; i < n; ++i) if (n >= 8)
{ {
w |= static_cast<std::uint64_t>(p[i]) << (8u * i); const unsigned char* const last = p + n - 8;
while (p < last)
{
h = (h ^ read_eight_bytes(p)) * 0xBF58476D1CE4E5B9u;
h ^= h >> 29u;
p += 8;
}
w = read_eight_bytes(last);
} }
h = (h ^ w) * 0x94D049BB133111EBu; else if (n >= 4)
{
std::uint32_t head = 0;
std::uint32_t tail = 0;
std::memcpy(&head, p, 4);
std::memcpy(&tail, p + n - 4, 4);
w = static_cast<std::uint64_t>(head) | (static_cast<std::uint64_t>(tail) << 32u);
}
else if (n >= 2)
{
std::uint16_t head = 0;
std::uint16_t tail = 0;
std::memcpy(&head, p, 2);
std::memcpy(&tail, p + n - 2, 2);
w = static_cast<std::uint64_t>(head) | (static_cast<std::uint64_t>(tail) << 16u);
}
else if (n == 1)
{
w = p[0];
}
// (a product only carries bits upwards, and the slot is taken from the low
// bits: the high bits of the last word are folded down before the multiplication)
h = (h ^ w ^ (w >> 29u)) * 0x94D049BB133111EBu;
return h ^ (h >> 31u); return h ^ (h >> 31u);
} }
/// whether the n <= 16 bytes at a and b are equal, as two overlapping words
/// (the first and the last 8, 4, or 2 bytes) and without a call
NLOHMANN_VIEW_ALWAYS_INLINE bool same_short_bytes(const unsigned char* a, const unsigned char* b, std::size_t n) noexcept
{
if (n >= 8)
{
std::uint64_t x0 = 0;
std::uint64_t x1 = 0;
std::uint64_t y0 = 0;
std::uint64_t y1 = 0;
std::memcpy(&x0, a, 8);
std::memcpy(&x1, a + n - 8, 8);
std::memcpy(&y0, b, 8);
std::memcpy(&y1, b + n - 8, 8);
return x0 == y0 && x1 == y1;
}
if (n >= 4)
{
std::uint32_t x0 = 0;
std::uint32_t x1 = 0;
std::uint32_t y0 = 0;
std::uint32_t y1 = 0;
std::memcpy(&x0, a, 4);
std::memcpy(&x1, a + n - 4, 4);
std::memcpy(&y0, b, 4);
std::memcpy(&y1, b + n - 4, 4);
return x0 == y0 && x1 == y1;
}
if (n >= 2)
{
std::uint16_t x0 = 0;
std::uint16_t x1 = 0;
std::uint16_t y0 = 0;
std::uint16_t y1 = 0;
std::memcpy(&x0, a, 2);
std::memcpy(&x1, a + n - 2, 2);
std::memcpy(&y0, b, 2);
std::memcpy(&y1, b + n - 2, 2);
return x0 == y0 && x1 == y1;
}
return n == 0 || *a == *b;
}
/// build the table of a large object /// build the table of a large object
inline void build_object_index(document_data& d, node* obj) inline void build_object_index(document_data& d, node* obj)
{ {
@@ -132,6 +201,27 @@ inline const node* find_indexed(const document_data& d, const node* obj, const c
const document_data::object_index& ix = d.indexes[obj->extra - 1u]; const document_data::object_index& ix = d.indexes[obj->extra - 1u];
const std::uint32_t* const table = d.index_slots.data() + ix.start; const std::uint32_t* const table = d.index_slots.data() + ix.start;
std::size_t slot = conditional_static_cast<std::size_t>(key_hash(key, n)) & ix.mask; std::size_t slot = conditional_static_cast<std::size_t>(key_hash(key, n)) & ix.mask;
const auto* const k_bytes = reinterpret_cast<const unsigned char*>(key); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
// (the loop for longer keys comes first, as the code is laid out; it is not the more common case)
if (NLOHMANN_VIEW_UNLIKELY(n <= 16))
{
// (up to 16 bytes, the bytes are compared as two words, without a call)
for (std::size_t distance = 0; distance <= index_max_displacement; ++distance)
{
const std::uint32_t s = table[slot];
if (s == 0)
{
return nullptr;
}
const node* const k = obj + s;
if (k->len == n && same_short_bytes(reinterpret_cast<const unsigned char*>(d.str(*k)), k_bytes, n)) // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
{
return k;
}
slot = (slot + 1) & ix.mask;
}
return nullptr;
}
for (std::size_t distance = 0; distance <= index_max_displacement; ++distance) for (std::size_t distance = 0; distance <= index_max_displacement; ++distance)
{ {
const std::uint32_t s = table[slot]; const std::uint32_t s = table[slot];
include/nlohmann/detail/view/pointer.hpp
+50 -14
View File
@@ -32,15 +32,51 @@ enum class pointer_mode
contains, ///< as basic_json::contains(): no exceptions at all contains, ///< as basic_json::contains(): no exceptions at all
}; };
/// the outcome of reading an array index: detail::parse_array_index(), shared with json_pointer /// the outcome of reading an array index from a reference token
using index_status = detail::array_index_status; enum class index_status
{
ok,
leading_zero, ///< parse_error.106
not_number, ///< parse_error.109
unresolved, ///< out_of_range.404
too_large, ///< out_of_range.410
};
/// reads an array index like json_pointer::array_index (RFC 6901, Sect. 4), /// reads an array index like json_pointer::array_index (RFC 6901, Sect. 4),
/// but reports errors instead of throwing them /// but reports errors instead of throwing them
template<typename StringType> template<typename StringType>
index_status array_index(const StringType& s, std::size_t& idx) noexcept index_status array_index(const StringType& s, std::size_t& idx) noexcept
{ {
return detail::parse_array_index(s, idx); // as basic_json: the empty token and "-" (past the last element) are
// unresolved rather than not a number
if (s.empty() || (s.size() == 1 && s[0] == '-'))
{
return index_status::unresolved;
}
if (s[0] < '0' || s[0] > '9')
{
return index_status::not_number;
}
if (s.size() > 1 && s[0] == '0')
{
return index_status::leading_zero;
}
std::uint64_t v = 0;
for (const auto c : s)
{
const auto d = static_cast<unsigned>(static_cast<unsigned char>(c)) - '0';
if (d > 9 || v > ((std::numeric_limits<std::uint64_t>::max)() - d) / 10)
{
return index_status::unresolved; // not a number, or beyond unsigned long long
}
v = (v * 10) + d;
}
if (v >= (std::numeric_limits<std::size_t>::max)()) // (std::size_t converts to std::uint64_t implicitly)
{
return index_status::too_large;
}
idx = static_cast<std::size_t>(v);
return index_status::ok;
} }
/// throws the exception json_pointer::array_index throws for this status /// throws the exception json_pointer::array_index throws for this status
@@ -50,15 +86,15 @@ template<typename StringType>
switch (status) switch (status)
{ {
case index_status::leading_zero: case index_status::leading_zero:
throw_parse_error(exception_id::pointer_index_leading_zero, concat("array index '", s, "' must not begin with '0'")); throw_parse_error(106, concat("array index '", s, "' must not begin with '0'"));
case index_status::not_a_number: case index_status::not_number:
throw_parse_error(exception_id::pointer_index_not_a_number, concat("array index '", s, "' is not a number")); throw_parse_error(109, concat("array index '", s, "' is not a number"));
case index_status::exceeds_size_type: case index_status::too_large:
throw_out_of_range(exception_id::value_out_of_range, concat("array index ", s, " exceeds size_type")); // LCOV_EXCL_LINE throw_out_of_range(410, concat("array index ", s, " exceeds size_type")); // LCOV_EXCL_LINE
case index_status::unresolved: case index_status::unresolved:
case index_status::ok: case index_status::ok:
default: default:
throw_out_of_range(exception_id::pointer_unresolved, concat("unresolved reference token '", s, "'")); throw_out_of_range(404, concat("unresolved reference token '", s, "'"));
} }
} }
@@ -83,7 +119,7 @@ View resolve_pointer(View cur, const Tokens& tokens, pointer_mode mode)
{ {
if (mode == pointer_mode::checked) if (mode == pointer_mode::checked)
{ {
throw_key_not_found(token); throw_out_of_range(403, concat("key '", token, "' not found"));
} }
return View(); return View();
} }
@@ -95,7 +131,7 @@ View resolve_pointer(View cur, const Tokens& tokens, pointer_mode mode)
{ {
if (throwing) if (throwing)
{ {
throw_array_index_past_end(cur.size()); throw_out_of_range(402, concat("array index '-' (", std::to_string(cur.size()), ") is out of range"));
} }
return View(); return View();
} }
@@ -103,7 +139,7 @@ View resolve_pointer(View cur, const Tokens& tokens, pointer_mode mode)
const index_status status = array_index(token, idx); const index_status status = array_index(token, idx);
if (status != index_status::ok) if (status != index_status::ok)
{ {
const bool parse_error = status == index_status::leading_zero || status == index_status::not_a_number; const bool parse_error = status == index_status::leading_zero || status == index_status::not_number;
if (throwing || (mode == pointer_mode::value && parse_error)) if (throwing || (mode == pointer_mode::value && parse_error))
{ {
throw_array_index_error(status, token); throw_array_index_error(status, token);
@@ -114,7 +150,7 @@ View resolve_pointer(View cur, const Tokens& tokens, pointer_mode mode)
{ {
if (mode == pointer_mode::checked) if (mode == pointer_mode::checked)
{ {
throw_array_index_out_of_range(idx); throw_out_of_range(401, concat("array index ", std::to_string(idx), " is out of range"));
} }
return View(); return View();
} }
@@ -124,7 +160,7 @@ View resolve_pointer(View cur, const Tokens& tokens, pointer_mode mode)
{ {
if (throwing) if (throwing)
{ {
throw_out_of_range(exception_id::pointer_unresolved, concat("unresolved reference token '", token, "'")); throw_out_of_range(404, concat("unresolved reference token '", token, "'"));
} }
return View(); return View();
} }
include/nlohmann/detail/view/serializer.hpp
+382 -261
View File
@@ -23,6 +23,7 @@
#include <nlohmann/detail/view/macro_scope.hpp> #include <nlohmann/detail/view/macro_scope.hpp>
#include <nlohmann/detail/view/node.hpp> #include <nlohmann/detail/view/node.hpp>
#include <nlohmann/detail/view/number.hpp> #include <nlohmann/detail/view/number.hpp>
#include <nlohmann/detail/view/simd.hpp>
NLOHMANN_JSON_NAMESPACE_BEGIN NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail namespace detail
@@ -80,13 +81,6 @@ class output_buffer
put(s, N - 1); put(s, N - 1);
} }
void put_repeated(char c, std::size_t n)
{
reserve(n);
std::memset(m_pos, c, n);
m_pos += n;
}
/// the write position and the end of the writable space, for a writer /// the write position and the end of the writable space, for a writer
/// that keeps the position in a local variable (set_cursor() hands it back) /// that keeps the position in a local variable (set_cursor() hands it back)
char* cursor() const noexcept char* cursor() const noexcept
@@ -143,9 +137,27 @@ class output_buffer
/// image_check::bounds can have others, which are then written unchanged). /// image_check::bounds can have others, which are then written unchanged).
inline std::size_t plain_output_run(const unsigned char* s, std::size_t n) noexcept inline std::size_t plain_output_run(const unsigned char* s, std::size_t n) noexcept
{ {
std::size_t i = 0;
#if NLOHMANN_VIEW_SSE2
// (16 bytes per step; with NEON, the mask of a step costs more than it saves on the
// short runs that are typical here)
// NOLINTBEGIN(portability-simd-intrinsics)
for (; i + 16 <= n; i += 16)
{
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(s + i))); // NOLINT(bugprone-casting-through-void)
// (a byte is below 0x20 if it is its own minimum with 0x1F)
const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_cmpeq_epi8(_mm_min_epu8(in, _mm_set1_epi8(0x1F)), in));
const auto mask = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
if (mask != 0)
{
return i + static_cast<std::size_t>(count_trailing_zeros(mask));
}
}
// NOLINTEND(portability-simd-intrinsics)
#endif
constexpr std::uint64_t ones = 0x0101010101010101ull; constexpr std::uint64_t ones = 0x0101010101010101ull;
constexpr std::uint64_t high = 0x8080808080808080ull; constexpr std::uint64_t high = 0x8080808080808080ull;
std::size_t i = 0;
for (; i + 8 <= n; i += 8) for (; i + 8 <= n; i += 8)
{ {
const std::uint64_t v = read_eight_bytes(s + i); const std::uint64_t v = read_eight_bytes(s + i);
@@ -196,6 +208,70 @@ inline bool is_ascii_without_delete(const char* s, std::size_t n) noexcept
return true; return true;
} }
#if NLOHMANN_VIEW_VECTOR
/// the bytes of the 16 at p that ensure_ascii cannot copy (a quote, a
/// backslash, a control character, DEL, or not ASCII; a signed compare with
/// 0x20 finds control characters and non-ASCII bytes at once), one bit per
/// byte (x86) or one nibble per byte (NEON)
NLOHMANN_VIEW_ALWAYS_INLINE std::uint64_t ascii_stop_mask(const unsigned char* p) noexcept
{
#if NLOHMANN_VIEW_NEON
const uint8x16_t in = vld1q_u8(p);
const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))),
vorrq_u8(vceqq_u8(in, vdupq_n_u8(0x7F)), vcltq_s8(vreinterpretq_s8_u8(in), vdupq_n_s8(0x20))));
return vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0);
#else
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8(0x7F)), _mm_cmplt_epi8(in, _mm_set1_epi8(0x20))));
return static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
#endif
}
/// the width in bits of a byte in the mask of ascii_stop_mask()
constexpr unsigned ascii_stop_bits() noexcept
{
return NLOHMANN_VIEW_NEON ? 4u : 1u;
}
#endif
/// As find_ascii_copyable_run(), 16 bytes per step. end is the end of the
/// buffer that holds the n bytes at p (a short run is read as 16 bytes if
/// they are in the buffer).
inline std::size_t ascii_copyable_run(const unsigned char* p, std::size_t n, const unsigned char* end) noexcept
{
#if NLOHMANN_VIEW_VECTOR
if (n >= 16)
{
std::size_t i = 0;
for (; i + 16 <= n; i += 16)
{
const std::uint64_t mask = ascii_stop_mask(p + i);
if (mask != 0)
{
return i + (static_cast<std::size_t>(count_trailing_zeros(mask)) / ascii_stop_bits());
}
}
if (i == n)
{
return n;
}
// the last 16 bytes: those before i are known to be copyable
const std::uint64_t mask = ascii_stop_mask(p + n - 16);
return mask != 0 ? n - 16 + (static_cast<std::size_t>(count_trailing_zeros(mask)) / ascii_stop_bits()) : n;
}
if (end - p >= 16)
{
// (the bytes after the run are masked out)
const std::uint64_t mask = ascii_stop_mask(p) & ((std::uint64_t{1} << (n * ascii_stop_bits())) - 1u);
return mask != 0 ? static_cast<std::size_t>(count_trailing_zeros(mask)) / ascii_stop_bits() : n;
}
#else
static_cast<void>(end);
#endif
return find_ascii_copyable_run(p, n);
}
/// A stack that starts in a buffer of the caller (a local array) and moves to /// A stack that starts in a buffer of the caller (a local array) and moves to
/// the heap (a vector of the caller) only when that is full, so that dumps of /// the heap (a vector of the caller) only when that is full, so that dumps of
/// shallow documents need no allocation. The top is a pointer, as in /// shallow documents need no allocation. The top is a pointer, as in
@@ -275,6 +351,39 @@ NLOHMANN_VIEW_ALWAYS_INLINE char* put_literal(char* w, const char (&s)[N]) noexc
return w + N - 1; return w + N - 1;
} }
/// the depth of the innermost open container, counted only for indented output
template<bool Counted>
struct depth_counter
{
std::size_t value = 0;
void enter() noexcept
{
++value;
}
std::size_t leave() noexcept
{
return --value;
}
std::size_t get() const noexcept
{
return value;
}
};
template<>
struct depth_counter<false>
{
void enter() noexcept {}
std::size_t leave() noexcept // NOLINT(readability-convert-member-functions-to-static): same interface as depth_counter<true>
{
return 0;
}
std::size_t get() const noexcept // NOLINT(readability-convert-member-functions-to-static): same interface as depth_counter<true>
{
return 0;
}
};
/// how the view's dump() writes a value /// how the view's dump() writes a value
struct dump_style struct dump_style
{ {
@@ -311,139 +420,95 @@ class view_serializer
void dump(const node* root) void dump(const node* root)
{ {
if (!m_style.pretty) if (m_style.ensure_ascii)
{ {
if (m_style.ensure_ascii) dump_ascii(root);
{
dump_compact_ascii(root);
}
else if (m_style.source_numbers)
{
dump_compact<true, false>(root);
}
else
{
dump_compact<false, false>(root);
}
return;
} }
struct frame else if (m_style.pretty)
{ {
const node* pos; ///< next element, or key of the next member dump_pretty(root);
const node* end; }
bool object; else if (m_style.source_numbers)
bool first; ///< nothing written yet
};
std::array<frame, 32> buffer; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::vector<frame> heap;
small_stack<frame> stack(buffer.data(), buffer.size(), heap);
const node* n = root;
for (;;)
{ {
// write the value at n dump_text<true, false, false>(root);
if (is_container(*n)) }
{ else
const bool object = n->kind == static_cast<std::uint8_t>(value_t::object); {
if (n->len == 0) dump_text<false, false, false>(root);
{
m_out.put(object ? "{}" : "[]", 2);
}
else
{
m_out.put(object ? '{' : '[');
stack.push_back(frame{nav::first(m_doc, n), nav::end(m_doc, n), object, true});
}
}
else
{
write_scalar(*n);
}
// go to the next value: close finished containers, then separate
for (;;)
{
if (stack.empty())
{
m_out.finish();
return;
}
if (stack.back().pos == stack.back().end)
{
const bool object = stack.back().object;
stack.pop_back();
newline(stack.size());
m_out.put(object ? '}' : ']');
continue;
}
frame& f = stack.back();
if (!f.first)
{
m_out.put(',');
}
f.first = false;
newline(stack.size());
if (f.object)
{
write_string(*f.pos);
if (m_style.pretty)
{
m_out.put(": ");
}
else
{
m_out.put(':');
}
n = nav::value(f.pos + 1);
f.pos = document_data::after(f.pos + 1);
}
else
{
n = nav::value(f.pos);
f.pos = document_data::after(f.pos);
}
break;
}
} }
} }
private: private:
/// dump_text() with ensure_ascii, out of line: the walk without it is
/// faster when it is not merged with this one into dump()
NLOHMANN_VIEW_NOINLINE void dump_ascii(const node* root)
{
if (m_style.pretty)
{
if (m_style.source_numbers)
{
dump_text<true, true, true>(root);
}
else
{
dump_text<false, true, true>(root);
}
}
else if (m_style.source_numbers)
{
dump_text<true, false, true>(root);
}
else
{
dump_text<false, false, true>(root);
}
}
/// dump_text() with indentation, out of line: the compact walk is faster
/// when it is not merged with the indented ones into dump()
NLOHMANN_VIEW_NOINLINE void dump_pretty(const node* root)
{
if (m_style.source_numbers)
{
dump_text<true, true, false>(root);
}
else
{
dump_text<false, true, false>(root);
}
}
/// a string that is not a plain string of the source (decoded, or written /// a string that is not a plain string of the source (decoded, or written
/// by an edit), without ensure_ascii: runs without characters to escape /// by an edit), without ensure_ascii: runs without characters to escape
/// are copied /// are copied. If validate is set, the string must be valid UTF-8: the
NLOHMANN_VIEW_NOINLINE void write_decoded(const node& n) /// runs are found by the lexer's scan, which validates the sequences, and
/// invalid UTF-8 throws what basic_json::dump() throws. (The compact output
/// writes the bytes of a damaged image unchanged, and scans without
/// validating them.)
NLOHMANN_VIEW_NOINLINE void write_decoded(const node& n, bool validate)
{ {
const auto* const s = reinterpret_cast<const unsigned char*>(m_doc.str(n)); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast) const auto* const s = reinterpret_cast<const unsigned char*>(m_doc.str(n)); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
m_out.put('"'); m_out.put('"');
for (std::size_t i = 0; i < n.len;) for (std::size_t i = 0; i < n.len;)
{ {
const std::size_t run = plain_output_run(s + i, n.len - i); const std::size_t run = validate ? string_bulk_run(s + i, n.len - i) : plain_output_run(s + i, n.len - i);
if (run != 0) if (run != 0)
{ {
m_out.put(reinterpret_cast<const char*>(s + i), run); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast) m_out.put(reinterpret_cast<const char*>(s + i), run); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
i += run; i += run;
continue; continue;
} }
if (validate && s[i] >= 0x80)
{
invalid_utf8(s, n.len); // (the scan stopped at ill-formed UTF-8)
}
write_codepoint<false>(s[i], s + i, 1); // a quote, a backslash, or a control character write_codepoint<false>(s[i], s + i, 1); // a quote, a backslash, or a control character
++i; ++i;
} }
m_out.put('"'); m_out.put('"');
} }
/// dump_compact() with ensure_ascii, out of line: the walk without it is /// whether dump_text() copies a string or key from the source: one of
/// faster when it is not merged with this one into dump()
NLOHMANN_VIEW_NOINLINE void dump_compact_ascii(const node* root)
{
if (m_style.source_numbers)
{
dump_compact<true, true>(root);
}
else
{
dump_compact<false, true>(root);
}
}
/// whether dump_compact() copies a string or key from the source: one of
/// the source, and with ensure_ascii one without bytes >= 0x7F (an overload /// the source, and with ensure_ascii one without bytes >= 0x7F (an overload
/// each, so that the walk without ensure_ascii is compiled exactly as /// each, so that the walk without ensure_ascii is compiled exactly as
/// before: with GCC, a condition that only folds away later changed its /// before: with GCC, a condition that only folds away later changed its
@@ -458,13 +523,155 @@ class view_serializer
return (n.flags & node_flags::storage) == 0 && is_ascii_without_delete(src + n.off, n.len); return (n.flags & node_flags::storage) == 0 && is_ascii_without_delete(src + n.off, n.len);
} }
/// a string with ensure_ascii (dump_compact()), out of line /*!
NLOHMANN_VIEW_NOINLINE void write_ascii_string(const node& n) @brief a string with ensure_ascii: the characters that are not printable ASCII as \\uXXXX
(a character beyond U+FFFF as a surrogate pair); strings of a document
are valid UTF-8, except in a damaged image loaded with image_check::bounds,
for which this throws what basic_json::dump() throws for the string
*/
NLOHMANN_VIEW_NOINLINE void write_ascii(const node& n)
{ {
write_string(n); const auto* const s = reinterpret_cast<const unsigned char*>(m_doc.str(n)); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
const std::size_t size = n.len;
char* w = m_out.cursor();
char* lim = m_out.limit();
// room for k bytes
const auto room = [&](std::size_t k)
{
const auto left = static_cast<std::size_t>(lim - w);
if (NLOHMANN_VIEW_UNLIKELY(left < k))
{
m_out.set_cursor(w);
m_out.reserve(k);
w = m_out.cursor();
lim = m_out.limit();
}
};
static constexpr const char* hex = "0123456789abcdef";
// \\uXXXX
const auto u_escape = [&](std::uint32_t u)
{
w[0] = '\\';
w[1] = 'u';
w[2] = hex[(u >> 12u) & 0xFu];
w[3] = hex[(u >> 8u) & 0xFu];
w[4] = hex[(u >> 4u) & 0xFu];
w[5] = hex[u & 0xFu];
w += 6;
};
room(1);
*w++ = '"';
std::size_t i = 0;
// (a string of the source starts with a run that has nothing to escape, 16 bytes per step)
const std::size_t prefix = (n.flags & node_flags::storage) == 0 ? ascii_copyable_run(s, size, reinterpret_cast<const unsigned char*>(m_doc.src + m_doc.size)) : 0; // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
if (prefix != 0)
{
room(prefix);
std::memcpy(w, s, prefix);
w += prefix;
i = prefix;
}
while (i < size)
{
// (a character that needs an escape sequence is not worth a call)
const std::size_t run = is_ascii_copyable(s[i]) ? find_ascii_copyable_run(s + i, size - i) : 0;
if (run != 0)
{
room(run);
std::memcpy(w, s + i, run);
w += run;
i += run;
continue;
}
room(12);
if (s[i] >= 0x80)
{
const unsigned char lead = s[i];
// (the two-byte sequences of Latin text and the three-byte sequences of most other
// text, with continuation bytes of any value: the ranges that are restricted (E0,
// ED, and the sequences of four bytes) go through validate_one_utf8())
if (lead >= 0xE1 && lead <= 0xEF && lead != 0xED && size - i >= 3 && (s[i + 1] & 0xC0u) == 0x80 && (s[i + 2] & 0xC0u) == 0x80)
{
u_escape(((lead & 0x0Fu) << 12u) | ((s[i + 1] & 0x3Fu) << 6u) | (s[i + 2] & 0x3Fu));
i += 3;
continue;
}
if (lead >= 0xC2 && lead <= 0xDF && size - i >= 2 && (s[i + 1] & 0xC0u) == 0x80)
{
u_escape(((lead & 0x1Fu) << 6u) | (s[i + 1] & 0x3Fu));
i += 2;
continue;
}
const std::size_t len = validate_one_utf8(s + i, size - i);
if (NLOHMANN_VIEW_UNLIKELY(len == 0))
{
m_out.set_cursor(w);
invalid_utf8(s, size);
return; // LCOV_EXCL_LINE (invalid_utf8 throws)
}
std::uint32_t codepoint = s[i] & (0xFFu >> (len + 1));
for (std::size_t k = 1; k < len; ++k)
{
codepoint = (codepoint << 6u) | (s[i + k] & 0x3Fu);
}
i += len;
if (codepoint <= 0xFFFF)
{
u_escape(codepoint);
}
else
{
u_escape(0xD7C0u + (codepoint >> 10u));
u_escape(0xDC00u + (codepoint & 0x3FFu));
}
continue;
}
// a quote, a backslash, or a control character (or DEL)
const unsigned char c = s[i++];
char escape = 0;
switch (c)
{
case 0x08:
escape = 'b';
break;
case 0x09:
escape = 't';
break;
case 0x0A:
escape = 'n';
break;
case 0x0C:
escape = 'f';
break;
case 0x0D:
escape = 'r';
break;
case 0x22:
escape = '"';
break;
case 0x5C:
escape = '\\';
break;
default:
break;
}
if (escape != 0)
{
w[0] = '\\';
w[1] = escape;
w += 2;
}
else
{
u_escape(c);
}
}
room(1);
*w++ = '"';
m_out.set_cursor(w);
} }
/// the copies of dump_compact() that are not fixed-size moves (long /// the copies of dump_text() that are not fixed-size moves (long
/// strings, or near the end of the source); out of line, so that the /// strings, or near the end of the source); out of line, so that the
/// compiler does not merge the fixed-size moves into this call /// compiler does not merge the fixed-size moves into this call
NLOHMANN_VIEW_NOINLINE static void copy_long(char* to, const char* from, std::size_t n) noexcept NLOHMANN_VIEW_NOINLINE static void copy_long(char* to, const char* from, std::size_t n) noexcept
@@ -481,7 +688,7 @@ class view_serializer
} }
/*! /*!
@brief the compact output (the default dump(), or with ensure_ascii) @brief the output, compact (the default dump()) or indented, with or without ensure_ascii
The same walk as dump(), with the write position in a local variable The same walk as dump(), with the write position in a local variable
(stores through char pointers would otherwise force a reload of the (stores through char pointers would otherwise force a reload of the
@@ -489,11 +696,11 @@ class view_serializer
the source copied by fixed-size moves of 32 bytes where the source has the source copied by fixed-size moves of 32 bytes where the source has
that many bytes left, instead of a library call per token. The buffer that many bytes left, instead of a library call per token. The buffer
keeps 64 bytes of slack for the overshoot. With ensure_ascii (Ascii), a keeps 64 bytes of slack for the overshoot. With ensure_ascii (Ascii), a
string of the source is copied only if it has no byte >= 0x7F (see string of the source is copied only if it has nothing to escape (see
is_ascii_without_delete()); any other string is written by write_string(). copied()); any other string is written by write_ascii().
*/ */
template<bool SourceNumbers, bool Ascii> template<bool SourceNumbers, bool Pretty, bool Ascii>
void dump_compact(const node* root) void dump_text(const node* root)
{ {
struct frame struct frame
{ {
@@ -508,6 +715,16 @@ class view_serializer
const char* const src_end = src + m_doc.size; const char* const src_end = src + m_doc.size;
char* w = m_out.cursor(); char* w = m_out.cursor();
char* lim = m_out.limit(); char* lim = m_out.limit();
// (indented output) a line break and 63 indentation characters, from
// which the line break of a level of up to 63 characters is a copy
std::array<char, 64> ruler; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): filled below if used
if (Pretty)
{
ruler.fill(m_style.indent_char);
ruler[0] = '\n';
}
const std::size_t indent = m_style.indent;
depth_counter<Pretty> level; // depth of the innermost open container (indented output)
// room for n bytes and the slack // room for n bytes and the slack
const auto room = [&](std::size_t n) const auto room = [&](std::size_t n)
{ {
@@ -539,6 +756,22 @@ class view_serializer
} }
w += n; w += n;
}; };
// (indented output) the line break and the indentation of a level
const auto newline = [&](std::size_t depth)
{
const std::size_t count = 1 + (depth * indent);
room(count);
if (count <= ruler.size())
{
std::memcpy(w, ruler.data(), ruler.size());
}
else
{
*w = '\n';
std::memset(w + 1, m_style.indent_char, count - 1);
}
w += count;
};
// a string that is not a plain string of the source (out of line, so // a string that is not a plain string of the source (out of line, so
// that the cursor stays in a register here) // that the cursor stays in a register here)
const auto escaped = [&](const node & n) const auto escaped = [&](const node & n)
@@ -546,11 +779,11 @@ class view_serializer
m_out.set_cursor(w); m_out.set_cursor(w);
if (enabled(Ascii)) if (enabled(Ascii))
{ {
write_ascii_string(n); write_ascii(n);
} }
else else
{ {
write_decoded(n); write_decoded(n, Pretty);
} }
w = m_out.cursor(); w = m_out.cursor();
lim = m_out.limit(); lim = m_out.limit();
@@ -651,6 +884,7 @@ class view_serializer
{ {
*w++ = object ? '{' : '['; *w++ = object ? '{' : '[';
stack.push_back(cur); stack.push_back(cur);
level.enter();
if (Editable) if (Editable)
{ {
cur = frame{nav::first(m_doc, n), nav::end(m_doc, n), object}; cur = frame{nav::first(m_doc, n), nav::end(m_doc, n), object};
@@ -693,6 +927,10 @@ class view_serializer
{ {
break; break;
} }
if (Pretty)
{
newline(level.leave());
}
room(1); room(1);
*w++ = cur.object ? '}' : ']'; *w++ = cur.object ? '}' : ']';
cur = stack.back(); cur = stack.back();
@@ -701,13 +939,17 @@ class view_serializer
room(1); room(1);
*w++ = ','; *w++ = ',';
} }
if (Pretty)
{
newline(level.get());
}
const node* const at = Editable ? cur.pos : n; const node* const at = Editable ? cur.pos : n;
if (cur.object) if (cur.object)
{ {
const node& key = *at; const node& key = *at;
if (copied(key, src, std::integral_constant<bool, Ascii>())) if (copied(key, src, std::integral_constant<bool, Ascii>()))
{ {
room(key.len + 3); room(key.len + (Pretty ? 4 : 3));
*w++ = '"'; *w++ = '"';
copy(src + key.off, key.len); copy(src + key.off, key.len);
w[0] = '"'; w[0] = '"';
@@ -717,9 +959,13 @@ class view_serializer
else else
{ {
escaped(key); escaped(key);
room(1); room(Pretty ? 2 : 1);
*w++ = ':'; *w++ = ':';
} }
if (Pretty)
{
*w++ = ' ';
}
if (Editable) if (Editable)
{ {
n = nav::value(at + 1); n = nav::value(at + 1);
@@ -738,69 +984,6 @@ class view_serializer
} }
} }
void newline(std::size_t level)
{
if (m_style.pretty)
{
m_out.put('\n');
m_out.put_repeated(m_style.indent_char, level * m_style.indent);
}
}
void write_scalar(const node& n)
{
switch (static_cast<value_t>(n.kind))
{
case value_t::null:
m_out.put("null");
break;
case value_t::boolean:
if ((n.flags & node_flags::is_true) != 0)
{
m_out.put("true");
}
else
{
m_out.put("false");
}
break;
case value_t::string:
write_string(n);
break;
case value_t::number_integer:
case value_t::number_unsigned:
{
const char* const token = m_doc.str(n);
const std::uint32_t len = number_length(n);
if (!m_style.source_numbers && len == 2 && token[0] == '-' && token[1] == '0')
{
m_out.put('0'); // parse() reads -0 as the integer 0
}
else
{
m_out.put(token, len);
}
break;
}
case value_t::number_float:
if (m_style.source_numbers && (n.flags & node_flags::storage) != node_flags::edited)
{
m_out.put(m_doc.str(n), n.len); // (a float set by an edit is written as with shortest)
}
else
{
write_float_node(n);
}
break;
case value_t::object: // LCOV_EXCL_LINE (containers are written by dump())
case value_t::array: // LCOV_EXCL_LINE
case value_t::binary: // LCOV_EXCL_LINE (not in a document)
case value_t::discarded: // LCOV_EXCL_LINE
default: // LCOV_EXCL_LINE
break; // LCOV_EXCL_LINE
}
}
/// a float node as dump() writes it /// a float node as dump() writes it
void write_float_node(const node& n) void write_float_node(const node& n)
{ {
@@ -890,89 +1073,27 @@ class view_serializer
m_out.put("null"); m_out.put("null");
return; return;
} }
write_float(x, std::integral_constant<bool, ::nlohmann::detail::has_native_float_format<number_float_t>::value>()); static constexpr bool is_ieee_single_or_double
= std::numeric_limits<number_float_t>::is_iec559
&& ((std::numeric_limits<number_float_t>::digits == 24 && std::numeric_limits<number_float_t>::max_exponent == 128)
|| (std::numeric_limits<number_float_t>::digits == 53 && std::numeric_limits<number_float_t>::max_exponent == 1024));
write_float(x, std::integral_constant<bool, is_ieee_single_or_double>());
} }
void write_float(number_float_t x, std::true_type /*has_native_float_format*/) void write_float(number_float_t x, std::true_type /*is_ieee_single_or_double*/)
{ {
std::array<char, 64> buf{}; std::array<char, 64> buf{};
const char* const end = ::nlohmann::detail::to_chars(buf.data(), buf.data() + buf.size(), x); const char* const end = ::nlohmann::detail::to_chars(buf.data(), buf.data() + buf.size(), x);
m_out.put(buf.data(), static_cast<std::size_t>(end - buf.data())); m_out.put(buf.data(), static_cast<std::size_t>(end - buf.data()));
} }
void write_float(number_float_t x, std::false_type /*has_native_float_format*/) void write_float(number_float_t x, std::false_type /*is_ieee_single_or_double*/)
{ {
// other types (e.g. long double) are rare: the library writes them // other types (e.g. long double) are rare: the library writes them
const string_t s = BasicJsonType(x).dump(); const string_t s = BasicJsonType(x).dump();
m_out.put(s.data(), s.size()); m_out.put(s.data(), s.size());
} }
void write_string(const node& n)
{
const char* const s = m_doc.str(n);
m_out.put('"');
if ((n.flags & node_flags::storage) == 0 && (!m_style.ensure_ascii || is_ascii_without_delete(s, n.len)))
{
// a string of the source without escape sequences has nothing to
// escape (with ensure_ascii, if it is printable ASCII)
m_out.put(s, n.len);
}
else if (m_style.ensure_ascii)
{
write_escaped<true>(reinterpret_cast<const unsigned char*>(s), n.len); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
}
else
{
write_escaped<false>(reinterpret_cast<const unsigned char*>(s), n.len); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
}
m_out.put('"');
}
/// as serializer::dump_escaped(); strings of a document are valid UTF-8,
/// except in a damaged image loaded with image_check::bounds, for which
/// this throws what basic_json::dump() throws for the string
template<bool EnsureAscii>
void write_escaped(const unsigned char* s, std::size_t n)
{
std::size_t i = 0;
while (i < n)
{
std::size_t run = 0;
if (!EnsureAscii)
{
run = string_bulk_run(s + i, n - i);
}
else if (is_ascii_copyable(s[i]))
{
run = find_ascii_copyable_run(s + i, n - i);
}
if (run != 0)
{
m_out.put(reinterpret_cast<const char*>(s + i), run); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
i += run;
continue;
}
std::uint32_t codepoint = s[i];
std::size_t len = 1;
if (codepoint >= 0x80)
{
len = validate_one_utf8(s + i, n - i);
if (NLOHMANN_VIEW_UNLIKELY(len == 0))
{
invalid_utf8(s, n);
return;
}
codepoint &= 0xFFu >> (len + 1);
for (std::size_t k = 1; k < len; ++k)
{
codepoint = (codepoint << 6u) | (s[i + k] & 0x3Fu);
}
}
write_codepoint<EnsureAscii>(codepoint, s + i, len);
i += len;
}
}
/// throw what basic_json::dump() throws for a string that is not valid UTF-8 /// throw what basic_json::dump() throws for a string that is not valid UTF-8
NLOHMANN_VIEW_NOINLINE static void invalid_utf8(const unsigned char* s, std::size_t n) NLOHMANN_VIEW_NOINLINE static void invalid_utf8(const unsigned char* s, std::size_t n)
{ {
@@ -1027,8 +1148,8 @@ class view_serializer
void write_u_escape(std::uint32_t u) void write_u_escape(std::uint32_t u)
{ {
std::array<char, 6> e = {{'\\', 'u', 0, 0, 0, 0}}; static constexpr const char* hex = "0123456789abcdef";
hex4_lower(static_cast<std::uint16_t>(u), &e[2]); const std::array<char, 6> e = {{'\\', 'u', hex[(u >> 12u) & 0xFu], hex[(u >> 8u) & 0xFu], hex[(u >> 4u) & 0xFu], hex[u & 0xFu]}};
m_out.put(e.data(), e.size()); m_out.put(e.data(), e.size());
} }
include/nlohmann/detail/view/simd.hpp
+28
View File
@@ -116,6 +116,34 @@ NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned
} }
return p; return p;
} }
/*!
@brief copy 16 bytes from p to w, and find the first quote, backslash,
control character, or non-ASCII byte among them
The bytes are copied whether or not they are all plain (the caller has room
for 16 bytes and keeps only the plain ones).
@return the offset of that byte, or 16 if there is none
*/
NLOHMANN_VIEW_ALWAYS_INLINE unsigned copy_plain_run16(const unsigned char* p, char* w) noexcept
{
#if NLOHMANN_VIEW_NEON
const uint8x16_t in = vld1q_u8(p);
vst1q_u8(reinterpret_cast<std::uint8_t*>(w), in); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))),
vcltq_s8(vreinterpretq_s8_u8(in), vdupq_n_s8(0x20)));
const std::uint64_t bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0);
return bits != 0 ? static_cast<unsigned>(count_trailing_zeros(bits) >> 2u) : 16u;
#else
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
_mm_storeu_si128(static_cast<__m128i*>(static_cast<void*>(w)), in); // NOLINT(bugprone-casting-through-void)
const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_cmplt_epi8(in, _mm_set1_epi8(0x20)));
const auto bits = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
return bits != 0 ? static_cast<unsigned>(count_trailing_zeros(bits)) : 16u;
#endif
}
#endif #endif
#if NLOHMANN_VIEW_SSSE3_DISPATCH #if NLOHMANN_VIEW_SSSE3_DISPATCH
include/nlohmann/detail/view/value.hpp
+3 -3
View File
@@ -60,7 +60,7 @@ NLOHMANN_VIEW_ALWAYS_INLINE T arithmetic_value(const document_data& d, const nod
case value_t::binary: case value_t::binary:
case value_t::discarded: case value_t::discarded:
default: default:
throw_type_error(exception_id::type_mismatch, "type must be number, but is ", value_type_name(static_cast<value_t>(n.kind))); throw_type_error(302, "type must be number, but is ", value_type_name(static_cast<value_t>(n.kind)));
} }
} }
@@ -70,7 +70,7 @@ std::vector<U, A> vector_value(const View& v)
{ {
if (NLOHMANN_VIEW_UNLIKELY(!v.is_array())) if (NLOHMANN_VIEW_UNLIKELY(!v.is_array()))
{ {
throw_type_error(exception_id::type_mismatch, "type must be array, but is ", v.type_name()); throw_type_error(302, "type must be array, but is ", v.type_name());
} }
std::vector<U, A> r; std::vector<U, A> r;
r.reserve(v.size()); r.reserve(v.size());
@@ -88,7 +88,7 @@ Map map_value(const View& v)
{ {
if (NLOHMANN_VIEW_UNLIKELY(!v.is_object())) if (NLOHMANN_VIEW_UNLIKELY(!v.is_object()))
{ {
throw_type_error(exception_id::type_mismatch, "type must be object, but is ", v.type_name()); throw_type_error(302, "type must be object, but is ", v.type_name());
} }
Map r; Map r;
for (auto it = v.begin(); it != v.end(); ++it) for (auto it = v.begin(); it != v.end(); ++it)
include/nlohmann/json.hpp
+358 -305
View File
File diff suppressed because it is too large. Load diff
include/nlohmann/json_view.hpp
+107 -29
View File
@@ -30,6 +30,7 @@
#include <array> // array #include <array> // array
#include <cstddef> // nullptr_t, size_t // IWYU pragma: keep #include <cstddef> // nullptr_t, size_t // IWYU pragma: keep
#include <cstdint> // uint8_t, uint32_t #include <cstdint> // uint8_t, uint32_t
#include <cstdio> // FILE
#include <cstring> // memcpy, strlen #include <cstring> // memcpy, strlen
#include <istream> // istream // IWYU pragma: keep #include <istream> // istream // IWYU pragma: keep
#include <iterator> // distance, input_iterator_tag, iterator_traits #include <iterator> // distance, input_iterator_tag, iterator_traits
@@ -267,7 +268,7 @@ class basic_json_view
{ {
return {}; return {};
} }
detail::view::throw_type_error(detail::exception_id::subscript_wrong_type, "cannot use operator[] with a string argument with ", type_name()); detail::view::throw_type_error(305, "cannot use operator[] with a string argument with ", type_name());
} }
return lookup(key); return lookup(key);
} }
@@ -293,9 +294,9 @@ class basic_json_view
{ {
return {}; return {};
} }
detail::view::throw_type_error(detail::exception_id::subscript_wrong_type, "cannot use operator[] with a numeric argument with ", type_name()); detail::view::throw_type_error(305, "cannot use operator[] with a numeric argument with ", type_name());
} }
return idx < m_node->len ? basic_json_view(m_doc, navigation::value(detail::view::element_at<Editable>(*m_doc, m_node, idx))) : basic_json_view(); return idx < m_node->len ? element(idx) : basic_json_view();
} }
/// any other integer type (int, unsigned, long, std::int64_t, ...; a /// any other integer type (int, unsigned, long, std::int64_t, ...; a
@@ -326,12 +327,12 @@ class basic_json_view
{ {
if (NLOHMANN_VIEW_UNLIKELY(!is_object())) if (NLOHMANN_VIEW_UNLIKELY(!is_object()))
{ {
detail::view::throw_type_error(detail::exception_id::at_wrong_type, "cannot use at() with ", type_name()); detail::view::throw_type_error(304, "cannot use at() with ", type_name());
} }
const basic_json_view r = lookup(key); const basic_json_view r = lookup(key);
if (NLOHMANN_VIEW_UNLIKELY(r.is_discarded())) if (NLOHMANN_VIEW_UNLIKELY(r.is_discarded()))
{ {
detail::view::throw_key_not_found(std::string(key.data(), key.size())); detail::view::throw_out_of_range(403, detail::concat("key '", std::string(key.data(), key.size()), "' not found"));
} }
return r; return r;
} }
@@ -352,13 +353,13 @@ class basic_json_view
{ {
if (NLOHMANN_VIEW_UNLIKELY(!is_array())) if (NLOHMANN_VIEW_UNLIKELY(!is_array()))
{ {
detail::view::throw_type_error(detail::exception_id::at_wrong_type, "cannot use at() with ", type_name()); detail::view::throw_type_error(304, "cannot use at() with ", type_name());
} }
if (NLOHMANN_VIEW_UNLIKELY(idx >= m_node->len)) if (NLOHMANN_VIEW_UNLIKELY(idx >= m_node->len))
{ {
detail::view::throw_array_index_out_of_range(idx); detail::view::throw_out_of_range(401, detail::concat("array index ", std::to_string(idx), " is out of range"));
} }
return basic_json_view(m_doc, navigation::value(detail::view::element_at<Editable>(*m_doc, m_node, idx))); return element(idx);
} }
/// any other integer type, see operator[]; negative values are out of /// any other integer type, see operator[]; negative values are out of
@@ -384,7 +385,7 @@ class basic_json_view
{ {
if (NLOHMANN_VIEW_UNLIKELY(!is_object())) if (NLOHMANN_VIEW_UNLIKELY(!is_object()))
{ {
detail::view::throw_type_error(detail::exception_id::value_wrong_type, "cannot use value() with ", type_name()); detail::view::throw_type_error(306, "cannot use value() with ", type_name());
} }
const basic_json_view r = lookup(key); const basic_json_view r = lookup(key);
return r.is_discarded() ? default_value : r.template get<T>(); return r.is_discarded() ? default_value : r.template get<T>();
@@ -392,7 +393,13 @@ class basic_json_view
string_t value(string_view_t key, const char* default_value) const string_t value(string_view_t key, const char* default_value) const
{ {
return value(key, string_t(default_value)); if (NLOHMANN_VIEW_UNLIKELY(!is_object()))
{
detail::view::throw_type_error(306, "cannot use value() with ", type_name());
}
// (the default is made only if there is no such member)
const basic_json_view r = lookup(key);
return r.is_discarded() ? string_t(default_value) : r.template get<string_t>();
} }
/// the value a JSON pointer refers to converted to T, or the default /// the value a JSON pointer refers to converted to T, or the default
@@ -403,7 +410,7 @@ class basic_json_view
{ {
if (NLOHMANN_VIEW_UNLIKELY(!is_structured())) if (NLOHMANN_VIEW_UNLIKELY(!is_structured()))
{ {
detail::view::throw_type_error(detail::exception_id::value_wrong_type, "cannot use value() with ", type_name()); detail::view::throw_type_error(306, "cannot use value() with ", type_name());
} }
const basic_json_view r = detail::view::resolve_pointer(*this, detail::json_pointer_access::reference_tokens(ptr), detail::view::pointer_mode::value); const basic_json_view r = detail::view::resolve_pointer(*this, detail::json_pointer_access::reference_tokens(ptr), detail::view::pointer_mode::value);
return r.is_discarded() ? default_value : r.template get<T>(); return r.is_discarded() ? default_value : r.template get<T>();
@@ -411,7 +418,12 @@ class basic_json_view
string_t value(const json_pointer& ptr, const char* default_value) const string_t value(const json_pointer& ptr, const char* default_value) const
{ {
return value(ptr, string_t(default_value)); if (NLOHMANN_VIEW_UNLIKELY(!is_structured()))
{
detail::view::throw_type_error(306, "cannot use value() with ", type_name());
}
const basic_json_view r = detail::view::resolve_pointer(*this, detail::json_pointer_access::reference_tokens(ptr), detail::view::pointer_mode::value);
return r.is_discarded() ? string_t(default_value) : r.template get<string_t>();
} }
/// the first element or member value; a primitive value itself. Throws /// the first element or member value; a primitive value itself. Throws
@@ -421,7 +433,7 @@ class basic_json_view
const iterator it = begin(); const iterator it = begin();
if (NLOHMANN_VIEW_UNLIKELY(it == end())) if (NLOHMANN_VIEW_UNLIKELY(it == end()))
{ {
detail::view::throw_invalid_iterator(detail::exception_id::iterator_value_unavailable, "cannot get value"); detail::view::throw_invalid_iterator(214, "cannot get value");
} }
return *it; return *it;
} }
@@ -433,7 +445,8 @@ class basic_json_view
{ {
if (is_structured() && m_node->len != 0) if (is_structured() && m_node->len != 0)
{ {
return basic_json_view(m_doc, navigation::value(detail::view::last_child<Editable>(*m_doc, m_node) + (is_object() ? 1 : 0))); return unedited() ? basic_json_view(m_doc, detail::view::last_child<false>(*m_doc, m_node) + (is_object() ? 1 : 0))
: basic_json_view(m_doc, navigation::value(detail::view::last_child<Editable>(*m_doc, m_node) + (is_object() ? 1 : 0)));
} }
return front(); return front();
} }
@@ -450,7 +463,7 @@ class basic_json_view
{ {
return end(); return end();
} }
const node* const k = detail::view::find_member<Editable>(*m_doc, m_node, key.data(), key.size()); const node* const k = find_key(key);
return k != nullptr ? iterator(m_doc, k, true) : end(); return k != nullptr ? iterator(m_doc, k, true) : end();
} }
@@ -467,7 +480,7 @@ class basic_json_view
/// whether this is an object with a member with this key /// whether this is an object with a member with this key
bool contains(string_view_t key) const bool contains(string_view_t key) const
{ {
return is_object() && detail::view::find_member<Editable>(*m_doc, m_node, key.data(), key.size()) != nullptr; return is_object() && find_key(key) != nullptr;
} }
bool contains(const char* key) const bool contains(const char* key) const
@@ -573,7 +586,7 @@ class basic_json_view
{ {
if (NLOHMANN_VIEW_UNLIKELY(!is_string())) if (NLOHMANN_VIEW_UNLIKELY(!is_string()))
{ {
detail::view::throw_type_error(detail::exception_id::type_mismatch, "type must be string, but is ", type_name()); detail::view::throw_type_error(302, "type must be string, but is ", type_name());
} }
return {m_doc->str(*m_node), m_node->len}; return {m_doc->str(*m_node), m_node->len};
} }
@@ -585,7 +598,7 @@ class basic_json_view
{ {
if (NLOHMANN_VIEW_UNLIKELY(!is_number())) if (NLOHMANN_VIEW_UNLIKELY(!is_number()))
{ {
detail::view::throw_type_error(detail::exception_id::type_mismatch, "type must be number, but is ", type_name()); detail::view::throw_type_error(302, "type must be number, but is ", type_name());
} }
return {m_doc->str(*m_node), detail::view::number_length(*m_node)}; return {m_doc->str(*m_node), detail::view::number_length(*m_node)};
} }
@@ -627,7 +640,14 @@ class basic_json_view
// the compact writer keeps 64 bytes of slack, so that it does not grow // the compact writer keeps 64 bytes of slack, so that it does not grow
// the buffer just before the end // the buffer just before the end
const std::size_t estimate = source_extent() + (style.pretty ? source_extent() / 2 : 0) + 160; const std::size_t estimate = source_extent() + (style.pretty ? source_extent() / 2 : 0) + 160;
detail::view::view_serializer<BasicJsonType, Editable>(*m_doc, out, estimate, style).dump(m_node); if (unedited())
{
detail::view::view_serializer<BasicJsonType, false>(*m_doc, out, estimate, style).dump(m_node);
}
else
{
detail::view::view_serializer<BasicJsonType, Editable>(*m_doc, out, estimate, style).dump(m_node);
}
return out; return out;
} }
@@ -707,7 +727,7 @@ class basic_json_view
{ {
return BasicJsonType(value_t::discarded); return BasicJsonType(value_t::discarded);
} }
return detail::view::materialize<BasicJsonType, Editable>(*m_doc, m_node); return unedited() ? detail::view::materialize<BasicJsonType, false>(*m_doc, m_node) : detail::view::materialize<BasicJsonType, Editable>(*m_doc, m_node);
} }
/// byte offset of this value in the source text (for strings: of the /// byte offset of this value in the source text (for strings: of the
@@ -723,8 +743,12 @@ class basic_json_view
template<typename, bool> friend class basic_json_document; template<typename, bool> friend class basic_json_document;
template<typename, bool> friend class basic_json_view; template<typename, bool> friend class basic_json_view;
template<typename, typename> friend class detail::view::editor; template<typename, typename> friend class detail::view::editor;
template<typename, typename> friend class detail::view::view_side;
friend iterator; friend iterator;
/// whether the view is one of an editable document
static constexpr bool is_editable = Editable;
basic_json_view(const document_data* d, const node* n) noexcept basic_json_view(const document_data* d, const node* n) noexcept
: m_doc(d), m_node(n) : m_doc(d), m_node(n)
{} {}
@@ -785,17 +809,57 @@ class basic_json_view
/// (object required) /// (object required)
NLOHMANN_VIEW_ALWAYS_INLINE basic_json_view lookup(string_view_t key) const noexcept NLOHMANN_VIEW_ALWAYS_INLINE basic_json_view lookup(string_view_t key) const noexcept
{ {
if (unedited())
{
const node* const k = detail::view::find_member<false>(*m_doc, m_node, key.data(), key.size());
return k != nullptr ? basic_json_view(m_doc, k + 1) : basic_json_view();
}
const node* const k = detail::view::find_member<Editable>(*m_doc, m_node, key.data(), key.size()); const node* const k = detail::view::find_member<Editable>(*m_doc, m_node, key.data(), key.size());
return k != nullptr ? basic_json_view(m_doc, navigation::value(k + 1)) : basic_json_view(); return k != nullptr ? basic_json_view(m_doc, navigation::value(k + 1)) : basic_json_view();
} }
/// whether the document has not been edited yet: until the first edit, its index is walked like that of a
/// read-only document (nothing is moved or linked), which is cheaper
NLOHMANN_VIEW_ALWAYS_INLINE bool unedited() const noexcept
{
return !navigation::editable() || m_doc->edits == nullptr;
}
/// the member of this object with @a key (the last one if the key occurs more than once), or nullptr
NLOHMANN_VIEW_ALWAYS_INLINE const node* find_key(string_view_t key) const
{
return find_key(key, std::integral_constant<bool, Editable> {});
}
NLOHMANN_VIEW_ALWAYS_INLINE const node* find_key(string_view_t key, std::true_type /*editable*/) const
{
// (an editable document without edits is looked up as one that cannot be edited)
return unedited() ? detail::view::find_member<false>(*m_doc, m_node, key.data(), key.size())
: detail::view::find_member<true>(*m_doc, m_node, key.data(), key.size());
}
NLOHMANN_VIEW_ALWAYS_INLINE const node* find_key(string_view_t key, std::false_type /*editable*/) const
{
return detail::view::find_member<false>(*m_doc, m_node, key.data(), key.size());
}
/// the element of an array at an index below its size
NLOHMANN_VIEW_ALWAYS_INLINE basic_json_view element(size_type idx) const
{
if (unedited())
{
return basic_json_view(m_doc, detail::view::element_at<false>(*m_doc, m_node, idx));
}
return basic_json_view(m_doc, navigation::value(detail::view::element_at<Editable>(*m_doc, m_node, idx)));
}
// --- get() dispatch --- // --- get() dispatch ---
bool get_impl(detail::view::value_tag<bool> /*unused*/, detail::priority_tag<2> /*unused*/) const bool get_impl(detail::view::value_tag<bool> /*unused*/, detail::priority_tag<2> /*unused*/) const
{ {
if (NLOHMANN_VIEW_UNLIKELY(!is_boolean())) if (NLOHMANN_VIEW_UNLIKELY(!is_boolean()))
{ {
detail::view::throw_type_error(detail::exception_id::type_mismatch, "type must be boolean, but is ", type_name()); detail::view::throw_type_error(302, "type must be boolean, but is ", type_name());
} }
return (m_node->flags & detail::view::node_flags::is_true) != 0; return (m_node->flags & detail::view::node_flags::is_true) != 0;
} }
@@ -805,7 +869,7 @@ class basic_json_view
{ {
if (NLOHMANN_VIEW_UNLIKELY(m_node == nullptr)) if (NLOHMANN_VIEW_UNLIKELY(m_node == nullptr))
{ {
detail::view::throw_type_error(detail::exception_id::type_mismatch, "type must be number, but is ", type_name()); detail::view::throw_type_error(302, "type must be number, but is ", type_name());
} }
return detail::view::arithmetic_value<T, BasicJsonType>(*m_doc, *m_node); return detail::view::arithmetic_value<T, BasicJsonType>(*m_doc, *m_node);
} }
@@ -814,7 +878,7 @@ class basic_json_view
{ {
if (NLOHMANN_VIEW_UNLIKELY(!is_null())) if (NLOHMANN_VIEW_UNLIKELY(!is_null()))
{ {
detail::view::throw_type_error(detail::exception_id::type_mismatch, "type must be null, but is ", type_name()); detail::view::throw_type_error(302, "type must be null, but is ", type_name());
} }
return nullptr; return nullptr;
} }
@@ -1123,7 +1187,7 @@ class basic_json_document
{ {
if (NLOHMANN_VIEW_UNLIKELY(!m_data || m_data->discarded)) if (NLOHMANN_VIEW_UNLIKELY(!m_data || m_data->discarded))
{ {
detail::view::throw_type_error(detail::exception_id::document_image_unsupported, "cannot save a discarded json_document"); detail::view::throw_type_error(320, "cannot save a discarded json_document");
} }
return detail::view::save_image(*m_data); return detail::view::save_image(*m_data);
} }
@@ -1224,7 +1288,7 @@ class basic_json_document
const std::size_t idx = pointer_index(token); const std::size_t idx = pointer_index(token);
if (idx >= 0xFFFFFFFFu) if (idx >= 0xFFFFFFFFu)
{ {
detail::view::throw_array_index_out_of_range(idx); detail::view::throw_out_of_range(401, detail::concat("array index ", std::to_string(idx), " is out of range"));
} }
for (std::size_t i = 0; i < idx; ++i) for (std::size_t i = 0; i < idx; ++i)
{ {
@@ -1280,7 +1344,7 @@ class basic_json_document
{ {
if (ptr.empty()) if (ptr.empty())
{ {
detail::view::throw_out_of_range(detail::exception_id::patch_on_root, "JSON pointer has no parent"); detail::view::throw_out_of_range(405, "JSON pointer has no parent");
} }
const view_type parent = root().at(ptr.parent_pointer()); const view_type parent = root().at(ptr.parent_pointer());
const auto& token = ptr.back(); const auto& token = ptr.back();
@@ -1300,7 +1364,7 @@ class basic_json_document
static_assert(Editable, "only an editable document can be edited: use basic_json_document<BasicJsonType, true> (json_editable_document)"); static_assert(Editable, "only an editable document can be edited: use basic_json_document<BasicJsonType, true> (json_editable_document)");
if (NLOHMANN_VIEW_UNLIKELY(!m_data || m_data->discarded)) if (NLOHMANN_VIEW_UNLIKELY(!m_data || m_data->discarded))
{ {
detail::view::throw_invalid_iterator(detail::exception_id::iterator_from_other_value, "view does not belong to this document"); detail::view::throw_invalid_iterator(202, "view does not belong to this document");
} }
return detail::view::editor<BasicJsonType, view_type>(*m_data); return detail::view::editor<BasicJsonType, view_type>(*m_data);
} }
@@ -1317,7 +1381,7 @@ class basic_json_document
{ {
if (idx < 0) if (idx < 0)
{ {
detail::view::throw_array_index_out_of_range(idx); detail::view::throw_out_of_range(401, detail::concat("array index ", std::to_string(idx), " is out of range"));
} }
return static_cast<std::size_t>(idx); return static_cast<std::size_t>(idx);
} }
@@ -1528,7 +1592,8 @@ class basic_json_document
// NOLINTBEGIN(readability-avoid-nested-conditional-operator): a constant expression of C++11 // NOLINTBEGIN(readability-avoid-nested-conditional-operator): a constant expression of C++11
append_input_impl(buf, std::forward<T>(input), std::integral_constant < int, detail::is_contiguous_byte_container<D>::value ? 0 append_input_impl(buf, std::forward<T>(input), std::integral_constant < int, detail::is_contiguous_byte_container<D>::value ? 0
: std::is_base_of<std::istream, D>::value ? 1 : std::is_base_of<std::istream, D>::value ? 1
: detail::view::is_adapter_input<D>::value ? 2 : 3 > {}); : detail::view::is_adapter_input<D>::value ? 2
: detail::view::is_c_stream<D>::value ? 4 : 3 > {});
// NOLINTEND(readability-avoid-nested-conditional-operator) // NOLINTEND(readability-avoid-nested-conditional-operator)
} }
@@ -1550,6 +1615,19 @@ class basic_json_document
detail::view::append_adapter(buf, std::move(input.ia)); detail::view::append_adapter(buf, std::move(input.ia));
} }
#ifndef JSON_NO_IO
static void append_input_impl(std::string& buf, std::FILE* file, std::integral_constant<int, 4> /*C stream*/)
{
if (file == nullptr)
{
// (as before: the library's adapter decides what a null stream yields)
detail::view::append_adapter(buf, detail::input_adapter(file));
return;
}
detail::view::append_file(buf, file);
}
#endif
template<typename T> template<typename T>
static void append_input_impl(std::string& buf, T&& input, std::integral_constant<int, 3> /*other*/) static void append_input_impl(std::string& buf, T&& input, std::integral_constant<int, 3> /*other*/)
{ {
include/nlohmann/ordered_map.hpp
+14 -15
View File
@@ -125,19 +125,6 @@ private:
Container::pop_back(); Container::pop_back();
} }
/// @brief remove the entry with the given key; returns the number of removed entries
template<class K>
size_type erase_impl(const K& key)
{
const auto it = find_impl(*this, key);
if (it != this->end())
{
erase_at(it);
return 1;
}
return 0;
}
public: public:
template<class V, detail::enable_if_t< template<class V, detail::enable_if_t<
detail::is_constructible<T, V>::value, int> = 0> detail::is_constructible<T, V>::value, int> = 0>
@@ -227,14 +214,26 @@ public:
size_type erase(const key_type& key) size_type erase(const key_type& key)
{ {
return erase_impl(key); const auto it = find_impl(*this, key);
if (it != this->end())
{
erase_at(it);
return 1;
}
return 0;
} }
template<class KeyType, detail::enable_if_t< template<class KeyType, detail::enable_if_t<
detail::is_usable_as_key_type<key_compare, key_type, KeyType>::value, int> = 0> detail::is_usable_as_key_type<key_compare, key_type, KeyType>::value, int> = 0>
size_type erase(KeyType && key) // NOLINT(cppcoreguidelines-missing-std-forward) size_type erase(KeyType && key) // NOLINT(cppcoreguidelines-missing-std-forward)
{ {
return erase_impl(key); const auto it = find_impl(*this, key);
if (it != this->end())
{
erase_at(it);
return 1;
}
return 0;
} }
iterator erase(iterator pos) iterator erase(iterator pos)
single_include/nlohmann/json.hpp
Generated
+2736 -1492
View File
File diff suppressed because it is too large. Load diff
single_include/nlohmann/json_view.hpp
Generated
+2344 -871
View File
File diff suppressed because it is too large. Load diff
src/modules/json.cppm
+1
View File
@@ -60,6 +60,7 @@ inline namespace json_literals
namespace detail namespace detail
{ {
using NLOHMANN_JSON_NAMESPACE::detail::json_sax_dom_callback_parser; using NLOHMANN_JSON_NAMESPACE::detail::json_sax_dom_callback_parser;
using NLOHMANN_JSON_NAMESPACE::detail::json_sax_dom_parser;
using NLOHMANN_JSON_NAMESPACE::detail::unknown_size; using NLOHMANN_JSON_NAMESPACE::detail::unknown_size;
} // namespace detail } // namespace detail
tests/src/unit-class_lexer.cpp
-115
View File
@@ -2042,118 +2042,3 @@ TEST_CASE("string scanning kernels")
CHECK(nlohmann::detail::read_eight_bytes(bytes.data() + 8) == 0xFF0F0E0D0C0B0A09u); CHECK(nlohmann::detail::read_eight_bytes(bytes.data() + 8) == 0xFF0F0E0D0C0B0A09u);
CHECK(nlohmann::detail::read_eight_bytes(reinterpret_cast<const char*>(bytes.data()) + 3) == 0x0B0A090807060504u); CHECK(nlohmann::detail::read_eight_bytes(reinterpret_cast<const char*>(bytes.data()) + 3) == 0x0B0A090807060504u);
} }
TEST_CASE("find_ascii_copyable_run agrees with is_ascii_copyable for every byte")
{
// the SWAR mask in find_ascii_copyable_run() encodes the predicate a
// second time; check every byte value at every position of a 24-byte buffer
// (three words) and for several lengths, with the byte at the end or inside
for (unsigned int value = 0; value < 256; ++value)
{
for (std::size_t position = 0; position < 24; ++position)
{
std::array<unsigned char, 24> buffer{};
buffer.fill(static_cast<unsigned char>('a'));
buffer[position] = static_cast<unsigned char>(value);
const bool copyable = nlohmann::detail::is_ascii_copyable(static_cast<unsigned char>(value));
for (const std::size_t n :
{
position, position + 1, std::size_t{24}
})
{
if (n > 24)
{
continue;
}
const std::size_t expected = (copyable || n <= position) ? n : position;
CAPTURE(value)
CAPTURE(position)
CAPTURE(n)
CHECK(nlohmann::detail::find_ascii_copyable_run(buffer.data(), n) == expected);
}
}
}
}
TEST_CASE("UTF-8 validators accept the same sequences")
{
// The library checks UTF-8 in several places: detail::decode (serializer),
// the per-lead-byte switch of the lexer (json::accept), validate_one_utf8
// (bulk string scan), utf8_shape, and the byte path of the BON8 reader.
// They must accept exactly the same sequences (RFC 3629, section 4).
using nlohmann::json;
// bytes for the third and fourth position: the first two pairs are checked
// against all validators, the others only against the cheap ones
const std::array<std::array<unsigned char, 2>, 6> tails =
{
{
{{0x80, 0x80}}, {{0xBF, 0xBF}}, {{0x7F, 0x80}}, {{0xC0, 0x80}}, {{0x80, 0x7F}}, {{0x80, 0xC0}}
}
};
for (unsigned lead = 0x80; lead <= 0xFF; ++lead)
{
const auto shape = nlohmann::detail::utf8_shape(static_cast<std::uint8_t>(lead));
for (unsigned second = 0; second <= 0xFF; ++second)
{
for (std::size_t t = 0; t < tails.size(); ++t)
{
const std::array<unsigned char, 4> bytes = {{static_cast<unsigned char>(lead), static_cast<unsigned char>(second), tails[t][0], tails[t][1]}};
// expectation from the shape: the sequence is bytes[0..length)
const std::size_t length = shape.length;
bool expected = length >= 2 && second >= shape.second_lo && second <= shape.second_hi;
for (std::size_t i = 2; expected && i < length; ++i)
{
expected = bytes[i] >= 0x80 && bytes[i] <= 0xBF;
}
// the DFA of the serializer
bool dfa = length >= 2;
if (dfa)
{
std::uint8_t state = nlohmann::detail::UTF8_ACCEPT;
std::uint32_t codep = 0;
for (std::size_t i = 0; i < length; ++i)
{
nlohmann::detail::decode(state, codep, bytes[i]);
}
dfa = state == nlohmann::detail::UTF8_ACCEPT;
}
// the bulk path: exactly the sequence, and one byte short
const std::size_t bulk = nlohmann::detail::validate_one_utf8(bytes.data(), length);
const std::size_t bulk_short = length > 0 ? nlohmann::detail::validate_one_utf8(bytes.data(), length - 1) : 0;
CAPTURE(lead)
CAPTURE(second)
CAPTURE(t)
CHECK((dfa == expected && bulk == (expected ? length : 0) && bulk_short == 0));
if (t < 2 && length >= 2)
{
// the lexer's byte path
const std::string sequence(reinterpret_cast<const char*>(bytes.data()), length); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
CHECK(json::accept("\"" + sequence + "\"") == expected);
// the byte path of the BON8 reader (a string ends at 0xFF)
if (lead >= 0xC2 && lead <= 0xF7)
{
std::vector<std::uint8_t> packed(bytes.begin(), bytes.begin() + static_cast<std::ptrdiff_t>(length));
packed.push_back(0xFF);
const json j = json::from_bon8(packed, true, false);
if (expected)
{
CHECK(j == json(sequence));
}
else
{
CHECK_FALSE((j.is_string() && j.get<std::string>() == sequence));
}
}
}
}
}
}
}
tests/src/unit-class_parser.cpp
+78
View File
@@ -2952,3 +2952,81 @@ TEST_CASE("diagnostic positions: value lifetime, input adapters, and SAX")
} }
} }
#endif #endif
TEST_CASE("accept() agrees with parse() on floats around the overflow limit")
{
// accept() does not convert a float that cannot overflow; the tokens
// around the limit (and beyond it) must still be rejected exactly as
// parse() rejects them
const std::vector<const char*> numbers =
{
"1e0", "123.456e-5", "1e-400", "-1e-400", "1e307", "1e308", "1e309", "-1e309",
"10e307", "100e307", "0.1e309", "0.1e310", "1E+308", "1E+309",
"1.7976931348623157e308", "1.7976931348623159e308", "1.8e308", "-1.7976931348623157e308",
"12345678901234567e291", "12345678901234567e292", "123456789012345678e290", "123456789012345678e291",
"1234567890123456789e289", "1234567890123456789e290", "9999999999999999999e289", "9999999999999999999e290",
"0.00000000000000000001e328", "0.00000000000000000001e329"
};
for (const char* number : numbers)
{
for (const std::string& text :
{
std::string(number), "[" + std::string(number) + "]", "{\"a\":" + std::string(number) + "}"
})
{
INFO(text);
const bool parses = !json::parse(text, nullptr, false).is_discarded();
CHECK(json::accept(text) == parses);
}
}
}
namespace
{
// a DOM parser that reports parse errors without throwing, as in the sax_no_exception example
template<typename BasicJsonType>
class quiet_dom_parser : public nlohmann::detail::json_sax_dom_parser<BasicJsonType, nlohmann::detail::string_input_adapter_type>
{
public:
explicit quiet_dom_parser(BasicJsonType& j)
: nlohmann::detail::json_sax_dom_parser<BasicJsonType, nlohmann::detail::string_input_adapter_type>(j, false)
{}
bool parse_error(std::size_t /*unused*/, const std::string& /*unused*/, const nlohmann::detail::exception& /*unused*/)
{
return false;
}
};
template<typename BasicJsonType>
std::string read_before_error(const std::string& text)
{
BasicJsonType j;
quiet_dom_parser<BasicJsonType> sax(j);
CHECK(!BasicJsonType::sax_parse(text, &sax));
return j.dump();
}
} // namespace
TEST_CASE("a json_sax_dom_parser keeps the values read before a parse error")
{
// a json_sax_dom_parser that a user constructs stores every value in place, so a
// handler that does not throw on a parse error sees what was read before it
const std::vector<std::pair<std::string, std::string>> cases =
{
{"[1,2,3,]", "[1,2,3]"},
{"[[1,[2,3],4],[5,", "[[1,[2,3],4],[5]]"},
{R"({"a":1,"b":[1,2,{"c":)", R"({"a":1,"b":[1,2,{"c":null}]})"},
{R"({"a":1,"a":)", R"({"a":1})"},
{"[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,]", "[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]"},
{R"({"k0":0,"k1":1,"k2":{"x":[true,)", R"({"k0":0,"k1":1,"k2":{"x":[true]}})"},
{R"([{},[],{"a":[]},x)", R"([{},[],{"a":[]}])"},
};
for (const auto& c : cases)
{
CAPTURE(c.first)
CHECK(read_before_error<json>(c.first) == c.second);
CHECK(read_before_error<nlohmann::ordered_json>(c.first) == c.second);
}
}
tests/src/unit-json_view.cpp
+127
View File
@@ -185,6 +185,33 @@ TEST_CASE("json_view")
// duplicate keys: the last value, at the position of the first key // duplicate keys: the last value, at the position of the first key
CHECK(materialized<json_document>(R"({"a":1,"b":2,"a":3})") == json::parse(R"({"a":1,"b":2,"a":3})")); CHECK(materialized<json_document>(R"({"a":1,"b":2,"a":3})") == json::parse(R"({"a":1,"b":2,"a":3})"));
CHECK(materialized<ordered_json_document>(R"({"a":1,"b":2,"a":3})").dump() == R"({"a":3,"b":2})"); CHECK(materialized<ordered_json_document>(R"({"a":1,"b":2,"a":3})").dump() == R"({"a":3,"b":2})");
// scalars (made without the handler), and nesting around the depth up to which the stack of open containers is local
for (const char* scalar :
{"null", "true", "false", "0", "-7", "18446744073709551615", "1.5e300", "-0.0", "\"\"", "\"short\"", "\"a string that is longer than fifteen bytes\"", R"("\u00e4\n")"
})
{
CAPTURE(scalar)
CHECK(materialized<json_document>(scalar) == json::parse(scalar));
const std::string in_array = std::string("[1,") + scalar + "]";
const json_document d = json_document::parse(in_array);
CHECK(d.root()[1].materialize() == json::parse(scalar));
CHECK(d.root()[1].get<json>() == json::parse(scalar));
}
for (const std::size_t levels :
{
std::size_t{1}, std::size_t{15}, std::size_t{16}, std::size_t{17}, std::size_t{32}, std::size_t{33}
})
{
CAPTURE(levels)
const std::string nested = std::string(levels, '[') + R"({"k":[1,{"j":2}]})" + std::string(levels, ']');
CHECK(materialized<json_document>(nested) == json::parse(nested));
std::string wide = R"({"a":)";
wide += nested;
wide += R"(,"b":)";
wide += nested;
wide += "}";
CHECK(materialized<json_document>(wide) == json::parse(wide));
}
// very deep nesting (iterative, as parse()) // very deep nesting (iterative, as parse())
const std::string deep = std::string(100000, '[') + std::string(100000, ']'); const std::string deep = std::string(100000, '[') + std::string(100000, ']');
// (one check per step, so that a failure shows which step it is) // (one check per step, so that a failure shows which step it is)
@@ -410,6 +437,31 @@ TEST_CASE("json_view")
std::istringstream rvalue_source("[1]"); std::istringstream rvalue_source("[1]");
CHECK(json_document::parse_copy(std::move(rvalue_source)).owns_source()); CHECK(json_document::parse_copy(std::move(rvalue_source)).owns_source());
// a C stream is read in blocks as well (more than one block, and a
// text whose length is a multiple of the block size)
for (const std::size_t extra :
{
std::size_t{0}, std::size_t{1}
})
{
std::FILE* long_file = std::tmpfile(); // NOLINT(cppcoreguidelines-owning-memory)
if (long_file != nullptr)
{
std::string padded = text;
padded.append(((4096 - (padded.size() % 4096)) % 4096) + extra, ' ');
CHECK(std::fwrite(padded.data(), 1, padded.size(), long_file) == padded.size());
CHECK(std::fseek(long_file, 0, SEEK_SET) == 0);
const json_document from_long_file = json_document::parse(long_file);
CHECK(from_long_file.source() == padded);
CHECK(from_long_file.root().materialize() == json::parse(text));
CHECK(std::fclose(long_file) == 0); // NOLINT(cppcoreguidelines-owning-memory)
}
}
{
json_document from_null;
CHECK_THROWS_AS(from_null = json_document::parse(static_cast<std::FILE*>(nullptr)), json::parse_error&);
}
// a stream without a buffer is an empty input // a stream without a buffer is an empty input
std::istream no_buffer(nullptr); std::istream no_buffer(nullptr);
CHECK(json_document::parse(no_buffer, false).is_discarded()); CHECK(json_document::parse(no_buffer, false).is_discarded());
@@ -1006,6 +1058,49 @@ TEST_CASE("json_view element access and iteration")
#endif #endif
} }
SECTION("arrays whose elements have no subtrees: index and back()")
{
// (the elements are adjacent in the index; an array, an object, or an
// empty container anywhere makes the lookup skip element by element)
const std::vector<std::string> texts =
{
"[]", "[1]", "[1,2,3]", "[1,\"two\",3.5,true,null,-7,18446744073709551615]", "[[]]", "[{}]", "[1,[2],3]",
"[1,2,{\"a\":3}]", "[[1,2],3,4]", "[1,2,3,[4,5,6,7,8],9,10,11,12]", "[{\"a\":[1]},2]", R"(["a\u00e4","b"])"
};
for (const std::string& text : texts)
{
CAPTURE(text)
const json_document d = json_document::parse(text);
const json_view v = d.root();
const json expected = json::parse(text);
for (std::size_t i = 0; i < expected.size(); ++i)
{
CHECK(v[i].materialize() == expected[i]);
CHECK(v.at(i).materialize() == expected[i]);
}
CHECK(v[expected.size()].is_discarded());
if (!expected.empty())
{
CHECK(v.back().materialize() == expected.back());
}
}
// a long array (the direct lookup is not limited to short ones), also with a container at its end
std::string text = "[0";
for (int i = 1; i < 1000; ++i)
{
text += "," + std::to_string(i);
}
const json_document scalars = json_document::parse(text + "]");
CHECK(scalars.root()[999].get<int>() == 999);
CHECK(scalars.root()[500].get<int>() == 500);
CHECK(scalars.root().at(1).get<int>() == 1);
CHECK(scalars.root().back().get<int>() == 999);
const json_document container_last = json_document::parse(text + ",[1000]]");
CHECK(container_last.root()[999].get<int>() == 999);
CHECK(container_last.root()[1000][0].get<int>() == 1000);
CHECK(container_last.root().back()[0].get<int>() == 1000);
}
SECTION("iterators") SECTION("iterators")
{ {
const json_document d = json_document::parse(R"({"x":[1,{"y":2}],"z":null})"); const json_document d = json_document::parse(R"({"x":[1,{"y":2}],"z":null})");
@@ -1045,6 +1140,28 @@ TEST_CASE("json_view element access and iteration")
CHECK((*first++).key() == "0"); CHECK((*first++).key() == "0");
CHECK((*first).key() == "1"); CHECK((*first).key() == "1");
CHECK(++first == items.end()); CHECK(++first == items.end());
// the keys of array elements are their indexes, whether or not they are asked for
std::string numbers = "[0";
for (int i = 1; i < 1234; ++i)
{
numbers += "," + std::to_string(i);
}
const json_document array = json_document::parse(numbers + "]");
std::size_t position = 0;
for (const auto& item : array.root().items())
{
CHECK(item.value().get<std::size_t>() == position);
if (position % 3 != 0) // (not for every item: values alone do not make the key)
{
CHECK(item.key() == std::to_string(position));
CHECK(item.key().data() == item.key().data()); // (the same text every time)
const json_view::item copy = item;
CHECK(copy.key() == std::to_string(position));
}
++position;
}
CHECK(position == 1234);
#ifdef JSON_HAS_CPP_17 #ifdef JSON_HAS_CPP_17
std::string pairs; std::string pairs;
for (const auto [key, value] : d.root().items()) for (const auto [key, value] : d.root().items())
@@ -1383,7 +1500,17 @@ TEST_CASE("json_view values")
CHECK(exception_of([&] { static_cast<void>(v.value("s", 0)); }) == without_path(exception_of([&] { static_cast<void>(j.value("s", 0)); }))); CHECK(exception_of([&] { static_cast<void>(v.value("s", 0)); }) == without_path(exception_of([&] { static_cast<void>(j.value("s", 0)); })));
CHECK(exception_of([&] { static_cast<void>(v["n"].value("x", 0)); }) == without_path(exception_of([&] { static_cast<void>(j["n"].value("x", 0)); }))); CHECK(exception_of([&] { static_cast<void>(v["n"].value("x", 0)); }) == without_path(exception_of([&] { static_cast<void>(j["n"].value("x", 0)); })));
CHECK(exception_of([&] { static_cast<void>(v["n"].value(json::json_pointer("/x"), 0)); }) == without_path(exception_of([&] { static_cast<void>(j["n"].value(json::json_pointer("/x"), 0)); }))); CHECK(exception_of([&] { static_cast<void>(v["n"].value(json::json_pointer("/x"), 0)); }) == without_path(exception_of([&] { static_cast<void>(j["n"].value(json::json_pointer("/x"), 0)); })));
// string defaults: the member is not a string, or the value is neither an object nor an array
CHECK(exception_of([&] { static_cast<void>(v.value("n", "default")); }) == without_path(exception_of([&] { static_cast<void>(j.value("n", "default")); })));
CHECK(exception_of([&] { static_cast<void>(v["n"].value("x", "default")); }) == without_path(exception_of([&] { static_cast<void>(j["n"].value("x", "default")); })));
CHECK(exception_of([&] { static_cast<void>(v.value(json::json_pointer("/n"), "default")); }) == without_path(exception_of([&] { static_cast<void>(j.value(json::json_pointer("/n"), "default")); })));
CHECK(exception_of([&] { static_cast<void>(v["s"].value(json::json_pointer("/x"), "default")); }) == without_path(exception_of([&] { static_cast<void>(j["s"].value(json::json_pointer("/x"), "default")); })));
#endif #endif
// (defaults and members longer than the short string optimization)
CHECK(v.value("s", "a default longer than fifteen characters") == "text");
CHECK(v.value("missing", "a default longer than fifteen characters") == "a default longer than fifteen characters");
CHECK(v.value(json::json_pointer("/s"), "a default longer than fifteen characters") == "text");
CHECK(v.value(json::json_pointer("/zz"), "a default longer than fifteen characters") == "a default longer than fifteen characters");
} }
} }
tests/src/unit-json_view_builder.cpp
+52
View File
@@ -523,6 +523,58 @@ TEST_CASE("json_view builder: strings across vector blocks")
} }
} }
TEST_CASE("json_view builder: escapes between runs of text")
{
// The text between escapes is copied 16 bytes at a time where there are
// that many bytes left, and with the UTF-8 check where there is
// non-ASCII text. Runs of every length from 0 to 40 between valid and
// invalid escapes, UTF-8 sequences, and control characters, in strings
// that are cut at all places near the end of the text.
const std::array<const char*, 9> valid = {{"\\n", "\\\"", "\\\\", "\\/", "\\t", "\\u00e4", "\\u20ac", "\\ud83d\\ude00", "\\u0000"}};
const std::array<const char*, 9> invalid = {{"\\x", "\\ud800", "\\udc00", "\\u12G4", "\\u12", "\\ud83dx", "\\ud83d\\u0041", "\\", "\x01"}};
const std::array<const char*, 6> utf8 = {{"\xc3\xa4", "\xe2\x82\xac", "\xf0\x9f\x98\x80", "\xc3", "\xff", "\x80"}};
std::mt19937 rng(2026); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed): reproducible
for (std::size_t run = 0; run <= 40; ++run)
{
for (int i = 0; i < 60; ++i)
{
std::string content = std::string(rng() % 20, 'a');
const auto pieces = 1 + (rng() % 6);
for (unsigned k = 0; k < pieces; ++k)
{
content += valid[rng() % valid.size()];
content += std::string(k == 0 ? run : rng() % 41, static_cast<char>('b' + (rng() % 20)));
switch (rng() % 8)
{
case 0:
content += utf8[rng() % 3];
break;
case 1:
content += utf8[rng() % utf8.size()];
break;
case 2:
content += invalid[rng() % invalid.size()];
break;
default:
break;
}
}
check_string(content);
check_string(content.substr(0, content.size() - (rng() % 4)));
}
}
// the text before the first escape, and runs and escapes at the end of the text
for (std::size_t before = 0; before <= 40; ++before)
{
for (const char* escape : valid)
{
check_string(std::string(before, 'a') + escape);
check_string(std::string(before, 'a') + escape + std::string(before % 19, 'z'));
check_string(std::string(before, 'a') + escape + escape + std::string(before % 19, 'z'));
}
}
}
TEST_CASE("json_view node integer bits") TEST_CASE("json_view node integer bits")
{ {
using nlohmann::detail::view::integer_bits; using nlohmann::detail::view::integer_bits;
tests/src/unit-json_view_dump.cpp
+449
View File
@@ -21,6 +21,7 @@ using nlohmann::ordered_json_view;
using json_view_test::generator; using json_view_test::generator;
using json_view_test::has_duplicate_keys; using json_view_test::has_duplicate_keys;
#include <algorithm>
#include <array> #include <array>
#include <cmath> #include <cmath>
#include <cstddef> #include <cstddef>
@@ -29,6 +30,7 @@ using json_view_test::has_duplicate_keys;
#include <iomanip> #include <iomanip>
#include <map> #include <map>
#include <random> #include <random>
#include <set>
#include <sstream> #include <sstream>
#include <string> #include <string>
#include <vector> #include <vector>
@@ -186,6 +188,133 @@ TEST_CASE("json_view dump")
CHECK(deep_document.root().dump() == deep); CHECK(deep_document.root().dump() == deep);
} }
SECTION("ensure_ascii: a character that needs an escape at each position of a string")
{
// string lengths around the width of the vector steps, in a string value, a key, and a value near the end of the text
for (const char* const special :
{
"\xC3\xA9", "\xE4\xB8\xAD", "\xF0\x9F\x98\x80", "\x7F", "\\n", "\\u0001", "\\\"", "\\\\"
})
{
for (std::size_t length = 0; length <= 40; ++length)
{
for (std::size_t position = 0; position <= length; ++position)
{
std::string content(length, 'a');
content.insert(position, special);
const std::array<std::string, 3> texts =
{
{
"\"" + content + "\"",
"[\"" + content + "\"]",
"{\"" + content + "\":1}",
}
};
for (const std::string& text : texts)
{
CAPTURE(text)
const ordered_json_document d = ordered_json_document::parse(text);
const ordered_json j = ordered_json::parse(text);
CHECK(d.root().dump(-1, ' ', true) == j.dump(-1, ' ', true));
CHECK(d.root().dump(2, ' ', true) == j.dump(2, ' ', true));
CHECK(d.root().dump() == j.dump());
}
}
}
}
}
SECTION("ensure_ascii: every character of the Basic Multilingual Plane and some beyond")
{
// (the two- and three-byte sequences, including the ranges that need a check: E0, ED, EF BF BF)
std::string text = "[";
std::size_t in_chunk = 0;
const auto append = [&](std::uint32_t cp)
{
if (cp < 0x80)
{
if (cp < 0x20 || cp == '"' || cp == '\\')
{
return;
}
text += static_cast<char>(cp);
}
else if (cp < 0x800)
{
text += static_cast<char>(0xC0 | (cp >> 6));
text += static_cast<char>(0x80 | (cp & 0x3F));
}
else if (cp < 0x10000)
{
text += static_cast<char>(0xE0 | (cp >> 12));
text += static_cast<char>(0x80 | ((cp >> 6) & 0x3F));
text += static_cast<char>(0x80 | (cp & 0x3F));
}
else
{
text += static_cast<char>(0xF0 | (cp >> 18));
text += static_cast<char>(0x80 | ((cp >> 12) & 0x3F));
text += static_cast<char>(0x80 | ((cp >> 6) & 0x3F));
text += static_cast<char>(0x80 | (cp & 0x3F));
}
};
for (std::uint32_t cp = 0x20; cp < 0x10400; ++cp)
{
if (cp >= 0xD800 && cp < 0xE000)
{
continue;
}
if (in_chunk == 0)
{
text += text.size() > 1 ? ",\"" : "\"";
}
append(cp);
if (++in_chunk == 37)
{
text += "\"";
in_chunk = 0;
}
}
text += in_chunk != 0 ? "\"]" : "]";
const ordered_json_document d = ordered_json_document::parse(text);
const ordered_json j = ordered_json::parse(text);
CHECK(d.root().dump(-1, ' ', true) == j.dump(-1, ' ', true));
CHECK(d.root().dump(2, ' ', true) == j.dump(2, ' ', true));
CHECK(d.root().dump() == j.dump());
CHECK(d.root().dump(2) == j.dump(2));
}
SECTION("deep nesting, indented")
{
// indentations of up to and beyond the width of the writer's fixed-size copy
for (const int depth :
{
1, 5, 20, 70, 300
})
{
std::string text;
for (int i = 0; i < depth; ++i)
{
text += i % 2 == 0 ? "[1," : R"({"k":2,"\u00e9":)";
}
text += "[]";
for (int i = depth - 1; i >= 0; --i)
{
text += i % 2 == 0 ? "]" : "}";
}
const ordered_json_document d = ordered_json_document::parse(text);
const ordered_json j = ordered_json::parse(text);
for (const int indent :
{
0, 1, 2, 3, 8
})
{
CHECK(d.root().dump(indent) == j.dump(indent));
CHECK(d.root().dump(indent, '\t') == j.dump(indent, '\t'));
}
}
}
SECTION("the output buffer of a small value is small") SECTION("the output buffer of a small value is small")
{ {
// an escaped key after the value: its node lies in the arena, so the // an escaped key after the value: its node lies in the arena, so the
@@ -345,6 +474,35 @@ TEST_CASE("json_view comparison")
} }
} }
SECTION("objects whose first keys differ or repeat")
{
const std::array<const char*, 17> texts =
{
{
R"({"a":1,"b":2})", R"({"a":1,"c":2})", R"({"b":1,"a":2})", R"({"a":1,"a":2,"b":3})", R"({"a":2,"b":3})",
R"({"a":1,"b":1,"a":2})", R"({"a":2,"b":1})", R"({"a":1,"a":2})", R"({"b":1})", R"({"a":1,"b":2,"c":3,"d":4,"e":5})",
R"({"a":1,"b":2,"c":3,"d":4,"f":5})", R"({"a":1,"b":2,"c":3,"a":4,"e":5})", R"({"a":4,"b":2,"c":3,"e":5})",
R"({"\u0061":1,"b":2})", R"({})", R"([{"a":1,"b":2}])", R"([{"a":1,"c":2}])"
}
};
for (const char* const x : texts)
{
for (const char* const y : texts)
{
CAPTURE(x)
CAPTURE(y)
const ordered_json_document dx = ordered_json_document::parse(x);
const ordered_json_document dy = ordered_json_document::parse(y);
const ordered_json_editable_document ex = ordered_json_editable_document::parse(x);
const ordered_json jx = ordered_json::parse(x);
const ordered_json jy = ordered_json::parse(y);
CHECK((dx.root() == dy.root()) == (jx == jy));
CHECK((dx.root() == jy) == (jx == jy));
CHECK((ex.root() == dy.root()) == (jx == jy));
}
}
}
SECTION("numbers, duplicate keys, member order") SECTION("numbers, duplicate keys, member order")
{ {
const auto same = [](const char* x, const char* y) const auto same = [](const char* x, const char* y)
@@ -385,6 +543,151 @@ TEST_CASE("json_view comparison")
CHECK(!(null_document.root() == discarded)); CHECK(!(null_document.root() == discarded));
} }
SECTION("values that are the same text, or the same value written differently, at any depth")
{
// nested arrays and objects around depths where the walk over equal text gives up
for (const int depth :
{
1, 5, 47, 48, 49, 50, 60, 300
})
{
CAPTURE(depth)
std::string head;
std::string tail;
for (int i = 0; i < depth; ++i)
{
head += i % 2 == 0 ? R"({"k":[1,"s",{"x":true},)" : R"([null,2.5,{"y":)";
tail.insert(0, i % 2 == 0 ? "]}" : "}]");
}
// the same text, one number changed, and one number written differently
const auto with_number = [&](const char* number)
{
std::string result = head;
result += number;
result += tail;
return result;
};
const std::string text = with_number("7");
const std::string changed = with_number("8");
const std::string spelled = with_number("7.0");
const json_document a = json_document::parse(text);
const json_document b = json_document::parse(text);
const json_document c = json_document::parse(changed);
const json_document d = json_document::parse(spelled);
CHECK(a.root() == b.root());
CHECK(a.root() != c.root());
CHECK(c.root() != a.root());
CHECK(a.root() == d.root());
CHECK(d.root() == a.root());
CHECK(a.root() == json::parse(text));
CHECK(a.root() != json::parse(changed));
CHECK(json::parse(spelled) == a.root());
const nlohmann::json_editable_document e = nlohmann::json_editable_document::parse(text);
CHECK(a.root() == e.root());
CHECK(e.root() == a.root());
CHECK(c.root() != e.root());
CHECK(e.root() == e.root());
const ordered_json_document oa = ordered_json_document::parse(text);
const ordered_json_document oc = ordered_json_document::parse(changed);
CHECK(oa.root() != oc.root());
CHECK(oa.root() == ordered_json::parse(text));
CHECK(oc.root() != ordered_json::parse(text));
}
// not the same text, but the same value
const auto same = [](const char* x, const char* y)
{
const json_document dx = json_document::parse(x);
const json_document dy = json_document::parse(y);
const bool equal = dx.root() == dy.root();
CHECK(equal == (json::parse(x) == json::parse(y)));
CHECK(equal == (dx.root() == json::parse(y)));
return equal;
};
CHECK(same(R"({"a":1,"a":2})", R"({"a":3,"a":2})"));
CHECK(same(R"({"a":1,"b":[1,{"c":2,"c":3}]})", R"({"b":[1,{"c":9,"c":3}],"a":1})"));
CHECK(!same(R"({"a":1,"a":2})", R"({"a":2,"a":3})"));
CHECK(same("[1.0,-0,0,1e2]", "[1.00,0,-0,100.0]"));
CHECK(same(R"(["\u00e9","\n"])", "[\"\xc3\xa9\",\"\\n\"]"));
CHECK(!same(R"({"a":[1,2,{"b":null}]})", R"({"a":[1,2,{"b":false}]})"));
CHECK(!same(R"({"a":[1,2,{"b":null}]})", R"({"a":[1,2,{"c":null}]})"));
CHECK(!same(R"({"a":[1,2,{"b":null}]})", R"({"a":[1,2,{"b":null,"c":1}]})"));
// edited documents: moved sequences, links, and values of new nodes
nlohmann::json_editable_document edited = nlohmann::json_editable_document::parse(R"({"a":[1,2,3],"b":{"c":"d"},"e":0.5})");
const json_document plain = json_document::parse(R"({"a":[1,2,3,4],"b":{"c":"x","y":null},"e":0.5,"z":[]})");
edited.push_back(edited.root()["a"], 4);
edited.set(edited.root()["b"], "c", "x");
edited.set(edited.root()["b"], "y", nullptr);
edited.set(edited.root(), "z", json::array());
CHECK(edited.root() == plain.root());
CHECK(plain.root() == edited.root());
edited.erase(edited.root()["a"], 0);
CHECK(edited.root() != plain.root());
CHECK(edited.root()["a"] == json::parse("[2,3,4]"));
const json_document tail = json_document::parse("[2,3,4]");
CHECK(edited.root()["a"] == tail.root());
}
SECTION("large documents, with a difference at any place")
{
const auto make = [](std::size_t count, std::size_t changed, bool swap_members)
{
std::string text = R"({"b":1,"list":[)";
for (std::size_t i = 0; i < count; ++i)
{
text += (i != 0 ? "," : "");
const std::string id = std::to_string(i == changed ? i + 1 : i);
if (swap_members && i % 3 == 0)
{
text += R"({"v":[1.5,"s",{"k":)";
text += id;
text += R"(}],"id":)";
text += id;
text += "}";
}
else
{
text += R"({"id":)";
text += id;
text += R"(,"v":[1.5,"s",{"k":)";
text += id;
text += "}]}";
}
}
return text + R"(],"a":{"y":2,"x":3}})";
};
const std::size_t count = 3000;
for (const std::size_t changed :
{
count, std::size_t{0}, std::size_t{1}, count / 2, count - 1
})
{
for (const bool swap :
{
false, true
})
{
CAPTURE(changed)
CAPTURE(swap)
const std::string text_a = make(count, count, false);
const std::string text_b = make(count, changed, swap);
const json_document a = json_document::parse(text_a);
const json_document b = json_document::parse(text_b);
const bool equal = changed == count;
CHECK((a.root() == b.root()) == equal);
CHECK((b.root() == a.root()) == equal);
CHECK((a.root() == json::parse(text_b)) == equal);
CHECK((json::parse(text_b) == a.root()) == equal);
CHECK((a.root()["list"] == b.root()["list"]) == equal);
const ordered_json_document oa = ordered_json_document::parse(text_a);
const ordered_json_document ob = ordered_json_document::parse(text_b);
CHECK((oa.root() == ob.root()) == (equal && !swap));
CHECK((oa.root() == ordered_json::parse(text_b)) == (equal && !swap));
}
}
}
SECTION("deep nesting") SECTION("deep nesting")
{ {
const std::string deep = std::string(100000, '[') + std::string(100000, ']'); const std::string deep = std::string(100000, '[') + std::string(100000, ']');
@@ -598,6 +901,152 @@ TEST_CASE("json_view large objects")
CHECK(without_collisions.memory_usage() >= with_collisions.memory_usage() + (table_size * sizeof(std::uint32_t))); CHECK(without_collisions.memory_usage() >= with_collisions.memory_usage() + (table_size * sizeof(std::uint32_t)));
} }
SECTION("keys that differ in their last bytes are spread over the table")
{
// The hash must spread the usual families of keys (numbered names, hex
// strings, dates, short names): a key farther than index_max_displacement
// from its home slot would leave the object without a table. The keys
// are placed with linear probing, as the table does.
const auto max_displacement = [](const std::vector<std::string>& keys)
{
std::size_t slots = 16;
while (slots < 2 * keys.size())
{
slots *= 2;
}
std::vector<bool> used(slots, false);
std::size_t farthest = 0;
for (const std::string& key : keys)
{
std::size_t i = nlohmann::detail::conditional_static_cast<std::size_t>(nlohmann::detail::view::key_hash(key.data(), key.size())) & (slots - 1);
std::size_t distance = 0;
while (used[i])
{
i = (i + 1) & (slots - 1);
++distance;
}
used[i] = true;
farthest = (std::max)(farthest, distance);
}
return farthest;
};
// the keys of a printf-like pattern with one %d, %0Nd or %0Nx, formatted with a stream
// (a format string that is not a literal is a Flawfinder error)
const auto numbered = [](const char* format, int count, int step)
{
const std::string pattern(format);
const std::size_t percent = pattern.find('%');
std::size_t conversion = percent + 1;
std::size_t width = 0;
while (pattern[conversion] >= '0' && pattern[conversion] <= '9')
{
width = (width * 10) + static_cast<std::size_t>(pattern[conversion] - '0');
++conversion;
}
const bool hex = pattern[conversion] == 'x';
std::vector<std::string> keys;
for (int i = 0; i < count; ++i)
{
std::ostringstream key;
key << pattern.substr(0, percent) << std::setfill('0') << std::setw(static_cast<int>(width))
<< (hex ? std::hex : std::dec) << (i * step) << pattern.substr(conversion + 1);
keys.push_back(key.str());
}
return keys;
};
for (const char* format :
{"key%d", "%d", "item_%04d", "a%05d", "%03dxyz", "id%d", "user_%d", "%d_suffix", "0x%04x", "A%07d", "12345678%d", "1234567%d", "123456789%d",
"configuration_parameter_value_%04d_extra", "2026-01-%02dT00:00:00Z"
})
{
CAPTURE(format)
CHECK(max_displacement(numbered(format, std::string(format).find("%02d") != std::string::npos ? 60 : 1000, 1)) < nlohmann::detail::view::index_max_displacement); // NOLINT(abseil-string-find-str-contains)
CHECK(max_displacement(numbered(format, 600, 7)) < nlohmann::detail::view::index_max_displacement);
}
std::mt19937_64 rng(7); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed)
for (const std::size_t length :
{
std::size_t{1}, std::size_t{2}, std::size_t{3}, std::size_t{5}, std::size_t{7}, std::size_t{8}, std::size_t{9}, std::size_t{12}, std::size_t{16}, std::size_t{17}, std::size_t{33}
})
{
CAPTURE(length)
std::set<std::string> distinct;
// (there are 26 and 676 keys of one and two letters)
std::size_t wanted = 700;
if (length == 1)
{
wanted = 26;
}
else if (length == 2)
{
wanted = 500;
}
while (distinct.size() < wanted)
{
std::string key(length, 'a');
for (char& c : key)
{
c = static_cast<char>('a' + (rng() % 26));
}
distinct.insert(key);
}
CHECK(max_displacement(std::vector<std::string>(distinct.begin(), distinct.end())) < nlohmann::detail::view::index_max_displacement);
}
}
SECTION("keys of every length that differ in one byte")
{
// In a large object, a key is found by comparing its bytes with those of the member the
// table points to: up to 16 bytes as two overlapping words, longer ones with memcmp.
// Keys of every length from 0 to 40, and keys that differ from them in one byte.
std::mt19937_64 rng(2026); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed)
std::vector<std::string> keys;
std::set<std::string> distinct;
for (std::size_t length = 0; length <= 40; ++length)
{
for (int copy = 0; copy < 5; ++copy)
{
std::string key(length, 'a');
for (char& c : key)
{
c = static_cast<char>('a' + (rng() % 26));
}
if (distinct.insert(key).second)
{
keys.push_back(key);
}
}
}
std::string text = "{";
for (std::size_t i = 0; i < keys.size(); ++i)
{
text += (i != 0 ? ",\"" : "\"") + keys[i] + "\":" + std::to_string(i);
}
text += "}";
REQUIRE(keys.size() >= 128); // (a table is built for 128 members)
const json_document d = json_document::parse(text);
const json_view v = d.root();
for (std::size_t i = 0; i < keys.size(); ++i)
{
CAPTURE(keys[i])
CHECK(v[keys[i]].get<std::size_t>() == i);
CHECK(v.at(keys[i]).get<std::size_t>() == i);
CHECK(v.contains(keys[i]));
for (std::size_t at = 0; at < keys[i].size(); ++at)
{
std::string other = keys[i];
other[at] = other[at] == 'z' ? 'y' : 'z';
CAPTURE(other)
CHECK(v.contains(other) == (distinct.count(other) != 0));
}
CHECK(v.contains(keys[i] + "a") == (distinct.count(keys[i] + "a") != 0));
if (!keys[i].empty())
{
CHECK(v.contains(keys[i].substr(1)) == (distinct.count(keys[i].substr(1)) != 0));
}
}
}
SECTION("shrink_to_fit releases the tables' spare capacity") SECTION("shrink_to_fit releases the tables' spare capacity")
{ {
const auto make_text = [](int objects, int members) const auto make_text = [](int objects, int members)
tests/src/unit-json_view_edit.cpp
+109
View File
@@ -661,6 +661,115 @@ TEST_CASE("json_view edits: replacing arrays and objects by scalars")
CHECK(d.root().materialize() == expected); CHECK(d.root().materialize() == expected);
} }
TEST_CASE("json_view edits: arrays whose elements have no subtrees")
{
// such an array is indexed directly while it is in its parsed layout; edits
// that replace elements by scalars keep it, others move it
json_editable_document d = json_editable_document::parse("[10,20,30,40]");
json expected = json::array({10, 20, 30, 40});
const auto check = [&]
{
CHECK(d.root().size() == expected.size());
for (std::size_t i = 0; i < expected.size(); ++i)
{
CHECK(d.root()[i].materialize() == expected[i]);
CHECK(d.root().at(i).materialize() == expected[i]);
}
CHECK(d.root()[expected.size()].is_discarded());
CHECK(d.root().back().materialize() == expected.back());
CHECK(d.root().dump() == expected.dump());
};
check();
d.set(d.root()[1], "x");
expected[1] = "x";
check();
d.set(d.root()[3], 2.5);
expected[3] = 2.5;
check();
d.set(d.root()[2], json::array({1, 2}));
expected[2] = json::array({1, 2});
check();
d.push_back(d.root(), 50);
expected.push_back(50);
check();
d.insert(d.root(), 0, true);
expected.insert(expected.begin(), true);
check();
d.erase(d.root(), 1);
expected.erase(expected.begin() + 1);
check();
d.set(d.root()[2], nullptr);
expected[2] = nullptr;
check();
}
TEST_CASE("json_view edits: materializing values nested deeper than the document was parsed")
{
// the stack of open containers of materialize() is sized from the nesting of the parsed text;
// a value added by an edit can be deeper than that (and than the stack that is local to the call)
json_editable_document d = json_editable_document::parse("[1,[2]]");
json expected = json::array({1, json::array({2})});
json nested = 3;
for (int level = 0; level < 40; ++level)
{
nested = json::array({nested, json{{"k", level}}});
}
d.push_back(d.root(), nested);
expected.push_back(nested);
CHECK(d.root().materialize() == expected);
CHECK(d.root()[2].materialize() == nested);
d.set(d.root()[1][0], nested);
expected[1][0] = nested;
CHECK(d.root().materialize() == expected);
CHECK(d.root().get<json>() == expected);
}
TEST_CASE("json_view edits: an editable document before and after its first edit")
{
// until the first edit, an editable document is read like a read-only one; views made before the edit
// keep working after it
const std::string text = R"({"a":[1,2,{"x":"y"},4],"b":{"c":true,"d":null},"e":"text","f":[]})";
const json_document read_only = json_document::parse(text);
json_editable_document d = json_editable_document::parse(text);
const auto root = d.root();
const auto same_as_read_only = [&]
{
CHECK(root.dump() == read_only.root().dump());
CHECK(root.dump(2) == read_only.root().dump(2));
CHECK(root.materialize() == read_only.root().materialize());
CHECK(root["a"][2]["x"].get_string() == "y");
CHECK(root["a"].at(3).get<int>() == 4);
CHECK(root["a"].back().get<int>() == 4);
CHECK(root.contains("b"));
CHECK(!root.contains("zz"));
CHECK(root.find("e")->get_string() == "text");
CHECK(root.find("zz") == root.end());
CHECK(root["b"]["c"].get<bool>());
CHECK(root.value("e", "none") == "text");
std::size_t count = 0;
for (const auto m : root["a"])
{
CHECK(m.materialize() == read_only.root()["a"][count].materialize());
++count;
}
CHECK(count == 4);
};
same_as_read_only();
// an edit that changes a scalar in place, then one that moves an array
d.set(root["e"], "other");
CHECK(root["e"].get_string() == "other");
CHECK(root["a"][2]["x"].get_string() == "y");
CHECK(root["a"].back().get<int>() == 4);
d.push_back(root["f"], 7);
CHECK(root["f"][0].get<int>() == 7);
CHECK(root["f"].back().get<int>() == 7);
CHECK(root.dump() == R"({"a":[1,2,{"x":"y"},4],"b":{"c":true,"d":null},"e":"other","f":[7]})");
d.set(root["a"], 2, json::array({1, 2}));
CHECK(root["a"][2][1].get<int>() == 2);
CHECK(root["a"].dump() == "[1,2,[1,2],4]");
CHECK(root.materialize() == json::parse(R"({"a":[1,2,[1,2],4],"b":{"c":true,"d":null},"e":"other","f":[7]})"));
}
namespace namespace
{ {
// reads of "a" in an object that repeats it: the first member wins, however the // reads of "a" in an object that repeats it: the first member wins, however the
tests/src/unit-json_view_image.cpp
+78
View File
@@ -682,6 +682,84 @@ TEST_CASE("json_view images: ownership")
#if !defined(JSON_NOEXCEPTION) #if !defined(JSON_NOEXCEPTION)
TEST_CASE("json_view images: errors") TEST_CASE("json_view images: errors")
{ {
SECTION("float tokens: a changed byte is accepted if the text is still a number with the digits the node records")
{
for (const char* const token :
{
"-12.375", "0.5", "123456.0", "-0.0", "9999999999999999999.5", "0.000000000000000000000001", "1.25e10", "5E-3"
})
{
CAPTURE(token)
const std::string text = std::string("[") + token + "]";
const json_document d = json_document::parse(text);
const std::vector<std::uint8_t> image = d.save();
const std::size_t at = text_at(image) + 1; // the first byte of the token
const std::uint16_t recorded = node_at(image, 1).extra;
CHECK(loaded_dump(image) == json::parse(text).dump());
for (std::size_t pos = 0; token[pos] != '\0'; ++pos)
{
for (unsigned int byte = 0; byte < 256; ++byte)
{
std::vector<std::uint8_t> changed = image;
changed[at + pos] = static_cast<std::uint8_t>(byte);
std::string t = token;
t[pos] = static_cast<char>(byte);
// the digits of the integer part and of the fraction of the changed token
std::size_t integer = 0;
std::size_t fraction = 0;
std::size_t k = (t[0] == '-') ? 1 : 0;
for (; k < t.size() && t[k] >= '0' && t[k] <= '9'; ++k)
{
++integer;
}
if (k < t.size() && t[k] == '.')
{
for (++k; k < t.size() && t[k] >= '0' && t[k] <= '9'; ++k)
{
++fraction;
}
}
// (as a JSON text, the token may have white space around it)
const bool float_token = json::accept(t) && t.find_first_of(std::string(" \t\r\n\0", 5)) == std::string::npos && (t.find('.') != std::string::npos || t.find_first_of("eE") != std::string::npos) // NOLINT(abseil-string-find-str-contains)
&& ((integer | (fraction << 8U)) == recorded);
CAPTURE(pos)
CAPTURE(byte)
std::string result;
try
{
result = loaded_dump(changed);
}
catch (const json::parse_error& e)
{
CHECK(std::string(e.what()) == check_failed);
result = "rejected";
}
CHECK((result != "rejected") == float_token);
}
}
}
}
SECTION("invalid UTF-8 of a decoded string: the compact dump writes it, the others throw")
{
const json_document d = json_document::parse(R"({"k\n":"\u00e9x"})");
std::vector<std::uint8_t> image = d.save();
const std::size_t arena = text_at(image) + header_size_field(image, 16) + 1;
image[arena + 3] = 0x28; // the key is "k\n"; the value starts at 3: 0xC3 0x28 is invalid
const json_document loaded = json_document::load(image, image_check::bounds);
CHECK(loaded.root().dump().size() == 13);
for (const int indent :
{
0, 2
})
{
CHECK_THROWS_AS(loaded.root().dump(indent), json::type_error);
CHECK_THROWS_AS(loaded.root().dump(indent, ' ', true), json::type_error);
}
CHECK_THROWS_AS(loaded.root().dump(-1, ' ', true), json::type_error);
CHECK_THROWS_WITH_AS(loaded.root().dump(2), "[json.exception.type_error.316] invalid UTF-8 byte at index 1: 0x28", json::type_error&);
}
SECTION("a literal as the root: dump() after loading") SECTION("a literal as the root: dump() after loading")
{ {
for (const char* text : for (const char* text :
tests/src/unit-ordered_json.cpp
+278
View File
@@ -12,6 +12,11 @@
using nlohmann::json; using nlohmann::json;
using nlohmann::ordered_json; using nlohmann::ordered_json;
#include <array>
#include <cstddef>
#include <cstdint>
#include <string>
TEST_CASE("ordered_json") TEST_CASE("ordered_json")
{ {
json j; json j;
@@ -237,3 +242,276 @@ TEST_CASE("regression test for issue #5673 - ordered_json::emplace with a non-rv
CHECK(oj.dump() == "{\"a\":1}"); CHECK(oj.dump() == "{\"a\":1}");
} }
} }
namespace
{
// the sizes around which the DOM parser changes how it stores the elements of
// an array (16) and the members of an object in a vector (32, and 64 for the
// key index)
const std::array<std::size_t, 17> container_sizes = {{0, 1, 2, 3, 15, 16, 17, 18, 31, 32, 33, 34, 63, 64, 65, 66, 200}};
template<typename Json>
std::string array_text(const std::size_t n, const std::size_t depth)
{
std::string text = "[";
for (std::size_t i = 0; i < n; ++i)
{
text += (i == 0 ? "" : ",");
// alternate scalars, empty containers and nested containers
switch (i % 4)
{
case 0:
text += std::to_string(i);
break;
case 1:
text += "\"s" + std::to_string(i) + "\"";
break;
case 2:
text += depth == 0 ? "[]" : array_text<Json>(n / 2, depth - 1);
break;
default:
text += depth == 0 ? "{}" : "{\"a\":" + array_text<Json>(n / 3, depth - 1) + ",\"b\":null}";
break;
}
}
return text + "]";
}
template<typename Json>
std::string object_text(const std::size_t n, const std::size_t depth, const bool duplicates)
{
std::string text = "{";
for (std::size_t i = 0; i < n; ++i)
{
text += (i == 0 ? "" : ",");
text += "\"key" + std::to_string(i) + "\":";
switch (i % 4)
{
case 0:
text += std::to_string(i);
break;
case 1:
text += depth == 0 ? "[]" : array_text<Json>(n / 2, depth - 1);
break;
case 2:
text += depth == 0 ? "{}" : object_text<Json>(n / 2, depth - 1, duplicates);
break;
default:
text += "\"value\"";
break;
}
}
if (duplicates)
{
// keys that are there already, the first and the last one, with other values
text += n == 0 ? "" : ",";
text += R"("key0":[1,2],"key)";
text += std::to_string(n - 1);
text += R"(":{"again":true})";
}
return text + "}";
}
template<typename Json>
void check_same_dump(const std::string& text)
{
// the text is what dump() writes for the parsed value, and the value
// survives a second round trip: arrays keep their elements in order, an
// ordered_json keeps its members in order of appearance
const Json parsed = Json::parse(text);
const Json again = Json::parse(parsed.dump());
CHECK(parsed == again);
CHECK(parsed.dump() == again.dump());
CHECK(Json::accept(text));
}
} // namespace
TEST_CASE("parsing containers of sizes around the limits of the DOM parser")
{
SECTION("arrays")
{
for (const std::size_t n : container_sizes)
{
const std::string text = array_text<json>(n, 2);
INFO("size ", n);
const json parsed = json::parse(text);
CHECK(parsed.is_array());
CHECK(parsed.size() == n);
for (std::size_t i = 0; i < n; i += 4)
{
CHECK(parsed[i] == i);
}
check_same_dump<json>(text);
check_same_dump<ordered_json>(text);
CHECK(ordered_json::parse(text).dump() == parsed.dump());
}
}
SECTION("objects")
{
for (const bool duplicates :
{
false, true
})
{
for (const std::size_t n : container_sizes)
{
const std::string text = object_text<json>(n, 2, duplicates);
INFO("size ", n, " duplicates ", duplicates);
const json parsed = json::parse(text);
const ordered_json ordered = ordered_json::parse(text);
CHECK(parsed.is_object());
CHECK(ordered.is_object());
CHECK(parsed.size() == ordered.size());
// the same members, in the order of their first appearance
std::size_t i = 0;
for (auto it = ordered.begin(); it != ordered.end(); ++it, ++i)
{
if (i < n)
{
CHECK(it.key() == "key" + std::to_string(i));
}
CHECK(parsed.at(it.key()) == json(it.value()));
}
if (duplicates && n > 1)
{
// the later value replaced the earlier one in its place
CHECK(ordered.at("key0") == ordered_json::parse("[1,2]"));
CHECK(ordered.at("key" + std::to_string(n - 1)) == ordered_json::parse("{\"again\":true}"));
CHECK(ordered.begin().key() == "key0");
CHECK(ordered.size() == n);
}
check_same_dump<json>(text);
check_same_dump<ordered_json>(text);
}
}
}
SECTION("objects with many different and repeated keys")
{
// 300 members of which a third repeat an earlier key
std::string text = "{";
for (std::size_t i = 0; i < 300; ++i)
{
text += (i == 0 ? "" : ",");
text += "\"k" + std::to_string(i % 200) + "\":" + std::to_string(i);
}
text += '}';
const ordered_json ordered = ordered_json::parse(text);
const json parsed = json::parse(text);
CHECK(ordered.size() == 200);
CHECK(parsed.size() == 200);
std::size_t i = 0;
for (auto it = ordered.begin(); it != ordered.end(); ++it, ++i)
{
CHECK(it.key() == "k" + std::to_string(i));
// the last value written
CHECK(it.value() == (i < 100 ? i + 200 : i));
CHECK(parsed.at(it.key()) == json(it.value()));
}
}
}
namespace
{
// a deterministic source of structure for parsing tests: builds a JSON text
// and, at the same time, the value it has to parse to
class tree_generator
{
public:
explicit tree_generator(std::uint32_t seed) : state(seed) {}
template<typename Json>
Json make(std::string& text, const std::size_t depth)
{
switch (next(depth == 0 ? 5 : 9))
{
case 0:
text += "null";
return Json(nullptr);
case 1:
text += "true";
return Json(true);
case 2:
{
const int n = static_cast<int>(next(2000)) - 1000;
text += std::to_string(n);
return Json(n);
}
case 3:
text += "\"text\"";
return Json("text");
case 4:
text += "2.5";
return Json(2.5);
case 5:
case 6:
case 7:
{
Json array = Json::array();
text += '[';
const std::size_t n = size();
for (std::size_t i = 0; i < n; ++i)
{
text += (i == 0 ? "" : ",");
array.push_back(make<Json>(text, depth - 1));
}
text += ']';
return array;
}
default:
{
Json object = Json::object();
text += '{';
const std::size_t n = size();
for (std::size_t i = 0; i < n; ++i)
{
const std::string key = "k" + std::to_string(i);
text += (i == 0 ? "\"" : ",\"") + key + "\":";
object[key] = make<Json>(text, depth - 1);
}
text += '}';
return object;
}
}
}
private:
std::uint32_t next(const std::uint32_t bound)
{
state = (state * 1664525U) + 1013904223U;
return (state >> 8) % bound;
}
std::size_t size()
{
return container_sizes[next(static_cast<std::uint32_t>(container_sizes.size()))];
}
std::uint32_t state;
};
} // namespace
TEST_CASE("parsing random trees of arrays and objects")
{
for (std::uint32_t seed = 1; seed <= 150; ++seed)
{
INFO("seed ", seed);
std::string text;
std::string ordered_text;
const json expected = tree_generator(seed).make<json>(text, 3);
const auto ordered_expected = tree_generator(seed).make<ordered_json>(ordered_text, 3);
CHECK(text == ordered_text);
CHECK(json::parse(text) == expected);
CHECK(json::accept(text));
const ordered_json ordered = ordered_json::parse(text);
CHECK(ordered == ordered_expected);
CHECK(ordered.dump() == ordered_expected.dump());
CHECK(json::parse(text).dump() == expected.dump());
}
}