The files with code adapted from fast_float, Zmij and simdjson carried the
authors' SPDX copyright lines, but their file comments did not say what
was taken or under which license: number_parse.hpp, to_chars.hpp,
simd_scan.hpp and view/simd.hpp now do, as zmij.hpp and the yyjson-based
view files already did. README and the license page now also name
fast_float's eight-digit parsing and the lexer's use of simdjson's UTF-8
check. Comments only.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document the contract of object keys passed to json_sax::key and key callbacks
Describe what the name passed to json_sax::key and to a parser callback
for parse_event_t::key holds when parsing JSON text: escapes are decoded
to UTF-8, different spellings of a name yield identical strings, the name
is validated before the call (invalid escapes or UTF-8 give a parse error
and no key event), no normalization or case conversion is applied, the
event precedes the ':' check, and duplicate names are reported once per
occurrence. Also note that sax_parse passes binary-format keys unchanged.
See discussion #5955.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add a test for the object key contract and correct the BON8 note
Pin the guarantees documented for json_sax::key and the key event of
parser callbacks: decoded names, validation before delivery, delivery
before the colon, one event per repeated name, and the binary formats.
BON8 rejects ill-formed UTF-8 in a name even with sax_parse, so the
note no longer lists it with the formats that pass bytes through.
See discussion #5955.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the GCC build of the object key contract test
GCC 11 in C++11 mode rejects a default argument on a lambda parameter
(-Wc++14-extensions) and ignores a static_cast<void> on the
warn_unused_result of parse(). Pass the input format explicitly and bind
the parse result to a variable.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add version history for the object key contract
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: source positions in materialize() with JSON_DIAGNOSTIC_POSITIONS
With JSON_DIAGNOSTIC_POSITIONS, materialize() now gives every value the
start_pos()/end_pos() that basic_json::parse() would give it, so exceptions
about materialized values name the same bytes (#5864).
- json_sax_dom_parser takes the lexer type as a defaulted template
parameter; materialize() passes position_source, which reports the
positions it derives from the index. diagnostic_positions treats npos as
"no position" and keeps ranges all-or-nothing.
- Strings with escapes keep the offset of their opening quote plus one in
`next` (only with the macro); materialize() scans from there to the closing
quote. The end of an array or object is found by scanning from the end of
its last element over whitespace, comments and a trailing comma.
- In editable documents, values written by edits, arrays and objects whose
elements changed, and containers whose last element has no position get
none. Images keep the flag is_new, and load() checks that `next` of a
string is within the text.
Nothing changes on the parse path without the macro. The view's own
exceptions still carry no byte range, so the tests that compare them with
those of materialized values skip it under the macro, as they skip the path
under JSON_DIAGNOSTICS.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Indent comment in unit-json_view_from_json.cpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review: clearer position code and docs
- position_source gets clear(); set_scalar() returns early for new values,
switches on node_kind, and uses no local copies; the one position_source
per materialize() call is documented
- no npos + 1 for containers without a position; a comment on why the end
of a container is reported as its closing bracket
- less duplication under #if in materialize() and the builder
- diagnostic_positions sets npos explicitly
- docs: positions of basic_json values after changes, positions in images
depend on how the document was parsed, when next is set, and the cost of
positions while parsing
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Share the comment scanners of the view builder with materialize()
skip_line_comment() and find_block_comment_end() in scan.hpp are used by
the builder's comment() and by position_source::container_end(), which
had its own copy.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review: if/else for container positions, clearer next field
- the opening bracket of a container sets positions with an if/else instead
of a ternary, and the frame takes its start from positions
- the closing bracket of a container is found in nested if/else; the parent's
last_end is taken from positions, so it is npos whenever no range is set
- architecture.md: the next field reads "or 0 if it was not; all other
values: 0" instead of "else 0; otherwise 0"
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix GCC unused-result and clang-tidy raw-string findings in unit-json_view_positions.cpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix self-aliasing in json_pointer::operator/= and patch_inplace
json_pointer::operator/=(const json_pointer&) passed iterators into its
own token vector to insert() when a pointer was appended to itself
(p /= p), which is undefined behavior; libstdc++ produced "/a/b//" for
"/a/b". The pointer now appends a copy in that case.
basic_json::patch_inplace iterated the patch while changing *this, so
j.patch_inplace(j) read freed memory. It now applies a copy of the
patch when the patch is the value itself. A patch stored inside the
value remains undefined behavior and is documented as such; copying the
patch unconditionally would double the time of a large patch.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix JSON_DIAGNOSTICS test crash and clang-tidy hit in patch_inplace self-aliasing
The spare-capacity test reallocated non-empty elements through get_ref(), which leaves stale parent pointers under JSON_DIAGNOSTICS; reserve while the array is empty instead. Suppress performance-unnecessary-copy-initialization for the copy that is needed because *this is modified.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Detect a patch stored inside the patched value with JSON_DIAGNOSTICS
patch_inplace applied a copy only when the patch was the value itself. With JSON_DIAGNOSTICS the parent pointers make it cheap to also detect a patch nested inside the value, so that case is handled the same way. Without diagnostics it stays documented undefined behavior.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Silence misc-const-correctness for patch_aliases_this
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy findings in the operator/= self-append test
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Spell out that numbers comparing equal produce no operation even if one is
zero and the other negative zero, in diff.md and features/json_patch.md.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep the value unchanged when operator[] with a JSON pointer throws
The non-const operator[](json_pointer) is documented with strong exception
safety, but get_unchecked created intermediate values while walking the
pointer and kept them when a later reference token failed, e.g.
json j; j["/a/99999999999999999999999"_json_pointer] threw 404 and left
{"a":[]}; padding an existing array or appending with "-" behaved alike,
and so did std::length_error/std::bad_alloc from a huge index.
get_unchecked now only walks existing values up to the last reference
token. The last token's key insertion or array growth is strongly
exception safe by itself. Any other change goes through
get_unchecked_create, which records the one change it makes to an existing
value (null converted, key inserted, array grown) in a RAII guard and
reverts it if a later step throws; everything else it creates lies below
that change. The guard needs no try/catch, so JSON_NOEXCEPTION builds are
unaffected. Successful calls create the same values as before.
Docs: state the guarantee in the exceptions list and version history of
operator[], fix the complexity of the pointer overloads of operator[] and
at, and warn that padding arrays up to an untrusted index allocates memory
proportional to the index (operator[].md, untrusted_input.md).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the strongly exception-safe pointer operator[] faster than before
The previous commit made get_unchecked strongly exception safe but slower
with GCC 16 when it creates values (up to +6 % instructions for appending
with "-"). This restructures it so that no benchmark row is slower than
before the strong guarantee, with GCC 16, Clang 22 and Apple Clang.
- get_unchecked walks existing values in one loop that inserts nothing.
The last reference token is resolved in that walk: "-" appends with
emplace_back(), an index grows the array with operator[], and a missing
key is inserted with emplace_hint() at the position the walk's
lower_bound() found, so the key is searched once.
- With std::less<string_t> or std::less<> as comparator, the walk checks
the key at the lower bound for equality instead of equivalence, which is
the same for these comparators and cheaper.
- get_unchecked_create (not inlined) handles all other changes. Its guard
swaps a null value with a fresh null value instead of copying it and
reverts the change in a non-inlined function, so a successful call pays
one compare for it. As every null value converted afterwards is a fresh
one, it is converted in place instead of assigning a new array or object.
- Keys are added to new objects with emplace(), which also replaces the
basic_json::operator[] call for ordered_json.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix rollback of pointer operator[] for unusual object and string types
Follow-up to the strong exception guarantee of the non-const
operator[](json_pointer):
- get_unchecked_create assumed that emplace() always inserts the key that
the walk did not find. For object types whose lookup and emplace()
disagree (e.g. a copied fifo_map), nothing was inserted, the rollback
erased the existing member, and the walk continued with a value of the
wrong type (assertion, type_error.305, or a crash). It now checks the
result of emplace(), arms the rollback only if a key was inserted, and
otherwise continues the walk with the member found, as basic_json's
operator[] does: such pointers throw out_of_range.404 and leave the value
unchanged, as before the strong guarantee.
- The rollback erased the inserted key by key, which converts string_t to
object_t::key_type and may allocate inside a noexcept function
(std::terminate on bad_alloc with custom key types). It now erases by the
iterator emplace() returned, and shrinks a grown array with erase() at
the end instead of resize(). The documentation states the remaining
requirement for custom template parameters.
- Creating values no longer calls string_t::begin(), so the non-const
operator[](json_pointer) compiles with string types that lack it (as the
alternative string type of the tests).
- Correct the @throw lists of get_unchecked and get_unchecked_create.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review comments on operator[](json_pointer) rollback
- undo() shrinks the array with resize() instead of erase()
- comments say what "will be" done in get_unchecked_create
- tests: avoid a clang-tidy performance-unnecessary-copy-initialization
hit (unit-alt-string) and the word "access" in section names, which
Flawfinder flags as race/access
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Let the pointer rollback guard call undo() unconditionally
The check in the destructor and JSON_HEDLEY_NEVER_INLINE on undo() made no
measurable difference: instruction counts of j[ptr] are identical with
GCC 13 -O3 and within 0.4 % with clang, timings are within noise.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: new design for the documentation site
Apply the brand to the Material for MkDocs site: Inter and JetBrains Mono
served by the site (upright and real italic), brand colours, blue header band
with the syntax pattern, wordmark in the header, new home page (hero, cards,
animated feature tour), 404 page, polish for tabs, search, footer, focus
rings, tables and code blocks, and theme features (code copy, previous/next
footer, TOC follow, search suggestions).
Draw the number railroad diagram, the parser callback events and the iterator
illustrations as inline SVGs that follow the light and dark scheme. The
generator scripts are in docs/mkdocs/scripts/figures; the feature slideshow
(json-light.svg, json-dark.svg, replacing json.gif on the home page) is built
by docs/mkdocs/scripts/slideshow. Decision nodes in five flowcharts become
hexagons, which need far less room.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: add figures, table captions and a macro chooser
- Figures with numbered captions: SAX stepper, json_view layout and edit
stepper, exception map, JSON pointer and patch walkers, QA pipeline,
scaling and binary size charts, roadmap lanes, release timeline,
integration decision tree and namespace builder
- Table captions (nj-table-caption) on 9 pages
- Compile times: modules and PCH charts, filesystem section, re-measured
Figure 1, links between features/modules.md and compile_times.md
- Interactive macro chooser for features/arbitrary_types.md
- Link check: exclude URLs that answer the plugin with 400/401/419 or are
dead (customers.md)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: macro chooser generates adl_serializer for non-default-constructible types
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: cross-link concept pages and API pages in See also sections
Add See also links in both directions between the concept pages (features,
home, integration) and the API pages, write API links in code font, give every
entry a comment after a dash, and show the callback event figure on the parser
callbacks page.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: address review comments on the new design
- Home: add a "Latest release" button
- Sponsors: update the list and group it by tier
- Features: list all feature pages on the index page
- Releases: add highlights per release, link the 4.0 roadmap, add the first commit to the timeline
- Exceptions: prefix all example messages with the exception name
- API: no code font for format names in the See also links of the to_/from_ pages
- Merge Patch: show the document before, the patch, and the result
- Replace "--" with typographic dashes, mark external links with an icon, center Mermaid diagrams
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- Size the buffer of decoded strings on its first growth from the input (a
quarter of it plus 16 bytes, at most 4 KiB) instead of 272 bytes, so that a
document with escapes does not grow it twice, with a zero-filled copy each
time.
- Allocate the node index of documents with an estimate of up to 1024 nodes
(was 256) together with the header, which saves one allocation. When the
estimate is too low (dense numbers), that block stays unused: at most 16 KiB
instead of 4 KiB. shrink_to_fit() cannot release it separately; the
documentation says so.
Fresh parse of a 2.5 KB minified tweet: 993 to 958 ns (-3.5 %), traverse and
select -3 %, with no row consistently slower (Apple M1 Max, 30 rounds, three
runs). Dense numeric documents parse about 9 % faster and keep up to 16 KiB
more memory.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: writing generic code for json_view
A "Writing generic code" section on the json_view page: why a function
taking json_view accepts no other view, the three signatures for
editable, read-only, or either kind of view (passed by value), why the
modes do not convert, is_json_view, and how a from_json() reads views
(constrained to json values and views; the macros and the library's
conversions do it already) with what differs then (duplicate keys,
JSON_DIAGNOSTICS paths).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: name the documents the generic view function accepts
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: views of read-only documents convert to editable views
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: the library's std conversions read views directly
The library's from_json() overloads (containers, maps, std::array, C
arrays, pairs, tuples, optional, valarray, forward_list, enums, binary,
paths, and scalars) accept views: their traits use the view's
basic_json type, and the few accesses that differ (get_ptr) go through
small helpers. from_json() of a view now finds them; only the probe
still looks up by ADL only. get<T>() of a view thus converts these types
without materialize(), with the same values and exceptions; from an
object, a repeated key keeps the last value, as parse() does.
The view keeps its direct paths for vectors of the types it converts
directly (faster than from_json()) and for containers of views; its
other vector and map paths are replaced by from_json(). For basic_json,
std::valarray and std::forward_list are read in one pass (40 % and 3.5 %
fewer instructions); all other conversions are unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: enable_if_not_view_t instead of '!' in the from_json() templates
astyle spaces out the template argument list when it contains a negation;
the alias spells the condition with negation<> and leaves the lines alone.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: compact the template parameter lists astyle had spaced out
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
get<T>() and get_to() of a view call T's from_json() with the view itself
when that from_json() accepts views but not an unrelated probe type, and
fall back to materialize() otherwise. The from_json() functions of the
NLOHMANN_DEFINE_TYPE_*, NLOHMANN_DEFINE_DERIVED_TYPE_* and
NLOHMANN_JSON_SERIALIZE_ENUM(_STRICT) macros accept basic_json values and
views; a derived type whose base reads only json reads the base part
through materialize(). The library's own from_json() overloads are not
called with views: from_json() of a view (and of the probe) only looks up
functions by ADL.
New public trait nlohmann::is_json_view<T>, and the member type
basic_json_type of basic_json_view. With duplicate keys, the members of
such a T are those the view's lookups find (the first ones).
bench_decode.cpp measures decoding into structs (4-6x faster than
materialize().get<T>() on twitter, citm_catalog, and synthetic records).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The view's vector code follows JSON_NO_SIMD, like the lexer and the float
writer, and skips the CPU check of SSSE3 when the code is compiled for
SSSE3 anyway (__SSSE3__), like the lexer. JSON_VIEW_NO_SIMD and
JSON_VIEW_USE_SSSE3 and their ABI tags _vns and _vssse3 are removed; they
were never released. JSON_NO_SIMD keeps its tag _ns.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The image format is little-endian only. Instead of throwing
type_error.320 at runtime, save_image(), load_image() and the members
save() and load() do not exist on a big-endian target, so a call fails to
compile, like set() on a read-only document. type_error.320 remains for
saving a discarded document.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: view_mode enum instead of bool Editable, node_kind for node::kind
basic_json_document and basic_json_view take a view_mode (read_only, the
default, or editable) instead of a bool: basic_json_document<json, true>
did not say what true means. The shortcut aliases are unchanged.
The kind of a node is a node_kind (the value_t values plus link, 10 as
before), which removes the static_cast<std::uint8_t>(value_t::...) of
every comparison. The node and image layouts are unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: node_kind in the node writers, spell out read_only, comment fixes
The builder and the edit functions take and store a node_kind instead of
casting a value_t; the numbering assertions and the comments now speak of
node_kind, and the link kind notes that it follows value_t::discarded.
The read-only aliases name view_mode::read_only like the editable ones.
view_mode.md says the mode is a compile-time property of the type.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: views of read-only documents convert to editable views
A read-only document has no edits, so a view of it reads correctly as a
view of an editable document before its first edit. The implicit
conversion lets a function taking a json_editable_view accept the views
of both modes. There is no conversion the other way: a read-only view of
an editable document would read wrong values after the next edit.
The mode comparisons in the cross-mode operator==/!= use an
is_other_mode trait, which astyle keeps compact.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: fix BON8 documentation gaps found in the format review
Verified each claim with small C++ probes before writing it.
- index.md: BON8 serialization is also incomplete for invalid UTF-8.
- bon8.md: document duplicate keys (last wins), the error ids for
malformed input incl. 0xF5..0xF7 + 0x80..0xBF, exact string
termination wording, wide integers and discarded values in the
unsupported list, and title two untitled admonitions.
- exceptions.md: to_bon8 in type_error.316, BON8 in parse_error.112
(with example) and .113, wide integers in out_of_range.407.
- binary_values.md: bytes 0..39 take one byte in BON8.
- from_bon8.md: note on strict UTF-8 and duplicate keys.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review of the BON8 documentation
Restore the correct statement that marker bytes cannot begin a UTF-8 character (0x80..0xBF occur in valid UTF-8 as continuation bytes), and reword two sentences.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: drop the discarded-value remark from the BON8 page
Discarded values are not mentioned on the individual format pages.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: do not list invalid UTF-8 strings as a BON8 serialization gap
JSON text must be valid UTF-8, and to_bon8 rejects ill-formed strings by design (type_error.316), as documented on the BON8 page. Restore the original cell and realign the table.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read ND-arrays of half floats and enforce the char limit in ND-arrays
BJData N-dimensional arrays with element type 'h' (float16) failed to
read with a misleading parse_error.112, because the reader's dtype name
table lacked 'h'. It now maps to "half", the JData name. The writer still
does not pack "half"; such an object is written as a plain object and
reads back equal.
BJData limits 'C' (char) to 0..127 in Draft 2 and Draft 3, and the 1-D
reader enforces this, but ND-arrays with _ArrayType_ "char" wrote values
up to 255 as 'C' elements and read them back without a check. The writer
now only packs values in 0..127 (others fall back to a plain object that
round-trips), and the reader rejects 'C' elements above 127 with
parse_error.113 like the 1-D reader.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: correct and complete the BJData documentation
Verify the BJData chapter against the library and the Draft 2/3/4
specifications and fix what did not match:
- bjdata.md: correct the list of unused markers (N, not Z), the ND-array
example (3x2, not 2x3), the conditions under which an object is packed
as an ND-array (exactly three keys, row vectors, float-only elements for
single/double, char 0..127, byte only in Draft 3), the binary table row
and notes (which flag/draft combinations write [$B), the "$" type
restriction (Draft 3, Z is excluded too), the H marker in the
deserialization table, and the claim of a complete deserialization
mapping. Document the supported drafts, that Draft 4 (SoA containers,
extension type E) and column-major ND-arrays are rejected, ND-arrays of
half floats, duplicate object keys, and link the Draft 3 specifications
instead of master.
- Add an API page for basic_json::bjdata_version_t with an example, and
link it from the nav, the API index and to_bjdata.
- Update binary_formats/index.md and binary_values.md, which still claimed
that BJData has no binary type, and fix a wrong byte comment there.
- Document parse_error.115 and the out_of_range.408 cases for from_bjdata,
the precise other_error.502 condition for to_bjdata, and reword the note
on truncated unsigned integers in exceptions.md (only ND-array elements
were truncated before 3.13.0).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate single_include and apply astyle
Result of make amalgamate (including make pretty) for the preceding changes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review of the BJData documentation
The $ type restriction exists since Draft 2 (Draft 3 added B), $ requires a count, ND char above 127 is rejected on read, and the new API page is listed in alphabetical order in the nav.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: format tables, use the Oxford comma
Reformat the tables changed by this PR (BJData serialization and
deserialization mappings, completeness and binary-value tables) so all
rows are aligned, and add the Oxford comma to lists of three or more
items in the added text.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Write and read the inner length of BSON binary subtype 0x02
The BSON specification defines the payload of the "binary (old)" subtype
0x02 as an int32 followed by that many bytes. to_bson wrote the plain
payload instead, so decoders such as pymongo and libbson-based readers
rejected the bytes, and from_bson returned the inner length as part of the
payload (documented as lenient behavior in gh-5333).
The writer now emits the outer length (payload + 4), the subtype, the inner
int32 holding the payload size, and the payload. The reader requires an
outer length of at least 4 and an inner length equal to the outer length
minus 4 (the cases the bson-corpus lists as decode errors), reports parse
error 112 otherwise, and returns only the payload. Both directions change
together so that the round trip keeps working.
The single-reservation fast path for scalar entries declines subtype 0x02
because it has no room for the inner length.
This is a deliberate exception to the opt-in rule for behavior changes:
the previous output was invalid BSON.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Check the declared BSON document size before reading its content
The BSON reader compared a document's declared size with the number of bytes
read only when it reached the terminator. The declared size never bounded
the reads, so a SAX handler saw the keys and values of a document whose size
was too small before parse error 112 was reported.
The container frame now stores the position where the document ends
according to its declared size. open_bson_document rejects a size below 5
(size prefix plus terminator) and an embedded document that does not end
before the terminator of its enclosing document; the element loop rejects an
element whose type byte leaves no room for the name terminator and the
document terminator. The terminator check stays and still catches a declared
size that is too large. No input that was accepted before is rejected now;
documents with a wrong size are reported earlier, with fewer SAX events, a
different byte position and, in three cases, a different message. The error
id (parse_error.112) is unchanged and strict=false still does not skip the
check.
The tests that asserted the old messages and positions are updated, with the
outer sizes of two inputs corrected so that they still exercise the embedded
document. New tests cover the SAX events, a size below 5, and an embedded
document that fits exactly.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document BSON subtype 0x02, document size checks, and lenient reads
Describe the inner length of binary subtype 0x02 that to_bson now writes and
from_bson checks and removes. Add the document size checks (parse_error.112,
not affected by strict=false) to bson.md and from_bson.md, and extend the
list of lenient reads with the UUID and MD5 payload lengths (not checked),
duplicate keys (last value wins), embedded NUL bytes in strings, and the
ignored trailing bytes with strict=false. State that the members of an object
are written in iteration order.
Correct the statement that element names are never validated on read: they
are checked and sanitized like string values by the error handler. Add the
BSON message to the examples of out_of_range.415 and type_error.321, the
subtype limit of 255 to the binary values page, CBOR to the formats that use
byte_container_with_subtype, and fix a typo in the incomplete mapping box.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate single_include and apply astyle
Result of make amalgamate (including make pretty) for the preceding changes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review of the BSON documentation
Describe exactly when a too small document size is detected and note that subtype 2 data written by earlier versions is not readable.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: use the Oxford comma, drop discarded-value remarks
Add the serial comma in a list added by this PR and remove the BSON
example of type_error.321 (with the type_error.317 remark) so the docs do
not mention discarded values for BSON; restore the original singular
heading of that section. No tables were changed by this PR.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix -Weffc++ in the BSON recorder test and a clang-tidy use-auto hit
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy readability-redundant-member-init in BSON test
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Validate the optimized-container type marker for count 0 in UBJSON and BJData
The '$' type marker of an optimized container was only checked when an
element was read, so containers with count 0 accepted any marker:
`[$x#i0`, `{$x#i0` and `[$E#i0` parsed as empty containers although a
non-empty container with the same marker is rejected and the UBJSON
specification requires a legal marker after '$'.
get_ubjson_size_type() now checks the marker right after '$' against the
allowed set (UBJSON: Z N T F i U I l L d D H C S [ {; BJData: B U i u I m
l M L h d D C, with the existing exclusion of T F Z N S H [ {) and reports
parse_error.112 "marker ... is not a permitted optimized array type".
Input that was accepted and is now rejected: an optimized container with
count 0 and an invalid '$' type marker. Valid input is unaffected. Two
existing tests that used an invalid marker incidentally ("[$#" and a
negative-count case) were adapted.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Correct and complete the UBJSON documentation
Fix statements that disagree with the behavior (checked with probes):
- `Z` is null; the unused marker is `N`.
- `$` requires `#`; use_type with use_size=false throws other_error.502 only
for non-empty arrays, objects and binary values.
- NaN and infinity are written as IEEE 754 float64 and read back unchanged;
the UBJSON specification encodes infinity as null.
- `H` is documented (semantics, parse_error.115, out_of_range.406); the
"complete mapping" claim is qualified.
- use_type "same type" means same marker.
- Long Z/T/F arrays lose only the `$` annotation; the `#` count and the
element markers are kept.
- from_ubjson lists the 112, 113, 115, 406 and 408 causes that occur for
UBJSON, without the BJData-only ND-array wording.
- Duplicate object keys keep the last value; the `$` type marker is
validated even for count 0.
- Smaller fixes: float32 row, "map" -> "object", https link, exception 110
strict case, to_ubjson history of discarded values.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate single_include and apply astyle
Result of make amalgamate (including make pretty) for the preceding changes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review of the UBJSON documentation
Clarify that empty containers carry no $ annotation of their own and note that a counted N object is not supported.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: format tables, use the Oxford comma, drop discarded-value remarks
Reformat the two UBJSON mapping tables that this PR changed (uniform column
widths, padded cells), use the serial comma in lists of three or more in the
added text, and restore the original type_error.321 bullet in to_ubjson.md.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix narrowing of the optimized-container count in BJData and UBJSON tests
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document MessagePack ext types and fix related doc errors
Add an "Extension types (ext, fixext)" section to the MessagePack page:
ext/fixext are read as binary, the signed ext type is exposed as an
unsigned subtype 0..255 (-1 reads as 255), Timestamp is not interpreted,
subtypes are always written as ext/fixext, and subtypes above 255 throw
out_of_range.415. Add the 0xC1 row to the deserialization table, note
that duplicate keys keep the last value, and correct the NaN/Infinity
explanation, the "31 bytes / 15 elements" wording, the ill-formed UTF-8
sentence, and the generic 0xC1-key error message.
Add fixext16 and the unsigned subtype mapping to binary_t.md and
binary_values.md, reword "maximum of the MessagePack ext type" in
to_msgpack.md and exceptions.md, and mark MessagePack deserialization
as incomplete (non-string keys) in the binary formats overview.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review of the MessagePack documentation
Name parse_error.113 for a 0xC1 map key, move the duplicate-key sentence out of the ext section, and keep the 0xC1 row in marker order.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Skip CBOR tags in front of map keys and validate indefinite text chunks
A tag in front of a text string map key was rejected with parse_error.113
even with cbor_tag_handler_t::ignore or store, although tags in front of
values are handled. The tags are now skipped iteratively before the key is
read; the handler error still rejects them.
The chunks of an indefinite-length text string were concatenated before
the single UTF-8 check, so a code point split over two chunks, or an
invalid chunk, was accepted. RFC 8949, Section 3.2.3 requires every chunk
to be valid on its own. With error_handler_t::strict, replace and ignore
each chunk is now checked; keep is unchanged. The chunks are appended with
append(ptr, len), so StringType needs nothing new.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: correct and extend the CBOR documentation
Fix the negative-integer rows (they used MessagePack boundaries), the
number_integer range of the 0x1B row, and the half-precision entry in the
list of unused types; correct the NaN history (0xF9 since 3.8.0) and the
tagged-item wording (only the tag directly in front of a byte string is
stored, and only in the 0xD8..0xDB form). Document that to_cbor is not
RFC 8949 deterministic, that duplicate keys are accepted (last wins),
tags in front of keys, per-chunk UTF-8 validation, the exceptions
parse_error.113, out_of_range.406/408 of from_cbor, the pointer and
C array input traps, the partially written output on exceptions, and the
store/ignore results in the binary values example. Drop the claim that
the CBOR reader needs StringType::append(const StringType&) and
BinaryType::insert(pos, first, last).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate single_include and apply astyle
Result of make amalgamate (including make pretty) for the preceding changes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review of the CBOR changes
Cite RFC 8949 Section 5.3.1 for duplicate keys, describe the out_of_range.408 condition precisely, rewrap a long line, and update the stale get_string comment (per-chunk UTF-8 check). single_include regenerated.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: drop discarded-value remarks
Remove the mention that a discarded value cannot be serialized to CBOR
(type_error.321) from the exception-safety and mapping sections. The
integer table and all added prose already follow the table-format and
Oxford-comma rules.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* create objects for tokens that cannot be array indices (#5357)
When operator[](json_pointer) traverses a level that does not exist yet,
the null value is turned into an array or an object depending on the
reference token. The check only tested whether all characters are digits,
so tokens that can never be a valid array index selected an array and
then failed:
- "01" (and any other token with a leading '0') threw parse_error.106
- the empty token threw out_of_range.404
Both tokens are valid object keys, and both work when the level already
exists as an object, so creating the level changed the outcome. Test the
token against the RFC 6901, Sect. 4 grammar for array indices instead, so
that such tokens create an object. This only affects pointers that threw
before.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Clarify that the token 0 is a valid array index in operator[] docs
The note described valid array indices as "a nonempty sequence of digits
without a leading 0", which read literally excludes the token "0" itself,
contradicting both the code and the /foo/0/0/0 example in the next
sentence. Spell out the RFC 6901 grammar instead.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Throw out_of_range.411 for JSON Patch add into a null parent (#5917)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MKJf48vnGtTE25KscmjfaW
Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com>
* docs: make the diff() and patch() documentation accurate; add tests
Document the exceptions to the diff round-trip invariant (NaN, discarded
values, number types), the path parameter, non-minimal patches, array and
object operation order, the ordered_json fallback, binary values in
serialized patches, the real complexity, duplicate member names in
operations, test on missing paths, and the exact out_of_range.403 message.
Add tests for these behaviors.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Fix inaccurate or missing statements about unflatten (third round-trip
failure, empty token, typo), value (type_error.306 before pointer lookup),
contains (pointer overload), parent_pointer and operator<< (bad_alloc),
out_of_range.410 example, template parameter requirements (resize(n)), and
document the URI fragment form and non-unique member names of RFC 6901.
Add tests for the unflatten and value() cases.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: byte layout sections, glossary, and QA page updates
- Move "Byte layout" of each binary format to its own section before
"Serialization" and describe the idea of the format.
- Glossary: add LLDB and other terms used throughout the documentation.
- quality_assurance: correct Clang Static Analyzer and CMake entries; add
CodeQL, Semgrep, Flawfinder, CIFuzz, OpenSSF Scorecard, REUSE, Meson/SwiftPM,
and reproducibility checks.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: drop unused length colour from CBOR and MessagePack byte layout legends
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document clang-tidy exclusions and drop unneeded ones
Re-enable every excluded check in a run of clang-tidy 22 over all tests
and remove the seven that never fire (cert-err58-cpp,
clang-analyzer-webkit.NoUncountedMemberChecker,
cppcoreguidelines-virtual-class-destructor, google-readability-function-size,
hicpp-function-size, hicpp-no-assembler, readability-function-size).
Document the reason for every remaining exclusion at the top of the file.
Exclude in advance the checks that only exist in clang-tidy 23 and are
noise here (bugprone-signed-bitwise, *-explicit-constructor,
readability-trailing-comma, modernize-use-string-view), plus
modernize-use-integer-sign-comparison, which needs C++20 std::cmp_less
and flags deliberate casts.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy findings in documentation examples and replace redundant aliases
Apply clang-tidy fixes to the documentation examples:
- performance-avoid-endl: std::endl -> '\n'
- readability-qualified-auto: auto -> auto* / const auto*
- performance-for-range-copy: copy -> const reference
- google-readability-casting: C-style cast -> static_cast
Replace redundant aliases (readability-redundant-qualified-alias) such as
"using json = nlohmann::json;" with "using nlohmann::json;" in the
documentation, README, examples, tests, and benchmarks. Aliases at class
scope and aliases of other types (with_base_class_t etc.) are kept.
No changes to the library or its public API. All changed examples were
compiled and run; their output matches the stored .output files (except
max_size and meta, which depend on the platform and compiler).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Format doc examples with the pinned astyle
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Docs: fill documentation gaps found in user questions
Add pages on class-hierarchy/smart-pointer/variant/chrono recipes, working
without exceptions, and large, streamed and partial input. Extend the FAQ
(23 entries, Known bugs first), parse error documentation (empty-input
checklist, line/column, diagnostic positions), element access, conversions,
creating values, ADL troubleshooting, memory, and build/distribution notes.
Add 14 examples with generated output. Correct the statement that parse errors
cannot be recovered from with -fno-exceptions.
The topics come from recurring questions on Stack Overflow, Reddit, Hacker
News, YouTube, and the project's GitHub discussions and issues.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: declare range-for lists in variables to keep astyle quiet
astyle reformats a braced init list placed directly in a range-for
header; declare the list first so check_amalgamation passes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: cross-reference json_view and json_document for 3.13.0
Document the new zero-copy document/view API across the docs: add a
"Documents and views" section to the basic_json API hub linking to all
specializations, point parsing/conversion/modifying pages to the view-based
alternatives, add document/view sections to feature pages, update the
landing page to mention json_view.hpp/json_literals.hpp, and cross-link
debugging/exceptions.
Also document wide-string inputs (and ill-formed UTF-16/32 rejection),
add parse/accept examples for wide strings, fix related doc gaps, and
align features/parsing and FAQ with the added support.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: fix article before std::istream in FAQ
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: clarify document inputs, decoding, get<T>() and editing
Address review comments: say which inputs are borrowed and that a wide
string pointer is an extra input, which values are decoded during parsing,
that get<T>() reads only built-in types directly, all forms of set() and
that the document's insert() is array-only, and how to store wide strings.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: integers are converted while parsing a document
Also say "They accept every input" in basic_json_document::parse.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- Make the iterator range illustrations legible in dark mode (prefers-color-scheme
rules inside the SVGs).
- Add a "Byte layout" section with a color-coded byte table to each binary format
page (BJData, BON8, BSON, CBOR, MessagePack, UBJSON), and highlight the smallest
size per file in the comparison table.
- Add "See also" sections to the overview pages that lacked one.
- Split the release notes into one page per release, grouped by minor version on
the overview page, with an "API changes" section per release generated from the
API pages' version histories (hooks/api_changes.py).
- Fact-check the release notes and version histories against the release tags and
correct wrong or misleading entries (e.g., json_serializer 2.1.0, merge_patch
3.1.0, items, count, operator[], to_bjdata, std::string_view key support).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read C streams into a json_document in blocks
json_document::parse(FILE*) (and read(), accept(), parse_copy()) went through the
library's file input adapter and appended the text character by character
with fgetc. Read it with fread in blocks instead, like the std::istream
overload already does.
Parsing citm_catalog.json (1.7 MB) from a FILE*:
Apple M1: 80.6 M -> 8.4 M instructions, 4.8 ms -> 0.55 ms (parse of the same text from a std::string: 8.7 M)
Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 47.0 M -> 13.7 M / 45.8 M -> 14.2 M
The parsed documents and the error cases are identical (checked over the test corpus
and 30 mutants of each file). A null FILE* is handled as before (by the library's adapter).
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Index arrays without subtrees directly
operator[](idx), at(idx) and back() skipped the elements of an array one
after the other, whatever they were. In an array none of whose elements is
an array or object (numbers, strings, booleans, null) the elements are
adjacent nodes, which the array node's own sizes tell: its subtree has
exactly one node more than it has elements. Such an element is now the
node idx after the array node; any other array is walked as before.
Reading r[i] for every i of an array of 3000 integers:
Apple M1: 28.8 M instructions, 3.1 ms -> 0.1 M instructions, 4 us
Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 84.1 M -> 3.1 M / 75.0 M -> 3.0 M
Arrays that contain arrays are unchanged.
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Skip the bounds checks of number and string tokens in NUL-terminated input
For a NUL-terminated input (std::string, C string, string literal), the
parser reads the terminator instead of comparing p with the end, via
cursor::cur(). Number and string scanning still tested `p != e` before each
look at the next byte (after the integer digits for '.', for 'e', after
the exponent sign and digits, and after a string for its closing quote),
although the terminator can never be one of those characters. They now use
cur(), which is the plain load for NUL-terminated input and the bounds check
for ptr+len input, as before.
Parsing, Apple M1 instructions per parse: twitter 3.126 M -> 3.089 M (-1.2 %),
citm_catalog 8.70 M -> 8.58 M (-1.3 %), jeopardy 306.1 M -> 302.5 M (-1.2 %), signed
integers 281 M -> 266 M (-5.4 %), canada and floats -0.1 % / -0.9 %.
Linux x86-64 Ir, GCC 13 / Clang 18: twitter -0.5 % / -0.0 %, signed integers +0.2 % / -0.7 %.
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the index text of items() on arrays lazy
basic_json_view::items() on an array formatted every element's index with
std::to_string when the item was made, whether or not the loop asks for the
key. The item now keeps the position and makes the text on the first key()
call. Objects are not affected.
Iterating over the items of an array of 3000 numbers and reading only the values:
Apple M1: 25.5 us -> 4.6 us (0.45 M -> ~0 net instructions)
Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 14.9 M -> 4.1 M / 15.3 M -> 4.6 M
Reading key() costs what it did (24.5 us).
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the default of value() with a C string only if the member is missing
basic_json_view::value(key, "text") and value(pointer, "text") built a
std::string from the default first and passed it on, so a call allocated
for the default even when the member exists, and copied it again when it
does not. They now look the member up and make the result directly: from the
member if there is one, else from the C string. Behavior, including the
exceptions, is the same.
Two calls (member found, member missing) with a 40 character default, Apple M1:
1.6 K -> 0.7 K instructions
Linux x86-64 Ir (500 rounds), GCC 13 / Clang 18: 2.46 M -> 2.22 M / 2.40 M -> 2.15 M
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Materialize scalars directly and keep the stack of small subtrees local
basic_json_view::materialize() (and get<BasicJsonType>()) always set up the
SAX handler that parse() uses and a std::vector for the open containers,
sized from the document's deepest nesting. For a scalar, which is most of what
is materialized one value at a time, neither is needed: the value is made
directly (the same constructors the handler ends up calling). For arrays and
objects the stack of open containers is now a local array for the first 16
levels, and the vector is only used for deeper documents.
Materializing five scalars of a small document, Apple M1: 3.6 K -> 0.9 K instructions
Linux x86-64 Ir (500 rounds), GCC 13 / Clang 18: 2.96 M -> 2.37 M / 2.90 M -> 2.29 M
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Copy the text between escapes while scanning it
A string that contains an escape is decoded by builder::slow_string(). After
each escape it scanned the next run of plain bytes (8 or 16 byte-by-byte
steps, then a vector loop), and then copied that run to the decode arena in a
second pass. The run is now copied 16 bytes at a time while it is scanned
(one vector load, a store of the same 16 bytes, and the mask of the bytes
that end the run), and the write position lives in a local variable, not in
the builder, since a store of a character may alias the builder's members and
made the compiler reload it after every escape. A run that ends at a quote
needs no scan, and non-ASCII text goes through the UTF-8 check as before;
the portable code is unchanged.
Parsing, Apple M1 instructions: jeopardy.json 304.0 M -> 293.8 M (-3.4 %), 3000 strings with
escapes 1.72 M -> 1.54 M (-10 %), twitter -0.6 %.
Linux x86-64 Ir, GCC 13 / Clang 18: escaped strings -7.6 % / -11.4 %, twitter 0.0 % / 0.0 %.
Documents and error messages are identical on 900 k random strings of escapes, UTF-8 and control
characters (also under ASan), and on mutants of the test corpus.
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Hash the last bytes of a key of a large object without a loop
key_hash() mixed a key eight bytes at a time and then packed the remaining
0-7 bytes one by one, in a loop. The last word of a key of eight or more
bytes is now read as the last eight bytes of the key (overlapping the word
before), and a key of 2-7 bytes as two overlapping words of four or two
bytes. A product only moves bits upwards, and the slot is taken from the low
bits: the high bits of the last word are folded down before the
multiplication, since keys that differ only in their last characters ("user_17",
"user_18") would otherwise share slots. The hash is not stored, so its values
may change.
Looking up every key of an object of 1000 random keys, Apple M1 instructions per 1000 lookups:
keys of 5 bytes 175 K -> 93 K, hex strings of 8 bytes 177 K -> 168 K, keys of 12 bytes 196 K -> 186 K
Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 5-byte keys -4.7 % / -8.8 %, 12-byte keys -3.3 % / -7.4 %.
Building the tables while parsing such objects is cheaper as well (5-byte keys: 13.3 us -> 10.1 us).
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Compare short keys of a large object without a call of memcmp
find_indexed() compared the key of a member the table points to with
memcmp, a library call for any length. Up to 16 bytes, the bytes are now
compared as two overlapping words (the first and the last 8, 4, or 2 bytes),
as find_member() already does for the objects without a table; longer keys
still use memcmp.
Looking up every key of an object of 1000 random keys, Apple M1 instructions per 1000 lookups
(with the previous commit): keys of 5 bytes 93 K -> 72 K, hex strings of 8 bytes 168 K -> 148 K,
keys of 12 bytes 186 K -> 170 K, keys of 24 bytes 198 K -> 193 K (unchanged, memcmp).
Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 5-byte keys -7.2 % / -7.3 %, 12-byte keys -5.9 % / -6.6 %.
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read an unedited editable document like a read-only one
Views of a json_editable_document followed edits at every step: the
lookup, the element access, back(), dump() and materialize() were built
for nodes that can be moved or linked, though no edit changes the index until
the first one (every edit creates the edit state before it touches a node).
Until then these now take the read-only code: one check of the document's
edit state per call, not per member or per node. After the first edit
nothing changes. Views that were made earlier keep working, as the check is
made when they are used.
Unedited editable document, Apple M1: dump() of twitter.json 1.40 M -> 1.22 M instructions (-12.6 %);
lookups unchanged. Linux x86-64 Ir, GCC 13 / Clang 18: dump -1.7 % / -1.0 %, lookups -0.1 % / 0.0 %.
Walking with iterators is unchanged, and so is dump() after an edit.
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Format the numbered test keys of json_view's index with a stream
The keys were made with snprintf() and a format from a list, which
Flawfinder reports as an error (CWE-134, a format string that is not a
literal). A stream now formats them; the 30000 keys of the test are the
same as before.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Put the key number of the json_view index test in parentheses
astyle formatted i * step after << as a pointer declaration (i* step),
which failed the amalgamation check.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the compiler and lint findings of json_view's read side and tests
- find() and contains() dispatch on Editable for the member lookup; for
documents that cannot be edited both branches were the same call
(GCC -Wduplicated-branches)
- json_view.hpp includes <cstdio> for the FILE overload (cpplint)
- the arena check macro of the builder is on one line (cpplint read it
as an if body without braces)
- tests: no nested conditional (also MSVC C4018), no useless cast,
parentheses around mixed operators, raw string literals, += instead
of temporary concatenations, and NOLINT for the fixed seeds and a
find() != npos, as elsewhere in the tests
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Grow the decode arena through a function and store its position back on every return
slow_string() kept its write position in locals and grew the arena through
the macro NLOHMANN_VIEW_ENSURE. The macro is now the always-inline member
arena_ensure() (a lambda is not inlined at every call site by clang, which
then keeps the position in memory: +11 % instructions on escape-heavy
strings), and a guard stores the position back into m_arena_write on every
return instead of by hand on the successful one. Instruction counts are
unchanged (clang) or 1 % lower (GCC).
Also word the complexity of back() as that of at().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Since #5877, the checked-in nlohmann_json.natvis only covers namespaces
with up to two ABI tags. The generator gets options to produce a file
for other configurations:
- --abi-tags TAGS adds the namespaces of one configuration (repeatable)
- --all generates every combination of ABI tags
- --list-tags prints the tags with the macros that enable them
- --output FILE writes somewhere else than the repository root
`make natvis NATVIS_ARGS="..."` passes the options through. The default
output is unchanged. The options are documented in the generator's
README, the debugging page, and FILES.md, and checked in CI.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Scan strings with SSE2, SSSE3, or NEON
The string fast path found the end of a run of plain characters 8 bytes at
a time (SWAR) and validated non-ASCII text one sequence at a time, calling
validate_one_utf8() out of line for each.
The vector code of json_view's scanner, in the new shared header
detail/input/simd_scan.hpp, now does both 16 bytes at a time: SSE2 finds
the end of a run on x86-64, and the UTF-8 check of Keiser and Lemire runs
with NEON on AArch64 and with SSSE3 on x86-64. SSSE3 is not part of x86-64,
so the check is compiled for it with a function attribute and used when the
CPU has it (cached in a constant-initialized atomic), else the scalar check;
code compiled with -mssse3 skips the CPU check. The first word of a run is
still checked with SWAR (the first four with NEON, where a vector mask
costs more), as most keys and short values end there, and the scalar
helpers are always inlined. On anything else, the scalar code is unchanged.
The new macro JSON_NO_SIMD selects the portable code everywhere, including
the SSE2/NEON digit formatting of the floating-point writer. The lexer tests
now also run with JSON_NO_SIMD and, on x86-64, with -mssse3, and a new test
places every byte that ends a run at each offset across several 16-byte
blocks, after ASCII and after CJK text.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep the string scan from slowing down short strings and dump() on Arm
Three benchmarks were 7-9% slower than develop on Apple M1 (Apple Clang,
-O3 -flto): Accept/jeopardy, Dump/citm_catalog, and Dump/jeopardy.
- dump(): the serializer passes a string without its closing quote, so
plain_run_end() ran out of input and called the out-of-line NEON loop
for every string, with fewer than 16 bytes left. The loop is now only
called for 16 bytes or more.
- accept(): no string of jeopardy.json reaches the vector UTF-8 check,
yet compiling it into the lexer's scan function made it 5% slower: the
function runs once per token and now saved more registers and kept a
stack slot. lexer_string_bulk_run() checks the first four words inline
and continues out of line, and the serializer keeps string_bulk_run()
inline, which is faster there.
- The byte that ends a run is read from the scanned buffer instead of
through get(), which Apple Clang merged with the other get() calls that
load the read position again right after it was stored (3%).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Define string_bulk_run_from() only where lexer_string_bulk_run() calls it
With NEON and simdutf in C++17, lexer_string_bulk_run() takes the
string_bulk_run() path, so the out-of-line helper was unused.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not dispatch the SSSE3 string scan with GCC's C++20 modules
GCC does not keep the target("ssse3") attribute on functions it reads
back from a module interface, so the module build (ci_module_cpp20,
gcc:latest) rejected the intrinsics in vector_utf8_run() with "needs isa
option -mssse3". With __cpp_modules, GCC now uses the portable UTF-8 check
like other compilers without the target attribute; the SSE2 scan stays,
and -mssse3 or more still compiles the SSSE3 check directly. Clang is not
affected.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Include <cpuid.h> only once when json.hpp and json_view.hpp are both used
The SSSE3 dispatch of the string scan and the one of json_view each
include <cpuid.h>. Before GCC 11 the header has no include guard, so a
translation unit that includes both headers fails with GCC 10:
"redefinition of 'unsigned int __get_cpuid_max(unsigned int, unsigned int*)'".
Guard both includes with NLOHMANN_JSON_CPUID_INCLUDED.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add an ABI tag for JSON_NO_SIMD
JSON_NO_SIMD selects between two definitions of the internal scanning and
formatting functions. Like JSON_VIEW_NO_SIMD (_vns), it now appends _ns to
the inline namespace, so translation units built with and without it can
be linked into one program without ODR violations.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The generator wrote a namespace entry for every combination of ABI tags,
so nlohmann_json.natvis doubled with each new tag. With the 12 tags on
develop it had reached 57.9 MB, and one more tag would push it past
GitHub's 100 MB file limit. It now covers the default namespace, every
single tag, and every pair of tags (versioned and unversioned), which is
158 entries / 1 MB with 12 tags.
Taken over from #5868. Refs #5863.
Signed-off-by: Kartikey <kg704925@gmail.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Kartikey <kg704925@gmail.com>
Niels Lohmann