mirror of
https://github.com/nlohmann/json.git
synced 2026-11-15 19:13:35 +00:00
253e4c392654d15aa45fb0ccfc6355301fcba93d
5532
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Niels Lohmann
|
253e4c3926 |
Bound what the DOM parsers reserve for arrays that announce a size (#6082)
* Bound the total number of valueless UBJSON elements per input An optimized UBJSON array of type Z, T or F has no payload, so its declared count is not bounded by the input. The limit of 1048576 applied to each array on its own, so 9 bytes (`[$Z#l` and a count) produced 1048576 values and every further 9 bytes another 1048576: 362 bytes allocated 661 MB in a DOM, and a SAX handler received 1M events per 9 bytes. The reader now adds the counts of all such arrays of one input and rejects the input with out_of_range.408 when the sum exceeds 1048576 plus 4 for every byte read so far. The base is the per-array limit, so an array that was accepted on its own before is still accepted at the start of an input. These arrays add about 80 bytes of memory per input byte (measured 77; one `Z` per byte needs 32), plus the base; the memory of a document as a whole is not bounded by this. The check runs once per valueless array, before its start event, with plain size_t arithmetic that cannot overflow for a 32-bit size_t. Containers with a payload type and objects (every member needs a key) are bounded by the input already and are unchanged. BJData rejects these types. to_ubjson counts the elements of the typed arrays it writes in one call and writes an array in the unoptimized form (`[#n Z Z ...`) once it would bring the total above 1048576, which a reader accepts at any input size. The check is made only for a typed array of a valueless type, so BJData and other arrays do not pay for it. Everything to_ubjson writes in version 3.13.0 can therefore be read back. Two consequences: - to_ubjson output changes for values with more than 1048576 elements in arrays of type Z, T or F: such an array is written at one byte per element, e.g. 270000 rows of 100 nulls take 26.8 MB instead of 1.35 MB. - Output of version 3.12.0 and earlier (typed arrays without such a total) is rejected by from_ubjson when the arrays hold more than 1048576 elements plus 4 per byte of input. 3.12.0 writes a row of up to 255 nulls in 5 bytes (6 above 255), so a table is rejected from 1075 rows of 1000 nulls or 209723 rows of 25 (accepted: 1000 rows of 1000, 209000 rows of 25); rows of 20 or fewer are never rejected. Measured on the test data: no file of json_test_data (converted with to_ubjson in all three modes, 846 round trips) holds an array of this kind with more than 0.17 values per byte. All of them are read back. Reproducer (K arrays of 1048576 nulls), Apple clang -O2: K=40 (362 bytes): before 41943040 values, 664 MB; after rejected, 33 MB K=2 (20 bytes): before 2097152 values, 53 MB; after rejected, 33 MB The new tests count events with a SAX handler where they can: building and comparing million-element values made the first version take 176 s under Valgrind, the current one about 22 s. Behaviour change: an input that holds more than 1048576 elements in arrays of type Z, T or F at more than 4 per byte is now rejected. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Report an empty array for UBJSON `[$N#<n>` instead of n elements A no-op (N) is not a value, so an optimized array of type N holds no element whatever count it declares. The reader skipped the elements but still passed the declared count to the SAX parser: a handler was told start_array(n) and no element followed, and the DOM parser reserved min(n, 16384) values (256 KB) for an array that stays empty. Nine bytes (`[$N#l 0x7FFFFFFF`) therefore committed 256 KB: 2000 of them in one input, 18 KB, committed 512 MB of heap (capacity 163,840,000 values for the 10000 arrays of the original probe: 2.6 GB for 70 KB of input). Untouched pages keep the resident size small, but the commit charge and a 32-bit address space are consumed. The reader now announces 0 elements for an optimized array of type N. The loop that skips the elements is unchanged (it never ran). A count that does not fit into std::size_t is still rejected by the count reader; a count that is too large for the array type of the DOM (before: out_of_range.408 "excessive array size" from the DOM handler) is accepted now, because the array is empty. Behaviour change: sax_parse reports start_array(0) for `[$N#n` (before: n); from_ubjson no longer throws out_of_range.408 for an N-typed array whose count exceeds the maximum size of the array type. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Bound what the DOM parsers reserve for arrays that announce a size Cause: #5476 made the DOM parsers reserve storage (up to 16384 elements) for a counted array of CBOR, MessagePack, UBJSON and BJData when it opens, before any input backs the count. The arrays that are open at the same time keep their storage until the parse ends, so N nested headers that announce 16384 elements and are never completed hold N * 16384 * sizeof(json): 100,000 nested CBOR headers `99 40 00` (300,000 bytes) peaked at 1.65 GiB, 3.12.0 at 12 MiB; the same for MessagePack `DC 40 00`, UBJSON `[#I 16384` and BJData. Fix: json_sax_dom_error_state, the base of the three DOM parsers, keeps a credit of elements that may still be reserved: 65,536 at the start plus 2 for every byte the reader has read. An array that announces no more than the credit left is reserved as before (one comparison and one subtraction per counted array); a larger one is reserved what is left and grows as its elements arrive. binary_reader tells a DOM parser (a handler whose pointer converts to json_sax_dom_error_state*; a handler that derives from one privately or protectedly, or any other SAX handler, is not told anything and reserves as in the base of this change) where to find its count of characters read for the duration of sax_parse(). An RAII guard puts back the previous binding on every exit, also by exception, so the parser never keeps a pointer to a reader that is gone (unbound, only the per-array cap of 16384 applies); the public SAX interface is unchanged, and start_array() still reports the announced number. A bound per array by the remaining input would not help, since the same bytes would back every open level; the bound has to be over all arrays. The elements of an array take at least one byte of input each, so the credit is more than honest input needs: none of the 19 inputs measured (canada, citm_catalog, twitter, mesh, update-center, random, numbers, animation, ik_rig, instruments, geo_response, jeopardy and 7 synthetic ones, each as CBOR, MessagePack, UBJSON and BJData, with and without size/type optimization) was given less than it announced. A value that nests five or more arrays of 16384 elements, each with an array as its first element, is the exception: the later ones grow as before. Measurements (clang -O2, macOS arm64, peak memory footprint of `/usr/bin/time -l`, reproducer: n nested headers): n = 20,000 n = 100,000 develop fix 3.12.0 develop fix 3.12.0 CBOR 338.7 5.9 8.2 1687.7 23.3 11.8 MiB MessagePack 338.9 6.0 7.3 1687.9 23.8 11.9 MiB UBJSON 339.0 7.6 8.0 1687.9 30.1 11.6 MiB BJData 339.0 7.3 7.9 1688.0 30.8 11.6 MiB That is at most (65,536 + 2 per input byte) values: 1 MiB plus 32 bytes per input byte with 16-byte values (a value is 24 bytes with JSON_DIAGNOSTICS and up to 40 with JSON_DIAGNOSTIC_POSITIONS), as documented in untrusted_input.md. Lower fixed caps do not get there: a cap of 64 elements still peaks at 113 MiB (n = 100,000) and costs 4.7 % on arrays of 300 elements (mat300.cbor) and 5.0 % on arrays of 10,000 (arrays_of_big.cbor). Reader instructions retired (min of 7, 108 rows: 18 inputs in the 6 forms CBOR, MessagePack, UBJSON and BJData plain and optimized; jeopardy, 46 MB, is within 0.1 %), compared with the base of this change: -0.19 % to +0.37 % in the formats that announce sizes (canada.cbor 81.9 M -> 82.1 M, many_small.cbor 333.5 M -> 334.2 M); the benefit of #5476 is kept (without any reserve canada.cbor takes 116.8 M, +42.6 %, many_small.cbor +71.5 %, mat300.cbor +81.6 %). Rows of plain UBJSON and BJData, whose arrays announce no size, move by -2.7 % to +2.6 % without any code of theirs changing: the base alternates between two instruction counts per process (452 M and 463 M for many_pairs.ubjson, equal in cycles), and a struct with three unused members added to the same class reproduces the 474 M mode. Tests (CBOR, MessagePack, UBJSON, BJData): 200 nested headers that announce 16384 elements peak at 52,433,600 bytes of DOM allocations on the base and below 128 Ki values with the fix (scaled by sizeof(value), so that it holds with the diagnostic macros) (counting allocator, no process memory measured); an array that completes is still reserved in one allocation; CBOR also checks that a DOM parser is not bound to a finished parse (ASan: stack-use-after-scope without the guard), that wrappers derived publicly (also with a parse inside the parse), privately and protectedly compile and are bound or not as described, that an array that got less than it announced decodes and that a SAX handler is told the announced size. The new tests take 4.6 s under Valgrind at -O0 (gcc:12, all four files). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Clarify the reservation note for announced array counts Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Report an empty array for UBJSON [$N#<n>] instead of n elements (#6079) * Report an empty array for UBJSON `[$N#<n>` instead of n elements A no-op (N) is not a value, so an optimized array of type N holds no element whatever count it declares. The reader skipped the elements but still passed the declared count to the SAX parser: a handler was told start_array(n) and no element followed, and the DOM parser reserved min(n, 16384) values (256 KB) for an array that stays empty. Nine bytes (`[$N#l 0x7FFFFFFF`) therefore committed 256 KB: 2000 of them in one input, 18 KB, committed 512 MB of heap (capacity 163,840,000 values for the 10000 arrays of the original probe: 2.6 GB for 70 KB of input). Untouched pages keep the resident size small, but the commit charge and a 32-bit address space are consumed. The reader now announces 0 elements for an optimized array of type N. The loop that skips the elements is unchanged (it never ran). A count that does not fit into std::size_t is still rejected by the count reader; a count that is too large for the array type of the DOM (before: out_of_range.408 "excessive array size" from the DOM handler) is accepted now, because the array is empty. Behaviour change: sax_parse reports start_array(0) for `[$N#n` (before: n); from_ubjson no longer throws out_of_range.408 for an N-typed array whose count exceeds the maximum size of the array type. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Clarify the reservation note for announced array counts Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget (#6083) * Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget The whole-input budget for the elements of optimized UBJSON arrays of type Z, T and F (1048576 plus 4 per byte read) rejects files written by 3.12.0 that have long arrays of these types, such as 1100 rows of 1000 nulls with a `$Z` annotation, and there was no way to read them. The per-byte rate is now the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE. Semantics: - default 4, so the default build is unchanged (the generated code of from_ubjson is identical at -O2 up to source line numbers); - the value must be a positive integer: 0, negative, non-integral, bool, character and wider-than-uintmax_t values fail to compile with a static_assert that names the macro; - a value that does not fit into std::size_t is the largest size_t, so one spelling works for 32- and 64-bit targets; - the limit base + rate * bytes_read saturates at the largest size_t instead of wrapping around, for any rate and input size; a rate of 131072 (2^20 / 8) or more cannot be exceeded by any input (the shortest header of an array of more than 32767 elements is 8 bytes), which switches the whole-input budget off; - the limit of one array (1048576) and the base (1048576) stay fixed constants, so the writer fallback still follows the base only and a reader with any rate reads everything the writer produces; - the macro is not part of the ABI tag and must have the same value in every translation unit; it is documented that way. The memory the arrays can take in a DOM is about 20 bytes per rate and byte of input (measured, macOS, containers of 9 * rate elements: 25 bytes per input byte at rate 1, 81 at 4, 312 at 16, 1226 at 64, 5493 at 256) plus about 33 MB for the base. A 3.12.0 row of N values has a header of b bytes (5 up to 255, 6 up to 32767, 8 above), so a table is read completely if N <= rate * b, and otherwise up to (1048576 + h * rate, h = outer header: 6, 7 or 9 bytes) / (N - rate * b) rows; 1100 rows of 1000 nulls need a rate of 8. The new page of the macro documents the formulas, the measurements, and two examples that are run by the documentation checks. The tests of the budget move to tests/src/unit-ubjson_valueless.cpp, which derives every boundary from the macro and is built with the rates 1, 1000000 and 0xFFFFFFFFFFFFFFFF in addition to the default (the last two check that nothing is refused and that the limit does not wrap around); they fail with the unchanged header for rate 1, 1000000 and the maximum. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Refuse char8_t for JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE and clarify its documentation Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use fseek instead of rewind in the UBJSON FILE* test clang-tidy (bugprone-unsafe-functions) rejects rewind() because it has no error detection. Check the result of fseek() instead. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Charge an announced array only what is reserved for it An array that announced more than reserve_cap() elements but no more than the credit left was charged the announced count, while reserve_array() reserved at most reserve_cap() of them. An outer array of 20000 elements therefore left a third nested array of 16384 with 12796 instead of the full reservation. reserve_count() now caps the count first and charges that, and the DOM parsers reserve it through reserve_array_capped(), which does not cap again, so the fast path costs the same as before (instructions retired within +-0.06 % on canada, citm_catalog and twitter in CBOR, MessagePack and UBJSON). Also re-wrap the comment of valueless_elements_base. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
73ec296320 |
Bound the total number of valueless UBJSON elements per input (#6078)
* Bound the total number of valueless UBJSON elements per input An optimized UBJSON array of type Z, T or F has no payload, so its declared count is not bounded by the input. The limit of 1048576 applied to each array on its own, so 9 bytes (`[$Z#l` and a count) produced 1048576 values and every further 9 bytes another 1048576: 362 bytes allocated 661 MB in a DOM, and a SAX handler received 1M events per 9 bytes. The reader now adds the counts of all such arrays of one input and rejects the input with out_of_range.408 when the sum exceeds 1048576 plus 4 for every byte read so far. The base is the per-array limit, so an array that was accepted on its own before is still accepted at the start of an input. These arrays add about 80 bytes of memory per input byte (measured 77; one `Z` per byte needs 32), plus the base; the memory of a document as a whole is not bounded by this. The check runs once per valueless array, before its start event, with plain size_t arithmetic that cannot overflow for a 32-bit size_t. Containers with a payload type and objects (every member needs a key) are bounded by the input already and are unchanged. BJData rejects these types. to_ubjson counts the elements of the typed arrays it writes in one call and writes an array in the unoptimized form (`[#n Z Z ...`) once it would bring the total above 1048576, which a reader accepts at any input size. The check is made only for a typed array of a valueless type, so BJData and other arrays do not pay for it. Everything to_ubjson writes in version 3.13.0 can therefore be read back. Two consequences: - to_ubjson output changes for values with more than 1048576 elements in arrays of type Z, T or F: such an array is written at one byte per element, e.g. 270000 rows of 100 nulls take 26.8 MB instead of 1.35 MB. - Output of version 3.12.0 and earlier (typed arrays without such a total) is rejected by from_ubjson when the arrays hold more than 1048576 elements plus 4 per byte of input. 3.12.0 writes a row of up to 255 nulls in 5 bytes (6 above 255), so a table is rejected from 1075 rows of 1000 nulls or 209723 rows of 25 (accepted: 1000 rows of 1000, 209000 rows of 25); rows of 20 or fewer are never rejected. Measured on the test data: no file of json_test_data (converted with to_ubjson in all three modes, 846 round trips) holds an array of this kind with more than 0.17 values per byte. All of them are read back. Reproducer (K arrays of 1048576 nulls), Apple clang -O2: K=40 (362 bytes): before 41943040 values, 664 MB; after rejected, 33 MB K=2 (20 bytes): before 2097152 values, 53 MB; after rejected, 33 MB The new tests count events with a SAX handler where they can: building and comparing million-element values made the first version take 176 s under Valgrind, the current one about 22 s. Behaviour change: an input that holds more than 1048576 elements in arrays of type Z, T or F at more than 4 per byte is now rejected. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Report an empty array for UBJSON [$N#<n>] instead of n elements (#6079) * Report an empty array for UBJSON `[$N#<n>` instead of n elements A no-op (N) is not a value, so an optimized array of type N holds no element whatever count it declares. The reader skipped the elements but still passed the declared count to the SAX parser: a handler was told start_array(n) and no element followed, and the DOM parser reserved min(n, 16384) values (256 KB) for an array that stays empty. Nine bytes (`[$N#l 0x7FFFFFFF`) therefore committed 256 KB: 2000 of them in one input, 18 KB, committed 512 MB of heap (capacity 163,840,000 values for the 10000 arrays of the original probe: 2.6 GB for 70 KB of input). Untouched pages keep the resident size small, but the commit charge and a 32-bit address space are consumed. The reader now announces 0 elements for an optimized array of type N. The loop that skips the elements is unchanged (it never ran). A count that does not fit into std::size_t is still rejected by the count reader; a count that is too large for the array type of the DOM (before: out_of_range.408 "excessive array size" from the DOM handler) is accepted now, because the array is empty. Behaviour change: sax_parse reports start_array(0) for `[$N#n` (before: n); from_ubjson no longer throws out_of_range.408 for an N-typed array whose count exceeds the maximum size of the array type. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Clarify the reservation note for announced array counts Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use fseek instead of rewind in the UBJSON FILE* test clang-tidy (bugprone-unsafe-functions) rejects rewind() because it has no error detection. Check the result of fseek() instead. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
79f6513cec |
Fix VS 2015 build of unit-bon8: compare tellg() outside doctest decomposition (#6128)
VS 2015 cannot find an operator== for doctest's Expression_lhs<std::fpos> and int, introduced in #6119. Evaluate the comparison as a plain bool. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
81888ff9f0 |
Report a missing 0xFF after a final BON8 string also without strict (#6119)
A BON8 string ends at the first byte that cannot continue it. When a
string ends a value and is not terminated by 0xFF, that byte belongs to
whatever follows the value, but the reader has already taken it from the
input and keeps it in its push-back buffer. With strict = true the
end-of-input check reports it; with strict = false the buffer was
discarded, so a following value in a stream lost its first byte or bytes
without an error ({0x61, 0x90} read "a" and then dropped the 0 that
follows).
Report parse_error.110 "string at the end of the value is not terminated
by 0xFF" when the push-back buffer is not empty once a BON8 value is
complete and strict is false, as bon8.md requires the final 0xFF. Output
of to_bon8 always has it, so canonical input is not affected.
Fixes #6110.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
Niels Lohmann
|
38cb5d0573 |
docs: output growth with nesting depth and ordered_json objects with many keys (#6118)
untrusted_input.md did not say that indented dump() and flatten() produce results that grow with the nesting depth times the size of the value (80 KB of input give 600 MB of dump(1)), nor that the binary readers take quadratic time for objects with many keys when they read into ordered_json. Add a warning to "Nesting depth", a section "Objects with many keys in ordered_json", and both points to the checklist. The warning in ordered_map.md still said that parsing JSON text into ordered_json is quadratic; that is no longer true. Name the cases that still are (filling an object key by key, the binary readers) and replace the table with current measurements. dump.md and flatten.md called their complexity linear in the value; it is linear in the result, which can be larger by a factor of the depth. Fixes #6108. Refs #6104. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
72966ae73d |
Force inlining of the serializer helpers only in optimized builds (#6117)
The seven serializer helpers marked JSON_HEDLEY_ALWAYS_INLINE are inlined at -O0 as well, and GCC and Clang give every inlined copy its own stack slots there. That made the recursive frame of dump_container() 2304 bytes with Clang 21 (2048 with GCC 16), against 480 bytes in 3.12.0, so dump() of a value nested beyond the default recursion depth limit needed 288 KiB of stack in an unoptimized build and crashed in a 128 KiB musl thread. Define JSON_INTERNAL_SERIALIZER_INLINE as JSON_HEDLEY_ALWAYS_INLINE when __OPTIMIZE__ (or _MSC_VER) is defined and as plain inline otherwise, and use it for these helpers. The frame drops to 320 bytes and dump() needs 48 KiB at the default limit. The -O2 object code is unchanged with Clang 21 and GCC 16. Update the stack tables in json_recursion_depth_limit.md and untrusted_input.md: copying and conversion are now the most demanding operations at -O0. Fixes #6116. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
dependabot[bot]
|
5bad6c518b |
build(deps): bump jsdom in /docs/mkdocs/scripts/mermaid (#6102)
Bumps [jsdom](https://github.com/jsdom/jsdom) from 30.1.3 to 31.0.0. - [Release notes](https://github.com/jsdom/jsdom/releases) - [Commits](https://github.com/jsdom/jsdom/compare/v30.1.3...v31.0.0) --- updated-dependencies: - dependency-name: jsdom dependency-version: 31.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
641e312f96 |
build(deps): bump watchdog from 7.0.0 to 7.1.0 in /tools/serve_header (#6101)
Bumps [watchdog](https://github.com/gorakhargosh/watchdog) from 7.0.0 to 7.1.0. - [Release notes](https://github.com/gorakhargosh/watchdog/releases) - [Changelog](https://github.com/gorakhargosh/watchdog/blob/master/changelog.rst) - [Commits](https://github.com/gorakhargosh/watchdog/compare/v7.0.0...v7.1.0) --- updated-dependencies: - dependency-name: watchdog dependency-version: 7.1.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Niels Lohmann
|
2755952d36 |
Fix develop CI: GCC 4.8, Clang 3.4 and clang-tidy errors (#6100)
* Fix develop CI: GCC 4.8, Clang 3.4 and clang-tidy errors - unit-json_patch.cpp: move multi-line raw strings out of CHECK(), which GCC 4.8 cannot parse inside a macro argument - unit-recursion_depth_limit.cpp: value-initialize the const std::hash (Clang 3.4 rejects default initialization of a const object) - json_view: silence modernize-use-transparent-functors (std::less<> is C++14), bugprone-suspicious-stringview-data-usage (the length comes from the node) and the deliberate use-after-clear() / owning-memory hits in unit-json_view_builder.cpp; use a raw string literal there Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy errors in the std::u8string tests Test the lvalue path with the const original instead of a mutable copy and the rvalue path with a temporary (misc-const-correctness), and mark the intentional char8_t array with NOLINT (avoid-c-arrays). Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
cdfb130fa6 |
Read a std::u8string from a JSON string and store it as one in every configuration (#6075)
* Read a std::u8string from a JSON string and store it as one in every configuration
In C++20, json(std::u8string) was stored as a JSON string only when
std::filesystem::path support was available (the to_json overload sat
inside the JSON_HAS_FILESYSTEM guard), and there was no matching
from_json, so get<std::u8string>() threw type_error.302 ("type must be
array, but is string"). Without std::filesystem, the same expression
still produced an array of code units (the 3.12.0 behavior).
Move the to_json overload out of the filesystem guard so that a
std::u8string is a string in every configuration, build the value
directly from the code units (this also works for a custom string_t
such as the tests' alt_string), and add a from_json overload that reads
a JSON string. The array of numbers written by 3.12.0 is still accepted;
other types throw type_error.302 ("type must be string or array").
std::u8string_view and char8_t arrays are unchanged.
The tests now run in C++20/23 (the file mentions JSON_HAS_CPP_20/23),
also without std::filesystem (test-conversions2_no_filesystem), and with
ordered_json and a custom string type. The new behavior is documented.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fill the std::u8string with resize and indexing instead of push_back
The push_back loop is not vectorized: reading a 1 kB string took about
2.2 us against 0.39 us with resize and direct element access (clang -O2).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Write the std::u8string from_json template header in the compact form
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Ignore libstdc++ char to char8_t sign change in the clang sanitizer job
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
Niels Lohmann
|
17e3de0fcc |
docs: add version-history notes for fixed start_pos and error positions (#6089)
Two bugs of 3.12.0 are fixed on develop without a note on the pages. With JSON_DIAGNOSTIC_POSITIONS, start_pos() of a string that contains escape sequences was too large; the position is now that of the opening quote. A parse error for an unexpected number that is followed by a line feed reported column 0 (for "1 2" and a line feed, line 1, column 0); for "1", line feed, "2", line feed it now reports line 2, column 1. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
07aa3d1d69 |
docs: fix NaN ordering, FILE* locking and binary integer-range statements (#6088)
* docs: state that NaN ordering inside containers depends on the standard When the element-wise comparison of arrays or objects reaches a NaN and a number (or another NaN) before any other difference, the result depends on the language standard. Before C++20 the pair counts as equivalent and the following elements (or the sizes) decide, so [1] <= [NaN] is true. Since C++20 the comparison is unordered and <, <=, >, >= all yield false. Document this on operator<, <=, >, >= and <=>. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: fix the FILE* notes of parse() and the locking promise The performance note of parse() said a FILE* is read one character at a time. With glibc and the Apple C library the characters in the FILE's buffer are scanned in bulk; only elsewhere is it one character at a time. The file is locked once only where the C library has unlocked stdio functions (not on musl, MinGW, newlib, with JSON_DISABLE_UNLOCKED_FILE_IO, or under MSan). Qualify the promise on parse.md, sax_parse.md, the parsing page and the performance page. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: fix integer ranges in the MessagePack and BJData tables number_integer_t is std::int64_t, so number_integer cannot hold values above 9223372036854775807; those are number_unsigned. The MessagePack row for uint 64 listed the range up to 18446744073709551615, and the BJData table had a uint64 row for number_integer that can never be selected. The CBOR table was already corrected. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: name the macros that turn off FILE* locking and bulk scans parse() and sax_parse() now say that the FILE* is locked unless JSON_DISABLE_UNLOCKED_FILE_IO is defined to 1, and the input-source table and the performance page name JSON_DISABLE_STREAM_BULK_SCAN and JSON_DISABLE_UNLOCKED_FILE_IO with what each turns off. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
dependabot[bot]
|
03ad4a3e7f |
build(deps): bump the codeql-action group with 3 updates (#6065)
Bumps the codeql-action group with 3 updates: [github/codeql-action/init](https://github.com/github/codeql-action), [github/codeql-action/analyze](https://github.com/github/codeql-action) and [github/codeql-action/upload-sarif](https://github.com/github/codeql-action). Updates `github/codeql-action/init` from 4.39.2 to 4.39.3 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/1fc58d04a4d4374f921bdac573c3811c240f9c26...33d7b85da1a04171e211befe2c739303b9a4c066) Updates `github/codeql-action/analyze` from 4.39.2 to 4.39.3 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/1fc58d04a4d4374f921bdac573c3811c240f9c26...33d7b85da1a04171e211befe2c739303b9a4c066) Updates `github/codeql-action/upload-sarif` from 4.39.2 to 4.39.3 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/1fc58d04a4d4374f921bdac573c3811c240f9c26...33d7b85da1a04171e211befe2c739303b9a4c066) --- updated-dependencies: - dependency-name: github/codeql-action/init dependency-version: 4.39.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/analyze dependency-version: 4.39.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/upload-sarif dependency-version: 4.39.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Niels Lohmann
|
e0b76bac31 |
Guard the image part of the alt_string test against big-endian targets (#6074)
The test for json_view documents over an alternative string type called alt_editable_document::load() and save(). Both exist only on little-endian targets, so the test suite did not compile on big-endian targets such as s390x. Guard the two image lines with the condition the other image tests use (unit-json_view_image.cpp, unit-json_view_positions.cpp). Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
02d57db31e |
Do not compare an edited NaN equal to itself in json_view (#6085)
Two views compared equal when the tokens of two floats were equal, and the same node was equal to itself. An edit writes a NaN as the token "nan", so a view of it (or of an array holding it) was equal to itself and to another NaN, while json(NaN) == json(NaN) and view == json are false. The token shortcut now excludes the edited "nan" token (found by the missing digit layout), dispatched on the document mode by a tag: a read-only document cannot hold a NaN, and its walk is unchanged (instruction count and cycles on canada, citm and twitter within noise). For editable documents the check costs about 2 % instructions on canada (cycles within noise). The same-node shortcut is kept for read-only documents and for editable ones that were never edited; a document with edits is walked, so comparing it with itself is linear instead of constant. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
9d35fe6d21 |
Reject integer tokens in float nodes of fully checked images (#6084)
image_check::full accepted a float node whose token is an integer literal that fits 64 bits (e.g. 10446744073709551616 changed from 18446744073709551616). The parser stores an integer node for such a token, so the loaded document differed from what parsing the same text gives. check_float_ranges now requires an integer token of a float node not to fit uint64_t (or int64_t when negative). Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
53d097f980 |
docs: correct the json_view dump() guarantee for loaded images (#6087)
dump() said a view never throws type_error.316 because the text is always validated. A document loaded with image_check::bounds or none can hold invalid UTF-8: dump() writes it unchanged, and dump(ensure_ascii) throws type_error.316. (The matching correction of number_token() is part of the read() documentation change below which this commit is stacked.) Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
94331d1050 |
Document that read() must not be given a view of the document itself (#6086)
* Document that read() must not be given a view of the document itself basic_json_document::read() releases the previous text, the decoded strings and the edits before it parses its input. An input that points into them (source(), get_string(), number_token(), or a string of an edited document) is read after its memory was cleared or freed. A guard that copies such input costs about 1 ns per read() (5 % of a read of a 7-byte input), which this API exists to avoid, so state the precondition in read.md, lifetime.md and the pages of the views that hand out such strings instead, and show how to copy the string first. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Assert in debug builds that read() is not given its own text read() releases the document's text, decoded strings, owned image and edits before it parses, so an input that points into them is read after release. Without NDEBUG, read() now asserts that a borrowed input, a copied contiguous input and (before C++17) a string_ref input are not held by the document: its owned text, its decoded strings, an owned image, and the current buffer of edited strings. Release builds keep the documented precondition without a check, which would cost 2-4.5 % per read() of a small message. JSON_ASSERT is undefined at the end of json.hpp, so the view uses assert() through a new internal NLOHMANN_VIEW_ASSERT. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
a7b377e9ec |
docs: state what the converting constructor does on a string/key mismatch (#6092)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
abab6f2f76 |
docs: qualify push_back({value}) and += {value} brace-init behavior (#6091)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
b5967d0ff9 |
Export the editing API of json_view from the C++20 module (#6090)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
1dae5a66b1 |
Add JSON_RECURSION_DEPTH_LIMIT to set how deep value walks recurse (#6081)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
0660d78b9c |
Correct the UTF-8 claims for binary input in the security guide (#6080)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
0d38e99d83 |
Resume patch operations from the common prefix in constant time per step (#6077)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
be08f42456 |
Fix parse(container, function) for plain functions as callback (#6073)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
93d2dd2c69 |
ci: pin Semgrep version without hashes (#6072)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
017a3ce001 |
ci: revert importlib-metadata to 8.7.1 to fix Semgrep install (#6064)
The Dependabot bump to 10.1.0 (#6059) conflicts with the pinned opentelemetry-api 1.37.0 (importlib-metadata<8.8.0), so the Semgrep job failed with ResolutionImpossible. Ignore importlib-metadata in Dependabot since semgrep pins it indirectly. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels LohmannandClaude Sonnet 5.5
|
8d591bf720 |
Read iterator pairs of std::string and std::vector as pointer ranges (#6029)
* Read iterator pairs of std::string and std::vector as pointer ranges Before C++20, a pair of iterators of a contiguous container cannot be told from any other random-access iterator, so json::parse(s.begin(), s.end()) on a std::string or std::vector<char> was read one character at a time, without the bulk scans of strings, numbers and indentation that the container itself gets (the container overload of input_adapter() already maps it to a pointer range). The iterators of std::string and of a std::vector of 1-byte integral elements are now matched exactly and mapped to the pointer range they cover; an empty range is not dereferenced. The character type stays the same, so the results and error positions do not change. parse(first, last) of a std::string gets 1.7-2.0x as fast in C++17 on an Apple M1 Max, accept(first, last) 2.7-3.1x, and from_cbor(first, last) of a std::vector 8-23% faster. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: clang-tidy, unused result, Infer false positive in tests - unit-deserialization.cpp: raw string literals, and assign the result of parse() in CHECK_THROWS_AS (-Wunused-result) - unit-bson.cpp: mark Infer's NULLPTR_DEREFERENCE on from_bson() with an iterator pair as a false positive Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix implicit char conversion in iterator-pair test The test built std::vector<unsigned char> and <signed char> from a char range, which -fsanitize=integer reports as an implicit conversion. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: sweep_allocator::construct() accepts pointers to const Recent libc++ (emsdk latest) probes construct(const T*, const T&) in an unevaluated context while checking for trivial copies; the static_cast to void* in the noexcept specification then fails to compile. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: guard sweep_allocator::construct() against const types instead of const_cast The const_cast added in the previous commit is rejected by cppcoreguidelines-pro-type-const-cast (clang-tidy). Make construct() not viable for const U instead, which is what libc++ probes for. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: astyle and clang-tidy for sweep_allocator::construct() astyle 3.4.13 reformats the enable_if<!...> default template argument (Check amalgamation). Delete the const overload instead, which makes the libc++ probe for const objects fail in the SFINAE context just the same and needs no <type_traits>. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Iterator ranges: astyle-safe trait members; reserve in the test helper The byte check and the container match are static constexpr members instead of integral_constant arguments and a default template argument with comparisons, which astyle spaced out. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
Niels LohmannandClaude Sonnet 5.5
|
2a4aec3e69 |
SWAR and vector scans: test for special bytes in fewer operations (#6033)
* SWAR: test for special bytes in the reduced form The usual test for a zero byte, (x - ones) & ~x & high, needs `& ~x` only to keep bytes from 0x80 out. The tests for quotes, backslashes and control characters in string_scan.hpp and the serializer also flag every byte from 0x80 with `v & high`, so the `& ~x` terms can be dropped. Clang finds this reduction itself; GCC does not and emits 24 instead of 16 instructions. The masks are the same for 1.3 billion random words and for every pair of adjacent byte values. With GCC 15 on x86-64: accept() -2 to -4 %, sax_parse() -2 %, dump() -3 %. Clang is unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Vector scans: one compare for the quote and the control characters The SSE2/NEON tests for the bytes a string run stops at compared with the quote, with the backslash and with 0x20 separately. Flipping bit 1 maps 0x00-0x1F onto itself and the quote onto 0x20, and every other ASCII byte to 33 or more, so (c ^ 2) < 33 finds the quote and the control characters with one compare: signed where the bytes from 0x80 stop the run too, unsigned (or with a saturating subtraction / minimum on SSE2) where they do not. This saves one vector operation per 16 bytes in all twelve vector tests: the lexer's vector_plain_run() and vector_utf8_run(), dump()'s copy_plain_halves() and copy_plain_vectors(), and json_view's scanner and serializer. The DEL tests of the ensure_ascii paths stay. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: sweep_allocator::construct() accepts pointers to const Recent libc++ (emsdk latest) probes construct(const T*, const T&) in an unevaluated context while checking for trivial copies; the static_cast to void* in the noexcept specification then fails to compile. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz (cherry picked from commit 794bf70823c12a2a1cb2a8a95f34dfb6357746a8) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: guard sweep_allocator::construct() against const types instead of const_cast The const_cast added in the previous commit is rejected by cppcoreguidelines-pro-type-const-cast (clang-tidy). Make construct() not viable for const U instead, which is what libc++ probes for. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz (cherry picked from commit de099fa607001b710862fd1bbecef3609c54cbc4) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: astyle and clang-tidy for sweep_allocator::construct() astyle 3.4.13 reformats the enable_if<!...> default template argument (Check amalgamation). Delete the const overload instead, which makes the libc++ probe for const objects fail in the SFINAE context just the same and needs no <type_traits>. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
Niels Lohmann
|
8c0b48088c |
ci: ignore Dependabot bumps of packages pinned by semgrep (#6062)
semgrep requires exact or compatible-release versions of mcp, boltons and the opentelemetry packages. Bumping them alone makes the Semgrep job fail at pip install (see #6054, #6055, #6057, #6058, #6060, #6061), so leave them to move with the semgrep bump. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
dependabot[bot]
|
0dbeb39811 |
build(deps): bump importlib-metadata in /cmake/requirements (#6059)
Bumps [importlib-metadata](https://github.com/python/importlib_metadata) from 8.7.1 to 10.1.0. - [Release notes](https://github.com/python/importlib_metadata/releases) - [Changelog](https://github.com/python/importlib_metadata/blob/main/NEWS.rst) - [Commits](https://github.com/python/importlib_metadata/compare/v8.7.1...v10.1.0) --- updated-dependencies: - dependency-name: importlib-metadata dependency-version: 10.1.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
d430252840 |
build(deps): bump the codeql-action group with 3 updates (#6056)
Bumps the codeql-action group with 3 updates: [github/codeql-action/init](https://github.com/github/codeql-action), [github/codeql-action/analyze](https://github.com/github/codeql-action) and [github/codeql-action/upload-sarif](https://github.com/github/codeql-action). Updates `github/codeql-action/init` from 4.39.1 to 4.39.2 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/9b3254085494b55a0f7c4c2b3d2d52cc712ce370...1fc58d04a4d4374f921bdac573c3811c240f9c26) Updates `github/codeql-action/analyze` from 4.39.1 to 4.39.2 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/9b3254085494b55a0f7c4c2b3d2d52cc712ce370...1fc58d04a4d4374f921bdac573c3811c240f9c26) Updates `github/codeql-action/upload-sarif` from 4.39.1 to 4.39.2 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/9b3254085494b55a0f7c4c2b3d2d52cc712ce370...1fc58d04a4d4374f921bdac573c3811c240f9c26) --- updated-dependencies: - dependency-name: github/codeql-action/init dependency-version: 4.39.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/analyze dependency-version: 4.39.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action - dependency-name: github/codeql-action/upload-sarif dependency-version: 4.39.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: codeql-action ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Niels Lohmann
|
9ed8f85b21 |
build(deps): bump semgrep from 1.168.0 to 1.184.0 in /cmake/requirements (#6052)
Regenerate the hash-pinned requirements file. This raises the transitive pins pyjwt 2.13.0 -> 2.15.1 and mcp 1.23.3 -> 1.29.0 (also click 8.1.8 -> 8.4.2, new structlog 26.1.0), closing 18 of the 23 open Dependabot alerts. The remaining alerts cannot be fixed until Semgrep relaxes its pins: mcp needs 1.30.0 (Semgrep pins mcp==1.29.0) and opentelemetry-api needs 1.45.1 (Semgrep pins ~=1.37.0). Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
88ab2d56f5 |
docs: update the README for the release (#6051)
* docs: update the README for the release - Show the animated SVG feature slideshow (light and dark variant) instead of json.gif. - Thank 20 further contributors and extend two existing entries. - Fix the bad_serializer example (invalid template header, second function named to_json instead of from_json, missing template keyword). - Add "Zero-copy views" to the table of contents, link ordered_json to its API page, and point the speed design goal to the performance page. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * chore: remove the stale ChangeLog.md and docs/json.gif ChangeLog.md was last generated in January 2025 and still ends at "unreleased (2024-12-22)"; 3.12.0 shipped without an entry. The release notes live on the GitHub releases page and in docs/mkdocs/docs/home/releases/. Remove the file, the Makefile target that called github_changelog_generator, and the README credit for that tool. docs/json.gif is an old copy of the feature slideshow that nothing references; the home page uses the SVGs built by docs/mkdocs/scripts/slideshow/. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * chore: remove the unreferenced docs/mkdocs/docs/images/json.gif The README header and the home page use the SVG slideshow built by docs/mkdocs/scripts/slideshow/. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
5901d861ff |
json_view: keep tokens without an integer digit out of the float batch (#6049)
dump() batches float tokens of 16 or 17 digits with at least 8 fraction digits. A damaged image loaded with image_check::bounds can record a token with no digit before the point and 17 after it, which no JSON number has. Its q = -17 is outside the range of locate_token(), which then read fives[17] past its 17 entries; on x86-64 the resulting digits had 18 digits and failed the assertion of write_decimal() (OSS-Fuzz issue 583898810, json_view_image_fuzzer). The batch now takes 8 to 16 fraction digits, which every valid token of 16 or 17 digits has, so it batches the same tokens of valid documents. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
eac1b4343b |
ci: install the remaining pip dependencies with --require-hashes (#6050)
Install Semgrep from a new hash-pinned requirements file in cmake/requirements (covered by Dependabot) instead of a version-only "pip install semgrep==1.168.0", and pass --require-hashes to the astyle and natvis installs in check_amalgamation.yml, whose requirements files got their hashes in #6046. Addresses the Scorecard Pinned-Dependencies code-scanning alerts for semgrep.yml and check_amalgamation.yml. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
a78e2a6b10 |
docs: blog post: remeasure on the 3.13.0 candidate and describe the latest performance work (#6041)
* docs: blog post: describe the performance work of #6020, #6028-#6031 and #6033
Theme 1 gets the iterator pairs of std::string and std::vector, a new
subsection on the fused contiguous parser after yyjson's reader, and the
FILE buffer window after gnulib's freadptr(). Theme 2 shows the reduced
SWAR test and the (c ^ 2) < 33 vector compare. Themes 4 and 6 get the
binary strings without a zero fill and the 64 KiB window of the vector
sink. Appendices H and J list the new pull requests and their checks.
The numbers of the suite, the timeline, the comparison and the compile
times are not remeasured yet.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: blog post: remeasure on develop with #6029, #6031, #6033 and #6039
Rerun on develop
|
||
|
dependabot[bot]
|
ffe77931b0 |
build(deps): bump mkdocs-git-revision-date-localized-plugin (#6047)
Bumps [mkdocs-git-revision-date-localized-plugin](https://github.com/timvink/mkdocs-git-revision-date-localized-plugin) from 1.6.2 to 1.6.3. - [Release notes](https://github.com/timvink/mkdocs-git-revision-date-localized-plugin/releases) - [Commits](https://github.com/timvink/mkdocs-git-revision-date-localized-plugin/compare/v1.6.2...v1.6.3) --- updated-dependencies: - dependency-name: mkdocs-git-revision-date-localized-plugin dependency-version: 1.6.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
2e52aa372b |
build(deps): bump watchdog from 6.0.0 to 7.0.0 in /tools/serve_header (#6048)
Bumps [watchdog](https://github.com/gorakhargosh/watchdog) from 6.0.0 to 7.0.0. - [Release notes](https://github.com/gorakhargosh/watchdog/releases) - [Changelog](https://github.com/gorakhargosh/watchdog/blob/master/changelog.rst) - [Commits](https://github.com/gorakhargosh/watchdog/compare/v6.0.0...v7.0.0) --- updated-dependencies: - dependency-name: watchdog dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
667fc8e3ab |
build(deps): bump katex and mermaid in /docs/mkdocs/scripts/mermaid (#6043)
Bumps [katex](https://github.com/KaTeX/KaTeX) to 0.18.10 and updates ancestor dependency [mermaid](https://github.com/mermaid-js/mermaid). These dependencies need to be updated together. Updates `katex` from 0.16.47 to 0.18.10 - [Release notes](https://github.com/KaTeX/KaTeX/releases) - [Changelog](https://github.com/KaTeX/KaTeX/blob/main/CHANGELOG.md) - [Commits](https://github.com/KaTeX/KaTeX/compare/v0.16.47...v0.18.10) Updates `mermaid` from 11.17.2 to 12.3.0 - [Release notes](https://github.com/mermaid-js/mermaid/releases) - [Commits](https://github.com/mermaid-js/mermaid/compare/mermaid@11.17.2...mermaid@12.3.0) --- updated-dependencies: - dependency-name: katex dependency-version: 0.18.10 dependency-type: indirect - dependency-name: mermaid dependency-version: 12.3.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Niels LohmannandClaude Sonnet 5.5
|
33f15727ce |
Parse JSON in one block of memory with a fused pointer-based parser (#6031)
* Parse JSON in one block of memory with a fused pointer-based parser The parser and the lexer exchange one token at a time, and the lexer keeps its state (the current character, a pending unget, where the token starts) in members, which every character pays for: json::accept() of twitter.json took 2.8x as long as json_document::accept(). detail::contiguous_parser reads text in one block of memory through a local pointer, with the grammar, the whitespace and the tokens in one loop (after yyjson's reader), and drives a SAX handler: parse()'s DOM handler, accept()'s handler, or the user's handler of sax_parse(). It is used with the default options (no callback, no comments, no trailing commas; sax_parse() with strict = true). Errors are reported by the library parser, so the exception, its message and its position stay the same. contiguous_parser stops at the token that the library parser would reject, before reporting it, so the events up to the error are the same; then: - parse() runs the library parser with a handler that only reports the error (json_sax_error_reporter), with the lexer of parse(), which is instantiated anyway; - sax_parse() runs it with a handler that forwards only parse_error() to the user's handler (json_sax_error_forwarder), so no event arrives twice; - accept() returns the verdict. A handler that returns false stops the parse as before. An error at the end of a large input costs parse() about 1.2x of what it cost before. On an Apple M1 Max: accept() 1.6-2.0x as fast, parse() 10-16% faster, ordered_json 11-16%, sax_parse() with a user's handler 1.36-1.46x, small messages 12-22% (parse()) and 2.1-2.4x (accept()). On x86-64 (GCC 15, Clang 21): accept() 1.6-1.9x, parse() 5-17%, ordered_json 12-19%, sax_parse() 1.32-1.57x. A translation unit that parses a string compiles 3% (-O0) and 9% (-O2) slower. unit-contiguous_parser.cpp compares the contiguous path with the byte path (std::list iterators) on corner cases, generated documents, their truncations and mutations, deep nesting, and narrow number types: the same values with the same number types and float bits, exception messages, accept() verdicts and sax_parse() events. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: unused parameter without exceptions, cpplint, BUILD.bazel - json_sax_error_reporter::parse_error: mark the exception unused, as JSON_THROW does not use it with JSON_NOEXCEPTION - unit-contiguous_parser.cpp: compare the exceptions of both paths only when exceptions are enabled (invalid input aborted the test otherwise) - silence cpplint's readability/braces false positives on the if statements inside the parser's macros - add contiguous_parser.hpp to BUILD.bazel Signed-off-by: Niels Lohmann <mail@nlohmann.me> * accept(): skip digits without computing their value accept() needs the value of a number only for a float whose exponent may reach the largest value. skip_digits() finds the end of the digits without the multiply-adds of accumulate_digits(), and once an 8-byte word fails the digit test, the remaining loop needs no bounds check: a non-digit is in that word. The value is computed afterwards for the rare floats that need it. accept() of canada.json: -16 % on Apple M1, -22 % with GCC 15 and -21 % with Clang 21 on x86-64; files with many numbers -9 to -15 %. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: mark parser callback in unit-contiguous_parser noexcept Fixes the -Werror=noexcept failure with GCC 26 (ci_test_standards_gcc (26)). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: sweep_allocator::construct() accepts pointers to const Recent libc++ (emsdk latest) probes construct(const T*, const T&) in an unevaluated context while checking for trivial copies; the static_cast to void* in the noexcept specification then fails to compile. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz (cherry picked from commit 794bf70823c12a2a1cb2a8a95f34dfb6357746a8) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: guard sweep_allocator::construct() against const types instead of const_cast The const_cast added in the previous commit is rejected by cppcoreguidelines-pro-type-const-cast (clang-tidy). Make construct() not viable for const U instead, which is what libc++ probes for. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz (cherry picked from commit de099fa607001b710862fd1bbecef3609c54cbc4) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: astyle and clang-tidy for sweep_allocator::construct() astyle 3.4.13 reformats the enable_if<!...> default template argument (Check amalgamation). Delete the const overload instead, which makes the libc++ probe for const objects fail in the SFINAE context just the same and needs no <type_traits>. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Contiguous parser: review fixes (wording, macros, diagnostic positions) - "default parser" instead of "library parser" in comments and tests - the three comments in json.hpp say the same thing - current() replaces the repeated `p != end ? *p : 0` - the backslashes of each macro are aligned - the docs say that JSON_DIAGNOSTIC_POSITIONS turns the fast path off Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
Niels Lohmann
|
b2626e25f6 |
ci: pin pip dependencies by hash (#6046)
Add sha256 hashes for every release file to the requirements of the astyle and natvis tools, and install Meson in the ubuntu workflow from a new hashed requirements file with --require-hashes instead of an unpinned "pip install meson". pip verifies all installs from these files once they contain hashes, including the ones of the Makefile and cmake/ci.cmake. Addresses the Scorecard Pinned-Dependencies alerts for ubuntu.yml; the check_amalgamation.yml flags follow once develop has the hashes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
77a608c2ce |
ci: run CodeQL with build-mode none (#6045)
Autobuild compiled the whole test suite. The resulting database was so large that the hosted runner repeatedly lost communication with the server while importing the TRAP files (about 80 minutes in), which left failed CodeQL analyses on develop and on pull requests. Analyze the sources without building them instead. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
6f01030c4e |
docs: fix json_view links in the 3.13.0 performance blog post (#6044)
#6018 split features/json_view.md into features/json_view/ sub-pages while #6022 added a blog post linking to the old page, so `mkdocs build --strict` failed on develop (Publish documentation run 42577793091). Point the links to json_view/index.md and json_view/editing.md. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
6a852ab96d |
Split the json_view feature page into sub-pages (#6018)
* Split the json_view feature page into sub-pages
The single features/json_view.md page grew to ~600 lines covering eight
topics. Split it into features/json_view/ with an overview page and
pages for ownership and lifetime, differences from basic_json, reading
values, generic code, editing, and images.
The new differences page collects what was spread over several sections
("What is the same", "What is different", "Duplicate keys", and notes in
the conversion and serialization sections) under one summary table, and
adds the parts of the basic_json interface a view does not have. The
other pages now link to it instead of restating the rules.
Internal links across the documentation point to the new pages.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Docs: fix stale comments and trim repeated text in json_view pages
- fig-jsonview.js/.css: point the header comments at the new pages
- reading.md: views' strings are valid as long as the document (and a
borrowed source text), as lifetime.md says
- reading.md, generic_code.md: link to differences.md instead of
restating the member-order and repeated-key rules
- differences.md: drop the third mention of ordering comparisons, name
rbegin()/rend() in the iterator row
- exceptions.md: link editable documents to editing.md
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
Niels Lohmann
|
7757de3a58 |
Iterator: test for objects before arrays (#6039)
#5835 replaced the switch on the value type in operator*, operator->, operator++, operator-- and operator== of iter_impl with an if chain that tests for an array first. Every step over an object then pays for a failed test and a taken branch: with the iterator inlined, walking twitter.json took 14% more cycles than with 3.12.0, and citm_catalog.json 3.5% more. Test for objects first. Walking every value of the three files now takes fewer cycles than with 3.12.0 (Apple M1 Max, Apple clang 21): citm_catalog.json 1.21x, twitter.json 1.04x, canada.json 1.17x as fast. canada.json, which holds only arrays, gives back part of the gain of #5835 (develop: 1.62x); the other orders tried (arrays first without hints, objects first without hints, a switch) each left one of the files slower than 3.12.0. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
2c7474affc |
docs: add the blog post "How nlohmann/json 3.13.0 got 3× faster" (#6022)
* docs: add the blog post "How 3.13.0 got faster" A post for the documentation (Home > Blog) on how 3.13.0 became faster: eight themes with the pull requests behind them, the ideas taken from other libraries, what did not work, how the fast paths are tested, and the compile-time cost. Every number comes from a run on 2026-10-27 (3.12.0 against develop plus the open non-draft PRs, Apple M1 Max) or from the linked PR. - docs/mkdocs/docs/home/blog/how_3_13_0_got_faster.md: the post. - docs/mkdocs/scripts/figures/perf_blog.py: draws its 12 figures (overview, eight timeline panels, the SWAR test, the cliffs, json_view throughput) as inline SVGs in the nj-charts style; data/perf_timeline.csv holds the timeline. - tests/benchmarks/history: history.cpp and history.py time one fixed program against the header of every commit that changed it, interleaved, and write one CSV row per commit with its PR. The timeline of the post comes from it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: compare json_document with Glaze, sonic, RapidJSON and Rust parsers Extends the comparison in "How 3.13.0 got faster" from yyjson, simdjson and Boost.JSON to eleven engines: sonic-cpp, RapidJSON and Glaze (generic DOM, lazy_json, reading into structs) in C++, sonic-rs, simd-json and serde_json in Rust. Measured on an Apple M1 Max (Apple clang 21, Rust 1.99) and an x86-64 Haswell server (GCC 13, Clang 18). Every engine must return the same values, string bytes and bit-identical number sums before it is timed. The post now answers whether json_document is the fastest parser: it is the fastest document parser at parse+dump everywhere and at parsing on Apple silicon, ties with simdjson's DOM on x86-64, and loses to simdjson On-Demand and Glaze structs when only a few fields are read. tests/benchmarks/json_view/others holds the programs (bench_all.cpp, the Rust crate with its Cargo.lock), summarize.py and the pinned versions and checksums. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: captions, test-file table and figure fixes for the 3.13.0 post - caption every table and figure - describe what makes twitter.json, citm_catalog.json, canada.json and jeopardy.json hard, and what each timeline panel exercises - split the json_view layout figure into the tree and the index, with the example text above; fix the arrows into the string boxes - arrows and a "less is better" axis label in the cliffs figure - name nlohmann/json in the comparison tables and chart - licenses of the adapted code in the thanks section Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: byline and licenses for the 3.13.0 post - author and date below the title, styled as .nj-byline; its link gets no external-link icon - the license of every library in the thanks section, checked against each project's license files; credit sonic-rs for QUOTE_TAB, which orjson adapted from it Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: dump comparison and split acknowledgments in the 3.13.0 post - a dump workload in bench_all and bench_rs: each engine writes a document parsed beforehand, and every output must read back with bit-identical numbers; an optional argument selects one workload - the post's comparison gains a write table, a throughput chart, a dump row for all three configurations, and the x86-64 cases it loses - the thanks section is split into adapted code and borrowed ideas Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: x86-64 dump numbers with #5908's branch-free token digits The dump of canada.json with GCC on x86-64 lost to yyjson by 0.78 because GCC compiled #5908's token-digit selections as branches; |
||
|
dependabot[bot]
|
6343775000 |
build(deps): bump reuse from 6.2.0 to 6.3.0 in /cmake/requirements (#6035)
Bumps [reuse](https://github.com/RichardLitt/standard-readme) from 6.2.0 to 6.3.0. - [Release notes](https://github.com/RichardLitt/standard-readme/releases) - [Commits](https://github.com/RichardLitt/standard-readme/commits) --- updated-dependencies: - dependency-name: reuse dependency-version: 6.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Niels Lohmann
|
7fc8312c64 | Fix ci_test_clang_sanitizer: build the CBOR test string from bytes (#6037) | ||
|
Niels LohmannandClaude Sonnet 5.5
|
18138227dc |
Scan the buffer of a FILE in bulk with glibc and the Apple C library (#6030)
* Scan the buffer of a FILE in bulk with glibc and the Apple C library Parsing from a FILE* still read one character at a time through getc_unlocked(), while a std::istream has its get area scanned in bulk. file_input_adapter now offers the lexer the same kind of window into the buffer of the FILE: it reads the fields that the C library's own getc_unlocked() macro reads (glibc: _IO_read_ptr and _IO_read_end; Apple: _p and _r) and consumes exactly the characters getc_unlocked() would return, so the file is left where it was left before. The part of the current token in a buffer that is about to be refilled is kept for error messages, as for streams. The window needs the file locked for the whole parse, so JSON_DISABLE_UNLOCKED_FILE_IO and MemorySanitizer builds turn it off; uClibc, which defines __GLIBC__ but has other fields, does not get it. parse() from a FILE* gets 2.0-2.6x as fast on an Apple M1 Max and 1.7-2.2x on x86-64 (glibc, GCC 15 and Clang 21), accept() 2.9-4.4x; that is about as fast as parsing from a std::ifstream. The new tests read with FILE buffers of 1 to 70 bytes and unbuffered files, after ungetc(), with SAX positions and with binary strings across the buffer's edges. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: no setvbuf() buffer below 2 bytes on Windows The Microsoft C runtime rejects a _IOFBF buffer size below 2 as an invalid parameter and ends the process (0xc0000409), which made test-file_input fail on all MSVC and clang-cl jobs. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: astyle formatting in unit-file_input.cpp Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: limit the multi-character ungetc() test to glibc and Apple The Windows C runtime only supports one pushed-back character, which broke the Windows CI jobs. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: sweep_allocator::construct() accepts pointers to const Recent libc++ (emsdk latest) probes construct(const T*, const T&) in an unevaluated context while checking for trivial copies; the static_cast to void* in the noexcept specification then fails to compile. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz (cherry picked from commit 794bf70823c12a2a1cb2a8a95f34dfb6357746a8) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: guard sweep_allocator::construct() against const types instead of const_cast The const_cast added in the previous commit is rejected by cppcoreguidelines-pro-type-const-cast (clang-tidy). Make construct() not viable for const U instead, which is what libc++ probes for. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz (cherry picked from commit de099fa607001b710862fd1bbecef3609c54cbc4) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: astyle and clang-tidy for sweep_allocator::construct() astyle 3.4.13 reformats the enable_if<!...> default template argument (Check amalgamation). Delete the const overload instead, which makes the libc++ probe for const objects fail in the SFINAE context just the same and needs no <type_traits>. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |