mirror of
https://github.com/nlohmann/json.git
synced 2026-11-09 16:21:35 +00:00
b6070980a189572357dec7c26f0a83fa31cbfdd0
233
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Claude
|
b6070980a1 |
Merge develop into claude/fix-issue-3989-db7e45
Resolve the conflict in binary_reader.hpp: develop dropped the document_size < 0 check in check_bson_document_size(), because sizes below 5 are rejected before they are stored. Here such a size is only stored after it was repaired, and the !size_repaired guard skips the check, so the check is dropped here as well. Regenerate the single headers. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEVkSyiaFoX2FtPpsU |
||
|
Niels LohmannandClaude Sonnet 5.5
|
951271127b |
Binary readers: append strings straight from contiguous input (#6028)
* Binary readers: append strings straight from contiguous input get_bytes() read every string into a zero-filled resize() and copied the bytes over it, so each byte was written twice. A string that the input holds in one block (bulk_remaining() at least its length) is now appended from it directly. The claimed length is checked against the input first, so a huge claimed length still allocates nothing it cannot fill; other inputs and binary values keep the chunked loop. from_cbor() and from_msgpack() of twitter.json and citm_catalog.json get 5.5-9.1% faster on an Apple M1 Max and 2.7-8.6% on x86-64 (GCC 15, Clang 21); canada.json, which has few strings, is unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Binary readers: keep the tag argument of append_from_block() astyle-proof Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: bugprone-signed-char-misuse in append_from_block Convert the (positive) length to its unsigned type before widening it to std::size_t, as clang-tidy flags the int8_t to std::size_t cast. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: sweep_allocator::construct() accepts pointers to const Recent libc++ (emsdk latest) probes construct(const T*, const T&) in an unevaluated context while checking for trivial copies; the static_cast to void* in the noexcept specification then fails to compile. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz (cherry picked from commit 794bf70823c12a2a1cb2a8a95f34dfb6357746a8) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: guard sweep_allocator::construct() against const types instead of const_cast The const_cast added in the previous commit is rejected by cppcoreguidelines-pro-type-const-cast (clang-tidy). Make construct() not viable for const U instead, which is what libc++ probes for. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz (cherry picked from commit de099fa607001b710862fd1bbecef3609c54cbc4) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: astyle and clang-tidy for sweep_allocator::construct() astyle 3.4.13 reformats the enable_if<!...> default template argument (Check amalgamation). Delete the const overload instead, which makes the libc++ probe for const objects fail in the SFINAE context just the same and needs no <type_traits>. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
Niels Lohmann
|
e48800f09d |
Binary reader and writer: remove checks that invariants make dead (#6001)
* Binary reader and writer: remove checks that invariants make dead
Branch coverage of the test suite and of 24k mutated inputs showed
these conditions as never taken; each is implied by a check made
earlier on the same path:
- check_bson_document_size: `document_size < 0` (the size was rejected
below 5 before it was stored)
- get_ubjson_ndarray_size: `second != 'N'` ('N' is rejected as a type
marker before `second` can be set)
- bjd_type_name() == nullptr for an ndarray dtype: every marker
is_valid_optimized_type() accepts for BJData has a name (now an assert)
- the early is_bjd_excluded_optimized_type() check in
get_ubjson_size_type(): is_valid_optimized_type() rejects the same
eight markers with the same error
- `input_format == bjdata` next to `is_ndarray`, bit 8 of the type, or a
'B'/'u'/'m'/'M' marker: these only arise for BJData
- `first != npos` next to bit 8: the bit is only set after a count
- MessagePack writer: the `<= max` halves of the negative int8/16/32
ladder (the ladder is in the `else` of `n >= 0`)
No change in behaviour: a build that aborts on each removed side passes
the test suite and the fuzz corpora, and a 1.6M-input differential run
of the reader against the original headers found no difference.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Remove is_bjd_excluded_optimized_type(), which has no caller left; fix two comments
is_valid_optimized_type() rejects the markers BJData excludes, so the reader
no longer needs its own list; the test now checks is_valid_optimized_type()
of a BJData reader against the writer's list. The BSON size comment now says
that open_bson_document() rejects a size below 5.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the BJData reader in the lookup test const (misc-const-correctness)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
Niels Lohmann
|
efc437ef6e |
UBJSON/BJData/MessagePack reader: reject a container count equal to SIZE_MAX (#6002)
* UBJSON/BJData reader: reject a container count equal to SIZE_MAX
The reader marks a container whose count is not known by npos, which
is SIZE_MAX. A count of exactly that value was stored unchanged and
then read as "no count", so the BJData input
[ # M ff ff ff ff ff ff ff ff i 01 ]
parsed as [1] instead of throwing out_of_range.408 as a count of
2^64-2 does. Where size_t is 32 bits, the same holds for 'm' and 'L'
counts of 2^32-1, in UBJSON too.
The counts of 'm', 'M' and the signed types now go through
store_ubjson_count(), which rejects npos with "integer value overflow"
(the message a count beyond size_t already gets), as the CBOR reader
does in get_cbor_container_size(). An ND-array dimension of 2^64-1 gets
the same message now, where it was "excessive ndarray size caused
overflow" (the product check) on 64-bit platforms.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* MessagePack reader: reject an array/map 32 length equal to SIZE_MAX
On a platform with a 32-bit std::size_t, an array 32 or map 32 length of
0xFFFFFFFF equals unknown_size(), so start_array()/start_object() were
told the size was unknown and skipped the max_size() check. Route both
through the CBOR length check, renamed get_container_size(), which
already rejects that value.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
Niels Lohmann
|
6d5581c4e8 |
CBOR/MessagePack readers: leave one-element arrays to the general path (#6008)
enter_cbor_array and enter_msgpack_array called the out-of-line scalar run for every non-empty array. When the array's element is an object or an array, as on every level of [[...]] or the single ring of a GeoJSON polygon, the call reads nothing; it made 1,000 arrays nested in each other cost 4 % more instructions (GCC 13) and up to 8 % more time (Apple clang) than without the run. #5975 had fixed that with an inline pre-check, which was later dropped after measuring canada only. The run is now entered only for arrays of two or more elements: the same comparison as before, so no other row changes. Arrays of one scalar, which the run read faster, now take the general path; they are rare in documents (one in citm_catalog, none in twitter or canada). The new benchmark FromBinaryShape/singles measures them, next to FromBinaryShape/nested. Instructions per repetition, new/old, GCC 13 callgrind: nested arrays 0.973 (CBOR) and 0.971 (MessagePack), arrays of one scalar 1.037 and 1.021, every other row within 0.15 %. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
405b8dcb13 |
BSON reader: recover from the document size checks made on reading the size
Since #5930, a size below 5, a size that exceeds the enclosing document, and a size too small for the document's content are reported before the document is read. When recovering, such a document is now read up to its terminator, within the enclosing document, and its size is not reported a second time. BJData now checks an ndarray's type before the object that would hold it is begun, so that test expects null. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
02d2fa8605 |
Merge remote-tracking branch 'origin/develop' into claude/fix-issue-3989-db7e45
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
b2674a4a2f | BSON reader: check array element names in place in strict mode (#5984) | ||
|
Niels Lohmann
|
f022e3c59c |
Binary readers and writers: faster UTF-8 checks of strings (#5982)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
fb515ddc34 |
UBJSON/BJData readers: read optimized arrays of numbers in one loop (#5981)
* UBJSON/BJData readers: read optimized arrays of numbers in one loop An array with a type marker for a fixed-size number ($U, $i, $I, $l, $L, $d, $D, and BJData's $u, $m, $M), including the data of a BJData ndarray, gives its elements no markers. When the input is contiguous, read the elements that are in the input in one loop - load, byte swap for the format's byte order, emit - instead of one trip through parse_ubjson_internal and get_ubjson_value per element. Elements the input ends before, other types, and non-contiguous inputs are read as before. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * UBJSON/BJData tests: assign the parsed value instead of casting it to void GCC reports -Werror=unused-result for static_cast<void>(json::from_ubjson(...)) and json::from_bjdata(...): a cast to void does not silence warn_unused_result on GCC. Assign to a json, as the other tests do. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
edf8d17601 |
MessagePack reader: read runs of scalar array elements in one loop (#5980)
* MessagePack reader: read runs of scalar array elements in one loop As for CBOR: when an array opens and the input is contiguous, read the integers, floats, booleans, and nils that follow in a tight loop instead of going through parse_msgpack_internal and parse_msgpack_value for every element. The loop stops at any other kind of value and when fewer than 9 bytes are left; those values are read by the general path as before. Non-contiguous inputs are not affected. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * MessagePack reader: enter the scalar run only when a scalar follows As for CBOR: enter_msgpack_array tests the first byte and the remaining length inline and calls the out-of-line run only for a scalar, so arrays of maps or strings no longer pay for a call that reads nothing; the run's loop condition moves to the end (do-while). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * MessagePack reader: mark the initial accepted in the scalar run as a false-positive dead store ci_infer reports DEAD_STORE for 'bool accepted = true;' in the MessagePack scalar run, as in the CBOR one, because every case assigns accepted before it is read. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * MessagePack tests: assign the parsed value instead of casting it to void GCC (ci_test_gcc) reports -Werror=unused-result for static_cast<void>(json::from_msgpack(...)): a cast to void does not silence warn_unused_result on GCC. Assign to a json, as the other tests in the file do. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * MessagePack reader: drop the scalar pre-check before the run enter_msgpack_array now calls the run directly, and parse_msgpack_scalar_run tests the remaining count and length at the top of a while loop instead of relying on msgpack_scalar_follows(); its own switch already leaves on any value that is not a scalar. Instructions per repetition, new/old: SAX canada 0.946 (clang) and 0.973 (GCC 13), DOM read canada 0.991 / 0.992; no row more than 0.3 % higher. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * MessagePack reader: say why the initial accepted in the scalar run stays As for CBOR: Infer's DEAD_STORE report is right, not a false positive. The initializer is kept for clang-tidy's cppcoreguidelines-init-variables. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * MessagePack reader: give the scalar run the bulk tag instead of a wrapper As for CBOR: parse_msgpack_scalar_run takes the bulk-scan tag itself and is the function kept out of line, so parse_msgpack_array_start goes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * MessagePack reader: start the scalar run's accepted at false As for CBOR: Infer ignores dead stores of 0, so the suppression goes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * MessagePack reader: let the scalar run find its array itself As for CBOR: parse_msgpack_scalar_run reads container_stack.back() in its bulk version, so the caller no longer evaluates it for input that is not contiguous. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
aecbef3949 |
Binary readers: swap the bytes of floats with a single byteswap (#5976)
* Binary readers: swap the bytes of floats with a single byteswap byte_swap reversed float and double values byte by byte. GCC does not merge that loop into one bswap, and C++23's std::byteswap accepts only integers. Overloads for float and double now copy the bits into a uint32_t/uint64_t, call detail::byteswap, and copy them back. This applies to every binary reader that uses get_number. In the CBOR reader, the single- and double-precision float cases call a non-inlined helper. Without it, GCC stopped inlining the stream adapter's get_character() into parse_cbor_internal, which made reading from a std::istream about 0.5 % slower. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * CBOR reader: inline get_cbor_float() and always inline get() instead The never-inlined get_cbor_float() costs a call for every float read from input without bulk access: reading canada.json from CBOR through a FILE* became 5.6 % slower on Apple clang in the benchmark suite. The helper existed only because GCC 13 stopped inlining the stream adapter's get_character() into parse_cbor_internal. Always inlining binary_reader::get() fixes that as well. callgrind, all binary formats on buffer, std::istringstream and FILE* input (canada and citm_catalog): GCC 13 0.986-1.003, GCC 16 0.994-1.006, Clang 23 identical. Apple clang suite: FromBinaryFile/cbor/canada 0.939, geomean of the 53 binary reader rows 0.998. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
6c3ab966d3 |
CBOR reader: read runs of scalar array elements in one loop (#5975)
* CBOR reader: read runs of scalar array elements in one loop When a definite-length CBOR array opens and the input is contiguous, read the integers, floats, booleans, and nulls that follow in a tight loop instead of going through parse_cbor_internal's container logic and parse_cbor_value's dispatch for every element. The loop stops at any other kind of value and when fewer than 9 bytes are left; those values are read by the general path exactly as before. Non-contiguous inputs are not affected (tag dispatch on the bulk-scan capability). Also pass a member empty string as the number source to number_float instead of constructing a std::string for every float. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * CBOR reader: address review of the scalar-run reader - parse_cbor_scalar_run: set the size explicitly in every case that reads a value, instead of relying on a default that every other case overwrites. - Remove cbor_scalar_follows(): parse_cbor_scalar_run already returns on the same conditions, and without the pre-check SAX canada needs another 8.5 % fewer instructions with GCC 13 (0.915 against the previous version), with no row more than 0.5 % worse (Mac and GCC). - Restore the comment on why parse_cbor_internal copies the container frame. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * CBOR reader: enter the scalar run only when a scalar follows An array of containers or strings called the out-of-line run start once per array only to leave it at once; on nested data (arrays of objects) that call made reading slower than before the run reader. enter_cbor_array now tests the first byte and the remaining length inline and calls the run only for a scalar, so the run's loop condition moves to the end (do-while). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * CBOR reader: mark the initial accepted in the scalar run as a false-positive dead store ci_infer reports DEAD_STORE for 'bool accepted = true;' in parse_cbor_scalar_run because every case assigns accepted before it is read. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * CBOR reader: drop the scalar pre-check before the run enter_cbor_array calls the run directly again, and parse_cbor_scalar_run tests the remaining count and length at the top of a while loop; its own switch already leaves on any value that is not a scalar. The MessagePack reader is built the same way. Instructions per repetition, new/old: clang SAX canada 0.983, read canada 0.993; GCC 13 read canada 0.997, the other rows at most 1.004. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * CBOR reader: say why the initial accepted in the scalar run stays Infer's DEAD_STORE report is right, not a false positive: every case that reaches the read assigns accepted. The initializer is kept for clang-tidy's cppcoreguidelines-init-variables. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * CBOR reader: give the scalar run the bulk tag instead of a wrapper parse_cbor_scalar_run takes the bulk-scan tag itself and is the function that is kept out of line, so the one-line parse_cbor_array_start wrapper goes, as the UBJSON typed run already does. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * CBOR reader: start the scalar run's accepted at false Infer ignores dead stores of 0 (--liveness-ignored-constant defaults to "0"), and false is 0 to it, so the suppression and its comment go. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * CBOR reader: let the scalar run find its array itself parse_cbor_scalar_run reads container_stack.back() in its bulk version, so the caller no longer evaluates it for input that is not contiguous. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
76001c6b4c |
BSON reader: find C-string terminators 8 bytes at a time, skip array element names (#5968)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
60107eda3f |
Read BJData ND-arrays of float16, restrict ND-array char to 0..127, correct BJData docs (#5931)
* Read ND-arrays of half floats and enforce the char limit in ND-arrays BJData N-dimensional arrays with element type 'h' (float16) failed to read with a misleading parse_error.112, because the reader's dtype name table lacked 'h'. It now maps to "half", the JData name. The writer still does not pack "half"; such an object is written as a plain object and reads back equal. BJData limits 'C' (char) to 0..127 in Draft 2 and Draft 3, and the 1-D reader enforces this, but ND-arrays with _ArrayType_ "char" wrote values up to 255 as 'C' elements and read them back without a check. The writer now only packs values in 0..127 (others fall back to a plain object that round-trips), and the reader rejects 'C' elements above 127 with parse_error.113 like the 1-D reader. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: correct and complete the BJData documentation Verify the BJData chapter against the library and the Draft 2/3/4 specifications and fix what did not match: - bjdata.md: correct the list of unused markers (N, not Z), the ND-array example (3x2, not 2x3), the conditions under which an object is packed as an ND-array (exactly three keys, row vectors, float-only elements for single/double, char 0..127, byte only in Draft 3), the binary table row and notes (which flag/draft combinations write [$B), the "$" type restriction (Draft 3, Z is excluded too), the H marker in the deserialization table, and the claim of a complete deserialization mapping. Document the supported drafts, that Draft 4 (SoA containers, extension type E) and column-major ND-arrays are rejected, ND-arrays of half floats, duplicate object keys, and link the Draft 3 specifications instead of master. - Add an API page for basic_json::bjdata_version_t with an example, and link it from the nav, the API index and to_bjdata. - Update binary_formats/index.md and binary_values.md, which still claimed that BJData has no binary type, and fix a wrong byte comment there. - Document parse_error.115 and the out_of_range.408 cases for from_bjdata, the precise other_error.502 condition for to_bjdata, and reword the note on truncated unsigned integers in exceptions.md (only ND-array elements were truncated before 3.13.0). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate single_include and apply astyle Result of make amalgamate (including make pretty) for the preceding changes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review of the BJData documentation The $ type restriction exists since Draft 2 (Draft 3 added B), $ requires a count, ND char above 127 is rejected on read, and the new API page is listed in alphabetical order in the nav. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: format tables, use the Oxford comma Reformat the tables changed by this PR (BJData serialization and deserialization mappings, completeness and binary-value tables) so all rows are aligned, and add the Oxford comma to lists of three or more items in the added text. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
dc768a7d5d |
Fix BSON binary subtype 0x02 and check the declared document size earlier (#5930)
* Write and read the inner length of BSON binary subtype 0x02 The BSON specification defines the payload of the "binary (old)" subtype 0x02 as an int32 followed by that many bytes. to_bson wrote the plain payload instead, so decoders such as pymongo and libbson-based readers rejected the bytes, and from_bson returned the inner length as part of the payload (documented as lenient behavior in gh-5333). The writer now emits the outer length (payload + 4), the subtype, the inner int32 holding the payload size, and the payload. The reader requires an outer length of at least 4 and an inner length equal to the outer length minus 4 (the cases the bson-corpus lists as decode errors), reports parse error 112 otherwise, and returns only the payload. Both directions change together so that the round trip keeps working. The single-reservation fast path for scalar entries declines subtype 0x02 because it has no room for the inner length. This is a deliberate exception to the opt-in rule for behavior changes: the previous output was invalid BSON. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Check the declared BSON document size before reading its content The BSON reader compared a document's declared size with the number of bytes read only when it reached the terminator. The declared size never bounded the reads, so a SAX handler saw the keys and values of a document whose size was too small before parse error 112 was reported. The container frame now stores the position where the document ends according to its declared size. open_bson_document rejects a size below 5 (size prefix plus terminator) and an embedded document that does not end before the terminator of its enclosing document; the element loop rejects an element whose type byte leaves no room for the name terminator and the document terminator. The terminator check stays and still catches a declared size that is too large. No input that was accepted before is rejected now; documents with a wrong size are reported earlier, with fewer SAX events, a different byte position and, in three cases, a different message. The error id (parse_error.112) is unchanged and strict=false still does not skip the check. The tests that asserted the old messages and positions are updated, with the outer sizes of two inputs corrected so that they still exercise the embedded document. New tests cover the SAX events, a size below 5, and an embedded document that fits exactly. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Document BSON subtype 0x02, document size checks, and lenient reads Describe the inner length of binary subtype 0x02 that to_bson now writes and from_bson checks and removes. Add the document size checks (parse_error.112, not affected by strict=false) to bson.md and from_bson.md, and extend the list of lenient reads with the UUID and MD5 payload lengths (not checked), duplicate keys (last value wins), embedded NUL bytes in strings, and the ignored trailing bytes with strict=false. State that the members of an object are written in iteration order. Correct the statement that element names are never validated on read: they are checked and sanitized like string values by the error handler. Add the BSON message to the examples of out_of_range.415 and type_error.321, the subtype limit of 255 to the binary values page, CBOR to the formats that use byte_container_with_subtype, and fix a typo in the incomplete mapping box. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate single_include and apply astyle Result of make amalgamate (including make pretty) for the preceding changes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review of the BSON documentation Describe exactly when a too small document size is detected and note that subtype 2 data written by earlier versions is not readable. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: use the Oxford comma, drop discarded-value remarks Add the serial comma in a list added by this PR and remove the BSON example of type_error.321 (with the type_error.317 remark) so the docs do not mention discarded values for BSON; restore the original singular heading of that section. No tables were changed by this PR. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix -Weffc++ in the BSON recorder test and a clang-tidy use-auto hit Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy readability-redundant-member-init in BSON test Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
0c3570d44a |
Validate the UBJSON/BJData container type marker for count 0; fix UBJSON docs (#5929)
* Validate the optimized-container type marker for count 0 in UBJSON and BJData
The '$' type marker of an optimized container was only checked when an
element was read, so containers with count 0 accepted any marker:
`[$x#i0`, `{$x#i0` and `[$E#i0` parsed as empty containers although a
non-empty container with the same marker is rejected and the UBJSON
specification requires a legal marker after '$'.
get_ubjson_size_type() now checks the marker right after '$' against the
allowed set (UBJSON: Z N T F i U I l L d D H C S [ {; BJData: B U i u I m
l M L h d D C, with the existing exclusion of T F Z N S H [ {) and reports
parse_error.112 "marker ... is not a permitted optimized array type".
Input that was accepted and is now rejected: an optimized container with
count 0 and an invalid '$' type marker. Valid input is unaffected. Two
existing tests that used an invalid marker incidentally ("[$#" and a
negative-count case) were adapted.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Correct and complete the UBJSON documentation
Fix statements that disagree with the behavior (checked with probes):
- `Z` is null; the unused marker is `N`.
- `$` requires `#`; use_type with use_size=false throws other_error.502 only
for non-empty arrays, objects and binary values.
- NaN and infinity are written as IEEE 754 float64 and read back unchanged;
the UBJSON specification encodes infinity as null.
- `H` is documented (semantics, parse_error.115, out_of_range.406); the
"complete mapping" claim is qualified.
- use_type "same type" means same marker.
- Long Z/T/F arrays lose only the `$` annotation; the `#` count and the
element markers are kept.
- from_ubjson lists the 112, 113, 115, 406 and 408 causes that occur for
UBJSON, without the BJData-only ND-array wording.
- Duplicate object keys keep the last value; the `$` type marker is
validated even for count 0.
- Smaller fixes: float32 row, "map" -> "object", https link, exception 110
strict case, to_ubjson history of discarded values.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate single_include and apply astyle
Result of make amalgamate (including make pretty) for the preceding changes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review of the UBJSON documentation
Clarify that empty containers carry no $ annotation of their own and note that a counted N object is not supported.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: format tables, use the Oxford comma, drop discarded-value remarks
Reformat the two UBJSON mapping tables that this PR changed (uniform column
widths, padded cells), use the serial comma in lists of three or more in the
added text, and restore the original type_error.321 bullet in to_ubjson.md.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix narrowing of the optimized-container count in BJData and UBJSON tests
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
Niels Lohmann
|
d5da84249f |
Skip CBOR tags in front of map keys, validate indefinite text chunks, fix CBOR docs (#5927)
* Skip CBOR tags in front of map keys and validate indefinite text chunks A tag in front of a text string map key was rejected with parse_error.113 even with cbor_tag_handler_t::ignore or store, although tags in front of values are handled. The tags are now skipped iteratively before the key is read; the handler error still rejects them. The chunks of an indefinite-length text string were concatenated before the single UTF-8 check, so a code point split over two chunks, or an invalid chunk, was accepted. RFC 8949, Section 3.2.3 requires every chunk to be valid on its own. With error_handler_t::strict, replace and ignore each chunk is now checked; keep is unchanged. The chunks are appended with append(ptr, len), so StringType needs nothing new. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: correct and extend the CBOR documentation Fix the negative-integer rows (they used MessagePack boundaries), the number_integer range of the 0x1B row, and the half-precision entry in the list of unused types; correct the NaN history (0xF9 since 3.8.0) and the tagged-item wording (only the tag directly in front of a byte string is stored, and only in the 0xD8..0xDB form). Document that to_cbor is not RFC 8949 deterministic, that duplicate keys are accepted (last wins), tags in front of keys, per-chunk UTF-8 validation, the exceptions parse_error.113, out_of_range.406/408 of from_cbor, the pointer and C array input traps, the partially written output on exceptions, and the store/ignore results in the binary values example. Drop the claim that the CBOR reader needs StringType::append(const StringType&) and BinaryType::insert(pos, first, last). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate single_include and apply astyle Result of make amalgamate (including make pretty) for the preceding changes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review of the CBOR changes Cite RFC 8949 Section 5.3.1 for duplicate keys, describe the out_of_range.408 condition precisely, rewrap a long line, and update the stale get_string comment (per-chunk UTF-8 check). single_include regenerated. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: drop discarded-value remarks Remove the mention that a discarded value cannot be serialized to CBOR (type_error.321) from the exception-safety and mapping sections. The integer table and all added prose already follow the table-format and Oxford-comma rules. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
a6c5d970f3 |
Merge branch 'develop' into claude/fix-issue-3989-db7e45
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
fe5e401870 |
Consolidate duplicated detail helpers and bodies (#5891) (#5901)
* Add shared detail helpers for the #5891 cleanup Add detail::is_primitive(value_t), detail::hex4_lower(), detail::utf8_shape(), detail::input_format_name(), detail::empty_input_message(), the exception ids 116, 319, 320 and 416 to detail::exception_id, and make detail::is_ascii_copyable() constexpr. Later commits switch the duplicated code over to them. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.4: BSON writers share one write-pass UTF-8 sanitizer (#5891) write_bson_entry_header and write_bson_string re-derived the error-handler policy differently. Both now use sanitize_utf8_for_bson_write, which treats strict like keep (the size pass already validated) and sanitizes replace/ignore into the bytes the size pass measured. No context plumbing is needed. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.1: one MessagePack container prefix helper (#5891) Array and object prefixes differed only in the fix marker and the 16/32-bit markers; both now delegate to write_msgpack_container_prefix. Output is byte-identical. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.2: one iterative walker for CBOR and MessagePack (#5891) write_cbor_iterative and write_msgpack_iterative (and their *_value_or_push helpers) were identical but for the head, scalar, and key writers. They are now one template over a small policy struct, so everything inlines. BON8 keeps its own walker (open string run, closing marker), noted in a comment. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.3, 2.1.8: one UBJSON container start helper, shared frame base (#5891) write_ubjson_start_array/_object are now write_ubjson_start_container<IsArray>, keeping the array-only begin()+1 scan and excessive_valueless rule. ubjson_frame derives from binary_container_frame instead of duplicating it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.4: write_bytes helper for the byte-string write sites (#5891) The 13 oa.write_characters(reinterpret_cast<const CharType*>(x.data()), n) sites now call write_bytes(const void*, size_t); copy_characters sites and the buffer-based writes are untouched. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.5: shared object-key writer for CBOR and MessagePack (#5891) write_cbor_key and write_msgpack_key were the same SFINAE pair apart from the string/scalar writers; both now delegate to write_object_key<Walk>. UBJSON key and to_format_length stay as they are. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.10: writer-side format names from input_format_name (#5891) The CBOR/MessagePack/UBJSON/BSON/BJData/BON8 literals in throw_on_discarded and throw_integer_out_of_range calls now come from detail::input_format_name; the texts are identical, and UBJSON vs BJData still follows use_bjdata. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * serializer: constexpr ascii table, hex4_lower, dedupe helpers (#5891) 1.1: build ascii_copyable() at compile time from is_ascii_copyable and test find_ascii_copyable_run against it for every byte. 1.2/2.5.4: use detail::hex4_lower in both \uXXXX writers. 2.1.6: factor the repeated flush/emit blocks in dump_escaped_impl into always-inline helpers. 2.1.7: comment why the escape copy does not use put_buffer. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.6: use string_t-aware to_string/concat in meta() (#5891) Replaces the four std::to_string sites; rendered text unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.7: use detail::is_primitive in json.hpp switches (#5891) has_no_children, set_parents, max_size and the two diff walkers. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.1: factor insert()/iterator throw helpers (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.2: share swap() overloads via swap_impl (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.4: factor primitive reset and range check in erase()/range ctor (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.6: share push_back array bodies via push_back_array_impl (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.7: route parse/accept/sax_parse through *_impl helpers (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.8: share the vector-returning to_* bodies via to_binary_vector (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.10: merge destroy_string/destroy_binary into destroy_pointer (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.1/1.7: exceptions: share what_impl, flatten diagnostics switch (#5891) The four create() bodies build their message through one protected static helper; the diagnostics parent walk handles array/object and ignores all other types. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.2/2.4.3/1.7: json_pointer: share slash and escape checks, drop primitive case lists (#5891) The leading-slash and invalid '~' escape checks are shared helpers; the reference-token walks test object and array first and treat everything else as primitive. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.7/2.4.9: iter_impl, iteration_proxy: drop primitive case lists, share compare preamble (#5891) Object and array stay the first branches; everything else is the primitive case. operator== and operator< share the different-container check. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.5/2.4.6: from_json: share get_number_value and reserve_if_possible (#5891) The three number cases of get_arithmetic_value and the arithmetic from_json share one helper (boolean stays accepted only by the latter); the array and object reserve overload pairs become one. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.8: ordered_map: share erase_impl between the key erase overloads (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.7: json_pointer: flatten walk drops the primitive case list (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.2/1.3/1.5/2.5.1/2.5.2/2.5.5: share helpers in the view code (#5891) - detail::exception_id at every numeric id; throw_* helpers take it - hex_byte, hex4_lower, utf8_shape replace local hex/table code - throw_array_index_out_of_range/_past_end, throw_key_not_found - has_native_float_format replaces the hand-written IEEE check No change of messages, ids or output. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.5: use utf8_shape in get_bon8_string, add exhaustive UTF-8 validator test (#5891) get_bon8_string takes the valid range of the second byte from utf8_shape() (0xF5..0xF7 stay errors, unget behavior unchanged). validate_one_utf8 keeps its if-chain (hot bulk path); the new test checks decode, validate_one_utf8, utf8_shape, json::accept and the BON8 reader against each other for every lead and second byte. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.1: get_cbor_sized_bytes for the CBOR chunk getters (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.2: get_string_with_length for the plain-order string sites (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.3, 2.3.4, 2.3.10: unsupported_key_type, empty_input_message, input_format_name in the readers (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.7, 2.3.9: read_object_key and leave_container_and_check for CBOR/MessagePack/BON8 (#5891) UBJSON keys (different getter signature) and its container tail stay as is. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.5: json_sax_dom_error_state base for the DOM SAX parsers (#5891) Holds errored, allow_exceptions, parse_error() and is_errored(). The scalar handlers stay in each class (parse hot path). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.6: table of control-character messages in the lexer (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.8: finish_input helper for the strict-EOF block in the parser (#5891) sax_parse keeps its if (result) guard and return value. Also adds the clang-tidy NOLINT for the 2.3.6 message table. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.5.7: share parse_array_index between json_pointer and json_view (#5891) Move json_pointer::parse_array_index() and its status enum to namespace detail (json_pointer keeps a forwarding member), and let the json_view pointer resolution use it instead of its own copy of the RFC 6901 index grammar. Both accepted the same tokens with the same outcome. Also update the comment on detail::decode(), which listed four of the five UTF-8 validators and now points to the test that keeps them in sync. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.5: keep the inlined number cases in from_json(ArithmeticType&) (#5891) Calling get_number_value() from from_json(ArithmeticType&) made GetArray<std::array<int, 1000>> 2.4x slower (it changes the inlining of the std::array conversion), so that overload keeps its own switch, with a comment. get_arithmetic_value() still uses the shared helper. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.5.7: force-inline detail::parse_array_index (#5891) Moving parse_array_index out of json_pointer stopped clang from inlining it into the pointer resolution (JsonPointerAt/citm_catalog +17%, JsonPointerAt/twitter +8%); JSON_HEDLEY_ALWAYS_INLINE restores the baseline. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.5: keep get_bon8_string's own UTF-8 ranges (#5891) Taking the second-byte range from utf8_shape() made BON8 decoding 2-3% slower (FromBinaryShape/scalars, FromBinaryBuffer, alignment-pinned builds), so the byte path keeps its hand-written ranges. The exhaustive validator test still checks that it accepts the same sequences as the other validators. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI failures: constexpr, NRVO, cppcheck, IWYU (#5891) - json_sax_dom_error_state::is_errored() cannot be constexpr: the class is not a literal type (GCC 4.8, GCC -Wpedantic, Clang 3.5). - to_binary_vector: destroy the vector writer before returning, as the output_vector_sink sets the size of the vector in its destructor. Without NRVO (MSVC Debug in C++14 mode), the vector was moved out before it was resized. - parser::sax_parse: return false instead of result after the early exit (cppcheck identicalConditionAfterEarlyExit). - json.hpp: export detail/exceptions.hpp for IWYU, so json_view.hpp can name exception_id without including a detail header. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review: throw helpers take input_format_t, simpler type checks - throw_on_discarded() and throw_integer_out_of_range() take an input_format_t and call input_format_name() themselves - has_no_children(), set_parents() and max_size() check array, then object, instead of calling is_primitive() first - parse_impl() chains like accept_impl(); sax_parse_impl() uses if/else - std::array for the control character messages; negation<> in write_object_key(); clarify the is_primitive() comment Signed-off-by: Niels Lohmann <mail@nlohmann.me> * json_view: use exception_id in the value() overloads Signed-off-by: Niels Lohmann <mail@nlohmann.me> * json_sax: assert the invariant of place_container for the static analyzer Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
a763c9eb83 |
Merge branch 'develop' into claude/fix-issue-3989-db7e45
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
9c50bdd140 |
Make the DOM parser cheaper per container and per member (#5882)
* Reserve the parser's nesting stack once The stack of open arrays and objects was a std::vector<std::uint8_t> that allocated at the first push_back and then reallocated at 1, 2, 4, 8 bytes, so a document nested four levels deep made four allocations before it parsed a value. A document that starts with a container now reserves 32 levels in one allocation; deeper documents grow as before. The loop that reads the tokens is the code it was, so large documents are not affected (a stack object with its own storage, and a reserve at every container, both cost the token loop instructions or cycles). Fixed cost of parsing a 124-byte object (macOS instructions): accept -7.6 %, DOM -2.5 %; no change for a single number. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Build arrays of the DOM parser in one exact-size allocation Reading an array of unknown length appended its elements to a std::vector one at a time, so a small array reallocated (and moved its elements) at sizes 1, 2, 4, ... The DOM SAX parser now collects the elements of arrays whose length is unknown on one stack shared by all open arrays and moves them into an array of exactly the right size when the closing bracket arrives. An array that grows past 16 elements switches to appending to its own vector, so large arrays pay no second copy. Binary formats, which announce the length, and builds with JSON_DIAGNOSTICS or JSON_DIAGNOSTIC_POSITIONS (whose values need their final place from the start) keep storing into the array directly. The stacks of open containers are reserved on first use instead of growing from one element. The binary readers get a DOM parser of their own, json_sax_dom_binary_parser: it is the plain parser that stores every value directly in its container (the code the readers used before). Their containers announce their size, so the bookkeeping of the collecting parser (frame entries, the destination of a finished container) was pure overhead for them: with the collecting parser FromBinaryShape nested was 3-12 % and containers up to 5 % more instructions (GCC) than before. With the plain parser Linux Ir of FromBinaryShape and FromBinaryBuffer is within 0.8 % of develop (Clang: identical). canada.json has 56 000 two-element arrays that took three allocations each and now take two. DOM parse Ir on Linux against develop (GCC / Clang): canada -7.0 % / -7.7 %, citm -0.8 % / -1.6 %, 124-byte object -5.4 % / -5.5 %; accept and SAX are untouched. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Reuse one value for the key events of the callback parser parse() with a callback built a new string value for every key it passed to the callback (a heap allocation and release per key), and copied the key a second time into the object. The callback parser now keeps one string value for key events, which it assigns, and moves the key into the object. Callback parse, Linux Ir (GCC / Clang): twitter -13.3 % / -11.7 %, citm -9.5 % / -7.6 %. (Earlier macOS measurement: twitter -19.5 % instructions.) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remember where the DOM parser stores the next value The DOM SAX parser looked up the innermost open container for every value: the end of its frame stack, then the object or array pointer in the frame. Compilers have to reload all of that after each value is stored, since a store of a value may alias the frame. The parser now keeps the innermost container's kind and, for an array that stores directly, the array itself in members, updated when a container opens or closes. Flat arrays of numbers are the case that mattered: parsing floats.json (1 000 000 doubles) into a DOM took 2 % more cycles than before the previous commit on macOS and now takes 0.4 % fewer than the code before it (signed_ints.json -0.8 %). Linux Ir is unchanged (canada -6.1 % GCC / -6.9 % Clang, citm -0.3 % / -0.8 %, twitter +0.2 % / 0.0 %). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Look up the keys of big ordered_json objects in a hash table ordered_map keeps its elements in a vector and searches it linearly for every key it adds, so parsing an object with n keys into ordered_json took O(n^2) key comparisons: 33 billion instructions for one object with 30 000 keys, where json needs 245 million. From 64 keys on, the DOM parser now keeps a hash table of the keys of the open object (hashes and positions only; the elements stay in the vector) and finds duplicate keys there. The result is the same, including the position of a key that occurs twice. Smaller objects, std::map based objects and key types without std::hash are untouched. ordered_json parse, macOS instructions: object with 1 000 keys 64.8 M -> 14.6 M, 10 000 keys 5 104 M -> 38.7 M, 30 000 keys 33 358 M -> 89 M. Linux Ir of an object with 2 000 keys: 43.0 M -> 1.9 M (GCC), 37.9 M -> 1.8 M (Clang). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Trim the work the DOM parser does per container and per member Opening and closing an array or object cost the DOM parser more than before the exact-size array change, which showed on documents of many small objects (jeopardy.json: +1.4 % Ir with GCC). This takes it back: - the stack of open containers keeps its first eight entries in the parser object instead of in a vector, so it needs no capacity check beyond the push itself and no allocation for typical documents; - a container is created in the place it ends up in, instead of as a temporary that is then moved there; - a new object member, which is null, is constructed over the null instead of being swapped with it (only with the default base class of basic_json); - the key index that big ordered_json objects use is only a part of the stack entries of objects that can have one; - an array that outgrows the stack reserves twice what it has so far, as a vector that grew by itself would have. DOM parse Ir of the series up to here against develop, Linux GCC / Clang: canada -6.6 % / -7.3 %, citm +1.2 % / +0.2 %, jeopardy +0.4 % / -0.1 %, 124-byte object -6.5 % / -6.2 %. (twitter measures +6.8 % / +6.6 % at this point in steady state, almost all of it in glibc's malloc/consolidate for the new allocation sizes; with the last commit of the series it is -0.1 % / 0.0 %.) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Test parsing containers of the sizes where the DOM parser changes storage The DOM parser collects the elements of an array on a stack (up to 16) and the members of an ordered_json object (up to 32) before it moves them into the container, and uses a hash table for keys from 64 members on. Parse arrays and objects of the sizes around these limits, nested into each other, with repeated keys, and as random trees that are built next to their text, and compare json and ordered_json against the value built through the API. The tests pass on the code from before these changes as well. The random trees are what finds the mistakes that are easy to make here: while the collecting change was being written, a collected array that outgrew the stack at the moment its parent had just reached the limit left its elements in the parent, and only these tests (142 failed assertions) showed it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not convert floats that cannot overflow in accept() accept() only has to find out whether a float token is finite: json_sax_acceptor ignores the value, and the parser rejects only a float that overflows. A float whose digits fit into 19 digits and whose decimal exponent is at most max_exponent10 - 19 is below 10^max_exponent10 and cannot overflow, so the lexer, told by accept() that the value is not needed, skips the conversion for those. A token that may overflow, with more digits, or in another float format is converted as before, so 1e309 and its neighbours are rejected as they were (the new test compares accept() with parse() on tokens around the limit). json::accept(): canada.json Ir -8.6 % (GCC) / -10.9 % (Clang), 124-byte object -3.2 %; cycles on the M1 (release benchmarks, LTO): canada -7.8 %, floats.json -9.0 %. The check tests the accept() flag first, so the parser that needs the value pays one load and branch: SaxParse canada +0.35 % Ir (GCC and Clang), DOM canada +0.4 %. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Collect the members of ordered_json objects on a stack until they are known An ordered_json object keeps its members in a vector and appends every new member to it, so reading an object of unknown size reallocated the vector at 1, 2, 4, 8, ... members, and ordered_map copies all keys on every reallocation. The DOM SAX parser now collects the members of such objects on one stack shared by all open objects, as it does the elements of arrays, and moves them into a vector of exactly the right size when the closing brace arrives. An object that gets past 32 members switches to appending to its own vector (and, from 64, to the key index of the earlier commit). The members are the same, in the same order, and a key that occurs twice keeps the place of its first occurrence. std::map based objects, objects whose size the input announces (binary formats), and builds with JSON_DIAGNOSTICS or JSON_DIAGNOSTIC_POSITIONS are untouched. What only ordered objects need (the collected members, the key indexes) is a member of the parser only for them, and the code that stores a value tries the array that takes its elements directly first. An array that outgrows the stack now reserves 32 elements, the capacity a vector that grew by itself has at that point; with the larger reserve before, parsing floats.json (1 000 000 numbers) was 2 % slower on Apple M1 than before this series. ordered_json parse, Linux Ir against develop (GCC / Clang): citm -11.6 % / -11.7 %, twitter -8.4 % / -8.4 %, jeopardy (12 000 objects of 7 keys) -10.5 % / -8.0 %, canada -3.9 % / -5.2 %, 124-byte object -20.9 % / -16.5 %. macOS time (M1, earlier base): citm -19 %, twitter -13 %, jeopardy -13 %, canada -14 %, 124-byte object -26 %. json parse: steady-state twitter -0.1 % (see the previous commits), citm -1.2 %. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Rename the key index's match callback so Flawfinder doesn't read it as std::equal Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Bind the parser test's strings by reference in the range-for loop GCC and Clang 14 (-Wrange-loop-construct) reported the copy. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Collect container values only in basic_json's own parse functions A json_sax_dom_parser that a user constructs (as in the sax_no_exception example) must hold the values read before a parse error, as on develop: its value is used after the error. The values of containers of unknown size are now collected on a stack only with the new template parameter Collect (default false), which json::parse() and the json_view materialization set; without it, every value is stored in place as it arrives. A test checks the values kept after an error for json and ordered_json. Also fixes the clang-tidy, cpplint and Infer findings: the frame stack keeps its first frames in a std::array (left uninitialized) and the others in a std::unique_ptr, auto for create<>(), NOLINT for member initializers -Weffc++ needs, and an @infer-ignore for the nullptr that place_container() does not use. json::parse() and ordered_json::parse() instructions are unchanged (Apple clang; twitter, citm, canada, jeopardy: -0.03 % to +0.25 %). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Clarify the DOM parser's collection stacks and rename reenter_enclosing_container() Address review comments: explain why a closed container erases only the top of the shared stack, set the payload before the type, rename the helper that makes the enclosing container current again, and declare the parser's state stack right before reserving it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Export json_sax_dom_parser from the C++20 module for MSVC MSVC needs the detail classes that basic_json befriends to be visible in the module (#3970, as json_sax_dom_callback_parser); with its new template parameter, json_sax_dom_parser failed in ci_module_cpp20 (C2039, not a member of nlohmann::detail). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep the overflow-limit numbers of the parser test in a std::vector clang-tidy (cppcoreguidelines-avoid-c-arrays) rejected the C array. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the clang-tidy findings of the ordered_json parser tests The container sizes are a std::array, the expected text uses raw string literals, the generator's arithmetic is in parentheses, auto for the result of make<>(), and single characters are appended as characters. clang-tidy now reports nothing on the lines the branch adds to unit-ordered_json.cpp and unit-class_parser.cpp (apart from checks CI's clang-tidy does not have). Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
6a0cf2ba2f |
Recover by error kind instead of by error message
The lexer records an error_kind with each error_message, and recover_token() dispatches on it instead of comparing message prefixes. add_codepoint() now uses encode_utf8(), and the surrogate pair formula is shared by scan_string() and recover_low_surrogate(). The stubs for AllowRecovery == false select via negation, which astyle leaves alone. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
c9fdfc39db |
Merge remote-tracking branch 'origin/develop' into claude/fix-issue-3989-db7e45
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
4da73dc5ff |
Fix compiler warnings on develop from the API performance stack (#5858)
These PRs were merged while their CI runs were cancelled, so the warnings went unnoticed. All of them fail the -Werror builds: - json_pointer.hpp (#5836): size_t += int under GCC -Wsign-conversion - binary_reader.hpp (#5839): char_int_type passed as char under GCC -Wconversion - json.hpp (#5838): @throw without a description under clang -Wdocumentation - string_escape.hpp (#5836): escape_into() was inserted between escape() and its documentation, so the @return was attached to a void function - unit-diagnostics.cpp (#5832): clang -Wnrvo in a lambda without an explicit return type - unit-to_json-rvalue.cpp (#5837): GCC -Wnoexcept (C++20) and clang -Wunneeded-member-function on the test type Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
870bdec01f |
Merge remote-tracking branch 'origin/develop' into claude/fix-issue-3989-db7e45
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
ed86019a21 |
Move binary reader error construction out of line (#5839)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
545477ffa8 |
Merge remote-tracking branch 'origin/develop' into claude/fix-issue-3989-db7e45
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
1649d9eda4 |
Use and extend the detail helpers to remove duplicated code (#5783)
* Use and extend the detail helpers to remove duplicated code Library: - binary_reader: format bytes with hex_byte() instead of snprintf - add throw_type_must_be() for the 20 copies of type_error.302 - binary_reader: add last_byte_error()/unexpected_byte() for the 32 "parse error at the last read byte" sites (replaces bon8_error) - json_sax: add check_container_size() for out_of_range.408 and diagnostic_positions::set_container_start/_end() - json_pointer: add throw_no_parent() (405) and throw_unresolved() (404) Tests: - unit-class_parser uses the shared utils::SaxCountdown - move SaxEventLogger (and its ExitAfter* variants) from unit-class_parser and unit-deserialization into the new tests/src/test_sax.hpp Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Factor out more repeated error paths and test boilerplate Library: - add throw_cannot_use_with() for the 34 copies of type_error.304-312 "cannot use X with Y" - iter_impl: add throw_cannot_get_value() (invalid_iterator.214) - parser: add syntax_error() for the 12 parse_error.101 sites - ordered_map: share the four at() bodies via at_impl() - json_sax_dom_callback_parser: add pop_container() for end_object() and end_array() - binary_reader: build the two UBJSON/BJData length-type messages with concat() and last_byte_error() Tests: - move same_value(), the NDEBUG guard, and step 0 (parse without exceptions) of the seven fuzzer drivers into tests/src/fuzzer_common.hpp Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Name the exception ids and address review comments Add detail::exception_id, a scoped enum with one named enumerator per documented exception id, and use it for every id in the library. The create() functions get an overload for it; the int overloads stay for user code. Following the review of #5783: add binary_reader::invalid_byte() and length_type_error(), basic_json::throw_subscript_wrong_type(), move the fuzzer includes and the using-declaration into fuzzer_common.hpp, and rename test_sax.hpp to sax_event_loggers.hpp. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Silence -Wweak-vtables for the SAX event loggers The loggers moved from anonymous namespaces in the test files into sax_event_loggers.hpp, so clang now warns that their vtables are emitted in every translation unit. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Silence MSVC 2015 C4100 in parser::syntax_error for static SAX::parse_error Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Inline invalid_byte() into its call sites The wrapper only fixed the message string of unexpected_byte(), which is the same kind of per-argument helper that was declined for throw_type_must_be(). Call unexpected_byte("invalid byte", ...) directly. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
1d2dea7769 |
Do not force inlining of the recovery stubs with MSVC
|
||
|
Niels Lohmann
|
f8ff0942b2 |
Keep the MinGW test objects below the linker's section limit
The MinGW clang job links with a GNU ld that cannot relocate an object with more than 32767 sections. unit-regression2, unit-regression3 and unit-msgpack went over it: - At -O0, the parser and binary reader for parse(), accept() and from_*() still emitted the stubs that stand in for recovering, and the parser referenced its recovery helpers from code that never runs. The stubs are now always inlined, and the parser reaches its helpers through the same allow_recovery tag as the stubs, so that this code is not even instantiated there. This removes about 300 sections from unit-msgpack. - Every sax_parse call instantiates the parser and binary reader that recover from errors. The tests that call it move from unit-regression2.cpp (#3989) and unit-regression3.cpp (#5676) into unit-sax_parse.cpp, like unit-explicit_instantiation.cpp got a file of its own (#5511), and CONTRIBUTING.md points there. Section counts of the MinGW Debug objects (clang 19, -g0), develop / PR: unit-msgpack 32281 / 32558, unit-regression2 31272 / 31278, unit-regression3 32392 / 27462, unit-sax_parse - / 19836. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
dcf7e2222c |
Merge remote-tracking branch 'origin/develop' into claude/fix-issue-3989-db7e45
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
069ace74af |
Round-trip BJData ND-array annotations exactly (single precision, key order) (#5707)
Squashed onto develop from: - Round-trip BJData ND-array annotations exactly (single precision, key order) Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
a2766df787 |
Merge remote-tracking branch 'origin/develop' into claude/fix-issue-3989-db7e45
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels LohmannandJoseph.Demarest
|
ff6f3d7d4a |
Reject nested indefinite-length CBOR string chunks (#5766)
* fix(cbor): reject nested indefinite string chunks Signed-off-by: Joseph.Demarest <joseph@demarest.dev> Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review comments on nested indefinite-length CBOR strings Rename is_chunk to inside_indefinite, update the stale test section names, and use lowercase comments like the surrounding code. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Joseph.Demarest <joseph@demarest.dev> Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Joseph.Demarest <joseph@demarest.dev> |
||
|
fhgffy
|
69874e4544 |
Fix NUL bytes in UBJSON/BJData high-precision numbers (#5760)
* Fix NUL-terminated UBJSON and BJData high-precision payloads Signed-off-by: fhgffy <102001626+fhgffy@users.noreply.github.com> * Use English comments for the high-precision NUL fix Signed-off-by: fhgffy <102001626+fhgffy@users.noreply.github.com> * Track NUL bytes while reading high-precision payloads Signed-off-by: fhgffy <102001626+fhgffy@users.noreply.github.com> * Drop dates from code comments. Signed-off-by: fhgffy <102001626+fhgffy@users.noreply.github.com> * Report a NUL in a high-precision number where it is read Return the parse error from the read loop so the byte offset points at the NUL, and drop the separate check after lexing. Signed-off-by: fhgffy <102001626+fhgffy@users.noreply.github.com> * Use fixed text for the high-precision NUL error 2026-10-06: Use the known zero byte directly instead of formatting and concatenating it. Preserve the SAX token, error message, and byte offset. Signed-off-by: fhgffy <102001626+fhgffy@users.noreply.github.com> --------- Signed-off-by: fhgffy <102001626+fhgffy@users.noreply.github.com> |
||
|
Niels Lohmann
|
632f5369fe |
Merge remote-tracking branch 'origin/develop' into claude/fix-issue-3989-db7e45
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
8a142bc436 |
Fix CI on develop after #5600, #5607, and #5755 (#5764)
* Fix CI on develop after #5600, #5607, and #5755 - binary_reader: cast the result of -1 - number back to number_integer_t, because a number_integer_t narrower than int is promoted to int, which GCC's -Warith-conversion rejects (ci_test_gcc, ci_test_standards_gcc) - JSON_DELETE_DEPRECATED_FUNCTIONS: declare the deleted stream operators as function templates at namespace scope, because GCC < 5 rejects deleted friend functions ("can't initialize friend function") and Clang 7-9 report a redefinition when a class template has a deleted friend function - docs: give the examples of JSON_USE_OBJECTS_FOR_ENUM_KEYED_MAPS "Example:" titles and add the page to the docset (style_check) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Ignore libstdc++'s <format> sign change in the sanitizer job libstdc++ 14's <format> initializes a size_t parameter with -1 (GCC bug 119429), so every std::format call fails ci_test_clang_sanitizer under -fsanitize=integer (test-std-format_cpp20). Exclude only the implicit-integer-sign-change check and only that header via -fsanitize-ignorelist. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the AppVeyor (MSVC 2015-2019) build - binary_reader: emit_signed/emit_unsigned pass integers that do not fit the number types to emit_float as long double, because MSVC's <cmath> has no integer overloads of std::isfinite (C2668 'fpclassify'), from #5607 - scalar comparisons: the friend operators take the JSON type for their noexcept from their parameter, because MSVC 2015/2017 take basic_json as the class template there (C3203) and MSVC 2019 16.0 does not see member types or template parameters, from #5751 - unit-conversions2: skip the !is_nothrow_constructible static_assert for std::optional on MSVC 2017, which evaluates the conditional noexcept as true (C2607), from #5754 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Re-amalgamate Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Avoid MSVC 2015's C4800 for enums with underlying type bool MSVC 2015 warns about any conversion to bool (C4800), even with an explicit cast, so the enum conversions from #5754 (#5671) failed the AppVeyor build with /WX. Convert to bool by comparing with zero via the new detail::bool_aware_static_cast, and keep doctest from printing the enum in the test. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Skip the #5650 allocator test on MSVC 2015 debug builds MSVC 2015's debug STL constructs the containers' debug proxies through the allocator in noexcept constructors, so countdown_allocator's failing construction crashes test-allocator (SIGSEGV) instead of throwing std::bad_alloc. Use the guard #5585 uses for the same reason. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Skip deleted-function detection checks on MSVC 2015 MSVC 2015 does not treat selecting a deleted function in decltype as a substitution failure, so the detection traits in unit-delete_deprecated_functions (#5755) and the integral-key checks in unit-element_access2 (#5657) report deleted overloads as callable there. Calling them still fails to compile. Skip those checks for _MSC_VER < 1910, and use the stream operators for real in the runtime section, so MSVC 2015 still compiles them with the macro set. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Split the Visual Studio 2017 AppVeyor jobs in two The VS 2017 jobs hit AppVeyor's 60-minute limit per job while still compiling the tests (77 of about 108 test targets after 58 minutes). They pass /std:c++17 for everything anyway, so build only the C++17 variant of each test (JSON_TestStandards=17), and split the unit test files across two jobs each with the new JSON_TestShard=<index>/<count> option, which keeps every <count>-th test file starting at <index>. The extra variants of single test files are built in shard 0 only. CMAKE_OPTIONS is no longer quoted in appveyor.yml, so that it can hold more than one option. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix std::terminate when converting std::optional with MSVC 2017 MSVC 2017 evaluates std::is_nothrow_assignable<json&, const T&> as true even if T's to_json throws, so to_json(json&, const std::optional<T>&) was noexcept there and the exception from #5642's test called std::terminate instead of propagating. Make that conversion never noexcept on MSVC 2017; all other compilers keep the exact condition. The static_asserts on the condition are skipped for MSVC 2017; the runtime check that the exception propagates still runs there. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
c2619bf460 |
Merge remote-tracking branch 'origin/develop' into claude/fix-issue-3989-db7e45
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
6b4b825af2 |
Handle numbers that do not fit narrow number types in the binary readers (#5607)
* Handle numbers that do not fit narrow number types in the binary readers With custom number types narrower than the values in a binary document, for example basic_json<..., std::int32_t, std::uint32_t, float>, every binary reader (CBOR, MessagePack, UBJSON, BJData, BSON, BON8) passed the decoded number to the SAX interface with an implicit conversion: the integer 5000000000 silently became 705032704, and a finite double such as 1e300 became infinity. The lexer handles the same values in JSON text: an integer that fits neither integer type is stored as number_float_t, and a finite number that overflows number_float_t is rejected with out_of_range.406. Pass every number read from binary input through three helpers that apply the lexer's rules: - emit_signed(): number_integer_t, else number_unsigned_t for a non-negative value, else number_float_t - emit_unsigned(): number_unsigned_t, else number_float_t - emit_float(): out_of_range.406 if a finite value overflows number_float_t; infinity and NaN are passed on For consistency, a CBOR negative integer below the range of number_integer_t is now stored as number_float_t, like a too small integer in JSON text, instead of being rejected with parse_error.112. With the default number types, this is the only change in behavior. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix MSVC and clang 3.5 in the narrow number type test MSVC types 3000000000 and 5000000000 as unsigned long, so json(-3000000000) triggered C4146 (unary minus on an unsigned type), which /WX turns into an error. Use LL literals, as elsewhere in the tests. clang 3.5 cannot convert the lambdas in the braced initializer of the format table to function pointers. Use named functions instead. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Check integer-to-float fallbacks for overflow in the binary readers emit_signed, emit_unsigned, and the CBOR negative integer fallback now pass their number_float_t fallback through emit_float, so a value that overflows number_float_t is rejected with out_of_range.406 like a floating-point value, instead of silently becoming infinity. This only matters for a number_float_t that cannot represent 2^64, such as a half-precision type. The CBOR value -1 - n is computed as long double so that emit_float sees a finite value. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use the input_format member instead of passing the format to binary_reader helpers The helpers (get_number, get_to, get_string, get_binary, get_bytes, emit_signed, emit_unsigned, emit_float, unexpect_eof, exception_message) are members of binary_reader, which already stores the format it was constructed with, so the parameter was redundant. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
351aeb7440 |
Merge remote-tracking branch 'origin/develop' into HEAD
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
d8d47be4a5 |
Fix CI on develop after merging the ready-to-merge PRs (#5754)
* Keep the serializer conversion for objects whose keys cannot be converted #5591 added a test converting nlohmann::json into a basic_json whose string type cannot be constructed from std::string. That instantiates convert_iteratively(), whose members.emplace_back(next.key(), ...) needs exactly that key conversion, and broke the build of unit-alt-string. Dispatch on the key's constructibility and leave such conversions to the serializers, as the levels above the nesting bound already do (#3425). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the remaining CI failures on develop - unit-wstring: with a 16-bit wchar_t (Windows), a lone surrogate is reported as the ill-formed byte 0xFF since #5704; the std::wstring expectations still had the previous <U+0000>. - ci_single_binaries: json_literals.hpp (#5610) and json.hpp include each other on purpose, and IWYU, not following the cycle, asks to replace json.hpp with json_fwd.hpp. Report its findings without failing the build, as already done for json.hpp. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the library warnings and noexcept specifications from the merged PRs - binary_reader: rename the error_handler constructor parameter, which shadowed the member (-Wshadow, -Wshadow-field-in-constructor; #5746) - basic_json(copy_construct_tag, ...): declare it noexcept when copying the base class is (GCC 16 -Wnoexcept; #5690) - the scalar-on-left legacy comparison operators: noexcept only when converting the scalar is, like their member counterparts (#5682, #5751) - compare_leaves: use std::is_eq/is_lt/is_gt instead of comparing a std::partial_ordering with 0 (-Wzero-as-null-pointer-constant; #5686) - serializer: silence MSVC C4127 for the EnsureAscii template parameter (#5741, #5746) - clang-tidy: return the sanitized reference in binary_writer, take the key of ordered_map::find_impl by const reference (#5727), and mark the switches over parse_array_index (#5728) - ordered_map: keep <memory> for std::allocator (IWYU) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Split unit-conversions.cpp so MinGW can link it clang 18 with the MinGW linker failed to link test-conversions_cpp17 ("relocation truncated to fit: IMAGE_REL_AMD64_REL32"). As windows.yml recommends, keep the objects small by splitting the test file. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the tests added by the merged PRs for all CI configurations - discard the results of dump() and from_*() in CHECK_THROWS with utils::ignore_return_value (GCC -Werror=unused-result) - give unit-bson's huge_string_t a default constructor (MSVC C2512, GCC 5, clang 3.5) - unit-disabled_exceptions: use the literals namespace when the global UDLs are off (ci_test_noglobaludls; #5700) - unit-binary_utf8_strict: expect the JSON pointer prefix with JSON_DIAGNOSTICS (#5741) - skip the tests that rely on exceptions under JSON_NOEXCEPTION (#5678, #5732) - clang-tidy and clang -Werror: static test data, CAPTURE(...);, const-correctness, use-after-move alias, unused conversion operator, a missing <iterator> include Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Title the macro examples and add JSON_STRICT_BINARY_UTF8 to the docset The documentation style check requires "Example: ..." titles on pages with several examples (#5741, #5591) and a docset entry for every macro page. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate BUILD.bazel and nlohmann_json.natvis Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5746 added detail/output/error_handler.hpp and #5741 the json_abi_sbu8 ABI tag. * Install libidn11 for the CMake 3.5.0 binary in ci_cmake_flags Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5733 moved ci_cmake_options from ubuntu:focal to ubuntu:24.04, which no longer ships libidn.so.11; the CMake 3.5.0 release binary links against it, so every ci_cmake_flags run has failed since. Install focal's libidn11 package for that matrix entry only. * Suppress Infer's false STACK_VARIABLE_ADDRESS_ESCAPE in get_impl get_impl() returns its local by value. A test added by the merged PRs instantiates it with a type Infer misreads, so ci_infer reported the 2021 code for the first time. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
7b0af0072e |
Merge remote-tracking branch 'origin/develop' into claude/fix-issue-3989-db7e45
Signed-off-by: Niels Lohmann <mail@nlohmann.me> # Conflicts: # include/nlohmann/detail/input/binary_reader.hpp # include/nlohmann/detail/string_utils.hpp # single_include/nlohmann/json.hpp |
||
|
Niels Lohmann
|
73e9eae3c1 |
Add an error_handler parameter for UTF-8 to the binary readers and writers (#5746)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
f56b418c56 |
Follow each binary format's UTF-8 rule: strict writers (CBOR/UBJSON/BJData/BSON), lenient readers (#5741)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
1143da4faa |
Fix CI: keep the binary readers' recovery code out of from_*()
The MinGW job ("clang (20.1.8)", Debug) failed to link test-msgpack_cpp17
with "relocation truncated to fit: IMAGE_REL_AMD64_REL32 against
`.rdata'". Its GNU ld (MinGW 12.2.0) cannot link an object file with
more than 32767 sections. Reproduced with clang 20.1.8 and binutils
2.40: unit-msgpack.cpp's object has 31958 sections on develop and had
33210 with #3989; binutils 2.44 links both.
The readers of from_*() never recover, but unoptimized builds still
emitted the recovery code their dead branches referenced: about 20
helpers per reader instantiation, plus the lexer's recovery functions
through the UBJSON high-precision number repair. Now:
- Errors are reported with report_error() only. The helpers that act on
the SAX parser's answer (repair_requested(), resync(), value_failed(),
close_open_containers(), and the repairs of high-precision numbers and
unsupported BSON elements) have one trivial variant for readers that do
not recover, so the recovery code is not referenced there.
- Object keys that are not strings and BSON elements whose end is lost
are marked at the error site and skipped by resync() in the reading
loops; report_repairable_error(), report_error_repairable_if(),
report_bson_element_error(), and bon8_error_repairable_if() are gone.
- is_cbor_item_head() is a free function, and the BSON element type is
formatted as on develop again.
The object now has 32638 sections and links. Behavior is unchanged
(regression tests, binary format suites, and fuzzing), and an optimized
program that only calls from_*() is now 256 bytes smaller than on
develop.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
Niels Lohmann
|
ab52c98f71 |
Merge remote-tracking branch 'origin/develop' into claude/fix-issue-3989-db7e45
Signed-off-by: Niels Lohmann <mail@nlohmann.me> # Conflicts: # docs/mkdocs/docs/api/json_sax/parse_error.md # docs/mkdocs/docs/features/parsing/index.md |
||
|
Niels Lohmann
|
4d46bce4e1 |
Fix CI on develop (#5750)
* Fix CI configuration broken by recent merges and tool updates - gcc_flags.cmake: drop -Wexperimental-fmv-target, which GCC 16 accepts only on aarch64; amd64 rejects it, so every GCC job failed while checking the compiler. - ci_get_cmake: add VERBATIM so the checksum pipeline is passed to the shell intact (the unescaped `$'` broke the generated Makefile and build.ninja, failing ci_cmake_flags and ci_module_cpp20); match the SHA-256 entry case-insensitively, as CMake 3.5.0 lists the archive as "Linux-x86_64"; and unpack with --strip-components, as that archive's top-level directory is spelled "Linux" too. - ci_single_binaries: compile json.hpp's TU without IWYU's --error, as the comment above the gate already intends. - tests: restore -Wno-deprecated-declarations for all non-MSVC compilers (#5737 kept it for GCC only), as several tests call deprecated functions on purpose; include thirdparty/fifo_map as SYSTEM. - .clang-tidy: set misc-use-internal-linkage.AnalyzeTypes to false; clang-tidy 22.1 extended the check to classes and enums and flagged 100 test helper types. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix library warnings and a JSON_DIAGNOSTICS parent bug - binary_reader: pass integers to sax->number_integer() through conditional_static_cast<number_integer_t>, making the existing narrowing for a narrow number_integer_t explicit (MSVC C4244 and GCC -Wconversion/-Warith-conversion with the int16_t test from #5694); mark two Infer DEAD_STORE false positives with @infer-ignore. - to_json: set the parents of an array built from a C++20 range view after all elements are in place; a reallocating push_back moved the earlier elements and left their parent pointers stale, failing the JSON_DIAGNOSTICS invariant assertion. - json.hpp: suppress MSVC C4127 for the new is_ordered_map check in diff(), like the three existing ones; spell out std::formatter::parse's return and iterator types for clang-tidy 22.1. - number_parse: make the Eisel-Lemire digit counter unsigned (GCC -Wstrict-overflow). - string_utils: take encode_utf8's callable by const reference (cppcoreguidelines-missing-std-forward) and drop a \u from its doc comment (-Wdocumentation-unknown-command). - ordered_map: include <memory> for std::allocator (cpplint). Ran make amalgamate. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix tests failing in CI on develop - unit-allocator: skip the #5640 test under MSVC STL iterator debugging, where containers allocate a debug proxy in noexcept move constructors and a failing allocation terminates; move a decrement out of an if condition (bugprone-inc-dec-in-conditions). - unit-conversions: expect the "(/0)" path with JSON_DIAGNOSTICS; compare strict enums via get<>() rather than through the noexcept operator==(ScalarType, json), which bugprone-exception-escape flags. - unit-alt-string: suppress -Wexit-time-destructors for the strict enum macro and misc-use-internal-linkage for its enum. - unit-bjdata: call the static lookup functions through the type and pass unsigned char (-Wsign-conversion on amd64). - Mark Infer false positives with @infer-ignore in unit-diagnostics, unit-pointer_access, unit-udt, and unit-conversions. - Smaller clang-tidy 22.1 findings in unit-class_parser, unit-constructor2, unit-custom-base-class, unit-locale-cpp, and unit-noexcept. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
6aec1a085f |
Report BON8 input that ends after a UTF-8 lead byte as truncated (#5677)
A lead byte (0xC2..0xF7) inside a string begins either another character (if a continuation byte follows) or an integer (otherwise). When the input ended right after the lead byte, the reader took the missing byte as "not a continuation byte", ended the string before the lead byte, and treated the lead byte as the start of the next value. With strict=false, a message cut off there was therefore read as a shorter value: the 11 bytes of "😀😀é" cut after 9 bytes gave "😀😀", and ["aé"] cut after 3 of its 5 bytes gave ["a"]. With strict=true, the input was rejected with a misleading message ("expected end of input"), or, for a key, with parse_error.112 instead of 110. Either reading of the lead byte leaves the message incomplete: a string at the end of a message must be terminated by 0xFF, so the lead byte cannot belong to a following message. Report parse_error.110 (unexpected end of input) for strings and keys, as the comment on get_bon8_string() already requires and as the reference decoder (HikoGUI) does. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
e5a89d671f |
Fix lint debt: enum-macro NOLINTs, doctest as SYSTEM, no-op analyzer (#5737)
* Drop stale LCOV_EXCL_LINE from the json_pointer out_of_range.410 throw The comment said the size_type overflow check in array_index() is only triggered on special platforms like 32-bit, and the throw was excluded from coverage. On 64-bit platforms the check is true for SIZE_MAX itself, and unit-json_pointer.cpp has asserted that case four times since #5395, so the line is executed in the coverage job. Reword the comment and remove the exclusion marker so the coverage report notices if the tests stop reaching it. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Name all three C-array check aliases in the enum-macro NOLINTs NLOHMANN_JSON_SERIALIZE_ENUM(_STRICT) suppressed the c-array warning under modernize-avoid-c-arrays only, but clang-tidy emits the same diagnostic under the aliases cppcoreguidelines-avoid-c-arrays and hicpp-avoid-c-arrays too. Any user running those checks got a false positive at every macro expansion, and our own tests needed a local NOLINT at each call site to work around it. Name all three aliases in the four macro comments instead, and drop the now-redundant c-array names from the five test call-site NOLINTs. Comment-only change; behavior, the public API, and the ABI do not change. Ran make amalgamate. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Include doctest as a SYSTEM directory instead of disabling warnings for all tests test_main added -Wno-deprecated and -Wno-float-equal as PUBLIC compile options for every non-MSVC compiler, so they were applied to every translation unit, library headers included, and silenced the CI warnings meant to check the library's own -Wfloat-equal pragmas. The only code that actually needed the suppression was the vendored doctest.h, which was included as a normal (non-SYSTEM) directory. Include thirdparty/doctest as SYSTEM for test_main, matching what tests/abi/CMakeLists.txt already does, and drop the two suppressions from both targets. Verified locally that unit-comparison, unit-conversions and unit-constructor1 compile clean with -Werror -Weverything and doctest as -isystem, and that CMake still configures with JSON_BuildTests=ON. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove the no-op ci_clang_analyze target ci_clang_analyze configured the build with the real compiler and only then wrapped ninja with scan-build. scan-build intercepts compiles by overriding CC/CXX, but build.ninja already had the compiler path baked in from the configure step, so every run bypassed the analyzer: CI logs show "No bugs found" after a normal build, never an analysis. The job also used Debian's frozen clang-tools-14 rather than the image's own clang, and CLANG_ANALYZER_CHECKS still named three valist.* checkers that current clang merged into security.VAList. ci_clang_tidy already runs every clang-analyzer-* check (via .clang-tidy's "Checks: '*'") with warnings as errors, so nothing is lost by removing the dead job. Delete ci_clang_analyze, CLANG_ANALYZER_CHECKS and the SCAN_BUILD_TOOL lookup from cmake/ci.cmake, drop it from the ubuntu.yml ci_static_analysis_clang matrix, and drop the now-unused clang-tools apt package (iwyu stays for ci_single_binaries). Reword quality_assurance.md and assurance_case.md, which described the dead job as a working control, to say the Clang Static Analyzer checks run through clang-tidy. Verified that `cmake -DJSON_CI=ON` still configures cleanly and that ci_clang_analyze no longer appears in the generated build or in any CMake/workflow file. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Re-enable portability-template-virtual-member-function; remove redundant forwards .clang-tidy disabled three checks "to get the CI going" (#4489, 2024-11-13): portability-template-virtual-member-function, bugprone-use-after-move and its alias hicpp-invalid-access-moved. portability-template-virtual-member-function only flagged output_stream_adapter::write_character/write_characters; annotate both with NOLINT and re-enable the check. bugprone-use-after-move flagged several double forwards that have no effect at runtime: - from_json.hpp calls std::forward<BasicJsonType>(j).at(Idx) inside pack expansions; at() has no ref-qualified overloads and always returns an lvalue reference, so the forward is a no-op. Replace with plain j.at(Idx) in all four places. - the move constructor forwards the whole object to its base class and then reads other's members. That is item 9 of #5724 (together with its cppcheck suppressions) and is left to that change. - input_adapters.hpp forwards the container twice on purpose, so the begin/end iterator types match adapter_type; annotate with NOLINT and a comment instead of changing behavior. The check still flags the move constructor (see above) and two sites in at(KeyType&&) (both overloads, json.hpp, in the throw's string_t(std::forward<KeyType>(key)) after find(std::forward<KeyType>(key))). Open PR #5689 rewrites that hunk, so bugprone-use-after-move (and hicpp-invalid-access-moved) stay disabled for now, with a comment explaining why; re-enable them once #5689 and the #5724 move-constructor change have landed. Also resolve the portability-avoid-pragma-once TODO: single_include never has #pragma once (amalgamate.py strips it) and every supported compiler accepts it in include/, so keep it disabled with an explanatory comment instead of a TODO. Fix the stale "json.hpp, around line 1265" comment in unit-class_parser.cpp, which now points at the move constructor's actual line. Behavior, the public API and the ABI do not change. Verified with clang-tidy 22.1.8 that portability-template-virtual-member-function now reports nothing, that bugprone-use-after-move/ hicpp-invalid-access-moved report only the known at(KeyType&&) and move-constructor sites, and that unit-custom-base-class, unit-constructor1, unit-conversions, unit-element_access2, unit-class_parser and unit-diagnostic-positions (JSON_DIAGNOSTIC_POSITIONS=1) compile under ASan/UBSan and pass with the same assertion counts as before. Ran make amalgamate. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix stale and malformed NOLINT comments json_sax.hpp named "-warnings-as-errors" in the NOLINT list on the two JSON_ASSERT(false) lines; that is the suffix clang-tidy appends to a diagnostic tag under WarningsAsErrors, not a check name, and every other JSON_ASSERT(false) omits it. unit-capacity.cpp carried 30 "// NOLINT(misc-const-correctness)" comments on "json j = ...;" declarations that are all used with non-const members afterwards, so the check has nothing to report there. unit-constructor2.cpp used a blanket "// NOLINT: access after move is OK here" on a use-after-move that hides every check on the line; naming bugprone-use-after-move and hicpp-invalid-access-moved keeps the intent once those checks are re-enabled (#5724). Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 10 * Remove stale .clang-tidy entries -google-runtime-references disabled a check that neither clang-tidy 22.1.8 nor 23.1.2 lists under --list-checks -checks='*'; it was removed upstream. The commented-out HeaderFilterRegex line has been unused since the active HeaderFilterRegex was introduced in #2561 (2021). Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 11 * Remove the GCC C++20 -Wignored-attributes pragma in json.hpp The pragma (added in #5164) claimed to work around the C++ modules redefinition errors of #5103, but #5103 is about hard errors (e.g. "redefinition of std::__is_constant_evaluated()", conflicting std::integral_constant) that ignoring a warning cannot suppress; they are traced to GCC PR 124430 and reproduce with <map> or <string> instead of json.hpp too. A GCC 16.2 -std=gnu++20 -fmodules build following #5103's repro steps still fails with the pragma in place, and a build of all test TUs with GCC_CXXFLAGS (which enable -Wignored-attributes) and the pragma removed produces no such warning. The block only hid a warning class from GCC C++20 users while suggesting #5103 was handled. Overlaps #5610, whose hunks touch the closing half of this pragma to insert the json_literals.hpp include. Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 9 * Fix stale doxygen comments hidden by the -Wdocumentation pragma macro_scope.hpp ignores -Wdocumentation and -Wdocumentation-unknown-command for the whole library, which also hides genuine documentation mistakes: - detail::unescape() documented "@return unescaped string" but returns void and unescapes its argument in place; reworded to "@param[in,out] s string to unescape in place" and dropped the bogus @return. - basic_json::get()'s copy-conversion overload wrote "converted to @tparam ValueType" inside @return, which Doxygen and Clang parse as a second, malformed @tparam; changed to "@a ValueType", matching the two other get() overloads a few lines above that already use it. This narrows the gap the -Wdocumentation pragma needs to cover; fully replacing the Doxygen-only commands it also hides (item 2c) is left for after #5267. Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 2 * Fix -Wextra-semi-stmt at its actual source, not assert() clang_flags.cmake blamed the global -Wno-extra-semi-stmt on assert(), but assert() expands to an expression under glibc and libc++ and does not trigger this warning. unit-assert_macro.cpp overrides JSON_ASSERT with "{if (!(x)) ++assert_counter; }", a bare block followed by a semicolon at every JSON_ASSERT(...) call site in the library; that was the actual source of 151 of the 208 -Wextra-semi-stmt sites found in a Clang 22 -Weverything sweep of the test suite with the flag removed. Switched to the standard do/while(false) macro idiom, which does not expand to a statement-plus-semicolon, and corrected the comment to name the remaining source instead: vendored Doctest's CAPTURE(x) shim, which already ends in a semicolon. Verified with clang++ -Wextra-semi-stmt (plus the file's other CI ignores) that unit-assert_macro.cpp now compiles without any -Wextra-semi-stmt diagnostic. Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 8 (step 1 of 2; step 2 covers the CAPTURE() call sites) * Drop the redundant semicolon from CAPTURE() call sites; remove -Wno-extra-semi-stmt doctest_compatibility.h defines CAPTURE(x) as DOCTEST_CAPTURE(x); (with a trailing semicolon baked into the macro), specifically so call sites do not need to add one themselves; most of the ~267 call sites already follow that convention. The remaining 64 call sites across 20 files wrote "CAPTURE(x);" anyway, turning into a statement plus an empty statement and triggering -Wextra-semi-stmt. Dropped the redundant semicolon at each of those sites. With item 6 having already made vendored Doctest a SYSTEM include, and this the last known source of -Wextra-semi-stmt findings, removed the flag from clang_flags.cmake entirely. Verified with clang++ -Wextra-semi-stmt (plus the file's other CI ignores) that all 20 touched files, plus a file with no CAPTURE() use (unit-json_pointer.cpp), compile without any -Wextra-semi-stmt diagnostic. Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 8 (step 2 of 2) * Switch ci_static_analysis_clang off the frozen LLVM 22 dev image ubuntu.yml pinned the clang-tidy/clang-tidy-sanitizer/single-binaries job to silkeh/clang:dev, a tag last pushed 2026-02-18 that reports "clang version 22.0.0 (...+20251015...)", a pre-release snapshot from before the LLVM 22 release; the maintainer now updates dev-unstable, 22, and latest instead. Switched to silkeh/clang:22, matching the other clang jobs on :latest. Verified with clang-tidy 22.1.8 (the image's actual version) against this repository's .clang-tidy and library headers what the release image newly reports compared to :dev: - readability-redundant-typename fires at ~250 sites across the _cpp20-relevant conversion/to_chars headers; the library targets C++11 and keeps the typenames, so the check is disabled in .clang-tidy, matching how the file already handles checks that don't fit a C++11 codebase. - misc-anonymous-namespace-in-header fires on the two anonymous namespaces in from_json.hpp and to_json.hpp; added the alias to their existing NOLINT (cert-dcl59-cpp, fuchsia-header-anon-namespaces, google-build-namespaces). - bugprone-std-namespace-modification fires on every addition to namespace std: the std::hash, std::formatter and std::swap overloads in json.hpp, and the std::tuple_size/std::tuple_element specializations in iteration_proxy.hpp (this last file is not named in #5725's item 5, found by actually running clang-tidy 22.1.8 against the current tree). All six are legal, deliberate additions to namespace std (explicit/partial specializations of std types, or the pre-C++20 std::swap overload); annotated each with the check name next to its existing cert-dcl58-cpp NOLINT. - modernize-avoid-c-style-cast reported nothing new. Also added clang++-22/21, clang-tidy-22/21, g++-16 and gcov-16 to the find_program search lists in ci.cmake so a local "maximal warnings" configure prefers the current toolchain version over an older one on PATH. #5725 item 5 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate cmake/gcc_flags.cmake for GCC 16.2.0 GCC_CXXFLAGS was generated for GCC 15.1.0, but ci_test_gcc and ci_test_gcc_cxx{11..26} now run in gcc:latest, currently GCC 16.2.0, so the "maximal warnings" job was missing warnings introduced since 15.1.0 while carrying entries GCC 16 treats as duplicates or no-ops. Regenerated with https://github.com/nlohmann/gcc_flags (patched locally to not crash on an option whose "-x c++ <opt> -" probe fails before it reads stdin, e.g. -Wabi=; the tool otherwise raises BrokenPipeError instead of recording the option as an error) run against g++ 16.2.0 in the official gcc:16 Docker image, keeping the documented -Wno-* exclusions and the same alphabetical placement scheme as before. Also added three GCC 16 warnings the generator cannot discover on its own because it only probes value ranges/lists it finds in the -Q option name itself, not in the enum choices --help=warnings documents separately: - -Wbidi-chars=any, -Wleading-whitespace=spaces: manually verified these compile cleanly with g++ 16.2.0. - -Wstrict-flex-arrays: deliberately NOT added, unlike the other two. Without -fstrict-flex-arrays (which the library does not enable, as it would change codegen for flexible array members), GCC prints "'-Wstrict-flex-arrays' is ignored when '-fstrict-flex-arrays' is not present" on every translation unit, and under our -Werror that note itself aborts the build. This differs from the harmless no-op warnings already kept in the file (-Whsa, -Wsynth, -Wunreachable-code, -Wunsafe-loop-optimizations), which emit nothing; #5725 item 7 named -Wstrict-flex-arrays as one of the flags GCC 16 adds, but did not anticipate this failure mode. Verified: compiled the library header and a representative set of test translation units (including ones touched by items 1, 3, 8, 9, 10 of this issue) with the regenerated GCC_CXXFLAGS plus -Werror under g++ 16.2.0 at -std=c++11 through -std=c++26, with zero warnings; ran the full local test suite (129/129 passing, unrelated to this compiler) as a regression check. CI must still confirm the actual ci_test_gcc / ci_test_standards_gcc targets end to end, since this was verified with direct g++ invocations rather than through the CMake/ CXXFLAGS environment-variable plumbing in ci.cmake. #5725 item 7 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Avoid std::basic_string<CharType> for non-character output_adapter CharType output_adapter<CharType, StringType> defaulted StringType to std::basic_string<CharType>, and (with JSON_NO_IO undefined) always declared a std::basic_ostream<CharType>&-taking constructor. For CharType with no non-deprecated std::char_traits specialization (only std::uint8_t is ever used this way, by the binary writers), simply naming either type - as an unused default template argument, or as an unused, never-called constructor's parameter type - instantiates std::char_traits<CharType> merely to name it, which some standard libraries mark deprecated: with the library-wide -Wdocumentation pragma (item 2's other half, left for a later commit) temporarily removed, an Apple clang 21 / libc++ TU calling json::to_cbor(j, vec) with std::vector<std::uint8_t>& got one -Wdeprecated-declarations warning per binary writer at the old output_adapters.hpp:193. Replaced the eager std::basic_string<CharType> / std::basic_ostream <CharType> defaults with a bool-tagged partial specialization (not std::conditional, which requires naming both branches' types up front regardless of which is selected, reproducing the same warning) that only ever names std::basic_string<CharType> / std::basic_ostream <CharType> when CharType is actually one of char, wchar_t, char16_t, char32_t, or (with __cpp_lib_char8_t) char8_t. For any other CharType, output_adapter's StringType and ostream-constructor parameter fall back to two distinct empty placeholder types, kept distinct so the two constructor overloads do not collide into a single redeclaration. Public API / behavior: passing a std::basic_string<std::uint8_t>& or std::basic_ostream<std::uint8_t>& directly to a binary writer's output_adapter now fails to compile instead of compiling with a deprecation warning; this was neither documented nor tested. All documented uses (std::vector<CharType>, std::basic_ostream<CharType> and StringType for character CharType) are unaffected. Verified with Apple clang 21 / libc++, with the two -Wdocumentation* "ignored" pragma lines in macro_scope.hpp temporarily removed and -std=c++11/c++20 plus the project's -Weverything flag set: calling to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson/to_bon8 on a std::vector<std::uint8_t> now produces no char_traits<unsigned char> (or any other) deprecation warning, while the char-based string- and ostream-adapter paths, and a to_cbor/from_cbor round trip, still compile and run correctly; also verified with GCC 16.2.0. Ran the full local test suite, including the binary-format unit tests (unit-cbor, unit-msgpack, unit-ubjson, unit-bjdata, unit-bson, unit-bon8, unit-binary_writer_sinks, unit-binary_formats, unit-custom-binary-type): 129/129 passing. #5725 item 2 (step a) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove the library-wide -Wdocumentation pragma; fix what it hid macro_scope.hpp / macro_unscope.hpp pushed and popped a Clang diagnostic region over the entire library that ignored -Wdocumentation and -Wdocumentation-unknown-command. Removed both pragmas and fixed every finding a full -Wdocumentation (which implies -Wdocumentation-unknown-command and -Wdocumentation-deprecated-sync) build reports, so the library now compiles clean under Clang's documentation checks without a blanket suppression. Overlaps #5267, which is still open and edits a nearby doc block (json.hpp's get()/get_impl() @return, already fixed in the item 2 step (b) commit of this branch); this commit does not touch that block again. Unknown Doxygen alias commands (Doxyfile removed in #3071, so these were never rendered by anything) rewritten as plain prose, keeping the same information: - @requirement REQ-JSON-01 / REQ-JSON-02 (iter_impl.hpp, json_reverse_iterator.hpp): now "This class satisfies the following concept requirements (REQ-JSON-0N):". - @liveexample{prose,example-id} (three sites in json.hpp): kept the prose, dropped the command wrapper and the trailing example-id (docs/mkdocs/docs/examples/*.cpp still exist and are used directly by the rendered docs, not through this in-header alias) and unescaped the "\," commas that were only needed for the old alias's comma-separated argument syntax. - @complexity X (json.hpp x4, json_pointer.hpp x2, serializer.hpp x1): now "Complexity: X". Backslash sequences Clang's comment lexer tried to parse as commands, escaped to render as literal backslashes: - lexer.hpp get_codepoint(): two `\u` occurrences. - binary_reader.hpp get_bson_cstr() / get_bson_cstr_bulk(): two `\x00` occurrences. - serializer.hpp: three `\uXXXX` occurrences (constructor @param, append_codepoint_to_string_buffer() @brief, and the ensure_ascii member comment). One finding remained after all of the above: Clang reports "declaration is marked with '@deprecated' command but does not have a deprecation attribute" on the deprecated sax_parse(span_input_adapter&&, ...) overload, even though JSON_HEDLEY_DEPRECATED_FOR does expand to __attribute__((deprecated(...))) for Clang. Several isolated reproductions of this exact declaration shape - doc comment, template<>, two stacked __attribute__ macros, an overload set sharing the name - did not reproduce the warning, so this looks like a Clang comment/declaration-association quirk specific to this overload inside the much larger basic_json class template, not an actual documentation defect. Rather than keep the pragma library-wide for one Clang false positive, added a tightly scoped -Wdocumentation-deprecated-sync push/pop around just that overload. Verified with Apple clang 21 and the project's actual -Weverything flag set (cmake/clang_flags.cmake) on the full header at -std=c++11 and -std=c++20: zero -Wdocumentation* diagnostics. Also compiled clean with GCC 16.2.0 (the pragmas are already __clang__-gated, so this only confirms no unrelated breakage). Ran make check-amalgamation and the full local test suite: 129/129 passing. #5725 item 2 (step c) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Take the JSON value by const reference in the array and tuple from_json paths Review feedback on #5737 (gregmarr): once the no-op std::forward calls are gone, the forwarding references have no purpose. from_json_fn passes the value as const BasicJsonType&, so these functions were only ever instantiated with a const lvalue anyway. The std::array, std::pair and std::tuple overloads of from_json and their helpers now take const BasicJsonType& and pass j on unchanged. Because the deduced BasicJsonType is now the plain type, tuple_type and the static_assert name const BasicJsonType& explicitly, so the reference checks are unchanged: get<std::tuple<const std::string&>>() still works, and get<std::tuple<std::string&>>() still fails the same static_assert. from_json_tuple_get_impl keeps its forwarding reference, since tuple_type calls it through std::declval. Behavior, the public API and the ABI do not change. unit-conversions, unit-constructor1, unit-udt, unit-udt_macro, unit-regression1/2/3, unit-deserialization, unit-noexcept, unit-items, unit-allocator, unit-custom-object-type, unit-ordered_json2 and unit-brace-init-copy-semantics pass at C++11, C++17 and C++20 with unchanged assertion counts. Ran make amalgamate. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |