mirror of
https://github.com/nlohmann/json.git
synced 2026-11-11 16:40:54 +00:00
e0a12df00a6912b28e09f0437f240e67dacc1cfe
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Niels LohmannandClaude Sonnet 5.5
|
33f15727ce |
Parse JSON in one block of memory with a fused pointer-based parser (#6031)
* Parse JSON in one block of memory with a fused pointer-based parser The parser and the lexer exchange one token at a time, and the lexer keeps its state (the current character, a pending unget, where the token starts) in members, which every character pays for: json::accept() of twitter.json took 2.8x as long as json_document::accept(). detail::contiguous_parser reads text in one block of memory through a local pointer, with the grammar, the whitespace and the tokens in one loop (after yyjson's reader), and drives a SAX handler: parse()'s DOM handler, accept()'s handler, or the user's handler of sax_parse(). It is used with the default options (no callback, no comments, no trailing commas; sax_parse() with strict = true). Errors are reported by the library parser, so the exception, its message and its position stay the same. contiguous_parser stops at the token that the library parser would reject, before reporting it, so the events up to the error are the same; then: - parse() runs the library parser with a handler that only reports the error (json_sax_error_reporter), with the lexer of parse(), which is instantiated anyway; - sax_parse() runs it with a handler that forwards only parse_error() to the user's handler (json_sax_error_forwarder), so no event arrives twice; - accept() returns the verdict. A handler that returns false stops the parse as before. An error at the end of a large input costs parse() about 1.2x of what it cost before. On an Apple M1 Max: accept() 1.6-2.0x as fast, parse() 10-16% faster, ordered_json 11-16%, sax_parse() with a user's handler 1.36-1.46x, small messages 12-22% (parse()) and 2.1-2.4x (accept()). On x86-64 (GCC 15, Clang 21): accept() 1.6-1.9x, parse() 5-17%, ordered_json 12-19%, sax_parse() 1.32-1.57x. A translation unit that parses a string compiles 3% (-O0) and 9% (-O2) slower. unit-contiguous_parser.cpp compares the contiguous path with the byte path (std::list iterators) on corner cases, generated documents, their truncations and mutations, deep nesting, and narrow number types: the same values with the same number types and float bits, exception messages, accept() verdicts and sax_parse() events. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: unused parameter without exceptions, cpplint, BUILD.bazel - json_sax_error_reporter::parse_error: mark the exception unused, as JSON_THROW does not use it with JSON_NOEXCEPTION - unit-contiguous_parser.cpp: compare the exceptions of both paths only when exceptions are enabled (invalid input aborted the test otherwise) - silence cpplint's readability/braces false positives on the if statements inside the parser's macros - add contiguous_parser.hpp to BUILD.bazel Signed-off-by: Niels Lohmann <mail@nlohmann.me> * accept(): skip digits without computing their value accept() needs the value of a number only for a float whose exponent may reach the largest value. skip_digits() finds the end of the digits without the multiply-adds of accumulate_digits(), and once an 8-byte word fails the digit test, the remaining loop needs no bounds check: a non-digit is in that word. The value is computed afterwards for the rare floats that need it. accept() of canada.json: -16 % on Apple M1, -22 % with GCC 15 and -21 % with Clang 21 on x86-64; files with many numbers -9 to -15 %. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: mark parser callback in unit-contiguous_parser noexcept Fixes the -Werror=noexcept failure with GCC 26 (ci_test_standards_gcc (26)). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: sweep_allocator::construct() accepts pointers to const Recent libc++ (emsdk latest) probes construct(const T*, const T&) in an unevaluated context while checking for trivial copies; the static_cast to void* in the noexcept specification then fails to compile. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz (cherry picked from commit 794bf70823c12a2a1cb2a8a95f34dfb6357746a8) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: guard sweep_allocator::construct() against const types instead of const_cast The const_cast added in the previous commit is rejected by cppcoreguidelines-pro-type-const-cast (clang-tidy). Make construct() not viable for const U instead, which is what libc++ probes for. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz (cherry picked from commit de099fa607001b710862fd1bbecef3609c54cbc4) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: astyle and clang-tidy for sweep_allocator::construct() astyle 3.4.13 reformats the enable_if<!...> default template argument (Check amalgamation). Delete the const overload instead, which makes the libc++ probe for const objects fail in the SFINAE context just the same and needs no <type_traits>. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Contiguous parser: review fixes (wording, macros, diagnostic positions) - "default parser" instead of "library parser" in comments and tests - the three comments in json.hpp say the same thing - current() replaces the repeated `p != end ? *p : 0` - the backslashes of each macro are aligned - the docs say that JSON_DIAGNOSTIC_POSITIONS turns the fast path off Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
Niels Lohmann
|
6a852ab96d |
Split the json_view feature page into sub-pages (#6018)
* Split the json_view feature page into sub-pages
The single features/json_view.md page grew to ~600 lines covering eight
topics. Split it into features/json_view/ with an overview page and
pages for ownership and lifetime, differences from basic_json, reading
values, generic code, editing, and images.
The new differences page collects what was spread over several sections
("What is the same", "What is different", "Duplicate keys", and notes in
the conversion and serialization sections) under one summary table, and
adds the parts of the basic_json interface a view does not have. The
other pages now link to it instead of restating the rules.
Internal links across the documentation point to the new pages.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Docs: fix stale comments and trim repeated text in json_view pages
- fig-jsonview.js/.css: point the header comments at the new pages
- reading.md: views' strings are valid as long as the document (and a
borrowed source text), as lifetime.md says
- reading.md, generic_code.md: link to differences.md instead of
restating the member-order and repeated-key rules
- differences.md: drop the third mention of ordering comparisons, name
rbegin()/rend() in the iterator row
- exceptions.md: link editable documents to editing.md
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
Niels LohmannandClaude Sonnet 5.5
|
18138227dc |
Scan the buffer of a FILE in bulk with glibc and the Apple C library (#6030)
* Scan the buffer of a FILE in bulk with glibc and the Apple C library Parsing from a FILE* still read one character at a time through getc_unlocked(), while a std::istream has its get area scanned in bulk. file_input_adapter now offers the lexer the same kind of window into the buffer of the FILE: it reads the fields that the C library's own getc_unlocked() macro reads (glibc: _IO_read_ptr and _IO_read_end; Apple: _p and _r) and consumes exactly the characters getc_unlocked() would return, so the file is left where it was left before. The part of the current token in a buffer that is about to be refilled is kept for error messages, as for streams. The window needs the file locked for the whole parse, so JSON_DISABLE_UNLOCKED_FILE_IO and MemorySanitizer builds turn it off; uClibc, which defines __GLIBC__ but has other fields, does not get it. parse() from a FILE* gets 2.0-2.6x as fast on an Apple M1 Max and 1.7-2.2x on x86-64 (glibc, GCC 15 and Clang 21), accept() 2.9-4.4x; that is about as fast as parsing from a std::ifstream. The new tests read with FILE buffers of 1 to 70 bytes and unbuffered files, after ungetc(), with SAX positions and with binary strings across the buffer's edges. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: no setvbuf() buffer below 2 bytes on Windows The Microsoft C runtime rejects a _IOFBF buffer size below 2 as an invalid parameter and ends the process (0xc0000409), which made test-file_input fail on all MSVC and clang-cl jobs. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: astyle formatting in unit-file_input.cpp Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: limit the multi-character ungetc() test to glibc and Apple The Windows C runtime only supports one pushed-back character, which broke the Windows CI jobs. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: sweep_allocator::construct() accepts pointers to const Recent libc++ (emsdk latest) probes construct(const T*, const T&) in an unevaluated context while checking for trivial copies; the static_cast to void* in the noexcept specification then fails to compile. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz (cherry picked from commit 794bf70823c12a2a1cb2a8a95f34dfb6357746a8) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: guard sweep_allocator::construct() against const types instead of const_cast The const_cast added in the previous commit is rejected by cppcoreguidelines-pro-type-const-cast (clang-tidy). Make construct() not viable for const U instead, which is what libc++ probes for. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz (cherry picked from commit de099fa607001b710862fd1bbecef3609c54cbc4) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: astyle and clang-tidy for sweep_allocator::construct() astyle 3.4.13 reformats the enable_if<!...> default template argument (Check amalgamation). Delete the const overload instead, which makes the libc++ probe for const objects fail in the SFINAE context just the same and needs no <type_traits>. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
Niels Lohmann
|
a49242f088 | Docs: reading and writing files; patch_inplace() is not atomic (#6017) | ||
|
Niels Lohmann
|
a47170346a |
docs: new design for the documentation site (#5934)
* docs: new design for the documentation site Apply the brand to the Material for MkDocs site: Inter and JetBrains Mono served by the site (upright and real italic), brand colours, blue header band with the syntax pattern, wordmark in the header, new home page (hero, cards, animated feature tour), 404 page, polish for tabs, search, footer, focus rings, tables and code blocks, and theme features (code copy, previous/next footer, TOC follow, search suggestions). Draw the number railroad diagram, the parser callback events and the iterator illustrations as inline SVGs that follow the light and dark scheme. The generator scripts are in docs/mkdocs/scripts/figures; the feature slideshow (json-light.svg, json-dark.svg, replacing json.gif on the home page) is built by docs/mkdocs/scripts/slideshow. Decision nodes in five flowcharts become hexagons, which need far less room. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: add figures, table captions and a macro chooser - Figures with numbered captions: SAX stepper, json_view layout and edit stepper, exception map, JSON pointer and patch walkers, QA pipeline, scaling and binary size charts, roadmap lanes, release timeline, integration decision tree and namespace builder - Table captions (nj-table-caption) on 9 pages - Compile times: modules and PCH charts, filesystem section, re-measured Figure 1, links between features/modules.md and compile_times.md - Interactive macro chooser for features/arbitrary_types.md - Link check: exclude URLs that answer the plugin with 400/401/419 or are dead (customers.md) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: macro chooser generates adl_serializer for non-default-constructible types Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: cross-link concept pages and API pages in See also sections Add See also links in both directions between the concept pages (features, home, integration) and the API pages, write API links in code font, give every entry a comment after a dash, and show the callback event figure on the parser callbacks page. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: address review comments on the new design - Home: add a "Latest release" button - Sponsors: update the list and group it by tier - Features: list all feature pages on the index page - Releases: add highlights per release, link the 4.0 roadmap, add the first commit to the timeline - Exceptions: prefix all example messages with the exception name - API: no code font for format names in the See also links of the to_/from_ pages - Merge Patch: show the document before, the patch, and the result - Replace "--" with typographic dashes, mark external links with an icon, center Mermaid diagrams Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
ffd4242dc2 |
Docs: fill documentation gaps found in user questions (#5907)
* Docs: fill documentation gaps found in user questions Add pages on class-hierarchy/smart-pointer/variant/chrono recipes, working without exceptions, and large, streamed and partial input. Extend the FAQ (23 entries, Known bugs first), parse error documentation (empty-input checklist, line/column, diagnostic positions), element access, conversions, creating values, ADL troubleshooting, memory, and build/distribution notes. Add 14 examples with generated output. Correct the statement that parse errors cannot be recovered from with -fno-exceptions. The topics come from recurring questions on Stack Overflow, Reddit, Hacker News, YouTube, and the project's GitHub discussions and issues. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: declare range-for lists in variables to keep astyle quiet astyle reformats a braced init list placed directly in a range-for header; declare the list first so check_amalgamation passes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
cc917d7123 |
docs: cross-reference json_view and json_document for 3.13.0 (#5892)
* docs: cross-reference json_view and json_document for 3.13.0 Document the new zero-copy document/view API across the docs: add a "Documents and views" section to the basic_json API hub linking to all specializations, point parsing/conversion/modifying pages to the view-based alternatives, add document/view sections to feature pages, update the landing page to mention json_view.hpp/json_literals.hpp, and cross-link debugging/exceptions. Also document wide-string inputs (and ill-formed UTF-16/32 rejection), add parse/accept examples for wide strings, fix related doc gaps, and align features/parsing and FAQ with the added support. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: fix article before std::istream in FAQ Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: clarify document inputs, decoding, get<T>() and editing Address review comments: say which inputs are borrowed and that a wide string pointer is an extra input, which values are decoded during parsing, that get<T>() reads only built-in types directly, all forms of set() and that the document's insert() is array-only, and how to store wide strings. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: integers are converted while parsing a document Also say "They accept every input" in basic_json_document::parse. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
e9970066c5 |
Scan strings with SSE2, SSSE3, or NEON (#5853)
* Scan strings with SSE2, SSSE3, or NEON The string fast path found the end of a run of plain characters 8 bytes at a time (SWAR) and validated non-ASCII text one sequence at a time, calling validate_one_utf8() out of line for each. The vector code of json_view's scanner, in the new shared header detail/input/simd_scan.hpp, now does both 16 bytes at a time: SSE2 finds the end of a run on x86-64, and the UTF-8 check of Keiser and Lemire runs with NEON on AArch64 and with SSSE3 on x86-64. SSSE3 is not part of x86-64, so the check is compiled for it with a function attribute and used when the CPU has it (cached in a constant-initialized atomic), else the scalar check; code compiled with -mssse3 skips the CPU check. The first word of a run is still checked with SWAR (the first four with NEON, where a vector mask costs more), as most keys and short values end there, and the scalar helpers are always inlined. On anything else, the scalar code is unchanged. The new macro JSON_NO_SIMD selects the portable code everywhere, including the SSE2/NEON digit formatting of the floating-point writer. The lexer tests now also run with JSON_NO_SIMD and, on x86-64, with -mssse3, and a new test places every byte that ends a run at each offset across several 16-byte blocks, after ASCII and after CJK text. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep the string scan from slowing down short strings and dump() on Arm Three benchmarks were 7-9% slower than develop on Apple M1 (Apple Clang, -O3 -flto): Accept/jeopardy, Dump/citm_catalog, and Dump/jeopardy. - dump(): the serializer passes a string without its closing quote, so plain_run_end() ran out of input and called the out-of-line NEON loop for every string, with fewer than 16 bytes left. The loop is now only called for 16 bytes or more. - accept(): no string of jeopardy.json reaches the vector UTF-8 check, yet compiling it into the lexer's scan function made it 5% slower: the function runs once per token and now saved more registers and kept a stack slot. lexer_string_bulk_run() checks the first four words inline and continues out of line, and the serializer keeps string_bulk_run() inline, which is faster there. - The byte that ends a run is read from the scanned buffer instead of through get(), which Apple Clang merged with the other get() calls that load the read position again right after it was stored (3%). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Define string_bulk_run_from() only where lexer_string_bulk_run() calls it With NEON and simdutf in C++17, lexer_string_bulk_run() takes the string_bulk_run() path, so the out-of-line helper was unused. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not dispatch the SSSE3 string scan with GCC's C++20 modules GCC does not keep the target("ssse3") attribute on functions it reads back from a module interface, so the module build (ci_module_cpp20, gcc:latest) rejected the intrinsics in vector_utf8_run() with "needs isa option -mssse3". With __cpp_modules, GCC now uses the portable UTF-8 check like other compilers without the target attribute; the SSE2 scan stays, and -mssse3 or more still compiles the SSSE3 check directly. Clang is not affected. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Include <cpuid.h> only once when json.hpp and json_view.hpp are both used The SSSE3 dispatch of the string scan and the one of json_view each include <cpuid.h>. Before GCC 11 the header has no include guard, so a translation unit that includes both headers fails with GCC 10: "redefinition of 'unsigned int __get_cpuid_max(unsigned int, unsigned int*)'". Guard both includes with NLOHMANN_JSON_CPUID_INCLUDED. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Add an ABI tag for JSON_NO_SIMD JSON_NO_SIMD selects between two definitions of the internal scanning and formatting functions. Like JSON_VIEW_NO_SIMD (_vns), it now appends _ns to the inline namespace, so translation units built with and without it can be linked into one program without ODR violations. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
4da932f518 |
Make parsing from std::istream and FILE* fast (#5828)
* Make parsing from std::istream and FILE* fast
Squashed onto develop
|
||
|
Niels Lohmann
|
63c10a51fc |
Review and extend the documentation, and check it in CI (#5638)
* Review and extend the documentation, and check it in CI A review of all documentation pages found factual errors, dead links, missing cross-references, and gaps in examples. This fixes them and adds checks so the same problems are caught automatically. Fixes: - wrong signatures and version histories (operator!= C++20 member, binary() subtype type, get<PointerType>(), JSON_NO_THREAD_LOCAL, ...) - stale descriptions (number parsing since #5283, UBJSON table, SAX example that no longer compiled, tsl::ordered_map advice) - dead internal and external links; repology.org badges (the domain is suspended) replaced by badges that query the registries directly - deprecation notes link the migration guide; the guide itself fixed Additions: - "See also" sections, cross-references, 25 runnable examples, 12 Mermaid diagrams, new API pages for json_pointer::operator<=> and byte_container_with_subtype::operator==/!= - landing page, guides for untrusted input and performance - "unreleased" badge after versions newer than the latest release Checks: - strict documentation build (broken links/anchors fail it); CI and the publish workflow fetch the full history the build needs - weekly external link check, Mermaid syntax check in CI - check_structure.py: example titles, heading levels, alt texts, header links, docset index coverage; its unused-example check works again - all examples produce the same output on every platform Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep the customer links that could not be fixed A dead link on the customers page is still the evidence of where the use of the library was documented. Keep the original URLs of the entries without a working replacement (Marne, Cisco Webex Desk Camera, Philips Hue, CyberArk) and exclude exactly these URLs from the link check. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Correct the duplicate-key recipe's claim about SAX positions The SAX interface's key() receives no position either; only parse_error() does. Also note that the recipe does not report the path to the repeated key (see discussion #5085). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Say the library is available as a single header and mention json_fwd.hpp Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Correct documentation errors found while hunting for bugs - patch/patch_inplace: list the JSON pointer errors parse_error.106-109 and out_of_range.402/404, and quote the actual parse_error.105 message. - unflatten: list parse_error.106/107/108 and out_of_range.404. - to_bson: list out_of_range.415 (binary subtype above 255) and note that 412 and 415 are new in 3.13.0. - to_string: state that string_t must be convertible to std::string, also in the StringType requirements table. - JSON Lines: a `while (input >> j)` loop also throws after the last value for concatenated JSON values; show a loop that works for both. - BON8: a string gets 0xFF only if nothing follows it in the message; a string at the end of an array or object is ended by 0xFE. - custom_string_type.hpp: add operator+=(char), which the "Always required" list asks for (json_pointer::to_string, flatten, unflatten, and diff did not compile), and an ADL int_to_string for diff and items. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Cache the release headers with functools.lru_cache Codacy (Pylint) flagged the mutable default argument that header() used as its cache. functools.lru_cache keeps the same memoization without it. The script's output is unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
abbe52d6de |
Add JSON_PRECISE_STREAM_POSITION to leave the character that terminates a number in the stream (#5344)
* docs: qualify the operator>> stream positioning guarantee operator>>'s notes state that it leaves the stream positioned right after the parsed value, so that concatenated JSON values can be read back to back. That does not hold when the value is a number: a number is only terminated by the character that follows it, and the lexer's unget() is simulated (it rewinds only the lexer's own bookkeeping), so that character stays consumed from the stream. Document the actual behaviour: the guarantee holds for all value types except numbers, which must be followed by whitespace. Also qualify the cross-reference on the JSON Lines page, which repeated the unqualified claim. Documentation only; the behaviour itself is tracked in #5340. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * fix: restore the character that terminates a number (#5340) operator>> is documented to leave the stream positioned right after the parsed value, so that concatenated JSON values can be read back to back. That did not hold for numbers: a number is only terminated by the character following it, and lexer::scan_number() reads that character and calls unget() -- which is simulated and rewinds only the lexer's own bookkeeping. input_stream_adapter consumes via sbumpc() with no matching sungetc(), so the terminating character stayed consumed and the next extraction started one byte too late ('1true' left the stream at 'rue'). Propagating unget() to the adapter directly does not work: next_unget makes the following get() replay the cached character, so the terminator would be delivered twice. Instead, restore the still-pending character once at the end of a non-strict parse, where the input is handed back to the caller: - input_stream_adapter gains unget_character() (sungetc()) and advertises it via supports_unget, detected the same way as supports_seek. - lexer::restore_pending_unget() turns a pending simulated unget of a real (non-EOF) character into a real one and clears next_unget so the character is not also replayed. It is a no-op for adapters that cannot unget, and reports failure when sungetc() fails, in which case the input is left as it was before. - parser calls it on the three non-strict paths, i.e. for operator>> and sax_parse(strict = false). Strict parse()/accept() are unaffected: they require the input to end after the value, so the character is consumed by the end-of-input check anyway. Parse error messages and reported positions are unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * tests: fix CI failures in the #5340 test helpers Four CI failures, all in the new test code: - GCC (-Werror=useless-cast): drop the `json(...)` wrapper around `json::parse(...)`, which already returns a `json`. - GCC (-Werror=unused-result): assign the discarded `json::parse()` result to a dummy, the idiom used elsewhere in the test suite, and catch `json::parse_error&` for consistency. - clang-tidy (google-default-arguments): remove the default argument from the `pbackfail()` override; `sungetc()` supplies the base declaration's default. - MSVC (bad allocation): `no_putback_streambuf::underflow()` set a one-character get area without advancing `m_pos`, so an implementation whose `istream::get` peeks before it bumps re-read the same character forever. Keep no get area at all: `underflow()` peeks, `uflow()` consumes, and `sungetc()` still always lands in `pbackfail()`, which is what the test needs. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * fix: leave the character that terminates a number in the input Read the character following a number without consuming it, instead of consuming it and putting it back. input_stream_adapter now peeks with sgetc() and only steps over the character when the next one is requested or when the adapter is destroyed, so releasing it cannot fail - no putback position is required from the streambuf. Suggested by gregmarr in #5344. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: match the version history wording to the peek-based fix Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: drop the whitespace-separator caveat from the parsing pages The caveat added in #5343 describes the behavior this branch fixes: a number no longer consumes the character that terminates it, so concatenated values need no separator. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * refactor: split the strict and non-strict paths in parser Folding the release_lookahead() call into the existing strict check left the "in strict mode" comment on an else-if branch, and made the strict condition in sax_parse() redundant with the branch it followed. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Put the stream position fix behind JSON_PRECISE_STREAM_POSITION Leaving the character that terminates a number in the stream is observable: reading "1,2,3" with repeated operator>> works today only because the comma after each number is swallowed, and std::getline after a number skips the line break. Both break with the fix, so make it opt-in for 3.x, as suggested by @gregmarr in the review. - JSON_PRECISE_STREAM_POSITION (default 0) selects the peek-based input_stream_adapter. Without it, the adapter is the consuming one from develop and has no supports_lookahead, so lexer::release_lookahead() and the parser's calls to it compile to nothing. - The macro changes input_stream_adapter's layout and member functions, so it gets the ABI tag _psp, after _bics. The ABI config tests, the natvis generator, and nlohmann_json.natvis (regenerated) know the tag. - The tests for the fix move to unit-precise-stream-position.cpp, which defines the macro itself and runs in every build, and gain the two cases above. unit-deserialization.cpp pins the default behavior instead. - The docs describe the default behavior again and point to the new macro page; version history says "added in 3.13.0, planned default in 4.0.0". Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
KBSandmanon
|
78821cd9c2 |
docs: document standards compliance and parse() vs operator>> strictness (#5326)
* docs: document RFC 8259 / JSONTestSuite compliance and parse() vs operator>> strictness The compliance story lived only in tests/src/unit-testsuites.cpp, so drive-by comparisons kept claiming the library "does not fully pass JSONTestSuite". Make it discoverable: - README: add a "Standards compliance" note stating that both nst JSONTestSuite revisions run in CI, that all mandatory y_/n_ cases pass through the strict parse() entry point, and listing the deliberate implementation-defined i_ choices (unbounded nesting, silent BOM stripping, noncharacters forwarded, strict rejection of invalid UTF-8 and lone surrogates, out_of_range.406 on numeric overflow). - features/parsing: add a "Strictness and trailing data" section documenting that parse() is strict and rejects trailing data while operator>> follows relaxed iostream semantics (parses one value and leaves the stream positioned after it) -- the single place a naive test yields a "non-compliant" result. Documentation only; no parser behavior change. Closes #5290. Signed-off-by: manon <youdie006@users.noreply.github.com> * docs: correct test-data vendoring and parse()/operator>> claims per review - README: the JSONTestSuite data is downloaded from nlohmann/json_test_data at configure time, not vendored/committed; say so. - README: only the updated suite runs y_ and n_ cases through strict parse(); the original suite's y_ cases go through operator>>. Narrow the claim. - parsing/index.md and operator_gtgt.md: note that operator>> consumes a number's terminating byte, so concatenated numbers must be whitespace-separated (1 2 works, 1true does not); structural and literal values are unaffected. Signed-off-by: manon <youdie006@users.noreply.github.com> --------- Signed-off-by: manon <youdie006@users.noreply.github.com> Co-authored-by: manon <youdie006@users.noreply.github.com> |
||
|
Niels Lohmann
|
c034480c22 |
📝 add more docs (#5231)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
b21c345179 |
Reorganize directories (#3462)
* 🚚 move files * 🚚 rename doc folder to docs * 🚚 rename test folder to tests |