The string fast path found the end of a run of plain characters 8 bytes at
a time (SWAR) and validated non-ASCII text one sequence at a time, calling
validate_one_utf8() out of line for each.
The vector code of json_view's scanner, in the new shared header
detail/input/simd_scan.hpp, now does both 16 bytes at a time: SSE2 finds
the end of a run on x86-64, and the UTF-8 check of Keiser and Lemire runs
with NEON on AArch64 and with SSSE3 on x86-64. SSSE3 is not part of x86-64,
so the check is compiled for it with a function attribute and used when the
CPU has it (cached in a constant-initialized atomic), else the scalar check;
code compiled with -mssse3 skips the CPU check. The first word of a run is
still checked with SWAR (the first four with NEON, where a vector mask
costs more), as most keys and short values end there, and the scalar
helpers are always inlined. On anything else, the scalar code is unchanged.
The new macro JSON_NO_SIMD selects the portable code everywhere, including
the SSE2/NEON digit formatting of the floating-point writer. The lexer tests
now also run with JSON_NO_SIMD and, on x86-64, with -mssse3, and a new test
places every byte that ends a run at each offset across several 16-byte
blocks, after ASCII and after CJK text.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Squashed onto develop cd6b428ef (CI fixes from #5860). Original commits:
- Let the lexer's bulk fast paths run on a window of the input
- Scan the get area of an input stream in bulk
- Lock a FILE once per parse instead of once per character
- Add benchmarks for parsing from a FILE* and for many values from a stream
- Avoid implicit unsigned char to char conversion in stream tests
- Make file_input_adapter::unlock() safe to call on a moved-from adapter
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- Discarded values: the legacy macro only changes <= and >=; != is true by default
- Out-of-range numbers are rejected while parsing, not stored as NaN
- merge_diff round trip: point to the ordered_json caveat
- ensure_ascii example shows the escaped output
- out_of_range.405 is thrown for "remove"/"move" on the root, not "add"
- CBOR stores binary subtypes as tags
- Integer/float comparison uses exact values, not conversion
- diff, flatten, and the binary writers are depth-bounded
- FAQ: max_digits10 for serialization, trailing commas are supported
- out_of_range.407 is still thrown by to_bon8
- FetchContent_MakeAvailable needs CMake 3.14
- json_fwd.hpp contents, operator< link, RFC 7396 for merge patch
- README: dump() wording, merge_diff, MSYS2/build2, patch comment typos,
operator[] link, embedded CMake example aligned with the docs
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Review and extend the documentation, and check it in CI
A review of all documentation pages found factual errors, dead links,
missing cross-references, and gaps in examples. This fixes them and adds
checks so the same problems are caught automatically.
Fixes:
- wrong signatures and version histories (operator!= C++20 member,
binary() subtype type, get<PointerType>(), JSON_NO_THREAD_LOCAL, ...)
- stale descriptions (number parsing since #5283, UBJSON table, SAX
example that no longer compiled, tsl::ordered_map advice)
- dead internal and external links; repology.org badges (the domain is
suspended) replaced by badges that query the registries directly
- deprecation notes link the migration guide; the guide itself fixed
Additions:
- "See also" sections, cross-references, 25 runnable examples, 12
Mermaid diagrams, new API pages for json_pointer::operator<=> and
byte_container_with_subtype::operator==/!=
- landing page, guides for untrusted input and performance
- "unreleased" badge after versions newer than the latest release
Checks:
- strict documentation build (broken links/anchors fail it); CI and
the publish workflow fetch the full history the build needs
- weekly external link check, Mermaid syntax check in CI
- check_structure.py: example titles, heading levels, alt texts,
header links, docset index coverage; its unused-example check works
again
- all examples produce the same output on every platform
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep the customer links that could not be fixed
A dead link on the customers page is still the evidence of where the
use of the library was documented. Keep the original URLs of the entries
without a working replacement (Marne, Cisco Webex Desk Camera, Philips
Hue, CyberArk) and exclude exactly these URLs from the link check.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Correct the duplicate-key recipe's claim about SAX positions
The SAX interface's key() receives no position either; only parse_error()
does. Also note that the recipe does not report the path to the repeated
key (see discussion #5085).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Say the library is available as a single header and mention json_fwd.hpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Correct documentation errors found while hunting for bugs
- patch/patch_inplace: list the JSON pointer errors parse_error.106-109
and out_of_range.402/404, and quote the actual parse_error.105 message.
- unflatten: list parse_error.106/107/108 and out_of_range.404.
- to_bson: list out_of_range.415 (binary subtype above 255) and note
that 412 and 415 are new in 3.13.0.
- to_string: state that string_t must be convertible to std::string, also
in the StringType requirements table.
- JSON Lines: a `while (input >> j)` loop also throws after the last value
for concatenated JSON values; show a loop that works for both.
- BON8: a string gets 0xFF only if nothing follows it in the message; a
string at the end of an array or object is ended by 0xFE.
- custom_string_type.hpp: add operator+=(char), which the "Always
required" list asks for (json_pointer::to_string, flatten, unflatten,
and diff did not compile), and an ADL int_to_string for diff and items.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Cache the release headers with functools.lru_cache
Codacy (Pylint) flagged the mutable default argument that header() used
as its cache. functools.lru_cache keeps the same memoization without it.
The script's output is unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: qualify the operator>> stream positioning guarantee
operator>>'s notes state that it leaves the stream positioned right
after the parsed value, so that concatenated JSON values can be read
back to back. That does not hold when the value is a number: a number
is only terminated by the character that follows it, and the lexer's
unget() is simulated (it rewinds only the lexer's own bookkeeping),
so that character stays consumed from the stream.
Document the actual behaviour: the guarantee holds for all value types
except numbers, which must be followed by whitespace. Also qualify the
cross-reference on the JSON Lines page, which repeated the unqualified
claim.
Documentation only; the behaviour itself is tracked in #5340.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* fix: restore the character that terminates a number (#5340)
operator>> is documented to leave the stream positioned right after the
parsed value, so that concatenated JSON values can be read back to back.
That did not hold for numbers: a number is only terminated by the
character following it, and lexer::scan_number() reads that character
and calls unget() -- which is simulated and rewinds only the lexer's own
bookkeeping. input_stream_adapter consumes via sbumpc() with no matching
sungetc(), so the terminating character stayed consumed and the next
extraction started one byte too late ('1true' left the stream at 'rue').
Propagating unget() to the adapter directly does not work: next_unget
makes the following get() replay the cached character, so the terminator
would be delivered twice. Instead, restore the still-pending character
once at the end of a non-strict parse, where the input is handed back to
the caller:
- input_stream_adapter gains unget_character() (sungetc()) and advertises
it via supports_unget, detected the same way as supports_seek.
- lexer::restore_pending_unget() turns a pending simulated unget of a
real (non-EOF) character into a real one and clears next_unget so the
character is not also replayed. It is a no-op for adapters that cannot
unget, and reports failure when sungetc() fails, in which case the
input is left as it was before.
- parser calls it on the three non-strict paths, i.e. for operator>> and
sax_parse(strict = false).
Strict parse()/accept() are unaffected: they require the input to end
after the value, so the character is consumed by the end-of-input check
anyway. Parse error messages and reported positions are unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* tests: fix CI failures in the #5340 test helpers
Four CI failures, all in the new test code:
- GCC (-Werror=useless-cast): drop the `json(...)` wrapper around
`json::parse(...)`, which already returns a `json`.
- GCC (-Werror=unused-result): assign the discarded `json::parse()`
result to a dummy, the idiom used elsewhere in the test suite, and
catch `json::parse_error&` for consistency.
- clang-tidy (google-default-arguments): remove the default argument
from the `pbackfail()` override; `sungetc()` supplies the base
declaration's default.
- MSVC (bad allocation): `no_putback_streambuf::underflow()` set a
one-character get area without advancing `m_pos`, so an implementation
whose `istream::get` peeks before it bumps re-read the same character
forever. Keep no get area at all: `underflow()` peeks, `uflow()`
consumes, and `sungetc()` still always lands in `pbackfail()`, which
is what the test needs.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* fix: leave the character that terminates a number in the input
Read the character following a number without consuming it, instead of
consuming it and putting it back. input_stream_adapter now peeks with
sgetc() and only steps over the character when the next one is requested
or when the adapter is destroyed, so releasing it cannot fail - no
putback position is required from the streambuf.
Suggested by gregmarr in #5344.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: match the version history wording to the peek-based fix
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: drop the whitespace-separator caveat from the parsing pages
The caveat added in #5343 describes the behavior this branch fixes: a
number no longer consumes the character that terminates it, so
concatenated values need no separator.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* refactor: split the strict and non-strict paths in parser
Folding the release_lookahead() call into the existing strict check left
the "in strict mode" comment on an else-if branch, and made the strict
condition in sax_parse() redundant with the branch it followed.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Put the stream position fix behind JSON_PRECISE_STREAM_POSITION
Leaving the character that terminates a number in the stream is observable:
reading "1,2,3" with repeated operator>> works today only because the comma
after each number is swallowed, and std::getline after a number skips the
line break. Both break with the fix, so make it opt-in for 3.x, as suggested
by @gregmarr in the review.
- JSON_PRECISE_STREAM_POSITION (default 0) selects the peek-based
input_stream_adapter. Without it, the adapter is the consuming one from
develop and has no supports_lookahead, so lexer::release_lookahead() and
the parser's calls to it compile to nothing.
- The macro changes input_stream_adapter's layout and member functions, so
it gets the ABI tag _psp, after _bics. The ABI config tests, the natvis
generator, and nlohmann_json.natvis (regenerated) know the tag.
- The tests for the fix move to unit-precise-stream-position.cpp, which
defines the macro itself and runs in every build, and gain the two cases
above. unit-deserialization.cpp pins the default behavior instead.
- The docs describe the default behavior again and point to the new macro
page; version history says "added in 3.13.0, planned default in 4.0.0".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: document RFC 8259 / JSONTestSuite compliance and parse() vs operator>> strictness
The compliance story lived only in tests/src/unit-testsuites.cpp, so
drive-by comparisons kept claiming the library "does not fully pass
JSONTestSuite". Make it discoverable:
- README: add a "Standards compliance" note stating that both nst
JSONTestSuite revisions run in CI, that all mandatory y_/n_ cases pass
through the strict parse() entry point, and listing the deliberate
implementation-defined i_ choices (unbounded nesting, silent BOM
stripping, noncharacters forwarded, strict rejection of invalid UTF-8
and lone surrogates, out_of_range.406 on numeric overflow).
- features/parsing: add a "Strictness and trailing data" section
documenting that parse() is strict and rejects trailing data while
operator>> follows relaxed iostream semantics (parses one value and
leaves the stream positioned after it) -- the single place a naive
test yields a "non-compliant" result.
Documentation only; no parser behavior change. Closes#5290.
Signed-off-by: manon <youdie006@users.noreply.github.com>
* docs: correct test-data vendoring and parse()/operator>> claims per review
- README: the JSONTestSuite data is downloaded from nlohmann/json_test_data at
configure time, not vendored/committed; say so.
- README: only the updated suite runs y_ and n_ cases through strict parse();
the original suite's y_ cases go through operator>>. Narrow the claim.
- parsing/index.md and operator_gtgt.md: note that operator>> consumes a number's
terminating byte, so concatenated numbers must be whitespace-separated (1 2
works, 1true does not); structural and literal values are unaffected.
Signed-off-by: manon <youdie006@users.noreply.github.com>
---------
Signed-off-by: manon <youdie006@users.noreply.github.com>
Co-authored-by: manon <youdie006@users.noreply.github.com>
operator>>'s notes state that it leaves the stream positioned right
after the parsed value, so that concatenated JSON values can be read
back to back. That does not hold when the value is a number: a number
is only terminated by the character that follows it, and the lexer's
unget() is simulated (it rewinds only the lexer's own bookkeeping),
so that character stays consumed from the stream.
Document the actual behaviour: the guarantee holds for all value types
except numbers, which must be followed by whitespace. Also qualify the
cross-reference on the JSON Lines page, which repeated the unqualified
claim.
Documentation only; the behaviour itself is tracked in #5340.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 📝 Fix documentation gaps found in a full GitHub Discussions review
Reviewed all 1008 GitHub Discussions (2020-2026) for recurring questions
that better or more visible documentation would have avoided. Adds/expands
documentation for ~26 distinct gaps, including:
- New "Debugging" page collecting natvis, GDB pretty printer, LLDB status,
and JSON_DIAGNOSTICS pointers (previously scattered/undiscoverable)
- Thread-safety and schema-validation FAQ entries
- StringType's char-based requirement (no wstring/u16string/u32string)
- Brace-initialization-yields-arrays warning directly on the constructor
reference page (previously only in the FAQ, missed by users reading
the constructor docs)
- std::any exclusion from get<T>(), with a manual-dispatch example
- Non-string-keyed std::map serializing as an array of pairs
- ordered_json compatibility with NLOHMANN_DEFINE_TYPE_* macros
(already worked, was undocumented)
- std::array truncation on size-mismatched conversion (no exception)
- static_cast vs. get<std::optional<T>>() divergence
- Recipe for omitting a std::optional field instead of emitting null
- No built-in nesting-depth limit during parsing + a callback-based
workaround recipe
- Recipe for streaming a large homogeneous array via parser callbacks
- operator>> stream-position semantics for concatenated JSON values
- JSON Pointer array-vs-object creation rule for non-existing paths
- CMake target name (nlohmann_json_modules) needed to link C++20 modules
- ESP-IDF/PlatformIO: no official package, link to a community fork
- get(key, default) as the Python dict.get() equivalent
- reserve() recipe for pre-allocating array capacity
- JSONC as an alias for the existing ignore_comments/ignore_trailing_commas
combination (distinct from the unsupported JSON5)
- items() dereferenced-element type: decltype() idiom + detail-namespace
stability caveat
- Various macro/type-conversion limitations (MSGPACK_DEFINE_ARRAY
equivalent, char-array round-tripping, ADL serializer macro gap)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 🎨 fix format
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 📝 Document a duplicate-object-key rejection recipe
RFC 8259 leaves handling of duplicate object keys to the implementation;
this library silently keeps only the last value for a repeated key.
Discussion #5085 asked for an opt-in rejection mode. Decision: don't
change library behavior, but document the existing parser-callback
workaround instead.
Adds a "Recipe: rejecting duplicate object keys" section to
parser_callbacks.md, adapted from a community-contributed workaround.
Fixed an off-by-one bug in the original snippet: object_start reports
the depth of the object's parent, while key events inside that object
report depth+1, so indexing the per-depth key set with the same depth
in both places caused an out-of-bounds access on nested objects.
Verified the published snippet compiles and behaves correctly for flat
duplicates, nested duplicates, sibling objects sharing key names, and
arrays of objects.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Cross-link the duplicate-key recipe with the existing object_t behavior docs
object_t.md and features/types/index.md already document that duplicate
object keys resolve to an unspecified value (RFC 8259 leaves this to the
implementation). The new recipe's intro overstated this as a guaranteed
"last value wins" rule, which isn't true in general -- parsing text keeps
the last value, but constructing from an initializer list keeps the first.
Reworded the recipe to point at object_t's "unspecified" behavior instead
of asserting a specific rule, and added cross-links from both existing
pages to the new recipe.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Turn the duplicate-key recipe into a standalone, compiled example
Replace the inline code fence in the "rejecting duplicate object keys"
recipe with a proper docs/mkdocs/docs/examples/*.cpp + .output pair,
included via --8<-- like every other example on the site. The .output
file was generated by running it through the project's actual example
build (docs/Makefile: single_include, -std=c++11, -DJSON_USE_GLOBAL_UDLS=0)
and cross-checked with `make check_output`, and the source passes the
pinned astyle 3.4.13 formatting unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add operator<< for json_pointer
* Deprecate json_pointer::operator string_t()
* Update documentation
* Move operator<<(basic_json) example
* Add example
* Add mkdocs-redirects
* Move operator<< and operator>> doc pages out of basic_json/
* Rename JSON pointer operator_string to operator_string_t
* Add unit test
Niels Lohmann