The key hash is not seeded, so keys chosen to collide made building the table quadratic (20,000 colliding keys took 470 ms to parse). A key may now sit at most 64 slots from its home slot; if a key would sit further away, the table is dropped and the object is searched linearly. Lookups stop after the same distance.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
GCC ignores the target attribute in modules, so the SSSE3 dispatch is
disabled for the module interface (the check stays portable, SSE2 is kept).
Make the 8-vs-16 byte unrolling condition in scan_string_run a
preprocessor/template split to avoid a constant condition (C4127).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Speed up json_view's parser with SIMD scanning and a hash table
for large objects.
Long runs of string bytes are scanned 16 bytes at a time with NEON
(AArch64, GCC and Clang) and SSE2 (x86-64), both baseline
instruction sets. Keys keep 16 table checks before the vector
loop, because their lengths repeat from record to record; string
values get 8, because their lengths vary more. Non-ASCII text is
validated 16 bytes at a time with simdjson's "lookup4" check
(Keiser and Lemire, 2021), with NEON on AArch64 and, on x86-64,
with SSSE3. SSSE3 is not part of baseline x86-64, so the check is
compiled for SSSE3 with a function attribute and used only where
CPUID reports it, which all x86-64 CPUs since about 2011 do; the
answer is cached in a statically initialized atomic, so there is
no guard of a local static and no global constructor. The same
input is accepted either way. JSON_VIEW_NO_SIMD selects the
portable code.
On x86-64, string runs are now checked vector-first: one SSE2
compare from the first byte finds the end of most keys and short
values, instead of a branch per byte for the first 8-16 bytes.
AArch64 keeps the byte-wise steps, where a NEON mask costs more and
the branches predict well. Entering an object or array no longer
stalls: open() stores the parent's frame field by field instead of
building it on the stack and reading it back with wider loads,
which waited for the narrower stores to retire.
Objects with 128 members or more get an open-addressing hash table
built when the object closes, so operator[], at(), find(),
contains(), count(), value(), and JSON pointers take constant time
on average in such objects; of duplicate keys, the first is kept,
as for the linear search. The idea comes from Boost.JSON.
simdjson is credited in simd.hpp's SPDX block, the README, and
license.md.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
finish() copied the whole output when the buffer was more than twice as
large as the result (citm dump +11%). The tighter source_extent()
estimate already keeps the buffer of a small value small.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The source extent of a value was read from the next node, falling back to the rest of the document when that node held a decoded string. dump() of a small value could thus allocate a buffer as large as the document. Skip a few such nodes, cap the fallback estimate, and shrink a buffer that is much larger than its output.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Add basic_json_view::dump() and the comparison operators, and read
floats from the parser's digit layout instead of rescanning the
token.
dump(indent, indent_char, ensure_ascii, number_format) writes a
value the way ordered_json::parse(text).dump() writes it for the
same arguments: members in document order, all of them should a
key occur more than once; strings escaped by the same rules, using
the library's scanning kernels; floats written with the library's
to_chars conversion, so the output equals basic_json's byte for
byte; integers copied from the source, where they are already
canonical, except -0, which parse() reads as 0. There is no
error_handler argument, because the view only holds valid UTF-8.
number_format::source copies numbers exactly as they appear in the
source (e.g. "1.50", "1E2", "-0"), which basic_json cannot provide.
operator<< takes the indentation from the stream width, as for
basic_json. The writer walks iteratively, so nesting depth is
limited by memory only.
operator== and operator!= compare two views, or a view and a
basic_json value in either order, by the rules basic_json's
operator== uses: numbers compare by value across their types,
objects compare by their members with duplicate keys resolved as
parse() resolves them, member order matters only where the object
type keeps one, and discarded views compare as discarded basic_json
values do, including under JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON.
Nothing is materialized except single scalars.
While parsing, the view now records where the integer digits, the
fraction digits, and the exponent of a float token are, so floats
and doubles with at most 19 digits are read from that layout with
the library's decimal_to_float() instead of rescanning the token.
Both round correctly, so the values are those of parse(). get<double>(),
materialize(), dump(), and the comparisons all use it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Develop's #5800 made basic_json report a JSON pointer array index such as
"/a" as parse_error.109 instead of out_of_range.404, as longer ones were;
the view's array_index() follows, with "-" and the empty token still
unresolved.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The same changes as on the dump branch, where they were first made, so
that this branch passes clang-tidy on its own.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
__forceinline makes MSVC report C4714 for function templates it cannot
inline (get<std::string>), an error under /WX. The forced inlining is
only a performance hint, so use plain inline there.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Looking up the last member of a duplicate key cannot stop at a match, so
every lookup scanned the whole object (1.6 to 3.4 times slower for small
objects). operator[](key), at, find, value, contains, count, and JSON
pointer resolution return the first member again, as yyjson and simdjson
do; materialize() and get<map>() keep the last value, as parse().
The documentation says so in the feature page and on each lookup page,
and explains how to get the value parse() would give. The integer index
templates and the discarded chaining of operator[] stay.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Lookups (operator[], at, find, contains, count, value, JSON pointers)
return the last member of a duplicate key, as materialize() and
parse() keep it.
* operator[] on a discarded view returns a discarded view instead of
throwing, so v["a"]["b"] is safe for a missing "a".
* operator[] and at() take any integer type (not only int and size_t),
fixing ambiguous calls with unsigned, long, std::int64_t, ...
* Fix the operator[] documentation, which claimed a discarded view for
a type mismatch where type_error.305 is thrown.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Give basic_json_view the read-only access functions of basic_json:
operator[] and at() with keys and indices, front()/back(), find(),
contains(), count(), begin()/end() and cbegin()/cend(), items()
with structured bindings from C++17 on, and type_name().
Exceptions have the ids and messages of the const functions of
basic_json. Where basic_json has undefined behavior the view
answers safely: operator[] with a missing key or an out-of-range
index returns a discarded view, and front()/back() of an empty
container throw invalid_iterator.214. Objects are iterated in
document order, and all members are visited; duplicate-key lookups
find the first member (as yyjson and simdjson do), while parse(),
materialize(), and the map conversions keep the last value, as
parse() does. Keys of up to 16 bytes are compared with two
overlapping loads.
Add value conversions: get<T>()/get_to() for arithmetic types,
bool, nullptr_t, strings (std::basic_string copied,
string_view_t without a copy), BasicJsonType, views, std::vector,
and maps with string keys; get_string() for the string without a
copy; number_token() for the number exactly as written in the
source; value() with keys and JSON pointers; and operator[]/at()/
contains() with JSON pointers. Everything else, including types
with from_json(), goes through materialize() of that subtree.
get<T>() of arithmetic types is inlined down to the conversion, so
reading an integer needs no call.
detail::json_pointer_access exposes a pointer's reference tokens
to code outside basic_json.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- inline_nodes is null when the document has no inline nodes, instead of
pointing past the end of the allocation; shrink_to_fit() no longer
passes it to memcpy() for an empty node array
- is_byte_char and is_wide_char use conjunction/disjunction, which
astyle leaves unspaced
- append_stream() uses (std::max) for the count read
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Changes promised in the replies to the review of #5620:
- read() reuses the owned copy: a copied input (an rvalue byte container,
a stream, FILE*, wide characters, an iterator range) is appended to the
cleared owned buffer instead of a fresh std::string being move-assigned
over it. A moved-in std::string still replaces it.
- The error replay runs the library parser with a SAX handler that stores
nothing and throws the reported exception, instead of
BasicJsonType::parse(), which built a DOM up to the error.
- Pointers to and arrays of wide characters, and FILE*, take the input
adapter path (converted to UTF-8 into an owned buffer) instead of
failing a static_assert. Wide pointers are NUL-terminated.
- Streams are read in blocks through rdbuf() into the buffer, instead of
character by character through the input adapter; eofbit is set at the
end, as by the library's stream adapter.
- tape/tape_size/tape_cap are now nodes/node_count/node_capacity.
- The builder's members get descriptive, m_-prefixed names.
- The shallow frame stack is a std::array (in a wrapper with a
user-provided constructor: -Weffc++ wants the member initialized, and
value-initializing the 64 frames costs up to 30 % on small documents).
- The value dispatch macro has no NEXT parameter; the switch is left with
break, and the macro is undefined after its last use.
- literal() and digits_exceed() deduce the length from the string literal;
each literal has one helper; the two `if (!is_float)` blocks are merged.
- arena_ensure() sits right before the writes; the decode arena is a
std::vector<char>.
- The builder records the deepest nesting, and materialize() sizes its
stack from it once (growing it only if a subtree is deeper).
- Comment fixes (hex4, slow_string, the NUL rule, put_utf8, 0 - m, the
arena resize, grow()) and formatting (macro backslashes, digit cases,
one power of ten per line, value_t assertion next to is_container()).
- build_with's template parameters follow the builder's order.
Greg's idea of keeping the source offset in `off` and the arena offset in
`next` for escaped strings was measured and not adopted: string access got
7-18 % slower with a select, and 3.5-18 % slower with a branch-free field
pick (Apple M-series, clang -O3), parsing was unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
emit() moves the node array when it grows, and the subtraction read base
in the same expression, so the order was unspecified. MSVC Release builds
without forced inlining read the old base; the container index then
pointed outside the array and close() wrote out of bounds.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The check rejects inputs of 0xFFFFFFF0 bytes or more, but the exception
message and the documentation said 4 GiB. Name the limit once
(max_input_size), and state 4 GiB minus 16 bytes in the message and the
documentation. Test the limit with a container that only claims the size.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
reserve(n) and the growth of the index computed n * sizeof(node) without a
check, which wraps around on 32-bit targets for inputs of about 1 GiB and
allocates a too small array. Throw std::bad_alloc for a count beyond the
address space and clamp the growth step to it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The non-little-endian path of the node writer wrote the integer's native
word over len and next, so len got the high half there. Compose and split
the value explicitly (len is the low half, next the high half); the
little-endian path stays a plain memcpy.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
json_document::parse(std::move(const_string)) failed to compile with
"no matching read_kind": only a non-const rvalue std::string can be moved
from. Treat a const rvalue as a copied byte container.
parse() does not accept everything BasicJsonType::parse() does: a FILE*
and pointers to or arrays of wide characters are rejected at compile time.
List the supported inputs instead.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
parse(ptr, len) compiled: len converted to allow_exceptions, and ptr was
read as a C string, past the end of a buffer without a terminating NUL.
Delete the overloads of parse, parse_copy, accept, and read that take an
integer other than bool where the flags are expected.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Drop the empty braced NSDMIs of the std::string members: old Clang
rejects the defaulted constructor when it is used by a member
initializer before the end of the class definition.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Add json_document and json_view, a read-only, zero-copy index of a
JSON text, as the first public slice of the zero-copy view (#5295).
A parse produces a flat array of 16-byte nodes in document order,
one per value and one per object key. Strings stay in the source
text; escaped strings are decoded into an arena. Integers are
converted while their digits are in the cache; floats keep only
their digit layout and are converted on read. Containers store the
size of their subtree, so a reader can step over one in constant
time. A document makes a handful of allocations, however many
values it has.
The parser accepts exactly what json::parse accepts, with every
combination of ignore_comments and ignore_trailing_commas, with and
without a trailing NUL, and under JSON_STRICT_NUL_HANDLING. It is
portable C++11 and does not depend on byte order.
basic_json_document adds parse, parse_copy, accept, read (reuses a
document's memory), root, is_discarded, source, owns_source,
node_count, memory_usage, and shrink_to_fit. basic_json_view adds
type, the is_* queries, operator bool, size, empty, materialize,
and source_offset. A parse error throws the same exception
basic_json::parse would throw for the same input, message and
position included.
detail::abi_config keeps JSON_STRICT_NUL_HANDLING and
JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON readable after json.hpp
undefines them, in the ABI namespace so they always match the
basic_json in use.
A NUL byte that ends a // comment is the end of the input, as in
parse() since #5696.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Fixes every CI failure develop has had since the API performance stack (#5831-#5842) was merged: compiler warnings and errors (Clang, GCC 4.9/11, MSVC 2015/2017/2022), clang-tidy, cpplint, Cppcheck, IWYU, the examples check, the no-exceptions, diagnostic-positions, implicit-conversions and strict-NUL test configurations, and the MinGW link (now with LLD).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
These PRs were merged while their CI runs were cancelled, so the warnings
went unnoticed. All of them fail the -Werror builds:
- json_pointer.hpp (#5836): size_t += int under GCC -Wsign-conversion
- binary_reader.hpp (#5839): char_int_type passed as char under GCC
-Wconversion
- json.hpp (#5838): @throw without a description under clang
-Wdocumentation
- string_escape.hpp (#5836): escape_into() was inserted between escape()
and its documentation, so the @return was attached to a void function
- unit-diagnostics.cpp (#5832): clang -Wnrvo in a lambda without an
explicit return type
- unit-to_json-rvalue.cpp (#5837): GCC -Wnoexcept (C++20) and clang
-Wunneeded-member-function on the test type
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
MSVC 2015 warns C4554 on the shifts with casts in the fractional part
of zmij::to_shortest despite the parentheses; with /WX every target
fails to build. Name the shift amounts as constants instead.
The shortest-digits test built the candidates with one digit fewer via
a stream with std::setprecision(digits.size() - 2); the standard library
of MSVC 2017 ignores a precision of 0, so for two-digit values the
"shorter" candidates were the value itself. Truncate the digits
instead: the value lies within half a unit of the last digit, so the
truncated digits and their successor are the closest candidates.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Escape characters below 0x60 from a table straight into the write buffer
When the bulk scanner in dump_escaped_impl() stops at a control
character, a quote or a backslash, that byte used to go through the
UTF-8 decoder and a switch that wrote the escape byte by byte into
string_buffer, which was then copied into write_buffer at the next run.
A byte below 0x60 at a character boundary is a complete code point, so
it now skips the decoder: its escape sequence comes from a 96-entry
table of 8-byte rows (sequence padded with zeros, length in the last
byte), and all 8 bytes are copied straight into write_buffer with one
room check. The check sits inside the existing character-boundary
block, so it runs once per scanner stop and not once per byte; testing
every byte cost about 6% on twitter.json with ensure_ascii.
The switch in the decoder's accept branch lost its cases for
characters below 0x60, which can no longer reach it.
Output is byte-identical. The string-buffer boundary tests now use
\u-escaped non-ASCII characters, since simple escapes no longer reach
string_buffer, and a new test case checks every ASCII character at
every position against a reference escaper.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Avoid raw string literals with backslashes in CHECK macros
MSVC's traditional preprocessor mis-lexes raw string literals inside
macro arguments and reports illegal escape sequences (C2017, C3850).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Sweep invalid UTF-8 at every position through dump()
Compare dump() of strings with ill-formed UTF-8 (stray continuation
bytes, invalid lead bytes, overlong encodings, surrogates, code points
above U+10FFFF, and sequences cut short by ASCII, a quote, a backslash,
a lead byte, or the end of the string) with an independent reference
serializer, for every error handler and with ensure_ascii on and off.
The pattern is put at every position of short strings to cross the
strides of the bulk scanners, and of longer ones to cross the end of the
512-byte string buffer.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Cppcheck 2.20 rejects invocations of unknown function-like macros in #if
conditions instead of evaluating them as 0. default_defines.hpp expands
the Hedley macros to the compiler's feature-test operators (__has_attribute
etc.), so define them as 0 in a generated include file.
It also reports a false bufferAccessOutOfBounds in the float overload of
float_from_bits, sizing the parameter as the double overload's; use the
size of the source instead (equal by the static_assert).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Allow the same type for number_integer_t and number_unsigned_t
basic_json did not compile with NumberUnsignedType == NumberIntegerType
because overloads on both types collided. Such a basic_json now stores
every integer as number_integer: the colliding overloads take a
placeholder type, unsigned C++ integers are converted to number_integer_t,
and value_t::number_unsigned maps to number_integer.
Also fixes two problems this configuration would hit: the unsigned
integer parser accepted values that wrap around to negative numbers, and
dump() overflowed when computing the absolute value of the minimum.
A repeated NumberFloatType gets a static_assert.
Refs #2573
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use json::with_integers_t in the test and the docs
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* ✅ skip patch test on GCC 5 to avoid internal compiler error
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI: name the unsigned overload type through template parameters
basic_json used number_unsigned_overload_t<basic_json> while the class was
still incomplete. MSVC rejected it as a recursive alias declaration and GCC
4.9, 5 and 7 crashed with an internal compiler error. basic_json now uses
number_unsigned_overload_type<NumberIntegerType, NumberUnsignedType>, which
also makes the GCC 5 workaround in the patch test unnecessary.
Infer reported a null dereference for the number_unsigned case in
from_json, which cannot happen for a basic_json whose number_unsigned_t is
number_integer_t; assert it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review: conjunction/negation trait, astyle-friendly alias, docs table order
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
patch_inplace() built its error messages inline. In particular get_value()
concatenated the context-sensitive "operation 'x'" string on every call, even
when nothing was wrong. The patch exceptions now come from NEVER_INLINE
helpers: the index and key errors reuse basic_json's existing
throw_array_index_out_of_range/throw_key_not_found, and the seven
patch-specific messages get one helper each in detail/exceptions.hpp. Ids
and messages are unchanged, and unit-throw_messages pins them (with and
without JSON_DIAGNOSTICS).
citm_catalog patch: -6.6 % (Apple clang), -9.5 % (GCC 16).
The other remaining throw sites (parser::syntax_error, binary_writer,
serializer, input_adapters, enum_key_duplicate, format_error) were prototyped
and measured: no gain beyond noise, so they stay as they are.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Speed up container destruction, iterator access and diff()
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Destroy the first levels of a value by recursion
The pointer-reversing walk in destroy_container() avoids the call stack but
moves a value around for every container it visits. Recurse for the first
recursion_depth_limit() levels instead - no allocation, no thread-local, a
bounded frame - and hand anything deeper to the walk. Destroy twitter
0.49 -> 0.36 ms, citm 0.80 -> 0.74 ms (3.12.0: 0.47 / 1.00).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the value lifecycle inline-friendly
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Compare standard containers without a thread-local depth count
operator== on objects and arrays recursed through the containers' own
operator== and counted the depth in a thread-local variable. Where reading a
thread-local costs a call (macOS), that added about 10% to comparing documents
with many small containers. For the standard containers, walk objects and
arrays directly, compare scalars and strings inline, and pass the depth as an
argument, falling back to compare_iteratively beyond the depth limit. Other
container types keep the previous path.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review: owns_memory() helper, leaf compare naming, astyle
- Add detail::owns_memory(value_t), a switch listing every enumerator so
-Wswitch-enum flags it when a type is added; destroy() and the copy
constructor use it instead of hand-written comparisons. Clang emits the
same bitmask test as before.
- Drop the parentheses around the scalar operands in
JSON_IMPLEMENT_SCALAR_CASES; they are only needed for the container
operands (#1530), which the comment now says.
- Build equal_bounded_containers from detail::conjunction so astyle no
longer pads the template brackets.
- Rename equal_scalar() to equal_leaf() and document why it also compares
strings, unlike JSON_IMPLEMENT_SCALAR_FAST.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix quadratic parent updates for ordered_json with JSON_DIAGNOSTICS
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Only trust containers known to keep their elements in place
Address review: decide whether an insertion may have moved the other
members by naming the containers that are known not to, instead of the
ones that are known to.
- detail::has_stable_elements: std::map, std::multimap,
std::unordered_map and std::unordered_multimap; object_capacity()
reports 0 for them.
- detail::is_nlohmann_ordered_map: only the real nlohmann::ordered_map
(which only appends) reports its capacity().
- Any other object type reports unknown_size(), so it gets a full
set_parents() on every insertion. Before, an object type without
capacity() was treated as never moving, and one with capacity() (a
sorted flat map, say) was trusted to move only on reallocation.
- set_parents_after_object_erase() re-parents unless the object type is
stable; the unused set_parent() is removed.
- Comments say "the object type" instead of "ordered_json".
- New test with an object type that moves its members without changing
its capacity; it fails without this change.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move elements when converting rvalue containers to JSON
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Simplify the rvalue to_json traits
Use negation<disjunction<...>> instead of a conjunction of negations and
keep the template argument lists compact (astyle only keeps the spaced
'< ... >' form when it is written that way).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Speed up array-to-container from_json
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Hint the end when converting an object to a standard container
from_json for map-like containers called emplace() without a hint, which
searches the tree for every key. The keys of the source object are sorted, so
inserting with the end as the hint is constant time, as it was in 3.12.0 via
std::inserter. get<std::map<...>>() of a 25,000 entry object: 4.2 ms -> 3.25 ms
(3.12.0: 3.7 ms).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fall back to emplace() in object from_json for containers without emplace_hint()
ordered_map has no hinted insert, so the insert(end(), ...) call broke
from_json into types holding an ordered_map (regression test #5122).
Use emplace_hint() where available and plain emplace() otherwise.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Speed up JSON pointer parsing, formatting, comparison and lookup
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Restore unflatten() speed after the linear-time rewrite
The prefix tree introduced for linear unflatten() kept a std::map per prefix,
copied each token into it and looked every token up a second time while
creating the values. Replace the per-prefix maps by one open-addressing table
keyed by (parent, token) that refers to the parsed tokens, skip the lookups for
the prefix shared with the previous key (flattened objects are mostly sorted),
and derive the prefix numbers of the second pass from parent links instead of
lookups.
Time and memory stay linear in the pointer depth, and the result is still
independent of the iteration order of the flattened object.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Put [[noreturn]] before __attribute__ on the JSON pointer throw helpers
Clang 18 on Linux rejects a C++11 attribute list after a GNU
__attribute__ ("an attribute list cannot appear here").
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make flatten() linear for ordered_json
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use priority_tag instead of a C-style variadic overload in add_flattened
clang-tidy flags the ellipsis fallback (cert-dcl50-cpp,
modernize-avoid-variadic-functions).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Write doubles with the shortest digits (Żmij), digits in registers
Write doubles with the conversion of Zmij by Victor Zverovich (MIT),
ported to C++11 (detail/conversions/zmij.hpp). It finds the
shortest decimal that reads back as the same double, and the
closest one if there are several. Grisu2, used until now, is fast
but not always shortest: it sometimes writes a 17th digit where 16
suffice, or a last digit that is not the closest. The layout is
unchanged (1.5, 100.0, 1e+100, -0.0); float keeps Grisu2.
Digits are converted eight at a time with the BCD conversion of
Xiang JunBo, as in Zmij, and written with one byte swap per eight
digits and fixed-size moves instead of per-digit loops. Leading and
trailing zeros are counted from those bytes. to_chars() uses a
local buffer when the caller's is shorter than the 41 bytes this
may write. The powers of ten come from the number-parsing table,
adjusted where it holds values rounded up, and extended with Zmij's
compressed tables beyond 10^308.
write_shortest() converts its 16 digits in one vector register
(SSE2 on x86-64, NEON on 64-bit Arm, both baseline) and inserts the
decimal point inside the register, avoiding a store-forwarding
stall that cost about 25% of the time to write a double. dump()
writes floats and integers straight into the serializer's write
buffer instead of copying them from a member buffer, and small
integers eight digits at a time. read_eight_bytes() and
parse_eight_digits() are marked always-inline, which GCC had been
calling out of line in the number-parsing loops.
Of one million random doubles, about 0.14% are now written with
different digits, always to a value that still reads back as the
same double.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI findings in the Zmij writer and its tests
- bit_ops.hpp: include macro_scope.hpp (JSON_HEDLEY_ALWAYS_INLINE); fixes IWYU
- to_chars.hpp: C4100 for the unused parameter in release builds, clang-tidy
sign comparison, cpplint runtime/int, GCC -Wstrict-overflow (unsigned abs)
- unit-to_chars.cpp: parse with the library instead of strtod (MinGW's strtod
rounds some 16/17 digit inputs wrongly); no floating-point std::to_chars
with icpc
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Select the Zmij conversion by a trait, not by the double overload
The non-template overloads for double asserted binary64 doubles wherever json.hpp was included, so the library no longer compiled where double is not IEEE 754 binary64 (AVR, -fshort-double). A trait now picks Zmij for any binary64 type, including a long double of that format (MSVC, Apple Arm), and Grisu2 for the others. Remove the unused write_short_decimal, powers_of_ten_16, zmij::decimal, zmij::to_decimal, and shortest_digits(double).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Credit Zmij's authors in to_chars.hpp, the README, and the license page
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Silence MSVC C4127 for a platform-constant check in the to_chars test
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Speed up the lexer: own float parser, string scan, and \u table
Give the library its own correctly rounded float converter for
binary32 and binary64 (IEEE 754), and speed up the lexer's string
and escape scanning.
The converter splits a number token into sign, significand, and
decimal exponent, then tries Clinger's fast path, then a templated
Eisel-Lemire step, and falls back to an exact big-integer digit
comparison for tokens with more than 19 significant digits whose two
candidate values round differently. This replaces std::from_chars
and strtod/strtof for both formats, so parsed values no longer
depend on the C/C++ library or the current locale. The strtold
fallback kept for other long double formats (x87, binary128) now
also copies a multi-byte decimal point correctly, fixing #5660.
eisel_lemire() and decimal_to_float() are always inlined so callers
keep the whole conversion in their hot loop.
The string-scanning kernels in string_scan.hpp find a stop byte with
the trailing-zero count of the SWAR mask instead of a byte loop, and
scalar_string_bulk_run() validates a run of multi-byte UTF-8
sequences one after another instead of re-searching after each one.
get_codepoint() decodes a contiguous \uXXXX escape with one table
lookup per byte instead of four range-checked get() calls; the
streaming path and all error positions are unchanged.
Adds 508 generated hard float-parsing cases with expected binary32
and binary64 bits, and kernel-comparison tests for the string scans
and the escape table against byte-by-byte references.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI warnings in own float parser
- pow5_table.hpp: pow5_128_largest_power was unused in this branch's
own code (GCC -Werror=unused-const-variable); tie it to the table
size with a static_assert instead of removing it, since a later
branch in the stack (json-view/23-zmij) uses it.
- number_parse.hpp: rename the local variable `copy` to `buffer` to
satisfy cpplint's build/include_what_you_use check.
- unit-class_lexer.cpp: extend the NOLINT list on the seeded mt19937
with bugprone-random-generator-seed, and parenthesize
`8 * sizeof(Bits) - 1` for clang-tidy.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Load string scan words with memcpy and use MSVC bit-scan intrinsics
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Reword float conversion notes for the string type and number_float_t docs
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy 22 findings in unit-class_lexer.cpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Mark the unsigned long of the MSVC bit scans for cpplint
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Drop the position updates in the bulk \u path, which #5809 made unused
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review: std::any_of in has_nonzero_digit, comment punctuation
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use data() instead of c_str() in the multi-byte decimal point path
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Pretty-printed JSON indents every line with a run of spaces, which
skip_whitespace() read one character at a time. For contiguous input
(bulk_scan) the lexer now skips the spaces after a newline 8 bytes at a
time directly in the input buffer, including the leading spaces of the
last, partial word. This bypasses get(), which is safe because
bulk_scan implies the lazy position from the previous commit: nothing
is counted per character.
Compared to lazy positions alone, pretty-printed input (twitter,
citm_catalog with indent 2 and 4) parses up to 13% faster and accept()
gets up to 30% faster (GCC 13 and clang 18 on x86-64, Apple clang 21 on
arm64). Minified input is unchanged within noise.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Derive the lexer position lazily for seekable input
The lexer counted the read position on every character: two increments
and a newline check per get(), all on members kept in memory. For
seekable input (pointers, strings, other random-access byte ranges) the
position can be derived instead: chars_read_total is the adapter's read
count (including reads at EOF) minus a pending unget, and line and
column are recomputed from the consumed input only when get_position()
is called, i.e. on parse errors. JSON_DIAGNOSTIC_POSITIONS uses the new
O(1) get_chars_read_total(). Stream, file, wide-string and user-defined
adapters keep the eager counters.
Parsing pretty-printed input gets 8-12% faster and accept() 12-27%
(canada, citm_catalog, twitter; clang 21 arm64 and GCC 13 x86-64).
Error messages, byte offsets, SAX positions and diagnostic positions are
unchanged (differential comparison against develop on 548k inputs).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move the std::vector<std::byte> binary type tests out of unit-regression3
The #4804/#4209 tests instantiated a second basic_json specialization
(with_binary_t<std::vector<std::byte>>), including its parser and CBOR
reader. With clang 18, the C++20 object of unit-regression3.cpp had
32656 COFF sections on develop; the lazy position helpers pushed it to
32975, past what the MinGW linker of the Windows clang jobs handles
("relocation truncated to fit: IMAGE_REL_AMD64_REL32 against .rdata").
unit-custom-binary-type.cpp already instantiates the same type, so the
tests move there; unit-regression3.cpp drops to 27616 sections.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Report the parse errors of the position test via SAX
The test caught json::parse_error, so it aborted in the
ci_test_noexceptions build, where JSON_NOEXCEPTION turns the throw into
std::abort(). sax_parse() hands the error position and the exception
object to parse_error() without throwing, so the test now records them
there and runs in every configuration.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Stop the BON8 writer overflowing the stack on deep values
to_bon8() recursed once per nesting level, so a value the iterative
BON8 reader accepts (e.g. ~24k nested one-element arrays) crashed on
the way back out. #5781 bounded the CBOR, MessagePack, and UBJSON/BJData
writers, but BON8 was merged before it and was not covered.
Apply the same scheme: recurse for the first recursion_depth_limit()
levels, then finish the value with write_bon8_iterative, which keeps
the open containers on a heap stack (reusing binary_container_frame)
and writes the 0xFE closer when it leaves a container with more than
four elements. The output is byte-for-byte unchanged.
Fixes https://issues.oss-fuzz.com/issues/572238015
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Merge the array and object branches of the BON8 iterative writer
write_bon8_iterative and write_bon8_value_or_push handled arrays and
objects in separate branches that repeated the end-of-container check,
the 0xFE closer and the marker computation. Share those parts and branch
only where arrays and objects really differ. The output is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hz7VJi1FTKr6gpseLErbbS
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Restore v3.12.0 support for custom object key types
Custom object_t types whose key_type is not string_t compiled with
v3.12.0 for several APIs that unreleased changes broke:
- to_bson failed for every custom key type (#5553 kept a const string_t*
to the key); the nested entry's header is now written where the entry
is found.
- Copying deep values (and parse, merge_patch, update, insert) required
operator== on keys (#5389); keys without one are now paired via find().
- to_cbor/to_msgpack required an implicit conversion to string_t (#5746,
#5328); keys without one go through a temporary basic_json again.
- at() required a conversion to string_t for its error message (#5727);
other keys are passed to concat() unchanged again.
The new unit-custom-object-key-type.cpp covers five key types with
different capabilities.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use the with_object_t alias for the custom object key test types
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Avoid floating-point equality in custom key type test
GCC with -Werror=float-equal rejects comparing the double value with ==.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Write the head of nested BSON elements in one helper
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Suppress bugprone-return-const-ref-from-parameter in key_for_message
The reference is only passed to concat() within the full-expression that
holds the key, like the similar helpers in binary_writer.hpp.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the value of the custom test key types private
clang-tidy (cppcoreguidelines-non-private-member-variables-in-classes)
rejects the protected member; the derived key types use a protected
accessor instead.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Pass keys with data() and size() unchanged into the at() miss message
key_for_message() converted every key that string_t can be constructed
from, so a miss on a string_t or string_view key copied it before
concat() copied it again. Keys that concat() can append through data()
and size() are now passed through; only other keys (string literals,
key types that just convert to string_t) are converted.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
write_bon8_value silently skipped discarded values, but the array/object
count marker still counted them, so [1, discarded] produced 82 91: a marker
announcing two elements followed by one. Throw type_error.321 like the other
binary writers do, at any nesting level.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Niels Lohmann