The review of #5620 renamed document_data's tape/tape_size/tape_cap to
nodes/node_count/node_capacity and made the decode arena a
std::vector<char>; adapt the code of this change to it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
From gregmarr's review of #5630:
- alloc_nodes() gives a request of half of the next chunk or more a chunk
of its own, so that the rest of the current chunk stays available for
smaller requests instead of being abandoned.
- The comments on edit_state, append_text(), and reserve_moved() say that
views stay valid across every edit (not just the first), that n is 0 for
empty strings, and why both capacities are checked.
- The architecture page names flag bits 3, 4, and 5 (linked) separately
and says that a discarded view has no node; erase.md rewords overload 3
and says why it never throws invalid_iterator.202.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The review of #5620 renamed document_data's tape/tape_size/tape_cap to
nodes/node_count/node_capacity and made the decode arena a
std::vector<char>; adapt the code of this change to it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
json.hpp, which object_index.hpp already includes, provides it; included
directly, the json_view amalgamation (which only treats nlohmann/json.hpp
as external) copied the json internals into json_view.hpp a second time,
so the fuzzers, which include both single headers, failed with
redefinitions (make_void, integer_sequence, priority_tag, ...).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Qt defines slots as a macro, so a Qt header included before
json_view.hpp broke the declarations of the local variable (see #5865).
The test variable is renamed as well.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The review of #5620 renamed document_data's tape/tape_size/tape_cap to
nodes/node_count/node_capacity and made the decode arena a
std::vector<char>; adapt the code of this change to it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The overload takes the literal by array reference so that its length is a
template parameter, like scan_literal() and the C array overloads of
to_json() and from_json().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
json_view.hpp uses std::istream and std::ostream itself, but IWYU sees
them through the exported json.hpp and asked to remove both includes,
which fails ci_single_binaries (single_json_view).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The review of #5620 renamed document_data's tape/tape_size/tape_cap to
nodes/node_count/node_capacity and made the decode arena a
std::vector<char>; adapt the code of this change to it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Develop's #5800 made basic_json report a JSON pointer array index such as
"/a" as parse_error.109 instead of out_of_range.404, as longer ones were;
the view's array_index() follows, with "-" and the empty token still
unresolved.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
memcpy() must not be passed a null pointer even for zero bytes, which
arena_run() did for an empty run while the arena was still empty
(ci_test_clang_sanitizer: "null pointer passed as argument 1" in
builder.hpp). Only the slow path can see such a run.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Qt defines emit as an empty macro, so a Qt header included before
json_view.hpp broke the declaration of the builder's emit(), see
issue #5865. It is now called push_node().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- inline_nodes is null when the document has no inline nodes, instead of
pointing past the end of the allocation; shrink_to_fit() no longer
passes it to memcpy() for an empty node array
- is_byte_char and is_wide_char use conjunction/disjunction, which
astyle leaves unspaced
- append_stream() uses (std::max) for the count read
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Changes promised in the replies to the review of #5620:
- read() reuses the owned copy: a copied input (an rvalue byte container,
a stream, FILE*, wide characters, an iterator range) is appended to the
cleared owned buffer instead of a fresh std::string being move-assigned
over it. A moved-in std::string still replaces it.
- The error replay runs the library parser with a SAX handler that stores
nothing and throws the reported exception, instead of
BasicJsonType::parse(), which built a DOM up to the error.
- Pointers to and arrays of wide characters, and FILE*, take the input
adapter path (converted to UTF-8 into an owned buffer) instead of
failing a static_assert. Wide pointers are NUL-terminated.
- Streams are read in blocks through rdbuf() into the buffer, instead of
character by character through the input adapter; eofbit is set at the
end, as by the library's stream adapter.
- tape/tape_size/tape_cap are now nodes/node_count/node_capacity.
- The builder's members get descriptive, m_-prefixed names.
- The shallow frame stack is a std::array (in a wrapper with a
user-provided constructor: -Weffc++ wants the member initialized, and
value-initializing the 64 frames costs up to 30 % on small documents).
- The value dispatch macro has no NEXT parameter; the switch is left with
break, and the macro is undefined after its last use.
- literal() and digits_exceed() deduce the length from the string literal;
each literal has one helper; the two `if (!is_float)` blocks are merged.
- arena_ensure() sits right before the writes; the decode arena is a
std::vector<char>.
- The builder records the deepest nesting, and materialize() sizes its
stack from it once (growing it only if a subtree is deeper).
- Comment fixes (hex4, slow_string, the NUL rule, put_utf8, 0 - m, the
arena resize, grow()) and formatting (macro backslashes, digit cases,
one power of ten per line, value_t assertion next to is_container()).
- build_with's template parameters follow the builder's order.
Greg's idea of keeping the source offset in `off` and the arena offset in
`next` for escaped strings was measured and not adopted: string access got
7-18 % slower with a select, and 3.5-18 % slower with a branch-free field
pick (Apple M-series, clang -O3), parsing was unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make parsing from std::istream and FILE* fast
Squashed onto develop cd6b428ef (CI fixes from #5860). Original commits:
- Let the lexer's bulk fast paths run on a window of the input
- Scan the get area of an input stream in bulk
- Lock a FILE once per parse instead of once per character
- Add benchmarks for parsing from a FILE* and for many values from a stream
- Avoid implicit unsigned char to char conversion in stream tests
- Make file_input_adapter::unlock() safe to call on a moved-from adapter
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add ABI tags for JSON_DISABLE_STREAM_BULK_SCAN and JSON_DISABLE_UNLOCKED_FILE_IO
Both macros select between two definitions of an input adapter, so
translation units compiled with different settings violated the ODR when
linked together. Like the other ABI-affecting macros, they now append a
tag to the inline namespace: _dsbs and _dufio. Their defaults move to
abi_macros.hpp so that json_fwd.hpp sees them, and the natvis file is
regenerated for the new namespaces.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Skip the exception-only checks of the stream input tests without exceptions
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not use Qt's keyword macros as identifiers
The prefix tree used by unflatten() (#5836) had a member named slots, which
Qt defines as an empty macro, so including a Qt header before this library
failed to compile. Rename it to table and add a test that includes the
library with Qt's keyword macros (slots, signals, emit, foreach, forever)
defined.
Fixes#5865
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep the Qt keyword macros in the test from failing -Wunused-macros
Clang (-Werror) rejected the macros that the test defines but the library
must not use, and clang-tidy asked for parentheses around the arguments
of the Q_FOREACH imitation, where the variable is a declaration.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Fixes every CI failure develop has had since the API performance stack (#5831-#5842) was merged: compiler warnings and errors (Clang, GCC 4.9/11, MSVC 2015/2017/2022), clang-tidy, cpplint, Cppcheck, IWYU, the examples check, the no-exceptions, diagnostic-positions, implicit-conversions and strict-NUL test configurations, and the MinGW link (now with LLD).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
less_general() (#5840) calls operator<(value_t, value_t), which exists in
C++20 only for GCC. Explicit instantiation of basic_json compiles every
member, so clang-cl and MSVC failed on unit-explicit_instantiation and
unit-regression2 in C++20 mode. Its only caller, operator<, is defined
only without three-way comparison, as it was before #5840.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
These PRs were merged while their CI runs were cancelled, so the warnings
went unnoticed. All of them fail the -Werror builds:
- json_pointer.hpp (#5836): size_t += int under GCC -Wsign-conversion
- binary_reader.hpp (#5839): char_int_type passed as char under GCC
-Wconversion
- json.hpp (#5838): @throw without a description under clang
-Wdocumentation
- string_escape.hpp (#5836): escape_into() was inserted between escape()
and its documentation, so the @return was attached to a void function
- unit-diagnostics.cpp (#5832): clang -Wnrvo in a lambda without an
explicit return type
- unit-to_json-rvalue.cpp (#5837): GCC -Wnoexcept (C++20) and clang
-Wunneeded-member-function on the test type
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
MSVC 2015 warns C4554 on the shifts with casts in the fractional part
of zmij::to_shortest despite the parentheses; with /WX every target
fails to build. Name the shift amounts as constants instead.
The shortest-digits test built the candidates with one digit fewer via
a stream with std::setprecision(digits.size() - 2); the standard library
of MSVC 2017 ignores a precision of 0, so for two-digit values the
"shorter" candidates were the value itself. Truncate the digits
instead: the value lies within half a unit of the last digit, so the
truncated digits and their successor are the closest candidates.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Escape characters below 0x60 from a table straight into the write buffer
When the bulk scanner in dump_escaped_impl() stops at a control
character, a quote or a backslash, that byte used to go through the
UTF-8 decoder and a switch that wrote the escape byte by byte into
string_buffer, which was then copied into write_buffer at the next run.
A byte below 0x60 at a character boundary is a complete code point, so
it now skips the decoder: its escape sequence comes from a 96-entry
table of 8-byte rows (sequence padded with zeros, length in the last
byte), and all 8 bytes are copied straight into write_buffer with one
room check. The check sits inside the existing character-boundary
block, so it runs once per scanner stop and not once per byte; testing
every byte cost about 6% on twitter.json with ensure_ascii.
The switch in the decoder's accept branch lost its cases for
characters below 0x60, which can no longer reach it.
Output is byte-identical. The string-buffer boundary tests now use
\u-escaped non-ASCII characters, since simple escapes no longer reach
string_buffer, and a new test case checks every ASCII character at
every position against a reference escaper.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Avoid raw string literals with backslashes in CHECK macros
MSVC's traditional preprocessor mis-lexes raw string literals inside
macro arguments and reports illegal escape sequences (C2017, C3850).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Sweep invalid UTF-8 at every position through dump()
Compare dump() of strings with ill-formed UTF-8 (stray continuation
bytes, invalid lead bytes, overlong encodings, surrogates, code points
above U+10FFFF, and sequences cut short by ASCII, a quote, a backslash,
a lead byte, or the end of the string) with an independent reference
serializer, for every error handler and with ensure_ascii on and off.
The pattern is put at every position of short strings to cross the
strides of the bulk scanners, and of longer ones to cross the end of the
512-byte string buffer.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Cppcheck 2.20 rejects invocations of unknown function-like macros in #if
conditions instead of evaluating them as 0. default_defines.hpp expands
the Hedley macros to the compiler's feature-test operators (__has_attribute
etc.), so define them as 0 in a generated include file.
It also reports a false bufferAccessOutOfBounds in the float overload of
float_from_bits, sizing the parameter as the double overload's; use the
size of the source instead (equal by the static_assert).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Allow the same type for number_integer_t and number_unsigned_t
basic_json did not compile with NumberUnsignedType == NumberIntegerType
because overloads on both types collided. Such a basic_json now stores
every integer as number_integer: the colliding overloads take a
placeholder type, unsigned C++ integers are converted to number_integer_t,
and value_t::number_unsigned maps to number_integer.
Also fixes two problems this configuration would hit: the unsigned
integer parser accepted values that wrap around to negative numbers, and
dump() overflowed when computing the absolute value of the minimum.
A repeated NumberFloatType gets a static_assert.
Refs #2573
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use json::with_integers_t in the test and the docs
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* ✅ skip patch test on GCC 5 to avoid internal compiler error
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI: name the unsigned overload type through template parameters
basic_json used number_unsigned_overload_t<basic_json> while the class was
still incomplete. MSVC rejected it as a recursive alias declaration and GCC
4.9, 5 and 7 crashed with an internal compiler error. basic_json now uses
number_unsigned_overload_type<NumberIntegerType, NumberUnsignedType>, which
also makes the GCC 5 workaround in the patch test unnecessary.
Infer reported a null dereference for the number_unsigned case in
from_json, which cannot happen for a basic_json whose number_unsigned_t is
number_integer_t; assert it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review: conjunction/negation trait, astyle-friendly alias, docs table order
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
patch_inplace() built its error messages inline. In particular get_value()
concatenated the context-sensitive "operation 'x'" string on every call, even
when nothing was wrong. The patch exceptions now come from NEVER_INLINE
helpers: the index and key errors reuse basic_json's existing
throw_array_index_out_of_range/throw_key_not_found, and the seven
patch-specific messages get one helper each in detail/exceptions.hpp. Ids
and messages are unchanged, and unit-throw_messages pins them (with and
without JSON_DIAGNOSTICS).
citm_catalog patch: -6.6 % (Apple clang), -9.5 % (GCC 16).
The other remaining throw sites (parser::syntax_error, binary_writer,
serializer, input_adapters, enum_key_duplicate, format_error) were prototyped
and measured: no gain beyond noise, so they stay as they are.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Speed up container destruction, iterator access and diff()
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Destroy the first levels of a value by recursion
The pointer-reversing walk in destroy_container() avoids the call stack but
moves a value around for every container it visits. Recurse for the first
recursion_depth_limit() levels instead - no allocation, no thread-local, a
bounded frame - and hand anything deeper to the walk. Destroy twitter
0.49 -> 0.36 ms, citm 0.80 -> 0.74 ms (3.12.0: 0.47 / 1.00).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the value lifecycle inline-friendly
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Compare standard containers without a thread-local depth count
operator== on objects and arrays recursed through the containers' own
operator== and counted the depth in a thread-local variable. Where reading a
thread-local costs a call (macOS), that added about 10% to comparing documents
with many small containers. For the standard containers, walk objects and
arrays directly, compare scalars and strings inline, and pass the depth as an
argument, falling back to compare_iteratively beyond the depth limit. Other
container types keep the previous path.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review: owns_memory() helper, leaf compare naming, astyle
- Add detail::owns_memory(value_t), a switch listing every enumerator so
-Wswitch-enum flags it when a type is added; destroy() and the copy
constructor use it instead of hand-written comparisons. Clang emits the
same bitmask test as before.
- Drop the parentheses around the scalar operands in
JSON_IMPLEMENT_SCALAR_CASES; they are only needed for the container
operands (#1530), which the comment now says.
- Build equal_bounded_containers from detail::conjunction so astyle no
longer pads the template brackets.
- Rename equal_scalar() to equal_leaf() and document why it also compares
strings, unlike JSON_IMPLEMENT_SCALAR_FAST.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix quadratic parent updates for ordered_json with JSON_DIAGNOSTICS
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Only trust containers known to keep their elements in place
Address review: decide whether an insertion may have moved the other
members by naming the containers that are known not to, instead of the
ones that are known to.
- detail::has_stable_elements: std::map, std::multimap,
std::unordered_map and std::unordered_multimap; object_capacity()
reports 0 for them.
- detail::is_nlohmann_ordered_map: only the real nlohmann::ordered_map
(which only appends) reports its capacity().
- Any other object type reports unknown_size(), so it gets a full
set_parents() on every insertion. Before, an object type without
capacity() was treated as never moving, and one with capacity() (a
sorted flat map, say) was trusted to move only on reallocation.
- set_parents_after_object_erase() re-parents unless the object type is
stable; the unused set_parent() is removed.
- Comments say "the object type" instead of "ordered_json".
- New test with an object type that moves its members without changing
its capacity; it fails without this change.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move elements when converting rvalue containers to JSON
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Simplify the rvalue to_json traits
Use negation<disjunction<...>> instead of a conjunction of negations and
keep the template argument lists compact (astyle only keeps the spaced
'< ... >' form when it is written that way).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Speed up array-to-container from_json
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Hint the end when converting an object to a standard container
from_json for map-like containers called emplace() without a hint, which
searches the tree for every key. The keys of the source object are sorted, so
inserting with the end as the hint is constant time, as it was in 3.12.0 via
std::inserter. get<std::map<...>>() of a 25,000 entry object: 4.2 ms -> 3.25 ms
(3.12.0: 3.7 ms).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fall back to emplace() in object from_json for containers without emplace_hint()
ordered_map has no hinted insert, so the insert(end(), ...) call broke
from_json into types holding an ordered_map (regression test #5122).
Use emplace_hint() where available and plain emplace() otherwise.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Speed up JSON pointer parsing, formatting, comparison and lookup
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Restore unflatten() speed after the linear-time rewrite
The prefix tree introduced for linear unflatten() kept a std::map per prefix,
copied each token into it and looked every token up a second time while
creating the values. Replace the per-prefix maps by one open-addressing table
keyed by (parent, token) that refers to the parsed tokens, skip the lookups for
the prefix shared with the previous key (flattened objects are mostly sorted),
and derive the prefix numbers of the second pass from parent links instead of
lookups.
Time and memory stay linear in the pointer depth, and the result is still
independent of the iteration order of the flattened object.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Put [[noreturn]] before __attribute__ on the JSON pointer throw helpers
Clang 18 on Linux rejects a C++11 attribute list after a GNU
__attribute__ ("an attribute list cannot appear here").
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make flatten() linear for ordered_json
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use priority_tag instead of a C-style variadic overload in add_flattened
clang-tidy flags the ellipsis fallback (cert-dcl50-cpp,
modernize-avoid-variadic-functions).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Niels Lohmann