tests/benchmarks/json_view/ holds a comparison with other
libraries, answering the question users ask when they pick one.
It is not built by CMake and not run by CI.
- bench_view.cpp: parse, traverse, select and dump of twitter,
citm_catalog, canada, jeopardy, a tweet and an RPC request,
against json_view, yyjson, simdjson, Boost.JSON and json::parse,
all agreeing and run interleaved.
- bench_corpus.cpp: parse, traverse and dump of any file list.
- bench_edit.cpp: parses, edits through each library's own API,
and serializes; all outputs must describe the same value.
- compare.py: builds both programs with system or pinned,
SHA-256-checked downloads, and writes results with the commit,
CPU, OS, compiler and library versions.
- README.md and a workflow that runs on demand or via a PR label.
Fairness fixes folded in: each engine runs once untimed before
each timed round, so the next one no longer pays for the
previous one's cleanup; dump is also compared with source numbers
against yyjson's raw-number mode; a "simdjson DOM (fresh)" column
and a reused-document column for json_view were added;
compare.py's paths are absolute; downloads use a .part file and a
failed checksum removes the archive.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add dump() and comparisons to json_view
Add basic_json_view::dump() and the comparison operators, and read
floats from the parser's digit layout instead of rescanning the
token.
dump(indent, indent_char, ensure_ascii, number_format) writes a
value the way ordered_json::parse(text).dump() writes it for the
same arguments: members in document order, all of them should a
key occur more than once; strings escaped by the same rules, using
the library's scanning kernels; floats written with the library's
to_chars conversion, so the output equals basic_json's byte for
byte; integers copied from the source, where they are already
canonical, except -0, which parse() reads as 0. There is no
error_handler argument, because the view only holds valid UTF-8.
number_format::source copies numbers exactly as they appear in the
source (e.g. "1.50", "1E2", "-0"), which basic_json cannot provide.
operator<< takes the indentation from the stream width, as for
basic_json. The writer walks iteratively, so nesting depth is
limited by memory only.
operator== and operator!= compare two views, or a view and a
basic_json value in either order, by the rules basic_json's
operator== uses: numbers compare by value across their types,
objects compare by their members with duplicate keys resolved as
parse() resolves them, member order matters only where the object
type keeps one, and discarded views compare as discarded basic_json
values do, including under JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON.
Nothing is materialized except single scalars.
While parsing, the view now records where the integer digits, the
fraction digits, and the exponent of a float token are, so floats
and doubles with at most 19 digits are read from that layout with
the library's decimal_to_float() instead of rescanning the token.
Both round correctly, so the values are those of parse(). get<double>(),
materialize(), dump(), and the comparisons all use it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Avoid raw string with escapes inside CHECK macro (MSVC C2017)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Size the dump buffer of a small value by its own extent
The source extent of a value was read from the next node, falling back to the rest of the document when that node held a decoded string. dump() of a small value could thus allocate a buffer as large as the document. Skip a few such nodes, cap the fallback estimate, and shrink a buffer that is much larger than its output.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Hold the documents of the dump and comparison tests in names
root() of a temporary document is deleted: its views would dangle.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not shrink the output of dump()
finish() copied the whole output when the buffer was more than twice as
large as the result (citm dump +11%). The tighter source_extent()
estimate already keeps the buffer of a small value small.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document and test first-wins lookups next to dump() and ==
dump() writes every member and == resolves duplicate keys as parse()
does, whereas lookups find the first member of a duplicate key.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the clang-tidy 22 findings in json_view and its tests
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Split the dump and comparison tests off unit-json_view.cpp
The MinGW linker of the Windows clang jobs cannot link object files with more
than 32767 sections ("relocation truncated to fit: IMAGE_REL_AMD64_REL32
against `.rdata'"). unit-json_view.cpp reaches that limit as the stack
grows, so its "json_view dump" and "json_view comparison" test cases move
into unit-json_view_dump.cpp. The test generator and has_duplicate_keys()
that both files use move into json_view_test_helpers.hpp.
The new file mentions JSON_HAS_CPP_17, so it is built for C++17 like the file
it was split from.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make json_view.hpp pass include-what-you-use
ci_single_binaries runs IWYU with --error on every header. For json_view.hpp
it suggested adding <array> (std::array is used), the headers that json.hpp
already provides (abi_config, abi_macros, input_adapters, json_pointer,
cpp_future, string_concat, value_t, json_fwd), and <version> for
std::nullptr_t, and removing <cstddef>.
- include <array>
- keep <cstddef> (nullptr_t, size_t; IWYU attributes them to <version> and <cstring>)
- tell IWYU not to suggest the headers that json.hpp provides: the amalgamated
json_view.hpp only includes json.hpp, so including them here would duplicate
their definitions
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Follow the renames from the review of the view builder
The review of #5620 renamed document_data's tape/tape_size/tape_cap to
nodes/node_count/node_capacity and made the decode arena a
std::vector<char>; adapt the code of this change to it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep the stream includes of json_view.hpp for IWYU
json_view.hpp uses std::istream and std::ostream itself, but IWYU sees
them through the exported json.hpp and asked to remove both includes,
which fails ci_single_binaries (single_json_view).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address the review of json_view dump() and comparisons
From gregmarr's review of #5624:
- The output buffer takes string literals without a length.
- write_float() names the IEEE single/double condition once, as the
library's serializer does.
- equal() checks the types first and compares with !=; the reference to
the top frame is noted as dangling after pop_back().
- dump.md: one paragraph per case in the notes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Exclude all json_view test files from Infer's reports
The same std::unique_ptr false positive (USE_AFTER_DELETE) also shows for
two json_documents parsed in one lambda in unit-json_view_dump.cpp
(ci_infer). It affects any test that holds several documents, so the
exclusion now covers tests/src/unit-json_view*.cpp; ASan and Valgrind
cover these tests.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Allow the string literal parameter of the view serializer's put() for clang-tidy
The overload takes the literal by array reference so that its length is a
template parameter, like scan_literal() and the C array overloads of
to_json() and from_json().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Bind the serializer's frame only after the check for its end
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add json_document and json_view: node index, parser, and document
Add json_document and json_view, a read-only, zero-copy index of a
JSON text, as the first public slice of the zero-copy view (#5295).
A parse produces a flat array of 16-byte nodes in document order,
one per value and one per object key. Strings stay in the source
text; escaped strings are decoded into an arena. Integers are
converted while their digits are in the cache; floats keep only
their digit layout and are converted on read. Containers store the
size of their subtree, so a reader can step over one in constant
time. A document makes a handful of allocations, however many
values it has.
The parser accepts exactly what json::parse accepts, with every
combination of ignore_comments and ignore_trailing_commas, with and
without a trailing NUL, and under JSON_STRICT_NUL_HANDLING. It is
portable C++11 and does not depend on byte order.
basic_json_document adds parse, parse_copy, accept, read (reuses a
document's memory), root, is_discarded, source, owns_source,
node_count, memory_usage, and shrink_to_fit. basic_json_view adds
type, the is_* queries, operator bool, size, empty, materialize,
and source_offset. A parse error throws the same exception
basic_json::parse would throw for the same input, message and
position included.
detail::abi_config keeps JSON_STRICT_NUL_HANDLING and
JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON readable after json.hpp
undefines them, in the ABI namespace so they always match the
basic_json in use.
A NUL byte that ends a // comment is the end of the input, as in
parse() since #5696.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix document_data for old Clang (3.4-3.6)
Drop the empty braced NSDMIs of the std::string members: old Clang
rejects the defaulted constructor when it is used by a member
initializer before the end of the class definition.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Exclude libstdc++'s string_view comparison from the sign-change check
basic_string_view::_S_compare stores the difference of two lengths in a
signed difference_type on purpose, which -fsanitize=integer reports for
every comparison with a shorter view (test-json_view_cpp17).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Reject integer lengths in json_document::parse
parse(ptr, len) compiled: len converted to allow_exceptions, and ptr was
read as a C string, past the end of a buffer without a terminating NUL.
Delete the overloads of parse, parse_copy, accept, and read that take an
integer other than bool where the flags are expected.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Copy const rvalue strings and document the accepted inputs
json_document::parse(std::move(const_string)) failed to compile with
"no matching read_kind": only a non-const rvalue std::string can be moved
from. Treat a const rvalue as a copied byte container.
parse() does not accept everything BasicJsonType::parse() does: a FILE*
and pointers to or arrays of wide characters are rejected at compile time.
List the supported inputs instead.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Remove the conversion to bool from json_view
explicit operator bool meant "refers to a value", which silently differs
from what a basic_json converts to. Use !v.is_discarded() instead.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Delete json_document::root() on temporary documents
auto v = json_document::parse(text).root() compiled and left the view
dangling. Delete the overload for rvalue documents, take a named document
in the tests, and document the lifetime rule.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Store node integers by halves on big-endian targets
The non-little-endian path of the node writer wrote the integer's native
word over len and next, so len got the high half there. Compose and split
the value explicitly (len is the low half, next the high half); the
little-endian path stays a plain memcpy.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Check the node array size for overflow
reserve(n) and the growth of the index computed n * sizeof(node) without a
check, which wraps around on 32-bit targets for inputs of about 1 GiB and
allocates a too small array. Throw std::bad_alloc for a count beyond the
address space and clamp the growth step to it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* State the exact input size limit of json_document
The check rejects inputs of 0xFFFFFFF0 bytes or more, but the exception
message and the documentation said 4 GiB. Name the limit once
(max_input_size), and state 4 GiB minus 16 bytes in the message and the
documentation. Test the limit with a container that only claims the size.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Remove the unused legacy_discarded_value_comparison from abi_config
Nothing reads it: the view does not compare values.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Align the banner comment of json_view.hpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fuzz json_document with exact-size buffers and parse options
The fuzzer only parsed a std::string with default options. Also parse an
exact-size byte vector, which has no NUL after its last byte and takes the
bounds-checked path, and derive ignore_comments and ignore_trailing_commas
from the first input byte, comparing against json::parse and json::accept
with the same options.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Give document_data a user-provided constructor
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy findings in unit-json_view_builder.cpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Annotate intentional patterns in unit-json_view.cpp for clang-tidy
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Suppress Flawfinder's read() finding on the deleted overload
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Check the steps of the deep-nesting test separately
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make max_input_size a function (GCC -Wunused-const-variable)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make json_view.hpp pass include-what-you-use
ci_single_binaries runs IWYU with --error on every header. For json_view.hpp
it suggested adding <array> (std::array is used), the headers that json.hpp
already provides (abi_config, abi_macros, input_adapters, json_pointer,
cpp_future, string_concat, value_t, json_fwd), and <version> for
std::nullptr_t, and removing <cstddef>.
- include <array>
- keep <cstddef> (nullptr_t, size_t; IWYU attributes them to <version> and <cstring>)
- tell IWYU not to suggest the headers that json.hpp provides: the amalgamated
json_view.hpp only includes json.hpp, so including them here would duplicate
their definitions
- export json.hpp, keep macro_unscope.hpp, and drop the unused forward declaration of the document class
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read the node array base after emit() in the view builder's open()
emit() moves the node array when it grows, and the subtraction read base
in the same expression, so the order was unspecified. MSVC Release builds
without forced inlining read the old base; the container index then
pointed outside the array and close() wrote out of bounds.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Silence a clang-tidy use-after-move finding in the json_view test
const_text is const, so std::move does not move from it; the second call
is part of the const rvalue test.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address the review of the view builder and document
Changes promised in the replies to the review of #5620:
- read() reuses the owned copy: a copied input (an rvalue byte container,
a stream, FILE*, wide characters, an iterator range) is appended to the
cleared owned buffer instead of a fresh std::string being move-assigned
over it. A moved-in std::string still replaces it.
- The error replay runs the library parser with a SAX handler that stores
nothing and throws the reported exception, instead of
BasicJsonType::parse(), which built a DOM up to the error.
- Pointers to and arrays of wide characters, and FILE*, take the input
adapter path (converted to UTF-8 into an owned buffer) instead of
failing a static_assert. Wide pointers are NUL-terminated.
- Streams are read in blocks through rdbuf() into the buffer, instead of
character by character through the input adapter; eofbit is set at the
end, as by the library's stream adapter.
- tape/tape_size/tape_cap are now nodes/node_count/node_capacity.
- The builder's members get descriptive, m_-prefixed names.
- The shallow frame stack is a std::array (in a wrapper with a
user-provided constructor: -Weffc++ wants the member initialized, and
value-initializing the 64 frames costs up to 30 % on small documents).
- The value dispatch macro has no NEXT parameter; the switch is left with
break, and the macro is undefined after its last use.
- literal() and digits_exceed() deduce the length from the string literal;
each literal has one helper; the two `if (!is_float)` blocks are merged.
- arena_ensure() sits right before the writes; the decode arena is a
std::vector<char>.
- The builder records the deepest nesting, and materialize() sizes its
stack from it once (growing it only if a subtree is deeper).
- Comment fixes (hex4, slow_string, the NUL rule, put_utf8, 0 - m, the
arena resize, grow()) and formatting (macro backslashes, digit cases,
one power of ten per line, value_t assertion next to is_container()).
- build_with's template parameters follow the builder's order.
Greg's idea of keeping the source offset in `off` and the arena offset in
`next` for escaped strings was measured and not adopted: string access got
7-18 % slower with a select, and 3.5-18 % slower with a branch-free field
pick (Apple M-series, clang -O3), parsing was unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Apply the review's wording fixes to the json_view documentation
- Copying is disabled because of its cost, not because of two documents
borrowing the same buffer; the unique_ptr follows from move-only.
- Inputs: streams, FILE*, and wide characters (strings, arrays, pointers)
are all supported and owned; point to parse_copy() for owned copies.
- The error replay only runs for failing input, builds no values, and is
skipped with allow_exceptions = false.
- read() writes copied inputs into the previous owned copy.
- Views can dangle like std::string_view, and why there is no
shared/weak pointer design.
- The 4 GiB limit (16-byte nodes, 16-byte scanner steps), streams being
read to the end first, which macros apply, why materialize() has no
source positions, and wording nits (accept() vs basic_json::accept(),
is_binary, is_discarded, see-also lists, size table, "back into").
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Say that json_view converts integers while parsing
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address the review: null inline nodes, trait spelling, and max() in json_view
- inline_nodes is null when the document has no inline nodes, instead of
pointing past the end of the allocation; shrink_to_fit() no longer
passes it to memcpy() for an empty node array
- is_byte_char and is_wide_char use conjunction/disjunction, which
astyle leaves unspaced
- append_stream() uses (std::max) for the count read
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Include materialize.hpp in the view builder test
The test calls detail::view::materialize(), which only the single header
pulled in; the multiple-header build (JSON_TEST_USING_MULTIPLE_HEADERS)
failed with "'materialize' is not a member of detail::view".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Rename emit() in the view builder for Qt compatibility
Qt defines emit as an empty macro, so a Qt header included before
json_view.hpp broke the declaration of the builder's emit(), see
issue #5865. It is now called push_node().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not copy an empty run into an unallocated arena
memcpy() must not be passed a null pointer even for zero bytes, which
arena_run() did for an empty run while the arena was still empty
(ci_test_clang_sanitizer: "null pointer passed as argument 1" in
builder.hpp). Only the slow path can see such a run.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Exclude the view builder test from Infer's reports
Infer's std::unique_ptr model confuses the values that the test's build()
helper returns by value and reports USE_AFTER_DELETE for one of them after
the destructor of an unrelated temporary (ci_infer, 12 reports, all in
tests/src/unit-json_view_builder.cpp). AddressSanitizer is clean on the
same tests.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Skip the deleted-overload checks of json_view on MSVC 2015
MSVC 2015 does not treat selecting a deleted function in decltype as a
substitution failure, so is_detected reports the deleted
parse/parse_copy/accept/read(ptr, len) overloads as callable and the
static_asserts in "integer arguments do not compile" fail (AppVeyor
VS2015 jobs). Calling them still fails to compile there. Guard the
negative checks as unit-delete_deprecated_functions.cpp does.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make parsing from std::istream and FILE* fast
Squashed onto develop cd6b428ef (CI fixes from #5860). Original commits:
- Let the lexer's bulk fast paths run on a window of the input
- Scan the get area of an input stream in bulk
- Lock a FILE once per parse instead of once per character
- Add benchmarks for parsing from a FILE* and for many values from a stream
- Avoid implicit unsigned char to char conversion in stream tests
- Make file_input_adapter::unlock() safe to call on a moved-from adapter
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add ABI tags for JSON_DISABLE_STREAM_BULK_SCAN and JSON_DISABLE_UNLOCKED_FILE_IO
Both macros select between two definitions of an input adapter, so
translation units compiled with different settings violated the ODR when
linked together. Like the other ABI-affecting macros, they now append a
tag to the inline namespace: _dsbs and _dufio. Their defaults move to
abi_macros.hpp so that json_fwd.hpp sees them, and the natvis file is
regenerated for the new namespaces.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Skip the exception-only checks of the stream input tests without exceptions
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Speed up array-to-container from_json
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Hint the end when converting an object to a standard container
from_json for map-like containers called emplace() without a hint, which
searches the tree for every key. The keys of the source object are sorted, so
inserting with the end as the hint is constant time, as it was in 3.12.0 via
std::inserter. get<std::map<...>>() of a 25,000 entry object: 4.2 ms -> 3.25 ms
(3.12.0: 3.7 ms).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fall back to emplace() in object from_json for containers without emplace_hint()
ordered_map has no hinted insert, so the insert(end(), ...) call broke
from_json into types holding an ordered_map (regression test #5122).
Use emplace_hint() where available and plain emplace() otherwise.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Update Google Benchmark to 1.10.0
Bump the pinned release and its archive checksum. Reword the comment and
README: the version is fixed for reproducible builds and updated like any
other dependency; results stay comparable because both sides of a
comparison are built with the same version.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not name the Google Benchmark version in the README
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move instead of deep-copy ordered_json values when an object grows
ordered_map keeps its elements in a std::vector<std::pair<const Key, T>>.
With a std::string key, that pair is not nothrow move constructible (the
const key has to be copied), so std::vector copies every element when it
reallocates. For ordered_json, this deep-copies every member value an
object already holds, including whole nested subtrees, on each growth
step.
Grow the storage in ordered_map instead, copying the keys and moving the
values. This happens in two phases, so the strong exception guarantee is
kept without try/catch. The first phase may throw, but only touches a
temporary buffer: it copies the keys, value-initializes the values, and
constructs the new element. The second phase moves the values (noexcept)
and swaps the buffers. Because the new element is constructed before any
value is moved, arguments that refer to elements of the container stay
valid, as with std::vector. Types that cannot take this path keep the
std::vector behavior.
Parsing into ordered_json (ParseStringOrdered, Apple M1 Max, clang -O3):
twitter 3.20 -> 1.70 ms, citm_catalog 7.73 -> 3.67 ms, jeopardy 219 ->
177 ms, canada unchanged. The number of allocations for twitter and
citm_catalog drops by two thirds.
Also add ParseStringOrdered rows to the benchmarks, and document the
growth behavior and the exception safety of ordered_map.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI: skip std::pair noexcept assumptions on EDG-based compilers
ci_icpc and ci_nvhpc failed to compile unit-ordered_map.cpp: the static
assertion that std::pair<const std::string, ordered_json> is not nothrow
move-constructible fails there. The EDG front end (Intel icpc 2021.10,
NVIDIA nvc++ 25.5) considers the defaulted move constructor of
std::pair<const Key, T> noexcept even if copying Key can throw. With these
compilers, std::vector already moves such elements itself when it grows,
and ordered_map correctly leaves growing to it.
The same misjudgement makes std::vector call std::terminate when a key copy
throws during growth, so the exception-safety test with throwing_key would
abort on these compilers as well.
Skip the static assertion and the exception-safety section when __EDG__ is
defined. Verified with icpc 2021.10 (-std=gnu++11) and nvc++ 25.5 (C++11 and
C++17) on Compiler Explorer: unit-ordered_map and unit-disabled_exceptions
build and pass.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add BON8 support
Add to_bon8/from_bon8 and input_format_t::bon8 for BON8, a binary format
that uses the byte values that cannot begin a UTF-8 character as type
markers, so strings need no length prefix. It is the most compact of the
supported binary formats on the benchmark files.
The reader is non-recursive like the other binary readers. A string ends
at the first byte that cannot continue it, so the reader hands the one or
two bytes it reads past a string back to the value that follows. The
writer produces the canonical representation of the specification, except
for NFC normalization; its output is identical to that of the reference
implementation (HikoGUI) on all files of the test data.
The round-trip tests need the .bon8 files of json_test_data 3.2.0.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review comments
- Reuse detail::validate_one_utf8 to check strings in to_bon8; the error
now names the first byte of the invalid sequence.
- Document that to_bon8 leaves bytes in the output adapter on an
exception, and that string_open is only an output of write_bon8_marker.
- Explain why the pushback buffer of the BON8 reader cannot overflow.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Select the BON8 float prefix by type
get_bon8_float_prefix only depends on the type of its argument, so make
the type a template parameter instead of passing an unused value.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Rename a test variable that Flawfinder mistakes for read()
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the BON8 CI failures
- compare the float in write_bon8_float with number_float_t constants,
so GCC does not warn about a float-to-double conversion
- mark check_bon8_utf8's context as used when exceptions are disabled
- choose the compact float prefix in a helper rather than with nested
conditional operators (clang-tidy)
- use auto for the cast in the BON8 integer reader (clang-tidy)
- write the int32 minimum test values as long long literals (MSVC C4146)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Amalgamate
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read BON8 strings in bulk from contiguous input
- copy the valid UTF-8 of a string in one step when the input is
contiguous (twitter.json is read in 1.68 instead of 2.52 ms,
jeopardy.json in 196 instead of 297 ms, close to CBOR and MessagePack)
- share the new valid_utf8_prefix() with the writer's UTF-8 check, which
now skips ASCII 8 bytes at a time
- let the fuzzer check that contiguous and stream input give the same
value or error, and test both paths in the unit tests
- clarify that a second 0xFF after a string is an empty string
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Link the BON8 functions from the other binary format pages
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Name the bulk scan flag after the input, not BON8
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read BSON keys in bulk from contiguous input
BSON keys (and array indices) are C-style strings, which were read byte
by byte. For contiguous input they are now read up to their \x00-byte in
one step, using the same bulk_scan flag as BON8 strings: twitter.json is
read in 1.46 instead of 2.01 ms, citm_catalog.json in 2.93 instead of
3.33 ms, jeopardy.json in 182 instead of 207 ms. canada.json, whose keys
are almost all one-digit array indices, takes 2 % longer.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the BON8 CI failures of the bulk-read tests
- skip the contiguous-versus-stream tests of BON8 strings and BSON keys
when exceptions are disabled: they catch the parse errors of invalid
input, and without exceptions the library aborts instead
- use static_cast for the int64 test value (google-readability-casting)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move the explicit basic_json instantiation into its own test file
Linking test-regression3_cpp20 with clang and MinGW failed with
"relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata'",
as test-regression2 did before #5511. The explicit instantiation of
basic_json<> for #4825 compiles every member function, including the
BON8 reader and writer, into that object, and it was already close to
the limit (2,226,104 bytes on develop, 2,234,960 with BON8; clang -O1,
C++20).
Give the instantiation a file of its own: unit-regression3 is now
1,594,736 bytes and unit-explicit_instantiation 1,095,064. The new file
mentions JSON_HAS_CPP_17 and JSON_HAS_CPP_20 so it keeps being built
for the C++17 standard the regression was about.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Convert the bytes of the BON8 test strings explicitly
The str() helper constructed a std::string from a byte range, which
converts each unsigned char implicitly; -fsanitize=integer reports that
for bytes of 0x80 and above (ci_test_clang_sanitizer).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document the benchmarks, and make them build and compare versions again
The benchmark project hasn't configured since #4793: download_test_data.cmake
compiles cmake/detect_libcpp_version.cpp relative to CMAKE_SOURCE_DIR, which
is tests/benchmarks when that is the top-level project, so try_run fails and
so does `make run_benchmarks`. The path is now relative to the module itself,
which is the same file for the main build.
The Dump benchmark discarded dump()'s result, which is [[nodiscard]] by now;
it warned, and left the optimizer free to shorten the loop. The result is
now kept with benchmark::DoNotOptimize.
A new cache variable, JSON_BENCHMARK_INCLUDE_DIR, names the directory holding
the nlohmann/json.hpp to benchmark (single_include by default, as before),
so the same benchmarks can be built against two versions and compared.
tests/benchmarks/README.md documents what is measured, how to build and run
the benchmarks, how to read the output, and how to compare two versions with
Google Benchmark's compare.py; it recommends doing so by hand before a
release rather than in CI.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Point ci_benchmarks at tests/benchmarks
The target has configured ${PROJECT_SOURCE_DIR}/benchmarks since it was
added in #2561, but the benchmarks live in tests/benchmarks.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Pin Google Benchmark to release 1.9.5
The benchmarks fetched Google Benchmark's main branch, so two builds on
different days could measure with different library code, and CMake 3.30
and later warn that the single-argument FetchContent_Populate() is
deprecated. Fetch the 1.9.5 release archive, verified by its SHA-256,
with FetchContent_MakeAvailable() instead. That needs CMake 3.14; Google
Benchmark itself already needed 3.13.
Its -Werror is switched off, so a newer compiler's new warnings cannot
break the pinned release, and its install rules are no longer added.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The benchmark suite covered parsing JSON text, dumping, and serializing to
CBOR, but only one binary read: FromMsgpack. Nothing measured from_cbor,
from_ubjson, from_bjdata or from_bson, so a change to binary_reader.hpp had
no baseline to be compared against.
Add read benchmarks for every format, in the two shapes that matter: from a
contiguous buffer, which is what most callers pass, and from a FILE*, which
is what FromMsgpack already measures and which compiles to different code.
FromMsgpack itself is left untouched so its numbers stay comparable across
releases. The input is derived at setup time by serializing a parsed test
file, because the test data repository ships JSON only.
The test files are wide and shallow, but the readers' cost is per container,
so add three value shapes they do not cover -- deeply nested containers, many
sibling containers, and one flat array of numbers -- plus an indefinite-length
CBOR string, a form the writer never emits and which therefore has to be
assembled by hand. UBJSON and BJData are also captured in their size- and
type-annotated form, which the readers handle in a separate code path.
BSON requires an object at the top level, so it cannot reuse the array-rooted
test files; it is captured on the object-rooted ones, and the shapes are
wrapped in an object so every format measures the same value. The setup marks
the benchmark as skipped rather than letting the exception escape if that
requirement is ever violated.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Every input in the benchmark corpus is minified or only lightly spaced, so
none of them exercise the lexer's whitespace handling. Real-world JSON is
frequently indented - configuration files, pretty-printed API responses,
anything kept under version control - where the insignificant whitespace can
outweigh the data itself.
Add a ParseIndented family that re-serializes each document with an
indentation and parses that. The content is identical to the matching
ParseString row, so the pair isolates the cost of the whitespace alone.
Measured here, parsing the indented form costs 16-34% more than the minified
form of the same document, which nothing in the suite currently reports.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* multibyte binary reader
* wide_string_input_adapter fallback to get_character
Update input_adapters.hpp
* Update json.hpp
* Add from msgpack test
* Test for broken msgpack with stream, address some warnings
* Reading binary number from wchar as an error, address warnings
* Not casting float to int, it violates strict aliasing rule
Starting with CMake 3.27, deprecation warnings are issued
when asking for policy settings for CMake 3.4 or earlier.
The cmake_minimum_required() command accepts a version
range, which allows NEW policy settings up to the upper end
of that range to be used, but without raising the minimum
CMake version above the bottom of that range. This means
NEW policy settings will be used where available, without
requiring them. This change updates the project's
cmake_minimum_required() calls to use a version range to
extend the upper policy version to 3.14 where it wasn't already
at that version or higher. This prevents the deprecation warning
from CMake 3.27, and gives breathing space before a future
CMake release will start issuing similar deprecation warnings
again.
* Add benchmark for cbor serialization and cbor binary data serialization
* Use std::vector::insert method instead of std::copy in output_vector_adapter::write_characters
This change increases a lot the performance when writing lots of binary data.
Niels Lohmann