mirror of
https://github.com/nlohmann/json.git
synced 2026-11-09 16:21:35 +00:00
7757de3a58e23534486bf4f16827ad26154cc9a4
33
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Niels Lohmann
|
2c7474affc |
docs: add the blog post "How nlohmann/json 3.13.0 got 3× faster" (#6022)
* docs: add the blog post "How 3.13.0 got faster" A post for the documentation (Home > Blog) on how 3.13.0 became faster: eight themes with the pull requests behind them, the ideas taken from other libraries, what did not work, how the fast paths are tested, and the compile-time cost. Every number comes from a run on 2026-10-27 (3.12.0 against develop plus the open non-draft PRs, Apple M1 Max) or from the linked PR. - docs/mkdocs/docs/home/blog/how_3_13_0_got_faster.md: the post. - docs/mkdocs/scripts/figures/perf_blog.py: draws its 12 figures (overview, eight timeline panels, the SWAR test, the cliffs, json_view throughput) as inline SVGs in the nj-charts style; data/perf_timeline.csv holds the timeline. - tests/benchmarks/history: history.cpp and history.py time one fixed program against the header of every commit that changed it, interleaved, and write one CSV row per commit with its PR. The timeline of the post comes from it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: compare json_document with Glaze, sonic, RapidJSON and Rust parsers Extends the comparison in "How 3.13.0 got faster" from yyjson, simdjson and Boost.JSON to eleven engines: sonic-cpp, RapidJSON and Glaze (generic DOM, lazy_json, reading into structs) in C++, sonic-rs, simd-json and serde_json in Rust. Measured on an Apple M1 Max (Apple clang 21, Rust 1.99) and an x86-64 Haswell server (GCC 13, Clang 18). Every engine must return the same values, string bytes and bit-identical number sums before it is timed. The post now answers whether json_document is the fastest parser: it is the fastest document parser at parse+dump everywhere and at parsing on Apple silicon, ties with simdjson's DOM on x86-64, and loses to simdjson On-Demand and Glaze structs when only a few fields are read. tests/benchmarks/json_view/others holds the programs (bench_all.cpp, the Rust crate with its Cargo.lock), summarize.py and the pinned versions and checksums. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: captions, test-file table and figure fixes for the 3.13.0 post - caption every table and figure - describe what makes twitter.json, citm_catalog.json, canada.json and jeopardy.json hard, and what each timeline panel exercises - split the json_view layout figure into the tree and the index, with the example text above; fix the arrows into the string boxes - arrows and a "less is better" axis label in the cliffs figure - name nlohmann/json in the comparison tables and chart - licenses of the adapted code in the thanks section Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: byline and licenses for the 3.13.0 post - author and date below the title, styled as .nj-byline; its link gets no external-link icon - the license of every library in the thanks section, checked against each project's license files; credit sonic-rs for QUOTE_TAB, which orjson adapted from it Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: dump comparison and split acknowledgments in the 3.13.0 post - a dump workload in bench_all and bench_rs: each engine writes a document parsed beforehand, and every output must read back with bit-identical numbers; an optional argument selects one workload - the post's comparison gains a write table, a throughput chart, a dump row for all three configurations, and the x86-64 cases it loses - the thanks section is split into adapted code and borrowed ideas Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: x86-64 dump numbers with #5908's branch-free token digits The dump of canada.json with GCC on x86-64 lost to yyjson by 0.78 because GCC compiled #5908's token-digit selections as branches; |
||
|
Niels Lohmann
|
6d5581c4e8 |
CBOR/MessagePack readers: leave one-element arrays to the general path (#6008)
enter_cbor_array and enter_msgpack_array called the out-of-line scalar run for every non-empty array. When the array's element is an object or an array, as on every level of [[...]] or the single ring of a GeoJSON polygon, the call reads nothing; it made 1,000 arrays nested in each other cost 4 % more instructions (GCC 13) and up to 8 % more time (Apple clang) than without the run. #5975 had fixed that with an inline pre-check, which was later dropped after measuring canada only. The run is now entered only for arrays of two or more elements: the same comparison as before, so no other row changes. Arrays of one scalar, which the run read faster, now take the general path; they are rare in documents (one in citm_catalog, none in twitter or canada). The new benchmark FromBinaryShape/singles measures them, next to FromBinaryShape/nested. Instructions per repetition, new/old, GCC 13 callgrind: nested arrays 0.973 (CBOR) and 0.971 (MessagePack), arrays of one scalar 1.037 and 1.021, every other row within 0.15 %. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
d20c4dcd89 |
json_view benchmarks: compare fairly and state what each row means (#5914)
- bench_view: add "(editable)" rows for json_editable_document and a parse+dump workload, so the cost of lazy float conversion shows next to the parse row (the traverse workload already includes the parse). Check that all engines' dumps agree where outputs are comparable. - compare.py: add --aligned, which also builds and runs with aligned code and reports both tables, so code-layout artifacts become visible. Off by default; the default output is unchanged. Not run end to end yet. - README: say which engines are like-for-like peers (yyjson, simdjson DOM) and which are mutable reference points (Boost.JSON, json::parse, the editable json_view); explain the lazy float conversion and the eager prototype that was measured and dropped. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
e5fe4ebdd2 |
json_view: from_json() of the macros reads views directly (#5922)
get<T>() and get_to() of a view call T's from_json() with the view itself when that from_json() accepts views but not an unrelated probe type, and fall back to materialize() otherwise. The from_json() functions of the NLOHMANN_DEFINE_TYPE_*, NLOHMANN_DEFINE_DERIVED_TYPE_* and NLOHMANN_JSON_SERIALIZE_ENUM(_STRICT) macros accept basic_json values and views; a derived type whose base reads only json reads the base part through materialize(). The library's own from_json() overloads are not called with views: from_json() of a view (and of the probe) only looks up functions by ADL. New public trait nlohmann::is_json_view<T>, and the member type basic_json_type of basic_json_view. With duplicate keys, the members of such a T are those the view's lookups find (the first ones). bench_decode.cpp measures decoding into structs (4-6x faster than materialize().get<T>() on twitter, citm_catalog, and synthetic records). Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
330fbead59 |
Document clang-tidy exclusions, drop unneeded ones, and clean up doc examples (#5909)
* Document clang-tidy exclusions and drop unneeded ones Re-enable every excluded check in a run of clang-tidy 22 over all tests and remove the seven that never fire (cert-err58-cpp, clang-analyzer-webkit.NoUncountedMemberChecker, cppcoreguidelines-virtual-class-destructor, google-readability-function-size, hicpp-function-size, hicpp-no-assembler, readability-function-size). Document the reason for every remaining exclusion at the top of the file. Exclude in advance the checks that only exist in clang-tidy 23 and are noise here (bugprone-signed-bitwise, *-explicit-constructor, readability-trailing-comma, modernize-use-string-view), plus modernize-use-integer-sign-comparison, which needs C++20 std::cmp_less and flags deliberate casts. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy findings in documentation examples and replace redundant aliases Apply clang-tidy fixes to the documentation examples: - performance-avoid-endl: std::endl -> '\n' - readability-qualified-auto: auto -> auto* / const auto* - performance-for-range-copy: copy -> const reference - google-readability-casting: C-style cast -> static_cast Replace redundant aliases (readability-redundant-qualified-alias) such as "using json = nlohmann::json;" with "using nlohmann::json;" in the documentation, README, examples, tests, and benchmarks. Aliases at class scope and aliases of other types (with_base_class_t etc.) are kept. No changes to the library or its public API. All changed examples were compiled and run; their output matches the stored .output files (except max_size and meta, which depend on the platform and compiler). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Format doc examples with the pinned astyle Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
2dc1e54856 |
Compare json_view with yyjson, simdjson, and Boost.JSON (#5627)
* Compare json_view with yyjson, simdjson, and Boost.JSON tests/benchmarks/json_view/ holds a comparison with other libraries, answering the question users ask when they pick one. It is not built by CMake and not run by CI. - bench_view.cpp: parse, traverse, select and dump of twitter, citm_catalog, canada, jeopardy, a tweet and an RPC request, against json_view, yyjson, simdjson, Boost.JSON and json::parse, all agreeing and run interleaved. - bench_corpus.cpp: parse, traverse and dump of any file list. - bench_edit.cpp: parses, edits through each library's own API, and serializes; all outputs must describe the same value. - compare.py: builds both programs with system or pinned, SHA-256-checked downloads, and writes results with the commit, CPU, OS, compiler and library versions. - README.md and a workflow that runs on demand or via a PR label. Fairness fixes folded in: each engine runs once untimed before each timed round, so the next one no longer pays for the previous one's cleanup; dump is also compared with source numbers against yyjson's raw-number mode; a "simdjson DOM (fresh)" column and a reused-document column for json_view were added; compare.py's paths are absolute; downloads use a .part file and a failed checksum removes the archive. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make the downloads of json_view's compare.py robust Hash the archives in chunks instead of reading them into memory, download with a timeout, extract into a temporary directory that is renamed into place only after success (a half-extracted directory was trusted forever), and split CXX and CC into arguments so that values like 'ccache g++' work. Note at the pins that a SHA-256 mismatch of a GitHub tag archive means that GitHub regenerated it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Add a timeout and a concurrency group to the json_view benchmarks Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Check the CSV file and the round count in the json_view benchmarks fopen() of the CSV output was not checked, so an unwritable directory made fprintf() write to a null FILE*; the round count is parsed with strtol and clamped instead of atoi. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
898e6ee81d |
Add benchmarks for SAX parsing, value operations, JSON Pointer, and conversions (#5845)
Squashed onto develop
|
||
|
Niels Lohmann
|
4c3a3eb5b5 |
Add dump() and comparisons to json_view (#5624)
* Add dump() and comparisons to json_view Add basic_json_view::dump() and the comparison operators, and read floats from the parser's digit layout instead of rescanning the token. dump(indent, indent_char, ensure_ascii, number_format) writes a value the way ordered_json::parse(text).dump() writes it for the same arguments: members in document order, all of them should a key occur more than once; strings escaped by the same rules, using the library's scanning kernels; floats written with the library's to_chars conversion, so the output equals basic_json's byte for byte; integers copied from the source, where they are already canonical, except -0, which parse() reads as 0. There is no error_handler argument, because the view only holds valid UTF-8. number_format::source copies numbers exactly as they appear in the source (e.g. "1.50", "1E2", "-0"), which basic_json cannot provide. operator<< takes the indentation from the stream width, as for basic_json. The writer walks iteratively, so nesting depth is limited by memory only. operator== and operator!= compare two views, or a view and a basic_json value in either order, by the rules basic_json's operator== uses: numbers compare by value across their types, objects compare by their members with duplicate keys resolved as parse() resolves them, member order matters only where the object type keeps one, and discarded views compare as discarded basic_json values do, including under JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON. Nothing is materialized except single scalars. While parsing, the view now records where the integer digits, the fraction digits, and the exponent of a float token are, so floats and doubles with at most 19 digits are read from that layout with the library's decimal_to_float() instead of rescanning the token. Both round correctly, so the values are those of parse(). get<double>(), materialize(), dump(), and the comparisons all use it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Avoid raw string with escapes inside CHECK macro (MSVC C2017) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Size the dump buffer of a small value by its own extent The source extent of a value was read from the next node, falling back to the rest of the document when that node held a decoded string. dump() of a small value could thus allocate a buffer as large as the document. Skip a few such nodes, cap the fallback estimate, and shrink a buffer that is much larger than its output. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Hold the documents of the dump and comparison tests in names root() of a temporary document is deleted: its views would dangle. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not shrink the output of dump() finish() copied the whole output when the buffer was more than twice as large as the result (citm dump +11%). The tighter source_extent() estimate already keeps the buffer of a small value small. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Document and test first-wins lookups next to dump() and == dump() writes every member and == resolves duplicate keys as parse() does, whereas lookups find the first member of a duplicate key. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the clang-tidy 22 findings in json_view and its tests Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Split the dump and comparison tests off unit-json_view.cpp The MinGW linker of the Windows clang jobs cannot link object files with more than 32767 sections ("relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata'"). unit-json_view.cpp reaches that limit as the stack grows, so its "json_view dump" and "json_view comparison" test cases move into unit-json_view_dump.cpp. The test generator and has_duplicate_keys() that both files use move into json_view_test_helpers.hpp. The new file mentions JSON_HAS_CPP_17, so it is built for C++17 like the file it was split from. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make json_view.hpp pass include-what-you-use ci_single_binaries runs IWYU with --error on every header. For json_view.hpp it suggested adding <array> (std::array is used), the headers that json.hpp already provides (abi_config, abi_macros, input_adapters, json_pointer, cpp_future, string_concat, value_t, json_fwd), and <version> for std::nullptr_t, and removing <cstddef>. - include <array> - keep <cstddef> (nullptr_t, size_t; IWYU attributes them to <version> and <cstring>) - tell IWYU not to suggest the headers that json.hpp provides: the amalgamated json_view.hpp only includes json.hpp, so including them here would duplicate their definitions Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate the amalgamated headers Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Follow the renames from the review of the view builder The review of #5620 renamed document_data's tape/tape_size/tape_cap to nodes/node_count/node_capacity and made the decode arena a std::vector<char>; adapt the code of this change to it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep the stream includes of json_view.hpp for IWYU json_view.hpp uses std::istream and std::ostream itself, but IWYU sees them through the exported json.hpp and asked to remove both includes, which fails ci_single_binaries (single_json_view). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address the review of json_view dump() and comparisons From gregmarr's review of #5624: - The output buffer takes string literals without a length. - write_float() names the IEEE single/double condition once, as the library's serializer does. - equal() checks the types first and compares with !=; the reference to the top frame is noted as dangling after pop_back(). - dump.md: one paragraph per case in the notes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate the amalgamated headers Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Exclude all json_view test files from Infer's reports The same std::unique_ptr false positive (USE_AFTER_DELETE) also shows for two json_documents parsed in one lambda in unit-json_view_dump.cpp (ci_infer). It affects any test that holds several documents, so the exclusion now covers tests/src/unit-json_view*.cpp; ASan and Valgrind cover these tests. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Allow the string literal parameter of the view serializer's put() for clang-tidy The overload takes the literal by array reference so that its length is a template parameter, like scan_literal() and the C array overloads of to_json() and from_json(). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Bind the serializer's frame only after the check for its end Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate the amalgamated headers Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
122d718330 |
Add json_document and json_view: node index, parser, and document (#5620)
* Add json_document and json_view: node index, parser, and document Add json_document and json_view, a read-only, zero-copy index of a JSON text, as the first public slice of the zero-copy view (#5295). A parse produces a flat array of 16-byte nodes in document order, one per value and one per object key. Strings stay in the source text; escaped strings are decoded into an arena. Integers are converted while their digits are in the cache; floats keep only their digit layout and are converted on read. Containers store the size of their subtree, so a reader can step over one in constant time. A document makes a handful of allocations, however many values it has. The parser accepts exactly what json::parse accepts, with every combination of ignore_comments and ignore_trailing_commas, with and without a trailing NUL, and under JSON_STRICT_NUL_HANDLING. It is portable C++11 and does not depend on byte order. basic_json_document adds parse, parse_copy, accept, read (reuses a document's memory), root, is_discarded, source, owns_source, node_count, memory_usage, and shrink_to_fit. basic_json_view adds type, the is_* queries, operator bool, size, empty, materialize, and source_offset. A parse error throws the same exception basic_json::parse would throw for the same input, message and position included. detail::abi_config keeps JSON_STRICT_NUL_HANDLING and JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON readable after json.hpp undefines them, in the ABI namespace so they always match the basic_json in use. A NUL byte that ends a // comment is the end of the input, as in parse() since #5696. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix document_data for old Clang (3.4-3.6) Drop the empty braced NSDMIs of the std::string members: old Clang rejects the defaulted constructor when it is used by a member initializer before the end of the class definition. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Exclude libstdc++'s string_view comparison from the sign-change check basic_string_view::_S_compare stores the difference of two lengths in a signed difference_type on purpose, which -fsanitize=integer reports for every comparison with a shorter view (test-json_view_cpp17). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Reject integer lengths in json_document::parse parse(ptr, len) compiled: len converted to allow_exceptions, and ptr was read as a C string, past the end of a buffer without a terminating NUL. Delete the overloads of parse, parse_copy, accept, and read that take an integer other than bool where the flags are expected. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Copy const rvalue strings and document the accepted inputs json_document::parse(std::move(const_string)) failed to compile with "no matching read_kind": only a non-const rvalue std::string can be moved from. Treat a const rvalue as a copied byte container. parse() does not accept everything BasicJsonType::parse() does: a FILE* and pointers to or arrays of wide characters are rejected at compile time. List the supported inputs instead. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove the conversion to bool from json_view explicit operator bool meant "refers to a value", which silently differs from what a basic_json converts to. Use !v.is_discarded() instead. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Delete json_document::root() on temporary documents auto v = json_document::parse(text).root() compiled and left the view dangling. Delete the overload for rvalue documents, take a named document in the tests, and document the lifetime rule. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Store node integers by halves on big-endian targets The non-little-endian path of the node writer wrote the integer's native word over len and next, so len got the high half there. Compose and split the value explicitly (len is the low half, next the high half); the little-endian path stays a plain memcpy. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Check the node array size for overflow reserve(n) and the growth of the index computed n * sizeof(node) without a check, which wraps around on 32-bit targets for inputs of about 1 GiB and allocates a too small array. Throw std::bad_alloc for a count beyond the address space and clamp the growth step to it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * State the exact input size limit of json_document The check rejects inputs of 0xFFFFFFF0 bytes or more, but the exception message and the documentation said 4 GiB. Name the limit once (max_input_size), and state 4 GiB minus 16 bytes in the message and the documentation. Test the limit with a container that only claims the size. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove the unused legacy_discarded_value_comparison from abi_config Nothing reads it: the view does not compare values. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Align the banner comment of json_view.hpp Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fuzz json_document with exact-size buffers and parse options The fuzzer only parsed a std::string with default options. Also parse an exact-size byte vector, which has no NUL after its last byte and takes the bounds-checked path, and derive ignore_comments and ignore_trailing_commas from the first input byte, comparing against json::parse and json::accept with the same options. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Give document_data a user-provided constructor Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy findings in unit-json_view_builder.cpp Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Annotate intentional patterns in unit-json_view.cpp for clang-tidy Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Suppress Flawfinder's read() finding on the deleted overload Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Check the steps of the deep-nesting test separately Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make max_input_size a function (GCC -Wunused-const-variable) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make json_view.hpp pass include-what-you-use ci_single_binaries runs IWYU with --error on every header. For json_view.hpp it suggested adding <array> (std::array is used), the headers that json.hpp already provides (abi_config, abi_macros, input_adapters, json_pointer, cpp_future, string_concat, value_t, json_fwd), and <version> for std::nullptr_t, and removing <cstddef>. - include <array> - keep <cstddef> (nullptr_t, size_t; IWYU attributes them to <version> and <cstring>) - tell IWYU not to suggest the headers that json.hpp provides: the amalgamated json_view.hpp only includes json.hpp, so including them here would duplicate their definitions - export json.hpp, keep macro_unscope.hpp, and drop the unused forward declaration of the document class Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read the node array base after emit() in the view builder's open() emit() moves the node array when it grows, and the subtraction read base in the same expression, so the order was unspecified. MSVC Release builds without forced inlining read the old base; the container index then pointed outside the array and close() wrote out of bounds. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Silence a clang-tidy use-after-move finding in the json_view test const_text is const, so std::move does not move from it; the second call is part of the const rvalue test. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate the amalgamated headers Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address the review of the view builder and document Changes promised in the replies to the review of #5620: - read() reuses the owned copy: a copied input (an rvalue byte container, a stream, FILE*, wide characters, an iterator range) is appended to the cleared owned buffer instead of a fresh std::string being move-assigned over it. A moved-in std::string still replaces it. - The error replay runs the library parser with a SAX handler that stores nothing and throws the reported exception, instead of BasicJsonType::parse(), which built a DOM up to the error. - Pointers to and arrays of wide characters, and FILE*, take the input adapter path (converted to UTF-8 into an owned buffer) instead of failing a static_assert. Wide pointers are NUL-terminated. - Streams are read in blocks through rdbuf() into the buffer, instead of character by character through the input adapter; eofbit is set at the end, as by the library's stream adapter. - tape/tape_size/tape_cap are now nodes/node_count/node_capacity. - The builder's members get descriptive, m_-prefixed names. - The shallow frame stack is a std::array (in a wrapper with a user-provided constructor: -Weffc++ wants the member initialized, and value-initializing the 64 frames costs up to 30 % on small documents). - The value dispatch macro has no NEXT parameter; the switch is left with break, and the macro is undefined after its last use. - literal() and digits_exceed() deduce the length from the string literal; each literal has one helper; the two `if (!is_float)` blocks are merged. - arena_ensure() sits right before the writes; the decode arena is a std::vector<char>. - The builder records the deepest nesting, and materialize() sizes its stack from it once (growing it only if a subtree is deeper). - Comment fixes (hex4, slow_string, the NUL rule, put_utf8, 0 - m, the arena resize, grow()) and formatting (macro backslashes, digit cases, one power of ten per line, value_t assertion next to is_container()). - build_with's template parameters follow the builder's order. Greg's idea of keeping the source offset in `off` and the arena offset in `next` for escaped strings was measured and not adopted: string access got 7-18 % slower with a select, and 3.5-18 % slower with a branch-free field pick (Apple M-series, clang -O3), parsing was unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Apply the review's wording fixes to the json_view documentation - Copying is disabled because of its cost, not because of two documents borrowing the same buffer; the unique_ptr follows from move-only. - Inputs: streams, FILE*, and wide characters (strings, arrays, pointers) are all supported and owned; point to parse_copy() for owned copies. - The error replay only runs for failing input, builds no values, and is skipped with allow_exceptions = false. - read() writes copied inputs into the previous owned copy. - Views can dangle like std::string_view, and why there is no shared/weak pointer design. - The 4 GiB limit (16-byte nodes, 16-byte scanner steps), streams being read to the end first, which macros apply, why materialize() has no source positions, and wording nits (accept() vs basic_json::accept(), is_binary, is_discarded, see-also lists, size table, "back into"). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Say that json_view converts integers while parsing Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address the review: null inline nodes, trait spelling, and max() in json_view - inline_nodes is null when the document has no inline nodes, instead of pointing past the end of the allocation; shrink_to_fit() no longer passes it to memcpy() for an empty node array - is_byte_char and is_wide_char use conjunction/disjunction, which astyle leaves unspaced - append_stream() uses (std::max) for the count read Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Include materialize.hpp in the view builder test The test calls detail::view::materialize(), which only the single header pulled in; the multiple-header build (JSON_TEST_USING_MULTIPLE_HEADERS) failed with "'materialize' is not a member of detail::view". Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Rename emit() in the view builder for Qt compatibility Qt defines emit as an empty macro, so a Qt header included before json_view.hpp broke the declaration of the builder's emit(), see issue #5865. It is now called push_node(). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not copy an empty run into an unallocated arena memcpy() must not be passed a null pointer even for zero bytes, which arena_run() did for an empty run while the arena was still empty (ci_test_clang_sanitizer: "null pointer passed as argument 1" in builder.hpp). Only the slow path can see such a run. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Exclude the view builder test from Infer's reports Infer's std::unique_ptr model confuses the values that the test's build() helper returns by value and reports USE_AFTER_DELETE for one of them after the destructor of an unrelated temporary (ci_infer, 12 reports, all in tests/src/unit-json_view_builder.cpp). AddressSanitizer is clean on the same tests. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Skip the deleted-overload checks of json_view on MSVC 2015 MSVC 2015 does not treat selecting a deleted function in decltype as a substitution failure, so is_detected reports the deleted parse/parse_copy/accept/read(ptr, len) overloads as callable and the static_asserts in "integer arguments do not compile" fail (AppVeyor VS2015 jobs). Calling them still fails to compile there. Guard the negative checks as unit-delete_deprecated_functions.cpp does. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
4da932f518 |
Make parsing from std::istream and FILE* fast (#5828)
* Make parsing from std::istream and FILE* fast
Squashed onto develop
|
||
|
Niels Lohmann
|
6ba53e63dd |
Speed up array-to-container from_json (#5833)
* Speed up array-to-container from_json Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Hint the end when converting an object to a standard container from_json for map-like containers called emplace() without a hint, which searches the tree for every key. The keys of the source object are sorted, so inserting with the end as the hint is constant time, as it was in 3.12.0 via std::inserter. get<std::map<...>>() of a 25,000 entry object: 4.2 ms -> 3.25 ms (3.12.0: 3.7 ms). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fall back to emplace() in object from_json for containers without emplace_hint() ordered_map has no hinted insert, so the insert(end(), ...) call broke from_json into types holding an ordered_map (regression test #5122). Use emplace_hint() where available and plain emplace() otherwise. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
1c92a3c5de |
Update Google Benchmark to 1.10.0 (#5823)
* Update Google Benchmark to 1.10.0 Bump the pinned release and its archive checksum. Reword the comment and README: the version is fixed for reproducible builds and updated like any other dependency; results stay comparable because both sides of a comparison are built with the same version. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not name the Google Benchmark version in the README Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
65af260117 |
Move instead of deep-copy ordered_json values when an object grows (#5609)
* Move instead of deep-copy ordered_json values when an object grows ordered_map keeps its elements in a std::vector<std::pair<const Key, T>>. With a std::string key, that pair is not nothrow move constructible (the const key has to be copied), so std::vector copies every element when it reallocates. For ordered_json, this deep-copies every member value an object already holds, including whole nested subtrees, on each growth step. Grow the storage in ordered_map instead, copying the keys and moving the values. This happens in two phases, so the strong exception guarantee is kept without try/catch. The first phase may throw, but only touches a temporary buffer: it copies the keys, value-initializes the values, and constructs the new element. The second phase moves the values (noexcept) and swaps the buffers. Because the new element is constructed before any value is moved, arguments that refer to elements of the container stay valid, as with std::vector. Types that cannot take this path keep the std::vector behavior. Parsing into ordered_json (ParseStringOrdered, Apple M1 Max, clang -O3): twitter 3.20 -> 1.70 ms, citm_catalog 7.73 -> 3.67 ms, jeopardy 219 -> 177 ms, canada unchanged. The number of allocations for twitter and citm_catalog drops by two thirds. Also add ParseStringOrdered rows to the benchmarks, and document the growth behavior and the exception safety of ordered_map. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: skip std::pair noexcept assumptions on EDG-based compilers ci_icpc and ci_nvhpc failed to compile unit-ordered_map.cpp: the static assertion that std::pair<const std::string, ordered_json> is not nothrow move-constructible fails there. The EDG front end (Intel icpc 2021.10, NVIDIA nvc++ 25.5) considers the defaulted move constructor of std::pair<const Key, T> noexcept even if copying Key can throw. With these compilers, std::vector already moves such elements itself when it grows, and ordered_map correctly leaves growing to it. The same misjudgement makes std::vector call std::terminate when a key copy throws during growth, so the exception-safety test with throwing_key would abort on these compilers as well. Skip the static assertion and the exception-safety section when __EDG__ is defined. Verified with icpc 2021.10 (-std=gnu++11) and nvc++ 25.5 (C++11 and C++17) on Compiler Explorer: unit-ordered_map and unit-disabled_exceptions build and pass. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
1e101ecac1 |
Add BON8 support (#2998)
* Add BON8 support Add to_bon8/from_bon8 and input_format_t::bon8 for BON8, a binary format that uses the byte values that cannot begin a UTF-8 character as type markers, so strings need no length prefix. It is the most compact of the supported binary formats on the benchmark files. The reader is non-recursive like the other binary readers. A string ends at the first byte that cannot continue it, so the reader hands the one or two bytes it reads past a string back to the value that follows. The writer produces the canonical representation of the specification, except for NFC normalization; its output is identical to that of the reference implementation (HikoGUI) on all files of the test data. The round-trip tests need the .bon8 files of json_test_data 3.2.0. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review comments - Reuse detail::validate_one_utf8 to check strings in to_bon8; the error now names the first byte of the invalid sequence. - Document that to_bon8 leaves bytes in the output adapter on an exception, and that string_open is only an output of write_bon8_marker. - Explain why the pushback buffer of the BON8 reader cannot overflow. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Select the BON8 float prefix by type get_bon8_float_prefix only depends on the type of its argument, so make the type a template parameter instead of passing an unused value. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Rename a test variable that Flawfinder mistakes for read() Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the BON8 CI failures - compare the float in write_bon8_float with number_float_t constants, so GCC does not warn about a float-to-double conversion - mark check_bon8_utf8's context as used when exceptions are disabled - choose the compact float prefix in a helper rather than with nested conditional operators (clang-tidy) - use auto for the cast in the BON8 integer reader (clang-tidy) - write the int32 minimum test values as long long literals (MSVC C4146) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read BON8 strings in bulk from contiguous input - copy the valid UTF-8 of a string in one step when the input is contiguous (twitter.json is read in 1.68 instead of 2.52 ms, jeopardy.json in 196 instead of 297 ms, close to CBOR and MessagePack) - share the new valid_utf8_prefix() with the writer's UTF-8 check, which now skips ASCII 8 bytes at a time - let the fuzzer check that contiguous and stream input give the same value or error, and test both paths in the unit tests - clarify that a second 0xFF after a string is an empty string Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Link the BON8 functions from the other binary format pages Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Name the bulk scan flag after the input, not BON8 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read BSON keys in bulk from contiguous input BSON keys (and array indices) are C-style strings, which were read byte by byte. For contiguous input they are now read up to their \x00-byte in one step, using the same bulk_scan flag as BON8 strings: twitter.json is read in 1.46 instead of 2.01 ms, citm_catalog.json in 2.93 instead of 3.33 ms, jeopardy.json in 182 instead of 207 ms. canada.json, whose keys are almost all one-digit array indices, takes 2 % longer. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the BON8 CI failures of the bulk-read tests - skip the contiguous-versus-stream tests of BON8 strings and BSON keys when exceptions are disabled: they catch the parse errors of invalid input, and without exceptions the library aborts instead - use static_cast for the int64 test value (google-readability-casting) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move the explicit basic_json instantiation into its own test file Linking test-regression3_cpp20 with clang and MinGW failed with "relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata'", as test-regression2 did before #5511. The explicit instantiation of basic_json<> for #4825 compiles every member function, including the BON8 reader and writer, into that object, and it was already close to the limit (2,226,104 bytes on develop, 2,234,960 with BON8; clang -O1, C++20). Give the instantiation a file of its own: unit-regression3 is now 1,594,736 bytes and unit-explicit_instantiation 1,095,064. The new file mentions JSON_HAS_CPP_17 and JSON_HAS_CPP_20 so it keeps being built for the C++17 standard the regression was about. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Convert the bytes of the BON8 test strings explicitly The str() helper constructed a std::string from a byte range, which converts each unsigned char implicitly; -fsanitize=integer reports that for bytes of 0x80 and above (ci_test_clang_sanitizer). Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
ec4bdc398a |
Document the benchmarks, make them build and compare versions again, and pin Google Benchmark (#5556)
* Document the benchmarks, and make them build and compare versions again The benchmark project hasn't configured since #4793: download_test_data.cmake compiles cmake/detect_libcpp_version.cpp relative to CMAKE_SOURCE_DIR, which is tests/benchmarks when that is the top-level project, so try_run fails and so does `make run_benchmarks`. The path is now relative to the module itself, which is the same file for the main build. The Dump benchmark discarded dump()'s result, which is [[nodiscard]] by now; it warned, and left the optimizer free to shorten the loop. The result is now kept with benchmark::DoNotOptimize. A new cache variable, JSON_BENCHMARK_INCLUDE_DIR, names the directory holding the nlohmann/json.hpp to benchmark (single_include by default, as before), so the same benchmarks can be built against two versions and compared. tests/benchmarks/README.md documents what is measured, how to build and run the benchmarks, how to read the output, and how to compare two versions with Google Benchmark's compare.py; it recommends doing so by hand before a release rather than in CI. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Point ci_benchmarks at tests/benchmarks The target has configured ${PROJECT_SOURCE_DIR}/benchmarks since it was added in #2561, but the benchmarks live in tests/benchmarks. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Pin Google Benchmark to release 1.9.5 The benchmarks fetched Google Benchmark's main branch, so two builds on different days could measure with different library code, and CMake 3.30 and later warn that the single-argument FetchContent_Populate() is deprecated. Fetch the 1.9.5 release archive, verified by its SHA-256, with FetchContent_MakeAvailable() instead. That needs CMake 3.14; Google Benchmark itself already needed 3.13. Its -Werror is switched off, so a newer compiler's new warnings cannot break the pinned release, and its install rules are no longer added. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
d2514a46f7 |
Add benchmarks for the binary readers (#5510)
The benchmark suite covered parsing JSON text, dumping, and serializing to CBOR, but only one binary read: FromMsgpack. Nothing measured from_cbor, from_ubjson, from_bjdata or from_bson, so a change to binary_reader.hpp had no baseline to be compared against. Add read benchmarks for every format, in the two shapes that matter: from a contiguous buffer, which is what most callers pass, and from a FILE*, which is what FromMsgpack already measures and which compiles to different code. FromMsgpack itself is left untouched so its numbers stay comparable across releases. The input is derived at setup time by serializing a parsed test file, because the test data repository ships JSON only. The test files are wide and shallow, but the readers' cost is per container, so add three value shapes they do not cover -- deeply nested containers, many sibling containers, and one flat array of numbers -- plus an indefinite-length CBOR string, a form the writer never emits and which therefore has to be assembled by hand. UBJSON and BJData are also captured in their size- and type-annotated form, which the readers handle in a separate code path. BSON requires an object at the top level, so it cannot reuse the array-rooted test files; it is captured on the object-rooted ones, and the shapes are wrapped in an object so every format measures the same value. The setup marks the benchmark as skipped rather than letting the exception escape if that requirement is ever violated. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
471584616f |
Benchmark parsing of pretty-printed JSON (#5456)
Every input in the benchmark corpus is minified or only lightly spaced, so none of them exercise the lexer's whitespace handling. Real-world JSON is frequently indented - configuration files, pretty-printed API responses, anything kept under version control - where the insignificant whitespace can outweigh the data itself. Add a ParseIndented family that re-serializes each document with an indentation and parses that. The content is identical to the matching ParseString row, so the pair isolates the cost of the whitespace alone. Measured here, parsing the indented form costs 16-34% more than the minified form of the same document, which nothing in the suite currently reports. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
515d994acb |
📄 adjust year (#5044)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
54be9b04f0 | 📄 update REUSE (#4960) | ||
|
Niels Lohmann
|
1705bfe914 |
🔖 set version to 3.12.0 (#4727)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
f06604fce0 |
Bump the copyright years (#4606)
* 📄 bump the copyright years Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 📄 bump the copyright years Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 📄 bump the copyright years Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com> |
||
|
Niels Lohmann
|
1b9a9d1f21 |
Update licenses (#4521)
* 📄 update licenses * 📄 update licenses |
||
|
Tianyi Chen
|
935c6eeb5a |
Optimize binary get_number implementation by reading multiple bytes at once (#4391)
* multibyte binary reader * wide_string_input_adapter fallback to get_character Update input_adapters.hpp * Update json.hpp * Add from msgpack test * Test for broken msgpack with stream, address some warnings * Reading binary number from wchar as an error, address warnings * Not casting float to int, it violates strict aliasing rule |
||
|
Niels Lohmann
|
9cca280a4d | JSON for Modern C++ 3.11.3 (#4222) | ||
|
Niels Lohmann
|
f56c6e2e30 | Update documentation for the next release (#4216) | ||
|
Craig Scott
|
fac07e22c5 |
Accept NEW CMake policies up to CMake 3.14 (#4112)
Starting with CMake 3.27, deprecation warnings are issued when asking for policy settings for CMake 3.4 or earlier. The cmake_minimum_required() command accepts a version range, which allows NEW policy settings up to the upper end of that range to be used, but without raising the minimum CMake version above the bottom of that range. This means NEW policy settings will be used where available, without requiring them. This change updates the project's cmake_minimum_required() calls to use a version range to extend the upper policy version to 3.14 where it wasn't already at that version or higher. This prevents the deprecation warning from CMake 3.27, and gives breathing space before a future CMake release will start issuing similar deprecation warnings again. |
||
|
Niels Lohmann
|
836b7beca4 | Fix CI, again (#4083) | ||
|
Niels Lohmann
|
9d69186291 | 🔖 set version to 3.11.2 | ||
|
Niels Lohmann
|
f2020da0dd | 🔖 set version to 3.11.1 | ||
|
Niels Lohmann
|
ce0e13ccea | 🔖 set version to 3.11.0 | ||
|
Niels LohmannandFlorian Albrechtskirchinger
|
527da54dcb |
Use REUSE framework (#3546)
* 📄 add licenses * 👷 add REUSE compliance check * 📝 add badge for REUSE Co-authored-by: Florian Albrechtskirchinger <falbrechtskirchinger@gmail.com> |
||
|
Romain Reignier
|
d4daaa897f |
Optimize output vector adapter write (#3569)
* Add benchmark for cbor serialization and cbor binary data serialization * Use std::vector::insert method instead of std::copy in output_vector_adapter::write_characters This change increases a lot the performance when writing lots of binary data. |
||
|
Niels Lohmann
|
b21c345179 |
Reorganize directories (#3462)
* 🚚 move files * 🚚 rename doc folder to docs * 🚚 rename test folder to tests |