Niels Lohmann
2c7474affc
* docs: add the blog post "How 3.13.0 got faster" A post for the documentation (Home > Blog) on how 3.13.0 became faster: eight themes with the pull requests behind them, the ideas taken from other libraries, what did not work, how the fast paths are tested, and the compile-time cost. Every number comes from a run on 2026-10-27 (3.12.0 against develop plus the open non-draft PRs, Apple M1 Max) or from the linked PR. - docs/mkdocs/docs/home/blog/how_3_13_0_got_faster.md: the post. - docs/mkdocs/scripts/figures/perf_blog.py: draws its 12 figures (overview, eight timeline panels, the SWAR test, the cliffs, json_view throughput) as inline SVGs in the nj-charts style; data/perf_timeline.csv holds the timeline. - tests/benchmarks/history: history.cpp and history.py time one fixed program against the header of every commit that changed it, interleaved, and write one CSV row per commit with its PR. The timeline of the post comes from it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: compare json_document with Glaze, sonic, RapidJSON and Rust parsers Extends the comparison in "How 3.13.0 got faster" from yyjson, simdjson and Boost.JSON to eleven engines: sonic-cpp, RapidJSON and Glaze (generic DOM, lazy_json, reading into structs) in C++, sonic-rs, simd-json and serde_json in Rust. Measured on an Apple M1 Max (Apple clang 21, Rust 1.99) and an x86-64 Haswell server (GCC 13, Clang 18). Every engine must return the same values, string bytes and bit-identical number sums before it is timed. The post now answers whether json_document is the fastest parser: it is the fastest document parser at parse+dump everywhere and at parsing on Apple silicon, ties with simdjson's DOM on x86-64, and loses to simdjson On-Demand and Glaze structs when only a few fields are read. tests/benchmarks/json_view/others holds the programs (bench_all.cpp, the Rust crate with its Cargo.lock), summarize.py and the pinned versions and checksums. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: captions, test-file table and figure fixes for the 3.13.0 post - caption every table and figure - describe what makes twitter.json, citm_catalog.json, canada.json and jeopardy.json hard, and what each timeline panel exercises - split the json_view layout figure into the tree and the index, with the example text above; fix the arrows into the string boxes - arrows and a "less is better" axis label in the cliffs figure - name nlohmann/json in the comparison tables and chart - licenses of the adapted code in the thanks section Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: byline and licenses for the 3.13.0 post - author and date below the title, styled as .nj-byline; its link gets no external-link icon - the license of every library in the thanks section, checked against each project's license files; credit sonic-rs for QUOTE_TAB, which orjson adapted from it Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: dump comparison and split acknowledgments in the 3.13.0 post - a dump workload in bench_all and bench_rs: each engine writes a document parsed beforehand, and every output must read back with bit-identical numbers; an optional argument selects one workload - the post's comparison gains a write table, a throughput chart, a dump row for all three configurations, and the x86-64 cases it loses - the thanks section is split into adapted code and borrowed ideas Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: x86-64 dump numbers with #5908's branch-free token digits The dump of canada.json with GCC on x86-64 lost to yyjson by 0.78 because GCC compiled #5908's token-digit selections as branches;9fad526888fixed that. The post's x86-64 dump and parse+dump rows of canada.json are now measured with it (GCC 0.95, Clang 0.91 against yyjson), theme 5 tells the story, and "How we measured" says which rows include the commit. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Compare JSON libraries by use case in the json_view harness bench_all and bench_rs now time use cases (validate, sum, select, edit, minify, write) instead of library functions. Every library may solve a use case in every way it offers; each variant's result is checked, and three malformed inputs give its validation level (full, no UTF-8, partial). summarize.py reports each library's fastest fully validating variant and a faster one at a lower level, as text and as JSON. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: restructure the 3.13.0 post around themes and use cases - deeper technical detail and code in the eight themes; pull request numbers move to the appendix, timeline steps are labelled by theme - comparison with other libraries by use case, as heatmaps per platform, with validation levels and the caveats next to the results - appendix: test files, all 182 suite rows with a histogram, timeline steps, algorithmic fixes, per-file tables, behaviour changes, setup, pull requests by theme Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Measure each library's fastest modes in the json_view harness Address the points of a review from the other libraries' side: - simdjson 5.0.8 (On-Demand structs, key selectors) and Glaze 9.1.0 - number-skipping modes: yyjson and RapidJSON raw numbers, Boost.JSON number_precision::none - reused parsers and allocators for yyjson, RapidJSON and Boost.JSON - Boost.JSON parse_into, Glaze generic with double numbers, and the same struct types for nlohmann, Glaze, simdjson and serde - minified copies of twitter.json and citm_catalog.json - a fourth probe (trailing garbage) and Glaze "(strict)" variants - a new random variant order in every round, as a variant's time could depend on the one before it by up to 15 % on the Mac - --files for other test files and --conformance for JSONTestSuite, with conformance-rs.sh for the Rust program summarize.py also reports each library's best reused variant. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: rerun the library comparison of the 3.13.0 post with fairer rules The figures and Appendix E use the reruns with each library's fastest modes. Skipping number conversion while reading is now allowed for every library; copying number text is still not chosen when writing. The text now says who set the rules, where libraries that check less win select, how minified files change the picture, and that the reused jeopardy.json claim holds with every library reused. A new section covers 19 held-out files and JSONTestSuite, including where json_document loses, and lists what was not measured. Every comparison number was recomputed; Glaze is credited in the intro. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: measure the 3.13.0 post against develop with #5964 and #5908 The suite (Appendix B, the short-version table, the overview and distribution figures) and the timeline endpoint now measure developd20c4dcd89with #5964 and #5908 merged. The geometric mean stays 2.9x (2.95); Dump/signed_ints and Dump/unsigned_ints gain 1.5x and 2.1x from #5964; GetVectorDouble/floats drops from 10.3x to 7.0x because the old candidate's suite binary happened to compile that loop faster (a standalone build takes 0.97 ms with both headers). The timeline endpoint is the 10-27 value scaled by the ratio of the new to the old candidate, both measured on 10-30, so that it stays comparable with the other points of the timeline. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: add three figures to the 3.13.0 post - short strings: two overlapping 16-byte loads and stores - eight digits at a time: the three multiplications, lane by lane - validate on the 19 held-out files, per file, Mac and x86-64 The prose the figures replace is trimmed. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: measure the 3.13.0 post against develop with all open pull requests The baseline is now develop9c349b1d7cwith every open non-draft pull request merged, including #5968, #5970, #5971, #5975, #5976, #5978, #5979, #5980, #5981 and #5982. The suite was rerun on 11-01 with every step repeated when the load rose above 5 while it ran; the comparison with other libraries and the held-out files were rerun on 10-31. - suite: 3.0x geometric mean (median 2.9x); Appendix B, the overview and the distribution regenerated - timeline endpoint, Appendix D and the cliffs figure updated - the held-out figure compares json_document with the fastest other library - new paragraphs for the pull requests listed above Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: measure the 3.13.0 post against develop with the open pull requests of 11-02 Suite, timeline endpoint, Appendix D and the scalar-run numbers now come from developb2674a4a2fwith #5990, #5991, #5992, #5996, #5997, #5998, #5999 and #6000 merged. The scalar runs are measured against a build without them on develop's code, which no longer checks for a scalar before entering the loop. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: measure the 3.13.0 post against develop with all open pull requests of 11-02 Suite, timeline endpoint, Appendix D and the scalar-run numbers now come from developb2674a4a2fwith #5990-#5992, #5996-#6006 and #6008 merged. With #6008, arrays of one element skip the scalar run, so nested arrays no longer pay for it. The suite ran three rounds per version; the round of 3.12.0 and of 3.13.0 that differed from the other two was dropped, as the setup table says. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: fix the facts the review of the 3.13.0 post found wrong or imprecise Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: re-measure the 3.13.0 post and apply the review Measure everything again against develop with the open pull requests of 11-03 (suite, timeline, comparison on the Mac and on x86-64, compile times, stream gap), add the suite on an Intel MacBook Pro as a check, mark the ways that keep number text as "N" in the use-case charts, move the loop-shape box to theme 5, and apply the style and wording fixes of the review. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: make the comparison of the 3.13.0 post fairer Three reviews of "How it compares" (a maintainer audit of the harness, a methodology review, a reader's red team) led to a full rerun on the Mac and x86-64 with simdjson 5.1.1 and new ways in the harness: - in-place parses with the copy made before the timer ("(copy untimed)", never chosen as a library's best) and json_document::parse_copy(); - RapidJSON through StringStream, and with default precision in validate, which reads no number; - BENCH_ONLY takes several prefixes. The post now gives the copy's real cost (11-14 us for twitter.json on the Mac, not 30), what yyjson gains without it, the run-to-run noise, the LTO check, the single-file losses that were missing, and qualifies the headline claim. Every number in the section and Appendix E is recomputed; the NUL-byte remark moved out of the body. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Format bench_all.cpp with astyle The amalgamation check runs astyle over tests/ and failed on bench_all.cpp. The variant tables keep one variant per line between *INDENT-OFF* and *INDENT-ON*; astyle formats the rest, and the list of use cases becomes a named array so the loop header stays on one line. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Print the history benchmark's results with a stream Flawfinder flagged the printf in history.cpp (CWE-134), although its format string is a constant. The stream prints the same text, so history.py reads it as before. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: address review comments on the performance blog post - fix wording: "are new", "for each input source", "the strings", "the last zero to seven bytes", "by" Danila Kutenin, "the result of that query" - introduce `json_document` where it first gets a detail, say that the `-mbranches` flag is not used, explain why the position and token bookkeeping overloads are empty, and say that a step's percentage is relative to the time before the step and that a median of 4 is the mean of the two middle values - write percentages without a space before `%` in the post and in the price figure (regenerated) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: wrap the prose of the performance blog post at 120 columns Only line breaks change: the rendered page is identical after whitespace normalization. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: blog post: address the second round of review comments Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: blog post: note that Boost.JSON's tree lead comes from its arena The comparison builds Boost.JSON's tree in a monotonic_resource. With its default allocator and precise numbers, boost::json::parse() takes as long as json::parse() on twitter.json, 5% longer on citm_catalog.json and 17% less time on canada.json (M1, C++23 -O3, best of 3 x 30 runs; the arena saves Boost.JSON 28-41%). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: blog post: comparing already passes the depth; copying measured slower without the counter operator== walks the standard containers itself and passes the depth as an argument (#5840). A copy that does the same, inserting element by element, measured 4-16% slower on an M1 Max with Apple clang, so copying keeps the thread_local counter. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: blog post: name every user of the depth counter and every input the document copies The "Bounded recursion" paragraph said only copying and equality for custom containers use the thread_local counter. Converting from another basic_json specialization and the ordering comparisons use it too; equality for the standard containers passes the depth instead. The "Strings stay in the input" bullet named only parse_copy() and an rvalue std::string as owning inputs. Streams, FILE* and other input that is not one contiguous block of bytes are read into an owned buffer as well. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Performance history
The benchmarks in tests/benchmarks compare two versions. The program here answers a different
question: which commit made something faster or slower. history.cpp is one fixed benchmark that uses only API
3.12.0 already has. history.py compiles it against the single_include of every first-parent commit that changed
json.hpp or json_view.hpp, runs all of them interleaved, and writes one CSV row per commit with its pull request.
The timeline in the post How 3.13.0 got faster comes from
it. It is not built by CMake or run by CI.
Workloads
| name | what it does |
|---|---|
parse_twitter, parse_citm, parse_canada |
json::parse() of a std::string (citm_catalog.json as distributed is indented by four spaces) |
parse_citm_compact |
the same with a minified copy of citm_catalog.json |
parse_twitter_ifstream |
json::parse() of an std::ifstream, opening the file included |
accept_twitter |
json::accept() |
view_parse_* |
json_document::parse(); only where the header has json_view.hpp |
dump_twitter, dump_citm, dump_canada, dump4_citm |
dump() and dump(4) |
from_cbor_twitter, to_cbor_twitter |
read and write CBOR |
copy_destroy_citm |
copy a parsed document and destroy the copy |
equal_citm |
copy a parsed document and compare it with == |
Each workload runs in batches of at least 20 ms; the program prints the median of 9 batches in microseconds per call.
Running it
mkdir -p ~/history/data
cp <json_test_data>/nativejson-benchmark/{twitter,citm_catalog,canada}.json ~/history/data/
python3 history.py build --repo ../../.. --work ~/history # about 300 builds, a few minutes in parallel
caffeinate -i python3 history.py run --work ~/history --rounds 4 # about 1 h 45 min on an M1 Max (about 4 s per binary and round)
python3 history.py summarize --work ~/history > history.csv
--extra DIR adds one more header directory, such as an amalgamation of develop with open pull requests merged
in. run waits whenever the 1-minute load average is 4 or more.
Reading the results
A single commit can move a workload by a few percent through code layout alone: the functions it did not touch move to other addresses. Read a change as a step only if it is larger than that and stays. To check one, time the two commits against each other with more rounds, or compare instruction counts.