wt/json
1
0 Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-11-09 16:21:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: add the blog post "How nlohmann/json 3.13.0 got 3× faster" (#6022)

Browse Source 
* docs: add the blog post "How 3.13.0 got faster"

A post for the documentation (Home > Blog) on how 3.13.0 became faster:
eight themes with the pull requests behind them, the ideas taken from other
libraries, what did not work, how the fast paths are tested, and the
compile-time cost. Every number comes from a run on 2026-10-27 (3.12.0 against
develop plus the open non-draft PRs, Apple M1 Max) or from the linked PR.

- docs/mkdocs/docs/home/blog/how_3_13_0_got_faster.md: the post.
- docs/mkdocs/scripts/figures/perf_blog.py: draws its 12 figures (overview,
  eight timeline panels, the SWAR test, the cliffs, json_view throughput) as
  inline SVGs in the nj-charts style; data/perf_timeline.csv holds the
  timeline.
- tests/benchmarks/history: history.cpp and history.py time one fixed program
  against the header of every commit that changed it, interleaved, and write
  one CSV row per commit with its PR. The timeline of the post comes from it.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: compare json_document with Glaze, sonic, RapidJSON and Rust parsers

Extends the comparison in "How 3.13.0 got faster" from yyjson, simdjson and
Boost.JSON to eleven engines: sonic-cpp, RapidJSON and Glaze (generic DOM,
lazy_json, reading into structs) in C++, sonic-rs, simd-json and serde_json in
Rust. Measured on an Apple M1 Max (Apple clang 21, Rust 1.99) and an x86-64
Haswell server (GCC 13, Clang 18). Every engine must return the same values,
string bytes and bit-identical number sums before it is timed.

The post now answers whether json_document is the fastest parser: it is the
fastest document parser at parse+dump everywhere and at parsing on Apple
silicon, ties with simdjson's DOM on x86-64, and loses to simdjson On-Demand
and Glaze structs when only a few fields are read.

tests/benchmarks/json_view/others holds the programs (bench_all.cpp, the Rust
crate with its Cargo.lock), summarize.py and the pinned versions and checksums.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: captions, test-file table and figure fixes for the 3.13.0 post

- caption every table and figure
- describe what makes twitter.json, citm_catalog.json, canada.json and
  jeopardy.json hard, and what each timeline panel exercises
- split the json_view layout figure into the tree and the index, with
  the example text above; fix the arrows into the string boxes
- arrows and a "less is better" axis label in the cliffs figure
- name nlohmann/json in the comparison tables and chart
- licenses of the adapted code in the thanks section

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: byline and licenses for the 3.13.0 post

- author and date below the title, styled as .nj-byline; its link gets
  no external-link icon
- the license of every library in the thanks section, checked against
  each project's license files; credit sonic-rs for QUOTE_TAB, which
  orjson adapted from it

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: dump comparison and split acknowledgments in the 3.13.0 post

- a dump workload in bench_all and bench_rs: each engine writes a
  document parsed beforehand, and every output must read back with
  bit-identical numbers; an optional argument selects one workload
- the post's comparison gains a write table, a throughput chart, a dump
  row for all three configurations, and the x86-64 cases it loses
- the thanks section is split into adapted code and borrowed ideas

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: x86-64 dump numbers with #5908's branch-free token digits

The dump of canada.json with GCC on x86-64 lost to yyjson by 0.78 because
GCC compiled #5908's token-digit selections as branches; 9fad526888 fixed
that. The post's x86-64 dump and parse+dump rows of canada.json are now
measured with it (GCC 0.95, Clang 0.91 against yyjson), theme 5 tells the
story, and "How we measured" says which rows include the commit.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Compare JSON libraries by use case in the json_view harness

bench_all and bench_rs now time use cases (validate, sum, select, edit,
minify, write) instead of library functions. Every library may solve a
use case in every way it offers; each variant's result is checked, and
three malformed inputs give its validation level (full, no UTF-8,
partial). summarize.py reports each library's fastest fully validating
variant and a faster one at a lower level, as text and as JSON.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: restructure the 3.13.0 post around themes and use cases

- deeper technical detail and code in the eight themes; pull request
  numbers move to the appendix, timeline steps are labelled by theme
- comparison with other libraries by use case, as heatmaps per
  platform, with validation levels and the caveats next to the results
- appendix: test files, all 182 suite rows with a histogram, timeline
  steps, algorithmic fixes, per-file tables, behaviour changes, setup,
  pull requests by theme

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Measure each library's fastest modes in the json_view harness

Address the points of a review from the other libraries' side:

- simdjson 5.0.8 (On-Demand structs, key selectors) and Glaze 9.1.0
- number-skipping modes: yyjson and RapidJSON raw numbers, Boost.JSON
  number_precision::none
- reused parsers and allocators for yyjson, RapidJSON and Boost.JSON
- Boost.JSON parse_into, Glaze generic with double numbers, and the
  same struct types for nlohmann, Glaze, simdjson and serde
- minified copies of twitter.json and citm_catalog.json
- a fourth probe (trailing garbage) and Glaze "(strict)" variants
- a new random variant order in every round, as a variant's time could
  depend on the one before it by up to 15 % on the Mac
- --files for other test files and --conformance for JSONTestSuite,
  with conformance-rs.sh for the Rust program

summarize.py also reports each library's best reused variant.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: rerun the library comparison of the 3.13.0 post with fairer rules

The figures and Appendix E use the reruns with each library's fastest
modes. Skipping number conversion while reading is now allowed for
every library; copying number text is still not chosen when writing.

The text now says who set the rules, where libraries that check less
win select, how minified files change the picture, and that the
reused jeopardy.json claim holds with every library reused. A new
section covers 19 held-out files and JSONTestSuite, including where
json_document loses, and lists what was not measured. Every
comparison number was recomputed; Glaze is credited in the intro.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: measure the 3.13.0 post against develop with #5964 and #5908

The suite (Appendix B, the short-version table, the overview and distribution
figures) and the timeline endpoint now measure develop d20c4dcd89 with #5964
and #5908 merged. The geometric mean stays 2.9x (2.95); Dump/signed_ints and
Dump/unsigned_ints gain 1.5x and 2.1x from #5964; GetVectorDouble/floats drops
from 10.3x to 7.0x because the old candidate's suite binary happened to compile
that loop faster (a standalone build takes 0.97 ms with both headers).

The timeline endpoint is the 10-27 value scaled by the ratio of the new to the
old candidate, both measured on 10-30, so that it stays comparable with the
other points of the timeline.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: add three figures to the 3.13.0 post

- short strings: two overlapping 16-byte loads and stores
- eight digits at a time: the three multiplications, lane by lane
- validate on the 19 held-out files, per file, Mac and x86-64

The prose the figures replace is trimmed.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: measure the 3.13.0 post against develop with all open pull requests

The baseline is now develop 9c349b1d7c with every open non-draft pull
request merged, including #5968, #5970, #5971, #5975, #5976, #5978,
#5979, #5980, #5981 and #5982. The suite was rerun on 11-01 with every
step repeated when the load rose above 5 while it ran; the comparison
with other libraries and the held-out files were rerun on 10-31.

- suite: 3.0x geometric mean (median 2.9x); Appendix B, the overview
  and the distribution regenerated
- timeline endpoint, Appendix D and the cliffs figure updated
- the held-out figure compares json_document with the fastest other
  library
- new paragraphs for the pull requests listed above

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: measure the 3.13.0 post against develop with the open pull requests of 11-02

Suite, timeline endpoint, Appendix D and the scalar-run numbers now come from
develop b2674a4a2f with #5990, #5991, #5992, #5996, #5997, #5998, #5999 and
#6000 merged. The scalar runs are measured against a build without them on
develop's code, which no longer checks for a scalar before entering the loop.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: measure the 3.13.0 post against develop with all open pull requests of 11-02

Suite, timeline endpoint, Appendix D and the scalar-run numbers now come from
develop b2674a4a2f with #5990-#5992, #5996-#6006 and #6008 merged. With #6008,
arrays of one element skip the scalar run, so nested arrays no longer pay for
it. The suite ran three rounds per version; the round of 3.12.0 and of 3.13.0
that differed from the other two was dropped, as the setup table says.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: fix the facts the review of the 3.13.0 post found wrong or imprecise

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: re-measure the 3.13.0 post and apply the review

Measure everything again against develop with the open pull requests of
11-03 (suite, timeline, comparison on the Mac and on x86-64, compile
times, stream gap), add the suite on an Intel MacBook Pro as a check,
mark the ways that keep number text as "N" in the use-case charts, move
the loop-shape box to theme 5, and apply the style and wording fixes of
the review.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: make the comparison of the 3.13.0 post fairer

Three reviews of "How it compares" (a maintainer audit of the harness, a
methodology review, a reader's red team) led to a full rerun on the Mac and
x86-64 with simdjson 5.1.1 and new ways in the harness:

- in-place parses with the copy made before the timer ("(copy untimed)",
  never chosen as a library's best) and json_document::parse_copy();
- RapidJSON through StringStream, and with default precision in validate,
  which reads no number;
- BENCH_ONLY takes several prefixes.

The post now gives the copy's real cost (11-14 us for twitter.json on the
Mac, not 30), what yyjson gains without it, the run-to-run noise, the LTO
check, the single-file losses that were missing, and qualifies the headline
claim. Every number in the section and Appendix E is recomputed; the
NUL-byte remark moved out of the body.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Format bench_all.cpp with astyle

The amalgamation check runs astyle over tests/ and failed on bench_all.cpp. The variant tables keep one variant per
line between *INDENT-OFF* and *INDENT-ON*; astyle formats the rest, and the list of use cases becomes a named array so
the loop header stays on one line.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Print the history benchmark's results with a stream

Flawfinder flagged the printf in history.cpp (CWE-134), although its format string is a constant. The stream prints the
same text, so history.py reads it as before.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: address review comments on the performance blog post

- fix wording: "are new", "for each input source", "the strings", "the last zero to seven bytes", "by" Danila
  Kutenin, "the result of that query"
- introduce `json_document` where it first gets a detail, say that the `-mbranches` flag is not used, explain
  why the position and token bookkeeping overloads are empty, and say that a step's percentage is relative to
  the time before the step and that a median of 4 is the mean of the two middle values
- write percentages without a space before `%` in the post and in the price figure (regenerated)

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: wrap the prose of the performance blog post at 120 columns

Only line breaks change: the rendered page is identical after whitespace normalization.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: blog post: address the second round of review comments

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: blog post: note that Boost.JSON's tree lead comes from its arena

The comparison builds Boost.JSON's tree in a monotonic_resource. With its
default allocator and precise numbers, boost::json::parse() takes as long
as json::parse() on twitter.json, 5% longer on citm_catalog.json and 17%
less time on canada.json (M1, C++23 -O3, best of 3 x 30 runs; the arena
saves Boost.JSON 28-41%).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: blog post: comparing already passes the depth; copying measured slower without the counter

operator== walks the standard containers itself and passes the depth as an
argument (#5840). A copy that does the same, inserting element by element,
measured 4-16% slower on an M1 Max with Apple clang, so copying keeps the
thread_local counter.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: blog post: name every user of the depth counter and every input the document copies

The "Bounded recursion" paragraph said only copying and equality for custom
containers use the thread_local counter. Converting from another basic_json
specialization and the ordering comparisons use it too; equality for the
standard containers passes the depth instead.

The "Strings stay in the input" bullet named only parse_copy() and an rvalue
std::string as owning inputs. Streams, FILE* and other input that is not one
contiguous block of bytes are read into an owned buffer as well.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann authored and GitHub committed 2026-11-09 08:11:10 +01:00
1 parent 6343775000
commit 2c7474affc
51 files changed
+73792 -18
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/benchmarks/README.md
+3
View File
@@ -138,6 +138,9 @@ the `View*` benchmarks (in `src/benchmarks_view.cpp`) are only built if the dire
Only benchmarks present in both result files are compared, so for older releases, either filter the benchmarks or
build that release's own `tests/benchmarks` against its own header.
To find out *which commit* changed a number, [`history/`](history/README.md) times one fixed program against the
header of every commit that changed it.
## Getting stable numbers
- Build and run both versions on the same machine with the same compiler and Google Benchmark, one right after the
tests/benchmarks/history/README.md
+43
View File
@@ -0,0 +1,43 @@
# Performance history
The benchmarks in [`tests/benchmarks`](../README.md) compare two versions. The program here answers a different
question: *which commit* made something faster or slower. `history.cpp` is one fixed benchmark that uses only API
3.12.0 already has. `history.py` compiles it against the `single_include` of every first-parent commit that changed
`json.hpp` or `json_view.hpp`, runs all of them interleaved, and writes one CSV row per commit with its pull request.
The timeline in the post [How 3.13.0 got faster](https://json.nlohmann.me/home/blog/how_3_13_0_got_faster/) comes from
it. It is not built by CMake or run by CI.
## Workloads
| name | what it does |
|---|---|
| `parse_twitter`, `parse_citm`, `parse_canada` | `json::parse()` of a `std::string` (`citm_catalog.json` as distributed is indented by four spaces) |
| `parse_citm_compact` | the same with a minified copy of `citm_catalog.json` |
| `parse_twitter_ifstream` | `json::parse()` of an `std::ifstream`, opening the file included |
| `accept_twitter` | `json::accept()` |
| `view_parse_*` | `json_document::parse()`; only where the header has `json_view.hpp` |
| `dump_twitter`, `dump_citm`, `dump_canada`, `dump4_citm` | `dump()` and `dump(4)` |
| `from_cbor_twitter`, `to_cbor_twitter` | read and write CBOR |
| `copy_destroy_citm` | copy a parsed document and destroy the copy |
| `equal_citm` | copy a parsed document and compare it with `==` |
Each workload runs in batches of at least 20 ms; the program prints the median of 9 batches in microseconds per call.
## Running it
```sh
mkdir -p ~/history/data
cp <json_test_data>/nativejson-benchmark/{twitter,citm_catalog,canada}.json ~/history/data/
python3 history.py build --repo ../../.. --work ~/history # about 300 builds, a few minutes in parallel
caffeinate -i python3 history.py run --work ~/history --rounds 4 # about 1 h 45 min on an M1 Max (about 4 s per binary and round)
python3 history.py summarize --work ~/history > history.csv
```
`--extra DIR` adds one more header directory, such as an amalgamation of `develop` with open pull requests merged
in. `run` waits whenever the 1-minute load average is 4 or more.
## Reading the results
A single commit can move a workload by a few percent through code layout alone: the functions it did not touch
move to other addresses. Read a change as a step only if it is larger than that and stays. To check one, time the
two commits against each other with more rounds, or compare instruction counts.
tests/benchmarks/history/history.cpp
+98
View File
@@ -0,0 +1,98 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
// One fixed benchmark, compiled against the json.hpp of every commit that changed it (see README.md).
// It uses only API that 3.12.0 has; the json_document rows exist only where the header has json_view.hpp.
// Usage: history <dir with twitter.json, citm_catalog.json, canada.json and citm_compact.json>
// Prints one line per workload: the name and the median of 9 batches in microseconds per call.
#include <nlohmann/json.hpp>
#if __has_include(<nlohmann/json_view.hpp>) && !defined(HISTORY_NO_VIEW)
#include <nlohmann/json_view.hpp>
#define HISTORY_VIEW 1
#endif
#include <algorithm>
#include <chrono>
#include <fstream>
#include <functional>
#include <iomanip>
#include <iostream>
#include <sstream>
#include <string>
#include <vector>
using nlohmann::json;
static std::string slurp(const std::string& p)
{
std::ifstream f(p, std::ios::binary);
std::stringstream s;
s << f.rdbuf();
return s.str();
}
static volatile std::size_t sink;
static void bench(const char* name, const std::function<std::size_t()>& f)
{
using clk = std::chrono::steady_clock;
sink = f(); // warm up
std::size_t n = 1;
for (;;)
{
auto t0 = clk::now();
for (std::size_t i = 0; i < n; ++i)
{
sink = f();
}
double dt = std::chrono::duration<double>(clk::now() - t0).count();
if (dt > 0.02)
{
break;
}
n *= 2;
}
std::vector<double> v;
for (int b = 0; b < 9; ++b)
{
auto t0 = clk::now();
for (std::size_t i = 0; i < n; ++i)
{
sink = f();
}
v.push_back(std::chrono::duration<double, std::micro>(clk::now() - t0).count() / n);
}
std::sort(v.begin(), v.end());
std::cout << name << ' ' << std::fixed << std::setprecision(3) << v[4] << '\n';
}
int main(int, char** argv)
{
std::string d = argv[1];
const std::string tw = slurp(d + "/twitter.json"), ci = slurp(d + "/citm_catalog.json"),
ca = slurp(d + "/canada.json"), cii = slurp(d + "/citm_compact.json");
const json jtw = json::parse(tw), jci = json::parse(ci), jca = json::parse(ca);
const std::vector<std::uint8_t> cbtw = json::to_cbor(jtw);
bench("parse_twitter", [&] { return json::parse(tw).size(); });
bench("parse_citm", [&] { return json::parse(ci).size(); });
bench("parse_canada", [&] { return json::parse(ca).size(); });
bench("parse_citm_compact", [&] { return json::parse(cii).size(); });
bench("parse_twitter_ifstream", [&] { std::ifstream f(d + "/twitter.json"); return json::parse(f).size(); });
bench("accept_twitter", [&] { return static_cast<std::size_t>(json::accept(tw)); });
#ifdef HISTORY_VIEW
bench("view_parse_twitter", [&] { const auto doc = nlohmann::json_document::parse(tw); return doc.root().size(); });
bench("view_parse_citm", [&] { const auto doc = nlohmann::json_document::parse(ci); return doc.root().size(); });
bench("view_parse_canada", [&] { const auto doc = nlohmann::json_document::parse(ca); return doc.root().size(); });
#endif
bench("dump_twitter", [&] { return jtw.dump().size(); });
bench("dump_citm", [&] { return jci.dump().size(); });
bench("dump_canada", [&] { return jca.dump().size(); });
bench("dump4_citm", [&] { return jci.dump(4).size(); });
bench("from_cbor_twitter", [&] { return json::from_cbor(cbtw).size(); });
bench("to_cbor_twitter", [&] { return json::to_cbor(jtw).size(); });
bench("copy_destroy_citm", [&] { json c = jci; return c.size(); });
bench("equal_citm", [&] { json c = jci; return static_cast<std::size_t>(c == jci); });
}
tests/benchmarks/history/history.py
Executable
+151
View File
@@ -0,0 +1,151 @@
#!/usr/bin/env python3
# __ _____ _____ _____
# __| | __| | | | JSON for Modern C++ (supporting code)
# | | |__ | | | | | | version 3.12.0
# |_____|_____|_____|_|___| https://github.com/nlohmann/json
#
# SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
# SPDX-License-Identifier: MIT
"""Time one fixed benchmark against the json.hpp of every commit that changed it.
Usage:
history.py build --repo REPO --work DIR [--from v3.12.0] [--to origin/develop] [--extra INCLUDE_DIR]
history.py run --work DIR [--rounds 4]
history.py summarize --work DIR [--repo REPO] > history.csv
build: lists the first-parent commits between --from and --to that changed single_include/nlohmann/json.hpp or
json_view.hpp, extracts single_include/ of each (and of --from itself), and compiles history.cpp against it into
DIR/bin/NNN, NNN being the line of the commit in DIR/commits.txt. --extra adds one more header directory at the end,
for example an amalgamation of develop with open pull requests merged in. It also writes the input files to DIR/data
(the nativejson-benchmark files from the test data, which must be in DIR/data already, plus a minified copy of
citm_catalog.json).
run: runs every binary once per round, in a new random order each round, and only while the 1-minute load average is
below 4. Output: DIR/res/NNN-R.txt.
summarize: the median over the rounds per binary and workload, as CSV with the commit, its date and its pull request.
Only the Python 3 standard library is used; a C++17 compiler (c++) and git are needed.
"""
import argparse
import concurrent.futures
import csv
import json
import os
import random
import re
import statistics
import subprocess
import sys
import time
from pathlib import Path
HERE = Path(__file__).resolve().parent
FLAGS = ["-std=c++17", "-O3", "-DNDEBUG", "-w"]
def git(repo, *args):
return subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True, text=True).stdout
def build_one(work, idx, rev, repo):
hdr = work / "hdr" / idx
hdr.mkdir(parents=True, exist_ok=True)
if rev.startswith("dir:"):
subprocess.run(["cp", "-R", rev[4:] + "/nlohmann", str(hdr)], check=True)
else:
archive = subprocess.run(["git", "-C", str(repo), "archive", rev, "single_include/nlohmann"],
check=True, capture_output=True).stdout
subprocess.run(["tar", "-x", "-C", str(hdr), "--strip-components=1"], input=archive, check=True)
out = work / "bin" / idx
cmd = ["c++", *FLAGS, "-I", str(hdr), str(HERE / "history.cpp"), "-o", str(out)]
# json_view.hpp of the first commits that had it may not compile with history.cpp: time json.hpp alone then
if subprocess.run(cmd, capture_output=True).returncode != 0:
if subprocess.run(cmd[:-4] + ["-DHISTORY_NO_VIEW"] + cmd[-4:], capture_output=True).returncode != 0:
return "FAIL %s %s" % (idx, rev)
return None
def build(a):
work, repo = Path(a.work), Path(a.repo)
(work / "bin").mkdir(parents=True, exist_ok=True)
revs = [a.__dict__["from"]] + git(repo, "log", "--first-parent", "--reverse", "--format=%H",
"%s..%s" % (a.__dict__["from"], a.to), "--",
"single_include/nlohmann/json.hpp", "single_include/nlohmann/json_view.hpp").split()
if a.extra:
revs.append("dir:" + str(Path(a.extra).resolve()))
(work / "commits.txt").write_text("\n".join(revs) + "\n")
data = work / "data"
citm = json.loads((data / "citm_catalog.json").read_text(encoding="utf-8"))
(data / "citm_compact.json").write_text(json.dumps(citm, separators=(",", ":"), ensure_ascii=False), encoding="utf-8")
with concurrent.futures.ThreadPoolExecutor(os.cpu_count()) as ex:
for r in ex.map(lambda p: build_one(work, "%03d" % p[0], p[1], repo), enumerate(revs)):
if r:
print(r, file=sys.stderr)
def load():
out = subprocess.run(["sysctl", "-n", "vm.loadavg"], capture_output=True, text=True).stdout
return float(out.split()[1]) if out else os.getloadavg()[0]
def run(a):
work = Path(a.work)
(work / "res").mkdir(exist_ok=True)
bins = sorted(p.name for p in (work / "bin").iterdir())
for r in range(1, a.rounds + 1):
random.shuffle(bins)
for b in bins:
while load() >= 4:
time.sleep(15)
out = subprocess.run([str(work / "bin" / b), str(work / "data")], check=True, capture_output=True, text=True)
(work / "res" / ("%s-%d.txt" % (b, r))).write_text(out.stdout)
print("round %d done" % r, file=sys.stderr)
def summarize(a):
work = Path(a.work)
revs = (work / "commits.txt").read_text().split()
times, names = {}, []
for f in sorted((work / "res").glob("*.txt")):
idx = int(f.name.split("-")[0])
for line in f.read_text().splitlines():
name, us = line.split()
if name not in names:
names.append(name)
times.setdefault((idx, name), []).append(float(us))
w = csv.writer(sys.stdout, lineterminator="\n")
w.writerow(["index", "commit", "date", "pr", "subject"] + names)
for idx, rev in enumerate(revs):
if rev.startswith("dir:"):
sha, date, pr, subject = "candidate", "", "", "develop + open pull requests"
else:
sha, date, subject = git(a.repo, "log", "-1", "--format=%h%x09%cs%x09%s", rev).strip().split("\t")
m = re.search(r"\(#(\d+)\)$", subject)
pr = m.group(1) if m else ""
w.writerow([idx, sha, date, pr, subject] +
["%.3f" % statistics.median(times[(idx, n)]) if (idx, n) in times else "" for n in names])
def main():
p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
sub = p.add_subparsers(dest="cmd", required=True)
b = sub.add_parser("build")
b.add_argument("--repo", required=True)
b.add_argument("--work", required=True)
b.add_argument("--from", default="v3.12.0")
b.add_argument("--to", default="origin/develop")
b.add_argument("--extra")
r = sub.add_parser("run")
r.add_argument("--work", required=True)
r.add_argument("--rounds", type=int, default=4)
s = sub.add_parser("summarize")
s.add_argument("--work", required=True)
s.add_argument("--repo", default=str(HERE.parents[2]))
a = p.parse_args()
{"build": build, "run": run, "summarize": summarize}[a.cmd](a)
if __name__ == "__main__":
main()
tests/benchmarks/json_view/others/README.md
+249
View File
@@ -0,0 +1,249 @@
# json_document against other JSON libraries
[`compare.py`](../compare.py) compares `json_document` with yyjson, simdjson and Boost.JSON. The programs here add
sonic-cpp, RapidJSON and Glaze in C++, and sonic-rs, simd-json and serde_json in Rust, for the post
[How 3.13.0 got faster](https://json.nlohmann.me/home/blog/how_3_13_0_got_faster/#how-it-compares). They are not
built by CMake or run by CI.
## What is measured
The programs compare **use cases**, not library functions. A library may implement a use case in every reasonable way
(another API, SAX events, in-situ parsing, a struct binding); each such way is a *variant*, named
`<library>: <variant>` (`yyjson: insitu (copy)`, `RapidJSON: SAX`, `nlohmann json_document: accept`). The report shows
each library's fastest correct variant. `nlohmann json` (`basic_json`) and `nlohmann json_document` (with
`json_view`) count as two libraries.
| use case | what it does |
|---|---|
| `validate` | is this text valid JSON? returns a bool |
| `sum` | visit every value, convert every number, touch every string and key; sum all numbers in document order |
| `select` | read a few fields of every record (the fields of `bench_view.cpp`, e.g. `id`, `user.screen_name` and `retweet_count` of every status) |
| `edit` | change some values of every record and write the document as compact JSON (see below) |
| `minify` | text in, compact text out |
| `write` | write a document parsed beforehand as compact JSON; only the writing is timed |
The edits:
| file | edit |
|---|---|
| `twitter.json` | for every element of `statuses`: `retweet_count` + 1, erase `entities`, add `"seen": true` |
| status | the same edit on the one status |
| `citm_catalog.json` | for every element of `performances`: `start` + 1, erase `seatMapImage`, add `"seen": true` |
| `canada.json` | `features[0].properties.name` = `"Kanada"`; append `[0.5,0.25]` to the first ring (`geometry.coordinates[0]`) of every feature |
| `jeopardy.json` | for every record: `value` = `null`, erase `air_date`, add `"seen": true` |
The files are `twitter.json`, `citm_catalog.json`, `canada.json` (nativejson-benchmark), `jeopardy.json`, the first
status of `twitter.json` (`status`; written by `json::dump()`, timed in batches of 200), and `twitter_min` and
`citm_catalog_min`: `twitter.json` and `citm_catalog.json` with all whitespace outside strings removed, built in memory at
startup by the same small function in both programs (`strip_ws`; the bytes inside strings are untouched). The minified
files use the select and edit of their originals.
### Input contract
The input is a `const std::string` holding the file, with at least 64 bytes of spare capacity (`reserve`), so libraries
that need padding (simdjson) can use it without a copy. A variant that modifies the input (in-situ parsing: yyjson
`YYJSON_READ_INSITU`, RapidJSON `ParseInsitu`, simd-json) copies it inside the timed call; its name says `(copy)`.
The C++ in-situ variants also run as `(copy untimed)`: the copy is made before the timer starts and freed after it
stops, as for a program that reads a file into a buffer it owns and no longer needs. They are reported but never
chosen as a library's best. `json_document`'s `parse_copy` is the opposite case: a document that owns a copy of its
text.
Each timed call starts from the text and frees everything it built, except variants named `(reused)` or
`(reused allocator)`, which keep a parser, document or allocator across calls (simdjson's reused parsers,
`json_document::read()`, Boost.JSON's reset `parser` and `basic_parser`, yyjson's dynamic allocator, RapidJSON's
user-buffer allocators, simd-json's buffers). These are reported but never chosen as a library's best;
`summarize.py` reports each library's best reused variant separately. In Rust, the input is the file as bytes (`&[u8]`), so every library
has to validate UTF-8 itself.
### Correctness rules
Before timing, every variant runs on the file and must produce the right result:
- `validate`: `true`.
- `sum`: the same number of values, the same number of string and key bytes, and a sum of all numbers that is
bit-identical to `json_document`'s (document order), so every number is converted exactly; variants that visit
objects in another order (`nlohmann json: parse`, simd-json and serde_json values) are compared within 1e-9.
- `select`: the same value as `json_document`. A variant that does not convert numbers (`(raw numbers)`,
`(numbers: none)`) has to convert the numbers it selects exactly itself.
- `edit`, `minify`, `write`: the output, read with `json::parse`, compares equal (`==`) to the expected document (for
`edit`, the edit applied to a `json` value; floats compare exactly, so every float must be written exactly). In
Rust, `serde_json::Value` (with `float_roundtrip`) is the reference.
A variant that gives a wrong result on a file, or crashes on it, is printed as
`excluded: <file> <use case> | <variant> | <reason>` and not timed.
### Validation levels
Except for `write`, every variant is also probed with four malformed inputs derived from each file: the file
truncated by one byte (after trailing whitespace), the file followed by ` ]x` (trailing garbage), a stray `,` before
the first `}` (before the last `]` in a file without objects), and a `0xFF` byte inside a string value (the first in
the second half of the file, or the last one; never a key, so that a variant that accepts the byte still finds the
keys it looks up; a file without string values gets no `0xFF` probe). The probes run in a child process, so a crash is caught. The result is the
variant's *validation level*, printed in its cell (`sonic-cpp: Document [no UTF-8] 380.1us`):
| level | meaning |
|---|---|
| `full` | rejects all four probes |
| `no UTF-8` | rejects the structural probes, accepts the `0xFF` byte |
| `partial` | accepts the truncated input, the trailing garbage or the stray comma, or crashes on any probe |
| `n/a` | `write`: there is no input to validate (counts as `full`) |
What a variant accepts is printed as `level: <file> <use case> | <variant> | <level> | <what>`. A variant's level in
the summary is the worst it got for any file. UTF-8 validation is enabled where a library has a switch (RapidJSON
`kParseValidateEncodingFlag`; the `(no encoding check)` variants leave it off). sonic-cpp 1.0.2 cannot validate
UTF-8 at all; serde_json does not validate UTF-8 in the parts it skips (`IgnoredAny`, unknown struct fields) unless
the input is checked with `std::str::from_utf8` first (the `from_utf8 + ...` variants).
The sum and select reference values are printed as `check ... reference:` lines; `summarize.py` warns if the C++ and
Rust logs disagree.
The four probes do not cover everything a parser can get wrong. `--conformance <dir>` runs every `validate` variant
on every file of [JSONTestSuite](https://github.com/nst/JSONTestSuite)'s `test_parsing` (json_test_data has it as
`nst_json_testsuite2/test_parsing`) and prints `conformance: <file> | <variant> | accepted|rejected|crashed`; a `y_`
file must be accepted, an `n_` file rejected, and an `i_` file may go either way. bench_all runs each call in a child
process (a call that hangs for 60 s counts as crashed); a stack overflow aborts bench_rs, so `conformance-rs.sh`
runs it one file per process and, for a file that aborts, one variant per process (`CONF_VARIANT`). The trailing
garbage probe was added after this check: Glaze reads anything after the value unless `validate_trailing_whitespace`
is set.
### Variants
Level on the Mac in brackets where it is not `full` (`P` = partial, `U` = no UTF-8); every `write` variant is `n/a`.
| library | validate | sum | select | edit | minify | write |
|---|---|---|---|---|---|---|
| nlohmann json_document | `accept`, `parse`, `read (reused)` | `parse`, `parse_copy`, `read (reused)` | `parse`, `parse_copy`, `structs (from_json)`, `read (reused)` | `editable`, `editable (source numbers)` (`json_editable_document`: set/erase, `dump` with `number_format::source`) | `parse + dump`, `parse_copy + dump`, `parse + dump (source numbers)`, `read + dump (reused)` | `dump`, `dump (source numbers)` |
| nlohmann json | `accept`, `parse` | `parse`, `sax` | `parse`, `structs (from_json)` | `parse`, `ordered_json` | `parse + dump` | `dump` |
| yyjson | `read`, `insitu (copy)`, `insitu (copy untimed)`, `read (raw numbers)`, `read (reused allocator)` | `read`, `insitu (copy)`, `insitu (copy untimed)`, `read (reused allocator)` | `read`, `insitu (copy)`, `insitu (copy untimed)`, `read (raw numbers)`, `read (reused allocator)` | `mut copy`, `insitu (copy) + mut copy` (`yyjson_doc_mut_copy`), `insitu (copy untimed) + mut copy` | `read + write`, `insitu (copy) + write`, `insitu (copy untimed) + write`, `read (raw numbers) + write`, `read + write (reused allocator)` | `write` |
| simdjson | `DOM`, `DOM (reused)` | `DOM`, `DOM (reused)`, `On-Demand`, `On-Demand (reused)` | `DOM`, `DOM (reused)`, `On-Demand` [P], `On-Demand (reused)` [P], `On-Demand structs` [P], `On-Demand (key selectors)` [P] | n/a (no mutable document) | `DOM + to_string`, `DOM (reused) + to_string`, `minify` [P] | `to_string` |
| RapidJSON | `SAX` (null handler), `Document`, `insitu (copy)`, `insitu (copy untimed)`, `SAX (string stream)`, `SAX (default precision)`, `SAX (string stream, default precision)`, `SAX (raw numbers)`, `Document (reused allocator)` | `Document`, `SAX`, `insitu (copy)`, `insitu (copy untimed)`, `SAX (string stream)`, `Document (reused allocator)` | `Document`, `insitu (copy)`, `insitu (copy untimed)`, `Document (string stream)`, `Document (raw numbers)`, `Document (reused allocator)` | `Document`, `insitu (copy)`, `insitu (copy untimed)`, `Document (string stream)` | `Document + Writer`, `SAX to Writer`, `SAX to Writer (string stream)`, `SAX to Writer (raw numbers)`, `Document + Writer (reused allocator)` | `Writer` |
| RapidJSON, `(no encoding check)` | the same [U] | the same [U] | the same [U] | the same [U] | the same [U] | |
| sonic-cpp | `Document` [U] | `Document` [U] | `Document` [U] | `Document` [U] | `Document + Serialize` [U] | `Serialize` |
| Glaze | `validate_json`, `generic` [P], `generic (double)` [P], `generic (strict)`, `generic (double, strict)` | `generic` [P], `generic (double)` [P], `generic (strict)`, `generic (double, strict)`, `lazy_json` [P], `lazy_json (string_view)` (x) | `structs` [P], `structs (validate_skipped)`, `generic` [P], `generic (double)` [P], `generic (strict)`, `generic (double, strict)`, `lazy_json` [P], `lazy_json (string_view)` (x) | `generic` [P], `generic (double)` (x), `generic (strict)`, `generic (double, strict)` (x) | `generic` [P], `generic (double)` (x), `generic (strict)`, `generic (double, strict)` (x), `minify_json` [P] | `generic`, `generic (double)` (x), `generic (strict)`, `generic (double, strict)` (x) |
| Boost.JSON | `SAX` (`basic_parser`), `parse`, `SAX (numbers: none)`, `parse (numbers: none)`, `SAX (reused)`, `parser (reused)` | `parse`, `SAX`, `SAX (reused)`, `parser (reused)` | `parse`, `parse (numbers: none)` (x), `parser (reused)`, `parse_into structs` | `parse` | `parse + serialize`, `parser (reused) + serialize` | `serialize` |
| serde_json | `IgnoredAny` [U], `from_utf8 + IgnoredAny`, `Value` | `visitor`, `Value` | `structs` [U], `from_utf8 + structs`, `Value` | `Value` | `Value` | `Value` |
| sonic-rs | `IgnoredAny`, `Value` | `visitor`, `Value` | `structs`, `Value` | `Value` | `Value` | `Value` |
| simd-json | `tape (copy)`, `borrowed (copy)`, `tape (reused)` | `visitor (copy)`, `borrowed (copy)`, `borrowed (reused)` | `structs (copy)`, `borrowed (copy)` | `owned (copy)`, `borrowed (copy)` | `borrowed (copy)` | `borrowed` |
(x): excluded for a wrong result on some file, and so in the summary for every file: `generic (double)` loses the
64-bit ids of `twitter.json` when it writes them; `lazy_json (string_view)` reads the raw bytes of a string, so escaped
strings have the wrong length (`sum`, and the `"` in 19,077 `jeopardy.json` categories in `select`);
`numbers: none` reads every number as 0.
`structs` read into types that declare only the fields `select` uses and hold the same data: nlohmann/json's
`from_json()` of the `NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE` macro (from a `json` value or from a `json_document`'s view)
reads into `nsel`, where the arrays and objects that are only counted (`seatCategories`, the values of `events`,
`coordinates`) are a `count_only` holding their size. Glaze reads into the same types (reflected; `count_only` has a
reader that skips and counts the elements; `error_on_unknown_keys = false`, and with `validate_skipped` to validate
what it skips), and so does simdjson On-Demand (`get<T>()` with `tag_invoke`, fields in document order,
`count_elements()`/`count_fields()`). serde derive reads into the same fields with a `Count` that counts with
`IgnoredAny` (and a `BTreeMap`, as `std::map`). Boost.JSON's `parse_into` has no type that skips any value, so its
counted members only count what it converts into them: `seatCategories` into a counting sequence of an empty described
struct, `coordinates` into three levels of counting sequences down to `double`, and the `events` values into an empty
described struct, so their members are not counted. `key selectors` are simdjson 5.0's `object::for_each<"k1",
...>`, which walks each object once and matches keys with a compile-time perfect hash. `visitor` is a serde
`DeserializeSeed` that sums without building a value; `SAX` and `sax` are the libraries' event interfaces with a
summing (or, for `validate`, an empty) handler. RapidJSON parses with `kParseFullPrecisionFlag` (correctly rounded
floats; without it, sums and written floats are wrong) and Boost.JSON with `number_precision::precise`, as every other
library rounds correctly by default; only `validate`, which reads no number, also has RapidJSON's `(default
precision)`. RapidJSON's `Document::Parse(data, size)` and our `SAX` read through `EncodedInputStream<MemoryStream>`;
`(string stream)` reads the NUL-terminated string through `StringStream`, as `Document::Parse(const char*)` does,
where RapidJSON has a SIMD string scanner (used only without `kParseValidateEncodingFlag`).
Variants named `(raw numbers)` or `(numbers: none)` use a library's mode that validates numbers without converting
them (yyjson `YYJSON_READ_NUMBER_AS_RAW`, RapidJSON `kParseNumbersAsStringsFlag`, Boost.JSON
`number_precision::none`), as `json_document` converts a number only when it is read. In `select` they convert the
selected numbers themselves (`std::from_chars`); Boost.JSON's `none` leaves no number text in its DOM, so it cannot.
yyjson writes a raw number verbatim, so `read (raw numbers) + write` is exact. sonic-cpp 1.0.2 has no parse flags,
and Glaze's generic DOM has no raw number mode (`validate_json` converts no number anyway). `(reused allocator)`:
yyjson's `yyjson_alc_dyn_new()`, RapidJSON's `MemoryPoolAllocator`s over user buffers for the values and the parse
stack; Boost.JSON's `parser (reused)` builds its value in a `monotonic_resource` over a buffer kept across calls. These
buffers grow after a call that needed more, so the timed calls do not allocate. `generic (double)` is `glz::generic`,
which stores every number as a double. The Glaze `(strict)` variants read with `validate_trailing_whitespace` (else
Glaze accepts anything after the value) and write with `escape_control_characters` (else Glaze writes a string's
control characters, e.g. U+0001, unescaped, which is not JSON; `gsoc-2018.json` has some).
| library | version | notes |
|---|---|---|
| `json_document`, `json` | this checkout | |
| yyjson | 0.13.0 | |
| simdjson | 5.1.1 | the single-header files of the tag |
| sonic-cpp | 1.0.2 | |
| RapidJSON | master 24b5e7a8b27f (2025-02-05) | |
| Glaze | 9.1.0 | `generic_u64` and `generic` (owning DOMs, member order kept) |
| Boost.JSON | 1.92.0 | monotonic resource |
| sonic-rs, simd-json, serde_json | see `rust/Cargo.lock` | serde_json with `float_roundtrip` |
SHA-256 of the downloads:
```
55f8a675397c505f9bee6c160e1c26b0085868f185d04d7a7727531fdbff4bd4 glaze-v9.1.0.tar.gz
30edf9e47c417f653546c39295fdb66ba5f89fb37eb3780122c6ee66605e5837 sonic-cpp v1.0.2.tar.gz
2d2601a82d2d3b7e143a3c8d43ef616671391034bc46891a9816b79cf2d3e7a8 rapidjson 24b5e7a8b27f42fa16b96fc70aade9106cf7102f.tar.gz
34e0f62a2bc11ab20d601e8ca1cc2b2079503aa45119a19133d89d19b94a0fae yyjson 0.13.0.tar.gz
f3e5ee5f39051960b5ce65762ee4b6919caf270675eb52240ebd92f21a5bcefd simdjson v5.1.1 singleheader/simdjson.h
3ea5f501cef6c2e16e28095b566feeb56021ca473aaf50aad32102f45d734195 simdjson v5.1.1 singleheader/simdjson.cpp
c4a3b310ddd2472416e091067166b0713be97c63f38c212c484ada022fd296ce boost_1_92_0.tar.gz
```
### Timing
Every variant of a use case runs interleaved in every round, each timed call after an untimed one, and the best
round is reported (the method of `bench_view.cpp`). The order of the variants changes in every round (a shuffle with a
fixed seed; the first round keeps the listed order): on the Mac, a variant's time can depend on which variant ran
before it, by up to 15 %, and a fixed order would give each variant the same neighbour in every round. `jeopardy.json` (50 MB) gets a fifth of the rounds (at least 3,
at most the given number).
## Building and running
All libraries except yyjson and simdjson are used header-only. With the downloads unpacked into `DL`
(`compare.py --download` fetches yyjson and Boost into its build directory, and simdjson 4.6.11, so take simdjson
5.1.1's `singleheader` files from its tag):
```sh
cc -O3 -DNDEBUG -c $DL/yyjson-0.13.0/src/yyjson.c -o yyjson.o
c++ -std=c++17 -O3 -DNDEBUG -c $DL/simdjson-5.1.1/singleheader/simdjson.cpp -o simdjson.o
c++ -std=c++23 -O3 -DNDEBUG -DRAPIDJSON_NEON \
-I ../../../../include -I $DL/yyjson-0.13.0/src -I $DL/simdjson-5.1.1/singleheader -I $DL/boost_1_92_0 \
-I $DL/glaze-9.1.0/include -I $DL/rapidjson/include -I $DL/sonic/include \
bench_all.cpp yyjson.o simdjson.o -o bench_all
./bench_all <json_test_data> 30 > mac-cpp-1.log
cd rust && cargo build --release --locked && ./target/release/bench_rs <json_test_data> 30 > ../mac-rs-1.log
python3 summarize.py mac-cpp-*.log --rust mac-rs-*.log --json mac.json
```
With GCC 16 and Homebrew Clang on macOS, build the three steps with the same flags plus
`--sysroot=$(xcrun --show-sdk-path)` (`gcc-16`, `g++-16`) or `-isysroot $(xcrun --show-sdk-path)`
(`/opt/homebrew/opt/llvm/bin/clang`, `clang++`). The key selectors and simdjson's `tag_invoke` need C++20, so
`bench_all.cpp` needs `-std=c++20` or newer.
Both programs take `<json_test_data> [rounds] [file] [use case]`: `./bench_all <json_test_data> 30 canada edit` times
only the edit of `canada.json`; `""` stands for all files; the file names are `twitter`, `citm_catalog`, `canada`,
`jeopardy`, `status`, `twitter_min` and `citm_catalog_min`. The checks cover only the selected files and use cases.
bench_all uses `fork()` for the probes and the crash check (macOS, Linux).
`--files <dir> [rounds] [file] [use case]` runs `validate`, `sum`, `minify` and `write` (`select` and `edit` are
defined only for the files above) on every `*.json` file of a directory, named by their stem; files under 500 KB are
timed in batches of about 500 KB of input. It is meant for files the libraries were not tuned on, such as the
`jsonexamples` of [simdjson-data](https://github.com/simdjson/simdjson-data) without `twitter.json`,
`citm_catalog.json` and `canada.json`. `summarize.py` takes such logs as well.
On x86-64, use `-march=haswell -DRAPIDJSON_SSE42` instead of `-DRAPIDJSON_NEON` (sonic-cpp needs AVX2), pin the
process to one core, and time one file per process. Glaze 9 needs `std::expected`, which Clang 18 and older cannot
use from GCC 13's standard library (Clang 19 and newer can); `-DBENCH_NO_GLAZE=1` builds without it. For Rust, build
with `RUSTFLAGS="-C target-cpu=haswell"` to match `-march=haswell`.
## Reading the results
A log line holds the file, the use case, and one cell per variant: `| <library>: <variant> [<level>] <time>us
(<ratio>x)`, the ratio relative to the first variant of the line. `summarize.py` prints, for every use case and file,
(a) per library the best `full` variant and, if faster, the best variant at any level, with their ratios to the best
`full` `nlohmann json_document` variant (`(reused)` and `(reused allocator)` variants are never a library's best; a
`reused:` line below the library shows its best reused variant in the same way), (b) all variants with their levels
and times, and (c) what the variants below `full` accept, and the excluded variants. `--json` writes
`{use case: {file: {library: {"best": {"variant", "us", "level"} or null, "best_any": {...}, "best_reused": {...} or
null, "best_reused_any": {...} or null, "variants": {variant: {"us", "level"}}, "excluded": {...}}}}}` for the chart
scripts (`best_reused`: the best full reused variant; `best_reused_any`: at any level). Several logs are combined by taking each cell's best
time; a variant excluded for one file is left out for every file of that use case.
On a shared machine, differences of a few percent are noise: run twice and compare.
tests/benchmarks/json_view/others/bench_all.cpp
+3562
View File
File diff suppressed because it is too large. Load diff
tests/benchmarks/json_view/others/conformance-rs.sh
Executable
+14
View File
@@ -0,0 +1,14 @@
#!/bin/bash
# the Rust conformance run, one file per process; a file that aborts the process (stack overflow) is rerun one
# variant per process, and a variant that aborts is printed as crashed
B=$1; D=$2
VARS=$($B --conformance $D 1 y_structure_lonely_int.json | sed -n 's/^conformance: [^|]* | \(.*\) | [a-z]*$/\1/p')
for f in $(ls $D | grep '\.json$'); do
out=$($B --conformance $D 1 "$f" 2>/dev/null)
if [ $? = 0 ]; then echo "$out" | grep '^conformance:'; continue; fi
while IFS= read -r v; do
o=$(CONF_VARIANT="$v" $B --conformance $D 1 "$f" 2>/dev/null)
if [ $? = 0 ]; then echo "$o" | grep '^conformance:'; else echo "conformance: $f | $v | crashed"; fi
done <<< "$VARS"
done
echo "done: conformance"
tests/benchmarks/json_view/others/rust/Cargo.lock
Generated
+627
View File
@@ -0,0 +1,627 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "ahash"
version = "0.8.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75"
dependencies = [
"cfg-if",
"getrandom",
"once_cell",
"version_check",
"zerocopy",
]
[[package]]
name = "allocator-api2"
version = "0.2.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923"
[[package]]
name = "autocfg"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
[[package]]
name = "bench_rs"
version = "0.1.0"
dependencies = [
"serde",
"serde_json",
"simd-json",
"sonic-rs",
]
[[package]]
name = "bumpalo"
version = "3.20.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649"
[[package]]
name = "bytes"
version = "1.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04"
[[package]]
name = "cfg-if"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
[[package]]
name = "equivalent"
version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "00d174d5400e5e8fd687ad1049e2f578285fa914201b1af7e8b112a4546bd826"
[[package]]
name = "faststr"
version = "0.2.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ca7d44d22004409a61c393afb3369c8f7bb74abcae49fe249ee01dcc3002113"
dependencies = [
"bytes",
"rkyv",
"serde",
"simdutf8",
]
[[package]]
name = "float-cmp"
version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b09cf3155332e944990140d967ff5eceb70df778b34f77d8075db46e4704e6d8"
dependencies = [
"num-traits",
]
[[package]]
name = "foldhash"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb"
[[package]]
name = "futures-core"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e"
[[package]]
name = "futures-task"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cd417de3d1d015fc3bfd2b1ea46dfc7bab72ef86f1cc7cc9c78e728b34a6d1fd"
[[package]]
name = "futures-util"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc"
dependencies = [
"futures-core",
"futures-task",
"pin-project-lite",
"slab",
]
[[package]]
name = "getrandom"
version = "0.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd"
dependencies = [
"cfg-if",
"libc",
"r-efi",
"wasip2",
]
[[package]]
name = "halfbrown"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c7ed2f2edad8a14c8186b847909a41fbb9c3eafa44f88bd891114ed5019da09"
dependencies = [
"hashbrown 0.16.1",
"serde",
]
[[package]]
name = "hashbrown"
version = "0.16.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100"
dependencies = [
"allocator-api2",
"equivalent",
"foldhash",
]
[[package]]
name = "hashbrown"
version = "0.17.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a"
[[package]]
name = "indexmap"
version = "2.14.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cc4e190f5d26ca7051642629da2c52fc03bde85a03197c99408dcd291734c855"
dependencies = [
"equivalent",
"hashbrown 0.17.1",
]
[[package]]
name = "itoa"
version = "1.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
[[package]]
name = "js-sys"
version = "0.3.106"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7883d941dae510fb2d978fc3fe018c71c9e2892fd38854de3e8b92c2e5ad9cc5"
dependencies = [
"cfg-if",
"futures-util",
"wasm-bindgen",
]
[[package]]
name = "libc"
version = "0.2.191"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "65cd87c9a6d60536580b316ad985726104730a5151c99845f2490428ef91240a"
[[package]]
name = "memchr"
version = "2.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
[[package]]
name = "munge"
version = "0.4.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5e17401f259eba956ca16491461b6e8f72913a0a114e39736ce404410f915a0c"
dependencies = [
"munge_macro",
]
[[package]]
name = "munge_macro"
version = "0.4.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4568f25ccbd45ab5d5603dc34318c1ec56b117531781260002151b8530a9f931"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]]
name = "num-traits"
version = "0.2.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841"
dependencies = [
"autocfg",
]
[[package]]
name = "once_cell"
version = "1.21.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
[[package]]
name = "pin-project-lite"
version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd"
[[package]]
name = "proc-macro2"
version = "1.0.107"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9"
dependencies = [
"unicode-ident",
]
[[package]]
name = "ptr_meta"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "743da816b98c921cdbe8628ef7381b76f25ecf4da599fc80aca90eae7ef70cc0"
dependencies = [
"ptr_meta_derive",
]
[[package]]
name = "ptr_meta_derive"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1c8d9ca532f185d5d4db7a7c9d51420b452168ea1c2b913953281bd6fe1fcbd0"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.9",
]
[[package]]
name = "quote"
version = "1.0.47"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001"
dependencies = [
"proc-macro2",
]
[[package]]
name = "r-efi"
version = "5.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f"
[[package]]
name = "rancor"
version = "0.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b534442d0fcdb55d66f373d9cac6d33b6293a2335bc2136dbd06ce0e87d2572"
dependencies = [
"ptr_meta",
]
[[package]]
name = "ref-cast"
version = "1.0.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7e440fb4e4b4147295338efb76001ab9e4efc0e5839df2c47fc5ac2381d365c3"
dependencies = [
"ref-cast-impl",
]
[[package]]
name = "ref-cast-impl"
version = "1.0.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92ecd8964f8453721699a1ed72037b0db49ce2f5a5138486ee89bed6f67cdf3a"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.9",
]
[[package]]
name = "rend"
version = "0.5.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "663ba70707f96e871406fe10d68128412e619b06d1d47cb91c3a4c6501176240"
[[package]]
name = "rkyv"
version = "0.8.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d9776093b7ca170454ab1406954f7b7d97a57c51dc6c0642957fb2ef25c2d399"
dependencies = [
"bytes",
"hashbrown 0.17.1",
"indexmap",
"munge",
"ptr_meta",
"rancor",
"rend",
"rkyv_derive",
"tinyvec",
"uuid",
]
[[package]]
name = "rkyv_derive"
version = "0.8.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1c25ef604ac7dd839d44d64648952ea23c97866f124ff671b0ed2cf3ad9bb06e"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.9",
]
[[package]]
name = "rustversion"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f"
[[package]]
name = "ryu"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
[[package]]
name = "serde"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
dependencies = [
"serde_core",
"serde_derive",
]
[[package]]
name = "serde_core"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.9",
]
[[package]]
name = "serde_json"
version = "1.0.154"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e7e9cc8b1b85264074fbcc02a88680c4096b1e47df8f739dceb03bf482f04bd6"
dependencies = [
"itoa",
"memchr",
"serde",
"serde_core",
"zmij",
]
[[package]]
name = "simd-json"
version = "0.19.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e8f2e4d523646f7da6dde3983bd18ede0babc21c258ab708b3dd93ea0259719d"
dependencies = [
"halfbrown",
"serde",
"serde_json",
"simdutf8",
"value-trait",
]
[[package]]
name = "simdutf8"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e"
[[package]]
name = "slab"
version = "0.4.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"
[[package]]
name = "sonic-number"
version = "0.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d7c350a05f295f922fc153a3af8b443c10db70904a9291a624ddb6058c6110d0"
dependencies = [
"cfg-if",
]
[[package]]
name = "sonic-rs"
version = "0.5.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "756586caeeb21126a57c17d1ef2b9386e2613ba784aae0f27104f5e9982335ec"
dependencies = [
"ahash",
"bumpalo",
"bytes",
"cfg-if",
"faststr",
"itoa",
"ref-cast",
"serde",
"simdutf8",
"sonic-number",
"sonic-simd",
"thiserror",
"zmij",
]
[[package]]
name = "sonic-simd"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f99e664ecd2d85a68c87e3c7a3cfe691f647ea9e835de984aba4d54a41f817d4"
dependencies = [
"cfg-if",
]
[[package]]
name = "syn"
version = "2.0.119"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "syn"
version = "3.0.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d78c8dee4c7bf0e14673097256fed6142ce9d3b85a408189d07482442145823b"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "thiserror"
version = "2.0.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09e52cb86a36cede5cb101bf8908837b3e4c6e5e59fe7fd85c23fb56200d189e"
dependencies = [
"thiserror-impl",
]
[[package]]
name = "thiserror-impl"
version = "2.0.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fe5197923287db20a58125f0bc85c062f7f2c892de97b18c356f9efb14b28524"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.9",
]
[[package]]
name = "tinyvec"
version = "1.13.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fd3ca314f692efd6c868f8408f53fe444634a845f96c028b97d35f6a1f79f0ee"
[[package]]
name = "unicode-ident"
version = "1.0.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a2c754d6c33795a1c324727428e5a7dedb5b06195f9890bdbcba760d3e246563"
[[package]]
name = "uuid"
version = "1.29.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4a0e2d11620c50695fe12bbbfde96b3477fe606e7284c22d505c12ce51214514"
dependencies = [
"js-sys",
"wasm-bindgen",
]
[[package]]
name = "value-trait"
version = "0.12.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "545a3d7089f11f1741d12963c057adf9443fc24325110a6a6daf582e825fd51b"
dependencies = [
"float-cmp",
"halfbrown",
"itoa",
"ryu",
]
[[package]]
name = "version_check"
version = "0.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
[[package]]
name = "wasip2"
version = "1.0.4+wasi-0.2.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b67efb37e106e55ce722a510d6b5f9c17f083e5fc79afc2badeb12cc313d9487"
dependencies = [
"wit-bindgen",
]
[[package]]
name = "wasm-bindgen"
version = "0.2.129"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9bb54f33acc68fd454578d9820b0bde1a1a3d17aa17bb7b6595806d02886d409"
dependencies = [
"cfg-if",
"once_cell",
"rustversion",
"wasm-bindgen-macro",
"wasm-bindgen-shared",
]
[[package]]
name = "wasm-bindgen-macro"
version = "0.2.129"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2e29d0c35b16e224a7eeb5cd2d25e3e1968fbd65604117b44d3b789d00ee8535"
dependencies = [
"quote",
"wasm-bindgen-macro-support",
]
[[package]]
name = "wasm-bindgen-macro-support"
version = "0.2.129"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6f501a8bc3719dba86ef8ae4728879c08001bea749eb1333ac5b91e040e2a6b7"
dependencies = [
"bumpalo",
"proc-macro2",
"quote",
"syn 3.0.9",
"wasm-bindgen-shared",
]
[[package]]
name = "wasm-bindgen-shared"
version = "0.2.129"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "23f0c9c52aa7cd7d77769a4cfe2a9adb1b331f489a41d912ce14513d5ab995c6"
dependencies = [
"unicode-ident",
]
[[package]]
name = "wit-bindgen"
version = "0.57.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e"
[[package]]
name = "zerocopy"
version = "0.8.63"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e5fe1f8f1b06191a00962174c61aa5005e0bb391a6d80d07e24d115c01a92ed8"
dependencies = [
"zerocopy-derive",
]
[[package]]
name = "zerocopy-derive"
version = "0.8.63"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "863ad3ac83293fb4d740aedbfdc9240dd8d1a50c1099acd76ce80ce7c7230c7f"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]]
name = "zmij"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
tests/benchmarks/json_view/others/rust/Cargo.toml
+15
View File
@@ -0,0 +1,15 @@
[package]
name = "bench_rs"
version = "0.1.0"
edition = "2021"
[dependencies]
simd-json = "*"
sonic-rs = "*"
serde = { version = "1", features = ["derive"] }
serde_json = { version = "*", features = ["float_roundtrip"] }
[profile.release]
opt-level = 3
lto = "fat"
codegen-units = 1
tests/benchmarks/json_view/others/rust/src/main.rs
+968
View File
@@ -0,0 +1,968 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
// The Rust counterpart of bench_all.cpp: the same use cases (validate, sum, select, edit, minify, write), files,
// checks (right result; validation level from malformed-input probes) and timing method (every variant interleaved in every round, each
// timed call after an untimed one, best round); see README.md. The input is the file as bytes (&[u8]); simd-json
// parses in place, so its variants copy the input inside the timed call.
//
// Usage: bench_rs <json_test_data dir> [rounds] [document] [use case]
use serde::de::{DeserializeOwned, DeserializeSeed, IgnoredAny, MapAccess, SeqAccess, Visitor};
use simd_json::prelude::*;
use sonic_rs::{JsonContainerTrait, JsonValueMutTrait, JsonValueTrait};
use std::cell::RefCell;
use std::panic::{catch_unwind, AssertUnwindSafe};
use std::time::Instant;
#[derive(Default, Debug, Clone, Copy)]
struct Stats {
num: f64,
str_: usize,
nodes: usize,
}
// ---- sum: walk a DOM, visit every value, convert every number, touch every string and key
fn walk_simd(v: &simd_json::BorrowedValue, st: &mut Stats) {
st.nodes += 1;
if let Some(o) = v.as_object() {
for (k, x) in o.iter() {
st.str_ += k.len();
walk_simd(x, st);
}
} else if let Some(a) = v.as_array() {
for x in a.iter() {
walk_simd(x, st);
}
} else if let Some(s) = v.as_str() {
st.str_ += s.len();
} else if v.is_u64() {
st.num += v.as_u64().unwrap() as f64;
} else if v.is_i64() {
st.num += v.as_i64().unwrap() as f64;
} else if v.is_f64() {
st.num += v.as_f64().unwrap();
}
}
fn walk_sonic(v: &sonic_rs::Value, st: &mut Stats) {
st.nodes += 1;
if let Some(o) = v.as_object() {
for (k, x) in o.iter() {
st.str_ += k.len();
walk_sonic(x, st);
}
} else if let Some(a) = v.as_array() {
for x in a.iter() {
walk_sonic(x, st);
}
} else if let Some(s) = v.as_str() {
st.str_ += s.len();
} else if v.is_number() {
if let Some(u) = v.as_u64() {
st.num += u as f64;
} else if let Some(i) = v.as_i64() {
st.num += i as f64;
} else {
st.num += v.as_f64().unwrap();
}
}
}
fn walk_serde(v: &serde_json::Value, st: &mut Stats) {
st.nodes += 1;
match v {
serde_json::Value::Object(o) => {
for (k, x) in o.iter() {
st.str_ += k.len();
walk_serde(x, st);
}
}
serde_json::Value::Array(a) => {
for x in a.iter() {
walk_serde(x, st);
}
}
serde_json::Value::String(s) => st.str_ += s.len(),
serde_json::Value::Number(n) => {
if let Some(u) = n.as_u64() {
st.num += u as f64;
} else if let Some(i) = n.as_i64() {
st.num += i as f64;
} else {
st.num += n.as_f64().unwrap();
}
}
_ => {}
}
}
// the same visit without a DOM: a serde visitor, driven by any serde deserializer, in document order
struct Sum<'a>(&'a mut Stats);
struct Key<'a>(&'a mut Stats);
impl<'de> DeserializeSeed<'de> for Sum<'_> {
type Value = ();
fn deserialize<D: serde::Deserializer<'de>>(self, d: D) -> Result<(), D::Error> {
d.deserialize_any(self)
}
}
impl<'de> Visitor<'de> for Sum<'_> {
type Value = ();
fn expecting(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
f.write_str("a JSON value")
}
fn visit_unit<E: serde::de::Error>(self) -> Result<(), E> {
self.0.nodes += 1;
Ok(())
}
fn visit_bool<E: serde::de::Error>(self, _: bool) -> Result<(), E> {
self.0.nodes += 1;
Ok(())
}
fn visit_i64<E: serde::de::Error>(self, v: i64) -> Result<(), E> {
self.0.nodes += 1;
self.0.num += v as f64;
Ok(())
}
fn visit_u64<E: serde::de::Error>(self, v: u64) -> Result<(), E> {
self.0.nodes += 1;
self.0.num += v as f64;
Ok(())
}
fn visit_f64<E: serde::de::Error>(self, v: f64) -> Result<(), E> {
self.0.nodes += 1;
self.0.num += v;
Ok(())
}
fn visit_str<E: serde::de::Error>(self, v: &str) -> Result<(), E> {
self.0.nodes += 1;
self.0.str_ += v.len();
Ok(())
}
fn visit_seq<A: SeqAccess<'de>>(self, mut a: A) -> Result<(), A::Error> {
let st = self.0;
st.nodes += 1;
while a.next_element_seed(Sum(&mut *st))?.is_some() {}
Ok(())
}
fn visit_map<A: MapAccess<'de>>(self, mut a: A) -> Result<(), A::Error> {
let st = self.0;
st.nodes += 1;
while a.next_key_seed(Key(&mut *st))?.is_some() {
a.next_value_seed(Sum(&mut *st))?;
}
Ok(())
}
}
impl<'de> DeserializeSeed<'de> for Key<'_> {
type Value = ();
fn deserialize<D: serde::Deserializer<'de>>(self, d: D) -> Result<(), D::Error> {
d.deserialize_str(self)
}
}
impl<'de> Visitor<'de> for Key<'_> {
type Value = ();
fn expecting(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
f.write_str("a key")
}
fn visit_str<E: serde::de::Error>(self, v: &str) -> Result<(), E> {
self.0.str_ += v.len();
Ok(())
}
}
fn sum_serde(s: &[u8], st: &mut Stats) -> bool {
let mut de = serde_json::Deserializer::from_slice(s);
Sum(st).deserialize(&mut de).is_ok() && de.end().is_ok()
}
fn sum_sonic(s: &[u8], st: &mut Stats) -> bool {
let mut de = sonic_rs::Deserializer::from_slice(s);
Sum(st).deserialize(&mut de).is_ok() && de.end().is_ok()
}
fn sum_simd(s: &[u8], st: &mut Stats) -> bool {
let mut b = s.to_vec();
match simd_json::Deserializer::from_slice(&mut b) {
Ok(mut de) => Sum(st).deserialize(&mut de).is_ok(),
Err(_) => false,
}
}
// ---- select: read a few fields of every record (the fields of bench_view.cpp)
// on a DOM; serde_json's is the reference
macro_rules! pick_value {
($fname:ident, $t:ty) => {
fn $fname(name: &str, r: &$t) -> f64 {
fn size(v: &$t) -> usize {
if let Some(a) = v.as_array() {
a.len()
} else {
v.as_object().unwrap().len()
}
}
fn status(acc: &mut f64, s: &$t) {
*acc += s["id"].as_u64().unwrap() as f64;
*acc += s["user"]["screen_name"].as_str().unwrap().len() as f64;
*acc += s["retweet_count"].as_i64().unwrap() as f64;
}
let mut acc = 0.0;
match name {
"twitter" => {
for s in r["statuses"].as_array().unwrap().iter() {
status(&mut acc, s);
}
}
"status" => status(&mut acc, r),
"citm_catalog" => {
for p in r["performances"].as_array().unwrap().iter() {
acc += (p["id"].as_u64().unwrap() + p["eventId"].as_u64().unwrap() + size(&p["seatCategories"]) as u64) as f64;
}
acc += size(&r["events"]) as f64;
}
"canada" => {
let g = &r["features"][0]["geometry"];
acc += (r["type"].as_str().unwrap().len() + g["type"].as_str().unwrap().len() + size(&g["coordinates"])) as f64;
}
"jeopardy" => {
for q in r.as_array().unwrap().iter() {
acc += if q["round"].as_str().unwrap() == "Final Jeopardy!" { 1.0 } else { 0.0 };
acc += q["category"].as_str().unwrap().len() as f64;
}
}
_ => {}
}
acc
}
};
}
pick_value!(pick_serde, serde_json::Value);
pick_value!(pick_sonic, sonic_rs::Value);
pick_value!(pick_simd, simd_json::BorrowedValue<'_>);
// into structs that declare only the fields the selection uses, the same data as nlohmann/json's from_json() structs in
// bench_all.cpp (nsel); unknown keys are skipped; Count counts the elements of an array or the members of an object
// (nlohmann's count_only)
mod sel {
use serde::de::{Deserializer, IgnoredAny, MapAccess, SeqAccess, Visitor};
use serde::Deserialize;
use std::collections::BTreeMap;
pub struct Count(pub usize);
impl<'de> Deserialize<'de> for Count {
fn deserialize<D: Deserializer<'de>>(d: D) -> Result<Count, D::Error> {
struct V;
impl<'de> Visitor<'de> for V {
type Value = Count;
fn expecting(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
f.write_str("an array or an object")
}
fn visit_seq<A: SeqAccess<'de>>(self, mut a: A) -> Result<Count, A::Error> {
let mut n = 0;
while a.next_element::<IgnoredAny>()?.is_some() {
n += 1;
}
Ok(Count(n))
}
fn visit_map<A: MapAccess<'de>>(self, mut a: A) -> Result<Count, A::Error> {
let mut n = 0;
while a.next_entry::<IgnoredAny, IgnoredAny>()?.is_some() {
n += 1;
}
Ok(Count(n))
}
}
d.deserialize_any(V)
}
}
#[derive(Deserialize)]
pub struct User {
pub screen_name: String,
}
#[derive(Deserialize)]
pub struct Status {
pub id: u64,
pub user: User,
pub retweet_count: i64,
}
#[derive(Deserialize)]
pub struct Twitter {
pub statuses: Vec<Status>,
}
#[derive(Deserialize)]
pub struct Performance {
pub id: u64,
#[serde(rename = "eventId")]
pub event_id: u64,
#[serde(rename = "seatCategories")]
pub seat_categories: Count,
}
#[derive(Deserialize)]
pub struct Citm {
pub events: BTreeMap<String, Count>,
pub performances: Vec<Performance>,
}
#[derive(Deserialize)]
pub struct Geometry {
#[serde(rename = "type")]
pub type_: String,
pub coordinates: Count,
}
#[derive(Deserialize)]
pub struct Feature {
pub geometry: Geometry,
}
#[derive(Deserialize)]
pub struct Canada {
#[serde(rename = "type")]
pub type_: String,
pub features: Vec<Feature>,
}
#[derive(Deserialize)]
pub struct Question {
pub category: String,
pub round: String,
}
}
trait De {
fn de<T: DeserializeOwned>(s: &[u8]) -> Option<T>;
}
struct SerdeJson;
struct SerdeJsonStr; // checks UTF-8 with std::str::from_utf8 first, then reads from the &str
struct SonicRs;
struct SimdJson;
impl De for SerdeJson {
fn de<T: DeserializeOwned>(s: &[u8]) -> Option<T> {
serde_json::from_slice(s).ok()
}
}
impl De for SerdeJsonStr {
fn de<T: DeserializeOwned>(s: &[u8]) -> Option<T> {
serde_json::from_str(std::str::from_utf8(s).ok()?).ok()
}
}
impl De for SonicRs {
fn de<T: DeserializeOwned>(s: &[u8]) -> Option<T> {
sonic_rs::from_slice(s).ok()
}
}
impl De for SimdJson {
fn de<T: DeserializeOwned>(s: &[u8]) -> Option<T> {
let mut b = s.to_vec();
simd_json::serde::from_slice(&mut b).ok()
}
}
fn pick_structs<D: De>(name: &str, s: &[u8]) -> Option<f64> {
fn status(acc: &mut f64, x: &sel::Status) {
*acc += x.id as f64;
*acc += x.user.screen_name.len() as f64;
*acc += x.retweet_count as f64;
}
let mut acc = 0.0;
match name {
"twitter" => {
let t: sel::Twitter = D::de(s)?;
for x in &t.statuses {
status(&mut acc, x);
}
}
"status" => {
let x: sel::Status = D::de(s)?;
status(&mut acc, &x);
}
"citm_catalog" => {
let c: sel::Citm = D::de(s)?;
for p in &c.performances {
acc += (p.id + p.event_id + p.seat_categories.0 as u64) as f64;
}
acc += c.events.len() as f64;
}
"canada" => {
let c: sel::Canada = D::de(s)?;
let g = &c.features[0].geometry;
acc += (c.type_.len() + g.type_.len() + g.coordinates.0) as f64;
}
"jeopardy" => {
let q: Vec<sel::Question> = D::de(s)?;
for x in &q {
acc += if x.round == "Final Jeopardy!" { 1.0 } else { 0.0 };
acc += x.category.len() as f64;
}
}
_ => {}
}
Some(acc)
}
// ---- edit: the same edits as bench_all.cpp, then write compact JSON
fn edit_serde(name: &str, r: &mut serde_json::Value) {
fn status(s: &mut serde_json::Value) {
let o = s.as_object_mut().unwrap();
let rc = o["retweet_count"].as_i64().unwrap();
o.insert("retweet_count".to_string(), (rc + 1).into());
o.remove("entities");
o.insert("seen".to_string(), true.into());
}
match name {
"twitter" => {
for s in r["statuses"].as_array_mut().unwrap() {
status(s);
}
}
"status" => status(r),
"citm_catalog" => {
for p in r["performances"].as_array_mut().unwrap() {
let o = p.as_object_mut().unwrap();
let start = o["start"].as_i64().unwrap();
o.insert("start".to_string(), (start + 1).into());
o.remove("seatMapImage");
o.insert("seen".to_string(), true.into());
}
}
"canada" => {
r["features"][0]["properties"]["name"] = "Kanada".into();
for f in r["features"].as_array_mut().unwrap() {
f["geometry"]["coordinates"][0].as_array_mut().unwrap().push(serde_json::json!([0.5, 0.25]));
}
}
"jeopardy" => {
for q in r.as_array_mut().unwrap() {
let o = q.as_object_mut().unwrap();
o.insert("value".to_string(), serde_json::Value::Null);
o.remove("air_date");
o.insert("seen".to_string(), true.into());
}
}
_ => {}
}
}
fn edit_sonic(name: &str, r: &mut sonic_rs::Value) {
fn status(s: &mut sonic_rs::Value) {
let o = s.as_object_mut().unwrap();
let rc = o.get(&"retweet_count").unwrap().as_i64().unwrap();
o.insert("retweet_count", rc + 1);
o.remove(&"entities");
o.insert("seen", true);
}
match name {
"twitter" => {
for s in r.get_mut("statuses").unwrap().as_array_mut().unwrap().iter_mut() {
status(s);
}
}
"status" => status(r),
"citm_catalog" => {
for p in r.get_mut("performances").unwrap().as_array_mut().unwrap().iter_mut() {
let o = p.as_object_mut().unwrap();
let start = o.get(&"start").unwrap().as_i64().unwrap();
o.insert("start", start + 1);
o.remove(&"seatMapImage");
o.insert("seen", true);
}
}
"canada" => {
let fs = r.get_mut("features").unwrap().as_array_mut().unwrap();
fs[0].get_mut("properties").unwrap().as_object_mut().unwrap().insert("name", "Kanada");
for f in fs.iter_mut() {
let ring = f.get_mut("geometry").unwrap().get_mut("coordinates").unwrap().get_mut(0).unwrap();
ring.as_array_mut().unwrap().push(sonic_rs::json!([0.5, 0.25]));
}
}
"jeopardy" => {
for q in r.as_array_mut().unwrap().iter_mut() {
let o = q.as_object_mut().unwrap();
o.insert("value", sonic_rs::Value::new_null());
o.remove(&"air_date");
o.insert("seen", true);
}
}
_ => {}
}
}
macro_rules! edit_simd {
($fname:ident, $t:ty) => {
fn $fname(name: &str, r: &mut $t) {
fn status(s: &mut $t) {
let o = s.as_object_mut().unwrap();
let rc = o.get("retweet_count").unwrap().as_i64().unwrap();
o.insert("retweet_count".into(), (rc + 1).into());
o.remove("entities");
o.insert("seen".into(), true.into());
}
match name {
"twitter" => {
for s in r.as_object_mut().unwrap().get_mut("statuses").unwrap().as_array_mut().unwrap().iter_mut() {
status(s);
}
}
"status" => status(r),
"citm_catalog" => {
for p in r.as_object_mut().unwrap().get_mut("performances").unwrap().as_array_mut().unwrap().iter_mut() {
let o = p.as_object_mut().unwrap();
let start = o.get("start").unwrap().as_i64().unwrap();
o.insert("start".into(), (start + 1).into());
o.remove("seatMapImage");
o.insert("seen".into(), true.into());
}
}
"canada" => {
let fs = r.as_object_mut().unwrap().get_mut("features").unwrap().as_array_mut().unwrap();
fs[0].as_object_mut().unwrap().get_mut("properties").unwrap().as_object_mut().unwrap().insert("name".into(), "Kanada".into());
for f in fs.iter_mut() {
let g = f.as_object_mut().unwrap().get_mut("geometry").unwrap().as_object_mut().unwrap();
let ring = &mut g.get_mut("coordinates").unwrap().as_array_mut().unwrap()[0];
ring.as_array_mut().unwrap().push(vec![<$t>::from(0.5), <$t>::from(0.25)].into());
}
}
"jeopardy" => {
for q in r.as_array_mut().unwrap().iter_mut() {
let o = q.as_object_mut().unwrap();
o.insert("value".into(), <$t>::Static(simd_json::StaticNode::Null));
o.remove("air_date");
o.insert("seen".into(), true.into());
}
}
_ => {}
}
}
};
}
edit_simd!(edit_simd_owned, simd_json::OwnedValue);
edit_simd!(edit_simd_borrowed, simd_json::BorrowedValue<'_>);
// ---- driver
struct Res {
st: Stats,
acc: f64,
want: bool,
out: Vec<u8>,
}
impl Res {
fn new(want: bool) -> Res {
Res { st: Stats::default(), acc: 0.0, want, out: Vec::new() }
}
}
fn emit(r: &mut Res, o: Vec<u8>) -> bool {
r.acc = o.len() as f64;
if r.want {
r.out = o;
}
true
}
struct Variant<'a> {
name: &'static str,
f: Box<dyn Fn(&[u8], &mut Res) -> bool + 'a>,
any_order: bool, // sum: visits objects in another order than the document's
}
fn v<'a>(name: &'static str, f: impl Fn(&[u8], &mut Res) -> bool + 'a) -> Variant<'a> {
Variant { name, f: Box::new(f), any_order: false }
}
fn v_any<'a>(name: &'static str, f: impl Fn(&[u8], &mut Res) -> bool + 'a) -> Variant<'a> {
Variant { name, f: Box::new(f), any_order: true }
}
fn run(var: &Variant, s: &[u8], r: &mut Res) -> bool {
catch_unwind(AssertUnwindSafe(|| (var.f)(s, r))).unwrap_or(false)
}
static mut SINK: f64 = 0.0;
fn sink(x: f64) {
unsafe { std::ptr::write_volatile(std::ptr::addr_of_mut!(SINK), x) }
}
// the input contract of bench_all.cpp: at least 64 bytes of spare capacity
fn padded(s: &[u8]) -> Vec<u8> {
let mut r = Vec::with_capacity(s.len() + 64);
r.extend_from_slice(s);
r
}
// the minified inputs (twitter_min, citm_catalog_min): the whitespace outside strings removed, the bytes inside strings
// untouched (the same function as strip_ws in bench_all.cpp)
fn strip_ws(s: &[u8]) -> Vec<u8> {
let mut r = Vec::with_capacity(s.len());
let mut in_string = false;
let mut i = 0;
while i < s.len() {
let ch = s[i];
if in_string {
r.push(ch);
if ch == b'\\' && i + 1 < s.len() {
i += 1;
r.push(s[i]);
} else if ch == b'"' {
in_string = false;
}
} else if ch != b' ' && ch != b'\n' && ch != b'\r' && ch != b'\t' {
r.push(ch);
in_string = ch == b'"';
}
i += 1;
}
r
}
// malformed inputs derived from a valid document, as in bench_all.cpp (0xFF in a string value, not a key)
// (what, text, is the 0xFF probe)
fn malformed(s: &[u8]) -> Vec<(&'static str, Vec<u8>, bool)> {
let (mut first_close, mut mid_quote, mut last_quote) = (None, None, None);
let mut in_string = false;
let mut stack: Vec<u8> = Vec::new();
let mut prev = 0u8;
let mut i = 0;
while i < s.len() {
let ch = s[i];
if in_string {
if ch == b'\\' {
i += 1;
} else if ch == b'"' {
in_string = false;
}
} else if ch == b'"' {
in_string = true;
let value = prev == b':' || (stack.last() == Some(&b'[') && (prev == b'[' || prev == b','));
if value {
last_quote = Some(i);
if mid_quote.is_none() && i >= s.len() / 2 {
mid_quote = Some(i);
}
}
prev = b'"';
} else if ch == b'{' || ch == b'[' {
stack.push(ch);
prev = ch;
} else if ch == b'}' || ch == b']' {
if ch == b'}' && first_close.is_none() {
first_close = Some(i);
}
stack.pop();
prev = ch;
} else if ch == b':' || ch == b',' {
prev = ch;
}
i += 1;
}
let mut t = s.to_vec();
while matches!(t.last(), Some(b' ' | b'\n' | b'\r' | b'\t')) {
t.pop();
}
let mut trailing = t.clone();
trailing.extend_from_slice(b" ]x"); // after the value
t.pop();
let mut out = vec![("truncated input", padded(&t), false), ("trailing garbage", padded(&trailing), false)];
let mut t = s.to_vec();
if let Some(c) = first_close {
t.insert(c, b',');
out.push(("a stray comma before '}'", padded(&t), false));
} else {
// a file without objects (--files): the comma goes before the last ']'
let c = (0..t.len()).rev().find(|&i| t[i] == b']').unwrap();
t.insert(c, b',');
out.push(("a stray comma before ']'", padded(&t), false));
}
// a file without string values (--files) gets no UTF-8 probe
if let Some(q) = mid_quote.or(last_quote) {
let mut t = s.to_vec();
t.insert(q + 1, 0xFF);
out.push(("invalid UTF-8 (0xFF in a string)", padded(&t), true));
}
out
}
fn same_text(out: &[u8], expected: &serde_json::Value) -> bool {
serde_json::from_slice::<serde_json::Value>(out).map(|v| v == *expected).unwrap_or(false)
}
fn main() {
let args: Vec<String> = std::env::args().collect();
// --files <dir>: every *.json file of the directory instead of the files below, without select and edit;
// --conformance <dir>: every validate variant on every *.json file of the directory (JSONTestSuite), no timing
let files = args.len() > 1 && args[1] == "--files";
let conformance = args.len() > 1 && args[1] == "--conformance";
let a = if files || conformance { 2 } else { 1 };
if args.len() <= a {
eprintln!("usage: {0} <json_test_data dir> [rounds] [document] [use case]\n {0} --files <dir> [rounds] [document] [use case]\n {0} --conformance <dir>", args[0]);
std::process::exit(1);
}
let t = format!("{}/", args[a]);
let rounds: usize = args.get(a + 1).map(|r| r.parse().unwrap()).unwrap_or(30).max(1);
let only: String = args.get(a + 2).cloned().unwrap_or_default();
let only_uc: String = if conformance { "validate".to_string() } else { args.get(a + 3).cloned().unwrap_or_default() };
// (file, text, batch, the document whose select and edit apply)
let mut docs: Vec<(String, Vec<u8>, usize, String)> = Vec::new();
if files || conformance {
let mut paths: Vec<std::path::PathBuf> = std::fs::read_dir(&t).unwrap().map(|e| e.unwrap().path()).filter(|p| p.extension().map_or(false, |x| x == "json")).collect();
paths.sort();
for p in paths {
// small files are timed in batches of about 500 KB of input, as status is
let text = padded(&std::fs::read(&p).unwrap());
let name = if conformance { p.file_name() } else { p.file_stem() }.unwrap().to_string_lossy().to_string();
let batch = (500000 / text.len().max(1)).max(1);
docs.push((name.clone(), text, batch, name));
}
} else {
for f in ["nativejson-benchmark/twitter.json", "nativejson-benchmark/citm_catalog.json", "nativejson-benchmark/canada.json", "jeopardy/jeopardy.json"] {
let name = f.split('/').nth(1).unwrap().trim_end_matches(".json").to_string();
docs.push((name.clone(), padded(&std::fs::read(format!("{}{}", t, f)).unwrap()), 1, name));
}
let tw: serde_json::Value = serde_json::from_slice(&docs[0].1).unwrap();
docs.push(("status".to_string(), padded(serde_json::to_string(&tw["statuses"][0]).unwrap().as_bytes()), 200, "status".to_string()));
let (tw_min, citm_min) = (padded(&strip_ws(&docs[0].1)), padded(&strip_ws(&docs[1].1)));
docs.push(("twitter_min".to_string(), tw_min, 1, "twitter".to_string()));
docs.push(("citm_catalog_min".to_string(), citm_min, 1, "citm_catalog".to_string()));
}
std::panic::set_hook(Box::new(|_| {})); // a panic is a failed call, reported by the checks
let simd_reused = RefCell::new((simd_json::Buffers::default(), Vec::<u8>::new()));
let mut excluded_total = 0;
for (file, s, batch, base) in &docs {
if !only.is_empty() && *file != only {
continue;
}
let (file, name) = (file.as_str(), base.as_str());
let s = s.as_slice();
let bad = if conformance { Vec::new() } else { malformed(s) };
for uc in ["validate", "sum", "select", "edit", "minify", "write"] {
if (!only_uc.is_empty() && uc != only_uc) || (files && (uc == "select" || uc == "edit")) {
continue;
}
// write: each library's document, parsed once outside the timer
let mut wbuf: Vec<u8>;
let (w_serde, w_sonic, w_simd) = if uc == "write" {
wbuf = s.to_vec();
(
Some(serde_json::from_slice::<serde_json::Value>(s).unwrap()),
Some(sonic_rs::from_slice::<sonic_rs::Value>(s).unwrap()),
Some(simd_json::to_borrowed_value(&mut wbuf).unwrap()),
)
} else {
(None, None, None)
};
let reuse = &simd_reused;
let all: Vec<Variant> = match uc {
"validate" => vec![
v("serde_json: IgnoredAny", |s, _| serde_json::from_slice::<IgnoredAny>(s).is_ok()),
v("serde_json: from_utf8 + IgnoredAny", |s, _| std::str::from_utf8(s).ok().map_or(false, |t| serde_json::from_str::<IgnoredAny>(t).is_ok())),
v("serde_json: Value", |s, _| serde_json::from_slice::<serde_json::Value>(s).is_ok()),
v("sonic-rs: IgnoredAny", |s, _| sonic_rs::from_slice::<IgnoredAny>(s).is_ok()),
v("sonic-rs: Value", |s, _| sonic_rs::from_slice::<sonic_rs::Value>(s).is_ok()),
v("simd-json: tape (copy)", |s, _| { let mut b = s.to_vec(); simd_json::to_tape(&mut b).is_ok() }),
v("simd-json: borrowed (copy)", |s, _| { let mut b = s.to_vec(); let ok = simd_json::to_borrowed_value(&mut b).is_ok(); ok }),
v("simd-json: tape (reused)", move |s, _| {
let mut g = reuse.borrow_mut();
let (ref mut bufs, ref mut b) = *g;
b.clear();
b.extend_from_slice(s);
simd_json::to_tape_with_buffers(b, bufs).is_ok()
}),
],
"sum" => vec![
v("serde_json: visitor", |s, r| sum_serde(s, &mut r.st)),
v_any("serde_json: Value", |s, r| serde_json::from_slice::<serde_json::Value>(s).map(|x| walk_serde(&x, &mut r.st)).is_ok()),
v("sonic-rs: visitor", |s, r| sum_sonic(s, &mut r.st)),
v("sonic-rs: Value", |s, r| sonic_rs::from_slice::<sonic_rs::Value>(s).map(|x| walk_sonic(&x, &mut r.st)).is_ok()),
v("simd-json: visitor (copy)", |s, r| sum_simd(s, &mut r.st)),
v_any("simd-json: borrowed (copy)", |s, r| { let mut b = s.to_vec(); simd_json::to_borrowed_value(&mut b).map(|x| walk_simd(&x, &mut r.st)).is_ok() }),
v_any("simd-json: borrowed (reused)", move |s, r| {
let mut g = reuse.borrow_mut();
let (ref mut bufs, ref mut b) = *g;
b.clear();
b.extend_from_slice(s);
simd_json::to_borrowed_value_with_buffers(b, bufs).map(|x| walk_simd(&x, &mut r.st)).is_ok()
}),
],
"select" => vec![
v("serde_json: structs", move |s, r| pick_structs::<SerdeJson>(name, s).map(|a| r.acc = a).is_some()),
v("serde_json: from_utf8 + structs", move |s, r| pick_structs::<SerdeJsonStr>(name, s).map(|a| r.acc = a).is_some()),
v("serde_json: Value", move |s, r| serde_json::from_slice::<serde_json::Value>(s).map(|x| r.acc = pick_serde(name, &x)).is_ok()),
v("sonic-rs: structs", move |s, r| pick_structs::<SonicRs>(name, s).map(|a| r.acc = a).is_some()),
v("sonic-rs: Value", move |s, r| sonic_rs::from_slice::<sonic_rs::Value>(s).map(|x| r.acc = pick_sonic(name, &x)).is_ok()),
v("simd-json: structs (copy)", move |s, r| pick_structs::<SimdJson>(name, s).map(|a| r.acc = a).is_some()),
v("simd-json: borrowed (copy)", move |s, r| { let mut b = s.to_vec(); let ok = simd_json::to_borrowed_value(&mut b).map(|x| r.acc = pick_simd(name, &x)).is_ok(); ok }),
],
"edit" => vec![
v("serde_json: Value", move |s, r| match serde_json::from_slice::<serde_json::Value>(s) {
Ok(mut x) => { edit_serde(name, &mut x); emit(r, serde_json::to_vec(&x).unwrap()) }
Err(_) => false,
}),
v("sonic-rs: Value", move |s, r| match sonic_rs::from_slice::<sonic_rs::Value>(s) {
Ok(mut x) => { edit_sonic(name, &mut x); emit(r, sonic_rs::to_vec(&x).unwrap()) }
Err(_) => false,
}),
v("simd-json: owned (copy)", move |s, r| { let mut b = s.to_vec(); match simd_json::to_owned_value(&mut b) {
Ok(mut x) => { edit_simd_owned(name, &mut x); emit(r, simd_json::to_vec(&x).unwrap()) }
Err(_) => false,
}}),
v("simd-json: borrowed (copy)", move |s, r| { let mut b = s.to_vec(); let ok = match simd_json::to_borrowed_value(&mut b) {
Ok(mut x) => { edit_simd_borrowed(name, &mut x); emit(r, simd_json::to_vec(&x).unwrap()) }
Err(_) => false,
}; ok }),
],
"minify" => vec![
v("serde_json: Value", |s, r| serde_json::from_slice::<serde_json::Value>(s).map(|x| emit(r, serde_json::to_vec(&x).unwrap())).unwrap_or(false)),
v("sonic-rs: Value", |s, r| sonic_rs::from_slice::<sonic_rs::Value>(s).map(|x| emit(r, sonic_rs::to_vec(&x).unwrap())).unwrap_or(false)),
v("simd-json: borrowed (copy)", |s, r| { let mut b = s.to_vec(); simd_json::to_borrowed_value(&mut b).map(|x| emit(r, simd_json::to_vec(&x).unwrap())).unwrap_or(false) }),
],
_ => {
let (a, b, c) = (w_serde.as_ref().unwrap(), w_sonic.as_ref().unwrap(), w_simd.as_ref().unwrap());
vec![
v("serde_json: Value", move |_, r| emit(r, serde_json::to_vec(a).unwrap())),
v("sonic-rs: Value", move |_, r| emit(r, sonic_rs::to_vec(b).unwrap())),
v("simd-json: borrowed", move |_, r| emit(r, simd_json::to_vec(c).unwrap())),
]
}
};
// the reference result: serde_json's visitor (sum, document order), Value (select, edit, minify, write)
let mut ref_st = Stats::default();
let mut ref_acc = 0.0;
let mut ref_json = serde_json::Value::Null;
match uc {
"sum" => {
sum_serde(s, &mut ref_st);
println!("check {:<13} {:<9} reference: nodes {} str {} num {:.16e}", file, uc, ref_st.nodes, ref_st.str_, ref_st.num);
}
"select" => {
ref_acc = pick_serde(name, &serde_json::from_slice(s).unwrap());
println!("check {:<13} {:<9} reference: {:.16e}", file, uc, ref_acc);
}
"validate" => {}
_ => {
ref_json = serde_json::from_slice(s).unwrap();
if uc == "edit" {
edit_serde(name, &mut ref_json);
}
}
}
if conformance {
// CONF_VARIANT=<name>: only this variant (a stack overflow aborts the process, see conformance.sh)
let only_var = std::env::var("CONF_VARIANT").unwrap_or_default();
for var in all.iter().filter(|v| only_var.is_empty() || v.name == only_var) {
let mut r = Res::new(true);
let verdict = match catch_unwind(AssertUnwindSafe(|| (var.f)(s, &mut r))) {
Ok(true) => "accepted",
Ok(false) => "rejected",
Err(_) => "crashed",
};
println!("conformance: {} | {} | {}", file, var.name, verdict);
}
continue;
}
let n_all = all.len();
let mut vs: Vec<Variant> = Vec::new();
let mut levels: Vec<&str> = Vec::new();
for var in all {
let mut r = Res::new(true);
let mut why = String::new();
let mut panicked = false;
if !catch_unwind(AssertUnwindSafe(|| (var.f)(s, &mut r))).unwrap_or_else(|_| {
panicked = true;
false
}) {
why = if panicked { "crashes on the valid input" } else { "rejects the valid input" }.to_string();
} else if uc == "sum"
&& !(r.st.nodes == ref_st.nodes
&& r.st.str_ == ref_st.str_
&& if var.any_order { (r.st.num - ref_st.num).abs() <= 1e-9 * ref_st.num.abs() } else { r.st.num == ref_st.num })
{
why = format!("wrong result (nodes {} str {} num {:.16e})", r.st.nodes, r.st.str_, r.st.num);
} else if uc == "select" && r.acc != ref_acc {
why = format!("wrong result ({:.16e})", r.acc);
} else if (uc == "edit" || uc == "minify" || uc == "write") && !same_text(&r.out, &ref_json) {
why = "the output does not read back as the expected document".to_string();
}
if why.is_empty() {
// the validation level from the malformed-input probes, as in bench_all.cpp
let mut level = "n/a";
if uc != "write" {
let (mut structural, mut utf8) = (false, false);
let mut notes: Vec<String> = Vec::new();
for (what, b, is_utf8) in &bad {
let mut rb = Res::new(true);
match catch_unwind(AssertUnwindSafe(|| (var.f)(b, &mut rb))) {
Ok(false) => continue,
Ok(true) => {
notes.push(format!("accepts {}", what));
if *is_utf8 { utf8 = true } else { structural = true }
}
Err(_) => {
notes.push(format!("crashes on {}", what));
structural = true;
}
}
}
level = if structural { "partial" } else if utf8 { "no UTF-8" } else { "full" };
if !notes.is_empty() {
println!("level: {} {} | {} | {} | {}", file, uc, var.name, level, notes.join("; "));
}
}
vs.push(var);
levels.push(level);
} else {
excluded_total += 1;
println!("excluded: {} {} | {} | {}", file, uc, var.name, why);
}
}
println!("check {:<13} {:<9} {} of {} variants give the right result", file, uc, vs.len(), n_all);
if vs.is_empty() {
continue;
}
let r = if s.len() > 10_000_000 { rounds.min((rounds / 5).max(3)) } else { rounds };
let mut best = vec![f64::MAX; vs.len()];
// a new order of the variants in every round (as in bench_all.cpp): a Fisher-Yates shuffle with xorshift64
let mut order: Vec<usize> = (0..vs.len()).collect();
let mut rng: u64 = 12345;
for i in 0..r {
if i > 0 {
for j in (1..order.len()).rev() {
rng ^= rng << 13;
rng ^= rng >> 7;
rng ^= rng << 17;
order.swap(j, (rng % (j as u64 + 1)) as usize);
}
}
for &k in &order {
let var = &vs[k];
let mut res = Res::new(false);
run(var, s, &mut res); // untimed: the previous variant's leftovers are cleaned up here
sink(res.st.num + res.acc);
let t0 = Instant::now();
for _ in 0..*batch {
let mut rt = Res::new(false);
let ok = run(var, s, &mut rt);
sink(if ok { rt.st.num + rt.acc } else { -1.0 });
}
let ns = t0.elapsed().as_nanos() as f64 / *batch as f64;
best[k] = best[k].min(ns);
}
}
print!("{:<13} {:<9}", file, uc);
for (k, var) in vs.iter().enumerate() {
print!(" | {} [{}] {:.1}us ({:.2}x)", var.name, levels[k], best[k] / 1e3, best[k] / best[0]);
}
println!();
}
}
println!("done: {} variant(s) excluded", excluded_total);
}
tests/benchmarks/json_view/others/summarize.py
Executable
+229
View File
@@ -0,0 +1,229 @@
#!/usr/bin/env python3
# __ _____ _____ _____
# __| | __| | | | JSON for Modern C++ (supporting code)
# | | |__ | | | | | | version 3.12.0
# |_____|_____|_____|_|___| https://github.com/nlohmann/json
#
# SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
# SPDX-License-Identifier: MIT
"""Summarize runs of bench_all and bench_rs by use case.
Usage: summarize.py LOG... [--rust LOG...] [--json OUT.json]
Every variant carries a validation level from the malformed-input probes: "full" (rejects truncated input, a stray
comma and invalid UTF-8), "no UTF-8" (accepts only the invalid UTF-8), "partial" (accepts or crashes on a structural
error, or crashes on any probe), or "n/a" (write: no input to validate; counts as full). A variant's level is the
worst it got for any file. Prints, for every use case and file,
(a) per library the best "full" variant and, if faster, the best variant at any level, with their ratios to the
best "full" nlohmann json_document variant (variants named "(reused)" or "(reused allocator)" keep a parser,
document or allocator across calls and are never chosen as a library's best; neither are "(copy untimed)"
variants, which copy their input before the timer starts), and in a "reused:" line the library's best reused
variant in the same way,
(b) all variants with their levels and times,
(c) the variants below "full" with what the probes found, and the variants excluded for wrong results.
--json writes {use case: {file: {library: {"best": {"variant", "us", "level"} or null, "best_any": {...},
"best_reused": {...} or null, "best_reused_any": {...} or null, "variants": {variant: {"us", "level"}}, "excluded":
{variant: reason}}}}} for the chart scripts ("best_reused" is the best full reused variant, "best_reused_any" the best
reused variant at any level); variant names there omit the "<library>: " prefix.
Several logs are combined by taking each cell's best time. A variant excluded for one file (or in one log) is excluded
for every file of that use case. The reference values that bench_all and bench_rs print for sum and select are
compared, so both programs are known to have checked against the same numbers.
"""
import json
import re
import sys
USE_CASES = ["validate", "sum", "select", "edit", "minify", "write"]
DOCS = ["twitter", "citm_catalog", "canada", "jeopardy", "status", "twitter_min", "citm_catalog_min"]
OURS = "nlohmann json_document"
RANK = {"full": 0, "n/a": 0, "no UTF-8": 1, "partial": 2}
CELL = re.compile(r"^(.+) \[(full|no UTF-8|partial|n/a)\] ([\d.]+)us(?: \(([\d.]+)x\))?$")
REF = re.compile(r"^check (\S+)\s+(\S+)\s+reference: (.*)$")
def library(name):
return name.split(": ", 1)[0]
def variant(name):
return name.split(": ", 1)[1] if ": " in name else name
def is_reused(name):
return re.search(r"\(reused[ )]", name) is not None
def is_untimed_copy(name):
return "(copy untimed)" in name
def worse(a, b):
return a if a is not None and (b is None or RANK[a] >= RANK[b]) else b
def parse(path, cells, levels, notes, excluded, refs):
for line in open(path, encoding="utf-8"):
line = line.rstrip("\n")
if line.startswith(("excluded: ", "level: ")):
kind, rest = line.split(": ", 1)
fields = rest.split(" | ")
doc, uc = fields[0].split()
name = fields[1]
target = excluded if kind == "excluded" else notes
for why in fields[-1].split("; "):
target.setdefault((uc, name), {}).setdefault(why, set()).add(doc)
if kind == "level":
levels[(uc, name)] = worse(levels.get((uc, name)), fields[2])
continue
m = REF.match(line)
if m:
refs.setdefault((m[1], m[2]), {})[path] = m[3]
continue
if line.startswith(("check", "done", "WARNING")) or " | " not in line:
continue
parts = line.split(" | ")
head = parts[0].split()
if len(head) != 2 or head[1] not in USE_CASES:
continue
for c in parts[1:]:
m = CELL.match(c.strip())
if m:
key = (head[0], head[1], m[1])
if head[0] not in DOCS: # bench_all --files: the documents are the files of a directory
DOCS.append(head[0])
cells[key] = min(cells.get(key, float("inf")), float(m[3]))
levels[(head[1], m[1])] = worse(levels.get((head[1], m[1])), m[2])
def check_refs(refs):
"""The sum and select references of all logs must agree (bench_rs prints Rust's float formatting)."""
def numbers(text):
return [float(x) for x in re.findall(r"[-+]?\d[\d.]*(?:e[-+]?\d+)?", text)]
ok = True
for (doc, uc), by_log in sorted(refs.items()):
values = {tuple(numbers(t)) for t in by_log.values()}
if len(values) > 1:
ok = False
print("WARNING: the %s %s references differ: %s" % (doc, uc, "; ".join("%s: %s" % kv for kv in by_log.items())))
return ok
def is_full(level):
return RANK[level] == 0
def main():
args = sys.argv[1:]
out_json = None
if "--json" in args:
i = args.index("--json")
out_json = args[i + 1]
del args[i:i + 2]
cpp = args[:args.index("--rust")] if "--rust" in args else args
rust = args[args.index("--rust") + 1:] if "--rust" in args else []
if not cpp and not rust:
sys.exit(__doc__)
cells, levels, notes, excluded, refs = {}, {}, {}, {}, {}
for p in cpp + rust:
parse(p, cells, levels, notes, excluded, refs)
check_refs(refs)
result = {}
def docs_of(uc):
return [d for d in DOCS if any(k[0] == d and k[1] == uc for k in cells)]
def cell_of(doc, uc):
return {n: t for (d, u, n), t in cells.items() if d == doc and u == uc and (uc, n) not in excluded}
def show(n, t, ours, uc):
ratio = "%5.2f" % (t / ours) if ours else " -"
return "%-40s %-10s %11.1f us %s" % (variant(n), "[%s]" % levels[(uc, n)], t, ratio)
print("# (a) Per library: the best full variant, and the best at any level if faster (ratio to the best full %s"
" variant; below 1: faster)" % OURS)
for uc in USE_CASES:
if not docs_of(uc):
continue
print("\n== " + uc)
for doc in docs_of(uc):
cell = cell_of(doc, uc)
libs = {}
for n, t in cell.items():
libs.setdefault(library(n), {})[n] = t
entry = result.setdefault(uc, {}).setdefault(doc, {})
rows = []
def pick(vs):
"""the best full variant and the best variant at any level"""
full = {n: t for n, t in vs.items() if is_full(levels[(uc, n)])}
return (min(full.items(), key=lambda x: x[1]) if full else None,
min(vs.items(), key=lambda x: x[1]) if vs else None)
def described(x):
return {"variant": variant(x[0]), "us": x[1], "level": levels[(uc, x[0])]} if x else None
for lib, vs in libs.items():
b, a = pick({n: t for n, t in vs.items() if not is_reused(n) and not is_untimed_copy(n)})
rb, ra = pick({n: t for n, t in vs.items() if is_reused(n)})
entry[lib] = {
"best": described(b),
"best_any": described(a),
"best_reused": described(rb),
"best_reused_any": described(ra),
"variants": {variant(n): {"us": t, "level": levels[(uc, n)]} for n, t in sorted(vs.items(), key=lambda x: x[1])},
}
rows.append((lib, b, a, rb, ra))
for (u, n), whys in excluded.items():
if u == uc:
lib = entry.setdefault(library(n), {"best": None, "best_any": None, "best_reused": None, "best_reused_any": None, "variants": {}})
lib.setdefault("excluded", {})[variant(n)] = "; ".join(sorted(whys))
ours = next((b[1] for lib, b, a, rb, ra in rows if lib == OURS and b), None)
print("%s%s" % (doc, "" if ours else " (no full %s result)" % OURS))
rows.sort(key=lambda r: (r[1] is None, r[1][1] if r[1] else r[2][1] if r[2] else 0))
for lib, b, a, rb, ra in rows:
line = " %-24s %s" % (lib, show(*b, ours, uc) if b else "%-40s %-10s %14s %5s" % ("(none)", "", "", ""))
if a and (not b or a[0] != b[0]):
line += " | faster: %s" % show(*a, ours, uc)
print(line)
if ra:
line = " %-22s %s" % ("reused:", show(*rb, ours, uc) if rb else "%-40s %-10s %14s %5s" % ("(none)", "", "", ""))
if not rb or ra[0] != rb[0]:
line += " | faster: %s" % show(*ra, ours, uc)
print(line)
print("\n# (b) All variants")
for uc in USE_CASES:
if not docs_of(uc):
continue
print("\n== " + uc)
for doc in docs_of(uc):
cell = cell_of(doc, uc)
ours = min((t for n, t in cell.items() if library(n) == OURS and not is_reused(n) and is_full(levels[(uc, n)])), default=None)
print(doc)
for n, t in sorted(cell.items(), key=lambda x: x[1]):
ratio = "%5.2f" % (t / ours) if ours else " -"
print(" %-66s %-10s %11.1f us %s" % (n, "[%s]" % levels[(uc, n)], t, ratio))
print("\n# (c) Variants below full, and excluded variants")
for uc in USE_CASES:
rows = sorted((n, whys) for (u, n), whys in notes.items() if u == uc)
ex = sorted((n, whys) for (u, n), whys in excluded.items() if u == uc)
if not rows and not ex:
continue
print("\n== " + uc)
for n, whys in rows:
for why, docs in sorted(whys.items()):
print(" %-58s %-10s %s (%s)" % (n, "[%s]" % levels.get((uc, n), "?"), why, ", ".join(d for d in DOCS if d in docs)))
for n, whys in ex:
for why, docs in sorted(whys.items()):
print(" %-58s %-10s %s (%s)" % (n, "excluded", why, ", ".join(d for d in DOCS if d in docs)))
if out_json:
with open(out_json, "w", encoding="utf-8") as f:
json.dump(result, f, indent=2)
f.write("\n")
if __name__ == "__main__":
main()