wt/json
1
0 Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-10-22 11:36:14 +00:00
Code Issues Packages Projects Releases Wiki Activity
5,365 Commits 16 Branches 57 Tags
89bd5d63e19dd315edc4924bc4b4cecb8664a7d7
Commit Graph
3 Commits
Author SHA1 Message Date
Niels Lohmann 3fb7d9c710 Check the float tokens of a json_document image without a sort and a scan of every character 
load(image_check::full) checked every float token of an image by sorting
the token ranges, scanning the token character by character, and comparing
the digit layout with the node's. Now:

- the ranges are sorted only if they are not sorted already (save() writes
  them in the order of the nodes, so they are);
- a token of the shape [-]digits.digits whose digit counts are the ones the
  node records (nearly all of them) is accepted by checking its digits eight
  at a time, without the general scan;
- the general scan skips runs of digits eight at a time.

Damaged tokens are still rejected as before: the check accepts exactly the
tokens that the general scan accepts.

Instructions of load(image_check::full) (develop -> now):

  macOS arm64        canada 44.6M -> 30.9M (-31 %), jeopardy 11.3M -> 10.7M,
                     citm 10.4M -> 10.0M, twitter 5.06M -> 4.94M
  Linux GCC 13       canada -50 %, jeopardy -43 %, twitter -31 %, citm -28 %
  Linux Clang 18     canada -42 %, jeopardy -48 %, twitter -35 %, citm -32 %

image_check::bounds is unchanged (within 0.4 %).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-22 09:30:44 +02:00
Niels Lohmann 4155e2938f Write json_view's dump() without a call or conversion per token (#5633) 
* Write json_view's dump() without a call or conversion per token

The default dump() (no indentation, no ensure_ascii) gets its own
writer that makes the same walk and produces the same output:

- the write position stays in a local variable instead of a
  member, so the compiler keeps it in a register across stores
  through aliasing char pointers;
- strings and number tokens are copied with fixed-size 32-byte
  moves wherever enough source bytes remain, instead of one
  memcpy call per token;
- the innermost open container lives in local variables; a stack
  that starts as a local array of 32 entries holds the rest;
- unedited documents are walked through the node array in order,
  and integer tokens are read from the source directly.

On top of that, float tokens of at most 15 significant digits are
written straight from their digits via zmij::to_shortest() and
write_shortest(), without converting to a double and back: such
decimals are farther apart than a double's rounding interval, so
the token's digits are the double's shortest digits. Tokens of
16+ digits, or edited values, still go through decimal_to_float().
The view's own NEON write_decimal() is removed in favor of the
shared writer, and the dump output now grows in 64 KiB steps
instead of being resized to its estimate at once.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix MSVC C4127 and Infer finding in the view serializer

Test the template options through enabled() so that conditions combined
with them are not constant, and mark the nav::value() null dereference
as a false positive like materialize.hpp does.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix out-of-bounds read when dumping a float token of an unchecked image

A float node of an image loaded with image_check::bounds can hold a
token whose bytes are not digits, so its value need not have the digit
count recorded in the node. write_double_at() passed that count to
write_short_decimal(), which indexes its table of powers of ten by it:
an assertion failure in debug builds, an out-of-bounds read in release
builds. Use the counted overload only if the value has exactly that many
digits, otherwise count them.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Tidy the view serializer: doxygen block placement, enabled() for a constant

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Use a named document in the dump test: root() of a temporary does not compile

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Test write_short_decimal and powers_of_ten_16 against to_chars

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix clang-tidy 22 findings in unit-to_chars.cpp

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Write doubles from their bits only where double is IEEE 754 binary64

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Split the value and pointer edit tests off unit-json_view_edit.cpp

The MinGW linker of the Windows clang jobs cannot link object files with more
than 32767 sections ("relocation truncated to fit: IMAGE_REL_AMD64_REL32
against `.rdata'"). unit-json_view_edit.cpp has 35409 sections since the fast
dump instantiates more code for editable documents, so its test cases "views
and values", "deeply nested values", "pointers below a null value", and
"strings of other documents are checked" move into
unit-json_view_edit_values.cpp. The helper exception_of_call() that both
files use moves into json_view_test_helpers.hpp.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Move the dump tests of this branch to unit-json_view_dump.cpp

The dump and comparison tests moved to their own file in the dump pull
request; the tests of the fast dump() join them there.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Regenerate the amalgamated headers

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Address review: one powers-of-ten table, literal sizes from the type

- dtoa_impl::powers_of_ten() (10^0 to 10^19, one entry per line)
  replaces powers_of_ten_16() and the table of int_pow10()
- put_literal() takes the length of a string literal from its type
- small_stack holds the heap vector by reference
- the growth step of the output buffer is written in hex

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Explain why the growth of the dump's stack is static

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Regenerate the amalgamated headers

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-21 10:21:30 +02:00
Niels Lohmann 4971200b54 Add images of json_documents: save() and load() (#5632) 
* Add images of json_documents: save() and load()

An image is a document stored so that loading it needs no
parsing: save() writes the node index, the text and the decoded
strings; the static load() reads an image written by save().

load() takes a pointer and size, a borrowed vector, or an owned
rvalue vector; the nodes are copied so they are aligned and can
be edited, while the text and decoded strings stay in the image.

image_check controls how much load() trusts the input: full
checks structure, bounds, strings and numbers, the parser's own
guarantees; bounds checks structure and bounds only; none skips
all checks, for images from a trusted source.

Layout is little-endian only ("NJVI" header, nodes, text, decoded
strings), following the idea of zero-copy formats such as
FlatBuffers and YaFF; the check follows FlatBuffers' Verifier.

New errors: parse_error.116 for a malformed image or a failed
check, type_error.320 for a discarded document or a big-endian
target.

A dedicated fuzzer and 6,000 seeded corruptions, checked under
ASan/UBSan, found and fixed two gaps: unchecked reserved header
fields, and unbounded null/boolean offsets that could make
dump() throw std::length_error.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix MSVC errors in image test and add image_check doc anchor

Move the raw string literal out of the CHECK macro (MSVC preprocessor),
cast 64-bit header fields to std::size_t (C4244 on 32-bit), and give the
image_check section in load.md a real anchor for the mkdocs strict build.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Check each distinct string and float range once in the full image check

The full check scanned the contents of every string node and every float token over its own range, so many nodes pointing to one large range made load() quadratic. The structural walk now only collects the ranges; after it, each kind is sorted and checked once per distinct range. Ranges of one kind that overlap without being identical are rejected, as save() never writes them (nodes that share a value share the whole range). The cost is linear in the size of the image plus sorting the ranges.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Adapt the image tests to named documents and last-wins lookups

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Test that the targets of links never reach an image

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Test lookups in loaded images with duplicate and colliding keys

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Release the list of large objects once load() has built the indexes

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Test that invalid UTF-8 of a loaded image is not copied into an editable document

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Document that editable documents refuse invalid UTF-8 copied from a loaded image

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Expect first-wins lookups in the image tests

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Avoid useless casts of header fields in the image tests

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix clang-tidy 22 findings in unit-json_view_image.cpp and image.hpp

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix MSVC C4244 and C2017 in the image test

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix the indentation of a loop in the image test

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Make json_view.hpp pass include-what-you-use

ci_single_binaries runs IWYU with --error on every header. For json_view.hpp
it suggested adding <array> (std::array is used), the headers that json.hpp
already provides (abi_config, abi_macros, input_adapters, json_pointer,
cpp_future, string_concat, value_t, json_fwd), and <version> for
std::nullptr_t, and removing <cstddef>.

- include <array>
- keep <cstddef> (nullptr_t, size_t; IWYU attributes them to <version> and <cstring>)
- tell IWYU not to suggest the headers that json.hpp provides: the amalgamated
  json_view.hpp only includes json.hpp, so including them here would duplicate
  their definitions
- drop edit_storage.hpp, which edit.hpp includes

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Silence a clang-tidy C++17 finding in the image test

bugprone-suspicious-stringview-data-usage flags data() of the source view
under C++17, although the length is passed and checked by the REQUIRE
before it.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Build the image link test without exceptions

load_result() exists only with exceptions; with JSON_NOEXCEPTION the test
loads the image with the check directly (a failed check aborts).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Regenerate the amalgamated headers

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Follow the renames from the review of the view builder

The review of #5620 renamed document_data's tape/tape_size/tape_cap to
nodes/node_count/node_capacity and made the decode arena a
std::vector<char>; adapt the code of this change to it.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Rename the emit lambda in the image writer for Qt compatibility

Qt defines emit as an empty macro, so a Qt header included before
json_view.hpp broke the lambda's declaration (see #5865). The slots
variables in the image tests are renamed as well.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Address the review of json_document images

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Regenerate the amalgamated headers

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-21 10:21:29 +02:00