Add images of json_documents: save() and load() (#5632)

* Add images of json_documents: save() and load() An image is a document stored so that loading it needs no parsing: save() writes the node index, the text and the decoded strings; the static load() reads an image written by save(). load() takes a pointer and size, a borrowed vector, or an owned rvalue vector; the nodes are copied so they are aligned and can be edited, while the text and decoded strings stay in the image. image_check controls how much load() trusts the input: full checks structure, bounds, strings and numbers, the parser's own guarantees; bounds checks structure and bounds only; none skips all checks, for images from a trusted source. Layout is little-endian only ("NJVI" header, nodes, text, decoded strings), following the idea of zero-copy formats such as FlatBuffers and YaFF; the check follows FlatBuffers' Verifier. New errors: parse_error.116 for a malformed image or a failed check, type_error.320 for a discarded document or a big-endian target. A dedicated fuzzer and 6,000 seeded corruptions, checked under ASan/UBSan, found and fixed two gaps: unchecked reserved header fields, and unbounded null/boolean offsets that could make dump() throw std::length_error. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix MSVC errors in image test and add image_check doc anchor Move the raw string literal out of the CHECK macro (MSVC preprocessor), cast 64-bit header fields to std::size_t (C4244 on 32-bit), and give the image_check section in load.md a real anchor for the mkdocs strict build. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Check each distinct string and float range once in the full image check The full check scanned the contents of every string node and every float token over its own range, so many nodes pointing to one large range made load() quadratic. The structural walk now only collects the ranges; after it, each kind is sorted and checked once per distinct range. Ranges of one kind that overlap without being identical are rejected, as save() never writes them (nodes that share a value share the whole range). The cost is linear in the size of the image plus sorting the ranges. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Adapt the image tests to named documents and last-wins lookups Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Test that the targets of links never reach an image Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Test lookups in loaded images with duplicate and colliding keys Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Release the list of large objects once load() has built the indexes Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Test that invalid UTF-8 of a loaded image is not copied into an editable document Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Document that editable documents refuse invalid UTF-8 copied from a loaded image Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Expect first-wins lookups in the image tests Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Avoid useless casts of header fields in the image tests Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy 22 findings in unit-json_view_image.cpp and image.hpp Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix MSVC C4244 and C2017 in the image test Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the indentation of a loop in the image test Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make json_view.hpp pass include-what-you-use ci_single_binaries runs IWYU with --error on every header. For json_view.hpp it suggested adding <array> (std::array is used), the headers that json.hpp already provides (abi_config, abi_macros, input_adapters, json_pointer, cpp_future, string_concat, value_t, json_fwd), and <version> for std::nullptr_t, and removing <cstddef>. - include <array> - keep <cstddef> (nullptr_t, size_t; IWYU attributes them to <version> and <cstring>) - tell IWYU not to suggest the headers that json.hpp provides: the amalgamated json_view.hpp only includes json.hpp, so including them here would duplicate their definitions - drop edit_storage.hpp, which edit.hpp includes Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Silence a clang-tidy C++17 finding in the image test bugprone-suspicious-stringview-data-usage flags data() of the source view under C++17, although the length is passed and checked by the REQUIRE before it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Build the image link test without exceptions load_result() exists only with exceptions; with JSON_NOEXCEPTION the test loads the image with the check directly (a failed check aborts). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate the amalgamated headers Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Follow the renames from the review of the view builder The review of #5620 renamed document_data's tape/tape_size/tape_cap to nodes/node_count/node_capacity and made the decode arena a std::vector<char>; adapt the code of this change to it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Rename the emit lambda in the image writer for Qt compatibility Qt defines emit as an empty macro, so a Qt header included before json_view.hpp broke the lambda's declaration (see #5865). The slots variables in the image tests are renamed as well. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address the review of json_document images Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate the amalgamated headers Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-21 11:29:53 +00:00 · 2026-10-21 10:21:29 +02:00
1 parent 823eee3b59
commit 4971200b54
@@ -49,14 +49,21 @@ bookkeeping edits need, and calling any of them on one fails to compile (`#!cpp

 - **view_type** - the type of view returned by [`root()`](root.md) (`#!cpp basic_json_view<BasicJsonType, Editable>`)
 - **value_t** - the JSON type enumeration, see [`basic_json::value_t`](../basic_json/value_t.md)
+- **image_check** - how [`load()`](load.md) validates an image (`full`, `bounds`, `none`), see [`image_check`](load.md#image_check)

 ## Member functions

 - [(constructor)](basic_json_document.md)
+
+### Parsing
+
 - [**parse**](parse.md) (_static_) - deserialize from a compatible input, borrowing or owning it as appropriate
 - [**parse_copy**](parse_copy.md) (_static_) - deserialize a copy of a compatible input
 - [**accept**](accept.md) (_static_) - check whether the input is valid JSON
 - [**read**](read.md) - (re-)parse into this document, reusing its memory
+
+### Access
+
 - [**root**](root.md) - the view of the root value
 - [**is_discarded**](is_discarded.md) - return whether the last parse failed
 - [**source**](source.md) - the parsed text
@@ -64,6 +71,14 @@ bookkeeping edits need, and calling any of them on one fails to compile (`#!cpp
 - [**node_count**](node_count.md) - the number of index entries (values plus object keys)
 - [**memory_usage**](memory_usage.md) - the number of bytes held by the document
 - [**shrink_to_fit**](shrink_to_fit.md) - release unused index capacity
+
+### Images
+
+- [**save**](save.md) - the document as an image that `load()` reads without parsing
+- [**load**](load.md) (_static_) - read an image written by `save()`
+
+### Edits
+
 - [**set**](set.md) - replace a value, or set an object member, an array element, or the value a JSON pointer refers
  to (`#!cpp Editable` documents only)
 - [**push_back**](push_back.md) - append to an array (`#!cpp Editable` documents only)
@@ -0,0 +1,179 @@
+# <small>nlohmann::basic_json_document::</small>load
+
+```cpp
+// (1)
+static basic_json_document load(const std::uint8_t* image, std::size_t size,
+                                 const image_check check = image_check::full);
+
+// (2)
+static basic_json_document load(const std::vector<std::uint8_t>& image,
+                                 const image_check check = image_check::full);
+
+// (3)
+static basic_json_document load(std::vector<std::uint8_t>&& image,
+                                 const image_check check = image_check::full);
+```
+
+1. Reads an image [`save()`](save.md) wrote, from a pointer and a byte count. The image is **borrowed**: `image`
+   must stay alive and unchanged for as long as the returned document, and any view taken from it, is used.
+2. Reads an image from a `#!cpp std::vector`. Also **borrowed** -- equivalent to overload 1 called with
+   `#!cpp image.data()` and `#!cpp image.size()`.
+3. Reads an image, keeping the vector instead of copying it: `image` is moved into the document (no copy), which
+   then owns it for as long as it needs the text and the decoded strings. [`owns_source()`](owns_source.md) is
+   `#!cpp true` afterward.
+
+In every overload, the node index is copied into storage the document itself owns -- so that it is properly aligned,
+and, for an [editable](index.md#edits) document, can be edited -- while the text and the decoded strings stay in
+`image`. The hash indexes [large objects](../../features/json_view.md) use for lookup are rebuilt, exactly as after
+parsing.
+
+## Parameters
+
+`image` (in)
+:   the image [`save()`](save.md) wrote (overloads 1 and 2), or one to take ownership of (overload 3)
+
+`size` (in)
+:   the number of bytes at `image` (overload 1)
+
+`check` (in)
+:   how thoroughly to validate `image` before trusting it; see [`image_check`](#image_check) below (optional,
+    `#!cpp image_check::full` by default)
+
+## Return value
+
+The document read from the image.
+
+## Exception safety
+
+Overloads 1 and 2 give the strong guarantee: `image` is only read, never written, so a thrown exception leaves the
+caller's buffer untouched.
+
+Overload 3 moves `image` into the document *before* validating it, so that a good image is kept without a copy. If
+loading then fails, the partially built document -- and the vector now inside it -- is discarded along with the
+exception, and `image` itself is left **empty**, not restored to what was passed in. Move a copy in instead, or
+validate with overload 2 first, if the original vector must survive a failed load.
+
+## Exceptions
+
+On a big-endian target, throws [`type_error.320`](../../home/exceptions.md#jsonexceptiontype_error320) -- the same
+exception [`save()`](save.md#exceptions) throws there, since the image format is little-endian only.
+
+Otherwise throws [`parse_error.116`](../../home/exceptions.md#jsonexceptionparse_error116) if `image` is not one
+`save()` could have written, or fails the requested `check`:
+
+| message               | when                                                                                                |
+|------------------------|------------------------------------------------------------------------------------------------------|
+| `too short`            | `image` is `#!cpp nullptr`, or `size` is smaller than the 64-byte header                             |
+| `unknown format`       | the header's magic bytes or version do not match, or a reserved header field is not zero             |
+| `sizes out of range`   | the node count, text size, or decoded-string size the header describes does not fit `size`, or the `#!cpp '\0'` after the text or after the decoded strings is missing |
+| `the check failed`     | `check` is not `#!cpp image_check::none`, and the image fails it -- see [`image_check`](#image_check) |
+
+!!! failure "Example messages"
+
+    ```
+    [json.exception.parse_error.116] parse error: invalid json_document image: too short
+    ```
+    ```
+    [json.exception.parse_error.116] parse error: invalid json_document image: unknown format
+    ```
+    ```
+    [json.exception.parse_error.116] parse error: invalid json_document image: sizes out of range
+    ```
+    ```
+    [json.exception.parse_error.116] parse error: invalid json_document image: the check failed
+    ```
+
+## Complexity
+
+Linear in the number of nodes, which are always copied into the document. `#!cpp image_check::none` does not read
+the text or the decoded strings. `#!cpp image_check::bounds` reads one byte of the text for each float token (its sign,
+to check the recorded digits against the token's length), and nothing else of the text or the decoded strings.
+
+With `#!cpp image_check::full`, linear in the size of the image (the nodes, the text, and the decoded strings), plus
+sorting the ranges of the strings and of the float tokens (at most one per node): the contents of each distinct range
+are checked once, however many nodes refer to it.
+
+## Notes
+
+### `image_check` { #image_check }
+
+```cpp
+using image_check = detail::view::image_check;
+
+enum class image_check
+{
+    full,
+    bounds,
+    none
+};
+```
+
+How thoroughly `load()` validates `image` before trusting it.
+
+`full` (the default)
+:   Checks everything the parser itself guarantees: structure and bounds; that every string is valid UTF-8 (and, for a
+    string still in the source text, that it contains no quote, backslash, or control character); and that every number
+    token is well-formed and matches the value stored for it. Strings and float tokens may share a range only if the
+    ranges are identical (as nodes that share a value do), and never overlap otherwise. Reading and serializing a
+    checked image is safe and always produces valid JSON, exactly as for a parsed document. This is the right choice for
+    an image from anything you do not fully control -- a file, a cache shared with other processes, a peer on the
+    network.
+
+`bounds`
+:   Checks structure and bounds only: that every offset and count in the node index stays inside the image. So
+    `load()`, [`root()`](root.md), element access, and [`materialize()`](../basic_json_view/materialize.md) can never
+    read outside the image, even on a damaged one. It does *not* guarantee that the image describes valid JSON: a
+    string that a `full` check would have rejected can make [`dump()`](../basic_json_view/dump.md) write invalid UTF-8
+    or invalid JSON, or throw [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316), and a number can
+    read back with a value that does not match how it is spelled. An [editable document](../json_editable_document.md)
+    does not take such a string over: [`set`](set.md), [`push_back`](push_back.md) and [`insert`](insert.md) throw
+    `type_error.316` when they copy it from a view of the loaded document, and leave the editable document unchanged.
+    `bounds` skips scanning the text and the decoded strings, so it fits a cache your own process just wrote and reads
+    straight back, where damage would mean a bug or a hardware fault rather than adversarial input.
+
+`none`
+:   Checks nothing. Reading a damaged image is undefined behavior, so use it only for an image you trust as much as
+    your own memory.
+
+**Lifetime.** Overloads 1 and 2 borrow `image`: it must stay alive and byte-for-byte unchanged for as long as the
+returned document, and any [view](../basic_json_view/index.md) taken from it, is used -- exactly like a document
+[`parse()`](parse.md) borrowed its input for. Overload 3 avoids this by keeping the vector itself; see
+[`owns_source`](owns_source.md).
+
+!!! warning "Experimental"
+
+    The image format is versioned but not yet stable, and may change in an incompatible way before it is declared
+    stable; `load()` already rejects an image written by a different format version with `parse_error.116`
+    ("unknown format"). Use images to cache a document within one build of the library, or to hand one to another
+    process running the *same* build on the *same* (little-endian) machine -- not as a long-term storage format.
+
+## Examples
+
+??? example "Caching a document, ownership, and a rejected image"
+
+    The example below saves a parsed document as an image, checks that `load()` reproduces the original
+    [`dump()`](../basic_json_view/dump.md) without parsing, and shows the difference between
+    `load(std::move(image))` (owned) and `load(image)` (borrowed). It then damages one byte of the image and shows
+    `image_check::full` rejecting it with `parse_error.116`, while `image_check::bounds` -- meant for a cache the
+    process already trusts -- still reads it without going out of bounds.
+
+    ```cpp
+    --8<-- "examples/basic_json_document__load.cpp"
+    ```
+
+    Output:
+
+    ```json
+    --8<-- "examples/basic_json_document__load.output"
+    ```
+
+## See also
+
+- [save](save.md) - write the document as an image
+- [owns_source](owns_source.md) - return whether the document holds its own copy of the text
+- [parse](parse.md) - deserialize from JSON text instead of an image
+- [Images](../../features/json_view.md#images) - why and when to use images
+
+## Version history
+
+- Added in version 3.13.0.
@@ -149,6 +149,7 @@ buffer of a given length, pass a pair of pointers: `#!cpp parse(ptr, ptr + len)`
 - [accept](accept.md) - check whether the input is valid JSON
 - [read](read.md) - (re-)parse into this document, reusing its memory
 - [owns_source](owns_source.md) - return whether the document holds its own copy of the text
+- [load](load.md) - read a document from an image instead of parsing JSON text
 - [`BasicJsonType::parse`](../basic_json/parse.md) - the corresponding function of `basic_json`

 ## Version history
@@ -80,6 +80,7 @@ compile; see [`parse`](parse.md#notes).

 - [parse](parse.md) - deserialize from a compatible input
 - [root](root.md) - the view of the root value
+- [load](load.md) - read a document from an image instead of parsing JSON text

 ## Version history

@@ -0,0 +1,104 @@
+# <small>nlohmann::basic_json_document::</small>save
+
+```cpp
+std::vector<std::uint8_t> save() const;
+```
+
+Writes the document as an *image*: a byte buffer that [`load`](load.md) reads back without parsing. The image holds
+the node index, the source text (plus, for an edited document, the number tokens edits wrote), and the decoded
+strings (plus the strings edits wrote) -- everything [`root()`](root.md) needs, with nothing left to parse.
+
+An edited document is written in its *current* state, with its values in document order, the way the library's own
+parser would have produced them for that JSON text: a member [`set`](set.md) added goes at the end, an
+[`erase`](erase.md)d member leaves no trace, and a float that is not finite (NaN or positive/negative infinity)
+becomes null, the same substitution [`dump()`](../basic_json_view/dump.md) makes. The same document always saves to
+the same bytes -- also across `BasicJsonType` and `#!cpp Editable`, since the image reflects document order and
+values only, not which specialization produced them.
+
+## Return value
+
+The image, as a `#!cpp std::vector<std::uint8_t>`. Pass it, or a pointer to its data together with its size, to
+[`load`](load.md) to read the document back.
+
+## Exception safety
+
+Strong guarantee: `save()` does not modify `#!cpp *this` (it is `#!cpp const`), so if it throws, the document is left
+exactly as it was, and the partially built image is discarded with the exception.
+
+## Exceptions
+
+Throws [`type_error.320`](../../home/exceptions.md#jsonexceptiontype_error320) if the document is
+[discarded](is_discarded.md) -- a default-constructed document, or one a failed [`parse()`](parse.md)/
+[`read()`](read.md) with `allow_exceptions == false` left discarded.
+
+On a big-endian target, throws `type_error.320` with a different message instead: the image format is little-endian
+only (see [Notes](#notes)).
+
+Throws [`out_of_range.416`](../../home/exceptions.md#jsonexceptionout_of_range416) if the node count, the text, or
+the decoded strings of the image would individually reach 4 GiB -- the same 32-bit offsets
+[`parse()`](parse.md#exceptions) and, for edits, [`set`](set.md)/[`push_back`](push_back.md) are already limited to.
+
+!!! failure "Example messages"
+
+    ```
+    [json.exception.type_error.320] cannot save a discarded json_document
+    ```
+    ```
+    [json.exception.type_error.320] json_document images need a little-endian target
+    ```
+    ```
+    [json.exception.out_of_range.416] images of 4 GiB or more are not supported by json_document
+    ```
+
+## Complexity
+
+Linear in the size of the document: the number of nodes, plus the length of the text and the decoded strings that end
+up in the image.
+
+## Notes
+
+**Format.** The image begins with a 64-byte header (the magic bytes `#!cpp "NJVI"`, a version number, the node count,
+and the sizes of the text and the decoded strings, all little-endian), followed by the nodes
+([16 bytes each](../../home/architecture.md#node-index-of-json-views)), the text and a `#!cpp '\0'`, and the decoded
+strings and a `#!cpp '\0'`. [`load`](load.md) checks the header, and the sizes it describes, before reading anything
+else -- see [`load`'s Exceptions](load.md#exceptions).
+
+!!! warning "Experimental"
+
+    The image format is versioned but not yet stable: it may change in an incompatible way before it is declared
+    stable. Use images to cache a document within one build of the library, or to hand one to another process running
+    the *same* build on the *same* (little-endian) machine -- not as a long-term storage format. Keep the original
+    JSON text if you need to read a saved document back with a future library version.
+
+**Little-endian only.** The image is written as raw little-endian bytes, with no byte-swapping. `save()` (and
+[`load`](load.md)) throw `type_error.320` on a big-endian target rather than silently produce bytes a big-endian
+reader could not interpret correctly.
+
+## Examples
+
+??? example "Caching a parsed document as an image"
+
+    The example below saves a parsed configuration as an image -- the way a service might cache one to answer later
+    requests without parsing the text again -- and confirms that loading it back gives exactly the same result as
+    parsing did, and that saving is deterministic.
+
+    ```cpp
+    --8<-- "examples/basic_json_document__save.cpp"
+    ```
+
+    Output:
+
+    ```json
+    --8<-- "examples/basic_json_document__save.output"
+    ```
+
+## See also
+
+- [load](load.md) - read an image written by `save()`
+- [owns_source](owns_source.md) - return whether the document holds its own copy of the text
+- [`basic_json_view::dump`](../basic_json_view/dump.md) - serialize the document to JSON text instead of an image
+- [Images](../../features/json_view.md#images) - why and when to use images
+
+## Version history
+
+- Added in version 3.13.0.