* Add editable json_documents: set, push_back, insert, and erase basic_json_document gets a second template parameter, Editable (false by default), plus the aliases json_editable_document, json_editable_view, ordered_json_editable_document and ordered_json_editable_view. Editable documents can change values and structure without rewriting the source text: set()/push_back() on values, keys, array indices and JSON pointers; insert() before an array element; erase() of an object key, array index or JSON pointer. New values and element sequences go into edit storage that the document owns and never moves, so views keep referring to their value across edits and a parsed node never moves. Read-only documents walk the plain node array and are unaffected. Strings are checked for UTF-8 on entry, so dump() of an editable document never throws type_error.316. Binary values cannot be stored (type_error.319). A seeded differential test applies random edits to an editable document and to the equivalent ordered_json and compares both after every step. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix old clang and MSVC C4127 in editable json documents Value-initialize the const std::less in find_parent (clang 3.4/3.6 do not implement DR 253), and test the Editable template argument through a function to avoid MSVC C4127. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Copy nested values into a document without recursion Counting and copying the nodes of a view or a basic_json value into an editable document recursed once per nesting level, so that a deeply nested value overflowed the stack. The four functions now walk the value with an explicit stack, as materialize() does. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not leave a half-assigned container when set_moved() throws assign() rewrote the kind, length and extent of a slot before set_moved() registered its new element sequence. set_moved() can throw (std::bad_alloc from reserving the bookkeeping vectors) for a slot that is not moved yet, which left a container without the moved flag that showed its old children. The bookkeeping is now reserved first (reserve_moved()), so that nothing after the first write to the slot can throw. block_of() reserves before it marks anything for the same reason. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Check strings copied from another document for valid UTF-8 An editable document only holds valid UTF-8, but copy_scalar() copied the strings and keys of a view of another document unchecked. A document of a weaker check (or a borrowed text that changed after parsing) could therefore bring ill-formed UTF-8 into it. The copy is checked now, with the error that dump() reports for the string; copies within the same document stay unchecked. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Allow the edit arena to grow up to 4 GiB - 1 bytes append_text() doubled the capacity of the arena and rejected the result if it exceeded 4 GiB, so that an arena of more than 2 GiB could not grow even though the 32-bit offsets of nodes address 4 GiB - 1 bytes. The capacity is now clamped to that limit (text_capacity()), and an append is rejected only if the bytes themselves do not fit. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Set a null value below a pointer as basic_json does set(json_pointer, value) turned a null parent into an object for every last reference token, so "/a/0" and "/a/-" below {"a":null} made {"a":{"0":1}}, where basic_json's operator[](json_pointer) makes an array. A null parent now becomes an array for "-" and for digit tokens (filled with nulls up to the index) and an object otherwise. An invalid index is reported before the parent changes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Replace a container by a scalar in constant time once it is linked Replacing an array or object that spans several nodes by a scalar looks up its parent from the root (find_parent), which is linear in the size of the document: setting every element of a 20,000 element array took more than half a second. The parent only has to switch to links once; afterward the extent of the replaced value no longer matters. Nodes that an entry of a moved sequence links to are now marked (node_flags::linked), and assign() skips the lookup for them. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Update the json_view banner and type_error.319 for editable documents The banner still described a read-only view, and the documentation of type_error.319 listed set and push_back only, although insert rejects binary values as well. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Assign the member that lookups find when set() meets duplicate keys An editable document reads the last member of a repeated key (as the read-only document does), but set(object, key, value) assigned the first one: a view taken from object[key] before the call did not show the new value. It now assigns the last member's value, keeps the key at the position of its first occurrence (where materialize() puts it), and drops the other members. The seeded differential test now also edits documents whose objects repeat keys and compares every lookup (operator[], at, find, value, contains, count, and JSON pointers) with the parsed basic_json value. A targeted test covers duplicates before and after edits that move the object, in large objects with an index, in moved arrays, and in values copied from other documents. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Assign the first member when set() meets duplicate keys again Lookups find the first member of a repeated key again, so set(object, key, value) assigns that member (keeping the key where it is) and drops the others, as documented before, and a view taken from object[key] shows the new value. This undoes the code and documentation changes of5cbb8e6d6. Lookups in edited objects (navigation of editable documents) stop at the first match, like those of parsed objects. The tests that commit added expect the first member from lookups now. In the seeded differential test, the documents with repeated keys repeat them after the real members; the reference holds the first members, which the edits address and the lookups are compared with, while materialize() is compared with what parse() makes of the document's dump. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix MSVC C4127 in the lookups of editable json documents Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy 22 findings in the json_view headers Silence bugprone-casting-through-void for the SSE loads (a reinterpret_cast would trip -Wcast-align=strict), use auto for a cast initialiser, add parentheses to a mixed expression, and name bugprone-std-namespace-modification in the NOLINTs of the tuple_size/tuple_element specialisations. Signed-off-by: Niels Lohmann <mail@nlohmann.me> (cherry picked from commit0effbaa7a2) * Fix clang-tidy 22 findings in unit-json_view_edit.cpp and edit.hpp Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep a raw string with a backslash out of a CHECK (MSVC C2017) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make json_view.hpp pass include-what-you-use ci_single_binaries runs IWYU with --error on every header. For json_view.hpp it suggested adding <array> (std::array is used), and the headers that json.hpp already provides (abi_config, abi_macros, input_adapters, json_pointer, cpp_future, string_concat, value_t, json_fwd), plus <version> for std::nullptr_t, and removing <cstddef> and edit_storage.hpp. - include <array> - keep <cstddef> (nullptr_t, size_t; IWYU attributes them to <version> and <cstring>) - tell IWYU not to suggest the headers that json.hpp provides: the amalgamated json_view.hpp only includes json.hpp, so including them here would duplicate their definitions - drop edit_storage.hpp, which edit.hpp includes Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep the view's edit headers clean for clang-tidy in C++17 mode clang-tidy 22 checks the headers with -std=gnu++17 as well, where it asks for a transparent comparator and std::make_unique. The regions map uses its default comparator, and the allocation of the edit state keeps new (the library is C++11) with a NOLINT. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate the amalgamated headers Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Follow the renames from the review of the view builder The review of #5620 renamed document_data's tape/tape_size/tape_cap to nodes/node_count/node_capacity and made the decode arena a std::vector<char>; adapt the code of this change to it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Expect count() to count every member in the edit tests Since #5622, count() counts every member with the key, so documents with duplicate keys no longer give 1. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Count matching keys as unsigned in the json_view edit tests Like in the access test, adding the int result of the conditional to a std::size_t fails -Werror=sign-conversion with GCC. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address the review of erase() in editable json documents From gregmarr's review of #5630: - invalid_iterator.202 is thrown by overloads 1 and 2 only; the page said "every overload" and then excluded 3 in parentheses. - The note on links says what moves (the links after the removed element, as insert() moves them the other way) and that every iterator over the container is invalidated, not only those after the removed element. - edit.hpp compares keys with document_data::key_equals(). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate the amalgamated headers Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address the review of editable json documents From gregmarr's review of #5630: - set() with a JSON pointer below a null value checks for "-" before scanning the token for digits. - node_flags::storage is defined as escaped | edited, and the comment on document_data::base says that the fourth slot is unused. - source_extent() names its root check. - The integer-index overloads of set(), insert(), and erase() use a named trait instead of a negated condition, which astyle formats like the library's other enable_ifs. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate the amalgamated headers Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address the rest of the review of editable json documents From gregmarr's review of #5630: - alloc_nodes() gives a request of half of the next chunk or more a chunk of its own, so that the rest of the current chunk stays available for smaller requests instead of being abandoned. - The comments on edit_state, append_text(), and reserve_moved() say that views stay valid across every edit (not just the first), that n is 0 for empty strings, and why both capacities are checked. - The architecture page names flag bits 3, 4, and 5 (linked) separately and says that a discarded view has no node; erase.md rewords overload 3 and says why it never throws invalid_iterator.202. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Call alloc_nodes() before reading chunks.back() in the json_view chunk test The operands of == are unsequenced, and MSVC evaluated d->edits->chunks.back() before alloc_nodes() added the new chunk, so the check compared the result with the previous chunk and failed on msvc-arm64 (Debug and Release). Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
297 lines
12 KiB
C++
297 lines
12 KiB
C++
// __ _____ _____ _____
|
|
// __| | __| | | | JSON for Modern C++
|
|
// | | |__ | | | | | | version 3.12.0
|
|
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
|
|
//
|
|
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
|
|
// SPDX-License-Identifier: MIT
|
|
|
|
#pragma once
|
|
|
|
#include <array> // array
|
|
#include <cstddef> // size_t
|
|
#include <cstdint> // uint32_t
|
|
#include <cstring> // memcmp, memcpy
|
|
#include <limits> // numeric_limits
|
|
#include <map> // map
|
|
#include <memory> // unique_ptr
|
|
#include <new> // bad_alloc, operator new, placement new
|
|
#include <string> // string
|
|
#include <vector> // vector
|
|
|
|
#include <nlohmann/json.hpp>
|
|
#include <nlohmann/detail/view/macro_scope.hpp>
|
|
#include <nlohmann/detail/view/node.hpp>
|
|
|
|
NLOHMANN_JSON_NAMESPACE_BEGIN
|
|
namespace detail
|
|
{
|
|
namespace view
|
|
{
|
|
|
|
/// storage of a parsed document; heap-allocated (header and an initial node
|
|
/// array in one block) so that views survive moves of the owning document
|
|
struct document_data
|
|
{
|
|
const char* src = nullptr;
|
|
std::size_t size = 0;
|
|
node* nodes = nullptr; ///< the node index (see node.hpp)
|
|
std::size_t node_count = 0;
|
|
std::size_t node_capacity = 0;
|
|
node* inline_nodes = nullptr; ///< node array allocated together with this header
|
|
std::size_t inline_capacity = 0;
|
|
std::size_t max_depth = 0; ///< deepest nesting of arrays and objects (the root container is depth 1)
|
|
std::vector<char> arena; ///< decoded strings that contained escapes (no terminator: they are handed out with their length)
|
|
std::string owned; ///< owned copy of the input, if any (a std::string, so that its NUL serves as the parser's sentinel)
|
|
|
|
// hash indexes of the keys of large objects (see object_index.hpp)
|
|
static constexpr std::uint32_t index_min_members = 128;
|
|
struct object_index
|
|
{
|
|
std::size_t start; ///< first slot in index_slots
|
|
std::uint32_t mask; ///< slot count - 1 (a power of two minus one)
|
|
};
|
|
std::vector<object_index> indexes;
|
|
std::vector<std::uint32_t> index_slots;
|
|
std::vector<std::uint32_t> large_objects; ///< positions of the objects to index (noted while parsing)
|
|
std::array<const char*, 4> base = {{nullptr, nullptr, nullptr, nullptr}}; ///< string bases: source, arena, edit arena, unused (indexed by flags & node_flags::storage)
|
|
bool discarded = true;
|
|
|
|
/// The storage of edits (editable documents only; see edit_storage.hpp).
|
|
/// No edit moves or resizes the parsed index, and the nodes of values
|
|
/// written by edits never move either, so a view stays valid whenever it
|
|
/// was taken, before the first edit or between later ones: an array/object
|
|
/// whose elements change gets node_flags::moved, and its elements then live
|
|
/// in a separate sequence (a header node, then the entries), whose entries
|
|
/// link to the values.
|
|
struct edit_state
|
|
{
|
|
std::vector<node*> moved{}; ///< element sequences of moved arrays/objects (header node first) // NOLINT(readability-redundant-member-init)
|
|
std::vector<std::size_t> moved_cap{}; ///< capacity in nodes of a growable block; 0: a fixed sequence (a new value) // NOLINT(readability-redundant-member-init)
|
|
std::vector<std::unique_ptr<node[]>> chunks{}; ///< storage of new values and blocks; never moved // NOLINT(readability-redundant-member-init,cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
|
|
std::map<const node*, node*> regions{}; ///< new arrays/objects: root -> container that uses it as its element sequence (nullptr: linked from a block) // NOLINT(readability-redundant-member-init)
|
|
node* chunk_cur = nullptr;
|
|
node* chunk_end = nullptr;
|
|
std::size_t chunk_next = 64;
|
|
std::vector<std::unique_ptr<char[]>> texts{}; ///< edit arena, the current buffer last; earlier ones stay alive for string views // NOLINT(readability-redundant-member-init,cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
|
|
std::size_t text_used = 0;
|
|
std::size_t text_cap = 0;
|
|
std::size_t bytes = 0; ///< memory held by edits
|
|
};
|
|
std::unique_ptr<edit_state> edits; ///< created by the first edit
|
|
|
|
/// One allocation for the header and room for `count` nodes, so that a
|
|
/// small document needs one allocation instead of two. Large documents get
|
|
/// a separate node array instead (so it can be trimmed). When a document
|
|
/// outgrows the inline array, that array stays unused: at most 256 nodes
|
|
/// (4 KiB), and only for documents whose estimate was too small.
|
|
static document_data* create(std::size_t count)
|
|
{
|
|
count = count <= 256 ? count : 0;
|
|
void* mem = ::operator new (sizeof(document_data) + (count * sizeof(node)));
|
|
auto* d = new (mem) document_data(); // NOLINT(cppcoreguidelines-owning-memory): owned by the returned pointer, freed by deleter
|
|
// (aligned: sizeof is a multiple of the alignment; through void*, as GCC's -Wcast-align wants)
|
|
// (null without inline nodes, not a pointer past the end of the allocation)
|
|
d->inline_nodes = count == 0 ? nullptr : static_cast<node*>(static_cast<void*>(static_cast<char*>(mem) + sizeof(document_data))); // NOLINT(bugprone-casting-through-void)
|
|
d->inline_capacity = count;
|
|
d->nodes = d->inline_nodes;
|
|
d->node_capacity = count;
|
|
return d;
|
|
}
|
|
|
|
struct deleter
|
|
{
|
|
void operator()(document_data* d) const noexcept
|
|
{
|
|
d->~document_data();
|
|
::operator delete (d);
|
|
}
|
|
};
|
|
|
|
/// user-provided so that the class-type members can be initialized in the
|
|
/// member initialization list (-Weffc++ asks for it, and old GCC rejects a
|
|
/// defaulted constructor whose exception specification differs from the
|
|
/// implicit one); they cannot take default member initializers, which old
|
|
/// Clang (3.4-3.6) rejects. The other members have default member
|
|
/// initializers. noexcept: create() constructs into raw memory and could not
|
|
/// release it if this threw (the std::vector and std::string default
|
|
/// constructors do not allocate).
|
|
document_data() noexcept
|
|
: arena() // NOLINT(readability-redundant-member-init)
|
|
, owned() // NOLINT(readability-redundant-member-init)
|
|
, indexes() // NOLINT(readability-redundant-member-init)
|
|
, index_slots() // NOLINT(readability-redundant-member-init)
|
|
, large_objects() // NOLINT(readability-redundant-member-init)
|
|
, edits() // NOLINT(readability-redundant-member-init)
|
|
{}
|
|
document_data(const document_data&) = delete;
|
|
document_data(document_data&&) = delete;
|
|
document_data& operator=(const document_data&) = delete;
|
|
document_data& operator=(document_data&&) = delete;
|
|
~document_data()
|
|
{
|
|
release();
|
|
}
|
|
|
|
void release() noexcept
|
|
{
|
|
if (nodes != inline_nodes)
|
|
{
|
|
::operator delete (nodes);
|
|
}
|
|
nodes = inline_nodes;
|
|
node_capacity = inline_capacity;
|
|
}
|
|
|
|
/// the largest node count whose size in bytes fits a std::size_t
|
|
static constexpr std::size_t max_nodes() noexcept
|
|
{
|
|
return (std::numeric_limits<std::size_t>::max)() / sizeof(node);
|
|
}
|
|
|
|
[[noreturn]] NLOHMANN_VIEW_NOINLINE static void throw_bad_alloc()
|
|
{
|
|
NLOHMANN_VIEW_THROW(std::bad_alloc());
|
|
}
|
|
|
|
/// make room for n nodes; keeps the first node_count nodes (throws
|
|
/// std::bad_alloc for a count that does not fit the address space,
|
|
/// instead of wrapping around in n * sizeof(node))
|
|
void reserve(std::size_t n)
|
|
{
|
|
if (n <= node_capacity)
|
|
{
|
|
return;
|
|
}
|
|
if (NLOHMANN_VIEW_UNLIKELY(n > max_nodes()))
|
|
{
|
|
throw_bad_alloc();
|
|
}
|
|
node* fresh = static_cast<node*>(::operator new (n * sizeof(node)));
|
|
if (node_count != 0)
|
|
{
|
|
std::memcpy(fresh, nodes, node_count * sizeof(node));
|
|
}
|
|
release();
|
|
nodes = fresh;
|
|
node_capacity = n;
|
|
}
|
|
|
|
const char* str(const node& n) const noexcept
|
|
{
|
|
return base[n.flags & node_flags::storage] + n.off;
|
|
}
|
|
|
|
/// whether the string of node k (a key) is the n bytes at key
|
|
bool key_equals(const node& k, const char* key, std::size_t n) const noexcept
|
|
{
|
|
return k.len == n && (n == 0 || std::memcmp(str(k), key, n) == 0);
|
|
}
|
|
|
|
/// the node after n's subtree (containers span `next` nodes, scalars one)
|
|
static NLOHMANN_VIEW_ALWAYS_INLINE const node* after(const node* n) noexcept
|
|
{
|
|
return n + (is_container(*n) ? n->next : 1u);
|
|
}
|
|
|
|
/// first element (array) or first key (object) of a container
|
|
static NLOHMANN_VIEW_ALWAYS_INLINE const node* first_child(const node* n) noexcept
|
|
{
|
|
return n + 1;
|
|
}
|
|
|
|
/// end of the elements of a container
|
|
static NLOHMANN_VIEW_ALWAYS_INLINE const node* child_end(const node* n) noexcept
|
|
{
|
|
return n + n->next;
|
|
}
|
|
|
|
/// (editable documents) first element or key, also of a moved container
|
|
NLOHMANN_VIEW_ALWAYS_INLINE const node* first_child_edited(const node* n) const noexcept
|
|
{
|
|
return NLOHMANN_VIEW_LIKELY((n->flags & node_flags::moved) == 0) ? n + 1 : edits->moved[n->off] + 1;
|
|
}
|
|
|
|
/// (editable documents) end of the elements, also of a moved container
|
|
NLOHMANN_VIEW_ALWAYS_INLINE const node* child_end_edited(const node* n) const noexcept
|
|
{
|
|
if (NLOHMANN_VIEW_LIKELY((n->flags & node_flags::moved) == 0))
|
|
{
|
|
return n + n->next;
|
|
}
|
|
const node* const h = edits->moved[n->off];
|
|
return h + h->next;
|
|
}
|
|
|
|
/// (editable documents) the value at an element position: entries of
|
|
/// moved sequences are links. The link case is out of line, so that this
|
|
/// compiles to a predicted branch rather than a select that delays the
|
|
/// following loads.
|
|
static NLOHMANN_VIEW_ALWAYS_INLINE const node* deref(const node* n) noexcept
|
|
{
|
|
return NLOHMANN_VIEW_LIKELY(n->kind != kind_link) ? n : follow_link(n);
|
|
}
|
|
|
|
static NLOHMANN_VIEW_NOINLINE const node* follow_link(const node* n) noexcept
|
|
{
|
|
return link_target(*n);
|
|
}
|
|
};
|
|
|
|
/// How the index is walked: views of read-only documents follow the node
|
|
/// array alone and compile without any of the edit handling; views of
|
|
/// editable documents also follow moved element sequences and links.
|
|
template<bool Editable>
|
|
struct navigation
|
|
{
|
|
/// whether the walk follows edits (the template argument as a runtime
|
|
/// condition: `Editable && ...` is a constant condition for MSVC, C4127)
|
|
static NLOHMANN_VIEW_ALWAYS_INLINE bool editable() noexcept
|
|
{
|
|
return false;
|
|
}
|
|
|
|
static NLOHMANN_VIEW_ALWAYS_INLINE const node* first(const document_data& /*d*/, const node* n) noexcept
|
|
{
|
|
return n + 1;
|
|
}
|
|
|
|
static NLOHMANN_VIEW_ALWAYS_INLINE const node* end(const document_data& /*d*/, const node* n) noexcept
|
|
{
|
|
return n + n->next;
|
|
}
|
|
|
|
static NLOHMANN_VIEW_ALWAYS_INLINE const node* value(const node* n) noexcept
|
|
{
|
|
return n;
|
|
}
|
|
};
|
|
|
|
template<>
|
|
struct navigation<true>
|
|
{
|
|
static NLOHMANN_VIEW_ALWAYS_INLINE bool editable() noexcept
|
|
{
|
|
return true;
|
|
}
|
|
|
|
static NLOHMANN_VIEW_ALWAYS_INLINE const node* first(const document_data& d, const node* n) noexcept
|
|
{
|
|
return d.first_child_edited(n);
|
|
}
|
|
|
|
static NLOHMANN_VIEW_ALWAYS_INLINE const node* end(const document_data& d, const node* n) noexcept
|
|
{
|
|
return d.child_end_edited(n);
|
|
}
|
|
|
|
static NLOHMANN_VIEW_ALWAYS_INLINE const node* value(const node* n) noexcept
|
|
{
|
|
return document_data::deref(n);
|
|
}
|
|
};
|
|
|
|
} // namespace view
|
|
} // namespace detail
|
|
NLOHMANN_JSON_NAMESPACE_END
|