wt/json
1
0 Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-11-09 16:21:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

json_view: source positions in materialize() with JSON_DIAGNOSTIC_POSITIONS (#5945)

Browse Source 
* json_view: source positions in materialize() with JSON_DIAGNOSTIC_POSITIONS

With JSON_DIAGNOSTIC_POSITIONS, materialize() now gives every value the
start_pos()/end_pos() that basic_json::parse() would give it, so exceptions
about materialized values name the same bytes (#5864).

- json_sax_dom_parser takes the lexer type as a defaulted template
  parameter; materialize() passes position_source, which reports the
  positions it derives from the index. diagnostic_positions treats npos as
  "no position" and keeps ranges all-or-nothing.
- Strings with escapes keep the offset of their opening quote plus one in
  `next` (only with the macro); materialize() scans from there to the closing
  quote. The end of an array or object is found by scanning from the end of
  its last element over whitespace, comments and a trailing comma.
- In editable documents, values written by edits, arrays and objects whose
  elements changed, and containers whose last element has no position get
  none. Images keep the flag is_new, and load() checks that `next` of a
  string is within the text.

Nothing changes on the parse path without the macro. The view's own
exceptions still carry no byte range, so the tests that compare them with
those of materialized values skip it under the macro, as they skip the path
under JSON_DIAGNOSTICS.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Indent comment in unit-json_view_from_json.cpp

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Address review: clearer position code and docs

- position_source gets clear(); set_scalar() returns early for new values,
  switches on node_kind, and uses no local copies; the one position_source
  per materialize() call is documented
- no npos + 1 for containers without a position; a comment on why the end
  of a container is reported as its closing bracket
- less duplication under #if in materialize() and the builder
- diagnostic_positions sets npos explicitly
- docs: positions of basic_json values after changes, positions in images
  depend on how the document was parsed, when next is set, and the cost of
  positions while parsing

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Share the comment scanners of the view builder with materialize()

skip_line_comment() and find_block_comment_end() in scan.hpp are used by
the builder's comment() and by position_source::container_end(), which
had its own copy.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Address review: if/else for container positions, clearer next field

- the opening bracket of a container sets positions with an if/else instead
  of a ternary, and the frame takes its start from positions
- the closing bracket of a container is found in nested if/else; the parent's
  last_end is taken from positions, so it is npos whenever no range is set
- architecture.md: the next field reads "or 0 if it was not; all other
  values: 0" instead of "else 0; otherwise 0"

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix GCC unused-result and clang-tidy raw-string findings in unit-json_view_positions.cpp

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann authored and GitHub committed 2026-10-28 20:24:49 +01:00
1 parent 091c10d4a5
commit 54f6b3ef06
17 files changed
+1101 -113
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

include/nlohmann/detail/view/image.hpp
+18 -12
View File
@@ -80,7 +80,8 @@ constexpr std::uint64_t image_limit = 0xFFFFFFF0u;
/// Copy the current structure of an edited document into nodes in document
/// order, as the parser would have written them. Text written by edits is
/// appended to text_tail (number tokens) and arena_tail (strings); floats that
/// are not finite become null, as dump() writes them.
/// are not finite become null, as dump() writes them. Nodes keep the flag
/// is_new: they have no source position (JSON_DIAGNOSTIC_POSITIONS).
inline void compact_nodes(const document_data& d, std::size_t arena_size, std::vector<node>& out, std::string& text_tail, std::string& arena_tail)
{
struct frame
@@ -103,16 +104,18 @@ inline void compact_nodes(const document_data& d, std::size_t arena_size, std::v
arena_tail.append(d.str(s), s.len);
r.flags = node_flags::escaped;
}
r.flags = static_cast<std::uint8_t>(r.flags | (s.flags & node_flags::is_new));
return r;
};
const auto copy_node = [&](const node * v)
{
node r = *v;
const auto is_new = static_cast<std::uint8_t>(v->flags & node_flags::is_new);
switch (static_cast<value_t>(v->kind))
{
case value_t::object:
case value_t::array:
r.flags = 0;
r.flags = is_new;
r.extra = 0;
r.off = (v->flags & (node_flags::moved | node_flags::is_new)) != 0 ? 0 : v->off;
r.len = 0; // counted below
@@ -129,7 +132,7 @@ inline void compact_nodes(const document_data& d, std::size_t arena_size, std::v
r.off = static_cast<std::uint32_t>(d.size + text_tail.size());
text_tail.append(d.str(*v), number_length(*v));
}
r.flags = 0;
r.flags = is_new;
break;
case value_t::number_float:
if ((v->flags & node_flags::storage) == node_flags::edited)
@@ -139,22 +142,23 @@ inline void compact_nodes(const document_data& d, std::size_t arena_size, std::v
{
r = node{}; // nan and infinity: null, as dump() writes them
r.kind = node_kind::null;
r.flags = is_new;
break;
}
r.off = static_cast<std::uint32_t>(d.size + text_tail.size());
text_tail.append(t, v->len);
r.extra = 0xFFFFu; // the digit layout is not recorded
}
r.flags = 0;
r.flags = is_new;
break;
case value_t::boolean:
r.flags = static_cast<std::uint8_t>(v->flags & node_flags::is_true);
r.flags = static_cast<std::uint8_t>(v->flags & (node_flags::is_true | node_flags::is_new));
break;
case value_t::null:
case value_t::binary:
case value_t::discarded:
default:
r.flags = 0;
r.flags = is_new;
break;
}
out.push_back(r);
@@ -515,13 +519,15 @@ inline bool check_image(const node* nodes, std::size_t count, const unsigned cha
std::vector<float_range> floats;
const auto check_string = [&](const node & n) -> bool
{
if ((n.flags & ~node_flags::escaped) != 0 || n.extra != 0)
if ((n.flags & ~(node_flags::escaped | node_flags::is_new)) != 0 || n.extra != 0)
{
return false;
}
const bool decoded = (n.flags & node_flags::escaped) != 0;
const std::size_t limit = decoded ? arena_size : text_size;
if (n.off > limit || n.len > limit - n.off)
// (next: the source offset of a decoded string's opening quote plus
// one, or 0; see node)
if (n.off > limit || n.len > limit - n.off || n.next > text_size)
{
return false;
}
@@ -593,13 +599,13 @@ inline bool check_image(const node* nodes, std::size_t count, const unsigned cha
{
case value_t::null:
// (the offset of a literal is read to size the output of dump())
if (n.flags != 0 || n.extra != 0 || n.off > text_size)
if ((n.flags & ~node_flags::is_new) != 0 || n.extra != 0 || n.off > text_size)
{
return false;
}
break;
case value_t::boolean:
if ((n.flags & ~node_flags::is_true) != 0 || n.extra != 0 || n.off > text_size)
if ((n.flags & ~(node_flags::is_true | node_flags::is_new)) != 0 || n.extra != 0 || n.off > text_size)
{
return false;
}
@@ -613,7 +619,7 @@ inline bool check_image(const node* nodes, std::size_t count, const unsigned cha
case value_t::number_integer:
case value_t::number_unsigned:
case value_t::number_float:
if (n.flags != 0 || !number_in_bounds(n))
if ((n.flags & ~node_flags::is_new) != 0 || !number_in_bounds(n))
{
return false;
}
@@ -635,7 +641,7 @@ inline bool check_image(const node* nodes, std::size_t count, const unsigned cha
case value_t::object:
{
const std::size_t limit = stack.empty() ? count : stack.back().end;
if (n.flags != 0 || n.extra != 0 || n.next == 0 || n.next > limit - i || n.off > text_size)
if ((n.flags & ~node_flags::is_new) != 0 || n.extra != 0 || n.next == 0 || n.next > limit - i || n.off > text_size)
{
return false;
}