wt/json
1
0 Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-10-29 12:40:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

json_view: source positions in materialize() with JSON_DIAGNOSTIC_POSITIONS (#5945)

Browse Source 
* json_view: source positions in materialize() with JSON_DIAGNOSTIC_POSITIONS

With JSON_DIAGNOSTIC_POSITIONS, materialize() now gives every value the
start_pos()/end_pos() that basic_json::parse() would give it, so exceptions
about materialized values name the same bytes (#5864).

- json_sax_dom_parser takes the lexer type as a defaulted template
  parameter; materialize() passes position_source, which reports the
  positions it derives from the index. diagnostic_positions treats npos as
  "no position" and keeps ranges all-or-nothing.
- Strings with escapes keep the offset of their opening quote plus one in
  `next` (only with the macro); materialize() scans from there to the closing
  quote. The end of an array or object is found by scanning from the end of
  its last element over whitespace, comments and a trailing comma.
- In editable documents, values written by edits, arrays and objects whose
  elements changed, and containers whose last element has no position get
  none. Images keep the flag is_new, and load() checks that `next` of a
  string is within the text.

Nothing changes on the parse path without the macro. The view's own
exceptions still carry no byte range, so the tests that compare them with
those of materialized values skip it under the macro, as they skip the path
under JSON_DIAGNOSTICS.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Indent comment in unit-json_view_from_json.cpp

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Address review: clearer position code and docs

- position_source gets clear(); set_scalar() returns early for new values,
  switches on node_kind, and uses no local copies; the one position_source
  per materialize() call is documented
- no npos + 1 for containers without a position; a comment on why the end
  of a container is reported as its closing bracket
- less duplication under #if in materialize() and the builder
- diagnostic_positions sets npos explicitly
- docs: positions of basic_json values after changes, positions in images
  depend on how the document was parsed, when next is set, and the cost of
  positions while parsing

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Share the comment scanners of the view builder with materialize()

skip_line_comment() and find_block_comment_end() in scan.hpp are used by
the builder's comment() and by position_source::container_end(), which
had its own copy.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Address review: if/else for container positions, clearer next field

- the opening bracket of a container sets positions with an if/else instead
  of a ternary, and the frame takes its start from positions
- the closing bracket of a container is found in nested if/else; the parent's
  last_end is taken from positions, so it is npos whenever no range is set
- architecture.md: the next field reads "or 0 if it was not; all other
  values: 0" instead of "else 0; otherwise 0"

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix GCC unused-result and clang-tidy raw-string findings in unit-json_view_positions.cpp

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann authored and GitHub committed 2026-10-28 20:24:49 +01:00
1 parent 091c10d4a5
commit 54f6b3ef06
17 files changed
+1101 -113
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

single_include/nlohmann/json_view.hpp
Generated
+322 -39
View File
@@ -269,7 +269,7 @@ struct node
std::uint16_t extra; ///< numbers: integer digits (low byte) and fraction digits (high byte), 255 = "many"; objects: number of the object's key hash index (1-based, 0 = none); otherwise 0
std::uint32_t off; ///< source offset (string content, number token, literal, bracket); arena offset if escaped/edited; number of the element sequence if moved
std::uint32_t len; ///< string: decoded bytes; float: token bytes; array/object: element count
std::uint32_t next; ///< array/object: number of nodes of the subtree (its extent in the enclosing sequence)
std::uint32_t next; ///< array/object: number of nodes of the subtree (its extent in the enclosing sequence); escaped string: source offset of the opening quote plus one (JSON_DIAGNOSTIC_POSITIONS; 0: unknown)
};
static_assert(sizeof(node) == 16, "node must stay 16 bytes");
@@ -1185,6 +1185,32 @@ NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* skip_digits(const unsigned char
return p;
}
/// advance over the text of a line comment (after its "//") to the line
/// break or the end (with NulIsEnd also a null byte, which is left for the
/// caller to see, as in parse())
template<bool NulIsEnd = false>
inline const unsigned char* skip_line_comment(const unsigned char* p, const unsigned char* e) noexcept
{
while (p != e && *p != '\n' && *p != '\r' && !(NulIsEnd && *p == 0))
{
++p;
}
return p;
}
/// the "*/" that ends a block comment whose text starts at p (after its "/*");
/// if it is unterminated, the end (with NulIsEnd also a null byte), which is
/// where the caller reports the error
template<bool NulIsEnd = false>
inline const unsigned char* find_block_comment_end(const unsigned char* p, const unsigned char* e) noexcept
{
while (p != e && !(NulIsEnd && *p == 0) && !(*p == '*' && p + 1 != e && p[1] == '/'))
{
++p;
}
return p;
}
/// powers of ten up to 10^19 as integers
inline std::uint64_t int_pow10(unsigned k) noexcept
{
@@ -1382,31 +1408,16 @@ class builder
}
if (p[1] == '/')
{
p += 2;
// (as in parse(), a null byte is the end of the input, so it is
// left for the caller to see)
while (p != m_end && *p != '\n' && *p != '\r' && !(NulIsEnd && *p == 0))
{
++p;
}
return p;
return skip_line_comment<NulIsEnd>(p + 2, m_end);
}
if (p[1] == '*')
{
p += 2;
for (;;)
p = find_block_comment_end<NulIsEnd>(p + 2, m_end);
if (p == m_end || *p != '*')
{
if (p == m_end || (NulIsEnd && *p == 0))
{
return fail_at(error_code::comment_unterminated, p);
}
if (*p == '*' && p + 1 != m_end && p[1] == '/')
{
p += 2;
return p;
}
++p;
return fail_at(error_code::comment_unterminated, p);
}
return p + 2;
}
++p;
return fail_at(error_code::comment_start, p);
@@ -2402,7 +2413,14 @@ indent_done:
return false;
}
p = r.p;
push_node(node_kind::string, node_flags::escaped, 0, r.start, r.len);
#if JSON_DIAGNOSTIC_POSITIONS
// the decoded string is in the arena: next keeps the source offset
// of the opening quote plus one (0: unknown) for materialize()
const std::uint64_t second = static_cast<std::uint64_t>(r.len) | (static_cast<std::uint64_t>(s - b) << 32);
#else
const std::uint64_t second = r.len;
#endif
push_node(node_kind::string, node_flags::escaped, 0, r.start, second);
return true;
}
};
@@ -4943,7 +4961,8 @@ constexpr std::uint64_t image_limit = 0xFFFFFFF0u;
/// Copy the current structure of an edited document into nodes in document
/// order, as the parser would have written them. Text written by edits is
/// appended to text_tail (number tokens) and arena_tail (strings); floats that
/// are not finite become null, as dump() writes them.
/// are not finite become null, as dump() writes them. Nodes keep the flag
/// is_new: they have no source position (JSON_DIAGNOSTIC_POSITIONS).
inline void compact_nodes(const document_data& d, std::size_t arena_size, std::vector<node>& out, std::string& text_tail, std::string& arena_tail)
{
struct frame
@@ -4966,16 +4985,18 @@ inline void compact_nodes(const document_data& d, std::size_t arena_size, std::v
arena_tail.append(d.str(s), s.len);
r.flags = node_flags::escaped;
}
r.flags = static_cast<std::uint8_t>(r.flags | (s.flags & node_flags::is_new));
return r;
};
const auto copy_node = [&](const node * v)
{
node r = *v;
const auto is_new = static_cast<std::uint8_t>(v->flags & node_flags::is_new);
switch (static_cast<value_t>(v->kind))
{
case value_t::object:
case value_t::array:
r.flags = 0;
r.flags = is_new;
r.extra = 0;
r.off = (v->flags & (node_flags::moved | node_flags::is_new)) != 0 ? 0 : v->off;
r.len = 0; // counted below
@@ -4992,7 +5013,7 @@ inline void compact_nodes(const document_data& d, std::size_t arena_size, std::v
r.off = static_cast<std::uint32_t>(d.size + text_tail.size());
text_tail.append(d.str(*v), number_length(*v));
}
r.flags = 0;
r.flags = is_new;
break;
case value_t::number_float:
if ((v->flags & node_flags::storage) == node_flags::edited)
@@ -5002,22 +5023,23 @@ inline void compact_nodes(const document_data& d, std::size_t arena_size, std::v
{
r = node{}; // nan and infinity: null, as dump() writes them
r.kind = node_kind::null;
r.flags = is_new;
break;
}
r.off = static_cast<std::uint32_t>(d.size + text_tail.size());
text_tail.append(t, v->len);
r.extra = 0xFFFFu; // the digit layout is not recorded
}
r.flags = 0;
r.flags = is_new;
break;
case value_t::boolean:
r.flags = static_cast<std::uint8_t>(v->flags & node_flags::is_true);
r.flags = static_cast<std::uint8_t>(v->flags & (node_flags::is_true | node_flags::is_new));
break;
case value_t::null:
case value_t::binary:
case value_t::discarded:
default:
r.flags = 0;
r.flags = is_new;
break;
}
out.push_back(r);
@@ -5378,13 +5400,15 @@ inline bool check_image(const node* nodes, std::size_t count, const unsigned cha
std::vector<float_range> floats;
const auto check_string = [&](const node & n) -> bool
{
if ((n.flags & ~node_flags::escaped) != 0 || n.extra != 0)
if ((n.flags & ~(node_flags::escaped | node_flags::is_new)) != 0 || n.extra != 0)
{
return false;
}
const bool decoded = (n.flags & node_flags::escaped) != 0;
const std::size_t limit = decoded ? arena_size : text_size;
if (n.off > limit || n.len > limit - n.off)
// (next: the source offset of a decoded string's opening quote plus
// one, or 0; see node)
if (n.off > limit || n.len > limit - n.off || n.next > text_size)
{
return false;
}
@@ -5456,13 +5480,13 @@ inline bool check_image(const node* nodes, std::size_t count, const unsigned cha
{
case value_t::null:
// (the offset of a literal is read to size the output of dump())
if (n.flags != 0 || n.extra != 0 || n.off > text_size)
if ((n.flags & ~node_flags::is_new) != 0 || n.extra != 0 || n.off > text_size)
{
return false;
}
break;
case value_t::boolean:
if ((n.flags & ~node_flags::is_true) != 0 || n.extra != 0 || n.off > text_size)
if ((n.flags & ~(node_flags::is_true | node_flags::is_new)) != 0 || n.extra != 0 || n.off > text_size)
{
return false;
}
@@ -5476,7 +5500,7 @@ inline bool check_image(const node* nodes, std::size_t count, const unsigned cha
case value_t::number_integer:
case value_t::number_unsigned:
case value_t::number_float:
if (n.flags != 0 || !number_in_bounds(n))
if ((n.flags & ~node_flags::is_new) != 0 || !number_in_bounds(n))
{
return false;
}
@@ -5498,7 +5522,7 @@ inline bool check_image(const node* nodes, std::size_t count, const unsigned cha
case value_t::object:
{
const std::size_t limit = stack.empty() ? count : stack.back().end;
if (n.flags != 0 || n.extra != 0 || n.next == 0 || n.next > limit - i || n.off > text_size)
if ((n.flags & ~node_flags::is_new) != 0 || n.extra != 0 || n.next == 0 || n.next > limit - i || n.off > text_size)
{
return false;
}
@@ -6101,6 +6125,8 @@ NLOHMANN_JSON_NAMESPACE_END
// #include <nlohmann/detail/view/number.hpp>
// #include <nlohmann/detail/view/scan.hpp>
NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail
@@ -6108,6 +6134,189 @@ namespace detail
namespace view
{
#if JSON_DIAGNOSTIC_POSITIONS
/*!
@brief the source positions of the values materialize() replays
Stands in for the lexer of json_sax_dom_parser, which reads the positions of
the values it stores from these three functions (diagnostic_positions). npos
is a value without a source position. One position_source serves a whole
materialize() call: before each value is handed to the handler, the range of
that value is set (or cleared), overwriting that of the value before.
*/
struct position_source
{
/// the token, of which diagnostic_positions only reads the length
struct token
{
std::size_t length;
std::size_t size() const noexcept
{
return length;
}
};
std::size_t start = std::string::npos; ///< the first byte of the next value
std::size_t end = std::string::npos; ///< one past the next value (start_object/start_array: one past the bracket)
std::size_t get_chars_read_total() const noexcept
{
return end;
}
std::size_t get_token_start_position() const noexcept
{
return start;
}
token get_string() const noexcept
{
return token{end - start};
}
/// the range [first, last), or none if either is npos
void set(std::size_t first, std::size_t last) noexcept
{
if (first == std::string::npos || last == std::string::npos)
{
clear();
return;
}
start = first;
end = last;
}
/// no range: the next value has no source position
void clear() noexcept
{
start = std::string::npos;
end = std::string::npos;
}
/// one past the closing quote of the string whose opening quote is at
/// src[quote], or npos (the scan is bounded by size, as images can be
/// loaded without a check)
static std::size_t string_end(const char* src, std::size_t size, std::size_t quote) noexcept
{
for (std::size_t i = quote + 1; i < size; ++i)
{
if (src[i] == '\\')
{
++i;
}
else if (src[i] == '"')
{
return i + 1;
}
}
return std::string::npos;
}
/// one past the closing bracket of the array or object whose last element
/// ends at src[pos] (or whose opening bracket is at src[pos - 1]), or npos;
/// in between, there can only be whitespace, comments, and a trailing comma
/// (the source was validated when it was parsed)
static std::size_t container_end(const char* src, std::size_t size, std::size_t pos) noexcept
{
const auto* const b = reinterpret_cast<const unsigned char*>(src); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast): the text as bytes, as the builder reads it
const unsigned char* const e = b + size;
while (pos < size)
{
switch (src[pos])
{
case ' ':
case '\t':
case '\n':
case '\r':
case ',':
++pos;
break;
case ']':
case '}':
return pos + 1;
case '/':
if (pos + 1 < size && src[pos + 1] == '/')
{
pos = static_cast<std::size_t>(skip_line_comment(b + pos + 2, e) - b);
break;
}
if (pos + 1 < size && src[pos + 1] == '*')
{
// (unterminated: pos is size, which ends the loop)
pos = static_cast<std::size_t>(find_block_comment_end(b + pos + 2, e) - b) + 2;
break;
}
return std::string::npos;
default:
return std::string::npos;
}
}
return std::string::npos;
}
/// the range of the scalar n of document d
void set_scalar(const document_data& d, const node& n) noexcept
{
if ((n.flags & node_flags::is_new) != 0)
{
clear();
return;
}
std::size_t first = std::string::npos;
std::size_t last = std::string::npos;
switch (n.kind)
{
case node_kind::string:
if ((n.flags & node_flags::storage) == 0)
{
// off is the first byte after the opening quote
if (n.off != 0)
{
first = n.off - 1;
last = n.off + std::size_t{n.len} + 1;
}
}
else if ((n.flags & node_flags::storage) == node_flags::escaped && n.next != 0)
{
// next is the offset of the opening quote plus one
first = n.next - 1;
last = string_end(d.base[0], d.size, first);
}
break;
case node_kind::number_integer:
case node_kind::number_unsigned:
case node_kind::number_float:
if ((n.flags & node_flags::storage) == 0)
{
first = n.off;
last = n.off + number_length(n);
}
break;
case node_kind::boolean:
first = n.off;
last = n.off + ((n.flags & node_flags::is_true) != 0 ? 4u : 5u);
break;
case node_kind::null:
first = n.off;
last = n.off + 4u;
break;
case node_kind::object:
case node_kind::array:
case node_kind::link:
default:
break;
}
// (a crafted image can point past the text)
if (last == std::string::npos || last > d.size)
{
clear();
}
else
{
set(first, last);
}
}
};
#endif
/*!
@brief the basic_json value of the subtree at n
@@ -6115,14 +6324,24 @@ The subtree is replayed into the SAX handler that parse() uses to build its
values, so the result is the value parse() would produce: duplicate keys keep
the last value, and with JSON_DIAGNOSTICS the parent pointers are set. It is
iterative, so the nesting depth is limited by memory only, as for parse().
Without a lexer the handler records no source positions
(JSON_DIAGNOSTIC_POSITIONS).
With JSON_DIAGNOSTIC_POSITIONS, the values get their source positions as with
parse(): position_source stands in for the lexer. Values written by edits have
none (npos), and neither has an array or object whose elements changed, or
whose last element has none (the end of its closing bracket is found by
scanning from the end of the last element).
*/
template<typename BasicJsonType, bool Editable>
BasicJsonType materialize(const document_data& d, const node* n)
{
using string_t = typename BasicJsonType::string_t;
using sax_t = json_sax_dom_parser<BasicJsonType, iterator_input_adapter<const char*>, true>;
using input_t = iterator_input_adapter<const char*>;
#if JSON_DIAGNOSTIC_POSITIONS
using lexer_t = position_source;
#else
using lexer_t = lexer<BasicJsonType, input_t>; // (the default: unused, as no lexer is passed)
#endif
using sax_t = json_sax_dom_parser<BasicJsonType, input_t, true, lexer_t>;
using nav = navigation<Editable>;
// the open containers: their next element (or key of the next member),
@@ -6134,9 +6353,15 @@ BasicJsonType materialize(const document_data& d, const node* n)
const node* pos;
const node* end;
bool object;
#if JSON_DIAGNOSTIC_POSITIONS
std::size_t start; ///< the opening bracket (npos: unknown)
std::size_t last_end; ///< the end of the last element read (npos: unknown)
#endif
};
#if !JSON_DIAGNOSTIC_POSITIONS
// a scalar needs neither the handler nor the stack of open containers
// (with positions, it goes through the handler, which sets them)
switch (static_cast<value_t>(n->kind))
{
case value_t::string:
@@ -6158,10 +6383,16 @@ BasicJsonType materialize(const document_data& d, const node* n)
default:
break;
}
#endif
BasicJsonType result;
#if JSON_DIAGNOSTIC_POSITIONS
position_source positions;
sax_t sax(result, true, &positions);
#else
sax_t sax(result, true);
const string_t no_token{};
#endif
const string_t no_token {};
// (shallow subtrees on the stack: a call for a small value then allocates only for the value)
std::array<frame, 16> shallow; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): filled as containers open
std::vector<frame> deep{};
@@ -6176,6 +6407,16 @@ BasicJsonType materialize(const document_data& d, const node* n)
std::size_t depth = 0;
for (;;)
{
#if JSON_DIAGNOSTIC_POSITIONS
if (!is_container(*n))
{
positions.set_scalar(d, *n);
if (depth != 0)
{
open[depth - 1].last_end = positions.end;
}
}
#endif
// false positive: n comes from nav::value(), which never returns null for a valid index
// @infer-ignore NULLPTR_DEREFERENCE
switch (static_cast<value_t>(n->kind))
@@ -6184,6 +6425,17 @@ BasicJsonType materialize(const document_data& d, const node* n)
case value_t::array:
{
const bool object = n->kind == node_kind::object;
#if JSON_DIAGNOSTIC_POSITIONS
// (moved containers are also new)
if ((n->flags & node_flags::is_new) != 0 || n->off >= d.size)
{
positions.clear();
}
else
{
positions.set(std::size_t{n->off}, std::size_t{n->off} + 1);
}
#endif
if (object)
{
sax.start_object(n->len);
@@ -6200,7 +6452,13 @@ BasicJsonType materialize(const document_data& d, const node* n)
open = deep.data();
capacity = deep.size();
}
open[depth++] = frame{nav::first(d, n), nav::end(d, n), object};
// (with positions: until an element is read, the container's
// own bracket is the last thing read)
open[depth++] = frame{nav::first(d, n), nav::end(d, n), object
#if JSON_DIAGNOSTIC_POSITIONS
, positions.start, positions.end
#endif
};
break;
}
case value_t::string:
@@ -6241,6 +6499,31 @@ BasicJsonType materialize(const document_data& d, const node* n)
frame& f = open[depth - 1];
if (f.pos == f.end)
{
#if JSON_DIAGNOSTIC_POSITIONS
if (f.start == std::string::npos || f.last_end == std::string::npos)
{
positions.clear();
}
else
{
const std::size_t end = position_source::container_end(d.base[0], d.size, f.last_end);
if (end == std::string::npos)
{
positions.clear();
}
else
{
// the closing bracket, as the lexer reports it (end_object()
// and end_array() only read its end; the start of the
// container was set when it was opened)
positions.set(end - 1, end);
}
}
if (depth > 1)
{
open[depth - 2].last_end = positions.end;
}
#endif
if (f.object)
{
sax.end_object();