wt/json
1
0 Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-11-13 17:00:29 +00:00
Code Issues Packages Projects Releases Wiki Activity

Contiguous parser: review fixes (wording, macros, diagnostic positions)

Browse Source 
- "default parser" instead of "library parser" in comments and tests
- the three comments in json.hpp say the same thing
- current() replaces the repeated `p != end ? *p : 0`
- the backslashes of each macro are aligned
- the docs say that JSON_DIAGNOSTIC_POSITIONS turns the fast path off

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann committed 2026-11-07 10:16:20 +01:00
1 parent 7960f33af8
commit 181f41487f
6 files changed
+294 -274
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/mkdocs/docs/api/macros/json_diagnostic_positions.md
+3
View File
@@ -34,6 +34,9 @@ or quotes.
Note that enabling this macro increases the size of every JSON value by two `std::size_t` fields and adds slight runtime Note that enabling this macro increases the size of every JSON value by two `std::size_t` fields and adds slight runtime
overhead to parsing, copying JSON value objects, and the generation of error messages for exceptions. It also causes overhead to parsing, copying JSON value objects, and the generation of error messages for exceptions. It also causes
these values to be reported in those error messages. these values to be reported in those error messages.
In addition, `parse()`, `accept()`, and `sax_parse()` no longer use the
[faster path for input in one block of memory](../../features/parsing/index.md#input-sources-and-performance), so
parsing a string or a buffer is slower than without the macro.
If [`JSON_DIAGNOSTICS`](json_diagnostics.md) is also set, the message contains the path and the byte range, such as If [`JSON_DIAGNOSTICS`](json_diagnostics.md) is also set, the message contains the path and the byte range, such as
`(/1) (bytes 4-8)`. An exception about the root value has no path and only the range, such as `(bytes 0-4)`. `(/1) (bytes 4-8)`. An exception about the root value has no path and only the range, such as `(bytes 0-4)`.
docs/mkdocs/docs/features/parsing/index.md
+3
View File
@@ -84,6 +84,9 @@ Input in one block of memory is read even faster with the default options (no
[parser callback](../../api/basic_json/parser_callback_t.md), no comments, no trailing commas, and for `sax_parse()` [parser callback](../../api/basic_json/parser_callback_t.md), no comments, no trailing commas, and for `sax_parse()`
`strict = true`): `parse()`, `accept()`, and `sax_parse()` then read the grammar and the tokens in one loop over the `strict = true`): `parse()`, `accept()`, and `sax_parse()` then read the grammar and the tokens in one loop over the
memory. Only after an error do `parse()` and `sax_parse()` read the input again, to report the error exactly as before. memory. Only after an error do `parse()` and `sax_parse()` read the input again, to report the error exactly as before.
This faster path is not used when [`JSON_DIAGNOSTIC_POSITIONS`](../../api/macros/json_diagnostic_positions.md) is
enabled, because the positions are recorded by the default parser: the input is then parsed as before, at the same speed
as before.
Parsing a file from a `#!cpp std::ifstream` therefore takes about 1.15 to 1.45 times as long as reading the whole file Parsing a file from a `#!cpp std::ifstream` therefore takes about 1.15 to 1.45 times as long as reading the whole file
into a `#!cpp std::string` first and parsing that, but it needs no memory for a copy of the file. The stream is never read into a `#!cpp std::string` first and parsing that, but it needs no memory for a copy of the file. The stream is never read
include/nlohmann/detail/input/contiguous_parser.hpp
+137 -131
View File
@@ -37,8 +37,8 @@ namespace detail
@brief a parser for JSON text in one contiguous block, for parse(), accept(), @brief a parser for JSON text in one contiguous block, for parse(), accept(),
and sax_parse() and sax_parse()
The parser and the lexer exchange one token at a time, and the lexer keeps its The default parser and lexer exchange one token at a time, and the lexer keeps
state (the current character, a pending unget, where the token starts) in its state (the current character, a pending unget, where the token starts) in
members, which every character pays for. This parser reads the block through a members, which every character pays for. This parser reads the block through a
local pointer instead, and the grammar, the whitespace and the tokens are one local pointer instead, and the grammar, the whitespace and the tokens are one
loop, after yyjson's reader: the state stays in registers, and the expected loop, after yyjson's reader: the state stays in registers, and the expected
@@ -46,13 +46,13 @@ next token is tested where it is expected instead of being dispatched on.
It reports what it reads to a SAX handler: the one parse() uses to build the It reports what it reads to a SAX handler: the one parse() uses to build the
value, the one accept() uses, or the user's handler of sax_parse(). On a byte value, the one accept() uses, or the user's handler of sax_parse(). On a byte
the library parser would reject, it stops before reporting the token, and the default parser would reject, it stops before reporting the token, and
returns false: the events up to there are those the library parser reports. returns false: the events up to there are those the default parser reports.
The caller then runs the library parser on the same input to report the error The caller then runs the default parser on the same input to report the error
with its exact message and position (report_contiguous_error(), with its exact message and position (report_contiguous_error(),
json_sax_error_forwarder), and parse() builds no value from what was read. json_sax_error_forwarder), and parse() builds no value from what was read.
What is accepted, and the values reported, are those of the library parser What is accepted, and the values reported, are those of the default parser
with default options: the same number grammar and conversions, the same UTF-8 with default options: the same number grammar and conversions, the same UTF-8
and escape rules, a UTF-8 byte order mark at the start, and, without and escape rules, a UTF-8 byte order mark at the start, and, without
JSON_STRICT_NUL_HANDLING, a NUL byte as the end of the input. JSON_STRICT_NUL_HANDLING, a NUL byte as the end of the input.
@@ -106,28 +106,28 @@ class contiguous_parser
std::vector<std::uint64_t> outer; std::vector<std::uint64_t> outer;
// the current byte, or 0 at the end (no JSON token starts with 0) // the current byte, or 0 at the end (no JSON token starts with 0)
#define JSON_CONTIGUOUS_CUR() (p != e ? *p : 0) #define JSON_CONTIGUOUS_CUR() current(p, e)
// skip whitespace; most tokens have none before them // skip whitespace; most tokens have none before them
#define JSON_CONTIGUOUS_WS() \ #define JSON_CONTIGUOUS_WS() \
do \ do \
{ \ { \
if (JSON_HEDLEY_UNLIKELY(JSON_CONTIGUOUS_CUR() <= ' ')) \ if (JSON_HEDLEY_UNLIKELY(JSON_CONTIGUOUS_CUR() <= ' ')) \
{ \ { \
p = skip_whitespace(p, e); \ p = skip_whitespace(p, e); \
} \ } \
} \ } \
while (false) while (false)
#define JSON_CONTIGUOUS_OPEN(is_object) \ #define JSON_CONTIGUOUS_OPEN(is_object) \
do \ do \
{ \ { \
if (JSON_HEDLEY_UNLIKELY(depth != 0 && (depth % 64) == 0)) /* NOLINT(readability/braces) */ \ if (JSON_HEDLEY_UNLIKELY(depth != 0 && (depth % 64) == 0)) /* NOLINT(readability/braces) */ \
{ \ { \
outer.push_back(kinds); \ outer.push_back(kinds); \
kinds = 0; \ kinds = 0; \
} \ } \
kinds = (kinds << 1u) | static_cast<std::uint64_t>(is_object); \ kinds = (kinds << 1u) | static_cast<std::uint64_t>(is_object); \
++depth; \ ++depth; \
} \ } \
while (false) while (false)
JSON_CONTIGUOUS_WS(); JSON_CONTIGUOUS_WS();
@@ -160,107 +160,107 @@ class contiguous_parser
// a value in an array or an object: expanded once for each, so that // a value in an array or an object: expanded once for each, so that
// each dispatch has its own branch history (arrays tend to hold values // each dispatch has its own branch history (arrays tend to hold values
// of one kind) and the code after a scalar knows where it is // of one kind) and the code after a scalar knows where it is
#define JSON_CONTIGUOUS_VALUE \ #define JSON_CONTIGUOUS_VALUE \
switch (JSON_CONTIGUOUS_CUR()) \ switch (JSON_CONTIGUOUS_CUR()) \
{ \ { \
case '"': \ case '"': \
if (JSON_HEDLEY_UNLIKELY(!string(p, false))) \ if (JSON_HEDLEY_UNLIKELY(!string(p, false))) \
{ \ { \
return false; \ return false; \
} \ } \
break; \ break; \
case '{': \ case '{': \
++p; \ ++p; \
if (JSON_HEDLEY_UNLIKELY(!report(sax->start_object(detail::unknown_size())))) \ if (JSON_HEDLEY_UNLIKELY(!report(sax->start_object(detail::unknown_size())))) \
{ \ { \
return false; \ return false; \
} \ } \
JSON_CONTIGUOUS_OPEN(1u); \ JSON_CONTIGUOUS_OPEN(1u); \
goto object_first; \ goto object_first; \
case '[': \ case '[': \
++p; \ ++p; \
if (JSON_HEDLEY_UNLIKELY(!report(sax->start_array(detail::unknown_size())))) \ if (JSON_HEDLEY_UNLIKELY(!report(sax->start_array(detail::unknown_size())))) \
{ \ { \
return false; \ return false; \
} \ } \
JSON_CONTIGUOUS_OPEN(0u); \ JSON_CONTIGUOUS_OPEN(0u); \
goto array_first; \ goto array_first; \
case '-': \ case '-': \
case '0': \ case '0': \
case '1': \ case '1': \
case '2': \ case '2': \
case '3': \ case '3': \
case '4': \ case '4': \
case '5': \ case '5': \
case '6': \ case '6': \
case '7': \ case '7': \
case '8': \ case '8': \
case '9': \ case '9': \
if (JSON_HEDLEY_UNLIKELY(!number(p))) \ if (JSON_HEDLEY_UNLIKELY(!number(p))) \
{ \ { \
return false; \ return false; \
} \ } \
break; \ break; \
case 't': \ case 't': \
if (JSON_HEDLEY_UNLIKELY(!literal(p, "true") || !report(sax->boolean(true)))) \ if (JSON_HEDLEY_UNLIKELY(!literal(p, "true") || !report(sax->boolean(true)))) \
{ \ { \
return false; \ return false; \
} \ } \
break; \ break; \
case 'f': \ case 'f': \
if (JSON_HEDLEY_UNLIKELY(!literal(p, "false") || !report(sax->boolean(false)))) \ if (JSON_HEDLEY_UNLIKELY(!literal(p, "false") || !report(sax->boolean(false)))) \
{ \ { \
return false; \ return false; \
} \ } \
break; \ break; \
case 'n': \ case 'n': \
if (JSON_HEDLEY_UNLIKELY(!literal(p, "null") || !report(sax->null()))) \ if (JSON_HEDLEY_UNLIKELY(!literal(p, "null") || !report(sax->null()))) \
{ \ { \
return false; \ return false; \
} \ } \
break; \ break; \
default: \ default: \
return false; \ return false; \
} }
// what may follow a value in an array: a comma and the next value, or the end // what may follow a value in an array: a comma and the next value, or the end
#define JSON_CONTIGUOUS_AFTER_ELEMENT \ #define JSON_CONTIGUOUS_AFTER_ELEMENT \
JSON_CONTIGUOUS_WS(); \ JSON_CONTIGUOUS_WS(); \
if (JSON_HEDLEY_LIKELY(JSON_CONTIGUOUS_CUR() == ',')) /* NOLINT(readability/braces) */ \ if (JSON_HEDLEY_LIKELY(JSON_CONTIGUOUS_CUR() == ',')) /* NOLINT(readability/braces) */ \
{ \ { \
++p; \ ++p; \
JSON_CONTIGUOUS_WS(); \ JSON_CONTIGUOUS_WS(); \
goto array_value; \ goto array_value; \
} \ } \
if (JSON_HEDLEY_LIKELY(JSON_CONTIGUOUS_CUR() == ']')) /* NOLINT(readability/braces) */ \ if (JSON_HEDLEY_LIKELY(JSON_CONTIGUOUS_CUR() == ']')) /* NOLINT(readability/braces) */ \
{ \ { \
++p; \ ++p; \
if (JSON_HEDLEY_UNLIKELY(!report(sax->end_array()))) \ if (JSON_HEDLEY_UNLIKELY(!report(sax->end_array()))) \
{ \ { \
return false; \ return false; \
} \ } \
goto close; \ goto close; \
} \ } \
return false; return false;
// what may follow a value in an object: a comma and the next member, or the end // what may follow a value in an object: a comma and the next member, or the end
#define JSON_CONTIGUOUS_AFTER_MEMBER \ #define JSON_CONTIGUOUS_AFTER_MEMBER \
JSON_CONTIGUOUS_WS(); \ JSON_CONTIGUOUS_WS(); \
if (JSON_HEDLEY_LIKELY(JSON_CONTIGUOUS_CUR() == ',')) /* NOLINT(readability/braces) */ \ if (JSON_HEDLEY_LIKELY(JSON_CONTIGUOUS_CUR() == ',')) /* NOLINT(readability/braces) */ \
{ \ { \
++p; \ ++p; \
JSON_CONTIGUOUS_WS(); \ JSON_CONTIGUOUS_WS(); \
goto object_key; \ goto object_key; \
} \ } \
if (JSON_HEDLEY_LIKELY(JSON_CONTIGUOUS_CUR() == '}')) /* NOLINT(readability/braces) */ \ if (JSON_HEDLEY_LIKELY(JSON_CONTIGUOUS_CUR() == '}')) /* NOLINT(readability/braces) */ \
{ \ { \
++p; \ ++p; \
if (JSON_HEDLEY_UNLIKELY(!report(sax->end_object()))) \ if (JSON_HEDLEY_UNLIKELY(!report(sax->end_object()))) \
{ \ { \
return false; \ return false; \
} \ } \
goto close; \ goto close; \
} \ } \
return false; return false;
array_first: array_first:
@@ -387,6 +387,12 @@ done:
return KeepValues; return KeepValues;
} }
/// the byte at @a p, or 0 at the end @a e (no JSON token starts with 0)
static JSON_HEDLEY_ALWAYS_INLINE uchar current(const uchar* p, const uchar* e) noexcept
{
return p != e ? *p : 0;
}
/// the first byte at or after p that is not whitespace, or e /// the first byte at or after p that is not whitespace, or e
static JSON_HEDLEY_NEVER_INLINE const uchar* skip_whitespace(const uchar* p, const uchar* e) noexcept static JSON_HEDLEY_NEVER_INLINE const uchar* skip_whitespace(const uchar* p, const uchar* e) noexcept
{ {
@@ -436,7 +442,7 @@ done:
/// a JSON text that is a single scalar /// a JSON text that is a single scalar
bool scalar(const uchar*& p) bool scalar(const uchar*& p)
{ {
switch (p != end ? *p : 0) switch (current(p, end))
{ {
case '"': case '"':
return string(p, false); return string(p, false);
@@ -624,7 +630,7 @@ done:
contiguous path (lexer::scan_number_bulk_contiguous()) contiguous path (lexer::scan_number_bulk_contiguous())
Integers that do not fit their type become floats; floats that overflow Integers that do not fit their type become floats; floats that overflow
are an error, which the library parser reports. are an error, which the default parser reports.
*/ */
JSON_HEDLEY_ALWAYS_INLINE bool number(const uchar*& pos) JSON_HEDLEY_ALWAYS_INLINE bool number(const uchar*& pos)
{ {
@@ -773,7 +779,7 @@ done:
} }
if (JSON_HEDLEY_UNLIKELY(!std::isfinite(value))) if (JSON_HEDLEY_UNLIKELY(!std::isfinite(value)))
{ {
return false; // the library parser reports the overflow return false; // the default parser reports the overflow
} }
return report(sax->number_float(value, number_text(data, data + i))); return report(sax->number_float(value, number_text(data, data + i)));
} }
@@ -811,7 +817,7 @@ struct can_parse_contiguous
@brief report the input behind @a adapter, one block of memory, to @a sax @brief report the input behind @a adapter, one block of memory, to @a sax
@return whether the input is valid and was reported completely; if not, @return whether the input is valid and was reported completely; if not,
nothing was consumed from @a adapter, and the caller has the library nothing was consumed from @a adapter, and the caller has the default
parser report the error parser report the error
*/ */
template<typename BasicJsonType, bool KeepValues, typename SAX, typename InputAdapterType> template<typename BasicJsonType, bool KeepValues, typename SAX, typename InputAdapterType>
@@ -820,7 +826,7 @@ bool parse_contiguous(InputAdapterType& adapter, SAX& sax, std::true_type /*cont
const std::size_t size = adapter.bulk_remaining(); const std::size_t size = adapter.bulk_remaining();
if (size == 0) if (size == 0)
{ {
return false; // the library parser reports the empty input return false; // the default parser reports the empty input
} }
const auto* const first = reinterpret_cast<const unsigned char*>(adapter.bulk_data()); const auto* const first = reinterpret_cast<const unsigned char*>(adapter.bulk_data());
contiguous_parser<BasicJsonType, SAX, KeepValues> p(first, first + size, &sax); contiguous_parser<BasicJsonType, SAX, KeepValues> p(first, first + size, &sax);
@@ -831,7 +837,7 @@ bool parse_contiguous(InputAdapterType& adapter, SAX& sax, std::true_type /*cont
@brief a SAX handler that only reports a parse error, as the handler of parse() @brief a SAX handler that only reports a parse error, as the handler of parse()
does: it throws the exception, unless exceptions are not allowed does: it throws the exception, unless exceptions are not allowed
After contiguous_parser has rejected an input, the library parser is run again After contiguous_parser has rejected an input, the default parser is run again
with this handler to report the error: the error, its message and its position with this handler to report the error: the error, its message and its position
are those of parse(), without building the values read up to it. are those of parse(), without building the values read up to it.
*/ */
@@ -868,7 +874,7 @@ struct sax_ignores_number_text<json_sax_error_reporter<BasicJsonType>> : std::tr
rejected, as parse() reports it: throw its exception, or, without rejected, as parse() reports it: throw its exception, or, without
exceptions, return true exceptions, return true
@return false if the library parser accepts the input after all (which would @return false if the default parser accepts the input after all (which would
be a disagreement of the two parsers): the caller then parses it be a disagreement of the two parsers): the caller then parses it
*/ */
template<typename BasicJsonType, typename InputAdapterType> template<typename BasicJsonType, typename InputAdapterType>
@@ -922,7 +928,7 @@ bool parse_contiguous(InputAdapterType& /*adapter*/, BasicJsonType& /*result*/,
@return whether @a accepted is the result of accept(); false if the input is @return whether @a accepted is the result of accept(); false if the input is
not one block of memory, in which case the caller checks it. not one block of memory, in which case the caller checks it.
contiguous_parser accepts exactly what the library parser accepts. contiguous_parser accepts exactly what the default parser accepts.
*/ */
template<typename BasicJsonType, typename InputAdapterType> template<typename BasicJsonType, typename InputAdapterType>
bool accept_contiguous(InputAdapterType& adapter, bool& accepted, std::true_type /*contiguous*/) bool accept_contiguous(InputAdapterType& adapter, bool& accepted, std::true_type /*contiguous*/)
@@ -943,9 +949,9 @@ bool accept_contiguous(InputAdapterType& /*adapter*/, bool& /*accepted*/, std::f
handler of the user handler of the user
After contiguous_parser has delivered the events of the input to the user's After contiguous_parser has delivered the events of the input to the user's
handler and rejected it, the library parser is run again with this handler: handler and rejected it, the default parser is run again with this handler:
the user's handler gets the error, its message and its position from the the user's handler gets the error, its message and its position from the
library parser, as before, but no event twice. default parser, as before, but no event twice.
*/ */
template<typename BasicJsonType, typename SAX> template<typename BasicJsonType, typename SAX>
class json_sax_error_forwarder : public json_sax_acceptor<BasicJsonType> class json_sax_error_forwarder : public json_sax_acceptor<BasicJsonType>
@@ -962,7 +968,7 @@ class json_sax_error_forwarder : public json_sax_acceptor<BasicJsonType>
return user->parse_error(position, last_token, ex); return user->parse_error(position, last_token, ex);
} }
/// whether the library parser reported an error /// whether the default parser reported an error
bool reported() const noexcept bool reported() const noexcept
{ {
return forwarded; return forwarded;
@@ -981,8 +987,8 @@ struct sax_ignores_number_text<json_sax_error_forwarder<BasicJsonType, SAX>> : s
a user's handler @a sax a user's handler @a sax
contiguous_parser reports to @a sax what it reads, and stops at the token that contiguous_parser reports to @a sax what it reads, and stops at the token that
the library parser would reject, before reporting it: the events up to an error the default parser would reject, before reporting it: the events up to an error
are the same. The error itself is then reported by the library parser (see are the same. The error itself is then reported by the default parser (see
json_sax_error_forwarder). json_sax_error_forwarder).
@return whether @a result is the result of sax_parse(); false if the input is @return whether @a result is the result of sax_parse(); false if the input is
@@ -1003,14 +1009,14 @@ bool sax_parse_contiguous(InputAdapterType& adapter, SAX* sax, bool& result, std
} }
if (p.aborted()) if (p.aborted())
{ {
// the handler returned false, as with the library parser // the handler returned false, as with the default parser
result = false; result = false;
return true; return true;
} }
} }
json_sax_error_forwarder<BasicJsonType, SAX> forwarder(sax); json_sax_error_forwarder<BasicJsonType, SAX> forwarder(sax);
result = parser<BasicJsonType, InputAdapterType>(InputAdapterType(adapter), nullptr, true, false, false).sax_parse(&forwarder, true); result = parser<BasicJsonType, InputAdapterType>(InputAdapterType(adapter), nullptr, true, false, false).sax_parse(&forwarder, true);
// the library parser must reject the input as well (the events after the // the default parser must reject the input as well (the events after the
// byte contiguous_parser stopped at cannot be delivered any more) // byte contiguous_parser stopped at cannot be delivered any more)
JSON_ASSERT(forwarder.reported()); JSON_ASSERT(forwarder.reported());
return true; return true;
include/nlohmann/json.hpp
+6 -5
View File
@@ -3111,7 +3111,7 @@ public:
{ {
basic_json result; basic_json result;
// JSON in one block of memory, with the default options, is read by // JSON in one block of memory, with the default options, is read by
// the faster contiguous_parser; on an error, the library parser reads // the faster contiguous_parser; on an error, the default parser reads
// it again to report it (see contiguous_parser) // it again to report it (see contiguous_parser)
using contiguous = std::integral_constant<bool, detail::can_parse_contiguous<basic_json, typename std::decay<InputAdapterType>::type>::value>; using contiguous = std::integral_constant<bool, detail::can_parse_contiguous<basic_json, typename std::decay<InputAdapterType>::type>::value>;
if (cb == nullptr && !ignore_comments && !ignore_trailing_commas if (cb == nullptr && !ignore_comments && !ignore_trailing_commas
@@ -3129,8 +3129,8 @@ public:
const bool ignore_comments, const bool ignore_comments,
const bool ignore_trailing_commas) const bool ignore_trailing_commas)
{ {
// JSON in one block of memory is checked by the faster // JSON in one block of memory, with the default options, is checked by
// contiguous_parser (see parse_impl()) // the faster contiguous_parser (see parse_impl())
using adapter_t = typename std::decay<InputAdapterType>::type; using adapter_t = typename std::decay<InputAdapterType>::type;
bool accepted = false; bool accepted = false;
if (!ignore_comments && !ignore_trailing_commas if (!ignore_comments && !ignore_trailing_commas
@@ -3154,8 +3154,9 @@ public:
{ {
if (format == input_format_t::json) if (format == input_format_t::json)
{ {
// JSON in one block of memory is read by the faster // JSON in one block of memory, with the default options, is read by
// contiguous_parser, with the default options (see parse_impl()) // the faster contiguous_parser; on an error, the default parser
// reads it again to report it (see parse_impl())
bool result = false; bool result = false;
if (strict && !ignore_comments && !ignore_trailing_commas if (strict && !ignore_comments && !ignore_trailing_commas
&& detail::sax_parse_contiguous<basic_json>(ia, sax, result, std::integral_constant<bool, detail::can_parse_contiguous<basic_json, InputAdapterType>::value> {})) && detail::sax_parse_contiguous<basic_json>(ia, sax, result, std::integral_constant<bool, detail::can_parse_contiguous<basic_json, InputAdapterType>::value> {}))
single_include/nlohmann/json.hpp
Generated
+143 -136
View File
@@ -23903,8 +23903,8 @@ namespace detail
@brief a parser for JSON text in one contiguous block, for parse(), accept(), @brief a parser for JSON text in one contiguous block, for parse(), accept(),
and sax_parse() and sax_parse()
The parser and the lexer exchange one token at a time, and the lexer keeps its The default parser and lexer exchange one token at a time, and the lexer keeps
state (the current character, a pending unget, where the token starts) in its state (the current character, a pending unget, where the token starts) in
members, which every character pays for. This parser reads the block through a members, which every character pays for. This parser reads the block through a
local pointer instead, and the grammar, the whitespace and the tokens are one local pointer instead, and the grammar, the whitespace and the tokens are one
loop, after yyjson's reader: the state stays in registers, and the expected loop, after yyjson's reader: the state stays in registers, and the expected
@@ -23912,13 +23912,13 @@ next token is tested where it is expected instead of being dispatched on.
It reports what it reads to a SAX handler: the one parse() uses to build the It reports what it reads to a SAX handler: the one parse() uses to build the
value, the one accept() uses, or the user's handler of sax_parse(). On a byte value, the one accept() uses, or the user's handler of sax_parse(). On a byte
the library parser would reject, it stops before reporting the token, and the default parser would reject, it stops before reporting the token, and
returns false: the events up to there are those the library parser reports. returns false: the events up to there are those the default parser reports.
The caller then runs the library parser on the same input to report the error The caller then runs the default parser on the same input to report the error
with its exact message and position (report_contiguous_error(), with its exact message and position (report_contiguous_error(),
json_sax_error_forwarder), and parse() builds no value from what was read. json_sax_error_forwarder), and parse() builds no value from what was read.
What is accepted, and the values reported, are those of the library parser What is accepted, and the values reported, are those of the default parser
with default options: the same number grammar and conversions, the same UTF-8 with default options: the same number grammar and conversions, the same UTF-8
and escape rules, a UTF-8 byte order mark at the start, and, without and escape rules, a UTF-8 byte order mark at the start, and, without
JSON_STRICT_NUL_HANDLING, a NUL byte as the end of the input. JSON_STRICT_NUL_HANDLING, a NUL byte as the end of the input.
@@ -23972,28 +23972,28 @@ class contiguous_parser
std::vector<std::uint64_t> outer; std::vector<std::uint64_t> outer;
// the current byte, or 0 at the end (no JSON token starts with 0) // the current byte, or 0 at the end (no JSON token starts with 0)
#define JSON_CONTIGUOUS_CUR() (p != e ? *p : 0) #define JSON_CONTIGUOUS_CUR() current(p, e)
// skip whitespace; most tokens have none before them // skip whitespace; most tokens have none before them
#define JSON_CONTIGUOUS_WS() \ #define JSON_CONTIGUOUS_WS() \
do \ do \
{ \ { \
if (JSON_HEDLEY_UNLIKELY(JSON_CONTIGUOUS_CUR() <= ' ')) \ if (JSON_HEDLEY_UNLIKELY(JSON_CONTIGUOUS_CUR() <= ' ')) \
{ \ { \
p = skip_whitespace(p, e); \ p = skip_whitespace(p, e); \
} \ } \
} \ } \
while (false) while (false)
#define JSON_CONTIGUOUS_OPEN(is_object) \ #define JSON_CONTIGUOUS_OPEN(is_object) \
do \ do \
{ \ { \
if (JSON_HEDLEY_UNLIKELY(depth != 0 && (depth % 64) == 0)) /* NOLINT(readability/braces) */ \ if (JSON_HEDLEY_UNLIKELY(depth != 0 && (depth % 64) == 0)) /* NOLINT(readability/braces) */ \
{ \ { \
outer.push_back(kinds); \ outer.push_back(kinds); \
kinds = 0; \ kinds = 0; \
} \ } \
kinds = (kinds << 1u) | static_cast<std::uint64_t>(is_object); \ kinds = (kinds << 1u) | static_cast<std::uint64_t>(is_object); \
++depth; \ ++depth; \
} \ } \
while (false) while (false)
JSON_CONTIGUOUS_WS(); JSON_CONTIGUOUS_WS();
@@ -24026,107 +24026,107 @@ class contiguous_parser
// a value in an array or an object: expanded once for each, so that // a value in an array or an object: expanded once for each, so that
// each dispatch has its own branch history (arrays tend to hold values // each dispatch has its own branch history (arrays tend to hold values
// of one kind) and the code after a scalar knows where it is // of one kind) and the code after a scalar knows where it is
#define JSON_CONTIGUOUS_VALUE \ #define JSON_CONTIGUOUS_VALUE \
switch (JSON_CONTIGUOUS_CUR()) \ switch (JSON_CONTIGUOUS_CUR()) \
{ \ { \
case '"': \ case '"': \
if (JSON_HEDLEY_UNLIKELY(!string(p, false))) \ if (JSON_HEDLEY_UNLIKELY(!string(p, false))) \
{ \ { \
return false; \ return false; \
} \ } \
break; \ break; \
case '{': \ case '{': \
++p; \ ++p; \
if (JSON_HEDLEY_UNLIKELY(!report(sax->start_object(detail::unknown_size())))) \ if (JSON_HEDLEY_UNLIKELY(!report(sax->start_object(detail::unknown_size())))) \
{ \ { \
return false; \ return false; \
} \ } \
JSON_CONTIGUOUS_OPEN(1u); \ JSON_CONTIGUOUS_OPEN(1u); \
goto object_first; \ goto object_first; \
case '[': \ case '[': \
++p; \ ++p; \
if (JSON_HEDLEY_UNLIKELY(!report(sax->start_array(detail::unknown_size())))) \ if (JSON_HEDLEY_UNLIKELY(!report(sax->start_array(detail::unknown_size())))) \
{ \ { \
return false; \ return false; \
} \ } \
JSON_CONTIGUOUS_OPEN(0u); \ JSON_CONTIGUOUS_OPEN(0u); \
goto array_first; \ goto array_first; \
case '-': \ case '-': \
case '0': \ case '0': \
case '1': \ case '1': \
case '2': \ case '2': \
case '3': \ case '3': \
case '4': \ case '4': \
case '5': \ case '5': \
case '6': \ case '6': \
case '7': \ case '7': \
case '8': \ case '8': \
case '9': \ case '9': \
if (JSON_HEDLEY_UNLIKELY(!number(p))) \ if (JSON_HEDLEY_UNLIKELY(!number(p))) \
{ \ { \
return false; \ return false; \
} \ } \
break; \ break; \
case 't': \ case 't': \
if (JSON_HEDLEY_UNLIKELY(!literal(p, "true") || !report(sax->boolean(true)))) \ if (JSON_HEDLEY_UNLIKELY(!literal(p, "true") || !report(sax->boolean(true)))) \
{ \ { \
return false; \ return false; \
} \ } \
break; \ break; \
case 'f': \ case 'f': \
if (JSON_HEDLEY_UNLIKELY(!literal(p, "false") || !report(sax->boolean(false)))) \ if (JSON_HEDLEY_UNLIKELY(!literal(p, "false") || !report(sax->boolean(false)))) \
{ \ { \
return false; \ return false; \
} \ } \
break; \ break; \
case 'n': \ case 'n': \
if (JSON_HEDLEY_UNLIKELY(!literal(p, "null") || !report(sax->null()))) \ if (JSON_HEDLEY_UNLIKELY(!literal(p, "null") || !report(sax->null()))) \
{ \ { \
return false; \ return false; \
} \ } \
break; \ break; \
default: \ default: \
return false; \ return false; \
} }
// what may follow a value in an array: a comma and the next value, or the end // what may follow a value in an array: a comma and the next value, or the end
#define JSON_CONTIGUOUS_AFTER_ELEMENT \ #define JSON_CONTIGUOUS_AFTER_ELEMENT \
JSON_CONTIGUOUS_WS(); \ JSON_CONTIGUOUS_WS(); \
if (JSON_HEDLEY_LIKELY(JSON_CONTIGUOUS_CUR() == ',')) /* NOLINT(readability/braces) */ \ if (JSON_HEDLEY_LIKELY(JSON_CONTIGUOUS_CUR() == ',')) /* NOLINT(readability/braces) */ \
{ \ { \
++p; \ ++p; \
JSON_CONTIGUOUS_WS(); \ JSON_CONTIGUOUS_WS(); \
goto array_value; \ goto array_value; \
} \ } \
if (JSON_HEDLEY_LIKELY(JSON_CONTIGUOUS_CUR() == ']')) /* NOLINT(readability/braces) */ \ if (JSON_HEDLEY_LIKELY(JSON_CONTIGUOUS_CUR() == ']')) /* NOLINT(readability/braces) */ \
{ \ { \
++p; \ ++p; \
if (JSON_HEDLEY_UNLIKELY(!report(sax->end_array()))) \ if (JSON_HEDLEY_UNLIKELY(!report(sax->end_array()))) \
{ \ { \
return false; \ return false; \
} \ } \
goto close; \ goto close; \
} \ } \
return false; return false;
// what may follow a value in an object: a comma and the next member, or the end // what may follow a value in an object: a comma and the next member, or the end
#define JSON_CONTIGUOUS_AFTER_MEMBER \ #define JSON_CONTIGUOUS_AFTER_MEMBER \
JSON_CONTIGUOUS_WS(); \ JSON_CONTIGUOUS_WS(); \
if (JSON_HEDLEY_LIKELY(JSON_CONTIGUOUS_CUR() == ',')) /* NOLINT(readability/braces) */ \ if (JSON_HEDLEY_LIKELY(JSON_CONTIGUOUS_CUR() == ',')) /* NOLINT(readability/braces) */ \
{ \ { \
++p; \ ++p; \
JSON_CONTIGUOUS_WS(); \ JSON_CONTIGUOUS_WS(); \
goto object_key; \ goto object_key; \
} \ } \
if (JSON_HEDLEY_LIKELY(JSON_CONTIGUOUS_CUR() == '}')) /* NOLINT(readability/braces) */ \ if (JSON_HEDLEY_LIKELY(JSON_CONTIGUOUS_CUR() == '}')) /* NOLINT(readability/braces) */ \
{ \ { \
++p; \ ++p; \
if (JSON_HEDLEY_UNLIKELY(!report(sax->end_object()))) \ if (JSON_HEDLEY_UNLIKELY(!report(sax->end_object()))) \
{ \ { \
return false; \ return false; \
} \ } \
goto close; \ goto close; \
} \ } \
return false; return false;
array_first: array_first:
@@ -24253,6 +24253,12 @@ done:
return KeepValues; return KeepValues;
} }
/// the byte at @a p, or 0 at the end @a e (no JSON token starts with 0)
static JSON_HEDLEY_ALWAYS_INLINE uchar current(const uchar* p, const uchar* e) noexcept
{
return p != e ? *p : 0;
}
/// the first byte at or after p that is not whitespace, or e /// the first byte at or after p that is not whitespace, or e
static JSON_HEDLEY_NEVER_INLINE const uchar* skip_whitespace(const uchar* p, const uchar* e) noexcept static JSON_HEDLEY_NEVER_INLINE const uchar* skip_whitespace(const uchar* p, const uchar* e) noexcept
{ {
@@ -24302,7 +24308,7 @@ done:
/// a JSON text that is a single scalar /// a JSON text that is a single scalar
bool scalar(const uchar*& p) bool scalar(const uchar*& p)
{ {
switch (p != end ? *p : 0) switch (current(p, end))
{ {
case '"': case '"':
return string(p, false); return string(p, false);
@@ -24490,7 +24496,7 @@ done:
contiguous path (lexer::scan_number_bulk_contiguous()) contiguous path (lexer::scan_number_bulk_contiguous())
Integers that do not fit their type become floats; floats that overflow Integers that do not fit their type become floats; floats that overflow
are an error, which the library parser reports. are an error, which the default parser reports.
*/ */
JSON_HEDLEY_ALWAYS_INLINE bool number(const uchar*& pos) JSON_HEDLEY_ALWAYS_INLINE bool number(const uchar*& pos)
{ {
@@ -24639,7 +24645,7 @@ done:
} }
if (JSON_HEDLEY_UNLIKELY(!std::isfinite(value))) if (JSON_HEDLEY_UNLIKELY(!std::isfinite(value)))
{ {
return false; // the library parser reports the overflow return false; // the default parser reports the overflow
} }
return report(sax->number_float(value, number_text(data, data + i))); return report(sax->number_float(value, number_text(data, data + i)));
} }
@@ -24677,7 +24683,7 @@ struct can_parse_contiguous
@brief report the input behind @a adapter, one block of memory, to @a sax @brief report the input behind @a adapter, one block of memory, to @a sax
@return whether the input is valid and was reported completely; if not, @return whether the input is valid and was reported completely; if not,
nothing was consumed from @a adapter, and the caller has the library nothing was consumed from @a adapter, and the caller has the default
parser report the error parser report the error
*/ */
template<typename BasicJsonType, bool KeepValues, typename SAX, typename InputAdapterType> template<typename BasicJsonType, bool KeepValues, typename SAX, typename InputAdapterType>
@@ -24686,7 +24692,7 @@ bool parse_contiguous(InputAdapterType& adapter, SAX& sax, std::true_type /*cont
const std::size_t size = adapter.bulk_remaining(); const std::size_t size = adapter.bulk_remaining();
if (size == 0) if (size == 0)
{ {
return false; // the library parser reports the empty input return false; // the default parser reports the empty input
} }
const auto* const first = reinterpret_cast<const unsigned char*>(adapter.bulk_data()); const auto* const first = reinterpret_cast<const unsigned char*>(adapter.bulk_data());
contiguous_parser<BasicJsonType, SAX, KeepValues> p(first, first + size, &sax); contiguous_parser<BasicJsonType, SAX, KeepValues> p(first, first + size, &sax);
@@ -24697,7 +24703,7 @@ bool parse_contiguous(InputAdapterType& adapter, SAX& sax, std::true_type /*cont
@brief a SAX handler that only reports a parse error, as the handler of parse() @brief a SAX handler that only reports a parse error, as the handler of parse()
does: it throws the exception, unless exceptions are not allowed does: it throws the exception, unless exceptions are not allowed
After contiguous_parser has rejected an input, the library parser is run again After contiguous_parser has rejected an input, the default parser is run again
with this handler to report the error: the error, its message and its position with this handler to report the error: the error, its message and its position
are those of parse(), without building the values read up to it. are those of parse(), without building the values read up to it.
*/ */
@@ -24734,7 +24740,7 @@ struct sax_ignores_number_text<json_sax_error_reporter<BasicJsonType>> : std::tr
rejected, as parse() reports it: throw its exception, or, without rejected, as parse() reports it: throw its exception, or, without
exceptions, return true exceptions, return true
@return false if the library parser accepts the input after all (which would @return false if the default parser accepts the input after all (which would
be a disagreement of the two parsers): the caller then parses it be a disagreement of the two parsers): the caller then parses it
*/ */
template<typename BasicJsonType, typename InputAdapterType> template<typename BasicJsonType, typename InputAdapterType>
@@ -24788,7 +24794,7 @@ bool parse_contiguous(InputAdapterType& /*adapter*/, BasicJsonType& /*result*/,
@return whether @a accepted is the result of accept(); false if the input is @return whether @a accepted is the result of accept(); false if the input is
not one block of memory, in which case the caller checks it. not one block of memory, in which case the caller checks it.
contiguous_parser accepts exactly what the library parser accepts. contiguous_parser accepts exactly what the default parser accepts.
*/ */
template<typename BasicJsonType, typename InputAdapterType> template<typename BasicJsonType, typename InputAdapterType>
bool accept_contiguous(InputAdapterType& adapter, bool& accepted, std::true_type /*contiguous*/) bool accept_contiguous(InputAdapterType& adapter, bool& accepted, std::true_type /*contiguous*/)
@@ -24809,9 +24815,9 @@ bool accept_contiguous(InputAdapterType& /*adapter*/, bool& /*accepted*/, std::f
handler of the user handler of the user
After contiguous_parser has delivered the events of the input to the user's After contiguous_parser has delivered the events of the input to the user's
handler and rejected it, the library parser is run again with this handler: handler and rejected it, the default parser is run again with this handler:
the user's handler gets the error, its message and its position from the the user's handler gets the error, its message and its position from the
library parser, as before, but no event twice. default parser, as before, but no event twice.
*/ */
template<typename BasicJsonType, typename SAX> template<typename BasicJsonType, typename SAX>
class json_sax_error_forwarder : public json_sax_acceptor<BasicJsonType> class json_sax_error_forwarder : public json_sax_acceptor<BasicJsonType>
@@ -24828,7 +24834,7 @@ class json_sax_error_forwarder : public json_sax_acceptor<BasicJsonType>
return user->parse_error(position, last_token, ex); return user->parse_error(position, last_token, ex);
} }
/// whether the library parser reported an error /// whether the default parser reported an error
bool reported() const noexcept bool reported() const noexcept
{ {
return forwarded; return forwarded;
@@ -24847,8 +24853,8 @@ struct sax_ignores_number_text<json_sax_error_forwarder<BasicJsonType, SAX>> : s
a user's handler @a sax a user's handler @a sax
contiguous_parser reports to @a sax what it reads, and stops at the token that contiguous_parser reports to @a sax what it reads, and stops at the token that
the library parser would reject, before reporting it: the events up to an error the default parser would reject, before reporting it: the events up to an error
are the same. The error itself is then reported by the library parser (see are the same. The error itself is then reported by the default parser (see
json_sax_error_forwarder). json_sax_error_forwarder).
@return whether @a result is the result of sax_parse(); false if the input is @return whether @a result is the result of sax_parse(); false if the input is
@@ -24869,14 +24875,14 @@ bool sax_parse_contiguous(InputAdapterType& adapter, SAX* sax, bool& result, std
} }
if (p.aborted()) if (p.aborted())
{ {
// the handler returned false, as with the library parser // the handler returned false, as with the default parser
result = false; result = false;
return true; return true;
} }
} }
json_sax_error_forwarder<BasicJsonType, SAX> forwarder(sax); json_sax_error_forwarder<BasicJsonType, SAX> forwarder(sax);
result = parser<BasicJsonType, InputAdapterType>(InputAdapterType(adapter), nullptr, true, false, false).sax_parse(&forwarder, true); result = parser<BasicJsonType, InputAdapterType>(InputAdapterType(adapter), nullptr, true, false, false).sax_parse(&forwarder, true);
// the library parser must reject the input as well (the events after the // the default parser must reject the input as well (the events after the
// byte contiguous_parser stopped at cannot be delivered any more) // byte contiguous_parser stopped at cannot be delivered any more)
JSON_ASSERT(forwarder.reported()); JSON_ASSERT(forwarder.reported());
return true; return true;
@@ -41973,7 +41979,7 @@ public:
{ {
basic_json result; basic_json result;
// JSON in one block of memory, with the default options, is read by // JSON in one block of memory, with the default options, is read by
// the faster contiguous_parser; on an error, the library parser reads // the faster contiguous_parser; on an error, the default parser reads
// it again to report it (see contiguous_parser) // it again to report it (see contiguous_parser)
using contiguous = std::integral_constant<bool, detail::can_parse_contiguous<basic_json, typename std::decay<InputAdapterType>::type>::value>; using contiguous = std::integral_constant<bool, detail::can_parse_contiguous<basic_json, typename std::decay<InputAdapterType>::type>::value>;
if (cb == nullptr && !ignore_comments && !ignore_trailing_commas if (cb == nullptr && !ignore_comments && !ignore_trailing_commas
@@ -41991,8 +41997,8 @@ public:
const bool ignore_comments, const bool ignore_comments,
const bool ignore_trailing_commas) const bool ignore_trailing_commas)
{ {
// JSON in one block of memory is checked by the faster // JSON in one block of memory, with the default options, is checked by
// contiguous_parser (see parse_impl()) // the faster contiguous_parser (see parse_impl())
using adapter_t = typename std::decay<InputAdapterType>::type; using adapter_t = typename std::decay<InputAdapterType>::type;
bool accepted = false; bool accepted = false;
if (!ignore_comments && !ignore_trailing_commas if (!ignore_comments && !ignore_trailing_commas
@@ -42016,8 +42022,9 @@ public:
{ {
if (format == input_format_t::json) if (format == input_format_t::json)
{ {
// JSON in one block of memory is read by the faster // JSON in one block of memory, with the default options, is read by
// contiguous_parser, with the default options (see parse_impl()) // the faster contiguous_parser; on an error, the default parser
// reads it again to report it (see parse_impl())
bool result = false; bool result = false;
if (strict && !ignore_comments && !ignore_trailing_commas if (strict && !ignore_comments && !ignore_trailing_commas
&& detail::sax_parse_contiguous<basic_json>(ia, sax, result, std::integral_constant<bool, detail::can_parse_contiguous<basic_json, InputAdapterType>::value> {})) && detail::sax_parse_contiguous<basic_json>(ia, sax, result, std::integral_constant<bool, detail::can_parse_contiguous<basic_json, InputAdapterType>::value> {}))
tests/src/unit-contiguous_parser.cpp
+2 -2
View File
@@ -201,7 +201,7 @@ void check_same_events(const std::string& text, const std::list<char>& bytes)
CHECK(r1 == r2); CHECK(r1 == r2);
CHECK(contiguous.log() == byte_path.log()); CHECK(contiguous.log() == byte_path.log());
} }
// a handler that asks to go on after an error (the library parser then // a handler that asks to go on after an error (the default parser then
// reports what follows, e.g., the missing end of the input) // reports what follows, e.g., the missing end of the input)
event_log contiguous(static_cast<std::size_t>(-1), true); event_log contiguous(static_cast<std::size_t>(-1), true);
event_log byte_path(static_cast<std::size_t>(-1), true); event_log byte_path(static_cast<std::size_t>(-1), true);
@@ -545,7 +545,7 @@ TEST_CASE("contiguous parser")
CHECK_FALSE(narrow_json::accept("34028236e31")); CHECK_FALSE(narrow_json::accept("34028236e31"));
} }
SECTION("callbacks, comments and trailing commas take the library parser") SECTION("callbacks, comments and trailing commas take the default parser")
{ {
// (the results are those of the options; the contiguous parser does not support them) // (the results are those of the options; the contiguous parser does not support them)
CHECK(json::parse("[1,]", nullptr, true, false, true) == json({1})); CHECK(json::parse("[1,]", nullptr, true, false, true) == json({1}));