Compare commits

..
Author SHA1 Message Date
Niels Lohmann 0efa09cf49 Fix clang-tidy performance-inefficient-string-concatenation in unit-json_view_builder
Fixes the ci_clang_tidy failure at tests/src/unit-json_view_builder.cpp:549.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-31 16:19:56 +01:00
Niels Lohmann 6491563cbb json_view: skip the digits of a fraction 16 bytes at a time (SSE2)
The digits after the decimal point were skipped by skip_digits(), 16 compare-and-branch
steps in a row: about 3.5 instructions per digit, and an exit branch that is
mispredicted when the lengths of the fractions vary. On x86 with SSE2, the
fraction is now skipped by skip_fraction_digits(): 16 bytes are compared at
once and the end of the run is taken from the bit mask, which is about a third
of the instructions for a fraction of 12 digits. The last bytes of the input
are skipped one at a time as before. Where there is no SSE2 (AArch64, other
targets, JSON_NO_SIMD) skip_fraction_digits() is skip_digits(): nothing changes
there. Integer digits keep skip_digits(), whose exit is predictable.

GCC unrolls the loop over the last bytes in every inlined copy of the new
function, which made the parse loop 4 KiB larger and parse() of documents with
hardly any fractions about 4 % slower. For GCC that loop is therefore out of
line (skip_digits_tail()); Clang is faster with it inline.

json_document::parse(), A/B in one binary, 6 code layouts, x86-64 Haswell,
time after / before (GCC 16.2 / Clang 23):

- numbers.json (10,001 floats): 0.715 / 0.680
- canada.json: 0.778 / 0.790
- mesh.json: 0.908 / 0.883
- marine_ik.json: 0.937 / 0.926
- mesh.pretty.json: 0.923 / 0.960

Files with few or no fractions and files with fractions of a fixed length
(20,000 prices with two decimals, 60,000 coordinates with six): between 0.98 and
1.014 on both compilers, except citm_catalog.json with GCC (1.05: the
instruction count is +0.3 %, and that row varies by +-7 % between builds of
unrelated variants on this machine).

New test: fractions of every length from 0 to 40 digits followed by every kind of
byte that ends a number or lies next to the digits in ASCII, in several
positions, and cut in every place near the end of the text, against json::parse()
(the length of the number is part of what is compared), for all option
combinations and with and without a terminating NUL. Checked on x86-64 with GCC 16.2,
Clang 23 and Apple clang under Rosetta, and with a deliberate off-by-one at the 16th byte.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-30 22:17:00 +01:00
9 changed files with 253 additions and 267 deletions

No files matched your search

+42 -92
View File
@@ -209,13 +209,8 @@ constexpr bool input_adapter_bulk_window(std::false_type /*detected*/)
@brief lexical analysis @brief lexical analysis
This class organizes the lexical analysis during JSON deserialization. This class organizes the lexical analysis during JSON deserialization.
DiscardValues: the caller only needs the token types and never looks at the values of number and string tokens (only
accept(), which parses through json_sax_acceptor). A template parameter rather than a member, so that the lexer of
parse() is compiled without the paths of accept(): with both string scans in one scan(), parse() was up to 2 % slower
on Apple M1.
*/ */
template<typename BasicJsonType, typename InputAdapterType, bool DiscardValues = false> template<typename BasicJsonType, typename InputAdapterType>
class lexer : public lexer_base<BasicJsonType> class lexer : public lexer_base<BasicJsonType>
{ {
using number_integer_t = typename BasicJsonType::number_integer_t; using number_integer_t = typename BasicJsonType::number_integer_t;
@@ -273,9 +268,10 @@ class lexer : public lexer_base<BasicJsonType>
public: public:
using token_type = typename lexer_base<BasicJsonType>::token_type; using token_type = typename lexer_base<BasicJsonType>::token_type;
explicit lexer(InputAdapterType&& adapter, bool ignore_comments_ = false) noexcept explicit lexer(InputAdapterType&& adapter, bool ignore_comments_ = false, bool discard_number_values_ = false) noexcept
: ia(std::move(adapter)) : ia(std::move(adapter))
, ignore_comments(ignore_comments_) , ignore_comments(ignore_comments_)
, discard_number_values(discard_number_values_)
{ {
// the token string of an error before the first token is everything // the token string of an error before the first token is everything
// read so far // read so far
@@ -392,8 +388,6 @@ class lexer : public lexer_base<BasicJsonType>
checks if it is inside the range. If a violation was detected, set up an checks if it is inside the range. If a violation was detected, set up an
error message and return false. Otherwise, return true. error message and return false. Otherwise, return true.
@param[in] keep whether the bytes are added to token_buffer (false for
accept(), see scan())
@param[in] ranges list of integers; interpreted as list of pairs of @param[in] ranges list of integers; interpreted as list of pairs of
inclusive lower and upper bound, respectively inclusive lower and upper bound, respectively
@@ -402,18 +396,17 @@ class lexer : public lexer_base<BasicJsonType>
@return true if and only if no range violation was detected @return true if and only if no range violation was detected
*/ */
template<bool Keep> bool next_byte_in_range(std::initializer_list<char_int_type> ranges)
bool next_byte_in_range(std::integral_constant<bool, Keep> keep, std::initializer_list<char_int_type> ranges)
{ {
JSON_ASSERT(ranges.size() == 2 || ranges.size() == 4 || ranges.size() == 6); JSON_ASSERT(ranges.size() == 2 || ranges.size() == 4 || ranges.size() == 6);
add_string_char(keep, current); add(current);
for (auto range = ranges.begin(); range != ranges.end(); ++range) for (auto range = ranges.begin(); range != ranges.end(); ++range)
{ {
get(); get();
if (JSON_HEDLEY_LIKELY(*range <= current && current <= *(++range))) // NOLINT(bugprone-inc-dec-in-conditions) if (JSON_HEDLEY_LIKELY(*range <= current && current <= *(++range))) // NOLINT(bugprone-inc-dec-in-conditions)
{ {
add_string_char(keep, current); add(current);
} }
else else
{ {
@@ -426,16 +419,14 @@ class lexer : public lexer_base<BasicJsonType>
} }
/// contiguous input: bulk-append the run of ordinary characters and complete /// contiguous input: bulk-append the run of ordinary characters and complete
/// well-formed UTF-8 sequences starting at the current read position (only /// well-formed UTF-8 sequences starting at the current read position, then
/// skip it if @a keep is false), then
/// read the first byte that needs individual handling (the closing quote, an /// read the first byte that needs individual handling (the closing quote, an
/// escape, a control character, or an ill-formed UTF-8 byte) like get(). /// escape, a control character, or an ill-formed UTF-8 byte) like get().
/// That byte is taken from the buffer that was just scanned. Through get(), /// That byte is taken from the buffer that was just scanned. Through get(),
/// Apple Clang merged this read with the other get() calls of the scan /// Apple Clang merged this read with the other get() calls of the scan
/// function, which load the read position again right after it was stored /// function, which load the read position again right after it was stored
/// (3% of json::accept() on jeopardy.json on Apple M1). /// (3% of json::accept() on jeopardy.json on Apple M1).
template<bool Keep> char_int_type get_after_string_run(std::true_type /*bulk*/)
char_int_type get_after_string_run(std::true_type /*bulk*/, std::integral_constant<bool, Keep> keep)
{ {
// a pending unget must be consumed through the normal path first // a pending unget must be consumed through the normal path first
if (next_unget) if (next_unget)
@@ -453,7 +444,7 @@ class lexer : public lexer_base<BasicJsonType>
const std::size_t pos = lexer_string_bulk_run(data, remaining); const std::size_t pos = lexer_string_bulk_run(data, remaining);
if (pos != 0) if (pos != 0)
{ {
append_string_run(keep, data, pos); token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), pos);
bulk_consume(reinterpret_cast<const char*>(data), pos); bulk_consume(reinterpret_cast<const char*>(data), pos);
} }
if (pos == remaining) if (pos == remaining)
@@ -497,8 +488,7 @@ class lexer : public lexer_base<BasicJsonType>
} }
/// streaming input: no bulk fast path /// streaming input: no bulk fast path
template<bool Keep> char_int_type get_after_string_run(std::false_type /*bulk*/)
char_int_type get_after_string_run(std::false_type /*bulk*/, std::integral_constant<bool, Keep> /*keep*/)
{ {
return get(); return get();
} }
@@ -512,18 +502,13 @@ class lexer : public lexer_base<BasicJsonType>
may contain \0 bytes), and token_buffer.size() is the number of bytes in the may contain \0 bytes), and token_buffer.size() is the number of bytes in the
string. string.
@param[in] keep whether the string is copied into token_buffer; if false
(accept(), see scan()), it is only checked, with the same
errors, and token_buffer stays empty
@return token_type::value_string if string could be successfully scanned, @return token_type::value_string if string could be successfully scanned,
token_type::parse_error otherwise token_type::parse_error otherwise
@note In case of errors, variable error_message contains a textual @note In case of errors, variable error_message contains a textual
description. description.
*/ */
template<bool Keep> token_type scan_string()
token_type scan_string(std::integral_constant<bool, Keep> keep)
{ {
// reset token_buffer (ignore opening quote) // reset token_buffer (ignore opening quote)
reset(); reset();
@@ -535,7 +520,7 @@ class lexer : public lexer_base<BasicJsonType>
{ {
// bulk-consume ordinary characters from contiguous input, then // bulk-consume ordinary characters from contiguous input, then
// get the next character, the one that ends the run // get the next character, the one that ends the run
switch (get_after_string_run(std::integral_constant<bool, bulk_scan>(), keep)) switch (get_after_string_run(std::integral_constant<bool, bulk_scan>()))
{ {
// end of file while parsing the string // end of file while parsing the string
case char_traits<char_type>::eof(): case char_traits<char_type>::eof():
@@ -557,35 +542,35 @@ class lexer : public lexer_base<BasicJsonType>
{ {
// quotation mark // quotation mark
case '\"': case '\"':
add_string_char(keep, '\"'); add('\"');
break; break;
// reverse solidus // reverse solidus
case '\\': case '\\':
add_string_char(keep, '\\'); add('\\');
break; break;
// solidus // solidus
case '/': case '/':
add_string_char(keep, '/'); add('/');
break; break;
// backspace // backspace
case 'b': case 'b':
add_string_char(keep, '\b'); add('\b');
break; break;
// form feed // form feed
case 'f': case 'f':
add_string_char(keep, '\f'); add('\f');
break; break;
// line feed // line feed
case 'n': case 'n':
add_string_char(keep, '\n'); add('\n');
break; break;
// carriage return // carriage return
case 'r': case 'r':
add_string_char(keep, '\r'); add('\r');
break; break;
// tab // tab
case 't': case 't':
add_string_char(keep, '\t'); add('\t');
break; break;
// unicode escapes // unicode escapes
@@ -653,7 +638,10 @@ class lexer : public lexer_base<BasicJsonType>
JSON_ASSERT(0x00 <= codepoint && codepoint <= 0x10FFFF); JSON_ASSERT(0x00 <= codepoint && codepoint <= 0x10FFFF);
// translate codepoint into bytes // translate codepoint into bytes
add_codepoint(keep, codepoint); encode_utf8(static_cast<std::uint32_t>(codepoint), [this](std::uint32_t byte)
{
add(static_cast<char_int_type>(byte));
});
break; break;
} }
@@ -836,7 +824,7 @@ class lexer : public lexer_base<BasicJsonType>
case 0x7E: case 0x7E:
case 0x7F: case 0x7F:
{ {
add_string_char(keep, current); add(current);
break; break;
} }
@@ -872,7 +860,7 @@ class lexer : public lexer_base<BasicJsonType>
case 0xDE: case 0xDE:
case 0xDF: case 0xDF:
{ {
if (JSON_HEDLEY_UNLIKELY(!next_byte_in_range(keep, {0x80, 0xBF}))) if (JSON_HEDLEY_UNLIKELY(!next_byte_in_range({0x80, 0xBF})))
{ {
return token_type::parse_error; return token_type::parse_error;
} }
@@ -882,7 +870,7 @@ class lexer : public lexer_base<BasicJsonType>
// U+0800..U+0FFF: bytes E0 A0..BF 80..BF // U+0800..U+0FFF: bytes E0 A0..BF 80..BF
case 0xE0: case 0xE0:
{ {
if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range(keep, {0xA0, 0xBF, 0x80, 0xBF})))) if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range({0xA0, 0xBF, 0x80, 0xBF}))))
{ {
return token_type::parse_error; return token_type::parse_error;
} }
@@ -906,7 +894,7 @@ class lexer : public lexer_base<BasicJsonType>
case 0xEE: case 0xEE:
case 0xEF: case 0xEF:
{ {
if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range(keep, {0x80, 0xBF, 0x80, 0xBF})))) if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range({0x80, 0xBF, 0x80, 0xBF}))))
{ {
return token_type::parse_error; return token_type::parse_error;
} }
@@ -916,7 +904,7 @@ class lexer : public lexer_base<BasicJsonType>
// U+D000..U+D7FF: bytes ED 80..9F 80..BF // U+D000..U+D7FF: bytes ED 80..9F 80..BF
case 0xED: case 0xED:
{ {
if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range(keep, {0x80, 0x9F, 0x80, 0xBF})))) if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range({0x80, 0x9F, 0x80, 0xBF}))))
{ {
return token_type::parse_error; return token_type::parse_error;
} }
@@ -926,7 +914,7 @@ class lexer : public lexer_base<BasicJsonType>
// U+10000..U+3FFFF F0 90..BF 80..BF 80..BF // U+10000..U+3FFFF F0 90..BF 80..BF 80..BF
case 0xF0: case 0xF0:
{ {
if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range(keep, {0x90, 0xBF, 0x80, 0xBF, 0x80, 0xBF})))) if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range({0x90, 0xBF, 0x80, 0xBF, 0x80, 0xBF}))))
{ {
return token_type::parse_error; return token_type::parse_error;
} }
@@ -938,7 +926,7 @@ class lexer : public lexer_base<BasicJsonType>
case 0xF2: case 0xF2:
case 0xF3: case 0xF3:
{ {
if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range(keep, {0x80, 0xBF, 0x80, 0xBF, 0x80, 0xBF})))) if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range({0x80, 0xBF, 0x80, 0xBF, 0x80, 0xBF}))))
{ {
return token_type::parse_error; return token_type::parse_error;
} }
@@ -948,7 +936,7 @@ class lexer : public lexer_base<BasicJsonType>
// U+100000..U+10FFFF F4 80..8F 80..BF 80..BF // U+100000..U+10FFFF F4 80..8F 80..BF 80..BF
case 0xF4: case 0xF4:
{ {
if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range(keep, {0x80, 0x8F, 0x80, 0xBF, 0x80, 0xBF})))) if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range({0x80, 0x8F, 0x80, 0xBF, 0x80, 0xBF}))))
{ {
return token_type::parse_error; return token_type::parse_error;
} }
@@ -1439,7 +1427,7 @@ scan_number_done:
token_type convert_number(token_type number_type, std::size_t mantissa_end) token_type convert_number(token_type number_type, std::size_t mantissa_end)
{ {
// accept() only needs to know whether the input is valid, so it sets // accept() only needs to know whether the input is valid, so it sets
// discard_values (see json.hpp), and an integer token whose // discard_number_values (see json.hpp), and an integer token whose
// digit count shows that it fits is reported without calling // digit count shows that it fits is reported without calling
// convert_integer(). A number with up to 18 digits always fits into // convert_integer(). A number with up to 18 digits always fits into
// both std::uint64_t and std::int64_t (18 nines is about 1e18, below // both std::uint64_t and std::int64_t (18 nines is about 1e18, below
@@ -1454,7 +1442,7 @@ scan_number_done:
// json_sax_acceptor, whose number callbacks discard their argument and // json_sax_acceptor, whose number callbacks discard their argument and
// return true, and the parser rejects neither integers nor finite // return true, and the parser rejects neither integers nor finite
// floats. value_unsigned/value_integer are left unset here, so a caller // floats. value_unsigned/value_integer are left unset here, so a caller
// that reads the converted value must not set discard_values. // that reads the converted value must not set discard_number_values.
// //
// On contiguous input, scan_number_bulk_contiguous() converts integer // On contiguous input, scan_number_bulk_contiguous() converts integer
// tokens itself and does not pass them to this function, unless // tokens itself and does not pass them to this function, unless
@@ -1462,7 +1450,7 @@ scan_number_done:
// reached for input without bulk access (e.g. streams), with // reached for input without bulk access (e.g. streams), with
// JSON_DIAGNOSTIC_POSITIONS, or when scan_number_bulk_contiguous() // JSON_DIAGNOSTIC_POSITIONS, or when scan_number_bulk_contiguous()
// falls back to scan_number(). // falls back to scan_number().
if (discard_values()) if (discard_number_values)
{ {
constexpr std::size_t safe_digit_count = 18; constexpr std::size_t safe_digit_count = 18;
if (number_type == token_type::value_unsigned && token_buffer.size() <= safe_digit_count) if (number_type == token_type::value_unsigned && token_buffer.size() <= safe_digit_count)
@@ -1713,7 +1701,7 @@ scan_number_done:
// w < 10^19, so a value below 10^max_exponent10 cannot overflow, and a // w < 10^19, so a value below 10^max_exponent10 cannot overflow, and a
// value that underflows is a finite number as well. Only a token that // value that underflows is a finite number as well. Only a token that
// may overflow has to be converted to find out. // may overflow has to be converted to find out.
if (JSON_HEDLEY_UNLIKELY(discard_values() && s.exponent + 19 <= std::numeric_limits<number_float_t>::max_exponent10)) if (JSON_HEDLEY_UNLIKELY(discard_number_values && s.exponent + 19 <= std::numeric_limits<number_float_t>::max_exponent10))
{ {
value_float = 0; value_float = 0;
return true; return true;
@@ -2045,34 +2033,6 @@ scan_number_done:
token_buffer.push_back(static_cast<typename string_t::value_type>(c)); token_buffer.push_back(static_cast<typename string_t::value_type>(c));
} }
/// add a character of a string token; accept() checks the string but does
/// not keep its value (@a keep is false)
void add_string_char(std::true_type /*keep*/, char_int_type c)
{
token_buffer.push_back(static_cast<typename string_t::value_type>(c));
}
static void add_string_char(std::false_type /*keep*/, char_int_type /*c*/) noexcept {}
/// add the UTF-8 bytes of a code point from a \\u escape to a string token
void add_codepoint(std::true_type /*keep*/, int codepoint)
{
encode_utf8(static_cast<std::uint32_t>(codepoint), [this](std::uint32_t byte)
{
add(static_cast<char_int_type>(byte));
});
}
static void add_codepoint(std::false_type /*keep*/, int /*codepoint*/) noexcept {}
/// append a run of ordinary string characters (see get_after_string_run())
void append_string_run(std::true_type /*keep*/, const unsigned char* data, std::size_t n)
{
token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), n);
}
static void append_string_run(std::false_type /*keep*/, const unsigned char* /*data*/, std::size_t /*n*/) noexcept {}
public: public:
///////////////////// /////////////////////
// value getters // value getters
@@ -2467,9 +2427,7 @@ scan_number_done:
// string // string
case '\"': case '\"':
// accept() (DiscardValues) checks strings without copying them return scan_string();
// into token_buffer
return scan_string(keep_strings {});
// number // number
case '-': case '-':
@@ -2554,19 +2512,11 @@ scan_number_done:
std::size_t decimal_point_position = std::string::npos; std::size_t decimal_point_position = std::string::npos;
/// whether the caller only needs the token types and never looks at the /// whether the caller only needs the token types and never looks at the
/// values of number and string tokens; DiscardValues, set only by accept(), which parses /// converted numeric values; set only by accept(), which parses through
/// through json_sax_acceptor. When set, convert_number() skips converting /// json_sax_acceptor. When set, convert_number() skips converting integer
/// integer tokens whose digit count guarantees that they fit into 64 bits /// tokens whose digit count guarantees that they fit into 64 bits (see
/// (see there), floats are converted only when they may overflow (see /// there)
/// convert_accumulated_float()), and strings are checked without being const bool discard_number_values = false;
/// copied into token_buffer (see scan())
static bool discard_values() noexcept
{
return DiscardValues;
}
/// the tag of scan_string(): whether string tokens are kept
using keep_strings = typename std::conditional<DiscardValues, std::false_type, std::true_type>::type;
}; };
} // namespace detail } // namespace detail
+5 -4
View File
@@ -72,14 +72,14 @@ This class implements a parser for JSON text. Nested arrays and objects are trac
stack instead of recursion, so deeply nested input does not exhaust the call stack, and what is read stack instead of recursion, so deeply nested input does not exhaust the call stack, and what is read
is reported as SAX events. is reported as SAX events.
*/ */
template<typename BasicJsonType, typename InputAdapterType, bool DiscardValues = false> template<typename BasicJsonType, typename InputAdapterType>
class parser class parser
{ {
using number_integer_t = typename BasicJsonType::number_integer_t; using number_integer_t = typename BasicJsonType::number_integer_t;
using number_unsigned_t = typename BasicJsonType::number_unsigned_t; using number_unsigned_t = typename BasicJsonType::number_unsigned_t;
using number_float_t = typename BasicJsonType::number_float_t; using number_float_t = typename BasicJsonType::number_float_t;
using string_t = typename BasicJsonType::string_t; using string_t = typename BasicJsonType::string_t;
using lexer_t = lexer<BasicJsonType, InputAdapterType, DiscardValues>; using lexer_t = lexer<BasicJsonType, InputAdapterType>;
using token_type = typename lexer_t::token_type; using token_type = typename lexer_t::token_type;
public: public:
@@ -88,9 +88,10 @@ class parser
parser_callback_t<BasicJsonType> cb = nullptr, parser_callback_t<BasicJsonType> cb = nullptr,
const bool allow_exceptions_ = true, const bool allow_exceptions_ = true,
const bool ignore_comments = false, const bool ignore_comments = false,
const bool ignore_trailing_commas_ = false) const bool ignore_trailing_commas_ = false,
const bool discard_number_values_ = false)
: callback(std::move(cb)) : callback(std::move(cb))
, m_lexer(std::move(adapter), ignore_comments) , m_lexer(std::move(adapter), ignore_comments, discard_number_values_)
, allow_exceptions(allow_exceptions_) , allow_exceptions(allow_exceptions_)
, ignore_trailing_commas(ignore_trailing_commas_) , ignore_trailing_commas(ignore_trailing_commas_)
{ {
+1 -1
View File
@@ -1082,7 +1082,7 @@ indent_done:
{ {
++p; ++p;
const unsigned char* const f0 = p; const unsigned char* const f0 = p;
p = skip_digits(p, e); p = skip_fraction_digits(p, e);
if (NLOHMANN_VIEW_UNLIKELY(p == f0)) if (NLOHMANN_VIEW_UNLIKELY(p == f0))
{ {
return fail(error_code::number_after_dot); return fail(error_code::number_after_dot);
+51
View File
@@ -198,6 +198,57 @@ inline const unsigned char* find_block_comment_end(const unsigned char* p, const
return p; return p;
} }
/// The digits of a fraction, whose number varies from one number to the next: skip_digits() with 16 bytes compared at
/// once and the end of the run taken from a bit mask (SSE2), instead of a compare and a branch for each digit. That
/// is about a third of the instructions for a fraction of 12 digits, and no branch that is mispredicted when the
/// lengths vary. Where there is no SSE2, skip_digits() is used.
#if NLOHMANN_VIEW_SSE2
// The last bytes of the input, one at a time. GCC unrolls this loop in every inlined copy of the function below, which
// makes the parse loop 4 KiB larger and slows down documents that have hardly any fractions; out of line, it does
// not. Clang does better with the loop inline (a call in the parse loop costs it more than the unrolling).
#if defined(__GNUC__) && !defined(__clang__)
inline NLOHMANN_VIEW_NOINLINE const unsigned char* skip_digits_tail(const unsigned char* p, const unsigned char* e) noexcept
{
while (p != e && is_digit(*p))
{
++p;
}
return p;
}
#else
NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* skip_digits_tail(const unsigned char* p, const unsigned char* e) noexcept
{
while (p != e && is_digit(*p))
{
++p;
}
return p;
}
#endif
NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* skip_fraction_digits(const unsigned char* p, const unsigned char* e) noexcept
{
while (e - p >= 16)
{
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
// a digit is above '/' and below ':' (signed compares: a byte with the top bit set is negative, and neither)
const __m128i digits = _mm_and_si128(_mm_cmpgt_epi8(in, _mm_set1_epi8('/')), _mm_cmplt_epi8(in, _mm_set1_epi8(':')));
const auto not_digit = static_cast<std::uint64_t>(~static_cast<unsigned>(_mm_movemask_epi8(digits)) & 0xFFFFu);
if (not_digit != 0)
{
return p + count_trailing_zeros(not_digit);
}
p += 16;
}
return skip_digits_tail(p, e);
}
#else
NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* skip_fraction_digits(const unsigned char* p, const unsigned char* e) noexcept
{
return skip_digits(p, e);
}
#endif
/// powers of ten up to 10^19 as integers /// powers of ten up to 10^19 as integers
inline std::uint64_t int_pow10(unsigned k) noexcept inline std::uint64_t int_pow10(unsigned k) noexcept
{ {
+5 -6
View File
@@ -150,7 +150,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
// can be restored when json_pointer backwards compatibility is removed // can be restored when json_pointer backwards compatibility is removed
// friend ::nlohmann::json_pointer<StringType>; // friend ::nlohmann::json_pointer<StringType>;
template<typename BasicJsonType, typename InputType, bool DiscardValues> template<typename BasicJsonType, typename InputType>
friend class ::nlohmann::detail::parser; friend class ::nlohmann::detail::parser;
friend ::nlohmann::detail::serializer<basic_json>; friend ::nlohmann::detail::serializer<basic_json>;
template<typename BasicJsonType> template<typename BasicJsonType>
@@ -183,11 +183,12 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
detail::parser_callback_t<basic_json>cb = nullptr, detail::parser_callback_t<basic_json>cb = nullptr,
const bool allow_exceptions = true, const bool allow_exceptions = true,
const bool ignore_comments = false, const bool ignore_comments = false,
const bool ignore_trailing_commas = false const bool ignore_trailing_commas = false,
const bool discard_number_values = false
) )
{ {
return ::nlohmann::detail::parser<basic_json, InputAdapterType>(std::move(adapter), return ::nlohmann::detail::parser<basic_json, InputAdapterType>(std::move(adapter),
std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas); std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas, discard_number_values);
} }
private: private:
@@ -3116,9 +3117,7 @@ public:
const bool ignore_comments, const bool ignore_comments,
const bool ignore_trailing_commas) const bool ignore_trailing_commas)
{ {
// the parser and lexer that discard the values of tokens (see lexer) return parser(std::forward<InputAdapterType>(adapter), nullptr, false, ignore_comments, ignore_trailing_commas, true).accept(true);
return ::nlohmann::detail::parser<basic_json, typename std::decay<InputAdapterType>::type, true>(std::forward<InputAdapterType>(adapter),
nullptr, false, ignore_comments, ignore_trailing_commas).accept(true);
} }
/// generate SAX events from the input behind an already created adapter /// generate SAX events from the input behind an already created adapter
+52 -102
View File
@@ -13065,13 +13065,8 @@ constexpr bool input_adapter_bulk_window(std::false_type /*detected*/)
@brief lexical analysis @brief lexical analysis
This class organizes the lexical analysis during JSON deserialization. This class organizes the lexical analysis during JSON deserialization.
DiscardValues: the caller only needs the token types and never looks at the values of number and string tokens (only
accept(), which parses through json_sax_acceptor). A template parameter rather than a member, so that the lexer of
parse() is compiled without the paths of accept(): with both string scans in one scan(), parse() was up to 2 % slower
on Apple M1.
*/ */
template<typename BasicJsonType, typename InputAdapterType, bool DiscardValues = false> template<typename BasicJsonType, typename InputAdapterType>
class lexer : public lexer_base<BasicJsonType> class lexer : public lexer_base<BasicJsonType>
{ {
using number_integer_t = typename BasicJsonType::number_integer_t; using number_integer_t = typename BasicJsonType::number_integer_t;
@@ -13129,9 +13124,10 @@ class lexer : public lexer_base<BasicJsonType>
public: public:
using token_type = typename lexer_base<BasicJsonType>::token_type; using token_type = typename lexer_base<BasicJsonType>::token_type;
explicit lexer(InputAdapterType&& adapter, bool ignore_comments_ = false) noexcept explicit lexer(InputAdapterType&& adapter, bool ignore_comments_ = false, bool discard_number_values_ = false) noexcept
: ia(std::move(adapter)) : ia(std::move(adapter))
, ignore_comments(ignore_comments_) , ignore_comments(ignore_comments_)
, discard_number_values(discard_number_values_)
{ {
// the token string of an error before the first token is everything // the token string of an error before the first token is everything
// read so far // read so far
@@ -13248,8 +13244,6 @@ class lexer : public lexer_base<BasicJsonType>
checks if it is inside the range. If a violation was detected, set up an checks if it is inside the range. If a violation was detected, set up an
error message and return false. Otherwise, return true. error message and return false. Otherwise, return true.
@param[in] keep whether the bytes are added to token_buffer (false for
accept(), see scan())
@param[in] ranges list of integers; interpreted as list of pairs of @param[in] ranges list of integers; interpreted as list of pairs of
inclusive lower and upper bound, respectively inclusive lower and upper bound, respectively
@@ -13258,18 +13252,17 @@ class lexer : public lexer_base<BasicJsonType>
@return true if and only if no range violation was detected @return true if and only if no range violation was detected
*/ */
template<bool Keep> bool next_byte_in_range(std::initializer_list<char_int_type> ranges)
bool next_byte_in_range(std::integral_constant<bool, Keep> keep, std::initializer_list<char_int_type> ranges)
{ {
JSON_ASSERT(ranges.size() == 2 || ranges.size() == 4 || ranges.size() == 6); JSON_ASSERT(ranges.size() == 2 || ranges.size() == 4 || ranges.size() == 6);
add_string_char(keep, current); add(current);
for (auto range = ranges.begin(); range != ranges.end(); ++range) for (auto range = ranges.begin(); range != ranges.end(); ++range)
{ {
get(); get();
if (JSON_HEDLEY_LIKELY(*range <= current && current <= *(++range))) // NOLINT(bugprone-inc-dec-in-conditions) if (JSON_HEDLEY_LIKELY(*range <= current && current <= *(++range))) // NOLINT(bugprone-inc-dec-in-conditions)
{ {
add_string_char(keep, current); add(current);
} }
else else
{ {
@@ -13282,16 +13275,14 @@ class lexer : public lexer_base<BasicJsonType>
} }
/// contiguous input: bulk-append the run of ordinary characters and complete /// contiguous input: bulk-append the run of ordinary characters and complete
/// well-formed UTF-8 sequences starting at the current read position (only /// well-formed UTF-8 sequences starting at the current read position, then
/// skip it if @a keep is false), then
/// read the first byte that needs individual handling (the closing quote, an /// read the first byte that needs individual handling (the closing quote, an
/// escape, a control character, or an ill-formed UTF-8 byte) like get(). /// escape, a control character, or an ill-formed UTF-8 byte) like get().
/// That byte is taken from the buffer that was just scanned. Through get(), /// That byte is taken from the buffer that was just scanned. Through get(),
/// Apple Clang merged this read with the other get() calls of the scan /// Apple Clang merged this read with the other get() calls of the scan
/// function, which load the read position again right after it was stored /// function, which load the read position again right after it was stored
/// (3% of json::accept() on jeopardy.json on Apple M1). /// (3% of json::accept() on jeopardy.json on Apple M1).
template<bool Keep> char_int_type get_after_string_run(std::true_type /*bulk*/)
char_int_type get_after_string_run(std::true_type /*bulk*/, std::integral_constant<bool, Keep> keep)
{ {
// a pending unget must be consumed through the normal path first // a pending unget must be consumed through the normal path first
if (next_unget) if (next_unget)
@@ -13309,7 +13300,7 @@ class lexer : public lexer_base<BasicJsonType>
const std::size_t pos = lexer_string_bulk_run(data, remaining); const std::size_t pos = lexer_string_bulk_run(data, remaining);
if (pos != 0) if (pos != 0)
{ {
append_string_run(keep, data, pos); token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), pos);
bulk_consume(reinterpret_cast<const char*>(data), pos); bulk_consume(reinterpret_cast<const char*>(data), pos);
} }
if (pos == remaining) if (pos == remaining)
@@ -13353,8 +13344,7 @@ class lexer : public lexer_base<BasicJsonType>
} }
/// streaming input: no bulk fast path /// streaming input: no bulk fast path
template<bool Keep> char_int_type get_after_string_run(std::false_type /*bulk*/)
char_int_type get_after_string_run(std::false_type /*bulk*/, std::integral_constant<bool, Keep> /*keep*/)
{ {
return get(); return get();
} }
@@ -13368,18 +13358,13 @@ class lexer : public lexer_base<BasicJsonType>
may contain \0 bytes), and token_buffer.size() is the number of bytes in the may contain \0 bytes), and token_buffer.size() is the number of bytes in the
string. string.
@param[in] keep whether the string is copied into token_buffer; if false
(accept(), see scan()), it is only checked, with the same
errors, and token_buffer stays empty
@return token_type::value_string if string could be successfully scanned, @return token_type::value_string if string could be successfully scanned,
token_type::parse_error otherwise token_type::parse_error otherwise
@note In case of errors, variable error_message contains a textual @note In case of errors, variable error_message contains a textual
description. description.
*/ */
template<bool Keep> token_type scan_string()
token_type scan_string(std::integral_constant<bool, Keep> keep)
{ {
// reset token_buffer (ignore opening quote) // reset token_buffer (ignore opening quote)
reset(); reset();
@@ -13391,7 +13376,7 @@ class lexer : public lexer_base<BasicJsonType>
{ {
// bulk-consume ordinary characters from contiguous input, then // bulk-consume ordinary characters from contiguous input, then
// get the next character, the one that ends the run // get the next character, the one that ends the run
switch (get_after_string_run(std::integral_constant<bool, bulk_scan>(), keep)) switch (get_after_string_run(std::integral_constant<bool, bulk_scan>()))
{ {
// end of file while parsing the string // end of file while parsing the string
case char_traits<char_type>::eof(): case char_traits<char_type>::eof():
@@ -13413,35 +13398,35 @@ class lexer : public lexer_base<BasicJsonType>
{ {
// quotation mark // quotation mark
case '\"': case '\"':
add_string_char(keep, '\"'); add('\"');
break; break;
// reverse solidus // reverse solidus
case '\\': case '\\':
add_string_char(keep, '\\'); add('\\');
break; break;
// solidus // solidus
case '/': case '/':
add_string_char(keep, '/'); add('/');
break; break;
// backspace // backspace
case 'b': case 'b':
add_string_char(keep, '\b'); add('\b');
break; break;
// form feed // form feed
case 'f': case 'f':
add_string_char(keep, '\f'); add('\f');
break; break;
// line feed // line feed
case 'n': case 'n':
add_string_char(keep, '\n'); add('\n');
break; break;
// carriage return // carriage return
case 'r': case 'r':
add_string_char(keep, '\r'); add('\r');
break; break;
// tab // tab
case 't': case 't':
add_string_char(keep, '\t'); add('\t');
break; break;
// unicode escapes // unicode escapes
@@ -13509,7 +13494,10 @@ class lexer : public lexer_base<BasicJsonType>
JSON_ASSERT(0x00 <= codepoint && codepoint <= 0x10FFFF); JSON_ASSERT(0x00 <= codepoint && codepoint <= 0x10FFFF);
// translate codepoint into bytes // translate codepoint into bytes
add_codepoint(keep, codepoint); encode_utf8(static_cast<std::uint32_t>(codepoint), [this](std::uint32_t byte)
{
add(static_cast<char_int_type>(byte));
});
break; break;
} }
@@ -13692,7 +13680,7 @@ class lexer : public lexer_base<BasicJsonType>
case 0x7E: case 0x7E:
case 0x7F: case 0x7F:
{ {
add_string_char(keep, current); add(current);
break; break;
} }
@@ -13728,7 +13716,7 @@ class lexer : public lexer_base<BasicJsonType>
case 0xDE: case 0xDE:
case 0xDF: case 0xDF:
{ {
if (JSON_HEDLEY_UNLIKELY(!next_byte_in_range(keep, {0x80, 0xBF}))) if (JSON_HEDLEY_UNLIKELY(!next_byte_in_range({0x80, 0xBF})))
{ {
return token_type::parse_error; return token_type::parse_error;
} }
@@ -13738,7 +13726,7 @@ class lexer : public lexer_base<BasicJsonType>
// U+0800..U+0FFF: bytes E0 A0..BF 80..BF // U+0800..U+0FFF: bytes E0 A0..BF 80..BF
case 0xE0: case 0xE0:
{ {
if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range(keep, {0xA0, 0xBF, 0x80, 0xBF})))) if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range({0xA0, 0xBF, 0x80, 0xBF}))))
{ {
return token_type::parse_error; return token_type::parse_error;
} }
@@ -13762,7 +13750,7 @@ class lexer : public lexer_base<BasicJsonType>
case 0xEE: case 0xEE:
case 0xEF: case 0xEF:
{ {
if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range(keep, {0x80, 0xBF, 0x80, 0xBF})))) if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range({0x80, 0xBF, 0x80, 0xBF}))))
{ {
return token_type::parse_error; return token_type::parse_error;
} }
@@ -13772,7 +13760,7 @@ class lexer : public lexer_base<BasicJsonType>
// U+D000..U+D7FF: bytes ED 80..9F 80..BF // U+D000..U+D7FF: bytes ED 80..9F 80..BF
case 0xED: case 0xED:
{ {
if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range(keep, {0x80, 0x9F, 0x80, 0xBF})))) if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range({0x80, 0x9F, 0x80, 0xBF}))))
{ {
return token_type::parse_error; return token_type::parse_error;
} }
@@ -13782,7 +13770,7 @@ class lexer : public lexer_base<BasicJsonType>
// U+10000..U+3FFFF F0 90..BF 80..BF 80..BF // U+10000..U+3FFFF F0 90..BF 80..BF 80..BF
case 0xF0: case 0xF0:
{ {
if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range(keep, {0x90, 0xBF, 0x80, 0xBF, 0x80, 0xBF})))) if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range({0x90, 0xBF, 0x80, 0xBF, 0x80, 0xBF}))))
{ {
return token_type::parse_error; return token_type::parse_error;
} }
@@ -13794,7 +13782,7 @@ class lexer : public lexer_base<BasicJsonType>
case 0xF2: case 0xF2:
case 0xF3: case 0xF3:
{ {
if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range(keep, {0x80, 0xBF, 0x80, 0xBF, 0x80, 0xBF})))) if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range({0x80, 0xBF, 0x80, 0xBF, 0x80, 0xBF}))))
{ {
return token_type::parse_error; return token_type::parse_error;
} }
@@ -13804,7 +13792,7 @@ class lexer : public lexer_base<BasicJsonType>
// U+100000..U+10FFFF F4 80..8F 80..BF 80..BF // U+100000..U+10FFFF F4 80..8F 80..BF 80..BF
case 0xF4: case 0xF4:
{ {
if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range(keep, {0x80, 0x8F, 0x80, 0xBF, 0x80, 0xBF})))) if (JSON_HEDLEY_UNLIKELY(!(next_byte_in_range({0x80, 0x8F, 0x80, 0xBF, 0x80, 0xBF}))))
{ {
return token_type::parse_error; return token_type::parse_error;
} }
@@ -14295,7 +14283,7 @@ scan_number_done:
token_type convert_number(token_type number_type, std::size_t mantissa_end) token_type convert_number(token_type number_type, std::size_t mantissa_end)
{ {
// accept() only needs to know whether the input is valid, so it sets // accept() only needs to know whether the input is valid, so it sets
// discard_values (see json.hpp), and an integer token whose // discard_number_values (see json.hpp), and an integer token whose
// digit count shows that it fits is reported without calling // digit count shows that it fits is reported without calling
// convert_integer(). A number with up to 18 digits always fits into // convert_integer(). A number with up to 18 digits always fits into
// both std::uint64_t and std::int64_t (18 nines is about 1e18, below // both std::uint64_t and std::int64_t (18 nines is about 1e18, below
@@ -14310,7 +14298,7 @@ scan_number_done:
// json_sax_acceptor, whose number callbacks discard their argument and // json_sax_acceptor, whose number callbacks discard their argument and
// return true, and the parser rejects neither integers nor finite // return true, and the parser rejects neither integers nor finite
// floats. value_unsigned/value_integer are left unset here, so a caller // floats. value_unsigned/value_integer are left unset here, so a caller
// that reads the converted value must not set discard_values. // that reads the converted value must not set discard_number_values.
// //
// On contiguous input, scan_number_bulk_contiguous() converts integer // On contiguous input, scan_number_bulk_contiguous() converts integer
// tokens itself and does not pass them to this function, unless // tokens itself and does not pass them to this function, unless
@@ -14318,7 +14306,7 @@ scan_number_done:
// reached for input without bulk access (e.g. streams), with // reached for input without bulk access (e.g. streams), with
// JSON_DIAGNOSTIC_POSITIONS, or when scan_number_bulk_contiguous() // JSON_DIAGNOSTIC_POSITIONS, or when scan_number_bulk_contiguous()
// falls back to scan_number(). // falls back to scan_number().
if (discard_values()) if (discard_number_values)
{ {
constexpr std::size_t safe_digit_count = 18; constexpr std::size_t safe_digit_count = 18;
if (number_type == token_type::value_unsigned && token_buffer.size() <= safe_digit_count) if (number_type == token_type::value_unsigned && token_buffer.size() <= safe_digit_count)
@@ -14569,7 +14557,7 @@ scan_number_done:
// w < 10^19, so a value below 10^max_exponent10 cannot overflow, and a // w < 10^19, so a value below 10^max_exponent10 cannot overflow, and a
// value that underflows is a finite number as well. Only a token that // value that underflows is a finite number as well. Only a token that
// may overflow has to be converted to find out. // may overflow has to be converted to find out.
if (JSON_HEDLEY_UNLIKELY(discard_values() && s.exponent + 19 <= std::numeric_limits<number_float_t>::max_exponent10)) if (JSON_HEDLEY_UNLIKELY(discard_number_values && s.exponent + 19 <= std::numeric_limits<number_float_t>::max_exponent10))
{ {
value_float = 0; value_float = 0;
return true; return true;
@@ -14901,34 +14889,6 @@ scan_number_done:
token_buffer.push_back(static_cast<typename string_t::value_type>(c)); token_buffer.push_back(static_cast<typename string_t::value_type>(c));
} }
/// add a character of a string token; accept() checks the string but does
/// not keep its value (@a keep is false)
void add_string_char(std::true_type /*keep*/, char_int_type c)
{
token_buffer.push_back(static_cast<typename string_t::value_type>(c));
}
static void add_string_char(std::false_type /*keep*/, char_int_type /*c*/) noexcept {}
/// add the UTF-8 bytes of a code point from a \\u escape to a string token
void add_codepoint(std::true_type /*keep*/, int codepoint)
{
encode_utf8(static_cast<std::uint32_t>(codepoint), [this](std::uint32_t byte)
{
add(static_cast<char_int_type>(byte));
});
}
static void add_codepoint(std::false_type /*keep*/, int /*codepoint*/) noexcept {}
/// append a run of ordinary string characters (see get_after_string_run())
void append_string_run(std::true_type /*keep*/, const unsigned char* data, std::size_t n)
{
token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), n);
}
static void append_string_run(std::false_type /*keep*/, const unsigned char* /*data*/, std::size_t /*n*/) noexcept {}
public: public:
///////////////////// /////////////////////
// value getters // value getters
@@ -15323,9 +15283,7 @@ scan_number_done:
// string // string
case '\"': case '\"':
// accept() (DiscardValues) checks strings without copying them return scan_string();
// into token_buffer
return scan_string(keep_strings {});
// number // number
case '-': case '-':
@@ -15410,19 +15368,11 @@ scan_number_done:
std::size_t decimal_point_position = std::string::npos; std::size_t decimal_point_position = std::string::npos;
/// whether the caller only needs the token types and never looks at the /// whether the caller only needs the token types and never looks at the
/// values of number and string tokens; DiscardValues, set only by accept(), which parses /// converted numeric values; set only by accept(), which parses through
/// through json_sax_acceptor. When set, convert_number() skips converting /// json_sax_acceptor. When set, convert_number() skips converting integer
/// integer tokens whose digit count guarantees that they fit into 64 bits /// tokens whose digit count guarantees that they fit into 64 bits (see
/// (see there), floats are converted only when they may overflow (see /// there)
/// convert_accumulated_float()), and strings are checked without being const bool discard_number_values = false;
/// copied into token_buffer (see scan())
static bool discard_values() noexcept
{
return DiscardValues;
}
/// the tag of scan_string(): whether string tokens are kept
using keep_strings = typename std::conditional<DiscardValues, std::false_type, std::true_type>::type;
}; };
} // namespace detail } // namespace detail
@@ -22614,14 +22564,14 @@ This class implements a parser for JSON text. Nested arrays and objects are trac
stack instead of recursion, so deeply nested input does not exhaust the call stack, and what is read stack instead of recursion, so deeply nested input does not exhaust the call stack, and what is read
is reported as SAX events. is reported as SAX events.
*/ */
template<typename BasicJsonType, typename InputAdapterType, bool DiscardValues = false> template<typename BasicJsonType, typename InputAdapterType>
class parser class parser
{ {
using number_integer_t = typename BasicJsonType::number_integer_t; using number_integer_t = typename BasicJsonType::number_integer_t;
using number_unsigned_t = typename BasicJsonType::number_unsigned_t; using number_unsigned_t = typename BasicJsonType::number_unsigned_t;
using number_float_t = typename BasicJsonType::number_float_t; using number_float_t = typename BasicJsonType::number_float_t;
using string_t = typename BasicJsonType::string_t; using string_t = typename BasicJsonType::string_t;
using lexer_t = lexer<BasicJsonType, InputAdapterType, DiscardValues>; using lexer_t = lexer<BasicJsonType, InputAdapterType>;
using token_type = typename lexer_t::token_type; using token_type = typename lexer_t::token_type;
public: public:
@@ -22630,9 +22580,10 @@ class parser
parser_callback_t<BasicJsonType> cb = nullptr, parser_callback_t<BasicJsonType> cb = nullptr,
const bool allow_exceptions_ = true, const bool allow_exceptions_ = true,
const bool ignore_comments = false, const bool ignore_comments = false,
const bool ignore_trailing_commas_ = false) const bool ignore_trailing_commas_ = false,
const bool discard_number_values_ = false)
: callback(std::move(cb)) : callback(std::move(cb))
, m_lexer(std::move(adapter), ignore_comments) , m_lexer(std::move(adapter), ignore_comments, discard_number_values_)
, allow_exceptions(allow_exceptions_) , allow_exceptions(allow_exceptions_)
, ignore_trailing_commas(ignore_trailing_commas_) , ignore_trailing_commas(ignore_trailing_commas_)
{ {
@@ -36995,7 +36946,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
// can be restored when json_pointer backwards compatibility is removed // can be restored when json_pointer backwards compatibility is removed
// friend ::nlohmann::json_pointer<StringType>; // friend ::nlohmann::json_pointer<StringType>;
template<typename BasicJsonType, typename InputType, bool DiscardValues> template<typename BasicJsonType, typename InputType>
friend class ::nlohmann::detail::parser; friend class ::nlohmann::detail::parser;
friend ::nlohmann::detail::serializer<basic_json>; friend ::nlohmann::detail::serializer<basic_json>;
template<typename BasicJsonType> template<typename BasicJsonType>
@@ -37028,11 +36979,12 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
detail::parser_callback_t<basic_json>cb = nullptr, detail::parser_callback_t<basic_json>cb = nullptr,
const bool allow_exceptions = true, const bool allow_exceptions = true,
const bool ignore_comments = false, const bool ignore_comments = false,
const bool ignore_trailing_commas = false const bool ignore_trailing_commas = false,
const bool discard_number_values = false
) )
{ {
return ::nlohmann::detail::parser<basic_json, InputAdapterType>(std::move(adapter), return ::nlohmann::detail::parser<basic_json, InputAdapterType>(std::move(adapter),
std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas); std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas, discard_number_values);
} }
private: private:
@@ -39961,9 +39913,7 @@ public:
const bool ignore_comments, const bool ignore_comments,
const bool ignore_trailing_commas) const bool ignore_trailing_commas)
{ {
// the parser and lexer that discard the values of tokens (see lexer) return parser(std::forward<InputAdapterType>(adapter), nullptr, false, ignore_comments, ignore_trailing_commas, true).accept(true);
return ::nlohmann::detail::parser<basic_json, typename std::decay<InputAdapterType>::type, true>(std::forward<InputAdapterType>(adapter),
nullptr, false, ignore_comments, ignore_trailing_commas).accept(true);
} }
/// generate SAX events from the input behind an already created adapter /// generate SAX events from the input behind an already created adapter
+52 -1
View File
@@ -1218,6 +1218,57 @@ inline const unsigned char* find_block_comment_end(const unsigned char* p, const
return p; return p;
} }
/// The digits of a fraction, whose number varies from one number to the next: skip_digits() with 16 bytes compared at
/// once and the end of the run taken from a bit mask (SSE2), instead of a compare and a branch for each digit. That
/// is about a third of the instructions for a fraction of 12 digits, and no branch that is mispredicted when the
/// lengths vary. Where there is no SSE2, skip_digits() is used.
#if NLOHMANN_VIEW_SSE2
// The last bytes of the input, one at a time. GCC unrolls this loop in every inlined copy of the function below, which
// makes the parse loop 4 KiB larger and slows down documents that have hardly any fractions; out of line, it does
// not. Clang does better with the loop inline (a call in the parse loop costs it more than the unrolling).
#if defined(__GNUC__) && !defined(__clang__)
inline NLOHMANN_VIEW_NOINLINE const unsigned char* skip_digits_tail(const unsigned char* p, const unsigned char* e) noexcept
{
while (p != e && is_digit(*p))
{
++p;
}
return p;
}
#else
NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* skip_digits_tail(const unsigned char* p, const unsigned char* e) noexcept
{
while (p != e && is_digit(*p))
{
++p;
}
return p;
}
#endif
NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* skip_fraction_digits(const unsigned char* p, const unsigned char* e) noexcept
{
while (e - p >= 16)
{
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
// a digit is above '/' and below ':' (signed compares: a byte with the top bit set is negative, and neither)
const __m128i digits = _mm_and_si128(_mm_cmpgt_epi8(in, _mm_set1_epi8('/')), _mm_cmplt_epi8(in, _mm_set1_epi8(':')));
const auto not_digit = static_cast<std::uint64_t>(~static_cast<unsigned>(_mm_movemask_epi8(digits)) & 0xFFFFu);
if (not_digit != 0)
{
return p + count_trailing_zeros(not_digit);
}
p += 16;
}
return skip_digits_tail(p, e);
}
#else
NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* skip_fraction_digits(const unsigned char* p, const unsigned char* e) noexcept
{
return skip_digits(p, e);
}
#endif
/// powers of ten up to 10^19 as integers /// powers of ten up to 10^19 as integers
inline std::uint64_t int_pow10(unsigned k) noexcept inline std::uint64_t int_pow10(unsigned k) noexcept
{ {
@@ -2324,7 +2375,7 @@ indent_done:
{ {
++p; ++p;
const unsigned char* const f0 = p; const unsigned char* const f0 = p;
p = skip_digits(p, e); p = skip_fraction_digits(p, e);
if (NLOHMANN_VIEW_UNLIKELY(p == f0)) if (NLOHMANN_VIEW_UNLIKELY(p == f0))
{ {
return fail(error_code::number_after_dot); return fail(error_code::number_after_dot);
-61
View File
@@ -2982,67 +2982,6 @@ TEST_CASE("accept() agrees with parse() on floats around the overflow limit")
} }
} }
TEST_CASE("accept() agrees with parse() on strings, which it checks without copying them")
{
// accept() scans strings and object keys with a separate instantiation of
// the string scanner that does not copy them into the token buffer; it
// must accept and reject exactly what parse() does
const std::vector<std::string> pieces =
{
"", "abc", R"(\"\\\/\b\f\n\r\t)", R"(\u0041\u00e4\u20AC)", "\\ud83d\\ude00", "\\uD83D\\uDE00",
"\\ud800", "\\udc00", "\\ud800\\u0041", "\\ud800x", "\\udbff\\udfff", "\\u12", "\\u12g4", "\\x", "\\",
"\xc3\xa4", "\xe2\x82\xac", "\xf0\x9f\x98\x80", "\xf4\x8f\xbf\xbf", "\xc0\xaf", "\xed\xa0\x80", "\xf4\x90\x80\x80",
"\xff", "\xe2\x82", "\xc3", "\x01", "\x1f", "\x7f", "\n", std::string(1, '\0')
};
for (std::size_t run = 0; run <= 70; ++run)
{
for (const std::string& piece : pieces)
{
// a run of ordinary characters puts the piece at every offset of the bulk scanners
const std::string str = std::string(run, 'a') + piece;
std::string opened = "\"";
opened += str;
std::string closed = opened;
closed += '"';
std::vector<std::string> texts;
texts.push_back(closed);
texts.emplace_back("[");
texts.back() += closed;
texts.back() += ']';
texts.emplace_back("{");
texts.back() += closed;
texts.back() += ":1}";
texts.emplace_back("{\"k\":");
texts.back() += closed;
texts.back() += ',';
texts.back() += closed;
texts.back() += ":[";
texts.back() += closed;
texts.back() += "]}";
texts.emplace_back("[");
texts.back() += opened;
for (const std::string& text : texts)
{
CAPTURE(text)
for (int options = 0; options < 4; ++options)
{
const bool ignore_comments = (options & 1) != 0;
const bool ignore_trailing_commas = (options & 2) != 0;
const bool parses = !json::parse(text, nullptr, false, ignore_comments, ignore_trailing_commas).is_discarded();
CHECK(json::accept(text, ignore_comments, ignore_trailing_commas) == parses);
std::istringstream stream(text);
CHECK(json::accept(stream, ignore_comments, ignore_trailing_commas) == parses);
const std::list<char> chars(text.begin(), text.end());
CHECK(json::accept(chars.begin(), chars.end(), ignore_comments, ignore_trailing_commas) == parses);
}
}
}
}
}
namespace namespace
{ {
// a DOM parser that reports parse errors without throwing, as in the sax_no_exception example // a DOM parser that reports parse errors without throwing, as in the sax_no_exception example
+45
View File
@@ -523,6 +523,51 @@ TEST_CASE("json_view builder: strings across vector blocks")
} }
} }
TEST_CASE("json_view builder: fraction digits")
{
// The digits of a fraction are skipped 16 bytes at a time where there are
// that many left (SSE2), and one at a time after that. Fractions of every
// length from 0 to 40 digits, followed by every kind of byte that ends a
// number or lies next to the digits in ASCII ('/' and ':'), as values, as
// keys' neighbours, and at the very end of the text (a vector without a
// terminating NUL). The length of the number is part of the value that
// the test compares with json::parse().
const std::string followers = ",]}/:.eE+-x \n\t\"09";
std::mt19937 rng(2028); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed): reproducible
for (std::size_t digits = 0; digits <= 40; ++digits)
{
for (int i = 0; i < 3; ++i)
{
std::string fraction;
for (std::size_t k = 0; k < digits; ++k)
{
fraction += static_cast<char>('0' + (rng() % 10));
}
const std::string number = std::string(rng() % 2 == 0 ? "-" : "") + std::to_string(rng() % 100000) + "." + fraction;
check_same(number);
check_same("[" + number + "]");
std::string pair = "[" + number;
pair += ",";
pair += number;
pair += "]";
check_same(pair);
check_same("{\"a\":" + number + "}");
for (const char follower : followers)
{
check_same(number + follower);
check_same("[" + number + follower + "]");
check_same("[" + number + follower + std::string(rng() % 20, '1'));
}
// everything near the end of the text: the number cut in every place
for (std::size_t cut = 1; cut <= 18 && cut <= number.size(); ++cut)
{
check_same(number.substr(0, number.size() - cut));
check_same("[" + number.substr(0, number.size() - cut));
}
}
}
}
TEST_CASE("json_view builder: escapes between runs of text") TEST_CASE("json_view builder: escapes between runs of text")
{ {
// The text between escapes is copied 16 bytes at a time where there are // The text between escapes is copied 16 bytes at a time where there are