Compare commits

..
Author SHA1 Message Date
Niels Lohmann 759a847178 Fix CI: bugprone-signed-char-misuse in append_from_block
Convert the (positive) length to its unsigned type before widening it
to std::size_t, as clang-tidy flags the int8_t to std::size_t cast.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-11-06 12:13:09 +01:00
Niels Lohmann a33af0a51b Binary readers: keep the tag argument of append_from_block() astyle-proof
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-11-06 08:40:38 +01:00
Niels Lohmann baf0bee38f Binary readers: append strings straight from contiguous input
get_bytes() read every string into a zero-filled resize() and copied the
bytes over it, so each byte was written twice. A string that the input holds
in one block (bulk_remaining() at least its length) is now appended from it
directly. The claimed length is checked against the input first, so a huge
claimed length still allocates nothing it cannot fill; other inputs and
binary values keep the chunked loop.

from_cbor() and from_msgpack() of twitter.json and citm_catalog.json get
5.5-9.1% faster on an Apple M1 Max and 2.7-8.6% on x86-64 (GCC 15, Clang 21);
canada.json, which has few strings, is unchanged.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-11-06 06:43:54 +01:00
12 changed files with 92 additions and 2757 deletions

No files matched your search

-1
View File
@@ -30,7 +30,6 @@ cc_library(
"include/nlohmann/detail/exceptions.hpp", "include/nlohmann/detail/exceptions.hpp",
"include/nlohmann/detail/hash.hpp", "include/nlohmann/detail/hash.hpp",
"include/nlohmann/detail/input/binary_reader.hpp", "include/nlohmann/detail/input/binary_reader.hpp",
"include/nlohmann/detail/input/contiguous_parser.hpp",
"include/nlohmann/detail/input/input_adapters.hpp", "include/nlohmann/detail/input/input_adapters.hpp",
"include/nlohmann/detail/input/json_sax.hpp", "include/nlohmann/detail/input/json_sax.hpp",
"include/nlohmann/detail/input/lexer.hpp", "include/nlohmann/detail/input/lexer.hpp",
@@ -48,8 +48,8 @@ The default value is `0`: streams are scanned in bulk.
!!! note "Performance" !!! note "Performance"
Parsing from a `std::ifstream` or `std::istringstream` is typically 1.5 to 2 times as fast as before, and reading Parsing from a `std::ifstream` or `std::istringstream` is typically 1.5 to 2 times as fast as before, and reading
many small values from one stream with `operator>>` 1.4 to 4 times as fast. Parsing from a stream then takes about many small values from one stream with `operator>>` 1.4 to 4 times as fast. Parsing from a stream is then within
1.15 to 1.45 times as long as first reading the whole file into a `std::string` and parsing that. about 10 to 20 % of first reading the whole file into a `std::string` and parsing that.
A stream buffer without a get area (one that only overrides `underflow()` and `uflow()` and hands out every A stream buffer without a get area (one that only overrides `underflow()` and `uflow()` and hands out every
character from there) offers nothing to scan in bulk, so it is read one character at a time as before. character from there) offers nothing to scan in bulk, so it is read one character at a time as before.
+5 -5
View File
@@ -32,11 +32,11 @@ json read_json_file(const std::string& path)
column 1 instead of saying that the file is missing. A relative path is resolved against the working directory of the column 1 instead of saying that the file is missing. A relative path is resolved against the working directory of the
running program, which is often not the directory of the source file (see the running program, which is often not the directory of the source file (see the
[FAQ](../home/faq.md#parsing-a-file-fails-with-unexpected-end-of-input)). [FAQ](../home/faq.md#parsing-a-file-fails-with-unexpected-end-of-input)).
- **Read the file into a string only if the memory is no concern.** Parsing from a `#!cpp std::ifstream` takes about - **Don't read the file into a string first.** Parsing from a `#!cpp std::ifstream` is only about 10 to 20 % slower
1.15 to 1.45 times as long as reading the whole file into a string with one `read()` and parsing that, but it needs no than reading the whole file into a string with one `read()` and parsing that, and it needs no memory for the copy (see
memory for the copy (see [input sources and performance](parsing/index.md#input-sources-and-performance)). The common [input sources and performance](parsing/index.md#input-sources-and-performance)). The common way to read a file with
way to read a file with `#!cpp std::istreambuf_iterator<char>` is so slow that it takes about twice as long as parsing `#!cpp std::istreambuf_iterator<char>` is so slow that it takes about twice as long as parsing from the stream. Read
from the stream. This advice is for the file into a string only if you need the text itself, for example for an error message. This advice is for
`#!cpp json::parse`. [`json_document`](json_view.md) reads a stream into a buffer that it owns anyway, so there is no `#!cpp json::parse`. [`json_document`](json_view.md) reads a stream into a buffer that it owns anyway, so there is no
copy to save: reading the file into a `#!cpp std::string` yourself and passing that as an lvalue, which the document copy to save: reading the file into a `#!cpp std::string` yourself and passing that as an lvalue, which the document
borrows, was about 6 to 12 % faster than parsing from the stream, with the same single buffer. borrows, was about 6 to 12 % faster than parsing from the stream, with the same single buffer.
+2 -7
View File
@@ -80,13 +80,8 @@ them faster than others:
| `#!cpp FILE*` | one character at a time, with the file locked once for the whole parse | | `#!cpp FILE*` | one character at a time, with the file locked once for the whole parse |
| other iterator ranges (e.g., `#!cpp std::list<char>`, `#!cpp std::istreambuf_iterator<char>`), wide strings | one character at a time | | other iterator ranges (e.g., `#!cpp std::list<char>`, `#!cpp std::istreambuf_iterator<char>`), wide strings | one character at a time |
Input in one block of memory is read even faster with the default options (no Parsing a file from a `#!cpp std::ifstream` is therefore only about 10 to 20 % slower than reading the whole file into a
[parser callback](../../api/basic_json/parser_callback_t.md), no comments, no trailing commas, and for `sax_parse()` `#!cpp std::string` first and parsing that, and it needs no memory for a copy of the file. The stream is never read
`strict = true`): `parse()`, `accept()`, and `sax_parse()` then read the grammar and the tokens in one loop over the
memory. Only after an error do `parse()` and `sax_parse()` read the input again, to report the error exactly as before.
Parsing a file from a `#!cpp std::ifstream` therefore takes about 1.15 to 1.45 times as long as reading the whole file
into a `#!cpp std::string` first and parsing that, but it needs no memory for a copy of the file. The stream is never read
ahead: after parsing, it is positioned exactly as if every character had been read one at a time, so ahead: after parsing, it is positioned exactly as if every character had been read one at a time, so
[`operator>>`](../../api/operator_gtgt.md) can still read several values from one stream (see also [`operator>>`](../../api/operator_gtgt.md) can still read several values from one stream (see also
[JSON Lines](json_lines.md)). [JSON Lines](json_lines.md)).
+3 -3
View File
@@ -29,9 +29,9 @@ up. Nothing is read ahead, so the stream is left exactly where it would be after
[`operator>>`](../api/operator_gtgt.md) can still read several values from one stream. A stream buffer without a get [`operator>>`](../api/operator_gtgt.md) can still read several values from one stream. A stream buffer without a get
area is read one character at a time. A `#!cpp FILE*` is still read one character at a time, but `file_input_adapter` area is read one character at a time. A `#!cpp FILE*` is still read one character at a time, but `file_input_adapter`
locks the file once per parse instead of once per character. See locks the file once per parse instead of once per character. See
[input sources and performance](parsing/index.md#input-sources-and-performance) for the full table. Parsing from a [input sources and performance](parsing/index.md#input-sources-and-performance) for the full table. According to the
stream takes about 1.15 to 1.45 times as long as parsing an equivalent `#!cpp std::string`, which `parse()` reads with description of [#5828](https://github.com/nlohmann/json/pull/5828), parsing from a stream is within about 10 to 20 % of
a parser of its own (see there). parsing an equivalent `#!cpp std::string`.
Inputs that are still read one character at a time: iterator ranges that are not contiguous (e.g. Inputs that are still read one character at a time: iterator ranges that are not contiguous (e.g.
`#!cpp std::list<char>` or `#!cpp std::istreambuf_iterator<char>`), wide strings, and user-defined adapters. An `#!cpp std::list<char>` or `#!cpp std::istreambuf_iterator<char>`), wide strings, and user-defined adapters. An
@@ -4887,6 +4887,14 @@ class binary_reader
const char* context, const char* context,
ContainerType& result) ContainerType& result)
{ {
// a string that is in the block of input as a whole is appended from
// it, without the zero-filling resize() below
constexpr bool from_block = bulk_scan && std::is_same<ContainerType, string_t>::value;
if (append_from_block(len, result, std::integral_constant<bool, from_block> {}))
{
return true;
}
// upper bound on the number of bytes read (and allocated) per chunk // upper bound on the number of bytes read (and allocated) per chunk
constexpr std::size_t chunk_size = 4096; constexpr std::size_t chunk_size = 4096;
@@ -4923,6 +4931,36 @@ class binary_reader
return true; return true;
} }
/// append the @a len bytes at the current position to @a result, if the
/// input holds them in one block; false (with nothing read) otherwise
template<typename NumberType>
bool append_from_block(const NumberType len, string_t& result, std::true_type /*block*/)
{
if (len <= 0)
{
return false;
}
// (len > 0, so the unsigned type holds it; casting a signed char
// straight to std::size_t would trip bugprone-signed-char-misuse)
const auto ulen = static_cast<typename std::make_unsigned<NumberType>::type>(len);
const std::size_t remaining = ia.bulk_remaining();
if (static_cast<std::uintmax_t>(ulen) > static_cast<std::uintmax_t>(remaining))
{
return false; // (the loop reports the premature end)
}
const auto n = static_cast<std::size_t>(ulen);
result.append(reinterpret_cast<const typename string_t::value_type*>(ia.bulk_data()), n);
ia.bulk_skip(n);
chars_read += n;
return true;
}
template<typename NumberType, typename ContainerType>
static bool append_from_block(const NumberType /*len*/, ContainerType& /*result*/, std::false_type /*block*/) noexcept
{
return false;
}
/*! /*!
@param[in] context further context information (for diagnostics) @param[in] context further context information (for diagnostics)
@return whether the last read character is not EOF @return whether the last read character is not EOF
File diff suppressed because it is too large. Load diff
@@ -316,30 +316,6 @@ JSON_HEDLEY_ALWAYS_INLINE std::size_t accumulate_digits(const char* data, std::s
return i; return i;
} }
/// the index of the first byte at or after @a i that is not a digit: the
/// digits of accumulate_digits() without their value
JSON_HEDLEY_ALWAYS_INLINE std::size_t skip_digits(const char* data, std::size_t i, std::size_t avail) noexcept
{
while (avail - i >= 8)
{
if (!is_eight_digits(read_eight_bytes(data + i)))
{
// a byte of the word is not a digit: no bounds checks before it
while (static_cast<unsigned char>(data[i] - '0') <= 9)
{
++i;
}
return i;
}
i += 8;
}
while (i < avail && data[i] >= '0' && data[i] <= '9')
{
++i;
}
return i;
}
/// whether [first, last) contains a digit other than '0'; a plain loop, as /// whether [first, last) contains a digit other than '0'; a plain loop, as
/// GCC calls std::any_of out of line even for the usual empty range /// GCC calls std::any_of out of line even for the usual empty range
JSON_HEDLEY_ALWAYS_INLINE bool has_nonzero_digit(const char* first, const char* last) noexcept JSON_HEDLEY_ALWAYS_INLINE bool has_nonzero_digit(const char* first, const char* last) noexcept
@@ -520,10 +520,7 @@ JSON_HEDLEY_NEVER_INLINE inline std::size_t string_bulk_run_from(const unsigned
} }
#endif #endif
// string_bulk_run() for the lexer. The run must be the longest one: the lexer // string_bulk_run() for the lexer. With NEON, only the first four words are
// would only lose speed with a shorter run, but contiguous_parser rejects a
// string whose run ends at a byte other than a quote or a backslash, so a
// shorter run would reject valid input there. With NEON, only the first four words are
// checked inline (most keys and short values end there), and longer runs and // checked inline (most keys and short values end there), and longer runs and
// non-ASCII text continue out of line. The lexer's scan function runs once per // non-ASCII text continue out of line. The lexer's scan function runs once per
// token, and the vector code inlined into it needs more registers, which every // token, and the vector code inlined into it needs more registers, which every
+1 -28
View File
@@ -58,7 +58,6 @@
#include <nlohmann/detail/exceptions.hpp> // IWYU pragma: export #include <nlohmann/detail/exceptions.hpp> // IWYU pragma: export
#include <nlohmann/detail/hash.hpp> #include <nlohmann/detail/hash.hpp>
#include <nlohmann/detail/input/binary_reader.hpp> #include <nlohmann/detail/input/binary_reader.hpp>
#include <nlohmann/detail/input/contiguous_parser.hpp>
#include <nlohmann/detail/input/input_adapters.hpp> #include <nlohmann/detail/input/input_adapters.hpp>
#include <nlohmann/detail/input/json_sax.hpp> #include <nlohmann/detail/input/json_sax.hpp>
#include <nlohmann/detail/input/lexer.hpp> #include <nlohmann/detail/input/lexer.hpp>
@@ -3110,15 +3109,6 @@ public:
const bool ignore_trailing_commas) const bool ignore_trailing_commas)
{ {
basic_json result; basic_json result;
// JSON in one block of memory, with the default options, is read by
// the faster contiguous_parser; on an error, the library parser reads
// it again to report it (see contiguous_parser)
using contiguous = std::integral_constant<bool, detail::can_parse_contiguous<basic_json, typename std::decay<InputAdapterType>::type>::value>;
if (cb == nullptr && !ignore_comments && !ignore_trailing_commas
&& detail::parse_contiguous(adapter, result, allow_exceptions, contiguous {}))
{
return result;
}
parser(std::forward<InputAdapterType>(adapter), std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas).parse(true, result); parser(std::forward<InputAdapterType>(adapter), std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas).parse(true, result);
return result; return result;
} }
@@ -3129,17 +3119,8 @@ public:
const bool ignore_comments, const bool ignore_comments,
const bool ignore_trailing_commas) const bool ignore_trailing_commas)
{ {
// JSON in one block of memory is checked by the faster
// contiguous_parser (see parse_impl())
using adapter_t = typename std::decay<InputAdapterType>::type;
bool accepted = false;
if (!ignore_comments && !ignore_trailing_commas
&& detail::accept_contiguous<basic_json>(adapter, accepted, std::integral_constant<bool, detail::can_parse_contiguous<basic_json, adapter_t>::value> {}))
{
return accepted;
}
// the parser and lexer that discard the values of tokens (see lexer) // the parser and lexer that discard the values of tokens (see lexer)
return ::nlohmann::detail::parser<basic_json, adapter_t, true>(std::forward<InputAdapterType>(adapter), return ::nlohmann::detail::parser<basic_json, typename std::decay<InputAdapterType>::type, true>(std::forward<InputAdapterType>(adapter),
nullptr, false, ignore_comments, ignore_trailing_commas).accept(true); nullptr, false, ignore_comments, ignore_trailing_commas).accept(true);
} }
@@ -3154,14 +3135,6 @@ public:
{ {
if (format == input_format_t::json) if (format == input_format_t::json)
{ {
// JSON in one block of memory is read by the faster
// contiguous_parser, with the default options (see parse_impl())
bool result = false;
if (strict && !ignore_comments && !ignore_trailing_commas
&& detail::sax_parse_contiguous<basic_json>(ia, sax, result, std::integral_constant<bool, detail::can_parse_contiguous<basic_json, InputAdapterType>::value> {}))
{
return result;
}
return parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict); return parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict);
} }
return detail::binary_reader<basic_json, InputAdapterType, SAX>(std::move(ia), format).sax_parse(sax, strict, tag_handler); return detail::binary_reader<basic_json, InputAdapterType, SAX>(std::move(ia), format).sax_parse(sax, strict, tag_handler);
+40 -1093
View File
File diff suppressed because it is too large. Load diff
-564
View File
@@ -1,564 +0,0 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
// parse() and accept() read text in one block of memory with
// detail::contiguous_parser, and everything else (here: std::list iterators)
// with the lexer's byte path. These tests check that the two agree on every
// input: the same value, with the same number types and float bits, or the
// same exception with the same message.
#include "doctest_compatibility.h"
#include <nlohmann/json.hpp>
using nlohmann::json;
using nlohmann::ordered_json;
#include <cstddef>
#include <cstdint>
#include <cstring>
#include <list>
#include <map>
#include <random>
#include <string>
#include <type_traits>
#include <vector>
namespace
{
/// the value with its number types and float bits; below 100 levels, the
/// library's dump() (whose recursion is bounded) describes the rest
template<typename Json>
std::string describe_value(const Json& j, int depth = 0)
{
std::string out;
if (depth >= 100)
{
out = j.dump();
return out;
}
switch (j.type())
{
case Json::value_t::number_float:
{
const double d = j.template get<double>();
std::uint64_t bits = 0;
std::memcpy(&bits, &d, sizeof(bits));
out += "f" + std::to_string(bits);
break;
}
case Json::value_t::number_integer:
out += "i" + j.dump();
break;
case Json::value_t::number_unsigned:
out += "u" + j.dump();
break;
case Json::value_t::array:
out += "[";
for (const auto& e : j)
{
out += describe_value(e, depth + 1) + ",";
}
out += "]";
break;
case Json::value_t::object:
out += "{";
for (auto it = j.begin(); it != j.end(); ++it)
{
out += Json(it.key()).dump() + ":" + describe_value(it.value(), depth + 1) + ",";
}
out += "}";
break;
case Json::value_t::null:
case Json::value_t::boolean:
case Json::value_t::string:
case Json::value_t::binary:
case Json::value_t::discarded:
default:
out += j.dump();
break;
}
return out;
}
#if !defined(JSON_NOEXCEPTION)
template<typename Json, typename Parse>
std::string outcome(Parse parse)
{
try
{
return describe_value(parse());
}
catch (const typename Json::exception& e)
{
return std::string("exception: ") + e.what();
}
}
#endif
/// every SAX event with its values, the text of float tokens, and the parse
/// error with its position, token and message; returns false after
/// @a stop_after events (to stop the parse)
class event_log : public nlohmann::json_sax<json>
{
public:
explicit event_log(std::size_t stop_after_ = static_cast<std::size_t>(-1), bool continue_after_error_ = false)
: stop_after(stop_after_), continue_after_error(continue_after_error_)
{}
bool null() override
{
return add("null");
}
bool boolean(bool val) override
{
return add(val ? "true" : "false");
}
bool number_integer(number_integer_t val) override
{
return add("i" + std::to_string(val));
}
bool number_unsigned(number_unsigned_t val) override
{
return add("u" + std::to_string(val));
}
bool number_float(number_float_t val, const string_t& s) override
{
std::uint64_t bits = 0;
std::memcpy(&bits, &val, sizeof(bits));
return add("f" + std::to_string(bits) + " " + s);
}
bool string(string_t& val) override
{
return add("s" + val);
}
bool binary(binary_t& /*val*/) override
{
return add("binary");
}
bool start_object(std::size_t elements) override
{
return add("{" + std::to_string(elements));
}
bool key(string_t& val) override
{
return add("k" + val);
}
bool end_object() override
{
return add("}");
}
bool start_array(std::size_t elements) override
{
return add("[" + std::to_string(elements));
}
bool end_array() override
{
return add("]");
}
bool parse_error(std::size_t position, const std::string& last_token, const nlohmann::detail::exception& ex) override
{
events.push_back("error " + std::to_string(position) + " " + last_token + " " + ex.what());
return continue_after_error;
}
const std::vector<std::string>& log() const noexcept
{
return events;
}
private:
bool add(const std::string& event)
{
events.push_back(event);
return events.size() != stop_after;
}
std::vector<std::string> events {}; // NOLINT(readability-redundant-member-init)
std::size_t stop_after;
bool continue_after_error;
};
/// sax_parse() of the contiguous path against the byte path, also with a
/// handler that stops the parse
void check_same_events(const std::string& text, const std::list<char>& bytes)
{
for (const std::size_t stop_after :
{
static_cast<std::size_t>(-1), std::size_t{1}, std::size_t{3}
})
{
CAPTURE(stop_after)
event_log contiguous(stop_after);
event_log byte_path(stop_after);
const bool r1 = json::sax_parse(text, &contiguous);
const bool r2 = json::sax_parse(bytes.begin(), bytes.end(), &byte_path);
CHECK(r1 == r2);
CHECK(contiguous.log() == byte_path.log());
}
// a handler that asks to go on after an error (the library parser then
// reports what follows, e.g., the missing end of the input)
event_log contiguous(static_cast<std::size_t>(-1), true);
event_log byte_path(static_cast<std::size_t>(-1), true);
const bool r1 = json::sax_parse(text, &contiguous);
const bool r2 = json::sax_parse(bytes.begin(), bytes.end(), &byte_path);
CHECK(r1 == r2);
CHECK(contiguous.log() == byte_path.log());
}
/// the contiguous path (a std::string) against the byte path (std::list iterators)
template<typename Json>
void check_same(const std::string& text)
{
CAPTURE(text)
const std::list<char> bytes(text.begin(), text.end());
#if !defined(JSON_NOEXCEPTION)
// with exceptions: the value, or the same exception for both
const std::string contiguous = outcome<Json>([&]
{
return Json::parse(text);
});
const std::string byte_path = outcome<Json>([&]
{
return Json::parse(bytes.begin(), bytes.end());
});
CHECK(contiguous == byte_path);
#endif
// without exceptions: the value, or discarded for both
const Json c = Json::parse(text, nullptr, false);
const Json b = Json::parse(bytes.begin(), bytes.end(), nullptr, false);
CHECK(c.is_discarded() == b.is_discarded());
CHECK(Json::accept(text) == Json::accept(bytes.begin(), bytes.end()));
if (std::is_same<Json, json>::value)
{
check_same_events(text, bytes);
}
// a pointer range without a terminating NUL
std::vector<char> exact(text.begin(), text.end());
if (!exact.empty())
{
CHECK(Json::accept(exact.data(), exact.data() + exact.size()) == Json::accept(bytes.begin(), bytes.end()));
}
}
void check_both(const std::string& text)
{
check_same<json>(text);
check_same<ordered_json>(text);
}
/// a random JSON text, mostly valid, with the corner cases of the parser
class generator
{
public:
explicit generator(std::uint32_t seed) : rng(seed) {}
std::string document(int depth)
{
std::string out;
value(out, depth);
return out;
}
private:
std::uint32_t pick(std::uint32_t n)
{
return static_cast<std::uint32_t>(rng() % n);
}
void whitespace(std::string& out)
{
static const char* const choices[] = {"", "", "", " ", "\n", "\n ", "\t", "\r\n ", "\n ", " "}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
out += choices[pick(10)];
}
void number(std::string& out)
{
static const char* const choices[] = // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
{
"0", "-0", "1", "-1", "42", "123456789012345678", "-123456789012345678", "1234567890123456789",
"-1234567890123456789", "9223372036854775807", "-9223372036854775808", "-9223372036854775809",
"18446744073709551615", "18446744073709551616", "123456789012345678901234567890", "0.0", "-0.0",
"1.5", "-65.613616999999977", "1e10", "1E+10", "1e-10", "2.2250738585072014e-308", "4.9e-324",
"1e-400", "1.7976931348623157e308", "1e308", "1e309", "-1e309", "123.456e-789", "0.1e1",
"12345678901234567890.5", "1.00000000000000011102230246251565404236316680908203125",
"9007199254740993", "0.30000000000000004", "1e22", "1e23", "100000000000000000000000",
"00", "01", "-", "1.", ".5", "1e", "1e+", "--1", "+1", "1.5e3.2", "0x10", "1_000"
};
out += choices[pick(sizeof(choices) / sizeof(choices[0]))];
}
void string(std::string& out)
{
static const char* const pieces[] = // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
{
"a", "key", "a longer piece of plain ASCII text for the vector paths", "\\n", "\\t", "\\\"",
"\\\\", "\\/", "\\b", "\\f", "\\r", "\\u0000", "\\u00e4", "\\u20AC", "\\uD83D\\uDE00",
"\\uD83D", "\\uDE00", "\\uD83Dx", "\\u12", "\\x", "\xC3\xA4", "\xE2\x82\xAC", "\xF0\x9F\x98\x80",
"\xC0\xAF", "\xED\xA0\x80", "\xF4\x90\x80\x80", "\x80", "\xFF", "\x01", "\x1F", "\x7F", "\t"
};
out += '"';
const auto n = pick(6);
for (std::uint32_t i = 0; i < n; ++i)
{
// mostly well-formed pieces
out += pieces[pick(5) == 0 ? pick(sizeof(pieces) / sizeof(pieces[0])) : pick(20)];
}
out += '"';
}
void value(std::string& out, int depth)
{
const auto kind = depth > 0 ? pick(9) : 3 + pick(6);
switch (kind)
{
case 0:
case 1:
{
out += '[';
whitespace(out);
const auto n = pick(6);
for (std::uint32_t i = 0; i < n; ++i)
{
if (i != 0)
{
out += ',';
whitespace(out);
}
value(out, depth - 1);
whitespace(out);
}
out += ']';
break;
}
case 2:
{
out += '{';
whitespace(out);
const auto n = pick(6);
for (std::uint32_t i = 0; i < n; ++i)
{
if (i != 0)
{
out += ',';
whitespace(out);
}
// a few duplicate keys
if (pick(4) == 0)
{
out += "\"dup\"";
}
else
{
string(out);
}
whitespace(out);
out += pick(2) == 0 ? ":" : ": ";
whitespace(out);
value(out, depth - 1);
whitespace(out);
}
out += '}';
break;
}
case 3:
case 4:
number(out);
break;
case 5:
case 6:
string(out);
break;
case 7:
out += pick(2) == 0 ? "true" : "false";
break;
default:
out += "null";
break;
}
}
std::mt19937 rng;
};
} // namespace
TEST_CASE("contiguous parser")
{
SECTION("fixed inputs")
{
const char* const inputs[] = // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
{
"", " ", "\n", "null", "true", "false", "nul", "tru", "fals", "nullx", "[]", "{}", "[", "{", "]", "}",
"[1,]", "[,1]", R"({"a":1,})", R"({"a" 1})", R"({"a":})", "{1:2}", "[1 2]", R"("")", R"("abc)", R"("\")",
"\xEF\xBB\xBF[1]", "\xEF\xBB[1]", "\xEF[1]", "\xEF\xBB\xBF", " [1] ", "[1]x", "[1] [2]", "1 2",
R"({"a":1,"a":2})", R"({"b":1,"a":2,"b":3})", R"(["\u0000"])", "[1e400]", "[-1e400]",
R"({"a": [1, 2, {"b": null}], "c": "d"})", "[true,false,null]", "[[[[[]]]]]",
};
for (const char* input : inputs)
{
check_both(input);
}
// NUL bytes: by default the end of the input (between values), else an error
check_both(std::string("[1]\0garbage", 11));
check_both(std::string("[1,\0]", 5));
check_both(std::string("\"a\0b\"", 5));
check_both(std::string("\0", 1));
check_both(std::string("1\0", 2));
}
SECTION("deep nesting")
{
// across the 64-level words of the nesting stack
const std::vector<std::size_t> depths = {63, 64, 65, 127, 128, 129, 200, 1000, 10000};
for (const std::size_t depth : depths)
{
check_both(std::string(depth, '[') + std::string(depth, ']'));
std::string objects;
for (std::size_t i = 0; i < depth; ++i)
{
objects += (i % 2 == 0) ? "{\"k\":" : "[";
}
objects += "1";
for (std::size_t i = depth; i-- > 0;)
{
objects += (i % 2 == 0) ? "}" : "]";
}
check_both(objects);
// unbalanced
check_both(std::string(depth, '[') + std::string(depth - 1, ']'));
check_both(std::string(depth, '[') + std::string(depth + 1, ']'));
}
// a value after a container that closes a 64-level word
std::string mixed = std::string(64, '[') + "1" + std::string(63, ']') + ",{\"a\":2}]";
check_both(mixed);
mixed = "{\"x\":" + std::string(64, '[') + std::string(64, ']') + ",\"y\":[1]}";
check_both(mixed);
}
SECTION("generated documents and their mutations")
{
generator gen(20261105u);
std::mt19937 rng(42u); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed): reproducible
const char alphabet[] = "{}[]:,\"\\ \t\n\r0123456789-+.eEtrufalsn/u\x80\xbf\xc3\xe2\xed\xf0\xff"; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
for (int round = 0; round < 1500; ++round)
{
const std::string text = gen.document(1 + (round % 5));
check_both(text);
// every truncation of short documents, and a few mutations
if (text.size() < 60)
{
for (std::size_t n = 0; n < text.size(); ++n)
{
check_same<json>(text.substr(0, n));
}
}
for (int m = 0; m < 3 && !text.empty(); ++m)
{
std::string mutated = text;
const std::size_t pos = rng() % mutated.size();
switch (rng() % 3)
{
case 0:
mutated[pos] = alphabet[rng() % (sizeof(alphabet) - 1)];
break;
case 1:
mutated.insert(pos, 1, alphabet[rng() % (sizeof(alphabet) - 1)]);
break;
default:
mutated.erase(pos, 1);
break;
}
check_same<json>(mutated);
}
}
}
SECTION("narrow number types")
{
// integers that do not fit become floats, floats are converted to float
using narrow_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int32_t, std::uint32_t, float>;
const std::vector<std::string> inputs =
{
"[2147483647, 2147483648, -2147483648, -2147483649, 4294967295, 4294967296, 18446744073709551615]",
"[1.5, 3.4028234663852886e38, 3.4028235677973366e38, 1e39, -1e39, 1e-46, 0.1, 16777217]",
"[-0, 0.0, 123456789012345678901234567890, -9223372036854775809]", "1e39", "4294967296", "-2147483649"
};
for (const auto& input : inputs)
{
check_same<narrow_json>(input);
}
generator gen(17u);
for (int round = 0; round < 200; ++round)
{
check_same<narrow_json>(gen.document(1 + (round % 4)));
}
}
SECTION("number corner cases at the end of the input")
{
// the number ends at the end of the block (no closing bracket after it)
for (const char* number :
{"0", "-0", "1", "12345678", "123456789", "1234567890123456789", "18446744073709551616",
"1.5", "1e5", "1e", "1.", "-", "1.5e-3", "123456789012345678901234567890e-10"
})
{
check_both(number);
check_both(std::string("[") + number + "]");
check_both(std::string("{\"n\":") + number + "}");
}
}
SECTION("accept() of floats near the largest value")
{
// accept() skips digits without their value and computes it only for
// a float whose exponent may reach the largest value
for (const char* number :
{"17976931348623158e292", "17976931348623159e292", "1797693134862315.8e293", "1797693134862315.9e293",
"0.17976931348623158e309", "0.17976931348623159e309", "-17976931348623159e292", "9999999999999999999e289"
})
{
check_both(number);
check_both(std::string("[") + number + "]");
}
CHECK(json::accept("17976931348623158e292"));
CHECK_FALSE(json::accept("17976931348623159e292"));
CHECK(json::accept("[1797693134862315.8e293]"));
CHECK_FALSE(json::accept("[1797693134862315.9e293]"));
CHECK(json::accept("0.17976931348623158e309"));
CHECK_FALSE(json::accept("0.17976931348623159e309"));
CHECK_FALSE(json::accept("-17976931348623159e292"));
using narrow_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int32_t, std::uint32_t, float>;
check_same<narrow_json>("34028235e31");
check_same<narrow_json>("34028236e31");
CHECK(narrow_json::accept("34028235e31"));
CHECK_FALSE(narrow_json::accept("34028236e31"));
}
SECTION("callbacks, comments and trailing commas take the library parser")
{
// (the results are those of the options; the contiguous parser does not support them)
CHECK(json::parse("[1,]", nullptr, true, false, true) == json({1}));
CHECK(json::parse("[1/*c*/]", nullptr, true, true) == json({1}));
CHECK(json::accept("[1,]", false, true));
CHECK(json::accept("// c\n[1]", true));
int calls = 0;
const json j = json::parse("[1,2]", [&calls](int /*depth*/, json::parse_event_t /*event*/, json& /*parsed*/)
{
++calls;
return true;
});
CHECK(j == json({1, 2}));
CHECK(calls == 4);
}
}