Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
759a847178 | ||
|
|
a33af0a51b | ||
|
|
baf0bee38f |
No files matched your search
@@ -30,7 +30,6 @@ cc_library(
|
|||||||
"include/nlohmann/detail/exceptions.hpp",
|
"include/nlohmann/detail/exceptions.hpp",
|
||||||
"include/nlohmann/detail/hash.hpp",
|
"include/nlohmann/detail/hash.hpp",
|
||||||
"include/nlohmann/detail/input/binary_reader.hpp",
|
"include/nlohmann/detail/input/binary_reader.hpp",
|
||||||
"include/nlohmann/detail/input/contiguous_parser.hpp",
|
|
||||||
"include/nlohmann/detail/input/input_adapters.hpp",
|
"include/nlohmann/detail/input/input_adapters.hpp",
|
||||||
"include/nlohmann/detail/input/json_sax.hpp",
|
"include/nlohmann/detail/input/json_sax.hpp",
|
||||||
"include/nlohmann/detail/input/lexer.hpp",
|
"include/nlohmann/detail/input/lexer.hpp",
|
||||||
|
|||||||
@@ -48,8 +48,8 @@ The default value is `0`: streams are scanned in bulk.
|
|||||||
!!! note "Performance"
|
!!! note "Performance"
|
||||||
|
|
||||||
Parsing from a `std::ifstream` or `std::istringstream` is typically 1.5 to 2 times as fast as before, and reading
|
Parsing from a `std::ifstream` or `std::istringstream` is typically 1.5 to 2 times as fast as before, and reading
|
||||||
many small values from one stream with `operator>>` 1.4 to 4 times as fast. Parsing from a stream then takes about
|
many small values from one stream with `operator>>` 1.4 to 4 times as fast. Parsing from a stream is then within
|
||||||
1.15 to 1.45 times as long as first reading the whole file into a `std::string` and parsing that.
|
about 10 to 20 % of first reading the whole file into a `std::string` and parsing that.
|
||||||
|
|
||||||
A stream buffer without a get area (one that only overrides `underflow()` and `uflow()` and hands out every
|
A stream buffer without a get area (one that only overrides `underflow()` and `uflow()` and hands out every
|
||||||
character from there) offers nothing to scan in bulk, so it is read one character at a time as before.
|
character from there) offers nothing to scan in bulk, so it is read one character at a time as before.
|
||||||
|
|||||||
@@ -32,11 +32,11 @@ json read_json_file(const std::string& path)
|
|||||||
column 1 instead of saying that the file is missing. A relative path is resolved against the working directory of the
|
column 1 instead of saying that the file is missing. A relative path is resolved against the working directory of the
|
||||||
running program, which is often not the directory of the source file (see the
|
running program, which is often not the directory of the source file (see the
|
||||||
[FAQ](../home/faq.md#parsing-a-file-fails-with-unexpected-end-of-input)).
|
[FAQ](../home/faq.md#parsing-a-file-fails-with-unexpected-end-of-input)).
|
||||||
- **Read the file into a string only if the memory is no concern.** Parsing from a `#!cpp std::ifstream` takes about
|
- **Don't read the file into a string first.** Parsing from a `#!cpp std::ifstream` is only about 10 to 20 % slower
|
||||||
1.15 to 1.45 times as long as reading the whole file into a string with one `read()` and parsing that, but it needs no
|
than reading the whole file into a string with one `read()` and parsing that, and it needs no memory for the copy (see
|
||||||
memory for the copy (see [input sources and performance](parsing/index.md#input-sources-and-performance)). The common
|
[input sources and performance](parsing/index.md#input-sources-and-performance)). The common way to read a file with
|
||||||
way to read a file with `#!cpp std::istreambuf_iterator<char>` is so slow that it takes about twice as long as parsing
|
`#!cpp std::istreambuf_iterator<char>` is so slow that it takes about twice as long as parsing from the stream. Read
|
||||||
from the stream. This advice is for
|
the file into a string only if you need the text itself, for example for an error message. This advice is for
|
||||||
`#!cpp json::parse`. [`json_document`](json_view.md) reads a stream into a buffer that it owns anyway, so there is no
|
`#!cpp json::parse`. [`json_document`](json_view.md) reads a stream into a buffer that it owns anyway, so there is no
|
||||||
copy to save: reading the file into a `#!cpp std::string` yourself and passing that as an lvalue, which the document
|
copy to save: reading the file into a `#!cpp std::string` yourself and passing that as an lvalue, which the document
|
||||||
borrows, was about 6 to 12 % faster than parsing from the stream, with the same single buffer.
|
borrows, was about 6 to 12 % faster than parsing from the stream, with the same single buffer.
|
||||||
|
|||||||
@@ -80,13 +80,8 @@ them faster than others:
|
|||||||
| `#!cpp FILE*` | one character at a time, with the file locked once for the whole parse |
|
| `#!cpp FILE*` | one character at a time, with the file locked once for the whole parse |
|
||||||
| other iterator ranges (e.g., `#!cpp std::list<char>`, `#!cpp std::istreambuf_iterator<char>`), wide strings | one character at a time |
|
| other iterator ranges (e.g., `#!cpp std::list<char>`, `#!cpp std::istreambuf_iterator<char>`), wide strings | one character at a time |
|
||||||
|
|
||||||
Input in one block of memory is read even faster with the default options (no
|
Parsing a file from a `#!cpp std::ifstream` is therefore only about 10 to 20 % slower than reading the whole file into a
|
||||||
[parser callback](../../api/basic_json/parser_callback_t.md), no comments, no trailing commas, and for `sax_parse()`
|
`#!cpp std::string` first and parsing that, and it needs no memory for a copy of the file. The stream is never read
|
||||||
`strict = true`): `parse()`, `accept()`, and `sax_parse()` then read the grammar and the tokens in one loop over the
|
|
||||||
memory. Only after an error do `parse()` and `sax_parse()` read the input again, to report the error exactly as before.
|
|
||||||
|
|
||||||
Parsing a file from a `#!cpp std::ifstream` therefore takes about 1.15 to 1.45 times as long as reading the whole file
|
|
||||||
into a `#!cpp std::string` first and parsing that, but it needs no memory for a copy of the file. The stream is never read
|
|
||||||
ahead: after parsing, it is positioned exactly as if every character had been read one at a time, so
|
ahead: after parsing, it is positioned exactly as if every character had been read one at a time, so
|
||||||
[`operator>>`](../../api/operator_gtgt.md) can still read several values from one stream (see also
|
[`operator>>`](../../api/operator_gtgt.md) can still read several values from one stream (see also
|
||||||
[JSON Lines](json_lines.md)).
|
[JSON Lines](json_lines.md)).
|
||||||
|
|||||||
@@ -29,9 +29,9 @@ up. Nothing is read ahead, so the stream is left exactly where it would be after
|
|||||||
[`operator>>`](../api/operator_gtgt.md) can still read several values from one stream. A stream buffer without a get
|
[`operator>>`](../api/operator_gtgt.md) can still read several values from one stream. A stream buffer without a get
|
||||||
area is read one character at a time. A `#!cpp FILE*` is still read one character at a time, but `file_input_adapter`
|
area is read one character at a time. A `#!cpp FILE*` is still read one character at a time, but `file_input_adapter`
|
||||||
locks the file once per parse instead of once per character. See
|
locks the file once per parse instead of once per character. See
|
||||||
[input sources and performance](parsing/index.md#input-sources-and-performance) for the full table. Parsing from a
|
[input sources and performance](parsing/index.md#input-sources-and-performance) for the full table. According to the
|
||||||
stream takes about 1.15 to 1.45 times as long as parsing an equivalent `#!cpp std::string`, which `parse()` reads with
|
description of [#5828](https://github.com/nlohmann/json/pull/5828), parsing from a stream is within about 10 to 20 % of
|
||||||
a parser of its own (see there).
|
parsing an equivalent `#!cpp std::string`.
|
||||||
|
|
||||||
Inputs that are still read one character at a time: iterator ranges that are not contiguous (e.g.
|
Inputs that are still read one character at a time: iterator ranges that are not contiguous (e.g.
|
||||||
`#!cpp std::list<char>` or `#!cpp std::istreambuf_iterator<char>`), wide strings, and user-defined adapters. An
|
`#!cpp std::list<char>` or `#!cpp std::istreambuf_iterator<char>`), wide strings, and user-defined adapters. An
|
||||||
|
|||||||
@@ -4887,6 +4887,14 @@ class binary_reader
|
|||||||
const char* context,
|
const char* context,
|
||||||
ContainerType& result)
|
ContainerType& result)
|
||||||
{
|
{
|
||||||
|
// a string that is in the block of input as a whole is appended from
|
||||||
|
// it, without the zero-filling resize() below
|
||||||
|
constexpr bool from_block = bulk_scan && std::is_same<ContainerType, string_t>::value;
|
||||||
|
if (append_from_block(len, result, std::integral_constant<bool, from_block> {}))
|
||||||
|
{
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
// upper bound on the number of bytes read (and allocated) per chunk
|
// upper bound on the number of bytes read (and allocated) per chunk
|
||||||
constexpr std::size_t chunk_size = 4096;
|
constexpr std::size_t chunk_size = 4096;
|
||||||
|
|
||||||
@@ -4923,6 +4931,36 @@ class binary_reader
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// append the @a len bytes at the current position to @a result, if the
|
||||||
|
/// input holds them in one block; false (with nothing read) otherwise
|
||||||
|
template<typename NumberType>
|
||||||
|
bool append_from_block(const NumberType len, string_t& result, std::true_type /*block*/)
|
||||||
|
{
|
||||||
|
if (len <= 0)
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
// (len > 0, so the unsigned type holds it; casting a signed char
|
||||||
|
// straight to std::size_t would trip bugprone-signed-char-misuse)
|
||||||
|
const auto ulen = static_cast<typename std::make_unsigned<NumberType>::type>(len);
|
||||||
|
const std::size_t remaining = ia.bulk_remaining();
|
||||||
|
if (static_cast<std::uintmax_t>(ulen) > static_cast<std::uintmax_t>(remaining))
|
||||||
|
{
|
||||||
|
return false; // (the loop reports the premature end)
|
||||||
|
}
|
||||||
|
const auto n = static_cast<std::size_t>(ulen);
|
||||||
|
result.append(reinterpret_cast<const typename string_t::value_type*>(ia.bulk_data()), n);
|
||||||
|
ia.bulk_skip(n);
|
||||||
|
chars_read += n;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename NumberType, typename ContainerType>
|
||||||
|
static bool append_from_block(const NumberType /*len*/, ContainerType& /*result*/, std::false_type /*block*/) noexcept
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
/*!
|
/*!
|
||||||
@param[in] context further context information (for diagnostics)
|
@param[in] context further context information (for diagnostics)
|
||||||
@return whether the last read character is not EOF
|
@return whether the last read character is not EOF
|
||||||
|
|||||||
File diff suppressed because it is too large.
Load diff
@@ -316,30 +316,6 @@ JSON_HEDLEY_ALWAYS_INLINE std::size_t accumulate_digits(const char* data, std::s
|
|||||||
return i;
|
return i;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// the index of the first byte at or after @a i that is not a digit: the
|
|
||||||
/// digits of accumulate_digits() without their value
|
|
||||||
JSON_HEDLEY_ALWAYS_INLINE std::size_t skip_digits(const char* data, std::size_t i, std::size_t avail) noexcept
|
|
||||||
{
|
|
||||||
while (avail - i >= 8)
|
|
||||||
{
|
|
||||||
if (!is_eight_digits(read_eight_bytes(data + i)))
|
|
||||||
{
|
|
||||||
// a byte of the word is not a digit: no bounds checks before it
|
|
||||||
while (static_cast<unsigned char>(data[i] - '0') <= 9)
|
|
||||||
{
|
|
||||||
++i;
|
|
||||||
}
|
|
||||||
return i;
|
|
||||||
}
|
|
||||||
i += 8;
|
|
||||||
}
|
|
||||||
while (i < avail && data[i] >= '0' && data[i] <= '9')
|
|
||||||
{
|
|
||||||
++i;
|
|
||||||
}
|
|
||||||
return i;
|
|
||||||
}
|
|
||||||
|
|
||||||
/// whether [first, last) contains a digit other than '0'; a plain loop, as
|
/// whether [first, last) contains a digit other than '0'; a plain loop, as
|
||||||
/// GCC calls std::any_of out of line even for the usual empty range
|
/// GCC calls std::any_of out of line even for the usual empty range
|
||||||
JSON_HEDLEY_ALWAYS_INLINE bool has_nonzero_digit(const char* first, const char* last) noexcept
|
JSON_HEDLEY_ALWAYS_INLINE bool has_nonzero_digit(const char* first, const char* last) noexcept
|
||||||
|
|||||||
@@ -520,10 +520,7 @@ JSON_HEDLEY_NEVER_INLINE inline std::size_t string_bulk_run_from(const unsigned
|
|||||||
}
|
}
|
||||||
#endif
|
#endif
|
||||||
|
|
||||||
// string_bulk_run() for the lexer. The run must be the longest one: the lexer
|
// string_bulk_run() for the lexer. With NEON, only the first four words are
|
||||||
// would only lose speed with a shorter run, but contiguous_parser rejects a
|
|
||||||
// string whose run ends at a byte other than a quote or a backslash, so a
|
|
||||||
// shorter run would reject valid input there. With NEON, only the first four words are
|
|
||||||
// checked inline (most keys and short values end there), and longer runs and
|
// checked inline (most keys and short values end there), and longer runs and
|
||||||
// non-ASCII text continue out of line. The lexer's scan function runs once per
|
// non-ASCII text continue out of line. The lexer's scan function runs once per
|
||||||
// token, and the vector code inlined into it needs more registers, which every
|
// token, and the vector code inlined into it needs more registers, which every
|
||||||
|
|||||||
@@ -58,7 +58,6 @@
|
|||||||
#include <nlohmann/detail/exceptions.hpp> // IWYU pragma: export
|
#include <nlohmann/detail/exceptions.hpp> // IWYU pragma: export
|
||||||
#include <nlohmann/detail/hash.hpp>
|
#include <nlohmann/detail/hash.hpp>
|
||||||
#include <nlohmann/detail/input/binary_reader.hpp>
|
#include <nlohmann/detail/input/binary_reader.hpp>
|
||||||
#include <nlohmann/detail/input/contiguous_parser.hpp>
|
|
||||||
#include <nlohmann/detail/input/input_adapters.hpp>
|
#include <nlohmann/detail/input/input_adapters.hpp>
|
||||||
#include <nlohmann/detail/input/json_sax.hpp>
|
#include <nlohmann/detail/input/json_sax.hpp>
|
||||||
#include <nlohmann/detail/input/lexer.hpp>
|
#include <nlohmann/detail/input/lexer.hpp>
|
||||||
@@ -3110,15 +3109,6 @@ public:
|
|||||||
const bool ignore_trailing_commas)
|
const bool ignore_trailing_commas)
|
||||||
{
|
{
|
||||||
basic_json result;
|
basic_json result;
|
||||||
// JSON in one block of memory, with the default options, is read by
|
|
||||||
// the faster contiguous_parser; on an error, the library parser reads
|
|
||||||
// it again to report it (see contiguous_parser)
|
|
||||||
using contiguous = std::integral_constant<bool, detail::can_parse_contiguous<basic_json, typename std::decay<InputAdapterType>::type>::value>;
|
|
||||||
if (cb == nullptr && !ignore_comments && !ignore_trailing_commas
|
|
||||||
&& detail::parse_contiguous(adapter, result, allow_exceptions, contiguous {}))
|
|
||||||
{
|
|
||||||
return result;
|
|
||||||
}
|
|
||||||
parser(std::forward<InputAdapterType>(adapter), std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas).parse(true, result);
|
parser(std::forward<InputAdapterType>(adapter), std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas).parse(true, result);
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
@@ -3129,17 +3119,8 @@ public:
|
|||||||
const bool ignore_comments,
|
const bool ignore_comments,
|
||||||
const bool ignore_trailing_commas)
|
const bool ignore_trailing_commas)
|
||||||
{
|
{
|
||||||
// JSON in one block of memory is checked by the faster
|
|
||||||
// contiguous_parser (see parse_impl())
|
|
||||||
using adapter_t = typename std::decay<InputAdapterType>::type;
|
|
||||||
bool accepted = false;
|
|
||||||
if (!ignore_comments && !ignore_trailing_commas
|
|
||||||
&& detail::accept_contiguous<basic_json>(adapter, accepted, std::integral_constant<bool, detail::can_parse_contiguous<basic_json, adapter_t>::value> {}))
|
|
||||||
{
|
|
||||||
return accepted;
|
|
||||||
}
|
|
||||||
// the parser and lexer that discard the values of tokens (see lexer)
|
// the parser and lexer that discard the values of tokens (see lexer)
|
||||||
return ::nlohmann::detail::parser<basic_json, adapter_t, true>(std::forward<InputAdapterType>(adapter),
|
return ::nlohmann::detail::parser<basic_json, typename std::decay<InputAdapterType>::type, true>(std::forward<InputAdapterType>(adapter),
|
||||||
nullptr, false, ignore_comments, ignore_trailing_commas).accept(true);
|
nullptr, false, ignore_comments, ignore_trailing_commas).accept(true);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -3154,14 +3135,6 @@ public:
|
|||||||
{
|
{
|
||||||
if (format == input_format_t::json)
|
if (format == input_format_t::json)
|
||||||
{
|
{
|
||||||
// JSON in one block of memory is read by the faster
|
|
||||||
// contiguous_parser, with the default options (see parse_impl())
|
|
||||||
bool result = false;
|
|
||||||
if (strict && !ignore_comments && !ignore_trailing_commas
|
|
||||||
&& detail::sax_parse_contiguous<basic_json>(ia, sax, result, std::integral_constant<bool, detail::can_parse_contiguous<basic_json, InputAdapterType>::value> {}))
|
|
||||||
{
|
|
||||||
return result;
|
|
||||||
}
|
|
||||||
return parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict);
|
return parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict);
|
||||||
}
|
}
|
||||||
return detail::binary_reader<basic_json, InputAdapterType, SAX>(std::move(ia), format).sax_parse(sax, strict, tag_handler);
|
return detail::binary_reader<basic_json, InputAdapterType, SAX>(std::move(ia), format).sax_parse(sax, strict, tag_handler);
|
||||||
|
|||||||
Generated
+40
-1093
File diff suppressed because it is too large.
Load diff
@@ -1,564 +0,0 @@
|
|||||||
// __ _____ _____ _____
|
|
||||||
// __| | __| | | | JSON for Modern C++ (supporting code)
|
|
||||||
// | | |__ | | | | | | version 3.12.0
|
|
||||||
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
|
|
||||||
//
|
|
||||||
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
|
|
||||||
// SPDX-License-Identifier: MIT
|
|
||||||
|
|
||||||
// parse() and accept() read text in one block of memory with
|
|
||||||
// detail::contiguous_parser, and everything else (here: std::list iterators)
|
|
||||||
// with the lexer's byte path. These tests check that the two agree on every
|
|
||||||
// input: the same value, with the same number types and float bits, or the
|
|
||||||
// same exception with the same message.
|
|
||||||
|
|
||||||
#include "doctest_compatibility.h"
|
|
||||||
|
|
||||||
#include <nlohmann/json.hpp>
|
|
||||||
using nlohmann::json;
|
|
||||||
using nlohmann::ordered_json;
|
|
||||||
|
|
||||||
#include <cstddef>
|
|
||||||
#include <cstdint>
|
|
||||||
#include <cstring>
|
|
||||||
#include <list>
|
|
||||||
#include <map>
|
|
||||||
#include <random>
|
|
||||||
#include <string>
|
|
||||||
#include <type_traits>
|
|
||||||
#include <vector>
|
|
||||||
|
|
||||||
namespace
|
|
||||||
{
|
|
||||||
|
|
||||||
/// the value with its number types and float bits; below 100 levels, the
|
|
||||||
/// library's dump() (whose recursion is bounded) describes the rest
|
|
||||||
template<typename Json>
|
|
||||||
std::string describe_value(const Json& j, int depth = 0)
|
|
||||||
{
|
|
||||||
std::string out;
|
|
||||||
if (depth >= 100)
|
|
||||||
{
|
|
||||||
out = j.dump();
|
|
||||||
return out;
|
|
||||||
}
|
|
||||||
switch (j.type())
|
|
||||||
{
|
|
||||||
case Json::value_t::number_float:
|
|
||||||
{
|
|
||||||
const double d = j.template get<double>();
|
|
||||||
std::uint64_t bits = 0;
|
|
||||||
std::memcpy(&bits, &d, sizeof(bits));
|
|
||||||
out += "f" + std::to_string(bits);
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
case Json::value_t::number_integer:
|
|
||||||
out += "i" + j.dump();
|
|
||||||
break;
|
|
||||||
case Json::value_t::number_unsigned:
|
|
||||||
out += "u" + j.dump();
|
|
||||||
break;
|
|
||||||
case Json::value_t::array:
|
|
||||||
out += "[";
|
|
||||||
for (const auto& e : j)
|
|
||||||
{
|
|
||||||
out += describe_value(e, depth + 1) + ",";
|
|
||||||
}
|
|
||||||
out += "]";
|
|
||||||
break;
|
|
||||||
case Json::value_t::object:
|
|
||||||
out += "{";
|
|
||||||
for (auto it = j.begin(); it != j.end(); ++it)
|
|
||||||
{
|
|
||||||
out += Json(it.key()).dump() + ":" + describe_value(it.value(), depth + 1) + ",";
|
|
||||||
}
|
|
||||||
out += "}";
|
|
||||||
break;
|
|
||||||
case Json::value_t::null:
|
|
||||||
case Json::value_t::boolean:
|
|
||||||
case Json::value_t::string:
|
|
||||||
case Json::value_t::binary:
|
|
||||||
case Json::value_t::discarded:
|
|
||||||
default:
|
|
||||||
out += j.dump();
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
return out;
|
|
||||||
}
|
|
||||||
|
|
||||||
#if !defined(JSON_NOEXCEPTION)
|
|
||||||
template<typename Json, typename Parse>
|
|
||||||
std::string outcome(Parse parse)
|
|
||||||
{
|
|
||||||
try
|
|
||||||
{
|
|
||||||
return describe_value(parse());
|
|
||||||
}
|
|
||||||
catch (const typename Json::exception& e)
|
|
||||||
{
|
|
||||||
return std::string("exception: ") + e.what();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
#endif
|
|
||||||
|
|
||||||
/// every SAX event with its values, the text of float tokens, and the parse
|
|
||||||
/// error with its position, token and message; returns false after
|
|
||||||
/// @a stop_after events (to stop the parse)
|
|
||||||
class event_log : public nlohmann::json_sax<json>
|
|
||||||
{
|
|
||||||
public:
|
|
||||||
explicit event_log(std::size_t stop_after_ = static_cast<std::size_t>(-1), bool continue_after_error_ = false)
|
|
||||||
: stop_after(stop_after_), continue_after_error(continue_after_error_)
|
|
||||||
{}
|
|
||||||
|
|
||||||
bool null() override
|
|
||||||
{
|
|
||||||
return add("null");
|
|
||||||
}
|
|
||||||
bool boolean(bool val) override
|
|
||||||
{
|
|
||||||
return add(val ? "true" : "false");
|
|
||||||
}
|
|
||||||
bool number_integer(number_integer_t val) override
|
|
||||||
{
|
|
||||||
return add("i" + std::to_string(val));
|
|
||||||
}
|
|
||||||
bool number_unsigned(number_unsigned_t val) override
|
|
||||||
{
|
|
||||||
return add("u" + std::to_string(val));
|
|
||||||
}
|
|
||||||
bool number_float(number_float_t val, const string_t& s) override
|
|
||||||
{
|
|
||||||
std::uint64_t bits = 0;
|
|
||||||
std::memcpy(&bits, &val, sizeof(bits));
|
|
||||||
return add("f" + std::to_string(bits) + " " + s);
|
|
||||||
}
|
|
||||||
bool string(string_t& val) override
|
|
||||||
{
|
|
||||||
return add("s" + val);
|
|
||||||
}
|
|
||||||
bool binary(binary_t& /*val*/) override
|
|
||||||
{
|
|
||||||
return add("binary");
|
|
||||||
}
|
|
||||||
bool start_object(std::size_t elements) override
|
|
||||||
{
|
|
||||||
return add("{" + std::to_string(elements));
|
|
||||||
}
|
|
||||||
bool key(string_t& val) override
|
|
||||||
{
|
|
||||||
return add("k" + val);
|
|
||||||
}
|
|
||||||
bool end_object() override
|
|
||||||
{
|
|
||||||
return add("}");
|
|
||||||
}
|
|
||||||
bool start_array(std::size_t elements) override
|
|
||||||
{
|
|
||||||
return add("[" + std::to_string(elements));
|
|
||||||
}
|
|
||||||
bool end_array() override
|
|
||||||
{
|
|
||||||
return add("]");
|
|
||||||
}
|
|
||||||
bool parse_error(std::size_t position, const std::string& last_token, const nlohmann::detail::exception& ex) override
|
|
||||||
{
|
|
||||||
events.push_back("error " + std::to_string(position) + " " + last_token + " " + ex.what());
|
|
||||||
return continue_after_error;
|
|
||||||
}
|
|
||||||
|
|
||||||
const std::vector<std::string>& log() const noexcept
|
|
||||||
{
|
|
||||||
return events;
|
|
||||||
}
|
|
||||||
|
|
||||||
private:
|
|
||||||
bool add(const std::string& event)
|
|
||||||
{
|
|
||||||
events.push_back(event);
|
|
||||||
return events.size() != stop_after;
|
|
||||||
}
|
|
||||||
|
|
||||||
std::vector<std::string> events {}; // NOLINT(readability-redundant-member-init)
|
|
||||||
std::size_t stop_after;
|
|
||||||
bool continue_after_error;
|
|
||||||
};
|
|
||||||
|
|
||||||
/// sax_parse() of the contiguous path against the byte path, also with a
|
|
||||||
/// handler that stops the parse
|
|
||||||
void check_same_events(const std::string& text, const std::list<char>& bytes)
|
|
||||||
{
|
|
||||||
for (const std::size_t stop_after :
|
|
||||||
{
|
|
||||||
static_cast<std::size_t>(-1), std::size_t{1}, std::size_t{3}
|
|
||||||
})
|
|
||||||
{
|
|
||||||
CAPTURE(stop_after)
|
|
||||||
event_log contiguous(stop_after);
|
|
||||||
event_log byte_path(stop_after);
|
|
||||||
const bool r1 = json::sax_parse(text, &contiguous);
|
|
||||||
const bool r2 = json::sax_parse(bytes.begin(), bytes.end(), &byte_path);
|
|
||||||
CHECK(r1 == r2);
|
|
||||||
CHECK(contiguous.log() == byte_path.log());
|
|
||||||
}
|
|
||||||
// a handler that asks to go on after an error (the library parser then
|
|
||||||
// reports what follows, e.g., the missing end of the input)
|
|
||||||
event_log contiguous(static_cast<std::size_t>(-1), true);
|
|
||||||
event_log byte_path(static_cast<std::size_t>(-1), true);
|
|
||||||
const bool r1 = json::sax_parse(text, &contiguous);
|
|
||||||
const bool r2 = json::sax_parse(bytes.begin(), bytes.end(), &byte_path);
|
|
||||||
CHECK(r1 == r2);
|
|
||||||
CHECK(contiguous.log() == byte_path.log());
|
|
||||||
}
|
|
||||||
|
|
||||||
/// the contiguous path (a std::string) against the byte path (std::list iterators)
|
|
||||||
template<typename Json>
|
|
||||||
void check_same(const std::string& text)
|
|
||||||
{
|
|
||||||
CAPTURE(text)
|
|
||||||
const std::list<char> bytes(text.begin(), text.end());
|
|
||||||
|
|
||||||
#if !defined(JSON_NOEXCEPTION)
|
|
||||||
// with exceptions: the value, or the same exception for both
|
|
||||||
const std::string contiguous = outcome<Json>([&]
|
|
||||||
{
|
|
||||||
return Json::parse(text);
|
|
||||||
});
|
|
||||||
const std::string byte_path = outcome<Json>([&]
|
|
||||||
{
|
|
||||||
return Json::parse(bytes.begin(), bytes.end());
|
|
||||||
});
|
|
||||||
CHECK(contiguous == byte_path);
|
|
||||||
#endif
|
|
||||||
|
|
||||||
// without exceptions: the value, or discarded for both
|
|
||||||
const Json c = Json::parse(text, nullptr, false);
|
|
||||||
const Json b = Json::parse(bytes.begin(), bytes.end(), nullptr, false);
|
|
||||||
CHECK(c.is_discarded() == b.is_discarded());
|
|
||||||
|
|
||||||
CHECK(Json::accept(text) == Json::accept(bytes.begin(), bytes.end()));
|
|
||||||
if (std::is_same<Json, json>::value)
|
|
||||||
{
|
|
||||||
check_same_events(text, bytes);
|
|
||||||
}
|
|
||||||
// a pointer range without a terminating NUL
|
|
||||||
std::vector<char> exact(text.begin(), text.end());
|
|
||||||
if (!exact.empty())
|
|
||||||
{
|
|
||||||
CHECK(Json::accept(exact.data(), exact.data() + exact.size()) == Json::accept(bytes.begin(), bytes.end()));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
void check_both(const std::string& text)
|
|
||||||
{
|
|
||||||
check_same<json>(text);
|
|
||||||
check_same<ordered_json>(text);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// a random JSON text, mostly valid, with the corner cases of the parser
|
|
||||||
class generator
|
|
||||||
{
|
|
||||||
public:
|
|
||||||
explicit generator(std::uint32_t seed) : rng(seed) {}
|
|
||||||
|
|
||||||
std::string document(int depth)
|
|
||||||
{
|
|
||||||
std::string out;
|
|
||||||
value(out, depth);
|
|
||||||
return out;
|
|
||||||
}
|
|
||||||
|
|
||||||
private:
|
|
||||||
std::uint32_t pick(std::uint32_t n)
|
|
||||||
{
|
|
||||||
return static_cast<std::uint32_t>(rng() % n);
|
|
||||||
}
|
|
||||||
|
|
||||||
void whitespace(std::string& out)
|
|
||||||
{
|
|
||||||
static const char* const choices[] = {"", "", "", " ", "\n", "\n ", "\t", "\r\n ", "\n ", " "}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
|
|
||||||
out += choices[pick(10)];
|
|
||||||
}
|
|
||||||
|
|
||||||
void number(std::string& out)
|
|
||||||
{
|
|
||||||
static const char* const choices[] = // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
|
|
||||||
{
|
|
||||||
"0", "-0", "1", "-1", "42", "123456789012345678", "-123456789012345678", "1234567890123456789",
|
|
||||||
"-1234567890123456789", "9223372036854775807", "-9223372036854775808", "-9223372036854775809",
|
|
||||||
"18446744073709551615", "18446744073709551616", "123456789012345678901234567890", "0.0", "-0.0",
|
|
||||||
"1.5", "-65.613616999999977", "1e10", "1E+10", "1e-10", "2.2250738585072014e-308", "4.9e-324",
|
|
||||||
"1e-400", "1.7976931348623157e308", "1e308", "1e309", "-1e309", "123.456e-789", "0.1e1",
|
|
||||||
"12345678901234567890.5", "1.00000000000000011102230246251565404236316680908203125",
|
|
||||||
"9007199254740993", "0.30000000000000004", "1e22", "1e23", "100000000000000000000000",
|
|
||||||
"00", "01", "-", "1.", ".5", "1e", "1e+", "--1", "+1", "1.5e3.2", "0x10", "1_000"
|
|
||||||
};
|
|
||||||
out += choices[pick(sizeof(choices) / sizeof(choices[0]))];
|
|
||||||
}
|
|
||||||
|
|
||||||
void string(std::string& out)
|
|
||||||
{
|
|
||||||
static const char* const pieces[] = // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
|
|
||||||
{
|
|
||||||
"a", "key", "a longer piece of plain ASCII text for the vector paths", "\\n", "\\t", "\\\"",
|
|
||||||
"\\\\", "\\/", "\\b", "\\f", "\\r", "\\u0000", "\\u00e4", "\\u20AC", "\\uD83D\\uDE00",
|
|
||||||
"\\uD83D", "\\uDE00", "\\uD83Dx", "\\u12", "\\x", "\xC3\xA4", "\xE2\x82\xAC", "\xF0\x9F\x98\x80",
|
|
||||||
"\xC0\xAF", "\xED\xA0\x80", "\xF4\x90\x80\x80", "\x80", "\xFF", "\x01", "\x1F", "\x7F", "\t"
|
|
||||||
};
|
|
||||||
out += '"';
|
|
||||||
const auto n = pick(6);
|
|
||||||
for (std::uint32_t i = 0; i < n; ++i)
|
|
||||||
{
|
|
||||||
// mostly well-formed pieces
|
|
||||||
out += pieces[pick(5) == 0 ? pick(sizeof(pieces) / sizeof(pieces[0])) : pick(20)];
|
|
||||||
}
|
|
||||||
out += '"';
|
|
||||||
}
|
|
||||||
|
|
||||||
void value(std::string& out, int depth)
|
|
||||||
{
|
|
||||||
const auto kind = depth > 0 ? pick(9) : 3 + pick(6);
|
|
||||||
switch (kind)
|
|
||||||
{
|
|
||||||
case 0:
|
|
||||||
case 1:
|
|
||||||
{
|
|
||||||
out += '[';
|
|
||||||
whitespace(out);
|
|
||||||
const auto n = pick(6);
|
|
||||||
for (std::uint32_t i = 0; i < n; ++i)
|
|
||||||
{
|
|
||||||
if (i != 0)
|
|
||||||
{
|
|
||||||
out += ',';
|
|
||||||
whitespace(out);
|
|
||||||
}
|
|
||||||
value(out, depth - 1);
|
|
||||||
whitespace(out);
|
|
||||||
}
|
|
||||||
out += ']';
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
case 2:
|
|
||||||
{
|
|
||||||
out += '{';
|
|
||||||
whitespace(out);
|
|
||||||
const auto n = pick(6);
|
|
||||||
for (std::uint32_t i = 0; i < n; ++i)
|
|
||||||
{
|
|
||||||
if (i != 0)
|
|
||||||
{
|
|
||||||
out += ',';
|
|
||||||
whitespace(out);
|
|
||||||
}
|
|
||||||
// a few duplicate keys
|
|
||||||
if (pick(4) == 0)
|
|
||||||
{
|
|
||||||
out += "\"dup\"";
|
|
||||||
}
|
|
||||||
else
|
|
||||||
{
|
|
||||||
string(out);
|
|
||||||
}
|
|
||||||
whitespace(out);
|
|
||||||
out += pick(2) == 0 ? ":" : ": ";
|
|
||||||
whitespace(out);
|
|
||||||
value(out, depth - 1);
|
|
||||||
whitespace(out);
|
|
||||||
}
|
|
||||||
out += '}';
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
case 3:
|
|
||||||
case 4:
|
|
||||||
number(out);
|
|
||||||
break;
|
|
||||||
case 5:
|
|
||||||
case 6:
|
|
||||||
string(out);
|
|
||||||
break;
|
|
||||||
case 7:
|
|
||||||
out += pick(2) == 0 ? "true" : "false";
|
|
||||||
break;
|
|
||||||
default:
|
|
||||||
out += "null";
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
std::mt19937 rng;
|
|
||||||
};
|
|
||||||
|
|
||||||
} // namespace
|
|
||||||
|
|
||||||
TEST_CASE("contiguous parser")
|
|
||||||
{
|
|
||||||
SECTION("fixed inputs")
|
|
||||||
{
|
|
||||||
const char* const inputs[] = // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
|
|
||||||
{
|
|
||||||
"", " ", "\n", "null", "true", "false", "nul", "tru", "fals", "nullx", "[]", "{}", "[", "{", "]", "}",
|
|
||||||
"[1,]", "[,1]", R"({"a":1,})", R"({"a" 1})", R"({"a":})", "{1:2}", "[1 2]", R"("")", R"("abc)", R"("\")",
|
|
||||||
"\xEF\xBB\xBF[1]", "\xEF\xBB[1]", "\xEF[1]", "\xEF\xBB\xBF", " [1] ", "[1]x", "[1] [2]", "1 2",
|
|
||||||
R"({"a":1,"a":2})", R"({"b":1,"a":2,"b":3})", R"(["\u0000"])", "[1e400]", "[-1e400]",
|
|
||||||
R"({"a": [1, 2, {"b": null}], "c": "d"})", "[true,false,null]", "[[[[[]]]]]",
|
|
||||||
};
|
|
||||||
for (const char* input : inputs)
|
|
||||||
{
|
|
||||||
check_both(input);
|
|
||||||
}
|
|
||||||
|
|
||||||
// NUL bytes: by default the end of the input (between values), else an error
|
|
||||||
check_both(std::string("[1]\0garbage", 11));
|
|
||||||
check_both(std::string("[1,\0]", 5));
|
|
||||||
check_both(std::string("\"a\0b\"", 5));
|
|
||||||
check_both(std::string("\0", 1));
|
|
||||||
check_both(std::string("1\0", 2));
|
|
||||||
}
|
|
||||||
|
|
||||||
SECTION("deep nesting")
|
|
||||||
{
|
|
||||||
// across the 64-level words of the nesting stack
|
|
||||||
const std::vector<std::size_t> depths = {63, 64, 65, 127, 128, 129, 200, 1000, 10000};
|
|
||||||
for (const std::size_t depth : depths)
|
|
||||||
{
|
|
||||||
check_both(std::string(depth, '[') + std::string(depth, ']'));
|
|
||||||
std::string objects;
|
|
||||||
for (std::size_t i = 0; i < depth; ++i)
|
|
||||||
{
|
|
||||||
objects += (i % 2 == 0) ? "{\"k\":" : "[";
|
|
||||||
}
|
|
||||||
objects += "1";
|
|
||||||
for (std::size_t i = depth; i-- > 0;)
|
|
||||||
{
|
|
||||||
objects += (i % 2 == 0) ? "}" : "]";
|
|
||||||
}
|
|
||||||
check_both(objects);
|
|
||||||
// unbalanced
|
|
||||||
check_both(std::string(depth, '[') + std::string(depth - 1, ']'));
|
|
||||||
check_both(std::string(depth, '[') + std::string(depth + 1, ']'));
|
|
||||||
}
|
|
||||||
// a value after a container that closes a 64-level word
|
|
||||||
std::string mixed = std::string(64, '[') + "1" + std::string(63, ']') + ",{\"a\":2}]";
|
|
||||||
check_both(mixed);
|
|
||||||
mixed = "{\"x\":" + std::string(64, '[') + std::string(64, ']') + ",\"y\":[1]}";
|
|
||||||
check_both(mixed);
|
|
||||||
}
|
|
||||||
|
|
||||||
SECTION("generated documents and their mutations")
|
|
||||||
{
|
|
||||||
generator gen(20261105u);
|
|
||||||
std::mt19937 rng(42u); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed): reproducible
|
|
||||||
const char alphabet[] = "{}[]:,\"\\ \t\n\r0123456789-+.eEtrufalsn/u\x80\xbf\xc3\xe2\xed\xf0\xff"; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
|
|
||||||
for (int round = 0; round < 1500; ++round)
|
|
||||||
{
|
|
||||||
const std::string text = gen.document(1 + (round % 5));
|
|
||||||
check_both(text);
|
|
||||||
// every truncation of short documents, and a few mutations
|
|
||||||
if (text.size() < 60)
|
|
||||||
{
|
|
||||||
for (std::size_t n = 0; n < text.size(); ++n)
|
|
||||||
{
|
|
||||||
check_same<json>(text.substr(0, n));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
for (int m = 0; m < 3 && !text.empty(); ++m)
|
|
||||||
{
|
|
||||||
std::string mutated = text;
|
|
||||||
const std::size_t pos = rng() % mutated.size();
|
|
||||||
switch (rng() % 3)
|
|
||||||
{
|
|
||||||
case 0:
|
|
||||||
mutated[pos] = alphabet[rng() % (sizeof(alphabet) - 1)];
|
|
||||||
break;
|
|
||||||
case 1:
|
|
||||||
mutated.insert(pos, 1, alphabet[rng() % (sizeof(alphabet) - 1)]);
|
|
||||||
break;
|
|
||||||
default:
|
|
||||||
mutated.erase(pos, 1);
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
check_same<json>(mutated);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
SECTION("narrow number types")
|
|
||||||
{
|
|
||||||
// integers that do not fit become floats, floats are converted to float
|
|
||||||
using narrow_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int32_t, std::uint32_t, float>;
|
|
||||||
const std::vector<std::string> inputs =
|
|
||||||
{
|
|
||||||
"[2147483647, 2147483648, -2147483648, -2147483649, 4294967295, 4294967296, 18446744073709551615]",
|
|
||||||
"[1.5, 3.4028234663852886e38, 3.4028235677973366e38, 1e39, -1e39, 1e-46, 0.1, 16777217]",
|
|
||||||
"[-0, 0.0, 123456789012345678901234567890, -9223372036854775809]", "1e39", "4294967296", "-2147483649"
|
|
||||||
};
|
|
||||||
for (const auto& input : inputs)
|
|
||||||
{
|
|
||||||
check_same<narrow_json>(input);
|
|
||||||
}
|
|
||||||
generator gen(17u);
|
|
||||||
for (int round = 0; round < 200; ++round)
|
|
||||||
{
|
|
||||||
check_same<narrow_json>(gen.document(1 + (round % 4)));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
SECTION("number corner cases at the end of the input")
|
|
||||||
{
|
|
||||||
// the number ends at the end of the block (no closing bracket after it)
|
|
||||||
for (const char* number :
|
|
||||||
{"0", "-0", "1", "12345678", "123456789", "1234567890123456789", "18446744073709551616",
|
|
||||||
"1.5", "1e5", "1e", "1.", "-", "1.5e-3", "123456789012345678901234567890e-10"
|
|
||||||
})
|
|
||||||
{
|
|
||||||
check_both(number);
|
|
||||||
check_both(std::string("[") + number + "]");
|
|
||||||
check_both(std::string("{\"n\":") + number + "}");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
SECTION("accept() of floats near the largest value")
|
|
||||||
{
|
|
||||||
// accept() skips digits without their value and computes it only for
|
|
||||||
// a float whose exponent may reach the largest value
|
|
||||||
for (const char* number :
|
|
||||||
{"17976931348623158e292", "17976931348623159e292", "1797693134862315.8e293", "1797693134862315.9e293",
|
|
||||||
"0.17976931348623158e309", "0.17976931348623159e309", "-17976931348623159e292", "9999999999999999999e289"
|
|
||||||
})
|
|
||||||
{
|
|
||||||
check_both(number);
|
|
||||||
check_both(std::string("[") + number + "]");
|
|
||||||
}
|
|
||||||
CHECK(json::accept("17976931348623158e292"));
|
|
||||||
CHECK_FALSE(json::accept("17976931348623159e292"));
|
|
||||||
CHECK(json::accept("[1797693134862315.8e293]"));
|
|
||||||
CHECK_FALSE(json::accept("[1797693134862315.9e293]"));
|
|
||||||
CHECK(json::accept("0.17976931348623158e309"));
|
|
||||||
CHECK_FALSE(json::accept("0.17976931348623159e309"));
|
|
||||||
CHECK_FALSE(json::accept("-17976931348623159e292"));
|
|
||||||
|
|
||||||
using narrow_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int32_t, std::uint32_t, float>;
|
|
||||||
check_same<narrow_json>("34028235e31");
|
|
||||||
check_same<narrow_json>("34028236e31");
|
|
||||||
CHECK(narrow_json::accept("34028235e31"));
|
|
||||||
CHECK_FALSE(narrow_json::accept("34028236e31"));
|
|
||||||
}
|
|
||||||
|
|
||||||
SECTION("callbacks, comments and trailing commas take the library parser")
|
|
||||||
{
|
|
||||||
// (the results are those of the options; the contiguous parser does not support them)
|
|
||||||
CHECK(json::parse("[1,]", nullptr, true, false, true) == json({1}));
|
|
||||||
CHECK(json::parse("[1/*c*/]", nullptr, true, true) == json({1}));
|
|
||||||
CHECK(json::accept("[1,]", false, true));
|
|
||||||
CHECK(json::accept("// c\n[1]", true));
|
|
||||||
int calls = 0;
|
|
||||||
const json j = json::parse("[1,2]", [&calls](int /*depth*/, json::parse_event_t /*event*/, json& /*parsed*/)
|
|
||||||
{
|
|
||||||
++calls;
|
|
||||||
return true;
|
|
||||||
});
|
|
||||||
CHECK(j == json({1, 2}));
|
|
||||||
CHECK(calls == 4);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Reference in new issue
Block a user