Compare commits

...
8 Commits
Author SHA1 Message Date
Niels Lohmann 30e542e52a Declare the namespace-scope constants in to_chars.hpp inline (#5776)
MSVC warns with C5260 that kAlpha and kGamma have internal linkage when
the header is used as a header unit or through a module. JSON_INLINE_VARIABLE
makes them inline variables from C++17 on.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-07 09:03:18 +02:00
Niels Lohmann 24dd7c63f0 Re-amalgamate after #5740 (#5777)
#5740 changed include/nlohmann/json.hpp without regenerating
single_include/nlohmann/json.hpp.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-07 09:03:17 +02:00
Niels Lohmann e6f32bd28a Stricter fuzzer checks and boundary-value tests for buffers (#5774)
* Check in the fuzzers that parsing without exceptions agrees

Each fuzzer driver now also parses its input with allow_exceptions =
false. That call must never throw a parse_error, must return a discarded
value where parsing with exceptions fails, and must return the same value
where it succeeds. Values are compared by their dump(), because NaN is
not equal to itself.

A plain !is_discarded() assertion, as suggested in #3642, would never
fail: the drivers parse with exceptions, so a result can never be
discarded.

tests/fuzzing.md describes the checks and notes that OSS-Fuzz and
CIFuzz already run LeakSanitizer, because their default address
sanitizer includes it.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Use JSON_HAS_RANGE_VIEW_CONVERSION in the range view regression tests

#5728 combined the JSON_HAS_RANGES and MinGW conditions into
JSON_HAS_RANGE_VIEW_CONVERSION, but three test guards still spelled
them out.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Test the serializer's buffers at their boundaries

The dump() indent overflow survived full line coverage because the tests
grew its buffer by only one step. This adds tests that land exactly on,
and one past, the limits of the other two serializer buffers:

- write_buffer (1024 bytes): strings of 1023, 1024 and 1025 bytes at the
  top level, and of 1022 and 1023 bytes inside an array, so that both
  guards in put_string() are hit at their boundary. Each is checked for
  dump() and for stream output.
- string_buffer (512 bytes, flushed when fewer than 13 bytes remain):
  runs of two-byte escapes, and a surrogate pair written with 14 bytes of
  room, right after a flush, and one escape later.
- The 8-byte bulk scan from the serializer side: 0 to 17 plain bytes
  followed by a quote, a control character, or a non-ASCII character.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Test the chunked string and binary reads of all binary formats

The binary readers read strings and binary values in chunks of 4096
bytes. Only CBOR tested lengths around that size. MessagePack, UBJSON,
BJData and BSON now round-trip lengths 0, 1, 4095, 4096, 4097, 8192 and
100000 from vector and pointer input, and must report a truncated
payload as a parse error.

UBJSON reads binary values as arrays of numbers, so it is tested with
strings only. BJData binary values reach the chunked read only in
Draft 3. BON8 decodes strings byte by byte and does not use this path.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-07 08:50:12 +02:00
Niels Lohmann e4e7d657ef Document the API stability guarantee in the roadmap (#5775)
* Document what is not covered by the API stability guarantee

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Move the API stability guarantee to the roadmap

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Note that exceptions to the API stability rules are documented in the release notes

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-07 08:49:51 +02:00
Niels Lohmann ef570827e3 Point the README's fuzzing badge to the current OSS-Fuzz tracker (#5773)
OSS-Fuzz moved its issues from bugs.chromium.org to issues.oss-fuzz.com.
The old link now redirects to the new tracker but drops the project
filter, so it showed every project's issues.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-07 07:39:44 +02:00
Niels Lohmann 1c754cfe31 Copy a pair-shaped array value under JSON_BRACE_INIT_COPY_SEMANTICS (#5701)
With JSON_BRACE_INIT_COPY_SEMANTICS enabled, single-element brace
initialization from a JSON value decided whether to copy the value or
build an object by inspecting the value's runtime shape: a two-element
array whose first element is a string, such as ["key", 42], was turned
into an object instead of being copied. This made the behavior depend
on the element's content, and it did not distinguish an existing value
of this shape from a nested braced pair written in the source, such as
the inner {"key", "value"} of {{"key", "value"}}.

json_ref now records whether it was constructed from a braced list
(true only for the std::initializer_list<json_ref> constructor used
for nested braced lists) or from a value. The initializer-list
constructor uses this to copy or move a single non-braced-list element
before deciding whether the list describes an object, so a JSON value
is always copied regardless of its shape, while a braced pair written
in the source still creates an object.

Fixes #5662.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-07 07:39:05 +02:00
Niels LohmannandJoseph.Demarest ff6f3d7d4a Reject nested indefinite-length CBOR string chunks (#5766)
* fix(cbor): reject nested indefinite string chunks

Signed-off-by: Joseph.Demarest <joseph@demarest.dev>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Address review comments on nested indefinite-length CBOR strings

Rename is_chunk to inside_indefinite, update the stale test section
names, and use lowercase comments like the surrounding code.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Joseph.Demarest <joseph@demarest.dev>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Joseph.Demarest <joseph@demarest.dev>
2026-10-07 07:38:52 +02:00
Mohd Quamar TyagiandNiels Lohmann 675e519966 Allow SAX parsing of tagged CBOR (#5740)
* Allow SAX parsing of tagged CBOR

Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>

* Consolidate sax_parse overloads with default tag_handler parameter

Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>

* Add version history entry for tag_handler in sax_parse documentation

---------

Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Niels Lohmann <mail@nlohmann.me>
2026-10-07 07:38:24 +02:00
28 changed files with 941 additions and 134 deletions

No files matched your search

+3
View File
@@ -205,6 +205,9 @@ API of the 3.x.y version is broken. This includes:
- Changing access specifiers.
- Changing default arguments.
What is and is not covered by this guarantee is described in the
[roadmap](https://json.nlohmann.me/community/roadmap/#api-stability).
Although these guidelines may seem restrictive, they are essential for maintaining the library’s utility.
Breaking changes may be introduced when they are guarded with a feature macro such as
+1 -1
View File
@@ -7,7 +7,7 @@
[![Coverage Status](https://coveralls.io/repos/github/nlohmann/json/badge.svg?branch=develop)](https://coveralls.io/github/nlohmann/json?branch=develop)
[![Coverity Scan Build Status](https://scan.coverity.com/projects/5550/badge.svg)](https://scan.coverity.com/projects/nlohmann-json)
[![Codacy Badge](https://app.codacy.com/project/badge/Grade/e0d1a9d5d6fd46fcb655c4cb930bb3e8)](https://app.codacy.com/gh/nlohmann/json/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)
[![Fuzzing Status](https://oss-fuzz-build-logs.storage.googleapis.com/badges/json.svg)](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:json)
[![Fuzzing Status](https://oss-fuzz-build-logs.storage.googleapis.com/badges/json.svg)](https://issues.oss-fuzz.com/issues?q=project:json)
[![Try online](https://img.shields.io/badge/try-online-blue.svg)](https://wandbox.org/permlink/1mp10JbaANo6FUc7)
[![Documentation](https://img.shields.io/badge/docs-mkdocs-blue.svg)](https://json.nlohmann.me)
[![GitHub license](https://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/nlohmann/json/develop/LICENSE.MIT)
@@ -9,10 +9,10 @@ enum class cbor_tag_handler_t
};
```
This enumeration is used in the [`from_cbor`](from_cbor.md) function to choose how to treat tags:
This enumeration is used in [`from_cbor`](from_cbor.md) and [`sax_parse`](sax_parse.md) to choose how to treat tags:
error
: throw a `parse_error` exception in case of a tag
: report a parse error in case of a tag (the `from_cbor` overloads throw a `parse_error` exception by default)
ignore
: ignore tags
+10 -3
View File
@@ -8,7 +8,8 @@ static bool sax_parse(InputType&& i,
input_format_t format = input_format_t::json,
const bool strict = true,
const bool ignore_comments = false,
const bool ignore_trailing_commas = false);
const bool ignore_trailing_commas = false,
const cbor_tag_handler_t tag_handler = cbor_tag_handler_t::error);
// (2)
template<class IteratorType, class SAX, class SentinelType = IteratorType>
@@ -17,13 +18,14 @@ static bool sax_parse(IteratorType first, SentinelType last,
input_format_t format = input_format_t::json,
const bool strict = true,
const bool ignore_comments = false,
const bool ignore_trailing_commas = false);
const bool ignore_trailing_commas = false,
const cbor_tag_handler_t tag_handler = cbor_tag_handler_t::error);
```
Read from input and generate SAX events
1. Read from a compatible input.
2. Read from a pair of character iterators, or an iterator and a sentinel of a different type (C++20 ranges support)
2. Read from a pair of character iterators, or an iterator and a sentinel of a different type (C++20 ranges support).
The value_type of the iterator must be an integral type with a size of 1, 2, or 4 bytes, which will be interpreted
respectively as UTF-8, UTF-16, and UTF-32. If `SentinelType` differs from `IteratorType`, it must be comparable to
@@ -82,6 +84,10 @@ The SAX event lister must follow the interface of [`json_sax`](../json_sax/index
: whether trailing commas in arrays or objects should be ignored and treated like whitespace (`#!cpp true`) or yield a parse error
(`#!cpp false`); (optional, `#!cpp false` by default)
`tag_handler` (in)
: how to handle CBOR tags; see [`cbor_tag_handler_t`](cbor_tag_handler_t.md). Ignored for formats other than CBOR
(optional, `cbor_tag_handler_t::error` by default).
`first` (in)
: iterator to the start of a character range
@@ -137,6 +143,7 @@ A UTF-8 byte order mark is silently ignored.
- Added in version 3.2.0.
- Ignoring comments via `ignore_comments` added in version 3.9.0.
- Added `ignore_trailing_commas` in version 3.13.0.
- Added `tag_handler` in version 3.13.0.
- Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
- Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
- `JSON_PRECISE_STREAM_POSITION` added in version 3.13.0 to optionally leave a `#!cpp std::istream` positioned right
@@ -50,9 +50,11 @@ The default value is `0` (disabled — existing behavior is preserved).
```
Code that relies on these producing arrays must use `json::array()` instead (see below). Lists with more than one
element, and a single `[string, value]` pair such as `{{"key", "value"}}`, which still creates an object, are not
affected. The library's own conversions are not affected either: for example, `std::tuple<int>{5}` still becomes
`[5]`.
element, and a single `[string, value]` pair *written as a braced list*, such as `{{"key", "value"}}`, which still
creates an object, are not affected. This exception is based on how the pair is written, not on the shape of its
value: an existing JSON value that happens to be a two-element array with a string as its first element, such as
`json arr = {"key", 42};`, is still copied by `json j{arr};` rather than turned into an object. The library's own
conversions are not affected either: for example, `std::tuple<int>{5}` still becomes `[5]`.
!!! note "ABI compatibility"
+27 -3
View File
@@ -25,9 +25,7 @@ work items are tracked in the [GitHub milestones](https://github.com/nlohmann/js
## What the project will not do
- **Break the public API of version 3.x.** See the
[contribution guidelines](https://github.com/nlohmann/json/blob/develop/.github/CONTRIBUTING.md#break-the-public-api)
for what counts as a breaking change.
- **Break the public API of version 3.x.** See [API stability](#api-stability) for what this covers.
- **Require a newer C++ standard than C++11.**
- **Break JSON conformance** or enable non-standard extensions by default.
- **Add dependencies** or require a build step. The library remains header-only, and the single header
@@ -35,6 +33,32 @@ work items are tracked in the [GitHub milestones](https://github.com/nlohmann/js
- **Trade simplicity for speed or memory efficiency.** Performance improvements are welcome, but the library is not
meant to compete with the fastest JSON libraries, see [Design goals](../home/design_goals.md).
## API stability
Releases follow [semantic versioning](https://semver.org): a minor or patch release of version 3.x does not break code
that uses the public API. In particular, a 3.x release does not:
- change the signature of a function (its parameter types, return type, number of parameters, or the const-ness of a
member function);
- remove or rename a function or class;
- change which exceptions a function throws, or the [exception ids](../home/exceptions.md);
- change access specifiers or default arguments.
Exceptions to these rules, for instance when fixing a bug requires changing the exception a function throws, are
documented in the [release notes](../home/releases.md).
The following are **not** part of the public API and may change in any release, including patch releases:
- The text of exception messages returned by `what()`. Use the [exception id](../home/exceptions.md) to tell errors
apart.
- The ABI, including `sizeof(basic_json)` and the memory layout of its values. Recompile your code when you upgrade the
library. The [versioned inline namespace](../features/namespace.md) turns mixing versions into a link error.
- Everything in namespace `nlohmann::detail`, and macros and type traits that are not documented in the
[API reference](../api/basic_json/index.md).
Changes that would break the public API are only added behind a macro whose default keeps the 3.x behavior, see
[Version 4.0](#version-40).
## Version 4.0
There is no release date for version 4.0 yet. Proposals that need a major version, for instance stricter type
@@ -131,6 +131,7 @@ The library maps CBOR types to JSON value types as follows:
| Byte string | binary | 0x59 |
| Byte string | binary | 0x5A |
| Byte string | binary | 0x5B |
| Byte string | binary | 0x5F |
| UTF-8 string | string | 0x60..0x77 |
| UTF-8 string | string | 0x78 |
| UTF-8 string | string | 0x79 |
@@ -156,6 +157,9 @@ The library maps CBOR types to JSON value types as follows:
| Single-Precision Float | number_float | 0xFA |
| Double-Precision Float | number_float | 0xFB |
Indefinite-length UTF-8 strings (0x7F) and byte strings (0x5F) are supported. Each chunk must be a definite-length
string of the same major type, as required by [RFC 8949, Section 3.2.3](https://www.rfc-editor.org/rfc/rfc8949.html#section-3.2.3).
!!! warning "Incomplete mapping"
The mapping is **incomplete** in the sense that not all CBOR types can be converted to a JSON value. The following CBOR types are not supported and will yield parse errors:
@@ -308,8 +308,8 @@ boundaries compute_boundaries(FloatType value)
//
// -e <= 60 or e >= -60 := alpha
constexpr int kAlpha = -60;
constexpr int kGamma = -32;
JSON_INLINE_VARIABLE constexpr int kAlpha = -60;
JSON_INLINE_VARIABLE constexpr int kGamma = -32;
struct cached_power // c = f * 2^e ~= 10^k
{
+50 -40
View File
@@ -1072,6 +1072,20 @@ class binary_reader
}
}
/*!
@brief reports a nested indefinite-length CBOR string or byte array
@param[in] type_name name of the rejected string type
@param[in] context parsing context for the error message
@return whether the SAX consumer accepts the parse error
*/
bool cbor_indefinite_string_error(const char* type_name, const char* context)
{
auto last_token = get_token_string();
return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read,
exception_message(concat("indefinite-length ", type_name,
" is not allowed inside indefinite-length ", type_name, "; last byte: 0x", last_token), context), nullptr));
}
/*!
@brief reads a definite-length CBOR string
@@ -1081,12 +1095,13 @@ class binary_reader
into the same string.
@param[out] result string the bytes are appended to
@param[in] inside_indefinite whether the bytes belong to an indefinite-length string
@return whether string creation completed
@pre @a current is not EOF
*/
bool get_cbor_string_chunk(string_t& result)
bool get_cbor_string_chunk(string_t& result, const bool inside_indefinite)
{
switch (current)
{
@@ -1147,7 +1162,7 @@ class binary_reader
{
auto last_token = get_token_string();
return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read,
exception_message(concat("expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0x", last_token), "string"), nullptr));
exception_message(concat("expected length specification (0x60-0x7B)", inside_indefinite ? "" : " or indefinite string type (0x7F)", "; last byte: 0x", last_token), "string"), nullptr));
}
}
}
@@ -1165,13 +1180,9 @@ class binary_reader
*/
bool get_cbor_string(string_t& result, const char* context = "string")
{
// number of indefinite-length strings that have been opened and not
// closed yet. RFC 8949, Section 3.2.3 does not permit nesting them,
// but this reader has always accepted it, so the open levels are
// counted instead of recursed through, which overflowed the stack for
// an input of repeated 0x7F bytes (see #5104). Every chunk is appended
// to the same result, so no per-level state is needed.
std::size_t open = 0;
// read chunks iteratively, but reject a second indefinite-length
// level as required by RFC 8949, Section 3.2.3
bool indefinite = false;
while (true)
{
@@ -1182,29 +1193,28 @@ class binary_reader
if (current == 0x7F) // UTF-8 string (indefinite length)
{
++open;
get();
continue;
}
// a break marker closes the innermost indefinite-length string;
// outside of one it is not a string and falls through to the error
if (open != 0 && current == 0xFF)
{
if (--open == 0)
if (JSON_HEDLEY_UNLIKELY(indefinite))
{
return check_string_utf8(result, context);
return cbor_indefinite_string_error("string", "string");
}
indefinite = true;
get();
continue;
}
if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result)))
// a break marker closes the indefinite-length string; outside
// of one it falls through to the error below
if (indefinite && current == 0xFF)
{
return check_string_utf8(result, context);
}
if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result, indefinite)))
{
return false;
}
if (open == 0)
if (!indefinite)
{
return check_string_utf8(result, context);
}
@@ -1296,12 +1306,13 @@ class binary_reader
read into the same byte array.
@param[out] result byte array the bytes are appended to
@param[in] inside_indefinite whether the bytes belong to an indefinite-length string
@return whether byte array creation completed
@pre @a current is not EOF
*/
bool get_cbor_binary_chunk(binary_t& result)
bool get_cbor_binary_chunk(binary_t& result, const bool inside_indefinite)
{
switch (current)
{
@@ -1366,7 +1377,7 @@ class binary_reader
{
auto last_token = get_token_string();
return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read,
exception_message(concat("expected length specification (0x40-0x5B) or indefinite binary array type (0x5F); last byte: 0x", last_token), "binary"), nullptr));
exception_message(concat("expected length specification (0x40-0x5B)", inside_indefinite ? "" : " or indefinite binary array type (0x5F)", "; last byte: 0x", last_token), "binary"), nullptr));
}
}
}
@@ -1384,9 +1395,9 @@ class binary_reader
*/
bool get_cbor_binary(binary_t& result)
{
// the open indefinite-length byte arrays are counted rather than
// recursed through, for the reason given in @ref get_cbor_string
std::size_t open = 0;
// read chunks iteratively, but reject a second indefinite-length
// level as required by RFC 8949, Section 3.2.3
bool indefinite = false;
while (true)
{
@@ -1397,29 +1408,28 @@ class binary_reader
if (current == 0x5F) // Binary data (indefinite length)
{
++open;
get();
continue;
}
// a break marker closes the innermost indefinite-length byte
// array; outside of one it falls through to the error below
if (open != 0 && current == 0xFF)
{
if (--open == 0)
if (JSON_HEDLEY_UNLIKELY(indefinite))
{
return true;
return cbor_indefinite_string_error("binary array", "binary");
}
indefinite = true;
get();
continue;
}
if (JSON_HEDLEY_UNLIKELY(!get_cbor_binary_chunk(result)))
// a break marker closes the indefinite-length string; outside
// of one it falls through to the error below
if (indefinite && current == 0xFF)
{
return true;
}
if (JSON_HEDLEY_UNLIKELY(!get_cbor_binary_chunk(result, indefinite)))
{
return false;
}
if (open == 0)
if (!indefinite)
{
return true;
}
+9
View File
@@ -34,6 +34,7 @@ class json_ref
json_ref(std::initializer_list<json_ref> init)
: owned_value(init)
, braced_list(true)
{}
template <
@@ -69,9 +70,17 @@ class json_ref
return &** this;
}
/// whether the value was written as a braced list, such as {"key", 1},
/// rather than given as a value
bool is_braced_list() const noexcept
{
return braced_list;
}
private:
mutable value_type owned_value = nullptr;
value_type const* value_ref = nullptr;
bool braced_list = false;
};
} // namespace detail
+21 -6
View File
@@ -2252,6 +2252,18 @@ public:
bool type_deduction = true,
value_t manual_type = value_t::array)
{
#if JSON_BRACE_INIT_COPY_SEMANTICS
// a single element that is a value rather than a braced list is
// copied or moved as is, whatever its content looks like
if (type_deduction && init.size() == 1 && !init.begin()->is_braced_list())
{
*this = init.begin()->moved_or_copied();
set_parents();
assert_invariant();
return;
}
#endif
// check if each element is an array with two elements whose first
// element is a string
bool is_an_object = std::all_of(init.begin(), init.end(),
@@ -5671,12 +5683,13 @@ public:
input_format_t format = input_format_t::json,
const bool strict = true,
const bool ignore_comments = false,
const bool ignore_trailing_commas = false)
const bool ignore_trailing_commas = false,
const cbor_tag_handler_t tag_handler = cbor_tag_handler_t::error)
{
auto ia = detail::input_adapter(std::forward<InputType>(i));
return format == input_format_t::json
? parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict)
: detail::binary_reader<basic_json, decltype(ia), SAX>(std::move(ia), format).sax_parse(sax, strict);
: detail::binary_reader<basic_json, decltype(ia), SAX>(std::move(ia), format).sax_parse(sax, strict, tag_handler);
}
/// @brief generate SAX events (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -5688,12 +5701,13 @@ public:
input_format_t format = input_format_t::json,
const bool strict = true,
const bool ignore_comments = false,
const bool ignore_trailing_commas = false)
const bool ignore_trailing_commas = false,
const cbor_tag_handler_t tag_handler = cbor_tag_handler_t::error)
{
auto ia = detail::input_adapter(std::move(first), std::move(last));
return format == input_format_t::json
? parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict)
: detail::binary_reader<basic_json, decltype(ia), SAX>(std::move(ia), format).sax_parse(sax, strict);
: detail::binary_reader<basic_json, decltype(ia), SAX>(std::move(ia), format).sax_parse(sax, strict, tag_handler);
}
/// @brief generate SAX events
@@ -5721,7 +5735,8 @@ public:
input_format_t format = input_format_t::json,
const bool strict = true,
const bool ignore_comments = false,
const bool ignore_trailing_commas = false)
const bool ignore_trailing_commas = false,
const cbor_tag_handler_t tag_handler = cbor_tag_handler_t::error)
#if JSON_DELETE_DEPRECATED_FUNCTIONS
= delete;
#else
@@ -5731,7 +5746,7 @@ public:
// NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
? parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict)
// NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
: detail::binary_reader<basic_json, decltype(ia), SAX>(std::move(ia), format).sax_parse(sax, strict);
: detail::binary_reader<basic_json, decltype(ia), SAX>(std::move(ia), format).sax_parse(sax, strict, tag_handler);
}
#endif
#if defined(__clang__)
+82 -48
View File
@@ -14768,6 +14768,20 @@ class binary_reader
}
}
/*!
@brief reports a nested indefinite-length CBOR string or byte array
@param[in] type_name name of the rejected string type
@param[in] context parsing context for the error message
@return whether the SAX consumer accepts the parse error
*/
bool cbor_indefinite_string_error(const char* type_name, const char* context)
{
auto last_token = get_token_string();
return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read,
exception_message(concat("indefinite-length ", type_name,
" is not allowed inside indefinite-length ", type_name, "; last byte: 0x", last_token), context), nullptr));
}
/*!
@brief reads a definite-length CBOR string
@@ -14777,12 +14791,13 @@ class binary_reader
into the same string.
@param[out] result string the bytes are appended to
@param[in] inside_indefinite whether the bytes belong to an indefinite-length string
@return whether string creation completed
@pre @a current is not EOF
*/
bool get_cbor_string_chunk(string_t& result)
bool get_cbor_string_chunk(string_t& result, const bool inside_indefinite)
{
switch (current)
{
@@ -14843,7 +14858,7 @@ class binary_reader
{
auto last_token = get_token_string();
return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read,
exception_message(concat("expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0x", last_token), "string"), nullptr));
exception_message(concat("expected length specification (0x60-0x7B)", inside_indefinite ? "" : " or indefinite string type (0x7F)", "; last byte: 0x", last_token), "string"), nullptr));
}
}
}
@@ -14861,13 +14876,9 @@ class binary_reader
*/
bool get_cbor_string(string_t& result, const char* context = "string")
{
// number of indefinite-length strings that have been opened and not
// closed yet. RFC 8949, Section 3.2.3 does not permit nesting them,
// but this reader has always accepted it, so the open levels are
// counted instead of recursed through, which overflowed the stack for
// an input of repeated 0x7F bytes (see #5104). Every chunk is appended
// to the same result, so no per-level state is needed.
std::size_t open = 0;
// read chunks iteratively, but reject a second indefinite-length
// level as required by RFC 8949, Section 3.2.3
bool indefinite = false;
while (true)
{
@@ -14878,29 +14889,28 @@ class binary_reader
if (current == 0x7F) // UTF-8 string (indefinite length)
{
++open;
get();
continue;
}
// a break marker closes the innermost indefinite-length string;
// outside of one it is not a string and falls through to the error
if (open != 0 && current == 0xFF)
{
if (--open == 0)
if (JSON_HEDLEY_UNLIKELY(indefinite))
{
return check_string_utf8(result, context);
return cbor_indefinite_string_error("string", "string");
}
indefinite = true;
get();
continue;
}
if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result)))
// a break marker closes the indefinite-length string; outside
// of one it falls through to the error below
if (indefinite && current == 0xFF)
{
return check_string_utf8(result, context);
}
if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result, indefinite)))
{
return false;
}
if (open == 0)
if (!indefinite)
{
return check_string_utf8(result, context);
}
@@ -14992,12 +15002,13 @@ class binary_reader
read into the same byte array.
@param[out] result byte array the bytes are appended to
@param[in] inside_indefinite whether the bytes belong to an indefinite-length string
@return whether byte array creation completed
@pre @a current is not EOF
*/
bool get_cbor_binary_chunk(binary_t& result)
bool get_cbor_binary_chunk(binary_t& result, const bool inside_indefinite)
{
switch (current)
{
@@ -15062,7 +15073,7 @@ class binary_reader
{
auto last_token = get_token_string();
return sax->parse_error(chars_read, last_token, parse_error::create(113, chars_read,
exception_message(concat("expected length specification (0x40-0x5B) or indefinite binary array type (0x5F); last byte: 0x", last_token), "binary"), nullptr));
exception_message(concat("expected length specification (0x40-0x5B)", inside_indefinite ? "" : " or indefinite binary array type (0x5F)", "; last byte: 0x", last_token), "binary"), nullptr));
}
}
}
@@ -15080,9 +15091,9 @@ class binary_reader
*/
bool get_cbor_binary(binary_t& result)
{
// the open indefinite-length byte arrays are counted rather than
// recursed through, for the reason given in @ref get_cbor_string
std::size_t open = 0;
// read chunks iteratively, but reject a second indefinite-length
// level as required by RFC 8949, Section 3.2.3
bool indefinite = false;
while (true)
{
@@ -15093,29 +15104,28 @@ class binary_reader
if (current == 0x5F) // Binary data (indefinite length)
{
++open;
get();
continue;
}
// a break marker closes the innermost indefinite-length byte
// array; outside of one it falls through to the error below
if (open != 0 && current == 0xFF)
{
if (--open == 0)
if (JSON_HEDLEY_UNLIKELY(indefinite))
{
return true;
return cbor_indefinite_string_error("binary array", "binary");
}
indefinite = true;
get();
continue;
}
if (JSON_HEDLEY_UNLIKELY(!get_cbor_binary_chunk(result)))
// a break marker closes the indefinite-length string; outside
// of one it falls through to the error below
if (indefinite && current == 0xFF)
{
return true;
}
if (JSON_HEDLEY_UNLIKELY(!get_cbor_binary_chunk(result, indefinite)))
{
return false;
}
if (open == 0)
if (!indefinite)
{
return true;
}
@@ -21088,6 +21098,7 @@ class json_ref
json_ref(std::initializer_list<json_ref> init)
: owned_value(init)
, braced_list(true)
{}
template <
@@ -21123,9 +21134,17 @@ class json_ref
return &** this;
}
/// whether the value was written as a braced list, such as {"key", 1},
/// rather than given as a value
bool is_braced_list() const noexcept
{
return braced_list;
}
private:
mutable value_type owned_value = nullptr;
value_type const* value_ref = nullptr;
bool braced_list = false;
};
} // namespace detail
@@ -24426,8 +24445,8 @@ boundaries compute_boundaries(FloatType value)
//
// -e <= 60 or e >= -60 := alpha
constexpr int kAlpha = -60;
constexpr int kGamma = -32;
JSON_INLINE_VARIABLE constexpr int kAlpha = -60;
JSON_INLINE_VARIABLE constexpr int kGamma = -32;
struct cached_power // c = f * 2^e ~= 10^k
{
@@ -29444,6 +29463,18 @@ public:
bool type_deduction = true,
value_t manual_type = value_t::array)
{
#if JSON_BRACE_INIT_COPY_SEMANTICS
// a single element that is a value rather than a braced list is
// copied or moved as is, whatever its content looks like
if (type_deduction && init.size() == 1 && !init.begin()->is_braced_list())
{
*this = init.begin()->moved_or_copied();
set_parents();
assert_invariant();
return;
}
#endif
// check if each element is an array with two elements whose first
// element is a string
bool is_an_object = std::all_of(init.begin(), init.end(),
@@ -32863,12 +32894,13 @@ public:
input_format_t format = input_format_t::json,
const bool strict = true,
const bool ignore_comments = false,
const bool ignore_trailing_commas = false)
const bool ignore_trailing_commas = false,
const cbor_tag_handler_t tag_handler = cbor_tag_handler_t::error)
{
auto ia = detail::input_adapter(std::forward<InputType>(i));
return format == input_format_t::json
? parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict)
: detail::binary_reader<basic_json, decltype(ia), SAX>(std::move(ia), format).sax_parse(sax, strict);
: detail::binary_reader<basic_json, decltype(ia), SAX>(std::move(ia), format).sax_parse(sax, strict, tag_handler);
}
/// @brief generate SAX events (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -32880,12 +32912,13 @@ public:
input_format_t format = input_format_t::json,
const bool strict = true,
const bool ignore_comments = false,
const bool ignore_trailing_commas = false)
const bool ignore_trailing_commas = false,
const cbor_tag_handler_t tag_handler = cbor_tag_handler_t::error)
{
auto ia = detail::input_adapter(std::move(first), std::move(last));
return format == input_format_t::json
? parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict)
: detail::binary_reader<basic_json, decltype(ia), SAX>(std::move(ia), format).sax_parse(sax, strict);
: detail::binary_reader<basic_json, decltype(ia), SAX>(std::move(ia), format).sax_parse(sax, strict, tag_handler);
}
/// @brief generate SAX events
@@ -32913,7 +32946,8 @@ public:
input_format_t format = input_format_t::json,
const bool strict = true,
const bool ignore_comments = false,
const bool ignore_trailing_commas = false)
const bool ignore_trailing_commas = false,
const cbor_tag_handler_t tag_handler = cbor_tag_handler_t::error)
#if JSON_DELETE_DEPRECATED_FUNCTIONS
= delete;
#else
@@ -32923,7 +32957,7 @@ public:
// NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
? parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict)
// NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg)
: detail::binary_reader<basic_json, decltype(ia), SAX>(std::move(ia), format).sax_parse(sax, strict);
: detail::binary_reader<basic_json, decltype(ia), SAX>(std::move(ia), format).sax_parse(sax, strict, tag_handler);
}
#endif
#if defined(__clang__)
+14 -1
View File
@@ -3,6 +3,14 @@
Each parser of the library (JSON, BJData, BON8, BSON, CBOR, MessagePack, and UBJSON) can be fuzz tested. Currently,
[libFuzzer](https://llvm.org/docs/LibFuzzer.html) and [afl++](https://github.com/AFLplusplus/AFLplusplus) are supported.
## What the fuzzers check
Each fuzzer driver (`tests/src/fuzzer-parse_*.cpp`) parses its input twice: once with `allow_exceptions = false` and
once with exceptions. Both calls must agree. Where parsing with exceptions fails, the call without exceptions must
return a discarded value (or throw the same kind of non-parse error), and it must never throw a `parse_error`. Where
parsing succeeds, both calls must return the same value. The drivers then serialize the value, parse the result back,
and check that nothing was lost. The drivers check all of this with `assert`, so they refuse to build with `NDEBUG`.
## Corpus creation
For most effective fuzzing, a [corpus](https://llvm.org/docs/LibFuzzer.html#corpus) should be provided. A corpus is a
@@ -54,6 +62,9 @@ Then pass the corpus directory as command-line argument (assuming it is located
The fuzzer should be able to run indefinitely without crashing. In case of a crash, the tested input is dumped into
a file starting with `crash-`.
To also detect memory leaks, build with AddressSanitizer (`FUZZER_ENGINE="-fsanitize=fuzzer,address"`): libFuzzer then
runs LeakSanitizer by default (`-detect_leaks=1`). LeakSanitizer is not available with Apple Clang on macOS.
## afl++
To use afl++, you need to pass `-fsanitize=fuzzer` as `FUZZER_ENGINE`. It will be replaced by a `libAFLDriver.a` to
@@ -76,7 +87,9 @@ directory `out`.
The library is further fuzz-tested 24/7 by Google's [OSS-Fuzz project](https://github.com/google/oss-fuzz). It uses
the same `fuzzers` target as above and also relies on the `FUZZER_ENGINE` variable. See the used
[build script](https://github.com/google/oss-fuzz/blob/master/projects/json/build.sh) for more information.
[build script](https://github.com/google/oss-fuzz/blob/master/projects/json/build.sh) for more information. Its default
`address` sanitizer includes LeakSanitizer, so OSS-Fuzz and the CIFuzz workflow (`.github/workflows/cifuzz.yml`) report
memory leaks, too.
In case the build at OSS-Fuzz fails, an issue will be created automatically.
+39 -1
View File
@@ -10,7 +10,9 @@
This file implements a parser test suitable for fuzz testing. Given a byte
array data, it performs the following steps:
- j0 = from_bjdata(data, allow_exceptions = false)
- j1 = from_bjdata(data)
- assert(j0 is discarded if parsing j1 fails, and j0 == j1 otherwise)
- vec2 = to_bjdata(j1, use_size = false, use_type = false)
- vec3 = to_bjdata(j1, use_size = true, use_type = false)
- vec4 = to_bjdata(j1, use_size = true, use_type = true)
@@ -59,6 +61,13 @@ drivers.
using json = nlohmann::json;
// compares dumps rather than values, because NaN != NaN; keep writes strings
// byte for byte, so ill-formed UTF-8 that a binary reader accepts cannot throw
static bool same_value(const json& lhs, const json& rhs)
{
return lhs.dump(-1, ' ', false, json::error_handler_t::keep) == rhs.dump(-1, ' ', false, json::error_handler_t::keep);
}
// value-stable comparison for the round-trip checks below; see the note
// above on why this compares dump()s rather than the json values directly
static bool is_value_stable(const json& lhs, const json& rhs)
@@ -69,11 +78,37 @@ static bool is_value_stable(const json& lhs, const json& rhs)
// see http://llvm.org/docs/LibFuzzer.html
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
std::vector<uint8_t> const vec1(data, data + size);
// step 0: parse input without exceptions; a parse error must then be
// reported as a discarded value, never thrown
json j_noexcept;
bool noexcept_threw = false;
try
{
j_noexcept = json::from_bjdata(vec1, true, false);
}
catch (const json::parse_error&)
{
assert(false);
}
catch (const json::exception&)
{
// type and out-of-range errors are not parse errors and still throw
noexcept_threw = true;
}
// whether step 1 succeeded; if not, the catch blocks below check that
// step 0 failed, too
bool parsed = false;
try
{
// step 1: parse input
std::vector<uint8_t> const vec1(data, data + size);
json const j1 = json::from_bjdata(vec1);
parsed = true;
// without exceptions, the same input must give the same value
assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
try
{
@@ -107,14 +142,17 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
catch (const json::parse_error&)
{
// parse errors are ok, because input may be random bytes
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
catch (const json::type_error&)
{
// type errors can occur during parsing, too
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
catch (const json::out_of_range&)
{
// out of range errors may happen if provided sizes are excessive
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
// return 0 - non-zero return values are reserved for future use
+39 -1
View File
@@ -10,7 +10,9 @@
This file implements a parser test suitable for fuzz testing. Given a byte
array data, it performs the following steps:
- j0 = from_bon8(data, allow_exceptions = false)
- j1 = from_bon8(data)
- assert(j0 is discarded if parsing j1 fails, and j0 == j1 otherwise)
- vec = to_bon8(j1)
- j2 = from_bon8(vec)
- assert(to_bon8(j2) == vec)
@@ -34,6 +36,13 @@ drivers.
using json = nlohmann::json;
// compares dumps rather than values, because NaN != NaN; keep writes strings
// byte for byte, so ill-formed UTF-8 that a binary reader accepts cannot throw
static bool same_value(const json& lhs, const json& rhs)
{
return lhs.dump(-1, ' ', false, json::error_handler_t::keep) == rhs.dump(-1, ' ', false, json::error_handler_t::keep);
}
namespace
{
// the serialization of the value read from @a input, or the error message
@@ -61,11 +70,37 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
assert(read_bon8(std::vector<uint8_t>(data, data + size)) == read_bon8(stream));
}
std::vector<uint8_t> const vec1(data, data + size);
// step 0: parse input without exceptions; a parse error must then be
// reported as a discarded value, never thrown
json j_noexcept;
bool noexcept_threw = false;
try
{
j_noexcept = json::from_bon8(vec1, true, false);
}
catch (const json::parse_error&)
{
assert(false);
}
catch (const json::exception&)
{
// type and out-of-range errors are not parse errors and still throw
noexcept_threw = true;
}
// whether step 1 succeeded; if not, the catch blocks below check that
// step 0 failed, too
bool parsed = false;
try
{
// step 1: parse input
std::vector<uint8_t> const vec1(data, data + size);
json const j1 = json::from_bon8(vec1);
parsed = true;
// without exceptions, the same input must give the same value
assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
try
{
@@ -87,14 +122,17 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
catch (const json::parse_error&)
{
// parse errors are ok, because input may be random bytes
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
catch (const json::type_error&)
{
// type errors can occur during parsing, too
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
catch (const json::out_of_range&)
{
// out of range errors may happen if provided sizes are excessive
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
// return 0 - non-zero return values are reserved for future use
+39 -1
View File
@@ -10,7 +10,9 @@
This file implements a parser test suitable for fuzz testing. Given a byte
array data, it performs the following steps:
- j0 = from_bson(data, allow_exceptions = false)
- j1 = from_bson(data)
- assert(j0 is discarded if parsing j1 fails, and j0 == j1 otherwise)
- vec = to_bson(j1)
- j2 = from_bson(vec)
- assert(to_bson(j2) == vec)
@@ -29,14 +31,47 @@ drivers.
using json = nlohmann::json;
// compares dumps rather than values, because NaN != NaN; keep writes strings
// byte for byte, so ill-formed UTF-8 that a binary reader accepts cannot throw
static bool same_value(const json& lhs, const json& rhs)
{
return lhs.dump(-1, ' ', false, json::error_handler_t::keep) == rhs.dump(-1, ' ', false, json::error_handler_t::keep);
}
// see http://llvm.org/docs/LibFuzzer.html
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
std::vector<uint8_t> const vec1(data, data + size);
// step 0: parse input without exceptions; a parse error must then be
// reported as a discarded value, never thrown
json j_noexcept;
bool noexcept_threw = false;
try
{
j_noexcept = json::from_bson(vec1, true, false);
}
catch (const json::parse_error&)
{
assert(false);
}
catch (const json::exception&)
{
// type and out-of-range errors are not parse errors and still throw
noexcept_threw = true;
}
// whether step 1 succeeded; if not, the catch blocks below check that
// step 0 failed, too
bool parsed = false;
try
{
// step 1: parse input
std::vector<uint8_t> const vec1(data, data + size);
json const j1 = json::from_bson(vec1);
parsed = true;
// without exceptions, the same input must give the same value
assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
try
{
@@ -58,14 +93,17 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
catch (const json::parse_error&)
{
// parse errors are ok, because input may be random bytes
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
catch (const json::type_error&)
{
// type errors can occur during parsing, too
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
catch (const json::out_of_range&)
{
// out of range errors can occur during parsing, too
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
// return 0 - non-zero return values are reserved for future use
+39 -1
View File
@@ -10,7 +10,9 @@
This file implements a parser test suitable for fuzz testing. Given a byte
array data, it performs the following steps:
- j0 = from_cbor(data, allow_exceptions = false)
- j1 = from_cbor(data)
- assert(j0 is discarded if parsing j1 fails, and j0 == j1 otherwise)
- vec = to_cbor(j1)
- j2 = from_cbor(vec)
- assert(to_cbor(j2) == vec)
@@ -29,14 +31,47 @@ drivers.
using json = nlohmann::json;
// compares dumps rather than values, because NaN != NaN; keep writes strings
// byte for byte, so ill-formed UTF-8 that a binary reader accepts cannot throw
static bool same_value(const json& lhs, const json& rhs)
{
return lhs.dump(-1, ' ', false, json::error_handler_t::keep) == rhs.dump(-1, ' ', false, json::error_handler_t::keep);
}
// see http://llvm.org/docs/LibFuzzer.html
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
std::vector<uint8_t> const vec1(data, data + size);
// step 0: parse input without exceptions; a parse error must then be
// reported as a discarded value, never thrown
json j_noexcept;
bool noexcept_threw = false;
try
{
j_noexcept = json::from_cbor(vec1, true, false);
}
catch (const json::parse_error&)
{
assert(false);
}
catch (const json::exception&)
{
// type and out-of-range errors are not parse errors and still throw
noexcept_threw = true;
}
// whether step 1 succeeded; if not, the catch blocks below check that
// step 0 failed, too
bool parsed = false;
try
{
// step 1: parse input
std::vector<uint8_t> const vec1(data, data + size);
json const j1 = json::from_cbor(vec1);
parsed = true;
// without exceptions, the same input must give the same value
assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
try
{
@@ -58,14 +93,17 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
catch (const json::parse_error&)
{
// parse errors are ok, because input may be random bytes
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
catch (const json::type_error&)
{
// type errors can occur during parsing, too
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
catch (const json::out_of_range&)
{
// out of range errors can occur during parsing, too
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
// return 0 - non-zero return values are reserved for future use
+36
View File
@@ -10,7 +10,9 @@
This file implements a parser test suitable for fuzz testing. Given a byte
array data, it performs the following steps:
- j0 = parse(data, allow_exceptions = false)
- j1 = parse(data)
- assert(j0 is discarded if parsing j1 fails, and j0 == j1 otherwise)
- s1 = serialize(j1)
- j2 = parse(s1)
- s2 = serialize(j2)
@@ -30,13 +32,45 @@ drivers.
using json = nlohmann::json;
// compares dumps rather than values, because NaN != NaN; keep writes strings
// byte for byte, so ill-formed UTF-8 that a binary reader accepts cannot throw
static bool same_value(const json& lhs, const json& rhs)
{
return lhs.dump(-1, ' ', false, json::error_handler_t::keep) == rhs.dump(-1, ' ', false, json::error_handler_t::keep);
}
// see http://llvm.org/docs/LibFuzzer.html
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// step 0: parse input without exceptions; a parse error must then be
// reported as a discarded value, never thrown
json j_noexcept;
bool noexcept_threw = false;
try
{
j_noexcept = json::parse(data, data + size, nullptr, false);
}
catch (const json::parse_error&)
{
assert(false);
}
catch (const json::exception&)
{
// type and out-of-range errors are not parse errors and still throw
noexcept_threw = true;
}
// whether step 1 succeeded; if not, the catch blocks below check that
// step 0 failed, too
bool parsed = false;
try
{
// step 1: parse input
json const j1 = json::parse(data, data + size);
parsed = true;
// without exceptions, the same input must give the same value
assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
try
{
@@ -63,10 +97,12 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
catch (const json::parse_error&)
{
// parse errors are ok, because input may be random bytes
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
catch (const json::out_of_range&)
{
// out of range errors may happen if provided sizes are excessive
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
// return 0 - non-zero return values are reserved for future use
+39 -1
View File
@@ -10,7 +10,9 @@
This file implements a parser test suitable for fuzz testing. Given a byte
array data, it performs the following steps:
- j0 = from_msgpack(data, allow_exceptions = false)
- j1 = from_msgpack(data)
- assert(j0 is discarded if parsing j1 fails, and j0 == j1 otherwise)
- vec = to_msgpack(j1)
- j2 = from_msgpack(vec)
- assert(to_msgpack(j2) == vec)
@@ -29,14 +31,47 @@ drivers.
using json = nlohmann::json;
// compares dumps rather than values, because NaN != NaN; keep writes strings
// byte for byte, so ill-formed UTF-8 that a binary reader accepts cannot throw
static bool same_value(const json& lhs, const json& rhs)
{
return lhs.dump(-1, ' ', false, json::error_handler_t::keep) == rhs.dump(-1, ' ', false, json::error_handler_t::keep);
}
// see http://llvm.org/docs/LibFuzzer.html
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
std::vector<uint8_t> const vec1(data, data + size);
// step 0: parse input without exceptions; a parse error must then be
// reported as a discarded value, never thrown
json j_noexcept;
bool noexcept_threw = false;
try
{
j_noexcept = json::from_msgpack(vec1, true, false);
}
catch (const json::parse_error&)
{
assert(false);
}
catch (const json::exception&)
{
// type and out-of-range errors are not parse errors and still throw
noexcept_threw = true;
}
// whether step 1 succeeded; if not, the catch blocks below check that
// step 0 failed, too
bool parsed = false;
try
{
// step 1: parse input
std::vector<uint8_t> const vec1(data, data + size);
json const j1 = json::from_msgpack(vec1);
parsed = true;
// without exceptions, the same input must give the same value
assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
try
{
@@ -58,14 +93,17 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
catch (const json::parse_error&)
{
// parse errors are ok, because input may be random bytes
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
catch (const json::type_error&)
{
// type errors can occur during parsing, too
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
catch (const json::out_of_range&)
{
// out of range errors may happen if provided sizes are excessive
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
// return 0 - non-zero return values are reserved for future use
+39 -1
View File
@@ -10,7 +10,9 @@
This file implements a parser test suitable for fuzz testing. Given a byte
array data, it performs the following steps:
- j0 = from_ubjson(data, allow_exceptions = false)
- j1 = from_ubjson(data)
- assert(j0 is discarded if parsing j1 fails, and j0 == j1 otherwise)
- vec2 = to_ubjson(j1, use_size = false, use_type = false)
- vec3 = to_ubjson(j1, use_size = true, use_type = false)
- vec4 = to_ubjson(j1, use_size = true, use_type = true)
@@ -38,14 +40,47 @@ drivers.
using json = nlohmann::json;
// compares dumps rather than values, because NaN != NaN; keep writes strings
// byte for byte, so ill-formed UTF-8 that a binary reader accepts cannot throw
static bool same_value(const json& lhs, const json& rhs)
{
return lhs.dump(-1, ' ', false, json::error_handler_t::keep) == rhs.dump(-1, ' ', false, json::error_handler_t::keep);
}
// see http://llvm.org/docs/LibFuzzer.html
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
std::vector<uint8_t> const vec1(data, data + size);
// step 0: parse input without exceptions; a parse error must then be
// reported as a discarded value, never thrown
json j_noexcept;
bool noexcept_threw = false;
try
{
j_noexcept = json::from_ubjson(vec1, true, false);
}
catch (const json::parse_error&)
{
assert(false);
}
catch (const json::exception&)
{
// type and out-of-range errors are not parse errors and still throw
noexcept_threw = true;
}
// whether step 1 succeeded; if not, the catch blocks below check that
// step 0 failed, too
bool parsed = false;
try
{
// step 1: parse input
std::vector<uint8_t> const vec1(data, data + size);
json const j1 = json::from_ubjson(vec1);
parsed = true;
// without exceptions, the same input must give the same value
assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
try
{
@@ -77,14 +112,17 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
catch (const json::parse_error&)
{
// parse errors are ok, because input may be random bytes
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
catch (const json::type_error&)
{
// type errors can occur during parsing, too
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
catch (const json::out_of_range&)
{
// out of range errors may happen if provided sizes are excessive
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
}
// return 0 - non-zero return values are reserved for future use
+46
View File
@@ -4603,3 +4603,49 @@ TEST_CASE("issue #5648 - from_bjdata(ptr, len) must read len bytes, not treat pt
CHECK(json::from_bjdata(packed.data(), packed.size(), false) == j);
#endif
}
TEST_CASE("BJData large strings and binaries (chunked reader)")
{
// Strings share get_ubjson_string() -> get_string() -> get_bytes() with
// plain UBJSON. Binary values are different: only a Draft 3 optimized
// array (type marker 'B') is read back as a binary value, through
// get_binary() -> get_bytes() (see parse_ubjson_internal()'s "If BJData
// type marker is 'B'" branch); Draft 2 (the default) writes a binary
// value as a plain array of uint8_t numbers instead (see the "round trip
// of a binary value is value-stable, not byte-stable" test above), which
// never reaches get_bytes(). Both reads happen in bounded chunks
// (binary_reader.hpp, chunk_size == 4096); check lengths around and
// beyond that size, for both vector (iterator) and pointer inputs.
for (const std::size_t len :
{
std::size_t{0}, std::size_t{1}, std::size_t{4095}, std::size_t{4096},
std::size_t{4097}, std::size_t{8192}, std::size_t{100000}
})
{
CAPTURE(len)
// string
const json j_string = std::string(len, 'x');
const std::vector<std::uint8_t> v_string = json::to_bjdata(j_string);
CHECK(json::from_bjdata(v_string) == j_string);
// pointer input exercises the std::memcpy fast path
CHECK(json::from_bjdata(reinterpret_cast<const char*>(v_string.data()),
reinterpret_cast<const char*>(v_string.data()) + v_string.size()) == j_string);
// binary, forced into the Draft 3 optimized ('B' marker) encoding
const json j_binary = json::binary(std::vector<std::uint8_t>(len, 0xCD));
const std::vector<std::uint8_t> v_binary = json::to_bjdata(j_binary, true, true, json::bjdata_version_t::draft3);
CHECK(json::from_bjdata(v_binary) == j_binary);
CHECK(json::from_bjdata(reinterpret_cast<const char*>(v_binary.data()),
reinterpret_cast<const char*>(v_binary.data()) + v_binary.size()) == j_binary);
// a truncated payload must still be reported as an error
if (len > 16)
{
std::vector<std::uint8_t> truncated = v_string;
truncated.resize(truncated.size() - 8);
json _;
CHECK_THROWS_AS(_ = json::from_bjdata(truncated), json::parse_error);
}
}
}
@@ -72,6 +72,28 @@ TEST_CASE("JSON_BRACE_INIT_COPY_SEMANTICS")
CHECK(j7 == json::array({1, 2}));
}
SECTION("single-element brace initialization copies a pair-shaped array value (#5662)")
{
// a JSON value that happens to be a 2-element array whose first
// element is a string must still be copied, not turned into an
// object; only a braced list written in the source, such as the
// inner {"key", "value"} of {{"key", "value"}}, describes an object
json const pair_shaped = json::array({"key", 42});
json const j1{pair_shaped};
CHECK(j1.is_array());
CHECK(j1 == pair_shaped);
json const j2 = {pair_shaped};
CHECK(j2.is_array());
CHECK(j2 == pair_shaped);
// the same holds for an rvalue of the same shape
json const j3{json::array({"key", 42})};
CHECK(j3.is_array());
CHECK(j3 == pair_shaped);
}
SECTION("what the macro does not change")
{
// lists with more than one element are unaffected
+44
View File
@@ -1993,3 +1993,47 @@ TEST_CASE("Invalid document size handling")
CHECK(json::from_bson(v, true, false).is_discarded());
}
}
TEST_CASE("BSON large strings and binaries (chunked reader)")
{
// get_bson_string()/get_bson_binary() both read through get_string()/
// get_binary(), which read in bounded chunks (binary_reader.hpp,
// chunk_size == 4096); make sure roundtripping is correct for lengths
// around and beyond that chunk size, for both vector (iterator) and
// pointer inputs. BSON only accepts an object at the top level, so the
// string/binary value is wrapped in one.
for (const std::size_t len :
{
std::size_t{0}, std::size_t{1}, std::size_t{4095}, std::size_t{4096},
std::size_t{4097}, std::size_t{8192}, std::size_t{100000}
})
{
CAPTURE(len)
// string
const json j_string = {{"k", std::string(len, 'x')}};
const std::vector<std::uint8_t> v_string = json::to_bson(j_string);
CHECK(json::from_bson(v_string) == j_string);
// pointer input exercises the std::memcpy fast path
CHECK(json::from_bson(reinterpret_cast<const char*>(v_string.data()),
reinterpret_cast<const char*>(v_string.data()) + v_string.size()) == j_string);
// binary (BSON binary values always carry a subtype, so give one
// explicitly; otherwise from_bson() would round-trip to subtype 0
// rather than back to the original "no subtype" value)
const json j_binary = {{"k", json::binary(std::vector<std::uint8_t>(len, 0xCD), std::uint8_t{0})}};
const std::vector<std::uint8_t> v_binary = json::to_bson(j_binary);
CHECK(json::from_bson(v_binary) == j_binary);
CHECK(json::from_bson(reinterpret_cast<const char*>(v_binary.data()),
reinterpret_cast<const char*>(v_binary.data()) + v_binary.size()) == j_binary);
// a truncated payload must still be reported as an error
if (len > 16)
{
std::vector<std::uint8_t> truncated = v_string;
truncated.resize(truncated.size() - 8);
json _;
CHECK_THROWS_AS(_ = json::from_bson(truncated), json::parse_error);
}
}
}
+23 -16
View File
@@ -1699,7 +1699,7 @@ TEST_CASE("CBOR")
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0xA1, 0x61, 0X61})), "[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing CBOR value: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0xBF, 0x61, 0X61})), "[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing CBOR value: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0x5F})), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing CBOR binary: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0x5F, 0x00})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR binary: expected length specification (0x40-0x5B) or indefinite binary array type (0x5F); last byte: 0x00", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0x5F, 0x00})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR binary: expected length specification (0x40-0x5B); last byte: 0x00", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0x41})), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing CBOR binary: unexpected end of input", json::parse_error&);
CHECK(json::from_cbor(std::vector<uint8_t>({0x18}), true, false).is_discarded());
@@ -2305,22 +2305,21 @@ TEST_CASE("CBOR indefinite-length strings do not recurse per chunk")
{
// Reading an indefinite-length string or byte array used to call itself
// once per chunk, so a payload of repeated 0x7F (or 0x5F) bytes exhausted
// the call stack before any of the input was rejected. The open levels are
// counted now, and the levels below prove the reader still reads the same
// values and reports the same errors at the same byte offsets.
// the call stack before any of the input was rejected. Nested indefinite
// chunks are now rejected at the second byte, without recursing.
json _;
SECTION("many open levels are reported, not crashed on")
SECTION("nested levels are rejected, not crashed on")
{
const std::vector<uint8_t> input(200000, 0x7F);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(input), "[json.exception.parse_error.110] parse error at byte 200001: syntax error while parsing CBOR string: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(input), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: indefinite-length string is not allowed inside indefinite-length string; last byte: 0x7F", json::parse_error&);
CHECK(json::from_cbor(input, true, false).is_discarded());
}
SECTION("many open levels are reported, not crashed on (binary)")
SECTION("nested levels are rejected, not crashed on (binary)")
{
const std::vector<uint8_t> input(200000, 0x5F);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(input), "[json.exception.parse_error.110] parse error at byte 200001: syntax error while parsing CBOR binary: unexpected end of input", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(input), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR binary: indefinite-length binary array is not allowed inside indefinite-length binary array; last byte: 0x5F", json::parse_error&);
CHECK(json::from_cbor(input, true, false).is_discarded());
}
@@ -2328,22 +2327,22 @@ TEST_CASE("CBOR indefinite-length strings do not recurse per chunk")
{
CHECK(json::from_cbor(std::vector<uint8_t>({0x7F, 0xFF})) == json(""));
CHECK(json::from_cbor(std::vector<uint8_t>({0x7F, 0x61, 0x61, 0xFF})) == json("a"));
// nested indefinite-length strings are concatenated across levels
CHECK(json::from_cbor(std::vector<uint8_t>({0x7F, 0x7F, 0x61, 0x61, 0xFF, 0x61, 0x62, 0xFF})) == json("ab"));
CHECK(json::from_cbor(std::vector<uint8_t>({0x7F, 0x7F, 0x7F, 0x61, 0x7A, 0xFF, 0xFF, 0xFF})) == json("z"));
// empty and nonempty definite-length chunks concatenate in order
CHECK(json::from_cbor(std::vector<uint8_t>({0x7F, 0x61, 'a', 0x60, 0x61, 'b', 0x61, 'c', 0xFF})) == json("abc"));
CHECK(json::from_cbor(std::vector<uint8_t>({0xA1, 0x7F, 0x61, 0x61, 0xFF, 0x01})) == json({{"a", 1}}));
}
SECTION("chunks are still concatenated (binary)")
{
CHECK(json::from_cbor(std::vector<uint8_t>({0x5F, 0x41, 0x61, 0xFF})) == json::binary({0x61}));
CHECK(json::from_cbor(std::vector<uint8_t>({0x5F, 0x5F, 0x41, 0x61, 0xFF, 0x41, 0x62, 0xFF})) == json::binary({0x61, 0x62}));
CHECK(json::from_cbor(std::vector<uint8_t>({0x5F, 0xFF})) == json::binary({}));
CHECK(json::from_cbor(std::vector<uint8_t>({0x5F, 0x41, 0x61, 0x40, 0x41, 0x62, 0x41, 0x63, 0xFF})) == json::binary({0x61, 0x62, 0x63}));
}
SECTION("a chunk that is not a string is still rejected")
{
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0x7F, 0x7F, 0x00})), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing CBOR string: expected length specification (0x60-0x7B) or indefinite string type (0x7F); last byte: 0x00", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0x5F, 0x5F, 0x00})), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing CBOR binary: expected length specification (0x40-0x5B) or indefinite binary array type (0x5F); last byte: 0x00", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0x7F, 0x00})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: expected length specification (0x60-0x7B); last byte: 0x00", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<uint8_t>({0x5F, 0x00})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR binary: expected length specification (0x40-0x5B); last byte: 0x00", json::parse_error&);
}
SECTION("a break marker outside an indefinite-length string is not a string")
@@ -2896,9 +2895,17 @@ TEST_CASE("examples from RFC 8949 Appendix A")
{
const auto packed = utils::read_binary_file(TEST_DATA_DIRECTORY "/binary_data/cbor_binary.cbor");
json j;
CHECK_NOTHROW(j = json::from_cbor(packed));
// the fixture's tail contains nested indefinite-length byte strings.
CHECK_THROWS_WITH_AS(j = json::from_cbor(packed), "[json.exception.parse_error.113] parse error at byte 513: syntax error while parsing CBOR binary: indefinite-length binary array is not allowed inside indefinite-length binary array; last byte: 0x5F", json::parse_error&);
const auto expected = utils::read_binary_file(TEST_DATA_DIRECTORY "/binary_data/cbor_binary.out");
// keep the byte-for-byte decoding check for its valid prefix: the first
// 512 encoded bytes contain 468 payload bytes in definite-length chunks.
auto valid_prefix = packed;
valid_prefix.resize(512);
valid_prefix.push_back(0xFF);
auto expected = utils::read_binary_file(TEST_DATA_DIRECTORY "/binary_data/cbor_binary.out");
expected.resize(468);
CHECK_NOTHROW(j = json::from_cbor(valid_prefix));
CHECK(j == json::binary(expected));
// 0xd8
+40
View File
@@ -2482,3 +2482,43 @@ TEST_CASE("MessagePack numbers use the active union member (see #5644)")
CHECK(json::from_msgpack(result) == j);
}
}
TEST_CASE("MessagePack large strings and binaries (chunked reader)")
{
// get_msgpack_string()/get_msgpack_binary() both read through get_binary(),
// which reads in bounded chunks (binary_reader.hpp, chunk_size == 4096);
// make sure roundtripping is correct for lengths around and beyond that
// chunk size, for both vector (iterator) and pointer inputs.
for (const std::size_t len :
{
std::size_t{0}, std::size_t{1}, std::size_t{4095}, std::size_t{4096},
std::size_t{4097}, std::size_t{8192}, std::size_t{100000}
})
{
CAPTURE(len)
// string
const json j_string = std::string(len, 'x');
const std::vector<std::uint8_t> v_string = json::to_msgpack(j_string);
CHECK(json::from_msgpack(v_string) == j_string);
// pointer input exercises the std::memcpy fast path
CHECK(json::from_msgpack(reinterpret_cast<const char*>(v_string.data()),
reinterpret_cast<const char*>(v_string.data()) + v_string.size()) == j_string);
// binary
const json j_binary = json::binary(std::vector<std::uint8_t>(len, 0xCD));
const std::vector<std::uint8_t> v_binary = json::to_msgpack(j_binary);
CHECK(json::from_msgpack(v_binary) == j_binary);
CHECK(json::from_msgpack(reinterpret_cast<const char*>(v_binary.data()),
reinterpret_cast<const char*>(v_binary.data()) + v_binary.size()) == j_binary);
// a truncated payload must still be reported as an error
if (len > 16)
{
std::vector<std::uint8_t> truncated = v_string;
truncated.resize(truncated.size() - 8);
json _;
CHECK_THROWS_AS(_ = json::from_msgpack(truncated), json::parse_error);
}
}
}
+51 -3
View File
@@ -533,7 +533,7 @@ TEST_CASE("regression tests 3")
}
#endif
#if JSON_HAS_RANGES && !defined(__MINGW32__)
#if JSON_HAS_RANGE_VIEW_CONVERSION
SECTION("issue #4916 - constructing array from C++20 ranges view does not work")
{
std::vector<int> nums{1, 2, 37, 42, 21};
@@ -548,7 +548,7 @@ TEST_CASE("regression tests 3")
#endif
// owning_view is not available in libstdc++ < 12
#if JSON_HAS_RANGES && !defined(__MINGW32__) && !(defined(__GLIBCXX__) && _GLIBCXX_RELEASE < 12)
#if JSON_HAS_RANGE_VIEW_CONVERSION && !(defined(__GLIBCXX__) && _GLIBCXX_RELEASE < 12)
SECTION("issue #4916 - constructing array from prvalue C++20 ranges view (owning_view)")
{
json const j(std::vector<int> {1, 2, 37, 42, 21} | std::views::filter([](int i)
@@ -560,7 +560,7 @@ TEST_CASE("regression tests 3")
}
#endif
#if JSON_HAS_RANGES && !defined(__MINGW32__)
#if JSON_HAS_RANGE_VIEW_CONVERSION
SECTION("issue #4916 - constructing array from C++20 transform view (prvalue elements)")
{
std::vector<int> nums{1, 2, 3};
@@ -862,6 +862,46 @@ TEST_CASE("issue #5338 - truncated CBOR tagged binary subtype is rejected")
}
}
TEST_CASE("issue #5676 - SAX parsing of CBOR tags")
{
const json expected = json::binary({1, 2, 3}, 42);
const auto cbor = json::to_cbor(expected);
nlohmann::detail::json_sax_acceptor<json> acceptor;
CHECK_FALSE(json::sax_parse(cbor, &acceptor, json::input_format_t::cbor));
CHECK_FALSE(json::sax_parse(cbor, &acceptor, json::input_format_t::cbor,
true, false, false, json::cbor_tag_handler_t::error));
CHECK(json::sax_parse(cbor, &acceptor, json::input_format_t::cbor,
true, false, false, json::cbor_tag_handler_t::ignore));
json parsed;
nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::string_input_adapter_type> sax(parsed);
CHECK(json::sax_parse(cbor, &sax, json::input_format_t::cbor,
true, false, false, json::cbor_tag_handler_t::store));
CHECK(parsed == expected);
json iterator_parsed;
nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::string_input_adapter_type> iterator_sax(iterator_parsed);
CHECK(json::sax_parse(cbor.begin(), cbor.end(), &iterator_sax, json::input_format_t::cbor,
true, false, false, json::cbor_tag_handler_t::store));
CHECK(iterator_parsed == expected);
json span_parsed;
nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::string_input_adapter_type> span_sax(span_parsed);
CHECK(json::sax_parse(nlohmann::detail::span_input_adapter(cbor.data(), cbor.size()), &span_sax,
json::input_format_t::cbor, true, false, false, json::cbor_tag_handler_t::store));
CHECK(span_parsed == expected);
const std::string text = "null";
CHECK(json::sax_parse(text, &acceptor, json::input_format_t::json,
true, false, false, json::cbor_tag_handler_t::store));
CHECK(json::sax_parse(text.begin(), text.end(), &acceptor, json::input_format_t::json,
true, false, false, json::cbor_tag_handler_t::store));
CHECK(json::sax_parse(nlohmann::detail::span_input_adapter(text.data(), text.size()), &acceptor,
json::input_format_t::json, true, false, false, json::cbor_tag_handler_t::store));
}
TEST_CASE("issue #5402 - update(merge_objects=true) overwrites a primitive with an object")
{
json t = {{"k", 1}};
@@ -920,4 +960,12 @@ TEST_CASE("regression test #5476 - array type without reserve()")
}
}
TEST_CASE("issue #5317 - nested indefinite-length CBOR string chunks are rejected")
{
json _;
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<std::uint8_t>({0x7F, 0x7F, 0x61, 0x61, 0xFF, 0xFF})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR string: indefinite-length string is not allowed inside indefinite-length string; last byte: 0x7F", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<std::uint8_t>({0x5F, 0x5F, 0x41, 0x61, 0xFF, 0xFF})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR binary: indefinite-length binary array is not allowed inside indefinite-length binary array; last byte: 0x5F", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<std::uint8_t>({0xA1, 0x7F, 0x7F, 0xFF, 0xFF, 0x01})), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing CBOR string: indefinite-length string is not allowed inside indefinite-length string; last byte: 0x7F", json::parse_error&);
}
DOCTEST_CLANG_SUPPRESS_WARNING_POP
+175
View File
@@ -809,3 +809,178 @@ TEST_CASE("serializer buffers are flushed mid-string and mid-binary")
CHECK(j.dump(2) == "{\n \"bytes\": [" + expected_pretty_bytes + "],\n \"subtype\": null\n}");
}
}
TEST_CASE("serialization boundary values for the write buffer")
{
// write_buffer is a std::array<char, 1024> (write_buffer_size). put_string()
// guards it with two checks, and each must be exercised exactly on and one
// past its own boundary: a heap overflow in a different manual buffer path
// (the dump(1100) indent buffer) once survived 100% line coverage because
// every test that touched it only ever grew the buffer by a single step,
// never landing on the exact edge of the comparison that protects it.
//
// - straight-through: put_string() bypasses write_buffer entirely and
// writes directly to the output adapter once `length >= write_buffer.size()`.
// - flush-then-copy: otherwise, if `write_buffer_pos + length > write_buffer.size()`,
// put_string() flushes what is pending and then memcpy's the new run into
// the freshly emptied buffer.
SECTION("top-level string exercises the straight-through guard (length >= 1024)")
{
// dump() of a bare string writes the opening quote with put_char()
// (write_buffer_pos: 0 -> 1), then the body with put_string(). With
// write_buffer_pos == 1, `1 + length > 1024` and `length >= 1024` flip
// together at length 1024, so 1023/1024/1025 cover "just under",
// "exactly at" and "just over" the guard in one move: 1023 is copied
// into the buffer (filling it exactly), 1024 and 1025 bypass it.
for (const std::size_t len :
{
std::size_t{1023}, std::size_t{1024}, std::size_t{1025}
})
{
CAPTURE(len)
const std::string body(len, 'a');
const json j = body;
const std::string expected = '"' + body + '"';
CHECK(j.dump() == expected);
std::ostringstream o;
o << j;
CHECK(o.str() == expected);
}
}
SECTION("string nested in an array exercises the flush-then-copy guard")
{
// json::array({body}) writes '[' then '"' before the body, so
// write_buffer_pos == 2 when put_string() is entered for it. The
// body's last byte then lands at logical offset 2 + len: len == 1022
// lands exactly on offset 1024 (2 + 1022 == write_buffer.size(), so the
// strict "> " guard does not fire and the body fits snugly), while
// len == 1023 lands one past it at offset 1025 (2 + 1023 > 1024),
// which must flush what's pending before copying the body in.
for (const std::size_t len :
{
std::size_t{1022}, std::size_t{1023}
})
{
CAPTURE(len)
const std::string body(len, 'a');
const json j = json::array({body});
const std::string expected = "[\"" + body + "\"]";
CHECK(j.dump() == expected);
std::ostringstream o;
o << j;
CHECK(o.str() == expected);
CHECK(json::parse(j.dump()) == j);
}
}
}
TEST_CASE("serialization boundary values for the string buffer")
{
// string_buffer is a std::array<char, 512>. dump_escaped_impl() flushes it
// mid-string once fewer than 13 bytes remain (`string_buffer.size() - bytes
// < 13`), 13 being one more than the most a single code point can ever
// write at once (a surrogate pair: two back-to-back "\uXXXX" escapes, 12
// bytes). Every write into string_buffer that this check protects happens
// in steps of 2 (a simple "\\x" escape) or 6 (one "\uXXXX" unit), so
// `bytes` only ever takes even values at the point the check runs - the
// tightest values actually reachable are therefore 498 (512 - 498 == 14,
// one simple escape away from the threshold) and 500 (512 - 500 == 12,
// where the flush fires immediately and resets bytes to 0).
SECTION("a run of 2-byte escapes lands bytes on, and one step past, the flush threshold")
{
for (const int count :
{
249, 250, 251
})
{
CAPTURE(count)
const json j = std::string(static_cast<std::size_t>(count), '\n');
std::string expected = "\"";
for (int i = 0; i < count; ++i)
{
expected += "\\n";
}
expected += '"';
CHECK(j.dump() == expected);
}
}
SECTION("an ASCII prefix leaves the tightest reachable margin before a 12-byte surrogate pair")
{
// U+1F600 (the "\xF0\x9F\x98\x80" UTF-8 bytes) is dumped under
// ensure_ascii as the 12-byte surrogate pair "\ud83d\ude00"; that
// write happens in a single step with no intermediate flush check, so
// it is the write most exposed by an off-by-one in the "< 13" guard.
// A prefix of 249 newlines leaves exactly 14 bytes of headroom
// (512 - 498), the smallest margin the guard ever actually allows
// into a new code point; 250 newlines instead trigger the guard's own
// flush first, so the emoji starts from a freshly emptied (512-byte)
// buffer, and 251 repeats that with one more escape already past the
// reset. Together they cover the margin the guard allows landing on,
// one step before, and one step after - all must still produce the
// identical, correct escapes.
for (const int prefix_count :
{
249, 250, 251
})
{
CAPTURE(prefix_count)
const std::string prefix(static_cast<std::size_t>(prefix_count), '\n');
const std::string emoji = "\xF0\x9F\x98\x80";
const json j = prefix + emoji;
std::string expected_prefix;
for (int i = 0; i < prefix_count; ++i)
{
expected_prefix += "\\n";
}
// newline escaping does not depend on ensure_ascii: only the
// emoji differs (raw UTF-8 bytes vs. a \u-escaped surrogate pair)
CHECK(j.dump(-1, ' ', false) == '"' + expected_prefix + emoji + '"');
CHECK(j.dump(-1, ' ', true) == '"' + expected_prefix + "\\ud83d\\ude00\"");
CHECK(json::parse(j.dump(-1, ' ', true)) == j);
CHECK(json::parse(j.dump(-1, ' ', false)) == j);
}
}
SECTION("SWAR bulk-copy stride: k plain bytes followed by a byte handled individually")
{
// string_bulk_run()/find_ascii_copyable_run() (string_scan.hpp) scan 8
// bytes at a time and fall back to a byte-at-a-time tail scan for
// what is left over. k from 0 to 17 spans zero, one and two full
// 8-byte strides plus a 1-byte tail, so every possible stopping point
// within and right after the SIMD stride is covered.
for (std::size_t k = 0; k <= 17; ++k)
{
CAPTURE(k)
const std::string prefix(k, 'a');
// (a) the run is stopped by a quote that must itself be escaped
{
const json j = prefix + "\"";
CHECK(j.dump() == '"' + prefix + "\\\"" + '"');
}
// (b) the run is stopped by a control character
{
const json j = prefix + "\x01";
CHECK(j.dump() == '"' + prefix + "\\u0001" + '"');
}
// (c) the run is stopped by a non-ASCII byte under ensure_ascii
{
const json j = prefix + "\xC3\xA9"; // prefix + 'é'
CHECK(j.dump(-1, ' ', true) == '"' + prefix + "\\u00e9" + '"');
}
}
}
}
+40
View File
@@ -3308,3 +3308,43 @@ TEST_CASE("UBJSON and BJData integer markers at every range edge")
}
}
}
TEST_CASE("UBJSON large strings (chunked reader)")
{
// get_ubjson_string() reads through get_string(), which reads in bounded
// chunks (binary_reader.hpp, chunk_size == 4096); make sure roundtripping
// is correct for lengths around and beyond that chunk size, for both
// vector (iterator) and pointer inputs.
//
// A binary value is not included here: plain UBJSON (unlike BJData, see
// the "BJData large strings and binaries" test) has no reader-side binary
// type, so even the optimized uint8_t-array encoding of a binary value is
// read back element-by-element as a JSON array of numbers rather than
// through get_binary() - it never reaches the chunked path this test is
// about (see the "roundtrip only works to an array of numbers" case
// above).
for (const std::size_t len :
{
std::size_t{0}, std::size_t{1}, std::size_t{4095}, std::size_t{4096},
std::size_t{4097}, std::size_t{8192}, std::size_t{100000}
})
{
CAPTURE(len)
const json j_string = std::string(len, 'x');
const std::vector<std::uint8_t> v_string = json::to_ubjson(j_string);
CHECK(json::from_ubjson(v_string) == j_string);
// pointer input exercises the std::memcpy fast path
CHECK(json::from_ubjson(reinterpret_cast<const char*>(v_string.data()),
reinterpret_cast<const char*>(v_string.data()) + v_string.size()) == j_string);
// a truncated payload must still be reported as an error
if (len > 16)
{
std::vector<std::uint8_t> truncated = v_string;
truncated.resize(truncated.size() - 8);
json _;
CHECK_THROWS_AS(_ = json::from_ubjson(truncated), json::parse_error);
}
}
}