Compare commits

..
Author SHA1 Message Date
Niels Lohmann eed5792dbd json_view: no per-object allocations in ordered_json comparisons
For an object type that keeps the order of its members (ordered_json),
resolve_indices() allocated a vector per compared object, and
resolve_members() two. They now use one working vector owned by equal(),
whose capacity is reused, and resolve_indices() needs none when the keys
are unique. resolve_members() sorts small objects by insertion, as the
other paths do.

Instructions per comparison (callgrind, GCC 13 and clang 18, x86-64):
twitter, ordered_json view == ordered_json: 0.939 / 0.941; 20000 small
objects with a duplicate key: 0.741-0.768; all other rows (json,
citm_catalog, canada, view == view) within 0.988-1.003.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-25 21:56:14 +01:00
Niels Lohmann d4faace387 Merge develop into perf/view-compare
Develop fixed the VS 2015 build of the DOM parser itself (#5916): its
json_sax.hpp is kept, so this branch no longer changes the file.
single_include is a fresh amalgamation.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-25 12:13:26 +01:00
Niels Lohmann 165f91ed3c Build the DOM parser with Visual Studio 2015 again
Since #5882, the three VS 2015 jobs of AppVeyor fail on develop:
- the detection of keys that can be indexed instantiated std::hash for
  every string type, which VS 2015 rejects with a static_assert for a type
  it does not hash (alt_string): std::hash is now looked at only for
  objects in a vector
- the frame stack copied its frames with std::copy to a raw pointer, which
  VS 2015 reports as unsafe (C4996, an error with /WX): a loop now

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-25 12:04:43 +01:00
Niels Lohmann 7833865dbd Merge develop into perf/view-compare
single_include is a fresh amalgamation.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-25 12:00:01 +01:00
Niels Lohmann 9707519b24 Move the json_view comparison tests into their own file
With the tests this branch adds, the object file of unit-json_view_dump.cpp
got too large for the MinGW x64 assembler ("file too big"). The test case
"json_view comparison" is now unit-json_view_compare.cpp, unchanged and
also built with C++17.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-24 20:03:58 +02:00
Niels Lohmann 2d040c89d0 Merge develop into perf/view-compare
Develop silenced the same Infer false positive in materialize() (#5905):
its version is kept. single_include is a fresh amalgamation.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-24 17:14:02 +02:00
Niels Lohmann 182b31f902 Mark Infer's null dereference in materialize() as a false positive
Infer (ci_infer, also red on develop since #5882) reports a null
dereference in json_sax_dom_parser::null() called from materialize(). The
member pointer the parser stores a value of an object in is set by the
key() before it, so it is not null there.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-24 15:40:31 +02:00
Niels Lohmann 6775994761 Merge develop into perf/view-compare
Resolves the conflict in unit-json_view_image.cpp, where develop (#5889)
and this branch each added a section to "json_view images: errors": both
are kept. single_include is a fresh amalgamation.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-24 07:48:54 +02:00
Niels Lohmann 70d1baa57a Remove a nested duplicate guard from the json_view image test
clang-tidy reported the inner #if !defined(JSON_NOEXCEPTION) as
redundant (the test case is in the same #if) and a find() != npos, which
gets the NOLINT the other tests use.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 18:53:34 +02:00
Niels Lohmann 536672e927 Dispatch on the compile-time conditions of the json_view comparison
MSVC (/W4 /WX) reported C4127 (conditional expression is constant) for
the template parameter EA and the object type's ordering in an if
condition; both are now tag-dispatched.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 09:44:39 +02:00
Niels Lohmann 55cdc93ff8 Keep the byte comparison helpers in compare.hpp
Address review comments: bytes_equal() and bytes_less() move from node.hpp
to compare.hpp, their only user; key_less() no longer splits the comment of
resolve_indices() from its function; and the loop counters of bytes_equal()
and all_digits() are scoped to their loops.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 06:47:22 +02:00
Niels Lohmann db5bd7fbcc Fix the compiler and lint failures of the json_view comparison
- all_digits(): the lambda declares its constant itself; MSVC does not
  let a lambda without captures read a constexpr local of the function
- equal(): the frame's "in document order" index is a namespace-scope
  constant (document_order); the local constexpr npos in the default
  member initializers of the local struct crashed GCC 9 and MinGW GCC
  and was not found by MSVC
- keys_differ_early() keeps the first keys in std::arrays (cpplint:
  variable-length arrays; clang-tidy: C arrays), and the basic_json
  side's first_keys() is not noexcept (key() may throw)
- the numbers of one type are compared in an if chain instead of a
  nested conditional operator
- the dump test avoids C arrays and temporary string concatenations

View comparison instructions are unchanged (Apple clang, twitter, citm,
canada: -0.1 % to +0.1 %).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-22 20:13:22 +02:00
Niels Lohmann 89bd5d63e1 Rename names that Flawfinder takes for the file function open()
The comparison cursors' open() is now enter(), and a test's open and
close strings are head and tail.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-22 13:27:54 +02:00
Niels Lohmann 3fb7d9c710 Check the float tokens of a json_document image without a sort and a scan of every character
load(image_check::full) checked every float token of an image by sorting
the token ranges, scanning the token character by character, and comparing
the digit layout with the node's. Now:

- the ranges are sorted only if they are not sorted already (save() writes
  them in the order of the nodes, so they are);
- a token of the shape [-]digits.digits whose digit counts are the ones the
  node records (nearly all of them) is accepted by checking its digits eight
  at a time, without the general scan;
- the general scan skips runs of digits eight at a time.

Damaged tokens are still rejected as before: the check accepts exactly the
tokens that the general scan accepts.

Instructions of load(image_check::full) (develop -> now):

  macOS arm64        canada 44.6M -> 30.9M (-31 %), jeopardy 11.3M -> 10.7M,
                     citm 10.4M -> 10.0M, twitter 5.06M -> 4.94M
  Linux GCC 13       canada -50 %, jeopardy -43 %, twitter -31 %, citm -28 %
  Linux Clang 18     canada -42 %, jeopardy -48 %, twitter -35 %, citm -32 %

image_check::bounds is unchanged (within 0.4 %).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-22 09:30:44 +02:00
Niels Lohmann fa8134ae85 Compare json_view values without a vector per container and a basic_json per number
operator== of views (view/view, view/json, and editable documents) built two
vectors of members for every object it entered, sorted them with stable_sort
for every object, and converted each number to a basic_json value. It now:

- keeps the members of all open objects in one vector (reserved with
  geometric growth, so that nested objects do not reallocate), and orders
  them by insertion sort for up to 16 members, or not at all when the keys
  ascend already;
- compares numbers of one type directly, and builds a basic_json only for
  numbers of different types;
- compares short texts (keys, numbers, most strings) with two overlapping
  loads (bytes_equal(), in node.hpp);
- first walks both documents in lockstep, in the order of their text, which
  finds most pairs of equal values without the work of the comparison. The
  walk only enters an object if the comparison would visit its members in the
  same order (the object type keeps the order, the keys ascend, or all values
  are scalars), so a difference is never found later than before, and it
  records the containers it found different, so that the comparison does not
  walk them again;
- rejects values of different types and arrays of different sizes before any
  of this, and, for an object type that keeps the order, objects whose first
  (up to four) members have different keys, which ends a comparison that
  differs near the start without preparing the members.

Results are unchanged: numbers compare by value across types, duplicate keys
resolve as parse() resolves them, and objects compare in document order for
ordered_json and by key otherwise.

Instructions of one comparison of a document with an equal copy, macOS arm64
(develop -> now), json_document / ordered_json_document / editable:

  twitter    16.1M -> 7.0M   17.7M -> 1.5M    16.6M -> 7.3M
  citm       45.0M -> 3.4M   47.0M -> 3.2M    45.7M -> 4.3M
  canada    111.4M -> 13.7M  111.4M -> 13.7M  113.3M -> 15.0M
  jeopardy   26.2M -> 3.7M   27.3M -> 3.7M    26.6M -> 12.1M

and of a view with a basic_json value:

  twitter    11.4M -> 5.9M   11.2M -> 7.3M    11.6M -> 5.9M
  citm       33.9M -> 9.7M   32.6M -> 9.1M    34.6M -> 9.8M
  canada     92.9M -> 32.1M  92.9M -> 32.1M   93.9M -> 32.8M

and Linux instruction counts (GCC 13 / Clang 18) of the same comparisons:

  document with an equal copy   json_document     ordered_json_document   editable
    twitter                     -49 % / -52 %     -89 % / -89 %           -48 % / -50 %
    citm                        -90 % / -89 %     -91 % / -90 %           -88 % / -87 %
    jeopardy (4,000 questions)  -85 % / -85 %     -86 % / -86 %           -46 % / -49 %
  view with a basic_json value
    twitter                     -41 % / -45 %     -28 % / -35 %           -45 % / -44 %
    citm                        -63 % / -65 %     -63 % / -66 %           -65 % / -64 %
    jeopardy                    -39 % / -43 %     -14 % / -24 %           -44 % / -43 %

A sweep that places one difference at about 2,100 positions of eight
documents (a token at an even spacing, the first and last tokens, the first,
middle and last member of the root removed, one appended, the document
wrapped) in five kinds of comparison (view/view of read-only and of editable
documents, view/json, json_document, editable/view) finds the same results as
before and no placement slower on the Mac (Clang); on Linux no placement is
slower than 3 % (Clang and GCC).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-22 09:30:42 +02:00
25 changed files with 2935 additions and 1304 deletions

No files matched your search

+2 -3
View File
@@ -90,9 +90,8 @@ type `#!cpp binary_t*` must be dereferenced.
- MessagePack - MessagePack
- If a subtype is given and the binary array contains exactly 1, 2, 4, 8, or 16 elements, the fixext family (fixext1, - If a subtype is given and the binary array contains exactly 1, 2, 4, 8, or 16 elements, the fixext family (fixext1,
fixext2, fixext4, fixext8, fixext16) is used. For other sizes, the ext family (ext8, ext16, ext32) is used. The fixext2, fixext4, fixext8) is used. For other sizes, the ext family (ext8, ext16, ext32) is used. The subtype is
subtype is then written as the signed 8-bit ext type: subtypes 128 to 255 map to the negative (reserved) ext then added as a signed 8-bit integer.
types, and on reading the ext type is exposed as an unsigned subtype from 0 to 255.
- If no subtype is given, the bin family (bin8, bin16, bin32) is used. - If no subtype is given, the bin family (bin8, bin16, bin32) is used.
- BSON - BSON
@@ -15,13 +15,11 @@ error
: report a parse error in case of a tag (the `from_cbor` overloads throw a `parse_error` exception by default) : report a parse error in case of a tag (the `from_cbor` overloads throw a `parse_error` exception by default)
ignore ignore
: ignore tags (also in front of a text string map key) : ignore tags
store store
: store tagged byte strings (for bytes 0xd8..0xdb) as binary values with the tag as subtype; other tagged values are : store tagged byte strings (for bytes 0xd8..0xdb) as binary values with the tag as subtype; other tagged values are
read as if the tag were ignored. If several tags precede a byte string, only the tag directly in front of it is read as if the tag were ignored. If several tags precede a byte string, only the innermost one is stored.
stored, and only when it has the form 0xd8..0xdb (a tag 0..23 written as 0xc0..0xd7 is skipped). Tags in front of a
text string map key are skipped.
## Examples ## Examples
+4 -12
View File
@@ -32,10 +32,8 @@ The exact mapping and its limitations are described on a [dedicated page](../../
- an `std::istream` object - an `std::istream` object
- a `FILE` pointer - a `FILE` pointer
- a C-style array of characters (its terminating NUL is read as trailing data: `parse_error.110` unless `strict` is - a C-style array of characters
`false`) - a pointer to a null-terminated string of single byte characters
- a pointer to a null-terminated string of single byte characters (read up to the first 0x00 byte, which is a valid
CBOR byte)
- a container `obj` for which `begin(obj)` and `end(obj)` produce a valid pair of iterators - a container `obj` for which `begin(obj)` and `end(obj)` produce a valid pair of iterators
(as found via ADL or member functions, with semantics compatible to `std::begin` and `std::end`) (as found via ADL or member functions, with semantics compatible to `std::begin` and `std::end`)
@@ -91,14 +89,8 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
- Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if unsupported features from CBOR were - Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if unsupported features from CBOR were
used in the given input or if the input is not valid CBOR used in the given input or if the input is not valid CBOR
- Throws [parse_error.113](../../home/exceptions.md#jsonexceptionparse_error113) if a map key is not a string (keys of - Throws [parse_error.113](../../home/exceptions.md#jsonexceptionparse_error113) if a map key is not a string (keys of
other types are not supported, as JSON object keys are always strings), if the head of a string or byte string is other types are not supported, as JSON object keys are always strings), or if a string value or object key is not
malformed (including a chunk of the wrong type or an indefinite-length string nested inside an indefinite-length valid UTF-8 and `error_handler` is `strict`
string), or if a string value or object key (or one chunk of an indefinite-length string) is not valid UTF-8 and
`error_handler` is `strict`
- Throws [out_of_range.408](../../home/exceptions.md#jsonexceptionout_of_range408) if an array or map announces
a length that `std::size_t` cannot hold, or that equals `std::size_t(-1)` (for example `9B FF FF FF FF FF FF FF FF`)
- Throws [out_of_range.406](../../home/exceptions.md#jsonexceptionout_of_range406) if a floating-point number does not
fit into `number_float_t`
## Complexity ## Complexity
+4 -11
View File
@@ -79,18 +79,11 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
- Throws [parse_error.110](../../home/exceptions.md#jsonexceptionparse_error110) if the given input ends prematurely or - Throws [parse_error.110](../../home/exceptions.md#jsonexceptionparse_error110) if the given input ends prematurely or
the end of the file was not reached when `strict` was set to true the end of the file was not reached when `strict` was set to true
- Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if an invalid type marker is read - Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if a parse error occurs
(including a type marker after `$` of an optimized container, even if its count is 0), or if `$` is not followed by `#`
- Throws [parse_error.113](../../home/exceptions.md#jsonexceptionparse_error113) if a string could not be parsed - Throws [parse_error.113](../../home/exceptions.md#jsonexceptionparse_error113) if a string could not be parsed
successfully (negative or non-integer length, `C` followed by a byte above 0x7F), if the count of an optimized successfully, or if a string value or object key is not valid UTF-8 and `error_handler` is `strict`
container is negative, or if a string value or object key is not valid UTF-8 and `error_handler` is `strict` - Throws [out_of_range.408](../../home/exceptions.md#jsonexceptionout_of_range408) if the size of an optimized container
- Throws [parse_error.115](../../home/exceptions.md#jsonexceptionparse_error115) if the text of a high-precision number or n-dimensional array cannot be represented by `std::size_t`
(`H`) is not a valid JSON number
- Throws [out_of_range.406](../../home/exceptions.md#jsonexceptionout_of_range406) if a high-precision number, or a
float64 when `number_float_t` is `float`, does not fit into `number_float_t`
- Throws [out_of_range.408](../../home/exceptions.md#jsonexceptionout_of_range408) if the count of an optimized
container exceeds the capacity of the container (or of `std::size_t`), or if an optimized array of `Z`, `T`, or `F` has
more than 1,048,576 elements
## Complexity ## Complexity
+1 -2
View File
@@ -42,8 +42,7 @@ The exact mapping and its limitations are described on a [dedicated page](../../
## Exception safety ## Exception safety
Strong guarantee: if an exception is thrown, there are no changes in the JSON value. Bytes that were already written to Strong guarantee: if an exception is thrown, there are no changes in the JSON value.
the output adapter are not taken back.
## Exceptions ## Exceptions
@@ -53,7 +53,7 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
value, array, or object exceeds 4294967295, the maximum MessagePack can store; example: value, array, or object exceeds 4294967295, the maximum MessagePack can store; example:
`"MessagePack length 4294967296 exceeds maximum of 4294967295"` `"MessagePack length 4294967296 exceeds maximum of 4294967295"`
- Throws [`out_of_range.415`](../../home/exceptions.md#jsonexceptionout_of_range415) if the subtype of a binary value - Throws [`out_of_range.415`](../../home/exceptions.md#jsonexceptionout_of_range415) if the subtype of a binary value
exceeds 255, the largest value the library can store in the one-byte MessagePack ext type; example: exceeds 255, the maximum of the MessagePack ext type; example:
`"subtype 70000 is too large for the MessagePack ext type (max 255)"` `"subtype 70000 is too large for the MessagePack ext type (max 255)"`
- Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is - Throws [type_error.316](../../home/exceptions.md#jsonexceptiontype_error316) if a string or object key in `j` is
not valid UTF-8 and `error_handler` is `strict` not valid UTF-8 and `error_handler` is `strict`
@@ -11,7 +11,7 @@ extremely small code sizes, fairly small message size, and extensibility without
## Byte layout ## Byte layout
CBOR encodes every data item as a *head* followed by optional content. The first byte of the head holds the *major type* in its upper three bits (unsigned integer, negative integer, byte string, text string, array, map, tag, or simple/float) and an *additional information* value in its lower five bits. Additional information below 24 is the argument itself (for example, a small integer or a short length); the values 24 to 27 announce that the argument follows in 1, 2, 4, or 8 big-endian bytes. Strings, arrays, and maps are prefixed with their length (or count of elements/pairs), and content follows directly. A reader of definite-length items knows how many items to expect without a terminator; indefinite-length items (additional information 31) are instead closed by the "break" byte 0xFF, which `from_cbor` reads but `to_cbor` never writes. CBOR encodes every data item as a *head* followed by optional content. The first byte of the head holds the *major type* in its upper three bits (unsigned integer, negative integer, byte string, text string, array, map, tag, or simple/float) and an *additional information* value in its lower five bits. Additional information below 24 is the argument itself (for example, a small integer or a short length); the values 24 to 27 announce that the argument follows in 1, 2, 4, or 8 big-endian bytes. Strings, arrays, and maps are prefixed with their length (or count of elements/pairs), and content follows directly, so a reader always knows how many items to expect without a terminator.
Consider the following JSON value (shown formatted for readability): Consider the following JSON value (shown formatted for readability):
@@ -52,16 +52,16 @@ The library uses the following mapping from JSON values types to CBOR types acco
| null | `null` | Null | 0xF6 | | null | `null` | Null | 0xF6 |
| boolean | `true` | True | 0xF5 | | boolean | `true` | True | 0xF5 |
| boolean | `false` | False | 0xF4 | | boolean | `false` | False | 0xF4 |
| number_integer | -9223372036854775808..-4294967297 | Negative integer (8 bytes follow) | 0x3B | | number_integer | -9223372036854775808..-2147483649 | Negative integer (8 bytes follow) | 0x3B |
| number_integer | -4294967296..-65537 | Negative integer (4 bytes follow) | 0x3A | | number_integer | -2147483648..-32769 | Negative integer (4 bytes follow) | 0x3A |
| number_integer | -65536..-257 | Negative integer (2 bytes follow) | 0x39 | | number_integer | -32768..-129 | Negative integer (2 bytes follow) | 0x39 |
| number_integer | -256..-25 | Negative integer (1 byte follow) | 0x38 | | number_integer | -128..-25 | Negative integer (1 byte follow) | 0x38 |
| number_integer | -24..-1 | Negative integer | 0x20..0x37 | | number_integer | -24..-1 | Negative integer | 0x20..0x37 |
| number_integer | 0..23 | Integer | 0x00..0x17 | | number_integer | 0..23 | Integer | 0x00..0x17 |
| number_integer | 24..255 | Unsigned integer (1 byte follow) | 0x18 | | number_integer | 24..255 | Unsigned integer (1 byte follow) | 0x18 |
| number_integer | 256..65535 | Unsigned integer (2 bytes follow) | 0x19 | | number_integer | 256..65535 | Unsigned integer (2 bytes follow) | 0x19 |
| number_integer | 65536..4294967295 | Unsigned integer (4 bytes follow) | 0x1A | | number_integer | 65536..4294967295 | Unsigned integer (4 bytes follow) | 0x1A |
| number_integer | 4294967296..9223372036854775807 | Unsigned integer (8 bytes follow) | 0x1B | | number_integer | 4294967296..18446744073709551615 | Unsigned integer (8 bytes follow) | 0x1B |
| number_unsigned | 0..23 | Integer | 0x00..0x17 | | number_unsigned | 0..23 | Integer | 0x00..0x17 |
| number_unsigned | 24..255 | Unsigned integer (1 byte follow) | 0x18 | | number_unsigned | 24..255 | Unsigned integer (1 byte follow) | 0x18 |
| number_unsigned | 256..65535 | Unsigned integer (2 bytes follow) | 0x19 | | number_unsigned | 256..65535 | Unsigned integer (2 bytes follow) | 0x19 |
@@ -95,24 +95,7 @@ see "binary" cells in the table above.
!!! success "Complete mapping" !!! success "Complete mapping"
The mapping is **complete** in the sense that any JSON value type can be converted to a CBOR value, with one The mapping is **complete** in the sense that any JSON value type can be converted to a CBOR value.
exception: an integer outside the range of CBOR integers (only possible with a number type wider than 64 bits)
throws [`out_of_range.407`](../../home/exceptions.md#jsonexceptionout_of_range407).
!!! info "Not a deterministic encoding"
`to_cbor` does not produce the deterministic encoding of
[RFC 8949, Section 4.2.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-4.2.1), and not always the preferred
serialization of Section 4.1:
- floating-point numbers are written as single precision (0xFA) when that is exact and as double precision (0xFB)
otherwise; half precision (0xF9) is used only for `NaN` and `Infinity`/`-Infinity` (`1.0` is `fa 3f 80 00 00`);
- a binary subtype 0..23 is written as `d8 xx` instead of the one-byte tag 0xC0..0xD7 (`d8 01 41 01`);
- object members are written in the order of `object_t` (for the default `std::map`: sorted by the key string, not
by the encoded key bytes).
Integers and lengths use the shortest form, and indefinite-length items are never written. Decoders must accept all
of this; the RFC does not require the preferred serialization.
!!! info "NaN/infinity handling" !!! info "NaN/infinity handling"
@@ -122,8 +105,9 @@ see "binary" cells in the table above.
!!! note !!! note
Prior to version 3.8.0, NaN and Infinity were instead serialized as a CBOR double-precision float (type 0xFB, Prior to version 3.13.0, NaN and Infinity were instead serialized as a CBOR double-precision float (type 0xFB,
9 bytes total). 9 bytes total), because the check used to select a smaller encoding compared magnitudes with NaN, which is
always `false` and caused the intended half-precision path to be skipped.
!!! info "Unused CBOR types" !!! info "Unused CBOR types"
@@ -140,6 +124,7 @@ see "binary" cells in the table above.
- expected conversions (0xD5..0xD7) - expected conversions (0xD5..0xD7)
- simple values (0xE0..0xF3, 0xF8) - simple values (0xE0..0xF3, 0xF8)
- undefined (0xF7) - undefined (0xF7)
- half-precision floats (0xF9)
- break (0xFF) - break (0xFF)
!!! info "Tagged items" !!! info "Tagged items"
@@ -241,20 +226,10 @@ string of the same major type, as required by [RFC 8949, Section 3.2.3](https://
([RFC 8392](https://www.rfc-editor.org/rfc/rfc8392.html)), cannot be read with this library and need a ([RFC 8392](https://www.rfc-editor.org/rfc/rfc8392.html)), cannot be read with this library and need a
general-purpose CBOR library instead. general-purpose CBOR library instead.
A tag in front of a text string key is handled like a tag in front of a value: it is skipped with
`cbor_tag_handler_t::ignore` or `cbor_tag_handler_t::store` and rejected with the default `error`. A tag does not
turn a key of another type into a text string.
A map with duplicate keys is well-formed, but invalid under
[RFC 8949, Section 5.3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-5.3.1). `from_cbor` accepts it and the
last value wins; the [SAX interface](../parsing/sax_interface.md) reports every occurrence.
!!! warning "Ill-formed UTF-8 in text strings" !!! warning "Ill-formed UTF-8 in text strings"
[RFC 8949, Section 3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-3.1) requires CBOR text strings (major [RFC 8949, Section 3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-3.1) requires CBOR text strings (major
type 3) to be valid UTF-8 (a string with ill-formed UTF-8 is "well-formed but invalid"), and type 3) to be valid UTF-8, but leaves it up to the decoder whether to enforce this, so checking is opt-in: with the
[Section 5.3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-5.3.1) leaves it up to the decoder whether to
verify this, so checking is opt-in: with the
[`error_handler`](../../api/basic_json/from_cbor.md) parameter left at `keep` (the default), `from_cbor()` accepts a [`error_handler`](../../api/basic_json/from_cbor.md) parameter left at `keep` (the default), `from_cbor()` accepts a
text string (object keys included) whose bytes are not valid UTF-8 and hands them back unchanged. Passing text string (object keys included) whose bytes are not valid UTF-8 and hands them back unchanged. Passing
`error_handler_t::strict` makes `from_cbor()` check and throw `error_handler_t::strict` makes `from_cbor()` check and throw
@@ -267,14 +242,9 @@ string of the same major type, as required by [RFC 8949, Section 3.2.3](https://
[`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md) is enabled), it throws the same exception [`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md) is enabled), it throws the same exception
instead. Byte strings (major type 2) are unaffected, since they are not required to hold text. instead. Byte strings (major type 2) are unaffected, since they are not required to hold text.
The chunks of an indefinite-length text string are checked one by one, as required by
[RFC 8949, Section 3.2.3](https://www.rfc-editor.org/rfc/rfc8949.html#section-3.2.3): a code point cannot be spread
over two chunks. With `error_handler_t::strict` such a string throws `parse_error.113`; `replace` and `ignore`
sanitize each chunk on its own.
!!! warning "Tagged items" !!! warning "Tagged items"
Tagged items (0xC0..0xDB) will throw a parse error by default. They can be ignored by passing `cbor_tag_handler_t::ignore` to function `from_cbor`, in which case the tag is skipped and the enclosed data item is parsed on its own. Passing `cbor_tag_handler_t::store` to function `from_cbor` stores tagged byte strings (for bytes 0xd8..0xdb) as binary values with the tag as subtype; other tagged values are read as if the tag were ignored. If several tags precede a byte string, only the tag directly in front of it is stored, and only when it has the form 0xD8..0xDB (a tag 0..23 written as 0xC0..0xD7 is skipped); for example, `D8 05 C1 41 AA` yields subtype `null`, whereas `C1 D8 05 41 AA` yields 5. Tags in front of a text string map key are skipped as well. Note that no tag is ever interpreted: for instance, a text string tagged with tag 0 (date/time) stays a string. Tagged items (0xC0..0xDB) will throw a parse error by default. They can be ignored by passing `cbor_tag_handler_t::ignore` to function `from_cbor`, in which case the tag is skipped and the enclosed data item is parsed on its own. Passing `cbor_tag_handler_t::store` to function `from_cbor` stores tagged byte strings (for bytes 0xd8..0xdb) as binary values with the tag as subtype; other tagged values are read as if the tag were ignored. If several tags precede a byte string, only the innermost one is stored. Note that no tag is ever interpreted: for instance, a text string tagged with tag 0 (date/time) stays a string.
??? example "Example: deserialize a JSON value from CBOR" ??? example "Example: deserialize a JSON value from CBOR"
@@ -22,7 +22,7 @@ to efficiently encode JSON values to byte vectors and to decode such vectors.
| BON8 | incomplete: no unsigned integers above int64 | complete | | BON8 | incomplete: no unsigned integers above int64 | complete |
| BSON | incomplete: top-level value must be an object | incomplete, but all JSON types are supported | | BSON | incomplete: top-level value must be an object | incomplete, but all JSON types are supported |
| CBOR | complete | incomplete, but all JSON types are supported | | CBOR | complete | incomplete, but all JSON types are supported |
| MessagePack | complete | incomplete, but all JSON types are supported | | MessagePack | complete | complete |
| UBJSON | complete | complete | | UBJSON | complete | complete |
### Binary values ### Binary values
@@ -11,7 +11,7 @@ extra byte in addition to the strings themselves.
## Byte layout ## Byte layout
MessagePack encodes each value as a one-byte *format marker*, optionally followed by a length or the payload. The marker range is partitioned to keep common values small: small non-negative integers (0 to 127) and small negative integers (-32 to -1) are the marker byte itself; short strings, arrays, and maps (up to 31 bytes for strings and up to 15 elements for arrays and maps) hide their length in the marker's lower bits. Larger values use dedicated markers that announce the width (8, 16, 32, or 64 bits) of the length or number that follows, in big-endian byte order. Like CBOR, containers are length-prefixed, so no terminator is needed. MessagePack encodes each value as a one-byte *format marker*, optionally followed by a length or the payload. The marker range is partitioned to keep common values small: small non-negative integers (0 to 127) and small negative integers (-32 to -1) are the marker byte itself; short strings, arrays, and maps (up to 31 or 15 elements) hide their length in the marker's lower bits. Larger values use dedicated markers that announce the width (8, 16, 32, or 64 bits) of the length or number that follows, in big-endian byte order. Like CBOR, containers are length-prefixed, so no terminator is needed.
Consider the following JSON value (shown formatted for readability): Consider the following JSON value (shown formatted for readability):
@@ -109,9 +109,8 @@ specification:
!!! note !!! note
Prior to version 3.13.0, NaN and Infinity were instead serialized as a MessagePack float 64 (type 0xCB, 9 bytes Prior to version 3.13.0, NaN and Infinity were instead serialized as a MessagePack float 64 (type 0xCB, 9 bytes
total), because the check used to select the smaller float 32 encoding failed for NaN (all comparisons are `false`) total), because the check used to select the smaller float 32 encoding compared magnitudes with NaN, which is
and for Infinity and -Infinity (outside the float range checked against `std::numeric_limits<float>::lowest()` and always `false` and caused the float 32 path to be skipped.
`max()`), so the float 32 path was skipped.
??? example "Example: serialize a JSON value to MessagePack" ??? example "Example: serialize a JSON value to MessagePack"
@@ -136,7 +135,6 @@ The library maps MessagePack types to JSON value types as follows:
| fixarray | array | 0x90..0x9F | | fixarray | array | 0x90..0x9F |
| fixstr | string | 0xA0..0xBF | | fixstr | string | 0xA0..0xBF |
| nil | `null` | 0xC0 | | nil | `null` | 0xC0 |
| (never used) | – | 0xC1 |
| false | `false` | 0xC2 | | false | `false` | 0xC2 |
| true | `true` | 0xC3 | | true | `true` | 0xC3 |
| float 32 | number_float | 0xCA | | float 32 | number_float | 0xCA |
@@ -169,30 +167,6 @@ The library maps MessagePack types to JSON value types as follows:
| fixext 16 | binary | 0xD8 | | fixext 16 | binary | 0xD8 |
| negative fixint | number_integer | 0xE0-0xFF | | negative fixint | number_integer | 0xE0-0xFF |
The byte 0xC1 is reserved as "never used" by the specification. As a value it is rejected with
[`parse_error.112`](../../home/exceptions.md#jsonexceptionparse_error112) (`invalid byte: 0xC1`); as a map key it is
rejected with [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) and the generic "expected length
specification" message, because the key is read as a string.
Duplicate keys in a map are accepted; the last value wins.
### Extension types (ext, fixext)
MessagePack ext and fixext values are read as binary values, and the ext type byte is stored as the binary subtype.
Although the specification defines the type as a signed 8-bit integer (0 to 127 for applications, -128 to -1 reserved,
with -1 being the Timestamp extension), the library reads it as an unsigned byte: a negative ext type `n` becomes
subtype `256+n`, so ext type -1 is read as subtype 255 and -128 as subtype 128. The Timestamp extension is not
interpreted; it is returned as binary with subtype 255. Use
[`has_subtype()`](../../api/byte_container_with_subtype/has_subtype.md) to tell a `bin` value (no subtype) from an ext
value (subtype, even if it is 0).
When serializing, a binary value with a subtype is always written as fixext (1, 2, 4, 8, or 16 bytes) or ext, even
for subtype 0 (`d6 00 01 02 03 04`); a binary value without a subtype is written as bin. Subtypes 128 to 255 are
written as the reserved negative ext types (255 as -1), so `json::binary({1,2,3,4}, 255)` becomes
`d6 ff 01 02 03 04`, which other MessagePack libraries decode as a 32-bit timestamp. A subtype above 255 throws
[`out_of_range.415`](../../home/exceptions.md#jsonexceptionout_of_range415). The payload length is limited to
4294967295 bytes like all other lengths.
!!! info !!! info
Any MessagePack output created by `to_msgpack` can be successfully parsed by `from_msgpack`. Any MessagePack output created by `to_msgpack` can be successfully parsed by `from_msgpack`.
@@ -203,8 +177,7 @@ written as the reserved negative ext types (255 as -1), so `json::binary({1,2,3,
JSON-compatible [profile](https://github.com/msgpack/msgpack/blob/master/spec.md#profile) sketched in the JSON-compatible [profile](https://github.com/msgpack/msgpack/blob/master/spec.md#profile) sketched in the
MessagePack specification, this library restricts map keys to `str` values. Maps with keys of any other type are MessagePack specification, this library restricts map keys to `str` values. Maps with keys of any other type are
rejected with a [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) exception (or, with rejected with a [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) exception (or, with
`allow_exceptions` set to `false`, a discarded value). For most key types the message names the type that was `allow_exceptions` set to `false`, a discarded value) naming the type of the key that was found, for instance:
found, for instance:
``` ```
[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack object key: only string keys are supported, but found nil; last byte: 0xC0 [json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack object key: only string keys are supported, but found nil; last byte: 0xC0
@@ -216,7 +189,7 @@ written as the reserved negative ext types (255 as -1), so `json::binary({1,2,3,
!!! warning "Ill-formed UTF-8 in string values" !!! warning "Ill-formed UTF-8 in string values"
The MessagePack specification explicitly allows a `str` value (`fixstr`, `str 8`, `str 16`, `str 32`) to contain The MessagePack specification explicitly allows a `str` value (`fixstr`, `str 8`, `str 16`, `str 32`) to contain
a byte sequence that is not valid UTF-8, and recommends that a deserializer provide a way to obtain the original bytes. a byte sequence that is not valid UTF-8, and expects a deserializer to hand the original bytes back unchanged.
This library follows that by default: with its This library follows that by default: with its
[`error_handler`](../../api/basic_json/from_msgpack.md) parameter left at `keep` (the default), [`error_handler`](../../api/basic_json/from_msgpack.md) parameter left at `keep` (the default),
`from_msgpack()` reads `str` bytes (object keys included) as-is, without validating them, so such a value `from_msgpack()` reads `str` bytes (object keys included) as-is, without validating them, so such a value
@@ -5,11 +5,11 @@ achieve the generality of JSON, combined with being much easier to process than
!!! abstract "References" !!! abstract "References"
- [UBJSON Website](https://ubjson.org) - [UBJSON Website](http://ubjson.org)
## Byte layout ## Byte layout
UBJSON is built from just a one-byte, human-readable ASCII *type marker* (for example, `i` for int8, `S` for string, `[` for array) followed by the payload of that type. Strings and the like are prefixed with their length, which is itself a UBJSON integer value including its marker. Numbers are stored in big-endian byte order. Containers can be written in two ways: as an *unoptimized* container that is opened with `[` or `{` and closed with `]` or `}`, or as an *optimized* container that declares a count (`#`) after the opening marker, optionally preceded by an element type (`$`); an element type is only allowed together with a count. An optimized container saves the closing marker and, with a type, the per-element markers. UBJSON is built from just a one-byte, human-readable ASCII *type marker* (for example, `i` for int8, `S` for string, `[` for array) followed by the payload of that type. Strings and the like are prefixed with their length, which is itself a UBJSON integer value including its marker. Numbers are stored in big-endian byte order. Containers can be written in two ways: as an *unoptimized* container that is opened with `[` or `{` and closed with `]` or `}`, or as an *optimized* container that declares an element type (`$`) and/or a count (`#`) after the opening marker, which saves per-element markers and (for a count) the closing marker.
Consider the following JSON value (shown formatted for readability): Consider the following JSON value (shown formatted for readability):
@@ -45,7 +45,7 @@ Consider the following JSON value (shown formatted for readability):
The library uses the following mapping from JSON values types to UBJSON types according to the UBJSON specification: The library uses the following mapping from JSON values types to UBJSON types according to the UBJSON specification:
| JSON value type | value/range | UBJSON type | marker | | JSON value type | value/range | UBJSON type | marker |
|-----------------|-------------------------------------------|--------------------------------------------------|-----------| |-----------------|-------------------------------------------|----------------|--------|
| null | `null` | null | `Z` | | null | `null` | null | `Z` |
| boolean | `true` | true | `T` | | boolean | `true` | true | `T` |
| boolean | `false` | false | `F` | | boolean | `false` | false | `F` |
@@ -63,10 +63,10 @@ The library uses the following mapping from JSON values types to UBJSON types ac
| number_unsigned | 32768..2147483647 | int32 | `l` | | number_unsigned | 32768..2147483647 | int32 | `l` |
| number_unsigned | 2147483648..9223372036854775807 | int64 | `L` | | number_unsigned | 2147483648..9223372036854775807 | int64 | `L` |
| number_unsigned | 9223372036854775808..18446744073709551615 | high-precision | `H` | | number_unsigned | 9223372036854775808..18446744073709551615 | high-precision | `H` |
| number_float | *any value* | float64 (float32 if `number_float_t` is `float`) | `D` (`d`) | | number_float | *any value* | float64 | `D` |
| string | *with shortest length indicator* | string | `S` | | string | *with shortest length indicator* | string | `S` |
| array | *see notes on optimized format* | array | `[` | | array | *see notes on optimized format* | array | `[` |
| object | *see notes on optimized format* | object | `{` | | object | *see notes on optimized format* | map | `{` |
!!! success "Complete mapping" !!! success "Complete mapping"
@@ -93,27 +93,20 @@ The library uses the following mapping from JSON values types to UBJSON types ac
The following markers are not used in the conversion: The following markers are not used in the conversion:
- `N`: no-op values are never written (they are skipped when reading). - `Z`: no-op values are not created.
- `C`: single-byte strings are serialized with `S` markers. - `C`: single-byte strings are serialized with `S` markers.
!!! info "NaN/infinity handling" !!! info "NaN/infinity handling"
NaN and infinity are written as IEEE 754 float64 values (`D`) and read back unchanged. This differs from If NaN or Infinity are stored inside a JSON number, they are serialized properly. This behavior differs from the
[`dump()`](../../api/basic_json/dump.md), which serializes them to `null`, and from the UBJSON specification, which [`dump()`](../../api/basic_json/dump.md) function which serializes NaN or Infinity to `null`.
encodes infinity as `null` (it does not mention NaN); other UBJSON readers may therefore not accept these values.
!!! info "Optimized formats" !!! info "Optimized formats"
The optimized formats for containers are supported: Parameter `use_size` adds size information to the beginning of a The optimized formats for containers are supported: Parameter `use_size` adds size information to the beginning of a
container and removes the closing marker. Parameter `use_type` further checks whether all elements of a container container and removes the closing marker. Parameter `use_type` further checks whether all elements of a container
have the same type and adds the type marker to the beginning of the container. The `use_type` parameter must only be have the same type and adds the type marker to the beginning of the container. The `use_type` parameter must only be
used together with `use_size = true`: combining `use_type` with `use_size = false` throws used together with `use_size = true`.
[`other_error.502`](../../home/exceptions.md#jsonexceptionother_error502) if the value contains a non-empty array,
object, or binary value.
Two elements have the same type if they are written with the same type marker. For example, `[1, 2]` is written with
the type `i`, but `[1, 200]` (`i`, `U`), `[true, false]` (`T`, `F`), and `[1, 2.5]` (`i`, `D`) are not; an empty
container is never given a `$` annotation of its own (an array of empty arrays is still typed `[`).
Note that `use_size = true` alone may result in larger representations - the benefit of this parameter is that the Note that `use_size = true` alone may result in larger representations - the benefit of this parameter is that the
receiving side is immediately informed on the number of elements of the container. receiving side is immediately informed on the number of elements of the container.
@@ -122,9 +115,8 @@ The library uses the following mapping from JSON values types to UBJSON types ac
already is the value. Its declared count is therefore the only thing that decides how much memory the receiving side already is the value. Its declared count is therefore the only thing that decides how much memory the receiving side
allocates, and a handful of bytes can describe billions of elements. `from_ubjson` rejects such an array with allocates, and a handful of bytes can describe billions of elements. `from_ubjson` rejects such an array with
[`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408) when the count exceeds 1,048,576 [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408) when the count exceeds 1,048,576
(`1 << 20`), and `to_ubjson` writes longer arrays of these types in the counted but untyped form: the `$` type marker is (`1 << 20`), and `to_ubjson` writes longer arrays of these types without the annotation, so any value it produces
dropped, the `#` count is kept, and every element is written with its own marker (`[#l 1048577 Z Z ...`), so any value can be read back.
it produces can be read back. Objects and untyped counted arrays are not subject to this limit.
!!! info "Binary values" !!! info "Binary values"
@@ -149,7 +141,7 @@ The library uses the following mapping from JSON values types to UBJSON types ac
The library maps UBJSON types to JSON value types as follows: The library maps UBJSON types to JSON value types as follows:
| UBJSON type | JSON value type | marker | | UBJSON type | JSON value type | marker |
|----------------|--------------------------------------------------|--------| |-------------|-----------------------------------------|--------|
| no-op | *no value, next value is read* | `N` | | no-op | *no value, next value is read* | `N` |
| null | `null` | `Z` | | null | `null` | `Z` |
| false | `false` | `F` | | false | `false` | `F` |
@@ -163,39 +155,12 @@ The library maps UBJSON types to JSON value types as follows:
| int64 | number_integer | `L` | | int64 | number_integer | `L` |
| string | string | `S` | | string | string | `S` |
| char | string | `C` | | char | string | `C` |
| high-precision | number_unsigned, number_integer, or number_float | `H` |
| array | array (optimized values are supported) | `[` | | array | array (optimized values are supported) | `[` |
| object | object (optimized values are supported) | `{` | | object | object (optimized values are supported) | `{` |
!!! info "High-precision numbers"
The payload of `H` is a length-prefixed string that is parsed as a JSON number (surrounding whitespace is ignored).
Integer text in the range of `number_unsigned_t` (non-negative) or `number_integer_t` (negative) is stored as such;
any other valid number text (larger or smaller integers, fractions, and exponents such as `1e2`) is stored as
`number_float`, so precision may be lost. Text that is not a valid JSON number raises
[`parse_error.115`](../../home/exceptions.md#jsonexceptionparse_error115), a number outside the range of
`number_float_t` (`1e999`) raises [`out_of_range.406`](../../home/exceptions.md#jsonexceptionout_of_range406).
!!! success "Complete mapping" !!! success "Complete mapping"
Every well-formed UBJSON value can be converted to a JSON value, with these exceptions: a high-precision number The mapping is **complete** in the sense that any UBJSON value can be converted to a JSON value.
outside the range of `number_float_t` ([`out_of_range.406`](../../home/exceptions.md#jsonexceptionout_of_range406)),
an optimized array of `Z`, `T`, or `F` with more than 1,048,576 elements, and a count that exceeds the capacity of the
container ([`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408)). High-precision numbers that
do not fit an integer type are stored as floating-point numbers and may lose precision. Ill-formed input is rejected
with a parse error.
!!! info "Duplicate object keys"
If an object contains the same key more than once, the last value is kept (the UBJSON specification does not
address duplicate keys).
!!! info "Type marker of optimized containers"
The type marker after `$` must be one of `Z`, `N`, `T`, `F`, `i`, `U`, `I`, `l`, `L`, `d`, `D`, `H`, `C`, `S`, `[`, or
`{`. It is checked even if the count is 0; any other marker raises
[`parse_error.112`](../../home/exceptions.md#jsonexceptionparse_error112). A counted object typed `N` (`{$N#i1 ...`)
is not supported.
!!! warning "Ill-formed UTF-8 in string values and object keys" !!! warning "Ill-formed UTF-8 in string values and object keys"
+2 -14
View File
@@ -301,8 +301,6 @@ representation using the length of the byte array.
[`json::from_cbor`](../api/basic_json/from_cbor.md) (see [`json::from_cbor`](../api/basic_json/from_cbor.md) (see
[`cbor_tag_handler_t`](../api/basic_json/cbor_tag_handler_t.md)). [`cbor_tag_handler_t`](../api/basic_json/cbor_tag_handler_t.md)).
With `ignore`, the subtype is dropped (`"subtype": null`); with `store`, the tag is kept (`"subtype": 42`):
```json ```json
{ {
"binary": { "binary": {
@@ -312,21 +310,11 @@ representation using the length of the byte array.
} }
``` ```
```json
{
"binary": {
"bytes": [202, 254, 186, 190],
"subtype": 42
}
}
```
### MessagePack ### MessagePack
[MessagePack](binary_formats/messagepack.md) supports binary values and subtypes. If a subtype is given, the ext family [MessagePack](binary_formats/messagepack.md) supports binary values and subtypes. If a subtype is given, the ext family
is used. The library will choose the smallest representation among fixext1, fixext2, fixext4, fixext8, fixext16, ext8, is used. The library will choose the smallest representation among fixext1, fixext2, fixext4, fixext8, ext8, ext16, and
ext16, and ext32. The subtype is then written as the signed 8-bit ext type: subtypes 128 to 255 map to the negative ext32. The subtype is then added as a signed 8-bit integer.
(reserved) ext types, and on reading the ext type is exposed as an unsigned subtype from 0 to 255.
If no subtype is given, the bin family (bin8, bin16, bin32) is used. If no subtype is given, the bin family (bin8, bin16, bin32) is used.
@@ -368,7 +368,8 @@ using array_t = ArrayType<basic_json, AllocatorType<basic_json>>;
`#!cpp std::from_chars` is not available or declines the token, and `std::strtold` reads up to the null character. A type whose `data()` `#!cpp std::from_chars` is not available or declines the token, and `std::strtold` reads up to the null character. A type whose `data()`
is not null-terminated does not fail to compile; with such a `NumberFloatType` it can silently misparse is not null-terminated does not fail to compile; with such a `NumberFloatType` it can silently misparse
floating-point numbers. floating-point numbers.
- `append(const char*, size_type)`, used by [`dump`](../../api/basic_json/dump.md) and by the binary readers. The library's internal string concatenation additionally has - `append(const char*, size_type)`, used by [`dump`](../../api/basic_json/dump.md), and `append(const StringType&)`,
used by the CBOR reader for indefinite-length strings. The library's internal string concatenation additionally has
to append a `#!cpp char` and a `#!cpp const char*`; for each it selects between `append(arg)`, `#!cpp operator+=`, to append a `#!cpp char` and a `#!cpp const char*`; for each it selects between `append(arg)`, `#!cpp operator+=`,
`append(first, last)`, and `append(data, size)`. `append(first, last)`, and `append(data, size)`.
- The comparison operator `==` against another `StringType`, and `<` for use as a key of the chosen - The comparison operator `==` against another `StringType`, and `<` for use as a key of the chosen
@@ -677,7 +678,8 @@ using binary_t = nlohmann::byte_container_with_subtype<BinaryType>;
[`StringType`](#stringtype), whose non-`#!cpp const` `data()` is C++17 and later only. [`StringType`](#stringtype), whose non-`#!cpp const` `data()` is C++17 and later only.
- Default-constructible, copy-constructible, and move-constructible. - Default-constructible, copy-constructible, and move-constructible.
- Member functions `size()`, `empty()`, `data()`, `resize()`, `operator[]`, `back()`, `begin()`, `end()`, `cbegin()`, - Member functions `size()`, `empty()`, `data()`, `resize()`, `operator[]`, `back()`, `begin()`, `end()`, `cbegin()`,
and `cend()` with random-access iterators. `push_back()` is **not** required. and `cend()` with random-access iterators, and `insert(pos, first, last)`, which the CBOR reader uses to join the
chunks of an indefinite-length byte string. `push_back()` is **not** required.
- Comparison operators: `==` is used by - Comparison operators: `==` is used by
[`byte_container_with_subtype`](../../api/byte_container_with_subtype/index.md), the relational operators by [`byte_container_with_subtype`](../../api/byte_container_with_subtype/index.md), the relational operators by
[`basic_json`'s comparison operators](../../api/basic_json/operator_le.md). [`basic_json`'s comparison operators](../../api/basic_json/operator_le.md).
+5 -6
View File
@@ -330,7 +330,7 @@ where they cannot be resolved.
### json.exception.parse_error.110 ### json.exception.parse_error.110
When parsing a [binary format](../features/binary_formats/index.md), the byte vector ends before the complete value has When parsing a [binary format](../features/binary_formats/index.md), the byte vector ends before the complete value has
been read, or (with `strict = true`) further bytes follow the value. been read.
!!! failure "Example messages" !!! failure "Example messages"
@@ -1045,8 +1045,8 @@ or the encoded length for [CBOR](../features/binary_formats/cbor.md).
The exception is also thrown for a [UBJSON](../features/binary_formats/ubjson.md) array of a type that is encoded by its The exception is also thrown for a [UBJSON](../features/binary_formats/ubjson.md) array of a type that is encoded by its
marker alone (`Z`, `T` or `F`) whose declared count exceeds 1,048,576 (`1 << 20`). Such an array has no payload, so its marker alone (`Z`, `T` or `F`) whose declared count exceeds 1,048,576 (`1 << 20`). Such an array has no payload, so its
count alone decides how much memory is allocated, and a handful of bytes would otherwise describe billions of values. count alone decides how much memory is allocated, and a handful of bytes would otherwise describe billions of values.
[`to_ubjson`](../api/basic_json/to_ubjson.md) writes longer arrays of these types without the `$` type annotation (the `#` [`to_ubjson`](../api/basic_json/to_ubjson.md) writes longer arrays of these types without the size and type annotation,
count and one marker per element are kept), so any value it produces can still be read back. so any value it produces can still be read back.
!!! failure "Example messages" !!! failure "Example messages"
@@ -1153,7 +1153,7 @@ A JSON Patch `move` operation's `"from"` location is a proper prefix of its `"pa
### json.exception.out_of_range.415 ### json.exception.out_of_range.415
MessagePack's ext type and BSON's binary subtype are each stored in a single byte. This exception is thrown when serializing a MessagePack's ext type and BSON's binary subtype are each stored in a single byte. This exception is thrown when serializing a
[`byte_container_with_subtype`](../api/byte_container_with_subtype/index.md) whose subtype exceeds 255, the largest value the library can store in that byte. [`byte_container_with_subtype`](../api/byte_container_with_subtype/index.md) whose subtype exceeds 255.
!!! failure "Example message" !!! failure "Example message"
@@ -1229,8 +1229,7 @@ A JSON Patch operation 'test' failed. The unsuccessful operation is also printed
### json.exception.other_error.502 ### json.exception.other_error.502
[`to_ubjson`](../api/basic_json/to_ubjson.md) and [`to_bjdata`](../api/basic_json/to_bjdata.md) were called with [`to_ubjson`](../api/basic_json/to_ubjson.md) and [`to_bjdata`](../api/basic_json/to_bjdata.md) were called with
`use_type = true` but `use_size = false` for a value that contains a non-empty array, object, or binary value. UBJSON `use_type = true` but `use_size = false`. UBJSON requires a size marker (`#`) after a type marker (`$`).
requires a size marker (`#`) after a type marker (`$`); scalars and empty containers do not throw.
!!! failure "Example message" !!! failure "Example message"
+11 -121
View File
@@ -1218,10 +1218,6 @@ class binary_reader
// level as required by RFC 8949, Section 3.2.3 // level as required by RFC 8949, Section 3.2.3
bool indefinite = false; bool indefinite = false;
// the chunk currently being validated; only used for the chunks of an
// indefinite-length string when the UTF-8 check can do something
string_t chunk;
while (true) while (true)
{ {
if (JSON_HEDLEY_UNLIKELY(!unexpect_eof("string"))) if (JSON_HEDLEY_UNLIKELY(!unexpect_eof("string")))
@@ -1241,82 +1237,26 @@ class binary_reader
} }
// a break marker closes the indefinite-length string; outside // a break marker closes the indefinite-length string; outside
// of one it falls through to the error below. Every chunk has // of one it falls through to the error below
// been checked on its own, so there is nothing left to check.
if (indefinite && current == 0xFF) if (indefinite && current == 0xFF)
{ {
return true; return check_string_utf8(result, context);
}
if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result, indefinite)))
{
return false;
} }
if (!indefinite) if (!indefinite)
{ {
return get_cbor_string_chunk(result, false) && check_string_utf8(result, context); return check_string_utf8(result, context);
}
if (error_handler == error_handler_t::keep)
{
// nothing to check: read the chunk directly into the result
if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result, true)))
{
return false;
}
}
else
{
// RFC 8949, Section 3.2.3: each chunk is a text string of its
// own and must be valid UTF-8 by itself, i.e., a code point
// cannot be spread over two chunks
chunk.clear();
if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(chunk, true) || !check_string_utf8(chunk, context)))
{
return false;
}
result.append(chunk.data(), chunk.size());
} }
get(); get();
} }
} }
/*!
@brief skips the tags in front of a CBOR object key
A tag may wrap any data item, so it may also wrap a key. Like in front of a
value, the tags are skipped for the handlers ignore and store; with the
handler error they are left to @ref get_cbor_object_key, which rejects the
key. The tag loop is iterative, so the number of tags is not limited by the
recursion depth.
@param[in] tag_handler how CBOR tags should be treated
@return whether the tags could be skipped (false on EOF in a tag argument)
@pre @a current is the first byte of the key
*/
bool skip_cbor_key_tags(const cbor_tag_handler_t tag_handler)
{
if (tag_handler == cbor_tag_handler_t::error)
{
return true;
}
// 0xC0..0xDB: tag with the tag number in the head (0xC0..0xD7) or in
// the 1, 2, 4, or 8 bytes that follow (0xD8..0xDB)
while (current >= 0xC0 && current <= 0xDB)
{
if (current >= 0xD8)
{
std::uint64_t tag_to_ignore{};
if (JSON_HEDLEY_UNLIKELY(!get_cbor_argument(tag_to_ignore)))
{
return false;
}
}
get();
}
return true;
}
/*! /*!
@brief reads a CBOR object key @brief reads a CBOR object key
@@ -1683,10 +1623,6 @@ class binary_reader
if (top.is_object) if (top.is_object)
{ {
if (JSON_HEDLEY_UNLIKELY(!skip_cbor_key_tags(tag_handler)))
{
return false;
}
if (JSON_HEDLEY_UNLIKELY(!read_object_key<&binary_reader::get_cbor_object_key>(key))) if (JSON_HEDLEY_UNLIKELY(!read_object_key<&binary_reader::get_cbor_object_key>(key)))
{ {
return false; return false;
@@ -3000,14 +2936,6 @@ class binary_reader
return false; return false;
} }
// validate the type marker up front: otherwise it would only be
// checked when an element is read, and a container with count 0
// would accept any marker
if (JSON_HEDLEY_UNLIKELY(!is_valid_optimized_type(result.second)))
{
return optimized_type_error();
}
get_ignore_noop(); get_ignore_noop();
if (JSON_HEDLEY_UNLIKELY(current != '#')) if (JSON_HEDLEY_UNLIKELY(current != '#'))
{ {
@@ -4202,9 +4130,9 @@ class binary_reader
// sequence), UBJSON, BJData, or BSON requires a decoder to reject // sequence), UBJSON, BJData, or BSON requires a decoder to reject
// ill-formed UTF-8. Checking (and, with @ref error_handler_t::strict, // ill-formed UTF-8. Checking (and, with @ref error_handler_t::strict,
// rejecting, or with `replace`/`ignore`, sanitizing) is opt-in via // rejecting, or with `replace`/`ignore`, sanitizing) is opt-in via
// @ref error_handler, applied by @ref check_string_utf8 at the call // @ref error_handler, applied once the whole string (all chunks of
// site to the whole string or, for an indefinite-length CBOR string, // an indefinite-length CBOR string included) has been assembled, by
// to each chunk on its own (RFC 8949, Section 3.2.3). // @ref check_string_utf8 at the call site.
return get_bytes(len, "string", result); return get_bytes(len, "string", result);
} }
@@ -4504,44 +4432,6 @@ class binary_reader
std::size_t bon8_pushback_size = 0; std::size_t bon8_pushback_size = 0;
JSON_PRIVATE_UNLESS_TESTED: JSON_PRIVATE_UNLESS_TESTED:
/*!
@brief check whether @a marker is a type marker that may follow '$'
UBJSON: Z N T F i U I l L d D H C S [ {; BJData: the numeric markers and C
(BJData's excluded markers are rejected separately).
*/
bool is_valid_optimized_type(const char_int_type marker) const noexcept
{
switch (marker)
{
case 'i':
case 'U':
case 'I':
case 'l':
case 'L':
case 'd':
case 'D':
case 'C':
return true;
case 'Z':
case 'N':
case 'T':
case 'F':
case 'H':
case 'S':
case '[':
case '{':
return input_format != input_format_t::bjdata;
case 'B':
case 'u':
case 'm':
case 'M':
case 'h':
return input_format == input_format_t::bjdata;
default:
return false;
}
}
/*! /*!
@brief whether @a marker is excluded from BJData's optimized ND-array types @brief whether @a marker is excluded from BJData's optimized ND-array types
@return whether @a marker is one of 'F', 'H', 'N', 'S', 'T', 'Z', '[', '{' @return whether @a marker is one of 'F', 'H', 'N', 'S', 'T', 'Z', '[', '{'
File diff suppressed because it is too large. Load diff
+97 -15
View File
@@ -255,6 +255,81 @@ struct number_token
bool is_float; ///< a fraction or an exponent bool is_float; ///< a fraction or an exponent
}; };
/// advance over ASCII digits, eight bytes at a time
inline const unsigned char* digit_run(const unsigned char* p, const unsigned char* e) noexcept
{
constexpr std::uint64_t ones = 0x0101010101010101ull;
while (e - p >= 8)
{
// (the digits become 0..9; a byte above 9, or with bit 7 set, is no digit. A carry
// out of a byte with bit 7 set only affects the bytes above, after the first one
// that count_trailing_zeros finds.)
const std::uint64_t x = read_eight_bytes(p) ^ (ones * 0x30u);
const std::uint64_t not_digit = ((x + (ones * 0x76u)) | x) & (ones * 0x80u);
if (not_digit != 0)
{
return p + (static_cast<std::size_t>(count_trailing_zeros(not_digit)) / 8);
}
p += 8;
}
while (p != e && is_digit(*p))
{
++p;
}
return p;
}
/// whether the n bytes at p are ASCII digits (n > 0)
inline bool all_digits(const unsigned char* p, std::size_t n) noexcept
{
const auto digits = [](const unsigned char* q)
{
constexpr std::uint64_t ones = 0x0101010101010101ull;
const std::uint64_t x = read_eight_bytes(q) ^ (ones * 0x30u);
return (((x + (ones * 0x76u)) | x) & (ones * 0x80u)) == 0;
};
if (n >= 8)
{
for (std::size_t i = 0; i + 8 < n; i += 8)
{
if (!digits(p + i))
{
return false;
}
}
return digits(p + n - 8); // (overlapping the bytes before: they are digits)
}
for (std::size_t i = 0; i < n; ++i)
{
if (!is_digit(p[i]))
{
return false;
}
}
return true;
}
/// Whether a float token has the shape that the parser records for most of
/// them: [-]digits.digits, with the numbers of digits of the node's layout
/// (extra). Then the token is a number of that layout, and the general scan
/// that follows is not needed.
inline bool plain_float_token(const unsigned char* s, std::size_t len, std::uint16_t extra) noexcept
{
const std::size_t int_digits = extra & 0xFFu;
const std::size_t frac_digits = extra >> 8u;
if (int_digits == 0 || int_digits >= 255 || frac_digits == 0 || frac_digits >= 255 || len == 0)
{
return false;
}
const std::size_t negative = *s == '-' ? 1 : 0;
if (len != negative + int_digits + 1 + frac_digits)
{
return false;
}
const unsigned char* const p = s + negative;
return !(int_digits > 1 && *p == '0') && p[int_digits] == '.' && all_digits(p, int_digits) && all_digits(p + int_digits + 1, frac_digits);
}
/// whether [s, s + len) is a JSON number (the grammar the parser accepts) /// whether [s, s + len) is a JSON number (the grammar the parser accepts)
inline bool scan_number_token(const unsigned char* s, std::size_t len, number_token& t) inline bool scan_number_token(const unsigned char* s, std::size_t len, number_token& t)
{ {
@@ -273,11 +348,7 @@ inline bool scan_number_token(const unsigned char* s, std::size_t len, number_to
} }
else if (*p >= '1' && *p <= '9') else if (*p >= '1' && *p <= '9')
{ {
do p = digit_run(p + 1, e);
{
++p;
}
while (p != e && is_digit(*p));
} }
else else
{ {
@@ -289,10 +360,7 @@ inline bool scan_number_token(const unsigned char* s, std::size_t len, number_to
if (p != e && *p == '.') if (p != e && *p == '.')
{ {
const unsigned char* const f0 = ++p; const unsigned char* const f0 = ++p;
while (p != e && is_digit(*p)) p = digit_run(p, e);
{
++p;
}
if (p == f0) if (p == f0)
{ {
return false; return false;
@@ -403,10 +471,15 @@ struct float_range
template<typename Check> template<typename Check>
bool check_distinct_ranges(std::vector<byte_range>& ranges, Check check) bool check_distinct_ranges(std::vector<byte_range>& ranges, Check check)
{ {
std::sort(ranges.begin(), ranges.end(), [](const byte_range & a, const byte_range & b) // (save() writes the ranges in the order of the nodes, which is sorted already)
const auto before = [](const byte_range & a, const byte_range & b)
{ {
return a.off != b.off ? a.off < b.off : a.len < b.len; return a.off != b.off ? a.off < b.off : a.len < b.len;
}); };
if (!std::is_sorted(ranges.begin(), ranges.end(), before))
{
std::sort(ranges.begin(), ranges.end(), before);
}
std::size_t end = 0; std::size_t end = 0;
for (std::size_t i = 0; i < ranges.size(); ++i) for (std::size_t i = 0; i < ranges.size(); ++i)
{ {
@@ -428,7 +501,8 @@ bool check_distinct_ranges(std::vector<byte_range>& ranges, Check check)
/// record layouts that match it), tokens must not overlap. /// record layouts that match it), tokens must not overlap.
inline bool check_float_ranges(std::vector<float_range>& ranges, const unsigned char* text) inline bool check_float_ranges(std::vector<float_range>& ranges, const unsigned char* text)
{ {
std::sort(ranges.begin(), ranges.end(), [](const float_range & a, const float_range & b) // (save() writes the tokens in the order of the nodes, which is sorted already)
const auto before = [](const float_range & a, const float_range & b)
{ {
if (a.off != b.off) if (a.off != b.off)
{ {
@@ -439,7 +513,11 @@ inline bool check_float_ranges(std::vector<float_range>& ranges, const unsigned
return a.len < b.len; return a.len < b.len;
} }
return a.extra < b.extra; return a.extra < b.extra;
}); };
if (!std::is_sorted(ranges.begin(), ranges.end(), before))
{
std::sort(ranges.begin(), ranges.end(), before);
}
std::size_t end = 0; std::size_t end = 0;
std::size_t i = 0; std::size_t i = 0;
while (i < ranges.size()) while (i < ranges.size())
@@ -449,14 +527,18 @@ inline bool check_float_ranges(std::vector<float_range>& ranges, const unsigned
{ {
return false; return false;
} }
number_token t{};
const unsigned char* const s = text + r.off; const unsigned char* const s = text + r.off;
std::uint16_t layout = r.extra;
if (!plain_float_token(s, r.len, r.extra))
{
number_token t{};
if (!scan_number_token(s, r.len, t) || !float_token_finite(s, r.len, t)) if (!scan_number_token(s, r.len, t) || !float_token_finite(s, r.len, t))
{ {
return false; return false;
} }
// the digit layout the parser records (or "many", as compaction writes it) // the digit layout the parser records (or "many", as compaction writes it)
const std::uint16_t layout = float_layout(t); layout = float_layout(t);
}
for (; i < ranges.size() && ranges[i].off == r.off && ranges[i].len == r.len; ++i) for (; i < ranges.size() && ranges[i].off == r.off && ranges[i].len == r.len; ++i)
{ {
if (ranges[i].extra != layout && ranges[i].extra != 0xFFFFu) if (ranges[i].extra != layout && ranges[i].extra != 0xFFFFu)
+4
View File
@@ -743,8 +743,12 @@ class basic_json_view
template<typename, bool> friend class basic_json_document; template<typename, bool> friend class basic_json_document;
template<typename, bool> friend class basic_json_view; template<typename, bool> friend class basic_json_view;
template<typename, typename> friend class detail::view::editor; template<typename, typename> friend class detail::view::editor;
template<typename, typename> friend class detail::view::view_side;
friend iterator; friend iterator;
/// whether the view is one of an editable document
static constexpr bool is_editable = Editable;
basic_json_view(const document_data* d, const node* n) noexcept basic_json_view(const document_data* d, const node* n) noexcept
: m_doc(d), m_node(n) : m_doc(d), m_node(n)
{} {}
+11 -121
View File
@@ -18685,10 +18685,6 @@ class binary_reader
// level as required by RFC 8949, Section 3.2.3 // level as required by RFC 8949, Section 3.2.3
bool indefinite = false; bool indefinite = false;
// the chunk currently being validated; only used for the chunks of an
// indefinite-length string when the UTF-8 check can do something
string_t chunk;
while (true) while (true)
{ {
if (JSON_HEDLEY_UNLIKELY(!unexpect_eof("string"))) if (JSON_HEDLEY_UNLIKELY(!unexpect_eof("string")))
@@ -18708,82 +18704,26 @@ class binary_reader
} }
// a break marker closes the indefinite-length string; outside // a break marker closes the indefinite-length string; outside
// of one it falls through to the error below. Every chunk has // of one it falls through to the error below
// been checked on its own, so there is nothing left to check.
if (indefinite && current == 0xFF) if (indefinite && current == 0xFF)
{ {
return true; return check_string_utf8(result, context);
}
if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result, indefinite)))
{
return false;
} }
if (!indefinite) if (!indefinite)
{ {
return get_cbor_string_chunk(result, false) && check_string_utf8(result, context); return check_string_utf8(result, context);
}
if (error_handler == error_handler_t::keep)
{
// nothing to check: read the chunk directly into the result
if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(result, true)))
{
return false;
}
}
else
{
// RFC 8949, Section 3.2.3: each chunk is a text string of its
// own and must be valid UTF-8 by itself, i.e., a code point
// cannot be spread over two chunks
chunk.clear();
if (JSON_HEDLEY_UNLIKELY(!get_cbor_string_chunk(chunk, true) || !check_string_utf8(chunk, context)))
{
return false;
}
result.append(chunk.data(), chunk.size());
} }
get(); get();
} }
} }
/*!
@brief skips the tags in front of a CBOR object key
A tag may wrap any data item, so it may also wrap a key. Like in front of a
value, the tags are skipped for the handlers ignore and store; with the
handler error they are left to @ref get_cbor_object_key, which rejects the
key. The tag loop is iterative, so the number of tags is not limited by the
recursion depth.
@param[in] tag_handler how CBOR tags should be treated
@return whether the tags could be skipped (false on EOF in a tag argument)
@pre @a current is the first byte of the key
*/
bool skip_cbor_key_tags(const cbor_tag_handler_t tag_handler)
{
if (tag_handler == cbor_tag_handler_t::error)
{
return true;
}
// 0xC0..0xDB: tag with the tag number in the head (0xC0..0xD7) or in
// the 1, 2, 4, or 8 bytes that follow (0xD8..0xDB)
while (current >= 0xC0 && current <= 0xDB)
{
if (current >= 0xD8)
{
std::uint64_t tag_to_ignore{};
if (JSON_HEDLEY_UNLIKELY(!get_cbor_argument(tag_to_ignore)))
{
return false;
}
}
get();
}
return true;
}
/*! /*!
@brief reads a CBOR object key @brief reads a CBOR object key
@@ -19150,10 +19090,6 @@ class binary_reader
if (top.is_object) if (top.is_object)
{ {
if (JSON_HEDLEY_UNLIKELY(!skip_cbor_key_tags(tag_handler)))
{
return false;
}
if (JSON_HEDLEY_UNLIKELY(!read_object_key<&binary_reader::get_cbor_object_key>(key))) if (JSON_HEDLEY_UNLIKELY(!read_object_key<&binary_reader::get_cbor_object_key>(key)))
{ {
return false; return false;
@@ -20467,14 +20403,6 @@ class binary_reader
return false; return false;
} }
// validate the type marker up front: otherwise it would only be
// checked when an element is read, and a container with count 0
// would accept any marker
if (JSON_HEDLEY_UNLIKELY(!is_valid_optimized_type(result.second)))
{
return optimized_type_error();
}
get_ignore_noop(); get_ignore_noop();
if (JSON_HEDLEY_UNLIKELY(current != '#')) if (JSON_HEDLEY_UNLIKELY(current != '#'))
{ {
@@ -21669,9 +21597,9 @@ class binary_reader
// sequence), UBJSON, BJData, or BSON requires a decoder to reject // sequence), UBJSON, BJData, or BSON requires a decoder to reject
// ill-formed UTF-8. Checking (and, with @ref error_handler_t::strict, // ill-formed UTF-8. Checking (and, with @ref error_handler_t::strict,
// rejecting, or with `replace`/`ignore`, sanitizing) is opt-in via // rejecting, or with `replace`/`ignore`, sanitizing) is opt-in via
// @ref error_handler, applied by @ref check_string_utf8 at the call // @ref error_handler, applied once the whole string (all chunks of
// site to the whole string or, for an indefinite-length CBOR string, // an indefinite-length CBOR string included) has been assembled, by
// to each chunk on its own (RFC 8949, Section 3.2.3). // @ref check_string_utf8 at the call site.
return get_bytes(len, "string", result); return get_bytes(len, "string", result);
} }
@@ -21971,44 +21899,6 @@ class binary_reader
std::size_t bon8_pushback_size = 0; std::size_t bon8_pushback_size = 0;
JSON_PRIVATE_UNLESS_TESTED: JSON_PRIVATE_UNLESS_TESTED:
/*!
@brief check whether @a marker is a type marker that may follow '$'
UBJSON: Z N T F i U I l L d D H C S [ {; BJData: the numeric markers and C
(BJData's excluded markers are rejected separately).
*/
bool is_valid_optimized_type(const char_int_type marker) const noexcept
{
switch (marker)
{
case 'i':
case 'U':
case 'I':
case 'l':
case 'L':
case 'd':
case 'D':
case 'C':
return true;
case 'Z':
case 'N':
case 'T':
case 'F':
case 'H':
case 'S':
case '[':
case '{':
return input_format != input_format_t::bjdata;
case 'B':
case 'u':
case 'm':
case 'M':
case 'h':
return input_format == input_format_t::bjdata;
default:
return false;
}
}
/*! /*!
@brief whether @a marker is excluded from BJData's optimized ND-array types @brief whether @a marker is excluded from BJData's optimized ND-array types
@return whether @a marker is one of 'F', 'H', 'N', 'S', 'T', 'Z', '[', '{' @return whether @a marker is one of 'F', 'H', 'N', 'S', 'T', 'Z', '[', '{'
+1386 -434
View File
File diff suppressed because it is too large. Load diff
+2 -59
View File
@@ -3134,7 +3134,7 @@ TEST_CASE("BJData")
std::vector<uint8_t> const v1 = {'[', '#', 'i', 0xF1}; std::vector<uint8_t> const v1 = {'[', '#', 'i', 0xF1};
std::vector<uint8_t> const v2 = {'[', '$', 'I', '#', 'i', 0xF2}; std::vector<uint8_t> const v2 = {'[', '$', 'I', '#', 'i', 0xF2};
std::vector<uint8_t> const v3 = {'[', '$', 'I', '#', '[', 'i', 0xF4, 'i', 0x02, ']'}; std::vector<uint8_t> const v3 = {'[', '$', 'I', '#', '[', 'i', 0xF4, 'i', 0x02, ']'};
std::vector<uint8_t> const v4 = {'[', '$', 'l', '#', 'i', 0xF7}; std::vector<uint8_t> const v4 = {'[', '$', 0xF6, '#', 'i', 0xF7};
std::vector<uint8_t> const v5 = {'[', '$', 'I', '#', '[', 'i', 0xF5, 'i', 0xF1, ']'}; std::vector<uint8_t> const v5 = {'[', '$', 'I', '#', '[', 'i', 0xF5, 'i', 0xF1, ']'};
std::vector<uint8_t> const v6 = {'[', '#', '[', 'i', 0xF3, 'i', 0x02, ']'}; std::vector<uint8_t> const v6 = {'[', '#', '[', 'i', 0xF3, 'i', 0x02, ']'};
@@ -3455,7 +3455,7 @@ TEST_CASE("BJData")
CHECK(json::from_bjdata(v0, true, false).is_discarded()); CHECK(json::from_bjdata(v0, true, false).is_discarded());
std::vector<uint8_t> const vi = {'[', '$', '#'}; std::vector<uint8_t> const vi = {'[', '$', '#'};
CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vi), "[json.exception.parse_error.112] parse error at byte 3: syntax error while parsing BJData type: marker 0x23 is not a permitted optimized array type", json::parse_error&); CHECK_THROWS_WITH_AS(_ = json::from_bjdata(vi), "[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing BJData value: unexpected end of input", json::parse_error&);
CHECK(json::from_bjdata(vi, true, false).is_discarded()); CHECK(json::from_bjdata(vi, true, false).is_discarded());
std::vector<uint8_t> const vU = {'[', '$', 'U'}; std::vector<uint8_t> const vU = {'[', '$', 'U'};
@@ -4622,60 +4622,3 @@ TEST_CASE("BJData large strings and binaries (chunked reader)")
} }
} }
} }
TEST_CASE("BJData optimized container type marker is validated for count 0")
{
// the '$' type marker must be a legal marker even if no element follows
SECTION("invalid marker")
{
// 'Z', 'T', 'F', 'N', 'S', 'H', '[' and '{' are excluded in BJData
for (const char marker :
{'x', 'E', 'Z', 'T', 'F', 'N', 'S', 'H', '[', '{'
})
{
for (const char open :
{'[', '{'
})
{
for (const int count :
{
0, 1
})
{
CAPTURE(marker)
CAPTURE(open)
CAPTURE(count)
const std::vector<std::uint8_t> v = {static_cast<std::uint8_t>(open), '$', static_cast<std::uint8_t>(marker), '#', 'i', static_cast<std::uint8_t>(count)};
json _;
CHECK_THROWS_AS(_ = json::from_bjdata(v), json::parse_error&);
CHECK(json::from_bjdata(v, true, false).is_discarded());
}
}
}
}
SECTION("error message")
{
const std::vector<std::uint8_t> v = {'[', '$', 'x', '#', 'i', 0};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_bjdata(v), "[json.exception.parse_error.112] parse error at byte 3: syntax error while parsing BJData type: marker 0x78 is not a permitted optimized array type", json::parse_error&);
}
SECTION("valid marker")
{
for (const char marker :
{'i', 'U', 'I', 'u', 'l', 'm', 'L', 'M', 'h', 'd', 'D', 'C'
})
{
CAPTURE(marker)
const std::vector<std::uint8_t> va = {'[', '$', static_cast<std::uint8_t>(marker), '#', 'i', 0};
const std::vector<std::uint8_t> vo = {'{', '$', static_cast<std::uint8_t>(marker), '#', 'i', 0};
CHECK(json::from_bjdata(va) == json::array());
CHECK(json::from_bjdata(vo) == json::object());
}
// an optimized array of bytes is read as a binary value
const std::vector<std::uint8_t> vb = {'[', '$', 'B', '#', 'i', 0};
CHECK(json::from_bjdata(vb) == json::binary({}));
}
}
+3 -148
View File
@@ -1807,93 +1807,6 @@ TEST_CASE("CBOR")
CHECK(json::from_cbor(std::vector<uint8_t>({0xa1, 0xff, 0x01}), true, false).is_discarded()); CHECK(json::from_cbor(std::vector<uint8_t>({0xa1, 0xff, 0x01}), true, false).is_discarded());
} }
SECTION("tag in front of a key")
{
const auto ignore = json::cbor_tag_handler_t::ignore;
const auto store = json::cbor_tag_handler_t::store;
const json expected = {{"a", 10}};
// the tags are skipped in front of a text string key, like in
// front of a value
const std::vector<std::vector<std::uint8_t>> tagged =
{
{0xA1, 0xC0, 0x61, 0x61, 0x0A}, // tag in the head
{0xA1, 0xD8, 0x05, 0x61, 0x61, 0x0A}, // 1-byte tag number
{0xA1, 0xD9, 0x01, 0x2C, 0x61, 0x61, 0x0A}, // 2-byte tag number
{0xA1, 0xDA, 0x00, 0x01, 0x00, 0x00, 0x61, 0x61, 0x0A}, // 4-byte tag number
{0xA1, 0xDB, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x61, 0x61, 0x0A}, // 8-byte tag number
{0xA1, 0xC0, 0xD8, 0x05, 0xC1, 0x61, 0x61, 0x0A}, // chained tags
{0xBF, 0xC0, 0x61, 0x61, 0x0A, 0xFF}, // indefinite-length map
{0xBF, 0x61, 0x62, 0x01, 0xC0, 0x61, 0x61, 0x0A, 0xFF}, // second key tagged
{0xA1, 0xC0, 0x7F, 0x61, 0x61, 0xFF, 0x0A}, // indefinite-length text key
{0xA1, 0xC0, 0x61, 0x61, 0xC0, 0x0A}, // tagged key and value
};
for (const auto& input : tagged)
{
CAPTURE(input)
json const j_ignore = json::from_cbor(input, true, true, ignore);
json const j_store = json::from_cbor(input, true, true, store);
CHECK(j_ignore.at("a") == 10);
CHECK(j_store.at("a") == 10);
CHECK(j_ignore == j_store);
}
CHECK(json::from_cbor(tagged[0], true, true, ignore) == expected);
CHECK(json::from_cbor(tagged[1], true, true, store) == expected);
// nested maps
CHECK(json::from_cbor(std::vector<std::uint8_t>({0xA1, 0xC0, 0x61, 0x61, 0xA1, 0xD8, 0x05, 0x61, 0x62, 0x01}), true, true, ignore) == json{{"a", {{"b", 1}}}});
// the tag handler error still rejects every tag
for (const auto& input : tagged)
{
CAPTURE(input)
json _;
CHECK_THROWS_AS(_ = json::from_cbor(input), json::parse_error&);
}
{
json _;
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<std::uint8_t>({0xA1, 0xD8, 0x05, 0x61, 0x61, 0x0A})), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing CBOR object key: only string keys are supported, but found a tag; last byte: 0xD8", json::parse_error&);
}
// a tag does not turn a key of another type into a text string
{
json _;
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<std::uint8_t>({0xA1, 0xC0, 0x01, 0x01}), true, true, ignore), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing CBOR object key: only string keys are supported, but found an unsigned integer; last byte: 0x01", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<std::uint8_t>({0xA1, 0xD8, 0x05, 0x41, 0x61, 0x01}), true, true, store), "[json.exception.parse_error.113] parse error at byte 4: syntax error while parsing CBOR object key: only string keys are supported, but found a byte string; last byte: 0x41", json::parse_error&);
CHECK(json::from_cbor(std::vector<std::uint8_t>({0xA1, 0xC0, 0x01, 0x01}), true, false, ignore).is_discarded());
}
// end of input after the tag, or inside the tag number
{
json _;
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<std::uint8_t>({0xA1, 0xC0}), true, true, ignore), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing CBOR string: unexpected end of input", json::parse_error&);
CHECK_THROWS_AS(_ = json::from_cbor(std::vector<std::uint8_t>({0xA1, 0xD8}), true, true, ignore), json::parse_error&);
CHECK_THROWS_AS(_ = json::from_cbor(std::vector<std::uint8_t>({0xA1, 0xDB, 0x00, 0x00}), true, true, store), json::parse_error&);
CHECK_THROWS_AS(_ = json::from_cbor(std::vector<std::uint8_t>({0xBF, 0xC0}), true, true, store), json::parse_error&);
}
// SAX: the key event is reported without the tag (start_object,
// key, number_unsigned, end_object are four events)
{
const std::vector<std::uint8_t> input = {0xA1, 0xC0, 0x61, 0x61, 0x0A};
SaxCountdown four(4);
CHECK(json::sax_parse(input, &four, json::input_format_t::cbor, true, false, false, ignore));
SaxCountdown three(3);
CHECK(!json::sax_parse(input, &three, json::input_format_t::cbor, true, false, false, ignore));
SaxCountdown error_handler(4);
CHECK(!json::sax_parse(input, &error_handler, json::input_format_t::cbor));
}
// a long chain of tags needs no recursion
{
std::vector<std::uint8_t> input = {0xA1};
input.insert(input.end(), 100000, 0xC0);
input.insert(input.end(), {0x61, 0x61, 0x0A});
CHECK(json::from_cbor(input, true, true, ignore) == expected);
}
}
SECTION("non-string key (see #2766 and #3381)") SECTION("non-string key (see #2766 and #3381)")
{ {
// only text strings map to JSON object keys; any other key is // only text strings map to JSON object keys; any other key is
@@ -2002,10 +1915,9 @@ TEST_CASE("CBOR")
{ {
json _; json _;
// with the default error handler keep, the chunks are concatenated // the chunks are concatenated as is, without checking that each
// as is, without checking that each chunk is valid UTF-8 on its // chunk is valid UTF-8 on its own (RFC 8949, Section 3.2.3), so
// own (RFC 8949, Section 3.2.3), so a code point split across two // a code point split across two chunks yields a valid string
// chunks yields a valid string
CHECK_NOTHROW(_ = json::from_cbor(std::vector<uint8_t>({0x7f, 0x61, 0xc3, 0x61, 0xa9, 0xff}))); CHECK_NOTHROW(_ = json::from_cbor(std::vector<uint8_t>({0x7f, 0x61, 0xc3, 0x61, 0xa9, 0xff})));
CHECK(_ == "\xc3\xa9"); CHECK(_ == "\xc3\xa9");
CHECK(_.dump() == "\"\xc3\xa9\""); CHECK(_.dump() == "\"\xc3\xa9\"");
@@ -2025,63 +1937,6 @@ TEST_CASE("CBOR")
CHECK(json::from_cbor(std::vector<uint8_t>({0x7f, 0x62, 0xc3, 0xa9, 0x62, 0xc3, 0xb6, 0xff})) == "\xc3\xa9\xc3\xb6"); CHECK(json::from_cbor(std::vector<uint8_t>({0x7f, 0x62, 0xc3, 0xa9, 0x62, 0xc3, 0xb6, 0xff})) == "\xc3\xa9\xc3\xb6");
} }
SECTION("ill-formed UTF-8 in a chunk of an indefinite-length string (RFC 8949, Section 3.2.3)")
{
const auto strict = json::error_handler_t::strict;
const auto replace = json::error_handler_t::replace;
const auto ignore = json::error_handler_t::ignore;
const auto keep = json::error_handler_t::keep;
const auto tags = json::cbor_tag_handler_t::error;
// a code point split over two chunks
const std::vector<std::uint8_t> split = {0x7f, 0x61, 0xc3, 0x61, 0xa9, 0xff};
// a 4-byte code point split over two chunks
const std::vector<std::uint8_t> split_emoji = {0x7f, 0x62, 0xf0, 0x9f, 0x62, 0x98, 0x80, 0xff};
// the same in an object key
const std::vector<std::uint8_t> split_key = {0xa1, 0x7f, 0x61, 0xc3, 0x61, 0xa9, 0xff, 0x01};
// a valid chunk followed by one with a truncated code point
const std::vector<std::uint8_t> truncated = {0x7f, 0x62, 0x41, 0xc3, 0x61, 0xa9, 0xff};
// strict: the string is invalid
for (const auto& input :
{
split, split_emoji, split_key, truncated
})
{
CAPTURE(input)
json _;
CHECK_THROWS_AS(_ = json::from_cbor(input, true, true, tags, strict), json::parse_error&);
CHECK(json::from_cbor(input, true, false, tags, strict).is_discarded());
}
{
json _;
CHECK_THROWS_WITH_AS(_ = json::from_cbor(split, true, true, tags, strict), "[json.exception.parse_error.113] parse error at byte 3: syntax error while parsing CBOR string: invalid string: ill-formed UTF-8 byte", json::parse_error&);
CHECK_THROWS_WITH_AS(_ = json::from_cbor(split_key, true, true, tags, strict), "[json.exception.parse_error.113] parse error at byte 4: syntax error while parsing CBOR key: invalid string: ill-formed UTF-8 byte", json::parse_error&);
}
// replace and ignore: every chunk is sanitized on its own
CHECK(json::from_cbor(split, true, true, tags, replace) == "\xEF\xBF\xBD\xEF\xBF\xBD");
CHECK(json::from_cbor(split, true, true, tags, ignore) == "");
CHECK(json::from_cbor(truncated, true, true, tags, replace) == "A\xEF\xBF\xBD\xEF\xBF\xBD");
CHECK(json::from_cbor(truncated, true, true, tags, ignore) == "A");
CHECK(json::from_cbor(split_key, true, true, tags, replace) == json{{"\xEF\xBF\xBD\xEF\xBF\xBD", 1}});
// keep: unchanged, the chunks are concatenated as is
CHECK(json::from_cbor(split, true, true, tags, keep) == "\xc3\xa9");
CHECK(json::from_cbor(split_emoji, true, true, tags, keep) == "\xf0\x9f\x98\x80");
// conforming input is unaffected by the error handler
const std::vector<std::uint8_t> valid = {0x7f, 0x62, 0xc3, 0xa9, 0x64, 0xf0, 0x9f, 0x98, 0x80, 0x60, 0xff};
for (const auto handler :
{
strict, replace, ignore, keep
})
{
CHECK(json::from_cbor(valid, true, true, tags, handler) == "\xc3\xa9\xf0\x9f\x98\x80");
CHECK(json::from_cbor(std::vector<std::uint8_t>({0x7f, 0xff}), true, true, tags, handler) == "");
}
}
SECTION("many chunks in indefinite-length string") SECTION("many chunks in indefinite-length string")
{ {
constexpr std::size_t chunks = 100000; constexpr std::size_t chunks = 100000;
+364
View File
@@ -0,0 +1,364 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#include "doctest_compatibility.h"
#include <nlohmann/json_view.hpp>
using nlohmann::json;
using nlohmann::ordered_json;
using nlohmann::json_document;
using nlohmann::ordered_json_editable_document;
using nlohmann::json_view;
using nlohmann::ordered_json_document;
#include "json_view_test_helpers.hpp"
using json_view_test::generator;
#include <array>
#include <cstddef>
#include <string>
#include <vector>
// These tests were split off unit-json_view_dump.cpp, whose object file got too
// large for the MinGW assembler ("file too big"). Like that file, this one is
// also built with C++17, as it mentions JSON_HAS_CPP_17.
TEST_CASE("json_view comparison")
{
SECTION("equality of the values parse() produces")
{
generator g;
std::vector<std::string> texts;
for (int i = 0; i < 600; ++i)
{
std::string text;
g.value(text, 0);
texts.push_back(text);
// the same value written differently: sorted keys, canonical numbers
texts.push_back(json::parse(text).dump(1));
}
for (std::size_t i = 0; i + 2 < texts.size(); ++i)
{
for (std::size_t k = i; k < i + 3; ++k)
{
CAPTURE(texts[i])
CAPTURE(texts[k])
const json_document a = json_document::parse(texts[i]);
const json_document b = json_document::parse(texts[k]);
const json ja = json::parse(texts[i]);
const json jb = json::parse(texts[k]);
CHECK((a.root() == b.root()) == (ja == jb));
CHECK((a.root() != b.root()) == (ja != jb));
CHECK((a.root() == jb) == (ja == jb));
CHECK((jb == a.root()) == (ja == jb));
CHECK((a.root() != jb) == (ja != jb));
CHECK((jb != a.root()) == (ja != jb));
// ordered_json compares members in order
const ordered_json_document oa = ordered_json_document::parse(texts[i]);
const ordered_json_document ob = ordered_json_document::parse(texts[k]);
const ordered_json oja = ordered_json::parse(texts[i]);
const ordered_json ojb = ordered_json::parse(texts[k]);
CHECK((oa.root() == ob.root()) == (oja == ojb));
CHECK((oa.root() == ojb) == (oja == ojb));
}
}
}
SECTION("objects whose first keys differ or repeat")
{
const std::array<const char*, 17> texts =
{
{
R"({"a":1,"b":2})", R"({"a":1,"c":2})", R"({"b":1,"a":2})", R"({"a":1,"a":2,"b":3})", R"({"a":2,"b":3})",
R"({"a":1,"b":1,"a":2})", R"({"a":2,"b":1})", R"({"a":1,"a":2})", R"({"b":1})", R"({"a":1,"b":2,"c":3,"d":4,"e":5})",
R"({"a":1,"b":2,"c":3,"d":4,"f":5})", R"({"a":1,"b":2,"c":3,"a":4,"e":5})", R"({"a":4,"b":2,"c":3,"e":5})",
R"({"\u0061":1,"b":2})", R"({})", R"([{"a":1,"b":2}])", R"([{"a":1,"c":2}])"
}
};
for (const char* const x : texts)
{
for (const char* const y : texts)
{
CAPTURE(x)
CAPTURE(y)
const ordered_json_document dx = ordered_json_document::parse(x);
const ordered_json_document dy = ordered_json_document::parse(y);
const ordered_json_editable_document ex = ordered_json_editable_document::parse(x);
const ordered_json jx = ordered_json::parse(x);
const ordered_json jy = ordered_json::parse(y);
CHECK((dx.root() == dy.root()) == (jx == jy));
CHECK((dx.root() == jy) == (jx == jy));
CHECK((ex.root() == dy.root()) == (jx == jy));
}
}
}
SECTION("objects with duplicate keys, small and large")
{
// n members with keys in descending order; with duplicates, some keys
// repeat later (the first after unique ones in key order); in reverse,
// the members are in ascending key order
const auto object = [](int n, bool duplicates, bool reverse, int value)
{
std::string s = "{";
for (int i = 0; i < n; ++i)
{
const int k = reverse ? i : n - 1 - i;
s += (i != 0 ? ",\"k" : "\"k") + std::to_string(100 + k) + "\":" + std::to_string(value + k);
}
if (duplicates)
{
for (int k = n / 2; k < n; k += 3)
{
s += ",\"k" + std::to_string(100 + k) + "\":" + std::to_string(value + k);
}
}
return s + "}";
};
std::vector<std::string> texts;
for (const int n :
{
3, 16, 17, 40
})
{
for (const int flags :
{
0, 1, 2, 3
})
{
texts.push_back(object(n, (flags & 1) != 0, (flags & 2) != 0, 0));
}
texts.push_back(object(n, true, false, 1)); // other values
}
// many objects in one comparison (working memory reused across objects)
std::string array = "[";
std::string array_other = "[";
for (std::size_t i = 0; i < texts.size(); ++i)
{
array += (i != 0 ? "," : "") + texts[i];
array_other += (i != 0 ? "," : "") + texts[texts.size() - 1 - i];
}
texts.push_back(array + "]");
texts.push_back(array_other + "]");
for (const std::string& x : texts)
{
for (const std::string& y : texts)
{
CAPTURE(x)
CAPTURE(y)
const ordered_json_document dx = ordered_json_document::parse(x);
const ordered_json_document dy = ordered_json_document::parse(y);
const ordered_json jx = ordered_json::parse(x);
const ordered_json jy = ordered_json::parse(y);
CHECK((dx.root() == dy.root()) == (jx == jy));
CHECK((dx.root() == jy) == (jx == jy));
const json_document ux = json_document::parse(x);
const json uy = json::parse(y);
CHECK((ux.root() == uy) == (json::parse(x) == uy));
}
}
}
SECTION("numbers, duplicate keys, member order")
{
const auto same = [](const char* x, const char* y)
{
const json_document dx = json_document::parse(x);
const json_document dy = json_document::parse(y);
return dx.root() == dy.root();
};
CHECK(same("1", "1.0"));
CHECK(same("[1, -1, 2.5]", "[1.0, -1.0, 25e-1]"));
CHECK(!same("1", "1.5"));
CHECK(same("18446744073709551615", "18446744073709551615"));
CHECK(same(R"({"a": 1, "a": 2})", R"({"a": 2})"));
CHECK(!same(R"({"a": 1, "a": 2})", R"({"a": 1})"));
CHECK(same(R"({"a": 1, "b": 2})", R"({"b": 2, "a": 1})"));
CHECK(!same(R"({"a": 1})", R"({"a": 1, "b": 2})"));
CHECK(!same("[1, 2]", "[2, 1]"));
CHECK(!same("\"a\"", "\"b\""));
CHECK(same("\"\\u00e9\"", "\"\xc3\xa9\""));
CHECK(!same("null", "false"));
CHECK(!same("[]", "{}"));
const ordered_json_document dup = ordered_json_document::parse(R"({"a": 1, "b": 2, "a": 3})");
const ordered_json_document last = ordered_json_document::parse(R"({"a": 3, "b": 2})");
CHECK(dup.root() == last.root());
// (== compares as parse() resolves duplicates, a lookup finds the first)
CHECK(dup.root()["a"].materialize() == 1);
CHECK(dup.root()["a"] != last.root()["a"]);
const ordered_json_document ab = ordered_json_document::parse(R"({"a": 1, "b": 2})");
const ordered_json_document ba = ordered_json_document::parse(R"({"b": 2, "a": 1})");
CHECK(ab.root() != ba.root());
// discarded values compare as basic_json's do
const json discarded(json::value_t::discarded);
CHECK((json_view() == json_view()) == (discarded == discarded)); // NOLINT(readability-container-size-empty): operator== is tested
CHECK((json_view() == discarded) == (discarded == discarded)); // NOLINT(readability-container-size-empty)
const json_document null_document = json_document::parse("null");
CHECK(!(json_view() == null_document.root())); // NOLINT(readability-container-size-empty)
CHECK(!(null_document.root() == discarded));
}
SECTION("values that are the same text, or the same value written differently, at any depth")
{
// nested arrays and objects around depths where the walk over equal text gives up
for (const int depth :
{
1, 5, 47, 48, 49, 50, 60, 300
})
{
CAPTURE(depth)
std::string head;
std::string tail;
for (int i = 0; i < depth; ++i)
{
head += i % 2 == 0 ? R"({"k":[1,"s",{"x":true},)" : R"([null,2.5,{"y":)";
tail.insert(0, i % 2 == 0 ? "]}" : "}]");
}
// the same text, one number changed, and one number written differently
const auto with_number = [&](const char* number)
{
std::string result = head;
result += number;
result += tail;
return result;
};
const std::string text = with_number("7");
const std::string changed = with_number("8");
const std::string spelled = with_number("7.0");
const json_document a = json_document::parse(text);
const json_document b = json_document::parse(text);
const json_document c = json_document::parse(changed);
const json_document d = json_document::parse(spelled);
CHECK(a.root() == b.root());
CHECK(a.root() != c.root());
CHECK(c.root() != a.root());
CHECK(a.root() == d.root());
CHECK(d.root() == a.root());
CHECK(a.root() == json::parse(text));
CHECK(a.root() != json::parse(changed));
CHECK(json::parse(spelled) == a.root());
const nlohmann::json_editable_document e = nlohmann::json_editable_document::parse(text);
CHECK(a.root() == e.root());
CHECK(e.root() == a.root());
CHECK(c.root() != e.root());
CHECK(e.root() == e.root());
const ordered_json_document oa = ordered_json_document::parse(text);
const ordered_json_document oc = ordered_json_document::parse(changed);
CHECK(oa.root() != oc.root());
CHECK(oa.root() == ordered_json::parse(text));
CHECK(oc.root() != ordered_json::parse(text));
}
// not the same text, but the same value
const auto same = [](const char* x, const char* y)
{
const json_document dx = json_document::parse(x);
const json_document dy = json_document::parse(y);
const bool equal = dx.root() == dy.root();
CHECK(equal == (json::parse(x) == json::parse(y)));
CHECK(equal == (dx.root() == json::parse(y)));
return equal;
};
CHECK(same(R"({"a":1,"a":2})", R"({"a":3,"a":2})"));
CHECK(same(R"({"a":1,"b":[1,{"c":2,"c":3}]})", R"({"b":[1,{"c":9,"c":3}],"a":1})"));
CHECK(!same(R"({"a":1,"a":2})", R"({"a":2,"a":3})"));
CHECK(same("[1.0,-0,0,1e2]", "[1.00,0,-0,100.0]"));
CHECK(same(R"(["\u00e9","\n"])", "[\"\xc3\xa9\",\"\\n\"]"));
CHECK(!same(R"({"a":[1,2,{"b":null}]})", R"({"a":[1,2,{"b":false}]})"));
CHECK(!same(R"({"a":[1,2,{"b":null}]})", R"({"a":[1,2,{"c":null}]})"));
CHECK(!same(R"({"a":[1,2,{"b":null}]})", R"({"a":[1,2,{"b":null,"c":1}]})"));
// edited documents: moved sequences, links, and values of new nodes
nlohmann::json_editable_document edited = nlohmann::json_editable_document::parse(R"({"a":[1,2,3],"b":{"c":"d"},"e":0.5})");
const json_document plain = json_document::parse(R"({"a":[1,2,3,4],"b":{"c":"x","y":null},"e":0.5,"z":[]})");
edited.push_back(edited.root()["a"], 4);
edited.set(edited.root()["b"], "c", "x");
edited.set(edited.root()["b"], "y", nullptr);
edited.set(edited.root(), "z", json::array());
CHECK(edited.root() == plain.root());
CHECK(plain.root() == edited.root());
edited.erase(edited.root()["a"], 0);
CHECK(edited.root() != plain.root());
CHECK(edited.root()["a"] == json::parse("[2,3,4]"));
const json_document tail = json_document::parse("[2,3,4]");
CHECK(edited.root()["a"] == tail.root());
}
SECTION("large documents, with a difference at any place")
{
const auto make = [](std::size_t count, std::size_t changed, bool swap_members)
{
std::string text = R"({"b":1,"list":[)";
for (std::size_t i = 0; i < count; ++i)
{
text += (i != 0 ? "," : "");
const std::string id = std::to_string(i == changed ? i + 1 : i);
if (swap_members && i % 3 == 0)
{
text += R"({"v":[1.5,"s",{"k":)";
text += id;
text += R"(}],"id":)";
text += id;
text += "}";
}
else
{
text += R"({"id":)";
text += id;
text += R"(,"v":[1.5,"s",{"k":)";
text += id;
text += "}]}";
}
}
return text + R"(],"a":{"y":2,"x":3}})";
};
const std::size_t count = 3000;
for (const std::size_t changed :
{
count, std::size_t{0}, std::size_t{1}, count / 2, count - 1
})
{
for (const bool swap :
{
false, true
})
{
CAPTURE(changed)
CAPTURE(swap)
const std::string text_a = make(count, count, false);
const std::string text_b = make(count, changed, swap);
const json_document a = json_document::parse(text_a);
const json_document b = json_document::parse(text_b);
const bool equal = changed == count;
CHECK((a.root() == b.root()) == equal);
CHECK((b.root() == a.root()) == equal);
CHECK((a.root() == json::parse(text_b)) == equal);
CHECK((json::parse(text_b) == a.root()) == equal);
CHECK((a.root()["list"] == b.root()["list"]) == equal);
const ordered_json_document oa = ordered_json_document::parse(text_a);
const ordered_json_document ob = ordered_json_document::parse(text_b);
CHECK((oa.root() == ob.root()) == (equal && !swap));
CHECK((oa.root() == ordered_json::parse(text_b)) == (equal && !swap));
}
}
}
SECTION("deep nesting")
{
const std::string deep = std::string(100000, '[') + std::string(100000, ']');
const json_document a = json_document::parse(deep);
const json_document b = json_document::parse(deep);
CHECK(a.root() == b.root());
CHECK(a.root() == json::parse(deep));
const std::string other = std::string(100000, '[') + "1" + std::string(100000, ']');
const json_document c = json_document::parse(other);
CHECK(a.root() != c.root());
}
}
-95
View File
@@ -432,101 +432,6 @@ TEST_CASE("json_view dump")
} }
} }
TEST_CASE("json_view comparison")
{
SECTION("equality of the values parse() produces")
{
generator g;
std::vector<std::string> texts;
for (int i = 0; i < 600; ++i)
{
std::string text;
g.value(text, 0);
texts.push_back(text);
// the same value written differently: sorted keys, canonical numbers
texts.push_back(json::parse(text).dump(1));
}
for (std::size_t i = 0; i + 2 < texts.size(); ++i)
{
for (std::size_t k = i; k < i + 3; ++k)
{
CAPTURE(texts[i])
CAPTURE(texts[k])
const json_document a = json_document::parse(texts[i]);
const json_document b = json_document::parse(texts[k]);
const json ja = json::parse(texts[i]);
const json jb = json::parse(texts[k]);
CHECK((a.root() == b.root()) == (ja == jb));
CHECK((a.root() != b.root()) == (ja != jb));
CHECK((a.root() == jb) == (ja == jb));
CHECK((jb == a.root()) == (ja == jb));
CHECK((a.root() != jb) == (ja != jb));
CHECK((jb != a.root()) == (ja != jb));
// ordered_json compares members in order
const ordered_json_document oa = ordered_json_document::parse(texts[i]);
const ordered_json_document ob = ordered_json_document::parse(texts[k]);
const ordered_json oja = ordered_json::parse(texts[i]);
const ordered_json ojb = ordered_json::parse(texts[k]);
CHECK((oa.root() == ob.root()) == (oja == ojb));
CHECK((oa.root() == ojb) == (oja == ojb));
}
}
}
SECTION("numbers, duplicate keys, member order")
{
const auto same = [](const char* x, const char* y)
{
const json_document dx = json_document::parse(x);
const json_document dy = json_document::parse(y);
return dx.root() == dy.root();
};
CHECK(same("1", "1.0"));
CHECK(same("[1, -1, 2.5]", "[1.0, -1.0, 25e-1]"));
CHECK(!same("1", "1.5"));
CHECK(same("18446744073709551615", "18446744073709551615"));
CHECK(same(R"({"a": 1, "a": 2})", R"({"a": 2})"));
CHECK(!same(R"({"a": 1, "a": 2})", R"({"a": 1})"));
CHECK(same(R"({"a": 1, "b": 2})", R"({"b": 2, "a": 1})"));
CHECK(!same(R"({"a": 1})", R"({"a": 1, "b": 2})"));
CHECK(!same("[1, 2]", "[2, 1]"));
CHECK(!same("\"a\"", "\"b\""));
CHECK(same("\"\\u00e9\"", "\"\xc3\xa9\""));
CHECK(!same("null", "false"));
CHECK(!same("[]", "{}"));
const ordered_json_document dup = ordered_json_document::parse(R"({"a": 1, "b": 2, "a": 3})");
const ordered_json_document last = ordered_json_document::parse(R"({"a": 3, "b": 2})");
CHECK(dup.root() == last.root());
// (== compares as parse() resolves duplicates, a lookup finds the first)
CHECK(dup.root()["a"].materialize() == 1);
CHECK(dup.root()["a"] != last.root()["a"]);
const ordered_json_document ab = ordered_json_document::parse(R"({"a": 1, "b": 2})");
const ordered_json_document ba = ordered_json_document::parse(R"({"b": 2, "a": 1})");
CHECK(ab.root() != ba.root());
// discarded values compare as basic_json's do
const json discarded(json::value_t::discarded);
CHECK((json_view() == json_view()) == (discarded == discarded)); // NOLINT(readability-container-size-empty): operator== is tested
CHECK((json_view() == discarded) == (discarded == discarded)); // NOLINT(readability-container-size-empty)
const json_document null_document = json_document::parse("null");
CHECK(!(json_view() == null_document.root())); // NOLINT(readability-container-size-empty)
CHECK(!(null_document.root() == discarded));
}
SECTION("deep nesting")
{
const std::string deep = std::string(100000, '[') + std::string(100000, ']');
const json_document a = json_document::parse(deep);
const json_document b = json_document::parse(deep);
CHECK(a.root() == b.root());
CHECK(a.root() == json::parse(deep));
const std::string other = std::string(100000, '[') + "1" + std::string(100000, ']');
const json_document c = json_document::parse(other);
CHECK(a.root() != c.root());
}
}
TEST_CASE("json_view large objects") TEST_CASE("json_view large objects")
{ {
// objects with 128 members or more are looked up with a hash index // objects with 128 members or more are looked up with a hash index
+58
View File
@@ -682,6 +682,64 @@ TEST_CASE("json_view images: ownership")
#if !defined(JSON_NOEXCEPTION) #if !defined(JSON_NOEXCEPTION)
TEST_CASE("json_view images: errors") TEST_CASE("json_view images: errors")
{ {
SECTION("float tokens: a changed byte is accepted if the text is still a number with the digits the node records")
{
for (const char* const token :
{
"-12.375", "0.5", "123456.0", "-0.0", "9999999999999999999.5", "0.000000000000000000000001", "1.25e10", "5E-3"
})
{
CAPTURE(token)
const std::string text = std::string("[") + token + "]";
const json_document d = json_document::parse(text);
const std::vector<std::uint8_t> image = d.save();
const std::size_t at = text_at(image) + 1; // the first byte of the token
const std::uint16_t recorded = node_at(image, 1).extra;
CHECK(loaded_dump(image) == json::parse(text).dump());
for (std::size_t pos = 0; token[pos] != '\0'; ++pos)
{
for (unsigned int byte = 0; byte < 256; ++byte)
{
std::vector<std::uint8_t> changed = image;
changed[at + pos] = static_cast<std::uint8_t>(byte);
std::string t = token;
t[pos] = static_cast<char>(byte);
// the digits of the integer part and of the fraction of the changed token
std::size_t integer = 0;
std::size_t fraction = 0;
std::size_t k = (t[0] == '-') ? 1 : 0;
for (; k < t.size() && t[k] >= '0' && t[k] <= '9'; ++k)
{
++integer;
}
if (k < t.size() && t[k] == '.')
{
for (++k; k < t.size() && t[k] >= '0' && t[k] <= '9'; ++k)
{
++fraction;
}
}
// (as a JSON text, the token may have white space around it)
const bool float_token = json::accept(t) && t.find_first_of(std::string(" \t\r\n\0", 5)) == std::string::npos && (t.find('.') != std::string::npos || t.find_first_of("eE") != std::string::npos) // NOLINT(abseil-string-find-str-contains)
&& ((integer | (fraction << 8U)) == recorded);
CAPTURE(pos)
CAPTURE(byte)
std::string result;
try
{
result = loaded_dump(changed);
}
catch (const json::parse_error& e)
{
CHECK(std::string(e.what()) == check_failed);
result = "rejected";
}
CHECK((result != "rejected") == float_token);
}
}
}
}
SECTION("invalid UTF-8 of a decoded string: the compact dump writes it, the others throw") SECTION("invalid UTF-8 of a decoded string: the compact dump writes it, the others throw")
{ {
const json_document d = json_document::parse(R"({"k\n":"\u00e9x"})"); const json_document d = json_document::parse(R"({"k\n":"\u00e9x"})");
+1 -53
View File
@@ -2023,7 +2023,7 @@ TEST_CASE("UBJSON")
CHECK(json::from_ubjson(v0, true, false).is_discarded()); CHECK(json::from_ubjson(v0, true, false).is_discarded());
std::vector<uint8_t> const vi = {'[', '$', '#'}; std::vector<uint8_t> const vi = {'[', '$', '#'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vi), "[json.exception.parse_error.112] parse error at byte 3: syntax error while parsing UBJSON type: marker 0x23 is not a permitted optimized array type", json::parse_error&); CHECK_THROWS_WITH_AS(_ = json::from_ubjson(vi), "[json.exception.parse_error.110] parse error at byte 4: syntax error while parsing UBJSON value: unexpected end of input", json::parse_error&);
CHECK(json::from_ubjson(vi, true, false).is_discarded()); CHECK(json::from_ubjson(vi, true, false).is_discarded());
std::vector<uint8_t> const vT = {'[', '$', 'T'}; std::vector<uint8_t> const vT = {'[', '$', 'T'};
@@ -3348,55 +3348,3 @@ TEST_CASE("UBJSON large strings (chunked reader)")
} }
} }
} }
TEST_CASE("UBJSON optimized container type marker is validated for count 0")
{
// the '$' type marker must be a legal marker even if no element follows
SECTION("invalid marker")
{
for (const char marker :
{'x', 'E', 'B', 'u', 'm', 'M', 'h'
})
{
for (const char open :
{'[', '{'
})
{
for (const int count :
{
0, 1
})
{
CAPTURE(marker)
CAPTURE(open)
CAPTURE(count)
const std::vector<std::uint8_t> v = {static_cast<std::uint8_t>(open), '$', static_cast<std::uint8_t>(marker), '#', 'i', static_cast<std::uint8_t>(count)};
json _;
CHECK_THROWS_AS(_ = json::from_ubjson(v), json::parse_error&);
CHECK(json::from_ubjson(v, true, false).is_discarded());
}
}
}
}
SECTION("error message")
{
const std::vector<std::uint8_t> v = {'[', '$', 'x', '#', 'i', 0};
json _;
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v), "[json.exception.parse_error.112] parse error at byte 3: syntax error while parsing UBJSON type: marker 0x78 is not a permitted optimized array type", json::parse_error&);
}
SECTION("valid marker")
{
for (const char marker :
{'Z', 'N', 'T', 'F', 'i', 'U', 'I', 'l', 'L', 'd', 'D', 'H', 'C', 'S', '[', '{'
})
{
CAPTURE(marker)
const std::vector<std::uint8_t> va = {'[', '$', static_cast<std::uint8_t>(marker), '#', 'i', 0};
const std::vector<std::uint8_t> vo = {'{', '$', static_cast<std::uint8_t>(marker), '#', 'i', 0};
CHECK(json::from_ubjson(va) == json::array());
CHECK(json::from_ubjson(vo) == json::object());
}
}
}