docs: correct and extend the CBOR documentation
Fix the negative-integer rows (they used MessagePack boundaries), the number_integer range of the 0x1B row, and the half-precision entry in the list of unused types; correct the NaN history (0xF9 since 3.8.0) and the tagged-item wording (only the tag directly in front of a byte string is stored, and only in the 0xD8..0xDB form). Document that to_cbor is not RFC 8949 deterministic, that duplicate keys are accepted (last wins), tags in front of keys, per-chunk UTF-8 validation, the exceptions parse_error.113, out_of_range.406/408 of from_cbor, the pointer and C array input traps, the partially written output on exceptions, and the store/ignore results in the binary values example. Drop the claim that the CBOR reader needs StringType::append(const StringType&) and BinaryType::insert(pos, first, last). Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
6 files changed
+76
-24
No files matched your search
@@ -15,11 +15,12 @@ error
|
||||
: report a parse error in case of a tag (the `from_cbor` overloads throw a `parse_error` exception by default)
|
||||
|
||||
ignore
|
||||
: ignore tags
|
||||
: ignore tags (also in front of a text string map key)
|
||||
|
||||
store
|
||||
: store tagged byte strings (for bytes 0xd8..0xdb) as binary values with the tag as subtype; other tagged values are
|
||||
read as if the tag were ignored. If several tags precede a byte string, only the innermost one is stored.
|
||||
read as if the tag were ignored. If several tags precede a byte string, only the tag directly in front of it is stored, and only when it has the form
|
||||
0xd8..0xdb (a tag 0..23 written as 0xc0..0xd7 is skipped). Tags in front of a text string map key are skipped.
|
||||
|
||||
## Examples
|
||||
|
||||
|
||||
@@ -32,8 +32,10 @@ The exact mapping and its limitations are described on a [dedicated page](../../
|
||||
|
||||
- an `std::istream` object
|
||||
- a `FILE` pointer
|
||||
- a C-style array of characters
|
||||
- a pointer to a null-terminated string of single byte characters
|
||||
- a C-style array of characters (its terminating NUL is read as trailing data: `parse_error.110` unless `strict` is
|
||||
`false`)
|
||||
- a pointer to a null-terminated string of single byte characters (read up to the first 0x00 byte, which is a valid
|
||||
CBOR byte)
|
||||
- a container `obj` for which `begin(obj)` and `end(obj)` produce a valid pair of iterators
|
||||
(as found via ADL or member functions, with semantics compatible to `std::begin` and `std::end`)
|
||||
|
||||
@@ -89,8 +91,14 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
|
||||
- Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if unsupported features from CBOR were
|
||||
used in the given input or if the input is not valid CBOR
|
||||
- Throws [parse_error.113](../../home/exceptions.md#jsonexceptionparse_error113) if a map key is not a string (keys of
|
||||
other types are not supported, as JSON object keys are always strings), or if a string value or object key is not
|
||||
valid UTF-8 and `error_handler` is `strict`
|
||||
other types are not supported, as JSON object keys are always strings), if the head of a string or byte string is
|
||||
malformed (including a chunk of the wrong type or an indefinite-length string nested inside an indefinite-length
|
||||
string), or if a string value or object key (or one chunk of an indefinite-length string) is not valid UTF-8 and
|
||||
`error_handler` is `strict`
|
||||
- Throws [out_of_range.408](../../home/exceptions.md#jsonexceptionout_of_range408) if an array or map announces
|
||||
more elements than `std::size_t` can hold (for example `9B FF FF FF FF FF FF FF FF`)
|
||||
- Throws [out_of_range.406](../../home/exceptions.md#jsonexceptionout_of_range406) if a floating-point number does not
|
||||
fit into `number_float_t`
|
||||
|
||||
## Complexity
|
||||
|
||||
|
||||
@@ -42,7 +42,9 @@ The exact mapping and its limitations are described on a [dedicated page](../../
|
||||
|
||||
## Exception safety
|
||||
|
||||
Strong guarantee: if an exception is thrown, there are no changes in the JSON value.
|
||||
Strong guarantee: if an exception is thrown, there are no changes in the JSON value. Bytes that were already written to
|
||||
the output adapter are not taken back; for `[1,2,<discarded>]`, three bytes are written before
|
||||
[`type_error.321`](../../home/exceptions.md#jsonexceptiontype_error321) is thrown.
|
||||
|
||||
## Exceptions
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ extremely small code sizes, fairly small message size, and extensibility without
|
||||
|
||||
## Byte layout
|
||||
|
||||
CBOR encodes every data item as a *head* followed by optional content. The first byte of the head holds the *major type* in its upper three bits (unsigned integer, negative integer, byte string, text string, array, map, tag, or simple/float) and an *additional information* value in its lower five bits. Additional information below 24 is the argument itself (for example, a small integer or a short length); the values 24 to 27 announce that the argument follows in 1, 2, 4, or 8 big-endian bytes. Strings, arrays, and maps are prefixed with their length (or count of elements/pairs), and content follows directly, so a reader always knows how many items to expect without a terminator.
|
||||
CBOR encodes every data item as a *head* followed by optional content. The first byte of the head holds the *major type* in its upper three bits (unsigned integer, negative integer, byte string, text string, array, map, tag, or simple/float) and an *additional information* value in its lower five bits. Additional information below 24 is the argument itself (for example, a small integer or a short length); the values 24 to 27 announce that the argument follows in 1, 2, 4, or 8 big-endian bytes. Strings, arrays, and maps are prefixed with their length (or count of elements/pairs), and content follows directly. A reader of definite-length items knows how many items to expect without a terminator; indefinite-length items (additional information 31) are instead closed by the "break" byte 0xFF, which `from_cbor` reads but `to_cbor` never writes.
|
||||
|
||||
Consider the following JSON value (shown formatted for readability):
|
||||
|
||||
@@ -52,16 +52,16 @@ The library uses the following mapping from JSON values types to CBOR types acco
|
||||
| null | `null` | Null | 0xF6 |
|
||||
| boolean | `true` | True | 0xF5 |
|
||||
| boolean | `false` | False | 0xF4 |
|
||||
| number_integer | -9223372036854775808..-2147483649 | Negative integer (8 bytes follow) | 0x3B |
|
||||
| number_integer | -2147483648..-32769 | Negative integer (4 bytes follow) | 0x3A |
|
||||
| number_integer | -32768..-129 | Negative integer (2 bytes follow) | 0x39 |
|
||||
| number_integer | -128..-25 | Negative integer (1 byte follow) | 0x38 |
|
||||
| number_integer | -9223372036854775808..-4294967297 | Negative integer (8 bytes follow) | 0x3B |
|
||||
| number_integer | -4294967296..-65537 | Negative integer (4 bytes follow) | 0x3A |
|
||||
| number_integer | -65536..-257 | Negative integer (2 bytes follow) | 0x39 |
|
||||
| number_integer | -256..-25 | Negative integer (1 byte follow) | 0x38 |
|
||||
| number_integer | -24..-1 | Negative integer | 0x20..0x37 |
|
||||
| number_integer | 0..23 | Integer | 0x00..0x17 |
|
||||
| number_integer | 24..255 | Unsigned integer (1 byte follow) | 0x18 |
|
||||
| number_integer | 256..65535 | Unsigned integer (2 bytes follow) | 0x19 |
|
||||
| number_integer | 65536..4294967295 | Unsigned integer (4 bytes follow) | 0x1A |
|
||||
| number_integer | 4294967296..18446744073709551615 | Unsigned integer (8 bytes follow) | 0x1B |
|
||||
| number_integer | 4294967296..9223372036854775807 | Unsigned integer (8 bytes follow) | 0x1B |
|
||||
| number_unsigned | 0..23 | Integer | 0x00..0x17 |
|
||||
| number_unsigned | 24..255 | Unsigned integer (1 byte follow) | 0x18 |
|
||||
| number_unsigned | 256..65535 | Unsigned integer (2 bytes follow) | 0x19 |
|
||||
@@ -95,7 +95,26 @@ see "binary" cells in the table above.
|
||||
|
||||
!!! success "Complete mapping"
|
||||
|
||||
The mapping is **complete** in the sense that any JSON value type can be converted to a CBOR value.
|
||||
The mapping is **complete** in the sense that any JSON value type can be converted to a CBOR value, with two
|
||||
exceptions: a [`discarded`](../../api/basic_json/is_discarded.md) value cannot be serialized and throws
|
||||
[`type_error.321`](../../home/exceptions.md#jsonexceptiontype_error321), and an integer outside the range of CBOR
|
||||
integers (only possible with a number type wider than 64 bits) throws
|
||||
[`out_of_range.407`](../../home/exceptions.md#jsonexceptionout_of_range407).
|
||||
|
||||
!!! info "Not a deterministic encoding"
|
||||
|
||||
`to_cbor` does not produce the deterministic encoding of
|
||||
[RFC 8949, Section 4.2.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-4.2.1), and not always the preferred
|
||||
serialization of Section 4.1:
|
||||
|
||||
- floating-point numbers are written as single precision (0xFA) when that is exact and as double precision (0xFB)
|
||||
otherwise; half precision (0xF9) is used only for `NaN` and `Infinity`/`-Infinity` (`1.0` is `fa 3f 80 00 00`);
|
||||
- a binary subtype 0..23 is written as `d8 xx` instead of the one-byte tag 0xC0..0xD7 (`d8 01 41 01`);
|
||||
- object members are written in the order of `object_t` (for the default `std::map`: sorted by the key string, not
|
||||
by the encoded key bytes).
|
||||
|
||||
Integers and lengths use the shortest form, and indefinite-length items are never written. Decoders must accept all
|
||||
of this; the RFC does not require the preferred serialization.
|
||||
|
||||
!!! info "NaN/infinity handling"
|
||||
|
||||
@@ -105,9 +124,8 @@ see "binary" cells in the table above.
|
||||
|
||||
!!! note
|
||||
|
||||
Prior to version 3.13.0, NaN and Infinity were instead serialized as a CBOR double-precision float (type 0xFB,
|
||||
9 bytes total), because the check used to select a smaller encoding compared magnitudes with NaN, which is
|
||||
always `false` and caused the intended half-precision path to be skipped.
|
||||
Prior to version 3.8.0, NaN and Infinity were instead serialized as a CBOR double-precision float (type 0xFB,
|
||||
9 bytes total).
|
||||
|
||||
!!! info "Unused CBOR types"
|
||||
|
||||
@@ -124,7 +142,6 @@ see "binary" cells in the table above.
|
||||
- expected conversions (0xD5..0xD7)
|
||||
- simple values (0xE0..0xF3, 0xF8)
|
||||
- undefined (0xF7)
|
||||
- half-precision floats (0xF9)
|
||||
- break (0xFF)
|
||||
|
||||
!!! info "Tagged items"
|
||||
@@ -226,10 +243,20 @@ string of the same major type, as required by [RFC 8949, Section 3.2.3](https://
|
||||
([RFC 8392](https://www.rfc-editor.org/rfc/rfc8392.html)), cannot be read with this library and need a
|
||||
general-purpose CBOR library instead.
|
||||
|
||||
A tag in front of a text string key is handled like a tag in front of a value: it is skipped with
|
||||
`cbor_tag_handler_t::ignore` or `cbor_tag_handler_t::store` and rejected with the default `error`. A tag does not
|
||||
turn a key of another type into a text string.
|
||||
|
||||
A map with duplicate keys is well-formed, but invalid under
|
||||
[RFC 8949, Section 5.4](https://www.rfc-editor.org/rfc/rfc8949.html#section-5.4). `from_cbor` accepts it and the
|
||||
last value wins; the [SAX interface](../parsing/sax_interface.md) reports every occurrence.
|
||||
|
||||
!!! warning "Ill-formed UTF-8 in text strings"
|
||||
|
||||
[RFC 8949, Section 3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-3.1) requires CBOR text strings (major
|
||||
type 3) to be valid UTF-8, but leaves it up to the decoder whether to enforce this, so checking is opt-in: with the
|
||||
type 3) to be valid UTF-8 (a string with ill-formed UTF-8 is "well-formed but invalid"), and
|
||||
[Section 5.3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-5.3.1) leaves it up to the decoder whether to
|
||||
verify this, so checking is opt-in: with the
|
||||
[`error_handler`](../../api/basic_json/from_cbor.md) parameter left at `keep` (the default), `from_cbor()` accepts a
|
||||
text string (object keys included) whose bytes are not valid UTF-8 and hands them back unchanged. Passing
|
||||
`error_handler_t::strict` makes `from_cbor()` check and throw
|
||||
@@ -242,9 +269,14 @@ string of the same major type, as required by [RFC 8949, Section 3.2.3](https://
|
||||
[`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md) is enabled), it throws the same exception
|
||||
instead. Byte strings (major type 2) are unaffected, since they are not required to hold text.
|
||||
|
||||
The chunks of an indefinite-length text string are checked one by one, as required by
|
||||
[RFC 8949, Section 3.2.3](https://www.rfc-editor.org/rfc/rfc8949.html#section-3.2.3): a code point cannot be spread
|
||||
over two chunks. With `error_handler_t::strict` such a string throws `parse_error.113`; `replace` and `ignore`
|
||||
sanitize each chunk on its own.
|
||||
|
||||
!!! warning "Tagged items"
|
||||
|
||||
Tagged items (0xC0..0xDB) will throw a parse error by default. They can be ignored by passing `cbor_tag_handler_t::ignore` to function `from_cbor`, in which case the tag is skipped and the enclosed data item is parsed on its own. Passing `cbor_tag_handler_t::store` to function `from_cbor` stores tagged byte strings (for bytes 0xd8..0xdb) as binary values with the tag as subtype; other tagged values are read as if the tag were ignored. If several tags precede a byte string, only the innermost one is stored. Note that no tag is ever interpreted: for instance, a text string tagged with tag 0 (date/time) stays a string.
|
||||
Tagged items (0xC0..0xDB) will throw a parse error by default. They can be ignored by passing `cbor_tag_handler_t::ignore` to function `from_cbor`, in which case the tag is skipped and the enclosed data item is parsed on its own. Passing `cbor_tag_handler_t::store` to function `from_cbor` stores tagged byte strings (for bytes 0xd8..0xdb) as binary values with the tag as subtype; other tagged values are read as if the tag were ignored. If several tags precede a byte string, only the tag directly in front of it is stored, and only when it has the form 0xD8..0xDB (a tag 0..23 written as 0xC0..0xD7 is skipped); for example, `D8 05 C1 41 AA` yields subtype `null`, whereas `C1 D8 05 41 AA` yields 5. Tags in front of a text string map key are skipped as well. Note that no tag is ever interpreted: for instance, a text string tagged with tag 0 (date/time) stays a string.
|
||||
|
||||
??? example "Example: deserialize a JSON value from CBOR"
|
||||
|
||||
|
||||
@@ -301,6 +301,8 @@ representation using the length of the byte array.
|
||||
[`json::from_cbor`](../api/basic_json/from_cbor.md) (see
|
||||
[`cbor_tag_handler_t`](../api/basic_json/cbor_tag_handler_t.md)).
|
||||
|
||||
With `ignore`, the subtype is dropped (`"subtype": null`); with `store`, the tag is kept (`"subtype": 42`):
|
||||
|
||||
```json
|
||||
{
|
||||
"binary": {
|
||||
@@ -310,6 +312,15 @@ representation using the length of the byte array.
|
||||
}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"binary": {
|
||||
"bytes": [202, 254, 186, 190],
|
||||
"subtype": 42
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### MessagePack
|
||||
|
||||
[MessagePack](binary_formats/messagepack.md) supports binary values and subtypes. If a subtype is given, the ext family
|
||||
|
||||
@@ -368,8 +368,7 @@ using array_t = ArrayType<basic_json, AllocatorType<basic_json>>;
|
||||
`#!cpp std::from_chars` is not available or declines the token, and `std::strtold` reads up to the null character. A type whose `data()`
|
||||
is not null-terminated does not fail to compile; with such a `NumberFloatType` it can silently misparse
|
||||
floating-point numbers.
|
||||
- `append(const char*, size_type)`, used by [`dump`](../../api/basic_json/dump.md), and `append(const StringType&)`,
|
||||
used by the CBOR reader for indefinite-length strings. The library's internal string concatenation additionally has
|
||||
- `append(const char*, size_type)`, used by [`dump`](../../api/basic_json/dump.md) and by the binary readers. The library's internal string concatenation additionally has
|
||||
to append a `#!cpp char` and a `#!cpp const char*`; for each it selects between `append(arg)`, `#!cpp operator+=`,
|
||||
`append(first, last)`, and `append(data, size)`.
|
||||
- The comparison operator `==` against another `StringType`, and `<` for use as a key of the chosen
|
||||
@@ -678,8 +677,7 @@ using binary_t = nlohmann::byte_container_with_subtype<BinaryType>;
|
||||
[`StringType`](#stringtype), whose non-`#!cpp const` `data()` is C++17 and later only.
|
||||
- Default-constructible, copy-constructible, and move-constructible.
|
||||
- Member functions `size()`, `empty()`, `data()`, `resize()`, `operator[]`, `back()`, `begin()`, `end()`, `cbegin()`,
|
||||
and `cend()` with random-access iterators, and `insert(pos, first, last)`, which the CBOR reader uses to join the
|
||||
chunks of an indefinite-length byte string. `push_back()` is **not** required.
|
||||
and `cend()` with random-access iterators. `push_back()` is **not** required.
|
||||
- Comparison operators: `==` is used by
|
||||
[`byte_container_with_subtype`](../../api/byte_container_with_subtype/index.md), the relational operators by
|
||||
[`basic_json`'s comparison operators](../../api/basic_json/operator_le.md).
|
||||
|
||||
Reference in new issue
Block a user