wt/json
1
0 Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-10-27 12:25:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

deploy: 94fe31eb9b

Browse Source
This commit is contained in:
nlohmann committed 2026-10-26 21:24:35 +00:00
1 parent 41c601a7d1
commit d2e264bd84
1400 files changed
+11859 -5440
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

features/binary_formats/ubjson.md
+222 -189
View File
@@ -5,177 +5,11 @@ achieve the generality of JSON, combined with being much easier to process than
!!! abstract "References"
- [UBJSON Website](http://ubjson.org)
- [UBJSON Website](https://ubjson.org)
## Serialization
## Byte layout
The library uses the following mapping from JSON values types to UBJSON types according to the UBJSON specification:
| JSON value type | value/range | UBJSON type | marker |
|-----------------|-------------------------------------------|----------------|--------|
| null | `null` | null | `Z` |
| boolean | `true` | true | `T` |
| boolean | `false` | false | `F` |
| number_integer | -9223372036854775808..-2147483649 | int64 | `L` |
| number_integer | -2147483648..-32769 | int32 | `l` |
| number_integer | -32768..-129 | int16 | `I` |
| number_integer | -128..127 | int8 | `i` |
| number_integer | 128..255 | uint8 | `U` |
| number_integer | 256..32767 | int16 | `I` |
| number_integer | 32768..2147483647 | int32 | `l` |
| number_integer | 2147483648..9223372036854775807 | int64 | `L` |
| number_unsigned | 0..127 | int8 | `i` |
| number_unsigned | 128..255 | uint8 | `U` |
| number_unsigned | 256..32767 | int16 | `I` |
| number_unsigned | 32768..2147483647 | int32 | `l` |
| number_unsigned | 2147483648..9223372036854775807 | int64 | `L` |
| number_unsigned | 9223372036854775808..18446744073709551615 | high-precision | `H` |
| number_float | *any value* | float64 | `D` |
| string | *with shortest length indicator* | string | `S` |
| array | *see notes on optimized format* | array | `[` |
| object | *see notes on optimized format* | map | `{` |
!!! success "Complete mapping"
The mapping is **complete** in the sense that any JSON value type can be converted to a UBJSON value.
Any UBJSON output created by `to_ubjson` can be successfully parsed by `from_ubjson`.
!!! warning "Size constraints"
The following values can **not** be converted to a UBJSON value:
- strings with more than 9223372036854775807 bytes (theoretical)
!!! warning "UTF-8 validation of string values and object keys"
UBJSON's required string encoding is UTF-8. By default (the [`error_handler`](../../api/basic_json/to_ubjson.md)
parameter left at `keep`), `to_ubjson()` writes the bytes of string values and object keys unchanged, even if they
are not valid UTF-8. With `error_handler_t::strict`, it throws
[`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for ill-formed UTF-8 instead;
`replace`/`ignore` sanitize the string. [`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md)
makes `strict` the default.
!!! info "Unused UBJSON markers"
The following markers are not used in the conversion:
- `Z`: no-op values are not created.
- `C`: single-byte strings are serialized with `S` markers.
!!! info "NaN/infinity handling"
If NaN or Infinity are stored inside a JSON number, they are serialized properly. This behavior differs from the
[`dump()`](../../api/basic_json/dump.md) function which serializes NaN or Infinity to `null`.
!!! info "Optimized formats"
The optimized formats for containers are supported: Parameter `use_size` adds size information to the beginning of a
container and removes the closing marker. Parameter `use_type` further checks whether all elements of a container
have the same type and adds the type marker to the beginning of the container. The `use_type` parameter must only be
used together with `use_size = true`.
Note that `use_size = true` alone may result in larger representations - the benefit of this parameter is that the
receiving side is immediately informed on the number of elements of the container.
An array whose type marker is `Z` (null), `T` (true) or `F` (false) stores no payload at all, because the marker
already is the value. Its declared count is therefore the only thing that decides how much memory the receiving side
allocates, and a handful of bytes can describe billions of elements. `from_ubjson` rejects such an array with
[`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408) when the count exceeds 1,048,576
(`1 << 20`), and `to_ubjson` writes longer arrays of these types without the annotation, so any value it produces
can be read back.
!!! info "Binary values"
If the JSON data contains the binary type, the value stored is a list of integers, as suggested by the UBJSON
documentation. In particular, this means that serialization and the deserialization of a JSON containing binary
values into UBJSON and back will result in a different JSON object.
??? example "Example: serialize JSON values to UBJSON, with and without size/type optimization"
```cpp
#include <iostream>
#include <iomanip>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
using namespace nlohmann::literals;
// function to print UBJSON's diagnostic format
void print_byte(uint8_t byte)
{
if (32 < byte and byte < 128)
{
std::cout << (char)byte;
}
else
{
std::cout << (int)byte;
}
}
int main()
{
// create a JSON value
json j = R"({"compact": true, "schema": false})"_json;
// serialize it to UBJSON
std::vector<std::uint8_t> v = json::to_ubjson(j);
// print the vector content
for (auto& byte : v)
{
print_byte(byte);
}
std::cout << std::endl;
// create an array of numbers
json array = {1, 2, 3, 4, 5, 6, 7, 8};
// serialize it to UBJSON using default representation
std::vector<std::uint8_t> v_array = json::to_ubjson(array);
// serialize it to UBJSON using size optimization
std::vector<std::uint8_t> v_array_size = json::to_ubjson(array, true);
// serialize it to UBJSON using type optimization
std::vector<std::uint8_t> v_array_size_and_type = json::to_ubjson(array, true, true);
// print the vector contents
for (auto& byte : v_array)
{
print_byte(byte);
}
std::cout << std::endl;
for (auto& byte : v_array_size)
{
print_byte(byte);
}
std::cout << std::endl;
for (auto& byte : v_array_size_and_type)
{
print_byte(byte);
}
std::cout << std::endl;
}
```
Output:
```c
{i7compactTi6schemaF}
[i1i2i3i4i5i6i7i8]
[#i8i1i2i3i4i5i6i7i8
[$i#i812345678
```
### Byte layout
UBJSON is built from just a one-byte, human-readable ASCII *type marker* (for example, `i` for int8, `S` for string, `[` for array) followed by the payload of that type. Strings and the like are prefixed with their length, which is itself a UBJSON integer value including its marker. Numbers are stored in big-endian byte order. Containers can be written in two ways: as an *unoptimized* container that is opened with `[` or `{` and closed with `]` or `}`, or as an *optimized* container that declares a count (`#`) after the opening marker, optionally preceded by an element type (`$`); an element type is only allowed together with a count. An optimized container saves the closing marker and, with a type, the per-element markers.
Consider the following JSON value (shown formatted for readability):
@@ -206,31 +40,230 @@ Consider the following JSON value (shown formatted for readability):
<tr><td><code>}</code></td><td><span class="bl bl-t">7D</span></td><td>object end (<code>}</code>)</td></tr>
</tbody></table></div>
## Serialization
The library uses the following mapping from JSON values types to UBJSON types according to the UBJSON specification:
| JSON value type | value/range | UBJSON type | marker |
|-----------------|-------------------------------------------|--------------------------------------------------|-----------|
| null | `null` | null | `Z` |
| boolean | `true` | true | `T` |
| boolean | `false` | false | `F` |
| number_integer | -9223372036854775808..-2147483649 | int64 | `L` |
| number_integer | -2147483648..-32769 | int32 | `l` |
| number_integer | -32768..-129 | int16 | `I` |
| number_integer | -128..127 | int8 | `i` |
| number_integer | 128..255 | uint8 | `U` |
| number_integer | 256..32767 | int16 | `I` |
| number_integer | 32768..2147483647 | int32 | `l` |
| number_integer | 2147483648..9223372036854775807 | int64 | `L` |
| number_unsigned | 0..127 | int8 | `i` |
| number_unsigned | 128..255 | uint8 | `U` |
| number_unsigned | 256..32767 | int16 | `I` |
| number_unsigned | 32768..2147483647 | int32 | `l` |
| number_unsigned | 2147483648..9223372036854775807 | int64 | `L` |
| number_unsigned | 9223372036854775808..18446744073709551615 | high-precision | `H` |
| number_float | *any value* | float64 (float32 if `number_float_t` is `float`) | `D` (`d`) |
| string | *with shortest length indicator* | string | `S` |
| array | *see notes on optimized format* | array | `[` |
| object | *see notes on optimized format* | object | `{` |
!!! success "Complete mapping"
The mapping is **complete** in the sense that any JSON value type can be converted to a UBJSON value.
Any UBJSON output created by `to_ubjson` can be successfully parsed by `from_ubjson`.
!!! warning "Size constraints"
The following values can **not** be converted to a UBJSON value:
- strings with more than 9223372036854775807 bytes (theoretical)
!!! warning "UTF-8 validation of string values and object keys"
UBJSON's required string encoding is UTF-8. By default (the [`error_handler`](../../api/basic_json/to_ubjson.md)
parameter left at `keep`), `to_ubjson()` writes the bytes of string values and object keys unchanged, even if they
are not valid UTF-8. With `error_handler_t::strict`, it throws
[`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for ill-formed UTF-8 instead;
`replace`/`ignore` sanitize the string. [`JSON_STRICT_BINARY_UTF8`](../../api/macros/json_strict_binary_utf8.md)
makes `strict` the default.
!!! info "Unused UBJSON markers"
The following markers are not used in the conversion:
- `N`: no-op values are never written (they are skipped when reading).
- `C`: single-byte strings are serialized with `S` markers.
!!! info "NaN/infinity handling"
NaN and infinity are written as IEEE 754 float64 values (`D`) and read back unchanged. This differs from
[`dump()`](../../api/basic_json/dump.md), which serializes them to `null`, and from the UBJSON specification, which
encodes infinity as `null` (it does not mention NaN); other UBJSON readers may therefore not accept these values.
!!! info "Optimized formats"
The optimized formats for containers are supported: Parameter `use_size` adds size information to the beginning of a
container and removes the closing marker. Parameter `use_type` further checks whether all elements of a container
have the same type and adds the type marker to the beginning of the container. The `use_type` parameter must only be
used together with `use_size = true`: combining `use_type` with `use_size = false` throws
[`other_error.502`](../../home/exceptions.md#jsonexceptionother_error502) if the value contains a non-empty array,
object, or binary value.
Two elements have the same type if they are written with the same type marker. For example, `[1, 2]` is written with
the type `i`, but `[1, 200]` (`i`, `U`), `[true, false]` (`T`, `F`), and `[1, 2.5]` (`i`, `D`) are not; an empty
container is never given a `$` annotation of its own (an array of empty arrays is still typed `[`).
Note that `use_size = true` alone may result in larger representations - the benefit of this parameter is that the
receiving side is immediately informed on the number of elements of the container.
An array whose type marker is `Z` (null), `T` (true) or `F` (false) stores no payload at all, because the marker
already is the value. Its declared count is therefore the only thing that decides how much memory the receiving side
allocates, and a handful of bytes can describe billions of elements. `from_ubjson` rejects such an array with
[`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408) when the count exceeds 1,048,576
(`1 << 20`), and `to_ubjson` writes longer arrays of these types in the counted but untyped form: the `$` type marker is
dropped, the `#` count is kept, and every element is written with its own marker (`[#l 1048577 Z Z ...`), so any value
it produces can be read back. Objects and untyped counted arrays are not subject to this limit.
!!! info "Binary values"
If the JSON data contains the binary type, the value stored is a list of integers, as suggested by the UBJSON
documentation. In particular, this means that serialization and the deserialization of a JSON containing binary
values into UBJSON and back will result in a different JSON object.
??? example "Example: serialize JSON values to UBJSON, with and without size/type optimization"
```cpp
#include <iostream>
#include <iomanip>
#include <nlohmann/json.hpp>
using nlohmann::json;
using namespace nlohmann::literals;
// function to print UBJSON's diagnostic format
void print_byte(uint8_t byte)
{
if (32 < byte and byte < 128)
{
std::cout << static_cast<char>(byte);
}
else
{
std::cout << static_cast<int>(byte);
}
}
int main()
{
// create a JSON value
json j = R"({"compact": true, "schema": false})"_json;
// serialize it to UBJSON
std::vector<std::uint8_t> v = json::to_ubjson(j);
// print the vector content
for (auto& byte : v)
{
print_byte(byte);
}
std::cout << '\n';
// create an array of numbers
json array = {1, 2, 3, 4, 5, 6, 7, 8};
// serialize it to UBJSON using default representation
std::vector<std::uint8_t> v_array = json::to_ubjson(array);
// serialize it to UBJSON using size optimization
std::vector<std::uint8_t> v_array_size = json::to_ubjson(array, true);
// serialize it to UBJSON using type optimization
std::vector<std::uint8_t> v_array_size_and_type = json::to_ubjson(array, true, true);
// print the vector contents
for (auto& byte : v_array)
{
print_byte(byte);
}
std::cout << '\n';
for (auto& byte : v_array_size)
{
print_byte(byte);
}
std::cout << '\n';
for (auto& byte : v_array_size_and_type)
{
print_byte(byte);
}
std::cout << '\n';
}
```
Output:
```c
{i7compactTi6schemaF}
[i1i2i3i4i5i6i7i8]
[#i8i1i2i3i4i5i6i7i8
[$i#i812345678
```
## Deserialization
The library maps UBJSON types to JSON value types as follows:
| UBJSON type | JSON value type | marker |
|-------------|-----------------------------------------|--------|
| no-op | *no value, next value is read* | `N` |
| null | `null` | `Z` |
| false | `false` | `F` |
| true | `true` | `T` |
| float32 | number_float | `d` |
| float64 | number_float | `D` |
| uint8 | number_unsigned | `U` |
| int8 | number_integer | `i` |
| int16 | number_integer | `I` |
| int32 | number_integer | `l` |
| int64 | number_integer | `L` |
| string | string | `S` |
| char | string | `C` |
| array | array (optimized values are supported) | `[` |
| object | object (optimized values are supported) | `{` |
| UBJSON type | JSON value type | marker |
|----------------|--------------------------------------------------|--------|
| no-op | *no value, next value is read* | `N` |
| null | `null` | `Z` |
| false | `false` | `F` |
| true | `true` | `T` |
| float32 | number_float | `d` |
| float64 | number_float | `D` |
| uint8 | number_unsigned | `U` |
| int8 | number_integer | `i` |
| int16 | number_integer | `I` |
| int32 | number_integer | `l` |
| int64 | number_integer | `L` |
| string | string | `S` |
| char | string | `C` |
| high-precision | number_unsigned, number_integer, or number_float | `H` |
| array | array (optimized values are supported) | `[` |
| object | object (optimized values are supported) | `{` |
!!! info "High-precision numbers"
The payload of `H` is a length-prefixed string that is parsed as a JSON number (surrounding whitespace is ignored).
Integer text in the range of `number_unsigned_t` (non-negative) or `number_integer_t` (negative) is stored as such;
any other valid number text (larger or smaller integers, fractions, and exponents such as `1e2`) is stored as
`number_float`, so precision may be lost. Text that is not a valid JSON number raises
[`parse_error.115`](../../home/exceptions.md#jsonexceptionparse_error115), a number outside the range of
`number_float_t` (`1e999`) raises [`out_of_range.406`](../../home/exceptions.md#jsonexceptionout_of_range406).
!!! success "Complete mapping"
The mapping is **complete** in the sense that any UBJSON value can be converted to a JSON value.
Every well-formed UBJSON value can be converted to a JSON value, with these exceptions: a high-precision number
outside the range of `number_float_t` ([`out_of_range.406`](../../home/exceptions.md#jsonexceptionout_of_range406)),
an optimized array of `Z`, `T`, or `F` with more than 1,048,576 elements, and a count that exceeds the capacity of the
container ([`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408)). High-precision numbers that
do not fit an integer type are stored as floating-point numbers and may lose precision. Ill-formed input is rejected
with a parse error.
!!! info "Duplicate object keys"
If an object contains the same key more than once, the last value is kept (the UBJSON specification does not
address duplicate keys).
!!! info "Type marker of optimized containers"
The type marker after `$` must be one of `Z`, `N`, `T`, `F`, `i`, `U`, `I`, `l`, `L`, `d`, `D`, `H`, `C`, `S`, `[`, or
`{`. It is checked even if the count is 0; any other marker raises
[`parse_error.112`](../../home/exceptions.md#jsonexceptionparse_error112). A counted object typed `N` (`{$N#i1 ...`)
is not supported.
!!! warning "Ill-formed UTF-8 in string values and object keys"
@@ -252,7 +285,7 @@ The library maps UBJSON types to JSON value types as follows:
#include <iomanip>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
using nlohmann::json;
int main()
{
@@ -266,7 +299,7 @@ The library maps UBJSON types to JSON value types as follows:
json j = json::from_ubjson(v);
// print the deserialized JSON value
std::cout << std::setw(2) << j << std::endl;
std::cout << std::setw(2) << j << '\n';
}
```