wt/json
1
0 Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-10-25 12:04:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: byte layout sections, glossary entries, and QA page update (#5910)

Browse Source 
* docs: byte layout sections, glossary, and QA page updates

- Move "Byte layout" of each binary format to its own section before
  "Serialization" and describe the idea of the format.
- Glossary: add LLDB and other terms used throughout the documentation.
- quality_assurance: correct Clang Static Analyzer and CMake entries; add
  CodeQL, Semgrep, Flawfinder, CIFuzz, OpenSSF Scorecard, REUSE, Meson/SwiftPM,
  and reproducibility checks.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: drop unused length colour from CBOR and MessagePack byte layout legends

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann authored and GitHub committed 2026-10-25 12:08:14 +01:00
1 parent 330fbead59
commit 1cf5492bcd
8 files changed
+252 -216
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/mkdocs/docs/community/quality_assurance.md
+27 -3
View File
@@ -188,10 +188,13 @@ Note: Some modern features (like C++20 ranges or filesystem support) may be disa
```
- [x] The code is checked with the latest [Cppcheck](https://cppcheck.sourceforge.io) with all warnings enabled.
- [x] The code is checked with the latest [Clang Static Analyzer](https://clang-analyzer.llvm.org) with 89 enabled
rules.
- [x] The code is checked with the latest [Clang Static Analyzer](https://clang-analyzer.llvm.org) with 52 enabled
checkers.
- [x] The code is checked with [Infer](https://fbinfer.com).
- [x] The code is checked with [Codacy](https://app.codacy.com/gh/nlohmann/json/dashboard).
- [x] The code is scanned with [CodeQL](https://codeql.github.com), [Semgrep](https://semgrep.dev), and
[Flawfinder](https://dwheeler.com/flawfinder/) on every push and pull request and on a schedule; the findings are
reported to GitHub's code scanning.
## Dynamic analysis
@@ -204,6 +207,8 @@ Note: Some modern features (like C++20 ranges or filesystem support) may be disa
- [x] The test suite is executed with [Valgrind](https://valgrind.org) (Memcheck) to detect memory leaks.
- [x] The test suite is executed with [Sanitizers](https://github.com/google/sanitizers) (address sanitizer, undefined
behavior sanitizer, integer overflow detection, nullability violations).
- [x] The fuzz targets are run with [CIFuzz](https://google.github.io/oss-fuzz/getting-started/continuous-integration/)
on every pull request to catch regressions before they are merged.
## Dependencies
@@ -223,6 +228,8 @@ Note: Some modern features (like C++20 ranges or filesystem support) may be disa
No release is made while such an alert is open.
- [x] Third-party code included in the repository for testing, such as [doctest](https://github.com/doctest/doctest),
is updated manually.
- [x] The repository is evaluated with the [OpenSSF Scorecard](https://securityscorecards.dev) on a schedule and on
every push to the default branch, and the results are reported to GitHub's code scanning.
## Style check
@@ -258,4 +265,21 @@ Note: Some modern features (like C++20 ranges or filesystem support) may be disa
- [x] The library is tested against relevant CMake versions:
- CMake 3.5 (the earliest supported)
- CMake 3.31.6 (the latest 3.x release)
- CMake 4.0.0 (a very recent release)
- CMake 4.0.0 (the first 4.x release)
!!! success "Requirement: Other build systems"
The library can be used with build systems and package managers other than CMake.
- [x] The library can be consumed with [Meson](https://json.nlohmann.me/integration/package_managers/) and the Swift
Package Manager; the installation with Meson and the resolution of `Package.swift` are tested in the CI.
!!! success "Requirement: Reproducible and verifiable builds"
The library can be built and tested without network access or a Git checkout, and its licensing is machine-readable.
- [x] The test suite is executed from an exported source tree without the `.git` directory and with the test data
being available offline.
- [x] The test suite is executed without the tests that change installed files to check that builds are reproducible.
- [x] The repository is checked for compliance with the [REUSE specification](https://reuse.software) so that the
license and copyright of every file is machine-readable.
- [x] The documentation is built with warnings treated as errors, and its links are checked regularly.
docs/mkdocs/docs/features/binary_formats/bjdata.md
+33 -35
View File
@@ -17,6 +17,39 @@ simple processing.
- [BJData Specification](https://neurojson.org/bjdata/draft2)
## Byte layout
BJData follows the structure of UBJSON: a one-byte, human-readable ASCII *type marker* followed by the payload of that type, with strings prefixed by a length that is itself an integer value. In contrast to UBJSON, numbers are stored in little-endian byte order, and additional markers cover unsigned 16/32/64-bit integers, half-precision floats, and bytes. Containers can be written unoptimized (opened with `[` or `{`, closed with `]` or `}`) or optimized with a declared element type (`$`) and/or count (`#`). BJData extends the optimized form: the count can be a list of dimensions, which stores an N-dimensional packed array as a flat sequence of values without any per-element markers.
Consider the following JSON value (shown formatted for readability):
```json
{
"id": 300,
"name": "json",
"pi": 3.14,
"tags": [
"a",
"b"
]
}
```
`to_bjdata` encodes it to 51 bytes (shown in hexadecimal; the encoding contains no whitespace). The colors mark the role of each byte: <span class="bl bl-t">type/marker</span> <span class="bl bl-l">length/count/size</span> <span class="bl bl-d">data</span>.
<div class="byte-layout" markdown="0"><table>
<thead><tr><th>JSON</th><th>Bytes</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>{</code></td><td><span class="bl bl-t">7B</span></td><td>object start (<code>{</code>)</td></tr>
<tr><td><code>&quot;id&quot;: 300,</code></td><td><span class="bl bl-l">69 02</span> <span class="bl bl-d">69 64</span> <span class="bl bl-t">49</span> <span class="bl bl-d">2C 01</span></td><td>key: length 2 as int8 (<code>i</code>), then UTF-8 bytes; value: int16 (<code>I</code>), little-endian</td></tr>
<tr><td><code>&quot;name&quot;: &quot;json&quot;,</code></td><td><span class="bl bl-l">69 04</span> <span class="bl bl-d">6E 61 6D 65</span> <span class="bl bl-t">53</span> <span class="bl bl-l">69 04</span> <span class="bl bl-d">6A 73 6F 6E</span></td><td>key: length 4; value: string (<code>S</code>), length 4 as int8, UTF-8 bytes</td></tr>
<tr><td><code>&quot;pi&quot;: 3.14,</code></td><td><span class="bl bl-l">69 02</span> <span class="bl bl-d">70 69</span> <span class="bl bl-t">44</span> <span class="bl bl-d">1F 85 EB 51 B8 1E 09 40</span></td><td>key: length 2; value: float64 (<code>D</code>), little-endian</td></tr>
<tr><td><code>&quot;tags&quot;: [</code></td><td><span class="bl bl-l">69 04</span> <span class="bl bl-d">74 61 67 73</span> <span class="bl bl-t">5B</span></td><td>key: length 4; value: array start (<code>[</code>)</td></tr>
<tr><td><code>&quot;a&quot;,</code></td><td><span class="bl bl-t">53</span> <span class="bl bl-l">69 01</span> <span class="bl bl-d">61</span></td><td>string, length 1</td></tr>
<tr><td><code>&quot;b&quot;</code></td><td><span class="bl bl-t">53</span> <span class="bl bl-l">69 01</span> <span class="bl bl-d">62</span></td><td>string, length 1</td></tr>
<tr><td><code>]</code></td><td><span class="bl bl-t">5D</span></td><td>array end (<code>]</code>)</td></tr>
<tr><td><code>}</code></td><td><span class="bl bl-t">7D</span></td><td>object end (<code>}</code>)</td></tr>
</tbody></table></div>
## Serialization
The library uses the following mapping from JSON values types to BJData types according to the BJData specification:
@@ -190,41 +223,6 @@ The library uses the following mapping from JSON values types to BJData types ac
--8<-- "examples/to_bjdata.output"
```
### Byte layout
Consider the following JSON value (shown formatted for readability):
```json
{
"id": 300,
"name": "json",
"pi": 3.14,
"tags": [
"a",
"b"
]
}
```
`to_bjdata` encodes it to 51 bytes (shown in hexadecimal; the encoding contains no whitespace). The colors mark the role of each byte: <span class="bl bl-t">type/marker</span> <span class="bl bl-l">length/count/size</span> <span class="bl bl-d">data</span>.
<div class="byte-layout" markdown="0"><table>
<thead><tr><th>JSON</th><th>Bytes</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>{</code></td><td><span class="bl bl-t">7B</span></td><td>object start (<code>{</code>)</td></tr>
<tr><td><code>&quot;id&quot;: 300,</code></td><td><span class="bl bl-l">69 02</span> <span class="bl bl-d">69 64</span> <span class="bl bl-t">49</span> <span class="bl bl-d">2C 01</span></td><td>key: length 2 as int8 (<code>i</code>), then UTF-8 bytes; value: int16 (<code>I</code>), little-endian</td></tr>
<tr><td><code>&quot;name&quot;: &quot;json&quot;,</code></td><td><span class="bl bl-l">69 04</span> <span class="bl bl-d">6E 61 6D 65</span> <span class="bl bl-t">53</span> <span class="bl bl-l">69 04</span> <span class="bl bl-d">6A 73 6F 6E</span></td><td>key: length 4; value: string (<code>S</code>), length 4 as int8, UTF-8 bytes</td></tr>
<tr><td><code>&quot;pi&quot;: 3.14,</code></td><td><span class="bl bl-l">69 02</span> <span class="bl bl-d">70 69</span> <span class="bl bl-t">44</span> <span class="bl bl-d">1F 85 EB 51 B8 1E 09 40</span></td><td>key: length 2; value: float64 (<code>D</code>), little-endian</td></tr>
<tr><td><code>&quot;tags&quot;: [</code></td><td><span class="bl bl-l">69 04</span> <span class="bl bl-d">74 61 67 73</span> <span class="bl bl-t">5B</span></td><td>key: length 4; value: array start (<code>[</code>)</td></tr>
<tr><td><code>&quot;a&quot;,</code></td><td><span class="bl bl-t">53</span> <span class="bl bl-l">69 01</span> <span class="bl bl-d">61</span></td><td>string, length 1</td></tr>
<tr><td><code>&quot;b&quot;</code></td><td><span class="bl bl-t">53</span> <span class="bl bl-l">69 01</span> <span class="bl bl-d">62</span></td><td>string, length 1</td></tr>
<tr><td><code>]</code></td><td><span class="bl bl-t">5D</span></td><td>array end (<code>]</code>)</td></tr>
<tr><td><code>}</code></td><td><span class="bl bl-t">7D</span></td><td>object end (<code>}</code>)</td></tr>
</tbody></table></div>
## Deserialization
The library maps BJData types to JSON value types as follows:
docs/mkdocs/docs/features/binary_formats/bon8.md
+33 -35
View File
@@ -10,6 +10,39 @@ values -1.0, 0.0, and 1.0 take a single byte, and arrays and objects with up to
- [BON8 specification](https://github.com/hikoworks/hikogui/blob/main/docs/BON8.md)
- [Reference implementation](https://github.com/hikoworks/hikogui/blob/main/src/hikogui/codec/BON8.hpp) in HikoGUI
## Byte layout
BON8 exploits the fact that some byte values can never start a UTF-8 character. These bytes serve as *type markers*, so strings can be stored as plain UTF-8 without a length prefix: a string ends at the first byte that cannot continue it (often the marker of the next value). Small integers, `true`, `false`, `null`, and a few common floats are a single byte. Arrays and objects are introduced by a marker; those with up to four elements encode their size in the marker and need no terminator, while larger ones are closed by the end marker `0xFE`.
Consider the following JSON value (shown formatted for readability):
```json
{
"id": 300,
"name": "json",
"pi": 3.14,
"tags": [
"a",
"b"
]
}
```
`to_bon8` encodes it to 35 bytes (shown in hexadecimal; the encoding contains no whitespace). The colors mark the role of each byte: <span class="bl bl-t">type/marker</span> <span class="bl bl-l">length/count/size</span> <span class="bl bl-d">data</span>.
<div class="byte-layout" markdown="0"><table>
<thead><tr><th>JSON</th><th>Bytes</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>{</code></td><td><span class="bl bl-t">8A</span></td><td>object with 4 members</td></tr>
<tr><td><code>&quot;id&quot;: 300,</code></td><td><span class="bl bl-d">69 64</span> <span class="bl bl-t">C4</span> <span class="bl bl-d">04</span></td><td>key: UTF-8 bytes; the string ends at the next byte, which cannot continue a UTF-8 character; value: 2-byte positive integer (lead byte, then a byte below 0x80)</td></tr>
<tr><td><code>&quot;name&quot;: &quot;json&quot;,</code></td><td><span class="bl bl-d">6E 61 6D 65</span> <span class="bl bl-l">FF</span> <span class="bl bl-d">6A 73 6F 6E</span> <span class="bl bl-l">FF</span></td><td>key: UTF-8 bytes; 0xFF terminates the string as another string follows; value: UTF-8 bytes; terminated as a string key follows</td></tr>
<tr><td><code>&quot;pi&quot;: 3.14,</code></td><td><span class="bl bl-d">70 69</span> <span class="bl bl-t">8F</span> <span class="bl bl-d">40 09 1E B8 51 EB 85 1F</span></td><td>key: UTF-8 bytes; value: binary64 (IEEE 754, big-endian); its marker also ends <code>&quot;pi&quot;</code></td></tr>
<tr><td><code>&quot;tags&quot;: [</code></td><td><span class="bl bl-d">74 61 67 73</span> <span class="bl bl-t">82</span></td><td>key: UTF-8 bytes; value: array with 2 elements; its marker also ends <code>&quot;tags&quot;</code></td></tr>
<tr><td><code>&quot;a&quot;,</code></td><td><span class="bl bl-d">61</span> <span class="bl bl-l">FF</span></td><td>UTF-8 bytes; terminated as a string follows</td></tr>
<tr><td><code>&quot;b&quot;</code></td><td><span class="bl bl-d">62</span> <span class="bl bl-l">FF</span></td><td>UTF-8 bytes; terminated as nothing follows</td></tr>
<tr><td><code>]</code></td><td></td><td>no end marker: the element count was given up front</td></tr>
<tr><td><code>}</code></td><td></td><td>no end marker: the member count was given up front</td></tr>
</tbody></table></div>
## Serialization
The library uses the following mapping from JSON values types to BON8 types according to the BON8 specification:
@@ -105,41 +138,6 @@ serialized as 0x81 0x65 0xFF, but `[1,2,3,4,"e"]` as 0x85 0x91 0x92 0x93 0x94 0x
--8<-- "examples/to_bon8.output"
```
### Byte layout
Consider the following JSON value (shown formatted for readability):
```json
{
"id": 300,
"name": "json",
"pi": 3.14,
"tags": [
"a",
"b"
]
}
```
`to_bon8` encodes it to 35 bytes (shown in hexadecimal; the encoding contains no whitespace). The colors mark the role of each byte: <span class="bl bl-t">type/marker</span> <span class="bl bl-l">length/count/size</span> <span class="bl bl-d">data</span>.
<div class="byte-layout" markdown="0"><table>
<thead><tr><th>JSON</th><th>Bytes</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>{</code></td><td><span class="bl bl-t">8A</span></td><td>object with 4 members</td></tr>
<tr><td><code>&quot;id&quot;: 300,</code></td><td><span class="bl bl-d">69 64</span> <span class="bl bl-t">C4</span> <span class="bl bl-d">04</span></td><td>key: UTF-8 bytes; the string ends at the next byte, which cannot continue a UTF-8 character; value: 2-byte positive integer (lead byte, then a byte below 0x80)</td></tr>
<tr><td><code>&quot;name&quot;: &quot;json&quot;,</code></td><td><span class="bl bl-d">6E 61 6D 65</span> <span class="bl bl-l">FF</span> <span class="bl bl-d">6A 73 6F 6E</span> <span class="bl bl-l">FF</span></td><td>key: UTF-8 bytes; 0xFF terminates the string as another string follows; value: UTF-8 bytes; terminated as a string key follows</td></tr>
<tr><td><code>&quot;pi&quot;: 3.14,</code></td><td><span class="bl bl-d">70 69</span> <span class="bl bl-t">8F</span> <span class="bl bl-d">40 09 1E B8 51 EB 85 1F</span></td><td>key: UTF-8 bytes; value: binary64 (IEEE 754, big-endian); its marker also ends <code>&quot;pi&quot;</code></td></tr>
<tr><td><code>&quot;tags&quot;: [</code></td><td><span class="bl bl-d">74 61 67 73</span> <span class="bl bl-t">82</span></td><td>key: UTF-8 bytes; value: array with 2 elements; its marker also ends <code>&quot;tags&quot;</code></td></tr>
<tr><td><code>&quot;a&quot;,</code></td><td><span class="bl bl-d">61</span> <span class="bl bl-l">FF</span></td><td>UTF-8 bytes; terminated as a string follows</td></tr>
<tr><td><code>&quot;b&quot;</code></td><td><span class="bl bl-d">62</span> <span class="bl bl-l">FF</span></td><td>UTF-8 bytes; terminated as nothing follows</td></tr>
<tr><td><code>]</code></td><td></td><td>no end marker: the element count was given up front</td></tr>
<tr><td><code>}</code></td><td></td><td>no end marker: the member count was given up front</td></tr>
</tbody></table></div>
## Deserialization
The library maps BON8 types to JSON value types as follows:
docs/mkdocs/docs/features/binary_formats/bson.md
+33 -36
View File
@@ -10,6 +10,39 @@ representation of data types that are not part of the JSON spec. For example, BS
- [BSON Specification](http://bsonspec.org/spec.html) - the specification
## Byte layout
A BSON document is a length-prefixed sequence of *elements*. The first four bytes hold the total size of the document (little-endian int32), followed by the elements, and a final `0x00` byte ends the document. Each element starts with a one-byte type, followed by its key as a NUL-terminated string and the value, whose encoding depends on the type. Arrays are documents whose keys are the decimal indices `"0"`, `"1"`, ...; nesting is expressed by embedding documents. Because every document announces its size, a reader can skip over it without parsing its content. The top level of a BSON value is always a document, which is why only JSON objects can be serialized.
Consider the following JSON value (shown formatted for readability):
```json
{
"id": 300,
"name": "json",
"pi": 3.14,
"tags": [
"a",
"b"
]
}
```
`to_bson` encodes it to 69 bytes (shown in hexadecimal; the encoding contains no whitespace). The colors mark the role of each byte: <span class="bl bl-t">type/marker</span> <span class="bl bl-l">length/count/size</span> <span class="bl bl-d">data</span>.
<div class="byte-layout" markdown="0"><table>
<thead><tr><th>JSON</th><th>Bytes</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>{</code></td><td><span class="bl bl-l">45 00 00 00</span></td><td>document size: 69 bytes (int32, little-endian)</td></tr>
<tr><td><code>&quot;id&quot;: 300,</code></td><td><span class="bl bl-t">10</span> <span class="bl bl-d">69 64 00</span> <span class="bl bl-d">2C 01 00 00</span></td><td>int32 element; key as NUL-terminated string; value (little-endian)</td></tr>
<tr><td><code>&quot;name&quot;: &quot;json&quot;,</code></td><td><span class="bl bl-t">02</span> <span class="bl bl-d">6E 61 6D 65 00</span> <span class="bl bl-l">05 00 00 00</span> <span class="bl bl-d">6A 73 6F 6E 00</span></td><td>string element; key; length incl. NUL (int32); NUL-terminated UTF-8 bytes</td></tr>
<tr><td><code>&quot;pi&quot;: 3.14,</code></td><td><span class="bl bl-t">01</span> <span class="bl bl-d">70 69 00</span> <span class="bl bl-d">1F 85 EB 51 B8 1E 09 40</span></td><td>double element; key; value (little-endian)</td></tr>
<tr><td><code>&quot;tags&quot;: [</code></td><td><span class="bl bl-t">04</span> <span class="bl bl-d">74 61 67 73 00</span> <span class="bl bl-l">17 00 00 00</span></td><td>array element; key; embedded document of 23 bytes (keys are the indices)</td></tr>
<tr><td><code>&quot;a&quot;,</code></td><td><span class="bl bl-t">02</span> <span class="bl bl-d">30 00</span> <span class="bl bl-l">02 00 00 00</span> <span class="bl bl-d">61 00</span></td><td>string element; key <code>&quot;0&quot;</code>; length 2; <code>a</code> + NUL</td></tr>
<tr><td><code>&quot;b&quot;</code></td><td><span class="bl bl-t">02</span> <span class="bl bl-d">31 00</span> <span class="bl bl-l">02 00 00 00</span> <span class="bl bl-d">62 00</span></td><td>string element; key <code>&quot;1&quot;</code>; length 2; <code>b</code> + NUL</td></tr>
<tr><td><code>]</code></td><td><span class="bl bl-t">00</span></td><td>end of the array document</td></tr>
<tr><td><code>}</code></td><td><span class="bl bl-t">00</span></td><td>end of the document</td></tr>
</tbody></table></div>
## Serialization
The library uses the following mapping from JSON values types to BSON types:
@@ -60,42 +93,6 @@ The library uses the following mapping from JSON values types to BSON types:
--8<-- "examples/to_bson.output"
```
### Byte layout
Consider the following JSON value (shown formatted for readability):
```json
{
"id": 300,
"name": "json",
"pi": 3.14,
"tags": [
"a",
"b"
]
}
```
`to_bson` encodes it to 69 bytes (shown in hexadecimal; the encoding contains no whitespace). The colors mark the role of each byte: <span class="bl bl-t">type/marker</span> <span class="bl bl-l">length/count/size</span> <span class="bl bl-d">data</span>.
<div class="byte-layout" markdown="0"><table>
<thead><tr><th>JSON</th><th>Bytes</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>{</code></td><td><span class="bl bl-l">45 00 00 00</span></td><td>document size: 69 bytes (int32, little-endian)</td></tr>
<tr><td><code>&quot;id&quot;: 300,</code></td><td><span class="bl bl-t">10</span> <span class="bl bl-d">69 64 00</span> <span class="bl bl-d">2C 01 00 00</span></td><td>int32 element; key as NUL-terminated string; value (little-endian)</td></tr>
<tr><td><code>&quot;name&quot;: &quot;json&quot;,</code></td><td><span class="bl bl-t">02</span> <span class="bl bl-d">6E 61 6D 65 00</span> <span class="bl bl-l">05 00 00 00</span> <span class="bl bl-d">6A 73 6F 6E 00</span></td><td>string element; key; length incl. NUL (int32); NUL-terminated UTF-8 bytes</td></tr>
<tr><td><code>&quot;pi&quot;: 3.14,</code></td><td><span class="bl bl-t">01</span> <span class="bl bl-d">70 69 00</span> <span class="bl bl-d">1F 85 EB 51 B8 1E 09 40</span></td><td>double element; key; value (little-endian)</td></tr>
<tr><td><code>&quot;tags&quot;: [</code></td><td><span class="bl bl-t">04</span> <span class="bl bl-d">74 61 67 73 00</span> <span class="bl bl-l">17 00 00 00</span></td><td>array element; key; embedded document of 23 bytes (keys are the indices)</td></tr>
<tr><td><code>&quot;a&quot;,</code></td><td><span class="bl bl-t">02</span> <span class="bl bl-d">30 00</span> <span class="bl bl-l">02 00 00 00</span> <span class="bl bl-d">61 00</span></td><td>string element; key <code>&quot;0&quot;</code>; length 2; <code>a</code> + NUL</td></tr>
<tr><td><code>&quot;b&quot;</code></td><td><span class="bl bl-t">02</span> <span class="bl bl-d">31 00</span> <span class="bl bl-l">02 00 00 00</span> <span class="bl bl-d">62 00</span></td><td>string element; key <code>&quot;1&quot;</code>; length 2; <code>b</code> + NUL</td></tr>
<tr><td><code>]</code></td><td><span class="bl bl-t">00</span></td><td>end of the array document</td></tr>
<tr><td><code>}</code></td><td><span class="bl bl-t">00</span></td><td>end of the document</td></tr>
</tbody></table></div>
## Deserialization
The library maps BSON record types to JSON value types as follows:
docs/mkdocs/docs/features/binary_formats/cbor.md
+33 -35
View File
@@ -9,6 +9,39 @@ extremely small code sizes, fairly small message size, and extensibility without
- [CBOR Playground](http://cbor.me) - an interactive webpage to translate between JSON and CBOR
- [RFC 8949](https://www.rfc-editor.org/rfc/rfc8949.html) - the CBOR specification
## Byte layout
CBOR encodes every data item as a *head* followed by optional content. The first byte of the head holds the *major type* in its upper three bits (unsigned integer, negative integer, byte string, text string, array, map, tag, or simple/float) and an *additional information* value in its lower five bits. Additional information below 24 is the argument itself (for example, a small integer or a short length); the values 24 to 27 announce that the argument follows in 1, 2, 4, or 8 big-endian bytes. Strings, arrays, and maps are prefixed with their length (or count of elements/pairs), and content follows directly, so a reader always knows how many items to expect without a terminator.
Consider the following JSON value (shown formatted for readability):
```json
{
"id": 300,
"name": "json",
"pi": 3.14,
"tags": [
"a",
"b"
]
}
```
`to_cbor` encodes it to 39 bytes (shown in hexadecimal; the encoding contains no whitespace). The colors mark the role of each byte: <span class="bl bl-t">type/marker</span> <span class="bl bl-d">data</span>.
<div class="byte-layout" markdown="0"><table>
<thead><tr><th>JSON</th><th>Bytes</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>{</code></td><td><span class="bl bl-t">A4</span></td><td>map with 4 key/value pairs</td></tr>
<tr><td><code>&quot;id&quot;: 300,</code></td><td><span class="bl bl-t">62</span> <span class="bl bl-d">69 64</span> <span class="bl bl-t">19</span> <span class="bl bl-d">01 2C</span></td><td>key: text string of length 2, then its UTF-8 bytes; value: unsigned integer, 2 bytes follow (big-endian)</td></tr>
<tr><td><code>&quot;name&quot;: &quot;json&quot;,</code></td><td><span class="bl bl-t">64</span> <span class="bl bl-d">6E 61 6D 65</span> <span class="bl bl-t">64</span> <span class="bl bl-d">6A 73 6F 6E</span></td><td>key: text string of length 4; value: text string of length 4</td></tr>
<tr><td><code>&quot;pi&quot;: 3.14,</code></td><td><span class="bl bl-t">62</span> <span class="bl bl-d">70 69</span> <span class="bl bl-t">FB</span> <span class="bl bl-d">40 09 1E B8 51 EB 85 1F</span></td><td>key: text string of length 2; value: double-precision float (IEEE 754, big-endian)</td></tr>
<tr><td><code>&quot;tags&quot;: [</code></td><td><span class="bl bl-t">64</span> <span class="bl bl-d">74 61 67 73</span> <span class="bl bl-t">82</span></td><td>key: text string of length 4; value: array with 2 elements</td></tr>
<tr><td><code>&quot;a&quot;,</code></td><td><span class="bl bl-t">61</span> <span class="bl bl-d">61</span></td><td>text string of length 1</td></tr>
<tr><td><code>&quot;b&quot;</code></td><td><span class="bl bl-t">61</span> <span class="bl bl-d">62</span></td><td>text string of length 1</td></tr>
<tr><td><code>]</code></td><td></td><td>no end marker: the element count was given up front</td></tr>
<tr><td><code>}</code></td><td></td><td>no end marker: the member count was given up front</td></tr>
</tbody></table></div>
## Serialization
The library uses the following mapping from JSON values types to CBOR types according to the CBOR specification
@@ -110,41 +143,6 @@ see "binary" cells in the table above.
--8<-- "examples/to_cbor.output"
```
### Byte layout
Consider the following JSON value (shown formatted for readability):
```json
{
"id": 300,
"name": "json",
"pi": 3.14,
"tags": [
"a",
"b"
]
}
```
`to_cbor` encodes it to 39 bytes (shown in hexadecimal; the encoding contains no whitespace). The colors mark the role of each byte: <span class="bl bl-t">type/marker</span> <span class="bl bl-l">length/count/size</span> <span class="bl bl-d">data</span>.
<div class="byte-layout" markdown="0"><table>
<thead><tr><th>JSON</th><th>Bytes</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>{</code></td><td><span class="bl bl-t">A4</span></td><td>map with 4 key/value pairs</td></tr>
<tr><td><code>&quot;id&quot;: 300,</code></td><td><span class="bl bl-t">62</span> <span class="bl bl-d">69 64</span> <span class="bl bl-t">19</span> <span class="bl bl-d">01 2C</span></td><td>key: text string of length 2, then its UTF-8 bytes; value: unsigned integer, 2 bytes follow (big-endian)</td></tr>
<tr><td><code>&quot;name&quot;: &quot;json&quot;,</code></td><td><span class="bl bl-t">64</span> <span class="bl bl-d">6E 61 6D 65</span> <span class="bl bl-t">64</span> <span class="bl bl-d">6A 73 6F 6E</span></td><td>key: text string of length 4; value: text string of length 4</td></tr>
<tr><td><code>&quot;pi&quot;: 3.14,</code></td><td><span class="bl bl-t">62</span> <span class="bl bl-d">70 69</span> <span class="bl bl-t">FB</span> <span class="bl bl-d">40 09 1E B8 51 EB 85 1F</span></td><td>key: text string of length 2; value: double-precision float (IEEE 754, big-endian)</td></tr>
<tr><td><code>&quot;tags&quot;: [</code></td><td><span class="bl bl-t">64</span> <span class="bl bl-d">74 61 67 73</span> <span class="bl bl-t">82</span></td><td>key: text string of length 4; value: array with 2 elements</td></tr>
<tr><td><code>&quot;a&quot;,</code></td><td><span class="bl bl-t">61</span> <span class="bl bl-d">61</span></td><td>text string of length 1</td></tr>
<tr><td><code>&quot;b&quot;</code></td><td><span class="bl bl-t">61</span> <span class="bl bl-d">62</span></td><td>text string of length 1</td></tr>
<tr><td><code>]</code></td><td></td><td>no end marker: the element count was given up front</td></tr>
<tr><td><code>}</code></td><td></td><td>no end marker: the member count was given up front</td></tr>
</tbody></table></div>
## Deserialization
The library maps CBOR types to JSON value types as follows:
docs/mkdocs/docs/features/binary_formats/messagepack.md
+33 -35
View File
@@ -9,6 +9,39 @@ extra byte in addition to the strings themselves.
- [MessagePack website](https://msgpack.org)
- [MessagePack specification](https://github.com/msgpack/msgpack/blob/master/spec.md)
## Byte layout
MessagePack encodes each value as a one-byte *format marker*, optionally followed by a length or the payload. The marker range is partitioned to keep common values small: small non-negative integers (0 to 127) and small negative integers (-32 to -1) are the marker byte itself; short strings, arrays, and maps (up to 31 or 15 elements) hide their length in the marker's lower bits. Larger values use dedicated markers that announce the width (8, 16, 32, or 64 bits) of the length or number that follows, in big-endian byte order. Like CBOR, containers are length-prefixed, so no terminator is needed.
Consider the following JSON value (shown formatted for readability):
```json
{
"id": 300,
"name": "json",
"pi": 3.14,
"tags": [
"a",
"b"
]
}
```
`to_msgpack` encodes it to 39 bytes (shown in hexadecimal; the encoding contains no whitespace). The colors mark the role of each byte: <span class="bl bl-t">type/marker</span> <span class="bl bl-d">data</span>.
<div class="byte-layout" markdown="0"><table>
<thead><tr><th>JSON</th><th>Bytes</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>{</code></td><td><span class="bl bl-t">84</span></td><td>map with 4 key/value pairs</td></tr>
<tr><td><code>&quot;id&quot;: 300,</code></td><td><span class="bl bl-t">A2</span> <span class="bl bl-d">69 64</span> <span class="bl bl-t">CD</span> <span class="bl bl-d">01 2C</span></td><td>key: fixstr of length 2, then its UTF-8 bytes; value: uint 16 (big-endian)</td></tr>
<tr><td><code>&quot;name&quot;: &quot;json&quot;,</code></td><td><span class="bl bl-t">A4</span> <span class="bl bl-d">6E 61 6D 65</span> <span class="bl bl-t">A4</span> <span class="bl bl-d">6A 73 6F 6E</span></td><td>key: fixstr of length 4; value: fixstr of length 4</td></tr>
<tr><td><code>&quot;pi&quot;: 3.14,</code></td><td><span class="bl bl-t">A2</span> <span class="bl bl-d">70 69</span> <span class="bl bl-t">CB</span> <span class="bl bl-d">40 09 1E B8 51 EB 85 1F</span></td><td>key: fixstr of length 2; value: float 64 (IEEE 754, big-endian)</td></tr>
<tr><td><code>&quot;tags&quot;: [</code></td><td><span class="bl bl-t">A4</span> <span class="bl bl-d">74 61 67 73</span> <span class="bl bl-t">92</span></td><td>key: fixstr of length 4; value: fixarray with 2 elements</td></tr>
<tr><td><code>&quot;a&quot;,</code></td><td><span class="bl bl-t">A1</span> <span class="bl bl-d">61</span></td><td>fixstr of length 1</td></tr>
<tr><td><code>&quot;b&quot;</code></td><td><span class="bl bl-t">A1</span> <span class="bl bl-d">62</span></td><td>fixstr of length 1</td></tr>
<tr><td><code>]</code></td><td></td><td>no end marker: the element count was given up front</td></tr>
<tr><td><code>}</code></td><td></td><td>no end marker: the member count was given up front</td></tr>
</tbody></table></div>
## Serialization
The library uses the following mapping from JSON values types to MessagePack types according to the MessagePack
@@ -91,41 +124,6 @@ specification:
--8<-- "examples/to_msgpack.output"
```
### Byte layout
Consider the following JSON value (shown formatted for readability):
```json
{
"id": 300,
"name": "json",
"pi": 3.14,
"tags": [
"a",
"b"
]
}
```
`to_msgpack` encodes it to 39 bytes (shown in hexadecimal; the encoding contains no whitespace). The colors mark the role of each byte: <span class="bl bl-t">type/marker</span> <span class="bl bl-l">length/count/size</span> <span class="bl bl-d">data</span>.
<div class="byte-layout" markdown="0"><table>
<thead><tr><th>JSON</th><th>Bytes</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>{</code></td><td><span class="bl bl-t">84</span></td><td>map with 4 key/value pairs</td></tr>
<tr><td><code>&quot;id&quot;: 300,</code></td><td><span class="bl bl-t">A2</span> <span class="bl bl-d">69 64</span> <span class="bl bl-t">CD</span> <span class="bl bl-d">01 2C</span></td><td>key: fixstr of length 2, then its UTF-8 bytes; value: uint 16 (big-endian)</td></tr>
<tr><td><code>&quot;name&quot;: &quot;json&quot;,</code></td><td><span class="bl bl-t">A4</span> <span class="bl bl-d">6E 61 6D 65</span> <span class="bl bl-t">A4</span> <span class="bl bl-d">6A 73 6F 6E</span></td><td>key: fixstr of length 4; value: fixstr of length 4</td></tr>
<tr><td><code>&quot;pi&quot;: 3.14,</code></td><td><span class="bl bl-t">A2</span> <span class="bl bl-d">70 69</span> <span class="bl bl-t">CB</span> <span class="bl bl-d">40 09 1E B8 51 EB 85 1F</span></td><td>key: fixstr of length 2; value: float 64 (IEEE 754, big-endian)</td></tr>
<tr><td><code>&quot;tags&quot;: [</code></td><td><span class="bl bl-t">A4</span> <span class="bl bl-d">74 61 67 73</span> <span class="bl bl-t">92</span></td><td>key: fixstr of length 4; value: fixarray with 2 elements</td></tr>
<tr><td><code>&quot;a&quot;,</code></td><td><span class="bl bl-t">A1</span> <span class="bl bl-d">61</span></td><td>fixstr of length 1</td></tr>
<tr><td><code>&quot;b&quot;</code></td><td><span class="bl bl-t">A1</span> <span class="bl bl-d">62</span></td><td>fixstr of length 1</td></tr>
<tr><td><code>]</code></td><td></td><td>no end marker: the element count was given up front</td></tr>
<tr><td><code>}</code></td><td></td><td>no end marker: the member count was given up front</td></tr>
</tbody></table></div>
## Deserialization
The library maps MessagePack types to JSON value types as follows:
docs/mkdocs/docs/features/binary_formats/ubjson.md
+33 -35
View File
@@ -7,6 +7,39 @@ achieve the generality of JSON, combined with being much easier to process than
- [UBJSON Website](http://ubjson.org)
## Byte layout
UBJSON is built from just a one-byte, human-readable ASCII *type marker* (for example, `i` for int8, `S` for string, `[` for array) followed by the payload of that type. Strings and the like are prefixed with their length, which is itself a UBJSON integer value including its marker. Numbers are stored in big-endian byte order. Containers can be written in two ways: as an *unoptimized* container that is opened with `[` or `{` and closed with `]` or `}`, or as an *optimized* container that declares an element type (`$`) and/or a count (`#`) after the opening marker, which saves per-element markers and (for a count) the closing marker.
Consider the following JSON value (shown formatted for readability):
```json
{
"id": 300,
"name": "json",
"pi": 3.14,
"tags": [
"a",
"b"
]
}
```
`to_ubjson` encodes it to 51 bytes (shown in hexadecimal; the encoding contains no whitespace). The colors mark the role of each byte: <span class="bl bl-t">type/marker</span> <span class="bl bl-l">length/count/size</span> <span class="bl bl-d">data</span>.
<div class="byte-layout" markdown="0"><table>
<thead><tr><th>JSON</th><th>Bytes</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>{</code></td><td><span class="bl bl-t">7B</span></td><td>object start (<code>{</code>)</td></tr>
<tr><td><code>&quot;id&quot;: 300,</code></td><td><span class="bl bl-l">69 02</span> <span class="bl bl-d">69 64</span> <span class="bl bl-t">49</span> <span class="bl bl-d">01 2C</span></td><td>key: length 2 as int8 (<code>i</code>), then UTF-8 bytes; value: int16 (<code>I</code>), big-endian</td></tr>
<tr><td><code>&quot;name&quot;: &quot;json&quot;,</code></td><td><span class="bl bl-l">69 04</span> <span class="bl bl-d">6E 61 6D 65</span> <span class="bl bl-t">53</span> <span class="bl bl-l">69 04</span> <span class="bl bl-d">6A 73 6F 6E</span></td><td>key: length 4; value: string (<code>S</code>), length 4 as int8, UTF-8 bytes</td></tr>
<tr><td><code>&quot;pi&quot;: 3.14,</code></td><td><span class="bl bl-l">69 02</span> <span class="bl bl-d">70 69</span> <span class="bl bl-t">44</span> <span class="bl bl-d">40 09 1E B8 51 EB 85 1F</span></td><td>key: length 2; value: float64 (<code>D</code>), big-endian</td></tr>
<tr><td><code>&quot;tags&quot;: [</code></td><td><span class="bl bl-l">69 04</span> <span class="bl bl-d">74 61 67 73</span> <span class="bl bl-t">5B</span></td><td>key: length 4; value: array start (<code>[</code>)</td></tr>
<tr><td><code>&quot;a&quot;,</code></td><td><span class="bl bl-t">53</span> <span class="bl bl-l">69 01</span> <span class="bl bl-d">61</span></td><td>string, length 1</td></tr>
<tr><td><code>&quot;b&quot;</code></td><td><span class="bl bl-t">53</span> <span class="bl bl-l">69 01</span> <span class="bl bl-d">62</span></td><td>string, length 1</td></tr>
<tr><td><code>]</code></td><td><span class="bl bl-t">5D</span></td><td>array end (<code>]</code>)</td></tr>
<tr><td><code>}</code></td><td><span class="bl bl-t">7D</span></td><td>object end (<code>}</code>)</td></tr>
</tbody></table></div>
## Serialization
The library uses the following mapping from JSON values types to UBJSON types according to the UBJSON specification:
@@ -103,41 +136,6 @@ The library uses the following mapping from JSON values types to UBJSON types ac
--8<-- "examples/to_ubjson.output"
```
### Byte layout
Consider the following JSON value (shown formatted for readability):
```json
{
"id": 300,
"name": "json",
"pi": 3.14,
"tags": [
"a",
"b"
]
}
```
`to_ubjson` encodes it to 51 bytes (shown in hexadecimal; the encoding contains no whitespace). The colors mark the role of each byte: <span class="bl bl-t">type/marker</span> <span class="bl bl-l">length/count/size</span> <span class="bl bl-d">data</span>.
<div class="byte-layout" markdown="0"><table>
<thead><tr><th>JSON</th><th>Bytes</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>{</code></td><td><span class="bl bl-t">7B</span></td><td>object start (<code>{</code>)</td></tr>
<tr><td><code>&quot;id&quot;: 300,</code></td><td><span class="bl bl-l">69 02</span> <span class="bl bl-d">69 64</span> <span class="bl bl-t">49</span> <span class="bl bl-d">01 2C</span></td><td>key: length 2 as int8 (<code>i</code>), then UTF-8 bytes; value: int16 (<code>I</code>), big-endian</td></tr>
<tr><td><code>&quot;name&quot;: &quot;json&quot;,</code></td><td><span class="bl bl-l">69 04</span> <span class="bl bl-d">6E 61 6D 65</span> <span class="bl bl-t">53</span> <span class="bl bl-l">69 04</span> <span class="bl bl-d">6A 73 6F 6E</span></td><td>key: length 4; value: string (<code>S</code>), length 4 as int8, UTF-8 bytes</td></tr>
<tr><td><code>&quot;pi&quot;: 3.14,</code></td><td><span class="bl bl-l">69 02</span> <span class="bl bl-d">70 69</span> <span class="bl bl-t">44</span> <span class="bl bl-d">40 09 1E B8 51 EB 85 1F</span></td><td>key: length 2; value: float64 (<code>D</code>), big-endian</td></tr>
<tr><td><code>&quot;tags&quot;: [</code></td><td><span class="bl bl-l">69 04</span> <span class="bl bl-d">74 61 67 73</span> <span class="bl bl-t">5B</span></td><td>key: length 4; value: array start (<code>[</code>)</td></tr>
<tr><td><code>&quot;a&quot;,</code></td><td><span class="bl bl-t">53</span> <span class="bl bl-l">69 01</span> <span class="bl bl-d">61</span></td><td>string, length 1</td></tr>
<tr><td><code>&quot;b&quot;</code></td><td><span class="bl bl-t">53</span> <span class="bl bl-l">69 01</span> <span class="bl bl-d">62</span></td><td>string, length 1</td></tr>
<tr><td><code>]</code></td><td><span class="bl bl-t">5D</span></td><td>array end (<code>]</code>)</td></tr>
<tr><td><code>}</code></td><td><span class="bl bl-t">7D</span></td><td>object end (<code>}</code>)</td></tr>
</tbody></table></div>
## Deserialization
The library maps UBJSON types to JSON value types as follows:
docs/mkdocs/includes/glossary.md
+27 -2
View File
@@ -1,5 +1,7 @@
<!-- https://squidfunk.github.io/mkdocs-material/reference/tooltips/#adding-a-glossary -->
*[AArch64]: 64-bit execution state of the Arm architecture
*[ABI]: Application Binary Interface
*[ADL]: Argument-dependent lookup
*[API]: Application Programming Interface
*[ASCII]: American Standard Code for Information Interchange
@@ -10,28 +12,51 @@
*[CBOR]: Concise Binary Object Representation
*[CC0]: Creative Commons Zero
*[CI]: Continuous Integration
*[CPU]: Central Processing Unit
*[CUDA]: Compute Unified Device Architecture
*[CVE]: Common Vulnerabilities and Exposures
*[DOM]: Document Object Model
*[EOF]: End of File
*[FAQ]: Frequently Asked Questions
*[GCC]: GNU Compiler Collection
*[GDB]: GNU Debugger
*[GiB]: Gibibyte (2^30 bytes)
*[HTTP]: Hypertext Transfer Protocol
*[ICC]: Intel C++ Compiler
*[IDE]: Integrated Development Environment
*[IEEE]: Institute of Electrical and Electronics Engineers
*[ISO]: International Organization for Standardization
*[IWYU]: Include What You Use
*[JSON]: JavaScript Object Notation
*[JSONC]: JSON with Comments
*[JWCC]: JSON With Commas and Comments
*[LLDB]: LLVM Debugger
*[LTS]: Long-Term Support
*[MinGW]: Minimalist GNU for Windows
*[MIT]: Massachusetts Institute of Technology
*[MSVC]: Microsoft Visual C++
*[MsgPack]: MessagePack
*[MSVC]: Microsoft Visual C++
*[NaN]: Not a Number
*[NASA]: National Aeronautics and Space Administration
*[NDK]: Native Development Kit
*[NaN]: Not a Number
*[NEON]: Arm SIMD instruction set extension
*[NUL]: Null character (U+0000, byte value 0)
*[ODR]: One Definition Rule
*[REUSE]: Software licensing specification by the Free Software Foundation Europe
*[RFC]: Request for Comments
*[RTTI]: Runtime Type Information
*[SAX]: Simple API for XML
*[SDK]: Software Development Kit
*[SFINAE]: Substitution failure is not an error
*[SGX]: Software Guard Extensions
*[SHA]: Secure Hash Algorithm
*[SIMD]: Single Instruction, Multiple Data
*[SPDX]: Software Package Data Exchange
*[SSE2]: Streaming SIMD Extensions 2
*[SSSE3]: Supplemental Streaming SIMD Extensions 3
*[STL]: Standard Template Library
*[UBJSON]: Universal Binary JSON
*[UDL]: User-defined literal
*[UDLs]: User-defined literals
*[URL]: Uniform Resource Locator
*[UTF]: Unicode Transformation Format