docs: byte layout sections, glossary entries, and QA page update (#5910)
* docs: byte layout sections, glossary, and QA page updates - Move "Byte layout" of each binary format to its own section before "Serialization" and describe the idea of the format. - Glossary: add LLDB and other terms used throughout the documentation. - quality_assurance: correct Clang Static Analyzer and CMake entries; add CodeQL, Semgrep, Flawfinder, CIFuzz, OpenSSF Scorecard, REUSE, Meson/SwiftPM, and reproducibility checks. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: drop unused length colour from CBOR and MessagePack byte layout legends Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann authored and GitHub committed
1 parent
330fbead59
commit
1cf5492bcd
8 files changed
+252
-216
No files matched your search
@@ -10,6 +10,39 @@ values -1.0, 0.0, and 1.0 take a single byte, and arrays and objects with up to
|
||||
- [BON8 specification](https://github.com/hikoworks/hikogui/blob/main/docs/BON8.md)
|
||||
- [Reference implementation](https://github.com/hikoworks/hikogui/blob/main/src/hikogui/codec/BON8.hpp) in HikoGUI
|
||||
|
||||
## Byte layout
|
||||
|
||||
BON8 exploits the fact that some byte values can never start a UTF-8 character. These bytes serve as *type markers*, so strings can be stored as plain UTF-8 without a length prefix: a string ends at the first byte that cannot continue it (often the marker of the next value). Small integers, `true`, `false`, `null`, and a few common floats are a single byte. Arrays and objects are introduced by a marker; those with up to four elements encode their size in the marker and need no terminator, while larger ones are closed by the end marker `0xFE`.
|
||||
|
||||
Consider the following JSON value (shown formatted for readability):
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 300,
|
||||
"name": "json",
|
||||
"pi": 3.14,
|
||||
"tags": [
|
||||
"a",
|
||||
"b"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`to_bon8` encodes it to 35 bytes (shown in hexadecimal; the encoding contains no whitespace). The colors mark the role of each byte: <span class="bl bl-t">type/marker</span> <span class="bl bl-l">length/count/size</span> <span class="bl bl-d">data</span>.
|
||||
|
||||
<div class="byte-layout" markdown="0"><table>
|
||||
<thead><tr><th>JSON</th><th>Bytes</th><th>Meaning</th></tr></thead><tbody>
|
||||
<tr><td><code>{</code></td><td><span class="bl bl-t">8A</span></td><td>object with 4 members</td></tr>
|
||||
<tr><td><code>"id": 300,</code></td><td><span class="bl bl-d">69 64</span> <span class="bl bl-t">C4</span> <span class="bl bl-d">04</span></td><td>key: UTF-8 bytes; the string ends at the next byte, which cannot continue a UTF-8 character; value: 2-byte positive integer (lead byte, then a byte below 0x80)</td></tr>
|
||||
<tr><td><code>"name": "json",</code></td><td><span class="bl bl-d">6E 61 6D 65</span> <span class="bl bl-l">FF</span> <span class="bl bl-d">6A 73 6F 6E</span> <span class="bl bl-l">FF</span></td><td>key: UTF-8 bytes; 0xFF terminates the string as another string follows; value: UTF-8 bytes; terminated as a string key follows</td></tr>
|
||||
<tr><td><code>"pi": 3.14,</code></td><td><span class="bl bl-d">70 69</span> <span class="bl bl-t">8F</span> <span class="bl bl-d">40 09 1E B8 51 EB 85 1F</span></td><td>key: UTF-8 bytes; value: binary64 (IEEE 754, big-endian); its marker also ends <code>"pi"</code></td></tr>
|
||||
<tr><td><code>"tags": [</code></td><td><span class="bl bl-d">74 61 67 73</span> <span class="bl bl-t">82</span></td><td>key: UTF-8 bytes; value: array with 2 elements; its marker also ends <code>"tags"</code></td></tr>
|
||||
<tr><td><code>"a",</code></td><td><span class="bl bl-d">61</span> <span class="bl bl-l">FF</span></td><td>UTF-8 bytes; terminated as a string follows</td></tr>
|
||||
<tr><td><code>"b"</code></td><td><span class="bl bl-d">62</span> <span class="bl bl-l">FF</span></td><td>UTF-8 bytes; terminated as nothing follows</td></tr>
|
||||
<tr><td><code>]</code></td><td></td><td>no end marker: the element count was given up front</td></tr>
|
||||
<tr><td><code>}</code></td><td></td><td>no end marker: the member count was given up front</td></tr>
|
||||
</tbody></table></div>
|
||||
|
||||
## Serialization
|
||||
|
||||
The library uses the following mapping from JSON values types to BON8 types according to the BON8 specification:
|
||||
@@ -105,41 +138,6 @@ serialized as 0x81 0x65 0xFF, but `[1,2,3,4,"e"]` as 0x85 0x91 0x92 0x93 0x94 0x
|
||||
--8<-- "examples/to_bon8.output"
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
### Byte layout
|
||||
|
||||
Consider the following JSON value (shown formatted for readability):
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 300,
|
||||
"name": "json",
|
||||
"pi": 3.14,
|
||||
"tags": [
|
||||
"a",
|
||||
"b"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`to_bon8` encodes it to 35 bytes (shown in hexadecimal; the encoding contains no whitespace). The colors mark the role of each byte: <span class="bl bl-t">type/marker</span> <span class="bl bl-l">length/count/size</span> <span class="bl bl-d">data</span>.
|
||||
|
||||
<div class="byte-layout" markdown="0"><table>
|
||||
<thead><tr><th>JSON</th><th>Bytes</th><th>Meaning</th></tr></thead><tbody>
|
||||
<tr><td><code>{</code></td><td><span class="bl bl-t">8A</span></td><td>object with 4 members</td></tr>
|
||||
<tr><td><code>"id": 300,</code></td><td><span class="bl bl-d">69 64</span> <span class="bl bl-t">C4</span> <span class="bl bl-d">04</span></td><td>key: UTF-8 bytes; the string ends at the next byte, which cannot continue a UTF-8 character; value: 2-byte positive integer (lead byte, then a byte below 0x80)</td></tr>
|
||||
<tr><td><code>"name": "json",</code></td><td><span class="bl bl-d">6E 61 6D 65</span> <span class="bl bl-l">FF</span> <span class="bl bl-d">6A 73 6F 6E</span> <span class="bl bl-l">FF</span></td><td>key: UTF-8 bytes; 0xFF terminates the string as another string follows; value: UTF-8 bytes; terminated as a string key follows</td></tr>
|
||||
<tr><td><code>"pi": 3.14,</code></td><td><span class="bl bl-d">70 69</span> <span class="bl bl-t">8F</span> <span class="bl bl-d">40 09 1E B8 51 EB 85 1F</span></td><td>key: UTF-8 bytes; value: binary64 (IEEE 754, big-endian); its marker also ends <code>"pi"</code></td></tr>
|
||||
<tr><td><code>"tags": [</code></td><td><span class="bl bl-d">74 61 67 73</span> <span class="bl bl-t">82</span></td><td>key: UTF-8 bytes; value: array with 2 elements; its marker also ends <code>"tags"</code></td></tr>
|
||||
<tr><td><code>"a",</code></td><td><span class="bl bl-d">61</span> <span class="bl bl-l">FF</span></td><td>UTF-8 bytes; terminated as a string follows</td></tr>
|
||||
<tr><td><code>"b"</code></td><td><span class="bl bl-d">62</span> <span class="bl bl-l">FF</span></td><td>UTF-8 bytes; terminated as nothing follows</td></tr>
|
||||
<tr><td><code>]</code></td><td></td><td>no end marker: the element count was given up front</td></tr>
|
||||
<tr><td><code>}</code></td><td></td><td>no end marker: the member count was given up front</td></tr>
|
||||
</tbody></table></div>
|
||||
|
||||
## Deserialization
|
||||
|
||||
The library maps BON8 types to JSON value types as follows:
|
||||
|
||||
Reference in new issue
Block a user