245 lines
15 KiB
Markdown
245 lines
15 KiB
Markdown
# MessagePack
|
||
|
||
MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves.
|
||
|
||
References
|
||
|
||
- [MessagePack website](https://msgpack.org)
|
||
- [MessagePack specification](https://github.com/msgpack/msgpack/blob/master/spec.md)
|
||
|
||
## Byte layout
|
||
|
||
MessagePack encodes each value as a one-byte *format marker*, optionally followed by a length or the payload. The marker range is partitioned to keep common values small: small non-negative integers (0 to 127) and small negative integers (-32 to -1) are the marker byte itself; short strings, arrays, and maps (up to 31 bytes for strings and up to 15 elements for arrays and maps) hide their length in the marker's lower bits. Larger values use dedicated markers that announce the width (8, 16, 32, or 64 bits) of the length or number that follows, in big-endian byte order. Like CBOR, containers are length-prefixed, so no terminator is needed.
|
||
|
||
Consider the following JSON value (shown formatted for readability):
|
||
|
||
```
|
||
{
|
||
"id": 300,
|
||
"name": "json",
|
||
"pi": 3.14,
|
||
"tags": [
|
||
"a",
|
||
"b"
|
||
]
|
||
}
|
||
```
|
||
|
||
`to_msgpack` encodes it to 39 bytes (shown in hexadecimal; the encoding contains no whitespace). The colors mark the role of each byte: type/marker data.
|
||
|
||
| JSON | Bytes | Meaning |
|
||
| ----------------- | ----------------------------------- | -------------------------------------------------------------------------- |
|
||
| `{` | 84 | map with 4 key/value pairs |
|
||
| `"id": 300,` | A2 69 64 CD 01 2C | key: fixstr of length 2, then its UTF-8 bytes; value: uint 16 (big-endian) |
|
||
| `"name": "json",` | A4 6E 61 6D 65 A4 6A 73 6F 6E | key: fixstr of length 4; value: fixstr of length 4 |
|
||
| `"pi": 3.14,` | A2 70 69 CB 40 09 1E B8 51 EB 85 1F | key: fixstr of length 2; value: float 64 (IEEE 754, big-endian) |
|
||
| `"tags": [` | A4 74 61 67 73 92 | key: fixstr of length 4; value: fixarray with 2 elements |
|
||
| `"a",` | A1 61 | fixstr of length 1 |
|
||
| `"b"` | A1 62 | fixstr of length 1 |
|
||
| `]` | | no end marker: the element count was given up front |
|
||
| `}` | | no end marker: the member count was given up front |
|
||
|
||
## Serialization
|
||
|
||
The library uses the following mapping from JSON values types to MessagePack types according to the MessagePack specification:
|
||
|
||
| JSON value type | value/range | MessagePack type | first byte |
|
||
| --------------- | ---------------------------------------- | ---------------- | ---------- |
|
||
| null | `null` | nil | 0xC0 |
|
||
| boolean | `true` | true | 0xC3 |
|
||
| boolean | `false` | false | 0xC2 |
|
||
| number_integer | -9223372036854775808..-2147483649 | int64 | 0xD3 |
|
||
| number_integer | -2147483648..-32769 | int32 | 0xD2 |
|
||
| number_integer | -32768..-129 | int16 | 0xD1 |
|
||
| number_integer | -128..-33 | int8 | 0xD0 |
|
||
| number_integer | -32..-1 | negative fixint | 0xE0..0xFF |
|
||
| number_integer | 0..127 | positive fixint | 0x00..0x7F |
|
||
| number_integer | 128..255 | uint 8 | 0xCC |
|
||
| number_integer | 256..65535 | uint 16 | 0xCD |
|
||
| number_integer | 65536..4294967295 | uint 32 | 0xCE |
|
||
| number_integer | 4294967296..18446744073709551615 | uint 64 | 0xCF |
|
||
| number_unsigned | 0..127 | positive fixint | 0x00..0x7F |
|
||
| number_unsigned | 128..255 | uint 8 | 0xCC |
|
||
| number_unsigned | 256..65535 | uint 16 | 0xCD |
|
||
| number_unsigned | 65536..4294967295 | uint 32 | 0xCE |
|
||
| number_unsigned | 4294967296..18446744073709551615 | uint 64 | 0xCF |
|
||
| number_float | *any value representable by a float* | float 32 | 0xCA |
|
||
| number_float | *any value NOT representable by a float* | float 64 | 0xCB |
|
||
| string | *length*: 0..31 | fixstr | 0xA0..0xBF |
|
||
| string | *length*: 32..255 | str 8 | 0xD9 |
|
||
| string | *length*: 256..65535 | str 16 | 0xDA |
|
||
| string | *length*: 65536..4294967295 | str 32 | 0xDB |
|
||
| array | *size*: 0..15 | fixarray | 0x90..0x9F |
|
||
| array | *size*: 16..65535 | array 16 | 0xDC |
|
||
| array | *size*: 65536..4294967295 | array 32 | 0xDD |
|
||
| object | *size*: 0..15 | fix map | 0x80..0x8F |
|
||
| object | *size*: 16..65535 | map 16 | 0xDE |
|
||
| object | *size*: 65536..4294967295 | map 32 | 0xDF |
|
||
| binary | *size*: 0..255 | bin 8 | 0xC4 |
|
||
| binary | *size*: 256..65535 | bin 16 | 0xC5 |
|
||
| binary | *size*: 65536..4294967295 | bin 32 | 0xC6 |
|
||
|
||
Complete mapping
|
||
|
||
The mapping is **complete** in the sense that any JSON value type can be converted to a MessagePack value.
|
||
|
||
Any MessagePack output created by `to_msgpack` can be successfully parsed by `from_msgpack`.
|
||
|
||
Size constraints
|
||
|
||
The following values can **not** be converted to a MessagePack value:
|
||
|
||
- strings with more than 4294967295 bytes
|
||
- byte strings with more than 4294967295 bytes
|
||
- arrays with more than 4294967295 elements
|
||
- objects with more than 4294967295 elements
|
||
|
||
Serializing such a value throws [`out_of_range.412`](https://json.nlohmann.me/home/exceptions/#jsonexceptionout_of_range412).
|
||
|
||
NaN/infinity handling
|
||
|
||
`NaN`, `Infinity`, and `-Infinity` are serialized as a MessagePack float 32 (type 0xCA, 5 bytes total), regardless of magnitude, in contrast to the [dump](https://json.nlohmann.me/api/basic_json/dump/index.md) function which serializes NaN or Infinity to `null`.
|
||
|
||
Note
|
||
|
||
Prior to version 3.13.0 unreleased, NaN and Infinity were instead serialized as a MessagePack float 64 (type 0xCB, 9 bytes total), because the check used to select the smaller float 32 encoding failed for NaN (all comparisons are `false`) and for Infinity and -Infinity (outside the float range checked against `std::numeric_limits<float>::lowest()` and `max()`), so the float 32 path was skipped.
|
||
|
||
Example: serialize a JSON value to MessagePack
|
||
|
||
```
|
||
#include <iostream>
|
||
#include <iomanip>
|
||
#include <nlohmann/json.hpp>
|
||
|
||
using nlohmann::json;
|
||
using namespace nlohmann::literals;
|
||
|
||
int main()
|
||
{
|
||
// create a JSON value
|
||
json j = R"({"compact": true, "schema": 0})"_json;
|
||
|
||
// serialize it to MessagePack
|
||
std::vector<std::uint8_t> v = json::to_msgpack(j);
|
||
|
||
// print the vector content
|
||
for (auto& byte : v)
|
||
{
|
||
std::cout << "0x" << std::hex << std::setw(2) << std::setfill('0') << static_cast<int>(byte) << " ";
|
||
}
|
||
std::cout << '\n';
|
||
}
|
||
```
|
||
|
||
Output:
|
||
|
||
```
|
||
0x82 0xa7 0x63 0x6f 0x6d 0x70 0x61 0x63 0x74 0xc3 0xa6 0x73 0x63 0x68 0x65 0x6d 0x61 0x00
|
||
```
|
||
|
||
## Deserialization
|
||
|
||
The library maps MessagePack types to JSON value types as follows:
|
||
|
||
| MessagePack type | JSON value type | first byte |
|
||
| ---------------- | --------------- | ---------- |
|
||
| positive fixint | number_unsigned | 0x00..0x7F |
|
||
| fixmap | object | 0x80..0x8F |
|
||
| fixarray | array | 0x90..0x9F |
|
||
| fixstr | string | 0xA0..0xBF |
|
||
| nil | `null` | 0xC0 |
|
||
| (never used) | – | 0xC1 |
|
||
| false | `false` | 0xC2 |
|
||
| true | `true` | 0xC3 |
|
||
| float 32 | number_float | 0xCA |
|
||
| float 64 | number_float | 0xCB |
|
||
| uint 8 | number_unsigned | 0xCC |
|
||
| uint 16 | number_unsigned | 0xCD |
|
||
| uint 32 | number_unsigned | 0xCE |
|
||
| uint 64 | number_unsigned | 0xCF |
|
||
| int 8 | number_integer | 0xD0 |
|
||
| int 16 | number_integer | 0xD1 |
|
||
| int 32 | number_integer | 0xD2 |
|
||
| int 64 | number_integer | 0xD3 |
|
||
| str 8 | string | 0xD9 |
|
||
| str 16 | string | 0xDA |
|
||
| str 32 | string | 0xDB |
|
||
| array 16 | array | 0xDC |
|
||
| array 32 | array | 0xDD |
|
||
| map 16 | object | 0xDE |
|
||
| map 32 | object | 0xDF |
|
||
| bin 8 | binary | 0xC4 |
|
||
| bin 16 | binary | 0xC5 |
|
||
| bin 32 | binary | 0xC6 |
|
||
| ext 8 | binary | 0xC7 |
|
||
| ext 16 | binary | 0xC8 |
|
||
| ext 32 | binary | 0xC9 |
|
||
| fixext 1 | binary | 0xD4 |
|
||
| fixext 2 | binary | 0xD5 |
|
||
| fixext 4 | binary | 0xD6 |
|
||
| fixext 8 | binary | 0xD7 |
|
||
| fixext 16 | binary | 0xD8 |
|
||
| negative fixint | number_integer | 0xE0-0xFF |
|
||
|
||
The byte 0xC1 is reserved as "never used" by the specification. As a value it is rejected with [`parse_error.112`](https://json.nlohmann.me/home/exceptions/#jsonexceptionparse_error112) (`invalid byte: 0xC1`); as a map key it is rejected with [`parse_error.113`](https://json.nlohmann.me/home/exceptions/#jsonexceptionparse_error113) and the generic "expected length specification" message, because the key is read as a string.
|
||
|
||
Duplicate keys in a map are accepted; the last value wins.
|
||
|
||
### Extension types (ext, fixext)
|
||
|
||
MessagePack ext and fixext values are read as binary values, and the ext type byte is stored as the binary subtype. Although the specification defines the type as a signed 8-bit integer (0 to 127 for applications, -128 to -1 reserved, with -1 being the Timestamp extension), the library reads it as an unsigned byte: a negative ext type `n` becomes subtype `256+n`, so ext type -1 is read as subtype 255 and -128 as subtype 128. The Timestamp extension is not interpreted; it is returned as binary with subtype 255. Use [`has_subtype()`](https://json.nlohmann.me/api/byte_container_with_subtype/has_subtype/index.md) to tell a `bin` value (no subtype) from an ext value (subtype, even if it is 0).
|
||
|
||
When serializing, a binary value with a subtype is always written as fixext (1, 2, 4, 8, or 16 bytes) or ext, even for subtype 0 (`d6 00 01 02 03 04`); a binary value without a subtype is written as bin. Subtypes 128 to 255 are written as the reserved negative ext types (255 as -1), so `json::binary({1,2,3,4}, 255)` becomes `d6 ff 01 02 03 04`, which other MessagePack libraries decode as a 32-bit timestamp. A subtype above 255 throws [`out_of_range.415`](https://json.nlohmann.me/home/exceptions/#jsonexceptionout_of_range415). The payload length is limited to 4294967295 bytes like all other lengths.
|
||
|
||
Info
|
||
|
||
Any MessagePack output created by `to_msgpack` can be successfully parsed by `from_msgpack`.
|
||
|
||
Object keys
|
||
|
||
MessagePack allows map keys of any type, whereas JSON only allows strings as keys in object values. Like the JSON-compatible [profile](https://github.com/msgpack/msgpack/blob/master/spec.md#profile) sketched in the MessagePack specification, this library restricts map keys to `str` values. Maps with keys of any other type are rejected with a [`parse_error.113`](https://json.nlohmann.me/home/exceptions/#jsonexceptionparse_error113) exception (or, with `allow_exceptions` set to `false`, a discarded value). For most key types the message names the type that was found, for instance:
|
||
|
||
```
|
||
[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing MessagePack object key: only string keys are supported, but found nil; last byte: 0xC0
|
||
```
|
||
|
||
This applies to the [SAX interface](https://json.nlohmann.me/features/parsing/sax_interface/index.md) as well, as the key is read before it is passed on. Such input needs a general-purpose MessagePack library instead.
|
||
|
||
Ill-formed UTF-8 in string values
|
||
|
||
The MessagePack specification explicitly allows a `str` value (`fixstr`, `str 8`, `str 16`, `str 32`) to contain a byte sequence that is not valid UTF-8, and recommends that a deserializer provide a way to obtain the original bytes. This library follows that by default: with its [`error_handler`](https://json.nlohmann.me/api/basic_json/from_msgpack/index.md) parameter left at `keep` (the default), `from_msgpack()` reads `str` bytes (object keys included) as-is, without validating them, so such a value round-trips through `from_msgpack(to_msgpack(j))` byte for byte. Passing `error_handler_t::strict` makes `from_msgpack()` check anyway and throw [`parse_error.113`](https://json.nlohmann.me/home/exceptions/#jsonexceptionparse_error113) for ill-formed UTF-8, and `replace`/`ignore` sanitize the string instead of keeping it. `to_msgpack()` also writes `str` bytes as-is by default, since the specification permits it; its [`error_handler`](https://json.nlohmann.me/api/basic_json/to_msgpack/index.md) parameter can be set to `strict` to throw [`type_error.316`](https://json.nlohmann.me/home/exceptions/#jsonexceptiontype_error316) instead, or to `replace`/`ignore` to sanitize the string, for instance for a decoder that rejects ill-formed UTF-8. However, [`dump()`](https://json.nlohmann.me/api/basic_json/dump/index.md) still requires valid UTF-8 and throws [`type_error.316`](https://json.nlohmann.me/home/exceptions/#jsonexceptiontype_error316) for a value read this way with the default `keep` handler, unless an error handler is passed that replaces or ignores the ill-formed bytes.
|
||
|
||
Example: deserialize a JSON value from MessagePack
|
||
|
||
```
|
||
#include <iostream>
|
||
#include <iomanip>
|
||
#include <nlohmann/json.hpp>
|
||
|
||
using nlohmann::json;
|
||
|
||
int main()
|
||
{
|
||
// create byte vector
|
||
std::vector<std::uint8_t> v = {0x82, 0xa7, 0x63, 0x6f, 0x6d, 0x70, 0x61, 0x63,
|
||
0x74, 0xc3, 0xa6, 0x73, 0x63, 0x68, 0x65, 0x6d,
|
||
0x61, 0x00
|
||
};
|
||
|
||
// deserialize it with MessagePack
|
||
json j = json::from_msgpack(v);
|
||
|
||
// print the deserialized JSON value
|
||
std::cout << std::setw(2) << j << '\n';
|
||
}
|
||
```
|
||
|
||
Output:
|
||
|
||
```
|
||
{
|
||
"compact": true,
|
||
"schema": 0
|
||
}
|
||
```
|