Niels Lohmann
73d116547a
to_cbor(), to_ubjson(), to_bjdata(), and to_bson() wrote a string or object
key with ill-formed UTF-8 byte for byte, but from_cbor()/from_ubjson()/
from_bjdata()/from_bson() reject such strings with parse_error.113 since
d19f7f5dc (#5185, not yet released): a value that serialized without error
could not be read back by the same library (#5651).
CBOR (RFC 8949 Section 5), UBJSON, BJData, and BSON all require text strings
and object keys/element names to be valid UTF-8, so their writers now
validate and throw type_error.316, like to_bon8() already does. For BSON,
the check runs in the size-computation pass, before any byte is written, the
same way the binary subtype check works. check_bon8_utf8() is renamed to
check_utf8() since it is now shared by all of these writers.
MessagePack's specification explicitly allows a str object to contain an
invalid byte sequence and expects a deserializer to hand the bytes back
unchanged, so its writer is unaffected and from_msgpack() (object keys
included) no longer validates UTF-8, restoring its pre-#5185 behavior.
dump() still rejects ill-formed UTF-8 with type_error.316 unless an error
handler is passed.
Docs: add the type_error.316 exception to to_cbor/to_ubjson/to_bjdata/to_bson,
document the writer-side check on the cbor/bson/ubjson/bjdata format pages,
and rewrite the MessagePack UTF-8 warning to describe the round-trip and
dump() behavior instead of a validation requirement the spec does not have.
Tests: add ill-formed value/key cases (invalid byte, truncated sequence,
encoded surrogate, overlong encoding) expecting type_error.316 to
unit-cbor.cpp, unit-ubjson.cpp, unit-bjdata.cpp, and unit-bson.cpp (which
also checks the output vector stays empty), and turn unit-msgpack.cpp's
former parse_error.113 ill-formed-UTF-8 tests into byte-for-byte round-trip
tests for both values and keys.
This text was written by Claude Code.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
6.6 KiB
6.6 KiB
BSON
BSON, short for Binary JSON, is a binary-encoded serialization of JSON-like documents. Like JSON, BSON supports the embedding of documents and arrays within other documents and arrays. BSON also contains extensions that allow representation of data types that are not part of the JSON spec. For example, BSON has a Date type and a BinData type.
!!! abstract "References"
- [BSON Website](http://bsonspec.org) - the main source on BSON
- [BSON Specification](http://bsonspec.org/spec.html) - the specification
Serialization
The library uses the following mapping from JSON values types to BSON types:
| JSON value type | value/range | BSON type | marker |
|---|---|---|---|
| null | null |
null | 0x0A |
| boolean | true, false |
boolean | 0x08 |
| number_integer | -9223372036854775808..-2147483649 | int64 | 0x12 |
| number_integer | -2147483648..2147483647 | int32 | 0x10 |
| number_integer | 2147483648..9223372036854775807 | int64 | 0x12 |
| number_unsigned | 0..2147483647 | int32 | 0x10 |
| number_unsigned | 2147483648..9223372036854775807 | int64 | 0x12 |
| number_unsigned | 9223372036854775808..18446744073709551615 | uint64 | 0x11 |
| number_float | any value | double | 0x01 |
| string | any value | string | 0x02 |
| array | any value | document | 0x04 |
| object | any value | document | 0x03 |
| binary | any value | binary | 0x05 |
!!! warning "Incomplete mapping"
The mapping is **incomplete**, since only JSON-objects (and things contained therein) can be serialized to BSON.
Also, keys may not contain U+0000, since they are serialized a zero-terminated c-strings.
!!! warning "BSON type 0x11 interoperability"
The BSON specification defines type `0x11` as a Timestamp. This library uses marker `0x11` when serializing
`number_unsigned` values in the range `9223372036854775808..18446744073709551615`. Other BSON implementations may
therefore interpret these values as Timestamps instead of unsigned integers.
!!! info "Binary values without a subtype"
BSON requires every binary value to have a subtype. If a binary value has no subtype, this library serializes it
with the generic subtype `0x00`. After deserialization, `has_subtype()` returns `true` and `subtype()` returns `0`.
As a result, serializing and deserializing a JSON object containing such a value produces a different JSON object,
even though the binary data is unchanged.
??? example
```cpp
--8<-- "examples/to_bson.cpp"
```
Output:
```c
--8<-- "examples/to_bson.output"
```
Deserialization
The library maps BSON record types to JSON value types as follows:
| BSON type | BSON marker byte | JSON value type |
|---|---|---|
| double | 0x01 | number_float |
| string | 0x02 | string |
| document | 0x03 | object |
| array | 0x04 | array |
| binary | 0x05 | binary |
| undefined | 0x06 | unsupported |
| ObjectId | 0x07 | unsupported |
| boolean | 0x08 | boolean |
| UTC Date-Time | 0x09 | unsupported |
| null | 0x0A | null |
| Regular Expr. | 0x0B | unsupported |
| DB Pointer | 0x0C | unsupported |
| JavaScript Code | 0x0D | unsupported |
| Symbol | 0x0E | unsupported |
| JavaScript Code w/ scope | 0x0F | unsupported |
| int32 | 0x10 | number_integer |
| uint64(Timestamp) | 0x11 | number_unsigned |
| int64 | 0x12 | number_integer |
| 128-bit decimal float | 0x13 | unsupported |
| Max Key | 0x7F | unsupported |
| Min Key | 0xFF | unsupported |
!!! warning "Incomplete mapping"
The mapping is **incomplete**. The unsupported mappings are indicated in the table above.
!!! note "Handling of BSON type 0x11"
This library deserializes BSON type `0x11` (Timestamp) as a `number_unsigned` value. The 64-bit value is preserved,
but the Timestamp type information is not.
!!! warning "Lenient BSON input handling"
The BSON reader is lenient in a few areas where the BSON specification is more restrictive:
- array element keys are not checked against the required decimal sequence (`0`, `1`, `2`, ...),
- any non-zero byte is accepted as `true` for the boolean type, and
- the payload for binary subtype `0x02` is returned as-is, including its inner length prefix.
If BSON input must be validated for strict specification compliance, validate it separately before passing it to
`from_bson()`.
!!! warning "UTF-8 validation of string values"
The BSON specification requires `string` values (type `0x02`) to be valid UTF-8. This library validates the
bytes of every such string at decode time and rejects ill-formed UTF-8 with a
[`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) exception (or, with `allow_exceptions`
set to `false`, a discarded value), rather than only failing later when the resulting value is dumped. Element
(key) names and `binary` values (type `0x05`) are unaffected and are never validated on read, since they are read
byte-by-byte as a C string, or are not required to hold text, respectively. `to_bson()` validates both string
values and element names and throws
[`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316) for ill-formed UTF-8 in either, so an
object with such a key or value cannot be produced in the first place, even though `from_bson()` would accept it
from another source.
??? example
```cpp
--8<-- "examples/from_bson.cpp"
```
Output:
```json
--8<-- "examples/from_bson.output"
```