# Architecture
This page gives a high-level overview of the library's architecture. It should help new contributors to get an idea of
the used concepts and where to make changes.
## Overview
The library is built around a single class template, [`nlohmann::basic_json`](../api/basic_json/index.md). A
`basic_json` value is a node in a tree of JSON values. All other components either create such a tree from an input
(parsing), write a tree to an output (serialization), or give access to it (iterators, JSON Pointer, conversions).
```mermaid
flowchart LR
input[/"input
(string, stream,
iterator range, file)"/]
ia["input adapter"]
lexer["lexer"]
parser["parser"]
breader["binary_reader"]
sax["SAX interface"]
value[("basic_json
value tree")]
serializer["serializer"]
bwriter["binary_writer"]
oa["output adapter"]
output[/"output
(string, stream,
vector)"/]
input --> ia
ia --> lexer --> parser --> sax
ia --> breader --> sax
sax --> value
value --> serializer --> oa
value --> bwriter --> oa
oa --> output
```
Data flow through the library: input runs left to right through an input adapter and the lexer and parser (JSON text) or the `binary_reader` (binary formats) into SAX events, which build the value tree; the `serializer` or the `binary_writer` writes the tree to an output adapter.
- **JSON text** is read by an [input adapter](#input-adapters), tokenized by the lexer, and turned into SAX events by the parser. - **Binary formats** (BJData, BON8, BSON, CBOR, MessagePack, UBJSON) are read by an input adapter and turned into the same SAX events by the `binary_reader`. - A [SAX consumer](#sax-interface) receives the events. The one used by [`parse`](../api/basic_json/parse.md) builds a `basic_json` value tree. - The `serializer` (JSON text) or the `binary_writer` (binary formats) writes a value tree to an [output adapter](#output-adapters). ## Source layout The public headers are in [`include/nlohmann`](https://github.com/nlohmann/json/tree/develop/include/nlohmann): - [`json.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/json.hpp) defines class [`basic_json`](../api/basic_json/index.md). - [`json_fwd.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/json_fwd.hpp) contains forward declarations. - [`adl_serializer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/adl_serializer.hpp), [`byte_container_with_subtype.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/byte_container_with_subtype.hpp), and [`ordered_map.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/ordered_map.hpp) define [`adl_serializer`](../api/adl_serializer/index.md), [`byte_container_with_subtype`](../api/byte_container_with_subtype/index.md), and [`ordered_map`](../api/ordered_map.md). - [`json_view.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/json_view.hpp) is a separate, optional header that defines [`basic_json_document`](../api/basic_json_document/index.md) and [`basic_json_view`](../api/basic_json_view/index.md), a flat-index, read-only, non-owning way to look at a parsed JSON text; see [Zero-copy JSON views](../features/json_view.md). It builds on `json.hpp` internals (it requires the same library version) and has its own `detail/view/` subdirectory. Everything else lives in [`detail/`](https://github.com/nlohmann/json/tree/develop/include/nlohmann/detail) and namespace `nlohmann::detail`, which is not part of the public API. Paths below are relative to `include/nlohmann`. | Component | Location | |-----------|----------| | Value type enumeration | [`detail/value_t.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/value_t.hpp) | | Input adapters | [`detail/input/input_adapters.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/input_adapters.hpp) | | Lexer | [`detail/input/lexer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/lexer.hpp), [`detail/input/number_parse.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/number_parse.hpp), [`detail/input/string_scan.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/string_scan.hpp), [`detail/input/simd_scan.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/simd_scan.hpp) | | Parser | [`detail/input/parser.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/parser.hpp) | | SAX interface and DOM builders | [`detail/input/json_sax.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/json_sax.hpp) | | Binary format readers | [`detail/input/binary_reader.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/binary_reader.hpp) | | JSON serializer | [`detail/output/serializer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/output/serializer.hpp), [`detail/conversions/to_chars.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/conversions/to_chars.hpp) | | Binary format writers | [`detail/output/binary_writer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/output/binary_writer.hpp) | | Output adapters | [`detail/output/output_adapters.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/output/output_adapters.hpp) | | Iterators | [`detail/iterators/`](https://github.com/nlohmann/json/tree/develop/include/nlohmann/detail/iterators) | | Conversions from/to arbitrary types | [`detail/conversions/from_json.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/conversions/from_json.hpp), [`detail/conversions/to_json.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/conversions/to_json.hpp) | | JSON Pointer | [`detail/json_pointer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/json_pointer.hpp) | | Exceptions | [`detail/exceptions.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/exceptions.hpp) | | Type traits and C++ feature backports | [`detail/meta/`](https://github.com/nlohmann/json/tree/develop/include/nlohmann/detail/meta) | | Macros | [`detail/macro_scope.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/macro_scope.hpp), [`detail/macro_unscope.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/macro_unscope.hpp), [`detail/abi_macros.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/abi_macros.hpp) | The single-header version [`single_include/nlohmann/json.hpp`](https://github.com/nlohmann/json/blob/develop/single_include/nlohmann/json.hpp) is generated from these files with `make amalgamate` and must not be edited by hand. The same command also generates [`single_include/nlohmann/json_view.hpp`](https://github.com/nlohmann/json/blob/develop/single_include/nlohmann/json_view.hpp) from `json_view.hpp` and `detail/view/`. ## Template parameters [`basic_json`](../api/basic_json/index.md) is parameterized by the types it uses to store values and to convert from and to other types: | Template parameter | Default | Used for | |----------------------|-----------------------------|-------------------------------------------------------------------| | `ObjectType` | `std::map` | objects, see [`object_t`](../api/basic_json/object_t.md) | | `ArrayType` | `std::vector` | arrays, see [`array_t`](../api/basic_json/array_t.md) | | `StringType` | `std::string` | strings and object keys, see [`string_t`](../api/basic_json/string_t.md) | | `BooleanType` | `bool` | Booleans, see [`boolean_t`](../api/basic_json/boolean_t.md) | | `NumberIntegerType` | `std::int64_t` | signed integers, see [`number_integer_t`](../api/basic_json/number_integer_t.md) | | `NumberUnsignedType` | `std::uint64_t` | unsigned integers, see [`number_unsigned_t`](../api/basic_json/number_unsigned_t.md) | | `NumberFloatType` | `double` | floating-point numbers, see [`number_float_t`](../api/basic_json/number_float_t.md) | | `AllocatorType` | `std::allocator` | allocating objects, arrays, strings, and binary values | | `JSONSerializer` | `adl_serializer` | conversions from/to other types, see [`adl_serializer`](../api/adl_serializer/index.md) | | `BinaryType` | `std::vectorLayout of one 16-byte node of a JSON view index. Each row is 32 bits wide, and the numbers are the bit offsets of the fields `kind`, `flags`, `extra`, `off`, `len`, and `next`.
| Bytes | Field | Type | Contents | |-------|---------|------------|-------------------------------------------------------------------------------------------------------------------------------| | 0 | `kind` | `uint8_t` | the type, numbered as [`value_t`](../api/basic_json/value_t.md): 0 null, 1 object, 2 array, 3 string, 4 boolean, 5 signed integer, 6 unsigned integer, 7 float; 10 for a link (see below). There is no node for a discarded value: a [discarded](../api/basic_json_view/is_discarded.md) view has none | | 1 | `flags` | `uint8_t` | bits 0-1: where a string's bytes (or a number's token) are (0: the source text, 1: the buffer of decoded strings, for strings with escapes, 2: the edit buffer); bit 2: the value of a boolean; bit 3: moved, bit 4: new, bit 5: linked (see below) | | 2-3 | `extra` | `uint16_t` | numbers: the number of integer digits (low byte) and fraction digits (high byte), 255 for more; objects: the number of their key hash index (1-based), or 0; otherwise 0 | | 4-7 | `off` | `uint32_t` | where the value starts: the first byte after a string's opening quote (or its position in the buffer of decoded strings), the first byte of a number or literal, the bracket of an array or object | | 8-11 | `len` | `uint32_t` | strings: the length after decoding; floats and literals: the length of the token; arrays and objects: the number of elements | | 12-15 | `next` | `uint32_t` | arrays and objects: the number of nodes of the subtree, including the node itself; strings with escapes: the position of the opening quote in the source plus one if the document was parsed with [`JSON_DIAGNOSTIC_POSITIONS`](../api/macros/json_diagnostic_positions.md), or 0 if it was not; all other values: 0 |Fields of a node, by their byte offset within the 16-byte node. Integers use bytes 8-15 differently, see below.
- **Integers** keep their converted value in bytes 8-15 instead of `len` and `next`; the length of their token follows from the number of digits in `extra` (and the sign). Non-negative integers are unsigned integers, as with [`parse`](../api/basic_json/parse.md). - **Floats** keep only their token. The digit layout in `extra` lets the conversion read the digits without scanning the token again, and only when the value is read. - **Object members** are the node of the key (a string) followed by the nodes of the value. - **Navigation** needs no pointers: the elements of an array or object follow its node, and the node after a value's subtree is `next` nodes further for an array or object, and the next node otherwise (`document_data::after`). Views step from element to element this way and skip whole subtrees in constant time. - **Offsets** are 32 bits wide, so a document is limited to 4294967279 bytes, 4 GiB minus 16 bytes (a margin below 2^32 for positions one scanner step past the end of the text; `out_of_range.416`). - **Source positions** ([`JSON_DIAGNOSTIC_POSITIONS`](../api/macros/json_diagnostic_positions.md)) are derived by [`materialize()`](../api/basic_json_view/materialize.md) and cost almost nothing while parsing: the only work is storing the offset of the opening quote of a string with escapes in its `next`, which is otherwise unused. A scalar's range follows from `off` and its length, a string with escapes is scanned from its opening quote (`next`) to the closing one, and the end of an array or object is found by scanning from the end of its last element over whitespace, comments, and a trailing comma. `materialize()` hands these positions to the SAX builder of `parse()` in place of its lexer. - **Large objects** (128 members or more) get a hash index for their keys after parsing ([`detail/view/object_index.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/view/object_index.hpp)): an open-addressing table whose slots hold the distance from the object's node to a key's node, so that a lookup does not compare every key. Of duplicate keys the table leads to the first, as a linear search does. The object's `extra` holds the number of its table. Only 65,535 tables fit into `extra`; objects beyond them are searched linearly. For example, `#!json {"a": [1, 2.5]}` becomes five nodes. Each node's elements follow it, and `next` leads from an array or object past its subtree: ```mermaid flowchart LR n0["0: objectThe five nodes of `#!json {"a": [1, 2.5]}` in document order. Solid arrows lead from each node to the one after it; dotted arrows show where `next` leads from the object and the array, past their subtrees.
| Node | `kind` | `extra` | `off` | `len` | `next` | |------|----------------------|---------|-------|-------|--------| | 0 | 1 (object) | 0 | 0 | 1 | 5 | | 1 | 3 (string) | 0 | 2 | 1 | 0 | | 2 | 2 (array) | 0 | 6 | 2 | 3 | | 3 | 6 (unsigned integer) | 0x0001 | 7 | - | - | | 4 | 7 (float) | 0x0101 | 10 | 3 | 0 |Fields of the five nodes of `#!json {"a": [1, 2.5]}`. `off` is the byte offset in the text, and - marks the bytes 8-15 that hold the integer's value instead of `len` and `next`.
All `flags` are 0. The integer's bytes 8-15 hold its value, 1; its `extra` says it has one digit. The float's `extra` says it has one integer and one fraction digit, and its `len` is that of the token `2.5`. Editable documents ([`json_editable_document`](../api/json_editable_document.md), [`detail/view/edit.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/view/edit.hpp) and [`detail/view/edit_storage.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/view/edit_storage.hpp)) never write the source text and never move or resize the parsed index, so views stay valid while the document is edited: - A new scalar is written over its node. Its text (a string, or the token of a number as `dump()` writes it) goes to the edit buffer, which `flags` bits 0-1 then name. - An array or object whose elements change gets the flag *moved* (bit 3): its elements then live in a separate sequence (a header node, then the entries), whose number is in `off`. The entries are links (`kind` 10), whose bytes 8-15 hold the address of the value's node, so values never move. - A node written by an edit gets the flag *new* (bit 4): it has no position in the source text. - A value that an entry of such a sequence links to gets the flag *linked* (bit 5): its extent in the parsed layout (`next`) no longer matters, so a scalar can replace an array or object there in place. - Views of read-only documents compile without any of this: how views walk the index is a template parameter (`navigationThe input adapters as a class diagram. The dotted arrows mean that each adapter implements the input adapter concept; the diamond shows that `wide_string_input_adapter` holds an `iterator_input_adapter`.
## SAX interface The parser does not build values itself. It reports what it reads as events to a [SAX](../features/parsing/sax_interface.md) consumer, which implements the interface [`json_sax`](../api/json_sax/index.md): `null`, `boolean`, `number_integer`, `number_unsigned`, `number_float`, `string`, `binary`, `start_object`, `key`, `end_object`, `start_array`, `end_array`, and `parse_error`. The library comes with two consumers in `detail/input/json_sax.hpp`: - `json_sax_dom_parser` builds a [`basic_json`](../api/basic_json/index.md) value tree. [`parse`](../api/basic_json/parse.md) uses it. - `json_sax_dom_callback_parser` does the same, but calls a [parser callback](../features/parsing/parser_callbacks.md) for each event, which can skip values. `parse` uses it when a callback is given. The `binary_reader` emits the same events for binary formats, so [`sax_parse`](../api/basic_json/sax_parse.md) works with a user-defined consumer for JSON and for all binary formats alike. The [SAX event stepper](../features/parsing/sax_interface.md#sax-event-stepper) shows the events for a small input. ## Output adapters Output is written via **output adapters**: ```cpp void write_character(CharType c); void write_characters(const CharType* s, std::size_t length); ``` The `serializer` (used by [`dump`](../api/basic_json/dump.md) and [`operator<<`](../api/operator_ltlt.md)) and the `binary_writer` (used by the `to_*` functions) write to one of these adapters: - `output_vector_adapter` appends to a `std::vector`. - `output_stream_adapter` writes to a `std::ostream`. - `output_string_adapter` appends to a string. They implement the abstract class `output_adapter_protocol`. The class `output_adapter` creates the one that matches its argument and holds it in a `std::shared_ptr` (`output_adapter_t`): ```mermaid classDiagram class output_adapter_protocol~CharType~ { <The output adapters as a class diagram. The three adapters derive from `output_adapter_protocol` (solid arrows), and `output_adapter` holds one of them through a `shared_ptr` (diamond).
The `binary_writer` does not call an adapter for every byte. The `to_*` functions that return a `std::vector` write into the vector through a sink (`output_vector_sink`) that is not virtual. The functions that take an output adapter write through a sink (`output_adapter_sink`) that collects the output in a buffer of one kibibyte and passes it to the adapter in blocks, so that the adapter is called (and a stream locked) once per block instead of once per value. ## Value conversion Values are converted from and to other types with the `JSONSerializer` template parameter. The default, [`adl_serializer`](../api/adl_serializer/index.md), calls the free functions ```cpp template