This commit is contained in:
nlohmann committed 2026-10-27 12:14:00 +00:00
1 parent b36f8a9821
commit 11a863958e
1180 files changed
+14211 -6025

No files matched your search

@@ -1,4 +1,4 @@
<!doctype html><html lang=en class=no-js> <head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1"><meta name=author content="Niels Lohmann"><link href="https://json.nlohmann.me/features/parsing/large_and_partial_input/" rel="canonical"><link href=../json_lines/ rel=prev><link href=../parse_exceptions/ rel=next><link rel=icon href=../../../assets/images/favicon.png><meta name=generator content="mkdocs-1.6.1, mkdocs-material-9.7.7"><title>Large, Streamed, and Partial Input - JSON for Modern C++</title><link rel=stylesheet href=../../../assets/stylesheets/main.ec1eaa64.min.css><link rel=stylesheet href=../../../assets/stylesheets/palette.ab4e12ef.min.css><link rel="stylesheet" href="../../../assets/external/fonts.googleapis.com/css.61a430c9.css"><style>:root{--md-text-font:"Roboto";--md-code-font:"JetBrains Mono"}</style><link rel=stylesheet href=../../../css/custom.css><script>__md_scope=new URL("../../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script></head> <body dir=ltr data-md-color-scheme=default data-md-color-primary=indigo data-md-color-accent=indigo> <input class=md-toggle data-md-toggle=drawer type=checkbox id=__drawer autocomplete=off> <input class=md-toggle data-md-toggle=search type=checkbox id=__search autocomplete=off> <label class=md-overlay for=__drawer></label> <div data-md-component=skip> <a href=#large-streamed-and-partial-input class=md-skip> Skip to content </a> </div> <div data-md-component=announce> </div> <header class=md-header data-md-component=header> <nav class="md-header__inner md-grid" aria-label=Header> <a href=../../.. title="JSON for Modern C++" class="md-header__button md-logo" aria-label="JSON for Modern C++" data-md-component=logo> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg> </a> <label class="md-header__button md-icon" for=__drawer> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M3 6h18v2H3zm0 5h18v2H3zm0 5h18v2H3z"/></svg> </label> <div class=md-header__title data-md-component=header-title> <div class=md-header__ellipsis> <div class=md-header__topic> <span class=md-ellipsis> JSON for Modern C++ </span> </div> <div class=md-header__topic data-md-component=header-topic> <span class=md-ellipsis> Large, Streamed, and Partial Input </span> </div> </div> </div> <form class=md-header__option data-md-component=palette> <input class=md-option data-md-color-media="(prefers-color-scheme: light)" data-md-color-scheme=default data-md-color-primary=indigo data-md-color-accent=indigo aria-label="Switch to dark mode" type=radio name=__palette id=__palette_0> <label class="md-header__button md-icon" title="Switch to dark mode" for=__palette_1 hidden> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M12 8a4 4 0 0 0-4 4 4 4 0 0 0 4 4 4 4 0 0 0 4-4 4 4 0 0 0-4-4m0 10a6 6 0 0 1-6-6 6 6 0 0 1 6-6 6 6 0 0 1 6 6 6 6 0 0 1-6 6m8-9.31V4h-4.69L12 .69 8.69 4H4v4.69L.69 12 4 15.31V20h4.69L12 23.31 15.31 20H20v-4.69L23.31 12z"/></svg> </label> <input class=md-option data-md-color-media="(prefers-color-scheme: dark)" data-md-color-scheme=slate data-md-color-primary=indigo data-md-color-accent=indigo aria-label="Switch to light mode" type=radio name=__palette id=__palette_1> <label class="md-header__button md-icon" title="Switch to light mode" for=__palette_0 hidden> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M12 18c-.89 0-1.74-.2-2.5-.55C11.56 16.5 13 14.42 13 12s-1.44-4.5-3.5-5.45C10.26 6.2 11.11 6 12 6a6 6 0 0 1 6 6 6 6 0 0 1-6 6m8-9.31V4h-4.69L12 .69 8.69 4H4v4.69L.69 12 4 15.31V20h4.69L12 23.31 15.31 20H20v-4.69L23.31 12z"/></svg> </label> </form> <script>var palette=__md_get("__palette");if(palette&&palette.color){if("(prefers-color-scheme)"===palette.color.media){var media=matchMedia("(prefers-color-scheme: light)"),input=document.querySelector(media.matches?"[data-md-color-media='(prefers-color-scheme: light)']":"[data-md-color-media='(prefers-color-scheme: dark)']");palette.color.media=input.getAttribute("data-md-color-media"),palette.color.scheme=input.getAttribute("data-md-color-scheme"),palette.color.primary=input.getAttribute("data-md-color-primary"),palette.color.accent=input.getAttribute("data-md-color-accent")}for(var[key,value]of Object.entries(palette.color))document.body.setAttribute("data-md-color-"+key,value)}</script> <label class="md-header__button md-icon" for=__search> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0 1 16 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.52 6.52 0 0 1 9.5 16 6.5 6.5 0 0 1 3 Line truncated
<!doctype html><html lang=en class=no-js> <head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1"><meta name=description content="JSON for Modern C++ is a header-only C++11 library that makes JSON a first-class data type."><meta name=author content="Niels Lohmann"><link href="https://json.nlohmann.me/features/parsing/large_and_partial_input/" rel="canonical"><link href=../json_lines/ rel=prev><link href=../parse_exceptions/ rel=next><link rel=icon href=../../../assets/images/favicon.png><meta name=generator content="mkdocs-1.6.1, mkdocs-material-9.7.7"><title>Large, Streamed, and Partial Input - JSON for Modern C++</title><link rel=stylesheet href=../../../assets/stylesheets/main.ec1eaa64.min.css><link rel=stylesheet href=../../../assets/stylesheets/palette.ab4e12ef.min.css><link rel=stylesheet href=../../../css/custom.css><link rel=stylesheet href=../../../css/brand.css><link rel=stylesheet href=../../../css/fig-sax.css><link rel=stylesheet href=../../../css/fig-jsonview.css><link rel=stylesheet href=../../../css/fig-exceptions.css><link rel=stylesheet href=../../../css/fig-pointer.css><link rel=stylesheet href=../../../css/fig-charts.css><link rel=stylesheet href=../../../css/fig-integration.css><link rel=stylesheet href=../../../css/fig-macros.css><script>__md_scope=new URL("../../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script></head> <body dir=ltr data-md-color-scheme=default data-md-color-primary=custom data-md-color-accent=custom> <input class=md-toggle data-md-toggle=drawer type=checkbox id=__drawer autocomplete=off> <input class=md-toggle data-md-toggle=search type=checkbox id=__search autocomplete=off> <label class=md-overlay for=__drawer></label> <div data-md-component=skip> <a href=#large-streamed-and-partial-input class=md-skip> Skip to content </a> </div> <div data-md-component=announce> </div> <header class=md-header data-md-component=header> <nav class="md-header__inner md-grid" aria-label=Header> <a href=../../.. title="JSON for Modern C++" class="md-header__button md-logo" aria-label="JSON for Modern C++" data-md-component=logo> <img src=../../../images/wordmark-header.svg alt=logo> </a> <label class="md-header__button md-icon" for=__drawer> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M3 6h18v2H3zm0 5h18v2H3zm0 5h18v2H3z"/></svg> </label> <div class=md-header__title data-md-component=header-title> <div class=md-header__ellipsis> <div class=md-header__topic> <span class=md-ellipsis> JSON for Modern C++ </span> </div> <div class=md-header__topic data-md-component=header-topic> <span class=md-ellipsis> Large, Streamed, and Partial Input </span> </div> </div> </div> <form class=md-header__option data-md-component=palette> <input class=md-option data-md-color-media="(prefers-color-scheme: light)" data-md-color-scheme=default data-md-color-primary=custom data-md-color-accent=custom aria-label="Switch to dark mode" type=radio name=__palette id=__palette_0> <label class="md-header__button md-icon" title="Switch to dark mode" for=__palette_1 hidden> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M12 8a4 4 0 0 0-4 4 4 4 0 0 0 4 4 4 4 0 0 0 4-4 4 4 0 0 0-4-4m0 10a6 6 0 0 1-6-6 6 6 0 0 1 6-6 6 6 0 0 1 6 6 6 6 0 0 1-6 6m8-9.31V4h-4.69L12 .69 8.69 4H4v4.69L.69 12 4 15.31V20h4.69L12 23.31 15.31 20H20v-4.69L23.31 12z"/></svg> </label> <input class=md-option data-md-color-media="(prefers-color-scheme: dark)" data-md-color-scheme=slate data-md-color-primary=custom data-md-color-accent=custom aria-label="Switch to light mode" type=radio name=__palette id=__palette_1> <label class="md-header__button md-icon" title="Switch to light mode" for=__palette_0 hidden> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M12 18c-.89 0-1.74-.2-2.5-.55C11.56 16.5 13 14.42 13 12s-1.44-4.5-3.5-5.45C10.26 6.2 11.11 6 12 6a6 6 0 0 1 6 6 6 6 0 0 1-6 6m8-9.31V4h-4.69L12 .69 8.69 4H4v4.69L.69 12 4 15.31V20h4.69L12 23.31 15.31 20H20v-4.69L23.31 12z"/></svg> </label> </form> <script>var palette=__md_get("__palette");if(palette&&palette.color){if("(prefers-color-scheme)"===palette.color.media){var media=matchMedia("(prefers-color-scheme: light)"),input=document.querySelector(media.matches?"[data-md-color-media='(prefers-color-scheme: light)']":"[data-md-color-media='(prefers-color-scheme: dark)']");palette.color.media=input.getAttribute("data-md-color-media"),palette.color.scheme=input.getAttribute("data-md-color-scheme"),palette.color.primary=input.getAttribute("data-md-color-primary"),palette.color.accent=input.getAttribute("data-md-color-accent")}for(var[key,value]of Object.entries(palette.color))document.body.setAttribute("data-md-color-"+key,value)}</script> <label class="md-header__button md-icon" for=__searchLine truncated
A["my input is..."] --&gt; B{"several JSON values in one stream?"}
B --&gt;|"yes"| C["read them one at a time"]
A --&gt; D{"arriving in chunks (socket, pipe)?"}
@@ -6,7 +6,7 @@
A --&gt; F{"too large to keep in memory?"}
F --&gt;|"yes"| G["discard parts with a callback, or use SAX"]
A --&gt; H{"only a small part is needed?"}
H --&gt;|"yes"| I["stop early with SAX, or read through a view"]</code></pre> <h2 id=several-values-in-one-stream>Several values in one stream<a class=headerlink href=#several-values-in-one-stream title="Permanent link">&para;</a></h2> <p>For newline-delimited input, read line by line and parse each line, as described in <a href=../json_lines/ ><abbr title="JavaScript Object Notation">JSON</abbr> Lines</a>. For values that are simply concatenated, <a href=../../../api/operator_gtgt/ ><code>operator&gt;&gt;</code></a> reads one value and leaves the stream positioned after it. Check for the end of the stream before each read, because a read at the end throws:</p> <details class=example> <summary>Example: read concatenated values</summary> <div class=highlight><pre><span></span><code><span class=cp>#include</span><span class=w> </span><span class=cpf>&lt;iostream&gt;</span>
H --&gt;|"yes"| I["stop early with SAX, or read through a view"]</code></pre> <p class=nj-caption>Questions about the input and the approach this page describes for each; an input can match more than one question.</p> <h2 id=several-values-in-one-stream>Several values in one stream<a class=headerlink href=#several-values-in-one-stream title="Permanent link">&para;</a></h2> <p>For newline-delimited input, read line by line and parse each line, as described in <a href=../json_lines/ ><abbr title="JavaScript Object Notation">JSON</abbr> Lines</a>. For values that are simply concatenated, <a href=../../../api/operator_gtgt/ ><code>operator&gt;&gt;</code></a> reads one value and leaves the stream positioned after it. Check for the end of the stream before each read, because a read at the end throws:</p> <details class=example> <summary>Example: read concatenated values</summary> <div class=highlight><pre><span></span><code><span class=cp>#include</span><span class=w> </span><span class=cpf>&lt;iostream&gt;</span>
<span class=cp>#include</span><span class=w> </span><span class=cpf>&lt;sstream&gt;</span>
<span class=cp>#include</span><span class=w> </span><span class=cpf>&lt;nlohmann/json.hpp&gt;</span>
@@ -108,7 +108,7 @@ chunk ist": [1, 2 -&gt; incomplete
chunk , 3]} -&gt; complete
{"list":[1,2,3],"name":"Mary"}
[json.exception.parse_error.101] parse error at line 1, column 5: syntax error while parsing value - unexpected ']'; expected '[', '{', or a literal
</code></pre></div> </details> <p>The example feeds a document in five chunks. The first four are reported as incomplete, and a syntax error is reported as such, not as missing data.</p> <div class="admonition warning"> <p class=admonition-title>Limits of buffering</p> <ul> <li>The buffer must contain exactly one value. Data that belongs to the next value is a syntax error. Use framing if values follow each other without a separator.</li> <li>A number at the end of the buffer is a complete value, even if more digits are still to come (<code>12</code> may be the start of <code>123</code>). Buffering cannot detect this; use framing for top-level numbers.</li> <li>Every attempt parses the buffer from the beginning, so the total work grows quadratically with the size of the value. Limit the size of the buffer for input you do not trust (see <a href=../untrusted_input/ >untrusted input</a>).</li> </ul> </div> <h2 id=input-too-large-for-memory>Input too large for memory<a class=headerlink href=#input-too-large-for-memory title="Permanent link">&para;</a></h2> <p>The input does not have to become one huge <code>json</code> value. The two tools below process it piece by piece.</p> <h3 id=handle-and-discard-elements-with-a-callback>Handle and discard elements with a callback<a class=headerlink href=#handle-and-discard-elements-with-a-callback title="Permanent link">&para;</a></h3> <p>A <a href=../parser_callbacks/ >parser callback</a> is called for every finished value. If it returns <code class=highlight><span class=nb>false</span></code>, the value is discarded and its memory is released at once. For a file that consists of one huge array (or one huge object) of records, handle each record at depth 1 and discard it:</p> <details class=example> <summary>Example: handle and discard each record of a large array</summary> <div class=highlight><pre><span></span><code><span class=cp>#include</span><span class=w> </span><span class=cpf>&lt;iostream&gt;</span>
</code></pre></div> </details> <p>The example feeds a document in five chunks. The first four are reported as incomplete, and a syntax error is reported as such, not as missing data.</p> <p>The chunks of the example, and the bytes each attempt parses again:</p> <div class=nj-fig-wrap> <p><svg class="nj-fig nj-fig--sax-buf" viewbox="0 0 700 442" role=img aria-labelledby="fig-sax-buf-title fig-sax-buf-desc" xmlns=http://www.w3.org/2000/svg> <title id=fig-sax-buf-title>Buffering chunks until the value is complete: five chunks of {&quot;name&quot;: &quot;Mary&quot;, &quot;list&quot;: [1, 2, 3]} appended to a buffer.</title> <desc id=fig-sax-buf-desc>After chunks 1 to 4 the buffer has 4, 12, 19, and 30 bytes, and parsing it fails at byte 5, 13, 20, and 31: one past the end, so the input is incomplete and the program waits for more. After chunk 5 the buffer of 35 bytes parses as one complete value. For the input [1, ] of 5 bytes the error is at byte 5, before the end, so it is a real syntax error. Every attempt parses the buffer from byte 1, so 4 + 12 + 19 + 30 + 35 = 100 bytes are parsed for a 35-byte value, and the work grows quadratically.</desc> <text class=fg-h x=16 y=24>chunk</text> <text class=fg-h x=96 y=24>buffer, parsed from byte 1 each time</text> <text class=fg-h x=420 y=24>result</text> <text class="fg-note fg-note--l" x=16 y=67>chunk 1</text> <rect class=nj-sax-new x=96 y=50 width=36.6 height=24 rx=4 /> <text class=nj-sax-buf x=96 y=67 xml:space=preserve>{&quot;na</text> <path class="nj-sax-caret nj-sax-s-wait" d="M133.8 76 l-5 8 h10 z"/> <text class="nj-sax-status nj-sax-s-wait" x=420 y=61>error at byte 5 = size + 1</text> <text class="fg-note fg-note--l" x=420 y=77>incomplete: wait for more</text> <text class="fg-note fg-note--l" x=16 y=117>chunk 2</text> <rect class=nj-sax-old x=93 y=100 width=36.6 height=24 rx=4 /> <rect class=nj-sax-new x=129.6 y=100 width=70.2 height=24 rx=4 /> <text class=nj-sax-buf x=96 y=117 xml:space=preserve>{&quot;name&quot;: &quot;Ma</text> <path class="nj-sax-caret nj-sax-s-wait" d="M201 126 l-5 8 h10 z"/> <text class="nj-sax-status nj-sax-s-wait" x=420 y=111>error at byte 13 = size + 1</text> <text class="fg-note fg-note--l" x=420 y=127>incomplete: wait for more</text> <text class="fg-note fg-note--l" x=16 y=167>chunk 3</text> <rect class=nj-sax-old x=93 y=150 width=103.8 height=24 rx=4 /> <rect class=nj-sax-new x=196.8 y=150 width=61.8 height=24 rx=4 /> <text class=nj-sax-buf x=96 y=167 xml:space=preserve>{&quot;name&quot;: &quot;Mary&quot;, &quot;l</text> <path class="nj-sax-caret nj-sax-s-wait" d="M259.8 176 l-5 8 h10 z"/> <text class="nj-sax-status nj-sax-s-wait" x=420 y=161>error at byte 20 = size + 1</text> <text class="fg-note fg-note--l" x=420 y=177>incomplete: wait for more</text> <text class="fg-note fg-note--l" x=16 y=217>chunk 4</text> <rect class=nj-sax-old x=93 y=200 width=162.6 height=24 rx=4 /> <rect class=nj-sax-new x=255.6 y=200 width=95.4 height=24 rx=4 /> <text class=nj-sax-buf x=96 y=217 xml:space=preserve>{&quot;name&quot;: &quot;Mary&quot;, &quot;list&quot;: [1, 2</text> <path class="nj-sax-caret nj-sax-s-wait" d="M352.2 226 l-5 8 h10 z"/> <text class="nj-sax-status nj-sax-s-wait" x=420 y=211>error at byte 31 = size + 1</text> <text class="fg-note fg-note--l" x=420 y=227>incomplete: wait for more</text> <text class="fg-note fg-note--l" x=16 y=267>chunk 5</text> <rect class=nj-sax-old x=93 y=250 width=255 height=24 rx=4 /> <rect class=nj-sax-new x=348 y=250 width=45 height=24 rx=4 /> <text class=nj-sax-buf x=96 y=267 xml:space=preserve>{&quot;name&quot;: &quot;Mary&quot;, &quot;list&quot;: [1, 2, 3]}</text> <text class="nj-sax-status nj-sax-s-ok" x=420 y=261>parsed, size 35</text> <text class="fg-note fg-note--l" x=420 y=277>complete: one value</text> <path class=nj-sax-sep d="M16 290 H684"/> <text class="fg-note fg-note--l" x=16 y=325>other input</text> <rect class=nj-sax-new x=96 y=308 width=45 height=24 rx=4 /> <text class=nj-sax-buf x=96 y=325 xml:space=preserve>[1, ]</text> <path class="nj-sax-caret nj-sax-s-err" d="M133.8 334 l-5 8 h10 z"/> <text class="nj-sax-status nj-sax-s-err" x=420 y=319>error at byte 5, size 5</text> <text class="fg-note fg-note--l" x=420 y=335>before the end: a real error</text> <text class="fg-note fg-note--l" x=16 y=366>Bytes parsed: 4 + 12 + 19 + 30 + 35 = 100 for a value of 35 bytes.</text> <text class="fg-note fg-note--l" x=16 y=386>Every attempt starts again at byte 1, so the total work grows</text> <text class="fg-note fg-note--l" x=16 y=406>quadratically with the size of the value.</text> </svg></p> </div> <p class=nj-caption>Each row is one parse attempt on the growing buffer: an error at the byte after the end of the buffer means the value is incomplete, an error before the end is a real syntax error. The last line adds up the bytes parsed over all attempts.</p> <div class="admonition warning"> <p class=admonition-title>Limits of buffering</p> <ul> <li>The buffer must contain exactly one value. Data that bLine truncated
<span class=cp>#include</span><span class=w> </span><span class=cpf>&lt;sstream&gt;</span>
<span class=cp>#include</span><span class=w> </span><span class=cpf>&lt;nlohmann/json.hpp&gt;</span>
@@ -222,4 +222,4 @@ result: []
<span class=p>}</span>
</code></pre></div> <p>Output:</p> <div class=highlight><pre><span></span><code>parse completed: false
id: 42
</code></pre></div> </details> <p>Note that <code>sax_parse</code> returns <code class=highlight><span class=nb>false</span></code> both when a handler stopped the parse and when the input is invalid, so record the reason in the handler (here: <code>id</code>) if you need to tell the two apart.</p> <h3 id=read-parts-of-a-large-document-with-a-view>Read parts of a large document with a view<a class=headerlink href=#read-parts-of-a-large-document-with-a-view title="Permanent link">&para;</a></h3> <p>If the whole text is available and you only need to read some of it, a <a href=../../json_view/ ><code>json_document</code></a> from <code>&lt;nlohmann/json_view.hpp&gt;</code> (added in version 3.13.0 <span class=unreleased-version title="Not part of a release yet; the latest release is 3.12.0.">unreleased</span>) avoids the <code>basic_json</code> tree altogether: it keeps the text, builds a flat index of the values, and gives read-only <a href=../../../api/basic_json_view/ >views</a> of the parts you look at. Only what you <a href=../../../api/basic_json_view/materialize/ ><code>materialize()</code></a> becomes a <code>basic_json</code> value. See <a href=../../json_view/ >Zero-copy <abbr title="JavaScript Object Notation">JSON</abbr> views</a> for details and for the lifetime rules.</p> <h2 id=see-also>See also<a class=headerlink href=#see-also title="Permanent link">&para;</a></h2> <ul> <li><a href=../ >Parsing</a> - overview of the parsing functions</li> <li><a href=../json_lines/ ><abbr title="JavaScript Object Notation">JSON</abbr> Lines</a> - newline-delimited <abbr title="JavaScript Object Notation">JSON</abbr></li> <li><a href=../parser_callbacks/ >Parser callbacks</a> - discard values while parsing</li> <li><a href=../sax_interface/ ><abbr title="Simple API for XML">SAX</abbr> interface</a> - event-based parsing</li> <li><a href=../../json_view/ >Zero-copy <abbr title="JavaScript Object Notation">JSON</abbr> views</a> - read parts of a large document without building a tree</li> <li><a href=../untrusted_input/ >Untrusted input</a> - limits for input you do not control</li> </ul> <!-- https://squidfunk.github.io/mkdocs-material/reference/tooltips/#adding-a-glossary --> <aside class=md-source-file> <span class=md-source-file__fact> <span class=md-icon title="Last update"> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M21 13.1c-.1 0-.3.1-.4.2l-1 1 2.1 2.1 1-1c.2-.2.2-.6 0-.8l-1.3-1.3c-.1-.1-.2-.2-.4-.2m-1.9 1.8-6.1 6V23h2.1l6.1-6.1zM12.5 7v5.2l4 2.4-1 1L11 13V7zM11 21.9c-5.1-.5-9-4.8-9-9.9C2 6.5 6.5 2 12 2c5.3 0 9.6 4.1 10 9.3-.3-.1-.6-.2-1-.2s-.7.1-1 .2C19.6 7.2 16.2 4 12 4c-4.4 0-8 3.6-8 8 0 4.1 3.1 7.5 7.1 7.9l-.1.2z"/></svg> </span> <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date" title="October 25, 2026 11:07:13 UTC">October 25, 2026</span> </span> </aside> </article> </div> <script>var tabs=__md_get("__tabs");if(Array.isArray(tabs))e:for(var set of document.querySelectorAll(".tabbed-set")){var labels=set.querySelector(".tabbed-labels");for(var tab of tabs)for(var label of labels.getElementsByTagName("label"))if(label.innerText.trim()===tab){var input=document.getElementById(label.htmlFor);input.checked=!0;continue e}}</script> <script>var target=document.getElementById(location.hash.slice(1));target&&target.name&&(target.checked=target.name.startsWith("__tabbed_"))</script> </div> <button type=button class="md-top md-icon" data-md-component=top hidden> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M13 20h-2V8l-5.5 5.5-1.42-1.42L12 4.16l7.92 7.92-1.42 1.42L13 8z"/></svg> Back to top </button> </main> <footer class=md-footer> <div class="md-footer-meta md-typeset"> <div class="md-footer-meta__inner md-grid"> <div class=md-copyright> <div class=md-copyright__highlight> Copyright &copy; 2013-2026 Niels Lohmann </div> </div> <div class=md-social> <a href="https://github.com/nlohmann" target="_blank" rel="noopener" title="github.com" class="md-social__link"> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 512 512"><!-- Font Awesome Free 7.1.0 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2025 Fonticons, Inc.--><path d="M173.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6m-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3m44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9M252.8 8C114.1 8 8 113.3 8 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.Line truncated
</code></pre></div> </details> <p>Note that <code>sax_parse</code> returns <code class=highlight><span class=nb>false</span></code> both when a handler stopped the parse and when the input is invalid, so record the reason in the handler (here: <code>id</code>) if you need to tell the two apart.</p> <h3 id=read-parts-of-a-large-document-with-a-view>Read parts of a large document with a view<a class=headerlink href=#read-parts-of-a-large-document-with-a-view title="Permanent link">&para;</a></h3> <p>If the whole text is available and you only need to read some of it, a <a href=../../json_view/ ><code>json_document</code></a> from <code>&lt;nlohmann/json_view.hpp&gt;</code> (added in version 3.13.0 <span class=unreleased-version title="Not part of a release yet; the latest release is 3.12.0.">unreleased</span>) avoids the <code>basic_json</code> tree altogether: it keeps the text, builds a flat index of the values, and gives read-only <a href=../../../api/basic_json_view/ >views</a> of the parts you look at. Only what you <a href=../../../api/basic_json_view/materialize/ ><code>materialize()</code></a> becomes a <code>basic_json</code> value. See <a href=../../json_view/ >Zero-copy <abbr title="JavaScript Object Notation">JSON</abbr> views</a> for details and for the lifetime rules.</p> <h2 id=see-also>See also<a class=headerlink href=#see-also title="Permanent link">&para;</a></h2> <ul> <li><a href=../ >Parsing</a> - overview of the parsing functions</li> <li><a href=../json_lines/ ><abbr title="JavaScript Object Notation">JSON</abbr> Lines</a> - newline-delimited <abbr title="JavaScript Object Notation">JSON</abbr></li> <li><a href=../parser_callbacks/ >Parser callbacks</a> - discard values while parsing</li> <li><a href=../sax_interface/ ><abbr title="Simple API for XML">SAX</abbr> interface</a> - event-based parsing</li> <li><a href=../../json_view/ >Zero-copy <abbr title="JavaScript Object Notation">JSON</abbr> views</a> - read parts of a large document without building a tree</li> <li><a href=../untrusted_input/ >Untrusted input</a> - limits for input you do not control</li> <li><a href=../../../api/operator_gtgt/ ><code>operator&gt;&gt;</code></a> - deserialize from stream</li> <li><a href=../../../api/basic_json/parse/ ><code>parse</code></a> - deserialize from a compatible input</li> <li><a href=../../../api/basic_json/sax_parse/ ><code>sax_parse</code></a> - generate <abbr title="Simple API for XML">SAX</abbr> events</li> <li><a href=../../../api/macros/json_precise_stream_position/ ><code>JSON_PRECISE_STREAM_POSITION</code></a> - leave the stream positioned right after a parsed number</li> <li><a href=../../../api/macros/json_disable_stream_bulk_scan/ ><code>JSON_DISABLE_STREAM_BULK_SCAN</code></a> - read a <code>std::istream</code> one character at a time</li> </ul> <!-- https://squidfunk.github.io/mkdocs-material/reference/tooltips/#adding-a-glossary --> <aside class=md-source-file> <span class=md-source-file__fact> <span class=md-icon title="Last update"> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M21 13.1c-.1 0-.3.1-.4.2l-1 1 2.1 2.1 1-1c.2-.2.2-.6 0-.8l-1.3-1.3c-.1-.1-.2-.2-.4-.2m-1.9 1.8-6.1 6V23h2.1l6.1-6.1zM12.5 7v5.2l4 2.4-1 1L11 13V7zM11 21.9c-5.1-.5-9-4.8-9-9.9C2 6.5 6.5 2 12 2c5.3 0 9.6 4.1 10 9.3-.3-.1-.6-.2-1-.2s-.7.1-1 .2C19.6 7.2 16.2 4 12 4c-4.4 0-8 3.6-8 8 0 4.1 3.1 7.5 7.1 7.9l-.1.2z"/></svg> </span> <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date" title="October 27, 2026 06:41:42 UTC">October 27, 2026</span> </span> </aside> </article> </div> <script>var tabs=__md_get("__tabs");if(Array.isArray(tabs))e:for(var set of document.querySelectorAll(".tabbed-set")){var labels=set.querySelector(".tabbed-labels");for(var tab of tabs)for(var label of labels.getElementsByTagName("label"))if(label.innerText.trim()===tab){var input=document.getElementById(label.htmlFor);input.checked=!0;continue e}}</script> <script>var target=document.getElementById(location.hash.slice(1));target&&target.name&&(target.checked=target.name.startsWith("__tabbed_"))</script> </div> <button type=button class="md-top md-icon" data-md-component=top hidden> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M13 20h-2V8l-5.5 5.5-1.42-1.42L12 4.16l7.92 7.92-1.42 1.42L13 8z"/></svg> Back to top </button> </main> <footer class=md-footer> <nav class="md-footer__inner md-grid" aria-label=Footer> <a href=../json_lines/ class="md-footer__link md-footer__link--prev" aria-label="Previous: JSON Lines"> <div class="md-footer__button md-icon"> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11z"/></svg> </div> <div class=md-footer__title> <span class=md-footer__direction> Previous </span> <div class=md-ellipsis> JSON Lines </div> </div> </a> <a href=../parse_exceptions/ class="md-footer__link md-footer__link--next" aria-label="Next: Parsing and Exceptions"> <div class=md-foLine truncated
@@ -14,6 +14,8 @@ flowchart TD
H -->|"yes"| I["stop early with SAX, or read through a view"]
```
Questions about the input and the approach this page describes for each; an input can match more than one question.
## Several values in one stream
For newline-delimited input, read line by line and parse each line, as described in [JSON Lines](https://json.nlohmann.me/features/parsing/json_lines/index.md). For values that are simply concatenated, [`operator>>`](https://json.nlohmann.me/api/operator_gtgt/index.md) reads one value and leaves the stream positioned after it. Check for the end of the stream before each read, because a read at the end throws:
@@ -162,6 +164,10 @@ chunk , 3]} -> complete
The example feeds a document in five chunks. The first four are reported as incomplete, and a syntax error is reported as such, not as missing data.
The chunks of the example, and the bytes each attempt parses again:
Each row is one parse attempt on the growing buffer: an error at the byte after the end of the buffer means the value is incomplete, an error before the end is a real syntax error. The last line adds up the bytes parsed over all attempts.
Limits of buffering
- The buffer must contain exactly one value. Data that belongs to the next value is a syntax error. Use framing if values follow each other without a separator.
@@ -172,6 +178,10 @@ Limits of buffering
The input does not have to become one huge `json` value. The two tools below process it piece by piece.
How the memory use develops while a large array of records is parsed (illustrative, not a measurement):
Memory use over the number of records parsed: `parse()` keeps the whole tree, `json_document` the text plus its index, a discarding callback one record at a time, and SAX only what the handler keeps.
### Handle and discard elements with a callback
A [parser callback](https://json.nlohmann.me/features/parsing/parser_callbacks/index.md) is called for every finished value. If it returns `false`, the value is discarded and its memory is released at once. For a file that consists of one huge array (or one huge object) of records, handle each record at depth 1 and discard it:
@@ -330,3 +340,8 @@ If the whole text is available and you only need to read some of it, a [`json_do
- [SAX interface](https://json.nlohmann.me/features/parsing/sax_interface/index.md) - event-based parsing
- [Zero-copy JSON views](https://json.nlohmann.me/features/json_view/index.md) - read parts of a large document without building a tree
- [Untrusted input](https://json.nlohmann.me/features/parsing/untrusted_input/index.md) - limits for input you do not control
- [`operator>>`](https://json.nlohmann.me/api/operator_gtgt/index.md) - deserialize from stream
- [`parse`](https://json.nlohmann.me/api/basic_json/parse/index.md) - deserialize from a compatible input
- [`sax_parse`](https://json.nlohmann.me/api/basic_json/sax_parse/index.md) - generate SAX events
- [`JSON_PRECISE_STREAM_POSITION`](https://json.nlohmann.me/api/macros/json_precise_stream_position/index.md) - leave the stream positioned right after a parsed number
- [`JSON_DISABLE_STREAM_BULK_SCAN`](https://json.nlohmann.me/api/macros/json_disable_stream_bulk_scan/index.md) - read a `std::istream` one character at a time