deploy: 94fe31eb9b
This commit is contained in:
1 parent
41c601a7d1
commit
d2e264bd84
1400 files changed
+11859
-5440
No files matched your search
@@ -0,0 +1,225 @@
|
||||
<!doctype html><html lang=en class=no-js> <head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1"><meta name=author content="Niels Lohmann"><link href="https://json.nlohmann.me/features/parsing/large_and_partial_input/" rel="canonical"><link href=../json_lines/ rel=prev><link href=../parse_exceptions/ rel=next><link rel=icon href=../../../assets/images/favicon.png><meta name=generator content="mkdocs-1.6.1, mkdocs-material-9.7.7"><title>Large, Streamed, and Partial Input - JSON for Modern C++</title><link rel=stylesheet href=../../../assets/stylesheets/main.ec1eaa64.min.css><link rel=stylesheet href=../../../assets/stylesheets/palette.ab4e12ef.min.css><link rel="stylesheet" href="../../../assets/external/fonts.googleapis.com/css.61a430c9.css"><style>:root{--md-text-font:"Roboto";--md-code-font:"JetBrains Mono"}</style><link rel=stylesheet href=../../../css/custom.css><script>__md_scope=new URL("../../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script></head> <body dir=ltr data-md-color-scheme=default data-md-color-primary=indigo data-md-color-accent=indigo> <input class=md-toggle data-md-toggle=drawer type=checkbox id=__drawer autocomplete=off> <input class=md-toggle data-md-toggle=search type=checkbox id=__search autocomplete=off> <label class=md-overlay for=__drawer></label> <div data-md-component=skip> <a href=#large-streamed-and-partial-input class=md-skip> Skip to content </a> </div> <div data-md-component=announce> </div> <header class=md-header data-md-component=header> <nav class="md-header__inner md-grid" aria-label=Header> <a href=../../.. title="JSON for Modern C++" class="md-header__button md-logo" aria-label="JSON for Modern C++" data-md-component=logo> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg> </a> <label class="md-header__button md-icon" for=__drawer> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M3 6h18v2H3zm0 5h18v2H3zm0 5h18v2H3z"/></svg> </label> <div class=md-header__title data-md-component=header-title> <div class=md-header__ellipsis> <div class=md-header__topic> <span class=md-ellipsis> JSON for Modern C++ </span> </div> <div class=md-header__topic data-md-component=header-topic> <span class=md-ellipsis> Large, Streamed, and Partial Input </span> </div> </div> </div> <form class=md-header__option data-md-component=palette> <input class=md-option data-md-color-media="(prefers-color-scheme: light)" data-md-color-scheme=default data-md-color-primary=indigo data-md-color-accent=indigo aria-label="Switch to dark mode" type=radio name=__palette id=__palette_0> <label class="md-header__button md-icon" title="Switch to dark mode" for=__palette_1 hidden> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M12 8a4 4 0 0 0-4 4 4 4 0 0 0 4 4 4 4 0 0 0 4-4 4 4 0 0 0-4-4m0 10a6 6 0 0 1-6-6 6 6 0 0 1 6-6 6 6 0 0 1 6 6 6 6 0 0 1-6 6m8-9.31V4h-4.69L12 .69 8.69 4H4v4.69L.69 12 4 15.31V20h4.69L12 23.31 15.31 20H20v-4.69L23.31 12z"/></svg> </label> <input class=md-option data-md-color-media="(prefers-color-scheme: dark)" data-md-color-scheme=slate data-md-color-primary=indigo data-md-color-accent=indigo aria-label="Switch to light mode" type=radio name=__palette id=__palette_1> <label class="md-header__button md-icon" title="Switch to light mode" for=__palette_0 hidden> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M12 18c-.89 0-1.74-.2-2.5-.55C11.56 16.5 13 14.42 13 12s-1.44-4.5-3.5-5.45C10.26 6.2 11.11 6 12 6a6 6 0 0 1 6 6 6 6 0 0 1-6 6m8-9.31V4h-4.69L12 .69 8.69 4H4v4.69L.69 12 4 15.31V20h4.69L12 23.31 15.31 20H20v-4.69L23.31 12z"/></svg> </label> </form> <script>var palette=__md_get("__palette");if(palette&&palette.color){if("(prefers-color-scheme)"===palette.color.media){var media=matchMedia("(prefers-color-scheme: light)"),input=document.querySelector(media.matches?"[data-md-color-media='(prefers-color-scheme: light)']":"[data-md-color-media='(prefers-color-scheme: dark)']");palette.color.media=input.getAttribute("data-md-color-media"),palette.color.scheme=input.getAttribute("data-md-color-scheme"),palette.color.primary=input.getAttribute("data-md-color-primary"),palette.color.accent=input.getAttribute("data-md-color-accent")}for(var[key,value]of Object.entries(palette.color))document.body.setAttribute("data-md-color-"+key,value)}</script> <label class="md-header__button md-icon" for=__search> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0 1 16 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.52 6.52 0 0 1 9.5 16 6.5 6.5 0 0 1 3 Line truncated
|
||||
A["my input is..."] --> B{"several JSON values in one stream?"}
|
||||
B -->|"yes"| C["read them one at a time"]
|
||||
A --> D{"arriving in chunks (socket, pipe)?"}
|
||||
D -->|"yes"| E["frame the messages, or buffer until complete"]
|
||||
A --> F{"too large to keep in memory?"}
|
||||
F -->|"yes"| G["discard parts with a callback, or use SAX"]
|
||||
A --> H{"only a small part is needed?"}
|
||||
H -->|"yes"| I["stop early with SAX, or read through a view"]</code></pre> <h2 id=several-values-in-one-stream>Several values in one stream<a class=headerlink href=#several-values-in-one-stream title="Permanent link">¶</a></h2> <p>For newline-delimited input, read line by line and parse each line, as described in <a href=../json_lines/ ><abbr title="JavaScript Object Notation">JSON</abbr> Lines</a>. For values that are simply concatenated, <a href=../../../api/operator_gtgt/ ><code>operator>></code></a> reads one value and leaves the stream positioned after it. Check for the end of the stream before each read, because a read at the end throws:</p> <details class=example> <summary>Example: read concatenated values</summary> <div class=highlight><pre><span></span><code><span class=cp>#include</span><span class=w> </span><span class=cpf><iostream></span>
|
||||
<span class=cp>#include</span><span class=w> </span><span class=cpf><sstream></span>
|
||||
<span class=cp>#include</span><span class=w> </span><span class=cpf><nlohmann/json.hpp></span>
|
||||
|
||||
<span class=k>using</span><span class=w> </span><span class=n>json</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=n>nlohmann</span><span class=o>::</span><span class=n>json</span><span class=p>;</span>
|
||||
|
||||
<span class=kt>int</span><span class=w> </span><span class=nf>main</span><span class=p>()</span>
|
||||
<span class=p>{</span>
|
||||
<span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>istringstream</span><span class=w> </span><span class=n>input</span><span class=p>(</span><span class=sa>R</span><span class=s>"</span><span class=dl>(</span><span class=s>{"a":1} [2,3] 4 "s"</span><span class=dl>)</span><span class=s>"</span><span class=p>);</span>
|
||||
|
||||
<span class=w> </span><span class=c1>// check for the end of the stream before each read, because a read at the end throws</span>
|
||||
<span class=w> </span><span class=n>json</span><span class=w> </span><span class=n>j</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=k>while</span><span class=w> </span><span class=p>(</span><span class=n>input</span><span class=w> </span><span class=o>>></span><span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>ws</span><span class=w> </span><span class=o>&&</span><span class=w> </span><span class=n>input</span><span class=p>.</span><span class=n>peek</span><span class=p>()</span><span class=w> </span><span class=o>!=</span><span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>char_traits</span><span class=o><</span><span class=kt>char</span><span class=o>>::</span><span class=n>eof</span><span class=p>())</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=n>input</span><span class=w> </span><span class=o>>></span><span class=w> </span><span class=n>j</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>cout</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=n>j</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=sc>'\n'</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=p>}</span>
|
||||
</code></pre></div> <p>Output:</p> <div class=highlight><pre><span></span><code>{"a":1}
|
||||
[2,3]
|
||||
4
|
||||
"s"
|
||||
</code></pre></div> </details> <p>The same works for binary formats. <a href=../../../api/basic_json/from_cbor/ ><code>from_cbor</code></a> and its relatives accept <code class=highlight><span class=n>strict</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=nb>false</span></code> to stop after the first value and leave the rest of the stream untouched; <code>tellg()</code> then reports how many bytes the value took:</p> <details class=example> <summary>Example: read one <abbr title="Concise Binary Object Representation">CBOR</abbr> value and find out how many bytes it used</summary> <div class=highlight><pre><span></span><code><span class=cp>#include</span><span class=w> </span><span class=cpf><iostream></span>
|
||||
<span class=cp>#include</span><span class=w> </span><span class=cpf><sstream></span>
|
||||
<span class=cp>#include</span><span class=w> </span><span class=cpf><nlohmann/json.hpp></span>
|
||||
|
||||
<span class=k>using</span><span class=w> </span><span class=n>json</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=n>nlohmann</span><span class=o>::</span><span class=n>json</span><span class=p>;</span>
|
||||
|
||||
<span class=kt>int</span><span class=w> </span><span class=nf>main</span><span class=p>()</span>
|
||||
<span class=p>{</span>
|
||||
<span class=w> </span><span class=c1>// a CBOR value followed by other data</span>
|
||||
<span class=w> </span><span class=k>const</span><span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>vector</span><span class=o><</span><span class=n>std</span><span class=o>::</span><span class=kt>uint8_t</span><span class=o>></span><span class=w> </span><span class=n>cbor</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=n>json</span><span class=o>::</span><span class=n>to_cbor</span><span class=p>(</span><span class=n>json</span><span class=p>{</span><span class=mi>1</span><span class=p>,</span><span class=w> </span><span class=mi>2</span><span class=p>,</span><span class=w> </span><span class=mi>3</span><span class=p>});</span>
|
||||
<span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>string</span><span class=w> </span><span class=n>data</span><span class=p>(</span><span class=n>cbor</span><span class=p>.</span><span class=n>begin</span><span class=p>(),</span><span class=w> </span><span class=n>cbor</span><span class=p>.</span><span class=n>end</span><span class=p>());</span>
|
||||
<span class=w> </span><span class=n>data</span><span class=w> </span><span class=o>+=</span><span class=w> </span><span class=s>"TRAILING"</span><span class=p>;</span>
|
||||
|
||||
<span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>istringstream</span><span class=w> </span><span class=n>input</span><span class=p>(</span><span class=n>data</span><span class=p>);</span>
|
||||
<span class=w> </span><span class=n>json</span><span class=w> </span><span class=n>j</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=n>json</span><span class=o>::</span><span class=n>from_cbor</span><span class=p>(</span><span class=n>input</span><span class=p>,</span><span class=w> </span><span class=nb>false</span><span class=p>);</span><span class=w> </span><span class=c1>// strict = false: stop after the first value</span>
|
||||
|
||||
<span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>cout</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=s>"value: "</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=n>j</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=sc>'\n'</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>cout</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=s>"bytes used: "</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=n>input</span><span class=p>.</span><span class=n>tellg</span><span class=p>()</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=sc>'\n'</span><span class=p>;</span>
|
||||
|
||||
<span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>string</span><span class=w> </span><span class=n>rest</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=n>input</span><span class=w> </span><span class=o>>></span><span class=w> </span><span class=n>rest</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>cout</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=s>"remaining input: "</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=n>rest</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=sc>'\n'</span><span class=p>;</span>
|
||||
<span class=p>}</span>
|
||||
</code></pre></div> <p>Output:</p> <div class=highlight><pre><span></span><code>value: [1,2,3]
|
||||
bytes used: 4
|
||||
remaining input: TRAILING
|
||||
</code></pre></div> </details> <h2 id=input-arriving-in-chunks>Input arriving in chunks<a class=headerlink href=#input-arriving-in-chunks title="Permanent link">¶</a></h2> <p>A socket, pipe, or <abbr title="Hypertext Transfer Protocol">HTTP</abbr> body hands over data in pieces of arbitrary size, and a piece can end in the middle of a number, a string, or an object. The parser does not keep state between calls, so there are two ways to deal with this.</p> <p><strong>Frame the messages.</strong> The most robust approach is a protocol that says where a value ends: one value per line (<a href=../json_lines/ ><abbr title="JavaScript Object Notation">JSON</abbr> Lines</a>), a length prefix, or a delimiter. Collect the bytes of one message and parse them when the frame is complete. Prefer this whenever you control the protocol.</p> <p><strong>Buffer until the value is complete.</strong> If the data has no framing, append every chunk to a buffer and try to parse it. A parse error at the very end of the buffer means that the input ended prematurely, while an error anywhere else is a real syntax error. The member <code>byte</code> of the <a href=../../../home/exceptions/#jsonexceptionparse_error101><code>parse_error</code></a> exception is one past the buffer size in the first case:</p> <details class=example> <summary>Example: buffer chunks until the value is complete</summary> <div class=highlight><pre><span></span><code><span class=cp>#include</span><span class=w> </span><span class=cpf><iostream></span>
|
||||
<span class=cp>#include</span><span class=w> </span><span class=cpf><nlohmann/json.hpp></span>
|
||||
|
||||
<span class=k>using</span><span class=w> </span><span class=n>json</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=n>nlohmann</span><span class=o>::</span><span class=n>json</span><span class=p>;</span>
|
||||
|
||||
<span class=n>std</span><span class=o>::</span><span class=n>string</span><span class=w> </span><span class=n>buffer</span><span class=p>;</span>
|
||||
|
||||
<span class=c1>// call for every chunk; returns true and sets `value` once the buffer holds one complete JSON value</span>
|
||||
<span class=kt>bool</span><span class=w> </span><span class=nf>push</span><span class=p>(</span><span class=k>const</span><span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>string</span><span class=o>&</span><span class=w> </span><span class=n>chunk</span><span class=p>,</span><span class=w> </span><span class=n>json</span><span class=o>&</span><span class=w> </span><span class=n>value</span><span class=p>)</span>
|
||||
<span class=p>{</span>
|
||||
<span class=w> </span><span class=n>buffer</span><span class=w> </span><span class=o>+=</span><span class=w> </span><span class=n>chunk</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=k>try</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=n>value</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=n>json</span><span class=o>::</span><span class=n>parse</span><span class=p>(</span><span class=n>buffer</span><span class=p>);</span>
|
||||
<span class=w> </span><span class=n>buffer</span><span class=p>.</span><span class=n>clear</span><span class=p>();</span>
|
||||
<span class=w> </span><span class=k>return</span><span class=w> </span><span class=nb>true</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=w> </span><span class=k>catch</span><span class=w> </span><span class=p>(</span><span class=k>const</span><span class=w> </span><span class=n>json</span><span class=o>::</span><span class=n>parse_error</span><span class=o>&</span><span class=w> </span><span class=n>e</span><span class=p>)</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=k>if</span><span class=w> </span><span class=p>(</span><span class=n>e</span><span class=p>.</span><span class=kt>byte</span><span class=w> </span><span class=o>==</span><span class=w> </span><span class=n>buffer</span><span class=p>.</span><span class=n>size</span><span class=p>()</span><span class=w> </span><span class=o>+</span><span class=w> </span><span class=mi>1</span><span class=p>)</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=k>return</span><span class=w> </span><span class=nb>false</span><span class=p>;</span><span class=w> </span><span class=c1>// the input ends prematurely: wait for more data</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=w> </span><span class=k>throw</span><span class=p>;</span><span class=w> </span><span class=c1>// a real syntax error</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=p>}</span>
|
||||
|
||||
<span class=kt>int</span><span class=w> </span><span class=nf>main</span><span class=p>()</span>
|
||||
<span class=p>{</span>
|
||||
<span class=w> </span><span class=n>json</span><span class=w> </span><span class=n>value</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=k>const</span><span class=w> </span><span class=k>auto</span><span class=w> </span><span class=n>chunks</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=p>{</span><span class=s>"{</span><span class=se>\"</span><span class=s>na"</span><span class=p>,</span><span class=w> </span><span class=s>"me</span><span class=se>\"</span><span class=s>: </span><span class=se>\"</span><span class=s>Ma"</span><span class=p>,</span><span class=w> </span><span class=s>"ry</span><span class=se>\"</span><span class=s>, </span><span class=se>\"</span><span class=s>l"</span><span class=p>,</span><span class=w> </span><span class=s>"ist</span><span class=se>\"</span><span class=s>: [1, 2"</span><span class=p>,</span><span class=w> </span><span class=s>", 3]}"</span><span class=p>};</span>
|
||||
<span class=w> </span><span class=k>for</span><span class=w> </span><span class=p>(</span><span class=k>const</span><span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>string</span><span class=w> </span><span class=n>chunk</span><span class=w> </span><span class=o>:</span><span class=w> </span><span class=n>chunks</span><span class=p>)</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>cout</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=s>"chunk "</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=n>chunk</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=s>" -> "</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=p>(</span><span class=n>push</span><span class=p>(</span><span class=n>chunk</span><span class=p>,</span><span class=w> </span><span class=n>value</span><span class=p>)</span><span class=w> </span><span class=o>?</span><span class=w> </span><span class=s>"complete"</span><span class=w> </span><span class=o>:</span><span class=w> </span><span class=s>"incomplete"</span><span class=p>)</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=sc>'\n'</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>cout</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=n>value</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=sc>'\n'</span><span class=p>;</span>
|
||||
|
||||
<span class=w> </span><span class=c1>// a syntax error in the middle is not mistaken for missing data</span>
|
||||
<span class=w> </span><span class=k>try</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=n>push</span><span class=p>(</span><span class=s>"[1, ]"</span><span class=p>,</span><span class=w> </span><span class=n>value</span><span class=p>);</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=w> </span><span class=k>catch</span><span class=w> </span><span class=p>(</span><span class=k>const</span><span class=w> </span><span class=n>json</span><span class=o>::</span><span class=n>parse_error</span><span class=o>&</span><span class=w> </span><span class=n>e</span><span class=p>)</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>cout</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=n>e</span><span class=p>.</span><span class=n>what</span><span class=p>()</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=sc>'\n'</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=p>}</span>
|
||||
</code></pre></div> <p>Output:</p> <div class=highlight><pre><span></span><code>chunk {"na -> incomplete
|
||||
chunk me": "Ma -> incomplete
|
||||
chunk ry", "l -> incomplete
|
||||
chunk ist": [1, 2 -> incomplete
|
||||
chunk , 3]} -> complete
|
||||
{"list":[1,2,3],"name":"Mary"}
|
||||
[json.exception.parse_error.101] parse error at line 1, column 5: syntax error while parsing value - unexpected ']'; expected '[', '{', or a literal
|
||||
</code></pre></div> </details> <p>The example feeds a document in five chunks. The first four are reported as incomplete, and a syntax error is reported as such, not as missing data.</p> <div class="admonition warning"> <p class=admonition-title>Limits of buffering</p> <ul> <li>The buffer must contain exactly one value. Data that belongs to the next value is a syntax error. Use framing if values follow each other without a separator.</li> <li>A number at the end of the buffer is a complete value, even if more digits are still to come (<code>12</code> may be the start of <code>123</code>). Buffering cannot detect this; use framing for top-level numbers.</li> <li>Every attempt parses the buffer from the beginning, so the total work grows quadratically with the size of the value. Limit the size of the buffer for input you do not trust (see <a href=../untrusted_input/ >untrusted input</a>).</li> </ul> </div> <h2 id=input-too-large-for-memory>Input too large for memory<a class=headerlink href=#input-too-large-for-memory title="Permanent link">¶</a></h2> <p>The input does not have to become one huge <code>json</code> value. The two tools below process it piece by piece.</p> <h3 id=handle-and-discard-elements-with-a-callback>Handle and discard elements with a callback<a class=headerlink href=#handle-and-discard-elements-with-a-callback title="Permanent link">¶</a></h3> <p>A <a href=../parser_callbacks/ >parser callback</a> is called for every finished value. If it returns <code class=highlight><span class=nb>false</span></code>, the value is discarded and its memory is released at once. For a file that consists of one huge array (or one huge object) of records, handle each record at depth 1 and discard it:</p> <details class=example> <summary>Example: handle and discard each record of a large array</summary> <div class=highlight><pre><span></span><code><span class=cp>#include</span><span class=w> </span><span class=cpf><iostream></span>
|
||||
<span class=cp>#include</span><span class=w> </span><span class=cpf><sstream></span>
|
||||
<span class=cp>#include</span><span class=w> </span><span class=cpf><nlohmann/json.hpp></span>
|
||||
|
||||
<span class=k>using</span><span class=w> </span><span class=n>json</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=n>nlohmann</span><span class=o>::</span><span class=n>json</span><span class=p>;</span>
|
||||
|
||||
<span class=kt>void</span><span class=w> </span><span class=nf>handle</span><span class=p>(</span><span class=k>const</span><span class=w> </span><span class=n>json</span><span class=o>&</span><span class=w> </span><span class=n>record</span><span class=p>)</span>
|
||||
<span class=p>{</span>
|
||||
<span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>cout</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=s>"handling "</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=n>record</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=sc>'\n'</span><span class=p>;</span>
|
||||
<span class=p>}</span>
|
||||
|
||||
<span class=kt>int</span><span class=w> </span><span class=nf>main</span><span class=p>()</span>
|
||||
<span class=p>{</span>
|
||||
<span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>istringstream</span><span class=w> </span><span class=n>file</span><span class=p>(</span><span class=sa>R</span><span class=s>"</span><span class=dl>(</span><span class=s>[{"id": 1}, {"id": 2}, {"id": 3}]</span><span class=dl>)</span><span class=s>"</span><span class=p>);</span>
|
||||
|
||||
<span class=w> </span><span class=n>json</span><span class=w> </span><span class=n>result</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=n>json</span><span class=o>::</span><span class=n>parse</span><span class=p>(</span><span class=n>file</span><span class=p>,</span><span class=w> </span><span class=p>[](</span><span class=kt>int</span><span class=w> </span><span class=n>depth</span><span class=p>,</span><span class=w> </span><span class=n>json</span><span class=o>::</span><span class=n>parse_event_t</span><span class=w> </span><span class=n>event</span><span class=p>,</span><span class=w> </span><span class=n>json</span><span class=w> </span><span class=o>&</span><span class=w> </span><span class=n>record</span><span class=p>)</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=k>if</span><span class=w> </span><span class=p>(</span><span class=n>depth</span><span class=w> </span><span class=o>==</span><span class=w> </span><span class=mi>1</span><span class=w> </span><span class=o>&&</span><span class=w> </span><span class=n>event</span><span class=w> </span><span class=o>==</span><span class=w> </span><span class=n>json</span><span class=o>::</span><span class=n>parse_event_t</span><span class=o>::</span><span class=n>object_end</span><span class=p>)</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=n>handle</span><span class=p>(</span><span class=n>record</span><span class=p>);</span>
|
||||
<span class=w> </span><span class=k>return</span><span class=w> </span><span class=nb>false</span><span class=p>;</span><span class=w> </span><span class=c1>// discard the record</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=w> </span><span class=k>return</span><span class=w> </span><span class=nb>true</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=p>});</span>
|
||||
|
||||
<span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>cout</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=s>"result: "</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=n>result</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=sc>'\n'</span><span class=p>;</span>
|
||||
<span class=p>}</span>
|
||||
</code></pre></div> <p>Output:</p> <div class=highlight><pre><span></span><code>handling {"id":1}
|
||||
handling {"id":2}
|
||||
handling {"id":3}
|
||||
result: []
|
||||
</code></pre></div> </details> <p>Only one record is in memory at any time. The same recipe, with more detail, is in the <a href=../parser_callbacks/#recipe-streaming-a-large-homogeneous-array>parser callbacks</a> page. To skip entire members you do not need, discard them when their key is read; see the <a href=../parser_callbacks/ >example</a> for <code>parse_event_t::key</code>.</p> <h3 id=extract-a-few-values-and-stop-with-sax>Extract a few values and stop with <abbr title="Simple API for XML">SAX</abbr><a class=headerlink href=#extract-a-few-values-and-stop-with-sax title="Permanent link">¶</a></h3> <p>With the <a href=../sax_interface/ ><abbr title="Simple API for XML">SAX</abbr> interface</a>, no value is built at all, and <strong>returning <code class=highlight><span class=nb>false</span></code> from any handler function stops the parser</strong>. <a href=../../../api/basic_json/sax_parse/ ><code>sax_parse</code></a> then returns <code class=highlight><span class=nb>false</span></code>, and the rest of the input is not read. The example finds the first <code>id</code> in a document without parsing the remainder:</p> <details class=example> <summary>Example: stop parsing as soon as a value is found</summary> <div class=highlight><pre><span></span><code><span class=cp>#include</span><span class=w> </span><span class=cpf><iostream></span>
|
||||
<span class=cp>#include</span><span class=w> </span><span class=cpf><nlohmann/json.hpp></span>
|
||||
|
||||
<span class=k>using</span><span class=w> </span><span class=n>json</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=n>nlohmann</span><span class=o>::</span><span class=n>json</span><span class=p>;</span>
|
||||
|
||||
<span class=k>struct</span><span class=w> </span><span class=nc>find_id</span><span class=w> </span><span class=o>:</span><span class=w> </span><span class=n>nlohmann</span><span class=o>::</span><span class=n>json_sax</span><span class=o><</span><span class=n>json</span><span class=o>></span>
|
||||
<span class=p>{</span>
|
||||
<span class=w> </span><span class=kt>bool</span><span class=w> </span><span class=n>next_is_id</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=nb>false</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=n>json</span><span class=o>::</span><span class=n>number_unsigned_t</span><span class=w> </span><span class=n>id</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=mi>0</span><span class=p>;</span>
|
||||
|
||||
<span class=w> </span><span class=kt>bool</span><span class=w> </span><span class=nf>key</span><span class=p>(</span><span class=n>std</span><span class=o>::</span><span class=n>string</span><span class=o>&</span><span class=w> </span><span class=n>name</span><span class=p>)</span><span class=w> </span><span class=k>override</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=n>next_is_id</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=p>(</span><span class=n>name</span><span class=w> </span><span class=o>==</span><span class=w> </span><span class=s>"id"</span><span class=p>);</span>
|
||||
<span class=w> </span><span class=k>return</span><span class=w> </span><span class=nb>true</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
|
||||
<span class=w> </span><span class=kt>bool</span><span class=w> </span><span class=nf>number_unsigned</span><span class=p>(</span><span class=n>json</span><span class=o>::</span><span class=n>number_unsigned_t</span><span class=w> </span><span class=n>value</span><span class=p>)</span><span class=w> </span><span class=k>override</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=k>if</span><span class=w> </span><span class=p>(</span><span class=n>next_is_id</span><span class=p>)</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=n>id</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=n>value</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=k>return</span><span class=w> </span><span class=nb>false</span><span class=p>;</span><span class=w> </span><span class=c1>// found: stop parsing</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=w> </span><span class=k>return</span><span class=w> </span><span class=nb>true</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
|
||||
<span class=w> </span><span class=kt>bool</span><span class=w> </span><span class=nf>null</span><span class=p>()</span><span class=w> </span><span class=k>override</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=k>return</span><span class=w> </span><span class=nb>true</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=w> </span><span class=kt>bool</span><span class=w> </span><span class=nf>boolean</span><span class=p>(</span><span class=kt>bool</span><span class=w> </span><span class=cm>/*val*/</span><span class=p>)</span><span class=w> </span><span class=k>override</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=k>return</span><span class=w> </span><span class=nb>true</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=w> </span><span class=kt>bool</span><span class=w> </span><span class=nf>number_integer</span><span class=p>(</span><span class=n>json</span><span class=o>::</span><span class=n>number_integer_t</span><span class=w> </span><span class=cm>/*val*/</span><span class=p>)</span><span class=w> </span><span class=k>override</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=k>return</span><span class=w> </span><span class=nb>true</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=w> </span><span class=kt>bool</span><span class=w> </span><span class=nf>number_float</span><span class=p>(</span><span class=n>json</span><span class=o>::</span><span class=n>number_float_t</span><span class=w> </span><span class=cm>/*val*/</span><span class=p>,</span><span class=w> </span><span class=k>const</span><span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>string</span><span class=o>&</span><span class=w> </span><span class=cm>/*s*/</span><span class=p>)</span><span class=w> </span><span class=k>override</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=k>return</span><span class=w> </span><span class=nb>true</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=w> </span><span class=kt>bool</span><span class=w> </span><span class=nf>string</span><span class=p>(</span><span class=n>std</span><span class=o>::</span><span class=n>string</span><span class=o>&</span><span class=w> </span><span class=cm>/*val*/</span><span class=p>)</span><span class=w> </span><span class=k>override</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=k>return</span><span class=w> </span><span class=nb>true</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=w> </span><span class=kt>bool</span><span class=w> </span><span class=nf>binary</span><span class=p>(</span><span class=n>json</span><span class=o>::</span><span class=n>binary_t</span><span class=o>&</span><span class=w> </span><span class=cm>/*val*/</span><span class=p>)</span><span class=w> </span><span class=k>override</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=k>return</span><span class=w> </span><span class=nb>true</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=w> </span><span class=kt>bool</span><span class=w> </span><span class=nf>start_object</span><span class=p>(</span><span class=n>std</span><span class=o>::</span><span class=kt>size_t</span><span class=w> </span><span class=cm>/*elements*/</span><span class=p>)</span><span class=w> </span><span class=k>override</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=k>return</span><span class=w> </span><span class=nb>true</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=w> </span><span class=kt>bool</span><span class=w> </span><span class=nf>end_object</span><span class=p>()</span><span class=w> </span><span class=k>override</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=k>return</span><span class=w> </span><span class=nb>true</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=w> </span><span class=kt>bool</span><span class=w> </span><span class=nf>start_array</span><span class=p>(</span><span class=n>std</span><span class=o>::</span><span class=kt>size_t</span><span class=w> </span><span class=cm>/*elements*/</span><span class=p>)</span><span class=w> </span><span class=k>override</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=k>return</span><span class=w> </span><span class=nb>true</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=w> </span><span class=kt>bool</span><span class=w> </span><span class=nf>end_array</span><span class=p>()</span><span class=w> </span><span class=k>override</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=k>return</span><span class=w> </span><span class=nb>true</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=w> </span><span class=kt>bool</span><span class=w> </span><span class=nf>parse_error</span><span class=p>(</span><span class=n>std</span><span class=o>::</span><span class=kt>size_t</span><span class=w> </span><span class=cm>/*position*/</span><span class=p>,</span><span class=w> </span><span class=k>const</span><span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>string</span><span class=o>&</span><span class=w> </span><span class=cm>/*last_token*/</span><span class=p>,</span><span class=w> </span><span class=k>const</span><span class=w> </span><span class=n>json</span><span class=o>::</span><span class=n>exception</span><span class=o>&</span><span class=w> </span><span class=cm>/*ex*/</span><span class=p>)</span><span class=w> </span><span class=k>override</span>
|
||||
<span class=w> </span><span class=p>{</span>
|
||||
<span class=w> </span><span class=k>return</span><span class=w> </span><span class=nb>false</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=p>}</span>
|
||||
<span class=p>};</span>
|
||||
|
||||
<span class=kt>int</span><span class=w> </span><span class=nf>main</span><span class=p>()</span>
|
||||
<span class=p>{</span>
|
||||
<span class=w> </span><span class=n>find_id</span><span class=w> </span><span class=n>handler</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=k>const</span><span class=w> </span><span class=kt>bool</span><span class=w> </span><span class=n>completed</span><span class=w> </span><span class=o>=</span><span class=w> </span><span class=n>json</span><span class=o>::</span><span class=n>sax_parse</span><span class=p>(</span><span class=sa>R</span><span class=s>"</span><span class=dl>(</span><span class=s>{"x": [1, 2, 3], "id": 42, "rest": [4, 5, 6]}</span><span class=dl>)</span><span class=s>"</span><span class=p>,</span><span class=w> </span><span class=o>&</span><span class=n>handler</span><span class=p>);</span>
|
||||
|
||||
<span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>cout</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>boolalpha</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>cout</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=s>"parse completed: "</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=n>completed</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=sc>'\n'</span><span class=p>;</span>
|
||||
<span class=w> </span><span class=n>std</span><span class=o>::</span><span class=n>cout</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=s>"id: "</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=n>handler</span><span class=p>.</span><span class=n>id</span><span class=w> </span><span class=o><<</span><span class=w> </span><span class=sc>'\n'</span><span class=p>;</span>
|
||||
<span class=p>}</span>
|
||||
</code></pre></div> <p>Output:</p> <div class=highlight><pre><span></span><code>parse completed: false
|
||||
id: 42
|
||||
</code></pre></div> </details> <p>Note that <code>sax_parse</code> returns <code class=highlight><span class=nb>false</span></code> both when a handler stopped the parse and when the input is invalid, so record the reason in the handler (here: <code>id</code>) if you need to tell the two apart.</p> <h3 id=read-parts-of-a-large-document-with-a-view>Read parts of a large document with a view<a class=headerlink href=#read-parts-of-a-large-document-with-a-view title="Permanent link">¶</a></h3> <p>If the whole text is available and you only need to read some of it, a <a href=../../json_view/ ><code>json_document</code></a> from <code><nlohmann/json_view.hpp></code> (added in version 3.13.0 <span class=unreleased-version title="Not part of a release yet; the latest release is 3.12.0.">unreleased</span>) avoids the <code>basic_json</code> tree altogether: it keeps the text, builds a flat index of the values, and gives read-only <a href=../../../api/basic_json_view/ >views</a> of the parts you look at. Only what you <a href=../../../api/basic_json_view/materialize/ ><code>materialize()</code></a> becomes a <code>basic_json</code> value. See <a href=../../json_view/ >Zero-copy <abbr title="JavaScript Object Notation">JSON</abbr> views</a> for details and for the lifetime rules.</p> <h2 id=see-also>See also<a class=headerlink href=#see-also title="Permanent link">¶</a></h2> <ul> <li><a href=../ >Parsing</a> - overview of the parsing functions</li> <li><a href=../json_lines/ ><abbr title="JavaScript Object Notation">JSON</abbr> Lines</a> - newline-delimited <abbr title="JavaScript Object Notation">JSON</abbr></li> <li><a href=../parser_callbacks/ >Parser callbacks</a> - discard values while parsing</li> <li><a href=../sax_interface/ ><abbr title="Simple API for XML">SAX</abbr> interface</a> - event-based parsing</li> <li><a href=../../json_view/ >Zero-copy <abbr title="JavaScript Object Notation">JSON</abbr> views</a> - read parts of a large document without building a tree</li> <li><a href=../untrusted_input/ >Untrusted input</a> - limits for input you do not control</li> </ul> <!-- https://squidfunk.github.io/mkdocs-material/reference/tooltips/#adding-a-glossary --> <aside class=md-source-file> <span class=md-source-file__fact> <span class=md-icon title="Last update"> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M21 13.1c-.1 0-.3.1-.4.2l-1 1 2.1 2.1 1-1c.2-.2.2-.6 0-.8l-1.3-1.3c-.1-.1-.2-.2-.4-.2m-1.9 1.8-6.1 6V23h2.1l6.1-6.1zM12.5 7v5.2l4 2.4-1 1L11 13V7zM11 21.9c-5.1-.5-9-4.8-9-9.9C2 6.5 6.5 2 12 2c5.3 0 9.6 4.1 10 9.3-.3-.1-.6-.2-1-.2s-.7.1-1 .2C19.6 7.2 16.2 4 12 4c-4.4 0-8 3.6-8 8 0 4.1 3.1 7.5 7.1 7.9l-.1.2z"/></svg> </span> <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date" title="October 25, 2026 11:07:13 UTC">October 25, 2026</span> </span> </aside> </article> </div> <script>var tabs=__md_get("__tabs");if(Array.isArray(tabs))e:for(var set of document.querySelectorAll(".tabbed-set")){var labels=set.querySelector(".tabbed-labels");for(var tab of tabs)for(var label of labels.getElementsByTagName("label"))if(label.innerText.trim()===tab){var input=document.getElementById(label.htmlFor);input.checked=!0;continue e}}</script> <script>var target=document.getElementById(location.hash.slice(1));target&&target.name&&(target.checked=target.name.startsWith("__tabbed_"))</script> </div> <button type=button class="md-top md-icon" data-md-component=top hidden> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M13 20h-2V8l-5.5 5.5-1.42-1.42L12 4.16l7.92 7.92-1.42 1.42L13 8z"/></svg> Back to top </button> </main> <footer class=md-footer> <div class="md-footer-meta md-typeset"> <div class="md-footer-meta__inner md-grid"> <div class=md-copyright> <div class=md-copyright__highlight> Copyright © 2013-2026 Niels Lohmann </div> </div> <div class=md-social> <a href="https://github.com/nlohmann" target="_blank" rel="noopener" title="github.com" class="md-social__link"> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 512 512"><!-- Font Awesome Free 7.1.0 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2025 Fonticons, Inc.--><path d="M173.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6m-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3m44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9M252.8 8C114.1 8 8 113.3 8 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.Line truncated
|
||||
@@ -0,0 +1,332 @@
|
||||
# Large, Streamed, and Partial Input
|
||||
|
||||
[`parse`](https://json.nlohmann.me/api/basic_json/parse/index.md) reads exactly one complete JSON value and builds it in memory. It cannot be paused and resumed, and it does not hand back a half-parsed value when the input ends early. This page shows what to do when the input does not fit that model: it arrives in pieces, it contains several values, or it is too large to hold as a `json` value.
|
||||
|
||||
```
|
||||
flowchart TD
|
||||
A["my input is..."] --> B{"several JSON values in one stream?"}
|
||||
B -->|"yes"| C["read them one at a time"]
|
||||
A --> D{"arriving in chunks (socket, pipe)?"}
|
||||
D -->|"yes"| E["frame the messages, or buffer until complete"]
|
||||
A --> F{"too large to keep in memory?"}
|
||||
F -->|"yes"| G["discard parts with a callback, or use SAX"]
|
||||
A --> H{"only a small part is needed?"}
|
||||
H -->|"yes"| I["stop early with SAX, or read through a view"]
|
||||
```
|
||||
|
||||
## Several values in one stream
|
||||
|
||||
For newline-delimited input, read line by line and parse each line, as described in [JSON Lines](https://json.nlohmann.me/features/parsing/json_lines/index.md). For values that are simply concatenated, [`operator>>`](https://json.nlohmann.me/api/operator_gtgt/index.md) reads one value and leaves the stream positioned after it. Check for the end of the stream before each read, because a read at the end throws:
|
||||
|
||||
Example: read concatenated values
|
||||
|
||||
```
|
||||
#include <iostream>
|
||||
#include <sstream>
|
||||
#include <nlohmann/json.hpp>
|
||||
|
||||
using json = nlohmann::json;
|
||||
|
||||
int main()
|
||||
{
|
||||
std::istringstream input(R"({"a":1} [2,3] 4 "s")");
|
||||
|
||||
// check for the end of the stream before each read, because a read at the end throws
|
||||
json j;
|
||||
while (input >> std::ws && input.peek() != std::char_traits<char>::eof())
|
||||
{
|
||||
input >> j;
|
||||
std::cout << j << '\n';
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```
|
||||
{"a":1}
|
||||
[2,3]
|
||||
4
|
||||
"s"
|
||||
```
|
||||
|
||||
The same works for binary formats. [`from_cbor`](https://json.nlohmann.me/api/basic_json/from_cbor/index.md) and its relatives accept `strict = false` to stop after the first value and leave the rest of the stream untouched; `tellg()` then reports how many bytes the value took:
|
||||
|
||||
Example: read one CBOR value and find out how many bytes it used
|
||||
|
||||
```
|
||||
#include <iostream>
|
||||
#include <sstream>
|
||||
#include <nlohmann/json.hpp>
|
||||
|
||||
using json = nlohmann::json;
|
||||
|
||||
int main()
|
||||
{
|
||||
// a CBOR value followed by other data
|
||||
const std::vector<std::uint8_t> cbor = json::to_cbor(json{1, 2, 3});
|
||||
std::string data(cbor.begin(), cbor.end());
|
||||
data += "TRAILING";
|
||||
|
||||
std::istringstream input(data);
|
||||
json j = json::from_cbor(input, false); // strict = false: stop after the first value
|
||||
|
||||
std::cout << "value: " << j << '\n';
|
||||
std::cout << "bytes used: " << input.tellg() << '\n';
|
||||
|
||||
std::string rest;
|
||||
input >> rest;
|
||||
std::cout << "remaining input: " << rest << '\n';
|
||||
}
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```
|
||||
value: [1,2,3]
|
||||
bytes used: 4
|
||||
remaining input: TRAILING
|
||||
```
|
||||
|
||||
## Input arriving in chunks
|
||||
|
||||
A socket, pipe, or HTTP body hands over data in pieces of arbitrary size, and a piece can end in the middle of a number, a string, or an object. The parser does not keep state between calls, so there are two ways to deal with this.
|
||||
|
||||
**Frame the messages.** The most robust approach is a protocol that says where a value ends: one value per line ([JSON Lines](https://json.nlohmann.me/features/parsing/json_lines/index.md)), a length prefix, or a delimiter. Collect the bytes of one message and parse them when the frame is complete. Prefer this whenever you control the protocol.
|
||||
|
||||
**Buffer until the value is complete.** If the data has no framing, append every chunk to a buffer and try to parse it. A parse error at the very end of the buffer means that the input ended prematurely, while an error anywhere else is a real syntax error. The member `byte` of the [`parse_error`](https://json.nlohmann.me/home/exceptions/#jsonexceptionparse_error101) exception is one past the buffer size in the first case:
|
||||
|
||||
Example: buffer chunks until the value is complete
|
||||
|
||||
```
|
||||
#include <iostream>
|
||||
#include <nlohmann/json.hpp>
|
||||
|
||||
using json = nlohmann::json;
|
||||
|
||||
std::string buffer;
|
||||
|
||||
// call for every chunk; returns true and sets `value` once the buffer holds one complete JSON value
|
||||
bool push(const std::string& chunk, json& value)
|
||||
{
|
||||
buffer += chunk;
|
||||
try
|
||||
{
|
||||
value = json::parse(buffer);
|
||||
buffer.clear();
|
||||
return true;
|
||||
}
|
||||
catch (const json::parse_error& e)
|
||||
{
|
||||
if (e.byte == buffer.size() + 1)
|
||||
{
|
||||
return false; // the input ends prematurely: wait for more data
|
||||
}
|
||||
throw; // a real syntax error
|
||||
}
|
||||
}
|
||||
|
||||
int main()
|
||||
{
|
||||
json value;
|
||||
const auto chunks = {"{\"na", "me\": \"Ma", "ry\", \"l", "ist\": [1, 2", ", 3]}"};
|
||||
for (const std::string chunk : chunks)
|
||||
{
|
||||
std::cout << "chunk " << chunk << " -> " << (push(chunk, value) ? "complete" : "incomplete") << '\n';
|
||||
}
|
||||
std::cout << value << '\n';
|
||||
|
||||
// a syntax error in the middle is not mistaken for missing data
|
||||
try
|
||||
{
|
||||
push("[1, ]", value);
|
||||
}
|
||||
catch (const json::parse_error& e)
|
||||
{
|
||||
std::cout << e.what() << '\n';
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```
|
||||
chunk {"na -> incomplete
|
||||
chunk me": "Ma -> incomplete
|
||||
chunk ry", "l -> incomplete
|
||||
chunk ist": [1, 2 -> incomplete
|
||||
chunk , 3]} -> complete
|
||||
{"list":[1,2,3],"name":"Mary"}
|
||||
[json.exception.parse_error.101] parse error at line 1, column 5: syntax error while parsing value - unexpected ']'; expected '[', '{', or a literal
|
||||
```
|
||||
|
||||
The example feeds a document in five chunks. The first four are reported as incomplete, and a syntax error is reported as such, not as missing data.
|
||||
|
||||
Limits of buffering
|
||||
|
||||
- The buffer must contain exactly one value. Data that belongs to the next value is a syntax error. Use framing if values follow each other without a separator.
|
||||
- A number at the end of the buffer is a complete value, even if more digits are still to come (`12` may be the start of `123`). Buffering cannot detect this; use framing for top-level numbers.
|
||||
- Every attempt parses the buffer from the beginning, so the total work grows quadratically with the size of the value. Limit the size of the buffer for input you do not trust (see [untrusted input](https://json.nlohmann.me/features/parsing/untrusted_input/index.md)).
|
||||
|
||||
## Input too large for memory
|
||||
|
||||
The input does not have to become one huge `json` value. The two tools below process it piece by piece.
|
||||
|
||||
### Handle and discard elements with a callback
|
||||
|
||||
A [parser callback](https://json.nlohmann.me/features/parsing/parser_callbacks/index.md) is called for every finished value. If it returns `false`, the value is discarded and its memory is released at once. For a file that consists of one huge array (or one huge object) of records, handle each record at depth 1 and discard it:
|
||||
|
||||
Example: handle and discard each record of a large array
|
||||
|
||||
```
|
||||
#include <iostream>
|
||||
#include <sstream>
|
||||
#include <nlohmann/json.hpp>
|
||||
|
||||
using json = nlohmann::json;
|
||||
|
||||
void handle(const json& record)
|
||||
{
|
||||
std::cout << "handling " << record << '\n';
|
||||
}
|
||||
|
||||
int main()
|
||||
{
|
||||
std::istringstream file(R"([{"id": 1}, {"id": 2}, {"id": 3}])");
|
||||
|
||||
json result = json::parse(file, [](int depth, json::parse_event_t event, json & record)
|
||||
{
|
||||
if (depth == 1 && event == json::parse_event_t::object_end)
|
||||
{
|
||||
handle(record);
|
||||
return false; // discard the record
|
||||
}
|
||||
return true;
|
||||
});
|
||||
|
||||
std::cout << "result: " << result << '\n';
|
||||
}
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```
|
||||
handling {"id":1}
|
||||
handling {"id":2}
|
||||
handling {"id":3}
|
||||
result: []
|
||||
```
|
||||
|
||||
Only one record is in memory at any time. The same recipe, with more detail, is in the [parser callbacks](https://json.nlohmann.me/features/parsing/parser_callbacks/#recipe-streaming-a-large-homogeneous-array) page. To skip entire members you do not need, discard them when their key is read; see the [example](https://json.nlohmann.me/features/parsing/parser_callbacks/index.md) for `parse_event_t::key`.
|
||||
|
||||
### Extract a few values and stop with SAX
|
||||
|
||||
With the [SAX interface](https://json.nlohmann.me/features/parsing/sax_interface/index.md), no value is built at all, and **returning `false` from any handler function stops the parser**. [`sax_parse`](https://json.nlohmann.me/api/basic_json/sax_parse/index.md) then returns `false`, and the rest of the input is not read. The example finds the first `id` in a document without parsing the remainder:
|
||||
|
||||
Example: stop parsing as soon as a value is found
|
||||
|
||||
```
|
||||
#include <iostream>
|
||||
#include <nlohmann/json.hpp>
|
||||
|
||||
using json = nlohmann::json;
|
||||
|
||||
struct find_id : nlohmann::json_sax<json>
|
||||
{
|
||||
bool next_is_id = false;
|
||||
json::number_unsigned_t id = 0;
|
||||
|
||||
bool key(std::string& name) override
|
||||
{
|
||||
next_is_id = (name == "id");
|
||||
return true;
|
||||
}
|
||||
|
||||
bool number_unsigned(json::number_unsigned_t value) override
|
||||
{
|
||||
if (next_is_id)
|
||||
{
|
||||
id = value;
|
||||
return false; // found: stop parsing
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
bool null() override
|
||||
{
|
||||
return true;
|
||||
}
|
||||
bool boolean(bool /*val*/) override
|
||||
{
|
||||
return true;
|
||||
}
|
||||
bool number_integer(json::number_integer_t /*val*/) override
|
||||
{
|
||||
return true;
|
||||
}
|
||||
bool number_float(json::number_float_t /*val*/, const std::string& /*s*/) override
|
||||
{
|
||||
return true;
|
||||
}
|
||||
bool string(std::string& /*val*/) override
|
||||
{
|
||||
return true;
|
||||
}
|
||||
bool binary(json::binary_t& /*val*/) override
|
||||
{
|
||||
return true;
|
||||
}
|
||||
bool start_object(std::size_t /*elements*/) override
|
||||
{
|
||||
return true;
|
||||
}
|
||||
bool end_object() override
|
||||
{
|
||||
return true;
|
||||
}
|
||||
bool start_array(std::size_t /*elements*/) override
|
||||
{
|
||||
return true;
|
||||
}
|
||||
bool end_array() override
|
||||
{
|
||||
return true;
|
||||
}
|
||||
bool parse_error(std::size_t /*position*/, const std::string& /*last_token*/, const json::exception& /*ex*/) override
|
||||
{
|
||||
return false;
|
||||
}
|
||||
};
|
||||
|
||||
int main()
|
||||
{
|
||||
find_id handler;
|
||||
const bool completed = json::sax_parse(R"({"x": [1, 2, 3], "id": 42, "rest": [4, 5, 6]})", &handler);
|
||||
|
||||
std::cout << std::boolalpha;
|
||||
std::cout << "parse completed: " << completed << '\n';
|
||||
std::cout << "id: " << handler.id << '\n';
|
||||
}
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```
|
||||
parse completed: false
|
||||
id: 42
|
||||
```
|
||||
|
||||
Note that `sax_parse` returns `false` both when a handler stopped the parse and when the input is invalid, so record the reason in the handler (here: `id`) if you need to tell the two apart.
|
||||
|
||||
### Read parts of a large document with a view
|
||||
|
||||
If the whole text is available and you only need to read some of it, a [`json_document`](https://json.nlohmann.me/features/json_view/index.md) from `<nlohmann/json_view.hpp>` (added in version 3.13.0 unreleased) avoids the `basic_json` tree altogether: it keeps the text, builds a flat index of the values, and gives read-only [views](https://json.nlohmann.me/api/basic_json_view/index.md) of the parts you look at. Only what you [`materialize()`](https://json.nlohmann.me/api/basic_json_view/materialize/index.md) becomes a `basic_json` value. See [Zero-copy JSON views](https://json.nlohmann.me/features/json_view/index.md) for details and for the lifetime rules.
|
||||
|
||||
## See also
|
||||
|
||||
- [Parsing](https://json.nlohmann.me/features/parsing/index.md) - overview of the parsing functions
|
||||
- [JSON Lines](https://json.nlohmann.me/features/parsing/json_lines/index.md) - newline-delimited JSON
|
||||
- [Parser callbacks](https://json.nlohmann.me/features/parsing/parser_callbacks/index.md) - discard values while parsing
|
||||
- [SAX interface](https://json.nlohmann.me/features/parsing/sax_interface/index.md) - event-based parsing
|
||||
- [Zero-copy JSON views](https://json.nlohmann.me/features/json_view/index.md) - read parts of a large document without building a tree
|
||||
- [Untrusted input](https://json.nlohmann.me/features/parsing/untrusted_input/index.md) - limits for input you do not control
|
||||
Reference in new issue
Block a user