json_view: indented and ensure_ascii dumps through the fast writer (#5889)
* Write the indented output of json_view's dump() with the fast writer
The compact writer of dump() (cursor in a register, fixed-size copies of
tokens) served dump(-1) only; every dump(indent), and everything with
ensure_ascii that was indented, went through a generic walk that calls put()
for each token and put_repeated() (a memset call) for each line break. The
fast writer now writes indented output, too, with or without ensure_ascii, and
the generic walk is gone. A line break is a copy of a 64-byte ruler (a line
break and 63 indentation characters) when the line is at most 64 bytes, and a
store plus memset beyond that. The depth is counted only in the indented
instantiations (counting it in the compact one cost an editable document 5 %
with Clang). The indented walks are out of line (dump_pretty()): inlined
into dump() next to the compact ones, they made compact dumps of integer
arrays 7 % slower.
A decoded string (one with escape sequences) of a damaged image loaded with
image_check::bounds can contain invalid UTF-8. As before, the compact dump
writes such bytes unchanged, and the indented dump throws type_error.316; its
runs are found by the lexer's validating scan (a scalar validation of every
sequence made dump(2) of decoded CJK text twice as slow as before).
dump(2) instructions per call, macOS arm64 (before -> after):
twitter citm canada jeopardy(4k)
json 3.75M -> 1.85M 8.28M -> 3.93M 67.1M -> 56.0M 9.56M -> 4.81M
editable 3.82M -> 1.84M 8.48M -> 4.02M 68.1M -> 56.2M 9.66M -> 4.75M
Linux instructions (GCC 13 / Clang 18), dump(2) and dump(2, ' ', true):
twitter -25 % / -30 % -11 % / -11 %
citm -29 % / -28 % -24 % / -24 %
jeopardy(4k) -25 % / -32 % -20 % / -24 %
CJK text with \u escapes -6 % / -11 % -2 % / -2 %
The compact dump is unchanged (within +0.4 % on Linux, within noise on the Mac).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Scan the strings of json_view's dump() with ensure_ascii 16 bytes at a time
With ensure_ascii, a string of the source that has a byte >= 0x7F was written
by a generic path: put() per piece and a validate-and-decode step for every
character beyond ASCII. It now goes through a writer with the cursor in a
local variable. The part of the string that has nothing to escape is found
16 bytes at a time (SSE2 or NEON; the existing 8-byte scan without vector
support) and copied; strings that are ASCII throughout are still decided by
the 8-byte check of dump() with ensure_ascii (a 16-byte scan for them was
2-5 % slower for short strings). The two-byte sequences (lead C2..DF) and
the three-byte sequences whose continuation bytes may have any value (lead
E1..EF except ED) are decoded with a few compares; E0, ED, four-byte
sequences, and damaged input go through validate_one_utf8() as before, so
damaged images still throw type_error.316.
dump(-1, ' ', true) instructions per call, macOS arm64 (develop -> now);
the compact dump without ensure_ascii and dumps of ASCII text are unchanged:
twitter 5.42M -> 3.57M citm 3.28M -> 3.24M jeopardy(4k) 5.78M -> 5.65M
CJK text, raw 66.5M -> 35.3M escaped \u 32.7M -> 17.0M
Latin text with accents 11.5M -> 9.4M ASCII text 4.40M -> 4.40M
Linux instructions (GCC 13 / Clang 18), dump(-1, ' ', true):
twitter -9 % / -33 % CJK text -14 % / -47 % Latin text -8 % / -18 %
citm -0.2 % / -0.4 % ASCII text -0.1 % / -0.3 %
The compact dump without ensure_ascii is unchanged (canada 0.0 %).
The unit test covers every character of the Basic Multilingual Plane, and a
character that needs an escape at each position of strings of 0..40 bytes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Find the end of the plain run of a decoded string 16 bytes at a time with SSE2
The strings of a document that had escape sequences are written in runs
without a quote, backslash, or control character; the compact dump found the
end of a run 8 bytes at a time with a SWAR test. With SSE2 (x86-64) it now
compares 16 bytes per step. NEON keeps the SWAR code: there the mask of a
step costs more than it saves on the short runs that are typical here.
Instructions of dump(), Linux (GCC 13 / Clang 18):
twitter.json 1.99M -> 1.88M (-5.2 %) / 1.97M -> 1.92M (-3.0 %)
CJK text with \u escapes 5.98M -> 4.52M (-24.5 %) / 5.30M -> 4.25M (-19.9 %)
ASCII text with a few escapes 4.28M -> 4.09M (-4.4 %) / 4.24M -> 4.11M (-3.2 %)
jeopardy (4,000 questions) 5.08M -> 4.78M (-5.8 %) / 4.97M -> 4.80M (-3.4 %)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Rename depth_counter's open() and close() to enter() and leave()
Flawfinder took them for the file functions open() and close().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the clang-tidy findings in json_view's dump and its tests
- the SSE2 loop of the string scan is within NOLINTBEGIN/END
(portability-simd-intrinsics), as in to_chars.hpp
- depth_counter<false>'s leave() and get() keep the interface of
depth_counter<true> instead of being static
- write_ascii() compares the room left as a size_t variable
- the test's texts are a std::array, and a JSON fragment with escapes
is a raw string literal
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move dump_ascii() and dump_pretty() next to dump()
Address a review comment: the two dispatchers that dump() calls now follow
it directly.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann authored and GitHub committed
1 parent
1e9222f955
commit
e9c4fe3cdf
4 files changed
+894
-512
No files matched your search
@@ -186,6 +186,133 @@ TEST_CASE("json_view dump")
|
||||
CHECK(deep_document.root().dump() == deep);
|
||||
}
|
||||
|
||||
SECTION("ensure_ascii: a character that needs an escape at each position of a string")
|
||||
{
|
||||
// string lengths around the width of the vector steps, in a string value, a key, and a value near the end of the text
|
||||
for (const char* const special :
|
||||
{
|
||||
"\xC3\xA9", "\xE4\xB8\xAD", "\xF0\x9F\x98\x80", "\x7F", "\\n", "\\u0001", "\\\"", "\\\\"
|
||||
})
|
||||
{
|
||||
for (std::size_t length = 0; length <= 40; ++length)
|
||||
{
|
||||
for (std::size_t position = 0; position <= length; ++position)
|
||||
{
|
||||
std::string content(length, 'a');
|
||||
content.insert(position, special);
|
||||
const std::array<std::string, 3> texts =
|
||||
{
|
||||
{
|
||||
"\"" + content + "\"",
|
||||
"[\"" + content + "\"]",
|
||||
"{\"" + content + "\":1}",
|
||||
}
|
||||
};
|
||||
for (const std::string& text : texts)
|
||||
{
|
||||
CAPTURE(text)
|
||||
const ordered_json_document d = ordered_json_document::parse(text);
|
||||
const ordered_json j = ordered_json::parse(text);
|
||||
CHECK(d.root().dump(-1, ' ', true) == j.dump(-1, ' ', true));
|
||||
CHECK(d.root().dump(2, ' ', true) == j.dump(2, ' ', true));
|
||||
CHECK(d.root().dump() == j.dump());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
SECTION("ensure_ascii: every character of the Basic Multilingual Plane and some beyond")
|
||||
{
|
||||
// (the two- and three-byte sequences, including the ranges that need a check: E0, ED, EF BF BF)
|
||||
std::string text = "[";
|
||||
std::size_t in_chunk = 0;
|
||||
const auto append = [&](std::uint32_t cp)
|
||||
{
|
||||
if (cp < 0x80)
|
||||
{
|
||||
if (cp < 0x20 || cp == '"' || cp == '\\')
|
||||
{
|
||||
return;
|
||||
}
|
||||
text += static_cast<char>(cp);
|
||||
}
|
||||
else if (cp < 0x800)
|
||||
{
|
||||
text += static_cast<char>(0xC0 | (cp >> 6));
|
||||
text += static_cast<char>(0x80 | (cp & 0x3F));
|
||||
}
|
||||
else if (cp < 0x10000)
|
||||
{
|
||||
text += static_cast<char>(0xE0 | (cp >> 12));
|
||||
text += static_cast<char>(0x80 | ((cp >> 6) & 0x3F));
|
||||
text += static_cast<char>(0x80 | (cp & 0x3F));
|
||||
}
|
||||
else
|
||||
{
|
||||
text += static_cast<char>(0xF0 | (cp >> 18));
|
||||
text += static_cast<char>(0x80 | ((cp >> 12) & 0x3F));
|
||||
text += static_cast<char>(0x80 | ((cp >> 6) & 0x3F));
|
||||
text += static_cast<char>(0x80 | (cp & 0x3F));
|
||||
}
|
||||
};
|
||||
for (std::uint32_t cp = 0x20; cp < 0x10400; ++cp)
|
||||
{
|
||||
if (cp >= 0xD800 && cp < 0xE000)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
if (in_chunk == 0)
|
||||
{
|
||||
text += text.size() > 1 ? ",\"" : "\"";
|
||||
}
|
||||
append(cp);
|
||||
if (++in_chunk == 37)
|
||||
{
|
||||
text += "\"";
|
||||
in_chunk = 0;
|
||||
}
|
||||
}
|
||||
text += in_chunk != 0 ? "\"]" : "]";
|
||||
const ordered_json_document d = ordered_json_document::parse(text);
|
||||
const ordered_json j = ordered_json::parse(text);
|
||||
CHECK(d.root().dump(-1, ' ', true) == j.dump(-1, ' ', true));
|
||||
CHECK(d.root().dump(2, ' ', true) == j.dump(2, ' ', true));
|
||||
CHECK(d.root().dump() == j.dump());
|
||||
CHECK(d.root().dump(2) == j.dump(2));
|
||||
}
|
||||
|
||||
SECTION("deep nesting, indented")
|
||||
{
|
||||
// indentations of up to and beyond the width of the writer's fixed-size copy
|
||||
for (const int depth :
|
||||
{
|
||||
1, 5, 20, 70, 300
|
||||
})
|
||||
{
|
||||
std::string text;
|
||||
for (int i = 0; i < depth; ++i)
|
||||
{
|
||||
text += i % 2 == 0 ? "[1," : R"({"k":2,"\u00e9":)";
|
||||
}
|
||||
text += "[]";
|
||||
for (int i = depth - 1; i >= 0; --i)
|
||||
{
|
||||
text += i % 2 == 0 ? "]" : "}";
|
||||
}
|
||||
const ordered_json_document d = ordered_json_document::parse(text);
|
||||
const ordered_json j = ordered_json::parse(text);
|
||||
for (const int indent :
|
||||
{
|
||||
0, 1, 2, 3, 8
|
||||
})
|
||||
{
|
||||
CHECK(d.root().dump(indent) == j.dump(indent));
|
||||
CHECK(d.root().dump(indent, '\t') == j.dump(indent, '\t'));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
SECTION("the output buffer of a small value is small")
|
||||
{
|
||||
// an escaped key after the value: its node lies in the arena, so the
|
||||
|
||||
@@ -682,6 +682,26 @@ TEST_CASE("json_view images: ownership")
|
||||
#if !defined(JSON_NOEXCEPTION)
|
||||
TEST_CASE("json_view images: errors")
|
||||
{
|
||||
SECTION("invalid UTF-8 of a decoded string: the compact dump writes it, the others throw")
|
||||
{
|
||||
const json_document d = json_document::parse(R"({"k\n":"\u00e9x"})");
|
||||
std::vector<std::uint8_t> image = d.save();
|
||||
const std::size_t arena = text_at(image) + header_size_field(image, 16) + 1;
|
||||
image[arena + 3] = 0x28; // the key is "k\n"; the value starts at 3: 0xC3 0x28 is invalid
|
||||
const json_document loaded = json_document::load(image, image_check::bounds);
|
||||
CHECK(loaded.root().dump().size() == 13);
|
||||
for (const int indent :
|
||||
{
|
||||
0, 2
|
||||
})
|
||||
{
|
||||
CHECK_THROWS_AS(loaded.root().dump(indent), json::type_error);
|
||||
CHECK_THROWS_AS(loaded.root().dump(indent, ' ', true), json::type_error);
|
||||
}
|
||||
CHECK_THROWS_AS(loaded.root().dump(-1, ' ', true), json::type_error);
|
||||
CHECK_THROWS_WITH_AS(loaded.root().dump(2), "[json.exception.type_error.316] invalid UTF-8 byte at index 1: 0x28", json::type_error&);
|
||||
}
|
||||
|
||||
SECTION("a literal as the root: dump() after loading")
|
||||
{
|
||||
for (const char* text :
|
||||
|
||||
Reference in new issue
Block a user