Compare commits

..
Author SHA1 Message Date
Niels Lohmann 86049706a0 Rename depth_counter's open() and close() to enter() and leave()
Flawfinder took them for the file functions open() and close().

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-22 13:18:11 +02:00
Niels Lohmann cc946adf68 Find the end of the plain run of a decoded string 16 bytes at a time with SSE2
The strings of a document that had escape sequences are written in runs
without a quote, backslash, or control character; the compact dump found the
end of a run 8 bytes at a time with a SWAR test. With SSE2 (x86-64) it now
compares 16 bytes per step. NEON keeps the SWAR code: there the mask of a
step costs more than it saves on the short runs that are typical here.

Instructions of dump(), Linux (GCC 13 / Clang 18):

  twitter.json                  1.99M -> 1.88M (-5.2 %) / 1.97M -> 1.92M (-3.0 %)
  CJK text with \u escapes      5.98M -> 4.52M (-24.5 %) / 5.30M -> 4.25M (-19.9 %)
  ASCII text with a few escapes 4.28M -> 4.09M (-4.4 %) / 4.24M -> 4.11M (-3.2 %)
  jeopardy (4,000 questions)    5.08M -> 4.78M (-5.8 %) / 4.97M -> 4.80M (-3.4 %)

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-22 09:30:39 +02:00
Niels Lohmann 272d07b48b Scan the strings of json_view's dump() with ensure_ascii 16 bytes at a time
With ensure_ascii, a string of the source that has a byte >= 0x7F was written
by a generic path: put() per piece and a validate-and-decode step for every
character beyond ASCII. It now goes through a writer with the cursor in a
local variable. The part of the string that has nothing to escape is found
16 bytes at a time (SSE2 or NEON; the existing 8-byte scan without vector
support) and copied; strings that are ASCII throughout are still decided by
the 8-byte check of dump() with ensure_ascii (a 16-byte scan for them was
2-5 % slower for short strings). The two-byte sequences (lead C2..DF) and
the three-byte sequences whose continuation bytes may have any value (lead
E1..EF except ED) are decoded with a few compares; E0, ED, four-byte
sequences, and damaged input go through validate_one_utf8() as before, so
damaged images still throw type_error.316.

dump(-1, ' ', true) instructions per call, macOS arm64 (develop -> now);
the compact dump without ensure_ascii and dumps of ASCII text are unchanged:

  twitter 5.42M -> 3.57M    citm 3.28M -> 3.24M     jeopardy(4k) 5.78M -> 5.65M
  CJK text, raw 66.5M -> 35.3M    escaped \u 32.7M -> 17.0M
  Latin text with accents 11.5M -> 9.4M    ASCII text 4.40M -> 4.40M

Linux instructions (GCC 13 / Clang 18), dump(-1, ' ', true):

  twitter -9 % / -33 %    CJK text -14 % / -47 %    Latin text -8 % / -18 %
  citm -0.2 % / -0.4 %    ASCII text -0.1 % / -0.3 %

The compact dump without ensure_ascii is unchanged (canada 0.0 %).

The unit test covers every character of the Basic Multilingual Plane, and a
character that needs an escape at each position of strings of 0..40 bytes.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-22 09:30:36 +02:00
Niels Lohmann ef45593bda Write the indented output of json_view's dump() with the fast writer
The compact writer of dump() (cursor in a register, fixed-size copies of
tokens) served dump(-1) only; every dump(indent), and everything with
ensure_ascii that was indented, went through a generic walk that calls put()
for each token and put_repeated() (a memset call) for each line break. The
fast writer now writes indented output, too, with or without ensure_ascii, and
the generic walk is gone. A line break is a copy of a 64-byte ruler (a line
break and 63 indentation characters) when the line is at most 64 bytes, and a
store plus memset beyond that. The depth is counted only in the indented
instantiations (counting it in the compact one cost an editable document 5 %
with Clang). The indented walks are out of line (dump_pretty()): inlined
into dump() next to the compact ones, they made compact dumps of integer
arrays 7 % slower.

A decoded string (one with escape sequences) of a damaged image loaded with
image_check::bounds can contain invalid UTF-8. As before, the compact dump
writes such bytes unchanged, and the indented dump throws type_error.316; its
runs are found by the lexer's validating scan (a scalar validation of every
sequence made dump(2) of decoded CJK text twice as slow as before).

dump(2) instructions per call, macOS arm64 (before -> after):

               twitter         citm            canada          jeopardy(4k)
  json         3.75M -> 1.85M  8.28M -> 3.93M  67.1M -> 56.0M  9.56M -> 4.81M
  editable     3.82M -> 1.84M  8.48M -> 4.02M  68.1M -> 56.2M  9.66M -> 4.75M

Linux instructions (GCC 13 / Clang 18), dump(2) and dump(2, ' ', true):

  twitter      -25 % / -30 %      -11 % / -11 %
  citm         -29 % / -28 %      -24 % / -24 %
  jeopardy(4k) -25 % / -32 %      -20 % / -24 %
  CJK text with \u escapes  -6 % / -11 %   -2 % / -2 %

The compact dump is unchanged (within +0.4 % on Linux, within noise on the Mac).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-22 09:30:34 +02:00
11 changed files with 1304 additions and 4766 deletions

No files matched your search

-5
View File
@@ -339,11 +339,6 @@ The `serializer` (used by [`dump`](../api/basic_json/dump.md) and [`operator<<`]
- `output_stream_adapter` writes to a `std::ostream`. - `output_stream_adapter` writes to a `std::ostream`.
- `output_string_adapter` appends to a string. - `output_string_adapter` appends to a string.
The `binary_writer` does not call an adapter for every byte. The `to_*` functions that return a `std::vector` write
into the vector through a sink (`output_vector_sink`) that is not virtual. The functions that take an output adapter
write through a sink (`output_adapter_sink`) that collects the output in a buffer of one kibibyte and passes it to the
adapter in blocks, so that the adapter is called (and a stream locked) once per block instead of once per value.
## Value conversion ## Value conversion
Values are converted from and to other types with the `JSONSerializer` template parameter. The default, Values are converted from and to other types with the `JSONSerializer` template parameter. The default,
File diff suppressed because it is too large. Load diff
@@ -55,66 +55,6 @@ using output_adapter_t = std::shared_ptr<output_adapter_protocol<CharType>>;
/// output_vector_sink also appends writes of at least this size with insert() /// output_vector_sink also appends writes of at least this size with insert()
constexpr std::size_t max_reserve_size = 256; constexpr std::size_t max_reserve_size = 256;
/// @brief copy @a length characters, at most 32, without a call of memcpy()
///
/// Short strings are the typical payload of a binary value, and a call of
/// memcpy() costs more than copying them. A copy of 16 to 32 characters is two
/// possibly overlapping 16-byte copies, one of 8 to 15 two 8-byte copies, and
/// so on down to a single character; each is a fixed-size load and store.
/// Nothing outside of [src, src + length) is read or written.
template<typename CharType>
inline void copy_short(CharType* dst, const CharType* src, const std::size_t length, std::true_type /*is_byte*/) noexcept
{
JSON_ASSERT(length <= 32);
if (length >= 16)
{
std::array<char, 16> first; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::array<char, 16> last; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::memcpy(first.data(), src, 16);
std::memcpy(last.data(), src + (length - 16), 16);
std::memcpy(dst, first.data(), 16);
std::memcpy(dst + (length - 16), last.data(), 16);
}
else if (length >= 8)
{
std::array<char, 8> first; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::array<char, 8> last; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::memcpy(first.data(), src, 8);
std::memcpy(last.data(), src + (length - 8), 8);
std::memcpy(dst, first.data(), 8);
std::memcpy(dst + (length - 8), last.data(), 8);
}
else if (length >= 4)
{
std::array<char, 4> first; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::array<char, 4> last; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::memcpy(first.data(), src, 4);
std::memcpy(last.data(), src + (length - 4), 4);
std::memcpy(dst, first.data(), 4);
std::memcpy(dst + (length - 4), last.data(), 4);
}
else if (length != 0)
{
// 1 to 3: the first, the middle, and the last character
const CharType a = src[0];
const CharType b = src[length >> 1];
const CharType c = src[length - 1];
dst[0] = a;
dst[length >> 1] = b;
dst[length - 1] = c;
}
}
/// @brief copy @a length characters, of a character type wider than a byte
template<typename CharType>
inline void copy_short(CharType* dst, const CharType* src, const std::size_t length, std::false_type /*is_byte*/) noexcept
{
std::memcpy(dst, src, length * sizeof(CharType));
}
/// the number of characters an empty vector is grown to by output_vector_sink
constexpr std::size_t min_vector_sink_capacity = 64;
/// @brief non-virtual output sink writing into a std::vector /// @brief non-virtual output sink writing into a std::vector
/// ///
/// This sink is not part of the virtual output_adapter_protocol hierarchy: it is /// This sink is not part of the virtual output_adapter_protocol hierarchy: it is
@@ -138,7 +78,7 @@ class output_vector_sink
{ {
public: public:
explicit output_vector_sink(std::vector<CharType, AllocatorType>& vec) noexcept explicit output_vector_sink(std::vector<CharType, AllocatorType>& vec) noexcept
: v(&vec), cursor(vec.data() + vec.size()), limit(cursor) : v(&vec), position(vec.size())
{} {}
output_vector_sink(const output_vector_sink&) = delete; output_vector_sink(const output_vector_sink&) = delete;
@@ -146,7 +86,7 @@ class output_vector_sink
output_vector_sink& operator=(output_vector_sink&&) = delete; output_vector_sink& operator=(output_vector_sink&&) = delete;
output_vector_sink(output_vector_sink&& other) noexcept output_vector_sink(output_vector_sink&& other) noexcept
: v(other.v), cursor(other.cursor), limit(other.limit) : v(other.v), position(other.position)
{ {
other.v = nullptr; other.v = nullptr;
} }
@@ -155,7 +95,7 @@ class output_vector_sink
{ {
if (v != nullptr) if (v != nullptr)
{ {
v->resize(position()); v->resize(position);
} }
} }
@@ -164,11 +104,11 @@ class output_vector_sink
JSON_HEDLEY_ALWAYS_INLINE JSON_HEDLEY_ALWAYS_INLINE
void write_character(CharType c) void write_character(CharType c)
{ {
if (JSON_HEDLEY_UNLIKELY(cursor == limit)) if (JSON_HEDLEY_UNLIKELY(position == v->size()))
{ {
grow(1); grow(1);
} }
*cursor++ = c; (*v)[position++] = c;
} }
// no JSON_HEDLEY_NON_NULL here: binary_writer legitimately passes a null // no JSON_HEDLEY_NON_NULL here: binary_writer legitimately passes a null
@@ -177,7 +117,7 @@ class output_vector_sink
JSON_HEDLEY_ALWAYS_INLINE JSON_HEDLEY_ALWAYS_INLINE
void write_characters(const CharType* s, std::size_t length) void write_characters(const CharType* s, std::size_t length)
{ {
if (JSON_HEDLEY_UNLIKELY(space() < length)) if (JSON_HEDLEY_UNLIKELY(v->size() - position < length))
{ {
if (length >= max_reserve_size) if (length >= max_reserve_size)
{ {
@@ -186,15 +126,10 @@ class output_vector_sink
} }
grow(length); grow(length);
} }
if (length <= 32) if (length != 0)
{ {
copy_short(cursor, s, length, std::integral_constant<bool, sizeof(CharType) == 1>()); std::memcpy(v->data() + position, s, length * sizeof(CharType));
cursor += length; position += length;
}
else
{
std::memcpy(cursor, s, length * sizeof(CharType));
cursor += length;
} }
} }
@@ -209,37 +144,21 @@ class output_vector_sink
CharType* reserve(const std::size_t length) CharType* reserve(const std::size_t length)
{ {
JSON_ASSERT(length <= max_reserve_size); JSON_ASSERT(length <= max_reserve_size);
if (JSON_HEDLEY_UNLIKELY(space() < length)) if (JSON_HEDLEY_UNLIKELY(v->size() - position < length))
{ {
grow(length); grow(length);
} }
return cursor; return v->data() + position;
} }
/// keep the first @a length characters written to the space from reserve() /// keep the first @a length characters written to the space from reserve()
JSON_HEDLEY_ALWAYS_INLINE JSON_HEDLEY_ALWAYS_INLINE
void commit(const std::size_t length) noexcept void commit(const std::size_t length) noexcept
{ {
cursor += length; position += length;
} }
/// nothing to pass on: the characters are in the vector (see output_adapter_sink)
void flush() noexcept
{}
private: private:
/// number of characters of *v written so far
std::size_t position() const noexcept
{
return static_cast<std::size_t>(cursor - v->data());
}
/// number of characters that fit the vector behind the write position
std::size_t space() const noexcept
{
return static_cast<std::size_t>(limit - cursor);
}
/// make room for at least @a length more characters after the position; /// make room for at least @a length more characters after the position;
/// kept out of line so that the write functions stay small enough for /// kept out of line so that the write functions stay small enough for
/// GCC to inline them into the writer /// GCC to inline them into the writer
@@ -247,15 +166,12 @@ class output_vector_sink
void grow(const std::size_t length) void grow(const std::size_t length)
{ {
// use the capacity the vector already has, else at least double it; // use the capacity the vector already has, else at least double it;
// never less than min_vector_sink_capacity, so that a small value does not // never less than 64 characters, so that a small value does not
// reallocate for every few bytes // reallocate for every few bytes
const std::size_t written = position(); const std::size_t needed = position + length;
const std::size_t needed = written + length; v->resize((std::max)(std::size_t{64}, needed <= v->capacity()
v->resize((std::max)(min_vector_sink_capacity, needed <= v->capacity()
? v->capacity() ? v->capacity()
: (std::max)(needed, v->capacity() * 2))); : (std::max)(needed, v->capacity() * 2)));
cursor = v->data() + written;
limit = v->data() + v->size();
} }
/// append @a length (at least max_reserve_size) characters that do not fit /// append @a length (at least max_reserve_size) characters that do not fit
@@ -265,24 +181,20 @@ class output_vector_sink
JSON_HEDLEY_NEVER_INLINE JSON_HEDLEY_NEVER_INLINE
void append(const CharType* s, const std::size_t length) void append(const CharType* s, const std::size_t length)
{ {
const std::size_t written = position(); v->resize(position);
v->resize(written); if (v->capacity() - position < length)
if (v->capacity() - written < length)
{ {
// grow geometrically, with room for the writes that follow, so // grow geometrically, with room for the writes that follow, so
// that the next grow() need not reallocate right away // that the next grow() need not reallocate right away
v->reserve((std::max)(v->capacity() * 2, written + length + max_reserve_size)); v->reserve((std::max)(v->capacity() * 2, position + length + max_reserve_size));
} }
v->insert(v->end(), s, s + length); v->insert(v->end(), s, s + length);
cursor = v->data() + v->size(); position = v->size();
limit = cursor;
} }
std::vector<CharType, AllocatorType>* v; std::vector<CharType, AllocatorType>* v;
/// the write position: in *v, at the number of characters written so far /// number of characters of *v written so far
CharType* cursor; std::size_t position;
/// the end of *v
CharType* limit;
}; };
/// output adapter for byte vectors /// output adapter for byte vectors
@@ -365,17 +277,9 @@ class output_string_adapter : public output_adapter_protocol<CharType>
/// ///
/// Wraps the polymorphic output_adapter_t so the same binary_writer template can /// Wraps the polymorphic output_adapter_t so the same binary_writer template can
/// also target arbitrary adapters (output streams, strings, user-provided /// also target arbitrary adapters (output streams, strings, user-provided
/// adapters) via the `output_adapter`-based overloads. /// adapters) via the `output_adapter`-based overloads. Each write still goes
/// /// through one virtual call, exactly as before; only the concrete sinks above
/// A write through the adapter is a virtual call, and for a stream a call with /// avoid it.
/// a sentry on top, which costs more than encoding the few bytes of most values.
/// The sink therefore collects what is written in a buffer and passes it on in
/// one piece when it is full, a write is too large for it, or @ref flush is
/// called. Everything written reaches the adapter in the same order and with
/// the same bytes; the destructor passes on what is left, so that a write that
/// throws leaves the adapter with exactly the bytes written before it. Only
/// @ref flush can report an error of the adapter (the destructor must not
/// throw), so it has to be called once the value is written.
template<typename CharType> template<typename CharType>
class output_adapter_sink class output_adapter_sink
{ {
@@ -386,121 +290,36 @@ class output_adapter_sink
JSON_ASSERT(oa); JSON_ASSERT(oa);
} }
output_adapter_sink(const output_adapter_sink&) = delete;
output_adapter_sink& operator=(const output_adapter_sink&) = delete;
output_adapter_sink& operator=(output_adapter_sink&&) = delete;
output_adapter_sink(output_adapter_sink&& other) noexcept
: oa(std::move(other.oa)), used(other.used)
{
std::memcpy(buffer.data(), other.buffer.data(), other.used * sizeof(CharType));
other.used = 0;
}
~output_adapter_sink()
{
// a normal return has flushed already, so this is for a write that threw
if (used != 0)
{
JSON_TRY
{
flush();
}
JSON_CATCH(...)
{
// nothing can be done about an adapter that throws here
}
}
}
JSON_HEDLEY_ALWAYS_INLINE
void write_character(CharType c) void write_character(CharType c)
{ {
if (JSON_HEDLEY_UNLIKELY(used == buffer_size)) oa->write_character(c);
{
flush();
}
buffer[used++] = c;
} }
// no JSON_HEDLEY_NON_NULL: forwards (null, 0) for empty payloads, exactly as // no JSON_HEDLEY_NON_NULL: forwards (null, 0) for empty payloads, exactly as
// the type-erased path already did before this sink existed // the type-erased path already did before this sink existed
JSON_HEDLEY_ALWAYS_INLINE
void write_characters(const CharType* s, std::size_t length) void write_characters(const CharType* s, std::size_t length)
{ {
if (JSON_HEDLEY_UNLIKELY(buffer_size - used < length)) oa->write_characters(s, length);
{
write_large(s, length);
return;
}
if (length <= 32)
{
copy_short(buffer.data() + used, s, length, std::integral_constant<bool, sizeof(CharType) == 1>());
used += length;
}
else
{
std::memcpy(buffer.data() + used, s, length * sizeof(CharType));
used += length;
}
} }
/// space for writing up to @a length characters (see /// space for writing up to @a length characters (see
/// output_vector_sink::reserve) /// output_vector_sink::reserve)
JSON_HEDLEY_ALWAYS_INLINE CharType* reserve(const std::size_t length) noexcept
CharType* reserve(const std::size_t length)
{ {
JSON_ASSERT(length <= max_reserve_size); JSON_ASSERT(length <= max_reserve_size);
if (JSON_HEDLEY_UNLIKELY(buffer_size - used < length)) static_cast<void>(length);
{ return reserved.data();
flush();
}
return buffer.data() + used;
} }
/// keep the first @a length characters written to the space from reserve() /// write the first @a length characters written to the space from reserve()
JSON_HEDLEY_ALWAYS_INLINE void commit(const std::size_t length)
void commit(const std::size_t length) noexcept
{ {
used += length; oa->write_characters(reserved.data(), length);
}
/// pass everything written so far on to the adapter
void flush()
{
if (used != 0)
{
// reset first: the characters count as written even if the adapter throws
const std::size_t length = used;
used = 0;
oa->write_characters(buffer.data(), length);
}
} }
private: private:
/// a write that does not fit the space left in the buffer
JSON_HEDLEY_NEVER_INLINE
void write_large(const CharType* s, const std::size_t length)
{
flush();
if (length >= buffer_size)
{
oa->write_characters(s, length);
}
else
{
std::memcpy(buffer.data(), s, length * sizeof(CharType));
used = length;
}
}
/// large enough for reserve() at any time
static constexpr std::size_t buffer_size = 4 * max_reserve_size;
output_adapter_t<CharType> oa; output_adapter_t<CharType> oa;
/// number of characters of @ref buffer not yet passed on std::array<CharType, max_reserve_size> reserved{};
std::size_t used = 0;
std::array<CharType, buffer_size> buffer{};
}; };
/// @brief whether std::basic_string<CharType> has a non-deprecated std::char_traits /// @brief whether std::basic_string<CharType> has a non-deprecated std::char_traits
+361 -247
View File
@@ -23,6 +23,7 @@
#include <nlohmann/detail/view/macro_scope.hpp> #include <nlohmann/detail/view/macro_scope.hpp>
#include <nlohmann/detail/view/node.hpp> #include <nlohmann/detail/view/node.hpp>
#include <nlohmann/detail/view/number.hpp> #include <nlohmann/detail/view/number.hpp>
#include <nlohmann/detail/view/simd.hpp>
NLOHMANN_JSON_NAMESPACE_BEGIN NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail namespace detail
@@ -80,13 +81,6 @@ class output_buffer
put(s, N - 1); put(s, N - 1);
} }
void put_repeated(char c, std::size_t n)
{
reserve(n);
std::memset(m_pos, c, n);
m_pos += n;
}
/// the write position and the end of the writable space, for a writer /// the write position and the end of the writable space, for a writer
/// that keeps the position in a local variable (set_cursor() hands it back) /// that keeps the position in a local variable (set_cursor() hands it back)
char* cursor() const noexcept char* cursor() const noexcept
@@ -143,9 +137,25 @@ class output_buffer
/// image_check::bounds can have others, which are then written unchanged). /// image_check::bounds can have others, which are then written unchanged).
inline std::size_t plain_output_run(const unsigned char* s, std::size_t n) noexcept inline std::size_t plain_output_run(const unsigned char* s, std::size_t n) noexcept
{ {
std::size_t i = 0;
#if NLOHMANN_VIEW_SSE2
// (16 bytes per step; with NEON, the mask of a step costs more than it saves on the
// short runs that are typical here)
for (; i + 16 <= n; i += 16)
{
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(s + i))); // NOLINT(bugprone-casting-through-void)
// (a byte is below 0x20 if it is its own minimum with 0x1F)
const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_cmpeq_epi8(_mm_min_epu8(in, _mm_set1_epi8(0x1F)), in));
const auto mask = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
if (mask != 0)
{
return i + static_cast<std::size_t>(count_trailing_zeros(mask));
}
}
#endif
constexpr std::uint64_t ones = 0x0101010101010101ull; constexpr std::uint64_t ones = 0x0101010101010101ull;
constexpr std::uint64_t high = 0x8080808080808080ull; constexpr std::uint64_t high = 0x8080808080808080ull;
std::size_t i = 0;
for (; i + 8 <= n; i += 8) for (; i + 8 <= n; i += 8)
{ {
const std::uint64_t v = read_eight_bytes(s + i); const std::uint64_t v = read_eight_bytes(s + i);
@@ -196,6 +206,70 @@ inline bool is_ascii_without_delete(const char* s, std::size_t n) noexcept
return true; return true;
} }
#if NLOHMANN_VIEW_VECTOR
/// the bytes of the 16 at p that ensure_ascii cannot copy (a quote, a
/// backslash, a control character, DEL, or not ASCII; a signed compare with
/// 0x20 finds control characters and non-ASCII bytes at once), one bit per
/// byte (x86) or one nibble per byte (NEON)
NLOHMANN_VIEW_ALWAYS_INLINE std::uint64_t ascii_stop_mask(const unsigned char* p) noexcept
{
#if NLOHMANN_VIEW_NEON
const uint8x16_t in = vld1q_u8(p);
const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))),
vorrq_u8(vceqq_u8(in, vdupq_n_u8(0x7F)), vcltq_s8(vreinterpretq_s8_u8(in), vdupq_n_s8(0x20))));
return vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0);
#else
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8(0x7F)), _mm_cmplt_epi8(in, _mm_set1_epi8(0x20))));
return static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
#endif
}
/// the width in bits of a byte in the mask of ascii_stop_mask()
constexpr unsigned ascii_stop_bits() noexcept
{
return NLOHMANN_VIEW_NEON ? 4u : 1u;
}
#endif
/// As find_ascii_copyable_run(), 16 bytes per step. end is the end of the
/// buffer that holds the n bytes at p (a short run is read as 16 bytes if
/// they are in the buffer).
inline std::size_t ascii_copyable_run(const unsigned char* p, std::size_t n, const unsigned char* end) noexcept
{
#if NLOHMANN_VIEW_VECTOR
if (n >= 16)
{
std::size_t i = 0;
for (; i + 16 <= n; i += 16)
{
const std::uint64_t mask = ascii_stop_mask(p + i);
if (mask != 0)
{
return i + (static_cast<std::size_t>(count_trailing_zeros(mask)) / ascii_stop_bits());
}
}
if (i == n)
{
return n;
}
// the last 16 bytes: those before i are known to be copyable
const std::uint64_t mask = ascii_stop_mask(p + n - 16);
return mask != 0 ? n - 16 + (static_cast<std::size_t>(count_trailing_zeros(mask)) / ascii_stop_bits()) : n;
}
if (end - p >= 16)
{
// (the bytes after the run are masked out)
const std::uint64_t mask = ascii_stop_mask(p) & ((std::uint64_t{1} << (n * ascii_stop_bits())) - 1u);
return mask != 0 ? static_cast<std::size_t>(count_trailing_zeros(mask)) / ascii_stop_bits() : n;
}
#else
static_cast<void>(end);
#endif
return find_ascii_copyable_run(p, n);
}
/// A stack that starts in a buffer of the caller (a local array) and moves to /// A stack that starts in a buffer of the caller (a local array) and moves to
/// the heap (a vector of the caller) only when that is full, so that dumps of /// the heap (a vector of the caller) only when that is full, so that dumps of
/// shallow documents need no allocation. The top is a pointer, as in /// shallow documents need no allocation. The top is a pointer, as in
@@ -275,6 +349,39 @@ NLOHMANN_VIEW_ALWAYS_INLINE char* put_literal(char* w, const char (&s)[N]) noexc
return w + N - 1; return w + N - 1;
} }
/// the depth of the innermost open container, counted only for indented output
template<bool Counted>
struct depth_counter
{
std::size_t value = 0;
void enter() noexcept
{
++value;
}
std::size_t leave() noexcept
{
return --value;
}
std::size_t get() const noexcept
{
return value;
}
};
template<>
struct depth_counter<false>
{
void enter() noexcept {}
std::size_t leave() noexcept
{
return 0;
}
std::size_t get() const noexcept
{
return 0;
}
};
/// how the view's dump() writes a value /// how the view's dump() writes a value
struct dump_style struct dump_style
{ {
@@ -311,139 +418,95 @@ class view_serializer
void dump(const node* root) void dump(const node* root)
{ {
if (!m_style.pretty) if (m_style.ensure_ascii)
{ {
if (m_style.ensure_ascii) dump_ascii(root);
{
dump_compact_ascii(root);
}
else if (m_style.source_numbers)
{
dump_compact<true, false>(root);
}
else
{
dump_compact<false, false>(root);
}
return;
} }
struct frame else if (m_style.pretty)
{ {
const node* pos; ///< next element, or key of the next member dump_pretty(root);
const node* end; }
bool object; else if (m_style.source_numbers)
bool first; ///< nothing written yet
};
std::array<frame, 32> buffer; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::vector<frame> heap;
small_stack<frame> stack(buffer.data(), buffer.size(), heap);
const node* n = root;
for (;;)
{ {
// write the value at n dump_text<true, false, false>(root);
if (is_container(*n)) }
{ else
const bool object = n->kind == static_cast<std::uint8_t>(value_t::object); {
if (n->len == 0) dump_text<false, false, false>(root);
{
m_out.put(object ? "{}" : "[]", 2);
}
else
{
m_out.put(object ? '{' : '[');
stack.push_back(frame{nav::first(m_doc, n), nav::end(m_doc, n), object, true});
}
}
else
{
write_scalar(*n);
}
// go to the next value: close finished containers, then separate
for (;;)
{
if (stack.empty())
{
m_out.finish();
return;
}
if (stack.back().pos == stack.back().end)
{
const bool object = stack.back().object;
stack.pop_back();
newline(stack.size());
m_out.put(object ? '}' : ']');
continue;
}
frame& f = stack.back();
if (!f.first)
{
m_out.put(',');
}
f.first = false;
newline(stack.size());
if (f.object)
{
write_string(*f.pos);
if (m_style.pretty)
{
m_out.put(": ");
}
else
{
m_out.put(':');
}
n = nav::value(f.pos + 1);
f.pos = document_data::after(f.pos + 1);
}
else
{
n = nav::value(f.pos);
f.pos = document_data::after(f.pos);
}
break;
}
} }
} }
private: private:
/// a string that is not a plain string of the source (decoded, or written /// a string that is not a plain string of the source (decoded, or written
/// by an edit), without ensure_ascii: runs without characters to escape /// by an edit), without ensure_ascii: runs without characters to escape
/// are copied /// are copied. If validate is set, the string must be valid UTF-8: the
NLOHMANN_VIEW_NOINLINE void write_decoded(const node& n) /// runs are found by the lexer's scan, which validates the sequences, and
/// invalid UTF-8 throws what basic_json::dump() throws. (The compact output
/// writes the bytes of a damaged image unchanged, and scans without
/// validating them.)
NLOHMANN_VIEW_NOINLINE void write_decoded(const node& n, bool validate)
{ {
const auto* const s = reinterpret_cast<const unsigned char*>(m_doc.str(n)); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast) const auto* const s = reinterpret_cast<const unsigned char*>(m_doc.str(n)); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
m_out.put('"'); m_out.put('"');
for (std::size_t i = 0; i < n.len;) for (std::size_t i = 0; i < n.len;)
{ {
const std::size_t run = plain_output_run(s + i, n.len - i); const std::size_t run = validate ? string_bulk_run(s + i, n.len - i) : plain_output_run(s + i, n.len - i);
if (run != 0) if (run != 0)
{ {
m_out.put(reinterpret_cast<const char*>(s + i), run); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast) m_out.put(reinterpret_cast<const char*>(s + i), run); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
i += run; i += run;
continue; continue;
} }
if (validate && s[i] >= 0x80)
{
invalid_utf8(s, n.len); // (the scan stopped at ill-formed UTF-8)
}
write_codepoint<false>(s[i], s + i, 1); // a quote, a backslash, or a control character write_codepoint<false>(s[i], s + i, 1); // a quote, a backslash, or a control character
++i; ++i;
} }
m_out.put('"'); m_out.put('"');
} }
/// dump_compact() with ensure_ascii, out of line: the walk without it is /// dump_text() with indentation, out of line: the compact walk is faster
/// faster when it is not merged with this one into dump() /// when it is not merged with the indented ones into dump()
NLOHMANN_VIEW_NOINLINE void dump_compact_ascii(const node* root) NLOHMANN_VIEW_NOINLINE void dump_pretty(const node* root)
{ {
if (m_style.source_numbers) if (m_style.source_numbers)
{ {
dump_compact<true, true>(root); dump_text<true, true, false>(root);
} }
else else
{ {
dump_compact<false, true>(root); dump_text<false, true, false>(root);
} }
} }
/// whether dump_compact() copies a string or key from the source: one of /// dump_text() with ensure_ascii, out of line: the walk without it is
/// faster when it is not merged with this one into dump()
NLOHMANN_VIEW_NOINLINE void dump_ascii(const node* root)
{
if (m_style.pretty)
{
if (m_style.source_numbers)
{
dump_text<true, true, true>(root);
}
else
{
dump_text<false, true, true>(root);
}
}
else if (m_style.source_numbers)
{
dump_text<true, false, true>(root);
}
else
{
dump_text<false, false, true>(root);
}
}
/// whether dump_text() copies a string or key from the source: one of
/// the source, and with ensure_ascii one without bytes >= 0x7F (an overload /// the source, and with ensure_ascii one without bytes >= 0x7F (an overload
/// each, so that the walk without ensure_ascii is compiled exactly as /// each, so that the walk without ensure_ascii is compiled exactly as
/// before: with GCC, a condition that only folds away later changed its /// before: with GCC, a condition that only folds away later changed its
@@ -458,13 +521,154 @@ class view_serializer
return (n.flags & node_flags::storage) == 0 && is_ascii_without_delete(src + n.off, n.len); return (n.flags & node_flags::storage) == 0 && is_ascii_without_delete(src + n.off, n.len);
} }
/// a string with ensure_ascii (dump_compact()), out of line /*!
NLOHMANN_VIEW_NOINLINE void write_ascii_string(const node& n) @brief a string with ensure_ascii: the characters that are not printable ASCII as \\uXXXX
(a character beyond U+FFFF as a surrogate pair); strings of a document
are valid UTF-8, except in a damaged image loaded with image_check::bounds,
for which this throws what basic_json::dump() throws for the string
*/
NLOHMANN_VIEW_NOINLINE void write_ascii(const node& n)
{ {
write_string(n); const auto* const s = reinterpret_cast<const unsigned char*>(m_doc.str(n)); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
const std::size_t size = n.len;
char* w = m_out.cursor();
char* lim = m_out.limit();
// room for k bytes
const auto room = [&](std::size_t k)
{
if (NLOHMANN_VIEW_UNLIKELY(static_cast<std::size_t>(lim - w) < k))
{
m_out.set_cursor(w);
m_out.reserve(k);
w = m_out.cursor();
lim = m_out.limit();
}
};
static constexpr const char* hex = "0123456789abcdef";
// \\uXXXX
const auto u_escape = [&](std::uint32_t u)
{
w[0] = '\\';
w[1] = 'u';
w[2] = hex[(u >> 12u) & 0xFu];
w[3] = hex[(u >> 8u) & 0xFu];
w[4] = hex[(u >> 4u) & 0xFu];
w[5] = hex[u & 0xFu];
w += 6;
};
room(1);
*w++ = '"';
std::size_t i = 0;
// (a string of the source starts with a run that has nothing to escape, 16 bytes per step)
const std::size_t prefix = (n.flags & node_flags::storage) == 0 ? ascii_copyable_run(s, size, reinterpret_cast<const unsigned char*>(m_doc.src + m_doc.size)) : 0; // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
if (prefix != 0)
{
room(prefix);
std::memcpy(w, s, prefix);
w += prefix;
i = prefix;
}
while (i < size)
{
// (a character that needs an escape sequence is not worth a call)
const std::size_t run = is_ascii_copyable(s[i]) ? find_ascii_copyable_run(s + i, size - i) : 0;
if (run != 0)
{
room(run);
std::memcpy(w, s + i, run);
w += run;
i += run;
continue;
}
room(12);
if (s[i] >= 0x80)
{
const unsigned char lead = s[i];
// (the two-byte sequences of Latin text and the three-byte sequences of most other
// text, with continuation bytes of any value: the ranges that are restricted (E0,
// ED, and the sequences of four bytes) go through validate_one_utf8())
if (lead >= 0xE1 && lead <= 0xEF && lead != 0xED && size - i >= 3 && (s[i + 1] & 0xC0u) == 0x80 && (s[i + 2] & 0xC0u) == 0x80)
{
u_escape(((lead & 0x0Fu) << 12u) | ((s[i + 1] & 0x3Fu) << 6u) | (s[i + 2] & 0x3Fu));
i += 3;
continue;
}
if (lead >= 0xC2 && lead <= 0xDF && size - i >= 2 && (s[i + 1] & 0xC0u) == 0x80)
{
u_escape(((lead & 0x1Fu) << 6u) | (s[i + 1] & 0x3Fu));
i += 2;
continue;
}
const std::size_t len = validate_one_utf8(s + i, size - i);
if (NLOHMANN_VIEW_UNLIKELY(len == 0))
{
m_out.set_cursor(w);
invalid_utf8(s, size);
return; // LCOV_EXCL_LINE (invalid_utf8 throws)
}
std::uint32_t codepoint = s[i] & (0xFFu >> (len + 1));
for (std::size_t k = 1; k < len; ++k)
{
codepoint = (codepoint << 6u) | (s[i + k] & 0x3Fu);
}
i += len;
if (codepoint <= 0xFFFF)
{
u_escape(codepoint);
}
else
{
u_escape(0xD7C0u + (codepoint >> 10u));
u_escape(0xDC00u + (codepoint & 0x3FFu));
}
continue;
}
// a quote, a backslash, or a control character (or DEL)
const unsigned char c = s[i++];
char escape = 0;
switch (c)
{
case 0x08:
escape = 'b';
break;
case 0x09:
escape = 't';
break;
case 0x0A:
escape = 'n';
break;
case 0x0C:
escape = 'f';
break;
case 0x0D:
escape = 'r';
break;
case 0x22:
escape = '"';
break;
case 0x5C:
escape = '\\';
break;
default:
break;
}
if (escape != 0)
{
w[0] = '\\';
w[1] = escape;
w += 2;
}
else
{
u_escape(c);
}
}
room(1);
*w++ = '"';
m_out.set_cursor(w);
} }
/// the copies of dump_compact() that are not fixed-size moves (long /// the copies of dump_text() that are not fixed-size moves (long
/// strings, or near the end of the source); out of line, so that the /// strings, or near the end of the source); out of line, so that the
/// compiler does not merge the fixed-size moves into this call /// compiler does not merge the fixed-size moves into this call
NLOHMANN_VIEW_NOINLINE static void copy_long(char* to, const char* from, std::size_t n) noexcept NLOHMANN_VIEW_NOINLINE static void copy_long(char* to, const char* from, std::size_t n) noexcept
@@ -481,7 +685,7 @@ class view_serializer
} }
/*! /*!
@brief the compact output (the default dump(), or with ensure_ascii) @brief the output, compact (the default dump()) or indented, with or without ensure_ascii
The same walk as dump(), with the write position in a local variable The same walk as dump(), with the write position in a local variable
(stores through char pointers would otherwise force a reload of the (stores through char pointers would otherwise force a reload of the
@@ -489,11 +693,11 @@ class view_serializer
the source copied by fixed-size moves of 32 bytes where the source has the source copied by fixed-size moves of 32 bytes where the source has
that many bytes left, instead of a library call per token. The buffer that many bytes left, instead of a library call per token. The buffer
keeps 64 bytes of slack for the overshoot. With ensure_ascii (Ascii), a keeps 64 bytes of slack for the overshoot. With ensure_ascii (Ascii), a
string of the source is copied only if it has no byte >= 0x7F (see string of the source is copied only if it has nothing to escape (see
is_ascii_without_delete()); any other string is written by write_string(). copied()); any other string is written by write_ascii().
*/ */
template<bool SourceNumbers, bool Ascii> template<bool SourceNumbers, bool Pretty, bool Ascii>
void dump_compact(const node* root) void dump_text(const node* root)
{ {
struct frame struct frame
{ {
@@ -508,6 +712,16 @@ class view_serializer
const char* const src_end = src + m_doc.size; const char* const src_end = src + m_doc.size;
char* w = m_out.cursor(); char* w = m_out.cursor();
char* lim = m_out.limit(); char* lim = m_out.limit();
// (indented output) a line break and 63 indentation characters, from
// which the line break of a level of up to 63 characters is a copy
std::array<char, 64> ruler; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): filled below if used
if (Pretty)
{
ruler.fill(m_style.indent_char);
ruler[0] = '\n';
}
const std::size_t indent = m_style.indent;
depth_counter<Pretty> level; // depth of the innermost open container (indented output)
// room for n bytes and the slack // room for n bytes and the slack
const auto room = [&](std::size_t n) const auto room = [&](std::size_t n)
{ {
@@ -539,6 +753,22 @@ class view_serializer
} }
w += n; w += n;
}; };
// (indented output) the line break and the indentation of a level
const auto newline = [&](std::size_t depth)
{
const std::size_t count = 1 + (depth * indent);
room(count);
if (count <= ruler.size())
{
std::memcpy(w, ruler.data(), ruler.size());
}
else
{
*w = '\n';
std::memset(w + 1, m_style.indent_char, count - 1);
}
w += count;
};
// a string that is not a plain string of the source (out of line, so // a string that is not a plain string of the source (out of line, so
// that the cursor stays in a register here) // that the cursor stays in a register here)
const auto escaped = [&](const node & n) const auto escaped = [&](const node & n)
@@ -546,11 +776,11 @@ class view_serializer
m_out.set_cursor(w); m_out.set_cursor(w);
if (enabled(Ascii)) if (enabled(Ascii))
{ {
write_ascii_string(n); write_ascii(n);
} }
else else
{ {
write_decoded(n); write_decoded(n, Pretty);
} }
w = m_out.cursor(); w = m_out.cursor();
lim = m_out.limit(); lim = m_out.limit();
@@ -651,6 +881,7 @@ class view_serializer
{ {
*w++ = object ? '{' : '['; *w++ = object ? '{' : '[';
stack.push_back(cur); stack.push_back(cur);
level.enter();
if (Editable) if (Editable)
{ {
cur = frame{nav::first(m_doc, n), nav::end(m_doc, n), object}; cur = frame{nav::first(m_doc, n), nav::end(m_doc, n), object};
@@ -693,6 +924,10 @@ class view_serializer
{ {
break; break;
} }
if (Pretty)
{
newline(level.leave());
}
room(1); room(1);
*w++ = cur.object ? '}' : ']'; *w++ = cur.object ? '}' : ']';
cur = stack.back(); cur = stack.back();
@@ -701,13 +936,17 @@ class view_serializer
room(1); room(1);
*w++ = ','; *w++ = ',';
} }
if (Pretty)
{
newline(level.get());
}
const node* const at = Editable ? cur.pos : n; const node* const at = Editable ? cur.pos : n;
if (cur.object) if (cur.object)
{ {
const node& key = *at; const node& key = *at;
if (copied(key, src, std::integral_constant<bool, Ascii>())) if (copied(key, src, std::integral_constant<bool, Ascii>()))
{ {
room(key.len + 3); room(key.len + (Pretty ? 4 : 3));
*w++ = '"'; *w++ = '"';
copy(src + key.off, key.len); copy(src + key.off, key.len);
w[0] = '"'; w[0] = '"';
@@ -717,9 +956,13 @@ class view_serializer
else else
{ {
escaped(key); escaped(key);
room(1); room(Pretty ? 2 : 1);
*w++ = ':'; *w++ = ':';
} }
if (Pretty)
{
*w++ = ' ';
}
if (Editable) if (Editable)
{ {
n = nav::value(at + 1); n = nav::value(at + 1);
@@ -738,69 +981,6 @@ class view_serializer
} }
} }
void newline(std::size_t level)
{
if (m_style.pretty)
{
m_out.put('\n');
m_out.put_repeated(m_style.indent_char, level * m_style.indent);
}
}
void write_scalar(const node& n)
{
switch (static_cast<value_t>(n.kind))
{
case value_t::null:
m_out.put("null");
break;
case value_t::boolean:
if ((n.flags & node_flags::is_true) != 0)
{
m_out.put("true");
}
else
{
m_out.put("false");
}
break;
case value_t::string:
write_string(n);
break;
case value_t::number_integer:
case value_t::number_unsigned:
{
const char* const token = m_doc.str(n);
const std::uint32_t len = number_length(n);
if (!m_style.source_numbers && len == 2 && token[0] == '-' && token[1] == '0')
{
m_out.put('0'); // parse() reads -0 as the integer 0
}
else
{
m_out.put(token, len);
}
break;
}
case value_t::number_float:
if (m_style.source_numbers && (n.flags & node_flags::storage) != node_flags::edited)
{
m_out.put(m_doc.str(n), n.len); // (a float set by an edit is written as with shortest)
}
else
{
write_float_node(n);
}
break;
case value_t::object: // LCOV_EXCL_LINE (containers are written by dump())
case value_t::array: // LCOV_EXCL_LINE
case value_t::binary: // LCOV_EXCL_LINE (not in a document)
case value_t::discarded: // LCOV_EXCL_LINE
default: // LCOV_EXCL_LINE
break; // LCOV_EXCL_LINE
}
}
/// a float node as dump() writes it /// a float node as dump() writes it
void write_float_node(const node& n) void write_float_node(const node& n)
{ {
@@ -911,72 +1091,6 @@ class view_serializer
m_out.put(s.data(), s.size()); m_out.put(s.data(), s.size());
} }
void write_string(const node& n)
{
const char* const s = m_doc.str(n);
m_out.put('"');
if ((n.flags & node_flags::storage) == 0 && (!m_style.ensure_ascii || is_ascii_without_delete(s, n.len)))
{
// a string of the source without escape sequences has nothing to
// escape (with ensure_ascii, if it is printable ASCII)
m_out.put(s, n.len);
}
else if (m_style.ensure_ascii)
{
write_escaped<true>(reinterpret_cast<const unsigned char*>(s), n.len); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
}
else
{
write_escaped<false>(reinterpret_cast<const unsigned char*>(s), n.len); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
}
m_out.put('"');
}
/// as serializer::dump_escaped(); strings of a document are valid UTF-8,
/// except in a damaged image loaded with image_check::bounds, for which
/// this throws what basic_json::dump() throws for the string
template<bool EnsureAscii>
void write_escaped(const unsigned char* s, std::size_t n)
{
std::size_t i = 0;
while (i < n)
{
std::size_t run = 0;
if (!EnsureAscii)
{
run = string_bulk_run(s + i, n - i);
}
else if (is_ascii_copyable(s[i]))
{
run = find_ascii_copyable_run(s + i, n - i);
}
if (run != 0)
{
m_out.put(reinterpret_cast<const char*>(s + i), run); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
i += run;
continue;
}
std::uint32_t codepoint = s[i];
std::size_t len = 1;
if (codepoint >= 0x80)
{
len = validate_one_utf8(s + i, n - i);
if (NLOHMANN_VIEW_UNLIKELY(len == 0))
{
invalid_utf8(s, n);
return;
}
codepoint &= 0xFFu >> (len + 1);
for (std::size_t k = 1; k < len; ++k)
{
codepoint = (codepoint << 6u) | (s[i + k] & 0x3Fu);
}
}
write_codepoint<EnsureAscii>(codepoint, s + i, len);
i += len;
}
}
/// throw what basic_json::dump() throws for a string that is not valid UTF-8 /// throw what basic_json::dump() throws for a string that is not valid UTF-8
NLOHMANN_VIEW_NOINLINE static void invalid_utf8(const unsigned char* s, std::size_t n) NLOHMANN_VIEW_NOINLINE static void invalid_utf8(const unsigned char* s, std::size_t n)
{ {
+18 -42
View File
@@ -6321,7 +6321,7 @@ public:
const error_handler_t error_handler = detail::binary_writer_default_error_handler()) const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{ {
std::vector<std::uint8_t> result; std::vector<std::uint8_t> result;
result.reserve(detail::binary_initial_capacity(j)); result.reserve(detail::binary_reserve_hint(j));
vector_writer(result, error_handler).write_cbor(j); vector_writer(result, error_handler).write_cbor(j);
return result; return result;
} }
@@ -6331,9 +6331,7 @@ public:
static void to_cbor(const basic_json& j, detail::output_adapter<std::uint8_t> o, static void to_cbor(const basic_json& j, detail::output_adapter<std::uint8_t> o,
const error_handler_t error_handler = detail::binary_writer_default_error_handler()) const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{ {
binary_writer<std::uint8_t> writer(o, error_handler); binary_writer<std::uint8_t>(o, error_handler).write_cbor(j);
writer.write_cbor(j);
writer.flush();
} }
/// @brief create a CBOR serialization of a given JSON value /// @brief create a CBOR serialization of a given JSON value
@@ -6341,9 +6339,7 @@ public:
static void to_cbor(const basic_json& j, detail::output_adapter<char> o, static void to_cbor(const basic_json& j, detail::output_adapter<char> o,
const error_handler_t error_handler = detail::binary_writer_default_error_handler()) const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{ {
binary_writer<char> writer(o, error_handler); binary_writer<char>(o, error_handler).write_cbor(j);
writer.write_cbor(j);
writer.flush();
} }
/// @brief create a MessagePack serialization of a given JSON value /// @brief create a MessagePack serialization of a given JSON value
@@ -6352,7 +6348,7 @@ public:
const error_handler_t error_handler = error_handler_t::keep) const error_handler_t error_handler = error_handler_t::keep)
{ {
std::vector<std::uint8_t> result; std::vector<std::uint8_t> result;
result.reserve(detail::binary_initial_capacity(j)); result.reserve(detail::binary_reserve_hint(j));
vector_writer(result, error_handler).write_msgpack(j); vector_writer(result, error_handler).write_msgpack(j);
return result; return result;
} }
@@ -6362,9 +6358,7 @@ public:
static void to_msgpack(const basic_json& j, detail::output_adapter<std::uint8_t> o, static void to_msgpack(const basic_json& j, detail::output_adapter<std::uint8_t> o,
const error_handler_t error_handler = error_handler_t::keep) const error_handler_t error_handler = error_handler_t::keep)
{ {
binary_writer<std::uint8_t> writer(o, error_handler); binary_writer<std::uint8_t>(o, error_handler).write_msgpack(j);
writer.write_msgpack(j);
writer.flush();
} }
/// @brief create a MessagePack serialization of a given JSON value /// @brief create a MessagePack serialization of a given JSON value
@@ -6372,9 +6366,7 @@ public:
static void to_msgpack(const basic_json& j, detail::output_adapter<char> o, static void to_msgpack(const basic_json& j, detail::output_adapter<char> o,
const error_handler_t error_handler = error_handler_t::keep) const error_handler_t error_handler = error_handler_t::keep)
{ {
binary_writer<char> writer(o, error_handler); binary_writer<char>(o, error_handler).write_msgpack(j);
writer.write_msgpack(j);
writer.flush();
} }
/// @brief create a UBJSON serialization of a given JSON value /// @brief create a UBJSON serialization of a given JSON value
@@ -6385,7 +6377,7 @@ public:
const error_handler_t error_handler = detail::binary_writer_default_error_handler()) const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{ {
std::vector<std::uint8_t> result; std::vector<std::uint8_t> result;
result.reserve(detail::binary_initial_capacity(j)); result.reserve(detail::binary_reserve_hint(j));
vector_writer(result, error_handler).write_ubjson(j, use_size, use_type); vector_writer(result, error_handler).write_ubjson(j, use_size, use_type);
return result; return result;
} }
@@ -6396,9 +6388,7 @@ public:
const bool use_size = false, const bool use_type = false, const bool use_size = false, const bool use_type = false,
const error_handler_t error_handler = detail::binary_writer_default_error_handler()) const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{ {
binary_writer<std::uint8_t> writer(o, error_handler); binary_writer<std::uint8_t>(o, error_handler).write_ubjson(j, use_size, use_type);
writer.write_ubjson(j, use_size, use_type);
writer.flush();
} }
/// @brief create a UBJSON serialization of a given JSON value /// @brief create a UBJSON serialization of a given JSON value
@@ -6407,9 +6397,7 @@ public:
const bool use_size = false, const bool use_type = false, const bool use_size = false, const bool use_type = false,
const error_handler_t error_handler = detail::binary_writer_default_error_handler()) const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{ {
binary_writer<char> writer(o, error_handler); binary_writer<char>(o, error_handler).write_ubjson(j, use_size, use_type);
writer.write_ubjson(j, use_size, use_type);
writer.flush();
} }
/// @brief create a BJData serialization of a given JSON value /// @brief create a BJData serialization of a given JSON value
@@ -6421,7 +6409,7 @@ public:
const error_handler_t error_handler = detail::binary_writer_default_error_handler()) const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{ {
std::vector<std::uint8_t> result; std::vector<std::uint8_t> result;
result.reserve(detail::binary_initial_capacity(j)); result.reserve(detail::binary_reserve_hint(j));
vector_writer(result, error_handler).write_ubjson(j, use_size, use_type, true, true, version); vector_writer(result, error_handler).write_ubjson(j, use_size, use_type, true, true, version);
return result; return result;
} }
@@ -6433,9 +6421,7 @@ public:
const bjdata_version_t version = bjdata_version_t::draft2, const bjdata_version_t version = bjdata_version_t::draft2,
const error_handler_t error_handler = detail::binary_writer_default_error_handler()) const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{ {
binary_writer<std::uint8_t> writer(o, error_handler); binary_writer<std::uint8_t>(o, error_handler).write_ubjson(j, use_size, use_type, true, true, version);
writer.write_ubjson(j, use_size, use_type, true, true, version);
writer.flush();
} }
/// @brief create a BJData serialization of a given JSON value /// @brief create a BJData serialization of a given JSON value
@@ -6445,9 +6431,7 @@ public:
const bjdata_version_t version = bjdata_version_t::draft2, const bjdata_version_t version = bjdata_version_t::draft2,
const error_handler_t error_handler = detail::binary_writer_default_error_handler()) const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{ {
binary_writer<char> writer(o, error_handler); binary_writer<char>(o, error_handler).write_ubjson(j, use_size, use_type, true, true, version);
writer.write_ubjson(j, use_size, use_type, true, true, version);
writer.flush();
} }
/// @brief create a BSON serialization of a given JSON value /// @brief create a BSON serialization of a given JSON value
@@ -6456,7 +6440,7 @@ public:
const error_handler_t error_handler = detail::binary_writer_default_error_handler()) const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{ {
std::vector<std::uint8_t> result; std::vector<std::uint8_t> result;
result.reserve(detail::binary_initial_capacity(j)); result.reserve(detail::binary_reserve_hint(j));
vector_writer(result, error_handler).write_bson(j); vector_writer(result, error_handler).write_bson(j);
return result; return result;
} }
@@ -6466,9 +6450,7 @@ public:
static void to_bson(const basic_json& j, detail::output_adapter<std::uint8_t> o, static void to_bson(const basic_json& j, detail::output_adapter<std::uint8_t> o,
const error_handler_t error_handler = detail::binary_writer_default_error_handler()) const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{ {
binary_writer<std::uint8_t> writer(o, error_handler); binary_writer<std::uint8_t>(o, error_handler).write_bson(j);
writer.write_bson(j);
writer.flush();
} }
/// @brief create a BSON serialization of a given JSON value /// @brief create a BSON serialization of a given JSON value
@@ -6476,9 +6458,7 @@ public:
static void to_bson(const basic_json& j, detail::output_adapter<char> o, static void to_bson(const basic_json& j, detail::output_adapter<char> o,
const error_handler_t error_handler = detail::binary_writer_default_error_handler()) const error_handler_t error_handler = detail::binary_writer_default_error_handler())
{ {
binary_writer<char> writer(o, error_handler); binary_writer<char>(o, error_handler).write_bson(j);
writer.write_bson(j);
writer.flush();
} }
/// @brief create a BON8 serialization of a given JSON value /// @brief create a BON8 serialization of a given JSON value
@@ -6486,7 +6466,7 @@ public:
static std::vector<std::uint8_t> to_bon8(const basic_json& j) static std::vector<std::uint8_t> to_bon8(const basic_json& j)
{ {
std::vector<std::uint8_t> result; std::vector<std::uint8_t> result;
result.reserve(detail::binary_initial_capacity(j)); result.reserve(detail::binary_reserve_hint(j));
vector_writer(result).write_bon8(j); vector_writer(result).write_bon8(j);
return result; return result;
} }
@@ -6495,18 +6475,14 @@ public:
/// @sa https://json.nlohmann.me/api/basic_json/to_bon8/ /// @sa https://json.nlohmann.me/api/basic_json/to_bon8/
static void to_bon8(const basic_json& j, detail::output_adapter<std::uint8_t> o) static void to_bon8(const basic_json& j, detail::output_adapter<std::uint8_t> o)
{ {
binary_writer<std::uint8_t> writer(o); binary_writer<std::uint8_t>(o).write_bon8(j);
writer.write_bon8(j);
writer.flush();
} }
/// @brief create a BON8 serialization of a given JSON value /// @brief create a BON8 serialization of a given JSON value
/// @sa https://json.nlohmann.me/api/basic_json/to_bon8/ /// @sa https://json.nlohmann.me/api/basic_json/to_bon8/
static void to_bon8(const basic_json& j, detail::output_adapter<char> o) static void to_bon8(const basic_json& j, detail::output_adapter<char> o)
{ {
binary_writer<char> writer(o); binary_writer<char>(o).write_bon8(j);
writer.write_bon8(j);
writer.flush();
} }
/// @brief create a JSON value from an input in CBOR format /// @brief create a JSON value from an input in CBOR format
+218 -1973
View File
File diff suppressed because it is too large. Load diff
+362 -247
View File
@@ -6089,6 +6089,8 @@ NLOHMANN_JSON_NAMESPACE_END
// #include <nlohmann/detail/view/number.hpp> // #include <nlohmann/detail/view/number.hpp>
// #include <nlohmann/detail/view/simd.hpp>
NLOHMANN_JSON_NAMESPACE_BEGIN NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail namespace detail
@@ -6146,13 +6148,6 @@ class output_buffer
put(s, N - 1); put(s, N - 1);
} }
void put_repeated(char c, std::size_t n)
{
reserve(n);
std::memset(m_pos, c, n);
m_pos += n;
}
/// the write position and the end of the writable space, for a writer /// the write position and the end of the writable space, for a writer
/// that keeps the position in a local variable (set_cursor() hands it back) /// that keeps the position in a local variable (set_cursor() hands it back)
char* cursor() const noexcept char* cursor() const noexcept
@@ -6209,9 +6204,25 @@ class output_buffer
/// image_check::bounds can have others, which are then written unchanged). /// image_check::bounds can have others, which are then written unchanged).
inline std::size_t plain_output_run(const unsigned char* s, std::size_t n) noexcept inline std::size_t plain_output_run(const unsigned char* s, std::size_t n) noexcept
{ {
std::size_t i = 0;
#if NLOHMANN_VIEW_SSE2
// (16 bytes per step; with NEON, the mask of a step costs more than it saves on the
// short runs that are typical here)
for (; i + 16 <= n; i += 16)
{
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(s + i))); // NOLINT(bugprone-casting-through-void)
// (a byte is below 0x20 if it is its own minimum with 0x1F)
const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_cmpeq_epi8(_mm_min_epu8(in, _mm_set1_epi8(0x1F)), in));
const auto mask = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
if (mask != 0)
{
return i + static_cast<std::size_t>(count_trailing_zeros(mask));
}
}
#endif
constexpr std::uint64_t ones = 0x0101010101010101ull; constexpr std::uint64_t ones = 0x0101010101010101ull;
constexpr std::uint64_t high = 0x8080808080808080ull; constexpr std::uint64_t high = 0x8080808080808080ull;
std::size_t i = 0;
for (; i + 8 <= n; i += 8) for (; i + 8 <= n; i += 8)
{ {
const std::uint64_t v = read_eight_bytes(s + i); const std::uint64_t v = read_eight_bytes(s + i);
@@ -6262,6 +6273,70 @@ inline bool is_ascii_without_delete(const char* s, std::size_t n) noexcept
return true; return true;
} }
#if NLOHMANN_VIEW_VECTOR
/// the bytes of the 16 at p that ensure_ascii cannot copy (a quote, a
/// backslash, a control character, DEL, or not ASCII; a signed compare with
/// 0x20 finds control characters and non-ASCII bytes at once), one bit per
/// byte (x86) or one nibble per byte (NEON)
NLOHMANN_VIEW_ALWAYS_INLINE std::uint64_t ascii_stop_mask(const unsigned char* p) noexcept
{
#if NLOHMANN_VIEW_NEON
const uint8x16_t in = vld1q_u8(p);
const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))),
vorrq_u8(vceqq_u8(in, vdupq_n_u8(0x7F)), vcltq_s8(vreinterpretq_s8_u8(in), vdupq_n_s8(0x20))));
return vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0);
#else
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8(0x7F)), _mm_cmplt_epi8(in, _mm_set1_epi8(0x20))));
return static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
#endif
}
/// the width in bits of a byte in the mask of ascii_stop_mask()
constexpr unsigned ascii_stop_bits() noexcept
{
return NLOHMANN_VIEW_NEON ? 4u : 1u;
}
#endif
/// As find_ascii_copyable_run(), 16 bytes per step. end is the end of the
/// buffer that holds the n bytes at p (a short run is read as 16 bytes if
/// they are in the buffer).
inline std::size_t ascii_copyable_run(const unsigned char* p, std::size_t n, const unsigned char* end) noexcept
{
#if NLOHMANN_VIEW_VECTOR
if (n >= 16)
{
std::size_t i = 0;
for (; i + 16 <= n; i += 16)
{
const std::uint64_t mask = ascii_stop_mask(p + i);
if (mask != 0)
{
return i + (static_cast<std::size_t>(count_trailing_zeros(mask)) / ascii_stop_bits());
}
}
if (i == n)
{
return n;
}
// the last 16 bytes: those before i are known to be copyable
const std::uint64_t mask = ascii_stop_mask(p + n - 16);
return mask != 0 ? n - 16 + (static_cast<std::size_t>(count_trailing_zeros(mask)) / ascii_stop_bits()) : n;
}
if (end - p >= 16)
{
// (the bytes after the run are masked out)
const std::uint64_t mask = ascii_stop_mask(p) & ((std::uint64_t{1} << (n * ascii_stop_bits())) - 1u);
return mask != 0 ? static_cast<std::size_t>(count_trailing_zeros(mask)) / ascii_stop_bits() : n;
}
#else
static_cast<void>(end);
#endif
return find_ascii_copyable_run(p, n);
}
/// A stack that starts in a buffer of the caller (a local array) and moves to /// A stack that starts in a buffer of the caller (a local array) and moves to
/// the heap (a vector of the caller) only when that is full, so that dumps of /// the heap (a vector of the caller) only when that is full, so that dumps of
/// shallow documents need no allocation. The top is a pointer, as in /// shallow documents need no allocation. The top is a pointer, as in
@@ -6341,6 +6416,39 @@ NLOHMANN_VIEW_ALWAYS_INLINE char* put_literal(char* w, const char (&s)[N]) noexc
return w + N - 1; return w + N - 1;
} }
/// the depth of the innermost open container, counted only for indented output
template<bool Counted>
struct depth_counter
{
std::size_t value = 0;
void enter() noexcept
{
++value;
}
std::size_t leave() noexcept
{
return --value;
}
std::size_t get() const noexcept
{
return value;
}
};
template<>
struct depth_counter<false>
{
void enter() noexcept {}
std::size_t leave() noexcept
{
return 0;
}
std::size_t get() const noexcept
{
return 0;
}
};
/// how the view's dump() writes a value /// how the view's dump() writes a value
struct dump_style struct dump_style
{ {
@@ -6377,139 +6485,95 @@ class view_serializer
void dump(const node* root) void dump(const node* root)
{ {
if (!m_style.pretty) if (m_style.ensure_ascii)
{ {
if (m_style.ensure_ascii) dump_ascii(root);
{
dump_compact_ascii(root);
}
else if (m_style.source_numbers)
{
dump_compact<true, false>(root);
}
else
{
dump_compact<false, false>(root);
}
return;
} }
struct frame else if (m_style.pretty)
{ {
const node* pos; ///< next element, or key of the next member dump_pretty(root);
const node* end; }
bool object; else if (m_style.source_numbers)
bool first; ///< nothing written yet
};
std::array<frame, 32> buffer; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::vector<frame> heap;
small_stack<frame> stack(buffer.data(), buffer.size(), heap);
const node* n = root;
for (;;)
{ {
// write the value at n dump_text<true, false, false>(root);
if (is_container(*n)) }
{ else
const bool object = n->kind == static_cast<std::uint8_t>(value_t::object); {
if (n->len == 0) dump_text<false, false, false>(root);
{
m_out.put(object ? "{}" : "[]", 2);
}
else
{
m_out.put(object ? '{' : '[');
stack.push_back(frame{nav::first(m_doc, n), nav::end(m_doc, n), object, true});
}
}
else
{
write_scalar(*n);
}
// go to the next value: close finished containers, then separate
for (;;)
{
if (stack.empty())
{
m_out.finish();
return;
}
if (stack.back().pos == stack.back().end)
{
const bool object = stack.back().object;
stack.pop_back();
newline(stack.size());
m_out.put(object ? '}' : ']');
continue;
}
frame& f = stack.back();
if (!f.first)
{
m_out.put(',');
}
f.first = false;
newline(stack.size());
if (f.object)
{
write_string(*f.pos);
if (m_style.pretty)
{
m_out.put(": ");
}
else
{
m_out.put(':');
}
n = nav::value(f.pos + 1);
f.pos = document_data::after(f.pos + 1);
}
else
{
n = nav::value(f.pos);
f.pos = document_data::after(f.pos);
}
break;
}
} }
} }
private: private:
/// a string that is not a plain string of the source (decoded, or written /// a string that is not a plain string of the source (decoded, or written
/// by an edit), without ensure_ascii: runs without characters to escape /// by an edit), without ensure_ascii: runs without characters to escape
/// are copied /// are copied. If validate is set, the string must be valid UTF-8: the
NLOHMANN_VIEW_NOINLINE void write_decoded(const node& n) /// runs are found by the lexer's scan, which validates the sequences, and
/// invalid UTF-8 throws what basic_json::dump() throws. (The compact output
/// writes the bytes of a damaged image unchanged, and scans without
/// validating them.)
NLOHMANN_VIEW_NOINLINE void write_decoded(const node& n, bool validate)
{ {
const auto* const s = reinterpret_cast<const unsigned char*>(m_doc.str(n)); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast) const auto* const s = reinterpret_cast<const unsigned char*>(m_doc.str(n)); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
m_out.put('"'); m_out.put('"');
for (std::size_t i = 0; i < n.len;) for (std::size_t i = 0; i < n.len;)
{ {
const std::size_t run = plain_output_run(s + i, n.len - i); const std::size_t run = validate ? string_bulk_run(s + i, n.len - i) : plain_output_run(s + i, n.len - i);
if (run != 0) if (run != 0)
{ {
m_out.put(reinterpret_cast<const char*>(s + i), run); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast) m_out.put(reinterpret_cast<const char*>(s + i), run); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
i += run; i += run;
continue; continue;
} }
if (validate && s[i] >= 0x80)
{
invalid_utf8(s, n.len); // (the scan stopped at ill-formed UTF-8)
}
write_codepoint<false>(s[i], s + i, 1); // a quote, a backslash, or a control character write_codepoint<false>(s[i], s + i, 1); // a quote, a backslash, or a control character
++i; ++i;
} }
m_out.put('"'); m_out.put('"');
} }
/// dump_compact() with ensure_ascii, out of line: the walk without it is /// dump_text() with indentation, out of line: the compact walk is faster
/// faster when it is not merged with this one into dump() /// when it is not merged with the indented ones into dump()
NLOHMANN_VIEW_NOINLINE void dump_compact_ascii(const node* root) NLOHMANN_VIEW_NOINLINE void dump_pretty(const node* root)
{ {
if (m_style.source_numbers) if (m_style.source_numbers)
{ {
dump_compact<true, true>(root); dump_text<true, true, false>(root);
} }
else else
{ {
dump_compact<false, true>(root); dump_text<false, true, false>(root);
} }
} }
/// whether dump_compact() copies a string or key from the source: one of /// dump_text() with ensure_ascii, out of line: the walk without it is
/// faster when it is not merged with this one into dump()
NLOHMANN_VIEW_NOINLINE void dump_ascii(const node* root)
{
if (m_style.pretty)
{
if (m_style.source_numbers)
{
dump_text<true, true, true>(root);
}
else
{
dump_text<false, true, true>(root);
}
}
else if (m_style.source_numbers)
{
dump_text<true, false, true>(root);
}
else
{
dump_text<false, false, true>(root);
}
}
/// whether dump_text() copies a string or key from the source: one of
/// the source, and with ensure_ascii one without bytes >= 0x7F (an overload /// the source, and with ensure_ascii one without bytes >= 0x7F (an overload
/// each, so that the walk without ensure_ascii is compiled exactly as /// each, so that the walk without ensure_ascii is compiled exactly as
/// before: with GCC, a condition that only folds away later changed its /// before: with GCC, a condition that only folds away later changed its
@@ -6524,13 +6588,154 @@ class view_serializer
return (n.flags & node_flags::storage) == 0 && is_ascii_without_delete(src + n.off, n.len); return (n.flags & node_flags::storage) == 0 && is_ascii_without_delete(src + n.off, n.len);
} }
/// a string with ensure_ascii (dump_compact()), out of line /*!
NLOHMANN_VIEW_NOINLINE void write_ascii_string(const node& n) @brief a string with ensure_ascii: the characters that are not printable ASCII as \\uXXXX
(a character beyond U+FFFF as a surrogate pair); strings of a document
are valid UTF-8, except in a damaged image loaded with image_check::bounds,
for which this throws what basic_json::dump() throws for the string
*/
NLOHMANN_VIEW_NOINLINE void write_ascii(const node& n)
{ {
write_string(n); const auto* const s = reinterpret_cast<const unsigned char*>(m_doc.str(n)); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
const std::size_t size = n.len;
char* w = m_out.cursor();
char* lim = m_out.limit();
// room for k bytes
const auto room = [&](std::size_t k)
{
if (NLOHMANN_VIEW_UNLIKELY(static_cast<std::size_t>(lim - w) < k))
{
m_out.set_cursor(w);
m_out.reserve(k);
w = m_out.cursor();
lim = m_out.limit();
}
};
static constexpr const char* hex = "0123456789abcdef";
// \\uXXXX
const auto u_escape = [&](std::uint32_t u)
{
w[0] = '\\';
w[1] = 'u';
w[2] = hex[(u >> 12u) & 0xFu];
w[3] = hex[(u >> 8u) & 0xFu];
w[4] = hex[(u >> 4u) & 0xFu];
w[5] = hex[u & 0xFu];
w += 6;
};
room(1);
*w++ = '"';
std::size_t i = 0;
// (a string of the source starts with a run that has nothing to escape, 16 bytes per step)
const std::size_t prefix = (n.flags & node_flags::storage) == 0 ? ascii_copyable_run(s, size, reinterpret_cast<const unsigned char*>(m_doc.src + m_doc.size)) : 0; // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
if (prefix != 0)
{
room(prefix);
std::memcpy(w, s, prefix);
w += prefix;
i = prefix;
}
while (i < size)
{
// (a character that needs an escape sequence is not worth a call)
const std::size_t run = is_ascii_copyable(s[i]) ? find_ascii_copyable_run(s + i, size - i) : 0;
if (run != 0)
{
room(run);
std::memcpy(w, s + i, run);
w += run;
i += run;
continue;
}
room(12);
if (s[i] >= 0x80)
{
const unsigned char lead = s[i];
// (the two-byte sequences of Latin text and the three-byte sequences of most other
// text, with continuation bytes of any value: the ranges that are restricted (E0,
// ED, and the sequences of four bytes) go through validate_one_utf8())
if (lead >= 0xE1 && lead <= 0xEF && lead != 0xED && size - i >= 3 && (s[i + 1] & 0xC0u) == 0x80 && (s[i + 2] & 0xC0u) == 0x80)
{
u_escape(((lead & 0x0Fu) << 12u) | ((s[i + 1] & 0x3Fu) << 6u) | (s[i + 2] & 0x3Fu));
i += 3;
continue;
}
if (lead >= 0xC2 && lead <= 0xDF && size - i >= 2 && (s[i + 1] & 0xC0u) == 0x80)
{
u_escape(((lead & 0x1Fu) << 6u) | (s[i + 1] & 0x3Fu));
i += 2;
continue;
}
const std::size_t len = validate_one_utf8(s + i, size - i);
if (NLOHMANN_VIEW_UNLIKELY(len == 0))
{
m_out.set_cursor(w);
invalid_utf8(s, size);
return; // LCOV_EXCL_LINE (invalid_utf8 throws)
}
std::uint32_t codepoint = s[i] & (0xFFu >> (len + 1));
for (std::size_t k = 1; k < len; ++k)
{
codepoint = (codepoint << 6u) | (s[i + k] & 0x3Fu);
}
i += len;
if (codepoint <= 0xFFFF)
{
u_escape(codepoint);
}
else
{
u_escape(0xD7C0u + (codepoint >> 10u));
u_escape(0xDC00u + (codepoint & 0x3FFu));
}
continue;
}
// a quote, a backslash, or a control character (or DEL)
const unsigned char c = s[i++];
char escape = 0;
switch (c)
{
case 0x08:
escape = 'b';
break;
case 0x09:
escape = 't';
break;
case 0x0A:
escape = 'n';
break;
case 0x0C:
escape = 'f';
break;
case 0x0D:
escape = 'r';
break;
case 0x22:
escape = '"';
break;
case 0x5C:
escape = '\\';
break;
default:
break;
}
if (escape != 0)
{
w[0] = '\\';
w[1] = escape;
w += 2;
}
else
{
u_escape(c);
}
}
room(1);
*w++ = '"';
m_out.set_cursor(w);
} }
/// the copies of dump_compact() that are not fixed-size moves (long /// the copies of dump_text() that are not fixed-size moves (long
/// strings, or near the end of the source); out of line, so that the /// strings, or near the end of the source); out of line, so that the
/// compiler does not merge the fixed-size moves into this call /// compiler does not merge the fixed-size moves into this call
NLOHMANN_VIEW_NOINLINE static void copy_long(char* to, const char* from, std::size_t n) noexcept NLOHMANN_VIEW_NOINLINE static void copy_long(char* to, const char* from, std::size_t n) noexcept
@@ -6547,7 +6752,7 @@ class view_serializer
} }
/*! /*!
@brief the compact output (the default dump(), or with ensure_ascii) @brief the output, compact (the default dump()) or indented, with or without ensure_ascii
The same walk as dump(), with the write position in a local variable The same walk as dump(), with the write position in a local variable
(stores through char pointers would otherwise force a reload of the (stores through char pointers would otherwise force a reload of the
@@ -6555,11 +6760,11 @@ class view_serializer
the source copied by fixed-size moves of 32 bytes where the source has the source copied by fixed-size moves of 32 bytes where the source has
that many bytes left, instead of a library call per token. The buffer that many bytes left, instead of a library call per token. The buffer
keeps 64 bytes of slack for the overshoot. With ensure_ascii (Ascii), a keeps 64 bytes of slack for the overshoot. With ensure_ascii (Ascii), a
string of the source is copied only if it has no byte >= 0x7F (see string of the source is copied only if it has nothing to escape (see
is_ascii_without_delete()); any other string is written by write_string(). copied()); any other string is written by write_ascii().
*/ */
template<bool SourceNumbers, bool Ascii> template<bool SourceNumbers, bool Pretty, bool Ascii>
void dump_compact(const node* root) void dump_text(const node* root)
{ {
struct frame struct frame
{ {
@@ -6574,6 +6779,16 @@ class view_serializer
const char* const src_end = src + m_doc.size; const char* const src_end = src + m_doc.size;
char* w = m_out.cursor(); char* w = m_out.cursor();
char* lim = m_out.limit(); char* lim = m_out.limit();
// (indented output) a line break and 63 indentation characters, from
// which the line break of a level of up to 63 characters is a copy
std::array<char, 64> ruler; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): filled below if used
if (Pretty)
{
ruler.fill(m_style.indent_char);
ruler[0] = '\n';
}
const std::size_t indent = m_style.indent;
depth_counter<Pretty> level; // depth of the innermost open container (indented output)
// room for n bytes and the slack // room for n bytes and the slack
const auto room = [&](std::size_t n) const auto room = [&](std::size_t n)
{ {
@@ -6605,6 +6820,22 @@ class view_serializer
} }
w += n; w += n;
}; };
// (indented output) the line break and the indentation of a level
const auto newline = [&](std::size_t depth)
{
const std::size_t count = 1 + (depth * indent);
room(count);
if (count <= ruler.size())
{
std::memcpy(w, ruler.data(), ruler.size());
}
else
{
*w = '\n';
std::memset(w + 1, m_style.indent_char, count - 1);
}
w += count;
};
// a string that is not a plain string of the source (out of line, so // a string that is not a plain string of the source (out of line, so
// that the cursor stays in a register here) // that the cursor stays in a register here)
const auto escaped = [&](const node & n) const auto escaped = [&](const node & n)
@@ -6612,11 +6843,11 @@ class view_serializer
m_out.set_cursor(w); m_out.set_cursor(w);
if (enabled(Ascii)) if (enabled(Ascii))
{ {
write_ascii_string(n); write_ascii(n);
} }
else else
{ {
write_decoded(n); write_decoded(n, Pretty);
} }
w = m_out.cursor(); w = m_out.cursor();
lim = m_out.limit(); lim = m_out.limit();
@@ -6717,6 +6948,7 @@ class view_serializer
{ {
*w++ = object ? '{' : '['; *w++ = object ? '{' : '[';
stack.push_back(cur); stack.push_back(cur);
level.enter();
if (Editable) if (Editable)
{ {
cur = frame{nav::first(m_doc, n), nav::end(m_doc, n), object}; cur = frame{nav::first(m_doc, n), nav::end(m_doc, n), object};
@@ -6759,6 +6991,10 @@ class view_serializer
{ {
break; break;
} }
if (Pretty)
{
newline(level.leave());
}
room(1); room(1);
*w++ = cur.object ? '}' : ']'; *w++ = cur.object ? '}' : ']';
cur = stack.back(); cur = stack.back();
@@ -6767,13 +7003,17 @@ class view_serializer
room(1); room(1);
*w++ = ','; *w++ = ',';
} }
if (Pretty)
{
newline(level.get());
}
const node* const at = Editable ? cur.pos : n; const node* const at = Editable ? cur.pos : n;
if (cur.object) if (cur.object)
{ {
const node& key = *at; const node& key = *at;
if (copied(key, src, std::integral_constant<bool, Ascii>())) if (copied(key, src, std::integral_constant<bool, Ascii>()))
{ {
room(key.len + 3); room(key.len + (Pretty ? 4 : 3));
*w++ = '"'; *w++ = '"';
copy(src + key.off, key.len); copy(src + key.off, key.len);
w[0] = '"'; w[0] = '"';
@@ -6783,9 +7023,13 @@ class view_serializer
else else
{ {
escaped(key); escaped(key);
room(1); room(Pretty ? 2 : 1);
*w++ = ':'; *w++ = ':';
} }
if (Pretty)
{
*w++ = ' ';
}
if (Editable) if (Editable)
{ {
n = nav::value(at + 1); n = nav::value(at + 1);
@@ -6804,69 +7048,6 @@ class view_serializer
} }
} }
void newline(std::size_t level)
{
if (m_style.pretty)
{
m_out.put('\n');
m_out.put_repeated(m_style.indent_char, level * m_style.indent);
}
}
void write_scalar(const node& n)
{
switch (static_cast<value_t>(n.kind))
{
case value_t::null:
m_out.put("null");
break;
case value_t::boolean:
if ((n.flags & node_flags::is_true) != 0)
{
m_out.put("true");
}
else
{
m_out.put("false");
}
break;
case value_t::string:
write_string(n);
break;
case value_t::number_integer:
case value_t::number_unsigned:
{
const char* const token = m_doc.str(n);
const std::uint32_t len = number_length(n);
if (!m_style.source_numbers && len == 2 && token[0] == '-' && token[1] == '0')
{
m_out.put('0'); // parse() reads -0 as the integer 0
}
else
{
m_out.put(token, len);
}
break;
}
case value_t::number_float:
if (m_style.source_numbers && (n.flags & node_flags::storage) != node_flags::edited)
{
m_out.put(m_doc.str(n), n.len); // (a float set by an edit is written as with shortest)
}
else
{
write_float_node(n);
}
break;
case value_t::object: // LCOV_EXCL_LINE (containers are written by dump())
case value_t::array: // LCOV_EXCL_LINE
case value_t::binary: // LCOV_EXCL_LINE (not in a document)
case value_t::discarded: // LCOV_EXCL_LINE
default: // LCOV_EXCL_LINE
break; // LCOV_EXCL_LINE
}
}
/// a float node as dump() writes it /// a float node as dump() writes it
void write_float_node(const node& n) void write_float_node(const node& n)
{ {
@@ -6977,72 +7158,6 @@ class view_serializer
m_out.put(s.data(), s.size()); m_out.put(s.data(), s.size());
} }
void write_string(const node& n)
{
const char* const s = m_doc.str(n);
m_out.put('"');
if ((n.flags & node_flags::storage) == 0 && (!m_style.ensure_ascii || is_ascii_without_delete(s, n.len)))
{
// a string of the source without escape sequences has nothing to
// escape (with ensure_ascii, if it is printable ASCII)
m_out.put(s, n.len);
}
else if (m_style.ensure_ascii)
{
write_escaped<true>(reinterpret_cast<const unsigned char*>(s), n.len); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
}
else
{
write_escaped<false>(reinterpret_cast<const unsigned char*>(s), n.len); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
}
m_out.put('"');
}
/// as serializer::dump_escaped(); strings of a document are valid UTF-8,
/// except in a damaged image loaded with image_check::bounds, for which
/// this throws what basic_json::dump() throws for the string
template<bool EnsureAscii>
void write_escaped(const unsigned char* s, std::size_t n)
{
std::size_t i = 0;
while (i < n)
{
std::size_t run = 0;
if (!EnsureAscii)
{
run = string_bulk_run(s + i, n - i);
}
else if (is_ascii_copyable(s[i]))
{
run = find_ascii_copyable_run(s + i, n - i);
}
if (run != 0)
{
m_out.put(reinterpret_cast<const char*>(s + i), run); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
i += run;
continue;
}
std::uint32_t codepoint = s[i];
std::size_t len = 1;
if (codepoint >= 0x80)
{
len = validate_one_utf8(s + i, n - i);
if (NLOHMANN_VIEW_UNLIKELY(len == 0))
{
invalid_utf8(s, n);
return;
}
codepoint &= 0xFFu >> (len + 1);
for (std::size_t k = 1; k < len; ++k)
{
codepoint = (codepoint << 6u) | (s[i + k] & 0x3Fu);
}
}
write_codepoint<EnsureAscii>(codepoint, s + i, len);
i += len;
}
}
/// throw what basic_json::dump() throws for a string that is not valid UTF-8 /// throw what basic_json::dump() throws for a string that is not valid UTF-8
NLOHMANN_VIEW_NOINLINE static void invalid_utf8(const unsigned char* s, std::size_t n) NLOHMANN_VIEW_NOINLINE static void invalid_utf8(const unsigned char* s, std::size_t n)
{ {
-272
View File
@@ -492,275 +492,3 @@ TEST_CASE("binary_reserve_hint never over-reserves")
CHECK(nlohmann::detail::binary_reserve_hint(json({{"a", 1}, {"b", 2}})) == 5); CHECK(nlohmann::detail::binary_reserve_hint(json({{"a", 1}, {"b", 2}})) == 5);
} }
} }
namespace
{
// numbers around every boundary of the integer encodings, and floats of each
// kind (exact in float, needing double, special values)
std::vector<json> boundary_numbers()
{
std::vector<json> numbers;
for (int k = 0; k < 64; ++k)
{
const std::uint64_t p = std::uint64_t{1} << k;
for (const std::uint64_t d :
{
p - 1, p, p + 1
})
{
numbers.emplace_back(d);
if (d <= static_cast<std::uint64_t>((std::numeric_limits<std::int64_t>::max)()))
{
numbers.emplace_back(static_cast<std::int64_t>(d));
numbers.emplace_back(-static_cast<std::int64_t>(d));
numbers.emplace_back(-static_cast<std::int64_t>(d) - 1);
}
}
}
numbers.emplace_back((std::numeric_limits<std::int64_t>::min)());
numbers.emplace_back((std::numeric_limits<std::uint64_t>::max)());
for (const double f :
{
0.0, -0.0, 1.0, -1.0, 0.5, 0.1, 3.25, 1e300, -1e-300, 3.4028234663852886e38, 1e39,
std::numeric_limits<double>::infinity(), -std::numeric_limits<double>::infinity(),
std::numeric_limits<double>::quiet_NaN()
})
{
numbers.emplace_back(f);
}
return numbers;
}
// the concatenation of the encodings of the elements of @a numbers[first, first + count)
template<typename Single>
std::vector<std::uint8_t> concat_singles(const std::vector<json>& numbers, const std::size_t first, const std::size_t count, Single single)
{
std::vector<std::uint8_t> bytes;
for (std::size_t i = first; i < first + count; ++i)
{
const auto one = single(numbers[i]);
bytes.insert(bytes.end(), one.begin(), one.end());
}
return bytes;
}
} // namespace
// Runs of numbers in arrays are encoded straight into space reserved from the
// sink, by code that is separate from the encoding of a single value. Both have
// to agree on every number: an array is its head and the encodings of its
// elements, one after the other.
TEST_CASE("runs of numbers in arrays are encoded as the numbers one by one")
{
const std::vector<json> numbers = boundary_numbers();
// arrays of every length that a one-byte head covers, so that runs are as
// long as, and longer and shorter than, what fits one reservation
for (const std::size_t count :
{
1u, 2u, 3u, 5u, 14u, 15u, 23u
})
{
for (std::size_t first = 0; first + count <= numbers.size(); first += 7)
{
const json array(std::vector<json>(numbers.begin() + static_cast<std::ptrdiff_t>(first),
numbers.begin() + static_cast<std::ptrdiff_t>(first + count)));
CAPTURE(array.dump(-1, ' ', false, json::error_handler_t::replace))
std::vector<std::uint8_t> expected{static_cast<std::uint8_t>(0x80 + count)};
auto elements = concat_singles(numbers, first, count, [](const json & j)
{
return json::to_cbor(j);
});
expected.insert(expected.end(), elements.begin(), elements.end());
CHECK(json::to_cbor(array) == expected);
expected.assign(1, static_cast<std::uint8_t>(count <= 15 ? 0x90 + count : 0xDC));
if (count > 15)
{
expected.push_back(0);
expected.push_back(static_cast<std::uint8_t>(count));
}
elements = concat_singles(numbers, first, count, [](const json & j)
{
return json::to_msgpack(j);
});
expected.insert(expected.end(), elements.begin(), elements.end());
CHECK(json::to_msgpack(array) == expected);
for (const bool bjdata :
{
false, true
})
{
expected.assign(1, static_cast<std::uint8_t>('['));
elements = concat_singles(numbers, first, count, [bjdata](const json & j)
{
return bjdata ? json::to_bjdata(j) : json::to_ubjson(j);
});
expected.insert(expected.end(), elements.begin(), elements.end());
expected.push_back(static_cast<std::uint8_t>(']'));
CHECK((bjdata ? json::to_bjdata(array) : json::to_ubjson(array)) == expected);
}
// BON8 has no integers above the int64 range
bool representable = true;
for (std::size_t i = first; i < first + count; ++i)
{
representable = representable && bon8_representable(numbers[i]);
}
if (representable)
{
expected.assign(1, static_cast<std::uint8_t>(count <= 4 ? 0x80 + count : 0x85));
elements = concat_singles(numbers, first, count, [](const json & j)
{
return json::to_bon8(j);
});
expected.insert(expected.end(), elements.begin(), elements.end());
if (count > 4)
{
expected.push_back(0xFE);
}
CHECK(json::to_bon8(array) == expected);
}
else
{
CHECK_THROWS_AS(json::to_bon8(array), json::out_of_range&);
}
}
}
}
// An array that is written in more than one reservation (more numbers than fit
// one), and arrays of an index with more digits, round trip for every format.
TEST_CASE("large arrays and many digits of indices round trip")
{
json array = json::array();
for (int i = 0; i < 12000; ++i)
{
switch (i % 4)
{
case 0:
array.push_back(i);
break;
case 1:
array.push_back(-i);
break;
case 2:
array.push_back(i * 0.25);
break;
default:
array.push_back(i % 8 == 3);
break;
}
}
json document = json::object();
document["numbers"] = array;
document["nested"] = json::array({array, "text", array});
CHECK(json::from_cbor(json::to_cbor(document)) == document);
CHECK(json::from_msgpack(json::to_msgpack(document)) == document);
CHECK(json::from_ubjson(json::to_ubjson(document)) == document);
CHECK(json::from_ubjson(json::to_ubjson(document, true, true)) == document);
CHECK(json::from_bjdata(json::to_bjdata(document)) == document);
CHECK(json::from_bson(json::to_bson(document)) == document);
CHECK(json::from_bon8(json::to_bon8(document)) == document);
}
// Strings and keys are written with their head in one reservation up to a
// length; around that length, and with every kind of head, they must read back.
TEST_CASE("strings of every head length round trip")
{
json document = json::object();
json array = json::array();
for (const std::size_t length :
{
0u, 1u, 2u, 3u, 4u, 7u, 8u, 15u, 16u, 17u, 23u, 24u, 25u, 31u, 32u, 33u, 63u, 127u, 128u, 129u, 200u, 245u, 246u, 247u, 248u, 249u, 250u, 251u, 252u, 253u, 254u, 255u, 256u, 257u, 300u, 1000u, 65535u, 65536u, 70000u
})
{
const std::string text(length, 'x');
array.push_back(text);
document[std::string(length, 'k')] = text;
}
document["array"] = array;
CHECK(json::from_cbor(json::to_cbor(document)) == document);
CHECK(json::from_msgpack(json::to_msgpack(document)) == document);
CHECK(json::from_ubjson(json::to_ubjson(document)) == document);
CHECK(json::from_ubjson(json::to_ubjson(document, true, true)) == document);
CHECK(json::from_bjdata(json::to_bjdata(document)) == document);
CHECK(json::from_bson(json::to_bson(document)) == document);
CHECK(json::from_bon8(json::to_bon8(document)) == document);
}
// What a write that throws leaves behind in an output adapter is what was written
// before it, whatever the adapter and however the adapter sink buffers.
TEST_CASE("a write that throws leaves the same bytes in every kind of output")
{
json array = json::array();
for (int i = 0; i < 700; ++i)
{
array.push_back(i * 1000);
array.push_back("text " + std::to_string(i));
}
array.push_back("bad \xFF byte");
json document = json::object();
document["first"] = array;
document["second"] = 1;
const auto strict = json::error_handler_t::strict;
SECTION("cbor")
{
std::vector<std::uint8_t> as_vector;
std::string as_string;
std::ostringstream as_stream;
CHECK_THROWS_AS(json::to_cbor(document, as_vector, strict), json::type_error&);
CHECK_THROWS_AS(json::to_cbor(document, as_string, strict), json::type_error&);
CHECK_THROWS_AS(json::to_cbor(document, as_stream, strict), json::type_error&);
CHECK(!as_vector.empty());
CHECK(std::string(as_vector.begin(), as_vector.end()) == as_string);
CHECK(as_stream.str() == as_string);
}
SECTION("msgpack")
{
std::vector<std::uint8_t> as_vector;
std::string as_string;
std::ostringstream as_stream;
CHECK_THROWS_AS(json::to_msgpack(document, as_vector, strict), json::type_error&);
CHECK_THROWS_AS(json::to_msgpack(document, as_string, strict), json::type_error&);
CHECK_THROWS_AS(json::to_msgpack(document, as_stream, strict), json::type_error&);
CHECK(!as_vector.empty());
CHECK(std::string(as_vector.begin(), as_vector.end()) == as_string);
CHECK(as_stream.str() == as_string);
}
SECTION("ubjson")
{
std::vector<std::uint8_t> as_vector;
std::string as_string;
std::ostringstream as_stream;
CHECK_THROWS_AS(json::to_ubjson(document, as_vector, false, false, strict), json::type_error&);
CHECK_THROWS_AS(json::to_ubjson(document, as_string, false, false, strict), json::type_error&);
CHECK_THROWS_AS(json::to_ubjson(document, as_stream, false, false, strict), json::type_error&);
CHECK(!as_vector.empty());
CHECK(std::string(as_vector.begin(), as_vector.end()) == as_string);
CHECK(as_stream.str() == as_string);
}
SECTION("bson")
{
// the size pass finds the string before anything is written
std::vector<std::uint8_t> as_vector;
std::string as_string;
std::ostringstream as_stream;
CHECK_THROWS_AS(json::to_bson(document, as_vector, strict), json::type_error&);
CHECK_THROWS_AS(json::to_bson(document, as_string, strict), json::type_error&);
CHECK_THROWS_AS(json::to_bson(document, as_stream, strict), json::type_error&);
CHECK(as_vector.empty());
CHECK(as_string.empty());
CHECK(as_stream.str().empty());
}
}
-49
View File
@@ -285,55 +285,6 @@ TEST_CASE("custom number types: integers beyond 64 bits")
CHECK(wide_json::to_bjdata(wide_json::array({uint_big, uint_big}), true, true) == expected_bjdata_array); CHECK(wide_json::to_bjdata(wide_json::array({uint_big, uint_big}), true, true) == expected_bjdata_array);
} }
SECTION("integers in arrays and objects")
{
// runs of numbers and scalar object members are encoded by loops of
// their own for the number types up to 64 bits; a wider number type
// takes the generic path, which checks the range of every integer
const wide_json run = wide_json::array({make_int(1), make_int(-3), make_uint(7), wide_json(1.5), int_big, make_int(5)});
const wide_json run_negative = wide_json::array({make_int(1), make_int(-3), int_big_negative});
const wide_json run_unsigned = wide_json::array({make_uint(1), make_uint(7), uint_big});
const wide_json members = wide_json::object({{"a", make_int(1)}, {"b", wide_json(1.5)}, {"c", int_big}});
const std::vector<wide_json> failing = {run, run_negative, run_unsigned, members, wrap(run), wrap(members)};
for (const auto& value : failing)
{
CHECK_THROWS_AS(_ = wide_json::to_cbor(value), out_of_range&);
CHECK_THROWS_AS(_ = wide_json::to_msgpack(value), out_of_range&);
CHECK_THROWS_AS(_ = wide_json::to_bson(wrap(value)), out_of_range&);
CHECK_THROWS_AS(_ = wide_json::to_bon8(value), out_of_range&);
}
// a failing write leaves no more than the elements before the failing one
std::vector<std::uint8_t> out;
CHECK_THROWS_AS(wide_json::to_cbor(run, out), out_of_range&);
// (the head of the array, 1, -3, 7, and 1.5 as a single-precision float)
CHECK(out == std::vector<std::uint8_t>({0x86, 0x01, 0x22, 0x07, 0xFA, 0x3F, 0xC0, 0x00, 0x00}));
out.clear();
CHECK_THROWS_AS(wide_json::to_bon8(run, out), out_of_range&);
CHECK(!out.empty());
// the same values that fit are written exactly as with 64-bit number types
const wide_json fits = wide_json::array({make_int(1), make_int(-3), make_uint(7), wide_json(1.5), make_int(min_int64), make_int(max_int64), make_int(200), make_int(-70000)});
const nlohmann::json fits64 = nlohmann::json::array({1, -3, 7, 1.5, (std::numeric_limits<std::int64_t>::min)(), (std::numeric_limits<std::int64_t>::max)(), 200, -70000});
CHECK(wide_json::to_cbor(fits) == nlohmann::json::to_cbor(fits64));
CHECK(wide_json::to_msgpack(fits) == nlohmann::json::to_msgpack(fits64));
CHECK(wide_json::to_ubjson(fits) == nlohmann::json::to_ubjson(fits64));
CHECK(wide_json::to_ubjson(fits, true, true) == nlohmann::json::to_ubjson(fits64, true, true));
CHECK(wide_json::to_bjdata(fits) == nlohmann::json::to_bjdata(fits64));
CHECK(wide_json::to_bjdata(fits, true, true) == nlohmann::json::to_bjdata(fits64, true, true));
CHECK(wide_json::to_bon8(fits) == nlohmann::json::to_bon8(fits64));
CHECK(wide_json::to_bson(wrap(fits)) == nlohmann::json::to_bson(nlohmann::json::object({{"v", fits64}})));
const wide_json fit_members = wide_json::object({{"a", make_int(1)}, {"b", make_int(-300)}, {"c", make_uint(max_uint64)}, {"d", wide_json(2.5)}});
const nlohmann::json fit_members64 = nlohmann::json::object({{"a", 1}, {"b", -300}, {"c", (std::numeric_limits<std::uint64_t>::max)()}, {"d", 2.5}});
CHECK(wide_json::to_cbor(fit_members) == nlohmann::json::to_cbor(fit_members64));
CHECK(wide_json::to_msgpack(fit_members) == nlohmann::json::to_msgpack(fit_members64));
CHECK(wide_json::to_ubjson(fit_members) == nlohmann::json::to_ubjson(fit_members64));
CHECK(wide_json::to_bjdata(fit_members) == nlohmann::json::to_bjdata(fit_members64));
CHECK(wide_json::to_bson(fit_members) == nlohmann::json::to_bson(fit_members64));
}
SECTION("BJData ND-array") SECTION("BJData ND-array")
{ {
// an ND-array element beyond 64 bits does not fit any dtype, so the // an ND-array element beyond 64 bits does not fit any dtype, so the
+125
View File
@@ -186,6 +186,131 @@ TEST_CASE("json_view dump")
CHECK(deep_document.root().dump() == deep); CHECK(deep_document.root().dump() == deep);
} }
SECTION("ensure_ascii: a character that needs an escape at each position of a string")
{
// string lengths around the width of the vector steps, in a string value, a key, and a value near the end of the text
for (const char* const special :
{
"\xC3\xA9", "\xE4\xB8\xAD", "\xF0\x9F\x98\x80", "\x7F", "\\n", "\\u0001", "\\\"", "\\\\"
})
{
for (std::size_t length = 0; length <= 40; ++length)
{
for (std::size_t position = 0; position <= length; ++position)
{
std::string content(length, 'a');
content.insert(position, special);
const std::string texts[] =
{
"\"" + content + "\"",
"[\"" + content + "\"]",
"{\"" + content + "\":1}",
};
for (const std::string& text : texts)
{
CAPTURE(text)
const ordered_json_document d = ordered_json_document::parse(text);
const ordered_json j = ordered_json::parse(text);
CHECK(d.root().dump(-1, ' ', true) == j.dump(-1, ' ', true));
CHECK(d.root().dump(2, ' ', true) == j.dump(2, ' ', true));
CHECK(d.root().dump() == j.dump());
}
}
}
}
}
SECTION("ensure_ascii: every character of the Basic Multilingual Plane and some beyond")
{
// (the two- and three-byte sequences, including the ranges that need a check: E0, ED, EF BF BF)
std::string text = "[";
std::size_t in_chunk = 0;
const auto append = [&](std::uint32_t cp)
{
if (cp < 0x80)
{
if (cp < 0x20 || cp == '"' || cp == '\\')
{
return;
}
text += static_cast<char>(cp);
}
else if (cp < 0x800)
{
text += static_cast<char>(0xC0 | (cp >> 6));
text += static_cast<char>(0x80 | (cp & 0x3F));
}
else if (cp < 0x10000)
{
text += static_cast<char>(0xE0 | (cp >> 12));
text += static_cast<char>(0x80 | ((cp >> 6) & 0x3F));
text += static_cast<char>(0x80 | (cp & 0x3F));
}
else
{
text += static_cast<char>(0xF0 | (cp >> 18));
text += static_cast<char>(0x80 | ((cp >> 12) & 0x3F));
text += static_cast<char>(0x80 | ((cp >> 6) & 0x3F));
text += static_cast<char>(0x80 | (cp & 0x3F));
}
};
for (std::uint32_t cp = 0x20; cp < 0x10400; ++cp)
{
if (cp >= 0xD800 && cp < 0xE000)
{
continue;
}
if (in_chunk == 0)
{
text += text.size() > 1 ? ",\"" : "\"";
}
append(cp);
if (++in_chunk == 37)
{
text += "\"";
in_chunk = 0;
}
}
text += in_chunk != 0 ? "\"]" : "]";
const ordered_json_document d = ordered_json_document::parse(text);
const ordered_json j = ordered_json::parse(text);
CHECK(d.root().dump(-1, ' ', true) == j.dump(-1, ' ', true));
CHECK(d.root().dump(2, ' ', true) == j.dump(2, ' ', true));
CHECK(d.root().dump() == j.dump());
CHECK(d.root().dump(2) == j.dump(2));
}
SECTION("deep nesting, indented")
{
// indentations of up to and beyond the width of the writer's fixed-size copy
for (const int depth :
{
1, 5, 20, 70, 300
})
{
std::string text;
for (int i = 0; i < depth; ++i)
{
text += i % 2 == 0 ? "[1," : "{\"k\":2,\"\\u00e9\":";
}
text += "[]";
for (int i = depth - 1; i >= 0; --i)
{
text += i % 2 == 0 ? "]" : "}";
}
const ordered_json_document d = ordered_json_document::parse(text);
const ordered_json j = ordered_json::parse(text);
for (const int indent :
{
0, 1, 2, 3, 8
})
{
CHECK(d.root().dump(indent) == j.dump(indent));
CHECK(d.root().dump(indent, '\t') == j.dump(indent, '\t'));
}
}
}
SECTION("the output buffer of a small value is small") SECTION("the output buffer of a small value is small")
{ {
// an escaped key after the value: its node lies in the arena, so the // an escaped key after the value: its node lies in the arena, so the
+20
View File
@@ -682,6 +682,26 @@ TEST_CASE("json_view images: ownership")
#if !defined(JSON_NOEXCEPTION) #if !defined(JSON_NOEXCEPTION)
TEST_CASE("json_view images: errors") TEST_CASE("json_view images: errors")
{ {
SECTION("invalid UTF-8 of a decoded string: the compact dump writes it, the others throw")
{
const json_document d = json_document::parse(R"({"k\n":"\u00e9x"})");
std::vector<std::uint8_t> image = d.save();
const std::size_t arena = text_at(image) + header_size_field(image, 16) + 1;
image[arena + 3] = 0x28; // the key is "k\n"; the value starts at 3: 0xC3 0x28 is invalid
const json_document loaded = json_document::load(image, image_check::bounds);
CHECK(loaded.root().dump().size() == 13);
for (const int indent :
{
0, 2
})
{
CHECK_THROWS_AS(loaded.root().dump(indent), json::type_error);
CHECK_THROWS_AS(loaded.root().dump(indent, ' ', true), json::type_error);
}
CHECK_THROWS_AS(loaded.root().dump(-1, ' ', true), json::type_error);
CHECK_THROWS_WITH_AS(loaded.root().dump(2), "[json.exception.type_error.316] invalid UTF-8 byte at index 1: 0x28", json::type_error&);
}
SECTION("a literal as the root: dump() after loading") SECTION("a literal as the root: dump() after loading")
{ {
for (const char* text : for (const char* text :