Binary readers: append strings straight from contiguous input (#6028)

* Binary readers: append strings straight from contiguous input

get_bytes() read every string into a zero-filled resize() and copied the
bytes over it, so each byte was written twice. A string that the input holds
in one block (bulk_remaining() at least its length) is now appended from it
directly. The claimed length is checked against the input first, so a huge
claimed length still allocates nothing it cannot fill; other inputs and
binary values keep the chunked loop.

from_cbor() and from_msgpack() of twitter.json and citm_catalog.json get
5.5-9.1% faster on an Apple M1 Max and 2.7-8.6% on x86-64 (GCC 15, Clang 21);
canada.json, which has few strings, is unchanged.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Binary readers: keep the tag argument of append_from_block() astyle-proof

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix CI: bugprone-signed-char-misuse in append_from_block

Convert the (positive) length to its unsigned type before widening it
to std::size_t, as clang-tidy flags the int8_t to std::size_t cast.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Tests: sweep_allocator::construct() accepts pointers to const

Recent libc++ (emsdk latest) probes construct(const T*, const T&) in an
unevaluated context while checking for trivial copies; the static_cast to
void* in the noexcept specification then fails to compile.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
(cherry picked from commit 794bf70823c12a2a1cb2a8a95f34dfb6357746a8)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Tests: guard sweep_allocator::construct() against const types instead of const_cast

The const_cast added in the previous commit is rejected by
cppcoreguidelines-pro-type-const-cast (clang-tidy). Make construct() not
viable for const U instead, which is what libc++ probes for.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
(cherry picked from commit de099fa607001b710862fd1bbecef3609c54cbc4)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix CI: astyle and clang-tidy for sweep_allocator::construct()

astyle 3.4.13 reformats the enable_if<!...> default template argument
(Check amalgamation). Delete the const overload instead, which makes the
libc++ probe for const objects fail in the SFINAE context just the same
and needs no <type_traits>.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Niels LohmannandClaude Sonnet 5.5 authored and GitHub committed 2026-11-07 10:10:50 +01:00
1 parent cbb9e14889
commit 951271127b
2 files changed
+76

No files matched your search

@@ -4887,6 +4887,14 @@ class binary_reader
const char* context,
ContainerType& result)
{
// a string that is in the block of input as a whole is appended from
// it, without the zero-filling resize() below
constexpr bool from_block = bulk_scan && std::is_same<ContainerType, string_t>::value;
if (append_from_block(len, result, std::integral_constant<bool, from_block> {}))
{
return true;
}
// upper bound on the number of bytes read (and allocated) per chunk
constexpr std::size_t chunk_size = 4096;
@@ -4923,6 +4931,36 @@ class binary_reader
return true;
}
/// append the @a len bytes at the current position to @a result, if the
/// input holds them in one block; false (with nothing read) otherwise
template<typename NumberType>
bool append_from_block(const NumberType len, string_t& result, std::true_type /*block*/)
{
if (len <= 0)
{
return false;
}
// (len > 0, so the unsigned type holds it; casting a signed char
// straight to std::size_t would trip bugprone-signed-char-misuse)
const auto ulen = static_cast<typename std::make_unsigned<NumberType>::type>(len);
const std::size_t remaining = ia.bulk_remaining();
if (static_cast<std::uintmax_t>(ulen) > static_cast<std::uintmax_t>(remaining))
{
return false; // (the loop reports the premature end)
}
const auto n = static_cast<std::size_t>(ulen);
result.append(reinterpret_cast<const typename string_t::value_type*>(ia.bulk_data()), n);
ia.bulk_skip(n);
chars_read += n;
return true;
}
template<typename NumberType, typename ContainerType>
static bool append_from_block(const NumberType /*len*/, ContainerType& /*result*/, std::false_type /*block*/) noexcept
{
return false;
}
/*!
@param[in] context further context information (for diagnostics)
@return whether the last read character is not EOF
+38
View File
@@ -22903,6 +22903,14 @@ class binary_reader
const char* context,
ContainerType& result)
{
// a string that is in the block of input as a whole is appended from
// it, without the zero-filling resize() below
constexpr bool from_block = bulk_scan && std::is_same<ContainerType, string_t>::value;
if (append_from_block(len, result, std::integral_constant<bool, from_block> {}))
{
return true;
}
// upper bound on the number of bytes read (and allocated) per chunk
constexpr std::size_t chunk_size = 4096;
@@ -22939,6 +22947,36 @@ class binary_reader
return true;
}
/// append the @a len bytes at the current position to @a result, if the
/// input holds them in one block; false (with nothing read) otherwise
template<typename NumberType>
bool append_from_block(const NumberType len, string_t& result, std::true_type /*block*/)
{
if (len <= 0)
{
return false;
}
// (len > 0, so the unsigned type holds it; casting a signed char
// straight to std::size_t would trip bugprone-signed-char-misuse)
const auto ulen = static_cast<typename std::make_unsigned<NumberType>::type>(len);
const std::size_t remaining = ia.bulk_remaining();
if (static_cast<std::uintmax_t>(ulen) > static_cast<std::uintmax_t>(remaining))
{
return false; // (the loop reports the premature end)
}
const auto n = static_cast<std::size_t>(ulen);
result.append(reinterpret_cast<const typename string_t::value_type*>(ia.bulk_data()), n);
ia.bulk_skip(n);
chars_read += n;
return true;
}
template<typename NumberType, typename ContainerType>
static bool append_from_block(const NumberType /*len*/, ContainerType& /*result*/, std::false_type /*block*/) noexcept
{
return false;
}
/*!
@param[in] context further context information (for diagnostics)
@return whether the last read character is not EOF