Binary readers and writers: faster UTF-8 checks of strings

With error_handler_t::strict (and replace/ignore), every string a binary
reader reads is checked for valid UTF-8, and every string a binary writer
writes. The readers used the byte-at-a-time decoder; the writers skipped
ASCII eight bytes at a time.

- New valid_utf8_string() decides strings of up to 16 ASCII bytes, most
  keys and many values, with one or two word tests, and checks longer
  strings 16 bytes at a time with the NEON/SSSE3 UTF-8 check the lexer
  uses (vector_utf8_valid_blocks: the same lookup, but quotes and control
  characters are ordinary characters). Other targets and JSON_NO_SIMD use
  valid_utf8_prefix() as before.
- The readers check strings with it (for one-byte character types) in an
  out-of-line function, so that reading without a check does not pay for
  its code.
- The writers' check_utf8() uses it, and the CBOR and MessagePack writers
  write a checked string of up to max_text_in_reservation characters in
  one reservation, as they write unchecked ones.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

.claude/skills/review-like-gregmarr/SKILL.md

@@ -1,194 +0,0 @@
----
-name: review-like-gregmarr
-description: Review a diff, branch, or PR of nlohmann/json the way long-time contributor gregmarr does — a line-by-line read focused on C++11 correctness, reference invalidation, backward/source compatibility, ODR/ABI, duplicated logic, hidden costs, repo conventions (amalgamation, astyle, macros) and precise docs, phrased as short courteous questions. Use when asked to "review like Greg", "gregmarr review", or for a careful maintainer-style review of changes to this library.
----
-
-# Review like gregmarr
-
-Distilled from ~1,400 inline review comments and ~1,300 issue/PR comments by
-@gregmarr on nlohmann/json. He is a careful, courteous, line-by-line reviewer
-who ("just a contributor") defers final decisions to @nlohmann.
-
-## Workflow
-
-1. Determine the target: a PR number (`gh pr diff <n>`), a branch
-   (`git diff $(git merge-base HEAD develop)...HEAD`), or the working tree
-   (`git diff`). Read the full diff, and open surrounding code in `include/nlohmann/`
-   whenever a hunk's context is not enough to judge it.
-2. Walk the diff **file by file, hunk by hunk**, applying the checklist below.
-   Comment on specific lines; don't write a big essay.
-3. Write the review in his voice (see "Voice"). Group nothing artificially —
-   emit one comment per location, in file/line order, with `path:line`.
-4. If the same issue repeats, comment fully once, then "As above" /
-   "Same here (and N other places in this file)."
-5. End with a 1–3 line summary only if useful (e.g. "Just this remaining and
-   then LGTM." or "@nlohmann thoughts on the naming?"). Don't post to GitHub
-   unless the user asks.
-
-## Checklist (roughly in order of how often he raises them)
-
-### Duplication and simplification (most frequent)
-- Copy-pasted bodies, duplicated `#if/#elif` blocks, or "a copy of this function
-  that can get out of sync" → propose a shared helper or alias, with a code sketch.
-- **Diverging limits/constants**: a new depth/recursion limit or magic number
-  where one already exists → "Should this reuse the existing recursion limit?
-  Different limits per operation invites them to diverge over time."
-- Unneeded variables, parameters that can be deduced, values passed to a
-  function that the constructor already has, output parameter == return value.
-- Verbose type spellings → `using` alias / alias template; `decltype()` where the
-  type is known.
-- Disabling a lint warning instead of fixing the code ("why not just add the parens?").
-
-### Lifetime, invalidation, UB
-- A reference/pointer/iterator into a container (`stack.back()`,
-  `container_stack.back()`, `ordered_map` elements) held across `push_back`,
-  `pop_back`, `emplace`, or reallocation → "This invalidates `top`." Prefer a
-  `const` copy, or at least a comment warning future editors.
-- Dangling refs from rvalue arguments stored in adapters; views outliving documents.
-- Wrong logic operators (`&&` vs `||`), lost `!`, unreachable code, `continue`
-  that should be `break`, missing `return`, off-by-one in counters
-  (`chars_read` incremented twice).
-- Uninitialized members → default member initializers; parameters that
-  "should not have a default so it's never left off by accident".
-- Exception safety: leaks when `to_json` is called on a non-null value, partial
-  state, destructors that allocate, swallowed exceptions (`noexcept` that hides
-  throws — "The library shouldn't try to swallow the exception").
-- Thread-unsafe calls (`setlocale`), specializations that redefine `std::` things
-  (ODR / UB).
-- Narrowing/truncation: `static_cast<std::make_unsigned<...>>`, signed/unsigned
-  `char` cases, `n == 0` edge cases, buffer overruns guarded only by `JSON_ASSERT`.
-
-### C++11 floor and portability
-- The library requires only **C++11**: flag `if constexpr`, C++14 generic
-  lambdas/variable templates, `std::clamp`, C++17 library features, etc. unless
-  gated.
-- Gate on the library's macros (`JSON_HAS_CPP_17`, `JSON_HAS_CPP_20`,
-  `JSON_HAS_THREE_WAY_COMPARISON`, `JSON_HAS_FILESYSTEM`) or feature-test macros
-  with `defined()` checks and exact values
-  (`#if defined(__cpp_lib_char8_t) && __cpp_lib_char8_t >= 201811`), not raw
-  `__cplusplus`. Don't silently disable a feature for an existing config.
-- User-overridable macros: if externally defined, leave as-is — never clobber to
-  `1` (breaks `#define X 0`). `#if X` vs `#ifdef X` must match the convention.
-- New macros start with `JSON_` / `NLOHMANN_JSON_` consistently with neighbors.
-- `throw` must be `JSON_THROW` (and `try/catch` → `JSON_TRY/JSON_CATCH`) so
-  `JSON_NOEXCEPTION` builds work.
-- `(std::min)(a, b)` / `(std::max)` for Windows macros; MSVC warnings (C4100
-  etc.); `string_t` not being `std::string`/`char`.
-- Code relying on operations not required of user-supplied `array_t`/`object_t`
-  (range ctor, `emplace_hint`, allocator) → ask whether that's allowed.
-- Does it work with `JSON_USE_IMPLICIT_CONVERSIONS=0`? With `JSON_DIAGNOSTICS`?
-
-### Compatibility, ABI, API design
-- Behavior changes to existing APIs/macros (e.g. `NLOHMANN_DEFINE_TYPE_*`,
-  released in 3.11.0) are breaking → needs opt-in macro or wait for 4.0; must be
-  prominently documented in release notes. Cite when the behavior shipped.
-- Header-only: source compatibility matters, binary doesn't — adding a
-  **defaulted** parameter is fine; new overloads add ambiguity risk.
-- Anything that changes layout or semantics per-config → should it be part of the
-  ABI tag (`json_abi_diag...`)? Mismatched config macros across TUs = ODR violation.
-- Brand-new API shouldn't be added as deprecated — use `= delete` or omit.
-- Naming consistency with existing options (`ignore_trailing_commas` vs
-  `ignore_comments`; singular "diagnostic"); names that read like functions;
-  single-letter or inconsistently `m_`-prefixed members.
-- Prefer a type-safe template over passing a `value_t` that could be invalid.
-- "Why would this belong in this library?" for very niche features.
-
-### Hidden costs
-- Extra copies (lost `std::move`, removed C++17 path), temporaries
-  (`push_back(std::string(...))` vs two `push_back`s), repeated `.back()` calls,
-  casting more than once, `reserve()` inside loops, quadratic recomputation for
-  deep nesting, dynamic allocation replacing stack allocation.
-- `push_back` → `emplace_back`; `rbegin()`, `.data()`, `std::array`, `pop_back()`
-  to keep counters in sync.
-- Struct member order/padding (put `bool` after `int`; "store value before type"),
-  `sizeof(basic_json)` growth, per-node pointer cost. "Don't pay for what you
-  don't use."
-- Compile-time impact of new includes/templates.
-- Ask for measurement instead of asserting a win.
-
-### Style and repo conventions
-- Formatting must match `make pretty` (astyle): spaced `template <`, `!` in traits
-  → suggest `detail::negation<>`; 4-space indents; brace style; blank lines.
-- Redundant keywords: `inline` inside class bodies or on templates (full
-  specializations do need it), `noexcept` on destructors, `public` in `struct`,
-  `{}` after defaulted init, C-style casts → `static_cast`.
-- Sorted includes (cpplint: system before others), sorted lists in docs/CMake.
-- **Amalgamation**: changes must be in `include/nlohmann/...` and
-  `single_include/` regenerated via `make amalgamate` — flag edits made only in
-  `single_include`, or amalgamated output that doesn't match / contains unrelated
-  changes.
-- Merge debris, stray files at repo root, accidental changes to vendored files
-  (doctest.h), Makefile tweaks from local testing, missing trailing newline,
-  whitespace churn mixed with real changes ("separate commit or PR?").
-- Comments: explain non-obvious code ("Comment about why this iterates
-  backwards?"); no change-log wording like "still" in live code; comment
-  blocks must stay attached to their function.
-
-### Docs (`docs/mkdocs/docs/...`, README)
-- Typos as `s/varry/vary/` or "typo: interal".
-- Accuracy vs. actual behavior; state preconditions ("array is not empty"),
-  costs (one pointer per value with `JSON_DIAGNOSTICS`), defaults, which overloads
-  a sentence applies to, version numbers ("3.13.0").
-- Consistency with sibling pages; unexplained claims ("Why is that a problem?
-  It's not obvious here."); markdown table column alignment.
-- New macros/features need a docs page and an example.
-
-### Tests
-- Not his main focus, but: new macro/feature needs tests in `tests/src/unit-*.cpp`
-  that always build with the option (coverage build); test both with and
-  without the option; tests should use different values than the
-  initial ones; put tests in the existing file instead of duplicating a SAX class;
-  static_asserts where possible.
-
-### Usually NOT commented on
-Commit messages/PR titles, licensing, local variable naming bikeshedding,
-demanding benchmarks, rejecting the overall feature idea. CI YAML gets
-comments only for merge debris.
-
-## Voice
-
-- **Mostly questions**: "Is there a reason…?", "Should this…?", "Would it make
-  sense to…?", "Why not…?", "Any reason not to…?", "Is this needed?",
-  "Intentional?"
-- **Flat statements only for clear bugs**, with the reason: "This `if` will
-  never be true. You want `||` not `&&`." "Remove the noexcept. The library
-  shouldn't try to swallow the exception."
-- **Terse when trivial**: "`emplace_back`", "typo: alse", "astyle, negation",
-  "as above", "Lost a blank line here."
-- **Concrete sketches** in plain fenced code blocks introduced with "Maybe like
-  this?" / "Something like this:" / "Untested:". Rarely ```suggestion blocks.
-- **Hedge taste**: "minor nit:", "Personally I prefer…", "I'm on the fence…",
-  "not something for which I would hold up the PR", "(If this was just moved,
-  ignore this.)", "not the fault of this PR".
-- Cite evidence: cppreference, godbolt, the standard, related issues/PRs by
-  number, existing code ("see `NLOHMANN_JSON_ABI_TAG_DIAGNOSTICS`").
-- Light humor allowed sparingly (":)", "astyle strikes again", "Ugh").
-- Praise is rare and short ("Good change.", "Looks reasonable.").
-- Defer design calls: "@nlohmann has the final say here."
-- Never invent facts about the code: if unsure, phrase it as a question rather than a claim.
-
-## Example output
-
-```
-include/nlohmann/detail/input/parser.hpp:212
-This invalidates `top`. Make it a `const` copy so it can't be read after the
-`push_back` by a future change?
-
-include/nlohmann/detail/input/parser.hpp:240
-Should this reuse the existing recursion limit rather than adding a new one?
-Different limits per operation invites them to diverge over time.
-
-include/nlohmann/detail/macro_scope.hpp:88
-If this has been externally defined, we should leave it as-is. This will break
-if someone has `#define JSON_HAS_FOO 0`.
-
-include/nlohmann/json.hpp:1534
-`throw` → `JSON_THROW`, or this can't be used by people that disable exceptions.
-
-single_include/nlohmann/json.hpp:9001
-Looks like this was edited directly. Changes go in `include/` and then
-`make amalgamate`.
-
-docs/mkdocs/docs/api/basic_json/back.md:12
-Should this list the precondition that the array is not empty?
-```

include/nlohmann/detail/input/binary_reader.hpp

@@ -4293,6 +4293,19 @@ class binary_reader
         return get_bytes(len, "string", result);
     }
 
+    /// whether @a s is valid UTF-8: ASCII eight bytes at a time, the rest
+    /// sequence by sequence (as the binary writers check strings)
+    static bool is_valid_utf8_string(const string_t& s, std::true_type /*is_byte*/) noexcept
+    {
+        return valid_utf8_string(reinterpret_cast<const unsigned char*>(s.data()), s.size());
+    }
+
+    /// whether @a s, of a character type wider than a byte, is valid UTF-8
+    static bool is_valid_utf8_string(const string_t& s, std::false_type /*is_byte*/) noexcept
+    {
+        return is_valid_utf8(s);
+    }
+
     /*!
     @brief validate a decoded text string (value or object key) against @ref error_handler
 
@@ -4312,7 +4325,16 @@ class binary_reader
     */
     bool check_string_utf8(string_t& result, const char* context)
     {
-        if (error_handler == error_handler_t::keep || is_valid_utf8(result))
+        return error_handler == error_handler_t::keep || check_string_utf8_handler(result, context);
+    }
+
+    /// @ref check_string_utf8 for an error handler other than @ref
+    /// error_handler_t::keep; out of line, so that reading strings without a
+    /// check does not pay for the code of the check
+    JSON_HEDLEY_NEVER_INLINE
+    bool check_string_utf8_handler(string_t& result, const char* context)
+    {
+        if (is_valid_utf8_string(result, std::integral_constant<bool, sizeof(typename string_t::value_type) == 1>()))
         {
             return true;
         }

include/nlohmann/detail/input/simd_scan.hpp

@@ -322,6 +322,78 @@ JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline bool vector_utf8_run(cons
     block_out = block;
     return false;
 }
+
+/*!
+@brief skip 16-byte blocks of well-formed UTF-8 at p (a character boundary)
+
+Like vector_utf8_run, but quotes, backslashes, and control characters are
+ordinary characters. Returns the start of the first block in which the check
+found an error (which may belong to a sequence begun in the block before) or
+where fewer than 16 bytes are left; a sequence may be cut at that point, so the
+caller resumes at its start, one sequence at a time.
+*/
+JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline const unsigned char* vector_utf8_valid_blocks(const unsigned char* block, const unsigned char* e) noexcept
+{
+    using lookup = utf8_lookup4<>;
+#if JSON_SCAN_NEON
+    const uint8x16_t t1h = vld1q_u8(lookup::byte_1_high.data());
+    const uint8x16_t t1l = vld1q_u8(lookup::byte_1_low.data());
+    const uint8x16_t t2h = vld1q_u8(lookup::byte_2_high.data());
+    uint8x16_t prev = vdupq_n_u8(0);
+    while (e - block >= 16)
+    {
+        const uint8x16_t in = vld1q_u8(block);
+        if (vmaxvq_u8(vorrq_u8(in, prev)) < 0x80)
+        {
+            // ASCII here, and no sequence from the block before to finish
+            prev = in;
+            block += 16;
+            continue;
+        }
+        const uint8x16_t prev1 = vextq_u8(prev, in, 15);
+        const uint8x16_t sc = vandq_u8(vandq_u8(vqtbl1q_u8(t1h, vshrq_n_u8(prev1, 4)), vqtbl1q_u8(t1l, vandq_u8(prev1, vdupq_n_u8(0x0F)))), vqtbl1q_u8(t2h, vshrq_n_u8(in, 4)));
+        const uint8x16_t must23 = vorrq_u8(vqsubq_u8(vextq_u8(prev, in, 14), vdupq_n_u8(0xE0 - 0x80)), vqsubq_u8(vextq_u8(prev, in, 13), vdupq_n_u8(0xF0 - 0x80)));
+        const uint8x16_t err = veorq_u8(vandq_u8(must23, vdupq_n_u8(0x80)), sc);
+        if (vmaxvq_u8(err) != 0)
+        {
+            break;
+        }
+        prev = in;
+        block += 16;
+    }
+#else
+    const __m128i t1h = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_1_high.data()))); // NOLINT(bugprone-casting-through-void)
+    const __m128i t1l = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_1_low.data()))); // NOLINT(bugprone-casting-through-void)
+    const __m128i t2h = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_2_high.data()))); // NOLINT(bugprone-casting-through-void)
+    const __m128i nibble = _mm_set1_epi8(0x0F);
+    const __m128i zero = _mm_setzero_si128();
+    __m128i prev = zero;
+    while (e - block >= 16)
+    {
+        const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void)
+        if (_mm_movemask_epi8(_mm_or_si128(in, prev)) == 0)
+        {
+            prev = in;
+            block += 16;
+            continue;
+        }
+        const __m128i prev1 = _mm_alignr_epi8(in, prev, 15);
+        const __m128i sc = _mm_and_si128(_mm_and_si128(_mm_shuffle_epi8(t1h, _mm_and_si128(_mm_srli_epi16(prev1, 4), nibble)),
+                                         _mm_shuffle_epi8(t1l, _mm_and_si128(prev1, nibble))),
+                                         _mm_shuffle_epi8(t2h, _mm_and_si128(_mm_srli_epi16(in, 4), nibble)));
+        const __m128i must23 = _mm_or_si128(_mm_subs_epu8(_mm_alignr_epi8(in, prev, 14), _mm_set1_epi8(0xE0 - 0x80)),
+                                            _mm_subs_epu8(_mm_alignr_epi8(in, prev, 13), _mm_set1_epi8(0xF0 - 0x80)));
+        const __m128i err = _mm_xor_si128(_mm_and_si128(must23, _mm_set1_epi8(static_cast<char>(-128))), sc);
+        if (_mm_movemask_epi8(_mm_cmpeq_epi8(err, zero)) != 0xFFFF)
+        {
+            break;
+        }
+        prev = in;
+        block += 16;
+    }
+#endif
+    return block;
+}
 #endif
 
 }  // namespace detail

include/nlohmann/detail/input/string_scan.hpp

@@ -236,6 +236,64 @@ inline std::size_t valid_utf8_prefix(const unsigned char* data, std::size_t n) n
     return pos;
 }
 
+/*!
+@brief whether [data, data+n) is valid UTF-8, for checking whole strings
+
+The same answer as valid_utf8_prefix(data, n) == n. Short ASCII strings, the
+most common keys and values, are decided in one test, and long strings are
+checked 16 bytes at a time where the vector check is available.
+*/
+inline bool valid_utf8_string(const unsigned char* data, std::size_t n) noexcept
+{
+    constexpr std::uint64_t high = 0x8080808080808080ull;
+    if (n <= 16)
+    {
+        // two overlapping words cover 8 to 16 bytes, one partial word fewer
+        std::uint64_t first = 0;
+        std::uint64_t last = 0;
+        if (n >= 8)
+        {
+            std::memcpy(&first, data, sizeof(first));
+            std::memcpy(&last, data + n - 8, sizeof(last));
+        }
+        else if (n != 0)
+        {
+            std::memcpy(&first, data, n);
+        }
+        if (((first | last) & high) == 0)
+        {
+            return true;
+        }
+    }
+#if JSON_SCAN_VECTOR_UTF8
+    else if (n >= 32
+#if JSON_SCAN_SSSE3_DISPATCH
+             && cpu_has_ssse3()
+#endif
+            )
+    {
+        const unsigned char* block = vector_utf8_valid_blocks(data, data + n);
+        // the rest from the start of a sequence that may be cut at block
+        for (int i = 1; i <= 3 && block - i >= data; ++i)
+        {
+            const unsigned char c = block[-i];
+            if (c < 0x80)
+            {
+                break;
+            }
+            if (c >= 0xC0)
+            {
+                block -= i;
+                break;
+            }
+        }
+        const auto checked = static_cast<std::size_t>(block - data);
+        return checked + valid_utf8_prefix(block, n - checked) == n;
+    }
+#endif
+    return valid_utf8_prefix(data, n) == n;
+}
+
 #if JSON_SCAN_VECTOR_UTF8
 /// the end of scan_string_vector from block, where the vector loop stopped
 /// (ill-formed UTF-8, or fewer than 16 bytes left): one byte or sequence at a

include/nlohmann/detail/output/binary_writer.hpp

@@ -3080,6 +3080,25 @@ class binary_writer
         }
     }
 
+    /// write the head and the characters of @a value, at most
+    /// max_text_in_reservation of them, in one reservation
+    JSON_HEDLEY_ALWAYS_INLINE void write_cbor_string_reserved(const string_t& value)
+    {
+        CharType* const out = oa.reserve(value.size() + 2);
+        CharType* p = out;
+        if (value.size() <= 0x17)
+        {
+            *p++ = to_char_type(static_cast<std::uint8_t>(0x60 + value.size()));
+        }
+        else
+        {
+            *p++ = to_char_type(0x78);
+            *p++ = to_char_type(static_cast<std::uint8_t>(value.size()));
+        }
+        copy_characters(p, reinterpret_cast<const CharType*>(value.data()), value.size());
+        oa.commit(static_cast<std::size_t>(p - out) + value.size());
+    }
+
     /*!
     @brief write a CBOR text string
 
@@ -3101,19 +3120,7 @@ class binary_writer
             // the string is written as it is: head and characters in one reservation
             if (JSON_HEDLEY_LIKELY(value.size() <= max_text_in_reservation))
             {
-                CharType* const out = oa.reserve(value.size() + 2);
-                CharType* p = out;
-                if (value.size() <= 0x17)
-                {
-                    *p++ = to_char_type(static_cast<std::uint8_t>(0x60 + value.size()));
-                }
-                else
-                {
-                    *p++ = to_char_type(0x78);
-                    *p++ = to_char_type(static_cast<std::uint8_t>(value.size()));
-                }
-                copy_characters(p, reinterpret_cast<const CharType*>(value.data()), value.size());
-                oa.commit(static_cast<std::size_t>(p - out) + value.size());
+                write_cbor_string_reserved(value);
                 return;
             }
         }
@@ -3127,6 +3134,13 @@ class binary_writer
     JSON_HEDLEY_NEVER_INLINE
     void write_cbor_checked_string(const string_t& value, const BasicJsonType& context)
     {
+        if (error_handler == error_handler_t::strict && value.size() <= max_text_in_reservation)
+        {
+            check_utf8(value, context);
+            write_cbor_string_reserved(value);
+            return;
+        }
+
         string_t storage;
         const string_t& sanitized = sanitize_utf8_for_write(value, context, storage);
 
@@ -3141,6 +3155,27 @@ class binary_writer
     // MsgPack //
     /////////////
 
+    /// write the head and the characters of @a value, at most
+    /// max_text_in_reservation of them, in one reservation
+    JSON_HEDLEY_ALWAYS_INLINE void write_msgpack_string_reserved(const string_t& value)
+    {
+        CharType* const out = oa.reserve(value.size() + 2);
+        CharType* p = out;
+        if (value.size() <= 31)
+        {
+            // fixstr
+            *p++ = to_char_type(static_cast<std::uint8_t>(0xA0 | value.size()));
+        }
+        else
+        {
+            // str 8
+            *p++ = to_char_type(0xD9);
+            *p++ = to_char_type(static_cast<std::uint8_t>(value.size()));
+        }
+        copy_characters(p, reinterpret_cast<const CharType*>(value.data()), value.size());
+        oa.commit(static_cast<std::size_t>(p - out) + value.size());
+    }
+
     /*!
     @brief write a MessagePack str
 
@@ -3158,21 +3193,7 @@ class binary_writer
             // the string is written as it is: head and characters in one reservation
             if (JSON_HEDLEY_LIKELY(value.size() <= max_text_in_reservation))
             {
-                CharType* const out = oa.reserve(value.size() + 2);
-                CharType* p = out;
-                if (value.size() <= 31)
-                {
-                    // fixstr
-                    *p++ = to_char_type(static_cast<std::uint8_t>(0xA0 | value.size()));
-                }
-                else
-                {
-                    // str 8
-                    *p++ = to_char_type(0xD9);
-                    *p++ = to_char_type(static_cast<std::uint8_t>(value.size()));
-                }
-                copy_characters(p, reinterpret_cast<const CharType*>(value.data()), value.size());
-                oa.commit(static_cast<std::size_t>(p - out) + value.size());
+                write_msgpack_string_reserved(value);
                 return;
             }
         }
@@ -3185,6 +3206,13 @@ class binary_writer
     JSON_HEDLEY_NEVER_INLINE
     void write_msgpack_checked_string(const string_t& value, const BasicJsonType& context)
     {
+        if (error_handler == error_handler_t::strict && value.size() <= max_text_in_reservation)
+        {
+            check_utf8(value, context);
+            write_msgpack_string_reserved(value);
+            return;
+        }
+
         string_t storage;
         const string_t& sanitized = sanitize_utf8_for_write(value, context, storage);
 
@@ -4794,10 +4822,9 @@ class binary_writer
     static void check_utf8(const string_t& s, const BasicJsonType& context)
     {
         const auto* data = reinterpret_cast<const unsigned char*>(s.data());
-        const std::size_t valid = valid_utf8_prefix(data, s.size());
-        if (JSON_HEDLEY_UNLIKELY(valid != s.size()))
+        if (JSON_HEDLEY_UNLIKELY(!valid_utf8_string(data, s.size())))
         {
-            throw_invalid_utf8(data, valid, context);
+            throw_invalid_utf8(data, valid_utf8_prefix(data, s.size()), context);
         }
     }
 

single_include/nlohmann/json.hpp

@@ -12377,6 +12377,78 @@ JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline bool vector_utf8_run(cons
     block_out = block;
     return false;
 }
+
+/*!
+@brief skip 16-byte blocks of well-formed UTF-8 at p (a character boundary)
+
+Like vector_utf8_run, but quotes, backslashes, and control characters are
+ordinary characters. Returns the start of the first block in which the check
+found an error (which may belong to a sequence begun in the block before) or
+where fewer than 16 bytes are left; a sequence may be cut at that point, so the
+caller resumes at its start, one sequence at a time.
+*/
+JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline const unsigned char* vector_utf8_valid_blocks(const unsigned char* block, const unsigned char* e) noexcept
+{
+    using lookup = utf8_lookup4<>;
+#if JSON_SCAN_NEON
+    const uint8x16_t t1h = vld1q_u8(lookup::byte_1_high.data());
+    const uint8x16_t t1l = vld1q_u8(lookup::byte_1_low.data());
+    const uint8x16_t t2h = vld1q_u8(lookup::byte_2_high.data());
+    uint8x16_t prev = vdupq_n_u8(0);
+    while (e - block >= 16)
+    {
+        const uint8x16_t in = vld1q_u8(block);
+        if (vmaxvq_u8(vorrq_u8(in, prev)) < 0x80)
+        {
+            // ASCII here, and no sequence from the block before to finish
+            prev = in;
+            block += 16;
+            continue;
+        }
+        const uint8x16_t prev1 = vextq_u8(prev, in, 15);
+        const uint8x16_t sc = vandq_u8(vandq_u8(vqtbl1q_u8(t1h, vshrq_n_u8(prev1, 4)), vqtbl1q_u8(t1l, vandq_u8(prev1, vdupq_n_u8(0x0F)))), vqtbl1q_u8(t2h, vshrq_n_u8(in, 4)));
+        const uint8x16_t must23 = vorrq_u8(vqsubq_u8(vextq_u8(prev, in, 14), vdupq_n_u8(0xE0 - 0x80)), vqsubq_u8(vextq_u8(prev, in, 13), vdupq_n_u8(0xF0 - 0x80)));
+        const uint8x16_t err = veorq_u8(vandq_u8(must23, vdupq_n_u8(0x80)), sc);
+        if (vmaxvq_u8(err) != 0)
+        {
+            break;
+        }
+        prev = in;
+        block += 16;
+    }
+#else
+    const __m128i t1h = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_1_high.data()))); // NOLINT(bugprone-casting-through-void)
+    const __m128i t1l = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_1_low.data()))); // NOLINT(bugprone-casting-through-void)
+    const __m128i t2h = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_2_high.data()))); // NOLINT(bugprone-casting-through-void)
+    const __m128i nibble = _mm_set1_epi8(0x0F);
+    const __m128i zero = _mm_setzero_si128();
+    __m128i prev = zero;
+    while (e - block >= 16)
+    {
+        const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void)
+        if (_mm_movemask_epi8(_mm_or_si128(in, prev)) == 0)
+        {
+            prev = in;
+            block += 16;
+            continue;
+        }
+        const __m128i prev1 = _mm_alignr_epi8(in, prev, 15);
+        const __m128i sc = _mm_and_si128(_mm_and_si128(_mm_shuffle_epi8(t1h, _mm_and_si128(_mm_srli_epi16(prev1, 4), nibble)),
+                                         _mm_shuffle_epi8(t1l, _mm_and_si128(prev1, nibble))),
+                                         _mm_shuffle_epi8(t2h, _mm_and_si128(_mm_srli_epi16(in, 4), nibble)));
+        const __m128i must23 = _mm_or_si128(_mm_subs_epu8(_mm_alignr_epi8(in, prev, 14), _mm_set1_epi8(0xE0 - 0x80)),
+                                            _mm_subs_epu8(_mm_alignr_epi8(in, prev, 13), _mm_set1_epi8(0xF0 - 0x80)));
+        const __m128i err = _mm_xor_si128(_mm_and_si128(must23, _mm_set1_epi8(static_cast<char>(-128))), sc);
+        if (_mm_movemask_epi8(_mm_cmpeq_epi8(err, zero)) != 0xFFFF)
+        {
+            break;
+        }
+        prev = in;
+        block += 16;
+    }
+#endif
+    return block;
+}
 #endif
 
 }  // namespace detail
@@ -12604,6 +12676,64 @@ inline std::size_t valid_utf8_prefix(const unsigned char* data, std::size_t n) n
     return pos;
 }
 
+/*!
+@brief whether [data, data+n) is valid UTF-8, for checking whole strings
+
+The same answer as valid_utf8_prefix(data, n) == n. Short ASCII strings, the
+most common keys and values, are decided in one test, and long strings are
+checked 16 bytes at a time where the vector check is available.
+*/
+inline bool valid_utf8_string(const unsigned char* data, std::size_t n) noexcept
+{
+    constexpr std::uint64_t high = 0x8080808080808080ull;
+    if (n <= 16)
+    {
+        // two overlapping words cover 8 to 16 bytes, one partial word fewer
+        std::uint64_t first = 0;
+        std::uint64_t last = 0;
+        if (n >= 8)
+        {
+            std::memcpy(&first, data, sizeof(first));
+            std::memcpy(&last, data + n - 8, sizeof(last));
+        }
+        else if (n != 0)
+        {
+            std::memcpy(&first, data, n);
+        }
+        if (((first | last) & high) == 0)
+        {
+            return true;
+        }
+    }
+#if JSON_SCAN_VECTOR_UTF8
+    else if (n >= 32
+#if JSON_SCAN_SSSE3_DISPATCH
+             && cpu_has_ssse3()
+#endif
+            )
+    {
+        const unsigned char* block = vector_utf8_valid_blocks(data, data + n);
+        // the rest from the start of a sequence that may be cut at block
+        for (int i = 1; i <= 3 && block - i >= data; ++i)
+        {
+            const unsigned char c = block[-i];
+            if (c < 0x80)
+            {
+                break;
+            }
+            if (c >= 0xC0)
+            {
+                block -= i;
+                break;
+            }
+        }
+        const auto checked = static_cast<std::size_t>(block - data);
+        return checked + valid_utf8_prefix(block, n - checked) == n;
+    }
+#endif
+    return valid_utf8_prefix(data, n) == n;
+}
+
 #if JSON_SCAN_VECTOR_UTF8
 /// the end of scan_string_vector from block, where the vector loop stopped
 /// (ill-formed UTF-8, or fewer than 16 bytes left): one byte or sequence at a
@@ -22069,6 +22199,19 @@ class binary_reader
         return get_bytes(len, "string", result);
     }
 
+    /// whether @a s is valid UTF-8: ASCII eight bytes at a time, the rest
+    /// sequence by sequence (as the binary writers check strings)
+    static bool is_valid_utf8_string(const string_t& s, std::true_type /*is_byte*/) noexcept
+    {
+        return valid_utf8_string(reinterpret_cast<const unsigned char*>(s.data()), s.size());
+    }
+
+    /// whether @a s, of a character type wider than a byte, is valid UTF-8
+    static bool is_valid_utf8_string(const string_t& s, std::false_type /*is_byte*/) noexcept
+    {
+        return is_valid_utf8(s);
+    }
+
     /*!
     @brief validate a decoded text string (value or object key) against @ref error_handler
 
@@ -22088,7 +22231,16 @@ class binary_reader
     */
     bool check_string_utf8(string_t& result, const char* context)
     {
-        if (error_handler == error_handler_t::keep || is_valid_utf8(result))
+        return error_handler == error_handler_t::keep || check_string_utf8_handler(result, context);
+    }
+
+    /// @ref check_string_utf8 for an error handler other than @ref
+    /// error_handler_t::keep; out of line, so that reading strings without a
+    /// check does not pay for the code of the check
+    JSON_HEDLEY_NEVER_INLINE
+    bool check_string_utf8_handler(string_t& result, const char* context)
+    {
+        if (is_valid_utf8_string(result, std::integral_constant<bool, sizeof(typename string_t::value_type) == 1>()))
         {
             return true;
         }
@@ -30306,6 +30458,25 @@ class binary_writer
         }
     }
 
+    /// write the head and the characters of @a value, at most
+    /// max_text_in_reservation of them, in one reservation
+    JSON_HEDLEY_ALWAYS_INLINE void write_cbor_string_reserved(const string_t& value)
+    {
+        CharType* const out = oa.reserve(value.size() + 2);
+        CharType* p = out;
+        if (value.size() <= 0x17)
+        {
+            *p++ = to_char_type(static_cast<std::uint8_t>(0x60 + value.size()));
+        }
+        else
+        {
+            *p++ = to_char_type(0x78);
+            *p++ = to_char_type(static_cast<std::uint8_t>(value.size()));
+        }
+        copy_characters(p, reinterpret_cast<const CharType*>(value.data()), value.size());
+        oa.commit(static_cast<std::size_t>(p - out) + value.size());
+    }
+
     /*!
     @brief write a CBOR text string
 
@@ -30327,19 +30498,7 @@ class binary_writer
             // the string is written as it is: head and characters in one reservation
             if (JSON_HEDLEY_LIKELY(value.size() <= max_text_in_reservation))
             {
-                CharType* const out = oa.reserve(value.size() + 2);
-                CharType* p = out;
-                if (value.size() <= 0x17)
-                {
-                    *p++ = to_char_type(static_cast<std::uint8_t>(0x60 + value.size()));
-                }
-                else
-                {
-                    *p++ = to_char_type(0x78);
-                    *p++ = to_char_type(static_cast<std::uint8_t>(value.size()));
-                }
-                copy_characters(p, reinterpret_cast<const CharType*>(value.data()), value.size());
-                oa.commit(static_cast<std::size_t>(p - out) + value.size());
+                write_cbor_string_reserved(value);
                 return;
             }
         }
@@ -30353,6 +30512,13 @@ class binary_writer
     JSON_HEDLEY_NEVER_INLINE
     void write_cbor_checked_string(const string_t& value, const BasicJsonType& context)
     {
+        if (error_handler == error_handler_t::strict && value.size() <= max_text_in_reservation)
+        {
+            check_utf8(value, context);
+            write_cbor_string_reserved(value);
+            return;
+        }
+
         string_t storage;
         const string_t& sanitized = sanitize_utf8_for_write(value, context, storage);
 
@@ -30367,6 +30533,27 @@ class binary_writer
     // MsgPack //
     /////////////
 
+    /// write the head and the characters of @a value, at most
+    /// max_text_in_reservation of them, in one reservation
+    JSON_HEDLEY_ALWAYS_INLINE void write_msgpack_string_reserved(const string_t& value)
+    {
+        CharType* const out = oa.reserve(value.size() + 2);
+        CharType* p = out;
+        if (value.size() <= 31)
+        {
+            // fixstr
+            *p++ = to_char_type(static_cast<std::uint8_t>(0xA0 | value.size()));
+        }
+        else
+        {
+            // str 8
+            *p++ = to_char_type(0xD9);
+            *p++ = to_char_type(static_cast<std::uint8_t>(value.size()));
+        }
+        copy_characters(p, reinterpret_cast<const CharType*>(value.data()), value.size());
+        oa.commit(static_cast<std::size_t>(p - out) + value.size());
+    }
+
     /*!
     @brief write a MessagePack str
 
@@ -30384,21 +30571,7 @@ class binary_writer
             // the string is written as it is: head and characters in one reservation
             if (JSON_HEDLEY_LIKELY(value.size() <= max_text_in_reservation))
             {
-                CharType* const out = oa.reserve(value.size() + 2);
-                CharType* p = out;
-                if (value.size() <= 31)
-                {
-                    // fixstr
-                    *p++ = to_char_type(static_cast<std::uint8_t>(0xA0 | value.size()));
-                }
-                else
-                {
-                    // str 8
-                    *p++ = to_char_type(0xD9);
-                    *p++ = to_char_type(static_cast<std::uint8_t>(value.size()));
-                }
-                copy_characters(p, reinterpret_cast<const CharType*>(value.data()), value.size());
-                oa.commit(static_cast<std::size_t>(p - out) + value.size());
+                write_msgpack_string_reserved(value);
                 return;
             }
         }
@@ -30411,6 +30584,13 @@ class binary_writer
     JSON_HEDLEY_NEVER_INLINE
     void write_msgpack_checked_string(const string_t& value, const BasicJsonType& context)
     {
+        if (error_handler == error_handler_t::strict && value.size() <= max_text_in_reservation)
+        {
+            check_utf8(value, context);
+            write_msgpack_string_reserved(value);
+            return;
+        }
+
         string_t storage;
         const string_t& sanitized = sanitize_utf8_for_write(value, context, storage);
 
@@ -32020,10 +32200,9 @@ class binary_writer
     static void check_utf8(const string_t& s, const BasicJsonType& context)
     {
         const auto* data = reinterpret_cast<const unsigned char*>(s.data());
-        const std::size_t valid = valid_utf8_prefix(data, s.size());
-        if (JSON_HEDLEY_UNLIKELY(valid != s.size()))
+        if (JSON_HEDLEY_UNLIKELY(!valid_utf8_string(data, s.size())))
         {
-            throw_invalid_utf8(data, valid, context);
+            throw_invalid_utf8(data, valid_utf8_prefix(data, s.size()), context);
         }
     }
 

tests/src/unit-guard-page.cpp

@@ -403,6 +403,7 @@ TEST_CASE("guard page")
                     CHECK(nlohmann::detail::find_string_special(p, n) <= n);
                     CHECK(nlohmann::detail::find_ascii_copyable_run(p, n) <= n);
                     CHECK(nlohmann::detail::valid_utf8_prefix(p, n) <= n);
+                    CHECK(nlohmann::detail::valid_utf8_string(p, n) == (nlohmann::detail::valid_utf8_prefix(p, n) == n));
 #if defined(JSON_USE_SIMDUTF) && defined(JSON_HAS_CPP_17)
                     CHECK(nlohmann::detail::find_string_delimiter(p, n) <= n);
 #endif

tests/src/unit-unicode.cpp

@@ -13,6 +13,7 @@ using nlohmann::json;
 
 #include <algorithm>
 #include <fstream>
+#include <random>
 #include <string>
 #include "make_test_data_available.hpp"
 #include "test_utils.hpp"
@@ -1631,3 +1632,40 @@ TEST_CASE("Unicode RFC 3629")
 }
 
 DOCTEST_CLANG_SUPPRESS_WARNING_POP
+
+TEST_CASE("UTF-8 check of whole strings")
+{
+    // valid_utf8_string() decides short strings with a word test and long ones
+    // 16 bytes at a time where a vector check is available; it must agree with
+    // the byte-wise check for every mix of ASCII, sequences of each length,
+    // ill-formed and truncated sequences, at every length and offset
+    const char* const pieces[] =
+    {
+        "a", "\x7F", "\xC3\xA9", "\xE2\x82\xAC", "\xE0\xA0\x80", "\xEF\xBF\xBF", "\xF0\x9F\x98\x80", "\xF4\x8F\xBF\xBF",
+        "\x80", "\xC0\xAF", "\xC3", "\xE2\x82", "\xED\xA0\x80", "\xF0\x9F", "\xF4\x90\x80\x80", "\xFF"
+    };
+    // std::mt19937's sequence is fixed by the standard; no distribution is used
+    std::mt19937 rng(5);
+    std::size_t valid = 0;
+    for (int i = 0; i < 100000; ++i)
+    {
+        std::string s;
+        const std::size_t length = rng() % 100;
+        const std::uint32_t ascii = rng() % 101;
+        while (s.size() < length)
+        {
+            // mostly well-formed pieces (the first eight), so that long valid
+            // strings are common
+            s += (rng() % 100 < ascii) ? std::string("x") : std::string(pieces[rng() % (rng() % 8 == 0 ? 16 : 8)]);
+        }
+        const auto* data = reinterpret_cast<const unsigned char*>(s.data());
+        const bool expected = nlohmann::detail::is_valid_utf8(s);
+        valid += expected ? 1 : 0;
+        CAPTURE(s)
+        CHECK(nlohmann::detail::valid_utf8_string(data, s.size()) == expected);
+        CHECK((nlohmann::detail::valid_utf8_prefix(data, s.size()) == s.size()) == expected);
+    }
+    // both outcomes are common enough to be tested
+    CHECK(valid > 20000);
+    CHECK(valid < 80000);
+}