Compare commits

..
Author SHA1 Message Date
Niels Lohmann 9bee955345 perf: check whole UTF-8 strings without waiting for each block's result
valid_utf8_string() checks long strings 16 bytes at a time in
vector_utf8_valid_blocks(), which left its loop at the first block with
an error. Clang compiles "if (error) break; block += 16;" into a
conditional select on the block pointer, so each block waited for the
complete check of the block before: 28 cycles for 16 bytes on Apple M1.

The errors of all blocks are now collected and tested once at the end.
ASCII blocks are skipped in a loop of their own that needs none of the
lookup tables, and from the first block that is not ASCII, eight blocks
are checked without testing for ASCII in between, which was a
mispredicted branch for text with a few non-ASCII characters. If any
block had an error, the caller finds it one sequence at a time from the
start of the string, as before from the block with the error.

The check is used by the binary writers and readers with an error
handler other than keep, and always by BON8. On Apple M1 (Apple Clang),
it needs 0.27 instead of 1.78 cycles per byte for non-ASCII text, and
to_msgpack() with error_handler_t::strict is 24 to 43 % faster for such
text. With GCC 15 on x86-64, where the loop was not affected, the check
itself is 13 to 35 % faster.

A new test puts well-formed and ill-formed sequences at every offset of
strings that are longer than eight blocks.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-11-10 10:07:56 +01:00
32 changed files with 3682 additions and 751 deletions

No files matched your search

-7
View File
@@ -60,10 +60,3 @@ updates:
interval: daily interval: daily
cooldown: cooldown:
default-days: 7 default-days: 7
ignore:
# semgrep pins these packages to exact or compatible-release versions (see its METADATA);
# bumping them on their own makes the Semgrep job fail with ResolutionImpossible, so
# they only move together with a semgrep release
- dependency-name: mcp
- dependency-name: boltons
- dependency-name: "opentelemetry-*"
+2 -2
View File
@@ -38,7 +38,7 @@ jobs:
# Initializes the CodeQL tools for scanning. # Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL - name: Initialize CodeQL
uses: github/codeql-action/init@1fc58d04a4d4374f921bdac573c3811c240f9c26 # v4.39.2 uses: github/codeql-action/init@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
with: with:
languages: c-cpp languages: c-cpp
# Analyze the sources without compiling them. Autobuild compiled the # Analyze the sources without compiling them. Autobuild compiled the
@@ -47,4 +47,4 @@ jobs:
build-mode: none build-mode: none
- name: Perform CodeQL Analysis - name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@1fc58d04a4d4374f921bdac573c3811c240f9c26 # v4.39.2 uses: github/codeql-action/analyze@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
+1 -1
View File
@@ -48,6 +48,6 @@ jobs:
output: 'flawfinder_results.sarif' output: 'flawfinder_results.sarif'
- name: Upload analysis results to GitHub Security tab - name: Upload analysis results to GitHub Security tab
uses: github/codeql-action/upload-sarif@1fc58d04a4d4374f921bdac573c3811c240f9c26 # v4.39.2 uses: github/codeql-action/upload-sarif@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
with: with:
sarif_file: ${{github.workspace}}/flawfinder_results.sarif sarif_file: ${{github.workspace}}/flawfinder_results.sarif
+1 -1
View File
@@ -80,6 +80,6 @@ jobs:
# Upload the results to GitHub's code scanning dashboard. # Upload the results to GitHub's code scanning dashboard.
- name: "Upload to code-scanning" - name: "Upload to code-scanning"
uses: github/codeql-action/upload-sarif@1fc58d04a4d4374f921bdac573c3811c240f9c26 # v4.39.2 uses: github/codeql-action/upload-sarif@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
with: with:
sarif_file: results.sarif sarif_file: results.sarif
+1 -1
View File
@@ -65,7 +65,7 @@ jobs:
# Upload SARIF file generated in previous step # Upload SARIF file generated in previous step
- name: Upload SARIF file - name: Upload SARIF file
uses: github/codeql-action/upload-sarif@1fc58d04a4d4374f921bdac573c3811c240f9c26 # v4.39.2 uses: github/codeql-action/upload-sarif@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
with: with:
sarif_file: semgrep.sarif sarif_file: semgrep.sarif
if: always() if: always()
+3272
View File
File diff suppressed because it is too large. Load diff
+18 -1
View File
@@ -1,4 +1,4 @@
.PHONY: all amalgamate check-amalgamation check_build_options pretty install_astyle clean release update_hedley update_hedley_undef BUILD.bazel natvis macro_builder_check run_benchmarks pvs_studio serve_header reuse spdx .PHONY: all amalgamate check-amalgamation check_build_options pretty install_astyle clean ChangeLog.md release update_hedley update_hedley_undef BUILD.bazel natvis macro_builder_check run_benchmarks pvs_studio serve_header reuse spdx
########################################################################## ##########################################################################
# configuration # configuration
@@ -40,6 +40,7 @@ MACRO_SCOPE_HPP=include/nlohmann/detail/macro_scope.hpp
all: all:
@echo "amalgamate - amalgamate files single_include/nlohmann/json{,_fwd,_literals,_view}.hpp from the include/nlohmann sources" @echo "amalgamate - amalgamate files single_include/nlohmann/json{,_fwd,_literals,_view}.hpp from the include/nlohmann sources"
@echo "BUILD.bazel - regenerate the Bazel BUILD file from the include/nlohmann sources" @echo "BUILD.bazel - regenerate the Bazel BUILD file from the include/nlohmann sources"
@echo "ChangeLog.md - generate ChangeLog file"
@echo "check-amalgamation - check whether sources have been amalgamated and BUILD.bazel is up to date" @echo "check-amalgamation - check whether sources have been amalgamated and BUILD.bazel is up to date"
@echo "check_build_options - check that the Meson build and the pkg-config files offer the options of the CMake target" @echo "check_build_options - check that the Meson build and the pkg-config files offer the options of the CMake target"
@echo "clean - remove built files" @echo "clean - remove built files"
@@ -174,6 +175,22 @@ check-amalgamation:
BUILD.bazel: BUILD.bazel:
cmake -P cmake/scripts/gen_bazel_build_file.cmake cmake -P cmake/scripts/gen_bazel_build_file.cmake
##########################################################################
# ChangeLog
##########################################################################
# Create a ChangeLog based on the git log using the GitHub Changelog Generator
# (<https://github.com/github-changelog-generator/github-changelog-generator>).
# variable to control the diffs between the last released version and the current repository state
NEXT_VERSION ?= "unreleased"
ChangeLog.md:
github_changelog_generator -o ChangeLog.md --user nlohmann --project json --simple-list --release-url https://github.com/nlohmann/json/releases/tag/%s --future-release $(NEXT_VERSION)
$(SED) -i 's|https://github.com/nlohmann/json/releases/tag/HEAD|https://github.com/nlohmann/json/tree/HEAD|' ChangeLog.md
$(SED) -i '2i All notable changes to this project will be documented in this file. This project adheres to [Semantic Versioning](http://semver.org/).' ChangeLog.md
########################################################################## ##########################################################################
# Release files # Release files
########################################################################## ##########################################################################
+9 -34
View File
@@ -1,9 +1,4 @@
<a href="https://github.com/nlohmann/json/releases"> [![JSON for Modern C++](docs/mkdocs/docs/images/json.gif)](https://github.com/nlohmann/json/releases)
<picture>
<source media="(prefers-color-scheme: dark)" srcset="docs/mkdocs/docs/images/json-dark.svg">
<img src="docs/mkdocs/docs/images/json-light.svg" alt="JSON for Modern C++: animated feature slideshow">
</picture>
</a>
[![Build Status](https://ci.appveyor.com/api/projects/status/1acb366xfyg3qybk/branch/develop?svg=true)](https://ci.appveyor.com/project/nlohmann/json) [![Build Status](https://ci.appveyor.com/api/projects/status/1acb366xfyg3qybk/branch/develop?svg=true)](https://ci.appveyor.com/project/nlohmann/json)
[![Ubuntu](https://github.com/nlohmann/json/workflows/Ubuntu/badge.svg)](https://github.com/nlohmann/json/actions?query=workflow%3AUbuntu) [![Ubuntu](https://github.com/nlohmann/json/workflows/Ubuntu/badge.svg)](https://github.com/nlohmann/json/actions?query=workflow%3AUbuntu)
@@ -44,7 +39,6 @@
- [Conversions to/from arbitrary types](#arbitrary-types-conversions) - [Conversions to/from arbitrary types](#arbitrary-types-conversions)
- [Specializing enum conversion](#specializing-enum-conversion) - [Specializing enum conversion](#specializing-enum-conversion)
- [Binary formats (BSON, CBOR, MessagePack, UBJSON, BJData, and BON8)](#binary-formats-bson-cbor-messagepack-ubjson-bjdata-and-bon8) - [Binary formats (BSON, CBOR, MessagePack, UBJSON, BJData, and BON8)](#binary-formats-bson-cbor-messagepack-ubjson-bjdata-and-bon8)
- [Zero-copy views](#zero-copy-views)
- [Customers](#customers) - [Customers](#customers)
- [Ecosystem](#ecosystem) - [Ecosystem](#ecosystem)
- [Supported compilers](#supported-compilers) - [Supported compilers](#supported-compilers)
@@ -74,7 +68,7 @@ Other aspects were not so important to us:
- **Memory efficiency**. Each JSON object has an overhead of one pointer (the maximal size of a union) and one enumeration element (1 byte). The default generalization uses the following C++ data types: `std::string` for strings, `int64_t`, `uint64_t` or `double` for numbers, `std::map` for objects, `std::vector` for arrays, and `bool` for Booleans. However, you can template the generalized class `basic_json` to your needs. - **Memory efficiency**. Each JSON object has an overhead of one pointer (the maximal size of a union) and one enumeration element (1 byte). The default generalization uses the following C++ data types: `std::string` for strings, `int64_t`, `uint64_t` or `double` for numbers, `std::map` for objects, `std::vector` for arrays, and `bool` for Booleans. However, you can template the generalized class `basic_json` to your needs.
- **Speed**. There are certainly [faster JSON libraries](https://github.com/miloyip/nativejson-benchmark#parsing-time) out there. However, if your goal is to speed up your development by adding JSON support with a single header, then this library is the way to go. If you know how to use a `std::vector` or `std::map`, you are already set. The [performance](https://json.nlohmann.me/features/performance/) page shows how to get the most out of the library. - **Speed**. There are certainly [faster JSON libraries](https://github.com/miloyip/nativejson-benchmark#parsing-time) out there. However, if your goal is to speed up your development by adding JSON support with a single header, then this library is the way to go. If you know how to use a `std::vector` or `std::map`, you are already set.
See the [contribution guidelines](https://github.com/nlohmann/json/blob/develop/.github/CONTRIBUTING.md#please-dont) for more information. See the [contribution guidelines](https://github.com/nlohmann/json/blob/develop/.github/CONTRIBUTING.md#please-dont) for more information.
@@ -1058,7 +1052,7 @@ struct less_than_32_serializer {
Be **very** careful when reimplementing your serializer, you can stack overflow if you don't pay attention: Be **very** careful when reimplementing your serializer, you can stack overflow if you don't pay attention:
```cpp ```cpp
template <typename T, typename = void> template <typename T, void>
struct bad_serializer struct bad_serializer
{ {
template <typename BasicJsonType> template <typename BasicJsonType>
@@ -1069,10 +1063,10 @@ struct bad_serializer
} }
template <typename BasicJsonType> template <typename BasicJsonType>
static void from_json(const BasicJsonType& j, T& value) { static void to_json(const BasicJsonType& j, T& value) {
// this calls BasicJsonType::json_serializer<T>::from_json(j, value) // this calls BasicJsonType::json_serializer<T>::from_json(j, value)
// if BasicJsonType::json_serializer == bad_serializer ... oops! // if BasicJsonType::json_serializer == bad_serializer ... oops!
value = j.template get<T>(); // oops! value = j.get<T>(); // oops!
} }
}; };
``` ```
@@ -1775,7 +1769,7 @@ I deeply appreciate the help of the following people.
330. [AniketDhemare](https://github.com/AniketDhemare) fixed a version in the documentation. 330. [AniketDhemare](https://github.com/AniketDhemare) fixed a version in the documentation.
331. [Philip Müller](https://github.com/philip-paul-mueller) fixed an example. 331. [Philip Müller](https://github.com/philip-paul-mueller) fixed an example.
332. [Leila Shcheglova](https://github.com/LeilaShcheglova) fixed a warning in a test. 332. [Leila Shcheglova](https://github.com/LeilaShcheglova) fixed a warning in a test.
333. [Alex Prabhat Bara](https://github.com/alexprabhat99) fixed a function name in the documentation and avoided temporary `basic_json` values for CBOR and MessagePack object keys. 333. [Alex Prabhat Bara](https://github.com/alexprabhat99) fixed a function name in the documentation.
334. [laterlaugh](https://github.com/laterlaugh) fixed some typos. 334. [laterlaugh](https://github.com/laterlaugh) fixed some typos.
335. [Yuanhao Jia](https://github.com/MrJia1997) fixed the GDB pretty printer. 335. [Yuanhao Jia](https://github.com/MrJia1997) fixed the GDB pretty printer.
336. [Fallen_Breath](https://github.com/Fallen-Breath) fixed an example for JSON Pointer. 336. [Fallen_Breath](https://github.com/Fallen-Breath) fixed an example for JSON Pointer.
@@ -1853,7 +1847,7 @@ I deeply appreciate the help of the following people.
408. [Luke Banicevic](https://github.com/banaboi) fixed corrupt BSON output for lengths exceeding `INT32_MAX`, cleaned up the BSON writer, and improved the documentation. 408. [Luke Banicevic](https://github.com/banaboi) fixed corrupt BSON output for lengths exceeding `INT32_MAX`, cleaned up the BSON writer, and improved the documentation.
409. [Patrick Armstrong](https://github.com/Patrick10199) updated the CBOR references and the half-precision float assertions. 409. [Patrick Armstrong](https://github.com/Patrick10199) updated the CBOR references and the half-precision float assertions.
410. [Yash Bavadiya](https://github.com/xevrion) added checks to all BSON reads. 410. [Yash Bavadiya](https://github.com/xevrion) added checks to all BSON reads.
411. [hum4nBeing](https://github.com/hum4nBeing) and [AJ369ninja](https://github.com/AJ369ninja) fixed the overflow handling of high-precision numbers in UBJSON. 411. [hum4nBeing](https://github.com/hum4nBeing) fixed the overflow handling of high-precision numbers in UBJSON.
412. [tomatotomata](https://github.com/tomatotomata) added checks for reading CBOR tagged subtypes. 412. [tomatotomata](https://github.com/tomatotomata) added checks for reading CBOR tagged subtypes.
413. [YingqiDuan](https://github.com/YingqiDuan) documented the BSON interoperability. 413. [YingqiDuan](https://github.com/YingqiDuan) documented the BSON interoperability.
414. [KBS](https://github.com/youdie006) documented the standards compliance and the strictness of `parse()` and `operator>>`. 414. [KBS](https://github.com/youdie006) documented the standards compliance and the strictness of `parse()` and `operator>>`.
@@ -1867,26 +1861,6 @@ I deeply appreciate the help of the following people.
422. [Avionic Harshit](https://github.com/avionicharshit-byte) made `diff()` linear when an array shrinks. 422. [Avionic Harshit](https://github.com/avionicharshit-byte) made `diff()` linear when an array shrinks.
423. [Qatadaha Bin Matloob](https://github.com/qatcod) fixed comparisons between integers and floats and fixed unparsable BJData output. 423. [Qatadaha Bin Matloob](https://github.com/qatcod) fixed comparisons between integers and floats and fixed unparsable BJData output.
424. [Wu Shuwen](https://github.com/dajiaohuang) removed an unused include. 424. [Wu Shuwen](https://github.com/dajiaohuang) removed an unused include.
425. [Alexander Lanin](https://github.com/AlexanderLanin) improved the error message of `get_to()` for `const` arguments.
426. [satelliteprogrammer](https://github.com/satelliteprogrammer) added `merge_diff()` to create JSON Merge Patches (RFC 7396).
427. [Jeremy Nimmer](https://github.com/jwnimmer-tri) removed `alwayslink` from the Bazel build.
428. [Dadi Reddy Sai Praneeth Reddy](https://github.com/dsp0redy) made `to_msgpack()` throw when a size exceeds the MessagePack limit.
429. [ayush-singh-0601](https://github.com/ayush-singh-0601) reduced the runtime of the test suite.
430. [Kartikey Negi](https://github.com/ReturnKartikey) fixed the CBOR tag handling of tagged values that are not byte strings.
431. [Michiel van Slobbe](https://github.com/mvanslobbe) sped up the parser by checking for the expected separator first.
432. [Mohd Quamar Tyagi](https://github.com/Tyagiquamar) allowed SAX parsing of tagged CBOR.
433. [fhgffy](https://github.com/fhgffy) rejected NUL bytes in UBJSON and BJData high-precision numbers and fixed a `-Wswitch-enum` warning.
434. [Suyog Verma](https://github.com/commit-msuyog) used MSVC intrinsics for the 128-bit multiplication in the floating-point serialization.
435. [Bas van Dijk](https://github.com/basdedas) fixed a TU-local exposure of `UTF8_ACCEPT` and `UTF8_REJECT` in C++20 modules.
436. [Christoph Grüninger](https://github.com/gruenich) fixed a leak and a crash when an allocation throws during parsing.
437. [Amelia Barnaby](https://github.com/ameliabarnabyhub) made serializing discarded values to binary formats throw instead of writing corrupt output.
438. [Vesko Karaganev](https://github.com/veselink1) made the destructor use the provided allocator.
439. [Michael Sam](https://github.com/michaelsam94) kept allocation failures from escaping the `noexcept` destructor.
440. [Mihnea Magheru](https://github.com/sakuntalle) added a natvis visualizer for `detail::json_default_base`.
441. [Joseph Demarest](https://github.com/JosephDemarest) rejected nested indefinite-length CBOR strings.
442. [Afonso Januário](https://github.com/afonsojanu) made `std::hash<basic_json>` consistent with `operator==` for numbers.
443. [Kartik Gupta](https://github.com/KartikeyG-world) limited the natvis namespace variants to two ABI tags.
444. [manop55555](https://github.com/manop55555) reported a buffer over-read in `dump()` with large indentation values.
Thanks a lot for helping out! Please [let me know](mailto:mail@nlohmann.me) if I forgot someone. Thanks a lot for helping out! Please [let me know](mailto:mail@nlohmann.me) if I forgot someone.
@@ -1909,6 +1883,7 @@ The library itself consists of a single header file licensed under the MIT licen
- [**cpplint**](https://github.com/cpplint/cpplint) for style checks - [**cpplint**](https://github.com/cpplint/cpplint) for style checks
- [**doctest**](https://github.com/doctest/doctest) for the unit tests - [**doctest**](https://github.com/doctest/doctest) for the unit tests
- [**Flawfinder**](https://dwheeler.com/flawfinder/) for static analysis - [**Flawfinder**](https://dwheeler.com/flawfinder/) for static analysis
- [**GitHub Changelog Generator**](https://github.com/github-changelog-generator/github-changelog-generator) to generate the [ChangeLog](https://github.com/nlohmann/json/blob/develop/ChangeLog.md)
- [**Google Benchmark**](https://github.com/google/benchmark) to implement the benchmarks - [**Google Benchmark**](https://github.com/google/benchmark) to implement the benchmarks
- [**Hedley**](https://nemequ.github.io/hedley/) to avoid re-inventing several compiler-agnostic feature macros - [**Hedley**](https://nemequ.github.io/hedley/) to avoid re-inventing several compiler-agnostic feature macros
- [**include-what-you-use**](https://include-what-you-use.org) to check the included headers - [**include-what-you-use**](https://include-what-you-use.org) to check the included headers
@@ -1975,7 +1950,7 @@ For more information, see [JSON With Commas and Comments (JWCC)](https://nigelta
By default, the library does not preserve the **insertion order of object elements**. This is standards-compliant, as the [JSON standard](https://tools.ietf.org/html/rfc8259.html) defines objects as "an unordered collection of zero or more name/value pairs". By default, the library does not preserve the **insertion order of object elements**. This is standards-compliant, as the [JSON standard](https://tools.ietf.org/html/rfc8259.html) defines objects as "an unordered collection of zero or more name/value pairs".
If you do want to preserve the insertion order, you can try the type [`nlohmann::ordered_json`](https://json.nlohmann.me/api/ordered_json/). Alternatively, you can use a more sophisticated ordered map like [`tsl::ordered_map`](https://github.com/Tessil/ordered-map) ([integration](https://github.com/nlohmann/json/issues/546#issuecomment-304447518)) or [`nlohmann::fifo_map`](https://github.com/nlohmann/fifo_map) ([integration](https://github.com/nlohmann/json/issues/485#issuecomment-333652309)). If you do want to preserve the insertion order, you can try the type [`nlohmann::ordered_json`](https://github.com/nlohmann/json/issues/2179). Alternatively, you can use a more sophisticated ordered map like [`tsl::ordered_map`](https://github.com/Tessil/ordered-map) ([integration](https://github.com/nlohmann/json/issues/546#issuecomment-304447518)) or [`nlohmann::fifo_map`](https://github.com/nlohmann/fifo_map) ([integration](https://github.com/nlohmann/json/issues/485#issuecomment-333652309)).
See the [**documentation on object order**](https://json.nlohmann.me/features/object_order/) for more information. See the [**documentation on object order**](https://json.nlohmann.me/features/object_order/) for more information.
+22 -24
View File
@@ -2,7 +2,7 @@
# hashes must list every dependency, so this file was generated with # hashes must list every dependency, so this file was generated with
# uv pip compile requirements.in --generate-hashes --python-version 3.14 \ # uv pip compile requirements.in --generate-hashes --python-version 3.14 \
# --python-platform x86_64-manylinux_2_39 --no-header --no-annotate # --python-platform x86_64-manylinux_2_39 --no-header --no-annotate
# from a requirements.in containing only "semgrep==1.184.0". Dependabot keeps it current. # from a requirements.in containing only "semgrep==1.168.0". Dependabot keeps it current.
annotated-types==0.8.0 \ annotated-types==0.8.0 \
--hash=sha256:13b2beaad985e05e2d6407ee4c4f35590b11f8d693a258a561055cac8f64cab7 \ --hash=sha256:13b2beaad985e05e2d6407ee4c4f35590b11f8d693a258a561055cac8f64cab7 \
--hash=sha256:f072f4d804ea359e4eaf198b1af7a8b0943881a87f31bb764f8bf219bb9419e0 --hash=sha256:f072f4d804ea359e4eaf198b1af7a8b0943881a87f31bb764f8bf219bb9419e0
@@ -295,9 +295,9 @@ charset-normalizer==3.5.2 \
--hash=sha256:fcff63213e8e6e47770541a4607175404f47cbb3ebea7b6058cc82d524a0e424 \ --hash=sha256:fcff63213e8e6e47770541a4607175404f47cbb3ebea7b6058cc82d524a0e424 \
--hash=sha256:fd1fbe0f116b6e55da77aca2c6ddcddcfac2186cbf78bdebf40fc156efca389d \ --hash=sha256:fd1fbe0f116b6e55da77aca2c6ddcddcfac2186cbf78bdebf40fc156efca389d \
--hash=sha256:fe9753dfee015c570d73df76f899f18444d41388bffcde097deba51c4fadbb9f --hash=sha256:fe9753dfee015c570d73df76f899f18444d41388bffcde097deba51c4fadbb9f
click==8.4.2 \ click==8.1.8 \
--hash=sha256:9a6cea6e60b17ebe0a44c5cc636d94f09bd66142c1cd7d8b4cd731c4917a15f6 \ --hash=sha256:63c132bbbed01578a06712a2d1f497bb62d9c1c0d329b7903a866228027263b2 \
--hash=sha256:e6f9f66136c816745b9d65817da91d61d957fb16e02e4dcd0552553c5a197b76 --hash=sha256:ed53c9d8990d83c2a27deae68e4ee337473f6330c040a31d4225c9574d16096a
click-option-group==0.5.9 \ click-option-group==0.5.9 \
--hash=sha256:ad2599248bd373e2e19bec5407967c3eec1d0d4fc4a5e77b08a0481e75991080 \ --hash=sha256:ad2599248bd373e2e19bec5407967c3eec1d0d4fc4a5e77b08a0481e75991080 \
--hash=sha256:f94ed2bc4cf69052e0f29592bd1e771a1789bd7bfc482dd0bc482134aff95823 --hash=sha256:f94ed2bc4cf69052e0f29592bd1e771a1789bd7bfc482dd0bc482134aff95823
@@ -391,9 +391,9 @@ httpx-sse==0.4.3 \
idna==3.20 \ idna==3.20 \
--hash=sha256:a7db850025b95ded1eae8a46181a1a6c56c92c96f0e2b005d9ff8dc0210cab44 \ --hash=sha256:a7db850025b95ded1eae8a46181a1a6c56c92c96f0e2b005d9ff8dc0210cab44 \
--hash=sha256:ab7ae7122974553370f0bdb919e1a960b2cd1bc1ef0276416d896db81c14582c --hash=sha256:ab7ae7122974553370f0bdb919e1a960b2cd1bc1ef0276416d896db81c14582c
importlib-metadata==10.1.0 \ importlib-metadata==8.7.1 \
--hash=sha256:82b04047fa09792c1a8cfeddfe08bec79e4b13c29c8469b0d3949860f61017e8 \ --hash=sha256:49fef1ae6440c182052f407c8d34a68f72efc36db9ca90dc0113398f2fdde8bb \
--hash=sha256:cf45ce66a929c64e994185a809c04840d0501db17e663810d993d0f4829265c7 --hash=sha256:5a1f80bf1daa489495071efbb095d75a634cf28a8bc299581244063b53176151
jsonschema==4.25.1 \ jsonschema==4.25.1 \
--hash=sha256:3fba0169e345c7175110351d456342c364814cfcf3b964ba4587f22915230a63 \ --hash=sha256:3fba0169e345c7175110351d456342c364814cfcf3b964ba4587f22915230a63 \
--hash=sha256:e4a9655ce0da0c0b67a085847e00a3a51449e1157f4f75e9fb5aa545e122eb85 --hash=sha256:e4a9655ce0da0c0b67a085847e00a3a51449e1157f4f75e9fb5aa545e122eb85
@@ -403,9 +403,9 @@ jsonschema-specifications==2025.9.1 \
markdown-it-py==4.2.0 \ markdown-it-py==4.2.0 \
--hash=sha256:04a21681d6fbb623de53f6f364d352309d4094dd4194040a10fd51833e418d49 \ --hash=sha256:04a21681d6fbb623de53f6f364d352309d4094dd4194040a10fd51833e418d49 \
--hash=sha256:9f7ebbcd14fe59494226453aed97c1070d83f8d24b6fc3a3bcf9a38092641c4a --hash=sha256:9f7ebbcd14fe59494226453aed97c1070d83f8d24b6fc3a3bcf9a38092641c4a
mcp==1.29.0 \ mcp==1.23.3 \
--hash=sha256:52d01f334de1868cc3bb2d6604931126a67631f99a6c5d3b82ba47290315ec36 \ --hash=sha256:32768af4b46a1b4f7df34e2bfdf5c6011e7b63d7f1b0e321d0fdef4cd6082031 \
--hash=sha256:f5a075bb611f23d6f4d080c6a1699fa62772eebc562ba9e66b306ddde1c755f7 --hash=sha256:b3b0da2cc949950ce1259c7bfc1b081905a51916fcd7c8182125b85e70825201
mdurl==0.1.2 \ mdurl==0.1.2 \
--hash=sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8 \ --hash=sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8 \
--hash=sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba --hash=sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba
@@ -609,9 +609,9 @@ pydantic-settings==2.16.2 \
pygments==2.21.0 \ pygments==2.21.0 \
--hash=sha256:2363c69b61c4a97c838da3b130dcd6468f4848992b21a82f2a63ec34377137d9 \ --hash=sha256:2363c69b61c4a97c838da3b130dcd6468f4848992b21a82f2a63ec34377137d9 \
--hash=sha256:610ca751c9bc2492b38eb9a38a7fbc93edbbb2d7182edaf34e66ae493dee5c8c --hash=sha256:610ca751c9bc2492b38eb9a38a7fbc93edbbb2d7182edaf34e66ae493dee5c8c
pyjwt==2.15.1 \ pyjwt==2.13.0 \
--hash=sha256:42d59d631f7768a1028a64c7ff581a9bf7519804daf91fc5b6c56e30eec5e193 \ --hash=sha256:41571c89ca91598c79e8ef18a2d07367d4810fbbd6f637794879baf1b7703423 \
--hash=sha256:4f259e80cdfb6b3fc18a7de51fd1ef9ec79652f25019bae68975ca2468a34df8 --hash=sha256:66adcc2aff09b3f1bbd95fc1e1577df8ac8723c978552fd43304c8a290ac5728
python-dotenv==1.2.4 \ python-dotenv==1.2.4 \
--hash=sha256:42269a8a5b3fd54ffa6f3d84b18abed50064717576b4ecf03dc4a55d8aa04fdc \ --hash=sha256:42269a8a5b3fd54ffa6f3d84b18abed50064717576b4ecf03dc4a55d8aa04fdc \
--hash=sha256:f0d53e69935a851c0dcc78f3ab7aaccd8cabef0b92382b576b824212902873c0 --hash=sha256:f0d53e69935a851c0dcc78f3ab7aaccd8cabef0b92382b576b824212902873c0
@@ -824,23 +824,21 @@ ruamel-yaml-clib==0.2.15 \
semantic-version==2.10.0 \ semantic-version==2.10.0 \
--hash=sha256:bdabb6d336998cbb378d4b9db3a4b56a1e3235701dc05ea2690d9a997ed5041c \ --hash=sha256:bdabb6d336998cbb378d4b9db3a4b56a1e3235701dc05ea2690d9a997ed5041c \
--hash=sha256:de78a3b8e0feda74cabc54aab2da702113e33ac9d9eb9d2389bcf1f58b7d9177 --hash=sha256:de78a3b8e0feda74cabc54aab2da702113e33ac9d9eb9d2389bcf1f58b7d9177
semgrep==1.184.0 \ semgrep==1.168.0 \
--hash=sha256:0ce3064175a44236dd9fd35603627e3f56139ac61e13ca967bae2b76b2cec9bf \ --hash=sha256:09dfacb0530ed4a17bd2deb7914e9a25fc3581d5d84d5365cdac77bbebed8081 \
--hash=sha256:0fc8a58e9a62de59ce644dde792dfc36764245d0139ab85632f66a8e0846a8c5 \ --hash=sha256:1a5d6b5b47347b22bcb1d6f15a5e51a21e734244884624c3494e09bc1de955cd \
--hash=sha256:3c86b2961b4663d1338fb965de038c7869571661971b78e18e9346ab57e1e20a \ --hash=sha256:86e99b095b80492b6cfe5087bf4a3b0175a1132c9c9dcd25a96c3745cc58d85a \
--hash=sha256:42a59e95433fd05a0908f8083e25656c90994c65549e483e0de8a1a0004a7adf \ --hash=sha256:a072b1734b5c54e39cbbe957b10cf0b83a114b9b9fc762b4ea51afe2f114cd11 \
--hash=sha256:cf82d77eb911f921902f3e9d098fb97604c5419c0a87cf7de7ab2944b05e2ceb \ --hash=sha256:c4a664f4dda097fbdf657d64f79217e94c4d87502c16c991fac569f07aaecb8e \
--hash=sha256:d847abb8fb6fdde77d9e30a6bc869dbfcb68eff4142dcd77897c52635a7600d9 \ --hash=sha256:d288699c4c056cc3d7cf82830e00f9c07fac33227c3a5290a2113bd1be63da46 \
--hash=sha256:f794d7c6d33126b6881f80d069583ef35b52bd796d882a5fa249ce1124aba3f3 --hash=sha256:e6b85d84e815ff86f56dcb6b4a6370b98f1bc0e402144daafa9b41aa00ea9a90 \
--hash=sha256:fa22f0a41ee3857ecec1b3883b2920b1d41d07ef1bdd7e216da3c519210ae5fb
sse-starlette==3.7.0 \ sse-starlette==3.7.0 \
--hash=sha256:326416da2ab52e9a41b3f409d3ff897eda45aa68c9f3b7b17d0913aab8d219c3 \ --hash=sha256:326416da2ab52e9a41b3f409d3ff897eda45aa68c9f3b7b17d0913aab8d219c3 \
--hash=sha256:f1d3136caa12d129284fca3d2ded79bc9af943320b6472712955fea52a6d3eea --hash=sha256:f1d3136caa12d129284fca3d2ded79bc9af943320b6472712955fea52a6d3eea
starlette==1.10.0 \ starlette==1.10.0 \
--hash=sha256:478c0ab3b4ced6e82719849c118fa406f0278a3499c1a898cd15cd31bdc38ce4 \ --hash=sha256:478c0ab3b4ced6e82719849c118fa406f0278a3499c1a898cd15cd31bdc38ce4 \
--hash=sha256:b8399098d4c1e3522734f7235dc79b1fc967f96cf139c81e0fa30479e0685f9b --hash=sha256:b8399098d4c1e3522734f7235dc79b1fc967f96cf139c81e0fa30479e0685f9b
structlog==26.1.0 \
--hash=sha256:e081a26d6c373e6d201eca24eede26d8ffab07f88f477822e679183428d3d91e \
--hash=sha256:f63a716cbd1b1291cf7661de7794b455acfa4c43c5bcf1630e6ad5ddc1adb3b7
tomli==2.4.1 \ tomli==2.4.1 \
--hash=sha256:01f520d4f53ef97964a240a035ec2a869fe1a37dde002b57ebc4417a27ccd853 \ --hash=sha256:01f520d4f53ef97964a240a035ec2a869fe1a37dde002b57ebc4417a27ccd853 \
--hash=sha256:0d85819802132122da43cb86656f8d1f8c6587d54ae7dcaf30e90533028b49fe \ --hash=sha256:0d85819802132122da43cb86656f8d1f8c6587d54ae7dcaf30e90533028b49fe \
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

@@ -67,11 +67,7 @@ basic_json(basic_json&& other) noexcept;
- **objects**: [`object_t`](object_t.md) and all kinds of compatible associative containers such as `std::map`, - **objects**: [`object_t`](object_t.md) and all kinds of compatible associative containers such as `std::map`,
`std::unordered_map`, `std::multimap`, and `std::unordered_multimap` with a `key_type` compatible to `string_t` `std::unordered_map`, `std::multimap`, and `std::unordered_multimap` with a `key_type` compatible to `string_t`
and a `value_type` from which a `basic_json` value can be constructed. and a `value_type` from which a `basic_json` value can be constructed.
- **strings**: `string_t`, string literals, and all compatible string containers can be used. Since version 3.13.0, - **strings**: `string_t`, string literals, and all compatible string containers can be used.
this includes `#!cpp std::u8string` (C++20): its code units are stored as a JSON string and
[`get<std::u8string>()`](get.md) reads it back (in version 3.12.0, a `#!cpp std::u8string` was stored as an array
of numbers; `get<std::u8string>()` still accepts such an array). `#!cpp std::u8string_view` and `#!cpp char8_t`
arrays are not strings; they are stored as arrays of code units.
!!! note "Rvalue containers are moved from" !!! note "Rvalue containers are moved from"
-6
View File
@@ -130,12 +130,6 @@ overload (3).
`#!cpp json(true).get<double>()` throw. This is the behavior of all 3.x versions; `#!cpp json(true).get<double>()` throw. This is the behavior of all 3.x versions;
[`basic_json_view::get<T>()`](../basic_json_view/get.md) behaves the same. [`basic_json_view::get<T>()`](../basic_json_view/get.md) behaves the same.
!!! note "`std::u8string` conversions"
Since version 3.13.0, `#!cpp get<std::u8string>()` (C++20) reads a JSON string; it also accepts an array of numbers,
the form in which version 3.12.0 stored a `#!cpp std::u8string`. Any other JSON type throws
[`type_error.302`](../../home/exceptions.md#jsonexceptiontype_error302) ("type must be string or array").
!!! note "`std::optional` conversions" !!! note "`std::optional` conversions"
Prior to version 3.13.0, `#!cpp get<std::optional<T>>()` (and other conversions to `std::optional<T>`) failed to Prior to version 3.13.0, `#!cpp get<std::optional<T>>()` (and other conversions to `std::optional<T>`) failed to
@@ -425,7 +425,7 @@ using array_t = ArrayType<basic_json, AllocatorType<basic_json>>;
| Type | Reason | | Type | Reason |
|----------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------| |----------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
| `#!cpp std::wstring`, `#!cpp std::u16string`, `#!cpp std::u32string` | the character type is not one byte wide | | `#!cpp std::wstring`, `#!cpp std::u16string`, `#!cpp std::u32string` | the character type is not one byte wide |
| `#!cpp std::u8string` | one byte wide, but `#!cpp char8_t` is not `#!cpp char`-compatible (as a *value*, it converts to and from a JSON string) | | `#!cpp std::u8string` | one byte wide, but `#!cpp char8_t` is not `#!cpp char`-compatible |
| `absl::Cord` | no `value_type`, and the storage is not contiguous | | `absl::Cord` | no `value_type`, and the storage is not contiguous |
| `QString` | no `append(const char*, size_type)`; its `QChar` is also two bytes wide, though that is never diagnosed | | `QString` | no `append(const char*, size_type)`; its `QChar` is also two bytes wide, though that is never diagnosed |
@@ -8,16 +8,6 @@ These changes are part of the `develop` branch, but not of a release yet.
exception is thrown during the conversion, the source may be partially moved-from, so the strong exception guarantee exception is thrown during the conversion, the source may be partially moved-from, so the strong exception guarantee
no longer holds for the source of an rvalue conversion (the JSON value itself is unaffected). Lvalues and `const` no longer holds for the source of an rvalue conversion (the JSON value itself is unaffected). Lvalues and `const`
objects are still copied. See [`basic_json`](../../api/basic_json/basic_json.md). objects are still copied. See [`basic_json`](../../api/basic_json/basic_json.md).
- In C++20 (library support for `char8_t`, `__cpp_lib_char8_t`), a `#!cpp std::u8string` is a JSON **string**:
`json j = std::u8string(u8"abc");` stores `"abc"`, and `#!cpp j.get<std::u8string>()` reads it back. Version 3.12.0
stored a `#!cpp std::u8string` as an array of code units (`[97,98,99]`); on the 3.13.0 development branch it was
already stored as a string if `std::filesystem` is available, but could not be read back. Now the result no longer
depends on whether `std::filesystem::path` is supported, and `#!cpp get<std::u8string>()` accepts both a JSON string
and the array of numbers written by 3.12.0. **Consequence:** JSON documents produced from a `#!cpp std::u8string` by
3.12.0 (arrays) are still read, but code that checks `is_array()` on such a value, or that reads it as
`#!cpp std::vector<char8_t>` / `#!cpp std::vector<std::uint8_t>`, now sees a string; to keep the array form, convert
explicitly, e.g., `#!cpp json(std::vector<char8_t>(s.begin(), s.end()))`. `#!cpp std::u8string_view` and
`#!cpp char8_t` arrays are unchanged (arrays of code units). See [`basic_json`](../../api/basic_json/basic_json.md).
## API changes ## API changes
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

@@ -785,37 +785,6 @@ void from_json(const BasicJsonType& j, std::unordered_map<Key, Value, Hash, KeyE
from_json_pair_array_to_map(j, m); from_json_pair_array_to_map(j, m);
} }
#if defined(__cpp_lib_char8_t)
// a std::u8string is read from a JSON string; the array of code units written by
// versions before 3.13.0 is still accepted
template < typename BasicJsonType, typename Tr, typename Allocator,
enable_if_t < std::is_same<typename json_type_of_t<BasicJsonType>::string_t::value_type, char>::value, int > = 0 >
inline void from_json(const BasicJsonType& j, std::basic_string<char8_t, Tr, Allocator>& s)
{
if (j.is_string())
{
const auto& str = string_value(j);
const char* const data = str.data();
const std::size_t size = str.size();
std::basic_string<char8_t, Tr, Allocator> ret(s.get_allocator());
ret.reserve(size);
for (std::size_t i = 0; i < size; ++i)
{
ret.push_back(static_cast<char8_t>(static_cast<unsigned char>(data[i])));
}
s = std::move(ret);
}
else if (j.is_array())
{
from_json_array_impl(j, s, priority_tag<3> {});
}
else
{
throw_type_must_be("string or array", j);
}
}
#endif
#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM #if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
// Workaround for MSVC 19.51 (and possibly later): in large cpp files, the compiler may fail to resolve with generic has_from_json (issue #4996) // Workaround for MSVC 19.51 (and possibly later): in large cpp files, the compiler may fail to resolve with generic has_from_json (issue #4996)
@@ -625,20 +625,16 @@ inline void to_json(BasicJsonType& j, const T& t)
to_json_tuple_impl(j, t, make_index_sequence<std::tuple_size<T>::value> {}); to_json_tuple_impl(j, t, make_index_sequence<std::tuple_size<T>::value> {});
} }
// a std::u8string is a JSON string (it was an array of code units before 3.13.0); #if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
// defined outside the std::filesystem guard so that the result does not depend
// on whether std::filesystem::path is supported
#if defined(__cpp_lib_char8_t) #if defined(__cpp_lib_char8_t)
template<typename BasicJsonType, typename Tr, typename Allocator> template<typename BasicJsonType, typename Tr, typename Allocator>
inline void to_json(BasicJsonType& j, const std::basic_string<char8_t, Tr, Allocator>& s) inline void to_json(BasicJsonType& j, const std::basic_string<char8_t, Tr, Allocator>& s)
{ {
using string_t = typename BasicJsonType::string_t; using OtherAllocator = typename std::allocator_traits<Allocator>::template rebind_alloc<char>;
// char may alias any object; the code units are copied unchanged j = std::basic_string<char, std::char_traits<char>, OtherAllocator>(s.begin(), s.end(), s.get_allocator());
j = string_t(reinterpret_cast<const char*>(s.data()), s.size());
} }
#endif #endif
#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
// Workaround for MSVC 19.51 (and possibly later): in large cpp files, the compiler may fail to resolve with generic has_to_json (issue #4996) // Workaround for MSVC 19.51 (and possibly later): in large cpp files, the compiler may fail to resolve with generic has_to_json (issue #4996)
template<typename BasicJsonType> template<typename BasicJsonType>
struct has_to_json<BasicJsonType, std_fs::path, void> : std::true_type {}; struct has_to_json<BasicJsonType, std_fs::path, void> : std::true_type {};
@@ -1348,64 +1348,6 @@ struct iterator_input_adapter_factory<IteratorType, SentinelType, enable_if_t<is
} }
}; };
/*!
@brief whether @a IteratorType is an iterator of std::string or of a
std::vector of 1-byte elements
Their elements are contiguous, but before C++20 that cannot be detected from
the iterator type (std::contiguous_iterator), so an iterator pair of them would
be read one character at a time through the iterators. They are read as the
pointer range they are instead, as the containers themselves are (see the
container overload of input_adapter()). The types are matched exactly, so no
other iterator can be mistaken for one. (std::basic_string is only matched for
char: other character types need std::char_traits specializations that the
standard library need not have.)
*/
template<typename CharType>
struct is_byte_char_type
{
// a member, not an integral_constant argument, which astyle misformats
static constexpr bool value = sizeof(CharType) == 1 && std::is_integral<CharType>::value && !std::is_same<CharType, bool>::value;
};
template<typename IteratorType, typename CharType, bool IsByte = is_byte_char_type<CharType>::value>
struct is_std_contiguous_byte_iterator_impl : std::false_type {};
// (the containers are only named for byte types)
template<typename IteratorType, typename CharType>
struct is_std_contiguous_byte_iterator_impl<IteratorType, CharType, true>
{
static constexpr bool value = !std::is_pointer<IteratorType>::value
&& (std::is_same<IteratorType, typename std::vector<CharType>::iterator>::value
|| std::is_same<IteratorType, typename std::vector<CharType>::const_iterator>::value
|| std::is_same<IteratorType, std::string::iterator>::value
|| std::is_same<IteratorType, std::string::const_iterator>::value);
};
template<typename IteratorType>
struct is_std_contiguous_byte_iterator
: is_std_contiguous_byte_iterator_impl<IteratorType, typename std::remove_cv<typename std::iterator_traits<IteratorType>::value_type>::type> {};
template<typename IteratorType>
struct iterator_input_adapter_factory<IteratorType, IteratorType, enable_if_t<is_std_contiguous_byte_iterator<IteratorType>::value>>
{
using iterator_type = IteratorType;
using sentinel_type = IteratorType;
using char_type = typename std::remove_cv<typename std::iterator_traits<iterator_type>::value_type>::type;
using adapter_type = iterator_input_adapter<const char_type*>;
static adapter_type create(IteratorType first, IteratorType last)
{
if (first == last)
{
// (an empty range must not be dereferenced)
return adapter_type(nullptr, nullptr);
}
const char_type* const begin = std::addressof(*first);
return adapter_type(begin, begin + (last - first));
}
};
// General purpose iterator-based input (iterator+sentinel pair; SentinelType // General purpose iterator-based input (iterator+sentinel pair; SentinelType
// defaults to IteratorType for the common same-type case, but may differ for // defaults to IteratorType for the common same-type case, but may differ for
// C++20 ranges-style iterator+sentinel pairs). Only enable for types that can // C++20 ranges-style iterator+sentinel pairs). Only enable for types that can
+83 -48
View File
@@ -92,12 +92,8 @@ namespace detail
character, or not ASCII, 16 bytes per step character, or not ASCII, 16 bytes per step
Stops at such a byte, or where fewer than 16 bytes are left (the caller tells Stops at such a byte, or where fewer than 16 bytes are left (the caller tells
the two apart). One signed compare finds the quote, the control characters and the two apart). A signed compare with 0x20 finds control characters and
the non-ASCII bytes at once: flipping bit 1 maps 0x00-0x1F onto itself and the non-ASCII bytes at once.
quote (0x22) onto 0x20, keeps 0x80-0xFF negative, and maps every other byte to
33 or more, so these are exactly the bytes with (int8_t)(c ^ 2) < 33 (see
D. Lemire, "Detect control characters, quotes and backslashes efficiently
using SWAR", 2025). The backslash needs its own compare.
*/ */
JSON_HEDLEY_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned char* p, const unsigned char* e) noexcept JSON_HEDLEY_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned char* p, const unsigned char* e) noexcept
{ {
@@ -105,8 +101,8 @@ JSON_HEDLEY_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned c
{ {
#if JSON_SCAN_NEON #if JSON_SCAN_NEON
const uint8x16_t in = vld1q_u8(p); const uint8x16_t in = vld1q_u8(p);
const uint8x16_t special = vorrq_u8(vceqq_u8(in, vdupq_n_u8('\\')), const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))),
vcltq_s8(vreinterpretq_s8_u8(veorq_u8(in, vdupq_n_u8(2))), vdupq_n_s8(33))); vcltq_s8(vreinterpretq_s8_u8(in), vdupq_n_s8(0x20)));
// one nibble per byte (the usual NEON replacement of x86's movemask, see // one nibble per byte (the usual NEON replacement of x86's movemask, see
// D. Kutenin, "Porting x86 vector bitmask optimizations to Arm NEON", 2022) // D. Kutenin, "Porting x86 vector bitmask optimizations to Arm NEON", 2022)
const std::uint64_t bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0); const std::uint64_t bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0);
@@ -116,8 +112,8 @@ JSON_HEDLEY_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned c
} }
#else #else
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void) const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
const __m128i special = _mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('\\')), const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_cmplt_epi8(_mm_xor_si128(in, _mm_set1_epi8(2)), _mm_set1_epi8(33))); _mm_cmplt_epi8(in, _mm_set1_epi8(0x20)));
const auto bits = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special))); const auto bits = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
if (bits != 0) if (bits != 0)
{ {
@@ -254,8 +250,7 @@ JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline bool vector_utf8_run(cons
while (e - block >= 16) while (e - block >= 16)
{ {
const uint8x16_t in = vld1q_u8(block); const uint8x16_t in = vld1q_u8(block);
// the quote and the control characters: (c ^ 2) < 33 (see vector_plain_run()) const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))), vcltq_u8(in, vdupq_n_u8(0x20)));
const uint8x16_t special = vorrq_u8(vceqq_u8(in, vdupq_n_u8('\\')), vcltq_u8(veorq_u8(in, vdupq_n_u8(2)), vdupq_n_u8(33)));
const uint8x16_t prev1 = vextq_u8(prev, in, 15); const uint8x16_t prev1 = vextq_u8(prev, in, 15);
const uint8x16_t sc = vandq_u8(vandq_u8(vqtbl1q_u8(t1h, vshrq_n_u8(prev1, 4)), vqtbl1q_u8(t1l, vandq_u8(prev1, vdupq_n_u8(0x0F)))), vqtbl1q_u8(t2h, vshrq_n_u8(in, 4))); const uint8x16_t sc = vandq_u8(vandq_u8(vqtbl1q_u8(t1h, vshrq_n_u8(prev1, 4)), vqtbl1q_u8(t1l, vandq_u8(prev1, vdupq_n_u8(0x0F)))), vqtbl1q_u8(t2h, vshrq_n_u8(in, 4)));
const uint8x16_t must23 = vorrq_u8(vqsubq_u8(vextq_u8(prev, in, 14), vdupq_n_u8(0xE0 - 0x80)), vqsubq_u8(vextq_u8(prev, in, 13), vdupq_n_u8(0xF0 - 0x80))); const uint8x16_t must23 = vorrq_u8(vqsubq_u8(vextq_u8(prev, in, 14), vdupq_n_u8(0xE0 - 0x80)), vqsubq_u8(vextq_u8(prev, in, 13), vdupq_n_u8(0xF0 - 0x80)));
@@ -295,10 +290,8 @@ JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline bool vector_utf8_run(cons
while (e - block >= 16) while (e - block >= 16)
{ {
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void) const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void)
// the quote and the control characters: (c ^ 2) < 33, that is, const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
// (c ^ 2) - 32 saturates to 0 (see vector_plain_run()) _mm_cmpeq_epi8(_mm_subs_epu8(in, _mm_set1_epi8(0x1F)), zero)); // in < 0x20
const __m128i special = _mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('\\')),
_mm_cmpeq_epi8(_mm_subs_epu8(_mm_xor_si128(in, _mm_set1_epi8(2)), _mm_set1_epi8(32)), zero));
const __m128i prev1 = _mm_alignr_epi8(in, prev, 15); const __m128i prev1 = _mm_alignr_epi8(in, prev, 15);
const __m128i sc = _mm_and_si128(_mm_and_si128(_mm_shuffle_epi8(t1h, _mm_and_si128(_mm_srli_epi16(prev1, 4), nibble)), const __m128i sc = _mm_and_si128(_mm_and_si128(_mm_shuffle_epi8(t1h, _mm_and_si128(_mm_srli_epi16(prev1, 4), nibble)),
_mm_shuffle_epi8(t1l, _mm_and_si128(prev1, nibble))), _mm_shuffle_epi8(t1l, _mm_and_si128(prev1, nibble))),
@@ -331,15 +324,23 @@ JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline bool vector_utf8_run(cons
} }
/*! /*!
@brief skip 16-byte blocks of well-formed UTF-8 at p (a character boundary) @brief vector_utf8_valid_blocks() from the first block that is not ASCII
Like vector_utf8_run, but quotes, backslashes, and control characters are @a block is that block, and @a first is where the check began (only ASCII up
ordinary characters. Returns the start of the first block in which the check to @a block). Returns where fewer than 16 bytes are left, or @a first if the
found an error (which may belong to a sequence begun in the block before) or check found an error in any block.
where fewer than 16 bytes are left; a sequence may be cut at that point, so the
caller resumes at its start, one sequence at a time. From a block that is not ASCII on, eight blocks are checked without testing
for ASCII again: in text with a few non-ASCII characters (accented letters),
every other block has one, and the test would be a mispredicted branch for
most blocks. After them, ASCII blocks are skipped with one test again.
The errors of all blocks are collected and tested once at the end. A test
after each block lets the compiler advance the position with a conditional
select on its result (Apple Clang 17 does), and each block then waits for the
check of the block before: 28 cycles for 16 bytes instead of 4 on Apple M1.
*/ */
JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline const unsigned char* vector_utf8_valid_blocks(const unsigned char* block, const unsigned char* e) noexcept JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline const unsigned char* vector_utf8_check_blocks(const unsigned char* first, const unsigned char* block, const unsigned char* e) noexcept
{ {
using lookup = utf8_lookup4<>; using lookup = utf8_lookup4<>;
#if JSON_SCAN_NEON #if JSON_SCAN_NEON
@@ -347,27 +348,29 @@ JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline const unsigned char* vect
const uint8x16_t t1l = vld1q_u8(lookup::byte_1_low.data()); const uint8x16_t t1l = vld1q_u8(lookup::byte_1_low.data());
const uint8x16_t t2h = vld1q_u8(lookup::byte_2_high.data()); const uint8x16_t t2h = vld1q_u8(lookup::byte_2_high.data());
uint8x16_t prev = vdupq_n_u8(0); uint8x16_t prev = vdupq_n_u8(0);
uint8x16_t error = vdupq_n_u8(0);
while (e - block >= 16) while (e - block >= 16)
{ {
const uint8x16_t in = vld1q_u8(block); const uint8x16_t ahead = vld1q_u8(block);
if (vmaxvq_u8(vorrq_u8(in, prev)) < 0x80) if (vmaxvq_u8(vorrq_u8(ahead, prev)) < 0x80)
{ {
// ASCII here, and no sequence from the block before to finish // ASCII here, and no sequence from the block before to finish
prev = in; prev = ahead;
block += 16; block += 16;
continue; continue;
} }
const uint8x16_t prev1 = vextq_u8(prev, in, 15); for (int blocks = 8; blocks != 0 && e - block >= 16; --blocks)
const uint8x16_t sc = vandq_u8(vandq_u8(vqtbl1q_u8(t1h, vshrq_n_u8(prev1, 4)), vqtbl1q_u8(t1l, vandq_u8(prev1, vdupq_n_u8(0x0F)))), vqtbl1q_u8(t2h, vshrq_n_u8(in, 4)));
const uint8x16_t must23 = vorrq_u8(vqsubq_u8(vextq_u8(prev, in, 14), vdupq_n_u8(0xE0 - 0x80)), vqsubq_u8(vextq_u8(prev, in, 13), vdupq_n_u8(0xF0 - 0x80)));
const uint8x16_t err = veorq_u8(vandq_u8(must23, vdupq_n_u8(0x80)), sc);
if (vmaxvq_u8(err) != 0)
{ {
break; const uint8x16_t in = vld1q_u8(block);
const uint8x16_t prev1 = vextq_u8(prev, in, 15);
const uint8x16_t sc = vandq_u8(vandq_u8(vqtbl1q_u8(t1h, vshrq_n_u8(prev1, 4)), vqtbl1q_u8(t1l, vandq_u8(prev1, vdupq_n_u8(0x0F)))), vqtbl1q_u8(t2h, vshrq_n_u8(in, 4)));
const uint8x16_t must23 = vorrq_u8(vqsubq_u8(vextq_u8(prev, in, 14), vdupq_n_u8(0xE0 - 0x80)), vqsubq_u8(vextq_u8(prev, in, 13), vdupq_n_u8(0xF0 - 0x80)));
error = vorrq_u8(error, veorq_u8(vandq_u8(must23, vdupq_n_u8(0x80)), sc));
prev = in;
block += 16;
} }
prev = in;
block += 16;
} }
return vmaxvq_u8(error) != 0 ? first : block;
#else #else
const __m128i t1h = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_1_high.data()))); // NOLINT(bugprone-casting-through-void) const __m128i t1h = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_1_high.data()))); // NOLINT(bugprone-casting-through-void)
const __m128i t1l = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_1_low.data()))); // NOLINT(bugprone-casting-through-void) const __m128i t1l = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_1_low.data()))); // NOLINT(bugprone-casting-through-void)
@@ -375,30 +378,62 @@ JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline const unsigned char* vect
const __m128i nibble = _mm_set1_epi8(0x0F); const __m128i nibble = _mm_set1_epi8(0x0F);
const __m128i zero = _mm_setzero_si128(); const __m128i zero = _mm_setzero_si128();
__m128i prev = zero; __m128i prev = zero;
__m128i error = zero;
while (e - block >= 16) while (e - block >= 16)
{ {
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void) const __m128i ahead = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void)
if (_mm_movemask_epi8(_mm_or_si128(in, prev)) == 0) if (_mm_movemask_epi8(_mm_or_si128(ahead, prev)) == 0)
{ {
prev = in; prev = ahead;
block += 16; block += 16;
continue; continue;
} }
const __m128i prev1 = _mm_alignr_epi8(in, prev, 15); for (int blocks = 8; blocks != 0 && e - block >= 16; --blocks)
const __m128i sc = _mm_and_si128(_mm_and_si128(_mm_shuffle_epi8(t1h, _mm_and_si128(_mm_srli_epi16(prev1, 4), nibble)),
_mm_shuffle_epi8(t1l, _mm_and_si128(prev1, nibble))),
_mm_shuffle_epi8(t2h, _mm_and_si128(_mm_srli_epi16(in, 4), nibble)));
const __m128i must23 = _mm_or_si128(_mm_subs_epu8(_mm_alignr_epi8(in, prev, 14), _mm_set1_epi8(0xE0 - 0x80)),
_mm_subs_epu8(_mm_alignr_epi8(in, prev, 13), _mm_set1_epi8(0xF0 - 0x80)));
const __m128i err = _mm_xor_si128(_mm_and_si128(must23, _mm_set1_epi8(static_cast<char>(-128))), sc);
if (_mm_movemask_epi8(_mm_cmpeq_epi8(err, zero)) != 0xFFFF)
{ {
break; const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void)
const __m128i prev1 = _mm_alignr_epi8(in, prev, 15);
const __m128i sc = _mm_and_si128(_mm_and_si128(_mm_shuffle_epi8(t1h, _mm_and_si128(_mm_srli_epi16(prev1, 4), nibble)),
_mm_shuffle_epi8(t1l, _mm_and_si128(prev1, nibble))),
_mm_shuffle_epi8(t2h, _mm_and_si128(_mm_srli_epi16(in, 4), nibble)));
const __m128i must23 = _mm_or_si128(_mm_subs_epu8(_mm_alignr_epi8(in, prev, 14), _mm_set1_epi8(0xE0 - 0x80)),
_mm_subs_epu8(_mm_alignr_epi8(in, prev, 13), _mm_set1_epi8(0xF0 - 0x80)));
error = _mm_or_si128(error, _mm_xor_si128(_mm_and_si128(must23, _mm_set1_epi8(static_cast<char>(-128))), sc));
prev = in;
block += 16;
}
}
return _mm_movemask_epi8(_mm_cmpeq_epi8(error, zero)) != 0xFFFF ? first : block;
#endif
}
/*!
@brief skip 16-byte blocks of well-formed UTF-8 at p (a character boundary)
Like vector_utf8_run, but quotes, backslashes, and control characters are
ordinary characters. Returns where fewer than 16 bytes are left; a sequence may
be cut at that point, so the caller resumes at its start, one sequence at a
time. If the check found an error in any block, @a block itself is returned,
for the caller to find the error one sequence at a time.
ASCII blocks need one test each and none of the tables of the UTF-8 check,
which starts at the first block that is not ASCII (see
vector_utf8_check_blocks()). Out of line, like the check.
*/
JSON_HEDLEY_NEVER_INLINE inline const unsigned char* vector_utf8_valid_blocks(const unsigned char* block, const unsigned char* e) noexcept
{
const unsigned char* const first = block;
while (e - block >= 16)
{
#if JSON_SCAN_NEON
if (JSON_HEDLEY_UNLIKELY(vmaxvq_u8(vld1q_u8(block)) >= 0x80))
#else
if (JSON_HEDLEY_UNLIKELY(_mm_movemask_epi8(_mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block)))) != 0)) // NOLINT(bugprone-casting-through-void)
#endif
{
return vector_utf8_check_blocks(first, block, e);
} }
prev = in;
block += 16; block += 16;
} }
#endif
return block; return block;
} }
#endif #endif
+10 -6
View File
@@ -58,10 +58,11 @@ inline std::uint64_t swar_string_special(std::uint64_t v) noexcept
constexpr std::uint64_t high = 0x8080808080808080ull; constexpr std::uint64_t high = 0x8080808080808080ull;
const std::uint64_t q = v ^ 0x2222222222222222ull; // '"' (0x22) const std::uint64_t q = v ^ 0x2222222222222222ull; // '"' (0x22)
const std::uint64_t b = v ^ 0x5C5C5C5C5C5C5C5Cull; // '\\' (0x5C) const std::uint64_t b = v ^ 0x5C5C5C5C5C5C5C5Cull; // '\\' (0x5C)
// The usual test for a zero byte is (x - ones) & ~x & high. The `& ~x` const std::uint64_t has_quote = (q - ones) & ~q & high;
// only removes bytes from 0x80, and `v & high` flags those anyway, so it const std::uint64_t has_backslash = (b - ones) & ~b & high;
// can be dropped. Clang finds this reduction itself; GCC does not. const std::uint64_t has_control = (v - 0x2020202020202020ull) & ~v & high; // < 0x20
return ((q - ones) | (b - ones) | (v - 0x2020202020202020ull) | v) & high; const std::uint64_t has_non_ascii = v & high; // >= 0x80
return has_quote | has_backslash | has_control | has_non_ascii;
} }
// return the index of the first is_string_special() byte in [data, data+n), or // return the index of the first is_string_special() byte in [data, data+n), or
@@ -142,8 +143,11 @@ inline std::size_t find_ascii_copyable_run(const unsigned char* data, std::size_
const std::uint64_t q = v ^ 0x2222222222222222ull; // '"' (0x22) const std::uint64_t q = v ^ 0x2222222222222222ull; // '"' (0x22)
const std::uint64_t b = v ^ 0x5C5C5C5C5C5C5C5Cull; // '\\' (0x5C) const std::uint64_t b = v ^ 0x5C5C5C5C5C5C5C5Cull; // '\\' (0x5C)
const std::uint64_t d = v ^ 0x7F7F7F7F7F7F7F7Full; // DEL (0x7F) const std::uint64_t d = v ^ 0x7F7F7F7F7F7F7F7Full; // DEL (0x7F)
// (see swar_string_special(): the bytes from 0x80 are stops anyway) const std::uint64_t stop = ((q - ones) & ~q & high) // == '"'
const std::uint64_t stop = ((q - ones) | (b - ones) | (d - ones) | (v - 0x2020202020202020ull) | v) & high; | ((b - ones) & ~b & high) // == '\\'
| ((d - ones) & ~d & high) // == 0x7F
| ((v - 0x2020202020202020ull) & ~v & high) // < 0x20
| (v & high); // >= 0x80
if (stop != 0) if (stop != 0)
{ {
// the lowest flagged byte is the first one to stop at (see // the lowest flagged byte is the first one to stop at (see
+18 -22
View File
@@ -724,8 +724,10 @@ class serializer
constexpr std::uint64_t ones = 0x0101010101010101ull; constexpr std::uint64_t ones = 0x0101010101010101ull;
const std::uint64_t q = v ^ 0x2222222222222222ull; // '"' const std::uint64_t q = v ^ 0x2222222222222222ull; // '"'
const std::uint64_t b = v ^ 0x5C5C5C5C5C5C5C5Cull; // '\\' const std::uint64_t b = v ^ 0x5C5C5C5C5C5C5C5Cull; // '\\'
// (see swar_string_special(): the bytes from 0x80 are flagged anyway) return (((q - ones) & ~q) | ((b - ones) & ~b) // == '"', == '\\'
return ((q - ones) | (b - ones) | (v - 0x2020202020202020ull) | v | (v + ones)) & 0x8080808080808080ull; | ((v - 0x2020202020202020ull) & ~v) // < 0x20
| v | (v + ones)) // >= 0x7F
& 0x8080808080808080ull;
} }
template<typename WordType> template<typename WordType>
@@ -834,10 +836,8 @@ class serializer
const uint8x8_t lo = vld1_u8(reinterpret_cast<const std::uint8_t*>(p)); const uint8x8_t lo = vld1_u8(reinterpret_cast<const std::uint8_t*>(p));
const uint8x8_t hi = vld1_u8(reinterpret_cast<const std::uint8_t*>(p + n - 8)); const uint8x8_t hi = vld1_u8(reinterpret_cast<const std::uint8_t*>(p + n - 8));
const uint8x16_t in = vcombine_u8(lo, hi); const uint8x16_t in = vcombine_u8(lo, hi);
// (int8_t)(c ^ 2) < 33: the quote, the control characters and the const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))),
// non-ASCII bytes (see vector_plain_run()) vorrq_u8(vcltq_s8(vreinterpretq_s8_u8(in), vdupq_n_s8(0x20)), vceqq_u8(in, vdupq_n_u8(0x7F))));
const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('\\')), vceqq_u8(in, vdupq_n_u8(0x7F))),
vcltq_s8(vreinterpretq_s8_u8(veorq_u8(in, vdupq_n_u8(2))), vdupq_n_s8(33)));
if (vmaxvq_u8(special) != 0) if (vmaxvq_u8(special) != 0)
{ {
return false; return false;
@@ -848,8 +848,8 @@ class serializer
const __m128i lo = _mm_loadl_epi64(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void) const __m128i lo = _mm_loadl_epi64(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
const __m128i hi = _mm_loadl_epi64(static_cast<const __m128i*>(static_cast<const void*>(p + n - 8))); // NOLINT(bugprone-casting-through-void) const __m128i hi = _mm_loadl_epi64(static_cast<const __m128i*>(static_cast<const void*>(p + n - 8))); // NOLINT(bugprone-casting-through-void)
const __m128i in = _mm_unpacklo_epi64(lo, hi); const __m128i in = _mm_unpacklo_epi64(lo, hi);
const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('\\')), _mm_cmpeq_epi8(in, _mm_set1_epi8(0x7F))), const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_cmplt_epi8(_mm_xor_si128(in, _mm_set1_epi8(2)), _mm_set1_epi8(33))); _mm_or_si128(_mm_cmplt_epi8(in, _mm_set1_epi8(0x20)), _mm_cmpeq_epi8(in, _mm_set1_epi8(0x7F))));
if (_mm_movemask_epi8(special) != 0) if (_mm_movemask_epi8(special) != 0)
{ {
return false; return false;
@@ -868,13 +868,10 @@ class serializer
#if JSON_SCAN_NEON #if JSON_SCAN_NEON
const uint8x16_t a = vld1q_u8(reinterpret_cast<const std::uint8_t*>(p)); const uint8x16_t a = vld1q_u8(reinterpret_cast<const std::uint8_t*>(p));
const uint8x16_t b = vld1q_u8(reinterpret_cast<const std::uint8_t*>(p + n - 16)); const uint8x16_t b = vld1q_u8(reinterpret_cast<const std::uint8_t*>(p + n - 16));
// (int8_t)(c ^ 2) < 33: the quote, the control characters and the const uint8x16_t special = vorrq_u8(vorrq_u8(vorrq_u8(vceqq_u8(a, vdupq_n_u8('"')), vceqq_u8(a, vdupq_n_u8('\\'))),
// non-ASCII bytes (see vector_plain_run()) vorrq_u8(vceqq_u8(b, vdupq_n_u8('"')), vceqq_u8(b, vdupq_n_u8('\\')))),
const uint8x16_t flip = vdupq_n_u8(2); vorrq_u8(vorrq_u8(vcltq_s8(vreinterpretq_s8_u8(a), vdupq_n_s8(0x20)), vcltq_s8(vreinterpretq_s8_u8(b), vdupq_n_s8(0x20))),
const int8x16_t limit = vdupq_n_s8(33); vorrq_u8(vceqq_u8(a, vdupq_n_u8(0x7F)), vceqq_u8(b, vdupq_n_u8(0x7F)))));
const uint8x16_t special = vorrq_u8(vorrq_u8(vorrq_u8(vceqq_u8(a, vdupq_n_u8('\\')), vceqq_u8(b, vdupq_n_u8('\\'))),
vorrq_u8(vceqq_u8(a, vdupq_n_u8(0x7F)), vceqq_u8(b, vdupq_n_u8(0x7F)))),
vorrq_u8(vcltq_s8(vreinterpretq_s8_u8(veorq_u8(a, flip)), limit), vcltq_s8(vreinterpretq_s8_u8(veorq_u8(b, flip)), limit)));
if (vmaxvq_u8(special) != 0) if (vmaxvq_u8(special) != 0)
{ {
return false; return false;
@@ -884,15 +881,14 @@ class serializer
#else #else
const __m128i a = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void) const __m128i a = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
const __m128i b = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p + n - 16))); // NOLINT(bugprone-casting-through-void) const __m128i b = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p + n - 16))); // NOLINT(bugprone-casting-through-void)
// (int8_t)(c ^ 2) < 33: the quote, the control characters and the const __m128i quote = _mm_set1_epi8('"');
// non-ASCII bytes (see vector_plain_run())
const __m128i backslash = _mm_set1_epi8('\\'); const __m128i backslash = _mm_set1_epi8('\\');
const __m128i space = _mm_set1_epi8(0x20);
const __m128i del = _mm_set1_epi8(0x7F); const __m128i del = _mm_set1_epi8(0x7F);
const __m128i flip = _mm_set1_epi8(2); const __m128i special = _mm_or_si128(_mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(a, quote), _mm_cmpeq_epi8(a, backslash)),
const __m128i limit = _mm_set1_epi8(33); _mm_or_si128(_mm_cmpeq_epi8(b, quote), _mm_cmpeq_epi8(b, backslash))),
const __m128i special = _mm_or_si128(_mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(a, backslash), _mm_cmpeq_epi8(b, backslash)), _mm_or_si128(_mm_or_si128(_mm_cmplt_epi8(a, space), _mm_cmplt_epi8(b, space)),
_mm_or_si128(_mm_cmpeq_epi8(a, del), _mm_cmpeq_epi8(b, del))), _mm_or_si128(_mm_cmpeq_epi8(a, del), _mm_cmpeq_epi8(b, del))));
_mm_or_si128(_mm_cmplt_epi8(_mm_xor_si128(a, flip), limit), _mm_cmplt_epi8(_mm_xor_si128(b, flip), limit)));
if (_mm_movemask_epi8(special) != 0) if (_mm_movemask_epi8(special) != 0)
{ {
return false; return false;
+14 -20
View File
@@ -152,12 +152,9 @@ inline std::size_t plain_output_run(const unsigned char* s, std::size_t n) noexc
for (; i + 16 <= n; i += 16) for (; i + 16 <= n; i += 16)
{ {
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(s + i))); // NOLINT(bugprone-casting-through-void) const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(s + i))); // NOLINT(bugprone-casting-through-void)
// the quote and the control characters are the bytes with // (a byte is below 0x20 if it is its own minimum with 0x1F)
// (c ^ 2) <= 32, that is, those that are their own minimum with 32 const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
// (see vector_plain_run()) _mm_cmpeq_epi8(_mm_min_epu8(in, _mm_set1_epi8(0x1F)), in));
const __m128i flipped = _mm_xor_si128(in, _mm_set1_epi8(2));
const __m128i special = _mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('\\')),
_mm_cmpeq_epi8(_mm_min_epu8(flipped, _mm_set1_epi8(32)), flipped));
const auto mask = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special))); const auto mask = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
if (mask != 0) if (mask != 0)
{ {
@@ -220,21 +217,20 @@ inline bool is_ascii_without_delete(const char* s, std::size_t n) noexcept
#if NLOHMANN_VIEW_VECTOR #if NLOHMANN_VIEW_VECTOR
/// the bytes of the 16 at p that ensure_ascii cannot copy (a quote, a /// the bytes of the 16 at p that ensure_ascii cannot copy (a quote, a
/// backslash, a control character, DEL, or not ASCII; the signed compare /// backslash, a control character, DEL, or not ASCII; a signed compare with
/// (int8_t)(c ^ 2) < 33 finds the quote, the control characters and the /// 0x20 finds control characters and non-ASCII bytes at once), one bit per
/// non-ASCII bytes at once, see vector_plain_run()), one bit per byte (x86) or /// byte (x86) or one nibble per byte (NEON)
/// one nibble per byte (NEON)
NLOHMANN_VIEW_ALWAYS_INLINE std::uint64_t ascii_stop_mask(const unsigned char* p) noexcept NLOHMANN_VIEW_ALWAYS_INLINE std::uint64_t ascii_stop_mask(const unsigned char* p) noexcept
{ {
#if NLOHMANN_VIEW_NEON #if NLOHMANN_VIEW_NEON
const uint8x16_t in = vld1q_u8(p); const uint8x16_t in = vld1q_u8(p);
const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('\\')), vceqq_u8(in, vdupq_n_u8(0x7F))), const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))),
vcltq_s8(vreinterpretq_s8_u8(veorq_u8(in, vdupq_n_u8(2))), vdupq_n_s8(33))); vorrq_u8(vceqq_u8(in, vdupq_n_u8(0x7F)), vcltq_s8(vreinterpretq_s8_u8(in), vdupq_n_s8(0x20))));
return vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0); return vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0);
#else #else
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void) const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('\\')), _mm_cmpeq_epi8(in, _mm_set1_epi8(0x7F))), const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_cmplt_epi8(_mm_xor_si128(in, _mm_set1_epi8(2)), _mm_set1_epi8(33))); _mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8(0x7F)), _mm_cmplt_epi8(in, _mm_set1_epi8(0x20))));
return static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special))); return static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
#endif #endif
} }
@@ -397,7 +393,7 @@ struct depth_counter<false>
/*! /*!
@brief where the double x nearest to a token w * 10^q lies, for 10^15 <= w < @brief where the double x nearest to a token w * 10^q lies, for 10^15 <= w <
10^17 and -16 <= q <= -1 (Eisel-Lemire, then a fixed-point position) 10^17 and -17 <= q <= -1 (Eisel-Lemire, then a fixed-point position)
dump() writes the shortest decimal in the rounding interval of x (the closest dump() writes the shortest decimal in the rounding interval of x (the closest
one if there are several, see zmij::to_shortest()). Its digits are those of a one if there are several, see zmij::to_shortest()). Its digits are those of a
@@ -1262,15 +1258,13 @@ class view_serializer
} }
for (; p != stop && batch.count != float_batch::capacity(); ++p) for (; p != stop && batch.count != float_batch::capacity(); ++p)
{ {
// a token of the source: 16 or 17 digits, 8 to 16 after the // a token of the source: 16 or 17 digits, at least 8 after the
// point, no exponent, 32 bytes to read (17 after the point, with // point, no exponent, 32 bytes to read
// no digit before it, only in a damaged image loaded with
// image_check::bounds: beyond the range of locate_token())
const unsigned frac_digits = p->extra >> 8u; const unsigned frac_digits = p->extra >> 8u;
const unsigned int_digits = p->extra & 0xFFu; const unsigned int_digits = p->extra & 0xFFu;
const unsigned count = int_digits + frac_digits; const unsigned count = int_digits + frac_digits;
if (p->kind == node_kind::number_float && (p->flags & node_flags::storage) == 0 if (p->kind == node_kind::number_float && (p->flags & node_flags::storage) == 0
&& count - 16u <= 1u && frac_digits - 8u <= 8u && src_end - (src + p->off) >= 33) && count - 16u <= 1u && frac_digits >= 8 && src_end - (src + p->off) >= 33)
{ {
const auto* const token = reinterpret_cast<const unsigned char*>(src + p->off); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast) const auto* const token = reinterpret_cast<const unsigned char*>(src + p->off); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
const unsigned sign = token[0] == '-' ? 1 : 0; const unsigned sign = token[0] == '-' ? 1 : 0;
+13 -20
View File
@@ -91,10 +91,8 @@ namespace view
character, or not ASCII, 16 bytes per step character, or not ASCII, 16 bytes per step
Stops at such a byte, or where fewer than 16 bytes are left (the caller tells Stops at such a byte, or where fewer than 16 bytes are left (the caller tells
the two apart). One signed compare finds the quote, the control characters and the two apart). A signed compare with 0x20 finds control characters and
the non-ASCII bytes at once: these are exactly the bytes with non-ASCII bytes at once.
(int8_t)(c ^ 2) < 33 (see detail::vector_plain_run()). The backslash needs its
own compare.
*/ */
NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned char* p, const unsigned char* e) noexcept NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned char* p, const unsigned char* e) noexcept
{ {
@@ -102,8 +100,8 @@ NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned
{ {
#if NLOHMANN_VIEW_NEON #if NLOHMANN_VIEW_NEON
const uint8x16_t in = vld1q_u8(p); const uint8x16_t in = vld1q_u8(p);
const uint8x16_t special = vorrq_u8(vceqq_u8(in, vdupq_n_u8('\\')), const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))),
vcltq_s8(vreinterpretq_s8_u8(veorq_u8(in, vdupq_n_u8(2))), vdupq_n_s8(33))); vcltq_s8(vreinterpretq_s8_u8(in), vdupq_n_s8(0x20)));
// one nibble per byte (the usual NEON replacement of x86's movemask, see // one nibble per byte (the usual NEON replacement of x86's movemask, see
// D. Kutenin, "Porting x86 vector bitmask optimizations to Arm NEON", 2022) // D. Kutenin, "Porting x86 vector bitmask optimizations to Arm NEON", 2022)
const std::uint64_t bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0); const std::uint64_t bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0);
@@ -113,8 +111,8 @@ NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned
} }
#else #else
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void) const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
const __m128i special = _mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('\\')), const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_cmplt_epi8(_mm_xor_si128(in, _mm_set1_epi8(2)), _mm_set1_epi8(33))); _mm_cmplt_epi8(in, _mm_set1_epi8(0x20)));
const auto bits = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special))); const auto bits = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
if (bits != 0) if (bits != 0)
{ {
@@ -140,17 +138,15 @@ NLOHMANN_VIEW_ALWAYS_INLINE unsigned copy_plain_run16(const unsigned char* p, ch
#if NLOHMANN_VIEW_NEON #if NLOHMANN_VIEW_NEON
const uint8x16_t in = vld1q_u8(p); const uint8x16_t in = vld1q_u8(p);
vst1q_u8(reinterpret_cast<std::uint8_t*>(w), in); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast) vst1q_u8(reinterpret_cast<std::uint8_t*>(w), in); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
// (int8_t)(c ^ 2) < 33: the quote, the control characters and the const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))),
// non-ASCII bytes (see vector_plain_run()) vcltq_s8(vreinterpretq_s8_u8(in), vdupq_n_s8(0x20)));
const uint8x16_t special = vorrq_u8(vceqq_u8(in, vdupq_n_u8('\\')),
vcltq_s8(vreinterpretq_s8_u8(veorq_u8(in, vdupq_n_u8(2))), vdupq_n_s8(33)));
const std::uint64_t bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0); const std::uint64_t bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0);
return bits != 0 ? static_cast<unsigned>(count_trailing_zeros(bits) >> 2u) : 16u; return bits != 0 ? static_cast<unsigned>(count_trailing_zeros(bits) >> 2u) : 16u;
#else #else
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void) const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
_mm_storeu_si128(static_cast<__m128i*>(static_cast<void*>(w)), in); // NOLINT(bugprone-casting-through-void) _mm_storeu_si128(static_cast<__m128i*>(static_cast<void*>(w)), in); // NOLINT(bugprone-casting-through-void)
const __m128i special = _mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('\\')), const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_cmplt_epi8(_mm_xor_si128(in, _mm_set1_epi8(2)), _mm_set1_epi8(33))); _mm_cmplt_epi8(in, _mm_set1_epi8(0x20)));
const auto bits = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special))); const auto bits = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
return bits != 0 ? static_cast<unsigned>(count_trailing_zeros(bits)) : 16u; return bits != 0 ? static_cast<unsigned>(count_trailing_zeros(bits)) : 16u;
#endif #endif
@@ -310,8 +306,7 @@ NLOHMANN_VIEW_SSSE3_TARGET NLOHMANN_VIEW_NOINLINE NLOHMANN_VIEW_NOCLONE inline c
while (e - block >= 16) while (e - block >= 16)
{ {
const uint8x16_t in = vld1q_u8(block); const uint8x16_t in = vld1q_u8(block);
// the quote and the control characters: (c ^ 2) < 33 (see vector_plain_run()) const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))), vcltq_u8(in, vdupq_n_u8(0x20)));
const uint8x16_t special = vorrq_u8(vceqq_u8(in, vdupq_n_u8('\\')), vcltq_u8(veorq_u8(in, vdupq_n_u8(2)), vdupq_n_u8(33)));
const uint8x16_t prev1 = vextq_u8(prev, in, 15); const uint8x16_t prev1 = vextq_u8(prev, in, 15);
const uint8x16_t sc = vandq_u8(vandq_u8(vqtbl1q_u8(t1h, vshrq_n_u8(prev1, 4)), vqtbl1q_u8(t1l, vandq_u8(prev1, vdupq_n_u8(0x0F)))), vqtbl1q_u8(t2h, vshrq_n_u8(in, 4))); const uint8x16_t sc = vandq_u8(vandq_u8(vqtbl1q_u8(t1h, vshrq_n_u8(prev1, 4)), vqtbl1q_u8(t1l, vandq_u8(prev1, vdupq_n_u8(0x0F)))), vqtbl1q_u8(t2h, vshrq_n_u8(in, 4)));
const uint8x16_t must23 = vorrq_u8(vqsubq_u8(vextq_u8(prev, in, 14), vdupq_n_u8(0xE0 - 0x80)), vqsubq_u8(vextq_u8(prev, in, 13), vdupq_n_u8(0xF0 - 0x80))); const uint8x16_t must23 = vorrq_u8(vqsubq_u8(vextq_u8(prev, in, 14), vdupq_n_u8(0xE0 - 0x80)), vqsubq_u8(vextq_u8(prev, in, 13), vdupq_n_u8(0xF0 - 0x80)));
@@ -350,10 +345,8 @@ NLOHMANN_VIEW_SSSE3_TARGET NLOHMANN_VIEW_NOINLINE NLOHMANN_VIEW_NOCLONE inline c
while (e - block >= 16) while (e - block >= 16)
{ {
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void) const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void)
// the quote and the control characters: (c ^ 2) < 33, that is, const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
// (c ^ 2) - 32 saturates to 0 (see vector_plain_run()) _mm_cmpeq_epi8(_mm_subs_epu8(in, _mm_set1_epi8(0x1F)), zero)); // in < 0x20
const __m128i special = _mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('\\')),
_mm_cmpeq_epi8(_mm_subs_epu8(_mm_xor_si128(in, _mm_set1_epi8(2)), _mm_set1_epi8(32)), zero));
const __m128i prev1 = _mm_alignr_epi8(in, prev, 15); const __m128i prev1 = _mm_alignr_epi8(in, prev, 15);
const __m128i sc = _mm_and_si128(_mm_and_si128(_mm_shuffle_epi8(t1h, _mm_and_si128(_mm_srli_epi16(prev1, 4), nibble)), const __m128i sc = _mm_and_si128(_mm_and_si128(_mm_shuffle_epi8(t1h, _mm_and_si128(_mm_srli_epi16(prev1, 4), nibble)),
_mm_shuffle_epi8(t1l, _mm_and_si128(prev1, nibble))), _mm_shuffle_epi8(t1l, _mm_and_si128(prev1, nibble))),
+114 -172
View File
@@ -6928,37 +6928,6 @@ void from_json(const BasicJsonType& j, std::unordered_map<Key, Value, Hash, KeyE
from_json_pair_array_to_map(j, m); from_json_pair_array_to_map(j, m);
} }
#if defined(__cpp_lib_char8_t)
// a std::u8string is read from a JSON string; the array of code units written by
// versions before 3.13.0 is still accepted
template < typename BasicJsonType, typename Tr, typename Allocator,
enable_if_t < std::is_same<typename json_type_of_t<BasicJsonType>::string_t::value_type, char>::value, int > = 0 >
inline void from_json(const BasicJsonType& j, std::basic_string<char8_t, Tr, Allocator>& s)
{
if (j.is_string())
{
const auto& str = string_value(j);
const char* const data = str.data();
const std::size_t size = str.size();
std::basic_string<char8_t, Tr, Allocator> ret(s.get_allocator());
ret.reserve(size);
for (std::size_t i = 0; i < size; ++i)
{
ret.push_back(static_cast<char8_t>(static_cast<unsigned char>(data[i])));
}
s = std::move(ret);
}
else if (j.is_array())
{
from_json_array_impl(j, s, priority_tag<3> {});
}
else
{
throw_type_must_be("string or array", j);
}
}
#endif
#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM #if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
// Workaround for MSVC 19.51 (and possibly later): in large cpp files, the compiler may fail to resolve with generic has_from_json (issue #4996) // Workaround for MSVC 19.51 (and possibly later): in large cpp files, the compiler may fail to resolve with generic has_from_json (issue #4996)
@@ -8360,20 +8329,16 @@ inline void to_json(BasicJsonType& j, const T& t)
to_json_tuple_impl(j, t, make_index_sequence<std::tuple_size<T>::value> {}); to_json_tuple_impl(j, t, make_index_sequence<std::tuple_size<T>::value> {});
} }
// a std::u8string is a JSON string (it was an array of code units before 3.13.0); #if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
// defined outside the std::filesystem guard so that the result does not depend
// on whether std::filesystem::path is supported
#if defined(__cpp_lib_char8_t) #if defined(__cpp_lib_char8_t)
template<typename BasicJsonType, typename Tr, typename Allocator> template<typename BasicJsonType, typename Tr, typename Allocator>
inline void to_json(BasicJsonType& j, const std::basic_string<char8_t, Tr, Allocator>& s) inline void to_json(BasicJsonType& j, const std::basic_string<char8_t, Tr, Allocator>& s)
{ {
using string_t = typename BasicJsonType::string_t; using OtherAllocator = typename std::allocator_traits<Allocator>::template rebind_alloc<char>;
// char may alias any object; the code units are copied unchanged j = std::basic_string<char, std::char_traits<char>, OtherAllocator>(s.begin(), s.end(), s.get_allocator());
j = string_t(reinterpret_cast<const char*>(s.data()), s.size());
} }
#endif #endif
#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
// Workaround for MSVC 19.51 (and possibly later): in large cpp files, the compiler may fail to resolve with generic has_to_json (issue #4996) // Workaround for MSVC 19.51 (and possibly later): in large cpp files, the compiler may fail to resolve with generic has_to_json (issue #4996)
template<typename BasicJsonType> template<typename BasicJsonType>
struct has_to_json<BasicJsonType, std_fs::path, void> : std::true_type {}; struct has_to_json<BasicJsonType, std_fs::path, void> : std::true_type {};
@@ -10492,64 +10457,6 @@ struct iterator_input_adapter_factory<IteratorType, SentinelType, enable_if_t<is
} }
}; };
/*!
@brief whether @a IteratorType is an iterator of std::string or of a
std::vector of 1-byte elements
Their elements are contiguous, but before C++20 that cannot be detected from
the iterator type (std::contiguous_iterator), so an iterator pair of them would
be read one character at a time through the iterators. They are read as the
pointer range they are instead, as the containers themselves are (see the
container overload of input_adapter()). The types are matched exactly, so no
other iterator can be mistaken for one. (std::basic_string is only matched for
char: other character types need std::char_traits specializations that the
standard library need not have.)
*/
template<typename CharType>
struct is_byte_char_type
{
// a member, not an integral_constant argument, which astyle misformats
static constexpr bool value = sizeof(CharType) == 1 && std::is_integral<CharType>::value && !std::is_same<CharType, bool>::value;
};
template<typename IteratorType, typename CharType, bool IsByte = is_byte_char_type<CharType>::value>
struct is_std_contiguous_byte_iterator_impl : std::false_type {};
// (the containers are only named for byte types)
template<typename IteratorType, typename CharType>
struct is_std_contiguous_byte_iterator_impl<IteratorType, CharType, true>
{
static constexpr bool value = !std::is_pointer<IteratorType>::value
&& (std::is_same<IteratorType, typename std::vector<CharType>::iterator>::value
|| std::is_same<IteratorType, typename std::vector<CharType>::const_iterator>::value
|| std::is_same<IteratorType, std::string::iterator>::value
|| std::is_same<IteratorType, std::string::const_iterator>::value);
};
template<typename IteratorType>
struct is_std_contiguous_byte_iterator
: is_std_contiguous_byte_iterator_impl<IteratorType, typename std::remove_cv<typename std::iterator_traits<IteratorType>::value_type>::type> {};
template<typename IteratorType>
struct iterator_input_adapter_factory<IteratorType, IteratorType, enable_if_t<is_std_contiguous_byte_iterator<IteratorType>::value>>
{
using iterator_type = IteratorType;
using sentinel_type = IteratorType;
using char_type = typename std::remove_cv<typename std::iterator_traits<iterator_type>::value_type>::type;
using adapter_type = iterator_input_adapter<const char_type*>;
static adapter_type create(IteratorType first, IteratorType last)
{
if (first == last)
{
// (an empty range must not be dereferenced)
return adapter_type(nullptr, nullptr);
}
const char_type* const begin = std::addressof(*first);
return adapter_type(begin, begin + (last - first));
}
};
// General purpose iterator-based input (iterator+sentinel pair; SentinelType // General purpose iterator-based input (iterator+sentinel pair; SentinelType
// defaults to IteratorType for the common same-type case, but may differ for // defaults to IteratorType for the common same-type case, but may differ for
// C++20 ranges-style iterator+sentinel pairs). Only enable for types that can // C++20 ranges-style iterator+sentinel pairs). Only enable for types that can
@@ -12454,12 +12361,8 @@ namespace detail
character, or not ASCII, 16 bytes per step character, or not ASCII, 16 bytes per step
Stops at such a byte, or where fewer than 16 bytes are left (the caller tells Stops at such a byte, or where fewer than 16 bytes are left (the caller tells
the two apart). One signed compare finds the quote, the control characters and the two apart). A signed compare with 0x20 finds control characters and
the non-ASCII bytes at once: flipping bit 1 maps 0x00-0x1F onto itself and the non-ASCII bytes at once.
quote (0x22) onto 0x20, keeps 0x80-0xFF negative, and maps every other byte to
33 or more, so these are exactly the bytes with (int8_t)(c ^ 2) < 33 (see
D. Lemire, "Detect control characters, quotes and backslashes efficiently
using SWAR", 2025). The backslash needs its own compare.
*/ */
JSON_HEDLEY_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned char* p, const unsigned char* e) noexcept JSON_HEDLEY_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned char* p, const unsigned char* e) noexcept
{ {
@@ -12467,8 +12370,8 @@ JSON_HEDLEY_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned c
{ {
#if JSON_SCAN_NEON #if JSON_SCAN_NEON
const uint8x16_t in = vld1q_u8(p); const uint8x16_t in = vld1q_u8(p);
const uint8x16_t special = vorrq_u8(vceqq_u8(in, vdupq_n_u8('\\')), const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))),
vcltq_s8(vreinterpretq_s8_u8(veorq_u8(in, vdupq_n_u8(2))), vdupq_n_s8(33))); vcltq_s8(vreinterpretq_s8_u8(in), vdupq_n_s8(0x20)));
// one nibble per byte (the usual NEON replacement of x86's movemask, see // one nibble per byte (the usual NEON replacement of x86's movemask, see
// D. Kutenin, "Porting x86 vector bitmask optimizations to Arm NEON", 2022) // D. Kutenin, "Porting x86 vector bitmask optimizations to Arm NEON", 2022)
const std::uint64_t bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0); const std::uint64_t bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0);
@@ -12478,8 +12381,8 @@ JSON_HEDLEY_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned c
} }
#else #else
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void) const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
const __m128i special = _mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('\\')), const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_cmplt_epi8(_mm_xor_si128(in, _mm_set1_epi8(2)), _mm_set1_epi8(33))); _mm_cmplt_epi8(in, _mm_set1_epi8(0x20)));
const auto bits = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special))); const auto bits = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
if (bits != 0) if (bits != 0)
{ {
@@ -12616,8 +12519,7 @@ JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline bool vector_utf8_run(cons
while (e - block >= 16) while (e - block >= 16)
{ {
const uint8x16_t in = vld1q_u8(block); const uint8x16_t in = vld1q_u8(block);
// the quote and the control characters: (c ^ 2) < 33 (see vector_plain_run()) const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))), vcltq_u8(in, vdupq_n_u8(0x20)));
const uint8x16_t special = vorrq_u8(vceqq_u8(in, vdupq_n_u8('\\')), vcltq_u8(veorq_u8(in, vdupq_n_u8(2)), vdupq_n_u8(33)));
const uint8x16_t prev1 = vextq_u8(prev, in, 15); const uint8x16_t prev1 = vextq_u8(prev, in, 15);
const uint8x16_t sc = vandq_u8(vandq_u8(vqtbl1q_u8(t1h, vshrq_n_u8(prev1, 4)), vqtbl1q_u8(t1l, vandq_u8(prev1, vdupq_n_u8(0x0F)))), vqtbl1q_u8(t2h, vshrq_n_u8(in, 4))); const uint8x16_t sc = vandq_u8(vandq_u8(vqtbl1q_u8(t1h, vshrq_n_u8(prev1, 4)), vqtbl1q_u8(t1l, vandq_u8(prev1, vdupq_n_u8(0x0F)))), vqtbl1q_u8(t2h, vshrq_n_u8(in, 4)));
const uint8x16_t must23 = vorrq_u8(vqsubq_u8(vextq_u8(prev, in, 14), vdupq_n_u8(0xE0 - 0x80)), vqsubq_u8(vextq_u8(prev, in, 13), vdupq_n_u8(0xF0 - 0x80))); const uint8x16_t must23 = vorrq_u8(vqsubq_u8(vextq_u8(prev, in, 14), vdupq_n_u8(0xE0 - 0x80)), vqsubq_u8(vextq_u8(prev, in, 13), vdupq_n_u8(0xF0 - 0x80)));
@@ -12657,10 +12559,8 @@ JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline bool vector_utf8_run(cons
while (e - block >= 16) while (e - block >= 16)
{ {
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void) const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void)
// the quote and the control characters: (c ^ 2) < 33, that is, const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
// (c ^ 2) - 32 saturates to 0 (see vector_plain_run()) _mm_cmpeq_epi8(_mm_subs_epu8(in, _mm_set1_epi8(0x1F)), zero)); // in < 0x20
const __m128i special = _mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('\\')),
_mm_cmpeq_epi8(_mm_subs_epu8(_mm_xor_si128(in, _mm_set1_epi8(2)), _mm_set1_epi8(32)), zero));
const __m128i prev1 = _mm_alignr_epi8(in, prev, 15); const __m128i prev1 = _mm_alignr_epi8(in, prev, 15);
const __m128i sc = _mm_and_si128(_mm_and_si128(_mm_shuffle_epi8(t1h, _mm_and_si128(_mm_srli_epi16(prev1, 4), nibble)), const __m128i sc = _mm_and_si128(_mm_and_si128(_mm_shuffle_epi8(t1h, _mm_and_si128(_mm_srli_epi16(prev1, 4), nibble)),
_mm_shuffle_epi8(t1l, _mm_and_si128(prev1, nibble))), _mm_shuffle_epi8(t1l, _mm_and_si128(prev1, nibble))),
@@ -12693,15 +12593,23 @@ JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline bool vector_utf8_run(cons
} }
/*! /*!
@brief skip 16-byte blocks of well-formed UTF-8 at p (a character boundary) @brief vector_utf8_valid_blocks() from the first block that is not ASCII
Like vector_utf8_run, but quotes, backslashes, and control characters are @a block is that block, and @a first is where the check began (only ASCII up
ordinary characters. Returns the start of the first block in which the check to @a block). Returns where fewer than 16 bytes are left, or @a first if the
found an error (which may belong to a sequence begun in the block before) or check found an error in any block.
where fewer than 16 bytes are left; a sequence may be cut at that point, so the
caller resumes at its start, one sequence at a time. From a block that is not ASCII on, eight blocks are checked without testing
for ASCII again: in text with a few non-ASCII characters (accented letters),
every other block has one, and the test would be a mispredicted branch for
most blocks. After them, ASCII blocks are skipped with one test again.
The errors of all blocks are collected and tested once at the end. A test
after each block lets the compiler advance the position with a conditional
select on its result (Apple Clang 17 does), and each block then waits for the
check of the block before: 28 cycles for 16 bytes instead of 4 on Apple M1.
*/ */
JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline const unsigned char* vector_utf8_valid_blocks(const unsigned char* block, const unsigned char* e) noexcept JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline const unsigned char* vector_utf8_check_blocks(const unsigned char* first, const unsigned char* block, const unsigned char* e) noexcept
{ {
using lookup = utf8_lookup4<>; using lookup = utf8_lookup4<>;
#if JSON_SCAN_NEON #if JSON_SCAN_NEON
@@ -12709,27 +12617,29 @@ JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline const unsigned char* vect
const uint8x16_t t1l = vld1q_u8(lookup::byte_1_low.data()); const uint8x16_t t1l = vld1q_u8(lookup::byte_1_low.data());
const uint8x16_t t2h = vld1q_u8(lookup::byte_2_high.data()); const uint8x16_t t2h = vld1q_u8(lookup::byte_2_high.data());
uint8x16_t prev = vdupq_n_u8(0); uint8x16_t prev = vdupq_n_u8(0);
uint8x16_t error = vdupq_n_u8(0);
while (e - block >= 16) while (e - block >= 16)
{ {
const uint8x16_t in = vld1q_u8(block); const uint8x16_t ahead = vld1q_u8(block);
if (vmaxvq_u8(vorrq_u8(in, prev)) < 0x80) if (vmaxvq_u8(vorrq_u8(ahead, prev)) < 0x80)
{ {
// ASCII here, and no sequence from the block before to finish // ASCII here, and no sequence from the block before to finish
prev = in; prev = ahead;
block += 16; block += 16;
continue; continue;
} }
const uint8x16_t prev1 = vextq_u8(prev, in, 15); for (int blocks = 8; blocks != 0 && e - block >= 16; --blocks)
const uint8x16_t sc = vandq_u8(vandq_u8(vqtbl1q_u8(t1h, vshrq_n_u8(prev1, 4)), vqtbl1q_u8(t1l, vandq_u8(prev1, vdupq_n_u8(0x0F)))), vqtbl1q_u8(t2h, vshrq_n_u8(in, 4)));
const uint8x16_t must23 = vorrq_u8(vqsubq_u8(vextq_u8(prev, in, 14), vdupq_n_u8(0xE0 - 0x80)), vqsubq_u8(vextq_u8(prev, in, 13), vdupq_n_u8(0xF0 - 0x80)));
const uint8x16_t err = veorq_u8(vandq_u8(must23, vdupq_n_u8(0x80)), sc);
if (vmaxvq_u8(err) != 0)
{ {
break; const uint8x16_t in = vld1q_u8(block);
const uint8x16_t prev1 = vextq_u8(prev, in, 15);
const uint8x16_t sc = vandq_u8(vandq_u8(vqtbl1q_u8(t1h, vshrq_n_u8(prev1, 4)), vqtbl1q_u8(t1l, vandq_u8(prev1, vdupq_n_u8(0x0F)))), vqtbl1q_u8(t2h, vshrq_n_u8(in, 4)));
const uint8x16_t must23 = vorrq_u8(vqsubq_u8(vextq_u8(prev, in, 14), vdupq_n_u8(0xE0 - 0x80)), vqsubq_u8(vextq_u8(prev, in, 13), vdupq_n_u8(0xF0 - 0x80)));
error = vorrq_u8(error, veorq_u8(vandq_u8(must23, vdupq_n_u8(0x80)), sc));
prev = in;
block += 16;
} }
prev = in;
block += 16;
} }
return vmaxvq_u8(error) != 0 ? first : block;
#else #else
const __m128i t1h = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_1_high.data()))); // NOLINT(bugprone-casting-through-void) const __m128i t1h = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_1_high.data()))); // NOLINT(bugprone-casting-through-void)
const __m128i t1l = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_1_low.data()))); // NOLINT(bugprone-casting-through-void) const __m128i t1l = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_1_low.data()))); // NOLINT(bugprone-casting-through-void)
@@ -12737,30 +12647,62 @@ JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline const unsigned char* vect
const __m128i nibble = _mm_set1_epi8(0x0F); const __m128i nibble = _mm_set1_epi8(0x0F);
const __m128i zero = _mm_setzero_si128(); const __m128i zero = _mm_setzero_si128();
__m128i prev = zero; __m128i prev = zero;
__m128i error = zero;
while (e - block >= 16) while (e - block >= 16)
{ {
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void) const __m128i ahead = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void)
if (_mm_movemask_epi8(_mm_or_si128(in, prev)) == 0) if (_mm_movemask_epi8(_mm_or_si128(ahead, prev)) == 0)
{ {
prev = in; prev = ahead;
block += 16; block += 16;
continue; continue;
} }
const __m128i prev1 = _mm_alignr_epi8(in, prev, 15); for (int blocks = 8; blocks != 0 && e - block >= 16; --blocks)
const __m128i sc = _mm_and_si128(_mm_and_si128(_mm_shuffle_epi8(t1h, _mm_and_si128(_mm_srli_epi16(prev1, 4), nibble)),
_mm_shuffle_epi8(t1l, _mm_and_si128(prev1, nibble))),
_mm_shuffle_epi8(t2h, _mm_and_si128(_mm_srli_epi16(in, 4), nibble)));
const __m128i must23 = _mm_or_si128(_mm_subs_epu8(_mm_alignr_epi8(in, prev, 14), _mm_set1_epi8(0xE0 - 0x80)),
_mm_subs_epu8(_mm_alignr_epi8(in, prev, 13), _mm_set1_epi8(0xF0 - 0x80)));
const __m128i err = _mm_xor_si128(_mm_and_si128(must23, _mm_set1_epi8(static_cast<char>(-128))), sc);
if (_mm_movemask_epi8(_mm_cmpeq_epi8(err, zero)) != 0xFFFF)
{ {
break; const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void)
const __m128i prev1 = _mm_alignr_epi8(in, prev, 15);
const __m128i sc = _mm_and_si128(_mm_and_si128(_mm_shuffle_epi8(t1h, _mm_and_si128(_mm_srli_epi16(prev1, 4), nibble)),
_mm_shuffle_epi8(t1l, _mm_and_si128(prev1, nibble))),
_mm_shuffle_epi8(t2h, _mm_and_si128(_mm_srli_epi16(in, 4), nibble)));
const __m128i must23 = _mm_or_si128(_mm_subs_epu8(_mm_alignr_epi8(in, prev, 14), _mm_set1_epi8(0xE0 - 0x80)),
_mm_subs_epu8(_mm_alignr_epi8(in, prev, 13), _mm_set1_epi8(0xF0 - 0x80)));
error = _mm_or_si128(error, _mm_xor_si128(_mm_and_si128(must23, _mm_set1_epi8(static_cast<char>(-128))), sc));
prev = in;
block += 16;
}
}
return _mm_movemask_epi8(_mm_cmpeq_epi8(error, zero)) != 0xFFFF ? first : block;
#endif
}
/*!
@brief skip 16-byte blocks of well-formed UTF-8 at p (a character boundary)
Like vector_utf8_run, but quotes, backslashes, and control characters are
ordinary characters. Returns where fewer than 16 bytes are left; a sequence may
be cut at that point, so the caller resumes at its start, one sequence at a
time. If the check found an error in any block, @a block itself is returned,
for the caller to find the error one sequence at a time.
ASCII blocks need one test each and none of the tables of the UTF-8 check,
which starts at the first block that is not ASCII (see
vector_utf8_check_blocks()). Out of line, like the check.
*/
JSON_HEDLEY_NEVER_INLINE inline const unsigned char* vector_utf8_valid_blocks(const unsigned char* block, const unsigned char* e) noexcept
{
const unsigned char* const first = block;
while (e - block >= 16)
{
#if JSON_SCAN_NEON
if (JSON_HEDLEY_UNLIKELY(vmaxvq_u8(vld1q_u8(block)) >= 0x80))
#else
if (JSON_HEDLEY_UNLIKELY(_mm_movemask_epi8(_mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block)))) != 0)) // NOLINT(bugprone-casting-through-void)
#endif
{
return vector_utf8_check_blocks(first, block, e);
} }
prev = in;
block += 16; block += 16;
} }
#endif
return block; return block;
} }
#endif #endif
@@ -12812,10 +12754,11 @@ inline std::uint64_t swar_string_special(std::uint64_t v) noexcept
constexpr std::uint64_t high = 0x8080808080808080ull; constexpr std::uint64_t high = 0x8080808080808080ull;
const std::uint64_t q = v ^ 0x2222222222222222ull; // '"' (0x22) const std::uint64_t q = v ^ 0x2222222222222222ull; // '"' (0x22)
const std::uint64_t b = v ^ 0x5C5C5C5C5C5C5C5Cull; // '\\' (0x5C) const std::uint64_t b = v ^ 0x5C5C5C5C5C5C5C5Cull; // '\\' (0x5C)
// The usual test for a zero byte is (x - ones) & ~x & high. The `& ~x` const std::uint64_t has_quote = (q - ones) & ~q & high;
// only removes bytes from 0x80, and `v & high` flags those anyway, so it const std::uint64_t has_backslash = (b - ones) & ~b & high;
// can be dropped. Clang finds this reduction itself; GCC does not. const std::uint64_t has_control = (v - 0x2020202020202020ull) & ~v & high; // < 0x20
return ((q - ones) | (b - ones) | (v - 0x2020202020202020ull) | v) & high; const std::uint64_t has_non_ascii = v & high; // >= 0x80
return has_quote | has_backslash | has_control | has_non_ascii;
} }
// return the index of the first is_string_special() byte in [data, data+n), or // return the index of the first is_string_special() byte in [data, data+n), or
@@ -12896,8 +12839,11 @@ inline std::size_t find_ascii_copyable_run(const unsigned char* data, std::size_
const std::uint64_t q = v ^ 0x2222222222222222ull; // '"' (0x22) const std::uint64_t q = v ^ 0x2222222222222222ull; // '"' (0x22)
const std::uint64_t b = v ^ 0x5C5C5C5C5C5C5C5Cull; // '\\' (0x5C) const std::uint64_t b = v ^ 0x5C5C5C5C5C5C5C5Cull; // '\\' (0x5C)
const std::uint64_t d = v ^ 0x7F7F7F7F7F7F7F7Full; // DEL (0x7F) const std::uint64_t d = v ^ 0x7F7F7F7F7F7F7F7Full; // DEL (0x7F)
// (see swar_string_special(): the bytes from 0x80 are stops anyway) const std::uint64_t stop = ((q - ones) & ~q & high) // == '"'
const std::uint64_t stop = ((q - ones) | (b - ones) | (d - ones) | (v - 0x2020202020202020ull) | v) & high; | ((b - ones) & ~b & high) // == '\\'
| ((d - ones) & ~d & high) // == 0x7F
| ((v - 0x2020202020202020ull) & ~v & high) // < 0x20
| (v & high); // >= 0x80
if (stop != 0) if (stop != 0)
{ {
// the lowest flagged byte is the first one to stop at (see // the lowest flagged byte is the first one to stop at (see
@@ -37334,8 +37280,10 @@ class serializer
constexpr std::uint64_t ones = 0x0101010101010101ull; constexpr std::uint64_t ones = 0x0101010101010101ull;
const std::uint64_t q = v ^ 0x2222222222222222ull; // '"' const std::uint64_t q = v ^ 0x2222222222222222ull; // '"'
const std::uint64_t b = v ^ 0x5C5C5C5C5C5C5C5Cull; // '\\' const std::uint64_t b = v ^ 0x5C5C5C5C5C5C5C5Cull; // '\\'
// (see swar_string_special(): the bytes from 0x80 are flagged anyway) return (((q - ones) & ~q) | ((b - ones) & ~b) // == '"', == '\\'
return ((q - ones) | (b - ones) | (v - 0x2020202020202020ull) | v | (v + ones)) & 0x8080808080808080ull; | ((v - 0x2020202020202020ull) & ~v) // < 0x20
| v | (v + ones)) // >= 0x7F
& 0x8080808080808080ull;
} }
template<typename WordType> template<typename WordType>
@@ -37444,10 +37392,8 @@ class serializer
const uint8x8_t lo = vld1_u8(reinterpret_cast<const std::uint8_t*>(p)); const uint8x8_t lo = vld1_u8(reinterpret_cast<const std::uint8_t*>(p));
const uint8x8_t hi = vld1_u8(reinterpret_cast<const std::uint8_t*>(p + n - 8)); const uint8x8_t hi = vld1_u8(reinterpret_cast<const std::uint8_t*>(p + n - 8));
const uint8x16_t in = vcombine_u8(lo, hi); const uint8x16_t in = vcombine_u8(lo, hi);
// (int8_t)(c ^ 2) < 33: the quote, the control characters and the const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))),
// non-ASCII bytes (see vector_plain_run()) vorrq_u8(vcltq_s8(vreinterpretq_s8_u8(in), vdupq_n_s8(0x20)), vceqq_u8(in, vdupq_n_u8(0x7F))));
const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('\\')), vceqq_u8(in, vdupq_n_u8(0x7F))),
vcltq_s8(vreinterpretq_s8_u8(veorq_u8(in, vdupq_n_u8(2))), vdupq_n_s8(33)));
if (vmaxvq_u8(special) != 0) if (vmaxvq_u8(special) != 0)
{ {
return false; return false;
@@ -37458,8 +37404,8 @@ class serializer
const __m128i lo = _mm_loadl_epi64(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void) const __m128i lo = _mm_loadl_epi64(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
const __m128i hi = _mm_loadl_epi64(static_cast<const __m128i*>(static_cast<const void*>(p + n - 8))); // NOLINT(bugprone-casting-through-void) const __m128i hi = _mm_loadl_epi64(static_cast<const __m128i*>(static_cast<const void*>(p + n - 8))); // NOLINT(bugprone-casting-through-void)
const __m128i in = _mm_unpacklo_epi64(lo, hi); const __m128i in = _mm_unpacklo_epi64(lo, hi);
const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('\\')), _mm_cmpeq_epi8(in, _mm_set1_epi8(0x7F))), const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_cmplt_epi8(_mm_xor_si128(in, _mm_set1_epi8(2)), _mm_set1_epi8(33))); _mm_or_si128(_mm_cmplt_epi8(in, _mm_set1_epi8(0x20)), _mm_cmpeq_epi8(in, _mm_set1_epi8(0x7F))));
if (_mm_movemask_epi8(special) != 0) if (_mm_movemask_epi8(special) != 0)
{ {
return false; return false;
@@ -37478,13 +37424,10 @@ class serializer
#if JSON_SCAN_NEON #if JSON_SCAN_NEON
const uint8x16_t a = vld1q_u8(reinterpret_cast<const std::uint8_t*>(p)); const uint8x16_t a = vld1q_u8(reinterpret_cast<const std::uint8_t*>(p));
const uint8x16_t b = vld1q_u8(reinterpret_cast<const std::uint8_t*>(p + n - 16)); const uint8x16_t b = vld1q_u8(reinterpret_cast<const std::uint8_t*>(p + n - 16));
// (int8_t)(c ^ 2) < 33: the quote, the control characters and the const uint8x16_t special = vorrq_u8(vorrq_u8(vorrq_u8(vceqq_u8(a, vdupq_n_u8('"')), vceqq_u8(a, vdupq_n_u8('\\'))),
// non-ASCII bytes (see vector_plain_run()) vorrq_u8(vceqq_u8(b, vdupq_n_u8('"')), vceqq_u8(b, vdupq_n_u8('\\')))),
const uint8x16_t flip = vdupq_n_u8(2); vorrq_u8(vorrq_u8(vcltq_s8(vreinterpretq_s8_u8(a), vdupq_n_s8(0x20)), vcltq_s8(vreinterpretq_s8_u8(b), vdupq_n_s8(0x20))),
const int8x16_t limit = vdupq_n_s8(33); vorrq_u8(vceqq_u8(a, vdupq_n_u8(0x7F)), vceqq_u8(b, vdupq_n_u8(0x7F)))));
const uint8x16_t special = vorrq_u8(vorrq_u8(vorrq_u8(vceqq_u8(a, vdupq_n_u8('\\')), vceqq_u8(b, vdupq_n_u8('\\'))),
vorrq_u8(vceqq_u8(a, vdupq_n_u8(0x7F)), vceqq_u8(b, vdupq_n_u8(0x7F)))),
vorrq_u8(vcltq_s8(vreinterpretq_s8_u8(veorq_u8(a, flip)), limit), vcltq_s8(vreinterpretq_s8_u8(veorq_u8(b, flip)), limit)));
if (vmaxvq_u8(special) != 0) if (vmaxvq_u8(special) != 0)
{ {
return false; return false;
@@ -37494,15 +37437,14 @@ class serializer
#else #else
const __m128i a = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void) const __m128i a = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
const __m128i b = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p + n - 16))); // NOLINT(bugprone-casting-through-void) const __m128i b = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p + n - 16))); // NOLINT(bugprone-casting-through-void)
// (int8_t)(c ^ 2) < 33: the quote, the control characters and the const __m128i quote = _mm_set1_epi8('"');
// non-ASCII bytes (see vector_plain_run())
const __m128i backslash = _mm_set1_epi8('\\'); const __m128i backslash = _mm_set1_epi8('\\');
const __m128i space = _mm_set1_epi8(0x20);
const __m128i del = _mm_set1_epi8(0x7F); const __m128i del = _mm_set1_epi8(0x7F);
const __m128i flip = _mm_set1_epi8(2); const __m128i special = _mm_or_si128(_mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(a, quote), _mm_cmpeq_epi8(a, backslash)),
const __m128i limit = _mm_set1_epi8(33); _mm_or_si128(_mm_cmpeq_epi8(b, quote), _mm_cmpeq_epi8(b, backslash))),
const __m128i special = _mm_or_si128(_mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(a, backslash), _mm_cmpeq_epi8(b, backslash)), _mm_or_si128(_mm_or_si128(_mm_cmplt_epi8(a, space), _mm_cmplt_epi8(b, space)),
_mm_or_si128(_mm_cmpeq_epi8(a, del), _mm_cmpeq_epi8(b, del))), _mm_or_si128(_mm_cmpeq_epi8(a, del), _mm_cmpeq_epi8(b, del))));
_mm_or_si128(_mm_cmplt_epi8(_mm_xor_si128(a, flip), limit), _mm_cmplt_epi8(_mm_xor_si128(b, flip), limit)));
if (_mm_movemask_epi8(special) != 0) if (_mm_movemask_epi8(special) != 0)
{ {
return false; return false;
+27 -40
View File
@@ -759,10 +759,8 @@ namespace view
character, or not ASCII, 16 bytes per step character, or not ASCII, 16 bytes per step
Stops at such a byte, or where fewer than 16 bytes are left (the caller tells Stops at such a byte, or where fewer than 16 bytes are left (the caller tells
the two apart). One signed compare finds the quote, the control characters and the two apart). A signed compare with 0x20 finds control characters and
the non-ASCII bytes at once: these are exactly the bytes with non-ASCII bytes at once.
(int8_t)(c ^ 2) < 33 (see detail::vector_plain_run()). The backslash needs its
own compare.
*/ */
NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned char* p, const unsigned char* e) noexcept NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned char* p, const unsigned char* e) noexcept
{ {
@@ -770,8 +768,8 @@ NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned
{ {
#if NLOHMANN_VIEW_NEON #if NLOHMANN_VIEW_NEON
const uint8x16_t in = vld1q_u8(p); const uint8x16_t in = vld1q_u8(p);
const uint8x16_t special = vorrq_u8(vceqq_u8(in, vdupq_n_u8('\\')), const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))),
vcltq_s8(vreinterpretq_s8_u8(veorq_u8(in, vdupq_n_u8(2))), vdupq_n_s8(33))); vcltq_s8(vreinterpretq_s8_u8(in), vdupq_n_s8(0x20)));
// one nibble per byte (the usual NEON replacement of x86's movemask, see // one nibble per byte (the usual NEON replacement of x86's movemask, see
// D. Kutenin, "Porting x86 vector bitmask optimizations to Arm NEON", 2022) // D. Kutenin, "Porting x86 vector bitmask optimizations to Arm NEON", 2022)
const std::uint64_t bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0); const std::uint64_t bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0);
@@ -781,8 +779,8 @@ NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned
} }
#else #else
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void) const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
const __m128i special = _mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('\\')), const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_cmplt_epi8(_mm_xor_si128(in, _mm_set1_epi8(2)), _mm_set1_epi8(33))); _mm_cmplt_epi8(in, _mm_set1_epi8(0x20)));
const auto bits = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special))); const auto bits = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
if (bits != 0) if (bits != 0)
{ {
@@ -808,17 +806,15 @@ NLOHMANN_VIEW_ALWAYS_INLINE unsigned copy_plain_run16(const unsigned char* p, ch
#if NLOHMANN_VIEW_NEON #if NLOHMANN_VIEW_NEON
const uint8x16_t in = vld1q_u8(p); const uint8x16_t in = vld1q_u8(p);
vst1q_u8(reinterpret_cast<std::uint8_t*>(w), in); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast) vst1q_u8(reinterpret_cast<std::uint8_t*>(w), in); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
// (int8_t)(c ^ 2) < 33: the quote, the control characters and the const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))),
// non-ASCII bytes (see vector_plain_run()) vcltq_s8(vreinterpretq_s8_u8(in), vdupq_n_s8(0x20)));
const uint8x16_t special = vorrq_u8(vceqq_u8(in, vdupq_n_u8('\\')),
vcltq_s8(vreinterpretq_s8_u8(veorq_u8(in, vdupq_n_u8(2))), vdupq_n_s8(33)));
const std::uint64_t bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0); const std::uint64_t bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0);
return bits != 0 ? static_cast<unsigned>(count_trailing_zeros(bits) >> 2u) : 16u; return bits != 0 ? static_cast<unsigned>(count_trailing_zeros(bits) >> 2u) : 16u;
#else #else
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void) const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
_mm_storeu_si128(static_cast<__m128i*>(static_cast<void*>(w)), in); // NOLINT(bugprone-casting-through-void) _mm_storeu_si128(static_cast<__m128i*>(static_cast<void*>(w)), in); // NOLINT(bugprone-casting-through-void)
const __m128i special = _mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('\\')), const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_cmplt_epi8(_mm_xor_si128(in, _mm_set1_epi8(2)), _mm_set1_epi8(33))); _mm_cmplt_epi8(in, _mm_set1_epi8(0x20)));
const auto bits = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special))); const auto bits = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
return bits != 0 ? static_cast<unsigned>(count_trailing_zeros(bits)) : 16u; return bits != 0 ? static_cast<unsigned>(count_trailing_zeros(bits)) : 16u;
#endif #endif
@@ -978,8 +974,7 @@ NLOHMANN_VIEW_SSSE3_TARGET NLOHMANN_VIEW_NOINLINE NLOHMANN_VIEW_NOCLONE inline c
while (e - block >= 16) while (e - block >= 16)
{ {
const uint8x16_t in = vld1q_u8(block); const uint8x16_t in = vld1q_u8(block);
// the quote and the control characters: (c ^ 2) < 33 (see vector_plain_run()) const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))), vcltq_u8(in, vdupq_n_u8(0x20)));
const uint8x16_t special = vorrq_u8(vceqq_u8(in, vdupq_n_u8('\\')), vcltq_u8(veorq_u8(in, vdupq_n_u8(2)), vdupq_n_u8(33)));
const uint8x16_t prev1 = vextq_u8(prev, in, 15); const uint8x16_t prev1 = vextq_u8(prev, in, 15);
const uint8x16_t sc = vandq_u8(vandq_u8(vqtbl1q_u8(t1h, vshrq_n_u8(prev1, 4)), vqtbl1q_u8(t1l, vandq_u8(prev1, vdupq_n_u8(0x0F)))), vqtbl1q_u8(t2h, vshrq_n_u8(in, 4))); const uint8x16_t sc = vandq_u8(vandq_u8(vqtbl1q_u8(t1h, vshrq_n_u8(prev1, 4)), vqtbl1q_u8(t1l, vandq_u8(prev1, vdupq_n_u8(0x0F)))), vqtbl1q_u8(t2h, vshrq_n_u8(in, 4)));
const uint8x16_t must23 = vorrq_u8(vqsubq_u8(vextq_u8(prev, in, 14), vdupq_n_u8(0xE0 - 0x80)), vqsubq_u8(vextq_u8(prev, in, 13), vdupq_n_u8(0xF0 - 0x80))); const uint8x16_t must23 = vorrq_u8(vqsubq_u8(vextq_u8(prev, in, 14), vdupq_n_u8(0xE0 - 0x80)), vqsubq_u8(vextq_u8(prev, in, 13), vdupq_n_u8(0xF0 - 0x80)));
@@ -1018,10 +1013,8 @@ NLOHMANN_VIEW_SSSE3_TARGET NLOHMANN_VIEW_NOINLINE NLOHMANN_VIEW_NOCLONE inline c
while (e - block >= 16) while (e - block >= 16)
{ {
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void) const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void)
// the quote and the control characters: (c ^ 2) < 33, that is, const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
// (c ^ 2) - 32 saturates to 0 (see vector_plain_run()) _mm_cmpeq_epi8(_mm_subs_epu8(in, _mm_set1_epi8(0x1F)), zero)); // in < 0x20
const __m128i special = _mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('\\')),
_mm_cmpeq_epi8(_mm_subs_epu8(_mm_xor_si128(in, _mm_set1_epi8(2)), _mm_set1_epi8(32)), zero));
const __m128i prev1 = _mm_alignr_epi8(in, prev, 15); const __m128i prev1 = _mm_alignr_epi8(in, prev, 15);
const __m128i sc = _mm_and_si128(_mm_and_si128(_mm_shuffle_epi8(t1h, _mm_and_si128(_mm_srli_epi16(prev1, 4), nibble)), const __m128i sc = _mm_and_si128(_mm_and_si128(_mm_shuffle_epi8(t1h, _mm_and_si128(_mm_srli_epi16(prev1, 4), nibble)),
_mm_shuffle_epi8(t1l, _mm_and_si128(prev1, nibble))), _mm_shuffle_epi8(t1l, _mm_and_si128(prev1, nibble))),
@@ -8607,12 +8600,9 @@ inline std::size_t plain_output_run(const unsigned char* s, std::size_t n) noexc
for (; i + 16 <= n; i += 16) for (; i + 16 <= n; i += 16)
{ {
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(s + i))); // NOLINT(bugprone-casting-through-void) const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(s + i))); // NOLINT(bugprone-casting-through-void)
// the quote and the control characters are the bytes with // (a byte is below 0x20 if it is its own minimum with 0x1F)
// (c ^ 2) <= 32, that is, those that are their own minimum with 32 const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
// (see vector_plain_run()) _mm_cmpeq_epi8(_mm_min_epu8(in, _mm_set1_epi8(0x1F)), in));
const __m128i flipped = _mm_xor_si128(in, _mm_set1_epi8(2));
const __m128i special = _mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('\\')),
_mm_cmpeq_epi8(_mm_min_epu8(flipped, _mm_set1_epi8(32)), flipped));
const auto mask = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special))); const auto mask = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
if (mask != 0) if (mask != 0)
{ {
@@ -8675,21 +8665,20 @@ inline bool is_ascii_without_delete(const char* s, std::size_t n) noexcept
#if NLOHMANN_VIEW_VECTOR #if NLOHMANN_VIEW_VECTOR
/// the bytes of the 16 at p that ensure_ascii cannot copy (a quote, a /// the bytes of the 16 at p that ensure_ascii cannot copy (a quote, a
/// backslash, a control character, DEL, or not ASCII; the signed compare /// backslash, a control character, DEL, or not ASCII; a signed compare with
/// (int8_t)(c ^ 2) < 33 finds the quote, the control characters and the /// 0x20 finds control characters and non-ASCII bytes at once), one bit per
/// non-ASCII bytes at once, see vector_plain_run()), one bit per byte (x86) or /// byte (x86) or one nibble per byte (NEON)
/// one nibble per byte (NEON)
NLOHMANN_VIEW_ALWAYS_INLINE std::uint64_t ascii_stop_mask(const unsigned char* p) noexcept NLOHMANN_VIEW_ALWAYS_INLINE std::uint64_t ascii_stop_mask(const unsigned char* p) noexcept
{ {
#if NLOHMANN_VIEW_NEON #if NLOHMANN_VIEW_NEON
const uint8x16_t in = vld1q_u8(p); const uint8x16_t in = vld1q_u8(p);
const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('\\')), vceqq_u8(in, vdupq_n_u8(0x7F))), const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))),
vcltq_s8(vreinterpretq_s8_u8(veorq_u8(in, vdupq_n_u8(2))), vdupq_n_s8(33))); vorrq_u8(vceqq_u8(in, vdupq_n_u8(0x7F)), vcltq_s8(vreinterpretq_s8_u8(in), vdupq_n_s8(0x20))));
return vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0); return vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0);
#else #else
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void) const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('\\')), _mm_cmpeq_epi8(in, _mm_set1_epi8(0x7F))), const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_cmplt_epi8(_mm_xor_si128(in, _mm_set1_epi8(2)), _mm_set1_epi8(33))); _mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8(0x7F)), _mm_cmplt_epi8(in, _mm_set1_epi8(0x20))));
return static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special))); return static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
#endif #endif
} }
@@ -8852,7 +8841,7 @@ struct depth_counter<false>
/*! /*!
@brief where the double x nearest to a token w * 10^q lies, for 10^15 <= w < @brief where the double x nearest to a token w * 10^q lies, for 10^15 <= w <
10^17 and -16 <= q <= -1 (Eisel-Lemire, then a fixed-point position) 10^17 and -17 <= q <= -1 (Eisel-Lemire, then a fixed-point position)
dump() writes the shortest decimal in the rounding interval of x (the closest dump() writes the shortest decimal in the rounding interval of x (the closest
one if there are several, see zmij::to_shortest()). Its digits are those of a one if there are several, see zmij::to_shortest()). Its digits are those of a
@@ -9717,15 +9706,13 @@ class view_serializer
} }
for (; p != stop && batch.count != float_batch::capacity(); ++p) for (; p != stop && batch.count != float_batch::capacity(); ++p)
{ {
// a token of the source: 16 or 17 digits, 8 to 16 after the // a token of the source: 16 or 17 digits, at least 8 after the
// point, no exponent, 32 bytes to read (17 after the point, with // point, no exponent, 32 bytes to read
// no digit before it, only in a damaged image loaded with
// image_check::bounds: beyond the range of locate_token())
const unsigned frac_digits = p->extra >> 8u; const unsigned frac_digits = p->extra >> 8u;
const unsigned int_digits = p->extra & 0xFFu; const unsigned int_digits = p->extra & 0xFFu;
const unsigned count = int_digits + frac_digits; const unsigned count = int_digits + frac_digits;
if (p->kind == node_kind::number_float && (p->flags & node_flags::storage) == 0 if (p->kind == node_kind::number_float && (p->flags & node_flags::storage) == 0
&& count - 16u <= 1u && frac_digits - 8u <= 8u && src_end - (src + p->off) >= 33) && count - 16u <= 1u && frac_digits >= 8 && src_end - (src + p->off) >= 33)
{ {
const auto* const token = reinterpret_cast<const unsigned char*>(src + p->off); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast) const auto* const token = reinterpret_cast<const unsigned char*>(src + p->off); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
const unsigned sign = token[0] == '-' ? 1 : 0; const unsigned sign = token[0] == '-' ? 1 : 0;
-10
View File
@@ -364,16 +364,6 @@ json_test_add_test_for(src/unit-class_parser.cpp
MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force} MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force}
) )
# test the conversions again without std::filesystem::path support: a
# std::u8string must be a JSON string in this configuration, too
json_test_set_test_options(test-conversions2_no_filesystem
COMPILE_DEFINITIONS JSON_HAS_FILESYSTEM=0 JSON_HAS_EXPERIMENTAL_FILESYSTEM=0
)
json_test_add_test_for(src/unit-conversions2.cpp
NAME test-conversions2_no_filesystem
MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force}
)
# test diagnostic positions again without regular diagnostics (JSON pointer paths) # test diagnostic positions again without regular diagnostics (JSON pointer paths)
json_test_set_test_options(test-diagnostic-positions_only json_test_set_test_options(test-diagnostic-positions_only
COMPILE_DEFINITIONS JSON_DIAGNOSTICS=0 COMPILE_DEFINITIONS JSON_DIAGNOSTICS=0
-16
View File
@@ -536,20 +536,4 @@ TEST_CASE("alternative string type with json_view")
} }
} }
// a std::u8string is a JSON string, whatever the string type of the JSON value
// JSON_HAS_CPP_20 JSON_HAS_CPP_23 (do not remove; see note at top of file)
#if defined(__cpp_lib_char8_t)
TEST_CASE("alternative string type and std::u8string")
{
const std::u8string s = u8"abc";
const alt_json j = s;
CHECK(j.is_string());
CHECK(j.get<alt_string>() == alt_string("abc"));
CHECK(j.get<std::u8string>() == s);
// the array of code units written by versions before 3.13.0
CHECK(alt_json::array({0x61, 0x62, 0x63}).get<std::u8string>() == s);
}
#endif
DOCTEST_CLANG_SUPPRESS_WARNING_POP DOCTEST_CLANG_SUPPRESS_WARNING_POP
-2
View File
@@ -1674,8 +1674,6 @@ TEST_CASE("BSON input that cannot be read is discarded by every overload")
json _; json _;
CHECK_THROWS_AS(_ = json::from_bson(input.begin(), input.end()), json::parse_error&); CHECK_THROWS_AS(_ = json::from_bson(input.begin(), input.end()), json::parse_error&);
CHECK(json::from_bson(input, true, false).is_discarded()); CHECK(json::from_bson(input, true, false).is_discarded());
// false positive: Infer takes std::integral_constant<bool, false>::value for a pointer
// @infer-ignore NULLPTR_DEREFERENCE
CHECK(json::from_bson(input.begin(), input.end(), true, false).is_discarded()); CHECK(json::from_bson(input.begin(), input.end(), true, false).is_discarded());
#ifndef JSON_TEST_DEPRECATED_FUNCTIONS_DELETED #ifndef JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
CHECK(json::from_bson(input.data(), input.size(), true, false).is_discarded()); CHECK(json::from_bson(input.data(), input.size(), true, false).is_discarded());
+15 -93
View File
@@ -27,12 +27,10 @@ using nlohmann::json;
#include <deque> #include <deque>
#include <forward_list> #include <forward_list>
#include <list> #include <list>
#include <map>
#include <set> #include <set>
#include <unordered_map> #include <unordered_map>
#include <unordered_set> #include <unordered_set>
#include <valarray> #include <valarray>
#include <vector>
// NLOHMANN_JSON_SERIALIZE_ENUM uses a static std::pair // NLOHMANN_JSON_SERIALIZE_ENUM uses a static std::pair
DOCTEST_CLANG_SUPPRESS_WARNING_PUSH DOCTEST_CLANG_SUPPRESS_WARNING_PUSH
@@ -709,115 +707,39 @@ TEST_CASE("std::filesystem::path")
} }
#endif #endif
// a std::u8string is a JSON string since 3.13.0 (it was an array of code units // the ADL to_json overload for std::u8string only exists under the same guard
// before); independent of whether std::filesystem::path is supported, see the // as std::filesystem::path support (it is otherwise only reached indirectly,
// test-conversions2_no_filesystem target // via std::filesystem::path::u8string()) -- mirror both #if conditions from
// JSON_HAS_CPP_20 JSON_HAS_CPP_23 (do not remove; see note at top of file) // include/nlohmann/detail/conversions/to_json.hpp exactly
#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
#if defined(__cpp_lib_char8_t) #if defined(__cpp_lib_char8_t)
TEST_CASE("std::u8string") TEST_CASE("std::u8string")
{ {
// use \u universal-character-names (rather than raw \x byte escapes
// or literal non-ASCII source bytes) to compose the multi-byte UTF-8
// encoding -- MSVC treats \x escapes used that way inside a u8
// literal as a nonstandard extension (warning C5321), which some of
// our CI configs promote to an error; \u is portable and produces
// the exact same encoded bytes without depending on the source
// file's encoding
const std::u8string utf8 = u8"P\u011B\u0161ina";
SECTION("ascii") SECTION("ascii")
{ {
const std::u8string s = u8"Path"; const std::u8string s = u8"Path";
json const j = s; json const j = s;
CHECK(j.is_string());
CHECK(j.template get<std::string>() == "Path"); CHECK(j.template get<std::string>() == "Path");
} }
SECTION("utf-8") SECTION("utf-8")
{ {
json const j = utf8; // use \u universal-character-names (rather than raw \x byte escapes
// or literal non-ASCII source bytes) to compose the multi-byte UTF-8
// encoding -- MSVC treats \x escapes used that way inside a u8
// literal as a nonstandard extension (warning C5321), which some of
// our CI configs promote to an error; \u is portable and produces
// the exact same encoded bytes without depending on the source
// file's encoding
const std::u8string s = u8"P\u011B\u0161ina";
json const j = s;
CHECK(j.is_string());
CHECK(j.template get<std::string>() == "P\xc4\x9b\xc5\xa1ina"); CHECK(j.template get<std::string>() == "P\xc4\x9b\xc5\xa1ina");
} }
SECTION("round trip")
{
json const j = utf8;
CHECK(j.template get<std::u8string>() == utf8);
std::u8string back;
j.get_to(back);
CHECK(back == utf8);
// from a string
json const j_string = "P\xc4\x9b\xc5\xa1ina";
CHECK(j_string.template get<std::u8string>() == utf8);
// empty, and with an embedded NUL
CHECK(json(std::u8string()).template get<std::u8string>().empty());
const std::u8string with_nul(u8"a\0b", 3);
CHECK(json(with_nul).template get<std::u8string>() == with_nul);
// the lvalue and rvalue paths, and in containers
std::u8string lvalue = utf8;
CHECK(json(lvalue).template get<std::u8string>() == utf8);
CHECK(json(std::move(lvalue)).template get<std::u8string>() == utf8);
const std::vector<std::u8string> vec = {utf8, u8"x"};
json const j_vec = vec;
CHECK(j_vec.is_array());
CHECK(j_vec[0].is_string());
CHECK(j_vec.template get<std::vector<std::u8string>>() == vec);
const std::map<std::string, std::u8string> map = {{"k", utf8}};
CHECK(json(map).template get<std::map<std::string, std::u8string>>() == map);
}
SECTION("array of code units (written by versions before 3.13.0)")
{
json const j = json::array({0x50, 0xC4, 0x9B, 0xC5, 0xA1, 0x69, 0x6E, 0x61});
CHECK(j.template get<std::u8string>() == utf8);
CHECK(json::array().template get<std::u8string>().empty());
CHECK(json::parse("[97, 98]").template get<std::u8string>() == u8"ab");
std::u8string s = u8"overwritten";
j.get_to(s);
CHECK(s == utf8);
}
SECTION("other types")
{
#if !defined(JSON_NOEXCEPTION)
CHECK_THROWS_WITH_AS(json(1).template get<std::u8string>(), "[json.exception.type_error.302] type must be string or array, but is number", json::type_error);
CHECK_THROWS_WITH_AS(json(nullptr).template get<std::u8string>(), "[json.exception.type_error.302] type must be string or array, but is null", json::type_error);
CHECK_THROWS_AS(json::object().template get<std::u8string>(), json::type_error);
// an element that is not a number
CHECK_THROWS_AS(json::array({"a"}).template get<std::u8string>(), json::type_error);
#endif
}
SECTION("ordered_json")
{
nlohmann::ordered_json const j = utf8;
CHECK(j.is_string());
CHECK(j.template get<std::string>() == "P\xc4\x9b\xc5\xa1ina");
CHECK(j.template get<std::u8string>() == utf8);
CHECK(nlohmann::ordered_json::array({0x61, 0x62}).template get<std::u8string>() == u8"ab");
}
SECTION("std::u8string_view and char8_t arrays are unchanged (arrays of code units)")
{
const std::u8string_view view = utf8;
json const j_view = view;
CHECK(j_view.is_array());
CHECK(j_view.size() == 8);
const char8_t arr[] = u8"ab";
json const j_arr = arr;
CHECK(j_arr == json::array({0x61, 0x62, 0}));
}
} }
#endif #endif
#endif
#if !defined(JSON_NOEXCEPTION) #if !defined(JSON_NOEXCEPTION)
namespace namespace
-75
View File
@@ -32,7 +32,6 @@ using nlohmann::json;
#include <iostream> #include <iostream>
#include <iterator> #include <iterator>
#include <list>
#include <sstream> #include <sstream>
#include <string> #include <string>
#include <valarray> #include <valarray>
@@ -1410,77 +1409,3 @@ TEST_CASE("indentation runs of every length around the 8-byte skip")
CHECK(parse_result(indented("[\n~1]", std::string(17, ' ')), true) == "[1]"); CHECK(parse_result(indented("[\n~1]", std::string(17, ' ')), true) == "[1]");
} }
#endif #endif
#if !defined(JSON_NOEXCEPTION)
namespace
{
/// the value of parsing [first, last), or the message of the exception
template<typename Iterator>
std::string outcome_of(Iterator first, Iterator last)
{
try
{
return json::parse(first, last).dump();
}
catch (const json::exception& e)
{
return e.what();
}
}
/// the bytes of @a text as a vector of @a T (explicit casts: no implicit char conversion)
template<typename T>
std::vector<T> bytes_as(const std::string& text)
{
std::vector<T> result;
result.reserve(text.size());
for (const char c : text)
{
result.push_back(static_cast<T>(c));
}
return result;
}
/// a std::string, std::vector<char>, <unsigned char> and <signed char> as
/// iterator pairs: the same as the byte path (std::list iterators)
void check_iterator_pairs(const std::string& text)
{
CAPTURE(text)
const std::list<char> bytes(text.begin(), text.end());
const std::string expected = outcome_of(bytes.begin(), bytes.end());
const bool expected_accept = json::accept(bytes.begin(), bytes.end());
std::string s = text;
CHECK(outcome_of(s.begin(), s.end()) == expected);
CHECK(outcome_of(s.cbegin(), s.cend()) == expected);
CHECK(json::accept(s.cbegin(), s.cend()) == expected_accept);
const std::vector<char> vc(text.begin(), text.end());
CHECK(outcome_of(vc.begin(), vc.end()) == expected);
const std::vector<unsigned char> vu = bytes_as<unsigned char>(text);
CHECK(outcome_of(vu.begin(), vu.end()) == expected);
CHECK(json::accept(vu.begin(), vu.end()) == expected_accept);
const std::vector<signed char> vs = bytes_as<signed char>(text);
CHECK(outcome_of(vs.begin(), vs.end()) == expected);
}
} // namespace
TEST_CASE("deserialization from iterator pairs of std::string and std::vector")
{
// these are read as the pointer range they are, also before C++20
for (const char* text :
{
"", " ", R"([1, 2.5, "a", true, null, {"k": ["v"]}])", "[1, 2", R"({"a" 1})", "\"\xC3\xA4\"", "\"\xC3\"", "[1]x", "1e400"
})
{
check_iterator_pairs(text);
}
check_iterator_pairs(std::string("[1]" "\0" "x", 5));
// an empty range is not dereferenced
std::string empty;
CHECK_FALSE(json::accept(empty.begin(), empty.end()));
json _;
CHECK_THROWS_AS(_ = json::parse(empty.begin(), empty.end()), json::parse_error&);
}
#endif
-38
View File
@@ -1165,44 +1165,6 @@ TEST_CASE("json_view images: check")
} }
} }
SECTION("float tokens with 17 digits after the point and none before it")
{
// No JSON number has this layout. dump() batches tokens of 16 or 17
// digits, but only those with 8 to 16 fraction digits are in its
// range: 17 read past the table of powers of five (caught by a
// bounds-checked std::array), and on x86-64 wrote 18 digits
// (the assertion of write_decimal()).
const std::vector<std::uint8_t> img = json_document::parse("[1.2345678901234567]" + std::string(40, ' ')).save();
node n = node_at(img, 1);
REQUIRE(n.len == 18);
n.extra = 17u << 8u;
std::vector<std::uint8_t> b = img;
set_node(b, 1, n);
std::memcpy(b.data() + text_at(img) + n.off, ".99999999999999999", 18);
const json_document d = json_document::load(b, image_check::bounds);
const std::string dumped = d.root().dump();
CAPTURE(dumped)
CHECK(json::parse(dumped)[0].is_number());
CHECK(dumped == d.root().materialize().dump());
// (the input of OSS-Fuzz issue 583898810: the token is " " and
// 15 nines, then "1{")
const std::vector<std::uint8_t> fuzzed =
{
0x4E, 0x4A, 0x56, 0x49, 0x01, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x22, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x07, 0x10, 0x00, 0x11, 0x00, 0x00, 0x00, 0x00, 0x12, 0x00, 0x00, 0x00, 0x20, 0x20, 0x20, 0x20,
0x20, 0x39, 0x39, 0x39, 0x39, 0x39, 0x39, 0x39, 0x39, 0x39, 0x39, 0x39, 0x39, 0x39, 0x39, 0x39,
0x31, 0x7B, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20,
0x20, 0x20, 0x00, 0x00
};
CHECK(load_result(fuzzed, image_check::full) == check_failed);
const json_document f = json_document::load(fuzzed, image_check::bounds);
CHECK_NOTHROW(f.root().dump());
}
SECTION("nodes that share a range") SECTION("nodes that share a range")
{ {
// [big string, then n strings made to point to the big string]: every // [big string, then n strings made to point to the big string]: every
+57
View File
@@ -1669,3 +1669,60 @@ TEST_CASE("UTF-8 check of whole strings")
CHECK(valid > 20000); CHECK(valid > 20000);
CHECK(valid < 80000); CHECK(valid < 80000);
} }
TEST_CASE("UTF-8 check of long strings")
{
// Where a vector check is available, valid_utf8_string() skips ASCII 16
// bytes at a time, checks eight blocks from the first one that is not
// ASCII without testing for ASCII in between, and collects the errors of
// all blocks until the end. A sequence of each length and each kind of
// ill-formed sequence is put at every offset of strings that are longer
// than eight blocks: before, inside, at the end of, and after these
// blocks, and across every block boundary.
const char* const pieces[] = // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
{
"\xC3\xA9", "\xE2\x82\xAC", "\xE0\xA0\x80", "\xEF\xBF\xBF", "\xF0\x9F\x98\x80", "\xF4\x8F\xBF\xBF",
"\x80", "\xC0\xAF", "\xC3", "\xE2\x82", "\xED\xA0\x80", "\xF0\x9F", "\xF0\x9F\x98", "\xF4\x90\x80\x80", "\xFF"
};
// the text around the piece: ASCII only, a sequence at the start (so that
// the check of eight blocks starts there), and a sequence in every block
const char* const fillers[] = {"", "\xC3\xA9", "\xE2\x82\xAC"}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
std::size_t valid = 0;
std::size_t invalid = 0;
for (const char* const piece : pieces)
{
for (std::size_t filler = 0; filler < 3; ++filler)
{
std::string base;
if (filler == 1)
{
base += fillers[1];
}
while (base.size() < 300)
{
base += (filler == 2 && base.size() % 16 == 5) ? std::string(fillers[2]) : std::string("x");
}
for (std::size_t offset = 0; offset <= base.size(); ++offset)
{
// (not inside a sequence of the text around the piece)
if (offset < base.size() && (static_cast<unsigned char>(base[offset]) & 0xC0u) == 0x80u)
{
continue;
}
std::string s = base;
s.insert(offset, piece);
const auto* data = reinterpret_cast<const unsigned char*>(s.data());
const bool expected = nlohmann::detail::is_valid_utf8(s);
(expected ? valid : invalid) += 1;
CAPTURE(offset)
CAPTURE(filler)
CAPTURE(piece)
CHECK(nlohmann::detail::valid_utf8_string(data, s.size()) == expected);
CHECK((nlohmann::detail::valid_utf8_prefix(data, s.size()) == s.size()) == expected);
}
}
}
// the six well-formed pieces are valid at every offset, the others at none
CHECK(valid * 9 == invalid * 6);
CHECK(valid > 5000);
}