Merge branch develop into claude/perf-iterator-ranges

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

.github/workflows/codeql-analysis.yml

@@ -41,10 +41,11 @@ jobs:
       uses: github/codeql-action/init@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
       with:
         languages: c-cpp
-        # Analyze the sources without compiling them. Autobuild compiled the
-        # whole test suite and the resulting database was so large that the
-        # runner often lost communication with the server while importing it.
-        build-mode: none
+
+    # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
+    # If this step fails, then you should remove it and run the build manually (see below)
+    - name: Autobuild
+      uses: github/codeql-action/autobuild@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
 
     - name: Perform CodeQL Analysis
       uses: github/codeql-action/analyze@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1

docs/mkdocs/docs/home/blog/how_3_13_0_got_faster.md

@@ -22,7 +22,7 @@ Since 2016, the [README](https://github.com/nlohmann/json#readme) of this librar
 That sentence is still true, but the gap is a lot smaller now. Swap the 3.12.0 header for the 3.13.0 one, recompile, and
 the 182 benchmarks of the library's own suite run 3.0× as fast on an Apple M1 Max (geometric mean; 3.1× on an Intel
 MacBook Pro from 2016). Not a line of your code changes. Parsing typical documents got 3.1–4.3× as fast, and writing
-them 3.0–6.7×. The new version also adds [`json_document`](../../features/json_view/index.md), a second way to hold a parsed
+them 3.0–6.7×. The new version also adds [`json_document`](../../features/json_view.md), a second way to hold a parsed
 document, which competes with the fastest C++ and Rust JSON libraries; [How it compares](#how-it-compares) has the
 numbers, including where it loses. The price is compile time: two small example programs take 34–62% longer to compile.
 A precompiled header or a C++20 module wins back most or all of that in debug builds, and 41–91% of it in optimized
@@ -397,7 +397,7 @@ runs one word first. Inlining the vector code into the lexer cost 5% of validati
 no string of that file is long enough to reach it: the scan function, which runs once per token, saved more registers
 and needed a stack slot.
 
-[`json_document`](../../features/json_view/index.md), the new document type of `json_view.hpp`, also skips the digits after a
+[`json_document`](../../features/json_view.md), the new document type of `json_view.hpp`, also skips the digits after a
 decimal point 16 bytes at a time with SSE2: a compare and a bit mask instead of a branch per digit, about 33% of the
 instructions for a 12-digit fraction. Parsing `canada.json` got 22% faster on x86-64, a file of 10,000 floats 28–32%.
 
@@ -1132,7 +1132,7 @@ million small objects 1.39×.
 After all of the above, [`json::parse()`](../../api/basic_json/parse.md) is still bound by what it builds: one heap
 allocation per string, array and object, and a `std::map` node per member, all of which have to be freed again. Many
 programs only parse a document, read some fields and throw it away, or change a few values and write it back. For them,
-3.13.0 adds [`json_document`](../../features/json_view/index.md) and `json_view` in a new header, `<nlohmann/json_view.hpp>`.
+3.13.0 adds [`json_document`](../../features/json_view.md) and `json_view` in a new header, `<nlohmann/json_view.hpp>`.
 The two figures below show what each parser makes of this 54-byte text:
 
 ```json
@@ -1215,7 +1215,7 @@ static_assert(sizeof(node) == 16, "node must stay 16 bytes");
 ### It can be edited, too
 
 A read-only index is the easy case, and a fair objection is that `json_document` is only fast because it can't change.
-[`json_editable_document`](../../features/json_view/editing.md) has `set()`, `insert()`, `erase()` and
+[`json_editable_document`](../../features/json_view.md#editing-a-document) has `set()`, `insert()`, `erase()` and
 `push_back()`. An edit never writes into the source text and never moves the parsed index:
 
 - a new scalar is written over its own node, with new string and number text in an append-only arena;

include/nlohmann/detail/input/input_adapters.hpp

@@ -1348,6 +1348,64 @@ struct iterator_input_adapter_factory<IteratorType, SentinelType, enable_if_t<is
     }
 };
 
+/*!
+@brief whether @a IteratorType is an iterator of std::string or of a
+       std::vector of 1-byte elements
+
+Their elements are contiguous, but before C++20 that cannot be detected from
+the iterator type (std::contiguous_iterator), so an iterator pair of them would
+be read one character at a time through the iterators. They are read as the
+pointer range they are instead, as the containers themselves are (see the
+container overload of input_adapter()). The types are matched exactly, so no
+other iterator can be mistaken for one. (std::basic_string is only matched for
+char: other character types need std::char_traits specializations that the
+standard library need not have.)
+*/
+template<typename CharType>
+struct is_byte_char_type
+{
+    // a member, not an integral_constant argument, which astyle misformats
+    static constexpr bool value = sizeof(CharType) == 1 && std::is_integral<CharType>::value && !std::is_same<CharType, bool>::value;
+};
+
+template<typename IteratorType, typename CharType, bool IsByte = is_byte_char_type<CharType>::value>
+struct is_std_contiguous_byte_iterator_impl : std::false_type {};
+
+// (the containers are only named for byte types)
+template<typename IteratorType, typename CharType>
+struct is_std_contiguous_byte_iterator_impl<IteratorType, CharType, true>
+{
+    static constexpr bool value = !std::is_pointer<IteratorType>::value
+                                  && (std::is_same<IteratorType, typename std::vector<CharType>::iterator>::value
+                                      || std::is_same<IteratorType, typename std::vector<CharType>::const_iterator>::value
+                                      || std::is_same<IteratorType, std::string::iterator>::value
+                                      || std::is_same<IteratorType, std::string::const_iterator>::value);
+};
+
+template<typename IteratorType>
+struct is_std_contiguous_byte_iterator
+    : is_std_contiguous_byte_iterator_impl<IteratorType, typename std::remove_cv<typename std::iterator_traits<IteratorType>::value_type>::type> {};
+
+template<typename IteratorType>
+struct iterator_input_adapter_factory<IteratorType, IteratorType, enable_if_t<is_std_contiguous_byte_iterator<IteratorType>::value>>
+{
+    using iterator_type = IteratorType;
+    using sentinel_type = IteratorType;
+    using char_type = typename std::remove_cv<typename std::iterator_traits<iterator_type>::value_type>::type;
+    using adapter_type = iterator_input_adapter<const char_type*>;
+
+    static adapter_type create(IteratorType first, IteratorType last)
+    {
+        if (first == last)
+        {
+            // (an empty range must not be dereferenced)
+            return adapter_type(nullptr, nullptr);
+        }
+        const char_type* const begin = std::addressof(*first);
+        return adapter_type(begin, begin + (last - first));
+    }
+};
+
 // General purpose iterator-based input (iterator+sentinel pair; SentinelType
 // defaults to IteratorType for the common same-type case, but may differ for
 // C++20 ranges-style iterator+sentinel pairs). Only enable for types that can

single_include/nlohmann/json.hpp

@@ -10457,6 +10457,64 @@ struct iterator_input_adapter_factory<IteratorType, SentinelType, enable_if_t<is
     }
 };
 
+/*!
+@brief whether @a IteratorType is an iterator of std::string or of a
+       std::vector of 1-byte elements
+
+Their elements are contiguous, but before C++20 that cannot be detected from
+the iterator type (std::contiguous_iterator), so an iterator pair of them would
+be read one character at a time through the iterators. They are read as the
+pointer range they are instead, as the containers themselves are (see the
+container overload of input_adapter()). The types are matched exactly, so no
+other iterator can be mistaken for one. (std::basic_string is only matched for
+char: other character types need std::char_traits specializations that the
+standard library need not have.)
+*/
+template<typename CharType>
+struct is_byte_char_type
+{
+    // a member, not an integral_constant argument, which astyle misformats
+    static constexpr bool value = sizeof(CharType) == 1 && std::is_integral<CharType>::value && !std::is_same<CharType, bool>::value;
+};
+
+template<typename IteratorType, typename CharType, bool IsByte = is_byte_char_type<CharType>::value>
+struct is_std_contiguous_byte_iterator_impl : std::false_type {};
+
+// (the containers are only named for byte types)
+template<typename IteratorType, typename CharType>
+struct is_std_contiguous_byte_iterator_impl<IteratorType, CharType, true>
+{
+    static constexpr bool value = !std::is_pointer<IteratorType>::value
+                                  && (std::is_same<IteratorType, typename std::vector<CharType>::iterator>::value
+                                      || std::is_same<IteratorType, typename std::vector<CharType>::const_iterator>::value
+                                      || std::is_same<IteratorType, std::string::iterator>::value
+                                      || std::is_same<IteratorType, std::string::const_iterator>::value);
+};
+
+template<typename IteratorType>
+struct is_std_contiguous_byte_iterator
+    : is_std_contiguous_byte_iterator_impl<IteratorType, typename std::remove_cv<typename std::iterator_traits<IteratorType>::value_type>::type> {};
+
+template<typename IteratorType>
+struct iterator_input_adapter_factory<IteratorType, IteratorType, enable_if_t<is_std_contiguous_byte_iterator<IteratorType>::value>>
+{
+    using iterator_type = IteratorType;
+    using sentinel_type = IteratorType;
+    using char_type = typename std::remove_cv<typename std::iterator_traits<iterator_type>::value_type>::type;
+    using adapter_type = iterator_input_adapter<const char_type*>;
+
+    static adapter_type create(IteratorType first, IteratorType last)
+    {
+        if (first == last)
+        {
+            // (an empty range must not be dereferenced)
+            return adapter_type(nullptr, nullptr);
+        }
+        const char_type* const begin = std::addressof(*first);
+        return adapter_type(begin, begin + (last - first));
+    }
+};
+
 // General purpose iterator-based input (iterator+sentinel pair; SentinelType
 // defaults to IteratorType for the common same-type case, but may differ for
 // C++20 ranges-style iterator+sentinel pairs). Only enable for types that can

tests/src/unit-bson.cpp

@@ -1674,6 +1674,8 @@ TEST_CASE("BSON input that cannot be read is discarded by every overload")
     json _;
     CHECK_THROWS_AS(_ = json::from_bson(input.begin(), input.end()), json::parse_error&);
     CHECK(json::from_bson(input, true, false).is_discarded());
+    // false positive: Infer takes std::integral_constant<bool, false>::value for a pointer
+    // @infer-ignore NULLPTR_DEREFERENCE
     CHECK(json::from_bson(input.begin(), input.end(), true, false).is_discarded());
 #ifndef JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
     CHECK(json::from_bson(input.data(), input.size(), true, false).is_discarded());

tests/src/unit-deserialization.cpp

@@ -32,6 +32,7 @@ using nlohmann::json;
 
 #include <iostream>
 #include <iterator>
+#include <list>
 #include <sstream>
 #include <string>
 #include <valarray>
@@ -1409,3 +1410,77 @@ TEST_CASE("indentation runs of every length around the 8-byte skip")
     CHECK(parse_result(indented("[\n~1]", std::string(17, ' ')), true) == "[1]");
 }
 #endif
+
+#if !defined(JSON_NOEXCEPTION)
+namespace
+{
+/// the value of parsing [first, last), or the message of the exception
+template<typename Iterator>
+std::string outcome_of(Iterator first, Iterator last)
+{
+    try
+    {
+        return json::parse(first, last).dump();
+    }
+    catch (const json::exception& e)
+    {
+        return e.what();
+    }
+}
+
+/// the bytes of @a text as a vector of @a T (explicit casts: no implicit char conversion)
+template<typename T>
+std::vector<T> bytes_as(const std::string& text)
+{
+    std::vector<T> result;
+    result.reserve(text.size());
+    for (const char c : text)
+    {
+        result.push_back(static_cast<T>(c));
+    }
+    return result;
+}
+
+/// a std::string, std::vector<char>, <unsigned char> and <signed char> as
+/// iterator pairs: the same as the byte path (std::list iterators)
+void check_iterator_pairs(const std::string& text)
+{
+    CAPTURE(text)
+    const std::list<char> bytes(text.begin(), text.end());
+    const std::string expected = outcome_of(bytes.begin(), bytes.end());
+    const bool expected_accept = json::accept(bytes.begin(), bytes.end());
+
+    std::string s = text;
+    CHECK(outcome_of(s.begin(), s.end()) == expected);
+    CHECK(outcome_of(s.cbegin(), s.cend()) == expected);
+    CHECK(json::accept(s.cbegin(), s.cend()) == expected_accept);
+
+    const std::vector<char> vc(text.begin(), text.end());
+    CHECK(outcome_of(vc.begin(), vc.end()) == expected);
+    const std::vector<unsigned char> vu = bytes_as<unsigned char>(text);
+    CHECK(outcome_of(vu.begin(), vu.end()) == expected);
+    CHECK(json::accept(vu.begin(), vu.end()) == expected_accept);
+    const std::vector<signed char> vs = bytes_as<signed char>(text);
+    CHECK(outcome_of(vs.begin(), vs.end()) == expected);
+}
+}  // namespace
+
+TEST_CASE("deserialization from iterator pairs of std::string and std::vector")
+{
+    // these are read as the pointer range they are, also before C++20
+    for (const char* text :
+            {
+                "", " ", R"([1, 2.5, "a", true, null, {"k": ["v"]}])", "[1, 2", R"({"a" 1})", "\"\xC3\xA4\"", "\"\xC3\"", "[1]x", "1e400"
+            })
+    {
+        check_iterator_pairs(text);
+    }
+    check_iterator_pairs(std::string("[1]" "\0" "x", 5));
+
+    // an empty range is not dereferenced
+    std::string empty;
+    CHECK_FALSE(json::accept(empty.begin(), empty.end()));
+    json _;
+    CHECK_THROWS_AS(_ = json::parse(empty.begin(), empty.end()), json::parse_error&);
+}
+#endif