Fix CI: astyle formatting in unit-file_input.cpp

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

BUILD.bazel

@@ -30,7 +30,6 @@ cc_library(
         "include/nlohmann/detail/exceptions.hpp",
         "include/nlohmann/detail/hash.hpp",
         "include/nlohmann/detail/input/binary_reader.hpp",
-        "include/nlohmann/detail/input/contiguous_parser.hpp",
         "include/nlohmann/detail/input/input_adapters.hpp",
         "include/nlohmann/detail/input/json_sax.hpp",
         "include/nlohmann/detail/input/lexer.hpp",

docs/mkdocs/docs/api/macros/json_disable_stream_bulk_scan.md

@@ -48,8 +48,8 @@ The default value is `0`: streams are scanned in bulk.
 !!! note "Performance"
 
     Parsing from a `std::ifstream` or `std::istringstream` is typically 1.5 to 2 times as fast as before, and reading
-    many small values from one stream with `operator>>` 1.4 to 4 times as fast. Parsing from a stream then takes about
+    many small values from one stream with `operator>>` 1.4 to 4 times as fast. Parsing from a stream is then within
-    1.15 to 1.45 times as long as first reading the whole file into a `std::string` and parsing that.
+    about 10 to 20 % of first reading the whole file into a `std::string` and parsing that.
 
     A stream buffer without a get area (one that only overrides `underflow()` and `uflow()` and hands out every
     character from there) offers nothing to scan in bulk, so it is read one character at a time as before.

docs/mkdocs/docs/api/macros/json_disable_unlocked_file_io.md

@@ -5,7 +5,9 @@
 ```
 
 When defined to `1`, parsing from a `#!cpp FILE*` reads every character with `std::fgetc`, as in version 3.12.0 and
-earlier, instead of locking the file once and reading it with the unlocked stdio functions.
+earlier, instead of locking the file once and reading it with the unlocked stdio functions. This also turns off the
+scan of the `FILE`'s buffer in bulk (see below), which [`JSON_DISABLE_STREAM_BULK_SCAN`](json_disable_stream_bulk_scan.md)
+does not affect.
 
 It does not change any result: values, exceptions, and the position of the file after parsing are the same either way.
 Only the speed and the locking of the file differ (see the notes on locking below).
@@ -37,9 +39,17 @@ The default value is `0`: the file is locked once and read with the unlocked std
     | Microsoft (MSVC, clang-cl)                                 | `_lock_file`, `_fgetc_nolock`, `_unlock_file`       |
     | others (e.g., musl, MinGW, newlib)                         | `std::fgetc`, as before                             |
 
-    This makes parsing from a `FILE*` 3 to 4 times as fast with the Apple C library and 1.1 to 1.3 times as fast with
+    With glibc and the Apple C library, the parser also scans the characters that the `FILE` already holds in its
-    glibc, whose locking is cheaper. The binary formats ([`from_cbor`](../basic_json/from_cbor.md),
+    buffer in bulk, as it scans the get area of a stream (see
-    [`from_msgpack`](../basic_json/from_msgpack.md), ...) read from a `FILE*` the same way.
+    [`JSON_DISABLE_STREAM_BULK_SCAN`](json_disable_stream_bulk_scan.md)): it reads the same fields of the `FILE` that
+    the C library's own `getc_unlocked` reads, and it consumes exactly the characters that `getc_unlocked` would have
+    returned, so the file is left where it was left before.
+
+    Locking the file once makes parsing from a `FILE*` 3 to 4 times as fast with the Apple C library and 1.1 to 1.3
+    times as fast with glibc, whose locking is cheaper. Scanning the buffer in bulk makes `parse()` another 1.7 to 2.6
+    times as fast (and `accept()` 2.9 to 4.4 times), about as fast as parsing from a `std::ifstream`. The binary
+    formats ([`from_cbor`](../basic_json/from_cbor.md), [`from_msgpack`](../basic_json/from_msgpack.md), ...) read
+    from a `FILE*` the same way.
 
 !!! note "Locking"
 

docs/mkdocs/docs/features/files.md

@@ -32,11 +32,11 @@ json read_json_file(const std::string& path)
   column 1 instead of saying that the file is missing. A relative path is resolved against the working directory of the
   running program, which is often not the directory of the source file (see the
   [FAQ](../home/faq.md#parsing-a-file-fails-with-unexpected-end-of-input)).
-- **Read the file into a string only if the memory is no concern.** Parsing from a `#!cpp std::ifstream` takes about
+- **Don't read the file into a string first.** Parsing from a `#!cpp std::ifstream` is only about 10 to 20 % slower
-  1.15 to 1.45 times as long as reading the whole file into a string with one `read()` and parsing that, but it needs no
+  than reading the whole file into a string with one `read()` and parsing that, and it needs no memory for the copy (see
-  memory for the copy (see [input sources and performance](parsing/index.md#input-sources-and-performance)). The common
+  [input sources and performance](parsing/index.md#input-sources-and-performance)). The common way to read a file with
-  way to read a file with `#!cpp std::istreambuf_iterator<char>` is so slow that it takes about twice as long as parsing
+  `#!cpp std::istreambuf_iterator<char>` is so slow that it takes about twice as long as parsing from the stream. Read
-  from the stream. This advice is for
+  the file into a string only if you need the text itself, for example for an error message. This advice is for
   `#!cpp json::parse`. [`json_document`](json_view.md) reads a stream into a buffer that it owns anyway, so there is no
   copy to save: reading the file into a `#!cpp std::string` yourself and passing that as an lvalue, which the document
   borrows, was about 6 to 12 % faster than parsing from the stream, with the same single buffer.

docs/mkdocs/docs/features/parsing/index.md

@@ -77,16 +77,11 @@ them faster than others:
 |--------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------|
 | strings, string views, string literals, contiguous containers, pointer or iterator ranges over contiguous memory | the whole input is scanned in bulk: runs of string characters, numbers, and indentation are recognized several bytes at a time |
 | `#!cpp std::istream` (`#!cpp std::ifstream`, `#!cpp std::istringstream`, ...)            | the characters the stream buffer holds are scanned in bulk like a string; the stream buffer is asked for more when they are used up |
-| `#!cpp FILE*`                                                                              | one character at a time, with the file locked once for the whole parse              |
+| `#!cpp FILE*`                                                                              | with glibc and the Apple C library, the characters the `FILE` holds in its buffer are scanned in bulk like a stream's; elsewhere one character at a time; the file is locked once for the whole parse |
 | other iterator ranges (e.g., `#!cpp std::list<char>`, `#!cpp std::istreambuf_iterator<char>`), wide strings | one character at a time                                                             |
 
-Input in one block of memory is read even faster with the default options (no
+Parsing a file from a `#!cpp std::ifstream` is therefore only about 10 to 20 % slower than reading the whole file into a
-[parser callback](../../api/basic_json/parser_callback_t.md), no comments, no trailing commas, and for `sax_parse()`
+`#!cpp std::string` first and parsing that, and it needs no memory for a copy of the file. The stream is never read
-`strict = true`): `parse()`, `accept()`, and `sax_parse()` then read the grammar and the tokens in one loop over the
-memory. Only after an error do `parse()` and `sax_parse()` read the input again, to report the error exactly as before.
-
-Parsing a file from a `#!cpp std::ifstream` therefore takes about 1.15 to 1.45 times as long as reading the whole file
-into a `#!cpp std::string` first and parsing that, but it needs no memory for a copy of the file. The stream is never read
 ahead: after parsing, it is positioned exactly as if every character had been read one at a time, so
 [`operator>>`](../../api/operator_gtgt.md) can still read several values from one stream (see also
 [JSON Lines](json_lines.md)).

docs/mkdocs/docs/features/performance.md

@@ -27,11 +27,11 @@ A `#!cpp std::istream` (including `#!cpp std::ifstream`) gets the same bulk scan
 characters its stream buffer already holds (the *get area*) and asks the stream buffer for more only when they are used
 up. Nothing is read ahead, so the stream is left exactly where it would be after reading one character at a time, and
 [`operator>>`](../api/operator_gtgt.md) can still read several values from one stream. A stream buffer without a get
-area is read one character at a time. A `#!cpp FILE*` is still read one character at a time, but `file_input_adapter`
+area is read one character at a time. A `#!cpp FILE*` is locked once per parse instead of once per character, and with
-locks the file once per parse instead of once per character. See
+glibc and the Apple C library, the characters in its buffer are scanned in bulk as a stream's get area is. See
-[input sources and performance](parsing/index.md#input-sources-and-performance) for the full table. Parsing from a
+[input sources and performance](parsing/index.md#input-sources-and-performance) for the full table. According to the
-stream takes about 1.15 to 1.45 times as long as parsing an equivalent `#!cpp std::string`, which `parse()` reads with
+description of [#5828](https://github.com/nlohmann/json/pull/5828), parsing from a stream is within about 10 to 20 % of
-a parser of its own (see there).
+parsing an equivalent `#!cpp std::string`.
 
 Inputs that are still read one character at a time: iterator ranges that are not contiguous (e.g.
 `#!cpp std::list<char>` or `#!cpp std::istreambuf_iterator<char>`), wide strings, and user-defined adapters. An

include/nlohmann/detail/input/input_adapters.hpp

@@ -58,6 +58,19 @@
         #define JSON_UNLOCKED_FILE_IO 0
     #endif
     #undef JSON_UNLOCKED_FILE_IO_MSAN
+
+    // Whether file_input_adapter scans the buffer of the FILE directly, as
+    // getc_unlocked() reads it: the fields it uses are those that the C
+    // library's own getc_unlocked() macro uses (glibc: _IO_read_ptr and
+    // _IO_read_end; Apple: _p and _r). It needs the file locked for the whole
+    // parse, as JSON_UNLOCKED_FILE_IO == 1 has it (so JSON_DISABLE_UNLOCKED_FILE_IO
+    // and MemorySanitizer builds turn it off).
+    // (uClibc defines __GLIBC__ as well, but its FILE has other fields)
+    #if JSON_UNLOCKED_FILE_IO == 1 && ((defined(__GLIBC__) && !defined(__UCLIBC__)) || defined(__APPLE__))
+        #define JSON_FILE_WINDOW 1
+    #else
+        #define JSON_FILE_WINDOW 0
+    #endif
 #endif  // JSON_NO_IO
 
 NLOHMANN_JSON_NAMESPACE_BEGIN
@@ -179,8 +192,12 @@ class token_carry_buffer
     char* m_capacity_end = nullptr;
 };
 /*!
-Input adapter for stdio file access. This adapter read only 1 byte and do not use any
+Input adapter for stdio file access. The file is locked for the lifetime of the
- buffer. This adapter is a very low level adapter.
+adapter and read with the unlocked stdio functions (see
+JSON_DISABLE_UNLOCKED_FILE_IO). With glibc and the Apple C library, the
+characters in the buffer of the FILE are also scanned in bulk, as the get area
+of a stream is (see input_stream_adapter): exactly the characters getc_unlocked()
+would return are consumed, so the file is left where it was left before.
 */
 class file_input_adapter
 {
@@ -199,6 +216,12 @@ class file_input_adapter
     file_input_adapter(const file_input_adapter&) = delete;
     file_input_adapter(file_input_adapter&& rhs) noexcept
         : m_file(rhs.m_file)
+#if JSON_FILE_WINDOW
+        , token_carry(std::move(rhs.token_carry))
+        , token_segment(rhs.token_segment)
+        , current_in_buffer(rhs.current_in_buffer)
+        , tracking_token(rhs.tracking_token)
+#endif
     {
         // the lock is owned by the thread, not the adapter: hand it over
         rhs.m_file = nullptr;
@@ -210,6 +233,20 @@ class file_input_adapter
         unlock();
     }
 
+#if JSON_FILE_WINDOW
+    std::char_traits<char>::int_type get_character()
+    {
+        // what getc_unlocked() does, while the buffer has characters
+        const char* const next = buffer_ptr();
+        if (JSON_HEDLEY_LIKELY(next != buffer_end()))
+        {
+            current_in_buffer = true;
+            buffer_bump(1);
+            return std::char_traits<char>::to_int_type(*next);
+        }
+        return get_character_slow();
+    }
+#else
     std::char_traits<char>::int_type get_character() noexcept
     {
         // the file is locked for the lifetime of the adapter, so the
@@ -222,6 +259,7 @@ class file_input_adapter
         return std::fgetc(m_file);
 #endif
     }
+#endif
 
     // returns the number of characters successfully read
     template<class T>
@@ -230,7 +268,134 @@ class file_input_adapter
         return fread(dest, 1, sizeof(T) * count, m_file);
     }
 
+#if JSON_FILE_WINDOW
+    // The lexer may scan the buffer of the FILE directly, as the window of
+    // input_stream_adapter: the buffer ends wherever the C library's buffer
+    // ends, and is refilled by the next get_character().
+    static constexpr bool supports_bulk_scan = true;
+    static constexpr bool bulk_window = true;
+
+    // Pointer to the next unread character; only valid when bulk_remaining() > 0.
+    const char* bulk_data() const
+    {
+        return buffer_ptr();
+    }
+
+    // Number of unread characters in the buffer.
+    std::size_t bulk_remaining() const
+    {
+        if (!current_in_buffer)
+        {
+            return 0;
+        }
+        return static_cast<std::size_t>(buffer_end() - buffer_ptr());
+    }
+
+    // Consume @a n characters previously inspected via bulk_data().
+    void bulk_skip(std::size_t n)
+    {
+        JSON_ASSERT(n <= bulk_remaining());
+        buffer_bump(n);
+    }
+
+    // The adapter keeps the current token for the lexer's error messages, as
+    // input_stream_adapter does.
+    static constexpr bool supports_token_tracking = true;
+
+    void begin_token() noexcept
+    {
+        tracking_token = true;
+        token_carry.clear();
+        token_segment = buffer_ptr();
+    }
+
+    void mark_token_start(std::char_traits<char>::int_type c)
+    {
+        tracking_token = true;
+        token_carry.clear();
+        if (current_in_buffer)
+        {
+            token_segment = buffer_ptr() - 1;
+        }
+        else
+        {
+            token_segment = nullptr;
+            token_carry.push_back(std::char_traits<char>::to_char_type(c));
+        }
+    }
+
+    void copy_token(std::vector<char>& out, bool drop_last) const
+    {
+        token_carry.copy_to(out);
+        if (token_segment != nullptr)
+        {
+            out.insert(out.end(), token_segment, buffer_ptr());
+        }
+        if (drop_last && !out.empty())
+        {
+            out.pop_back();
+        }
+    }
+#endif
+
   private:
+#if JSON_FILE_WINDOW
+    // the buffer of the FILE: [buffer_ptr(), buffer_end()) are the characters
+    // getc_unlocked() would return next
+#if defined(__GLIBC__)
+    const char* buffer_ptr() const noexcept
+    {
+        return m_file->_IO_read_ptr;
+    }
+    const char* buffer_end() const noexcept
+    {
+        return m_file->_IO_read_end;
+    }
+    void buffer_bump(std::size_t n) noexcept
+    {
+        m_file->_IO_read_ptr += n;
+    }
+#else
+    const char* buffer_ptr() const noexcept
+    {
+        return reinterpret_cast<const char*>(m_file->_p);
+    }
+    const char* buffer_end() const noexcept
+    {
+        return m_file->_r > 0 ? reinterpret_cast<const char*>(m_file->_p) + m_file->_r : reinterpret_cast<const char*>(m_file->_p);
+    }
+    void buffer_bump(std::size_t n) noexcept
+    {
+        m_file->_p += n;
+        m_file->_r -= static_cast<int>(n);
+    }
+#endif
+
+    // the buffer is exhausted: getc_unlocked() refills it
+    JSON_HEDLEY_NEVER_INLINE std::char_traits<char>::int_type get_character_slow()
+    {
+        if (token_segment != nullptr)
+        {
+            token_carry.append(token_segment, buffer_end());
+            token_segment = nullptr;
+        }
+        const auto res = getc_unlocked(m_file);
+        if (res == EOF)
+        {
+            current_in_buffer = false;
+            return std::char_traits<char>::eof();
+        }
+        // the character was taken from the refilled buffer, at buffer_ptr() - 1
+        current_in_buffer = true;
+        if (tracking_token)
+        {
+            token_segment = buffer_ptr() - 1;
+        }
+        return res;
+    }
+
+#endif
+
     // Lock the file once for the whole parse instead of once per character.
     // The lock is recursive, so fread() and the user's own stdio calls on the
     // same thread keep working.
@@ -259,6 +424,16 @@ class file_input_adapter
 
     /// the file pointer to read from
     std::FILE* m_file;
+#if JSON_FILE_WINDOW
+    /// the part of the current token that was in earlier buffers
+    token_carry_buffer token_carry{}; // NOLINT(readability-redundant-member-init)
+    /// start of the current token's part in the buffer, or nullptr
+    const char* token_segment = nullptr;
+    /// whether the character last returned by get_character() is in the buffer
+    bool current_in_buffer = false;
+    /// whether a token is tracked at all; the binary readers never start one
+    bool tracking_token = false;
+#endif
 };
 
 /*!

include/nlohmann/detail/input/number_parse.hpp

@@ -316,30 +316,6 @@ JSON_HEDLEY_ALWAYS_INLINE std::size_t accumulate_digits(const char* data, std::s
     return i;
 }
 
-/// the index of the first byte at or after @a i that is not a digit: the
-/// digits of accumulate_digits() without their value
-JSON_HEDLEY_ALWAYS_INLINE std::size_t skip_digits(const char* data, std::size_t i, std::size_t avail) noexcept
-{
-    while (avail - i >= 8)
-    {
-        if (!is_eight_digits(read_eight_bytes(data + i)))
-        {
-            // a byte of the word is not a digit: no bounds checks before it
-            while (static_cast<unsigned char>(data[i] - '0') <= 9)
-            {
-                ++i;
-            }
-            return i;
-        }
-        i += 8;
-    }
-    while (i < avail && data[i] >= '0' && data[i] <= '9')
-    {
-        ++i;
-    }
-    return i;
-}
-
 /// whether [first, last) contains a digit other than '0'; a plain loop, as
 /// GCC calls std::any_of out of line even for the usual empty range
 JSON_HEDLEY_ALWAYS_INLINE bool has_nonzero_digit(const char* first, const char* last) noexcept

include/nlohmann/detail/input/string_scan.hpp

@@ -520,10 +520,7 @@ JSON_HEDLEY_NEVER_INLINE inline std::size_t string_bulk_run_from(const unsigned
 }
 #endif
 
-// string_bulk_run() for the lexer. The run must be the longest one: the lexer
+// string_bulk_run() for the lexer. With NEON, only the first four words are
-// would only lose speed with a shorter run, but contiguous_parser rejects a
-// string whose run ends at a byte other than a quote or a backslash, so a
-// shorter run would reject valid input there. With NEON, only the first four words are
 // checked inline (most keys and short values end there), and longer runs and
 // non-ASCII text continue out of line. The lexer's scan function runs once per
 // token, and the vector code inlined into it needs more registers, which every

include/nlohmann/detail/macro_unscope.hpp

@@ -22,6 +22,7 @@
 #undef JSON_DISABLE_ENUM_SERIALIZATION
 #undef JSON_DISABLE_TUPLE_REFERENCE_CONVERSION
 #undef JSON_UNLOCKED_FILE_IO
+#undef JSON_FILE_WINDOW
 #undef JSON_DTOA_SSE2
 #undef JSON_DTOA_NEON
 #undef JSON_LITTLE_ENDIAN_TARGET

include/nlohmann/json.hpp

@@ -58,7 +58,6 @@
 #include <nlohmann/detail/exceptions.hpp> // IWYU pragma: export
 #include <nlohmann/detail/hash.hpp>
 #include <nlohmann/detail/input/binary_reader.hpp>
-#include <nlohmann/detail/input/contiguous_parser.hpp>
 #include <nlohmann/detail/input/input_adapters.hpp>
 #include <nlohmann/detail/input/json_sax.hpp>
 #include <nlohmann/detail/input/lexer.hpp>
@@ -3110,15 +3109,6 @@ public:
                                  const bool ignore_trailing_commas)
     {
         basic_json result;
-        // JSON in one block of memory, with the default options, is read by
-        // the faster contiguous_parser; on an error, the library parser reads
-        // it again to report it (see contiguous_parser)
-        using contiguous = std::integral_constant<bool, detail::can_parse_contiguous<basic_json, typename std::decay<InputAdapterType>::type>::value>;
-        if (cb == nullptr && !ignore_comments && !ignore_trailing_commas
-                && detail::parse_contiguous(adapter, result, allow_exceptions, contiguous {}))
-        {
-            return result;
-        }
         parser(std::forward<InputAdapterType>(adapter), std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas).parse(true, result);
         return result;
     }
@@ -3129,17 +3119,8 @@ public:
                             const bool ignore_comments,
                             const bool ignore_trailing_commas)
     {
-        // JSON in one block of memory is checked by the faster
-        // contiguous_parser (see parse_impl())
-        using adapter_t = typename std::decay<InputAdapterType>::type;
-        bool accepted = false;
-        if (!ignore_comments && !ignore_trailing_commas
-                && detail::accept_contiguous<basic_json>(adapter, accepted, std::integral_constant<bool, detail::can_parse_contiguous<basic_json, adapter_t>::value> {}))
-        {
-            return accepted;
-        }
         // the parser and lexer that discard the values of tokens (see lexer)
-        return ::nlohmann::detail::parser<basic_json, adapter_t, true>(std::forward<InputAdapterType>(adapter),
+        return ::nlohmann::detail::parser<basic_json, typename std::decay<InputAdapterType>::type, true>(std::forward<InputAdapterType>(adapter),
                 nullptr, false, ignore_comments, ignore_trailing_commas).accept(true);
     }
 
@@ -3154,14 +3135,6 @@ public:
     {
         if (format == input_format_t::json)
         {
-            // JSON in one block of memory is read by the faster
-            // contiguous_parser, with the default options (see parse_impl())
-            bool result = false;
-            if (strict && !ignore_comments && !ignore_trailing_commas
-                    && detail::sax_parse_contiguous<basic_json>(ia, sax, result, std::integral_constant<bool, detail::can_parse_contiguous<basic_json, InputAdapterType>::value> {}))
-            {
-                return result;
-            }
             return parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict);
         }
         return detail::binary_reader<basic_json, InputAdapterType, SAX>(std::move(ia), format).sax_parse(sax, strict, tag_handler);

tests/src/unit-contiguous_parser.cpp

@@ -1,564 +0,0 @@
-//     __ _____ _____ _____
-//  __|  |   __|     |   | |  JSON for Modern C++ (supporting code)
-// |  |  |__   |  |  | | | |  version 3.12.0
-// |_____|_____|_____|_|___|  https://github.com/nlohmann/json
-//
-// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
-// SPDX-License-Identifier: MIT
-
-// parse() and accept() read text in one block of memory with
-// detail::contiguous_parser, and everything else (here: std::list iterators)
-// with the lexer's byte path. These tests check that the two agree on every
-// input: the same value, with the same number types and float bits, or the
-// same exception with the same message.
-
-#include "doctest_compatibility.h"
-
-#include <nlohmann/json.hpp>
-using nlohmann::json;
-using nlohmann::ordered_json;
-
-#include <cstddef>
-#include <cstdint>
-#include <cstring>
-#include <list>
-#include <map>
-#include <random>
-#include <string>
-#include <type_traits>
-#include <vector>
-
-namespace
-{
-
-/// the value with its number types and float bits; below 100 levels, the
-/// library's dump() (whose recursion is bounded) describes the rest
-template<typename Json>
-std::string describe_value(const Json& j, int depth = 0)
-{
-    std::string out;
-    if (depth >= 100)
-    {
-        out = j.dump();
-        return out;
-    }
-    switch (j.type())
-    {
-        case Json::value_t::number_float:
-        {
-            const double d = j.template get<double>();
-            std::uint64_t bits = 0;
-            std::memcpy(&bits, &d, sizeof(bits));
-            out += "f" + std::to_string(bits);
-            break;
-        }
-        case Json::value_t::number_integer:
-            out += "i" + j.dump();
-            break;
-        case Json::value_t::number_unsigned:
-            out += "u" + j.dump();
-            break;
-        case Json::value_t::array:
-            out += "[";
-            for (const auto& e : j)
-            {
-                out += describe_value(e, depth + 1) + ",";
-            }
-            out += "]";
-            break;
-        case Json::value_t::object:
-            out += "{";
-            for (auto it = j.begin(); it != j.end(); ++it)
-            {
-                out += Json(it.key()).dump() + ":" + describe_value(it.value(), depth + 1) + ",";
-            }
-            out += "}";
-            break;
-        case Json::value_t::null:
-        case Json::value_t::boolean:
-        case Json::value_t::string:
-        case Json::value_t::binary:
-        case Json::value_t::discarded:
-        default:
-            out += j.dump();
-            break;
-    }
-    return out;
-}
-
-#if !defined(JSON_NOEXCEPTION)
-template<typename Json, typename Parse>
-std::string outcome(Parse parse)
-{
-    try
-    {
-        return describe_value(parse());
-    }
-    catch (const typename Json::exception& e)
-    {
-        return std::string("exception: ") + e.what();
-    }
-}
-#endif
-
-/// every SAX event with its values, the text of float tokens, and the parse
-/// error with its position, token and message; returns false after
-/// @a stop_after events (to stop the parse)
-class event_log : public nlohmann::json_sax<json>
-{
-  public:
-    explicit event_log(std::size_t stop_after_ = static_cast<std::size_t>(-1), bool continue_after_error_ = false)
-        : stop_after(stop_after_), continue_after_error(continue_after_error_)
-    {}
-
-    bool null() override
-    {
-        return add("null");
-    }
-    bool boolean(bool val) override
-    {
-        return add(val ? "true" : "false");
-    }
-    bool number_integer(number_integer_t val) override
-    {
-        return add("i" + std::to_string(val));
-    }
-    bool number_unsigned(number_unsigned_t val) override
-    {
-        return add("u" + std::to_string(val));
-    }
-    bool number_float(number_float_t val, const string_t& s) override
-    {
-        std::uint64_t bits = 0;
-        std::memcpy(&bits, &val, sizeof(bits));
-        return add("f" + std::to_string(bits) + " " + s);
-    }
-    bool string(string_t& val) override
-    {
-        return add("s" + val);
-    }
-    bool binary(binary_t& /*val*/) override
-    {
-        return add("binary");
-    }
-    bool start_object(std::size_t elements) override
-    {
-        return add("{" + std::to_string(elements));
-    }
-    bool key(string_t& val) override
-    {
-        return add("k" + val);
-    }
-    bool end_object() override
-    {
-        return add("}");
-    }
-    bool start_array(std::size_t elements) override
-    {
-        return add("[" + std::to_string(elements));
-    }
-    bool end_array() override
-    {
-        return add("]");
-    }
-    bool parse_error(std::size_t position, const std::string& last_token, const nlohmann::detail::exception& ex) override
-    {
-        events.push_back("error " + std::to_string(position) + " " + last_token + " " + ex.what());
-        return continue_after_error;
-    }
-
-    const std::vector<std::string>& log() const noexcept
-    {
-        return events;
-    }
-
-  private:
-    bool add(const std::string& event)
-    {
-        events.push_back(event);
-        return events.size() != stop_after;
-    }
-
-    std::vector<std::string> events {}; // NOLINT(readability-redundant-member-init)
-    std::size_t stop_after;
-    bool continue_after_error;
-};
-
-/// sax_parse() of the contiguous path against the byte path, also with a
-/// handler that stops the parse
-void check_same_events(const std::string& text, const std::list<char>& bytes)
-{
-    for (const std::size_t stop_after :
-            {
-                static_cast<std::size_t>(-1), std::size_t{1}, std::size_t{3}
-            })
-    {
-        CAPTURE(stop_after)
-        event_log contiguous(stop_after);
-        event_log byte_path(stop_after);
-        const bool r1 = json::sax_parse(text, &contiguous);
-        const bool r2 = json::sax_parse(bytes.begin(), bytes.end(), &byte_path);
-        CHECK(r1 == r2);
-        CHECK(contiguous.log() == byte_path.log());
-    }
-    // a handler that asks to go on after an error (the library parser then
-    // reports what follows, e.g., the missing end of the input)
-    event_log contiguous(static_cast<std::size_t>(-1), true);
-    event_log byte_path(static_cast<std::size_t>(-1), true);
-    const bool r1 = json::sax_parse(text, &contiguous);
-    const bool r2 = json::sax_parse(bytes.begin(), bytes.end(), &byte_path);
-    CHECK(r1 == r2);
-    CHECK(contiguous.log() == byte_path.log());
-}
-
-/// the contiguous path (a std::string) against the byte path (std::list iterators)
-template<typename Json>
-void check_same(const std::string& text)
-{
-    CAPTURE(text)
-    const std::list<char> bytes(text.begin(), text.end());
-
-#if !defined(JSON_NOEXCEPTION)
-    // with exceptions: the value, or the same exception for both
-    const std::string contiguous = outcome<Json>([&]
-    {
-        return Json::parse(text);
-    });
-    const std::string byte_path = outcome<Json>([&]
-    {
-        return Json::parse(bytes.begin(), bytes.end());
-    });
-    CHECK(contiguous == byte_path);
-#endif
-
-    // without exceptions: the value, or discarded for both
-    const Json c = Json::parse(text, nullptr, false);
-    const Json b = Json::parse(bytes.begin(), bytes.end(), nullptr, false);
-    CHECK(c.is_discarded() == b.is_discarded());
-
-    CHECK(Json::accept(text) == Json::accept(bytes.begin(), bytes.end()));
-    if (std::is_same<Json, json>::value)
-    {
-        check_same_events(text, bytes);
-    }
-    // a pointer range without a terminating NUL
-    std::vector<char> exact(text.begin(), text.end());
-    if (!exact.empty())
-    {
-        CHECK(Json::accept(exact.data(), exact.data() + exact.size()) == Json::accept(bytes.begin(), bytes.end()));
-    }
-}
-
-void check_both(const std::string& text)
-{
-    check_same<json>(text);
-    check_same<ordered_json>(text);
-}
-
-/// a random JSON text, mostly valid, with the corner cases of the parser
-class generator
-{
-  public:
-    explicit generator(std::uint32_t seed) : rng(seed) {}
-
-    std::string document(int depth)
-    {
-        std::string out;
-        value(out, depth);
-        return out;
-    }
-
-  private:
-    std::uint32_t pick(std::uint32_t n)
-    {
-        return static_cast<std::uint32_t>(rng() % n);
-    }
-
-    void whitespace(std::string& out)
-    {
-        static const char* const choices[] = {"", "", "", " ", "\n", "\n        ", "\t", "\r\n  ", "\n                  ", "  "}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
-        out += choices[pick(10)];
-    }
-
-    void number(std::string& out)
-    {
-        static const char* const choices[] = // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
-        {
-            "0", "-0", "1", "-1", "42", "123456789012345678", "-123456789012345678", "1234567890123456789",
-            "-1234567890123456789", "9223372036854775807", "-9223372036854775808", "-9223372036854775809",
-            "18446744073709551615", "18446744073709551616", "123456789012345678901234567890", "0.0", "-0.0",
-            "1.5", "-65.613616999999977", "1e10", "1E+10", "1e-10", "2.2250738585072014e-308", "4.9e-324",
-            "1e-400", "1.7976931348623157e308", "1e308", "1e309", "-1e309", "123.456e-789", "0.1e1",
-            "12345678901234567890.5", "1.00000000000000011102230246251565404236316680908203125",
-            "9007199254740993", "0.30000000000000004", "1e22", "1e23", "100000000000000000000000",
-            "00", "01", "-", "1.", ".5", "1e", "1e+", "--1", "+1", "1.5e3.2", "0x10", "1_000"
-        };
-        out += choices[pick(sizeof(choices) / sizeof(choices[0]))];
-    }
-
-    void string(std::string& out)
-    {
-        static const char* const pieces[] = // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
-        {
-            "a", "key", "a longer piece of plain ASCII text for the vector paths", "\\n", "\\t", "\\\"",
-            "\\\\", "\\/", "\\b", "\\f", "\\r", "\\u0000", "\\u00e4", "\\u20AC", "\\uD83D\\uDE00",
-            "\\uD83D", "\\uDE00", "\\uD83Dx", "\\u12", "\\x", "\xC3\xA4", "\xE2\x82\xAC", "\xF0\x9F\x98\x80",
-            "\xC0\xAF", "\xED\xA0\x80", "\xF4\x90\x80\x80", "\x80", "\xFF", "\x01", "\x1F", "\x7F", "\t"
-        };
-        out += '"';
-        const auto n = pick(6);
-        for (std::uint32_t i = 0; i < n; ++i)
-        {
-            // mostly well-formed pieces
-            out += pieces[pick(5) == 0 ? pick(sizeof(pieces) / sizeof(pieces[0])) : pick(20)];
-        }
-        out += '"';
-    }
-
-    void value(std::string& out, int depth)
-    {
-        const auto kind = depth > 0 ? pick(9) : 3 + pick(6);
-        switch (kind)
-        {
-            case 0:
-            case 1:
-            {
-                out += '[';
-                whitespace(out);
-                const auto n = pick(6);
-                for (std::uint32_t i = 0; i < n; ++i)
-                {
-                    if (i != 0)
-                    {
-                        out += ',';
-                        whitespace(out);
-                    }
-                    value(out, depth - 1);
-                    whitespace(out);
-                }
-                out += ']';
-                break;
-            }
-            case 2:
-            {
-                out += '{';
-                whitespace(out);
-                const auto n = pick(6);
-                for (std::uint32_t i = 0; i < n; ++i)
-                {
-                    if (i != 0)
-                    {
-                        out += ',';
-                        whitespace(out);
-                    }
-                    // a few duplicate keys
-                    if (pick(4) == 0)
-                    {
-                        out += "\"dup\"";
-                    }
-                    else
-                    {
-                        string(out);
-                    }
-                    whitespace(out);
-                    out += pick(2) == 0 ? ":" : ": ";
-                    whitespace(out);
-                    value(out, depth - 1);
-                    whitespace(out);
-                }
-                out += '}';
-                break;
-            }
-            case 3:
-            case 4:
-                number(out);
-                break;
-            case 5:
-            case 6:
-                string(out);
-                break;
-            case 7:
-                out += pick(2) == 0 ? "true" : "false";
-                break;
-            default:
-                out += "null";
-                break;
-        }
-    }
-
-    std::mt19937 rng;
-};
-
-} // namespace
-
-TEST_CASE("contiguous parser")
-{
-    SECTION("fixed inputs")
-    {
-        const char* const inputs[] = // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
-        {
-            "", " ", "\n", "null", "true", "false", "nul", "tru", "fals", "nullx", "[]", "{}", "[", "{", "]", "}",
-            "[1,]", "[,1]", R"({"a":1,})", R"({"a" 1})", R"({"a":})", "{1:2}", "[1 2]", R"("")", R"("abc)", R"("\")",
-            "\xEF\xBB\xBF[1]", "\xEF\xBB[1]", "\xEF[1]", "\xEF\xBB\xBF", " [1] ", "[1]x", "[1] [2]", "1 2",
-            R"({"a":1,"a":2})", R"({"b":1,"a":2,"b":3})", R"(["\u0000"])", "[1e400]", "[-1e400]",
-            R"({"a": [1, 2, {"b": null}], "c": "d"})", "[true,false,null]", "[[[[[]]]]]",
-        };
-        for (const char* input : inputs)
-        {
-            check_both(input);
-        }
-
-        // NUL bytes: by default the end of the input (between values), else an error
-        check_both(std::string("[1]\0garbage", 11));
-        check_both(std::string("[1,\0]", 5));
-        check_both(std::string("\"a\0b\"", 5));
-        check_both(std::string("\0", 1));
-        check_both(std::string("1\0", 2));
-    }
-
-    SECTION("deep nesting")
-    {
-        // across the 64-level words of the nesting stack
-        const std::vector<std::size_t> depths = {63, 64, 65, 127, 128, 129, 200, 1000, 10000};
-        for (const std::size_t depth : depths)
-        {
-            check_both(std::string(depth, '[') + std::string(depth, ']'));
-            std::string objects;
-            for (std::size_t i = 0; i < depth; ++i)
-            {
-                objects += (i % 2 == 0) ? "{\"k\":" : "[";
-            }
-            objects += "1";
-            for (std::size_t i = depth; i-- > 0;)
-            {
-                objects += (i % 2 == 0) ? "}" : "]";
-            }
-            check_both(objects);
-            // unbalanced
-            check_both(std::string(depth, '[') + std::string(depth - 1, ']'));
-            check_both(std::string(depth, '[') + std::string(depth + 1, ']'));
-        }
-        // a value after a container that closes a 64-level word
-        std::string mixed = std::string(64, '[') + "1" + std::string(63, ']') + ",{\"a\":2}]";
-        check_both(mixed);
-        mixed = "{\"x\":" + std::string(64, '[') + std::string(64, ']') + ",\"y\":[1]}";
-        check_both(mixed);
-    }
-
-    SECTION("generated documents and their mutations")
-    {
-        generator gen(20261105u);
-        std::mt19937 rng(42u); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed): reproducible
-        const char alphabet[] = "{}[]:,\"\\ \t\n\r0123456789-+.eEtrufalsn/u\x80\xbf\xc3\xe2\xed\xf0\xff"; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
-        for (int round = 0; round < 1500; ++round)
-        {
-            const std::string text = gen.document(1 + (round % 5));
-            check_both(text);
-            // every truncation of short documents, and a few mutations
-            if (text.size() < 60)
-            {
-                for (std::size_t n = 0; n < text.size(); ++n)
-                {
-                    check_same<json>(text.substr(0, n));
-                }
-            }
-            for (int m = 0; m < 3 && !text.empty(); ++m)
-            {
-                std::string mutated = text;
-                const std::size_t pos = rng() % mutated.size();
-                switch (rng() % 3)
-                {
-                    case 0:
-                        mutated[pos] = alphabet[rng() % (sizeof(alphabet) - 1)];
-                        break;
-                    case 1:
-                        mutated.insert(pos, 1, alphabet[rng() % (sizeof(alphabet) - 1)]);
-                        break;
-                    default:
-                        mutated.erase(pos, 1);
-                        break;
-                }
-                check_same<json>(mutated);
-            }
-        }
-    }
-
-    SECTION("narrow number types")
-    {
-        // integers that do not fit become floats, floats are converted to float
-        using narrow_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int32_t, std::uint32_t, float>;
-        const std::vector<std::string> inputs =
-        {
-            "[2147483647, 2147483648, -2147483648, -2147483649, 4294967295, 4294967296, 18446744073709551615]",
-            "[1.5, 3.4028234663852886e38, 3.4028235677973366e38, 1e39, -1e39, 1e-46, 0.1, 16777217]",
-            "[-0, 0.0, 123456789012345678901234567890, -9223372036854775809]", "1e39", "4294967296", "-2147483649"
-        };
-        for (const auto& input : inputs)
-        {
-            check_same<narrow_json>(input);
-        }
-        generator gen(17u);
-        for (int round = 0; round < 200; ++round)
-        {
-            check_same<narrow_json>(gen.document(1 + (round % 4)));
-        }
-    }
-
-    SECTION("number corner cases at the end of the input")
-    {
-        // the number ends at the end of the block (no closing bracket after it)
-        for (const char* number :
-                {"0", "-0", "1", "12345678", "123456789", "1234567890123456789", "18446744073709551616",
-                 "1.5", "1e5", "1e", "1.", "-", "1.5e-3", "123456789012345678901234567890e-10"
-                })
-        {
-            check_both(number);
-            check_both(std::string("[") + number + "]");
-            check_both(std::string("{\"n\":") + number + "}");
-        }
-    }
-
-    SECTION("accept() of floats near the largest value")
-    {
-        // accept() skips digits without their value and computes it only for
-        // a float whose exponent may reach the largest value
-        for (const char* number :
-                {"17976931348623158e292", "17976931348623159e292", "1797693134862315.8e293", "1797693134862315.9e293",
-                 "0.17976931348623158e309", "0.17976931348623159e309", "-17976931348623159e292", "9999999999999999999e289"
-                })
-        {
-            check_both(number);
-            check_both(std::string("[") + number + "]");
-        }
-        CHECK(json::accept("17976931348623158e292"));
-        CHECK_FALSE(json::accept("17976931348623159e292"));
-        CHECK(json::accept("[1797693134862315.8e293]"));
-        CHECK_FALSE(json::accept("[1797693134862315.9e293]"));
-        CHECK(json::accept("0.17976931348623158e309"));
-        CHECK_FALSE(json::accept("0.17976931348623159e309"));
-        CHECK_FALSE(json::accept("-17976931348623159e292"));
-
-        using narrow_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int32_t, std::uint32_t, float>;
-        check_same<narrow_json>("34028235e31");
-        check_same<narrow_json>("34028236e31");
-        CHECK(narrow_json::accept("34028235e31"));
-        CHECK_FALSE(narrow_json::accept("34028236e31"));
-    }
-
-    SECTION("callbacks, comments and trailing commas take the library parser")
-    {
-        // (the results are those of the options; the contiguous parser does not support them)
-        CHECK(json::parse("[1,]", nullptr, true, false, true) == json({1}));
-        CHECK(json::parse("[1/*c*/]", nullptr, true, true) == json({1}));
-        CHECK(json::accept("[1,]", false, true));
-        CHECK(json::accept("// c\n[1]", true));
-        int calls = 0;
-        const json j = json::parse("[1,2]", [&calls](int /*depth*/, json::parse_event_t /*event*/, json& /*parsed*/)
-        {
-            ++calls;
-            return true;
-        });
-        CHECK(j == json({1, 2}));
-        CHECK(calls == 4);
-    }
-}

tests/src/unit-file_input.cpp

@@ -49,6 +49,14 @@ file_ptr open_with(const char* name, const std::string& content)
     return {std::fopen(name, "rb"), &std::fclose};
 }
 
+// the smallest size setvbuf() accepts for _IOFBF: the Microsoft C runtime
+// rejects sizes below 2 as an invalid parameter, which ends the process
+#if defined(_WIN32)
+    constexpr int min_full_buffer = 2;
+#else
+    constexpr int min_full_buffer = 1;
+#endif
+
 // The stdio lock is recursive, so whether parsing released it can only be
 // seen from another thread: ftrylockfile() fails there while this thread
 // holds the lock. ftrylockfile() is POSIX; elsewhere, the check is skipped.
@@ -225,6 +233,129 @@ TEST_CASE("parsing from a FILE*")
         check_unlocked(f.get());
     }
 
+#if !defined(JSON_NOEXCEPTION)
+    SECTION("tokens across the edges of the file's buffer")
+    {
+        // the adapter reads the buffer of the FILE directly where it can (see
+        // JSON_FILE_WINDOW): with buffers of 1 to 70 bytes and none, every
+        // token crosses an edge somewhere; the results, the error messages
+        // and their positions must be those of the string
+        const std::vector<std::string> documents =
+        {
+            input,
+            R"({"long": "a string that is longer than the buffer of the file, with \"escapes\" and \u00e4", "n": [123456789012345678901, -0.000123e-45, 1.5E+300, 0, -0]})",
+            "[\n    \"indented\",\n    \"caf\xc3\xa9 \xe2\x82\xac \xf0\x9f\x98\x80\",\n    {\n        \"key\": 1\n    }\n]",
+            R"({"a": [1, 2, 3], "b": "unterminated)",
+            R"([1, 2, 3, 1e999])",
+            R"({"a": "invalid escape \x"})",
+            "[\"invalid UTF-8 \xc3\x28\"]",
+            "[1, 2, 3] trailing",
+            R"([truex])",
+        };
+        for (const auto& document : documents)
+        {
+            std::string expected;
+            try
+            {
+                expected = json::parse(document).dump();
+            }
+            catch (const json::exception& e)
+            {
+                expected = e.what();
+            }
+            const bool expected_accept = json::accept(document);
+            for (int size = 0; size <= 70; ++size)
+            {
+                if (size != 0 && size < min_full_buffer)
+                {
+                    continue;
+                }
+                CAPTURE(document)
+                CAPTURE(size)
+                std::vector<char> buffer(static_cast<std::size_t>(size) + 1);
+                for (int pass = 0; pass < 2; ++pass)
+                {
+                    const file_ptr f = open_with(name, document);
+                    REQUIRE(f != nullptr);
+                    // size 0: unbuffered
+                    REQUIRE(std::setvbuf(f.get(), size == 0 ? nullptr : buffer.data(), size == 0 ? _IONBF : _IOFBF, static_cast<std::size_t>(size)) == 0);
+                    if (pass == 0)
+                    {
+                        std::string actual;
+                        try
+                        {
+                            actual = json::parse(f.get()).dump();
+                        }
+                        catch (const json::exception& e)
+                        {
+                            actual = e.what();
+                        }
+                        CHECK(actual == expected);
+                    }
+                    else
+                    {
+                        CHECK(json::accept(f.get()) == expected_accept);
+                    }
+                    check_unlocked(f.get());
+                }
+            }
+        }
+    }
+#endif
+
+#if !defined(JSON_NOEXCEPTION)
+    SECTION("characters pushed back with ungetc() before parsing")
+    {
+        // (glibc and Apple keep them in a separate buffer)
+        const file_ptr f = open_with(name, "2, 3]");
+        REQUIRE(f != nullptr);
+        REQUIRE(std::ungetc(',', f.get()) == ',');
+        REQUIRE(std::ungetc('1', f.get()) == '1');
+        REQUIRE(std::ungetc('[', f.get()) == '[');
+        CHECK(json::parse(f.get()) == json({1, 2, 3}));
+    }
+#endif
+
+    SECTION("SAX events across the edges of the file's buffer")
+    {
+        const std::string document = R"({"key": "a value that is longer than the buffer", "n": [1, -2.5e3, true, null]})";
+        std::vector<long> expected;
+        {
+            const file_ptr f = open_with(name, document);
+            REQUIRE(f != nullptr);
+            position_reading_sax sax(f.get());
+            CHECK(json::sax_parse(f.get(), &sax));
+            expected = sax.positions();
+        }
+        for (int size = min_full_buffer; size <= 20; ++size)
+        {
+            CAPTURE(size)
+            std::vector<char> buffer(static_cast<std::size_t>(size));
+            const file_ptr f = open_with(name, document);
+            REQUIRE(f != nullptr);
+            REQUIRE(std::setvbuf(f.get(), buffer.data(), _IOFBF, buffer.size()) == 0);
+            position_reading_sax sax(f.get());
+            CHECK(json::sax_parse(f.get(), &sax));
+            // the events happen at the same positions of the file
+            CHECK(sax.positions() == expected);
+        }
+    }
+
+    SECTION("binary strings across the edges of the file's buffer")
+    {
+        const json value = {{"a string that is longer than the buffer of the file", "and a value that is longer, too"}, {"n", {1, 2}}};
+        const std::vector<std::uint8_t> cbor = json::to_cbor(value);
+        for (int size = min_full_buffer; size <= 20; ++size)
+        {
+            CAPTURE(size)
+            std::vector<char> buffer(static_cast<std::size_t>(size));
+            const file_ptr f = open_with(name, std::string(cbor.begin(), cbor.end()));
+            REQUIRE(f != nullptr);
+            REQUIRE(std::setvbuf(f.get(), buffer.data(), _IOFBF, buffer.size()) == 0);
+            CHECK(json::from_cbor(f.get()) == value);
+        }
+    }
+
     SECTION("binary formats read from a FILE*")
     {
         const json j = {{"string", std::string(300, 'x')}, {"array", {1, 2, 3}}, {"number", 1.5}};