BSON writer: write to_bson(j) in one pass, with sizes written afterwards

BSON prefixes every document and array with its size. to_bson() therefore
walked the value twice: once to compute every size, once to write. When
the output is the vector that to_bson(j) returns, the writer now puts a
placeholder in front of each document and array and writes its size there
once the document is complete, so one walk is enough.

Every key and value is still checked as the size pass checked it, in the
same order, so the same exception is thrown; the partial output is
discarded with the vector. If a length would not fit BSON's 32-bit size
fields, the writer drops what it has written and writes the document in
two passes, which reports that error before writing anything, as before.

The two passes remain for output adapters (streams, strings, vectors
passed by reference), for error_handler_t::replace and ignore (which
change sizes), and for integer types wider than 64 bits.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

.github/CONTRIBUTING.md

@@ -154,11 +154,6 @@ and each page's title (H1) and declaration by
 [`docs/docset/generate_docset.py`](https://github.com/nlohmann/json/blob/develop/docs/docset/generate_docset.py);
 `make build` reports API pages that cannot be classified.
 
-A pull request that changes only the documentation, Markdown files, GitHub metadata, or license files skips the
-compiler workflows (Ubuntu, macOS, Windows); the documentation and license checks still run. See
-[`tools/check_path_filters`](https://github.com/nlohmann/json/tree/develop/tools/check_path_filters) for the list of
-skipped files and what to do when a compiler job starts reading one of them.
-
 ### Amalgamate the source code
 
 The single-header files

.github/workflows/macos.yml

@@ -7,25 +7,6 @@ on:
       - master
       - release/*
   pull_request:
-    # Skip pull requests that change only files no job here reads. The list is checked by
-    # tools/check_path_filters/check_path_filters.py (repository_checks.yml); change both together.
-    paths:
-      - '**'
-      - '!docs/**'
-      - '!**.md'
-      - '!.github/*'
-      - '!.github/ISSUE_TEMPLATE/**'
-      - '!.github/external_ci/**'
-      - '!.github/workflows/**'
-      - '!.reuse/**'
-      - '!LICENSES/**'
-      - '!CITATION.cff'
-      - '!REUSE.toml'
-      # read by tools/check_build_options in the ci_meson_install job (ubuntu.yml)
-      - 'docs/mkdocs/docs/integration/package_managers.md'
-      # built by the swiftpm job (macos.yml)
-      - 'docs/mkdocs/docs/integration/swift/**'
-      - '.github/workflows/macos.yml'
   workflow_dispatch:
 
 concurrency:

.github/workflows/repository_checks.yml

@@ -1,78 +0,0 @@
-name: Repository checks
-
-# Checks that concern files the compiler workflows (ubuntu.yml, macos.yml,
-# windows.yml) skip with their path filter: the documentation, the license
-# information, and the path filter itself. This workflow has no path filter.
-on:
-  push:
-    branches:
-      - develop
-      - master
-      - release/*
-  pull_request:
-  workflow_dispatch:
-
-permissions:
-  contents: read
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref || github.run_id }}
-  cancel-in-progress: true
-
-jobs:
-  check_path_filters:
-    runs-on: ubuntu-26.04
-    steps:
-      - name: Harden Runner
-        uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
-        with:
-          egress-policy: audit
-
-      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
-        with:
-          persist-credentials: false
-      - name: Check the path filters of the compiler workflows
-        run: python3 tools/check_path_filters/check_path_filters.py
-
-  ci_reuse_compliance:
-    runs-on: ubuntu-26.04
-    steps:
-      - name: Harden Runner
-        uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
-        with:
-          egress-policy: audit
-
-      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
-        with:
-          persist-credentials: false
-      - name: Get latest CMake and ninja
-        uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
-      - name: Run CMake
-        run: cmake -S . -B build -DJSON_CI=On
-      - name: Build
-        run: cmake --build build --target ci_reuse_compliance
-
-  ci_test_documentation:
-    runs-on: ubuntu-26.04
-    strategy:
-      matrix:
-        target: [ci_test_examples, ci_test_build_documentation, ci_test_documentation_mermaid]
-    steps:
-      - name: Harden Runner
-        uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
-        with:
-          egress-policy: audit
-
-      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
-        with:
-          persist-credentials: false
-          # the git-revision-date-localized plugin needs the history; a shallow clone makes the strict build fail
-          fetch-depth: ${{ matrix.target == 'ci_test_build_documentation' && '0' || '1' }}
-      - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
-        if: matrix.target == 'ci_test_documentation_mermaid'
-        with:
-          node-version: 24
-      - name: Run CMake
-        run: cmake -S . -B build -DJSON_CI=On
-      - name: Build
-        run: cmake --build build --target ${{ matrix.target }}

.github/workflows/ubuntu.yml

@@ -7,25 +7,6 @@ on:
       - master
       - release/*
   pull_request:
-    # Skip pull requests that change only files no job here reads. The list is checked by
-    # tools/check_path_filters/check_path_filters.py (repository_checks.yml); change both together.
-    paths:
-      - '**'
-      - '!docs/**'
-      - '!**.md'
-      - '!.github/*'
-      - '!.github/ISSUE_TEMPLATE/**'
-      - '!.github/external_ci/**'
-      - '!.github/workflows/**'
-      - '!.reuse/**'
-      - '!LICENSES/**'
-      - '!CITATION.cff'
-      - '!REUSE.toml'
-      # read by tools/check_build_options in the ci_meson_install job (ubuntu.yml)
-      - 'docs/mkdocs/docs/integration/package_managers.md'
-      # built by the swiftpm job (macos.yml)
-      - 'docs/mkdocs/docs/integration/swift/**'
-      - '.github/workflows/ubuntu.yml'
   workflow_dispatch:
 
 permissions:
@@ -128,7 +109,7 @@ jobs:
     runs-on: ubuntu-26.04
     strategy:
       matrix:
-        target: [ci_test_single_header, ci_cppcheck, ci_cpplint, ci_reproducible_tests, ci_non_git_tests, ci_offline_testdata, ci_test_valgrind]
+        target: [ci_test_single_header, ci_cppcheck, ci_cpplint, ci_reproducible_tests, ci_non_git_tests, ci_offline_testdata, ci_reuse_compliance, ci_test_valgrind]
     steps:
       - name: Harden Runner
         uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
@@ -468,6 +449,31 @@ jobs:
       - name: Build
         run: cmake --build build
 
+  ci_test_documentation:
+    runs-on: ubuntu-26.04
+    strategy:
+      matrix:
+        target: [ci_test_examples, ci_test_build_documentation, ci_test_documentation_mermaid]
+    steps:
+      - name: Harden Runner
+        uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
+        with:
+          egress-policy: audit
+
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+          # the git-revision-date-localized plugin needs the history; a shallow clone makes the strict build fail
+          fetch-depth: ${{ matrix.target == 'ci_test_build_documentation' && '0' || '1' }}
+      - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
+        if: matrix.target == 'ci_test_documentation_mermaid'
+        with:
+          node-version: 24
+      - name: Run CMake
+        run: cmake -S . -B build -DJSON_CI=On
+      - name: Build
+        run: cmake --build build --target ${{ matrix.target }}
+
   # Native Linux ARM64 runner with the latest GCC. The ubuntu-24.04-arm label is
   # only available for public repositories.
   ci_test_arm64:

.github/workflows/windows.yml

@@ -7,25 +7,6 @@ on:
       - master
       - release/*
   pull_request:
-    # Skip pull requests that change only files no job here reads. The list is checked by
-    # tools/check_path_filters/check_path_filters.py (repository_checks.yml); change both together.
-    paths:
-      - '**'
-      - '!docs/**'
-      - '!**.md'
-      - '!.github/*'
-      - '!.github/ISSUE_TEMPLATE/**'
-      - '!.github/external_ci/**'
-      - '!.github/workflows/**'
-      - '!.reuse/**'
-      - '!LICENSES/**'
-      - '!CITATION.cff'
-      - '!REUSE.toml'
-      # read by tools/check_build_options in the ci_meson_install job (ubuntu.yml)
-      - 'docs/mkdocs/docs/integration/package_managers.md'
-      # built by the swiftpm job (macos.yml)
-      - 'docs/mkdocs/docs/integration/swift/**'
-      - '.github/workflows/windows.yml'
   workflow_dispatch:
 
 permissions:

include/nlohmann/detail/output/binary_writer.hpp

@@ -2088,7 +2088,8 @@ class binary_writer
         /// arrays: the first index with one more digit; only maintained by
         /// @ref calc_bson_sizes
         std::uint64_t index_boundary = 10;
-        /// @ref calc_bson_sizes only: where its size goes in the table
+        /// @ref calc_bson_sizes: where its size goes in the table; @ref
+        /// write_bson_document_single_pass: the position of its size field
         std::size_t size_slot;
         /// @ref calc_bson_sizes only: the size of its entries seen so far
         std::size_t entries_size = 0;
@@ -2339,6 +2340,197 @@ class binary_writer
         }
     }
 
+    template<typename SinkType>
+    using sink_written_t = decltype(std::declval<const SinkType&>().written());
+
+    /*!
+    @brief whether BSON documents can be written in a single pass: the sink
+           can overwrite what it has written (output_vector_sink), and every
+           integer fits the 64 bits BSON has, so that writing an array of
+           numbers needs no range check
+    */
+    using bson_single_pass_sink = std::integral_constant < bool,
+          is_detected<sink_written_t, OutputSinkType>::value
+          && std::numeric_limits<typename BasicJsonType::number_integer_t>::is_specialized
+          && std::numeric_limits<typename BasicJsonType::number_integer_t>::digits <= std::numeric_limits<std::int64_t>::digits
+          && std::numeric_limits<typename BasicJsonType::number_unsigned_t>::is_specialized
+          && std::numeric_limits<typename BasicJsonType::number_unsigned_t>::digits <= std::numeric_limits<std::uint64_t>::digits >;
+
+    /// write a placeholder for the size of an object or array whose entries
+    /// follow; @return its position, for @ref patch_bson_size
+    std::size_t write_bson_size_placeholder()
+    {
+        const std::size_t pos = oa.written();
+        write_number<std::int32_t>(0, true);
+        return pos;
+    }
+
+    /// end the object or array whose size is at @a pos and write its size
+    /// there: everything from that field to the terminating null byte
+    /// @return false (and the size is not written) if it does not fit
+    bool close_bson_container(const std::size_t pos)
+    {
+        oa.write_character(to_char_type(0x00));
+        const std::size_t size = oa.written() - pos;
+        if (JSON_HEDLEY_UNLIKELY(!value_in_range_of<std::int32_t>(size)))
+        {
+            return false;
+        }
+        std::array<CharType, sizeof(std::int32_t)> bytes{};
+        store_little_endian<std::int32_t>(bytes.data(), static_cast<std::int32_t>(size));
+        oa.overwrite(pos, bytes.data(), bytes.size());
+        return true;
+    }
+
+    /*!
+    @brief check the value @a j of a BSON entry as @ref calc_bson_value_size
+           does, for @ref write_bson_document_single_pass
+    @return at least the size of the value
+    A number, boolean, or null needs no check there (its integers have at most
+    64 bits) and has at most 8 characters; a discarded value is rejected with
+    the same exception when it is written.
+    */
+    std::size_t check_bson_value(const BasicJsonType& j)
+    {
+        return (j.is_string() || j.is_binary()) ? calc_bson_value_size(j) : 8;
+    }
+
+    /// whether an entry of @a entry_size characters keeps the document that
+    /// begins at @a start, closed by @a open_containers null bytes, within
+    /// the 32-bit size of BSON
+    bool bson_entry_fits(const std::size_t start, const std::size_t entry_size, const std::size_t open_containers) const noexcept
+    {
+        const std::size_t limit = static_cast<std::size_t>((std::numeric_limits<std::int32_t>::max)());
+        const std::size_t written = oa.written() - start;
+        return written <= limit && entry_size <= limit - written && open_containers <= limit - written - entry_size;
+    }
+
+    /*!
+    @brief Serializes the JSON object @a document as a BSON document in a
+           single pass, writing the size of each object and array once it is
+           complete
+
+    Checks every key and value as @ref calc_bson_sizes does, in the same
+    order, so the same exception is thrown; the output written up to then is
+    discarded with the vector it was written to.
+
+    @return false as soon as the document or a string, binary value, object,
+            or array in it would be larger than the 32-bit BSON sizes allow;
+            nothing large has been written then, and the caller writes the
+            document in two passes, which reports that error
+    */
+    template < typename SinkType = OutputSinkType,
+               typename std::enable_if < is_detected<sink_written_t, SinkType>::value, int >::type = 0 >
+    bool write_bson_document_single_pass(const BasicJsonType& document)
+    {
+        const std::size_t start = oa.written();
+        // the object or array whose entries are being written (with the
+        // position of its size), and the ones it is in
+        bson_frame current(&document, write_bson_size_placeholder());
+        std::vector<bson_frame> parents;
+
+        while (true)
+        {
+            const BasicJsonType* nested = nullptr;
+            std::size_t nested_size_pos = 0;
+            if (current.value->is_object())
+            {
+                const auto& object = *current.value->m_data.m_value.object;
+                while (nested == nullptr && current.member != object.cend())
+                {
+                    const auto& el = *current.member;
+                    ++current.member;
+                    const std::size_t header_size = calc_bson_entry_header_size(el.first, el.second);
+                    const std::size_t value_size = el.second.is_structured() ? 0 : check_bson_value(el.second);
+                    if (JSON_HEDLEY_UNLIKELY(!bson_entry_fits(start, header_size + value_size, parents.size() + 1)))
+                    {
+                        return false;
+                    }
+                    if (el.second.is_structured())
+                    {
+                        write_bson_entry_header(el.first, el.second.is_object() ? 0x03 : 0x04);
+                        nested_size_pos = write_bson_size_placeholder();
+                        nested = &el.second;
+                    }
+                    else
+                    {
+                        write_bson_value(el.first, el.second);
+                    }
+                }
+            }
+            else
+            {
+                const auto& array = *current.value->m_data.m_value.array;
+                while (nested == nullptr && current.index < array.size())
+                {
+                    if (is_bson_fixed_size(array[current.index]))
+                    {
+                        write_bson_array_fixed_entries(current);
+                        if (current.index == array.size())
+                        {
+                            break;
+                        }
+                    }
+
+                    // a string, binary value, object or array
+                    const BasicJsonType& el = array[current.index];
+                    const string_t index_name(current.index_name.data(), current.index_name_size);
+                    current.next_index();
+                    const std::size_t value_size = el.is_structured() ? 0 : check_bson_value(el);
+                    if (JSON_HEDLEY_UNLIKELY(!bson_entry_fits(start, 2 + index_name.size() + value_size, parents.size() + 1)))
+                    {
+                        return false;
+                    }
+                    if (el.is_structured())
+                    {
+                        write_bson_entry_header(index_name, el.is_object() ? 0x03 : 0x04);
+                        nested_size_pos = write_bson_size_placeholder();
+                        nested = &el;
+                    }
+                    else
+                    {
+                        write_bson_value(index_name, el);
+                    }
+                }
+            }
+
+            if (nested != nullptr)
+            {
+                if (nested->is_array())
+                {
+                    // an array of booleans, numbers and null is written without a frame on the stack
+                    bson_frame leaf(nested, nested_size_pos);
+                    write_bson_array_fixed_entries(leaf);
+                    if (leaf.index == nested->m_data.m_value.array->size())
+                    {
+                        if (JSON_HEDLEY_UNLIKELY(!close_bson_container(nested_size_pos)))
+                        {
+                            return false;
+                        }
+                        continue;
+                    }
+                    parents.push_back(std::move(current));
+                    current = std::move(leaf);
+                    continue;
+                }
+                parents.push_back(std::move(current));
+                current = bson_frame(nested, nested_size_pos);
+                continue;
+            }
+
+            if (JSON_HEDLEY_UNLIKELY(!close_bson_container(current.size_slot)))
+            {
+                return false;
+            }
+            if (parents.empty())
+            {
+                return true;
+            }
+            current = std::move(parents.back());
+            parents.pop_back();
+        }
+    }
+
     /*!
     @brief Serializes the JSON object @a document as a BSON document
 
@@ -2350,6 +2542,39 @@ class binary_writer
     @pre       document.type() == value_t::object
     */
     void write_bson_document(const BasicJsonType& document)
+    {
+        write_bson_document(document, bson_single_pass_sink());
+    }
+
+    /// the sink can overwrite: one pass, unless the error handler replaces or
+    /// drops ill-formed UTF-8, which changes the sizes the checks compute
+    void write_bson_document(const BasicJsonType& document, std::true_type /*single_pass*/)
+    {
+        const std::size_t start = oa.written();
+        if ((error_handler == error_handler_t::keep || error_handler == error_handler_t::strict)
+                && write_bson_document_single_pass(document))
+        {
+            return;
+        }
+        // a length does not fit: start over, so that it is reported as the
+        // two passes report it, before anything is written
+        oa.rewind(start);
+        write_bson_document_two_pass(document);
+    }
+
+    void write_bson_document(const BasicJsonType& document, std::false_type /*single_pass*/)
+    {
+        write_bson_document_two_pass(document);
+    }
+
+    /*!
+    @brief Serializes the JSON object @a document as a BSON document: the
+           sizes of all objects and arrays first, then the entries
+
+    @param[in] document  the JSON object to serialize
+    @pre       document.type() == value_t::object
+    */
+    void write_bson_document_two_pass(const BasicJsonType& document)
     {
         std::vector<std::size_t> nested_sizes;
         const std::size_t document_size = calc_bson_sizes(document, nested_sizes);

include/nlohmann/detail/output/output_adapters.hpp

@@ -227,6 +227,29 @@ class output_vector_sink
     void flush() noexcept
     {}
 
+    /// the number of characters written so far, the position of the next one
+    std::size_t written() const noexcept
+    {
+        return position();
+    }
+
+    /// @brief replace @a length characters written before, from position @a pos
+    ///
+    /// Lets a writer put a length in front of data it has written since, such
+    /// as the size of a BSON document, without computing it in advance.
+    void overwrite(const std::size_t pos, const CharType* s, const std::size_t length) noexcept
+    {
+        JSON_ASSERT(pos + length <= position());
+        std::memcpy(v->data() + pos, s, length * sizeof(CharType));
+    }
+
+    /// drop the characters written from position @a pos on
+    void rewind(const std::size_t pos) noexcept
+    {
+        JSON_ASSERT(pos <= position());
+        cursor = v->data() + pos;
+    }
+
   private:
     /// number of characters of *v written so far
     std::size_t position() const noexcept

include/nlohmann/json_view.hpp

@@ -766,9 +766,6 @@ class basic_json_view
     template<typename, typename> friend class detail::view::view_side;
     friend iterator;
 
-    /// whether the view is one of an editable document
-    static constexpr bool is_editable = Editable;
-
     basic_json_view(const document_data* d, const node* n) noexcept
         : m_doc(d), m_node(n)
     {}

single_include/nlohmann/json.hpp

@@ -26870,6 +26870,29 @@ class output_vector_sink
     void flush() noexcept
     {}
 
+    /// the number of characters written so far, the position of the next one
+    std::size_t written() const noexcept
+    {
+        return position();
+    }
+
+    /// @brief replace @a length characters written before, from position @a pos
+    ///
+    /// Lets a writer put a length in front of data it has written since, such
+    /// as the size of a BSON document, without computing it in advance.
+    void overwrite(const std::size_t pos, const CharType* s, const std::size_t length) noexcept
+    {
+        JSON_ASSERT(pos + length <= position());
+        std::memcpy(v->data() + pos, s, length * sizeof(CharType));
+    }
+
+    /// drop the characters written from position @a pos on
+    void rewind(const std::size_t pos) noexcept
+    {
+        JSON_ASSERT(pos <= position());
+        cursor = v->data() + pos;
+    }
+
   private:
     /// number of characters of *v written so far
     std::size_t position() const noexcept
@@ -29314,7 +29337,8 @@ class binary_writer
         /// arrays: the first index with one more digit; only maintained by
         /// @ref calc_bson_sizes
         std::uint64_t index_boundary = 10;
-        /// @ref calc_bson_sizes only: where its size goes in the table
+        /// @ref calc_bson_sizes: where its size goes in the table; @ref
+        /// write_bson_document_single_pass: the position of its size field
         std::size_t size_slot;
         /// @ref calc_bson_sizes only: the size of its entries seen so far
         std::size_t entries_size = 0;
@@ -29565,6 +29589,197 @@ class binary_writer
         }
     }
 
+    template<typename SinkType>
+    using sink_written_t = decltype(std::declval<const SinkType&>().written());
+
+    /*!
+    @brief whether BSON documents can be written in a single pass: the sink
+           can overwrite what it has written (output_vector_sink), and every
+           integer fits the 64 bits BSON has, so that writing an array of
+           numbers needs no range check
+    */
+    using bson_single_pass_sink = std::integral_constant < bool,
+          is_detected<sink_written_t, OutputSinkType>::value
+          && std::numeric_limits<typename BasicJsonType::number_integer_t>::is_specialized
+          && std::numeric_limits<typename BasicJsonType::number_integer_t>::digits <= std::numeric_limits<std::int64_t>::digits
+          && std::numeric_limits<typename BasicJsonType::number_unsigned_t>::is_specialized
+          && std::numeric_limits<typename BasicJsonType::number_unsigned_t>::digits <= std::numeric_limits<std::uint64_t>::digits >;
+
+    /// write a placeholder for the size of an object or array whose entries
+    /// follow; @return its position, for @ref patch_bson_size
+    std::size_t write_bson_size_placeholder()
+    {
+        const std::size_t pos = oa.written();
+        write_number<std::int32_t>(0, true);
+        return pos;
+    }
+
+    /// end the object or array whose size is at @a pos and write its size
+    /// there: everything from that field to the terminating null byte
+    /// @return false (and the size is not written) if it does not fit
+    bool close_bson_container(const std::size_t pos)
+    {
+        oa.write_character(to_char_type(0x00));
+        const std::size_t size = oa.written() - pos;
+        if (JSON_HEDLEY_UNLIKELY(!value_in_range_of<std::int32_t>(size)))
+        {
+            return false;
+        }
+        std::array<CharType, sizeof(std::int32_t)> bytes{};
+        store_little_endian<std::int32_t>(bytes.data(), static_cast<std::int32_t>(size));
+        oa.overwrite(pos, bytes.data(), bytes.size());
+        return true;
+    }
+
+    /*!
+    @brief check the value @a j of a BSON entry as @ref calc_bson_value_size
+           does, for @ref write_bson_document_single_pass
+    @return at least the size of the value
+    A number, boolean, or null needs no check there (its integers have at most
+    64 bits) and has at most 8 characters; a discarded value is rejected with
+    the same exception when it is written.
+    */
+    std::size_t check_bson_value(const BasicJsonType& j)
+    {
+        return (j.is_string() || j.is_binary()) ? calc_bson_value_size(j) : 8;
+    }
+
+    /// whether an entry of @a entry_size characters keeps the document that
+    /// begins at @a start, closed by @a open_containers null bytes, within
+    /// the 32-bit size of BSON
+    bool bson_entry_fits(const std::size_t start, const std::size_t entry_size, const std::size_t open_containers) const noexcept
+    {
+        const std::size_t limit = static_cast<std::size_t>((std::numeric_limits<std::int32_t>::max)());
+        const std::size_t written = oa.written() - start;
+        return written <= limit && entry_size <= limit - written && open_containers <= limit - written - entry_size;
+    }
+
+    /*!
+    @brief Serializes the JSON object @a document as a BSON document in a
+           single pass, writing the size of each object and array once it is
+           complete
+
+    Checks every key and value as @ref calc_bson_sizes does, in the same
+    order, so the same exception is thrown; the output written up to then is
+    discarded with the vector it was written to.
+
+    @return false as soon as the document or a string, binary value, object,
+            or array in it would be larger than the 32-bit BSON sizes allow;
+            nothing large has been written then, and the caller writes the
+            document in two passes, which reports that error
+    */
+    template < typename SinkType = OutputSinkType,
+               typename std::enable_if < is_detected<sink_written_t, SinkType>::value, int >::type = 0 >
+    bool write_bson_document_single_pass(const BasicJsonType& document)
+    {
+        const std::size_t start = oa.written();
+        // the object or array whose entries are being written (with the
+        // position of its size), and the ones it is in
+        bson_frame current(&document, write_bson_size_placeholder());
+        std::vector<bson_frame> parents;
+
+        while (true)
+        {
+            const BasicJsonType* nested = nullptr;
+            std::size_t nested_size_pos = 0;
+            if (current.value->is_object())
+            {
+                const auto& object = *current.value->m_data.m_value.object;
+                while (nested == nullptr && current.member != object.cend())
+                {
+                    const auto& el = *current.member;
+                    ++current.member;
+                    const std::size_t header_size = calc_bson_entry_header_size(el.first, el.second);
+                    const std::size_t value_size = el.second.is_structured() ? 0 : check_bson_value(el.second);
+                    if (JSON_HEDLEY_UNLIKELY(!bson_entry_fits(start, header_size + value_size, parents.size() + 1)))
+                    {
+                        return false;
+                    }
+                    if (el.second.is_structured())
+                    {
+                        write_bson_entry_header(el.first, el.second.is_object() ? 0x03 : 0x04);
+                        nested_size_pos = write_bson_size_placeholder();
+                        nested = &el.second;
+                    }
+                    else
+                    {
+                        write_bson_value(el.first, el.second);
+                    }
+                }
+            }
+            else
+            {
+                const auto& array = *current.value->m_data.m_value.array;
+                while (nested == nullptr && current.index < array.size())
+                {
+                    if (is_bson_fixed_size(array[current.index]))
+                    {
+                        write_bson_array_fixed_entries(current);
+                        if (current.index == array.size())
+                        {
+                            break;
+                        }
+                    }
+
+                    // a string, binary value, object or array
+                    const BasicJsonType& el = array[current.index];
+                    const string_t index_name(current.index_name.data(), current.index_name_size);
+                    current.next_index();
+                    const std::size_t value_size = el.is_structured() ? 0 : check_bson_value(el);
+                    if (JSON_HEDLEY_UNLIKELY(!bson_entry_fits(start, 2 + index_name.size() + value_size, parents.size() + 1)))
+                    {
+                        return false;
+                    }
+                    if (el.is_structured())
+                    {
+                        write_bson_entry_header(index_name, el.is_object() ? 0x03 : 0x04);
+                        nested_size_pos = write_bson_size_placeholder();
+                        nested = &el;
+                    }
+                    else
+                    {
+                        write_bson_value(index_name, el);
+                    }
+                }
+            }
+
+            if (nested != nullptr)
+            {
+                if (nested->is_array())
+                {
+                    // an array of booleans, numbers and null is written without a frame on the stack
+                    bson_frame leaf(nested, nested_size_pos);
+                    write_bson_array_fixed_entries(leaf);
+                    if (leaf.index == nested->m_data.m_value.array->size())
+                    {
+                        if (JSON_HEDLEY_UNLIKELY(!close_bson_container(nested_size_pos)))
+                        {
+                            return false;
+                        }
+                        continue;
+                    }
+                    parents.push_back(std::move(current));
+                    current = std::move(leaf);
+                    continue;
+                }
+                parents.push_back(std::move(current));
+                current = bson_frame(nested, nested_size_pos);
+                continue;
+            }
+
+            if (JSON_HEDLEY_UNLIKELY(!close_bson_container(current.size_slot)))
+            {
+                return false;
+            }
+            if (parents.empty())
+            {
+                return true;
+            }
+            current = std::move(parents.back());
+            parents.pop_back();
+        }
+    }
+
     /*!
     @brief Serializes the JSON object @a document as a BSON document
 
@@ -29576,6 +29791,39 @@ class binary_writer
     @pre       document.type() == value_t::object
     */
     void write_bson_document(const BasicJsonType& document)
+    {
+        write_bson_document(document, bson_single_pass_sink());
+    }
+
+    /// the sink can overwrite: one pass, unless the error handler replaces or
+    /// drops ill-formed UTF-8, which changes the sizes the checks compute
+    void write_bson_document(const BasicJsonType& document, std::true_type /*single_pass*/)
+    {
+        const std::size_t start = oa.written();
+        if ((error_handler == error_handler_t::keep || error_handler == error_handler_t::strict)
+                && write_bson_document_single_pass(document))
+        {
+            return;
+        }
+        // a length does not fit: start over, so that it is reported as the
+        // two passes report it, before anything is written
+        oa.rewind(start);
+        write_bson_document_two_pass(document);
+    }
+
+    void write_bson_document(const BasicJsonType& document, std::false_type /*single_pass*/)
+    {
+        write_bson_document_two_pass(document);
+    }
+
+    /*!
+    @brief Serializes the JSON object @a document as a BSON document: the
+           sizes of all objects and arrays first, then the entries
+
+    @param[in] document  the JSON object to serialize
+    @pre       document.type() == value_t::object
+    */
+    void write_bson_document_two_pass(const BasicJsonType& document)
     {
         std::vector<std::size_t> nested_sizes;
         const std::size_t document_size = calc_bson_sizes(document, nested_sizes);

single_include/nlohmann/json_view.hpp

@@ -9807,9 +9807,6 @@ class basic_json_view
     template<typename, typename> friend class detail::view::view_side;
     friend iterator;
 
-    /// whether the view is one of an editable document
-    static constexpr bool is_editable = Editable;
-
     basic_json_view(const document_data* d, const node* n) noexcept
         : m_doc(d), m_node(n)
     {}

tests/src/unit-bson.cpp

@@ -2267,3 +2267,132 @@ TEST_CASE("BSON large strings and binaries (chunked reader)")
         }
     }
 }
+
+TEST_CASE("BSON written in one pass and in two passes")
+{
+    // to_bson() into a returned vector writes each size once the object or
+    // array is complete; through an output adapter, all sizes are computed
+    // first. Both must produce the same bytes and throw the same exceptions.
+    const auto two_pass = [](const json & j, const json::error_handler_t handler)
+    {
+        std::vector<std::uint8_t> result;
+        json::to_bson(j, result, handler);
+        return result;
+    };
+
+    const auto check_same = [&](const json & j, const json::error_handler_t handler)
+    {
+        CAPTURE(j.dump(-1, ' ', true, json::error_handler_t::replace))
+        CAPTURE(static_cast<int>(handler))
+#if !defined(JSON_NOEXCEPTION)
+        std::string one_pass_error;
+        std::string two_pass_error;
+        std::vector<std::uint8_t> one;
+        std::vector<std::uint8_t> two;
+        try
+        {
+            one = json::to_bson(j, handler);
+        }
+        catch (const json::exception& e)
+        {
+            one_pass_error = e.what();
+        }
+        try
+        {
+            two = two_pass(j, handler);
+        }
+        catch (const json::exception& e)
+        {
+            two_pass_error = e.what();
+        }
+        CHECK(one_pass_error == two_pass_error);
+        CHECK(one == two);
+#else
+        CHECK(json::to_bson(j, handler) == two_pass(j, handler));
+#endif
+    };
+
+    const json::error_handler_t handlers[] = {json::error_handler_t::keep, json::error_handler_t::strict,
+                                              json::error_handler_t::replace, json::error_handler_t::ignore
+                                             };
+
+    SECTION("corpus")
+    {
+        for (const auto& value : utils::round_trip_corpus::values())
+        {
+            const json document = value.is_object() ? value : json{{"value", value}};
+            for (const auto handler : handlers)
+            {
+                check_same(document, handler);
+            }
+        }
+    }
+
+    SECTION("nesting and arrays of every length class")
+    {
+        json document = json::object();
+        document["empty object"] = json::object();
+        document["empty array"] = json::array();
+        for (const std::size_t n :
+                {
+                    std::size_t(1), std::size_t(9), std::size_t(10), std::size_t(11), std::size_t(99), std::size_t(100), std::size_t(101), std::size_t(1000)
+                })
+        {
+            json numbers = json::array();
+            json mixed = json::array();
+            for (std::size_t i = 0; i < n; ++i)
+            {
+                numbers.push_back(i % 3 == 0 ? json(static_cast<double>(i) / 3) : json(static_cast<std::int64_t>(i) * 1000003));
+                mixed.push_back(i % 4 == 0 ? json("s" + std::to_string(i)) : i % 4 == 1 ? json::array({i, json::object({{"k", i}})}) : i % 4 == 2 ? json(nullptr) : json(i % 2 == 0));
+            }
+            document["numbers " + std::to_string(n)] = numbers;
+            document["mixed " + std::to_string(n)] = mixed;
+            document["nested " + std::to_string(n)] = json::object({{"a", numbers}, {"b", json::array({mixed, json::object()})}});
+        }
+        document["binary"] = json::binary({1, 2, 3}, 0x80);
+        document["binary old"] = json::binary({1, 2, 3}, 0x02);
+        for (const auto handler : handlers)
+        {
+            check_same(document, handler);
+        }
+    }
+
+    SECTION("errors at every position")
+    {
+        // an entry that cannot be written, at each place among valid ones
+        const std::string bad_utf8 = "\xC3\x28";
+        const json bad_entries[] =
+        {
+            json::object({{std::string("a\0b", 3), 1}}),
+            json::object({{"s", bad_utf8}}),
+            json::object({{bad_utf8, 1}}),
+            json::array({1, 2, bad_utf8}),
+            json::object({{"b", json::binary({1}, 0x100)}}),
+            json::object({{"d", json(json::value_t::discarded)}}),
+            json::array({json(json::value_t::discarded)}),
+        };
+        for (const auto& bad : bad_entries)
+        {
+            for (std::size_t pos = 0; pos < 4; ++pos)
+            {
+                json document = json::object();
+                json list = json::array();
+                for (std::size_t i = 0; i < 4; ++i)
+                {
+                    if (i == pos)
+                    {
+                        document["x" + std::to_string(i)] = bad;
+                        list.push_back(bad);
+                    }
+                    document["n" + std::to_string(i)] = json::array({i, "str", json::object({{"k", i}})});
+                    list.push_back(i);
+                }
+                document["list"] = list;
+                for (const auto handler : handlers)
+                {
+                    check_same(document, handler);
+                }
+            }
+        }
+    }
+}

tools/check_path_filters/README.md

@@ -1,34 +0,0 @@
-# check_path_filters
-
-The compiler workflows [`ubuntu.yml`](../../.github/workflows/ubuntu.yml),
-[`macos.yml`](../../.github/workflows/macos.yml), and [`windows.yml`](../../.github/workflows/windows.yml) skip pull
-requests that change only files no compiler job reads: the documentation, Markdown files, GitHub metadata, other
-workflows, and the license files. Pushes to `develop`, `master`, and `release/*` always run them. The documentation,
-license, and path-filter checks run in [`repository_checks.yml`](../../.github/workflows/repository_checks.yml), which
-has no path filter.
-
-The filter starts with `'**'` and excludes these files explicitly, so a new file runs the compiler workflows unless it
-is excluded on purpose. [`check_path_filters.py`](check_path_filters.py) holds the filter (`PATHS`) and checks that
-
-- each of the three workflows has exactly this filter, with its own workflow file included,
-- the CI targets that read excluded files (`UNFILTERED_TARGETS`) do not run in these workflows, and
-- no file the filter keeps (sources, tests, build files, tools, workflows) mentions an excluded file or directory
-  outside a comment line, unless the mention is listed in `KNOWN_REFERENCES` with a reason.
-
-When the check reports a mention, decide whether a compiler job reads that file:
-
-- If it does, include the file in `PATHS` after the exclusions (as `package_managers.md` and the Swift example are) and
-  copy the new filter into the three workflows.
-- If it does not, add the mention to `KNOWN_REFERENCES` with the reason.
-
-Run the check with
-
-```shell
-python3 tools/check_path_filters/check_path_filters.py
-```
-
-To see which workflows a pull request would run, pass the changed files:
-
-```shell
-python3 tools/check_path_filters/check_path_filters.py --changed $(git diff --name-only origin/develop)
-```

tools/check_path_filters/check_path_filters.py

@@ -1,269 +0,0 @@
-#!/usr/bin/env python3
-"""Check the path filters of the compiler workflows.
-
-ubuntu.yml, macos.yml and windows.yml skip pull requests that change only files
-no compiler job reads (documentation, Markdown files, GitHub metadata, license
-files). The filter starts with '**' and excludes these files explicitly, so a
-new file runs the workflows unless it is excluded on purpose.
-
-The script checks that
-
-- each of the three workflows has exactly the filter defined in PATHS below,
-  with its own workflow file included, and
-- no tracked file that the filter keeps (sources, tests, build files, tools,
-  workflows) mentions a file the filter excludes, unless the mention is listed
-  in KNOWN_REFERENCES with the reason why it does not matter.
-
-A mention found this way must either be added to PATHS as an included path
-(the file is read by a compiler job) or to KNOWN_REFERENCES (it is not).
-
-With --changed FILE..., the script prints instead which workflows a pull
-request changing these files would run.
-"""
-
-import argparse
-import os
-import re
-import subprocess  # nosec B404
-import sys
-
-REPO_ROOT = os.path.normpath(os.path.join(sys.path[0], '..', '..'))
-SELF = 'tools/check_path_filters/check_path_filters.py'
-
-WORKFLOWS = ['.github/workflows/ubuntu.yml', '.github/workflows/macos.yml', '.github/workflows/windows.yml']
-
-# The pull_request paths filter of the workflows above; '{workflow}' is replaced
-# with the workflow's own file. GitHub applies the patterns in order: a file
-# matching a later pattern is included ('pattern') or excluded ('!pattern').
-PATHS = [
-    '**',
-    '!docs/**',
-    '!**.md',
-    '!.github/*',
-    '!.github/ISSUE_TEMPLATE/**',
-    '!.github/external_ci/**',
-    '!.github/workflows/**',
-    '!.reuse/**',
-    '!LICENSES/**',
-    '!CITATION.cff',
-    '!REUSE.toml',
-    # read by tools/check_build_options in the ci_meson_install job (ubuntu.yml)
-    'docs/mkdocs/docs/integration/package_managers.md',
-    # built by the swiftpm job (macos.yml)
-    'docs/mkdocs/docs/integration/swift/**',
-    '{workflow}',
-]
-
-# Mentions of excluded files that no job of the three workflows depends on:
-# (file that mentions it, the mention, reason)
-KNOWN_REFERENCES = [
-    ('cmake/ci.cmake', 'docs/', 'ci_test_examples; runs in repository_checks.yml, which has no path filter'),
-    ('cmake/ci.cmake', 'docs/mkdocs/', 'ci_test_build_documentation and ci_test_documentation_mermaid; they run in '
-                                       'repository_checks.yml, which has no path filter'),
-    ('cmake/ci.cmake', 'docs/mkdocs/docs/examples/', 'indentation check of ci_test_amalgamation; the same check runs in '
-                                                     'check_amalgamation.yml, which has no path filter'),
-    ('Makefile', 'docs/mkdocs/docs/examples/', 'make pretty; runs in check_amalgamation.yml, which has no path filter'),
-    ('Makefile', 'ChangeLog.md', 'make ChangeLog.md writes the file; no CI job runs it'),
-    ('Makefile', 'tests/fuzzing.md', 'help text'),
-    ('Makefile', 'tools/generate_natvis/README.md', 'help text'),
-    ('tests/src/make_test_data_available.hpp', 'README.md', 'a file of the external test data repository'),
-    ('tools/check_build_options/check_build_options.py', 'docs/', 'builds the path of package_managers.md, which '
-                                                                  'PATHS includes'),
-]
-
-
-# CI targets that read excluded files; they must run in a workflow without a path filter
-UNFILTERED_TARGETS = ['ci_test_examples', 'ci_test_build_documentation', 'ci_test_documentation_mermaid',
-                      'ci_test_amalgamation', 'ci_reuse_compliance']
-
-
-def git_files(root):
-    # fixed command, no user input; includes new files not yet added, so a local run
-    # sees what CI sees after the commit
-    out = subprocess.check_output(['git', 'ls-files', '-z', '--cached', '--others', '--exclude-standard'], cwd=root)  # nosec B603, B607
-    return [f for f in out.decode('utf-8').split('\0') if f]
-
-
-def glob_to_regex(pattern):
-    """Translate a GitHub Actions path pattern to a regular expression."""
-    regex = ''
-    i = 0
-    while i < len(pattern):
-        if pattern.startswith('**', i):
-            regex += '.*'
-            i += 2
-        elif pattern[i] == '*':
-            regex += '[^/]*'
-            i += 1
-        elif pattern[i] == '?':
-            regex += '[^/]'
-            i += 1
-        else:
-            regex += re.escape(pattern[i])
-            i += 1
-    return re.compile(regex + r'\Z')
-
-
-def compile_filter(paths, workflow):
-    return [(p.startswith('!'), glob_to_regex(p.lstrip('!').replace('{workflow}', workflow))) for p in paths]
-
-
-def included(compiled, path):
-    result = False
-    for negated, regex in compiled:
-        if regex.match(path):
-            result = not negated
-    return result
-
-
-def read_filter(root, workflow):
-    """Return the pull_request paths list of a workflow file."""
-    with open(os.path.join(root, workflow), encoding='utf-8') as f:
-        lines = f.read().splitlines()
-    paths = None
-    in_pull_request = False
-    for line in lines:
-        if re.match(r'^  pull_request:', line):
-            in_pull_request = True
-            continue
-        if in_pull_request:
-            if re.match(r'^  \S', line) or re.match(r'^\S', line):
-                break
-            if re.match(r'^    paths:\s*$', line):
-                paths = []
-                continue
-            m = re.match(r"^      - '(.*)'\s*(#.*)?$", line)
-            if paths is not None and m:
-                paths.append(m.group(1))
-            elif paths is not None and line.strip() and not line.strip().startswith('#'):
-                break
-    return paths
-
-
-TOKEN = re.compile(r'(["\'/]?)([\w.-]+(?:/[\w.-]+)*)')
-
-
-COMMENT_LINE = re.compile(r'^[ \t]*(//|#|/?\*|--).*$', re.MULTILINE)
-
-
-class Mentions:
-    """Finds the words and paths in a text that name an excluded file or directory."""
-
-    def __init__(self, files, excluded, is_excluded):
-        self.files = set(files)
-        self.excluded = excluded
-        self.dirs = {'/'.join(f.split('/')[:n]) for f in files for n in range(1, f.count('/') + 1)}
-        # an excluded directory: a new file in it would be excluded
-        self.excluded_dirs = {d for d in self.dirs if is_excluded(d + '/new_file')}
-        # base names that only excluded files have, e.g. 'ChangeLog.md' but not 'CMakeLists.txt'
-        self.names = {os.path.basename(f) for f in excluded} - {os.path.basename(f) for f in files if f not in excluded}
-
-    def find(self, text):
-        """Yield (offset, mention) for every word or path in text that names an excluded file or directory."""
-        for m in TOKEN.finditer(text):
-            parts = m.group(2).rstrip('/').split('/')
-            # a single word names a directory only in a path context, e.g. 'docs' in os.path.join('docs', ...)
-            path_context = len(parts) > 1 or m.group(1)
-            for i in range(len(parts)):
-                path = '/'.join(parts[i:])
-                if path in self.files:
-                    if path in self.excluded:
-                        yield m.start(2), path
-                    break
-                if path in self.dirs and path_context:
-                    if path in self.excluded_dirs:
-                        yield m.start(2), path + '/'
-                    break
-            else:
-                if parts[-1] in self.names and '.' in parts[-1]:
-                    yield m.start(2), parts[-1]
-
-
-def known(path, mention):
-    return any(path == file and mention == known_mention for file, known_mention, _ in KNOWN_REFERENCES)
-
-
-def check_workflows(root):
-    """Return the errors in the filters of the workflows and in the targets they run."""
-    errors = []
-    for workflow in WORKFLOWS:
-        expected = [p.replace('{workflow}', workflow) for p in PATHS]
-        actual = read_filter(root, workflow)
-        if actual != expected:
-            errors.append('{}: the pull_request paths filter differs from tools/check_path_filters/check_path_filters.py\n'
-                          '  expected: {}\n  actual:   {}'.format(workflow, expected, actual))
-        with open(os.path.join(root, workflow), encoding='utf-8') as f:
-            text = f.read()
-        errors += ['{}: runs {}, which reads excluded files; run it in repository_checks.yml'.format(workflow, target)
-                   for target in UNFILTERED_TARGETS if re.search(r'\b' + target + r'\b', text)]
-    return errors
-
-
-def scanned_text(root, path):
-    """Return the text of a kept file without comment lines and without the filter itself, or None to skip it."""
-    # third-party code and Markdown files read nothing; this script lists the excluded files
-    if path.startswith('tests/thirdparty/') or path.endswith('.md') or path == SELF:
-        return None
-    try:
-        with open(os.path.join(root, path), encoding='utf-8') as f:
-            # a mention in a comment line reads nothing; keep the line numbers
-            text = COMMENT_LINE.sub('', f.read())
-    except (UnicodeDecodeError, OSError):
-        return None
-    if path in WORKFLOWS:
-        # the filter itself lists the excluded files
-        for p in PATHS:
-            text = text.replace("      - '{}'".format(p.replace('{workflow}', path)), '')
-    return text
-
-
-def check_mentions(root, files, excluded, finder):
-    """Return an error for every mention of an excluded file in a kept file that KNOWN_REFERENCES does not list."""
-    errors = []
-    for path in sorted(set(files) - excluded):
-        text = scanned_text(root, path)
-        reported = set()
-        for offset, mention in finder.find(text or ''):
-            if mention not in reported and not known(path, mention):
-                reported.add(mention)
-                line = text.count('\n', 0, offset) + 1
-                errors.append('{}:{}: mentions excluded {}; include it in PATHS if a compiler job reads it, '
-                              'or add it to KNOWN_REFERENCES'.format(path, line, mention))
-    return errors
-
-
-def check(root):
-    files = git_files(root)
-    union = [compile_filter(PATHS, w) for w in WORKFLOWS]
-
-    def is_excluded(path):
-        return not any(included(c, path) for c in union)
-
-    excluded = {f for f in files if is_excluded(f)}
-    errors = check_workflows(root) + check_mentions(root, files, excluded, Mentions(files, excluded, is_excluded))
-    for e in errors:
-        print(e)
-    if not errors:
-        print('path filters OK: {} of {} tracked files skip the compiler workflows'.format(len(excluded), len(files)))
-    return 1 if errors else 0
-
-
-def explain(changed):
-    for workflow in WORKFLOWS:
-        compiled = compile_filter(PATHS, workflow)
-        hits = [f for f in changed if included(compiled, f)]
-        print('{}: {}'.format(workflow, 'runs ({})'.format(', '.join(hits[:3]) + (' ...' if len(hits) > 3 else ''))
-                                       if hits else 'skipped'))
-    return 0
-
-
-def main():
-    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
-    parser.add_argument('--root', default=REPO_ROOT, help='repository root')
-    parser.add_argument('--changed', nargs='+', metavar='FILE', help='show which workflows these changed files run')
-    args = parser.parse_args()
-    return explain(args.changed) if args.changed else check(args.root)
-
-
-if __name__ == '__main__':
-    sys.exit(main())