Read a std::u8string from a JSON string and store it as one in every configuration

In C++20, json(std::u8string) was stored as a JSON string only when
std::filesystem::path support was available (the to_json overload sat
inside the JSON_HAS_FILESYSTEM guard), and there was no matching
from_json, so get<std::u8string>() threw type_error.302 ("type must be
array, but is string"). Without std::filesystem, the same expression
still produced an array of code units (the 3.12.0 behavior).

Move the to_json overload out of the filesystem guard so that a
std::u8string is a string in every configuration, build the value
directly from the code units (this also works for a custom string_t
such as the tests' alt_string), and add a from_json overload that reads
a JSON string. The array of numbers written by 3.12.0 is still accepted;
other types throw type_error.302 ("type must be string or array").
std::u8string_view and char8_t arrays are unchanged.

The tests now run in C++20/23 (the file mentions JSON_HAS_CPP_20/23),
also without std::filesystem (test-conversions2_no_filesystem), and with
ordered_json and a custom string type. The new behavior is documented.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/basic_json.md

@@ -67,7 +67,11 @@ basic_json(basic_json&& other) noexcept;
     - **objects**: [`object_t`](object_t.md) and all kinds of compatible associative containers such as `std::map`,
      `std::unordered_map`, `std::multimap`, and `std::unordered_multimap` with a `key_type` compatible to `string_t`
      and a `value_type` from which a `basic_json` value can be constructed.
-    - **strings**: `string_t`, string literals, and all compatible string containers can be used.
+    - **strings**: `string_t`, string literals, and all compatible string containers can be used. Since version 3.13.0,
+      this includes `#!cpp std::u8string` (C++20): its code units are stored as a JSON string and
+      [`get<std::u8string>()`](get.md) reads it back (in version 3.12.0, a `#!cpp std::u8string` was stored as an array
+      of numbers; `get<std::u8string>()` still accepts such an array). `#!cpp std::u8string_view` and `#!cpp char8_t`
+      arrays are not strings; they are stored as arrays of code units.
 
     !!! note "Rvalue containers are moved from"
 

docs/mkdocs/docs/api/basic_json/get.md

@@ -130,6 +130,12 @@ overload (3).
     `#!cpp json(true).get<double>()` throw. This is the behavior of all 3.x versions;
     [`basic_json_view::get<T>()`](../basic_json_view/get.md) behaves the same.
 
+!!! note "`std::u8string` conversions"
+
+    Since version 3.13.0, `#!cpp get<std::u8string>()` (C++20) reads a JSON string; it also accepts an array of numbers,
+    the form in which version 3.12.0 stored a `#!cpp std::u8string`. Any other JSON type throws
+    [`type_error.302`](../../home/exceptions.md#jsonexceptiontype_error302) ("type must be string or array").
+
 !!! note "`std::optional` conversions"
 
     Prior to version 3.13.0, `#!cpp get<std::optional<T>>()` (and other conversions to `std::optional<T>`) failed to

docs/mkdocs/docs/features/types/template_parameters.md

@@ -425,7 +425,7 @@ using array_t = ArrayType<basic_json, AllocatorType<basic_json>>;
 | Type                                                                 | Reason                                                                                                  |
 |----------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
 | `#!cpp std::wstring`, `#!cpp std::u16string`, `#!cpp std::u32string` | the character type is not one byte wide                                                                 |
-| `#!cpp std::u8string`                                                | one byte wide, but `#!cpp char8_t` is not `#!cpp char`-compatible                                       |
+| `#!cpp std::u8string`                                                | one byte wide, but `#!cpp char8_t` is not `#!cpp char`-compatible (as a *value*, it converts to and from a JSON string) |
 | `absl::Cord`                                                         | no `value_type`, and the storage is not contiguous                                                      |
 | `QString`                                                            | no `append(const char*, size_type)`; its `QChar` is also two bytes wide, though that is never diagnosed |
 

docs/mkdocs/docs/home/releases/unreleased.md

@@ -8,6 +8,16 @@ These changes are part of the `develop` branch, but not of a release yet.
   exception is thrown during the conversion, the source may be partially moved-from, so the strong exception guarantee
   no longer holds for the source of an rvalue conversion (the JSON value itself is unaffected). Lvalues and `const`
   objects are still copied. See [`basic_json`](../../api/basic_json/basic_json.md).
+- In C++20 (library support for `char8_t`, `__cpp_lib_char8_t`), a `#!cpp std::u8string` is a JSON **string**:
+  `json j = std::u8string(u8"abc");` stores `"abc"`, and `#!cpp j.get<std::u8string>()` reads it back. Version 3.12.0
+  stored a `#!cpp std::u8string` as an array of code units (`[97,98,99]`); on the 3.13.0 development branch it was
+  already stored as a string if `std::filesystem` is available, but could not be read back. Now the result no longer
+  depends on whether `std::filesystem::path` is supported, and `#!cpp get<std::u8string>()` accepts both a JSON string
+  and the array of numbers written by 3.12.0. **Consequence:** JSON documents produced from a `#!cpp std::u8string` by
+  3.12.0 (arrays) are still read, but code that checks `is_array()` on such a value, or that reads it as
+  `#!cpp std::vector<char8_t>` / `#!cpp std::vector<std::uint8_t>`, now sees a string; to keep the array form, convert
+  explicitly, e.g., `#!cpp json(std::vector<char8_t>(s.begin(), s.end()))`. `#!cpp std::u8string_view` and
+  `#!cpp char8_t` arrays are unchanged (arrays of code units). See [`basic_json`](../../api/basic_json/basic_json.md).
 
 ## API changes
 

include/nlohmann/detail/conversions/from_json.hpp

@@ -785,6 +785,37 @@ void from_json(const BasicJsonType& j, std::unordered_map<Key, Value, Hash, KeyE
     from_json_pair_array_to_map(j, m);
 }
 
+#if defined(__cpp_lib_char8_t)
+// a std::u8string is read from a JSON string; the array of code units written by
+// versions before 3.13.0 is still accepted
+template < typename BasicJsonType, typename Tr, typename Allocator,
+           enable_if_t < std::is_same<typename json_type_of_t<BasicJsonType>::string_t::value_type, char>::value, int > = 0 >
+inline void from_json(const BasicJsonType& j, std::basic_string<char8_t, Tr, Allocator>& s)
+{
+    if (j.is_string())
+    {
+        const auto& str = string_value(j);
+        const char* const data = str.data();
+        const std::size_t size = str.size();
+        std::basic_string<char8_t, Tr, Allocator> ret(s.get_allocator());
+        ret.reserve(size);
+        for (std::size_t i = 0; i < size; ++i)
+        {
+            ret.push_back(static_cast<char8_t>(static_cast<unsigned char>(data[i])));
+        }
+        s = std::move(ret);
+    }
+    else if (j.is_array())
+    {
+        from_json_array_impl(j, s, priority_tag<3> {});
+    }
+    else
+    {
+        throw_type_must_be("string or array", j);
+    }
+}
+#endif
+
 #if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
 
 // Workaround for MSVC 19.51 (and possibly later): in large cpp files, the compiler may fail to resolve with generic has_from_json (issue #4996)

include/nlohmann/detail/conversions/to_json.hpp

@@ -625,16 +625,20 @@ inline void to_json(BasicJsonType& j, const T& t)
     to_json_tuple_impl(j, t, make_index_sequence<std::tuple_size<T>::value> {});
 }
 
-#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
+// a std::u8string is a JSON string (it was an array of code units before 3.13.0);
+// defined outside the std::filesystem guard so that the result does not depend
+// on whether std::filesystem::path is supported
 #if defined(__cpp_lib_char8_t)
 template<typename BasicJsonType, typename Tr, typename Allocator>
 inline void to_json(BasicJsonType& j, const std::basic_string<char8_t, Tr, Allocator>& s)
 {
-    using OtherAllocator = typename std::allocator_traits<Allocator>::template rebind_alloc<char>;
+    using string_t = typename BasicJsonType::string_t;
-    j = std::basic_string<char, std::char_traits<char>, OtherAllocator>(s.begin(), s.end(), s.get_allocator());
+    // char may alias any object; the code units are copied unchanged
+    j = string_t(reinterpret_cast<const char*>(s.data()), s.size());
 }
 #endif
 
+#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
 // Workaround for MSVC 19.51 (and possibly later): in large cpp files, the compiler may fail to resolve with generic has_to_json (issue #4996)
 template<typename BasicJsonType>
 struct has_to_json<BasicJsonType, std_fs::path, void> : std::true_type {};

include/nlohmann/detail/json_pointer.hpp

@@ -852,6 +852,139 @@ class json_pointer
     }
 
   private:
+    /*!
+    @brief the pointer prefixes of a flattened object that denote arrays
+
+    @ref unflatten creates arrays exactly below the prefixes that have a
+    reference token 0 among their children, and objects everywhere else.
+    Deciding this up front keeps the result independent of the order in which
+    the flattened object is iterated, which is unspecified for some object
+    types.
+
+    A prefix is identified by the text of the key it is the beginning of:
+    escaping is unambiguous, so equal prefixes of valid keys are equal texts.
+    The set refers to the keys, which must outlive it.
+    */
+    class array_prefix_set
+    {
+      public:
+        void insert(const char* data, std::size_t size)
+        {
+            // flattened objects are usually sorted, so most keys that have a
+            // 0 yield the prefix of the previous one
+            if (last_data != nullptr && last_size == size && std::memcmp(last_data, data, size) == 0)
+            {
+                return;
+            }
+            last_data = data;
+            last_size = size;
+
+            if (table.empty())
+            {
+                table.resize(16);
+            }
+            const std::size_t hash = hash_of(data, size);
+            const std::size_t mask = table.size() - 1;
+            std::size_t i = hash & mask;
+            while (table[i].data != nullptr)
+            {
+                if (table[i].hash == hash && table[i].size == size && std::memcmp(table[i].data, data, size) == 0)
+                {
+                    return;
+                }
+                i = (i + 1) & mask;
+            }
+            table[i] = slot(hash, data, size);
+
+            // keep the load factor below 1/2
+            if (++used * 2 > table.size())
+            {
+                grow();
+            }
+        }
+
+        bool empty() const noexcept
+        {
+            return used == 0;
+        }
+
+        bool contains(const char* data, std::size_t size) const noexcept
+        {
+            if (used == 0)
+            {
+                return false;
+            }
+            const std::size_t hash = hash_of(data, size);
+            const std::size_t mask = table.size() - 1;
+            for (std::size_t i = hash & mask; table[i].data != nullptr; i = (i + 1) & mask)
+            {
+                if (table[i].hash == hash && table[i].size == size && std::memcmp(table[i].data, data, size) == 0)
+                {
+                    return true;
+                }
+            }
+            return false;
+        }
+
+      private:
+        struct slot
+        {
+            slot() noexcept = default;
+
+            slot(std::size_t h, const char* d, std::size_t s) noexcept
+                : hash(h), data(d), size(s)
+            {}
+
+            std::size_t hash = 0;
+            const char* data = nullptr; // nullptr: the slot is empty
+            std::size_t size = 0;
+        };
+
+        static std::size_t hash_of(const char* data, std::size_t size) noexcept
+        {
+            std::uint64_t h = 0x9E3779B97F4A7C15ULL ^ size;
+            std::size_t i = 0;
+            for (; i + 8 <= size; i += 8)
+            {
+                std::uint64_t w = 0;
+                std::memcpy(&w, data + i, 8);
+                h = (h ^ w) * 0x100000001B3ULL;
+                h ^= h >> 32u;
+            }
+            std::uint64_t w = 0;
+            if (i < size)
+            {
+                std::memcpy(&w, data + i, size - i);
+            }
+            h = (h ^ w) * 0x9E3779B97F4A7C15ULL;
+            return detail::conditional_static_cast<std::size_t>(h ^ (h >> 29u));
+        }
+
+        void grow()
+        {
+            std::vector<slot> old(table.size() * 2);
+            old.swap(table);
+            const std::size_t mask = table.size() - 1;
+            for (const auto& s : old)
+            {
+                if (s.data != nullptr)
+                {
+                    std::size_t i = s.hash & mask;
+                    while (table[i].data != nullptr)
+                    {
+                        i = (i + 1) & mask;
+                    }
+                    table[i] = s;
+                }
+            }
+        }
+
+        std::vector<slot> table{};
+        std::size_t used = 0;
+        const char* last_data = nullptr;
+        std::size_t last_size = 0;
+    };
+
     /// @brief the length of the longest common prefix of the first @a limit bytes of @a a and @a b
     static std::size_t common_prefix_length(const char* a, const char* b, const std::size_t limit) noexcept
     {
@@ -874,354 +1007,6 @@ class json_pointer
         return i;
     }
 
-    /*!
-    @brief the prefixes of a flattened object that are marked as arrays, for
-           prefixes that are not short
-
-    A prefix is identified by the tokens it consists of: a node of a tree
-    stands for the prefix of its parent followed by a token, and is looked up
-    by its parent and the text of the token. The node of a prefix is found by
-    walking the tokens of the key from the root, so identifying all prefixes
-    of a key together costs one pass over the key, however long the prefixes
-    are, and no prefix is ever compared byte by byte as a whole. The nodes
-    along the key that was walked last are kept, and the walk of the next key
-    starts below the beginning it has in common with that key. The tree refers
-    to the keys, which must outlive it.
-    */
-    class prefix_tree
-    {
-      public:
-        /// @brief set the seed of the hash function, before anything is added
-        void reseed(const std::uint64_t seed_value) noexcept
-        {
-            seed = seed_value;
-        }
-
-        /// @brief mark the first @a prefix_size bytes of the key of @a key_size bytes at @a data
-        void mark(const char* data, std::size_t key_size, std::size_t prefix_size)
-        {
-            nodes[walk(data, key_size, prefix_size)].marked = true;
-            any_marked = true;
-        }
-
-        bool is_marked(const char* data, std::size_t key_size, std::size_t prefix_size)
-        {
-            return any_marked && nodes[walk(data, key_size, prefix_size)].marked;
-        }
-
-        bool empty() const noexcept
-        {
-            return !any_marked;
-        }
-
-        /// @brief the hash of the first @a size bytes at @a data, for the seed @a seed_value
-        static std::size_t hash_bytes(const char* data, std::size_t size, const std::uint64_t seed_value) noexcept
-        {
-            std::uint64_t h = seed_value ^ size;
-            std::size_t i = 0;
-            for (; i + 8 <= size; i += 8)
-            {
-                std::uint64_t w = 0;
-                std::memcpy(&w, data + i, 8);
-                h = (h ^ w) * 0x100000001B3ULL;
-                h ^= h >> 32u;
-            }
-            std::uint64_t w = 0;
-            if (i < size)
-            {
-                std::memcpy(&w, data + i, size - i);
-            }
-            h = (h ^ w) * 0x9E3779B97F4A7C15ULL;
-            return detail::conditional_static_cast<std::size_t>(h ^ (h >> 29u));
-        }
-
-      private:
-        struct node
-        {
-            std::size_t parent = 0;
-            std::size_t hash = 0;
-            const char* token = nullptr;
-            std::size_t token_size = 0;
-            bool marked = false;
-        };
-
-        /// @brief a slot of the hash table; the index of a node, 0 if the slot is empty
-        struct bucket
-        {
-            std::size_t index = 0;
-        };
-
-        /// @brief the node after a token of the key that was walked last, and where the token ends
-        struct step
-        {
-            std::size_t node = 0;
-            std::size_t end = 0;
-        };
-
-        /// @brief the index of the node of the prefix that ends at the slash (or the end) at @a prefix_size
-        std::size_t walk(const char* data, std::size_t key_size, std::size_t prefix_size)
-        {
-            if (nodes.empty())
-            {
-                nodes.emplace_back();
-                table.resize(16);
-                chain.emplace_back();
-            }
-
-            if (data != last_key)
-            {
-                // keep the nodes within the beginning that this key has in common
-                // with the key they were found for; a prefix that ends inside a
-                // token of this key is not one of its prefixes
-                const std::size_t same = common_prefix_length(data, last_key, (std::min)(key_size, last_key_size));
-                std::size_t keep = chain.size();
-                while (keep > 1 && (chain[keep - 1].end > same || (chain[keep - 1].end == same && same < key_size && data[same] != '/')))
-                {
-                    --keep;
-                }
-                chain.resize(keep);
-                last_key = data;
-                last_key_size = key_size;
-            }
-
-            // walk the tokens that are not walked yet
-            while (chain.back().end < prefix_size && chain.back().end < key_size)
-            {
-                const std::size_t start = chain.back().end + 1;
-                const void* const slash = std::memchr(data + start, '/', key_size - start);
-                const std::size_t end = slash == nullptr ? key_size : static_cast<std::size_t>(static_cast<const char*>(slash) - data);
-                step next;
-                next.node = child(chain.back().node, data + start, end - start);
-                next.end = end;
-                chain.push_back(next);
-            }
-
-            if (chain.back().end == prefix_size)
-            {
-                return chain.back().node;
-            }
-            // an earlier call walked further
-            const auto it = std::lower_bound(chain.begin(), chain.end(), prefix_size, [](const step & s, std::size_t size)
-            {
-                return s.end < size;
-            });
-            return it->node;
-        }
-
-        std::size_t slot_of(std::size_t parent, std::size_t hash) const noexcept
-        {
-            std::uint64_t x = static_cast<std::uint64_t>(hash) ^ (static_cast<std::uint64_t>(parent) * 0x9E3779B97F4A7C15ULL);
-            x ^= x >> 32u;
-            x *= 0xD6E8FEB86659FD93ULL;
-            x ^= x >> 32u;
-            return static_cast<std::size_t>(x) & (table.size() - 1);
-        }
-
-        /// @brief the node of the prefix of the node @a parent followed by the token, which is added if it is not there
-        std::size_t child(std::size_t parent, const char* token, std::size_t token_size)
-        {
-            const std::size_t hash = hash_bytes(token, token_size, seed);
-            std::size_t i = slot_of(parent, hash);
-            while (table[i].index != 0)
-            {
-                const node& n = nodes[table[i].index];
-                if (n.parent == parent && n.hash == hash && n.token_size == token_size && std::memcmp(n.token, token, token_size) == 0)
-                {
-                    return table[i].index;
-                }
-                i = (i + 1) & (table.size() - 1);
-            }
-
-            node created;
-            created.parent = parent;
-            created.hash = hash;
-            created.token = token;
-            created.token_size = token_size;
-            nodes.push_back(created);
-            table[i].index = nodes.size() - 1;
-
-            // keep the load factor below 1/2
-            if (nodes.size() * 2 > table.size())
-            {
-                std::vector<bucket> old(table.size() * 2);
-                old.swap(table);
-                for (std::size_t k = 1; k < nodes.size(); ++k)
-                {
-                    std::size_t j = slot_of(nodes[k].parent, nodes[k].hash);
-                    while (table[j].index != 0)
-                    {
-                        j = (j + 1) & (table.size() - 1);
-                    }
-                    table[j].index = k;
-                }
-            }
-            return nodes.size() - 1;
-        }
-
-        std::uint64_t seed = 0;
-        /// @brief the nodes; the first is the root, the empty prefix
-        std::vector<node> nodes{};
-        /// @brief the nodes (not the root) in a hash table
-        std::vector<bucket> table{};
-        /// @brief the root, and the node after each token of the key that was walked last
-        std::vector<step> chain{};
-        const char* last_key = nullptr;
-        std::size_t last_key_size = 0;
-        bool any_marked = false;
-    };
-
-    /*!
-    @brief the pointer prefixes of a flattened object that denote arrays
-
-    @ref unflatten creates arrays exactly below the prefixes that have a
-    reference token 0 among their children, and objects everywhere else.
-    Deciding this up front keeps the result independent of the order in which
-    the flattened object is iterated, which is unspecified for some object
-    types.
-
-    A prefix is identified by the text of the key it is the beginning of:
-    escaping is unambiguous, so equal prefixes of valid keys are equal texts.
-    The set refers to the keys, which must outlive it. Short prefixes are
-    kept in a hash table and compared as texts, which costs at most the length
-    of the bound. Longer prefixes are kept in a @ref prefix_tree, because
-    hashing and comparing every one of the prefixes of a key with many tokens
-    from its beginning would take quadratic time.
-
-    The hash function depends on a seed that is derived from the address of a
-    local variable, which address space layout randomization changes between
-    processes, so keys cannot be crafted to collide in advance; the result
-    does not depend on the seed. Where the addresses are not randomized, or
-    when the seed is known, keys can collide: k short prefixes then cost
-    k^2 / 2 comparisons of fewer than 64 bytes each. The long prefixes are not
-    hashed as a whole.
-    */
-    class array_prefix_set
-    {
-      public:
-        /// @brief choose the seed of the hash function, before anything is added: it is derived from the address that is passed
-        void reseed(const void* address) noexcept
-        {
-            // NOLINTNEXTLINE(cppcoreguidelines-pro-type-reinterpret-cast): the address is only a source of entropy
-            std::uint64_t x = static_cast<std::uint64_t>(reinterpret_cast<std::uintptr_t>(address)) ^ 0x9E3779B97F4A7C15ULL;
-            x *= 0xBF58476D1CE4E5B9ULL;
-            seed = x ^ (x >> 31u);
-            tree.reseed(seed);
-        }
-
-        /// @brief add the first @a prefix_size bytes of the key of @a key_size bytes at @a data
-        void insert(const char* data, std::size_t key_size, std::size_t prefix_size)
-        {
-            if (prefix_size >= short_limit)
-            {
-                tree.mark(data, key_size, prefix_size);
-                return;
-            }
-
-            // flattened objects are usually sorted, so most keys that have a
-            // 0 yield the prefix of the previous one
-            if (last_data != nullptr && last_size == prefix_size && std::memcmp(last_data, data, prefix_size) == 0)
-            {
-                return;
-            }
-            last_data = data;
-            last_size = prefix_size;
-
-            const std::size_t hash = prefix_tree::hash_bytes(data, prefix_size, seed);
-            if (table.empty())
-            {
-                table.resize(16);
-            }
-            const std::size_t mask = table.size() - 1;
-            std::size_t i = hash & mask;
-            while (table[i].data != nullptr)
-            {
-                if (table[i].hash == hash && table[i].size == prefix_size && std::memcmp(table[i].data, data, prefix_size) == 0)
-                {
-                    return;
-                }
-                i = (i + 1) & mask;
-            }
-            table[i] = slot(hash, data, prefix_size);
-
-            // keep the load factor below 1/2
-            if (++used * 2 > table.size())
-            {
-                grow();
-            }
-        }
-
-        bool empty() const noexcept
-        {
-            return used == 0 && tree.empty();
-        }
-
-        /// @brief whether the first @a prefix_size bytes of the key of @a key_size bytes at @a data were added
-        bool contains(const char* data, std::size_t key_size, std::size_t prefix_size)
-        {
-            if (prefix_size >= short_limit)
-            {
-                return tree.is_marked(data, key_size, prefix_size);
-            }
-            if (used == 0)
-            {
-                return false;
-            }
-            const std::size_t hash = prefix_tree::hash_bytes(data, prefix_size, seed);
-            const std::size_t mask = table.size() - 1;
-            for (std::size_t i = hash & mask; table[i].data != nullptr; i = (i + 1) & mask)
-            {
-                if (table[i].hash == hash && table[i].size == prefix_size && std::memcmp(table[i].data, data, prefix_size) == 0)
-                {
-                    return true;
-                }
-            }
-            return false;
-        }
-
-      private:
-        /// @brief prefixes of fewer bytes are kept in the hash table
-        static constexpr std::size_t short_limit = 64;
-
-        struct slot
-        {
-            slot() noexcept = default;
-
-            slot(std::size_t h, const char* d, std::size_t s) noexcept
-                : hash(h), data(d), size(s)
-            {}
-
-            std::size_t hash = 0;
-            const char* data = nullptr; // nullptr: the slot is empty
-            std::size_t size = 0;
-        };
-
-        void grow()
-        {
-            std::vector<slot> old(table.size() * 2);
-            old.swap(table);
-            const std::size_t mask = table.size() - 1;
-            for (const auto& s : old)
-            {
-                if (s.data != nullptr)
-                {
-                    std::size_t i = s.hash & mask;
-                    while (table[i].data != nullptr)
-                    {
-                        i = (i + 1) & mask;
-                    }
-                    table[i] = s;
-                }
-            }
-        }
-
-        std::uint64_t seed = 0;
-        prefix_tree tree{};
-        std::vector<slot> table{};
-        std::size_t used = 0;
-        const char* last_data = nullptr;
-        std::size_t last_size = 0;
-    };
-
     /*!
     @brief check that @a key is a valid JSON pointer, and add the prefixes of
            it that denote arrays to @a arrays
@@ -1267,7 +1052,7 @@ class json_pointer
                 // a token 0 makes the pointer before it an array
                 if (i - begin == 1 && data[begin] == '0')
                 {
-                    arrays.insert(data, size, begin - 1);
+                    arrays.insert(data, begin - 1);
                 }
                 begin = i + 1;
             }
@@ -2277,7 +2062,6 @@ class json_pointer
         // token 0 among their children; the values below them are arrays,
         // all others are objects (see array_prefix_set)
         array_prefix_set arrays;
-        arrays.reseed(&arrays);
         const string_t* previous_key = nullptr;
         for (const auto& element : *value.m_data.m_value.object)
         {
@@ -2369,7 +2153,7 @@ class json_pointer
                     case detail::value_t::null:
                     {
                         // a first child 0 makes the value an array without a look-up
-                        if ((token_size == 1 && data[pos + 1] == '0') || arrays.contains(data, key.size(), pos))
+                        if ((token_size == 1 && data[pos + 1] == '0') || arrays.contains(data, pos))
                         {
                             // some reference token below this position is 0, so the
                             // value is an array

single_include/nlohmann/json.hpp

@@ -6928,6 +6928,37 @@ void from_json(const BasicJsonType& j, std::unordered_map<Key, Value, Hash, KeyE
     from_json_pair_array_to_map(j, m);
 }
 
+#if defined(__cpp_lib_char8_t)
+// a std::u8string is read from a JSON string; the array of code units written by
+// versions before 3.13.0 is still accepted
+template < typename BasicJsonType, typename Tr, typename Allocator,
+           enable_if_t < std::is_same<typename json_type_of_t<BasicJsonType>::string_t::value_type, char>::value, int > = 0 >
+inline void from_json(const BasicJsonType& j, std::basic_string<char8_t, Tr, Allocator>& s)
+{
+    if (j.is_string())
+    {
+        const auto& str = string_value(j);
+        const char* const data = str.data();
+        const std::size_t size = str.size();
+        std::basic_string<char8_t, Tr, Allocator> ret(s.get_allocator());
+        ret.reserve(size);
+        for (std::size_t i = 0; i < size; ++i)
+        {
+            ret.push_back(static_cast<char8_t>(static_cast<unsigned char>(data[i])));
+        }
+        s = std::move(ret);
+    }
+    else if (j.is_array())
+    {
+        from_json_array_impl(j, s, priority_tag<3> {});
+    }
+    else
+    {
+        throw_type_must_be("string or array", j);
+    }
+}
+#endif
+
 #if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
 
 // Workaround for MSVC 19.51 (and possibly later): in large cpp files, the compiler may fail to resolve with generic has_from_json (issue #4996)
@@ -8329,16 +8360,20 @@ inline void to_json(BasicJsonType& j, const T& t)
     to_json_tuple_impl(j, t, make_index_sequence<std::tuple_size<T>::value> {});
 }
 
-#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
+// a std::u8string is a JSON string (it was an array of code units before 3.13.0);
+// defined outside the std::filesystem guard so that the result does not depend
+// on whether std::filesystem::path is supported
 #if defined(__cpp_lib_char8_t)
 template<typename BasicJsonType, typename Tr, typename Allocator>
 inline void to_json(BasicJsonType& j, const std::basic_string<char8_t, Tr, Allocator>& s)
 {
-    using OtherAllocator = typename std::allocator_traits<Allocator>::template rebind_alloc<char>;
+    using string_t = typename BasicJsonType::string_t;
-    j = std::basic_string<char, std::char_traits<char>, OtherAllocator>(s.begin(), s.end(), s.get_allocator());
+    // char may alias any object; the code units are copied unchanged
+    j = string_t(reinterpret_cast<const char*>(s.data()), s.size());
 }
 #endif
 
+#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
 // Workaround for MSVC 19.51 (and possibly later): in large cpp files, the compiler may fail to resolve with generic has_to_json (issue #4996)
 template<typename BasicJsonType>
 struct has_to_json<BasicJsonType, std_fs::path, void> : std::true_type {};
@@ -27027,6 +27062,139 @@ class json_pointer
     }
 
   private:
+    /*!
+    @brief the pointer prefixes of a flattened object that denote arrays
+
+    @ref unflatten creates arrays exactly below the prefixes that have a
+    reference token 0 among their children, and objects everywhere else.
+    Deciding this up front keeps the result independent of the order in which
+    the flattened object is iterated, which is unspecified for some object
+    types.
+
+    A prefix is identified by the text of the key it is the beginning of:
+    escaping is unambiguous, so equal prefixes of valid keys are equal texts.
+    The set refers to the keys, which must outlive it.
+    */
+    class array_prefix_set
+    {
+      public:
+        void insert(const char* data, std::size_t size)
+        {
+            // flattened objects are usually sorted, so most keys that have a
+            // 0 yield the prefix of the previous one
+            if (last_data != nullptr && last_size == size && std::memcmp(last_data, data, size) == 0)
+            {
+                return;
+            }
+            last_data = data;
+            last_size = size;
+
+            if (table.empty())
+            {
+                table.resize(16);
+            }
+            const std::size_t hash = hash_of(data, size);
+            const std::size_t mask = table.size() - 1;
+            std::size_t i = hash & mask;
+            while (table[i].data != nullptr)
+            {
+                if (table[i].hash == hash && table[i].size == size && std::memcmp(table[i].data, data, size) == 0)
+                {
+                    return;
+                }
+                i = (i + 1) & mask;
+            }
+            table[i] = slot(hash, data, size);
+
+            // keep the load factor below 1/2
+            if (++used * 2 > table.size())
+            {
+                grow();
+            }
+        }
+
+        bool empty() const noexcept
+        {
+            return used == 0;
+        }
+
+        bool contains(const char* data, std::size_t size) const noexcept
+        {
+            if (used == 0)
+            {
+                return false;
+            }
+            const std::size_t hash = hash_of(data, size);
+            const std::size_t mask = table.size() - 1;
+            for (std::size_t i = hash & mask; table[i].data != nullptr; i = (i + 1) & mask)
+            {
+                if (table[i].hash == hash && table[i].size == size && std::memcmp(table[i].data, data, size) == 0)
+                {
+                    return true;
+                }
+            }
+            return false;
+        }
+
+      private:
+        struct slot
+        {
+            slot() noexcept = default;
+
+            slot(std::size_t h, const char* d, std::size_t s) noexcept
+                : hash(h), data(d), size(s)
+            {}
+
+            std::size_t hash = 0;
+            const char* data = nullptr; // nullptr: the slot is empty
+            std::size_t size = 0;
+        };
+
+        static std::size_t hash_of(const char* data, std::size_t size) noexcept
+        {
+            std::uint64_t h = 0x9E3779B97F4A7C15ULL ^ size;
+            std::size_t i = 0;
+            for (; i + 8 <= size; i += 8)
+            {
+                std::uint64_t w = 0;
+                std::memcpy(&w, data + i, 8);
+                h = (h ^ w) * 0x100000001B3ULL;
+                h ^= h >> 32u;
+            }
+            std::uint64_t w = 0;
+            if (i < size)
+            {
+                std::memcpy(&w, data + i, size - i);
+            }
+            h = (h ^ w) * 0x9E3779B97F4A7C15ULL;
+            return detail::conditional_static_cast<std::size_t>(h ^ (h >> 29u));
+        }
+
+        void grow()
+        {
+            std::vector<slot> old(table.size() * 2);
+            old.swap(table);
+            const std::size_t mask = table.size() - 1;
+            for (const auto& s : old)
+            {
+                if (s.data != nullptr)
+                {
+                    std::size_t i = s.hash & mask;
+                    while (table[i].data != nullptr)
+                    {
+                        i = (i + 1) & mask;
+                    }
+                    table[i] = s;
+                }
+            }
+        }
+
+        std::vector<slot> table{};
+        std::size_t used = 0;
+        const char* last_data = nullptr;
+        std::size_t last_size = 0;
+    };
+
     /// @brief the length of the longest common prefix of the first @a limit bytes of @a a and @a b
     static std::size_t common_prefix_length(const char* a, const char* b, const std::size_t limit) noexcept
     {
@@ -27049,354 +27217,6 @@ class json_pointer
         return i;
     }
 
-    /*!
-    @brief the prefixes of a flattened object that are marked as arrays, for
-           prefixes that are not short
-
-    A prefix is identified by the tokens it consists of: a node of a tree
-    stands for the prefix of its parent followed by a token, and is looked up
-    by its parent and the text of the token. The node of a prefix is found by
-    walking the tokens of the key from the root, so identifying all prefixes
-    of a key together costs one pass over the key, however long the prefixes
-    are, and no prefix is ever compared byte by byte as a whole. The nodes
-    along the key that was walked last are kept, and the walk of the next key
-    starts below the beginning it has in common with that key. The tree refers
-    to the keys, which must outlive it.
-    */
-    class prefix_tree
-    {
-      public:
-        /// @brief set the seed of the hash function, before anything is added
-        void reseed(const std::uint64_t seed_value) noexcept
-        {
-            seed = seed_value;
-        }
-
-        /// @brief mark the first @a prefix_size bytes of the key of @a key_size bytes at @a data
-        void mark(const char* data, std::size_t key_size, std::size_t prefix_size)
-        {
-            nodes[walk(data, key_size, prefix_size)].marked = true;
-            any_marked = true;
-        }
-
-        bool is_marked(const char* data, std::size_t key_size, std::size_t prefix_size)
-        {
-            return any_marked && nodes[walk(data, key_size, prefix_size)].marked;
-        }
-
-        bool empty() const noexcept
-        {
-            return !any_marked;
-        }
-
-        /// @brief the hash of the first @a size bytes at @a data, for the seed @a seed_value
-        static std::size_t hash_bytes(const char* data, std::size_t size, const std::uint64_t seed_value) noexcept
-        {
-            std::uint64_t h = seed_value ^ size;
-            std::size_t i = 0;
-            for (; i + 8 <= size; i += 8)
-            {
-                std::uint64_t w = 0;
-                std::memcpy(&w, data + i, 8);
-                h = (h ^ w) * 0x100000001B3ULL;
-                h ^= h >> 32u;
-            }
-            std::uint64_t w = 0;
-            if (i < size)
-            {
-                std::memcpy(&w, data + i, size - i);
-            }
-            h = (h ^ w) * 0x9E3779B97F4A7C15ULL;
-            return detail::conditional_static_cast<std::size_t>(h ^ (h >> 29u));
-        }
-
-      private:
-        struct node
-        {
-            std::size_t parent = 0;
-            std::size_t hash = 0;
-            const char* token = nullptr;
-            std::size_t token_size = 0;
-            bool marked = false;
-        };
-
-        /// @brief a slot of the hash table; the index of a node, 0 if the slot is empty
-        struct bucket
-        {
-            std::size_t index = 0;
-        };
-
-        /// @brief the node after a token of the key that was walked last, and where the token ends
-        struct step
-        {
-            std::size_t node = 0;
-            std::size_t end = 0;
-        };
-
-        /// @brief the index of the node of the prefix that ends at the slash (or the end) at @a prefix_size
-        std::size_t walk(const char* data, std::size_t key_size, std::size_t prefix_size)
-        {
-            if (nodes.empty())
-            {
-                nodes.emplace_back();
-                table.resize(16);
-                chain.emplace_back();
-            }
-
-            if (data != last_key)
-            {
-                // keep the nodes within the beginning that this key has in common
-                // with the key they were found for; a prefix that ends inside a
-                // token of this key is not one of its prefixes
-                const std::size_t same = common_prefix_length(data, last_key, (std::min)(key_size, last_key_size));
-                std::size_t keep = chain.size();
-                while (keep > 1 && (chain[keep - 1].end > same || (chain[keep - 1].end == same && same < key_size && data[same] != '/')))
-                {
-                    --keep;
-                }
-                chain.resize(keep);
-                last_key = data;
-                last_key_size = key_size;
-            }
-
-            // walk the tokens that are not walked yet
-            while (chain.back().end < prefix_size && chain.back().end < key_size)
-            {
-                const std::size_t start = chain.back().end + 1;
-                const void* const slash = std::memchr(data + start, '/', key_size - start);
-                const std::size_t end = slash == nullptr ? key_size : static_cast<std::size_t>(static_cast<const char*>(slash) - data);
-                step next;
-                next.node = child(chain.back().node, data + start, end - start);
-                next.end = end;
-                chain.push_back(next);
-            }
-
-            if (chain.back().end == prefix_size)
-            {
-                return chain.back().node;
-            }
-            // an earlier call walked further
-            const auto it = std::lower_bound(chain.begin(), chain.end(), prefix_size, [](const step & s, std::size_t size)
-            {
-                return s.end < size;
-            });
-            return it->node;
-        }
-
-        std::size_t slot_of(std::size_t parent, std::size_t hash) const noexcept
-        {
-            std::uint64_t x = static_cast<std::uint64_t>(hash) ^ (static_cast<std::uint64_t>(parent) * 0x9E3779B97F4A7C15ULL);
-            x ^= x >> 32u;
-            x *= 0xD6E8FEB86659FD93ULL;
-            x ^= x >> 32u;
-            return static_cast<std::size_t>(x) & (table.size() - 1);
-        }
-
-        /// @brief the node of the prefix of the node @a parent followed by the token, which is added if it is not there
-        std::size_t child(std::size_t parent, const char* token, std::size_t token_size)
-        {
-            const std::size_t hash = hash_bytes(token, token_size, seed);
-            std::size_t i = slot_of(parent, hash);
-            while (table[i].index != 0)
-            {
-                const node& n = nodes[table[i].index];
-                if (n.parent == parent && n.hash == hash && n.token_size == token_size && std::memcmp(n.token, token, token_size) == 0)
-                {
-                    return table[i].index;
-                }
-                i = (i + 1) & (table.size() - 1);
-            }
-
-            node created;
-            created.parent = parent;
-            created.hash = hash;
-            created.token = token;
-            created.token_size = token_size;
-            nodes.push_back(created);
-            table[i].index = nodes.size() - 1;
-
-            // keep the load factor below 1/2
-            if (nodes.size() * 2 > table.size())
-            {
-                std::vector<bucket> old(table.size() * 2);
-                old.swap(table);
-                for (std::size_t k = 1; k < nodes.size(); ++k)
-                {
-                    std::size_t j = slot_of(nodes[k].parent, nodes[k].hash);
-                    while (table[j].index != 0)
-                    {
-                        j = (j + 1) & (table.size() - 1);
-                    }
-                    table[j].index = k;
-                }
-            }
-            return nodes.size() - 1;
-        }
-
-        std::uint64_t seed = 0;
-        /// @brief the nodes; the first is the root, the empty prefix
-        std::vector<node> nodes{};
-        /// @brief the nodes (not the root) in a hash table
-        std::vector<bucket> table{};
-        /// @brief the root, and the node after each token of the key that was walked last
-        std::vector<step> chain{};
-        const char* last_key = nullptr;
-        std::size_t last_key_size = 0;
-        bool any_marked = false;
-    };
-
-    /*!
-    @brief the pointer prefixes of a flattened object that denote arrays
-
-    @ref unflatten creates arrays exactly below the prefixes that have a
-    reference token 0 among their children, and objects everywhere else.
-    Deciding this up front keeps the result independent of the order in which
-    the flattened object is iterated, which is unspecified for some object
-    types.
-
-    A prefix is identified by the text of the key it is the beginning of:
-    escaping is unambiguous, so equal prefixes of valid keys are equal texts.
-    The set refers to the keys, which must outlive it. Short prefixes are
-    kept in a hash table and compared as texts, which costs at most the length
-    of the bound. Longer prefixes are kept in a @ref prefix_tree, because
-    hashing and comparing every one of the prefixes of a key with many tokens
-    from its beginning would take quadratic time.
-
-    The hash function depends on a seed that is derived from the address of a
-    local variable, which address space layout randomization changes between
-    processes, so keys cannot be crafted to collide in advance; the result
-    does not depend on the seed. Where the addresses are not randomized, or
-    when the seed is known, keys can collide: k short prefixes then cost
-    k^2 / 2 comparisons of fewer than 64 bytes each. The long prefixes are not
-    hashed as a whole.
-    */
-    class array_prefix_set
-    {
-      public:
-        /// @brief choose the seed of the hash function, before anything is added: it is derived from the address that is passed
-        void reseed(const void* address) noexcept
-        {
-            // NOLINTNEXTLINE(cppcoreguidelines-pro-type-reinterpret-cast): the address is only a source of entropy
-            std::uint64_t x = static_cast<std::uint64_t>(reinterpret_cast<std::uintptr_t>(address)) ^ 0x9E3779B97F4A7C15ULL;
-            x *= 0xBF58476D1CE4E5B9ULL;
-            seed = x ^ (x >> 31u);
-            tree.reseed(seed);
-        }
-
-        /// @brief add the first @a prefix_size bytes of the key of @a key_size bytes at @a data
-        void insert(const char* data, std::size_t key_size, std::size_t prefix_size)
-        {
-            if (prefix_size >= short_limit)
-            {
-                tree.mark(data, key_size, prefix_size);
-                return;
-            }
-
-            // flattened objects are usually sorted, so most keys that have a
-            // 0 yield the prefix of the previous one
-            if (last_data != nullptr && last_size == prefix_size && std::memcmp(last_data, data, prefix_size) == 0)
-            {
-                return;
-            }
-            last_data = data;
-            last_size = prefix_size;
-
-            const std::size_t hash = prefix_tree::hash_bytes(data, prefix_size, seed);
-            if (table.empty())
-            {
-                table.resize(16);
-            }
-            const std::size_t mask = table.size() - 1;
-            std::size_t i = hash & mask;
-            while (table[i].data != nullptr)
-            {
-                if (table[i].hash == hash && table[i].size == prefix_size && std::memcmp(table[i].data, data, prefix_size) == 0)
-                {
-                    return;
-                }
-                i = (i + 1) & mask;
-            }
-            table[i] = slot(hash, data, prefix_size);
-
-            // keep the load factor below 1/2
-            if (++used * 2 > table.size())
-            {
-                grow();
-            }
-        }
-
-        bool empty() const noexcept
-        {
-            return used == 0 && tree.empty();
-        }
-
-        /// @brief whether the first @a prefix_size bytes of the key of @a key_size bytes at @a data were added
-        bool contains(const char* data, std::size_t key_size, std::size_t prefix_size)
-        {
-            if (prefix_size >= short_limit)
-            {
-                return tree.is_marked(data, key_size, prefix_size);
-            }
-            if (used == 0)
-            {
-                return false;
-            }
-            const std::size_t hash = prefix_tree::hash_bytes(data, prefix_size, seed);
-            const std::size_t mask = table.size() - 1;
-            for (std::size_t i = hash & mask; table[i].data != nullptr; i = (i + 1) & mask)
-            {
-                if (table[i].hash == hash && table[i].size == prefix_size && std::memcmp(table[i].data, data, prefix_size) == 0)
-                {
-                    return true;
-                }
-            }
-            return false;
-        }
-
-      private:
-        /// @brief prefixes of fewer bytes are kept in the hash table
-        static constexpr std::size_t short_limit = 64;
-
-        struct slot
-        {
-            slot() noexcept = default;
-
-            slot(std::size_t h, const char* d, std::size_t s) noexcept
-                : hash(h), data(d), size(s)
-            {}
-
-            std::size_t hash = 0;
-            const char* data = nullptr; // nullptr: the slot is empty
-            std::size_t size = 0;
-        };
-
-        void grow()
-        {
-            std::vector<slot> old(table.size() * 2);
-            old.swap(table);
-            const std::size_t mask = table.size() - 1;
-            for (const auto& s : old)
-            {
-                if (s.data != nullptr)
-                {
-                    std::size_t i = s.hash & mask;
-                    while (table[i].data != nullptr)
-                    {
-                        i = (i + 1) & mask;
-                    }
-                    table[i] = s;
-                }
-            }
-        }
-
-        std::uint64_t seed = 0;
-        prefix_tree tree{};
-        std::vector<slot> table{};
-        std::size_t used = 0;
-        const char* last_data = nullptr;
-        std::size_t last_size = 0;
-    };
-
     /*!
     @brief check that @a key is a valid JSON pointer, and add the prefixes of
            it that denote arrays to @a arrays
@@ -27442,7 +27262,7 @@ class json_pointer
                 // a token 0 makes the pointer before it an array
                 if (i - begin == 1 && data[begin] == '0')
                 {
-                    arrays.insert(data, size, begin - 1);
+                    arrays.insert(data, begin - 1);
                 }
                 begin = i + 1;
             }
@@ -28452,7 +28272,6 @@ class json_pointer
         // token 0 among their children; the values below them are arrays,
         // all others are objects (see array_prefix_set)
         array_prefix_set arrays;
-        arrays.reseed(&arrays);
         const string_t* previous_key = nullptr;
         for (const auto& element : *value.m_data.m_value.object)
         {
@@ -28544,7 +28363,7 @@ class json_pointer
                     case detail::value_t::null:
                     {
                         // a first child 0 makes the value an array without a look-up
-                        if ((token_size == 1 && data[pos + 1] == '0') || arrays.contains(data, key.size(), pos))
+                        if ((token_size == 1 && data[pos + 1] == '0') || arrays.contains(data, pos))
                         {
                             // some reference token below this position is 0, so the
                             // value is an array

tests/CMakeLists.txt

@@ -364,6 +364,16 @@ json_test_add_test_for(src/unit-class_parser.cpp
     MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force}
 )
 
+# test the conversions again without std::filesystem::path support: a
+# std::u8string must be a JSON string in this configuration, too
+json_test_set_test_options(test-conversions2_no_filesystem
+    COMPILE_DEFINITIONS JSON_HAS_FILESYSTEM=0 JSON_HAS_EXPERIMENTAL_FILESYSTEM=0
+)
+json_test_add_test_for(src/unit-conversions2.cpp
+    NAME test-conversions2_no_filesystem
+    MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force}
+)
+
 # test diagnostic positions again without regular diagnostics (JSON pointer paths)
 json_test_set_test_options(test-diagnostic-positions_only
     COMPILE_DEFINITIONS JSON_DIAGNOSTICS=0

tests/src/unit-alt-string.cpp

@@ -536,4 +536,20 @@ TEST_CASE("alternative string type with json_view")
     }
 }
 
+// a std::u8string is a JSON string, whatever the string type of the JSON value
+// JSON_HAS_CPP_20 JSON_HAS_CPP_23 (do not remove; see note at top of file)
+#if defined(__cpp_lib_char8_t)
+TEST_CASE("alternative string type and std::u8string")
+{
+    const std::u8string s = u8"abc";
+    const alt_json j = s;
+    CHECK(j.is_string());
+    CHECK(j.get<alt_string>() == alt_string("abc"));
+    CHECK(j.get<std::u8string>() == s);
+
+    // the array of code units written by versions before 3.13.0
+    CHECK(alt_json::array({0x61, 0x62, 0x63}).get<std::u8string>() == s);
+}
+#endif
+
 DOCTEST_CLANG_SUPPRESS_WARNING_POP

tests/src/unit-conversions2.cpp

@@ -27,10 +27,12 @@ using nlohmann::json;
 #include <deque>
 #include <forward_list>
 #include <list>
+#include <map>
 #include <set>
 #include <unordered_map>
 #include <unordered_set>
 #include <valarray>
+#include <vector>
 
 // NLOHMANN_JSON_SERIALIZE_ENUM uses a static std::pair
 DOCTEST_CLANG_SUPPRESS_WARNING_PUSH
@@ -707,38 +709,114 @@ TEST_CASE("std::filesystem::path")
 }
 #endif
 
-// the ADL to_json overload for std::u8string only exists under the same guard
+// a std::u8string is a JSON string since 3.13.0 (it was an array of code units
-// as std::filesystem::path support (it is otherwise only reached indirectly,
+// before); independent of whether std::filesystem::path is supported, see the
-// via std::filesystem::path::u8string()) -- mirror both #if conditions from
+// test-conversions2_no_filesystem target
-// include/nlohmann/detail/conversions/to_json.hpp exactly
+// JSON_HAS_CPP_20 JSON_HAS_CPP_23 (do not remove; see note at top of file)
-#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
 #if defined(__cpp_lib_char8_t)
 TEST_CASE("std::u8string")
 {
+    // use \u universal-character-names (rather than raw \x byte escapes
+    // or literal non-ASCII source bytes) to compose the multi-byte UTF-8
+    // encoding -- MSVC treats \x escapes used that way inside a u8
+    // literal as a nonstandard extension (warning C5321), which some of
+    // our CI configs promote to an error; \u is portable and produces
+    // the exact same encoded bytes without depending on the source
+    // file's encoding
+    const std::u8string utf8 = u8"P\u011B\u0161ina";
+
     SECTION("ascii")
     {
         const std::u8string s = u8"Path";
         json const j = s;
 
+        CHECK(j.is_string());
         CHECK(j.template get<std::string>() == "Path");
     }
 
     SECTION("utf-8")
     {
-        // use \u universal-character-names (rather than raw \x byte escapes
+        json const j = utf8;
-        // or literal non-ASCII source bytes) to compose the multi-byte UTF-8
-        // encoding -- MSVC treats \x escapes used that way inside a u8
-        // literal as a nonstandard extension (warning C5321), which some of
-        // our CI configs promote to an error; \u is portable and produces
-        // the exact same encoded bytes without depending on the source
-        // file's encoding
-        const std::u8string s = u8"P\u011B\u0161ina";
-        json const j = s;
 
+        CHECK(j.is_string());
         CHECK(j.template get<std::string>() == "P\xc4\x9b\xc5\xa1ina");
     }
-}
+
+    SECTION("round trip")
+    {
+        json const j = utf8;
+        CHECK(j.template get<std::u8string>() == utf8);
+
+        std::u8string back;
+        j.get_to(back);
+        CHECK(back == utf8);
+
+        // from a string
+        json const j_string = "P\xc4\x9b\xc5\xa1ina";
+        CHECK(j_string.template get<std::u8string>() == utf8);
+
+        // empty, and with an embedded NUL
+        CHECK(json(std::u8string()).template get<std::u8string>().empty());
+        const std::u8string with_nul(u8"a\0b", 3);
+        CHECK(json(with_nul).template get<std::u8string>() == with_nul);
+
+        // the lvalue and rvalue paths, and in containers
+        std::u8string lvalue = utf8;
+        CHECK(json(lvalue).template get<std::u8string>() == utf8);
+        CHECK(json(std::move(lvalue)).template get<std::u8string>() == utf8);
+        const std::vector<std::u8string> vec = {utf8, u8"x"};
+        json const j_vec = vec;
+        CHECK(j_vec.is_array());
+        CHECK(j_vec[0].is_string());
+        CHECK(j_vec.template get<std::vector<std::u8string>>() == vec);
+        const std::map<std::string, std::u8string> map = {{"k", utf8}};
+        CHECK(json(map).template get<std::map<std::string, std::u8string>>() == map);
+    }
+
+    SECTION("array of code units (written by versions before 3.13.0)")
+    {
+        json const j = json::array({0x50, 0xC4, 0x9B, 0xC5, 0xA1, 0x69, 0x6E, 0x61});
+        CHECK(j.template get<std::u8string>() == utf8);
+        CHECK(json::array().template get<std::u8string>().empty());
+        CHECK(json::parse("[97, 98]").template get<std::u8string>() == u8"ab");
+
+        std::u8string s = u8"overwritten";
+        j.get_to(s);
+        CHECK(s == utf8);
+    }
+
+    SECTION("other types")
+    {
+#if !defined(JSON_NOEXCEPTION)
+        CHECK_THROWS_WITH_AS(json(1).template get<std::u8string>(), "[json.exception.type_error.302] type must be string or array, but is number", json::type_error);
+        CHECK_THROWS_WITH_AS(json(nullptr).template get<std::u8string>(), "[json.exception.type_error.302] type must be string or array, but is null", json::type_error);
+        CHECK_THROWS_AS(json::object().template get<std::u8string>(), json::type_error);
+        // an element that is not a number
+        CHECK_THROWS_AS(json::array({"a"}).template get<std::u8string>(), json::type_error);
 #endif
+    }
+
+    SECTION("ordered_json")
+    {
+        nlohmann::ordered_json const j = utf8;
+        CHECK(j.is_string());
+        CHECK(j.template get<std::string>() == "P\xc4\x9b\xc5\xa1ina");
+        CHECK(j.template get<std::u8string>() == utf8);
+        CHECK(nlohmann::ordered_json::array({0x61, 0x62}).template get<std::u8string>() == u8"ab");
+    }
+
+    SECTION("std::u8string_view and char8_t arrays are unchanged (arrays of code units)")
+    {
+        const std::u8string_view view = utf8;
+        json const j_view = view;
+        CHECK(j_view.is_array());
+        CHECK(j_view.size() == 8);
+
+        const char8_t arr[] = u8"ab";
+        json const j_arr = arr;
+        CHECK(j_arr == json::array({0x61, 0x62, 0}));
+    }
+}
 #endif
 
 #if !defined(JSON_NOEXCEPTION)

tests/src/unit-json_pointer.cpp

@@ -23,10 +23,8 @@ using nlohmann::json;
     using namespace nlohmann::literals; // NOLINT(google-build-using-namespace)
 #endif
 
-#include <algorithm>
 #include <cerrno>
 #include <cstdlib>
-#include <ctime>
 #include <limits>
 #include <map>
 #include <random>
@@ -1562,294 +1560,3 @@ TEST_CASE("JSON patch with moves and copies")
     CHECK_THROWS_AS(inplace.patch_inplace(json::parse(R"([{"op":"move","from":"","path":""}])")), json::out_of_range&);
     CHECK(inplace == doc);
 }
-
-namespace
-{
-// the keys of a shape of flattened objects with long keys; `zero` is the
-// token that is repeated `m` times
-// 0: one key, 1: one `zero` followed by tokens 1, 2: the token 1 followed by
-// tokens `zero`, 3 and 4: two keys with a long prefix, apart from other keys,
-// 5: the same for two long branches, 6: more of the keys of 3
-std::vector<std::string> shape_keys(const int shape, const std::size_t m, const std::string& zero)
-{
-    std::string repeated;
-    for (std::size_t i = 0; i < m; ++i)
-    {
-        repeated += "/" + zero;
-    }
-    std::vector<std::string> keys;
-    switch (shape)
-    {
-        case 0:
-            keys.push_back(repeated);
-            break;
-        case 1:
-        {
-            std::string key = "/" + zero;
-            for (std::size_t i = 1; i < m; ++i)
-            {
-                key += "/1";
-            }
-            keys.push_back(key);
-            break;
-        }
-        case 2:
-        {
-            std::string key = "/1";
-            for (std::size_t i = 1; i < m; ++i)
-            {
-                key += "/" + zero;
-            }
-            keys.push_back(key);
-            break;
-        }
-        case 3:
-        case 4:
-            keys = {repeated + "/a0", "/10", repeated + "/a1", "/11"};
-            break;
-        case 5:
-            keys = {"/x" + repeated + "/a0", "/y" + repeated + "/a0", "/x" + repeated + "/a1", "/y" + repeated + "/a1"};
-            break;
-        case 6:
-            for (int j = 0; j < 4; ++j)
-            {
-                keys.push_back(repeated + "/a" + std::to_string(j));
-                keys.push_back("/1" + std::to_string(j));
-            }
-            break;
-        default:
-            break;
-    }
-    return keys;
-}
-
-// the best of `runs` times in seconds that std::clock() measures to unflatten the keys,
-// which an ordered_json holds in the order they are given (shape 4 uses a
-// json, which sorts them)
-double unflatten_cpu_time(const int shape, const std::size_t m, const int runs, const std::string& zero)
-{
-    const std::vector<std::string> keys = shape_keys(shape, m, zero);
-    nlohmann::ordered_json ordered = nlohmann::ordered_json::object();
-    json sorted = json::object();
-    for (const auto& key : keys)
-    {
-        ordered[key] = 1;
-        sorted[key] = 1;
-    }
-
-    double best = 1e30;
-    for (int i = 0; i < runs; ++i)
-    {
-        const std::clock_t start = std::clock();
-        if (shape == 4)
-        {
-            const json nested = sorted.unflatten();
-        }
-        else
-        {
-            const nlohmann::ordered_json nested = ordered.unflatten();
-        }
-        best = (std::min)(best, static_cast<double>(std::clock() - start) / static_cast<double>(CLOCKS_PER_SEC));
-    }
-    return best;
-}
-} // namespace
-
-TEST_CASE("unflatten of keys with many reference tokens")
-{
-    using nlohmann::ordered_json;
-
-    SECTION("which prefixes become arrays")
-    {
-        // sorted keys that share a long beginning
-        CHECK(json({{"/x/y/q/1", 5}, {"/x/y/w/0", 4}, {"/x/y/w/1", 3}, {"/x/y/z/0", 1}, {"/x/y/z/1", 2}}).unflatten() ==
-        json::parse(R"({"x":{"y":{"q":{"1":5},"w":[4,3],"z":[1,2]}}})"));
-
-        // prefixes that differ in the position of a slash only
-        CHECK(json({{"/a/0/x", 1}, {"/a/1/y", 3}, {"/ab/1", 2}}).unflatten() ==
-        json::parse(R"({"a":[{"x":1},{"y":3}],"ab":{"1":2}})"));
-        CHECK(json({{"/a/b/1", 1}, {"/ab/0", 2}}).unflatten() ==
-        json::parse(R"({"a":{"b":{"1":1}},"ab":[2]})"));
-
-        // tokens longer than one word, escapes, and a token 0 below a token 1
-        const std::string t = "token_with_more_than_eight_bytes";
-        CHECK(json({{"/" + t + "/1/b/0", 1}, {"/" + t + "/0", 2}, {"/" + t + "/2/a~1b", 3}}).unflatten() ==
-        json::parse(R"({"token_with_more_than_eight_bytes":[2,{"b":[1]},{"a/b":3}]})"));
-
-        // prefixes of more than 64 bytes, that share a long beginning
-        std::string deep;
-        json expected = {{"a", {1, 2}}, {"b", {{"1", 3}}}, {"c", {{"1", {{"x", {4}}}}}}};
-        for (int i = 0; i < 30; ++i)
-        {
-            deep += "/seg";
-            expected = {{"seg", expected}};
-        }
-        CHECK(json({{deep + "/c/1/x/0", 4}, {deep + "/a/1", 2}, {deep + "/b/1", 3}, {deep + "/a/0", 1}}).unflatten() == expected);
-        CHECK(json(ordered_json({{deep + "/c/1/x/0", 4}, {deep + "/a/1", 2}, {deep + "/b/1", 3}, {deep + "/a/0", 1}}).unflatten()) == expected);
-        CHECK(json(ordered_json({{deep + "/a/1", 2}, {deep + "/a/0", 1}, {deep + "/b/1", 3}, {deep + "/c/1/x/0", 4}}).unflatten()) == expected);
-
-        // prefixes of every length around the limit up to which they are hashed directly
-        for (std::size_t length = 40; length < 90; ++length)
-        {
-            const std::string name(length, 'x');
-            const json flat = {{"/" + name + "/a/0", 1}, {"/" + name + "/a/1", 2}, {"/" + name + "/b/1", 3}, {"/" + name + "/c/1/0", 4}};
-            CHECK(flat.unflatten() == json({{name, {{"a", {1, 2}}, {"b", {{"1", 3}}}, {"c", {{"1", {4}}}}}}}));
-        }
-
-        // only the parent of a token 0 is an array, however deep the key is
-        CHECK(json({{"/0/1/1/1", 1}}).unflatten() == json::parse(R"([{"1":{"1":{"1":1}}}])"));
-        CHECK(json({{"/1/1/1/0", 1}}).unflatten() == json::parse(R"({"1":{"1":{"1":[1]}}})"));
-        CHECK(json({{"/1/0/1/0", 1}}).unflatten() == json::parse(R"({"1":[{"1":[1]}]})"));
-    }
-
-    SECTION("the result does not depend on the order of the keys")
-    {
-        // random documents without empty containers and without the key "0" in
-        // an object: flatten() and unflatten() are inverse to each other
-        std::mt19937 gen(20261110); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed): reproducible
-        const std::vector<std::string> names = {"a", "b", "ab", "1", "2", "10", "a/b", "a~b", "token_with_more_than_eight_bytes", "x"};
-
-        struct builder
-        {
-            std::mt19937& gen;
-            const std::vector<std::string>& names;
-
-            json operator()(int depth) const
-            {
-                const int kind = std::uniform_int_distribution<int>(0, 9)(gen);
-                if (depth == 0 || kind < 3)
-                {
-                    return kind;
-                }
-                json result = kind < 6 ? json::array() : json::object();
-                const int size = std::uniform_int_distribution<int>(1, 4)(gen);
-                for (int i = 0; i < size; ++i)
-                {
-                    if (kind < 6)
-                    {
-                        result.push_back((*this)(depth - 1));
-                    }
-                    else
-                    {
-                        result[names[std::uniform_int_distribution<std::size_t>(0, names.size() - 1)(gen)]] = (*this)(depth - 1);
-                    }
-                }
-                return result;
-            }
-        };
-        const builder build{gen, names};
-
-        for (int round = 0; round < 200; ++round)
-        {
-            const json doc = build(6);
-            const json flat = doc.flatten();
-            CHECK(flat.unflatten() == doc);
-
-            // the same entries in shuffled order, in an object that keeps it
-            std::vector<std::pair<std::string, json>> entries;
-            for (const auto& el : flat.items())
-            {
-                entries.emplace_back(el.key(), el.value());
-            }
-            std::shuffle(entries.begin(), entries.end(), gen);
-            ordered_json shuffled = ordered_json::object();
-            for (const auto& e : entries)
-            {
-                shuffled[e.first] = e.second;
-            }
-            CHECK(json(shuffled.unflatten()) == doc);
-        }
-    }
-
-    SECTION("keys with long prefixes, in every order")
-    {
-        // prefixes of more than 64 bytes that share a long beginning, and
-        // continue with tokens that start the same way
-        std::string deep;
-        for (int i = 0; i < 30; ++i)
-        {
-            deep += "/seg";
-        }
-        const std::string other = deep.substr(0, 70) + "/x" + deep.substr(70);
-        std::vector<std::string> keys = {deep + "/a/0", deep + "/ab/1", deep + "/ab/x", deep + "/abc/0/y/0", other + "/b/1/0", deep + "/abc/1"};
-        std::sort(keys.begin(), keys.end());
-
-        // the result for the keys in sorted order, which json iterates in
-        json sorted = json::object();
-        for (const auto& key : keys)
-        {
-            sorted[key] = 1;
-        }
-        const json expected = sorted.unflatten();
-        CHECK(expected.is_object());
-
-        // ordered_json iterates in the order the keys were added
-        std::vector<std::size_t> order(keys.size());
-        for (std::size_t i = 0; i < order.size(); ++i)
-        {
-            order[i] = i;
-        }
-        std::size_t permutations = 0;
-        bool all_equal = true;
-        do
-        {
-            nlohmann::ordered_json flat = nlohmann::ordered_json::object();
-            for (const auto i : order)
-            {
-                flat[keys[i]] = 1;
-            }
-            all_equal = all_equal && json(flat.unflatten()) == expected;
-            ++permutations;
-        }
-        while (std::next_permutation(order.begin(), order.end()));
-        CHECK(permutations == 720);
-        CHECK(all_equal);
-    }
-
-    SECTION("time is linear in the number of tokens")
-    {
-        // A bound in seconds would depend on the machine. Instead, the time is
-        // compared with the time for the same keys with the token 0 replaced by
-        // a token that is not a number: there is no array, so no prefix is
-        // looked up. The ratio is 1 to 3 for a linear algorithm, and between
-        // 12 and 100 when looking up a prefix takes time proportional to its
-        // length. The time that std::clock() measures is mostly the time of the
-        // processor, which a busy machine does not inflate (on MSVC, it is the
-        // time that has passed), and the best of two or three runs counts.
-        for (int shape = 0; shape < 7; ++shape)
-        {
-            // the shapes with several keys, in an order that does not sort them,
-            // need more tokens to be slow enough to tell
-            const bool many = shape == 3 || shape == 5 || shape == 6;
-            const std::size_t m = many ? 40000 : 10000;
-            const int runs = many ? 2 : 3;
-
-            // a busy machine may spoil one comparison, hardly two
-            bool linear = false;
-            double t_array = 0;
-            double t_object = 0;
-            for (int attempt = 0; attempt < 2 && !linear; ++attempt)
-            {
-                t_array = unflatten_cpu_time(shape, m, runs, "0");
-                t_object = (std::max)(unflatten_cpu_time(shape, m, runs, "a"), 1e-3);
-                linear = t_array < 6 * t_object;
-            }
-            CAPTURE(shape)
-            CAPTURE(t_array)
-            CAPTURE(t_object)
-            CHECK(linear);
-        }
-
-        // the nesting is as expected: an array exactly below each token 0
-        const json nested_deep = json::object({{shape_keys(0, 1000, "0")[0], 1}}).unflatten();
-        std::size_t depth = 0;
-        for (const json* p = &nested_deep; p->is_array(); p = &p->back())
-        {
-            ++depth;
-        }
-        CHECK(depth == 1000);
-        const json nested_ones = json::object({{shape_keys(1, 1000, "0")[0], 1}}).unflatten();
-        CHECK(nested_ones.is_array());
-        CHECK(nested_ones.back().is_object());
-    }
-}