Compare commits

..
Author SHA1 Message Date
Niels Lohmann af23f07f8c Fix parse(container, function) for plain functions as callback
Passing a plain function or function pointer as parser callback, as in json::parse(std::string("[1,2]"), cb), stopped compiling in C++11/14/17. The iterator+sentinel overloads added for C++20 ranges are constrained by can_compare_ne, i.e. whether `first != last` is well-formed. The callback's parameter types make basic_json an associated class, so ADL finds the friend operator!=(const_reference, ScalarType): the container converts to basic_json and the function pointer is a scalar. The sentinel overload was thereby selected and failed hard in its body.

can_compare_ne now excludes function pointers and functions as sentinel type, since a callback is never a sentinel. This shared constraint guards parse, accept, sax_parse and all from_* overloads. Checking IteratorType via std::iterator_traits was avoided on purpose: it would instantiate iterator_traits for every two-argument call, which is a hard error on standard libraries with a non-SFINAE-friendly iterator_traits.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-11-11 09:24:05 +01:00
12 changed files with 88 additions and 229 deletions

No files matched your search

@@ -67,11 +67,7 @@ basic_json(basic_json&& other) noexcept;
- **objects**: [`object_t`](object_t.md) and all kinds of compatible associative containers such as `std::map`,
`std::unordered_map`, `std::multimap`, and `std::unordered_multimap` with a `key_type` compatible to `string_t`
and a `value_type` from which a `basic_json` value can be constructed.
- **strings**: `string_t`, string literals, and all compatible string containers can be used. Since version 3.13.0,
this includes `#!cpp std::u8string` (C++20): its code units are stored as a JSON string and
[`get<std::u8string>()`](get.md) reads it back (in version 3.12.0, a `#!cpp std::u8string` was stored as an array
of numbers; `get<std::u8string>()` still accepts such an array). `#!cpp std::u8string_view` and `#!cpp char8_t`
arrays are not strings; they are stored as arrays of code units.
- **strings**: `string_t`, string literals, and all compatible string containers can be used.
!!! note "Rvalue containers are moved from"
-6
View File
@@ -130,12 +130,6 @@ overload (3).
`#!cpp json(true).get<double>()` throw. This is the behavior of all 3.x versions;
[`basic_json_view::get<T>()`](../basic_json_view/get.md) behaves the same.
!!! note "`std::u8string` conversions"
Since version 3.13.0, `#!cpp get<std::u8string>()` (C++20) reads a JSON string; it also accepts an array of numbers,
the form in which version 3.12.0 stored a `#!cpp std::u8string`. Any other JSON type throws
[`type_error.302`](../../home/exceptions.md#jsonexceptiontype_error302) ("type must be string or array").
!!! note "`std::optional` conversions"
Prior to version 3.13.0, `#!cpp get<std::optional<T>>()` (and other conversions to `std::optional<T>`) failed to
@@ -425,7 +425,7 @@ using array_t = ArrayType<basic_json, AllocatorType<basic_json>>;
| Type | Reason |
|----------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
| `#!cpp std::wstring`, `#!cpp std::u16string`, `#!cpp std::u32string` | the character type is not one byte wide |
| `#!cpp std::u8string` | one byte wide, but `#!cpp char8_t` is not `#!cpp char`-compatible (as a *value*, it converts to and from a JSON string) |
| `#!cpp std::u8string` | one byte wide, but `#!cpp char8_t` is not `#!cpp char`-compatible |
| `absl::Cord` | no `value_type`, and the storage is not contiguous |
| `QString` | no `append(const char*, size_type)`; its `QChar` is also two bytes wide, though that is never diagnosed |
@@ -8,16 +8,6 @@ These changes are part of the `develop` branch, but not of a release yet.
exception is thrown during the conversion, the source may be partially moved-from, so the strong exception guarantee
no longer holds for the source of an rvalue conversion (the JSON value itself is unaffected). Lvalues and `const`
objects are still copied. See [`basic_json`](../../api/basic_json/basic_json.md).
- In C++20 (library support for `char8_t`, `__cpp_lib_char8_t`), a `#!cpp std::u8string` is a JSON **string**:
`json j = std::u8string(u8"abc");` stores `"abc"`, and `#!cpp j.get<std::u8string>()` reads it back. Version 3.12.0
stored a `#!cpp std::u8string` as an array of code units (`[97,98,99]`); on the 3.13.0 development branch it was
already stored as a string if `std::filesystem` is available, but could not be read back. Now the result no longer
depends on whether `std::filesystem::path` is supported, and `#!cpp get<std::u8string>()` accepts both a JSON string
and the array of numbers written by 3.12.0. **Consequence:** JSON documents produced from a `#!cpp std::u8string` by
3.12.0 (arrays) are still read, but code that checks `is_array()` on such a value, or that reads it as
`#!cpp std::vector<char8_t>` / `#!cpp std::vector<std::uint8_t>`, now sees a string; to keep the array form, convert
explicitly, e.g., `#!cpp json(std::vector<char8_t>(s.begin(), s.end()))`. `#!cpp std::u8string_view` and
`#!cpp char8_t` arrays are unchanged (arrays of code units). See [`basic_json`](../../api/basic_json/basic_json.md).
## API changes
@@ -785,37 +785,6 @@ void from_json(const BasicJsonType& j, std::unordered_map<Key, Value, Hash, KeyE
from_json_pair_array_to_map(j, m);
}
#if defined(__cpp_lib_char8_t)
// a std::u8string is read from a JSON string; the array of code units written by
// versions before 3.13.0 is still accepted
template < typename BasicJsonType, typename Tr, typename Allocator,
enable_if_t < std::is_same<typename json_type_of_t<BasicJsonType>::string_t::value_type, char>::value, int > = 0 >
inline void from_json(const BasicJsonType& j, std::basic_string<char8_t, Tr, Allocator>& s)
{
if (j.is_string())
{
const auto& str = string_value(j);
const char* const data = str.data();
const std::size_t size = str.size();
std::basic_string<char8_t, Tr, Allocator> ret(s.get_allocator());
ret.reserve(size);
for (std::size_t i = 0; i < size; ++i)
{
ret.push_back(static_cast<char8_t>(static_cast<unsigned char>(data[i])));
}
s = std::move(ret);
}
else if (j.is_array())
{
from_json_array_impl(j, s, priority_tag<3> {});
}
else
{
throw_type_must_be("string or array", j);
}
}
#endif
#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
// Workaround for MSVC 19.51 (and possibly later): in large cpp files, the compiler may fail to resolve with generic has_from_json (issue #4996)
@@ -625,20 +625,16 @@ inline void to_json(BasicJsonType& j, const T& t)
to_json_tuple_impl(j, t, make_index_sequence<std::tuple_size<T>::value> {});
}
// a std::u8string is a JSON string (it was an array of code units before 3.13.0);
// defined outside the std::filesystem guard so that the result does not depend
// on whether std::filesystem::path is supported
#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
#if defined(__cpp_lib_char8_t)
template<typename BasicJsonType, typename Tr, typename Allocator>
inline void to_json(BasicJsonType& j, const std::basic_string<char8_t, Tr, Allocator>& s)
{
using string_t = typename BasicJsonType::string_t;
// char may alias any object; the code units are copied unchanged
j = string_t(reinterpret_cast<const char*>(s.data()), s.size());
using OtherAllocator = typename std::allocator_traits<Allocator>::template rebind_alloc<char>;
j = std::basic_string<char, std::char_traits<char>, OtherAllocator>(s.begin(), s.end(), s.get_allocator());
}
#endif
#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
// Workaround for MSVC 19.51 (and possibly later): in large cpp files, the compiler may fail to resolve with generic has_to_json (issue #4996)
template<typename BasicJsonType>
struct has_to_json<BasicJsonType, std_fs::path, void> : std::true_type {};
@@ -1312,15 +1312,23 @@ struct can_compare_ne_either_order : std::integral_constant < bool,
can_compare_ne_impl<IteratorType, SentinelType>::value ||
can_compare_ne_reversed<IteratorType, SentinelType>::value > {};
// std::nullptr_t is excluded explicitly: a literal `nullptr` passed as a
// trailing default argument (e.g. parse(s, nullptr, ...)) must never be
// mistaken for a sentinel, and some compilers (e.g. GCC 4.8) unreliably
// SFINAE the `operator!=` detection above for std::nullptr_t against
// container/string types, which would otherwise make such calls ambiguous
// with the compatible-input overload.
// Two kinds of SentinelType are excluded explicitly:
//
// - std::nullptr_t: a literal `nullptr` passed as a trailing default argument
// (e.g. parse(s, nullptr, ...)) must never be mistaken for a sentinel, and
// some compilers (e.g. GCC 4.8) unreliably SFINAE the `operator!=` detection
// above for std::nullptr_t against container/string types, which would
// otherwise make such calls ambiguous with the compatible-input overload.
// - function pointers and functions: a parser callback passed as plain function
// (e.g. parse(s, cb)) is never a sentinel. The callback's parameter types
// make basic_json an associated class, so ADL finds the friend
// operator!=(const_reference, ScalarType) of basic_json, which makes
// `s != cb` well-formed (both convert to basic_json) and would select the
// iterator/sentinel overload.
template<typename IteratorType, typename SentinelType>
struct can_compare_ne : std::integral_constant < bool,
!std::is_same<SentinelType, std::nullptr_t>::value &&
!std::is_function<typename std::remove_pointer<SentinelType>::type>::value &&
can_compare_ne_either_order<IteratorType, SentinelType>::value > {};
template<typename T>
+17 -44
View File
@@ -6928,37 +6928,6 @@ void from_json(const BasicJsonType& j, std::unordered_map<Key, Value, Hash, KeyE
from_json_pair_array_to_map(j, m);
}
#if defined(__cpp_lib_char8_t)
// a std::u8string is read from a JSON string; the array of code units written by
// versions before 3.13.0 is still accepted
template < typename BasicJsonType, typename Tr, typename Allocator,
enable_if_t < std::is_same<typename json_type_of_t<BasicJsonType>::string_t::value_type, char>::value, int > = 0 >
inline void from_json(const BasicJsonType& j, std::basic_string<char8_t, Tr, Allocator>& s)
{
if (j.is_string())
{
const auto& str = string_value(j);
const char* const data = str.data();
const std::size_t size = str.size();
std::basic_string<char8_t, Tr, Allocator> ret(s.get_allocator());
ret.reserve(size);
for (std::size_t i = 0; i < size; ++i)
{
ret.push_back(static_cast<char8_t>(static_cast<unsigned char>(data[i])));
}
s = std::move(ret);
}
else if (j.is_array())
{
from_json_array_impl(j, s, priority_tag<3> {});
}
else
{
throw_type_must_be("string or array", j);
}
}
#endif
#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
// Workaround for MSVC 19.51 (and possibly later): in large cpp files, the compiler may fail to resolve with generic has_from_json (issue #4996)
@@ -8360,20 +8329,16 @@ inline void to_json(BasicJsonType& j, const T& t)
to_json_tuple_impl(j, t, make_index_sequence<std::tuple_size<T>::value> {});
}
// a std::u8string is a JSON string (it was an array of code units before 3.13.0);
// defined outside the std::filesystem guard so that the result does not depend
// on whether std::filesystem::path is supported
#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
#if defined(__cpp_lib_char8_t)
template<typename BasicJsonType, typename Tr, typename Allocator>
inline void to_json(BasicJsonType& j, const std::basic_string<char8_t, Tr, Allocator>& s)
{
using string_t = typename BasicJsonType::string_t;
// char may alias any object; the code units are copied unchanged
j = string_t(reinterpret_cast<const char*>(s.data()), s.size());
using OtherAllocator = typename std::allocator_traits<Allocator>::template rebind_alloc<char>;
j = std::basic_string<char, std::char_traits<char>, OtherAllocator>(s.begin(), s.end(), s.get_allocator());
}
#endif
#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
// Workaround for MSVC 19.51 (and possibly later): in large cpp files, the compiler may fail to resolve with generic has_to_json (issue #4996)
template<typename BasicJsonType>
struct has_to_json<BasicJsonType, std_fs::path, void> : std::true_type {};
@@ -10456,15 +10421,23 @@ struct can_compare_ne_either_order : std::integral_constant < bool,
can_compare_ne_impl<IteratorType, SentinelType>::value ||
can_compare_ne_reversed<IteratorType, SentinelType>::value > {};
// std::nullptr_t is excluded explicitly: a literal `nullptr` passed as a
// trailing default argument (e.g. parse(s, nullptr, ...)) must never be
// mistaken for a sentinel, and some compilers (e.g. GCC 4.8) unreliably
// SFINAE the `operator!=` detection above for std::nullptr_t against
// container/string types, which would otherwise make such calls ambiguous
// with the compatible-input overload.
// Two kinds of SentinelType are excluded explicitly:
//
// - std::nullptr_t: a literal `nullptr` passed as a trailing default argument
// (e.g. parse(s, nullptr, ...)) must never be mistaken for a sentinel, and
// some compilers (e.g. GCC 4.8) unreliably SFINAE the `operator!=` detection
// above for std::nullptr_t against container/string types, which would
// otherwise make such calls ambiguous with the compatible-input overload.
// - function pointers and functions: a parser callback passed as plain function
// (e.g. parse(s, cb)) is never a sentinel. The callback's parameter types
// make basic_json an associated class, so ADL finds the friend
// operator!=(const_reference, ScalarType) of basic_json, which makes
// `s != cb` well-formed (both convert to basic_json) and would select the
// iterator/sentinel overload.
template<typename IteratorType, typename SentinelType>
struct can_compare_ne : std::integral_constant < bool,
!std::is_same<SentinelType, std::nullptr_t>::value &&
!std::is_function<typename std::remove_pointer<SentinelType>::type>::value &&
can_compare_ne_either_order<IteratorType, SentinelType>::value > {};
template<typename T>
-10
View File
@@ -364,16 +364,6 @@ json_test_add_test_for(src/unit-class_parser.cpp
MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force}
)
# test the conversions again without std::filesystem::path support: a
# std::u8string must be a JSON string in this configuration, too
json_test_set_test_options(test-conversions2_no_filesystem
COMPILE_DEFINITIONS JSON_HAS_FILESYSTEM=0 JSON_HAS_EXPERIMENTAL_FILESYSTEM=0
)
json_test_add_test_for(src/unit-conversions2.cpp
NAME test-conversions2_no_filesystem
MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force}
)
# test diagnostic positions again without regular diagnostics (JSON pointer paths)
json_test_set_test_options(test-diagnostic-positions_only
COMPILE_DEFINITIONS JSON_DIAGNOSTICS=0
-16
View File
@@ -536,20 +536,4 @@ TEST_CASE("alternative string type with json_view")
}
}
// a std::u8string is a JSON string, whatever the string type of the JSON value
// JSON_HAS_CPP_20 JSON_HAS_CPP_23 (do not remove; see note at top of file)
#if defined(__cpp_lib_char8_t)
TEST_CASE("alternative string type and std::u8string")
{
const std::u8string s = u8"abc";
const alt_json j = s;
CHECK(j.is_string());
CHECK(j.get<alt_string>() == alt_string("abc"));
CHECK(j.get<std::u8string>() == s);
// the array of code units written by versions before 3.13.0
CHECK(alt_json::array({0x61, 0x62, 0x63}).get<std::u8string>() == s);
}
#endif
DOCTEST_CLANG_SUPPRESS_WARNING_POP
+37
View File
@@ -3232,3 +3232,40 @@ TEST_CASE("object keys are delivered decoded and validated")
CHECK(keys_of_format(std::string("\x87\xC3\x28\x91"), json::input_format_t::bon8).empty());
}
}
namespace
{
// a plain function (not a lambda or std::function) used as parser callback
bool keep_everything_callback(int /*depth*/, json::parse_event_t /*event*/, json& /*parsed*/)
{
return true;
}
} // namespace
TEST_CASE("a plain function is accepted as parser callback for containers")
{
// ADL finds basic_json's friend operator!= for "container != function
// pointer", which must not make the iterator/sentinel overloads viable
const json expected = {1, 2};
const std::string text = "[1,2]";
const std::vector<std::uint8_t> bytes(text.begin(), text.end());
const std::vector<char> chars(text.begin(), text.end());
CHECK(json::parse(text, keep_everything_callback) == expected);
CHECK(json::parse(text, &keep_everything_callback) == expected);
CHECK(json::parse(std::string("[1,2]"), keep_everything_callback) == expected);
CHECK(json::parse(bytes, keep_everything_callback) == expected);
CHECK(json::parse(bytes, &keep_everything_callback) == expected);
CHECK(json::parse(chars, keep_everything_callback) == expected);
CHECK(json::parse(text.begin(), text.end(), keep_everything_callback) == expected);
// the sentinel overloads (shared constraint of parse, accept, sax_parse and
// the from_* functions) are not viable for a container and a function
// pointer, but still are for iterator pairs
using callback_pointer = decltype(&keep_everything_callback);
CHECK_FALSE((nlohmann::detail::can_compare_ne<std::string, callback_pointer>::value));
CHECK_FALSE((nlohmann::detail::can_compare_ne<std::vector<std::uint8_t>, callback_pointer>::value));
CHECK((nlohmann::detail::can_compare_ne<std::string::const_iterator, std::string::const_iterator>::value));
CHECK((nlohmann::detail::can_compare_ne<const char*, const char*>::value));
}
+15 -93
View File
@@ -27,12 +27,10 @@ using nlohmann::json;
#include <deque>
#include <forward_list>
#include <list>
#include <map>
#include <set>
#include <unordered_map>
#include <unordered_set>
#include <valarray>
#include <vector>
// NLOHMANN_JSON_SERIALIZE_ENUM uses a static std::pair
DOCTEST_CLANG_SUPPRESS_WARNING_PUSH
@@ -709,115 +707,39 @@ TEST_CASE("std::filesystem::path")
}
#endif
// a std::u8string is a JSON string since 3.13.0 (it was an array of code units
// before); independent of whether std::filesystem::path is supported, see the
// test-conversions2_no_filesystem target
// JSON_HAS_CPP_20 JSON_HAS_CPP_23 (do not remove; see note at top of file)
// the ADL to_json overload for std::u8string only exists under the same guard
// as std::filesystem::path support (it is otherwise only reached indirectly,
// via std::filesystem::path::u8string()) -- mirror both #if conditions from
// include/nlohmann/detail/conversions/to_json.hpp exactly
#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
#if defined(__cpp_lib_char8_t)
TEST_CASE("std::u8string")
{
// use \u universal-character-names (rather than raw \x byte escapes
// or literal non-ASCII source bytes) to compose the multi-byte UTF-8
// encoding -- MSVC treats \x escapes used that way inside a u8
// literal as a nonstandard extension (warning C5321), which some of
// our CI configs promote to an error; \u is portable and produces
// the exact same encoded bytes without depending on the source
// file's encoding
const std::u8string utf8 = u8"P\u011B\u0161ina";
SECTION("ascii")
{
const std::u8string s = u8"Path";
json const j = s;
CHECK(j.is_string());
CHECK(j.template get<std::string>() == "Path");
}
SECTION("utf-8")
{
json const j = utf8;
// use \u universal-character-names (rather than raw \x byte escapes
// or literal non-ASCII source bytes) to compose the multi-byte UTF-8
// encoding -- MSVC treats \x escapes used that way inside a u8
// literal as a nonstandard extension (warning C5321), which some of
// our CI configs promote to an error; \u is portable and produces
// the exact same encoded bytes without depending on the source
// file's encoding
const std::u8string s = u8"P\u011B\u0161ina";
json const j = s;
CHECK(j.is_string());
CHECK(j.template get<std::string>() == "P\xc4\x9b\xc5\xa1ina");
}
SECTION("round trip")
{
json const j = utf8;
CHECK(j.template get<std::u8string>() == utf8);
std::u8string back;
j.get_to(back);
CHECK(back == utf8);
// from a string
json const j_string = "P\xc4\x9b\xc5\xa1ina";
CHECK(j_string.template get<std::u8string>() == utf8);
// empty, and with an embedded NUL
CHECK(json(std::u8string()).template get<std::u8string>().empty());
const std::u8string with_nul(u8"a\0b", 3);
CHECK(json(with_nul).template get<std::u8string>() == with_nul);
// the lvalue and rvalue paths, and in containers
std::u8string lvalue = utf8;
CHECK(json(lvalue).template get<std::u8string>() == utf8);
CHECK(json(std::move(lvalue)).template get<std::u8string>() == utf8);
const std::vector<std::u8string> vec = {utf8, u8"x"};
json const j_vec = vec;
CHECK(j_vec.is_array());
CHECK(j_vec[0].is_string());
CHECK(j_vec.template get<std::vector<std::u8string>>() == vec);
const std::map<std::string, std::u8string> map = {{"k", utf8}};
CHECK(json(map).template get<std::map<std::string, std::u8string>>() == map);
}
SECTION("array of code units (written by versions before 3.13.0)")
{
json const j = json::array({0x50, 0xC4, 0x9B, 0xC5, 0xA1, 0x69, 0x6E, 0x61});
CHECK(j.template get<std::u8string>() == utf8);
CHECK(json::array().template get<std::u8string>().empty());
CHECK(json::parse("[97, 98]").template get<std::u8string>() == u8"ab");
std::u8string s = u8"overwritten";
j.get_to(s);
CHECK(s == utf8);
}
SECTION("other types")
{
#if !defined(JSON_NOEXCEPTION)
CHECK_THROWS_WITH_AS(json(1).template get<std::u8string>(), "[json.exception.type_error.302] type must be string or array, but is number", json::type_error);
CHECK_THROWS_WITH_AS(json(nullptr).template get<std::u8string>(), "[json.exception.type_error.302] type must be string or array, but is null", json::type_error);
CHECK_THROWS_AS(json::object().template get<std::u8string>(), json::type_error);
// an element that is not a number
CHECK_THROWS_AS(json::array({"a"}).template get<std::u8string>(), json::type_error);
#endif
}
SECTION("ordered_json")
{
nlohmann::ordered_json const j = utf8;
CHECK(j.is_string());
CHECK(j.template get<std::string>() == "P\xc4\x9b\xc5\xa1ina");
CHECK(j.template get<std::u8string>() == utf8);
CHECK(nlohmann::ordered_json::array({0x61, 0x62}).template get<std::u8string>() == u8"ab");
}
SECTION("std::u8string_view and char8_t arrays are unchanged (arrays of code units)")
{
const std::u8string_view view = utf8;
json const j_view = view;
CHECK(j_view.is_array());
CHECK(j_view.size() == 8);
const char8_t arr[] = u8"ab";
json const j_arr = arr;
CHECK(j_arr == json::array({0x61, 0x62, 0}));
}
}
#endif
#endif
#if !defined(JSON_NOEXCEPTION)
namespace