Docs: clarify file reading speed, scope of file replacement, cross-device rename

- The 10 to 20 % figure compares parsing from a std::ifstream with reading
  the file into a string with one read() and parsing that; reading with
  istreambuf_iterator is about twice as slow as parsing from the stream.
- Say that replacing a file is general C++ advice, not library behaviour.
- std::filesystem::rename fails across file systems (EXDEV) instead of
  falling back to a copy.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/patch_inplace.md

@@ -15,7 +15,9 @@ the patch. This function applies a JSON patch in place and returns void.
 
 ## Exception safety
 
-No guarantees, value may be corrupted by an unsuccessful patch operation.
+No guarantees: if an operation fails, the operations before it stay applied, and a failing `move` operation may already
+have removed the value at its "from" location. Use [`patch`](patch.md) if the value must be unchanged after a failed
+patch.
 
 ## Exceptions
 
@@ -62,8 +64,19 @@ the JSON value, the complexity can usually be neglected.
 ## Notes
 
 Unlike [`patch`](patch.md), `patch_inplace` applies the operation "in place" and no copy of the JSON value is created.
-That makes it faster for large documents by avoiding the copy. However, the JSON value might be corrupted if the
+That makes it faster for large documents by avoiding the copy.
-function throws an exception.
+
+!!! warning "Not atomic"
+
+    `patch_inplace` applies the operations one after the other and does not undo them when one fails. After an
+    exception, the value holds the result of the operations before the failing one, and a failing `move` operation
+    (a "remove" followed by an "add") may already have removed the value at its "from" location; see the second example
+    below. [RFC 6902](https://tools.ietf.org/html/rfc6902#section-5) requires that a patch with a failing operation is
+    not deemed successful, but leaves open what happens to the document.
+
+    If the value must stay unchanged when the patch fails, use [`patch`](patch.md), which applies the patch to a copy:
+    `#!cpp j = j.patch(json_patch);`. Validating the patch beforehand is not enough, as "test" operations and missing
+    paths are only detected while the patch is applied.
 
 Duplicate member names inside an operation object are resolved by the parser before the patch is applied (the last one
 is kept, see [`object_t`](object_t.md#behavior)), so `#!json {"op": "add", "op": "remove", "path": "/a"}` is a

docs/mkdocs/docs/features/files.md

@@ -0,0 +1,127 @@
+# Reading and writing files
+
+The library has no functions that take a file name. It reads from and writes to streams, so files are handled with
+`#!cpp std::ifstream` and `#!cpp std::ofstream` (or a `#!cpp FILE*` when reading). This page collects the recipes for
+the common cases and the mistakes that show up again and again.
+
+## Reading a JSON file
+
+Pass the stream to [`parse`](../api/basic_json/parse.md) directly, and check that the file was opened:
+
+```cpp
+#include <fstream>
+#include <stdexcept>
+#include <string>
+#include <nlohmann/json.hpp>
+
+using json = nlohmann::json;
+
+json read_json_file(const std::string& path)
+{
+    std::ifstream f(path);
+    if (!f.is_open())
+    {
+        throw std::runtime_error("cannot open " + path);
+    }
+    return json::parse(f);
+}
+```
+
+- **Check that the file is open.** A stream that failed to open looks like an empty input, so `parse` reports
+  [`parse_error.101`](../home/exceptions.md#jsonexceptionparse_error101) ("unexpected end of input") at line 1,
+  column 1 instead of saying that the file is missing. A relative path is resolved against the working directory of the
+  running program, which is often not the directory of the source file (see the
+  [FAQ](../home/faq.md#parsing-a-file-fails-with-unexpected-end-of-input)).
+- **Don't read the file into a string first.** Parsing from a `#!cpp std::ifstream` is only about 10 to 20 % slower
+  than reading the whole file into a string with one `read()` and parsing that, and it needs no memory for the copy (see
+  [input sources and performance](parsing/index.md#input-sources-and-performance)). The common way to read a file with
+  `#!cpp std::istreambuf_iterator<char>` is so slow that it takes about twice as long as parsing from the stream. Read
+  the file into a string only if you need the text itself, for example for an error message.
+- `parse` requires the file to hold exactly one JSON value. Files with one value per line are read with
+  [JSON Lines](parsing/json_lines.md); files too large for memory are covered in
+  [Large, streamed, and partial input](parsing/large_and_partial_input.md).
+
+## Writing a JSON file
+
+[`dump`](../api/basic_json/dump.md) the value into a string first, then write the string:
+
+```cpp
+void write_json_file(const std::string& path, const json& j)
+{
+    const std::string text = j.dump(4) + '\n';  // may throw: nothing is written yet
+
+    std::ofstream f(path, std::ios::binary);
+    if (!f.is_open())
+    {
+        throw std::runtime_error("cannot open " + path);
+    }
+    f << text;
+    f.close();
+    if (f.fail())
+    {
+        throw std::runtime_error("cannot write " + path);
+    }
+}
+```
+
+- **Serialize before opening the file.** Opening a `#!cpp std::ofstream` truncates the file. `dump` and
+  [`operator<<`](../api/operator_ltlt.md) throw [`type_error.316`](../home/exceptions.md#jsonexceptiontype_error316)
+  when a string is not valid UTF-8 (see [handling invalid UTF-8](serialization.md#handling-invalid-utf-8)), and
+  `#!cpp f << std::setw(4) << j` may already have written part of the value when it throws. Serializing into a string
+  first leaves the old file untouched in that case.
+- **Check the stream after closing it.** A full disk or a missing permission sets the stream's error state; it does not
+  throw unless you enabled exceptions on the stream with `exceptions()`.
+- `std::ios::binary` writes the text as it is. Without it, Windows writes `\r\n` for every newline of a pretty-printed
+  value.
+
+## Replacing a file without corrupting it
+
+The library never writes a file itself, so this section is not about the library. It is general C++ advice for a
+problem users of the library keep running into.
+
+A program that crashes or is killed while it writes leaves a truncated file behind. To make sure readers see either the
+old or the new contents, write to a temporary file in the same directory and rename it over the original
+(C++17, `#!cpp <filesystem>`):
+
+```cpp
+#include <filesystem>
+
+void replace_json_file(const std::filesystem::path& path, const json& j)
+{
+    std::filesystem::path tmp = path;
+    tmp += ".tmp";  // use a unique name if several processes may write the file
+
+    write_json_file(tmp.string(), j);
+    std::filesystem::rename(tmp, path);  // replaces path
+}
+```
+
+The temporary file must be in the same directory (more precisely, on the same file system) as the target: a rename
+across file systems does not fall back to a copy, it fails and throws `#!cpp std::filesystem::filesystem_error`. On
+POSIX systems, the rename atomically replaces the file. Standard C++ has no way to flush a file to the disk (`fsync`),
+so this protects against a crashed program, but not necessarily against a power loss.
+
+## Binary formats
+
+The [binary formats](binary_formats/index.md) work the same way, with the streams opened in binary mode:
+
+```cpp
+// read
+std::ifstream in("data.cbor", std::ios::binary);
+json j = json::from_cbor(in);
+
+// write
+const std::vector<std::uint8_t> bytes = json::to_cbor(j);
+std::ofstream out("data.cbor", std::ios::binary);
+out.write(reinterpret_cast<const char*>(bytes.data()), static_cast<std::streamsize>(bytes.size()));
+```
+
+Without `std::ios::binary`, Windows turns the bytes `0x0D 0x0A` into `0x0A` and may stop at a byte `0x1A` while reading,
+and turns `0x0A` into `0x0D 0x0A` while writing, which corrupts the data.
+
+## See also
+
+- [Parsing](parsing/index.md) - the inputs `parse` accepts
+- [Serialization](serialization.md) - the options of `dump`
+- [`parse`](../api/basic_json/parse.md), [`dump`](../api/basic_json/dump.md)
+- [`operator>>`](../api/operator_gtgt.md), [`operator<<`](../api/operator_ltlt.md)

docs/mkdocs/docs/features/index.md

@@ -46,6 +46,8 @@ C++ types, and finally serialize it again.
   [BJData](binary_formats/bjdata.md), [BON8](binary_formats/bon8.md), [BSON](binary_formats/bson.md),
   [CBOR](binary_formats/cbor.md), [MessagePack](binary_formats/messagepack.md), or [UBJSON](binary_formats/ubjson.md).
 - [Binary values](binary_values.md) — store and exchange raw byte sequences.
+- [Reading and writing files](files.md) — read and write files safely, including replacing a file without
+  corrupting it.
 
 ## How values are stored and configured
 

docs/mkdocs/docs/features/json_patch.md

@@ -29,7 +29,8 @@ other three continue on its result.
 
     [`patch`](../api/basic_json/patch.md) leaves the original value unchanged and returns the patched result as a copy.
     If the document is large and the original value is no longer needed,
-    [`patch_inplace`](../api/basic_json/patch_inplace.md) applies the same operations in place instead.
+    [`patch_inplace`](../api/basic_json/patch_inplace.md) applies the same operations in place instead. Unlike
+    `patch`, it is not atomic: if an operation fails, the operations before it stay applied.
 
 !!! note "Duplicate member names"
 

docs/mkdocs/docs/features/parsing/index.md

@@ -35,6 +35,8 @@ std::ifstream f("example.json");
 json data = json::parse(f);
 ```
 
+See [Reading and writing files](../files.md) for checking that the file was opened and for writing files safely.
+
 The input must be encoded in UTF-8; other encodings are not supported. The exception is wide-character input, which
 is read as UTF-16 or UTF-32 — depending on the character type — and converted to UTF-8; see the
 [`parse`](../../api/basic_json/parse.md) page for an example. A single input may contain only one JSON value.

docs/mkdocs/docs/features/serialization.md

@@ -17,6 +17,9 @@ To write a value directly to a stream (for example, a file or `#!cpp std::cout`)
 std::cout << j << std::endl;
 ```
 
+To write a file without leaving a truncated file behind when serialization fails, see
+[Reading and writing files](files.md#writing-a-json-file).
+
 !!! note "String, not raw value"
 
     `dump` always returns a **JSON text**. Serializing a JSON string therefore includes the surrounding quotes and

docs/mkdocs/docs/home/faq.md

@@ -110,7 +110,7 @@ in the source files that need it, instantiate `basic_json` once, and consider pr
 The parser did not see any character: the file was most likely not opened. A relative path is resolved against the
 working directory of the running program, which is often the build directory of your IDE, not the source directory.
 Check `is_open()` and try an absolute path. Other causes are an empty file and a stream that was already read. See
-[`parse_error.101`](exceptions.md#jsonexceptionparse_error101).
+[`parse_error.101`](exceptions.md#jsonexceptionparse_error101) and [Reading and writing files](../features/files.md).
 
 ### Input that arrives in chunks
 

docs/mkdocs/mkdocs.yml

@@ -130,6 +130,7 @@ nav:
           - features/element_access/unchecked_access.md
           - features/element_access/checked_access.md
           - features/element_access/default_value.md
+      - features/files.md
       - features/iterators.md
       - features/merge_patch.md
       - features/json_patch.md

tests/src/unit-allocator.cpp

@@ -1038,14 +1038,6 @@ template<class T>
 struct sweep_allocator
 {
     using value_type = T;
-    // GCC 4.8's std::map takes these from the allocator instead of from
-    // std::allocator_traits
-    using pointer = T*;
-    using const_pointer = const T*;
-    using reference = T&;
-    using const_reference = const T&;
-    using size_type = std::size_t;
-    using difference_type = std::ptrdiff_t;
 
     sweep_allocator() = default;
     template<class U> sweep_allocator(const sweep_allocator<U>& /*unused*/) noexcept {}
@@ -1163,14 +1155,9 @@ TEST_CASE("parse() survives a failing allocation at every point (#5985)")
 
     // With iterator debugging (MSVC STL debug builds, also used by clang-cl),
     // containers allocate a debug proxy through the allocator inside their
-    // noexcept default and move constructors, so failing that allocation
+    // noexcept move constructors, so failing that allocation terminates the
-    // terminates the program instead of throwing std::bad_alloc. Nothing to
+    // program instead of throwing std::bad_alloc. Nothing to check there.
-    // check there.
 #if !(defined(_ITERATOR_DEBUG_LEVEL) && _ITERATOR_DEBUG_LEVEL > 0)
-    // Clang 3.4 and 3.5 leak memory when an element constructor throws
-    // inside libstdc++'s containers (a std::map copy, a std::vector
-    // reallocation), without any json value involved.
-#if !(defined(__clang__) && !defined(__apple_build_version__) && __clang_major__ == 3 && __clang_minor__ < 6)
     SECTION("parser callback")
     {
         const auto keep_all = [](int /*depth*/, sweep_json::parse_event_t /*event*/, sweep_json& /*parsed*/) noexcept
@@ -1191,6 +1178,7 @@ TEST_CASE("parse() survives a failing allocation at every point (#5985)")
             return sweep_json::parse(text, drop_d);
         });
     }
+#endif
 
     SECTION("sax_parse into an existing value")
     {
@@ -1204,7 +1192,5 @@ TEST_CASE("parse() survives a failing allocation at every point (#5985)")
             return value;
         });
     }
-#endif
-#endif
 }
 #endif

tests/src/unit-class_lexer.cpp

@@ -2222,10 +2222,9 @@ TEST_CASE("serialization of short strings")
     {
         CAPTURE(s)
         const json j = s;
-        const std::string quoted = "\"" + s + "\""; // NOLINT(performance-inefficient-string-concatenation)
+        CHECK(j.dump() == "\"" + s + "\"");
-        CHECK(j.dump() == quoted);
+        CHECK(j.dump(-1, ' ', true) == "\"" + s + "\"");
-        CHECK(j.dump(-1, ' ', true) == quoted);
+        CHECK(json({{s, s}}).dump() == "{\"" + s + "\":\"" + s + "\"}");
-        CHECK(json({{s, s}}).dump() == "{" + quoted + ":" + quoted + "}"); // NOLINT(performance-inefficient-string-concatenation)
     }
 
     // a character that needs an escape

tests/src/unit-custom-object-key-type.cpp

@@ -160,12 +160,12 @@ class strict_key
     }
 
   private:
-    std::string m_value;
+    std::string m_value{};
 };
 
 strict_key make_strict_key(const std::string& key)
 {
-    return {key.c_str()};
+    return strict_key(key.c_str());
 }
 
 template<class Key, class Value, class Compare, class Allocator>
@@ -224,9 +224,7 @@ TEST_CASE("converting a value with keys that cannot be made from a std::string n
         }
     }
 
-    // direct-initialization: a copy-initialization makes GCC 4.8 instantiate the
+    const strict_json converted = value;
-    // copy constructor, which needs a range constructor strict_object lacks
-    const strict_json converted(value);
 
     // at() needs to make a std::string of a key for its error message
     const auto member = [](const strict_json & object, const char* name) -> const strict_json &

tests/src/unit-json_view.cpp

@@ -105,53 +105,19 @@ template<typename T>
 struct failing_allocator
 {
     using value_type = T;
-    // GCC 4.8's std::map takes these from the allocator instead of from
-    // std::allocator_traits
-    using pointer = T*;
-    using const_pointer = const T*;
-    using reference = T&;
-    using const_reference = const T&;
-    using size_type = std::size_t;
-    using difference_type = std::ptrdiff_t;
-
-    template<typename U>
-    struct rebind
-    {
-        using other = failing_allocator<U>;
-    };
 
     failing_allocator() = default;
 
     template<typename U>
     explicit failing_allocator(const failing_allocator<U>& /*other*/) noexcept {}
 
-    [[noreturn]] T* allocate(std::size_t /*n*/) // NOLINT(readability-convert-member-functions-to-static)
+    T* allocate(std::size_t /*n*/) // NOLINT(readability-convert-member-functions-to-static)
     {
         throw std::bad_alloc();
     }
 
     void deallocate(T* /*p*/, std::size_t /*n*/) noexcept {} // NOLINT(readability-convert-member-functions-to-static)
 
-    // GCC 4.8's std::map calls these on the allocator directly instead of
-    // going through std::allocator_traits
-    std::size_t max_size() const noexcept // NOLINT(readability-convert-member-functions-to-static)
-    {
-        return (std::numeric_limits<std::size_t>::max)() / sizeof(T);
-    }
-
-    template<typename U, typename... Args>
-    void construct(U* p, Args&& ... args) // NOLINT(readability-convert-member-functions-to-static)
-    {
-        ::new (static_cast<void*>(p)) U(std::forward<Args>(args)...);
-    }
-
-    template<typename U>
-    void destroy(U* p) noexcept // NOLINT(readability-convert-member-functions-to-static)
-    {
-        static_cast<void>(p); // unused for trivially destructible types
-        p->~U();
-    }
-
     template<typename U>
     bool operator==(const failing_allocator<U>& /*other*/) const noexcept
     {
@@ -304,7 +270,7 @@ TEST_CASE("json_view")
                     "\"", "\"abc", "\"\\x\"", "\"\\u12\"", "\"\\ud800\"", "\"\\udc00\"", "\"\x01\"", "\"\xff\"", "\"\xc3\"", "[1]x", "/", "/*", "[\n  1,\n  x\n]", // NOLINT(modernize-raw-string-literal)
                     "1e400", "-1e400", "[1.7976931348623159e308]", "{\"a\":\n{\"b\": [1, 2,\n 3 x]}}",
                     // (an error after every kind of value: the error is found by replaying the parser)
-                    R"([null, true, -1, 1, 1.5, "s", {"a": 1} 2])", R"({"a": null, "b": -1, "c": 1.5, "d": "s", "e": {}, 1})"
+                    "[null, true, -1, 1, 1.5, \"s\", {\"a\": 1} 2]", "{\"a\": null, \"b\": -1, \"c\": 1.5, \"d\": \"s\", \"e\": {}, 1}"
                 })
         {
             CAPTURE(text)
@@ -1590,7 +1556,7 @@ TEST_CASE("json_view values")
         CHECK_THROWS_WITH_AS(not_an_object.root().get<view_map>(), json_view_test::message("type_error.302", "", 0, 5, "type must be object, but is array").c_str(), json::type_error&);
 #if !defined(JSON_NOEXCEPTION)
         // (and one that cannot allocate its nodes)
-        using failing_map = std::map<std::string, json_view, std::less<std::string>, failing_allocator<std::pair<const std::string, json_view>>>; // NOLINT(modernize-use-transparent-functors)
+        using failing_map = std::map<std::string, json_view, std::less<std::string>, failing_allocator<std::pair<const std::string, json_view>>>;
         const json_document one_member = json_document::parse(R"({"a":1})");
         CHECK_THROWS_AS(one_member.root().get<failing_map>(), std::bad_alloc&);
 #endif

tests/src/unit-locale-cpp.cpp

@@ -525,7 +525,7 @@ struct locale_switching_token
         {
             switched = std::setlocale(LC_NUMERIC, switch_to.c_str()) != nullptr;
         }
-        return &text[0]; // NOLINT(readability-container-data-pointer): std::string::data() is const before C++17
+        return &text[0];
     }
 
     std::string text;
@@ -647,7 +647,7 @@ TEST_CASE("convert_float_locale_aware")
 
             // "C" -> comma locale: strtod stops at the '.', so the conversion is repeated
             {
-                locale_switching_token token("1.5", comma_locale);
+                locale_switching_token token("1.5", comma_locale.c_str());
                 double value = 0;
                 nlohmann::detail::convert_float_locale_aware(token, 1, value);
                 if (token.switched)
@@ -660,7 +660,7 @@ TEST_CASE("convert_float_locale_aware")
             // the same for the other floating-point types
             REQUIRE(std::setlocale(LC_NUMERIC, "C") != nullptr);
             {
-                locale_switching_token token("2.25", comma_locale);
+                locale_switching_token token("2.25", comma_locale.c_str());
                 float value = 0;
                 nlohmann::detail::convert_float_locale_aware(token, 1, value);
                 if (token.switched)
@@ -670,7 +670,7 @@ TEST_CASE("convert_float_locale_aware")
             }
             REQUIRE(std::setlocale(LC_NUMERIC, "C") != nullptr);
             {
-                locale_switching_token token("2.25", comma_locale);
+                locale_switching_token token("2.25", comma_locale.c_str());
                 long double value = 0;
                 nlohmann::detail::convert_float_locale_aware(token, 1, value);
                 if (token.switched)

tests/src/unit-msgpack-cpp17.cpp

@@ -122,8 +122,7 @@ TEST_CASE("JSON text with std::byte")
     SECTION("invalid text")
     {
         CHECK_FALSE(json::accept(to_bytes("[1,\n2,")));
-        json _;
+        CHECK_THROWS_WITH_AS(json::parse(to_bytes("[1,\n2,")),
-        CHECK_THROWS_WITH_AS(_ = json::parse(to_bytes("[1,\n2,")),
                              "[json.exception.parse_error.101] parse error at line 2, column 3: syntax error while parsing value - unexpected end of input; expected '[', '{', or a literal", json::parse_error&);
     }
 }

tests/src/unit-ordered_json.cpp

@@ -541,7 +541,6 @@ TEST_CASE("object_key_index")
         };
 
         std::vector<std::size_t> hashes;
-        hashes.reserve(initial);
         for (std::size_t i = 0; i < initial; ++i)
         {
             hashes.push_back(hash_of(i));

tests/src/unit-regression1.cpp

@@ -1533,7 +1533,7 @@ TEST_CASE("regression tests 1")
         {
             my_json::object_t top = orig.get_ref<const my_json::object_t&>();
             top.at("x") = my_json::parse(R"({"a":{"b":1},"n":null,"arr":[10,20],"objs":[{"m":5},{"m":6}]})");
-            return my_json(std::move(top)); // NOLINT(modernize-return-braced-init-list): {top} would be an array
+            return my_json(std::move(top));
         };
 
         SECTION("existing members")

tests/src/unit-stream-bulk-scan.cpp

@@ -120,7 +120,6 @@ class oversized_get_area_streambuf : public std::streambuf
   private:
     std::array<char, 8> m_data{{'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'}};
 };
-
 #endif
 
 // how a test reads from the stream
@@ -464,18 +463,15 @@ TEST_CASE("parsing from a std::istream scans the get area in bulk")
         CHECK(token.empty());
     }
 
-#if SIZE_MAX > UINT32_MAX
-    // a get area larger than an int needs a 64-bit std::size_t (checked by the
-    // preprocessor, as a constant condition triggers MSVC's C4127)
     SECTION("skipping more characters than gbump() takes at once")
     {
         // gbump() takes an int, but a get area can be larger
-        const std::size_t claimed = std::size_t{3} << 30u;
+        const std::size_t claimed = sizeof(std::size_t) < 8 ? 0 : std::size_t{3} << 30u;
         oversized_get_area_streambuf buf(claimed);
-        if (static_cast<std::size_t>(buf.in_avail()) != claimed)
+        if (claimed == 0 || static_cast<std::size_t>(buf.in_avail()) != claimed)
         {
-            // a standard library whose streambuf stores the size of the get
+            // a 32-bit size_t, or a standard library whose streambuf stores
-            // area in an int (MSVC)
+            // the size of the get area in an int (MSVC)
             MESSAGE("a get area larger than an int is not supported");
         }
         else
@@ -495,7 +491,6 @@ TEST_CASE("parsing from a std::istream scans the get area in bulk")
             buf.restore_get_area();
         }
     }
-#endif
 #endif
 
     SECTION("SAX parsing from a stream")