With error_handler_t::strict (and replace/ignore), every string a binary
reader reads is checked for valid UTF-8, and every string a binary writer
writes. The readers used the byte-at-a-time decoder; the writers skipped
ASCII eight bytes at a time.
- New valid_utf8_string() decides strings of up to 16 ASCII bytes, most
keys and many values, with one or two word tests, and checks longer
strings 16 bytes at a time with the NEON/SSSE3 UTF-8 check the lexer
uses (vector_utf8_valid_blocks: the same lookup, but quotes and control
characters are ordinary characters). Other targets and JSON_NO_SIMD use
valid_utf8_prefix() as before.
- The readers check strings with it (for one-byte character types) in an
out-of-line function, so that reading without a check does not pay for
its code.
- The writers' check_utf8() uses it, and the CBOR and MessagePack writers
write a checked string of up to max_text_in_reservation characters in
one reservation, as they write unchecked ones.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Niels Lohmann
22d2013b13