The SWAR string scan, the bulk number scan, and the indentation skip were
only enabled for adapters that support seeking (contiguous random-access
input), because they rely on the lazy token and position tracking that
reconstructs both from the consumed input on demand.
This decouples the bulk paths from seeking, so that an adapter can expose
a contiguous block that is only a window into its input (bulk_window):
- bulk_consume() does the bookkeeping that get() would have done for the
skipped characters when token and position are tracked eagerly.
- A number that reaches the end of a window is finished character by
character, because it may continue after the window.
- After a bulk-scanned number, a window adapter reads the terminating
character and ungets it, as scan_number() does, so that the input
handed back to the user is consumed exactly as before.
- An adapter can keep the current token itself (supports_token_tracking,
begin_token(), mark_token_start(), copy_token()) so the lexer need not
copy every character for its error messages.
No adapter declares any of this yet, so the behavior is unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Niels Lohmann
80d00eda50