Compare commits

..
Author SHA1 Message Date
Niels Lohmann 069057c43b Fix implicit-conversion sanitizer failure in unit-class_parser.cpp
Constructing a std::string from uint8_t iterators converts values >= 0x80
implicitly to char, which ci_test_clang_sanitizer rejects. Convert each
byte with an explicit static_cast.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-29 13:31:46 +01:00
Niels Lohmann 2aecd86b37 Document the contract of object keys passed to json_sax::key and key callbacks (#5956)
* Document the contract of object keys passed to json_sax::key and key callbacks

Describe what the name passed to json_sax::key and to a parser callback
for parse_event_t::key holds when parsing JSON text: escapes are decoded
to UTF-8, different spellings of a name yield identical strings, the name
is validated before the call (invalid escapes or UTF-8 give a parse error
and no key event), no normalization or case conversion is applied, the
event precedes the ':' check, and duplicate names are reported once per
occurrence. Also note that sax_parse passes binary-format keys unchanged.

See discussion #5955.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Add a test for the object key contract and correct the BON8 note

Pin the guarantees documented for json_sax::key and the key event of
parser callbacks: decoded names, validation before delivery, delivery
before the colon, one event per repeated name, and the binary formats.
BON8 rejects ill-formed UTF-8 in a name even with sax_parse, so the
note no longer lists it with the formats that pass bytes through.

See discussion #5955.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix the GCC build of the object key contract test

GCC 11 in C++11 mode rejects a default argument on a lambda parameter
(-Wc++14-extensions) and ignores a static_cast<void> on the
warn_unused_result of parse(). Pass the input format explicitly and bind
the parse result to a variable.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Add version history for the object key contract

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-29 07:37:33 +01:00
Niels Lohmann 5eccbbd2bc Fix clang-tidy misc-redundant-expression in unit-json_view_image.cpp (#5958)
Split the combined has_save/has_load static_asserts so clang-tidy no
longer sees both sides of && as equivalent (develop ci_clang_tidy failure).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-29 07:36:11 +01:00
12 changed files with 204 additions and 446 deletions

No files matched your search

-5
View File
@@ -154,11 +154,6 @@ and each page's title (H1) and declaration by
[`docs/docset/generate_docset.py`](https://github.com/nlohmann/json/blob/develop/docs/docset/generate_docset.py);
`make build` reports API pages that cannot be classified.
A pull request that changes only the documentation, Markdown files, GitHub metadata, or license files skips the
compiler workflows (Ubuntu, macOS, Windows); the documentation and license checks still run. See
[`tools/check_path_filters`](https://github.com/nlohmann/json/tree/develop/tools/check_path_filters) for the list of
skipped files and what to do when a compiler job starts reading one of them.
### Amalgamate the source code
The single-header files
-19
View File
@@ -7,25 +7,6 @@ on:
- master
- release/*
pull_request:
# Skip pull requests that change only files no job here reads. The list is checked by
# tools/check_path_filters/check_path_filters.py (repository_checks.yml); change both together.
paths:
- '**'
- '!docs/**'
- '!**.md'
- '!.github/*'
- '!.github/ISSUE_TEMPLATE/**'
- '!.github/external_ci/**'
- '!.github/workflows/**'
- '!.reuse/**'
- '!LICENSES/**'
- '!CITATION.cff'
- '!REUSE.toml'
# read by tools/check_build_options in the ci_meson_install job (ubuntu.yml)
- 'docs/mkdocs/docs/integration/package_managers.md'
# built by the swiftpm job (macos.yml)
- 'docs/mkdocs/docs/integration/swift/**'
- '.github/workflows/macos.yml'
workflow_dispatch:
concurrency:
-78
View File
@@ -1,78 +0,0 @@
name: Repository checks
# Checks that concern files the compiler workflows (ubuntu.yml, macos.yml,
# windows.yml) skip with their path filter: the documentation, the license
# information, and the path filter itself. This workflow has no path filter.
on:
push:
branches:
- develop
- master
- release/*
pull_request:
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.ref || github.run_id }}
cancel-in-progress: true
jobs:
check_path_filters:
runs-on: ubuntu-26.04
steps:
- name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1
with:
egress-policy: audit
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Check the path filters of the compiler workflows
run: python3 tools/check_path_filters/check_path_filters.py
ci_reuse_compliance:
runs-on: ubuntu-26.04
steps:
- name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1
with:
egress-policy: audit
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ci_reuse_compliance
ci_test_documentation:
runs-on: ubuntu-26.04
strategy:
matrix:
target: [ci_test_examples, ci_test_build_documentation, ci_test_documentation_mermaid]
steps:
- name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1
with:
egress-policy: audit
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# the git-revision-date-localized plugin needs the history; a shallow clone makes the strict build fail
fetch-depth: ${{ matrix.target == 'ci_test_build_documentation' && '0' || '1' }}
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
if: matrix.target == 'ci_test_documentation_mermaid'
with:
node-version: 24
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ${{ matrix.target }}
+26 -20
View File
@@ -7,25 +7,6 @@ on:
- master
- release/*
pull_request:
# Skip pull requests that change only files no job here reads. The list is checked by
# tools/check_path_filters/check_path_filters.py (repository_checks.yml); change both together.
paths:
- '**'
- '!docs/**'
- '!**.md'
- '!.github/*'
- '!.github/ISSUE_TEMPLATE/**'
- '!.github/external_ci/**'
- '!.github/workflows/**'
- '!.reuse/**'
- '!LICENSES/**'
- '!CITATION.cff'
- '!REUSE.toml'
# read by tools/check_build_options in the ci_meson_install job (ubuntu.yml)
- 'docs/mkdocs/docs/integration/package_managers.md'
# built by the swiftpm job (macos.yml)
- 'docs/mkdocs/docs/integration/swift/**'
- '.github/workflows/ubuntu.yml'
workflow_dispatch:
permissions:
@@ -128,7 +109,7 @@ jobs:
runs-on: ubuntu-26.04
strategy:
matrix:
target: [ci_test_single_header, ci_cppcheck, ci_cpplint, ci_reproducible_tests, ci_non_git_tests, ci_offline_testdata, ci_test_valgrind]
target: [ci_test_single_header, ci_cppcheck, ci_cpplint, ci_reproducible_tests, ci_non_git_tests, ci_offline_testdata, ci_reuse_compliance, ci_test_valgrind]
steps:
- name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1
@@ -468,6 +449,31 @@ jobs:
- name: Build
run: cmake --build build
ci_test_documentation:
runs-on: ubuntu-26.04
strategy:
matrix:
target: [ci_test_examples, ci_test_build_documentation, ci_test_documentation_mermaid]
steps:
- name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1
with:
egress-policy: audit
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# the git-revision-date-localized plugin needs the history; a shallow clone makes the strict build fail
fetch-depth: ${{ matrix.target == 'ci_test_build_documentation' && '0' || '1' }}
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
if: matrix.target == 'ci_test_documentation_mermaid'
with:
node-version: 24
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ${{ matrix.target }}
# Native Linux ARM64 runner with the latest GCC. The ubuntu-24.04-arm label is
# only available for public repositories.
ci_test_arm64:
-19
View File
@@ -7,25 +7,6 @@ on:
- master
- release/*
pull_request:
# Skip pull requests that change only files no job here reads. The list is checked by
# tools/check_path_filters/check_path_filters.py (repository_checks.yml); change both together.
paths:
- '**'
- '!docs/**'
- '!**.md'
- '!.github/*'
- '!.github/ISSUE_TEMPLATE/**'
- '!.github/external_ci/**'
- '!.github/workflows/**'
- '!.reuse/**'
- '!LICENSES/**'
- '!CITATION.cff'
- '!REUSE.toml'
# read by tools/check_build_options in the ci_meson_install job (ubuntu.yml)
- 'docs/mkdocs/docs/integration/package_managers.md'
# built by the swiftpm job (macos.yml)
- 'docs/mkdocs/docs/integration/swift/**'
- '.github/workflows/windows.yml'
workflow_dispatch:
permissions:
@@ -63,6 +63,15 @@ called:
Whether the JSON value which called the function during parsing should be kept (`#!cpp true`) or not (`#!cpp false`). In
the latter case, it is skipped completely, or replaced by `null` if it is the top-level value.
## Notes
For a `parse_event_t::key` event, `parsed` holds the decoded object name, with the same guarantees as the `val`
parameter of [`json_sax::key`](../json_sax/key.md#notes): all escape sequences are decoded and `\uXXXX` escapes are
stored as UTF-8, so different spellings of the same name yield identical strings; the name is validated completely
before the callback is called, so a name with an invalid escape sequence or ill-formed UTF-8 causes a parse error and
no `key` event; no Unicode normalization or case conversion is applied; and a name that occurs several times in the
same object yields one `key` event per occurrence.
## Examples
??? example "Example: skip an object key while parsing"
+21
View File
@@ -19,6 +19,23 @@ Whether parsing should proceed.
It is safe to move the passed object key value.
When parsing JSON text, `val` is the decoded object name, not its spelling in the input:
- All escape sequences (`\"`, `\\`, `\/`, `\b`, `\f`, `\n`, `\r`, `\t`, `\uXXXX`, and surrogate pairs) are
decoded, and `\uXXXX` escapes are stored as UTF-8. Different spellings of the same name, such as `#!json "/"`
and `#!json "\/"`, or `#!json "\u00e9"` and `#!json "\u00E9"`, yield identical strings. An escaped `\u0000` yields
an embedded NUL byte.
- The name is scanned and validated completely before `key` is called. If it contains an invalid escape sequence, an
unpaired surrogate, a control character, or ill-formed UTF-8, the parse fails with a
[`parse_error`](../../home/exceptions.md#parse-errors), and `key` is not called for it.
- No Unicode normalization or case conversion is applied: names that differ in any byte are different strings.
- `key` is called before the following `:` is read, so a parse error may still follow for the same member.
- A name that occurs several times in the same object is passed once per occurrence.
For the binary formats, object keys contain no escape sequences. For CBOR, MessagePack, UBJSON, and BSON, `val` holds
the bytes stored in the input, and [`sax_parse`](../basic_json/sax_parse.md) passes them unchanged, without checking them
for valid UTF-8. BON8 encodes characters itself, so an ill-formed name is a parse error and `key` is not called for it.
## Examples
??? example
@@ -45,3 +62,7 @@ It is safe to move the passed object key value.
## Version history
- Added in version 3.2.0.
- The behavior described in the notes above (decoded names, validation before `key` is called, no normalization, `key`
before the `:` check, one call per duplicate, bytes of binary formats passed unchanged) is the same in all versions
from 3.2.0 on, and for the `key` event of a [parser callback](../basic_json/parser_callback_t.md) from 3.0.0 on. BSON
is only supported from version 3.4.0, and BON8 only from version 3.13.0.
@@ -64,6 +64,9 @@ bool parse_error(std::size_t position, const std::string& last_token, const json
The return value of each function determines whether parsing should proceed.
Object keys are passed to `key` decoded and validated: escape sequences are already resolved, so different spellings of
the same name yield the same string. See [`key`](../../api/json_sax/key.md#notes) for the exact guarantees.
## Stepping through the events
The events for a small input, step by step, and what two different handlers do with them. Press *Play* or step with *Next*:
+141
View File
@@ -3030,3 +3030,144 @@ TEST_CASE("a json_sax_dom_parser keeps the values read before a parse error")
CHECK(read_before_error<nlohmann::ordered_json>(c.first) == c.second);
}
}
TEST_CASE("object keys are delivered decoded and validated")
{
// json_sax::key and the key event of a parser callback see the decoded name, and
// only after the whole name was scanned and validated
const auto keys_of_format = [](const std::string & text, const json::input_format_t format)
{
SaxEventLogger l;
json::sax_parse(text, &l, format);
std::vector<std::string> keys;
for (const auto& e : l.events)
{
if (e.compare(0, 4, "key(") == 0)
{
keys.push_back(e.substr(4, e.size() - 5));
}
}
return keys;
};
const auto keys_of = [&keys_of_format](const std::string & text)
{
return keys_of_format(text, json::input_format_t::json);
};
const auto callback_keys_of = [](const std::string & text)
{
std::vector<std::string> keys;
const json::parser_callback_t cb = [&keys](int /*depth*/, json::parse_event_t event, json & parsed)
{
if (event == json::parse_event_t::key)
{
keys.push_back(parsed.get<std::string>());
}
return true;
};
const json parsed = json::parse(text, cb, false);
static_cast<void>(parsed);
return keys;
};
SECTION("different spellings of a name yield the same string")
{
const std::string e_acute = "\xc3\xa9";
const std::string text = std::string(R"({"/":1,"\/":2,"\u002f":3,"\u00e9":4,"\u00E9":5,")") + e_acute + R"(":6})";
const std::vector<std::string> expected = {"/", "/", "/", e_acute, e_acute, e_acute};
CHECK(keys_of(text) == expected);
CHECK(callback_keys_of(text) == expected);
}
SECTION("escape sequences are decoded")
{
const std::string simple = R"({"\"\\\/\b\f\n\r\t":1})";
const std::vector<std::string> simple_expected = {std::string("\"\\/\b\f\n\r\t")};
CHECK(keys_of(simple) == simple_expected);
CHECK(callback_keys_of(simple) == simple_expected);
const std::string nul = R"({"\u0000":1})";
const std::vector<std::string> nul_expected = {std::string(1, '\0')};
CHECK(keys_of(nul) == nul_expected);
CHECK(callback_keys_of(nul) == nul_expected);
const std::string pair = R"({"\ud83d\ude00":1})";
const std::vector<std::string> pair_expected = {"\xf0\x9f\x98\x80"};
CHECK(keys_of(pair) == pair_expected);
CHECK(callback_keys_of(pair) == pair_expected);
}
SECTION("no normalization and no case conversion")
{
const std::string text = std::string("{\"\xc3\xa9\":1,\"e\xcc\x81\":2,\"A\":3,\"a\":4}");
const std::vector<std::string> expected = {"\xc3\xa9", "e\xcc\x81", "A", "a"};
CHECK(keys_of(text) == expected);
CHECK(callback_keys_of(text) == expected);
}
SECTION("a name that is not valid is not delivered")
{
const std::vector<std::string> texts =
{
R"({"\u12G4":1})", // malformed escape
R"({"\udc00":1})", // lone low surrogate
R"({"\ud800":1})", // lone high surrogate
R"({"\ud800\u0041":1})", // high surrogate not followed by a low one
R"({"\x":1})", // unknown escape
std::string("{\"a\x01\":1}"), // control character
std::string("{\"\xff\":1}") // ill-formed UTF-8
};
for (const auto& text : texts)
{
CAPTURE(text)
CHECK(!json::accept(text));
CHECK(keys_of(text).empty());
CHECK(callback_keys_of(text).empty());
}
}
SECTION("a name is delivered before the following colon is read")
{
const std::string text = R"({"a" 1})";
const std::vector<std::string> expected = {"a"};
CHECK(!json::accept(text));
CHECK(keys_of(text) == expected);
CHECK(callback_keys_of(text) == expected);
}
SECTION("a repeated name is delivered once per occurrence")
{
const std::string text = R"({"a":1,"a":2})";
const std::vector<std::string> expected = {"a", "a"};
CHECK(keys_of(text) == expected);
CHECK(callback_keys_of(text) == expected);
}
SECTION("binary formats pass the stored bytes through")
{
const std::string stored = "\xff\\";
const std::vector<std::string> expected = {stored};
const auto keys_of_bytes = [&keys_of_format](const std::vector<std::uint8_t>& v, json::input_format_t format)
{
std::string text;
for (const auto byte : v)
{
text.push_back(static_cast<char>(byte));
}
return keys_of_format(text, format);
};
CHECK(keys_of_bytes({0xA1, 0x62, 0xFF, 0x5C, 0x01}, json::input_format_t::cbor) == expected);
CHECK(keys_of_bytes({0x81, 0xA2, 0xFF, 0x5C, 0x01}, json::input_format_t::msgpack) == expected);
CHECK(keys_of_bytes({'{', 'i', 2, 0xFF, 0x5C, 'i', 1, '}'}, json::input_format_t::ubjson) == expected);
CHECK(keys_of_bytes({0x0D, 0, 0, 0, 0x10, 0xFF, 0x5C, 0, 1, 0, 0, 0, 0}, json::input_format_t::bson) == expected);
}
SECTION("BON8 rejects ill-formed UTF-8 in a name")
{
const std::vector<std::string> valid = {"\\"};
CHECK(keys_of_format(std::string("\x87\x5C\x91"), json::input_format_t::bon8) == valid);
CHECK(keys_of_format(std::string("\x87\xC3\x28\x91"), json::input_format_t::bon8).empty());
}
}
+4 -2
View File
@@ -185,8 +185,10 @@ std::uint32_t rng()
TEST_CASE("json_view images: round trips")
{
static_assert(json_view_image_test::has_save<json_document>::value && json_view_image_test::has_load<json_document>::value, "save() and load() on a little-endian target");
static_assert(json_view_image_test::has_save<ordered_json_editable_document>::value && json_view_image_test::has_load<ordered_json_editable_document>::value, "save() and load() on a little-endian target");
static_assert(json_view_image_test::has_save<json_document>::value, "save() on a little-endian target");
static_assert(json_view_image_test::has_load<json_document>::value, "load() on a little-endian target");
static_assert(json_view_image_test::has_save<ordered_json_editable_document>::value, "save() on a little-endian target");
static_assert(json_view_image_test::has_load<ordered_json_editable_document>::value, "load() on a little-endian target");
SECTION("small documents")
{
-34
View File
@@ -1,34 +0,0 @@
# check_path_filters
The compiler workflows [`ubuntu.yml`](../../.github/workflows/ubuntu.yml),
[`macos.yml`](../../.github/workflows/macos.yml), and [`windows.yml`](../../.github/workflows/windows.yml) skip pull
requests that change only files no compiler job reads: the documentation, Markdown files, GitHub metadata, other
workflows, and the license files. Pushes to `develop`, `master`, and `release/*` always run them. The documentation,
license, and path-filter checks run in [`repository_checks.yml`](../../.github/workflows/repository_checks.yml), which
has no path filter.
The filter starts with `'**'` and excludes these files explicitly, so a new file runs the compiler workflows unless it
is excluded on purpose. [`check_path_filters.py`](check_path_filters.py) holds the filter (`PATHS`) and checks that
- each of the three workflows has exactly this filter, with its own workflow file included,
- the CI targets that read excluded files (`UNFILTERED_TARGETS`) do not run in these workflows, and
- no file the filter keeps (sources, tests, build files, tools, workflows) mentions an excluded file or directory
outside a comment line, unless the mention is listed in `KNOWN_REFERENCES` with a reason.
When the check reports a mention, decide whether a compiler job reads that file:
- If it does, include the file in `PATHS` after the exclusions (as `package_managers.md` and the Swift example are) and
copy the new filter into the three workflows.
- If it does not, add the mention to `KNOWN_REFERENCES` with the reason.
Run the check with
```shell
python3 tools/check_path_filters/check_path_filters.py
```
To see which workflows a pull request would run, pass the changed files:
```shell
python3 tools/check_path_filters/check_path_filters.py --changed $(git diff --name-only origin/develop)
```
@@ -1,269 +0,0 @@
#!/usr/bin/env python3
"""Check the path filters of the compiler workflows.
ubuntu.yml, macos.yml and windows.yml skip pull requests that change only files
no compiler job reads (documentation, Markdown files, GitHub metadata, license
files). The filter starts with '**' and excludes these files explicitly, so a
new file runs the workflows unless it is excluded on purpose.
The script checks that
- each of the three workflows has exactly the filter defined in PATHS below,
with its own workflow file included, and
- no tracked file that the filter keeps (sources, tests, build files, tools,
workflows) mentions a file the filter excludes, unless the mention is listed
in KNOWN_REFERENCES with the reason why it does not matter.
A mention found this way must either be added to PATHS as an included path
(the file is read by a compiler job) or to KNOWN_REFERENCES (it is not).
With --changed FILE..., the script prints instead which workflows a pull
request changing these files would run.
"""
import argparse
import os
import re
import subprocess # nosec B404
import sys
REPO_ROOT = os.path.normpath(os.path.join(sys.path[0], '..', '..'))
SELF = 'tools/check_path_filters/check_path_filters.py'
WORKFLOWS = ['.github/workflows/ubuntu.yml', '.github/workflows/macos.yml', '.github/workflows/windows.yml']
# The pull_request paths filter of the workflows above; '{workflow}' is replaced
# with the workflow's own file. GitHub applies the patterns in order: a file
# matching a later pattern is included ('pattern') or excluded ('!pattern').
PATHS = [
'**',
'!docs/**',
'!**.md',
'!.github/*',
'!.github/ISSUE_TEMPLATE/**',
'!.github/external_ci/**',
'!.github/workflows/**',
'!.reuse/**',
'!LICENSES/**',
'!CITATION.cff',
'!REUSE.toml',
# read by tools/check_build_options in the ci_meson_install job (ubuntu.yml)
'docs/mkdocs/docs/integration/package_managers.md',
# built by the swiftpm job (macos.yml)
'docs/mkdocs/docs/integration/swift/**',
'{workflow}',
]
# Mentions of excluded files that no job of the three workflows depends on:
# (file that mentions it, the mention, reason)
KNOWN_REFERENCES = [
('cmake/ci.cmake', 'docs/', 'ci_test_examples; runs in repository_checks.yml, which has no path filter'),
('cmake/ci.cmake', 'docs/mkdocs/', 'ci_test_build_documentation and ci_test_documentation_mermaid; they run in '
'repository_checks.yml, which has no path filter'),
('cmake/ci.cmake', 'docs/mkdocs/docs/examples/', 'indentation check of ci_test_amalgamation; the same check runs in '
'check_amalgamation.yml, which has no path filter'),
('Makefile', 'docs/mkdocs/docs/examples/', 'make pretty; runs in check_amalgamation.yml, which has no path filter'),
('Makefile', 'ChangeLog.md', 'make ChangeLog.md writes the file; no CI job runs it'),
('Makefile', 'tests/fuzzing.md', 'help text'),
('Makefile', 'tools/generate_natvis/README.md', 'help text'),
('tests/src/make_test_data_available.hpp', 'README.md', 'a file of the external test data repository'),
('tools/check_build_options/check_build_options.py', 'docs/', 'builds the path of package_managers.md, which '
'PATHS includes'),
]
# CI targets that read excluded files; they must run in a workflow without a path filter
UNFILTERED_TARGETS = ['ci_test_examples', 'ci_test_build_documentation', 'ci_test_documentation_mermaid',
'ci_test_amalgamation', 'ci_reuse_compliance']
def git_files(root):
# fixed command, no user input; includes new files not yet added, so a local run
# sees what CI sees after the commit
out = subprocess.check_output(['git', 'ls-files', '-z', '--cached', '--others', '--exclude-standard'], cwd=root) # nosec B603, B607
return [f for f in out.decode('utf-8').split('\0') if f]
def glob_to_regex(pattern):
"""Translate a GitHub Actions path pattern to a regular expression."""
regex = ''
i = 0
while i < len(pattern):
if pattern.startswith('**', i):
regex += '.*'
i += 2
elif pattern[i] == '*':
regex += '[^/]*'
i += 1
elif pattern[i] == '?':
regex += '[^/]'
i += 1
else:
regex += re.escape(pattern[i])
i += 1
return re.compile(regex + r'\Z')
def compile_filter(paths, workflow):
return [(p.startswith('!'), glob_to_regex(p.lstrip('!').replace('{workflow}', workflow))) for p in paths]
def included(compiled, path):
result = False
for negated, regex in compiled:
if regex.match(path):
result = not negated
return result
def read_filter(root, workflow):
"""Return the pull_request paths list of a workflow file."""
with open(os.path.join(root, workflow), encoding='utf-8') as f:
lines = f.read().splitlines()
paths = None
in_pull_request = False
for line in lines:
if re.match(r'^ pull_request:', line):
in_pull_request = True
continue
if in_pull_request:
if re.match(r'^ \S', line) or re.match(r'^\S', line):
break
if re.match(r'^ paths:\s*$', line):
paths = []
continue
m = re.match(r"^ - '(.*)'\s*(#.*)?$", line)
if paths is not None and m:
paths.append(m.group(1))
elif paths is not None and line.strip() and not line.strip().startswith('#'):
break
return paths
TOKEN = re.compile(r'(["\'/]?)([\w.-]+(?:/[\w.-]+)*)')
COMMENT_LINE = re.compile(r'^[ \t]*(//|#|/?\*|--).*$', re.MULTILINE)
class Mentions:
"""Finds the words and paths in a text that name an excluded file or directory."""
def __init__(self, files, excluded, is_excluded):
self.files = set(files)
self.excluded = excluded
self.dirs = {'/'.join(f.split('/')[:n]) for f in files for n in range(1, f.count('/') + 1)}
# an excluded directory: a new file in it would be excluded
self.excluded_dirs = {d for d in self.dirs if is_excluded(d + '/new_file')}
# base names that only excluded files have, e.g. 'ChangeLog.md' but not 'CMakeLists.txt'
self.names = {os.path.basename(f) for f in excluded} - {os.path.basename(f) for f in files if f not in excluded}
def find(self, text):
"""Yield (offset, mention) for every word or path in text that names an excluded file or directory."""
for m in TOKEN.finditer(text):
parts = m.group(2).rstrip('/').split('/')
# a single word names a directory only in a path context, e.g. 'docs' in os.path.join('docs', ...)
path_context = len(parts) > 1 or m.group(1)
for i in range(len(parts)):
path = '/'.join(parts[i:])
if path in self.files:
if path in self.excluded:
yield m.start(2), path
break
if path in self.dirs and path_context:
if path in self.excluded_dirs:
yield m.start(2), path + '/'
break
else:
if parts[-1] in self.names and '.' in parts[-1]:
yield m.start(2), parts[-1]
def known(path, mention):
return any(path == file and mention == known_mention for file, known_mention, _ in KNOWN_REFERENCES)
def check_workflows(root):
"""Return the errors in the filters of the workflows and in the targets they run."""
errors = []
for workflow in WORKFLOWS:
expected = [p.replace('{workflow}', workflow) for p in PATHS]
actual = read_filter(root, workflow)
if actual != expected:
errors.append('{}: the pull_request paths filter differs from tools/check_path_filters/check_path_filters.py\n'
' expected: {}\n actual: {}'.format(workflow, expected, actual))
with open(os.path.join(root, workflow), encoding='utf-8') as f:
text = f.read()
errors += ['{}: runs {}, which reads excluded files; run it in repository_checks.yml'.format(workflow, target)
for target in UNFILTERED_TARGETS if re.search(r'\b' + target + r'\b', text)]
return errors
def scanned_text(root, path):
"""Return the text of a kept file without comment lines and without the filter itself, or None to skip it."""
# third-party code and Markdown files read nothing; this script lists the excluded files
if path.startswith('tests/thirdparty/') or path.endswith('.md') or path == SELF:
return None
try:
with open(os.path.join(root, path), encoding='utf-8') as f:
# a mention in a comment line reads nothing; keep the line numbers
text = COMMENT_LINE.sub('', f.read())
except (UnicodeDecodeError, OSError):
return None
if path in WORKFLOWS:
# the filter itself lists the excluded files
for p in PATHS:
text = text.replace(" - '{}'".format(p.replace('{workflow}', path)), '')
return text
def check_mentions(root, files, excluded, finder):
"""Return an error for every mention of an excluded file in a kept file that KNOWN_REFERENCES does not list."""
errors = []
for path in sorted(set(files) - excluded):
text = scanned_text(root, path)
reported = set()
for offset, mention in finder.find(text or ''):
if mention not in reported and not known(path, mention):
reported.add(mention)
line = text.count('\n', 0, offset) + 1
errors.append('{}:{}: mentions excluded {}; include it in PATHS if a compiler job reads it, '
'or add it to KNOWN_REFERENCES'.format(path, line, mention))
return errors
def check(root):
files = git_files(root)
union = [compile_filter(PATHS, w) for w in WORKFLOWS]
def is_excluded(path):
return not any(included(c, path) for c in union)
excluded = {f for f in files if is_excluded(f)}
errors = check_workflows(root) + check_mentions(root, files, excluded, Mentions(files, excluded, is_excluded))
for e in errors:
print(e)
if not errors:
print('path filters OK: {} of {} tracked files skip the compiler workflows'.format(len(excluded), len(files)))
return 1 if errors else 0
def explain(changed):
for workflow in WORKFLOWS:
compiled = compile_filter(PATHS, workflow)
hits = [f for f in changed if included(compiled, f)]
print('{}: {}'.format(workflow, 'runs ({})'.format(', '.join(hits[:3]) + (' ...' if len(hits) > 3 else ''))
if hits else 'skipped'))
return 0
def main():
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument('--root', default=REPO_ROOT, help='repository root')
parser.add_argument('--changed', nargs='+', metavar='FILE', help='show which workflows these changed files run')
args = parser.parse_args()
return explain(args.changed) if args.changed else check(args.root)
if __name__ == '__main__':
sys.exit(main())