Niels Lohmann
4c3e92b96e
The 19 macOS and 32 Windows jobs ran on every push to every pull request. They now use the same sets as the Ubuntu jobs: - PR set (16): four Visual Studio 2022 and two Visual Studio 2026 configurations, ARM64 Debug, Clang 20.1.8 with MinGW, MinGW x86, the C++20 module test, Xcode 16.0, 16.4, 26.3 and 26.6, C++14 on the latest Xcode, and the Swift Package Manager check. - full set (34): every other Visual Studio configuration, Clang version, clang-cl, Xcode version and C++ standard. - nightly set (1): 64-bit MinGW, which takes about 68 minutes. A failed nightly run opens or updates an issue. No configuration is removed: tools/ci_jobs/ci_jobs.py --compare lists 157 configurations before and after, and all jobs keep their runner, steps and environment. ci_jobs.py now checks all three workflows and expands matrices that consist only of include entries correctly. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
27 KiB
27 KiB
Quality assurance
Ensuring quality is paramount for this project, particularly because numerous other projects depend on it. Each commit to the library undergoes rigorous checks against the following requirements, and any violations will result in a failed build.
The overview shows the checks a change passes on its way from a push or pull request to a release; the sections below describe each of them.
--8<-- "images/qa_pipeline.svg"
The checks grouped by the stage at which they run, from push or pull request through the merge into develop to a release. The badge on each card says whether the check runs on every push or PR, on a schedule, or continuously.
How CI runs
Not every job runs for every change. The compiler jobs on Ubuntu, macOS, and Windows are split into three sets, so a pull request gets fast feedback and the long jobs run when they matter:
| Set | When it runs | What it contains |
|---|---|---|
| PR set | every push to a pull request, and every push to develop, master, and release/* |
GCC and Clang with all warnings, Clang-Tidy, sanitizers, IWYU, cpplint, the oldest and newest GCC and Clang, common CMake options, Meson, Emscripten, four Visual Studio 2022 and two Visual Studio 2026 configurations, MinGW, Clang on Windows, ARM64 Windows, five Xcode versions, Swift Package Manager; with the checks that always run, they would have caught at least 92 % of the failing commits of September and October 2026 |
| full set | after the PR set passed: on pull requests with the label 🚀 ready to merge, and on every push | every other compiler, version, C++ standard, and CMake option, the Clang Static Analyzer, Cppcheck, and the C++20 module tests |
| nightly set | every night on develop, on every push to master and release/*, and when started by hand |
Infer, Valgrind, coverage, the Intel and NVIDIA compilers, CUDA, ARM64 Linux, and 64-bit MinGW; a failure opens an issue |
The documentation and license checks and the amalgamation check run on every push and pull request; AppVeyor (Visual
Studio 2015, 2017, and 2019) has its own schedule. A pull request that changes only the documentation, Markdown files, GitHub metadata, or license files
skips the compiler workflows (see tools/check_path_filters).
A maintainer adds the label 🚀 ready to merge when a pull request is ready; the full set then runs, and again on every later push to that pull request. A pull request is merged only when the full set passed.
??? abstract "Jobs of the compiler workflows and their sets"
The table is generated by [`tools/ci_jobs`](https://github.com/nlohmann/json/tree/develop/tools/ci_jobs), which
also checks in CI that it is up to date and that every job belongs to exactly one set.
<!-- ci_jobs: begin -->
| Set | Workflow | Job | Configurations |
|-----|----------|-----|----------------|
| PR set | macos.yml | `macos-15` | 16.0; 16.4; 26.3 |
| PR set | macos.yml | `macos-26` | 26.6 |
| PR set | macos.yml | `xcode_standards` | 14 |
| PR set | macos.yml | `swiftpm` | - |
| PR set | ubuntu.yml | `ci_test_gcc` | - |
| PR set | ubuntu.yml | `ci_meson_install` | - |
| PR set | ubuntu.yml | `ci_static_analysis_ubuntu` | ci_cpplint |
| PR set | ubuntu.yml | `ci_static_analysis_clang` | ci_test_clang; ci_clang_tidy; ci_test_clang_sanitizer; ci_single_binaries |
| PR set | ubuntu.yml | `ci_cmake_options` | ci_test_diagnostics; ci_test_diagnostic_positions; ci_test_noexceptions; ci_test_noglobaludls |
| PR set | ubuntu.yml | `ci_test_compilers_gcc_old` | 4.8 |
| PR set | ubuntu.yml | `ci_test_compilers_clang` | 3.4 |
| PR set | ubuntu.yml | `ci_test_standards_gcc` | 26 |
| PR set | ubuntu.yml | `ci_test_standards_clang` | 26, libcxx |
| PR set | ubuntu.yml | `ci_emscripten` | - |
| PR set | windows.yml | `mingw` | x86 |
| PR set | windows.yml | `msvc` | Debug, x64, default; Release, x64, default; Debug, Win32, default; Release, Win32, latest |
| PR set | windows.yml | `msvc-vs2026` | Debug, x64, latest; Release, Win32, default |
| PR set | windows.yml | `msvc-arm64` | Debug |
| PR set | windows.yml | `clang` | 20.1.8 |
| PR set | windows.yml | `ci_module_cpp20` | - |
| full set | macos.yml | `macos-15_full` | 16.1; 16.2; 16.3; 26.0.1; 26.1.1; 26.2 |
| full set | macos.yml | `macos-26_full` | 26.4.1; 26.5 |
| full set | macos.yml | `xcode_standards_full` | 11; 17; 20; 23; 26 |
| full set | ubuntu.yml | `ci_static_analysis_ubuntu_full` | ci_test_single_header; ci_cppcheck; ci_reproducible_tests; ci_non_git_tests; ci_offline_testdata |
| full set | ubuntu.yml | `ci_static_analysis_clang_full` | ci_clang_analyze |
| full set | ubuntu.yml | `ci_cmake_options_full` | ci_cmake_flags; ci_test_noimplicitconversions; ci_test_legacycomparison; ci_test_disableenumserialization; ci_test_disabletuplereferenceconversion; ci_test_skiplibraryversioncheck; ci_test_simdutf; ci_test_strict_nul_handling; ci_test_delete_deprecated_functions; ci_test_no_thread_local |
| full set | ubuntu.yml | `ci_test_compilers_gcc_old_full` | 4.9; 5; 6 |
| full set | ubuntu.yml | `ci_test_compilers_gcc_full` | 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; latest |
| full set | ubuntu.yml | `ci_test_compilers_clang_full` | 3.5; 3.6; 3.7; 3.8; 3.9; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15-bullseye; 16; 17; 18; 19; 20; 21; 22; latest |
| full set | ubuntu.yml | `ci_test_standards_gcc_full` | 11; 14; 17; 20; 23 |
| full set | ubuntu.yml | `ci_test_standards_clang_full` | 11, libcxx; 11, libstdcxx; 14, libcxx; 14, libstdcxx; 17, libcxx; 17, libstdcxx; 20, libcxx; 20, libstdcxx; 23, libcxx; 23, libstdcxx; 26, libstdcxx |
| full set | ubuntu.yml | `ci_module_cpp20` | gcc:latest; silkeh/clang:latest |
| full set | windows.yml | `msvc_full` | Debug, Win32, latest; Debug, x64, latest; Release, Win32, default; Release, x64, latest |
| full set | windows.yml | `msvc-vs2026_full` | Debug, Win32, default; Debug, Win32, latest; Debug, x64, default; Release, Win32, latest; Release, x64, default; Release, x64, latest |
| full set | windows.yml | `msvc-arm64_full` | Release |
| full set | windows.yml | `clang_full` | 11.0.1; 12.0.1; 13.0.1; 14.0.6; 15.0.7; 16.0.6; 18.1.8; 19.1.7 |
| full set | windows.yml | `clang-cl_full` | Win32; x64 |
| nightly set | ubuntu.yml | `ci_infer` | - |
| nightly set | ubuntu.yml | `ci_static_analysis_ubuntu_nightly` | ci_test_valgrind |
| nightly set | ubuntu.yml | `ci_test_coverage` | - |
| nightly set | ubuntu.yml | `ci_cuda_example` | 11.8.0; 12.1.1; 12.6.3 |
| nightly set | ubuntu.yml | `ci_icpc` | - |
| nightly set | ubuntu.yml | `ci_icpx` | - |
| nightly set | ubuntu.yml | `ci_nvhpc` | - |
| nightly set | ubuntu.yml | `ci_test_arm64` | - |
| nightly set | windows.yml | `mingw_nightly` | x64 |
<!-- ci_jobs: end -->
C++ language compliance and compiler compatibility
!!! success "Requirement: Compiler support"
Any compiler with complete C++11 support can compile the library without warnings.
Note: C++20 modules support may hit compiler-specific issues not covered by the general compiler matrix below. See Modules for known issues and workarounds.
Note: Some modern features (like C++20 ranges or filesystem support) may be disabled on specific broken or incomplete toolchains even when standard feature-test macros indicate support. See JSON_HAS_RANGES, JSON_HAS_RANGE_VIEW_CONVERSION, and JSON_HAS_FILESYSTEM for details on known exclusions.
-
The library is compiled with 50+ different C++ compilers with different operating systems and platforms, including the oldest versions known to compile the library.
??? abstract "Compilers used in continuous integration"
| Compiler | Architecture | Operating System | CI | |----------------------------------------------|--------------|-----------------------------------|-----------| | AppleClang 16.0.0.16000026; Xcode 16 | arm64 | macOS 15.2 (Sequoia) | GitHub | | AppleClang 16.0.0.16000026; Xcode 16.1 | arm64 | macOS 15.2 (Sequoia) | GitHub | | AppleClang 16.0.0.16000026; Xcode 16.2 | arm64 | macOS 15.2 (Sequoia) | GitHub | | AppleClang 17.0.0.17000013; Xcode 16.3 | arm64 | macOS 15.5 (Sequoia) | GitHub | | AppleClang 17.0.0.17000013; Xcode 16.4 | arm64 | macOS 15.5 (Sequoia) | GitHub | | AppleClang 17.0.0.17000319; Xcode 26.0.1 | arm64 | macOS 15.5 (Sequoia) | GitHub | | AppleClang 17.0.0.17000404; Xcode 26.1.1 | arm64 | macOS 15.7.9 (Sequoia) | GitHub | | AppleClang 17.0.0.17000603; Xcode 26.2 | arm64 | macOS 15.7.9 (Sequoia) | GitHub | | AppleClang 17.0.0.17000604; Xcode 26.3 | arm64 | macOS 15.7.9 (Sequoia) | GitHub | | AppleClang 21.0.0.21000099; Xcode 26.4.1 | arm64 | macOS 26.6.2 (Tahoe) | GitHub | | AppleClang 21.0.0.21000101; Xcode 26.5 | arm64 | macOS 26.6.2 (Tahoe) | GitHub | | AppleClang 21.0.0.21000101; Xcode 26.6 | arm64 | macOS 26.6.2 (Tahoe) | GitHub | | Clang 3.4.2 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 3.5.2 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 3.6.2 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 3.7.1 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 3.8.1 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 3.9.1 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 4.0.1 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 5.0.2 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 6.0.1 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 7.1.0 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 8.0.1 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 9.0.1 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 10.0.1 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 11.0.1 with GNU-like command-line | x86_64 | Windows Server 2022 (Build 20348) | GitHub | | Clang 11.1.0 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 12.0.1 with GNU-like command-line | x86_64 | Windows Server 2022 (Build 20348) | GitHub | | Clang 12.0.1 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 13.0.1 with GNU-like command-line | x86_64 | Windows Server 2022 (Build 20348) | GitHub | | Clang 13.0.1 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 14.0.6 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 14.0.6 with GNU-like command-line | x86_64 | Windows Server 2022 (Build 20348) | GitHub | | Clang 15.0.7 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 15.0.7 with GNU-like command-line | x86_64 | Windows Server 2022 (Build 20348) | GitHub | | Clang 16.0.6 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 16.0.6 with GNU-like command-line | x86_64 | Windows Server 2022 (Build 20348) | GitHub | | Clang 17.0.6 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 18.1.8 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 18.1.8 with GNU-like command-line | x86_64 | Windows Server 2022 (Build 20348) | GitHub | | Clang 19.1.5 with MSVC-like command-line | x86_64 | Windows Server 2022 (Build 20348) | GitHub | | Clang 19.1.7 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 19.1.7 with GNU-like command-line | x86_64 | Windows Server 2022 (Build 20348) | GitHub | | Clang 20.1.1 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 20.1.8 with GNU-like command-line | x86_64 | Windows Server 2022 (Build 20348) | GitHub | | Clang 21.1.8 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | Clang 22.1.8 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | CUDA 11.8.0 (nvcc) | x86_64 | Ubuntu 22.04 LTS | GitHub | | CUDA 12.1.1 (nvcc) | x86_64 | Ubuntu 22.04 LTS | GitHub | | CUDA 12.6.3 (nvcc) | x86_64 | Ubuntu 22.04 LTS | GitHub | | Emscripten 4.0.6 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | GNU 4.8.5 | x86_64 | Ubuntu 20.04 LTS | GitHub | | GNU 4.9.3 | x86_64 | Ubuntu 20.04 LTS | GitHub | | GNU 5.5.0 | x86_64 | Ubuntu 20.04 LTS | GitHub | | GNU 6.4.0 | x86_64 | Ubuntu 20.04 LTS | GitHub | | GNU 7.5.0 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | GNU 8.5.0 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | GNU 9.3.0 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | GNU 9.4.0 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | GNU 9.5.0 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | GNU 10.5.0 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | GNU 11.4.0 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | GNU 11.5.0 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | GNU 12.2.0 (MinGW-W64 i686-ucrt-posix-dwarf) | x86_64 | Windows Server 2022 (Build 20348) | GitHub | | GNU 12.2.0 (MinGW-W64 x86_64-ucrt-posix-seh) | x86_64 | Windows Server 2022 (Build 20348) | GitHub | | GNU 12.4.0 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | GNU 13.3.0 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | GNU 14.2.0 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | GNU 15.1.0 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | GNU 16.2.0 | x86_64 | Ubuntu 22.04.1 LTS | GitHub | | GNU 16.1.0 | arm64 | Ubuntu 24.04 | GitHub | | icpc (ICC) 2021.10.0 20230609 | x86_64 | Ubuntu 22.04 LTS | GitHub | | icpx (Intel oneAPI DPC++/C++) 2025.3.2 | x86_64 | Ubuntu 24.04 LTS | GitHub | | nvc++ (NVIDIA HPC SDK) 25.5-0 | x86_64 | Ubuntu 22.04 LTS | GitHub | | MSVC 19.0.24241.7 | x86 | Windows 8.1 | AppVeyor | | MSVC 19.16.27035.0 | x86 | Windows-10 (Build 14393) | AppVeyor | | MSVC 19.29.30157.0 | x86 | Windows-10 (Build 17763) | AppVeyor | | MSVC 19.44.35207.0 | arm64 | Windows 11 (Build 26200) | GitHub | | MSVC 19.44.35214.0 | x86 | Windows Server 2022 (Build 20348) | GitHub | | MSVC 19.44.35214.0 | x86_64 | Windows Server 2022 (Build 20348) | GitHub | | MSVC 19.51.36231.0 | x86 | Windows Server 2025 (Build 26100) | GitHub | | MSVC 19.51.36231.0 | x86_64 | Windows Server 2025 (Build 26100) | GitHub | -
The library is compiled with all C++ language revisions (C++11, C++14, C++17, C++20, C++23, and C++26) to detect and fix language deprecations early.
-
The library is checked for compiler warnings:
-
On Clang,
-Weverythingis used with 7 exceptions.??? abstract "Clang warnings"
```cmake --8<-- "../../../cmake/clang_flags.cmake" ``` -
On GCC, 300+ warnings are enabled with 8 exceptions.
??? abstract "GCC warnings"
```cmake --8<-- "../../../cmake/gcc_flags.cmake" ```
-
C++ standard library compliance
!!! success "Requirement: No prerequisites"
The library has no prerequisites other than the Standard Template Library (STL).
- The library is compiled and tested with both libc++ and libstdc++ to detect subtle differences or incompatibilities.
- The code checked with Include What You Use (IWYU) that all required standard headers are included.
- On Windows, the library is compiled with
<Windows.h>being included to detect and avoid common bugs (seeunit-windows_h.cpp). - The library is compiled with Qt's keyword macros (
slots,signals,emit,foreach,forever) being defined to make sure it does not use them as identifiers (seeunit-qt_keywords.cpp). - The library is compiled with exceptions disabled to support alternative means of error handling.
Stable public API
!!! success "Requirement: Stable public API"
Any change to the library does not break the public API.
- All public API functions are tested with a variety of arguments.
- The library is compiled and tested with different template arguments for number, string, array, and object types.
- Unit tests cover all lines of the code base.
- Every exception of the library is thrown in the test suite, and the error messages and exception ids are checked.
!!! success "Requirement: Complete documentation"
The public API is extensively documented.
- Every public API function has a dedicated page in the API reference documentation with a self-contained code example.
- All examples in the documentation are tested, and changes in their output are treated as an error.
Robust input processing
!!! success "Requirement: Standards compliance"
The library is compliant to JSON as defined in [RFC 8259](https://datatracker.ietf.org/doc/html/rfc8259).
- The lexer is tested with all valid Unicode code points and all prefixes of all invalid Unicode code points.
- The parser is tested against extensive correctness suites for JSON compliance.
- In addition, the library is continuously fuzz-tested at OSS-Fuzz where the library is checked against billions of inputs.
- Every crash reported by OSS-Fuzz is fixed together with a unit test that reproduces it, and the fix references the OSS-Fuzz issue. The round-trip checks of the fuzzer drivers are also part of the unit tests. See the fuzz testing documentation.
Static analysis
!!! success "Requirement: State-of-the-art code analysis"
The code is checked with state-of-the-art static code analysis tools.
-
The code is checked with the latest Clang-Tidy.
??? abstract "Clang-Tidy configuration (.clang-tidy)"
```ini --8<-- "../../../.clang-tidy" ``` -
The code is checked with the latest Cppcheck with all warnings enabled.
-
The code is checked with the latest Clang Static Analyzer with 52 enabled checkers.
-
The code is checked with Infer.
-
The code is checked with Codacy.
-
The code is scanned with CodeQL, Semgrep, and Flawfinder on every push and pull request and on a schedule; the findings are reported to GitHub's code scanning.
Dynamic analysis
!!! success "Requirement: Correctness"
The library is checked for memory correctness and absence of undefined behavior.
- The test suite is executed with enabled runtime assertions to check invariants and preconditions of functions to detect undefined behavior.
- The test suite is executed with Valgrind (Memcheck) to detect memory leaks.
- The test suite is executed with Sanitizers (address sanitizer, undefined behavior sanitizer, integer overflow detection, nullability violations).
- The fuzz targets are run with CIFuzz on every pull request to catch regressions before they are merged.
Dependencies
!!! success "Requirement: No vulnerable dependencies"
The library has no dependencies besides the C++ standard library. The tools used to build, test, and document it
are kept free of known vulnerabilities.
- GitHub Actions are pinned to a commit hash, and the Python packages used by the documentation and the tools are pinned to exact versions.
- Dependabot checks these dependencies daily and proposes updates as pull requests.
- Every pull request is checked with the dependency review action. A pull request that adds a dependency with a known vulnerability of any severity fails this check and is not merged.
- Vulnerability alerts for dependencies are fixed or dismissed with a documented reason before the next release. No release is made while such an alert is open.
- Third-party code included in the repository for testing, such as doctest, is updated manually.
- The repository is evaluated with the OpenSSF Scorecard on a schedule and on every push to the default branch, and the results are reported to GitHub's code scanning.
Style check
!!! success "Requirement: Common code style"
A common code style is used throughout all code files of the library.
-
The code is formatted with Artistic Style (astyle) against a style configuration that is also enforced in the CI.
??? abstract "Astyle configuration (tools/astyle/.astylerc)"
```ini --8<-- "../../../tools/astyle/.astylerc" ``` -
The code style is checked with cpplint with 61 enabled rules.
Simple integration
!!! success "Requirement: Single header"
The library can be used by adding a single header to a C++ project.
- An amalgamation script is used to check if the source code is exposed as a self-contained single-header file.
- The test suite is checked against the amalgamated source file as well as the individual source file.
!!! success "Requirement: CMake as primary development tool"
All library functions are exposed and usable by CMake.
-
All library options are exposed as CMake options and tested.
-
The library is tested against relevant CMake versions:
-
CMake 3.5 (the earliest supported)
-
CMake 3.31.6 (the latest 3.x release)
-
CMake 4.0.0 (the first 4.x release) !!! success "Requirement: Other build systems"
The library can be used with build systems and package managers other than CMake.
-
-
The library can be consumed with Meson and the Swift Package Manager; the installation with Meson and the resolution of
Package.swiftare tested in the CI.
!!! success "Requirement: Reproducible and verifiable builds"
The library can be built and tested without network access or a Git checkout, and its licensing is machine-readable.
- The test suite is executed from an exported source tree without the
.gitdirectory and with the test data being available offline. - The test suite is executed without the tests that change installed files to check that builds are reproducible.
- The repository is checked for compliance with the REUSE specification so that the license and copyright of every file is machine-readable.
- The documentation is built with warnings treated as errors, and its links are checked regularly.