feat(cccl): integrate missing CCCL directories — python/, ci/, .agent/, docs/, test/
Sparse-checkout from NVIDIA/cccl main branch to complete cccl_upstream: Added: - python/cuda_cccl/ (226 files) — Python bindings for device-level algorithms Critical for muh toolchain: cuda.compute.reduce_into, scan, radix_sort, etc. Includes 204 .py files with full test coverage for all 27 algorithms - ci/ (163 files) — Build/test infrastructure build_cub.sh, test_cub.sh, build_and_test_targets.sh, matrix.yaml Directly maps to our [INFRA-CI] and [INFRA-BUILD] items - .agent/skills/ (7 files) — NVIDIA's own agent skills for CCCL cccl-style/SKILL.md, cccl-test/SKILL.md, sass-diff/SKILL.md - docs/ (491 files) — Official CCCL documentation CI references, CMake guides, Python compute docs, libcudacxx PTX docs - test/ (12 files) — Top-level integration tests (cuda_smoke, stdpar) - Root configs: .clang-format, .clang-tidy, CONTRIBUTING.md, pyproject.toml - CLAUDE.md symlink → AGENTS.md (NVIDIA's standard) cccl_upstream now mirrors full NVIDIA/cccl structure: Before: 42M (cub + thrust + libcudacxx + cudax + c + examples + benchmarks) After: 53M (+python +ci +docs +.agent +test +configs) This completes the CCCL base needed for: - [muh-bench] items: ci/util/build_and_test_targets.sh for targeted builds - [CCCL-verify] items: python/cuda_cccl/tests/ as reference implementations - [CCCL-test] items: ci/test_cub.sh, ci/test_thrust.sh - Agent workflow: .agent/skills/ for consistent style and test patterns
This commit is contained in:
@@ -0,0 +1,118 @@
|
||||
.. _infra-ci-change-detection:
|
||||
|
||||
Change detection
|
||||
================
|
||||
|
||||
CCCL's CI does not run every pull-request matrix job on every pull request.
|
||||
|
||||
The PR matrix is split into two sections: ``pull_request`` and ``pull_request_lite``.
|
||||
The first contains a representative sampling of supported configurations for each CCCL project.
|
||||
The lite version is designed to 'smoke test' in situations where full coverage isn't warranted,
|
||||
but some coverage is prudent.
|
||||
|
||||
This improves turnaround time, reduces hardware costs / energy use, and helps keep the
|
||||
runner queues short for CCCL and the other projects that we share the runners with.
|
||||
|
||||
Two files drive the decision. ``ci/inspect_changes.py`` contains the logic, and
|
||||
``ci/project_files_and_dependencies.yaml`` defines the project graph and file mappings.
|
||||
|
||||
Ultimately, two lists of projects are produced: ``FULL_BUILD`` and ``LITE_BUILD``.
|
||||
The workflow-build step reads these lists and assembles the final matrix by pulling the
|
||||
relevant jobs from the relevant matrix workflows.
|
||||
|
||||
The project graph
|
||||
-----------------
|
||||
|
||||
Each project in ``ci/project_files_and_dependencies.yaml`` declares how files
|
||||
map to it and how rebuilds propagate from it:
|
||||
|
||||
``include_regexes``
|
||||
Path patterns that mark this project dirty. Every pattern is a regex anchored
|
||||
to the repository root.
|
||||
|
||||
``exclude_regexes``
|
||||
Patterns that remove files from the included set.
|
||||
|
||||
``exclude_project_files``
|
||||
When one project is nested inside another (common for public/internal splits),
|
||||
this can be used to easily exclude the inner project's files from the outer project.
|
||||
|
||||
``full_dependencies``
|
||||
Projects whose direct changes force a full rebuild of this project.
|
||||
|
||||
``lite_dependencies``
|
||||
Projects whose changes trigger a reduced rebuild of this project.
|
||||
|
||||
``matrix_project``
|
||||
The name this project uses in ``ci/matrix.yaml``. A project without one is
|
||||
internal to change detection and never appears in the output lists.
|
||||
|
||||
The ``core`` project is a catch-all. It declares no ``include_regexes``. Any
|
||||
dirty file that no other project claims lands in ``core`` — a new top-level
|
||||
script, a ``CMakeLists.txt`` at the root, a ``ci/`` change. When ``core`` has
|
||||
any dirty file, every project goes to ``FULL_BUILD`` immediately. A CI
|
||||
infrastructure change rebuilds everything, because the extent of its impact
|
||||
cannot be easily evaluated.
|
||||
|
||||
Files that should never trigger CI are filtered first, before project matching,
|
||||
via the top-level ``ignore_regexes`` list. Readmes, ``docs/``, config files,
|
||||
non-CI ``.github/workflows/``, and their unholy ilk should be catalogued here
|
||||
lest they anger the gods.
|
||||
|
||||
The public/internal split
|
||||
-------------------------
|
||||
|
||||
Large libraries are split into two projects: a ``_public`` project matching only
|
||||
the public headers, and an ``_internal`` project matching everything else.
|
||||
``libcudacxx_public`` matches ``libcudacxx/include/``. ``libcudacxx_internal``
|
||||
matches ``libcudacxx/`` and excludes ``libcudacxx_public``'s files via
|
||||
``exclude_project_files``. The same pattern holds for CUB, Thrust, cudax, and
|
||||
the C parallel library.
|
||||
|
||||
The split exists to keep test and infrastructure churn from rebuilding the
|
||||
world. ``libcudacxx_internal`` lists ``libcudacxx_public`` as a full dependency,
|
||||
so a public-header change still rebuilds the library's own tests. But downstream
|
||||
libraries depend on ``libcudacxx_public``, not ``libcudacxx_internal``. A change
|
||||
to a libcudacxx unit test rebuilds libcudacxx and stops there. A change to a
|
||||
public header propagates outward to CUB, Thrust, and cudax — as a lite rebuild,
|
||||
not a full one.
|
||||
|
||||
Propagation
|
||||
-----------
|
||||
|
||||
Projects matched directly by a changed file always go to ``FULL_BUILD``,
|
||||
regardless of depth.
|
||||
|
||||
If a dirty project has any explicit "full" dependencies, those projects are also
|
||||
added to ``FULL_BUILD``.
|
||||
|
||||
From here, all downstream projects reachable via lite or full dependencies
|
||||
from any project in ``FULL_BUILD`` is added to ``LITE_BUILD``.
|
||||
|
||||
Output
|
||||
------
|
||||
|
||||
``inspect_changes.py`` emits two GitHub Actions outputs, each a space-separated
|
||||
list of ``matrix_project`` values:
|
||||
|
||||
``FULL_BUILD``
|
||||
Projects that build and test their full job set.
|
||||
|
||||
``LITE_BUILD``
|
||||
Projects that build and test a reduced set.
|
||||
|
||||
These feed the workflow-build step. The build reads the ``pull_request`` section
|
||||
of ``ci/matrix.yaml`` (see :ref:`infra-ci-matrix-yaml`) and keeps only entries
|
||||
whose project is in ``FULL_BUILD``, then reads the ``pull_request_lite`` section
|
||||
and keeps only entries whose project is in ``LITE_BUILD``, then concatenates the
|
||||
two. A project absent from both lists contributes no jobs.
|
||||
|
||||
To see the exact lists for any change, run ``inspect_changes.py`` locally:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ci/inspect_changes.py --refs origin/main HEAD
|
||||
|
||||
The script prints the dependency overview, the per-project dirty-file breakdown,
|
||||
and the final ``FULL_BUILD`` and ``LITE_BUILD`` values. Pass ``--file`` or
|
||||
``--stdin`` to supply a path list directly instead of diffing refs.
|
||||
103
cccl_upstream/docs/infrastructure/ci/references/ci_artifacts.rst
Normal file
103
cccl_upstream/docs/infrastructure/ci/references/ci_artifacts.rst
Normal file
@@ -0,0 +1,103 @@
|
||||
.. _infra-ci-artifacts:
|
||||
|
||||
Artifact system
|
||||
===============
|
||||
|
||||
The artifact system carries data between CI jobs. A producer job uploads its build outputs;
|
||||
consumer jobs download them to run tests. It wraps GitHub Actions' native ``upload-artifact``
|
||||
and ``download-artifact`` and is GitHub Actions-specific.
|
||||
|
||||
Artifacts are not mandatory. Projects that lean on the shared AWS sccache can let the cache
|
||||
serve build products to test jobs instead. They still pay off for larger projects: downloading
|
||||
one compressed archive of the test binaries is much faster than recompiling or fetching each
|
||||
target individually through the cache. Compare a test job's sccache build time against the cost
|
||||
of packing and downloading an archive to decide which a project should use.
|
||||
|
||||
Scripts
|
||||
-------
|
||||
|
||||
CI scripts call a small set of entry points under ``ci/util/artifacts/``. Each script's source
|
||||
is the authoritative reference for its full argument set.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 32 68
|
||||
|
||||
* - Script
|
||||
- Purpose
|
||||
* - ``stage.sh``
|
||||
- Add files matching a regex to a named artifact's staged set.
|
||||
* - ``unstage.sh``
|
||||
- Remove files from a staged set — prune what no consumer needs.
|
||||
* - ``upload_stage_packed.sh``
|
||||
- Pack a staged set into a compressed ``tar.zst`` and upload it.
|
||||
* - ``upload.sh``
|
||||
- Upload a file, or regex-matched files, as a plain unpacked artifact.
|
||||
* - ``download_packed.sh``
|
||||
- Download a packed artifact and extract it to a path.
|
||||
* - ``download.sh``
|
||||
- Download a plain artifact file to a path.
|
||||
|
||||
The other scripts under ``ci/util/artifacts/`` — ``register.sh``, ``pack.sh``,
|
||||
``upload_packed.sh``, and the ``upload/set_*.sh`` compression and retention knobs — are
|
||||
lower-level building blocks the entry points compose. ``upload_stage_packed.sh`` packs,
|
||||
registers, and sets compression on the archive itself, so producers rarely touch them directly.
|
||||
|
||||
Producers and consumers find each other through ``ci/util/workflow/`` helpers:
|
||||
``get_producer_id.sh`` resolves a consumer's producer, and ``has_consumers.sh`` /
|
||||
``get_consumers.sh`` let a producer skip work when nothing downstream needs its output.
|
||||
|
||||
Producer: stage and pack build outputs
|
||||
--------------------------------------
|
||||
|
||||
A producer stages the files a consumer will need, prunes what it does not, then packs and
|
||||
uploads the set as one archive. It first checks whether any consumer exists and exits early if
|
||||
not:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ci/util/workflow/has_consumers.sh "$JOB_ID" || exit 0
|
||||
|
||||
ci/util/artifacts/stage.sh "<artifact_name>" '<regex>' ['<regex>' ...]
|
||||
ci/util/artifacts/unstage.sh "<artifact_name>" '<regex>'
|
||||
ci/util/artifacts/upload_stage_packed.sh "<artifact_name>"
|
||||
|
||||
``stage.sh`` and ``unstage.sh`` build up the file set by inclusion and exclusion — regexes match
|
||||
against ``find`` within the stage path. ``upload_stage_packed.sh`` compresses the result and
|
||||
registers it for upload. CUB stages one packed artifact per launch-id variant (``no_lid``,
|
||||
``lid_0``–``lid_2``); ``ci/upload_cub_test_artifacts.sh`` is the authoritative example.
|
||||
|
||||
Consumer: resolve and download
|
||||
------------------------------
|
||||
|
||||
A consumer does not know its producer's job ID in advance. It resolves the producer from the
|
||||
run manifest, then fetches the packed outputs.
|
||||
|
||||
Resolve the producer with ``get_producer_id.sh`` — it loads the run manifest (``workflow.json``)
|
||||
on demand and returns the producer's job ID. The packed artifact name is project-specific and
|
||||
embeds the producer ID; the consuming CI script constructs it, so there are no hardcoded names.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
producer_id=$(ci/util/workflow/get_producer_id.sh)
|
||||
for tag in "${ARTIFACT_TAGS[@]}"; do
|
||||
ci/util/artifacts/download_packed.sh \
|
||||
"z_cub-test-artifacts-${DEVCONTAINER_NAME:?}-${producer_id}-${tag}" /home/coder/cccl
|
||||
done
|
||||
|
||||
See ``ci/test_cub.sh`` for the authoritative CUB form.
|
||||
|
||||
Result record
|
||||
-------------
|
||||
|
||||
Apart from build artifacts, every job records its own outcome. At exit it calls
|
||||
``ci/upload_job_result_artifacts.sh "$JOB_ID" $exit_code``, which uploads a ``zz_jobs-<job_id>``
|
||||
artifact containing a ``success`` file only when the exit code was zero. The ``ci:`` gate reads
|
||||
these records to compute the single pass/fail for the run — see :ref:`infra-ci-overview`.
|
||||
|
||||
Python wheels
|
||||
-------------
|
||||
|
||||
Python jobs exchange built wheels rather than packed test binaries. A producer uploads the wheel
|
||||
with ``upload.sh``; a consumer downloads it with ``download.sh``. Both resolve the filename
|
||||
through ``ci/util/workflow/get_wheel_artifact_name.sh``.
|
||||
160
cccl_upstream/docs/infrastructure/ci/references/ci_overview.rst
Normal file
160
cccl_upstream/docs/infrastructure/ci/references/ci_overview.rst
Normal file
@@ -0,0 +1,160 @@
|
||||
.. _infra-ci-overview:
|
||||
|
||||
CI overview
|
||||
===========
|
||||
|
||||
CCCL runs its build and test matrix on NVIDIA's self-hosted GitHub Actions runners.
|
||||
The pipeline turns one push into hundreds of jobs across CUDA Toolkit versions,
|
||||
host compilers, GPU architectures, C++ standards, and operating systems.
|
||||
|
||||
Triggering and the copy-pr-bot security model
|
||||
---------------------------------------------
|
||||
|
||||
CCCL's runners have access to NVIDIA infrastructure, so arbitrary fork code must
|
||||
never run on them unreviewed. The ``copy-pr-bot`` GitHub App enforces this. CI does
|
||||
not trigger on ``pull_request`` events. It triggers on pushes to ``pull-request/<N>``
|
||||
branches in the main repository, where ``<N>`` is the PR number. ``copy-pr-bot``
|
||||
owns those branches and only writes vetted commits to them.
|
||||
|
||||
For external contributors, CI does not begin until a maintainer leaves an
|
||||
``/ok to test [commit SHA]`` comment. ``copy-pr-bot`` verifies the SHA, copies the
|
||||
approved code to the ``pull-request/<N>`` branch, and the push to that branch starts
|
||||
CI. Every new commit on an external PR needs a fresh ``/ok to test``.
|
||||
|
||||
For NVIDIA enterprise members with signed commits, the bot establishes identity from
|
||||
the commit signature and copies pushes automatically. CI begins immediately on each
|
||||
push, with no comment required. See the `copy-pr-bot documentation
|
||||
<https://docs.gha-runners.nvidia.com/apps/copy-pr-bot/>`_ for signing setup.
|
||||
Draft PRs do not auto-trigger — ``auto_sync_draft: false`` in
|
||||
``.github/copy-pr-bot.yaml`` holds the bot back until the PR is ready.
|
||||
|
||||
SSH signing keys
|
||||
----------------
|
||||
|
||||
`Signed commits <https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits>`_
|
||||
are required for any internal NVIDIA contributors who want the convenience of CI running
|
||||
automatically whenever a commit is pushed to a branch (i.e., doesn't require using ``/ok to test``).
|
||||
|
||||
This is not required for external contributions, which will always require an explicit
|
||||
``/ok to test [commit SHA]`` comment from an approved account for each CI run.
|
||||
|
||||
To enable commit signing using your existing ssh key, set the following git options:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git config --global gpg.format ssh
|
||||
git config --global user.signingKey ~/.ssh/YOUR_PUBLIC_KEY_FILE_HERE.pub
|
||||
|
||||
# These settings are optional. They tell git to automatically sign all new commits and tags.
|
||||
# If these are set to false, use `git commit -S` to manually sign each commit.
|
||||
git config --global commit.gpgsign true
|
||||
git config --global tag.gpgsign true
|
||||
|
||||
Git is now configured to sign commits with your ssh key.
|
||||
|
||||
To complete the process, upload the public key to your `GitHub Signing Keys
|
||||
<https://github.com/settings/keys>`_ in your browser or using the ``gh`` CLI tool:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
gh ssh-key add ~/.ssh/YOUR_PUBLIC_KEY_FILE_HERE.pub --type signing
|
||||
|
||||
Make sure that the key is uploaded to 'Signing Keys', not just 'Authentication Keys'.
|
||||
The same key may be used for both.
|
||||
|
||||
Building the job matrix
|
||||
-----------------------
|
||||
|
||||
``ci/matrix.yaml`` is the authoritative matrix definition, described in detail at
|
||||
:ref:`infra-ci-matrix-yaml`. The ``build-workflow`` action
|
||||
(``.github/actions/workflow-build/``) reads it and produces the concrete job list for
|
||||
the run. The action selects a workflow key by trigger: ``pull_request`` for PRs,
|
||||
``nightly`` and ``weekly`` for scheduled runs.
|
||||
|
||||
Before expansion, ``ci/inspect_changes.py`` compares the PR against its merge base and
|
||||
reports which projects changed, the mechanism covered at
|
||||
:ref:`infra-ci-change-detection`. The action filters the ``pull_request`` matrix to the
|
||||
changed projects and pulls reduced ``pull_request_lite`` entries for projects only
|
||||
affected through a dependency. A change anywhere in CCCL infrastructure marks every
|
||||
project dirty and runs the full matrix. The expansion mechanics — dependency
|
||||
propagation, tag defaults, list explosion, ``job_map`` — are in
|
||||
``ci/matrix.yaml``'s ``tags``, ``jobs``, and ``projects`` sections and the
|
||||
``build-workflow.py`` action.
|
||||
|
||||
Expansion produces ``workflow.json``: named groups (for example, "CUB GCC"), each
|
||||
carrying a ``standalone`` array and a ``two_stage`` array. A second step splits these
|
||||
into four buckets along two axes — operating system and structure:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
linux_standalone linux_two_stage
|
||||
windows_standalone windows_two_stage
|
||||
|
||||
``ci-workflow-pull-request.yml`` consumes the four buckets as four parallel matrix
|
||||
dispatches, one per ``workflow-dispatch-<structure>-group-<os>.yml``. The split exists
|
||||
because Linux and Windows need different runner images and shell tooling, and because
|
||||
standalone and two-stage jobs have different dependency shapes. The workflow file
|
||||
mechanics are covered in full at :ref:`infra-ci-gha-workflows`.
|
||||
|
||||
Standalone and two-stage jobs
|
||||
-----------------------------
|
||||
|
||||
A standalone job builds and tests in one runner, sharing nothing with other jobs.
|
||||
|
||||
A two-stage job splits build from test into a generic single-producer multiple-consumer model.
|
||||
A single producer job creates artifacts (usually on a cheap non-GPU runner), which are then
|
||||
consumed in one or more test jobs, which may uses GPUs if needed.
|
||||
|
||||
Accelerating build times with sccache
|
||||
-------------------------------------
|
||||
|
||||
CCCL's CI uses a `heavily modified fork of sccache <https://github.com/rapidsai/sccache>`_ with
|
||||
improved CUDA support to cache compiler artifacts for files that haven't changed and dramatically
|
||||
accelerate build times. Local builds inside `CCCL's
|
||||
Dev Containers <https://github.com/NVIDIA/cccl/blob/main/.devcontainer/README.md>`_ can share the
|
||||
same cache such that local builds and CI jobs mutually benefit from accelerated build times.
|
||||
Follow the `GitHub Authentication
|
||||
<https://github.com/NVIDIA/cccl/blob/main/.devcontainer/README.md#optional-authenticate-with-github-for-sccache>`_
|
||||
guide to enable this feature.
|
||||
|
||||
PR Branch Protections
|
||||
---------------------
|
||||
|
||||
The pull request workflow produces a single job that gates PR mergability, named "CI."
|
||||
This sentinel job depends on and checks for success of every required top-level job in the
|
||||
workflow.
|
||||
|
||||
This test also checks for skip tags, override matrix modifications, open benchmark requests, etc.
|
||||
and will block merging until the tree is restored to a mergeable state.
|
||||
|
||||
Override and skip enforcement
|
||||
-----------------------------
|
||||
|
||||
A non-empty ``workflows.override`` in ``ci/matrix.yaml`` replaces the ``pull_request``
|
||||
matrix entirely, detailed at :ref:`infra-ci-override-matrix`. Use it to run a subset of jobs
|
||||
when a full run would be unnecessary / wasteful during iteration.
|
||||
The override deliberately blocks merge: the full suite must run before any PR lands,
|
||||
so the override must be emptied and re-pushed before acceptance testing.
|
||||
|
||||
Skip tags in the last commit message on the branch, documented at :ref:`infra-ci-skip-tags`, drop job
|
||||
groups for the next run. They also block merge while present, forcing a clean final run:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git commit -m "README tidy-up [skip-matrix][skip-vdc][skip-docs][skip-tpt]"
|
||||
git commit -m "Run PR benchmarks [bench-only]"
|
||||
|
||||
The recognized tags and their semantics are catalogued in :ref:`infra-ci-skip-tags`.
|
||||
``[bench-only]`` is shorthand for the common benchmark-request combination.
|
||||
|
||||
Reproducing a failure locally
|
||||
-----------------------------
|
||||
|
||||
CI jobs run the build and test scripts in ``ci/`` inside the devcontainers described in
|
||||
``.devcontainer/README.md``. A failing job's log names the exact container and script
|
||||
invocation. Pull the same container and run the same ``ci/build_<project>.sh`` or
|
||||
``ci/test_<project>.sh`` line to reproduce the CI environment, as walked through at
|
||||
:ref:`infra-ci-reproducing-locally`. For targeted single-test iteration,
|
||||
``ci/util/build_and_test_targets.sh`` builds and runs a named subset, covered at
|
||||
:ref:`infra-ci-targeted-builds`; for a regression hunt, see
|
||||
:doc:`/cccl/development/build_and_bisect_tools`.
|
||||
@@ -0,0 +1,66 @@
|
||||
.. _infra-ci-scripts:
|
||||
|
||||
CI scripts
|
||||
==========
|
||||
|
||||
The ``ci/`` directory holds the build, test, benchmark, and utility scripts that CI jobs
|
||||
invoke. The same scripts reproduce a CI environment locally — a failing job's log prints the
|
||||
exact script and arguments it ran. Scripts are organized by role:
|
||||
|
||||
- ``ci/`` root — per-project build and test entry points.
|
||||
- ``ci/util/`` — shared utilities, plus the artifact and workflow plumbing jobs use to pass data.
|
||||
- ``ci/bench/`` — benchmark drivers.
|
||||
- ``ci/windows/`` — PowerShell equivalents of the per-project scripts for Windows runners.
|
||||
|
||||
Per-project build and test scripts
|
||||
----------------------------------
|
||||
|
||||
Each project has a pair of scripts following a fixed naming convention:
|
||||
``ci/build_<project>.sh`` configures and builds it, ``ci/test_<project>.sh`` builds and runs
|
||||
its tests. Both source ``ci/build_common.sh`` for argument parsing and CMake setup, so every
|
||||
project shares one flag interface. Building tests needs no GPU; running them (usually) does.
|
||||
|
||||
Specialized build scripts cover configurations that fall outside the per-project pattern —
|
||||
stdpar, clang-tidy, Python wheels, NVRTC, and codegen verification among them. They live
|
||||
beside the per-project scripts in ``ci/`` and source the same common setup.
|
||||
|
||||
The flags come from ``ci/build_common.sh``: the host and CUDA compilers (``-cxx``, ``-cuda``),
|
||||
the C++ standard (``-std``), target architectures (``-arch``), forwarded CMake options
|
||||
(``-cmake-options``), and ``-configure`` to stop after configuration.
|
||||
``PARALLEL_LEVEL`` controls build parallelism. Run a script with ``-h`` for the authoritative
|
||||
flag list and current defaults.
|
||||
|
||||
When run locally, the test scripts will invoke the build script to ensure that the targets are
|
||||
available. In CI, they may download GHA artifacts instead.
|
||||
|
||||
::
|
||||
|
||||
./ci/test_cub.sh -cxx g++ -std 17 -arch "70;80;90"
|
||||
|
||||
For fast local iteration on a single target rather than a whole project, see
|
||||
:doc:`/cccl/development/build_and_bisect_tools`.
|
||||
|
||||
Utility scripts: ci/util/
|
||||
-------------------------
|
||||
|
||||
``ci/util/`` collects tooling shared across jobs: a targeted build-and-test runner
|
||||
(``build_and_test_targets.sh``), automated ``git bisect`` over a build/test command, command
|
||||
retry, peak-memory monitoring, and a mock job environment (``create_mock_job_env.sh``) that
|
||||
lets the artifact and workflow scripts run outside GitHub Actions. Run any script with ``-h``
|
||||
for its options.
|
||||
|
||||
Two subdirectories carry the producer/consumer plumbing for two-stage jobs:
|
||||
``ci/util/artifacts/`` uploads and downloads the files passed between jobs, and
|
||||
``ci/util/workflow/`` resolves producer/consumer relationships for the current run. Both are
|
||||
covered at :ref:`infra-ci-artifacts`.
|
||||
|
||||
``ci/inspect_changes.py`` reports which projects are dirty between two refs and drives full
|
||||
versus lite matrix selection; see :ref:`infra-ci-change-detection`.
|
||||
|
||||
Benchmark scripts: ci/bench/
|
||||
----------------------------
|
||||
|
||||
``ci/bench/`` holds the benchmark drivers: ``bench.sh`` builds and runs the suite for a
|
||||
configuration, and the ``compare_*`` scripts build two refs or two paths and diff the results.
|
||||
The comparison workflow is ``.github/workflows/bench.yml``. PR request syntax lives in
|
||||
``ci/bench.yaml``; ``ci/bench/README.md`` documents local usage and artifact layout.
|
||||
@@ -0,0 +1,161 @@
|
||||
.. _infra-ci-gha-workflows:
|
||||
|
||||
GitHub Actions workflows
|
||||
========================
|
||||
|
||||
Every CCCL workflow lives in ``.github/workflows/``. They fall into five purposes:
|
||||
the CI matrix, manual developer tools, the release pipeline, documentation
|
||||
deployment, and repository automation. This page catalogs each workflow, then
|
||||
explains the workflow-file mechanics that turn one push into hundreds of jobs.
|
||||
For the high-level CI narrative — triggering, change detection, the ``ci:`` gate —
|
||||
see :ref:`infra-ci-overview`.
|
||||
|
||||
Workflow reference
|
||||
------------------
|
||||
|
||||
CI matrix
|
||||
~~~~~~~~~
|
||||
|
||||
Three trigger workflows select a matrix key and run it. Six dispatcher workflows
|
||||
unfold the generated job list into GitHub matrix strategies. Contributors rarely
|
||||
invoke any of these directly; they fire on push or schedule.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 40 20 40
|
||||
|
||||
* - Workflow
|
||||
- Trigger
|
||||
- Purpose
|
||||
* - ``.github/workflows/ci-workflow-pull-request.yml``
|
||||
- Automatic
|
||||
- Runs the ``pull_request`` matrix on every push to a ``pull-request/<N>`` branch.
|
||||
* - ``.github/workflows/ci-workflow-nightly.yml``
|
||||
- Both
|
||||
- Runs the broader ``nightly`` matrix on a weekday schedule, with Slack notifications.
|
||||
* - ``.github/workflows/ci-workflow-weekly.yml``
|
||||
- Both
|
||||
- Runs the exhaustive ``weekly`` matrix on a Sunday schedule, with Slack notifications.
|
||||
* - ``.github/workflows/workflow-dispatch-standalone-group-linux.yml``
|
||||
- Automatic
|
||||
- Dispatches an array of standalone Linux jobs as a matrix.
|
||||
* - ``.github/workflows/workflow-dispatch-standalone-group-windows.yml``
|
||||
- Automatic
|
||||
- Dispatches an array of standalone Windows jobs as a matrix.
|
||||
* - ``.github/workflows/workflow-dispatch-two-stage-group-linux.yml``
|
||||
- Automatic
|
||||
- Dispatches an array of producer/consumer Linux chains as a matrix.
|
||||
* - ``.github/workflows/workflow-dispatch-two-stage-group-windows.yml``
|
||||
- Automatic
|
||||
- Dispatches an array of producer/consumer Windows chains as a matrix.
|
||||
* - ``.github/workflows/workflow-dispatch-two-stage-linux.yml``
|
||||
- Automatic
|
||||
- Executes one producer/consumer chain on Linux.
|
||||
* - ``.github/workflows/workflow-dispatch-two-stage-windows.yml``
|
||||
- Automatic
|
||||
- Executes one producer/consumer chain on Windows.
|
||||
|
||||
Manual tools
|
||||
~~~~~~~~~~~~
|
||||
|
||||
Contributors invoke these through ``workflow_dispatch`` inputs on the Actions tab.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 40 20 40
|
||||
|
||||
* - Workflow
|
||||
- Trigger
|
||||
- Purpose
|
||||
* - ``.github/workflows/bench.yml``
|
||||
- Both
|
||||
- Compares benchmark performance between two refs; also called by the PR workflow's bench dispatch.
|
||||
* - ``.github/workflows/git-bisect.yml``
|
||||
- Manual
|
||||
- Runs an automated ``git bisect`` across a commit range to locate a regression.
|
||||
|
||||
Documentation
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 40 20 40
|
||||
|
||||
* - Workflow
|
||||
- Trigger
|
||||
- Purpose
|
||||
* - ``.github/workflows/docs-deploy.yml``
|
||||
- Both
|
||||
- Builds the Sphinx docs and publishes them to GitHub Pages under a version-derived path.
|
||||
|
||||
|
||||
CI orchestration
|
||||
----------------
|
||||
|
||||
The build-workflow action
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The "build workflow from matrix" job runs the ``build-workflow`` action at
|
||||
``.github/actions/workflow-build/``. Its script ``build-workflow.py`` reads
|
||||
``ci/matrix.yaml`` together with the change-detection output from
|
||||
``ci/inspect_changes.py`` and produces ``workflow/workflow.json``. That file holds
|
||||
named groups — "CUB GCC", for example — each carrying a ``standalone`` array and a
|
||||
``two_stage`` array of fully expanded job specs.
|
||||
|
||||
A second script, ``prepare-workflow-dispatch.py``, splits ``workflow.json`` into
|
||||
``dispatch.json``. This is the structure the GHA workflow is actually dispatched over.
|
||||
|
||||
The four-bucket split
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
``dispatch.json`` carries four buckets, keyed by operating system crossed with job
|
||||
structure:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
linux_standalone linux_two_stage
|
||||
windows_standalone windows_two_stage
|
||||
|
||||
Each bucket holds an ordered ``keys`` list of group names and a ``jobs`` map from
|
||||
each key to its job specs:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
{
|
||||
"linux_two_stage": {
|
||||
"keys": ["CUB GCC", "Thrust Clang", ...],
|
||||
"jobs": {
|
||||
"CUB GCC": [{ "producers": [...], "consumers": [...] }],
|
||||
"Thrust Clang": [{ "producers": [...], "consumers": [...] }]
|
||||
}
|
||||
},
|
||||
"linux_standalone": { "keys": [...], "jobs": {...} },
|
||||
"windows_two_stage": { "keys": [...], "jobs": {...} },
|
||||
"windows_standalone": { "keys": [...], "jobs": {...} }
|
||||
}
|
||||
|
||||
``ci-workflow-pull-request.yml`` dispatches four parallel matrix jobs, one per
|
||||
bucket. Each fans out over its bucket's ``keys`` and calls the matching dispatcher
|
||||
workflow — ``workflow-dispatch-<structure>-group-<os>.yml``.
|
||||
|
||||
Standalone jobs
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
A standalone job builds and tests on one runner, exchanging no artifacts with other
|
||||
jobs. The group dispatcher
|
||||
(``workflow-dispatch-standalone-group-<os>.yml``) fans out over the bucket's job
|
||||
array; each matrix entry runs one CI script — ``ci/build_<project>.sh`` or
|
||||
``ci/test_<project>.sh`` — start to finish. See :ref:`infra-ci-scripts` for details.
|
||||
|
||||
Two-stage jobs
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
A typical two-stage workflow splits a build job from one or more test jobs. Each stage
|
||||
has different workloads and hardware requirements, and this division allows us to schedule
|
||||
each on appropriate hardware.
|
||||
|
||||
The producer runs on a GPU-less, CPU-heavy runner.
|
||||
It builds and uploads binaries through GHA artifacts (:ref:`infra-ci-artifacts`) or sccache.
|
||||
|
||||
Each consumer runs on a CPU-light, GPU-attached runner.
|
||||
They fetch the pre-compiled test binaries and execute them.
|
||||
18
cccl_upstream/docs/infrastructure/ci/references/index.rst
Normal file
18
cccl_upstream/docs/infrastructure/ci/references/index.rst
Normal file
@@ -0,0 +1,18 @@
|
||||
.. _ci-references-index:
|
||||
|
||||
CI Reference
|
||||
============
|
||||
|
||||
Reference pages for CI configuration, scripts, and outputs.
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
:maxdepth: 1
|
||||
|
||||
ci_overview
|
||||
change_detection
|
||||
matrix_yaml
|
||||
ci_scripts
|
||||
ci_artifacts
|
||||
gha_workflows
|
||||
skip_tags
|
||||
183
cccl_upstream/docs/infrastructure/ci/references/matrix_yaml.rst
Normal file
183
cccl_upstream/docs/infrastructure/ci/references/matrix_yaml.rst
Normal file
@@ -0,0 +1,183 @@
|
||||
.. _infra-ci-matrix-yaml:
|
||||
|
||||
matrix.yaml reference
|
||||
=====================
|
||||
|
||||
``ci/matrix.yaml`` is the authoritative definition of CCCL's CI job matrix. It declares
|
||||
the workflows, the toolchain and hardware vocabulary jobs draw from, and the per-workflow
|
||||
job entries that expand into individual GitHub Actions jobs.
|
||||
|
||||
Top-level keys
|
||||
--------------
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 24 76
|
||||
|
||||
* - Key
|
||||
- Purpose
|
||||
* - ``workflows``
|
||||
- Map of workflow name to a list of job entries. Holds ``override``, ``pull_request``,
|
||||
``pull_request_lite``, ``nightly``, ``weekly``, etc.
|
||||
* - ``devcontainer_version``
|
||||
- Image version tag for ``rapidsai/devcontainers``.
|
||||
* - ``cuda99_gcc_version``, ``cuda99_clang_version``
|
||||
- Compiler versions used for internal cuda99.X builds.
|
||||
* - ``all_stds``
|
||||
- Every C++ standard CCCL supports (used for ``std: 'all'`` et al).
|
||||
* - ``ctk_versions``
|
||||
- Map of supported CUDA Toolkit versions to supported standards and aliases.
|
||||
* - ``device_compilers``
|
||||
- Device compiler definitions (``nvcc``, ``clang``). Selected by the ``cudacxx`` tag.
|
||||
* - ``host_compilers``
|
||||
- Host compiler definitions (``gcc``, ``clang``, ``msvc``, ``nvhpc``) and their per-version
|
||||
standards. Selected by the ``cxx`` tag.
|
||||
* - ``jobs``
|
||||
- Job type definitions: GPU requirement, dependencies, and script invocation.
|
||||
* - ``projects``
|
||||
- Project definitions: supported standards, display name, and ``job_map`` expansions.
|
||||
* - ``gpus``
|
||||
- GPU runner pools and their ``sm`` value.
|
||||
* - ``tags``
|
||||
- The fields a job entry accepts, with ``required`` flags and defaults.
|
||||
|
||||
``exclude`` is nested under ``workflows``. Entries matching an ``exclude`` rule are removed
|
||||
from every workflow's generated matrix.
|
||||
|
||||
Workflow types
|
||||
--------------
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 24 76
|
||||
|
||||
* - Workflow
|
||||
- Purpose
|
||||
* - ``override``
|
||||
- Overrides the CI jobs run for the active PR when defined. Blocks merge while set.
|
||||
* - ``pull_request``
|
||||
- Default per-PR matrix.
|
||||
* - ``pull_request_lite``
|
||||
- Reduced matrix run when only an upstream dependency changed. (See :ref:`infra-ci-change-detection`.)
|
||||
* - ``nightly``
|
||||
- Extended scheduled matrix.
|
||||
* - ``weekly``
|
||||
- Broadest scheduled matrix, including ``sm: 'all-cccl'`` and compute-sanitizer coverage.
|
||||
* - ``python-wheels``
|
||||
- Python wheel build and test matrix.
|
||||
* - ``devcontainers``
|
||||
- Image-generation matrix. Entries catalog currently available devcontainer configs and map to no real jobs.
|
||||
|
||||
Job entry format
|
||||
----------------
|
||||
|
||||
A job entry is a YAML mapping. Array-valued fields expand to the cross-product of their
|
||||
elements; ``exclude`` rules are applied after expansion. The fields a user specifies are the
|
||||
tags in the ``tags`` section.
|
||||
|
||||
Current defaults for all tags are defined in the ``tags`` section of ``ci/matrix.yaml``.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 18 82
|
||||
|
||||
* - Field
|
||||
- Meaning
|
||||
* - ``jobs``
|
||||
- Job types to run (e.g. ``build``, ``test``, ``nvrtc``, ``verify_codegen``). Expanded by
|
||||
the project's ``job_map``. Dependencies added automatically. Required.
|
||||
* - ``project``
|
||||
- Project key from the ``projects`` section.
|
||||
* - ``ctk``
|
||||
- CUDA Toolkit version or alias from ``ctk_versions``.
|
||||
* - ``cxx``
|
||||
- Host compiler. A bare name resolves to the latest version (see below).
|
||||
* - ``cudacxx``
|
||||
- Device compiler from ``device_compilers``; rarely needed, defaulting to ``nvcc``.
|
||||
* - ``std``
|
||||
- C++ standard. Accepts an integer or the ``all`` / ``min`` / ``max`` / ``minmax`` shortcuts.
|
||||
* - ``cpu``
|
||||
- CPU architecture (``amd64``, ``arm64``).
|
||||
* - ``gpu``
|
||||
- GPU runner type from ``gpus``.
|
||||
* - ``sm``
|
||||
- GPU architectures, ``CMAKE_CUDA_ARCHITECTURES`` syntax. ``gpu`` targets the ``gpu`` tag's SM.
|
||||
Omitting ``sm`` defers architecture selection to defaults in build scripts and ``CMakePresets.json``.
|
||||
* - ``py_version``
|
||||
- Python version for Python jobs.
|
||||
* - ``args``
|
||||
- Arguments appended to the generated command. Forwards options to
|
||||
``ci/util/build_and_test_targets.sh`` for the ``target`` project, but works for any job.
|
||||
* - ``cmake_options``
|
||||
- Extra CMake defines, passed as ``-cmake_options "<value>"``.
|
||||
* - ``environment``
|
||||
- Environment variables injected into the job.
|
||||
|
||||
Computed internally, not user-specified:
|
||||
|
||||
- ``needs`` — defined in the ``jobs`` section. A ``test`` entry auto-generates its ``build`` producer.
|
||||
- ``gpu`` requirement, ``cuda_ext``, ``name``, and ``invoke`` script details - also taken from the ``jobs`` section.
|
||||
- ``force_producer_ctk`` — set in the ``jobs`` section to pin a producer build's CTK independent
|
||||
of the consumer's ``ctk`` tag. Used mainly for python packaging special cases.
|
||||
|
||||
Annotated example
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
::
|
||||
|
||||
- {jobs: ['test'], # build (auto-added) then test
|
||||
project: 'thrust', # only thrust
|
||||
std: 'max', # highest std supported by thrust, ctk, and compiler
|
||||
cxx: ['gcc', 'clang'], # expands to two jobs, latest gcc and latest clang
|
||||
gpu: 'h100', # test job runs on an h100 runner
|
||||
sm: 'gpu', # build for the SM of the h100 runner (sm_90)
|
||||
cmake_options: '-DCMAKE_CUDA_FLAGS="-lineinfo"'} # customize cmake config
|
||||
|
||||
This expands to two ``build`` jobs — one per compiler. Thrust's ``job_map`` expands ``test``
|
||||
to ``test_cpu`` and ``test_gpu``, so the two compilers produce four combined test jobs. The test
|
||||
jobs request h100 runners; the auto-generated ``build`` producers run on CPU runners using
|
||||
``-lineinfo`` for CUDA targets and only produce device code for the h100's SM90 arch.
|
||||
|
||||
In total this spawns six jobs:
|
||||
|
||||
- build gcc
|
||||
- build clang
|
||||
- test_cpu gcc
|
||||
- test_cpu clang
|
||||
- test_gpu gcc
|
||||
- test_gpu clang
|
||||
|
||||
C++ standard resolution
|
||||
-----------------------
|
||||
|
||||
``std`` accepts integers or four keywords resolved against the intersection of the standards
|
||||
supported by the selected CTK, host compiler, device compiler, and project:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 18 82
|
||||
|
||||
* - Value
|
||||
- Resolves to
|
||||
* - ``all``
|
||||
- One job per supported standard.
|
||||
* - ``min``
|
||||
- The lowest supported standard.
|
||||
* - ``max``
|
||||
- The highest supported standard.
|
||||
* - ``minmax``
|
||||
- The lowest and highest, two jobs.
|
||||
|
||||
Compiler version resolution
|
||||
---------------------------
|
||||
|
||||
A bare compiler name in ``cxx`` resolves to the latest version listed for that compiler in
|
||||
``host_compilers``. ``cxx: 'gcc'`` selects the highest ``gcc`` version under
|
||||
``host_compilers.gcc.versions``. Pin a version by naming it: ``cxx: 'gcc13'``.
|
||||
|
||||
Version aliases resolve the same way. ``cxx: 'msvc2022'`` maps to the ``msvc`` version whose
|
||||
``alias`` is ``2022``. CTK aliases follow the ``ctk_versions`` map: ``ctk: '13.X'`` selects the
|
||||
newest CTK 13 entry, ``ctk: 'nvhpc'`` selects the CTK shipped in the current NVHPC.
|
||||
|
||||
Each version entry declares its own supported standards. ``std`` resolution and the
|
||||
``exclude`` rules drop combinations a compiler version does not support.
|
||||
@@ -0,0 +1,69 @@
|
||||
.. _infra-ci-skip-tags:
|
||||
|
||||
Skip tags
|
||||
=========
|
||||
|
||||
Skip tags scope a pull-request CI run. Place a tag in the last commit message before pushing.
|
||||
The next PR run reads the last commit message and filters job groups accordingly. Combine tags
|
||||
with the override matrix in ``ci/matrix.yaml`` for finer control.
|
||||
|
||||
All ``[skip-*]`` and ``[bench-only]`` tags block merge while present. Remove them from the last
|
||||
commit before merging.
|
||||
|
||||
Tag reference
|
||||
-------------
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 30 50 20
|
||||
|
||||
* - Tag
|
||||
- Skips
|
||||
- Blocks merge
|
||||
* - ``[skip-matrix]``
|
||||
- All build and test jobs from ``ci/matrix.yaml``.
|
||||
- Yes
|
||||
* - ``[skip-vdc]``
|
||||
- All "Validate Devcontainer" jobs.
|
||||
- Yes
|
||||
* - ``[skip-docs]``
|
||||
- The documentation verification build.
|
||||
- Yes
|
||||
* - ``[skip-tpt]`` / ``[skip-third-party-testing]``
|
||||
- All third-party canary builds (MatX, PyTorch, RAPIDS).
|
||||
- Yes
|
||||
* - ``[skip-rapids]``
|
||||
- RAPIDS canary builds only.
|
||||
- Yes
|
||||
* - ``[skip-matx]``
|
||||
- MatX canary builds only.
|
||||
- Yes
|
||||
* - ``[skip-pytorch]``
|
||||
- PyTorch canary builds only.
|
||||
- Yes
|
||||
* - ``[bench-only]``
|
||||
- Equivalent to ``[skip-matrix][skip-vdc][skip-docs][skip-tpt]``.
|
||||
- Yes
|
||||
|
||||
``[skip-tpt]`` and ``[skip-third-party-testing]`` are aliases for the same tag.
|
||||
|
||||
``[bench-only]`` shorthand
|
||||
--------------------------
|
||||
|
||||
``[bench-only]`` expands to ``[skip-matrix][skip-vdc][skip-docs][skip-tpt]``. It skips all
|
||||
non-benchmark job groups. Benchmarks are triggered separately by modifying ``ci/bench.yaml``
|
||||
relative to ``ci/bench.template.yaml``.
|
||||
|
||||
Usage
|
||||
-----
|
||||
|
||||
Place one or more tags in the last commit message before pushing::
|
||||
|
||||
git commit -m "Run PR benchmarks [bench-only]"
|
||||
git commit -m "README tidy-up [skip-matrix][skip-vdc][skip-docs][skip-third-party-testing]"
|
||||
|
||||
Tags act on the last commit message of the branch. To change which tags apply, amend the last
|
||||
commit and re-push.
|
||||
|
||||
Remove all skip tags from the last commit before merging. The full CI suite must run on the final
|
||||
commit for the PR to land.
|
||||
@@ -0,0 +1,23 @@
|
||||
.. _infra-cmake-architecture-flags:
|
||||
|
||||
Architecture flags
|
||||
==================
|
||||
|
||||
CCCL builds device code for the SM architectures named in ``CMAKE_CUDA_ARCHITECTURES``, a
|
||||
semicolon-separated list of SM numbers, each optionally tagged ``-real`` (embed SASS) or
|
||||
``-virtual`` (embed PTX for JIT on newer GPUs). CMake also accepts ``native``, ``all``, and
|
||||
``all-major``. The
|
||||
`CUDA_ARCHITECTURES property <https://cmake.org/cmake/help/latest/prop_tgt/CUDA_ARCHITECTURES.html>`_
|
||||
documents the standard syntax and values.
|
||||
|
||||
CCCL adds two values, expanded at configure time by ``cccl_check_cuda_architectures()`` against
|
||||
the toolkit found at configure time:
|
||||
|
||||
- ``all-cccl`` — every architecture the current nvcc supports at or above CCCL's minimum.
|
||||
- ``all-major-cccl`` — one entry per major architecture at or above the minimum, carrying
|
||||
forward PTX on the highest.
|
||||
|
||||
Both resolve against the installed toolkit rather than a fixed table. The minimum supported
|
||||
architecture and the full expansion logic live in ``cmake/CCCLCheckCudaArchitectures.cmake``.
|
||||
Expansion runs only for a top-level CCCL build; downstream consumers pass a concrete list or
|
||||
``native``.
|
||||
@@ -0,0 +1,102 @@
|
||||
.. _infra-cmake-helper-modules:
|
||||
|
||||
CMake helper modules
|
||||
====================
|
||||
|
||||
CCCL's ``cmake/`` directory holds the helper modules, script templates, and the vendored
|
||||
CPM package manager that drive the build. Library ``CMakeLists.txt`` files include these
|
||||
modules to add executables, generate header tests, expand architecture lists, wire install
|
||||
rules, and fetch dependencies. Each entry covers the module's purpose and the functions or
|
||||
macros it provides.
|
||||
|
||||
Helper modules
|
||||
--------------
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 26 40 34
|
||||
|
||||
* - File
|
||||
- Purpose
|
||||
- Key functions / macros
|
||||
* - ``CCCLAddExecutable.cmake``
|
||||
- Add an executable with standard CCCL configuration, optional CTest registration, metatargets, and clang-tidy.
|
||||
- ``cccl_add_executable()``
|
||||
* - ``CCCLAddSubdir.cmake``
|
||||
- Pull the CCCL libraries into the build as in-tree subprojects via ``cccl_add_subdir_helper``, honoring ``CCCL_REQUIRED_COMPONENTS`` / ``CCCL_OPTIONAL_COMPONENTS``.
|
||||
- (calls ``cccl_add_subdir_helper()``)
|
||||
* - ``CCCLAddSubdirHelper.cmake``
|
||||
- Standardizes project/subproject behavior when included with ``add_subdirectory()`` by a consumer/CPM.
|
||||
- ``cccl_add_subdir_helper()``
|
||||
* - ``CCCLAddTidyTarget.cmake``
|
||||
- clang-tidy integration: global, per-subproject, and per-source analysis targets.
|
||||
- ``cccl_tidy_init()``, ``cccl_tidy_make_subproject_target()``, ``cccl_tidy_add_target()``
|
||||
* - ``CCCLBuildCompilerTargets.cmake``
|
||||
- Build the ``cccl.compiler_interface`` target carrying warning, RTTI, exception, and ptxas flags.
|
||||
- ``cccl_build_compiler_targets()``
|
||||
* - ``CCCLCheckCudaArchitectures.cmake``
|
||||
- Expand the special ``all-cccl`` and ``all-major-cccl`` values for ``CMAKE_CUDA_ARCHITECTURES`` against the current NVCC, filtered to the minimum CCCL-supported arch.
|
||||
- ``cccl_check_cuda_architectures()``
|
||||
* - ``CCCLClangdCompileInfo.cmake``
|
||||
- Enable ``CMAKE_EXPORT_COMPILE_COMMANDS`` and symlink ``compile_commands.json`` into the source tree for clangd.
|
||||
- (script; no public functions)
|
||||
* - ``CCCLConfigureTarget.cmake``
|
||||
- Apply common target properties: disable extensions, set and require the C++/CUDA standard, propagate dialect compile features, set output directories.
|
||||
- ``cccl_configure_target()``
|
||||
* - ``CCCLDevBuildChecks.cmake``
|
||||
- Enforce supported developer-build configuration: require matching ``CMAKE_CXX_STANDARD`` and ``CMAKE_CUDA_STANDARD``, default both to 17.
|
||||
- ``cccl_dev_build_checks()``
|
||||
* - ``CCCLEnsureMetaTargets.cmake``
|
||||
- Create the dot-path metatarget hierarchy so ``ninja cub.test`` builds all descendants of ``cub.test``.
|
||||
- ``cccl_ensure_metatargets()``
|
||||
* - ``CCCLGenerateHeaderTests.cmake``
|
||||
- Generate per-header compilation tests from a template to verify headers are self-contained, plus a link-check executable that catches missing ``inline`` markup.
|
||||
- ``cccl_generate_header_tests()``
|
||||
* - ``CCCLGetDependencies.cmake``
|
||||
- Fetch external and in-tree dependencies via ``find_package`` or CPM. NVBench SHA is pinned in ``CCCL_NVBENCH_SHA``.
|
||||
- ``cccl_get_<dependency>()``
|
||||
* - ``CCCLHideThirdPartyOptions.cmake``
|
||||
- Mark Catch2, CPM, FetchContent, and LLVM cache variables advanced to keep them out of the default cache view.
|
||||
- (script; ``mark_as_advanced`` only)
|
||||
* - ``CCCLInstallRules.cmake``
|
||||
- Generate header and CMake-config install rules per project, gated by a ``<project>_ENABLE_INSTALL_RULES`` cache option.
|
||||
- ``cccl_generate_install_rules()``
|
||||
* - ``CCCLTestParams.cmake``
|
||||
- Parse ``%PARAM%`` comments in test sources into the cartesian product of variant labels and preprocessor definitions. See :doc:`/cccl/development/testing` for usage.
|
||||
- ``cccl_parse_variant_params()``
|
||||
* - ``CCCLUtilities.cmake``
|
||||
- Shared utilities: non-fatal process execution, CPM-consumption compile tests, and expected-failure compile tests.
|
||||
- ``cccl_execute_non_fatal_process()``, ``cccl_add_compile_test()``, ``cccl_add_xfail_compile_target_test()``
|
||||
* - ``AppendOptionIfAvailable.cmake``
|
||||
- Append a compiler flag to a list only if a ``check_cxx_compiler_flag`` probe accepts it.
|
||||
- ``append_option_if_available()``
|
||||
* - ``CPM.cmake``
|
||||
- Vendored CPM.cmake package manager. Used by ``CCCLGetDependencies.cmake`` and by downstream consumers fetching CCCL.
|
||||
- ``CPMAddPackage()`` (third-party)
|
||||
|
||||
Install rule files
|
||||
------------------
|
||||
|
||||
``cmake/install/`` holds one file per installable project. Each calls
|
||||
``cccl_generate_install_rules()`` with that project's header subdirectories and packaging
|
||||
options.
|
||||
|
||||
Adding an executable
|
||||
--------------------
|
||||
|
||||
``cccl_add_executable()`` is the entry point most test and example ``CMakeLists.txt`` files
|
||||
use. It calls ``cccl_configure_target()`` for standard properties, registers metatargets via
|
||||
``cccl_ensure_metatargets()``, and adds a clang-tidy target via ``cccl_tidy_add_target()``.
|
||||
|
||||
::
|
||||
|
||||
cccl_add_executable(cub.test.device_reduce
|
||||
SOURCES test_device_reduce.cu
|
||||
ADD_CTEST
|
||||
)
|
||||
|
||||
``ADD_CTEST`` registers a CTest that runs the executable with no arguments. ``NO_METATARGETS``
|
||||
and ``NO_CLANG_TIDY`` opt out of those integrations. ``METATARGET_PATH`` overrides the dot-path
|
||||
(default: the target name). ``DIALECT`` forces a C++ standard for this target.
|
||||
|
||||
Architecture-flag expansion is covered in :ref:`infra-cmake-architecture-flags`.
|
||||
14
cccl_upstream/docs/infrastructure/cmake/references/index.rst
Normal file
14
cccl_upstream/docs/infrastructure/cmake/references/index.rst
Normal file
@@ -0,0 +1,14 @@
|
||||
.. _cmake-references:
|
||||
|
||||
CMake Reference
|
||||
===============
|
||||
|
||||
Reference pages for the CCCL CMake build system.
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
:maxdepth: 1
|
||||
|
||||
preset_reference
|
||||
cmake_modules
|
||||
architecture_flags
|
||||
@@ -0,0 +1,57 @@
|
||||
.. _infra-cmake-preset-reference:
|
||||
|
||||
Preset reference
|
||||
================
|
||||
|
||||
CCCL ships its CMake configurations as presets in ``CMakePresets.json``, using CMake's
|
||||
`preset format <https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html>`_. The file
|
||||
defines 40+ presets covering each library, multiple C++ standards, special build
|
||||
modes, and benchmarking.
|
||||
|
||||
Preset structure
|
||||
----------------
|
||||
|
||||
Each preset belongs to one of three sections:
|
||||
|
||||
- ``configurePresets`` set the configuration: enabled libraries, build type, CUDA
|
||||
architectures, C++ standard, and per-library options. Every configure preset
|
||||
inherits from the hidden ``base`` preset, which selects the Ninja generator,
|
||||
Release mode, ``all-major-cccl`` architectures, and disables all libraries by
|
||||
default. A named preset enables the libraries and options it needs.
|
||||
- ``buildPresets`` reference a configure preset by name. Some pin an explicit
|
||||
target list; most build everything the configuration enables.
|
||||
- ``testPresets`` reference a configure preset and add CTest filters. Filters
|
||||
select a subset of tests by name regex. CUB launcher-mode presets and Thrust
|
||||
GPU/CPU splits are examples of this pattern.
|
||||
|
||||
A configure preset, its build preset, and its test preset share a name. Run all
|
||||
three with the same ``<name>``.
|
||||
|
||||
Listing available presets
|
||||
-------------------------
|
||||
|
||||
``cmake --list-presets`` prints all configure presets::
|
||||
|
||||
cmake --list-presets
|
||||
|
||||
List build and test presets separately::
|
||||
|
||||
cmake --list-presets=build
|
||||
cmake --build --list-presets
|
||||
ctest --list-presets
|
||||
|
||||
Using a preset
|
||||
--------------
|
||||
|
||||
Configure, then build with the matching preset name::
|
||||
|
||||
cmake --preset cub-cpp17
|
||||
cmake --build --preset cub-cpp17
|
||||
|
||||
Run the test preset of the same name with ``ctest``::
|
||||
|
||||
ctest --preset cub-cpp17
|
||||
|
||||
Build output lands under ``build/<infix>/<presetName>/``, where the infix comes from
|
||||
the ``CCCL_BUILD_INFIX`` environment variable used for devcontainer isolation.
|
||||
Distinct presets use distinct build directories and do not collide.
|
||||
@@ -0,0 +1,52 @@
|
||||
.. _infra-devcontainer-overview:
|
||||
|
||||
Devcontainer overview
|
||||
=====================
|
||||
|
||||
CCCL devcontainers package a CUDA toolkit and host compiler into a Docker image
|
||||
that matches a CI environment exactly. The same image that builds and tests a
|
||||
project in GitHub Actions runs on a developer's machine. Reproducing a CI
|
||||
result locally requires no guesswork about toolchain versions: pick the
|
||||
combination, launch the container, and the compiler, CTK, and supporting tools
|
||||
are identical to what CI used.
|
||||
|
||||
Two uses: local development and CI
|
||||
----------------------------------
|
||||
|
||||
**Local development.** Open the repository in VSCode and select a devcontainer
|
||||
from the picker, or launch one directly with ``.devcontainer/launch.sh``.
|
||||
:ref:`infra-devcontainer-launch-sh-reference` covers ``launch.sh`` flags and the available
|
||||
combinations. ``launch.sh --docker`` runs the container without VSCode and drops
|
||||
into a shell or runs a script. Without ``--docker``, it opens the container in
|
||||
VSCode.
|
||||
|
||||
**CI.** GitHub Actions runs the identical images. A CI job for a given CTK and
|
||||
compiler combination builds and tests inside the same ``rapidsai/devcontainers``
|
||||
image a developer would launch locally.
|
||||
|
||||
Image source: rapidsai/devcontainers
|
||||
------------------------------------
|
||||
|
||||
The base images are built and published by the
|
||||
`rapidsai/devcontainers <https://github.com/rapidsai/devcontainers>`_ repository,
|
||||
not by CCCL. Each image bundles a CUDA toolkit, a host compiler, and
|
||||
development tooling. CCCL references these images by tag; it does not build them.
|
||||
|
||||
Image tags follow the pattern
|
||||
``rapidsai/devcontainers:<version>-cpp-<compiler><version>-cuda<ctk>[ext]``.
|
||||
The ``-cuda<ctk>`` segment is present for every combination except nvhpc, which
|
||||
bundles its own CUDA toolkit; nvhpc images omit it.
|
||||
The ``<version>`` tag is defined in ``ci/matrix.yaml`` under
|
||||
``devcontainer_version``. A generated ``.devcontainer/<combo>/devcontainer.json``
|
||||
contains the resolved tag for any given combination.
|
||||
|
||||
This upstream dependency gates toolchain changes. Adding a CUDA toolkit version
|
||||
or a host compiler requires the matching image to exist in rapidsai/devcontainers
|
||||
first. The sequence is:
|
||||
|
||||
#. Update rapidsai/devcontainers to build and publish images for the new
|
||||
combination.
|
||||
#. Update CCCL's ``ci/matrix.yaml`` to reference the new version.
|
||||
#. Regenerate CCCL's devcontainer configs to match all matrix workflow requirements.
|
||||
|
||||
:ref:`infra-devcontainer-adding-toolchain` has additional details on this process.
|
||||
@@ -0,0 +1,13 @@
|
||||
.. _infra-devcontainer-references-index:
|
||||
|
||||
Devcontainer Reference
|
||||
======================
|
||||
|
||||
Reference pages for the CCCL devcontainer environment.
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
:maxdepth: 1
|
||||
|
||||
devcontainer_overview
|
||||
launch_sh_reference
|
||||
@@ -0,0 +1,84 @@
|
||||
.. _infra-devcontainer-launch-sh-reference:
|
||||
|
||||
launch.sh reference
|
||||
===================
|
||||
|
||||
``.devcontainer/launch.sh`` launches a development container for a chosen CUDA
|
||||
toolkit and host compiler. It selects a generated ``devcontainer.json``, mounts
|
||||
the repository, and starts either a VSCode dev container or a raw Docker shell.
|
||||
Linux-only.
|
||||
|
||||
Flags
|
||||
-----
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 22 12 12 54
|
||||
|
||||
* - Flag
|
||||
- Type
|
||||
- Default
|
||||
- Description
|
||||
* - ``-c``, ``--cuda <VER>``
|
||||
- string
|
||||
- unset
|
||||
- CUDA toolkit version, e.g. ``12.9``. Combines with ``--host`` to select a
|
||||
container.
|
||||
* - ``-H``, ``--host <COMPILER>``
|
||||
- string
|
||||
- unset
|
||||
- Host compiler, e.g. ``gcc12``. Combines with ``--cuda`` to select a
|
||||
container.
|
||||
* - ``--cuda-ext``
|
||||
- flag
|
||||
- false
|
||||
- Select the extended-CTK-libraries image. Adds the ``ext`` suffix to the
|
||||
container name.
|
||||
* - ``-d``, ``--docker``
|
||||
- flag
|
||||
- false
|
||||
- Launch directly in Docker, bypassing VSCode.
|
||||
* - ``--gpus <REQUEST>``
|
||||
- string
|
||||
- inferred
|
||||
- GPU devices to attach, e.g. ``all``. Overrides the
|
||||
``hostRequirements.gpu`` value from ``devcontainer.json``.
|
||||
* - ``-e``, ``--env <LIST>``
|
||||
- list
|
||||
- none
|
||||
- Set additional container environment variables. Repeatable.
|
||||
* - ``-v``, ``--volume <LIST>``
|
||||
- list
|
||||
- none
|
||||
- Bind-mount an additional volume. Repeatable.
|
||||
* - ``-h``, ``--help``
|
||||
- flag
|
||||
- —
|
||||
- Print usage and exit.
|
||||
|
||||
Arguments after ``--`` pass through to the container as the command to run.
|
||||
|
||||
Container selection
|
||||
-------------------
|
||||
|
||||
With no ``--cuda`` and no ``--host``, ``launch.sh`` uses the top-level
|
||||
``.devcontainer/devcontainer.json`` (the default environment).
|
||||
|
||||
Set both ``--cuda`` and ``--host`` to select a generated container.
|
||||
With both set, ``launch.sh`` builds a directory name and loads
|
||||
``.devcontainer/<name>/devcontainer.json``:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
cuda<VER>[ext]-<COMPILER>
|
||||
|
||||
``<VER>`` comes from ``--cuda``, ``<COMPILER>`` from ``--host``, and the
|
||||
``ext`` suffix is added when ``--cuda-ext`` is set. Examples:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
cuda12.9-gcc12 # --cuda 12.9 --host gcc12
|
||||
cuda13.3ext-clang20 # --cuda 13.3 --host clang20 --cuda-ext
|
||||
|
||||
If the resolved ``devcontainer.json`` does not exist, ``launch.sh`` reports the
|
||||
unknown combination and exits non-zero.
|
||||
20
cccl_upstream/docs/infrastructure/index.rst
Normal file
20
cccl_upstream/docs/infrastructure/index.rst
Normal file
@@ -0,0 +1,20 @@
|
||||
.. _infra-index:
|
||||
|
||||
Infrastructure
|
||||
==============
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
:maxdepth: 2
|
||||
|
||||
ci/references/index
|
||||
cmake/references/index
|
||||
devcontainer/references/index
|
||||
|
||||
Build, test, and CI infrastructure for CCCL contributors and maintainers.
|
||||
|
||||
- :doc:`CI System <ci/references/index>`
|
||||
|
||||
- :doc:`CMake Build System <cmake/references/index>`
|
||||
|
||||
- :doc:`Devcontainer Environment <devcontainer/references/index>`
|
||||
Reference in New Issue
Block a user