feat(cccl): integrate missing CCCL directories — python/, ci/, .agent/, docs/, test/

Sparse-checkout from NVIDIA/cccl main branch to complete cccl_upstream:

Added:
- python/cuda_cccl/ (226 files) — Python bindings for device-level algorithms
  Critical for muh toolchain: cuda.compute.reduce_into, scan, radix_sort, etc.
  Includes 204 .py files with full test coverage for all 27 algorithms
- ci/ (163 files) — Build/test infrastructure
  build_cub.sh, test_cub.sh, build_and_test_targets.sh, matrix.yaml
  Directly maps to our [INFRA-CI] and [INFRA-BUILD] items
- .agent/skills/ (7 files) — NVIDIA's own agent skills for CCCL
  cccl-style/SKILL.md, cccl-test/SKILL.md, sass-diff/SKILL.md
- docs/ (491 files) — Official CCCL documentation
  CI references, CMake guides, Python compute docs, libcudacxx PTX docs
- test/ (12 files) — Top-level integration tests (cuda_smoke, stdpar)
- Root configs: .clang-format, .clang-tidy, CONTRIBUTING.md, pyproject.toml
- CLAUDE.md symlink → AGENTS.md (NVIDIA's standard)

cccl_upstream now mirrors full NVIDIA/cccl structure:
  Before: 42M (cub + thrust + libcudacxx + cudax + c + examples + benchmarks)
  After:  53M (+python +ci +docs +.agent +test +configs)

This completes the CCCL base needed for:
- [muh-bench] items: ci/util/build_and_test_targets.sh for targeted builds
- [CCCL-verify] items: python/cuda_cccl/tests/ as reference implementations
- [CCCL-test] items: ci/test_cub.sh, ci/test_thrust.sh
- Agent workflow: .agent/skills/ for consistent style and test patterns
This commit is contained in:
muh-bot
2026-08-07 02:34:33 +00:00
parent 3f97dca7ad
commit 2a7ca101d7
908 changed files with 121615 additions and 0 deletions

View File

@@ -0,0 +1,163 @@
.. _infra-ci-adding-coverage:
Adding CI coverage
==================
CCCL's CI matrix is defined in ``ci/matrix.yaml``. You add coverage by writing new entries: each
entry expands into one or more jobs through the cross-product of its array-valued fields. Place the
entries under the right workflow sections, then validate them with the override matrix before merge.
The field reference for every entry — ``jobs``, ``project``, ``ctk``, ``cxx``, ``std``, ``gpu``,
``sm``, ``cmake_options``, ``args`` — lives in the ``tags:`` and ``jobs:`` maps in
``ci/matrix.yaml``. Read those before authoring an entry.
Choose the workflow sections to update
--------------------------------------
``ci/matrix.yaml`` defines separate matrices per trigger under ``workflows:``.
.. list-table::
:header-rows: 1
:widths: 20 80
* - Section
- Runs on
* - ``pull_request``
- Full coverage for projects modified in a PR. See :ref:`infra-ci-change-detection`.
* - ``pull_request_lite``
- Light coverage for project downstream of those modified in a PR. See :ref:`infra-ci-change-detection`.
* - ``nightly``
- Scheduled nightly. Broad compiler and CTK coverage.
* - ``weekly``
- Scheduled weekly. Widest coverage, including ``all-cccl`` architecture builds.
Add the pull request matrix entry
---------------------------------
Each row targets the specific compiler, CTK, and GPU.
Keep it narrow. Every array field multiplies the job count.
This entry adds a Thrust test run pinned to one CTK and one GPU, across three host compilers:
.. code-block:: yaml
- {jobs: ['test'], project: 'thrust', ctk: '<ctk-name>', std: 'max', cxx: ['<cxx-name-1>', '<cxx-name-2>', '<cxx-name-3>'], gpu: '<gpu-name>'}
Use the CTK name from the ``ctk_versions:`` map in ``ci/matrix.yaml`` and the compiler name
from the ``host_compilers:`` map. Use the GPU pool name from the ``gpus:`` map.
Field by field:
- ``jobs: ['test']`` — runs the ``test`` job. ``test`` requires a GPU and auto-generates its
``build`` producer job (see the ``jobs:`` section). Other projects may have more specialized
options besides build+test, but these are the most common.
- ``project: 'thrust'`` — restricts the entry to one project. Omit to use the default
``['libcudacxx', 'cub', 'thrust']``.
- ``ctk: '<ctk-name>'`` — the CTK name from the ``ctk_versions:`` map. See ``ci/matrix.yaml``
for the current names and what toolkit versions they resolve to. Prefer the convention of
``<major>.X`` when requesting "the latest of this major version", eg. "13.X" instead of "13.2",
and only use exact versions when meaningfully required (packaging constraints, minimum versions, etc).
- ``std: 'max'`` — the highest C++ standard the project supports. Use ``min``, ``minmax``, or
``all`` for wider coverage, or the standard year if specifics are needed (e.g. ``[17, 23]``).
- ``cxx: [...]`` — one or more host compiler names. The array expands to one job per element.
- ``gpu: '<gpu-name>'`` — the GPU runner pool to use, see the ``gpus:`` map.
- ``sm: [...]`` — Request specific CUDA SM architectures (eg. '75' for Turing). Use
``sm: 'gpu'`` to build for only the arch needed by the requested ``gpu``.
Add a build-only entry the same way with ``jobs: ['build']`` and no ``gpu``. Build jobs run on
CPU-only runners.
Add the pull request lite matrix entry
--------------------------------------
Rare, but if this is important coverage that is cheap, it may be worth adding a ``pull_request_lite`` entry.
Jobs from this matrix are added to the PR run when an upstream internal dependency is modified.
The goal is to keep this matrix as light as possible, extensions should be rare and well justified.
Note that these jobs will **NOT** run as part of the PR that adds them, so they **MUST** be tested with
the override matrix before merge.
See :ref:`infra-ci-change-detection` for how CCCL's CI encodes these dependencies.
Add the corresponding nightly / weekly entries
----------------------------------------------
The ``nightly`` and ``weekly`` entries run on a schedule, and carry the exhaustive, broad coverage that would
be wasteful and excessive for PRs.
It does not include the pull request matrix, so the PR jobs must be replicated, and possibly extended, here.
Note that these jobs will **NOT** run as part of the PR that adds them, so they **MUST** be tested with
the override matrix before merge.
Group the new entries under the existing comment headers in each section. Keep CTK and project
groupings together so the matrix stays readable.
Update project_files_and_dependencies.yaml
------------------------------------------
The full details of this system are documented in :ref:`infra-ci-change-detection`.
``ci/inspect_changes.py`` reads ``ci/project_files_and_dependencies.yaml`` to decide which projects
a PR touched. Update this file when the new coverage involves a project or a source path the file
does not already track. Skip this step when adding configurations to an existing project's existing
paths.
Add or extend a project entry so changed files map to the right matrix project:
.. code-block:: yaml
my_project_public:
name: "My Project Public API" # public/API entries only include public headers.
lite_dependencies: [libcudacxx_public] # lite dependency on libcu++'s public headers (upstream only triggers lite PR coverage)
full_dependencies: []
include_regexes: ["my_project/include/"] # path to public headers
my_project_internal:
name: "My Project Tests/Infra" # Internal entries exclude public headers, include everything else.
matrix_project: "my_project" # Maps to the matrix.yaml project that will be triggered when project files change
lite_dependencies: []
full_dependencies: [my_project_public] # trigger the full PR coverage when public headers change.
# changes to transitive deps (eg. libcudacxx_public) will onlytrigger lite PR coverage.
include_regexes: ["my_project/"] # path to project root
exclude_project_files: [my_project_public] # ignore files matched by the public entry.
- ``matrix_project`` — ties the change-detection key to the ``projects:`` key in ``ci/matrix.yaml``.
Without it the project never enters the build list.
- ``include_regexes`` — paths that mark this project dirty, anchored to the repo root.
- ``lite_dependencies`` / ``full_dependencies`` - See the file comments or :ref:`infra-ci-change-detection` for details.
**Files matching no project fall into ``core`` and trigger a full build of everything.**
If you're adding files that should not ever trigger CI, add them to the top-level ``ignore_regexes`` list to exclude them from
change detection.
Test the entry with the override matrix
---------------------------------------
Validate new entries with ``workflows.override`` before merge. A non-empty ``override`` replaces the
entire ``pull_request`` matrix for the PR, so CI runs only the entries you are testing. The override
blocks merge until removed, which guarantees the full suite runs before the change lands.
#. **Copy the candidate entries into override.** Place the new ``pull_request`` and ``nightly``
entries under ``workflows.override`` in ``ci/matrix.yaml``:
.. code-block:: yaml
workflows:
override:
- {jobs: ['test'], project: 'thrust', ctk: '<ctk-name>', std: 'max', cxx: ['<cxx-name-1>', '<cxx-name-2>'], gpu: '<gpu-name>'}
#. **Trim unrelated jobs.** Add ``[skip-tpt][skip-docs]`` to the **last commit message** to drop
third-party tests (eg RAPIDS, MatX) and doc builds while iterating.
#. **Push and inspect.** The PR runs only the override entries. Confirm the jobs appear with the
expected compiler, CTK, and GPU, and that they pass.
#. **Reset before merge.** Empty ``workflows.override`` and remove the ``[skip-*]`` tags from the
last commit message (or push a new commit). The merge gate fails until both are clean.
For a tighter loop on a single test target, use ``project: 'target'`` with ``args`` forwarded to
``ci/util/build_and_test_targets.sh``. The commented examples at the top of ``ci/matrix.yaml`` show
the ``run_cpu`` and ``run_gpu`` invocation patterns. Reproduce any failing job locally with
``.devcontainer/launch.sh`` and the matching ``ci/`` build or test script (see
:ref:`infra-ci-reproducing-locally`).

View File

@@ -0,0 +1,70 @@
.. _infra-devcontainer-adding-toolchain:
Adding a new devcontainer toolchain
===================================
A toolchain is one CTK version paired with one host compiler. CCCL generates a
devcontainer config for every combination listed in the ``devcontainers:`` section of
``ci/matrix.yaml``. The generated configs live under ``.devcontainer/<name>/devcontainer.json``,
one directory per combination, all produced by ``.devcontainer/make_devcontainers.sh [--clean]``.
Run the steps in order: edit the matrix, regenerate the configs, then verify before merge. The
base image for the combination must already exist in the
`rapidsai/devcontainers <https://github.com/rapidsai/devcontainers>`_ project before the matrix
edit.
Check that the base image exists
--------------------------------
Every CCCL devcontainer is built on a published ``rapidsai/devcontainers`` image. Image tags
follow this pattern::
rapidsai/devcontainers:<devcontainer_version>-cpp-<compiler><version>-cuda<ctk>[ext]
The ``-cuda<ctk>`` segment is present for every combination except nvhpc, which bundles its
own CUDA toolkit and omits it.
The ``<devcontainer_version>`` value is the ``devcontainer_version:`` field in ``ci/matrix.yaml``.
The images are maintained in the https://github.com/rapidsai/devcontainers/ repo, in the top-level
matrix file. If new images are required for the coverage, submit a PR against `main`.
Add the combination to ci/matrix.yaml
-------------------------------------
The source-of-truth when generating devcontainer toolchains is the ``matrix.yaml`` file. All jobs
from all workflows are parsed, the toolchains extracted, and the ``.devcontainer/...`` directories built.
At the bottom of the workflows section of ``matrix.yaml`` is a ``devcontainers:`` section.
This is intended to be a living mirror of the available images in the `rapidsai/devcontainers` repo,
and is useful for quickly checking supported CTK / host compilers while editing the matrix.
Occasionally we'll need a devcontainer that isn't referenced in any workflow, and this section is the place to add it.
Make sure that your new toolchain is listed and documented here.
Regenerate the devcontainer configs
-----------------------------------
From the repository root, regenerate every ``.devcontainer/<name>/devcontainer.json`` from the updated matrix:
.. code-block:: bash
.devcontainer/make_devcontainers.sh --clean
The script reads all matrix workflow entries, expands aliases, and writes one directory per combination
using the naming pattern ``cuda<version>[ext]-<compiler><version>``.
It also updates the root ``.devcontainer/devcontainer.json`` default to the newest GCC + newest
CUDA combination.
Pass ``--clean`` to remove directories for combinations no longer in the matrix (recommended).
Never hand-edit a generated ``.devcontainer/<name>/devcontainer.json``. Edits are overwritten on
the next run. To change settings that apply to every combination, edit the root
``.devcontainer/devcontainer.json`` template, then rerun the generator to propagate the change.
Verify before merge
-------------------
Locally test launching the devcontainer using the appropriate ``.devcontainer/launch.sh`` invocation.
See :ref:`infra-devcontainer-launching` for details on launching and using the devcontainer.
The ``verify-devcontainers`` CI workflow reruns ``make_devcontainers.sh --verbose --clean``
and fails if the result differs from the committed files.

View File

@@ -0,0 +1,115 @@
Run compile-time benchmarks
===========================
Use the compile-time benchmark when you want to compare how a change affects
CUDA TU compile time. The common workflow compares the current tree against
``origin/main`` and reports the most important movements in generated public
include-check TUs.
For the full option, CSV, filter, and CI contract reference, see
:doc:`../references/compile_time`.
Run the common comparison
-------------------------
From the repository root, run:
.. code-block:: bash
ci/build_compile_time_bench.sh \
-baseline-ref origin/main \
-- --slices /path/to/slices.json
The CI slices live in ``ci/matrix.yaml`` under
``compile_time.pull_request[].slices``. To reproduce the PR shape locally, copy
those slice definitions to a JSON file shaped as:
.. code-block:: json
{
"slices": [
{
"id": "file-processing",
"title": "Direct file processing",
"filter": "file-processing",
"timing": "exclusive",
"sort": "total",
"top": 15,
"threshold": 0.2
}
]
}
The wrapper builds both the current tree and the baseline commit with the same
preset, target set, architecture, and other build arguments. It then writes
baseline reports, current reports, comparison CSVs, a ``summary.json`` manifest,
raw traces, and Perfetto-friendly traces under the preset build directory:
.. code-block:: text
build/<infix>/<preset>/compile_time/
Run a quick single-slice report
-------------------------------
For iteration after traces already exist, skip the build and regenerate only an
event report:
.. code-block:: bash
ci/build_compile_time_bench.sh -skip-build -- \
-f file-processing -e --sort total -n 25
Other useful built-in filters include:
- ``total-compilation``
- ``file-processing``
- ``scanning-function-body``
- ``template-instantiation``
- ``host-compiler``
- ``code-generation``
- ``all``
Interpret the PR comment
------------------------
The PR comment contains one section per configured slice. Within each slice,
regressions and improvements are intentionally separated. Rows are ranked by
total impact across matched traces, not just by the largest single-TU movement.
Important columns:
- ``Regression impact`` / ``Improvement impact``: absolute total-impact change
across matched traces, in seconds.
- ``Selected Δ``: signed movement in the selected metric for the event.
- ``Baseline`` / ``Current``: selected metric values on each side.
- ``Matched traces``: number of generated TUs where the event key was comparable.
Small movements are filtered by per-slice thresholds from ``ci/matrix.yaml``.
Those thresholds are intentionally non-zero to hide ordinary run-to-run noise.
Inspect traces in Perfetto
--------------------------
The wrapper prepares trace copies whose event names include the useful file or
symbol detail. Open files under:
.. code-block:: text
build/<infix>/<preset>/compile_time/perfetto_traces/
in Perfetto or another Chrome-trace-compatible viewer.
Skip compile-time benchmark telemetry
-------------------------------------
Compile-time benchmark jobs are informational and do not gate the aggregate PR
``CI`` job. For early iterations where the compile-time benchmark is unrelated,
append this case-sensitive tag to the commit message:
.. code-block:: text
[skip-compile-time-bench]
Remove the tag before requesting final review if the compile-time benchmark
scripts, workflow, or configuration changed.

View File

@@ -0,0 +1,13 @@
How Tos
=======
How-to guides for maintainers.
.. toctree::
:maxdepth: 1
../backport_process
compile_time
plan_work_with_github_issues
adding_ci_coverage
adding_new_devcontainer

View File

@@ -0,0 +1,108 @@
How To Plan Work with GitHub Issues
===================================
CCCL uses `GitHub issues <https://github.com/NVIDIA/cccl/issues>`__ and the
`CCCL GitHub Project <https://github.com/orgs/NVIDIA/projects/6>`__ as the source of truth to make
planned work visible, understandable, and coordinated.
This guide explains how CCCL plans work, records that
work in closable issues, and keeps priorities and ownership visible over time.
CCCL plans work in monthly sprints, with each sprint identifying the issues the
team intends to prioritize during that 4-week window.
Understand current planned work
-------------------------------
The `Current Sprint <https://github.com/orgs/NVIDIA/projects/6/views/47>`__ view in the GitHub Project contains
the planned work the team has agreed to prioritize now. It is not a complete
list of all engineering activity.
Planned work intentionally leaves capacity for reviews, support, debugging, and
other interrupt-driven work.
In-progress work is not automatically carried forward into the next planning
cycle. Each cycle is planned from current priorities.
Write a closable issue
----------------------
Every issue should answer:
"This issue can be closed when..."
Examples:
- a bug is fixed;
- a refactoring is complete;
- a benchmark is added;
- a design decision is documented;
- follow-on issues are created;
- an investigation summary is written.
Large efforts can start as tracking issues whose scope is still evolving. As
concrete work becomes clear, create sub-issues for work with independent close
conditions.
Propose planned work
--------------------
Sprint planning happens monthly. Team leads review the roadmap, current priorities,
and incoming requests to identify the highest-impact work for the upcoming
four-week planning window.
If you think work should be considered for the next planning cycle, raise it
during planning or ask a team lead to add it to the
`Sprint Planning <https://github.com/orgs/NVIDIA/projects/6/views/58>`__ view.
During planning, the team reviews proposed issues, confirms ownership,
identifies gaps or dependencies, and checks that issues are actionable and
up-to-date.
Confirmed planned work is moved to the
`Current Sprint <https://github.com/orgs/NVIDIA/projects/6/views/47>`__ view.
Keep assigned issues current
----------------------------
The assignee is responsible for driving the issue forward and keeping it
accurate while the work evolves.
Update the issue when:
- the title no longer describes the work clearly;
- the close condition changes;
- the work has been split into sub-issues;
- important context, decisions, or blockers appear;
- the issue is no longer relevant and should be closed.
Use issue comments for questions, decisions, blockers, and context that
others may need to find later. If a comment changes the issue's scope or close
condition, update the issue body as well.
Complete assigned issues
------------------------
Use the issue's close condition to decide when the work is done. Often, this
means opening a PR that completes the work described by the issue. Follow the
:doc:`contributing guidelines </cccl/contributing>` for how to prepare and
submit the PR.
Link the PR to the issue so GitHub can close the issue when the PR merges. For
example, include ``Fixes #123`` in the PR description.
Linked PRs also keep the GitHub Project status current. A linked draft PR keeps
the issue ``In Progress``. When the PR is ready for review, automation moves the
issue to ``Review``.
Plan release-critical work
--------------------------
Upcoming releases have corresponding
`GitHub milestones <https://github.com/NVIDIA/cccl/milestones>`__.
Team leads create release milestones and assign issues to them during planning.
Add an issue to a release milestone when the work is
important to complete and include in that release.
Not all current planned work is tied to a particular release.