feat(cccl): integrate missing CCCL directories — python/, ci/, .agent/, docs/, test/
Sparse-checkout from NVIDIA/cccl main branch to complete cccl_upstream: Added: - python/cuda_cccl/ (226 files) — Python bindings for device-level algorithms Critical for muh toolchain: cuda.compute.reduce_into, scan, radix_sort, etc. Includes 204 .py files with full test coverage for all 27 algorithms - ci/ (163 files) — Build/test infrastructure build_cub.sh, test_cub.sh, build_and_test_targets.sh, matrix.yaml Directly maps to our [INFRA-CI] and [INFRA-BUILD] items - .agent/skills/ (7 files) — NVIDIA's own agent skills for CCCL cccl-style/SKILL.md, cccl-test/SKILL.md, sass-diff/SKILL.md - docs/ (491 files) — Official CCCL documentation CI references, CMake guides, Python compute docs, libcudacxx PTX docs - test/ (12 files) — Top-level integration tests (cuda_smoke, stdpar) - Root configs: .clang-format, .clang-tidy, CONTRIBUTING.md, pyproject.toml - CLAUDE.md symlink → AGENTS.md (NVIDIA's standard) cccl_upstream now mirrors full NVIDIA/cccl structure: Before: 42M (cub + thrust + libcudacxx + cudax + c + examples + benchmarks) After: 53M (+python +ci +docs +.agent +test +configs) This completes the CCCL base needed for: - [muh-bench] items: ci/util/build_and_test_targets.sh for targeted builds - [CCCL-verify] items: python/cuda_cccl/tests/ as reference implementations - [CCCL-test] items: ci/test_cub.sh, ci/test_thrust.sh - Agent workflow: .agent/skills/ for consistent style and test patterns
This commit is contained in:
101
cccl_upstream/docs/contributors/how_tos/adding_build_target.rst
Normal file
101
cccl_upstream/docs/contributors/how_tos/adding_build_target.rst
Normal file
@@ -0,0 +1,101 @@
|
||||
.. _infra-cmake-adding-build-target:
|
||||
|
||||
Adding a build target
|
||||
=====================
|
||||
|
||||
In most cases, new C++ test and example sources are detected automatically.
|
||||
Their executables should appear after re-configuring / re-building.
|
||||
When in doubt, check the CMakeLists.txt in the source directory to discover the conventions.
|
||||
|
||||
This document is intended for the rare case where a developer needs to build project infrastructure from scratch.
|
||||
|
||||
``cccl_add_executable`` wraps ``add_executable`` with CCCL's standard target
|
||||
configuration: dialect handling, output directories, metatarget registration,
|
||||
and clang-tidy integration. Use it for every test, example, benchmark, and tool
|
||||
in the tree. The function lives in ``cmake/CCCLAddExecutable.cmake``.
|
||||
|
||||
Signature
|
||||
---------
|
||||
|
||||
::
|
||||
|
||||
cccl_add_executable(target_name
|
||||
SOURCES <source1> [source2 ...]
|
||||
[ADD_CTEST]
|
||||
[NO_METATARGETS]
|
||||
[NO_CLANG_TIDY]
|
||||
[METATARGET_PATH <path>]
|
||||
[DIALECT <standard>]
|
||||
)
|
||||
|
||||
The first positional argument is the target name. The remaining arguments are
|
||||
keyword options:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 25 15 60
|
||||
|
||||
* - Argument
|
||||
- Kind
|
||||
- Effect
|
||||
* - ``SOURCES``
|
||||
- Required
|
||||
- Source files for the executable. The function hard-fails with a fatal
|
||||
error if absent.
|
||||
* - ``ADD_CTEST``
|
||||
- Flag
|
||||
- Registers a CTest with the same name as the target, running the
|
||||
executable with no arguments.
|
||||
* - ``NO_METATARGETS``
|
||||
- Flag
|
||||
- Skips metatarget registration. The target builds only by its own name.
|
||||
* - ``NO_CLANG_TIDY``
|
||||
- Flag
|
||||
- Skips clang-tidy integration for these sources.
|
||||
* - ``METATARGET_PATH``
|
||||
- One value
|
||||
- Dotted path placing the target in the metatarget hierarchy. Defaults to
|
||||
``target_name``.
|
||||
* - ``DIALECT``
|
||||
- One value
|
||||
- C++ standard override for this target (for example ``17`` or ``20``).
|
||||
|
||||
Add a test target
|
||||
-----------------
|
||||
|
||||
**Step 1. Call the function with SOURCES and ADD_CTEST.** Name the target with
|
||||
its dotted hierarchy path so the metatarget system groups it correctly::
|
||||
|
||||
cccl_add_executable(cub.test.device_reduce
|
||||
SOURCES test_device_reduce.cu
|
||||
ADD_CTEST
|
||||
)
|
||||
|
||||
This creates the executable ``cub.test.device_reduce``, registers a CTest of the
|
||||
same name, and adds it to the ``cub`` and ``cub.test`` metatargets.
|
||||
|
||||
**Step 2. Link the target's dependencies.** ``cccl_add_executable`` configures
|
||||
the target but does not link libraries. Add them after the call::
|
||||
|
||||
target_link_libraries(cub.test.device_reduce
|
||||
PRIVATE
|
||||
cub.compiler_interface
|
||||
cccl.c2h
|
||||
)
|
||||
|
||||
Place the target in the hierarchy
|
||||
---------------------------------
|
||||
|
||||
By default the metatarget path equals the target name. A target named
|
||||
``foo.bar.baz`` builds via metatargets ``foo`` and ``foo.bar``. Running
|
||||
``ninja foo`` builds every descendant; ``ninja foo.bar`` builds that subtree.
|
||||
|
||||
Use ``METATARGET_PATH`` to decouple the target name from its hierarchy
|
||||
position. Rare; advanced use only.
|
||||
|
||||
Exclude a target from the hierarchy
|
||||
-----------------------------------
|
||||
|
||||
Pass ``NO_METATARGETS`` for targets that should not appear in the test
|
||||
hierarchy: benchmarks, standalone tools, and anything outside the
|
||||
build-everything-and-test workflow. Rare; advanced use only.
|
||||
14
cccl_upstream/docs/contributors/how_tos/index.rst
Normal file
14
cccl_upstream/docs/contributors/how_tos/index.rst
Normal file
@@ -0,0 +1,14 @@
|
||||
How Tos
|
||||
=======
|
||||
|
||||
How-to guides for contributors.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
install_build_test
|
||||
targeted_builds
|
||||
launching
|
||||
reproducing_ci
|
||||
override_matrix
|
||||
adding_build_target
|
||||
162
cccl_upstream/docs/contributors/how_tos/install_build_test.rst
Normal file
162
cccl_upstream/docs/contributors/how_tos/install_build_test.rst
Normal file
@@ -0,0 +1,162 @@
|
||||
.. _infra-install-build-test:
|
||||
|
||||
Install, Build, Test
|
||||
====================
|
||||
|
||||
CCCL provides several developer interfaces for working with the codebase.
|
||||
Purpose-built scripts exist that drive routine work, and using the wrong tool can waste hours of developer time.
|
||||
For example, the ``ci/test_*.sh`` scripts can take hours to run a full validation suite, while the
|
||||
``ci/util/build_and_test_targets.sh`` tool configures, builds, and runs a small subset in seconds.
|
||||
Prebuilt development containers simplify working in specific toolchains and environments.
|
||||
Core members can launch GitHub Actions benchmarking / bisection workflows that run on cloud infrastructure.
|
||||
|
||||
Pick a path by goal
|
||||
-------------------
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 34 38 14 14
|
||||
|
||||
* - Goal
|
||||
- Tools
|
||||
- Type
|
||||
- Availability
|
||||
* - Install CCCL headers to a prefix
|
||||
- ``ci/install_cccl.sh``
|
||||
- Script
|
||||
- Public
|
||||
* - Build and run a specific test
|
||||
- ``ci/util/build_and_test_targets.sh``
|
||||
- Script
|
||||
- Public
|
||||
* - Build or test an entire project
|
||||
- ``ci/build_<project>.sh`` / ``ci/test_<project>.sh``
|
||||
- Script
|
||||
- Public
|
||||
* - Bisect a regression
|
||||
- ``ci/util/git_bisect.sh``, git-bisect.yml
|
||||
- Script, GHA
|
||||
- Public / members
|
||||
* - Request a benchmark comparison
|
||||
- ``ci/bench/bench.sh``, bench.yml, ``ci/bench.yaml``
|
||||
- Script, GHA, PR tool
|
||||
- Public / members
|
||||
* - Custom build
|
||||
- ``cmake --preset``
|
||||
- Script
|
||||
- Public
|
||||
|
||||
Install only
|
||||
~~~~~~~~~~~~
|
||||
|
||||
``ci/install_cccl.sh <prefix>`` copies CCCL's headers and CMake config files into a prefix
|
||||
directory. CCCL is header-only, so the install has no build step and finishes in seconds.
|
||||
|
||||
Use it when a downstream project needs ``find_package(CCCL)`` against a fixed checkout, or when you
|
||||
want CCCL on a system include path without cloning into the consumer's tree.
|
||||
|
||||
Build and run a specific test
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
``ci/util/build_and_test_targets.sh`` configures one preset, builds the Ninja targets you name,
|
||||
and runs the CTest or lit tests you name. It is the fast-iteration path for a single test or a
|
||||
handful of targets.
|
||||
|
||||
::
|
||||
|
||||
./ci/util/build_and_test_targets.sh \
|
||||
--preset cub-cpp20 \
|
||||
--build-targets "cub.test.iterator" \
|
||||
--ctest-targets "cub.test.iterator"
|
||||
|
||||
Use it when you are fixing one test and want a tight edit-build-run loop. Building tests does not
|
||||
require a GPU; running them does.
|
||||
|
||||
Need a specific CTK or host compiler? Launch the matching container with ``.devcontainer/launch.sh``
|
||||
first, then run the script inside it. Valid toolchain combinations are in the
|
||||
``devcontainers:`` section of ``ci/matrix.yaml``; launching is covered at
|
||||
:ref:`infra-devcontainer-launching`.
|
||||
|
||||
Build or test an entire project
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
``ci/build_<project>.sh`` and ``ci/test_<project>.sh`` build or test a full project across a host
|
||||
compiler, C++ standard, and architecture set. These are the scripts CI runs, so they reproduce a
|
||||
CI job exactly.
|
||||
|
||||
::
|
||||
|
||||
./ci/build_cub.sh -cxx g++ -std 17 -arch "75;80;90"
|
||||
./ci/test_cub.sh -cxx g++ -std 17 -arch "75;80;90"
|
||||
|
||||
Use them to reproduce a CI failure or to validate a project end to end before pushing. A full
|
||||
project build takes hours; a targeted ``build_and_test_targets.sh`` run takes minutes. Test scripts
|
||||
require a GPU.
|
||||
|
||||
Need a specific toolchain? Run these inside a devcontainer launched with ``.devcontainer/launch.sh``
|
||||
(:ref:`infra-devcontainer-launching`). CI failure logs print the exact container and arguments to
|
||||
reproduce the job.
|
||||
|
||||
Bisect a regression
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
``ci/util/git_bisect.sh`` walks the commit history between a good and a bad ref, building and
|
||||
testing each candidate, until it pins the commit that introduced a regression. It takes the same
|
||||
``--preset``, ``--build-targets``, and ``--ctest-targets`` arguments as
|
||||
``build_and_test_targets.sh``.
|
||||
|
||||
::
|
||||
|
||||
./ci/util/git_bisect.sh \
|
||||
--preset cub-cpp20 \
|
||||
--build-targets "cub.test.iterator" \
|
||||
--ctest-targets "cub.test.iterator" \
|
||||
--good-ref v1.13.0 \
|
||||
--bad-ref origin/main
|
||||
|
||||
Use it when a test passes on an old ref and fails on ``main`` and you need the offending commit.
|
||||
``--repeat N`` re-runs multiple times to help catch intermittent failures.
|
||||
|
||||
Need a specific toolchain? Run the script inside a devcontainer launched with
|
||||
``.devcontainer/launch.sh`` (:ref:`infra-devcontainer-launching`).
|
||||
|
||||
Members can run the same bisect remotely on CI machines through the `Git Bisect workflow
|
||||
<https://github.com/NVIDIA/cccl/actions/workflows/git-bisect.yml>`_: choose a runner, set the good
|
||||
and bad refs and target arguments, and dispatch. Source is ``.github/workflows/git-bisect.yml``.
|
||||
|
||||
Request a benchmark comparison
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Compare benchmark results between two refs from the browser, a local script, or a pull request.
|
||||
|
||||
Members can dispatch the `Benchmark Compare workflow
|
||||
<https://github.com/NVIDIA/cccl/actions/workflows/bench.yml>`_ from the browser: choose "Run
|
||||
workflow", set the base and test refs and the CUB and Python filters, and it runs on the CI GPU
|
||||
pool. Source is ``.github/workflows/bench.yml``.
|
||||
|
||||
Locally, ``ci/bench/bench.sh <base> <test>`` runs the same comparison against checked-out refs::
|
||||
|
||||
./ci/bench/bench.sh origin/main HEAD --cub-filter "^cub\.bench\.copy\.memcpy\.base$"
|
||||
|
||||
It wraps ``ci/bench/compare_git_refs.sh`` and ``ci/bench/compare_paths.sh``; call those directly
|
||||
when you already have two checkouts.
|
||||
|
||||
To benchmark inside a PR, edit ``ci/bench.yaml`` to set GPUs and filters and push; PR CI detects the
|
||||
diff from ``ci/bench.template.yaml`` and dispatches the jobs. Reset ``ci/bench.yaml`` to match the
|
||||
template before merging. Argument behavior and artifact layout live in ``ci/bench/README.md``.
|
||||
|
||||
Full CMake control
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
``cmake --preset <name>`` configures a build directory directly.
|
||||
This is discouraged, but available for custom workflows / tool integrations.
|
||||
|
||||
::
|
||||
|
||||
cmake --preset all-dev
|
||||
cmake --build --preset all-dev
|
||||
ctest --preset all-dev
|
||||
|
||||
List the presets with ``cmake --list-presets``. The ``all-dev`` preset enables every library,
|
||||
test, and example against your native GPU; per-library presets like ``cub-cpp20`` scope the build
|
||||
to one library and standard. The :ref:`infra-cmake-preset-reference` catalogs the full preset set.
|
||||
89
cccl_upstream/docs/contributors/how_tos/launching.rst
Normal file
89
cccl_upstream/docs/contributors/how_tos/launching.rst
Normal file
@@ -0,0 +1,89 @@
|
||||
.. _infra-devcontainer-launching:
|
||||
|
||||
Launching a container
|
||||
=====================
|
||||
|
||||
``.devcontainer/launch.sh`` starts a CCCL development container with a chosen CUDA
|
||||
toolkit and host compiler, mounts the repo, and either opens VSCode or drops you into
|
||||
a shell. It is Linux-only (including WSL2). With no toolchain flags, it uses the default devcontainer
|
||||
in ``.devcontainer/devcontainer.json``, which uses the latest CTK + gcc.
|
||||
|
||||
Launch in VSCode
|
||||
----------------
|
||||
|
||||
**Open the default container.** Run the script with no flags.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
.devcontainer/launch.sh
|
||||
|
||||
The script copies the selected ``devcontainer.json`` into a temporary directory and
|
||||
opens VSCode against it.
|
||||
|
||||
**Select a toolchain.** Pass ``--cuda`` and ``--host`` to open a specific variant.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
.devcontainer/launch.sh --cuda <cuda-version> --host <host-compiler>
|
||||
|
||||
The temporary-directory copy lets you run multiple variants of the same environment
|
||||
side by side, each in its own VSCode window.
|
||||
|
||||
Launch directly in Docker
|
||||
-------------------------
|
||||
|
||||
**Drop into a shell.** Add ``--docker`` to skip VSCode and run a bash shell inside the
|
||||
container.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
.devcontainer/launch.sh --docker --cuda <cuda-version> --host <host-compiler>
|
||||
|
||||
The container mounts the repo at ``/home/coder/cccl`` and removes itself on exit. Any
|
||||
trailing arguments after the flags run as a command instead of an interactive shell.
|
||||
|
||||
Specify the toolchain
|
||||
---------------------
|
||||
|
||||
``--cuda`` selects the CUDA toolkit version. ``--host`` selects the host compiler.
|
||||
The two flags resolve to ``.devcontainer/cuda<cuda>-<host>/devcontainer.json``; an
|
||||
unknown combination exits with an error.
|
||||
|
||||
Valid values come from the devcontainers located under ``.devcontainer/``.
|
||||
|
||||
Pass through GPUs
|
||||
-----------------
|
||||
|
||||
**Add host GPUs.** Pass ``--gpus all`` to expose every host GPU to the container.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
.devcontainer/launch.sh --docker --cuda <cuda-version> --host <host-compiler> --gpus all
|
||||
|
||||
``--gpus`` takes any Docker GPU request string. It overrides the ``hostRequirements.gpu``
|
||||
default read from the devcontainer config. Without it, the container starts without GPU
|
||||
access, which is sufficient for building tests.
|
||||
|
||||
Launch from a git worktree
|
||||
--------------------------
|
||||
|
||||
``launch.sh`` handles linked worktrees automatically. A worktree's ``.git`` is a file
|
||||
pointing at the main repository's git directory, which the container cannot reach through
|
||||
the worktree mount alone. The script bind-mounts the main repo's git common directory at
|
||||
its host path so git operations resolve inside the container.
|
||||
|
||||
The ``cccl-build`` and ``cccl-wheelhouse`` Docker volumes are shared across all worktrees
|
||||
and the main checkout, so build artifacts collide between them. Do not run multiple
|
||||
worktree containers concurrently unless you are careful to avoid conflicts.
|
||||
|
||||
First-time git auth in a worktree needs the main checkout's ``.config/gh``. If the main
|
||||
checkout has never run its container, ``launch.sh`` warns that startup will block on an
|
||||
interactive ``gh auth login``. Launch the main checkout's container once, complete the
|
||||
login, then re-launch the worktree.
|
||||
|
||||
Forward SSH keys
|
||||
----------------
|
||||
|
||||
If ``SSH_AUTH_SOCK`` is set in your environment, ``launch.sh`` forwards the agent socket
|
||||
into the container automatically. No flag is required. Git operations over SSH inside the
|
||||
container use your host agent's keys.
|
||||
109
cccl_upstream/docs/contributors/how_tos/override_matrix.rst
Normal file
109
cccl_upstream/docs/contributors/how_tos/override_matrix.rst
Normal file
@@ -0,0 +1,109 @@
|
||||
.. _infra-ci-override-matrix:
|
||||
|
||||
Override matrix
|
||||
===============
|
||||
|
||||
The override matrix scopes a pull request's CI to a chosen subset of jobs.
|
||||
When the ``workflows.override`` key in ``ci/matrix.yaml`` is non-empty, it
|
||||
replaces the entire ``pull_request`` matrix. The PR runs only the override
|
||||
jobs, and branch protection blocks the merge until the override is empty
|
||||
again.
|
||||
|
||||
Use the override matrix to:
|
||||
|
||||
- Test a new compiler's nightly / weekly jobs from the PR before merging.
|
||||
- Validate a compiler-specific or GPU-specific fix against one combo.
|
||||
- Test CI infrastructure changes that need only a few jobs to validate.
|
||||
- Debug a nightly failure by running only the combos that failed.
|
||||
|
||||
The override matrix is a temporary scoping tool, not a permanent matrix
|
||||
edit. Every override entry must be removed before the PR lands.
|
||||
|
||||
Add an override entry
|
||||
---------------------
|
||||
|
||||
The override lives at the top of ``ci/matrix.yaml`` under
|
||||
``workflows.override``. The default value is empty. Override entries use the
|
||||
same syntax as ``pull_request`` entries.
|
||||
|
||||
**Step 1. Pick the combo to test.** Identify the exact job, project,
|
||||
compiler, CTK version, and GPU you need. For a compiler-specific fix, this is
|
||||
one ``cxx`` value. For a nightly failure, copy the failing entry from the
|
||||
``nightly`` workflow.
|
||||
|
||||
**Step 2. Add the entry under** ``override``. Edit ``ci/matrix.yaml`` and
|
||||
add one mapping under the ``override:`` key:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
workflows:
|
||||
override:
|
||||
- {jobs: ['test'], project: 'thrust', std: 'max', ctk: '<ctk>', cxx: '<compiler>', gpu: '<gpu>'}
|
||||
|
||||
pull_request:
|
||||
- <...>
|
||||
|
||||
Choose ``ctk``, ``cxx``, and ``gpu`` values from the existing
|
||||
``pull_request`` entries in ``ci/matrix.yaml``. Each field scopes the run:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 20 80
|
||||
|
||||
* - Field
|
||||
- Meaning
|
||||
* - ``jobs``
|
||||
- Job types to run. A ``test`` entry auto-generates any build jobs it
|
||||
depends on.
|
||||
* - ``project``
|
||||
- Which project to build or test (``thrust``, ``cub``, ``libcudacxx``, ``cudax``, ...).
|
||||
* - ``std``
|
||||
- C++ standard. ``max`` selects the highest standard the combo supports.
|
||||
* - ``ctk``
|
||||
- CUDA Toolkit version. A ``<major>.X`` suffix selects the newest image for that major version.
|
||||
* - ``cxx``
|
||||
- Host compiler. A single value runs one compiler; an array expands to several jobs.
|
||||
* - ``gpu``
|
||||
- GPU runner model. Required for ``test`` jobs.
|
||||
|
||||
Field defaults and the full tag list live in the ``tags`` section of
|
||||
``ci/matrix.yaml``.
|
||||
|
||||
**Step 3. Trim turnaround with targeted builds.** A full project build is
|
||||
slow. To build and run a single test target instead, use ``project:
|
||||
'target'`` and pass ``args`` to ``ci/util/build_and_test_targets.sh``:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
workflows:
|
||||
override:
|
||||
- {jobs: ['run_gpu'], project: 'target', ctk: '<ctk>', cxx: '<compiler>', gpu: '<gpu>',
|
||||
args: '--preset <preset> --build-targets "<target>" --ctest-targets "<target>"'}
|
||||
|
||||
The ``run_cpu`` and ``run_gpu`` jobs map directly to
|
||||
``build_and_test_targets.sh``. See that script for the available ``args``,
|
||||
covered in :doc:`/cccl/development/build_and_bisect_tools`.
|
||||
|
||||
**Step 4. Reduce overhead further with skip tags.** Combine the override
|
||||
with :ref:`[skip-*] tags <infra-ci-skip-tags>` in the last commit message to drop
|
||||
devcontainer, docs, and third-party canary jobs:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git commit -m "Debug <compiler> <project> failure [skip-vdc][skip-docs][skip-tpt]"
|
||||
|
||||
Run scoped CI and merge
|
||||
-----------------------
|
||||
|
||||
The override is temporary: scoped jobs first, full matrix before merge.
|
||||
|
||||
1. **Add the override entry and push.** Only the override jobs run; the
|
||||
``pull_request`` matrix is skipped while ``workflows.override`` is non-empty.
|
||||
|
||||
2. **Iterate until the override jobs pass.** Each push reruns only those
|
||||
jobs, keeping turnaround short.
|
||||
|
||||
3. **Empty the override and push again.** With ``workflows.override`` reset
|
||||
to empty, the full ``pull_request`` matrix runs — the suite that gates merge.
|
||||
|
||||
4. **Merge once the full matrix is green.**
|
||||
121
cccl_upstream/docs/contributors/how_tos/reproducing_ci.rst
Normal file
121
cccl_upstream/docs/contributors/how_tos/reproducing_ci.rst
Normal file
@@ -0,0 +1,121 @@
|
||||
.. _infra-ci-reproducing-locally:
|
||||
|
||||
Reproducing CI locally
|
||||
======================
|
||||
|
||||
A failing CI job prints everything needed to reproduce it. Open the failed
|
||||
job from the PR's checks list and expand the log.
|
||||
|
||||
On failure: reproduction block
|
||||
------------------------------
|
||||
|
||||
When a job exits non-zero, the runner emits an
|
||||
**Instructions to Reproduce CI Failure Locally** block. It contains two steps:
|
||||
|
||||
**Step 1** — clone the repository at the exact SHA under test:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git clone --branch <branch> --single-branch https://github.com/NVIDIA/cccl.git \
|
||||
&& cd cccl && git checkout <sha>
|
||||
|
||||
**Step 2** — launch the same container and re-run the same command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
.devcontainer/launch.sh -d -c <cuda> -H <host> -- <command>
|
||||
|
||||
Copy both lines verbatim. The ``-c`` and ``-H`` values are the CTK version and
|
||||
host compiler for this job. ``<command>`` is the ``ci/*.sh`` invocation the
|
||||
runner used, with all flags.
|
||||
|
||||
.. note::
|
||||
|
||||
GPU test jobs omit the GPU flag from the printed command. Add ``--gpus all``
|
||||
when reproducing a test job on a machine with a GPU.
|
||||
|
||||
Job Inputs block
|
||||
----------------
|
||||
|
||||
The **Job Inputs** group near the top of every log records the full job
|
||||
configuration, printed before the container launches:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 30 70
|
||||
|
||||
* - Field
|
||||
- Meaning
|
||||
* - ``Job command``
|
||||
- The ``ci/*.sh`` script and flags the runner will execute
|
||||
* - ``JOB_ID``
|
||||
- Unique job identifier; pass it to ``create_mock_job_env.sh`` to reproduce the job's
|
||||
environment locally
|
||||
* - ``JOB_CUDA``
|
||||
- CTK version (e.g. ``13.3``)
|
||||
* - ``JOB_HOST``
|
||||
- Host compiler identifier (e.g. ``gcc15``)
|
||||
* - ``JOB_IMAGE``
|
||||
- Full RAPIDS devcontainer image tag pulled for this job
|
||||
* - ``JOB_RUNNER``
|
||||
- Runner label; labels containing ``-gpu-`` mean the job used a GPU
|
||||
* - ``JOB_ENVIRONMENT``
|
||||
- Extra environment variables injected into the container
|
||||
|
||||
These are the same values that feed the ``launch.sh`` invocation in the
|
||||
failure block.
|
||||
|
||||
Override matrix entry
|
||||
---------------------
|
||||
|
||||
The **Override matrix entry** block, also printed before the container
|
||||
launches, contains a YAML snippet:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
- {jobs: [...], project: '...', ctk: '...', cxx: '...', ...}
|
||||
|
||||
Paste this into the ``workflows.override`` list in ``ci/matrix.yaml`` and push
|
||||
to re-run only that specific job in CI without waiting for the full matrix.
|
||||
See :ref:`infra-ci-override-matrix` for the override workflow.
|
||||
|
||||
Mock the job environment
|
||||
------------------------
|
||||
|
||||
.. note::
|
||||
|
||||
This step is only needed when debugging the CCCL workflow / artifact scripts (rare).
|
||||
More project contributors can safely skip this.
|
||||
|
||||
To run the ``ci/util/workflow/`` and ``ci/util/artifacts/`` scripts outside CI —
|
||||
debugging artifact upload, download, or producer resolution locally — recreate the
|
||||
job's environment with ``create_mock_job_env.sh``:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ci/util/create_mock_job_env.sh <run_id> <job_id>
|
||||
|
||||
Run it inside a devcontainer. It takes the ``<run_id>`` and ``<job_id>`` from the job
|
||||
log (``GITHUB_RUN_ID`` and ``JOB_ID``), sets the ``GITHUB_*`` variables CI exports,
|
||||
sources the workflow and artifact helpers, clears stale local artifact directories, and
|
||||
drops into a shell that mimics the in-container CI environment. Every job log prints the
|
||||
exact command under ``Mock with:``.
|
||||
|
||||
Tighten the loop
|
||||
----------------
|
||||
|
||||
The full ``ci/build_*.sh`` and ``ci/test_*.sh`` scripts build and run an
|
||||
entire project's test suite. Once the container reproduces the failure, use
|
||||
``ci/util/build_and_test_targets.sh`` to build and run only the failing target:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
.devcontainer/launch.sh --docker --cuda 13.0 --host gcc14 --gpus all -- \
|
||||
ci/util/build_and_test_targets.sh \
|
||||
--preset cub-cpp20 \
|
||||
--build-targets "cub.test.iterator" \
|
||||
--ctest-targets "cub.test.iterator"
|
||||
|
||||
This rebuilds one target instead of the full project. For preset, target, and
|
||||
lit-test flags, and for ``ci/util/git_bisect.sh`` to find the introducing commit,
|
||||
see :doc:`/cccl/development/build_and_bisect_tools`.
|
||||
75
cccl_upstream/docs/contributors/how_tos/targeted_builds.rst
Normal file
75
cccl_upstream/docs/contributors/how_tos/targeted_builds.rst
Normal file
@@ -0,0 +1,75 @@
|
||||
.. _infra-ci-targeted-builds:
|
||||
|
||||
Build and run targeted tests
|
||||
============================
|
||||
|
||||
The ``ci/build_*.sh`` and ``ci/test_*.sh`` scripts build and run all headers,
|
||||
tests, examples, etc for a single project. It is the right tool for reproducing
|
||||
a CI job, but it is slow when you are iterating on a single test.
|
||||
|
||||
``ci/util/build_and_test_targets.sh`` builds and runs a named subset of CMake
|
||||
targets against one preset. Use it to compile one test, run one CTest pattern,
|
||||
or execute one libcudacxx lit test without rebuilding the rest of the project.
|
||||
The full flag reference is in :doc:`/cccl/development/build_and_bisect_tools`.
|
||||
|
||||
Build a single CUB test
|
||||
-----------------------
|
||||
|
||||
CCCL tests usually have a single name for their CMake target, ninja target, and CTest target.
|
||||
It uniquely encodes the project, path, and test case, eg: ``cub.test.iterator``.
|
||||
|
||||
#. **Configure and build the target.** Pass the preset and the metatarget to
|
||||
``--build-targets``::
|
||||
|
||||
ci/util/build_and_test_targets.sh \
|
||||
--preset cub-cpp20 \
|
||||
--build-targets "cub.test.iterator"
|
||||
|
||||
With no ``--ctest-targets``, the script configures and compiles, then stops.
|
||||
Compiling a test does not require a GPU.
|
||||
|
||||
#. **Run the target.** Add ``--ctest-targets`` with a CTest ``-R`` regex. The
|
||||
metatarget name works directly as the pattern::
|
||||
|
||||
ci/util/build_and_test_targets.sh \
|
||||
--preset cub-cpp20 \
|
||||
--build-targets "cub.test.iterator" \
|
||||
--ctest-targets "cub.test.iterator"
|
||||
|
||||
If ``--build-targets`` is omitted, the script assumes the targets are already built and
|
||||
skips to testing. Running tests may require a GPU.
|
||||
|
||||
Run a libcudacxx lit test
|
||||
-------------------------
|
||||
|
||||
Some libcudacxx tests run under lit, not CTest. Pass lit test paths relative to
|
||||
``libcudacxx/test/libcudacxx/``.
|
||||
|
||||
#. **Execute one lit test.** Use ``--lit-tests`` with the test path::
|
||||
|
||||
ci/util/build_and_test_targets.sh \
|
||||
--preset libcudacxx \
|
||||
--lit-tests \
|
||||
"std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp"
|
||||
|
||||
#. **Precompile without running.** Use ``--lit-precompile-tests`` to compile the
|
||||
test with a no-op executor. This catches compile errors without a GPU::
|
||||
|
||||
ci/util/build_and_test_targets.sh \
|
||||
--preset libcudacxx \
|
||||
--lit-precompile-tests \
|
||||
"std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp"
|
||||
|
||||
Run inside a devcontainer
|
||||
-------------------------
|
||||
|
||||
To build against a specific CUDA toolkit and host compiler, wrap the invocation
|
||||
with ``.devcontainer/launch.sh -d``. Valid CTK and host compiler values are
|
||||
listed in the ``.devcontainer`` directory. Pass ``--gpus all`` when the run
|
||||
needs a device::
|
||||
|
||||
.devcontainer/launch.sh -d --cuda <CTK> --host <compiler> --gpus all -- \
|
||||
ci/util/build_and_test_targets.sh \
|
||||
--preset cub-cpp20 \
|
||||
--build-targets "cub.test.iterator" \
|
||||
--ctest-targets "cub.test.iterator"
|
||||
11
cccl_upstream/docs/contributors/index.rst
Normal file
11
cccl_upstream/docs/contributors/index.rst
Normal file
@@ -0,0 +1,11 @@
|
||||
.. _contributors-index:
|
||||
|
||||
Contributor Docs
|
||||
================
|
||||
|
||||
This section covers the build, test, and debug workflows a CCCL contributor uses day to day.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
how_tos/index
|
||||
Reference in New Issue
Block a user