feat(cccl): integrate missing CCCL directories — python/, ci/, .agent/, docs/, test/
Sparse-checkout from NVIDIA/cccl main branch to complete cccl_upstream: Added: - python/cuda_cccl/ (226 files) — Python bindings for device-level algorithms Critical for muh toolchain: cuda.compute.reduce_into, scan, radix_sort, etc. Includes 204 .py files with full test coverage for all 27 algorithms - ci/ (163 files) — Build/test infrastructure build_cub.sh, test_cub.sh, build_and_test_targets.sh, matrix.yaml Directly maps to our [INFRA-CI] and [INFRA-BUILD] items - .agent/skills/ (7 files) — NVIDIA's own agent skills for CCCL cccl-style/SKILL.md, cccl-test/SKILL.md, sass-diff/SKILL.md - docs/ (491 files) — Official CCCL documentation CI references, CMake guides, Python compute docs, libcudacxx PTX docs - test/ (12 files) — Top-level integration tests (cuda_smoke, stdpar) - Root configs: .clang-format, .clang-tidy, CONTRIBUTING.md, pyproject.toml - CLAUDE.md symlink → AGENTS.md (NVIDIA's standard) cccl_upstream now mirrors full NVIDIA/cccl structure: Before: 42M (cub + thrust + libcudacxx + cudax + c + examples + benchmarks) After: 53M (+python +ci +docs +.agent +test +configs) This completes the CCCL base needed for: - [muh-bench] items: ci/util/build_and_test_targets.sh for targeted builds - [CCCL-verify] items: python/cuda_cccl/tests/ as reference implementations - [CCCL-test] items: ci/test_cub.sh, ci/test_thrust.sh - Agent workflow: .agent/skills/ for consistent style and test patterns
This commit is contained in:
46
cccl_upstream/docs/maintainers/backport_process.rst
Normal file
46
cccl_upstream/docs/maintainers/backport_process.rst
Normal file
@@ -0,0 +1,46 @@
|
||||
How to commit fixes to release branches (backport process)
|
||||
==========================================================
|
||||
|
||||
This guide explains when and how to commit fixes to release branches.
|
||||
|
||||
After a release branch is created and before a release tag is finalized,
|
||||
maintainers may need to apply fixes to that release branch.
|
||||
|
||||
To keep ``main`` as the source of truth, each fix starts as a PR against
|
||||
``main``, then automation opens an equivalent *backport* PR against the
|
||||
relevant release branch.
|
||||
|
||||
|
||||
Backport Criteria
|
||||
-----------------
|
||||
|
||||
Before starting a backport, use the questions below to decide if a change is worth backporting:
|
||||
|
||||
- Does this fix a correctness bug (wrong result, UB, memory safety, data race, deadlock)?
|
||||
- Does this fix a crash?
|
||||
- Does this fix a regression?
|
||||
- Are users actively asking for this fix to be backported?
|
||||
- How likely are users to be affected by this? How many?
|
||||
- Is there a reasonable workaround?
|
||||
- How risky is this change?
|
||||
|
||||
Examples *not* worth backporting:
|
||||
|
||||
- Fixes to tests that do not impact the functionality of the library.
|
||||
- Fixes to infrastructure that does not impact the functionality of the library.
|
||||
|
||||
Steps
|
||||
-----
|
||||
|
||||
#. Create a PR with the fix against ``main`` via a PR following our :doc:`contributing guidelines </cccl/contributing>`.
|
||||
#. Add the label ``backport branch/X.Y.x`` to the PR.
|
||||
|
||||
- If the PR is already merged, you can still trigger a backport by commenting
|
||||
``/backport branch/X.Y.x`` on the merged PR.
|
||||
#. After merge to ``main``, confirm automation opens a backport PR targeting
|
||||
``branch/X.Y.x``.
|
||||
#. Review the generated backport PR for correctness and resolve any conflicts.
|
||||
#. Ensure all CI checks have passed and merge the backport PR into the target
|
||||
release branch.
|
||||
|
||||
- Only members of the GitHub team `cccl-release-owners <https://github.com/orgs/NVIDIA/teams/cccl-release-owners>` can merge PRs to release branches.
|
||||
32
cccl_upstream/docs/maintainers/branching_strategy.rst
Normal file
32
cccl_upstream/docs/maintainers/branching_strategy.rst
Normal file
@@ -0,0 +1,32 @@
|
||||
Git Methodology
|
||||
=========================
|
||||
|
||||
This page defines the canonical branch model used for CCCL development and
|
||||
release maintenance.
|
||||
|
||||
Canonical branches
|
||||
------------------
|
||||
|
||||
- ``main``
|
||||
|
||||
- The default development branch.
|
||||
- Updates should be made via pull requests following our :doc:`contributing guidelines </cccl/contributing>`.
|
||||
|
||||
- ``branch/X.Y.x``
|
||||
|
||||
- Branches from ``main`` meant for stabilizing and publishing tagged releases.
|
||||
- Created via release automation.
|
||||
- Changes should be made via the :doc:`backport process <backport_process>`.
|
||||
|
||||
- ``ctk/X.Y.x``
|
||||
|
||||
- Branches from ``branch/X.Y.x`` meant for representing exact contents of CUDA Toolkit Releases
|
||||
- Created by NVIDIA. May contain internally developed features that are additive with the release it was based on.
|
||||
- This is a read-only branch and is created when the CTK or relevant documentation becomes available.
|
||||
|
||||
Tagging conventions
|
||||
-------------------
|
||||
- ``vX.Y.Z``: finalized release tags on release branches
|
||||
- ``vX.Y.Z-ctkN.M.K``: finalized release tags exactly equivalent to the CCCL contents in NVIDIA CUDA Toolkit release `N.M.K`.
|
||||
- ``vX.Y.Z-rcN``: release-candidate tags for pre-release validation.
|
||||
- ``vX.Y.Z.dev``: the first commit of development for ``X.Y.Z`` (the commit that increments the library version)
|
||||
66
cccl_upstream/docs/maintainers/coderabbit.rst
Normal file
66
cccl_upstream/docs/maintainers/coderabbit.rst
Normal file
@@ -0,0 +1,66 @@
|
||||
CodeRabbit
|
||||
==========
|
||||
|
||||
This page explains how to configure and use CodeRabbit for CCCL pull request
|
||||
review. For the complete product documentation, see the
|
||||
`CodeRabbit documentation <https://docs.coderabbit.ai/>`__.
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
CCCL configures CodeRabbit through ``.coderabbit.yaml`` in the repository root.
|
||||
The configuration in the pull request branch is used for that review.
|
||||
|
||||
When setting up or updating CodeRabbit:
|
||||
|
||||
#. Keep repository-specific settings in ``.coderabbit.yaml``.
|
||||
#. Use ``@coderabbitai configuration`` on a pull request to inspect the
|
||||
resolved configuration. This is useful when checking whether CodeRabbit is
|
||||
using the expected repository settings for that pull request.
|
||||
#. Use ``@coderabbitai generate configuration`` to export the resolved
|
||||
configuration if a new baseline is needed. This is useful when moving
|
||||
settings into a reviewable repository configuration file.
|
||||
#. Keep configuration changes small and reviewable.
|
||||
#. Use ``reviews.path_instructions`` for path-specific review guidance.
|
||||
#. Use ``knowledge_base.code_guidelines.filePatterns`` for CCCL guidance files
|
||||
that CodeRabbit should read as review context.
|
||||
|
||||
The CCCL configuration should keep comments focused on correctness, API
|
||||
stability, performance, security, and other high-impact issues. Avoid enabling
|
||||
features that add noisy generated comments or code by default.
|
||||
|
||||
Pull Request Reviews
|
||||
--------------------
|
||||
|
||||
Automatic reviews may be disabled or restricted by the repository
|
||||
configuration. Maintainers can always request review explicitly from a pull
|
||||
request comment:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
@coderabbitai review
|
||||
|
||||
Use a full review when the pull request should be reviewed again from scratch:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
@coderabbitai full review
|
||||
|
||||
Other useful commands:
|
||||
|
||||
- ``@coderabbitai help`` shows the current command reference.
|
||||
- ``@coderabbitai configuration`` shows the active configuration.
|
||||
- ``@coderabbitai summary`` can be placed in the pull request description as a
|
||||
placeholder for the generated summary.
|
||||
- ``@coderabbitai pause`` and ``@coderabbitai resume`` pause or resume
|
||||
automatic review behavior. This is useful when a pull request is still being
|
||||
updated frequently and should not be reviewed again until it is ready.
|
||||
- ``@coderabbitai ignore`` can be placed in the pull request description to
|
||||
disable automatic reviews.
|
||||
|
||||
Review Guidance
|
||||
---------------
|
||||
|
||||
Treat CodeRabbit feedback as review assistance, not as a merge requirement by
|
||||
itself. Maintainers remain responsible for deciding whether comments are
|
||||
actionable and whether a pull request has adequate tests and CI coverage.
|
||||
163
cccl_upstream/docs/maintainers/how_tos/adding_ci_coverage.rst
Normal file
163
cccl_upstream/docs/maintainers/how_tos/adding_ci_coverage.rst
Normal file
@@ -0,0 +1,163 @@
|
||||
.. _infra-ci-adding-coverage:
|
||||
|
||||
Adding CI coverage
|
||||
==================
|
||||
|
||||
CCCL's CI matrix is defined in ``ci/matrix.yaml``. You add coverage by writing new entries: each
|
||||
entry expands into one or more jobs through the cross-product of its array-valued fields. Place the
|
||||
entries under the right workflow sections, then validate them with the override matrix before merge.
|
||||
|
||||
The field reference for every entry — ``jobs``, ``project``, ``ctk``, ``cxx``, ``std``, ``gpu``,
|
||||
``sm``, ``cmake_options``, ``args`` — lives in the ``tags:`` and ``jobs:`` maps in
|
||||
``ci/matrix.yaml``. Read those before authoring an entry.
|
||||
|
||||
Choose the workflow sections to update
|
||||
--------------------------------------
|
||||
|
||||
``ci/matrix.yaml`` defines separate matrices per trigger under ``workflows:``.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 20 80
|
||||
|
||||
* - Section
|
||||
- Runs on
|
||||
* - ``pull_request``
|
||||
- Full coverage for projects modified in a PR. See :ref:`infra-ci-change-detection`.
|
||||
* - ``pull_request_lite``
|
||||
- Light coverage for project downstream of those modified in a PR. See :ref:`infra-ci-change-detection`.
|
||||
* - ``nightly``
|
||||
- Scheduled nightly. Broad compiler and CTK coverage.
|
||||
* - ``weekly``
|
||||
- Scheduled weekly. Widest coverage, including ``all-cccl`` architecture builds.
|
||||
|
||||
|
||||
Add the pull request matrix entry
|
||||
---------------------------------
|
||||
|
||||
Each row targets the specific compiler, CTK, and GPU.
|
||||
Keep it narrow. Every array field multiplies the job count.
|
||||
|
||||
This entry adds a Thrust test run pinned to one CTK and one GPU, across three host compilers:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
- {jobs: ['test'], project: 'thrust', ctk: '<ctk-name>', std: 'max', cxx: ['<cxx-name-1>', '<cxx-name-2>', '<cxx-name-3>'], gpu: '<gpu-name>'}
|
||||
|
||||
Use the CTK name from the ``ctk_versions:`` map in ``ci/matrix.yaml`` and the compiler name
|
||||
from the ``host_compilers:`` map. Use the GPU pool name from the ``gpus:`` map.
|
||||
|
||||
Field by field:
|
||||
|
||||
- ``jobs: ['test']`` — runs the ``test`` job. ``test`` requires a GPU and auto-generates its
|
||||
``build`` producer job (see the ``jobs:`` section). Other projects may have more specialized
|
||||
options besides build+test, but these are the most common.
|
||||
- ``project: 'thrust'`` — restricts the entry to one project. Omit to use the default
|
||||
``['libcudacxx', 'cub', 'thrust']``.
|
||||
- ``ctk: '<ctk-name>'`` — the CTK name from the ``ctk_versions:`` map. See ``ci/matrix.yaml``
|
||||
for the current names and what toolkit versions they resolve to. Prefer the convention of
|
||||
``<major>.X`` when requesting "the latest of this major version", eg. "13.X" instead of "13.2",
|
||||
and only use exact versions when meaningfully required (packaging constraints, minimum versions, etc).
|
||||
- ``std: 'max'`` — the highest C++ standard the project supports. Use ``min``, ``minmax``, or
|
||||
``all`` for wider coverage, or the standard year if specifics are needed (e.g. ``[17, 23]``).
|
||||
- ``cxx: [...]`` — one or more host compiler names. The array expands to one job per element.
|
||||
- ``gpu: '<gpu-name>'`` — the GPU runner pool to use, see the ``gpus:`` map.
|
||||
- ``sm: [...]`` — Request specific CUDA SM architectures (eg. '75' for Turing). Use
|
||||
``sm: 'gpu'`` to build for only the arch needed by the requested ``gpu``.
|
||||
|
||||
Add a build-only entry the same way with ``jobs: ['build']`` and no ``gpu``. Build jobs run on
|
||||
CPU-only runners.
|
||||
|
||||
Add the pull request lite matrix entry
|
||||
--------------------------------------
|
||||
|
||||
Rare, but if this is important coverage that is cheap, it may be worth adding a ``pull_request_lite`` entry.
|
||||
Jobs from this matrix are added to the PR run when an upstream internal dependency is modified.
|
||||
The goal is to keep this matrix as light as possible, extensions should be rare and well justified.
|
||||
|
||||
Note that these jobs will **NOT** run as part of the PR that adds them, so they **MUST** be tested with
|
||||
the override matrix before merge.
|
||||
|
||||
See :ref:`infra-ci-change-detection` for how CCCL's CI encodes these dependencies.
|
||||
|
||||
Add the corresponding nightly / weekly entries
|
||||
----------------------------------------------
|
||||
|
||||
The ``nightly`` and ``weekly`` entries run on a schedule, and carry the exhaustive, broad coverage that would
|
||||
be wasteful and excessive for PRs.
|
||||
It does not include the pull request matrix, so the PR jobs must be replicated, and possibly extended, here.
|
||||
Note that these jobs will **NOT** run as part of the PR that adds them, so they **MUST** be tested with
|
||||
the override matrix before merge.
|
||||
|
||||
Group the new entries under the existing comment headers in each section. Keep CTK and project
|
||||
groupings together so the matrix stays readable.
|
||||
|
||||
Update project_files_and_dependencies.yaml
|
||||
------------------------------------------
|
||||
|
||||
The full details of this system are documented in :ref:`infra-ci-change-detection`.
|
||||
|
||||
``ci/inspect_changes.py`` reads ``ci/project_files_and_dependencies.yaml`` to decide which projects
|
||||
a PR touched. Update this file when the new coverage involves a project or a source path the file
|
||||
does not already track. Skip this step when adding configurations to an existing project's existing
|
||||
paths.
|
||||
|
||||
Add or extend a project entry so changed files map to the right matrix project:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
my_project_public:
|
||||
name: "My Project Public API" # public/API entries only include public headers.
|
||||
lite_dependencies: [libcudacxx_public] # lite dependency on libcu++'s public headers (upstream only triggers lite PR coverage)
|
||||
full_dependencies: []
|
||||
include_regexes: ["my_project/include/"] # path to public headers
|
||||
|
||||
my_project_internal:
|
||||
name: "My Project Tests/Infra" # Internal entries exclude public headers, include everything else.
|
||||
matrix_project: "my_project" # Maps to the matrix.yaml project that will be triggered when project files change
|
||||
lite_dependencies: []
|
||||
full_dependencies: [my_project_public] # trigger the full PR coverage when public headers change.
|
||||
# changes to transitive deps (eg. libcudacxx_public) will onlytrigger lite PR coverage.
|
||||
include_regexes: ["my_project/"] # path to project root
|
||||
exclude_project_files: [my_project_public] # ignore files matched by the public entry.
|
||||
|
||||
- ``matrix_project`` — ties the change-detection key to the ``projects:`` key in ``ci/matrix.yaml``.
|
||||
Without it the project never enters the build list.
|
||||
- ``include_regexes`` — paths that mark this project dirty, anchored to the repo root.
|
||||
- ``lite_dependencies`` / ``full_dependencies`` - See the file comments or :ref:`infra-ci-change-detection` for details.
|
||||
|
||||
**Files matching no project fall into ``core`` and trigger a full build of everything.**
|
||||
|
||||
If you're adding files that should not ever trigger CI, add them to the top-level ``ignore_regexes`` list to exclude them from
|
||||
change detection.
|
||||
|
||||
Test the entry with the override matrix
|
||||
---------------------------------------
|
||||
|
||||
Validate new entries with ``workflows.override`` before merge. A non-empty ``override`` replaces the
|
||||
entire ``pull_request`` matrix for the PR, so CI runs only the entries you are testing. The override
|
||||
blocks merge until removed, which guarantees the full suite runs before the change lands.
|
||||
|
||||
#. **Copy the candidate entries into override.** Place the new ``pull_request`` and ``nightly``
|
||||
entries under ``workflows.override`` in ``ci/matrix.yaml``:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
workflows:
|
||||
override:
|
||||
- {jobs: ['test'], project: 'thrust', ctk: '<ctk-name>', std: 'max', cxx: ['<cxx-name-1>', '<cxx-name-2>'], gpu: '<gpu-name>'}
|
||||
|
||||
#. **Trim unrelated jobs.** Add ``[skip-tpt][skip-docs]`` to the **last commit message** to drop
|
||||
third-party tests (eg RAPIDS, MatX) and doc builds while iterating.
|
||||
|
||||
#. **Push and inspect.** The PR runs only the override entries. Confirm the jobs appear with the
|
||||
expected compiler, CTK, and GPU, and that they pass.
|
||||
|
||||
#. **Reset before merge.** Empty ``workflows.override`` and remove the ``[skip-*]`` tags from the
|
||||
last commit message (or push a new commit). The merge gate fails until both are clean.
|
||||
|
||||
For a tighter loop on a single test target, use ``project: 'target'`` with ``args`` forwarded to
|
||||
``ci/util/build_and_test_targets.sh``. The commented examples at the top of ``ci/matrix.yaml`` show
|
||||
the ``run_cpu`` and ``run_gpu`` invocation patterns. Reproduce any failing job locally with
|
||||
``.devcontainer/launch.sh`` and the matching ``ci/`` build or test script (see
|
||||
:ref:`infra-ci-reproducing-locally`).
|
||||
@@ -0,0 +1,70 @@
|
||||
.. _infra-devcontainer-adding-toolchain:
|
||||
|
||||
Adding a new devcontainer toolchain
|
||||
===================================
|
||||
|
||||
A toolchain is one CTK version paired with one host compiler. CCCL generates a
|
||||
devcontainer config for every combination listed in the ``devcontainers:`` section of
|
||||
``ci/matrix.yaml``. The generated configs live under ``.devcontainer/<name>/devcontainer.json``,
|
||||
one directory per combination, all produced by ``.devcontainer/make_devcontainers.sh [--clean]``.
|
||||
|
||||
Run the steps in order: edit the matrix, regenerate the configs, then verify before merge. The
|
||||
base image for the combination must already exist in the
|
||||
`rapidsai/devcontainers <https://github.com/rapidsai/devcontainers>`_ project before the matrix
|
||||
edit.
|
||||
|
||||
Check that the base image exists
|
||||
--------------------------------
|
||||
|
||||
Every CCCL devcontainer is built on a published ``rapidsai/devcontainers`` image. Image tags
|
||||
follow this pattern::
|
||||
|
||||
rapidsai/devcontainers:<devcontainer_version>-cpp-<compiler><version>-cuda<ctk>[ext]
|
||||
|
||||
The ``-cuda<ctk>`` segment is present for every combination except nvhpc, which bundles its
|
||||
own CUDA toolkit and omits it.
|
||||
|
||||
The ``<devcontainer_version>`` value is the ``devcontainer_version:`` field in ``ci/matrix.yaml``.
|
||||
|
||||
The images are maintained in the https://github.com/rapidsai/devcontainers/ repo, in the top-level
|
||||
matrix file. If new images are required for the coverage, submit a PR against `main`.
|
||||
|
||||
Add the combination to ci/matrix.yaml
|
||||
-------------------------------------
|
||||
|
||||
The source-of-truth when generating devcontainer toolchains is the ``matrix.yaml`` file. All jobs
|
||||
from all workflows are parsed, the toolchains extracted, and the ``.devcontainer/...`` directories built.
|
||||
|
||||
At the bottom of the workflows section of ``matrix.yaml`` is a ``devcontainers:`` section.
|
||||
This is intended to be a living mirror of the available images in the `rapidsai/devcontainers` repo,
|
||||
and is useful for quickly checking supported CTK / host compilers while editing the matrix.
|
||||
Occasionally we'll need a devcontainer that isn't referenced in any workflow, and this section is the place to add it.
|
||||
Make sure that your new toolchain is listed and documented here.
|
||||
|
||||
Regenerate the devcontainer configs
|
||||
-----------------------------------
|
||||
|
||||
From the repository root, regenerate every ``.devcontainer/<name>/devcontainer.json`` from the updated matrix:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
.devcontainer/make_devcontainers.sh --clean
|
||||
|
||||
The script reads all matrix workflow entries, expands aliases, and writes one directory per combination
|
||||
using the naming pattern ``cuda<version>[ext]-<compiler><version>``.
|
||||
It also updates the root ``.devcontainer/devcontainer.json`` default to the newest GCC + newest
|
||||
CUDA combination.
|
||||
Pass ``--clean`` to remove directories for combinations no longer in the matrix (recommended).
|
||||
|
||||
Never hand-edit a generated ``.devcontainer/<name>/devcontainer.json``. Edits are overwritten on
|
||||
the next run. To change settings that apply to every combination, edit the root
|
||||
``.devcontainer/devcontainer.json`` template, then rerun the generator to propagate the change.
|
||||
|
||||
Verify before merge
|
||||
-------------------
|
||||
|
||||
Locally test launching the devcontainer using the appropriate ``.devcontainer/launch.sh`` invocation.
|
||||
See :ref:`infra-devcontainer-launching` for details on launching and using the devcontainer.
|
||||
|
||||
The ``verify-devcontainers`` CI workflow reruns ``make_devcontainers.sh --verbose --clean``
|
||||
and fails if the result differs from the committed files.
|
||||
115
cccl_upstream/docs/maintainers/how_tos/compile_time.rst
Normal file
115
cccl_upstream/docs/maintainers/how_tos/compile_time.rst
Normal file
@@ -0,0 +1,115 @@
|
||||
Run compile-time benchmarks
|
||||
===========================
|
||||
|
||||
Use the compile-time benchmark when you want to compare how a change affects
|
||||
CUDA TU compile time. The common workflow compares the current tree against
|
||||
``origin/main`` and reports the most important movements in generated public
|
||||
include-check TUs.
|
||||
|
||||
For the full option, CSV, filter, and CI contract reference, see
|
||||
:doc:`../references/compile_time`.
|
||||
|
||||
Run the common comparison
|
||||
-------------------------
|
||||
|
||||
From the repository root, run:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ci/build_compile_time_bench.sh \
|
||||
-baseline-ref origin/main \
|
||||
-- --slices /path/to/slices.json
|
||||
|
||||
The CI slices live in ``ci/matrix.yaml`` under
|
||||
``compile_time.pull_request[].slices``. To reproduce the PR shape locally, copy
|
||||
those slice definitions to a JSON file shaped as:
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"slices": [
|
||||
{
|
||||
"id": "file-processing",
|
||||
"title": "Direct file processing",
|
||||
"filter": "file-processing",
|
||||
"timing": "exclusive",
|
||||
"sort": "total",
|
||||
"top": 15,
|
||||
"threshold": 0.2
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
The wrapper builds both the current tree and the baseline commit with the same
|
||||
preset, target set, architecture, and other build arguments. It then writes
|
||||
baseline reports, current reports, comparison CSVs, a ``summary.json`` manifest,
|
||||
raw traces, and Perfetto-friendly traces under the preset build directory:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
build/<infix>/<preset>/compile_time/
|
||||
|
||||
Run a quick single-slice report
|
||||
-------------------------------
|
||||
|
||||
For iteration after traces already exist, skip the build and regenerate only an
|
||||
event report:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ci/build_compile_time_bench.sh -skip-build -- \
|
||||
-f file-processing -e --sort total -n 25
|
||||
|
||||
Other useful built-in filters include:
|
||||
|
||||
- ``total-compilation``
|
||||
- ``file-processing``
|
||||
- ``scanning-function-body``
|
||||
- ``template-instantiation``
|
||||
- ``host-compiler``
|
||||
- ``code-generation``
|
||||
- ``all``
|
||||
|
||||
Interpret the PR comment
|
||||
------------------------
|
||||
|
||||
The PR comment contains one section per configured slice. Within each slice,
|
||||
regressions and improvements are intentionally separated. Rows are ranked by
|
||||
total impact across matched traces, not just by the largest single-TU movement.
|
||||
|
||||
Important columns:
|
||||
|
||||
- ``Regression impact`` / ``Improvement impact``: absolute total-impact change
|
||||
across matched traces, in seconds.
|
||||
- ``Selected Δ``: signed movement in the selected metric for the event.
|
||||
- ``Baseline`` / ``Current``: selected metric values on each side.
|
||||
- ``Matched traces``: number of generated TUs where the event key was comparable.
|
||||
|
||||
Small movements are filtered by per-slice thresholds from ``ci/matrix.yaml``.
|
||||
Those thresholds are intentionally non-zero to hide ordinary run-to-run noise.
|
||||
|
||||
Inspect traces in Perfetto
|
||||
--------------------------
|
||||
|
||||
The wrapper prepares trace copies whose event names include the useful file or
|
||||
symbol detail. Open files under:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
build/<infix>/<preset>/compile_time/perfetto_traces/
|
||||
|
||||
in Perfetto or another Chrome-trace-compatible viewer.
|
||||
|
||||
Skip compile-time benchmark telemetry
|
||||
-------------------------------------
|
||||
|
||||
Compile-time benchmark jobs are informational and do not gate the aggregate PR
|
||||
``CI`` job. For early iterations where the compile-time benchmark is unrelated,
|
||||
append this case-sensitive tag to the commit message:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
[skip-compile-time-bench]
|
||||
|
||||
Remove the tag before requesting final review if the compile-time benchmark
|
||||
scripts, workflow, or configuration changed.
|
||||
13
cccl_upstream/docs/maintainers/how_tos/index.rst
Normal file
13
cccl_upstream/docs/maintainers/how_tos/index.rst
Normal file
@@ -0,0 +1,13 @@
|
||||
How Tos
|
||||
=======
|
||||
|
||||
How-to guides for maintainers.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
../backport_process
|
||||
compile_time
|
||||
plan_work_with_github_issues
|
||||
adding_ci_coverage
|
||||
adding_new_devcontainer
|
||||
@@ -0,0 +1,108 @@
|
||||
How To Plan Work with GitHub Issues
|
||||
===================================
|
||||
|
||||
CCCL uses `GitHub issues <https://github.com/NVIDIA/cccl/issues>`__ and the
|
||||
`CCCL GitHub Project <https://github.com/orgs/NVIDIA/projects/6>`__ as the source of truth to make
|
||||
planned work visible, understandable, and coordinated.
|
||||
|
||||
This guide explains how CCCL plans work, records that
|
||||
work in closable issues, and keeps priorities and ownership visible over time.
|
||||
|
||||
CCCL plans work in monthly sprints, with each sprint identifying the issues the
|
||||
team intends to prioritize during that 4-week window.
|
||||
|
||||
Understand current planned work
|
||||
-------------------------------
|
||||
|
||||
The `Current Sprint <https://github.com/orgs/NVIDIA/projects/6/views/47>`__ view in the GitHub Project contains
|
||||
the planned work the team has agreed to prioritize now. It is not a complete
|
||||
list of all engineering activity.
|
||||
|
||||
Planned work intentionally leaves capacity for reviews, support, debugging, and
|
||||
other interrupt-driven work.
|
||||
|
||||
In-progress work is not automatically carried forward into the next planning
|
||||
cycle. Each cycle is planned from current priorities.
|
||||
|
||||
Write a closable issue
|
||||
----------------------
|
||||
|
||||
Every issue should answer:
|
||||
|
||||
"This issue can be closed when..."
|
||||
|
||||
Examples:
|
||||
|
||||
- a bug is fixed;
|
||||
- a refactoring is complete;
|
||||
- a benchmark is added;
|
||||
- a design decision is documented;
|
||||
- follow-on issues are created;
|
||||
- an investigation summary is written.
|
||||
|
||||
Large efforts can start as tracking issues whose scope is still evolving. As
|
||||
concrete work becomes clear, create sub-issues for work with independent close
|
||||
conditions.
|
||||
|
||||
Propose planned work
|
||||
--------------------
|
||||
|
||||
Sprint planning happens monthly. Team leads review the roadmap, current priorities,
|
||||
and incoming requests to identify the highest-impact work for the upcoming
|
||||
four-week planning window.
|
||||
|
||||
If you think work should be considered for the next planning cycle, raise it
|
||||
during planning or ask a team lead to add it to the
|
||||
`Sprint Planning <https://github.com/orgs/NVIDIA/projects/6/views/58>`__ view.
|
||||
|
||||
During planning, the team reviews proposed issues, confirms ownership,
|
||||
identifies gaps or dependencies, and checks that issues are actionable and
|
||||
up-to-date.
|
||||
|
||||
Confirmed planned work is moved to the
|
||||
`Current Sprint <https://github.com/orgs/NVIDIA/projects/6/views/47>`__ view.
|
||||
|
||||
Keep assigned issues current
|
||||
----------------------------
|
||||
|
||||
The assignee is responsible for driving the issue forward and keeping it
|
||||
accurate while the work evolves.
|
||||
|
||||
Update the issue when:
|
||||
|
||||
- the title no longer describes the work clearly;
|
||||
- the close condition changes;
|
||||
- the work has been split into sub-issues;
|
||||
- important context, decisions, or blockers appear;
|
||||
- the issue is no longer relevant and should be closed.
|
||||
|
||||
Use issue comments for questions, decisions, blockers, and context that
|
||||
others may need to find later. If a comment changes the issue's scope or close
|
||||
condition, update the issue body as well.
|
||||
|
||||
Complete assigned issues
|
||||
------------------------
|
||||
|
||||
Use the issue's close condition to decide when the work is done. Often, this
|
||||
means opening a PR that completes the work described by the issue. Follow the
|
||||
:doc:`contributing guidelines </cccl/contributing>` for how to prepare and
|
||||
submit the PR.
|
||||
|
||||
Link the PR to the issue so GitHub can close the issue when the PR merges. For
|
||||
example, include ``Fixes #123`` in the PR description.
|
||||
|
||||
Linked PRs also keep the GitHub Project status current. A linked draft PR keeps
|
||||
the issue ``In Progress``. When the PR is ready for review, automation moves the
|
||||
issue to ``Review``.
|
||||
|
||||
Plan release-critical work
|
||||
--------------------------
|
||||
|
||||
Upcoming releases have corresponding
|
||||
`GitHub milestones <https://github.com/NVIDIA/cccl/milestones>`__.
|
||||
|
||||
Team leads create release milestones and assign issues to them during planning.
|
||||
Add an issue to a release milestone when the work is
|
||||
important to complete and include in that release.
|
||||
|
||||
Not all current planned work is tied to a particular release.
|
||||
10
cccl_upstream/docs/maintainers/index.rst
Normal file
10
cccl_upstream/docs/maintainers/index.rst
Normal file
@@ -0,0 +1,10 @@
|
||||
Maintainer Docs
|
||||
===============
|
||||
|
||||
This section documents the policies and procedures to maintain and release artifacts from the CCCL repository.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
how_tos/index
|
||||
references/index
|
||||
300
cccl_upstream/docs/maintainers/references/compile_time.rst
Normal file
300
cccl_upstream/docs/maintainers/references/compile_time.rst
Normal file
@@ -0,0 +1,300 @@
|
||||
Compile-time benchmark reference
|
||||
================================
|
||||
|
||||
The compile-time benchmark builds generated one-include CUDA translation units
|
||||
(TUs) and summarizes NVCC ``--fdevice-time-trace`` output. It is a TU
|
||||
compile-time benchmark, even when the current input set comes from public
|
||||
include-check targets.
|
||||
|
||||
Entry point
|
||||
-----------
|
||||
|
||||
The top-level entry point is:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ci/build_compile_time_bench.sh
|
||||
|
||||
The wrapper configures a caller-selected CMake preset with compile-time
|
||||
instrumentation, builds selected targets, prepares Perfetto-friendly trace
|
||||
copies, writes a generated-TU summary CSV, and emits one or more event summary
|
||||
CSVs.
|
||||
|
||||
Generated outputs
|
||||
-----------------
|
||||
|
||||
By default, outputs are written under:
|
||||
|
||||
- ``build/<infix>/<preset>/compile_time/tu_summary.csv``
|
||||
- ``build/<infix>/<preset>/compile_time/event_reports/``
|
||||
- ``build/<infix>/<preset>/compile_time/perfetto_traces/``
|
||||
|
||||
Raw NVCC traces are generated under:
|
||||
|
||||
- ``build/<infix>/<preset>/compile_time/raw_traces/``
|
||||
|
||||
When ``-baseline-ref`` is used, baseline raw traces are copied out of the
|
||||
temporary baseline worktree before cleanup and preserved under:
|
||||
|
||||
- ``build/<infix>/<preset>/compile_time/baseline_raw_traces/``
|
||||
|
||||
Build controls
|
||||
--------------
|
||||
|
||||
The wrapper accepts build-shape parameters so it behaves like other
|
||||
``ci/build*.sh`` entry points:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
PARALLEL_LEVEL=16 ci/build_compile_time_bench.sh \
|
||||
-preset all-dev \
|
||||
-target libcudacxx.test.public_headers \
|
||||
-cmake-options "-DCMAKE_CUDA_ARCHITECTURES=native"
|
||||
|
||||
Useful build options include:
|
||||
|
||||
- ``-preset <name>``
|
||||
- ``-cmake-options <args>``
|
||||
- ``-target <name>`` (repeatable; replaces the default public include-check target set)
|
||||
- ``-baseline-ref <commit-ish>`` (build a temporary baseline worktree for comparison)
|
||||
- ``-skip-configure``
|
||||
- ``-skip-build``
|
||||
|
||||
The default target set is the current public include-check target set:
|
||||
|
||||
- ``cub.headers.base``
|
||||
- ``thrust.cpp.cuda.headers.base``
|
||||
- ``libcudacxx.test.public_headers``
|
||||
|
||||
The one-include source TUs are created by those CMake targets themselves. The
|
||||
benchmark wrapper only enables extra compile options on generated CUDA TUs:
|
||||
``CCCL_COMPILE_TIME_GENERATE_DEVICE_TIME_TRACES`` writes NVCC device-time trace
|
||||
JSON, and ``CCCL_COMPILE_TIME_SAVE_PREPROCESSED_TUS`` preserves compiler
|
||||
preprocessed/temporary TU artifacts for the TU summary.
|
||||
|
||||
Event summaries
|
||||
---------------
|
||||
|
||||
Arguments after ``--`` are forwarded to
|
||||
``ci/compile_time/summarize_events.py`` after the raw trace directory. If no
|
||||
event-summary arguments are provided, the wrapper runs:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
-f file-processing -e -n 15
|
||||
|
||||
For CI-style multi-slice reports, pass a JSON slice file:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ci/build_compile_time_bench.sh -baseline-ref origin/main -- \
|
||||
--slices /path/to/slices.json
|
||||
|
||||
The slice file contains a ``slices`` array. Each slice has a stable ``id``,
|
||||
display ``title``, ``filter``, ``timing`` (``inclusive`` or ``exclusive``),
|
||||
``sort``, ``top``, and ``threshold`` in seconds. Multi-slice mode writes each
|
||||
slice under ``event_reports/<slice-id>/`` and writes a normalized
|
||||
``event_reports/summary.json`` manifest for PR comment rendering.
|
||||
|
||||
Empty slices are represented in the manifest and CSVs. Slices that match no
|
||||
events, have no matching trace files, or have no comparable event keys record
|
||||
warnings so they are visible in PR comments instead of looking like ordinary
|
||||
no-change results. Empty slices with no warnings are omitted recursively by the
|
||||
PR comment renderer.
|
||||
|
||||
Single-slice examples:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ci/build_compile_time_bench.sh -skip-build -- -f scanning-function-body -i -n 20
|
||||
ci/build_compile_time_bench.sh -skip-build -- -f template-instantiation -e -n 15 --tag templates
|
||||
ci/build_compile_time_bench.sh -skip-build -- -f 'Scanning|Instantiating' -i -n 25
|
||||
ci/build_compile_time_bench.sh -skip-build -- -f code-generation -i --scope-filter ""
|
||||
|
||||
Baseline comparisons
|
||||
--------------------
|
||||
|
||||
Pass ``-baseline-ref <commit-ish>`` to compare the current tree state against a
|
||||
baseline commit. The wrapper creates a temporary detached worktree for the
|
||||
baseline, builds both the current tree and the baseline with the same preset,
|
||||
targets, and common build options, then runs the requested event slice as a
|
||||
baseline/current comparison:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ci/build_compile_time_bench.sh \
|
||||
-baseline-ref origin/main \
|
||||
-- -f file-processing -e --sort total -n 25 --threshold 0.2
|
||||
|
||||
Comparison mode writes three subdirectories under
|
||||
``<preset-build-dir>/compile_time/event_reports/``:
|
||||
|
||||
- ``baseline/``: the normal report for the baseline traces
|
||||
- ``current/``: the normal report for the current traces
|
||||
- ``comparison/``: ``worse`` and ``better`` CSVs for the requested filter,
|
||||
timing, exclusivity, sort, and top-N slice
|
||||
|
||||
In multi-slice comparison mode, the same layout appears under each
|
||||
``event_reports/<slice-id>/`` directory.
|
||||
|
||||
Trace files are matched by relative path. Delta CSVs only compare event keys
|
||||
that appear in both sides of the same matched trace file. Unmatched child events
|
||||
are still counted in their matched parent event's exclusive cost, so disappearing
|
||||
or newly appearing nested work remains visible as a parent cost change instead
|
||||
of being subtracted away. If there are no comparable event keys, the comparison
|
||||
CSVs are still written with headers and no rows.
|
||||
|
||||
Pass ``--threshold <seconds>`` after the wrapper's ``--`` separator in
|
||||
comparison mode to omit ``worse`` / ``better`` rows whose total impact change is
|
||||
not greater than that threshold. Baseline/current reports use ``--sort`` for
|
||||
their own top-N ordering, but comparison reports are ranked by total impact
|
||||
across all matched traces so repeated small movements outrank a larger movement
|
||||
in only one trace.
|
||||
|
||||
When ``-baseline-ref`` is used, the wrapper treats the invocation as an event
|
||||
comparison and skips the generated-TU CSV unless ``-tu-csv`` is provided
|
||||
explicitly. To compare arbitrary trace directories outside the wrapper layout,
|
||||
run ``ci/compile_time/summarize_events.py`` directly.
|
||||
|
||||
Pull-request reporting
|
||||
----------------------
|
||||
|
||||
Compile-time PR reporting is configured in ``ci/matrix.yaml`` under
|
||||
``compile_time.pull_request``. Each config selects the GPU runner, devcontainer
|
||||
launch arguments, baseline ref, preset, targets, wrapper arguments, and report
|
||||
slices. ``ci/compile_time/parse_matrix.py`` validates that section and emits the
|
||||
GitHub Actions matrix for the reusable compile-time benchmark workflow.
|
||||
|
||||
The reusable workflow uploads:
|
||||
|
||||
- event report CSVs and ``summary.json``
|
||||
- current raw traces
|
||||
- baseline raw traces
|
||||
- Perfetto-friendly traces
|
||||
- the rendered PR comment body
|
||||
|
||||
``ci/compile_time/render_pr_comment.py`` renders the comment from
|
||||
``summary.json``. Regressions and improvements are rendered in separate
|
||||
``<details>`` sections and are never mixed in one table. Slice warnings are
|
||||
rendered separately. Empty sections with no warnings are omitted recursively.
|
||||
Sticky comments are keyed by ``compile-time-bench-<config-id>``; previous
|
||||
comments for the same config are archived as outdated when a new one is posted.
|
||||
|
||||
This reporting is informational and is not part of the aggregate branch
|
||||
protection ``CI`` job. Commit messages containing ``[skip-compile-time-bench]``
|
||||
skip compile-time benchmark dispatch.
|
||||
|
||||
Filters and scope filtering
|
||||
---------------------------
|
||||
|
||||
Built-in filter names include:
|
||||
|
||||
- ``file-processing``
|
||||
- ``scanning-function-body``
|
||||
- ``template-instantiation``
|
||||
- ``template-class-instantiation``
|
||||
- ``template-function-instantiation``
|
||||
- ``pending-instantiations``
|
||||
- ``frontend``
|
||||
- ``host-compiler``
|
||||
- ``code-generation``
|
||||
- ``optimizer``
|
||||
- ``total-compilation``
|
||||
- ``all``
|
||||
|
||||
Unknown filters are interpreted as case-insensitive regular expressions over
|
||||
event names and event details.
|
||||
|
||||
Symbol-like events, such as function parsing, template instantiation, function
|
||||
IR generation, and optimizer-function events, are scope-filtered by default to
|
||||
top-level CCCL-owned namespaces:
|
||||
|
||||
- ``cuda::``
|
||||
- ``thrust::``
|
||||
- ``cub::``
|
||||
- ``cccl::``
|
||||
|
||||
This keeps reports focused on CCCL symbols instead of system library symbols
|
||||
pulled into the same generated TU. The filter applies to demangled trace details
|
||||
and to decoded namespace prefixes from Itanium-mangled symbols in trace details.
|
||||
It does not filter path/phase events such as file processing, host compiler
|
||||
phases, or total compilation time.
|
||||
|
||||
Pass ``--scope-filter <regex>`` after the wrapper's ``--`` separator to choose
|
||||
a different case-sensitive symbol-scope regex. Pass an empty string to disable
|
||||
symbol-scope filtering:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ci/build_compile_time_bench.sh -skip-build -- \
|
||||
-f template-instantiation -i --scope-filter ""
|
||||
|
||||
``host-compiler`` matches the host compiler preprocessing / compiling events
|
||||
that appear in the device-time-trace output. ``total-compilation`` is a
|
||||
synthetic per-trace event whose inclusive time is the wall-clock span from the
|
||||
first timed trace event to the last timed trace event. This includes host
|
||||
compiler, cudafe, NVVM, fatbinary, and gaps visible inside the trace timeline,
|
||||
but it is not a separate external wall-clock measurement of untraced driver,
|
||||
Ninja, or process-launch overhead.
|
||||
|
||||
Compared to Ninja log timings, ``total-compilation`` generally undercounts each
|
||||
TU by a small, consistent amount because the trace span starts at the first
|
||||
timed event and ends at the last timed event rather than at process launch/exit.
|
||||
That makes it a good relative-comparison and ranking metric, but not an exact
|
||||
replacement for external wall-clock command duration.
|
||||
|
||||
Use ``--sort`` to choose the selected ranking metric:
|
||||
|
||||
- ``total``
|
||||
- ``avg``
|
||||
- ``avg-root-tu``
|
||||
- ``max``
|
||||
|
||||
Perfetto trace preparation
|
||||
--------------------------
|
||||
|
||||
The wrapper prepares Perfetto-friendly trace copies by default. To prepare an
|
||||
existing trace directory manually:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ci/compile_time/prepare_traces.py \
|
||||
--input build/<infix>/<preset>/compile_time/raw_traces \
|
||||
--output /tmp/compile_time_perfetto
|
||||
|
||||
CSV output
|
||||
----------
|
||||
|
||||
``summarize_tus.py`` writes:
|
||||
|
||||
- ``tu_input``
|
||||
- ``transitive_loc``
|
||||
- ``tu_source``
|
||||
- ``preprocessed_tu``
|
||||
|
||||
This CSV is a generated-TU input/LOC summary. Use the ``total-compilation``
|
||||
event filter when you need per-TU compile-time rankings from trace data.
|
||||
|
||||
``summarize_events.py`` writes stable event keys and both inclusive and
|
||||
exclusive metrics, including:
|
||||
|
||||
- ``event_name``
|
||||
- ``event_key`` (repo-root-relative path for project file-processing events)
|
||||
- ``selected_total_s``
|
||||
- ``selected_avg_per_event_s``
|
||||
- ``selected_avg_per_root_tu_s``
|
||||
- ``total_inclusive_s`` / ``total_exclusive_s``
|
||||
- ``event_count``
|
||||
- ``trace_count``
|
||||
- ``root_tu_count``
|
||||
|
||||
Comparison CSVs additionally include total-impact columns
|
||||
(``baseline_impact_s``, ``current_impact_s``, ``impact_delta_s``,
|
||||
``impact_magnitude_s``) plus selected-metric columns
|
||||
(``baseline_selected_s``, ``current_selected_s``, ``selected_delta_s``,
|
||||
``selected_magnitude_s``).
|
||||
|
||||
Notebook workflow
|
||||
-----------------
|
||||
|
||||
For exploratory analysis, use ``ci/compile_time/analytics.ipynb``.
|
||||
11
cccl_upstream/docs/maintainers/references/index.rst
Normal file
11
cccl_upstream/docs/maintainers/references/index.rst
Normal file
@@ -0,0 +1,11 @@
|
||||
References
|
||||
==========
|
||||
|
||||
Reference documentation for maintainers.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
../branching_strategy
|
||||
../coderabbit
|
||||
compile_time
|
||||
Reference in New Issue
Block a user