feat(cccl): integrate missing CCCL directories — python/, ci/, .agent/, docs/, test/
Sparse-checkout from NVIDIA/cccl main branch to complete cccl_upstream: Added: - python/cuda_cccl/ (226 files) — Python bindings for device-level algorithms Critical for muh toolchain: cuda.compute.reduce_into, scan, radix_sort, etc. Includes 204 .py files with full test coverage for all 27 algorithms - ci/ (163 files) — Build/test infrastructure build_cub.sh, test_cub.sh, build_and_test_targets.sh, matrix.yaml Directly maps to our [INFRA-CI] and [INFRA-BUILD] items - .agent/skills/ (7 files) — NVIDIA's own agent skills for CCCL cccl-style/SKILL.md, cccl-test/SKILL.md, sass-diff/SKILL.md - docs/ (491 files) — Official CCCL documentation CI references, CMake guides, Python compute docs, libcudacxx PTX docs - test/ (12 files) — Top-level integration tests (cuda_smoke, stdpar) - Root configs: .clang-format, .clang-tidy, CONTRIBUTING.md, pyproject.toml - CLAUDE.md symlink → AGENTS.md (NVIDIA's standard) cccl_upstream now mirrors full NVIDIA/cccl structure: Before: 42M (cub + thrust + libcudacxx + cudax + c + examples + benchmarks) After: 53M (+python +ci +docs +.agent +test +configs) This completes the CCCL base needed for: - [muh-bench] items: ci/util/build_and_test_targets.sh for targeted builds - [CCCL-verify] items: python/cuda_cccl/tests/ as reference implementations - [CCCL-test] items: ci/test_cub.sh, ci/test_thrust.sh - Agent workflow: .agent/skills/ for consistent style and test patterns
This commit is contained in:
muh-bot
103
cccl_upstream/docs/infrastructure/ci/references/ci_artifacts.rst
Normal file
103
cccl_upstream/docs/infrastructure/ci/references/ci_artifacts.rst
Normal file
@@ -0,0 +1,103 @@
|
||||
.. _infra-ci-artifacts:
|
||||
|
||||
Artifact system
|
||||
===============
|
||||
|
||||
The artifact system carries data between CI jobs. A producer job uploads its build outputs;
|
||||
consumer jobs download them to run tests. It wraps GitHub Actions' native ``upload-artifact``
|
||||
and ``download-artifact`` and is GitHub Actions-specific.
|
||||
|
||||
Artifacts are not mandatory. Projects that lean on the shared AWS sccache can let the cache
|
||||
serve build products to test jobs instead. They still pay off for larger projects: downloading
|
||||
one compressed archive of the test binaries is much faster than recompiling or fetching each
|
||||
target individually through the cache. Compare a test job's sccache build time against the cost
|
||||
of packing and downloading an archive to decide which a project should use.
|
||||
|
||||
Scripts
|
||||
-------
|
||||
|
||||
CI scripts call a small set of entry points under ``ci/util/artifacts/``. Each script's source
|
||||
is the authoritative reference for its full argument set.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 32 68
|
||||
|
||||
* - Script
|
||||
- Purpose
|
||||
* - ``stage.sh``
|
||||
- Add files matching a regex to a named artifact's staged set.
|
||||
* - ``unstage.sh``
|
||||
- Remove files from a staged set — prune what no consumer needs.
|
||||
* - ``upload_stage_packed.sh``
|
||||
- Pack a staged set into a compressed ``tar.zst`` and upload it.
|
||||
* - ``upload.sh``
|
||||
- Upload a file, or regex-matched files, as a plain unpacked artifact.
|
||||
* - ``download_packed.sh``
|
||||
- Download a packed artifact and extract it to a path.
|
||||
* - ``download.sh``
|
||||
- Download a plain artifact file to a path.
|
||||
|
||||
The other scripts under ``ci/util/artifacts/`` — ``register.sh``, ``pack.sh``,
|
||||
``upload_packed.sh``, and the ``upload/set_*.sh`` compression and retention knobs — are
|
||||
lower-level building blocks the entry points compose. ``upload_stage_packed.sh`` packs,
|
||||
registers, and sets compression on the archive itself, so producers rarely touch them directly.
|
||||
|
||||
Producers and consumers find each other through ``ci/util/workflow/`` helpers:
|
||||
``get_producer_id.sh`` resolves a consumer's producer, and ``has_consumers.sh`` /
|
||||
``get_consumers.sh`` let a producer skip work when nothing downstream needs its output.
|
||||
|
||||
Producer: stage and pack build outputs
|
||||
--------------------------------------
|
||||
|
||||
A producer stages the files a consumer will need, prunes what it does not, then packs and
|
||||
uploads the set as one archive. It first checks whether any consumer exists and exits early if
|
||||
not:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ci/util/workflow/has_consumers.sh "$JOB_ID" || exit 0
|
||||
|
||||
ci/util/artifacts/stage.sh "<artifact_name>" '<regex>' ['<regex>' ...]
|
||||
ci/util/artifacts/unstage.sh "<artifact_name>" '<regex>'
|
||||
ci/util/artifacts/upload_stage_packed.sh "<artifact_name>"
|
||||
|
||||
``stage.sh`` and ``unstage.sh`` build up the file set by inclusion and exclusion — regexes match
|
||||
against ``find`` within the stage path. ``upload_stage_packed.sh`` compresses the result and
|
||||
registers it for upload. CUB stages one packed artifact per launch-id variant (``no_lid``,
|
||||
``lid_0``–``lid_2``); ``ci/upload_cub_test_artifacts.sh`` is the authoritative example.
|
||||
|
||||
Consumer: resolve and download
|
||||
------------------------------
|
||||
|
||||
A consumer does not know its producer's job ID in advance. It resolves the producer from the
|
||||
run manifest, then fetches the packed outputs.
|
||||
|
||||
Resolve the producer with ``get_producer_id.sh`` — it loads the run manifest (``workflow.json``)
|
||||
on demand and returns the producer's job ID. The packed artifact name is project-specific and
|
||||
embeds the producer ID; the consuming CI script constructs it, so there are no hardcoded names.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
producer_id=$(ci/util/workflow/get_producer_id.sh)
|
||||
for tag in "${ARTIFACT_TAGS[@]}"; do
|
||||
ci/util/artifacts/download_packed.sh \
|
||||
"z_cub-test-artifacts-${DEVCONTAINER_NAME:?}-${producer_id}-${tag}" /home/coder/cccl
|
||||
done
|
||||
|
||||
See ``ci/test_cub.sh`` for the authoritative CUB form.
|
||||
|
||||
Result record
|
||||
-------------
|
||||
|
||||
Apart from build artifacts, every job records its own outcome. At exit it calls
|
||||
``ci/upload_job_result_artifacts.sh "$JOB_ID" $exit_code``, which uploads a ``zz_jobs-<job_id>``
|
||||
artifact containing a ``success`` file only when the exit code was zero. The ``ci:`` gate reads
|
||||
these records to compute the single pass/fail for the run — see :ref:`infra-ci-overview`.
|
||||
|
||||
Python wheels
|
||||
-------------
|
||||
|
||||
Python jobs exchange built wheels rather than packed test binaries. A producer uploads the wheel
|
||||
with ``upload.sh``; a consumer downloads it with ``download.sh``. Both resolve the filename
|
||||
through ``ci/util/workflow/get_wheel_artifact_name.sh``.
|
||||
Reference in New Issue
Block a user