2a7ca101d7
Sparse-checkout from NVIDIA/cccl main branch to complete cccl_upstream: Added: - python/cuda_cccl/ (226 files) — Python bindings for device-level algorithms Critical for muh toolchain: cuda.compute.reduce_into, scan, radix_sort, etc. Includes 204 .py files with full test coverage for all 27 algorithms - ci/ (163 files) — Build/test infrastructure build_cub.sh, test_cub.sh, build_and_test_targets.sh, matrix.yaml Directly maps to our [INFRA-CI] and [INFRA-BUILD] items - .agent/skills/ (7 files) — NVIDIA's own agent skills for CCCL cccl-style/SKILL.md, cccl-test/SKILL.md, sass-diff/SKILL.md - docs/ (491 files) — Official CCCL documentation CI references, CMake guides, Python compute docs, libcudacxx PTX docs - test/ (12 files) — Top-level integration tests (cuda_smoke, stdpar) - Root configs: .clang-format, .clang-tidy, CONTRIBUTING.md, pyproject.toml - CLAUDE.md symlink → AGENTS.md (NVIDIA's standard) cccl_upstream now mirrors full NVIDIA/cccl structure: Before: 42M (cub + thrust + libcudacxx + cudax + c + examples + benchmarks) After: 53M (+python +ci +docs +.agent +test +configs) This completes the CCCL base needed for: - [muh-bench] items: ci/util/build_and_test_targets.sh for targeted builds - [CCCL-verify] items: python/cuda_cccl/tests/ as reference implementations - [CCCL-test] items: ci/test_cub.sh, ci/test_thrust.sh - Agent workflow: .agent/skills/ for consistent style and test patterns
104 lines
4.5 KiB
ReStructuredText
104 lines
4.5 KiB
ReStructuredText
.. _infra-ci-artifacts:
|
||
|
||
Artifact system
|
||
===============
|
||
|
||
The artifact system carries data between CI jobs. A producer job uploads its build outputs;
|
||
consumer jobs download them to run tests. It wraps GitHub Actions' native ``upload-artifact``
|
||
and ``download-artifact`` and is GitHub Actions-specific.
|
||
|
||
Artifacts are not mandatory. Projects that lean on the shared AWS sccache can let the cache
|
||
serve build products to test jobs instead. They still pay off for larger projects: downloading
|
||
one compressed archive of the test binaries is much faster than recompiling or fetching each
|
||
target individually through the cache. Compare a test job's sccache build time against the cost
|
||
of packing and downloading an archive to decide which a project should use.
|
||
|
||
Scripts
|
||
-------
|
||
|
||
CI scripts call a small set of entry points under ``ci/util/artifacts/``. Each script's source
|
||
is the authoritative reference for its full argument set.
|
||
|
||
.. list-table::
|
||
:header-rows: 1
|
||
:widths: 32 68
|
||
|
||
* - Script
|
||
- Purpose
|
||
* - ``stage.sh``
|
||
- Add files matching a regex to a named artifact's staged set.
|
||
* - ``unstage.sh``
|
||
- Remove files from a staged set — prune what no consumer needs.
|
||
* - ``upload_stage_packed.sh``
|
||
- Pack a staged set into a compressed ``tar.zst`` and upload it.
|
||
* - ``upload.sh``
|
||
- Upload a file, or regex-matched files, as a plain unpacked artifact.
|
||
* - ``download_packed.sh``
|
||
- Download a packed artifact and extract it to a path.
|
||
* - ``download.sh``
|
||
- Download a plain artifact file to a path.
|
||
|
||
The other scripts under ``ci/util/artifacts/`` — ``register.sh``, ``pack.sh``,
|
||
``upload_packed.sh``, and the ``upload/set_*.sh`` compression and retention knobs — are
|
||
lower-level building blocks the entry points compose. ``upload_stage_packed.sh`` packs,
|
||
registers, and sets compression on the archive itself, so producers rarely touch them directly.
|
||
|
||
Producers and consumers find each other through ``ci/util/workflow/`` helpers:
|
||
``get_producer_id.sh`` resolves a consumer's producer, and ``has_consumers.sh`` /
|
||
``get_consumers.sh`` let a producer skip work when nothing downstream needs its output.
|
||
|
||
Producer: stage and pack build outputs
|
||
--------------------------------------
|
||
|
||
A producer stages the files a consumer will need, prunes what it does not, then packs and
|
||
uploads the set as one archive. It first checks whether any consumer exists and exits early if
|
||
not:
|
||
|
||
.. code-block:: bash
|
||
|
||
ci/util/workflow/has_consumers.sh "$JOB_ID" || exit 0
|
||
|
||
ci/util/artifacts/stage.sh "<artifact_name>" '<regex>' ['<regex>' ...]
|
||
ci/util/artifacts/unstage.sh "<artifact_name>" '<regex>'
|
||
ci/util/artifacts/upload_stage_packed.sh "<artifact_name>"
|
||
|
||
``stage.sh`` and ``unstage.sh`` build up the file set by inclusion and exclusion — regexes match
|
||
against ``find`` within the stage path. ``upload_stage_packed.sh`` compresses the result and
|
||
registers it for upload. CUB stages one packed artifact per launch-id variant (``no_lid``,
|
||
``lid_0``–``lid_2``); ``ci/upload_cub_test_artifacts.sh`` is the authoritative example.
|
||
|
||
Consumer: resolve and download
|
||
------------------------------
|
||
|
||
A consumer does not know its producer's job ID in advance. It resolves the producer from the
|
||
run manifest, then fetches the packed outputs.
|
||
|
||
Resolve the producer with ``get_producer_id.sh`` — it loads the run manifest (``workflow.json``)
|
||
on demand and returns the producer's job ID. The packed artifact name is project-specific and
|
||
embeds the producer ID; the consuming CI script constructs it, so there are no hardcoded names.
|
||
|
||
.. code-block:: bash
|
||
|
||
producer_id=$(ci/util/workflow/get_producer_id.sh)
|
||
for tag in "${ARTIFACT_TAGS[@]}"; do
|
||
ci/util/artifacts/download_packed.sh \
|
||
"z_cub-test-artifacts-${DEVCONTAINER_NAME:?}-${producer_id}-${tag}" /home/coder/cccl
|
||
done
|
||
|
||
See ``ci/test_cub.sh`` for the authoritative CUB form.
|
||
|
||
Result record
|
||
-------------
|
||
|
||
Apart from build artifacts, every job records its own outcome. At exit it calls
|
||
``ci/upload_job_result_artifacts.sh "$JOB_ID" $exit_code``, which uploads a ``zz_jobs-<job_id>``
|
||
artifact containing a ``success`` file only when the exit code was zero. The ``ci:`` gate reads
|
||
these records to compute the single pass/fail for the run — see :ref:`infra-ci-overview`.
|
||
|
||
Python wheels
|
||
-------------
|
||
|
||
Python jobs exchange built wheels rather than packed test binaries. A producer uploads the wheel
|
||
with ``upload.sh``; a consumer downloads it with ``download.sh``. Both resolve the filename
|
||
through ``ci/util/workflow/get_wheel_artifact_name.sh``.
|