104 lines
4.5 KiB
ReStructuredText
104 lines
4.5 KiB
ReStructuredText
|
.. _infra-ci-artifacts:
|
|||
|
|
|
|||
|
|
Artifact system
|
|||
|
|
===============
|
|||
|
|
|
|||
|
|
The artifact system carries data between CI jobs. A producer job uploads its build outputs;
|
|||
|
|
consumer jobs download them to run tests. It wraps GitHub Actions' native ``upload-artifact``
|
|||
|
|
and ``download-artifact`` and is GitHub Actions-specific.
|
|||
|
|
|
|||
|
|
Artifacts are not mandatory. Projects that lean on the shared AWS sccache can let the cache
|
|||
|
|
serve build products to test jobs instead. They still pay off for larger projects: downloading
|
|||
|
|
one compressed archive of the test binaries is much faster than recompiling or fetching each
|
|||
|
|
target individually through the cache. Compare a test job's sccache build time against the cost
|
|||
|
|
of packing and downloading an archive to decide which a project should use.
|
|||
|
|
|
|||
|
|
Scripts
|
|||
|
|
-------
|
|||
|
|
|
|||
|
|
CI scripts call a small set of entry points under ``ci/util/artifacts/``. Each script's source
|
|||
|
|
is the authoritative reference for its full argument set.
|
|||
|
|
|
|||
|
|
.. list-table::
|
|||
|
|
:header-rows: 1
|
|||
|
|
:widths: 32 68
|
|||
|
|
|
|||
|
|
* - Script
|
|||
|
|
- Purpose
|
|||
|
|
* - ``stage.sh``
|
|||
|
|
- Add files matching a regex to a named artifact's staged set.
|
|||
|
|
* - ``unstage.sh``
|
|||
|
|
- Remove files from a staged set — prune what no consumer needs.
|
|||
|
|
* - ``upload_stage_packed.sh``
|
|||
|
|
- Pack a staged set into a compressed ``tar.zst`` and upload it.
|
|||
|
|
* - ``upload.sh``
|
|||
|
|
- Upload a file, or regex-matched files, as a plain unpacked artifact.
|
|||
|
|
* - ``download_packed.sh``
|
|||
|
|
- Download a packed artifact and extract it to a path.
|
|||
|
|
* - ``download.sh``
|
|||
|
|
- Download a plain artifact file to a path.
|
|||
|
|
|
|||
|
|
The other scripts under ``ci/util/artifacts/`` — ``register.sh``, ``pack.sh``,
|
|||
|
|
``upload_packed.sh``, and the ``upload/set_*.sh`` compression and retention knobs — are
|
|||
|
|
lower-level building blocks the entry points compose. ``upload_stage_packed.sh`` packs,
|
|||
|
|
registers, and sets compression on the archive itself, so producers rarely touch them directly.
|
|||
|
|
|
|||
|
|
Producers and consumers find each other through ``ci/util/workflow/`` helpers:
|
|||
|
|
``get_producer_id.sh`` resolves a consumer's producer, and ``has_consumers.sh`` /
|
|||
|
|
``get_consumers.sh`` let a producer skip work when nothing downstream needs its output.
|
|||
|
|
|
|||
|
|
Producer: stage and pack build outputs
|
|||
|
|
--------------------------------------
|
|||
|
|
|
|||
|
|
A producer stages the files a consumer will need, prunes what it does not, then packs and
|
|||
|
|
uploads the set as one archive. It first checks whether any consumer exists and exits early if
|
|||
|
|
not:
|
|||
|
|
|
|||
|
|
.. code-block:: bash
|
|||
|
|
|
|||
|
|
ci/util/workflow/has_consumers.sh "$JOB_ID" || exit 0
|
|||
|
|
|
|||
|
|
ci/util/artifacts/stage.sh "<artifact_name>" '<regex>' ['<regex>' ...]
|
|||
|
|
ci/util/artifacts/unstage.sh "<artifact_name>" '<regex>'
|
|||
|
|
ci/util/artifacts/upload_stage_packed.sh "<artifact_name>"
|
|||
|
|
|
|||
|
|
``stage.sh`` and ``unstage.sh`` build up the file set by inclusion and exclusion — regexes match
|
|||
|
|
against ``find`` within the stage path. ``upload_stage_packed.sh`` compresses the result and
|
|||
|
|
registers it for upload. CUB stages one packed artifact per launch-id variant (``no_lid``,
|
|||
|
|
``lid_0``–``lid_2``); ``ci/upload_cub_test_artifacts.sh`` is the authoritative example.
|
|||
|
|
|
|||
|
|
Consumer: resolve and download
|
|||
|
|
------------------------------
|
|||
|
|
|
|||
|
|
A consumer does not know its producer's job ID in advance. It resolves the producer from the
|
|||
|
|
run manifest, then fetches the packed outputs.
|
|||
|
|
|
|||
|
|
Resolve the producer with ``get_producer_id.sh`` — it loads the run manifest (``workflow.json``)
|
|||
|
|
on demand and returns the producer's job ID. The packed artifact name is project-specific and
|
|||
|
|
embeds the producer ID; the consuming CI script constructs it, so there are no hardcoded names.
|
|||
|
|
|
|||
|
|
.. code-block:: bash
|
|||
|
|
|
|||
|
|
producer_id=$(ci/util/workflow/get_producer_id.sh)
|
|||
|
|
for tag in "${ARTIFACT_TAGS[@]}"; do
|
|||
|
|
ci/util/artifacts/download_packed.sh \
|
|||
|
|
"z_cub-test-artifacts-${DEVCONTAINER_NAME:?}-${producer_id}-${tag}" /home/coder/cccl
|
|||
|
|
done
|
|||
|
|
|
|||
|
|
See ``ci/test_cub.sh`` for the authoritative CUB form.
|
|||
|
|
|
|||
|
|
Result record
|
|||
|
|
-------------
|
|||
|
|
|
|||
|
|
Apart from build artifacts, every job records its own outcome. At exit it calls
|
|||
|
|
``ci/upload_job_result_artifacts.sh "$JOB_ID" $exit_code``, which uploads a ``zz_jobs-<job_id>``
|
|||
|
|
artifact containing a ``success`` file only when the exit code was zero. The ``ci:`` gate reads
|
|||
|
|
these records to compute the single pass/fail for the run — see :ref:`infra-ci-overview`.
|
|||
|
|
|
|||
|
|
Python wheels
|
|||
|
|
-------------
|
|||
|
|
|
|||
|
|
Python jobs exchange built wheels rather than packed test binaries. A producer uploads the wheel
|
|||
|
|
with ``upload.sh``; a consumer downloads it with ``download.sh``. Both resolve the filename
|
|||
|
|
through ``ci/util/workflow/get_wheel_artifact_name.sh``.
|