2a7ca101d7
Sparse-checkout from NVIDIA/cccl main branch to complete cccl_upstream: Added: - python/cuda_cccl/ (226 files) — Python bindings for device-level algorithms Critical for muh toolchain: cuda.compute.reduce_into, scan, radix_sort, etc. Includes 204 .py files with full test coverage for all 27 algorithms - ci/ (163 files) — Build/test infrastructure build_cub.sh, test_cub.sh, build_and_test_targets.sh, matrix.yaml Directly maps to our [INFRA-CI] and [INFRA-BUILD] items - .agent/skills/ (7 files) — NVIDIA's own agent skills for CCCL cccl-style/SKILL.md, cccl-test/SKILL.md, sass-diff/SKILL.md - docs/ (491 files) — Official CCCL documentation CI references, CMake guides, Python compute docs, libcudacxx PTX docs - test/ (12 files) — Top-level integration tests (cuda_smoke, stdpar) - Root configs: .clang-format, .clang-tidy, CONTRIBUTING.md, pyproject.toml - CLAUDE.md symlink → AGENTS.md (NVIDIA's standard) cccl_upstream now mirrors full NVIDIA/cccl structure: Before: 42M (cub + thrust + libcudacxx + cudax + c + examples + benchmarks) After: 53M (+python +ci +docs +.agent +test +configs) This completes the CCCL base needed for: - [muh-bench] items: ci/util/build_and_test_targets.sh for targeted builds - [CCCL-verify] items: python/cuda_cccl/tests/ as reference implementations - [CCCL-test] items: ci/test_cub.sh, ci/test_thrust.sh - Agent workflow: .agent/skills/ for consistent style and test patterns
36 lines
2.2 KiB
ReStructuredText
36 lines
2.2 KiB
ReStructuredText
.. _cccl-tma:
|
|
|
|
Tensor Memory Accelerator (TMA)
|
|
===============================
|
|
|
|
The Tensor Memory Accelerator (TMA) is a hardware feature available on Hopper (SM90) and newer GPUs
|
|
that enables efficient asynchronous memory copies of tensor data between global and (cluster) shared memory.
|
|
The use of TMA is required to reach SOL memory throughput for some workloads,
|
|
notable those where the necessary load/store vectorization, unrolling, or pipelining are limited by the register file or other factors.
|
|
CCCL offer several tools to help users leverage TMA in their applications.
|
|
|
|
In general, we recommend users to reach for high-level algorithms if they fit their problem.
|
|
Several algorithms, like ``cub::DeviceTransform``, ``cub::DeviceMerge``, ``cub::DeviceScan`` already use TMA internally today,
|
|
with many Thrust algorithms building on those.
|
|
And more algorithms will be added over time.
|
|
Relying on high level algorithms leaves the complexity of implementing and tuning TMA to CCCL team,
|
|
while providing users with safer interfaces, high productivity and SOL performance from the start.
|
|
|
|
If direct use of TMA is required to author new kernels, CCCL offers the following tools to help users get started,
|
|
from high-level to low-level:
|
|
|
|
- ``cub::BlockLoadToShared`` coming soon :)
|
|
- :ref:`cuda::memcpy_async <libcudacxx-extended-api-asynchronous-operations-memcpy-async>`
|
|
- :ref:`cuda::device::memcpy_async_tx <libcudacxx-extended-api-asynchronous-operations-memcpy-async-tx>`
|
|
- :ref:`cuda::ptx::cp_async_bulk* variants <libcudacxx-ptx-instructions>`
|
|
|
|
``cub::BlockLoadToShared`` and ``cuda::memcpy_async`` have fallback implementations for pre-Hopper GPUs,
|
|
using ``cp.async``/``LDGSTS`` on Ampere (SM80+) and ordinary loads/stores on older architectures.
|
|
Furthermore, they gracefully handle unaligned data and copying regions of arbitrary size.
|
|
|
|
The various ``cuda::ptx::cp_async_bulk*`` versions and ``cuda::device::memcpy_async_tx``
|
|
are thin wrappers of the corresponding PTX instructions
|
|
and provide no fallback path on older GPUs and also require the copied data to be aligned and sized appropriately.
|
|
|
|
Some further TMA-related utilities are provided by the :ref:`libcu++ extended API <libcudacxx-extended-api-tma>`.
|