feat(cccl): integrate missing CCCL directories — python/, ci/, .agent/, docs/, test/

Sparse-checkout from NVIDIA/cccl main branch to complete cccl_upstream:

Added:
- python/cuda_cccl/ (226 files) — Python bindings for device-level algorithms
  Critical for muh toolchain: cuda.compute.reduce_into, scan, radix_sort, etc.
  Includes 204 .py files with full test coverage for all 27 algorithms
- ci/ (163 files) — Build/test infrastructure
  build_cub.sh, test_cub.sh, build_and_test_targets.sh, matrix.yaml
  Directly maps to our [INFRA-CI] and [INFRA-BUILD] items
- .agent/skills/ (7 files) — NVIDIA's own agent skills for CCCL
  cccl-style/SKILL.md, cccl-test/SKILL.md, sass-diff/SKILL.md
- docs/ (491 files) — Official CCCL documentation
  CI references, CMake guides, Python compute docs, libcudacxx PTX docs
- test/ (12 files) — Top-level integration tests (cuda_smoke, stdpar)
- Root configs: .clang-format, .clang-tidy, CONTRIBUTING.md, pyproject.toml
- CLAUDE.md symlink → AGENTS.md (NVIDIA's standard)

cccl_upstream now mirrors full NVIDIA/cccl structure:
  Before: 42M (cub + thrust + libcudacxx + cudax + c + examples + benchmarks)
  After:  53M (+python +ci +docs +.agent +test +configs)

This completes the CCCL base needed for:
- [muh-bench] items: ci/util/build_and_test_targets.sh for targeted builds
- [CCCL-verify] items: python/cuda_cccl/tests/ as reference implementations
- [CCCL-test] items: ci/test_cub.sh, ci/test_thrust.sh
- Agent workflow: .agent/skills/ for consistent style and test patterns
This commit is contained in:
muh-bot
2026-08-07 02:34:33 +00:00
parent 3f97dca7ad
commit 2a7ca101d7
908 changed files with 121615 additions and 0 deletions

View File

@@ -0,0 +1,63 @@
.. _libcudacxx-extended-api-memory-align_down:
``cuda::align_down``
====================
Defined in the header ``<cuda/memory>``.
.. code:: cuda
namespace cuda {
template <typename T>
[[nodiscard]] __host__ __device__ inline
T* align_down(T* ptr, size_t alignment) noexcept;
} // namespace cuda
The function returns the original pointer or closest pointer smaller than ``ptr`` that is aligned to the specified alignment :math:`floor\left(\frac{ptr}{alignment}\right) * alignment`.
**Parameters**
- ``ptr``: The pointer.
- ``alignment``: The alignment.
**Return value**
- The original pointer or closest pointer smaller than ``ptr`` that is aligned to the specified alignment.
**Constraints**
- ``alignment`` must be a power of two.
- ``alignment >= alignof(T)``.
- ``ptr`` is aligned to ``alignof(T)``.
**Performance considerations**
- The function is optimized for compile-time values of ``alignment``.
- The function does not perform any operations if ``alignment == alignof(T)``.
- The function is translated to a single ``LOP3.LUT`` instruction for other values of ``alignment``.
- The returned pointer is decorated with ``__builtin_assume_aligned`` to help the compiler generate better code.
- The returned pointer maintains the same memory space, for example shared memory, as the input pointer.
Example
-------
.. code:: cuda
#include <cuda/memory>
__global__ void kernel(int* ptr) {
auto ptr_align16 = cuda::align_down(ptr, 16);
reinterpret_cast<int4*>(ptr_align16)[0] = int4{1, 2, 3, 4};
}
int main() {
int* ptr;
cudaMalloc(&ptr, 100 * sizeof(int));
kernel<<<1, 1>>>(ptr + 20);
cudaDeviceSynchronize();
return 0;
}
`See it on Godbolt 🔗 <https://godbolt.org/z/6acdx3KhY>`__

View File

@@ -0,0 +1,63 @@
.. _libcudacxx-extended-api-memory-align_up:
``cuda::align_up``
==================
Defined in the header ``<cuda/memory>``.
.. code:: cuda
namespace cuda {
template <typename T>
[[nodiscard]] __host__ __device__ inline
T* align_up(T* ptr, size_t alignment) noexcept;
} // namespace cuda
The function returns the original pointer or closest pointer larger than ``ptr`` that is aligned to the specified alignment :math:`ceil\left(\frac{ptr}{alignment}\right) * alignment`.
**Parameters**
- ``ptr``: The pointer.
- ``alignment``: The alignment.
**Return value**
- The original pointer or closest pointer larger than ``ptr`` that is aligned to the specified alignment.
**Constraints**
- ``alignment`` must be a power of two.
- ``alignment >= alignof(T)``.
- ``ptr`` is aligned to ``alignof(T)``.
**Performance considerations**
- The function is optimized for compile-time values of ``alignment``.
- The function does not perform any operations if ``alignment == alignof(T)``.
- The function is translated to ``LOP3.LUT`` + ``IADD.64`` instructions for other values of ``alignment``.
- The returned pointer is decorated with ``__builtin_assume_aligned`` to help the compiler generate better code.
- The returned pointer maintains the same memory space, for example shared memory, as the input pointer.
Example
-------
.. code:: cuda
#include <cuda/memory>
__global__ void kernel(int* ptr) {
auto ptr_align16 = cuda::align_up(ptr, 16);
reinterpret_cast<int4*>(ptr_align16)[0] = int4{1, 2, 3, 4};
}
int main() {
int* ptr;
cudaMalloc(&ptr, 100 * sizeof(int));
kernel<<<1, 1>>>(ptr);
cudaDeviceSynchronize();
return 0;
}
`See it on Godbolt 🔗 <https://godbolt.org/z/d8e5KETeE>`__

View File

@@ -0,0 +1,91 @@
.. _libcudacxx-extended-api-memory-aligned-size:
``cuda::aligned_size_t``
========================
Defined in headers ``<cuda/memory>``, ``<cuda/barrier>`` and ``<cuda/pipeline>``:
.. code:: cuda
template <cuda::std::size_t Alignment>
struct cuda::aligned_size_t {
static constexpr cuda::std::size_t align = Align;
cuda::std::size_t value;
__host__ __device__ explicit constexpr aligned_size(cuda::std::size_t size);
__host__ __device__ constexpr operator cuda::std::size_t();
};
The class template ``cuda::aligned_size_t`` is a *shape* representing an extent of bytes with a statically
defined (address and size) alignment.
*Preconditions*:
- The *address* of the extent of bytes must be aligned to an ``Alignment`` alignment boundary.
- The *size* of the extent of bytes must be a multiple of the ``Alignment``.
Template Parameters
-------------------
.. list-table::
:widths: 25 75
:header-rows: 0
* - ``Alignment``
- The address and size alignment of the byte extent.
Data Members
------------
.. list-table::
:widths: 25 75
:header-rows: 0
* - ``align``
- The alignment of the byte extent.
* - ``value``
- The size of the byte extent.
Member Functions
----------------
.. list-table::
:widths: 25 75
:header-rows: 0
* - ``(constructor)``
- Constructs an *aligned size*. If the ``size`` is not a multiple of ``Alignment`` the behavior is undefined.
* - ``(destructor)``
- Trivial implicit destructor.
* - ``operator=``
- Trivial implicit copy/move.
* - ``operator cuda::std::size_t``
- Implicit conversion to `cuda::std::size_t <https://en.cppreference.com/w/cpp/types/size_t>`__.
Notes
-----
If ``Alignment`` is not a `valid alignment <https://en.cppreference.com/w/c/language/object#Alignment>`_,
the behavior is undefined.
Example
-------
.. code:: cuda
#include <cuda/memory>
__global__ void example_kernel(void* dst, void* src, size_t size) {
cuda::barrier<cuda::thread_scope_system> bar;
init(&bar, 1);
// Implementation cannot make assumptions about alignment.
cuda::memcpy_async(dst, src, size, bar);
// Implementation can assume that dst and src are 16-bytes aligned,
// and that size is a multiple of 16, and may optimize accordingly.
cuda::memcpy_async(dst, src, cuda::aligned_size_t<16>(size), bar);
bar.arrive_and_wait();
}
`See it on Godbolt <https://godbolt.org/z/PWGdfTd7d>`_

View File

@@ -0,0 +1,45 @@
.. _libcudacxx-extended-api-memory-discard-memory:
``cuda::discard_memory``
========================
Defined in header ``<cuda/memory>``, ``<cuda/discard_memory>`` (deprecated since CCCL 3.2).
.. code:: cuda
__host__ __device__
void discard_memory(volatile void* ptr, size_t nbytes);
Discard modified cache lines without writing back the cached data to memory. The functionality enables using global memory as temporary scratch space. Does **not** generate any HW store operations.
Equivalent to ``memset(ptr, _indeterminate_, nbytes)``.
**Preconditions**
- ``ptr`` points to a valid allocation in *global memory* of size greater or equal to ``nbytes``.
Example
-------
This kernel needs a scratch pad that does not fit in shared memory, so it uses an allocation in global memory instead:
.. code:: cuda
#include <cuda/memory>
__device__ int compute(int* scratch, size_t N);
__global__ void kernel(const int* in, int* out, int* scratch, size_t N) {
// Each thread reads N elements into the scratch pad:
for (int i = 0; i < N; ++i) {
int idx = threadIdx.x + i * blockDim.x;
scratch[idx] = in[idx];
}
__syncthreads();
// All threads compute on the scratch pad:
int result = compute(scratch, N);
// All threads discard the scratch pad memory to _hint_ that it does not need to be flushed from the cache:
cuda::discard_memory(scratch + threadIdx.x * N, N * sizeof(int));
__syncthreads();
out[threadIdx.x] = result;
}

View File

@@ -0,0 +1,72 @@
.. _libcudacxx-extended-api-memory-get-device-address:
``cuda::get_device_address``
============================
Defined in the headers ``<cuda/memory>`` and ``<cuda/functional>``.
.. code:: cuda
namespace cuda {
template <typename T>
[[nodiscard]] __host__ __device__ inline
T* get_device_address(T& device_object); // (1)
template <typename T>
[[nodiscard]] __host__ inline
T* get_device_address(T& device_object, device_ref device); // (2)
} // namespace cuda
``cuda::get_device_address`` returns a valid pointer to a device object for the current (1) or ``device`` (2) device. It replaces uses of ``cudaGetSymbolAddress``, which requires an inout parameter.
**Parameters**
- ``device_object``: Reference to a device object. (1, 2)
- ``device``: Device for which the object's address shall be retrieved. (2)
**Constraints**
- ``device_object`` must be a ``__device__`` or ``__constant__`` decorated variable.
Example
-------
.. code:: cuda
#include <cuda/devices>
#include <cuda/memory>
__device__ int device_object[] = {42, 1337, -1, 0};
__global__ void example_kernel(int *data) { ... }
void example()
{
cuda::device_ref device{0};
{
T* host_address = cuda::std::addressof(device_object);
cudaPointerAttributes attributes;
cudaError_t status = cudaPointerGetAttributes(&attributes, host_address);
assert(status == cudaSuccess);
assert(attributes.devicePointer == nullptr);
// Calling a kernel with host_address would segfault
// example_kernel<<<1, 1>>>(host_address);
}
{
T* device_address = cuda::get_device_address(device_object, device);
cudaPointerAttributes attributes;
cudaError_t status = cudaPointerGetAttributes(&attributes, device_address);
assert(status == cudaSuccess);
assert(attributes.devicePointer == device_address);
// Safe to call a kernel
example_kernel<<<1, 1>>>(device_address);
}
}

View File

@@ -0,0 +1,116 @@
.. _libcudacxx-extended-api-memory-is_address_from:
``cuda::device::is_address_from`` and ``cuda::device::is_object_from``
======================================================================
Defined in the ``<cuda/memory>`` header.
.. code:: cuda
namespace cuda::device {
enum class address_space
{
global, // Global state space
shared, // Shared state space
constant, // Constant state space
local, // Local state space
grid_constant, // Kernel function parameter in the parameter state space
cluster_shared, // Cluster shared window within the shared state space
};
} // namespace cuda::device
Enumeration of device address spaces used with the ``is_address_from()`` and ``is_object_from()`` functions. See the `PTX ISA documentation for state spaces <https://docs.nvidia.com/cuda/parallel-thread-execution/#state-spaces>`_ for more details.
----
.. code:: cuda
namespace cuda::device {
[[nodiscard]] __device__ inline
bool is_address_from(const volatile void* ptr, address_space space) noexcept; // (1)
} // namespace cuda::device
Checks whether a generic-address pointer ``ptr`` is from the specified address space.
----
.. code:: cuda
namespace cuda::device {
template <typename T>
[[nodiscard]] __device__ inline
bool is_object_from(T& obj, address_space space) noexcept; // (2)
} // namespace cuda::device
Checks whether an object ``obj`` with a generic address is from the specified address space.
----
Unlike the corresponding CUDA intrinsic functions ``__isGlobal()``, ``__isShared()``, ``__isConstant()``, ``__isLocal()``, ``__isGridConstant()``, and ``__isClusterShared()``, ``is_address_from()`` and ``is_object_from()`` are portable across all compute capabilities and, in debug mode, also checks that the pointer is not null.
**Parameters**
- ``ptr``: The pointer. (1)
- ``obj``: The object. (2)
- ``space``: The address space. (1, 2)
**Return value**
- Returns ``true`` if the pointer (1) or object (2) is from the specified address space; ``false`` otherwise.
.. note::
If the GPU architecture does not support the requested address space, the function always returns ``false``.
**Preconditions**
- ``ptr`` must not be null. (1)
**Performance considerations**
- When available, the built-in functions (``__isGlobal()``, ``__isShared()``, ``__isConstant()``, ``__isLocal()``, ``__isGridConstant()``, or ``__isClusterShared()``) are used to determine the address space.
- If the memory space of the input pointer matches the requested address space,
the function marks the pointer as belonging to that address space. For example, a subsequent store to a generic address that maps to shared memory emits an ``STS`` SASS instruction rather than the generic ``ST`` instruction.
Example
-------
.. code:: cuda
#include <cuda/memory>
__device__ int global_var;
__constant__ int constant_var;
__global__ void kernel(const __grid_constant__ int grid_constant_var)
{
using cuda::device::address_space;
__shared__ int shared_var;
int local_var{};
assert(cuda::device::is_address_from(&global_var, address_space::global));
assert(cuda::device::is_address_from(&shared_var, address_space::shared));
assert(cuda::device::is_address_from(&constant_var, address_space::constant));
assert(cuda::device::is_address_from(&local_var, address_space::local));
assert(cuda::device::is_address_from(&grid_constant_var, address_space::grid_constant));
assert(cuda::device::is_object_from(global_var, address_space::global));
assert(cuda::device::is_object_from(shared_var, address_space::shared));
assert(cuda::device::is_object_from(constant_var, address_space::constant));
assert(cuda::device::is_object_from(local_var, address_space::local));
assert(cuda::device::is_object_from(grid_constant_var, address_space::grid_constant));
}
int main(int, char**)
{
kernel<<<1, 1>>>(42);
cudaDeviceSynchronize();
}
`See it on Godbolt 🔗 <https://godbolt.org/z/5ajhe37df>`__

View File

@@ -0,0 +1,59 @@
.. _libcudacxx-extended-api-memory-is_aligned:
``cuda::is_aligned``
====================
Defined in the header ``<cuda/memory>``.
.. code:: cuda
namespace cuda {
[[nodiscard]] __host__ __device__ inline
bool is_aligned(const void* ptr, size_t alignment) noexcept
} // namespace cuda
The function determines if a pointer is aligned to a specific alignment.
**Parameters**
- ``ptr``: The pointer.
- ``alignment``: The alignment.
**Return value**
- ``true`` if the pointer is aligned to the specified alignment, ``false`` otherwise.
**Constraints**
- ``alignment`` must be a power of two.
.. note::
The function is similar to the C++ standard library function `cuda::std::is_sufficiently_aligned() <https://en.cppreference.com/w/cpp/memory/is_sufficiently_aligned.html>`__ from the ``<cuda/std/memory>`` header. The differences are the following:
- ``cuda::is_aligned()`` doesn't have a template parameter and might be less expensive to compile.
- ``cuda::is_aligned()`` supports run-time values of ``alignment``.
- ``cuda::std::is_sufficiently_aligned()`` additionally checks the compatibility between the alignment of the pointer type and the specified alignment.
Example
-------
.. code:: cuda
#include <cuda/memory>
__global__ void kernel(const void* ptr) {
assert(cuda::is_aligned(ptr, 16));
}
int main() {
void* ptr;
cudaMalloc(&ptr, 100 * sizeof(int));
kernel<<<1, 1>>>(ptr);
cudaDeviceSynchronize();
return 0;
}
`See it on Godbolt 🔗 <https://godbolt.org/z/Tr45EoKsT>`__

View File

@@ -0,0 +1,87 @@
.. _libcudacxx-extended-api-memory-is_pointer_accessible:
``cuda::is_host_accessible``, ``cuda::is_device_accessible``, ``cuda::is_managed``
==================================================================================
Defined in the ``<cuda/memory>`` header.
.. code:: cuda
namespace cuda {
[[nodiscard]] inline
bool is_host_accessible(const void* ptr); // (1)
[[nodiscard]] inline
bool is_device_accessible(const void* ptr, device_ref device); // (2)
[[nodiscard]] inline
bool is_managed(const void* ptr); // (3)
} // namespace cuda
Determines whether the memory referenced by ``ptr`` is accessible from the host (1), from the specified ``device`` (2), or is backed by Unified Memory (managed memory) (3).
- ``is_device_accessible()`` also checks whether the memory is peer-accessible or allocated from a memory pool accessible to the specified ``device``.
- ``is_host_accessible()`` also checks whether the memory is allocated from a memory pool accessible to the host.
----
**Parameters**
- ``ptr``: A pointer to the memory location to query.
- ``device``: A ``device_ref`` that denotes the device to query. (2)
**Return value**
- ``true`` if the queried property (host access, device access, or managed allocation) holds; otherwise, ``false``.
.. note::
A ``__device__`` global array or variable cannot be used directly from host code without first retrieving its address with ``cudaGetSymbolAddress()``.
**Prerequisites**
- The functions are available only when the CUDA Toolkit is available.
**Exceptions**
- The functions throw a ``cuda::cuda_error`` if the underlying driver API calls fail. Note that these functions may also fail with error codes from previously launched asynchronous operations.
**Undefined Behavior**
- The functions have undefined behavior if the pointer is not valid, for example, an already freed pointer.
Example
-------
.. code:: cuda
#include <cassert>
#include <cuda/memory>
#include <cuda_runtime_api.h>
int main() {
cuda::device_ref dev{0};
void* host_ptr = nullptr;
void* device_ptr = nullptr;
void* managed_ptr = nullptr;
cudaMallocHost(&host_ptr, 1024);
cudaMalloc(&device_ptr, 1024);
cudaMallocManaged(&managed_ptr, 1024);
assert(cuda::is_host_accessible(host_ptr));
assert(!cuda::is_device_accessible(host_ptr, dev));
assert(cuda::is_device_accessible(device_ptr, dev));
assert(!cuda::is_host_accessible(device_ptr));
assert(cuda::is_host_accessible(managed_ptr));
assert(cuda::is_device_accessible(managed_ptr, dev));
assert(cuda::is_managed(managed_ptr));
cudaFreeHost(host_ptr);
cudaFree(device_ptr);
cudaFree(managed_ptr);
}

View File

@@ -0,0 +1,67 @@
.. _libcudacxx-extended-api-memory-ptr_in_range:
``cuda::ptr_in_range``
======================
Defined in the header ``<cuda/memory>``.
.. code:: cuda
namespace cuda {
template <typename T>
[[nodiscard]] __host__ __device__ constexpr
bool ptr_in_range(T* ptr, T* start, T* end) noexcept;
} // namespace cuda
Checks whether ``ptr`` lies inside the half-open interval ``[start, end)``.
**Template parameters**
- ``T``: The type of the pointer.
**Parameters**
- ``ptr``: The pointer being tested.
- ``start``: Pointer to the first element in the range.
- ``end``: Pointer to one past the last element in the range.
**Return value**
- ``true`` when the pointer lies in ``[start, end)``, ``false`` otherwise.
**Preconditions**
- ``end`` must be greater than or equal to ``start``.
Example
-------
.. code:: cuda
#include <cuda/memory>
__global__ void kernel(float* data, size_t count) {
float* first = data;
float* last = data + count;
float* elem_ptr = data + threadIdx.x;
if (cuda::ptr_in_range(elem_ptr, first, last)) {
*elem_ptr = static_cast<float>(threadIdx.x);
}
}
int main() {
size_t N = 32;
float* device_ptr = nullptr;
cudaMalloc(&device_ptr, N * sizeof(float));
kernel<<<1, N>>>(device_ptr, N);
cudaDeviceSynchronize();
cudaFree(device_ptr);
return 0;
}
`See it on Godbolt 🔗 <https://godbolt.org/z/sMz76hGEc>`__

View File

@@ -0,0 +1,70 @@
.. _libcudacxx-extended-api-memory-ptr_rebind:
``cuda::ptr_rebind``
====================
Defined in the header ``<cuda/memory>``.
.. code:: cuda
namespace cuda {
template <typename U, typename T>
[[nodiscard]] __host__ __device__
U* ptr_rebind(T* ptr) noexcept;
template <typename U, typename T>
[[nodiscard]] __host__ __device__
const U* ptr_rebind(const T* ptr) noexcept;
template <typename U, typename T>
[[nodiscard]] __host__ __device__
volatile U* ptr_rebind(volatile T* ptr) noexcept;
template <typename U, typename T>
[[nodiscard]] __host__ __device__
const volatile U* ptr_rebind(const volatile T* ptr) noexcept;
} // namespace cuda
The functions return the pointer ``ptr`` cast to type ``U*`` or ``const U*``. They are shorter and safer alternative to ``reinterpret_cast``.
**Parameters**
- ``ptr``: The pointer.
**Return value**
- The pointer cast to type ``U*`` or ``const U*``.
**Constraints**
- ``ptr`` must be aligned to ``alignof(U)`` and ``alignof(T)``.
**Performance considerations**
- The returned pointer is decorated with ``__builtin_assume_aligned`` with the ``alignof(U)`` value to help the compiler generate better code.
- The returned pointer maintains the same memory space, for example shared memory, as the input pointer.
Example
-------
.. code:: cuda
#include <cuda/memory>
#include <cuda/std/cstdint>
__global__ void kernel(const int* ptr, volatile int* ptr2) {
auto ptr_res1 = cuda::ptr_rebind<uint64_t>(ptr); // ptr_res1: const uint64_t*
auto ptr_res2 = cuda::ptr_rebind<uint64_t>(ptr2); // ptr_res2: volatile uint64_t*
}
int main() {
int* ptr;
cudaMalloc(&ptr, 100 * sizeof(int));
kernel<<<1, 1>>>(ptr);
cudaDeviceSynchronize();
return 0;
}
`See it on Godbolt 🔗 <https://godbolt.org/z/bavzabce9>`__

View File

@@ -0,0 +1,60 @@
.. _libcudacxx-extended-api-memory-ranges_overlap:
``cuda::ranges_overlap``
========================
Defined in the ``<cuda/memory>`` header.
.. code:: cuda
namespace cuda {
template <typename T>
[[nodiscard]] __host__ __device__ constexpr
bool ranges_overlap(T lhs_start, T lhs_end, T rhs_start, T rhs_end) noexcept;
} // namespace cuda
Returns ``true`` when the half-open byte ranges ``[lhs_start, lhs_end)`` and ``[rhs_start, rhs_end)`` intersect.
**Constraints**
- ``T`` must be a forward iterator.
**Parameters**
- ``lhs_start``: The beginning of the first range.
- ``lhs_end``: The end of the first range.
- ``rhs_start``: The beginning of the second range.
- ``rhs_end``: The end of the second range.
**Return value**
- ``true`` when the two ranges overlap, ``false`` otherwise.
**Performance considerations**
- The function is optimized when the ranges are contiguous and random access iterators.
Example
-------
.. code:: cuda
#include <cuda/memory>
#include <cuda/std/cassert>
__global__ void overlap_kernel() {
int arrayA[10];
int arrayB[10];
assert(cuda::ranges_overlap(arrayA + 2, arrayA + 7, arrayA, arrayA + 10)); // overlap
assert(!cuda::ranges_overlap(arrayA, arrayA + 10, arrayB, arrayB + 10)); // no overlap
}
int main() {
overlap_kernel<<<1, 1>>>();
cudaDeviceSynchronize();
return 0;
}
`See it on Godbolt 🔗 <https://godbolt.org/z/nasnWz9Tv>`__