dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

feat(cccl): integrate missing CCCL directories — python/, ci/, .agent/, docs/, test/

Browse Source 
Sparse-checkout from NVIDIA/cccl main branch to complete cccl_upstream:

Added:
- python/cuda_cccl/ (226 files) — Python bindings for device-level algorithms
  Critical for muh toolchain: cuda.compute.reduce_into, scan, radix_sort, etc.
  Includes 204 .py files with full test coverage for all 27 algorithms
- ci/ (163 files) — Build/test infrastructure
  build_cub.sh, test_cub.sh, build_and_test_targets.sh, matrix.yaml
  Directly maps to our [INFRA-CI] and [INFRA-BUILD] items
- .agent/skills/ (7 files) — NVIDIA's own agent skills for CCCL
  cccl-style/SKILL.md, cccl-test/SKILL.md, sass-diff/SKILL.md
- docs/ (491 files) — Official CCCL documentation
  CI references, CMake guides, Python compute docs, libcudacxx PTX docs
- test/ (12 files) — Top-level integration tests (cuda_smoke, stdpar)
- Root configs: .clang-format, .clang-tidy, CONTRIBUTING.md, pyproject.toml
- CLAUDE.md symlink → AGENTS.md (NVIDIA's standard)

cccl_upstream now mirrors full NVIDIA/cccl structure:
  Before: 42M (cub + thrust + libcudacxx + cudax + c + examples + benchmarks)
  After:  53M (+python +ci +docs +.agent +test +configs)

This completes the CCCL base needed for:
- [muh-bench] items: ci/util/build_and_test_targets.sh for targeted builds
- [CCCL-verify] items: python/cuda_cccl/tests/ as reference implementations
- [CCCL-test] items: ci/test_cub.sh, ci/test_thrust.sh
- Agent workflow: .agent/skills/ for consistent style and test patterns
This commit is contained in:
muh-bot
2026-08-07 02:34:33 +00:00
parent 3f97dca7ad
commit 2a7ca101d7
908 changed files with 121615 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

79
cccl_upstream/docs/cub/developer/block_scope.rst Normal file
View File

@@ -0,0 +1,79 @@
.. _cub-developer-guide-block-scope:
Block-scope
************
Overview
=========
Block-scope algorithms are provided by structures as well:
.. code-block:: c++
template <typename T,
int BLOCK_DIM_X,
BlockReduceAlgorithm ALGORITHM = BLOCK_REDUCE_WARP_REDUCTIONS,
int BLOCK_DIM_Y = 1,
int BLOCK_DIM_Z = 1>
class BlockReduce {
public:
struct TempStorage : Uninitialized<_TempStorage> {};
// (1) new constructor
__device__ __forceinline__ BlockReduce()
: temp_storage(PrivateStorage()),
linear_tid(RowMajorTid(BLOCK_DIM_X, BLOCK_DIM_Y, BLOCK_DIM_Z)) {}
__device__ __forceinline__ BlockReduce(TempStorage &temp_storage)
: temp_storage(temp_storage.Alias()),
linear_tid(RowMajorTid(BLOCK_DIM_X, BLOCK_DIM_Y, BLOCK_DIM_Z)) {}
};
While warp-scope algorithms only provide a single constructor that requires the user to provide temporary storage,
block-scope algorithms provide two constructors:
#. The default constructor that allocates the required shared memory internally.
#. The constructor that requires the user to provide temporary storage as argument.
In the case of the default constructor,
the block-level algorithm uses the ``PrivateStorage()`` member function to allocate the required shared memory.
This ensures that shared memory required by the algorithm is only allocated when the default constructor is actually called in user code.
If the default constructor is never called,
then the algorithm will not allocate superfluous shared memory.
.. code-block:: c++
__device__ __forceinline__ _TempStorage& PrivateStorage()
{
__shared__ _TempStorage private_storage;
return private_storage;
}
The ``__shared__`` memory has static semantic, so it's safe to return a reference here.
Specialization
====================================
Block-scope facilities usually expose algorithm selection to the user.
The algorithm is represented by the enumeration part of the API.
For the reduction case,
``BlockReduceAlgorithm`` is provided.
Specializations are stored in the ``cub/block/specializations`` directory.
Temporary storage usage
====================================
For block-scope algorithms,
it's unsafe to use temporary storage without synchronization:
.. code-block:: c++
using BlockReduce = cub::BlockReduce<int, 128> ;
__shared__ BlockReduce::TempStorage temp_storage;
int aggregate_1 = BlockReduce(temp_storage).Sum(thread_data_1);
// illegal, has to add `__syncthreads` between the two
int aggregate_2 = BlockReduce(temp_storage).Sum(thread_data_2);
// illegal, has to add `__syncthreads` between the two
foo(temp_storage);

530
cccl_upstream/docs/cub/developer/device_scope.rst Normal file
View File

@@ -0,0 +1,530 @@
.. _cub-developer-guide-device-scope:
Device-scope
*************
Overview
=========
Device-scope functionality is provided by classes called ``DeviceAlgorithm``,
where ``Algorithm`` is the implemented algorithm.
These classes then contain static member functions providing corresponding API entry points.
For example, device-level reduce will look like `cub::DeviceReduce::Sum`.
Here is a generic example:
.. code-block:: c++
struct DeviceAlgorithm {
// two step API
template <typename ...>
static cudaError_t Algorithm(void *d_temp_storage, size_t &temp_storage_bytes, ..., cudaStream_t stream = 0) {
// optional: minimal argument checking or setup to call dispatch layer
return detail::algorithm::dispatch(d_temp_storage, temp_storage_bytes, ..., stream);
}
// environment API
template <typename ..., typename Env = cuda::std::execution::env<>>
static cudaError_t Algorithm(..., const Env& env = {}) {
// optional: minimal argument checking or setup to call dispatch layer
using default_policy_selector = detail::algorithm::policy_selector_from_types<...>;
return dispatch_with_env_and_tuning<default_policy_selector>(
env, [&](auto policy_selector, void* storage, size_t& bytes, auto stream) {
return detail::algorithm::dispatch(d_temp_storage, temp_storage_bytes, ..., stream, policy_selector);
});
}
};
Device-scope APIs come in two flavors, two step APIs and environment APIs,
both return ``cudaError_t`` and take algorithm specific arguments.
The two step API accepts a ``stream`` as the last parameter (``NULL`` stream by default)
and the first two parameters are always ``void *d_temp_storage, size_t &temp_storage_bytes``.
The environment API just takes an environment as the last parameter (empty environment by default).
The implementation may consist of some minimal argument checking, but should forward as soon as possible to the dispatch layer.
Device-scope algorithms are implemented in files located in `cub/device/device_***.cuh`.
The two step API is called in two phases:
1. Temporary storage size is calculated and returned in ``size_t &temp_storage_bytes``.
2. ``temp_storage_bytes`` of memory is expected to be allocated and ``d_temp_storage`` is expected to be the pointer to this memory.
The following example illustrates this pattern:
.. code-block:: c++
// First call: Determine temporary device storage requirements
std::size_t temp_storage_bytes = 0;
cub::DeviceReduce::Sum(/*d_temp_storage*/ nullptr, temp_storage_bytes, d_in, d_out, num_items);
// Allocate temporary storage
thrust::device_vector<unsigned char> temp_storage(temp_storage_bytes, thrust::no_init);
// Second call: Perform algorithm
cub::DeviceReduce::Sum(temp_storage.data().get(), temp_storage_bytes, d_in, d_out, num_items);
.. warning::
Even if the algorithm doesn't need temporary storage as scratch space,
the overload with ``void *d_temp_storage, size_t &temp_storage_bytes``
still requires one byte of memory to be allocated.
The environment overloads just require a single call, but setting up the environment may be more complex:
.. code-block:: c++
// Setup environment, everything is optional
auto env = cuda::std::execution::env{
stream_ref,
memory_resource,
cuda::execution::require(requirements...),
cuda::execution::tune(policy_selectors....)
};
// Perform algorithm
cub::DeviceReduce::Sum(d_in, d_out, num_items, env);
The device layer handles the extraction of the:
* CUDA stream
* memory resource
* requirements
* guarantees (TODO(bgruber): are those public?)
* :ref:`tuning policy selectors <cub-policy-selectors>`
from the environment argument, or provide default values in case the environment does not contain them.
They typically use helper functions like ``dispatch_with_env`` and ``dispatch_with_env_and_tuning``.
Some CUB APIs require no temporary storage and may omit the ``void *d_temp_storage, size_t &temp_storage_bytes`` parameters.
Their environment overloads will also ignore any passed memory resource.
Dispatch layer
====================================
A dispatch function exists for each device-scope algorithm (e.g., ``detail::reduce::dispatch``),
and is located in ``cub/device/dispatch``.
Most device-scope algorithms share one dispatch function.
Only device-scope algorithms have dispatch functions, which are also referred to as the dispatch layer.
The dispatch layer follows a certain architecture.
The high-level control flow is represented by the code below.
A more precise description is given later.
..
TODO(bgruber): consider removing the numbers below. control flow is now easier to follow
.. code-block:: c++
// Device-scope API
cudaError_t cub::DeviceAlgorithm::Algorithm(d_temp_storage, temp_storage_bytes, ...) {
return detail::algorithm::dispatch(d_temp_storage, temp_storage_bytes, ...);
}
namespace detail::algorithm {
cudaError_t dispatch(
void *d_temp_storage, size_t &temp_storage_bytes,
...,
cudaStream_t stream, PolicySelector policy_selector = {}) {
cuda::compute_capability cc{};
ptx_compute_cap(cc);
const /*or constexpr*/ AlgorithmPolicy active_policy = policy_selector(cc);
// host-side implementation of algorithm, calls kernels
kernel<PolicySelector><<<grid_size, active_policy.threads_per_block>>>(...);
}
template <typename PolicySelector>
__launch_bounds__(int(current_policy<PolicySelector>().threads_per_block))
void kernel(...) {
static constexpr auto policy = current_policy<PolicySelector>();
using agent_policy = AgentPolicy<policy.threads_per_block, policy.items_per_thread, ...>;
using agent = AgentAlgorithm<agent_policy, InputIteratorT, OffsetT, ...>;
agent a{...};
a.Process();
}
template <int ThreadsPerBlock, ...>
struct AgentPolicy { // legacy
static constexpr threads_per_block = ThreadsPerBlock;
...
};
template <typename Policy, ...>
struct AlgorithmAgent {
void Process() { ... }
};
}
Let's look at each of the building blocks closer.
The dispatch function
------------------------------------
The dispatch function is typically a simple function template called ``dispatch`` inside an algorithm-specific namespace.
It basically receives the same parameters as the public API entry point,
but they may have already been modified, extended, or generalized (to map many public APIs to the same dispatch function).
The goal of the dispatch function is to setup the execution of the algorithm on the device
by selecting the appropriate policy for the current GPU's compute capability,
preparing temporary storage and shared memory, configuring kernel launches, launching kernels, etc.
There are two style of dispatch functions, depending on whether the policy is needed as runtime or compile-time value:
.. code-block:: c++
namespace detail::algorithm {
// Dispatch version A - runtime policy
cudaError_t dispatch(
void *d_temp_storage, size_t &temp_storage_bytes,
...,
cudaStream_t stream, PolicySelector policy_selector = {}) {
cuda::compute_capability cc{};
ptx_compute_cap(cc);
const auto active_policy = policy_selector(cc); // runtime-time policy
// host-side implementation of algorithm, calls kernels
kernel<PolicySelector><<<grid_size, active_policy.threads_per_block>>>(...);
}
}
The dispatch function starts by querying the target compute capability for which compiled GPU code (PTX or SASS) is available,
by calling ``ptx_compute_cap``.
If the host code does not require the policy for this compute capability at compile-time (version A),
we can just pass the compute capability to the :ref:`policy selector <cub-policy-selectors>` to obtain the tuning policy at runtime.
The values from the policy are then used to setup resources and the kernel.
The kernel is then instantiated using only the type of the policy selector (not a concrete tuning policy),
so there is only one kernel instantiation across all target architectures compiled for.
More on that later.
If the host code needs the policy as a compile-time value (version B), we have to use ``dispatch_compute_cap``.
dispatch_compute_cap
------------------------------------
``dispatch_compute_cap`` maps a runtime ``cuda::compute_capability`` to a compile-time policy value
and calls the user-provided functor ``f`` with a nullary callable (a ``policy_getter``)
that returns the policy as a compile-time constant.
The policy getter is necessary to work around a C++17 limitation.
.. code-block:: c++
namespace detail::algorithm {
// Dispatch version B - compile-time policy
cudaError_t dispatch(
void *d_temp_storage, size_t &temp_storage_bytes,
...,
cudaStream_t stream, PolicySelector policy_selector = {}) {
cuda::compute_capability cc{};
ptx_compute_cap(cc);
return dispatch_compute_cap(policy_selector, cc, [&](auto policy_getter) {
constexpr auto active_policy = policy_getter(); // compile-time policy
static_assert(active_policy.tile_size() * sizeof(T) <= 48 * 1024, "Not enough SMEM");
// host-side implementation of algorithm, calls kernels
kernel<PolicySelector><<<grid_size, active_policy.threads_per_block>>>(...);
});
}
template <typename PolicySelector, typename F>
cudaError_t dispatch_compute_cap(PolicySelector, cuda::compute_capability cc, F&& f) { // (2)
// fold over __CUDA_ARCH_LIST__, calling f with a policy_getter
// that returns the policy for the matching arch as a compile-time value
}
}
Inside the lambda, ``policy_getter()`` returns the selected policy as a constant expression,
so compile-time branching (i.e. ``if constexpr``) or static assertions using policy values is possible.
Internally, ``dispatch_compute_cap`` uses ``__CUDA_ARCH_LIST__`` / ``NV_TARGET_SM_INTEGER_LIST`` (or all known compute capabilities as fallback)
to create one instantiation of ``f`` per distinct value of ``policy_selector(cc)`` (not per compute capability).
This results in one template instantiation of ``f`` per distinct policy value.
The kernel is then again only instantiated once using the type of the policy selector.
Kernels
------------------------------------
Kernels are templated on the ``PolicySelector`` type,
which is stateless and the same for all target architectures compiled for.
.. code-block:: c++
namespace detail::algorithm {
template <typename PolicySelector, typename InputIteratorT, typename OffsetT, /* ... */>
__launch_bounds__(int(current_policy<PolicySelector>().reduce.threads_per_block))
void DeviceReduceKernel(InputIteratorT d_in, OffsetT num_items, /* ... */) {
static constexpr auto policy = current_policy<PolicySelector>();
using agent_policy = AgentPolicy<policy.threads_per_block, policy.items_per_thread, ...>;
using agent = AgentAlgorithm<agent_policy, InputIteratorT, OffsetT, ...>;
agent a{...};
a.Process();
}
template <typename PolicySelector>
constexpr auto current_policy() {
return PolicySelector{}(cuda::compute_capability{__CUDA_ARCH__ / 10}); // simplified
}
}
``PolicySelector`` must be stateless (``is_empty_v<PolicySelector>`` is ``true``),
so it can be default-constructed in device code where needed.
The utility function ``current_policy`` can only be called in device code.
It selects the target compute capability based on compiler macros of the current device compilation pass
and retrieves a tuning policy from the policy selector.
The kernel typically uses ``current_policy`` in two places,
to get the block size to define the launch bounds,
and inside the kernel to setup various sub algorithms (like agents).
Because C++17 does not allow to pass structs as Non-Type Template Parameters (NTTPs),
we cannot easily pass the policy around to other functions,
so it will be converted to legacy agent policies in many places.
Those are just structs with static data members holding the policy value as a type,
so they can be passed to templates.
Agent policy structs have historically been part of the public API, and should be removed in the future.
.. warning::
The kernel gets compiled for each target architecture (N many) that was provided to the compiler.
During each device pass, ``current_policy`` may return a different policy.
During the host pass, version A (runtime policy) compiles a single instantiation of the dispatch logic for all target architectures.
Version B (using ``dispatch_compute_cap``) compiles the dispatch logic for each distinct tuning policy (M many).
If we passed the selected tuning policy instead of the policy selector as a kernel template parameter,
the kernel template instantiation would be different for each tuning policy value and
we would compile O(M*N) kernels for version B instead of O(N).
Many kernels are short, since the functionality is extracted into the agent layer.
All the kernel does is derive the proper policy,
unwrap it to initialize the agent and call one of its ``Consume`` / ``Process`` functions.
Agents hold kernel bodies and are intended to be reused across multiple device-scope algorithms.
However, this is not a requirement and some kernels just contain their entire implementation themselves.
The default policy selector
------------------------------------
CUB contains a default policy selector for each dispatch function.
Because many dispatch functions are also used by CCCL.C,
which compiles them without proper type information,
we have to provide them in two forms,
a typeless and a typeful version.
.. code-block:: c++
struct AlgorithmPolicy { ... }; // unrelated to AgentPolicy
namespace detail::algorithm {
struct policy_selector {
type_t accum_t;
op_kind_t operation_t;
int offset_size;
int accum_size;
constexpr auto operator()(::cuda::compute_capability cc) const -> AlgorithmPolicy {
// parameter selection across target compute capabilities and input characteristics (possibly HUGE logic)
}
};
template <typename AccumT, typename OffsetT, typename ReductionOpT>
struct policy_selector_from_types {
constexpr auto operator()(cuda::compute_capability cc) const -> AlgorithmPolicy {
constexpr auto ps = policy_selector{
classify_type<AccumT>(),
classify_op<ReductionOpT>(),
int{sizeof(OffsetT)},
int{sizeof(AccumT)}
};
return ps(cc);
}
};
}
The ``policy_selector`` is intended to be used without template-parameter-based type information
and is thus suitable to be used in CCCL.C without a JIT compiler for host code.
It contains the necessary information on the algorithm's input as data members.
This policy is only used in the host code of the dispatch function.
Because it is not stateless anymore, CCCL.C overrides the kernel launcher used by CUB,
providing the kernel from a JIT-compiled instantiation that uses a proper stateless policy selector.
The kernel, and the dispatch function when not called from CCCL.C,
will use the second version of the policy selector, ``policy_selector_from_types``,
which offers proper template parameters to pass type information.
This type is stateless and delegates to a ``constexpr`` instance of the ``policy_selector``
with compile-time values derived from the template parameters.
The policy selection logic is thus the same for CCCL.C and CUB,
the type information is just provided differently.
Each dispatch function also has an associated concept for the policy selector it expects:
.. code-block:: c++
template <typename T, typename Policy>
concept policy_selector = requires(T pol_sel, cuda::compute_capability cc) {
requires std::regular<Policy>;
{ pol_sel(cc) } -> std::same_as<Policy>;
};
namespace detail::algorithm {
template <typename T>
concept algorithm_policy_selector = policy_selector<T, AlgorithmPolicy>;
}
The concept basically checks whether the policy selector can be called with a ``cuda::compute_capability``
and returns the expected policy struct.
The default policy selectors and related concept are defined in ``cub/device/dispatch/tuning/tuning_<algorithm>.cuh``.
Backward compatibility
------------------------------------
Legacy public dispatchers (e.g. ``DispatchReduce``) are deprecated.
They continue to work by translating the ``PolicyHub`` template parameter to the new ``policy_selector``
via a ``policy_selector_from_hub`` adapter:
.. code-block:: c++
template <typename PolicyHub>
struct policy_selector_from_hub {
constexpr auto operator()(cuda::compute_capability cc) const -> AlgorithmPolicy {
// conversion logic
}
};
This allows existing user code that passes custom policy hubs to dispatchers to continue working.
The legacy dispatchers, policy hubs and related logic are scheduled for removal in CCCL 4.0.
Policies
====================================
Policies describe the configuration of agents or just kernels with respect to performance.
They must not change functional behavior, but affect how work is mapped to the hardware
by defining certain parameters (items per thread, block size, etc.),
or choosing between algorithms.
Policies must be plain semiregular aggregates to allow using them during constant evaluation,
and with designated initializers:
.. code-block:: c++
struct AlgorithmPolicy {
int threads_per_block;
int items_per_thread;
cub::BlockLoadAlgorithm load_algorithm;
cub::CacheLoadModifier load_modifier;
friend constexpr bool operator==(const AlgorithmPolicy& lhs, const AlgorithmPolicy& rhs) { ... }
friend constexpr bool operator!=(const AlgorithmPolicy& lhs, const AlgorithmPolicy& rhs) { ... }
friend std::ostream& operator<<(std::ostream& os, const AlgorithmPolicy& p) { ... }
};
Tuning policies can have various complexities and contain nested structures.
Policies are defined in ``cub/device/dispatch/tuning/tuning_<algorithm>.cuh``.
Tunings
====================================
Because the values to parameterize an agent may vary a lot for different compile-time parameters,
the selection of values can involve complex logic.
Often, such tunings are found by experimentation or heuristic search.
See also :ref:`cub-tuning-infra`.
Tunings are expressed as logic and values inside the ``constexpr operator()`` of a policy selector.
Because of the complexity of some policy selectors, nested functions may be used.
Many policy selectors also implement a fallback logic,
where they try to find a matching tuning based on the input characteristics (policy selector data members),
but if no match is found, they fall back to an older target compute capability.
Here is an example:
.. code-block:: c++
struct sm90_tuning_values {
int items;
int threads;
int items_per_vec_load;
};
constexpr auto get_sm90_tuning(type_t accum_t, op_kind_t op, int offset_size, int accum_size)
-> std::optional<sm90_tuning_values> {
// provide tunings for certain operations, accumulator or offset types
if (op == op_kind_t::plus) {
if (accum_t == type_t::float32 && offset_size == 4 && accum_size == 4)
return sm90_tuning_values{16, 512, 2};
if (offset_size == 4 && accum_size == 8)
return sm90_tuning_values{15, 512, 2};
}
return {}; // no tuning available, causes fallback
}
struct policy_selector {
type_t accum_t;
op_kind_t operation_t;
int offset_size;
int accum_size;
constexpr auto operator()(cuda::compute_capability cc) const -> reduce_policy {
if (cc >= cuda::compute_capability{9, 0}) {
if (auto tuning = get_sm90_tuning(accum_t, operation_t, offset_size, accum_size)) {
return *tuning; // found a tuning, use it
}
// fall through to sm_80 if no matching tuning found
}
if (cc >= cuda::compute_capability{8, 0}) {
return { /* sm_80 default policy */ };
}
return { /* default policy for everything else */ };
}
};
In general, tunings are not exhaustive and usually only apply for specific combinations
of parameter values and a single compute capability.
This is because they originate from tuning benchmarks running for specific workloads on specific target architectures.
Generic fallbacks are often just retained from earlier days of CUB to not risk regressions,
or are based on heuristics trying to provide reasonable performance based on a model of the GPU architecture or algorithm.
Tunings for CUB algorithms reside in ``cub/device/dispatch/tuning/tuning_<algorithm>.cuh``.
Temporary storage usage
====================================
It's safe to reuse storage in the stream order:
.. code-block:: c++
cub::DeviceReduce::Sum(nullptr, storage_bytes, d_in, d_out, num_items, stream_1);
// allocate temp storage
cub::DeviceReduce::Sum(d_storage, storage_bytes, d_in, d_out, num_items, stream_1);
// fine not to synchronize stream
cub::DeviceReduce::Sum(d_storage, storage_bytes, d_in, d_out, num_items, stream_1);
// illegal, should call cudaStreamSynchronize(stream)
cub::DeviceReduce::Sum(d_storage, storage_bytes, d_in, d_out, num_items, stream_2);
Temporary storage management
====================================
Often times temporary storage for device-scope algorithms has a complex structure.
To simplify temporary storage management and make it safer,
we introduced ``cub::detail::temporary_storage::layout``:
.. code-block:: c++
cub::detail::temporary_storage::layout<2> storage_layout;
auto slot_1 = storage_layout.get_slot(0);
auto slot_2 = storage_layout.get_slot(1);
auto allocation_1 = slot_1->create_alias<int>();
auto allocation_2 = slot_1->create_alias<double>(42);
auto allocation_3 = slot_2->create_alias<char>(12);
if (condition)
{
allocation_1.grow(num_items);
}
if (d_temp_storage == nullptr)
{
temp_storage_bytes = storage_layout.get_size();
return;
}
storage_layout.map_to_buffer(d_temp_storage, temp_storage_bytes);
// different slots, safe to use simultaneously
use(allocation_1.get(), allocation_3.get(), stream);
// `allocation_2` alias `allocation_1`, safe to use in stream order
use(allocation_2.get(), stream);

17
cccl_upstream/docs/cub/developer/nvtx.rst Normal file
View File

@@ -0,0 +1,17 @@
.. _cub-developer-guide-nvtx:
NVTX
=====
The `NVIDIA Tools Extension SDK (NVTX) <https://nvidia.github.io/NVTX/>`_ is a cross-platform API
for annotating source code to provide contextual information to developer tools.
All device-scope algorithms in CUB are annotated with NVTX ranges,
allowing their start and stop to be visualized in profilers
like `NVIDIA Nsight Systems <https://developer.nvidia.com/nsight-systems>`_.
Only the public APIs available in the ``<cub/device/device_xxx.cuh>`` headers are annotated,
excluding direct calls to the dispatch layer.
NVTX annotations can be disabled by defining ``NVTX_DISABLE`` during compilation.
When CUB device algorithms are called on a stream subject to
`graph capture <https://developer.nvidia.com/blog/cuda-graphs/>`_,
the NVTX range is reported for the duration of capture (where no execution happens),
and not when a captured graph is executed later (the actual execution).

328
cccl_upstream/docs/cub/developer/test_overview.rst Normal file
View File

@@ -0,0 +1,328 @@
CUB Tests
###########################
.. warning::
CUB is in the progress of migrating to [Catch2](https://github.com/catchorg/Catch2) framework.
CUB tests rely on `CPM <https://github.com/cpm-cmake/CPM.cmake>`_ to fetch
`Catch2 <https://github.com/catchorg/Catch2>`_ that's used as our main testing framework.
Currently,
legacy tests coexist with Catch2 ones.
This guide is focused on new tests.
.. important::
Instead of including ``<catch2/catch.hpp>`` directly, use ``catch2_test_helper.h``.
.. code-block:: c++
#include <cub/block/block_scan.cuh>
#include <c2h/vector.h>
#include <c2h/catch2_test_helper.h>
Directory and File Naming
*************************************
Our tests can be found in the ``test`` directory.
Legacy tests have the following naming scheme: ``test_SCOPE_FACILITY.cu``.
For instance, here are the reduce tests:
.. code-block:: c++
test/test_warp_reduce.cu
test/test_block_reduce.cu
test/test_device_reduce.cu
Catch2-based tests have a different naming scheme: ``catch2_test_SCOPE_FACILITY.cu``.
The prefix is essential since that's how CMake finds tests
and distinguishes new tests from legacy ones.
Test Structure
*************************************
Base case
=====================================
Let's start with a simple example.
Say there's no need to cover many types with your test.
.. code-block:: c++
// 0) Define test name and tags
C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]")
{
using type = std::int32_t;
constexpr int threads_per_block = 256;
constexpr int num_items = threads_per_block;
// 1) Allocate device input
c2h::device_vector<type> d_input(num_items);
// 2) Generate 3 random input arrays using Catch2 helper
c2h::gen(C2H_SEED(3), d_input);
// 3) Allocate output array
c2h::device_vector<type> d_output(d_input.size());
// 4) Copy device input to host
c2h::host_vector<key_t> h_reference = d_input;
// 5) Compute reference output
std::ALGORITHM(
thrust::raw_pointer_cast(h_reference.data()),
thrust::raw_pointer_cast(h_reference.data()) + h_reference.size());
// 6) Compute CUB output
SCOPE_ALGORITHM<threads_per_block>(d_input.data(),
d_output.data(),
d_input.size());
// 7) Compare device and host results
REQUIRE( d_input == d_output );
}
We introduce test cases with the ``C2H_TEST`` macro in (0).
This macro always takes two string arguments - a free-form test name and
one or more tags. Then, in (1), we allocate device memory using ``c2h::device_vector``.
``c2h::device_vector`` and ``c2h::host_vector`` behave similarly to their Thrust counterparts,
but are modified to provide more stable behavior in some testing edge cases.
.. important::
Always use ``c2h::host_vector<T>``/``c2h::device_vector<T>``
instead of ``thrust::host_vector<T>``/``thrust::device_vector<T>``,
unless the test code is being used for documentation examples.
Similarly, any thrust algorithms that executed on the device must be invoked with the
`c2h::device_policy` execution policy (not shown here) to support the same edge cases.
The memory is filled with random data in (2).
Generator ``c2h::gen`` takes at least two parameters.
The first one is a random generator seed.
Instead of providing a single value, we use the ``C2H_SEED`` macro.
The macro expects a number of seeds that has to be generated.
In the example above, we require three random seeds to be generated.
This leads to the whole test being executed three times
with different seed values.
Later, in (3), we allocate device output and host reference.
In (4), we allocate and populate the host input data.
Then, we perform the reference computation on the host in (5).
.. important::
Standard library algorithms (``std::``) have to be used where possible when computing reference solutions.
Afterwards, we launch the corresponding CUB algorithm in (6).
At this point, we have a reference solution on the CPU and a CUB solution on the GPU.
The two can be compared using Catch2's ``REQUIRE`` macro, which stops execution upon failure (preferred).
Catch2 also offers the ``CHECK`` macro, which continues test execution if the check fails.
If your test has to cover floating point types,
it's sufficient to replace ``REQUIRE( a == b )`` with ``REQUIRE_APPROX_EQ(a, b)``.
.. important::
Using ``c2h::gen`` for producing input data is strongly advised.
Do not use ``assert`` in tests, which is usually only enabled in Debug mode,
and we run CUB tests in Release mode.
If a custom (non-fundamental) type has to be tested, the following helper class template should be used:
.. code-block:: c++
using type = c2h::custom_type_t<c2h::accumulateable_t,
c2h::equal_comparable_t>;
Here we enumerate all the type properties that we are interested in.
The produced type ends up having ``operator==`` (from ``equal_comparable_t``)
and ``operator+`` (from ``accumulateable_t``).
More properties are available.
If a property is missing, please add it to the existing set in ``c2h``
instead of writing a custom type from scratch.
Generators
=====================================
We often need to test CUB algorithms against different inputs or problem sizes.
If these are **runtime values**, we can use the Catch2 ``GENERATE`` macro:
.. code-block:: c++
C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]")
{
int num_items = GENERATE(1, 100, 1'000'000); // 0) Init. a variable with a generator
// ...
}
This will lead to the test being executed three times, once for each argument to ``GENERATE(...)``.
Multiple generators in a test inside the same scope will form the cartesian product of all combinations.
Please consult the `Catch2 documentation <https://github.com/catchorg/Catch2/blob/devel/docs/generators.md>`_
for more details.
``C2H_SEED(3)`` uses a generator expression internally.
Type Lists
=====================================
Since CUB is a generic library,
it's often required to test CUB algorithms against many types.
To do so,
it's sufficient to define a type list and provide it to the ``C2H_TEST`` macro.
This is useful for **compile-time** parameterization of tests.
.. code-block:: c++
// 0) Define type list
using types = c2h::type_list<std::uint8_t, std::int32_t>;
C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]",
types) // 1) Provide it to the test case
{
// 2) Access current type with `c2h::get`
using type = typename c2h::get<0, TestType>;
// ...
}
This will lead to the test being compiled (instantiated) and run twice.
The first run will cause ``type`` to be ``std::uint8_t``.
The second one will cause ``type`` to be ``std::uint32_t``.
.. warning::
It's important to use types from the ``<cstdint>`` header
instead of built-in types like ``char`` and ``int``.
Multidimensional Configuration Spaces
=====================================
In most cases, the input data type is not the only compile-time parameter we want to vary.
For instance, you might need to test a block algorithm for different data types
**and** different thread block sizes.
To do so, you can add another type list as follows:
.. code-block:: c++
using block_sizes = c2h::enum_type_list<int, 128, 256>;
using types = c2h::type_list<std::uint8_t, std::int32_t>;
C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]",
types, block_sizes)
{
using type = typename c2h::get<0, TestType>;
constexpr int threads_per_block = c2h::get<1, TestType>::value;
// ...
}
The code above leads to the following combinations being compiled:
- ``type = std::uint8_t``, ``threads_per_block = 128``
- ``type = std::uint8_t``, ``threads_per_block = 256``
- ``type = std::int32_t``, ``threads_per_block = 128``
- ``type = std::int32_t``, ``threads_per_block = 256``
As an example, the following test case includes both multidimensional configuration spaces
and multiple random sequence generations.
.. code-block:: c++
using block_sizes = c2h::enum_type_list<int, 128, 256>;
using types = c2h::type_list<std::uint8_t, std::int32_t>;
C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]",
types, block_sizes)
{
using type = typename c2h::get<0, TestType>;
constexpr int threads_per_block = c2h::get<1, TestType>::value;
// ...
c2h::device_vector<type> d_input(5);
c2h::gen(C2H_SEED(2), d_input);
}
The code above leads to the following combinations being compiled:
- ``type = std::uint8_t``, ``threads_per_block = 128``, 1st random generated input sequence
- ``type = std::uint8_t``, ``threads_per_block = 256``, 1st random generated input sequence
- ``type = std::int32_t``, ``threads_per_block = 128``, 1st random generated input sequence
- ``type = std::int32_t``, ``threads_per_block = 256``, 1st random generated input sequence
- ``type = std::uint8_t``, ``threads_per_block = 128``, 2nd random generated input sequence
- ``type = std::uint8_t``, ``threads_per_block = 256``, 2nd random generated input sequence
- ``type = std::int32_t``, ``threads_per_block = 128``, 2nd random generated input sequence
- ``type = std::int32_t``, ``threads_per_block = 256``, 2nd random generated input sequence
Each new generator multiplies the number of execution times by its number of seeds. That means
that if there were further more sequence generators (``c2h::gen(C2H_SEED(X), ...)``) on the
example above the test would execute X more times and so on.
Speedup Compilation Time
=====================================
Since type lists in the ``C2H_TEST`` form a Cartesian product,
compilation time grows quickly with every new dimension.
To keep the compilation process parallelized,
it's possible to rely on our ``%PARAM%`` machinery:
.. code-block:: c++
// %PARAM% BLOCK_SIZE bs 128:256
using block_sizes = c2h::enum_type_list<int, BLOCK_SIZE>;
using types = c2h::type_list<std::uint8_t, std::int32_t>;
C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]",
types, block_sizes)
{
using type = typename c2h::get<0, TestType>;
constexpr int threads_per_block = c2h::get<1, TestType>::value;
// ...
}
The comment with ``%PARAM%`` is recognized by our CMake scripts.
It leads to multiple executables being produced from a single test source.
.. code-block:: bash
bin/cub.test.scope_algorithm.bs_128
bin/cub.test.scope_algorithm.bs_256
Multiple ``%PARAM%`` comments can be specified forming another Cartesian product.
Final Test
=====================================
Let's consider the final test that illustrates all of the tools we discussed above:
.. code-block:: c++
// %PARAM% BLOCK_SIZE bs 128:256
using block_sizes = c2h::enum_type_list<int, BLOCK_SIZE>;
using types = c2h::type_list<std::uint8_t, std::int32_t>;
C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]",
types, block_sizes)
{
using type = typename c2h::get<0, TestType>;
constexpr int threads_per_block = c2h::get<1, TestType>::value;
constexpr int max_num_items = threads_per_block;
c2h::device_vector<type> d_input(
GENERATE_COPY(take(2, random(0, max_num_items))));
c2h::gen(C2H_SEED(3), d_input);
c2h::device_vector<type> d_output(d_input.size());
SCOPE_ALGORITHM<threads_per_block>(d_input.data(),
d_output.data(),
d_input.size());
REQUIRE( d_input == d_output );
const type expected_sum = 4;
const type sum = thrust::reduce(c2h::device_policy, d_output.cbegin(), d_output.cend());
REQUIRE( sum == expected_sum);
}
Apart from discussed tools, here we also rely on ``Catch2`` to generate random input sizes
in the range ``[0, max_num_items]`` for our input vector ``d_input``.
Overall, the test will produce two executables.
Each of these executables is going to generate ``2`` input problem sizes.
For each problem size, ``3`` random vectors are generated.
As a result, we have ``12`` different tests.
The code also demonstrates the syntax and usage of ``c2h::device_policy`` with a Thrust algorithm.

24
cccl_upstream/docs/cub/developer/thread_level.rst Normal file
View File

@@ -0,0 +1,24 @@
.. _cub-developer-guide-thread-level:
Thread-level
*************
In contrast to algorithms at the warp/block/device layer,
single threaded functionality like ``cub::ThreadReduce``
is typically implemented as a sequential function and rarely exposed to the user.
.. code-block:: c++
template <
int LENGTH,
typename T,
typename ReductionOp,
typename PrefixT,
typename AccumT = detail::accumulator_t<ReductionOp, PrefixT, T>>
__device__ __forceinline__ AccumT ThreadReduce(
T (&input)[LENGTH],
ReductionOp reduction_op,
PrefixT prefix)
{
return ...;
}

157
cccl_upstream/docs/cub/developer/warp_level.rst Normal file
View File

@@ -0,0 +1,157 @@
.. _cub-developer-guide-warp-level:
Warp-level
************************************
CUB warp-level algorithms are specialized for execution by threads in the same CUDA warp.
These algorithms may only be invoked by ``1 <= n <= 32`` *consecutive* threads in the same warp.
Overview
====================================
Warp-level functionality is provided by types (classes) to provide encapsulation and enable partial template specialization.
For example, :cpp:struct:`cub::WarpReduce` is a class template:
.. code-block:: c++
template <typename T,
int LOGICAL_WARP_THREADS = 32>
class WarpReduce {
// ...
// (1) define `_TempStorage` type
// ...
_TempStorage &temp_storage;
public:
// (2) wrap `_TempStorage` in uninitialized memory
struct TempStorage : Uninitialized<_TempStorage> {};
__device__ __forceinline__ WarpReduce(TempStorage &temp_storage)
// (3) reinterpret cast
: temp_storage(temp_storage.Alias())
{}
// (4) actual algorithms
__device__ __forceinline__ T Sum(T input);
};
In CUDA, the hardware warp size is 32 threads.
However, CUB enables warp-level algorithms on "logical" warps of ``1 <= n <= 32`` threads.
The size of the logical warp is required at compile time via the ``LOGICAL_WARP_THREADS`` non-type template parameter.
This value is defaulted to the hardware warp size of ``32``.
There is a vital difference in the behavior of warp-level algorithms that depends on the value of ``LOGICAL_WARP_THREADS``:
- If ``LOGICAL_WARP_THREADS`` is a power of two - warp is partitioned into *sub*-warps,
each reducing its data independently from other *sub*-warps.
The terminology used in CUB: ``32`` threads are called hardware warp.
Groups with less than ``32`` threads are called *logical* or *virtual* warp since it doesn't correspond directly to any hardware unit.
- If ``LOGICAL_WARP_THREADS`` is **not** a power of two - there's no partitioning.
That is, only the first logical warp executes algorithm.
.. TODO: Add diagram showing non-power of two logical warps.
Temporary storage usage
====================================
Warp-level algorithms require temporary storage for scratch space and inter-thread communication.
The temporary storage needed for a given instantiation of an algorithm is known at compile time
and is exposed through the ``TempStorage`` member type definition.
It is the caller's responsibility to create this temporary storage and provide it to the constructor of the algorithm type.
It is possible to reuse the same temporary storage for different algorithm invocations,
but it is unsafe to do so without first synchronizing to ensure the first invocation is complete.
.. TODO: Add more explanation of the `TempStorage` type and the `Uninitialized` wrapper.
.. TODO: Explain if `TempStorage` is required to be shared memory or not.
.. code-block:: c++
using WarpReduce = cub::WarpReduce<int>;
// Allocate WarpReduce shared memory for four warps
__shared__ WarpReduce::TempStorage temp_storage[4];
// Get this thread's warp id
int warp_id = threadIdx.x / 32;
int aggregate_1 = WarpReduce(temp_storage[warp_id]).Sum(thread_data_1);
// illegal, has to add `__syncwarp()` between the two
int aggregate_2 = WarpReduce(temp_storage[warp_id]).Sum(thread_data_2);
// illegal, has to add `__syncwarp()` between the two
foo(temp_storage[warp_id]);
Specialization
====================================
The goal of CUB is to provide users with algorithms that abstract the complexities of achieving speed-of-light performance across a variety of use cases and hardware.
It is a CUB developer's job to abstract this complexity from the user by providing a uniform interface that statically dispatches to the optimal code path.
This is usually accomplished via customizing the implementation based on compile time information like the logical warp size, the data type, and the target architecture.
For example, :cpp:struct:`cub::WarpReduce` dispatches to two different implementations based on if the logical warp size is a power of two (described above):
.. code-block:: c++
using InternalWarpReduce = cuda::std::conditional_t<
IS_POW_OF_TWO,
detail::WarpReduceShfl<T, LOGICAL_WARP_THREADS>, // shuffle-based implementation
detail::WarpReduceSmem<T, LOGICAL_WARP_THREADS>>; // smem-based implementation
Specializations provide different shared memory requirements,
so the actual ``_TempStorage`` type is defined as:
.. code-block:: c++
using _TempStorage = typename InternalWarpReduce::TempStorage;
and algorithm implementation look like:
.. code-block:: c++
__device__ __forceinline__ T Sum(T input, int valid_items) {
return InternalWarpReduce(temp_storage)
.Reduce(input, valid_items, ::cuda::std::plus<>{});
}
``__CUDA_ARCH__`` cannot be used because it is conflicting with the PTX dispatch refactoring and limited NVHPC support.
Due to this limitation, we can't specialize on the PTX version.
``NV_IF_TARGET`` shall be used by specializations instead:
.. code-block:: c++
template <typename T, int LOGICAL_WARP_THREADS>
struct WarpReduceShfl
{
template <typename ReductionOp>
__device__ __forceinline__ T ReduceImpl(T input, int valid_items,
ReductionOp reduction_op)
{
// ... base case (SM < 80) ...
}
template <class U = T>
__device__ __forceinline__
typename std::enable_if<std::is_same_v<int, U> ||
std::is_same_v<unsigned int, U>,
T>::type
ReduceImpl(T input,
int, // valid_items
::cuda::std::plus<>) // reduction_op
{
T output = input;
NV_IF_TARGET(NV_PROVIDES_SM_80,
(output = __reduce_add_sync(member_mask, input);),
(output = ReduceImpl<::cuda::std::plus<>>(
input, LOGICAL_WARP_THREADS, ::cuda::std::plus<>{});));
return output;
}
};
Specializations are stored in the ``cub/warp/specializations`` directory.