dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

feat(cccl): integrate missing CCCL directories — python/, ci/, .agent/, docs/, test/

Browse Source 
Sparse-checkout from NVIDIA/cccl main branch to complete cccl_upstream:

Added:
- python/cuda_cccl/ (226 files) — Python bindings for device-level algorithms
  Critical for muh toolchain: cuda.compute.reduce_into, scan, radix_sort, etc.
  Includes 204 .py files with full test coverage for all 27 algorithms
- ci/ (163 files) — Build/test infrastructure
  build_cub.sh, test_cub.sh, build_and_test_targets.sh, matrix.yaml
  Directly maps to our [INFRA-CI] and [INFRA-BUILD] items
- .agent/skills/ (7 files) — NVIDIA's own agent skills for CCCL
  cccl-style/SKILL.md, cccl-test/SKILL.md, sass-diff/SKILL.md
- docs/ (491 files) — Official CCCL documentation
  CI references, CMake guides, Python compute docs, libcudacxx PTX docs
- test/ (12 files) — Top-level integration tests (cuda_smoke, stdpar)
- Root configs: .clang-format, .clang-tidy, CONTRIBUTING.md, pyproject.toml
- CLAUDE.md symlink → AGENTS.md (NVIDIA's standard)

cccl_upstream now mirrors full NVIDIA/cccl structure:
  Before: 42M (cub + thrust + libcudacxx + cudax + c + examples + benchmarks)
  After:  53M (+python +ci +docs +.agent +test +configs)

This completes the CCCL base needed for:
- [muh-bench] items: ci/util/build_and_test_targets.sh for targeted builds
- [CCCL-verify] items: python/cuda_cccl/tests/ as reference implementations
- [CCCL-test] items: ci/test_cub.sh, ci/test_thrust.sh
- Agent workflow: .agent/skills/ for consistent style and test patterns
This commit is contained in:
muh-bot
2026-08-07 02:34:33 +00:00
parent 3f97dca7ad
commit 2a7ca101d7
908 changed files with 121615 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

310
cccl_upstream/docs/cccl/3.0_migration_guide.rst Normal file
View File

@@ -0,0 +1,310 @@
.. _cccl-3.0-migration-guide:
CCCL 2.x ‐ CCCL 3.0 migration guide
===================================
The CCCL team plans breaking changes carefully and only conducts them at major releases.
The CCCL 2.8 release came with many deprecations to prepare for the breaking changes conducted in CCCL 3.0.
This page summarizes the changes and helps migrating from CCCL 2.x to CCCL 3.0.
See also the `list of all deprecated APIs in CCCL 2.8 <https://github.com/NVIDIA/cccl/issues/3700>`_
and the `list of breaking changes in CCCL 3.0 <https://github.com/NVIDIA/cccl/issues/101>`_.
CUDA Toolkit changes
--------------------
CCCL is moving to its own include directory within the CUDA Toolkit. This may cause build failures and some initial confusion.
This section will have some suggestions and mitigations to help maintain builds across both CUDA12 and future releases.
The CTK-provided includes are changing in the following ways:
+-------------------------------+------------------------------------+
| **Before CUDA 13.0** | **After CUDA 13.0** |
+-------------------------------+------------------------------------+
| `${CTK_ROOT}/include/cuda/` | `${CTK_ROOT}/include/cccl/cuda/` |
+-------------------------------+------------------------------------+
| `${CTK_ROOT}/include/cub/` | `${CTK_ROOT}/include/cccl/cub/` |
+-------------------------------+------------------------------------+
| `${CTK_ROOT}/include/thrust/` | `${CTK_ROOT}/include/cccl/thrust/` |
+-------------------------------+------------------------------------+
Due to these changes, and the fact that NVCC by default includes its own directories, you may encounter errors when including
CCCL headers in source files that are compiled *only* by the host compiler.
For example, when compiling with GCC or MSVC alone, you may see ``<cuda/...>``, ``<cub/...>``, or ``<thrust/...>`` headers missing.
To mitigate this there are several solutions available depending on your build system:
- **DO NOT** prefix missing includes with ``<cccl/>`` -- This will break.
- CMake: link ``CCCL::CCCL`` to your target.
- Example: ``target_link_library(${MY_TARGET} PRIVATE CCCL::CCCL)``
- Non-CMake: Directly include the CUDA Toolkit's CCCL directory. (Make/Other)
- Example: Add CCCL as an include flag ``-I${CTK_ROOT}/include/cccl``
- Use a non-bundled CCCL. CCCL is available and maintained independently of the CTK.
- `See here for compatibility. <https://github.com/NVIDIA/cccl?tab=readme-ov-file#cuda-toolkit-ctk-compatibility>`_
Removed macros
--------------
* ``CUB_IS_INT128_ENABLED``: No replacement
* ``CUB_MAX(a, b)``: Use the ``cuda::std::max(a, b)`` function instead
* ``CUB_MIN(a, b)``: Use the ``cuda::std::min(a, b)`` function instead
* ``CUB_QUOTIENT_CEILING(a, b)``: Use ``cuda::ceil_div(a, b)`` instead
* ``CUB_QUOTIENT_FLOOR(a, b)``: Use plain integer division ``a / b`` instead
* ``CUB_ROUND_DOWN_NEAREST(a, b)``: Use ``cuda::round_down(a, b)`` instead
* ``CUB_ROUND_UP_NEAREST(a, b)``: Use ``cuda::round_up(a, b)`` instead
* ``CUB_RUNTIME_ENABLED``: No replacement
* ``CUB_USE_COOPERATIVE_GROUPS``: No replacement
* ``CUDA_CUB_RET_IF_FAIL``: No replacement
* ``[THRUST|CUB]_CLANG_VERSION``: No replacement
* ``[THRUST|CUB]_DEVICE_COMPILER*``: No replacement
* ``[THRUST|CUB]_GCC_VERSION``: No replacement
* ``[THRUST|CUB]_HOST_COMPILER*``: No replacement
* ``[THRUST|CUB]_INCLUDE_DEVICE_CODE``: No replacement
* ``[THRUST|CUB]_INCLUDE_HOST_CODE``: No replacement
* ``[THRUST|CUB]_IS_DEVICE_CODE``: No replacement
* ``[THRUST|CUB]_IS_HOST_CODE``: No replacement
* ``[THRUST|CUB]_MSVC_VERSION_FULL``: No replacement
* ``[THRUST|CUB]_MSVC_VERSION``: No replacement
* ``THRUST_CDP_DISPATCH``: No replacement (Support for CUDA Dynamic Parallelism V1 (CDPv1) has been removed, see below)
* ``THRUST_DECLTYPE_RETURNS_WITH_SFINAE_CONDITION``: No replacement
* ``THRUST_DECLTYPE_RETURNS``: No replacement
* ``THRUST_DEVICE_CODE``: No replacement
* ``THRUST_HOST_BACKEND``: Use ``THRUST_HOST_SYSTEM`` instead
* ``THRUST_INLINE_CONSTANT``: Use ``inline constexpr`` instead
* ``THRUST_INLINE_INTEGRAL_MEMBER_CONSTANT``: Use ``static constexpr`` instead
* ``THRUST_LEGACY_GCC``: No replacement
* ``THRUST_MODERN_GCC_REQUIRED_NO_ERROR``: No replacement
* ``THRUST_MODERN_GCC``: No replacement
* ``THRUST_MVCAP``: No replacement
* ``THRUST_NODISCARD``: Use ``[[nodiscard]]`` instead
* ``THRUST_RETOF1``: No replacement
* ``THRUST_RETOF2``: No replacement
* ``THRUST_RETOF``: No replacement
* ``THRUST_TUNING_ARCH``: No direct replacement. Use compiler-specific ``__CUDA_ARCH__`` (nvcc) or ``__NVCOMPILER_CUDA_ARCH__`` (nvc++) instead
Removed functions and classes
-----------------------------
* ``_ReadWriteBarrier`` and ``__thrust_compiler_fence``: Use ``cuda::atomic`` instead
* ``cub::*Kernel``: Any CUB kernel entrypoint is considered an implementation detail. No public exposure is provided.
* ``cub::Agent*``: CUB agents were considered implementation details and have all been moved to internal namespaces. No public exposure is provided.
* ``cub::AliasTemporaries``: No replacement
* ``cub::ArrayWrapper``: Use ``cuda::std::array`` instead
* ``cub::BAR``: No replacement
* ``cub::BaseTraits::CATEGORY``: Use the facilities from ``<cuda/std/type_traits>`` instead
* ``cub::BaseTraits::NULL_TYPE``: No replacement
* ``cub::BaseTraits::PRIMITIVE``: Use the facilities from ``<cuda/std/type_traits>`` instead
* ``cub::BFI``: Use ``cuda::bitfield_insert`` instead
* ``cub::BinaryOpHasIdxParam::HAS_PARAM``: Use ``cub::BinaryOpHasIdxParam::value`` instead
* ``cub::ConstantInputIterator``: Use ``thrust::constant_iterator`` instead
* ``cub::CountingInputIterator``: Use ``thrust::counting_iterator`` instead
* ``cub::CTA_SYNC_AND``: Use ``__syncthreads_and()`` instead
* ``cub::CTA_SYNC_OR``: Use ``__syncthreads_or()`` instead
* ``cub::CTA_SYNC``: Use ``__syncthreads()`` instead
* ``cub::Device*Policy``: Those policy hubs are considered implementation details. No public exposure is provided.
* ``cub::DeviceSpmv``: Use `cuSPARSE <https://docs.nvidia.com/cuda/cusparse>`_ instead
* ``cub::Difference``: Use ``cuda::std::minus`` instead
* ``cub::DivideAndRoundUp``: Use ``cuda::round_up`` instead
* ``cub::Division``: Use ``cuda::std::divides`` instead
* ``cub::Equality``: Use ``cuda::std::equal_to`` instead
* ``cub::FFMA_RZ``: No replacement
* ``cub::FMUL_RZ``: No replacement
* ``cub::FpLimits<T>``: Use ``cuda::std::numeric_limits<T>`` instead
* ``cub::GridBarrier``: Use the APIs from cooperative groups instead
* ``cub::GridBarrierLifetime``: Use the APIs from cooperative groups instead
* ``cub::IADD3``: No replacement
* ``cub::Inequality``: Use ``cuda::std::not_equal_to`` instead
* ``cub::Int2Type``: Use ``cuda::std::integral_constant`` instead
* ``cub::IterateThreadLoad``: No replacement
* ``cub::IterateThreadStore``: No replacement
* ``cub::KernelConfig``: No replacement
* ``cub::LaneId()``: Use ``cuda::ptx::get_sreg_laneid()`` instead
* ``cub::LaneMaskGe()``: Use ``cuda::ptx::get_sreg_lanemask_ge()`` instead
* ``cub::LaneMaskGt()``: Use ``cuda::ptx::get_sreg_lanemask_gt()`` instead
* ``cub::LaneMaskLe()``: Use ``cuda::ptx::get_sreg_lanemask_le()`` instead
* ``cub::LaneMaskLt()``: Use ``cuda::ptx::get_sreg_lanemask_lt()`` instead
* ``cub::MakePolicyWrapper``: No replacement
* ``cub::Max``: Use ``cuda::maximum`` instead
* ``cub::max``: Use ``cuda::std::max`` instead
* ``cub::MemBoundScaling``: No replacement
* ``cub::Min``: Use ``cuda::minimum`` instead
* ``cub::min``: Use ``cuda::std::min`` instead
* ``cub::Mutex``: Use ``std::mutex`` instead
* ``cub::PolicyWrapper``: No replacement
* ``cub::PRMT``: Use ``cuda::ptx::prmt()`` instead
* ``cub::RegBoundScaling``: No replacement
* ``cub::SHFL_IDX_SYNC``: Use ``__shfl_sync()`` instead
* ``cub::SHL_ADD``: No replacement
* ``cub::SHR_ADD``: No replacement
* ``cub::Sum``: Use ``cuda::std::plus`` instead
* ``cub::Swap(a, b)``: Use ``cuda::std::swap(a, b)`` instead
* ``cub::ThreadTrap()``: Use ``cuda::std::terminate()`` instead
* ``cub::TransformInputIterator``: Use ``thrust::transform_iterator`` instead
* ``cub::TripleChevronFactory``: No replacement for now, we are working on a new kernel launch facility
* ``cub::ValueCache``: No replacement
* ``cub::WARP_ALL``: Use ``__all_sync()`` instead
* ``cub::WARP_ANY``: Use ``__any_sync()`` instead
* ``cub::WARP_BALLOT``: Use ``__ballot_sync()`` instead
* ``cub::WARP_SYNC``: Use ``__syncwarp()`` instead
* ``cub::WarpId()``: Use ``cuda::ptx::get_sreg_warpid()`` instead
* ``thrust::*::[first_argument_type|second_argument_type|result_type]``: The nested aliases have been removed for all function object types: ``thrust::[plus|minus|multiplies|divides|modulus|negate|square|equal_to|not_equal_to|greater|less|greater_equal|less_equal|logical_and|logical_or|logical_not|bit_and|bit_or|bit_xor|identity|maximum|minimum|project1st|project2nd]``. No replacement.
* ``thrust::[unary|binary]_function``: No replacement. If you inherit from one of these types, just remove those base classes.
* ``thrust::[unary|binary]_traits``: No replacement.
* ``thrust::async::*``: No replacement for now. We are working on a C++26 senders implementation. For make a thrust algorithm skip syncing, use ``thrust::cuda::par_nosync`` as execution policy.
* ``thrust::bidirectional_universal_iterator_tag``: No replacement
* ``thrust::conjunction_value<Ts...>``: Use ``cuda::std::bool_constant<(Ts && ...)>`` instead
* ``thrust::conjunction_value_v<Ts...>``: Use a fold expression: ``Ts && ...`` instead
* ``thrust::cuda_cub::core::*``: Those are considered implementation details. No public exposure is provided.
* ``thrust::cuda_cub::counting_iterator_t``: Use ``thrust::counting_iterator`` instead
* ``thrust::cuda_cub::identity``: Use ``cuda::std::identity`` instead
* ``thrust::cuda_cub::launcher::triple_chevron``: No replacement for now, we are working on a new kernel launch facility
* ``thrust::cuda_cub::terminate``: Use ``cuda::std::terminate()`` instead
* ``thrust::cuda_cub::transform_input_iterator_t``: Use ``thrust::transform_iterator`` instead
* ``thrust::cuda_cub::transform_pair_of_input_iterators_t``: Use ``thrust::transform_iterator of a thrust::zip_iterator`` instead
* ``thrust::disjunction_value<Ts...>``: Use ``cuda::std::bool_constant<(Ts || ...)>`` instead
* ``thrust::disjunction_value_v<Ts...>``: Use a fold expression: ``Ts || ...`` instead
* ``thrust::forward_universal_iterator_tag``: No replacement
* ``thrust::identity<T>``: Use ``cuda::std::identity`` instead. If ``thrust::identity`` was used to perform a cast to ``T``, please define your own function object.
* ``thrust::input_universal_iterator_tag``: No replacement
* ``thrust::negation_value<T>``: Use ``cuda::std::bool_constant<!T>`` instead
* ``thrust::negation_value_v<T>``: Use a plain negation ``!T``
* ``thrust::not[1|2]``: Use ``cuda::std::not_fn`` instead
* ``thrust::null_type``: No replacement
* ``thrust::numeric_limits<T>``: Use ``cuda::std::numeric_limits<T>`` instead
* ``thrust::optional<T>``: Use ``cuda::std::optional<T>`` instead.
* ``thrust::output_universal_iterator_tag``: No replacement
* ``thrust::random_access_universal_iterator_tag``: No replacement
* ``thrust::remove_cvref[_t]``: Use ``cuda::std::remove_cvref[_t]`` instead
* ``thrust::void_t``: Use ``cuda::std::void_t`` instead
Deprecations with planned removal
---------------------------------
* ``CUB_LOG_SMEM_BANKS``: No replacement
* ``CUB_LOG_WARP_THREADS``: No replacement
* ``CUB_MAX_DEVICES``: No replacement
* ``CUB_PREFER_CONFLICT_OVER_PADDING``: No replacement
* ``CUB_PTX_LOG_SMEM_BANKS``: No replacement
* ``CUB_PTX_LOG_WARP_THREADS``: No replacement
* ``CUB_PTX_PREFER_CONFLICT_OVER_PADDING``: No replacement
* ``CUB_PTX_SMEM_BANKS``: No replacement
* ``CUB_PTX_SUBSCRIPTION_FACTOR``: No replacement
* ``CUB_PTX_WARP_THREADS``: No replacement
* ``CUB_SMEM_BANKS``: No replacement
* ``CUB_SUBSCRIPTION_FACTOR``: No replacement
* ``CUB_WARP_THREADS``: No replacement
* ``THRUST_FALSE``: No replacement
* ``THRUST_PREVENT_MACRO_SUBSTITUTION``: No replacement
* ``THRUST_STATIC_ASSERT(expr)``: Use ``static_assert(expr)`` instead
* ``THRUST_TRUE``: No replacement
* ``THRUST_UNKNOWN``: No replacement
* ``THRUST_UNUSED_VAR``: No replacement
* ``cub::BFE``: Use ``cuda::bitfield_extract`` instead
* ``cub::MergePathSearch``: No replacement
* ``cub::Traits<T>::Max()``: Use ``cuda::std::numeric_limits<T>::max()`` instead
* ``cub::Traits<T>::Min()``: Use ``cuda::std::numeric_limits<T>::min()`` instead
* ``thrust::iterator_difference[_t]<T>``: Use ``cuda::std::iterator_traits<T>::difference_type`` or ``cuda::std::iter_difference_t<T>`` instead
* ``thrust::iterator_pointer[_t]<T>``: Use ``cuda::std::iterator_traits<T>::pointer`` instead
* ``thrust::iterator_reference[_t]<T>``: Use ``cuda::std::iterator_traits<T>::reference`` or ``cuda::std::iter_reference_t<T>`` instead
* ``thrust::iterator_traits<T>``: Use ``cuda::std::iterator_traits<T>`` instead
* ``thrust::iterator_value[_t]<T>``: Use ``cuda::std::iterator_traits<T>::value_type`` or ``cuda::std::iter_value_t<T>`` instead
API breaks
----------
* ``cub::Block*``: All trailing ``int LEGACY_PTX_ARCH`` template parameters have been removed
* ``cub::CachingAllocator``: The constructor taking a trailing ``bool debug`` parameter has been removed
* ``cub::Device*``: All overloads with a trailing ``bool debug_synchronous`` parameter have been removed
* ``cub::Dispatch*``: All Boolean template parameters have been replaced by enumerations to increase readability
* ``cub::Dispatch*``: All policy hub template parameters have been moved to the back of the template parameters list
* ``cub::DispatchScan[ByKey]``: The offset type must be an unsigned type of at least 4-byte size
* ``cuda::ceil_div``: Now returns the common type of its arguments
* ``thrust::pair``: Is now an alias to ``cuda::std::pair`` and no longer a distinct type
* ``thrust::tabulate_output_iterator``: The ``value_type`` has been fixed to be ``void``
* ``thrust::transform_iterator``: Upon copying, will now always copy its contained function. If the contained function is neither copy constructible nor copy assignable, the iterator fails to compile when attempting to be copied.
* ``thrust::tuple``: Is now an alias to ``cuda::std::tuple`` and no longer a distinct type
* ``thrust::universal_host_pinned_memory_resource``: The alias has changed to a different memory resource, potentially changing pointer types derived from an allocator/container using this memory resource.
* The following Thrust function object types have been made aliases to the equally-named types in ``cuda::std::``: ``thrust::[plus|minus|multiplies|divides|modulus|negate|equal_to|not_equal_to|greater|less|greater_equal|less_equal|logical_and|logical_or|logical_not|bit_and|bit_or|bit_xor|identity|maximum|minimum]``. No replacement.
* ``CUB_DEFINE_DETECT_NESTED_TYPE``: The generated detector trait no longer provides a ``::VALUE`` member. Use ``::value`` instead.
Iterator traits
^^^^^^^^^^^^^^^
``cuda::std::iterator_traits`` will now correctly recognize user-provided specializations of ``std::iterator_traits``.
All of Thrust's iterator traits have been redefined in terms of ``cuda::std::iterator_traits``,
and users should prefer to use iterator traits from libcu++.
``thrust::iterator_traits`` can no longer be specialized.
Users should prefer to specialize ``cuda::std::iterator_traits`` instead of ``std::iterator_traits`` when necessary,
to make their iterators work equally in device code.
CUB Traits
^^^^^^^^^^
The functionality and internal use of ``cub::Traits`` has been minimized, because libcu++ provides better and standard alternatives.
Only the use in CUB's radix sort implementation for bit-twiddling remains.
Floating-point limits should be obtained using ``cuda::std::numeric_limits<T>`` instead of ``cub::FpLimits<T>``.
Classification of types should be done with the facilities from ``<cuda/std/type_traits>`` and ``<cuda/type_traits>``,
notably with ``cuda::std::is_signed[_v]``, ``cuda::std::is_integral[_v]``, etc.
There is an important difference for extended floating point types though:
Since ``cuda::std::is_floating_point[_v]`` will only recognize C++ standard floating point types,
``cuda::is_floating_point[_v]`` must be used to correctly classify extended floating point types like ``__half`` or ``__nv_bfloat16``.
``cub::BaseTraits`` and ``cub::Traits`` can no longer be specialized for custom types, and ``cub::FpLimits`` has been removed.
We acknowledge the need to provide user-defined floating point types though,
e.g., registering a custom half type with CUB to be used in radix sort.
Therefore, users can still specialize ``cub::NumericTraits`` for their custom floating point types,
inheriting from ``cub::BaseTraits`` and providing the necessary information for the type.
Additionally, the traits from libcu++ have to be specialized as well:
For example, a custom floating point type ``my_half`` could be registered with CUB and libcu++ like this:
.. code:: cpp
template <>
inline constexpr bool ::cuda::is_floating_point_v<my_half> = true;
template <>
class ::cuda::std::numeric_limits<my_half> {
public:
static constexpr bool is_specialized = true;
static __host__ __device__ my_half max() { return /* TODO */; }
static __host__ __device__ my_half min() { return /* TODO */; }
static __host__ __device__ my_half lowest() { return /* TODO */; }
};
template <>
struct CUB_NS_QUALIFIER::NumericTraits<my_half> : BaseTraits<FLOATING_POINT, true, uint16_t, my_half> {};
Behavioral changes
------------------
* ``cub::DeviceReduce::[Arg][Max|Min]``: Will now use ``cuda::std::numeric_limits<T>::[max|min]()`` instead of ``cub::Traits`` to determine the initial value
* ``cuda::std::mdspan``: The implementation was entirely rewritten and you may experience subtle behavioral changes
* ``thrust::transform_iterator``: The logic to determine the reference type has been reworked, especially wrt. to functions that return references to their own arguments (e.g., ``thrust::identity``).
* ``thrust::transform_iterator::difference_type``: The logic to select the difference type has been reworked. It's now either ``int`` or ``ptrdiff``.
ABI breaks
----------
* All of libcu++'s old ABI namespaces have been removed
Platform support
----------------
* At least C++17 is required
* At least clang 14 is required
* At least GCC 7 is required
* On Windows, at least Visual Studio 2019 is required (MSC_VER >= 1920)
* Intel ICC (``icpx``) is no longer supported
* At least CUDA Toolkit 12.0 is required
* Support for CUDA Dynamic Parallelism V1 (CDPv1) has been removed
* At least a GPU with compute capability 50 (Maxwell) is required

94
cccl_upstream/docs/cccl/config_macros.rst Normal file
View File

@@ -0,0 +1,94 @@
.. _cccl-config:
CCCL configuration macros
=========================
The CUDA Core Compute Libraries provide a set of macros to enable or disable specific features. These macros must be defined before any CCCL source file is included. The recommended way is to define them as the predefined compiler macros, for example:
.. code-block:: bash
nvcc -DCCCL_DISABLE_SOME_FEATURE src.cu
.. important::
These macros should be defined consistently in the whole project. Defining them only for some translation units may lead to unexpected compile time and runtime behaviour.
Assertion Control Macros
------------------------
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_ENABLE_ASSERTIONS | Enables assertions in both host and device code. Implied by compiling in debug mode. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_ENABLE_DEVICE_ASSERTIONS | Enables assertions in device code, independent of debug mode. Implied by ``CCCL_ENABLE_ASSERTIONS``. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_ENABLE_HOST_ASSERTIONS | Enables assertions in host code, independent of debug mode. Implied by ``CCCL_ENABLE_ASSERTIONS``. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
C++ Feature Control Macros
--------------------------
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_EXCEPTIONS | Disables throwing exceptions. Each ``throw`` is replaced by a call to ``cuda::std::terminate()``. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_RTTI | Disables use of runtime type information. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_IGNORE_MSVC_TRADITIONAL_PREPROCESSOR_WARNING | Disables diagnostics emitted when using MSVC's traditional preprocessor. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
CUDA Feature Control Macros
---------------------------
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_ARCH_DEPENDENT_NAMESPACE | Disables architecture dependent name mangling of kernels. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_CDP | Disables use of CUDA Dynamic Parallelism. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_CTK_COMPATIBILITY_CHECK | Disables the check whether NVCC's version matches the CUDA Toolkit version. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_DEVICE_RUNTIME | Disables use of CUDA device runtime APIs (``<cuda_device_runtime.h>``), thus makes some APIs that are ``__host__ __device__`` to be ``__host__`` only. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_LAUNCH_BOUNDS | Disables use of ``__launch_bounds__`` attribute. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_PDL | Disables use of Programmatic Dependent Launch. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
Deprecation Diagnostics Suppression Macros
------------------------------------------
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_IGNORE_DEPRECATED_API | Disables deprecated API diagnostics. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_IGNORE_DEPRECATED_COMPILER | Disables deprecated compiler diagnostics. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_IGNORE_DEPRECATED_CPP_DIALECT | Disables deprecated C++ dialect diagnostics. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_IGNORE_DEPRECATED_CUDA_BELOW_12 | Disables deprecated CUDA compiler diagnostics. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
Third Party Libraries Interoperability
--------------------------------------
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_DLPACK | Disables inclusion of DLPack header and APIs. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
Type Support
------------
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_BF16_SUPPORT | Disables use and library support for the ``__nv_bfloat16`` type. Also disables support for smaller NV floating point types. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_FLOAT128_SUPPORT | Disables use and library support for the ``__float128`` type. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_FP16_SUPPORT | Disables use and library support for the ``__half`` type. Also disables support for smaller NV floating point types. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_INT128_SUPPORT | Disables use and library support for the ``__int128`` type. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_LONG_DOUBLE_SUPPORT | Disables use and library support for the ``long double`` type. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_NVFP4_SUPPORT | Disables use and library support for the ``__nv_fp4_eNmM`` types. Also disables support for smaller NV floating point types. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_NVFP6_SUPPORT | Disables use and library support for the ``__nv_fp6_eNmM`` types. Also disables support for smaller NV floating point types. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_NVFP8_SUPPORT | Disables use and library support for the ``__nv_fp8_eNmM`` types. Also disables support for smaller NV floating point types. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_ENABLE_EXPERIMENTAL_HOST_ATOMICS_128B | Enables experimental support for 128b atomics in host code. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_GCC_HAS_EXTENDED_NUMERIC_LITERALS | Must be enabled in addition to passing ``-fext-numeric-literals`` with GCC to enable ``__float128`` support. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
.. note::
``<cuda_fpN.h>`` headers contain dependencies on other ``<cuda_fpM.h>`` headers, thus for example defining ``CCCL_DISABLE_BF16_SUPPORT`` will disable support for NVIDIA 8-bit, 6-bit and 4-bit floating point types, too.

14
cccl_upstream/docs/cccl/contributing.rst Normal file
View File

@@ -0,0 +1,14 @@
.. _cccl-contributing:
Contributing to the CUDA Core Compute Libraries
===============================================
.. toctree::
:maxdepth: 1
contributing/code_of_conduct
We welcome contributions - just send us a pull request!
You can find detailed instructions on `GitHub <https://github.com/NVIDIA/cccl/blob/main/CONTRIBUTING.md>`_.
libcu++ uses the `Apache License v2.0 with LLVM Exceptions <https://llvm.org/LICENSE.txt>`_.

105
cccl_upstream/docs/cccl/contributing/code_of_conduct.rst Normal file
View File

@@ -0,0 +1,105 @@
Code of Conduct
===============
Overview
--------
This document defines the Code of Conduct followed and enforced for
NVIDIA C++ Core Compute Libraries.
Intended Audience
~~~~~~~~~~~~~~~~~
- Community
- Developers
- Project Leads
Our Pledge
----------
In the interest of fostering an open and welcoming environment, we as
contributors and maintainers pledge to making participation in our
project and our community a harassment-free experience for everyone,
regardless of age, body size, disability, ethnicity, sex
characteristics, gender identity and expression, level of experience,
education, socio-economic status, nationality, personal appearance,
race, religion, or sexual identity and orientation.
Our Standards
-------------
Examples of behavior that contributes to creating a positive environment
include:
- Using welcoming and inclusive language.
- Being respectful of differing viewpoints and experiences.
- Gracefully accepting constructive criticism.
- Focusing on what is best for the community.
- Showing empathy towards other community members.
Examples of unacceptable behavior by participants include:
- The use of sexualized language or imagery and unwelcome sexual
attention or advances.
- Trolling, insulting/derogatory comments, and personal or political
attacks.
- Public or private harassment.
- Publishing others' private information, such as a physical or
electronic address, without explicit permission.
- Other conduct which could reasonably be considered inappropriate.
Our Responsibilities
--------------------
Project maintainers are responsible for clarifying the standards of
acceptable behavior and are expected to take appropriate and fair
corrective action in response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit,
or reject comments, commits, code, wiki edits, issues, and other
contributions that are not aligned to this Code of Conduct, or to ban
temporarily or permanently any contributor for other behaviors that they
deem inappropriate, threatening, offensive, or harmful.
Scope
-----
This Code of Conduct applies both within project spaces and in public
spaces when an individual is representing the project or its community.
Examples of representing a project or community include using an
official project email address, posting via an official social media
account, or acting as an appointed representative at an online or
offline event. Representation of a project may be further defined and
clarified by project maintainers.
Enforcement
-----------
Instances of abusive, harassing, or otherwise unacceptable behavior may
be reported by contacting cpp-conduct@nvidia.com. All complaints will be
reviewed and investigated and will result in a response that is deemed
necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an
incident. Further details of specific enforcement policies may be posted
separately.
Project maintainers who do not follow or enforce the Code of Conduct in
good faith may face temporary or permanent repercussions as determined
by other members of the project's leadership.
Attribution
-----------
This Code of Conduct was taken from the `NVIDIA
RAPIDS <https://docs.rapids.ai/resources/conduct/>`_ project, which was
adapted from the `Contributor Covenant version
1.4 <https://www.contributor-covenant.org/version/1/4/code-of-conduct.html>`_.
Please see this `FAQ <https://www.contributor-covenant.org/faq>`_ for
answers to common questions about this Code of Conduct.
Contact
-------
Please email cpp-conduct@nvidia.com for any Code of Conduct related
matters.

106
cccl_upstream/docs/cccl/determinism.rst Normal file
View File

@@ -0,0 +1,106 @@
.. _cccl-determinism:
Determinism
===========
Determinism describes whether an algorithm produces the *same result* every time it is run with the
same input. For many parallel algorithms this is not automatic. For reductions and scans, for example,
the order in which partial results are combined depends on how work is scheduled across thousands of
threads, and that schedule can change between launches or between GPUs. When the combining operator is
not perfectly associative — most notably floating-point addition, where ``(a + b) + c`` need not equal
``a + (b + c)`` — a different combining order yields a (slightly) different result, so the output is no
longer identical from one run to the next.
What counts as the "same result" is defined *per algorithm*. For reductions and scans it means a
*bitwise-identical* output. For other algorithms it can be weaker: a deterministic top-k, for example,
guarantees the same *set* of selected items, while the order of those items within the output is a
separate guarantee that an algorithm may expose on its own.
CCCL lets users state the determinism guarantee they need as an explicit *requirement* on an
algorithm, rather than relying on implementation-defined behavior. The library then either
selects an implementation that satisfies the requirement or rejects the call at compile time if the
requirement cannot be met for the given types and operator.
Determinism guarantees
----------------------
By *reproducible* we mean: given the same inputs, an algorithm returns the same output, in the sense
defined for that algorithm (see above). What the guarantees below differ in is the *scope* of that
reproducibility — across repeated runs, across hardware, or not at all. CCCL models three levels,
defined in ``cuda::execution::determinism``:
``not_guaranteed``
No reproducibility guarantee. The result is a valid answer, but it may differ from one invocation to
the next — even on the same GPU with the same input. This is usually the fastest option.
``run_to_run``
The result is reproducible across repeated runs *on the same GPU*, with the same input, build,
tuning, and launch configuration. It may still differ on a *different* GPU architecture.
``gpu_to_gpu``
The strongest guarantee: the result is reproducible across repeated runs *and across different GPU
architectures* — the same inputs yield the same bits whether the algorithm runs on, say, an Ampere or
a Hopper GPU. This is the most constrained option, is not available for every type/operator
combination, and is typically the slowest.
The guarantees are ordered from weakest to strongest:
``not_guaranteed`` ⊆ ``run_to_run`` ⊆ ``gpu_to_gpu``. A ``gpu_to_gpu`` result is also reproducible
run-to-run, and a ``run_to_run`` result is a valid (but stronger-than-required) answer wherever
``not_guaranteed`` would be accepted.
For types and operators that are exactly associative (see
:ref:`cuda::is_associative_v <libcudacxx-extended-api-functional-operator-properties>`; for example, integral
addition with well-known operators), every invocation is already reproducible across runs and GPUs, so the
stronger guarantees come for free and the library simply selects the fastest valid implementation.
.. warning::
``gpu_to_gpu``/``run_to_run`` reproducibility is guaranteed for a *fixed* CCCL and CUDA Toolkit version, not
across versions. If a policy selector is specified to change the used tuning, then reproducibility is only
guaranteed for identical tunings. The bitwise result may also change between CCCL or CUDA Toolkit releases as
algorithms, reduction structures, or tuning evolve.
Requesting a determinism guarantee
-----------------------------------
Determinism is expressed as a *requirement* and passed to an algorithm through its execution
environment using ``cuda::execution::require``:
.. code-block:: c++
#include <cuda/execution>
// Request run-to-run reproducibility for this call.
auto env = cuda::execution::require(cuda::execution::determinism::run_to_run);
The requirement may be combined with other environment properties — such as a stream or a memory
resource — into a single environment:
.. code-block:: c++
auto determinism = cuda::execution::require(cuda::execution::determinism::run_to_run);
auto env = cuda::std::execution::env{cuda::stream_ref{stream}, memory_resource, determinism};
Passing a determinism property *without* wrapping it in ``require`` is a compile-time error
(*"Determinism should be used inside requires to have an effect."*). ``require`` turns the property
into a *requirement*, which is what the algorithm honors — this prevents a stray determinism property
from being silently ignored.
If an algorithm cannot satisfy the requested guarantee for the given value type and operator, the call
fails to compile with a diagnostic explaining the constraint. If the guarantee can be satisfied by a
weaker-but-sufficient implementation (for example, an exactly-associative operator under
``gpu_to_gpu``), the library transparently selects it.
Where it is used
----------------
Determinism requirements are consumed today by several ``cub`` device algorithms. See the
:ref:`CUB determinism guide <cub-determinism>` for the per-algorithm support matrix, the exact
type/operator constraints, and some examples.
Further reading
---------------
- `Controlling Floating-Point Determinism in NVIDIA CCCL
<https://developer.nvidia.com/blog/controlling-floating-point-determinism-in-nvidia-cccl/>`_ — a
deeper walkthrough of the three guarantees and the implementation strategies behind them.

136
cccl_upstream/docs/cccl/development/build_and_bisect_tools.rst Normal file
View File

@@ -0,0 +1,136 @@
.. _build-and-bisect-tools:
Build and Bisect Utilities
==========================
``build_and_test_targets.sh``
-----------------------------
:file:`ci/util/build_and_test_targets.sh` configures, builds, and tests selected
CMake targets.
Options
~~~~~~~
- ``--preset <name>`` - choose a CMake preset.
- ``--cmake-options <str>`` - extra arguments for the preset configuration.
- ``--configure-override <cmd>`` - run a custom configuration command instead of
a preset. When used, ``--preset`` and ``--cmake-options`` are ignored.
- ``--build-targets <targets>`` - space separated Ninja targets. If omitted,
nothing builds.
- ``--ctest-targets <regex>`` - space separated CTest ``-R`` patterns. If
omitted, nothing runs.
- ``--lit-precompile-tests <paths>`` - space separated libcudacxx lit test paths
to precompile (no run). Paths are relative to ``libcudacxx/test/libcudacxx/``.
- ``--lit-tests <paths>`` - space separated libcudacxx lit test paths to execute.
Paths are relative to ``libcudacxx/test/libcudacxx/``.
- ``--custom-test-cmd <cmd>`` - arbitrary command executed after build/tests.
Combine with ``.devcontainer/launch.sh -d`` to reproduce CI commands inside a
container and choose a CUDA toolkit and host compiler:
``.devcontainer/launch.sh -d [--cuda <XX.Y>] [--host <compiler>] [--gpus all] -- <script>``
Examples
~~~~~~~~
Build a single CUB test locally::
ci/util/build_and_test_targets.sh \
--preset cub-cpp20 \
--build-targets "cub.cpp20.test.iterator"
Build the same test for SM90 using a CMake option::
ci/util/build_and_test_targets.sh \
--preset cub-cpp20 \
--cmake-options "-DCMAKE_CUDA_ARCHITECTURES=90" \
--build-targets "cub.cpp20.test.iterator"
Build the test for SM90 with a configure override::
ci/util/build_and_test_targets.sh \
--configure-override "ci/build_cub.sh -configure -arch 90" \
--build-targets "cub.cpp20.test.iterator"
Build **and run** a single CUB test locally::
ci/util/build_and_test_targets.sh \
--preset cub-cpp20 \
--build-targets "cub.cpp20.test.iterator" \
--ctest-targets "cub.cpp20.test.iterator"
Build and run a single CUB test in a devcontainer with specific CTK and host::
.devcontainer/launch.sh -d --cuda 12.3 --host gcc12 --gpus all -- \
ci/util/build_and_test_targets.sh \
--preset cub-cpp20 \
--build-targets "cub.cpp20.test.iterator" \
--ctest-targets "cub.cpp20.test.iterator"
Precompile the libcudacxx lit suite::
ci/util/build_and_test_targets.sh \
--preset libcudacxx \
--build-targets libcudacxx.test.lit.precompile
Precompile a single libcudacxx lit test (no execution)::
ci/util/build_and_test_targets.sh \
--preset libcudacxx \
--lit-precompile-tests \
"std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp"
Execute one or more libcudacxx lit tests::
ci/util/build_and_test_targets.sh \
--preset libcudacxx \
--lit-tests \
"std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp"
``git_bisect.sh``
-----------------
:file:`ci/util/git_bisect.sh` wraps ``git bisect`` around the build/test helper.
It accepts all ``build_and_test_targets.sh`` options plus:
- ``--good-ref <rev>`` - Optional; known good commit, tag, or branch. ``-Nd`` means
"N days ago." Defaults to the latest release version tag.
- ``--bad-ref <rev>`` - Optional; known bad commit. ``-Nd`` means "N days ago."
Defaults to ``origin/main``.
Examples
~~~~~~~~
Local CUB bisection from latest release to origin/main::
ci/util/git_bisect.sh \
--preset cub-cpp20 \
--build-targets "cub.cpp20.test.iterator" \
--ctest-targets "cub.cpp20.test.iterator"
Devcontainer CUB bisection from last week::
.devcontainer/launch.sh -d --cuda 12.3 --host gcc12 --gpus all -- \
ci/util/git_bisect.sh \
--preset cub-cpp20 \
--build-targets "cub.cpp20.test.iterator" \
--ctest-targets "cub.cpp20.test.iterator" \
--good-ref -7d
Compute-sanitizer example for regression introduced between 3-4 weeks ago::
.devcontainer/launch.sh -d --cuda 12.9 --host gcc13 --gpus all \
--env CCCL_TEST_MODE=compute-sanitizer-initcheck \
--env C2H_SEED_COUNT_OVERRIDE=1 \
-- ci/util/git_bisect.sh \
--preset "cub-cpp20" \
--build-targets "cub.cpp20.test.iterator" \
--ctest-targets "cub.cpp20.test.iterator" \
--good-ref -28d \
--bad-ref -21d
Workflow/Bisect
---------------
A ``Workflow/Bisect`` GitHub Actions job runs ``git_bisect.sh`` on a remote
runner. Launch it from **Actions → Git Bisect → Run workflow**. Provide
any desired runner label, refs, preset, targets, or launch arguments. The job
log streams bisect progress, and the run's **Summary** page renders the final
Markdown report with culprit commit, PR, reproduction steps, and more.

95
cccl_upstream/docs/cccl/development/debugger_setup.rst Normal file
View File

@@ -0,0 +1,95 @@
.. _cccl-development-module-debugger-setup:
=================
General Debugging
=================
Debugger Pretty Printers
========================
libcudacxx ships custom pretty printers for its types under
``libcudacxx/share/libcudacxx``. They render CCCL types in a readable form and, for
device-accessible data, copy the contents back to the host so the elements can be
inspected. Two independent implementations are provided:
- ``libcudacxx/share/libcudacxx/gdb`` - printers for GDB.
- ``libcudacxx/share/libcudacxx/lldb`` - printers for LLDB.
Each directory has an ``__init__.py`` entry point that registers every printer. The
repository root contains a ``.gdbinit`` and a ``.lldbinit`` that load the matching entry
point for you, so the simplest way to enable the printers is to let the debugger pick up
these files.
.. important::
``lldb`` and ``gdb`` only inspect the following locations when looking for init
dotfiles (in the given order):
#. Home directory config files (usually ``~/.lldbinit`` or ``~/.config/gdb/gdbinit`` on
Linux, but check the respective manuals for specifics).
#. The current working directory.
They do **not** walk up the directory stack like most tools. So if you have a
``.lldbinit`` in the parent directory, ``lldb`` will **not** load it. For this reason,
you **must** run the debugger from the root CCCL directory in order for automatic
loading of the pretty printers to work.
In addition to not loading parent directory dot-files, ``gdb`` or ``lldb`` will load not
dotfiles unless you explicitly allow them. The following sections explain how to enable
this for each debugger.
.. note::
The following is **not** needed when working inside a devcontainer. devcontainers
already have the following set up.
If they don't, and automatic loading of the pretty printers does not work, then this is
a bug and should be fixed.
It is only needed for bare metal builds.
GDB
---
By default GDB does not source a ``.gdbinit`` from the current directory, and it guards
auto-loaded scripts with the ``auto-load safe-path`` setting. Add the repository root to
your ``~/.gdbinit`` (or ``~/.config/gdb/gdbinit`` if you have ``XDG_CONFIG_HOME`` set) so
the project's ``.gdbinit`` is trusted and loaded::
add-auto-load-safe-path /absolute/path/to/cccl
set auto-load local-gdbinit on
Launch GDB from the repository root and the printers should register automatically.
Verify that the printers are active with ``info pretty-printer``.
To load the printers without depending on the working directory - for example
from a global ``~/.gdbinit`` - ``source`` the entry point by absolute path
instead::
source /absolute/path/to/cccl/libcudacxx/share/libcudacxx/gdb/__init__.py
``source`` runs the script directly and is not subject to the ``auto-load safe-path``
restriction.
LLDB
----
LLDB only reads ``.lldbinit`` from your home directory unless you opt in to loading one
from the current working directory. Enable that once in your ``~/.lldbinit`` (``lldb``
seemingly does not respect ``XDG_CONFIG_HOME``)::
settings set target.load-cwd-lldbinit true
This is a trust decision, since the local file runs arbitrary Python. Launch LLDB from the
repository root and the project's ``.lldbinit`` imports the formatters automatically.
To load the formatters without depending on the working directory, add the absolute path
to your ``~/.lldbinit`` instead::
command script import "/absolute/path/to/cccl/libcudacxx/share/libcudacxx/lldb/__init__.py"
The entry point's ``__lldb_init_module`` hook defines and enables an LLDB type category
for the formatters. Print any CCCL value with the usual commands (``v``, ``frame
variable``, or ``dwim-print``).

23
cccl_upstream/docs/cccl/development/index.rst Normal file
View File

@@ -0,0 +1,23 @@
.. _cccl-development-module:
CCCL Development Guide
======================
.. toctree::
:hidden:
:maxdepth: 1
macro
testing
build_and_bisect_tools
visibility
debugger_setup
This living document serves to describe the internal details and the development process of CCCL libraries.
Documentation:
- :doc:`CCCL Internal Macros <macro>`
- :doc:`CCCL Testing Utilities <testing>`
- :doc:`CCCL Bisect And Targeted Build/Test Helpers <build_and_bisect_tools>`
- :doc:`General Debugging <debugger_setup>`

553
cccl_upstream/docs/cccl/development/macro.rst Normal file
View File

@@ -0,0 +1,553 @@
.. _cccl-development-module-macros:
CCCL Internal Macros
====================
The document describes the main *internal* macros used by CCCL. They are not intended to be used by end users, but for development of CCCL features only. We reserve the right to change them at any time without warning.
----
Compiler Macros
---------------
**Host compiler macros**:
+------------------------------+---------------------------------------------+
| ``_CCCL_COMPILER(CLANG)`` | Clang |
+------------------------------+---------------------------------------------+
| ``_CCCL_COMPILER(GCC)`` | GCC |
+------------------------------+---------------------------------------------+
| ``_CCCL_COMPILER(NVHPC)`` | Nvidia HPC compiler |
+------------------------------+---------------------------------------------+
| ``_CCCL_COMPILER(MSVC)`` | Microsoft Visual Studio |
+------------------------------+---------------------------------------------+
| ``_CCCL_COMPILER(MSVC2019)`` | Microsoft Visual Studio 2019 |
+------------------------------+---------------------------------------------+
| ``_CCCL_COMPILER(MSVC2022)`` | Microsoft Visual Studio 2022 |
+------------------------------+---------------------------------------------+
The ``_CCCL_COMPILER`` function-like macro can also be used to check the version of a compiler.
.. code:: cpp
_CCCL_COMPILER(MSVC, <, 19, 24)
_CCCL_COMPILER(GCC, >=, 9)
*Note*: When used without specifying a minor version number, the macro will only test against
the compiler's major version number. For example, when the compiler is ``gcc-9.1``, the macro
``_CCCL_COMPILER(GCC, >, 9)`` will be ``false`` even though ``9.1`` is greater than ``9``.
**CUDA compiler macros**:
+--------------------------------+-------------------------+
| ``_CCCL_CUDA_COMPILER(NVCC)`` | Nvidia compiler |
+--------------------------------+-------------------------+
| ``_CCCL_CUDA_COMPILER(NVHPC)`` | Nvidia HPC compiler |
+--------------------------------+-------------------------+
| ``_CCCL_CUDA_COMPILER(NVRTC)`` | Nvidia Runtime Compiler |
+--------------------------------+-------------------------+
| ``_CCCL_CUDA_COMPILER(CLANG)`` | Clang |
+--------------------------------+-------------------------+
The ``_CCCL_CUDA_COMPILER`` function-like macro can also be used to check the version of a CUDA compiler.
.. code:: cpp
_CCCL_CUDA_COMPILER(NVCC, <, 12, 3)
_CCCL_CUDA_COMPILER(CLANG, >=, 14)
*Note*: ``_CCCL_CUDA_COMPILER(...)`` check may result in a ``true`` value even during the compilation of a C++ source
file. Use ``_CCCL_CUDA_COMPILATION()`` to check for the compilation of a CUDA source file.
**CUDA identification/version macros**:
+----------------------------------+------------------------------------------------------------------------------------------------+
| ``_CCCL_CUDA_COMPILATION()`` | CUDA code is being compiled |
+----------------------------------+------------------------------------------------------------------------------------------------+
| ``_CCCL_HOST_COMPILATION()`` | Compiling host code, ``true`` when executing the CUDA host pass or compiling a C++ source file |
+----------------------------------+------------------------------------------------------------------------------------------------+
| ``_CCCL_DEVICE_COMPILATION()`` | Compiling device code, ``true`` when executing the CUDA device pass |
+----------------------------------+------------------------------------------------------------------------------------------------+
| ``_CCCL_CUDACC_BELOW(12, 7)`` | CUDA version below 12.7 when compiling a CUDA source file |
+----------------------------------+------------------------------------------------------------------------------------------------+
| ``_CCCL_CUDACC_AT_LEAST(12, 7)`` | CUDA version at least 12.7 when compiling a CUDA source file |
+----------------------------------+------------------------------------------------------------------------------------------------+
*Note*: When compiling CUDA code with ``nvc++`` both ``_CCCL_HOST_COMPILATION()`` and ``_CCCL_DEVICE_COMPILATION()`` result in a ``true`` value.
**PTX macros**:
+----------------------+-------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_PTX_ARCH()`` | Alias of ``__CUDA_ARCH__`` with value equal to 0 if a CUDA compiler is not available |
+----------------------+-------------------------------------------------------------------------------------------------------------------+
| ``__cccl_ptx_isa`` | PTX ISA version available with the current CUDA compiler, e.g. PTX ISA 8.4 (``840``) is available from CUDA 12.4 |
+----------------------+-------------------------------------------------------------------------------------------------------------------+
*Note*: When compiling CUDA code with ``nvc++`` the ``_CCCL_PTX_ARCH()`` macro expands to ``0``.
----
Architecture Macros
-------------------
The following macros are used to check the target architecture. They comply with the compiler supported by the CUDA toolkit. Compilers outside the CUDA toolkit may define such macros in a different way.
+------------------------------+---------------------------------------------------+
| ``_CCCL_HOST_ARCH(ARM64)`` | ARM 64-bit, including MSVC emulation |
+------------------------------+---------------------------------------------------+
| ``_CCCL_HOST_ARCH(X86_64)`` | X86 64-bit. False on ARM 64-bit MSVC emulation |
+------------------------------+---------------------------------------------------+
----
OS Macros
---------
+-----------------------+---------------------------------+
| ``_CCCL_OS(WINDOWS)`` | Windows, including NVRTC LLP64 |
+-----------------------+---------------------------------+
| ``_CCCL_OS(LINUX)`` | Linux, including NVRTC LP64 |
+-----------------------+---------------------------------+
| ``_CCCL_OS(ANDROID)`` | Android |
+-----------------------+---------------------------------+
| ``_CCCL_OS(QNX)`` | QNX |
+-----------------------+---------------------------------+
----
Execution Space
---------------
**Functions**
+-----------------------+-----------------------+
| ``_CCCL_HOST`` | Host function |
+-----------------------+-----------------------+
| ``_CCCL_DEVICE`` | Device function |
+-----------------------+-----------------------+
| ``_CCCL_HOST_DEVICE`` | Host/Device function |
+-----------------------+-----------------------+
In addition, ``_CCCL_EXEC_CHECK_DISABLE`` disables the execution space check for the NVHPC compiler
**Target Macros**
+---------------------------------------------------------------------------------+--------------------------------------------------------------------------+
| ``NV_IF_TARGET(TARGET, (CODE))`` | Enable ``CODE`` only if ``TARGET`` is satisfied. |
+---------------------------------------------------------------------------------+--------------------------------------------------------------------------+
| ``NV_IF_ELSE_TARGET(TARGET, (IF_CODE), (ELSE_CODE))`` | Enable ``CODE_IF`` if ``TARGET`` is satisfied, ``CODE_ELSE`` otherwise. |
+---------------------------------------------------------------------------------+--------------------------------------------------------------------------+
| ``NV_DISPATCH_TARGET(TARGET1, (TARGET1_CODE), ..., TARGET_N, (TARGET_N_CODE))`` | Enable a single code block if any of ``TARGET_i`` is satisfied. |
+---------------------------------------------------------------------------------+--------------------------------------------------------------------------+
Possible ``TARGET`` values:
+---------------------------+-------------------------------------------------------------------+
| ``NV_ANY_TARGET`` | Any target |
+---------------------------+-------------------------------------------------------------------+
| ``NV_IS_HOST`` | Host-code target |
+---------------------------+-------------------------------------------------------------------+
| ``NV_IS_DEVICE`` | Device-code target |
+---------------------------+-------------------------------------------------------------------+
| ``NV_PROVIDES_SM_<VER>`` | SM architecture is at least ``VER``, e.g. ``NV_PROVIDES_SM_80`` |
+---------------------------+-------------------------------------------------------------------+
| ``NV_IS_EXACTLY_SM_<NN>`` | SM architecture is exactly ``VER``, e.g. ``NV_IS_EXACTLY_SM_80`` |
+---------------------------+-------------------------------------------------------------------+
Usage example:
.. code-block:: c++
NV_IF_TARGET(NV_IS_DEVICE, (auto x = threadIdx.x; return x;));
NV_IF_ELSE_TARGET(NV_IS_HOST, (return 0;), (auto x = threadIdx.x; return x;));
NV_DISPATCH_TARGET(NV_PROVIDES_SM_90, (return "Hopper+";),
NV_IS_EXACTLY_SM_75, (return "Turing";),
NV_IS_HOST, (return "Host";))
*Pitfalls*:
* All target macros generate the code in a local scope, i.e. ``{ code }``.
* ``NV_DISPATCH_TARGET`` is *NOT* a switch statement. It enables the code associated with the first condition satisfied.
* The target macros take ``code`` as an argument, so it is *not* possible to use any conditional compilation, .e.g ``#if _CCCL_STD_VER >= 20`` within a target macro
----
CUDA attributes
---------------
+------------------------------+----------------------------------------------------------+
| ``_CCCL_GRID_CONSTANT`` | Grid constant kernel parameter |
+------------------------------+----------------------------------------------------------+
| ``_CCCL_GLOBAL_CONSTANT`` | Host/device global scope constant (``inline constexpr``) |
+------------------------------+----------------------------------------------------------+
----
CUDA Toolkit macros
-------------------
+-------------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_HAS_CTK()`` | CUDA toolkit is available if ``_CCCL_CUDA_COMPILER()`` evaluates to a ``true`` value or if ``cuda_runtime_api.h`` was found |
+-------------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_CTK_BELOW`` | CUDA toolkit version below 12.7 |
+-------------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_CTK_AT_LEAST(12, 7)`` | CUDA toolkit version at least 12.7 |
+-------------------------------+-----------------------------------------------------------------------------------------------------------------------------+
Non-standard Types Support
--------------------------
+------------------------------+-------------------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_HAS_INT128()`` | ``__int128`` and ``__uint128_t`` for 128-bit integer are supported and enabled |
+------------------------------+-------------------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_HAS_NVFP8()`` | ``__nv_fp8_e5m2/__nv_fp8_e4m3/__nv_fp8_e8m0`` data types are supported and enabled. Prefer over ``__CUDA_FP8_TYPES_EXIST__`` |
+------------------------------+-------------------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_HAS_NVFP16()`` | ``__half/__half2`` data types are supported and enabled. Prefer over ``__CUDA_FP16_TYPES_EXIST__`` |
+------------------------------+-------------------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_HAS_NVBF16()`` | ``__nv_bfloat16/__nv_bfloat162`` data types are supported and enabled. Prefer over ``__CUDA_BF16_TYPES_EXIST__`` |
+------------------------------+-------------------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_HAS_FLOAT128()`` | ``__float128`` for 128-bit floating-point are supported and enabled |
+------------------------------+-------------------------------------------------------------------------------------------------------------------------------+
+-----------------------------------+-------------------------------------------------------------------------+
| ``CCCL_DISABLE_INT128_SUPPORT`` | Disable ``__int128/__uint128_t`` support |
+-----------------------------------+-------------------------------------------------------------------------+
| ``CCCL_DISABLE_NVFP8_SUPPORT`` | Disable ``__nv_fp8_e5m2/__nv_fp8_e4m3/__nv_fp8_e8m0`` support |
+-----------------------------------+-------------------------------------------------------------------------+
| ``CCCL_DISABLE_NVFP16_SUPPORT`` | Disable ``__half/__half2`` support |
+-----------------------------------+-------------------------------------------------------------------------+
| ``CCCL_DISABLE_NVBF16_SUPPORT`` | Disable ``__nv_bfloat16/__nv_bfloat162`` support |
+-----------------------------------+-------------------------------------------------------------------------+
| ``CCCL_DISABLE_FLOAT128_SUPPORT`` | Disable ``__float128`` support |
+-----------------------------------+-------------------------------------------------------------------------+
+-----------------------------------+-------------------------------------------------------------------------+
| ``_LIBCUDACXX_HAS_NVFP16()`` | ``__half/__half2`` host/device are supported (CUDA 12.2+) |
+-----------------------------------+-------------------------------------------------------------------------+
| ``_LIBCUDACXX_HAS_NVBF16()`` | ``__nv_bfloat16/__nv_bfloat162`` host/device are supported (CUDA 12.2+) |
+-----------------------------------+-------------------------------------------------------------------------+
----
C++ Language Macros
-------------------
The following macros are required only if the target C++ version does not support the corresponding attribute
+-----------------------------+----------------------------------------------------------+
| ``_CCCL_STD_VER`` | C++ standard version, e.g. ``#if _CCCL_STD_VER >= 2017`` |
+-----------------------------+----------------------------------------------------------+
| ``_CCCL_CONSTEXPR_CXX20`` | Enable ``constexpr`` for C++20 or newer |
+-----------------------------+----------------------------------------------------------+
| ``_CCCL_CONSTEXPR_CXX23`` | Enable ``constexpr`` for C++23 or newer |
+-----------------------------+----------------------------------------------------------+
| ``_CCCL_HAS_EXCEPTIONS()`` | Features can use exceptions, e.g ``bad_optional_access`` |
+-----------------------------+----------------------------------------------------------+
**Concept-like Macros**:
+------------------------+--------------------------------------------------------------------------------------------+
| ``_CCCL_TEMPLATE(X)`` | ``template`` clause |
+------------------------+--------------------------------------------------------------------------------------------+
| ``_CCCL_REQUIRES(X)`` | ``requires`` clause |
+------------------------+--------------------------------------------------------------------------------------------+
| ``_CCCL_AND`` | Traits conjunction only used with ``_CCCL_REQUIRES`` |
+------------------------+--------------------------------------------------------------------------------------------+
Usage example:
.. code-block:: c++
_CCCL_TEMPLATE(typename T)
_CCCL_REQUIRES(is_integral_v<T> _CCCL_AND(sizeof(T) > 1))
.. code-block:: c++
_CCCL_TEMPLATE(typename T)
_CCCL_REQUIRES(is_arithmetic_v<T> _CCCL_AND (!is_integral_v<T>))
**Portable feature testing**:
+--------------------------+--------------------------------------------------+
| ``_CCCL_HAS_BUILTIN(X)`` | Portable ``__has_builtin(X)`` |
+--------------------------+--------------------------------------------------+
| ``_CCCL_HAS_FEATURE(X)`` | Portable ``__has_feature(X)`` |
+--------------------------+--------------------------------------------------+
**Portable attributes**:
+----------------------------------+------------------------------------------------------------------------------+
| ``_CCCL_ASSUME(EXPR)`` | Portable ``[[assume]]`` attribute (before C++23) |
+----------------------------------+------------------------------------------------------------------------------+
| ``_CCCL_NO_UNIQUE_ADDRESS`` | Portable ``[[no_unique_address]]`` attribute |
+----------------------------------+------------------------------------------------------------------------------+
| ``CCCL_DEPRECATED`` | Portable ``[[deprecated]]`` attribute (before C++14) |
+----------------------------------+------------------------------------------------------------------------------+
| ``CCCL_DEPRECATED_BECAUSE(MSG)`` | Portable ``[[deprecated]]`` attribute with custom message (before C++14) |
+----------------------------------+------------------------------------------------------------------------------+
| ``_CCCL_FORCEINLINE`` | Portable "always inline" attribute |
+----------------------------------+------------------------------------------------------------------------------+
| ``_CCCL_PURE`` | Portable "pure" function attribute |
+----------------------------------+------------------------------------------------------------------------------+
| ``_CCCL_CONST`` | Portable "constant" function attribute |
+----------------------------------+------------------------------------------------------------------------------+
| ``_CCCL_LIFETIMEBOUND`` | Portable "lifetime bound" function attribute |
+----------------------------------+------------------------------------------------------------------------------+
**Portable Builtin Macros**:
+---------------------------------------+--------------------------------------------+
| ``_CCCL_UNREACHABLE()`` | Portable ``__builtin_unreachable()`` |
+---------------------------------------+--------------------------------------------+
| ``_CCCL_BUILTIN_EXPECT(X)`` | Portable ``__builtin_expected(X)`` |
+---------------------------------------+--------------------------------------------+
| ``_CCCL_BUILTIN_PREFETCH(X[, Y, Z])`` | Portable ``__builtin_prefetch(X, Y, Z)`` |
+---------------------------------------+--------------------------------------------+
**Portable Keyword Macros**
+-----------------------------+--------------------------------------------+
| ``_CCCL_RESTRICT`` | Portable ``restrict`` keyword |
+-----------------------------+--------------------------------------------+
| ``_CCCL_ALIGNAS(X)`` | Portable ``alignas(X)`` keyword (variable) |
+-----------------------------+--------------------------------------------+
| ``_CCCL_ALIGNAS_TYPE(X)`` | Portable ``alignas(X)`` keyword (type) |
+-----------------------------+--------------------------------------------+
| ``_CCCL_PRAGMA(X)`` | Portable ``_Pragma(X)`` keyword |
+-----------------------------+--------------------------------------------+
**Portable Pragma Macros**
+--------------------------------+-------------------------------------------+
| ``_CCCL_PRAGMA_UNROLL(N)`` | Portable ``#pragma unroll N`` pragma |
+--------------------------------+-------------------------------------------+
| ``_CCCL_PRAGMA_UNROLL_FULL()`` | Portable ``#pragma unroll`` pragma |
+--------------------------------+-------------------------------------------+
| ``_CCCL_PRAGMA_NOUNROLL()`` | Portable ``#pragma nounroll`` pragma |
+--------------------------------+-------------------------------------------+
**Conditional Constant Evaluation Macros**
In C++23, the ``if consteval`` statement (`link <https://en.cppreference.com/w/cpp/language/if.html>`_) was introduced. CCCL mimics the behaviour with a set of macros that expand to an implementation supported by the compiler. If the compiler doesn't support any kind of conditional constant evaluation, the macros expand to predefined fallback values.
+------------------------------------+-----------------------------------------------------------------------------------+
| ``_CCCL_IF_CONSTEVAL`` | Equivalent to ``if consteval`` statement (fallbacks to ``if constexpr (false)``) |
+------------------------------------+-----------------------------------------------------------------------------------+
| ``_CCCL_IF_CONSTEVAL_DEFAULT`` | Equivalent to ``if consteval`` statement (fallbacks to ``if constexpr (true)``) |
+------------------------------------+-----------------------------------------------------------------------------------+
| ``_CCCL_IF_NOT_CONSTEVAL`` | Equivalent to ``if !consteval`` statement (fallbacks to ``if constexpr (true)``) |
+------------------------------------+-----------------------------------------------------------------------------------+
| ``_CCCL_IF_NOT_CONSTEVAL_DEFAULT`` | Equivalent to ``if !consteval`` statement (fallbacks to ``if constexpr (false)``) |
+------------------------------------+-----------------------------------------------------------------------------------+
**Exception Macros**
CUDA doesn't support exceptions in device code, however, sometimes we need to write host/device functions that use exceptions on host and ``__trap()`` on device. CCCL provides a set of macros that should be used in place of the standard C++ keywords to make the code compile in both, host and device code.
+-----------------------------+------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_TRY`` | Replacement for the ``try`` keyword. |
+-----------------------------+------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_CATCH (X)`` | Replacement for the ``catch (/*X*/)`` statement. |
+-----------------------------+------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_CATCH_ALL`` | Replacement for the ``catch (...)`` statement. |
+-----------------------------+------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_THROW(X, ...)`` | Replacement for the ``throw X(...)`` expression. ``X`` must be fully qualified type, without the leading ``::``. |
+-----------------------------+------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_RETHROW`` | Replacement for the plain ``throw`` expression. |
+-----------------------------+------------------------------------------------------------------------------------------------------------------+
*Note*: The ``_CCCL_CATCH`` clause must always introduce a named variable, like: ``_CCCL_CATCH(const exception_type& var)``.
.. note::
``_CCCL_THROW`` requires to include the ``<stdexcept>`` header, regardless exceptions are enabled or not.
Example:
.. code-block:: c++
__host__ __device__ void* alloc(cuda::std::size_t nbytes)
{
if (void* ptr = cuda::std::malloc(nbytes))
{
return ptr;
}
_CCCL_THROW(std::bad_alloc); // on device calls cuda::std::terminate()
}
__host__ __device__ void do_something(int* buff)
{
_CCCL_THROW(std::runtime_error, "Something went wrong"); // on device calls cuda::std::terminate()
}
__host__ __device__ void fn(cuda::std::size_t n)
{
int* buff{};
_CCCL_TRY
{
buff = reinterpret_cast<int*>(alloc(n * sizeof(int)));
do_something(buff);
}
_CCCL_CATCH ([[maybe_unused]] const std::bad_alloc& e) // must be always named
{
std::fprintf(stderr, "Failed to allocate memory\n"); // We can directly call host-only functions
cuda::std::terminate();
}
_CCCL_CATCH_ALL // or _CCCL_CATCH_FALLTHOUGH
{
cuda::std::free(buff);
_CCCL_RETHROW;
}
}
__global__ void kernel()
{
fn(10);
}
int main()
{
fn(10);
return 0;
}
----
Visibility Macros
-----------------
+-------------------------------+-----------------------------------------------------------------------------------------------------+
| ``_CCCL_VISIBILITY_HIDDEN`` | Hidden visibility attribute (e.g. ``__attribute__((visibility("hidden")))``) |
+-------------------------------+-----------------------------------------------------------------------------------------------------+
| ``_CCCL_HIDE_FROM_ABI`` | Hidden visibility (i.e. ``inline``, not exported, not instantiated) |
+-------------------------------+-----------------------------------------------------------------------------------------------------+
| ``_CCCL_API`` | Host/device function with hidden visibility. Most CCCL functions are hidden with this attribute |
+-------------------------------+-----------------------------------------------------------------------------------------------------+
| ``_CCCL_HOST_API`` | Host function with hidden visibility. Most CCCL functions are hidden with this attribute |
+-------------------------------+-----------------------------------------------------------------------------------------------------+
| ``_CCCL_DEVICE_API`` | Device function with hidden visibility. Most CCCL functions are hidden with this attribute |
+-------------------------------+-----------------------------------------------------------------------------------------------------+
| ``_CCCL_KERNEL_ATTRIBUTES`` | Global function with hidden visibility. Most CCCL functions are hidden with this attribute |
+-------------------------------+-----------------------------------------------------------------------------------------------------+
----
Other Common Macros
-------------------
+-----------------------------+--------------------------------------------+
| ``_CCCL_TO_STRING(X)`` | ``X`` to literal string |
+-----------------------------+--------------------------------------------+
| ``_CCCL_DOXYGEN_INVOKED`` | Defined during Doxygen parsing |
+-----------------------------+--------------------------------------------+
----
Debugging Macros
----------------
+-----------------------------------+-------------------------------------------------------------------------------------------------------------+
| ``_CCCL_ASSERT(COND, MSG)`` | Portable, conditional CCCL `assert()` macro. Requires (``CCCL_ENABLE_ASSERTIONS`` or a debug build) |
+-----------------------------------+-------------------------------------------------------------------------------------------------------------+
| ``_CCCL_VERIFY(COND, MSG)`` | Portable, always-on `assert()` reserved for critical checks that are always required |
+-----------------------------------+-------------------------------------------------------------------------------------------------------------+
| ``_CCCL_ENABLE_ASSERTIONS`` | Enable assertions |
+-----------------------------------+-------------------------------------------------------------------------------------------------------------+
| ``CCCL_ENABLE_HOST_ASSERTIONS`` | Enable host-side assertions |
+-----------------------------------+-------------------------------------------------------------------------------------------------------------+
| ``CCCL_ENABLE_DEVICE_ASSERTIONS`` | Enable device-side assertions |
+-----------------------------------+-------------------------------------------------------------------------------------------------------------+
| ``_CCCL_ENABLE_DEBUG_MODE`` | Enable debug mode (and assertions) |
+-----------------------------------+-------------------------------------------------------------------------------------------------------------+
----
Warning Suppression Macros
--------------------------
+-----------------------------+--------------------------------------------+
| ``_CCCL_DIAG_PUSH`` | Portable ``#pragma push`` |
+-----------------------------+--------------------------------------------+
| ``_CCCL_DIAG_POP`` | Portable ``#pragma pop`` |
+-----------------------------+--------------------------------------------+
**Compiler-specific Suppression Macros**:
+-------------------------------------+-------------------------------------------------------------+
| ``_CCCL_DIAG_SUPPRESS_CLANG(X)`` | Suppress clang warning, e.g. ``"-Wattributes"`` |
+-------------------------------------+-------------------------------------------------------------+
| ``_CCCL_DIAG_SUPPRESS_GCC(X)`` | Suppress gcc warning, e.g. ``"-Wattributes"`` |
+-------------------------------------+-------------------------------------------------------------+
| ``_CCCL_DIAG_SUPPRESS_NVHPC(X)`` | Suppress nvhpc warning, e.g. ``expr_has_no_effect`` |
+-------------------------------------+-------------------------------------------------------------+
| ``_CCCL_DIAG_SUPPRESS_MSVC(X)`` | Suppress msvc warning, e.g. ``4127`` |
+-------------------------------------+-------------------------------------------------------------+
| ``_CCCL_BEGIN_NV_DIAG_SUPPRESS(X)`` | Start to suppress nvcc warning, e.g. ``177`` |
+-------------------------------------+-------------------------------------------------------------+
| ``_CCCL_END_NV_DIAG_SUPPRESS()`` | End to suppress nvcc warning |
+-------------------------------------+-------------------------------------------------------------+
Usage example:
.. code-block:: c++
_CCCL_DIAG_PUSH
_CCCL_DIAG_SUPPRESS_GCC("-Wattributes")
// code ..
_CCCL_DIAG_POP
----
Freestanding support
--------------------------
We - partially - support building CCCL headers in freestanding mode, for example JIT compilation with NVRTC.
+-----------------------------+----------------------------------------------------+
| ``_CCCL_HOSTED()`` | "Normal" compilation mode with host STL support |
+-----------------------------+----------------------------------------------------+
| ``_CCCL_FREESTANDING()`` | Freestanding compilation mode, no host STL support |
+-----------------------------+----------------------------------------------------+
| ``_CCCL_HOSTJIT()`` | Freestanding compilation mode, with host compiler |
+-----------------------------+----------------------------------------------------+
Usage example:
.. code-block:: c++
#if _CCCL_HOSTED()
# include <iostream> // Host STL header not available in freestanding
#endif // _CCCL_HOSTED()
// code ..
Similarly we also provide macros to detect which host standard library is available
+-----------------------------------+----------------------------------------------------+
| ``_CCCL_HAS_HOST_STD_LIB()`` | Whether a known host standard library is available |
+-----------------------------------+----------------------------------------------------+
| ``_CCCL_HOST_STD_LIB(LIBSTDCXX)`` | libstdc++ is available as host standard library |
+-----------------------------------+----------------------------------------------------+
| ``_CCCL_HOST_STD_LIB(LIBCXX)`` | libc++ is available as host standard library |
+-----------------------------------+----------------------------------------------------+
| ``_CCCL_HOST_STD_LIB(STL)`` | MSVC STL is available as host standard library |
+-----------------------------------+----------------------------------------------------+
Usage example:
.. code-block:: c++
#if _CCCL_HAS_HOST_STD_LIB()
_CCCL_BEGIN_NAMESPACE_STD
# if _CCCL_HOST_STD_LIB(STL)
template <class _Tp, size_t _Size>
class array;
# else // ^^^ _CCCL_HOST_STD_LIB(STL) ^^^ / vvv !_CCCL_HOST_STD_LIB(STL) vvv
template <class _Tp, size_t _Size>
struct array;
# endif // !_CCCL_HOST_STD_LIB(STL)
_CCCL_END_NAMESPACE_STD
#endif // _CCCL_HAS_HOST_STD_LIB()

138
cccl_upstream/docs/cccl/development/testing.rst Normal file
View File

@@ -0,0 +1,138 @@
.. _cccl-development-module-testing:
======================
CCCL Testing Utilities
======================
This document describes utilities provided for implementing the *internal* CCCL tests.
They are not intended to be used by end users, but for development of CCCL features only.
We reserve the right to change them at any time without warning.
----
-------------------------------------------------------------------
Test Variants: Generating Multiple Executables from a Single Source
-------------------------------------------------------------------
Some of CCCL's tests are very slow to build and are capable of exhausting RAM
during compilation/linking. To avoid such issues, large tests are split into
multiple executables to take advantage of parallel computation and reduce memory
usage.
CCCL facilitates this by providing a CMake-based solution for automatically generating multiple
test executables from a single source file. This is done by using one or more ``%PARAM%`` comments
in the test's source code, each of which defines a parameter that will be split across multiple
executables.
The CMake functions that implement this feature are in ``cmake/CCCLTestParams.cmake``.
An example of their usage is provided below.
Using ``%PARAM%``
-----------------
The ``%PARAM%`` hint provides an automated method of generating multiple test
executables from a single source file. To use it, add one or more special
comments to the test source file::
// %PARAM% [definition] [label] [values]
CMake will parse the source file and extract these comments, using them to
generate multiple test executables for the full cartesian product of values.
- ``definition`` will be used as a preprocessor definition name. By convention,
these begin with ``TEST_``.
- ``label`` is a short, human-readable label that will be used in the test
executable's name to identify the test variant.
- ``values`` is a colon-separated list of values used during test generation. Only
numeric values have been tested.
Example
*******
A source file containing the following hints::
// %PARAM% TEST_FOO foo 0:1:2
// %PARAM% TEST_LAUNCH lid 0:1
will generate six variants with unique preprocessor definitions:
+-----------------------------+-------------------------------------------+
| Executable Name | Preprocessor Definitions |
+=============================+===========================================+
| ``<name_base>.foo_0.lid_0`` | ``-DTEST_FOO=0 -DTEST_LAUNCH=0 VAR_ID=0`` |
+-----------------------------+-------------------------------------------+
| ``<name_base>.foo_0.lid_1`` | ``-DTEST_FOO=0 -DTEST_LAUNCH=1 VAR_ID=1`` |
+-----------------------------+-------------------------------------------+
| ``<name_base>.foo_1.lid_0`` | ``-DTEST_FOO=1 -DTEST_LAUNCH=0 VAR_ID=2`` |
+-----------------------------+-------------------------------------------+
| ``<name_base>.foo_1.lid_1`` | ``-DTEST_FOO=1 -DTEST_LAUNCH=1 VAR_ID=3`` |
+-----------------------------+-------------------------------------------+
| ``<name_base>.foo_2.lid_0`` | ``-DTEST_FOO=2 -DTEST_LAUNCH=0 VAR_ID=4`` |
+-----------------------------+-------------------------------------------+
| ``<name_base>.foo_2.lid_1`` | ``-DTEST_FOO=2 -DTEST_LAUNCH=1 VAR_ID=5`` |
+-----------------------------+-------------------------------------------+
Changing ``%PARAM%`` Hints
**************************
Since CMake does not automatically reconfigure the build when source files are
modified, CMake will need to be rerun manually whenever the ``%PARAM%`` comments
change.
Using the CMake Variant Functions
---------------------------------
``cmake/CCCLTestParams.cmake`` provides the functions that implement this functionality.
See that file for detailed documentation. An example of their usage is:
.. code-block:: cmake
set(test_src <path_to_source_file>)
set(test_name <test_name_derived_from_test_src>)
# Parse %PARAM% comments from the source file and generate lists of labels/definitions:
cccl_parse_variant_params("${test_src}" num_variants variant_labels variant_defs)
if (num_variants EQUAL 0)
# Add test with no variants named `test_name` here. Example:
add_executable("${test_name}" "${test_src}")
add_test(NAME "${test_name}" COMMAND "${test_name}")
else() # Has variants:
# Optional: log the detected variant info to CMake's VERBOSE output stream:
cccl_log_variant_params("${test_name}" ${num_variants} variant_labels variant_defs)
# Subtract 1 to support the inclusive endpoint of foreach(...RANGE...):
math(EXPR var_range_end "${num_variants} - 1")
foreach(var_idx RANGE ${var_range_end})
# Get the variant label and definitions for the current index:
cccl_get_variant_data(variant_labels variant_defs ${var_idx} var_label var_defs)
set(var_name "${test_name}.${var_label}")
# Add the test with the current variant label and definitions.
# Example:
add_executable("${var_name}" "${test_src}")
target_compile_definitions("${var_name}" PRIVATE ${var_defs})
add_test(NAME "${var_name}" COMMAND "${var_name}")
endforeach()
endif()
Debugging
---------
Running CMake with ``--log-level=VERBOSE`` will print out extra information about
all detected test variants.
Additional Info
---------------
Ideally, only parameters that directly influence template instantiations
should be split out in this way. If changing a parameter doesn't change a
template type, the same template instantiations will be compiled into multiple
executables. This defeats the purpose of splitting up the test since the
compiler will generate redundant code across the new split executables.
The best candidate parameters for splitting are input value types, rather than
integral parameters like ``BLOCK_THREADS``, etc. Splitting by value type allows more
infrastructure (data generation, validation) to be reused. Splitting other
parameters can cause build times to increase since type-related infrastructure
has to be rebuilt for each test variant.

63
cccl_upstream/docs/cccl/development/visibility.rst Normal file
View File

@@ -0,0 +1,63 @@
. _cccl-development-visibility:
Symbol Visibility
==================
.. toctree::
:glob:
:maxdepth: 1
visibility/host_stub_visibility
visibility/device_kernel_visibility
visibility/different_architectures
Using kernels in shared libraries is a known source of issues. This relates to the visibility of the kernel functions
and their host stubs, but also ODR violations that arise from compiling with different CUDA architectures.
To alleviate those issues we have derived the following rules to ensure that users can safely rely on CCCL features in
shared libraries.
1. Every kernel should be annotated as ``hidden`` through ``_CCCL_KERNEL_ATTRIBUTES``
2. Every function or type that eventually calls a kernel in a subsequent function call or member function must be put in
a namespace that disambiguates the CUDA architectures the library was compiled with.
3. It is important that an API accepting kernel pointers (e.g. ``triple_chevron``) always resides in the same
library as the code taking this pointers.
In the following we will give a more detailed overview over the different problems and why we settled on above rules.
Problem 1: Selecting the right kernel stub
-------------------------------------------
Consider a project that links two shared libraries ``lib_a`` and ``lib_b`` that involve a kernel call of some global
``kernel`` template. The compiler will generate a stub function that handles actually launching the kernel via the CUDA
runtime. Prior to CTK 13.0 that stub function has weak linkage, so if both libraries try to launch ``kernel`` only one
host stub will be selected and the other kernel launch might silently fail. See the compiler teams
`blog post <https://developer.nvidia.com/blog/cuda-c-compiler-updates-impacting-elf-visibility-and-linkage/>`_ about the
recent changes to kernel visibility.
A more detailed description can be found :ref:`here <cccl-development-visibility-host-stub-visibility>`.
Problem 2: Calling kernels from inside a shared library
--------------------------------------------------------
This is quite similar to Problem 1 above. Again a project links two shared libraries ``lib_a`` and ``lib_b``. However,
this time we call a library function ``foo`` that takes a function pointer to a kernel as an argument and invokes it.
If ``foo`` has weak external linkage we might end up calling ``lib_b::foo`` from inside ``lib_b`` instead of
``lib_a::foo``, or vice versa. The CUDA runtime from ``lib_a`` will not be able to call the kernel function pointer we
passed from ``lib_b``.
A more detailed description can be found :ref:`here <cccl-development-visibility-device-kernel-visibility>`.
Problem 3: Libraries compiled for different architectures
----------------------------------------------------------
This is orthogonal to the visibility of the functions themself but relates to ODR
(`_one definition rule_ <https://en.cppreference.com/w/cpp/language/definition.html>`_) violations in case libraries are
compiled for different architectures. As new architectures come out, we adopt new features to provide the best possible
performance for all existing architectures.
However, consider a kernel that relies on hardware dependent tuning or can leverage runtime features that are only
available on certain hardware. If we build 2 libraries for different architectures then the kernel implementation
will be different between the two libraries, but the kernel itself is mangled as the same symbol.
A more detailed description can be found :ref:`here <cccl-development-visibility-different-architectures>`.

133
cccl_upstream/docs/cccl/development/visibility/device_kernel_visibility.rst Normal file
View File

@@ -0,0 +1,133 @@
.. _cccl-development-visibility-device-kernel-visibility:
Device Kernel Visibility Issue
-------------------------------
Consider the following simple translation unit (TU):
.. code-block:: cpp
template <class T>
__global__ void kernel(T *val) {
::printf("kernel: set val = 42\n");
*val = 42;
}
int main() {
int *ptr{};
kernel<<<1, 1>>>(ptr);
}
The cuda compiler frontend will turn this into:
.. code-block:: cpp
template< class T>
static void __wrapper__device_stub_kernel(T *&ptr) {
::cudaLaunchKernel(0, 0, 0, 0, 0, 0);
}
// stub host function
template< class T>
void kernel(T *ptr) {
__wrapper__device_stub_kernel<T>(ptr);
}
int main() {
int *ptr{};
(__cudaPushCallConfiguration(1, 1)) ? (void)0 : kernel(ptr);
}
static void __device_stub__Z6kernelIiEvPT_(int *__par0) {
__cudaLaunchPrologue(1);
__cudaSetupArgSimple(__par0, 0UL);
__cudaLaunch(((char *)((void ( *)(int *))kernel )));
}
template<> void __wrapper__device_stub_kernel(int *&__cuda_0) {
__device_stub__Z6kernelIiEvPT_( (int *&)__cuda_0);
}
The CUDA runtime is going to use the address of ``template<> void kernel(T *ptr)`` (in the following ``h_kernel``)
as a key in the host stub function (``h_kernel``) - device function (``d_kernel``) mapping. This works fine if
there is only a single source of truth for the stub function ``h_kernel``.
However, imagine that there are two shared libraries: ``lib_a`` and ``lib_b`` both instantiating different ``kernel``
instances, e.g ``d_kernel<int>`` and ``d_kernel<size_t>``.
.. code-block:: cmake
project(DeviceKernelVisibility CUDA CXX)
add_executable(device_kernel_visibility main.cu)
add_library(lib_a SHARED tu_a.cu)
add_library(lib_b SHARED tu_b.cu)
target_link_libraries(device_kernel_visibility PRIVATE lib_a lib_b)
Each library will have it's own fatbinary: ``d_kernel<int>_a`` and ``d_kernel<size_t>_b`` as well as host stub functions
``h_kernel<int>_a`` and ``h_kernel<size_t>_b``.
=== ============= ============
lib host device
=== ============= ============
a 0xh_kernel_a 0xd_kernel_a
b 0xh_kernel_b 0xd_kernel_b
=== ============= ============
In contrast to
:ref:`Problem 1 <cccl-development-visibility-host-stub-visibility>` the host stubs will get a different mangled name
and so the right stub function will always be selected.
Now imagine that both libraries are going to defer launching of their kernels to a function ``foo`` common to both
``lib_a`` and ``lib_b``, which has weak external linkage. This might happen in ``CUB``, because it launches
kernels through the ``thrust::triple_chevron`` helper.
Similar to :ref:`Problem 1 <cccl-development-visibility-host-stub-visibility>` the linker will pick one of the two
weak symbols and subsequently ``lib_a`` will try to pass its own kernel ``d_kernel<int>_a`` to ``lib_b::foo``.
However, the CUDA runtime in ``lib_b`` will not find any kernel registered at the address of ``d_kernel<int>_a`` and
will fail to launch the kernel.
A simple example program that exemplifies this can be found
`on github <https://github.com/NVIDIA/cccl/tree/main/docs/cub/developer/visibility/examples/device_kernel_visibility>`_
.. code-block:: bash
./device_kernel_visibility/device_kernel_visibility
a: kernel stub address: 0x7fdec19e13eb <== launching kernel_a from a
a: kernel is in mapping: no error
b: launched kernel
a: kernel: set val = 42
a: synchronized stream
a: copied from device to host
a: out: 42
a: kernel was launched: out == 42 <== everything is fine
a: defers launch to b
b: kernel stub address: 0x7fdec19e13eb <== launch kernel_a from b
b: kernel NOT found in mapping: invalid device function <== kernel_a is not found in b mapping
b: FAILED to launch kernel <== unable to launch the kernel from b
b: synchronized stream
b: copied from device to host
b: out: 0
b: kernel was NOT actually launched: out != 42
b: kernel stub address: 0x7fdec19333eb <== launch kernel_b from b
b: kernel is in mapping: no error
b: launched kernel
b: kernel: set val = 42
b: synchronized stream
b: copied from device to host
b: out: 42
b: kernel was launched: out == 42 <== everything is fine
b: defers launch to a
a: kernel stub address: 0x7fdec19333eb <== launching kernel_b from a
a: kernel NOT found in mapping: invalid device function <== same issue as above
a: FAILED to launch kernel
b: kernel: set val = 42
a: synchronized stream
a: copied from device to host
a: out: 42
a: kernel was launched: out == 42 <== kernel launch somehow succeeded

186
cccl_upstream/docs/cccl/development/visibility/different_architectures.rst Normal file
View File

@@ -0,0 +1,186 @@
.. _cccl-development-visibility-different-architectures:
Linking TUs compiled with different architectures
--------------------------------------------------
Consider the following simple library:
.. code-block:: cpp
template <int... Archs>
__host__ __device__ constexpr int sum_archs() noexcept {
return (Archs + ... + 0);
}
// kernel with architecture dependent symbol name and functionality
template <class T, auto Archs = sum_archs<__CUDA_ARCH_LIST__>()>
__global__ void kernel(T *val) {
*val = sum_archs<__CUDA_ARCH_LIST__>();
}
__attribute__((visibility("hidden"))) inline int use_kernel() {
int *d_val{};
cudaMalloc(&d_val, sizeof(d_val));
kernel<<<1, 1>>>(d_val);
int ret;
if (cudaMemcpy(&ret, d_val, sizeof(size_t), cudaMemcpyDeviceToHost) !=
cudaSuccess) {
std::printf("c: FAILED to copy from device to host\n");
}
return ret;
}
template <class T = int>
struct some_class_with_kernel {
T val_;
some_class_with_kernel();
__forceinline__ some_class_with_kernel(T) { val_ = use_kernel(); }
};
We have a kernel that does some architecture dependent work. This could be relying on some hardware feature that is
dependent on the current architecture.
.. code-block:: cpp
#include "kernel.cuh"
int main() {
some_class_with_kernel with_inline{1};
std::printf("a: value of class with inlined constructor: %d\n",
with_inline.val_);
some_class_with_kernel from_library{};
std::printf("a: value of class with constructor from library: %d\n",
from_library.val_);
}
Importantly, one of the constructors for that class is put into a shared library, whereas the other one happens to be
inlined. If a user now links two different libraries, the outcome of the initialization of ``some_class_with_kernel``
will depend on whether the inlined constructor is called and which of the libraries is loaded first by the linker.
Even worse, the state of a class depends on whether the constructor has been inlined or not and the order in which
the linker loads the libraries.
.. code-block:: cmake
project(CUBVisDifferentArchitectures CUDA CXX)
add_library(cubvis_different_architectures_lib_a SHARED tu_a.cu)
set_target_properties(cubvis_different_architectures_lib_a PROPERTIES CUDA_ARCHITECTURES "86;90a")
add_library(cubvis_different_architectures_lib_b SHARED tu_b.cu)
set_target_properties(cubvis_different_architectures_lib_b PROPERTIES CUDA_ARCHITECTURES "75;86;90a")
add_executable(cubvis_different_architectures main.cu)
set_target_properties(cubvis_different_architectures PROPERTIES CUDA_ARCHITECTURES "75;86")
target_link_libraries(cubvis_different_architectures PRIVATE
cubvis_different_architectures_lib_a
cubvis_different_architectures_lib_b)
add_executable(cubvis_different_architectures_switched main.cu)
set_target_properties(cubvis_different_architectures_switched PROPERTIES CUDA_ARCHITECTURES "75;86")
target_link_libraries(cubvis_different_architectures_switched PRIVATE
cubvis_different_architectures_lib_b
cubvis_different_architectures_lib_a)
Execution the two libraries will result in the following:
.. code-block::
./different_architectures/different_architectures
a: value of class with inlined constructor: 1610 <<<--- from main
a: value of class with constructor from library: 1760 <<<--- from lib_a
./different_architectures/different_architectures_switched
a: value of class with inlined constructor: 1610 <<<--- from main
a: value of class with constructor from library: 2510 <<<--- from lib_b
One solution would be to bake the architectures into the symbol name of the class, either via a defaulted template
argument or an inline namespace. That way the usage of the non-inlined kernel would result in a linker error, because
we did not provide a matching implementation.
.. code-block::
tmpxft_00048dff_00000000-6_main.compute_86.cudafe1.cpp:(.text.startup+0xc0):
undefined reference to `some_class_with_kernel<int, 5120ul>::some_class_with_kernel()'
However, if all the functionality is within a non-inlined function we would still get different results, because all
kernel definitions would be internal to the respective library.
.. code-block:: cpp
// In tu_a.cu and tu_b.cu
void non_inlined_function() {
some_class_with_kernel with_inline{1};
std::printf("a: value of class with inlined constructor: %d\n",
with_inline.val_);
some_class_with_kernel from_library{};
std::printf("a: value of class with constructor from library: %d\n",
from_library.val_);
}
// In main.cu
#include "kernel.cuh"
void non_inlined_function();
int main() {
some_class_with_kernel with_inline{1};
std::printf("a: value of class with inlined constructor: %d\n",
with_inline.val_);
non_inlined_function();
}
Executing this binary will give us again:
.. code-block::
./different_architectures/different_architectures
a: value of class with inlined constructor: 1610 <<<--- from main
a: value of class with inlined constructor: 1760 <<<--- from lib_a
a: value of class with constructor from library: 1760 <<<--- from lib_a
./different_architectures/different_architectures_switched
a: value of class with inlined constructor: 1610 <<<--- from main
a: value of class with inlined constructor: 2510 <<<--- from lib_a
a: value of class with constructor from library: 2510 <<<--- from lib_b
So there is not functional way we can solve this problem generically, because the moment a user actually uses any type
of function that executes a kernel and puts that function into a shared library there is no guarantee which function
is selected. The same happens if the user builds a type
.. code-block:: cpp
class user_defined_with_kernel {
some_class_with_kernel val;
user_defined_with_kernel();
__forceinline__ user_defined_with_kernel(T input) : val(input)
{}
};
void function_that_uses_kernel_inside();
If ``user_defined_with_kernel`` is ever baked into a library we would be back with the same exact problem,
just one layer up. The user would need to know that ``some_class_with_kernel`` uses a kernel and then annotate *their*
classes and functions appropriately. This is neither realistic nor feasible.
Lets circle back to the previous statement: ``This is bad.`` Is it really though?
Lets look at the prime example ``thrust::device_vector``, which uses a kernel for initialization. What happens if we
accidentally run the kernel from another shared library compiled with different architectures? Worst case we are
eating some performance regressions because the kernel will not utilize advanced features of a new architecture,
but in the end the result of calling that kernel will not change the outcome.
This is because the kernel call is consistent *within* each library. As long as the user facing API does not rely on
specific internals of a kernel to be called -which it should not-, then any of the two libraries will do.
Finally, the architectures that are passed around in ``__CUDA_ARCH_LIST__`` do *not* discriminate architecture families.
There is currently no programmatic way to discriminate a library that has been compiled for ``SM90a`` from one that was
compiled for ``SM90``. This is because the architecture specific macros are only available on device not on host.

10
cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/CMakeLists.txt Normal file
View File

@@ -0,0 +1,10 @@
project(DeveloperGuideDeviceKernelVisibility CUDA CXX)
add_executable(device_kernel_visibility main.cu)
add_library(device_kernel_visibility_liba SHARED tu_a.cu)
add_library(device_kernel_visibility_lib_b SHARED tu_b.cu)
target_link_libraries(
device_kernel_visibility
PRIVATE device_kernel_visibility_lib_a device_kernel_visibility_lib_b
)

10
cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/kernel.cuh Normal file
View File

@@ -0,0 +1,10 @@
#pragma once
#include <cstdio>
template <class T>
__global__ void kernel(char ln, T* val)
{
printf("%c: kernel: set val = 42\n", ln);
*val = 42;
}

8
cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/main.cu Normal file
View File

@@ -0,0 +1,8 @@
void a();
void b();
int main()
{
a();
b();
}

132
cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/tu_a.cu Normal file
View File

@@ -0,0 +1,132 @@
#include "kernel.cuh"
void b_launch(void (*k)(char, size_t*), char c, size_t* d_out);
void a_launch(void (*k)(char, int*), char c, int* d_out)
{
void* ptr = reinterpret_cast<void*>(k);
printf("a: kernel stub address: %p\n", ptr);
cudaFunction_t func{};
if (cudaError_t error = cudaGetFuncBySymbol(&func, ptr))
{
printf("a: kernel NOT found in mapping: %s\n", cudaGetErrorString(error));
}
else
{
printf("a: kernel is in mapping: %s\n", cudaGetErrorString(error));
}
cudaMemset(d_out, 0, sizeof(int));
k<<<1, 1>>>(c, d_out);
if (cudaPeekAtLastError() != cudaSuccess)
{
printf("a: FAILED to launch kernel\n");
}
else
{
printf("a: launched kernel\n");
}
if (cudaStreamSynchronize(0) != cudaSuccess)
{
printf("a: FAILED to synchronize stream\n");
}
else
{
printf("a: synchronized stream\n");
}
int h_out{};
if (cudaMemcpy(&h_out, d_out, sizeof(int), cudaMemcpyDeviceToHost) != cudaSuccess)
{
printf("a: FAILED to copy from device to host\n");
}
else
{
printf("a: copied from device to host\n");
}
printf("a: out: %d\n", h_out);
if (h_out != 42)
{
printf("a: kernel was NOT actually launched: out != 42\n");
}
else
{
printf("a: kernel was launched: out == 42\n");
}
printf("\n");
}
void a()
{
cudaGetLastError();
size_t* d_out{};
cudaMalloc(&d_out, sizeof(size_t));
cudaMemset(d_out, 0, sizeof(size_t));
void* ptr = reinterpret_cast<void*>(kernel<size_t>);
printf("a: kernel stub address: %p\n", ptr);
cudaFunction_t func{};
if (cudaError_t error = cudaGetFuncBySymbol(&func, ptr))
{
printf("a: kernel NOT found in mapping: %s\n", cudaGetErrorString(error));
}
else
{
printf("a: kernel is in mapping: %s\n", cudaGetErrorString(error));
}
kernel<<<1, 1>>>('a', d_out);
if (cudaPeekAtLastError() != cudaSuccess)
{
printf("b: FAILED to launch kernel\n");
}
else
{
printf("b: launched kernel\n");
}
if (cudaStreamSynchronize(0) != cudaSuccess)
{
printf("a: FAILED to synchronize stream\n");
}
else
{
printf("a: synchronized stream\n");
}
size_t h_out{};
if (cudaMemcpy(&h_out, d_out, sizeof(size_t), cudaMemcpyDeviceToHost) != cudaSuccess)
{
printf("a: FAILED to copy from device to host\n");
}
else
{
printf("a: copied from device to host\n");
}
printf("a: out: %d\n", static_cast<int>(h_out));
if (h_out != 42)
{
printf("a: kernel was NOT actually launched: out != 42\n");
}
else
{
printf("a: kernel was launched: out == 42\n");
}
cudaMemset(d_out, 0, sizeof(size_t));
printf("\n");
printf("a: defers launch to b\n");
b_launch(kernel<size_t>, 'b', d_out);
}

132
cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/tu_b.cu Normal file
View File

@@ -0,0 +1,132 @@
#include "kernel.cuh"
void a_launch(void (*k)(char, int*), char c, int* d_out);
void b_launch(void (*k)(char, size_t*), char c, size_t* d_out)
{
void* ptr = reinterpret_cast<void*>(k);
printf("b: kernel stub address: %p\n", ptr);
cudaFunction_t func{};
if (cudaError_t error = cudaGetFuncBySymbol(&func, ptr))
{
printf("b: kernel NOT found in mapping: %s\n", cudaGetErrorString(error));
}
else
{
printf("b: kernel is in mapping: %s\n", cudaGetErrorString(error));
}
k<<<1, 1>>>(c, d_out);
if (cudaPeekAtLastError() != cudaSuccess)
{
printf("b: FAILED to launch kernel\n");
}
else
{
printf("b: launched kernel\n");
}
if (cudaStreamSynchronize(0) != cudaSuccess)
{
printf("b: FAILED to synchronize stream\n");
}
else
{
printf("b: synchronized stream\n");
}
size_t h_out{};
if (cudaMemcpy(&h_out, d_out, sizeof(size_t), cudaMemcpyDeviceToHost) != cudaSuccess)
{
printf("b: FAILED to copy from device to host\n");
}
else
{
printf("b: copied from device to host\n");
}
printf("b: out: %d\n", static_cast<int>(h_out));
if (h_out != 42)
{
printf("b: kernel was NOT actually launched: out != 42\n");
}
else
{
printf("b: kernel was launched: out == 42\n");
}
printf("\n");
}
void b()
{
cudaGetLastError();
int* d_out{};
cudaMalloc(&d_out, sizeof(int));
cudaMemset(d_out, 0, sizeof(int));
void* ptr = reinterpret_cast<void*>(kernel<int>);
printf("b: kernel stub address: %p\n", ptr);
cudaFunction_t func{};
if (cudaError_t error = cudaGetFuncBySymbol(&func, ptr))
{
printf("b: kernel NOT found in mapping: %s\n", cudaGetErrorString(error));
}
else
{
printf("b: kernel is in mapping: %s\n", cudaGetErrorString(error));
}
kernel<<<1, 1>>>('b', d_out);
if (cudaPeekAtLastError() != cudaSuccess)
{
printf("b: FAILED to launch kernel\n");
}
else
{
printf("b: launched kernel\n");
}
if (cudaStreamSynchronize(0) != cudaSuccess)
{
printf("b: FAILED to synchronize stream\n");
}
else
{
printf("b: synchronized stream\n");
}
int h_out{};
if (cudaMemcpy(&h_out, d_out, sizeof(int), cudaMemcpyDeviceToHost) != cudaSuccess)
{
printf("b: FAILED to copy from device to host\n");
}
else
{
printf("b: copied from device to host\n");
}
printf("b: out: %d\n", h_out);
if (h_out != 42)
{
printf("b: kernel was NOT actually launched: out != 42\n");
}
else
{
printf("b: kernel was launched: out == 42\n");
}
cudaMemset(d_out, 0, sizeof(int));
printf("\n");
printf("b: defers launch to a\n");
a_launch(kernel<int>, 'b', d_out);
}

35
cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/CMakeLists.txt Normal file
View File

@@ -0,0 +1,35 @@
project(DeveloperGuideDifferentArchitectures CUDA CXX)
add_library(different_architectures_lib_a SHARED tu_a.cu)
set_target_properties(
different_architectures_lib_a
PROPERTIES CUDA_ARCHITECTURES "86;90a"
)
add_library(different_architectures_lib_b SHARED tu_b.cu)
set_target_properties(
different_architectures_lib_b
PROPERTIES CUDA_ARCHITECTURES "75;86;90a"
)
add_executable(different_architectures main.cu)
set_target_properties(
different_architectures
PROPERTIES CUDA_ARCHITECTURES "75;86"
)
target_link_libraries(
different_architectures
PRIVATE different_architectures_lib_a different_architectures_lib_b
)
add_executable(different_architectures_switched main.cu kernel.cu)
set_target_properties(
different_architectures_switched
PROPERTIES CUDA_ARCHITECTURES "75;86"
)
target_link_libraries(
different_architectures_switched
PRIVATE different_architectures_lib_b different_architectures_lib_a
)

42
cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/kernel.cuh Normal file
View File

@@ -0,0 +1,42 @@
#pragma once
#include <cstdint>
#include <cstdio>
template <int... Archs>
__attribute__((visibility("hidden"))) __host__ __device__ constexpr int sum_archs()
{
return (Archs + ... + 0);
}
template <class T, auto Archs = sum_archs<__CUDA_ARCH_LIST__>()>
__attribute__((visibility("hidden"))) __global__ void kernel(char ln, T* val)
{
printf("%c: kernel: set val = %i\n", ln, sum_archs<__CUDA_ARCH_LIST__>());
*val = sum_archs<__CUDA_ARCH_LIST__>();
}
__attribute__((visibility("hidden"))) __forceinline__ int use_kernel()
{
int* d_val{};
cudaMalloc(&d_val, sizeof(size_t));
kernel<<<1, 1>>>(d_val);
int ret;
if (cudaMemcpy(&ret, d_val, sizeof(size_t), cudaMemcpyDeviceToHost) != cudaSuccess)
{
printf("c: FAILED to copy from device to host\n");
}
return ret;
}
template <class T = int>
struct some_class_with_kernel
{
T val_;
some_class_with_kernel();
__forceinline__ some_class_with_kernel(T)
{
val_ = use_kernel();
}
};

11
cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/main.cu Normal file
View File

@@ -0,0 +1,11 @@
#include "kernel.cuh"
void non_inlined_function();
int main()
{
some_class_with_kernel with_inline{1};
printf("a: value of class with inlined constructor: %d\n", with_inline.val_);
non_inlined_function();
}

16
cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/tu_a.cu Normal file
View File

@@ -0,0 +1,16 @@
#include "kernel.cuh"
template <class T>
some_class_with_kernel<T>::some_class_with_kernel()
{
val_ = use_kernel();
}
void non_inlined_function()
{
some_class_with_kernel with_inline{1};
printf("a: value of class with inlined constructor: %d\n", with_inline.val_);
some_class_with_kernel from_library{};
printf("a: value of class with constructor from library: %d\n", from_library.val_);
}

16
cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/tu_b.cu Normal file
View File

@@ -0,0 +1,16 @@
#include "kernel.cuh"
template <class T>
some_class_with_kernel<T>::some_class_with_kernel()
{
val_ = use_kernel();
}
void non_inlined_function()
{
some_class_with_kernel with_inline{1};
printf("a: value of class with inlined constructor: %d\n", with_inline.val_);
some_class_with_kernel from_library{};
printf("a: value of class with constructor from library: %d\n", from_library.val_);
}

10
cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/CMakeLists.txt Normal file
View File

@@ -0,0 +1,10 @@
project(DeveloperGuideHostStubVisibility CUDA CXX)
add_executable(host_stub_visibility main.cu)
add_library(host_stub_visibility_lib_a SHARED tu_a.cu)
add_library(host_stub_visibility_lib_b SHARED tu_b.cu)
target_link_libraries(
host_stub_visibility
PRIVATE host_stub_visibility_lib_a host_stub_visibility_lib_b
)

10
cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/kernel.cuh Normal file
View File

@@ -0,0 +1,10 @@
#pragma once
#include <cstdio>
template <class T>
__global__ void kernel(char ln, T* val)
{
printf("%c: kernel: set val = 42\n", ln);
*val = 42;
}

8
cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/main.cu Normal file
View File

@@ -0,0 +1,8 @@
void a();
void b();
int main()
{
a();
b();
}

64
cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/tu_a.cu Normal file
View File

@@ -0,0 +1,64 @@
#include "kernel.cuh"
void a()
{
int* d_out{};
cudaMalloc(&d_out, sizeof(int));
cudaMemset(d_out, 0, sizeof(int));
void* ptr = reinterpret_cast<void*>(kernel<int>);
printf("a: kernel stub address: %p\n", ptr);
cudaFunction_t func{};
if (cudaError_t error = cudaGetFuncBySymbol(&func, ptr))
{
printf("a: kernel NOT found in mapping: %s\n", cudaGetErrorString(error));
}
else
{
printf("a: kernel is in mapping: %s\n", cudaGetErrorString(error));
}
kernel<<<1, 1>>>('a', d_out);
if (cudaPeekAtLastError() != cudaSuccess)
{
printf("b: FAILED to launch kernel\n");
}
else
{
printf("b: launched kernel\n");
}
if (cudaStreamSynchronize(0) != cudaSuccess)
{
printf("a: FAILED to synchronize stream\n");
}
else
{
printf("a: synchronized stream\n");
}
int h_out{};
if (cudaMemcpy(&h_out, d_out, sizeof(int), cudaMemcpyDeviceToHost) != cudaSuccess)
{
printf("a: FAILED to copy from device to host\n");
}
else
{
printf("a: copied from device to host\n");
}
printf("a: out: %d\n", h_out);
if (h_out != 42)
{
printf("a: kernel was NOT actually launched: out != 42\n");
}
else
{
printf("a: kernel was launched: out == 42\n");
}
printf("\n");
}

65
cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/tu_b.cu Normal file
View File

@@ -0,0 +1,65 @@
#include "kernel.cuh"
void b()
{
int* d_out{};
cudaMalloc(&d_out, sizeof(int));
cudaMemset(d_out, 0, sizeof(int));
void* ptr = reinterpret_cast<void*>(kernel<int>);
printf("b: kernel stub address: %p\n", ptr);
cudaFunction_t func{};
if (cudaError_t error = cudaGetFuncBySymbol(&func, ptr))
{
printf("b: kernel NOT found in mapping: %s\n", cudaGetErrorString(error));
}
else
{
printf("b: kernel is in mapping: %s\n", cudaGetErrorString(error));
}
kernel<<<1, 1>>>('b', d_out);
if (cudaPeekAtLastError() != cudaSuccess)
{
printf("b: FAILED to launch kernel\n");
}
else
{
printf("b: launched kernel\n");
}
if (cudaStreamSynchronize(0) != cudaSuccess)
{
printf("b: FAILED to synchronize stream\n");
}
else
{
printf("b: synchronized stream\n");
}
int h_out{};
if (cudaMemcpy(&h_out, d_out, sizeof(int), cudaMemcpyDeviceToHost) != cudaSuccess)
{
printf("b: FAILED to copy from device to host\n");
}
else
{
printf("b: copied from device to host\n");
}
printf("b: out: %d\n", h_out);
if (h_out != 42)
{
printf("b: kernel was NOT actually launched: out != 42\n");
}
else
{
printf("b: kernel was launched: out == 42\n");
}
printf("\n");
}

115
cccl_upstream/docs/cccl/development/visibility/host_stub_visibility.rst Normal file
View File

@@ -0,0 +1,115 @@
.. _cccl-development-visibility-host-stub-visibility:
Host Stub Visibility Issue
---------------------------
Consider the following simple translation unit (TU):
.. code-block:: cpp
#include <cstdio>
#include <cuda/memory>
template <class T>
__global__ void kernel(T *val) {
printf("kernel: set val = 42\n");
*val = 42;
}
__device__ int val;
int main() {
kernel<<<1, 1>>>(cuda::get_device_address(val));
}
The CUDA compiler frontend will turn this into:
.. code-block:: cpp
template< class T>
static void __wrapper__device_stub_kernel(T *&ptr) {
::cudaLaunchKernel(0, 0, 0, 0, 0, 0);
}
// stub host function
template< class T>
void kernel(T *ptr) {
__wrapper__device_stub_kernel<T>(ptr);
}
int main() {
int *ptr{};
(__cudaPushCallConfiguration(1, 1)) ? (void)0 : kernel(ptr);
}
static void __device_stub__Z6kernelIiEvPT_(int *__par0) {
__cudaLaunchPrologue(1);
__cudaSetupArgSimple(__par0, 0UL);
__cudaLaunch(((char *)((void ( *)(int *))kernel )));
}
template<> void __wrapper__device_stub_kernel(int *&__cuda_0) {
__device_stub__Z6kernelIiEvPT_( (int *&)__cuda_0);
}
The CUDA runtime is going to use the address of ``template<> void kernel(T *ptr)`` (in the following ``h_kernel``)
as a key in the host stub function (``h_kernel``) - device function (``d_kernel``) mapping. This works fine if
there is only a single source of truth for the stub function ``h_kernel``.
However, imagine that there are two shared libraries: ``lib_a`` and ``lib_b`` both using the same ``kernel`` instance.
.. code-block:: cmake
project(HostStubVisibility CUDA CXX)
add_executable(host_stub_visibility main.cu)
add_library(lib_a SHARED tu_a.cu)
add_library(lib_b SHARED tu_b.cu)
target_link_libraries(host_stub_visibility PRIVATE lib_a lib_b)
Each library will have its own fatbinary: ``d_kernel_a`` and ``d_kernel_b``, but the compiler
generated host stub function ``h_kernel`` has weak external linkage, so after dynamic linkage, we'll end up having
only one of them.
=== ===================== ============
lib host device
=== ===================== ============
a 0xh_kernel_a 0xd_kernel_a
b 0xh_kernel_a <- issue 0xd_kernel_b
=== ===================== ============
Since there's a clash of stub function addresses, only one entry stored. When ``lib_b`` queries for the
kernel using its address of ``h_kernel``, it's visible, although it might point to ``lib_a``'s fatbinary.
The opposite case might happen as well, depending on loading order, linker etc and is undefined behavior.
Launching ``d_kernel`` from ``lib_b`` is not possible and leads to random errors. For instance, there seems to be
some per CUDART global state. When the ``__cudaPushCallConfiguration`` is called in ``lib_b``, it affects the state of
``cudart_b``, but the launch happens through ``h_kernel``, which is in ``lib_a``.
This sometimes leads to ``__global__ function call is not configured``. However, there might also be no error at all,
and the kernel launch is silently skipped.
A simple example program that exemplifies this can be found
`on github <https://github.com/NVIDIA/cccl/tree/main/docs/cub/developer/visibility/examples/host_stub_visibility>`_
.. code-block:: bash
:./host_stub_visibility/host_stub_visibility
a: kernel stub address: 0x7f43318a415d <== same address as in B
a: kernel is in mapping: no error <== kernel is found in the mapping
b: launched kernel
a: kernel: set val = 42
a: synchronized stream
a: copied from device to host
a: out: 42
a: kernel was launched: out == 42
b: kernel stub address: 0x7f43318a415d <== same address as in A
b: kernel is in mapping: no error <== kernel is found in the mapping
b: launched kernel
b: synchronized stream
b: copied from device to host
b: out: 0
b: kernel was NOT actually launched: out != 42 <== silent failure

12
cccl_upstream/docs/cccl/license.rst Normal file
View File

@@ -0,0 +1,12 @@
.. _cccl-license:
License
========
The CUDA Core Compute Libraries use a mix of several licenses, mostly for historical reasons.
libcu++ uses the `Apache License v2.0 with LLVM Exceptions <https://llvm.org/LICENSE.txt>`_.
CUB mostly uses the `3-Clause BSD License <https://github.com/NVIDIA/cccl/blob/main/cub/LICENSE.TXT>`_.
Thrust mostly the `Apache License v2.0 <https://github.com/NVIDIA/cccl/blob/main/thrust/LICENSE>`_.
Newly developed code will be under the `Apache License v2.0 with LLVM Exceptions <https://llvm.org/LICENSE.txt>`_.

35
cccl_upstream/docs/cccl/tma.rst Normal file
View File

@@ -0,0 +1,35 @@
.. _cccl-tma:
Tensor Memory Accelerator (TMA)
===============================
The Tensor Memory Accelerator (TMA) is a hardware feature available on Hopper (SM90) and newer GPUs
that enables efficient asynchronous memory copies of tensor data between global and (cluster) shared memory.
The use of TMA is required to reach SOL memory throughput for some workloads,
notable those where the necessary load/store vectorization, unrolling, or pipelining are limited by the register file or other factors.
CCCL offer several tools to help users leverage TMA in their applications.
In general, we recommend users to reach for high-level algorithms if they fit their problem.
Several algorithms, like ``cub::DeviceTransform``, ``cub::DeviceMerge``, ``cub::DeviceScan`` already use TMA internally today,
with many Thrust algorithms building on those.
And more algorithms will be added over time.
Relying on high level algorithms leaves the complexity of implementing and tuning TMA to CCCL team,
while providing users with safer interfaces, high productivity and SOL performance from the start.
If direct use of TMA is required to author new kernels, CCCL offers the following tools to help users get started,
from high-level to low-level:
- ``cub::BlockLoadToShared`` coming soon :)
- :ref:`cuda::memcpy_async <libcudacxx-extended-api-asynchronous-operations-memcpy-async>`
- :ref:`cuda::device::memcpy_async_tx <libcudacxx-extended-api-asynchronous-operations-memcpy-async-tx>`
- :ref:`cuda::ptx::cp_async_bulk* variants <libcudacxx-ptx-instructions>`
``cub::BlockLoadToShared`` and ``cuda::memcpy_async`` have fallback implementations for pre-Hopper GPUs,
using ``cp.async``/``LDGSTS`` on Ampere (SM80+) and ordinary loads/stores on older architectures.
Furthermore, they gracefully handle unaligned data and copying regions of arbitrary size.
The various ``cuda::ptx::cp_async_bulk*`` versions and ``cuda::device::memcpy_async_tx``
are thin wrappers of the corresponding PTX instructions
and provide no fallback path on older GPUs and also require the copied data to be aligned and sized appropriately.
Some further TMA-related utilities are provided by the :ref:`libcu++ extended API <libcudacxx-extended-api-tma>`.