diff --git a/cccl_upstream/.agent/skills/cccl-style/SKILL.md b/cccl_upstream/.agent/skills/cccl-style/SKILL.md
new file mode 100644
index 00000000..6d5b5eb0
--- /dev/null
+++ b/cccl_upstream/.agent/skills/cccl-style/SKILL.md
@@ -0,0 +1,14 @@
+---
+name: cccl-style
+description: Use when editing or reviewing CCCL code for style conventions; read common CCCL guidance and the path-specific references named by this skill.
+---
+
+# CCCL Style
+
+## Workflow
+
+1. Read `references/common.md` for guidance that applies across CCCL.
+2. For `libcudacxx/include/**/*`, also read `references/libcudacxx.md`.
+3. For `cudax/include/**/*`, also read `references/libcudacxx.md`.
+4. If no path-specific reference exists, follow nearby code and repository docs. Do not import rules from another subproject.
+5. Apply each reference only to its stated scope. Rules for one CCCL subproject do not automatically apply to another.
diff --git a/cccl_upstream/.agent/skills/cccl-style/references/common.md b/cccl_upstream/.agent/skills/cccl-style/references/common.md
new file mode 100644
index 00000000..e786db1a
--- /dev/null
+++ b/cccl_upstream/.agent/skills/cccl-style/references/common.md
@@ -0,0 +1,70 @@
+# Common CCCL Style Guidance
+
+Apply this guidance across CCCL unless a path-specific style reference says otherwise.
+
+## Naming Style
+
+- Macros: macro style, e.g. `MY_MACRO`.
+- Template parameters: PascalCase, e.g. `MyParameter`.
+- All other symbols: snake style, e.g. `my_variable`. The one exception is the CUB public API, which uses PascalCase.
+
+## Variables
+
+- All variables that are not modified must use `const`. This includes variables initialized by casts (`static_cast`, `reinterpret_cast`, `bit_cast`), function return values, and loop-invariant computations.
+- All variables that can be evaluated at compile-time must use `constexpr`.
+- All `constexpr` variables at namespace/global scope must use `inline`, including variable templates.
+- Consider using plural names for array, span, list, e.g. `int values[4]` instead of `int value[4]`.
+- Use uniform initialization for class constructors (not enforced to builtin types) and compile-time conversions, e.g. `constexpr auto x = int{sizeof(float)};`.
+
+## Headers
+
+- Files must include all headers related to the symbols that they are using.
+- Relying on transitive header inclusion is not allowed.
+- Unneeded headers must be removed.
+- All headers must have the correct license. This also applies to source files.
+- All header inclusions must use the syntax `<header>`.
+- Use forward declaration, namely `__fwd/header.h` or direct type declaration, when possible instead of including the implementation header.
+- Headers should be the most precise available, e.g. `#include <cuda/std/__type_traits/is_array.h>`.
+- Do not include headers in `cuda/std/__cccl/` directly; they are provided by `__config` or the prologue/epilogue mechanism.
+
+## Functions
+
+- Functions must be marked `_CCCL_HOST_API`, `_CCCL_DEVICE_API`, `_CCCL_HOST_DEVICE_API`, `_CCCL_TILE_API`, or `_CCCL_API`.
+- Non-template, non-`constexpr` functions must use `inline`.
+- Most functions with a non-void return type should use `[[nodiscard]]`; functions with known side effects may be exceptions.
+- Functions that do not throw exceptions must use `noexcept`.
+- Use `_CCCL_CONSTEVAL` when the function can only be evaluated at compile time.
+- Use C++20 concept macros instead of SFINAE, e.g. `_CCCL_TEMPLATE(...)` and `_CCCL_REQUIRES(...)`.
+
+## Function Calls And Types
+
+- In headers, apply global qualification where the subproject requires it:
+  - libcudacxx and cudax require free function calls to be fully qualified from the global namespace, e.g. `::cuda::ceil_div(...)`.
+  - CUB applies this rule only to calls to symbols under the `::cuda` namespace hierarchy; otherwise follow existing CUB qualification style.
+  - Thrust uses leading `::` for many symbols under the `::cuda` namespace hierarchy, but relies on ADL in many places and the blanket free-function qualification rule does not apply to those calls.
+- For covered calls, this includes calls to functions defined in the same namespace, e.g. inside `cuda::`, call `::cuda::ceil_div(...)`, not `ceil_div(...)`. This does not apply to (static) member functions of classes. The only exceptions for covered calls are functions that are supposed to be found through argument-dependent lookup (ADL), such as `::cuda::std::swap` and `::cuda::std::get`. Those functions can be called unqualified with a preceding `using ::cuda::std::get;`.
+- This global-qualification rule does not apply to source files such as tests and benchmarks.
+- In headers, apply type-name qualification where the subproject requires it:
+  - libcudacxx and cudax require type names to be fully qualified except when they are already declared in the current namespace or an enclosing one. Outside those namespaces, fully qualify `cuda::std` and standard integer type aliases such as `::cuda::std::size_t`.
+  - CUB applies this rule only to type names under the `::cuda` namespace hierarchy. Do not apply the libcudacxx/cudax blanket type-qualification rule to CUB namespaces or `detail` namespaces.
+  - Thrust does not apply the libcudacxx/cudax blanket type-qualification rule. It uses leading `::` for many `::cuda` and `::cuda::std` type names, but also uses Thrust namespace patterns; follow neighboring Thrust code.
+- A local `using` declaration, e.g. `using ::cuda::std::size_t;`, is acceptable to avoid repetition within a function body.
+- Static member functions of a class template inherit the class's namespace.
+
+## Comments
+
+- Commented code without a description is not allowed.
+
+## General Guidelines
+
+- The code must reuse `cuda/` or `cuda/std` functionalities as much as possible, including macros.
+- Try to use modern C++ as much as possible. The repository supports C++17 but many more recent functionalities have been backported with functions and macros.
+
+## Prevent Compiler Errors And Improve Compatibility
+
+- Remove unused code, variables, functions, types, template parameters, headers, etc.
+- Variables that are unsigned, or that can become unsigned after template instantiation, must not check for negative values directly. Use `cuda::std::is_unsigned_v<T> ? false : (var < 0)` instead.
+
+## Compiler Compatibility
+
+- Protect host-only code with `#if !_CCCL_COMPILER(NVRTC)`.
diff --git a/cccl_upstream/.agent/skills/cccl-style/references/libcudacxx.md b/cccl_upstream/.agent/skills/cccl-style/references/libcudacxx.md
new file mode 100644
index 00000000..c3295634
--- /dev/null
+++ b/cccl_upstream/.agent/skills/cccl-style/references/libcudacxx.md
@@ -0,0 +1,55 @@
+# libcudacxx Style Guidance
+
+Use this reference for `libcudacxx/include/**/*` and `cudax/include/**/*`.
+
+## Naming Style
+
+All non-public symbols must be C++ reserved identifiers:
+
+- `_` for macros and template parameters, e.g. `_MY_MACRO`, `_MyParameter`.
+- `__` for all other symbols, e.g. `__my_variable`.
+- Never use reserved keywords, such as `__in`, `__out`, or `__inout` as variables, parameters, or function names.
+- Avoid single-letter template parameter names. Wrong: `_T`; correct: `_Tp`.
+
+## Class / Struct
+
+- Data member names have postfix `_`, e.g. `class __myclass { int __data_; };`.
+- Constructor parameter names should match class/struct data member names without the postfix `_`, e.g. `class __myclass { __myclass(int __data) : __data_(__data) {} };`.
+
+## Functions
+
+- Use `constexpr` for functions that do not depend on run-time features, such as pointers.
+- If the return type is not explicit (`auto`), then a trailing return type is strongly preferred.
+
+## Headers
+
+- Use the correct license:
+  - `libcudacxx/include/cuda/std` files ported from LLVM libc++ use the LLVM license.
+  - `libcudacxx/include/cuda/` files use Apache License v2.0 with LLVM Exceptions.
+- Headers use include guards with names derived from the uppercase full path and closing `#endif` comments repeating the guard name.
+- Right after the include guard, include:
+
+```cpp
+#include <cuda/std/detail/__config>
+
+#if defined(_CCCL_IMPLICIT_SYSTEM_HEADER_GCC)
+#  pragma GCC system_header
+#elif defined(_CCCL_IMPLICIT_SYSTEM_HEADER_CLANG)
+#  pragma clang system_header
+#elif defined(_CCCL_IMPLICIT_SYSTEM_HEADER_MSVC)
+#  pragma system_header
+#endif // no system header
+```
+
+- The last included header before code must be `<cuda/std/__cccl/prologue.h>`, and `<cuda/std/__cccl/epilogue.h>` must appear at the end of the file.
+
+## Comments
+
+- Use Doxygen-style `//! @brief` comments.
+- Documented functions must include `//! @brief`, `//! @param[in/out/in,out]` for every parameter, and `//! @return` for non-void functions.
+- The `@brief/@param/@return` description must accurately reflect the current functionality of the function.
+
+## Compiler Compatibility
+
+- Do not use lambda expressions in device-only or host-device code.
+- Do not rely on deduction guides for initialization; use explicit template arguments instead.
diff --git a/cccl_upstream/.agent/skills/cccl-test/SKILL.md b/cccl_upstream/.agent/skills/cccl-test/SKILL.md
new file mode 100644
index 00000000..8fb2f2b6
--- /dev/null
+++ b/cccl_upstream/.agent/skills/cccl-test/SKILL.md
@@ -0,0 +1,13 @@
+---
+name: cccl-test
+description: Use when writing, updating, reviewing, or validating CCCL tests; read common CCCL test guidance and the path-specific references named by this skill.
+---
+
+# CCCL Test
+
+## Workflow
+
+1. Read `references/common.md` for guidance that applies across CCCL tests.
+2. For `libcudacxx/test/**/*`, also read `references/libcudacxx.md`.
+3. If no path-specific reference exists, follow nearby tests and repository docs. Do not import test rules from another subproject.
+4. Apply each reference only to its stated scope. Rules for one CCCL subproject do not automatically apply to another.
diff --git a/cccl_upstream/.agent/skills/cccl-test/references/common.md b/cccl_upstream/.agent/skills/cccl-test/references/common.md
new file mode 100644
index 00000000..01f20983
--- /dev/null
+++ b/cccl_upstream/.agent/skills/cccl-test/references/common.md
@@ -0,0 +1,31 @@
+# Common CCCL Test Guidance
+
+Apply this guidance across CCCL tests unless a path-specific test reference says otherwise.
+
+## Local Consistency
+
+- Read nearby tests first and mirror their directory layout, file names, helper types, includes, assertion style, and local gating or skip mechanisms.
+
+## Coverage
+
+- Cover relevant edge cases.
+- Cover relevant input and output types.
+- Cover error behavior when applicable.
+- Cover runtime and compile-time behavior when applicable.
+- Cover device and host behavior when applicable.
+
+## Test Structure
+
+- All tests must have the correct license banner.
+- Use the local test harness assertions and helpers.
+- Use compile-time checks for compile-time guarantees and constexpr coverage when relevant.
+- Negative tests should check the intended diagnostic or failure mode when the local harness supports it.
+
+## Portability
+
+- Prefer project test macros and helpers for compiler, dialect, exception, host/device, and platform probes instead of spelling ad hoc checks directly.
+- If a test is unsupported, expected to fail, disabled, or skipped on a platform, motivate it with a comment.
+
+## Validation
+
+- Use targeted test runs for the project and files being changed.
diff --git a/cccl_upstream/.agent/skills/cccl-test/references/libcudacxx.md b/cccl_upstream/.agent/skills/cccl-test/references/libcudacxx.md
new file mode 100644
index 00000000..14921f10
--- /dev/null
+++ b/cccl_upstream/.agent/skills/cccl-test/references/libcudacxx.md
@@ -0,0 +1,83 @@
+# libcudacxx Test
+
+## Organization
+
+- Put CUDA Standard Library tests under `libcudacxx/test/libcudacxx/std/...`.
+- Put CUDA-specific API tests under `libcudacxx/test/libcudacxx/cuda/...`, unless an adjacent `std/...` directory is clearly the established home for the functionality.
+- Read nearby tests first and mirror their directory layout, file names, helper types, includes, and lit gates.
+
+## Purpose
+
+- Validate libcudacxx functionality. It is fundamental to verify:
+
+  - Edge cases.
+  - Input and output types.
+  - Exception behavior.
+  - Runtime and constant-evaluation behavior.
+  - Device and host behavior.
+
+## Test kinds
+
+- `.pass.cpp`: compiles, links, runs, and returns 0.
+- `.compile.pass.cpp`: compiles correctly.
+- `.fail.cpp`: must fail compilation. Prefer precise `expected-error`, `expected-warning`, `expected-note`, or `expected-no-diagnostics` annotations when clang verify is supported.
+- `.runfail.cpp`: compiles and runs but must return non-zero.
+
+## Test structure
+
+- All tests must have the correct license banner.
+- Always include top level headers, never internal ones with `__` prefix.
+- Include support headers `"test_macros.h"`, `"test_iterators.h"`, `"test_comparisons.h"`, when needed.
+- Use `static_assert(...)` for compile-time guarantees and constexpr coverage.
+- Use `<cuda/std/cassert>` and `assert(...)` for runtime checks.
+- The `main` function must be present, dispatch runtime and static-evaluation tests, and return 0.
+
+## Style
+
+- Use `cuda::std` names, not `std::` names, unless the test is intentionally checking interoperability with host standard library types.
+- Do not fully qualify names in header includes unless the test is intentionally checking interoperability with host standard library types.
+- Mark helper functions that may run on host and device with `TEST_FUNC`; use `TEST_DEVICE_FUNC` for device-only helpers.
+  use `TEST_TILE_FUNC` for tile only helpers and `TEST_TILE_DEVICE_FUNC` for functions that can run on tile and device
+- `const`-qualification is discouraged.
+- Don't use `noexcept` for helper functions unless strictly necessary.
+- Do not use lambda expressions in host/device test code unless nearby tests already prove the pattern is supported.
+
+## Portability
+
+- Guard host-only or device-only behavior with `NV_IF_TARGET(NV_IS_HOST, (...))` and `NV_IF_TARGET(NV_IS_DEVICE, (...))` respectively.
+- Use `TEST_STD_VER`, `TEST_COMPILER`, `TEST_CUDA_COMPILER`, `TEST_HAS_EXCEPTIONS`, and `TEST_THROW` from `"test_macros.h"` instead of spelling compiler or dialect probes directly.
+- Unsupported platforms can be disabled with `UNSUPPORTED: <feature-name>` or `XFAIL: <feature-name>` lit directives. Some common feature names are `nvrtc`, `enable-tile`, `pre-sm-70`, `c++17`, `c++20`, `msvc`, `gcc-<version>`, or `clang-<version>`.
+
+  - Always motivate unsupported features with a comment.
+
+## Lit directives
+
+- Put lit directives near the top of the file before includes.
+- Common directives: `UNSUPPORTED:`, `XFAIL:`, `REQUIRES:`, `ADDITIONAL_COMPILE_DEFINITIONS:`, `ADDITIONAL_COMPILE_OPTIONS_HOST:`, `ADDITIONAL_COMPILE_OPTIONS_CUDA:`, `MODULES_DEFINES:`, and `CONSTEXPR_STEPS:`.
+- For diagnostics in `.fail.cpp`, annotate the exact line that should fail when possible:
+
+```cpp
+bad_expression(); // expected-error {{message fragment}}
+```
+
+- Prefer checking the intended diagnostic over accepting any compile failure.
+
+## Validation
+
+Use targeted libcudacxx lit runs. Paths passed to `--lit-precompile-tests` and `--lit-tests` are relative to `libcudacxx/test/libcudacxx/`.
+
+```bash
+ci/util/build_and_test_targets.sh \
+  --preset libcudacxx \
+  --lit-precompile-tests "std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp" \
+  --lit-tests "std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp"
+```
+
+If running lit directly, use the configured site file:
+
+```bash
+LIBCUDACXX_SITE_CONFIG=<path-to-cccl>/build/<preset>/libcudacxx/test/libcudacxx/lit.site.cfg \
+  lit -v libcudacxx/test/libcudacxx/<relative-test-path>
+```
+
+- Use `-Dexecutor=NoopExecutor()` for precompile-only validation when runtime execution is unavailable or GPU coverage is not required.
diff --git a/cccl_upstream/.agent/skills/sass-diff/SKILL.md b/cccl_upstream/.agent/skills/sass-diff/SKILL.md
new file mode 100644
index 00000000..4dd594e2
--- /dev/null
+++ b/cccl_upstream/.agent/skills/sass-diff/SKILL.md
@@ -0,0 +1,50 @@
+---
+name: sass-diff
+description: Use when asked to check for SASS (or PTX) changes between commits, branches, or a local changeset; guides normalization, comparison, and reporting of CUDA disassembly diffs.
+---
+
+# SASS Diffs
+
+Use this when asked to check for SASS changes between commits, branches or a local changeset.
+
+## Goal
+
+Detect relevant changes in generated CUDA machine code (i.e. SASS) while filtering noise from addresses, symbols, metadata, etc.
+Any non-trivial change must be detected.
+
+## Inputs to establish
+
+* Compilation target under test
+* The CUDA SM architectures to compile for. Try to detect this from the code and offer the user a list of suggestions.
+  The user must confirm or provide this list.
+* Baseline source (e.g. the previous commit/branch or the current commit without the changes in the working copy).
+* Comparison source (e.g.  the current commit/branch or the current commit with the changes in the working copy).
+* Whether a SASS (default) or PTX diff is requested.
+
+## Disassembly listing generation
+
+* Compile both, the baseline and comparison source, with the same compiler flags and options.
+  When not specified otherwise, lookup the options from `compile_commands.json`
+  or the current build system (i.e. CMake files).
+  Make sure the CUDA SM architectures (`CMAKE_CUDA_ARCHITECTURES`) are set to the user-provided or approved list.
+* Dump the disassembly from the binaries produced in the previous set using `cuobjdump -sass` or `cuobjdump -ptx`.
+
+## Comparison rules (what matters)
+
+Ignore as trivial:
+
+* Register renaming with identical instruction sequence and operands.
+* Pure label renumbering or reordering of identical basic blocks.
+* Formatting-only differences or reordered symbol tables.
+* Changes to symbol names (global function names)
+
+## Reporting
+
+* If any non-trivial change was detected, report the top 5 regions where a non-trivial change was detected,
+  including the name of the kernel they appeared in.
+* Provide a short summary of the diff type,
+  including opcode changes, memory access size/cache policy changes, control-flow changes, register-count changes,
+  spills/local memory, shared memory, and occupancy-relevant resource deltas.
+* Explicitly state if only noise was detected.
+* If you are not sure if the differences are impactful, show it and ask the user for guidance.
+* Keep the disassembly dumps available and tell the user where they can find them.
diff --git a/cccl_upstream/.clang-format b/cccl_upstream/.clang-format
new file mode 100644
index 00000000..bfcbb33c
--- /dev/null
+++ b/cccl_upstream/.clang-format
@@ -0,0 +1,205 @@
+# Note that we don't specify the language in this file because some files are
+# detected as Cpp, but others are detected as ObjC and we want this formatting
+# to apply to all types of files.
+BasedOnStyle: LLVM
+AccessModifierOffset: -2
+AlignAfterOpenBracket: Align
+AlignConsecutiveAssignments: Consecutive
+AlignConsecutiveBitFields: Consecutive
+AlignConsecutiveMacros: Consecutive
+AlignEscapedNewlines: Left
+AlignOperands: AlignAfterOperator
+AllowAllArgumentsOnNextLine: true
+AlignTrailingComments:
+  Kind: Never
+AllowAllParametersOfDeclarationOnNextLine: true
+AllowAllConstructorInitializersOnNextLine: true
+AllowShortBlocksOnASingleLine: false
+AllowShortCaseLabelsOnASingleLine: false
+AllowShortFunctionsOnASingleLine: Empty
+AllowShortIfStatementsOnASingleLine: Never
+AllowShortLambdasOnASingleLine: Empty
+AllowShortLoopsOnASingleLine: false
+AlwaysBreakAfterReturnType: None
+AlwaysBreakTemplateDeclarations: Yes
+AttributeMacros: [
+                  '_CCCL_ALIGNAS_TYPE',
+                  '_CCCL_ALIGNAS',
+                  '_CCCL_CONSTEXPR_CXX20',
+                  '_CCCL_CONSTEXPR_CXX23',
+                  '_CCCL_DECLSPEC_EMPTY_BASES',
+                  '_CCCL_DEVICE',
+                  '_CCCL_FORCEINLINE',
+                  '_CCCL_HIDE_FROM_ABI',
+                  '_CCCL_HOST_DEVICE',
+                  '_CCCL_DEDUCTION_GUIDE_ATTRIBUTES',
+                  '_CCCL_HOST',
+                  '_CCCL_KERNEL_ATTRIBUTES',
+                  '_CCCL_NO_UNIQUE_ADDRESS',
+                  '_CCCL_TYPE_VISIBILITY_DEFAULT',
+                  '_CCCL_TYPE_VISIBILITY_HIDDEN',
+                  '_CCCL_VISIBILITY_HIDDEN',
+                  '_CCCL_LAUNCH_BOUNDS',
+                  '_CCCL_BLOCK_SIZE',
+                  'CUB_RUNTIME_FUNCTION',
+                  'THRUST_RUNTIME_FUNCTION',
+                  'CCCL_DEPRECATED',
+                  'CCCL_DEPRECATED_BECAUSE',
+                  '_CCCL_DEPRECATED_IN_CXX20',
+                  '_CCCL_DEPRECATED_IN_CXX23',
+                  '_CCCL_API',
+                  '_CCCL_HOST_API',
+                  '_CCCL_DEVICE_API',
+                  '_CCCL_NODEBUG_API',
+                  '_CCCL_NODEBUG_HOST_API',
+                  '_CCCL_NODEBUG_DEVICE_API',
+                  '_CCCL_TRIVIAL_API',
+                  '_CCCL_TRIVIAL_HOST_API',
+                  '_CCCL_TRIVIAL_DEVICE_API',
+                  '_CCCL_PUBLIC_API',
+                  '_CCCL_PUBLIC_HOST_API',
+                  '_CCCL_PUBLIC_DEVICE_API',
+                 ]
+BinPackArguments: false
+BinPackParameters: false
+BreakBeforeBraces: Custom
+BraceWrapping:
+  AfterCaseLabel: false
+  AfterClass: true
+  AfterControlStatement: true
+  AfterEnum: true
+  AfterFunction: true
+  AfterNamespace: true
+  AfterStruct: true
+  AfterUnion: true
+  BeforeCatch: true
+  BeforeElse: true
+  IndentBraces: false
+  SplitEmptyFunction: false
+  SplitEmptyRecord: false
+BreakBeforeConceptDeclarations: true
+BreakBeforeBinaryOperators: NonAssignment
+BreakBeforeTernaryOperators: true
+BreakConstructorInitializers: BeforeComma
+BreakInheritanceList: BeforeComma
+ColumnLimit: 120
+CompactNamespaces: false
+ContinuationIndentWidth: 2
+EmptyLineAfterAccessModifier: Never
+EmptyLineBeforeAccessModifier: Always
+FixNamespaceComments: true
+IfMacros: [
+  '_CUB_WEAKEN_IF_CONSTEXPR_IF_COMPILED_FOR_CCCL_C',
+  '_CCCL_CATCH'
+]
+IndentWrappedFunctionNames: false
+IncludeBlocks:   Regroup
+IncludeCategories:
+  - Regex:           '^<cuda/std/__cccl/prologue.h>'
+    Priority:            0x7FFFFFFF
+    SortPriority:        0x7FFFFFFF
+  - Regex:           '^<cuda/std/__cccl/epilogue.h>'
+    Priority:            -0x7FFFFFFF
+    SortPriority:        -0x7FFFFFFF
+  - Regex:           '^<cuda/experimental/__execution/prologue.cuh>'
+    Priority:            0x7FFFFFFF
+    SortPriority:        0x7FFFFFFF
+  - Regex:           '^"insert_nested_NVTX_range_guard.h"'
+    Priority:            -1
+    SortPriority:        -1
+  - Regex:           '^<(cuda/std/detail/__config|cub/config.cuh|thrust/detail/config.h|thrust/system/cuda/config.h)'
+    Priority:            0
+    SortPriority:        0
+  - Regex:           '^<cub/'
+    Priority:            2
+    SortPriority:        1
+  - Regex:           '^<thrust/'
+    Priority:            3
+    SortPriority:        2
+  - Regex:           '^<cuda/experimental'
+    Priority:            5
+    SortPriority:        4
+  - Regex:           '^<cuda/'
+    Priority:            4
+    SortPriority:        3
+  - Regex:           '^<nv/'
+    Priority:            6
+    SortPriority:        5
+  - Regex:           '^<[a-z_]*>$'
+    Priority:            7
+    SortPriority:        6
+  - Regex:           '^<[a-z_]*\.[a-z]+>$'
+    Priority:            8
+    SortPriority:        7
+  - Regex:           '^<cuda'
+    Priority:            0
+    SortPriority:        0
+InsertBraces: true
+IndentCaseLabels: true
+InsertNewlineAtEOF: true
+InsertTrailingCommas: Wrapped
+IndentRequires: true
+IndentPPDirectives: AfterHash
+IndentWidth: 2
+KeepEmptyLines:
+  AtEndOfFile: true
+  AtStartOfBlock: false
+  AtStartOfFile: false
+MaxEmptyLinesToKeep: 1
+Macros:
+- _CCCL_TEMPLATE(...)=template<...>
+- _CCCL_REQUIRES(...)=requires (...)
+- _CUDAX_SEMI_PRIVATE=private
+NamespaceIndentation: None
+PackConstructorInitializers: Never
+PenaltyBreakAssignment: 30
+PenaltyBreakBeforeFirstCallParameter: 50
+PenaltyBreakComment: 0
+PenaltyBreakFirstLessLess: 0
+PenaltyBreakString: 70
+PenaltyBreakTemplateDeclaration: 0
+PenaltyExcessCharacter: 100
+PenaltyReturnTypeOnItsOwnLine: 90
+PenaltyIndentedWhitespace: 2
+PointerAlignment: Left
+ReflowComments: true
+RemoveSemicolon: false
+SortIncludes: CaseInsensitive
+SpaceAfterCStyleCast: true
+SpaceAfterLogicalNot: false
+SpaceAfterTemplateKeyword: true
+SpaceBeforeAssignmentOperators: true
+SpaceBeforeCpp11BracedList: false
+SpaceBeforeCtorInitializerColon: true
+SpaceBeforeInheritanceColon: true
+SpaceBeforeParens: ControlStatements
+SpaceBeforeRangeBasedForLoopColon: true
+SpaceInEmptyParentheses: false
+SpacesBeforeTrailingComments: 1
+SpacesInAngles: Never
+SpacesInCStyleCastParentheses: false
+SpacesInParentheses: false
+SpacesInSquareBrackets: false
+Standard: c++20
+StatementMacros: [
+  '_CCCL_EXEC_CHECK_DISABLE',
+  'CUB_NAMESPACE_BEGIN',
+  'CUB_NAMESPACE_END',
+  'THRUST_NAMESPACE_BEGIN',
+  'THRUST_NAMESPACE_END',
+  '_CCCL_BEGIN_NAMESPACE_CUDA_STD',
+  '_CCCL_END_NAMESPACE_CUDA_STD',
+  '_CCCL_BEGIN_NAMESPACE_CUDA_STD_NOVERSION',
+  '_CCCL_END_NAMESPACE_CUDA_STD_NOVERSION',
+  '_CCCL_BEGIN_NAMESPACE_CUDA_STD_RANGES',
+  '_CCCL_END_NAMESPACE_CUDA_STD_RANGES',
+  '_CCCL_BEGIN_NAMESPACE_CUDA_STD_RANGES_ABI',
+  '_CCCL_END_NAMESPACE_CUDA_STD_RANGES_ABI',
+  '_CCCL_BEGIN_NAMESPACE_CUDA_STD_VIEWS',
+  '_CCCL_END_NAMESPACE_CUDA_STD_VIEWS',
+  '_CCCL_BEGIN_NAMESPACE_CPO',
+  '_CCCL_END_NAMESPACE_CPO',
+]
+TabWidth: 2
+UseTab: Never
+WrapNamespaceBodyWithEmptyLines: Never
diff --git a/cccl_upstream/.clang-tidy b/cccl_upstream/.clang-tidy
new file mode 100644
index 00000000..ea9e95c4
--- /dev/null
+++ b/cccl_upstream/.clang-tidy
@@ -0,0 +1,324 @@
+# SPDX-FileCopyrightText: Copyright (c) 2024-2026 NVIDIA CORPORATION & AFFILIATES.
+#                         All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+---
+Checks:
+  # enable everything first
+  - 'performance-*'
+  - 'modernize-*'
+  - 'readability-*'
+  - 'clang-analyzer-*'
+  - 'clang-diagnostic-*'
+  - 'bugprone-*'
+  - 'misc-*'
+  - 'core-*'
+  - 'mpi-*'
+  - 'cert-*'
+  - 'portability-*'
+  - 'google-*'
+  - 'cppcoreguidelines-*'
+  - 'concurrency-*'
+  # TODO: BEGIN REMOVE ME
+  - '-*'
+  - 'performance-*'
+  - '-performance-enum-size'
+  - '-performance-unnecessary-value-param'
+  - 'modernize-*'
+  - '-modernize-use-integer-sign-comparison'
+  - '-modernize-use-nodiscard'
+  - '-modernize-return-braced-init-list'
+  - '-modernize-use-auto'
+  - 'clang-diagnostic-*'
+  - 'bugprone-*'
+  - '-bugprone-crtp-constructor-accessibility'
+  - '-bugprone-argument-comment'
+  # This is just noise
+  - '-bugprone-signed-char-misuse'
+  - '-bugprone-multi-level-implicit-pointer-conversion'
+  # END REMOVE ME
+  # TODO(jfaibussowit):
+  #
+  # Enable these once we move to C++20. Normally, clang-tidy does not suggest checks for
+  # cpp N+1 if you are compiling with -std=c++N. However some subprojects like c.parallel
+  # actually require cpp N+1 so clang-tidy emits diagnostics for any headers included by
+  # it.
+  - '-modernize-use-designated-initializers'
+  - '-modernize-use-constraints'
+  - '-modernize-use-ranges'
+  # Irrelevant for GPU code
+  - '-modernize-pass-by-value'
+  # HICPP is 99% aliased to other checks (mostly modernize-* and bugprone-*). We don't
+  # want to also enable it because then we need to duplicate the NOLINT. The only 2 checks
+  # that aren't aliases are:
+  #
+  # - hicpp-multiway-paths-covered. This check is useful for detecting degenerate if-else
+  #   branches, but the switch cases are IMO better handled by -Wswitch. To make -Wswitch
+  #   even stronger, we should ban default: cases entirely,
+  #
+  # - hicpp-signed-bitwise. This check is also covered by
+  #   clang.llvm.org/extra/clang-tidy/checks/clang-analyzer/core.BitwiseShift.html
+  #
+  # - 'hicpp-*'
+  #
+  # LLVM checks are extremely specific to the LLVM project and aren't suitable for
+  # downstream users.
+  #
+  # - 'llvm-*'
+  #
+  # then disable the stuff we don't want
+  - '-cert-dcl21-cpp'  # returning non-const from operator-- or operator++
+  - '-cert-dcl50-cpp'  # allow c-style variadics
+  # No reserved identifiers, both of these are aliased to bugprone-reserved-identifier,
+  # which we do enable. Leaving these enabled however, leads to needing to specify all
+  # three (bugprone-reserved-identifier, cert-dcl51-cpp, and cert-dcl37-c) in NOLINT lines
+  # which is a hassle. Since bugprone-reserved-identifier is enabled, the check still
+  # fires.
+  - '-cert-dcl51-cpp,-cert-dcl37-c,-cert-oop54-cpp'
+  # Covered by bugprone-throwing-static-initialization
+  - '-cert-err58-cpp'
+  - '-modernize-use-trailing-return-type'
+  - '-readability-function-cognitive-complexity'
+  - '-readability-implicit-bool-conversion'
+  - '-readability-braces-around-statements'
+  - '-readability-qualified-auto'
+  - '-readability-isolate-declaration'
+  - '-modernize-avoid-c-arrays'
+  - '-cppcoreguidelines-avoid-c-arrays'
+  - '-readability-named-parameter'
+  - '-readability-identifier-length'
+  - '-misc-non-private-member-variables-in-classes'
+  - '-bugprone-easily-swappable-parameters'
+  - '-bugprone-implicit-widening-of-multiplication-result'
+  - '-misc-include-cleaner'
+  - '-misc-header-include-cycle'
+  - '-modernize-macro-to-enum'
+  - '-cppcoreguidelines-macro-to-enum'
+  - '-misc-no-recursion'
+  - '-bugprone-dynamic-static-initializers'
+  - '-portability-avoid-pragma-once'
+  # Generally speaking if something is static then it has an express reason to be. 99% of
+  # the candidates identified by this check were because they just so happened not to
+  # touch any member variables, not because they are logically static. So we disable the
+  # check.
+  - '-readability-convert-member-functions-to-static'
+  # When running in a conda env, ignore options that may have been added by conda but are unused
+  - '-clang-diagnostic-unused-command-line-argument'
+  # Given
+  #
+  # std::make_pair<some_type, some_other_type>(...)
+  #
+  # Results in: error: for C++11-compatibility, use pair directly. But we don't care about
+  # C++11, and so we don't care about this warning.
+  - '-google-build-explicit-make-pair'
+  # This check is incredibly expensive for absolutely no reason, and since we a) use
+  # modern google-test and b) don't have clang-tidy enabled on our testing code, we don't
+  # need to enable it!
+  - '-google-upgrade-googletest-case'
+  # This warning will warn if you have redundant default-initializers for class
+  # members. For example:
+  #
+  # class Foo
+  # {
+  #   int x_{}; // WARNING: redundant initializer here
+  #
+  # public:
+  #   Foo(int x) : x_{x} { }
+  # };
+  #
+  # Since Foo can only ever be constructed with an explicit value for x_ via its constructor,
+  # the default initializer is technically redundant. However, if we change the definition
+  # of Foo to now allow a default ctor, then the initializer becomes non-redundant
+  # again. It is easier to just follow the rule of "always explicitly default initialize
+  # members" than to remember to change 2 places at once.
+  - '-readability-redundant-member-init'
+  # Alias for readability-enum-initial-value, disable this one because the readability-
+  # name is easier to understand, and we don't want to silence 2 things for the same
+  # warning.
+  - '-cert-int09-c'
+  # This one is potentially controversial. This check warns when iterating over unordered
+  # containers of pointers:
+  #
+  # {
+  #   int a = 1, b = 2;
+  #   std::unordered_set<int *> set = {&a, &b};
+  #
+  #   for (auto *i : set) { // iteration order not deterministic
+  #     f(i);
+  #   }
+  # }
+  #
+  # On the one hand, this is a clear case of non-determinism. But on the other hand, I
+  # feel it is obvious that a user does not care about the order of iteration
+  # because... they are using unordered containers!
+  - '-bugprone-nondeterministic-pointer-iteration-order'
+  - '-cppcoreguidelines-pro-type-reinterpret-cast'
+  # We don't use GSL
+  - '-cppcoreguidelines-owning-memory'
+  # Covered by modernize-use-override already
+  - '-cppcoreguidelines-explicit-virtual-functions'
+  # Covered by readability-magic-numbers
+  - '-cppcoreguidelines-avoid-magic-numbers'
+  # This check does more harm than good, as it appears to be very rudimentary. For
+  # example, it emits warnings saying that:
+  #
+  # #define DEFINE_OPERATOR(op)                            \
+  #   _CCCL_HOST_DEVICE custom_numeric operator op() const \
+  #   {                                                    \
+  #     return custom_numeric(op value[0]);                \
+  #   }
+  #
+  # Should be replaced by a template function, but this is not possible. We instead rely
+  # on best judgment of reviewers to catch macros that can be functions.
+  - '-cppcoreguidelines-macro-usage'
+  # This check does not understand more complex macro usage where brackets are not allowed
+  #
+  # error: macro replacement list should be enclosed in parentheses [bugprone-macro-parentheses]
+  # 36 | #define __THRUST_HOST_SYSTEM_INCLUDE(filename) <__THRUST_HOST_SYSTEM_ROOT/filename>
+  #    |                                                                           ^
+  #    |                                                                           (       )
+  #
+  # So better left disabled.
+  - '-bugprone-macro-parentheses'
+
+  # TODO: Enable for clang-tidy-22
+  - '-modernize-avoid-c-style-cast'
+  - '-bugprone-signed-bitwise'
+  - '-bugprone-std-namespace-modification'
+  - '-clang-diagnostic-deprecated-attributes'
+  - '-bugprone-throwing-static-initialization'
+  - '-bugprone-unchecked-string-to-number-conversion'
+  - '-bugprone-invalid-enum-default-initialization'
+  - '-bugprone-derived-method-shadowing-base-method'
+  - '-bugprone-unsafe-to-allow-exceptions'
+  - '-bugprone-sizeof-expression'
+  - '-modernize-use-nullptr'
+  - '-bugprone-std-exception-baseclass'
+  - '-modernize-avoid-variadic-functions'
+  - '-modernize-type-traits'
+  - '-bugprone-random-generator-seed'
+  - '-bugprone-command-processor'
+
+  # TODO: ironically enough, enable this someday
+  - '-google-readability-todo'
+  # Covered by modernize-avoid-c-style-cast, and also is buggier. It seems to fire on
+  # `_v`-style constexpr bools as well sometimes.
+  - '-google-readability-casting'
+  # Covered by performance-noexcept-move-constructor
+  - '-cppcoreguidelines-noexcept-move-operations'
+  # REVIEW ME: This warns about any usage of operator[], suggesting usage of .at() instead. Could
+  - '-cppcoreguidelines-pro-bounds-avoid-unchecked-container-access'
+  - '-cppcoreguidelines-narrowing-conversions'
+  # Covered by misc-unconventional-assign-operator
+  - '-cppcoreguidelines-c-copy-assignment-signature'
+  # Covered by performance-noexcept-swap
+  - '-cppcoreguidelines-noexcept-swap'
+  # Covered by modernize-use-default-member-init
+  - '-cppcoreguidelines-use-default-member-init'
+  # clang-tidy documentation says that for a given file, it matches the closest
+  # .clang-tidy up the directory stack and applies the configuration from it. However, by
+  # "it" `clang-tidy` means the closest `.clang-tidy` to the *source* file, not the file
+  # that was included by the source file.
+  #
+  # So `cuda/std/foo.h` included by `cccl/foo/bar/baz.cpp` will never "match" against
+  # `cccl/libcudacxx/.clang-tidy` because `clang-tidy` finds `cccl/.clang-tidy` instead.
+  #
+  # So we disable this for now. In the future we should add a custom check that bans
+  # reserved identifiers except symbols from libcu++.
+  - '-bugprone-reserved-identifier'
+WarningsAsErrors: '*'
+HeaderFileExtensions:
+  - ''
+  - h
+  - hh
+  - hpp
+  - hxx
+  - cuh
+  - inl
+  - ipp
+ImplementationFileExtensions:
+  - c
+  - cc
+  - cpp
+  - cxx
+  - cu
+SystemHeaders: false
+HeaderFilterRegex: '.*/(thrust|cub|cuda|nv|cccl|c2h|cudastf|test)/.*'
+ExtraArgsBefore:
+  # To match _CCCL_DOXYGEN_INVOKED
+  - '-D_CCCL_CLANG_TIDY_INVOKED=1'
+  - '-ftemplate-backtrace-limit=0'
+  - '-fmacro-backtrace-limit=0'
+ExtraArgs:
+  # These must come in ExtraArgs not ExtraArgsBefore, because they are overriding the
+  # effects of -Wall and -Werror, where the last flag "wins".
+  - '-Wno-unknown-warning-option'
+  - '-Wno-unknown-cuda-version'
+  - '-Wno-error=unused-command-line-argument'
+CheckOptions:
+  cert-dcl16-c.NewSuffixes: 'L;LL;LU;LLU'
+  cert-err33-c.CheckedFunctions: '::aligned_alloc;::asctime_s;::at_quick_exit;::atexit;::bsearch;::bsearch_s;::btowc;::c16rtomb;::c32rtomb;::calloc;::clock;::cnd_broadcast;::cnd_init;::cnd_signal;::cnd_timedwait;::cnd_wait;::ctime_s;::fclose;::fflush;::fgetc;::fgetpos;::fgets;::fgetwc;::fopen;::fopen_s;::fprintf;::fprintf_s;::fputc;::fputs;::fputwc;::fputws;::fread;::freopen;::freopen_s;::fscanf;::fscanf_s;::fseek;::fsetpos;::ftell;::fwprintf;::fwprintf_s;::fwrite;::fwscanf;::fwscanf_s;::getc;::getchar;::getenv;::getenv_s;::gets_s;::getwc;::getwchar;::gmtime;::gmtime_s;::localtime;::localtime_s;::malloc;::mbrtoc16;::mbrtoc32;::mbsrtowcs;::mbsrtowcs_s;::mbstowcs;::mbstowcs_s;::memchr;::mktime;::mtx_init;::mtx_lock;::mtx_timedlock;::mtx_trylock;::mtx_unlock;::printf_s;::putc;::putwc;::raise;::realloc;::remove;::rename;::scanf;::scanf_s;::setlocale;::setvbuf;::signal;::snprintf;::snprintf_s;::sprintf;::sprintf_s;::sscanf;::sscanf_s;::strchr;::strerror_s;::strftime;::strpbrk;::strrchr;::strstr;::strtod;::strtof;::strtoimax;::strtok;::strtok_s;::strtol;::strtold;::strtoll;::strtoul;::strtoull;::strtoumax;::strxfrm;::swprintf;::swprintf_s;::swscanf;::swscanf_s;::thrd_create;::thrd_detach;::thrd_join;::thrd_sleep;::time;::timespec_get;::tmpfile;::tmpfile_s;::tmpnam;::tmpnam_s;::tss_create;::tss_get;::tss_set;::ungetc;::ungetwc;::vfprintf;::vfprintf_s;::vfscanf;::vfscanf_s;::vfwprintf;::vfwprintf_s;::vfwscanf;::vfwscanf_s;::vprintf_s;::vscanf;::vscanf_s;::vsnprintf;::vsnprintf_s;::vsprintf;::vsprintf_s;::vsscanf;::vsscanf_s;::vswprintf;::vswprintf_s;::vswscanf;::vswscanf_s;::vwprintf_s;::vwscanf;::vwscanf_s;::wcrtomb;::wcschr;::wcsftime;::wcspbrk;::wcsrchr;::wcsrtombs;::wcsrtombs_s;::wcsstr;::wcstod;::wcstof;::wcstoimax;::wcstok;::wcstok_s;::wcstol;::wcstold;::wcstoll;::wcstombs;::wcstombs_s;::wcstoul;::wcstoull;::wcstoumax;::wcsxfrm;::wctob;::wctrans;::wctype;::wmemchr;::wprintf_s;::wscanf;::wscanf_s;'
+  llvm-else-after-return.WarnOnUnfixable: 'false'
+  cert-str34-c.DiagnoseSignedUnsignedCharComparisons: 'false'
+  cppcoreguidelines-non-private-member-variables-in-classes.IgnoreClassesWithAllMemberVariablesBeingPublic: 'true'
+  google-readability-braces-around-statements.ShortStatementLines: '1'
+  llvm-qualified-auto.AddConstToQualified: 'false'
+  llvm-else-after-return.WarnOnConditionVariables: 'false'
+  cert-oop54-cpp.WarnOnlyIfThisHasSuspiciousField: 'false'
+  performance-move-const-arg.CheckTriviallyCopyableMove: 'false'
+  performance-inefficient-string-concatenation.StrictMode: 'true'
+  readability-simplify-boolean-expr.ChainedConditionalReturn: 'true'
+  readability-simplify-boolean-expr.ChainedConditionalAssignment: 'true'
+  bugprone-dangling-handle.HandleClasses: '::cuda::std::span;::cuda::std::mdspan'
+  bugprone-unused-return-value.AllowCastToVoid: 'true'
+  readability-enum-initial-value.AllowExplicitZeroFirstInitialValue: 'false'
+  readability-enum-initial-value.AllowExplicitSequentialInitialValues: 'false'
+  readability-redundant-access-specifiers.CheckFirstDeclaration: 'true'
+  bugprone-lambda-function-name.IgnoreMacros: 'true'
+  # readability-identifier-naming.ClassCase: 'lower_case'
+  # readability-identifier-naming.UnionCase: 'lower_case'
+  # readability-identifier-naming.ClassConstantCase: 'UPPER_CASE'
+  # readability-identifier-naming.ClassIgnoredRegexp: 'tuple|has_.*|is_.*|as_.*|.*_tag|tag|.*_of'
+  # readability-identifier-naming.ConstantMemberCase: 'UPPER_CASE'
+  # readability-identifier-naming.ConstantMemberIgnoredRegexp: 'value'
+  # readability-identifier-naming.EnumCase: 'lower_case'
+  # readability-identifier-naming.EnumConstantCase: 'UPPER_CASE'
+  # readability-identifier-naming.FunctionCase: 'lower_case'
+  # readability-identifier-naming.GlobalConstantCase: 'UPPER_CASE'
+  # readability-identifier-naming.GlobalConstantIgnoredRegexp: '.*_v'
+  # readability-identifier-naming.LocalVariableCase: 'lower_case'
+  # # We want to allow constexpr auto MY_VAL1
+  # readability-identifier-naming.LocalVariableIgnoredRegexp: '[A-Z_0-9]+'
+  # readability-identifier-naming.MacroDefinitionCase: 'UPPER_CASE'
+  # # We want to allow MY_MACRO_PRIVATE_1_
+  # readability-identifier-naming.MacroDefinitionIgnoredRegexp: '[A-Z_0-9]+'
+  # readability-identifier-naming.NamespaceCase: 'lower_case'
+  # readability-identifier-naming.PrivateMemberCase: 'lower_case'
+  # readability-identifier-naming.PrivateMemberSuffix: '_'
+  # readability-identifier-naming.PrivateMethodCase: 'lower_case'
+  # readability-identifier-naming.PrivateMethodSuffix: '_'
+  # readability-identifier-naming.ProtectedMemberCase: 'lower_case'
+  # readability-identifier-naming.ProtectedMemberSuffix: '_'
+  # readability-identifier-naming.ProtectedMethodCase: 'lower_case'
+  # readability-identifier-naming.ProtectedMethodSuffix: '_'
+  # readability-identifier-naming.PublicMethodCase: 'lower_case'
+  # readability-identifier-naming.ScopedEnumCase: 'lower_case'
+  # readability-identifier-naming.ScopedEnumConstantCase: 'UPPER_CASE'
+  readability-magic-numbers.IgnoredIntegerValues: '0;1;2;3;4;5;6;7;8;9'
+  # There are some single-argument functions that we would not like to ignore
+  # (`to_string(bool provenance = false)` comes to mind), but setting this to false makes
+  # clang-tidy warn on a bunch of "obvious" calls, like `set_dim(1)` or
+  # `with_concurrent(true)`.
+  bugprone-argument-comment.IgnoreSingleArgument: true
+  bugprone-argument-comment.CommentBoolLiterals: true
+  bugprone-argument-comment.CommentIntegerLiterals: true
+  bugprone-argument-comment.CommentFloatLiterals: true
+  bugprone-argument-comment.CommentStringLiterals: true
+  bugprone-argument-comment.CommentCharacterLiterals: true
+  bugprone-argument-comment.CommentUserDefinedLiterals: true
+  bugprone-argument-comment.CommentNullPtrs: true
+  cppcoreguidelines-macro-usage.AllowedRegexp: '^DEBUG_.*|THRUST_PP_.*|_CCCL.*_PP_.*'
+  bugprone-reserved-identifier.AllowedIdentifiers: '^_CCCL.*'
+  performance-enum-size.EnumIgnoreList: '::cuda::std::.*'
+  performance-unnecessary-value-param.AllowedTypes: '.*(_ref(erence)?|_ptr|pointer|(_)?[Ii]t(erator)?|IteratorT|__half(2)?|(__nv_bfloat.*)|_view)$'
+  bugprone-exception-escape.CheckMain: false
+...
diff --git a/cccl_upstream/.pre-commit-config.yaml b/cccl_upstream/.pre-commit-config.yaml
new file mode 100644
index 00000000..eb6c68b2
--- /dev/null
+++ b/cccl_upstream/.pre-commit-config.yaml
@@ -0,0 +1,143 @@
+# Copyright (c) 2024, NVIDIA CORPORATION.
+ci:
+    autofix_commit_msg: |
+      [pre-commit.ci] auto code formatting
+    autofix_prs: false
+    autoupdate_branch: ''
+    autoupdate_commit_msg: '[pre-commit.ci] pre-commit autoupdate'
+    autoupdate_schedule: quarterly
+    skip: [mypy, secret-scan-trufflehog] # mypy fails on pre-commit.ci due to numpy stub incompatibility with Python 3.14
+    submodules: false
+
+repos:
+  # Runs first so a leaked credential blocks the commit before any formatter runs.
+  # Self-installing: the hook downloads a pinned, checksum-verified trufflehog on
+  # first use (no manual install). Skipped on pre-commit.ci; Pulse CI enforces server-side.
+  - repo: https://github.com/NVIDIA/security-workflows
+    rev: v0.2.0
+    hooks:
+      - id: secret-scan-trufflehog
+
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v5.0.0
+    hooks:
+      - id: end-of-file-fixer
+      - id: mixed-line-ending
+      - id: trailing-whitespace
+        # handled by clang-format
+        exclude_types: [c, c++, cuda]
+      - id: check-json
+      - id: check-toml
+      - id: pretty-format-json
+        args: ['--autofix', '--indent=2', '--no-sort-keys']
+      - id: check-symlinks
+      - id: check-executables-have-shebangs
+      - id: check-merge-conflict
+      - id: check-shebang-scripts-are-executable
+        # ruff checks this already
+        exclude_types: [python]
+      - id: check-yaml
+
+  - repo: https://github.com/shellcheck-py/shellcheck-py
+    rev: v0.11.0.1
+    hooks:
+      - id: shellcheck
+        exclude: |
+          (?x)^(
+            ^.*libcudacxx/cmake/config\.guess$
+          )
+
+  - repo: https://github.com/pre-commit/mirrors-clang-format
+    rev: v20.1.7
+    hooks:
+      - id: clang-format
+        types_or: [file]
+        files: |
+          (?x)^(
+            ^.*\.c$|
+            ^.*\.cpp$|
+            ^.*\.cu$|
+            ^.*\.cuh$|
+            ^.*\.cxx$|
+            ^.*\.h$|
+            ^.*\.hpp$|
+            ^.*\.inl$|
+            ^.*\.mm$|
+            ^libcudacxx/include/.*/[^.]*$
+          )
+        args: ["-fallback-style=none", "-style=file", "-i"]
+
+  # TODO/REMINDER: add the Ruff vscode extension to the devcontainers
+  # Ruff, the Python auto-correcting linter/formatter written in Rust
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.12.2
+    hooks:
+    - id: ruff  # linter
+    - id: ruff-format  # formatter
+
+  # CMake formatting
+  - repo: https://github.com/BlankSpruce/gersemi
+    rev: 0.23.1
+    hooks:
+    - id: gersemi
+
+  - repo: https://github.com/codespell-project/codespell
+    rev: v2.4.1
+    hooks:
+      - id: codespell
+        additional_dependencies: [tomli]
+        args: ["--toml", "pyproject.toml", "-I", ".codespell-ignore.txt"]
+        exclude: |
+          (?x)^(
+            build|
+            CITATION.md
+          )
+
+  - repo: https://github.com/pre-commit/mirrors-mypy
+    rev: 'v1.16.1'
+    hooks:
+      - id: mypy
+        # numpy is intentionally not installed here: numpy>=2.3 ships PEP 695
+        # `type` statements in its stubs that mypy rejects under
+        # python_version=3.10, and numpy<2.3 has no wheels for the Python that
+        # pre-commit.ci runs. numpy is instead ignored in the mypy config.
+        additional_dependencies: [types-cachetools]
+        args: ["--config-file=python/cuda_cccl/pyproject.toml",
+               "python/cuda_cccl/cuda/compute/"]
+        pass_filenames: false
+
+  - repo: https://github.com/sirosen/texthooks
+    rev: 0.7.1
+    hooks:
+      - id: fix-smartquotes
+      - id: fix-spaces
+      - id: forbid-bidi-controls
+      - id: fix-ligatures
+
+  - repo: local
+    hooks:
+      - id: check-shebang
+        name: check-shebang
+        entry: ci/util/pre-commit/check_shebang.py
+        language: python
+        types: [shell]
+        exclude: |
+          (?x)^(
+            ^.*libcudacxx/cmake/config\.guess$
+          )
+
+      - id: check-cub-test-macros
+        name: require CUB test memory classification
+        entry: ci/util/pre-commit/check_cub_test_macros.py
+        language: python
+        files: ^(cub/test/.*\.(cu|cuh|h)|ci/util/pre-commit/check_cub_test_macros\.py)$
+        types: [file]
+
+      - id: unprintable-unicode
+        name: unprintable-unicode
+        entry: ci/util/pre-commit/strip_unprintable.py
+        language: python
+        types: [text]
+
+default_language_version:
+  python: python3
diff --git a/cccl_upstream/CITATION.md b/cccl_upstream/CITATION.md
new file mode 100644
index 00000000..de886a97
--- /dev/null
+++ b/cccl_upstream/CITATION.md
@@ -0,0 +1,12 @@
+# Citation Guide
+
+## To Cite CCCL
+If you use CCCL in a publication, please use citations in the following format (BibTeX entry for LaTeX):
+```tex
+@Manual{,
+  title = {{CCCL}: {CUDA} {C++} {C}ore {L}ibraries},
+  author = {{CCCL Development Team}},
+  year = {2023},
+  url = {https://github.com/NVIDIA/cccl},
+}
+```
diff --git a/cccl_upstream/CLAUDE.md b/cccl_upstream/CLAUDE.md
new file mode 120000
index 00000000..47dc3e3d
--- /dev/null
+++ b/cccl_upstream/CLAUDE.md
@@ -0,0 +1 @@
+AGENTS.md
\ No newline at end of file
diff --git a/cccl_upstream/CODE_OF_CONDUCT.md b/cccl_upstream/CODE_OF_CONDUCT.md
new file mode 100644
index 00000000..c09b71de
--- /dev/null
+++ b/cccl_upstream/CODE_OF_CONDUCT.md
@@ -0,0 +1,84 @@
+# Contributor Covenant Code of Conduct
+
+## Overview
+
+Define the code of conduct followed and enforced for CCCL.
+
+### Intended audience
+
+Community | Developers | Project Leads
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+  advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting GitHub_Conduct@nvidia.com. All complaints will be reviewed and
+investigated and will result in a response that is deemed necessary and appropriate
+to the circumstances. The project team is obligated to maintain confidentiality with
+regard to the reporter of an incident. Further details of specific enforcement policies
+may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq
diff --git a/cccl_upstream/CONTRIBUTING.md b/cccl_upstream/CONTRIBUTING.md
new file mode 100644
index 00000000..ce9efa07
--- /dev/null
+++ b/cccl_upstream/CONTRIBUTING.md
@@ -0,0 +1,281 @@
+
+# Contributing to CCCL
+
+Thank you for your interest in contributing to the CUDA Core Compute Libraries (CCCL)!
+
+Looking for ideas for your first contribution? Check out: ![GitHub Issues or Pull Requests by label](https://img.shields.io/github/issues/nvidia/cccl/good%20first%20issue)
+
+
+## Getting Started
+
+1. **Fork & Clone the Repository**:
+
+   Fork the [CCCL GitHub Repository](https://github.com/nvidia/cccl) and clone the fork. For more information, check [GitHub's documentation on forking](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo) and [cloning a repository](https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/cloning-a-repository).
+
+2. **Set up Development Environment**:
+
+   CCCL uses Development Containers to provide a consistent development environment for both local development and CI. Contributors are strongly encouraged to use these containers as they simplify environment setup. See the [Dev Containers guide](.devcontainer/README.md) for instructions on how to quickly get up and running using dev containers with or without VSCode.
+
+## Making Changes
+
+1. **Create a New Branch**:
+
+   ```bash
+   git checkout -b your-feature-branch
+   ```
+
+2. **Make Changes**.
+
+3. **Build and Test**:
+
+   Ensure changes don't break existing functionality by building and running tests.
+
+   ```bash
+   ./ci/build_[thrust|cub|libcudacxx].sh -cxx <HOST_COMPILER> -std <CXX_STANDARD> -arch <GPU_ARCHS>
+
+   # test implies build
+   ./ci/test_[thrust|cub|libcudacxx].sh  -cxx <HOST_COMPILER> -std <CXX_STANDARD> -arch <GPU_ARCHS>
+   ```
+
+   For more details on building and testing, refer to the [Building and Testing](#building-and-testing) section below.
+
+4. **Commit Changes**:
+
+   ```bash
+   git commit -m "Brief description of the change"
+   ```
+
+### Developer Guides
+
+For more information about design and development practices for each CCCL component, refer to the following developer guides:
+
+#### CUB
+
+- [CUB Developer Guide](docs/cub/developer_overview.rst) - General overview of the design of CUB internals
+- [CUB Tests](docs/cub/test_overview.rst) - Overview of how to write CUB unit tests
+- [CUB Benchmarks](docs/cub/benchmarking.rst) - Overview of CUB's performance benchmarks
+- [CUB Tunings](docs/cub/tuning.rst) - Overview of CUB's performance tuning infrastructure
+
+#### Thrust
+
+Coming soon!
+
+#### libcudacxx
+
+Coming soon!
+
+## Building and Testing
+
+CCCL components are header-only libraries. This means there isn't a traditional build process for the library itself. However, before submitting contributions, it's a good idea to [build and run tests](#developer-guides).
+
+There are multiple options for building and running our tests. Which option you choose depends on your preferences and whether you are using [CCCL's DevContainers](.devcontainer/README.md) (highly recommended!).
+
+### Using Manual Build Scripts
+
+#### Building
+
+Use the build scripts provided in the `ci/` directory to build tests for each component. Building tests does not require a GPU.
+
+```bash
+   ci/build_[thrust|cub|libcudacxx].sh -cxx <HOST_COMPILER> -std <CXX_STANDARD> -arch <GPU_ARCHS>
+```
+- **HOST_COMPILER**: The desired host compiler (e.g., `g++`, `clang++`).
+- **CXX_STANDARD**: The C++ standard version (e.g., `17`, `20`).
+- **GPU_ARCHS**: A semicolon-separated list of CUDA GPU architectures (e.g., `"70;85;90"`). This uses the same syntax as CMake's [CUDA_ARCHITECTURES](https://cmake.org/cmake/help/latest/prop_tgt/CUDA_ARCHITECTURES.html#prop_tgt:CUDA_ARCHITECTURES):
+   - `70` - both PTX and SASS
+   - `70-real` - SASS only
+   - `70-virtual` - PTX only
+
+**Example:**
+```bash
+./ci/build_cub.sh -cxx g++ -std 17 -arch "70;75;80-virtual"
+```
+
+#### Testing
+
+Use the test scripts provided in the `ci/` directory to run tests for each component. These take the same arguments as the build scripts and will automatically build the tests if they haven't already been built. Running tests requires a GPU.
+
+```bash
+   ci/test_[thrust|cub|libcudacxx].sh -cxx <HOST_COMPILER> -std <CXX_STANDARD> -arch <GPU_ARCHS>
+```
+
+**Example:**
+
+```bash
+./ci/test_cub.sh -cxx g++ -std 17 -arch "70;75;80-virtual"
+```
+
+### Using CMake Presets
+
+[CMake Presets](https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html) are a set of configurations defined in a JSON file that specify project-wide build details for CMake. They provide a standardized and sharable way to configure, build, and test projects across different platforms and development environments. Presets are available from CMake versions 3.19 and later.
+
+There are three kinds of Presets
+
+- Configure Presets: specify options for the `cmake` command,
+
+- Build Presets: specify options for the `cmake --build` command,
+
+- Test Presets: specify options for the `ctest` command.
+
+In CCCL we provide many presets to be used out of the box. You can find the complete list in our corresponding [CMakePresets.json](./CMakePresets.json) file.
+
+These commands can be used to get lists of the configure, build, and test presets.
+
+```bash
+cmake --list-presets # Configure presets
+cmake --build --list-presets # Build presets
+ctest --list-presets # Test presets
+```
+
+While there is a lot of overlap, there may be differences between the configure, build, and test presets to support various testing workflows.
+
+The `dev` presets are intended as a base for general development while the others are useful for replicating CI failures.
+
+#### Using CMake Presets via Command Line
+
+CMake automatically generates the preset build directories. You can configure, build and test for a specific preset (e.g. `thrust-cpp17`) via cmake from the root directory by appending `--preset=thrust-cpp17` to the corresponding commands. For example:
+
+```bash
+cmake --preset=thrust-cpp17
+cmake --build --preset=thrust-cpp17
+ctest --preset=thrust-cpp17
+```
+
+That will create `build/<optional devcontainer name>/thrust-cpp17/` and build everything in there. The devcontainer name is inserted automatically on devcontainer builds to keep build artifacts separate for the different toolchains.
+
+It's also worth mentioning that additional cmake options can still be passed in and will override the preset settings.
+
+As a common example, the presets are currently always `60;70;80` for `CMAKE_CUDA_ARCHITECTURES`, but this can be overridden at configure time with something like:
+
+```bash
+cmake --preset=thrust-cpp20 "-DCMAKE_CUDA_ARCHITECTURES=89"
+```
+
+> **Note**: Either using the `cmake` command from within the root directory or from within the build directory works, but will behave in slightly different ways. Building and running tests from the build directory will compile every target and run all of the tests configured in the configure step. Doing so from the root directory using the `--preset=<test_preset>` option will build and run a subset of configured targets and tests.
+
+#### Using CMake Presets via VS Code GUI extension (Recommended when using DevContainers)
+
+The recommended way to use CMake Presets is via the VS Code extension [CMake Tools](https://marketplace.visualstudio.com/items?itemName=ms-vscode.cmake-tools), already included in [CCCL's DevContainers](.devcontainer/README.md). As soon as you install the extension you would be able to see the sidebar menu below.
+
+   ![cmaketools sidebar](/.devcontainer/img/cmaketools_sidebar.png)
+
+You can specify the desired CMake Preset by clicking the "Select Configure Preset" button under the "Configure" node (see image below).
+
+   ![cmaketools presets](.devcontainer/img/cmaketools_presets.png)
+
+After that you can select the default build target from the "Build" node. As soon as you expand it, a list will appear with all the available targets that are included within the preset you selected. For example if you had selected the `all-dev` preset VS Code will display all the available targets we have in cccl.
+
+   ![cmaketools presets](.devcontainer/img/cmaketools_targets.png)
+
+You can build the selected target by pressing the gear button ![gear](.devcontainer/img/build_button.png) at the bottom of the VS Code window.
+
+Alternatively you can select the desired target from either the "Debug" or "Launch" drop down menu (for debugging or running correspondingly). <b>In that case after you select the target and either press "Run" ![run](.devcontainer/img/run.png) or "Debug" ![debug](.devcontainer/img/debug.png) the target will build on its own before running without the user having to build it explicitly from the gear button.</b>
+
+---
+
+We encourage users who want to debug device code to install the [Nsight Visual Studio Code Edition extension](https://marketplace.visualstudio.com/items?itemName=NVIDIA.nsight-vscode-edition) that enables the VS Code frontend for `cuda-gdb`. <u>To use it you should launch from the sidebar menu instead of pressing the "Debug" button from the bottom menu</u>.
+
+![nsight](.devcontainer/img/nsight.png)
+
+## Creating a Pull Request
+
+1. Push changes to your fork
+2. Create a pull request targeting the `main` branch of the original CCCL repository. Refer to [GitHub's documentation](https://docs.github.com/en/github/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests) for more information on creating a pull request.
+3. Describe the purpose and context of the changes in the pull request description.
+
+### Documentation Preview
+
+Documentation previews allow reviewers to see how changes will appear on the live documentation site before merging. Previews are automatically generated for all pull requests and updated with every commit. To skip building documentation for a PR, include `[skip-docs]` in your commit message.
+
+The preview URL will be posted as a comment on your PR and automatically cleaned up when the PR is closed.
+
+### Checking for Performance Regressions
+
+Performance stability is a key goal for CCCL, especially for `Device*` algorithms in CUB. When modifying any functionality that could impact these algorithms, contributors are encouraged to verify that no performance regressions occur.
+
+This verification is a two-step process:
+
+1. Determine whether your changes affect the generated SASS code (details on how to do this are provided below).
+2. If the generated SASS code changes, run the benchmarks (see [CUB Benchmarks](docs/cub/benchmarking.rst)) to quantify potential performance implications.
+
+**Steps to check whether your changes generate different SASS code:**
+
+1. Identify the `Device*` algorithm(s) that may be affected by the change. This isn't always straightforward, and you will need to confirm whether any of the CUB algorithms depend on components modified by your changes. If your changes affect only certain GPU architectures, make sure those architectures are included in the list of architectures used during compilation (for example, by specifying them with the `-arch` flag when using the build scripts, or with `-DCMAKE_CUDA_ARCHITECTURES` when building with CMake).
+2. Navigate to the build directory, compile the benchmarks for the specific `Device*` algorithm(s) identified in step 1, and dump the SASS code. For example: `ninja cub.bench.radix_sort.keys.base && cuobjdump -sass ./bin/cub.bench.radix_sort.keys.base |c++filt > ./radix_sort.keys_after.sass`.
+3. Check out the `main` branch to compare against the baseline SASS code: `git checkout $(git merge-base HEAD upstream/main)`
+4. Dump the SASS code emitted on the `main` branch. For example: `ninja cub.bench.radix_sort.keys.base && cuobjdump -sass ./bin/cub.bench.radix_sort.keys.base |c++filt > ./radix_sort.keys_before.sass`.
+5. Check whether there are differences in the generated SASS output: `git diff --text --no-index --word-diff radix_sort.keys_before.sass radix_sort.keys_after.sass`
+
+## Code Formatting (pre-commit hooks)
+
+CCCL uses [pre-commit](https://pre-commit.com/) to execute all code linters and formatters. These
+tools ensure a consistent coding style throughout the project. Using pre-commit ensures that linter
+versions and options are aligned for all developers. Additionally, there is a CI check in place to
+enforce that committed code follows our standards.
+
+The linters used by CCCL are listed in `.pre-commit-config.yaml`.
+For example, C++ and CUDA code is formatted with [`clang-format`](https://clang.llvm.org/docs/ClangFormat.html).
+
+To use `pre-commit`, install via `conda` or `pip`:
+
+```bash
+conda config --add channels conda-forge
+conda install pre-commit
+```
+
+```bash
+pip install pre-commit
+```
+
+Then run pre-commit hooks before committing code:
+
+```bash
+pre-commit run
+```
+
+By default, pre-commit runs on staged files (only changes and additions that will be committed).
+To run pre-commit checks on all files, execute:
+
+```bash
+pre-commit run --all-files
+```
+
+Optionally, you may set up the pre-commit hooks to run automatically when you make a git commit. This can be done by running:
+
+```bash
+pre-commit install
+```
+
+Now code linters and formatters will be run each time you commit changes.
+
+You can skip these checks with `git commit --no-verify` or with the short version `git commit -n`.
+
+## Secret Scanning
+
+The `secret-scan-trufflehog` pre-commit hook scans staged files and installs TruffleHog on first run (use Git Bash on Windows). If it flags a secret, remove it before committing, or contact a maintainer if it's a false positive. Secrets are also scanned server-side in CI on `main`.
+
+## Continuous Integration (CI)
+
+CCCL's CI pipeline tests across various CUDA versions, compilers, and GPU architectures.
+For external contributors, the CI pipeline will not begin until a maintainer leaves an `/ok to test` comment. For members of the NVIDIA GitHub enterprise, the CI pipeline will begin immediately.
+For a detailed overview of CCCL's CI, see [CI overview](docs/infrastructure/ci/references/ci_overview.rst).
+
+There is a CI check for pre-commit, called [pre-commit.ci](pre-commit.ci).
+This enforces that all linters (such as `clang-format`) pass.
+If pre-commit.ci is failing, you can comment `pre-commit.ci autofix` on a pull request to trigger the auto-fixer.
+The auto-fixer will push a commit to your pull request that applies changes made by pre-commit hooks.
+
+## Review Process
+
+Once submitted, maintainers will be automatically assigned to review the pull request. They might suggest changes or improvements. Constructive feedback is a part of the collaborative process, aimed at ensuring the highest quality code.
+
+For constructive feedback and effective communication during reviews, we recommend following [Conventional Comments](https://conventionalcomments.org/).
+
+Further recommended reading for successful PR reviews:
+
+- [How to Do Code Reviews Like a Human (Part One)](https://mtlynch.io/human-code-reviews-1/)
+- [How to Do Code Reviews Like a Human (Part Two)](https://mtlynch.io/human-code-reviews-2/)
+
+## Thank You
+
+Your contributions enhance CCCL for the entire community. We appreciate your effort and collaboration!
diff --git a/cccl_upstream/SECURITY.md b/cccl_upstream/SECURITY.md
new file mode 100644
index 00000000..241adf66
--- /dev/null
+++ b/cccl_upstream/SECURITY.md
@@ -0,0 +1,14 @@
+## Security
+
+NVIDIA is dedicated to the security and trust of our software products and services, including all source code repositories managed through our organization.
+
+If you need to report a security issue, please use the appropriate contact points outlined below. **Please do not report security vulnerabilities through GitHub.**
+
+## Reporting Potential Security Vulnerability in an NVIDIA Product
+
+To report a potential security vulnerability in any NVIDIA product:
+- Web: [Security Vulnerability Submission Form](https://www.nvidia.com/object/submit-security-vulnerability.html)
+- E-Mail: psirt@nvidia.com
+    - We encourage you to use the following PGP key for secure email communication: [NVIDIA public PGP Key for communication](https://www.nvidia.com/en-us/security/pgp-key)
+    - Please include the following information:
+   	 - Product/Driver name and version/branch that contains the vulnerability
diff --git a/cccl_upstream/ci/CMakeLists.txt b/cccl_upstream/ci/CMakeLists.txt
new file mode 100644
index 00000000..0130ac18
--- /dev/null
+++ b/cccl_upstream/ci/CMakeLists.txt
@@ -0,0 +1 @@
+add_subdirectory("test")
diff --git a/cccl_upstream/ci/bench.template.yaml b/cccl_upstream/ci/bench.template.yaml
new file mode 100644
index 00000000..02f60b62
--- /dev/null
+++ b/cccl_upstream/ci/bench.template.yaml
@@ -0,0 +1,70 @@
+# # CCCL PR benchmark request config.
+#
+# ## Overview:
+#
+# This file is used to request benchmark comparisons in PR CI.
+#
+# This file must match ci/bench.template.yaml to merge.
+# CI branch protections will fail if they differ. Reset before merging.
+#
+# To update the defaults (e.g. new GPU pools), modify both this file and
+# ci/bench.template.yaml together in the same PR.
+#
+# !! Strongly consider appending the following to your **commit messages** while benchmarking. !!
+# This prevents wasteful non-benchmark CI jobs if they are not needed.
+#
+#     [bench-only]
+#
+# To skip compile-time benchmark telemetry on unrelated changes, use:
+#
+#     [skip-compile-time-bench]
+#
+# ## Quick start:
+#
+# 1. Add one or more benchmark regexes under benchmarks.filters.cub and/or
+#    benchmarks.filters.python.
+# 2. Enable at least one GPU by uncommenting or adding entries in benchmarks.gpus.
+# 3. Push and inspect the dispatched benchmark jobs/artifacts.
+# 4. Remove/reset benchmark-request edits before final merge.
+
+benchmarks:
+
+  # Benchmark filters grouped by project.
+  filters:
+    # CUB C++ benchmark filters (regex matched against ninja target names).
+    cub:
+      # Examples:
+      # - '^cub\.bench\.for_each\.base'
+      # - '^cub\.bench\.reduce\.(sum|min)\.'
+
+    # Python benchmark filters (regex matched against paths under benchmarks/).
+    python:
+      # Examples:
+      # - 'compute/reduce/sum\.py'
+      # - 'compute/transform/.*\.py'
+
+  # Select GPUs. These are limited and shared, be intentional and conservative.
+  gpus:
+    # - "t4"         # sm_75, 16 GB
+    # - "rtx2080"    # sm_75,  8 GB
+    # - "rtxa6000"   # sm_86, 48 GB
+    # - "l4"         # sm_89, 24 GB
+    # - "rtx4090"    # sm_89, 24 GB
+    # - "h100"       # sm_90, 80 GB
+    # - "rtxpro6000" # sm_120
+
+  # Extra .devcontainer/launch.sh -d args
+  # launch_args: "--cuda 13.3 --host gcc14"
+  launch_args: "" # Latest nvcc + gcc
+
+  # Advanced:
+  base_ref: "origin/main"
+  test_ref: "HEAD"
+  arch: "native"
+  nvbench_args: >-
+    --timeout 30
+    --skip-time 15e-6
+    --stopping-criterion entropy
+    --throttle-threshold 90
+    --throttle-recovery-delay 0.15
+  nvbench_compare_args: ""
diff --git a/cccl_upstream/ci/bench.yaml b/cccl_upstream/ci/bench.yaml
new file mode 100644
index 00000000..02f60b62
--- /dev/null
+++ b/cccl_upstream/ci/bench.yaml
@@ -0,0 +1,70 @@
+# # CCCL PR benchmark request config.
+#
+# ## Overview:
+#
+# This file is used to request benchmark comparisons in PR CI.
+#
+# This file must match ci/bench.template.yaml to merge.
+# CI branch protections will fail if they differ. Reset before merging.
+#
+# To update the defaults (e.g. new GPU pools), modify both this file and
+# ci/bench.template.yaml together in the same PR.
+#
+# !! Strongly consider appending the following to your **commit messages** while benchmarking. !!
+# This prevents wasteful non-benchmark CI jobs if they are not needed.
+#
+#     [bench-only]
+#
+# To skip compile-time benchmark telemetry on unrelated changes, use:
+#
+#     [skip-compile-time-bench]
+#
+# ## Quick start:
+#
+# 1. Add one or more benchmark regexes under benchmarks.filters.cub and/or
+#    benchmarks.filters.python.
+# 2. Enable at least one GPU by uncommenting or adding entries in benchmarks.gpus.
+# 3. Push and inspect the dispatched benchmark jobs/artifacts.
+# 4. Remove/reset benchmark-request edits before final merge.
+
+benchmarks:
+
+  # Benchmark filters grouped by project.
+  filters:
+    # CUB C++ benchmark filters (regex matched against ninja target names).
+    cub:
+      # Examples:
+      # - '^cub\.bench\.for_each\.base'
+      # - '^cub\.bench\.reduce\.(sum|min)\.'
+
+    # Python benchmark filters (regex matched against paths under benchmarks/).
+    python:
+      # Examples:
+      # - 'compute/reduce/sum\.py'
+      # - 'compute/transform/.*\.py'
+
+  # Select GPUs. These are limited and shared, be intentional and conservative.
+  gpus:
+    # - "t4"         # sm_75, 16 GB
+    # - "rtx2080"    # sm_75,  8 GB
+    # - "rtxa6000"   # sm_86, 48 GB
+    # - "l4"         # sm_89, 24 GB
+    # - "rtx4090"    # sm_89, 24 GB
+    # - "h100"       # sm_90, 80 GB
+    # - "rtxpro6000" # sm_120
+
+  # Extra .devcontainer/launch.sh -d args
+  # launch_args: "--cuda 13.3 --host gcc14"
+  launch_args: "" # Latest nvcc + gcc
+
+  # Advanced:
+  base_ref: "origin/main"
+  test_ref: "HEAD"
+  arch: "native"
+  nvbench_args: >-
+    --timeout 30
+    --skip-time 15e-6
+    --stopping-criterion entropy
+    --throttle-threshold 90
+    --throttle-recovery-delay 0.15
+  nvbench_compare_args: ""
diff --git a/cccl_upstream/ci/bench/README.md b/cccl_upstream/ci/bench/README.md
new file mode 100644
index 00000000..51de0faa
--- /dev/null
+++ b/cccl_upstream/ci/bench/README.md
@@ -0,0 +1,82 @@
+# Benchmark Compare Scripts
+
+This directory contains the scripts used by `.github/workflows/bench.yml` to compare benchmark results between two code states.
+
+## Scripts
+
+- `ci/bench/bench.sh`: CI-oriented wrapper that calls `ci/bench/compare_git_refs.sh`.
+- `ci/bench/compare_git_refs.sh`: checks out `<base-ref>` and `<test-ref>` in temporary worktrees, then forwards all remaining args to `ci/bench/compare_paths.sh`.
+- `ci/bench/compare_paths.sh`: configures/builds/runs CUB benchmarks and/or Python benchmarks in two source trees and runs comparison tools on produced JSON outputs.
+- `ci/bench/parse_bench_matrix.sh`: parses `ci/bench.yaml` and emits a dispatch matrix JSON object for `.github/workflows/bench.yml`.
+
+## Usage
+
+Compare CUB benchmarks between two refs:
+
+```bash
+"./ci/bench/bench.sh" "origin/main" "HEAD" \
+  --cub-filter "^cub\\.bench\\.copy\\.memcpy\\.base$"
+```
+
+Compare Python benchmarks between two refs:
+
+```bash
+"./ci/bench/bench.sh" "origin/main" "HEAD" \
+  --python-filter "compute/reduce/sum\\.py"
+```
+
+Run both CUB and Python benchmarks:
+
+```bash
+"./ci/bench/bench.sh" \
+  "origin/main" \
+  "HEAD" \
+  --arch "native" \
+  --nvbench-args "..." \
+  --cub-filter "^cub\\.bench\\.reduce\\..*$" \
+  --python-filter "compute/reduce/sum\\.py"
+```
+
+Compare already checked-out trees:
+
+```bash
+"./ci/bench/compare_paths.sh" \
+  "/path/to/base/cccl" \
+  "/path/to/test/cccl" \
+  --arch "native" \
+  --cub-filter "^cub\\.bench\\.copy\\.memcpy\\.base$" \
+  --python-filter "compute/transform/.*\\.py"
+```
+
+## Workflow Inputs
+
+In `.github/workflows/bench.yml`:
+
+- If `raw_args` is non-empty, it is parsed and passed directly to `ci/bench/bench.sh`.
+- Otherwise, args are assembled from `base_ref`, `test_ref`, `arch`, `cub_filters`, `python_filters`, `nvbench_args`, and `nvbench_compare_args`.
+- CUB filters are passed as `--cub-filter` flags. Python filters are passed as `--python-filter` flags.
+- Malformed quoted input (for example unmatched quotes) fails the workflow step.
+
+## Python Benchmarks
+
+Python benchmarks live under `python/cuda_cccl/benchmarks/` and use `cuda.bench` (the Python nvbench bindings). Each benchmark script outputs nvbench-compatible JSON.
+
+For Python benchmarks, `compare_paths.sh`:
+
+1. Creates isolated virtual environments for base and test trees.
+2. Installs `cuda-cccl[bench-cuXX]` (editable, from each worktree), which pulls in `cuda-bench`, `cupy`, and all other benchmark dependencies.
+3. Runs matching benchmark scripts in each venv.
+4. Compares results using `nvbench-compare` (installed with `cuda-bench`).
+
+Python filters are regex patterns matched against relative paths under `python/cuda_cccl/benchmarks/`, for example:
+- `compute/reduce/sum\.py` — single benchmark
+- `compute/transform/.*\.py` — all transform benchmarks
+
+## Artifacts
+
+`compare_paths.sh` writes a run directory under `${CCCL_BENCH_ARTIFACT_ROOT:-$(pwd)/bench-artifacts}` containing:
+
+- per-target JSON and markdown outputs for base/test runs,
+- grouped build logs (`build.base.log`, `build.test.log`), per-target run logs, and per-target compare logs (`compare.<target>.log`),
+- Python venv setup logs (`py.venv.base.log`, `py.venv.test.log`),
+- `summary.md` with run metadata and per-target collapsible full compare reports.
diff --git a/cccl_upstream/ci/bench/bench.sh b/cccl_upstream/ci/bench/bench.sh
new file mode 100755
index 00000000..5d08f25d
--- /dev/null
+++ b/cccl_upstream/ci/bench/bench.sh
@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+die() {
+  local message="$1"
+  local code="${2:-2}"
+  echo "${message}" >&2
+  exit "${code}"
+}
+
+usage() {
+  cat <<EOF
+Usage: $0 <base-ref> <test-ref> [compare_paths args...]
+
+Wrapper for ci/bench/compare_git_refs.sh.
+EOF
+}
+
+if [[ "${1:-}" == "-h" || "${1:-}" == "--help" ]]; then
+  usage
+  exit 0
+fi
+
+bench_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+"${bench_dir}/compare_git_refs.sh" "$@"
diff --git a/cccl_upstream/ci/bench/compare_git_refs.sh b/cccl_upstream/ci/bench/compare_git_refs.sh
new file mode 100755
index 00000000..a20aacd9
--- /dev/null
+++ b/cccl_upstream/ci/bench/compare_git_refs.sh
@@ -0,0 +1,137 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+die() {
+  local message="$1"
+  local code="${2:-2}"
+  echo "${message}" >&2
+  exit "${code}"
+}
+
+usage() {
+  cat <<EOF
+Usage: $0 <base-ref> <test-ref> [compare_paths args...]
+
+Compare benchmark performance between two git refs from the current CCCL repo.
+Each ref is checked out in an isolated worktree and compared via compare_paths.sh.
+EOF
+}
+
+display_label_for_ref() {
+  local ref="$1"
+  local short_sha="$2"
+  if [[ "${ref}" =~ ^[0-9a-fA-F]{7,40}$ ]]; then
+    printf "%s" "${short_sha}"
+    return 0
+  fi
+  if [[ "${ref}" == "HEAD" ]]; then
+    printf "%s" "${short_sha}"
+    return 0
+  fi
+  printf "%s" "${ref}"
+}
+
+resolve_ref_to_commit() {
+  local repo_root="$1"
+  local ref="$2"
+  local remote=""
+  local branch=""
+  local alternate_ref=""
+
+  if git -C "${repo_root}" rev-parse --verify "${ref}^{commit}" >/dev/null 2>&1; then
+    git -C "${repo_root}" rev-parse --verify "${ref}^{commit}"
+    return 0
+  fi
+
+  if [[ "${ref}" =~ ^([^/]+)/(.+)$ ]] && git -C "${repo_root}" remote get-url "${BASH_REMATCH[1]}" >/dev/null 2>&1; then
+    remote="${BASH_REMATCH[1]}"
+    branch="${BASH_REMATCH[2]}"
+    git -C "${repo_root}" fetch --no-tags "${remote}" \
+      "+refs/heads/${branch}:refs/remotes/${remote}/${branch}" >/dev/null 2>&1 || true
+  elif [[ "${ref}" != refs/* ]]; then
+    # Try unqualified refs as origin branches/tags.
+    git -C "${repo_root}" fetch --no-tags origin \
+      "+refs/heads/${ref}:refs/remotes/origin/${ref}" >/dev/null 2>&1 || true
+    git -C "${repo_root}" fetch --no-tags origin \
+      "refs/tags/${ref}:refs/tags/${ref}" >/dev/null 2>&1 || true
+    alternate_ref="origin/${ref}"
+  fi
+
+  # Final best-effort fetch for raw refs (e.g. refs/pull/* or specific SHAs).
+  git -C "${repo_root}" fetch --no-tags origin "${ref}" >/dev/null 2>&1 || true
+
+  if git -C "${repo_root}" rev-parse --verify "${ref}^{commit}" >/dev/null 2>&1; then
+    git -C "${repo_root}" rev-parse --verify "${ref}^{commit}"
+    return 0
+  fi
+
+  if [[ -n "${alternate_ref}" ]] && git -C "${repo_root}" rev-parse --verify "${alternate_ref}^{commit}" >/dev/null 2>&1; then
+    git -C "${repo_root}" rev-parse --verify "${alternate_ref}^{commit}"
+    return 0
+  fi
+
+  return 1
+}
+
+if [[ "${1:-}" == "-h" || "${1:-}" == "--help" ]]; then
+  usage
+  exit 0
+fi
+
+if [[ "$#" -lt 2 ]]; then
+  usage
+  exit 2
+fi
+
+base_ref="$1"
+test_ref="$2"
+shift 2
+compare_paths_args=("$@")
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+repo_root="$(cd "${ci_dir}/../.." && pwd)"
+
+if [[ ! -f "${repo_root}/cccl-version.json" ]]; then
+  die "This script must run from a CCCL checkout."
+fi
+
+if ! base_commit="$(resolve_ref_to_commit "${repo_root}" "${base_ref}")"; then
+  die "Unable to resolve base ref: ${base_ref}"
+fi
+if ! test_commit="$(resolve_ref_to_commit "${repo_root}" "${test_ref}")"; then
+  die "Unable to resolve test ref: ${test_ref}"
+fi
+
+base_short_sha="$(git -C "${repo_root}" rev-parse --short=12 "${base_commit}")"
+test_short_sha="$(git -C "${repo_root}" rev-parse --short=12 "${test_commit}")"
+base_label="$(display_label_for_ref "${base_ref}" "${base_short_sha}")"
+test_label="$(display_label_for_ref "${test_ref}" "${test_short_sha}")"
+
+worktree_root="$(mktemp -d "/tmp/cccl-bench-worktrees-XXXXXX")"
+base_path="${worktree_root}/base"
+test_path="${worktree_root}/test"
+
+cleanup() {
+  git -C "${repo_root}" worktree remove --force "${base_path}" >/dev/null 2>&1 || true
+  git -C "${repo_root}" worktree remove --force "${test_path}" >/dev/null 2>&1 || true
+  rm -rf "${worktree_root}"
+}
+trap cleanup EXIT
+
+echo "Creating worktree for base ref: ${base_ref}"
+git -C "${repo_root}" worktree add --detach "${base_path}" "${base_commit}" >/dev/null
+
+echo "Creating worktree for test ref: ${test_ref}"
+git -C "${repo_root}" worktree add --detach "${test_path}" "${test_commit}" >/dev/null
+
+compare_cmd=(
+  "${ci_dir}/compare_paths.sh"
+  "${base_path}"
+  "${test_path}"
+  "${compare_paths_args[@]}"
+)
+
+CCCL_BENCH_BASE_LABEL="${base_label}" \
+CCCL_BENCH_TEST_LABEL="${test_label}" \
+  "${compare_cmd[@]}"
diff --git a/cccl_upstream/ci/bench/compare_paths.sh b/cccl_upstream/ci/bench/compare_paths.sh
new file mode 100755
index 00000000..2eb45a4a
--- /dev/null
+++ b/cccl_upstream/ci/bench/compare_paths.sh
@@ -0,0 +1,1048 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+die() {
+  local message="$1"
+  local code="${2:-2}"
+  echo "${message}" >&2
+  exit "${code}"
+}
+
+usage() {
+  cat <<EOF
+Usage: $0 <base-path> <test-path> \
+  [--cub-filter "<regex>"] \
+  [--python-filter "<regex>"] \
+  [--arch "<arch>"] \
+  [--nvbench-args "<args>"] \
+  [--nvbench-compare-args "<args>"]
+
+Compare benchmark performance between two checked-out CCCL trees.
+
+At least one --cub-filter or --python-filter must be provided.
+CUB filters are regex patterns matched against ninja target names.
+Python filters are regex patterns matched against benchmark script paths
+under python/cuda_cccl/benchmarks/ (e.g. compute/reduce/sum.py).
+
+Arguments:
+  <base-path>  Path to baseline CCCL source tree.
+  <test-path>  Path to comparison CCCL source tree.
+
+Options:
+  --cub-filter <regex>      CUB benchmark regex filter (repeatable).
+  --python-filter <regex>   Python benchmark regex filter (repeatable).
+  --arch <arch>             CMAKE_CUDA_ARCHITECTURES for CUB builds.
+  --nvbench-args <args>     Extra args passed to benchmark binaries/scripts.
+  --nvbench-compare-args <args>  Extra args passed to nvbench_compare.
+
+Environment:
+  CCCL_BENCH_ARTIFACT_ROOT   Root directory for outputs.
+                             Default: "\$(pwd)/bench-artifacts"
+  CCCL_BENCH_ARTIFACT_TAG    Optional explicit artifact directory name.
+  CCCL_BENCH_BASE_LABEL      Optional label used in auto-generated artifact names.
+  CCCL_BENCH_TEST_LABEL      Optional label used in auto-generated artifact names.
+  CCCL_BENCH_BASE_BUILD_DIR  Optional preconfigured build tree for base path.
+  CCCL_BENCH_TEST_BUILD_DIR  Optional preconfigured build tree for test path.
+                             If either is set, both must be set.
+  CCCL_BENCH_GPU_NAME        Optional GPU name included in artifact directory names.
+  CCCL_BENCH_BUILD_ROOT      Root directory for generated build trees.
+                             Default: "/tmp/cccl-bench-builds"
+EOF
+}
+
+sanitize_label() {
+  local label="$1"
+  label="${label//[^a-zA-Z0-9._-]/_}"
+  label="${label#_}"
+  label="${label%_}"
+  label="${label:-unknown}"
+  printf "%s" "${label}"
+}
+
+resolve_repo_label() {
+  local repo_path="$1"
+  local branch=""
+  branch="$(git -C "${repo_path}" rev-parse --abbrev-ref HEAD 2>/dev/null || true)"
+  if [[ -n "${branch}" && "${branch}" != "HEAD" ]]; then
+    printf "%s" "${branch}"
+    return 0
+  fi
+  local short_sha=""
+  short_sha="$(git -C "${repo_path}" rev-parse --short=12 HEAD 2>/dev/null || true)"
+  if [[ -n "${short_sha}" ]]; then
+    printf "%s" "${short_sha}"
+    return 0
+  fi
+  basename "${repo_path}"
+}
+
+validate_repo_path() {
+  local repo_path="$1"
+  if [[ ! -d "${repo_path}" ]]; then
+    die "Path does not exist: ${repo_path}"
+  fi
+  if [[ ! -f "${repo_path}/cccl-version.json" ]]; then
+    die "Path is not a CCCL source tree: ${repo_path}"
+  fi
+}
+
+validate_filter_array() {
+  local -n _validate_filters_ref="$1"
+  local label="$2"
+  local filter=""
+  for filter in "${_validate_filters_ref[@]}"; do
+    grep -Eq -- "${filter}" <<< "" >/dev/null 2>&1 || {
+      [[ "$?" -eq 1 ]] || die "Invalid ${label} regex filter: ${filter}"
+    }
+  done
+}
+
+print_shell_command() {
+  # Print a shell-escaped command line so it can be copied and re-run directly.
+  # Usage: print_shell_command [--env "VAR=value" ...] cmd [args...]
+  local -a env_prefixes=()
+  while [[ "$#" -gt 0 && "$1" == --env ]]; do
+    shift
+    env_prefixes+=("$1")
+    shift
+  done
+  printf '$'
+  local item=""
+  for item in "${env_prefixes[@]}"; do
+    printf ' %q' "${item}"
+  done
+  for item in "$@"; do
+    printf ' %q' "${item}"
+  done
+  printf '\n'
+}
+
+# ============================================================================
+# CUB helpers
+# ============================================================================
+
+configure_build_tree() {
+  local src_path="$1"
+  local build_path="$2"
+  local side="$3"
+  local log_path="$4"
+  local target_arch="$5"
+  local -a cmake_cmd
+  cmake_cmd=(cmake --preset "cub-benchmark" -S "${src_path}" -B "${build_path}")
+  if [[ -n "${target_arch}" ]]; then
+    cmake_cmd+=("-DCMAKE_CUDA_ARCHITECTURES=${target_arch}")
+  fi
+  run_grouped_logged_command "[configure:${side}]" "${log_path}" "${cmake_cmd[@]}"
+}
+
+validate_build_dir() {
+  local build_path="$1"
+  local label="$2"
+  if [[ ! -d "${build_path}" ]]; then
+    die "Configured ${label} build tree does not exist: ${build_path}"
+  fi
+  if [[ ! -f "${build_path}/build.ninja" ]]; then
+    die "Configured ${label} build tree is missing build.ninja: ${build_path}"
+  fi
+}
+
+list_all_benchmark_targets() {
+  local build_path="$1"
+  ninja -C "${build_path}" -t targets all \
+    | awk -F':' '/^.*\.bench\./ { print $1 }' \
+    | sort -u
+}
+
+target_matches_filters() {
+  local target="$1"
+  local filter=""
+  if [[ "${#FILTERS[@]}" -eq 0 ]]; then
+    return 0
+  fi
+  for filter in "${FILTERS[@]}"; do
+    if grep -Eq -- "${filter}" <<< "${target}"; then
+      return 0
+    fi
+  done
+  return 1
+}
+
+resolve_compare_script() {
+  local build_path="$1"
+  local nvbench_src="${build_path}/_deps/nvbench-src"
+  local candidate=""
+  # Diff versions have the script at diff locations:
+  for candidate in \
+    "${nvbench_src}/python/scripts/nvbench_compare.py" \
+    "${nvbench_src}/scripts/nvbench_compare.py"; do
+    if [[ -f "${candidate}" ]]; then
+      printf "%s" "${candidate}"
+      return 0
+    fi
+  done
+  return 1
+}
+
+run_target_for_side() {
+  local side="$1"
+  local build_path="$2"
+  local target="$3"
+  local json_path="$4"
+  local md_path="$5"
+  local run_log="$6"
+  local binary_path="${build_path}/bin/${target}"
+  local -a bench_cmd
+
+  if [[ ! -x "${binary_path}" ]]; then
+    echo "Benchmark binary missing: ${binary_path}" >&2
+    return 127
+  fi
+
+  bench_cmd=(
+    "${binary_path}"
+    -d 0
+    "${NVBENCH_RUN_ARGS[@]}"
+    --json "${json_path}"
+    --md "${md_path}"
+  )
+
+  run_grouped_logged_command \
+    "[run:${side}] ${target}" \
+    "${run_log}" \
+    "${bench_cmd[@]}"
+}
+
+select_targets() {
+  local base_build_path="$1"
+  local test_build_path="$2"
+  local -n selected_targets_ref="$3"
+  local -a base_targets
+  local -a test_targets
+  local -a common_targets
+  local target=""
+
+  mapfile -t base_targets < <(list_all_benchmark_targets "${base_build_path}")
+  mapfile -t test_targets < <(list_all_benchmark_targets "${test_build_path}")
+
+  if [[ "${#base_targets[@]}" -eq 0 ]]; then
+    die "No CUB benchmark targets were found in base build tree." 1
+  fi
+  if [[ "${#test_targets[@]}" -eq 0 ]]; then
+    die "No CUB benchmark targets were found in test build tree." 1
+  fi
+
+  mapfile -t common_targets < <(
+    comm -12 \
+      <(printf "%s\n" "${base_targets[@]}" | sort -u) \
+      <(printf "%s\n" "${test_targets[@]}" | sort -u)
+  )
+
+  selected_targets_ref=()
+  for target in "${common_targets[@]}"; do
+    [[ -n "${target}" ]] || continue
+    if target_matches_filters "${target}"; then
+      selected_targets_ref+=("${target}")
+    fi
+  done
+
+  if [[ "${#selected_targets_ref[@]}" -eq 0 ]]; then
+    die "No CUB benchmark targets matched the supplied filters." 1
+  fi
+}
+
+# ============================================================================
+# Python helpers
+# ============================================================================
+
+detect_cuda_major_version() {
+  local cuda_major=""
+  if command -v nvcc >/dev/null 2>&1; then
+    cuda_major="$(nvcc --version 2>/dev/null | sed -n 's/.*release \([0-9]*\)\..*/\1/p')"
+  fi
+  if [[ -z "${cuda_major}" ]]; then
+    cuda_major="12"
+  fi
+  printf "%s" "${cuda_major}"
+}
+
+python_path_to_target_name() {
+  local py_path="$1"
+  # compute/reduce/sum.py -> py.compute.reduce.sum
+  local name="${py_path%.py}"
+  name="${name//\//.}"
+  printf "py.%s" "${name}"
+}
+
+list_all_python_benchmarks() {
+  local benchmarks_path="$1"
+  if [[ ! -d "${benchmarks_path}" ]]; then
+    return 0
+  fi
+  find "${benchmarks_path}" -name '*.py' -type f \
+    ! -name 'utils.py' \
+    ! -name 'run_benchmarks.py' \
+    ! -name 'device_side_benchmark.py' \
+    ! -name '__init__.py' \
+    ! -path '*/__pycache__/*' \
+    -printf '%P\n' \
+    | sort -u
+}
+
+python_target_matches_filters() {
+  local target="$1"
+  local filter=""
+  for filter in "${PYTHON_FILTERS[@]}"; do
+    if grep -Eq -- "${filter}" <<< "${target}"; then
+      return 0
+    fi
+  done
+  return 1
+}
+
+select_python_targets() {
+  local base_bench_path="$1"
+  local test_bench_path="$2"
+  local -n selected_py_targets_ref="$3"
+  local -a base_py_targets
+  local -a test_py_targets
+  local -a common_py_targets
+  local target=""
+
+  mapfile -t base_py_targets < <(list_all_python_benchmarks "${base_bench_path}")
+  mapfile -t test_py_targets < <(list_all_python_benchmarks "${test_bench_path}")
+
+  if [[ "${#base_py_targets[@]}" -eq 0 ]]; then
+    die "No Python benchmark scripts were found in base tree: ${base_bench_path}" 1
+  fi
+  if [[ "${#test_py_targets[@]}" -eq 0 ]]; then
+    die "No Python benchmark scripts were found in test tree: ${test_bench_path}" 1
+  fi
+
+  mapfile -t common_py_targets < <(
+    comm -12 \
+      <(printf "%s\n" "${base_py_targets[@]}" | sort -u) \
+      <(printf "%s\n" "${test_py_targets[@]}" | sort -u)
+  )
+
+  selected_py_targets_ref=()
+  for target in "${common_py_targets[@]}"; do
+    [[ -n "${target}" ]] || continue
+    if python_target_matches_filters "${target}"; then
+      selected_py_targets_ref+=("${target}")
+    fi
+  done
+
+  if [[ "${#selected_py_targets_ref[@]}" -eq 0 ]]; then
+    die "No Python benchmark scripts matched the supplied --python-filter patterns." 1
+  fi
+}
+
+setup_python_venv() {
+  local venv_path="$1"
+  local src_path="$2"
+  local side="$3"
+  local log_path="$4"
+  local cuda_major="$5"
+  local cuda_cccl_dir="${src_path}/python/cuda_cccl"
+
+  if [[ ! -d "${cuda_cccl_dir}" ]]; then
+    die "cuda_cccl source directory not found: ${cuda_cccl_dir}"
+  fi
+
+  local -a setup_cmds
+  setup_cmds=(
+    bash -c "
+      set -euo pipefail
+      python3 -m venv '${venv_path}'
+      '${venv_path}/bin/pip' install --upgrade pip
+      '${venv_path}/bin/pip' install -e '${cuda_cccl_dir}[bench-cu${cuda_major}]'
+      # nvbench-compare runtime deps (until cuda-bench declares them):
+      '${venv_path}/bin/pip' install colorama jsondiff tabulate
+    "
+  )
+
+  run_grouped_logged_command \
+    "[py-venv:${side}]" \
+    "${log_path}" \
+    "${setup_cmds[@]}"
+}
+
+run_python_target_for_side() {
+  local side="$1"
+  local venv_path="$2"
+  local script_path="$3"
+  local json_path="$4"
+  local md_path="$5"
+  local run_log="$6"
+  local -a bench_cmd
+
+  if [[ ! -f "${script_path}" ]]; then
+    echo "Python benchmark script missing: ${script_path}" >&2
+    return 127
+  fi
+
+  bench_cmd=(
+    "${venv_path}/bin/python"
+    "${script_path}"
+    -d 0
+    "${NVBENCH_RUN_ARGS[@]}"
+    --json "${json_path}"
+    --md "${md_path}"
+  )
+
+  run_grouped_logged_command \
+    "[py-run:${side}] ${script_path##*/benchmarks/}" \
+    "${run_log}" \
+    "${bench_cmd[@]}"
+}
+
+run_python_compare_target() {
+  local target_name="$1"
+  local venv_path="$2"
+  local base_json="$3"
+  local test_json="$4"
+  local compare_out="$5"
+  local compare_log="$6"
+
+  local label="[py-compare] ${target_name}"
+  local started_at=0
+  local elapsed_s=0
+  local rc=0
+  local -a compare_cmd
+  compare_cmd=("${venv_path}/bin/nvbench-compare" --no-color "${NVBENCH_COMPARE_ARGS[@]}" "${base_json}" "${test_json}")
+
+  : > "${compare_log}"
+  echo "::group::${label}"
+  print_shell_command "${compare_cmd[@]}"
+  started_at="${SECONDS}"
+  if "${compare_cmd[@]}" \
+    > >(tee "${compare_out}" | tee -a "${compare_log}") \
+    2> >(tee -a "${compare_log}" >&2); then
+    rc=0
+  else
+    rc=$?
+  fi
+  elapsed_s=$((SECONDS - started_at))
+  echo "::endgroup::"
+  if [[ "${rc}" -eq 0 ]]; then
+    echo "${label} completed in ${elapsed_s}s"
+  else
+    echo "${label} failed in ${elapsed_s}s (rc=${rc})"
+  fi
+  return "${rc}"
+}
+
+# ============================================================================
+# Common helpers
+# ============================================================================
+
+run_grouped_logged_command() {
+  local label="$1"
+  local log_path="$2"
+  shift 2
+  local started_at=0
+  local elapsed_s=0
+  local rc=0
+  local -a pipe_statuses
+
+  echo "::group::${label}"
+  print_shell_command "$@"
+  started_at="${SECONDS}"
+  set +o pipefail
+  "$@" 2>&1 | tee "${log_path}"
+  pipe_statuses=("${PIPESTATUS[@]}")
+  set -o pipefail
+  if [[ "${pipe_statuses[0]}" -ne 0 ]]; then
+    rc="${pipe_statuses[0]}"
+  elif [[ "${pipe_statuses[1]}" -ne 0 ]]; then
+    rc="${pipe_statuses[1]}"
+  fi
+  elapsed_s=$((SECONDS - started_at))
+  echo "::endgroup::"
+  if [[ "${rc}" -eq 0 ]]; then
+    echo "${label} completed in ${elapsed_s}s"
+  else
+    echo "${label} failed in ${elapsed_s}s (rc=${rc})"
+  fi
+  return "${rc}"
+}
+
+run_compare_target() {
+  local target="$1"
+  local compare_script="$2"
+  local compare_script_dir="$3"
+  local base_json="$4"
+  local test_json="$5"
+  local compare_out="$6"
+  local compare_log="$7"
+
+  local label="[compare] ${target}"
+  local started_at=0
+  local elapsed_s=0
+  local rc=0
+  local compare_pythonpath="${compare_script_dir}${PYTHONPATH:+:${PYTHONPATH}}"
+  local -a compare_cmd
+  compare_cmd=(python3 "${compare_script}" --no-color "${NVBENCH_COMPARE_ARGS[@]}" "${base_json}" "${test_json}")
+
+  : > "${compare_log}"
+  echo "::group::${label}"
+  print_shell_command --env "PYTHONPATH=${compare_pythonpath}" "${compare_cmd[@]}"
+  started_at="${SECONDS}"
+  if PYTHONPATH="${compare_pythonpath}" \
+    "${compare_cmd[@]}" \
+    > >(tee "${compare_out}" | tee -a "${compare_log}") \
+    2> >(tee -a "${compare_log}" >&2); then
+    rc=0
+  else
+    rc=$?
+  fi
+  elapsed_s=$((SECONDS - started_at))
+  echo "::endgroup::"
+  if [[ "${rc}" -eq 0 ]]; then
+    echo "${label} completed in ${elapsed_s}s"
+  else
+    echo "${label} failed in ${elapsed_s}s (rc=${rc})"
+  fi
+  return "${rc}"
+}
+
+parse_quoted_args_to_nul_file() {
+  local quoted_args="$1"
+  local output_file="$2"
+  local option_name="$3"
+
+  QUOTED_ARGS="${quoted_args}" OPTION_NAME="${option_name}" python3 - "${output_file}" <<'PY'
+import os
+import shlex
+import sys
+
+quoted = os.environ.get("QUOTED_ARGS", "")
+option_name = os.environ.get("OPTION_NAME", "--args")
+try:
+    parsed = shlex.split(quoted)
+except ValueError as exc:
+    print(f"Invalid {option_name}: {exc}", file=sys.stderr)
+    raise SystemExit(2)
+
+with open(sys.argv[1], "wb") as out_file:
+    for arg in parsed:
+        out_file.write(arg.encode("utf-8"))
+        out_file.write(b"\0")
+PY
+}
+
+parse_quoted_args_to_array() {
+  local -n _target_array_ref="$1"
+  local quoted_args="$2"
+  local option_label="$3"
+  local parsed_args_file=""
+
+  _target_array_ref=()
+  [[ -n "${quoted_args}" ]] || return 0
+
+  parsed_args_file="$(mktemp "/tmp/cccl-parsed-args-XXXXXX")"
+  if ! parse_quoted_args_to_nul_file "${quoted_args}" "${parsed_args_file}" "${option_label}"; then
+    rm -f "${parsed_args_file}"
+    return 2
+  fi
+  mapfile -d '' -t _target_array_ref < "${parsed_args_file}"
+  rm -f "${parsed_args_file}"
+}
+
+# ============================================================================
+# Summary
+# ============================================================================
+
+write_summary() {
+  local summary_file="$1"
+  local target=""
+  local compare_report_file=""
+  local reports_emitted=0
+
+  {
+    echo "# Benchmark Comparison Summary"
+    echo
+    echo "- Timestamp (UTC): ${timestamp}"
+    echo "- GPU name: ${CCCL_BENCH_GPU_NAME:-not specified}"
+    echo "- Base label: ${base_label_raw}"
+    echo "- Test label: ${test_label_raw}"
+    echo "- Base source path: \`${BASE_PATH}\`"
+    echo "- Test source path: \`${TEST_PATH}\`"
+    if [[ "${#FILTERS[@]}" -gt 0 ]]; then
+      echo "- Base build dir: \`${base_build_dir}\`"
+      echo "- Test build dir: \`${test_build_dir}\`"
+    fi
+    echo "- CUB targets selected: ${#selected_targets[@]}"
+    echo "- CUB comparisons attempted: ${compares_attempted}"
+    echo "- CUB comparisons succeeded: ${compares_succeeded}"
+    echo "- Python targets selected: ${#selected_py_targets[@]}"
+    echo "- Python comparisons attempted: ${py_compares_attempted}"
+    echo "- Python comparisons succeeded: ${py_compares_succeeded}"
+    echo "- Target arch: ${TARGET_ARCH:-preset-default}"
+    echo "- Artifact directory: \`${artifact_dir}\`"
+    echo
+
+    if [[ "${#FILTERS[@]}" -gt 0 ]]; then
+      echo "## CUB Filters"
+      for filter in "${FILTERS[@]}"; do
+        echo "- \`${filter}\`"
+      done
+      echo
+    fi
+
+    if [[ "${#PYTHON_FILTERS[@]}" -gt 0 ]]; then
+      echo "## Python Filters"
+      for filter in "${PYTHON_FILTERS[@]}"; do
+        echo "- \`${filter}\`"
+      done
+      echo
+    fi
+
+    if [[ "${#selected_targets[@]}" -gt 0 ]]; then
+      echo "## CUB Compare Reports"
+      for target in "${selected_targets[@]}"; do
+        compare_report_file="${artifact_dir}/compare/${target}.md"
+        if [[ ! -f "${compare_report_file}" ]]; then
+          continue
+        fi
+        reports_emitted=$((reports_emitted + 1))
+        echo
+        echo "### \`${target}\`"
+        echo
+        echo "<details><summary>Expand full compare output for \`${target}\`</summary>"
+        echo
+        cat "${compare_report_file}"
+        echo
+        echo "</details>"
+      done
+    fi
+
+    if [[ "${#selected_py_targets[@]}" -gt 0 ]]; then
+      echo
+      echo "## Python Compare Reports"
+      local py_target_path=""
+      local py_target_name=""
+      for py_target_path in "${selected_py_targets[@]}"; do
+        py_target_name="$(python_path_to_target_name "${py_target_path}")"
+        compare_report_file="${artifact_dir}/compare/${py_target_name}.md"
+        if [[ ! -f "${compare_report_file}" ]]; then
+          continue
+        fi
+        reports_emitted=$((reports_emitted + 1))
+        echo
+        echo "### \`${py_target_name}\` (\`${py_target_path}\`)"
+        echo
+        echo "<details><summary>Expand full compare output for \`${py_target_name}\`</summary>"
+        echo
+        cat "${compare_report_file}"
+        echo
+        echo "</details>"
+      done
+    fi
+
+    if [[ "${reports_emitted}" -eq 0 ]]; then
+      echo
+      echo "_No per-target compare reports were produced._"
+    fi
+  } > "${summary_file}"
+}
+
+# ============================================================================
+# CLI parsing
+# ============================================================================
+
+parse_cli_args() {
+  if [[ "${1:-}" == "-h" || "${1:-}" == "--help" ]]; then
+    usage
+    exit 0
+  fi
+  if [[ "$#" -lt 2 ]]; then
+    usage
+    exit 2
+  fi
+
+  BASE_PATH="$(realpath "$1")"
+  TEST_PATH="$(realpath "$2")"
+  shift 2
+
+  NVBENCH_ARGS_STRING=""
+  NVBENCH_COMPARE_ARGS_STRING=""
+  TARGET_ARCH=""
+  FILTERS=()
+  PYTHON_FILTERS=()
+  while [[ "$#" -gt 0 ]]; do
+    case "$1" in
+      --arch)
+        if [[ "$#" -lt 2 ]]; then
+          die "Missing value for --arch"
+        fi
+        TARGET_ARCH="$2"
+        shift 2
+        ;;
+      --nvbench-args)
+        if [[ "$#" -lt 2 ]]; then
+          die "Missing value for --nvbench-args"
+        fi
+        NVBENCH_ARGS_STRING="$2"
+        shift 2
+        ;;
+      --nvbench-compare-args)
+        if [[ "$#" -lt 2 ]]; then
+          die "Missing value for --nvbench-compare-args"
+        fi
+        NVBENCH_COMPARE_ARGS_STRING="$2"
+        shift 2
+        ;;
+      --cub-filter)
+        if [[ "$#" -lt 2 ]]; then
+          die "Missing value for --cub-filter"
+        fi
+        FILTERS+=("$2")
+        shift 2
+        ;;
+      --python-filter)
+        if [[ "$#" -lt 2 ]]; then
+          die "Missing value for --python-filter"
+        fi
+        PYTHON_FILTERS+=("$2")
+        shift 2
+        ;;
+      --)
+        shift
+        break
+        ;;
+      *)
+        die "Unknown option: $1"
+        ;;
+    esac
+  done
+}
+
+parse_cli_args "$@"
+
+declare -a NVBENCH_RUN_ARGS
+declare -a NVBENCH_COMPARE_ARGS
+parse_quoted_args_to_array NVBENCH_RUN_ARGS "${NVBENCH_ARGS_STRING}" "--nvbench-args" \
+  || die "Failed to parse --nvbench-args."
+parse_quoted_args_to_array NVBENCH_COMPARE_ARGS "${NVBENCH_COMPARE_ARGS_STRING}" "--nvbench-compare-args" \
+  || die "Failed to parse --nvbench-compare-args."
+
+validate_repo_path "${BASE_PATH}"
+validate_repo_path "${TEST_PATH}"
+validate_filter_array FILTERS "CUB"
+validate_filter_array PYTHON_FILTERS "Python"
+
+# ============================================================================
+# Common setup
+# ============================================================================
+
+timestamp="$(date -u +'%Y%m%dT%H%M%SZ')"
+base_label_raw="${CCCL_BENCH_BASE_LABEL:-$(resolve_repo_label "${BASE_PATH}")}"
+test_label_raw="${CCCL_BENCH_TEST_LABEL:-$(resolve_repo_label "${TEST_PATH}")}"
+base_label="$(sanitize_label "${base_label_raw}")"
+test_label="$(sanitize_label "${test_label_raw}")"
+
+artifact_root="${CCCL_BENCH_ARTIFACT_ROOT:-$(pwd)/bench-artifacts}"
+gpu_tag="${CCCL_BENCH_GPU_NAME:+$(sanitize_label "${CCCL_BENCH_GPU_NAME}")-}"
+artifact_tag="${CCCL_BENCH_ARTIFACT_TAG:-bench-${gpu_tag}${test_label}-${timestamp}-${base_label}}"
+artifact_tag="$(sanitize_label "${artifact_tag}")"
+artifact_dir="${artifact_root}/${artifact_tag}"
+
+build_root="${CCCL_BENCH_BUILD_ROOT:-/tmp/cccl-bench-builds}"
+build_token="$(sanitize_label "${test_label}-${timestamp}-${base_label}")"
+base_build_dir="${build_root}/base-${build_token}"
+test_build_dir="${build_root}/test-${build_token}"
+
+for subdir in base compare logs meta test; do
+  mkdir -p "${artifact_dir}/${subdir}"
+done
+mkdir -p "${build_root}"
+
+echo "Artifact directory: ${artifact_dir}"
+if [[ -n "${CCCL_BENCH_GPU_NAME:-}" ]]; then
+  echo "GPU name: ${CCCL_BENCH_GPU_NAME}"
+fi
+echo "Base source: ${BASE_PATH}"
+echo "Test source: ${TEST_PATH}"
+if [[ "${#FILTERS[@]}" -gt 0 ]]; then
+  echo "CUB filters:"
+  for filter in "${FILTERS[@]}"; do
+    echo "  - ${filter}"
+  done
+else
+  echo "CUB filters: (none)"
+fi
+if [[ "${#PYTHON_FILTERS[@]}" -gt 0 ]]; then
+  echo "Python filters:"
+  for filter in "${PYTHON_FILTERS[@]}"; do
+    echo "  - ${filter}"
+  done
+else
+  echo "Python filters: (none)"
+fi
+if [[ -n "${TARGET_ARCH}" ]]; then
+  echo "Target arch: ${TARGET_ARCH}"
+fi
+if [[ "${#NVBENCH_RUN_ARGS[@]}" -gt 0 ]]; then
+  echo "Extra run args:"
+  for arg in "${NVBENCH_RUN_ARGS[@]}"; do
+    echo "  - ${arg}"
+  done
+fi
+if [[ "${#NVBENCH_COMPARE_ARGS[@]}" -gt 0 ]]; then
+  echo "Extra compare args:"
+  for arg in "${NVBENCH_COMPARE_ARGS[@]}"; do
+    echo "  - ${arg}"
+  done
+fi
+
+any_failures=0
+compares_attempted=0
+compares_succeeded=0
+declare -a selected_targets=()
+py_compares_attempted=0
+py_compares_succeeded=0
+declare -a selected_py_targets=()
+
+# ============================================================================
+# CUB benchmark pipeline
+# ============================================================================
+
+if [[ "${#FILTERS[@]}" -gt 0 ]]; then
+  echo
+  echo "=== CUB Benchmark Pipeline ==="
+  echo
+
+  external_base_build_dir="${CCCL_BENCH_BASE_BUILD_DIR:-}"
+  external_test_build_dir="${CCCL_BENCH_TEST_BUILD_DIR:-}"
+  if [[ -n "${external_base_build_dir}" || -n "${external_test_build_dir}" ]]; then
+    if [[ -z "${external_base_build_dir}" || -z "${external_test_build_dir}" ]]; then
+      die "Both CCCL_BENCH_BASE_BUILD_DIR and CCCL_BENCH_TEST_BUILD_DIR must be set together."
+    fi
+    base_build_dir="$(realpath "${external_base_build_dir}")"
+    test_build_dir="$(realpath "${external_test_build_dir}")"
+    validate_build_dir "${base_build_dir}" "base"
+    validate_build_dir "${test_build_dir}" "test"
+    if [[ -n "${TARGET_ARCH}" ]]; then
+      echo "Warning: --arch is ignored when using preconfigured build directories." >&2
+    fi
+  fi
+
+  if [[ -z "${external_base_build_dir:-}" ]]; then
+    configure_build_tree "${BASE_PATH}" "${base_build_dir}" "base" "${artifact_dir}/logs/configure.base.log" "${TARGET_ARCH}"
+    configure_build_tree "${TEST_PATH}" "${test_build_dir}" "test" "${artifact_dir}/logs/configure.test.log" "${TARGET_ARCH}"
+  else
+    echo "[configure:base] skipped (using existing build tree)"
+    echo "[configure:test] skipped (using existing build tree)"
+  fi
+
+  select_targets "${base_build_dir}" "${test_build_dir}" selected_targets
+
+  printf "%s\n" "${selected_targets[@]}" > "${artifact_dir}/meta/selected_targets.txt"
+
+  compare_script="$(resolve_compare_script "${test_build_dir}" || true)"
+  if [[ -z "${compare_script}" ]]; then
+    compare_script="$(resolve_compare_script "${base_build_dir}" || true)"
+  fi
+  if [[ -z "${compare_script}" ]]; then
+    die "Unable to locate nvbench_compare.py in build dependencies." 1
+  fi
+  compare_script_dir="$(dirname "${compare_script}")"
+
+  base_build_all_rc=0
+  test_build_all_rc=0
+
+  if run_grouped_logged_command \
+    "[build:base]" \
+    "${artifact_dir}/logs/build.base.log" \
+    ninja -C "${base_build_dir}" "${selected_targets[@]}"; then
+    base_build_all_rc=0
+  else
+    base_build_all_rc=$?
+    any_failures=1
+  fi
+
+  if run_grouped_logged_command \
+    "[build:test]" \
+    "${artifact_dir}/logs/build.test.log" \
+    ninja -C "${test_build_dir}" "${selected_targets[@]}"; then
+    test_build_all_rc=0
+  else
+    test_build_all_rc=$?
+    any_failures=1
+  fi
+
+  for target in "${selected_targets[@]}"; do
+    base_target_run_rc=125
+    test_target_run_rc=125
+    base_run_log="${artifact_dir}/logs/run.base.${target}.log"
+    test_run_log="${artifact_dir}/logs/run.test.${target}.log"
+    compare_report_md="${artifact_dir}/compare/${target}.md"
+    compare_report_log="${artifact_dir}/logs/compare.${target}.log"
+
+    base_json="${artifact_dir}/base/${target}.json"
+    base_md="${artifact_dir}/base/${target}.md"
+    test_json="${artifact_dir}/test/${target}.json"
+    test_md="${artifact_dir}/test/${target}.md"
+
+    if [[ "${base_build_all_rc}" -eq 0 ]]; then
+      if run_target_for_side \
+        "base" \
+        "${base_build_dir}" \
+        "${target}" \
+        "${base_json}" \
+        "${base_md}" \
+        "${base_run_log}"; then
+        base_target_run_rc=0
+      else
+        base_target_run_rc=$?
+        any_failures=1
+      fi
+    fi
+
+    if [[ "${test_build_all_rc}" -eq 0 ]]; then
+      if run_target_for_side \
+        "test" \
+        "${test_build_dir}" \
+        "${target}" \
+        "${test_json}" \
+        "${test_md}" \
+        "${test_run_log}"; then
+        test_target_run_rc=0
+      else
+        test_target_run_rc=$?
+        any_failures=1
+      fi
+    fi
+
+    if [[ "${base_target_run_rc}" -eq 0 && "${test_target_run_rc}" -eq 0 ]]; then
+      compares_attempted=$((compares_attempted + 1))
+      if run_compare_target \
+        "${target}" \
+        "${compare_script}" \
+        "${compare_script_dir}" \
+        "${base_json}" \
+        "${test_json}" \
+        "${compare_report_md}" \
+        "${compare_report_log}"; then
+        compares_succeeded=$((compares_succeeded + 1))
+      else
+        any_failures=1
+      fi
+    fi
+  done
+fi
+
+# ============================================================================
+# Python benchmark pipeline
+# ============================================================================
+
+if [[ "${#PYTHON_FILTERS[@]}" -gt 0 ]]; then
+  echo
+  echo "=== Python Benchmark Pipeline ==="
+  echo
+
+  py_benchmarks_subdir="python/cuda_cccl/benchmarks"
+  base_py_bench_dir="${BASE_PATH}/${py_benchmarks_subdir}"
+  test_py_bench_dir="${TEST_PATH}/${py_benchmarks_subdir}"
+
+  if [[ ! -d "${base_py_bench_dir}" ]]; then
+    die "Python benchmarks directory not found in base tree: ${base_py_bench_dir}"
+  fi
+  if [[ ! -d "${test_py_bench_dir}" ]]; then
+    die "Python benchmarks directory not found in test tree: ${test_py_bench_dir}"
+  fi
+
+  cuda_major="$(detect_cuda_major_version)"
+  echo "Detected CUDA major version: ${cuda_major}"
+
+  base_py_venv="${build_root}/py-base-${build_token}"
+  test_py_venv="${build_root}/py-test-${build_token}"
+
+  setup_python_venv "${base_py_venv}" "${BASE_PATH}" "base" "${artifact_dir}/logs/py.venv.base.log" "${cuda_major}"
+  setup_python_venv "${test_py_venv}" "${TEST_PATH}" "test" "${artifact_dir}/logs/py.venv.test.log" "${cuda_major}"
+
+  select_python_targets "${base_py_bench_dir}" "${test_py_bench_dir}" selected_py_targets
+
+  # Append Python targets to the selected targets metadata file.
+  for py_target_path in "${selected_py_targets[@]}"; do
+    python_path_to_target_name "${py_target_path}" >> "${artifact_dir}/meta/selected_targets.txt"
+  done
+
+  for py_target_path in "${selected_py_targets[@]}"; do
+    py_target_name="$(python_path_to_target_name "${py_target_path}")"
+    base_py_target_run_rc=125
+    test_py_target_run_rc=125
+
+    base_py_json="${artifact_dir}/base/${py_target_name}.json"
+    base_py_md="${artifact_dir}/base/${py_target_name}.md"
+    test_py_json="${artifact_dir}/test/${py_target_name}.json"
+    test_py_md="${artifact_dir}/test/${py_target_name}.md"
+    base_py_run_log="${artifact_dir}/logs/run.base.${py_target_name}.log"
+    test_py_run_log="${artifact_dir}/logs/run.test.${py_target_name}.log"
+    compare_py_report_md="${artifact_dir}/compare/${py_target_name}.md"
+    compare_py_report_log="${artifact_dir}/logs/compare.${py_target_name}.log"
+
+    if run_python_target_for_side \
+      "base" \
+      "${base_py_venv}" \
+      "${base_py_bench_dir}/${py_target_path}" \
+      "${base_py_json}" \
+      "${base_py_md}" \
+      "${base_py_run_log}"; then
+      base_py_target_run_rc=0
+    else
+      base_py_target_run_rc=$?
+      any_failures=1
+    fi
+
+    if run_python_target_for_side \
+      "test" \
+      "${test_py_venv}" \
+      "${test_py_bench_dir}/${py_target_path}" \
+      "${test_py_json}" \
+      "${test_py_md}" \
+      "${test_py_run_log}"; then
+      test_py_target_run_rc=0
+    else
+      test_py_target_run_rc=$?
+      any_failures=1
+    fi
+
+    if [[ "${base_py_target_run_rc}" -eq 0 && "${test_py_target_run_rc}" -eq 0 ]]; then
+      py_compares_attempted=$((py_compares_attempted + 1))
+      if run_python_compare_target \
+        "${py_target_name}" \
+        "${test_py_venv}" \
+        "${base_py_json}" \
+        "${test_py_json}" \
+        "${compare_py_report_md}" \
+        "${compare_py_report_log}"; then
+        py_compares_succeeded=$((py_compares_succeeded + 1))
+      else
+        any_failures=1
+      fi
+    fi
+  done
+fi
+
+# ============================================================================
+# Summary and exit
+# ============================================================================
+
+summary_file="${artifact_dir}/summary.md"
+write_summary "${summary_file}"
+
+echo "Wrote summary: ${summary_file}"
+echo "Benchmark artifacts: ${artifact_dir}"
+echo
+echo "Main summary:"
+cat "${summary_file}"
+echo
+
+if [[ "${any_failures}" -ne 0 ]]; then
+  exit 1
+fi
+
+exit 0
diff --git a/cccl_upstream/ci/bench/parse_bench_matrix.sh b/cccl_upstream/ci/bench/parse_bench_matrix.sh
new file mode 100755
index 00000000..76ce2540
--- /dev/null
+++ b/cccl_upstream/ci/bench/parse_bench_matrix.sh
@@ -0,0 +1,80 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+die() {
+  local message="$1"
+  local code="${2:-2}"
+  echo "${message}" >&2
+  exit "${code}"
+}
+
+usage() {
+  cat <<EOF
+Usage: $0 [bench-yaml-path]
+
+Parse ci/bench.yaml and emit a GitHub Actions strategy matrix JSON object:
+  {"include":[...]}
+
+Each include entry maps one enabled GPU to a benchmark workflow invocation.
+EOF
+}
+
+if [[ "${1:-}" == "-h" || "${1:-}" == "--help" ]]; then
+  usage
+  exit 0
+fi
+
+bench_yaml_path="${1:-ci/bench.yaml}"
+if [[ ! -f "${bench_yaml_path}" ]]; then
+  die "Benchmark config file not found: ${bench_yaml_path}"
+fi
+
+command -v "yq" >/dev/null 2>&1 || die "'yq' is required to parse ${bench_yaml_path}."
+command -v "jq" >/dev/null 2>&1 || die "'jq' is required to build the dispatch matrix."
+
+if ! bench_cfg_json="$(yq -o=json '.benchmarks // {}' "${bench_yaml_path}" 2>&1)"; then
+  die "Failed to parse ${bench_yaml_path} as YAML: ${bench_cfg_json}"
+fi
+
+# Extract CUB and Python filter arrays (default to empty arrays).
+cub_filters_json="$(jq -c '.filters.cub // []' <<<"${bench_cfg_json}")"
+python_filters_json="$(jq -c '.filters.python // []' <<<"${bench_cfg_json}")"
+
+has_cub_filters="$(jq -e 'type == "array" and length > 0 and all(.[]; type == "string")' <<<"${cub_filters_json}" >/dev/null 2>&1 && echo true || echo false)"
+has_python_filters="$(jq -e 'type == "array" and length > 0 and all(.[]; type == "string")' <<<"${python_filters_json}" >/dev/null 2>&1 && echo true || echo false)"
+
+if [[ "${has_cub_filters}" != "true" && "${has_python_filters}" != "true" ]]; then
+  die "${bench_yaml_path} must define at least one string entry in benchmarks.filters.cub or benchmarks.filters.python."
+fi
+
+cub_filters_arg=""
+if [[ "${has_cub_filters}" == "true" ]]; then
+  cub_filters_arg="$(jq -r '.filters.cub | map(@sh) | join(" ")' <<<"${bench_cfg_json}")"
+fi
+
+python_filters_arg=""
+if [[ "${has_python_filters}" == "true" ]]; then
+  python_filters_arg="$(jq -r '.filters.python | map(@sh) | join(" ")' <<<"${bench_cfg_json}")"
+fi
+
+jq -cn \
+  --argjson cfg "${bench_cfg_json}" \
+  --arg cub_filters "${cub_filters_arg}" \
+  --arg python_filters "${python_filters_arg}" \
+  '{
+    "include": [
+      ($cfg.gpus // [])[] as $gpu
+      | {
+          "gpu": $gpu,
+          "launch_args": ($cfg.launch_args // ""),
+          "arch": ($cfg.arch // "native"),
+          "base_ref": ($cfg.base_ref // "origin/main"),
+          "test_ref": ($cfg.test_ref // "HEAD"),
+          "cub_filters": $cub_filters,
+          "python_filters": $python_filters,
+          "nvbench_args": ($cfg.nvbench_args // ""),
+          "nvbench_compare_args": ($cfg.nvbench_compare_args // "")
+        }
+    ]
+  }'
diff --git a/cccl_upstream/ci/build_cccl_c_parallel.sh b/cccl_upstream/ci/build_cccl_c_parallel.sh
new file mode 100755
index 00000000..4ebdacec
--- /dev/null
+++ b/cccl_upstream/ci/build_cccl_c_parallel.sh
@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+# shellcheck source=ci/build_common.sh
+source "$(dirname "${BASH_SOURCE[0]}")/build_common.sh"
+
+print_environment_details
+
+PRESET="cccl-c-parallel"
+
+CMAKE_OPTIONS=("-DCMAKE_CXX_STANDARD=${CXX_STANDARD}" "-DCMAKE_CUDA_STANDARD=${CXX_STANDARD}")
+
+configure_and_build_preset "CCCL C Parallel Library" "$PRESET" "${CMAKE_OPTIONS[@]}"
+
+print_time_summary
diff --git a/cccl_upstream/ci/build_cccl_c_stf.sh b/cccl_upstream/ci/build_cccl_c_stf.sh
new file mode 100755
index 00000000..5775ca7a
--- /dev/null
+++ b/cccl_upstream/ci/build_cccl_c_stf.sh
@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+# shellcheck source=ci/build_common.sh
+source "$(dirname "${BASH_SOURCE[0]}")/build_common.sh"
+
+print_environment_details
+
+PRESET="cccl-c-stf"
+
+CMAKE_OPTIONS=("-DCMAKE_CXX_STANDARD=${CXX_STANDARD}" "-DCMAKE_CUDA_STANDARD=${CXX_STANDARD}")
+
+configure_and_build_preset "CCCL C CUDASTF Library" "$PRESET" "${CMAKE_OPTIONS[@]}"
+
+print_time_summary
diff --git a/cccl_upstream/ci/build_common.sh b/cccl_upstream/ci/build_common.sh
new file mode 100755
index 00000000..43df0934
--- /dev/null
+++ b/cccl_upstream/ci/build_common.sh
@@ -0,0 +1,501 @@
+#!/usr/bin/env bash
+
+set -eo pipefail
+
+if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
+  echo "This script must be sourced, not executed directly." >&2
+  exit 1
+fi
+
+# Ensure the script is being executed in its containing directory
+cd "$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )";
+
+# Script defaults
+VERBOSE=${VERBOSE:-}
+HOST_COMPILER=${CXX:-g++} # $CXX if set, otherwise `g++`
+CXX_STANDARD=17
+CUDA_COMPILER=${CUDACXX:-nvcc} # $CUDACXX if set, otherwise `nvcc`
+CUDA_ARCHS= # Empty, use presets by default.
+GLOBAL_CMAKE_OPTIONS=()
+DISABLE_CUB_BENCHMARKS= # Enable to force-disable building CUB benchmarks.
+PEDANTIC=${PEDANTIC:-} # Enable strict warnings. Default: on in CI, off locally.
+CONFIGURE_ONLY=false
+CTEST_PARALLEL_LEVEL=1
+
+# Check if the correct number of arguments has been provided
+function usage {
+    echo "Usage: $0 [OPTIONS]"
+    echo
+    echo "The PARALLEL_LEVEL environment variable controls the amount of build parallelism. Default is the number of cores minus one."
+    echo
+    echo "Options:"
+    echo "  -v/-verbose: enable shell echo for debugging"
+    echo "  -configure: Only run cmake to configure, do not build or test."
+    echo "  -cuda: CUDA compiler (Defaults to \$CUDACXX if set, otherwise nvcc)"
+    echo "  -cxx: Host compiler (Defaults to \$CXX if set, otherwise g++)"
+    echo "  -std: CUDA/C++ standard (Defaults to 17)"
+    echo "  -arch: Target CUDA arches, e.g. \"60-real;70;80-virtual\" (Defaults to value in presets file)"
+    echo "  --test-par: CTest parallel level (Defaults to 1)"
+    echo "  -pedantic/--pedantic: Enable strict warnings-as-errors and expose CCCL header warnings (default in CI)"
+    echo "  -cmake-options: Additional options to pass to CMake"
+    echo
+    echo "Examples:"
+    echo "  $ PARALLEL_LEVEL=8 $0"
+    echo "  $ PARALLEL_LEVEL=8 $0 -cxx g++-9"
+    echo "  $ $0 -cxx clang++-8"
+    echo "  $ $0 -configure -arch 80"
+    echo "  $ $0 -cxx g++-8 -std 14 -arch 80-real -v -cuda /usr/local/bin/nvcc"
+    echo "  $ $0 -cmake-options \"-DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS=-Wfatal-errors\""
+    exit 1
+}
+
+# Check for required dependencies
+function check_required_dependencies() {
+    local missing_deps=()
+
+    # Check for essential tools
+    local required_tools=("cmake" "git" "jq" "ninja" "nproc")
+    for tool in "${required_tools[@]}"; do
+        command -v "$tool" &>/dev/null || missing_deps+=("$tool")
+    done
+
+    if [[ ${#missing_deps[@]} -ne 0 ]]; then
+        echo "❌ Error: Missing required dependencies:" >&2
+        printf "   • %s\n" "${missing_deps[@]}" >&2
+        echo >&2
+        exit 1
+    fi
+}
+
+# Parse options
+
+# Copy the args into a temporary array, since we will modify them and
+# the parent script may still need them.
+args=("$@")
+while [[ "${#args[@]}" -ne 0 ]]; do
+    case "${args[0]}" in
+    -v | --verbose | -verbose) VERBOSE=1; args=("${args[@]:1}");;
+    -configure) CONFIGURE_ONLY=true;   args=("${args[@]:1}");;
+    -cxx)  HOST_COMPILER="${args[1]}"; args=("${args[@]:2}");;
+    -std)  CXX_STANDARD="${args[1]}";  args=("${args[@]:2}");;
+    -cuda) CUDA_COMPILER="${args[1]}"; args=("${args[@]:2}");;
+    -arch) CUDA_ARCHS="${args[1]}";    args=("${args[@]:2}");;
+    --test-par) CTEST_PARALLEL_LEVEL="${args[1]}"; args=("${args[@]:2}");;
+    -pedantic | --pedantic) PEDANTIC=1; args=("${args[@]:1}");;
+    -disable-benchmarks) export DISABLE_CUB_BENCHMARKS=1; args=("${args[@]:1}");;
+    -cmake-options)
+        if [[ -n "${args[1]}" ]]; then
+            IFS=' ' read -ra split_args <<< "${args[1]}"
+            GLOBAL_CMAKE_OPTIONS+=("${split_args[@]}")
+            args=("${args[@]:2}")
+        else
+            echo "Error: No arguments provided for -cmake-options"
+            usage
+            # usage will exit 1 for us, so below exit 1 is unreachable, but it does not
+            # hurt and guards against changes in usage.
+            # shellcheck disable=SC2317
+            exit 1
+        fi
+        ;;
+    -h | -help | --help) usage ;;
+    *) echo "Unrecognized option: ${args[0]}"; usage ;;
+    esac
+done
+
+# Convert to full paths and validate compilers exist:
+function validate_and_resolve_compiler() {
+    local compiler_name="$1"
+    local compiler_var="$2"
+    local compiler_path
+
+    compiler_path=$(command -v "${compiler_var}" 2>/dev/null)
+    if [[ -z "$compiler_path" ]]; then
+        echo "❌ Error: ${compiler_name} '${compiler_var}' not found in PATH" >&2
+        exit 1
+    fi
+
+    echo "$compiler_path"
+}
+
+HOST_COMPILER=$(validate_and_resolve_compiler "Host compiler" "${HOST_COMPILER}")
+CUDA_COMPILER=$(validate_and_resolve_compiler "CUDA compiler" "${CUDA_COMPILER}")
+
+if [[ "$(basename "$CUDA_COMPILER")" == nvcc* ]]; then
+    NVCC_VERSION=$("$CUDA_COMPILER" --version | grep "release" | sed 's/.*, V//')
+    # Verify that we have an X.Y.Z version in case the output format changes:
+    if ! [[ "$NVCC_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
+        echo "❌ Error: Detected nvcc version is not a valid X.Y.Z triple: '$NVCC_VERSION'" >&2
+        echo "$CUDA_COMPILER --version" >&2 || :
+        $CUDA_COMPILER --version >&2 || :
+        exit 1
+    fi
+fi
+
+
+if [[ -n "${CUDA_ARCHS}" ]]; then
+    GLOBAL_CMAKE_OPTIONS+=("-DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCHS}")
+fi
+
+# Default to pedantic mode in CI
+if [[ -z "${PEDANTIC}" && -n "${GITHUB_ACTIONS:-}" ]]; then
+    PEDANTIC=1
+fi
+
+if [[ -n "${PEDANTIC}" ]]; then
+    GLOBAL_CMAKE_OPTIONS+=("-DCCCL_ENABLE_WERROR=ON" "-DCCCL_ENABLE_PRAGMA_SYSTEM_HEADER=OFF")
+else
+    GLOBAL_CMAKE_OPTIONS+=("-DCCCL_ENABLE_WERROR=OFF" "-DCCCL_ENABLE_PRAGMA_SYSTEM_HEADER=ON")
+fi
+
+if [[ -n "$VERBOSE" ]]; then
+    set -x
+fi
+
+# Check for required dependencies
+check_required_dependencies
+
+# Begin processing unsets after option parsing
+set -u
+
+N_CPUS="$(nproc --all --ignore=1)"
+readonly N_CPUS
+readonly PARALLEL_LEVEL="${PARALLEL_LEVEL:=${N_CPUS}}"
+
+if [[ -z ${CCCL_BUILD_INFIX+x} ]]; then
+    CCCL_BUILD_INFIX=""
+fi
+
+mkdir -p ../build
+# Absolute path to cccl/build
+BUILD_ROOT=$(cd "../build" && pwd)
+
+# Absolute path to per-devcontainer build directory
+BUILD_DIR="$BUILD_ROOT/$CCCL_BUILD_INFIX"
+
+# The most recent devcontainer build dir will always be symlinked to cccl/build/latest
+mkdir -p "$BUILD_DIR"
+rm -f "$BUILD_ROOT"/latest
+ln -sf "$BUILD_DIR" "$BUILD_ROOT"/latest
+
+# The more recent preset build dir will always be symlinked to:
+# cccl/preset-latest
+function symlink_latest_preset {
+    local PRESET=$1
+    mkdir -p "$BUILD_DIR/$PRESET"
+    rm -f "$BUILD_ROOT/preset-latest"
+    ln -sf "$BUILD_DIR/$PRESET" "$BUILD_ROOT/preset-latest"
+}
+
+# Now that BUILD_DIR exists, use readlink to canonicalize the path:
+BUILD_DIR=$(readlink -f "${BUILD_DIR}")
+
+# Prepare environment for CMake:
+export CMAKE_BUILD_PARALLEL_LEVEL="$((PARALLEL_LEVEL > N_CPUS ? N_CPUS : PARALLEL_LEVEL))"
+
+export CTEST_PARALLEL_LEVEL
+export CXX="${HOST_COMPILER}"
+export CUDACXX="${CUDA_COMPILER}"
+export CUDAHOSTCXX="${HOST_COMPILER}"
+export CXX_STANDARD
+
+# shellcheck source=ci/pretty_printing.sh
+source ./pretty_printing.sh
+
+# Kill any build / test steps that exceed this time, otherwise CI jobs may be
+# killed by GHA before they can upload logs / artifacts needed to reproduce the timeout.
+# Only applies when running inside GitHub Actions.
+# Note that this is per-build/test limit, not a total timeout for the entire job.
+: "${CCCL_CI_COMMAND_TIMEOUT:=5.5h}"
+
+print_environment_details() {
+  begin_group "⚙️ Environment Details"
+
+  echo "free -h:"
+  free -h || :
+
+  echo "nproc=$(nproc || :)"
+
+  echo "pwd=$(pwd)"
+
+  print_var_values \
+      BUILD_DIR \
+      CXX_STANDARD \
+      CXX \
+      CUDACXX \
+      CUDAHOSTCXX \
+      NVCC_VERSION \
+      CMAKE_BUILD_PARALLEL_LEVEL \
+      CTEST_PARALLEL_LEVEL \
+      CCCL_CI_COMMAND_TIMEOUT \
+      CCCL_CUDA_EXTENDED \
+      CCCL_BUILD_INFIX \
+      PEDANTIC \
+      GLOBAL_CMAKE_OPTIONS \
+      TBB_ROOT
+
+  echo "Current commit is:"
+  git log -1 --format=short || echo "Not a repository"
+
+  if command -v nvidia-smi &> /dev/null; then
+    nvidia-smi
+  else
+    echo "nvidia-smi not found"
+  fi
+
+  if command -v sccache &> /dev/null; then
+    sccache --version
+  else
+    echo "sccache not found"
+  fi
+
+  if command -v cmake &> /dev/null; then
+    cmake --version
+  else
+    echo "cmake not found"
+  fi
+
+  if command -v ctest &> /dev/null; then
+    ctest --version
+  else
+    echo "ctest not found"
+  fi
+
+  end_group "⚙️ Environment Details"
+}
+
+run_ci_timed_command() {
+    local group_name="${1:-}"
+    shift
+    local -a command=("$@")
+
+    if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+        if [[ -n "${CCCL_CI_COMMAND_TIMEOUT}" && "${CCCL_CI_COMMAND_TIMEOUT}" != "0" ]]; then
+            if command -v timeout &> /dev/null; then
+                run_command "${group_name}" timeout "${CCCL_CI_COMMAND_TIMEOUT}" "${command[@]}"
+                return $?
+            fi
+            echo "Warning: timeout not found; running without CI timeout." >&2
+        fi
+    fi
+
+    run_command "${group_name}" "${command[@]}"
+}
+
+fail_if_no_gpu() {
+    if ! nvidia-smi &> /dev/null; then
+        echo "Error: No NVIDIA GPU detected. Please ensure you have an NVIDIA GPU installed and the drivers are properly configured." >&2
+        exit 1
+    fi
+}
+
+function cccl_configure_preset_for_test() {
+    local test_preset=$1
+    local presets_file="${BUILD_ROOT}/../CMakePresets.json"
+    local configure_preset
+
+    if [[ ! -f "${presets_file}" ]]; then
+        echo "Error: CMakePresets.json not found: ${presets_file}" >&2
+        return 1
+    fi
+
+    configure_preset=$(
+        jq -r --arg t "${test_preset}" '
+            .testPresets[] | select(.name == $t) | .configurePreset // empty
+        ' "${presets_file}"
+    )
+
+    if [[ -z "${configure_preset}" ]]; then
+        configure_preset="${test_preset}"
+    fi
+
+    echo "${configure_preset}"
+}
+
+function cccl_smoke_tests_enabled() {
+    local configure_preset=$1
+    local cache_file="${BUILD_DIR}/${configure_preset}/CMakeCache.txt"
+
+    [[ -f "${cache_file}" ]] \
+        && grep -q '^CCCL_ENABLE_CUDA_SMOKE_TESTS:BOOL=ON' "${cache_file}"
+}
+
+function run_cuda_smoke_test() {
+    local BUILD_NAME=$1
+    local test_preset=$2
+
+    local configure_preset
+    configure_preset="$(cccl_configure_preset_for_test "${test_preset}")"
+    local smoke_bin="${BUILD_DIR}/${configure_preset}/bin/cccl.test.cuda_runtime_smoke"
+
+    if [[ -x "${smoke_bin}" ]]; then
+        run_ci_timed_command "CUDA smoke ${BUILD_NAME}" "${smoke_bin}" || return $?
+    elif cccl_smoke_tests_enabled "${configure_preset}"; then
+        echo "Error: CCCL_ENABLE_CUDA_SMOKE_TESTS=ON but smoke binary not found: ${smoke_bin}" >&2
+        return 1
+    fi
+}
+
+function print_test_time_summary()
+{
+    ctest_log=${1}
+
+    if [[ -f "${ctest_log}" ]]; then
+        begin_group "⏱️ Longest Test Steps"
+        # Only print the full output in CI:
+        if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+            cmake -DLOGFILE="${ctest_log}" -P ../cmake/PrintCTestRunTimes.cmake
+        else
+            # `|| :` to avoid `set -o pipefail` from triggering when `head` closes the pipe before `cmake` finishes.
+            # Otherwise the script will exit early with status 141 (SIGPIPE).
+            cmake -DLOGFILE="${ctest_log}" -P ../cmake/PrintCTestRunTimes.cmake | head -n 15 || :
+        fi
+        end_group "⏱️ Longest Test Steps"
+    fi
+}
+
+function configure_preset()
+{
+    local BUILD_NAME=$1
+    local PRESET=$2
+    shift 2
+    local CMAKE_OPTIONS=("$@")
+
+    local GROUP_NAME="🛠️  CMake Configure ${BUILD_NAME}"
+
+    symlink_latest_preset "$PRESET"
+
+    pushd .. > /dev/null
+    if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+      # Retry 5 times with 30 seconds between attempts to try to WAR network issues during CPM fetch on CI runners:
+      export RUN_COMMAND_RETRY_PARAMS=(5 30)
+    fi
+    status=0
+    SCCACHE_NO_DIST_COMPILE=1 run_command "$GROUP_NAME" cmake --preset="$PRESET" --log-level=VERBOSE "${CMAKE_OPTIONS[@]}" "${GLOBAL_CMAKE_OPTIONS[@]}" || status=$?
+    if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+        unset RUN_COMMAND_RETRY_PARAMS
+    fi
+    popd > /dev/null
+
+    if $CONFIGURE_ONLY; then
+        echo "${BUILD_NAME} configuration complete:"
+        echo "  Exit code:       ${status}"
+        echo "  CMake Preset:    ${PRESET}"
+        echo "  CMake Options:   ${CMAKE_OPTIONS[*]}"
+        echo "  Build Directory: ${BUILD_DIR}/${PRESET}"
+        exit "$status"
+    fi
+
+    return "$status"
+}
+
+function build_preset() {
+    local BUILD_NAME=$1
+    local PRESET=$2
+    # shellcheck disable=SC2034
+    local green="1;32"
+    # shellcheck disable=SC2034
+    local red="1;31"
+    local GROUP_NAME="🏗️  Build ${BUILD_NAME}"
+    shift 2
+    local BUILD_COMMANDS=("$@")
+
+    symlink_latest_preset "$PRESET"
+
+    if $CONFIGURE_ONLY; then
+        return 0
+    fi
+
+    local preset_dir="${BUILD_DIR}/${PRESET}"
+    local sccache_json="${preset_dir}/sccache_stats.json"
+    local memmon_log="${preset_dir}/memmon.log"
+
+    sccache -z > /dev/null || :
+
+    # Track memory usage on CI:
+    if [[ -n "${GITHUB_ACTIONS:-}" || -n "${MEMMON:-}" ]]; then
+      util/memmon.sh --start \
+          --log-threshold "${MEMMON_LOG_THRESHOLD:-2}" \
+          --print-threshold "${MEMMON_PRINT_THRESHOLD:-5}" \
+          --log-file "$memmon_log" \
+          --poll "${MEMMON_POLL_INTERVAL:-5}" \
+          || :
+    fi
+
+    pushd .. > /dev/null
+    status=0
+    run_ci_timed_command "$GROUP_NAME" cmake --build --parallel "$PARALLEL_LEVEL" --preset="$PRESET" ${VERBOSE:+-v} "${BUILD_COMMANDS[@]}" || status=$?
+    popd > /dev/null
+
+    if [[ -n "${GITHUB_ACTIONS:-}" || -n "${MEMMON:-}" ]]; then
+      util/memmon.sh --stop || :
+      run_command "📝 Memory Monitor Log" head -n20 "$memmon_log" || :
+    fi
+
+    # Only print detailed stats in actions workflow
+    if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+        sccache --show-adv-stats --stats-format=json > "${sccache_json}" || :
+        run_command "📊 sccache stats" sccache --show-adv-stats || :
+
+        begin_group "🥷 ninja build times"
+        echo "The \"weighted\" time is the elapsed time of each build step divided by the number
+              of tasks that were running in parallel. This makes it an excellent approximation
+              of how \"important\" a slow step was. A link that is entirely or mostly serialized
+              will have a weighted time that is the same or similar to its elapsed time. A
+              compile that runs in parallel with 999 other compiles will have a weighted time
+              that is tiny."
+        ./ninja_summary.py -C "${BUILD_DIR}"/"${PRESET}" || echo "Warning: ninja_summary.py failed to execute properly."
+        end_group
+    else
+      sccache -s || :
+    fi
+
+    return "$status"
+}
+
+function test_preset()
+{
+    local BUILD_NAME=$1
+    local PRESET=$2
+    local GPU_REQUIRED=${3:-true}
+
+    symlink_latest_preset "$PRESET"
+
+    if $CONFIGURE_ONLY; then
+        return 0
+    fi
+
+    if $GPU_REQUIRED; then
+        fail_if_no_gpu
+        if [[ -z "${CCCL_SKIP_CI_SMOKE:-}" ]]; then
+            run_cuda_smoke_test "${BUILD_NAME}" "${PRESET}" || return $?
+        fi
+    fi
+
+    local GROUP_NAME="🚀  Test ${BUILD_NAME}"
+
+    local preset_dir="${BUILD_DIR}/${PRESET}"
+    local ctest_log="${preset_dir}/ctest.log"
+
+    pushd .. > /dev/null
+    status=0
+    run_ci_timed_command "$GROUP_NAME" ctest --output-on-failure --output-log "${ctest_log}" --preset="$PRESET" || status=$?
+    popd > /dev/null
+
+    print_test_time_summary "${ctest_log}"
+
+    return "$status"
+}
+
+function configure_and_build_preset()
+{
+    local BUILD_NAME=$1
+    local PRESET=$2
+    shift 2
+    local CMAKE_OPTIONS=("$@")
+
+    configure_preset "$BUILD_NAME" "$PRESET" "${CMAKE_OPTIONS[@]}"
+
+    if ! $CONFIGURE_ONLY; then
+        build_preset "$BUILD_NAME" "$PRESET"
+    fi
+}
diff --git a/cccl_upstream/ci/build_compile_time_bench.sh b/cccl_upstream/ci/build_compile_time_bench.sh
new file mode 100755
index 00000000..c90b0215
--- /dev/null
+++ b/cccl_upstream/ci/build_compile_time_bench.sh
@@ -0,0 +1,370 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+repo_root="$(cd "${ci_dir}/.." && pwd)"
+tool_dir="${repo_root}/ci/compile_time"
+
+default_preset="all-dev"
+preset="${default_preset}"
+skip_configure=0
+skip_build=0
+prepare_perfetto=1
+run_ctadvisor=0
+write_tu_csv=1
+explicit_tu_csv=0
+tu_csv=""
+perfetto_output_dir=""
+baseline_ref=""
+baseline_worktree=""
+max_detail_len=180
+cloc_processes=0
+
+declare -a build_targets=()
+declare -a event_args=()
+declare -a common_args=()
+declare -a default_build_targets=(
+  "cub.headers.base"
+  "thrust.cpp.cuda.headers.base"
+  "libcudacxx.test.public_headers"
+)
+declare -a bench_cmake_args=(
+  "-DCMAKE_EXPORT_COMPILE_COMMANDS=ON"
+  "-DCMAKE_CUDA_COMPILER_LAUNCHER="
+  "-DCMAKE_CXX_COMPILER_LAUNCHER="
+  "-DCCCL_ENABLE_TESTING=OFF"
+  "-DCCCL_ENABLE_EXAMPLES=OFF"
+  "-DCCCL_ENABLE_BENCHMARKS=OFF"
+  "-DCCCL_ENABLE_C_PARALLEL=OFF"
+  "-DCCCL_ENABLE_C_EXPERIMENTAL_STF=OFF"
+  "-DCUB_ENABLE_TESTING=OFF"
+  "-DCUB_ENABLE_EXAMPLES=OFF"
+  "-DTHRUST_ENABLE_TESTING=OFF"
+  "-DTHRUST_ENABLE_EXAMPLES=OFF"
+  "-DTHRUST_MULTICONFIG_WORKLOAD=SMALL"
+  "-DTHRUST_MULTICONFIG_ENABLE_SYSTEM_OMP=OFF"
+  "-DTHRUST_MULTICONFIG_ENABLE_SYSTEM_TBB=OFF"
+  "-Dcudax_ENABLE_TESTING=OFF"
+  "-Dcudax_ENABLE_EXAMPLES=OFF"
+  "-Dcudax_ENABLE_CUDASTF=OFF"
+  "-Dcudax_ENABLE_CUFILE=OFF"
+  "-DCCCL_COMPILE_TIME_SAVE_PREPROCESSED_TUS=ON"
+  "-DCCCL_COMPILE_TIME_GENERATE_DEVICE_TIME_TRACES=ON"
+)
+
+usage() {
+  cat <<'EOF'
+Usage: ci/build_compile_time_bench.sh [options] [-- <event summary args>]
+
+Build options:
+  -preset <name>              CMake configure preset (default: all-dev)
+  -cmake-options <args>       Extra CMake configure options handled by ci/build_common.sh
+  -target <name>              Build target; repeatable
+                              (default: public include-check target set)
+  -baseline-ref <commit-ish>  Build this commit-ish as a comparison baseline
+  -skip-configure             Do not run cmake configure
+  -skip-build                 Do not run cmake --build
+  -cuda, -cxx, -std, -arch    Common compiler/standard/arch options from ci/build_common.sh
+
+Summary options:
+  -tu-csv <path>              Generated-TU summary CSV
+                              (default: <preset-build-dir>/compile_time/tu_summary.csv)
+  -no-tu-csv                  Do not write the generated-TU summary CSV
+  -prepare-perfetto           Prepare Perfetto-friendly trace copies (default)
+  -no-prepare-perfetto        Skip Perfetto trace preparation
+  -perfetto-output <path>     Perfetto trace output directory
+                              (default: <preset-build-dir>/compile_time/perfetto_traces)
+  -max-detail-len <n>         Max promoted detail length for Perfetto traces (default: 180)
+  -cloc-processes <n>         cloc process count for generated-TU summary CSV
+  -ctadvisor                  Print ctadvisor report for raw traces
+
+Event summary args:
+  Arguments after '--' are forwarded to ci/compile_time/summarize_events.py
+  after the raw trace directory. If omitted, the default summary is:
+    -f file-processing -e -n 15
+  Pass --slices <json-file> after '--' to emit multiple event report slices
+  and an event_reports/summary.json manifest.
+  With -baseline-ref, comparison-only options such as --threshold <seconds>
+  may also be passed after '--'. Baseline raw traces are preserved under
+  <preset-build-dir>/compile_time/baseline_raw_traces.
+
+Examples:
+  ci/build_compile_time_bench.sh
+  ci/build_compile_time_bench.sh -target cudax.headers.basic.no_stf -- -f scanning-function-body -i -n 20
+  ci/build_compile_time_bench.sh -preset cub -target cub.headers.base -- -f template-instantiation -e -n 15
+  ci/build_compile_time_bench.sh -baseline-ref origin/main -- -f file-processing -e --threshold 0.001
+EOF
+}
+
+status() { echo "[compile-time-bench] $*" >&2; }
+
+require_command() {
+  local command_name="$1"
+  command -v "${command_name}" >/dev/null \
+    || { echo "error: ${command_name} not found" >&2; exit 1; }
+}
+
+cleanup_baseline_worktree() {
+  if [[ -n "${baseline_worktree}" && -d "${baseline_worktree}" ]]; then
+    git -C "${repo_root}" worktree remove --force "${baseline_worktree}" >/dev/null 2>&1 \
+      || rm -rf "${baseline_worktree}"
+  fi
+}
+
+cleanup_baseline_worktree_and_exit() {
+  local exit_code="$1"
+  trap - EXIT HUP INT TERM
+  cleanup_baseline_worktree
+  exit "${exit_code}"
+}
+
+install_baseline_cleanup_traps() {
+  trap cleanup_baseline_worktree EXIT
+  trap 'cleanup_baseline_worktree_and_exit 129' HUP
+  trap 'cleanup_baseline_worktree_and_exit 130' INT
+  trap 'cleanup_baseline_worktree_and_exit 143' TERM
+}
+
+overlay_current_bench_file() {
+  local rel_path="$1"
+  mkdir -p "$(dirname "${baseline_worktree}/${rel_path}")"
+  rm -rf "${baseline_worktree:?}/${rel_path}"
+  ln -s "${repo_root}/${rel_path}" "${baseline_worktree}/${rel_path}"
+}
+
+overlay_current_bench_logic() {
+  overlay_current_bench_file "ci"
+  overlay_current_bench_file "CMakePresets.json"
+  overlay_current_bench_file "cmake/CCCLGenerateHeaderTests.cmake"
+}
+
+for arg in "$@"; do
+  case "$arg" in
+    -h|-help|--help) usage; exit 0 ;;
+    *) ;;
+  esac
+done
+
+new_args="$("${ci_dir}/util/extract_switches.sh" \
+  -skip-configure \
+  -skip-build \
+  -no-tu-csv \
+  -prepare-perfetto \
+  -no-prepare-perfetto \
+  -ctadvisor \
+  -- "$@")"
+
+declare -a new_args="(${new_args})"
+set -- "${new_args[@]}"
+while true; do
+  case "$1" in
+    -skip-configure) skip_configure=1; shift ;;
+    -skip-build) skip_build=1; shift ;;
+    -no-tu-csv) write_tu_csv=0; shift ;;
+    -prepare-perfetto) prepare_perfetto=1; shift ;;
+    -no-prepare-perfetto) prepare_perfetto=0; shift ;;
+    -ctadvisor) run_ctadvisor=1; shift ;;
+    --) shift; break ;;
+    *) echo "Unknown argument: $1" >&2; usage; exit 1 ;;
+  esac
+done
+
+while (($#)); do
+  case "$1" in
+    -preset) preset="$2"; shift 2 ;;
+    -target) build_targets+=("$2"); shift 2 ;;
+    -baseline-ref) baseline_ref="$2"; shift 2 ;;
+    -tu-csv) explicit_tu_csv=1; write_tu_csv=1; tu_csv="$2"; shift 2 ;;
+    -perfetto-output) perfetto_output_dir="$2"; shift 2 ;;
+    -max-detail-len) max_detail_len="$2"; shift 2 ;;
+    -cloc-processes) cloc_processes="$2"; shift 2 ;;
+    --) shift; event_args+=("$@"); break ;;
+    *) common_args+=("$1"); shift ;;
+  esac
+done
+
+set -- "${common_args[@]}"
+# shellcheck source=ci/build_common.sh
+source "${ci_dir}/build_common.sh"
+
+current_build_root="${BUILD_ROOT}"
+current_build_dir="${BUILD_DIR}"
+
+build_root_for_source() {
+  local source_root="$1"
+  mkdir -p "${source_root}/build"
+  (cd "${source_root}/build" && pwd)
+}
+
+build_dir_for_source() {
+  local source_root="$1"
+  local build_root="$2"
+  local build_dir="${build_root}/${CCCL_BUILD_INFIX}"
+  mkdir -p "${build_dir}"
+  readlink -f "${build_dir}"
+}
+
+run_bench_build() {
+  local source_root="$1"
+  local build_name="$2"
+  local build_root="$3"
+  local build_dir="$4"
+
+  (
+    # build_common.sh keeps the active build tree in these globals; override
+    # them only inside this subshell so current-tree state is restored after
+    # each build.
+    # shellcheck disable=SC2030
+    BUILD_ROOT="${build_root}"
+    # shellcheck disable=SC2030
+    BUILD_DIR="${build_dir}"
+    cd "${source_root}/ci"
+
+    if (( ! skip_configure )); then
+      status "Configuring preset '${preset}' with compile-time bench instrumentation (${build_name})..."
+      configure_preset "${build_name}" "${preset}" "${bench_cmake_args[@]}"
+    fi
+
+    if (( ! skip_build )); then
+      status "Building target(s) (${build_name}): ${build_targets[*]}"
+      build_preset "${build_name}" "${preset}" --target "${build_targets[@]}"
+    fi
+  )
+}
+
+prepare_perfetto_traces() {
+  local input_dir="$1"
+  local output_dir="$2"
+  local trace_repo_root="$3"
+  local label="$4"
+
+  status "Preparing Perfetto trace copies (${label})..."
+  rm -rf "${output_dir}"
+  "${tool_dir}/prepare_traces.py" \
+    --input "${input_dir}" \
+    --output "${output_dir}" \
+    --repo-root "${trace_repo_root}" \
+    --max-detail-len "${max_detail_len}"
+  status "Perfetto traces (${label}): ${output_dir}"
+}
+
+if ((${#build_targets[@]} == 0)); then
+  build_targets=("${default_build_targets[@]}")
+fi
+
+preset_build_dir="${current_build_dir}/${preset}"
+baseline_trace_dir=""
+
+if [[ -n "${baseline_ref}" ]]; then
+  baseline_commit="$(git -C "${repo_root}" rev-parse --verify "${baseline_ref}^{commit}")"
+  baseline_worktree="$(mktemp -d "${TMPDIR:-/tmp}/cccl-compile-time-baseline.XXXXXX")"
+  rmdir "${baseline_worktree}"
+  install_baseline_cleanup_traps
+  status "Creating baseline worktree for ${baseline_ref} (${baseline_commit})..."
+  git -C "${repo_root}" worktree add --detach "${baseline_worktree}" "${baseline_commit}" >/dev/null
+  overlay_current_bench_logic
+
+  baseline_build_root="$(build_root_for_source "${baseline_worktree}")"
+  baseline_build_dir="$(build_dir_for_source "${baseline_worktree}" "${baseline_build_root}")"
+  baseline_trace_dir="${baseline_build_dir}/${preset}/compile_time/raw_traces"
+fi
+
+report_root="${preset_build_dir}/compile_time"
+trace_dir="${report_root}/raw_traces"
+baseline_artifact_trace_dir="${report_root}/baseline_raw_traces"
+event_output_dir="${report_root}/event_reports"
+tu_csv="${tu_csv:-${report_root}/tu_summary.csv}"
+perfetto_output_dir="${perfetto_output_dir:-${report_root}/perfetto_traces}"
+if [[ -n "${baseline_ref}" && "${explicit_tu_csv}" -eq 0 ]]; then
+  write_tu_csv=0
+fi
+
+require_command python3
+if (( write_tu_csv )); then
+  require_command cloc
+fi
+if (( run_ctadvisor )); then
+  require_command ctadvisor
+fi
+
+run_bench_build "${repo_root}" "Compile-time Bench (current)" "${current_build_root}" "${current_build_dir}"
+if $CONFIGURE_ONLY; then
+  exit 0
+fi
+
+if [[ -n "${baseline_ref}" ]]; then
+  run_bench_build \
+    "${baseline_worktree}" \
+    "Compile-time Bench (baseline)" \
+    "${baseline_build_root}" \
+    "${baseline_build_dir}"
+fi
+
+shopt -s nullglob globstar
+trace_paths=("${trace_dir}"/**/*.json)
+(( ${#trace_paths[@]} > 0 )) \
+  || { echo "error: no device-time-trace JSON files found under ${trace_dir}" >&2; exit 1; }
+if [[ -n "${baseline_ref}" ]]; then
+  baseline_trace_paths=("${baseline_trace_dir}"/**/*.json)
+  (( ${#baseline_trace_paths[@]} > 0 )) \
+    || { echo "error: no device-time-trace JSON files found under ${baseline_trace_dir}" >&2; exit 1; }
+
+  status "Copying baseline raw traces to ${baseline_artifact_trace_dir}..."
+  rm -rf "${baseline_artifact_trace_dir}"
+  mkdir -p "${baseline_artifact_trace_dir}"
+  cp -a "${baseline_trace_dir}/." "${baseline_artifact_trace_dir}/"
+fi
+
+if (( prepare_perfetto )); then
+  if [[ -n "${baseline_ref}" ]]; then
+    rm -rf "${perfetto_output_dir}"
+    prepare_perfetto_traces \
+      "${trace_dir}" \
+      "${perfetto_output_dir}/current" \
+      "${repo_root}" \
+      "current"
+    prepare_perfetto_traces \
+      "${baseline_trace_dir}" \
+      "${perfetto_output_dir}/baseline" \
+      "${baseline_worktree}" \
+      "baseline"
+  else
+    prepare_perfetto_traces \
+      "${trace_dir}" \
+      "${perfetto_output_dir}" \
+      "${repo_root}" \
+      "current"
+  fi
+fi
+
+if (( write_tu_csv )); then
+  status "Writing generated-TU summary CSV..."
+  "${tool_dir}/summarize_tus.py" \
+    --build-dir "${preset_build_dir}" \
+    --output-csv "${tu_csv}" \
+    --cloc-processes "${cloc_processes}"
+  status "Generated-TU summary CSV: ${tu_csv}"
+fi
+
+declare -a summary_args=("${event_args[@]}")
+if ((${#summary_args[@]} == 0)); then
+  summary_args=(-f file-processing -e -n 15)
+fi
+summary_args+=(-o "${event_output_dir}")
+if [[ -n "${baseline_ref}" ]]; then
+  summary_args+=(
+    --baseline-dir "${baseline_trace_dir}"
+    --baseline-repo-root "${baseline_worktree}"
+  )
+fi
+status "Writing event summary..."
+"${tool_dir}/summarize_events.py" "${trace_dir}" "${summary_args[@]}"
+
+if (( run_ctadvisor )); then
+  status "Running ctadvisor over ${#trace_paths[@]} trace(s)..."
+  ctadvisor \
+    --trace-file-path "${trace_dir}" \
+    --header-advisor-entries 20 \
+    --thread-number "$(nproc --all --ignore=2)"
+fi
diff --git a/cccl_upstream/ci/build_cub.sh b/cccl_upstream/ci/build_cub.sh
new file mode 100755
index 00000000..d647e35b
--- /dev/null
+++ b/cccl_upstream/ci/build_cub.sh
@@ -0,0 +1,111 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+NO_LID=false
+LID0=false
+LID1=false
+LID2=false
+ARTIFACT_TAGS=()
+
+ci_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+
+new_args="$("${ci_dir}/util/extract_switches.sh" \
+  -no-lid \
+  -lid0 \
+  -lid1 \
+  -lid2 \
+  -- "$@")"
+
+declare -a new_args="(${new_args})"
+set -- "${new_args[@]}"
+while true; do
+  case "$1" in
+  -no-lid)
+    ARTIFACT_TAGS+=("no_lid")
+    NO_LID=true
+    shift
+    ;;
+  -lid0)
+    ARTIFACT_TAGS+=("lid_0")
+    LID0=true
+    shift
+    ;;
+  -lid1)
+    ARTIFACT_TAGS+=("lid_1")
+    LID1=true
+    shift
+    ;;
+  -lid2)
+    ARTIFACT_TAGS+=("lid_2")
+    LID2=true
+    shift
+    ;;
+  --)
+    shift
+    break
+    ;;
+  *)
+    echo "Unknown argument: $1"
+    exit 1
+    ;;
+  esac
+done
+
+# shellcheck source=ci/build_common.sh
+source "${ci_dir}/build_common.sh"
+
+print_environment_details
+
+ENABLE_CCCL_BENCHMARKS="false"
+ENABLE_CUB_RDC="false"
+
+if [[ "$CUDA_COMPILER" == *nvcc* ]]; then
+    ENABLE_CUB_RDC="true"
+    NVCC_VERSION=$($CUDA_COMPILER --version | grep release | awk '{print $6}' | cut -c2-)
+    if [[ -n "${DISABLE_CUB_BENCHMARKS}" ]]; then
+        echo "Benchmarks have been forcefully disabled."
+    else
+        ENABLE_CCCL_BENCHMARKS="true"
+        echo "nvcc version is $NVCC_VERSION. Building CUB benchmarks."
+    fi
+else
+    echo "Not building with NVCC, disabling RDC and benchmarks."
+fi
+
+if [[ "$HOST_COMPILER" == *icpc* || "$HOST_COMPILER" == *nvhpc* ]]; then
+    ENABLE_CCCL_BENCHMARKS="false"
+fi
+
+PRESET="cub"
+if $NO_LID; then
+    PRESET="cub-nolid"
+elif $LID0; then
+    PRESET="cub-lid0"
+elif $LID1; then
+    PRESET="cub-lid1"
+elif $LID2; then
+    PRESET="cub-lid2"
+fi
+
+CMAKE_OPTIONS=(
+    "-DCMAKE_CXX_STANDARD=$CXX_STANDARD"
+    "-DCMAKE_CUDA_STANDARD=$CXX_STANDARD"
+    "-DCCCL_ENABLE_BENCHMARKS=$ENABLE_CCCL_BENCHMARKS"
+    "-DCUB_ENABLE_RDC_TESTS=$ENABLE_CUB_RDC"
+)
+
+configure_and_build_preset "CUB" "$PRESET" "${CMAKE_OPTIONS[@]}"
+
+# Create test artifacts:
+if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+    if [[ ${#ARTIFACT_TAGS[@]} -gt 0 ]]; then
+        run_command "📦  Packaging test artifacts" \
+            "${ci_dir}/upload_cub_test_artifacts.sh" \
+            "${ARTIFACT_TAGS[@]}"
+    else
+        run_command "📦  Packaging test artifacts" "${ci_dir}/upload_cub_test_artifacts.sh"
+    fi
+fi
+
+print_time_summary
diff --git a/cccl_upstream/ci/build_cuda_cccl_python.sh b/cccl_upstream/ci/build_cuda_cccl_python.sh
new file mode 100755
index 00000000..a5994942
--- /dev/null
+++ b/cccl_upstream/ci/build_cuda_cccl_python.sh
@@ -0,0 +1,184 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+usage="Usage: $0 -py-version <python_version> [additional options...]"
+
+# shellcheck source=ci/util/python/common_arg_parser.sh
+source "$ci_dir/util/python/common_arg_parser.sh"
+parse_python_args "$@"
+
+# Check if py_version was provided (this script requires it)
+require_py_version "$usage" || exit 1
+
+echo "Docker socket: " "$(ls /var/run/docker.sock)"
+
+if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+  # Prepare mount points etc for getting artifacts in/out of the container.
+  # shellcheck source=ci/util/artifacts/common.sh
+  source "$ci_dir/util/artifacts/common.sh"
+  # Note that these mounts use the runner (not the devcontainer) filesystem for
+  # source directories because of docker-out-of-docker quirks.
+  # The workflow-job GH actions make sure that they exist before running any
+  # scripts.
+  action_mounts=(
+    --mount "type=bind,source=${ARTIFACT_ARCHIVES},target=${ARTIFACT_ARCHIVES}"
+    --mount "type=bind,source=${ARTIFACT_UPLOAD_STAGE},target=${ARTIFACT_UPLOAD_STAGE}"
+  )
+else
+  # If not running in GitHub Actions, we don't need to set up artifact mounts.
+  action_mounts=()
+fi
+
+# cuda_cccl must be built in a container that can produce manylinux wheels,
+# and has the CUDA toolkit installed. We use the rapidsai/ci-wheel image for this.
+# We build separate wheels using separate containers for each CUDA version,
+# then merge them into a single wheel.
+
+readonly cuda12_version=12.9.1
+readonly cuda13_version=13.1.1
+readonly devcontainer_version=26.04
+readonly devcontainer_distro=rockylinux8
+# Use a baseline Python tag for the rapidsai ci-wheel image. The requested
+# py_version is installed inside the container by setup_python_env (uv).
+# Pinning the image tag avoids relying on a per-py_version image being
+# published (e.g. py3.14 images may not yet exist).
+readonly devcontainer_python_version=3.10
+
+if [[ "$(uname -m)" == "aarch64" ]]; then
+  cuda12_image="rapidsai/ci-wheel:${devcontainer_version}-cuda${cuda12_version}-${devcontainer_distro}-py${devcontainer_python_version}-arm64"
+  cuda13_image="rapidsai/ci-wheel:${devcontainer_version}-cuda${cuda13_version}-${devcontainer_distro}-py${devcontainer_python_version}-arm64"
+else
+  cuda12_image="rapidsai/ci-wheel:${devcontainer_version}-cuda${cuda12_version}-${devcontainer_distro}-py${devcontainer_python_version}"
+  cuda13_image="rapidsai/ci-wheel:${devcontainer_version}-cuda${cuda13_version}-${devcontainer_distro}-py${devcontainer_python_version}"
+fi
+# shellcheck disable=SC2034
+readonly cuda12_image
+# shellcheck disable=SC2034
+readonly cuda13_image
+
+mkdir -p wheelhouse
+
+# Shared caches across the cu12 + cu13 wheel builds. Both jobs compile an
+# identical LLVM/clang tree (LLVM has no CUDA dep), so a shared ccache cuts
+# the second build's LLVM phase from ~10 min to under 2 min; a shared CPM
+# source cache skips the second LLVM git clone entirely.
+#
+# The `mkdir`s run inside the (dev)container where only the container-side
+# paths are visible. The docker bind-mount uses the host-side paths
+# (${HOST_WORKSPACE}) since the inner docker daemon is the host's.
+mkdir -p ./.ccache ./.cpm-cache
+host_ccache_dir="${HOST_WORKSPACE:?}/.ccache"
+host_cpm_cache_dir="${HOST_WORKSPACE:?}/.cpm-cache"
+
+for ctk in 12 13; do
+  image="cuda${ctk}_image"
+  image="${!image}"
+  echo "::group::⚒️ Building CUDA $ctk wheel on $image"
+  (
+    set -x
+    docker pull "$image"
+    docker run --rm -i \
+        --workdir /workspace/python/cuda_cccl \
+        --mount "type=bind,source=${HOST_WORKSPACE:?},target=/workspace/" \
+        --mount "type=bind,source=${host_ccache_dir},target=/root/.ccache" \
+        --mount "type=bind,source=${host_cpm_cache_dir},target=/root/.cpm-cache" \
+        "${action_mounts[@]}" \
+        --env "py_version=${py_version}" \
+        --env "GITHUB_ACTIONS=${GITHUB_ACTIONS:-}" \
+        --env "GITHUB_RUN_ID=${GITHUB_RUN_ID:-}" \
+        --env "JOB_ID=${JOB_ID:-}" \
+        --env "CCCL_PYTHON_USE_V2=${CCCL_PYTHON_USE_V2:-}" \
+        --env "CCCL_C_PARALLEL_SANITIZE_THREAD=${CCCL_C_PARALLEL_SANITIZE_THREAD:-}" \
+        --env "CCACHE_DIR=/root/.ccache" \
+        --env "CPM_SOURCE_CACHE=/root/.cpm-cache" \
+        "$image" \
+        /workspace/ci/build_cuda_cccl_wheel.sh
+    # Prevent GHA runners from exhausting available storage with leftover images:
+    if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+      docker rmi -f "$image"
+    fi
+  )
+  echo "::endgroup::"
+done
+
+echo "Merging CUDA wheels..."
+
+# Set up a Python environment for the merge/repair steps.
+source "$ci_dir/pyenv_helper.sh"
+setup_python_env "${py_version}"
+
+# Needed for unpacking and repacking wheels.
+python -m pip install wheel
+
+# Find the built wheels
+cu12_wheel=$(find wheelhouse -name "*cu12*.whl" | head -1)
+cu13_wheel=$(find wheelhouse -name "*cu13*.whl" | head -1)
+
+if [[ -z "$cu12_wheel" ]]; then
+  echo "Error: CUDA 12 wheel not found in wheelhouse/"
+  ls -la wheelhouse/
+  exit 1
+fi
+
+if [[ -z "$cu13_wheel" ]]; then
+  echo "Error: CUDA 13 wheel not found in wheelhouse/"
+  ls -la wheelhouse/
+  exit 1
+fi
+
+echo "Found CUDA 12 wheel: $cu12_wheel"
+echo "Found CUDA 13 wheel: $cu13_wheel"
+
+# Merge the wheels
+python python/cuda_cccl/merge_cuda_wheels.py "$cu12_wheel" "$cu13_wheel" --output-dir wheelhouse_merged
+
+# A ThreadSanitizer wheel links libtsan; keep it external (excluded) so it is
+# NOT bundled -- the TSan test lane LD_PRELOADs the runner's matching libtsan
+# instead. Harmless for normal builds (the .so has no libtsan dependency).
+tsan_exclude=()
+if [[ "${CCCL_C_PARALLEL_SANITIZE_THREAD:-}" =~ ^(1|true|TRUE|on|ON)$ ]]; then
+  tsan_exclude=(--exclude 'libtsan.so.2')
+fi
+
+# Install auditwheel and repair the merged wheel
+python -m pip install patchelf auditwheel
+for wheel in wheelhouse_merged/cuda_cccl-*.whl; do
+    echo "Repairing merged wheel: $wheel"
+    python -m auditwheel repair \
+        --exclude 'libnvrtc.so.12' \
+        --exclude 'libnvrtc.so.13' \
+        --exclude 'libnvJitLink.so.12' \
+        --exclude 'libnvJitLink.so.13' \
+        --exclude 'libcudart.so.12' \
+        --exclude 'libcudart.so.13' \
+        --exclude 'libcuda.so.1' \
+        "${tsan_exclude[@]}" \
+        "$wheel" \
+        --wheel-dir wheelhouse_final
+done
+
+# Clean up intermediate files and move only the final merged wheel to wheelhouse
+rm -rf wheelhouse/*  # Clean existing wheelhouse
+mkdir -p wheelhouse
+
+# Move only the final repaired merged wheel
+if ls wheelhouse_final/cuda_cccl-*.whl 1> /dev/null 2>&1; then
+    mv wheelhouse_final/cuda_cccl-*.whl wheelhouse/
+    echo "Final merged wheel moved to wheelhouse"
+else
+    echo "No final repaired wheel found, moving unrepaired merged wheel"
+    mv wheelhouse_merged/cuda_cccl-*.whl wheelhouse/
+fi
+
+# Clean up temporary directories
+rm -rf wheelhouse_merged wheelhouse_final
+
+echo "Final wheels in wheelhouse:"
+ls -la wheelhouse/
+
+if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+  wheel_artifact_name="$(ci/util/workflow/get_wheel_artifact_name.sh)"
+  ci/util/artifacts/upload.sh "$wheel_artifact_name" 'wheelhouse/.*'
+fi
diff --git a/cccl_upstream/ci/build_cuda_cccl_python_tsan.sh b/cccl_upstream/ci/build_cuda_cccl_python_tsan.sh
new file mode 100755
index 00000000..cec4273e
--- /dev/null
+++ b/cccl_upstream/ci/build_cuda_cccl_python_tsan.sh
@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+# Thin wrapper around build_cuda_cccl_python.sh that builds the cuda_cccl wheel
+# with ThreadSanitizer instrumentation on the c.parallel (v1) host code, for the
+# free-threaded (3.14t) TSan nightly lane. The shared build script honors
+# CCCL_C_PARALLEL_SANITIZE_THREAD (passes -DCCCL_C_PARALLEL_SANITIZE_THREAD=ON to
+# the wheel build and --exclude libtsan from the auditwheel repair).
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+export CCCL_C_PARALLEL_SANITIZE_THREAD=1
+"$ci_dir/build_cuda_cccl_python.sh" "$@"
diff --git a/cccl_upstream/ci/build_cuda_cccl_python_v2.sh b/cccl_upstream/ci/build_cuda_cccl_python_v2.sh
new file mode 100755
index 00000000..a62f2aa2
--- /dev/null
+++ b/cccl_upstream/ci/build_cuda_cccl_python_v2.sh
@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+# Thin wrapper around build_cuda_cccl_python.sh that builds the cuda_cccl
+# wheel against the HostJIT-based cccl.c.parallel.v2 library instead of the
+# legacy NVRTC v1 library. The shared build script honors CCCL_PYTHON_USE_V2.
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+export CCCL_PYTHON_USE_V2=1
+exec "$ci_dir/build_cuda_cccl_python.sh" "$@"
diff --git a/cccl_upstream/ci/build_cuda_cccl_wheel.sh b/cccl_upstream/ci/build_cuda_cccl_wheel.sh
new file mode 100755
index 00000000..5fe15685
--- /dev/null
+++ b/cccl_upstream/ci/build_cuda_cccl_wheel.sh
@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Target script for `docker run` command in build_cuda_cccl_python.sh
+# The /workspace pathnames are hard-wired here.
+
+# Toolchain version, pinned in one place. Every gcc-toolset package name and the
+# /opt/rh enable path below derive from it — the -fsanitize=thread libtsan must
+# come from the same toolset as the compiler, so these cannot drift apart.
+readonly gcc_toolset_version=13
+
+# Install the GCC toolset (needed for the build) and ccache (shared between
+# cu12 and cu13 builds via /root/.ccache bind-mount from the host).
+/workspace/ci/util/retry.sh 5 30 dnf -y install \
+  "gcc-toolset-${gcc_toolset_version}-gcc" "gcc-toolset-${gcc_toolset_version}-gcc-c++" ccache
+
+# ThreadSanitizer builds link libcccl.c.parallel.so with -fsanitize=thread, which
+# the linker resolves via the toolset's libtsan.so. That runtime lives in a
+# separate package not pulled in by -gcc-c++, so install it for the TSan lane.
+if [[ "${CCCL_C_PARALLEL_SANITIZE_THREAD:-}" =~ ^(1|true|TRUE|on|ON)$ ]]; then
+  /workspace/ci/util/retry.sh 5 30 dnf -y install "gcc-toolset-${gcc_toolset_version}-libtsan-devel"
+fi
+
+# When the caller bind-mounts a ccache dir, wire it through to CMake. This
+# transparently caches every compile, so the second wheel build (cu13 after
+# cu12, or vice versa) reuses the entire LLVM/clang object tree.
+if [[ -n "${CCACHE_DIR:-}" ]]; then
+  export CMAKE_C_COMPILER_LAUNCHER=ccache
+  export CMAKE_CXX_COMPILER_LAUNCHER=ccache
+  export CMAKE_CUDA_COMPILER_LAUNCHER=ccache
+  echo "ccache enabled: CCACHE_DIR=${CCACHE_DIR}"
+  ccache --version 2>&1 | head -1 || true
+  ccache --show-stats 2>&1 | head -5 || true
+fi
+echo -e "#!/usr/bin/env bash\nsource /opt/rh/gcc-toolset-${gcc_toolset_version}/enable" >/etc/profile.d/enable_devtools.sh
+# shellcheck disable=SC1091
+source /etc/profile.d/enable_devtools.sh
+
+# Check what's available
+command -v gcc
+gcc --version
+command -v nvcc
+nvcc --version
+
+# Set up Python environment
+# shellcheck source=ci/pyenv_helper.sh
+source /workspace/ci/pyenv_helper.sh
+# shellcheck disable=SC2154
+setup_python_env "${py_version}"
+command -v python
+python --version
+echo "Done setting up python env"
+
+# Figure out the version to use for the package, we need repo history
+if "$(git rev-parse --is-shallow-repository)"; then
+  git fetch --unshallow
+fi
+export PACKAGE_VERSION_PREFIX="0.1."
+package_version=$(/workspace/ci/generate_version.sh)
+echo "Using package version ${package_version}"
+# Override the version used by setuptools_scm to the custom version
+export SETUPTOOLS_SCM_PRETEND_VERSION_FOR_CUDA_CCCL="${package_version}"
+
+cd /workspace/python/cuda_cccl
+
+# Determine CUDA version from nvcc
+cuda_version=$(nvcc --version | grep -oP 'release \K[0-9]+\.[0-9]+' | cut -d. -f1)
+echo "Detected CUDA version: ${cuda_version}"
+
+# Configure compilers:
+CXX="$(command -v g++)"
+export CXX
+CUDACXX="$(command -v nvcc)"
+export CUDACXX
+CUDAHOSTCXX="$(command -v g++)"
+export CUDAHOSTCXX
+
+# When CCCL_PYTHON_USE_V2 is set (=1/true/on), build the wheel against the
+# HostJIT-based cccl.c.parallel.v2 library instead of the default v1.
+if [[ "${CCCL_PYTHON_USE_V2:-}" =~ ^(1|true|TRUE|on|ON)$ ]]; then
+  export CMAKE_ARGS="${CMAKE_ARGS:-} -DCCCL_PYTHON_USE_V2=ON"
+  echo "Building wheel with CCCL v2 backend: CMAKE_ARGS=${CMAKE_ARGS}"
+
+  # v2's hostjit links against libnvJitLink and libnvfatbin, which aren't in
+  # the base rapidsai/ci-wheel image. Install the matching CTK devel packages
+  # so CMake's FindCUDAToolkit picks them up. nvcc is on PATH; derive the
+  # version (e.g. "13-0") from it.
+  ctk_pkg_ver=$(nvcc --version 2>/dev/null \
+    | grep -oP 'release \K[0-9]+\.[0-9]+' | tr '.' '-')
+  if [[ -n "${ctk_pkg_ver}" ]]; then
+    echo "Installing libnvjitlink-devel-${ctk_pkg_ver} libnvfatbin-devel-${ctk_pkg_ver}..."
+    /workspace/ci/util/retry.sh 5 30 dnf -y install \
+      "libnvjitlink-devel-${ctk_pkg_ver}" \
+      "libnvfatbin-devel-${ctk_pkg_ver}"
+  else
+    echo "WARNING: could not derive CTK version from nvcc; skipping nvJitLink/nvfatbin install"
+  fi
+
+  # FindCUDAToolkit learned about CUDA::nvfatbin only in CMake 3.27. The base
+  # rapidsai/ci-wheel image ships an older CMake; install a newer one into
+  # the active venv so scikit-build-core picks it up over the system cmake.
+  echo "Pinning cmake>=3.27 for FindCUDAToolkit nvfatbin support..."
+  python -m pip install --upgrade 'cmake>=3.27'
+fi
+
+# When CCCL_C_PARALLEL_SANITIZE_THREAD is set (=1/true/on), instrument the
+# c.parallel (v1) host code with ThreadSanitizer for the free-threaded TSan
+# nightly lane. Host-only; the libtsan runtime stays external (the shared
+# build_cuda_cccl_python.sh --excludes it from auditwheel).
+if [[ "${CCCL_C_PARALLEL_SANITIZE_THREAD:-}" =~ ^(1|true|TRUE|on|ON)$ ]]; then
+  export CMAKE_ARGS="${CMAKE_ARGS:-} -DCCCL_C_PARALLEL_SANITIZE_THREAD=ON"
+  echo "Building wheel with ThreadSanitizer-instrumented c.parallel: CMAKE_ARGS=${CMAKE_ARGS}"
+fi
+
+# Build the wheel
+python -m pip wheel --no-deps --verbose --wheel-dir dist .
+
+# Rename wheel to include CUDA version suffix
+for wheel in dist/cuda_cccl-*.whl; do
+    if [[ -f "$wheel" ]]; then
+        base_name=$(basename "$wheel" .whl)
+        new_name="${base_name}.cu${cuda_version}.whl"
+        mv "$wheel" "dist/${new_name}"
+        echo "Renamed wheel to: ${new_name}"
+    fi
+done
+
+# Move wheel to output directory
+mkdir -p /workspace/wheelhouse
+mv dist/cuda_cccl-*.cu*.whl /workspace/wheelhouse/
diff --git a/cccl_upstream/ci/build_cudax.sh b/cccl_upstream/ci/build_cudax.sh
new file mode 100755
index 00000000..8d8794ed
--- /dev/null
+++ b/cccl_upstream/ci/build_cudax.sh
@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+# shellcheck source=ci/build_common.sh
+source "$(dirname "${BASH_SOURCE[0]}")/build_common.sh"
+
+print_environment_details
+
+# If the cudax_ENABLE_CUFILE variable is specified, we don't modify it. Otherwise if we got an nvcc binary, check the
+# nvcc version and if it's less than 12.9, disable the cuFile support. NVHPC Toolkit doesn't come with cuFile, too, so
+# we don't enable cuFile support if the host compiler is nvc++.
+if [[ -z "${cudax_ENABLE_CUFILE:-}" ]]; then
+  cudax_ENABLE_CUFILE="false"
+  if [[ -n "${NVCC_VERSION:-}" ]] && [[ "$(basename "${HOST_COMPILER}")" != "nvc++" ]]; then
+    if util/version_compare.sh "${NVCC_VERSION}" ge 12.9; then
+      cudax_ENABLE_CUFILE="true"
+    fi
+  fi
+fi
+
+PRESET="cudax"
+
+CMAKE_OPTIONS=(
+  "-Dcudax_ENABLE_CUFILE=${cudax_ENABLE_CUFILE}"
+  "-DCMAKE_CXX_STANDARD=${CXX_STANDARD}"
+  "-DCMAKE_CUDA_STANDARD=${CXX_STANDARD}"
+)
+
+configure_and_build_preset "CUDA Experimental" "$PRESET" "${CMAKE_OPTIONS[@]}"
+
+print_time_summary
diff --git a/cccl_upstream/ci/build_libcudacxx.sh b/cccl_upstream/ci/build_libcudacxx.sh
new file mode 100755
index 00000000..29a8708a
--- /dev/null
+++ b/cccl_upstream/ci/build_libcudacxx.sh
@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+
+# shellcheck source=ci/build_common.sh
+source "${ci_dir}/build_common.sh"
+
+print_environment_details
+
+PRESET="libcudacxx"
+CMAKE_OPTIONS=("-DCMAKE_CXX_STANDARD=${CXX_STANDARD}" "-DCMAKE_CUDA_STANDARD=${CXX_STANDARD}")
+
+upload_test_artifacts=false
+if [[ -n "${GITHUB_ACTIONS:-}" ]] && "${ci_dir}/util/workflow/has_consumers.sh"; then
+  upload_test_artifacts=true
+  export LIT_OPTS="${LIT_OPTS:+${LIT_OPTS} }-Dtest_executable_mode=build"
+fi
+
+configure_and_build_preset libcudacxx "$PRESET" "${CMAKE_OPTIONS[@]}"
+
+if $upload_test_artifacts; then
+  run_command "📦  Packaging test artifacts" "${ci_dir}/upload_libcudacxx_test_artifacts.sh"
+fi
+
+print_time_summary
diff --git a/cccl_upstream/ci/build_stdpar.sh b/cccl_upstream/ci/build_stdpar.sh
new file mode 100755
index 00000000..7a58326b
--- /dev/null
+++ b/cccl_upstream/ci/build_stdpar.sh
@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+# Ensure the script is being executed in the root cccl directory:
+cd "$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )/..";
+
+# Get the current CCCL info:
+readonly cccl_repo="${PWD}"
+readonly workdir="${cccl_repo}/test/stdpar"
+
+CXX_STANDARD=17
+
+args=("$@")
+while [[ "${#args[@]}" -ne 0 ]]; do
+    case "${args[0]}" in
+    -std)  CXX_STANDARD="${args[1]}";  args=("${args[@]:2}");;
+    *) echo "Unrecognized option: ${args[0]}"; exit 1 ;;
+    esac
+done
+
+mkdir -p "${workdir}"
+cd "${workdir}"
+
+# Configure and build
+rm -rf build
+
+cmake -B build -S . -G Ninja \
+  -DCMAKE_CXX_STANDARD="${CXX_STANDARD}" \
+  `# Explicitly compile for hopper since the CI machine does not have a gpu:` \
+  -DCMAKE_CXX_FLAGS="-gpu=cc90"
+
+# Disabled because `cmake --build -j ""` is invalid, but so is
+# `cmake --build -j8`. CMake expects a space between `-j` and
+# the numeric argument, or no argument at all.
+# shellcheck disable=SC2086
+cmake --build build -j ${PARALLEL_LEVEL:-}
diff --git a/cccl_upstream/ci/build_thrust.sh b/cccl_upstream/ci/build_thrust.sh
new file mode 100755
index 00000000..b3250614
--- /dev/null
+++ b/cccl_upstream/ci/build_thrust.sh
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+# shellcheck source=ci/build_common.sh
+source "$(dirname "${BASH_SOURCE[0]}")/build_common.sh"
+
+print_environment_details
+
+PRESET="thrust"
+
+CMAKE_OPTIONS=("-DCMAKE_CXX_STANDARD=$CXX_STANDARD" "-DCMAKE_CUDA_STANDARD=$CXX_STANDARD")
+
+configure_and_build_preset "Thrust" "$PRESET" "${CMAKE_OPTIONS[@]}"
+
+# Create test artifacts:
+if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+    run_command "📦  Packaging test artifacts" /home/coder/cccl/ci/upload_thrust_test_artifacts.sh
+fi
+
+print_time_summary
diff --git a/cccl_upstream/ci/build_tidy.sh b/cccl_upstream/ci/build_tidy.sh
new file mode 100755
index 00000000..a2e1a5cb
--- /dev/null
+++ b/cccl_upstream/ci/build_tidy.sh
@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+source "$(dirname "${BASH_SOURCE[0]}")/build_common.sh"
+
+print_environment_details
+
+BUILD_NAME="clang-tidy"
+PRESET="all-tidy"
+CMAKE_OPTIONS=("-DCMAKE_CXX_STANDARD=${CXX_STANDARD}" "-DCMAKE_CUDA_STANDARD=${CXX_STANDARD}")
+# Clang does not understand -G, passed by all-dev-debug which all-tidy derives from
+CMAKE_OPTIONS+=("-DCMAKE_CUDA_FLAGS=")
+# TODO(jfaibussowit)
+#
+# STF seems to trip clang-cuda up pretty heavily. It's unclear whether this is because STF
+# hasn't been compiled against clang-cuda before or whether it's an issue with clang-cuda
+# itself.
+CMAKE_OPTIONS+=("-Dcudax_ENABLE_CUDASTF=OFF")
+CMAKE_OPTIONS+=("-Dcudax_ENABLE_PLACES=OFF")
+
+# todo(dabayer): Re-enable OpenMP thrust builds for clang-tidy.
+CMAKE_OPTIONS+=("-DTHRUST_MULTICONFIG_ENABLE_SYSTEM_OMP=OFF")
+
+# Cannot use configure_and_build_preset because that does not allow us to pass additional
+# arguments to the build command.
+configure_preset "${BUILD_NAME}" "${PRESET}" "${CMAKE_OPTIONS[@]}"
+# Keep going after errors, we want CI to unearth all clang-tidy errors in one go
+BUILD_OPTIONS=(-- -k 0)
+build_preset "${BUILD_NAME}" "${PRESET}" "${BUILD_OPTIONS[@]}"
+
+print_time_summary
diff --git a/cccl_upstream/ci/compile_time/README.md b/cccl_upstream/ci/compile_time/README.md
new file mode 100644
index 00000000..96c417bd
--- /dev/null
+++ b/cccl_upstream/ci/compile_time/README.md
@@ -0,0 +1,77 @@
+# Compile-time benchmark CI contracts
+
+The compile-time benchmark CI flow is configured from `ci/matrix.yaml` under
+`compile_time.pull_request`.
+
+## Matrix schema
+
+Each config is a GitHub Actions matrix entry:
+
+```yaml
+compile_time:
+  pull_request:
+    - id: public-headers-gcc13
+      name: Public headers compile-time bench
+      gpu: rtx2080
+      launch_args: "--cuda 13.3 --host gcc13"
+      baseline_ref: origin/main
+      preset: all-dev
+      targets:
+        - cub.headers.base
+      args: "-arch native"
+      slices:
+        - id: total-compilation
+          title: TU total compilation
+          filter: total-compilation
+          timing: inclusive
+          sort: total
+          top: 15
+          threshold: 0.001
+```
+
+Required config fields are `id`, `name`, `gpu`, `launch_args`,
+`baseline_ref`, `preset`, `targets`, and `slices`. `args`, `comment`, and
+`artifact_retention_days` are optional.
+
+Required slice fields are `id`, `title`, `filter`, `timing`, `sort`, `top`, and
+`threshold`. Slice `children` may be used to group nested report sections in the
+PR comment. Empty slice sections are omitted recursively by the renderer unless
+the summary manifest carries warnings for that slice.
+
+`ci/compile_time/parse_matrix.py ci/matrix.yaml --workflow pull_request` emits
+the GitHub Actions matrix JSON. Missing or empty `compile_time.pull_request`
+emits `{"include":[]}`.
+
+In baseline comparisons, `threshold` is measured against the total selected
+inclusive/exclusive impact across all matched traces. The per-side reports still
+use `sort` for their own top-N ordering; comparison worse/better tables always
+rank by total impact so a change repeated across many traces is not hidden by a
+larger single-trace movement.
+
+## Report contract
+
+`summarize_events.py --slices <json>` writes per-slice CSVs under
+`event_reports/<slice-id>/` and writes a normalized `event_reports/summary.json`
+manifest. The manifest is the renderer contract; CSVs are human artifacts.
+Configured slices that match no events, have no matching trace files, or have no
+comparable event keys record warnings in the manifest so reporting failures are
+not presented as ordinary no-regression results.
+
+In comparison mode, the wrapper preserves:
+
+- current raw traces: `compile_time/raw_traces`
+- baseline raw traces: `compile_time/baseline_raw_traces`
+- Perfetto copies: `compile_time/perfetto_traces/current` and
+  `compile_time/perfetto_traces/baseline`
+
+## PR comments
+
+`render_pr_comment.py` reads `summary.json`, config metadata, and an artifacts
+URL, then writes the sticky PR comment body. Regressions and improvements are
+rendered in separate `<details>` blocks and are never mixed in one table.
+Warnings are rendered separately and keep their slice visible even when there
+are no regression/improvement rows.
+
+The reusable workflow uses the sticky-comment header
+`compile-time-bench-<config-id>` with `hide_and_recreate: true`, so previous
+comments for the same config are archived as outdated.
diff --git a/cccl_upstream/ci/compile_time/analytics.ipynb b/cccl_upstream/ci/compile_time/analytics.ipynb
new file mode 100644
index 00000000..369411a8
--- /dev/null
+++ b/cccl_upstream/ci/compile_time/analytics.ipynb
@@ -0,0 +1,333 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "id": "903704f7",
+      "metadata": {},
+      "source": [
+        "# Compile-Time Analytics\n",
+        "\n",
+        "This notebook helps you:\n",
+        "\n",
+        "1. Run `ci/build_compile_time_bench.sh` from the repo root.\n",
+        "2. Load and inspect an all-header processing CSV.\n",
+        "3. Explore high-impact headers by TU coverage and average processing time.\n",
+        "4. Build combined ranking scores to identify optimization targets.\n",
+        "\n",
+        "Expected CSV columns include:\n",
+        "- `header_path`\n",
+        "- `include_tu_count`\n",
+        "- `avg_process_time_s`\n",
+        "- `total_process_time_s`\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "ad58db28",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%pip install pandas matplotlib plotly nbformat"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "16ba6808",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "import shlex\n",
+        "import sys\n",
+        "from pathlib import Path\n",
+        "\n",
+        "import pandas as pd\n",
+        "\n",
+        "pd.set_option(\"display.max_colwidth\", 160)\n",
+        "pd.set_option(\"display.width\", 200)\n",
+        "pd.set_option(\"display.max_columns\", 20)\n",
+        "\n",
+        "REPO_ROOT = Path.cwd()\n",
+        "while REPO_ROOT != REPO_ROOT.parent and not (REPO_ROOT / \".git\").exists():\n",
+        "    REPO_ROOT = REPO_ROOT.parent\n",
+        "\n",
+        "if not (REPO_ROOT / \".git\").exists():\n",
+        "    raise RuntimeError(\n",
+        "        \"Could not locate repo root (.git). Start notebook from inside the CCCL repo.\"\n",
+        "    )\n",
+        "\n",
+        "print(f\"Repo root: {REPO_ROOT}\")\n",
+        "print(f\"Python executable: {sys.executable}\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "373652c8",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# --- Run build_compile_time_bench.sh (file-processing mode) ---\n",
+        "import subprocess\n",
+        "from pathlib import Path\n",
+        "\n",
+        "if \"REPO_ROOT\" not in globals():\n",
+        "    REPO_ROOT = Path.cwd()\n",
+        "    while REPO_ROOT != REPO_ROOT.parent and not (REPO_ROOT / \".git\").exists():\n",
+        "        REPO_ROOT = REPO_ROOT.parent\n",
+        "\n",
+        "output_csv = Path(os.environ.get(\"COMPILE_TIME_CSV\", \"/tmp/compile_time.csv\"))\n",
+        "cmd = [\n",
+        "    \"bash\",\n",
+        "    str(REPO_ROOT / \"ci\" / \"build_compile_time_bench.sh\"),\n",
+        "    *shlex.split(os.environ.get(\"COMPILE_TIME_BUILD_ARGS\", \"\")),\n",
+        "    \"--\",\n",
+        "    \"-f\",\n",
+        "    \"file-processing\",\n",
+        "    \"-e\",\n",
+        "    \"-n\",\n",
+        "    os.environ.get(\"COMPILE_TIME_TOP_N\", \"5000\"),\n",
+        "    \"--sort\",\n",
+        "    \"total\",\n",
+        "    \"--output-csv\",\n",
+        "    str(output_csv),\n",
+        "]\n",
+        "\n",
+        "print(\"Command:\")\n",
+        "print(\"  \" + \" \".join(shlex.quote(x) for x in cmd))\n",
+        "\n",
+        "subprocess.run(cmd, cwd=REPO_ROOT, check=True)\n",
+        "print(f\"\\nWrote: {output_csv}\")"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "12ad4c5c",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# --- Load CSV ---\n",
+        "csv_path = Path(os.environ.get(\"COMPILE_TIME_CSV\", \"/tmp/compile_time.csv\"))\n",
+        "if not csv_path.exists():\n",
+        "    raise FileNotFoundError(f\"Missing CSV: {csv_path}. Run the script first.\")\n",
+        "\n",
+        "df = pd.read_csv(csv_path)\n",
+        "\n",
+        "event_summary_cols = {\n",
+        "    \"event_name\",\n",
+        "    \"event_key\",\n",
+        "    \"root_tu_count\",\n",
+        "    \"selected_avg_per_root_tu_s\",\n",
+        "    \"selected_total_s\",\n",
+        "}\n",
+        "required_cols = [\n",
+        "    \"header_path\",\n",
+        "    \"include_tu_count\",\n",
+        "    \"avg_process_time_s\",\n",
+        "    \"total_process_time_s\",\n",
+        "]\n",
+        "\n",
+        "if event_summary_cols.issubset(df.columns):\n",
+        "    df[\"header_path\"] = df[\"event_key\"]\n",
+        "    df[\"include_tu_count\"] = df[\"root_tu_count\"]\n",
+        "    if (\n",
+        "        \"avg_inclusive_per_root_tu_s\" in df.columns\n",
+        "        and \"total_inclusive_s\" in df.columns\n",
+        "    ):\n",
+        "        df[\"avg_process_time_s\"] = df[\"avg_inclusive_per_root_tu_s\"]\n",
+        "        df[\"total_process_time_s\"] = df[\"total_inclusive_s\"]\n",
+        "    else:\n",
+        "        df[\"avg_process_time_s\"] = df[\"selected_avg_per_root_tu_s\"]\n",
+        "        df[\"total_process_time_s\"] = df[\"selected_total_s\"]\n",
+        "\n",
+        "missing = [c for c in required_cols if c not in df.columns]\n",
+        "if missing:\n",
+        "    raise ValueError(\n",
+        "        f\"CSV is not all-header processing output. Missing columns: {missing}. \"\n",
+        "        \"Run the script cell above to regenerate /tmp/compile_time.csv.\"\n",
+        "    )\n",
+        "\n",
+        "for col in [\"include_tu_count\", \"avg_process_time_s\", \"total_process_time_s\"]:\n",
+        "    df[col] = pd.to_numeric(df[col], errors=\"coerce\").fillna(0)\n",
+        "\n",
+        "print(f\"Rows: {len(df):,}\")\n",
+        "print(f\"Columns: {list(df.columns)}\")\n",
+        "df.head(5)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "1a9b36a8",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# --- Quick top-N views ---\n",
+        "TOP_N = 5\n",
+        "\n",
+        "print(\"Top by avg_process_time_s\")\n",
+        "display(\n",
+        "    df.nlargest(TOP_N, \"avg_process_time_s\")[\n",
+        "        [\n",
+        "            \"header_path\",\n",
+        "            \"avg_process_time_s\",\n",
+        "            \"include_tu_count\",\n",
+        "            \"total_process_time_s\",\n",
+        "        ]\n",
+        "    ]\n",
+        ")\n",
+        "\n",
+        "print(\"\\nTop by include_tu_count\")\n",
+        "display(\n",
+        "    df.nlargest(TOP_N, \"include_tu_count\")[\n",
+        "        [\n",
+        "            \"header_path\",\n",
+        "            \"include_tu_count\",\n",
+        "            \"avg_process_time_s\",\n",
+        "            \"total_process_time_s\",\n",
+        "        ]\n",
+        "    ]\n",
+        ")\n",
+        "\n",
+        "print(\"\\nTop by impact score (include_tu_count * avg_process_time_s)\")\n",
+        "df_score = df.copy()\n",
+        "df_score[\"impact_score\"] = df_score[\"include_tu_count\"] * df_score[\"avg_process_time_s\"]\n",
+        "display(\n",
+        "    df_score.nlargest(TOP_N, \"impact_score\")[\n",
+        "        [\n",
+        "            \"header_path\",\n",
+        "            \"impact_score\",\n",
+        "            \"include_tu_count\",\n",
+        "            \"avg_process_time_s\",\n",
+        "            \"total_process_time_s\",\n",
+        "        ]\n",
+        "    ]\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "id": "174eabd4",
+      "metadata": {},
+      "source": [
+        "### How to read this plot\n",
+        "\n",
+        "- Each point is one header seen in all-mode profiling.\n",
+        "- **X axis (`include_tu_count`)**: how many generated public-header TUs include this header at least once.\n",
+        "- **Y axis (`avg_process_time_s`)**: average time spent processing that header per including TU.\n",
+        "- Headers near the **upper-right** are usually the best optimization candidates because they are both widespread and expensive per include."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "9ddc0725",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# --- Interactive scatter (hover shows header name) ---\n",
+        "\n",
+        "import plotly.express as px\n",
+        "\n",
+        "plot_df = df.copy()\n",
+        "\n",
+        "total_headers = len(plot_df)\n",
+        "total_public_headers = int(plot_df[\"include_tu_count\"].max())\n",
+        "\n",
+        "fig = px.scatter(\n",
+        "    plot_df,\n",
+        "    x=\"include_tu_count\",\n",
+        "    y=\"avg_process_time_s\",\n",
+        "    hover_name=\"header_path\",\n",
+        "    hover_data={\n",
+        "        \"include_tu_count\": True,\n",
+        "        \"avg_process_time_s\": \":.6f\",\n",
+        "        \"total_process_time_s\": \":.3f\",\n",
+        "        \"header_path\": False,\n",
+        "    },\n",
+        "    opacity=0.6,\n",
+        "    title=(\n",
+        "        f\"Header include count vs avg processing time ({total_headers:,} total headers; \"\n",
+        "        f\"coverage measured across {total_public_headers} public headers)\"\n",
+        "    ),\n",
+        ")\n",
+        "\n",
+        "fig.update_layout(\n",
+        "    xaxis_title=(\n",
+        "        \"TU coverage: number of public headers that include this header \"\n",
+        "        f\"(out of {total_public_headers})\"\n",
+        "    ),\n",
+        "    yaxis_title=\"Average processing time per including TU (seconds)\",\n",
+        "    height=900,\n",
+        ")\n",
+        "fig.show()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "id": "f429b934",
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# --- Optional: run ctadvisor and parse expensive headers ---\n",
+        "run_ctadvisor = os.environ.get(\"COMPILE_TIME_RUN_CTADVISOR\") == \"1\"\n",
+        "CTADVISOR_ENTRIES = 10\n",
+        "CTADVISOR_THREADS = os.cpu_count() or 8\n",
+        "\n",
+        "trace_root = (\n",
+        "    REPO_ROOT\n",
+        "    / \"build\"\n",
+        "    / os.environ.get(\"CCCL_BUILD_INFIX\", \"cuda13.1-gcc14\")\n",
+        "    / os.environ.get(\"CCCL_COMPILE_TIME_PRESET\", \"all-dev\")\n",
+        "    / \"compile_time\"\n",
+        "    / \"raw_traces\"\n",
+        ")\n",
+        "ctadvisor_cmd = [\n",
+        "    \"ctadvisor\",\n",
+        "    \"--trace-file-path\",\n",
+        "    str(trace_root),\n",
+        "    \"--header-advisor-entries\",\n",
+        "    str(CTADVISOR_ENTRIES),\n",
+        "    \"--thread-number\",\n",
+        "    str(CTADVISOR_THREADS),\n",
+        "]\n",
+        "\n",
+        "if run_ctadvisor:\n",
+        "    print(\"Command:\")\n",
+        "    print(\"  \" + \" \".join(shlex.quote(x) for x in ctadvisor_cmd))\n",
+        "\n",
+        "    result = subprocess.run(\n",
+        "        ctadvisor_cmd, cwd=REPO_ROOT, check=True, capture_output=True, text=True\n",
+        "    )\n",
+        "    print(result.stdout)\n",
+        "else:\n",
+        "    print(\"Skipping ctadvisor. Set COMPILE_TIME_RUN_CTADVISOR=1 to run it.\")"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "cccl",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.12.3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 5
+}
diff --git a/cccl_upstream/ci/compile_time/parse_matrix.py b/cccl_upstream/ci/compile_time/parse_matrix.py
new file mode 100755
index 00000000..f9e20837
--- /dev/null
+++ b/cccl_upstream/ci/compile_time/parse_matrix.py
@@ -0,0 +1,272 @@
+#!/usr/bin/env python3
+
+import argparse
+import json
+import re
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+try:
+    import yaml
+except ModuleNotFoundError:
+    yaml = None
+    YAML_ERROR_TYPES: tuple[type[BaseException], ...] = ()
+else:
+    YAML_ERROR_TYPES = (yaml.YAMLError,)
+
+ID_RE = re.compile(r"^[a-z0-9][a-z0-9_.-]*$")
+TIMINGS = {"inclusive", "exclusive"}
+SORTS = {"total", "avg", "avg-root-tu", "max"}
+
+
+def die(message: str) -> None:
+    print(f"error: {message}", file=sys.stderr)
+    raise SystemExit(2)
+
+
+def require_mapping(value: Any, where: str) -> dict[str, Any]:
+    if not isinstance(value, dict):
+        die(f"{where} must be a mapping")
+    return value
+
+
+def require_field(mapping: dict[str, Any], field: str, where: str) -> Any:
+    if field not in mapping:
+        die(f"{where} is missing required field '{field}'")
+    return mapping[field]
+
+
+def require_string(value: Any, where: str, *, nonempty: bool = True) -> str:
+    if not isinstance(value, str):
+        die(f"{where} must be a string")
+    if nonempty and not value:
+        die(f"{where} must be non-empty")
+    return value
+
+
+def require_id(value: Any, where: str) -> str:
+    text = require_string(value, where)
+    if not ID_RE.fullmatch(text):
+        die(f"{where} must match {ID_RE.pattern}")
+    return text
+
+
+def require_string_list(value: Any, where: str) -> list[str]:
+    if not isinstance(value, list) or not value:
+        die(f"{where} must be a non-empty list")
+    strings: list[str] = []
+    for index, item in enumerate(value):
+        strings.append(require_string(item, f"{where}[{index}]"))
+    return strings
+
+
+def require_bool(value: Any, where: str) -> bool:
+    if not isinstance(value, bool):
+        die(f"{where} must be a boolean")
+    return value
+
+
+def require_positive_int(value: Any, where: str) -> int:
+    if isinstance(value, bool) or not isinstance(value, int) or value <= 0:
+        die(f"{where} must be a positive integer")
+    return value
+
+
+def validate_slice(
+    slice_data: Any,
+    *,
+    where: str,
+    seen_ids: set[str],
+) -> dict[str, Any]:
+    data = require_mapping(slice_data, where)
+    slice_id = require_id(require_field(data, "id", where), f"{where}.id")
+    if slice_id in seen_ids:
+        die(f"duplicate slice id '{slice_id}' in {where}")
+    seen_ids.add(slice_id)
+
+    title = require_string(require_field(data, "title", where), f"{where}.title")
+    filter_name = require_string(
+        require_field(data, "filter", where), f"{where}.filter"
+    )
+    timing = require_string(require_field(data, "timing", where), f"{where}.timing")
+    if timing not in TIMINGS:
+        die(f"{where}.timing must be one of {sorted(TIMINGS)}")
+    sort = require_string(require_field(data, "sort", where), f"{where}.sort")
+    if sort not in SORTS:
+        die(f"{where}.sort must be one of {sorted(SORTS)}")
+
+    top = require_field(data, "top", where)
+    if isinstance(top, bool) or not isinstance(top, int) or top <= 0:
+        die(f"{where}.top must be a positive integer")
+    threshold = require_field(data, "threshold", where)
+    if (
+        isinstance(threshold, bool)
+        or not isinstance(threshold, (int, float))
+        or threshold < 0
+    ):
+        die(f"{where}.threshold must be a non-negative number")
+
+    result: dict[str, Any] = {
+        "id": slice_id,
+        "title": title,
+        "filter": filter_name,
+        "timing": timing,
+        "sort": sort,
+        "top": top,
+        "threshold": threshold,
+    }
+    for optional in ("scope_filter", "exclusive_scope"):
+        if optional in data:
+            result[optional] = require_string(
+                data[optional], f"{where}.{optional}", nonempty=False
+            )
+
+    children = data.get("children", [])
+    if not isinstance(children, list):
+        die(f"{where}.children must be a list")
+    if children:
+        result["children"] = [
+            validate_slice(
+                child,
+                where=f"{where}.children[{index}]",
+                seen_ids=seen_ids,
+            )
+            for index, child in enumerate(children)
+        ]
+    return result
+
+
+def validate_config(
+    config_data: Any, *, where: str, seen_ids: set[str]
+) -> dict[str, Any]:
+    data = require_mapping(config_data, where)
+    config_id = require_id(require_field(data, "id", where), f"{where}.id")
+    if config_id in seen_ids:
+        die(f"duplicate compile_time config id '{config_id}'")
+    seen_ids.add(config_id)
+
+    targets = require_string_list(
+        require_field(data, "targets", where), f"{where}.targets"
+    )
+    slices = require_field(data, "slices", where)
+    if not isinstance(slices, list) or not slices:
+        die(f"{where}.slices must be a non-empty list")
+
+    slice_ids: set[str] = set()
+    normalized_slices = [
+        validate_slice(
+            slice_data,
+            where=f"{where}.slices[{index}]",
+            seen_ids=slice_ids,
+        )
+        for index, slice_data in enumerate(slices)
+    ]
+
+    return {
+        "id": config_id,
+        "name": require_string(require_field(data, "name", where), f"{where}.name"),
+        "gpu": require_string(require_field(data, "gpu", where), f"{where}.gpu"),
+        "launch_args": require_string(
+            require_field(data, "launch_args", where), f"{where}.launch_args"
+        ),
+        "baseline_ref": require_string(
+            require_field(data, "baseline_ref", where), f"{where}.baseline_ref"
+        ),
+        "preset": require_string(
+            require_field(data, "preset", where), f"{where}.preset"
+        ),
+        "targets": targets,
+        "args": require_string(data.get("args", ""), f"{where}.args", nonempty=False),
+        "comment": require_bool(data.get("comment", True), f"{where}.comment"),
+        "artifact_retention_days": require_positive_int(
+            data.get("artifact_retention_days", 14),
+            f"{where}.artifact_retention_days",
+        ),
+        "slices": normalized_slices,
+    }
+
+
+def matrix_entry(config: dict[str, Any]) -> dict[str, Any]:
+    config_id = config["id"]
+    return {
+        "id": config_id,
+        "name": config["name"],
+        "gpu": config["gpu"],
+        "launch_args": config["launch_args"],
+        "baseline_ref": config["baseline_ref"],
+        "preset": config["preset"],
+        "targets_json": json.dumps(config["targets"], separators=(",", ":")),
+        "args": config["args"],
+        "slices_json": json.dumps({"slices": config["slices"]}, separators=(",", ":")),
+        "comment": str(config["comment"]).lower(),
+        "artifact_retention_days": config["artifact_retention_days"],
+        "comment_header": f"compile-time-bench-{config_id}",
+    }
+
+
+def parse_matrix(path: Path, workflow: str) -> dict[str, Any]:
+    try:
+        if yaml is not None:
+            with path.open(encoding="utf-8") as f:
+                matrix = yaml.safe_load(f) or {}
+        else:
+            completed = subprocess.run(
+                ["yq", "-o=json", ".", path.as_posix()],
+                check=True,
+                text=True,
+                stdout=subprocess.PIPE,
+                stderr=subprocess.PIPE,
+            )
+            matrix = json.loads(completed.stdout or "{}")
+    except OSError as e:
+        die(f"failed to read {path}: {e}")
+    except subprocess.CalledProcessError as e:
+        die(f"failed to parse {path} with yq: {e.stderr.strip()}")
+    except json.JSONDecodeError as e:
+        die(f"failed to decode {path} as JSON: {e}")
+    except YAML_ERROR_TYPES as e:
+        die(f"failed to parse {path}: {e}")
+
+    compile_time = matrix.get("compile_time")
+    if compile_time is None:
+        return {"include": []}
+    compile_time = require_mapping(compile_time, "compile_time")
+    configs = compile_time.get(workflow, [])
+    if configs is None:
+        configs = []
+    if not isinstance(configs, list):
+        die(f"compile_time.{workflow} must be a list")
+    if not configs:
+        return {"include": []}
+
+    seen_ids: set[str] = set()
+    return {
+        "include": [
+            matrix_entry(
+                validate_config(
+                    config,
+                    where=f"compile_time.{workflow}[{index}]",
+                    seen_ids=seen_ids,
+                )
+            )
+            for index, config in enumerate(configs)
+        ]
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Parse ci/matrix.yaml compile_time entries for GitHub Actions."
+    )
+    parser.add_argument("matrix_yaml", type=Path)
+    parser.add_argument("--workflow", default="pull_request")
+    args = parser.parse_args()
+
+    json.dump(parse_matrix(args.matrix_yaml, args.workflow), sys.stdout)
+    print()
+
+
+if __name__ == "__main__":
+    main()
diff --git a/cccl_upstream/ci/compile_time/prepare_traces.py b/cccl_upstream/ci/compile_time/prepare_traces.py
new file mode 100755
index 00000000..672ffe5c
--- /dev/null
+++ b/cccl_upstream/ci/compile_time/prepare_traces.py
@@ -0,0 +1,167 @@
+#!/usr/bin/env python3
+
+import argparse
+import json
+from pathlib import Path
+
+DETAIL_EVENT_NAMES = {
+    "Code Generation Function",
+    "CodeGen Function",
+    "ExecuteCompiler",
+    "Frontend",
+    "Instantiating Template Class",
+    "Instantiating Template Function",
+    "InstantiateClass",
+    "InstantiateFunction",
+    "OptFunction",
+    "ParseClass",
+    "PerformPendingInstantiations",
+    "Processing Header File",
+    "RunPass",
+    "Scanning Function Body",
+    "Source",
+}
+
+DETAIL_PREFIXES_TO_STRIP = (
+    "libcudacxx/include/",
+    "cudax/include/",
+    "c/parallel/include/",
+)
+
+DETAIL_PREFIXES_TO_COLLAPSE = (
+    ("cub/cub/", "cub/"),
+    ("thrust/thrust/", "thrust/"),
+)
+
+
+def normalize_detail(detail: str, repo_root: Path) -> str:
+    detail_path = Path(detail)
+    if detail_path.is_absolute():
+        try:
+            rel = detail_path.resolve(strict=False).relative_to(repo_root)
+            detail = rel.as_posix()
+        except ValueError:
+            pass
+
+    for prefix in DETAIL_PREFIXES_TO_STRIP:
+        if detail.startswith(prefix):
+            detail = detail[len(prefix) :]
+            break
+
+    for prefix, replacement in DETAIL_PREFIXES_TO_COLLAPSE:
+        if detail.startswith(prefix):
+            detail = replacement + detail[len(prefix) :]
+            break
+
+    return detail
+
+
+def display_detail(detail: str, repo_root: Path, max_detail_len: int | None) -> str:
+    detail = normalize_detail(detail, repo_root)
+    if (
+        max_detail_len is not None
+        and max_detail_len > 0
+        and len(detail) > max_detail_len
+    ):
+        return detail[: max_detail_len - 1] + "..."
+    return detail
+
+
+def rewrite_event_name(
+    event: dict, repo_root: Path, max_detail_len: int | None
+) -> bool:
+    name = event.get("name")
+    if name not in DETAIL_EVENT_NAMES:
+        return False
+
+    args = event.get("args")
+    if not isinstance(args, dict):
+        return False
+
+    detail = args.get("detail")
+    if not detail:
+        return False
+
+    args.setdefault("original_name", name)
+    event["name"] = f"{name}: {display_detail(str(detail), repo_root, max_detail_len)}"
+    return True
+
+
+def prepare_trace(
+    input_path: Path, output_path: Path, repo_root: Path, max_detail_len: int | None
+) -> int:
+    with input_path.open(encoding="utf-8") as f:
+        trace = json.load(f)
+
+    rewritten = 0
+    for event in trace.get("traceEvents", []):
+        if rewrite_event_name(event, repo_root, max_detail_len):
+            rewritten += 1
+
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    with output_path.open("w", encoding="utf-8") as f:
+        json.dump(trace, f, separators=(",", ":"))
+
+    return rewritten
+
+
+def iter_input_traces(input_path: Path) -> list[Path]:
+    if input_path.is_file():
+        return [input_path]
+    return sorted(input_path.rglob("*.json"))
+
+
+def output_path_for(input_trace: Path, input_root: Path, output_path: Path) -> Path:
+    if input_root.is_file():
+        if output_path.is_dir() or not output_path.suffix:
+            return output_path / f"{input_trace.stem}.perfetto.json"
+        return output_path
+
+    rel = input_trace.relative_to(input_root)
+    return output_path / rel.parent / f"{rel.stem}.perfetto.json"
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Prepare NVCC device-time-trace JSON files for Perfetto by promoting args.detail into event names."
+    )
+    parser.add_argument(
+        "--input", required=True, type=Path, help="Input trace JSON file or directory"
+    )
+    parser.add_argument(
+        "--output", required=True, type=Path, help="Output trace JSON file or directory"
+    )
+    parser.add_argument(
+        "--repo-root", default=Path(__file__).resolve().parents[2], type=Path
+    )
+    parser.add_argument(
+        "--max-detail-len",
+        default=0,
+        type=int,
+        help="Truncate promoted detail text to this many characters; 0 keeps full details",
+    )
+    args = parser.parse_args()
+
+    input_path = args.input.resolve(strict=False)
+    output_path = args.output.resolve(strict=False)
+    repo_root = args.repo_root.resolve(strict=False)
+    max_detail_len = args.max_detail_len if args.max_detail_len > 0 else None
+
+    traces = iter_input_traces(input_path)
+    if not traces:
+        raise SystemExit(f"no JSON traces found under {args.input}")
+
+    total_rewritten = 0
+    for trace_path in traces:
+        total_rewritten += prepare_trace(
+            trace_path,
+            output_path_for(trace_path, input_path, output_path),
+            repo_root,
+            max_detail_len,
+        )
+
+    print(f"prepared {len(traces)} trace(s); renamed {total_rewritten} event(s)")
+
+
+if __name__ == "__main__":
+    main()
diff --git a/cccl_upstream/ci/compile_time/render_pr_comment.py b/cccl_upstream/ci/compile_time/render_pr_comment.py
new file mode 100755
index 00000000..6d86fd75
--- /dev/null
+++ b/cccl_upstream/ci/compile_time/render_pr_comment.py
@@ -0,0 +1,246 @@
+#!/usr/bin/env python3
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any
+
+
+def load_json(path: Path) -> dict[str, Any]:
+    with path.open(encoding="utf-8") as f:
+        payload = json.load(f)
+    if not isinstance(payload, dict):
+        raise SystemExit(f"{path} must contain a JSON object")
+    return payload
+
+
+def md_escape(value: object) -> str:
+    text = str(value)
+    return (
+        text.replace("&", "&amp;")
+        .replace("<", "&lt;")
+        .replace(">", "&gt;")
+        .replace("|", "\\|")
+        .replace("\n", " ")
+    )
+
+
+def md_code_span(value: object) -> str:
+    text = str(value).replace("\n", " ")
+    max_backtick_run = 0
+    current_backtick_run = 0
+    for char in text:
+        if char == "`":
+            current_backtick_run += 1
+            max_backtick_run = max(max_backtick_run, current_backtick_run)
+        else:
+            current_backtick_run = 0
+    delimiter = "`" * (max_backtick_run + 1)
+    if text.startswith("`") or text.endswith("`"):
+        text = f" {text} "
+    return f"{delimiter}{text}{delimiter}"
+
+
+def render_event_name(row: dict[str, Any]) -> str:
+    event_name = row.get("event_name", "")
+    event_key = row.get("event_key", "")
+    if event_key:
+        return f"{md_escape(event_name)}: {md_code_span(event_key)}"
+    return md_escape(event_name)
+
+
+def render_rows(rows: list[dict[str, Any]], *, direction: str) -> str:
+    delta_heading = (
+        "Regression impact" if direction == "worse" else "Improvement impact"
+    )
+    lines = [
+        f"| Rank | {delta_heading} | Selected Δ | Baseline | Current | Event | Matched traces |",
+        "| ---: | ---: | ---: | ---: | ---: | --- | ---: |",
+    ]
+    for row in rows:
+        lines.append(
+            "| {rank} | `{impact}` | `{selected_delta}` | `{baseline}` | `{current}` | {event} | {traces} |".format(
+                rank=md_escape(row.get("rank", "")),
+                impact=md_escape(row.get("impact_magnitude_s", "")),
+                selected_delta=md_escape(row.get("selected_delta_s", "")),
+                baseline=md_escape(row.get("baseline_selected_s", "")),
+                current=md_escape(row.get("current_selected_s", "")),
+                event=render_event_name(row),
+                traces=md_escape(row.get("matched_trace_count", "")),
+            )
+        )
+    return "\n".join(lines)
+
+
+def render_direction_details(
+    slice_title: str,
+    direction: str,
+    rows: list[dict[str, Any]],
+) -> str:
+    if not rows:
+        return ""
+    label = "Regressions" if direction == "worse" else "Improvements"
+    icon = "🔴" if direction == "worse" else "🟢"
+    return "\n".join(
+        [
+            "<details>",
+            f"<summary><strong>{icon} {md_escape(slice_title)} — {label}</strong></summary>",
+            "",
+            render_rows(rows, direction=direction),
+            "",
+            "</details>",
+        ]
+    )
+
+
+def render_warning_details(slice_title: str, warnings: list[Any]) -> str:
+    if not warnings:
+        return ""
+    lines = [
+        "<details open>",
+        f"<summary><strong>⚠️ {md_escape(slice_title)} — Warnings</strong></summary>",
+        "",
+    ]
+    lines.extend(f"- {md_escape(warning)}" for warning in warnings)
+    lines.extend(["", "</details>"])
+    return "\n".join(lines)
+
+
+def render_slice(slice_data: dict[str, Any], *, level: int = 3) -> str:
+    comparison = slice_data.get("comparison", {})
+    worse_rows = comparison.get("worse", {}).get("rows", [])
+    better_rows = comparison.get("better", {}).get("rows", [])
+    warnings = slice_data.get("warnings", [])
+    child_sections = [
+        rendered
+        for child in slice_data.get("children", [])
+        if (rendered := render_slice(child, level=level + 1))
+    ]
+    direct_sections = [
+        section
+        for section in (
+            render_warning_details(slice_data.get("title", "Slice"), warnings),
+            render_direction_details(
+                slice_data.get("title", "Slice"), "worse", worse_rows
+            ),
+            render_direction_details(
+                slice_data.get("title", "Slice"), "better", better_rows
+            ),
+        )
+        if section
+    ]
+    if not direct_sections and not child_sections:
+        return ""
+
+    heading_prefix = "#" * min(level, 6)
+    subtitle = (
+        f"`-f {slice_data.get('filter', '')}` "
+        f"`{slice_data.get('timing', '')}` "
+        f"`--sort {slice_data.get('sort', '')}`"
+    )
+    lines = [
+        f"{heading_prefix} {md_escape(slice_data.get('title', 'Slice'))}",
+        "",
+        subtitle,
+        "",
+    ]
+    lines.extend(join_sections(direct_sections))
+    if child_sections:
+        lines.extend(["", *join_sections(child_sections)])
+    return "\n".join(lines).strip()
+
+
+def join_sections(sections: list[str]) -> list[str]:
+    lines: list[str] = []
+    for section in sections:
+        if lines:
+            lines.append("")
+        lines.append(section)
+    return lines
+
+
+def count_rows(slice_data: dict[str, Any], direction: str) -> int:
+    comparison = slice_data.get("comparison", {})
+    total = len(comparison.get(direction, {}).get("rows", []))
+    return total + sum(
+        count_rows(child, direction) for child in slice_data.get("children", [])
+    )
+
+
+def count_warnings(slice_data: dict[str, Any]) -> int:
+    return len(slice_data.get("warnings", [])) + sum(
+        count_warnings(child) for child in slice_data.get("children", [])
+    )
+
+
+def render_comment(
+    summary: dict[str, Any],
+    config: dict[str, Any],
+    *,
+    artifacts_url: str,
+) -> str:
+    config_id = str(config["id"])
+    slices = summary.get("slices", [])
+    sections = [
+        section for slice_data in slices if (section := render_slice(slice_data))
+    ]
+    worse_count = sum(count_rows(slice_data, "worse") for slice_data in slices)
+    better_count = sum(count_rows(slice_data, "better") for slice_data in slices)
+    warning_count = sum(count_warnings(slice_data) for slice_data in slices)
+    result = (
+        f"**Result:** {worse_count} regression row(s), "
+        f"{better_count} improvement row(s) above threshold."
+    )
+    if warning_count:
+        result += f" {warning_count} warning(s)."
+
+    lines = [
+        f"<!-- cccl-compile-time-bench: {md_escape(config_id)} -->",
+        f"## ⏱️ CCCL compile-time benchmark comparison: {md_escape(config.get('name', config_id))}",
+        "",
+        result,
+        "",
+        "| Run | Value |",
+        "| --- | --- |",
+        f"| Config | {md_code_span(config_id)} |",
+        f"| Baseline | {md_code_span(config.get('baseline_ref', ''))} |",
+        f"| Preset | {md_code_span(config.get('preset', ''))} |",
+        f"| Targets | {md_code_span(', '.join(config.get('targets', [])))} |",
+        f"| GPU / launch args | {md_code_span(config.get('gpu', ''))} / {md_code_span(config.get('launch_args', ''))} |",
+        "",
+        f"**Artifacts:** [reports and traces]({artifacts_url})",
+        "",
+    ]
+    if sections:
+        lines.extend(join_sections(sections))
+    else:
+        lines.append(
+            "No compile-time benchmark changes exceeded the configured thresholds."
+        )
+    return "\n".join(lines).rstrip() + "\n"
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Render a GitHub PR comment from compile-time report JSON."
+    )
+    parser.add_argument("--summary", type=Path, required=True)
+    parser.add_argument("--config", type=Path, required=True)
+    parser.add_argument("--artifacts-url", required=True)
+    parser.add_argument("-o", "--output", type=Path)
+    args = parser.parse_args()
+
+    comment = render_comment(
+        load_json(args.summary),
+        load_json(args.config),
+        artifacts_url=args.artifacts_url,
+    )
+    if args.output:
+        args.output.parent.mkdir(parents=True, exist_ok=True)
+        args.output.write_text(comment, encoding="utf-8")
+    else:
+        print(comment, end="")
+
+
+if __name__ == "__main__":
+    main()
diff --git a/cccl_upstream/ci/compile_time/summarize_events.py b/cccl_upstream/ci/compile_time/summarize_events.py
new file mode 100755
index 00000000..f591bc87
--- /dev/null
+++ b/cccl_upstream/ci/compile_time/summarize_events.py
@@ -0,0 +1,1718 @@
+#!/usr/bin/env python3
+
+import argparse
+import csv
+import heapq
+import json
+import re
+from collections import defaultdict
+from dataclasses import dataclass, field, replace
+from pathlib import Path
+from typing import Any, Callable
+
+DEFAULT_SCOPE_FILTER = r"(^|[^A-Za-z0-9_:])(?:::)?(?:cuda|thrust|cub|cccl)::"
+SYMBOL_SCOPE_EVENT_NAMES = {
+    "Scanning Function Body",
+    "Instantiating Template Class",
+    "Instantiating Template Function",
+    "Generating Function IR",
+    "OptFunction",
+}
+ITANIUM_CV_QUALIFIERS = frozenset("KOVR")
+
+
+@dataclass(frozen=True)
+class FilterSpec:
+    label: str
+    description: str
+    matches: Callable[["TraceEvent"], bool]
+    default_exclusive_scope: str = "all"
+
+
+@dataclass(frozen=True)
+class ReportConfig:
+    slice_id: str
+    title: str
+    spec: FilterSpec
+    timing: str
+    exclusive_scope: str
+    sort_by: str
+    top_n: int
+    tag: str | None
+    threshold_us: float = 0.0
+    scope_filter: re.Pattern[str] | None = None
+
+
+@dataclass
+class TraceEvent:
+    name: str
+    detail: str
+    start_us: int
+    end_us: int
+    pid: int
+    tid: int
+    root_tu: str
+    synthetic: bool = False
+    children: list["TraceEvent"] = field(default_factory=list)
+
+    @property
+    def inclusive_us(self) -> int:
+        return self.end_us - self.start_us
+
+    def key(self, repo_root: Path) -> str:
+        if self.detail:
+            return normalize_detail(self.detail, repo_root)
+        return self.name
+
+
+@dataclass
+class EventStats:
+    event_name: str
+    event_key: str
+    event_count: int = 0
+    total_inclusive_us: int = 0
+    total_exclusive_us: int = 0
+    max_inclusive_us: int = 0
+    max_exclusive_us: int = 0
+    trace_paths: set[str] = field(default_factory=set)
+    root_tus: set[str] = field(default_factory=set)
+
+
+@dataclass
+class ComparisonStats:
+    event_name: str
+    event_key: str
+    baseline: EventStats
+    current: EventStats
+    matched_trace_paths: set[str] = field(default_factory=set)
+
+
+@dataclass(frozen=True)
+class ComparisonRow:
+    stats: ComparisonStats
+    baseline_impact_us: float
+    current_impact_us: float
+    baseline_metric_us: float
+    current_metric_us: float
+    impact_magnitude_us: float
+
+    @property
+    def impact_delta_us(self) -> float:
+        return self.current_impact_us - self.baseline_impact_us
+
+    @property
+    def selected_delta_us(self) -> float:
+        return self.current_metric_us - self.baseline_metric_us
+
+    @property
+    def selected_magnitude_us(self) -> float:
+        return abs(self.selected_delta_us)
+
+
+@dataclass(frozen=True)
+class ComparisonSide:
+    name: str
+    repo_root: Path
+    events: list[TraceEvent]
+    report_ids: set[tuple[str, str]]
+    child_ids: set[tuple[str, str]]
+
+
+@dataclass(frozen=True)
+class ComparisonInput:
+    name: str
+    trace_paths: dict[Path, Path]
+    repo_root: Path
+
+
+@dataclass(frozen=True)
+class ReportSide:
+    name: str
+    trace_paths: list[Path]
+    repo_root: Path
+    output_dir: Path
+
+
+@dataclass(frozen=True)
+class SliceRequest:
+    config: ReportConfig
+    filter_name: str
+    children: tuple["SliceRequest", ...] = ()
+
+
+def merged_interval_duration(intervals: list[tuple[int, int]]) -> int:
+    if not intervals:
+        return 0
+
+    sorted_intervals = sorted(intervals)
+    total = 0
+    merged_start, merged_end = sorted_intervals[0]
+    for start, end in sorted_intervals[1:]:
+        if start > merged_end:
+            total += merged_end - merged_start
+            merged_start, merged_end = start, end
+        else:
+            merged_end = max(merged_end, end)
+
+    total += merged_end - merged_start
+    return total
+
+
+def generated_tu_input(tu: str) -> str:
+    marker = "/headers/"
+    if marker not in tu:
+        return tu
+
+    rel = tu.split(marker, 1)[1]
+    parts = rel.split("/", 1)
+    if len(parts) != 2:
+        return tu
+
+    tu_input = parts[1]
+    for suffix in (".cu", ".cpp", ".cxx", ".cc", ".c"):
+        if tu_input.endswith(suffix):
+            return tu_input[: -len(suffix)]
+
+    return tu_input
+
+
+def normalize_detail(detail: str, repo_root: Path) -> str:
+    detail_path = Path(detail)
+    if detail_path.is_absolute():
+        try:
+            detail = detail_path.resolve(strict=False).relative_to(repo_root).as_posix()
+        except ValueError:
+            pass
+
+    return detail
+
+
+def normalize_project_file(detail: str, repo_root: Path) -> str | None:
+    detail_path = Path(detail)
+    if not detail_path.is_absolute():
+        return None
+
+    try:
+        detail = detail_path.resolve(strict=False).relative_to(repo_root).as_posix()
+    except ValueError:
+        return None
+
+    if detail.startswith("build/"):
+        return None
+    return detail
+
+
+def general_event_identity(event: TraceEvent, repo_root: Path) -> tuple[str, str]:
+    return (event.name, event.key(repo_root))
+
+
+def strip_angle_arguments(symbol: str) -> str:
+    stripped: list[str] = []
+    depth = 0
+    for char in symbol:
+        if char == "<":
+            depth += 1
+            continue
+        if char == ">" and depth:
+            depth -= 1
+            continue
+        if depth == 0:
+            stripped.append(char)
+    return "".join(stripped)
+
+
+def symbol_name_prefix(symbol: str) -> str:
+    before_parameters = strip_angle_arguments(symbol).split("(", 1)[0].strip()
+    if not before_parameters:
+        return symbol
+    if "::operator" in before_parameters:
+        operator_scope = before_parameters.rfind("::operator")
+        prefix_start = before_parameters.rfind(" ", 0, operator_scope)
+        return before_parameters[prefix_start + 1 :]
+    return before_parameters.rsplit(None, 1)[-1]
+
+
+def itanium_nested_scope_candidates(symbol: str) -> list[str]:
+    if not symbol.startswith("_Z"):
+        return []
+
+    candidates: list[str] = []
+    for nested_marker in (i for i, char in enumerate(symbol) if char == "N"):
+        index = nested_marker + 1
+        while index < len(symbol) and symbol[index] in ITANIUM_CV_QUALIFIERS:
+            index += 1
+
+        scopes: list[str] = []
+        while index < len(symbol) and symbol[index].isdigit():
+            length_start = index
+            while index < len(symbol) and symbol[index].isdigit():
+                index += 1
+            try:
+                component_length = int(symbol[length_start:index])
+            except ValueError:
+                break
+
+            component = symbol[index : index + component_length]
+            if len(component) != component_length:
+                break
+
+            scopes.append(component)
+            candidates.append("::".join(scopes) + "::")
+            index += component_length
+
+    return candidates
+
+
+def symbol_scope_candidates(event: TraceEvent) -> list[str]:
+    if event.name not in SYMBOL_SCOPE_EVENT_NAMES or not event.detail:
+        return []
+
+    candidates = [symbol_name_prefix(event.detail)]
+    if " [" in event.detail:
+        _, bracketed_symbol = event.detail.split(" [", 1)
+        candidates.append(symbol_name_prefix(bracketed_symbol.rstrip("]")))
+
+    for candidate in list(candidates):
+        candidates.extend(itanium_nested_scope_candidates(candidate))
+
+    return candidates
+
+
+def matches_scope_filter(
+    event: TraceEvent, scope_filter: re.Pattern[str] | None
+) -> bool:
+    if scope_filter is None or event.name not in SYMBOL_SCOPE_EVENT_NAMES:
+        return True
+    return any(
+        scope_filter.search(candidate) for candidate in symbol_scope_candidates(event)
+    )
+
+
+def matches_report_config(event: TraceEvent, config: ReportConfig) -> bool:
+    return config.spec.matches(event) and matches_scope_filter(
+        event, config.scope_filter
+    )
+
+
+def report_event_identity(
+    event: TraceEvent, config: ReportConfig, repo_root: Path
+) -> tuple[str, str] | None:
+    if not matches_report_config(event, config):
+        return None
+
+    if config.spec.label == "file-processing":
+        event_key = normalize_project_file(event.detail, repo_root)
+        if event_key is None:
+            return None
+    else:
+        event_key = event.key(repo_root)
+
+    return (event.name, event_key)
+
+
+def filter_event_identity(
+    event: TraceEvent, config: ReportConfig, repo_root: Path
+) -> tuple[str, str] | None:
+    if not matches_report_config(event, config):
+        return None
+    return general_event_identity(event, repo_root)
+
+
+def trace_root_tu(trace: dict, trace_path: Path) -> str:
+    input_files = trace.get("otherData", {}).get("inputFiles", [])
+    if input_files:
+        return generated_tu_input(input_files[0])
+    return trace_path.as_posix()
+
+
+def iter_trace_paths(trace_dir: Path) -> list[Path]:
+    return sorted(p for p in trace_dir.rglob("*.json") if p.is_file())
+
+
+def iter_duration_events(trace_path: Path, repo_root: Path) -> list[TraceEvent]:
+    with trace_path.open(encoding="utf-8") as f:
+        trace = json.load(f)
+
+    root_tu = normalize_detail(trace_root_tu(trace, trace_path), repo_root)
+    events: list[TraceEvent] = []
+    for event in trace.get("traceEvents", []):
+        if event.get("ph") not in (None, "X"):
+            continue
+        if "ts" not in event or "dur" not in event:
+            continue
+
+        name = str(event.get("name", ""))
+        if not name:
+            continue
+
+        args = event.get("args", {})
+        detail = ""
+        if isinstance(args, dict):
+            detail = str(args.get("detail", "") or "")
+
+        start_us = int(event["ts"])
+        dur_us = int(event["dur"])
+        events.append(
+            TraceEvent(
+                name=name,
+                detail=detail,
+                start_us=start_us,
+                end_us=start_us + dur_us,
+                pid=int(event.get("pid", 0)),
+                tid=int(event.get("tid", 0)),
+                root_tu=root_tu,
+            )
+        )
+
+    if events:
+        trace_start_us = min(event.start_us for event in events)
+        trace_end_us = max(event.end_us for event in events)
+        events.append(
+            TraceEvent(
+                name="Total Compilation Time",
+                detail=root_tu,
+                start_us=trace_start_us,
+                end_us=trace_end_us,
+                pid=-1,
+                tid=-1,
+                root_tu=root_tu,
+                synthetic=True,
+            )
+        )
+
+    return events
+
+
+def link_child_events(events: list[TraceEvent]) -> None:
+    grouped: dict[tuple[int, int], list[TraceEvent]] = defaultdict(list)
+    for event in events:
+        grouped[(event.pid, event.tid)].append(event)
+
+    for thread_events in grouped.values():
+        stack: list[TraceEvent] = []
+        for event in sorted(thread_events, key=lambda e: (e.start_us, -e.end_us)):
+            # Example: A=[0,10], B=[10,20]. A ended before B starts, so it is
+            # not B's parent.
+            while stack and stack[-1].end_us <= event.start_us:
+                stack.pop()
+            # Example: A=[0,100], B=[50,150]. B overlaps A but is not fully
+            # contained by A, so A is not B's parent.
+            while stack and not (
+                stack[-1].start_us <= event.start_us
+                and event.end_us <= stack[-1].end_us
+            ):
+                stack.pop()
+
+            # Example: A=[0,100], B=[0,100]. Identical-span events are not
+            # treated as nested children of each other.
+            if stack and (
+                stack[-1].start_us != event.start_us or stack[-1].end_us != event.end_us
+            ):
+                stack[-1].children.append(event)
+
+            stack.append(event)
+
+
+def read_trace_events(trace_path: Path, repo_root: Path) -> list[TraceEvent]:
+    events = iter_duration_events(trace_path, repo_root)
+    link_child_events(events)
+    return events
+
+
+def event_name_filter(
+    *, label: str, description: str, event_names: tuple[str, ...]
+) -> FilterSpec:
+    name_set = set(event_names)
+    return FilterSpec(
+        label=label,
+        description=description,
+        matches=lambda event: event.name in name_set,
+    )
+
+
+def any_event_filter() -> FilterSpec:
+    return FilterSpec(
+        label="all",
+        description="all raw duration events",
+        matches=lambda event: not event.synthetic,
+    )
+
+
+def regex_filter(pattern: str) -> FilterSpec:
+    compiled = re.compile(pattern, re.IGNORECASE)
+    return FilterSpec(
+        label=f"regex-{slugify(pattern)}",
+        description=f"event name or detail matches /{pattern}/i",
+        matches=lambda event: (
+            bool(compiled.search(event.name)) or bool(compiled.search(event.detail))
+        )
+        and not event.synthetic,
+    )
+
+
+def builtin_filters() -> dict[str, FilterSpec]:
+    filters: dict[str, FilterSpec] = {}
+
+    def add(spec: FilterSpec) -> None:
+        filters[spec.label] = spec
+
+    add(any_event_filter())
+    add(
+        FilterSpec(
+            label="file-processing",
+            description=(
+                "PHF trace events; exclusive time subtracts nested PHF events "
+                "to match the direct file-processing metric"
+            ),
+            matches=lambda event: event.name == "Processing Header File",
+            default_exclusive_scope="same-filter",
+        ),
+    )
+    add(
+        event_name_filter(
+            label="scanning-function-body",
+            description="Scanning Function Body events",
+            event_names=("Scanning Function Body",),
+        ),
+    )
+    add(
+        event_name_filter(
+            label="template-instantiation",
+            description="template class/function instantiation events",
+            event_names=(
+                "Instantiating Template Class",
+                "Instantiating Template Function",
+            ),
+        ),
+    )
+    add(
+        event_name_filter(
+            label="template-class-instantiation",
+            description="template class instantiation events",
+            event_names=("Instantiating Template Class",),
+        ),
+    )
+    add(
+        event_name_filter(
+            label="template-function-instantiation",
+            description="template function instantiation events",
+            event_names=("Instantiating Template Function",),
+        ),
+    )
+    add(
+        event_name_filter(
+            label="pending-instantiations",
+            description="pending template instantiation phase events",
+            event_names=("Generating Needed Template Instantiations",),
+        ),
+    )
+    add(
+        event_name_filter(
+            label="frontend",
+            description="front-end phase events",
+            event_names=(
+                "Front End Cleanup",
+                "CUDA C++ Front-End",
+            ),
+        ),
+    )
+    add(
+        event_name_filter(
+            label="host-compiler",
+            description="host compiler preprocessing and compiling events",
+            event_names=(
+                "g++ (preprocessing 1)",
+                "g++ (preprocessing 4)",
+                "g++ (compiling)",
+                "gcc (preprocessing 1)",
+                "gcc (preprocessing 4)",
+                "gcc (compiling)",
+            ),
+        ),
+    )
+    add(
+        event_name_filter(
+            label="code-generation",
+            description="code generation events",
+            event_names=(
+                "Generating Function IR",
+                "Generating NVVM IR",
+                "NVVM CodeGen",
+            ),
+        ),
+    )
+    add(
+        event_name_filter(
+            label="optimizer",
+            description="optimizer events",
+            event_names=(
+                "OptFunction",
+                "NVVM Optimizer",
+            ),
+        ),
+    )
+    add(
+        event_name_filter(
+            label="total-compilation",
+            description=(
+                "synthetic per-trace wall-clock span from first to last timed "
+                "trace event"
+            ),
+            event_names=("Total Compilation Time",),
+        ),
+    )
+
+    return filters
+
+
+def resolve_filter(filter_name: str) -> FilterSpec:
+    filters = builtin_filters()
+    normalized = filter_name.strip().lower()
+    if normalized in filters:
+        return filters[normalized]
+
+    return regex_filter(filter_name)
+
+
+def exclusive_child_events(event: TraceEvent, config: ReportConfig) -> list[TraceEvent]:
+    if config.exclusive_scope == "all":
+        return event.children
+    if config.exclusive_scope == "same-filter":
+        return [
+            child for child in event.children if matches_report_config(child, config)
+        ]
+    raise ValueError(f"unknown exclusive scope: {config.exclusive_scope}")
+
+
+def event_exclusive_us(event: TraceEvent, config: ReportConfig) -> int:
+    child_intervals = [
+        (child.start_us, child.end_us)
+        for child in exclusive_child_events(event, config)
+    ]
+    return max(0, event.inclusive_us - merged_interval_duration(child_intervals))
+
+
+def collect_stats(
+    trace_paths: list[Path],
+    repo_root: Path,
+    config: ReportConfig,
+) -> dict[tuple[str, str], EventStats]:
+    stats: dict[tuple[str, str], EventStats] = {}
+
+    for trace_path in trace_paths:
+        events = read_trace_events(trace_path, repo_root)
+        collect_trace_stats(
+            stats,
+            events,
+            trace_path.as_posix(),
+            repo_root,
+            report_config=config,
+            exclusive_config=config,
+        )
+
+    return stats
+
+
+def collect_trace_stats(
+    stats: dict[tuple[str, str], EventStats],
+    events: list[TraceEvent],
+    trace_path: str,
+    repo_root: Path,
+    *,
+    report_config: ReportConfig,
+    exclusive_config: ReportConfig,
+) -> None:
+    for event in events:
+        identity = report_event_identity(event, report_config, repo_root)
+        if identity is None:
+            continue
+        add_event_stats(
+            stats,
+            identity,
+            event.inclusive_us,
+            event_exclusive_us(event, exclusive_config),
+            trace_path,
+            event.root_tu,
+        )
+
+
+def add_event_stats(
+    stats: dict[tuple[str, str], EventStats],
+    identity: tuple[str, str],
+    inclusive_us: int,
+    exclusive_us: int,
+    trace_path: str,
+    root_tu: str,
+) -> None:
+    event_name, event_key = identity
+    event_stats = stats.setdefault(
+        identity, EventStats(event_name=event_name, event_key=event_key)
+    )
+    merge_event_stats(
+        event_stats,
+        EventStats(
+            event_name=event_name,
+            event_key=event_key,
+            event_count=1,
+            total_inclusive_us=inclusive_us,
+            total_exclusive_us=exclusive_us,
+            max_inclusive_us=inclusive_us,
+            max_exclusive_us=exclusive_us,
+            trace_paths={trace_path},
+            root_tus={root_tu},
+        ),
+    )
+
+
+def selected_total_us(stats: EventStats, timing: str) -> int:
+    if timing == "inclusive":
+        return stats.total_inclusive_us
+    if timing == "exclusive":
+        return stats.total_exclusive_us
+    raise ValueError(f"unknown timing: {timing}")
+
+
+def average_us(total_us: int, count: int) -> float:
+    if count == 0:
+        return 0.0
+    return total_us / count
+
+
+def selected_avg_us(stats: EventStats, timing: str) -> float:
+    return average_us(selected_total_us(stats, timing), stats.event_count)
+
+
+def selected_avg_per_root_tu_us(stats: EventStats, timing: str) -> float:
+    return average_us(selected_total_us(stats, timing), len(stats.root_tus))
+
+
+def selected_max_us(stats: EventStats, timing: str) -> int:
+    if timing == "inclusive":
+        return stats.max_inclusive_us
+    if timing == "exclusive":
+        return stats.max_exclusive_us
+    raise ValueError(f"unknown timing: {timing}")
+
+
+def selected_metric_us(stats: EventStats, timing: str, sort_by: str) -> float:
+    if sort_by == "total":
+        return float(selected_total_us(stats, timing))
+    if sort_by == "avg":
+        return selected_avg_us(stats, timing)
+    if sort_by == "avg-root-tu":
+        return selected_avg_per_root_tu_us(stats, timing)
+    if sort_by == "max":
+        return float(selected_max_us(stats, timing))
+    raise ValueError(f"unknown sort: {sort_by}")
+
+
+def trace_paths_by_relative_root(trace_dir: Path) -> dict[Path, Path]:
+    return {path.relative_to(trace_dir): path for path in iter_trace_paths(trace_dir)}
+
+
+def comparison_input(name: str, trace_dir: Path, repo_root: Path) -> ComparisonInput:
+    return ComparisonInput(
+        name=name,
+        trace_paths=trace_paths_by_relative_root(trace_dir),
+        repo_root=repo_root,
+    )
+
+
+def comparable_child_identities(
+    events: list[TraceEvent],
+    repo_root: Path,
+    config: ReportConfig,
+) -> set[tuple[str, str]]:
+    if config.exclusive_scope == "all":
+        return {general_event_identity(event, repo_root) for event in events}
+    if config.exclusive_scope == "same-filter":
+        return {
+            identity
+            for event in events
+            if (identity := filter_event_identity(event, config, repo_root)) is not None
+        }
+    raise ValueError(f"unknown exclusive scope: {config.exclusive_scope}")
+
+
+def comparable_report_filter(
+    config: ReportConfig,
+    repo_root: Path,
+    comparable_report_ids: set[tuple[str, str]],
+) -> FilterSpec:
+    def matches(event: TraceEvent) -> bool:
+        identity = report_event_identity(event, config, repo_root)
+        return identity is not None and identity in comparable_report_ids
+
+    return replace(config.spec, matches=matches)
+
+
+def comparable_child_filter(
+    config: ReportConfig,
+    repo_root: Path,
+    comparable_child_ids: set[tuple[str, str]],
+) -> FilterSpec:
+    def child_identity(event: TraceEvent) -> tuple[str, str] | None:
+        if config.exclusive_scope == "all":
+            return general_event_identity(event, repo_root)
+        if config.exclusive_scope == "same-filter":
+            return filter_event_identity(event, config, repo_root)
+        raise ValueError(f"unknown exclusive scope: {config.exclusive_scope}")
+
+    def matches(event: TraceEvent) -> bool:
+        identity = child_identity(event)
+        return identity is not None and identity in comparable_child_ids
+
+    return replace(config.spec, matches=matches)
+
+
+def merge_event_stats(target: EventStats, source: EventStats) -> None:
+    target.event_count += source.event_count
+    target.total_inclusive_us += source.total_inclusive_us
+    target.total_exclusive_us += source.total_exclusive_us
+    target.max_inclusive_us = max(target.max_inclusive_us, source.max_inclusive_us)
+    target.max_exclusive_us = max(target.max_exclusive_us, source.max_exclusive_us)
+    target.trace_paths.update(source.trace_paths)
+    target.root_tus.update(source.root_tus)
+
+
+def merge_comparison_side_stats(
+    comparison_stats: dict[tuple[str, str], ComparisonStats],
+    side_name: str,
+    side_stats: dict[tuple[str, str], EventStats],
+) -> None:
+    for identity, source_stats in side_stats.items():
+        event_name, event_key = identity
+        comparison = comparison_stats.setdefault(
+            identity,
+            ComparisonStats(
+                event_name=event_name,
+                event_key=event_key,
+                baseline=EventStats(event_name, event_key),
+                current=EventStats(event_name, event_key),
+            ),
+        )
+        target_stats = getattr(comparison, side_name)
+        merge_event_stats(target_stats, source_stats)
+        comparison.matched_trace_paths.update(source_stats.trace_paths)
+
+
+def read_comparison_side(
+    name: str,
+    trace_path: Path,
+    repo_root: Path,
+    config: ReportConfig,
+) -> ComparisonSide:
+    events = read_trace_events(trace_path, repo_root)
+    report_ids = {
+        identity
+        for event in events
+        if (identity := report_event_identity(event, config, repo_root)) is not None
+    }
+    child_ids = comparable_child_identities(events, repo_root, config)
+    return ComparisonSide(
+        name=name,
+        repo_root=repo_root,
+        events=events,
+        report_ids=report_ids,
+        child_ids=child_ids,
+    )
+
+
+def collect_comparison_stats(
+    baseline_trace_dir: Path,
+    current_trace_dir: Path,
+    baseline_repo_root: Path,
+    current_repo_root: Path,
+    config: ReportConfig,
+) -> tuple[dict[tuple[str, str], ComparisonStats], int]:
+    comparison_inputs = (
+        comparison_input("baseline", baseline_trace_dir, baseline_repo_root),
+        comparison_input("current", current_trace_dir, current_repo_root),
+    )
+    matched_rel_paths = sorted(
+        set.intersection(
+            *(
+                set(comparison_input.trace_paths)
+                for comparison_input in comparison_inputs
+            )
+        )
+    )
+    comparison_stats: dict[tuple[str, str], ComparisonStats] = {}
+
+    for rel_path in matched_rel_paths:
+        sides = tuple(
+            read_comparison_side(
+                comparison_input.name,
+                comparison_input.trace_paths[rel_path],
+                comparison_input.repo_root,
+                config,
+            )
+            for comparison_input in comparison_inputs
+        )
+
+        comparable_report_ids = set.intersection(*(side.report_ids for side in sides))
+        if not comparable_report_ids:
+            continue
+
+        comparable_child_ids = set.intersection(*(side.child_ids for side in sides))
+        rel_path_str = rel_path.as_posix()
+
+        for side in sides:
+            report_config = replace(
+                config,
+                spec=comparable_report_filter(
+                    config, side.repo_root, comparable_report_ids
+                ),
+            )
+            exclusive_config = replace(
+                config,
+                spec=comparable_child_filter(
+                    config, side.repo_root, comparable_child_ids
+                ),
+                exclusive_scope="same-filter",
+                scope_filter=(
+                    config.scope_filter
+                    if config.exclusive_scope == "same-filter"
+                    else None
+                ),
+            )
+            side_stats: dict[tuple[str, str], EventStats] = {}
+            collect_trace_stats(
+                side_stats,
+                side.events,
+                rel_path_str,
+                side.repo_root,
+                report_config=report_config,
+                exclusive_config=exclusive_config,
+            )
+            merge_comparison_side_stats(comparison_stats, side.name, side_stats)
+
+    return comparison_stats, len(matched_rel_paths)
+
+
+def sorted_rows(
+    stats: dict[tuple[str, str], EventStats], config: ReportConfig
+) -> list[EventStats]:
+    if config.sort_by not in ("total", "avg", "avg-root-tu", "max"):
+        raise ValueError(f"unknown sort: {config.sort_by}")
+
+    # Python has heapq.nsmallest rather than a C++-style partial_sort. The
+    # selected metric is negated here so "smallest" means "largest selected
+    # time", while the string tie-breakers keep their natural ascending order.
+    return heapq.nsmallest(
+        config.top_n,
+        stats.values(),
+        key=lambda item: (
+            -selected_metric_us(item, config.timing, config.sort_by),
+            item.event_name,
+            item.event_key,
+        ),
+    )
+
+
+def seconds(us: float | int) -> str:
+    return f"{us / 1_000_000.0:.6f}"
+
+
+def slugify(value: str) -> str:
+    slug = re.sub(r"[^A-Za-z0-9_.-]+", "-", value.strip()).strip("-").lower()
+    return slug or "report"
+
+
+def default_output_path(
+    output_dir: Path,
+    config: ReportConfig,
+) -> Path:
+    pieces = ["top", str(config.top_n), config.spec.label, config.timing]
+    if config.timing == "exclusive":
+        pieces.append(config.exclusive_scope)
+    pieces.append(f"by-{config.sort_by}")
+    if config.tag:
+        pieces.append(slugify(config.tag))
+    return output_dir / ("-".join(slugify(piece) for piece in pieces) + ".csv")
+
+
+def comparison_output_path(
+    output_dir: Path,
+    config: ReportConfig,
+    direction: str,
+) -> Path:
+    pieces = ["top", str(config.top_n), config.spec.label, config.timing]
+    if config.timing == "exclusive":
+        pieces.append(config.exclusive_scope)
+    pieces.extend([f"by-{config.sort_by}", direction])
+    if config.tag:
+        pieces.append(slugify(config.tag))
+    return output_dir / ("-".join(slugify(piece) for piece in pieces) + ".csv")
+
+
+def event_stats_csv_row(rank: int, row: EventStats, timing: str) -> dict[str, object]:
+    root_tu_count = len(row.root_tus)
+    return {
+        "rank": rank,
+        "event_name": row.event_name,
+        "event_key": row.event_key,
+        "selected_total_s": seconds(selected_total_us(row, timing)),
+        "selected_avg_per_event_s": seconds(selected_avg_us(row, timing)),
+        "selected_avg_per_root_tu_s": seconds(selected_avg_per_root_tu_us(row, timing)),
+        "selected_max_s": seconds(selected_max_us(row, timing)),
+        "total_inclusive_s": seconds(row.total_inclusive_us),
+        "avg_inclusive_per_event_s": seconds(
+            average_us(row.total_inclusive_us, row.event_count)
+        ),
+        "avg_inclusive_per_root_tu_s": seconds(
+            average_us(row.total_inclusive_us, root_tu_count)
+        ),
+        "max_inclusive_s": seconds(row.max_inclusive_us),
+        "total_exclusive_s": seconds(row.total_exclusive_us),
+        "avg_exclusive_per_event_s": seconds(
+            average_us(row.total_exclusive_us, row.event_count)
+        ),
+        "avg_exclusive_per_root_tu_s": seconds(
+            average_us(row.total_exclusive_us, root_tu_count)
+        ),
+        "max_exclusive_s": seconds(row.max_exclusive_us),
+        "event_count": row.event_count,
+        "trace_count": len(row.trace_paths),
+        "root_tu_count": root_tu_count,
+    }
+
+
+def write_csv(
+    output_csv: Path,
+    rows: list[EventStats],
+    timing: str,
+) -> None:
+    output_csv.parent.mkdir(parents=True, exist_ok=True)
+    with output_csv.open("w", newline="", encoding="utf-8") as f:
+        writer = csv.DictWriter(
+            f,
+            fieldnames=[
+                "rank",
+                "event_name",
+                "event_key",
+                "selected_total_s",
+                "selected_avg_per_event_s",
+                "selected_avg_per_root_tu_s",
+                "selected_max_s",
+                "total_inclusive_s",
+                "avg_inclusive_per_event_s",
+                "avg_inclusive_per_root_tu_s",
+                "max_inclusive_s",
+                "total_exclusive_s",
+                "avg_exclusive_per_event_s",
+                "avg_exclusive_per_root_tu_s",
+                "max_exclusive_s",
+                "event_count",
+                "trace_count",
+                "root_tu_count",
+            ],
+        )
+        writer.writeheader()
+        for rank, row in enumerate(rows, start=1):
+            writer.writerow(event_stats_csv_row(rank, row, timing))
+
+
+def comparison_row_dict(
+    rank: int, row: ComparisonRow, timing: str
+) -> dict[str, object]:
+    stats = row.stats
+    return {
+        "rank": rank,
+        "event_name": stats.event_name,
+        "event_key": stats.event_key,
+        "baseline_impact_s": seconds(row.baseline_impact_us),
+        "current_impact_s": seconds(row.current_impact_us),
+        "impact_delta_s": seconds(row.impact_delta_us),
+        "impact_magnitude_s": seconds(row.impact_magnitude_us),
+        "baseline_selected_s": seconds(row.baseline_metric_us),
+        "current_selected_s": seconds(row.current_metric_us),
+        "selected_delta_s": seconds(row.selected_delta_us),
+        "selected_magnitude_s": seconds(row.selected_magnitude_us),
+        "baseline_total_inclusive_s": seconds(stats.baseline.total_inclusive_us),
+        "current_total_inclusive_s": seconds(stats.current.total_inclusive_us),
+        "baseline_total_exclusive_s": seconds(stats.baseline.total_exclusive_us),
+        "current_total_exclusive_s": seconds(stats.current.total_exclusive_us),
+        "baseline_event_count": stats.baseline.event_count,
+        "current_event_count": stats.current.event_count,
+        "matched_trace_count": len(stats.matched_trace_paths),
+    }
+
+
+def write_json(path: Path, payload: dict[str, Any]) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    with path.open("w", encoding="utf-8") as f:
+        json.dump(payload, f, indent=2, sort_keys=True)
+
+
+def report_side(
+    name: str,
+    trace_dir: Path,
+    repo_root: Path,
+    output_dir: Path,
+) -> ReportSide:
+    trace_paths = iter_trace_paths(trace_dir)
+    if not trace_paths:
+        raise SystemExit(f"no JSON traces found under {trace_dir}")
+    return ReportSide(
+        name=name,
+        trace_paths=trace_paths,
+        repo_root=repo_root,
+        output_dir=output_dir,
+    )
+
+
+def write_side_report(
+    side: ReportSide,
+    config: ReportConfig,
+) -> tuple[Path, int]:
+    stats = collect_stats(side.trace_paths, side.repo_root, config)
+    rows = sorted_rows(stats, config) if stats else []
+    output_csv = default_output_path(side.output_dir, config)
+    write_csv(output_csv, rows, config.timing)
+    return output_csv, len(rows)
+
+
+def comparison_rows(
+    stats: dict[tuple[str, str], ComparisonStats],
+    config: ReportConfig,
+    direction: str,
+) -> list[ComparisonRow]:
+    if direction == "worse":
+        multiplier = 1
+    elif direction == "better":
+        multiplier = -1
+    else:
+        raise ValueError(f"unknown comparison direction: {direction}")
+
+    rows: list[ComparisonRow] = []
+    for comparison in stats.values():
+        baseline_impact = float(selected_total_us(comparison.baseline, config.timing))
+        current_impact = float(selected_total_us(comparison.current, config.timing))
+        baseline_metric = selected_metric_us(
+            comparison.baseline, config.timing, config.sort_by
+        )
+        current_metric = selected_metric_us(
+            comparison.current, config.timing, config.sort_by
+        )
+        delta = current_impact - baseline_impact
+        magnitude = multiplier * delta
+        if magnitude <= config.threshold_us:
+            continue
+        rows.append(
+            ComparisonRow(
+                comparison,
+                baseline_impact,
+                current_impact,
+                baseline_metric,
+                current_metric,
+                magnitude,
+            )
+        )
+
+    # Python has heapq.nsmallest rather than a C++-style partial_sort. The
+    # total-impact change is negated here so "smallest" means "largest
+    # requested aggregate movement across matched traces", while the string
+    # tie-breakers keep their natural ascending order.
+    return heapq.nsmallest(
+        config.top_n,
+        rows,
+        key=lambda row: (
+            -row.impact_magnitude_us,
+            row.stats.event_name,
+            row.stats.event_key,
+        ),
+    )
+
+
+def write_comparison_csv(
+    output_csv: Path,
+    rows: list[ComparisonRow],
+    timing: str,
+) -> None:
+    output_csv.parent.mkdir(parents=True, exist_ok=True)
+    with output_csv.open("w", newline="", encoding="utf-8") as f:
+        writer = csv.DictWriter(
+            f,
+            fieldnames=[
+                "rank",
+                "event_name",
+                "event_key",
+                "baseline_impact_s",
+                "current_impact_s",
+                "impact_delta_s",
+                "impact_magnitude_s",
+                "baseline_selected_s",
+                "current_selected_s",
+                "selected_delta_s",
+                "selected_magnitude_s",
+                "baseline_total_inclusive_s",
+                "current_total_inclusive_s",
+                "baseline_total_exclusive_s",
+                "current_total_exclusive_s",
+                "baseline_event_count",
+                "current_event_count",
+                "matched_trace_count",
+            ],
+        )
+        writer.writeheader()
+        for rank, row in enumerate(rows, start=1):
+            writer.writerow(comparison_row_dict(rank, row, timing))
+
+
+def print_filters() -> None:
+    for name, spec in sorted(builtin_filters().items()):
+        print(f"{name:32} {spec.description}")
+
+
+def compile_scope_filter(
+    pattern: str, parser: argparse.ArgumentParser
+) -> re.Pattern[str] | None:
+    if not pattern:
+        return None
+    try:
+        return re.compile(pattern)
+    except re.error as e:
+        parser.error(f"invalid --scope-filter regex: {e}")
+
+
+def report_config(
+    *,
+    slice_id: str,
+    title: str,
+    filter_name: str,
+    timing: str,
+    exclusive_scope: str,
+    sort_by: str,
+    top_n: int,
+    tag: str | None,
+    threshold_s: float,
+    scope_filter: re.Pattern[str] | None,
+) -> ReportConfig:
+    spec = resolve_filter(filter_name)
+    resolved_exclusive_scope = (
+        spec.default_exclusive_scope if exclusive_scope == "auto" else exclusive_scope
+    )
+    return ReportConfig(
+        slice_id=slice_id,
+        title=title,
+        spec=spec,
+        timing=timing,
+        exclusive_scope=resolved_exclusive_scope,
+        sort_by=sort_by,
+        top_n=top_n,
+        tag=tag,
+        threshold_us=threshold_s * 1_000_000.0,
+        scope_filter=scope_filter,
+    )
+
+
+def single_slice_request(
+    args: argparse.Namespace, parser: argparse.ArgumentParser
+) -> SliceRequest:
+    config = report_config(
+        slice_id=slugify(args.tag or args.filter),
+        title=args.tag or resolve_filter(args.filter).description,
+        filter_name=args.filter,
+        timing=args.timing,
+        exclusive_scope=args.exclusive_scope,
+        sort_by=args.sort,
+        top_n=args.top,
+        tag=args.tag,
+        threshold_s=args.threshold,
+        scope_filter=compile_scope_filter(args.scope_filter, parser),
+    )
+    return SliceRequest(config=config, filter_name=args.filter)
+
+
+def require_slice_field(slice_data: dict[str, Any], field_name: str, path: str) -> Any:
+    if field_name not in slice_data:
+        raise ValueError(f"{path}: missing required field '{field_name}'")
+    return slice_data[field_name]
+
+
+def validate_slice_id(value: Any, path: str) -> str:
+    if not isinstance(value, str) or not re.fullmatch(r"[a-z0-9][a-z0-9_.-]*", value):
+        raise ValueError(f"{path}: id must match ^[a-z0-9][a-z0-9_.-]*$")
+    return value
+
+
+def slice_request_from_json(
+    slice_data: dict[str, Any],
+    parser: argparse.ArgumentParser,
+    *,
+    path: str,
+    seen_ids: set[str],
+) -> SliceRequest:
+    if not isinstance(slice_data, dict):
+        raise ValueError(f"{path}: slice entry must be an object")
+
+    slice_id = validate_slice_id(require_slice_field(slice_data, "id", path), path)
+    if slice_id in seen_ids:
+        raise ValueError(f"{path}: duplicate slice id '{slice_id}'")
+    seen_ids.add(slice_id)
+
+    title = require_slice_field(slice_data, "title", path)
+    filter_name = require_slice_field(slice_data, "filter", path)
+    timing = require_slice_field(slice_data, "timing", path)
+    sort_by = require_slice_field(slice_data, "sort", path)
+    top_n = require_slice_field(slice_data, "top", path)
+    threshold = require_slice_field(slice_data, "threshold", path)
+    exclusive_scope = slice_data.get("exclusive_scope", "auto")
+    scope_filter_pattern = slice_data.get("scope_filter", DEFAULT_SCOPE_FILTER)
+
+    if not isinstance(title, str) or not title:
+        raise ValueError(f"{path}: title must be a non-empty string")
+    if not isinstance(filter_name, str) or not filter_name:
+        raise ValueError(f"{path}: filter must be a non-empty string")
+    if timing not in ("inclusive", "exclusive"):
+        raise ValueError(f"{path}: timing must be 'inclusive' or 'exclusive'")
+    if sort_by not in ("total", "avg", "avg-root-tu", "max"):
+        raise ValueError(f"{path}: unsupported sort '{sort_by}'")
+    if exclusive_scope not in ("auto", "all", "same-filter"):
+        raise ValueError(f"{path}: unsupported exclusive_scope '{exclusive_scope}'")
+    if isinstance(top_n, bool) or not isinstance(top_n, int) or top_n <= 0:
+        raise ValueError(f"{path}: top must be a positive integer")
+    if (
+        isinstance(threshold, bool)
+        or not isinstance(threshold, (int, float))
+        or threshold < 0
+    ):
+        raise ValueError(f"{path}: threshold must be a non-negative number")
+    if not isinstance(scope_filter_pattern, str):
+        raise ValueError(f"{path}: scope_filter must be a string")
+
+    children_data = slice_data.get("children", [])
+    if not isinstance(children_data, list):
+        raise ValueError(f"{path}: children must be a list")
+
+    config = report_config(
+        slice_id=slice_id,
+        title=title,
+        filter_name=filter_name,
+        timing=timing,
+        exclusive_scope=exclusive_scope,
+        sort_by=sort_by,
+        top_n=top_n,
+        tag=None,
+        threshold_s=float(threshold),
+        scope_filter=compile_scope_filter(scope_filter_pattern, parser),
+    )
+    children = tuple(
+        slice_request_from_json(
+            child,
+            parser,
+            path=f"{path}.children[{index}]",
+            seen_ids=seen_ids,
+        )
+        for index, child in enumerate(children_data)
+    )
+    return SliceRequest(config=config, filter_name=filter_name, children=children)
+
+
+def read_slice_requests(
+    slices_path: Path, parser: argparse.ArgumentParser
+) -> list[SliceRequest]:
+    try:
+        with slices_path.open(encoding="utf-8") as f:
+            payload = json.load(f)
+    except OSError as e:
+        parser.error(f"failed to read --slices file: {e}")
+    except json.JSONDecodeError as e:
+        parser.error(f"failed to parse --slices JSON: {e}")
+
+    slices_data = payload.get("slices") if isinstance(payload, dict) else payload
+    if not isinstance(slices_data, list) or not slices_data:
+        parser.error(
+            "--slices JSON must be a non-empty list or an object with a non-empty 'slices' list"
+        )
+
+    seen_ids: set[str] = set()
+    try:
+        return [
+            slice_request_from_json(
+                slice_data,
+                parser,
+                path=f"slices[{index}]",
+                seen_ids=seen_ids,
+            )
+            for index, slice_data in enumerate(slices_data)
+        ]
+    except ValueError as e:
+        parser.error(str(e))
+
+
+def run_slice_report(
+    request: SliceRequest,
+    *,
+    trace_dir: Path,
+    baseline_dir: Path | None,
+    repo_root: Path,
+    baseline_repo_root: Path,
+    output_dir: Path,
+    output_csv: Path | None,
+    allow_empty: bool,
+) -> dict[str, Any]:
+    config = request.config
+    slice_output_dir = output_dir
+    manifest: dict[str, Any] = {
+        "id": config.slice_id,
+        "title": config.title,
+        "filter": request.filter_name,
+        "filter_label": config.spec.label,
+        "timing": config.timing,
+        "exclusive_scope": config.exclusive_scope,
+        "sort": config.sort_by,
+        "top": config.top_n,
+        "threshold_s": config.threshold_us / 1_000_000.0,
+        "output_dir": slice_output_dir.as_posix(),
+        "children": [],
+    }
+
+    if baseline_dir is not None:
+        comparison_output_dir = slice_output_dir / "comparison"
+        report_sides = (
+            report_side(
+                "baseline",
+                baseline_dir,
+                baseline_repo_root,
+                slice_output_dir / "baseline",
+            ),
+            report_side("current", trace_dir, repo_root, slice_output_dir / "current"),
+        )
+        report_csvs: dict[str, tuple[Path, int]] = {}
+        for side in report_sides:
+            report_csvs[side.name] = write_side_report(side, config)
+
+        comparison_stats, matched_trace_count = collect_comparison_stats(
+            baseline_dir,
+            trace_dir,
+            baseline_repo_root,
+            repo_root,
+            config,
+        )
+        warnings: list[str] = []
+        for side_name, (_, row_count) in report_csvs.items():
+            if row_count == 0:
+                warnings.append(
+                    f"{side_name} report matched no events for this slice; "
+                    f"check filter '{request.filter_name}', scope filtering, "
+                    "and trace format"
+                )
+        if matched_trace_count == 0:
+            warnings.append(
+                "baseline and current trace directories have no matching trace files"
+            )
+        elif not comparison_stats:
+            warnings.append(
+                "baseline and current traces have no comparable event keys for this slice"
+            )
+
+        comparison_manifest: dict[str, Any] = {
+            "matched_trace_count": matched_trace_count,
+        }
+        wrote: list[tuple[Path, int]] = []
+        for direction in ("worse", "better"):
+            rows = comparison_rows(comparison_stats, config, direction)
+            comparison_csv = comparison_output_path(
+                comparison_output_dir,
+                config,
+                direction,
+            )
+            write_comparison_csv(comparison_csv, rows, config.timing)
+            row_dicts = [
+                comparison_row_dict(rank, row, config.timing)
+                for rank, row in enumerate(rows, start=1)
+            ]
+            comparison_manifest[direction] = {
+                "csv": comparison_csv.as_posix(),
+                "row_count": len(rows),
+                "rows": row_dicts,
+            }
+            wrote.append((comparison_csv, len(rows)))
+
+        manifest["reports"] = {
+            side: {"csv": path.as_posix(), "row_count": row_count}
+            for side, (path, row_count) in report_csvs.items()
+        }
+        manifest["comparison"] = comparison_manifest
+        if warnings:
+            manifest["warnings"] = warnings
+        trace_counts = ", ".join(
+            f"{len(side.trace_paths)} {side.name} trace(s)" for side in report_sides
+        )
+        print(
+            f"wrote slice '{config.slice_id}' baseline/current reports and "
+            f"comparison reports from {trace_counts}, "
+            f"{matched_trace_count} matched trace file(s):"
+        )
+        for side_name, (path, _) in report_csvs.items():
+            print(f"  {side_name}: {path}")
+        for path, row_count in wrote:
+            print(f"  comparison ({row_count} row(s)): {path}")
+        for warning in warnings:
+            print(f"  warning: {warning}")
+    else:
+        side = report_side("current", trace_dir, repo_root, slice_output_dir)
+        stats = collect_stats(side.trace_paths, side.repo_root, config)
+        if not stats and not allow_empty:
+            raise SystemExit(
+                f"no events matched filter '{request.filter_name}' "
+                f"in {len(side.trace_paths)} trace(s)"
+            )
+
+        rows = sorted_rows(stats, config) if stats else []
+        report_csv = (
+            output_csv
+            if output_csv is not None
+            else default_output_path(slice_output_dir, config)
+        )
+        write_csv(report_csv, rows, config.timing)
+        manifest["reports"] = {
+            "current": {"csv": report_csv.as_posix(), "row_count": len(rows)}
+        }
+        if not rows:
+            manifest["warnings"] = [
+                f"current report matched no events for filter '{request.filter_name}'"
+            ]
+        print(
+            f"wrote slice '{config.slice_id}' {len(rows)} row(s) "
+            f"from {len(side.trace_paths)} trace(s) to {report_csv}"
+        )
+
+    manifest["children"] = [
+        run_slice_report(
+            child,
+            trace_dir=trace_dir,
+            baseline_dir=baseline_dir,
+            repo_root=repo_root,
+            baseline_repo_root=baseline_repo_root,
+            output_dir=output_dir / child.config.slice_id,
+            output_csv=None,
+            allow_empty=allow_empty,
+        )
+        for child in request.children
+    ]
+    return manifest
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description=(
+            "Emit a top-N CSV report for events in NVCC --fdevice-time-trace JSON files."
+        )
+    )
+    parser.add_argument(
+        "trace_dir",
+        type=Path,
+        nargs="?",
+        help="directory containing device-time-trace JSON files",
+    )
+    parser.add_argument(
+        "-f",
+        "--filter",
+        default="file-processing",
+        help=(
+            "canonical event filter name or case-insensitive regex over event "
+            "name/detail (default: file-processing); use --list-filters to see "
+            "built-in filters"
+        ),
+    )
+    timing = parser.add_mutually_exclusive_group()
+    timing.add_argument(
+        "-i",
+        "--inclusive",
+        action="store_const",
+        const="inclusive",
+        dest="timing",
+        help="rank by inclusive event time",
+    )
+    timing.add_argument(
+        "-e",
+        "--exclusive",
+        action="store_const",
+        const="exclusive",
+        dest="timing",
+        help="rank by exclusive event time",
+    )
+    parser.set_defaults(timing="inclusive")
+    parser.add_argument("-n", "--top", type=int, default=15, help="number of rows")
+    parser.add_argument(
+        "--sort",
+        choices=("total", "avg", "avg-root-tu", "max"),
+        default="total",
+        help=(
+            "sort selected timing by total contribution, average event cost, "
+            "average per root TU, or max event cost"
+        ),
+    )
+    parser.add_argument(
+        "-o",
+        "--output-dir",
+        type=Path,
+        help="output directory (default: <trace-dir>/event_reports)",
+    )
+    parser.add_argument(
+        "--baseline-dir",
+        type=Path,
+        help=(
+            "optional baseline trace directory; writes baseline/current reports "
+            "and worse/better comparison CSVs under --output-dir"
+        ),
+    )
+    parser.add_argument(
+        "--baseline-repo-root",
+        type=Path,
+        help=(
+            "repository root that produced --baseline-dir traces (default: --repo-root)"
+        ),
+    )
+    parser.add_argument(
+        "--threshold",
+        type=float,
+        default=0.0,
+        help=(
+            "comparison-only minimum total-impact change, in seconds, "
+            "required for worse/better rows (default: 0)"
+        ),
+    )
+    parser.add_argument(
+        "--output-csv",
+        type=Path,
+        help="exact output CSV path; overrides generated file name inside --output-dir",
+    )
+    parser.add_argument(
+        "--slices",
+        type=Path,
+        help=(
+            "JSON file describing multiple report slices; writes each slice under "
+            "--output-dir/<slice-id> and emits --output-dir/summary.json"
+        ),
+    )
+    parser.add_argument(
+        "--exclusive-scope",
+        choices=("auto", "all", "same-filter"),
+        default="auto",
+        help=(
+            "exclusive timing scope; auto uses same-filter for file-processing "
+            "and all nested events for other filters"
+        ),
+    )
+    parser.add_argument(
+        "--scope-filter",
+        default=DEFAULT_SCOPE_FILTER,
+        help=(
+            "case-sensitive regex for symbol-scope reports; applies to demangled "
+            "symbols and decoded Itanium-mangled namespace prefixes for symbol-like "
+            "events only; pass an empty string to disable (default: CCCL top-level "
+            "namespaces)"
+        ),
+    )
+    parser.add_argument(
+        "--repo-root", default=Path(__file__).resolve().parents[2], type=Path
+    )
+    parser.add_argument(
+        "--tag",
+        help="optional suffix for the generated output filename",
+    )
+    parser.add_argument(
+        "--list-filters",
+        action="store_true",
+        help="print built-in filters and exit",
+    )
+    args = parser.parse_args()
+
+    if args.list_filters:
+        print_filters()
+        return
+
+    if args.trace_dir is None:
+        parser.error("trace_dir is required unless --list-filters is used")
+    if args.top <= 0:
+        parser.error("--top must be positive")
+    if args.threshold < 0:
+        parser.error("--threshold must be non-negative")
+    if args.slices is not None:
+        ignored_slice_options = (
+            (args.filter != parser.get_default("filter"), "--filter"),
+            (args.timing != parser.get_default("timing"), "--inclusive/--exclusive"),
+            (args.top != parser.get_default("top"), "--top"),
+            (args.sort != parser.get_default("sort"), "--sort"),
+            (args.threshold != parser.get_default("threshold"), "--threshold"),
+            (
+                args.exclusive_scope != parser.get_default("exclusive_scope"),
+                "--exclusive-scope",
+            ),
+            (args.scope_filter != parser.get_default("scope_filter"), "--scope-filter"),
+            (args.tag is not None, "--tag"),
+        )
+        ignored_names = [name for changed, name in ignored_slice_options if changed]
+        if ignored_names:
+            parser.error(
+                "--slices cannot be combined with single-slice option(s): "
+                + ", ".join(ignored_names)
+            )
+    if args.baseline_dir is None and args.threshold != 0:
+        parser.error("--threshold can only be used together with --baseline-dir")
+    if args.baseline_dir is not None and args.output_csv is not None:
+        parser.error("--output-csv cannot be used together with --baseline-dir")
+    if args.slices is not None and args.output_csv is not None:
+        parser.error("--output-csv cannot be used together with --slices")
+
+    trace_dir = args.trace_dir.resolve(strict=False)
+    baseline_dir = (
+        args.baseline_dir.resolve(strict=False) if args.baseline_dir else None
+    )
+    repo_root = args.repo_root.resolve(strict=False)
+    baseline_repo_root = (
+        args.baseline_repo_root.resolve(strict=False)
+        if args.baseline_repo_root
+        else repo_root
+    )
+    output_dir = (
+        args.output_dir.resolve(strict=False)
+        if args.output_dir
+        else trace_dir / "event_reports"
+    )
+    output_csv = args.output_csv.resolve(strict=False) if args.output_csv else None
+    multi_slice = args.slices is not None
+    requests = (
+        read_slice_requests(args.slices.resolve(strict=False), parser)
+        if args.slices is not None
+        else [single_slice_request(args, parser)]
+    )
+
+    manifest = {
+        "schema_version": 1,
+        "mode": "comparison" if baseline_dir is not None else "single",
+        "trace_dir": trace_dir.as_posix(),
+        "baseline_dir": baseline_dir.as_posix() if baseline_dir else None,
+        "repo_root": repo_root.as_posix(),
+        "baseline_repo_root": baseline_repo_root.as_posix(),
+        "slices": [],
+    }
+
+    for request in requests:
+        slice_output_dir = (
+            output_dir / request.config.slice_id if multi_slice else output_dir
+        )
+        manifest["slices"].append(
+            run_slice_report(
+                request,
+                trace_dir=trace_dir,
+                baseline_dir=baseline_dir,
+                repo_root=repo_root,
+                baseline_repo_root=baseline_repo_root,
+                output_dir=slice_output_dir,
+                output_csv=output_csv,
+                allow_empty=multi_slice,
+            )
+        )
+
+    summary_json = output_dir / "summary.json"
+    write_json(summary_json, manifest)
+    print(f"wrote summary manifest: {summary_json}")
+
+
+if __name__ == "__main__":
+    main()
diff --git a/cccl_upstream/ci/compile_time/summarize_tus.py b/cccl_upstream/ci/compile_time/summarize_tus.py
new file mode 100755
index 00000000..cdca232d
--- /dev/null
+++ b/cccl_upstream/ci/compile_time/summarize_tus.py
@@ -0,0 +1,147 @@
+#!/usr/bin/env python3
+
+import argparse
+import csv
+import subprocess
+from pathlib import Path
+
+GENERATED_TU_MARKER = "/headers/"
+GENERATED_TU_SOURCE_SUFFIXES = (".cu", ".cpp", ".cxx", ".cc", ".c")
+PREPROCESSED_TU_SUFFIX = ".cpp4.ii"
+PREPROCESSED_TU_SUFFIXES = (".cpp4.ii", ".ii")
+
+
+def strip_generated_tu_suffix(path_text: str) -> str:
+    for suffix in GENERATED_TU_SOURCE_SUFFIXES:
+        if path_text.endswith(suffix):
+            return path_text[: -len(suffix)]
+    return path_text
+
+
+def generated_tu_input(tu: Path) -> str:
+    parts = tu.as_posix().split(GENERATED_TU_MARKER, 1)
+    if len(parts) != 2:
+        return tu.as_posix()
+
+    rel = parts[1].split("/", 1)
+    if len(rel) != 2:
+        return tu.as_posix()
+
+    return strip_generated_tu_suffix(rel[1])
+
+
+def find_preprocessed_tus(build_dir: Path) -> list[Path]:
+    return sorted(
+        {
+            path
+            for suffix in PREPROCESSED_TU_SUFFIXES
+            for path in build_dir.glob(f"**/headers/**/*{suffix}")
+        }
+    )
+
+
+def tu_source_for_preprocessed_tu(pp_path: Path) -> Path:
+    pp_text = pp_path.as_posix()
+    for suffix in PREPROCESSED_TU_SUFFIXES:
+        if pp_text.endswith(suffix):
+            return Path(pp_text[: -len(suffix)])
+    return pp_path.with_suffix("")
+
+
+def run_cloc(preprocessed_tus: list[Path], processes: int) -> dict[str, int]:
+    if not preprocessed_tus:
+        return {}
+
+    command = [
+        "cloc",
+        "--csv",
+        "--by-file",
+        "--skip-uniqueness",
+        "--processes",
+        str(processes),
+        "--force-lang=C++,ii",
+        *[path.as_posix() for path in preprocessed_tus],
+    ]
+    result = subprocess.run(command, check=True, capture_output=True, text=True)
+
+    loc_by_file: dict[str, int] = {}
+    reader = csv.reader(result.stdout.splitlines())
+    for row in reader:
+        if len(row) < 5 or row[1] == "filename":
+            continue
+        try:
+            loc_by_file[row[1]] = int(row[4])
+        except ValueError:
+            continue
+    return loc_by_file
+
+
+def write_summary(
+    output_csv: Path,
+    preprocessed_tus: list[Path],
+    loc_by_file: dict[str, int],
+) -> None:
+    output_csv.parent.mkdir(parents=True, exist_ok=True)
+    with output_csv.open("w", newline="", encoding="utf-8") as f:
+        writer = csv.DictWriter(
+            f,
+            fieldnames=[
+                "tu_input",
+                "transitive_loc",
+                "tu_source",
+                "preprocessed_tu",
+            ],
+        )
+        writer.writeheader()
+        for pp_path in preprocessed_tus:
+            tu_path = tu_source_for_preprocessed_tu(pp_path)
+            writer.writerow(
+                {
+                    "tu_input": generated_tu_input(tu_path),
+                    "transitive_loc": loc_by_file.get(pp_path.as_posix(), 0),
+                    "tu_source": tu_path.as_posix(),
+                    "preprocessed_tu": pp_path.as_posix(),
+                }
+            )
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Summarize generated TU inputs and preprocessed LOC."
+    )
+    parser.add_argument("--build-dir", required=True, type=Path)
+    parser.add_argument("--output-csv", required=True, type=Path)
+    parser.add_argument(
+        "--cloc-processes",
+        type=int,
+        default=0,
+        help="cloc process count; 0 uses nproc --all --ignore=2 when available",
+    )
+    args = parser.parse_args()
+
+    build_dir = args.build_dir.resolve(strict=False)
+    preprocessed_tus = find_preprocessed_tus(build_dir)
+    if not preprocessed_tus:
+        raise SystemExit(f"no preprocessed generated TUs found under {build_dir}")
+
+    processes = args.cloc_processes
+    if processes <= 0:
+        try:
+            processes = int(
+                subprocess.check_output(
+                    ["nproc", "--all", "--ignore=2"], text=True
+                ).strip()
+            )
+        except (subprocess.SubprocessError, ValueError):
+            processes = 1
+
+    write_summary(
+        output_csv=args.output_csv,
+        preprocessed_tus=preprocessed_tus,
+        loc_by_file=run_cloc(preprocessed_tus, processes),
+    )
+    print(f"wrote {len(preprocessed_tus)} generated TU row(s) to {args.output_csv}")
+
+
+if __name__ == "__main__":
+    main()
diff --git a/cccl_upstream/ci/compile_time/test_summarize_events.py b/cccl_upstream/ci/compile_time/test_summarize_events.py
new file mode 100644
index 00000000..a6093a83
--- /dev/null
+++ b/cccl_upstream/ci/compile_time/test_summarize_events.py
@@ -0,0 +1,1731 @@
+#!/usr/bin/env python3
+
+import csv
+import json
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+from ci.compile_time import summarize_tus
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+SUMMARY_SCRIPT = REPO_ROOT / "ci" / "compile_time" / "summarize_events.py"
+PREPARE_SCRIPT = REPO_ROOT / "ci" / "compile_time" / "prepare_traces.py"
+PARSE_MATRIX_SCRIPT = REPO_ROOT / "ci" / "compile_time" / "parse_matrix.py"
+RENDER_COMMENT_SCRIPT = REPO_ROOT / "ci" / "compile_time" / "render_pr_comment.py"
+WRAPPER_SCRIPT = REPO_ROOT / "ci" / "build_compile_time_bench.sh"
+PULL_REQUEST_WORKFLOW = (
+    REPO_ROOT / ".github" / "workflows" / "ci-workflow-pull-request.yml"
+)
+
+
+def csv_rows(path: Path) -> list[dict[str, str]]:
+    with path.open(newline="", encoding="utf-8") as f:
+        return list(csv.DictReader(f))
+
+
+class TraceBuilder:
+    def __init__(self, root: Path):
+        self.root = root
+
+    def project_detail(self, rel: str) -> str:
+        return (self.root / rel).as_posix()
+
+    def event(
+        self,
+        name: str,
+        detail: str,
+        ts: int,
+        dur: int,
+        *,
+        pid: int = 1,
+        tid: int = 1,
+    ) -> dict:
+        return {
+            "ph": "X",
+            "pid": pid,
+            "tid": tid,
+            "name": name,
+            "ts": ts,
+            "dur": dur,
+            "args": {"detail": detail},
+        }
+
+    def write_trace(self, path: Path, events: list[dict], input_name: str) -> None:
+        path.parent.mkdir(parents=True, exist_ok=True)
+        path.write_text(
+            json.dumps(
+                {
+                    "traceEvents": events,
+                    "otherData": {
+                        "inputFiles": [f"/generated/headers/target/{input_name}.cu"]
+                    },
+                }
+            ),
+            encoding="utf-8",
+        )
+
+
+class SummarizeEventsBaselineCompareTest(unittest.TestCase):
+    def setUp(self) -> None:
+        self.tempdir = tempfile.TemporaryDirectory()
+        self.work = Path(self.tempdir.name)
+        self.traces = TraceBuilder(REPO_ROOT)
+        self.wrapper_infix = f"compile-time-test-{self.work.name}"
+        self.wrapper_preset = "all-dev"
+        self.wrapper_build_dir = (
+            REPO_ROOT / "build" / self.wrapper_infix / self.wrapper_preset
+        )
+
+    def tearDown(self) -> None:
+        shutil.rmtree(REPO_ROOT / "build" / self.wrapper_infix, ignore_errors=True)
+        self.tempdir.cleanup()
+
+    def comparison_csv(self, output_dir: Path, name: str) -> Path:
+        return output_dir / "comparison" / name
+
+    def wrapper_trace_dir(self) -> Path:
+        return self.wrapper_build_dir / "compile_time" / "raw_traces"
+
+    def wrapper_output_dir(self) -> Path:
+        return self.wrapper_build_dir / "compile_time" / "event_reports"
+
+    def assert_empty_csv(self, path: Path) -> None:
+        self.assertTrue(path.exists())
+        self.assertEqual(csv_rows(path), [])
+
+    def run_summary(
+        self,
+        current: Path,
+        baseline: Path,
+        output_dir: Path,
+        *args: str,
+        baseline_repo_root: Path | None = None,
+    ) -> subprocess.CompletedProcess[str]:
+        command = [
+            sys.executable,
+            SUMMARY_SCRIPT.as_posix(),
+            current.as_posix(),
+            "--baseline-dir",
+            baseline.as_posix(),
+            "-o",
+            output_dir.as_posix(),
+        ]
+        if baseline_repo_root is not None:
+            command.extend(["--baseline-repo-root", baseline_repo_root.as_posix()])
+        command.extend(args)
+
+        return subprocess.run(
+            command,
+            cwd=REPO_ROOT,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+    def run_wrapper(
+        self,
+        *args: str,
+        no_tu_csv: bool = True,
+        common_args: tuple[str, ...] = (),
+    ) -> subprocess.CompletedProcess[str]:
+        command = [
+            "bash",
+            WRAPPER_SCRIPT.as_posix(),
+            "-skip-configure",
+            "-skip-build",
+            "-no-prepare-perfetto",
+            *common_args,
+        ]
+        if no_tu_csv:
+            command.append("-no-tu-csv")
+        command.extend(["--", *args])
+        env = os.environ.copy()
+        env["CCCL_BUILD_INFIX"] = self.wrapper_infix
+        return subprocess.run(
+            command,
+            cwd=REPO_ROOT,
+            env=env,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+    def test_comparison_outputs_and_unmatched_children(self) -> None:
+        baseline = self.work / "baseline"
+        current = self.work / "current"
+        output = self.work / "reports"
+
+        parent = self.traces.project_detail("libcudacxx/include/cuda/std/parent.h")
+        matched_child = self.traces.project_detail(
+            "libcudacxx/include/cuda/std/matched_child.h"
+        )
+        better = self.traces.project_detail("libcudacxx/include/cuda/std/better.h")
+        baseline_only_child = self.traces.project_detail(
+            "libcudacxx/include/cuda/std/baseline_only_child.h"
+        )
+        baseline_only = self.traces.project_detail(
+            "libcudacxx/include/cuda/std/only_baseline.h"
+        )
+        current_only = self.traces.project_detail(
+            "libcudacxx/include/cuda/std/only_current.h"
+        )
+
+        self.traces.write_trace(
+            baseline / "target" / "match.json",
+            [
+                self.traces.event("Parent", parent, 0, 100),
+                self.traces.event("Child", matched_child, 10, 30),
+                self.traces.event("Child", baseline_only_child, 50, 20),
+                self.traces.event("Better", better, 200, 70),
+                self.traces.event("OnlyBaseline", baseline_only, 300, 20),
+            ],
+            "match",
+        )
+        self.traces.write_trace(
+            current / "target" / "match.json",
+            [
+                self.traces.event("Parent", parent, 0, 150),
+                self.traces.event("Child", matched_child, 10, 40),
+                self.traces.event("Better", better, 200, 40),
+                self.traces.event("OnlyCurrent", current_only, 300, 90),
+            ],
+            "match",
+        )
+        self.traces.write_trace(
+            baseline / "target" / "baseline_only_trace.json",
+            [self.traces.event("Parent", parent, 0, 999)],
+            "baseline_only_trace",
+        )
+        self.traces.write_trace(
+            current / "target" / "current_only_trace.json",
+            [self.traces.event("Parent", parent, 0, 999)],
+            "current_only_trace",
+        )
+
+        self.run_summary(
+            current,
+            baseline,
+            output,
+            "-f",
+            "all",
+            "-e",
+            "--sort",
+            "total",
+            "-n",
+            "10",
+            "--tag",
+            "synthetic",
+        )
+
+        worse = csv_rows(
+            self.comparison_csv(
+                output, "top-10-all-exclusive-all-by-total-worse-synthetic.csv"
+            )
+        )
+        better_rows = csv_rows(
+            self.comparison_csv(
+                output, "top-10-all-exclusive-all-by-total-better-synthetic.csv"
+            )
+        )
+
+        parent_row = next(row for row in worse if row["event_name"] == "Parent")
+        self.assertEqual(parent_row["baseline_selected_s"], "0.000070")
+        self.assertEqual(parent_row["current_selected_s"], "0.000110")
+        self.assertEqual(parent_row["impact_magnitude_s"], "0.000040")
+
+        better_row = next(row for row in better_rows if row["event_name"] == "Better")
+        self.assertEqual(better_row["baseline_selected_s"], "0.000070")
+        self.assertEqual(better_row["current_selected_s"], "0.000040")
+        self.assertEqual(better_row["impact_magnitude_s"], "0.000030")
+
+        all_comparison_keys = {row["event_key"] for row in worse + better_rows}
+        self.assertFalse(any("only_baseline" in key for key in all_comparison_keys))
+        self.assertFalse(any("only_current" in key for key in all_comparison_keys))
+
+    def test_comparison_threshold_filters_small_changes(self) -> None:
+        baseline = self.work / "baseline"
+        current = self.work / "current"
+        output = self.work / "reports"
+
+        small = self.traces.project_detail("libcudacxx/include/cuda/std/small.h")
+        large = self.traces.project_detail("libcudacxx/include/cuda/std/large.h")
+        self.traces.write_trace(
+            baseline / "target" / "match.json",
+            [
+                self.traces.event("Same", small, 0, 10),
+                self.traces.event("Same", large, 100, 10),
+            ],
+            "match",
+        )
+        self.traces.write_trace(
+            current / "target" / "match.json",
+            [
+                self.traces.event("Same", small, 0, 12),
+                self.traces.event("Same", large, 100, 20),
+            ],
+            "match",
+        )
+
+        self.run_summary(
+            current,
+            baseline,
+            output,
+            "-f",
+            "all",
+            "-i",
+            "--sort",
+            "total",
+            "-n",
+            "10",
+            "--threshold",
+            "0.000005",
+            "--tag",
+            "threshold",
+        )
+
+        worse = csv_rows(
+            self.comparison_csv(
+                output, "top-10-all-inclusive-by-total-worse-threshold.csv"
+            )
+        )
+        self.assertEqual(len(worse), 1)
+        self.assertTrue(worse[0]["event_key"].endswith("large.h"))
+        self.assertEqual(worse[0]["impact_magnitude_s"], "0.000010")
+
+    def test_comparison_ranks_by_total_impact_not_selected_metric(self) -> None:
+        baseline = self.work / "baseline"
+        current = self.work / "current"
+        output = self.work / "reports"
+
+        repeated = self.traces.project_detail("libcudacxx/include/cuda/std/repeated.h")
+        one_trace = self.traces.project_detail(
+            "libcudacxx/include/cuda/std/one_trace.h"
+        )
+        self.traces.write_trace(
+            baseline / "target" / "first.json",
+            [
+                self.traces.event("Same", repeated, 0, 10),
+                self.traces.event("Same", one_trace, 100, 10),
+            ],
+            "first",
+        )
+        self.traces.write_trace(
+            current / "target" / "first.json",
+            [
+                self.traces.event("Same", repeated, 0, 16),
+                self.traces.event("Same", one_trace, 100, 20),
+            ],
+            "first",
+        )
+        self.traces.write_trace(
+            baseline / "target" / "second.json",
+            [self.traces.event("Same", repeated, 0, 10)],
+            "second",
+        )
+        self.traces.write_trace(
+            current / "target" / "second.json",
+            [self.traces.event("Same", repeated, 0, 16)],
+            "second",
+        )
+
+        self.run_summary(
+            current,
+            baseline,
+            output,
+            "-f",
+            "all",
+            "-i",
+            "--sort",
+            "max",
+            "-n",
+            "10",
+            "--tag",
+            "impact",
+        )
+
+        worse = csv_rows(
+            self.comparison_csv(output, "top-10-all-inclusive-by-max-worse-impact.csv")
+        )
+        self.assertTrue(worse[0]["event_key"].endswith("repeated.h"))
+        self.assertEqual(worse[0]["impact_magnitude_s"], "0.000012")
+        self.assertEqual(worse[0]["selected_magnitude_s"], "0.000006")
+
+    def test_threshold_requires_comparison_mode(self) -> None:
+        traces = self.work / "traces"
+        output = self.work / "reports"
+        same = self.traces.project_detail("libcudacxx/include/cuda/std/same.h")
+        self.traces.write_trace(
+            traces / "target" / "same.json",
+            [self.traces.event("Same", same, 0, 10)],
+            "same",
+        )
+
+        completed = subprocess.run(
+            [
+                sys.executable,
+                SUMMARY_SCRIPT.as_posix(),
+                traces.as_posix(),
+                "-o",
+                output.as_posix(),
+                "--threshold",
+                "0.000001",
+            ],
+            cwd=REPO_ROOT,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        self.assertNotEqual(completed.returncode, 0)
+        self.assertIn(
+            "--threshold can only be used together with --baseline-dir",
+            completed.stderr,
+        )
+
+    def test_file_processing_same_filter_keeps_unmatched_child_in_parent_cost(
+        self,
+    ) -> None:
+        baseline = self.work / "baseline"
+        current = self.work / "current"
+        output = self.work / "reports"
+
+        parent = self.traces.project_detail("libcudacxx/include/cuda/std/parent.h")
+        matched_child = self.traces.project_detail(
+            "libcudacxx/include/cuda/std/matched_child.h"
+        )
+        unmatched_child = self.traces.project_detail(
+            "libcudacxx/include/cuda/std/baseline_only_child.h"
+        )
+
+        self.traces.write_trace(
+            baseline / "target" / "same.json",
+            [
+                self.traces.event("Processing Header File", parent, 0, 100),
+                self.traces.event("Processing Header File", matched_child, 10, 30),
+                self.traces.event("Processing Header File", unmatched_child, 50, 20),
+            ],
+            "same",
+        )
+        self.traces.write_trace(
+            current / "target" / "same.json",
+            [
+                self.traces.event("Processing Header File", parent, 0, 130),
+                self.traces.event("Processing Header File", matched_child, 10, 40),
+            ],
+            "same",
+        )
+
+        self.run_summary(
+            current,
+            baseline,
+            output,
+            "-f",
+            "file-processing",
+            "-e",
+            "--sort",
+            "total",
+            "-n",
+            "5",
+            "--tag",
+            "file-processing",
+        )
+        worse = csv_rows(
+            self.comparison_csv(
+                output,
+                "top-5-file-processing-exclusive-same-filter-by-total-worse-file-processing.csv",
+            )
+        )
+        parent_row = next(row for row in worse if row["event_key"].endswith("parent.h"))
+        self.assertEqual(parent_row["baseline_selected_s"], "0.000070")
+        self.assertEqual(parent_row["current_selected_s"], "0.000090")
+
+    def test_same_filter_exclusive_uses_direct_children_only(self) -> None:
+        traces = self.work / "traces"
+        output = self.work / "reports"
+
+        parent = self.traces.project_detail("libcudacxx/include/cuda/std/parent.h")
+        nested = self.traces.project_detail("libcudacxx/include/cuda/std/nested.h")
+
+        self.traces.write_trace(
+            traces / "target" / "direct.json",
+            [
+                self.traces.event("Processing Header File", parent, 0, 100),
+                self.traces.event("Scanning Function Body", "not-a-header", 10, 80),
+                self.traces.event("Processing Header File", nested, 20, 10),
+            ],
+            "direct",
+        )
+
+        subprocess.run(
+            [
+                sys.executable,
+                SUMMARY_SCRIPT.as_posix(),
+                traces.as_posix(),
+                "-o",
+                output.as_posix(),
+                "-f",
+                "file-processing",
+                "-e",
+                "--sort",
+                "total",
+                "-n",
+                "5",
+                "--tag",
+                "direct",
+            ],
+            cwd=REPO_ROOT,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        rows = csv_rows(
+            output / "top-5-file-processing-exclusive-same-filter-by-total-direct.csv"
+        )
+        parent_row = next(row for row in rows if row["event_key"].endswith("parent.h"))
+        self.assertEqual(parent_row["selected_total_s"], "0.000100")
+
+    def test_empty_comparisons_still_write_csvs(self) -> None:
+        baseline = self.work / "baseline"
+        current = self.work / "current"
+        output = self.work / "reports"
+
+        foo = self.traces.project_detail("libcudacxx/include/cuda/std/foo.h")
+        bar = self.traces.project_detail("libcudacxx/include/cuda/std/bar.h")
+        self.traces.write_trace(
+            baseline / "target" / "same.json",
+            [self.traces.event("Foo", foo, 0, 10)],
+            "same",
+        )
+        self.traces.write_trace(
+            current / "target" / "same.json",
+            [self.traces.event("Bar", bar, 0, 20)],
+            "same",
+        )
+        self.run_summary(
+            current,
+            baseline,
+            output,
+            "-f",
+            "all",
+            "-i",
+            "--sort",
+            "total",
+            "-n",
+            "5",
+            "--tag",
+            "empty",
+        )
+
+        self.assert_empty_csv(
+            self.comparison_csv(output, "top-5-all-inclusive-by-total-worse-empty.csv")
+        )
+        self.assert_empty_csv(
+            self.comparison_csv(output, "top-5-all-inclusive-by-total-better-empty.csv")
+        )
+        with (output / "summary.json").open(encoding="utf-8") as f:
+            manifest = json.load(f)
+        self.assertIn(
+            "no comparable event keys",
+            " ".join(manifest["slices"][0]["warnings"]),
+        )
+
+    def test_wrapper_forwarding_writes_current_report(self) -> None:
+        current = self.wrapper_trace_dir()
+        output = self.wrapper_output_dir()
+
+        same = self.traces.project_detail("libcudacxx/include/cuda/std/same.h")
+        self.traces.write_trace(
+            current / "target" / "current_name.json",
+            [self.traces.event("Same", same, 0, 100)],
+            "current_name",
+        )
+
+        self.run_wrapper(
+            "-f",
+            "all",
+            "-i",
+            "--sort",
+            "total",
+            "-n",
+            "5",
+            "--tag",
+            "wrapper",
+            common_args=("-arch", "70"),
+        )
+
+        self.assertTrue((output / "top-5-all-inclusive-by-total-wrapper.csv").exists())
+
+    def test_comparison_matches_paths_from_distinct_repo_roots(self) -> None:
+        baseline_repo = self.work / "baseline-src"
+        baseline = self.work / "baseline"
+        current = self.work / "current"
+        output = self.work / "reports"
+
+        baseline_traces = TraceBuilder(baseline_repo)
+        baseline_same = baseline_traces.project_detail(
+            "libcudacxx/include/cuda/std/same.h"
+        )
+        current_same = self.traces.project_detail("libcudacxx/include/cuda/std/same.h")
+        self.traces.write_trace(
+            current / "target" / "same.json",
+            [self.traces.event("Same", current_same, 0, 12)],
+            "same",
+        )
+        baseline_traces.write_trace(
+            baseline / "target" / "same.json",
+            [baseline_traces.event("Same", baseline_same, 0, 10)],
+            "same",
+        )
+
+        self.run_summary(
+            current,
+            baseline,
+            output,
+            "-f",
+            "all",
+            "-i",
+            "--sort",
+            "total",
+            "-n",
+            "5",
+            baseline_repo_root=baseline_repo,
+        )
+
+        worse = csv_rows(
+            self.comparison_csv(output, "top-5-all-inclusive-by-total-worse.csv")
+        )
+        self.assertEqual(worse[0]["event_key"], "libcudacxx/include/cuda/std/same.h")
+        self.assertEqual(worse[0]["impact_magnitude_s"], "0.000002")
+
+    def test_wrapper_uses_computed_event_output_dir(self) -> None:
+        current = self.wrapper_trace_dir()
+        wrapper_default = self.wrapper_output_dir()
+        explicit_output = self.work / "explicit-output"
+
+        same = self.traces.project_detail("libcudacxx/include/cuda/std/same.h")
+        self.traces.write_trace(
+            current / "target" / "same.json",
+            [self.traces.event("Same", same, 0, 12)],
+            "same",
+        )
+
+        self.run_wrapper(
+            f"--output-dir={explicit_output.as_posix()}",
+            "-f",
+            "all",
+            "-i",
+            "--sort",
+            "total",
+            "-n",
+            "5",
+            "--tag",
+            "equals-output",
+        )
+
+        self.assertTrue(
+            (
+                wrapper_default / "top-5-all-inclusive-by-total-equals-output.csv"
+            ).exists()
+        )
+        self.assertFalse(
+            (
+                explicit_output / "top-5-all-inclusive-by-total-equals-output.csv"
+            ).exists()
+        )
+
+    def test_sort_by_average_per_root_tu(self) -> None:
+        traces = self.work / "traces"
+        output = self.work / "reports"
+
+        one_tu = self.traces.project_detail("libcudacxx/include/cuda/std/one_tu.h")
+        two_tus = self.traces.project_detail("libcudacxx/include/cuda/std/two_tus.h")
+        self.traces.write_trace(
+            traces / "target" / "first.json",
+            [
+                self.traces.event("Same", one_tu, 0, 80),
+                self.traces.event("Same", two_tus, 100, 60),
+            ],
+            "first",
+        )
+        self.traces.write_trace(
+            traces / "target" / "second.json",
+            [self.traces.event("Same", two_tus, 0, 40)],
+            "second",
+        )
+
+        subprocess.run(
+            [
+                sys.executable,
+                SUMMARY_SCRIPT.as_posix(),
+                traces.as_posix(),
+                "-o",
+                output.as_posix(),
+                "-f",
+                "all",
+                "-i",
+                "--sort",
+                "avg-root-tu",
+                "-n",
+                "2",
+                "--tag",
+                "avg-root-tu",
+            ],
+            cwd=REPO_ROOT,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        rows = csv_rows(output / "top-2-all-inclusive-by-avg-root-tu-avg-root-tu.csv")
+        self.assertTrue(rows[0]["event_key"].endswith("one_tu.h"))
+        self.assertEqual(rows[0]["selected_avg_per_root_tu_s"], "0.000080")
+        self.assertTrue(rows[1]["event_key"].endswith("two_tus.h"))
+        self.assertEqual(rows[1]["selected_avg_per_root_tu_s"], "0.000050")
+
+    def test_host_compiler_filter_matches_host_phase_events(self) -> None:
+        traces = self.work / "traces"
+        output = self.work / "reports"
+
+        self.traces.write_trace(
+            traces / "target" / "host.json",
+            [
+                self.traces.event(
+                    "g++ (preprocessing 1)", "g++ (preprocessing 1)", 0, 10
+                ),
+                self.traces.event("gcc (compiling)", "gcc (compiling)", 20, 30),
+                self.traces.event("CUDA C++ Front-End", "frontend", 60, 40),
+            ],
+            "host",
+        )
+
+        subprocess.run(
+            [
+                sys.executable,
+                SUMMARY_SCRIPT.as_posix(),
+                traces.as_posix(),
+                "-o",
+                output.as_posix(),
+                "-f",
+                "host-compiler",
+                "-i",
+                "--sort",
+                "total",
+                "-n",
+                "5",
+                "--tag",
+                "host",
+            ],
+            cwd=REPO_ROOT,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        rows = csv_rows(output / "top-5-host-compiler-inclusive-by-total-host.csv")
+        self.assertEqual(
+            {row["event_name"] for row in rows},
+            {"g++ (preprocessing 1)", "gcc (compiling)"},
+        )
+
+    def test_scope_filter_defaults_to_cccl_demangled_symbols(self) -> None:
+        traces = self.work / "traces"
+        output = self.work / "reports"
+
+        self.traces.write_trace(
+            traces / "target" / "symbols.json",
+            [
+                self.traces.event(
+                    "Scanning Function Body",
+                    "std::basic_string_view::size() const noexcept",
+                    0,
+                    100,
+                ),
+                self.traces.event(
+                    "Scanning Function Body",
+                    "cuda::std::__4::basic_string_view::size() const noexcept",
+                    200,
+                    80,
+                ),
+                self.traces.event(
+                    "Scanning Function Body",
+                    'cuda::std::literals::operator ""sv(const char *, unsigned long)',
+                    400,
+                    60,
+                ),
+            ],
+            "symbols",
+        )
+
+        subprocess.run(
+            [
+                sys.executable,
+                SUMMARY_SCRIPT.as_posix(),
+                traces.as_posix(),
+                "-o",
+                output.as_posix(),
+                "-f",
+                "scanning-function-body",
+                "-i",
+                "--sort",
+                "total",
+                "-n",
+                "5",
+                "--tag",
+                "scope-default",
+            ],
+            cwd=REPO_ROOT,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        rows = csv_rows(
+            output / "top-5-scanning-function-body-inclusive-by-total-scope-default.csv"
+        )
+        self.assertEqual(len(rows), 2)
+        self.assertIn("cuda::std", rows[0]["event_key"])
+        self.assertIn("operator", rows[1]["event_key"])
+
+    def test_empty_scope_filter_disables_symbol_scope_filtering(self) -> None:
+        traces = self.work / "traces"
+        output = self.work / "reports"
+
+        self.traces.write_trace(
+            traces / "target" / "symbols.json",
+            [
+                self.traces.event(
+                    "Scanning Function Body",
+                    "std::basic_string_view::size() const noexcept",
+                    0,
+                    100,
+                ),
+                self.traces.event(
+                    "Scanning Function Body",
+                    "cuda::std::__4::basic_string_view::size() const noexcept",
+                    200,
+                    80,
+                ),
+            ],
+            "symbols",
+        )
+
+        subprocess.run(
+            [
+                sys.executable,
+                SUMMARY_SCRIPT.as_posix(),
+                traces.as_posix(),
+                "-o",
+                output.as_posix(),
+                "-f",
+                "scanning-function-body",
+                "-i",
+                "--sort",
+                "total",
+                "-n",
+                "5",
+                "--scope-filter",
+                "",
+                "--tag",
+                "scope-disabled",
+            ],
+            cwd=REPO_ROOT,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        rows = csv_rows(
+            output
+            / "top-5-scanning-function-body-inclusive-by-total-scope-disabled.csv"
+        )
+        self.assertEqual(len(rows), 2)
+        self.assertIn("std::basic_string_view", rows[0]["event_key"])
+        self.assertIn("cuda::std", rows[1]["event_key"])
+
+    def test_scope_filter_uses_reported_template_symbol_scope(self) -> None:
+        traces = self.work / "traces"
+        output = self.work / "reports"
+
+        self.traces.write_trace(
+            traces / "target" / "templates.json",
+            [
+                self.traces.event(
+                    "Instantiating Template Function",
+                    (
+                        "nvtx3::v1::domain::get "
+                        "[nvtx3::v1::domain::get<cuda::__4::__nvtx_cccl_domain>()]"
+                    ),
+                    0,
+                    100,
+                ),
+                self.traces.event(
+                    "Instantiating Template Function",
+                    "cub::detail::load [cub::detail::load<int>()]",
+                    200,
+                    80,
+                ),
+            ],
+            "templates",
+        )
+
+        subprocess.run(
+            [
+                sys.executable,
+                SUMMARY_SCRIPT.as_posix(),
+                traces.as_posix(),
+                "-o",
+                output.as_posix(),
+                "-f",
+                "template-instantiation",
+                "-i",
+                "--sort",
+                "total",
+                "-n",
+                "5",
+                "--tag",
+                "template-scope",
+            ],
+            cwd=REPO_ROOT,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        rows = csv_rows(
+            output
+            / "top-5-template-instantiation-inclusive-by-total-template-scope.csv"
+        )
+        self.assertEqual(len(rows), 1)
+        self.assertIn("cub::detail::load", rows[0]["event_key"])
+
+    def test_scope_filter_matches_mangled_cccl_namespaces(self) -> None:
+        traces = self.work / "traces"
+        output = self.work / "reports"
+
+        self.traces.write_trace(
+            traces / "target" / "mangled.json",
+            [
+                self.traces.event(
+                    "Generating Function IR",
+                    "_ZN4cuda3std3__43fooEv",
+                    0,
+                    100,
+                ),
+                self.traces.event(
+                    "Generating Function IR",
+                    "_ZN6thrust6detail3barEv",
+                    200,
+                    80,
+                ),
+                self.traces.event(
+                    "Generating Function IR",
+                    "_ZNSt6vectorIiE4sizeEv",
+                    400,
+                    200,
+                ),
+            ],
+            "mangled",
+        )
+
+        subprocess.run(
+            [
+                sys.executable,
+                SUMMARY_SCRIPT.as_posix(),
+                traces.as_posix(),
+                "-o",
+                output.as_posix(),
+                "-f",
+                "code-generation",
+                "-i",
+                "--sort",
+                "total",
+                "-n",
+                "5",
+                "--tag",
+                "mangled-scope",
+            ],
+            cwd=REPO_ROOT,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        rows = csv_rows(
+            output / "top-5-code-generation-inclusive-by-total-mangled-scope.csv"
+        )
+        self.assertEqual(
+            {row["event_key"] for row in rows},
+            {
+                "_ZN4cuda3std3__43fooEv",
+                "_ZN6thrust6detail3barEv",
+            },
+        )
+
+    def test_total_compilation_filter_uses_trace_wall_span(self) -> None:
+        traces = self.work / "traces"
+        output = self.work / "reports"
+
+        self.traces.write_trace(
+            traces / "target" / "total.json",
+            [
+                self.traces.event(
+                    "g++ (preprocessing 1)", "g++ (preprocessing 1)", 10, 20
+                ),
+                self.traces.event("CUDA C++ Front-End", "frontend", 50, 40),
+            ],
+            "total",
+        )
+
+        subprocess.run(
+            [
+                sys.executable,
+                SUMMARY_SCRIPT.as_posix(),
+                traces.as_posix(),
+                "-o",
+                output.as_posix(),
+                "-f",
+                "total-compilation",
+                "-i",
+                "--sort",
+                "total",
+                "-n",
+                "5",
+                "--tag",
+                "total",
+            ],
+            cwd=REPO_ROOT,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+        total_rows = csv_rows(
+            output / "top-5-total-compilation-inclusive-by-total-total.csv"
+        )
+        self.assertEqual(len(total_rows), 1)
+        self.assertEqual(total_rows[0]["event_name"], "Total Compilation Time")
+        self.assertEqual(total_rows[0]["selected_total_s"], "0.000080")
+
+        subprocess.run(
+            [
+                sys.executable,
+                SUMMARY_SCRIPT.as_posix(),
+                traces.as_posix(),
+                "-o",
+                output.as_posix(),
+                "-f",
+                "all",
+                "-i",
+                "--sort",
+                "total",
+                "-n",
+                "5",
+                "--tag",
+                "all-no-total",
+            ],
+            cwd=REPO_ROOT,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+        all_rows = csv_rows(output / "top-5-all-inclusive-by-total-all-no-total.csv")
+        self.assertFalse(
+            any(row["event_name"] == "Total Compilation Time" for row in all_rows)
+        )
+
+    def test_same_dir_comparison_is_empty(self) -> None:
+        traces = self.work / "traces"
+        output = self.work / "reports"
+        same = self.traces.project_detail("libcudacxx/include/cuda/std/same.h")
+        self.traces.write_trace(
+            traces / "target" / "same.json",
+            [self.traces.event("Same", same, 0, 10)],
+            "same",
+        )
+
+        self.run_summary(
+            traces,
+            traces,
+            output,
+            "-f",
+            "all",
+            "-i",
+            "--sort",
+            "max",
+            "-n",
+            "5",
+            "--tag",
+            "same",
+        )
+        self.assert_empty_csv(
+            self.comparison_csv(output, "top-5-all-inclusive-by-max-worse-same.csv")
+        )
+        self.assert_empty_csv(
+            self.comparison_csv(output, "top-5-all-inclusive-by-max-better-same.csv")
+        )
+
+    def test_multi_slice_writes_manifest_and_allows_empty_slice(self) -> None:
+        traces = self.work / "traces"
+        output = self.work / "reports"
+        slices = self.work / "slices.json"
+        same = self.traces.project_detail("libcudacxx/include/cuda/std/same.h")
+        self.traces.write_trace(
+            traces / "target" / "same.json",
+            [self.traces.event("Same", same, 0, 10)],
+            "same",
+        )
+        slices.write_text(
+            json.dumps(
+                {
+                    "slices": [
+                        {
+                            "id": "all-events",
+                            "title": "All events",
+                            "filter": "all",
+                            "timing": "inclusive",
+                            "sort": "total",
+                            "top": 5,
+                            "threshold": 0,
+                        },
+                        {
+                            "id": "empty-events",
+                            "title": "Empty events",
+                            "filter": "does-not-match",
+                            "timing": "inclusive",
+                            "sort": "total",
+                            "top": 5,
+                            "threshold": 0,
+                        },
+                    ]
+                }
+            ),
+            encoding="utf-8",
+        )
+
+        subprocess.run(
+            [
+                sys.executable,
+                SUMMARY_SCRIPT.as_posix(),
+                traces.as_posix(),
+                "-o",
+                output.as_posix(),
+                "--slices",
+                slices.as_posix(),
+            ],
+            cwd=REPO_ROOT,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        with (output / "summary.json").open(encoding="utf-8") as f:
+            manifest = json.load(f)
+
+        self.assertEqual(
+            [item["id"] for item in manifest["slices"]], ["all-events", "empty-events"]
+        )
+        self.assertEqual(manifest["slices"][0]["reports"]["current"]["row_count"], 1)
+        self.assertEqual(manifest["slices"][1]["reports"]["current"]["row_count"], 0)
+        self.assertTrue(
+            (
+                output
+                / "empty-events"
+                / "top-5-regex-does-not-match-inclusive-by-total.csv"
+            ).exists()
+        )
+
+
+class CompileTimeMatrixAndCommentTest(unittest.TestCase):
+    def setUp(self) -> None:
+        self.tempdir = tempfile.TemporaryDirectory()
+        self.work = Path(self.tempdir.name)
+
+    def tearDown(self) -> None:
+        self.tempdir.cleanup()
+
+    def test_parse_matrix_disabled_when_section_missing(self) -> None:
+        matrix = self.work / "matrix.yaml"
+        matrix.write_text("workflows: {}\n", encoding="utf-8")
+
+        completed = subprocess.run(
+            [
+                sys.executable,
+                PARSE_MATRIX_SCRIPT.as_posix(),
+                matrix.as_posix(),
+            ],
+            cwd=REPO_ROOT,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        self.assertEqual(json.loads(completed.stdout), {"include": []})
+
+    def test_parse_matrix_valid_config(self) -> None:
+        matrix = self.work / "matrix.yaml"
+        matrix.write_text(
+            """
+compile_time:
+  pull_request:
+    - id: public-headers
+      name: Public headers
+      gpu: rtx2080
+      launch_args: "--cuda 13.3 --host gcc13"
+      baseline_ref: origin/main
+      preset: all-dev
+      targets: [cub.headers.base]
+      args: "-arch native"
+      slices:
+        - id: total-compilation
+          title: TU total compilation
+          filter: total-compilation
+          timing: inclusive
+          sort: total
+          top: 15
+          threshold: 0.001
+""",
+            encoding="utf-8",
+        )
+
+        completed = subprocess.run(
+            [
+                sys.executable,
+                PARSE_MATRIX_SCRIPT.as_posix(),
+                matrix.as_posix(),
+            ],
+            cwd=REPO_ROOT,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        include = json.loads(completed.stdout)["include"]
+        self.assertEqual(len(include), 1)
+        self.assertEqual(
+            include[0]["comment_header"], "compile-time-bench-public-headers"
+        )
+        self.assertEqual(json.loads(include[0]["targets_json"]), ["cub.headers.base"])
+        self.assertEqual(
+            json.loads(include[0]["slices_json"])["slices"][0]["id"],
+            "total-compilation",
+        )
+
+    def test_parse_matrix_rejects_duplicate_slice_ids(self) -> None:
+        matrix = self.work / "matrix.yaml"
+        matrix.write_text(
+            """
+compile_time:
+  pull_request:
+    - id: public-headers
+      name: Public headers
+      gpu: rtx2080
+      launch_args: "--cuda 13.3 --host gcc13"
+      baseline_ref: origin/main
+      preset: all-dev
+      targets: [cub.headers.base]
+      slices:
+        - id: repeated
+          title: First
+          filter: all
+          timing: inclusive
+          sort: total
+          top: 15
+          threshold: 0
+        - id: repeated
+          title: Second
+          filter: all
+          timing: inclusive
+          sort: total
+          top: 15
+          threshold: 0
+""",
+            encoding="utf-8",
+        )
+
+        completed = subprocess.run(
+            [
+                sys.executable,
+                PARSE_MATRIX_SCRIPT.as_posix(),
+                matrix.as_posix(),
+            ],
+            cwd=REPO_ROOT,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        self.assertNotEqual(completed.returncode, 0)
+        self.assertIn("duplicate slice id", completed.stderr)
+
+    def test_parse_matrix_rejects_bool_numeric_fields(self) -> None:
+        matrix = self.work / "matrix.yaml"
+        matrix.write_text(
+            """
+compile_time:
+  pull_request:
+    - id: public-headers
+      name: Public headers
+      gpu: rtx2080
+      launch_args: "--cuda 13.3 --host gcc13"
+      baseline_ref: origin/main
+      preset: all-dev
+      targets: [cub.headers.base]
+      artifact_retention_days: true
+      slices:
+        - id: total-compilation
+          title: TU total compilation
+          filter: total-compilation
+          timing: inclusive
+          sort: total
+          top: true
+          threshold: false
+""",
+            encoding="utf-8",
+        )
+
+        completed = subprocess.run(
+            [
+                sys.executable,
+                PARSE_MATRIX_SCRIPT.as_posix(),
+                matrix.as_posix(),
+            ],
+            cwd=REPO_ROOT,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        self.assertNotEqual(completed.returncode, 0)
+        self.assertIn("positive integer", completed.stderr)
+
+    def test_pull_request_workflow_supports_compile_time_bench_skip_tag(self) -> None:
+        workflow = PULL_REQUEST_WORKFLOW.read_text(encoding="utf-8")
+
+        self.assertIn("[skip-compile-time-bench]", workflow)
+        self.assertIn("compile_time_enabled=false", workflow)
+        self.assertIn('compile_time_matrix={"include":[]}', workflow)
+
+    def test_render_comment_omits_empty_sections_and_splits_directions(self) -> None:
+        summary = self.work / "summary.json"
+        config = self.work / "config.json"
+        output = self.work / "comment.md"
+        summary.write_text(
+            json.dumps(
+                {
+                    "slices": [
+                        {
+                            "id": "nonempty",
+                            "title": "Nonempty",
+                            "filter": "all",
+                            "timing": "inclusive",
+                            "sort": "total",
+                            "comparison": {
+                                "worse": {
+                                    "rows": [
+                                        {
+                                            "rank": 1,
+                                            "event_name": "Scanning Function Body",
+                                            "event_key": "cuda::std::__4::same<char>(int)",
+                                            "baseline_selected_s": "0.000001",
+                                            "current_selected_s": "0.000003",
+                                            "impact_magnitude_s": "0.000010",
+                                            "selected_delta_s": "0.000002",
+                                            "matched_trace_count": 1,
+                                        }
+                                    ]
+                                },
+                                "better": {"rows": []},
+                            },
+                            "children": [
+                                {
+                                    "id": "empty-child",
+                                    "title": "Empty child",
+                                    "filter": "all",
+                                    "timing": "inclusive",
+                                    "sort": "total",
+                                    "comparison": {
+                                        "worse": {"rows": []},
+                                        "better": {"rows": []},
+                                    },
+                                    "children": [],
+                                }
+                            ],
+                        }
+                    ]
+                }
+            ),
+            encoding="utf-8",
+        )
+        config.write_text(
+            json.dumps(
+                {
+                    "id": "public-headers",
+                    "name": "Public headers",
+                    "baseline_ref": "origin/main",
+                    "preset": "all-dev",
+                    "targets": ["cub.headers.base"],
+                    "gpu": "rtx2080",
+                    "launch_args": "--cuda 13.3 --host gcc13",
+                }
+            ),
+            encoding="utf-8",
+        )
+
+        subprocess.run(
+            [
+                sys.executable,
+                RENDER_COMMENT_SCRIPT.as_posix(),
+                "--summary",
+                summary.as_posix(),
+                "--config",
+                config.as_posix(),
+                "--artifacts-url",
+                "https://example.test/artifacts",
+                "-o",
+                output.as_posix(),
+            ],
+            cwd=REPO_ROOT,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        rendered = output.read_text(encoding="utf-8")
+        self.assertIn("<!-- cccl-compile-time-bench: public-headers -->", rendered)
+        self.assertIn("Regressions", rendered)
+        self.assertIn("Regression impact", rendered)
+        self.assertIn(
+            "Scanning Function Body: `cuda::std::__4::same<char>(int)`",
+            rendered,
+        )
+        self.assertIn("0.000010", rendered)
+        self.assertNotIn("Improvements</strong>", rendered)
+        self.assertNotIn("Empty child", rendered)
+        self.assertIn("https://example.test/artifacts", rendered)
+
+    def test_render_comment_reports_empty_slice_warnings(self) -> None:
+        summary = self.work / "summary.json"
+        config = self.work / "config.json"
+        output = self.work / "comment.md"
+        summary.write_text(
+            json.dumps(
+                {
+                    "slices": [
+                        {
+                            "id": "empty",
+                            "title": "Empty slice",
+                            "filter": "typo-filter",
+                            "timing": "inclusive",
+                            "sort": "total",
+                            "warnings": [
+                                "baseline report matched no events for this slice"
+                            ],
+                            "comparison": {
+                                "worse": {"rows": []},
+                                "better": {"rows": []},
+                            },
+                            "children": [],
+                        }
+                    ]
+                }
+            ),
+            encoding="utf-8",
+        )
+        config.write_text(
+            json.dumps(
+                {
+                    "id": "public-headers",
+                    "name": "Public headers",
+                    "baseline_ref": "origin/main",
+                    "preset": "all-dev",
+                    "targets": ["cub.headers.base"],
+                    "gpu": "rtx2080",
+                    "launch_args": "--cuda 13.3 --host gcc13",
+                }
+            ),
+            encoding="utf-8",
+        )
+
+        subprocess.run(
+            [
+                sys.executable,
+                RENDER_COMMENT_SCRIPT.as_posix(),
+                "--summary",
+                summary.as_posix(),
+                "--config",
+                config.as_posix(),
+                "--artifacts-url",
+                "https://example.test/artifacts",
+                "-o",
+                output.as_posix(),
+            ],
+            cwd=REPO_ROOT,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        rendered = output.read_text(encoding="utf-8")
+        self.assertIn("1 warning(s)", rendered)
+        self.assertIn("Empty slice — Warnings", rendered)
+        self.assertIn("baseline report matched no events", rendered)
+        self.assertNotIn(
+            "No compile-time benchmark changes exceeded the configured thresholds.",
+            rendered,
+        )
+
+    def test_render_comment_separates_top_level_slice_sections(self) -> None:
+        summary = self.work / "summary.json"
+        config = self.work / "config.json"
+        output = self.work / "comment.md"
+        row = {
+            "rank": 1,
+            "event_name": "Same",
+            "event_key": "cuda/std/same",
+            "baseline_selected_s": "0.000001",
+            "current_selected_s": "0.000003",
+            "impact_magnitude_s": "0.000010",
+            "selected_delta_s": "0.000002",
+            "matched_trace_count": 1,
+        }
+        summary.write_text(
+            json.dumps(
+                {
+                    "slices": [
+                        {
+                            "id": "first",
+                            "title": "First",
+                            "filter": "all",
+                            "timing": "inclusive",
+                            "sort": "total",
+                            "comparison": {
+                                "worse": {"rows": [row]},
+                                "better": {"rows": []},
+                            },
+                            "children": [],
+                        },
+                        {
+                            "id": "second",
+                            "title": "Second",
+                            "filter": "all",
+                            "timing": "inclusive",
+                            "sort": "total",
+                            "comparison": {
+                                "worse": {"rows": [row]},
+                                "better": {"rows": []},
+                            },
+                            "children": [],
+                        },
+                    ]
+                }
+            ),
+            encoding="utf-8",
+        )
+        config.write_text(
+            json.dumps(
+                {
+                    "id": "public-headers",
+                    "name": "Public headers",
+                    "baseline_ref": "origin/main",
+                    "preset": "all-dev",
+                    "targets": ["cub.headers.base"],
+                    "gpu": "rtx2080",
+                    "launch_args": "--cuda 13.3 --host gcc13",
+                }
+            ),
+            encoding="utf-8",
+        )
+
+        subprocess.run(
+            [
+                sys.executable,
+                RENDER_COMMENT_SCRIPT.as_posix(),
+                "--summary",
+                summary.as_posix(),
+                "--config",
+                config.as_posix(),
+                "--artifacts-url",
+                "https://example.test/artifacts",
+                "-o",
+                output.as_posix(),
+            ],
+            cwd=REPO_ROOT,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        rendered = output.read_text(encoding="utf-8")
+        self.assertIn("</details>\n\n### Second", rendered)
+        self.assertNotIn("</details>\n### Second", rendered)
+
+
+class PrepareTracesTest(unittest.TestCase):
+    def setUp(self) -> None:
+        self.tempdir = tempfile.TemporaryDirectory()
+        self.work = Path(self.tempdir.name)
+
+    def tearDown(self) -> None:
+        self.tempdir.cleanup()
+
+    def test_promotes_project_details_into_perfetto_event_names(self) -> None:
+        input_dir = self.work / "raw"
+        output_dir = self.work / "perfetto"
+        trace_path = input_dir / "target" / "trace.json"
+        trace_path.parent.mkdir(parents=True, exist_ok=True)
+        trace_path.write_text(
+            json.dumps(
+                {
+                    "traceEvents": [
+                        {
+                            "ph": "X",
+                            "name": "Processing Header File",
+                            "args": {
+                                "detail": (
+                                    REPO_ROOT
+                                    / "libcudacxx/include/cuda/std/string_view"
+                                ).as_posix()
+                            },
+                        },
+                        {
+                            "ph": "X",
+                            "name": "Other Event",
+                            "args": {"detail": "not promoted"},
+                        },
+                    ]
+                }
+            ),
+            encoding="utf-8",
+        )
+
+        subprocess.run(
+            [
+                sys.executable,
+                PREPARE_SCRIPT.as_posix(),
+                "--input",
+                input_dir.as_posix(),
+                "--output",
+                output_dir.as_posix(),
+                "--repo-root",
+                REPO_ROOT.as_posix(),
+            ],
+            cwd=REPO_ROOT,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        output_trace = output_dir / "target" / "trace.perfetto.json"
+        with output_trace.open(encoding="utf-8") as f:
+            events = json.load(f)["traceEvents"]
+
+        self.assertEqual(
+            events[0]["name"], "Processing Header File: cuda/std/string_view"
+        )
+        self.assertEqual(events[0]["args"]["original_name"], "Processing Header File")
+        self.assertEqual(events[1]["name"], "Other Event")
+
+    def test_single_file_input_accepts_output_directory(self) -> None:
+        input_trace = self.work / "trace.json"
+        output_dir = self.work / "perfetto"
+        input_trace.write_text(
+            json.dumps(
+                {
+                    "traceEvents": [
+                        {
+                            "ph": "X",
+                            "name": "Processing Header File",
+                            "args": {"detail": "cuda/std/string_view"},
+                        }
+                    ]
+                }
+            ),
+            encoding="utf-8",
+        )
+
+        subprocess.run(
+            [
+                sys.executable,
+                PREPARE_SCRIPT.as_posix(),
+                "--input",
+                input_trace.as_posix(),
+                "--output",
+                output_dir.as_posix(),
+                "--repo-root",
+                REPO_ROOT.as_posix(),
+            ],
+            cwd=REPO_ROOT,
+            check=True,
+            text=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+        )
+
+        self.assertTrue((output_dir / "trace.perfetto.json").exists())
+
+
+class SummarizeTusTest(unittest.TestCase):
+    def test_generated_tu_input(self) -> None:
+        build_dir = Path("/tmp/build")
+        pp_path = (
+            build_dir
+            / "libcudacxx/test"
+            / "headers"
+            / "libcudacxx.test.public_headers"
+            / "cuda/std/string_view.cpp4.ii"
+        )
+
+        self.assertEqual(
+            summarize_tus.generated_tu_input(pp_path.with_name("string_view.cpp")),
+            "cuda/std/string_view",
+        )
+
+    def test_finds_nvcc_and_clang_preprocessed_tus(self) -> None:
+        with tempfile.TemporaryDirectory() as tmp:
+            build_dir = Path(tmp)
+            header_dir = build_dir / "target" / "headers" / "generated"
+            header_dir.mkdir(parents=True)
+            nvcc_tu = header_dir / "cuda_std_span.cpp4.ii"
+            clang_tu = header_dir / "cuda_std_string_view.ii"
+            nvcc_tu.write_text("", encoding="utf-8")
+            clang_tu.write_text("", encoding="utf-8")
+
+            self.assertEqual(
+                summarize_tus.find_preprocessed_tus(build_dir),
+                [nvcc_tu, clang_tu],
+            )
+            self.assertEqual(
+                summarize_tus.tu_source_for_preprocessed_tu(nvcc_tu),
+                header_dir / "cuda_std_span",
+            )
+            self.assertEqual(
+                summarize_tus.tu_source_for_preprocessed_tu(clang_tu),
+                header_dir / "cuda_std_string_view",
+            )
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/cccl_upstream/ci/compute-sanitizer-suppressions.xml b/cccl_upstream/ci/compute-sanitizer-suppressions.xml
new file mode 100644
index 00000000..37aa0fe5
--- /dev/null
+++ b/cccl_upstream/ci/compute-sanitizer-suppressions.xml
@@ -0,0 +1,453 @@
+<?xml version="1.0" encoding="utf-8"?>
+<ComputeSanitizerOutput>
+  <!--
+  thrust::equal reduces using an accumulator type tuple<bool, OffsetT>.
+  The padding bytes inside the tuple are not initialized.
+  This causes issues during `cub::ThreadLoad`, which loads the memory as aliased
+  machine words that are cast to the tuple type.
+  -->
+  <record>
+    <kind>Initcheck</kind>
+    <what>
+      <text>Uninitialized __global__ memory read of size 2 bytes</text>
+      <size>2</size>
+    </what>
+    <where>
+      <func>ThreadLoad</func>
+    </where>
+    <deviceStack>
+      <frame>
+        <func>UnrolledThreadLoadImpl</func>
+      </frame>
+      <frame>
+        <func>UnrolledThreadLoad</func>
+      </frame>
+      <frame>
+        <func>ThreadLoad</func>
+      </frame>
+      <frame>
+        <func>ThreadLoad</func>
+      </frame>
+    </deviceStack>
+    <hostStack>
+      <frame>
+        <module>.*libcuda.so.*</module>
+      </frame>
+      <frame>
+        <func>libcudart_static.*</func>
+      </frame>
+      <frame>
+        <func>cudaLaunchKernel</func>
+      </frame>
+      <frame>
+        <func>.*cub::.*::DeviceReduce.*.*thrust::.*find_if.*</func>
+      </frame>
+    </hostStack>
+  </record>
+  <!--
+  Similar to the above, thrust::equal copies a tuple<bool, OffsetT> from host -> device
+  with the result of the comparison. The padding bytes trigger host API initialization
+  errors during the cudaMemcpy.
+  Sadly, this is a very generic suppression that may hide real issues, but it's the best
+  we can do given the current tooling.
+  -->
+  <record>
+    <kind>InitcheckApiError</kind>
+    <level>Error</level>
+    <what>
+      <text>Host API uninitialized memory access</text>
+      <accessSize>16</accessSize>
+    </what>
+    <hostStack>
+      <saveLocation>error</saveLocation>
+      <frame>
+        <module>.*/libcuda.so.*</module>
+      </frame>
+      <frame>
+        <func>libcudart_static_.*</func>
+      </frame>
+      <frame>
+        <func>libcudart_static_.*</func>
+      </frame>
+      <frame>
+        <func>cudaMemcpyAsync</func>
+      </frame>
+      <frame>
+        <func>void C_A_T_C_H_T_E_M_P_L_A_T_E_T_E_S_T_F_U_N_C_.*</func>
+      </frame>
+    </hostStack>
+  </record>
+  <!-- Another variant of the above with a different catch2 dispatcher -->
+  <record>
+    <kind>InitcheckApiError</kind>
+    <level>Error</level>
+    <what>
+      <text>Host API uninitialized memory access</text>
+      <accessSize>16</accessSize>
+    </what>
+    <hostStack>
+      <saveLocation>error</saveLocation>
+      <frame>
+        <module>.*/libcuda.so.*</module>
+      </frame>
+      <frame>
+        <func>libcudart_static_.*</func>
+      </frame>
+      <frame>
+        <func>libcudart_static_.*</func>
+      </frame>
+      <frame>
+        <func>cudaMemcpyAsync</func>
+      </frame>
+      <frame>
+        <func>void CATCH2_INTERNAL_TEMPLATE_TEST.*</func>
+      </frame>
+    </hostStack>
+  </record>
+  <!-- Yet another instance of that tuple<Offset, bool> padding -->
+  <record>
+    <kind>InitcheckApiError</kind>
+    <level>Error</level>
+    <what>
+      <text>Host API uninitialized memory access</text>
+      <accessSize>16</accessSize>
+    </what>
+    <hostStack>
+      <saveLocation>error</saveLocation>
+      <frame>
+        <module>.*libcuda.so.*</module>
+      </frame>
+      <frame>
+        <func>libcudart_static.*</func>
+      </frame>
+      <frame>
+        <func>libcudart_static.*</func>
+      </frame>
+      <frame>
+        <func>cudaMemcpyAsync</func>
+      </frame>
+      <frame>
+        <func>.*thrust::.*(equal|operator==|mismatch|find_if).*</func>
+      </frame>
+    </hostStack>
+  </record>
+  <!-- Yet another instance of that tuple<Offset, bool> padding -->
+  <record>
+    <kind>InitcheckApiError</kind>
+    <level>Error</level>
+    <what>
+      <text>Host API uninitialized memory access</text>
+      <accessSize>16</accessSize>
+    </what>
+    <hostStack>
+      <saveLocation>error</saveLocation>
+      <frame>
+        <module>.*libcuda.so.*</module>
+      </frame>
+      <frame>
+        <func>libcudart_static.*</func>
+      </frame>
+      <frame>
+        <func>libcudart_static.*</func>
+      </frame>
+      <frame>
+        <func>cudaMemcpyAsync</func>
+      </frame>
+      <frame>
+        <func>bool binary_equal.*</func> <!-- Implementation detail of CUB's block radix sort tests, calls thrust::equal -->
+      </frame>
+    </hostStack>
+  </record>
+  <!-- Yet another instance of that tuple<Offset, bool> padding -->
+  <record>
+    <kind>InitcheckApiError</kind>
+    <level>Error</level>
+    <what>
+      <text>Host API uninitialized memory access</text>
+      <accessSize>16</accessSize>
+    </what>
+    <hostStack>
+      <saveLocation>error</saveLocation>
+      <frame>
+        <module>.*libcuda.so.*</module>
+      </frame>
+      <frame>
+        <func>libcudart_static.*</func>
+      </frame>
+      <frame>
+        <func>libcudart_static.*</func>
+      </frame>
+      <frame>
+        <func>cudaMemcpyAsync</func>
+      </frame>
+      <frame>
+        <!-- CUB's namespace_wrapped test has this issue inlined into main: -->
+        <func>main</func>
+        <module>.*cub.*test.namespace_wrapped</module>
+      </frame>
+    </hostStack>
+  </record>
+  <!-- Yet another instance of that tuple<Offset, bool> padding -->
+  <record>
+    <kind>InitcheckApiError</kind>
+    <what>
+      <text>Host API uninitialized memory access</text>
+      <accessSize>16</accessSize>
+    </what>
+    <hostStack>
+      <saveLocation>error</saveLocation>
+      <frame>
+        <module>.*libcuda.so.*</module>
+      </frame>
+      <frame>
+        <func>libcudart_static.*</func>
+      </frame>
+      <frame>
+        <func>libcudart_static.*</func>
+      </frame>
+      <frame>
+        <func>cudaMemcpyAsync</func>
+      </frame>
+      <frame>
+        <!-- The segmented sort test have several stacks that have padding bit issues. -->
+        <module>.*cub.*device_segmented_sort.*</module>
+      </frame>
+    </hostStack>
+  </record>
+  <!-- Yet another instance of that tuple<Offset, bool> padding -->
+  <record>
+    <kind>InitcheckApiError</kind>
+    <what>
+      <text>Host API uninitialized memory access</text>
+      <accessSize>16</accessSize>
+    </what>
+    <hostStack>
+      <saveLocation>error</saveLocation>
+      <frame>
+        <module>.*libcuda.so.*</module>
+      </frame>
+      <frame>
+        <func>libcudart_static.*</func>
+      </frame>
+      <frame>
+        <func>libcudart_static.*</func>
+      </frame>
+      <frame>
+        <func>cudaMemcpyAsync</func>
+      </frame>
+      <frame>
+        <!-- The RLE tests have several stacks that have padding bit issues. -->
+        <module>.*cub.*device_run_length_encode.*</module>
+      </frame>
+    </hostStack>
+  </record>
+  <!--
+  DeviceRunLengthEncode performs a WarpExchange::ScatterToStriped that 'discards'
+  elements by scattering them to the same (ignored) destination address. This
+  triggers a WAW race that we can safely ignore.
+  -->
+  <record>
+    <kind>Analysis</kind>
+    <level>Error</level>
+    <what>
+      <text>Race condition</text>
+      <source>
+        <direction>Write</direction>
+        <where>
+          <func>ScatterToStriped</func>
+        </where>
+      </source>
+      <destination>
+        <direction>Write</direction>
+        <where>
+          <func>ScatterToStriped</func>
+        </where>
+      </destination>
+      <destination>
+        <direction>Write</direction>
+        <where>
+          <func>ScatterToStriped</func>
+        </where>
+      </destination>
+      <destination>
+        <direction>Write</direction>
+        <where>
+          <func>ScatterToStriped</func>
+        </where>
+      </destination>
+    </what>
+  </record>
+  <!-- Another variation of the above -->
+  <record>
+    <kind>Analysis</kind>
+    <level>Error</level>
+    <what>
+      <text>Race condition</text>
+      <source>
+        <direction>Write</direction>
+        <where>
+          <func>ScatterToStriped</func>
+        </where>
+      </source>
+      <destination>
+        <direction>Write</direction>
+        <where>
+          <func>ScatterToStriped</func>
+        </where>
+      </destination>
+    </what>
+  </record>
+  <!-- Another variation of the above -->
+  <record>
+    <kind>Analysis</kind>
+    <level>Error</level>
+    <what>
+      <text>Race condition</text>
+      <source>
+        <direction>Write</direction>
+        <where>
+          <func>ScatterToStriped</func>
+        </where>
+      </source>
+      <destination>
+        <direction>Write</direction>
+        <where>
+          <func>ScatterToStriped</func>
+        </where>
+      </destination>
+      <destination>
+        <direction>Write</direction>
+        <where>
+          <func>ScatterToStriped</func>
+        </where>
+      </destination>
+    </what>
+  </record>
+  <!--
+  There are uninitialized padding bits inside cub::ConstantInputIterator,
+  which is basically a struct{ T value; ptrdiff_t offset; }.
+  -->
+  <record>
+    <kind>InitcheckApiError</kind>
+    <level>Error</level>
+    <what>
+      <text>Host API uninitialized memory access</text>
+      <accessSize>32</accessSize>
+    </what>
+    <hostStack>
+      <saveLocation>error</saveLocation>
+      <frame>
+        <module>.*libcuda.so.*</module>
+      </frame>
+      <frame>
+        <func>libcudart_static_.*</func>
+      </frame>
+      <frame>
+        <func>libcudart_static_.*</func>
+      </frame>
+      <frame>
+        <func>cudaMemcpyAsync</func>
+      </frame>
+      <frame>
+        <func>thrust.*vector_base.*ConstantInputIterator.*</func>
+      </frame>
+      <frame>
+        <func>void test_iterator.*ConstantInputIterator.*</func>
+      </frame>
+    </hostStack>
+  </record>
+  <!--
+  Similar to the above; cub::TransformInputIterator is a struct{TransformOp op; InputIterT iter;}.
+  In this case TransformOp is 1 byte, and InputIterT is an 8-byte pointer.
+  Padding bits strike again.
+  -->
+    <record>
+    <kind>InitcheckApiError</kind>
+    <level>Error</level>
+    <what>
+      <text>Host API uninitialized memory access</text>
+      <accessSize>32</accessSize>
+    </what>
+    <hostStack>
+      <saveLocation>error</saveLocation>
+      <frame>
+        <module>.*/libcuda.so.*</module>
+      </frame>
+      <frame>
+        <func>libcudart_static_.*</func>
+      </frame>
+      <frame>
+        <func>libcudart_static_.*</func>
+      </frame>
+      <frame>
+        <func>cudaMemcpyAsync</func>
+      </frame>
+      <frame>
+        <func>thrust.*vector_base.*TransformInputIterator.*</func>
+      </frame>
+      <frame>
+        <func>void test_iterator.*TransformInputIterator.*</func>
+      </frame>
+    </hostStack>
+  </record>
+  <!--
+  Same as above; this time InputIterT is a cub::TexObjInputIterator.
+  -->
+  <record>
+    <kind>InitcheckApiError</kind>
+    <level>Error</level>
+    <what>
+      <text>Host API uninitialized memory access</text>
+      <accessSize>64</accessSize>
+    </what>
+    <hostStack>
+      <saveLocation>error</saveLocation>
+      <frame>
+        <module>.*/libcuda.so.*</module>
+      </frame>
+      <frame>
+        <func>libcudart_static_.*</func>
+      </frame>
+      <frame>
+        <func>libcudart_static_.*</func>
+      </frame>
+      <frame>
+        <func>cudaMemcpyAsync</func>
+      </frame>
+      <frame>
+        <func>thrust.*vector_base.*TransformInputIterator.*</func>
+      </frame>
+      <frame>
+        <func>void test_iterator.*TransformInputIterator.*</func>
+      </frame>
+    </hostStack>
+  </record>
+  <!--
+  cub.test.device_reduce transfers cub::KeyValuePair<int, unwrap_value_t<...>> from
+  device -> host. This suppresses warnings about transferring padding bits inside
+  of the KeyValuePair.
+  -->
+  <record>
+    <kind>InitcheckApiError</kind>
+    <level>Error</level>
+    <what>
+      <text>Host API uninitialized memory access</text>
+    </what>
+    <hostStack>
+      <saveLocation>error</saveLocation>
+      <frame>
+        <module>.*libcuda.so.*</module>
+      </frame>
+      <frame>
+        <func>libcudart_static.*</func>
+      </frame>
+      <frame>
+        <func>libcudart_static.*</func>
+      </frame>
+      <frame>
+        <func>cudaMemcpyAsync</func>
+        <module>.*cub.*device_(segmented_|)reduce.*</module>
+      </frame>
+    </hostStack>
+  </record>
+</ComputeSanitizerOutput>
diff --git a/cccl_upstream/ci/generate_version.sh b/cccl_upstream/ci/generate_version.sh
new file mode 100755
index 00000000..cffe3785
--- /dev/null
+++ b/cccl_upstream/ci/generate_version.sh
@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+
+# Generate a version number string using metadata from git or JSON.
+# Use the PyPi package versioning convention for pre-release or
+# post release patches.
+
+# Set some defaults for variables.
+CCCL_BRANCH="${CCCL_BRANCH:-dev}"
+PACKAGE_VERSION_PREFIX="${PACKAGE_VERSION_PREFIX:-}"
+
+GIT_DESCRIBE_TAG=$(git describe --tags --match "v[0-9]*" --abbrev=0)
+GIT_DESCRIBE_NUMBER=$(git rev-list "${GIT_DESCRIBE_TAG}"..HEAD --count)
+
+JSON_VERSION=$(jq -r .full /workspace/cccl-version.json)
+
+# Generate a suffix depending on release or dev branch.
+PACKAGE_VERSION_SUFFIX=""
+if [[ "${GIT_DESCRIBE_NUMBER}" != "0" ]]; then
+  if [[ ${CCCL_BRANCH} == "dev" ]]; then
+    PACKAGE_VERSION_SUFFIX=".dev${GIT_DESCRIBE_NUMBER}"
+  else
+    PACKAGE_VERSION_SUFFIX=".post${GIT_DESCRIBE_NUMBER}"
+  fi
+fi
+
+# If using Git metadata this could generate it from the last tag.
+# VERSION="${GIT_DESCRIBE_TAG#v}.dev${GIT_DESCRIBE_NUMBER}"
+
+# Generate the version using a combination of JSON and git commit number.
+VERSION="${PACKAGE_VERSION_PREFIX}${JSON_VERSION}${PACKAGE_VERSION_SUFFIX}"
+
+echo -n "${VERSION}"
diff --git a/cccl_upstream/ci/inspect_changes.py b/cccl_upstream/ci/inspect_changes.py
new file mode 100755
index 00000000..cfd013ae
--- /dev/null
+++ b/cccl_upstream/ci/inspect_changes.py
@@ -0,0 +1,691 @@
+#!/usr/bin/env python3
+"""Identify dirty CCCL subprojects between two commits or explicit path lists."""
+
+from __future__ import annotations
+
+import argparse
+import os
+import re
+import subprocess
+import sys
+from collections import deque
+from dataclasses import dataclass, replace
+from pathlib import Path
+from typing import Dict, List, Optional, Sequence, Tuple
+
+import yaml
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+CONFIG_PATH = REPO_ROOT / "ci" / "project_files_and_dependencies.yaml"
+CORE_PROJECT_KEY = "core"
+
+
+class SummaryWriter:
+    """Utility for duplicating output to stdout and an optional summary file."""
+
+    def __init__(self, path: Optional[Path]):
+        self._handle = path.open("a", encoding="utf-8") if path else None
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc, tb):
+        if self._handle:
+            self._handle.close()
+
+    def log(self, line: str = "") -> None:
+        print(line)
+        if self._handle:
+            self._handle.write(line + "\n")
+
+
+@dataclass(frozen=True)
+class ProjectConfig:
+    """Per-project configuration derived from the YAML file."""
+
+    key: str
+    name: str
+    matrix_project: Optional[str]
+    include_regexes: Tuple[str, ...]
+    exclude_regexes: Tuple[str, ...]
+    exclude_project_files: Tuple[str, ...]
+    lite_dependencies: Tuple[str, ...]
+    full_dependencies: Tuple[str, ...]
+    transitive_lite_dependencies: Tuple[str, ...] = ()
+
+
+@dataclass(frozen=True)
+class Config:
+    """Aggregated configuration for change detection."""
+
+    projects: Dict[str, ProjectConfig]
+    project_keys: Tuple[str, ...]
+    ignore_regexes: Tuple[str, ...]
+
+    def project(self, key: str) -> ProjectConfig:
+        return self.projects[key]
+
+
+def parse_args(argv: Optional[Sequence[str]] = None) -> argparse.Namespace:
+    """Parse CLI arguments for determining dirty files."""
+    parser = argparse.ArgumentParser(
+        description="Identify which CCCL projects require rebuilds between two commits."
+    )
+    group = parser.add_mutually_exclusive_group(required=True)
+    group.add_argument(
+        "--refs",
+        nargs=2,
+        metavar=("BASE", "HEAD"),
+        help="Compare two refs using 'git diff --name-only' to determine dirty files",
+    )
+    group.add_argument(
+        "--file",
+        metavar="PATH",
+        help="Read dirty file paths (one per line) from PATH",
+    )
+    group.add_argument(
+        "--stdin",
+        action="store_true",
+        help="Read dirty file paths (one per line) from stdin",
+    )
+    parser.add_argument(
+        "--summary",
+        metavar="PATH",
+        default=None,
+        help="Optional path to write a markdown summary table",
+    )
+    return parser.parse_args(argv)
+
+
+def load_config(path: Path) -> Config:
+    raw = yaml.safe_load(path.read_text(encoding="utf-8"))
+    projects_raw = raw.get("projects")
+    if not isinstance(projects_raw, dict) or not projects_raw:
+        raise SystemExit(f"No projects defined in {path}")
+
+    def to_tuple(value: Optional[Sequence[str] | str]) -> Tuple[str, ...]:
+        if value is None:
+            return tuple()
+        if isinstance(value, (list, tuple)):
+            return tuple(value)
+        return (value,)
+
+    project_keys: List[str] = list(projects_raw.keys())
+    projects: Dict[str, ProjectConfig] = {}
+    for key in project_keys:
+        entry = projects_raw.get(key) or {}
+        name = entry.get("name", key)
+        matrix_project = entry.get("matrix_project")
+        include_regexes = to_tuple(entry.get("include_regexes", []))
+        exclude_regexes = to_tuple(entry.get("exclude_regexes", []))
+        exclude_project_files = to_tuple(entry.get("exclude_project_files"))
+        lite_dependencies = to_tuple(entry.get("lite_dependencies"))
+        full_dependencies = to_tuple(entry.get("full_dependencies"))
+
+        if key != CORE_PROJECT_KEY and not include_regexes:
+            raise SystemExit(
+                f"Project '{key}' must define at least one include_regex in {path}"
+            )
+
+        projects[key] = ProjectConfig(
+            key=key,
+            name=name,
+            matrix_project=matrix_project,
+            include_regexes=include_regexes,
+            exclude_regexes=exclude_regexes,
+            exclude_project_files=exclude_project_files,
+            lite_dependencies=lite_dependencies,
+            full_dependencies=full_dependencies,
+            transitive_lite_dependencies=tuple(),
+        )
+
+    if CORE_PROJECT_KEY not in projects:
+        raise SystemExit(f"Project configuration must define '{CORE_PROJECT_KEY}'")
+
+    dependency_graph = build_dependency_graph_raw(projects)
+    transitive_map = compute_transitive_dependencies(
+        project_keys, projects, dependency_graph
+    )
+
+    for key, transitive in transitive_map.items():
+        projects[key] = replace(projects[key], transitive_lite_dependencies=transitive)
+
+    ignore_regexes = tuple(raw.get("ignore_regexes", ()))
+
+    return Config(
+        projects=projects,
+        project_keys=tuple(project_keys),
+        ignore_regexes=ignore_regexes,
+    )
+
+
+def run_git(args: Sequence[str], *, capture_output: bool = True) -> str:
+    """Run a git command in the repo and optionally capture stdout."""
+    result = subprocess.run(
+        ["git", *args],
+        cwd=REPO_ROOT,
+        check=True,
+        text=True,
+        stdout=subprocess.PIPE if capture_output else None,
+        stderr=subprocess.PIPE if capture_output else None,
+    )
+    if capture_output:
+        return result.stdout.strip()
+    return ""
+
+
+def try_rev_parse(ref: str) -> str:
+    try:
+        return run_git(["rev-parse", ref])
+    except subprocess.CalledProcessError:
+        return ref
+
+
+def ensure_fetched(ref: str) -> None:
+    """Fetch a ref from origin ensuring availability for merge-base."""
+    run_git(["fetch", "origin", ref, "-q"], capture_output=False)
+
+
+def repo_is_shallow() -> bool:
+    """Return True when the repository has a shallow history."""
+    output = run_git(["rev-parse", "--is-shallow-repository"])
+    return output.strip().lower() == "true"
+
+
+def indent(text: str, prefix: str) -> str:
+    """Indent multi-line text for readable logging blocks."""
+    return "\n".join(f"{prefix}{line}" for line in text.splitlines())
+
+
+def anchor_regex(pattern: str) -> re.Pattern[str]:
+    """Anchor a path regex to the repository root."""
+    anchored = pattern if pattern.startswith("^") else f"^{pattern}"
+    return re.compile(anchored)
+
+
+def compile_patterns(patterns: Sequence[str]) -> Tuple[re.Pattern[str], ...]:
+    """Compile a list of regex strings into anchored patterns."""
+    return tuple(anchor_regex(pattern) for pattern in patterns)
+
+
+def matches_any(patterns: Sequence[re.Pattern[str]], path: str) -> bool:
+    """Return True when any compiled regex matches the given path."""
+    return any(pattern.search(path) for pattern in patterns)
+
+
+def build_dependency_graph_raw(
+    projects: Dict[str, ProjectConfig],
+) -> Dict[str, List[Tuple[str, str]]]:
+    """Return mapping of project -> [(dependency, type)]."""
+    graph: Dict[str, List[Tuple[str, str]]] = {}
+    for key, project in projects.items():
+        edges: List[Tuple[str, str]] = []
+        edges.extend((dep, "full") for dep in project.full_dependencies)
+        edges.extend((dep, "lite") for dep in project.lite_dependencies)
+        graph[key] = edges
+    return graph
+
+
+def compute_transitive_dependencies(
+    project_keys: Sequence[str],
+    projects: Dict[str, ProjectConfig],
+    graph: Dict[str, List[Tuple[str, str]]],
+) -> Dict[str, Tuple[str, ...]]:
+    """Return a dictionary of transitive dependencies for each project."""
+    result: Dict[str, Tuple[str, ...]] = {}
+
+    for key in project_keys:
+        visited: set[str] = set()
+        queue: deque[str] = deque(dep for dep, _ in graph.get(key, []))
+
+        while queue:
+            dep = queue.popleft()
+            if dep == key or dep in visited:
+                continue
+            visited.add(dep)
+            queue.extend(child for child, _ in graph.get(dep, []))
+
+        project_cfg = projects[key]
+        direct_full = set(project_cfg.full_dependencies)
+        direct_lite = set(project_cfg.lite_dependencies)
+        transitive = [
+            dep
+            for dep in project_keys
+            if dep in visited and dep not in direct_full and dep not in direct_lite
+        ]
+
+        result[key] = tuple(transitive)
+
+    return result
+
+
+def project_dirty_files(
+    project: ProjectConfig, dirty_files: Sequence[str]
+) -> List[str]:
+    """Collect dirty files that belong to the given project. Does not apply project exclusions yet."""
+    include_patterns = compile_patterns(project.include_regexes)
+    exclude_patterns = compile_patterns(project.exclude_regexes)
+
+    if not include_patterns:
+        return []
+
+    included = [path for path in dirty_files if matches_any(include_patterns, path)]
+
+    if exclude_patterns:
+        included = [
+            path for path in included if not matches_any(exclude_patterns, path)
+        ]
+
+    return included
+
+
+def build_project_dirty_map(
+    config: Config, dirty_files: Sequence[str]
+) -> Dict[str, List[str]]:
+    """Compute per-project dirty file lists, including a residual `core` list."""
+    project_files: Dict[str, List[str]] = {}
+
+    # First gather matches for every non-core project. Files may belong to multiple projects.
+    for key in config.project_keys:
+        if key == CORE_PROJECT_KEY:
+            continue
+        files = project_dirty_files(config.project(key), dirty_files)
+        project_files[key] = files
+
+    # Remove files that are owned by other projects when requested.
+    for key in config.project_keys:
+        if key == CORE_PROJECT_KEY:
+            continue
+        project = config.project(key)
+        if not project.exclude_project_files:
+            continue
+        excluded_sets = [
+            set(project_files.get(other, [])) for other in project.exclude_project_files
+        ]
+        if not excluded_sets:
+            continue
+        exclusions = set().union(*excluded_sets)
+        project_files[key] = [
+            path for path in project_files[key] if path not in exclusions
+        ]
+
+    matched_paths: set[str] = set()
+    for key in config.project_keys:
+        if key == CORE_PROJECT_KEY:
+            continue
+        matched_paths.update(project_files.get(key, []))
+
+    core_files = [path for path in dirty_files if path not in matched_paths]
+    project_files[CORE_PROJECT_KEY] = core_files
+    return project_files
+
+
+def write_output(key: str, value: str) -> None:
+    """Emit a GitHub Actions output key/value pair."""
+    line = f"{key}={value}"
+    print(line)
+    output_path = os.environ.get("GITHUB_OUTPUT")
+    if output_path:
+        with open(output_path, "a", encoding="utf-8") as handle:
+            handle.write(f"{line}\n")
+
+
+def collect_dirty_files(
+    args: argparse.Namespace, config: Config
+) -> Tuple[List[str], List[str]]:
+    """Return normalized dirty file paths and the subset filtered as ignored."""
+    if args.refs:
+        if repo_is_shallow():
+            run_git(["fetch", "origin", "--unshallow", "-q"], capture_output=False)
+
+        base_ref = try_rev_parse(args.refs[0])
+        head_ref = try_rev_parse(args.refs[1])
+
+        ensure_fetched(base_ref)
+        ensure_fetched(head_ref)
+
+        base_sha = run_git(["merge-base", base_ref, head_ref])
+        head_sha = run_git(["rev-parse", head_ref])
+
+        print(f"Base SHA: {base_sha}")
+        base_log = run_git(["log", "--oneline", "-1", base_sha])
+        if base_log:
+            print(indent(base_log, "  "))
+        print(f"HEAD SHA: {head_sha}")
+        head_log = run_git(["log", "--oneline", "-1", head_sha])
+        if head_log:
+            print(indent(head_log, "  "))
+
+        dirty_files = run_git(["diff", "--name-only", base_sha, head_sha]).splitlines()
+    elif args.file:
+        print(f"Dirty files provided via file: {args.file}")
+        with open(args.file, "r", encoding="utf-8") as handle:
+            dirty_files = [line.strip() for line in handle if line.strip()]
+    else:  # args.stdin
+        print("Dirty files provided via stdin")
+        dirty_files = [line.strip() for line in sys.stdin if line.strip()]
+
+    print()
+
+    dirty_files = [path for path in dirty_files if path]
+
+    ignore_patterns = compile_patterns(config.ignore_regexes)
+    if ignore_patterns:
+        kept: List[str] = []
+        ignored: List[str] = []
+        for path in dirty_files:
+            if matches_any(ignore_patterns, path):
+                ignored.append(path)
+            else:
+                kept.append(path)
+        return kept, ignored
+
+    return dirty_files, []
+
+
+def format_bullet_list(lines: Sequence[str], indent: str = "  ") -> List[str]:
+    """Convert each string into a markdown bullet line with the given indent."""
+    return [f"{indent}- {line}" for line in lines]
+
+
+def build_reverse_dependency_graph(config: Config) -> Dict[str, List[Tuple[str, bool]]]:
+    """Return mapping of dependency -> (dependent, requires_full_rebuild)."""
+    reverse: Dict[str, List[Tuple[str, bool]]] = {}
+
+    for project in config.projects.values():
+        for dep in project.full_dependencies:
+            reverse.setdefault(dep, []).append((project.key, True))
+        for dep in project.lite_dependencies:
+            reverse.setdefault(dep, []).append((project.key, False))
+        for dep in project.transitive_lite_dependencies:
+            reverse.setdefault(dep, []).append((project.key, False))
+
+    return reverse
+
+
+def propagate_dirty_projects(
+    config: Config,
+    initial_full: Sequence[str],  # Projects with dirty files
+    reverse_graph: Dict[
+        str, List[Tuple[str, bool]]
+    ],  # dependency -> [(dependent, requires_full_rebuild)]
+) -> Tuple[set[str], set[str]]:  # (full_set, lite_set)
+    """Propagate rebuild requirements through the reverse dependency graph."""
+    full_set: set[str] = set(initial_full)
+    lite_set: set[str] = set()
+    # BFS queue of (project_key, depth)
+    queue: deque[Tuple[str, int]] = deque((key, 0) for key in initial_full)
+    seen: set[Tuple[str, int]] = set(queue)
+
+    while queue:
+        current, depth = queue.popleft()
+        for dependent, edge_full in reverse_graph.get(current, []):
+            propagate_full = edge_full and depth == 0
+
+            if propagate_full:
+                if dependent in full_set:
+                    continue
+                if dependent in lite_set:
+                    lite_set.remove(dependent)
+                full_set.add(dependent)
+                print(
+                    "- Upstream dependency change detected (full rebuild): "
+                    f"'{dependent}' ({config.project(dependent).name}) depends on dirty project '{current}'"
+                )
+            else:
+                if dependent in full_set or dependent in lite_set:
+                    continue
+                lite_set.add(dependent)
+                print(
+                    "- Upstream dependency change detected (lite rebuild): "
+                    f"'{dependent}' ({config.project(dependent).name}) depends on dirty project '{current}'"
+                )
+
+            key = (dependent, depth + 1)
+            if key not in seen:
+                seen.add(key)
+                queue.append(key)
+
+    return full_set, lite_set
+
+
+def log_dependency_overview(config: Config) -> None:
+    """Pretty-print the dependency data for each project."""
+    print("Project Dependency Overview:")
+    for key in config.project_keys:
+        project = config.project(key)
+        header = f"- {project.name} (key={key}"
+        if project.matrix_project:
+            header += f" matrix_project={project.matrix_project}"
+        header += ")"
+        print(header)
+
+        if project.full_dependencies:
+            print("  - direct full deps:")
+            for line in format_bullet_list(project.full_dependencies, indent="      "):
+                print(line)
+        if project.lite_dependencies:
+            print("  - direct lite deps:")
+            for line in format_bullet_list(project.lite_dependencies, indent="      "):
+                print(line)
+        if project.transitive_lite_dependencies:
+            print("  - transitive deps:")
+            for line in format_bullet_list(
+                project.transitive_lite_dependencies, indent="      "
+            ):
+                print(line)
+    print()
+
+
+def build_dirty_sections(
+    config: Config,
+    project_dirty_map: Dict[str, List[str]],
+    ignored_files: Sequence[str],
+) -> List[Tuple[str, List[str]]]:
+    """Create (heading, files) tuples for non-empty dirty buckets."""
+    sections: List[Tuple[str, List[str]]] = []
+    for key in config.project_keys:
+        if key == CORE_PROJECT_KEY:
+            continue
+        files = project_dirty_map.get(key, [])
+        if files:
+            sections.append((f"{config.project(key).name} ({key})", list(files)))
+
+    core_files = project_dirty_map.get(CORE_PROJECT_KEY, [])
+    if core_files:
+        sections.append(
+            (
+                f"{config.project(CORE_PROJECT_KEY).name} ({CORE_PROJECT_KEY})",
+                list(core_files),
+            )
+        )
+
+    if ignored_files:
+        sections.append(("Ignored Files", list(ignored_files)))
+
+    return sections
+
+
+def log_dirty_files(
+    combined_dirty: Sequence[str],
+    sections: Sequence[Tuple[str, Sequence[str]]],
+) -> None:
+    """Emit dirty-file information in markdown list form."""
+    if sections:
+        print("Dirty files by project:")
+        for idx, (heading, files) in enumerate(sections):
+            print(f"{heading}:")
+            for line in format_bullet_list(files):
+                print(line)
+            if idx != len(sections) - 1:
+                print()
+        print()
+
+    print("All dirty files:")
+    if combined_dirty:
+        for line in format_bullet_list(combined_dirty):
+            print(line)
+    else:
+        print("  - (none)")
+    print()
+
+
+def determine_rebuild_sets(
+    config: Config,
+    project_dirty_map: Dict[str, List[str]],  # key -> dirty files
+    reverse_graph: Dict[
+        str, List[Tuple[str, bool]]
+    ],  # dependency -> [(dependent, requires_full_rebuild)]
+) -> Tuple[
+    Dict[str, str], set[str], set[str]
+]:  # (project_statuses, full_set, lite_set)
+    """Return project statuses plus the full/lite rebuild sets."""
+    core_dirty_files = project_dirty_map[CORE_PROJECT_KEY]
+
+    if core_dirty_files:
+        project_statuses: Dict[str, str] = {key: "Dirty" for key in config.project_keys}
+        return project_statuses, set(config.project_keys), set()
+
+    initially_dirty = [
+        key
+        for key in config.project_keys
+        if key != CORE_PROJECT_KEY and project_dirty_map.get(key)
+    ]
+    for key in initially_dirty:
+        print(f"- Changes detected in subproject '{key}' ({config.project(key).name})")
+
+    full_set, lite_set = propagate_dirty_projects(
+        config,
+        initially_dirty,
+        reverse_graph,
+    )
+
+    project_statuses: Dict[str, str] = {}
+    for key in config.project_keys:
+        if key in full_set:
+            project_statuses[key] = "Dirty"
+        elif key in lite_set:
+            project_statuses[key] = "Dirty Deps"
+        else:
+            project_statuses[key] = "Clean"
+
+    return project_statuses, full_set, lite_set
+
+
+def compute_outputs(
+    config: Config,
+    full_set: set[str],
+    lite_set: set[str],
+) -> Tuple[List[str], List[str]]:  # (FULL_BUILD, LITE_BUILD) matrix_project lists
+    """Convert rebuild sets into ordered matrix project lists."""
+    full_output = [
+        config.project(key).matrix_project
+        for key in config.project_keys
+        if key in full_set and config.project(key).matrix_project
+    ]
+    lite_output = [
+        config.project(key).matrix_project
+        for key in config.project_keys
+        if key in lite_set and config.project(key).matrix_project
+    ]
+    return full_output, lite_output
+
+
+def emit_outputs(full_output: Sequence[str], lite_output: Sequence[str]) -> None:
+    """Write FULL_BUILD/LITE_BUILD strings to stdout and GitHub outputs."""
+    print("Github Action Outputs:")
+    write_output("FULL_BUILD", " ".join(full_output))
+    write_output("LITE_BUILD", " ".join(lite_output))
+    print()
+
+
+def write_project_summary(
+    config: Config,
+    project_statuses: Dict[str, str],
+    writer: SummaryWriter,
+) -> None:
+    """Render the status table inside the summary section."""
+    name_width = max(len(config.project(key).name) for key in config.project_keys)
+    writer.log(f"| {'Project':<{name_width}} | Status     |")
+    writer.log(f"|{'-' * (name_width + 2)}|------------|")
+    for key in config.project_keys:
+        project = config.project(key)
+        status = project_statuses.get(key, "Unknown?")
+        writer.log(f"| {project.name:<{name_width}} | {status:<10} |")
+    writer.log()
+
+
+def write_summary_dirty_sections(
+    combined_dirty: Sequence[str],
+    sections: Sequence[Tuple[str, Sequence[str]]],
+    writer: SummaryWriter,
+) -> None:
+    """Render dirty-file details inside the summary section."""
+    writer.log("<details><summary><h4>👉 Dirty Files</h4></summary>")
+    if sections:
+        for idx, (heading, files) in enumerate(sections):
+            writer.log()
+            writer.log(f"{heading}:")
+            for line in format_bullet_list(files):
+                writer.log(line)
+
+    writer.log()
+    if combined_dirty:
+        writer.log("All dirty files:")
+        for line in format_bullet_list(combined_dirty):
+            writer.log(line)
+
+    writer.log()
+    writer.log("</details>")
+
+
+def main(argv: Optional[Sequence[str]] = None) -> int:
+    """Entrypoint used by the GitHub Action wrapper."""
+    args = parse_args(argv)
+    config = load_config(CONFIG_PATH)
+
+    summary_path = Path(args.summary) if args.summary else None
+
+    dirty_files, ignored_files = collect_dirty_files(args, config)
+
+    project_dirty_map = build_project_dirty_map(config, dirty_files)
+    reverse_graph = build_reverse_dependency_graph(config)
+
+    print(f"Subprojects: {' '.join(config.project_keys)}")
+    print()
+
+    log_dependency_overview(config)
+
+    combined_dirty = dirty_files + ignored_files
+    sections = build_dirty_sections(config, project_dirty_map, ignored_files)
+    log_dirty_files(combined_dirty, sections)
+
+    print("Checking for changes...")
+
+    project_statuses, full_set, lite_set = determine_rebuild_sets(
+        config,
+        project_dirty_map,
+        reverse_graph,
+    )
+
+    full_output, lite_output = compute_outputs(config, full_set, lite_set)
+
+    print()
+    emit_outputs(full_output, lite_output)
+
+    with SummaryWriter(summary_path) as summary_writer:
+        print("::group::Project Change Summary")
+        summary_writer.log(
+            "<details><summary><h3>👃 Inspect Project Changes</h3></summary>"
+        )
+        summary_writer.log()
+        write_project_summary(config, project_statuses, summary_writer)
+        write_summary_dirty_sections(combined_dirty, sections, summary_writer)
+        summary_writer.log("</details>")
+        print("::endgroup::")
+
+    return 0
+
+
+if __name__ == "__main__":  # pragma: no cover
+    sys.exit(main())
diff --git a/cccl_upstream/ci/install_cccl.sh b/cccl_upstream/ci/install_cccl.sh
new file mode 100755
index 00000000..41eddadd
--- /dev/null
+++ b/cccl_upstream/ci/install_cccl.sh
@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+
+set -eo pipefail
+
+target_dir=$(realpath "$1")
+mkdir -p "$target_dir"
+
+# Move script to the root directory of the project
+cd "$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )";
+
+source ./pretty_printing.sh
+
+# Check if the correct number of arguments has been provided
+function usage {
+    echo "Usage: $0 [OPTIONS] dir"
+    echo
+    echo "Installs CCCL to the provided directory"
+
+    echo "Options:"
+    echo "  -v/-verbose: Enable shell echo for debugging"
+    echo
+    echo "Examples:"
+    echo "  $ $0 ~/my/prefix"
+    exit 1
+}
+
+while [[ "$#" -gt 0 ]]; do
+    case "$1" in
+        --verbose) VERBOSE=true; ;;
+        -v) VERBOSE=true; ;;
+        *) break ;;
+    esac
+    shift
+done
+
+if [[ -n "$VERBOSE" ]]; then
+    set -x
+fi
+
+# Move to cccl/ dir
+pushd ".." > /dev/null
+GROUP_NAME="🛠️  CMake Configure CCCL - Install"
+run_command "$GROUP_NAME" cmake -G "Unix Makefiles" --preset install -DCMAKE_INSTALL_PREFIX="${target_dir}"
+status=$?
+
+GROUP_NAME="🏗️  Install CCCL"
+run_command "$GROUP_NAME" cmake --build --preset install --target install
+status=$?
+popd > /dev/null
diff --git a/cccl_upstream/ci/install_packaging.sh b/cccl_upstream/ci/install_packaging.sh
new file mode 100755
index 00000000..d8b257ef
--- /dev/null
+++ b/cccl_upstream/ci/install_packaging.sh
@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+# Time limit for installation steps (seconds)
+readonly install_time_limit=20
+
+ci_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+source "$ci_dir/pretty_printing.sh"
+
+cd "$ci_dir/.."
+
+# Prefixes for installations
+prefix_default=$(mktemp -d /tmp/cccl-default-XXXX)
+prefix_preset=$(mktemp -d /tmp/cccl-preset-XXXX)
+prefix_script=$(mktemp -d /tmp/cccl-script-XXXX)
+prefix_preset_unstable=$(mktemp -d /tmp/cccl-preset-unstable-XXXX)
+prefix_preset_unstable_only=$(mktemp -d /tmp/cccl-preset-unstable-only-XXXX)
+
+# Default configure + install
+default_start_time=$SECONDS
+run_command "Configure default" \
+  cmake -S . -B build/default -DCMAKE_INSTALL_PREFIX="$prefix_default"
+run_command "Install default" \
+  cmake --build build/default --target install
+default_time=$((SECONDS - default_start_time))
+
+# Preset configure + install
+preset_start_time=$SECONDS
+CCCL_BUILD_INFIX=preset \
+run_command "Configure preset" \
+  cmake --preset install -DCMAKE_INSTALL_PREFIX="$prefix_preset"
+CCCL_BUILD_INFIX=preset \
+run_command "Install preset" \
+  cmake --build --preset install --target install
+preset_time=$((SECONDS - preset_start_time))
+
+# Preset configure + install-unstable
+preset_unstable_start_time=$SECONDS
+CCCL_BUILD_INFIX=preset_unstable \
+run_command "Configure preset-unstable" \
+  cmake --preset install-unstable -DCMAKE_INSTALL_PREFIX="$prefix_preset_unstable"
+CCCL_BUILD_INFIX=preset_unstable \
+run_command "Install preset-unstable" \
+  cmake --build build/preset_unstable/install-unstable --target install
+preset_unstable_time=$((SECONDS - preset_unstable_start_time))
+
+# Preset configure + install-unstable-only
+preset_unstable_only_start_time=$SECONDS
+CCCL_BUILD_INFIX=preset_unstable_only \
+run_command "Configure preset-unstable-only" \
+  cmake --preset install-unstable-only -DCMAKE_INSTALL_PREFIX="$prefix_preset_unstable_only"
+CCCL_BUILD_INFIX=preset_unstable_only \
+run_command "Install preset-unstable-only" \
+  cmake --build build/preset_unstable_only/install-unstable-only \
+        --target install
+preset_unstable_only_time=$((SECONDS - preset_unstable_only_start_time))
+
+# Script install
+script_start_time=$SECONDS
+CCCL_BUILD_INFIX=script \
+run_command "Install script" \
+  ci/install_cccl.sh "$prefix_script"
+script_time=$((SECONDS - script_start_time))
+
+# Compare directories
+if ! diff -ruN "$prefix_default" "$prefix_preset" > /tmp/diff_default_preset.log; then
+  cat /tmp/diff_default_preset.log
+  echo -e "\e[0;31mDefault and preset installations differ\e[0m"
+  exit 1
+fi
+if ! diff -ruN "$prefix_default" "$prefix_script" > /tmp/diff_default_script.log; then
+  cat /tmp/diff_default_script.log
+  echo -e "\e[0;31mDefault and script installations differ\e[0m"
+  exit 1
+fi
+
+# Verify unstable-only contains only cudax headers and cmake config
+if find "$prefix_preset_unstable_only" -type f | \
+    sed "s|$prefix_preset_unstable_only/||" | \
+    grep -vE '^include/cuda/experimental/|^lib/cmake/+cudax/|^lib/cmake/+cccl/' | \
+    grep -q .; then
+  echo -e "\e[0;31minstall-unstable-only contained unexpected files\e[0m" >&2
+  exit 1
+fi
+
+# Verify install-unstable is union of install and install-unstable-only
+prefix_union=$(mktemp -d /tmp/cccl-union-XXXX)
+cp -a "$prefix_default/." "$prefix_union/"
+cp -a "$prefix_preset_unstable_only/." "$prefix_union/"
+if ! diff -ruN "$prefix_union" "$prefix_preset_unstable" > /tmp/diff_union_unstable.log; then
+  echo -e "\e[0;31mUnion of install and install-unstable-only did not match install-unstable\e[0m" >&2
+  cat /tmp/diff_union_unstable.log
+  exit 1
+fi
+
+print_time_summary
+
+# Verify times
+violations=()
+(( default_time > install_time_limit )) && violations+=("default_time=${default_time}s")
+(( preset_time > install_time_limit )) && violations+=("preset_time=${preset_time}s")
+(( preset_unstable_time > install_time_limit )) && violations+=("preset_unstable_time=${preset_unstable_time}s")
+(( preset_unstable_only_time > install_time_limit )) && violations+=("preset_unstable_only_time=${preset_unstable_only_time}s")
+(( script_time > install_time_limit )) && violations+=("script_time=${script_time}s")
+
+if (( ${#violations[@]} > 0 )); then
+  echo -e "\e[0;31mInstallation time limit (${install_time_limit}s) exceeded by: ${violations[*]}\e[0m" >&2
+  exit 1
+fi
+
+# Clean up
+rm -rf "$prefix_default" "$prefix_preset" "$prefix_script" \
+       "$prefix_preset_unstable" "$prefix_preset_unstable_only" \
+       "$prefix_union"
+
+echo -e "\e[0;32mAll installation tests passed successfully.\e[0m"
diff --git a/cccl_upstream/ci/matrix.yaml b/cccl_upstream/ci/matrix.yaml
new file mode 100644
index 00000000..5ea7a70b
--- /dev/null
+++ b/cccl_upstream/ci/matrix.yaml
@@ -0,0 +1,816 @@
+compile_time:
+  pull_request:
+    - id: public-headers-gcc13
+      name: Public headers compile-time bench
+      gpu: rtx2080
+      launch_args: "--cuda 13.3 --host gcc13"
+      baseline_ref: origin/main
+      preset: all-dev
+      targets:
+        - cub.headers.base
+        - thrust.cpp.cuda.headers.base
+        - libcudacxx.test.public_headers
+      args: "-arch native"
+      slices:
+        - id: total-compilation
+          title: TU total compilation
+          filter: total-compilation
+          timing: inclusive
+          sort: total
+          top: 15
+          threshold: 3.0
+        - id: file-processing
+          title: Direct file processing
+          filter: file-processing
+          timing: exclusive
+          sort: total
+          top: 15
+          threshold: 0.2
+        - id: function-bodies
+          title: Function-body parsing
+          filter: scanning-function-body
+          timing: inclusive
+          sort: total
+          top: 15
+          threshold: 0.25
+        - id: template-instantiation
+          title: Template instantiation
+          filter: template-instantiation
+          timing: inclusive
+          sort: total
+          top: 15
+          threshold: 0.2
+
+workflows:
+  # If any jobs appear here, they will be executed instead of `pull_request' for PRs.
+  # This is useful for limiting resource usage when a full matrix is not needed.
+  # The branch protection checks will fail when using this override workflow.
+  #
+  # Example:
+  # override:
+  #   # Full project build: slow, expensive
+  #   - { jobs: ['test'], project: 'thrust', std: 17, ctk: '12.X', cxx: ['gcc12', 'clang16'] }
+  #
+  #   # Build / run targeted tests: faster turnaround, less runner usage.
+  #   # Use project 'target'.
+  #   # args are passed to ci/util/build_and_test_targets.sh. See that script for available options.
+  #   - { jobs: ['run_cpu'], project: 'target', ctk: ['12.X', '13.X'], cxx: ['gcc', 'clang', 'msvc'],
+  #       args: '--preset cub-cpp20 --build-targets "cub.cpp20.test.iterator"' }
+  #   - { jobs: ['run_gpu'], project: 'target', ctk: ['12.X', '13.X'], cxx: ['gcc', 'clang'], gpu: 'rtxa6000',
+  #       args: '--preset cub-cpp20 --build-targets "cub.cpp20.test.iterator" --ctest-targets "cub.cpp20.test.iterator"' }
+  #   - { jobs: ['run_cpu'], project: 'target', ctk: ['12.X', '13.X'], cxx: ['gcc', 'clang', 'msvc'],
+  #       args: '--preset libcudacxx --lit-precompile-tests "cuda/utility/basic_any.pass.cpp"' }
+  #   - { jobs: ['run_gpu'], project: 'target', ctk: ['12.X', '13.X'], cxx: ['gcc', 'clang'], gpu: 'rtx2080',
+  #       args: '--preset libcudacxx --lit-tests "cuda/utility/basic_any.pass.cpp"' }
+  #
+  # IMPORTANT: Do NOT delete or remove the `override:` key below, even when it is empty.
+  override:
+
+  pull_request:
+    # Old CTK: Oldest/newest supported host compilers:
+    - {jobs: ['build'], std: 'minmax', ctk: '12.0', cxx: ['gcc12', 'clang14',            'msvc2019', 'msvc14.39']}
+    - {jobs: ['build'], project: ['libcudacxx', 'thrust'], std: 'minmax', ctk: '12.0', cxx: 'gcc7'}
+    - {jobs: ['build_nolid', 'build_lid1', 'build_lid2'], project: 'cub', std: 'minmax', ctk: '12.0', cxx: 'gcc7'}
+    # CTK12.0/GCC7 CUB host-launch builds are memory-heavy with benchmarks enabled; keep this shard below
+    # the 61 GiB linux-amd64-cpu16 runner limit.
+    - {jobs: ['build_lid0'], project: 'cub', std: 'minmax', ctk: '12.0', cxx: 'gcc7', environment: ['PARALLEL_LEVEL=8']}
+    - {jobs: ['build'], std: 'minmax', ctk: '12.X', cxx: ['gcc7',  'gcc14',   'clang14', 'clang19', 'msvc2019', 'msvc2022' ]}
+    - {jobs: ['build'], std: 'minmax', ctk: '13.0', cxx: ['gcc11', 'gcc15',   'clang15', 'clang20', 'msvc2019', 'msvc2022' ]}
+    # Old CTK: cudax has a different support matrix:
+    - {jobs: ['build'], project: 'cudax', ctk: '12.0', std: 'minmax', cxx: ['gcc9',  'gcc12', 'clang14',            'msvc14.39']}
+    - {jobs: ['build'], project: 'cudax', ctk: '12.X', std: 'minmax', cxx: ['gcc9',  'gcc14',   'clang14', 'clang19', 'msvc2022']}
+    - {jobs: ['build'], project: 'cudax', ctk: '13.0', std: 'minmax', cxx: ['gcc11', 'gcc15',   'clang15', 'clang20', 'msvc2022']}
+    # Current CTK build-only:
+    - {jobs: ['build'], std: 'minmax', cxx: ['gcc11', 'clang15', 'msvc2019'] } # Oldest
+    - {jobs: ['build'], std: 'max',    cxx: ['gcc12', 'gcc13', 'gcc14'] }
+    - {jobs: ['build'], std: 'max',    cxx: ['clang16', 'clang17', 'clang18', 'clang19', 'clang20'] }
+    - {jobs: ['build'], std: 'max',    cxx: ['msvc2022'] }
+    - {jobs: ['build'], std: 'all',    cxx: ['gcc', 'clang', 'msvc']} # Latest
+    # Current CTK build-only: cudax has a different support matrix:
+    - {jobs: ['build'], project: 'cudax', std: 'minmax', cxx: ['gcc11', 'clang15', 'msvc2022']} # Oldest
+    - {jobs: ['build'], project: 'cudax', std: 'max',    cxx: ['gcc12']}
+    - {jobs: ['build'], project: 'cudax', std: 'max',    cxx: ['clang16', 'clang17', 'clang18', 'clang19', 'clang20']}
+    - {jobs: ['build'], project: 'cudax', std: 'all',    cxx: ['gcc', 'clang', 'msvc']} # Newest
+    # Current CTK testing:
+    - {jobs: ['test'], project: 'thrust',     std: 'max', cxx: ['gcc', 'clang', 'msvc'], gpu: 'rtx4090'}
+    - {jobs: ['test'], project: ['libcudacxx', 'cudax'], std: 'max', cxx: ['gcc', 'clang', 'msvc'], gpu: 't4'}
+
+    - {jobs: ['test_nolid', 'test_lid0'], project: 'cub', std: 'max', cxx: ['gcc', 'clang', 'msvc'], gpu: 'rtxa6000'}
+    - {jobs: ['test_lid1',  'test_lid2'], project: 'cub', std: 'max', cxx: ['gcc'],                  gpu: 'rtxa6000'}
+    # H100 coverage:
+    - {jobs: ['test_nolid', 'test_lid0'], project: 'cub',                   std: 'max', gpu: 'h100' }
+    - {jobs: ['test_gpu'],                project: 'thrust',                std: 'max', gpu: 'h100' }
+    - {jobs: ['test'],                    project: ['libcudacxx', 'cudax'], std: 'max', gpu: 'h100' }
+    # Multi-GPU coverage:
+    - {jobs: ['test'], project: ['libcudacxx', 'cudax'], std: 'max', gpu: 'h100_2gpu', sm: 'gpu'}
+    # RTX PRO 6000 coverage (limited due to small number of runners):
+    - {jobs: ['test_lid0'], project: 'cub', std: 'max', cxx: 'gcc', gpu: 'rtxpro6000'}
+    # Misc:
+    - {jobs: ['build'], cpu: 'arm64', project: ['libcudacxx', 'cub', 'thrust', 'cudax'], std: 'max', cxx: ['gcc', 'clang']}
+    - {jobs: ['test_gpu'], project: 'thrust', cmake_options: '-DTHRUST_DISPATCH_TYPE=Force32bit', gpu: 'rtx4090'}
+    - {jobs: ['nvrtc'], project: 'libcudacxx', std: 'all', gpu: 'rtx2080', sm: 'gpu'}
+    - {jobs: ['verify_codegen'], project: 'libcudacxx'}
+    # c.parallel -- pinned to gcc13 / msvc2022 to match python
+    - {jobs: ['test'], project: 'cccl_c_parallel', ctk: '12.X', cxx: ['gcc13', 'msvc2022'], gpu: ['t4']}
+    - {jobs: ['test'], project: 'cccl_c_parallel', ctk: '13.X', cxx: ['gcc13', 'msvc2022'], gpu: ['rtx2080', 'l4', 'h100']}
+    # RTX PRO 6000 coverage (limited due to small number of runners):
+    - {jobs: ['test'], project: 'cccl_c_parallel', ctk: '13.X', cxx: ['gcc13'], gpu: ['rtxpro6000']}
+    # c.parallel v2 (HostJIT-based)
+    #
+    # For now, this is a separate job run for Linux/CUDA13.
+    # Eventually v2 will replace v1 as the default and run across the
+    # entire matrix. Currently blocked on libnvfatbin availability on
+    # Windows containers, and for CUDA <12.4.
+    - {jobs: ['test'], project: 'cccl_c_parallel_v2', ctk: '13.X', cxx: ['gcc13', 'msvc'], gpu: 'rtx2080'}
+    # Python against c.parallel v2 (HostJIT-based). Single point of coverage
+    # for the v2 Python path; the main `python` matrix continues to test
+    # against v1 until v2 replaces it.
+    - {jobs: ['test'], project: 'python_v2', ctk: '13.X', py_version: '3.14', gpu: 'l4', cxx: 'gcc13'}
+    - {jobs: ['test_py_compute_minimal'], project: 'python_v2', ctk: '13.X', py_version: '3.14t', gpu: 'l4', cxx: 'gcc13'}
+    # c.experimental.stf-- pinned to gcc13 to match python
+    - {jobs: ['test'], project: 'cccl_c_stf', ctk: ['12.X', '13.X'], cxx: 'gcc13', gpu: 't4'}
+    - {jobs: ['test'], project: 'cccl_c_stf', ctk: '13.X', cxx: 'gcc13', gpu: ['l4', 'h100']}
+    # Python -- pinned to gcc13 / msvc2022 for consistency across CTK images
+    - {jobs: ['test'], project: 'python', ctk: ['12.X',        '13.X'], py_version: ['3.10'], gpu: 'l4', cxx: ['gcc13', 'msvc2022']}
+    - {jobs: ['test'], project: 'python', ctk: ['12.0', '12.X','13.0', '13.X'], py_version: '3.14', gpu: 'l4', cxx: ['gcc13', 'msvc2022']}
+    - {jobs: ['test'], project: 'python', py_version: '3.14', gpu: 'h100', cxx: 'gcc13'}
+    - {jobs: ['test'], project: 'python', ctk: '13.X', py_version: '3.14', gpu: ['t4', 'rtxa6000', 'rtxpro6000'], cxx: 'gcc13'}
+    - {jobs: ['test_py_compute_minimal'], project: 'python', ctk: '13.X', py_version: '3.14', gpu: 'l4', cxx: 'gcc13'}
+    - {jobs: ['test_py_compute_minimal'], project: 'python', ctk: ['12.0', '12.X', '13.0', '13.X'], py_version: '3.14t', gpu: 'l4', cxx: ['gcc13', 'msvc2022']}
+    - {jobs: ['test_py_compute_minimal'], project: 'python_tsan', ctk: '13.X', py_version: '3.14t', gpu: 'l4', cxx: 'gcc13'}
+    # Deliberately unpinned: py_ctk_mode 'latest' skips the CTK pin so pip
+    # resolves the latest minor -- catching breakage a plain `pip install
+    # cuda-cccl[cu12]/[cu13]` would hit before the container CTK bumps.
+    - {jobs: ['test'], project: 'python', ctk: ['12.X', '13.X'], py_version: '3.14', gpu: 'l4', cxx: ['gcc13', 'msvc2022'], py_ctk_mode: 'latest'}
+    # sysctk: install the sysctk extras (system-provided CTK, no pip cuda-toolkit)
+    # rather than cu*.
+    # Full test on 3.14, minimal (numba-free) on 3.14t; all CTKs, Linux+Windows.
+    - {jobs: ['test'], project: 'python', ctk: ['12.0', '12.X', '13.0', '13.X'], py_version: '3.14', gpu: 'l4', cxx: ['gcc13', 'msvc2022'], py_ctk_mode: 'sysctk'}
+    - {jobs: ['test_py_compute_minimal'], project: 'python', ctk: ['12.0', '12.X', '13.0', '13.X'], py_version: '3.14t', gpu: 'l4', cxx: ['gcc13', 'msvc2022'], py_ctk_mode: 'sysctk'}
+    # cuda.cccl.headers: CPU-only (gpu:false), arch-insensitive
+    - {jobs: ['test_headers'], project: 'python', ctk: ['12.X', '13.X'], py_version: '3.14', cxx: ['gcc13', 'msvc2022'], py_ctk_mode: ['pinned', 'latest', 'sysctk']}
+    # CCCL packaging:
+    - {jobs: ['test'], project: 'packaging', ctk: '12.0', cxx: ['gcc10', 'clang14'], gpu: 't4', args: '-min-cmake'}
+    - {jobs: ['test'], project: 'packaging', ctk: '12.X', cxx: ['gcc10', 'clang14'], gpu: 't4'}
+    - {jobs: ['test'], project: 'packaging', ctk: '13.0', cxx: ['gcc15', 'clang20'], gpu: 't4', args: '-min-cmake'}
+    - {jobs: ['test'], project: 'packaging', ctk: '13.X', cxx: ['gcc',   'clang'],   gpu: 't4'}
+    - {jobs: ['install'], project: 'packaging'}
+    # NVBench Helper testing:
+    - {jobs: ['test'], project: 'nvbench_helper', ctk: ['12.0', '12.X'], cxx: ['gcc10', 'clang14'], gpu: 't4'}
+    - {jobs: ['test'], project: 'nvbench_helper', ctk: '13.0', cxx: ['gcc15', 'clang20'], gpu: 't4'}
+    - {jobs: ['test'], project: 'nvbench_helper', ctk: '13.X', cxx: ['gcc',   'clang'],   gpu: 't4'}
+    # NVHPC build
+    - {jobs: ['build'], cxx: 'nvhpc', ctk: 'nvhpc', std: 'max', project: ['libcudacxx', 'thrust', 'stdpar'], cpu: 'amd64'}
+    - {jobs: ['build_nolid'], cxx: 'nvhpc', ctk: 'nvhpc', std: 'max', project: 'cub', cpu: 'amd64'}
+    # clang-cuda
+    - {jobs: ['build'], cudacxx: 'clang', ctk: 'clang-cuda', cxx: 'clang-cuda',  std: 'all', sm: '75;80;90;100;120'}
+    - {jobs: ['build'], project: 'libcudacxx', cudacxx: 'clang', ctk: 'clang-cuda', cxx: 'clang-cuda',  std: 23, sm: '75;80;90;100;120'}
+    # libc++
+    # - arm64 for now as it's closest to android.
+    # - {jobs: ['build'], cpu: 'arm64', project: 'libcudacxx', std: 'all', cudacxx: 'clang', ctk: 'clang-cuda', cxx: 'clang-cuda', cmake_options: '-DCCCL_USE_LIBCXX=ON', sm: '75;80;90;100;120'}
+    # clang-tidy
+    #
+    # The precise value of sm is not important, but it is required for cmake to identify
+    # clang as a CUDA compiler (see
+    # https://discourse.cmake.org/t/cmake-cuda-clang-fails/8657/5).
+    #
+    # Standard being exactly "min" is required. clang-tidy may emit additional diagnostics
+    # for later C++ versions (for example, warning that you should use designated
+    # initializers in C++20 or higher).
+    - { jobs: ['build'], project: 'tidy', std: 'min', cxx: ['clang'], cudacxx: ['clang'], ctk: 'clang-cuda', sm: '75' }
+
+  # Used when an upstream project changes to reduce time spent smoke testing dependencies.
+  pull_request_lite:
+    # libcudacxx - Specialized, testing default SM
+    - {project: 'libcudacxx', jobs: ['test'],  std: 'max', cxx: ['gcc', 'msvc'], gpu: 'rtx2080', sm: 'gpu'}
+    - {project: 'libcudacxx', jobs: ['build'], std: 'max', cxx: 'clang'}
+    - {project: 'libcudacxx', jobs: ['build'], std: 'max', ctk: 'nvhpc', cxx: 'nvhpc'}
+    - {project: 'libcudacxx', jobs: ['build'], std: 'max', cudacxx: 'clang', ctk: 'clang-cuda', cxx: 'clang-cuda', sm: '70;80;90;100;120'}
+    - {project: 'libcudacxx', jobs: ['nvrtc'], std: 'max', gpu: 't4', sm: 'gpu'}
+    - {project: 'libcudacxx', jobs: ['verify_codegen']}
+    # CUB - Specialized, testing default SM
+    - {project: 'cub', jobs: ['test_nolid', 'test_lid0'], std: 'max', cxx: ['gcc', 'msvc'], gpu: 'rtxa6000', sm: 'gpu'}
+    - {project: 'cub', jobs: ['build_nolid', 'build_lid0'], std: 'max', cxx: 'clang'}
+    - {project: 'cub', jobs: ['build_nolid', 'build_lid0'], std: 'max', ctk: 'nvhpc', cxx: 'nvhpc'}
+    - {project: 'cub', jobs: ['build_nolid', 'build_lid0'], std: 'max', cudacxx: 'clang', ctk: 'clang-cuda', cxx: 'clang-cuda', sm: '75;80;90;100;120'}
+    # Thrust - Keep number of sm small. Kernel coverage is in CUB. This just tests dispatch / glue in lite mode:
+    - {project: 'thrust', jobs: ['test'], std: 'max', cxx: ['gcc', 'msvc'], gpu: 'rtx4090', sm: 'gpu'}
+    - {project: 'thrust', jobs: ['build'], std: 'max', cxx: 'clang', sm: '75;120'}
+    - {project: 'thrust', jobs: ['build'], std: 'max', ctk: 'nvhpc', cxx: 'nvhpc', sm: '75;120'}
+    - {project: 'thrust', jobs: ['build'], std: 'max', cudacxx: 'clang', ctk: 'clang-cuda', cxx: 'clang-cuda', sm: '75;120'}
+    # cudax
+    - {project: 'cudax', jobs: ['test'], std: 'max', cxx: ['gcc', 'msvc'], gpu: 'rtx2080', sm: 'gpu'}
+    - {project: 'cudax', jobs: ['build'], std: 'max', cxx: 'clang', sm: '75;120'}
+    - {project: 'cudax', jobs: ['build'], std: 'max', ctk: 'nvhpc', cxx: 'nvhpc', sm: '75;120'}
+    # stdpar
+    - {project: 'stdpar', jobs: ['build'], std: 'max', ctk: 'nvhpc', cxx: 'nvhpc'}
+    # Python + support
+    - {project: 'cccl_c_parallel', jobs: ['test'], ctk: '13.X', cxx: ['gcc13', 'msvc2022'], gpu: 'rtx2080', sm: 'gpu'}
+    - {project: 'cccl_c_parallel', jobs: ['test'], ctk: '13.X', cxx: 'gcc13', gpu: 'rtxpro6000', sm: 'gpu'}
+    - {project: 'cccl_c_stf',      jobs: ['test'], ctk: '13.X', cxx: 'gcc13',               gpu: 't4', sm: 'gpu'}
+    - {project: 'python', jobs: ['test'], ctk: '13.X', py_version: '3.14', gpu: 'l4', cxx: ['gcc13', 'msvc2022']}
+    - {project: 'python', jobs: ['test_headers'], ctk: '13.X', py_version: '3.14', cxx: ['gcc13', 'msvc2022']}
+    # Packaging / install
+    - {project: 'packaging', jobs: ['test'], ctk: '13.X', cxx: ['gcc', 'clang'], gpu: 'rtx2080', sm: 'gpu'}
+    - {project: 'packaging', jobs: ['test'], args: '-min-cmake', gpu: 't4', sm: 'gpu'}
+    - {project: 'packaging', jobs: ['install']}
+    # NVBench Helper testing:
+    - {project: 'nvbench_helper', jobs: ['test'], ctk: '13.X', cxx: ['gcc', 'clang'], gpu: 'rtx2080'}
+    # c.parallel v2 (HostJIT-based)
+    - {jobs: ['test'], project: 'cccl_c_parallel_v2', ctk: '13.X', cxx: ['gcc13'], gpu: 'rtx2080'}
+    # Python against c.parallel v2 (HostJIT-based)
+    - {jobs: ['test'], project: 'python_v2', ctk: '13.X', py_version: '3.14', gpu: 'l4', cxx: 'gcc13'}
+
+  nightly:
+    # CTK 12.0 full matrix build: default projects
+    - {jobs: ['build'], std: 'all', ctk: '12.0', cxx: ['gcc7', 'gcc8', 'gcc9', 'gcc10', 'gcc11', 'gcc12']}
+    - {jobs: ['build'], std: 'all', ctk: '12.0', cxx: ['clang14']}
+    - {jobs: ['build'], std: 'all', ctk: '12.0', cxx: ['msvc2019', 'msvc14.39']}
+    # CTK 12.X full matrix build: default projects
+    - {jobs: ['build'], std: 'all', ctk: '12.X', cxx: ['gcc7', 'gcc8', 'gcc9', 'gcc10', 'gcc11', 'gcc12', 'gcc13', 'gcc14']}
+    - {jobs: ['build'], std: 'all', ctk: '12.X', cxx: ['clang14', 'clang15', 'clang16', 'clang17', 'clang18', 'clang19']}
+    - {jobs: ['build'], std: 'all', ctk: '12.X', cxx: ['msvc2019', 'msvc2022']}
+    # CTK 13.0 full matrix build: default projects
+    - {jobs: ['build'], std: 'all', ctk: '13.0', cxx: ['gcc11', 'gcc12', 'gcc13', 'gcc14', 'gcc15']}
+    - {jobs: ['build'], std: 'all', ctk: '13.0', cxx: ['clang15', 'clang16', 'clang17', 'clang18', 'clang19', 'clang20']}
+    - {jobs: ['build'], std: 'all', ctk: '13.0', cxx: ['msvc2019', 'msvc2022']}
+    # CTK '13.X' full matrix build: default projects
+    - {jobs: ['build'], std: 'all', ctk: '13.X', cxx: ['gcc11', 'gcc12', 'gcc13', 'gcc14', 'gcc15']}
+    - {jobs: ['build'], std: 'all', ctk: '13.X', cxx: ['clang15', 'clang16', 'clang17', 'clang18', 'clang19', 'clang20', 'clang21']}
+    - {jobs: ['build'], std: 'all', ctk: '13.X', cxx: ['msvc2019', 'msvc2022', 'msvc2026']}
+    # CTK 12.0 full matrix build: cudax
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '12.0', cxx: ['gcc9', 'gcc10', 'gcc11', 'gcc12']}
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '12.0', cxx: ['clang14']}
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '12.0', cxx: ['msvc14.39']}
+    # CTK 12.X full matrix build: cudax
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '12.X', cxx: ['gcc9', 'gcc10', 'gcc11', 'gcc12', 'gcc13', 'gcc14']}
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '12.X', cxx: ['clang14', 'clang15', 'clang16', 'clang17', 'clang18', 'clang19']}
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '12.X', cxx: ['msvc2022']}
+    # CTK 13.0 full matrix build: cudax
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '13.0', cxx: ['gcc11', 'gcc12', 'gcc13', 'gcc14', 'gcc15']}
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '13.0', cxx: ['clang15', 'clang16', 'clang17', 'clang18', 'clang19', 'clang20']}
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '13.0', cxx: ['msvc2022']}
+    # CTK '13.X' full matrix build: cudax
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '13.X', cxx: ['gcc11', 'gcc12', 'gcc13', 'gcc14', 'gcc15']}
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '13.X', cxx: ['clang15', 'clang16', 'clang17', 'clang18', 'clang19', 'clang20', 'clang21']}
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '13.X', cxx: ['msvc2022', 'msvc2026']}
+    # CTK 12.X testing:
+    - {jobs: ['test'], project: 'libcudacxx', ctk: '12.X', std: 'max', cxx: ['gcc14', 'clang19', 'msvc2022'], gpu: 't4'}
+    - {jobs: ['test'], project: 'cub',        ctk: '12.X', std: 'max', cxx: ['gcc14', 'clang19', 'msvc2022'], gpu: 'rtxa6000'}
+    - {jobs: ['test'], project: 'thrust',     ctk: '12.X', std: 'max', cxx: ['gcc14', 'clang19', 'msvc2022'], gpu: 'rtx4090'}
+    - {jobs: ['test'], project: 'cudax',      ctk: '12.X', std: 'max', cxx: ['gcc14', 'clang19', 'msvc2022'], gpu: 'rtx2080'}
+    - {jobs: ['test'], project: ['libcudacxx', 'cub', 'thrust', 'cudax'], ctk: '12.X', std: 'max', cxx: 'gcc14', gpu: 'h100' }
+    # CTK '13.X' testing:
+    - {jobs: ['test'], project: 'libcudacxx', ctk: '13.X', std: 'max', cxx: ['gcc', 'clang', 'msvc'], gpu: 'rtx2080'}
+    - {jobs: ['test'], project: 'cub',        ctk: '13.X', std: 'max', cxx: ['gcc', 'clang', 'msvc'], gpu: 'rtxa6000'}
+    - {jobs: ['test'], project: 'thrust',     ctk: '13.X', std: 'max', cxx: ['gcc', 'clang', 'msvc'], gpu: 'rtx4090'}
+    - {jobs: ['test'], project: 'cudax',      ctk: '13.X', std: 'max', cxx: ['gcc', 'clang', 'msvc'], gpu: 't4'}
+    - {jobs: ['test'], project: ['libcudacxx', 'cub', 'thrust', 'cudax'], ctk: '13.X', std: 'max', gpu: 'h100' }
+      # RTX PRO 6000 coverage (limited due to small number of runners):
+    - {jobs: ['test_nolid', 'test_lid0'], project: ['cub', 'thrust'], std: 'max', cxx: 'gcc', gpu: 'rtxpro6000'}
+    # Misc:
+    - {jobs: ['build'], cpu: 'arm64', project: ['libcudacxx', 'cub', 'thrust', 'cudax'], ctk: '12.X', std: 'all', cxx: ['gcc14', 'clang19']}
+    - {jobs: ['build'], cpu: 'arm64', project: ['libcudacxx', 'cub', 'thrust', 'cudax'], ctk: '13.X', std: 'all', cxx: ['gcc', 'clang']}
+    - {jobs: ['test_gpu'], project: 'thrust', cmake_options: '-DTHRUST_DISPATCH_TYPE=Force32bit', gpu: 'rtx4090'}
+    - {jobs: ['test_gpu'], project: 'thrust', cmake_options: '-DTHRUST_DISPATCH_TYPE=Force64bit', gpu: 'rtx4090'}
+    - {jobs: ['limited'], project: 'cub', std: 17, gpu: 'rtx2080'}
+    # NVRTC tests don't currently support 12.0:
+    - {jobs: ['nvrtc'],          project: 'libcudacxx', ctk: [        '12.X', '13.0', '13.X'], cxx: 'gcc12', std: 'all', gpu: 'rtx2080', sm: 'gpu'}
+    - {jobs: ['verify_codegen'], project: 'libcudacxx'}
+    # c.parallel -- pinned to gcc13 / msvc2022 to match python
+    - {jobs: ['test'],  project: ['cccl_c_parallel'], ctk: '12.X', cxx: ['gcc13', 'msvc2022'], gpu: ['t4']}
+    - {jobs: ['test'],  project: ['cccl_c_parallel'], ctk: '13.X', cxx: ['gcc13', 'msvc2022'], gpu: ['rtx2080', 'l4', 'h100']}
+    # RTX PRO 6000 coverage (limited due to small number of runners):
+    - {jobs: ['test'], project: 'cccl_c_parallel', ctk: '13.X', cxx: ['gcc13'], gpu: ['rtxpro6000']}
+    # c.experimental.stf -- pinned to gcc13 to match python
+    - {jobs: ['test'],  project: ['cccl_c_stf'], ctk: '12.X', cxx: 'gcc13', gpu: ['rtx2080']}
+    - {jobs: ['test'],  project: ['cccl_c_stf'], ctk: '13.X', cxx: 'gcc13', gpu: ['t4', 'l4', 'h100']}
+    # Python -- pinned to gcc13 / msvc2022 on Linux for consistency across CTK images
+    - {jobs: ['test'], project: 'python', ctk: ['12.0', '12.X', '13.0', '13.X'], py_version: ['3.10', '3.11', '3.12', '3.13', '3.14'], gpu: 'l4', cxx: ['gcc13', 'msvc2022']}
+    - {jobs: ['test'], project: 'python', ctk: ['12.X', '13.X'], py_version: '3.14', gpu: 'h100', cxx: 'gcc13'}
+    - {jobs: ['test'], project: 'python', ctk: ['12.X', '13.X'], py_version: '3.14', gpu: ['t4', 'rtxa6000'], cxx: 'gcc13'}
+    - {jobs: ['test'], project: 'python', ctk: '13.X', py_version: '3.14', gpu: 'rtxpro6000', cxx: 'gcc13'}
+    # Python free-threaded (3.14t) minimal lanes -- mirrors the pull_request rows
+    # so FT regressions (e.g. from dependency bumps) surface between PRs.
+    - {jobs: ['test_py_compute_minimal'], project: 'python', ctk: ['12.0', '12.X', '13.0', '13.X'], py_version: '3.14t', gpu: 'l4', cxx: ['gcc13', 'msvc2022']}
+    - {jobs: ['test_py_compute_minimal'], project: 'python_v2', ctk: '13.X', py_version: '3.14t', gpu: 'l4', cxx: 'gcc13'}
+    - {jobs: ['test_py_compute_minimal'], project: 'python_tsan', ctk: '13.X', py_version: '3.14t', gpu: 'l4', cxx: 'gcc13'}
+    - {jobs: ['test'], project: 'python', ctk: ['12.0', '12.X', '13.0', '13.X'], py_version: '3.14', gpu: 'l4', cxx: ['gcc13', 'msvc2022'], py_ctk_mode: 'sysctk'}
+    - {jobs: ['test_py_compute_minimal'], project: 'python', ctk: ['12.0', '12.X', '13.0', '13.X'], py_version: '3.14t', gpu: 'l4', cxx: ['gcc13', 'msvc2022'], py_ctk_mode: 'sysctk'}
+    # cuda.cccl.headers (CPU-only): all CTK x source x OS, py endpoints 3.10 + 3.14
+    - {jobs: ['test_headers'], project: 'python', ctk: ['12.0', '12.X', '13.0', '13.X'], py_version: ['3.10', '3.14'], cxx: ['gcc13', 'msvc2022'], py_ctk_mode: ['pinned', 'latest', 'sysctk']}
+    # CCCL packaging:
+    - {jobs: ['test'], project: 'packaging', ctk: '12.0', cxx: ['gcc10', 'clang14'], gpu: 't4', args: '-min-cmake'}
+    - {jobs: ['test'], project: 'packaging', ctk: '12.X', cxx: ['gcc10', 'clang14'], gpu: 'rtx2080'}
+    - {jobs: ['test'], project: 'packaging', ctk: '13.0', cxx: ['gcc15', 'clang20'], gpu: 't4', args: '-min-cmake'}
+    - {jobs: ['test'], project: 'packaging', ctk: '13.X', cxx: ['gcc',   'clang'],   gpu: 'rtx2080'}
+    - {jobs: ['install'], project: 'packaging'}
+    # NVBench Helper testing:
+    - {jobs: ['test'], project: 'nvbench_helper', ctk: '12.0', cxx: ['gcc10', 'clang14'], gpu: 'rtx2080'}
+    - {jobs: ['test'], project: 'nvbench_helper', ctk: '12.X', cxx: ['gcc10', 'clang14'], gpu: 't4'}
+    - {jobs: ['test'], project: 'nvbench_helper', ctk: '13.0', cxx: ['gcc15', 'clang20'], gpu: 'rtx2080'}
+    - {jobs: ['test'], project: 'nvbench_helper', ctk: '13.X', cxx: ['gcc',   'clang'],   gpu: 't4'}
+    # NVHPC build
+    - {jobs: ['build'], cxx: 'nvhpc-prev', ctk: 'nvhpc-prev', std: 'all', project: ['libcudacxx', 'cub', 'thrust', 'cudax', 'stdpar'], cpu: ['amd64', 'arm64']}
+    - {jobs: ['build'], cxx: 'nvhpc',      ctk: 'nvhpc',      std: 'all', project: ['libcudacxx', 'cub', 'thrust', 'cudax', 'stdpar'], cpu: ['amd64', 'arm64']}
+    # clang-cuda
+    - {jobs: ['build'], cudacxx: 'clang', ctk: 'clang-cuda', cxx: 'clang-cuda', std: 'all', sm: '75;80;90;100;120'}
+    - {jobs: ['build'], cudacxx: 'clang', ctk: 'clang_preview-cuda', cxx: 'clang_preview-cuda', std: 'all', sm: '75;80;90;100;120'}
+    - {jobs: ['build'], project: 'libcudacxx', cudacxx: 'clang', ctk: 'clang-cuda', cxx: 'clang-cuda',  std: 23, sm: '75;80;90;100;120'}
+    - {jobs: ['build'], project: 'libcudacxx', cudacxx: 'clang', ctk: 'clang_preview-cuda', cxx: 'clang_preview-cuda', std: 23, sm: '75;80;90;100;120'}
+    # clang-tidy
+    - { jobs: ['build'], project: 'tidy', std: 'min', cxx: ['clang'], cudacxx: ['clang'], ctk: 'clang-cuda', sm: '75' }
+    # arch-specific and family-specific arch builds
+    - {jobs: ['build'], project: ['libcudacxx', 'cub', 'thrust', 'cudax'], std: 'all', sm: '90a;100a;103a;110a;120a;121a'}
+    - {jobs: ['build'], project: ['libcudacxx', 'cub', 'thrust', 'cudax'], std: 'all', sm: '100f;103f;110f;120f;121f'}
+
+  weekly:
+    # CTK 12.0 full matrix build: default projects
+    - {jobs: ['build'], std: 'all', ctk: '12.0', cxx: ['gcc7', 'gcc8', 'gcc9', 'gcc10', 'gcc11', 'gcc12']}
+    - {jobs: ['build'], std: 'all', ctk: '12.0', cxx: ['clang14']}
+    - {jobs: ['build'], std: 'all', ctk: '12.0', cxx: ['msvc2019', 'msvc14.39']}
+    # CTK 12.X full matrix build: default projects
+    - {jobs: ['build'], std: 'all', ctk: '12.X', cxx: ['gcc7', 'gcc8', 'gcc9', 'gcc10', 'gcc11', 'gcc12', 'gcc13', 'gcc14']}
+    - {jobs: ['build'], std: 'all', ctk: '12.X', cxx: ['clang14', 'clang15', 'clang16', 'clang17', 'clang18', 'clang19']}
+    - {jobs: ['build'], std: 'all', ctk: '12.X', cxx: ['msvc2019', 'msvc2022']}
+    # CTK 13.0 full matrix build: default projects
+    - {jobs: ['build'], std: 'all', ctk: '13.0', cxx: ['gcc11', 'gcc12', 'gcc13', 'gcc14', 'gcc15']}
+    - {jobs: ['build'], std: 'all', ctk: '13.0', cxx: ['clang15', 'clang16', 'clang17', 'clang18', 'clang19', 'clang20']}
+    - {jobs: ['build'], std: 'all', ctk: '13.0', cxx: ['msvc2019', 'msvc2022']}
+    # CTK '13.X' full matrix build: default projects
+    - {jobs: ['build'], std: 'all', ctk: '13.X', cxx: ['gcc11', 'gcc12', 'gcc13', 'gcc14', 'gcc15']}
+    - {jobs: ['build'], std: 'all', ctk: '13.X', cxx: ['clang15', 'clang16', 'clang17', 'clang18', 'clang19', 'clang20', 'clang21']}
+    - {jobs: ['build'], std: 'all', ctk: '13.X', cxx: ['msvc2019', 'msvc2022', 'msvc2026']}
+    # CTK 12.0 full matrix build: cudax
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '12.0', cxx: ['gcc9', 'gcc10', 'gcc11', 'gcc12']}
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '12.0', cxx: ['clang14']}
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '12.0', cxx: ['msvc14.39']}
+    # CTK 12.X full matrix build: cudax
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '12.X', cxx: ['gcc9', 'gcc10', 'gcc11', 'gcc12', 'gcc13', 'gcc14']}
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '12.X', cxx: ['clang14', 'clang15', 'clang16', 'clang17', 'clang18', 'clang19']}
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '12.X', cxx: ['msvc2022']}
+    # CTK 13.0 full matrix build: cudax
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '13.0', cxx: ['gcc11', 'gcc12', 'gcc13', 'gcc14', 'gcc15']}
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '13.0', cxx: ['clang15', 'clang16', 'clang17', 'clang18', 'clang19', 'clang20']}
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '13.0', cxx: ['msvc2022']}
+    # CTK '13.X' full matrix build: cudax
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '13.X', cxx: ['gcc11', 'gcc12', 'gcc13', 'gcc14', 'gcc15']}
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '13.X', cxx: ['clang15', 'clang16', 'clang17', 'clang18', 'clang19', 'clang20', 'clang21']}
+    - {jobs: ['build'], project: 'cudax', std: 'all', ctk: '13.X', cxx: ['msvc2022', 'msvc2026']}
+    # CTK 12.X testing:
+    - {jobs: ['test'], project: 'libcudacxx', ctk: '12.X', std: 'minmax', cxx: ['gcc14', 'clang19', 'msvc2022'], gpu: 'rtx2080'}
+    - {jobs: ['test'], project: 'cub',        ctk: '12.X', std: 'minmax', cxx: ['gcc14', 'clang19', 'msvc2022'], gpu: 'rtxa6000'}
+    - {jobs: ['test'], project: 'thrust',     ctk: '12.X', std: 'minmax', cxx: ['gcc14', 'clang19', 'msvc2022'], gpu: 'rtx4090'}
+    - {jobs: ['test'], project: 'cudax',      ctk: '12.X', std: 'minmax', cxx: ['gcc14', 'clang19', 'msvc2022'], gpu: 't4'}
+    - {jobs: ['test'], project: ['libcudacxx', 'cub', 'thrust', 'cudax'], ctk: '12.X', std: 'minmax', cxx: 'gcc14', gpu: 'h100' }
+    # CTK '13.X' testing:
+    - {jobs: ['test'], project: 'libcudacxx', ctk: '13.X', std: 'minmax', cxx: ['gcc', 'clang', 'msvc'], gpu: 't4'}
+    - {jobs: ['test'], project: 'cub',        ctk: '13.X', std: 'minmax', cxx: ['gcc', 'clang', 'msvc'], gpu: 'rtxa6000'}
+    - {jobs: ['test'], project: 'thrust',     ctk: '13.X', std: 'minmax', cxx: ['gcc', 'clang', 'msvc'], gpu: 'rtx4090'}
+    - {jobs: ['test'], project: 'cudax',      ctk: '13.X', std: 'minmax', cxx: ['gcc', 'clang', 'msvc'], gpu: 'rtx2080'}
+    - {jobs: ['test'], project: ['libcudacxx', 'cub', 'thrust', 'cudax'], ctk: '13.X', std: 'minmax', gpu: 'h100' }
+    # RTX PRO 6000 coverage (limited due to small number of runners):
+    - {jobs: ['test'], project: ['cub', 'thrust'], std: 'max', cxx: 'gcc', gpu: 'rtxpro6000'}
+    # Misc:
+    - {jobs: ['build'], cpu: 'arm64', project: ['libcudacxx', 'cub', 'thrust', 'cudax'], ctk: '12.X', std: 'all', cxx: ['gcc14', 'clang19']}
+    - {jobs: ['build'], cpu: 'arm64', project: ['libcudacxx', 'cub', 'thrust', 'cudax'], ctk: '13.X', std: 'all', cxx: ['gcc', 'clang']}
+    - {jobs: ['test_gpu'], project: 'thrust', cmake_options: '-DTHRUST_DISPATCH_TYPE=Force32bit', gpu: 'rtx4090'}
+    - {jobs: ['test_gpu'], project: 'thrust', cmake_options: '-DTHRUST_DISPATCH_TYPE=Force64bit', gpu: 'rtx4090'}
+    - {jobs: ['limited'], project: 'cub', std: 17, gpu: 'rtx2080'}
+    # sm: all-cccl:
+    - {jobs: ['build'], project: ['thrust', 'libcudacxx', 'cudax'], std: 'max', sm: 'all-cccl' }
+    - {jobs: ['build_nolid', 'build_lid0'], project: ['cub'], std: 'max', sm: 'all-cccl'}
+    # NVRTC tests don't currently support 12.0:
+    - {jobs: ['nvrtc'],          project: 'libcudacxx', ctk: [        '12.X', '13.0', '13.X'], cxx: 'gcc12', std: 'all', gpu: 'rtx2080', sm: 'gpu'}
+    - {jobs: ['verify_codegen'], project: 'libcudacxx'}
+    # c.parallel -- pinned to gcc13 / msvc2022 to match python
+    - {jobs: ['test'],  project: ['cccl_c_parallel'], ctk: '12.X', cxx: ['gcc13', 'msvc2022'], gpu: ['t4']}
+    - {jobs: ['test'],  project: ['cccl_c_parallel'], ctk: '13.X', cxx: ['gcc13', 'msvc2022'], gpu: ['rtx2080', 'l4', 'h100']}
+    # RTX PRO 6000 coverage (limited due to small number of runners):
+    - {jobs: ['test'], project: 'cccl_c_parallel', ctk: '13.X', cxx: ['gcc13'], gpu: ['rtxpro6000']}
+    # c.experimental.stf -- pinned to gcc13 to match python
+    - {jobs: ['test'],  project: ['cccl_c_stf'], ctk: '12.X', cxx: 'gcc13', gpu: ['rtx2080']}
+    - {jobs: ['test'],  project: ['cccl_c_stf'], ctk: '13.X', cxx: 'gcc13', gpu: ['t4', 'l4', 'h100']}
+    # Python -- pinned to gcc13 / msvc2022 for consistency across CTK images
+    - {jobs: ['test'], project: 'python', ctk: ['12.0', '12.X', '13.0', '13.X'], py_version: ['3.10', '3.11', '3.12', '3.13', '3.14'], gpu: 'l4', cxx: ['gcc13', 'msvc2022']}
+    - {jobs: ['test'], project: 'python', ctk: ['12.X', '13.X'], py_version: '3.14', gpu: 'h100', cxx: ['gcc13', 'msvc2022']}
+    - {jobs: ['test'], project: 'python', ctk: ['12.0', '12.X', '13.0', '13.X'], py_version: ['3.10', '3.14'], gpu: 't4', cxx: 'gcc13'}
+    - {jobs: ['test'], project: 'python', ctk: ['12.0', '12.X', '13.0', '13.X'], py_version: '3.14', gpu: 'rtxa6000', cxx: 'gcc13'}
+    - {jobs: ['test'], project: 'python', ctk: '13.X', py_version: '3.14', gpu: 'rtxpro6000', cxx: 'gcc13'}
+    - {jobs: ['test'], project: 'python', ctk: ['12.0', '12.X', '13.0', '13.X'], py_version: '3.14', gpu: 'l4', cxx: ['gcc13', 'msvc2022'], py_ctk_mode: 'sysctk'}
+    - {jobs: ['test_py_compute_minimal'], project: 'python', ctk: ['12.0', '12.X', '13.0', '13.X'], py_version: '3.14t', gpu: 'l4', cxx: ['gcc13', 'msvc2022'], py_ctk_mode: 'sysctk'}
+    # cuda.cccl.headers (CPU-only): all CTK x source x OS, py endpoints 3.10 + 3.14
+    - {jobs: ['test_headers'], project: 'python', ctk: ['12.0', '12.X', '13.0', '13.X'], py_version: ['3.10', '3.14'], cxx: ['gcc13', 'msvc2022'], py_ctk_mode: ['pinned', 'latest', 'sysctk']}
+    # CCCL packaging:
+    - {jobs: ['test'], project: 'packaging', ctk: '12.0', cxx: ['gcc10', 'clang14'], gpu: 't4', args: '-min-cmake'}
+    - {jobs: ['test'], project: 'packaging', ctk: '12.X', cxx: ['gcc10', 'clang14'], gpu: 'rtx2080'}
+    - {jobs: ['test'], project: 'packaging', ctk: '13.0', cxx: ['gcc15', 'clang20'],   gpu: 't4', args: '-min-cmake'}
+    - {jobs: ['test'], project: 'packaging', ctk: '13.X', cxx: ['gcc',   'clang'],   gpu: 'rtx2080'}
+    - {jobs: ['install'], project: 'packaging'}
+    # NVBench Helper:
+    - {jobs: ['test'], project: 'nvbench_helper', ctk: '12.0', cxx: ['gcc10', 'clang14'], gpu: 'rtx2080'}
+    - {jobs: ['test'], project: 'nvbench_helper', ctk: '12.X', cxx: ['gcc10', 'clang14'], gpu: 't4'}
+    - {jobs: ['test'], project: 'nvbench_helper', ctk: '13.0', cxx: ['gcc15', 'clang20'], gpu: 'rtx2080'}
+    - {jobs: ['test'], project: 'nvbench_helper', ctk: '13.X', cxx: ['gcc',   'clang'],   gpu: 't4'}
+    # NVHPC build
+    - {jobs: ['build'], cxx: 'nvhpc-prev', ctk: 'nvhpc-prev', std: 'all', project: ['libcudacxx', 'cub', 'thrust', 'cudax', 'stdpar'], cpu: ['amd64', 'arm64']}
+    - {jobs: ['build'], cxx: 'nvhpc',      ctk: 'nvhpc',      std: 'all', project: ['libcudacxx', 'cub', 'thrust', 'cudax', 'stdpar'], cpu: ['amd64', 'arm64']}
+    # clang-cuda
+    - {jobs: ['build'], cudacxx: 'clang', ctk: 'clang-cuda', cxx: 'clang-cuda', std: 'all', sm: '75;80;90;100;120'}
+    - {jobs: ['build'], cudacxx: 'clang', ctk: 'clang_preview-cuda', cxx: 'clang_preview-cuda', std: 'all', sm: '75;80;90;100;120'}
+    - {jobs: ['build'], project: 'libcudacxx', cudacxx: 'clang', ctk: 'clang-cuda', cxx: 'clang-cuda',  std: 23, sm: '75;80;90;100;120'}
+    - {jobs: ['build'], project: 'libcudacxx', cudacxx: 'clang', ctk: 'clang_preview-cuda', cxx: 'clang_preview-cuda', std: 23, sm: '75;80;90;100;120'}
+    # compute-sanitizer
+    - {jobs: ['compute_sanitizer'], project: 'cub', std: 'max', gpu: 'rtxa6000', sm: 'gpu', cmake_options: '-DCMAKE_CUDA_FLAGS=-lineinfo'}
+    # clang-tidy
+    - { jobs: ['build'], project: 'tidy', std: 'min', cxx: ['clang'], cudacxx: ['clang'], ctk: 'clang-cuda', sm: '75' }
+    # arch-specific and family-specific arch builds
+    - {jobs: ['build'], project: ['libcudacxx', 'cub', 'thrust', 'cudax'], std: 'all', sm: '90a;100a;103a;110a;120a;121a'}
+    - {jobs: ['build'], project: ['libcudacxx', 'cub', 'thrust', 'cudax'], std: 'all', sm: '100f;103f;110f;120f;121f'}
+
+  python-wheels:
+    - {jobs: ['test'], project: 'python', ctk: ['12.0', '12.X', '13.0', '13.X'], py_version: ['3.10', '3.11', '3.12', '3.13', '3.14'], gpu: 'l4', cxx: ['gcc13', 'msvc2022']}
+    - {jobs: ['test'], project: 'python', ctk: ['12.X', '13.X'], py_version: '3.14', gpu: 'h100', cxx: ['gcc13', 'msvc2022']}
+    - {jobs: ['test'], project: 'python', cpu: 'arm64', ctk: ['12.X', '13.X'], py_version: ['3.10', '3.11', '3.12', '3.13', '3.14'], gpu: 'l4', cxx: 'gcc13'}
+
+
+  # This is just used to ensure that we generate devcontainers for all images we build.
+  # These do not map to any actual jobs.
+  devcontainers:
+    - {jobs: ['dc'],     ctk: ['12.0', '12.X'                ], cxx: ['clang14']}
+    - {jobs: ['dc'],     ctk: ['12.0', '12.X'                ], cxx: ['gcc7', 'gcc8', 'gcc9', 'gcc10']}
+    - {jobs: ['dc'],     ctk: ['12.0', '12.X', '13.0', '13.X'], cxx: ['gcc11', 'gcc12']}
+    - {jobs: ['dc'],     ctk: [        '12.X', '13.0', '13.X'], cxx: ['gcc13', 'gcc14']}
+    - {jobs: ['dc'],     ctk: [                '13.0', '13.X'], cxx: ['gcc15']}
+    - {jobs: ['dc'],     ctk: [        '12.X', '13.0', '13.X'], cxx: ['clang15', 'clang16', 'clang17', 'clang18', 'clang19']}
+    - {jobs: ['dc_ext'], ctk: [        '12.X', '13.0', '13.X'], cxx: ['gcc14', 'clang20']}
+    # Clang21+CTK12.9 is currently only used for cuda-clang testing. nvcc 12.9 doesn't support clang21.
+    - {jobs: ['dc_ext'], ctk: [        '12.X',         '13.X'], cxx: ['clang21']}
+    - {jobs: ['dc_ext'], ctk: [                        '13.X'], cxx: ['gcc15']}
+    # Clang22+CTK12.9 is currently only used for clang-cuda testing. nvcc 12.9 doesn't support clang22.
+    - {jobs: ['dc_ext'], ctk: [        '12.X'                ], cxx: ['clang_preview22']}
+    # 12.0 python image, pinned at gcc13 for simplicity. CTK 12.0 doesn't really play nice with gcc13, but
+    # that doesn't matter for running python tests.
+    - {jobs: ['dc'], ctk: ['12.0'], cxx: 'gcc13'}
+    # NVHPC
+    - {jobs: ['dc'], cxx: 'nvhpc-prev', ctk: 'nvhpc-prev'}
+    - {jobs: ['dc'], cxx: 'nvhpc',      ctk: 'nvhpc'}
+
+  # Any generated jobs that match the entries in `exclude` will be removed from the final matrix for all workflows.
+  exclude:
+    # GPU runners are not available on Windows.
+    - {jobs: ['test', 'test_gpu', 'test_nolid', 'test_lid0', 'test_lid1', 'test_lid2'], cxx: ['msvc2019', 'msvc14.39', 'msvc2022', 'msvc2026']}
+    # cudax doesn't support C++17 on msvc:
+    - {project: 'cudax', std: 17, cxx: ['msvc2019', 'msvc14.39', 'msvc2022', 'msvc2026']}
+
+
+#############################################################################################
+
+
+# The version of the devcontainer images to use from https://hub.docker.com/r/rapidsai/devcontainers
+devcontainer_version: '26.10'
+
+# Compiler versions used for the cuda99.X internal builds:
+cuda99_gcc_version: 15
+cuda99_clang_version: 21
+
+# All supported C++ standards:
+all_stds: [17, 20]
+
+# Aliases:
+# - 12.X: Newest CTK 12.X version.
+# - 13.X: Newest CTK 13.X version.
+# - nvhpc: CTK shipped in newest NVHPC
+# - nvhpc-prev: CTK shipped in previous NVHPC
+# - pybuild: Selects image to use for python wheel builds' outer docker instance
+ctk_versions:
+  12.0: { stds: [17, 20] }
+  12.9: { stds: [17, 20], alias: ['12.X', 'pybuild', 'clang-cuda', 'clang_preview-cuda'] }
+  13.0: { stds: [17, 20] }
+  13.1: { stds: [17, 20], alias: ['nvhpc-prev']}
+  13.2: { stds: [17, 20], alias: ['nvhpc']}
+  13.3: { stds: [17, 20], alias: ['13.X'] }
+
+device_compilers:
+  nvcc: # Version / stds are taken from CTK
+    name: 'nvcc'
+    exe: 'nvcc'
+  clang: # Requires cxx=clang. Version / stds are taken from cxx compiler.
+    name: "ClangCUDA"
+    exe: 'clang++'
+
+host_compilers:
+  gcc:
+    name: 'GCC'
+    container_tag: 'gcc'
+    exe: 'g++'
+    versions:
+      7:  { stds: [17,   ] }
+      8:  { stds: [17,   ] }
+      9:  { stds: [17,   ] }
+      10: { stds: [17, 20] }
+      11: { stds: [17, 20] }
+      12: { stds: [17, 20] }
+      13: { stds: [17, 20] }
+      14: { stds: [17, 20] }
+      15: { stds: [17, 20] }
+  clang:
+    name: 'Clang'
+    container_tag: 'llvm'
+    exe: 'clang++'
+    versions:
+      14: { stds: [17, 20] }
+      15: { stds: [17, 20] }
+      16: { stds: [17, 20] }
+      17: { stds: [17, 20] }
+      18: { stds: [17, 20] }
+      19: { stds: [17, 20] }
+      20: { stds: [17, 20] }
+      21: { stds: [17, 20], alias: 'cuda' }
+  clang_preview:
+    name: 'Clang'
+    container_tag: 'llvm'
+    exe: 'clang++'
+    versions:
+      22: { stds: [17, 20], alias: 'cuda' }
+  msvc:
+    name: 'MSVC'
+    container_tag: 'cl'
+    exe: cl
+    versions:
+      '14.29': { stds: [17,   ], alias: '2019' }
+      '14.39': { stds: [17, 20] } # CTK 12.0 doesn't recognize >14.39 as MSVC 2022.
+      '14.44': { stds: [17, 20], alias: '2022' }
+      '14.50': { stds: [17, 20], alias: '2026' }
+
+  nvhpc:
+    name: 'NVHPC'
+    container_tag: 'nvhpc'
+    exe: nvc++
+    versions:
+      # !! Update the ctk_versions 'nvhpc*' aliases when updating NVHPC versions:
+      26.3: { stds: [17, 20], alias: 'prev' }
+      26.5: { stds: [17, 20] }
+
+# Jobs support the following properties:
+#
+# - name: The human-readable name of the job. Default is the capitalized job key.
+# - needs:
+#   - A list of jobs that must be completed before this job can run. Default is an empty list.
+#   - These jobs are automatically added if needed:
+#     - Eg. "jobs: ['test']" in the workflow def will also create the required 'build' jobs.
+# - gpu: Whether the job requires a GPU runner. Default is false.
+# - cuda_ext: Whether the job requires a devcontainer with extra CUDA libraries. Default is false.
+# - invoke:
+#   - Map the job type to the script invocation spec:
+#     - prefix: The script invocation prefix. Default is the job name.
+#     - args: Additional arguments to pass to the script. Default is no args.
+#   - The script is invoked either:
+#     linux:   `ci/windows/<spec[prefix]>_<project>.ps1 <spec[args]>`
+#     windows: `ci/<spec[prefix]>_<project>.sh <spec[args]>`
+# - force_producer_ctk:
+#   - If set, force the auto-generated producers for this job to use a specific CTK version.
+#   - By default, the autogenerated job's CTK version is determined by the consumer's `ctk` tag.
+#   - This is useful for testing the cross-testing major version compat.
+#   - E.g. "force_producer_ctk: '12.0'" on a test step will force the generated build step to use CTK 12.0.
+
+jobs:
+  # General:
+  build:        { gpu: false }
+  test:         { gpu: true, needs: 'build' }
+  install:      { gpu: false }
+
+  test_nobuild: { gpu: true, name: 'Test', invoke: { prefix: 'test' } }
+
+  compute_sanitizer: { gpu: true, name: 'ComputeSanitizer', needs: 'build', invoke: { prefix: 'test', args: '-compute-sanitizer' } }
+
+  # libcudacxx:
+  nvrtc: { gpu: true, name: 'NVRTC' }
+  verify_codegen: { gpu: false, name: 'VerifyCodegen' }
+
+  # CUB:
+  build_nolid: { name: 'BuildNoLaunch',     gpu: false, invoke: { prefix: 'build', args: '-no-lid'} }
+  build_lid0:  { name: 'BuildHostLaunch',   gpu: false, invoke: { prefix: 'build', args: '-lid0'} }
+  build_lid1:  { name: 'BuildDeviceLaunch', gpu: false, invoke: { prefix: 'build', args: '-lid1'} }
+  build_lid2:  { name: 'BuildGraphCapture', gpu: false, invoke: { prefix: 'build', args: '-lid2'} }
+  # NoLid -> The string `lid_X` doesn't appear in the test name. Mostly warp/block tests, old device tests, and examples.
+  test_nolid: { name: 'TestNoLaunch',      gpu: true, needs: 'build_nolid', invoke: { prefix: 'test', args: '-no-lid --test-par 8'} }
+  # CUB uses `lid` to indicate launch strategies: whether CUB algorithms are:
+  # - launched from the host (lid0):
+  test_lid0:  { name: 'HostLaunch',   gpu: true, needs: 'build_lid0', invoke: { prefix: 'test', args: '-lid0 --test-par 8'} }
+  # - launched from the device (lid1):
+  test_lid1:  { name: 'DeviceLaunch', gpu: true, needs: 'build_lid1', invoke: { prefix: 'test', args: '-lid1 --test-par 8'} }
+  # - captured in a CUDA graph for deferred launch (lid2):
+  test_lid2:  { name: 'GraphCapture', gpu: true, needs: 'build_lid2', invoke: { prefix: 'test', args: '-lid2 --test-par 8'} }
+  # Limited build reduces the number of runtime test cases, available device memory, etc, and may be used
+  # to reduce test runtime in limited environments.
+  limited:    { name: "SmallGMem",   gpu: true, needs: 'build', invoke: { prefix: 'test', args: '-limited'} }
+  # Compute sanitizer jobs:
+  compute_mem_nolid:  { name: 'CSMem-TestGPU',       gpu: true, needs: 'build_nolid', invoke: { prefix: 'test', args: '-compute-sanitizer-memcheck  -no-lid'} }
+  compute_mem_lid0:   { name: 'CSMem-HostLaunch',    gpu: true, needs: 'build_lid0',  invoke: { prefix: 'test', args: '-compute-sanitizer-memcheck  -lid0'} }
+  compute_race_nolid: { name: 'CSRace-TestGPU',      gpu: true, needs: 'build_nolid', invoke: { prefix: 'test', args: '-compute-sanitizer-racecheck -no-lid'} }
+  compute_race_lid0:  { name: 'CSRace-HostLaunch',   gpu: true, needs: 'build_lid0',  invoke: { prefix: 'test', args: '-compute-sanitizer-racecheck -lid0'} }
+  compute_init_nolid: { name: 'CSInit-TestGPU',      gpu: true, needs: 'build_nolid', invoke: { prefix: 'test', args: '-compute-sanitizer-initcheck -no-lid'} }
+  compute_init_lid0:  { name: 'CSInit-HostLaunch',   gpu: true, needs: 'build_lid0',  invoke: { prefix: 'test', args: '-compute-sanitizer-initcheck -lid0'} }
+  compute_sync_nolid: { name: 'CSSync-TestGPU',      gpu: true, needs: 'build_nolid', invoke: { prefix: 'test', args: '-compute-sanitizer-synccheck -no-lid'} }
+  compute_sync_lid0:  { name: 'CSSync-HostLaunch',   gpu: true, needs: 'build_lid0',  invoke: { prefix: 'test', args: '-compute-sanitizer-synccheck -lid0'} }
+
+  # Thrust:
+  test_cpu: { name: 'TestCPU', gpu: false, needs: 'build', invoke: { prefix: 'test', args: '-cpu-only'} }
+  test_gpu: { name: 'TestGPU', gpu: true,  needs: 'build', invoke: { prefix: 'test', args: '-gpu-only'} }
+
+  # Python:
+  build_py_wheel:   { name: "Build cuda.cccl",             gpu: false, invoke: { prefix: 'build_cuda_cccl'} }
+  test_headers:     { name: "Test cuda.cccl.headers",      gpu: false, needs: 'build_py_wheel', force_producer_ctk: "pybuild", invoke: { prefix: 'test_cuda_cccl_headers'} }
+  test_py_par:      { name: "Test cuda.compute",     gpu: true,  needs: 'build_py_wheel', force_producer_ctk: "pybuild", invoke: { prefix: 'test_cuda_compute'} }
+  test_py_compute_minimal: { name: "Test cuda.compute minimal", gpu: true, needs: 'build_py_wheel', force_producer_ctk: "pybuild", invoke: { prefix: 'test_cuda_compute_minimal'} }
+  test_py_examples: { name: "Test cuda.cccl.examples",     gpu: true,  needs: 'build_py_wheel', force_producer_ctk: "pybuild", invoke: { prefix: 'test_cuda_cccl_examples'} }
+
+  # Run jobs for 'target' project (ci/util/build_and_test_targets.sh):
+  run_cpu: { gpu: false }
+  run_gpu: { gpu: true  }
+
+  # Only used for generating devcontainers. No scripts actually exist for these:
+  dc:     { gpu: false }
+  dc_ext: { gpu: false, cuda_ext: true }
+
+# Projects have the following properties:
+#
+# Keys are project subdirectories names. These will also be used in script names.
+#
+# - stds: A list of C++ standards to test. Required.
+# - name: The human-readable name of the project. Default is the project key.
+# - job_map: Map general jobs to arrays of project-specific jobs.
+#            Useful for things like splitting cpu/gpu testing for a project.
+#            E.g. "job_map: { test: ['test_cpu', 'test_gpu'] }" replaces
+#            the "test" job with distinct "test_cpu" and "test_gpu" jobs.
+projects:
+  packaging:
+    name: 'CCCL Packaging'
+    stds: [17, 20]
+    job_map:
+      build: []
+      test: ['test_nobuild']
+  libcudacxx:
+    name: 'libcu++'
+    stds: [17, 20]
+  cub:
+    name: 'CUB'
+    stds: [17, 20]
+    job_map:
+      build: ['build_nolid', 'build_lid0', 'build_lid1', 'build_lid2']
+      test:  ['test_nolid',  'test_lid0',  'test_lid1',  'test_lid2']
+      compute_sanitizer:
+        - compute_mem_nolid
+        - compute_mem_lid0
+        - compute_race_nolid
+        - compute_race_lid0
+        - compute_init_nolid
+        - compute_init_lid0
+        - compute_sync_nolid
+        - compute_sync_lid0
+  thrust:
+    name: 'Thrust'
+    stds: [17, 20]
+    job_map: { test: ['test_cpu', 'test_gpu'] }
+  cudax:
+    stds: [17, 20]
+  stdpar:
+    name: 'NVHPC stdpar'
+    stds: [17, 20]
+  python:
+    name: "Python"
+    job_map:
+      build: ['build_py_wheel']
+      test:  ['test_py_par', 'test_py_examples']
+  python_v2:
+    name: "Python (cuda.compute on v2/HostJIT)"
+    # Only cuda.compute differs between v1 and v2; cuda.cccl.headers does not
+    # need separate coverage. Run the examples because they exercise
+    # cuda.compute against the v2 backend.
+    job_map:
+      build: ['build_py_wheel']
+      test:  ['test_py_par', 'test_py_examples']
+  python_tsan:
+    name: "Python (cuda.compute free-threaded ThreadSanitizer)"
+    # Runs per-PR (and nightly). The producer build_py_wheel runs
+    # build_cuda_cccl_python_tsan.sh (c.parallel v1 host code built with
+    # -fsanitize=thread); test_py_compute_minimal runs
+    # test_cuda_compute_minimal_python_tsan.sh (the FT stress + sweep under the
+    # TSan runtime). Deliberately absent from the `python-wheels` publish
+    # workflow, so these instrumented wheels never reach PyPI.
+    job_map:
+      build: ['build_py_wheel']
+      test:  ['test_py_compute_minimal']
+  cccl_c_parallel:
+    name: 'CCCL C Parallel'
+    stds: [20]
+  cccl_c_parallel_v2:
+    name: 'CCCL C Parallel v2 (HostJIT)'
+    stds: [20]
+    # test_cccl_c_parallel_v2.sh builds inline (no separate build script),
+    # so suppress the default test→build dependency. test_nobuild invokes
+    # test_<project>.sh directly without a producer build job.
+    job_map:
+      build: []
+      test: ['test_nobuild']
+  cccl_c_stf:
+    name: 'CCCL C CUDASTF'
+    stds: [20]
+  nvbench_helper:
+    name: 'NVBench Helper'
+    stds: [17] # Only builds on oldest arch for max compat.
+    job_map:
+      build: []
+      test: ['test_nobuild']
+
+  # Run specific build_and_test_targets.sh invocations across the CI matrix.
+  # Use the override workflow and supply arguments via the `args` tag.
+  # Example:
+  #   override:
+  #     - { jobs: ['run'], project: 'target', ctk: ['12.X', '13.X'], cxx: 'gcc', gpu: 'rtx2080',
+  #         args: '--preset cub-cpp20 --build-targets "cub.cpp20.test.iterator" --ctest-targets "cub.cpp20.test.iterator"' }
+  target:
+    name: 'Target'
+    stds: [17, 20]
+  bisect:
+    name: 'Bisect'
+    stds: [17, 20]
+  tidy:
+    name: 'clang-tidy'
+    stds: [17]
+
+# name -> Display name for generated job labels.
+# runner -> GPU/driver/count segment of the GHA runner label.
+# testing -> Runner with GPU is in a nv-gh-runners testing pool.
+gpus:
+  t4:         { name: 'T4',          sm: 75,  runner: 't4-latest-1'         } # 16 GB,  10 runners
+  rtx2080:    { name: 'RTX2080',     sm: 75,  runner: 'rtx2080-latest-1'    } #  8 GB,  12 runners
+  rtxa6000:   { name: 'RTXA6000',    sm: 86,  runner: 'rtxa6000-latest-1'   } # 48 GB,  12 runners
+  l4:         { name: 'L4',          sm: 89,  runner: 'l4-latest-1'         } # 24 GB,  48 runners
+  rtx4090:    { name: 'RTX4090',     sm: 89,  runner: 'rtx4090-latest-1'    } # 24 GB,  10 runners
+  h100:       { name: 'H100',        sm: 90,  runner: 'h100-latest-1'       } # 80 GB,  16 runners
+  h100_2gpu:  { name: 'H100 2-GPU',  sm: 90,  runner: 'h100-latest-2'       } # 2 x 80 GB
+  # Very small number of runners on loan from cuda-python while we wait for our order to arrive.
+  # Limit jobs on these:
+  rtxpro6000: { name: 'RTXPRO6000',  sm: 120, runner: 'rtxpro6000-latest-1' }
+
+# Tags are used to define a `matrix job` in the workflow section.
+#
+# Tags have the following options:
+#  - required: Whether the tag is required. Default is false.
+#  - default: The default value for the tag. Default is null.
+tags:
+   # An array of jobs (e.g. 'build', 'test', 'nvrtc', 'infra', 'verify_codegen', ...)
+   # See the `jobs` map.
+  jobs: { required: true }
+  # CUDA ToolKit version
+  # See the `ctks` map.
+  ctk: { default: '13.X' }
+  # CPU architecture
+  cpu: { default: 'amd64' }
+  # GPU model
+  gpu: { default: 'rtx2080' }
+  # Host compiler {name, version, exe}
+  # See the `host_compilers` map.
+  cxx: { default: 'gcc' }
+  # Device compiler.
+  # See the `device_compilers` map.
+  cudacxx: { default: 'nvcc' }
+  # Project name (e.g. libcudacxx, cub, thrust, cccl)
+  # See the `projects` map.
+  project: { default: ['libcudacxx', 'cub', 'thrust'] }
+  # Python version for Python builds/tests
+  py_version: { required: false }
+  # Python CTK-source mode for test lanes (Python-only):
+  #   'pinned' (default) -- pin cuda-toolkit to the container CTK minor
+  #   'latest'           -- unpinned; pip resolves the newest minor
+  #   'sysctk'           -- use the system toolkit; install the sysctk extras
+  # If set, passed to the script with `-ctk-mode <mode>`. Exploded if an array,
+  # e.g. `['pinned','sysctk']`.
+  py_ctk_mode: { required: false }
+  # C++ standard
+  # If set to 'all', all stds supported by the ctk/compilers/project are used.
+  # If set to 'min', 'max', or 'minmax', the minimum, maximum, or both stds are used.
+  # If set, will be passed to script with `-std <std>`.
+  std: { required: false }
+  # GPU architecture
+  # - If set, passed to script with `-arch <sm>`.
+  # - Format is the same as `CMAKE_CUDA_ARCHITECTURES`:
+  #   - PTX only: 70-virtual
+  #   - SASS only: 70-real
+  #   - Both: 70
+  # - Can pass multiple architectures via "60;70-real;80-virtual"
+  # - Defaults to use the settings in the CMakePresets.json file.
+  # - Will be exploded if an array, e.g. `sm: ['60;70;80;90', '90a']` creates two jobs.
+  # - Set to 'gpu' to only target the GPU in the `gpu` tag.
+  sm: { required: false }
+  # Additional CMake options to pass to the build.
+  # If set, passed to script with `-cmake_options "<cmake_options>"`.
+  cmake_options: { required: false }
+  # Environment variables to add
+  environment: { required: false, default: [] }
+  # Additional arguments appended to the generated command.
+  # Typically used with the `target` project to forward options to
+  # ci/util/build_and_test_targets.sh, but works with all CI jobs.
+  args: { required: false, default: "" }
diff --git a/cccl_upstream/ci/matx/build_matx.sh b/cccl_upstream/ci/matx/build_matx.sh
new file mode 100755
index 00000000..1bf687aa
--- /dev/null
+++ b/cccl_upstream/ci/matx/build_matx.sh
@@ -0,0 +1,113 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+readonly matx_repo=https://github.com/NVIDIA/MatX.git
+readonly matx_branch=main
+
+# Ensure the script is being executed in the root cccl directory:
+cd "$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )/../..";
+
+log_vars() {
+  for var in "$@"; do
+    echo "${var}=${!var}"
+  done
+}
+
+# Take two version strings and return the greater of the two:
+version_max() {
+  local v1="${1}"
+  local v2="${2}"
+  if ci/util/version_compare.sh "$v1" ge "$v2"; then
+    echo "$v1"
+  else
+    echo "$v2"
+  fi
+}
+
+# Get the current CCCL info:
+readonly cccl_repo="${PWD}"
+
+# Define CCCL_TAG to override the default CCCL SHA. Otherwise the current HEAD of the local checkout is used.
+echo "CCCL_TAG (override): ${CCCL_TAG-}";
+if test -n "${CCCL_TAG-}"; then
+    # If CCCL_TAG is defined, fetch it to the local checkout
+    git -C "${cccl_repo}" fetch origin "${CCCL_TAG}";
+    cccl_sha="$(git -C "${cccl_repo}" rev-parse FETCH_HEAD)";
+else
+    cccl_sha="$(git -C "${cccl_repo}" rev-parse HEAD)";
+fi
+
+cccl_repo_version="$(git -C "${cccl_repo}" describe "${cccl_sha}"| grep -Eo '[0-9]+\.[0-9]+\.[0-9]+')"
+readonly cccl_repo_version
+
+# Define CCCL_VERSION to override the version used by rapids-cmake to patch CCCL.
+echo "CCCL_VERSION (override): ${CCCL_VERSION-}";
+if test -n "${CCCL_VERSION-}"; then
+  readonly cccl_rapids_cmake_version="${CCCL_VERSION}"
+else
+  cccl_rapids_cmake_version="${cccl_repo_version}"
+  # shellcheck disable=SC2034
+  readonly cccl_rapids_cmake_version
+fi
+
+# If the current version is less than 2.8.0, use 2.8.0 for the rapids-cmake version.
+# This is to allow rapids-cmake to correctly patch the CCCL install rules on current `main`.
+cccl_version=$(version_max "${cccl_repo_version}" "2.8.0")
+readonly cccl_version
+
+readonly workdir="${cccl_repo}/build/${CCCL_BUILD_INFIX:-}/matx"
+readonly version_file="${workdir}/MatX/cmake/versions.json"
+readonly version_override_file="${workdir}/versions-override.json"
+
+log_vars \
+  matx_repo matx_branch \
+  cccl_repo cccl_sha cccl_repo_version cccl_rapids_cmake_version \
+  workdir \
+  version_file version_override_file
+
+mkdir -p "${workdir}"
+cd "${workdir}"
+
+# Python deps:
+pip install numpy
+
+# Clone MatX
+rm -rf MatX
+git clone "${matx_repo}" -b "${matx_branch}"
+
+cd MatX
+echo "MatX HEAD:"
+git log -1 --format=short
+cd ..
+
+# Write out version override file
+jq -r ".packages.CCCL *=
+  {
+    \"git_url\": \"${cccl_repo}\",
+    \"git_tag\": \"${cccl_sha}\",
+    \"version\": \"${cccl_version}\",
+    \"always_download\": true
+  }" \
+  "${version_file}" > "${version_override_file}"
+
+echo "Overriding MatX versions.json file:"
+cat "$version_override_file"
+
+# Configure and build
+rm -rf build
+
+SCCACHE_NO_DIST_COMPILE=1 cmake \
+  -B build -S MatX -G Ninja \
+  "-DCMAKE_CUDA_ARCHITECTURES=75;120" \
+  "-DRAPIDS_CMAKE_CPM_OVERRIDE_VERSION_FILE=${version_override_file}" \
+  -DMATX_BUILD_TESTS=ON \
+  -DMATX_BUILD_EXAMPLES=ON \
+  -DMATX_BUILD_BENCHMARKS=ON \
+  -DMATX_EN_CUTENSOR=ON
+
+# Disabled because `cmake --build -j ""` is invalid, but so is
+# `cmake --build -j8`. CMake expects a space between `-j` and
+# the numeric argument, or no argument at all.
+# shellcheck disable=SC2086
+time cmake --build build -j ${PARALLEL_LEVEL:-}
diff --git a/cccl_upstream/ci/ninja_summary.py b/cccl_upstream/ci/ninja_summary.py
new file mode 100755
index 00000000..e1316f8d
--- /dev/null
+++ b/cccl_upstream/ci/ninja_summary.py
@@ -0,0 +1,390 @@
+#!/usr/bin/env python3
+# Copyright (c) 2018 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+r"""Summarize the last ninja build, invoked with ninja's -C syntax.
+
+This script is designed to be automatically run after each ninja build in
+order to summarize the build's performance. Making build performance information
+more visible should make it easier to notice anomalies and opportunities. To use
+this script on Windows just set NINJA_SUMMARIZE_BUILD=1 and run autoninja.bat.
+
+On Linux you can get autoninja to invoke this script using this syntax:
+
+$ NINJA_SUMMARIZE_BUILD=1 autoninja -C out/Default/ chrome
+
+You can also call this script directly using ninja's syntax to specify the
+output directory of interest:
+
+> python3 post_build_ninja_summary.py -C out/Default
+
+Typical output looks like this:
+
+>ninja -C out\debug_component base
+ninja.exe -C out\debug_component base -j 960 -l 48  -d keeprsp
+ninja: Entering directory `out\debug_component'
+[1 processes, 1/1 @ 0.3/s : 3.092s ] Regenerating ninja files
+Longest build steps:
+       0.1 weighted s to build obj/base/base/trace_log.obj (6.7 s elapsed time)
+       0.2 weighted s to build nasm.exe, nasm.exe.pdb (0.2 s elapsed time)
+       0.3 weighted s to build obj/base/base/win_util.obj (12.4 s elapsed time)
+       1.2 weighted s to build base.dll, base.dll.lib (1.2 s elapsed time)
+Time by build-step type:
+       0.0 s weighted time to generate 6 .lib files (0.3 s elapsed time sum)
+       0.1 s weighted time to generate 25 .stamp files (1.2 s elapsed time sum)
+       0.2 s weighted time to generate 20 .o files (2.8 s elapsed time sum)
+       1.7 s weighted time to generate 4 PEFile (linking) files (2.0 s elapsed
+time sum)
+      23.9 s weighted time to generate 770 .obj files (974.8 s elapsed time sum)
+26.1 s weighted time (982.9 s elapsed time sum, 37.7x parallelism)
+839 build steps completed, average of 32.17/s
+
+If no gn clean has been done then results will be for the last non-NULL
+invocation of ninja. Ideas for future statistics, and implementations are
+appreciated.
+
+The "weighted" time is the elapsed time of each build step divided by the number
+of tasks that were running in parallel. This makes it an excellent approximation
+of how "important" a slow step was. A link that is entirely or mostly serialized
+will have a weighted time that is the same or similar to its elapsed time. A
+compile that runs in parallel with 999 other compiles will have a weighted time
+that is tiny."""
+
+import argparse
+import errno
+import fnmatch
+import os
+import subprocess
+import sys
+
+# The number of long build times to report:
+long_count = 10
+# The number of long times by extension to report
+long_ext_count = 10
+
+
+class Target:
+    """Represents a single line read for a .ninja_log file."""
+
+    def __init__(self, start, end):
+        """Creates a target object by passing in the start/end times in seconds
+        as a float."""
+        self.start = start
+        self.end = end
+        # A list of targets, appended to by the owner of this object.
+        self.targets = []
+        self.weighted_duration = 0.0
+
+    def Duration(self):
+        """Returns the task duration in seconds as a float."""
+        return self.end - self.start
+
+    def SetWeightedDuration(self, weighted_duration):
+        """Sets the duration, in seconds, passed in as a float."""
+        self.weighted_duration = weighted_duration
+
+    def WeightedDuration(self):
+        """Returns the task's weighted duration in seconds as a float.
+
+        Weighted_duration takes the elapsed time of the task and divides it
+        by how many other tasks were running at the same time. Thus, it
+        represents the approximate impact of this task on the total build time,
+        with serialized or serializing steps typically ending up with much
+        longer weighted durations.
+        weighted_duration should always be the same or shorter than duration.
+        """
+        # Allow for modest floating-point errors
+        epsilon = 0.000002
+        if self.weighted_duration > self.Duration() + epsilon:
+            print("%s > %s?" % (self.weighted_duration, self.Duration()))
+        assert self.weighted_duration <= self.Duration() + epsilon
+        return self.weighted_duration
+
+    def DescribeTargets(self):
+        """Returns a printable string that summarizes the targets."""
+        # Some build steps generate dozens of outputs - handle them sanely.
+        # The max_length was chosen so that it can fit most of the long
+        # single-target names, while minimizing word wrapping.
+        result = ", ".join(self.targets)
+        max_length = 65
+        if len(result) > max_length:
+            result = result[:max_length] + "..."
+        return result
+
+
+# Copied with some modifications from ninjatracing
+def ReadTargets(log, show_all):
+    """Reads all targets from .ninja_log file |log_file|, sorted by duration.
+
+    The result is a list of Target objects."""
+    header = log.readline()
+    # Handle empty ninja_log gracefully by silently returning an empty list of
+    # targets.
+    if not header:
+        return []
+    assert header == "# ninja log v6\n", "unrecognized ninja log version %r" % header
+    targets_dict = {}
+    last_end_seen = 0.0
+    for line in log:
+        parts = line.strip().split("\t")
+        if len(parts) != 5:
+            # If ninja.exe is rudely halted then the .ninja_log file may be
+            # corrupt. Silently continue.
+            continue
+        start, end, _, name, cmdhash = parts  # Ignore restat.
+        # Convert from integral milliseconds to float seconds.
+        start = int(start) / 1000.0
+        end = int(end) / 1000.0
+        if not show_all and end < last_end_seen:
+            # An earlier time stamp means that this step is the first in a new
+            # build, possibly an incremental build. Throw away the previous
+            # data so that this new build will be displayed independently.
+            # This has to be done by comparing end times because records are
+            # written to the .ninja_log file when commands complete, so end
+            # times are guaranteed to be in order, but start times are not.
+            targets_dict = {}
+        target = None
+        if cmdhash in targets_dict:
+            target = targets_dict[cmdhash]
+            if not show_all and (target.start != start or target.end != end):
+                # If several builds in a row just run one or two build steps
+                # then the end times may not go backwards so the last build may
+                # not be detected as such. However in many cases there will be a
+                # build step repeated in the two builds and the changed
+                # start/stop points for that command, identified by the hash,
+                # can be used to detect and reset the target dictionary.
+                targets_dict = {}
+                target = None
+        if not target:
+            targets_dict[cmdhash] = target = Target(start, end)
+        last_end_seen = end
+        target.targets.append(name)
+    return list(targets_dict.values())
+
+
+def GetExtension(target, extra_patterns):
+    """Return the file extension that best represents a target.
+
+    For targets that generate multiple outputs it is important to return a
+    consistent 'canonical' extension. Ultimately the goal is to group build steps
+    by type."""
+    for output in target.targets:
+        if extra_patterns:
+            for fn_pattern in extra_patterns.split(";"):
+                if fnmatch.fnmatch(output, "*" + fn_pattern + "*"):
+                    return fn_pattern
+        # Not a true extension, but a good grouping.
+        if output.endswith("type_mappings"):
+            extension = "type_mappings"
+            break
+
+        # Capture two extensions if present. For example: file.javac.jar should
+        # be distinguished from file.interface.jar.
+        root, ext1 = os.path.splitext(output)
+        _, ext2 = os.path.splitext(root)
+        extension = ext2 + ext1  # Preserve the order in the file name.
+
+        if len(extension) == 0:
+            extension = "(no extension found)"
+
+        if ext1 in [".pdb", ".dll", ".exe"]:
+            extension = "PEFile (linking)"
+            # Make sure that .dll and .exe are grouped together and that the
+            # .dll.lib files don't cause these to be listed as libraries
+            break
+        if ext1 in [".so", ".TOC"]:
+            extension = ".so (linking)"
+            # Attempt to identify linking, avoid identifying as '.TOC'
+            break
+        # Make sure .obj files don't get categorized as mojo files
+        if ext1 in [".obj", ".o"]:
+            break
+        # Jars are the canonical output of java targets.
+        if ext1 == ".jar":
+            break
+        # Normalize all mojo related outputs to 'mojo'.
+        if output.count(".mojom") > 0:
+            extension = "mojo"
+            break
+    return extension
+
+
+def SummarizeEntries(entries, extra_step_types, elapsed_time_sorting):
+    """Print a summary of the passed in list of Target objects."""
+
+    # Create a list that is in order by time stamp and has entries for the
+    # beginning and ending of each build step (one time stamp may have multiple
+    # entries due to multiple steps starting/stopping at exactly the same time).
+    # Iterate through this list, keeping track of which tasks are running at all
+    # times. At each time step calculate a running total for weighted time so
+    # that when each task ends its own weighted time can easily be calculated.
+    task_start_stop_times = []
+
+    earliest = -1
+    latest = 0
+    total_cpu_time = 0
+    for target in entries:
+        if earliest < 0 or target.start < earliest:
+            earliest = target.start
+        if target.end > latest:
+            latest = target.end
+        total_cpu_time += target.Duration()
+        task_start_stop_times.append((target.start, "start", target))
+        task_start_stop_times.append((target.end, "stop", target))
+    length = latest - earliest
+    weighted_total = 0.0
+
+    # Sort by the time/type records and ignore |target|
+    task_start_stop_times.sort(key=lambda times: times[:2])
+    # Now we have all task start/stop times sorted by when they happen. If a
+    # task starts and stops on the same time stamp then the start will come
+    # first because of the alphabet, which is important for making this work
+    # correctly.
+    # Track the tasks which are currently running.
+    running_tasks = {}
+    # Record the time we have processed up to so we know how to calculate time
+    # deltas.
+    last_time = task_start_stop_times[0][0]
+    # Track the accumulated weighted time so that it can efficiently be added
+    # to individual tasks.
+    last_weighted_time = 0.0
+    # Scan all start/stop events.
+    for event in task_start_stop_times:
+        time, action_name, target = event
+        # Accumulate weighted time up to now.
+        num_running = len(running_tasks)
+        if num_running > 0:
+            # Update the total weighted time up to this moment.
+            last_weighted_time += (time - last_time) / float(num_running)
+        if action_name == "start":
+            # Record the total weighted task time when this task starts.
+            running_tasks[target] = last_weighted_time
+        if action_name == "stop":
+            # Record the change in the total weighted task time while this task
+            # ran.
+            weighted_duration = last_weighted_time - running_tasks[target]
+            target.SetWeightedDuration(weighted_duration)
+            weighted_total += weighted_duration
+            del running_tasks[target]
+        last_time = time
+    assert len(running_tasks) == 0
+
+    # Warn if the sum of weighted times is off by more than half a second.
+    if abs(length - weighted_total) > 500:
+        print(
+            "Warning: Possible corrupt ninja log, results may be "
+            "untrustworthy. Length = %.3f, weighted total = %.3f"
+            % (length, weighted_total)
+        )
+
+    # Print the slowest build steps:
+    print("    Longest build steps:")
+    if elapsed_time_sorting:
+        entries.sort(key=lambda x: x.Duration())
+    else:
+        entries.sort(key=lambda x: x.WeightedDuration())
+    for target in entries[-long_count:]:
+        print(
+            "      %8.1f weighted s to build %s (%.1f s elapsed time)"
+            % (target.WeightedDuration(), target.DescribeTargets(), target.Duration())
+        )
+
+    # Sum up the time by file extension/type of the output file
+    count_by_ext = {}
+    time_by_ext = {}
+    weighted_time_by_ext = {}
+    # Scan through all of the targets to build up per-extension statistics.
+    for target in entries:
+        extension = GetExtension(target, extra_step_types)
+        time_by_ext[extension] = time_by_ext.get(extension, 0) + target.Duration()
+        weighted_time_by_ext[extension] = (
+            weighted_time_by_ext.get(extension, 0) + target.WeightedDuration()
+        )
+        count_by_ext[extension] = count_by_ext.get(extension, 0) + 1
+
+    print("    Time by build-step type:")
+    # Copy to a list with extension name and total time swapped, to (time, ext)
+    if elapsed_time_sorting:
+        weighted_time_by_ext_sorted = sorted((y, x) for (x, y) in time_by_ext.items())
+    else:
+        weighted_time_by_ext_sorted = sorted(
+            (y, x) for (x, y) in weighted_time_by_ext.items()
+        )
+    # Print the slowest build target types:
+    for time, extension in weighted_time_by_ext_sorted[-long_ext_count:]:
+        print(
+            "      %8.1f s weighted time to generate %d %s files "
+            "(%1.1f s elapsed time sum)"
+            % (time, count_by_ext[extension], extension, time_by_ext[extension])
+        )
+
+    print(
+        "    %.1f s weighted time (%.1f s elapsed time sum, %1.1fx "
+        "parallelism)" % (length, total_cpu_time, total_cpu_time * 1.0 / length)
+    )
+    print(
+        "    %d build steps completed, average of %1.2f/s"
+        % (len(entries), len(entries) / (length))
+    )
+
+
+def main():
+    log_file = ".ninja_log"
+    metrics_file = "siso_metrics.json"
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-C", dest="build_directory", help="Build directory.")
+    parser.add_argument(
+        "-s",
+        "--step-types",
+        help="semicolon separated fnmatch patterns for build-step grouping",
+    )
+    parser.add_argument(
+        "-e",
+        "--elapsed_time_sorting",
+        default=False,
+        action="store_true",
+        help="Sort output by elapsed time instead of weighted time",
+    )
+    parser.add_argument("--log-file", help="specific ninja log file to analyze.")
+    args, _extra_args = parser.parse_known_args()
+    if args.build_directory:
+        log_file = os.path.join(args.build_directory, log_file)
+        metrics_file = os.path.join(args.build_directory, metrics_file)
+    if args.log_file:
+        log_file = args.log_file
+    if not args.step_types:
+        # Offer a convenient way to add extra step types automatically,
+        # including when this script is run by autoninja. get() returns None if
+        # the variable isn't set.
+        args.step_types = os.environ.get("chromium_step_types")
+    if args.step_types:
+        # Make room for the extra build types.
+        global long_ext_count
+        long_ext_count += len(args.step_types.split(";"))
+
+    if os.path.exists(metrics_file):
+        # Automatically handle summarizing siso builds.
+        cmd = ["siso.bat" if "win32" in sys.platform else "siso"]
+        cmd.extend(["metrics", "summary"])
+        if args.build_directory:
+            cmd.extend(["-C", args.build_directory])
+        if args.step_types:
+            cmd.extend(["--step_types", args.step_types])
+        if args.elapsed_time_sorting:
+            cmd.append("--elapsed_time_sorting")
+        subprocess.run(cmd)
+    else:
+        try:
+            with open(log_file, "r") as log:
+                entries = ReadTargets(log, False)
+                if entries:
+                    SummarizeEntries(
+                        entries, args.step_types, args.elapsed_time_sorting
+                    )
+        except IOError:
+            print("Log file %r not found, no build summary created." % log_file)
+            return errno.ENOENT
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/cccl_upstream/ci/nvrtc_libcudacxx.sh b/cccl_upstream/ci/nvrtc_libcudacxx.sh
new file mode 100755
index 00000000..3959241d
--- /dev/null
+++ b/cccl_upstream/ci/nvrtc_libcudacxx.sh
@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+
+# shellcheck source=ci/build_common.sh
+source "$(dirname "${BASH_SOURCE[0]}")/build_common.sh"
+
+print_environment_details
+
+
+PRESET="libcudacxx-nvrtc"
+CMAKE_OPTIONS=("-DCMAKE_CXX_STANDARD=${CXX_STANDARD}" "-DCMAKE_CUDA_STANDARD=${CXX_STANDARD}")
+
+configure_and_build_preset "libcudacxx NVRTC" "$PRESET" "${CMAKE_OPTIONS[@]}"
+
+sccache -z > /dev/null || :
+test_preset "libcudacxx NVRTC" "${PRESET}"
+sccache --show-adv-stats || :
diff --git a/cccl_upstream/ci/pretty_printing.sh b/cccl_upstream/ci/pretty_printing.sh
new file mode 100644
index 00000000..ab0f2c8b
--- /dev/null
+++ b/cccl_upstream/ci/pretty_printing.sh
@@ -0,0 +1,122 @@
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# Print "ARG=${ARG}" for all args.
+function print_var_values() {
+    # Iterate through the arguments
+    for var_name in "$@"; do
+        if [[ -z "$var_name" ]]; then
+            echo "Usage: print_var_values <variable_name1> <variable_name2> ..."
+            return 1
+        fi
+
+        # Dereference the variable and print the result
+        echo "$var_name=${!var_name:-(undefined)}"
+    done
+}
+
+# begin_group: Start a named section of log output, possibly with color.
+# Usage: begin_group "Group Name" [Color]
+#   Group Name: A string specifying the name of the group.
+#   Color (optional): ANSI color code to set text color. Default is blue (1;34).
+function begin_group() {
+    # See options for colors here: https://gist.github.com/JBlond/2fea43a3049b38287e5e9cefc87b2124
+    local blue="34"
+    local name="${1:-}"
+    local color="${2:-$blue}"
+
+    if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+        echo -e "::group::\e[${color}m${name}\e[0m"
+    else
+        echo -e "\e[${color}m================== ${name} ======================\e[0m"
+    fi
+}
+
+# end_group: End a named section of log output and print status based on exit status.
+# Usage: end_group "Group Name" [Exit Status]
+#   Group Name: A string specifying the name of the group.
+#   Exit Status (optional): The exit status of the command run within the group. Default is 0.
+function end_group() {
+    local name="${1:-}"
+    local build_status="${2:-0}"
+    local duration="${3:-}"
+    local red="31"
+    local blue="34"
+
+    if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+        echo "::endgroup::"
+
+        if [[ "$build_status" -ne 0 ]]; then
+            echo -e "::error::\e[${red}m ${name} - Failed (⬆️ click above for full log ⬆️)\e[0m"
+        fi
+    else
+        if [[ "$build_status" -ne 0 ]]; then
+            echo -e "\e[${red}m================== End ${name} - Failed${duration:+ - Duration: ${duration}s} ==================\e[0m"
+        else
+            echo -e "\e[${blue}m================== End ${name} - Success${duration:+ - Duration: ${duration}s} ==================\n\e[0m"
+        fi
+    fi
+}
+
+declare -A command_durations
+
+# Runs a command within a named group, handles the exit status, and prints appropriate messages based on the result.
+# Usage: run_command "Group Name" command [arguments...]
+function run_command() {
+    local group_name="${1:-}"
+    shift
+    local command=("$@")
+    local status
+
+    begin_group "$group_name"
+    echo "Working directory: $(pwd)"
+    echo "Running command: ${command[*]}"
+    set +e
+    local start_time
+    start_time=$(date +%s)
+    # If RUN_COMMAND_RETRY_PARAMS is set to "<retries> <sleep_time>", use retry.sh to run the command:
+    if [[ -v RUN_COMMAND_RETRY_PARAMS && "${#RUN_COMMAND_RETRY_PARAMS[@]}" -eq 2 ]]; then
+        status=0
+        "$ci_dir/util/retry.sh" "${RUN_COMMAND_RETRY_PARAMS[@]}" "${command[@]}" || status=$?
+    else
+        status=0
+        "${command[@]}" || status=$?
+    fi
+    local end_time
+    end_time=$(date +%s)
+    set -e
+    local duration=$((end_time - start_time))
+    end_group "$group_name" "$status" "$duration"
+    command_durations["$group_name"]=$duration
+    return "$status"
+}
+
+function string_width() {
+    local str="$1"
+    echo "$str" | awk '{print length}'
+}
+
+function print_time_summary() {
+    local max_length=0
+    local group
+
+    # Find the longest group name for formatting
+    for group in "${!command_durations[@]}"; do
+        local group_length
+        group_length=$(echo "$group" | awk '{print length}')
+        if [[ "$group_length" -gt "$max_length" ]]; then
+            max_length=$group_length
+        fi
+    done
+
+    if [[ "$max_length" -eq 0 ]]; then
+        return
+    fi
+
+    echo "Time Summary:"
+    for group in "${!command_durations[@]}"; do
+        printf "%-${max_length}s : %s seconds\n" "$group" "${command_durations[$group]}"
+    done
+
+    # Clear the array of timing info
+    declare -gA command_durations=()
+}
diff --git a/cccl_upstream/ci/project_files_and_dependencies.yaml b/cccl_upstream/ci/project_files_and_dependencies.yaml
new file mode 100644
index 00000000..b3a18356
--- /dev/null
+++ b/cccl_upstream/ci/project_files_and_dependencies.yaml
@@ -0,0 +1,287 @@
+# Configuration for CCCL project change detection.
+# Projects are declared in this file along with their human-friendly names, path
+# matching patterns, and dependency relationships. Every path pattern is
+# evaluated as a regular expression anchored to the repository root.
+#
+# Changed files that satisfy a project's include_regexes, exclude_regexes, and
+# exclude_project_files checks will mark that project as "dirty". The current project
+# will be added to the FULL_BUILD if it has dirty files or if any of its
+# full_dependencies are dirty.
+#
+# Additionally, if any of a project's lite_dependencies are dirty, the current project
+# will be added to the LITE_BUILD.
+#
+# If any transitive dependencies are dirty (eg. a full or lite dependency has
+# dirty dependencies of its own), the current project will also be added to the LITE_BUILD.
+#
+# Projects with large dependency trees are split into "public" and "internal" sub-projects.
+# The "public" sub-project contains only the public API files, while the "internal"
+# sub-project contains tests, examples, and other non-public files. This allows changes to
+# internal files to avoid triggering rebuilds of dependent projects when the public API
+# is not changed.
+#
+# Layout:
+#   project_key:          mapping from project key to configuration
+#     name:               display name used in logs/reports
+#     matrix_project:     project name to use for matrix generation. If not provided,
+#                           the project is considered internal-only and will not be listed
+#                           in the FULL_BUILD or LITE_BUILD outputs.
+#     lite_dependencies:  Add the current project to the `lite` matrix if any of these
+#                           dependencies are dirty.
+#     full_dependencies:  Add the current project to the `full` matrix if any of these
+#                           dependencies are dirty.
+#     include_regexes:    regexes that match files for this project
+#     exclude_regexes:    regexes that exclude files from the included set
+#     exclude_project_files: list of other project keys whose files should be
+#                           excluded from this project
+
+projects:
+  # `core` collects any dirty files that are not matched by other projects. It
+  # never appears in dependency lists because the script treats it specially,
+  # triggering a full rebuild of all projects when any core file is dirty.
+  core:
+    name: "CCCL Infrastructure"
+
+  libcudacxx_public:
+    name: "libcu++ Public API"
+    lite_dependencies: ['thrust_public', 'cub_public']
+    full_dependencies: []
+    include_regexes: ["libcudacxx/include/"]
+
+  libcudacxx_internal:
+    name: "libcu++ Tests/Infra"
+    matrix_project: "libcudacxx"
+    lite_dependencies: [c2h]
+    full_dependencies: [libcudacxx_public]
+    include_regexes: ["libcudacxx/"]
+    exclude_project_files: [libcudacxx_public]
+
+  cub_public:
+    name: "CUB Public API"
+    lite_dependencies: [libcudacxx_public, thrust_public]
+    full_dependencies: []
+    include_regexes: ["cub/cub/"]
+
+  cub_internal:
+    name: "CUB Tests/Infra"
+    matrix_project: "cub"
+    lite_dependencies: [c2h, nvbench_helper]
+    full_dependencies: [cub_public]
+    include_regexes: ["cub/"]
+    exclude_project_files: [cub_public]
+
+  thrust_public:
+    name: "Thrust Public API"
+    lite_dependencies: [libcudacxx_public, cub_public]
+    full_dependencies: []
+    include_regexes: ["thrust/thrust/"]
+
+  thrust_internal:
+    name: "Thrust Tests/Infra"
+    matrix_project: "thrust"
+    lite_dependencies: [nvbench_helper]
+    full_dependencies: [thrust_public]
+    include_regexes: ["thrust/"]
+    exclude_project_files: [thrust_public]
+
+  cudax_public:
+    name: "CUDA Experimental Public API"
+    lite_dependencies: [libcudacxx_public, thrust_public, cub_public]
+    full_dependencies: []
+    include_regexes: ["cudax/include/"]
+
+  cudax_internal:
+    name: "CUDA Experimental Tests/Infra"
+    matrix_project: "cudax"
+    lite_dependencies: [c2h, nvbench_helper]
+    full_dependencies: [cudax_public]
+    include_regexes: ["cudax/"]
+    exclude_project_files: [cudax_public]
+
+  cccl_c_parallel_public:
+    name: "CCCL C Parallel Library Public API"
+    lite_dependencies: [libcudacxx_public, cub_public, thrust_public]
+    full_dependencies: []
+    include_regexes:
+      - "c/parallel/include/"
+      - "c/parallel/src/"
+
+  cccl_c_parallel_internal:
+    name: "CCCL C Parallel Library Tests/Infra"
+    matrix_project: "cccl_c_parallel"
+    lite_dependencies: [c2h]
+    full_dependencies: [cccl_c_parallel_public]
+    include_regexes: ["c/parallel/"]
+    exclude_project_files: [cccl_c_parallel_public]
+
+  cccl_c_parallel_v2:
+    name: "CCCL C Parallel Library v2 (HostJIT)"
+    matrix_project: "cccl_c_parallel_v2"
+    # v2 depends on libcudacxx, cub, and thrust headers (it JIT-compiles
+    # CUB's host+device code via HostJIT). Any change to those should trigger
+    # v2 to run.
+    lite_dependencies: [libcudacxx_public, cub_public, thrust_public, c2h]
+    full_dependencies: []
+    include_regexes:
+      - "c/parallel\\.v2/"
+
+  python_v2:
+    name: "Python (cuda.compute on v2/HostJIT)"
+    matrix_project: "python_v2"
+    # cccl_c_parallel_v2 already pulls in libcudacxx/cub/thrust, so listing
+    # it here transitively triggers python_v2 on any of those upstream
+    # changes too. Direct includes catch Python-only edits.
+    lite_dependencies: [cccl_c_parallel_v2]
+    full_dependencies: []
+    include_regexes:
+      - "python/cuda_cccl/"
+      - "pyproject\\.toml"
+
+  python_tsan:
+    name: "Python (cuda.compute free-threaded ThreadSanitizer)"
+    matrix_project: "python_tsan"
+    # Same v1 sources as `python`, rebuilt with ThreadSanitizer. Must be listed
+    # here or the per-PR python_tsan job is pruned as never-dirty and never runs.
+    # cccl_c_parallel_public covers c/parallel/{src,include}; c/parallel/CMakeLists.txt
+    # is listed directly because the TSan build config (the -fsanitize=thread option)
+    # lives there and is owned by cccl_c_parallel_internal, which this lane otherwise
+    # does not depend on -- without it a change to the TSan flags would skip this lane.
+    lite_dependencies: [cccl_c_parallel_public]
+    full_dependencies: []
+    include_regexes:
+      - "python/cuda_cccl/"
+      - "pyproject\\.toml"
+      - "c/parallel/CMakeLists\\.txt"
+
+  cccl_c_stf:
+    name: "CCCL C CUDASTF Library"
+    matrix_project: "cccl_c_stf"
+    lite_dependencies: [libcudacxx_public, cudax_public, c2h]
+    full_dependencies: []
+    include_regexes:  ["c/experimental/stf/"]
+    exclude_regexes: []
+
+  python:
+    name: "Python"
+    matrix_project: "python"
+    lite_dependencies: [cccl_c_parallel_public]
+    full_dependencies: []
+    include_regexes:
+    - "python/"
+    - "pyproject.toml"
+
+  packaging:
+    name: "CCCL Packaging"
+    matrix_project: "packaging"
+    # Anything that affects CMake packages / install rules:
+    lite_dependencies:
+      - libcudacxx_internal
+      - libcudacxx_public
+      - cub_internal
+      - cub_public
+      - thrust_internal
+      - thrust_public
+      - cudax_internal
+      - cudax_public
+    full_dependencies: []
+    include_regexes:
+      - "examples/"
+      - "test/cmake/"
+      - "ci/test/"
+
+  stdpar:
+    name: "stdpar"
+    matrix_project: "stdpar"
+    lite_dependencies: [thrust_public]
+    full_dependencies: []
+    include_regexes: ["test/stdpar/"]
+
+  c2h:
+    name: "Catch2Helper"
+    lite_dependencies: [libcudacxx_public, cub_public, thrust_public]
+    full_dependencies: []
+    include_regexes: ["c2h/"]
+
+  nvbench_helper:
+    name: "NVBench Helper"
+    matrix_project: "nvbench_helper"
+    lite_dependencies: [libcudacxx_public, cub_public, thrust_public]
+    full_dependencies: []
+    include_regexes:
+      - "nvbench_helper/"
+      # CCCLBenchmarkRegistry.cmake is effectively part of nvbench_helper:
+      - "benchmarks/cmake/"
+
+  nvrtcc:
+    name: "nvrtcc"
+    matrix_project: "nvrtcc"
+    lite_dependencies: []
+    full_dependencies: []
+    include_regexes: ["nvrtcc/"]
+
+  # This is a dummy project, and only really serves to encode dependencies. Any time any
+  # of the C/C++ projects are modified, we want to launch one (and only one) clang-tidy
+  # job for the whole of CCCL.
+  tidy:
+    name: "clang-tidy"
+    matrix_project: "tidy"
+    full_dependencies:
+      - libcudacxx_public
+      - libcudacxx_internal
+      - cub_public
+      - cub_internal
+      - thrust_public
+      - thrust_internal
+      - cudax_public
+      - cudax_internal
+      - cccl_c_parallel_public
+      - cccl_c_parallel_internal
+      - cccl_c_parallel_v2
+      - cccl_c_stf
+      - stdpar
+      - c2h
+      - nvrtcc
+    include_regexes: [".clang-tidy"]
+
+# Files matching any of these regexes will be ignored globally.
+ignore_regexes:
+  - '.+\.md$'
+  - '\.branch_notes/'
+  - '\.coderabbit\.yaml'
+  - '\.clang-format'
+  - '\.clangd'
+  # Do not add .clang-tidy to this list. If we modify .clang-tidy to either add or remove
+  # a check we still want CI to run the clang-tidy job even if it results in no C/C++ code
+  # changes.
+  #
+  # - '\.clang-tidy'
+  - '\.devcontainer/img'
+  - '\.git-blame-ignore-revs'
+  - '\.github/actions/docs-build'
+  - '\.github/CODEOWNERS'
+  - '\.github/copy-pr-bot\.yaml'
+  - '\.github/ISSUE_TEMPLATE/'
+  - '\.github/problem-matchers/problem-matcher\.json'
+  - '\.github/workflows/backport-prs\.yml'
+  - '\.github/workflows/bench.*\.yml'
+  - '\.github/workflows/compile-time-bench\.yml'
+  - '\.github/workflows/build-docs\.yml'
+  - '\.github/workflows/build-matx\.yml'
+  - '\.github/workflows/build-pytorch\.yml'
+  - '\.github/workflows/build-rapids\.yml'
+  - '\.github/workflows/git-bisect\.yml'
+  - '\.github/workflows/project_automation.*\.yml'
+  - '\.github/workflows/release.*\.yml'
+  - '\.github/workflows/triage_rotation\.yml'
+  - '\.github/workflows/update_branch_version\.yml'
+  - '\.github/workflows/verify-devcontainers\.yml'
+  - '\.gitignore'
+  - "benchmarks/scripts/" # Python+bash scripts for running benchmarks, not used in tests.
+  - 'cccl-version.json'
+  - 'ci/bench/'
+  - 'ci/bench.+yaml'
+  - 'ci/matx/'
+  - 'ci/pytorch/'
+  - 'ci/rapids/'
+  - 'docs/'
+  - 'LICENSE'
diff --git a/cccl_upstream/ci/pyenv_helper.sh b/cccl_upstream/ci/pyenv_helper.sh
new file mode 100644
index 00000000..fffe9edc
--- /dev/null
+++ b/cccl_upstream/ci/pyenv_helper.sh
@@ -0,0 +1,77 @@
+setup_python_env() {
+    local py_version=$1
+
+    # Source pretty_printing.sh for begin_group/end_group helpers
+    local script_dir
+    script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+    # shellcheck source=ci/pretty_printing.sh
+    source "${script_dir}/pretty_printing.sh"
+
+    begin_group "🐍 Setting up Python ${py_version} (uv)"
+
+    # Install uv if not present
+    if ! command -v uv &> /dev/null; then
+        curl -LsSf https://astral.sh/uv/install.sh | sh
+        export PATH="$HOME/.local/bin:$PATH"
+    fi
+
+    # Create a venv with the requested Python version.
+    # uv downloads a pre-built CPython binary automatically — no compilation needed.
+    uv venv --seed --python "${py_version}" "${HOME}/.cccl-venv"
+
+    # Windows venvs use Scripts/, Linux/macOS use bin/
+    if [[ -f "${HOME}/.cccl-venv/Scripts/activate" ]]; then
+        #shellcheck disable=SC1091
+        source "${HOME}/.cccl-venv/Scripts/activate"
+    else
+        #shellcheck disable=SC1091
+        source "${HOME}/.cccl-venv/bin/activate"
+    fi
+
+    end_group "🐍 Setting up Python ${py_version} (uv)"
+}
+
+# Pin the cuda-toolkit wheels to the container's CTK major.minor (read from nvcc)
+# via PIP_CONSTRAINT when the mode ($1) is "pinned" (the default; empty also means
+# pinned). "latest" and "sysctk" leave it unpinned; any other value is a hard
+# error. This is the lane's mode gate -- it runs before ctk_extra_flavor in every
+# script, so ctk_extra_flavor can assume the mode is already valid. Also sets and
+# exports cuda_version / cuda_major_version; the caller uses cuda_major_version in
+# the pip-extra name (e.g. minimal-cu${cuda_major_version}).
+pin_cuda_toolkit() {
+    cuda_version=$(nvcc --version | grep release | awk '{print $6}' | tr -d ',' | cut -d '.' -f 1-2 | cut -d 'V' -f 2)
+    cuda_major_version=$(echo "$cuda_version" | cut -d '.' -f 1)
+    export cuda_version cuda_major_version
+
+    local mode="${1:-pinned}"
+    case "${mode,,}" in
+        pinned)
+            export PIP_CONSTRAINT="${TMPDIR:-/tmp}/ctk-constraint.txt"
+            echo "cuda-toolkit==${cuda_version}.*" > "${PIP_CONSTRAINT}"
+            ;;
+        latest | sysctk)
+            # No pin. Clear any inherited constraint so it cannot affect the
+            # resolve (latest tests the newest minor; sysctk installs no
+            # cuda-toolkit wheel at all).
+            unset PIP_CONSTRAINT
+            ;;
+        *)
+            echo "ERROR: invalid ctk mode '${mode}' (expected pinned|latest|sysctk)" >&2
+            return 1
+            ;;
+    esac
+}
+
+# Echoes the pip-extra toolkit "flavor" for the mode ($1): "sysctk" when the mode
+# is sysctk (rely on the system-provided CUDA toolkit) or "cu" otherwise
+# (pip-installed toolkit). The mode is validated by pin_cuda_toolkit, which every
+# lane calls first. Combine with the CUDA major, e.g.
+# "minimal-$(ctk_extra_flavor "${ctk_mode}")${cuda_major_version}" -> minimal-sysctk12.
+ctk_extra_flavor() {
+    local mode="${1:-}"
+    if [[ "${mode,,}" == "sysctk" ]]; then
+        echo "sysctk"
+    else
+        echo "cu"
+    fi
+}
diff --git a/cccl_upstream/ci/pytorch/build_pytorch.sh b/cccl_upstream/ci/pytorch/build_pytorch.sh
new file mode 100755
index 00000000..fc08c8c2
--- /dev/null
+++ b/cccl_upstream/ci/pytorch/build_pytorch.sh
@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+readonly pytorch_repo=https://github.com/pytorch/pytorch.git
+readonly pytorch_branch=main
+
+# Ensure the script is being executed in the root cccl directory:
+cd "$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )/../..";
+readonly cccl_repo="${PWD}"
+
+log_vars() {
+  for var in "$@"; do
+    echo "${var}=${!var}"
+  done
+}
+
+# Define CCCL_TAG to override the default CCCL SHA. Otherwise the current HEAD of the local checkout is used.
+echo "CCCL_TAG (override): ${CCCL_TAG-}";
+if test -n "${CCCL_TAG-}"; then
+    # If CCCL_TAG is defined, fetch it to the local checkout
+    git -C "${cccl_repo}" fetch origin "${CCCL_TAG}";
+    cccl_sha="$(git -C "${cccl_repo}" rev-parse FETCH_HEAD)";
+else
+    cccl_sha="$(git -C "${cccl_repo}" rev-parse HEAD)";
+fi
+
+readonly workdir="${cccl_repo}/build/${CCCL_BUILD_INFIX:-}/pytorch"
+
+log_vars \
+  pytorch_repo pytorch_branch \
+  cccl_repo cccl_sha \
+  workdir
+
+mkdir -p "${workdir}"
+cd "${workdir}"
+echo "Working in ${workdir}"
+
+echo "::group::Cloning CCCL..."
+rm -rf cccl
+git clone "${cccl_repo}"
+git -C cccl checkout "${cccl_sha}"
+echo "CCCL HEAD:"
+git -C cccl log -1 --format=short
+echo "::endgroup::"
+
+# Setup a CUDA environment with the requested CCCL.
+# Use a local directory to avoid modifying the actual CUDA install:
+echo "::group::Setting up clone of CUDA environment with custom CCCL..."
+(
+  set -x
+  rm -rf ./cuda
+  cp -Hr /usr/local/cuda ./cuda
+  rm -rf ./cuda/include/cccl/*
+  cccl/ci/install_cccl.sh ./cccl-install > /dev/null
+  cp -r ./cccl-install/include/* ./cuda/include/cccl
+)
+export PATH="$PWD/cuda/bin:$PATH"
+export CUDA_HOME="$PWD/cuda"
+export CUDA_PATH="$PWD/cuda"
+command -v nvcc
+nvcc --version
+echo "::endgroup::"
+
+echo "::group::Cloning PyTorch..."
+rm -rf pytorch
+git clone "${pytorch_repo}" -b "${pytorch_branch}" --recursive --depth 1
+echo "PyTorch HEAD:"
+git -C pytorch log -1 --format=short
+echo "::endgroup::"
+
+echo "::group::Installing PyTorch build dependencies..."
+pytorch_root="$PWD/pytorch"
+export PYTHONPATH="${pytorch_root}:${pytorch_root}/tools:${PYTHONPATH:-}"
+pip install -r "${pytorch_root}/requirements-build.txt"
+echo "::endgroup::"
+
+echo "::group::Configuring PyTorch..."
+rm -rf build
+mkdir build
+declare -a cmake_args=(
+  "-DUSE_NCCL=OFF"
+  # Need to define this explicitly, torch's FindCUDA logic adds ancient arches if left undefined:
+  "-DTORCH_CUDA_ARCH_LIST=7.5;8.0;9.0;10.0;12.0"
+)
+SCCACHE_NO_DIST_COMPILE=1 cmake -S ./pytorch -B ./build -G Ninja "${cmake_args[@]}"
+echo "::endgroup::"
+
+# Verify that the configured build is using the custom CUDA dir for CTK and nvcc:
+if ! grep -q "CUDA_TOOLKIT_ROOT_DIR:PATH=$PWD/cuda" ./build/CMakeCache.txt; then
+    echo "Error: CUDA_TOOLKIT_ROOT_DIR does not point to the custom CUDA";
+    exit 1;
+fi
+if ! grep -q "CUDA_NVCC_EXECUTABLE:FILEPATH=$PWD/cuda/bin/nvcc" ./build/CMakeCache.txt; then
+    echo "Error: CUDA_NVCC_EXECUTABLE does not point to the custom CUDA";
+    exit 1;
+fi
+
+# This builds a bunch of unnecessary targets. Leaving here to use as a fallback if the
+# ninja target extraction below starts failing:
+# echo "::group::Building torch_cuda target..."
+# cmake --build ./build/ --target torch_cuda
+# echo "::endgroup::"
+
+# This cuts the number of built targets roughly in half:
+echo "::group::Extracting cuda targets from build.ninja..."
+# Query ninja for all object files built from CUDA source files
+# that are part of the torch_cuda library:
+ninja -C ./build -t query lib/libtorch_cuda.so |
+  grep -E "torch_cuda\\.dir/.*\\.cu\\.o$" |
+  sort | uniq | tee build/cuda_targets.txt
+# At the time this script was written, there were 311 cuda targets.
+# Check that there are at least 100 detected targets, otherwise fail.
+num_targets=$(wc -l < build/cuda_targets.txt)
+if test "$num_targets" -lt 100; then
+    echo "Error: extracted cuda targets count is less than 100! ($num_targets)";
+    echo "This likely indicates a failure to extract the targets from ninja.";
+    exit 1;
+fi
+echo "::endgroup::"
+
+echo "::group::Building $num_targets pytorch CUDA targets with custom CCCL..."
+torch_cuda_targets="$(xargs -a build/cuda_targets.txt)"
+declare -a torch_cuda_targets="($torch_cuda_targets)"
+
+time ninja -C ./build "${torch_cuda_targets[@]}" "-j${PARALLEL_LEVEL:-}"
+
+echo "::endgroup::"
+
+echo "PyTorch CUDA targets built successfully with custom CCCL."
diff --git a/cccl_upstream/ci/rapids/cuda13.3-conda/devcontainer.json b/cccl_upstream/ci/rapids/cuda13.3-conda/devcontainer.json
new file mode 100644
index 00000000..6c138a8e
--- /dev/null
+++ b/cccl_upstream/ci/rapids/cuda13.3-conda/devcontainer.json
@@ -0,0 +1,124 @@
+{
+  "image": "rapidsai/devcontainers:26.10-cpp-mambaforge",
+  "runArgs": [
+    "--init",
+    "--rm",
+    "--name",
+    "${localEnv:USER:anon}-${localWorkspaceFolderBasename}-rapids-26.10-cuda13.3-conda",
+    "--ulimit",
+    "nofile=500000"
+  ],
+  "hostRequirements": {
+    "gpu": "optional"
+  },
+  "containerEnv": {
+    "AWS_ROLE_ARN": "arn:aws:iam::279114543810:role/nv-gha-token-sccache-devs",
+    "CI": "${localEnv:CI}",
+    "CUDA_VERSION": "13.3",
+    "CUDAARCHS": "75-real",
+    "DEFAULT_CONDA_ENV": "rapids",
+    "DEVCONTAINER_UTILS_ENABLE_SCCACHE_DIST": "true",
+    "HISTFILE": "/home/coder/.cache/._bash_history",
+    "INFER_NUM_DEVICE_ARCHITECTURES": "1",
+    "LIBCUDF_KERNEL_CACHE_PATH": "/home/coder/cudf/cpp/build/latest/jitify_cache",
+    "MAX_DEVICE_OBJ_TO_COMPILE_IN_PARALLEL": "20",
+    "PYTHON_PACKAGE_MANAGER": "conda",
+    "PYTHONDONTWRITEBYTECODE": "1",
+    "PYTHONSAFEPATH": "1",
+    "PYTHONUNBUFFERED": "1",
+    "RAPIDS_cmake_GIT_REPO": "${localEnv:RAPIDS_cmake_GIT_REPO}",
+    "RAPIDS_cudf_GIT_REPO": "${localEnv:RAPIDS_cudf_GIT_REPO}",
+    "RAPIDS_cugraph_GIT_REPO": "${localEnv:RAPIDS_cugraph_GIT_REPO}",
+    "RAPIDS_cugraph_gnn_GIT_REPO": "${localEnv:RAPIDS_cugraph_gnn_GIT_REPO}",
+    "RAPIDS_cuml_GIT_REPO": "${localEnv:RAPIDS_cuml_GIT_REPO}",
+    "RAPIDS_cuopt_GIT_REPO": "${localEnv:RAPIDS_cuopt_GIT_REPO}",
+    "RAPIDS_cuvs_GIT_REPO": "${localEnv:RAPIDS_cuvs_GIT_REPO}",
+    "RAPIDS_kvikio_GIT_REPO": "${localEnv:RAPIDS_kvikio_GIT_REPO}",
+    "RAPIDS_LIBS": "${localEnv:RAPIDS_LIBS}",
+    "RAPIDS_nvforest_GIT_REPO": "${localEnv:RAPIDS_nvforest_GIT_REPO}",
+    "RAPIDS_raft_GIT_REPO": "${localEnv:RAPIDS_raft_GIT_REPO}",
+    "RAPIDS_rmm_GIT_REPO": "${localEnv:RAPIDS_rmm_GIT_REPO}",
+    "RAPIDS_ucxx_GIT_REPO": "${localEnv:RAPIDS_ucxx_GIT_REPO}",
+    "SCCACHE_BUCKET": "rapids-sccache-devs",
+    "SCCACHE_DIST_FALLBACK_TO_LOCAL_COMPILE": "${localEnv:SCCACHE_DIST_FALLBACK_TO_LOCAL_COMPILE:true}",
+    "SCCACHE_DIST_MAX_RETRIES": "${localEnv:SCCACHE_DIST_MAX_RETRIES:4}",
+    "SCCACHE_DIST_REQUEST_TIMEOUT": "${localEnv:SCCACHE_DIST_REQUEST_TIMEOUT:7140}",
+    "SCCACHE_IDLE_TIMEOUT": "${localEnv:SCCACHE_IDLE_TIMEOUT:0}",
+    "SCCACHE_REGION": "us-east-2",
+    "SCCACHE_S3_USE_PREPROCESSOR_CACHE_MODE": "true",
+    "SCCACHE_SERVER_LOG": "${localEnv:SCCACHE_SERVER_LOG:sccache=debug}"
+  },
+  "initializeCommand": [
+    "/bin/bash",
+    "-c",
+    "mkdir -m 0755 -p ${localWorkspaceFolder}/.{aws,cache,config,local/state} ${localWorkspaceFolder}/ci/rapids/.{conda,log/devcontainer-utils} ${localWorkspaceFolder}/ci/rapids/.repos/{rmm,kvikio,ucxx,cudf,rapidsmpf,raft,cuvs,nvforest,cuml,cugraph,cugraph-gnn,cuopt}"
+  ],
+  "postCreateCommand": [
+    "/bin/bash",
+    "-c",
+    "if [ ${CI:-false} = 'false' ]; then . /home/coder/cccl/ci/rapids/post-create-command.sh; fi; if test -z \"${DISABLE_SCCACHE:+x}\"; then echo \"export SCCACHE_DIST_URL='https://$(uname -m | sed -e 's/x86_/amd/' -e 's/aarch/arm/').linux.sccache.rapids.nvidia.com'\" >> /home/coder/.bashrc; fi"
+  ],
+  "postAttachCommand": [
+    "/bin/bash",
+    "-c",
+    "if [ ${CODESPACES:-false} = 'true' ]; then . devcontainer-utils-post-attach-command; fi"
+  ],
+  "workspaceFolder": "/home/coder/cccl",
+  "workspaceMount": "source=${localWorkspaceFolder},target=/home/coder/cccl,type=bind",
+  "mounts": [
+    "source=/etc/timezone,target=/etc/timezone,type=bind",
+    "source=/etc/localtime,target=/etc/localtime,type=bind",
+    "source=${localWorkspaceFolder}/.aws,target=/home/coder/.aws,type=bind",
+    "source=${localWorkspaceFolder}/.cache,target=/home/coder/.cache,type=bind",
+    "source=${localWorkspaceFolder}/.config,target=/home/coder/.config,type=bind",
+    "source=${localWorkspaceFolder}/ci/rapids/.repos/rmm,target=/home/coder/rmm,type=bind",
+    "source=${localWorkspaceFolder}/ci/rapids/.repos/kvikio,target=/home/coder/kvikio,type=bind",
+    "source=${localWorkspaceFolder}/ci/rapids/.repos/ucxx,target=/home/coder/ucxx,type=bind",
+    "source=${localWorkspaceFolder}/ci/rapids/.repos/cudf,target=/home/coder/cudf,type=bind",
+    "source=${localWorkspaceFolder}/ci/rapids/.repos/rapidsmpf,target=/home/coder/rapidsmpf,type=bind",
+    "source=${localWorkspaceFolder}/ci/rapids/.repos/raft,target=/home/coder/raft,type=bind",
+    "source=${localWorkspaceFolder}/ci/rapids/.repos/cuvs,target=/home/coder/cuvs,type=bind",
+    "source=${localWorkspaceFolder}/ci/rapids/.repos/nvforest,target=/home/coder/nvforest,type=bind",
+    "source=${localWorkspaceFolder}/ci/rapids/.repos/cuml,target=/home/coder/cuml,type=bind",
+    "source=${localWorkspaceFolder}/ci/rapids/.repos/cugraph,target=/home/coder/cugraph,type=bind",
+    "source=${localWorkspaceFolder}/ci/rapids/.repos/cugraph-gnn,target=/home/coder/cugraph-gnn,type=bind",
+    "source=${localWorkspaceFolder}/ci/rapids/.repos/cuopt,target=/home/coder/cuopt,type=bind",
+    "source=${localWorkspaceFolder}/ci/rapids/.conda,target=/home/coder/.conda,type=bind",
+    "source=${localWorkspaceFolder}/ci/rapids/.log/devcontainer-utils,target=/var/log/devcontainer-utils,type=bind"
+  ],
+  "features": {
+    "ghcr.io/rapidsai/devcontainers/features/rapids-build-utils:26.10": {}
+  },
+  "overrideFeatureInstallOrder": [
+    "ghcr.io/rapidsai/devcontainers/features/rapids-build-utils"
+  ],
+  "customizations": {
+    "vscode": {
+      "extensions": [
+        "augustocdias.tasks-shell-input",
+        "ms-python.flake8",
+        "nvidia.nsight-vscode-edition"
+      ],
+      "files.watcherExclude": {
+        "**/build/**": true,
+        "**/_skbuild/**": true,
+        "**/target/**": true,
+        "/home/coder/.aws/**/*": true,
+        "/home/coder/.cache/**/*": true,
+        "/home/coder/.conda/**/*": true,
+        "/home/coder/.local/share/**/*": true,
+        "/home/coder/.vscode-server/**/*": true
+      },
+      "search.exclude": {
+        "**/build/**": true,
+        "**/_skbuild/**": true,
+        "**/*.code-search": true,
+        "/home/coder/.aws/**/*": true,
+        "/home/coder/.cache/**/*": true,
+        "/home/coder/.conda/**/*": true,
+        "/home/coder/.local/share/**/*": true,
+        "/home/coder/.vscode-server/**/*": true
+      }
+    }
+  }
+}
diff --git a/cccl_upstream/ci/rapids/post-create-command.sh b/cccl_upstream/ci/rapids/post-create-command.sh
new file mode 100755
index 00000000..db427da1
--- /dev/null
+++ b/cccl_upstream/ci/rapids/post-create-command.sh
@@ -0,0 +1,146 @@
+#!/usr/bin/env bash
+
+PROJECT_MANIFEST_YML="${PROJECT_MANIFEST_YML:-"/opt/rapids-build-utils/manifest.yaml"}"
+
+_restore_original_manifest() {
+    if ! test -f "/tmp/manifest.yaml.orig"; then
+        cp "${PROJECT_MANIFEST_YML}" /tmp/manifest.yaml.orig;
+    fi
+    sudo cp /tmp/manifest.yaml.orig "${PROJECT_MANIFEST_YML}";
+}
+
+_apply_manifest_modifications() {
+    # Restore the original manifest.yaml
+    _restore_original_manifest;
+    # Remove unnecessary libs from manifest.yaml
+    _prune_libs_from_manifest;
+    # Update manifest.yaml repo git info
+    _update_repo_git_info;
+    # Create rapids-cmake override JSON file and update default CMake arguments in manifest.yaml
+    _create_rapids_cmake_override_json;
+    # Print the entire manifest.yaml after modifications
+    cat "${PROJECT_MANIFEST_YML}";
+    # Regenerate the RAPIDS build scripts from the new manifest.yaml
+    rapids-generate-scripts;
+}
+
+_prune_libs_from_manifest() {
+    local -;
+    set -euo pipefail;
+    if test -n "${RAPIDS_LIBS-}"; then
+        local -a filters="(${RAPIDS_LIBS})";
+        # prefix each element
+        filters=("${filters[@]/#/'"'}");
+        # suffix each element
+        filters=("${filters[@]/%/'",'}");
+        # Remove trailing comma
+        local -r filters_str="$(cut -d',' -f1-${#filters[@]} <<< "${filters[*]}")";
+        sudo yq -i ".repos |= filter(.cpp[].name | contains(${filters_str}))" "${PROJECT_MANIFEST_YML}";
+    fi
+}
+
+# shellcheck disable=SC2016
+_update_repo_git_info() {
+    local -;
+    set -euo pipefail;
+    yq '.repos[].name' "${PROJECT_MANIFEST_YML}" \
+  | xargs -r -n1 bash -c '
+    var="RAPIDS_${1//-/_}_GIT_REPO";
+    if test -v "${var}" && test -n "${!var}"; then
+        sudo yq -i "(.repos[] | select(.name == \"$1\") | .git) *= ${!var}" "${0}";
+    fi' "${PROJECT_MANIFEST_YML}";
+}
+
+_create_rapids_cmake_override_json() {
+    local -;
+    set -euo pipefail;
+    local rapids_cmake_tag=;
+    local rapids_cmake_upstream=;
+    if test -n "${RAPIDS_cmake_GIT_REPO-}"; then
+        rapids_cmake_tag="$(jq -r '.tag' <<< "${RAPIDS_cmake_GIT_REPO}")";
+        rapids_cmake_upstream="$(jq -r '.upstream' <<< "${RAPIDS_cmake_GIT_REPO}")";
+    else
+        rapids_cmake_tag="$(yq '.x-git-defaults.tag' /opt/rapids-build-utils/manifest.yaml)";
+        rapids_cmake_upstream="$(yq '.x-git-defaults.upstream' /opt/rapids-build-utils/manifest.yaml)";
+    fi
+
+    # Define CCCL_TAG to override the default CCCL SHA. Otherwise the current HEAD of the local checkout is used.
+    if test -n "${CCCL_TAG-}"; then
+        # If CCCL_TAG is defined, fetch it to the local checkout
+        git -C "${HOME}/cccl" fetch origin "${CCCL_TAG}";
+        cccl_sha="$(git -C "${HOME}/cccl" rev-parse FETCH_HEAD)";
+    else
+        cccl_sha="$(git -C "${HOME}/cccl" rev-parse HEAD)";
+    fi
+    echo "CCCL_VERSION: ${CCCL_VERSION-}";
+    echo "CCCL_TAG: ${CCCL_TAG-}";
+    echo "cccl_sha: ${cccl_sha}";
+
+    echo
+    echo "Replacing CCCL repo information in rapids-cmake versions.json:";
+    curl -fsSL -o- "https://raw.githubusercontent.com/${rapids_cmake_upstream}/rapids-cmake/${rapids_cmake_tag}/rapids-cmake/cpm/versions.json" \
+  | jq -r ".packages.CCCL *= {\"git_url\": \"${HOME}/cccl\", \"git_tag\": \"${cccl_sha}\", \"always_download\": true}" \
+  | jq -r "del(.packages.CCCL.url) | del(.packages.CCCL.url_hash)" \
+  > ~/rapids-cmake-override-versions-cccl-repo.json;
+
+    if test -n "${CCCL_VERSION-}"; then
+        echo "Patching CCCL_VERSION in rapids-cmake versions.json:";
+        jq -r ".packages.CCCL.version = \"${CCCL_VERSION}\"" ~/rapids-cmake-override-versions-cccl-repo.json \
+      > ~/rapids-cmake-override-versions.json;
+    else
+        echo "Using the default CCCL version in rapids-cmake versions.json:";
+        mv ~/rapids-cmake-override-versions-cccl-repo.json ~/rapids-cmake-override-versions.json;
+    fi
+
+    echo
+    echo "Final rapids-cmake-override-versions.json:";
+    cat ~/rapids-cmake-override-versions.json;
+
+    # Define default CMake args for each repo
+    local -a cmake_args=(BUILD_TESTS BUILD_BENCHMARKS BUILD_PRIMS_BENCH BUILD_CUGRAPH_MG_TESTS);
+    # Enable tests
+    cmake_args=("${cmake_args[@]/#/"-D"}");
+    cmake_args=("${cmake_args[@]/%/"=${RAPIDS_ENABLE_TESTS:-ON}"}");
+
+    # Always build RAFT shared lib
+    cmake_args+=("-DBUILD_SHARED_LIBS=ON");
+    cmake_args+=("-DRAFT_COMPILE_LIBRARY=ON");
+
+    # Tell rapids-cmake to use custom CCCL and cuCollections forks
+    cmake_args+=("-Drapids-cmake-branch=${rapids_cmake_tag}");
+    cmake_args+=("-Drapids-cmake-repo=${rapids_cmake_upstream}/rapids-cmake");
+    cmake_args+=("-DRAPIDS_CMAKE_CPM_DEFAULT_VERSION_FILE=${HOME}/rapids-cmake-override-versions.json");
+
+    # Inject the CMake args into manifest.yaml
+    sudo yq -i "(.repos[] | .cpp[] | .args.cmake) += \" ${cmake_args[*]}\"" "${PROJECT_MANIFEST_YML}";
+    sudo yq -i "(.repos[] | .python[] | .args.cmake) += \" ${cmake_args[*]}\"" "${PROJECT_MANIFEST_YML}";
+}
+
+_run_post_create_command() {
+    local -;
+    set -e;
+
+    # Install `rapids-build-utils` feature if it's not installed (i.e. if running locally via `.devcontainer/launch.sh -d`)
+    if ! test -f "${PROJECT_MANIFEST_YML}"; then
+        git clone --depth 1 --filter=blob:none --sparse https://github.com/rapidsai/devcontainers.git /tmp/rapidsai-devcontainers;
+        git -C /tmp/rapidsai-devcontainers sparse-checkout set features/src/rapids-build-utils;
+        (
+            cd /tmp/rapidsai-devcontainers/features/src/rapids-build-utils;
+            sudo bash ./install.sh;
+        )
+        rm -rf /tmp/rapidsai-devcontainers;
+    fi
+
+    # Modify manifest.yaml based on envvars
+    _apply_manifest_modifications;
+
+    # Clone all the repos
+    gh config set git_protocol https;
+    gh config set git_protocol https --host github.com;
+
+    clone-all -j "$(nproc --all)" -v -q --clone-upstream --single-branch --shallow-submodules --no-update-env;
+}
+
+if [[ "$(basename "${BASH_SOURCE[${#BASH_SOURCE[@]}-1]}")" = post-create-command.sh ]]; then
+    _run_post_create_command;
+fi
diff --git a/cccl_upstream/ci/rapids/rapids-entrypoint.sh b/cccl_upstream/ci/rapids/rapids-entrypoint.sh
new file mode 100755
index 00000000..b8b43112
--- /dev/null
+++ b/cccl_upstream/ci/rapids/rapids-entrypoint.sh
@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+
+# shellcheck disable=SC1091
+
+set -e;
+
+if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+    echo "::group::Cloning RAPIDS..."
+fi
+
+ci/rapids/post-create-command.sh;
+rapids-post-start-command -f;
+
+if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+    echo "::endgroup::"
+fi
+
+if test $# -gt 0; then
+    exec "$@";
+else
+    exec /bin/bash -li;
+fi
diff --git a/cccl_upstream/ci/run_cpu_bisect.sh b/cccl_upstream/ci/run_cpu_bisect.sh
new file mode 100755
index 00000000..3bc7dd3c
--- /dev/null
+++ b/cccl_upstream/ci/run_cpu_bisect.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+# CI wrapper for the `bisect` project build job.
+# Forwards all arguments to ci/util/git_bisect.sh to run a bisection
+# using the provided configuration/build/test options.
+
+set -euo pipefail
+
+ci_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+repo_dir=$(cd "${ci_dir}/.." && pwd)
+
+user_args=("$@")
+set --
+source "${ci_dir}/build_common.sh"
+set -- "${user_args[@]}"
+
+cd "${repo_dir}"
+cmd=("${ci_dir}/util/git_bisect.sh" "$@")
+printf '\033[34m%s\033[0m\n' "${cmd[*]}"
+"${cmd[@]}"
diff --git a/cccl_upstream/ci/run_cpu_target.sh b/cccl_upstream/ci/run_cpu_target.sh
new file mode 100755
index 00000000..81951887
--- /dev/null
+++ b/cccl_upstream/ci/run_cpu_target.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+# CI wrapper for the `target` project build job.
+# Forwards all arguments to ci/util/build_and_test_targets.sh to configure
+# and build selected targets on a CPU runner.
+
+set -euo pipefail
+
+ci_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+repo_dir=$(cd "${ci_dir}/.." && pwd)
+
+user_args=("$@")
+set --
+source "${ci_dir}/build_common.sh"
+set -- "${user_args[@]}"
+
+cd "${repo_dir}"
+cmd=("${ci_dir}/util/build_and_test_targets.sh" "$@")
+printf '\033[34m%s\033[0m\n' "${cmd[*]}"
+"${cmd[@]}"
diff --git a/cccl_upstream/ci/run_gpu_bisect.sh b/cccl_upstream/ci/run_gpu_bisect.sh
new file mode 100755
index 00000000..41a7ddc2
--- /dev/null
+++ b/cccl_upstream/ci/run_gpu_bisect.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+# CI wrapper for the `bisect` project test job.
+# Invokes ci/util/git_bisect.sh with the provided arguments to
+# run a bisection on a GPU runner.
+
+set -euo pipefail
+
+ci_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+repo_dir=$(cd "${ci_dir}/.." && pwd)
+
+user_args=("$@")
+set --
+source "${ci_dir}/build_common.sh"
+set -- "${user_args[@]}"
+
+cd "${repo_dir}"
+cmd=("${ci_dir}/util/git_bisect.sh" "$@")
+printf '\033[34m%s\033[0m\n' "${cmd[*]}"
+"${cmd[@]}"
diff --git a/cccl_upstream/ci/run_gpu_target.sh b/cccl_upstream/ci/run_gpu_target.sh
new file mode 100755
index 00000000..4bc25bfa
--- /dev/null
+++ b/cccl_upstream/ci/run_gpu_target.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+# CI wrapper for the `target` project test job.
+# Invokes ci/util/build_and_test_targets.sh with the provided arguments to
+# build and test selected targets on a GPU runner.
+
+set -euo pipefail
+
+ci_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+repo_dir=$(cd "${ci_dir}/.." && pwd)
+
+user_args=("$@")
+set --
+source "${ci_dir}/build_common.sh"
+set -- "${user_args[@]}"
+
+cd "${repo_dir}"
+cmd=("${ci_dir}/util/build_and_test_targets.sh" "$@")
+printf '\033[34m%s\033[0m\n' "${cmd[*]}"
+"${cmd[@]}"
diff --git a/cccl_upstream/ci/test/CMakeLists.txt b/cccl_upstream/ci/test/CMakeLists.txt
new file mode 100644
index 00000000..e801d6e1
--- /dev/null
+++ b/cccl_upstream/ci/test/CMakeLists.txt
@@ -0,0 +1 @@
+add_subdirectory("inspect_changes")
diff --git a/cccl_upstream/ci/test/inspect_changes/CMakeLists.txt b/cccl_upstream/ci/test/inspect_changes/CMakeLists.txt
new file mode 100644
index 00000000..fda2c0f4
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/CMakeLists.txt
@@ -0,0 +1,22 @@
+find_package(Python3 REQUIRED COMPONENTS Interpreter)
+
+file(GLOB output_files "${CMAKE_CURRENT_SOURCE_DIR}/*.output")
+foreach (output_file IN LISTS output_files)
+  get_filename_component(test_name "${output_file}" NAME_WE)
+  set(dirty_file "${CMAKE_CURRENT_SOURCE_DIR}/${test_name}.dirty_files")
+  if (NOT EXISTS "${dirty_file}")
+    message(FATAL_ERROR "Missing dirty file for ${test_name}")
+  endif()
+
+  add_test(
+    NAME cccl.ci.test.inspect_changes.${test_name}
+    # gersemi: off
+    COMMAND
+      "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/run_inspect_changes_test.py"
+        "--python" "${Python3_EXECUTABLE}"
+        "--script" "${PROJECT_SOURCE_DIR}/ci/inspect_changes.py"
+        "--dirty" "${dirty_file}"
+        "--expected" "${output_file}"
+    # gersemi: on
+  )
+endforeach()
diff --git a/cccl_upstream/ci/test/inspect_changes/c2h_dependency.dirty_files b/cccl_upstream/ci/test/inspect_changes/c2h_dependency.dirty_files
new file mode 100644
index 00000000..8938dfc5
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/c2h_dependency.dirty_files
@@ -0,0 +1 @@
+c2h/catch2_runner.cu
diff --git a/cccl_upstream/ci/test/inspect_changes/c2h_dependency.output b/cccl_upstream/ci/test/inspect_changes/c2h_dependency.output
new file mode 100644
index 00000000..92df1f7a
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/c2h_dependency.output
@@ -0,0 +1,2 @@
+FULL_BUILD=tidy
+LITE_BUILD=libcudacxx cub cudax cccl_c_parallel cccl_c_parallel_v2 python_v2 cccl_c_stf packaging
diff --git a/cccl_upstream/ci/test/inspect_changes/core_dirty.dirty_files b/cccl_upstream/ci/test/inspect_changes/core_dirty.dirty_files
new file mode 100644
index 00000000..5e71f5bf
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/core_dirty.dirty_files
@@ -0,0 +1 @@
+CMakePresets.json
diff --git a/cccl_upstream/ci/test/inspect_changes/core_dirty.output b/cccl_upstream/ci/test/inspect_changes/core_dirty.output
new file mode 100644
index 00000000..456e3158
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/core_dirty.output
@@ -0,0 +1,2 @@
+FULL_BUILD=libcudacxx cub thrust cudax cccl_c_parallel cccl_c_parallel_v2 python_v2 python_tsan cccl_c_stf python packaging stdpar nvbench_helper nvrtcc tidy
+LITE_BUILD=
diff --git a/cccl_upstream/ci/test/inspect_changes/ignored_only.dirty_files b/cccl_upstream/ci/test/inspect_changes/ignored_only.dirty_files
new file mode 100644
index 00000000..7d2fa473
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/ignored_only.dirty_files
@@ -0,0 +1 @@
+docs/index.rst
diff --git a/cccl_upstream/ci/test/inspect_changes/ignored_only.output b/cccl_upstream/ci/test/inspect_changes/ignored_only.output
new file mode 100644
index 00000000..4e9e2ba7
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/ignored_only.output
@@ -0,0 +1,2 @@
+FULL_BUILD=
+LITE_BUILD=
diff --git a/cccl_upstream/ci/test/inspect_changes/libcudacxx_both.dirty_files b/cccl_upstream/ci/test/inspect_changes/libcudacxx_both.dirty_files
new file mode 100644
index 00000000..a96b1b4e
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/libcudacxx_both.dirty_files
@@ -0,0 +1,2 @@
+libcudacxx/CMakeLists.txt
+libcudacxx/include/cuda/__device/device_ref.h
diff --git a/cccl_upstream/ci/test/inspect_changes/libcudacxx_both.output b/cccl_upstream/ci/test/inspect_changes/libcudacxx_both.output
new file mode 100644
index 00000000..eddc7707
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/libcudacxx_both.output
@@ -0,0 +1,2 @@
+FULL_BUILD=libcudacxx tidy
+LITE_BUILD=cub thrust cudax cccl_c_parallel cccl_c_parallel_v2 python_v2 python_tsan cccl_c_stf python packaging stdpar nvbench_helper
diff --git a/cccl_upstream/ci/test/inspect_changes/libcudacxx_internal_only.dirty_files b/cccl_upstream/ci/test/inspect_changes/libcudacxx_internal_only.dirty_files
new file mode 100644
index 00000000..879d2d5f
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/libcudacxx_internal_only.dirty_files
@@ -0,0 +1 @@
+libcudacxx/CMakeLists.txt
diff --git a/cccl_upstream/ci/test/inspect_changes/libcudacxx_internal_only.output b/cccl_upstream/ci/test/inspect_changes/libcudacxx_internal_only.output
new file mode 100644
index 00000000..8e8a2194
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/libcudacxx_internal_only.output
@@ -0,0 +1,2 @@
+FULL_BUILD=libcudacxx tidy
+LITE_BUILD=packaging
diff --git a/cccl_upstream/ci/test/inspect_changes/libcudacxx_public_only.dirty_files b/cccl_upstream/ci/test/inspect_changes/libcudacxx_public_only.dirty_files
new file mode 100644
index 00000000..421343ca
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/libcudacxx_public_only.dirty_files
@@ -0,0 +1 @@
+libcudacxx/include/cuda/__device/device_ref.h
diff --git a/cccl_upstream/ci/test/inspect_changes/libcudacxx_public_only.output b/cccl_upstream/ci/test/inspect_changes/libcudacxx_public_only.output
new file mode 100644
index 00000000..eddc7707
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/libcudacxx_public_only.output
@@ -0,0 +1,2 @@
+FULL_BUILD=libcudacxx tidy
+LITE_BUILD=cub thrust cudax cccl_c_parallel cccl_c_parallel_v2 python_v2 python_tsan cccl_c_stf python packaging stdpar nvbench_helper
diff --git a/cccl_upstream/ci/test/inspect_changes/libcudacxx_thrust.dirty_files b/cccl_upstream/ci/test/inspect_changes/libcudacxx_thrust.dirty_files
new file mode 100644
index 00000000..0e8b7d4e
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/libcudacxx_thrust.dirty_files
@@ -0,0 +1,3 @@
+libcudacxx/include/cuda/__device/device_ref.h
+thrust/thrust/version.h
+README.md
diff --git a/cccl_upstream/ci/test/inspect_changes/libcudacxx_thrust.output b/cccl_upstream/ci/test/inspect_changes/libcudacxx_thrust.output
new file mode 100644
index 00000000..00f10a62
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/libcudacxx_thrust.output
@@ -0,0 +1,2 @@
+FULL_BUILD=libcudacxx thrust tidy
+LITE_BUILD=cub cudax cccl_c_parallel cccl_c_parallel_v2 python_v2 python_tsan cccl_c_stf python packaging stdpar nvbench_helper
diff --git a/cccl_upstream/ci/test/inspect_changes/multiple_projects.dirty_files b/cccl_upstream/ci/test/inspect_changes/multiple_projects.dirty_files
new file mode 100644
index 00000000..a91f6065
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/multiple_projects.dirty_files
@@ -0,0 +1,2 @@
+python/cuda_cccl/pyproject.toml
+examples/basic/CMakeLists.txt
diff --git a/cccl_upstream/ci/test/inspect_changes/multiple_projects.output b/cccl_upstream/ci/test/inspect_changes/multiple_projects.output
new file mode 100644
index 00000000..9395cf0a
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/multiple_projects.output
@@ -0,0 +1,2 @@
+FULL_BUILD=python_v2 python_tsan python packaging
+LITE_BUILD=
diff --git a/cccl_upstream/ci/test/inspect_changes/no_changes.dirty_files b/cccl_upstream/ci/test/inspect_changes/no_changes.dirty_files
new file mode 100644
index 00000000..e69de29b
diff --git a/cccl_upstream/ci/test/inspect_changes/no_changes.output b/cccl_upstream/ci/test/inspect_changes/no_changes.output
new file mode 100644
index 00000000..4e9e2ba7
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/no_changes.output
@@ -0,0 +1,2 @@
+FULL_BUILD=
+LITE_BUILD=
diff --git a/cccl_upstream/ci/test/inspect_changes/packaging_only.dirty_files b/cccl_upstream/ci/test/inspect_changes/packaging_only.dirty_files
new file mode 100644
index 00000000..f1ec2e00
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/packaging_only.dirty_files
@@ -0,0 +1 @@
+examples/CMakeLists.txt
diff --git a/cccl_upstream/ci/test/inspect_changes/packaging_only.output b/cccl_upstream/ci/test/inspect_changes/packaging_only.output
new file mode 100644
index 00000000..a23e2374
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/packaging_only.output
@@ -0,0 +1,2 @@
+FULL_BUILD=packaging
+LITE_BUILD=
diff --git a/cccl_upstream/ci/test/inspect_changes/regenerate_outputs.sh b/cccl_upstream/ci/test/inspect_changes/regenerate_outputs.sh
new file mode 100755
index 00000000..464d09ba
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/regenerate_outputs.sh
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+project_root=$(cd "${script_dir}/../../.." && pwd)
+inspect_changes="${project_root}/ci/inspect_changes.py"
+
+if [[ ! -x "${inspect_changes}" ]]; then
+  echo "Error: ${inspect_changes} not found or not executable." >&2
+  exit 1
+fi
+
+cd "${script_dir}"
+
+for dirty_file in *.dirty_files; do
+  test_name=${dirty_file%.dirty_files}
+  output_file="${test_name}.output"
+  echo "Regenerating ${output_file}"
+  python "${inspect_changes}" --file "${dirty_file}" \
+    | awk '/^FULL_BUILD=/{print}/^LITE_BUILD=/{print}' > "${output_file}"
+done
diff --git a/cccl_upstream/ci/test/inspect_changes/run_inspect_changes_test.py b/cccl_upstream/ci/test/inspect_changes/run_inspect_changes_test.py
new file mode 100755
index 00000000..94dccd57
--- /dev/null
+++ b/cccl_upstream/ci/test/inspect_changes/run_inspect_changes_test.py
@@ -0,0 +1,81 @@
+#!/usr/bin/env python3
+"""Test harness for ci/inspect_changes.py."""
+
+from __future__ import annotations
+
+import argparse
+import difflib
+import subprocess
+import sys
+from pathlib import Path
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description="Run inspect_changes.py and compare output"
+    )
+    parser.add_argument(
+        "--script", required=True, type=Path, help="Path to inspect_changes.py"
+    )
+    parser.add_argument(
+        "--dirty", required=True, type=Path, help="File listing dirty files"
+    )
+    parser.add_argument(
+        "--expected", required=True, type=Path, help="Expected stdout contents"
+    )
+    parser.add_argument(
+        "--python", default=sys.executable, help="Python interpreter to use"
+    )
+    return parser.parse_args()
+
+
+def build_command(python: str, script: Path, dirty_file: Path) -> list[str]:
+    cmd = [python, str(script), "--file", str(dirty_file)]
+    return cmd
+
+
+def main() -> int:
+    args = parse_args()
+    cmd = build_command(args.python, args.script, args.dirty)
+
+    sys.stdout.write(f"COMMAND: {' '.join(cmd)}\n")
+    result = subprocess.run(cmd, capture_output=True, text=True)
+    sys.stdout.write("OUTPUT:\n")
+    sys.stdout.write(result.stdout)
+    if result.stdout and not result.stdout.endswith("\n"):
+        sys.stdout.write("\n")
+    if result.returncode != 0:
+        sys.stderr.write(result.stderr)
+        sys.stderr.write("\nCommand failed: {}\n".format(" ".join(cmd)))
+        return result.returncode
+
+    actual_lines = result.stdout.splitlines()
+    expected_lines = [
+        line.strip()
+        for line in args.expected.read_text(encoding="utf-8").splitlines()
+        if line.strip()
+    ]
+
+    missing = [line for line in expected_lines if line not in actual_lines]
+    if missing:
+        diff = "\n".join(
+            difflib.unified_diff(
+                expected_lines,
+                actual_lines,
+                fromfile="expected(subset)",
+                tofile="actual",
+                lineterm="",
+            )
+        )
+        if diff:
+            sys.stderr.write(diff + "\n\n")
+        sys.stderr.write("\nExpected lines missing from output:\n")
+        for line in missing:
+            sys.stderr.write(f"  {line}\n")
+        return 1
+
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
diff --git a/cccl_upstream/ci/test_cccl_c_parallel.sh b/cccl_upstream/ci/test_cccl_c_parallel.sh
new file mode 100755
index 00000000..01fe14af
--- /dev/null
+++ b/cccl_upstream/ci/test_cccl_c_parallel.sh
@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+
+source "$(dirname "${BASH_SOURCE[0]}")/build_common.sh"
+
+print_environment_details
+
+./build_cccl_c_parallel.sh "$@"
+
+PRESET="cccl-c-parallel"
+
+test_preset "CCCL C Parallel Library" "${PRESET}"
+
+print_time_summary
diff --git a/cccl_upstream/ci/test_cccl_c_parallel_v2.sh b/cccl_upstream/ci/test_cccl_c_parallel_v2.sh
new file mode 100755
index 00000000..02c893f4
--- /dev/null
+++ b/cccl_upstream/ci/test_cccl_c_parallel_v2.sh
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+source "$(dirname "${BASH_SOURCE[0]}")/build_common.sh"
+
+print_environment_details
+
+# libnvfatbin is required by hostjit but is not included in the base rapidsai devcontainer
+# image. Detect the installed CTK version and install the matching package if missing.
+if [[ "$(uname -s)" == "Linux" ]] && ! ldconfig -p 2>/dev/null | grep -q libnvfatbin; then
+  CTK_DEB_VER=$(nvcc --version 2>/dev/null \
+    | grep -oP 'release \K[0-9]+\.[0-9]+' | tr '.' '-')
+  if [[ -n "$CTK_DEB_VER" ]]; then
+    echo "Installing libnvfatbin-dev-${CTK_DEB_VER}..."
+    sudo apt-get update -y
+    sudo apt-get install -y --no-install-recommends "libnvfatbin-dev-${CTK_DEB_VER}"
+  else
+    echo "WARNING: could not determine CTK version; skipping libnvfatbin install"
+  fi
+fi
+
+PRESET="cccl-c-parallel-v2"
+
+CMAKE_OPTIONS=()
+if test -n "${CXX_STANDARD:+x}"; then
+    CMAKE_OPTIONS+=("-DCMAKE_CXX_STANDARD=${CXX_STANDARD}" "-DCMAKE_CUDA_STANDARD=${CXX_STANDARD}")
+fi
+
+configure_and_build_preset "CCCL C Parallel Library v2 (HostJIT)" "$PRESET" "${CMAKE_OPTIONS[@]}"
+
+test_preset "CCCL C Parallel Library v2 (HostJIT)" "$PRESET"
+
+print_time_summary
diff --git a/cccl_upstream/ci/test_cccl_c_stf.sh b/cccl_upstream/ci/test_cccl_c_stf.sh
new file mode 100755
index 00000000..410e7ae4
--- /dev/null
+++ b/cccl_upstream/ci/test_cccl_c_stf.sh
@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+
+source "$(dirname "${BASH_SOURCE[0]}")/build_common.sh"
+
+print_environment_details
+
+./build_cccl_c_stf.sh "$@"
+
+PRESET="cccl-c-stf"
+
+test_preset "CCCL C Parallel Library" "${PRESET}"
+
+print_time_summary
diff --git a/cccl_upstream/ci/test_cub.sh b/cccl_upstream/ci/test_cub.sh
new file mode 100755
index 00000000..d111be86
--- /dev/null
+++ b/cccl_upstream/ci/test_cub.sh
@@ -0,0 +1,142 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+NO_LID=false
+LID0=false
+LID1=false
+LID2=false
+LIMITED=false
+COMPUTE_SANITIZER=false
+ARTIFACT_TAGS=()
+
+ci_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+
+new_args="$("${ci_dir}/util/extract_switches.sh" \
+  -no-lid \
+  -lid0 \
+  -lid1 \
+  -lid2 \
+  -limited \
+  -compute-sanitizer-memcheck \
+  -compute-sanitizer-racecheck \
+  -compute-sanitizer-initcheck \
+  -compute-sanitizer-synccheck \
+  -- "$@")"
+
+declare -a new_args="(${new_args})"
+set -- "${new_args[@]}"
+while true; do
+  case "$1" in
+  -no-lid)
+    ARTIFACT_TAGS+=("no_lid")
+    NO_LID=true
+    shift
+    ;;
+  -lid0)
+    ARTIFACT_TAGS+=("lid_0")
+    LID0=true
+    shift
+    ;;
+  -lid1)
+    ARTIFACT_TAGS+=("lid_1")
+    LID1=true
+    shift
+    ;;
+  -lid2)
+    ARTIFACT_TAGS+=("lid_2")
+    LID2=true
+    shift
+    ;;
+  -limited)
+    # Pull all artifacts:
+    ARTIFACT_TAGS+=("no_lid" "lid_0" "lid_1" "lid_2")
+    LIMITED=true
+    shift
+    ;;
+  -compute-sanitizer-memcheck)
+    COMPUTE_SANITIZER=true
+    TOOL=memcheck
+    shift
+    ;;
+  -compute-sanitizer-racecheck)
+    COMPUTE_SANITIZER=true
+    TOOL=racecheck
+    shift
+    ;;
+  -compute-sanitizer-initcheck)
+    COMPUTE_SANITIZER=true
+    TOOL=initcheck
+    shift
+    ;;
+  -compute-sanitizer-synccheck)
+    COMPUTE_SANITIZER=true
+    TOOL=synccheck
+    shift
+    ;;
+  --)
+    shift
+    break
+    ;;
+  *)
+    echo "Unknown argument: $1"
+    exit 1
+    ;;
+  esac
+done
+
+if $LIMITED; then
+
+  export C2H_SEED_COUNT_OVERRIDE=1
+  readonly device_mem_GiB=8
+  export C2H_DEVICE_MEMORY_LIMIT=$((device_mem_GiB * 1024 * 1024 * 1024))
+  export C2H_DEBUG_CHECKED_ALLOC_FAILURES=1
+
+  echo "Configuring limited environment:"
+  echo "  C2H_SEED_COUNT_OVERRIDE=${C2H_SEED_COUNT_OVERRIDE}"
+  echo "  C2H_DEVICE_MEMORY_LIMIT=${C2H_DEVICE_MEMORY_LIMIT} (${device_mem_GiB} GiB)"
+  echo "  C2H_DEBUG_CHECKED_ALLOC_FAILURES=${C2H_DEBUG_CHECKED_ALLOC_FAILURES}"
+  echo
+fi
+
+# shellcheck source=ci/build_common.sh
+source "${ci_dir}/build_common.sh"
+
+print_environment_details
+
+
+if [[ -z "${GITHUB_ACTIONS:-}" ]]; then
+  ./build_cub.sh "$@"
+else
+  producer_id=$(util/workflow/get_producer_id.sh)
+  for tag in "${ARTIFACT_TAGS[@]}"; do
+    artifact="z_cub-test-artifacts-${DEVCONTAINER_NAME:?}-$producer_id-$tag"
+    run_command "📦  Unpacking artifact '$artifact'" \
+      "${ci_dir}/util/artifacts/download_packed.sh" "$artifact" /home/coder/cccl
+  done
+fi
+
+if $NO_LID; then
+  PRESETS=("cub-nolid")
+elif $LID0; then
+  PRESETS=("cub-lid0")
+elif $LID1; then
+  PRESETS=("cub-lid1")
+elif $LID2; then
+  PRESETS=("cub-lid2")
+else
+  PRESETS=("cub")
+fi
+
+if $COMPUTE_SANITIZER; then
+  echo "Setting CCCL_TEST_MODE=compute-sanitizer-${TOOL}"
+  export CCCL_TEST_MODE=compute-sanitizer-${TOOL}
+  echo "Setting C2H_SEED_COUNT_OVERRIDE=1"
+  export C2H_SEED_COUNT_OVERRIDE=1
+fi
+
+for PRESET in "${PRESETS[@]}"; do
+  test_preset "CUB (${PRESET})" "${PRESET}"
+done
+
+print_time_summary
diff --git a/cccl_upstream/ci/test_cuda_cccl_examples_python.sh b/cccl_upstream/ci/test_cuda_cccl_examples_python.sh
new file mode 100755
index 00000000..5a3e17e5
--- /dev/null
+++ b/cccl_upstream/ci/test_cuda_cccl_examples_python.sh
@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "$ci_dir/pyenv_helper.sh"
+
+# Parse common arguments
+source "$ci_dir/util/python/common_arg_parser.sh"
+parse_python_args "$@"
+# Pin cuda-toolkit to the container's CTK minor and set cuda_version /
+# cuda_major_version (-ctk-mode latest opts out). See pyenv_helper.sh.
+pin_cuda_toolkit "${ctk_mode}"
+
+# Setup Python environment
+setup_python_env "${py_version}"
+
+# Fetch or build the cuda_cccl wheel:
+if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+  wheel_artifact_name=$("$ci_dir/util/workflow/get_wheel_artifact_name.sh")
+  "$ci_dir/util/artifacts/download.sh" "${wheel_artifact_name}" /home/coder/cccl/
+else
+  "$ci_dir/build_cuda_cccl_python.sh" -py-version "${py_version}"
+fi
+
+# Install cuda_cccl, plus CuPy which the cuda.compute examples require, plus
+# pytest-benchmark for the host-overhead benchmark smoke test below. (cuda-bench,
+# for the throughput smoke, is installed best-effort further down since it does
+# not always ship a wheel for the newest Python.)
+CUDA_CCCL_WHEEL_PATH="$(ls /home/coder/cccl/wheelhouse/cuda_cccl-*.whl)"
+ctk_flavor="$(ctk_extra_flavor "${ctk_mode}")"
+python -m pip install "${CUDA_CCCL_WHEEL_PATH}[test-${ctk_flavor}${cuda_major_version}]" "cupy-cuda${cuda_major_version}x" pytest-benchmark
+
+# Run tests for parallel module
+cd "/home/coder/cccl/python/cuda_cccl/tests/"
+python -m pytest -n 6 test_examples.py
+
+# Smoke-test the host-overhead benchmark harness: run every benchmark case
+# exactly once (pass/fail only, no timing) so harness rot fails CI here instead
+# of silently surviving until someone runs the perf suite.
+cd "/home/coder/cccl/python/cuda_cccl/benchmarks/compute/host/"
+python -m pytest -v --benchmark-disable .
+
+# Smoke-test the throughput (nvbench) benchmarks the same way. --profile runs
+# each configuration once (no sampling); --quick uses the reduced quick_configs
+# axes (one dtype, smallest size) so every benchmark harness still imports,
+# registers, launches, and completes. cuda-bench does not always ship a wheel for
+# the newest Python, so skip the throughput smoke ONLY for that known no-wheel
+# case. Note pip prints "No matching distribution"/"Could not find a version"
+# even when the index is unreachable, so check for fetch/network failures first
+# and fail on those; any other install error fails the lane too rather than
+# silently passing. tee streams pip's output live while capturing it for the
+# grep checks below (pipefail keeps pip's exit status, not tee's).
+install_log="$(mktemp)"
+if python -m pip install "cuda-bench[cu${cuda_major_version}]" pyyaml 2>&1 | tee "${install_log}"; then
+  cd "/home/coder/cccl/python/cuda_cccl/benchmarks/compute/"
+  python run_benchmarks.py --py --profile --quick
+elif grep -qiE "Could not fetch URL|Retrying \(Retry|connection broken|Failed to establish a new connection|Name or service not known|timed out|SSLError|certificate verify failed|ProxyError" "${install_log}"; then
+  echo "::error::cuda-bench install failed because pip could not reach the package index (network/DNS/TLS/auth); not skipping." >&2
+  exit 1
+elif grep -qiE "No matching distribution found for cuda-bench|Could not find a version that satisfies the requirement cuda-bench" "${install_log}"; then
+  echo "::warning::cuda-bench has no wheel for Python ${py_version}; skipping the throughput benchmark smoke test."
+else
+  echo "::error::cuda-bench install failed for an unrecognized reason." >&2
+  exit 1
+fi
+rm -f "${install_log}"
diff --git a/cccl_upstream/ci/test_cuda_cccl_examples_python_v2.sh b/cccl_upstream/ci/test_cuda_cccl_examples_python_v2.sh
new file mode 100755
index 00000000..ae852085
--- /dev/null
+++ b/cccl_upstream/ci/test_cuda_cccl_examples_python_v2.sh
@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+# Run the cuda.compute examples against a wheel built with the v2 (HostJIT)
+# backend. Mirrors test_cuda_cccl_examples_python.sh; the only difference is
+# exporting CCCL_PYTHON_USE_V2 so the wheel build uses cccl.c.parallel.v2.
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+export CCCL_PYTHON_USE_V2=1
+exec "$ci_dir/test_cuda_cccl_examples_python.sh" "$@"
diff --git a/cccl_upstream/ci/test_cuda_cccl_headers_python.sh b/cccl_upstream/ci/test_cuda_cccl_headers_python.sh
new file mode 100755
index 00000000..ee39630d
--- /dev/null
+++ b/cccl_upstream/ci/test_cuda_cccl_headers_python.sh
@@ -0,0 +1,33 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+source "$ci_dir/pyenv_helper.sh"
+
+# Parse common arguments
+source "$ci_dir/util/python/common_arg_parser.sh"
+parse_python_args "$@"
+# Pin cuda-toolkit to the container's CTK minor and set cuda_version /
+# cuda_major_version (-ctk-mode latest opts out). See pyenv_helper.sh.
+pin_cuda_toolkit "${ctk_mode}"
+
+# Setup Python environment
+setup_python_env "${py_version}"
+
+# Fetch or build the cuda_cccl wheel:
+if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+  wheel_artifact_name=$("$ci_dir/util/workflow/get_wheel_artifact_name.sh")
+  "$ci_dir/util/artifacts/download.sh" "${wheel_artifact_name}" /home/coder/cccl/
+else
+  "$ci_dir/build_cuda_cccl_python.sh" -py-version "${py_version}"
+fi
+
+# Install cuda_cccl
+CUDA_CCCL_WHEEL_PATH="$(ls /home/coder/cccl/wheelhouse/cuda_cccl-*.whl)"
+ctk_flavor="$(ctk_extra_flavor "${ctk_mode}")"
+python -m pip install "${CUDA_CCCL_WHEEL_PATH}[test-${ctk_flavor}${cuda_major_version}]"
+
+# Run tests for core package
+cd "/home/coder/cccl/python/cuda_cccl/tests/"
+python -m pytest -n auto -v headers/
diff --git a/cccl_upstream/ci/test_cuda_compute_minimal_python.sh b/cccl_upstream/ci/test_cuda_compute_minimal_python.sh
new file mode 100755
index 00000000..fc2635d3
--- /dev/null
+++ b/cccl_upstream/ci/test_cuda_compute_minimal_python.sh
@@ -0,0 +1,75 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+repo_root="$(cd "$ci_dir/.." && pwd)"
+source "$ci_dir/pyenv_helper.sh"
+
+# Parse common arguments
+source "$ci_dir/util/python/common_arg_parser.sh"
+parse_python_args "$@"
+require_py_version "Usage: $0 -py-version <python_version>"
+
+# Pin cuda-toolkit to the container's CTK minor and set cuda_version /
+# cuda_major_version (-ctk-mode latest opts out). See pyenv_helper.sh.
+pin_cuda_toolkit "${ctk_mode}"
+
+# Setup Python environment
+setup_python_env "${py_version}"
+
+# Fetch or build the cuda_cccl wheel:
+if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+  wheel_artifact_name=$("$ci_dir/util/workflow/get_wheel_artifact_name.sh")
+  "$ci_dir/util/artifacts/download.sh" "${wheel_artifact_name}" "${repo_root}/"
+  wheelhouse_dir="${repo_root}/wheelhouse"
+else
+  "$ci_dir/build_cuda_cccl_python.sh" -py-version "${py_version}"
+  wheelhouse_dir="${repo_root}/wheelhouse"
+fi
+
+# Install cuda_cccl with the minimal CUDA extra. This intentionally avoids the
+# full cu*/sysctk* extras because those pull in numba/numba-cuda. The flavor is
+# "cu" (pip toolkit) or "sysctk" (system toolkit) per the -ctk-mode arg.
+CUDA_CCCL_WHEEL_PATH="$(ls "${wheelhouse_dir}"/cuda_cccl-*.whl)"
+ctk_flavor="$(ctk_extra_flavor "${ctk_mode}")"
+python -m pip install "${CUDA_CCCL_WHEEL_PATH}[minimal-${ctk_flavor}${cuda_major_version}]"
+python -m pip install pytest pytest-xdist
+
+cd "${repo_root}/python/cuda_cccl/tests/"
+python -m pytest -n 6 -v compute/test_no_numba.py
+if [[ "${py_version}" == "3.14t" ]]; then
+  # Select only tests that support the minimal extra so pytest does not collect
+  # tests that import numba-cuda and re-enable the GIL. These tests provide their
+  # own worker threads, so keep pytest itself in a single process.
+  # The serialization node-ids are module-skipped on the v2 backend today and
+  # will start running there automatically once v2 gains serialization support.
+  python -m pytest -n 0 -v \
+    compute/test_free_threading_stress.py \
+    compute/test_multi_cc_serialization.py::test_aot_build_result_load_failure_is_shared_and_retryable \
+    compute/test_multi_cc_serialization.py::test_aot_serialization_waits_for_canonical_first_load
+
+  # Broad thread-safety sweep (pytest-run-parallel): re-run the numba-free
+  # functional suite with each test executed concurrently across threads
+  # (barrier-synchronized start), stressing the process-wide build cache,
+  # single-flight coordination, and the Cython bindings from many threads at
+  # once. Complements test_free_threading_stress.py above, which targets specific
+  # shared-object scenarios by hand. -n 0 so the threads share one interpreter.
+  #
+  # --parallel-threads=2 matches CuPy's free-threading CI (the closest GPU
+  # precedent); a small fixed count bounds GPU-memory pressure from concurrent
+  # kernels and stays reproducible across runners, unlike =auto (the runner's
+  # logical-core count).
+  #
+  # pytest-run-parallel is only used by this sweep, so install it on the 3.14t
+  # path rather than for every minimal (e.g. non-free-threaded 3.14) run.
+  python -m pip install pytest-run-parallel
+
+  # Fail fast if the interpreter is not actually GIL-free (wrong build /
+  # PYTHON_GIL=1): pytest-run-parallel does NOT catch a GIL that is enabled from
+  # the start -- it would run threads GIL-serialized and pass vacuously. (A GIL
+  # *re-enabled mid-run* by a non-free-threaded import IS caught by the plugin,
+  # which is why we do not pass --ignore-gil-enabled.)
+  python -c "import sys; assert not sys._is_gil_enabled(), 'GIL is enabled; parallel sweep has no signal'"
+  python -m pytest -n 0 -v --parallel-threads=2 compute/test_no_numba.py
+fi
diff --git a/cccl_upstream/ci/test_cuda_compute_minimal_python_tsan.sh b/cccl_upstream/ci/test_cuda_compute_minimal_python_tsan.sh
new file mode 100755
index 00000000..26fc69ff
--- /dev/null
+++ b/cccl_upstream/ci/test_cuda_compute_minimal_python_tsan.sh
@@ -0,0 +1,107 @@
+#!/usr/bin/env bash
+
+# ThreadSanitizer variant of the minimal free-threaded cuda.compute lane.
+#
+# Installs the TSan-instrumented cuda_cccl wheel (produced by the `python_tsan`
+# build, which compiles c.parallel v1 host code with -fsanitize=thread) and runs
+# the free-threaded stress tests + the pytest-run-parallel sweep under the TSan
+# runtime. A real data race in c.parallel (e.g. build-owned state mutated at
+# launch and shared across threads) fails the nightly here instead of silently
+# corrupting results under some interleaving.
+#
+# Only meaningful on a free-threaded (3.14t) interpreter; the GIL would serialize
+# the threads and hide the very races we are looking for.
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+repo_root="$(cd "$ci_dir/.." && pwd)"
+source "$ci_dir/pyenv_helper.sh"
+
+# Parse common arguments
+source "$ci_dir/util/python/common_arg_parser.sh"
+parse_python_args "$@"
+require_py_version "Usage: $0 -py-version <python_version>"
+
+if [[ "${py_version}" != *t ]]; then
+  echo "ERROR: the TSan lane requires a free-threaded (…t) interpreter; got '${py_version}'." >&2
+  echo "On a GIL interpreter the sweep serializes and TSan has no signal." >&2
+  exit 1
+fi
+
+# Instrument c.parallel when this script builds the wheel itself (local runs). In
+# CI the wheel is the pre-built `python_tsan` artifact, already instrumented; the
+# export is harmless there.
+export CCCL_C_PARALLEL_SANITIZE_THREAD=1
+
+cuda_major_version=$(nvcc --version | grep release | awk '{print $6}' | tr -d ',' | cut -d '.' -f 1 | cut -d 'V' -f 2)
+
+# Setup Python environment
+setup_python_env "${py_version}"
+
+# Fetch or build the TSan-instrumented cuda_cccl wheel. Under project
+# `python_tsan`, get_wheel_artifact_name.sh resolves to the distinct `-tsan`
+# artifact, so this never grabs an uninstrumented wheel.
+if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+  wheel_artifact_name=$("$ci_dir/util/workflow/get_wheel_artifact_name.sh")
+  "$ci_dir/util/artifacts/download.sh" "${wheel_artifact_name}" "${repo_root}/"
+  wheelhouse_dir="${repo_root}/wheelhouse"
+else
+  "$ci_dir/build_cuda_cccl_python.sh" -py-version "${py_version}"
+  wheelhouse_dir="${repo_root}/wheelhouse"
+fi
+
+# minimal-cu* extra intentionally avoids numba/numba-cuda (which re-enable the
+# GIL). pytest-run-parallel drives the concurrent sweep.
+CUDA_CCCL_WHEEL_PATH="$(ls "${wheelhouse_dir}"/cuda_cccl-*.whl)"
+python -m pip install "${CUDA_CCCL_WHEEL_PATH}[minimal-cu${cuda_major_version}]"
+python -m pip install pytest pytest-xdist pytest-run-parallel
+
+# The instrumented .so links libtsan but keeps it external (auditwheel --exclude),
+# so the TSan runtime must be present from process start -- LD_PRELOAD the
+# runner's libtsan (same soname/major as the gcc-13 build). Without preload the
+# .so fails to load ("cannot allocate memory in static TLS block").
+tsan_runtime="$(gcc -print-file-name=libtsan.so.2)"
+if [[ ! -e "${tsan_runtime}" ]]; then
+  echo "ERROR: libtsan.so.2 not found (gcc -print-file-name returned '${tsan_runtime}')." >&2
+  exit 1
+fi
+
+# setarch -R disables ASLR for the process tree. Required: TSan reserves fixed
+# shadow-memory regions and aborts ("unexpected memory mapping") when ASLR drops
+# something into them (google/sanitizers#1686). Uses personality(ADDR_NO_RANDOMIZE)
+# -- if a runner's seccomp profile blocks it, this call fails and must be allowed
+# (e.g. --security-opt seccomp=unconfined on the job's container).
+#
+# ignore_noninstrumented_modules=1: only c.parallel is instrumented, so ignore
+# races inside uninstrumented CPython / CUDA libs (avoids boundary false
+# positives). halt_on_error=1: stop at the first race -- it is usually the root
+# cause, and later reports are typically downstream noise.
+# exitcode=66: exit non-zero when any (unsuppressed) race is found, failing the
+# job even though pytest itself passes.
+run_under_tsan() {
+  setarch -R env \
+    LD_PRELOAD="${tsan_runtime}" \
+    TSAN_OPTIONS="ignore_noninstrumented_modules=1 halt_on_error=1 history_size=7 exitcode=66" \
+    "$@"
+}
+
+# Fail fast if the interpreter is not actually GIL-free (wrong build /
+# PYTHON_GIL=1): the sweep would run GIL-serialized and pass vacuously.
+run_under_tsan python -c "import sys; assert not sys._is_gil_enabled(), 'GIL is enabled; TSan sweep has no signal'"
+
+cd "${repo_root}/python/cuda_cccl/tests/"
+
+# Hand-written free-threaded stress scenarios (they spawn their own worker
+# threads) + the serialization node-ids, all sharing one interpreter (-n 0).
+run_under_tsan python -m pytest -n 0 -v \
+  compute/test_free_threading_stress.py \
+  compute/test_multi_cc_serialization.py::test_aot_build_result_load_failure_is_shared_and_retryable \
+  compute/test_multi_cc_serialization.py::test_aot_serialization_waits_for_canonical_first_load
+
+# Broad sweep: run the numba-free functional suite with each test executed
+# concurrently across threads (barrier-synchronized), exercising many more
+# c.parallel algorithms under contention than the hand-written stress tests.
+# -n 0 so the threads share one interpreter; --parallel-threads=2 bounds
+# GPU-memory pressure and stays reproducible across runners.
+run_under_tsan python -m pytest -n 0 -v --parallel-threads=2 compute/test_no_numba.py
diff --git a/cccl_upstream/ci/test_cuda_compute_minimal_python_v2.sh b/cccl_upstream/ci/test_cuda_compute_minimal_python_v2.sh
new file mode 100755
index 00000000..a34c3e3e
--- /dev/null
+++ b/cccl_upstream/ci/test_cuda_compute_minimal_python_v2.sh
@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+# Run the minimal cuda.compute suite against a wheel built with the v2
+# (HostJIT) backend. The shared minimal script owns dependency installation and
+# selects the free-threading stress and serialization tests for Python 3.14t.
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+export CCCL_PYTHON_USE_V2=1
+exec "$ci_dir/test_cuda_compute_minimal_python.sh" "$@"
diff --git a/cccl_upstream/ci/test_cuda_compute_python.sh b/cccl_upstream/ci/test_cuda_compute_python.sh
new file mode 100755
index 00000000..01202944
--- /dev/null
+++ b/cccl_upstream/ci/test_cuda_compute_python.sh
@@ -0,0 +1,50 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "$ci_dir/pyenv_helper.sh"
+
+# Parse common arguments
+source "$ci_dir/util/python/common_arg_parser.sh"
+parse_python_args "$@"
+# Pin cuda-toolkit to the container's CTK minor and set cuda_version /
+# cuda_major_version (-ctk-mode latest opts out). See pyenv_helper.sh.
+pin_cuda_toolkit "${ctk_mode}"
+
+# Setup Python environment
+setup_python_env "${py_version}"
+
+# Fetch or build the cuda_cccl wheel:
+if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+  wheel_artifact_name=$("$ci_dir/util/workflow/get_wheel_artifact_name.sh")
+  "$ci_dir/util/artifacts/download.sh" "${wheel_artifact_name}" /home/coder/cccl/
+else
+  "$ci_dir/build_cuda_cccl_python.sh" -py-version "${py_version}"
+fi
+
+# Install cuda_cccl. The extra flavor is "cu" (pip-installed toolkit) or "sysctk"
+# (system-provided toolkit) depending on the -ctk-mode arg.
+CUDA_CCCL_WHEEL_PATH="$(ls /home/coder/cccl/wheelhouse/cuda_cccl-*.whl)"
+ctk_flavor="$(ctk_extra_flavor "${ctk_mode}")"
+python -m pip install "${CUDA_CCCL_WHEEL_PATH}[test-${ctk_flavor}${cuda_major_version}]"
+
+# Run tests for compute module.
+# On the v2 (HostJIT) backend, abort on first failure — the suite is still
+# stabilizing and a single early failure is enough signal to investigate
+# without scrolling through hundreds of subsequent passes.
+pytest_extra=()
+if [[ "${CCCL_PYTHON_USE_V2:-}" =~ ^(1|true|TRUE|on|ON)$ ]]; then
+  pytest_extra+=(-x)
+fi
+
+cd "/home/coder/cccl/python/cuda_cccl/tests/"
+if [[ "${CCCL_PYTHON_USE_V2:-}" =~ ^(1|true|TRUE|on|ON)$ ]]; then
+  # The test isolates itself in a fresh subprocess (LLVM initialization is
+  # process-wide and only cold once), but it carries the free_threading marker,
+  # so it must be selected by node-id here or the sweeps below never run it.
+  python -m pytest "${pytest_extra[@]}" -n 0 -v \
+    compute/test_free_threading_stress.py::test_v2_concurrent_cold_llvm_initialization
+fi
+python -m pytest "${pytest_extra[@]}" -n 6 -v compute/ -m "not large and not free_threading"
+python -m pytest "${pytest_extra[@]}" -n 0 -v compute/ -m "large and not free_threading"
diff --git a/cccl_upstream/ci/test_cuda_compute_python_v2.sh b/cccl_upstream/ci/test_cuda_compute_python_v2.sh
new file mode 100755
index 00000000..bd4dce57
--- /dev/null
+++ b/cccl_upstream/ci/test_cuda_compute_python_v2.sh
@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+# Run the cuda.compute pytest suite against a wheel built with the v2
+# (HostJIT) backend. Mirrors test_cuda_compute_python.sh; the only difference
+# is exporting CCCL_PYTHON_USE_V2 so the wheel build (and downstream pytest)
+# uses cccl.c.parallel.v2.
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+export CCCL_PYTHON_USE_V2=1
+exec "$ci_dir/test_cuda_compute_python.sh" "$@"
diff --git a/cccl_upstream/ci/test_cudax.sh b/cccl_upstream/ci/test_cudax.sh
new file mode 100755
index 00000000..7d9dac03
--- /dev/null
+++ b/cccl_upstream/ci/test_cudax.sh
@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+
+source "$(dirname "${BASH_SOURCE[0]}")/build_common.sh"
+
+print_environment_details
+
+./build_cudax.sh "$@"
+
+PRESET="cudax"
+
+test_preset "CUDA Experimental" "${PRESET}"
+
+print_time_summary
diff --git a/cccl_upstream/ci/test_libcudacxx.sh b/cccl_upstream/ci/test_libcudacxx.sh
new file mode 100755
index 00000000..5a6d28b7
--- /dev/null
+++ b/cccl_upstream/ci/test_libcudacxx.sh
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+
+# shellcheck source=ci/build_common.sh
+source "${ci_dir}/build_common.sh"
+
+print_environment_details
+
+PRESET="libcudacxx"
+CMAKE_OPTIONS=("-DCMAKE_CXX_STANDARD=${CXX_STANDARD}" "-DCMAKE_CUDA_STANDARD=${CXX_STANDARD}")
+
+if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+  producer_id="$(util/workflow/get_producer_id.sh)"
+  artifact="z_libcudacxx-test-artifacts-${DEVCONTAINER_NAME:?}-$producer_id"
+  run_command "📦  Unpacking artifact '$artifact'" \
+    "${ci_dir}/util/artifacts/download_packed.sh" "$artifact" /home/coder/cccl/
+else
+  "${ci_dir}/build_libcudacxx.sh" "$@"
+  configure_preset libcudacxx "$PRESET" "${CMAKE_OPTIONS[@]}"
+fi
+
+test_preset "libcudacxx (CTest)" "libcudacxx-ctest"
+
+sccache -z > /dev/null || :
+
+lit_test_name="libcudacxx (lit)"
+if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+  export LIT_OPTS="${LIT_OPTS:+${LIT_OPTS} }-Dtest_executable_mode=replay"
+  lit_test_name="libcudacxx (lit replay)"
+fi
+test_preset "${lit_test_name}" "libcudacxx-lit"
+
+sccache --show-adv-stats || :
+
+print_time_summary
diff --git a/cccl_upstream/ci/test_nvbench_helper.sh b/cccl_upstream/ci/test_nvbench_helper.sh
new file mode 100755
index 00000000..42193be2
--- /dev/null
+++ b/cccl_upstream/ci/test_nvbench_helper.sh
@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+
+# shellcheck source=ci/build_common.sh
+source "$(dirname "${BASH_SOURCE[0]}")/build_common.sh"
+
+print_environment_details
+
+PRESET="nvbench-helper"
+
+CMAKE_OPTIONS=()
+
+GPU_REQUIRED="true"
+
+configure_and_build_preset "NVBench Helper" "$PRESET" "${CMAKE_OPTIONS[@]}"
+test_preset "NVBench Helper" "$PRESET" "$GPU_REQUIRED"
+
+print_time_summary
diff --git a/cccl_upstream/ci/test_packaging.sh b/cccl_upstream/ci/test_packaging.sh
new file mode 100755
index 00000000..97fea776
--- /dev/null
+++ b/cccl_upstream/ci/test_packaging.sh
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(dirname "${BASH_SOURCE[0]}")"
+cccl_dir="$(realpath "${ci_dir}/..")"
+
+MIN_CMAKE=false
+minimum_cmake_version=3.18.0
+
+new_args="$("${ci_dir}/util/extract_switches.sh" -min-cmake -- "$@")"
+declare -a new_args="(${new_args})"
+set -- "${new_args[@]}"
+while true; do
+  case "$1" in
+  -min-cmake)
+    MIN_CMAKE=true
+    shift
+    ;;
+  --)
+    shift
+    break
+    ;;
+  *)
+    echo "Unknown argument: $1"
+    exit 1
+    ;;
+  esac
+done
+
+if $MIN_CMAKE; then
+  echo "Installing minimum CMake version v${minimum_cmake_version}..."
+  wget -q \
+    https://github.com/Kitware/CMake/releases/download/v"${minimum_cmake_version}"/cmake-"${minimum_cmake_version}"-Linux-x86_64.sh \
+    -O /tmp/cmake-install.sh
+  prefix=/tmp/cmake-${minimum_cmake_version}
+  mkdir -p "${prefix}"
+  bash /tmp/cmake-install.sh --skip-license --prefix="${prefix}"
+  export MIN_CTEST_COMMAND="${prefix}/bin/ctest"
+fi
+
+# Needs to happen after cmake is installed:
+# shellcheck source=ci/build_common.sh
+source "${ci_dir}/build_common.sh"
+cd "${ci_dir}"
+
+print_environment_details
+
+PRESET="packaging"
+
+CMAKE_OPTIONS=()
+
+GPU_REQUIRED="true"
+
+CMAKE_OPTIONS=("-DCCCL_EXAMPLE_CPM_REPOSITORY=${cccl_dir}")
+
+# Local -- build against the current repo's HEAD commit:
+if [[ -z "${GITHUB_ACTIONS:-}" ]]; then
+  CMAKE_OPTIONS+=("-DCCCL_EXAMPLE_CPM_TAG=HEAD")
+else
+  CMAKE_OPTIONS+=("-DCCCL_EXAMPLE_CPM_TAG=${GITHUB_SHA:?}")
+fi
+
+if [[ -n "${MIN_CTEST_COMMAND:-}" ]]; then
+  CMAKE_OPTIONS+=("-DCCCL_EXAMPLE_CTEST_COMMAND=${MIN_CTEST_COMMAND}")
+fi
+
+configure_and_build_preset "Packaging" "$PRESET" "${CMAKE_OPTIONS[@]}"
+test_preset "Packaging" "$PRESET" "$GPU_REQUIRED"
+
+print_time_summary
diff --git a/cccl_upstream/ci/test_python_common.sh b/cccl_upstream/ci/test_python_common.sh
new file mode 100644
index 00000000..e18d9b54
--- /dev/null
+++ b/cccl_upstream/ci/test_python_common.sh
@@ -0,0 +1,30 @@
+set -euo pipefail
+
+function list_environment {
+  begin_group "⚙️ Existing site-packages"
+  pip freeze
+  end_group "⚙️ Existing site-packages"
+}
+
+function run_tests {
+  module=$1
+
+  pushd "../python/${module}" >/dev/null
+
+  TEMP_VENV_DIR="/tmp/${module}_venv"
+  rm -rf "${TEMP_VENV_DIR}"
+  python -m venv "${TEMP_VENV_DIR}"
+  # shellcheck disable=SC1091
+  . "${TEMP_VENV_DIR}/bin/activate"
+  echo 'cuda-cccl @ file:///home/coder/cccl/python/cuda_cccl' > /tmp/cuda-cccl_constraints.txt
+  run_command "⚙️  Pip install ${module}" pip install -c /tmp/cuda-cccl_constraints.txt ".[test]"
+  begin_group "⚙️ ${module} site-packages"
+  pip freeze
+  end_group "⚙️ ${module} site-packages"
+  run_command "🚀  Pytest ${module}" pytest -n "${PARALLEL_LEVEL:-$(nproc --all --ignore=1)}" -v ./tests
+  deactivate
+
+  popd >/dev/null
+
+  print_time_summary
+}
diff --git a/cccl_upstream/ci/test_thrust.sh b/cccl_upstream/ci/test_thrust.sh
new file mode 100755
index 00000000..1b69622e
--- /dev/null
+++ b/cccl_upstream/ci/test_thrust.sh
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+CPU_ONLY=false
+GPU_ONLY=false
+
+ci_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+
+new_args="$("${ci_dir}/util/extract_switches.sh" -cpu-only -gpu-only -- "$@")"
+declare -a new_args="(${new_args})"
+set -- "${new_args[@]}"
+while true; do
+  case "$1" in
+  -cpu-only)
+    ARTIFACT_TAG="test_cpu"
+    CPU_ONLY=true
+    shift
+    ;;
+  -gpu-only)
+    ARTIFACT_TAG="test_gpu"
+    GPU_ONLY=true
+    shift
+    ;;
+  --)
+    shift
+    break
+    ;;
+  *)
+    echo "Unknown argument: $1"
+    exit 1
+    ;;
+  esac
+done
+
+# shellcheck source=ci/build_common.sh
+source "${ci_dir}/build_common.sh"
+
+print_environment_details
+
+if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+  if ! $CPU_ONLY && ! $GPU_ONLY; then
+    echo "Error: test_thrust.sh requires -cpu-only or -gpu-only in CI" >&2
+    exit 1
+  fi
+  producer_id="$(util/workflow/get_producer_id.sh)"
+  run_command "📦  Unpacking test artifacts" \
+    "${ci_dir}/util/artifacts/download_packed.sh" \
+      "z_thrust-test-artifacts-${DEVCONTAINER_NAME:?}-$producer_id-$ARTIFACT_TAG" \
+      /home/coder/cccl/
+else
+  ./build_thrust.sh "$@"
+fi
+
+declare -a PRESET_GPU_PAIRS=()
+
+if $CPU_ONLY; then
+  PRESET_GPU_PAIRS+=("thrust-cpu:false")
+elif $GPU_ONLY; then
+  PRESET_GPU_PAIRS+=("thrust-gpu:true")
+else
+  PRESET_GPU_PAIRS+=("thrust-cpu:false" "thrust-gpu:true")
+fi
+
+for pair in "${PRESET_GPU_PAIRS[@]}"; do
+  PRESET="${pair%%:*}"
+  GPU_REQUIRED="${pair##*:}"
+  test_preset "Thrust (${PRESET})" "${PRESET}" "${GPU_REQUIRED}"
+done
+
+print_time_summary
diff --git a/cccl_upstream/ci/update_rapids_version.sh b/cccl_upstream/ci/update_rapids_version.sh
new file mode 100755
index 00000000..b5804b71
--- /dev/null
+++ b/cccl_upstream/ci/update_rapids_version.sh
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+# Copyright (c) 2024-2025, NVIDIA CORPORATION.
+##########################
+# RAPIDS Version Updater #
+##########################
+
+## Usage
+# bash update_rapids_version.sh <new_version>
+
+# Format is YY.MM.PP - no leading 'v' or trailing 'a'
+NEXT_FULL_TAG=$1
+
+#Get <major>.<minor> for next version
+NEXT_MAJOR=$(echo "$NEXT_FULL_TAG" | awk '{split($0, a, "."); print a[1]}')
+NEXT_MINOR=$(echo "$NEXT_FULL_TAG" | awk '{split($0, a, "."); print a[2]}')
+# shellcheck disable=SC2034
+NEXT_PATCH=$(echo "$NEXT_FULL_TAG" | awk '{split($0, a, "."); print a[3]}')
+NEXT_SHORT_TAG=${NEXT_MAJOR}.${NEXT_MINOR}
+
+# Need to distutils-normalize the versions for some use cases
+NEXT_SHORT_TAG_PEP440=$(python -c "from packaging.version import Version; print(Version('${NEXT_SHORT_TAG}'))")
+
+echo "Updating RAPIDS and devcontainers to $NEXT_FULL_TAG"
+
+# Inplace sed replace; workaround for Linux and Mac
+function sed_runner() {
+    sed -i.bak ''"$1"'' "$2" && rm -f "${2}".bak
+}
+
+# Update CI files
+sed_runner "/devcontainer_version/ s/'[0-9.]*'/'${NEXT_SHORT_TAG}'/g" ci/matrix.yaml
+sed_runner "/devcontainer_version=/ s/=[0-9.]*/=${NEXT_SHORT_TAG}/g" ci/build_cuda_cccl_python.sh
+
+function update_devcontainer() {
+    sed_runner "s@rapidsai/devcontainers:[0-9.]*@rapidsai/devcontainers:${NEXT_SHORT_TAG}@g" "${1}"
+    sed_runner "s@rapidsai/devcontainers/features/rapids-build-utils:[0-9.]*@rapidsai/devcontainers/features/rapids-build-utils:${NEXT_SHORT_TAG_PEP440}@" "${1}"
+    sed_runner "s@\${localWorkspaceFolderBasename}-rapids-[0-9.]*@\${localWorkspaceFolderBasename}-rapids-${NEXT_SHORT_TAG}@g" "${1}"
+}
+
+# Update .devcontainer files
+find .devcontainer/ ci/rapids/ -type f -name devcontainer.json -print0 | while IFS= read -r -d '' filename; do
+    update_devcontainer "${filename}"
+done
diff --git a/cccl_upstream/ci/update_version.sh b/cccl_upstream/ci/update_version.sh
new file mode 100755
index 00000000..cd996b24
--- /dev/null
+++ b/cccl_upstream/ci/update_version.sh
@@ -0,0 +1,121 @@
+#!/usr/bin/env bash
+
+# Usage: ./update_version.sh [--dry-run] <major> <minor> <patch>
+# Example: ./update_version.sh --dry-run 2 2 1
+
+# Run in root cccl/
+cd "$(dirname "${BASH_SOURCE[0]}")/.." || exit
+
+DRY_RUN=false
+
+while [[ "$#" -gt 0 ]]; do
+    case "$1" in
+        --dry-run) DRY_RUN=true; ;;
+        *) break ;;
+    esac
+    shift
+done
+
+major="$1"
+minor="$2"
+patch="$3"
+
+if [[ -z "$major" ]] || [[ -z "$minor" ]] || [[ -z "$patch" ]]; then
+    echo "Usage: $0 [--dry-run] <major> <minor> <patch>"
+    exit 1
+fi
+
+# Version file paths
+REPO_VERSION_FILE="cccl-version.json"
+DOCS_VERSION_MD_FILE="docs/VERSION.md"
+CCCL_VERSION_FILE="libcudacxx/include/cuda/std/__cccl/version.h"
+THRUST_VERSION_FILE="thrust/thrust/version.h"
+CUB_VERSION_FILE="cub/cub/version.cuh"
+CCCL_CMAKE_VERSION_FILE="lib/cmake/cccl/cccl-config-version.cmake"
+CUB_CMAKE_VERSION_FILE="lib/cmake/cub/cub-config-version.cmake"
+LIBCUDACXX_CMAKE_VERSION_FILE="lib/cmake/libcudacxx/libcudacxx-config-version.cmake"
+THRUST_CMAKE_VERSION_FILE="lib/cmake/thrust/thrust-config-version.cmake"
+CUDAX_CMAKE_VERSION_FILE="lib/cmake/cudax/cudax-config-version.cmake"
+CUDA_CCCL_VERSION_FILE="python/cuda_cccl/cuda/cccl/_version.py"
+
+# Calculated version codes
+new_cccl_version=$((major * 1000000 + minor * 1000 + patch))     # MMMmmmppp
+new_thrust_cub_version=$((major * 100000 + minor * 100 + patch)) # MMMmmmpp
+
+# Fetch current version from file
+current_cccl_version=$(grep -oP "define CCCL_VERSION \K[0-9]+" "$CCCL_VERSION_FILE")
+
+# Fetch the latest tag from git and strip the 'v' prefix if present
+latest_tag=$(git tag --sort=-v:refname | head -n 1 | sed 's/^v//')
+
+# Since the tags and versions are numerically comparable, we cast them to integers
+latest_tag_version=$(echo "$latest_tag" | awk -F. '{ printf("%d%03d%03d", $1,$2,$3) }')
+
+echo "Running in $(pwd)"
+echo "New MMMmmmppp version: $new_cccl_version"
+echo "New MMMmmmpp  version: $new_thrust_cub_version"
+echo "Current CCCL version:  $current_cccl_version"
+echo "Latest git tag:        $latest_tag"
+
+# Check if new version is less than or equal to current or the latest tag
+if (( new_cccl_version < current_cccl_version )) || (( new_cccl_version < latest_tag_version )); then
+    echo "Error: New version $new_cccl_version is less than current version $current_cccl_version or latest git tag version $latest_tag_version."
+    exit 1
+fi
+
+update_file () {
+    local file=$1
+    local pattern=$2
+    local new_value=$3
+    if [[ "$DRY_RUN" = true ]]; then
+        local temp_file
+        temp_file=$(mktemp)
+        sed "s/$pattern/$new_value/g" "$file" > "$temp_file"
+        diff --color=auto -U 0 "$file" "$temp_file" || true
+        rm "$temp_file"
+    else
+        sed -i "s/$pattern/$new_value/" "$file"
+    fi
+}
+
+# Update version information in files:
+
+update_file "$REPO_VERSION_FILE" "  \"full\":.*," "  \"full\": \"$major.$minor.$patch\","
+update_file "$REPO_VERSION_FILE" "  \"major\":.*" "  \"major\": $major,"
+update_file "$REPO_VERSION_FILE" "  \"minor\":.*" "  \"minor\": $minor,"
+update_file "$REPO_VERSION_FILE" "  \"patch\":.*" "  \"patch\": $patch"
+
+update_file "$DOCS_VERSION_MD_FILE" ".*" "$major.$minor"
+
+update_file "$CCCL_VERSION_FILE" "^#define CCCL_VERSION \([0-9]\+\)" "#define CCCL_VERSION $new_cccl_version"
+update_file "$THRUST_VERSION_FILE" "^#define THRUST_VERSION \([0-9]\+\)" "#define THRUST_VERSION $new_thrust_cub_version"
+update_file "$CUB_VERSION_FILE" "^#define CUB_VERSION \([0-9]\+\)" "#define CUB_VERSION $new_thrust_cub_version"
+
+update_file "$CUB_CMAKE_VERSION_FILE" "set(CUB_VERSION_MAJOR \([0-9]\+\))" "set(CUB_VERSION_MAJOR $major)"
+update_file "$CUB_CMAKE_VERSION_FILE" "set(CUB_VERSION_MINOR \([0-9]\+\))" "set(CUB_VERSION_MINOR $minor)"
+update_file "$CUB_CMAKE_VERSION_FILE" "set(CUB_VERSION_PATCH \([0-9]\+\))" "set(CUB_VERSION_PATCH $patch)"
+
+update_file "$LIBCUDACXX_CMAKE_VERSION_FILE" "set(libcudacxx_VERSION_MAJOR \([0-9]\+\))" "set(libcudacxx_VERSION_MAJOR $major)"
+update_file "$LIBCUDACXX_CMAKE_VERSION_FILE" "set(libcudacxx_VERSION_MINOR \([0-9]\+\))" "set(libcudacxx_VERSION_MINOR $minor)"
+update_file "$LIBCUDACXX_CMAKE_VERSION_FILE" "set(libcudacxx_VERSION_PATCH \([0-9]\+\))" "set(libcudacxx_VERSION_PATCH $patch)"
+
+update_file "$THRUST_CMAKE_VERSION_FILE" "set(THRUST_VERSION_MAJOR \([0-9]\+\))" "set(THRUST_VERSION_MAJOR $major)"
+update_file "$THRUST_CMAKE_VERSION_FILE" "set(THRUST_VERSION_MINOR \([0-9]\+\))" "set(THRUST_VERSION_MINOR $minor)"
+update_file "$THRUST_CMAKE_VERSION_FILE" "set(THRUST_VERSION_PATCH \([0-9]\+\))" "set(THRUST_VERSION_PATCH $patch)"
+
+update_file "$CCCL_CMAKE_VERSION_FILE" "set(CCCL_VERSION_MAJOR \([0-9]\+\))" "set(CCCL_VERSION_MAJOR $major)"
+update_file "$CCCL_CMAKE_VERSION_FILE" "set(CCCL_VERSION_MINOR \([0-9]\+\))" "set(CCCL_VERSION_MINOR $minor)"
+update_file "$CCCL_CMAKE_VERSION_FILE" "set(CCCL_VERSION_PATCH \([0-9]\+\))" "set(CCCL_VERSION_PATCH $patch)"
+
+update_file "$CUDAX_CMAKE_VERSION_FILE" "set(cudax_VERSION_MAJOR \([0-9]\+\))" "set(cudax_VERSION_MAJOR $major)"
+update_file "$CUDAX_CMAKE_VERSION_FILE" "set(cudax_VERSION_MINOR \([0-9]\+\))" "set(cudax_VERSION_MINOR $minor)"
+update_file "$CUDAX_CMAKE_VERSION_FILE" "set(cudax_VERSION_PATCH \([0-9]\+\))" "set(cudax_VERSION_PATCH $patch)"
+
+update_file "$CUDA_CCCL_VERSION_FILE" "^__version__ = \"\([0-9.]\+\)\"" "__version__ = \"$major.$minor.$patch\""
+
+
+if [[ "$DRY_RUN" = true ]]; then
+    echo "Dry run completed. No changes made."
+else
+    echo "Version updated to $major.$minor.$patch"
+fi
diff --git a/cccl_upstream/ci/upload_cub_test_artifacts.sh b/cccl_upstream/ci/upload_cub_test_artifacts.sh
new file mode 100755
index 00000000..b734c292
--- /dev/null
+++ b/cccl_upstream/ci/upload_cub_test_artifacts.sh
@@ -0,0 +1,107 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+if [[ -z "${GITHUB_ACTIONS:-}" ]]; then
+  echo "This script must be run in a GitHub Actions environment." >&2
+  exit 1
+fi
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+readonly ci_dir
+repo_root="$(cd "${ci_dir}/.." && pwd)"
+readonly repo_root
+
+cd "$repo_root"
+
+if ! ci/util/workflow/has_consumers.sh; then
+  echo "No consumers found for this job. Exiting." >&2
+  exit 0
+fi
+
+if [[ "$#" -gt 0 ]]; then
+  preset_variants=("$@")
+else
+  # Figure out which artifacts need to be built:
+  consumers=$(ci/util/workflow/get_consumers.sh)
+  preset_variants=()
+  if grep -q "TestGPU" <<< "$consumers"; then
+    preset_variants+=("no_lid")
+  fi
+  if grep -q "HostLaunch" <<< "$consumers"; then
+    preset_variants+=("lid_0")
+  fi
+  if grep -q "DeviceLaunch" <<< "$consumers"; then
+    preset_variants+=("lid_1")
+  fi
+  if grep -q "GraphCapture" <<< "$consumers"; then
+    preset_variants+=("lid_2")
+  fi
+  # Limited jobs run the entire test suite:
+  if grep -q "SmallGMem" <<< "$consumers"; then
+    preset_variants+=("no_lid" "lid_0" "lid_1" "lid_2")
+  fi
+fi
+
+# Remove duplicates:
+mapfile -t preset_variants < <(echo "${preset_variants[*]}" | tr ' ' '\n' | sort -u)
+
+artifact_prefix="z_cub-test-artifacts-${DEVCONTAINER_NAME:?}-${JOB_ID:?}"
+
+# BUILD_INFIX is undefined on windows CI
+build_dir_regex="build${CCCL_BUILD_INFIX:+/$CCCL_BUILD_INFIX}/cub[^/]*"
+
+# Just collect the minimum set of files needed for running each ctest preset:
+for preset_variant in "${preset_variants[@]}"; do
+
+  # Shared across all presets:
+  ci/util/artifacts/stage.sh "$artifact_prefix-$preset_variant" \
+      "$build_dir_regex/build\.ninja$" \
+      "$build_dir_regex/.*rules\.ninja$" \
+      "$build_dir_regex/CMakeCache\.txt$" \
+      "$build_dir_regex/.*VerifyGlobs\.cmake$" \
+      "$build_dir_regex/.*CTestTestfile\.cmake$" \
+      "$build_dir_regex/cub/rapids-cmake/.*" > /dev/null
+
+  # Add per-preset executables:
+  if [[ "$preset_variant" == lid_* ]]; then
+    ci/util/artifacts/stage.sh  \
+        "$artifact_prefix-$preset_variant" \
+        "$build_dir_regex/bin/.*\.$preset_variant.*" > /dev/null
+    # The CUDA runtime smoke binary is invoked explicitly from build_common.sh
+    ci/util/artifacts/stage.sh  \
+        "$artifact_prefix-$preset_variant" \
+        "$build_dir_regex/bin/cccl\.test\.cuda_runtime_smoke$" > /dev/null
+    # Windows builds generate binaries for the header tests, remove these:
+    ci/util/artifacts/unstage.sh  \
+        "$artifact_prefix-$preset_variant" \
+        "$build_dir_regex/.*\.headers\..*" > /dev/null || :
+
+    ci/util/artifacts/upload_stage_packed.sh "$artifact_prefix-$preset_variant"
+  fi
+done
+
+if [[ " ${preset_variants[*]} " == *" no_lid "* ]]; then
+  # Initially add all binaries to no_lid, then remove the lid variants in later passes:
+  ci/util/artifacts/stage.sh \
+      "$artifact_prefix-no_lid" \
+      "$build_dir_regex/bin/.*" > /dev/null
+  # Remove the benchmarks if present, we don't run those in CI, just build them:
+  ci/util/artifacts/unstage.sh \
+      "$artifact_prefix-no_lid" \
+      "$build_dir_regex/.*\.bench\..*" > /dev/null || :
+  # Remove all lid variants:
+  ci/util/artifacts/unstage.sh \
+      "$artifact_prefix-no_lid" \
+      "$build_dir_regex/.*\.lid_[0-2].*" > /dev/null
+  # Windows builds generate binaries for the header tests, remove these:
+  ci/util/artifacts/unstage.sh  \
+      "$artifact_prefix-no_lid" \
+      "$build_dir_regex/.*\.headers\..*" > /dev/null || :
+  # These cubin outputs are needed for FileCheck tests in test/cubin-check
+  ci/util/artifacts/stage.sh \
+      "$artifact_prefix-no_lid" \
+      "$build_dir_regex/cub/test/cubin-check/.*\.cubin$" > /dev/null || :
+
+  ci/util/artifacts/upload_stage_packed.sh "$artifact_prefix-no_lid"
+fi
diff --git a/cccl_upstream/ci/upload_job_result_artifacts.sh b/cccl_upstream/ci/upload_job_result_artifacts.sh
new file mode 100755
index 00000000..a73bba1e
--- /dev/null
+++ b/cccl_upstream/ci/upload_job_result_artifacts.sh
@@ -0,0 +1,66 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+if [[ -z "${GITHUB_ACTIONS:-}" ]]; then
+  echo "This script must be run in a GitHub Actions environment." >&2
+  exit 1
+fi
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+readonly ci_dir
+repo_root="$(cd "${ci_dir}/.." && pwd)"
+readonly repo_root
+
+cd "$repo_root"
+
+usage=$(cat <<EOF
+Usage: $0 <job_id> <exit code>
+EOF
+)
+
+if [[ "$#" -ne 2 ]]; then
+  echo "Error: Invalid number of arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+job_id="$1"
+exit_code="$2"
+
+# Collect workflow-related artifacts -- success state, sccache info, etc.
+# These are unpacked and parsed in the workflow-results action.
+
+source ci/util/artifacts/common.sh
+
+# The root of the shared artifact structure:
+jobs_artifact_dir="$ARTIFACT_UPLOAD_STAGE/jobs"
+
+# This job's artifact directory:
+job_artifacts="$jobs_artifact_dir/$job_id"
+mkdir -p "$job_artifacts"
+
+if [[ "$exit_code" -eq 0 ]]; then
+  touch "$job_artifacts/success"
+fi
+
+# Finds a matching file in the root and copies it to the artifact directory.
+find_and_copy_job_artifact_from() {
+  root="$1"
+  name="$2"
+  if find "$root"/ -maxdepth 4 -name "$name" -type f -printf '' -quit 2>/dev/null; then
+    find "$root"/ -maxdepth 4 -name "$name" -type f -print0 | xargs -0 -P4 -I% cp -v % "$job_artifacts"/
+  else
+    echo "No file matching '$name' found in '$root'."
+    return 1
+  fi
+}
+
+find_and_copy_job_artifact_from /tmp  "sccache*.log"       || : # Nonfatal if not found
+find_and_copy_job_artifact_from build "sccache_stats.json" || : # Nonfatal if not found
+find_and_copy_job_artifact_from build ".ninja_log"         || : # Nonfatal if not found
+find_and_copy_job_artifact_from build "build.ninja"        || : # Nonfatal if not found
+find_and_copy_job_artifact_from build "rules.ninja"        || : # Nonfatal if not found
+find_and_copy_job_artifact_from build "ctest.log"          || : # Nonfatal if not found
+
+ci/util/artifacts/upload/register.sh "zz_jobs-$job_id" "$jobs_artifact_dir"
diff --git a/cccl_upstream/ci/upload_libcudacxx_test_artifacts.sh b/cccl_upstream/ci/upload_libcudacxx_test_artifacts.sh
new file mode 100755
index 00000000..34e2248c
--- /dev/null
+++ b/cccl_upstream/ci/upload_libcudacxx_test_artifacts.sh
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+if [[ -z "${GITHUB_ACTIONS:-}" ]]; then
+  echo "This script must be run in a GitHub Actions environment." >&2
+  exit 1
+fi
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+readonly ci_dir
+repo_root="$(cd "${ci_dir}/.." && pwd)"
+readonly repo_root
+
+cd "$repo_root"
+
+if ! ci/util/workflow/has_consumers.sh; then
+  echo "No consumers found for this job. Exiting." >&2
+  exit 0
+fi
+
+artifact="z_libcudacxx-test-artifacts-${DEVCONTAINER_NAME:?}-${JOB_ID:?}"
+
+# BUILD_INFIX is undefined on windows CI.
+build_dir_regex="build${CCCL_BUILD_INFIX:+/$CCCL_BUILD_INFIX}/libcudacxx[^/]*"
+lit_executable_regex="$build_dir_regex/libcudacxx/test/libcudacxx/test/.*Output/.*\.exe$"
+
+# Minimum CTest/lit metadata needed to run from the unpacked build tree.
+ci/util/artifacts/stage.sh "$artifact" \
+    "$build_dir_regex/build\.ninja$" \
+    "$build_dir_regex/.*rules\.ninja$" \
+    "$build_dir_regex/CMakeCache\.txt$" \
+    "$build_dir_regex/.*VerifyGlobs\.cmake$" \
+    "$build_dir_regex/.*CTestTestfile\.cmake$" \
+    "$build_dir_regex/libcudacxx/test/libcudacxx/lit\.site\.cfg$" \
+    > /dev/null
+
+# Test executables plus shared libraries/smoke binaries used by CTest/lit.
+ci/util/artifacts/stage.sh "$artifact" \
+    "$build_dir_regex/bin/.*" \
+    "$build_dir_regex/lib/.*" \
+    "$lit_executable_regex" \
+    > /dev/null
+
+if ! find . -type f -regex "\./$lit_executable_regex" -print -quit | grep -q .; then
+  echo "No lit test executables found for artifact '$artifact'." >&2
+  exit 1
+fi
+
+# Windows builds generate binaries for header tests that are never executed.
+ci/util/artifacts/unstage.sh \
+    "$artifact" \
+    "$build_dir_regex/.*\.headers\..*" > /dev/null || :
+
+ci/util/artifacts/upload_stage_packed.sh "$artifact"
diff --git a/cccl_upstream/ci/upload_thrust_test_artifacts.sh b/cccl_upstream/ci/upload_thrust_test_artifacts.sh
new file mode 100755
index 00000000..2065abe1
--- /dev/null
+++ b/cccl_upstream/ci/upload_thrust_test_artifacts.sh
@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+if [[ -z "${GITHUB_ACTIONS:-}" ]]; then
+  echo "This script must be run in a GitHub Actions environment." >&2
+  exit 1
+fi
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+readonly ci_dir
+repo_root="$(cd "${ci_dir}/.." && pwd)"
+readonly repo_root
+
+cd "$repo_root"
+
+if ! ci/util/workflow/has_consumers.sh; then
+  echo "No consumers found for this job. Exiting." >&2
+  exit 0
+fi
+
+# Figure out which artifact needs to be built:
+consumers=$(ci/util/workflow/get_consumers.sh)
+preset_variants=()
+if grep -q "TestGPU" <<< "$consumers"; then
+  preset_variants+=("test_gpu")
+fi
+if grep -q "TestCPU" <<< "$consumers"; then
+  preset_variants+=("test_cpu")
+fi
+
+artifact_prefix="z_thrust-test-artifacts-${DEVCONTAINER_NAME:?}-${JOB_ID:?}"
+
+# BUILD_INFIX is undefined on windows CI
+build_dir_regex="build${CCCL_BUILD_INFIX:+/$CCCL_BUILD_INFIX}/thrust[^/]*"
+
+# Just collect the minimum set of files needed for running each ctest preset:
+for preset_variant in "${preset_variants[@]}"; do
+  # Shared across all presets:
+  ci/util/artifacts/stage.sh "$artifact_prefix-$preset_variant" \
+      "$build_dir_regex/build\.ninja$" \
+      "$build_dir_regex/.*rules\.ninja$" \
+      "$build_dir_regex/CMakeCache\.txt$" \
+      "$build_dir_regex/.*VerifyGlobs\.cmake$" \
+      "$build_dir_regex/.*CTestTestfile\.cmake$" \
+      > /dev/null
+done
+
+if [[ " ${preset_variants[*]} " == *" test_cpu "* ]]; then
+  # Initially add all binaries, then remove all containing 'cuda' in the name:
+  ci/util/artifacts/stage.sh \
+      "$artifact_prefix-test_cpu" \
+      "$build_dir_regex/bin/.*" > /dev/null
+  ci/util/artifacts/unstage.sh \
+      "$artifact_prefix-test_cpu" \
+      "$build_dir_regex/bin/thrust\..*\.cuda\..*" > /dev/null
+
+  ci/util/artifacts/stage.sh \
+      "$artifact_prefix-test_cpu" \
+      "$build_dir_regex/lib/.*\.test\.framework\..*" > /dev/null
+  ci/util/artifacts/unstage.sh \
+      "$artifact_prefix-test_cpu" \
+      "$build_dir_regex/lib/.*\.cuda\.test\.framework\..*" > /dev/null
+
+  # Windows builds generate binaries for the header tests, remove these:
+  ci/util/artifacts/unstage.sh  \
+      "$artifact_prefix-test_cpu" \
+      "$build_dir_regex/.*\.headers\..*" > /dev/null || :
+
+  ci/util/artifacts/upload_stage_packed.sh "$artifact_prefix-test_cpu"
+fi
+
+if [[ " ${preset_variants[*]} " == *" test_gpu "* ]]; then
+  # Only binaries containing 'cuda':
+  ci/util/artifacts/stage.sh \
+      "$artifact_prefix-test_gpu" \
+      "$build_dir_regex/bin/thrust\..*\.cuda\..*" > /dev/null
+  ci/util/artifacts/stage.sh \
+      "$artifact_prefix-test_gpu" \
+      "$build_dir_regex/lib/.*\.cuda\.test\.framework\..*" > /dev/null
+  # The CUDA runtime smoke binary is invoked explicitly from build_common.sh
+  ci/util/artifacts/stage.sh \
+      "$artifact_prefix-test_gpu" \
+      "$build_dir_regex/bin/cccl\.test\.cuda_runtime_smoke$" > /dev/null
+
+  # Windows builds generate binaries for the header tests, remove these:
+  ci/util/artifacts/unstage.sh  \
+      "$artifact_prefix-test_gpu" \
+      "$build_dir_regex/.*\.headers\..*" > /dev/null || :
+
+
+  ci/util/artifacts/upload_stage_packed.sh "$artifact_prefix-test_gpu"
+fi
diff --git a/cccl_upstream/ci/util/artifacts/common.sh b/cccl_upstream/ci/util/artifacts/common.sh
new file mode 100755
index 00000000..d7c72013
--- /dev/null
+++ b/cccl_upstream/ci/util/artifacts/common.sh
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
+  echo "This script must be sourced, not executed directly." >&2
+  exit 1
+fi
+
+if [[ -z "${GITHUB_ACTIONS:-}" ]]; then
+  echo "This script must be run in a GitHub Actions environment." >&2
+  exit 1
+fi
+
+to_posix_path() {
+  local path="$1"
+
+  if [[ "$path" =~ ^([A-Za-z]):([\\/]?.*)$ ]]; then
+    local drive="${BASH_REMATCH[1]}"
+    local rest="${BASH_REMATCH[2]}"
+    rest="${rest//\\/\/}"
+    printf '/%s%s\n' "${drive,,}" "$rest"
+    return
+  fi
+
+  printf '%s\n' "$path"
+}
+
+runner_temp_posix="$(to_posix_path "${RUNNER_TEMP:-/tmp}")"
+
+export ARTIFACT_UPLOAD_STAGE="${runner_temp_posix}/artifact_upload_stage"
+export ARTIFACT_ARCHIVES="${runner_temp_posix}/artifact_archives"
+export ARTIFACT_UPLOAD_REGISTERY="${ARTIFACT_UPLOAD_STAGE}/artifact_upload_registry.json"
+
+mkdir -p "$ARTIFACT_UPLOAD_STAGE" "$ARTIFACT_ARCHIVES"
+
+if [[ ! -f "$ARTIFACT_UPLOAD_REGISTERY" ]]; then
+  echo "[]" > "$ARTIFACT_UPLOAD_REGISTERY"
+fi
diff --git a/cccl_upstream/ci/util/artifacts/download.sh b/cccl_upstream/ci/util/artifacts/download.sh
new file mode 100755
index 00000000..96f2ca01
--- /dev/null
+++ b/cccl_upstream/ci/util/artifacts/download.sh
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/artifacts/common.sh
+source "$ci_dir/util/artifacts/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 <name> [<path>]
+
+Download artifacts uploaded by other jobs in this CI run.
+
+Example Usage:
+  Download an artifact to the current directory:
+    $0 source_artifact.tar.gz
+
+  Download a packed artifact and extract it to the provided path:
+    $0 job-\$ID-products some/path/
+EOF
+)
+readonly usage
+
+if [[ "$#" -lt 1 ]]; then
+  echo "Error: Missing artifact name." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+readonly artifact_name="$1"
+
+if [[ "$#" -eq 1 ]]; then
+  artifact_path="./"
+else
+  artifact_path="$2"
+fi
+
+start=$SECONDS
+"$ci_dir/util/artifacts/download/fetch.sh" "$artifact_name" "$artifact_path"
+echo "Artifact '$artifact_name' downloaded to '$artifact_path' in $((SECONDS - start)) seconds."
diff --git a/cccl_upstream/ci/util/artifacts/download/fetch.sh b/cccl_upstream/ci/util/artifacts/download/fetch.sh
new file mode 100755
index 00000000..7694fe71
--- /dev/null
+++ b/cccl_upstream/ci/util/artifacts/download/fetch.sh
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/artifacts/common.sh
+source "$ci_dir/util/artifacts/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 <artifact_name> <target_directory>
+
+Downloads files from a named artifact from the current CI workflow run into the specified directory.
+
+Example Usages:
+  - $0 my_artifact.tar.gz ./
+  - $0 my_artifact /path/to/some/directory/
+EOF
+)
+readonly usage
+
+if [[ "$#" -lt 2 ]]; then
+  echo "Error: Missing arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+readonly artifact_name="$1"
+
+# Create the target directory and then get its absolute path
+mkdir -p "$2"
+target_directory="$(cd "$2" && pwd)"
+readonly target_directory
+
+echo "Downloading artifact '$artifact_name' to '$target_directory'"
+# shellcheck disable=SC2154
+"$ci_dir/util/retry.sh" 5 30 \
+  gh run download "${GITHUB_RUN_ID}" \
+    --name "$artifact_name" \
+    --dir "$target_directory"
diff --git a/cccl_upstream/ci/util/artifacts/download/unpack.sh b/cccl_upstream/ci/util/artifacts/download/unpack.sh
new file mode 100755
index 00000000..84c6f103
--- /dev/null
+++ b/cccl_upstream/ci/util/artifacts/download/unpack.sh
@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/artifacts/common.sh
+source "$ci_dir/util/artifacts/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 <artifact_name> <artifact_path>
+
+Unpacks a fetched packed artifact's tar.zst archive into the specified directory.
+
+Example Usages:
+  - $0 /tmp/my_artifact.tar.zst /tmp/my_artifact
+  - $0 /path/to/archive.tar.zst /path/to/extract/
+EOF
+)
+readonly usage
+
+if [[ "$#" -lt 2 ]]; then
+  echo "Error: Missing arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+if ! command -v zstd > /dev/null 2>&1; then
+  echo "Error: zstd not found." >&2
+  exit 1
+fi
+
+readonly artifact_name="$1"
+readonly artifact_path="$2"
+
+readonly artifact_archive="$ARTIFACT_ARCHIVES/${artifact_name}.tar.zst"
+
+echo "Unpacking artifact from '$artifact_archive' to '$artifact_path'"
+echo "Using zstd executable: $(command -v zstd)"
+
+# Create the artifact path directory if it doesn't exist
+mkdir -p "$artifact_path"
+
+zstd --decompress --threads=0 --stdout "$artifact_archive" \
+  | tar -xv -C "$artifact_path"
diff --git a/cccl_upstream/ci/util/artifacts/download_packed.sh b/cccl_upstream/ci/util/artifacts/download_packed.sh
new file mode 100755
index 00000000..5cf4a6f7
--- /dev/null
+++ b/cccl_upstream/ci/util/artifacts/download_packed.sh
@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/artifacts/common.sh
+source "$ci_dir/util/artifacts/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 <name> [<path>]
+
+Download and extracts a packed artifact uploaded by another job in this CI run.
+
+Example Usage:
+  Download an artifact to the current directory:
+    $0 source_artifact.tar.gz
+
+  Download a packed artifact and extract it to the provided path:
+    $0 job-\$ID-products build/
+EOF
+)
+readonly usage
+
+if [[ "$#" -lt 1 ]]; then
+  echo "Error: Missing artifact name." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+readonly artifact_name="$1"
+
+if [[ "$#" -eq 1 ]]; then
+  artifact_path="./"
+else
+  artifact_path="$2"
+fi
+
+start=$SECONDS
+"$ci_dir/util/artifacts/download/fetch.sh" "$artifact_name" "${ARTIFACT_ARCHIVES}"
+fetched=$SECONDS
+"$ci_dir/util/artifacts/download/unpack.sh" "$artifact_name" "$artifact_path"
+unpacked=$SECONDS
+
+echo "Artifact '$artifact_name' fetched in $((fetched - start)) seconds."
+echo "Artifact '$artifact_name' unpacked in $((unpacked - fetched)) seconds."
diff --git a/cccl_upstream/ci/util/artifacts/stage.sh b/cccl_upstream/ci/util/artifacts/stage.sh
new file mode 100755
index 00000000..30eed04d
--- /dev/null
+++ b/cccl_upstream/ci/util/artifacts/stage.sh
@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/artifacts/common.sh
+source "$ci_dir/util/artifacts/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 <artifact_name> <regex> [<regex> ...]
+
+Stages files matching the provided regexes path for upload under the specified artifact.
+Regexes are passed to the 'find' command's -regex option within the artifact stage path and implicitly
+start with '^\\./'.
+
+Staged files can be unstaged using the 'artifact/unstage.sh' script.
+All stage / unstage operations on the same artifact must be performed from the same working directory.
+
+Once a stage is complete, 'artifact/upload_stage_packed.sh' can be used to create a packed artifact
+from the stage. See also 'artifact/upload/pack.sh' and 'artifact/upload/build.sh' for more staging options.
+
+Example Usage:
+
+Stage built binaries and .cmake files in \${ARTIFACT_UPLOAD_STAGE}/test_artifacts for upload:
+  $0 test_artifacts 'bin/.*' 'lib/.*' '.*cmake$'
+EOF
+)
+readonly usage
+
+if [[ "$#" -lt 2 ]]; then
+  echo "Error: Missing arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+artifact_name="$1"
+shift
+regexes=("$@")
+
+artifact_stage_path="${ARTIFACT_UPLOAD_STAGE}/${artifact_name}"
+if [[ "$artifact_stage_path" != /* ]]; then
+  artifact_stage_path="$(pwd)/$artifact_stage_path"
+fi
+
+mkdir -p "$artifact_stage_path"
+
+artifact_index_file="$artifact_stage_path/artifact_index.txt"
+artifact_index_cwd="$artifact_stage_path/artifact_index_cwd.txt"
+
+if [[ -f "$artifact_index_cwd" ]]; then
+  # Check that the cwd matches the original staging directory if the index already exists:
+  if [[ "$(cat "$artifact_index_cwd")" != "$(pwd)" ]]; then
+    echo "Error: The current working directory has changed since the artifact was staged." >&2
+    echo "Cannot currently stage files from multiple source directories." >&2
+    exit 1
+  fi
+else
+  pwd > "$artifact_index_cwd"
+fi
+
+echo "Staging artifacts in '$artifact_stage_path'"
+for regex in "${regexes[@]}"; do
+  # Prepend './' to the regex for convenience. There's an implied ^ at the start of the find regex,
+  # and paths always start with ./, so this lets us match top level files directly.
+  regex="\\./$regex"
+  echo "Staging files matching regex: $regex"
+  find . -type f -regex "$regex" | tee -a "$artifact_index_file"
+  echo
+done
diff --git a/cccl_upstream/ci/util/artifacts/unstage.sh b/cccl_upstream/ci/util/artifacts/unstage.sh
new file mode 100755
index 00000000..ebf23507
--- /dev/null
+++ b/cccl_upstream/ci/util/artifacts/unstage.sh
@@ -0,0 +1,56 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/artifacts/common.sh
+source "$ci_dir/util/artifacts/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 <artifact_name> <regex> [<regex> ...]
+
+Unstages (removes) files matching the provided regexes from the specified artifact stage.
+Regexes follow the same rules as artifact/stage.sh.
+
+This may be used to remove files that were previously staged for upload before packing or building the artifacts.
+
+Example Usage:
+
+Unstage previously-staged built binaries and .cmake files from test_artifacts:
+  $0 test_artifacts 'bin/.*' 'lib/.*' '.*cmake$'
+EOF
+)
+readonly usage
+
+if [[ "$#" -lt 2 ]]; then
+  echo "Error: Missing arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+artifact_name="$1"
+shift
+regexes=("$@")
+
+artifact_stage_path="${ARTIFACT_UPLOAD_STAGE}/${artifact_name}"
+if [[ "$artifact_stage_path" != /* ]]; then
+  artifact_stage_path="$(pwd)/$artifact_stage_path"
+fi
+
+mkdir -p "$artifact_stage_path"
+
+artifact_index_file="$artifact_stage_path/artifact_index.txt"
+artifact_index_cwd="$artifact_stage_path/artifact_index_cwd.txt"
+
+pwd > "$artifact_index_cwd"
+
+echo "Unstaging artifacts in '$artifact_stage_path'"
+for regex in "${regexes[@]}"; do
+  # Modify regex for consistency with staging script:
+  regex="^\\./$regex"
+  echo "Unstaging files matching regex: $regex"
+  grep -E "$regex" "$artifact_index_file"
+  grep -v -E "$regex" "$artifact_index_file" > "${artifact_index_file}.tmp" && \
+    mv "${artifact_index_file}.tmp" "$artifact_index_file"
+done
diff --git a/cccl_upstream/ci/util/artifacts/upload.sh b/cccl_upstream/ci/util/artifacts/upload.sh
new file mode 100755
index 00000000..2a2c7f64
--- /dev/null
+++ b/cccl_upstream/ci/util/artifacts/upload.sh
@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/artifacts/common.sh
+source "$ci_dir/util/artifacts/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 <name> [<regex> ...]
+
+Creates an artifact consisting of a zip file containing a single file or set of regex matches.
+
+Regexes are passed to the $(command -v find) command's -regex option in the current directory.
+'./' is prepended to all regexes for convenience.
+The artifact will contain all matching files with paths relative to the current directory.
+
+If no regexes are provided, the artifact will be created from a file in the current directory.
+The file must have the same name as the artifact.
+
+Example Usage:
+
+  Create an artifact of the given file in the current directory using the filename as the artifact name:
+
+    $0 some_resource.log
+
+  Copy all files that match the regexes to a staging directory, and upload a artifact that zips this directory.
+
+    $0 job-\$JOB_ID-products \
+       'build\.ninja$' \
+       '.*rules\.ninja$' \
+       'CMakeCache\.txt$' \
+       '.*VerifyGlobs\.cmake$' \
+       '.*CTestTestfile\.cmake$' \
+       'bin/.*' \
+       'lib/.*'
+EOF
+)
+readonly usage
+
+if [[ "$#" -lt 1 ]]; then
+  echo "Error: Missing artifact name." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+readonly artifact_name="$1"
+
+# If no regexes are provided, use the artifact name as the path:
+if [[ "$#" -eq 1 ]]; then
+  "$ci_dir/util/artifacts/upload/register.sh" "$artifact_name" "$artifact_name"
+  exit
+fi
+
+shift
+
+"$ci_dir/util/artifacts/stage.sh" "$artifact_name" "$@" > /dev/null
+"$ci_dir/util/artifacts/upload_stage.sh" "$artifact_name"
diff --git a/cccl_upstream/ci/util/artifacts/upload/build.sh b/cccl_upstream/ci/util/artifacts/upload/build.sh
new file mode 100755
index 00000000..308e1d57
--- /dev/null
+++ b/cccl_upstream/ci/util/artifacts/upload/build.sh
@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/artifacts/common.sh
+source "$ci_dir/util/artifacts/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 <artifact_name>
+
+Builds a physical tree containing a staged artifact created using artifact/stage.sh / unstage.sh.
+
+The artifact root will be located at \${ARTIFACT_UPLOAD_STAGE}/<artifact_name>/<artifact_name>.
+EOF
+)
+readonly usage
+
+if [[ "$#" -ne 1 ]]; then
+  echo "Error: Invalid number of arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+readonly artifact_name="$1"
+readonly artifact_stage_path="${ARTIFACT_UPLOAD_STAGE}/${artifact_name}"
+readonly artifact_index_file="$artifact_stage_path/artifact_index.txt"
+readonly artifact_cwd_file="$artifact_stage_path/artifact_index_cwd.txt"
+readonly artifact_dir="${ARTIFACT_UPLOAD_STAGE}/${artifact_name}/${artifact_name}"
+artifact_cwd="$(cat "$artifact_cwd_file")"
+readonly artifact_cwd
+
+mkdir -p "$artifact_dir"
+
+echo "Building artifact '$artifact_name' in '$artifact_dir'"
+echo "Pulling artifacts from working directory: $artifact_cwd"
+
+(
+  cd "$artifact_cwd"
+  while IFS= read -r file; do
+    cp -v --parents "$file" "$artifact_dir"
+  done < "$artifact_index_file"
+)
diff --git a/cccl_upstream/ci/util/artifacts/upload/pack.sh b/cccl_upstream/ci/util/artifacts/upload/pack.sh
new file mode 100755
index 00000000..3410a107
--- /dev/null
+++ b/cccl_upstream/ci/util/artifacts/upload/pack.sh
@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/artifacts/common.sh
+source "$ci_dir/util/artifacts/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 <artifact_name>
+
+Packs a staged artifact (created using artifact/stage.sh) into a tar.zst archive.
+
+The archive will be generated from the staged index file in \${ARTIFACT_UPLOAD_STAGE}/<artifact_name> and
+saved to \${ARTIFACT_UPLOAD_STAGE}/<artifact_name>/<artifact_name>.tar.zst.
+
+Example Usages:
+  - $0 test_artifact
+EOF
+)
+readonly usage
+
+if [[ "$#" -ne 1 ]]; then
+  echo "Error: Invalid number of arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+if ! command -v zstd > /dev/null 2>&1; then
+  echo "Error: zstd not found." >&2
+  exit 1
+fi
+
+readonly artifact_name="$1"
+readonly artifact_stage_path="${ARTIFACT_UPLOAD_STAGE}/${artifact_name}"
+readonly artifact_index_file="$artifact_stage_path/artifact_index.txt"
+readonly artifact_cwd_file="$artifact_stage_path/artifact_index_cwd.txt"
+readonly artifact_archive="${ARTIFACT_UPLOAD_STAGE}/${artifact_name}/${artifact_name}.tar.zst"
+
+echo "Packing artifact '$artifact_stage_path' into '$artifact_archive'"
+echo "Using zstd: $(command -v zstd)"
+echo "Pulling artifacts from working directory: $(cat "$artifact_cwd_file")"
+
+tar -cv -C "$(cat "$artifact_cwd_file")" -T "$artifact_index_file" \
+  | zstd --compress --threads=0 \
+  > "$artifact_archive"
diff --git a/cccl_upstream/ci/util/artifacts/upload/print_matrix.sh b/cccl_upstream/ci/util/artifacts/upload/print_matrix.sh
new file mode 100755
index 00000000..7aacaad1
--- /dev/null
+++ b/cccl_upstream/ci/util/artifacts/upload/print_matrix.sh
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/artifacts/common.sh
+source "$ci_dir/util/artifacts/common.sh"
+
+usage=$(cat <<EOF
+Prints the Github Actions matrix for uploading all registered artifacts.
+EOF
+)
+readonly usage
+
+if [[ "$#" -ne 0 ]]; then
+  echo "$usage" >&2
+  exit 1
+fi
+
+jq -c '.' "${ARTIFACT_UPLOAD_REGISTERY:?}"
diff --git a/cccl_upstream/ci/util/artifacts/upload/register.sh b/cccl_upstream/ci/util/artifacts/upload/register.sh
new file mode 100755
index 00000000..babf3713
--- /dev/null
+++ b/cccl_upstream/ci/util/artifacts/upload/register.sh
@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/artifacts/common.sh
+source "$ci_dir/util/artifacts/common.sh"
+
+readonly artifact_compression_level=6
+readonly artifact_retention_days=7
+
+usage=$(cat <<EOF
+Usage: $0 <artifact_name> [<artifact_path>]
+
+Registers artifacts for upload. If path is not provided, it defaults to the artifact name.
+
+Compression level is set to $artifact_compression_level by default.
+Use 'upload/set_compression_level.sh' to change this after registering if needed.
+
+Default retention days is set to $artifact_retention_days.
+Use 'upload/set_retention_days.sh' after registering to change this if needed.
+
+Example Usages:
+  - $0 my_artifact.tar.gz # Assumes the artifact is in the current directory.
+  - $0 my_artifact /path/to/my_artifact.tar.gz
+  - $0 my_artifact /path/to/my_artifact_directory/
+EOF
+)
+readonly usage
+
+if [[ "$#" -lt 1 ]]; then
+  echo "Error: Missing artifact name." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+artifact_name="$1"
+artifact_path="${2:-$artifact_name}"
+
+# Ensure the artifact path is absolute
+if [[ "$artifact_path" != /* ]]; then
+  artifact_path="$(pwd)/$artifact_path"
+fi
+
+if [[ ! -e "$artifact_path" ]]; then
+  echo "Error: Artifact path '$artifact_path' does not exist." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+# Register the artifact:
+jq --arg name "$artifact_name" \
+  --arg path "$artifact_path" \
+  --arg retention_days "$artifact_retention_days" \
+  --argjson compression_level "$artifact_compression_level" \
+  '. += [{"name": $name, "path": $path, "retention_days": ($retention_days | tonumber), "compression_level": $compression_level}]' \
+  "$ARTIFACT_UPLOAD_REGISTERY" > "$ARTIFACT_UPLOAD_REGISTERY.tmp" && \
+  mv "$ARTIFACT_UPLOAD_REGISTERY.tmp" "$ARTIFACT_UPLOAD_REGISTERY"
+
+echo "Artifact '$artifact_name' registered for upload with path '$artifact_path'."
diff --git a/cccl_upstream/ci/util/artifacts/upload/set_compression_level.sh b/cccl_upstream/ci/util/artifacts/upload/set_compression_level.sh
new file mode 100755
index 00000000..0fbb5a01
--- /dev/null
+++ b/cccl_upstream/ci/util/artifacts/upload/set_compression_level.sh
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/artifacts/common.sh
+source "$ci_dir/util/artifacts/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 <artifact_name> <compression_level>
+
+Sets the compression level for an artifact registered for upload.
+
+Example Usage:
+  $0 some_huge_precompressed_archive 0
+  $0 some_many_small_uncompressed_files 10
+EOF
+)
+readonly usage
+
+if [[ "$#" -lt 2 ]]; then
+  echo "Error: Missing arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+artifact_name="$1"
+compression_level="$2"
+
+# Find the artifact entry and update its compression level
+jq --arg name "$artifact_name" --argjson compression_level "$compression_level" \
+  'map(if .name == $name then .compression_level = $compression_level else . end)' \
+  "$ARTIFACT_UPLOAD_REGISTERY" > "$ARTIFACT_UPLOAD_REGISTERY.tmp" && \
+  mv "$ARTIFACT_UPLOAD_REGISTERY.tmp" "$ARTIFACT_UPLOAD_REGISTERY"
diff --git a/cccl_upstream/ci/util/artifacts/upload/set_retention_days.sh b/cccl_upstream/ci/util/artifacts/upload/set_retention_days.sh
new file mode 100755
index 00000000..3db5a6c0
--- /dev/null
+++ b/cccl_upstream/ci/util/artifacts/upload/set_retention_days.sh
@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/artifacts/common.sh
+source "$ci_dir/util/artifacts/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 <artifact_name> <retention_days>
+
+Sets the retention days for an artifact registered for upload.
+
+Example Usage:
+  $0 some_huge_temporary_artifact 1
+  $0 some_small_useful_output 7
+  $0 some_long_term_artifact 30
+EOF
+)
+readonly usage
+
+if [[ "$#" -lt 2 ]]; then
+  echo "Error: Missing arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+artifact_name="$1"
+retention_days="$2"
+
+# Find the artifact entry and update its retention days
+jq --arg name "$artifact_name" --argjson retention_days "$retention_days" \
+  'map(if .name == $name then .retention_days = $retention_days else . end)' \
+  "$ARTIFACT_UPLOAD_REGISTERY" > "$ARTIFACT_UPLOAD_REGISTERY.tmp" && \
+  mv "$ARTIFACT_UPLOAD_REGISTERY.tmp" "$ARTIFACT_UPLOAD_REGISTERY"
diff --git a/cccl_upstream/ci/util/artifacts/upload_packed.sh b/cccl_upstream/ci/util/artifacts/upload_packed.sh
new file mode 100755
index 00000000..4fba3a17
--- /dev/null
+++ b/cccl_upstream/ci/util/artifacts/upload_packed.sh
@@ -0,0 +1,54 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/artifacts/common.sh
+source "$ci_dir/util/artifacts/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 <name> <regex> [<regex> ...]
+
+Create a compressed artifact, suitable for large, temporary files such as build products or test binaries
+that need to be quickly uploaded and downloaded between CI jobs. The artifact will exist of a
+zip file containing an <artifact_name>.tar.zst archive, packed with the parallel zstd.
+
+Regexes are passed to the $(command -v find) command's -regex option in the current directory.
+'./' is prepended to all regexes for convenience.
+The artifact will contain all matching files relative to the current directory.
+
+If no regexes are provided, the artifact will be created from a file in the current directory.
+The file must have the same name as the artifact.
+
+Example Usage:
+
+  Create an artifact of the given file in the current directory using the filename as the artifact name:
+
+    $0 some_resource.log
+
+  Copy all files that match the regexes to a staging directory, and upload a artifact that zips this directory.
+
+    $0 job-\$JOB_ID-products \
+       'build\.ninja$' \
+       '.*rules\.ninja$' \
+       'CMakeCache\.txt$' \
+       '.*VerifyGlobs\.cmake$' \
+       '.*CTestTestfile\.cmake$' \
+       'bin/.*' \
+       'lib/.*'
+EOF
+)
+readonly usage
+
+if [[ "$#" -lt 2 ]]; then
+  echo "Error: Invalid number of arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+readonly artifact_name="$1"
+shift
+
+"$ci_dir/util/artifacts/stage.sh" "$artifact_name" "$@" > /dev/null
+"$ci_dir/util/artifacts/upload_stage_packed.sh" "$artifact_name"
diff --git a/cccl_upstream/ci/util/artifacts/upload_stage.sh b/cccl_upstream/ci/util/artifacts/upload_stage.sh
new file mode 100755
index 00000000..ec778efc
--- /dev/null
+++ b/cccl_upstream/ci/util/artifacts/upload_stage.sh
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/artifacts/common.sh
+source "$ci_dir/util/artifacts/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 <name>
+
+Same as 'ci/util/artifacts/upload_packed.sh', but assumes that the stage has already been created using
+'ci/util/artifacts/stage.sh' and 'unstage.sh'. Performs the packing and registration steps only.
+EOF
+)
+readonly usage
+
+if [[ "$#" -ne 1 ]]; then
+  echo "Error: Invalid number of arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+readonly artifact_name="$1"
+readonly artifact_dir="$ARTIFACT_UPLOAD_STAGE/${artifact_name}/${artifact_name}"
+
+start=$SECONDS
+"$ci_dir/util/artifacts/upload/build.sh" "$artifact_name"
+"$ci_dir/util/artifacts/upload/register.sh" "$artifact_name" "$artifact_dir"
+echo "Artifact '$artifact_name' built in $((SECONDS - start)) seconds."
diff --git a/cccl_upstream/ci/util/artifacts/upload_stage_packed.sh b/cccl_upstream/ci/util/artifacts/upload_stage_packed.sh
new file mode 100755
index 00000000..d365e2bd
--- /dev/null
+++ b/cccl_upstream/ci/util/artifacts/upload_stage_packed.sh
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/artifacts/common.sh
+source "$ci_dir/util/artifacts/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 <name>
+
+Same as 'ci/util/artifacts/upload_packed.sh', but assumes that the stage has already been created using
+'ci/util/artifacts/stage.sh' and 'unstage.sh'. Performs the packing and registration steps only.
+EOF
+)
+readonly usage
+
+if [[ "$#" -ne 1 ]]; then
+  echo "Error: Invalid number of arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+readonly artifact_name="$1"
+# shellcheck disable=SC2154
+readonly artifact_archive="$ARTIFACT_UPLOAD_STAGE/${artifact_name}/${artifact_name}.tar.zst"
+
+start=$SECONDS
+"$ci_dir/util/artifacts/upload/pack.sh" "$artifact_name"
+"$ci_dir/util/artifacts/upload/register.sh" "$artifact_name" "$artifact_archive"
+# Already compressed while packing:
+"$ci_dir/util/artifacts/upload/set_compression_level.sh" "$artifact_name" 0 > /dev/null
+echo "Artifact '$artifact_name' packed in $((SECONDS - start)) seconds."
diff --git a/cccl_upstream/ci/util/build_and_test_targets.sh b/cccl_upstream/ci/util/build_and_test_targets.sh
new file mode 100755
index 00000000..7714b8e8
--- /dev/null
+++ b/cccl_upstream/ci/util/build_and_test_targets.sh
@@ -0,0 +1,165 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# This must be run from the cccl repo root, but the script may be relocated by git_bisect.sh.
+# Check that the current directory looks like the repo root:
+if [[ ! -f "./cccl-version.json" ]]; then
+  echo "This script must be run from the cccl repo root."
+  exit 1
+fi
+
+usage() {
+  cat <<USAGE
+Usage: $0 [--preset NAME | --configure-override CMD] [options]
+
+Options:
+  -h, --help                Show this help and exit
+  --preset NAME             CMake preset
+  --cmake-options STR       Extra options passed to CMake preset configure (optional)
+  --configure-override CMD  Command to run for configuration instead of cmake preset
+                            If set, --preset and --cmake-options will be ignored
+  --build-targets STR       Space separated ninja build targets (optional)
+                            If omitted, no targets will be built -- explicitly specify 'all' if needed.
+  --ctest-targets STR       Space separated CTest -R regex patterns (optional)
+                            If omitted, no tests will be run -- explicitly specify '.' to run all.
+  --lit-precompile-tests STR  Space-separated libcudacxx lit test paths to precompile without execution (optional)
+                              e.g. 'cuda/utility/basic_any.pass.cpp'
+  --lit-tests STR            Space-separated libcudacxx lit test paths to execute (optional)
+                              e.g. 'cuda/utility/basic_any.pass.cpp'
+  --custom-test-cmd CMD     Custom command run after build and tests (optional)
+USAGE
+}
+
+start_timestamp=${SECONDS}
+
+function elapsed_time {
+  local duration=$(( SECONDS - start_timestamp ))
+  local minutes=$(( duration / 60 ))
+  local seconds=$(( duration % 60 ))
+  printf "%dm%02ds" "$minutes" "$seconds"
+}
+
+PRESET=""
+BUILD_TARGETS=()
+CTEST_TARGETS=()
+LIT_PRECOMPILE_TESTS=()
+LIT_TESTS=()
+CMAKE_OPTIONS=()
+CONFIGURE_OVERRIDE=""
+CUSTOM_TEST_CMD=""
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    -h|--help) usage; exit 0 ;;
+    --preset)        PRESET="${2:-}"; shift 2 ;;
+    --build-targets) declare -a BUILD_TARGETS="(${2:-})"; shift 2 ;;
+    --ctest-targets) declare -a CTEST_TARGETS="(${2:-})"; shift 2 ;;
+    --lit-precompile-tests) declare -a LIT_PRECOMPILE_TESTS="(${2:-})"; shift 2 ;;
+    --lit-tests)     declare -a LIT_TESTS="(${2:-})"; shift 2 ;;
+    --cmake-options) declare -a CMAKE_OPTIONS="(${2:-})"; shift 2 ;;
+    --configure-override) CONFIGURE_OVERRIDE="${2:-}"; shift 2 ;;
+    --custom-test-cmd) CUSTOM_TEST_CMD="${2:-}"; shift 2 ;;
+    *) echo "Unknown argument: $1" >&2; usage; exit 2 ;;
+  esac
+done
+
+if [[ -z "${PRESET}" && -z "${CONFIGURE_OVERRIDE}" ]]; then
+  echo "::error:: --preset or --configure-override is required" >&2
+  usage
+  exit 2
+fi
+
+if [[ -n "${CONFIGURE_OVERRIDE}" ]]; then
+  if [[ -n "${PRESET}" ]]; then
+    echo "::warning:: --preset ignored due to --configure-override" >&2
+  fi
+  if [[ "${#CMAKE_OPTIONS[@]}" -gt 0 ]]; then
+    echo "::warning:: --cmake-options ignored due to --configure-override" >&2
+  fi
+fi
+
+echo "::group::⚙️ Testing $(git log --oneline | head -n1)"
+
+# Configure and parse the build directory from CMake output
+BUILD_DIR=""
+cmlog_file="$(mktemp /tmp/cmake-config-XXXXXX.log)"
+if [[ -n "${CONFIGURE_OVERRIDE}" ]]; then
+  if ! (set -x; eval "${CONFIGURE_OVERRIDE}") 2>&1 | tee "${cmlog_file}"; then
+    echo "::endgroup::"
+    echo -e "🔴📝 Configuration override failed ($(elapsed_time)):\n\t${CONFIGURE_OVERRIDE}"
+    exit 1
+  fi
+else
+  if ! (set -x; cmake --preset "${PRESET}" "${CMAKE_OPTIONS[@]}") 2>&1 | tee "${cmlog_file}"; then
+    echo "::endgroup::"
+    echo "🔴📝 CMake configure failed for preset ${PRESET} ($(elapsed_time))"
+    exit 1
+  fi
+fi
+BUILD_DIR=$(awk -F': ' '/-- Build files have been written to:/ {print $2}' "${cmlog_file}" | tail -n1)
+if [[ -z "${BUILD_DIR}" ]]; then
+  echo "::endgroup::"
+  echo "🔴‼️ Unable to determine build directory ($(elapsed_time))"
+  exit 1
+fi
+
+if [[ "${#BUILD_TARGETS[@]}" -gt 0 ]]; then
+  if ! (set -x; ninja -C "${BUILD_DIR}" "${BUILD_TARGETS[@]}"); then
+    echo "::endgroup::"
+    echo "🔴🛠️ Ninja build failed for targets ($(elapsed_time)): ${BUILD_TARGETS[*]@Q}"
+    exit 1
+  fi
+fi
+
+if [[ "${#CTEST_TARGETS[@]}" -gt 0 ]]; then
+  for t in "${CTEST_TARGETS[@]}"; do
+    if ! (set -x; ctest --test-dir "${BUILD_DIR}" -R "$t" -V --output-on-failure); then
+      echo "::endgroup::"
+      echo "🔴🔎 CTest failed for target $t ($(elapsed_time))"
+      exit 1
+    fi
+  done
+fi
+
+if [[ "${#LIT_PRECOMPILE_TESTS[@]}" -gt 0 || "${#LIT_TESTS[@]}" -gt 0 ]]; then
+  lit_site_cfg="${BUILD_DIR}/libcudacxx/test/libcudacxx/lit.site.cfg"
+  if [[ ! -f "${lit_site_cfg}" ]]; then
+    echo "::endgroup::"
+    echo "🔴🧪 LIT site config not found ($(elapsed_time)): ${lit_site_cfg}"
+    exit 1
+  fi
+fi
+
+if [[ "${#LIT_PRECOMPILE_TESTS[@]}" -gt 0 ]]; then
+  for t in "${LIT_PRECOMPILE_TESTS[@]}"; do
+    t_path="libcudacxx/test/libcudacxx/${t}"
+    if ! (set -x; LIBCUDACXX_SITE_CONFIG="${lit_site_cfg}" lit -v "-Dexecutor=NoopExecutor()" "${t_path}"); then
+      echo "::endgroup::"
+      echo "🔴🧪 LIT precompile failed ($(elapsed_time)): ${t}"
+      exit 1
+    fi
+  done
+fi
+
+if [[ "${#LIT_TESTS[@]}" -gt 0 ]]; then
+  for t in "${LIT_TESTS[@]}"; do
+    t_path="libcudacxx/test/libcudacxx/${t}"
+    if ! (set -x; LIBCUDACXX_SITE_CONFIG="${lit_site_cfg}" lit -v "${t_path}"); then
+      echo "::endgroup::"
+      echo "🔴🧪 LIT test failed ($(elapsed_time)): ${t}"
+      exit 1
+    fi
+  done
+fi
+
+if [[ -n "${CUSTOM_TEST_CMD}" ]]; then
+  if ! (set -x; eval "${CUSTOM_TEST_CMD}"); then
+    echo "::endgroup::"
+    echo "🔴🧪 Custom test command failed ($(elapsed_time)): ${CUSTOM_TEST_CMD}"
+    exit 1
+  fi
+fi
+
+echo "::endgroup::"
+echo "🟢✅ Passed ($(elapsed_time))"
+exit 0
diff --git a/cccl_upstream/ci/util/create_mock_job_env.sh b/cccl_upstream/ci/util/create_mock_job_env.sh
new file mode 100755
index 00000000..067966a7
--- /dev/null
+++ b/cccl_upstream/ci/util/create_mock_job_env.sh
@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+usage=$(cat <<EOF
+Usage: $0 <run id> <job_id>
+
+Allows the scripts in ci/util/workflow and ci/util/artifacts to run as though they are running in a CI environment.
+
+Create a mock job environment for testing purposes.
+
+The run id can be found in the workflow run URL, and the job_id can be found at the start of the Run Command job step.
+
+A new shell is spawned with the remote environment of a specific job from a specific workflow run.
+
+Environment variables are configured to mimic the CI environment.
+
+Caches and previously downloaded artifacts in /tmp are deleted to ensure a clean state.
+    !! Note that this does affect the caller's filesystem:
+       /tmp/workflow
+       /tmp/<artifact stages, archives, registry>
+       and similar caches will be deleted **from the caller's filesystem**.
+
+       This is usually fine, but be might overwrite files in-use by other mock environments.
+EOF
+)
+readonly usage
+
+if [[ "$#" -ne 2 ]]; then
+  echo "Error: Invalid number of arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+if [[ -n "${GITHUB_ACTIONS:-}" ]]; then
+  echo "$0: Detected another GITHUB_ACTIONS environment." >&2
+  echo "unset GITHUB_ACTIONS if this is intentional." >&2
+  exit 1
+fi
+
+if [[ -z "${DEVCONTAINER_NAME:-}" ]]; then
+  echo "This script must be run inside a devcontainer." >&2
+  exit 1
+else
+  echo "Running in devcontainer: $DEVCONTAINER_NAME"
+fi
+
+ci_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")/../" && pwd)
+
+export GITHUB_ACTIONS=true
+export GITHUB_RUN_ID="$1"
+export JOB_ID="$2"
+
+(
+  # shellcheck source=ci/util/workflow/common.sh
+  source "$ci_dir/util/workflow/common.sh"
+  # shellcheck source=ci/util/artifacts/common.sh
+  source "$ci_dir/util/artifacts/common.sh"
+
+  rm -rf "$WORKFLOW_DIR"
+  rm -rf "$ARTIFACT_ARCHIVES"
+  rm -rf "$ARTIFACT_UPLOAD_STAGE"
+  rm -rf "$ARTIFACT_UPLOAD_REGISTERY"
+)
+
+# Configure shell prompt:
+export PS0=""
+export PS1="<Mock Job: $GITHUB_RUN_ID $JOB_ID> [\u@\h \W]$ "
+export PROMPT_COMMAND=""
+
+
+echo "Starting new shell for emulating Job $JOB_ID in Run $GITHUB_RUN_ID".
+echo ""
+
+bash --norc --noprofile -i || :
+
+echo
+echo "Exiting mock job environment."
diff --git a/cccl_upstream/ci/util/extract_switches.sh b/cccl_upstream/ci/util/extract_switches.sh
new file mode 100755
index 00000000..b8217534
--- /dev/null
+++ b/cccl_upstream/ci/util/extract_switches.sh
@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+
+# Similar to getopt, but only extracts recognized switches and leaves all other arguments in place.
+#
+# Example Usage:
+#   new_args=$(extract_switches.sh -cpu-only -gpu-only -- "$@")
+#   declare -a new_args="(${new_args})"
+#   set -- "${new_args[@]}"
+#   while true; do
+#     case "$1" in
+#     -cpu-only) CPU_ONLY=true; shift;;
+#     -gpu-only) GPU_ONLY=true; shift;;
+#     --) shift; break;;
+#     *) echo "Unknown argument: $1"; exit 1;;
+#     esac
+#   done
+#
+# This leaves all unrecognized arguments in $@ for later parsing.
+
+# Parse switches
+switches=()
+for arg in "$@"; do
+  case "$arg" in
+  --help | -h)
+    cat <<"EOF" | cut -c 5-
+    Usage: extract_switches.sh <switch> [<switch> ...] -- <argv>
+
+    Sorts any recognized switches in argv to the front and returns the result.
+    Unrecognized switches are left in place.
+
+    Example Usage:
+      new_args="$(extract_switches.sh -cpu-only -gpu-only -- "$@")"
+      declare -a new_args="(${new_args})"
+      set -- "${new_args[@]}"
+      while true; do
+        case "$1" in
+        -cpu-only) CPU_ONLY=true; shift;;
+        -gpu-only) GPU_ONLY=true; shift;;
+        --) shift; break;;
+        *) echo "Unknown argument: $1"; exit 1;;
+        esac
+      done
+EOF
+    exit
+    ;;
+  --)
+    shift
+    break
+    ;;
+  *)
+    switches+=("$arg")
+    shift
+    ;;
+  esac
+done
+
+found_switches=()
+other_args=()
+for arg in "$@"; do
+  for switch in "${switches[@]}"; do
+    if [[ "$arg" = "$switch" ]]; then
+      found_switches+=("\"$arg\"")
+      continue 2
+    fi
+  done
+  other_args+=("\"$arg\"")
+done
+
+echo "${found_switches[*]} -- ${other_args[*]}"
diff --git a/cccl_upstream/ci/util/git_bisect.sh b/cccl_upstream/ci/util/git_bisect.sh
new file mode 100755
index 00000000..53120639
--- /dev/null
+++ b/cccl_upstream/ci/util/git_bisect.sh
@@ -0,0 +1,323 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$ci_dir/.."
+
+usage() {
+  cat <<USAGE
+Usage: $0 [--preset NAME | --configure-override CMD] [options]
+
+Generic Options:
+
+  -h, --help             Show this help and exit
+
+Bisection Options:
+
+  --good-ref STR         Good ref/sha/tag/branch. Defaults to latest release tag.
+                         Accepts '-Nd' (e.g., '-14d') to mean 'origin/main as of N days ago'.
+  --bad-ref STR          Bad ref/sha/tag/branch. Defaults to origin/main.
+                         Accepts '-Nd' (e.g., '-14d') to mean 'origin/main as of N days ago'.
+  --summary-file PATH    Markdown summary output path (optional)
+                         No summary file will be generated if this is omitted.
+Build / Test Options:
+
+  --preset NAME             CMake preset
+  --cmake-options STR       Extra options passed to CMake preset configure (optional)
+  --configure-override CMD  Command to run for configuration instead of cmake preset
+                            If set, --preset and --cmake-options will be ignored
+  --build-targets STR       Space separated ninja build targets (optional)
+                            If omitted, no targets will be built -- explicitly specify 'all' if needed.
+  --ctest-targets STR       Space separated CTest -R regex patterns (optional)
+                            If omitted, no tests will be run -- explicitly specify '.' to run all.
+  --lit-precompile-tests STR  Space-separated libcudacxx lit test paths to precompile without execution (optional)
+                              e.g. 'cuda/utility/basic_any.pass.cpp'
+  --lit-tests STR            Space-separated libcudacxx lit test paths to execute (optional)
+                              e.g. 'cuda/utility/basic_any.pass.cpp'
+  --custom-test-cmd CMD     Custom command run after build and tests (optional)
+  --repeat N               Re-run the build/test for passing commits N times (default: 1)
+USAGE
+}
+
+start_timestamp=${SECONDS}
+
+function elapsed_time {
+  local duration=$(( SECONDS - start_timestamp ))
+  local minutes=$(( duration / 60 ))
+  local seconds=$(( duration % 60 ))
+  printf "%dm%02ds" "$minutes" "$seconds"
+}
+
+GOOD_REF=""
+BAD_REF=""
+PRESET=""
+BUILD_TARGETS=""
+CTEST_TARGETS=""
+LIT_PRECOMPILE_TESTS=""
+LIT_TESTS=""
+SUMMARY_FILE=""
+CMAKE_OPTIONS=""
+CONFIGURE_OVERRIDE=""
+CUSTOM_TEST_CMD=""
+REPEAT=1
+
+# Basic arg parser (keep simple, no extras)
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    -h|--help) usage; exit 0 ;;
+    --preset)        PRESET="${2:-}"; shift 2 ;;
+    --good-ref)      GOOD_REF="${2:-}"; shift 2 ;;
+    --bad-ref)       BAD_REF="${2:-}"; shift 2 ;;
+    --build-targets) BUILD_TARGETS="${2:-}"; shift 2 ;;
+    --ctest-targets) CTEST_TARGETS="${2:-}"; shift 2 ;;
+    --lit-precompile-tests) LIT_PRECOMPILE_TESTS="${2:-}"; shift 2 ;;
+    --lit-tests)     LIT_TESTS="${2:-}"; shift 2 ;;
+    --cmake-options) CMAKE_OPTIONS="${2:-}"; shift 2 ;;
+    --configure-override) CONFIGURE_OVERRIDE="${2:-}"; shift 2 ;;
+    --summary-file)  SUMMARY_FILE="${2:-}"; shift 2 ;;
+    --custom-test-cmd) CUSTOM_TEST_CMD="${2:-}"; shift 2 ;;
+    --repeat)        REPEAT="${2:-}"; shift 2 ;;
+    *) echo "Unknown argument: $1" >&2; usage; exit 2 ;;
+  esac
+done
+
+if [[ -z "${PRESET}" && -z "${CONFIGURE_OVERRIDE}" ]]; then
+  echo "::error:: --preset or --configure-override is required" >&2
+  usage
+  exit 2
+fi
+
+if [[ -n "${CONFIGURE_OVERRIDE}" ]]; then
+  if [[ -n "${PRESET}" ]]; then
+    echo "::warning:: --preset ignored due to --configure-override" >&2
+  fi
+  if [[ -n "${CMAKE_OPTIONS}" ]]; then
+    echo "::warning:: --cmake-options ignored due to --configure-override" >&2
+  fi
+fi
+
+# Ensure the checkout has complete history and tags:
+git fetch --unshallow > /dev/null 2>&1 || :
+git fetch --tags > /dev/null 2>&1 || :
+
+# Resolve good and bad refs
+good_ref="${GOOD_REF}"
+bad_ref="${BAD_REF}"
+
+# Helper to resolve '-Nd' (N days ago on origin/main) to a SHA
+_resolve_days_ago() {
+  local spec="$1"
+  local base_branch="origin/main"
+  local n="${spec#-}"
+  n="${n%d}"
+  if [[ -z "$n" || ! "$n" =~ ^[0-9]+$ ]]; then
+    return 1
+  fi
+  local when
+  when=$(date -u -d "$n days ago" '+%Y-%m-%d %H:%M:%S %z')
+  git rev-list -n 1 --before="$when" "$base_branch"
+}
+
+# Resolve good_ref
+if [[ -z "$good_ref" ]]; then
+  good_ref=$(git tag --list 'v*' | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | sort -V | tail -n 1 || :)
+  echo "Good ref defaulted to last release: $good_ref"
+fi
+if [[ "$good_ref" =~ ^-[0-9]+d$ ]]; then
+  good_sha=$(_resolve_days_ago "$good_ref")
+  if [[ -z "$good_sha" ]]; then
+    echo "::error::Unable to resolve good_ref '$good_ref' to a commit on origin/main" >&2
+    exit 1
+  fi
+  echo "Resolved good_ref '$good_ref' to origin/main @ $good_sha"
+else
+  if [[ -z "$good_ref" ]]; then
+    echo "::error::Unable to determine good ref" >&2
+    exit 1
+  fi
+  good_sha=$(git rev-parse "$good_ref")
+fi
+
+# Resolve bad_ref
+if [[ -z "$bad_ref" ]]; then
+  bad_ref="origin/main"
+  echo "Bad ref defaulted to origin/main: $bad_ref"
+fi
+if [[ "$bad_ref" =~ ^-[0-9]+d$ ]]; then
+  bad_sha=$(_resolve_days_ago "$bad_ref")
+  if [[ -z "$bad_sha" ]]; then
+    echo "::error::Unable to resolve bad_ref '$bad_ref' to a commit on origin/main" >&2
+    exit 1
+  fi
+  echo "Resolved bad_ref '$bad_ref' to origin/main @ $bad_sha"
+else
+  bad_sha=$(git rev-parse "$bad_ref")
+fi
+
+# Copy the build-and-test runner to a temp file so it remains available as HEAD changes:
+tmp_runner="$(mktemp /tmp/build-and-test-XXXXXX.sh)"
+cp "${ci_dir}/util/build_and_test_targets.sh" "${tmp_runner}"
+chmod +x "${tmp_runner}"
+
+# If --repeat > 1, wrap the runner to repeat successful runs to detect flakiness.
+bisect_runner="${tmp_runner}"
+if [[ "${REPEAT}" =~ ^[0-9]+$ ]] && [[ "${REPEAT}" -gt 1 ]]; then
+  tmp_repeat="$(mktemp /tmp/build-and-test-repeat-XXXXXX.sh)"
+  cat > "${tmp_repeat}" <<EOF
+#!/usr/bin/env bash
+set -euo pipefail
+
+attempts=${REPEAT}
+for ((i=1; i<=attempts; ++i)); do
+  echo "::group::✅ build_and_test attempt \${i}/${REPEAT}"
+  "${tmp_runner}" "\$@" || {
+    echo "Attempt \${i} failed; marking commit as bad."
+    exit 1
+  }
+  echo "::endgroup::"
+done
+exit 0
+EOF
+  chmod +x "${tmp_repeat}"
+  bisect_runner="${tmp_repeat}"
+  echo "Repeating successful runs ${REPEAT} times to check for flakiness."
+fi
+
+# Writer that always prints to stdout and tees to file if provided
+write_summary() {
+  if [[ -n "${SUMMARY_FILE}" ]]; then
+    tee -a "${SUMMARY_FILE}"
+  else
+    cat
+  fi
+}
+
+echo "Starting bisect with:"
+echo "  BAD_SHA:  $bad_sha"
+echo "  GOOD_SHA: $good_sha"
+
+echo "::group::⚙️ Starting git bisect"
+(set -x; git bisect start "$bad_sha" "$good_sha")
+echo "::endgroup::"
+
+bisect_log="$(mktemp /tmp/git-bisect-log-XXXXXX.log)"
+bisect_output="$(mktemp /tmp/git-bisect-output-XXXXXX.log)"
+(
+  set -x
+  git bisect run "${bisect_runner}" \
+    --preset "${PRESET}" \
+    --build-targets "${BUILD_TARGETS}" \
+    --ctest-targets "${CTEST_TARGETS}" \
+    --lit-precompile-tests "${LIT_PRECOMPILE_TESTS}" \
+    --lit-tests "${LIT_TESTS}" \
+    --cmake-options "${CMAKE_OPTIONS}" \
+    --configure-override "${CONFIGURE_OVERRIDE}" \
+    --custom-test-cmd "${CUSTOM_TEST_CMD}" \
+    | tee "${bisect_output}" || :
+  git bisect log | tee "${bisect_log}" || :
+  git bisect reset || :
+)
+
+if grep -q " is the first bad commit" "${bisect_output}"; then
+  bad_commit=$(awk '/ is the first bad commit/ {print $1}' "${bisect_output}")
+  echo -e "\e[1;32mFound bad commit in $(elapsed_time): $bad_commit\e[0m"
+  found=true
+else
+  echo -e "\e[1;31mNo bad commit found ($(elapsed_time)).\e[0m"
+  found=false
+fi
+
+function print_repro {
+  echo "### ♻️ Reproduction Steps"
+  echo
+  echo '```bash'
+  if [[ -n "${LAUNCH_ARGS:-}" ]]; then
+    echo "  .devcontainer/launch.sh \\"
+    echo "    ${LAUNCH_ARGS} \\"
+    echo "    -- \\"
+  fi
+  echo "    ./ci/util/build_and_test_targets.sh \\"
+
+  declare -a build_test_vars=(
+    PRESET
+    CMAKE_OPTIONS
+    CONFIGURE_OVERRIDE
+    BUILD_TARGETS
+    CTEST_TARGETS
+    LIT_PRECOMPILE_TESTS
+    LIT_TESTS
+    CUSTOM_TEST_CMD
+  )
+
+  # only print the above vars if they're non-empty.
+  first=true
+  for var in "${build_test_vars[@]}"; do
+    if [[ -n "${!var:-}" ]]; then
+      flag="--${var,,}"
+      flag=${flag//_/-}  # replace _ with -
+      if ! $first; then
+        echo " \\" # Trailing "\" to escape newlines
+      fi
+      echo -n "       ${flag} \"${!var}\""
+      first=false
+    fi
+  done
+  echo # Final newline for last argument
+  echo '```'
+}
+
+if [[ "${found}" == "true" ]]; then
+  commit_info=$(git log "$bad_commit" -1 --pretty=format:'%h %s')
+  pr_ref=$(echo "$commit_info" | grep -oE '#[0-9]+' | head -n1 || :)
+  (
+    echo "## 🔎 Bisect Result"
+    echo
+    echo "- Culprit Commit: $commit_info"
+    if [[ -n "$pr_ref" ]]; then
+      pr_num=${pr_ref#\#}
+      echo "- Culprit PR: https://github.com/NVIDIA/cccl/pull/$pr_num"
+    fi
+    echo "- Commit SHA: $bad_commit"
+    echo "- Commit URL: https://github.com/NVIDIA/cccl/commit/${bad_commit}"
+    if [[ -n "${GHA_LOG_URL:-}" ]]; then
+      echo "- Bisection Logs: [GHA Job](${GHA_LOG_URL})"
+    fi
+    if [[ -n "${STEP_SUMMARY_URL:-}" ]]; then
+      echo "- Bisection Summary: [GHA Report](${STEP_SUMMARY_URL})"
+    fi
+    echo
+    print_repro
+    echo
+    echo "### ℹ️ Commit Details"
+    echo
+    echo '```'
+    git show "$bad_commit" --stat
+    echo '```'
+    echo
+    echo "### 🪵 Bisect Log"
+    echo
+    echo '```'
+    cat "${bisect_log}"
+    echo '```'
+  ) | write_summary
+else
+  (
+    echo "## ‼️ Bisect Failed"
+    echo
+    echo "git bisect did not resolve to a single commit."
+    echo
+    if [[ -n "${GHA_LOG_URL:-}" ]]; then
+      echo "- Bisection Logs: [GHA Job](${GHA_LOG_URL})"
+      echo "- Bisection Summary: [GHA Report](${STEP_SUMMARY_URL})"
+    fi
+    echo
+    print_repro
+    echo
+    echo "### 🪵 Bisect Log"
+    echo
+    echo '```'
+    cat "${bisect_log}"
+    echo '```'
+  ) | write_summary
+  exit 1
+fi
diff --git a/cccl_upstream/ci/util/manifest.sh b/cccl_upstream/ci/util/manifest.sh
new file mode 100755
index 00000000..79008aea
--- /dev/null
+++ b/cccl_upstream/ci/util/manifest.sh
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+
+# Do not enable `eo pipefail`
+# read relies on a quirk of unterminated/unquoted heredoc to capture input.
+# TODO: rewrite using printf if we can keep nice formatting.
+set -u
+
+path="$(realpath "$1")"
+outfile="$2"
+version="$3"
+platform="$4"
+
+read -r -d '' manifest << EOF
+{
+"schema-version": 1,
+"componentName": "cccl",
+"componentVersion": "$version",
+"platform": "$platform"
+}
+EOF
+
+read -r -d '' prog << 'EOF'
+split(.,"\n")
+  | select(length > 2)
+  | [{(.[]) : {"type": "header"}}]
+  | add
+  | $manifest + { "files" : . }
+EOF
+
+find "$path" -wholename '*include/*' -type f -printf '%P\n' \
+  | jq -s --raw-input --argjson manifest "$manifest" "$prog" > "${outfile}"
diff --git a/cccl_upstream/ci/util/memmon.sh b/cccl_upstream/ci/util/memmon.sh
new file mode 100755
index 00000000..63593af5
--- /dev/null
+++ b/cccl_upstream/ci/util/memmon.sh
@@ -0,0 +1,310 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+pid_file="/tmp/.memmon.pid"
+
+log_threshold="2"
+print_threshold="5"
+poll_interval="5"
+log_file="$PWD/memmon.log"
+mode=""
+
+usage() {
+  cat <<'USAGE'
+Monitors running processes for high memory usage, logging and reporting peaks above specified thresholds.
+
+Usage: memmon.sh (--start | --stop | --monitor | --help)
+        [--log-threshold <GB>]   # Write to log file if process exceeds this memory (default 2 GB)
+        [--print-threshold <GB>] # Print to stdout if process exceeds this memory (default 5 GB)
+        [--poll <seconds>]       # Poll interval for checking processes (default 5 seconds)
+        [--log-file <file>]      # Log file path (default ./memmon.log)
+
+Modes:
+
+  --start    Start monitoring in the background (writes pid to /tmp/.memmon.pid)
+  --stop     Stop monitoring (kills pid in /tmp/.memmon.pid)
+  --monitor  Run monitoring in the foreground (for testing/debugging)
+
+Example Session:
+
+  memmon.sh --start                 # Start monitoring
+  launch_memory_intensive_processes # Do work
+  memmon.sh --stop                  # Stop monitoring and write log
+  cat memmon.log                    # View log
+USAGE
+}
+
+error() {
+  echo "memmon: $*" >&2
+  exit 1
+}
+
+error_usage() {
+  echo "memmon: $*" >&2
+  usage
+  exit 1
+}
+
+ensure_absolute_log() {
+  case "$log_file" in
+    /*) return ;;
+    *)
+      local dir
+      dir="$(cd "$(dirname "$log_file")" && pwd)"
+      local base
+      base="$(basename "$log_file")"
+      log_file="$dir/$base"
+      ;;
+  esac
+}
+
+to_kib() {
+  awk -v gib="$1" 'BEGIN {printf "%.0f", gib * 1024 * 1024}'
+}
+
+format_gib() {
+  awk -v rss="$1" 'BEGIN {printf "%.3f", rss/1024/1024}'
+}
+
+format_threshold() {
+  awk -v val="$1" 'BEGIN {printf "%.3f", val + 0}'
+}
+
+declare -A MEMMON_MAX_RSS
+declare -A MEMMON_CMD
+declare -A MEMMON_TARGET
+
+get_cmdline() {
+  local pid="$1"
+  local cmdline_file="/proc/$pid/cmdline"
+  local raw
+
+  # Attempt to read from /proc first for accuracy
+  if [[ -r "$cmdline_file" ]]; then
+    raw="$(tr '\0' ' ' <"$cmdline_file" 2>/dev/null)"
+    # Clean up whitespace
+    raw="${raw//$'\n'/ }"
+    raw="${raw//$'\r'/ }"
+    raw="${raw//$'\t'/ }"
+    while [[ "$raw" == *' ' ]]; do
+      raw="${raw% }"
+    done
+    if [[ -n "$raw" ]]; then
+      printf '%s' "$raw"
+      return 0
+    fi
+  fi
+  # Fallback to ps if /proc is unavailable
+  raw="$(ps -wwp "$pid" -o command= 2>/dev/null | head -n1)"
+  # Clean up whitespace
+  raw="${raw//$'\n'/ }"
+  raw="${raw//$'\r'/ }"
+  raw="${raw//$'\t'/ }"
+  if [[ -n "$raw" ]]; then
+    printf '%s' "$raw"
+    return 0
+  fi
+  printf '[command unavailable]'
+}
+
+extract_target() {
+  local cmd="$1"
+  # Try to locate the name of the cmake target:
+  if [[ "$cmd" =~ CMakeFiles/([^[:space:]]*)\.dir ]]; then
+    printf '%s' "${BASH_REMATCH[1]}"
+  else
+    printf '-'
+  fi
+}
+
+start_memmon() {
+  # Check if already running
+  if [[ -f "$pid_file" ]]; then
+    local existing_pid
+    existing_pid="$(<"$pid_file")"
+    if kill -0 "$existing_pid" 2>/dev/null; then
+      error "already running (pid $existing_pid)"
+    fi
+    rm -f "$pid_file"
+  fi
+
+  ensure_absolute_log
+
+  # Start monitoring in the background
+  "$0" --monitor \
+       --log-threshold "$log_threshold" \
+       --print-threshold "$print_threshold" \
+       --poll "$poll_interval" \
+       --log-file "$log_file" &
+  local child_pid=$!
+  echo "$child_pid" >"$pid_file"
+  echo "memmon started (pid $child_pid, log-threshold ${log_threshold}GB, print-threshold ${print_threshold}GB, log $log_file)"
+}
+
+stop_memmon() {
+  if [[ ! -f "$pid_file" ]]; then
+    error "not running"
+  fi
+  local running_pid
+  running_pid="$(<"$pid_file")"
+  if ! kill -0 "$running_pid" 2>/dev/null; then
+    rm -f "$pid_file"
+    error "not running"
+  fi
+
+  # Attempt graceful shutdown
+  kill "$running_pid" 2>/dev/null || true
+
+  for _ in {1..20}; do
+    if ! kill -0 "$running_pid" 2>/dev/null; then
+      break
+    fi
+    sleep 0.25
+  done
+
+  # Force kill if still running
+  if kill -0 "$running_pid" 2>/dev/null; then
+    kill -9 "$running_pid" 2>/dev/null || true
+  fi
+
+  # Wait for pid file to be removed
+  for _ in {1..20}; do
+    [[ ! -f "$pid_file" ]] && break
+    sleep 0.25
+  done
+  [[ -f "$pid_file" ]] && rm -f "$pid_file"
+  echo "memmon stopped"
+}
+
+monitor_mem() {
+  MEMMON_MAX_RSS=()
+  MEMMON_CMD=()
+  MEMMON_TARGET=()
+
+  ensure_absolute_log
+
+  local log_threshold_kib
+  log_threshold_kib="$(to_kib "$log_threshold")"
+  local print_threshold_kib
+  print_threshold_kib="$(to_kib "$print_threshold")"
+
+  local running=true
+
+  cleanup() {
+    trap - INT TERM EXIT
+    mkdir -p "$(dirname "$log_file")"
+    {
+      printf "peak-mem | PID | target | command-line\n"
+      if [[ ${#MEMMON_MAX_RSS[@]} -eq 0 ]]; then
+        printf "No processes exceeded %s GB\n" "$(format_threshold "$log_threshold")"
+      else
+        local tmp
+        tmp="$(mktemp)"
+        for pid in "${!MEMMON_MAX_RSS[@]}"; do
+          printf "%s\t%s\t%s\t%s\n" "${MEMMON_MAX_RSS[$pid]}" "$pid" "${MEMMON_TARGET[$pid]}" "${MEMMON_CMD[$pid]}" >>"$tmp"
+        done
+        sort -nr -k1,1 "$tmp" | while IFS=$'\t' read -r peak pid target cmd; do
+          local mem_gib
+          mem_gib="$(format_gib "$peak")"
+          printf "%s GB | %s | %s | %s\n" "$mem_gib" "$pid" "$target" "$cmd"
+        done
+        rm -f "$tmp"
+      fi
+    } >"$log_file"
+    echo "memmon log written to $log_file"
+    rm -f "$pid_file"
+  }
+
+  trap 'running=false' INT TERM
+  trap cleanup EXIT
+
+  while $running; do
+    while read -r pid rss; do
+      [[ -z "$pid" || -z "$rss" ]] && continue
+      [[ "$pid" =~ ^[0-9]+$ ]] || continue
+      [[ "$rss" =~ ^[0-9]+$ ]] || continue
+      if (( rss >= log_threshold_kib )); then
+        local current=${MEMMON_MAX_RSS[$pid]:-0}
+        if (( rss > current )); then
+          MEMMON_MAX_RSS[$pid]=$rss
+          MEMMON_CMD[$pid]=$(get_cmdline "$pid")
+          MEMMON_TARGET[$pid]=$(extract_target "${MEMMON_CMD[$pid]}")
+          if (( rss >= print_threshold_kib )); then
+            local mem_gib
+            mem_gib="$(format_gib "$rss")"
+            printf 'memmon: %s GB | %s | %s | %s\n' "$mem_gib" "$pid" "${MEMMON_TARGET[$pid]}" "${MEMMON_CMD[$pid]}"
+          fi
+        fi
+      fi
+    done < <(ps -eo pid=,rss=)
+
+    if ! sleep "$poll_interval"; then
+      break
+    fi
+  done
+}
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --log-threshold)
+      [[ $# -lt 2 ]] && error "--log-threshold requires a value"
+      log_threshold="$2"
+      shift 2
+      ;;
+    --print-threshold)
+      [[ $# -lt 2 ]] && error "--print-threshold requires a value"
+      print_threshold="$2"
+      shift 2
+      ;;
+    --log-file)
+      [[ $# -lt 2 ]] && error "--log-file requires a value"
+      log_file="$2"
+      shift 2
+      ;;
+    --poll)
+      [[ $# -lt 2 ]] && error "--poll requires a value"
+      poll_interval="$2"
+      shift 2
+      ;;
+    --start)
+      [[ -n "$mode" ]] && error "Specify only one of --start or --stop"
+      mode="start"
+      shift
+      ;;
+    --stop)
+      [[ -n "$mode" ]] && error "Specify only one of --start or --stop"
+      mode="stop"
+      shift
+      ;;
+    --monitor)
+      mode="monitor"
+      shift
+      ;;
+    --help|-h)
+      usage
+      exit 0
+      ;;
+    *)
+      error_usage "Unknown option: $1"
+      ;;
+  esac
+done
+
+if [[ -z "$mode" ]]; then
+  error_usage "Must specify one of --start or --stop or --monitor"
+fi
+
+case "$mode" in
+  start)
+    start_memmon
+    ;;
+  stop)
+    stop_memmon
+    ;;
+  monitor)
+    monitor_mem
+    ;;
+  *)
+    error_usage "Unhandled mode: $mode"
+    ;;
+esac
diff --git a/cccl_upstream/ci/util/pre-commit/check_cub_test_macros.py b/cccl_upstream/ci/util/pre-commit/check_cub_test_macros.py
new file mode 100755
index 00000000..b3a4cb33
--- /dev/null
+++ b/cccl_upstream/ci/util/pre-commit/check_cub_test_macros.py
@@ -0,0 +1,139 @@
+#!/usr/bin/env python3
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import re
+import sys
+
+RAW_TEST_MACROS = (
+    "C2H_TEST",
+    "C2H_TEST_LIST",
+    "C2H_TEST_WITH_FIXTURE",
+    "C2H_TEST_LIST_WITH_FIXTURE",
+    "TEST_CASE",
+    "TEST_CASE_METHOD",
+    "SCENARIO",
+    "SCENARIO_METHOD",
+    "TEMPLATE_TEST_CASE",
+    "TEMPLATE_TEST_CASE_SIG",
+    "TEMPLATE_TEST_CASE_METHOD",
+    "TEMPLATE_TEST_CASE_METHOD_SIG",
+    "TEMPLATE_PRODUCT_TEST_CASE",
+    "TEMPLATE_PRODUCT_TEST_CASE_SIG",
+    "TEMPLATE_PRODUCT_TEST_CASE_METHOD",
+    "TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG",
+    "TEMPLATE_LIST_TEST_CASE",
+    "TEMPLATE_LIST_TEST_CASE_METHOD",
+)
+
+RAW_TEST_MACRO_RE = re.compile(
+    r"^[ \t]*(?P<macro>"
+    + "|".join(
+        re.escape(macro) for macro in sorted(RAW_TEST_MACROS, key=len, reverse=True)
+    )
+    + r")[ \t]*\(",
+    re.MULTILINE,
+)
+
+
+def remove_comments(source: str) -> str:
+    """Blank C++ comments while preserving line and column positions."""
+    result = list(source)
+    index = 0
+
+    while index < len(source):
+        if source.startswith("//", index):
+            end = source.find("\n", index)
+            if end == -1:
+                end = len(source)
+            for comment_index in range(index, end):
+                result[comment_index] = " "
+            index = end
+            continue
+
+        if source.startswith("/*", index):
+            end = source.find("*/", index + 2)
+            if end == -1:
+                end = len(source) - 2
+            for comment_index in range(index, min(end + 2, len(source))):
+                if source[comment_index] not in "\r\n":
+                    result[comment_index] = " "
+            index = end + 2
+            continue
+
+        if source[index] in {'"', "'"}:
+            quote = source[index]
+            index += 1
+            while index < len(source):
+                if source[index] == "\\":
+                    index += 2
+                    continue
+                if source[index] == quote:
+                    index += 1
+                    break
+                index += 1
+            continue
+
+        index += 1
+
+    return "".join(result)
+
+
+def self_test() -> bool:
+    fixtures = [
+        ('// C2H_TEST("x")', False),
+        ('/*\nTEST_CASE("x")\n*/', False),
+        ('const char* value = "TEST_CASE(";', False),
+        ('CUB_TEST("x", "[y]", CUB_SMALL)', False),
+        ('CUB_TEST_CASE("x", "[y]", CUB_LARGE)', False),
+        ('CUB_TEST_LIST("x", "[y]", CUB_SMALL, types)', False),
+    ]
+    fixtures.extend((f'{macro}("x")', True) for macro in RAW_TEST_MACROS)
+
+    for source, expected in fixtures:
+        found = bool(RAW_TEST_MACRO_RE.search(remove_comments(source)))
+        if found != expected:
+            expected_result = "match" if expected else "no match"
+            actual_result = "match" if found else "no match"
+            print(
+                "internal error: test-registration checker self-test failed for "
+                f"{source!r}: expected {expected_result}, found {actual_result}. "
+                "This is a problem with the checker, not the files being committed.",
+                file=sys.stderr,
+            )
+            return False
+
+    return True
+
+
+def check_file(filename: str) -> bool:
+    with open(filename, encoding="utf-8", errors="surrogateescape") as source_file:
+        source = source_file.read()
+
+    source_without_comments = remove_comments(source)
+    found_error = False
+    for match in RAW_TEST_MACRO_RE.finditer(source_without_comments):
+        line = source.count("\n", 0, match.start()) + 1
+        column = match.start("macro") - source.rfind("\n", 0, match.start("macro"))
+        print(
+            f"{filename}:{line}:{column}: {match.group('macro')} bypasses CUB "
+            "memory classification; use CUB_TEST, CUB_TEST_CASE, or CUB_TEST_LIST."
+        )
+        found_error = True
+
+    return found_error
+
+
+def main() -> int:
+    if not self_test():
+        return 2
+
+    found_error = False
+    for filename in sys.argv[1:]:
+        found_error = check_file(filename) or found_error
+    return int(found_error)
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/cccl_upstream/ci/util/pre-commit/check_shebang.py b/cccl_upstream/ci/util/pre-commit/check_shebang.py
new file mode 100755
index 00000000..a00e4a0f
--- /dev/null
+++ b/cccl_upstream/ci/util/pre-commit/check_shebang.py
@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+import re
+import sys
+
+
+def main() -> int:
+    # We pre-compile this regular expression as a micro-optimization for speed. The
+    # assumption is that the shebang is correct, so everything in the happy path should be
+    # as fast as possible.
+    FIRST_LINE_RE = re.compile(r"#!\s*/usr/bin/env.*")
+    ret = 0
+    for f in sys.argv[1:]:
+        with open(f) as fd:
+            first_line = fd.readline()
+
+        if not first_line.startswith("#!"):
+            # Not a shebang
+            continue
+
+        if FIRST_LINE_RE.match(first_line):
+            # Already correct
+            continue
+
+        ret = 1
+
+        if not (
+            m := re.match(r"#!\s*(?:/bin/(\w+)|/usr/bin/(\w+))\s*(.*)", first_line)
+        ):
+            # Not assert, pre-commit may compile with -O
+            raise AssertionError(f"Failed to match shebang for {first_line}")
+
+        fixed = f"#!/usr/bin/env {m[1] or m[2]}".rstrip()
+        if rest := m[3].strip():
+            fixed += f" {rest}"
+        fixed += "\n"
+
+        with open(f) as fd:
+            # Read the remaining lines, we need them in order to overwrite
+            lines = fd.readlines()
+
+        lines[0] = fixed
+
+        with open(f, "w") as fd:
+            fd.writelines(lines)
+
+    return ret
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/cccl_upstream/ci/util/pre-commit/strip_unprintable.py b/cccl_upstream/ci/util/pre-commit/strip_unprintable.py
new file mode 100755
index 00000000..6b6ec013
--- /dev/null
+++ b/cccl_upstream/ci/util/pre-commit/strip_unprintable.py
@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# strip_unprintable.py - Remove invisible / unprintable characters from text files.
+import re
+import sys
+from argparse import ArgumentParser, Namespace, RawDescriptionHelpFormatter
+
+# Canonical definition of what gets removed. One row per range group:
+#   (regex character-class fragment, human description)
+# The compiled character class and the --help listing are both derived from this
+# table, so adding or removing a range only needs to happen here. TAB (U+0009),
+# LF (U+000A), and CR (U+000D) are deliberately excluded from the C0 range. The
+# fragments use \x/\u escapes so the source file itself stays free of the very
+# characters this script removes.
+RANGES = (
+    (
+        r"\x00-\x08\x0b\x0c\x0e-\x1f\x7f",
+        "C0 controls / DEL (TAB, LF, CR preserved)",
+    ),
+    (r"\x80-\x9f", "C1 controls"),
+    (r"\xa0", "no-break space"),
+    (r"\u200b-\u200f", "zero-width space/joiners, bidi marks"),
+    (r"\u202a-\u202e", "bidi embedding/override"),
+    (r"\u2060-\u2064", "word joiner, invisible operators"),
+    (r"\ufeff", "BOM / zero-width no-break space"),
+)
+
+# Character class assembled from column 1 of the ranges table.
+BAD_RE = re.compile("[" + "".join(frag for frag, _ in RANGES) + "]")
+
+
+def parse_args() -> Namespace:
+    removed = "\n".join(f"  {frag}\t{desc}" for frag, desc in RANGES)
+    parser = ArgumentParser(
+        description=(
+            "Remove invisible / unprintable characters from text files, in place, "
+            "while preserving ordinary whitespace (TAB U+0009, LF U+000A, "
+            "CR U+000D)."
+        ),
+        epilog=f"Removed characters:\n{removed}",
+        formatter_class=RawDescriptionHelpFormatter,
+    )
+    parser.add_argument(
+        "--check",
+        action="store_true",
+        help=(
+            "Report offending files and their line/column locations; make no "
+            "edits. Exits non-zero if any unprintable characters are found."
+        ),
+    )
+    parser.add_argument("files", nargs="+", metavar="FILE")
+    return parser.parse_args()
+
+
+def check(files: list[str]) -> int:
+    ret = 0
+    for f in files:
+        with open(f, encoding="utf-8", errors="surrogateescape") as fd:
+            for lineno, line in enumerate(fd, start=1):
+                for m in BAD_RE.finditer(line):
+                    print(f"{f}:{lineno}:{m.start() + 1}: U+{ord(m.group()):04X}")
+                    ret = 1
+
+    return ret
+
+
+def strip(files: list[str]) -> int:
+    ret = 0
+    for f in files:
+        with open(f, encoding="utf-8", errors="surrogateescape") as fd:
+            original = fd.read()
+        stripped = BAD_RE.sub("", original)
+        if stripped != original:
+            with open(
+                f, "w", encoding="utf-8", errors="surrogateescape", newline=""
+            ) as fd:
+                fd.write(stripped)
+            ret = 1
+
+    return ret
+
+
+def main() -> int:
+    args = parse_args()
+    if args.check:
+        return check(args.files)
+    return strip(args.files)
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/cccl_upstream/ci/util/python/common_arg_parser.sh b/cccl_upstream/ci/util/python/common_arg_parser.sh
new file mode 100644
index 00000000..c7fd41e7
--- /dev/null
+++ b/cccl_upstream/ci/util/python/common_arg_parser.sh
@@ -0,0 +1,58 @@
+# Argument parser for Python CI scripts.
+parse_python_args() {
+    # Initialize variables
+    py_version=""
+    # ctk_mode carries the -ctk-mode value; empty means the default ("pinned").
+    ctk_mode=""
+
+    while [[ $# -gt 0 ]]; do
+        case $1 in
+            -py-version=*)
+                py_version="${1#*=}"
+                shift
+                ;;
+            -py-version)
+                if [[ $# -lt 2 ]]; then
+                    echo "Error: -py-version requires a value" >&2
+                    return 1
+                fi
+                py_version="$2"
+                shift 2
+                ;;
+            -ctk-mode=*)
+                ctk_mode="${1#*=}"
+                # Reject an explicit-but-empty value (e.g. `-ctk-mode=`): a lane
+                # that wants the default omits the flag entirely, so an empty
+                # value signals a malformed generated argument -- fail loudly.
+                if [[ -z "${ctk_mode}" ]]; then
+                    echo "Error: -ctk-mode requires a value" >&2
+                    return 1
+                fi
+                shift
+                ;;
+            -ctk-mode)
+                if [[ $# -lt 2 || -z "$2" ]]; then
+                    echo "Error: -ctk-mode requires a value" >&2
+                    return 1
+                fi
+                ctk_mode="$2"
+                shift 2
+                ;;
+            *)
+                # Unknown argument, ignore
+                shift
+                ;;
+        esac
+    done
+
+    # Export for use by the calling script (py_version and ctk_mode are its inputs).
+    export py_version ctk_mode
+}
+
+require_py_version() {
+    if [[ -z "$py_version" ]]; then
+        echo "Error: -py-version is required" >&2
+        [[ -n "$1" ]] && echo "$1" >&2
+        return 1
+    fi
+}
diff --git a/cccl_upstream/ci/util/retry.sh b/cccl_upstream/ci/util/retry.sh
new file mode 100755
index 00000000..171fd9e3
--- /dev/null
+++ b/cccl_upstream/ci/util/retry.sh
@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+
+if [[ "$#" -lt 3 ]]; then
+    echo "Usage: $0 num_tries sleep_time command [args...]"
+    echo "  num_tries: Number of attempts to run the command"
+    echo "  sleep_time: Time to wait between attempts (in seconds)"
+    echo "  command: The command to run"
+    echo "  args: Arguments to pass to the command"
+    exit 1
+fi
+
+num_tries=$1
+sleep_time=$2
+shift 2
+command=("${*@Q}")
+
+# Loop until the command succeeds or we reach the maximum number of attempts:
+for ((i=1; i<=num_tries; i++)); do
+    echo "Attempt ${i} of ${num_tries}: Running command '${command[*]}'"
+    status=0
+    eval "${command[*]}" || status=$?
+
+    if [[ "$status" -eq 0 ]]; then
+        echo "Command '${command[*]}' succeeded on attempt ${i}."
+        exit 0
+    else
+        echo "Command '${command[*]}' failed with status ${status}. Retrying in ${sleep_time} seconds..."
+        sleep "$sleep_time"
+    fi
+done
+echo "Command '${command[*]}' failed after ${num_tries} attempts."
+exit 1
diff --git a/cccl_upstream/ci/util/version_compare.sh b/cccl_upstream/ci/util/version_compare.sh
new file mode 100755
index 00000000..b60bf6b2
--- /dev/null
+++ b/cccl_upstream/ci/util/version_compare.sh
@@ -0,0 +1,85 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+usage=$(cat <<EOF
+Usage: $0 X.Y[.Z[.W[...]]] <compare_op> A.B[.C[.D[...]]]
+
+Compares two version strings with the specified operator.
+
+compare_ops:
+  lt - less than
+  le - less than or equal to
+  eq - equal to
+  ne - not equal to
+  ge - greater than or equal to
+  gt - greater than
+EOF
+)
+readonly usage
+
+if [[ "$#" -ne 3 ]]; then
+  echo "Error: Invalid arguments: $*" >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+version_a="$1"
+operator="$2"
+version_b="$3"
+
+# Validate operator
+if [[ ! "$operator" =~ ^(lt|le|eq|ne|ge|gt)$ ]]; then
+  echo "Error: Invalid operator '$operator'. Must be one of: lt, le, eq, ne, ge, gt." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+# Validate versions:
+version_regex='^[0-9]+(\.[0-9]+)*$'
+if [[ ! "$version_a" =~ $version_regex ]]; then
+  echo "Error: Invalid version string '$version_a'." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+if [[ ! "$version_b" =~ $version_regex ]]; then
+  echo "Error: Invalid version string '$version_b'." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+# Split versions into arrays
+IFS='.' read -r -a ver_a_parts <<< "$version_a"
+IFS='.' read -r -a ver_b_parts <<< "$version_b"
+max_length=${#ver_a_parts[@]}
+if [[ "${#ver_b_parts[@]}" -gt "$max_length" ]]; then
+  max_length=${#ver_b_parts[@]}
+fi
+
+# Compare each part
+for ((i=0; i<max_length; i++)); do
+  part_a=${ver_a_parts[i]:-0}
+  part_b=${ver_b_parts[i]:-0}
+  if ((part_a < part_b)); then
+    result="lt"
+    break
+  elif ((part_a > part_b)); then
+    result="gt"
+    break
+  else
+    result="eq"
+  fi
+done
+
+# Evaluate the comparison based on the operator
+case "$operator" in
+  lt) [[ "$result" == "lt" ]] ;;
+  le) [[ "$result" == "lt" || "$result" == "eq" ]] ;;
+  eq) [[ "$result" == "eq" ]] ;;
+  ne) [[ "$result" != "eq" ]] ;;
+  ge) [[ "$result" == "gt" || "$result" == "eq" ]] ;;
+  gt) [[ "$result" == "gt" ]] ;;
+  *) echo "Error: Unhandled operator '${operator}'." >&2; exit 1 ;;
+esac
+
+exit $?
diff --git a/cccl_upstream/ci/util/workflow/common.sh b/cccl_upstream/ci/util/workflow/common.sh
new file mode 100755
index 00000000..a5fa4de9
--- /dev/null
+++ b/cccl_upstream/ci/util/workflow/common.sh
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
+  echo "This script must be sourced, not executed directly." >&2
+  exit 1
+fi
+
+if [[ -z "${GITHUB_ACTIONS:-}" ]]; then
+  echo "This script must be run in a GitHub Actions environment." >&2
+  exit 1
+fi
+
+to_posix_path() {
+  local path="$1"
+
+  if [[ "$path" =~ ^([A-Za-z]):([\\/]?.*)$ ]]; then
+    local drive="${BASH_REMATCH[1]}"
+    local rest="${BASH_REMATCH[2]}"
+    rest="${rest//\\/\/}"
+    printf '/%s%s\n' "${drive,,}" "$rest"
+    return
+  fi
+
+  printf '%s\n' "$path"
+}
+
+runner_temp_posix="$(to_posix_path "${RUNNER_TEMP:-/tmp}")"
+
+export WORKFLOW_ARTIFACT="workflow"
+export WORKFLOW_DIR="${runner_temp_posix}/workflow"
+
+mkdir -p "$WORKFLOW_DIR"
diff --git a/cccl_upstream/ci/util/workflow/get_consumers.sh b/cccl_upstream/ci/util/workflow/get_consumers.sh
new file mode 100755
index 00000000..a4db72ff
--- /dev/null
+++ b/cccl_upstream/ci/util/workflow/get_consumers.sh
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/workflow/common.sh
+source "$ci_dir/util/workflow/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 [job_id]
+
+Return a json array of job definitions for all consumers of the specified producer job ID.
+If no job ID is provided, the \$JOB_ID environment variable is used.
+EOF
+)
+readonly usage
+
+if [[ "$#" -gt 1 ]]; then
+  echo "Error: Too many arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+job_id="${1:-${JOB_ID:-}}"
+
+if [[ -z "$job_id" ]]; then
+  echo "Error: No job ID provided and \$JOB_ID is not set." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+"${ci_dir}/util/workflow/initialize.sh"
+
+consumers=$(jq --arg job_id "$job_id" '
+  to_entries[]
+  | select(.value.two_stage)
+  | .value.two_stage[]
+  | select(any(.producers[]; .id == $job_id))
+  | .consumers
+' "$WORKFLOW_DIR/workflow.json")
+
+echo "$consumers"
diff --git a/cccl_upstream/ci/util/workflow/get_job_def.sh b/cccl_upstream/ci/util/workflow/get_job_def.sh
new file mode 100755
index 00000000..37df5311
--- /dev/null
+++ b/cccl_upstream/ci/util/workflow/get_job_def.sh
@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/workflow/common.sh
+source "$ci_dir/util/workflow/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 [job_id]
+
+Prints a json object containing the workflow job definition for the specified job ID.
+If no job ID is provided, the \$JOB_ID environment variable is used.
+If the job ID does not exist in the workflow, an error is raised.
+EOF
+)
+readonly usage
+
+if [[ "$#" -gt 1 ]]; then
+  echo "Error: Too many arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+job_id="${1:-${JOB_ID:-}}"
+
+if [[ -z "$job_id" ]]; then
+  echo "Error: No job ID provided and \$JOB_ID is not set." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+"${ci_dir}/util/workflow/initialize.sh"
+
+job_obj=$(jq --arg job_id "$job_id" '
+  to_entries[]
+  | .value
+  | (
+      (select(has("standalone")) | .standalone[] | select(.id == $job_id)) //
+      (select(has("two_stage")) | .two_stage[] | .producers[] | select(.id == $job_id)) //
+      (select(has("two_stage")) | .two_stage[] | .consumers[] | select(.id == $job_id))
+    )
+' "$WORKFLOW_DIR/workflow.json")
+
+if [[ -z "$job_obj" ]]; then
+  echo "Error: No job definition found for job ID '$job_id'." >&2
+  exit 1
+fi
+
+echo "$job_obj" | jq -r
diff --git a/cccl_upstream/ci/util/workflow/get_job_project.sh b/cccl_upstream/ci/util/workflow/get_job_project.sh
new file mode 100755
index 00000000..ebbfe1ec
--- /dev/null
+++ b/cccl_upstream/ci/util/workflow/get_job_project.sh
@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+readonly ci_dir
+
+usage=$(cat <<EOF
+Usage: $0 [job_id]
+
+Returns the name of the project built by the specified job id.
+If no job ID is provided, the \$JOB_ID environment variable is used.
+If the job ID does not exist in the workflow an error is raised.
+EOF
+)
+readonly usage
+
+if [[ "$#" -gt 1 ]]; then
+  echo "Error: Too many arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+job_id="${1:-${JOB_ID:-}}"
+
+if [[ -z "$job_id" ]]; then
+  echo "Error: No job ID provided and \$JOB_ID is not set." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+job_def=$("${ci_dir}/util/workflow/get_job_def.sh" "$job_id")
+project=$(echo "$job_def" | jq -r '.origin.matrix_job.project')
+echo "$project"
diff --git a/cccl_upstream/ci/util/workflow/get_producer_id.sh b/cccl_upstream/ci/util/workflow/get_producer_id.sh
new file mode 100755
index 00000000..5b6df658
--- /dev/null
+++ b/cccl_upstream/ci/util/workflow/get_producer_id.sh
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/workflow/common.sh
+source "$ci_dir/util/workflow/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 [job_id]
+
+Prints the job ID of the associated producer for the specified consumer job ID.
+If no job ID is provided, the \$JOB_ID environment variable is used.
+If the number of producers for the job is not exactly one, an error is raised.
+EOF
+)
+readonly usage
+
+if [[ "$#" -gt 1 ]]; then
+  echo "Error: Too many arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+job_id="${1:-${JOB_ID:-}}"
+
+if [[ -z "$job_id" ]]; then
+  echo "Error: No job ID provided and \$JOB_ID is not set." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+"${ci_dir}/util/workflow/initialize.sh"
+
+producers=$(jq --arg job_id "$job_id" '
+  to_entries[]
+  | select(.value.two_stage)
+  | .value.two_stage[]
+  | select(any(.consumers[]; .id == $job_id))
+  | .producers
+' "$WORKFLOW_DIR/workflow.json")
+
+producer_count=$(echo "$producers" | jq 'length')
+if [[ "$producer_count" -ne 1 ]]; then
+  echo "Error: Expected exactly one producer for job ID '$job_id', but found ${producer_count:-0}." >&2
+  exit 1
+fi
+
+producer_id=$(echo "$producers" | jq -r '.[0].id')
+if [[ -z "$producer_id" ]]; then
+  echo "Error: No producer ID found for job ID '$job_id'." >&2
+  exit 1
+fi
+echo "$producer_id"
diff --git a/cccl_upstream/ci/util/workflow/get_producers.sh b/cccl_upstream/ci/util/workflow/get_producers.sh
new file mode 100755
index 00000000..ec037421
--- /dev/null
+++ b/cccl_upstream/ci/util/workflow/get_producers.sh
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/workflow/common.sh
+source "$ci_dir/util/workflow/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 [job_id]
+
+Return a json array of job definitions for all producers of the specified consumer job ID.
+If no job ID is provided, the \$JOB_ID environment variable is used.
+EOF
+)
+readonly usage
+
+if [[ "$#" -gt 1 ]]; then
+  echo "Error: Too many arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+job_id="${1:-${JOB_ID:-}}"
+
+if [[ -z "$job_id" ]]; then
+  echo "Error: No job ID provided and \$JOB_ID is not set." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+"${ci_dir}/util/workflow/initialize.sh"
+
+producers=$(jq --arg job_id "$job_id" '
+  to_entries[]
+  | select(.value.two_stage)
+  | .value.two_stage[]
+  | select(any(.consumers[]; .id == $job_id))
+  | .producers
+' "$WORKFLOW_DIR/workflow.json")
+
+echo "$producers"
diff --git a/cccl_upstream/ci/util/workflow/get_stable_job_hash.sh b/cccl_upstream/ci/util/workflow/get_stable_job_hash.sh
new file mode 100755
index 00000000..7753f905
--- /dev/null
+++ b/cccl_upstream/ci/util/workflow/get_stable_job_hash.sh
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+readonly ci_dir
+
+usage=$(cat <<EOF
+Usage: $0 [job_id]
+
+Get a stable hash that identifies the job's toolchain, runner, image,
+name, and launch command, removing origin and per-run ids.
+If no job ID is provided, the \$JOB_ID environment variable is used.
+If the job ID does not exist in the workflow an error is raised.
+EOF
+)
+readonly usage
+
+if [[ "$#" -gt 1 ]]; then
+  echo "Error: Too many arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+job_id="${1:-${JOB_ID:-}}"
+
+if [[ -z "$job_id" ]]; then
+  echo "Error: No job ID provided and \$JOB_ID is not set." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+job_def=$("${ci_dir}/util/workflow/get_job_def.sh" "$job_id" | jq 'del(.id, .origin)')
+job_hash=$(echo "$job_def" | sha256sum | awk '{print $1}')
+echo "$job_hash"
diff --git a/cccl_upstream/ci/util/workflow/get_wheel_artifact_name.sh b/cccl_upstream/ci/util/workflow/get_wheel_artifact_name.sh
new file mode 100755
index 00000000..b1058c1e
--- /dev/null
+++ b/cccl_upstream/ci/util/workflow/get_wheel_artifact_name.sh
@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+readonly ci_dir
+
+usage=$(cat <<EOF
+Usage: $0 [job_id]
+
+Get the name of the wheel file that matches the specified job ID's configuration.
+If no job ID is provided, the \$JOB_ID environment variable is used.
+If the job ID does not exist in the workflow, or is not a python job, an error is raised.
+EOF
+)
+readonly usage
+
+if [[ "$#" -gt 1 ]]; then
+  echo "Error: Too many arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+job_id="${1:-${JOB_ID:-}}"
+
+if [[ -z "$job_id" ]]; then
+  echo "Error: No job ID provided and \$JOB_ID is not set." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+job_def=$("${ci_dir}/util/workflow/get_job_def.sh" "$job_id")
+
+py_version=$(echo "$job_def" | jq -r '.origin.matrix_job.py_version')
+host=$(echo "$job_def" | jq -r '.origin.matrix_job.cxx_family')
+if [[ "$host" == "MSVC" ]]; then
+  os="windows"
+else
+  os="linux"
+fi
+arch=$(echo "$job_def" | jq -r '.origin.matrix_job.cpu')
+project=$(echo "$job_def" | jq -r '.origin.matrix_job.project')
+
+for tag in "$py_version" "$os" "$arch"; do
+  if [[ -z "$tag" ]]; then
+    echo "Error: Missing required field in job definition for job ID '$job_id'." >&2
+    echo "$usage" >&2
+    echo >&2
+    "Job definition: $job_def" >&2
+    exit 1
+  fi
+done
+
+# v1 and v2 Python build jobs both run in the same workflow, so their wheel
+# artifacts must have distinct names or the second upload clobbers the first
+# and downstream test jobs grab the wrong wheel. v1 keeps its historical name
+# (the test-cpu-import workflow hardcodes it); v2 gets a "-v2" suffix.
+suffix=""
+if [[ "$project" == "python_v2" ]]; then
+  suffix="-v2"
+elif [[ "$project" == "python_tsan" ]]; then
+  # ThreadSanitizer-instrumented wheel (free-threaded TSan nightly lane). Must
+  # be distinct so its build doesn't clobber the normal wheel and the TSan test
+  # job doesn't grab an uninstrumented one.
+  suffix="-tsan"
+fi
+
+echo "wheel-cccl${suffix}-$os-$arch-py$py_version"
diff --git a/cccl_upstream/ci/util/workflow/has_consumers.sh b/cccl_upstream/ci/util/workflow/has_consumers.sh
new file mode 100755
index 00000000..7202ac82
--- /dev/null
+++ b/cccl_upstream/ci/util/workflow/has_consumers.sh
@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/workflow/common.sh
+source "$ci_dir/util/workflow/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 [job_id]
+
+Exits successfully if the specified job ID has consumers, otherwise exits with an error.
+If no job ID is provided, the \$JOB_ID environment variable is used.
+EOF
+)
+readonly usage
+
+if [[ "$#" -gt 1 ]]; then
+  echo "Error: Too many arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+job_id="${1:-${JOB_ID:-}}"
+
+if [[ -z "$job_id" ]]; then
+  echo "Error: No job ID provided and \$JOB_ID is not set." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+"${ci_dir}/util/workflow/initialize.sh"
+
+matching_producer=$(jq --arg job_id "$job_id" '
+  to_entries[]
+  | select(.value.two_stage)
+  | .value.two_stage[]
+  | .producers[]
+  | select(.id == $job_id)
+' "$WORKFLOW_DIR/workflow.json")
+
+if [[ -n "$matching_producer" ]]; then
+  exit 0
+else
+  exit 1
+fi
diff --git a/cccl_upstream/ci/util/workflow/has_producers.sh b/cccl_upstream/ci/util/workflow/has_producers.sh
new file mode 100755
index 00000000..72ee3143
--- /dev/null
+++ b/cccl_upstream/ci/util/workflow/has_producers.sh
@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/workflow/common.sh
+source "$ci_dir/util/workflow/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0 [job_id]
+
+Exits successfully if the specified job ID has producers, otherwise exits with an error.
+If no job ID is provided, the \$JOB_ID environment variable is used.
+EOF
+)
+readonly usage
+
+if [[ "$#" -gt 1 ]]; then
+  echo "Error: Too many arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+job_id="${1:-${JOB_ID:-}}"
+
+if [[ -z "$job_id" ]]; then
+  echo "Error: No job ID provided and \$JOB_ID is not set." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+"${ci_dir}/util/workflow/initialize.sh"
+
+matching_consumer=$(jq --arg job_id "$job_id" '
+  to_entries[]
+  | select(.value.two_stage)
+  | .value.two_stage[]
+  | .consumers[]
+  | select(.id == $job_id)
+' "$WORKFLOW_DIR/workflow.json")
+
+if [[ -n "$matching_consumer" ]]; then
+  exit 0
+else
+  exit 1
+fi
diff --git a/cccl_upstream/ci/util/workflow/initialize.sh b/cccl_upstream/ci/util/workflow/initialize.sh
new file mode 100755
index 00000000..f4f13bbb
--- /dev/null
+++ b/cccl_upstream/ci/util/workflow/initialize.sh
@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+ci_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../" && pwd)"
+readonly ci_dir
+# shellcheck source=ci/util/workflow/common.sh
+source "$ci_dir/util/workflow/common.sh"
+
+usage=$(cat <<EOF
+Usage: $0
+
+Downloads the workflow artifact and unpacks it to \$WORKFLOW_DIR, but only if it doesn't already exist.
+EOF
+)
+readonly usage
+
+if [[ "$#" -ne 0 ]]; then
+  echo "Error: This script does not take any arguments." >&2
+  echo "$usage" >&2
+  exit 1
+fi
+
+if [[ ! -f "$WORKFLOW_DIR/workflow.json" ]]; then
+  "$ci_dir/util/artifacts/download/fetch.sh" "$WORKFLOW_ARTIFACT" "$WORKFLOW_DIR" > /dev/null
+fi
diff --git a/cccl_upstream/ci/verify_codegen_libcudacxx.sh b/cccl_upstream/ci/verify_codegen_libcudacxx.sh
new file mode 100755
index 00000000..da7b999b
--- /dev/null
+++ b/cccl_upstream/ci/verify_codegen_libcudacxx.sh
@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+set -eo pipefail
+
+# Ensure the script is being executed in its containing directory
+cd "$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )";
+
+# TBD: verbose? any extra args?
+
+source ./pretty_printing.sh
+
+pushd .. > /dev/null
+GROUP_NAME="🛠️  CMake Configure Libcudacxx Codegen"
+run_command "$GROUP_NAME" cmake --preset libcudacxx-codegen
+status=$?
+popd > /dev/null
+
+pushd .. > /dev/null
+GROUP_NAME="🏗️  Build Libcudacxx Codegen"
+run_command "$GROUP_NAME" cmake --build --preset libcudacxx-codegen
+
+status=$?
+popd > /dev/null
+
+pushd .. > /dev/null
+GROUP_NAME="🚀  Test Libcudacxx Codegen"
+run_command "$GROUP_NAME" ctest --preset libcudacxx-codegen
+status=$?
+popd > /dev/null
diff --git a/cccl_upstream/ci/windows/build_cccl_c_parallel.ps1 b/cccl_upstream/ci/windows/build_cccl_c_parallel.ps1
new file mode 100644
index 00000000..19a83a55
--- /dev/null
+++ b/cccl_upstream/ci/windows/build_cccl_c_parallel.ps1
@@ -0,0 +1,28 @@
+Param(
+    [Parameter(Mandatory = $false)]
+    [Alias("arch")]
+    [string]$CUDA_ARCH = "",
+    [Parameter(Mandatory = $false)]
+    [Alias("cmake-options")]
+    [string]$CMAKE_OPTIONS = ""
+)
+
+$ErrorActionPreference = "Stop"
+
+$CURRENT_PATH = Split-Path $pwd -leaf
+If($CURRENT_PATH -ne "ci") {
+    Write-Host "Moving to ci folder"
+    pushd "$PSScriptRoot/.."
+}
+
+Remove-Module -Name build_common -ErrorAction SilentlyContinue
+Import-Module $PSScriptRoot/build_common.psm1 -ArgumentList @(20, $CUDA_ARCH, $CMAKE_OPTIONS)
+
+$PRESET = "cccl-c-parallel"
+$LOCAL_CMAKE_OPTIONS = ""
+
+configure_and_build_preset "CCCL C Parallel" $PRESET $LOCAL_CMAKE_OPTIONS
+
+If($CURRENT_PATH -ne "ci") {
+    popd
+}
diff --git a/cccl_upstream/ci/windows/build_cccl_c_parallel_v2.ps1 b/cccl_upstream/ci/windows/build_cccl_c_parallel_v2.ps1
new file mode 100644
index 00000000..529a37b1
--- /dev/null
+++ b/cccl_upstream/ci/windows/build_cccl_c_parallel_v2.ps1
@@ -0,0 +1,28 @@
+Param(
+    [Parameter(Mandatory = $false)]
+    [Alias("arch")]
+    [string]$CUDA_ARCH = "",
+    [Parameter(Mandatory = $false)]
+    [Alias("cmake-options")]
+    [string]$CMAKE_OPTIONS = ""
+)
+
+$ErrorActionPreference = "Stop"
+
+$CURRENT_PATH = Split-Path $pwd -leaf
+If($CURRENT_PATH -ne "ci") {
+    Write-Host "Moving to ci folder"
+    pushd "$PSScriptRoot/.."
+}
+
+Remove-Module -Name build_common -ErrorAction SilentlyContinue
+Import-Module $PSScriptRoot/build_common.psm1 -ArgumentList @(20, $CUDA_ARCH, $CMAKE_OPTIONS)
+
+$PRESET = "cccl-c-parallel-v2"
+$LOCAL_CMAKE_OPTIONS = ""
+
+configure_and_build_preset "CCCL C Parallel" $PRESET $LOCAL_CMAKE_OPTIONS
+
+If($CURRENT_PATH -ne "ci") {
+    popd
+}
diff --git a/cccl_upstream/ci/windows/build_common.psm1 b/cccl_upstream/ci/windows/build_common.psm1
new file mode 100644
index 00000000..7d6b9554
--- /dev/null
+++ b/cccl_upstream/ci/windows/build_common.psm1
@@ -0,0 +1,233 @@
+Param(
+    [Parameter(Mandatory = $false)]
+    [Alias("std")]
+    [ValidateNotNullOrEmpty()]
+    [ValidateSet(17, 20)]
+    [int]$CXX_STANDARD = 17,
+    [Parameter(Mandatory = $false)]
+    [Alias("arch")]
+    [string]$CUDA_ARCH = "",
+    [Parameter(Mandatory = $false)]
+    [Alias("cmake-options")]
+    [string]$CMAKE_OPTIONS = ""
+)
+
+$ErrorActionPreference = "Stop"
+
+# We need the full path to cl because otherwise cmake will replace CMAKE_CXX_COMPILER with the full path
+# and keep CMAKE_CUDA_HOST_COMPILER at "cl" which breaks our cmake script
+$script:HOST_COMPILER  = (Get-Command "cl").source -replace '\\','/'
+$script:PARALLEL_LEVEL = $env:NUMBER_OF_PROCESSORS
+
+Write-Host "=== Docker Container Resource Info ==="
+Write-Host "Number of Processors: $script:PARALLEL_LEVEL"
+Get-WmiObject Win32_OperatingSystem | ForEach-Object {
+    Write-Host ("Memory: total={0:N1} GB, free={1:N1} GB" -f ($_.TotalVisibleMemorySize / 1MB), ($_.FreePhysicalMemory / 1MB))
+}
+Write-Host "======================================"
+
+# Extract the CL version for export to build scripts:
+$script:CL_VERSION_STRING = & cl.exe /?
+if ($script:CL_VERSION_STRING -match "Version (\d+\.\d+)\.\d+") {
+    $CL_VERSION = [version]$matches[1]
+    Write-Host "Detected cl.exe version: $CL_VERSION"
+}
+
+$script:GLOBAL_CMAKE_OPTIONS = $CMAKE_OPTIONS
+if ($CUDA_ARCH) {
+    $script:GLOBAL_CMAKE_OPTIONS += ' "-DCMAKE_CUDA_ARCHITECTURES={0}"' -f $CUDA_ARCH
+}
+
+# Default to pedantic mode in CI (GitHub Actions)
+if ($env:GITHUB_ACTIONS) {
+    $script:GLOBAL_CMAKE_OPTIONS += ' "-DCCCL_ENABLE_WERROR=ON" "-DCCCL_ENABLE_PRAGMA_SYSTEM_HEADER=OFF"'
+} else {
+    $script:GLOBAL_CMAKE_OPTIONS += ' "-DCCCL_ENABLE_WERROR=OFF" "-DCCCL_ENABLE_PRAGMA_SYSTEM_HEADER=ON"'
+}
+
+if (-not $env:CCCL_BUILD_INFIX) {
+    $env:CCCL_BUILD_INFIX = ""
+}
+
+# Presets will be configured in this directory:
+$BUILD_DIR = "../build/$env:CCCL_BUILD_INFIX"
+
+If(!(test-path -PathType container "../build")) {
+    New-Item -ItemType Directory -Path "../build"
+}
+
+# The most recent build will always be symlinked to cccl/build/latest
+New-Item -ItemType Directory -Path "$BUILD_DIR" -Force
+
+# Convert to an absolute path:
+$BUILD_DIR = (Get-Item -Path "$BUILD_DIR").FullName
+
+# Prepare environment for CMake:
+$env:CMAKE_BUILD_PARALLEL_LEVEL = $PARALLEL_LEVEL
+$env:CTEST_PARALLEL_LEVEL = 1
+$env:CUDAHOSTCXX = $script:HOST_COMPILER
+$env:CXX = $script:HOST_COMPILER
+
+Write-Host "========================================"
+Write-Host "Begin build"
+Write-Host "pwd=$pwd"
+Write-Host "BUILD_DIR=$BUILD_DIR"
+Write-Host "CXX_STANDARD=$CXX_STANDARD"
+Write-Host "CXX=$env:CXX"
+Write-Host "CUDACXX=$env:CUDACXX"
+Write-Host "CUDAHOSTCXX=$env:CUDAHOSTCXX"
+Write-Host "TBB_ROOT=$env:TBB_ROOT"
+Write-Host "NVCC_VERSION=$NVCC_VERSION"
+Write-Host "CMAKE_BUILD_PARALLEL_LEVEL=$env:CMAKE_BUILD_PARALLEL_LEVEL"
+Write-Host "CTEST_PARALLEL_LEVEL=$env:CTEST_PARALLEL_LEVEL"
+Write-Host "CCCL_BUILD_INFIX=$env:CCCL_BUILD_INFIX"
+Write-Host "GLOBAL_CMAKE_OPTIONS=$script:GLOBAL_CMAKE_OPTIONS"
+Write-Host "Current commit is:"
+Write-Host "$(git log -1 --format=short)"
+Write-Host "========================================"
+
+cmake --version
+ctest --version
+
+function configure_preset {
+    Param(
+        [Parameter(Mandatory = $true)]
+        [ValidateNotNullOrEmpty()]
+        [string]$BUILD_NAME,
+        [Parameter(Mandatory = $true)]
+        [ValidateNotNullOrEmpty()]
+        [string]$PRESET,
+        [Parameter(Mandatory = $false)]
+        [string]$LOCAL_CMAKE_OPTIONS = ""
+    )
+
+    $step = "$BUILD_NAME (configure)"
+
+    # CMake must be invoked in the same directory as the presets file:
+    pushd ".."
+
+    # Echo and execute command to stdout:
+    $configure_command = "cmake --preset $PRESET --log-level VERBOSE"
+    if ($LOCAL_CMAKE_OPTIONS) {
+        $configure_command += " $LOCAL_CMAKE_OPTIONS"
+    }
+    if ($script:GLOBAL_CMAKE_OPTIONS) {
+        $configure_command += " $script:GLOBAL_CMAKE_OPTIONS"
+    }
+
+    Write-Host $configure_command
+    Invoke-Expression $configure_command
+    $test_result = $LastExitCode
+
+    If ($test_result -ne 0) {
+        throw "$step Failed"
+    }
+
+    popd
+    Write-Host "$step complete."
+}
+
+function build_preset {
+    Param(
+        [Parameter(Mandatory = $true)]
+        [ValidateNotNullOrEmpty()]
+        [string]$BUILD_NAME,
+        [Parameter(Mandatory = $true)]
+        [ValidateNotNullOrEmpty()]
+        [string]$PRESET
+    )
+
+    $step = "$BUILD_NAME (build)"
+
+    # CMake must be invoked in the same directory as the presets file:
+    pushd ".."
+
+    sccache -z >$null
+
+    cmake --build --preset $PRESET -v
+    $test_result = $LastExitCode
+
+    $preset_dir = "${BUILD_DIR}/${PRESET}"
+    $sccache_json = "${preset_dir}/sccache_stats.json"
+
+    sccache --show-adv-stats
+    sccache --show-adv-stats --stats-format=json > "${sccache_json}"
+
+    echo "$step complete"
+
+    If ($test_result -ne 0) {
+         throw "$step Failed"
+    }
+
+    popd
+}
+
+function test_preset {
+    Param(
+        [Parameter(Mandatory = $true)]
+        [ValidateNotNullOrEmpty()]
+        [string]$BUILD_NAME,
+        [Parameter(Mandatory = $true)]
+        [ValidateNotNullOrEmpty()]
+        [string]$PRESET
+    )
+
+    $step = "$BUILD_NAME (test)"
+
+    # CTest must be invoked in the same directory as the presets file:
+    pushd ".."
+
+    sccache -z >$null
+
+    ctest --preset $PRESET
+    $test_result = $LastExitCode
+
+    sccache --show-adv-stats
+
+    echo "$step complete"
+
+    If ($test_result -ne 0) {
+         throw "$step Failed"
+    }
+
+    popd
+}
+
+function configure_and_build_preset {
+    Param(
+        [Parameter(Mandatory = $true)]
+        [ValidateNotNullOrEmpty()]
+        [string]$BUILD_NAME,
+        [Parameter(Mandatory = $true)]
+        [ValidateNotNullOrEmpty()]
+        [string]$PRESET,
+        [Parameter(Mandatory = $false)]
+        [string]$LOCAL_CMAKE_OPTIONS = ""
+    )
+
+    configure_preset $BUILD_NAME $PRESET $LOCAL_CMAKE_OPTIONS
+    build_preset $BUILD_NAME $PRESET
+}
+
+function Invoke-Checked {
+    <#
+    .SYNOPSIS
+        Runs a script block and throws if the last native command in it exits
+        non-zero. $ErrorActionPreference = "Stop" does not make native commands
+        (python/pip/pytest/...) throw, so their $LASTEXITCODE must be checked
+        explicitly; this wraps that boilerplate into one call.
+    .EXAMPLE
+        Invoke-Checked { & $python -m pip install pytest } "pip install failed"
+    #>
+    param(
+        [Parameter(Mandatory, Position = 0)][scriptblock]$ScriptBlock,
+        [Parameter(Position = 1)][string]$ErrorMessage = "Native command failed"
+    )
+    & $ScriptBlock
+    if ($LASTEXITCODE -ne 0) {
+        throw "$ErrorMessage (exit code $LASTEXITCODE)"
+    }
+}
+
+Export-ModuleMember -Function configure_preset, build_preset, test_preset, configure_and_build_preset, Invoke-Checked
+Export-ModuleMember -Variable BUILD_DIR, CL_VERSION
diff --git a/cccl_upstream/ci/windows/build_common_python.psm1 b/cccl_upstream/ci/windows/build_common_python.psm1
new file mode 100644
index 00000000..ba778236
--- /dev/null
+++ b/cccl_upstream/ci/windows/build_common_python.psm1
@@ -0,0 +1,246 @@
+function Get-Python {
+    <#
+    .SYNOPSIS
+        Returns the path of the Python interpreter satisfying the supplied
+        version, installing it via uv if necessary.
+    .PARAMETER Version
+        A string in the form 'M.m' (e.g., '3.10', '3.13') or a free-threaded
+        version such as '3.14t'.
+    #>
+    [CmdletBinding()]
+    param(
+        [Parameter(Mandatory, Position = 0)]
+        [ValidatePattern('^\d+\.\d+t?$')]
+        [string]$Version
+    )
+
+    # Install uv if not present. uv downloads pre-built CPython binaries --
+    # no compilation, no build dependencies, no pyenv-win required.
+    if (-not (Get-Command uv -ErrorAction SilentlyContinue)) {
+        Write-Host "Installing uv..."
+        Invoke-RestMethod https://astral.sh/uv/install.ps1 | Invoke-Expression
+        # uv installs to $HOME\.local\bin on Windows
+        $uvBin = Join-Path $HOME '.local\bin'
+        $Env:PATH = $uvBin + ";" + $Env:PATH
+    }
+
+    Write-Host "Creating Python $Version venv via uv..."
+    $venvDir = Join-Path $HOME '.cccl-venv'
+    & uv venv --seed --python $Version $venvDir
+    if ($LASTEXITCODE -ne 0) {
+        throw [System.InvalidOperationException]::new(
+            "Failed to create Python $Version venv via uv."
+        )
+    }
+
+    $exe = Join-Path $venvDir 'Scripts\python.exe'
+    if (-not (Test-Path $exe)) {
+        throw [System.InvalidOperationException]::new(
+            "Could not find python.exe in venv at $exe"
+        )
+    }
+
+    Write-Host "Python $Version at: $exe"
+
+    # Add venv Scripts dir to PATH so bare `python` and `pip` work.
+    $scriptsDir = Join-Path $venvDir 'Scripts'
+    $Env:PATH = $scriptsDir + ";" + $Env:PATH
+
+    return $exe
+}
+
+function Get-RepoRoot {
+    return (Resolve-Path "$PSScriptRoot/../..")
+}
+
+function Get-CudaMajor {
+    <#
+    .SYNOPSIS
+        Gets the CUDA major version for this container instance (e.g. '12' or
+        '13').  Defaults to '13' if no match can be found.
+    #>
+    if ($env:CUDA_PATH) {
+        $nvcc = Join-Path $env:CUDA_PATH "bin/nvcc.exe"
+        if (Test-Path $nvcc) {
+            $out = & $nvcc --version 2>&1
+            $text = ($out -join "`n")
+            if ($text -match 'release\s+(\d+)\.') { return $Matches[1] }
+        }
+        # Fallback: parse major from CUDA_PATH like ...\v13.0 or ...\CUDA\13
+        $pathMatch = [regex]::Match($env:CUDA_PATH, 'v?(\d+)(?:\.\d+)?')
+        if ($pathMatch.Success) { return $pathMatch.Groups[1].Value }
+    }
+    return '13'
+}
+
+function Get-CudaVersion {
+    <#
+    .SYNOPSIS
+        Gets the CUDA major.minor version for this container instance (e.g.
+        '12.9' or '13.0').  Defaults to '13.0' if no match can be found.
+    #>
+    if ($env:CUDA_PATH) {
+        $nvcc = Join-Path $env:CUDA_PATH "bin/nvcc.exe"
+        if (Test-Path $nvcc) {
+            $out = & $nvcc --version 2>&1
+            $text = ($out -join "`n")
+            if ($text -match 'release\s+(\d+\.\d+)') { return $Matches[1] }
+        }
+        # Fallback: parse major.minor from CUDA_PATH like ...\v13.0
+        $pathMatch = [regex]::Match($env:CUDA_PATH, 'v?(\d+\.\d+)')
+        if ($pathMatch.Success) { return $pathMatch.Groups[1].Value }
+    }
+    return '13.0'
+}
+
+function Get-CtkTestMode {
+    <#
+    .SYNOPSIS
+        Validates and normalizes a CTK test mode passed as -Mode (forwarded from
+        the -ctk-mode arg): 'pinned' (default; empty means pinned), 'latest', or
+        'sysctk'. Throws on any other value (fail loud on a typo'd mode). Returns
+        the lowercased mode.
+    #>
+    param([string]$Mode = "")
+    if ([string]::IsNullOrEmpty($Mode)) { return "pinned" }
+    $Mode = $Mode.ToLowerInvariant()
+    if (-not ($Mode -in @("pinned", "latest", "sysctk"))) {
+        throw "Invalid ctk mode '$Mode' (expected pinned|latest|sysctk)"
+    }
+    return $Mode
+}
+
+function Set-CtkPin {
+    <#
+    .SYNOPSIS
+        Configures cuda-toolkit pinning for this lane per the -Mode arg (see
+        Get-CtkTestMode): 'pinned' (default) pins cuda-toolkit to the container's
+        CTK major.minor via PIP_CONSTRAINT; 'latest' and 'sysctk' leave it
+        unpinned ('sysctk' installs no cuda-toolkit wheel at all -- the
+        system-provided toolkit is used).
+    #>
+    param([string]$Mode = "")
+    if ((Get-CtkTestMode $Mode) -eq "pinned") {
+        $cudaVersion = Get-CudaVersion
+        $env:PIP_CONSTRAINT = Join-Path ([System.IO.Path]::GetTempPath()) "ctk-constraint.txt"
+        "cuda-toolkit==$cudaVersion.*" | Out-File -FilePath $env:PIP_CONSTRAINT -Encoding ascii
+    } else {
+        # latest / sysctk: no pin. Clear any inherited constraint so it cannot
+        # affect the resolve.
+        Remove-Item Env:\PIP_CONSTRAINT -ErrorAction SilentlyContinue
+    }
+}
+
+function Get-CtkExtraFlavor {
+    <#
+    .SYNOPSIS
+        Returns the pip-extra toolkit "flavor" for the given -Mode: 'sysctk' when
+        the mode is sysctk (rely on the system-provided CUDA toolkit) or 'cu'
+        otherwise (pip-installed toolkit). Combine with the CUDA major, e.g.
+        "minimal-$(Get-CtkExtraFlavor $CtkMode)$cudaMajor".
+    #>
+    param([string]$Mode = "")
+    if ((Get-CtkTestMode $Mode) -eq "sysctk") { return "sysctk" }
+    return "cu"
+}
+
+function Convert-ToUnixPath {
+    Param([Parameter(Mandatory = $true)][string]$p)
+    return ($p -replace "\\", "/")
+}
+
+function Get-CudaCcclWheel {
+    <#
+    .SYNOPSIS
+        Returns the path of the cuda-cccl wheel artifact to use in the context
+        of a GitHub Actions CI test script.
+    #>
+    Param()
+
+    $repoRoot = Get-RepoRoot
+    if ($env:GITHUB_ACTIONS) {
+        Push-Location $repoRoot
+        try {
+            $wheelArtifactName = (& bash -lc "ci/util/workflow/get_wheel_artifact_name.sh").Trim()
+            if (-not $wheelArtifactName) { throw 'Failed to resolve wheel artifact name' }
+            $repoRootPosix = Convert-ToUnixPath $repoRoot
+            # Ensure output from downloader goes to console, not function return pipeline
+            $null = (& bash -lc "ci/util/artifacts/download.sh $wheelArtifactName $repoRootPosix" 2>&1 | Out-Host)
+            if ($LASTEXITCODE -ne 0) { throw "Failed to download wheel artifact '$wheelArtifactName'" }
+        }
+        finally { Pop-Location }
+    }
+
+    $wheelhouse = Join-Path $repoRoot 'wheelhouse'
+    $wheelPath = Get-OnePathMatch -Path $wheelhouse -Pattern '^cuda_cccl-.*\.whl' -File
+    return $wheelPath
+}
+
+function Get-OnePathMatch {
+    <#
+    .SYNOPSIS
+        Returns a single path (file or directory) match for a given pattern,
+        throwing an error if there were no matches or more than one match.
+    #>
+    [CmdletBinding(DefaultParameterSetName = 'FileSet')]
+    param(
+        [Parameter(Mandatory)]
+        [string] $Path,
+
+        [Parameter(Mandatory)]
+        [string] $Pattern,
+
+        [Parameter(Mandatory, ParameterSetName = 'FileSet')]
+        [switch] $File,
+
+        [Parameter(Mandatory, ParameterSetName = 'DirSet')]
+        [switch] $Directory,
+
+        [switch] $Recurse
+    )
+
+    if (-not (Test-Path -LiteralPath $Path -PathType Container)) {
+        throw "Path not found or not a directory: $Path"
+    }
+
+    $gciArgs = @{
+        LiteralPath = $Path
+        ErrorAction = 'SilentlyContinue'
+    }
+
+    if ($Recurse) { $gciArgs['Recurse'] = $true }
+    if ($PSCmdlet.ParameterSetName -eq 'FileSet') {
+        $gciArgs['File'] = $true
+    }
+    else {
+        $gciArgs['Directory'] = $true
+    }
+
+    $pathMatches = @(
+        Get-ChildItem @gciArgs |
+        Where-Object { $_.Name -match $Pattern } |
+        Select-Object -ExpandProperty FullName
+    )
+
+    if ($pathMatches.Count -ne 1) {
+        $kind = if ($PSCmdlet.ParameterSetName -eq 'FileSet') { 'file' }
+        else { 'directory' }
+        $indented = ($pathMatches | ForEach-Object { "    $_" }) -join "`n"
+
+        $msg = @"
+Expected exactly one $kind name matching regex:
+  $Pattern
+under:
+  $Path
+Found:
+  $($pathMatches.Count)
+
+$indented
+"@
+        throw $msg
+    }
+
+    return $pathMatches[0]
+}
+
+Export-ModuleMember -Function Get-Python, Get-CudaMajor, Set-CtkPin, Get-CtkExtraFlavor, Convert-ToUnixPath, Get-RepoRoot, Get-CudaCcclWheel, Get-OnePathMatch
diff --git a/cccl_upstream/ci/windows/build_cub.ps1 b/cccl_upstream/ci/windows/build_cub.ps1
new file mode 100644
index 00000000..37db55dc
--- /dev/null
+++ b/cccl_upstream/ci/windows/build_cub.ps1
@@ -0,0 +1,72 @@
+Param(
+    [Parameter(Mandatory = $false)]
+    [Alias("std")]
+    [ValidateNotNullOrEmpty()]
+    [ValidateSet(17, 20)]
+    [int]$CXX_STANDARD = 17,
+    [Parameter(Mandatory = $false)]
+    [Alias("arch")]
+    [string]$CUDA_ARCH = "",
+    [Parameter(Mandatory = $false)]
+    [Alias("cmake-options")]
+    [string]$CMAKE_OPTIONS = "",
+    [Parameter(Mandatory = $false)]
+    [Alias("no-lid")]
+    [switch]$NO_LID_SWITCH = $false,
+    [Parameter(Mandatory = $false)]
+    [Alias("lid0")]
+    [switch]$LID0_SWITCH = $false,
+    [Parameter(Mandatory = $false)]
+    [Alias("lid1")]
+    [switch]$LID1_SWITCH = $false,
+    [Parameter(Mandatory = $false)]
+    [Alias("lid2")]
+    [switch]$LID2_SWITCH = $false
+)
+
+$ErrorActionPreference = "Stop"
+
+$CURRENT_PATH = Split-Path $pwd -leaf
+If($CURRENT_PATH -ne "ci") {
+    Write-Host "Moving to ci folder"
+    pushd "$PSScriptRoot/.."
+}
+
+Import-Module $PSScriptRoot/build_common.psm1 -ArgumentList @($CXX_STANDARD, $CUDA_ARCH, $CMAKE_OPTIONS)
+
+$PRESET = "cub"
+$artifactTags = @()
+
+if ($NO_LID_SWITCH) {
+    $artifactTags += "no_lid"
+    $PRESET = "cub-nolid"
+} elseif ($LID0_SWITCH) {
+    $artifactTags += "lid_0"
+    $PRESET = "cub-lid0"
+} elseif ($LID1_SWITCH) {
+    $artifactTags += "lid_1"
+    $PRESET = "cub-lid1"
+} elseif ($LID2_SWITCH) {
+    $artifactTags += "lid_2"
+    $PRESET = "cub-lid2"
+}
+$LOCAL_CMAKE_OPTIONS = "-DCMAKE_CXX_STANDARD=$CXX_STANDARD -DCMAKE_CUDA_STANDARD=$CXX_STANDARD"
+
+if ($CL_VERSION -lt [version]"19.20") {
+    $LOCAL_CMAKE_OPTIONS = "$LOCAL_CMAKE_OPTIONS -DCCCL_IGNORE_DEPRECATED_COMPILER=ON"
+}
+
+configure_and_build_preset "CUB" $PRESET $LOCAL_CMAKE_OPTIONS
+
+if ($env:GITHUB_ACTIONS) {
+    Write-Host "Packaging test artifacts..."
+    if ($artifactTags.Count -gt 0) {
+        & bash "./upload_cub_test_artifacts.sh" @artifactTags
+    } else {
+        & bash "./upload_cub_test_artifacts.sh"
+    }
+}
+
+If($CURRENT_PATH -ne "ci") {
+    popd
+}
diff --git a/cccl_upstream/ci/windows/build_cuda_cccl_python.ps1 b/cccl_upstream/ci/windows/build_cuda_cccl_python.ps1
new file mode 100644
index 00000000..e151c673
--- /dev/null
+++ b/cccl_upstream/ci/windows/build_cuda_cccl_python.ps1
@@ -0,0 +1,422 @@
+<#
+.SYNOPSIS
+    Build Python cuda-cccl wheels on Windows.
+
+.DESCRIPTION
+    This script is the Windows analog to the Linux ../build_cuda_cccl_python.sh
+    script.  It is responsible for building CUDA 12.x and CUDA 13.x wheels that
+    are then merged together into a singular cuda-cccl wheel.
+
+    A single CUDA 12.9 builder image (i.e. Docker devcontainer) is used to
+    build each distinct Python/MSVC combo.  Much like the Linux approach, this
+    script detects when launched via the outer 12.9 instance, builds a `cu12`
+    wheel, then dispatches a inner Docker instance (Docker-out-of-Docker) to
+    execute this script with `-OnlyCudaMajor 13 -SkipUpload` parameters, which
+    yields a `cu13` build.
+
+    Upon completion of the `cu13` build, the outer 12.9 container merges both
+    `cu12` and `cu13` wheels into a single cuda-cccl wheel, and uploads that
+    via the standard CCCL CI artifact upload mechanisms.
+
+.PARAMETER PyVersion
+    **Required.** The Python version to use for building the wheel, expressed
+    as `<major>.<minor>` (e.g. `3.11`) or a free-threaded version such as
+    `3.14t`.
+
+.PARAMETER OnlyCudaMajor
+    Optional. Restricts the build to a single CUDA major version (`12` or `13`).
+    When set, only that version is built and the *merge* step is skipped.
+
+.PARAMETER Cuda13Image
+    Optional. The Docker image name used for a nested build of the CUDA 13
+    wheel when the outer container defaults to CUDA 12.9.  The default value
+    matches the RAPIDS dev-container image that contains the required
+    toolchain: `rapidsai/devcontainers:26.06-cuda13.0-cl14.44-windows2022`.
+
+.PARAMETER SkipUpload
+    When set, prevents the final wheel(s) from being uploaded as a GitHub
+    Actions artifact even when the script detects it is running inside an
+    Action.
+
+.EXAMPLE
+    # Build a single cuda-cccl wheel for Python 3.13 (consisting of both CUDA
+    # 12 and 13 versions), and, if in CI, upload the resulting wheel as an
+    # artifact.
+    .\build_cuda_cccl_python.ps1 -PyVersion 3.11
+#>
+
+[CmdletBinding()]
+Param(
+    [Parameter(Mandatory = $true)]
+    [Alias("py-version")]
+    [ValidatePattern("^\d+\.\d+t?$")]
+    [string]$PyVersion,
+
+    [Parameter(Mandatory = $false)]
+    [ValidateSet('12', '13')]
+    [string]$OnlyCudaMajor,
+
+    [Parameter(Mandatory = $false)]
+    [string]$Cuda13Image = "rapidsai/devcontainers:26.06-cuda13.0-cl14.44-windows2022",
+
+    [Parameter(Mandatory = $false)]
+    [switch]$SkipUpload
+)
+
+$ErrorActionPreference = "Stop"
+
+# Import shared helpers.
+Import-Module "$PSScriptRoot/build_common.psm1"
+Import-Module "$PSScriptRoot/build_common_python.psm1" -Force
+
+# Resolve repo root from this script's location.
+$RepoRoot = Resolve-Path "$PSScriptRoot/../.."
+Write-Host "Repo root: $RepoRoot"
+
+# Get the full path to the python.exe for the version we need.
+Write-Host "Looking for Python version $PyVersion..."
+$PythonExe = Get-Python -Version $PyVersion
+Write-Host "Using Python: $PythonExe"
+& $PythonExe -m pip --version
+
+# Ensure MSVC is available.
+$clPath = (Get-Command cl).Source
+if (-not $clPath) {
+    throw "cl.exe not found in PATH. Run from a Developer PowerShell prompt."
+}
+Write-Host "Found cl.exe at: $clPath"
+
+function Resolve-CudaPathForMajor {
+    Param(
+        [Parameter(Mandatory = $true)]
+        [ValidateSet('12', '13')]
+        [string]$Major
+    )
+    $candidates = @()
+    Get-ChildItem Env: |
+    Where-Object { $_.Name -match "^CUDA_PATH_V${Major}_(\d+)$" } |
+    ForEach-Object {
+        $minor = [int]([regex]::Match(
+                $_.Name,
+                "^CUDA_PATH_V${Major}_(\d+)$"
+            ).Groups[1].Value)
+        $candidates += [PSCustomObject]@{
+            Minor = $minor;
+            Path  = $_.Value
+        }
+    }
+
+    if ($candidates.Count -gt 0) {
+        return ($candidates | Sort-Object -Property Minor -Descending |
+            Select-Object -First 1).Path
+    }
+
+    if ($env:CUDA_PATH) {
+        $maybe = $env:CUDA_PATH
+        $nvcc = Join-Path $maybe 'bin/nvcc.exe'
+        if (Test-Path $nvcc) {
+            $out = & $nvcc --version 2>&1
+            $text = ($out -join "`n")
+            if ($text -match 'release\s+(\d+)\.') {
+                if ($Matches[1] -eq $Major) {
+                    return $maybe
+                }
+            }
+        }
+    }
+
+    return $null
+}
+
+# If $OnlyCudaMajor is present, it means we're being launched from a
+# nested Docker container build (12.x launched a 13.x build via DooD).
+if ($OnlyCudaMajor) {
+    $CudaMajorsToBuild = @($OnlyCudaMajor)
+}
+else {
+    $CudaMajorsToBuild = @('12', '13')
+}
+$DoMerge = -not [bool]$OnlyCudaMajor
+
+# Base pip/CMake options
+$pipBaseConfigArgs = @(
+    '-C', 'cmake.define.CMAKE_C_COMPILER=cl.exe',
+    '-C', 'cmake.define.CMAKE_CXX_COMPILER=cl.exe'
+)
+
+$env:CMAKE_GENERATOR = "Ninja"
+
+# Ensure wheelhouse directories exist.
+$Wheelhouse = Join-Path $RepoRoot "wheelhouse"
+New-Item -ItemType Directory -Path $Wheelhouse -Force | Out-Null
+${null} = New-Item -ItemType Directory -Path (Join-Path $RepoRoot 'wheelhouse_cu12') -Force
+${null} = New-Item -ItemType Directory -Path (Join-Path $RepoRoot 'wheelhouse_cu13') -Force
+
+function Invoke-Cuda13NestedBuild {
+    <#
+    .SYNOPSIS
+        Run the nested Docker build for CUDA 13 when we are already inside a
+        CUDA 12 builder image.
+
+    .DESCRIPTION
+        This routine launches a Docker devcontainer CUDA 13 build for the given
+        Python version by way of Docker-out-of-Docker (DooD) facilities.
+    #>
+    [CmdletBinding()]
+    param (
+        [Parameter(Mandatory)] [string] $Cuda13Image,
+        [Parameter(Mandatory)] [string] $PyVersion,
+        [ValidateNotNullOrEmpty()] [string] $HostWorkspace = $env:HOST_WORKSPACE,
+        [ValidateNotNullOrEmpty()] [string] $ContainerWorkspace = $env:CONTAINER_WORKSPACE
+    )
+
+    # Validate required environment variables.
+    if (-not $HostWorkspace) {
+        throw "HOST_WORKSPACE env var is not set; required for DooD " +
+        "nested docker mounts on Windows."
+    }
+    if (-not $ContainerWorkspace) {
+        throw "CONTAINER_WORKSPACE env var is not set; required for " +
+        "DooD nested docker mounts on Windows."
+    }
+
+    # Validate Docker CLI availability.
+    if (-not (Get-Command docker -ErrorAction SilentlyContinue)) {
+        throw "docker CLI not found in the devcontainer image (required for DooD)."
+    }
+
+    Write-Host "Checking DooD connectivity..."
+    $dockerVersionOutput = & docker version 2>&1
+    $dockerExitCode = $LASTEXITCODE
+    $dockerVersionOutput | Out-Host
+    if ($dockerExitCode -ne 0) {
+        throw "DooD connectivity check failed (exit code $dockerExitCode). See Docker output above."
+    }
+    Write-Host "DooD appears to be working, continuing..."
+
+    # Detect outer-container resources so we can set sensible limits.
+    $os = Get-WmiObject -Class Win32_OperatingSystem
+    $totalGB = [math]::Floor($os.TotalVisibleMemorySize / 1MB) # KB -> GB
+    $procCount = [Environment]::ProcessorCount
+
+    # Leave a little head-room so the outer container doesn't starve
+    $memLimitGB = [math]::Max(2, [int]([math]::Floor($totalGB * 0.9)))
+    $cpuCount = [math]::Max(2, $procCount)
+
+    Write-Host "Launching nested Docker for CUDA 13 build using image: $Cuda13Image"
+    $targetFile = Join-Path $ContainerWorkspace 'ci\windows\build_cuda_cccl_python.ps1'
+    $dockerArgs = @(
+        'run', '--rm', '-i',
+        '--cpu-count', "$cpuCount",
+        '--memory', "${memLimitGB}g",
+        '--workdir', $ContainerWorkspace,
+        '--mount', "type=bind,source=$HostWorkspace,target=$ContainerWorkspace",
+        '--env', "py_version=$PyVersion",
+        '--env', "GITHUB_ACTIONS=$($env:GITHUB_ACTIONS)",
+        '--env', "GITHUB_RUN_ID=$($env:GITHUB_RUN_ID)",
+        '--env', "JOB_ID=$($env:JOB_ID)",
+        $Cuda13Image,
+        'PowerShell.exe', '-NoLogo', '-NoProfile', '-ExecutionPolicy', 'Bypass',
+        '-File', $targetFile,
+        '-py-version', $PyVersion,
+        '-OnlyCudaMajor', '13',
+        '-SkipUpload'
+    )
+
+    Write-Host ("About to invoke: docker " + ($dockerArgs -join ' '))
+    Invoke-Checked { & docker @dockerArgs } 'Nested CUDA 13 wheel build failed'
+}
+
+function Build-CudaCcclWheel {
+    <#
+    .SYNOPSIS
+        Perform the regular wheel build for a given CUDA major version.
+
+    .DESCRIPTION
+        This routine is used to build both CUDA 12 and CUDA 13 based wheels,
+        and is called from normal "outer" Docker containers, as well as the
+        "inner" nested ones.
+    #>
+    [CmdletBinding()]
+    param (
+        [Parameter(Mandatory)] [ValidateSet('12', '13')] [string] $Major,
+        [Parameter(Mandatory)] [string] $RepoRoot,
+        [Parameter(Mandatory)] [string] $PythonExe,
+        [Parameter(Mandatory)] [string[]] $PipBaseConfigArgs
+    )
+
+    # Resolve CUDA toolkit location for the requested major version.
+    $CudaPathForMajor = Resolve-CudaPathForMajor -Major $Major
+    if (-not $CudaPathForMajor) {
+        throw "CUDA Toolkit $Major not found. Ensure CUDA_PATH_V${Major}_* " +
+        "is set or matching toolkit is installed."
+    }
+
+    $NvccForMajor = Join-Path $CudaPathForMajor 'bin/nvcc.exe'
+    if (-not (Test-Path $NvccForMajor)) {
+        throw "nvcc not found at $NvccForMajor"
+    }
+
+    # Convert Windows paths to Unix-style for CMake
+    $NvccUnix = Convert-ToUnixPath $NvccForMajor
+    $CudaUnix = Convert-ToUnixPath $CudaPathForMajor
+
+    # Build the pip configuration arguments that inject the CUDA toolchain.
+    $pipConfigArgs = $PipBaseConfigArgs + @(
+        '-C', "cmake.define.CMAKE_CUDA_COMPILER=$NvccUnix",
+        '-C', "cmake.define.CUDAToolkit_ROOT=$CudaUnix"
+    )
+
+    $extra = "cu$Major"
+    # Use separate output directories for 12 vs 13.
+    $outDir = Join-Path $RepoRoot "wheelhouse_$extra"
+
+    Write-Host "Building cuda-cccl wheel for CUDA $Major at $CudaPathForMajor..."
+
+    # Run pip wheel to build the wheel.
+    $pythonArgs = @(
+        '-m', 'pip', 'wheel',
+        '-w', $outDir,
+        ".[${extra}]",
+        '-v'
+    ) + $pipConfigArgs
+
+    Write-Host ("python " + ($pythonArgs -join ' '))
+    Invoke-Checked { & $PythonExe @pythonArgs } "Wheel build failed for CUDA $Major"
+
+    # Normalise the wheel filename (append .cu12/.cu13) and prune duplicates.
+    $builtWheel = Get-OnePathMatch -Path $outDir `
+        -Pattern '^cuda_cccl-.*\.whl' `
+        -File
+    if (-not $builtWheel) {
+        throw "Failed to locate built wheel in $outDir for CUDA $Major"
+    }
+
+    $builtName = [System.IO.Path]::GetFileName($builtWheel)
+    if ($builtName -notmatch ".cu$Major\.whl$") {
+        $newName = ([System.IO.Path]::GetFileNameWithoutExtension($builtName)) `
+            + ".cu$Major.whl"
+        Write-Host "Renaming wheel to: $newName"
+        Rename-Item -Path $builtWheel -NewName $newName -Force
+    }
+
+    # Remove any stray wheels that lack the .cuXX suffix.
+    Get-ChildItem -Path $outDir -Filter 'cuda_cccl-*.whl' |
+    Where-Object { $_.Name -notmatch "\.cu$Major\.whl$" } |
+    ForEach-Object {
+        Write-Host "Removing duplicate wheel: $($_.FullName)"
+        Remove-Item -Force $_.FullName
+    }
+}
+
+# Main build entry code.
+Push-Location (Join-Path $RepoRoot 'python/cuda_cccl')
+try {
+    foreach ($major in $CudaMajorsToBuild) {
+
+        # Nested Docker build for CUDA 13 for when we are currently inside a
+        # CUDA 12 image.
+        if (-not $OnlyCudaMajor -and $major -eq '13' -and $Cuda13Image) {
+            Invoke-Cuda13NestedBuild `
+                -Cuda13Image $Cuda13Image `
+                -PyVersion $PyVersion
+
+            continue
+        }
+
+        # Perform a normal build for the current major version.  This may
+        # be invoked from either an "outer" or inner "nested" image.
+        Build-CudaCcclWheel `
+            -Major $major `
+            -RepoRoot $RepoRoot `
+            -PythonExe $PythonExe `
+            -PipBaseConfigArgs $pipBaseConfigArgs
+    }
+}
+finally {
+    Pop-Location
+}
+
+
+# Merge the two major-version wheels (if both were built).  This will fail if
+# either wheel can't be found.  This only runs on the outer (non-nested)
+# container image.
+if ($DoMerge) {
+
+    $Cu12Wheel = Get-OnePathMatch `
+        -Path (Join-Path $RepoRoot 'wheelhouse_cu12') `
+        -Pattern '^cuda_cccl-.*\.cu12\.whl' `
+        -File
+
+    $Cu13Wheel = Get-OnePathMatch `
+        -Path (Join-Path $RepoRoot 'wheelhouse_cu13') `
+        -Pattern '^cuda_cccl-.*\.cu13\.whl' `
+        -File
+
+    Write-Host "Found CUDA 12 wheel: $Cu12Wheel"
+    Write-Host "Found CUDA 13 wheel: $Cu13Wheel"
+
+    Write-Host 'Merging CUDA wheels...'
+    Invoke-Checked { & $PythonExe -m pip install wheel | Write-Host } 'Failed to install wheel for merging'
+
+    $WheelhouseMerged = Join-Path $RepoRoot 'wheelhouse_merged'
+    ${null} = New-Item -ItemType Directory -Path $WheelhouseMerged -Force
+
+    $mergePy = Join-Path $RepoRoot 'python/cuda_cccl/merge_cuda_wheels.py'
+    Invoke-Checked { & $PythonExe $mergePy $Cu12Wheel $Cu13Wheel --output-dir $WheelhouseMerged } 'Merging wheels failed'
+
+    # Clean up the per-major directories and move the merged wheel into the
+    # final location.
+    Get-ChildItem $Wheelhouse -Filter '*.whl' |
+    ForEach-Object {
+        Remove-Item -Force $_.FullName
+    }
+    $MergedWheel = Get-OnePathMatch `
+        -Path $WheelhouseMerged `
+        -Pattern '^cuda_cccl-.*\.whl' `
+        -File
+    Move-Item -Force $MergedWheel $Wheelhouse
+
+    Remove-Item $WheelhouseMerged -Recurse -Force -ErrorAction SilentlyContinue
+    Remove-Item (Join-Path $RepoRoot 'wheelhouse_cu12') `
+        -Recurse -Force -ErrorAction SilentlyContinue
+    Remove-Item (Join-Path $RepoRoot 'wheelhouse_cu13') `
+        -Recurse -Force -ErrorAction SilentlyContinue
+
+    Write-Host 'Final wheels in wheelhouse:'
+    Get-ChildItem $Wheelhouse -Filter '*.whl' |
+    ForEach-Object {
+        Write-Host " - $($_.Name)"
+    }
+}
+
+# If it turns out we need delvewheel, we'd handle it here, after the merging
+# of wheels.  The two DLLs that seem like they might be problematic are
+# msvc140p.dll, and dbghelp.dll.  The former comes from llvmlite, upon which
+# we depend.  Dbghelp.dll ships in C:\Windows\System32, but that will often
+# be a much older version compared to the one used by Visual Studio.  We only
+# use one symbol from Dbghelp.dll: UnDecorateSymbolName, which is used by
+# nvrtc.  If we encounter weird issues with c.parallel jit compilation and
+# nvrtc in the wild on Windows, an out-of-date Dbghelp.dll could possibly be
+# the culprit.
+#
+# For now, though, it doesn't appear to be necessary.
+
+# Optionally upload the wheel artifact.
+if ($env:GITHUB_ACTIONS -and -not $SkipUpload) {
+    Push-Location $RepoRoot
+    try {
+        Write-Host 'GITHUB_ACTIONS detected; uploading wheel artifact'
+        $wheelArtifactName = (& bash -lc "ci/util/workflow/get_wheel_artifact_name.sh").Trim()
+        if (-not $wheelArtifactName) {
+            throw 'Failed to resolve wheel artifact name'
+        }
+        Write-Host "Wheel artifact name: $wheelArtifactName"
+
+        $uploadCmd = "ci/util/artifacts/upload.sh $wheelArtifactName 'wheelhouse/.*'"
+        Invoke-Checked { & bash -lc $uploadCmd } 'Wheel artifact upload failed'
+    }
+    finally {
+        Pop-Location
+    }
+}
diff --git a/cccl_upstream/ci/windows/build_cudax.ps1 b/cccl_upstream/ci/windows/build_cudax.ps1
new file mode 100644
index 00000000..e7c44084
--- /dev/null
+++ b/cccl_upstream/ci/windows/build_cudax.ps1
@@ -0,0 +1,31 @@
+Param(
+    [Parameter(Mandatory = $false)]
+    [Alias("std")]
+    [ValidateNotNullOrEmpty()]
+    [ValidateSet(20)]
+    [int]$CXX_STANDARD = 20,
+    [Parameter(Mandatory = $false)]
+    [Alias("arch")]
+    [string]$CUDA_ARCH = "",
+    [Parameter(Mandatory = $false)]
+    [Alias("cmake-options")]
+    [string]$CMAKE_OPTIONS = ""
+)
+
+$CURRENT_PATH = Split-Path $pwd -leaf
+If($CURRENT_PATH -ne "ci") {
+    Write-Host "Moving to ci folder"
+    pushd "$PSScriptRoot/.."
+}
+
+Remove-Module -Name build_common -ErrorAction SilentlyContinue
+Import-Module $PSScriptRoot/build_common.psm1 -ArgumentList @($CXX_STANDARD, $CUDA_ARCH, $CMAKE_OPTIONS)
+
+$PRESET = "cudax"
+$LOCAL_CMAKE_OPTIONS = "-DCMAKE_CXX_STANDARD=$CXX_STANDARD -DCMAKE_CUDA_STANDARD=$CXX_STANDARD"
+
+configure_and_build_preset "CUDA Experimental" $PRESET $LOCAL_CMAKE_OPTIONS
+
+If($CURRENT_PATH -ne "ci") {
+    popd
+}
diff --git a/cccl_upstream/ci/windows/build_libcudacxx.ps1 b/cccl_upstream/ci/windows/build_libcudacxx.ps1
new file mode 100644
index 00000000..6b0aff2e
--- /dev/null
+++ b/cccl_upstream/ci/windows/build_libcudacxx.ps1
@@ -0,0 +1,48 @@
+Param(
+    [Parameter(Mandatory = $false)]
+    [Alias("std")]
+    [ValidateNotNullOrEmpty()]
+    [ValidateSet(17, 20)]
+    [int]$CXX_STANDARD = 17,
+    [Parameter(Mandatory = $false)]
+    [Alias("arch")]
+    [string]$CUDA_ARCH = "",
+    [Parameter(Mandatory = $false)]
+    [Alias("cmake-options")]
+    [string]$CMAKE_OPTIONS = ""
+)
+
+$ErrorActionPreference = "Stop"
+
+$CURRENT_PATH = Split-Path $pwd -leaf
+If($CURRENT_PATH -ne "ci") {
+    Write-Host "Moving to ci folder"
+    pushd "$PSScriptRoot/.."
+}
+
+Import-Module $PSScriptRoot/build_common.psm1 -ArgumentList @($CXX_STANDARD, $CUDA_ARCH, $CMAKE_OPTIONS)
+
+$PRESET = "libcudacxx"
+$LOCAL_CMAKE_OPTIONS = "-DCMAKE_CXX_STANDARD=$CXX_STANDARD -DCMAKE_CUDA_STANDARD=$CXX_STANDARD"
+
+$uploadTestArtifacts = $false
+if ($env:GITHUB_ACTIONS) {
+    & bash "./util/workflow/has_consumers.sh"
+    $uploadTestArtifacts = $LASTEXITCODE -eq 0
+    if ($uploadTestArtifacts) {
+        $env:LIT_OPTS = "$env:LIT_OPTS -Dtest_executable_mode=build".Trim()
+    }
+}
+
+configure_and_build_preset "libcudacxx" $PRESET $LOCAL_CMAKE_OPTIONS
+
+if ($uploadTestArtifacts) {
+    Write-Host "Packaging test artifacts..."
+    Invoke-Checked {
+        & bash "./upload_libcudacxx_test_artifacts.sh"
+    } "Packaging test artifacts failed"
+}
+
+If($CURRENT_PATH -ne "ci") {
+    popd
+}
diff --git a/cccl_upstream/ci/windows/build_thrust.ps1 b/cccl_upstream/ci/windows/build_thrust.ps1
new file mode 100644
index 00000000..840f1177
--- /dev/null
+++ b/cccl_upstream/ci/windows/build_thrust.ps1
@@ -0,0 +1,37 @@
+Param(
+    [Parameter(Mandatory = $false)]
+    [Alias("std")]
+    [ValidateNotNullOrEmpty()]
+    [ValidateSet(17, 20)]
+    [int]$CXX_STANDARD = 17,
+    [Parameter(Mandatory = $false)]
+    [Alias("arch")]
+    [string]$CUDA_ARCH = "",
+    [Parameter(Mandatory = $false)]
+    [Alias("cmake-options")]
+    [string]$CMAKE_OPTIONS = ""
+)
+
+$ErrorActionPreference = "Stop"
+
+$CURRENT_PATH = Split-Path $pwd -leaf
+If($CURRENT_PATH -ne "ci") {
+    Write-Host "Moving to ci folder"
+    pushd "$PSScriptRoot/.."
+}
+
+Import-Module $PSScriptRoot/build_common.psm1 -ArgumentList @($CXX_STANDARD, $CUDA_ARCH, $CMAKE_OPTIONS)
+
+$PRESET = "thrust"
+$LOCAL_CMAKE_OPTIONS = "-DCMAKE_CXX_STANDARD=$CXX_STANDARD -DCMAKE_CUDA_STANDARD=$CXX_STANDARD"
+
+configure_and_build_preset "Thrust" $PRESET $LOCAL_CMAKE_OPTIONS
+
+if ($env:GITHUB_ACTIONS) {
+    Write-Host "Packaging test artifacts..."
+    & bash "./upload_thrust_test_artifacts.sh"
+}
+
+If($CURRENT_PATH -ne "ci") {
+    popd
+}
diff --git a/cccl_upstream/ci/windows/run_cpu_bisect.ps1 b/cccl_upstream/ci/windows/run_cpu_bisect.ps1
new file mode 100644
index 00000000..911d38d9
--- /dev/null
+++ b/cccl_upstream/ci/windows/run_cpu_bisect.ps1
@@ -0,0 +1,41 @@
+Param(
+    [Parameter(ValueFromRemainingArguments = $true)]
+    [string[]]$PassthroughArgs
+)
+
+$ErrorActionPreference = "Stop"
+
+$CURRENT_PATH = Split-Path $pwd -leaf
+if ($CURRENT_PATH -ne "ci") {
+    Write-Host "Moving to ci folder"
+    pushd "$PSScriptRoot/.."
+}
+
+if ($null -eq $PassthroughArgs) {
+    $PassthroughArgs = @()
+}
+
+Import-Module "$PSScriptRoot/build_common.psm1"
+
+$ciDirWindows = (Resolve-Path (Join-Path $PSScriptRoot "..")).Path
+$repoDirWindows = (Resolve-Path (Join-Path $ciDirWindows "..")).Path
+
+$ciDir = $ciDirWindows -replace "\\", "/"
+$repoDir = $repoDirWindows -replace "\\", "/"
+$utilScript = "$ciDir/util/git_bisect.sh"
+
+$argString = if ($PassthroughArgs.Count -gt 0) { " " + ($PassthroughArgs -join " ") } else { "" }
+
+$bashCommand = "cd $repoDir; $utilScript$argString"
+Write-Host $bashCommand -ForegroundColor Blue
+
+& bash -lc $bashCommand
+$exitCode = $LASTEXITCODE
+
+if ($CURRENT_PATH -ne "ci") {
+    popd
+}
+
+if ($exitCode -ne 0) {
+    exit $exitCode
+}
diff --git a/cccl_upstream/ci/windows/run_cpu_target.ps1 b/cccl_upstream/ci/windows/run_cpu_target.ps1
new file mode 100644
index 00000000..db928595
--- /dev/null
+++ b/cccl_upstream/ci/windows/run_cpu_target.ps1
@@ -0,0 +1,41 @@
+Param(
+    [Parameter(ValueFromRemainingArguments = $true)]
+    [string[]]$PassthroughArgs
+)
+
+$ErrorActionPreference = "Stop"
+
+$CURRENT_PATH = Split-Path $pwd -leaf
+if ($CURRENT_PATH -ne "ci") {
+    Write-Host "Moving to ci folder"
+    pushd "$PSScriptRoot/.."
+}
+
+if ($null -eq $PassthroughArgs) {
+    $PassthroughArgs = @()
+}
+
+Import-Module "$PSScriptRoot/build_common.psm1"
+
+$ciDirWindows = (Resolve-Path (Join-Path $PSScriptRoot "..")).Path
+$repoDirWindows = (Resolve-Path (Join-Path $ciDirWindows "..")).Path
+
+$ciDir = $ciDirWindows -replace "\\", "/"
+$repoDir = $repoDirWindows -replace "\\", "/"
+$utilScript = "$ciDir/util/build_and_test_targets.sh"
+
+$argString = if ($PassthroughArgs.Count -gt 0) { " " + ($PassthroughArgs -join " ") } else { "" }
+
+$bashCommand = "cd $repoDir; $utilScript$argString"
+Write-Host $bashCommand -ForegroundColor Blue
+
+& bash -lc $bashCommand
+$exitCode = $LASTEXITCODE
+
+if ($CURRENT_PATH -ne "ci") {
+    popd
+}
+
+if ($exitCode -ne 0) {
+    exit $exitCode
+}
diff --git a/cccl_upstream/ci/windows/run_gpu_bisect.ps1 b/cccl_upstream/ci/windows/run_gpu_bisect.ps1
new file mode 100644
index 00000000..911d38d9
--- /dev/null
+++ b/cccl_upstream/ci/windows/run_gpu_bisect.ps1
@@ -0,0 +1,41 @@
+Param(
+    [Parameter(ValueFromRemainingArguments = $true)]
+    [string[]]$PassthroughArgs
+)
+
+$ErrorActionPreference = "Stop"
+
+$CURRENT_PATH = Split-Path $pwd -leaf
+if ($CURRENT_PATH -ne "ci") {
+    Write-Host "Moving to ci folder"
+    pushd "$PSScriptRoot/.."
+}
+
+if ($null -eq $PassthroughArgs) {
+    $PassthroughArgs = @()
+}
+
+Import-Module "$PSScriptRoot/build_common.psm1"
+
+$ciDirWindows = (Resolve-Path (Join-Path $PSScriptRoot "..")).Path
+$repoDirWindows = (Resolve-Path (Join-Path $ciDirWindows "..")).Path
+
+$ciDir = $ciDirWindows -replace "\\", "/"
+$repoDir = $repoDirWindows -replace "\\", "/"
+$utilScript = "$ciDir/util/git_bisect.sh"
+
+$argString = if ($PassthroughArgs.Count -gt 0) { " " + ($PassthroughArgs -join " ") } else { "" }
+
+$bashCommand = "cd $repoDir; $utilScript$argString"
+Write-Host $bashCommand -ForegroundColor Blue
+
+& bash -lc $bashCommand
+$exitCode = $LASTEXITCODE
+
+if ($CURRENT_PATH -ne "ci") {
+    popd
+}
+
+if ($exitCode -ne 0) {
+    exit $exitCode
+}
diff --git a/cccl_upstream/ci/windows/run_gpu_target.ps1 b/cccl_upstream/ci/windows/run_gpu_target.ps1
new file mode 100644
index 00000000..db928595
--- /dev/null
+++ b/cccl_upstream/ci/windows/run_gpu_target.ps1
@@ -0,0 +1,41 @@
+Param(
+    [Parameter(ValueFromRemainingArguments = $true)]
+    [string[]]$PassthroughArgs
+)
+
+$ErrorActionPreference = "Stop"
+
+$CURRENT_PATH = Split-Path $pwd -leaf
+if ($CURRENT_PATH -ne "ci") {
+    Write-Host "Moving to ci folder"
+    pushd "$PSScriptRoot/.."
+}
+
+if ($null -eq $PassthroughArgs) {
+    $PassthroughArgs = @()
+}
+
+Import-Module "$PSScriptRoot/build_common.psm1"
+
+$ciDirWindows = (Resolve-Path (Join-Path $PSScriptRoot "..")).Path
+$repoDirWindows = (Resolve-Path (Join-Path $ciDirWindows "..")).Path
+
+$ciDir = $ciDirWindows -replace "\\", "/"
+$repoDir = $repoDirWindows -replace "\\", "/"
+$utilScript = "$ciDir/util/build_and_test_targets.sh"
+
+$argString = if ($PassthroughArgs.Count -gt 0) { " " + ($PassthroughArgs -join " ") } else { "" }
+
+$bashCommand = "cd $repoDir; $utilScript$argString"
+Write-Host $bashCommand -ForegroundColor Blue
+
+& bash -lc $bashCommand
+$exitCode = $LASTEXITCODE
+
+if ($CURRENT_PATH -ne "ci") {
+    popd
+}
+
+if ($exitCode -ne 0) {
+    exit $exitCode
+}
diff --git a/cccl_upstream/ci/windows/test_cccl_c_parallel.ps1 b/cccl_upstream/ci/windows/test_cccl_c_parallel.ps1
new file mode 100644
index 00000000..593a3bec
--- /dev/null
+++ b/cccl_upstream/ci/windows/test_cccl_c_parallel.ps1
@@ -0,0 +1,31 @@
+Param(
+    [Parameter(Mandatory = $false)]
+    [Alias("arch")]
+    [string]$CUDA_ARCH = "",
+    [Parameter(Mandatory = $false)]
+    [Alias("cmake-options")]
+    [string]$CMAKE_OPTIONS = ""
+)
+
+$ErrorActionPreference = "Stop"
+
+$CURRENT_PATH = Split-Path $pwd -leaf
+If($CURRENT_PATH -ne "ci") {
+    Write-Host "Moving to ci folder"
+    pushd "$PSScriptRoot/.."
+}
+
+# Build first
+$buildCmd = "$PSScriptRoot/build_cccl_c_parallel.ps1 -arch '$CUDA_ARCH' -cmake-options '$CMAKE_OPTIONS'"
+Write-Host "Running: $buildCmd"
+Invoke-Expression $buildCmd
+
+Remove-Module -Name build_common -ErrorAction SilentlyContinue
+Import-Module -Name "$PSScriptRoot/build_common.psm1" -ArgumentList @(20, $CUDA_ARCH, $CMAKE_OPTIONS)
+
+$PRESET = "cccl-c-parallel"
+test_preset "CCCL C Parallel" "$PRESET"
+
+If($CURRENT_PATH -ne "ci") {
+    popd
+}
diff --git a/cccl_upstream/ci/windows/test_cccl_c_parallel_v2.ps1 b/cccl_upstream/ci/windows/test_cccl_c_parallel_v2.ps1
new file mode 100644
index 00000000..73d3a4d5
--- /dev/null
+++ b/cccl_upstream/ci/windows/test_cccl_c_parallel_v2.ps1
@@ -0,0 +1,30 @@
+Param(
+    [Parameter(Mandatory = $false)]
+    [Alias("arch")]
+    [string]$CUDA_ARCH = "",
+    [Parameter(Mandatory = $false)]
+    [Alias("cmake-options")]
+    [string]$CMAKE_OPTIONS = ""
+)
+
+$ErrorActionPreference = "Stop"
+
+$CURRENT_PATH = Split-Path $pwd -leaf
+If($CURRENT_PATH -ne "ci") {
+    Write-Host "Moving to ci folder"
+    pushd "$PSScriptRoot/.."
+}
+
+Remove-Module -Name build_common -ErrorAction SilentlyContinue
+Import-Module -Name "$PSScriptRoot/build_common.psm1" -ArgumentList @(20, $CUDA_ARCH, $CMAKE_OPTIONS)
+
+$PRESET = "cccl-c-parallel-v2"
+$LOCAL_CMAKE_OPTIONS = ""
+
+configure_and_build_preset "CCCL C Parallel v2 (HostJIT)" $PRESET $LOCAL_CMAKE_OPTIONS
+
+test_preset "CCCL C Parallel v2 (HostJIT)" "$PRESET"
+
+If($CURRENT_PATH -ne "ci") {
+    popd
+}
diff --git a/cccl_upstream/ci/windows/test_cub.ps1 b/cccl_upstream/ci/windows/test_cub.ps1
new file mode 100644
index 00000000..2685e145
--- /dev/null
+++ b/cccl_upstream/ci/windows/test_cub.ps1
@@ -0,0 +1,73 @@
+Param(
+    [Parameter(Mandatory = $false)]
+    [Alias("std")]
+    [ValidateNotNullOrEmpty()]
+    [ValidateSet(17, 20)]
+    [int]$CXX_STANDARD = 17,
+    [Parameter(Mandatory = $false)]
+    [Alias("arch")]
+    [string]$CUDA_ARCH = "",
+    [Parameter(Mandatory = $false)]
+    [Alias("no-lid")]
+    [switch]$NO_LID_SWITCH = $false,
+    [Parameter(Mandatory = $false)]
+    [Alias("lid0")]
+    [switch]$LID0_SWITCH = $false,
+    [Parameter(Mandatory = $false)]
+    [Alias("lid1")]
+    [switch]$LID1_SWITCH = $false,
+    [Parameter(Mandatory = $false)]
+    [Alias("lid2")]
+    [switch]$LID2_SWITCH = $false,
+    [Parameter(Mandatory = $false)]
+    [Alias("cmake-options")]
+    [string]$CMAKE_OPTIONS = ""
+)
+
+$ErrorActionPreference = "Stop"
+
+$CURRENT_PATH = Split-Path $pwd -leaf
+If($CURRENT_PATH -ne "ci") {
+    Write-Host "Moving to ci folder"
+    pushd "$PSScriptRoot/.."
+}
+
+Import-Module -Name "$PSScriptRoot/build_common.psm1" -ArgumentList @($CXX_STANDARD, $CUDA_ARCH, $CMAKE_OPTIONS)
+
+$PRESET = "cub"
+$artifactTag = ""
+$variantArg = ""
+if ($NO_LID_SWITCH) {
+    $artifactTag = "no_lid"
+    $PRESET = "cub-nolid"
+    $variantArg = "-no-lid"
+} elseif ($LID0_SWITCH) {
+    $artifactTag = "lid_0"
+    $PRESET = "cub-lid0"
+    $variantArg = "-lid0"
+} elseif ($LID1_SWITCH) {
+    $artifactTag = "lid_1"
+    $PRESET = "cub-lid1"
+    $variantArg = "-lid1"
+} elseif ($LID2_SWITCH) {
+    $artifactTag = "lid_2"
+    $PRESET = "cub-lid2"
+    $variantArg = "-lid2"
+}
+
+if ($env:GITHUB_ACTIONS -and $artifactTag) {
+    $producerId = (& bash "./util/workflow/get_producer_id.sh").Trim()
+    $artifactName = "z_cub-test-artifacts-$env:DEVCONTAINER_NAME-$producerId-$artifactTag"
+    Write-Host "Unpacking artifact '$artifactName'"
+    & bash "./util/artifacts/download_packed.sh" "$artifactName" "../"
+} else {
+    $buildCmd = "$PSScriptRoot/build_cub.ps1 -std $CXX_STANDARD -arch '$CUDA_ARCH' -cmake-options '$CMAKE_OPTIONS' $variantArg"
+    Write-Host "Running: $buildCmd"
+    Invoke-Expression $buildCmd
+}
+
+test_preset "CUB ($PRESET)" "$PRESET"
+
+If($CURRENT_PATH -ne "ci") {
+    popd
+}
diff --git a/cccl_upstream/ci/windows/test_cuda_cccl_examples_python.ps1 b/cccl_upstream/ci/windows/test_cuda_cccl_examples_python.ps1
new file mode 100644
index 00000000..bbc9d1bc
--- /dev/null
+++ b/cccl_upstream/ci/windows/test_cuda_cccl_examples_python.ps1
@@ -0,0 +1,50 @@
+Param(
+    [Parameter(Mandatory = $true)]
+    [Alias("py-version")]
+    [ValidatePattern("^\d+\.\d+t?$")]
+    [string]$PyVersion,
+
+    [Alias("ctk-mode")]
+    [string]$CtkMode = ""
+)
+
+$ErrorActionPreference = "Stop"
+
+# Import shared helpers
+Import-Module "$PSScriptRoot/build_common.psm1"
+Import-Module "$PSScriptRoot/build_common_python.psm1"
+
+$python = Get-Python -Version $PyVersion
+$cudaMajor = Get-CudaMajor
+$ctkFlavor = Get-CtkExtraFlavor $CtkMode
+
+# Pin cuda-toolkit to the container's CTK minor (-ctk-mode latest
+# opts out). See build_common_python.psm1.
+Set-CtkPin $CtkMode
+
+$repoRoot = Get-RepoRoot
+
+${wheelPath} = Get-CudaCcclWheel
+
+# pytest-benchmark is for the host-benchmark smoke test below.
+Invoke-Checked { & $python -m pip install -U pip pytest pytest-xdist pytest-benchmark } "Failed to install pytest / pytest-xdist / pytest-benchmark"
+# CuPy is required by the cuda.compute examples and is not part of the test extras
+Invoke-Checked { & $python -m pip install "${wheelPath}[test-$ctkFlavor$cudaMajor]" "cupy-cuda${cudaMajor}x" } "Failed to install cuda_cccl test extra / cupy"
+
+Push-Location (Join-Path $repoRoot "python/cuda_cccl/tests")
+try {
+    Invoke-Checked { & $python -m pytest -n 6 test_examples.py } "examples tests failed"
+}
+finally { Pop-Location }
+
+# Smoke-test the host-overhead benchmark harness: run every benchmark case
+# exactly once (pass/fail only, no timing) so harness rot fails CI here instead
+# of silently surviving until someone runs the perf suite. --benchmark-disable
+# makes pytest-benchmark invoke each benchmarked callable a single time. This
+# lane already installs cupy + numba (for the examples), which the benchmark
+# suite also needs, so only pytest-benchmark is added above.
+Push-Location (Join-Path $repoRoot "python/cuda_cccl/benchmarks/compute/host")
+try {
+    Invoke-Checked { & $python -m pytest -v --benchmark-disable . } "host benchmark smoke test failed"
+}
+finally { Pop-Location }
diff --git a/cccl_upstream/ci/windows/test_cuda_cccl_headers_python.ps1 b/cccl_upstream/ci/windows/test_cuda_cccl_headers_python.ps1
new file mode 100644
index 00000000..4176bc04
--- /dev/null
+++ b/cccl_upstream/ci/windows/test_cuda_cccl_headers_python.ps1
@@ -0,0 +1,36 @@
+Param(
+    [Parameter(Mandatory = $true)]
+    [Alias("py-version")]
+    [ValidatePattern("^\d+\.\d+t?$")]
+    [string]$PyVersion,
+
+    [Alias("ctk-mode")]
+    [string]$CtkMode = ""
+)
+
+$ErrorActionPreference = "Stop"
+
+# Import shared helpers
+Import-Module "$PSScriptRoot/build_common.psm1"
+Import-Module "$PSScriptRoot/build_common_python.psm1"
+
+$python = Get-Python -Version $PyVersion
+$cudaMajor = Get-CudaMajor
+$ctkFlavor = Get-CtkExtraFlavor $CtkMode
+
+# Pin cuda-toolkit to the container's CTK minor (-ctk-mode latest
+# opts out). See build_common_python.psm1.
+Set-CtkPin $CtkMode
+
+$repoRoot = Get-RepoRoot
+
+${wheelPath} = Get-CudaCcclWheel
+
+Invoke-Checked { & $python -m pip install -U pip pytest pytest-xdist } "Failed to install pytest / pytest-xdist"
+Invoke-Checked { & $python -m pip install "${wheelPath}[test-$ctkFlavor$cudaMajor]" } "Failed to install cuda_cccl test extra"
+
+Push-Location (Join-Path $repoRoot "python/cuda_cccl/tests")
+try {
+    Invoke-Checked { & $python -m pytest -n auto -v headers/ } "headers tests failed"
+}
+finally { Pop-Location }
diff --git a/cccl_upstream/ci/windows/test_cuda_compute_minimal_python.ps1 b/cccl_upstream/ci/windows/test_cuda_compute_minimal_python.ps1
new file mode 100644
index 00000000..1f64d670
--- /dev/null
+++ b/cccl_upstream/ci/windows/test_cuda_compute_minimal_python.ps1
@@ -0,0 +1,79 @@
+Param(
+    [Parameter(Mandatory = $true)]
+    [Alias("py-version")]
+    [ValidatePattern("^\d+\.\d+t?$")]
+    [string]$PyVersion,
+
+    [Alias("ctk-mode")]
+    [string]$CtkMode = ""
+)
+
+$ErrorActionPreference = "Stop"
+
+# Import shared helpers
+Import-Module "$PSScriptRoot/build_common.psm1"
+Import-Module "$PSScriptRoot/build_common_python.psm1"
+
+$python = Get-Python -Version $PyVersion
+$cudaMajor = Get-CudaMajor
+$ctkFlavor = Get-CtkExtraFlavor $CtkMode
+
+# Pin cuda-toolkit to the container's CTK minor (-ctk-mode latest
+# opts out). See build_common_python.psm1.
+Set-CtkPin $CtkMode
+
+$repoRoot = Get-RepoRoot
+
+$wheelPath = Get-CudaCcclWheel
+
+# Install cuda_cccl with the minimal CUDA extra. This intentionally avoids the
+# full cu* extras because those pull in numba/numba-cuda.
+Invoke-Checked { & $python -m pip install -U pip pytest pytest-xdist } "Failed to install pytest / pytest-xdist"
+Invoke-Checked { & $python -m pip install "$wheelPath[minimal-$ctkFlavor$cudaMajor]" } "Failed to install cuda_cccl minimal extra"
+
+Push-Location (Join-Path $repoRoot "python/cuda_cccl/tests")
+try {
+    Invoke-Checked { & $python -m pytest -n 6 -v compute/test_no_numba.py } "test_no_numba.py failed"
+
+    if ($PyVersion -eq "3.14t") {
+        # Select only tests that support the minimal extra so pytest does not
+        # collect tests that import numba-cuda and re-enable the GIL. These tests
+        # provide their own worker threads, so keep pytest itself in a single
+        # process. The serialization node-ids are module-skipped on the v2
+        # backend today and will start running there automatically once v2 gains
+        # serialization support.
+        Invoke-Checked {
+            & $python -m pytest -n 0 -v `
+                compute/test_free_threading_stress.py `
+                compute/test_multi_cc_serialization.py::test_aot_build_result_load_failure_is_shared_and_retryable `
+                compute/test_multi_cc_serialization.py::test_aot_serialization_waits_for_canonical_first_load
+        } "free-threading stress / serialization tests failed"
+
+        # Broad thread-safety sweep (pytest-run-parallel): re-run the numba-free
+        # functional suite with each test executed concurrently across threads
+        # (barrier-synchronized start), stressing the process-wide build cache,
+        # single-flight coordination, and the Cython bindings from many threads at
+        # once. Complements test_free_threading_stress.py above, which targets
+        # specific shared-object scenarios by hand. -n 0 so the threads share one
+        # interpreter.
+        #
+        # --parallel-threads=2 matches CuPy's free-threading CI (the closest GPU
+        # precedent); a small fixed count bounds GPU-memory pressure from
+        # concurrent kernels and stays reproducible across runners, unlike =auto
+        # (the runner's logical-core count).
+        #
+        # pytest-run-parallel is only used by this sweep, so install it on the
+        # 3.14t path rather than for every minimal (e.g. non-free-threaded 3.14)
+        # run.
+        Invoke-Checked { & $python -m pip install pytest-run-parallel } "Failed to install pytest-run-parallel"
+
+        # Fail fast if the interpreter is not actually GIL-free (wrong build /
+        # PYTHON_GIL=1): pytest-run-parallel does NOT catch a GIL that is enabled
+        # from the start -- it would run threads GIL-serialized and pass
+        # vacuously. (A GIL *re-enabled mid-run* by a non-free-threaded import IS
+        # caught by the plugin, which is why we do not pass --ignore-gil-enabled.)
+        Invoke-Checked { & $python -c "import sys; assert not sys._is_gil_enabled(), 'GIL is enabled; parallel sweep has no signal'" } "interpreter is not GIL-free; parallel sweep has no signal"
+        Invoke-Checked { & $python -m pytest -n 0 -v --parallel-threads=2 compute/test_no_numba.py } "parallel-threads sweep failed"
+    }
+}
+finally { Pop-Location }
diff --git a/cccl_upstream/ci/windows/test_cuda_compute_python.ps1 b/cccl_upstream/ci/windows/test_cuda_compute_python.ps1
new file mode 100644
index 00000000..9283af57
--- /dev/null
+++ b/cccl_upstream/ci/windows/test_cuda_compute_python.ps1
@@ -0,0 +1,37 @@
+Param(
+    [Parameter(Mandatory = $true)]
+    [Alias("py-version")]
+    [ValidatePattern("^\d+\.\d+t?$")]
+    [string]$PyVersion,
+
+    [Alias("ctk-mode")]
+    [string]$CtkMode = ""
+)
+
+$ErrorActionPreference = "Stop"
+
+# Import shared helpers
+Import-Module "$PSScriptRoot/build_common.psm1"
+Import-Module "$PSScriptRoot/build_common_python.psm1"
+
+$python = Get-Python -Version $PyVersion
+$cudaMajor = Get-CudaMajor
+$ctkFlavor = Get-CtkExtraFlavor $CtkMode
+
+# Pin cuda-toolkit to the container's CTK minor (-ctk-mode latest
+# opts out). See build_common_python.psm1.
+Set-CtkPin $CtkMode
+
+$repoRoot = Get-RepoRoot
+
+$wheelPath = Get-CudaCcclWheel
+
+Invoke-Checked { & $python -m pip install -U pip pytest pytest-xdist } "Failed to install pytest / pytest-xdist"
+Invoke-Checked { & $python -m pip install "$wheelPath[test-$ctkFlavor$cudaMajor]" } "Failed to install cuda_cccl test extra"
+
+Push-Location (Join-Path $repoRoot "python/cuda_cccl/tests")
+try {
+    Invoke-Checked { & $python -m pytest -n 6 -v compute/ -m "not large and not free_threading" } "compute tests (not large) failed"
+    Invoke-Checked { & $python -m pytest -n 0 -v compute/ -m "large and not free_threading" } "compute tests (large) failed"
+}
+finally { Pop-Location }
diff --git a/cccl_upstream/ci/windows/test_cudax.ps1 b/cccl_upstream/ci/windows/test_cudax.ps1
new file mode 100644
index 00000000..5c7e4873
--- /dev/null
+++ b/cccl_upstream/ci/windows/test_cudax.ps1
@@ -0,0 +1,35 @@
+Param(
+    [Parameter(Mandatory = $false)]
+    [Alias("std")]
+    [ValidateNotNullOrEmpty()]
+    [ValidateSet(20)]
+    [int]$CXX_STANDARD = 20,
+    [Parameter(Mandatory = $false)]
+    [Alias("arch")]
+    [string]$CUDA_ARCH = "",
+    [Parameter(Mandatory = $false)]
+    [Alias("cmake-options")]
+    [string]$CMAKE_OPTIONS = ""
+)
+
+$ErrorActionPreference = "Stop"
+
+$CURRENT_PATH = Split-Path $pwd -leaf
+If($CURRENT_PATH -ne "ci") {
+    Write-Host "Moving to ci folder"
+    pushd "$PSScriptRoot/.."
+}
+
+# Build first
+$buildCmd = "$PSScriptRoot/build_cudax.ps1 -std $CXX_STANDARD -arch '$CUDA_ARCH' -cmake-options '$CMAKE_OPTIONS'"
+Write-Host "Running: $buildCmd"
+Invoke-Expression $buildCmd
+
+Import-Module -Name "$PSScriptRoot/build_common.psm1" -ArgumentList @($CXX_STANDARD, $CUDA_ARCH, $CMAKE_OPTIONS)
+
+$PRESET = "cudax"
+test_preset "CUDA Experimental" "$PRESET"
+
+If($CURRENT_PATH -ne "ci") {
+    popd
+}
diff --git a/cccl_upstream/ci/windows/test_libcudacxx.ps1 b/cccl_upstream/ci/windows/test_libcudacxx.ps1
new file mode 100644
index 00000000..a348475e
--- /dev/null
+++ b/cccl_upstream/ci/windows/test_libcudacxx.ps1
@@ -0,0 +1,53 @@
+Param(
+    [Parameter(Mandatory = $false)]
+    [Alias("std")]
+    [ValidateNotNullOrEmpty()]
+    [ValidateSet(17, 20)]
+    [int]$CXX_STANDARD = 17,
+    [Parameter(Mandatory = $false)]
+    [Alias("arch")]
+    [string]$CUDA_ARCH = "",
+    [Parameter(Mandatory = $false)]
+    [Alias("cmake-options")]
+    [string]$CMAKE_OPTIONS = ""
+)
+
+$ErrorActionPreference = "Stop"
+
+$CURRENT_PATH = Split-Path $pwd -leaf
+If($CURRENT_PATH -ne "ci") {
+    Write-Host "Moving to ci folder"
+    pushd "$PSScriptRoot/.."
+}
+
+Import-Module -Name "$PSScriptRoot/build_common.psm1" -ArgumentList @($CXX_STANDARD, $CUDA_ARCH, $CMAKE_OPTIONS)
+
+if ($env:GITHUB_ACTIONS) {
+    $producerId = & bash "./util/workflow/get_producer_id.sh"
+    if ($LASTEXITCODE -ne 0) {
+        throw "Finding the producer job failed (exit code $LASTEXITCODE)"
+    }
+    $producerId = "$producerId".Trim()
+    $artifactName = "z_libcudacxx-test-artifacts-$env:DEVCONTAINER_NAME-$producerId"
+    Write-Host "Unpacking artifact '$artifactName'"
+    Invoke-Checked {
+        & bash "./util/artifacts/download_packed.sh" "$artifactName" "../"
+    } "Downloading test artifacts failed"
+} else {
+    $buildCmd = "$PSScriptRoot/build_libcudacxx.ps1 -std $CXX_STANDARD -arch '$CUDA_ARCH' -cmake-options '$CMAKE_OPTIONS'"
+    Write-Host "Running: $buildCmd"
+    Invoke-Expression $buildCmd
+}
+
+if ($env:GITHUB_ACTIONS) {
+    test_preset "libcudacxx (CTest)" "libcudacxx-ctest"
+    $env:LIT_OPTS = "$env:LIT_OPTS -Dtest_executable_mode=replay".Trim()
+    test_preset "libcudacxx (lit replay)" "libcudacxx-lit"
+} else {
+    test_preset "libcudacxx (CTest)" "libcudacxx-ctest-cpp${CXX_STANDARD}"
+    test_preset "libcudacxx (lit)"   "libcudacxx-lit-cpp${CXX_STANDARD}"
+}
+
+If($CURRENT_PATH -ne "ci") {
+    popd
+}
diff --git a/cccl_upstream/ci/windows/test_packaging.ps1 b/cccl_upstream/ci/windows/test_packaging.ps1
new file mode 100644
index 00000000..fae829a4
--- /dev/null
+++ b/cccl_upstream/ci/windows/test_packaging.ps1
@@ -0,0 +1,37 @@
+Param(
+    [Parameter(Mandatory = $false)]
+    [Alias("std")]
+    [ValidateNotNullOrEmpty()]
+    [ValidateSet(17, 20)]
+    [int]$CXX_STANDARD = 17,
+    [Parameter(Mandatory = $false)]
+    [Alias("arch")]
+    [string]$CUDA_ARCH = "",
+    [Parameter(Mandatory = $false)]
+    [Alias("cmake-options")]
+    [string]$CMAKE_OPTIONS = ""
+)
+
+$ErrorActionPreference = "Stop"
+
+$CURRENT_PATH = Split-Path $pwd -leaf
+If($CURRENT_PATH -ne "ci") {
+    Write-Host "Moving to ci folder"
+    pushd "$PSScriptRoot/.."
+}
+
+Import-Module $PSScriptRoot/build_common.psm1 -ArgumentList @($CXX_STANDARD, $CUDA_ARCH, $CMAKE_OPTIONS)
+
+$PRESET = "packaging"
+$LOCAL_CMAKE_OPTIONS = ""
+
+if ($env:GITHUB_SHA) {
+    $LOCAL_CMAKE_OPTIONS = '"-DCCCL_EXAMPLE_CPM_TAG={0}"' -f $env:GITHUB_SHA
+}
+
+configure_preset "Packaging" $PRESET $LOCAL_CMAKE_OPTIONS
+test_preset "Packaging" $PRESET
+
+If($CURRENT_PATH -ne "ci") {
+    popd
+}
diff --git a/cccl_upstream/ci/windows/test_thrust.ps1 b/cccl_upstream/ci/windows/test_thrust.ps1
new file mode 100644
index 00000000..0c71f1c0
--- /dev/null
+++ b/cccl_upstream/ci/windows/test_thrust.ps1
@@ -0,0 +1,62 @@
+Param(
+    [Parameter(Mandatory = $false)]
+    [Alias("std")]
+    [ValidateNotNullOrEmpty()]
+    [ValidateSet(17, 20)]
+    [int]$CXX_STANDARD = 17,
+    [Parameter(Mandatory = $false)]
+    [Alias("arch")]
+    [string]$CUDA_ARCH = "",
+    [Parameter(Mandatory = $false)]
+    [Alias("cpu-only")]
+    [switch]$CPU_ONLY = $false,
+    [Parameter(Mandatory = $false)]
+    [Alias("gpu-only")]
+    [switch]$GPU_ONLY = $false,
+    [Parameter(Mandatory = $false)]
+    [Alias("cmake-options")]
+    [string]$CMAKE_OPTIONS = ""
+)
+
+$ErrorActionPreference = "Stop"
+
+$CURRENT_PATH = Split-Path $pwd -leaf
+If($CURRENT_PATH -ne "ci") {
+    Write-Host "Moving to ci folder"
+    pushd "$PSScriptRoot/.."
+}
+
+if ($CPU_ONLY) {
+    $artifactTag = "test_cpu"
+    $presets = @("thrust-cpu")
+} elseif ($GPU_ONLY) {
+    $artifactTag = "test_gpu"
+    $presets = @("thrust-gpu")
+} else {
+    if ($env:GITHUB_ACTIONS) {
+        throw "Error: test_thrust.ps1 requires -cpu-only or -gpu-only in CI"
+    }
+    $artifactTag = ""
+    $presets = @("thrust-cpu", "thrust-gpu")
+}
+
+if ($env:GITHUB_ACTIONS -and $artifactTag) {
+    $producerId = (& bash "./util/workflow/get_producer_id.sh").Trim()
+    $artifactName = "z_thrust-test-artifacts-$env:DEVCONTAINER_NAME-$producerId-$artifactTag"
+    Write-Host "Unpacking artifact '$artifactName'"
+    & bash "./util/artifacts/download_packed.sh" "$artifactName" "../"
+} else {
+    $cmd = "$PSScriptRoot/build_thrust.ps1 -std $CXX_STANDARD -arch '$CUDA_ARCH' -cmake-options '$CMAKE_OPTIONS'"
+    Write-Host "Running: $cmd"
+    Invoke-Expression $cmd
+}
+
+Import-Module -Name "$PSScriptRoot/build_common.psm1" -ArgumentList @($CXX_STANDARD, $CUDA_ARCH, $CMAKE_OPTIONS)
+
+foreach ($preset in $presets) {
+    test_preset "Thrust ($preset)" $preset
+}
+
+If($CURRENT_PATH -ne "ci") {
+    popd
+}
diff --git a/cccl_upstream/docs/.gitignore b/cccl_upstream/docs/.gitignore
new file mode 100644
index 00000000..28a8ad3f
--- /dev/null
+++ b/cccl_upstream/docs/.gitignore
@@ -0,0 +1,8 @@
+_build
+_repo
+env
+api
+*png
+cubimg
+*/auto_api.rst
+*/api/
diff --git a/cccl_upstream/docs/404.html b/cccl_upstream/docs/404.html
new file mode 100644
index 00000000..fb0b4e3d
--- /dev/null
+++ b/cccl_upstream/docs/404.html
@@ -0,0 +1,41 @@
+<!DOCTYPE html>
+
+<html lang="en" data-content_root="./" >
+    <head>
+        <title>404 &#8212; CUDA Core Compute Libraries</title>
+    </head>
+    <ul>
+        <div id="linkGen"></div>
+    </ul>
+    <script type="module">
+        import Fuse from 'https://cdn.jsdelivr.net/npm/fuse.js@7.0.0/dist/fuse.mjs'
+        // This page assumes that the user is already in a versioned subtree
+        // It is no longer the root 404 handler and is instead delagated to.
+        var localUri = window.location.pathname;
+        var hostUrl = window.location.protocol + "//" + window.location.host
+
+        // Check if user query is missing a branch prefix, e.g. cccl/thrust/vector.h
+        var rootPathMatch = localUri.match(/^\/cccl(?:\/unstable|\/branch\/\d+\.\d+\.\d+)/)
+        // Remove leading prefix, this is fed as a URI search argument to the appropriate 404 handler.
+        var searchPathMatch = localUri.replace(/^\/cccl(?:\/unstable|\/branch\/\d+\.\d+\.\d+)?/, "")
+
+        // Project default root leads to unstable
+        var rootPath = "/cccl/unstable"
+
+        // If a root path was found, use it instead
+        if (rootPathMatch !== null) {
+            rootPath = rootPathMatch[0]
+        }
+
+        // User was trying to reach a path, forward to appropriate 404 helper
+        if (searchPathMatch) {
+            // Redirect to the helper page, including the query
+            window.location.href =
+                hostUrl + rootPath + "/404_helper.html?path=" + searchPathMatch
+        }
+        // Redirect to index otherwise
+        else {
+            window.location.href = hostUrl + rootPath
+        }
+    </script>
+</html>
diff --git a/cccl_upstream/docs/404_helper.inc.html b/cccl_upstream/docs/404_helper.inc.html
new file mode 100644
index 00000000..83814d7d
--- /dev/null
+++ b/cccl_upstream/docs/404_helper.inc.html
@@ -0,0 +1,63 @@
+<ul>
+    <div id="linkGen"></div>
+</ul>
+<script type="module">
+    import Fuse from 'https://cdn.jsdelivr.net/npm/fuse.js@7.0.0/dist/fuse.mjs'
+    // This page assumes that the user is already in a versioned subtree
+    // It is no longer the root 404 handler and is instead delagated to.
+    var localUri = window.location.pathname;
+    var hostUrl = window.location.protocol + "//" + window.location.host
+
+    let params = new URLSearchParams(document.location.search);
+
+    var rootPath = localUri.replace(/\/404_helper\.html/, "")
+
+    var rootUrl = hostUrl + rootPath
+    var pagelistUrl = rootUrl + "/pagelist.txt"
+
+    // // Branch is versioned if it leads with `unstable` or `vX.Y.Z`
+    // var isBranchVersioned = localUri.match(/^\/cccl\/(unstable|v\d+\.\d+\.\d+)/);
+
+    // // If the branch is not versioned, we redirect to a versioned branch keeping the URI.
+    // if (!isBranchVersioned) {
+    //     window.location.href = hostUrl + '/' + localUri.replace(/^cccl\//g, "cccl/unstable");
+    // }
+
+    var searchString = params.get("path");
+
+    if (searchString === "/" || searchString === null) {
+      window.location.href = rootUrl + "/index.html"
+    }
+
+    if (searchString !== null) {
+        fetch(pagelistUrl)
+            .then((response) => response.text())
+            .then((text) => {
+                const pagelist = text.split(",")
+
+                const options = {
+                  includeScore: true,
+                  threshold: 0.8,
+                  ignoreLocation: false
+                }
+
+                // cccl/device_vector.html => 'cccl 'device 'vector
+                console.log("query:" + searchString)
+                const fuse = new Fuse(pagelist, options)
+                // Include the ending token in the search to prioritize short matches
+                const result = fuse.search(searchString + "(end)", {limit: 10})
+
+                const listItem = '<li><a href=\"{2}{0}\">{0}</a> - Score: {1}</li>'.replace(/\{2\}/g, rootUrl)
+                document.getElementById('linkGen').innerHTML = ""
+                result.forEach(element => {
+                  const targetUri = element.item.replace("(end)", "")
+                  const targetScore = element.score.toFixed(2)
+                  document.getElementById('linkGen').innerHTML += listItem.replace(/\{0\}/g, targetUri).replace(/\{1\}/g, targetScore)
+                  // The match is near perfect, just link directly to it.
+                  if (targetScore <= 0.02 ) {
+                    window.location.href = rootUrl + targetUri
+                  }
+                });
+            })
+    }
+</script>
diff --git a/cccl_upstream/docs/404_helper.rst b/cccl_upstream/docs/404_helper.rst
new file mode 100644
index 00000000..a296dac3
--- /dev/null
+++ b/cccl_upstream/docs/404_helper.rst
@@ -0,0 +1,9 @@
+:orphan:
+
+404
+===
+
+The page you're seeking could not be found. Here is a list of similar pages:
+
+.. raw:: html
+  :file: 404_helper.inc.html
diff --git a/cccl_upstream/docs/VERSION.md b/cccl_upstream/docs/VERSION.md
new file mode 100644
index 00000000..d70c8f8d
--- /dev/null
+++ b/cccl_upstream/docs/VERSION.md
@@ -0,0 +1 @@
+3.6
diff --git a/cccl_upstream/docs/_ext/__init__.py b/cccl_upstream/docs/_ext/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/cccl_upstream/docs/_ext/auto_api_generator.py b/cccl_upstream/docs/_ext/auto_api_generator.py
new file mode 100644
index 00000000..11598b0a
--- /dev/null
+++ b/cccl_upstream/docs/_ext/auto_api_generator.py
@@ -0,0 +1,1453 @@
+"""
+Sphinx extension to automatically generate API reference pages from Doxygen XML.
+Replicates repo-docs' automatic API generation functionality.
+"""
+
+import os
+import xml.etree.ElementTree as ET
+from pathlib import Path
+
+from sphinx.application import Sphinx
+from sphinx.util import logging
+
+logger = logging.getLogger(__name__)
+
+# Symbols whose Doxygen XML produces C++ declarations that breathe/Sphinx
+# cannot parse.  Skip generating individual API pages for these to avoid
+# unfixable build warnings under -W.  Names must match the unqualified form
+# used by extract_namespace_items (i.e. without the project namespace prefix).
+_BREATHE_SKIP_SYMBOLS = frozenset(
+    {
+        # cuda::experimental::stf::get_executor_func_t — function-pointer typedef.
+        # Breathe renders as "pos4(*)(pos4, dim4, dim4)" which Sphinx's C++ parser
+        # rejects ("Expected end of definition").
+        # Listed both qualified (cudax) and unqualified (libcudacxx) since the
+        # generator uses different naming conventions per project.
+        "get_executor_func_t",
+        "cuda::experimental::stf::get_executor_func_t",
+        # cuda::experimental::stf::partition_fn_t — same issue as get_executor_func_t.
+        # Function-pointer typedef that Breathe renders in a form Sphinx rejects.
+        "partition_fn_t",
+        "cuda::experimental::stf::partition_fn_t",
+        # cuda::property_with_value — variable template using _CCCL_REQUIRES_EXPR
+        # with a typename(...) inside the expression.  Sphinx cannot parse the
+        # requires-expression expansion.
+        "property_with_value",
+        # cuda::has_property — variable template with _CCCL_REQUIRES_EXPR
+        # containing a const keyword inside the expression body.
+        "has_property",
+        # cuda::experimental::group — variable with complex type expression
+        # that Sphinx's C++ parser cannot handle.
+        "cuda::experimental::group",
+    }
+)
+
+
+def extract_param_summary(params):
+    """Extract a simplified parameter summary for section headers."""
+    if not params:
+        return ""
+
+    # Remove template details and namespaces for brevity
+    params = params.strip()
+    if params.startswith("(") and params.endswith(")"):
+        params = params[1:-1]
+
+    # If empty after removing parentheses
+    if not params.strip():
+        return ""
+
+    # Split by comma (handling nested templates/parentheses)
+    param_parts = []
+    depth = 0
+    current = []
+
+    for char in params:
+        if char in "<([":
+            depth += 1
+        elif char in ">)]":
+            depth -= 1
+        elif char == "," and depth == 0:
+            param_parts.append("".join(current).strip())
+            current = []
+            continue
+        current.append(char)
+
+    if current:
+        param_parts.append("".join(current).strip())
+
+    # Extract parameter names
+    param_names = []
+
+    for param in param_parts:
+        param = param.strip()
+
+        # Special case for execution policy - shorten for readability
+        if "execution_policy_base" in param:
+            param_names.append("exec")
+        else:
+            # Split by spaces and find the parameter name (typically the last word)
+            words = param.split()
+            if words:
+                # The parameter name is the last word
+                param_name = words[-1]
+                # Clean up reference/pointer markers
+                param_name = param_name.strip("&*,")
+
+                # Just use the parameter name as-is
+                if param_name:
+                    param_names.append(param_name)
+
+    return ", ".join(param_names)
+
+
+def extract_function_signatures(func_name, refids, xml_dir, namespace=""):
+    """Extract exact function signatures from Doxygen XML for overloaded functions."""
+    signatures = []
+    xml_path = Path(xml_dir)
+
+    if not xml_path.exists():
+        return signatures
+
+    # Extract the simple function name (without namespace) for comparison
+    simple_func_name = func_name.split("::")[-1] if "::" in func_name else func_name
+
+    # Parse both namespace and group XML files to get function signatures
+    # Functions can be defined in either location (often in group_*.xml for thrust/cub)
+    xml_files = list(xml_path.glob("namespace*.xml")) + list(
+        xml_path.glob("group*.xml")
+    )
+
+    for xml_file in xml_files:
+        try:
+            tree = ET.parse(xml_file)
+            root = tree.getroot()
+
+            # Find all function members with matching refids
+            for memberdef in root.findall('.//memberdef[@kind="function"]'):
+                member_refid = memberdef.get("id")
+                if member_refid in refids:
+                    # Get the function name
+                    name_elem = memberdef.find("name")
+                    if name_elem is not None and name_elem.text == simple_func_name:
+                        # Get the exact args string and definition
+                        argsstring_elem = memberdef.find("argsstring")
+                        definition_elem = memberdef.find("definition")
+
+                        if argsstring_elem is not None and argsstring_elem.text:
+                            # Get the full qualified name from definition if available
+                            if definition_elem is not None and definition_elem.text:
+                                # Definition includes namespace and return type
+                                # Extract just the qualified function name
+                                definition = definition_elem.text
+                                # Look for the function name in the definition
+                                if (
+                                    "::" in definition
+                                    and simple_func_name in definition
+                                ):
+                                    # Extract namespace::function from definition
+                                    parts = definition.split()
+                                    for part in parts:
+                                        if simple_func_name in part and "::" in part:
+                                            qualified_name = part
+                                            break
+                                    else:
+                                        # Use the original func_name if it has namespace, otherwise add namespace
+                                        qualified_name = (
+                                            func_name
+                                            if "::" in func_name
+                                            else (
+                                                f"{namespace}::{func_name}"
+                                                if namespace
+                                                else func_name
+                                            )
+                                        )
+                                else:
+                                    # Use the original func_name if it has namespace, otherwise add namespace
+                                    qualified_name = (
+                                        func_name
+                                        if "::" in func_name
+                                        else (
+                                            f"{namespace}::{func_name}"
+                                            if namespace
+                                            else func_name
+                                        )
+                                    )
+                            else:
+                                # Use the original func_name if it has namespace, otherwise add namespace
+                                qualified_name = (
+                                    func_name
+                                    if "::" in func_name
+                                    else (
+                                        f"{namespace}::{func_name}"
+                                        if namespace
+                                        else func_name
+                                    )
+                                )
+
+                            # Build the complete signature for breathe
+                            full_signature = (
+                                qualified_name + argsstring_elem.text.strip()
+                            )
+                            # Check if this signature is already in the list (avoid duplicates)
+                            if not any(sig[1] == full_signature for sig in signatures):
+                                signatures.append((member_refid, full_signature))
+        except Exception as e:
+            logger.debug(f"Failed to extract signatures from {xml_file}: {e}")
+
+    return signatures
+
+
+def extract_doxygen_items(xml_dir):
+    """Extract all items (classes, structs, functions, etc.) from Doxygen XML."""
+    items = {
+        "classes": [],
+        "structs": [],
+        "functions": [],
+        "typedefs": [],
+        "enums": [],
+        "variables": [],
+        "macros": [],
+        "function_groups": {},  # Group functions by name for overloads
+        "groups": [],  # Doxygen groups
+    }
+
+    xml_path = Path(xml_dir)
+    if not xml_path.exists():
+        return items
+
+    # Parse index.xml to get all compounds and members
+    index_file = xml_path / "index.xml"
+    if not index_file.exists():
+        return items
+
+    try:
+        tree = ET.parse(index_file)
+        root = tree.getroot()
+
+        # Get the namespace compound (e.g., cub, thrust, cuda::experimental)
+        namespace_compounds = []
+        for compound in root.findall('.//compound[@kind="namespace"]'):
+            name = compound.find("name").text
+            # Match primary namespaces and nested namespaces for cudax
+            if name in ["cub", "thrust", "cuda"] or name.startswith(
+                "cuda::experimental"
+            ):
+                namespace_compounds.append(compound)
+
+        # Extract classes and structs
+        all_classes = []
+        all_structs = []
+
+        for compound in root.findall('.//compound[@kind="class"]'):
+            name = compound.find("name").text
+            refid = compound.get("refid")
+
+            # Skip internal/detail classes
+            if "detail" in name.lower() or "__" in name:
+                continue
+
+            all_classes.append((name, refid))
+
+        for compound in root.findall('.//compound[@kind="struct"]'):
+            name = compound.find("name").text
+            refid = compound.get("refid")
+
+            # Skip internal/detail structs
+            if "detail" in name.lower() or "__" in name:
+                continue
+
+            all_structs.append((name, refid))
+
+        # Filter out nested classes/structs when their parent is also documented
+        # This prevents duplicate declarations in Sphinx
+        def is_nested_and_parent_exists(name, all_classes, all_structs):
+            """Check if this is a nested class/struct and its parent is also documented."""
+            if "::" not in name:
+                return False
+
+            # Get the parent name by removing the last component
+            parent_name = "::".join(name.split("::")[:-1])
+
+            # Check if parent exists in either classes or structs list
+            all_items = all_classes + all_structs
+            for item_name, _ in all_items:
+                if item_name == parent_name:
+                    return True
+            return False
+
+        # Filter classes (check against both classes and structs for parents)
+        for name, refid in all_classes:
+            if not is_nested_and_parent_exists(name, all_classes, all_structs):
+                items["classes"].append((name, refid))
+
+        # Filter structs (check against both classes and structs for parents)
+        for name, refid in all_structs:
+            if not is_nested_and_parent_exists(name, all_classes, all_structs):
+                items["structs"].append((name, refid))
+
+        # Extract groups and their members
+        for compound in root.findall('.//compound[@kind="group"]'):
+            name = compound.find("name").text
+            refid = compound.get("refid")
+            items["groups"].append((name, refid))
+
+            # Also extract typedefs that are members of groups
+            # We need to get the qualified name from the actual group XML file
+            for member in compound.findall('member[@kind="typedef"]'):
+                name_elem = member.find("name")
+                if name_elem is None:
+                    continue
+                simple_name = name_elem.text
+                typedef_refid = member.get("refid")
+
+                # Try to get the qualified name from the group XML file
+                qualified_name = simple_name  # Default to simple name
+                group_xml_file = xml_path / f"{refid}.xml"
+                if group_xml_file.exists():
+                    try:
+                        group_tree = ET.parse(group_xml_file)
+                        group_root = group_tree.getroot()
+                        # Find the typedef with matching refid
+                        typedef_elem = group_root.find(
+                            f'.//memberdef[@id="{typedef_refid}"]'
+                        )
+                        if typedef_elem is not None:
+                            qualifiedname_elem = typedef_elem.find("qualifiedname")
+                            if (
+                                qualifiedname_elem is not None
+                                and qualifiedname_elem.text
+                            ):
+                                qualified_name = qualifiedname_elem.text
+                    except Exception:
+                        pass
+
+                items["typedefs"].append((qualified_name, typedef_refid))
+
+        # Extract documented macros from file compounds. Doxygen records macros as
+        # file-level "define" members, not namespace members.
+        seen_macros = set()
+        for compound in root.findall('.//compound[@kind="file"]'):
+            refid = compound.get("refid")
+            if not refid:
+                continue
+
+            file_xml_file = xml_path / f"{refid}.xml"
+            if not file_xml_file.exists():
+                continue
+
+            try:
+                file_tree = ET.parse(file_xml_file)
+                file_root = file_tree.getroot()
+                for memberdef in file_root.findall('.//memberdef[@kind="define"]'):
+                    macro_refid = memberdef.get("id")
+                    name_elem = memberdef.find("name")
+                    if macro_refid is None or name_elem is None or not name_elem.text:
+                        continue
+
+                    has_description = any(
+                        "".join(description.itertext()).strip()
+                        for description in (
+                            memberdef.find("briefdescription"),
+                            memberdef.find("detaileddescription"),
+                        )
+                        if description is not None
+                    )
+                    if not has_description:
+                        continue
+
+                    macro_key = (name_elem.text, macro_refid)
+                    if macro_key in seen_macros:
+                        continue
+
+                    seen_macros.add(macro_key)
+                    items["macros"].append((name_elem.text, macro_refid))
+            except Exception as e:
+                logger.debug(f"Failed to parse macros from {file_xml_file}: {e}")
+
+        # Extract functions, typedefs, enums, and variables from namespaces
+        for namespace_compound in namespace_compounds:
+            namespace_name = namespace_compound.find("name").text
+
+            for member in namespace_compound.findall('member[@kind="function"]'):
+                name = member.find("name").text
+                refid = member.get("refid")
+                # Only include full namespace for nested namespaces (cudax)
+                # For simple namespaces like 'thrust', 'cub', just use the function name
+                if namespace_name and "::" in namespace_name:
+                    full_name = f"{namespace_name}::{name}"
+                else:
+                    full_name = name
+                items["functions"].append((full_name, refid))
+
+                # Also track function groups for overloads
+                if full_name not in items["function_groups"]:
+                    items["function_groups"][full_name] = []
+                items["function_groups"][full_name].append(refid)
+
+            for member in namespace_compound.findall('member[@kind="typedef"]'):
+                name = member.find("name").text
+                refid = member.get("refid")
+                # Only include full namespace for nested namespaces
+                if namespace_name and "::" in namespace_name:
+                    full_name = f"{namespace_name}::{name}"
+                else:
+                    full_name = name
+                items["typedefs"].append((full_name, refid))
+
+            for member in namespace_compound.findall('member[@kind="enum"]'):
+                name = member.find("name").text
+                refid = member.get("refid")
+                # Only include full namespace for nested namespaces
+                if namespace_name and "::" in namespace_name:
+                    full_name = f"{namespace_name}::{name}"
+                else:
+                    full_name = name
+                items["enums"].append((full_name, refid))
+
+            for member in namespace_compound.findall('member[@kind="variable"]'):
+                name = member.find("name").text
+                refid = member.get("refid")
+                # Only include full namespace for nested namespaces
+                if namespace_name and "::" in namespace_name:
+                    full_name = f"{namespace_name}::{name}"
+                else:
+                    full_name = name
+                items["variables"].append((full_name, refid))
+
+    except Exception as e:
+        logger.warning(f"Failed to parse Doxygen XML: {e}")
+
+    return items
+
+
+# Unlike other normal symbols, doxygen treats them as floating in the global namespace
+# (which, technically, is correct). As such, they aren't namespaced, and the name of the
+# generated files would just be <name of file containing macro>_<refid>, which is nearly
+# impossible to name in hand-written rst code.
+#
+# We therefore perform special mangling of the generated file name and prepend this prefix
+# to it.
+MACRO_FILE_NAME_PREFIX = "macro_"
+
+
+def format_macro_doc_filename(macro_name):
+    """Create a stable filename for a generated macro page. Macros are not namespace
+    members so their refid's (and generated pages) are just the name of the file and a
+    hash. This makes it impossible to refer to macros in hand-written rst code."""
+    macro_name = macro_name.casefold().replace(" ", "_")
+    return f"{MACRO_FILE_NAME_PREFIX}{macro_name}"
+
+
+def extract_doxygen_classes(xml_dir, project_name=None):
+    """Extract classes categorized by type for category pages."""
+    # Only CUB uses these specific categories
+    if project_name == "cub":
+        classes = {
+            "device": [],
+            "block": [],
+            "warp": [],
+            "grid": [],
+            "iterator": [],
+            "thread": [],
+            "utility": [],
+        }
+    else:
+        # Other projects don't need category pages
+        classes = {}
+
+    # Only categorize for CUB
+    if project_name == "cub":
+        items = extract_doxygen_items(xml_dir)
+
+        # Filter out internal implementation details
+        internal_patterns = [
+            "StoreInternal",
+            "LoadInternal",
+            "_TempStorage",
+            "TileDescriptor",
+        ]
+
+        # Categorize classes and structs
+        for name, refid in items["classes"] + items["structs"]:
+            # Skip internal implementation details
+            if any(pattern in name for pattern in internal_patterns):
+                continue
+
+            # Remove namespace prefixes for categorization
+            simple_name = name.split("::")[-1] if "::" in name else name
+
+            # Categorize based on name
+            if "Device" in simple_name:
+                classes["device"].append((name, refid))
+            elif "Block" in simple_name:
+                classes["block"].append((name, refid))
+            elif "Warp" in simple_name:
+                classes["warp"].append((name, refid))
+            elif "Grid" in simple_name:
+                classes["grid"].append((name, refid))
+            elif "Iterator" in simple_name.lower() or "iterator" in simple_name.lower():
+                classes["iterator"].append((name, refid))
+            elif any(
+                x in simple_name
+                for x in ["Traits", "Type", "Allocator", "Debug", "Caching"]
+            ):
+                classes["utility"].append((name, refid))
+
+    return classes
+
+
+def generate_api_page(category, classes, project_name):
+    """Generate RST content for an API category page."""
+
+    category_titles = {
+        "device": "Device-wide Primitives",
+        "block": "Block-wide Primitives",
+        "warp": "Warp-wide Primitives",
+        "grid": "Grid-level Primitives",
+        "iterator": "Iterator Utilities",
+        "thread": "Thread-level Primitives",
+        "utility": "Utility Components",
+    }
+
+    content = []
+    content.append(category_titles.get(category, f"{category.title()} API"))
+    content.append("=" * len(content[0]))
+    content.append("")
+    content.append(".. contents:: Table of Contents")
+    content.append("   :local:")
+    content.append("   :depth: 2")
+    content.append("")
+
+    # Sort classes by name
+    classes.sort(key=lambda x: x[0])
+
+    for class_name, refid in classes:
+        # Use the full name including namespace for display
+        display_name = class_name
+
+        content.append(display_name)
+        content.append("-" * len(display_name))
+        content.append("")
+
+        # Check if this is a struct by looking at the refid
+        # Doxygen uses 'struct' prefix in the refid for structs
+        directive = "doxygenstruct" if refid.startswith("struct") else "doxygenclass"
+
+        # Use the full qualified name including namespace
+        content.append(f".. {directive}:: {class_name}")
+
+        content.append(f"   :project: {project_name}")
+        content.append("   :members:")
+        content.append("   :undoc-members:")
+        content.append("")
+
+    return "\n".join(content)
+
+
+def generate_group_index_page(group_name, group_refid, project_name, xml_dir, api_dir):
+    """Generate RST content for a Doxygen group index page."""
+    content = []
+
+    # Add marker comment for auto-generated files
+    content.append(".. AUTO-GENERATED by auto_api_generator.py - DO NOT EDIT")
+    content.append("")
+    content.append(":orphan:")
+    content.append("")
+
+    # Parse the group XML to get more details
+    group_xml_file = Path(xml_dir) / f"{group_refid}.xml"
+    title = group_name
+    members = []
+    brief_description = ""
+
+    if group_xml_file.exists():
+        try:
+            tree = ET.parse(group_xml_file)
+            root = tree.getroot()
+            compounddef = root.find('.//compounddef[@kind="group"]')
+
+            if compounddef is not None:
+                # Get the title
+                title_elem = compounddef.find("title")
+                if title_elem is not None and title_elem.text:
+                    title = title_elem.text
+
+                # Get brief description if available
+                brief_elem = compounddef.find("briefdescription")
+                if brief_elem is not None:
+                    # Extract text from brief description
+                    brief_text = "".join(brief_elem.itertext()).strip()
+                    if brief_text:
+                        brief_description = brief_text
+
+                # Get all inner classes/structs
+                for innerclass in compounddef.findall("innerclass"):
+                    class_refid = innerclass.get("refid")
+                    class_name = innerclass.text
+                    members.append(("class", class_name, class_refid))
+
+                # Get all member functions, typedefs, variables, etc.
+                for sectiondef in compounddef.findall("sectiondef"):
+                    for memberdef in sectiondef.findall("memberdef"):
+                        member_name = memberdef.find("name")
+                        if member_name is None:
+                            continue
+                        member_kind = memberdef.get("kind")
+                        member_refid = memberdef.get("id")
+                        member_filename = (
+                            format_macro_doc_filename(member_name.text)
+                            if member_kind == "define"
+                            else member_refid
+                        )
+                        member_rst_file = Path(api_dir) / f"{member_filename}.rst"
+                        # only add refids for exists pages.
+                        # Refids corresponding to overloads
+                        # do not have associated RST file
+                        if not member_rst_file.exists():
+                            continue
+                        members.append((member_kind, member_name.text, member_filename))
+        except Exception as e:
+            logger.warning(f"Failed to parse group XML {group_xml_file}: {e}")
+
+    # Add title
+    content.append(title)
+    content.append("=" * len(title))
+    content.append("")
+
+    # Add brief description if available
+    if brief_description:
+        content.append(brief_description)
+        content.append("")
+
+    # Do NOT use doxygengroup directive to avoid duplicate declarations
+    # Instead, just provide a simple page with links to members
+
+    # Provide a list of links instead of a toctree to avoid duplicate-toctree warnings.
+    if members:
+        content.append("Members")
+        content.append("-------")
+        content.append("")
+        for member_kind, member_name, member_refid in members:
+            content.append(
+                format_doc_reference(member_name, member_refid, "", as_list_item=True)
+            )
+        content.append("")
+
+    return "\n".join(content)
+
+
+def generate_individual_api_page(class_name, refid, project_name):
+    """Generate RST content for a single class/struct API page."""
+    content = []
+
+    # Add marker comment for auto-generated files
+    content.append(".. AUTO-GENERATED by auto_api_generator.py - DO NOT EDIT")
+    content.append("")
+    content.append(":orphan:")
+    content.append("")
+
+    # Add title
+    content.append(class_name)
+    content.append("=" * len(class_name))
+    content.append("")
+
+    # Check if this is a struct by looking at the refid
+    directive = "doxygenstruct" if refid.startswith("struct") else "doxygenclass"
+
+    # Add the doxygen directive
+    content.append(f".. {directive}:: {class_name}")
+    content.append(f"   :project: {project_name}")
+    content.append("   :members:")
+    content.append("   :undoc-members:")
+    content.append("")
+
+    return "\n".join(content)
+
+
+def check_function_in_namespace(member_name, xml_dir, namespace):
+    """Check if a function is defined in the namespace XML (not just referenced)."""
+    namespace_xml = os.path.join(xml_dir, f"namespace{namespace}.xml")
+    if not os.path.exists(namespace_xml):
+        return False
+
+    try:
+        tree = ET.parse(namespace_xml)
+        root = tree.getroot()
+
+        # Look for actual function definitions, not just references
+        for memberdef in root.findall('.//memberdef[@kind="function"]'):
+            name_elem = memberdef.find("name")
+            if name_elem is not None and name_elem.text == member_name:
+                # Check if it has a definition (not just a reference)
+                definition = memberdef.find("definition")
+                if definition is not None and definition.text:
+                    return True
+        return False
+    except Exception:
+        return False
+
+
+def generate_member_api_page(
+    member_name,
+    member_type,
+    project_name,
+    refid=None,
+    overload_refids=None,
+    xml_dir=None,
+):
+    """Generate RST content for a single function/typedef/enum/variable/macro API page."""
+    content = []
+
+    # Add marker comment for auto-generated files
+    content.append(".. AUTO-GENERATED by auto_api_generator.py - DO NOT EDIT")
+    content.append("")
+    content.append(":orphan:")
+    content.append("")
+
+    # Map member types to Doxygen directives
+    directive_map = {
+        "function": "doxygenfunction",
+        "typedef": "doxygentypedef",
+        "enum": "doxygenenum",
+        "variable": "doxygenvariable",
+        "macro": "doxygendefine",
+    }
+
+    directive = directive_map.get(member_type, "doxygenfunction")
+
+    # For thrust and cub, we need to use the namespace-qualified name
+    # For cudax, the member_name already includes the namespace
+    if member_type == "macro":
+        qualified_name = member_name
+    elif project_name in ["thrust", "cub"]:
+        # If the member_name doesn't already include the namespace, add it
+        if "::" not in member_name:
+            qualified_name = f"{project_name}::{member_name}"
+        else:
+            qualified_name = member_name
+    else:
+        qualified_name = member_name
+
+    # Add title
+    content.append(f"{qualified_name}")
+    content.append("=" * (len(qualified_name) + 4))
+    content.append("")
+
+    if member_type == "function" and overload_refids:
+        # Check if functions are in a group
+        is_group_function = False
+        group_name = None
+
+        if overload_refids and "_1" in overload_refids[0]:
+            parts = overload_refids[0].split("_1")
+            if parts[0].startswith("group__"):
+                is_group_function = True
+                # Get the group refid (e.g., 'group__stream__compaction')
+                group_refid = parts[0]
+
+                # Look up the actual group name from index.xml
+                group_name = None
+                if xml_dir:
+                    index_xml = os.path.join(xml_dir, "index.xml")
+                    if os.path.exists(index_xml):
+                        try:
+                            tree = ET.parse(index_xml)
+                            root = tree.getroot()
+                            # Find the compound with this refid
+                            compound = root.find(f'.//compound[@refid="{group_refid}"]')
+                            if compound is not None:
+                                name_elem = compound.find("name")
+                                if name_elem is not None:
+                                    group_name = name_elem.text
+                        except Exception:
+                            pass
+
+                # Fallback: remove 'group__' prefix if lookup fails
+                if not group_name:
+                    group_name = group_refid[7:]
+
+        if is_group_function and group_name:
+            # For group functions with overloads, we need to handle them specially
+            if len(overload_refids) > 1 and xml_dir:
+                # Extract signatures for all overloads
+                signatures = extract_function_signatures(
+                    member_name, overload_refids, xml_dir, namespace=project_name
+                )
+
+                if signatures:
+                    content.append("Overloads")
+                    content.append("---------")
+                    content.append("")
+
+                    for idx, (refid, full_sig) in enumerate(signatures, 1):
+                        # Extract just the parameter list from the full signature
+                        simple_name = (
+                            member_name.split("::")[-1]
+                            if "::" in member_name
+                            else member_name
+                        )
+                        if simple_name in full_sig:
+                            # Use find instead of rfind to get the first occurrence
+                            sig_idx = full_sig.find(simple_name)
+                            if sig_idx != -1:
+                                params = full_sig[sig_idx + len(simple_name) :].strip()
+                                # Handle trailing noexcept specifier
+                                # Look for the closing parenthesis of the parameter list
+                                paren_count = 0
+                                param_end = -1
+                                for i, char in enumerate(params):
+                                    if char == "(":
+                                        paren_count += 1
+                                    elif char == ")":
+                                        paren_count -= 1
+                                        if paren_count == 0:
+                                            param_end = i + 1
+                                            break
+                                if param_end > 0:
+                                    # Keep only the parameter list (up to and including the closing parenthesis)
+                                    params = params[:param_end]
+
+                                # Create a simplified signature for the section header
+                                param_summary = extract_param_summary(params)
+
+                                # Add a section header for this overload
+                                content.append(f"``{simple_name}({param_summary})``")
+                                content.append(
+                                    "^" * (len(simple_name) + len(param_summary) + 6)
+                                )
+                                content.append("")
+
+                                # Use doxygenfunction with the specific parameter signature
+                                content.append(
+                                    f".. doxygenfunction:: {qualified_name}{params}"
+                                )
+                                content.append(f"   :project: {project_name}")
+                                content.append("")
+                else:
+                    # Fallback to using doxygengroup if we can't extract signatures
+                    content.append(f".. doxygengroup:: {group_name}")
+                    content.append(f"   :project: {project_name}")
+                    content.append("   :content-only:")
+                    content.append("")
+            else:
+                # Single overload from group - but there might be other overloads in the namespace
+                # Extract the signature to be specific
+                if xml_dir:
+                    signatures = extract_function_signatures(
+                        member_name, overload_refids, xml_dir, namespace=project_name
+                    )
+                    if signatures and len(signatures) == 1:
+                        refid, full_sig = signatures[0]
+                        simple_name = (
+                            member_name.split("::")[-1]
+                            if "::" in member_name
+                            else member_name
+                        )
+                        if simple_name in full_sig:
+                            # Use find instead of rfind to get the first occurrence
+                            sig_idx = full_sig.find(simple_name)
+                            if sig_idx != -1:
+                                params = full_sig[sig_idx + len(simple_name) :].strip()
+                                # Handle trailing noexcept specifier
+                                # Look for the closing parenthesis of the parameter list
+                                paren_count = 0
+                                param_end = -1
+                                for i, char in enumerate(params):
+                                    if char == "(":
+                                        paren_count += 1
+                                    elif char == ")":
+                                        paren_count -= 1
+                                        if paren_count == 0:
+                                            param_end = i + 1
+                                            break
+                                if param_end > 0:
+                                    # Keep only the parameter list (up to and including the closing parenthesis)
+                                    params = params[:param_end]
+                                content.append(
+                                    f".. doxygenfunction:: {qualified_name}{params}"
+                                )
+                                content.append(f"   :project: {project_name}")
+                                content.append("")
+                            else:
+                                # Fallback to simple
+                                content.append(f".. doxygenfunction:: {qualified_name}")
+                                content.append(f"   :project: {project_name}")
+                                content.append("")
+                        else:
+                            # Fallback to simple
+                            content.append(f".. doxygenfunction:: {qualified_name}")
+                            content.append(f"   :project: {project_name}")
+                            content.append("")
+                    else:
+                        # Fallback to simple
+                        content.append(f".. doxygenfunction:: {qualified_name}")
+                        content.append(f"   :project: {project_name}")
+                        content.append("")
+                else:
+                    # No xml_dir, use simple
+                    content.append(f".. doxygenfunction:: {qualified_name}")
+                    content.append(f"   :project: {project_name}")
+                    content.append("")
+        elif len(overload_refids) > 1 and xml_dir:
+            # For functions with multiple overloads in namespace, extract signatures
+            # For cudax, member_name already includes namespace, for others use project_name
+            if "::" in member_name:
+                # Extract namespace from the qualified name for cudax
+                namespace_parts = member_name.split("::")[:-1]
+                namespace_name = (
+                    "::".join(namespace_parts) if namespace_parts else project_name
+                )
+            else:
+                namespace_name = project_name
+            signatures = extract_function_signatures(
+                member_name, overload_refids, xml_dir, namespace=namespace_name
+            )
+
+            if signatures:
+                content.append("Overloads")
+                content.append("---------")
+                content.append("")
+
+                for idx, (refid, full_sig) in enumerate(signatures, 1):
+                    # Extract just the parameter list from the full signature
+                    # Look for the simple function name (without namespace) in the signature
+                    simple_name = (
+                        member_name.split("::")[-1]
+                        if "::" in member_name
+                        else member_name
+                    )
+                    if simple_name in full_sig:
+                        sig_idx = full_sig.rfind(simple_name)
+                        if sig_idx != -1:
+                            params = full_sig[sig_idx + len(simple_name) :].strip()
+
+                            # Create a simplified signature for the section header
+                            # Extract key parameter types for identification
+                            param_summary = extract_param_summary(params)
+
+                            # Add a section header for this overload
+                            # Use simple name for readability in headers
+                            content.append(f"``{simple_name}({param_summary})``")
+                            content.append(
+                                "^" * (len(simple_name) + len(param_summary) + 6)
+                            )
+                            content.append("")
+
+                            # Use doxygenfunction with the specific parameter signature and qualified name
+                            content.append(
+                                f".. doxygenfunction:: {qualified_name}{params}"
+                            )
+                            content.append(f"   :project: {project_name}")
+                            content.append("   :no-link:")
+                            content.append("")
+            else:
+                # Fallback to simple directive with qualified name
+                content.append(f".. {directive}:: {qualified_name}")
+                content.append(f"   :project: {project_name}")
+                content.append("")
+        else:
+            # Single function with qualified name
+            content.append(f".. {directive}:: {qualified_name}")
+            content.append(f"   :project: {project_name}")
+            content.append("")
+    elif member_type == "function":
+        # For single functions or when we don't have xml_dir, use qualified name
+        content.append(f".. {directive}:: {qualified_name}")
+        content.append(f"   :project: {project_name}")
+        content.append("")
+    else:
+        # For other types, use the qualified name
+        content.append(f".. {directive}:: {qualified_name}")
+        content.append(f"   :project: {project_name}")
+        content.append("")
+
+    return "\n".join(content)
+
+
+def generate_category_index(category, class_list, project_name):
+    """Generate an index page for a category with links to individual class pages."""
+    category_titles = {
+        "device": "Device-wide Primitives",
+        "block": "Block-wide Primitives",
+        "warp": "Warp-wide Primitives",
+        "grid": "Grid-level Primitives",
+        "iterator": "Iterator Utilities",
+        "thread": "Thread-level Primitives",
+        "utility": "Utility Components",
+    }
+
+    content = []
+
+    # Add marker comment for auto-generated files
+    content.append(".. AUTO-GENERATED by auto_api_generator.py - DO NOT EDIT")
+    content.append("")
+    content.append(":orphan:")
+    content.append("")
+
+    title = category_titles.get(category, f"{category.title()} API")
+    content.append(title)
+    content.append("=" * len(title))
+    content.append("")
+
+    # Add toctree for all classes in this category
+    content.append(".. toctree::")
+    content.append("   :maxdepth: 1")
+    content.append("   :hidden:")
+    content.append("")
+
+    # Sort classes by name
+    class_list.sort(key=lambda x: x[0])
+
+    for class_name, refid in class_list:
+        # Generate filename from refid (e.g., structcub_1_1DeviceAdjacentDifference)
+        filename = refid
+        content.append(f"   {filename}")
+
+    content.append("")
+    content.append(".. list-table::")
+    content.append("   :widths: 50 50")
+    content.append("   :header-rows: 1")
+    content.append("")
+    content.append("   * - Class/Struct")
+    content.append("     - Description")
+
+    for class_name, refid in class_list:
+        filename = refid
+        # Use format_doc_reference but without the list item marker
+        doc_ref = format_doc_reference(class_name, filename, "", as_list_item=False)
+        content.append(f"   * - {doc_ref}")
+        content.append("     - ")  # Description would go here if available
+
+    return "\n".join(content)
+
+
+def clean_template_name(name):
+    """Remove spaces around template parameters to avoid Sphinx parsing issues."""
+    # Remove spaces after '<' and before '>'
+    cleaned = name.replace("< ", "<").replace(" >", ">")
+    # Handle pointer types - remove space before *
+    cleaned = cleaned.replace(" *", "*")
+    # Handle spaces before commas in template parameters
+    while " ," in cleaned:
+        cleaned = cleaned.replace(" ,", ",")
+    # Handle spaces after commas in template parameters
+    while ", " in cleaned:
+        cleaned = cleaned.replace(", ", ",")
+    # Handle spaces after :: in nested namespaces
+    cleaned = cleaned.replace(":: ", "::")
+    return cleaned
+
+
+def format_doc_reference(name, refid, doc_prefix="", as_list_item=True):
+    """Format a documentation reference, handling template specializations."""
+    clean_name = clean_template_name(name)
+
+    # Build the reference path
+    ref_path = f"{doc_prefix}{refid}" if doc_prefix else refid
+
+    # Check if the name contains any angle brackets (template)
+    if "<" in clean_name:
+        # For any template specialization, just use the file reference without display name
+        # This avoids RST parsing issues with angle brackets in the display name
+        doc_ref = f":doc:`{ref_path}`"
+    else:
+        # For non-templates, use the full name as display
+        doc_ref = f":doc:`{clean_name} <{ref_path}>`"
+
+    # Return with or without list item marker
+    return f"* {doc_ref}" if as_list_item else doc_ref
+
+
+def generate_namespace_api_page(project_name, items, title=None, doc_prefix=""):
+    """Generate a comprehensive namespace API reference page."""
+    content = []
+
+    # Add marker comment for auto-generated files
+    content.append(".. AUTO-GENERATED by auto_api_generator.py - DO NOT EDIT")
+    content.append("")
+
+    # Determine namespace name
+    namespace_name = project_name  # e.g., 'cub', 'thrust', etc.
+
+    # Title - use provided title or default
+    if not title:
+        title = f"{project_name.upper()} API Reference"
+
+    content.append(title)
+    content.append("=" * len(title))
+    content.append("")
+
+    # Add namespace description
+    namespace_title = f"Namespace ``{namespace_name}``"
+    content.append(namespace_title)
+    # Make underline slightly longer to avoid "too short" warnings
+    content.append("-" * (len(namespace_title) + 2))
+    content.append("")
+
+    # Filter out internal implementation details
+    internal_patterns = [
+        "StoreInternal",
+        "LoadInternal",
+        "_TempStorage",
+        "TileDescriptor",
+    ]
+
+    # Classes section
+    filtered_classes = [
+        (name, refid)
+        for name, refid in items["classes"]
+        if not any(pattern in name for pattern in internal_patterns)
+    ]
+    if filtered_classes:
+        content.append("Classes")
+        content.append("~~~~~~~")
+        content.append("")
+
+        # Sort classes alphabetically
+        filtered_classes.sort(key=lambda x: x[0].lower())
+        for name, refid in filtered_classes:
+            content.append(format_doc_reference(name, refid, doc_prefix))
+        content.append("")
+
+    # Structs section
+    filtered_structs = [
+        (name, refid)
+        for name, refid in items["structs"]
+        if not any(pattern in name for pattern in internal_patterns)
+    ]
+    if filtered_structs:
+        content.append("Structs")
+        content.append("~~~~~~~")
+        content.append("")
+
+        # Sort structs alphabetically
+        filtered_structs.sort(key=lambda x: x[0].lower())
+        for name, refid in filtered_structs:
+            content.append(format_doc_reference(name, refid, doc_prefix))
+        content.append("")
+
+    # Functions section
+    if items.get("function_groups"):
+        content.append("Functions")
+        content.append("~~~~~~~~~")
+        content.append("")
+
+        # Sort functions alphabetically by name
+        sorted_functions = sorted(
+            items["function_groups"].keys(), key=lambda x: x.lower()
+        )
+        for func_name in sorted_functions:
+            # Use the first refid for the link
+            first_refid = items["function_groups"][func_name][0]
+            content.append(format_doc_reference(func_name, first_refid, doc_prefix))
+        content.append("")
+
+    # Typedefs section
+    if items["typedefs"]:
+        content.append("Type Definitions")
+        content.append("~~~~~~~~~~~~~~~~")
+        content.append("")
+
+        # Sort typedefs alphabetically
+        items["typedefs"].sort(key=lambda x: x[0].lower())
+        for name, refid in items["typedefs"]:
+            if name in _BREATHE_SKIP_SYMBOLS:
+                logger.info(
+                    f"Skipping typedef reference {name} (in _BREATHE_SKIP_SYMBOLS)"
+                )
+                continue
+            content.append(format_doc_reference(name, refid, doc_prefix))
+        content.append("")
+
+    # Enums section
+    if items["enums"]:
+        content.append("Enumerations")
+        content.append("~~~~~~~~~~~~")
+        content.append("")
+
+        # Sort enums alphabetically
+        items["enums"].sort(key=lambda x: x[0].lower())
+        for name, refid in items["enums"]:
+            content.append(format_doc_reference(name, refid, doc_prefix))
+        content.append("")
+
+    # Variables section
+    if items["variables"]:
+        content.append("Variables")
+        content.append("~~~~~~~~~")
+        content.append("")
+
+        # Sort variables alphabetically
+        items["variables"].sort(key=lambda x: x[0].lower())
+        for name, refid in items["variables"]:
+            if name in _BREATHE_SKIP_SYMBOLS:
+                logger.info(
+                    f"Skipping variable reference {name} (in _BREATHE_SKIP_SYMBOLS)"
+                )
+                continue
+            content.append(format_doc_reference(name, refid, doc_prefix))
+        content.append("")
+
+    # Macros section
+    if items["macros"]:
+        content.append("Macros")
+        content.append("~~~~~~")
+        content.append("")
+
+        # Sort macros alphabetically
+        items["macros"].sort(key=lambda x: x[0].lower())
+        for name, refid in items["macros"]:
+            if name in _BREATHE_SKIP_SYMBOLS:
+                logger.info(
+                    f"Skipping macro reference {name} (in _BREATHE_SKIP_SYMBOLS)"
+                )
+                continue
+            filename = format_macro_doc_filename(name)
+            content.append(format_doc_reference(name, filename, doc_prefix))
+        content.append("")
+
+    # Keep API reference pages as leaf nodes in sidebar navigation.
+    # Symbol/group/category pages are still discoverable on-page and via search.
+
+    return "\n".join(content)
+
+
+def generate_api_docs(app, config):
+    """Generate API documentation pages during Sphinx build."""
+
+    # Only generate for projects with breathe configuration
+    if not hasattr(config, "breathe_projects"):
+        return
+
+    for project_name, xml_dir in config.breathe_projects.items():
+        # Skip if XML directory doesn't exist
+        if not os.path.exists(xml_dir):
+            continue
+
+        # Extract all items from Doxygen XML
+        items = extract_doxygen_items(xml_dir)
+
+        # Also extract categorized classes for category pages
+        classes = extract_doxygen_classes(xml_dir, project_name)
+
+        # Determine output directory based on project
+        api_dir = None
+        if project_name == "cub":
+            api_dir = Path(app.srcdir) / "cub" / "api"
+        elif project_name == "thrust":
+            api_dir = Path(app.srcdir) / "thrust" / "api"
+        elif project_name == "libcudacxx":
+            api_dir = Path(app.srcdir) / "libcudacxx" / "api"
+        elif project_name == "cudax":
+            api_dir = Path(app.srcdir) / "cudax" / "api"
+
+        if not api_dir:
+            continue
+
+        # Clean up auto-generated files first
+        # Remove all auto-generated .rst files (those matching certain patterns)
+        if api_dir.exists():
+            logger.info(f"Cleaning up auto-generated files in {api_dir}")
+            # Remove files matching Doxygen reference patterns
+            for pattern in [
+                "class*.rst",
+                "struct*.rst",
+                "group*.rst",
+                "namespace*.rst",
+                f"{MACRO_FILE_NAME_PREFIX}*.rst",
+            ]:
+                for file in api_dir.glob(pattern):
+                    file.unlink()
+                    logger.debug(f"Removed {file}")
+
+            # Also remove category files that are auto-generated
+            for category in [
+                "device",
+                "block",
+                "warp",
+                "grid",
+                "iterator",
+                "thread",
+                "utility",
+            ]:
+                category_file = api_dir / f"{category}.rst"
+                if category_file.exists():
+                    # Check if it's auto-generated by looking for our marker
+                    try:
+                        with open(category_file, "r") as f:
+                            first_line = f.readline()
+                            if "AUTO-GENERATED" in first_line:
+                                category_file.unlink()
+                                logger.debug(f"Removed auto-generated {category_file}")
+                    except Exception:
+                        pass
+
+            # Remove index.rst if it's auto-generated
+            index_file = api_dir / "index.rst"
+            if index_file.exists():
+                try:
+                    with open(index_file, "r") as f:
+                        first_line = f.readline()
+                        if "AUTO-GENERATED" in first_line:
+                            index_file.unlink()
+                            logger.debug(f"Removed auto-generated {index_file}")
+                except Exception:
+                    pass
+
+        # Note: We no longer generate or clean up auto_api.rst files
+
+        # Create API directory if it doesn't exist
+        api_dir.mkdir(parents=True, exist_ok=True)
+
+        # Generate individual pages for each class/struct
+        # Skip internal implementation details that cause template warnings
+        internal_patterns = [
+            "StoreInternal",
+            "LoadInternal",
+            "_TempStorage",
+            "TileDescriptor",
+        ]
+
+        for name, refid in items["classes"] + items["structs"]:
+            # Skip internal implementation details
+            if any(pattern in name for pattern in internal_patterns):
+                logger.debug(f"Skipping internal implementation detail: {name}")
+                continue
+
+            # Note: We generate individual pages for all classes/structs,
+            # even if they're in groups, so that references work correctly
+
+            # Generate individual page
+            content = generate_individual_api_page(name, refid, project_name)
+            output_file = api_dir / f"{refid}.rst"
+
+            # Write the individual class page
+            with open(output_file, "w") as f:
+                f.write(content)
+            logger.info(f"Generated API page: {output_file}")
+
+        # Generate individual pages for functions (one per unique function name)
+        # Use function_groups to handle overloads
+        function_groups = items.get("function_groups", {})
+        for func_name in function_groups:
+            # Use the first refid as the filename for consistency
+            first_refid = function_groups[func_name][0]
+            content = generate_member_api_page(
+                func_name,
+                "function",
+                project_name,
+                refid=first_refid,
+                overload_refids=function_groups[func_name],
+                xml_dir=xml_dir,
+            )
+            output_file = api_dir / f"{first_refid}.rst"
+            with open(output_file, "w") as f:
+                f.write(content)
+            logger.info(f"Generated function API page: {output_file}")
+
+        # Generate individual pages for typedefs
+        for name, refid in items["typedefs"]:
+            if name in _BREATHE_SKIP_SYMBOLS:
+                logger.info(f"Skipping typedef {name} (in _BREATHE_SKIP_SYMBOLS)")
+                continue
+            content = generate_member_api_page(name, "typedef", project_name, refid)
+            output_file = api_dir / f"{refid}.rst"
+            with open(output_file, "w") as f:
+                f.write(content)
+            logger.info(f"Generated typedef API page: {output_file}")
+
+        # Generate individual pages for enums
+        for name, refid in items["enums"]:
+            content = generate_member_api_page(name, "enum", project_name, refid)
+            output_file = api_dir / f"{refid}.rst"
+            with open(output_file, "w") as f:
+                f.write(content)
+            logger.info(f"Generated enum API page: {output_file}")
+
+        # Generate individual pages for variables
+        for name, refid in items["variables"]:
+            if name in _BREATHE_SKIP_SYMBOLS:
+                logger.info(f"Skipping variable {name} (in _BREATHE_SKIP_SYMBOLS)")
+                continue
+            content = generate_member_api_page(name, "variable", project_name, refid)
+            output_file = api_dir / f"{refid}.rst"
+            with open(output_file, "w") as f:
+                f.write(content)
+            logger.info(f"Generated variable API page: {output_file}")
+
+        # Generate individual pages for macros
+        for name, refid in items["macros"]:
+            if name in _BREATHE_SKIP_SYMBOLS:
+                logger.info(f"Skipping macro {name} (in _BREATHE_SKIP_SYMBOLS)")
+                continue
+            content = generate_member_api_page(name, "macro", project_name, refid)
+            output_file = api_dir / f"{format_macro_doc_filename(name)}.rst"
+            with open(output_file, "w") as f:
+                f.write(content)
+            logger.info(f"Generated macro API page: {output_file}")
+
+        # Generate group index pages
+        for group_name, group_refid in items["groups"]:
+            content = generate_group_index_page(
+                group_name, group_refid, project_name, xml_dir, api_dir
+            )
+            output_file = api_dir / f"{group_refid}.rst"
+            with open(output_file, "w") as f:
+                f.write(content)
+            logger.info(f"Generated group index page: {output_file}")
+
+        # Generate the main namespace API reference page for api/index.rst
+        namespace_content = generate_namespace_api_page(project_name, items)
+        namespace_file = api_dir / "index.rst"
+        with open(namespace_file, "w") as f:
+            f.write(namespace_content)
+        logger.info(f"Generated namespace API reference: {namespace_file}")
+
+        # Note: We no longer generate auto_api.rst as api/index.rst serves the same purpose
+
+        # Generate category index pages (for backward compatibility)
+        # Create stub files for all categories to avoid toctree warnings
+        for category, class_list in classes.items():
+            output_file = api_dir / f"{category}.rst"
+
+            if class_list:
+                content = generate_category_index(category, class_list, project_name)
+            else:
+                # Create a minimal stub file for empty categories
+                category_titles = {
+                    "device": "Device-wide Primitives",
+                    "block": "Block-wide Primitives",
+                    "warp": "Warp-wide Primitives",
+                    "grid": "Grid-wide Primitives",
+                    "iterator": "Iterator Utilities",
+                    "thread": "Thread-level Primitives",
+                    "utility": "Utility Components",
+                }
+                title = category_titles.get(category, category.title())
+                content = f""".. AUTO-GENERATED by auto_api_generator.py - DO NOT EDIT
+
+:orphan:
+
+{title}
+{"=" * len(title)}
+
+.. note::
+   No items in this category.
+"""
+
+            # Write the category index
+            with open(output_file, "w") as f:
+                f.write(content)
+            logger.info(f"Generated category index: {output_file}")
+
+
+def setup(app: Sphinx):
+    """Setup the extension."""
+    app.connect("config-inited", generate_api_docs)
+
+    return {
+        "version": "1.0",
+        "parallel_read_safe": True,
+        "parallel_write_safe": True,
+    }
diff --git a/cccl_upstream/docs/_static/deduplicate_toc.js b/cccl_upstream/docs/_static/deduplicate_toc.js
new file mode 100644
index 00000000..1bcfa8a6
--- /dev/null
+++ b/cccl_upstream/docs/_static/deduplicate_toc.js
@@ -0,0 +1,32 @@
+// Clean up the "On this page" sidebar for C++ API pages.
+//
+// Two problems caused by Breathe's per-overload anchor generation:
+// 1. toc-h4 entries: Breathe adds a bare redundant "transform()" child anchor under each
+//    overload's section heading. Always remove them.
+// 2. Duplicate toc-h3 entries: when all overloads share the same display name
+//    (e.g. "ExclusiveSum()"), keep only the first occurrence.
+document.addEventListener('DOMContentLoaded', function() {
+  var tocNav = document.getElementById('pst-page-toc-nav');
+  if (!tocNav)
+    return;
+
+  tocNav.querySelectorAll('li.toc-h4').forEach(function(li) {
+    var label = li.textContent.trim();
+    if (label.endsWith('()')) {
+      li.remove();
+    }
+  });
+
+  var seen = new Set();
+  tocNav.querySelectorAll('li.toc-h3').forEach(function(li) {
+    var label = li.textContent.trim();
+    if (!label.endsWith(')')) {
+      return;
+    }
+    if (seen.has(label)) {
+      li.remove();
+    } else {
+      seen.add(label);
+    }
+  });
+});
diff --git a/cccl_upstream/docs/_static/nvidia-logo.png b/cccl_upstream/docs/_static/nvidia-logo.png
new file mode 100644
index 00000000..1779ad93
Binary files /dev/null and b/cccl_upstream/docs/_static/nvidia-logo.png differ
diff --git a/cccl_upstream/docs/_static/search_custom.css b/cccl_upstream/docs/_static/search_custom.css
new file mode 100644
index 00000000..a503afc0
--- /dev/null
+++ b/cccl_upstream/docs/_static/search_custom.css
@@ -0,0 +1,41 @@
+.cccl-search-breadcrumbs {
+  display: flex;
+  flex-wrap: wrap;
+  font-size: 0.72em;
+  line-height: normal;
+  list-style: none;
+  padding: 0;
+}
+
+.cccl-search-breadcrumbs .breadcrumb-item {
+  align-items: center;
+  display: flex;
+  font-weight: 700;
+  margin: 0;
+  padding: 0;
+  white-space: nowrap;
+}
+
+.cccl-search-breadcrumbs .breadcrumb-item a {
+  color: var(--pst-color-text-muted);
+  margin: 0.1875rem;
+  overflow-x: hidden;
+  text-decoration: none;
+  text-overflow: ellipsis;
+}
+
+.cccl-search-breadcrumbs .breadcrumb-item a:hover {
+  color: var(--pst-color-link-hover);
+  text-decoration: underline;
+  text-decoration-skip-ink: none;
+  text-decoration-thickness: max(3px, 0.1875rem, 0.12em);
+  text-underline-offset: 0.1578em;
+}
+
+.cccl-search-breadcrumbs .breadcrumb-item + .breadcrumb-item::before {
+  color: var(--pst-color-text-muted);
+  content: var(--pst-breadcrumb-divider);
+  font: var(--fa-font-solid);
+  font-size: 0.8rem;
+  padding: 0 0.5rem;
+}
diff --git a/cccl_upstream/docs/_static/search_postprocess.js b/cccl_upstream/docs/_static/search_postprocess.js
new file mode 100644
index 00000000..b8bae1d5
--- /dev/null
+++ b/cccl_upstream/docs/_static/search_postprocess.js
@@ -0,0 +1,265 @@
+"use strict";
+
+(function () {
+  const maxBreadcrumbResults = 10;
+
+  const decodeEntities = (value) =>
+    String(value || "")
+      .replace(/&lt;/g, "<")
+      .replace(/&gt;/g, ">")
+      .replace(/&amp;/g, "&")
+      .replace(/&quot;/g, '"')
+      .replace(/&#39;|&apos;/g, "'")
+      .replace(/&nbsp;/g, " ")
+      .replace(/&hellip;/g, "...");
+
+  const getDocLinkSuffix = () =>
+    (typeof DOCUMENTATION_OPTIONS !== "undefined" &&
+      DOCUMENTATION_OPTIONS.LINK_SUFFIX) ||
+    ".html";
+
+  const pageInfoCache = new Map();
+
+  const getDocTitle = (docName) => {
+    if (
+      typeof Search === "undefined" ||
+      !Search._index ||
+      !Array.isArray(Search._index.docnames) ||
+      !Array.isArray(Search._index.titles)
+    ) {
+      return null;
+    }
+
+    const index = Search._index.docnames.indexOf(docName);
+    return index >= 0 ? Search._index.titles[index] : null;
+  };
+
+  const getResultDocNameFromHref = (href) => {
+    if (!href) {
+      return null;
+    }
+
+    const withoutAnchor = String(href).split("#", 1)[0];
+    const linkSuffix = getDocLinkSuffix();
+    const suffixIndex = withoutAnchor.lastIndexOf(linkSuffix);
+    if (suffixIndex < 0) {
+      return null;
+    }
+
+    let docName = withoutAnchor.slice(0, suffixIndex);
+    if (docName.startsWith("./")) {
+      docName = docName.slice(2);
+    }
+
+    const contentRoot =
+      document?.documentElement?.dataset?.content_root || "";
+    if (contentRoot && docName.startsWith(contentRoot)) {
+      docName = docName.slice(contentRoot.length);
+    }
+
+    return docName.replace(/^\/+/, "");
+  };
+
+  const getPageInfo = async (docName) => {
+    if (pageInfoCache.has(docName)) {
+      return pageInfoCache.get(docName);
+    }
+
+    const infoPromise = (async () => {
+      const pageUrl = `${docName}${getDocLinkSuffix()}`;
+      const response = await fetch(pageUrl);
+      const html = await response.text();
+      const parsed = new DOMParser().parseFromString(html, "text/html");
+
+      const pageHeading = parsed.querySelector("h1");
+      const breadcrumbLinks = Array.from(
+        parsed.querySelectorAll(".breadcrumb-item a.nav-link"),
+      );
+      const breadcrumbs = breadcrumbLinks
+        .map((breadcrumbLink) => {
+          const rawHref = breadcrumbLink.getAttribute("href");
+          if (!rawHref) {
+            return null;
+          }
+
+          return {
+            href: new URL(rawHref, response.url).href,
+            title: breadcrumbLink.textContent?.trim() || null,
+          };
+        })
+        .filter((breadcrumb) => breadcrumb && breadcrumb.title);
+
+      return {
+        pageTitle:
+          pageHeading?.textContent?.replace(/#\s*$/, "").trim() ||
+          getDocTitle(docName) ||
+          null,
+        breadcrumbs,
+      };
+    })().catch(() => null);
+
+    pageInfoCache.set(docName, infoPromise);
+    return infoPromise;
+  };
+
+  const addBreadcrumbTrail = async (listItem) => {
+    if (
+      !listItem ||
+      listItem.dataset.ccclBreadcrumbsAttached === "true" ||
+      listItem.dataset.ccclBreadcrumbsPending === "true"
+    ) {
+      return;
+    }
+    listItem.dataset.ccclBreadcrumbsPending = "true";
+
+    const primaryLink = listItem.querySelector("a");
+    if (!primaryLink) {
+      delete listItem.dataset.ccclBreadcrumbsPending;
+      return;
+    }
+
+    const primaryTitle = primaryLink.textContent?.trim() || "";
+    const href = primaryLink.getAttribute("href");
+    const docName = getResultDocNameFromHref(href);
+    if (!docName) {
+      delete listItem.dataset.ccclBreadcrumbsPending;
+      return;
+    }
+
+    const pageInfo = await getPageInfo(docName);
+    if (!pageInfo) {
+      delete listItem.dataset.ccclBreadcrumbsPending;
+      return;
+    }
+
+    const pageTitle = pageInfo.pageTitle || getDocTitle(docName);
+    const breadcrumbs = [...(pageInfo.breadcrumbs || [])];
+    if (pageTitle && primaryTitle && pageTitle !== primaryTitle) {
+      breadcrumbs.push({
+        href: `${docName}${getDocLinkSuffix()}`,
+        title: pageTitle,
+      });
+    }
+
+    if (breadcrumbs.length === 0) {
+      delete listItem.dataset.ccclBreadcrumbsPending;
+      return;
+    }
+
+    const breadcrumbContainer = document.createElement("div");
+    breadcrumbContainer.className = "cccl-search-breadcrumbs";
+
+    breadcrumbs.forEach((breadcrumb) => {
+      const breadcrumbItem = document.createElement("span");
+      breadcrumbItem.className = "breadcrumb-item";
+      const breadcrumbLink = document.createElement("a");
+      breadcrumbLink.href = breadcrumb.href;
+      breadcrumbLink.textContent = breadcrumb.title;
+      breadcrumbItem.appendChild(breadcrumbLink);
+      breadcrumbContainer.appendChild(breadcrumbItem);
+    });
+
+    listItem.insertBefore(breadcrumbContainer, primaryLink.nextSibling);
+    listItem.dataset.ccclBreadcrumbsAttached = "true";
+    delete listItem.dataset.ccclBreadcrumbsPending;
+  };
+
+  const installResultDecorator = () => {
+    if (
+      typeof Search === "undefined" ||
+      Search.__ccclResultDecoratorInstalled ||
+      typeof MutationObserver === "undefined"
+    ) {
+      return;
+    }
+
+    const originalPerformSearch = Search.performSearch;
+    Search.performSearch = (...args) => {
+      const result = originalPerformSearch(...args);
+      const output = Search.output;
+      if (!output) {
+        return result;
+      }
+
+      const decorateTopResults = () => {
+        Array.from(output.querySelectorAll("li"))
+          .slice(0, maxBreadcrumbResults)
+          .forEach(addBreadcrumbTrail);
+      };
+
+      if (Search.__ccclResultsObserver) {
+        Search.__ccclResultsObserver.disconnect();
+      }
+
+      decorateTopResults();
+
+      const observer = new MutationObserver((mutations) => {
+        decorateTopResults();
+      });
+
+      observer.observe(output, { childList: true, subtree: true });
+      Search.__ccclResultsObserver = observer;
+      Search.__ccclResultDecoratorInstalled = true;
+      return result;
+    };
+  };
+
+  const installPostprocess = () => {
+    if (typeof Search === "undefined" || Search.__ccclDedupInstalled) {
+      return;
+    }
+
+    const originalPerformSearch = Search._performSearch;
+    Search._performSearch = (...args) => {
+      const results = originalPerformSearch(...args);
+
+      // Sphinx keeps results in low->high score order and displays via pop().
+      // Walk from the end so we see the best-ranked result first, but prefer
+      // canonical page links without anchors when collapsing duplicates.
+      const chosen = new Map();
+      for (let i = results.length - 1; i >= 0; --i) {
+        const result = results[i];
+        const title = String(result[1] || "").toLowerCase();
+        const filename = String(result[5] || "");
+        const key = `${filename}\0${title}`;
+        const anchor = String(result[2] || "");
+        const existing = chosen.get(key);
+        if (!existing) {
+          chosen.set(key, result);
+          continue;
+        }
+
+        const existingAnchor = String(existing[2] || "");
+        const prefersCurrent = existingAnchor && !anchor;
+        if (prefersCurrent) {
+          chosen.set(key, result);
+        }
+      }
+
+      const deduped = [];
+      const emitted = new Set();
+      for (let i = 0; i < results.length; ++i) {
+        const result = results[i];
+        const title = String(result[1] || "").toLowerCase();
+        const filename = String(result[5] || "");
+        const key = `${filename}\0${title}`;
+        if (emitted.has(key)) {
+          continue;
+        }
+        const winner = chosen.get(key);
+        if (winner) {
+          winner[1] = decodeEntities(winner[1]);
+          winner[3] = decodeEntities(winner[3]);
+          deduped.push(winner);
+          emitted.add(key);
+        }
+      }
+      return deduped;
+    };
+
+    Search.__ccclDedupInstalled = true;
+  };
+
+  installPostprocess();
+  installResultDecorator();
+})();
diff --git a/cccl_upstream/docs/_static/search_scorer.js b/cccl_upstream/docs/_static/search_scorer.js
new file mode 100644
index 00000000..81612e4f
--- /dev/null
+++ b/cccl_upstream/docs/_static/search_scorer.js
@@ -0,0 +1,224 @@
+"use strict";
+
+const _normalizeSearchSymbol = (value) =>
+  (value || "")
+    .toLowerCase()
+    .replace(/&lt;|&gt;|&amp;|&quot;|&#39;|&apos;|&nbsp;|&hellip;/g, " ")
+    .replace(/[^a-z0-9:]+/g, "");
+
+const _splitSymbolWords = (value) =>
+  (value || "")
+    .replace(/&lt;|&gt;|&amp;|&quot;|&#39;|&apos;|&nbsp;|&hellip;/g, " ")
+    .replace(/::/g, " ")
+    .replace(/([a-z0-9])([A-Z])/g, "$1 $2")
+    .replace(/([A-Z]+)([A-Z][a-z])/g, "$1 $2")
+    .toLowerCase()
+    .match(/[a-z0-9]+/g) || [];
+
+const _getSearchQuery = () => {
+  try {
+    return new URLSearchParams(window.location.search).get("q") || "";
+  } catch {
+    return "";
+  }
+};
+
+var Scorer = {
+  // Keep strong object-name bias at the base layer.
+  objNameMatch: 80,
+  objPartialMatch: 35,
+  objPrio: {
+    0: 25, // highest-priority API objects
+    1: 10,
+    2: -10,
+  },
+  objPrioDefault: 0,
+  title: 15,
+  partialTitle: 7,
+  term: 5,
+  partialTerm: 2,
+
+  score: (result) => {
+    const [docName, title, anchor, descr, baseScore, filename] = result;
+    let score = baseScore;
+
+    const trimmedTitle = (title || "").trim();
+    const trimmedAnchor = (anchor || "").trim();
+    const trimmedDescription = (descr || "").trim();
+    const trimmedFilename = (filename || "").trim();
+    const trimmedDocName = (docName || "").trim();
+
+    const lowerDescription = trimmedDescription.toLowerCase();
+    const lowerFilename = trimmedFilename.toLowerCase();
+    const lowerDocName = trimmedDocName.toLowerCase();
+    const query = _getSearchQuery().trim();
+    const lowerQuery = query.toLowerCase();
+    const normalizedQuery = _normalizeSearchSymbol(query);
+
+    const titleParts = trimmedTitle.split("::").filter(Boolean);
+    const symbolDepth = titleParts.length;
+    const leaf = titleParts.length
+      ? titleParts[titleParts.length - 1]
+      : trimmedTitle;
+    const parentTitle =
+      titleParts.length > 1 ? titleParts.slice(0, -1).join("::") : "";
+    const normalizedLeaf = _normalizeSearchSymbol(leaf);
+    const normalizedParent = _normalizeSearchSymbol(parentTitle);
+    const normalizedLeafOnlyQuery = _normalizeSearchSymbol(
+      query.includes("::") ? query.split("::").pop() : query,
+    );
+    const leafWords = _splitSymbolWords(leaf);
+    const queryWords = _splitSymbolWords(query);
+    const simpleQuery =
+      lowerQuery &&
+      !/[.:/_]/.test(lowerQuery) &&
+      /^[a-z0-9]+$/.test(lowerQuery);
+
+    const looksLikeNamespaceQualified =
+      /^[a-zA-Z_]\w*(::[a-zA-Z_]\w*)+/.test(trimmedTitle);
+
+    const isExactFunctionishTitle =
+      /::[A-Za-z_]\w*$/.test(trimmedTitle); // e.g. thrust::transform
+
+    const isClassishTitle =
+      /::[A-Z]\w*$/.test(trimmedTitle); // e.g. cub::DeviceRadixSort
+
+    const isParameterLike =
+      /(template parameter|function parameter)/i.test(trimmedDescription);
+    const isMemberLike =
+      /(C\+\+ (member|type|property))/i.test(trimmedDescription);
+    const isCallableLike =
+      /(C\+\+ function\b|C\+\+ class\b|C\+\+ struct\b)/i.test(trimmedDescription);
+    const isTopLevelSymbol = looksLikeNamespaceQualified && symbolDepth <= 2;
+    const isNestedSymbol = symbolDepth >= 3;
+    const isConstructorLike =
+      isNestedSymbol &&
+      _normalizeSearchSymbol(titleParts[symbolDepth - 2]) === normalizedLeaf;
+    const hasQueryInFilename =
+      lowerQuery && lowerFilename.includes(lowerQuery);
+    const hasQueryInDocName = lowerQuery && lowerDocName.includes(lowerQuery);
+    const isEnumeratorLike =
+      /(C\+\+ enumerator\b)/i.test(trimmedDescription) || /^[A-Z0-9_]+$/.test(leaf);
+    const isInternalHelperLike =
+      /(policy|dispatch|state|status|callback|preference|layout|runningprefixop|emptycallback|op)/i.test(
+        trimmedTitle,
+      ) ||
+      /(TileState|Policy|Dispatch|Callback|Preference|Layout|RunningPrefixOp|Status|EmptyCallback)/.test(
+        trimmedTitle,
+      );
+    const isPythonModuleLike = /(Python module\b)/i.test(trimmedDescription);
+    const leafStartsWithQueryWord =
+      queryWords.length === 1 && leafWords[0] === queryWords[0];
+    const leafEndsWithQueryWord =
+      queryWords.length === 1 &&
+      leafWords.length > 0 &&
+      leafWords[leafWords.length - 1] === queryWords[0];
+    const leafQueryRemainderWords = queryWords.length === 1
+      ? leafWords.filter((word) => word !== queryWords[0])
+      : [];
+    const hasCompactQueryWordRemainder =
+      leafStartsWithQueryWord &&
+      leafQueryRemainderWords.length > 0 &&
+      leafQueryRemainderWords.length <= 2;
+    const hasHelperSuffix =
+      /(Strategy|Policy|State|Status|Callback|Preference|Layout|Type|Op|Match|Functor|Tag|Traits|Descriptor|Counts)$/.test(
+        leaf,
+      );
+    // Strong bias toward actual API symbols.
+    if (isExactFunctionishTitle) score += 35;
+    if (isClassishTitle) score += 20;
+
+    // Small boost for anchored entries; these are often object targets.
+    if (trimmedAnchor) score += 5;
+
+    // Penalize taxonomy/concept pages that match lots of body text.
+    if (
+      lowerDescription.includes("thrust::") ||
+      lowerDescription.includes("cub::") ||
+      lowerDescription.includes("cuda::")
+    ) {
+      score += 8;
+    }
+
+    // Query-aware ranking: prefer canonical symbol pages over nested members.
+    if (normalizedQuery) {
+      if (normalizedLeaf === normalizedLeafOnlyQuery) {
+        score += isTopLevelSymbol ? 180 : 35;
+      } else if (
+        normalizedLeafOnlyQuery &&
+        normalizedLeaf.includes(normalizedLeafOnlyQuery)
+      ) {
+        score += 15;
+      }
+    }
+
+    if (isNestedSymbol) score -= 25;
+    if (isParameterLike) score -= 80;
+    if (isMemberLike) score -= 35;
+    if (isConstructorLike) score -= 30;
+    if (isCallableLike && isTopLevelSymbol) score += 20;
+
+    // Prefer libcudacxx/cuda symbols over thrust equivalents on ties.
+    if (
+      normalizedLeaf === normalizedLeafOnlyQuery &&
+      /^cuda::/.test(trimmedTitle)
+    ) {
+      score += 12;
+    }
+
+    // For plain keyword queries, prefer pages that match in title/path metadata.
+    if (simpleQuery) {
+      if (hasQueryInFilename || hasQueryInDocName) score += 45;
+      if (isInternalHelperLike) score -= 100;
+      if (isPythonModuleLike) score -= 30;
+      if (hasHelperSuffix) score -= 80;
+      if (isTopLevelSymbol && isCallableLike && !hasHelperSuffix) score += 40;
+      if (
+        leafStartsWithQueryWord &&
+        isTopLevelSymbol &&
+        !isEnumeratorLike &&
+        !isInternalHelperLike
+      ) {
+        score += 75;
+      }
+      if (
+        leafEndsWithQueryWord &&
+        isTopLevelSymbol &&
+        !isEnumeratorLike &&
+        !isInternalHelperLike
+      ) {
+        score += 45;
+      }
+
+      // For broad prefix-style queries like "block", prefer compact public API
+      // names over longer compound variants or helper-like extensions.
+      if (
+        hasCompactQueryWordRemainder &&
+        isTopLevelSymbol &&
+        !isEnumeratorLike &&
+        !isInternalHelperLike &&
+        !hasHelperSuffix
+      ) {
+        score += 70 - 15 * (leafQueryRemainderWords.length - 1);
+      }
+      // If a nested member matches the query but its parent symbol also does,
+      // prefer the parent page/class over the member overload.
+      if (
+        isNestedSymbol &&
+        normalizedLeaf === normalizedLeafOnlyQuery &&
+        normalizedParent.includes(normalizedLeafOnlyQuery)
+      ) {
+        score -= 70;
+      }
+      if (
+        isTopLevelSymbol &&
+        normalizedLeaf.includes(normalizedLeafOnlyQuery) &&
+        normalizedLeaf !== normalizedLeafOnlyQuery
+      ) {
+        score += 70;
+      }
+    }
+
+    return score;
+  },
+};
diff --git a/cccl_upstream/docs/_templates/search.html b/cccl_upstream/docs/_templates/search.html
new file mode 100644
index 00000000..9233b2c8
--- /dev/null
+++ b/cccl_upstream/docs/_templates/search.html
@@ -0,0 +1,12 @@
+{% extends "!nvidia_sphinx_theme/search.html" %}
+
+{% block extrahead %}
+{{ super() }}
+<link rel="stylesheet" href="{{ pathto('_static/search_custom.css', 1) }}" />
+{% endblock %}
+
+{% block scripts %}
+<script src="{{ pathto('_static/search_scorer.js', 1) }}"></script>
+{{ super() }}
+<script src="{{ pathto('_static/search_postprocess.js', 1) }}"></script>
+{% endblock %}
diff --git a/cccl_upstream/docs/cccl/3.0_migration_guide.rst b/cccl_upstream/docs/cccl/3.0_migration_guide.rst
new file mode 100644
index 00000000..e72e87b2
--- /dev/null
+++ b/cccl_upstream/docs/cccl/3.0_migration_guide.rst
@@ -0,0 +1,310 @@
+.. _cccl-3.0-migration-guide:
+
+CCCL 2.x ‐ CCCL 3.0 migration guide
+===================================
+
+The CCCL team plans breaking changes carefully and only conducts them at major releases.
+The CCCL 2.8 release came with many deprecations to prepare for the breaking changes conducted in CCCL 3.0.
+This page summarizes the changes and helps migrating from CCCL 2.x to CCCL 3.0.
+
+See also the `list of all deprecated APIs in CCCL 2.8 <https://github.com/NVIDIA/cccl/issues/3700>`_
+and the `list of breaking changes in CCCL 3.0 <https://github.com/NVIDIA/cccl/issues/101>`_.
+
+CUDA Toolkit changes
+--------------------
+
+CCCL is moving to its own include directory within the CUDA Toolkit. This may cause build failures and some initial confusion.
+This section will have some suggestions and mitigations to help maintain builds across both CUDA12 and future releases.
+
+The CTK-provided includes are changing in the following ways:
+
++-------------------------------+------------------------------------+
+| **Before CUDA 13.0**          | **After CUDA 13.0**                |
++-------------------------------+------------------------------------+
+| `${CTK_ROOT}/include/cuda/`   | `${CTK_ROOT}/include/cccl/cuda/`   |
++-------------------------------+------------------------------------+
+| `${CTK_ROOT}/include/cub/`    | `${CTK_ROOT}/include/cccl/cub/`    |
++-------------------------------+------------------------------------+
+| `${CTK_ROOT}/include/thrust/` | `${CTK_ROOT}/include/cccl/thrust/` |
++-------------------------------+------------------------------------+
+
+Due to these changes, and the fact that NVCC by default includes its own directories, you may encounter errors when including
+CCCL headers in source files that are compiled *only* by the host compiler.
+
+For example, when compiling with GCC or MSVC alone, you may see ``<cuda/...>``, ``<cub/...>``, or ``<thrust/...>`` headers missing.
+
+To mitigate this there are several solutions available depending on your build system:
+
+- **DO NOT** prefix missing includes with ``<cccl/>`` -- This will break.
+- CMake: link ``CCCL::CCCL`` to your target.
+  - Example: ``target_link_library(${MY_TARGET} PRIVATE CCCL::CCCL)``
+- Non-CMake: Directly include the CUDA Toolkit's CCCL directory. (Make/Other)
+  - Example: Add CCCL as an include flag ``-I${CTK_ROOT}/include/cccl``
+- Use a non-bundled CCCL. CCCL is available and maintained independently of the CTK.
+  - `See here for compatibility. <https://github.com/NVIDIA/cccl?tab=readme-ov-file#cuda-toolkit-ctk-compatibility>`_
+
+Removed macros
+--------------
+
+* ``CUB_IS_INT128_ENABLED``: No replacement
+* ``CUB_MAX(a, b)``: Use the ``cuda::std::max(a, b)`` function instead
+* ``CUB_MIN(a, b)``: Use the ``cuda::std::min(a, b)`` function instead
+* ``CUB_QUOTIENT_CEILING(a, b)``: Use ``cuda::ceil_div(a, b)`` instead
+* ``CUB_QUOTIENT_FLOOR(a, b)``: Use plain integer division ``a / b`` instead
+* ``CUB_ROUND_DOWN_NEAREST(a, b)``: Use ``cuda::round_down(a, b)`` instead
+* ``CUB_ROUND_UP_NEAREST(a, b)``: Use ``cuda::round_up(a, b)`` instead
+* ``CUB_RUNTIME_ENABLED``: No replacement
+* ``CUB_USE_COOPERATIVE_GROUPS``: No replacement
+* ``CUDA_CUB_RET_IF_FAIL``: No replacement
+* ``[THRUST|CUB]_CLANG_VERSION``: No replacement
+* ``[THRUST|CUB]_DEVICE_COMPILER*``: No replacement
+* ``[THRUST|CUB]_GCC_VERSION``: No replacement
+* ``[THRUST|CUB]_HOST_COMPILER*``: No replacement
+* ``[THRUST|CUB]_INCLUDE_DEVICE_CODE``: No replacement
+* ``[THRUST|CUB]_INCLUDE_HOST_CODE``: No replacement
+* ``[THRUST|CUB]_IS_DEVICE_CODE``: No replacement
+* ``[THRUST|CUB]_IS_HOST_CODE``: No replacement
+* ``[THRUST|CUB]_MSVC_VERSION_FULL``: No replacement
+* ``[THRUST|CUB]_MSVC_VERSION``: No replacement
+* ``THRUST_CDP_DISPATCH``: No replacement (Support for CUDA Dynamic Parallelism V1 (CDPv1) has been removed, see below)
+* ``THRUST_DECLTYPE_RETURNS_WITH_SFINAE_CONDITION``: No replacement
+* ``THRUST_DECLTYPE_RETURNS``: No replacement
+* ``THRUST_DEVICE_CODE``: No replacement
+* ``THRUST_HOST_BACKEND``: Use ``THRUST_HOST_SYSTEM`` instead
+* ``THRUST_INLINE_CONSTANT``: Use ``inline constexpr`` instead
+* ``THRUST_INLINE_INTEGRAL_MEMBER_CONSTANT``: Use ``static constexpr`` instead
+* ``THRUST_LEGACY_GCC``: No replacement
+* ``THRUST_MODERN_GCC_REQUIRED_NO_ERROR``: No replacement
+* ``THRUST_MODERN_GCC``: No replacement
+* ``THRUST_MVCAP``: No replacement
+* ``THRUST_NODISCARD``: Use ``[[nodiscard]]`` instead
+* ``THRUST_RETOF1``: No replacement
+* ``THRUST_RETOF2``: No replacement
+* ``THRUST_RETOF``: No replacement
+* ``THRUST_TUNING_ARCH``: No direct replacement. Use compiler-specific ``__CUDA_ARCH__`` (nvcc) or  ``__NVCOMPILER_CUDA_ARCH__`` (nvc++) instead
+
+
+Removed functions and classes
+-----------------------------
+
+* ``_ReadWriteBarrier`` and ``__thrust_compiler_fence``: Use ``cuda::atomic`` instead
+* ``cub::*Kernel``: Any CUB kernel entrypoint is considered an implementation detail. No public exposure is provided.
+* ``cub::Agent*``: CUB agents were considered implementation details and have all been moved to internal namespaces. No public exposure is provided.
+* ``cub::AliasTemporaries``: No replacement
+* ``cub::ArrayWrapper``: Use ``cuda::std::array`` instead
+* ``cub::BAR``: No replacement
+* ``cub::BaseTraits::CATEGORY``: Use the facilities from ``<cuda/std/type_traits>`` instead
+* ``cub::BaseTraits::NULL_TYPE``: No replacement
+* ``cub::BaseTraits::PRIMITIVE``: Use the facilities from ``<cuda/std/type_traits>`` instead
+* ``cub::BFI``: Use ``cuda::bitfield_insert`` instead
+* ``cub::BinaryOpHasIdxParam::HAS_PARAM``: Use ``cub::BinaryOpHasIdxParam::value`` instead
+* ``cub::ConstantInputIterator``: Use ``thrust::constant_iterator`` instead
+* ``cub::CountingInputIterator``: Use ``thrust::counting_iterator`` instead
+* ``cub::CTA_SYNC_AND``: Use ``__syncthreads_and()`` instead
+* ``cub::CTA_SYNC_OR``: Use ``__syncthreads_or()`` instead
+* ``cub::CTA_SYNC``: Use ``__syncthreads()`` instead
+* ``cub::Device*Policy``: Those policy hubs are considered implementation details. No public exposure is provided.
+* ``cub::DeviceSpmv``: Use `cuSPARSE <https://docs.nvidia.com/cuda/cusparse>`_ instead
+* ``cub::Difference``: Use ``cuda::std::minus`` instead
+* ``cub::DivideAndRoundUp``: Use ``cuda::round_up`` instead
+* ``cub::Division``: Use ``cuda::std::divides`` instead
+* ``cub::Equality``: Use ``cuda::std::equal_to`` instead
+* ``cub::FFMA_RZ``: No replacement
+* ``cub::FMUL_RZ``: No replacement
+* ``cub::FpLimits<T>``: Use ``cuda::std::numeric_limits<T>`` instead
+* ``cub::GridBarrier``: Use the APIs from cooperative groups instead
+* ``cub::GridBarrierLifetime``: Use the APIs from cooperative groups instead
+* ``cub::IADD3``: No replacement
+* ``cub::Inequality``: Use ``cuda::std::not_equal_to`` instead
+* ``cub::Int2Type``: Use ``cuda::std::integral_constant`` instead
+* ``cub::IterateThreadLoad``: No replacement
+* ``cub::IterateThreadStore``: No replacement
+* ``cub::KernelConfig``: No replacement
+* ``cub::LaneId()``: Use ``cuda::ptx::get_sreg_laneid()`` instead
+* ``cub::LaneMaskGe()``: Use ``cuda::ptx::get_sreg_lanemask_ge()`` instead
+* ``cub::LaneMaskGt()``: Use ``cuda::ptx::get_sreg_lanemask_gt()`` instead
+* ``cub::LaneMaskLe()``: Use ``cuda::ptx::get_sreg_lanemask_le()`` instead
+* ``cub::LaneMaskLt()``: Use ``cuda::ptx::get_sreg_lanemask_lt()`` instead
+* ``cub::MakePolicyWrapper``: No replacement
+* ``cub::Max``: Use ``cuda::maximum`` instead
+* ``cub::max``: Use ``cuda::std::max`` instead
+* ``cub::MemBoundScaling``: No replacement
+* ``cub::Min``: Use ``cuda::minimum`` instead
+* ``cub::min``: Use ``cuda::std::min`` instead
+* ``cub::Mutex``: Use ``std::mutex`` instead
+* ``cub::PolicyWrapper``: No replacement
+* ``cub::PRMT``: Use ``cuda::ptx::prmt()`` instead
+* ``cub::RegBoundScaling``: No replacement
+* ``cub::SHFL_IDX_SYNC``: Use ``__shfl_sync()`` instead
+* ``cub::SHL_ADD``: No replacement
+* ``cub::SHR_ADD``: No replacement
+* ``cub::Sum``: Use ``cuda::std::plus`` instead
+* ``cub::Swap(a, b)``: Use ``cuda::std::swap(a, b)`` instead
+* ``cub::ThreadTrap()``: Use ``cuda::std::terminate()`` instead
+* ``cub::TransformInputIterator``: Use ``thrust::transform_iterator`` instead
+* ``cub::TripleChevronFactory``: No replacement for now, we are working on a new kernel launch facility
+* ``cub::ValueCache``: No replacement
+* ``cub::WARP_ALL``: Use ``__all_sync()`` instead
+* ``cub::WARP_ANY``: Use ``__any_sync()`` instead
+* ``cub::WARP_BALLOT``: Use ``__ballot_sync()`` instead
+* ``cub::WARP_SYNC``: Use ``__syncwarp()`` instead
+* ``cub::WarpId()``: Use ``cuda::ptx::get_sreg_warpid()`` instead
+* ``thrust::*::[first_argument_type|second_argument_type|result_type]``: The nested aliases have been removed for all function object types: ``thrust::[plus|minus|multiplies|divides|modulus|negate|square|equal_to|not_equal_to|greater|less|greater_equal|less_equal|logical_and|logical_or|logical_not|bit_and|bit_or|bit_xor|identity|maximum|minimum|project1st|project2nd]``. No replacement.
+* ``thrust::[unary|binary]_function``: No replacement. If you inherit from one of these types, just remove those base classes.
+* ``thrust::[unary|binary]_traits``: No replacement.
+* ``thrust::async::*``: No replacement for now. We are working on a C++26 senders implementation. For make a thrust algorithm skip syncing, use ``thrust::cuda::par_nosync`` as execution policy.
+* ``thrust::bidirectional_universal_iterator_tag``: No replacement
+* ``thrust::conjunction_value<Ts...>``: Use ``cuda::std::bool_constant<(Ts && ...)>`` instead
+* ``thrust::conjunction_value_v<Ts...>``: Use a fold expression: ``Ts && ...`` instead
+* ``thrust::cuda_cub::core::*``: Those are considered implementation details. No public exposure is provided.
+* ``thrust::cuda_cub::counting_iterator_t``: Use ``thrust::counting_iterator`` instead
+* ``thrust::cuda_cub::identity``: Use ``cuda::std::identity`` instead
+* ``thrust::cuda_cub::launcher::triple_chevron``: No replacement for now, we are working on a new kernel launch facility
+* ``thrust::cuda_cub::terminate``: Use ``cuda::std::terminate()`` instead
+* ``thrust::cuda_cub::transform_input_iterator_t``: Use ``thrust::transform_iterator`` instead
+* ``thrust::cuda_cub::transform_pair_of_input_iterators_t``: Use ``thrust::transform_iterator of a thrust::zip_iterator`` instead
+* ``thrust::disjunction_value<Ts...>``: Use ``cuda::std::bool_constant<(Ts || ...)>`` instead
+* ``thrust::disjunction_value_v<Ts...>``: Use a fold expression: ``Ts || ...`` instead
+* ``thrust::forward_universal_iterator_tag``: No replacement
+* ``thrust::identity<T>``: Use ``cuda::std::identity`` instead. If ``thrust::identity`` was used to perform a cast to ``T``, please define your own function object.
+* ``thrust::input_universal_iterator_tag``: No replacement
+* ``thrust::negation_value<T>``: Use ``cuda::std::bool_constant<!T>`` instead
+* ``thrust::negation_value_v<T>``: Use a plain negation ``!T``
+* ``thrust::not[1|2]``: Use ``cuda::std::not_fn`` instead
+* ``thrust::null_type``: No replacement
+* ``thrust::numeric_limits<T>``: Use ``cuda::std::numeric_limits<T>`` instead
+* ``thrust::optional<T>``: Use ``cuda::std::optional<T>`` instead.
+* ``thrust::output_universal_iterator_tag``: No replacement
+* ``thrust::random_access_universal_iterator_tag``: No replacement
+* ``thrust::remove_cvref[_t]``: Use ``cuda::std::remove_cvref[_t]`` instead
+* ``thrust::void_t``: Use ``cuda::std::void_t`` instead
+
+
+Deprecations with planned removal
+---------------------------------
+
+* ``CUB_LOG_SMEM_BANKS``: No replacement
+* ``CUB_LOG_WARP_THREADS``: No replacement
+* ``CUB_MAX_DEVICES``: No replacement
+* ``CUB_PREFER_CONFLICT_OVER_PADDING``: No replacement
+* ``CUB_PTX_LOG_SMEM_BANKS``: No replacement
+* ``CUB_PTX_LOG_WARP_THREADS``: No replacement
+* ``CUB_PTX_PREFER_CONFLICT_OVER_PADDING``: No replacement
+* ``CUB_PTX_SMEM_BANKS``: No replacement
+* ``CUB_PTX_SUBSCRIPTION_FACTOR``: No replacement
+* ``CUB_PTX_WARP_THREADS``: No replacement
+* ``CUB_SMEM_BANKS``: No replacement
+* ``CUB_SUBSCRIPTION_FACTOR``: No replacement
+* ``CUB_WARP_THREADS``: No replacement
+* ``THRUST_FALSE``: No replacement
+* ``THRUST_PREVENT_MACRO_SUBSTITUTION``: No replacement
+* ``THRUST_STATIC_ASSERT(expr)``: Use ``static_assert(expr)`` instead
+* ``THRUST_TRUE``: No replacement
+* ``THRUST_UNKNOWN``: No replacement
+* ``THRUST_UNUSED_VAR``: No replacement
+* ``cub::BFE``: Use ``cuda::bitfield_extract`` instead
+* ``cub::MergePathSearch``: No replacement
+* ``cub::Traits<T>::Max()``: Use ``cuda::std::numeric_limits<T>::max()`` instead
+* ``cub::Traits<T>::Min()``: Use ``cuda::std::numeric_limits<T>::min()`` instead
+* ``thrust::iterator_difference[_t]<T>``: Use ``cuda::std::iterator_traits<T>::difference_type`` or ``cuda::std::iter_difference_t<T>`` instead
+* ``thrust::iterator_pointer[_t]<T>``: Use ``cuda::std::iterator_traits<T>::pointer`` instead
+* ``thrust::iterator_reference[_t]<T>``: Use ``cuda::std::iterator_traits<T>::reference`` or ``cuda::std::iter_reference_t<T>`` instead
+* ``thrust::iterator_traits<T>``: Use ``cuda::std::iterator_traits<T>`` instead
+* ``thrust::iterator_value[_t]<T>``: Use ``cuda::std::iterator_traits<T>::value_type`` or ``cuda::std::iter_value_t<T>`` instead
+
+
+API breaks
+----------
+
+* ``cub::Block*``: All trailing ``int LEGACY_PTX_ARCH`` template parameters have been removed
+* ``cub::CachingAllocator``: The constructor taking a trailing ``bool debug`` parameter has been removed
+* ``cub::Device*``: All overloads with a trailing ``bool debug_synchronous`` parameter have been removed
+* ``cub::Dispatch*``: All Boolean template parameters have been replaced by enumerations to increase readability
+* ``cub::Dispatch*``: All policy hub template parameters have been moved to the back of the template parameters list
+* ``cub::DispatchScan[ByKey]``: The offset type must be an unsigned type of at least 4-byte size
+* ``cuda::ceil_div``: Now returns the common type of its arguments
+* ``thrust::pair``: Is now an alias to ``cuda::std::pair`` and no longer a distinct type
+* ``thrust::tabulate_output_iterator``: The ``value_type`` has been fixed to be ``void``
+* ``thrust::transform_iterator``: Upon copying, will now always copy its contained function. If the contained function is neither copy constructible nor copy assignable, the iterator fails to compile when attempting to be copied.
+* ``thrust::tuple``: Is now an alias to ``cuda::std::tuple`` and no longer a distinct type
+* ``thrust::universal_host_pinned_memory_resource``: The alias has changed to a different memory resource, potentially changing pointer types derived from an allocator/container using this memory resource.
+* The following Thrust function object types have been made aliases to the equally-named types in ``cuda::std::``: ``thrust::[plus|minus|multiplies|divides|modulus|negate|equal_to|not_equal_to|greater|less|greater_equal|less_equal|logical_and|logical_or|logical_not|bit_and|bit_or|bit_xor|identity|maximum|minimum]``. No replacement.
+* ``CUB_DEFINE_DETECT_NESTED_TYPE``: The generated detector trait no longer provides a ``::VALUE`` member. Use ``::value`` instead.
+
+
+Iterator traits
+^^^^^^^^^^^^^^^
+
+``cuda::std::iterator_traits`` will now correctly recognize user-provided specializations of ``std::iterator_traits``.
+All of Thrust's iterator traits have been redefined in terms of ``cuda::std::iterator_traits``,
+and users should prefer to use iterator traits from libcu++.
+``thrust::iterator_traits`` can no longer be specialized.
+Users should prefer to specialize ``cuda::std::iterator_traits`` instead of ``std::iterator_traits`` when necessary,
+to make their iterators work equally in device code.
+
+
+CUB Traits
+^^^^^^^^^^
+
+The functionality and internal use of ``cub::Traits`` has been minimized, because libcu++ provides better and standard alternatives.
+Only the use in CUB's radix sort implementation for bit-twiddling remains.
+Floating-point limits should be obtained using ``cuda::std::numeric_limits<T>`` instead of ``cub::FpLimits<T>``.
+Classification of types should be done with the facilities from ``<cuda/std/type_traits>`` and ``<cuda/type_traits>``,
+notably with ``cuda::std::is_signed[_v]``, ``cuda::std::is_integral[_v]``, etc.
+There is an important difference for extended floating point types though:
+Since ``cuda::std::is_floating_point[_v]`` will only recognize C++ standard floating point types,
+``cuda::is_floating_point[_v]`` must be used to correctly classify extended floating point types like ``__half`` or ``__nv_bfloat16``.
+``cub::BaseTraits`` and ``cub::Traits`` can no longer be specialized for custom types, and ``cub::FpLimits`` has been removed.
+
+We acknowledge the need to provide user-defined floating point types though,
+e.g., registering a custom half type with CUB to be used in radix sort.
+Therefore, users can still specialize ``cub::NumericTraits`` for their custom floating point types,
+inheriting from ``cub::BaseTraits`` and providing the necessary information for the type.
+Additionally, the traits from libcu++ have to be specialized as well:
+
+For example, a custom floating point type ``my_half`` could be registered with CUB and libcu++ like this:
+
+.. code:: cpp
+
+    template <>
+    inline constexpr bool ::cuda::is_floating_point_v<my_half> = true;
+
+    template <>
+    class ::cuda::std::numeric_limits<my_half> {
+    public:
+      static constexpr bool is_specialized = true;
+      static __host__ __device__ my_half max()    { return /* TODO */; }
+      static __host__ __device__ my_half min()    { return /* TODO */; }
+      static __host__ __device__ my_half lowest() { return /* TODO */; }
+    };
+
+    template <>
+    struct CUB_NS_QUALIFIER::NumericTraits<my_half> : BaseTraits<FLOATING_POINT, true, uint16_t, my_half> {};
+
+
+Behavioral changes
+------------------
+
+* ``cub::DeviceReduce::[Arg][Max|Min]``: Will now use ``cuda::std::numeric_limits<T>::[max|min]()`` instead of ``cub::Traits`` to determine the initial value
+* ``cuda::std::mdspan``: The implementation was entirely rewritten and you may experience subtle behavioral changes
+* ``thrust::transform_iterator``: The logic to determine the reference type has been reworked, especially wrt. to functions that return references to their own arguments (e.g., ``thrust::identity``).
+* ``thrust::transform_iterator::difference_type``: The logic to select the difference type has been reworked. It's now either ``int`` or ``ptrdiff``.
+
+
+ABI breaks
+----------
+
+* All of libcu++'s old ABI namespaces have been removed
+
+
+Platform support
+----------------
+
+* At least C++17 is required
+* At least clang 14 is required
+* At least GCC 7 is required
+* On Windows, at least Visual Studio 2019 is required (MSC_VER >= 1920)
+* Intel ICC (``icpx``) is no longer supported
+* At least CUDA Toolkit 12.0 is required
+* Support for CUDA Dynamic Parallelism V1 (CDPv1) has been removed
+* At least a GPU with compute capability 50 (Maxwell) is required
diff --git a/cccl_upstream/docs/cccl/config_macros.rst b/cccl_upstream/docs/cccl/config_macros.rst
new file mode 100644
index 00000000..78f73598
--- /dev/null
+++ b/cccl_upstream/docs/cccl/config_macros.rst
@@ -0,0 +1,94 @@
+.. _cccl-config:
+
+CCCL configuration macros
+=========================
+
+The CUDA Core Compute Libraries provide a set of macros to enable or disable specific features. These macros must be defined before any CCCL source file is included. The recommended way is to define them as the predefined compiler macros, for example:
+
+.. code-block:: bash
+
+    nvcc -DCCCL_DISABLE_SOME_FEATURE src.cu
+
+.. important::
+    These macros should be defined consistently in the whole project. Defining them only for some translation units may lead to unexpected compile time and runtime behaviour.
+
+Assertion Control Macros
+------------------------
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_ENABLE_ASSERTIONS                            | Enables assertions in both host and device code. Implied by compiling in debug mode.                                                                   |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_ENABLE_DEVICE_ASSERTIONS                     | Enables assertions in device code, independent of debug mode. Implied by ``CCCL_ENABLE_ASSERTIONS``.                                                   |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_ENABLE_HOST_ASSERTIONS                       | Enables assertions in host code, independent of debug mode. Implied by ``CCCL_ENABLE_ASSERTIONS``.                                                     |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+C++ Feature Control Macros
+--------------------------
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_DISABLE_EXCEPTIONS                           | Disables throwing exceptions. Each ``throw`` is replaced by a call to ``cuda::std::terminate()``.                                                      |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_DISABLE_RTTI                                 | Disables use of runtime type information.                                                                                                              |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_IGNORE_MSVC_TRADITIONAL_PREPROCESSOR_WARNING | Disables diagnostics emitted when using MSVC's traditional preprocessor.                                                                               |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+CUDA Feature Control Macros
+---------------------------
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_DISABLE_ARCH_DEPENDENT_NAMESPACE             | Disables architecture dependent name mangling of kernels.                                                                                              |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_DISABLE_CDP                                  | Disables use of CUDA Dynamic Parallelism.                                                                                                              |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_DISABLE_CTK_COMPATIBILITY_CHECK              | Disables the check whether NVCC's version matches the CUDA Toolkit version.                                                                            |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_DISABLE_DEVICE_RUNTIME                       | Disables use of CUDA device runtime APIs (``<cuda_device_runtime.h>``), thus makes some APIs that are ``__host__ __device__`` to be ``__host__`` only. |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_DISABLE_LAUNCH_BOUNDS                        | Disables use of ``__launch_bounds__`` attribute.                                                                                                       |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_DISABLE_PDL                                  | Disables use of Programmatic Dependent Launch.                                                                                                         |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+Deprecation Diagnostics Suppression Macros
+------------------------------------------
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_IGNORE_DEPRECATED_API                        | Disables deprecated API diagnostics.                                                                                                                   |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_IGNORE_DEPRECATED_COMPILER                   | Disables deprecated compiler diagnostics.                                                                                                              |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_IGNORE_DEPRECATED_CPP_DIALECT                | Disables deprecated C++ dialect diagnostics.                                                                                                           |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_IGNORE_DEPRECATED_CUDA_BELOW_12              | Disables deprecated CUDA compiler diagnostics.                                                                                                         |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+Third Party Libraries Interoperability
+--------------------------------------
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_DISABLE_DLPACK                               | Disables inclusion of DLPack header and APIs.                                                                                                          |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+Type Support
+------------
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_DISABLE_BF16_SUPPORT                         | Disables use and library support for the ``__nv_bfloat16`` type. Also disables support for smaller NV floating point types.                            |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_DISABLE_FLOAT128_SUPPORT                     | Disables use and library support for the ``__float128`` type.                                                                                          |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_DISABLE_FP16_SUPPORT                         | Disables use and library support for the ``__half`` type. Also disables support for smaller NV floating point types.                                   |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_DISABLE_INT128_SUPPORT                       | Disables use and library support for the ``__int128`` type.                                                                                            |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_DISABLE_LONG_DOUBLE_SUPPORT                  | Disables use and library support for the ``long double`` type.                                                                                         |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_DISABLE_NVFP4_SUPPORT                        | Disables use and library support for the ``__nv_fp4_eNmM`` types. Also disables support for smaller NV floating point types.                           |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_DISABLE_NVFP6_SUPPORT                        | Disables use and library support for the ``__nv_fp6_eNmM`` types. Also disables support for smaller NV floating point types.                           |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_DISABLE_NVFP8_SUPPORT                        | Disables use and library support for the ``__nv_fp8_eNmM`` types. Also disables support for smaller NV floating point types.                           |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_ENABLE_EXPERIMENTAL_HOST_ATOMICS_128B        | Enables experimental support for 128b atomics in host code.                                                                                            |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+| CCCL_GCC_HAS_EXTENDED_NUMERIC_LITERALS            | Must be enabled in addition to passing ``-fext-numeric-literals`` with GCC to enable ``__float128`` support.                                           |
++---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+.. note::
+    ``<cuda_fpN.h>`` headers contain dependencies on other ``<cuda_fpM.h>`` headers, thus for example defining ``CCCL_DISABLE_BF16_SUPPORT`` will disable support for NVIDIA 8-bit, 6-bit and 4-bit floating point types, too.
diff --git a/cccl_upstream/docs/cccl/contributing.rst b/cccl_upstream/docs/cccl/contributing.rst
new file mode 100644
index 00000000..948e152f
--- /dev/null
+++ b/cccl_upstream/docs/cccl/contributing.rst
@@ -0,0 +1,14 @@
+.. _cccl-contributing:
+
+Contributing to the CUDA Core Compute Libraries
+===============================================
+
+.. toctree::
+   :maxdepth: 1
+
+   contributing/code_of_conduct
+
+We welcome contributions - just send us a pull request!
+You can find detailed instructions on `GitHub <https://github.com/NVIDIA/cccl/blob/main/CONTRIBUTING.md>`_.
+
+libcu++ uses the `Apache License v2.0 with LLVM Exceptions <https://llvm.org/LICENSE.txt>`_.
diff --git a/cccl_upstream/docs/cccl/contributing/code_of_conduct.rst b/cccl_upstream/docs/cccl/contributing/code_of_conduct.rst
new file mode 100644
index 00000000..5250320c
--- /dev/null
+++ b/cccl_upstream/docs/cccl/contributing/code_of_conduct.rst
@@ -0,0 +1,105 @@
+Code of Conduct
+===============
+
+Overview
+--------
+
+This document defines the Code of Conduct followed and enforced for
+NVIDIA C++ Core Compute Libraries.
+
+Intended Audience
+~~~~~~~~~~~~~~~~~
+
+-  Community
+-  Developers
+-  Project Leads
+
+Our Pledge
+----------
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our
+project and our community a harassment-free experience for everyone,
+regardless of age, body size, disability, ethnicity, sex
+characteristics, gender identity and expression, level of experience,
+education, socio-economic status, nationality, personal appearance,
+race, religion, or sexual identity and orientation.
+
+Our Standards
+-------------
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+-  Using welcoming and inclusive language.
+-  Being respectful of differing viewpoints and experiences.
+-  Gracefully accepting constructive criticism.
+-  Focusing on what is best for the community.
+-  Showing empathy towards other community members.
+
+Examples of unacceptable behavior by participants include:
+
+-  The use of sexualized language or imagery and unwelcome sexual
+   attention or advances.
+-  Trolling, insulting/derogatory comments, and personal or political
+   attacks.
+-  Public or private harassment.
+-  Publishing others' private information, such as a physical or
+   electronic address, without explicit permission.
+-  Other conduct which could reasonably be considered inappropriate.
+
+Our Responsibilities
+--------------------
+
+Project maintainers are responsible for clarifying the standards of
+acceptable behavior and are expected to take appropriate and fair
+corrective action in response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit,
+or reject comments, commits, code, wiki edits, issues, and other
+contributions that are not aligned to this Code of Conduct, or to ban
+temporarily or permanently any contributor for other behaviors that they
+deem inappropriate, threatening, offensive, or harmful.
+
+Scope
+-----
+
+This Code of Conduct applies both within project spaces and in public
+spaces when an individual is representing the project or its community.
+Examples of representing a project or community include using an
+official project email address, posting via an official social media
+account, or acting as an appointed representative at an online or
+offline event. Representation of a project may be further defined and
+clarified by project maintainers.
+
+Enforcement
+-----------
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may
+be reported by contacting cpp-conduct@nvidia.com. All complaints will be
+reviewed and investigated and will result in a response that is deemed
+necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an
+incident. Further details of specific enforcement policies may be posted
+separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in
+good faith may face temporary or permanent repercussions as determined
+by other members of the project's leadership.
+
+Attribution
+-----------
+
+This Code of Conduct was taken from the `NVIDIA
+RAPIDS <https://docs.rapids.ai/resources/conduct/>`_ project, which was
+adapted from the `Contributor Covenant version
+1.4 <https://www.contributor-covenant.org/version/1/4/code-of-conduct.html>`_.
+
+Please see this `FAQ <https://www.contributor-covenant.org/faq>`_ for
+answers to common questions about this Code of Conduct.
+
+Contact
+-------
+
+Please email cpp-conduct@nvidia.com for any Code of Conduct related
+matters.
diff --git a/cccl_upstream/docs/cccl/determinism.rst b/cccl_upstream/docs/cccl/determinism.rst
new file mode 100644
index 00000000..3e26c604
--- /dev/null
+++ b/cccl_upstream/docs/cccl/determinism.rst
@@ -0,0 +1,106 @@
+.. _cccl-determinism:
+
+Determinism
+===========
+
+Determinism describes whether an algorithm produces the *same result* every time it is run with the
+same input. For many parallel algorithms this is not automatic. For reductions and scans, for example,
+the order in which partial results are combined depends on how work is scheduled across thousands of
+threads, and that schedule can change between launches or between GPUs. When the combining operator is
+not perfectly associative — most notably floating-point addition, where ``(a + b) + c`` need not equal
+``a + (b + c)`` — a different combining order yields a (slightly) different result, so the output is no
+longer identical from one run to the next.
+
+What counts as the "same result" is defined *per algorithm*. For reductions and scans it means a
+*bitwise-identical* output. For other algorithms it can be weaker: a deterministic top-k, for example,
+guarantees the same *set* of selected items, while the order of those items within the output is a
+separate guarantee that an algorithm may expose on its own.
+
+CCCL lets users state the determinism guarantee they need as an explicit *requirement* on an
+algorithm, rather than relying on implementation-defined behavior. The library then either
+selects an implementation that satisfies the requirement or rejects the call at compile time if the
+requirement cannot be met for the given types and operator.
+
+Determinism guarantees
+----------------------
+
+By *reproducible* we mean: given the same inputs, an algorithm returns the same output, in the sense
+defined for that algorithm (see above). What the guarantees below differ in is the *scope* of that
+reproducibility — across repeated runs, across hardware, or not at all. CCCL models three levels,
+defined in ``cuda::execution::determinism``:
+
+``not_guaranteed``
+   No reproducibility guarantee. The result is a valid answer, but it may differ from one invocation to
+   the next — even on the same GPU with the same input. This is usually the fastest option.
+
+``run_to_run``
+   The result is reproducible across repeated runs *on the same GPU*, with the same input, build,
+   tuning, and launch configuration. It may still differ on a *different* GPU architecture.
+
+``gpu_to_gpu``
+   The strongest guarantee: the result is reproducible across repeated runs *and across different GPU
+   architectures* — the same inputs yield the same bits whether the algorithm runs on, say, an Ampere or
+   a Hopper GPU. This is the most constrained option, is not available for every type/operator
+   combination, and is typically the slowest.
+
+The guarantees are ordered from weakest to strongest:
+``not_guaranteed`` ⊆ ``run_to_run`` ⊆ ``gpu_to_gpu``. A ``gpu_to_gpu`` result is also reproducible
+run-to-run, and a ``run_to_run`` result is a valid (but stronger-than-required) answer wherever
+``not_guaranteed`` would be accepted.
+
+For types and operators that are exactly associative (see
+:ref:`cuda::is_associative_v <libcudacxx-extended-api-functional-operator-properties>`; for example, integral
+addition with well-known operators), every invocation is already reproducible across runs and GPUs, so the
+stronger guarantees come for free and the library simply selects the fastest valid implementation.
+
+.. warning::
+
+   ``gpu_to_gpu``/``run_to_run`` reproducibility is guaranteed for a *fixed* CCCL and CUDA Toolkit version, not
+   across versions. If a policy selector is specified to change the used tuning, then reproducibility is only
+   guaranteed for identical tunings. The bitwise result may also change between CCCL or CUDA Toolkit releases as
+   algorithms, reduction structures, or tuning evolve.
+
+Requesting a determinism guarantee
+-----------------------------------
+
+Determinism is expressed as a *requirement* and passed to an algorithm through its execution
+environment using ``cuda::execution::require``:
+
+.. code-block:: c++
+
+    #include <cuda/execution>
+
+    // Request run-to-run reproducibility for this call.
+    auto env = cuda::execution::require(cuda::execution::determinism::run_to_run);
+
+The requirement may be combined with other environment properties — such as a stream or a memory
+resource — into a single environment:
+
+.. code-block:: c++
+
+    auto determinism = cuda::execution::require(cuda::execution::determinism::run_to_run);
+    auto env         = cuda::std::execution::env{cuda::stream_ref{stream}, memory_resource, determinism};
+
+Passing a determinism property *without* wrapping it in ``require`` is a compile-time error
+(*"Determinism should be used inside requires to have an effect."*). ``require`` turns the property
+into a *requirement*, which is what the algorithm honors — this prevents a stray determinism property
+from being silently ignored.
+
+If an algorithm cannot satisfy the requested guarantee for the given value type and operator, the call
+fails to compile with a diagnostic explaining the constraint. If the guarantee can be satisfied by a
+weaker-but-sufficient implementation (for example, an exactly-associative operator under
+``gpu_to_gpu``), the library transparently selects it.
+
+Where it is used
+----------------
+
+Determinism requirements are consumed today by several ``cub`` device algorithms. See the
+:ref:`CUB determinism guide <cub-determinism>` for the per-algorithm support matrix, the exact
+type/operator constraints, and some examples.
+
+Further reading
+---------------
+
+- `Controlling Floating-Point Determinism in NVIDIA CCCL
+  <https://developer.nvidia.com/blog/controlling-floating-point-determinism-in-nvidia-cccl/>`_ — a
+  deeper walkthrough of the three guarantees and the implementation strategies behind them.
diff --git a/cccl_upstream/docs/cccl/development/build_and_bisect_tools.rst b/cccl_upstream/docs/cccl/development/build_and_bisect_tools.rst
new file mode 100644
index 00000000..f47d2bea
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/build_and_bisect_tools.rst
@@ -0,0 +1,136 @@
+.. _build-and-bisect-tools:
+
+Build and Bisect Utilities
+==========================
+
+``build_and_test_targets.sh``
+-----------------------------
+
+:file:`ci/util/build_and_test_targets.sh` configures, builds, and tests selected
+CMake targets.
+
+Options
+~~~~~~~
+- ``--preset <name>`` - choose a CMake preset.
+- ``--cmake-options <str>`` - extra arguments for the preset configuration.
+- ``--configure-override <cmd>`` - run a custom configuration command instead of
+  a preset. When used, ``--preset`` and ``--cmake-options`` are ignored.
+- ``--build-targets <targets>`` - space separated Ninja targets. If omitted,
+  nothing builds.
+- ``--ctest-targets <regex>`` - space separated CTest ``-R`` patterns. If
+  omitted, nothing runs.
+- ``--lit-precompile-tests <paths>`` - space separated libcudacxx lit test paths
+  to precompile (no run). Paths are relative to ``libcudacxx/test/libcudacxx/``.
+- ``--lit-tests <paths>`` - space separated libcudacxx lit test paths to execute.
+  Paths are relative to ``libcudacxx/test/libcudacxx/``.
+- ``--custom-test-cmd <cmd>`` - arbitrary command executed after build/tests.
+
+Combine with ``.devcontainer/launch.sh -d`` to reproduce CI commands inside a
+container and choose a CUDA toolkit and host compiler:
+``.devcontainer/launch.sh -d [--cuda <XX.Y>] [--host <compiler>] [--gpus all] -- <script>``
+
+Examples
+~~~~~~~~
+Build a single CUB test locally::
+
+  ci/util/build_and_test_targets.sh \
+    --preset cub-cpp20 \
+    --build-targets "cub.cpp20.test.iterator"
+
+Build the same test for SM90 using a CMake option::
+
+  ci/util/build_and_test_targets.sh \
+    --preset cub-cpp20 \
+    --cmake-options "-DCMAKE_CUDA_ARCHITECTURES=90" \
+    --build-targets "cub.cpp20.test.iterator"
+
+Build the test for SM90 with a configure override::
+
+  ci/util/build_and_test_targets.sh \
+    --configure-override "ci/build_cub.sh -configure -arch 90" \
+    --build-targets "cub.cpp20.test.iterator"
+
+Build **and run** a single CUB test locally::
+
+  ci/util/build_and_test_targets.sh \
+    --preset cub-cpp20 \
+    --build-targets "cub.cpp20.test.iterator" \
+    --ctest-targets "cub.cpp20.test.iterator"
+
+Build and run a single CUB test in a devcontainer with specific CTK and host::
+
+  .devcontainer/launch.sh -d --cuda 12.3 --host gcc12 --gpus all -- \
+    ci/util/build_and_test_targets.sh \
+      --preset cub-cpp20 \
+      --build-targets "cub.cpp20.test.iterator" \
+      --ctest-targets "cub.cpp20.test.iterator"
+
+Precompile the libcudacxx lit suite::
+
+  ci/util/build_and_test_targets.sh \
+    --preset libcudacxx \
+    --build-targets libcudacxx.test.lit.precompile
+
+Precompile a single libcudacxx lit test (no execution)::
+
+  ci/util/build_and_test_targets.sh \
+    --preset libcudacxx \
+    --lit-precompile-tests \
+      "std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp"
+
+Execute one or more libcudacxx lit tests::
+
+  ci/util/build_and_test_targets.sh \
+    --preset libcudacxx \
+    --lit-tests \
+      "std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp"
+
+``git_bisect.sh``
+-----------------
+
+:file:`ci/util/git_bisect.sh` wraps ``git bisect`` around the build/test helper.
+It accepts all ``build_and_test_targets.sh`` options plus:
+
+- ``--good-ref <rev>`` - Optional; known good commit, tag, or branch. ``-Nd`` means
+  "N days ago." Defaults to the latest release version tag.
+- ``--bad-ref <rev>`` - Optional; known bad commit. ``-Nd`` means "N days ago."
+  Defaults to ``origin/main``.
+
+Examples
+~~~~~~~~
+Local CUB bisection from latest release to origin/main::
+
+  ci/util/git_bisect.sh \
+    --preset cub-cpp20 \
+    --build-targets "cub.cpp20.test.iterator" \
+    --ctest-targets "cub.cpp20.test.iterator"
+
+Devcontainer CUB bisection from last week::
+
+  .devcontainer/launch.sh -d --cuda 12.3 --host gcc12 --gpus all -- \
+    ci/util/git_bisect.sh \
+      --preset cub-cpp20 \
+      --build-targets "cub.cpp20.test.iterator" \
+      --ctest-targets "cub.cpp20.test.iterator" \
+      --good-ref -7d
+
+Compute-sanitizer example for regression introduced between 3-4 weeks ago::
+
+  .devcontainer/launch.sh -d --cuda 12.9 --host gcc13 --gpus all \
+    --env CCCL_TEST_MODE=compute-sanitizer-initcheck \
+    --env C2H_SEED_COUNT_OVERRIDE=1 \
+    -- ci/util/git_bisect.sh \
+      --preset "cub-cpp20" \
+      --build-targets "cub.cpp20.test.iterator" \
+      --ctest-targets "cub.cpp20.test.iterator" \
+      --good-ref -28d \
+      --bad-ref -21d
+
+Workflow/Bisect
+---------------
+
+A ``Workflow/Bisect`` GitHub Actions job runs ``git_bisect.sh`` on a remote
+runner. Launch it from **Actions → Git Bisect → Run workflow**. Provide
+any desired runner label, refs, preset, targets, or launch arguments. The job
+log streams bisect progress, and the run's **Summary** page renders the final
+Markdown report with culprit commit, PR, reproduction steps, and more.
diff --git a/cccl_upstream/docs/cccl/development/debugger_setup.rst b/cccl_upstream/docs/cccl/development/debugger_setup.rst
new file mode 100644
index 00000000..8257df45
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/debugger_setup.rst
@@ -0,0 +1,95 @@
+.. _cccl-development-module-debugger-setup:
+
+=================
+General Debugging
+=================
+
+Debugger Pretty Printers
+========================
+
+libcudacxx ships custom pretty printers for its types under
+``libcudacxx/share/libcudacxx``. They render CCCL types in a readable form and, for
+device-accessible data, copy the contents back to the host so the elements can be
+inspected. Two independent implementations are provided:
+
+- ``libcudacxx/share/libcudacxx/gdb`` - printers for GDB.
+- ``libcudacxx/share/libcudacxx/lldb`` - printers for LLDB.
+
+Each directory has an ``__init__.py`` entry point that registers every printer. The
+repository root contains a ``.gdbinit`` and a ``.lldbinit`` that load the matching entry
+point for you, so the simplest way to enable the printers is to let the debugger pick up
+these files.
+
+.. important::
+
+   ``lldb`` and ``gdb`` only inspect the following locations when looking for init
+   dotfiles (in the given order):
+
+   #. Home directory config files (usually ``~/.lldbinit`` or ``~/.config/gdb/gdbinit`` on
+      Linux, but check the respective manuals for specifics).
+   #. The current working directory.
+
+   They do **not** walk up the directory stack like most tools. So if you have a
+   ``.lldbinit`` in the parent directory, ``lldb`` will **not** load it. For this reason,
+   you **must** run the debugger from the root CCCL directory in order for automatic
+   loading of the pretty printers to work.
+
+
+In addition to not loading parent directory dot-files, ``gdb`` or ``lldb`` will load not
+dotfiles unless you explicitly allow them. The following sections explain how to enable
+this for each debugger.
+
+.. note::
+
+   The following is **not** needed when working inside a devcontainer. devcontainers
+   already have the following set up.
+
+   If they don't, and automatic loading of the pretty printers does not work, then this is
+   a bug and should be fixed.
+
+   It is only needed for bare metal builds.
+
+GDB
+---
+
+By default GDB does not source a ``.gdbinit`` from the current directory, and it guards
+auto-loaded scripts with the ``auto-load safe-path`` setting. Add the repository root to
+your ``~/.gdbinit`` (or ``~/.config/gdb/gdbinit`` if you have ``XDG_CONFIG_HOME`` set) so
+the project's ``.gdbinit`` is trusted and loaded::
+
+  add-auto-load-safe-path /absolute/path/to/cccl
+  set auto-load local-gdbinit on
+
+Launch GDB from the repository root and the printers should register automatically.
+
+Verify that the printers are active with ``info pretty-printer``.
+
+To load the printers without depending on the working directory - for example
+from a global ``~/.gdbinit`` - ``source`` the entry point by absolute path
+instead::
+
+  source /absolute/path/to/cccl/libcudacxx/share/libcudacxx/gdb/__init__.py
+
+``source`` runs the script directly and is not subject to the ``auto-load safe-path``
+restriction.
+
+LLDB
+----
+
+LLDB only reads ``.lldbinit`` from your home directory unless you opt in to loading one
+from the current working directory. Enable that once in your ``~/.lldbinit`` (``lldb``
+seemingly does not respect ``XDG_CONFIG_HOME``)::
+
+  settings set target.load-cwd-lldbinit true
+
+This is a trust decision, since the local file runs arbitrary Python. Launch LLDB from the
+repository root and the project's ``.lldbinit`` imports the formatters automatically.
+
+To load the formatters without depending on the working directory, add the absolute path
+to your ``~/.lldbinit`` instead::
+
+  command script import "/absolute/path/to/cccl/libcudacxx/share/libcudacxx/lldb/__init__.py"
+
+The entry point's ``__lldb_init_module`` hook defines and enables an LLDB type category
+for the formatters. Print any CCCL value with the usual commands (``v``, ``frame
+variable``, or ``dwim-print``).
diff --git a/cccl_upstream/docs/cccl/development/index.rst b/cccl_upstream/docs/cccl/development/index.rst
new file mode 100644
index 00000000..a1ea0be1
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/index.rst
@@ -0,0 +1,23 @@
+.. _cccl-development-module:
+
+CCCL Development Guide
+======================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   macro
+   testing
+   build_and_bisect_tools
+   visibility
+   debugger_setup
+
+This living document serves to describe the internal details and the development process of CCCL libraries.
+
+Documentation:
+
+- :doc:`CCCL Internal Macros <macro>`
+- :doc:`CCCL Testing Utilities <testing>`
+- :doc:`CCCL Bisect And Targeted Build/Test Helpers <build_and_bisect_tools>`
+- :doc:`General Debugging <debugger_setup>`
diff --git a/cccl_upstream/docs/cccl/development/macro.rst b/cccl_upstream/docs/cccl/development/macro.rst
new file mode 100644
index 00000000..61070a57
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/macro.rst
@@ -0,0 +1,553 @@
+.. _cccl-development-module-macros:
+
+CCCL Internal Macros
+====================
+
+The document describes the main *internal* macros used by CCCL. They are not intended to be used by end users, but for development of CCCL features only. We reserve the right to change them at any time without warning.
+
+----
+
+Compiler Macros
+---------------
+
+**Host compiler macros**:
+
++------------------------------+---------------------------------------------+
+| ``_CCCL_COMPILER(CLANG)``    | Clang                                       |
++------------------------------+---------------------------------------------+
+| ``_CCCL_COMPILER(GCC)``      | GCC                                         |
++------------------------------+---------------------------------------------+
+| ``_CCCL_COMPILER(NVHPC)``    | Nvidia HPC compiler                         |
++------------------------------+---------------------------------------------+
+| ``_CCCL_COMPILER(MSVC)``     | Microsoft Visual Studio                     |
++------------------------------+---------------------------------------------+
+| ``_CCCL_COMPILER(MSVC2019)`` | Microsoft Visual Studio 2019                |
++------------------------------+---------------------------------------------+
+| ``_CCCL_COMPILER(MSVC2022)`` | Microsoft Visual Studio 2022                |
++------------------------------+---------------------------------------------+
+
+The ``_CCCL_COMPILER`` function-like macro can also be used to check the version of a compiler.
+
+.. code:: cpp
+
+   _CCCL_COMPILER(MSVC, <, 19, 24)
+   _CCCL_COMPILER(GCC, >=, 9)
+
+*Note*: When used without specifying a minor version number, the macro will only test against
+the compiler's major version number. For example, when the compiler is ``gcc-9.1``, the macro
+``_CCCL_COMPILER(GCC, >, 9)`` will be ``false`` even though ``9.1`` is greater than ``9``.
+
+**CUDA compiler macros**:
+
++--------------------------------+-------------------------+
+| ``_CCCL_CUDA_COMPILER(NVCC)``  | Nvidia compiler         |
++--------------------------------+-------------------------+
+| ``_CCCL_CUDA_COMPILER(NVHPC)`` | Nvidia HPC compiler     |
++--------------------------------+-------------------------+
+| ``_CCCL_CUDA_COMPILER(NVRTC)`` | Nvidia Runtime Compiler |
++--------------------------------+-------------------------+
+| ``_CCCL_CUDA_COMPILER(CLANG)`` | Clang                   |
++--------------------------------+-------------------------+
+
+The ``_CCCL_CUDA_COMPILER`` function-like macro can also be used to check the version of a CUDA compiler.
+
+.. code:: cpp
+
+   _CCCL_CUDA_COMPILER(NVCC, <, 12, 3)
+   _CCCL_CUDA_COMPILER(CLANG, >=, 14)
+
+*Note*: ``_CCCL_CUDA_COMPILER(...)`` check may result in a ``true`` value even during the compilation of a C++ source
+file. Use ``_CCCL_CUDA_COMPILATION()`` to check for the compilation of a CUDA source file.
+
+**CUDA identification/version macros**:
+
++----------------------------------+------------------------------------------------------------------------------------------------+
+| ``_CCCL_CUDA_COMPILATION()``     | CUDA code is being compiled                                                                    |
++----------------------------------+------------------------------------------------------------------------------------------------+
+| ``_CCCL_HOST_COMPILATION()``     | Compiling host code, ``true`` when executing the CUDA host pass or compiling a C++ source file |
++----------------------------------+------------------------------------------------------------------------------------------------+
+| ``_CCCL_DEVICE_COMPILATION()``   | Compiling device code, ``true`` when executing the CUDA device pass                            |
++----------------------------------+------------------------------------------------------------------------------------------------+
+| ``_CCCL_CUDACC_BELOW(12, 7)``    | CUDA version below 12.7 when compiling a CUDA source file                                      |
++----------------------------------+------------------------------------------------------------------------------------------------+
+| ``_CCCL_CUDACC_AT_LEAST(12, 7)`` | CUDA version at least 12.7 when compiling a CUDA source file                                   |
++----------------------------------+------------------------------------------------------------------------------------------------+
+
+*Note*: When compiling CUDA code with ``nvc++`` both ``_CCCL_HOST_COMPILATION()`` and ``_CCCL_DEVICE_COMPILATION()`` result in a ``true`` value.
+
+**PTX macros**:
+
++----------------------+-------------------------------------------------------------------------------------------------------------------+
+| ``_CCCL_PTX_ARCH()`` | Alias of ``__CUDA_ARCH__`` with value equal to 0 if a CUDA compiler is not available                              |
++----------------------+-------------------------------------------------------------------------------------------------------------------+
+| ``__cccl_ptx_isa``   | PTX ISA version available with the current CUDA compiler, e.g. PTX ISA 8.4 (``840``) is available from CUDA 12.4  |
++----------------------+-------------------------------------------------------------------------------------------------------------------+
+
+*Note*: When compiling CUDA code with ``nvc++`` the ``_CCCL_PTX_ARCH()`` macro expands to ``0``.
+
+----
+
+Architecture Macros
+-------------------
+
+The following macros are used to check the target architecture. They comply with the compiler supported by the CUDA toolkit. Compilers outside the CUDA toolkit may define such macros in a different way.
+
++------------------------------+---------------------------------------------------+
+| ``_CCCL_HOST_ARCH(ARM64)``   |  ARM 64-bit, including MSVC emulation             |
++------------------------------+---------------------------------------------------+
+| ``_CCCL_HOST_ARCH(X86_64)``  |  X86 64-bit. False on ARM 64-bit MSVC emulation   |
++------------------------------+---------------------------------------------------+
+
+----
+
+OS Macros
+---------
+
++-----------------------+---------------------------------+
+| ``_CCCL_OS(WINDOWS)`` | Windows, including NVRTC LLP64  |
++-----------------------+---------------------------------+
+| ``_CCCL_OS(LINUX)``   | Linux, including NVRTC LP64     |
++-----------------------+---------------------------------+
+| ``_CCCL_OS(ANDROID)`` | Android                         |
++-----------------------+---------------------------------+
+| ``_CCCL_OS(QNX)``     | QNX                             |
++-----------------------+---------------------------------+
+
+----
+
+Execution Space
+---------------
+
+**Functions**
+
++-----------------------+-----------------------+
+| ``_CCCL_HOST``        | Host function         |
++-----------------------+-----------------------+
+| ``_CCCL_DEVICE``      | Device function       |
++-----------------------+-----------------------+
+| ``_CCCL_HOST_DEVICE`` | Host/Device function  |
++-----------------------+-----------------------+
+
+In addition, ``_CCCL_EXEC_CHECK_DISABLE`` disables the execution space check for the NVHPC compiler
+
+**Target Macros**
+
++---------------------------------------------------------------------------------+--------------------------------------------------------------------------+
+| ``NV_IF_TARGET(TARGET, (CODE))``                                                | Enable ``CODE`` only if ``TARGET`` is satisfied.                         |
++---------------------------------------------------------------------------------+--------------------------------------------------------------------------+
+| ``NV_IF_ELSE_TARGET(TARGET, (IF_CODE), (ELSE_CODE))``                           | Enable ``CODE_IF`` if ``TARGET`` is satisfied, ``CODE_ELSE`` otherwise.  |
++---------------------------------------------------------------------------------+--------------------------------------------------------------------------+
+| ``NV_DISPATCH_TARGET(TARGET1, (TARGET1_CODE), ..., TARGET_N, (TARGET_N_CODE))`` | Enable a single code block if any of ``TARGET_i`` is satisfied.          |
++---------------------------------------------------------------------------------+--------------------------------------------------------------------------+
+
+Possible ``TARGET`` values:
+
++---------------------------+-------------------------------------------------------------------+
+| ``NV_ANY_TARGET``         | Any target                                                        |
++---------------------------+-------------------------------------------------------------------+
+| ``NV_IS_HOST``            | Host-code target                                                  |
++---------------------------+-------------------------------------------------------------------+
+| ``NV_IS_DEVICE``          | Device-code target                                                |
++---------------------------+-------------------------------------------------------------------+
+| ``NV_PROVIDES_SM_<VER>``  | SM architecture is at least ``VER``, e.g. ``NV_PROVIDES_SM_80``   |
++---------------------------+-------------------------------------------------------------------+
+| ``NV_IS_EXACTLY_SM_<NN>`` | SM architecture is exactly ``VER``, e.g. ``NV_IS_EXACTLY_SM_80``  |
++---------------------------+-------------------------------------------------------------------+
+
+Usage example:
+
+.. code-block:: c++
+
+    NV_IF_TARGET(NV_IS_DEVICE,    (auto x = threadIdx.x; return x;));
+    NV_IF_ELSE_TARGET(NV_IS_HOST, (return 0;), (auto x = threadIdx.x; return x;));
+    NV_DISPATCH_TARGET(NV_PROVIDES_SM_90,   (return "Hopper+";),
+                       NV_IS_EXACTLY_SM_75, (return "Turing";),
+                       NV_IS_HOST,          (return "Host";))
+
+*Pitfalls*:
+
+* All target macros generate the code in a local scope, i.e. ``{ code }``.
+* ``NV_DISPATCH_TARGET`` is *NOT* a switch statement. It enables the code associated with the first condition satisfied.
+* The target macros take ``code`` as an argument, so it is *not* possible to use any conditional compilation, .e.g ``#if _CCCL_STD_VER >= 20`` within a target macro
+
+----
+
+CUDA attributes
+---------------
+
++------------------------------+----------------------------------------------------------+
+| ``_CCCL_GRID_CONSTANT``      | Grid constant kernel parameter                           |
++------------------------------+----------------------------------------------------------+
+| ``_CCCL_GLOBAL_CONSTANT``    | Host/device global scope constant (``inline constexpr``) |
++------------------------------+----------------------------------------------------------+
+
+----
+
+CUDA Toolkit macros
+-------------------
+
++-------------------------------+-----------------------------------------------------------------------------------------------------------------------------+
+| ``_CCCL_HAS_CTK()``           | CUDA toolkit is available if ``_CCCL_CUDA_COMPILER()`` evaluates to a ``true`` value or if ``cuda_runtime_api.h`` was found |
++-------------------------------+-----------------------------------------------------------------------------------------------------------------------------+
+| ``_CCCL_CTK_BELOW``           | CUDA toolkit version below 12.7                                                                                             |
++-------------------------------+-----------------------------------------------------------------------------------------------------------------------------+
+| ``_CCCL_CTK_AT_LEAST(12, 7)`` | CUDA toolkit version at least 12.7                                                                                          |
++-------------------------------+-----------------------------------------------------------------------------------------------------------------------------+
+
+Non-standard Types Support
+--------------------------
+
++------------------------------+-------------------------------------------------------------------------------------------------------------------------------+
+| ``_CCCL_HAS_INT128()``       | ``__int128`` and ``__uint128_t`` for 128-bit integer are supported and enabled                                                |
++------------------------------+-------------------------------------------------------------------------------------------------------------------------------+
+| ``_CCCL_HAS_NVFP8()``        | ``__nv_fp8_e5m2/__nv_fp8_e4m3/__nv_fp8_e8m0`` data types are supported and enabled.  Prefer over ``__CUDA_FP8_TYPES_EXIST__`` |
++------------------------------+-------------------------------------------------------------------------------------------------------------------------------+
+| ``_CCCL_HAS_NVFP16()``       | ``__half/__half2`` data types are supported and enabled. Prefer over ``__CUDA_FP16_TYPES_EXIST__``                            |
++------------------------------+-------------------------------------------------------------------------------------------------------------------------------+
+| ``_CCCL_HAS_NVBF16()``       | ``__nv_bfloat16/__nv_bfloat162`` data types are supported and enabled.  Prefer over ``__CUDA_BF16_TYPES_EXIST__``             |
++------------------------------+-------------------------------------------------------------------------------------------------------------------------------+
+| ``_CCCL_HAS_FLOAT128()``     | ``__float128`` for 128-bit floating-point are supported and enabled                                                           |
++------------------------------+-------------------------------------------------------------------------------------------------------------------------------+
+
++-----------------------------------+-------------------------------------------------------------------------+
+| ``CCCL_DISABLE_INT128_SUPPORT``   | Disable ``__int128/__uint128_t`` support                                |
++-----------------------------------+-------------------------------------------------------------------------+
+| ``CCCL_DISABLE_NVFP8_SUPPORT``    | Disable ``__nv_fp8_e5m2/__nv_fp8_e4m3/__nv_fp8_e8m0`` support           |
++-----------------------------------+-------------------------------------------------------------------------+
+| ``CCCL_DISABLE_NVFP16_SUPPORT``   | Disable ``__half/__half2`` support                                      |
++-----------------------------------+-------------------------------------------------------------------------+
+| ``CCCL_DISABLE_NVBF16_SUPPORT``   | Disable ``__nv_bfloat16/__nv_bfloat162`` support                        |
++-----------------------------------+-------------------------------------------------------------------------+
+| ``CCCL_DISABLE_FLOAT128_SUPPORT`` | Disable ``__float128`` support                                          |
++-----------------------------------+-------------------------------------------------------------------------+
+
++-----------------------------------+-------------------------------------------------------------------------+
+| ``_LIBCUDACXX_HAS_NVFP16()``      | ``__half/__half2`` host/device are supported  (CUDA 12.2+)              |
++-----------------------------------+-------------------------------------------------------------------------+
+| ``_LIBCUDACXX_HAS_NVBF16()``      | ``__nv_bfloat16/__nv_bfloat162`` host/device are supported (CUDA 12.2+) |
++-----------------------------------+-------------------------------------------------------------------------+
+
+----
+
+C++ Language Macros
+-------------------
+
+The following macros are required only if the target C++ version does not support the corresponding attribute
+
++-----------------------------+----------------------------------------------------------+
+| ``_CCCL_STD_VER``           | C++ standard version, e.g. ``#if _CCCL_STD_VER >= 2017`` |
++-----------------------------+----------------------------------------------------------+
+| ``_CCCL_CONSTEXPR_CXX20``   | Enable ``constexpr`` for C++20 or newer                  |
++-----------------------------+----------------------------------------------------------+
+| ``_CCCL_CONSTEXPR_CXX23``   | Enable ``constexpr`` for C++23 or newer                  |
++-----------------------------+----------------------------------------------------------+
+| ``_CCCL_HAS_EXCEPTIONS()``  | Features can use exceptions, e.g ``bad_optional_access`` |
++-----------------------------+----------------------------------------------------------+
+
+**Concept-like Macros**:
+
++------------------------+--------------------------------------------------------------------------------------------+
+| ``_CCCL_TEMPLATE(X)``  | ``template`` clause                                                                        |
++------------------------+--------------------------------------------------------------------------------------------+
+| ``_CCCL_REQUIRES(X)``  | ``requires`` clause                                                                        |
++------------------------+--------------------------------------------------------------------------------------------+
+| ``_CCCL_AND``          | Traits conjunction only used with ``_CCCL_REQUIRES``                                       |
++------------------------+--------------------------------------------------------------------------------------------+
+
+Usage example:
+
+.. code-block:: c++
+
+    _CCCL_TEMPLATE(typename T)
+    _CCCL_REQUIRES(is_integral_v<T> _CCCL_AND(sizeof(T) > 1))
+
+.. code-block:: c++
+
+    _CCCL_TEMPLATE(typename T)
+    _CCCL_REQUIRES(is_arithmetic_v<T> _CCCL_AND (!is_integral_v<T>))
+
+
+**Portable feature testing**:
+
++--------------------------+--------------------------------------------------+
+| ``_CCCL_HAS_BUILTIN(X)`` |  Portable ``__has_builtin(X)``                   |
++--------------------------+--------------------------------------------------+
+| ``_CCCL_HAS_FEATURE(X)`` |  Portable ``__has_feature(X)``                   |
++--------------------------+--------------------------------------------------+
+
+**Portable attributes**:
+
++----------------------------------+------------------------------------------------------------------------------+
+| ``_CCCL_ASSUME(EXPR)``           | Portable ``[[assume]]`` attribute (before C++23)                             |
++----------------------------------+------------------------------------------------------------------------------+
+| ``_CCCL_NO_UNIQUE_ADDRESS``      | Portable ``[[no_unique_address]]`` attribute                                 |
++----------------------------------+------------------------------------------------------------------------------+
+| ``CCCL_DEPRECATED``              | Portable ``[[deprecated]]`` attribute (before C++14)                         |
++----------------------------------+------------------------------------------------------------------------------+
+| ``CCCL_DEPRECATED_BECAUSE(MSG)`` | Portable ``[[deprecated]]`` attribute with custom message (before C++14)     |
++----------------------------------+------------------------------------------------------------------------------+
+| ``_CCCL_FORCEINLINE``            | Portable "always inline" attribute                                           |
++----------------------------------+------------------------------------------------------------------------------+
+| ``_CCCL_PURE``                   | Portable "pure" function attribute                                           |
++----------------------------------+------------------------------------------------------------------------------+
+| ``_CCCL_CONST``                  | Portable "constant" function attribute                                       |
++----------------------------------+------------------------------------------------------------------------------+
+| ``_CCCL_LIFETIMEBOUND``          | Portable "lifetime bound" function attribute                                 |
++----------------------------------+------------------------------------------------------------------------------+
+
+**Portable Builtin Macros**:
+
++---------------------------------------+--------------------------------------------+
+| ``_CCCL_UNREACHABLE()``               | Portable ``__builtin_unreachable()``       |
++---------------------------------------+--------------------------------------------+
+| ``_CCCL_BUILTIN_EXPECT(X)``           | Portable ``__builtin_expected(X)``         |
++---------------------------------------+--------------------------------------------+
+| ``_CCCL_BUILTIN_PREFETCH(X[, Y, Z])`` | Portable ``__builtin_prefetch(X, Y, Z)``   |
++---------------------------------------+--------------------------------------------+
+
+**Portable Keyword Macros**
+
++-----------------------------+--------------------------------------------+
+| ``_CCCL_RESTRICT``          | Portable ``restrict`` keyword              |
++-----------------------------+--------------------------------------------+
+| ``_CCCL_ALIGNAS(X)``        | Portable ``alignas(X)`` keyword (variable) |
++-----------------------------+--------------------------------------------+
+| ``_CCCL_ALIGNAS_TYPE(X)``   | Portable ``alignas(X)`` keyword (type)     |
++-----------------------------+--------------------------------------------+
+| ``_CCCL_PRAGMA(X)``         | Portable ``_Pragma(X)`` keyword            |
++-----------------------------+--------------------------------------------+
+
+**Portable Pragma Macros**
+
++--------------------------------+-------------------------------------------+
+| ``_CCCL_PRAGMA_UNROLL(N)``     | Portable ``#pragma unroll N`` pragma      |
++--------------------------------+-------------------------------------------+
+| ``_CCCL_PRAGMA_UNROLL_FULL()`` | Portable ``#pragma unroll`` pragma        |
++--------------------------------+-------------------------------------------+
+| ``_CCCL_PRAGMA_NOUNROLL()``    | Portable ``#pragma nounroll`` pragma      |
++--------------------------------+-------------------------------------------+
+
+**Conditional Constant Evaluation Macros**
+
+In C++23, the ``if consteval`` statement (`link <https://en.cppreference.com/w/cpp/language/if.html>`_) was introduced. CCCL mimics the behaviour with a set of macros that expand to an implementation supported by the compiler. If the compiler doesn't support any kind of conditional constant evaluation, the macros expand to predefined fallback values.
+
++------------------------------------+-----------------------------------------------------------------------------------+
+| ``_CCCL_IF_CONSTEVAL``             | Equivalent to ``if consteval`` statement (fallbacks to ``if constexpr (false)``)  |
++------------------------------------+-----------------------------------------------------------------------------------+
+| ``_CCCL_IF_CONSTEVAL_DEFAULT``     | Equivalent to ``if consteval`` statement (fallbacks to ``if constexpr (true)``)   |
++------------------------------------+-----------------------------------------------------------------------------------+
+| ``_CCCL_IF_NOT_CONSTEVAL``         | Equivalent to ``if !consteval`` statement (fallbacks to ``if constexpr (true)``)  |
++------------------------------------+-----------------------------------------------------------------------------------+
+| ``_CCCL_IF_NOT_CONSTEVAL_DEFAULT`` | Equivalent to ``if !consteval`` statement (fallbacks to ``if constexpr (false)``) |
++------------------------------------+-----------------------------------------------------------------------------------+
+
+**Exception Macros**
+
+CUDA doesn't support exceptions in device code, however, sometimes we need to write host/device functions that use exceptions on host and ``__trap()`` on device. CCCL provides a set of macros that should be used in place of the standard C++ keywords to make the code compile in both, host and device code.
+
++-----------------------------+------------------------------------------------------------------------------------------------------------------+
+| ``_CCCL_TRY``               | Replacement for the ``try`` keyword.                                                                             |
++-----------------------------+------------------------------------------------------------------------------------------------------------------+
+| ``_CCCL_CATCH (X)``         | Replacement for the ``catch (/*X*/)`` statement.                                                                 |
++-----------------------------+------------------------------------------------------------------------------------------------------------------+
+| ``_CCCL_CATCH_ALL``         | Replacement for the ``catch (...)`` statement.                                                                   |
++-----------------------------+------------------------------------------------------------------------------------------------------------------+
+| ``_CCCL_THROW(X, ...)``     | Replacement for the ``throw X(...)`` expression. ``X`` must be fully qualified type, without the leading ``::``. |
++-----------------------------+------------------------------------------------------------------------------------------------------------------+
+| ``_CCCL_RETHROW``           | Replacement for the plain ``throw`` expression.                                                                  |
++-----------------------------+------------------------------------------------------------------------------------------------------------------+
+
+*Note*: The ``_CCCL_CATCH`` clause must always introduce a named variable, like: ``_CCCL_CATCH(const exception_type& var)``.
+
+.. note::
+
+  ``_CCCL_THROW`` requires to include the ``<stdexcept>`` header, regardless exceptions are enabled or not.
+
+Example:
+
+.. code-block:: c++
+
+    __host__ __device__ void* alloc(cuda::std::size_t nbytes)
+    {
+        if (void* ptr = cuda::std::malloc(nbytes))
+        {
+            return ptr;
+        }
+        _CCCL_THROW(std::bad_alloc); // on device calls cuda::std::terminate()
+    }
+
+    __host__ __device__ void do_something(int* buff)
+    {
+        _CCCL_THROW(std::runtime_error, "Something went wrong"); // on device calls cuda::std::terminate()
+    }
+
+    __host__ __device__ void fn(cuda::std::size_t n)
+    {
+        int* buff{};
+        _CCCL_TRY
+        {
+            buff = reinterpret_cast<int*>(alloc(n * sizeof(int)));
+            do_something(buff);
+        }
+        _CCCL_CATCH ([[maybe_unused]] const std::bad_alloc& e) // must be always named
+        {
+            std::fprintf(stderr, "Failed to allocate memory\n"); // We can directly call host-only functions
+            cuda::std::terminate();
+        }
+        _CCCL_CATCH_ALL // or _CCCL_CATCH_FALLTHOUGH
+        {
+            cuda::std::free(buff);
+            _CCCL_RETHROW;
+        }
+    }
+
+    __global__ void kernel()
+    {
+        fn(10);
+    }
+
+    int main()
+    {
+        fn(10);
+        return 0;
+    }
+
+----
+
+Visibility Macros
+-----------------
+
++-------------------------------+-----------------------------------------------------------------------------------------------------+
+| ``_CCCL_VISIBILITY_HIDDEN``   | Hidden visibility attribute (e.g. ``__attribute__((visibility("hidden")))``)                        |
++-------------------------------+-----------------------------------------------------------------------------------------------------+
+| ``_CCCL_HIDE_FROM_ABI``       | Hidden visibility (i.e. ``inline``, not exported, not instantiated)                                 |
++-------------------------------+-----------------------------------------------------------------------------------------------------+
+| ``_CCCL_API``                 | Host/device function with hidden visibility. Most CCCL functions are hidden with this attribute     |
++-------------------------------+-----------------------------------------------------------------------------------------------------+
+| ``_CCCL_HOST_API``            | Host function with hidden visibility. Most CCCL functions are hidden with this attribute            |
++-------------------------------+-----------------------------------------------------------------------------------------------------+
+| ``_CCCL_DEVICE_API``          | Device function with hidden visibility. Most CCCL functions are hidden with this attribute          |
++-------------------------------+-----------------------------------------------------------------------------------------------------+
+| ``_CCCL_KERNEL_ATTRIBUTES``   | Global function with hidden visibility. Most CCCL functions are hidden with this attribute          |
++-------------------------------+-----------------------------------------------------------------------------------------------------+
+
+----
+
+Other Common Macros
+-------------------
+
++-----------------------------+--------------------------------------------+
+| ``_CCCL_TO_STRING(X)``      | ``X`` to literal string                    |
++-----------------------------+--------------------------------------------+
+| ``_CCCL_DOXYGEN_INVOKED``   | Defined during Doxygen parsing             |
++-----------------------------+--------------------------------------------+
+
+----
+
+Debugging Macros
+----------------
+
++-----------------------------------+-------------------------------------------------------------------------------------------------------------+
+| ``_CCCL_ASSERT(COND, MSG)``       | Portable, conditional CCCL `assert()` macro. Requires (``CCCL_ENABLE_ASSERTIONS`` or a debug build)         |
++-----------------------------------+-------------------------------------------------------------------------------------------------------------+
+| ``_CCCL_VERIFY(COND, MSG)``       | Portable, always-on `assert()` reserved for critical checks that are always required                        |
++-----------------------------------+-------------------------------------------------------------------------------------------------------------+
+| ``_CCCL_ENABLE_ASSERTIONS``       | Enable assertions                                                                                           |
++-----------------------------------+-------------------------------------------------------------------------------------------------------------+
+| ``CCCL_ENABLE_HOST_ASSERTIONS``   | Enable host-side assertions                                                                                 |
++-----------------------------------+-------------------------------------------------------------------------------------------------------------+
+| ``CCCL_ENABLE_DEVICE_ASSERTIONS`` | Enable device-side assertions                                                                               |
++-----------------------------------+-------------------------------------------------------------------------------------------------------------+
+| ``_CCCL_ENABLE_DEBUG_MODE``       | Enable debug mode (and assertions)                                                                          |
++-----------------------------------+-------------------------------------------------------------------------------------------------------------+
+
+----
+
+Warning Suppression Macros
+--------------------------
+
++-----------------------------+--------------------------------------------+
+| ``_CCCL_DIAG_PUSH``         | Portable ``#pragma push``                  |
++-----------------------------+--------------------------------------------+
+| ``_CCCL_DIAG_POP``          | Portable ``#pragma pop``                   |
++-----------------------------+--------------------------------------------+
+
+**Compiler-specific Suppression Macros**:
+
++-------------------------------------+-------------------------------------------------------------+
+| ``_CCCL_DIAG_SUPPRESS_CLANG(X)``    | Suppress clang warning, e.g. ``"-Wattributes"``             |
++-------------------------------------+-------------------------------------------------------------+
+| ``_CCCL_DIAG_SUPPRESS_GCC(X)``      | Suppress gcc warning, e.g. ``"-Wattributes"``               |
++-------------------------------------+-------------------------------------------------------------+
+| ``_CCCL_DIAG_SUPPRESS_NVHPC(X)``    | Suppress nvhpc warning, e.g. ``expr_has_no_effect``         |
++-------------------------------------+-------------------------------------------------------------+
+| ``_CCCL_DIAG_SUPPRESS_MSVC(X)``     | Suppress msvc warning, e.g. ``4127``                        |
++-------------------------------------+-------------------------------------------------------------+
+| ``_CCCL_BEGIN_NV_DIAG_SUPPRESS(X)`` | Start to suppress nvcc warning, e.g. ``177``                |
++-------------------------------------+-------------------------------------------------------------+
+| ``_CCCL_END_NV_DIAG_SUPPRESS()``    | End to suppress nvcc warning                                |
++-------------------------------------+-------------------------------------------------------------+
+
+Usage example:
+
+.. code-block:: c++
+
+    _CCCL_DIAG_PUSH
+    _CCCL_DIAG_SUPPRESS_GCC("-Wattributes")
+    // code ..
+    _CCCL_DIAG_POP
+
+----
+
+Freestanding support
+--------------------------
+
+We - partially - support building CCCL headers in freestanding mode, for example JIT compilation with NVRTC.
+
++-----------------------------+----------------------------------------------------+
+| ``_CCCL_HOSTED()``          | "Normal" compilation mode with host STL support    |
++-----------------------------+----------------------------------------------------+
+| ``_CCCL_FREESTANDING()``    | Freestanding compilation mode, no host STL support |
++-----------------------------+----------------------------------------------------+
+| ``_CCCL_HOSTJIT()``         | Freestanding compilation mode, with host compiler  |
++-----------------------------+----------------------------------------------------+
+
+Usage example:
+
+.. code-block:: c++
+
+    #if _CCCL_HOSTED()
+    #  include <iostream> // Host STL header not available in freestanding
+    #endif // _CCCL_HOSTED()
+
+    // code ..
+
+Similarly we also provide macros to detect which host standard library is available
+
++-----------------------------------+----------------------------------------------------+
+| ``_CCCL_HAS_HOST_STD_LIB()``      | Whether a known host standard library is available |
++-----------------------------------+----------------------------------------------------+
+| ``_CCCL_HOST_STD_LIB(LIBSTDCXX)`` | libstdc++ is available as host standard library    |
++-----------------------------------+----------------------------------------------------+
+| ``_CCCL_HOST_STD_LIB(LIBCXX)``    | libc++ is available as host standard library       |
++-----------------------------------+----------------------------------------------------+
+| ``_CCCL_HOST_STD_LIB(STL)``       | MSVC STL is available as host standard library     |
++-----------------------------------+----------------------------------------------------+
+
+Usage example:
+
+.. code-block:: c++
+
+    #if _CCCL_HAS_HOST_STD_LIB()
+    _CCCL_BEGIN_NAMESPACE_STD
+
+    #  if _CCCL_HOST_STD_LIB(STL)
+    template <class _Tp, size_t _Size>
+    class array;
+    #  else // ^^^ _CCCL_HOST_STD_LIB(STL) ^^^ / vvv !_CCCL_HOST_STD_LIB(STL) vvv
+    template <class _Tp, size_t _Size>
+    struct array;
+    #  endif // !_CCCL_HOST_STD_LIB(STL)
+
+    _CCCL_END_NAMESPACE_STD
+    #endif // _CCCL_HAS_HOST_STD_LIB()
diff --git a/cccl_upstream/docs/cccl/development/testing.rst b/cccl_upstream/docs/cccl/development/testing.rst
new file mode 100644
index 00000000..ac4f2366
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/testing.rst
@@ -0,0 +1,138 @@
+.. _cccl-development-module-testing:
+
+======================
+CCCL Testing Utilities
+======================
+
+This document describes utilities provided for implementing the *internal* CCCL tests.
+They are not intended to be used by end users, but for development of CCCL features only.
+We reserve the right to change them at any time without warning.
+
+----
+
+-------------------------------------------------------------------
+Test Variants: Generating Multiple Executables from a Single Source
+-------------------------------------------------------------------
+
+Some of CCCL's tests are very slow to build and are capable of exhausting RAM
+during compilation/linking. To avoid such issues, large tests are split into
+multiple executables to take advantage of parallel computation and reduce memory
+usage.
+
+CCCL facilitates this by providing a CMake-based solution for automatically generating multiple
+test executables from a single source file. This is done by using one or more ``%PARAM%`` comments
+in the test's source code, each of which defines a parameter that will be split across multiple
+executables.
+
+The CMake functions that implement this feature are in ``cmake/CCCLTestParams.cmake``.
+An example of their usage is provided below.
+
+Using ``%PARAM%``
+-----------------
+
+The ``%PARAM%`` hint provides an automated method of generating multiple test
+executables from a single source file. To use it, add one or more special
+comments to the test source file::
+
+  // %PARAM% [definition] [label] [values]
+
+CMake will parse the source file and extract these comments, using them to
+generate multiple test executables for the full cartesian product of values.
+
+- ``definition`` will be used as a preprocessor definition name. By convention,
+  these begin with ``TEST_``.
+- ``label`` is a short, human-readable label that will be used in the test
+  executable's name to identify the test variant.
+- ``values`` is a colon-separated list of values used during test generation. Only
+  numeric values have been tested.
+
+Example
+*******
+
+A source file containing the following hints::
+
+  // %PARAM% TEST_FOO foo 0:1:2
+  // %PARAM% TEST_LAUNCH lid 0:1
+
+will generate six variants with unique preprocessor definitions:
+
++-----------------------------+-------------------------------------------+
+| Executable Name             | Preprocessor Definitions                  |
++=============================+===========================================+
+| ``<name_base>.foo_0.lid_0`` | ``-DTEST_FOO=0 -DTEST_LAUNCH=0 VAR_ID=0`` |
++-----------------------------+-------------------------------------------+
+| ``<name_base>.foo_0.lid_1`` | ``-DTEST_FOO=0 -DTEST_LAUNCH=1 VAR_ID=1`` |
++-----------------------------+-------------------------------------------+
+| ``<name_base>.foo_1.lid_0`` | ``-DTEST_FOO=1 -DTEST_LAUNCH=0 VAR_ID=2`` |
++-----------------------------+-------------------------------------------+
+| ``<name_base>.foo_1.lid_1`` | ``-DTEST_FOO=1 -DTEST_LAUNCH=1 VAR_ID=3`` |
++-----------------------------+-------------------------------------------+
+| ``<name_base>.foo_2.lid_0`` | ``-DTEST_FOO=2 -DTEST_LAUNCH=0 VAR_ID=4`` |
++-----------------------------+-------------------------------------------+
+| ``<name_base>.foo_2.lid_1`` | ``-DTEST_FOO=2 -DTEST_LAUNCH=1 VAR_ID=5`` |
++-----------------------------+-------------------------------------------+
+
+Changing ``%PARAM%`` Hints
+**************************
+
+Since CMake does not automatically reconfigure the build when source files are
+modified, CMake will need to be rerun manually whenever the ``%PARAM%`` comments
+change.
+
+Using the CMake Variant Functions
+---------------------------------
+
+``cmake/CCCLTestParams.cmake`` provides the functions that implement this functionality.
+See that file for detailed documentation. An example of their usage is:
+
+..  code-block:: cmake
+
+  set(test_src <path_to_source_file>)
+  set(test_name <test_name_derived_from_test_src>)
+
+  # Parse %PARAM% comments from the source file and generate lists of labels/definitions:
+  cccl_parse_variant_params("${test_src}" num_variants variant_labels variant_defs)
+
+  if (num_variants EQUAL 0)
+    # Add test with no variants named `test_name` here. Example:
+    add_executable("${test_name}" "${test_src}")
+    add_test(NAME "${test_name}" COMMAND "${test_name}")
+  else() # Has variants:
+    # Optional: log the detected variant info to CMake's VERBOSE output stream:
+    cccl_log_variant_params("${test_name}" ${num_variants} variant_labels variant_defs)
+
+    # Subtract 1 to support the inclusive endpoint of foreach(...RANGE...):
+    math(EXPR var_range_end "${num_variants} - 1")
+    foreach(var_idx RANGE ${var_range_end})
+      # Get the variant label and definitions for the current index:
+      cccl_get_variant_data(variant_labels variant_defs ${var_idx} var_label var_defs)
+      set(var_name "${test_name}.${var_label}")
+
+      # Add the test with the current variant label and definitions.
+      # Example:
+      add_executable("${var_name}" "${test_src}")
+      target_compile_definitions("${var_name}" PRIVATE ${var_defs})
+      add_test(NAME "${var_name}" COMMAND "${var_name}")
+    endforeach()
+  endif()
+
+Debugging
+---------
+
+Running CMake with ``--log-level=VERBOSE`` will print out extra information about
+all detected test variants.
+
+Additional Info
+---------------
+
+Ideally, only parameters that directly influence template instantiations
+should be split out in this way. If changing a parameter doesn't change a
+template type, the same template instantiations will be compiled into multiple
+executables. This defeats the purpose of splitting up the test since the
+compiler will generate redundant code across the new split executables.
+
+The best candidate parameters for splitting are input value types, rather than
+integral parameters like ``BLOCK_THREADS``, etc. Splitting by value type allows more
+infrastructure (data generation, validation) to be reused. Splitting other
+parameters can cause build times to increase since type-related infrastructure
+has to be rebuilt for each test variant.
diff --git a/cccl_upstream/docs/cccl/development/visibility.rst b/cccl_upstream/docs/cccl/development/visibility.rst
new file mode 100644
index 00000000..ee1f73ce
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility.rst
@@ -0,0 +1,63 @@
+. _cccl-development-visibility:
+
+Symbol Visibility
+==================
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   visibility/host_stub_visibility
+   visibility/device_kernel_visibility
+   visibility/different_architectures
+
+Using kernels in shared libraries is a known source of issues. This relates to the visibility of the kernel functions
+and their host stubs, but also ODR violations that arise from compiling with different CUDA architectures.
+
+To alleviate those issues we have derived the following rules to ensure that users can safely rely on CCCL features in
+shared libraries.
+
+1. Every kernel should be annotated as ``hidden`` through ``_CCCL_KERNEL_ATTRIBUTES``
+2. Every function or type that eventually calls a kernel in a subsequent function call or member function must be put in
+   a namespace that disambiguates the CUDA architectures the library was compiled with.
+3. It is important that an API accepting kernel pointers (e.g. ``triple_chevron``) always resides in the same
+   library as the code taking this pointers.
+
+In the following we will give a more detailed overview over the different problems and why we settled on above rules.
+
+Problem 1: Selecting the right kernel stub
+-------------------------------------------
+
+Consider a project that links two shared libraries ``lib_a`` and ``lib_b`` that involve a kernel  call of some global
+``kernel`` template. The compiler will generate a stub function that handles actually launching the kernel via the CUDA
+runtime. Prior to CTK 13.0 that stub function has weak linkage, so if both libraries try to launch ``kernel`` only one
+host stub will be selected and the other kernel launch might silently fail. See the compiler teams
+`blog post <https://developer.nvidia.com/blog/cuda-c-compiler-updates-impacting-elf-visibility-and-linkage/>`_ about the
+recent changes to kernel visibility.
+
+A more detailed description can be found :ref:`here <cccl-development-visibility-host-stub-visibility>`.
+
+Problem 2: Calling kernels from inside a shared library
+--------------------------------------------------------
+
+This is quite similar to Problem 1 above. Again a project links two shared libraries ``lib_a`` and ``lib_b``. However,
+this time we call a library function ``foo`` that takes a function pointer to a kernel as an argument and invokes it.
+If ``foo`` has weak external linkage we might end up calling ``lib_b::foo`` from inside ``lib_b`` instead of
+``lib_a::foo``, or vice versa. The CUDA runtime from ``lib_a`` will not be able to call the kernel function pointer we
+passed from ``lib_b``.
+
+A more detailed description can be found :ref:`here <cccl-development-visibility-device-kernel-visibility>`.
+
+Problem 3: Libraries compiled for different architectures
+----------------------------------------------------------
+
+This is orthogonal to the visibility of the functions themself but relates to ODR
+(`_one definition rule_ <https://en.cppreference.com/w/cpp/language/definition.html>`_) violations in case libraries are
+compiled for different architectures. As new architectures come out, we adopt new features to provide the best possible
+performance for all existing architectures.
+
+However, consider a kernel that relies on hardware dependent tuning or can leverage runtime features that are only
+available on certain hardware. If we build 2 libraries for different architectures then the kernel implementation
+will be different between the two libraries, but the kernel itself is mangled as the same symbol.
+
+A more detailed description can be found :ref:`here <cccl-development-visibility-different-architectures>`.
diff --git a/cccl_upstream/docs/cccl/development/visibility/device_kernel_visibility.rst b/cccl_upstream/docs/cccl/development/visibility/device_kernel_visibility.rst
new file mode 100644
index 00000000..d754b494
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility/device_kernel_visibility.rst
@@ -0,0 +1,133 @@
+.. _cccl-development-visibility-device-kernel-visibility:
+
+
+Device Kernel Visibility Issue
+-------------------------------
+
+Consider the following simple translation unit (TU):
+
+.. code-block:: cpp
+
+    template <class T>
+    __global__ void kernel(T *val) {
+        ::printf("kernel: set val = 42\n");
+        *val = 42;
+    }
+
+   int main() {
+     int *ptr{};
+     kernel<<<1, 1>>>(ptr);
+   }
+
+The cuda compiler frontend will turn this into:
+
+.. code-block:: cpp
+
+   template< class T>
+   static void __wrapper__device_stub_kernel(T *&ptr) {
+     ::cudaLaunchKernel(0, 0, 0, 0, 0, 0);
+   }
+
+   // stub host function
+   template< class T>
+   void kernel(T *ptr) {
+     __wrapper__device_stub_kernel<T>(ptr);
+   }
+
+   int main() {
+     int *ptr{};
+     (__cudaPushCallConfiguration(1, 1)) ? (void)0 : kernel(ptr);
+   }
+
+   static void __device_stub__Z6kernelIiEvPT_(int *__par0) {
+     __cudaLaunchPrologue(1);
+     __cudaSetupArgSimple(__par0, 0UL);
+     __cudaLaunch(((char *)((void ( *)(int *))kernel )));
+   }
+
+   template<> void __wrapper__device_stub_kernel(int *&__cuda_0) {
+     __device_stub__Z6kernelIiEvPT_( (int *&)__cuda_0);
+   }
+
+The CUDA runtime is going to use the address of ``template<> void kernel(T *ptr)`` (in the following ``h_kernel``)
+as a key in the host stub function (``h_kernel``) - device function (``d_kernel``) mapping. This works fine if
+there is only a single source of truth for the stub function ``h_kernel``.
+
+However, imagine that there are two shared libraries: ``lib_a`` and ``lib_b`` both instantiating different ``kernel``
+instances, e.g ``d_kernel<int>`` and ``d_kernel<size_t>``.
+
+.. code-block:: cmake
+
+    project(DeviceKernelVisibility CUDA CXX)
+
+    add_executable(device_kernel_visibility main.cu)
+    add_library(lib_a SHARED tu_a.cu)
+    add_library(lib_b SHARED tu_b.cu)
+    target_link_libraries(device_kernel_visibility PRIVATE lib_a lib_b)
+
+Each library will have it's own fatbinary: ``d_kernel<int>_a`` and ``d_kernel<size_t>_b`` as well as host stub functions
+``h_kernel<int>_a`` and ``h_kernel<size_t>_b``.
+
+=== ============= ============
+lib host          device
+=== ============= ============
+a   0xh_kernel_a  0xd_kernel_a
+b   0xh_kernel_b  0xd_kernel_b
+=== ============= ============
+
+In contrast to
+:ref:`Problem 1 <cccl-development-visibility-host-stub-visibility>` the host stubs will get a different mangled name
+and so the right stub function will always be selected.
+
+Now imagine that both libraries are going to defer launching of their kernels to a function ``foo`` common to both
+``lib_a`` and ``lib_b``, which has weak external linkage. This might happen in ``CUB``, because it launches
+kernels through the ``thrust::triple_chevron`` helper.
+
+Similar to :ref:`Problem 1 <cccl-development-visibility-host-stub-visibility>` the linker will pick one of the two
+weak symbols and subsequently ``lib_a`` will try to pass its own kernel ``d_kernel<int>_a`` to ``lib_b::foo``.
+
+However, the CUDA runtime in ``lib_b`` will not find any kernel registered at the address of ``d_kernel<int>_a`` and
+will fail to launch the kernel.
+
+A simple example program that exemplifies this can be found
+`on github <https://github.com/NVIDIA/cccl/tree/main/docs/cub/developer/visibility/examples/device_kernel_visibility>`_
+
+.. code-block:: bash
+
+   ./device_kernel_visibility/device_kernel_visibility
+   a: kernel stub address: 0x7fdec19e13eb                      <== launching kernel_a from a
+   a: kernel is in mapping: no error
+   b: launched kernel
+   a: kernel: set val = 42
+   a: synchronized stream
+   a: copied from device to host
+   a: out: 42
+   a: kernel was launched: out == 42                           <== everything is fine
+
+   a: defers launch to b
+   b: kernel stub address: 0x7fdec19e13eb                      <== launch kernel_a from b
+   b: kernel NOT found in mapping: invalid device function     <== kernel_a is not found in b mapping
+   b: FAILED to launch kernel                                  <== unable to launch the kernel from b
+   b: synchronized stream
+   b: copied from device to host
+   b: out: 0
+   b: kernel was NOT actually launched: out != 42
+
+   b: kernel stub address: 0x7fdec19333eb                      <== launch kernel_b from b
+   b: kernel is in mapping: no error
+   b: launched kernel
+   b: kernel: set val = 42
+   b: synchronized stream
+   b: copied from device to host
+   b: out: 42
+   b: kernel was launched: out == 42                           <== everything is fine
+
+   b: defers launch to a
+   a: kernel stub address: 0x7fdec19333eb                      <== launching kernel_b from a
+   a: kernel NOT found in mapping: invalid device function     <== same issue as above
+   a: FAILED to launch kernel
+   b: kernel: set val = 42
+   a: synchronized stream
+   a: copied from device to host
+   a: out: 42
+   a: kernel was launched: out == 42                           <==  kernel launch somehow succeeded
diff --git a/cccl_upstream/docs/cccl/development/visibility/different_architectures.rst b/cccl_upstream/docs/cccl/development/visibility/different_architectures.rst
new file mode 100644
index 00000000..8598664e
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility/different_architectures.rst
@@ -0,0 +1,186 @@
+.. _cccl-development-visibility-different-architectures:
+
+Linking TUs compiled with different architectures
+--------------------------------------------------
+
+Consider the following simple library:
+
+.. code-block:: cpp
+
+    template <int... Archs>
+    __host__ __device__ constexpr int sum_archs() noexcept {
+      return (Archs + ... + 0);
+    }
+
+    // kernel with architecture dependent symbol name and functionality
+    template <class T, auto Archs = sum_archs<__CUDA_ARCH_LIST__>()>
+    __global__ void kernel(T *val) {
+      *val = sum_archs<__CUDA_ARCH_LIST__>();
+    }
+
+    __attribute__((visibility("hidden"))) inline int use_kernel() {
+      int *d_val{};
+      cudaMalloc(&d_val, sizeof(d_val));
+      kernel<<<1, 1>>>(d_val);
+      int ret;
+      if (cudaMemcpy(&ret, d_val, sizeof(size_t), cudaMemcpyDeviceToHost) !=
+          cudaSuccess) {
+        std::printf("c: FAILED to copy from device to host\n");
+      }
+      return ret;
+    }
+
+    template <class T = int>
+    struct some_class_with_kernel {
+      T val_;
+
+      some_class_with_kernel();
+      __forceinline__ some_class_with_kernel(T) { val_ = use_kernel(); }
+    };
+
+We have a kernel that does some architecture dependent work. This could be relying on some hardware feature that is
+dependent on the current architecture.
+
+.. code-block:: cpp
+
+  #include "kernel.cuh"
+
+  int main() {
+    some_class_with_kernel with_inline{1};
+    std::printf("a: value of class with inlined constructor: %d\n",
+                with_inline.val_);
+
+    some_class_with_kernel from_library{};
+    std::printf("a: value of class with constructor from library: %d\n",
+                from_library.val_);
+  }
+
+Importantly, one of the constructors for that class is put into a shared library, whereas the other one happens to be
+inlined. If a user now links two different libraries, the outcome of the initialization of ``some_class_with_kernel``
+will depend on whether the inlined constructor is called and which of the libraries is loaded first by the linker.
+
+Even worse, the state of a class depends on whether the constructor has been inlined or not and the order in which
+the linker loads the libraries.
+
+.. code-block:: cmake
+
+  project(CUBVisDifferentArchitectures CUDA CXX)
+
+  add_library(cubvis_different_architectures_lib_a SHARED tu_a.cu)
+  set_target_properties(cubvis_different_architectures_lib_a PROPERTIES CUDA_ARCHITECTURES "86;90a")
+
+  add_library(cubvis_different_architectures_lib_b SHARED tu_b.cu)
+  set_target_properties(cubvis_different_architectures_lib_b PROPERTIES CUDA_ARCHITECTURES "75;86;90a")
+
+  add_executable(cubvis_different_architectures main.cu)
+  set_target_properties(cubvis_different_architectures PROPERTIES CUDA_ARCHITECTURES "75;86")
+
+  target_link_libraries(cubvis_different_architectures PRIVATE
+    cubvis_different_architectures_lib_a
+    cubvis_different_architectures_lib_b)
+
+  add_executable(cubvis_different_architectures_switched main.cu)
+  set_target_properties(cubvis_different_architectures_switched PROPERTIES CUDA_ARCHITECTURES "75;86")
+
+  target_link_libraries(cubvis_different_architectures_switched PRIVATE
+    cubvis_different_architectures_lib_b
+    cubvis_different_architectures_lib_a)
+
+Execution the two libraries will result in the following:
+
+.. code-block::
+
+  ./different_architectures/different_architectures
+  a: value of class with inlined constructor: 1610       <<<--- from main
+  a: value of class with constructor from library: 1760  <<<--- from lib_a
+
+  ./different_architectures/different_architectures_switched
+  a: value of class with inlined constructor: 1610       <<<--- from main
+  a: value of class with constructor from library: 2510  <<<--- from lib_b
+
+
+One solution would be to bake the architectures into the symbol name of the class, either via a defaulted template
+argument or an inline namespace. That way the usage of the non-inlined kernel would result in a linker error, because
+we did not provide a matching implementation.
+
+.. code-block::
+
+  tmpxft_00048dff_00000000-6_main.compute_86.cudafe1.cpp:(.text.startup+0xc0):
+  undefined reference to `some_class_with_kernel<int, 5120ul>::some_class_with_kernel()'
+
+However, if all the functionality is within a non-inlined function we would still get different results, because all
+kernel definitions would be internal to the respective library.
+
+.. code-block:: cpp
+
+  // In tu_a.cu and tu_b.cu
+  void non_inlined_function() {
+    some_class_with_kernel with_inline{1};
+    std::printf("a: value of class with inlined constructor: %d\n",
+                with_inline.val_);
+
+    some_class_with_kernel from_library{};
+    std::printf("a: value of class with constructor from library: %d\n",
+                from_library.val_);
+  }
+
+  // In main.cu
+  #include "kernel.cuh"
+
+  void non_inlined_function();
+
+  int main() {
+    some_class_with_kernel with_inline{1};
+    std::printf("a: value of class with inlined constructor: %d\n",
+                with_inline.val_);
+
+    non_inlined_function();
+  }
+
+Executing this binary will give us again:
+
+.. code-block::
+
+  ./different_architectures/different_architectures
+  a: value of class with inlined constructor: 1610       <<<--- from main
+  a: value of class with inlined constructor: 1760       <<<--- from lib_a
+  a: value of class with constructor from library: 1760  <<<--- from lib_a
+
+  ./different_architectures/different_architectures_switched
+  a: value of class with inlined constructor: 1610       <<<--- from main
+  a: value of class with inlined constructor: 2510       <<<--- from lib_a
+  a: value of class with constructor from library: 2510  <<<--- from lib_b
+
+So there is not functional way we can solve this problem generically, because the moment a user actually uses any type
+of function that executes a kernel and puts that function into a shared library there is no guarantee which function
+is selected. The same happens if the user builds a type
+
+.. code-block:: cpp
+
+  class user_defined_with_kernel {
+    some_class_with_kernel val;
+
+    user_defined_with_kernel();
+    __forceinline__ user_defined_with_kernel(T input) : val(input)
+    {}
+  };
+
+  void function_that_uses_kernel_inside();
+
+If ``user_defined_with_kernel`` is ever baked into a library we would be back with the same exact problem,
+just one layer up. The user would need to know that ``some_class_with_kernel`` uses a kernel and then annotate *their*
+classes and functions appropriately. This is neither realistic nor feasible.
+
+Lets circle back to the previous statement: ``This is bad.`` Is it really though?
+
+Lets look at the prime example ``thrust::device_vector``, which uses a kernel for initialization. What happens if we
+accidentally run the kernel from another shared library compiled with different architectures? Worst case we are
+eating some performance regressions because the kernel will not utilize advanced features of a new architecture,
+but in the end the result of calling that kernel will not change the outcome.
+
+This is because the kernel call is consistent *within* each library. As long as the user facing API does not rely on
+specific internals of a kernel to be called -which it should not-, then any of the two libraries will do.
+
+Finally, the architectures that are passed around in ``__CUDA_ARCH_LIST__`` do *not* discriminate architecture families.
+There is currently no programmatic way to discriminate a library that has been compiled for ``SM90a`` from one that was
+compiled for ``SM90``. This is because the architecture specific macros are only available on device not on host.
diff --git a/cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/CMakeLists.txt b/cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/CMakeLists.txt
new file mode 100644
index 00000000..9c714ec8
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/CMakeLists.txt
@@ -0,0 +1,10 @@
+project(DeveloperGuideDeviceKernelVisibility CUDA CXX)
+
+add_executable(device_kernel_visibility main.cu)
+add_library(device_kernel_visibility_liba SHARED tu_a.cu)
+add_library(device_kernel_visibility_lib_b SHARED tu_b.cu)
+
+target_link_libraries(
+  device_kernel_visibility
+  PRIVATE device_kernel_visibility_lib_a device_kernel_visibility_lib_b
+)
diff --git a/cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/kernel.cuh b/cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/kernel.cuh
new file mode 100644
index 00000000..61696afa
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/kernel.cuh
@@ -0,0 +1,10 @@
+#pragma once
+
+#include <cstdio>
+
+template <class T>
+__global__ void kernel(char ln, T* val)
+{
+  printf("%c: kernel: set val = 42\n", ln);
+  *val = 42;
+}
diff --git a/cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/main.cu b/cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/main.cu
new file mode 100644
index 00000000..123cf451
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/main.cu
@@ -0,0 +1,8 @@
+void a();
+void b();
+
+int main()
+{
+  a();
+  b();
+}
diff --git a/cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/tu_a.cu b/cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/tu_a.cu
new file mode 100644
index 00000000..c1326dc7
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/tu_a.cu
@@ -0,0 +1,132 @@
+#include "kernel.cuh"
+
+void b_launch(void (*k)(char, size_t*), char c, size_t* d_out);
+
+void a_launch(void (*k)(char, int*), char c, int* d_out)
+{
+  void* ptr = reinterpret_cast<void*>(k);
+
+  printf("a: kernel stub address: %p\n", ptr);
+
+  cudaFunction_t func{};
+  if (cudaError_t error = cudaGetFuncBySymbol(&func, ptr))
+  {
+    printf("a: kernel NOT found in mapping: %s\n", cudaGetErrorString(error));
+  }
+  else
+  {
+    printf("a: kernel is in mapping: %s\n", cudaGetErrorString(error));
+  }
+
+  cudaMemset(d_out, 0, sizeof(int));
+  k<<<1, 1>>>(c, d_out);
+
+  if (cudaPeekAtLastError() != cudaSuccess)
+  {
+    printf("a: FAILED to launch kernel\n");
+  }
+  else
+  {
+    printf("a: launched kernel\n");
+  }
+
+  if (cudaStreamSynchronize(0) != cudaSuccess)
+  {
+    printf("a: FAILED to synchronize stream\n");
+  }
+  else
+  {
+    printf("a: synchronized stream\n");
+  }
+
+  int h_out{};
+  if (cudaMemcpy(&h_out, d_out, sizeof(int), cudaMemcpyDeviceToHost) != cudaSuccess)
+  {
+    printf("a: FAILED to copy from device to host\n");
+  }
+  else
+  {
+    printf("a: copied from device to host\n");
+  }
+
+  printf("a: out: %d\n", h_out);
+  if (h_out != 42)
+  {
+    printf("a: kernel was NOT actually launched: out != 42\n");
+  }
+  else
+  {
+    printf("a: kernel was launched: out == 42\n");
+  }
+
+  printf("\n");
+}
+
+void a()
+{
+  cudaGetLastError();
+
+  size_t* d_out{};
+  cudaMalloc(&d_out, sizeof(size_t));
+  cudaMemset(d_out, 0, sizeof(size_t));
+
+  void* ptr = reinterpret_cast<void*>(kernel<size_t>);
+
+  printf("a: kernel stub address: %p\n", ptr);
+
+  cudaFunction_t func{};
+  if (cudaError_t error = cudaGetFuncBySymbol(&func, ptr))
+  {
+    printf("a: kernel NOT found in mapping: %s\n", cudaGetErrorString(error));
+  }
+  else
+  {
+    printf("a: kernel is in mapping: %s\n", cudaGetErrorString(error));
+  }
+
+  kernel<<<1, 1>>>('a', d_out);
+
+  if (cudaPeekAtLastError() != cudaSuccess)
+  {
+    printf("b: FAILED to launch kernel\n");
+  }
+  else
+  {
+    printf("b: launched kernel\n");
+  }
+
+  if (cudaStreamSynchronize(0) != cudaSuccess)
+  {
+    printf("a: FAILED to synchronize stream\n");
+  }
+  else
+  {
+    printf("a: synchronized stream\n");
+  }
+
+  size_t h_out{};
+  if (cudaMemcpy(&h_out, d_out, sizeof(size_t), cudaMemcpyDeviceToHost) != cudaSuccess)
+  {
+    printf("a: FAILED to copy from device to host\n");
+  }
+  else
+  {
+    printf("a: copied from device to host\n");
+  }
+
+  printf("a: out: %d\n", static_cast<int>(h_out));
+  if (h_out != 42)
+  {
+    printf("a: kernel was NOT actually launched: out != 42\n");
+  }
+  else
+  {
+    printf("a: kernel was launched: out == 42\n");
+  }
+
+  cudaMemset(d_out, 0, sizeof(size_t));
+  printf("\n");
+
+  printf("a: defers launch to b\n");
+  b_launch(kernel<size_t>, 'b', d_out);
+}
diff --git a/cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/tu_b.cu b/cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/tu_b.cu
new file mode 100644
index 00000000..2060280f
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/tu_b.cu
@@ -0,0 +1,132 @@
+#include "kernel.cuh"
+
+void a_launch(void (*k)(char, int*), char c, int* d_out);
+
+void b_launch(void (*k)(char, size_t*), char c, size_t* d_out)
+{
+  void* ptr = reinterpret_cast<void*>(k);
+
+  printf("b: kernel stub address: %p\n", ptr);
+
+  cudaFunction_t func{};
+  if (cudaError_t error = cudaGetFuncBySymbol(&func, ptr))
+  {
+    printf("b: kernel NOT found in mapping: %s\n", cudaGetErrorString(error));
+  }
+  else
+  {
+    printf("b: kernel is in mapping: %s\n", cudaGetErrorString(error));
+  }
+
+  k<<<1, 1>>>(c, d_out);
+
+  if (cudaPeekAtLastError() != cudaSuccess)
+  {
+    printf("b: FAILED to launch kernel\n");
+  }
+  else
+  {
+    printf("b: launched kernel\n");
+  }
+
+  if (cudaStreamSynchronize(0) != cudaSuccess)
+  {
+    printf("b: FAILED to synchronize stream\n");
+  }
+  else
+  {
+    printf("b: synchronized stream\n");
+  }
+
+  size_t h_out{};
+  if (cudaMemcpy(&h_out, d_out, sizeof(size_t), cudaMemcpyDeviceToHost) != cudaSuccess)
+  {
+    printf("b: FAILED to copy from device to host\n");
+  }
+  else
+  {
+    printf("b: copied from device to host\n");
+  }
+
+  printf("b: out: %d\n", static_cast<int>(h_out));
+  if (h_out != 42)
+  {
+    printf("b: kernel was NOT actually launched: out != 42\n");
+  }
+  else
+  {
+    printf("b: kernel was launched: out == 42\n");
+  }
+
+  printf("\n");
+}
+
+void b()
+{
+  cudaGetLastError();
+
+  int* d_out{};
+  cudaMalloc(&d_out, sizeof(int));
+  cudaMemset(d_out, 0, sizeof(int));
+
+  void* ptr = reinterpret_cast<void*>(kernel<int>);
+
+  printf("b: kernel stub address: %p\n", ptr);
+
+  cudaFunction_t func{};
+  if (cudaError_t error = cudaGetFuncBySymbol(&func, ptr))
+  {
+    printf("b: kernel NOT found in mapping: %s\n", cudaGetErrorString(error));
+  }
+  else
+  {
+    printf("b: kernel is in mapping: %s\n", cudaGetErrorString(error));
+  }
+
+  kernel<<<1, 1>>>('b', d_out);
+
+  if (cudaPeekAtLastError() != cudaSuccess)
+  {
+    printf("b: FAILED to launch kernel\n");
+  }
+  else
+  {
+    printf("b: launched kernel\n");
+  }
+
+  if (cudaStreamSynchronize(0) != cudaSuccess)
+  {
+    printf("b: FAILED to synchronize stream\n");
+  }
+  else
+  {
+    printf("b: synchronized stream\n");
+  }
+
+  int h_out{};
+  if (cudaMemcpy(&h_out, d_out, sizeof(int), cudaMemcpyDeviceToHost) != cudaSuccess)
+  {
+    printf("b: FAILED to copy from device to host\n");
+  }
+  else
+  {
+    printf("b: copied from device to host\n");
+  }
+
+  printf("b: out: %d\n", h_out);
+
+  if (h_out != 42)
+  {
+    printf("b: kernel was NOT actually launched: out != 42\n");
+  }
+  else
+  {
+    printf("b: kernel was launched: out == 42\n");
+  }
+
+  cudaMemset(d_out, 0, sizeof(int));
+  printf("\n");
+
+  printf("b: defers launch to a\n");
+  a_launch(kernel<int>, 'b', d_out);
+}
diff --git a/cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/CMakeLists.txt b/cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/CMakeLists.txt
new file mode 100644
index 00000000..214a7577
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/CMakeLists.txt
@@ -0,0 +1,35 @@
+project(DeveloperGuideDifferentArchitectures CUDA CXX)
+
+add_library(different_architectures_lib_a SHARED tu_a.cu)
+set_target_properties(
+  different_architectures_lib_a
+  PROPERTIES CUDA_ARCHITECTURES "86;90a"
+)
+
+add_library(different_architectures_lib_b SHARED tu_b.cu)
+set_target_properties(
+  different_architectures_lib_b
+  PROPERTIES CUDA_ARCHITECTURES "75;86;90a"
+)
+
+add_executable(different_architectures main.cu)
+set_target_properties(
+  different_architectures
+  PROPERTIES CUDA_ARCHITECTURES "75;86"
+)
+
+target_link_libraries(
+  different_architectures
+  PRIVATE different_architectures_lib_a different_architectures_lib_b
+)
+
+add_executable(different_architectures_switched main.cu kernel.cu)
+set_target_properties(
+  different_architectures_switched
+  PROPERTIES CUDA_ARCHITECTURES "75;86"
+)
+
+target_link_libraries(
+  different_architectures_switched
+  PRIVATE different_architectures_lib_b different_architectures_lib_a
+)
diff --git a/cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/kernel.cuh b/cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/kernel.cuh
new file mode 100644
index 00000000..968a35cc
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/kernel.cuh
@@ -0,0 +1,42 @@
+#pragma once
+
+#include <cstdint>
+#include <cstdio>
+
+template <int... Archs>
+__attribute__((visibility("hidden"))) __host__ __device__ constexpr int sum_archs()
+{
+  return (Archs + ... + 0);
+}
+
+template <class T, auto Archs = sum_archs<__CUDA_ARCH_LIST__>()>
+__attribute__((visibility("hidden"))) __global__ void kernel(char ln, T* val)
+{
+  printf("%c: kernel: set val = %i\n", ln, sum_archs<__CUDA_ARCH_LIST__>());
+  *val = sum_archs<__CUDA_ARCH_LIST__>();
+}
+
+__attribute__((visibility("hidden"))) __forceinline__ int use_kernel()
+{
+  int* d_val{};
+  cudaMalloc(&d_val, sizeof(size_t));
+  kernel<<<1, 1>>>(d_val);
+  int ret;
+  if (cudaMemcpy(&ret, d_val, sizeof(size_t), cudaMemcpyDeviceToHost) != cudaSuccess)
+  {
+    printf("c: FAILED to copy from device to host\n");
+  }
+  return ret;
+}
+
+template <class T = int>
+struct some_class_with_kernel
+{
+  T val_;
+
+  some_class_with_kernel();
+  __forceinline__ some_class_with_kernel(T)
+  {
+    val_ = use_kernel();
+  }
+};
diff --git a/cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/main.cu b/cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/main.cu
new file mode 100644
index 00000000..27b22e62
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/main.cu
@@ -0,0 +1,11 @@
+#include "kernel.cuh"
+
+void non_inlined_function();
+
+int main()
+{
+  some_class_with_kernel with_inline{1};
+  printf("a: value of class with inlined constructor: %d\n", with_inline.val_);
+
+  non_inlined_function();
+}
diff --git a/cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/tu_a.cu b/cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/tu_a.cu
new file mode 100644
index 00000000..d83b4f0e
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/tu_a.cu
@@ -0,0 +1,16 @@
+#include "kernel.cuh"
+
+template <class T>
+some_class_with_kernel<T>::some_class_with_kernel()
+{
+  val_ = use_kernel();
+}
+
+void non_inlined_function()
+{
+  some_class_with_kernel with_inline{1};
+  printf("a: value of class with inlined constructor: %d\n", with_inline.val_);
+
+  some_class_with_kernel from_library{};
+  printf("a: value of class with constructor from library: %d\n", from_library.val_);
+}
diff --git a/cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/tu_b.cu b/cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/tu_b.cu
new file mode 100644
index 00000000..d83b4f0e
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/tu_b.cu
@@ -0,0 +1,16 @@
+#include "kernel.cuh"
+
+template <class T>
+some_class_with_kernel<T>::some_class_with_kernel()
+{
+  val_ = use_kernel();
+}
+
+void non_inlined_function()
+{
+  some_class_with_kernel with_inline{1};
+  printf("a: value of class with inlined constructor: %d\n", with_inline.val_);
+
+  some_class_with_kernel from_library{};
+  printf("a: value of class with constructor from library: %d\n", from_library.val_);
+}
diff --git a/cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/CMakeLists.txt b/cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/CMakeLists.txt
new file mode 100644
index 00000000..fb56cfd2
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/CMakeLists.txt
@@ -0,0 +1,10 @@
+project(DeveloperGuideHostStubVisibility CUDA CXX)
+
+add_executable(host_stub_visibility main.cu)
+add_library(host_stub_visibility_lib_a SHARED tu_a.cu)
+add_library(host_stub_visibility_lib_b SHARED tu_b.cu)
+
+target_link_libraries(
+  host_stub_visibility
+  PRIVATE host_stub_visibility_lib_a host_stub_visibility_lib_b
+)
diff --git a/cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/kernel.cuh b/cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/kernel.cuh
new file mode 100644
index 00000000..61696afa
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/kernel.cuh
@@ -0,0 +1,10 @@
+#pragma once
+
+#include <cstdio>
+
+template <class T>
+__global__ void kernel(char ln, T* val)
+{
+  printf("%c: kernel: set val = 42\n", ln);
+  *val = 42;
+}
diff --git a/cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/main.cu b/cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/main.cu
new file mode 100644
index 00000000..123cf451
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/main.cu
@@ -0,0 +1,8 @@
+void a();
+void b();
+
+int main()
+{
+  a();
+  b();
+}
diff --git a/cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/tu_a.cu b/cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/tu_a.cu
new file mode 100644
index 00000000..4f3f08cc
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/tu_a.cu
@@ -0,0 +1,64 @@
+#include "kernel.cuh"
+
+void a()
+{
+  int* d_out{};
+  cudaMalloc(&d_out, sizeof(int));
+  cudaMemset(d_out, 0, sizeof(int));
+
+  void* ptr = reinterpret_cast<void*>(kernel<int>);
+
+  printf("a: kernel stub address: %p\n", ptr);
+
+  cudaFunction_t func{};
+  if (cudaError_t error = cudaGetFuncBySymbol(&func, ptr))
+  {
+    printf("a: kernel NOT found in mapping: %s\n", cudaGetErrorString(error));
+  }
+  else
+  {
+    printf("a: kernel is in mapping: %s\n", cudaGetErrorString(error));
+  }
+
+  kernel<<<1, 1>>>('a', d_out);
+
+  if (cudaPeekAtLastError() != cudaSuccess)
+  {
+    printf("b: FAILED to launch kernel\n");
+  }
+  else
+  {
+    printf("b: launched kernel\n");
+  }
+
+  if (cudaStreamSynchronize(0) != cudaSuccess)
+  {
+    printf("a: FAILED to synchronize stream\n");
+  }
+  else
+  {
+    printf("a: synchronized stream\n");
+  }
+
+  int h_out{};
+  if (cudaMemcpy(&h_out, d_out, sizeof(int), cudaMemcpyDeviceToHost) != cudaSuccess)
+  {
+    printf("a: FAILED to copy from device to host\n");
+  }
+  else
+  {
+    printf("a: copied from device to host\n");
+  }
+
+  printf("a: out: %d\n", h_out);
+  if (h_out != 42)
+  {
+    printf("a: kernel was NOT actually launched: out != 42\n");
+  }
+  else
+  {
+    printf("a: kernel was launched: out == 42\n");
+  }
+
+  printf("\n");
+}
diff --git a/cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/tu_b.cu b/cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/tu_b.cu
new file mode 100644
index 00000000..4c8f1b81
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/tu_b.cu
@@ -0,0 +1,65 @@
+#include "kernel.cuh"
+
+void b()
+{
+  int* d_out{};
+  cudaMalloc(&d_out, sizeof(int));
+  cudaMemset(d_out, 0, sizeof(int));
+
+  void* ptr = reinterpret_cast<void*>(kernel<int>);
+
+  printf("b: kernel stub address: %p\n", ptr);
+
+  cudaFunction_t func{};
+  if (cudaError_t error = cudaGetFuncBySymbol(&func, ptr))
+  {
+    printf("b: kernel NOT found in mapping: %s\n", cudaGetErrorString(error));
+  }
+  else
+  {
+    printf("b: kernel is in mapping: %s\n", cudaGetErrorString(error));
+  }
+
+  kernel<<<1, 1>>>('b', d_out);
+
+  if (cudaPeekAtLastError() != cudaSuccess)
+  {
+    printf("b: FAILED to launch kernel\n");
+  }
+  else
+  {
+    printf("b: launched kernel\n");
+  }
+
+  if (cudaStreamSynchronize(0) != cudaSuccess)
+  {
+    printf("b: FAILED to synchronize stream\n");
+  }
+  else
+  {
+    printf("b: synchronized stream\n");
+  }
+
+  int h_out{};
+  if (cudaMemcpy(&h_out, d_out, sizeof(int), cudaMemcpyDeviceToHost) != cudaSuccess)
+  {
+    printf("b: FAILED to copy from device to host\n");
+  }
+  else
+  {
+    printf("b: copied from device to host\n");
+  }
+
+  printf("b: out: %d\n", h_out);
+
+  if (h_out != 42)
+  {
+    printf("b: kernel was NOT actually launched: out != 42\n");
+  }
+  else
+  {
+    printf("b: kernel was launched: out == 42\n");
+  }
+
+  printf("\n");
+}
diff --git a/cccl_upstream/docs/cccl/development/visibility/host_stub_visibility.rst b/cccl_upstream/docs/cccl/development/visibility/host_stub_visibility.rst
new file mode 100644
index 00000000..b98ad872
--- /dev/null
+++ b/cccl_upstream/docs/cccl/development/visibility/host_stub_visibility.rst
@@ -0,0 +1,115 @@
+.. _cccl-development-visibility-host-stub-visibility:
+
+
+Host Stub Visibility Issue
+---------------------------
+
+Consider the following simple translation unit (TU):
+
+.. code-block:: cpp
+
+  #include <cstdio>
+  #include <cuda/memory>
+
+  template <class T>
+  __global__ void kernel(T *val) {
+      printf("kernel: set val = 42\n");
+      *val = 42;
+  }
+
+  __device__ int val;
+
+  int main() {
+
+     kernel<<<1, 1>>>(cuda::get_device_address(val));
+  }
+
+The CUDA compiler frontend will turn this into:
+
+.. code-block:: cpp
+
+   template< class T>
+   static void __wrapper__device_stub_kernel(T *&ptr) {
+     ::cudaLaunchKernel(0, 0, 0, 0, 0, 0);
+   }
+
+   // stub host function
+   template< class T>
+   void kernel(T *ptr) {
+     __wrapper__device_stub_kernel<T>(ptr);
+   }
+
+   int main() {
+     int *ptr{};
+     (__cudaPushCallConfiguration(1, 1)) ? (void)0 : kernel(ptr);
+   }
+
+   static void __device_stub__Z6kernelIiEvPT_(int *__par0) {
+     __cudaLaunchPrologue(1);
+     __cudaSetupArgSimple(__par0, 0UL);
+     __cudaLaunch(((char *)((void ( *)(int *))kernel )));
+   }
+
+   template<> void __wrapper__device_stub_kernel(int *&__cuda_0) {
+     __device_stub__Z6kernelIiEvPT_( (int *&)__cuda_0);
+   }
+
+The CUDA runtime is going to use the address of ``template<> void kernel(T *ptr)`` (in the following ``h_kernel``)
+as a key in the host stub function (``h_kernel``) - device function (``d_kernel``) mapping. This works fine if
+there is only a single source of truth for the stub function ``h_kernel``.
+
+However, imagine that there are two shared libraries: ``lib_a`` and ``lib_b`` both using the same ``kernel`` instance.
+
+.. code-block:: cmake
+
+    project(HostStubVisibility CUDA CXX)
+
+    add_executable(host_stub_visibility main.cu)
+    add_library(lib_a SHARED tu_a.cu)
+    add_library(lib_b SHARED tu_b.cu)
+    target_link_libraries(host_stub_visibility PRIVATE lib_a lib_b)
+
+Each library will have its own fatbinary: ``d_kernel_a`` and ``d_kernel_b``, but the compiler
+generated host stub function ``h_kernel`` has weak external linkage, so after dynamic linkage, we'll end up having
+only one of them.
+
+=== ===================== ============
+lib host                  device
+=== ===================== ============
+a   0xh_kernel_a          0xd_kernel_a
+b   0xh_kernel_a <- issue 0xd_kernel_b
+=== ===================== ============
+
+Since there's a clash of stub function addresses, only one entry stored. When ``lib_b`` queries for the
+kernel using its address of ``h_kernel``, it's visible, although it might point to ``lib_a``'s fatbinary.
+The opposite case might happen as well, depending on loading order, linker etc and is undefined behavior.
+
+Launching ``d_kernel`` from ``lib_b`` is not possible and leads to random errors. For instance, there seems to be
+some per CUDART global state. When the ``__cudaPushCallConfiguration`` is called in ``lib_b``, it affects the state of
+``cudart_b``, but the launch happens through ``h_kernel``, which is in ``lib_a``.
+
+This sometimes leads to ``__global__ function call is not configured``. However, there might also be no error at all,
+and the kernel launch is silently skipped.
+
+A simple example program that exemplifies this can be found
+`on github <https://github.com/NVIDIA/cccl/tree/main/docs/cub/developer/visibility/examples/host_stub_visibility>`_
+
+.. code-block:: bash
+
+   :./host_stub_visibility/host_stub_visibility
+   a: kernel stub address: 0x7f43318a415d           <== same address as in B
+   a: kernel is in mapping: no error                <== kernel is found in the mapping
+   b: launched kernel
+   a: kernel: set val = 42
+   a: synchronized stream
+   a: copied from device to host
+   a: out: 42
+   a: kernel was launched: out == 42
+
+   b: kernel stub address: 0x7f43318a415d           <== same address as in A
+   b: kernel is in mapping: no error                <== kernel is found in the mapping
+   b: launched kernel
+   b: synchronized stream
+   b: copied from device to host
+   b: out: 0
+   b: kernel was NOT actually launched: out != 42   <== silent failure
diff --git a/cccl_upstream/docs/cccl/license.rst b/cccl_upstream/docs/cccl/license.rst
new file mode 100644
index 00000000..2abdcb2d
--- /dev/null
+++ b/cccl_upstream/docs/cccl/license.rst
@@ -0,0 +1,12 @@
+.. _cccl-license:
+
+License
+========
+
+The CUDA Core Compute Libraries use a mix of several licenses, mostly for historical reasons.
+
+libcu++ uses the `Apache License v2.0 with LLVM Exceptions <https://llvm.org/LICENSE.txt>`_.
+CUB mostly uses the `3-Clause BSD License <https://github.com/NVIDIA/cccl/blob/main/cub/LICENSE.TXT>`_.
+Thrust mostly the `Apache License v2.0 <https://github.com/NVIDIA/cccl/blob/main/thrust/LICENSE>`_.
+
+Newly developed code will be under the `Apache License v2.0 with LLVM Exceptions <https://llvm.org/LICENSE.txt>`_.
diff --git a/cccl_upstream/docs/cccl/tma.rst b/cccl_upstream/docs/cccl/tma.rst
new file mode 100644
index 00000000..d4f9467d
--- /dev/null
+++ b/cccl_upstream/docs/cccl/tma.rst
@@ -0,0 +1,35 @@
+.. _cccl-tma:
+
+Tensor Memory Accelerator (TMA)
+===============================
+
+The Tensor Memory Accelerator (TMA) is a hardware feature available on Hopper (SM90) and newer GPUs
+that enables efficient asynchronous memory copies of tensor data between global and (cluster) shared memory.
+The use of TMA is required to reach SOL memory throughput for some workloads,
+notable those where the necessary load/store vectorization, unrolling, or pipelining are limited by the register file or other factors.
+CCCL offer several tools to help users leverage TMA in their applications.
+
+In general, we recommend users to reach for high-level algorithms if they fit their problem.
+Several algorithms, like ``cub::DeviceTransform``, ``cub::DeviceMerge``, ``cub::DeviceScan`` already use TMA internally today,
+with many Thrust algorithms building on those.
+And more algorithms will be added over time.
+Relying on high level algorithms leaves the complexity of implementing and tuning TMA to CCCL team,
+while providing users with safer interfaces, high productivity and SOL performance from the start.
+
+If direct use of TMA is required to author new kernels, CCCL offers the following tools to help users get started,
+from high-level to low-level:
+
+ - ``cub::BlockLoadToShared`` coming soon :)
+ - :ref:`cuda::memcpy_async <libcudacxx-extended-api-asynchronous-operations-memcpy-async>`
+ - :ref:`cuda::device::memcpy_async_tx <libcudacxx-extended-api-asynchronous-operations-memcpy-async-tx>`
+ - :ref:`cuda::ptx::cp_async_bulk* variants <libcudacxx-ptx-instructions>`
+
+``cub::BlockLoadToShared`` and ``cuda::memcpy_async`` have fallback implementations for pre-Hopper GPUs,
+using ``cp.async``/``LDGSTS`` on Ampere (SM80+) and ordinary loads/stores on older architectures.
+Furthermore, they gracefully handle unaligned data and copying regions of arbitrary size.
+
+The various ``cuda::ptx::cp_async_bulk*`` versions and ``cuda::device::memcpy_async_tx``
+are thin wrappers of the corresponding PTX instructions
+and provide no fallback path on older GPUs and also require the copied data to be aligned and sized appropriately.
+
+Some further TMA-related utilities are provided by the :ref:`libcu++ extended API <libcudacxx-extended-api-tma>`.
diff --git a/cccl_upstream/docs/conf.py b/cccl_upstream/docs/conf.py
new file mode 100644
index 00000000..11631510
--- /dev/null
+++ b/cccl_upstream/docs/conf.py
@@ -0,0 +1,284 @@
+# CCCL Documentation Configuration File
+# Generated to replace repo-docs with direct Sphinx usage
+
+import os
+import sys
+from datetime import datetime
+
+# Add extension directory to path
+sys.path.insert(0, os.path.abspath("_ext"))
+
+# Add Python CCCL package to path for autodoc
+python_package_path = os.path.abspath("../python/cuda_cccl")
+if os.path.exists(python_package_path):
+    sys.path.insert(0, python_package_path)
+
+# Note: numpy is installed as a real dependency (see requirements.txt)
+# This avoids issues with type annotations using union syntax (ndarray | type)
+
+# -- Project information -----------------------------------------------------
+
+project = "CUDA Core Compute Libraries"
+copyright = f"{datetime.now().year}, NVIDIA Corporation"
+author = "NVIDIA Corporation"
+
+# Version information
+_env_version = os.environ.get("SPHINX_CCCL_VER")
+if _env_version:
+    release = _env_version
+else:
+    try:
+        with open("VERSION.md", "r", encoding="utf-8") as f:
+            release = f.read().strip()
+    except Exception:
+        release = "unstable"
+
+version = release
+
+# -- General configuration ---------------------------------------------------
+
+extensions = [
+    "sphinx.ext.autodoc",
+    "sphinx.ext.autosummary",
+    "sphinx.ext.intersphinx",
+    "sphinx.ext.napoleon",
+    "sphinx.ext.extlinks",
+    "sphinx.ext.mathjax",
+    "sphinx.ext.graphviz",
+    "sphinx.ext.doctest",
+    "myst_parser",  # MyST parser for markdown support
+    "breathe",  # For Doxygen integration - has built-in embed:rst support
+    # "exhale",  # Disabled - causing build timeouts, API docs handled by breathe
+    "sphinx_design",  # For dropdown, card, and other directives
+    "sphinx_copybutton",
+    "nbsphinx",
+    # "rst_processor",  # Disabled - breathe handles embed:rst natively
+    "auto_api_generator",  # Automatically generate API reference pages from Doxygen XML
+]
+
+# Breathe configuration for Doxygen integration
+breathe_projects = {
+    "cub": "_build/doxygen/cub/xml",
+    "thrust": "_build/doxygen/thrust/xml",
+    "libcudacxx": "_build/doxygen/libcudacxx/xml",
+    "cudax": "_build/doxygen/cudax/xml",
+}
+
+breathe_default_project = "cub"
+breathe_default_members = ("members", "undoc-members")
+breathe_show_enumvalue_initializer = True
+breathe_domain_by_extension = {"cuh": "cpp", "h": "cpp", "hpp": "cpp"}
+
+# Configure cpp domain to handle cub namespace
+cpp_index_common_prefix = ["cub::"]
+
+# Preprocessor definitions for Breathe to handle CCCL macros
+cpp_id_attributes = [
+    "__device__",
+    "__host__",
+    "__global__",
+    "__forceinline__",
+    "_CCCL_HOST_DEVICE",
+    "_CCCL_DEVICE",
+    "_CCCL_HOST",
+    "_CCCL_FORCEINLINE",
+    "_CCCL_API",
+    "_CCCL_HOST_API",
+    "_CCCL_DEVICE_API",
+    "_CCCL_NODEBUG_API",
+    "_CCCL_NODEBUG_HOST_API",
+    "_CCCL_NODEBUG_DEVICE_API",
+    "_CCCL_TRIVIAL_API",
+    "_CCCL_TRIVIAL_HOST_API",
+    "_CCCL_TRIVIAL_DEVICE_API",
+]
+cpp_paren_attributes = ["__declspec", "__align__"]
+
+# Add support for .rst and .md files
+source_suffix = {
+    ".rst": "restructuredtext",
+    ".md": "markdown",
+}
+
+templates_path = ["_templates"]
+
+# Exclude patterns
+exclude_patterns = [
+    "_build",
+    "_repo",
+    "tools",
+    "VERSION.md",
+    "Thumbs.db",
+    ".DS_Store",
+    "env/**",  # Virtual environment
+    "**/.pytest_cache",
+    "**/__pycache__",
+    "*.pyc",
+    "*.pyo",
+]
+
+# -- Options for HTML output -------------------------------------------------
+
+html_theme = "nvidia_sphinx_theme"
+
+html_logo = "_static/nvidia-logo.png"
+
+html_baseurl = (
+    os.environ.get("CCCL_DOCS_BASE_URL", "https://nvidia.github.io/cccl/").rstrip("/")
+    + "/"
+)
+
+html_theme_options = {
+    "icon_links": [
+        {
+            "name": "GitHub",
+            "url": "https://github.com/NVIDIA/cccl",
+            "icon": "fa-brands fa-github",
+            "type": "fontawesome",
+        }
+    ],
+    "navigation_depth": 4,
+    "show_toc_level": 2,
+    "navbar_start": ["navbar-logo"],
+    "navbar_end": ["theme-switcher", "navbar-icon-links"],
+    "footer_start": ["copyright"],
+    "footer_end": ["sphinx-version"],
+    "sidebar_includehidden": True,
+    "collapse_navigation": False,
+    "switcher": {
+        "json_url": f"{html_baseurl}nv-versions.json",
+        "version_match": release,
+    },
+}
+
+html_static_path = ["_static"] if os.path.exists("_static") else []
+
+# Images directory
+if os.path.exists("img"):
+    html_static_path.append("img")
+
+html_js_files = ["deduplicate_toc.js"]
+
+html_title = "CUDA Core Compute Libraries"
+
+# -- Options for extensions --------------------------------------------------
+
+# Intersphinx mapping
+intersphinx_mapping = {
+    "python": ("https://docs.python.org/3/", None),
+    "numpy": ("https://numpy.org/doc/stable/", None),
+}
+
+# MyST parser configuration
+myst_enable_extensions = [
+    "colon_fence",
+    "deflist",
+    "html_image",
+]
+
+# Napoleon settings
+napoleon_google_docstring = True
+napoleon_numpy_docstring = True
+napoleon_include_init_with_doc = False
+napoleon_include_private_with_doc = False
+napoleon_include_special_with_doc = True
+napoleon_use_admonition_for_examples = False
+napoleon_use_admonition_for_notes = False
+napoleon_use_admonition_for_references = False
+napoleon_use_ivar = False
+napoleon_use_param = True
+napoleon_use_rtype = True
+napoleon_preprocess_types = False
+napoleon_type_aliases = None
+
+# Autodoc settings
+autodoc_default_options = {
+    "members": True,
+    "member-order": "bysource",
+    "special-members": "__init__",
+    "undoc-members": True,
+    "exclude-members": "__weakref__",
+}
+
+# Enable type hints to be shown in the documentation
+autodoc_type_hints = "description"
+autodoc_type_aliases = {
+    "Operator": "Operator",
+}
+
+# Set Python domain primary for intersphinx
+primary_domain = "py"
+
+# Mock imports for Python documentation - these modules may not be installed
+autodoc_mock_imports = [
+    "numba",
+    "numba.core",
+    "numba.core.cgutils",
+    "numba.core.extending",
+    "numba.core.typing",
+    "numba.core.typing.ctypes_utils",
+    "numba.core.typing.templates",
+    "numba.cuda",
+    "numba.cuda.cudadecl",
+    "numba.cuda.dispatcher",
+    "numba.extending",
+    "numba.types",
+    "cuda.bindings",
+    "cuda.bindings.driver",
+    "cuda.bindings.runtime",
+    "cuda.core",
+    "cuda.core.experimental",
+    "cuda.core.experimental._utils",
+    "cuda.core.experimental._utils.cuda_utils",
+    "cuda.pathfinder",
+    "llvmlite",
+    "llvmlite.ir",
+    # numpy is installed as a real dependency (see requirements.txt)
+    "numpydoc_test_module",  # Mock to avoid import errors
+    "cupy",
+    "cuda.compute._bindings",
+    "cuda.compute._bindings_impl",
+]
+
+# External links configuration
+extlinks = {
+    "github": ("https://github.com/NVIDIA/cccl/blob/main/%s", "%s"),
+}
+
+
+# Exhale not used - API documentation is handled directly through breathe directives
+
+# Napoleon configuration (handles NumPy-style docstrings)
+# Note: numpydoc settings removed as Napoleon is used instead
+
+# Config copybutton
+# Suppress specific warning categories that arise from breathe (Doxygen-to-Sphinx
+# bridge) limitations.  These cannot be fixed in our source headers or RST files.
+#
+# See also _BREATHE_SKIP_SYMBOLS in _ext/auto_api_generator.py for symbols that
+# are excluded from page generation entirely due to unparsable declarations.
+suppress_warnings = [
+    # Breathe walks each Doxygen XML file independently.  When a symbol appears
+    # in both a namespace XML and a class/group XML (which is normal for Doxygen),
+    # breathe emits the C++ declaration twice, triggering a duplicate-declaration
+    # warning.  There is no way to control this from our side without patching
+    # breathe's XML traversal.
+    "cpp.duplicate_declaration",
+    # When breathe expands doxygenfunction/doxygenvariable directives, it writes
+    # the resolved C++ signature into RST.  Signatures containing default argument
+    # values (e.g. ``= {}``) or complex SFINAE expressions produce RST that the
+    # docutils parser cannot handle (mismatched inline-literal markers, unexpected
+    # braces, etc.).  The source C++ is valid; the issue is that what breathe
+    # emits as RST is not valid RST.
+    "docutils",
+]
+
+copybutton_prompt_text = ">>> |$ |# "
+autosummary_imported_members = False
+autosummary_generate = True
+autoclass_content = "class"
+
+
+def setup(app):
+    if os.path.exists("_static/custom.css"):
+        app.add_css_file("custom.css")
diff --git a/cccl_upstream/docs/contributors/how_tos/adding_build_target.rst b/cccl_upstream/docs/contributors/how_tos/adding_build_target.rst
new file mode 100644
index 00000000..bfdc7e61
--- /dev/null
+++ b/cccl_upstream/docs/contributors/how_tos/adding_build_target.rst
@@ -0,0 +1,101 @@
+.. _infra-cmake-adding-build-target:
+
+Adding a build target
+=====================
+
+In most cases, new C++ test and example sources are detected automatically.
+Their executables should appear after re-configuring / re-building.
+When in doubt, check the CMakeLists.txt in the source directory to discover the conventions.
+
+This document is intended for the rare case where a developer needs to build project infrastructure from scratch.
+
+``cccl_add_executable`` wraps ``add_executable`` with CCCL's standard target
+configuration: dialect handling, output directories, metatarget registration,
+and clang-tidy integration. Use it for every test, example, benchmark, and tool
+in the tree. The function lives in ``cmake/CCCLAddExecutable.cmake``.
+
+Signature
+---------
+
+::
+
+    cccl_add_executable(target_name
+      SOURCES <source1> [source2 ...]
+      [ADD_CTEST]
+      [NO_METATARGETS]
+      [NO_CLANG_TIDY]
+      [METATARGET_PATH <path>]
+      [DIALECT <standard>]
+    )
+
+The first positional argument is the target name. The remaining arguments are
+keyword options:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 25 15 60
+
+   * - Argument
+     - Kind
+     - Effect
+   * - ``SOURCES``
+     - Required
+     - Source files for the executable. The function hard-fails with a fatal
+       error if absent.
+   * - ``ADD_CTEST``
+     - Flag
+     - Registers a CTest with the same name as the target, running the
+       executable with no arguments.
+   * - ``NO_METATARGETS``
+     - Flag
+     - Skips metatarget registration. The target builds only by its own name.
+   * - ``NO_CLANG_TIDY``
+     - Flag
+     - Skips clang-tidy integration for these sources.
+   * - ``METATARGET_PATH``
+     - One value
+     - Dotted path placing the target in the metatarget hierarchy. Defaults to
+       ``target_name``.
+   * - ``DIALECT``
+     - One value
+     - C++ standard override for this target (for example ``17`` or ``20``).
+
+Add a test target
+-----------------
+
+**Step 1. Call the function with SOURCES and ADD_CTEST.** Name the target with
+its dotted hierarchy path so the metatarget system groups it correctly::
+
+    cccl_add_executable(cub.test.device_reduce
+      SOURCES test_device_reduce.cu
+      ADD_CTEST
+    )
+
+This creates the executable ``cub.test.device_reduce``, registers a CTest of the
+same name, and adds it to the ``cub`` and ``cub.test`` metatargets.
+
+**Step 2. Link the target's dependencies.** ``cccl_add_executable`` configures
+the target but does not link libraries. Add them after the call::
+
+    target_link_libraries(cub.test.device_reduce
+      PRIVATE
+        cub.compiler_interface
+        cccl.c2h
+    )
+
+Place the target in the hierarchy
+---------------------------------
+
+By default the metatarget path equals the target name. A target named
+``foo.bar.baz`` builds via metatargets ``foo`` and ``foo.bar``. Running
+``ninja foo`` builds every descendant; ``ninja foo.bar`` builds that subtree.
+
+Use ``METATARGET_PATH`` to decouple the target name from its hierarchy
+position. Rare; advanced use only.
+
+Exclude a target from the hierarchy
+-----------------------------------
+
+Pass ``NO_METATARGETS`` for targets that should not appear in the test
+hierarchy: benchmarks, standalone tools, and anything outside the
+build-everything-and-test workflow. Rare; advanced use only.
diff --git a/cccl_upstream/docs/contributors/how_tos/index.rst b/cccl_upstream/docs/contributors/how_tos/index.rst
new file mode 100644
index 00000000..ffd43c38
--- /dev/null
+++ b/cccl_upstream/docs/contributors/how_tos/index.rst
@@ -0,0 +1,14 @@
+How Tos
+=======
+
+How-to guides for contributors.
+
+.. toctree::
+   :maxdepth: 1
+
+   install_build_test
+   targeted_builds
+   launching
+   reproducing_ci
+   override_matrix
+   adding_build_target
diff --git a/cccl_upstream/docs/contributors/how_tos/install_build_test.rst b/cccl_upstream/docs/contributors/how_tos/install_build_test.rst
new file mode 100644
index 00000000..05c6ca19
--- /dev/null
+++ b/cccl_upstream/docs/contributors/how_tos/install_build_test.rst
@@ -0,0 +1,162 @@
+.. _infra-install-build-test:
+
+Install, Build, Test
+====================
+
+CCCL provides several developer interfaces for working with the codebase.
+Purpose-built scripts exist that drive routine work, and using the wrong tool can waste hours of developer time.
+For example, the ``ci/test_*.sh`` scripts can take hours to run a full validation suite, while the
+``ci/util/build_and_test_targets.sh`` tool configures, builds, and runs a small subset in seconds.
+Prebuilt development containers simplify working in specific toolchains and environments.
+Core members can launch GitHub Actions benchmarking / bisection workflows that run on cloud infrastructure.
+
+Pick a path by goal
+-------------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 34 38 14 14
+
+   * - Goal
+     - Tools
+     - Type
+     - Availability
+   * - Install CCCL headers to a prefix
+     - ``ci/install_cccl.sh``
+     - Script
+     - Public
+   * - Build and run a specific test
+     - ``ci/util/build_and_test_targets.sh``
+     - Script
+     - Public
+   * - Build or test an entire project
+     - ``ci/build_<project>.sh`` / ``ci/test_<project>.sh``
+     - Script
+     - Public
+   * - Bisect a regression
+     - ``ci/util/git_bisect.sh``, git-bisect.yml
+     - Script, GHA
+     - Public / members
+   * - Request a benchmark comparison
+     - ``ci/bench/bench.sh``, bench.yml, ``ci/bench.yaml``
+     - Script, GHA, PR tool
+     - Public / members
+   * - Custom build
+     - ``cmake --preset``
+     - Script
+     - Public
+
+Install only
+~~~~~~~~~~~~
+
+``ci/install_cccl.sh <prefix>`` copies CCCL's headers and CMake config files into a prefix
+directory. CCCL is header-only, so the install has no build step and finishes in seconds.
+
+Use it when a downstream project needs ``find_package(CCCL)`` against a fixed checkout, or when you
+want CCCL on a system include path without cloning into the consumer's tree.
+
+Build and run a specific test
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``ci/util/build_and_test_targets.sh`` configures one preset, builds the Ninja targets you name,
+and runs the CTest or lit tests you name. It is the fast-iteration path for a single test or a
+handful of targets.
+
+::
+
+    ./ci/util/build_and_test_targets.sh \
+      --preset cub-cpp20 \
+      --build-targets "cub.test.iterator" \
+      --ctest-targets "cub.test.iterator"
+
+Use it when you are fixing one test and want a tight edit-build-run loop. Building tests does not
+require a GPU; running them does.
+
+Need a specific CTK or host compiler? Launch the matching container with ``.devcontainer/launch.sh``
+first, then run the script inside it. Valid toolchain combinations are in the
+``devcontainers:`` section of ``ci/matrix.yaml``; launching is covered at
+:ref:`infra-devcontainer-launching`.
+
+Build or test an entire project
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``ci/build_<project>.sh`` and ``ci/test_<project>.sh`` build or test a full project across a host
+compiler, C++ standard, and architecture set. These are the scripts CI runs, so they reproduce a
+CI job exactly.
+
+::
+
+    ./ci/build_cub.sh -cxx g++ -std 17 -arch "75;80;90"
+    ./ci/test_cub.sh  -cxx g++ -std 17 -arch "75;80;90"
+
+Use them to reproduce a CI failure or to validate a project end to end before pushing. A full
+project build takes hours; a targeted ``build_and_test_targets.sh`` run takes minutes. Test scripts
+require a GPU.
+
+Need a specific toolchain? Run these inside a devcontainer launched with ``.devcontainer/launch.sh``
+(:ref:`infra-devcontainer-launching`). CI failure logs print the exact container and arguments to
+reproduce the job.
+
+Bisect a regression
+~~~~~~~~~~~~~~~~~~~
+
+``ci/util/git_bisect.sh`` walks the commit history between a good and a bad ref, building and
+testing each candidate, until it pins the commit that introduced a regression. It takes the same
+``--preset``, ``--build-targets``, and ``--ctest-targets`` arguments as
+``build_and_test_targets.sh``.
+
+::
+
+    ./ci/util/git_bisect.sh \
+      --preset cub-cpp20 \
+      --build-targets "cub.test.iterator" \
+      --ctest-targets "cub.test.iterator" \
+      --good-ref v1.13.0 \
+      --bad-ref origin/main
+
+Use it when a test passes on an old ref and fails on ``main`` and you need the offending commit.
+``--repeat N`` re-runs multiple times to help catch intermittent failures.
+
+Need a specific toolchain? Run the script inside a devcontainer launched with
+``.devcontainer/launch.sh`` (:ref:`infra-devcontainer-launching`).
+
+Members can run the same bisect remotely on CI machines through the `Git Bisect workflow
+<https://github.com/NVIDIA/cccl/actions/workflows/git-bisect.yml>`_: choose a runner, set the good
+and bad refs and target arguments, and dispatch. Source is ``.github/workflows/git-bisect.yml``.
+
+Request a benchmark comparison
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Compare benchmark results between two refs from the browser, a local script, or a pull request.
+
+Members can dispatch the `Benchmark Compare workflow
+<https://github.com/NVIDIA/cccl/actions/workflows/bench.yml>`_ from the browser: choose "Run
+workflow", set the base and test refs and the CUB and Python filters, and it runs on the CI GPU
+pool. Source is ``.github/workflows/bench.yml``.
+
+Locally, ``ci/bench/bench.sh <base> <test>`` runs the same comparison against checked-out refs::
+
+    ./ci/bench/bench.sh origin/main HEAD --cub-filter "^cub\.bench\.copy\.memcpy\.base$"
+
+It wraps ``ci/bench/compare_git_refs.sh`` and ``ci/bench/compare_paths.sh``; call those directly
+when you already have two checkouts.
+
+To benchmark inside a PR, edit ``ci/bench.yaml`` to set GPUs and filters and push; PR CI detects the
+diff from ``ci/bench.template.yaml`` and dispatches the jobs. Reset ``ci/bench.yaml`` to match the
+template before merging. Argument behavior and artifact layout live in ``ci/bench/README.md``.
+
+Full CMake control
+~~~~~~~~~~~~~~~~~~
+
+``cmake --preset <name>`` configures a build directory directly.
+This is discouraged, but available for custom workflows / tool integrations.
+
+::
+
+    cmake --preset all-dev
+    cmake --build --preset all-dev
+    ctest --preset all-dev
+
+List the presets with ``cmake --list-presets``. The ``all-dev`` preset enables every library,
+test, and example against your native GPU; per-library presets like ``cub-cpp20`` scope the build
+to one library and standard. The :ref:`infra-cmake-preset-reference` catalogs the full preset set.
diff --git a/cccl_upstream/docs/contributors/how_tos/launching.rst b/cccl_upstream/docs/contributors/how_tos/launching.rst
new file mode 100644
index 00000000..07a0049d
--- /dev/null
+++ b/cccl_upstream/docs/contributors/how_tos/launching.rst
@@ -0,0 +1,89 @@
+.. _infra-devcontainer-launching:
+
+Launching a container
+=====================
+
+``.devcontainer/launch.sh`` starts a CCCL development container with a chosen CUDA
+toolkit and host compiler, mounts the repo, and either opens VSCode or drops you into
+a shell. It is Linux-only (including WSL2). With no toolchain flags, it uses the default devcontainer
+in ``.devcontainer/devcontainer.json``, which uses the latest CTK + gcc.
+
+Launch in VSCode
+----------------
+
+**Open the default container.** Run the script with no flags.
+
+.. code-block:: bash
+
+   .devcontainer/launch.sh
+
+The script copies the selected ``devcontainer.json`` into a temporary directory and
+opens VSCode against it.
+
+**Select a toolchain.** Pass ``--cuda`` and ``--host`` to open a specific variant.
+
+.. code-block:: bash
+
+   .devcontainer/launch.sh --cuda <cuda-version> --host <host-compiler>
+
+The temporary-directory copy lets you run multiple variants of the same environment
+side by side, each in its own VSCode window.
+
+Launch directly in Docker
+-------------------------
+
+**Drop into a shell.** Add ``--docker`` to skip VSCode and run a bash shell inside the
+container.
+
+.. code-block:: bash
+
+   .devcontainer/launch.sh --docker --cuda <cuda-version> --host <host-compiler>
+
+The container mounts the repo at ``/home/coder/cccl`` and removes itself on exit. Any
+trailing arguments after the flags run as a command instead of an interactive shell.
+
+Specify the toolchain
+---------------------
+
+``--cuda`` selects the CUDA toolkit version. ``--host`` selects the host compiler.
+The two flags resolve to ``.devcontainer/cuda<cuda>-<host>/devcontainer.json``; an
+unknown combination exits with an error.
+
+Valid values come from the devcontainers located under ``.devcontainer/``.
+
+Pass through GPUs
+-----------------
+
+**Add host GPUs.** Pass ``--gpus all`` to expose every host GPU to the container.
+
+.. code-block:: bash
+
+   .devcontainer/launch.sh --docker --cuda <cuda-version> --host <host-compiler> --gpus all
+
+``--gpus`` takes any Docker GPU request string. It overrides the ``hostRequirements.gpu``
+default read from the devcontainer config. Without it, the container starts without GPU
+access, which is sufficient for building tests.
+
+Launch from a git worktree
+--------------------------
+
+``launch.sh`` handles linked worktrees automatically. A worktree's ``.git`` is a file
+pointing at the main repository's git directory, which the container cannot reach through
+the worktree mount alone. The script bind-mounts the main repo's git common directory at
+its host path so git operations resolve inside the container.
+
+The ``cccl-build`` and ``cccl-wheelhouse`` Docker volumes are shared across all worktrees
+and the main checkout, so build artifacts collide between them. Do not run multiple
+worktree containers concurrently unless you are careful to avoid conflicts.
+
+First-time git auth in a worktree needs the main checkout's ``.config/gh``. If the main
+checkout has never run its container, ``launch.sh`` warns that startup will block on an
+interactive ``gh auth login``. Launch the main checkout's container once, complete the
+login, then re-launch the worktree.
+
+Forward SSH keys
+----------------
+
+If ``SSH_AUTH_SOCK`` is set in your environment, ``launch.sh`` forwards the agent socket
+into the container automatically. No flag is required. Git operations over SSH inside the
+container use your host agent's keys.
diff --git a/cccl_upstream/docs/contributors/how_tos/override_matrix.rst b/cccl_upstream/docs/contributors/how_tos/override_matrix.rst
new file mode 100644
index 00000000..050acf8c
--- /dev/null
+++ b/cccl_upstream/docs/contributors/how_tos/override_matrix.rst
@@ -0,0 +1,109 @@
+.. _infra-ci-override-matrix:
+
+Override matrix
+===============
+
+The override matrix scopes a pull request's CI to a chosen subset of jobs.
+When the ``workflows.override`` key in ``ci/matrix.yaml`` is non-empty, it
+replaces the entire ``pull_request`` matrix. The PR runs only the override
+jobs, and branch protection blocks the merge until the override is empty
+again.
+
+Use the override matrix to:
+
+- Test a new compiler's nightly / weekly jobs from the PR before merging.
+- Validate a compiler-specific or GPU-specific fix against one combo.
+- Test CI infrastructure changes that need only a few jobs to validate.
+- Debug a nightly failure by running only the combos that failed.
+
+The override matrix is a temporary scoping tool, not a permanent matrix
+edit. Every override entry must be removed before the PR lands.
+
+Add an override entry
+---------------------
+
+The override lives at the top of ``ci/matrix.yaml`` under
+``workflows.override``. The default value is empty. Override entries use the
+same syntax as ``pull_request`` entries.
+
+**Step 1. Pick the combo to test.** Identify the exact job, project,
+compiler, CTK version, and GPU you need. For a compiler-specific fix, this is
+one ``cxx`` value. For a nightly failure, copy the failing entry from the
+``nightly`` workflow.
+
+**Step 2. Add the entry under** ``override``. Edit ``ci/matrix.yaml`` and
+add one mapping under the ``override:`` key:
+
+.. code-block:: yaml
+
+   workflows:
+     override:
+       - {jobs: ['test'], project: 'thrust', std: 'max', ctk: '<ctk>', cxx: '<compiler>', gpu: '<gpu>'}
+
+     pull_request:
+       - <...>
+
+Choose ``ctk``, ``cxx``, and ``gpu`` values from the existing
+``pull_request`` entries in ``ci/matrix.yaml``. Each field scopes the run:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 80
+
+   * - Field
+     - Meaning
+   * - ``jobs``
+     - Job types to run. A ``test`` entry auto-generates any build jobs it
+       depends on.
+   * - ``project``
+     - Which project to build or test (``thrust``, ``cub``, ``libcudacxx``, ``cudax``, ...).
+   * - ``std``
+     - C++ standard. ``max`` selects the highest standard the combo supports.
+   * - ``ctk``
+     - CUDA Toolkit version. A ``<major>.X`` suffix selects the newest image for that major version.
+   * - ``cxx``
+     - Host compiler. A single value runs one compiler; an array expands to several jobs.
+   * - ``gpu``
+     - GPU runner model. Required for ``test`` jobs.
+
+Field defaults and the full tag list live in the ``tags`` section of
+``ci/matrix.yaml``.
+
+**Step 3. Trim turnaround with targeted builds.** A full project build is
+slow. To build and run a single test target instead, use ``project:
+'target'`` and pass ``args`` to ``ci/util/build_and_test_targets.sh``:
+
+.. code-block:: yaml
+
+   workflows:
+     override:
+       - {jobs: ['run_gpu'], project: 'target', ctk: '<ctk>', cxx: '<compiler>', gpu: '<gpu>',
+          args: '--preset <preset> --build-targets "<target>" --ctest-targets "<target>"'}
+
+The ``run_cpu`` and ``run_gpu`` jobs map directly to
+``build_and_test_targets.sh``. See that script for the available ``args``,
+covered in :doc:`/cccl/development/build_and_bisect_tools`.
+
+**Step 4. Reduce overhead further with skip tags.** Combine the override
+with :ref:`[skip-*] tags <infra-ci-skip-tags>` in the last commit message to drop
+devcontainer, docs, and third-party canary jobs:
+
+.. code-block:: bash
+
+   git commit -m "Debug <compiler> <project> failure [skip-vdc][skip-docs][skip-tpt]"
+
+Run scoped CI and merge
+-----------------------
+
+The override is temporary: scoped jobs first, full matrix before merge.
+
+1. **Add the override entry and push.** Only the override jobs run; the
+   ``pull_request`` matrix is skipped while ``workflows.override`` is non-empty.
+
+2. **Iterate until the override jobs pass.** Each push reruns only those
+   jobs, keeping turnaround short.
+
+3. **Empty the override and push again.** With ``workflows.override`` reset
+   to empty, the full ``pull_request`` matrix runs — the suite that gates merge.
+
+4. **Merge once the full matrix is green.**
diff --git a/cccl_upstream/docs/contributors/how_tos/reproducing_ci.rst b/cccl_upstream/docs/contributors/how_tos/reproducing_ci.rst
new file mode 100644
index 00000000..00d224da
--- /dev/null
+++ b/cccl_upstream/docs/contributors/how_tos/reproducing_ci.rst
@@ -0,0 +1,121 @@
+.. _infra-ci-reproducing-locally:
+
+Reproducing CI locally
+======================
+
+A failing CI job prints everything needed to reproduce it. Open the failed
+job from the PR's checks list and expand the log.
+
+On failure: reproduction block
+------------------------------
+
+When a job exits non-zero, the runner emits an
+**Instructions to Reproduce CI Failure Locally** block. It contains two steps:
+
+**Step 1** — clone the repository at the exact SHA under test:
+
+.. code-block:: bash
+
+   git clone --branch <branch> --single-branch https://github.com/NVIDIA/cccl.git \
+     && cd cccl && git checkout <sha>
+
+**Step 2** — launch the same container and re-run the same command:
+
+.. code-block:: bash
+
+   .devcontainer/launch.sh -d -c <cuda> -H <host> -- <command>
+
+Copy both lines verbatim. The ``-c`` and ``-H`` values are the CTK version and
+host compiler for this job. ``<command>`` is the ``ci/*.sh`` invocation the
+runner used, with all flags.
+
+.. note::
+
+   GPU test jobs omit the GPU flag from the printed command. Add ``--gpus all``
+   when reproducing a test job on a machine with a GPU.
+
+Job Inputs block
+----------------
+
+The **Job Inputs** group near the top of every log records the full job
+configuration, printed before the container launches:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 30 70
+
+   * - Field
+     - Meaning
+   * - ``Job command``
+     - The ``ci/*.sh`` script and flags the runner will execute
+   * - ``JOB_ID``
+     - Unique job identifier; pass it to ``create_mock_job_env.sh`` to reproduce the job's
+       environment locally
+   * - ``JOB_CUDA``
+     - CTK version (e.g. ``13.3``)
+   * - ``JOB_HOST``
+     - Host compiler identifier (e.g. ``gcc15``)
+   * - ``JOB_IMAGE``
+     - Full RAPIDS devcontainer image tag pulled for this job
+   * - ``JOB_RUNNER``
+     - Runner label; labels containing ``-gpu-`` mean the job used a GPU
+   * - ``JOB_ENVIRONMENT``
+     - Extra environment variables injected into the container
+
+These are the same values that feed the ``launch.sh`` invocation in the
+failure block.
+
+Override matrix entry
+---------------------
+
+The **Override matrix entry** block, also printed before the container
+launches, contains a YAML snippet:
+
+.. code-block:: yaml
+
+    - {jobs: [...], project: '...', ctk: '...', cxx: '...', ...}
+
+Paste this into the ``workflows.override`` list in ``ci/matrix.yaml`` and push
+to re-run only that specific job in CI without waiting for the full matrix.
+See :ref:`infra-ci-override-matrix` for the override workflow.
+
+Mock the job environment
+------------------------
+
+.. note::
+
+   This step is only needed when debugging the CCCL workflow / artifact scripts (rare).
+   More project contributors can safely skip this.
+
+To run the ``ci/util/workflow/`` and ``ci/util/artifacts/`` scripts outside CI —
+debugging artifact upload, download, or producer resolution locally — recreate the
+job's environment with ``create_mock_job_env.sh``:
+
+.. code-block:: bash
+
+   ci/util/create_mock_job_env.sh <run_id> <job_id>
+
+Run it inside a devcontainer. It takes the ``<run_id>`` and ``<job_id>`` from the job
+log (``GITHUB_RUN_ID`` and ``JOB_ID``), sets the ``GITHUB_*`` variables CI exports,
+sources the workflow and artifact helpers, clears stale local artifact directories, and
+drops into a shell that mimics the in-container CI environment. Every job log prints the
+exact command under ``Mock with:``.
+
+Tighten the loop
+----------------
+
+The full ``ci/build_*.sh`` and ``ci/test_*.sh`` scripts build and run an
+entire project's test suite. Once the container reproduces the failure, use
+``ci/util/build_and_test_targets.sh`` to build and run only the failing target:
+
+.. code-block:: bash
+
+   .devcontainer/launch.sh --docker --cuda 13.0 --host gcc14 --gpus all -- \
+     ci/util/build_and_test_targets.sh \
+       --preset cub-cpp20 \
+       --build-targets "cub.test.iterator" \
+       --ctest-targets "cub.test.iterator"
+
+This rebuilds one target instead of the full project. For preset, target, and
+lit-test flags, and for ``ci/util/git_bisect.sh`` to find the introducing commit,
+see :doc:`/cccl/development/build_and_bisect_tools`.
diff --git a/cccl_upstream/docs/contributors/how_tos/targeted_builds.rst b/cccl_upstream/docs/contributors/how_tos/targeted_builds.rst
new file mode 100644
index 00000000..2e75974f
--- /dev/null
+++ b/cccl_upstream/docs/contributors/how_tos/targeted_builds.rst
@@ -0,0 +1,75 @@
+.. _infra-ci-targeted-builds:
+
+Build and run targeted tests
+============================
+
+The ``ci/build_*.sh`` and ``ci/test_*.sh`` scripts build and run all headers,
+tests, examples, etc for a single project. It is the right tool for reproducing
+a CI job, but it is slow when you are iterating on a single test.
+
+``ci/util/build_and_test_targets.sh`` builds and runs a named subset of CMake
+targets against one preset. Use it to compile one test, run one CTest pattern,
+or execute one libcudacxx lit test without rebuilding the rest of the project.
+The full flag reference is in :doc:`/cccl/development/build_and_bisect_tools`.
+
+Build a single CUB test
+-----------------------
+
+CCCL tests usually have a single name for their CMake target, ninja target, and CTest target.
+It uniquely encodes the project, path, and test case, eg: ``cub.test.iterator``.
+
+#. **Configure and build the target.** Pass the preset and the metatarget to
+   ``--build-targets``::
+
+     ci/util/build_and_test_targets.sh \
+       --preset cub-cpp20 \
+       --build-targets "cub.test.iterator"
+
+   With no ``--ctest-targets``, the script configures and compiles, then stops.
+   Compiling a test does not require a GPU.
+
+#. **Run the target.** Add ``--ctest-targets`` with a CTest ``-R`` regex. The
+   metatarget name works directly as the pattern::
+
+     ci/util/build_and_test_targets.sh \
+       --preset cub-cpp20 \
+       --build-targets "cub.test.iterator" \
+       --ctest-targets "cub.test.iterator"
+
+   If ``--build-targets`` is omitted, the script assumes the targets are already built and
+   skips to testing. Running tests may require a GPU.
+
+Run a libcudacxx lit test
+-------------------------
+
+Some libcudacxx tests run under lit, not CTest. Pass lit test paths relative to
+``libcudacxx/test/libcudacxx/``.
+
+#. **Execute one lit test.** Use ``--lit-tests`` with the test path::
+
+     ci/util/build_and_test_targets.sh \
+       --preset libcudacxx \
+       --lit-tests \
+         "std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp"
+
+#. **Precompile without running.** Use ``--lit-precompile-tests`` to compile the
+   test with a no-op executor. This catches compile errors without a GPU::
+
+     ci/util/build_and_test_targets.sh \
+       --preset libcudacxx \
+       --lit-precompile-tests \
+         "std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp"
+
+Run inside a devcontainer
+-------------------------
+
+To build against a specific CUDA toolkit and host compiler, wrap the invocation
+with ``.devcontainer/launch.sh -d``. Valid CTK and host compiler values are
+listed in the ``.devcontainer`` directory. Pass ``--gpus all`` when the run
+needs a device::
+
+  .devcontainer/launch.sh -d --cuda <CTK> --host <compiler> --gpus all -- \
+    ci/util/build_and_test_targets.sh \
+      --preset cub-cpp20 \
+      --build-targets "cub.test.iterator" \
+      --ctest-targets "cub.test.iterator"
diff --git a/cccl_upstream/docs/contributors/index.rst b/cccl_upstream/docs/contributors/index.rst
new file mode 100644
index 00000000..d079ce1e
--- /dev/null
+++ b/cccl_upstream/docs/contributors/index.rst
@@ -0,0 +1,11 @@
+.. _contributors-index:
+
+Contributor Docs
+================
+
+This section covers the build, test, and debug workflows a CCCL contributor uses day to day.
+
+.. toctree::
+   :maxdepth: 1
+
+   how_tos/index
diff --git a/cccl_upstream/docs/cpp.rst b/cccl_upstream/docs/cpp.rst
new file mode 100644
index 00000000..81a89cc2
--- /dev/null
+++ b/cccl_upstream/docs/cpp.rst
@@ -0,0 +1,70 @@
+.. _cccl-cpp-libraries:
+
+CUDA C++ Core Libraries
+=======================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 3
+
+   libcudacxx/index
+   cub/index
+   thrust/index
+   cudax/index
+   cccl/tma
+   cccl/determinism
+   cccl/config_macros
+   cccl/3.0_migration_guide
+   cccl/development/index
+   cccl/contributing
+   cccl/license
+
+Welcome to the CUDA Core Compute Libraries (CCCL) libraries for C++.
+
+The concept for the  CCCL C++ librarires grew organically out of the Thrust,
+CUB, and libcudacxx projects that were developed independently over the years
+with a similar goal: to provide high-quality, high-performance, and
+easy-to-use C++ abstractions for CUDA developers. Naturally, there was a lot
+of overlap among the three projects, and it became clear the community would
+be better served by unifying them into a single repository.
+
+- :doc:`libcu++ <libcudacxx/index>`
+  is the CUDA C++ Standard Library. It provides an implementation of the C++
+  Standard Library that works in both host and device code. Additionally, it
+  provides abstractions for CUDA-specific hardware features like
+  synchronization primitives, cache control, atomics, and more.
+
+- :doc:`CUB <cub/index>`
+  is a lower-level, CUDA-specific library designed for speed-of-light parallel
+  algorithms across all GPU architectures. In addition to device-wide
+  algorithms, it provides *cooperative algorithms* like block-wide reduction
+  and warp-wide scan, providing CUDA kernel developers with building blocks to
+  create speed-of-light, custom kernels.
+
+- :doc:`Thrust <thrust/index>`
+  is the C++ parallel algorithms library which inspired the introduction of
+  parallel algorithms to the C++ Standard Library. Thrust's high-level
+  interface greatly enhances programmer productivity while enabling performance
+  portability between GPUs and multicore CPUs via configurable backends that
+  allow using multiple parallel programming frameworks (such as CUDA, TBB, and
+  OpenMP).
+
+- :doc:`Cuda Experimental <cudax/index>`
+  is a library of experimental features that are still in the design process.
+
+The main goal of the CCCL C++ libraries is to fill a similar role that the
+Standard C++ Library fills for Standard C++: provide general-purpose,
+speed-of-light tools to CUDA C++ developers, allowing them to focus on
+solving the problems that matter. Unifying these projects is the first step
+towards realizing that goal.
+
+CUDA C++ Tile support
+======================
+
+`CUDA Tile <https://developer.nvidia.com/cuda/tile#section-more-resources>`_ introduces a new way to program GPUs at a higher level than SIMT.
+
+Due to the additional constraints of tile compilation :doc:`CUB <cub/index>`, :doc:`Thrust <thrust/index>`
+and :doc:`Cuda Experimental <cudax/index>` are currently not supported in a tile program.
+
+There is partial support for :doc:`libcu++ <libcudacxx/index>` for tile programs.
+The current support matrix can be found in the :doc:`libcu++ documentation <libcudacxx/tile>`
diff --git a/cccl_upstream/docs/cub/Doxyfile b/cccl_upstream/docs/cub/Doxyfile
new file mode 100644
index 00000000..4d677c3c
--- /dev/null
+++ b/cccl_upstream/docs/cub/Doxyfile
@@ -0,0 +1,183 @@
+# Doxyfile for CUB
+
+PROJECT_NAME           = CUB
+OUTPUT_DIRECTORY       = ../_build/doxygen/cub
+CREATE_SUBDIRS         = NO
+GENERATE_HTML          = NO
+GENERATE_LATEX         = NO
+GENERATE_XML           = YES
+XML_OUTPUT             = xml
+XML_PROGRAMLISTING     = YES
+
+INPUT                  = ../../cub/cub \
+                         ../../cub/cub/thread \
+                         ../../cub/cub/warp \
+                         ../../cub/cub/block \
+                         ../../cub/cub/device \
+                         ../../cub/cub/device/dispatch/tuning \
+                         ../../cub/cub/grid \
+                         ../../cub/cub/iterator
+
+RECURSIVE              = YES
+EXCLUDE_PATTERNS       = */detail/* */dispatch/dispatch_* */dispatch/kernels/* */kernels/* */test/* */examples/*
+EXCLUDE_SYMBOLS        = *detail* CUB_DETAIL*
+
+FILE_PATTERNS          = *.cuh *.h
+EXTENSION_MAPPING      = cuh=C++ cu=C++
+
+# Documentation extraction settings
+EXTRACT_ALL            = YES
+EXTRACT_PRIVATE        = NO
+EXTRACT_STATIC         = YES
+EXTRACT_LOCAL_CLASSES  = YES
+HIDE_UNDOC_MEMBERS     = NO
+HIDE_UNDOC_CLASSES     = NO
+SHOW_INCLUDE_FILES     = YES
+INLINE_INHERITED_MEMB  = YES
+FULL_PATH_NAMES        = YES
+STRIP_FROM_PATH        = ../../cub
+STRIP_FROM_INC_PATH    = ../../cub
+SHORT_NAMES            = NO
+
+# Parsing settings
+JAVADOC_AUTOBRIEF      = YES
+QT_AUTOBRIEF           = NO
+MULTILINE_CPP_IS_BRIEF = NO
+INHERIT_DOCS           = YES
+SEPARATE_MEMBER_PAGES  = NO
+TAB_SIZE               = 4
+BUILTIN_STL_SUPPORT    = YES
+
+# Preprocessing
+ENABLE_PREPROCESSING   = YES
+MACRO_EXPANSION        = YES
+EXPAND_ONLY_PREDEF     = NO
+SEARCH_INCLUDES        = YES
+SKIP_FUNCTION_MACROS   = YES
+
+# IMPORTANT: Aliases for custom commands
+# The rst alias enables embedding reStructuredText in doxygen comments
+ALIASES = "rst=\verbatim embed:rst:leading-asterisk"
+ALIASES += "endrst=\endverbatim"
+ALIASES += "rststar=\verbatim embed:rst:leading-asterisk"
+ALIASES += "inlinerst=\verbatim embed:rst:inline"
+
+# Key aliases that are used within @rst blocks (from repo.toml)
+ALIASES += "smemwarpreuse=A subsequent ``__syncwarp()`` warp-wide barrier should be invoked after calling this method if the collective's temporary storage (e.g., ``temp_storage``) is to be reused or repurposed."
+ALIASES += "smemreuse=A subsequent ``__syncthreads()`` threadblock barrier should be invoked after calling this method if the collective's temporary storage (e.g., ``temp_storage``) is to be reused or repurposed."
+ALIASES += "smemreuse{1}=After any operation, a subsequent ``__syncthreads()`` barrier is required if the collective's \1 is to be reused or repurposed"
+ALIASES += "smemstorage{1}=The operations exposed by \1 require a temporary memory allocation of this nested type for thread communication. This opaque storage can be allocated directly using the ``__shared__`` keyword. Alternatively, it can be aliased to externally allocated memory (shared or global) or ``union``'d with other storage allocation types to facilitate memory reuse."
+ALIASES += "granularity=Efficiency is increased with increased granularity ``ITEMS_PER_THREAD``. Performance is also typically increased until the additional register pressure or shared memory allocation size causes SM occupancy to fall too low. Consider variants of ``cub::BlockLoad`` for efficiently gathering a :ref:`blocked arrangement <flexible-data-arrangement>` of elements across threads."
+ALIASES += "blocksize=The number of threads in the block is a multiple of the architecture's warp size"
+ALIASES += "ptxversion=The PTX compute capability for which to to specialize this collective, formatted as per the ``__CUDA_ARCH__`` macro (e.g., 750 for sm_75). Useful for determining the collective's storage requirements for a given device from the host. (Default: the value of ``__CUDA_ARCH__`` during the current compiler pass)"
+ALIASES += "blockcollective{1}=Every thread in the block uses the \1 class by first specializing the \1 type, then instantiating an instance with parameters for communication, and finally invoking one or more collective member functions."
+ALIASES += "warpcollective{1}=Every thread in the warp uses the \1 class by first specializing the \1 type, then instantiating an instance with parameters for communication, and finally invoking or more collective member functions."
+ALIASES += "devicestorage=Temporary storage for this operation. If ``d_temp_storage`` is ``nullptr``, the required size is written to ``temp_storage_bytes`` without dereferencing iterators or launching kernels. Otherwise, ``d_temp_storage`` must point to a device-accessible allocation of at least ``temp_storage_bytes`` bytes. No special alignment is required. See :ref:`device-temp-storage` for usage guidance."
+ALIASES += "devicestorageP=This operation requires a relatively small allocation of temporary device storage that is ``O(P)``, where ``P`` is the number of streaming multiprocessors on the device (and is typically a small constant relative to the input size ``N``)."
+ALIASES += "devicestorageNP=This operation requires an allocation of temporary device storage that is ``O(N+P)``, where ``N`` is the length of the input and ``P`` is the number of streaming multiprocessors on the device."
+ALIASES += "devicestorageNCP=This operation requires a relatively small allocation of temporary device storage that is ``O(N/C + P)``, where ``N`` is the length of the input, ``C`` is the number of concurrent threads that can be actively scheduled on each streaming multiprocessor (typically several thousand), and ``P`` is the number of streaming multiprocessors on the device."
+ALIASES += "cdp_class{1}= - Dynamic parallelism. \1 methods can be called within kernel code on devices in which CUDA dynamic parallelism is supported."
+ALIASES += "determinism{1}= - Determinism. The default reproducibility guarantee is ``\1``. A different guarantee can be requested through the execution environment with ``cuda::execution::require``. See :ref:`Determinism in CUB <cub-determinism>` for the supported guarantees."
+ALIASES += "iterator=(may be a simple pointer type)"
+ALIASES += "offset_size1=(Consider using 32-bit values as offsets/lengths/etc. For example, ``int`` will typically yield better performance than ``size_t`` in 64-bit memory mode.)"
+ALIASES += "offset_size2=Careful consideration should be given to the size of integer types used for offsets and lengths. Many (if not most) scenarios will only require 32-bit offsets (e.g., ``int``). 64-bit offset types (e.g., ``size_t`` on 64-bit memory mode) can consume a significant amount of thread storage resources, adversely affecting processor occupancy and performance."
+ALIASES += "rowmajor=For multi-dimensional blocks, threads are linearly ranked in row-major order."
+ALIASES += "blocked=Assumes a :ref:`blocked arrangement <flexible-data-arrangement>` of (*block-threads* * *items-per-thread*) items across the thread block, where *thread*\ :sub:`i` owns the *i*\ :sup:`th` range of *items-per-thread* contiguous items. For multi-dimensional thread blocks, a row-major thread ordering is assumed."
+ALIASES += "striped=Assumes a :ref:`striped arrangement <flexible-data-arrangement>` of (*block-threads* * *items-per-thread*) items across the thread block, where *thread*\ :sub:`i` owns items (*i*), (*i* + *block-threads*), ..., (*i* + (*block-threads* * (*items-per-thread* - 1))).  For multi-dimensional thread blocks, a row-major thread ordering is assumed."
+ALIASES += "warpstriped=Assumes a *warp-striped arrangement* of elements across threads, where warp\ :sub:`i` owns the *i*\ :sup:`th` range of (*warp-threads* * *items-per-thread*) contiguous items, and each thread owns items (*i*), (*i* + *warp-threads*), ..., (*i* + (*warp-threads* * (*items-per-thread* - 1)))."
+ALIASES += "linear_performance{1}=The work-complexity of \1 as a function of input size is linear, resulting in performance throughput that plateaus with problem sizes large enough to saturate the GPU."
+ALIASES += "plots_below=Performance plots for other scenarios can be found in the detailed method descriptions below."
+ALIASES += "identityzero=This operation assumes the value of obtained by the ``T``'s default constructor (or by zero-initialization if no user-defined default constructor exists) is suitable as the identity value \"zero\" for addition."
+ALIASES += "lookback=`decoupled look-back <https://research.nvidia.com/publication/single-pass-parallel-prefix-scan-decoupled-look-back>`_"
+
+# Predefined macros (based on repo.toml doxygen_predefined)
+PREDEFINED             = __device__= \
+                        __host__= \
+                        __global__= \
+                        __forceinline__= \
+                        "__declspec(x)=" \
+                        "__align__(x)=" \
+                        __cccl_lib_mdspan \
+                        "CUB_NAMESPACE_BEGIN=namespace cub {" \
+                        "CUB_NAMESPACE_END=}" \
+                        "CUB_NS_PREFIX=" \
+                        "CUB_NS_POSTFIX=" \
+                        "CUB_NS_QUALIFIER=cub::" \
+                        "CUB_DETAIL_MAGIC_NS_BEGIN=" \
+                        "CUB_DETAIL_MAGIC_NS_END=" \
+                        "_CCCL_AND=&&" \
+                        "_CCCL_CONCEPT=constexpr bool " \
+                        "_CCCL_CONSTEXPR_FRIEND=friend " \
+                        "_CCCL_CONSTEXPR_CXX20=constexpr" \
+                        "_CCCL_CONSTEXPR_CXX23=constexpr" \
+                        "_CCCL_CTK_AT_LEAST(x, y)=1" \
+                        "_CCCL_CTK_BELOW(x, y)=0" \
+                        "_CCCL_CUDACC_AT_LEAST(x, y)=1" \
+                        "_CCCL_CUDACC_BELOW(x, y)=0" \
+                        _CCCL_DEVICE= \
+                        _CCCL_DIAG_PUSH= \
+                        _CCCL_DIAG_POP= \
+                        "_CCCL_DIAG_SUPPRESS_CLANG(x)=" \
+                        "_CCCL_DIAG_SUPPRESS_GCC(x)=" \
+                        "_CCCL_DIAG_SUPPRESS_MSVC(x)=" \
+                        "_CCCL_DIAG_SUPPRESS_NVHPC(x)=" \
+                        _CCCL_DOXYGEN_INVOKED \
+                        _CCCL_EXEC_CHECK_DISABLE= \
+                        _CCCL_FORCEINLINE= \
+                        "_CCCL_GLOBAL_CONSTANT=inline constexpr" \
+                        "_CCCL_HAS_CTK()=1" \
+                        _CCCL_HIDE_FROM_ABI= \
+                        _CCCL_HOST= \
+                        _CCCL_HOST_DEVICE= \
+                        "_CCCL_REQUIRES(x)= ::cuda::std::enable_if_t<x, int> = 0>" \
+                        _CCCL_STD_VER=2020 \
+                        _CCCL_SUPPRESS_DEPRECATED_PUSH= \
+                        _CCCL_SUPPRESS_DEPRECATED_POP= \
+                        "_CCCL_TEMPLATE(x)=template<x, " \
+                        "_CCCL_TRAIT(x, y)=x<y>::value" \
+                        "_CCCL_TRAILING_REQUIRES(x)=-> x requires " \
+                        _CCCL_TYPE_VISIBILITY_DEFAULT= \
+                        _CCCL_TYPE_VISIBILITY_HIDDEN= \
+                        _CCCL_API=inline \
+                        _CCCL_HOST_DEVICE_API=inline \
+                        _CCCL_DEVICE_API=inline \
+                        _CCCL_HOST_API=inline \
+                        _CCCL_NODEBUG_API=inline \
+                        _CCCL_NODEBUG_DEVICE_API=inline \
+                        _CCCL_NODEBUG_HOST_API=inline \
+                        _CCCL_TRIVIAL_API=inline \
+                        _CCCL_TRIVIAL_DEVICE_API=inline \
+                        _CCCL_TRIVIAL_HOST_API=inline \
+                        _CCCL_VISIBILITY_DEFAULT= \
+                        _CCCL_VISIBILITY_HIDDEN= \
+                        _CCCL_LIFETIMEBOUND= \
+                        _CCCL_TRY=try \
+                        _CCCL_CATCH=catch \
+                        "_CCCL_CATCH_ALL=catch (...)" \
+                        "_CCCL_CATCH_FALLTHROUGH=" \
+                        _CCCL_PUBLIC_API=inline \
+                        _CCCL_PUBLIC_DEVICE_API=inline \
+                        _CCCL_PUBLIC_HOST_API=inline \
+                        "_CUDAX_CONSTEXPR_FRIEND=friend" \
+                        "_LIBCUDACXX_HAS_SPACESHIP_OPERATOR()=1" \
+                        "CCCL_DEPRECATED=" \
+                        "CCCL_DEPRECATED_BECAUSE(x)=" \
+                        "CCCL_IGNORE_DEPRECATED_CPP_DIALECT" \
+                        "CUB_DISABLE_NAMESPACE_MAGIC" \
+                        "CUB_IGNORE_NAMESPACE_MAGIC_ERROR" \
+                        "CUB_RUNTIME_FUNCTION=" \
+                        "THRUST_FWD(x)=x" \
+                        "THRUST_NAMESPACE_BEGIN=namespace thrust {" \
+                        "THRUST_NAMESPACE_END=}" \
+                        "THRUST_PREVENT_MACRO_SUBSTITUTION" \
+                        "_CCCL_HOSTED()=1" \
+                        _CCCL_DOXYGEN_INVOKED
+
+# Quiet mode
+QUIET                  = YES
+WARNINGS               = YES
+WARN_AS_ERROR          = FAIL_ON_WARNINGS
+WARN_IF_UNDOCUMENTED   = NO
+WARN_IF_DOC_ERROR      = YES
+WARN_IF_INCOMPLETE_DOC = NO
+WARN_NO_PARAMDOC       = NO
diff --git a/cccl_upstream/docs/cub/benchmarking.rst b/cccl_upstream/docs/cub/benchmarking.rst
new file mode 100644
index 00000000..917b0154
--- /dev/null
+++ b/cccl_upstream/docs/cub/benchmarking.rst
@@ -0,0 +1,376 @@
+Benchmarks
+*************************************
+
+.. TODO(bgruber): this guide applies to Thrust as well. We should rename it to "CCCL Benchmarks" and move it out of CUB
+
+CUB comes with a set of `NVBench <https://github.com/NVIDIA/nvbench>`_-based benchmarks for its algorithms,
+which can be used to measure the performance of CUB on your system on a variety of workloads.
+The integration with NVBench allows to archive and compare benchmark results,
+which is useful for continuous performance testing, detecting regressions, tuning, and optimization.
+This guide gives an introduction into CUB's benchmarking infrastructure.
+
+Building benchmarks
+--------------------------------------------------------------------------------
+
+CUB benchmarks are build as part of the CCCL CMake infrastructure.
+Starting from scratch:
+
+.. code-block:: bash
+
+    git clone https://github.com/NVIDIA/cccl.git
+    cd cccl
+    mkdir build
+    cd build
+    cmake .. --preset=benchmark
+
+You clone the repository, create a build directory and configure the build with CMake.
+The preset `benchmark` takes care of everything.
+
+.. TODO(bgruber): do we have a public NVIDIA maintained table I can link here instead?
+
+We use Ninja as CMake generator in this guide, but you can use any other generator you prefer.
+
+You can then proceed to build the benchmarks.
+
+You can list the available cmake build targets with, if you intend to only build selected benchmarks:
+
+.. code-block:: bash
+
+    ninja -t targets | grep '\.bench\.'
+    cub.bench.adjacent_difference.subtract_left.base: phony
+    cub.bench.copy.memcpy.base: phony
+    ...
+    cub.bench.transform.babelstream3.base: phony
+    cub.bench.transform_reduce.sum.base: phony
+
+We also provide a target to build all benchmarks:
+
+.. code-block:: bash
+
+    ninja cub.all.benches
+
+
+.. _cub-benchmarking-running:
+
+Running a benchmark
+--------------------------------------------------------------------------------
+
+After we built a benchmark, we can run it as follows:
+
+.. code-block:: bash
+
+    ./bin/cub.bench.adjacent_difference.subtract_left.base\
+        -d 0\
+        --stopping-criterion entropy\
+        --json base.json\
+        --md base.md
+
+In this command, `-d 0` indicates that we want to run on GPU 0 on our system.
+Setting `--stopping-criterion entropy` is advisable since it reduces runtime
+and increase confidence in the resulting data.
+It's not set as default yet, because NVBench is still evaluating it.
+By default, NVBench will print the benchmark results to the terminal as Markdown.
+`--json base.json` will save the detailed results in a JSON file as well for later use.
+`--md base.md` will save the Markdown output to a file as well,
+so you can easily view the results later without having to parse the JSON.
+More information on what command line options are available can be found in the
+`NVBench documentation <https://github.com/NVIDIA/nvbench/blob/main/docs/cli_help.md>`__.
+
+The expected terminal output is something along the following lines (also saved to `base.md`),
+shortened for brevity:
+
+.. code-block:: bash
+
+    # Log
+    Run:  [1/8] base [Device=0 T{ct}=I32 OffsetT{ct}=I32 Elements{io}=2^16]
+    Pass: Cold: 0.004571ms GPU, 0.009322ms CPU, 0.00s total GPU, 0.01s total wall, 334x
+    Run:  [2/8] base [Device=0 T{ct}=I32 OffsetT{ct}=I32 Elements{io}=2^20]
+    Pass: Cold: 0.015161ms GPU, 0.023367ms CPU, 0.01s total GPU, 0.02s total wall, 430x
+    ...
+    # Benchmark Results
+    | T{ct} | OffsetT{ct} |   Elements{io}   | Samples |  CPU Time  |  Noise  |  GPU Time  | Noise  | Elem/s  | GlobalMem BW | BWUtil |
+    |-------|-------------|------------------|---------|------------|---------|------------|--------|---------|--------------|--------|
+    |   I32 |         I32 |     2^16 = 65536 |    334x |   9.322 us | 104.44% |   4.571 us | 10.87% | 14.337G | 114.696 GB/s | 14.93% |
+    |   I32 |         I32 |   2^20 = 1048576 |    430x |  23.367 us | 327.68% |  15.161 us |  3.47% | 69.161G | 553.285 GB/s | 72.03% |
+    ...
+
+If you are only interested in a subset of workloads, you can restrict benchmarking as follows:
+
+.. code-block:: bash
+
+    ./bin/cub.bench.adjacent_difference.subtract_left.base ...\
+        -a 'T{ct}=I32'\
+        -a 'OffsetT{ct}=I32'\
+        -a 'Elements{io}[pow2]=[24,28]'\
+
+The `-a` option allows you to restrict the values for each axis available for the benchmark.
+See the `NVBench documentation <https://github.com/NVIDIA/nvbench/blob/main/docs/cli_help_axis.md>`__.
+for more information on how to specify the axis values.
+If the specified axis does not exist, the benchmark will terminate with an error.
+
+If you want to plot the benchmark results, you can use the following script:
+
+.. code-block:: bash
+
+    PYTHONPATH=./_deps/nvbench-src/python/scripts ./_deps/nvbench-src/python/scripts/nvbench_plot_bwutil.py base.json
+
+The `-a` option is supported to restrict the values for some axes as well,
+which is useful if you want to plot only a subset of workloads.
+Use the `-b` option to select a specific benchmark by name
+in case your JSON file contains results for multiple benchmarks.
+Multiple benchmarks are selected by repeating the `-b` option.
+
+.. code-block:: bash
+
+    PYTHONPATH=./_deps/nvbench-src/python/scripts ./_deps/nvbench-src/python/scripts/nvbench_plot_bwutil.py \
+        -b base -a Elements{io}[pow2]=28 base.json
+
+.. _cub-benchmarking-comparing:
+
+Comparing benchmark results
+--------------------------------------------------------------------------------
+
+Let's say you have a modification that you'd like to benchmark.
+To compare the performance you have to build and run the benchmark as described above for the unmodified code,
+saving the results to a JSON file, e.g. `base.json`.
+Then, you apply your code changes (e.g., switch to a different branch, git stash pop, apply a patch file, etc.),
+rebuild and rerun the benchmark, saving the results to a different JSON file, e.g. `new.json`.
+
+You can now compare the two result JSON files using, assuming you are still in your build directory:
+
+.. code-block:: bash
+
+    PYTHONPATH=./_deps/nvbench-src/python/scripts ./_deps/nvbench-src/python/scripts/nvbench_compare.py base.json new.json
+
+The `PYTHONPATH` environment variable may not be necessary in all cases.
+The script will print a Markdown report showing the runtime differences between each variant of the two benchmark run.
+This could look like this, again shortened for brevity:
+
+.. code-block:: bash
+
+    |  T{ct}  |  OffsetT{ct}  |  Elements{io}  |   Ref Time |   Ref Noise |   Cmp Time |   Cmp Noise |       Diff |   %Diff |  Status  |
+    |---------|---------------|----------------|------------|-------------|------------|-------------|------------|---------|----------|
+    |   I32   |      I32      |      2^16      |   4.571 us |      10.87% |   4.096 us |       0.00% |  -0.475 us | -10.39% |   FAIL   |
+    |   I32   |      I32      |      2^20      |  15.161 us |       3.47% |  15.143 us |       3.55% |  -0.018 us |  -0.12% |   PASS   |
+    ...
+
+In addition to showing the absolute and relative runtime difference,
+NVBench reports the noise of the measurements,
+which corresponds to the relative standard deviation.
+It then reports with statistical significance in the `Status` column
+how the runtime changed from the base to the new version.
+
+You can reduce the output to runs with larger differences using the `--threshold-diff` option,
+passing the minimum percentage for a run to be shown, e.g., `0.05` for 5%.
+
+.. code-block:: bash
+
+    PYTHONPATH=./_deps/nvbench-src/python/scripts ./_deps/nvbench-src/python/scripts/nvbench_compare.py \
+        --threshold-diff 0.05 base.json new.json
+
+You can also plot the comparison by adding the `--plot` argument.
+It's reasonable to combine this with the `-a` option again
+to restrict the values for some axes.
+
+.. code-block:: bash
+
+    PYTHONPATH=./_deps/nvbench-src/python/scripts ./_deps/nvbench-src/python/scripts/nvbench_compare.py \
+        -a Elements{io}[pow2]=28 --plot base.json new.json
+
+
+Running all benchmarks directly from the command line
+--------------------------------------------------------------------------------
+
+To get a full snapshot of CUB's performance, you can run all benchmarks and save the results.
+For example, inside a build directory you can run:
+
+.. code-block:: bash
+
+    ninja cub.all.benches
+    benchmarks=$(ls bin | grep cub.bench); n=$(echo $benchmarks | wc -w); i=1; \
+    for b in $benchmarks; do \
+      echo "=== Running $b ($i/$n) ==="; \
+      ./bin/$b -d 0 --stopping-criterion entropy --json $b.json --md $b.md; \
+      ((i++)); \
+    done
+
+This will generate one JSON and one Markdown file for each benchmark.
+You can archive those files for later comparison or analysis.
+
+
+Running all benchmarks via tuning scripts (alternative)
+--------------------------------------------------------------------------------
+
+The benchmark suite can also be run using the :ref:`tuning infrastructure <cub-tuning-infra>`.
+The tuning infrastructure handles building benchmarks itself, because it records the build times.
+Therefore, it's critical that you run it in a clean build directory without any build artifacts.
+Running cmake is enough. Alternatively, you can also clean your build directory.
+Furthermore, the tuning scripts require some additional python dependencies, which you have to install:
+
+.. code-block:: bash
+
+    ninja clean
+    pip install --user fpzip pandas scipy
+
+To select the appropriate CUDA GPU, first identify the GPU ID by running `nvidia-smi`, then set the
+desired GPU using `export CUDA_VISIBLE_DEVICES=x <https://docs.nvidia.com/cuda/cuda-c-programming-guide/#cuda-environment-variables>`_,
+where `x` is the ID of the GPU you want to use (e.g., `1`).
+This ensures your application uses only the specified GPU.
+We can then run the full benchmark suite from the build directory with:
+
+.. code-block:: bash
+
+    export CUDA_VISIBLE_DEVICES=0 # or any other GPU ID
+    PYTHONPATH=../benchmarks/scripts ../benchmarks/scripts/run.py
+
+You can expect the output to look like this:
+
+.. code-block:: bash
+
+    &&&& RUNNING bench
+    ctk:  12.2.140
+    cub:  812ba98d1
+    &&&& PERF cub_bench_adjacent_difference_subtract_left_base_T_ct__I32___OffsetT_ct__I32___Elements_io__pow2__16 4.095999884157209e-06 -sec
+    &&&& PERF cub_bench_adjacent_difference_subtract_left_base_T_ct__I32___OffsetT_ct__I32___Elements_io__pow2__20 1.2288000107218977e-05 -sec
+    &&&& PERF cub_bench_adjacent_difference_subtract_left_base_T_ct__I32___OffsetT_ct__I32___Elements_io__pow2__24 0.00016998399223666638 -sec
+    &&&& PERF cub_bench_adjacent_difference_subtract_left_base_T_ct__I32___OffsetT_ct__I32___Elements_io__pow2__28 0.002673664130270481 -sec
+    ...
+
+The tuning infrastructure will build and execute all benchmarks and their variants one after each other,
+reporting the time in seconds it took to execute the benchmarked region.
+
+It's also possible to benchmark a subset of algorithms and workloads, by running in a build directory:
+
+.. code-block:: bash
+
+    export CUDA_VISIBLE_DEVICES=0 # or any other GPU ID
+    PYTHONPATH=../benchmarks/scripts ../benchmarks/scripts/run.py -R '.*scan.exclusive.sum.*' -a 'Elements{io}[pow2]=[24,28]' -a 'T{ct}=I32'
+    &&&& RUNNING bench
+     ctk:  12.6.77
+    cccl:  v2.7.0-rc0-265-g32aa6aa5a
+    &&&& PERF cub_bench_scan_exclusive_sum_base_T_ct__I32___OffsetT_ct__U32___Elements_io__pow2__28 0.003194367978721857 -sec
+    &&&& PERF cub_bench_scan_exclusive_sum_base_T_ct__I32___OffsetT_ct__U64___Elements_io__pow2__28 0.00319383991882205 -sec
+    &&&& PASSED bench
+
+
+The `-R` option allows you to specify a regular expression for selecting benchmarks.
+The `-a` restricts the values for an axis across all benchmarks
+See the `NVBench documentation <https://github.com/NVIDIA/nvbench/blob/main/docs/cli_help_axis.md>`__.
+for more information on how to specify the axis values.
+Contrary to running a benchmark directly,
+the tuning infrastructure will just ignore an axis value if a benchmark does not support,
+run the benchmark regardless, and continue.
+
+The tuning infrastructure stores results in an SQLite database called :code:`cccl_meta_bench.db` in the build directory.
+This database persists across tuning runs.
+If you interrupt the benchmark script and then launch it again, only missing benchmark variants will be run.
+
+
+Comparing results of multiple tuning databases
+--------------------------------------------------------------------------------
+
+Benchmark results captured in different tuning databases can be compared as well:
+
+.. code-block:: bash
+
+    <cccl_git_root>/benchmarks/scripts/compare.py -o cccl_meta_bench1.db cccl_meta_bench2.db
+
+This will print a Markdown report showing the runtime differences and noise for each variant.
+
+Furthermore, you can plot the results, which requires additional python packages:
+
+.. code-block:: bash
+
+    pip install fpzip pandas matplotlib seaborn tabulate PyQt5 colorama
+
+You can plot one or more tuning databases as a bar chart or a box plot (add `--box`):
+
+.. code-block:: bash
+
+    <cccl_git_root>/benchmarks/scripts/sol.py cccl_meta_bench.db ...
+
+This is useful to display the current performance of CUB as captured in a single tuning database,
+or visually compare the performance of CUB across different tuning databases
+(from different points in time, on different GPUs, etc.).
+
+
+Dumping benchmark results from a tuning database
+--------------------------------------------------------------------------------
+
+The resulting database contains all samples, which can be extracted into JSON files:
+
+.. code-block:: bash
+
+    <cccl_git_root>/benchmarks/scripts/analyze.py -o ./cccl_meta_bench.db
+
+This will create a JSON file for each benchmark variant next to the database.
+For example:
+
+.. code-block:: bash
+
+    cat cub_bench_scan_exclusive_sum_base_T_ct__I32___OffsetT_ct__U32___Elements_io__pow2__28.json
+    [
+      {
+        "variant": "base ()",
+        "elapsed": 2.6299014091,
+        "center": 0.003194368,
+        "bw": 0.8754671386,
+        "samples": [
+          0.003152896,
+          0.0031549439,
+          ...
+        ],
+        "Elements{io}[pow2]": "28",
+        "base_samples": [
+          0.003152896,
+          0.0031549439,
+          ...
+        ],
+        "speedup": 1
+      }
+    ]
+
+
+Profiling benchmarks with Nsight Compute
+--------------------------------------------------------------------------------
+
+If you want to see profiling metrics on source code level,
+you have to recompile your benchmarks with the `-lineinfo` option.
+With cmake, you can just add `-DCMAKE_CUDA_FLAGS=-lineinfo` when invoking cmake in the `build` directory:
+
+.. code-block:: bash
+
+    cmake .. --preset=benchmark -DCMAKE_CUDA_FLAGS=-lineinfo
+
+To profile the kernels, use the `ncu` command.
+A typical invocation, if you work on a remote cluster, could look like this:
+
+.. code-block:: bash
+
+    ncu --set full --import-source yes -o base.ncu-rep -f ./bin/thrust.bench.transform.basic.base -d 0 --profile
+
+The option `--set full` instructs `ncu` to collect all metrics.
+This requires rerunning some kernels and takes more time.
+`--import-source yes` imports the source code into the report file,
+so you can see metrics not only in SASS but also in your source code,
+even if you copy the resulting report away from the source code.
+`-o base.ncu-rep` specifies the output file and `-f` overwrites the output file if it already exists.
+`--profile` tells NVBench to run only one iteration, which speeds up profiling.
+
+For inspecting the profiling report, we recommend using the GUI of Nsight Compute.
+If you run on a remote machine, you may want to copy the report `base.ncu-rep` back to your local workstation,
+before viewing the report using `ncu-ui`:
+
+.. code-block:: bash
+
+    scp <remote hostname>:<cccl repo directory>/build/base.ncu-rep .
+    ncu-ui base.ncu-rep
+
+The version of `ncu-ui` needs to be at least as high as the version of `ncu` used to create the report.
+
+Authoring benchmarks
+--------------------------------------------------------------------------------
+
+CUB's benchmarks serve a dual purpose.
+They are used to measure and compare the performance of CUB and to tune CUB's algorithms.
+More information on how to create new benchmarks is provided in the :ref:`CUB tuning infrastructure guide <cub-tuning-infra>`.
diff --git a/cccl_upstream/docs/cub/block_wide.rst b/cccl_upstream/docs/cub/block_wide.rst
new file mode 100644
index 00000000..333b56c6
--- /dev/null
+++ b/cccl_upstream/docs/cub/block_wide.rst
@@ -0,0 +1,26 @@
+.. _block-module:
+
+Block-Wide "Collective" Primitives
+==================================================
+
+.. toctree::
+   :glob:
+   :hidden:
+   :maxdepth: 2
+
+   api/block
+
+CUB block-level algorithms are specialized for execution by threads in the same CUDA thread block:
+
+* :cpp:class:`cub::BlockAdjacentDifference` computes the difference between adjacent items partitioned across a CUDA thread block
+* :cpp:class:`cub::BlockDiscontinuity` flags discontinuities within an ordered set of items partitioned across a CUDA thread block
+* :cpp:struct:`cub::BlockExchange` rearranges data partitioned across a CUDA thread block
+* :cpp:class:`cub::BlockHistogram` constructs block-wide histograms from data samples partitioned across a CUDA thread block
+* :cpp:class:`cub::BlockLoad` loads a linear segment of items from memory into a CUDA thread block
+* :cpp:class:`cub::BlockMergeSort` sorts items partitioned across a CUDA thread block
+* :cpp:class:`cub::BlockRadixSort` sorts items partitioned across a CUDA thread block using radix sorting method
+* :cpp:struct:`cub::BlockReduce` computes reduction of items partitioned across a CUDA thread block
+* :cpp:class:`cub::BlockRunLengthDecode` decodes a run-length encoded sequence partitioned across a CUDA thread block
+* :cpp:struct:`cub::BlockScan` computes a prefix scan of items partitioned across a CUDA thread block
+* :cpp:struct:`cub::BlockShuffle` shifts items partitioned across a CUDA thread block
+* :cpp:class:`cub::BlockStore` stores items partitioned across a CUDA thread block to a linear segment of memory
diff --git a/cccl_upstream/docs/cub/determinism.rst b/cccl_upstream/docs/cub/determinism.rst
new file mode 100644
index 00000000..301e1165
--- /dev/null
+++ b/cccl_upstream/docs/cub/determinism.rst
@@ -0,0 +1,76 @@
+.. _cub-determinism:
+
+Determinism
+===============
+
+Several ``cub`` device algorithms let you request a reproducibility guarantee for a call. The concepts
+behind the three guarantees — ``not_guaranteed``, ``run_to_run``, and ``gpu_to_gpu`` — and the meaning
+of *reproducibility* are described in the :ref:`CCCL determinism overview <cccl-determinism>`. This
+page documents how to request a guarantee for a CUB algorithm and which algorithms support which
+guarantees.
+
+Requesting a guarantee
+----------------------
+
+A determinism guarantee is passed to a device algorithm through its execution environment using
+``cuda::execution::require``. The example below requests run-to-run reproducibility for
+``cub::DeviceReduce::Sum``:
+
+.. literalinclude:: ../../cub/test/catch2_test_device_reduce_env_api.cu
+   :language: c++
+   :dedent:
+   :start-after: example-begin sum-env-determinism
+   :end-before: example-end sum-env-determinism
+
+The general rules for requesting a guarantee are described in the
+:ref:`CCCL determinism overview <cccl-determinism>`.
+
+Each CUB algorithm has its own default guarantee, applied when none is requested, and its own type and
+operator constraints for each guarantee, summarized below. Requesting a guarantee that an algorithm
+does not support is rejected at compile time.
+
+Support matrix
+--------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 30 18 18 18 16
+
+   * - Algorithm
+     - ``not_guaranteed``
+     - ``run_to_run``
+     - ``gpu_to_gpu``
+     - Default
+   * - ``cub::DeviceReduce`` (``Reduce``, ``Sum``, ``Min``, ``Max``, ``TransformReduce``, ...)
+     - Yes
+     - Yes
+     - Yes (partial)
+     - ``run_to_run``
+   * - ``cub::DeviceScan`` (``ExclusiveSum``, ``ExclusiveScan``, ``InclusiveSum``, ``InclusiveScan``, ...)
+     - Yes
+     - Yes (partial)
+     - Yes (partial)
+     - ``not_guaranteed``
+   * - ``cub::DeviceSegmentedReduce``
+     - Yes
+     - Yes
+     - No
+     - ``run_to_run``
+
+.. note::
+
+   The set of algorithms that accept determinism requirements, and the type/operator constraints for
+   each guarantee, are expanding over time. The matrix above reflects the current implementation.
+
+Algorithm-specific determinism models
+--------------------------------------
+
+The three guarantees describe the *scope* of reproducibility and fit most algorithms, where a
+reproducible result means a *bitwise-identical* output. A few algorithms still use the same three
+levels but extend the model with additional, algorithm-specific controls, documented on their own
+pages:
+
+- :ref:`cub::DeviceTopK <cub-topk-requirements>` — determinism applies to *set membership* (which
+  *K* items are selected) rather than a bitwise-identical buffer, and it adds tie-breaking
+  (``cuda::execution::tie_break``) and output-ordering (``cuda::execution::output_ordering``)
+  controls.
diff --git a/cccl_upstream/docs/cub/developer/block_scope.rst b/cccl_upstream/docs/cub/developer/block_scope.rst
new file mode 100644
index 00000000..da2fdaa6
--- /dev/null
+++ b/cccl_upstream/docs/cub/developer/block_scope.rst
@@ -0,0 +1,79 @@
+.. _cub-developer-guide-block-scope:
+
+Block-scope
+************
+
+Overview
+=========
+
+Block-scope algorithms are provided by structures as well:
+
+.. code-block:: c++
+
+    template <typename T,
+              int BLOCK_DIM_X,
+              BlockReduceAlgorithm ALGORITHM = BLOCK_REDUCE_WARP_REDUCTIONS,
+              int BLOCK_DIM_Y = 1,
+              int BLOCK_DIM_Z = 1>
+    class BlockReduce {
+    public:
+      struct TempStorage : Uninitialized<_TempStorage> {};
+
+      // (1) new constructor
+      __device__ __forceinline__ BlockReduce()
+          : temp_storage(PrivateStorage()),
+            linear_tid(RowMajorTid(BLOCK_DIM_X, BLOCK_DIM_Y, BLOCK_DIM_Z)) {}
+
+      __device__ __forceinline__ BlockReduce(TempStorage &temp_storage)
+          : temp_storage(temp_storage.Alias()),
+            linear_tid(RowMajorTid(BLOCK_DIM_X, BLOCK_DIM_Y, BLOCK_DIM_Z)) {}
+    };
+
+While warp-scope algorithms only provide a single constructor that requires the user to provide temporary storage,
+block-scope algorithms provide two constructors:
+
+    #. The default constructor that allocates the required shared memory internally.
+    #. The constructor that requires the user to provide temporary storage as argument.
+
+In the case of the default constructor,
+the block-level algorithm uses the ``PrivateStorage()`` member function to allocate the required shared memory.
+This ensures that shared memory required by the algorithm is only allocated when the default constructor is actually called in user code.
+If the default constructor is never called,
+then the algorithm will not allocate superfluous shared memory.
+
+.. code-block:: c++
+
+    __device__ __forceinline__ _TempStorage& PrivateStorage()
+    {
+      __shared__ _TempStorage private_storage;
+      return private_storage;
+    }
+
+The ``__shared__`` memory has static semantic, so it's safe to return a reference here.
+
+Specialization
+====================================
+
+Block-scope facilities usually expose algorithm selection to the user.
+The algorithm is represented by the enumeration part of the API.
+For the reduction case,
+``BlockReduceAlgorithm`` is provided.
+Specializations are stored in the ``cub/block/specializations`` directory.
+
+Temporary storage usage
+====================================
+
+For block-scope algorithms,
+it's unsafe to use temporary storage without synchronization:
+
+.. code-block:: c++
+
+    using BlockReduce = cub::BlockReduce<int, 128> ;
+
+    __shared__ BlockReduce::TempStorage temp_storage;
+
+    int aggregate_1 = BlockReduce(temp_storage).Sum(thread_data_1);
+    // illegal, has to add `__syncthreads` between the two
+    int aggregate_2 = BlockReduce(temp_storage).Sum(thread_data_2);
+    // illegal, has to add `__syncthreads` between the two
+    foo(temp_storage);
diff --git a/cccl_upstream/docs/cub/developer/device_scope.rst b/cccl_upstream/docs/cub/developer/device_scope.rst
new file mode 100644
index 00000000..a0d50136
--- /dev/null
+++ b/cccl_upstream/docs/cub/developer/device_scope.rst
@@ -0,0 +1,530 @@
+.. _cub-developer-guide-device-scope:
+
+Device-scope
+*************
+
+Overview
+=========
+
+Device-scope functionality is provided by classes called ``DeviceAlgorithm``,
+where ``Algorithm`` is the implemented algorithm.
+These classes then contain static member functions providing corresponding API entry points.
+For example, device-level reduce will look like `cub::DeviceReduce::Sum`.
+Here is a generic example:
+
+.. code-block:: c++
+
+    struct DeviceAlgorithm {
+      // two step API
+      template <typename ...>
+      static cudaError_t Algorithm(void *d_temp_storage, size_t &temp_storage_bytes, ..., cudaStream_t stream = 0) {
+        // optional: minimal argument checking or setup to call dispatch layer
+        return detail::algorithm::dispatch(d_temp_storage, temp_storage_bytes, ..., stream);
+      }
+
+      // environment API
+      template <typename ..., typename Env = cuda::std::execution::env<>>
+      static cudaError_t Algorithm(..., const Env& env = {}) {
+        // optional: minimal argument checking or setup to call dispatch layer
+        using default_policy_selector = detail::algorithm::policy_selector_from_types<...>;
+        return dispatch_with_env_and_tuning<default_policy_selector>(
+          env, [&](auto policy_selector, void* storage, size_t& bytes, auto stream) {
+            return detail::algorithm::dispatch(d_temp_storage, temp_storage_bytes, ..., stream, policy_selector);
+          });
+      }
+    };
+
+Device-scope APIs come in two flavors, two step APIs and environment APIs,
+both return ``cudaError_t`` and take algorithm specific arguments.
+The two step API accepts a ``stream`` as the last parameter (``NULL`` stream by default)
+and the first two parameters are always ``void *d_temp_storage, size_t &temp_storage_bytes``.
+The environment API just takes an environment as the last parameter (empty environment by default).
+The implementation may consist of some minimal argument checking, but should forward as soon as possible to the dispatch layer.
+Device-scope algorithms are implemented in files located in `cub/device/device_***.cuh`.
+
+The two step API is called in two phases:
+
+  1. Temporary storage size is calculated and returned in ``size_t &temp_storage_bytes``.
+  2. ``temp_storage_bytes`` of memory is expected to be allocated and ``d_temp_storage`` is expected to be the pointer to this memory.
+
+The following example illustrates this pattern:
+
+.. code-block:: c++
+
+    // First call: Determine temporary device storage requirements
+    std::size_t temp_storage_bytes = 0;
+    cub::DeviceReduce::Sum(/*d_temp_storage*/ nullptr, temp_storage_bytes, d_in, d_out, num_items);
+
+    // Allocate temporary storage
+    thrust::device_vector<unsigned char> temp_storage(temp_storage_bytes, thrust::no_init);
+
+    // Second call: Perform algorithm
+    cub::DeviceReduce::Sum(temp_storage.data().get(), temp_storage_bytes, d_in, d_out, num_items);
+
+.. warning::
+    Even if the algorithm doesn't need temporary storage as scratch space,
+    the overload with ``void *d_temp_storage, size_t &temp_storage_bytes``
+    still requires one byte of memory to be allocated.
+
+The environment overloads just require a single call, but setting up the environment may be more complex:
+
+.. code-block:: c++
+
+    // Setup environment, everything is optional
+    auto env = cuda::std::execution::env{
+      stream_ref,
+      memory_resource,
+      cuda::execution::require(requirements...),
+      cuda::execution::tune(policy_selectors....)
+    };
+
+    // Perform algorithm
+    cub::DeviceReduce::Sum(d_in, d_out, num_items, env);
+
+The device layer handles the extraction of the:
+
+* CUDA stream
+* memory resource
+* requirements
+* guarantees (TODO(bgruber): are those public?)
+* :ref:`tuning policy selectors <cub-policy-selectors>`
+
+from the environment argument, or provide default values in case the environment does not contain them.
+They typically use helper functions like ``dispatch_with_env`` and ``dispatch_with_env_and_tuning``.
+
+Some CUB APIs require no temporary storage and may omit the ``void *d_temp_storage, size_t &temp_storage_bytes`` parameters.
+Their environment overloads will also ignore any passed memory resource.
+
+Dispatch layer
+====================================
+
+A dispatch function exists for each device-scope algorithm (e.g., ``detail::reduce::dispatch``),
+and is located in ``cub/device/dispatch``.
+Most device-scope algorithms share one dispatch function.
+Only device-scope algorithms have dispatch functions, which are also referred to as the dispatch layer.
+
+The dispatch layer follows a certain architecture.
+The high-level control flow is represented by the code below.
+A more precise description is given later.
+
+..
+    TODO(bgruber): consider removing the numbers below. control flow is now easier to follow
+
+.. code-block:: c++
+
+    // Device-scope API
+    cudaError_t cub::DeviceAlgorithm::Algorithm(d_temp_storage, temp_storage_bytes, ...) {
+      return detail::algorithm::dispatch(d_temp_storage, temp_storage_bytes, ...);
+    }
+
+    namespace detail::algorithm {
+      cudaError_t dispatch(
+          void *d_temp_storage, size_t &temp_storage_bytes,
+          ...,
+          cudaStream_t stream, PolicySelector policy_selector = {}) {
+        cuda::compute_capability cc{};
+        ptx_compute_cap(cc);
+        const /*or constexpr*/ AlgorithmPolicy active_policy = policy_selector(cc);
+        // host-side implementation of algorithm, calls kernels
+        kernel<PolicySelector><<<grid_size, active_policy.threads_per_block>>>(...);
+      }
+
+      template <typename PolicySelector>
+      __launch_bounds__(int(current_policy<PolicySelector>().threads_per_block))
+      void kernel(...) {
+        static constexpr auto policy = current_policy<PolicySelector>();
+        using agent_policy = AgentPolicy<policy.threads_per_block, policy.items_per_thread, ...>;
+        using agent = AgentAlgorithm<agent_policy, InputIteratorT, OffsetT, ...>;
+        agent a{...};
+        a.Process();
+      }
+
+      template <int ThreadsPerBlock, ...>
+      struct AgentPolicy { // legacy
+        static constexpr threads_per_block = ThreadsPerBlock;
+        ...
+      };
+
+      template <typename Policy, ...>
+      struct AlgorithmAgent {
+        void Process() { ... }
+      };
+    }
+
+Let's look at each of the building blocks closer.
+
+The dispatch function
+------------------------------------
+
+The dispatch function is typically a simple function template called ``dispatch`` inside an algorithm-specific namespace.
+It basically receives the same parameters as the public API entry point,
+but they may have already been modified, extended, or generalized (to map many public APIs to the same dispatch function).
+The goal of the dispatch function is to setup the execution of the algorithm on the device
+by selecting the appropriate policy for the current GPU's compute capability,
+preparing temporary storage and shared memory, configuring kernel launches, launching kernels, etc.
+
+There are two style of dispatch functions, depending on whether the policy is needed as runtime or compile-time value:
+
+.. code-block:: c++
+
+    namespace detail::algorithm {
+      // Dispatch version A - runtime policy
+      cudaError_t dispatch(
+          void *d_temp_storage, size_t &temp_storage_bytes,
+          ...,
+          cudaStream_t stream, PolicySelector policy_selector = {}) {
+        cuda::compute_capability cc{};
+        ptx_compute_cap(cc);
+        const auto active_policy = policy_selector(cc); // runtime-time policy
+        // host-side implementation of algorithm, calls kernels
+        kernel<PolicySelector><<<grid_size, active_policy.threads_per_block>>>(...);
+      }
+    }
+
+The dispatch function starts by querying the target compute capability for which compiled GPU code (PTX or SASS) is available,
+by calling ``ptx_compute_cap``.
+If the host code does not require the policy for this compute capability at compile-time (version A),
+we can just pass the compute capability to the :ref:`policy selector <cub-policy-selectors>` to obtain the tuning policy at runtime.
+The values from the policy are then used to setup resources and the kernel.
+The kernel is then instantiated using only the type of the policy selector (not a concrete tuning policy),
+so there is only one kernel instantiation across all target architectures compiled for.
+More on that later.
+
+If the host code needs the policy as a compile-time value (version B), we have to use ``dispatch_compute_cap``.
+
+dispatch_compute_cap
+------------------------------------
+
+``dispatch_compute_cap`` maps a runtime ``cuda::compute_capability`` to a compile-time policy value
+and calls the user-provided functor ``f`` with a nullary callable (a ``policy_getter``)
+that returns the policy as a compile-time constant.
+The policy getter is necessary to work around a C++17 limitation.
+
+.. code-block:: c++
+
+    namespace detail::algorithm {
+      // Dispatch version B - compile-time policy
+      cudaError_t dispatch(
+          void *d_temp_storage, size_t &temp_storage_bytes,
+          ...,
+          cudaStream_t stream, PolicySelector policy_selector = {}) {
+        cuda::compute_capability cc{};
+        ptx_compute_cap(cc);
+        return dispatch_compute_cap(policy_selector, cc, [&](auto policy_getter) {
+          constexpr auto active_policy = policy_getter(); // compile-time policy
+          static_assert(active_policy.tile_size() * sizeof(T) <= 48 * 1024, "Not enough SMEM");
+          // host-side implementation of algorithm, calls kernels
+          kernel<PolicySelector><<<grid_size, active_policy.threads_per_block>>>(...);
+        });
+      }
+
+      template <typename PolicySelector, typename F>
+      cudaError_t dispatch_compute_cap(PolicySelector, cuda::compute_capability cc, F&& f) { // (2)
+        // fold over __CUDA_ARCH_LIST__, calling f with a policy_getter
+        // that returns the policy for the matching arch as a compile-time value
+      }
+    }
+
+Inside the lambda, ``policy_getter()`` returns the selected policy as a constant expression,
+so compile-time branching (i.e. ``if constexpr``) or static assertions using policy values is possible.
+
+Internally, ``dispatch_compute_cap`` uses ``__CUDA_ARCH_LIST__`` / ``NV_TARGET_SM_INTEGER_LIST`` (or all known compute capabilities as fallback)
+to create one instantiation of ``f`` per distinct value of ``policy_selector(cc)`` (not per compute capability).
+This results in one template instantiation of ``f`` per distinct policy value.
+The kernel is then again only instantiated once using the type of the policy selector.
+
+
+Kernels
+------------------------------------
+
+Kernels are templated on the ``PolicySelector`` type,
+which is stateless and the same for all target architectures compiled for.
+
+.. code-block:: c++
+
+    namespace detail::algorithm {
+      template <typename PolicySelector, typename InputIteratorT, typename OffsetT, /* ... */>
+      __launch_bounds__(int(current_policy<PolicySelector>().reduce.threads_per_block))
+      void DeviceReduceKernel(InputIteratorT d_in, OffsetT num_items, /* ... */) {
+        static constexpr auto policy = current_policy<PolicySelector>();
+        using agent_policy = AgentPolicy<policy.threads_per_block, policy.items_per_thread, ...>;
+        using agent = AgentAlgorithm<agent_policy, InputIteratorT, OffsetT, ...>;
+        agent a{...};
+        a.Process();
+      }
+
+      template <typename PolicySelector>
+      constexpr auto current_policy() {
+        return PolicySelector{}(cuda::compute_capability{__CUDA_ARCH__ / 10}); // simplified
+      }
+    }
+
+``PolicySelector`` must be stateless (``is_empty_v<PolicySelector>`` is ``true``),
+so it can be default-constructed in device code where needed.
+The utility function ``current_policy`` can only be called in device code.
+It selects the target compute capability based on compiler macros of the current device compilation pass
+and retrieves a tuning policy from the policy selector.
+
+The kernel typically uses ``current_policy`` in two places,
+to get the block size to define the launch bounds,
+and inside the kernel to setup various sub algorithms (like agents).
+
+Because C++17 does not allow to pass structs as Non-Type Template Parameters (NTTPs),
+we cannot easily pass the policy around to other functions,
+so it will be converted to legacy agent policies in many places.
+Those are just structs with static data members holding the policy value as a type,
+so they can be passed to templates.
+Agent policy structs have historically been part of the public API, and should be removed in the future.
+
+.. warning::
+    The kernel gets compiled for each target architecture (N many) that was provided to the compiler.
+    During each device pass, ``current_policy`` may return a different policy.
+    During the host pass, version A (runtime policy) compiles a single instantiation of the dispatch logic for all target architectures.
+    Version B (using ``dispatch_compute_cap``) compiles the dispatch logic for each distinct tuning policy (M many).
+    If we passed the selected tuning policy instead of the policy selector as a kernel template parameter,
+    the kernel template instantiation would be different for each tuning policy value and
+    we would compile O(M*N) kernels for version B instead of O(N).
+
+Many kernels are short, since the functionality is extracted into the agent layer.
+All the kernel does is derive the proper policy,
+unwrap it to initialize the agent and call one of its ``Consume`` / ``Process`` functions.
+Agents hold kernel bodies and are intended to be reused across multiple device-scope algorithms.
+However, this is not a requirement and some kernels just contain their entire implementation themselves.
+
+
+The default policy selector
+------------------------------------
+
+CUB contains a default policy selector for each dispatch function.
+Because many dispatch functions are also used by CCCL.C,
+which compiles them without proper type information,
+we have to provide them in two forms,
+a typeless and a typeful version.
+
+.. code-block:: c++
+
+    struct AlgorithmPolicy { ... }; // unrelated to AgentPolicy
+
+    namespace detail::algorithm {
+      struct policy_selector {
+        type_t accum_t;
+        op_kind_t operation_t;
+        int offset_size;
+        int accum_size;
+
+        constexpr auto operator()(::cuda::compute_capability cc) const -> AlgorithmPolicy {
+          // parameter selection across target compute capabilities and input characteristics (possibly HUGE logic)
+        }
+      };
+
+      template <typename AccumT, typename OffsetT, typename ReductionOpT>
+      struct policy_selector_from_types {
+        constexpr auto operator()(cuda::compute_capability cc) const -> AlgorithmPolicy {
+          constexpr auto ps = policy_selector{
+            classify_type<AccumT>(),
+            classify_op<ReductionOpT>(),
+            int{sizeof(OffsetT)},
+            int{sizeof(AccumT)}
+          };
+          return ps(cc);
+        }
+      };
+    }
+
+The ``policy_selector`` is intended to be used without template-parameter-based type information
+and is thus suitable to be used in CCCL.C without a JIT compiler for host code.
+It contains the necessary information on the algorithm's input as data members.
+This policy is only used in the host code of the dispatch function.
+Because it is not stateless anymore, CCCL.C overrides the kernel launcher used by CUB,
+providing the kernel from a JIT-compiled instantiation that uses a proper stateless policy selector.
+
+The kernel, and the dispatch function when not called from CCCL.C,
+will use the second version of the policy selector, ``policy_selector_from_types``,
+which offers proper template parameters to pass type information.
+This type is stateless and delegates to a ``constexpr`` instance of the ``policy_selector``
+with compile-time values derived from the template parameters.
+The policy selection logic is thus the same for CCCL.C and CUB,
+the type information is just provided differently.
+
+Each dispatch function also has an associated concept for the policy selector it expects:
+
+.. code-block:: c++
+
+    template <typename T, typename Policy>
+    concept policy_selector = requires(T pol_sel, cuda::compute_capability cc) {
+      requires std::regular<Policy>;
+      { pol_sel(cc) } -> std::same_as<Policy>;
+    };
+
+    namespace detail::algorithm {
+      template <typename T>
+      concept algorithm_policy_selector = policy_selector<T, AlgorithmPolicy>;
+    }
+
+The concept basically checks whether the policy selector can be called with a ``cuda::compute_capability``
+and returns the expected policy struct.
+The default policy selectors and related concept are defined in ``cub/device/dispatch/tuning/tuning_<algorithm>.cuh``.
+
+
+Backward compatibility
+------------------------------------
+
+Legacy public dispatchers (e.g. ``DispatchReduce``) are deprecated.
+They continue to work by translating the ``PolicyHub`` template parameter to the new ``policy_selector``
+via a ``policy_selector_from_hub`` adapter:
+
+.. code-block:: c++
+
+    template <typename PolicyHub>
+    struct policy_selector_from_hub {
+      constexpr auto operator()(cuda::compute_capability cc) const -> AlgorithmPolicy {
+        // conversion logic
+      }
+    };
+
+This allows existing user code that passes custom policy hubs to dispatchers to continue working.
+The legacy dispatchers, policy hubs and related logic are scheduled for removal in CCCL 4.0.
+
+
+Policies
+====================================
+
+Policies describe the configuration of agents or just kernels with respect to performance.
+They must not change functional behavior, but affect how work is mapped to the hardware
+by defining certain parameters (items per thread, block size, etc.),
+or choosing between algorithms.
+
+Policies must be plain semiregular aggregates to allow using them during constant evaluation,
+and with designated initializers:
+
+.. code-block:: c++
+
+    struct AlgorithmPolicy {
+      int threads_per_block;
+      int items_per_thread;
+      cub::BlockLoadAlgorithm load_algorithm;
+      cub::CacheLoadModifier load_modifier;
+
+      friend constexpr bool operator==(const AlgorithmPolicy& lhs, const AlgorithmPolicy& rhs) { ... }
+      friend constexpr bool operator!=(const AlgorithmPolicy& lhs, const AlgorithmPolicy& rhs) { ... }
+      friend std::ostream& operator<<(std::ostream& os, const AlgorithmPolicy& p) { ... }
+    };
+
+Tuning policies can have various complexities and contain nested structures.
+Policies are defined in ``cub/device/dispatch/tuning/tuning_<algorithm>.cuh``.
+
+
+Tunings
+====================================
+
+Because the values to parameterize an agent may vary a lot for different compile-time parameters,
+the selection of values can involve complex logic.
+Often, such tunings are found by experimentation or heuristic search.
+See also :ref:`cub-tuning-infra`.
+
+Tunings are expressed as logic and values inside the ``constexpr operator()`` of a policy selector.
+Because of the complexity of some policy selectors, nested functions may be used.
+Many policy selectors also implement a fallback logic,
+where they try to find a matching tuning based on the input characteristics (policy selector data members),
+but if no match is found, they fall back to an older target compute capability.
+Here is an example:
+
+.. code-block:: c++
+
+    struct sm90_tuning_values {
+      int items;
+      int threads;
+      int items_per_vec_load;
+    };
+
+    constexpr auto get_sm90_tuning(type_t accum_t, op_kind_t op, int offset_size, int accum_size)
+        -> std::optional<sm90_tuning_values> {
+      // provide tunings for certain operations, accumulator or offset types
+      if (op == op_kind_t::plus) {
+        if (accum_t == type_t::float32 && offset_size == 4 && accum_size == 4)
+          return sm90_tuning_values{16, 512, 2};
+        if (offset_size == 4 && accum_size == 8)
+          return sm90_tuning_values{15, 512, 2};
+      }
+      return {}; // no tuning available, causes fallback
+    }
+
+    struct policy_selector {
+      type_t accum_t;
+      op_kind_t operation_t;
+      int offset_size;
+      int accum_size;
+
+      constexpr auto operator()(cuda::compute_capability cc) const -> reduce_policy {
+        if (cc >= cuda::compute_capability{9, 0}) {
+          if (auto tuning = get_sm90_tuning(accum_t, operation_t, offset_size, accum_size)) {
+            return *tuning; // found a tuning, use it
+          }
+          // fall through to sm_80 if no matching tuning found
+        }
+        if (cc >= cuda::compute_capability{8, 0}) {
+          return { /* sm_80 default policy */ };
+        }
+        return { /* default policy for everything else */ };
+      }
+    };
+
+In general, tunings are not exhaustive and usually only apply for specific combinations
+of parameter values and a single compute capability.
+This is because they originate from tuning benchmarks running for specific workloads on specific target architectures.
+Generic fallbacks are often just retained from earlier days of CUB to not risk regressions,
+or are based on heuristics trying to provide reasonable performance based on a model of the GPU architecture or algorithm.
+
+Tunings for CUB algorithms reside in ``cub/device/dispatch/tuning/tuning_<algorithm>.cuh``.
+
+
+Temporary storage usage
+====================================
+
+It's safe to reuse storage in the stream order:
+
+.. code-block:: c++
+
+    cub::DeviceReduce::Sum(nullptr, storage_bytes, d_in, d_out, num_items, stream_1);
+    // allocate temp storage
+    cub::DeviceReduce::Sum(d_storage, storage_bytes, d_in, d_out, num_items, stream_1);
+    // fine not to synchronize stream
+    cub::DeviceReduce::Sum(d_storage, storage_bytes, d_in, d_out, num_items, stream_1);
+    // illegal, should call cudaStreamSynchronize(stream)
+    cub::DeviceReduce::Sum(d_storage, storage_bytes, d_in, d_out, num_items, stream_2);
+
+Temporary storage management
+====================================
+
+Often times temporary storage for device-scope algorithms has a complex structure.
+To simplify temporary storage management and make it safer,
+we introduced ``cub::detail::temporary_storage::layout``:
+
+.. code-block:: c++
+
+    cub::detail::temporary_storage::layout<2> storage_layout;
+
+    auto slot_1 = storage_layout.get_slot(0);
+    auto slot_2 = storage_layout.get_slot(1);
+
+    auto allocation_1 = slot_1->create_alias<int>();
+    auto allocation_2 = slot_1->create_alias<double>(42);
+    auto allocation_3 = slot_2->create_alias<char>(12);
+
+    if (condition)
+    {
+      allocation_1.grow(num_items);
+    }
+
+    if (d_temp_storage == nullptr)
+    {
+      temp_storage_bytes = storage_layout.get_size();
+      return;
+    }
+
+    storage_layout.map_to_buffer(d_temp_storage, temp_storage_bytes);
+
+    // different slots, safe to use simultaneously
+    use(allocation_1.get(), allocation_3.get(), stream);
+    // `allocation_2` alias `allocation_1`, safe to use in stream order
+    use(allocation_2.get(), stream);
diff --git a/cccl_upstream/docs/cub/developer/nvtx.rst b/cccl_upstream/docs/cub/developer/nvtx.rst
new file mode 100644
index 00000000..2bf618e6
--- /dev/null
+++ b/cccl_upstream/docs/cub/developer/nvtx.rst
@@ -0,0 +1,17 @@
+.. _cub-developer-guide-nvtx:
+
+NVTX
+=====
+
+The `NVIDIA Tools Extension SDK (NVTX) <https://nvidia.github.io/NVTX/>`_ is a cross-platform API
+for annotating source code to provide contextual information to developer tools.
+All device-scope algorithms in CUB are annotated with NVTX ranges,
+allowing their start and stop to be visualized in profilers
+like `NVIDIA Nsight Systems <https://developer.nvidia.com/nsight-systems>`_.
+Only the public APIs available in the ``<cub/device/device_xxx.cuh>`` headers are annotated,
+excluding direct calls to the dispatch layer.
+NVTX annotations can be disabled by defining ``NVTX_DISABLE`` during compilation.
+When CUB device algorithms are called on a stream subject to
+`graph capture <https://developer.nvidia.com/blog/cuda-graphs/>`_,
+the NVTX range is reported for the duration of capture (where no execution happens),
+and not when a captured graph is executed later (the actual execution).
diff --git a/cccl_upstream/docs/cub/developer/test_overview.rst b/cccl_upstream/docs/cub/developer/test_overview.rst
new file mode 100644
index 00000000..93992798
--- /dev/null
+++ b/cccl_upstream/docs/cub/developer/test_overview.rst
@@ -0,0 +1,328 @@
+CUB Tests
+###########################
+
+.. warning::
+    CUB is in the progress of migrating to [Catch2](https://github.com/catchorg/Catch2) framework.
+
+CUB tests rely on `CPM <https://github.com/cpm-cmake/CPM.cmake>`_ to fetch
+`Catch2 <https://github.com/catchorg/Catch2>`_ that's used as our main testing framework.
+
+Currently,
+legacy tests coexist with Catch2 ones.
+This guide is focused on new tests.
+
+.. important::
+    Instead of including ``<catch2/catch.hpp>`` directly, use ``catch2_test_helper.h``.
+
+.. code-block:: c++
+
+    #include <cub/block/block_scan.cuh>
+    #include <c2h/vector.h>
+    #include <c2h/catch2_test_helper.h>
+
+Directory and File Naming
+*************************************
+
+Our tests can be found in the ``test`` directory.
+Legacy tests have the following naming scheme: ``test_SCOPE_FACILITY.cu``.
+For instance, here are the reduce tests:
+
+.. code-block:: c++
+
+    test/test_warp_reduce.cu
+    test/test_block_reduce.cu
+    test/test_device_reduce.cu
+
+Catch2-based tests have a different naming scheme: ``catch2_test_SCOPE_FACILITY.cu``.
+
+The prefix is essential since that's how CMake finds tests
+and distinguishes new tests from legacy ones.
+
+Test Structure
+*************************************
+
+Base case
+=====================================
+Let's start with a simple example.
+Say there's no need to cover many types with your test.
+
+.. code-block:: c++
+
+    // 0) Define test name and tags
+    C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]")
+    {
+      using type = std::int32_t;
+      constexpr int threads_per_block = 256;
+      constexpr int num_items = threads_per_block;
+
+      // 1) Allocate device input
+      c2h::device_vector<type> d_input(num_items);
+
+      // 2) Generate 3 random input arrays using Catch2 helper
+      c2h::gen(C2H_SEED(3), d_input);
+
+      // 3) Allocate output array
+      c2h::device_vector<type> d_output(d_input.size());
+
+      // 4) Copy device input to host
+      c2h::host_vector<key_t> h_reference = d_input;
+
+      // 5) Compute reference output
+      std::ALGORITHM(
+          thrust::raw_pointer_cast(h_reference.data()),
+          thrust::raw_pointer_cast(h_reference.data()) + h_reference.size());
+
+      // 6) Compute CUB output
+      SCOPE_ALGORITHM<threads_per_block>(d_input.data(),
+                                         d_output.data(),
+                                         d_input.size());
+
+      // 7) Compare device and host results
+      REQUIRE( d_input == d_output );
+    }
+
+We introduce test cases with the ``C2H_TEST`` macro in (0).
+This macro always takes two string arguments - a free-form test name and
+one or more tags. Then, in (1), we allocate device memory using ``c2h::device_vector``.
+``c2h::device_vector`` and ``c2h::host_vector`` behave similarly to their Thrust counterparts,
+but are modified to provide more stable behavior in some testing edge cases.
+
+.. important::
+    Always use ``c2h::host_vector<T>``/``c2h::device_vector<T>``
+    instead of ``thrust::host_vector<T>``/``thrust::device_vector<T>``,
+    unless the test code is being used for documentation examples.
+
+Similarly, any thrust algorithms that executed on the device must be invoked with the
+`c2h::device_policy` execution policy (not shown here) to support the same edge cases.
+The memory is filled with random data in (2).
+
+Generator ``c2h::gen`` takes at least two parameters.
+The first one is a random generator seed.
+Instead of providing a single value, we use the ``C2H_SEED`` macro.
+The macro expects a number of seeds that has to be generated.
+In the example above, we require three random seeds to be generated.
+This leads to the whole test being executed three times
+with different seed values.
+
+Later, in (3), we allocate device output and host reference.
+In (4), we allocate and populate the host input data.
+Then, we perform the reference computation on the host in (5).
+
+.. important::
+    Standard library algorithms (``std::``) have to be used where possible when computing reference solutions.
+
+Afterwards, we launch the corresponding CUB algorithm in (6).
+At this point, we have a reference solution on the CPU and a CUB solution on the GPU.
+The two can be compared using Catch2's ``REQUIRE`` macro, which stops execution upon failure (preferred).
+Catch2 also offers the ``CHECK`` macro, which continues test execution if the check fails.
+
+If your test has to cover floating point types,
+it's sufficient to replace ``REQUIRE( a == b )`` with ``REQUIRE_APPROX_EQ(a, b)``.
+
+.. important::
+    Using ``c2h::gen`` for producing input data is strongly advised.
+
+Do not use ``assert`` in tests, which is usually only enabled in Debug mode,
+and we run CUB tests in Release mode.
+
+If a custom (non-fundamental) type has to be tested, the following helper class template should be used:
+
+.. code-block:: c++
+
+    using type = c2h::custom_type_t<c2h::accumulateable_t,
+                                    c2h::equal_comparable_t>;
+
+Here we enumerate all the type properties that we are interested in.
+The produced type ends up having ``operator==`` (from ``equal_comparable_t``)
+and ``operator+`` (from ``accumulateable_t``).
+More properties are available.
+If a property is missing, please add it to the existing set in ``c2h``
+instead of writing a custom type from scratch.
+
+Generators
+=====================================
+
+We often need to test CUB algorithms against different inputs or problem sizes.
+If these are **runtime values**, we can use the Catch2 ``GENERATE`` macro:
+
+.. code-block:: c++
+
+    C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]")
+    {
+      int num_items = GENERATE(1, 100, 1'000'000); // 0) Init. a variable with a generator
+      // ...
+    }
+
+This will lead to the test being executed three times, once for each argument to ``GENERATE(...)``.
+Multiple generators in a test inside the same scope will form the cartesian product of all combinations.
+Please consult the `Catch2 documentation <https://github.com/catchorg/Catch2/blob/devel/docs/generators.md>`_
+for more details.
+
+``C2H_SEED(3)`` uses a generator expression internally.
+
+
+Type Lists
+=====================================
+
+Since CUB is a generic library,
+it's often required to test CUB algorithms against many types.
+To do so,
+it's sufficient to define a type list and provide it to the ``C2H_TEST`` macro.
+This is useful for **compile-time** parameterization of tests.
+
+.. code-block:: c++
+
+    // 0) Define type list
+    using types = c2h::type_list<std::uint8_t, std::int32_t>;
+
+    C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]",
+            types) // 1) Provide it to the test case
+    {
+      // 2) Access current type with `c2h::get`
+      using type = typename c2h::get<0, TestType>;
+      // ...
+    }
+
+This will lead to the test being compiled (instantiated) and run twice.
+The first run will cause ``type`` to be ``std::uint8_t``.
+The second one will cause ``type`` to be ``std::uint32_t``.
+
+.. warning::
+    It's important to use types from the ``<cstdint>`` header
+    instead of built-in types like ``char`` and ``int``.
+
+Multidimensional Configuration Spaces
+=====================================
+
+In most cases, the input data type is not the only compile-time parameter we want to vary.
+For instance, you might need to test a block algorithm for different data types
+**and** different thread block sizes.
+To do so, you can add another type list as follows:
+
+.. code-block:: c++
+
+    using block_sizes = c2h::enum_type_list<int, 128, 256>;
+    using types = c2h::type_list<std::uint8_t, std::int32_t>;
+
+    C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]",
+             types, block_sizes)
+    {
+      using type = typename c2h::get<0, TestType>;
+      constexpr int threads_per_block = c2h::get<1, TestType>::value;
+      // ...
+    }
+
+The code above leads to the following combinations being compiled:
+
+- ``type = std::uint8_t``, ``threads_per_block = 128``
+- ``type = std::uint8_t``, ``threads_per_block = 256``
+- ``type = std::int32_t``, ``threads_per_block = 128``
+- ``type = std::int32_t``, ``threads_per_block = 256``
+
+As an example, the following test case includes both multidimensional configuration spaces
+and multiple random sequence generations.
+
+.. code-block:: c++
+
+    using block_sizes = c2h::enum_type_list<int, 128, 256>;
+    using types = c2h::type_list<std::uint8_t, std::int32_t>;
+
+    C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]",
+             types, block_sizes)
+    {
+      using type = typename c2h::get<0, TestType>;
+      constexpr int threads_per_block = c2h::get<1, TestType>::value;
+      // ...
+      c2h::device_vector<type> d_input(5);
+      c2h::gen(C2H_SEED(2), d_input);
+    }
+
+The code above leads to the following combinations being compiled:
+
+- ``type = std::uint8_t``, ``threads_per_block = 128``, 1st random generated input sequence
+- ``type = std::uint8_t``, ``threads_per_block = 256``, 1st random generated input sequence
+- ``type = std::int32_t``, ``threads_per_block = 128``, 1st random generated input sequence
+- ``type = std::int32_t``, ``threads_per_block = 256``, 1st random generated input sequence
+- ``type = std::uint8_t``, ``threads_per_block = 128``, 2nd random generated input sequence
+- ``type = std::uint8_t``, ``threads_per_block = 256``, 2nd random generated input sequence
+- ``type = std::int32_t``, ``threads_per_block = 128``, 2nd random generated input sequence
+- ``type = std::int32_t``, ``threads_per_block = 256``, 2nd random generated input sequence
+
+Each new generator multiplies the number of execution times by its number of seeds. That means
+that if there were further more sequence generators (``c2h::gen(C2H_SEED(X), ...)``) on the
+example above the test would execute X more times and so on.
+
+Speedup Compilation Time
+=====================================
+
+Since type lists in the ``C2H_TEST`` form a Cartesian product,
+compilation time grows quickly with every new dimension.
+To keep the compilation process parallelized,
+it's possible to rely on our ``%PARAM%`` machinery:
+
+.. code-block:: c++
+
+    // %PARAM% BLOCK_SIZE bs 128:256
+    using block_sizes = c2h::enum_type_list<int, BLOCK_SIZE>;
+    using types = c2h::type_list<std::uint8_t, std::int32_t>;
+
+    C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]",
+             types, block_sizes)
+    {
+      using type = typename c2h::get<0, TestType>;
+      constexpr int threads_per_block = c2h::get<1, TestType>::value;
+      // ...
+    }
+
+The comment with ``%PARAM%`` is recognized by our CMake scripts.
+It leads to multiple executables being produced from a single test source.
+
+.. code-block:: bash
+
+    bin/cub.test.scope_algorithm.bs_128
+    bin/cub.test.scope_algorithm.bs_256
+
+Multiple ``%PARAM%`` comments can be specified forming another Cartesian product.
+
+Final Test
+=====================================
+
+Let's consider the final test that illustrates all of the tools we discussed above:
+
+.. code-block:: c++
+
+    // %PARAM% BLOCK_SIZE bs 128:256
+    using block_sizes = c2h::enum_type_list<int, BLOCK_SIZE>;
+    using types = c2h::type_list<std::uint8_t, std::int32_t>;
+
+    C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]",
+             types, block_sizes)
+    {
+      using type = typename c2h::get<0, TestType>;
+      constexpr int threads_per_block = c2h::get<1, TestType>::value;
+      constexpr int max_num_items = threads_per_block;
+
+      c2h::device_vector<type> d_input(
+        GENERATE_COPY(take(2, random(0, max_num_items))));
+      c2h::gen(C2H_SEED(3), d_input);
+
+      c2h::device_vector<type> d_output(d_input.size());
+
+      SCOPE_ALGORITHM<threads_per_block>(d_input.data(),
+                                        d_output.data(),
+                                        d_input.size());
+
+      REQUIRE( d_input == d_output );
+
+      const type expected_sum = 4;
+      const type sum = thrust::reduce(c2h::device_policy, d_output.cbegin(), d_output.cend());
+      REQUIRE( sum == expected_sum);
+    }
+
+Apart from discussed tools, here we also rely on ``Catch2`` to generate random input sizes
+in the range ``[0, max_num_items]`` for our input vector ``d_input``.
+Overall, the test will produce two executables.
+Each of these executables is going to generate ``2`` input problem sizes.
+For each problem size, ``3`` random vectors are generated.
+As a result, we have ``12`` different tests.
+The code also demonstrates the syntax and usage of ``c2h::device_policy`` with a Thrust algorithm.
diff --git a/cccl_upstream/docs/cub/developer/thread_level.rst b/cccl_upstream/docs/cub/developer/thread_level.rst
new file mode 100644
index 00000000..e450593c
--- /dev/null
+++ b/cccl_upstream/docs/cub/developer/thread_level.rst
@@ -0,0 +1,24 @@
+.. _cub-developer-guide-thread-level:
+
+Thread-level
+*************
+
+In contrast to algorithms at the warp/block/device layer,
+single threaded functionality like ``cub::ThreadReduce``
+is typically implemented as a sequential function and rarely exposed to the user.
+
+.. code-block:: c++
+
+    template <
+        int         LENGTH,
+        typename    T,
+        typename    ReductionOp,
+        typename    PrefixT,
+        typename    AccumT = detail::accumulator_t<ReductionOp, PrefixT, T>>
+    __device__ __forceinline__ AccumT ThreadReduce(
+        T           (&input)[LENGTH],
+        ReductionOp reduction_op,
+        PrefixT     prefix)
+    {
+        return ...;
+    }
diff --git a/cccl_upstream/docs/cub/developer/warp_level.rst b/cccl_upstream/docs/cub/developer/warp_level.rst
new file mode 100644
index 00000000..23df31e2
--- /dev/null
+++ b/cccl_upstream/docs/cub/developer/warp_level.rst
@@ -0,0 +1,157 @@
+.. _cub-developer-guide-warp-level:
+
+Warp-level
+************************************
+
+CUB warp-level algorithms are specialized for execution by threads in the same CUDA warp.
+These algorithms may only be invoked by ``1 <= n <= 32`` *consecutive* threads in the same warp.
+
+Overview
+====================================
+
+Warp-level functionality is provided by types (classes) to provide encapsulation and enable partial template specialization.
+
+For example, :cpp:struct:`cub::WarpReduce` is a class template:
+
+.. code-block:: c++
+
+    template <typename T,
+              int      LOGICAL_WARP_THREADS = 32>
+    class WarpReduce {
+      // ...
+      // (1)   define `_TempStorage` type
+      // ...
+      _TempStorage &temp_storage;
+    public:
+
+      // (2)   wrap `_TempStorage` in uninitialized memory
+      struct TempStorage : Uninitialized<_TempStorage> {};
+
+      __device__ __forceinline__ WarpReduce(TempStorage &temp_storage)
+      // (3)   reinterpret cast
+        : temp_storage(temp_storage.Alias())
+      {}
+
+      // (4)   actual algorithms
+      __device__ __forceinline__ T Sum(T input);
+    };
+
+In CUDA, the hardware warp size is 32 threads.
+However, CUB enables warp-level algorithms on "logical" warps of ``1 <= n <= 32`` threads.
+The size of the logical warp is required at compile time via the ``LOGICAL_WARP_THREADS`` non-type template parameter.
+This value is defaulted to the hardware warp size of ``32``.
+There is a vital difference in the behavior of warp-level algorithms that depends on the value of ``LOGICAL_WARP_THREADS``:
+
+- If ``LOGICAL_WARP_THREADS`` is a power of two - warp is partitioned into *sub*-warps,
+  each reducing its data independently from other *sub*-warps.
+  The terminology used in CUB: ``32`` threads are called hardware warp.
+  Groups with less than ``32`` threads are called *logical* or *virtual* warp since it doesn't correspond directly to any hardware unit.
+- If ``LOGICAL_WARP_THREADS`` is **not** a power of two - there's no partitioning.
+  That is, only the first logical warp executes algorithm.
+
+.. TODO: Add diagram showing non-power of two logical warps.
+
+Temporary storage usage
+====================================
+
+Warp-level algorithms require temporary storage for scratch space and inter-thread communication.
+The temporary storage needed for a given instantiation of an algorithm is known at compile time
+and is exposed through the ``TempStorage`` member type definition.
+It is the caller's responsibility to create this temporary storage and provide it to the constructor of the algorithm type.
+It is possible to reuse the same temporary storage for different algorithm invocations,
+but it is unsafe to do so without first synchronizing to ensure the first invocation is complete.
+
+.. TODO: Add more explanation of the `TempStorage` type and the `Uninitialized` wrapper.
+.. TODO: Explain if `TempStorage` is required to be shared memory or not.
+
+
+.. code-block:: c++
+
+    using WarpReduce = cub::WarpReduce<int>;
+
+    // Allocate WarpReduce shared memory for four warps
+    __shared__ WarpReduce::TempStorage temp_storage[4];
+
+    // Get this thread's warp id
+    int warp_id = threadIdx.x / 32;
+    int aggregate_1 = WarpReduce(temp_storage[warp_id]).Sum(thread_data_1);
+    // illegal, has to add `__syncwarp()` between the two
+    int aggregate_2 = WarpReduce(temp_storage[warp_id]).Sum(thread_data_2);
+    // illegal, has to add `__syncwarp()` between the two
+    foo(temp_storage[warp_id]);
+
+
+Specialization
+====================================
+
+The goal of CUB is to provide users with algorithms that abstract the complexities of achieving speed-of-light performance across a variety of use cases and hardware.
+It is a CUB developer's job to abstract this complexity from the user by providing a uniform interface that statically dispatches to the optimal code path.
+This is usually accomplished via customizing the implementation based on compile time information like the logical warp size, the data type, and the target architecture.
+For example, :cpp:struct:`cub::WarpReduce` dispatches to two different implementations based on if the logical warp size is a power of two (described above):
+
+.. code-block:: c++
+
+    using InternalWarpReduce = cuda::std::conditional_t<
+      IS_POW_OF_TWO,
+      detail::WarpReduceShfl<T, LOGICAL_WARP_THREADS>,  // shuffle-based implementation
+      detail::WarpReduceSmem<T, LOGICAL_WARP_THREADS>>; // smem-based implementation
+
+Specializations provide different shared memory requirements,
+so the actual ``_TempStorage`` type is defined as:
+
+.. code-block:: c++
+
+    using _TempStorage = typename InternalWarpReduce::TempStorage;
+
+and algorithm implementation look like:
+
+.. code-block:: c++
+
+    __device__ __forceinline__ T Sum(T input, int valid_items) {
+      return InternalWarpReduce(temp_storage)
+          .Reduce(input, valid_items, ::cuda::std::plus<>{});
+    }
+
+
+
+``__CUDA_ARCH__`` cannot be used because it is conflicting with the PTX dispatch refactoring and limited NVHPC support.
+Due to  this limitation, we can't specialize on the PTX version.
+``NV_IF_TARGET`` shall be used by specializations instead:
+
+.. code-block:: c++
+
+    template <typename T, int LOGICAL_WARP_THREADS>
+    struct WarpReduceShfl
+    {
+
+
+    template <typename ReductionOp>
+    __device__ __forceinline__ T ReduceImpl(T input, int valid_items,
+                                            ReductionOp reduction_op)
+    {
+      // ... base case (SM < 80) ...
+    }
+
+    template <class U = T>
+    __device__ __forceinline__
+      typename std::enable_if<std::is_same_v<int, U> ||
+                              std::is_same_v<unsigned int, U>,
+                              T>::type
+        ReduceImpl(T input,
+                  int,               // valid_items
+                  ::cuda::std::plus<>) // reduction_op
+    {
+      T output = input;
+
+      NV_IF_TARGET(NV_PROVIDES_SM_80,
+                  (output = __reduce_add_sync(member_mask, input);),
+                  (output = ReduceImpl<::cuda::std::plus<>>(
+                        input, LOGICAL_WARP_THREADS, ::cuda::std::plus<>{});));
+
+      return output;
+    }
+
+
+    };
+
+Specializations are stored in the ``cub/warp/specializations`` directory.
diff --git a/cccl_upstream/docs/cub/developer_overview.rst b/cccl_upstream/docs/cub/developer_overview.rst
new file mode 100644
index 00000000..5158a8be
--- /dev/null
+++ b/cccl_upstream/docs/cub/developer_overview.rst
@@ -0,0 +1,138 @@
+Developer Overview
+##########################
+
+.. toctree::
+   :hidden:
+   :maxdepth: 2
+
+   developer/thread_level
+   developer/warp_level
+   developer/block_scope
+   developer/device_scope
+   developer/nvtx
+   developer/test_overview
+
+This living document serves as a guide to the design of the internal structure of CUB.
+
+CUB provides layered algorithms that correspond to the thread/warp/block/device hierarchy of threads in CUDA.
+There are distinct algorithms for each layer and higher-level layers build on top of those below.
+
+For example, CUB has four flavors of ``reduce``,
+one for each layer: ``ThreadReduce, WarpReduce, BlockReduce``, and ``DeviceReduce``.
+Each is unique in how it is invoked,
+how many threads participate,
+and on which thread(s) the result is valid.
+
+These layers naturally build on each other.
+For example, :cpp:struct:`cub::WarpReduce` uses :cpp:func:`cub::ThreadReduce`,
+:cpp:struct:`cub::BlockReduce` uses :cpp:struct:`cub::WarpReduce`, etc.
+
+:cpp:func:`cub::ThreadReduce`
+
+   - A normal function invoked and executed sequentially by a single thread that returns a valid result on that thread
+   - Single thread functions are usually an implementation detail and not exposed in CUB's public API
+
+:cpp:struct:`cub::WarpReduce` and :cpp:struct:`cub::BlockReduce`
+
+   - A "cooperative" function where threads concurrently invoke the same function to execute parallel work
+   - The function's return value is well-defined only on the "first" thread (lowest thread index)
+
+:cpp:struct:`cub::DeviceReduce`
+
+   - A normal function invoked by a single thread that spawns additional threads to execute parallel work
+   - Result is stored in the pointer provided to the function
+   - Function returns a ``cudaError_t`` error code
+   - Function does not synchronize the host with the device
+
+
+The table below provides a summary of these functions:
+
+.. list-table::
+    :class: table-no-stripes
+    :header-rows: 1
+
+    * - layer
+      - coop invocation
+      - parallel execution
+      - max threads
+      - valid result in
+    * - :cpp:func:`cub::ThreadReduce`
+      - :math:`-`
+      - :math:`-`
+      - :math:`1`
+      - invoking thread
+    * - :cpp:struct:`cub::WarpReduce`
+      - :math:`+`
+      - :math:`+`
+      - :math:`32`
+      - main thread
+    * - :cpp:struct:`cub::BlockReduce`
+      - :math:`+`
+      - :math:`+`
+      - :math:`1024`
+      - main thread
+    * - :cpp:struct:`cub::DeviceReduce`
+      - :math:`-`
+      - :math:`+`
+      - :math:`\infty`
+      - global memory
+
+The details of how each of these layers are implemented is described below.
+
+Common Patterns
+************************************
+
+While CUB's algorithms are unique at each layer,
+there are commonalities among all of them:
+
+    - Algorithm interfaces are provided as *types* (classes)\ [1]_
+    - Algorithms need temporary storage
+    - Algorithms dispatch to specialized implementations depending on compile-time and runtime information
+    - Cooperative algorithms require the number of threads at compile time (template parameter)
+
+Invoking any CUB algorithm follows the same general pattern:
+
+    #. Select the class for the desired algorithm
+    #. Query the temporary storage requirements
+    #. Allocate the temporary storage
+    #. Pass the temporary storage to the algorithm
+    #. Invoke it via the appropriate member function
+
+An example of :cpp:struct:`cub::BlockReduce` demonstrates these patterns in practice:
+
+.. code-block:: c++
+
+    __global__ void kernel(int* per_block_results)
+    {
+      // (1) Select the desired class
+      // `cub::BlockReduce` is a class template that must be instantiated for the
+      // input data type and the number of threads. Internally the class is
+      // specialized depending on the data type, number of threads, and hardware
+      // architecture. Type aliases are often used for convenience:
+      using BlockReduce = cub::BlockReduce<int, 128>;
+      // (2) Query the temporary storage
+      // The type and amount of temporary storage depends on the selected instantiation
+      using TempStorage = typename BlockReduce::TempStorage;
+      // (3) Allocate the temporary storage
+      __shared__ TempStorage temp_storage;
+      // (4) Pass the temporary storage
+      // Temporary storage is passed to the constructor of the `BlockReduce` class
+      BlockReduce block_reduce{temp_storage};
+      // (5) Invoke the algorithm
+      // The `Sum()` member function performs the sum reduction of `thread_data` across all 128 threads
+      int thread_data[4] = {1, 2, 3, 4};
+      int block_result = block_reduce.Sum(thread_data);
+
+      per_block_results[blockIdx.x] = block_result;
+    }
+
+.. [1] Algorithm interfaces are provided as classes because it provides encapsulation for things like temporary storage requirements and enables partial template specialization for customizing an algorithm for specific data types or number of threads.
+
+For more detailed descriptions of the respective algorithms levels see the individual sections below
+
+  - :ref:`thread-level algorithms<cub-developer-guide-thread-level>`
+  - :ref:`warp-level algorithms<cub-developer-guide-warp-level>`
+  - :ref:`block-scope algorithms<cub-developer-guide-block-scope>`
+  - :ref:`device-scope algorithms<cub-developer-guide-device-scope>`
+
+There is additional information for :ref:`nvtx ranges <cub-developer-guide-nvtx>`
diff --git a/cccl_upstream/docs/cub/device_topk_requirements.rst b/cccl_upstream/docs/cub/device_topk_requirements.rst
new file mode 100644
index 00000000..a3117f9c
--- /dev/null
+++ b/cccl_upstream/docs/cub/device_topk_requirements.rst
@@ -0,0 +1,323 @@
+:orphan:
+
+.. _cub-topk-requirements:
+
+Top-K: Determinism, Tie-Breaking, and Output Ordering
+======================================================
+
+This page describes how to control the result of the CUB top-k family of algorithms
+(:cpp:struct:`cub::DeviceTopK` and :cpp:struct:`cub::DeviceBatchedTopK`) through the execution
+environment. For :cpp:struct:`cub::DeviceBatchedTopK`, these requirements apply independently within
+each segment. The same requirement model applies to every ``MaxKeys`` / ``MinKeys`` / ``MaxPairs`` /
+``MinPairs`` entry point.
+
+Two orthogonal concerns
+-----------------------
+
+Top-k algorithms answer two separate questions:
+
+#. **Which items are returned?** (the result *set* / membership), controlled by
+   ``cuda::execution::determinism`` and, when deterministic, optionally refined by
+   ``cuda::execution::tie_break``.
+#. **In what order are those items written to the output?** (the result *sequence*), controlled
+   independently by ``cuda::execution::output_ordering``.
+
+Think of it this way: determinism (with an optional tie-break) first selects a *set* of *K* items.
+Output ordering then arranges that fixed set into the output buffer. Changing output ordering never
+changes *which* items are selected. Changing tie-breaking never dictates *how* equal-key items are
+sequenced in the output (unless you also request a stable ordering, as described below).
+
+**Determinism applies to set membership.** Even with a deterministic selection, the *positions* of
+the selected items in the output buffer may still vary unless you also request a specific output
+ordering. Non-determinism arises only when more elements compare equal at the selection boundary
+than there are remaining slots in the top-*K*. For example, with *K* = 3 and four elements tied for
+the third-largest position, the algorithm must choose three of the four, and that choice is the
+source of variability.
+
+**Output ordering applies to the result sequence.** Once the result set is fixed, output ordering
+specifies how those *K* items are laid out in the output buffer.
+
+.. _cub-topk-default-behavior:
+
+Default behavior
+----------------
+
+When you do **not** specify any of these requirements, the top-k algorithms provide their strongest
+reproducibility guarantees. The committed default contract is:
+
+* ``cuda::execution::determinism::gpu_to_gpu`` for a deterministic result set,
+* ``cuda::execution::tie_break::prefer_smaller_index`` to resolve ties at the selection boundary
+  toward the smaller (lower) source index,
+* ``cuda::execution::output_ordering::stable_sorted`` to write output sorted by key, with equal
+  keys ordered by source index.
+
+In other words, by default you get the same items, in the same positions, run after run and across
+GPUs of the same architecture. You opt **out** of these guarantees (by requiring weaker properties
+such as ``cuda::execution::determinism::not_guaranteed`` and
+``cuda::execution::output_ordering::unsorted``) to obtain faster implementations.
+
+``determinism`` and ``tie_break`` are coupled. You specify **both** of them (inside a single
+``cuda::execution::require(...)``) or **neither** (to take the default). A specified ``tie_break`` of
+``prefer_smaller_index`` or ``prefer_larger_index`` pins the result set across GPUs and therefore
+requires ``determinism::gpu_to_gpu``. See :ref:`cub-topk-set-membership` for the full table.
+
+.. note::
+
+   **Current support.** This initial API surface only implements the fully opted-out configuration.
+   For :cpp:struct:`cub::DeviceBatchedTopK` it must be requested **explicitly** as
+   ``cuda::execution::require(cuda::execution::determinism::not_guaranteed,
+   cuda::execution::tie_break::unspecified, cuda::execution::output_ordering::unsorted)``
+   (:cpp:struct:`cub::DeviceTopK` has no tie-break dimension yet and omits the ``tie_break`` token).
+   The algorithms ``static_assert`` for any other combination (including an empty, no-requirement
+   environment), so the deterministic default described above cannot yet be exercised in code. The
+   deterministic, tie-broken, and (stable-)sorted modes documented here define the committed long-term
+   contract and will become available (including as the no-requirement default) as those code paths
+   land.
+
+Requirements reference
+----------------------
+
+Determinism (``cuda::execution::determinism``)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. list-table::
+   :header-rows: 1
+   :widths: 25 75
+
+   * - Value
+     - Meaning
+   * - ``not_guaranteed``
+     - No reproducibility guarantee. Among tied elements at the selection boundary, any valid subset
+       may be returned. Enables the fastest implementations.
+   * - ``run_to_run``
+     - The result set is identical across repeated invocations on the same GPU with the same input.
+       The tie-breaking policy is implementation-defined. Pinning a specific tie-break is not
+       available at this level and requires ``gpu_to_gpu``.
+   * - ``gpu_to_gpu``
+     - The result set is identical across different GPUs of the same architecture. This is the only
+       level that may be combined with an explicit ``tie_break`` (``prefer_smaller_index`` or
+       ``prefer_larger_index``), which then fully pins the result set for a given input.
+
+Tie-break (``cuda::execution::tie_break``)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A specified ``tie_break`` of ``prefer_smaller_index`` or ``prefer_larger_index`` pins the result set
+across GPUs, so it requires ``determinism::gpu_to_gpu``. Pairing it with ``run_to_run`` or
+``not_guaranteed`` is rejected at compile time. ``determinism`` and ``tie_break`` must always be
+specified together (or both omitted to take the default). Use ``tie_break::unspecified`` to leave the
+boundary policy to the implementation, for example alongside ``not_guaranteed`` or ``run_to_run``.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 30 70
+
+   * - Value
+     - Meaning
+   * - ``unspecified``
+     - Any deterministic tie-break is acceptable, and the implementation chooses. Valid with any
+       determinism level (including ``not_guaranteed`` and ``run_to_run``).
+   * - ``prefer_smaller_index`` *(default)*
+     - Among elements that compare equal at the boundary, prefer those with the **smaller** source
+       index. Requires ``determinism::gpu_to_gpu``.
+   * - ``prefer_larger_index``
+     - Among elements that compare equal at the boundary, prefer those with the **larger** source
+       index. Requires ``determinism::gpu_to_gpu``.
+
+Output ordering (``cuda::execution::output_ordering``)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. list-table::
+   :header-rows: 1
+   :widths: 25 75
+
+   * - Value
+     - Meaning
+   * - ``unsorted``
+     - No guarantee on output order. The same result set may appear in different permutations across
+       runs.
+   * - ``sorted``
+     - Output is sorted by key value (descending for ``Max*``, ascending for ``Min*``). Among
+       elements with equal keys, the relative order is **unspecified**.
+   * - ``stable_sorted``
+     - Output is sorted by key value, and among equal keys the relative order matches the **input
+       order** (smaller source index first). With a fully pinned result set (an explicit
+       ``tie_break``) this fully determines the output, so the result is bit-identical even across
+       GPUs of the same architecture.
+
+Composing requirements
+----------------------
+
+Requirements compose into a single ``cuda::execution::require(...)`` argument, which is placed in the
+execution environment alongside other properties such as a stream:
+
+.. code-block:: c++
+
+   auto env = cuda::std::execution::env{
+     cuda::execution::require(
+       cuda::execution::determinism::gpu_to_gpu,
+       cuda::execution::tie_break::prefer_smaller_index,
+       cuda::execution::output_ordering::sorted),
+     stream_ref};
+
+.. _cub-topk-set-membership:
+
+Which items are selected?
+-------------------------
+
+Determinism and tie-break together control **set membership**. They are always specified as a pair
+(or both omitted to take the default). Rows below are the ``determinism`` requirement and columns are
+the paired ``tie_break`` requirement. Cells marked *(compile error)* are rejected by a ``static_assert``.
+
+.. list-table::
+   :header-rows: 1
+   :stub-columns: 1
+   :widths: 22 26 26 26
+
+   * - ``determinism``
+     - ``tie_break::unspecified``
+     - ``tie_break::prefer_smaller_index``
+     - ``tie_break::prefer_larger_index``
+   * - ``not_guaranteed``
+     - Non-deterministic (fast path)
+     - *(compile error)*
+     - *(compile error)*
+   * - ``run_to_run``
+     - Deterministic, implementation-defined tie-break
+     - *(compile error)*
+     - *(compile error)*
+   * - ``gpu_to_gpu``
+     - Deterministic, implementation-defined tie-break
+     - Deterministic, ties toward the **smaller** source index
+     - Deterministic, ties toward the **larger** source index
+
+Reading the table:
+
+* A specified ``tie_break`` of ``prefer_smaller_index`` or ``prefer_larger_index`` pins the result set
+  across GPUs, which is a ``gpu_to_gpu`` guarantee. Requesting it alongside ``not_guaranteed`` or
+  ``run_to_run`` is a compile error, because you must acknowledge the ``gpu_to_gpu`` determinism you
+  receive.
+* With ``tie_break::unspecified`` the implementation chooses the boundary policy. ``run_to_run`` and
+  ``gpu_to_gpu`` then differ only in *scope*: identical results on the same GPU versus across GPUs of
+  the same architecture.
+* Omitting **both** requirements selects the default (``gpu_to_gpu`` with ``prefer_smaller_index``),
+  which is the bottom-middle cell.
+
+.. note::
+
+   This determinism and tie_break pairing rule is currently enforced only by
+   :cpp:struct:`cub::DeviceBatchedTopK`. :cpp:struct:`cub::DeviceTopK` does not yet inspect
+   ``tie_break``, so it still accepts requirement combinations that ``cub::DeviceBatchedTopK`` rejects.
+   The same enforcement will be added to ``cub::DeviceTopK`` in the next major release of CCCL (4.0).
+
+Worked example: set membership x output ordering
+-------------------------------------------------
+
+Consider ``cub::DeviceTopK::MaxKeys`` with *K* = 3 on this input:
+
+.. code-block:: text
+
+   index :  0     1     2     3     4     5
+   key   : 10     8     8     8     6     5
+
+The top three keys are ``10`` and two ``8``\ s. Four elements compare equal at the boundary (the
+``8``\ s at indices 1, 2, 3), but only two can be kept. That is the tie. The notation ``key@index``
+identifies an element by both its key and its source position (for example ``8@2`` is the ``8`` at
+index 2).
+
+The table below shows **two runs on the same input** for each combination. Compare the two runs
+within a cell to see whether the output order varies. Compare across rows to see whether the set
+membership varies.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 28 24 24 24
+
+   * - ``require(...)``
+     - ``output_ordering::unsorted``
+     - ``output_ordering::sorted``
+     - ``output_ordering::stable_sorted``
+   * - ``determinism::not_guaranteed,``
+       ``tie_break::unspecified``
+     - | Run 1: ``[8@2, 10@0, 8@1]``
+       | Run 2: ``[8@3, 10@0, 8@1]``
+       | Different sets *and* orders
+     - | Run 1: ``[10@0, 8@2, 8@1]``
+       | Run 2: ``[10@0, 8@1, 8@3]``
+       | Different sets, sorted by key
+     - | Run 1: ``[10@0, 8@1, 8@2]``
+       | Run 2: ``[10@0, 8@1, 8@3]``
+       | Different sets, equal keys in input order
+   * - ``determinism::run_to_run,``
+       ``tie_break::unspecified``
+     - | Run 1: ``[8@3, 10@0, 8@1]``
+       | Run 2: ``[10@0, 8@1, 8@3]``
+       | Same set ``{10@0, 8@1, 8@3}``, order may vary
+     - | Run 1: ``[10@0, 8@3, 8@1]``
+       | Run 2: ``[10@0, 8@1, 8@3]``
+       | Same set, equal-key order unspecified
+     - | Run 1: ``[10@0, 8@1, 8@3]``
+       | Run 2: ``[10@0, 8@1, 8@3]``
+       | Same set, equal keys always in input order
+   * - ``determinism::gpu_to_gpu,``
+       ``tie_break::prefer_smaller_index``
+     - | Run 1: ``[8@2, 10@0, 8@1]``
+       | Run 2: ``[10@0, 8@1, 8@2]``
+       | Same set ``{10@0, 8@1, 8@2}``, order may vary
+     - | Run 1: ``[10@0, 8@2, 8@1]``
+       | Run 2: ``[10@0, 8@1, 8@2]``
+       | Same set, equal-key order unspecified
+     - | Run 1: ``[10@0, 8@1, 8@2]``
+       | Run 2: ``[10@0, 8@1, 8@2]``
+       | Same set, equal keys always in input order
+   * - ``determinism::gpu_to_gpu,``
+       ``tie_break::prefer_larger_index``
+     - | Run 1: ``[8@3, 10@0, 8@2]``
+       | Run 2: ``[10@0, 8@2, 8@3]``
+       | Same set ``{10@0, 8@2, 8@3}``, order may vary
+     - | Run 1: ``[10@0, 8@3, 8@2]``
+       | Run 2: ``[10@0, 8@2, 8@3]``
+       | Same set, equal-key order unspecified
+     - | Run 1: ``[10@0, 8@2, 8@3]``
+       | Run 2: ``[10@0, 8@2, 8@3]``
+       | Same set, equal keys always in input order
+
+Reading the matrix:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 45 55
+
+   * - Observation
+     - Where to look
+   * - Set membership varies across runs
+     - ``not_guaranteed`` row: Run 1 keeps ``8@2``, Run 2 keeps ``8@3``
+   * - Set membership fixed, order varies
+     - ``run_to_run`` + ``unsorted``: both runs return ``{10@0, 8@1, 8@3}`` in different permutations
+   * - Set membership fixed, sorted but unstable among equal keys
+     - ``run_to_run`` + ``sorted``: both runs start with ``10@0``, but ``8@1`` and ``8@3`` may swap
+   * - Fully pinned: same set and same order
+     - ``gpu_to_gpu`` + ``tie_break::prefer_smaller_index`` + ``stable_sorted``: both runs yield
+       ``[10@0, 8@1, 8@2]``
+   * - Tie-break changes the set, not just the order
+     - Compare ``prefer_smaller_index`` vs ``prefer_larger_index``: ``8@2`` vs ``8@3``
+
+Choosing requirements
+---------------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 55 45
+
+   * - Goal
+     - Suggested ``require(...)``
+   * - Maximum performance, exact result unimportant
+     - ``determinism::not_guaranteed, tie_break::unspecified, output_ordering::unsorted``
+   * - Reproducible result set, order does not matter
+     - ``determinism::run_to_run, tie_break::unspecified, output_ordering::unsorted``
+   * - Reproducible result set with an explicit boundary policy
+     - ``determinism::gpu_to_gpu, tie_break::prefer_{smaller,larger}_index, output_ordering::unsorted``
+   * - Reproducible, key-sorted output
+     - the above + ``output_ordering::sorted``
+   * - Reproducible, key-sorted output with input-order stability among ties
+     - the above + ``output_ordering::stable_sorted`` (a fully pinned set plus stable-sorted output
+       is bit-identical, including across GPUs)
diff --git a/cccl_upstream/docs/cub/device_wide.rst b/cccl_upstream/docs/cub/device_wide.rst
new file mode 100644
index 00000000..8659ef5a
--- /dev/null
+++ b/cccl_upstream/docs/cub/device_wide.rst
@@ -0,0 +1,120 @@
+.. _device-module:
+
+Device-Wide Primitives
+======================
+
+.. toctree::
+   :glob:
+   :hidden:
+   :maxdepth: 2
+
+   api/device
+
+Almost all of CUB's device-wide APIs come in two flavors:
+
+* the traditional two-phase style that requires calling the API twice and managing temporary storage explicitly,
+* and the newer single-phase style where temporary storage is obtained from a memory resource in the execution environment.
+
+Some APIs that do not require any temporary storage may also have a traditional single-phase form in addition to the newer environment-based one.
+
+.. _device-temp-storage:
+
+Two-Phase API (explicit temporary storage management)
++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+Traditional two-phase APIs can be recognized by taking ``void* d_temp_storage, size_t& temp_storage_bytes`` as their first two parameters.
+They follow a two-phase usage pattern that requires three steps:
+
+1. **Query Phase**: The algorithm is called the first time with ``d_temp_storage = nullptr`` to determine the required temporary storage size.
+   The required size is written to ``temp_storage_bytes`` without dereferencing iterators or launching kernels.
+2. **Temporary storage allocation**: The user is responsible for allocating device-accessible memory of at least ``temp_storage_bytes`` bytes.
+   No special alignment is required.
+3. **Execution Phase**: The algorithm is called the second time with ``d_temp_storage`` pointing to the allocated device memory, performing the actual operation.
+
+In principle, the query phase and execution phase must call the same CUB API.
+This means in detail:
+
+* **Template arguments**: The query call must use the same template arguments as the execution call, so they share the same template instantiation.
+* **Argument values**: Regarding function parameters, only the values of the ``d_temp_storage``, ``temp_storage_bytes``,
+  and problem-size related arguments (like number of elements, number of segments, segment sizes, etc.) may be read during the query phase.
+  No other parameters (like input/output iterators, initial values, etc.) are accessed during the query phase, so their values may be indeterminate.
+  During the query phase, the API will return before launching any kernels or touching user storage.
+* **Current device**: The computed temporary storage size is valid only when the execution phase runs on the same current CUDA device as the query.
+  Re-run the query if the current device changes between phases.
+
+Example pattern:
+
+.. literalinclude:: ../../cub/examples/device/example_device_reduce.cu
+   :language: c++
+   :dedent:
+   :start-after: example-begin temp-storage-query
+   :end-before: example-end temp-storage-query
+
+
+Environment API (single phase)
+++++++++++++++++++++++++++++++
+
+The environment-based API is available for all CUB device-wide algorithms.
+They remove the split of query/execute phase and manually obtaining the temporary storage.
+Instead, the temporary storage is automatically requested from a memory resource queried from the execution environment argument.
+The environment supports further properties like passing a stream or an execution requirement in addition to a memory resource.
+
+Key properties of the environment argument:
+
+- It is a defaulted parameter and appears as the last argument.
+- Streams like `cudaStream_t` or `cuda::stream_ref` can be passed as environments directly, or added to the environment.
+- You can select the memory resource (CCCL-provided or custom) used for internal allocations.
+- Supported algorithms accept determinism requirements (for example, ``cuda::execution::determinism::gpu_to_gpu``).
+- Multiple properties compose into a single centralized argument by wrapping them into a ``cuda::execution::env`` object.
+
+Example pattern:
+
+.. literalinclude:: ../../cub/examples/device/example_device_reduce_env.cu
+   :language: c++
+   :dedent:
+   :start-after: example-begin env-overload-setup
+   :end-before: example-end env-overload-setup
+
+.. literalinclude:: ../../cub/examples/device/example_device_reduce_env.cu
+   :language: c++
+   :dedent:
+   :start-after: example-begin env-overload-run
+   :end-before: example-end env-overload-run
+
+Further information on CUB execution environments can be found in
+:ref:`Execution Environments <cub-environment>`.
+
+
+API overview
+++++++++++++
+
+In the following, the various groups of CUB device-wide algorithms are listed,
+linking to their respective documentation.
+
+CUB device-level single-problem parallel algorithms:
+
+* :cpp:struct:`cub::DeviceAdjacentDifference` computes the difference between adjacent elements residing within device-accessible memory
+* :cpp:struct:`cub::DeviceFor` provides device-wide, parallel operations for iterating over data residing within device-accessible memory
+* :cpp:struct:`cub::DeviceHistogram` constructs histograms from data samples residing within device-accessible memory
+* :cpp:struct:`cub::DevicePartition` partitions data residing within device-accessible memory
+* :cpp:struct:`cub::DeviceMerge` merges two sorted sequences in device-accessible memory into a single one
+* :cpp:struct:`cub::DeviceMergeSort` sorts items residing within device-accessible memory
+* :cpp:struct:`cub::DeviceRadixSort` sorts items residing within device-accessible memory using radix sorting method
+* :cpp:struct:`cub::DeviceReduce` computes reduction of items residing within device-accessible memory
+* :cpp:struct:`cub::DeviceRunLengthEncode` demarcating "runs" of same-valued items within a sequence residing within device-accessible memory
+* :cpp:struct:`cub::DeviceScan` computes a prefix scan across a sequence of data items residing within device-accessible memory
+* :cpp:struct:`cub::DeviceSelect` compacts data residing within device-accessible memory
+* :cpp:struct:`cub::DeviceTransform` transforms elements from multiple input sequences into an output sequence
+* :cpp:struct:`cub::DeviceTopK` finds the largest (or smallest) K items from an unordered list residing within device-accessible memory
+
+
+CUB device-level segmented-problem (batched) parallel algorithms:
+
+* :cpp:struct:`cub::DeviceSegmentedSort` computes batched sort across non-overlapping sequences of data residing within device-accessible memory
+* :cpp:struct:`cub::DeviceSegmentedRadixSort` computes batched radix sort across non-overlapping sequences of data residing within device-accessible memory
+* :cpp:struct:`cub::DeviceSegmentedReduce` computes reductions across multiple sequences of data residing within device-accessible memory
+* :cpp:struct:`cub::DeviceSegmentedScan` computes prefix scans across multiple sequences of data residing within device-accessible memory
+* :cpp:struct:`cub::DeviceBatchedTopK` finds the largest (or smallest) K items from each of multiple unordered lists (segments) residing within device-accessible memory
+* :cpp:struct:`cub::DeviceCopy` provides device-wide, parallel operations for batched copying of data residing within device-accessible memory
+* :cpp:struct:`cub::DeviceMemcpy` provides device-wide, parallel operations for batched copying of data residing within device-accessible memory
+* :cpp:struct:`cub::DeviceFind` provides vectorized binary search algorithms
diff --git a/cccl_upstream/docs/cub/environment.rst b/cccl_upstream/docs/cub/environment.rst
new file mode 100644
index 00000000..73e5a062
--- /dev/null
+++ b/cccl_upstream/docs/cub/environment.rst
@@ -0,0 +1,253 @@
+.. _cub-environment:
+
+Execution Environments
+======================
+
+Most CUB device-wide algorithms accept an optional *execution environment* as their last
+argument. The environment is a single object that bundles everything CUB needs to know
+about *how* to run the algorithm.
+
+Part of what it carries is familiar: the CUDA stream, which CUB algorithms have always
+accepted, now simply travels inside the environment. The rest are controls that had no
+place in the classic API and are enabled by environments:
+:ref:`determinism requirements <cub-env-determinism>`,
+:ref:`custom tuning policy selectors <cub-env-tuning>`, and
+:ref:`memory resources <cub-env-memory-resource>`. All properties are optional and freely
+composable with each other.
+
+This page explains what the CUB environment APIs are, and how to use them.
+
+.. contents::
+   :local:
+   :depth: 2
+
+
+Why environments?
+-----------------
+
+The classic two-phase CUB API requires three steps: query temporary-storage size, allocate,
+then execute. That is fine for situations where precise control over the temporary storage
+allocation is required, or when the temporary storage size needs to be queried independently
+of allocating it. For example, when the temporary storage is combined with other storage, e.g.
+for an output, into a single allocation. But in many cases this is not needed and the two-step
+API is just repetitive boilerplate.
+
+The environment-based single-phase API collapses all of that into one call. The algorithm
+queries the environment for the properties it needs — for example, the stream to run on,
+or a memory resource to allocate temporary storage from — and then executes. Properties
+an algorithm does not use are simply ignored, which is what makes one environment safe to
+pass to many different algorithms:
+
+.. code-block:: c++
+
+   cub::DeviceReduce::Sum(d_input, d_output, num_items, env);
+
+.. note::
+
+   The environment argument is entirely
+   optional: since it is defaulted, an algorithm can be invoked with no temporary-storage
+   arguments and no environment at all, and every property falls back to its default
+   (see :ref:`Default behavior <cub-environment-fallback>`):
+
+   .. code-block:: c++
+
+      cub::DeviceReduce::Sum(d_input, d_output, num_items);
+
+
+.. _cub-env-building:
+
+Building an environment
+-----------------------
+
+.. list explicitly what types are valid environments
+Some types are already valid environments on their own and can be passed directly to an
+algorithm. The most common case is a stream: ``cuda::stream_ref`` (or a raw
+``cudaStream_t``) passed as the last argument is treated as an environment containing
+just that stream.
+
+.. literalinclude:: ../../cub/test/catch2_test_device_reduce_env_api.cu
+   :language: c++
+   :dedent:
+   :start-after: example-begin reduce-env-stream
+   :end-before: example-end reduce-env-stream
+
+When you need more than one property, combine them with ``cuda::std::execution::env``.
+Properties can be listed in any order. The example below
+composes a stream, a memory pool for temporary storage, and a determinism requirement
+(the required declarations are provided by ``<cuda/execution>``, ``<cuda/stream>``,
+``<cuda/memory_pool>``, and ``<cuda/devices>``):
+
+.. literalinclude:: ../../cub/examples/device/example_device_reduce_env.cu
+   :language: c++
+   :dedent:
+   :start-after: example-begin env-overload-setup
+   :end-before: example-end env-overload-setup
+
+.. literalinclude:: ../../cub/examples/device/example_device_reduce_env.cu
+   :language: c++
+   :dedent:
+   :start-after: example-begin env-overload-run
+   :end-before: example-end env-overload-run
+
+The same ``env`` object can be passed to multiple algorithm calls without rebuilding it each
+time, see :ref:`cub-environment-reuse`.
+
+
+How to use environments
+-----------------------
+
+The controls below are what environments enable beyond the classic API. A few algorithms
+additionally accept algorithm-specific controls — e.g. tie-breaking and output-ordering
+for :ref:`cub::DeviceTopK <cub-topk-requirements>`. The set of supported properties
+keeps growing.
+
+.. _cub-env-determinism:
+
+Determinism Requirements
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Use ``cuda::execution::require`` to request a guarantee, like a reproducible/deterministic execution:
+
+.. literalinclude:: ../../cub/test/catch2_test_device_reduce_env_api.cu
+   :language: c++
+   :dedent:
+   :start-after: example-begin reduce-env-determinism
+   :end-before: example-end reduce-env-determinism
+
+Multiple determinism levels are available. The meaning of each is described in the
+:ref:`CCCL determinism overview <cccl-determinism>`. Which algorithms support which levels,
+and each algorithm's default, are listed in the :ref:`CUB determinism support matrix
+<cub-determinism>`. Requesting a level an algorithm does not support is rejected at
+compile time.
+
+
+.. _cub-env-tuning:
+
+Custom Tuning Policy Selectors
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Pass a custom policy selector through the environment to override CUB's built-in tuning.
+A policy selector answers the question "which tuning parameters should this algorithm use
+on this GPU?": it is a function object that CUB calls with the GPU's compute capability,
+and that returns the tuning parameters — threads per block, items per thread, and so on —
+packaged as the algorithm's policy (here ``cub::MergePolicy``):
+
+.. literalinclude:: ../../cub/test/catch2_test_device_merge_env_api.cu
+   :language: c++
+   :dedent:
+   :start-after: example-begin merge-keys-policy-selector
+   :end-before: example-end merge-keys-policy-selector
+
+The selector is wrapped with ``cuda::execution::tune`` and passed as (part of) the
+environment:
+
+.. literalinclude:: ../../cub/test/catch2_test_device_merge_env_api.cu
+   :language: c++
+   :dedent:
+   :start-after: example-begin merge-keys-tuning
+   :end-before: example-end merge-keys-tuning
+
+.. seealso:: :ref:`cub-policy-selectors` - full guide on defining and composing policy
+   selectors, including the requirements a policy selector must satisfy.
+
+.. _cub-env-memory-resource:
+
+Memory Resources
+~~~~~~~~~~~~~~~~
+
+The memory resource controls where an algorithm's temporary storage is allocated from.
+Memory resource types are valid environments on their own, so they can be passed directly
+or composed with other properties:
+
+.. code-block:: c++
+
+   auto pool = cuda::device_default_memory_pool(cuda::devices[0]);
+
+   // Pass directly...
+   cub::DeviceReduce::Sum(d_input, d_output, num_items, pool);
+
+   // ...or compose with other properties
+   auto env = cuda::std::execution::env{stream, pool};
+   cub::DeviceReduce::Sum(d_input, d_output, num_items, env);
+
+Temporary storage is allocated from the memory resource on the algorithm's stream before
+execution and released on the same stream afterwards. When no memory resource is present
+in the environment, CUB falls back to a stream-ordered ``cudaMallocAsync``/``cudaFree``  allocator
+(see :ref:`Default behavior <cub-environment-fallback>`).
+
+.. TODO: Add Guarantees sub-section after #9278 is merged.
+
+.. _cub-environment-reuse:
+
+Reusing an environment across multiple calls
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+An ``env`` object can be built once and passed to as many algorithm
+calls as you like:
+
+.. code-block:: c++
+
+   auto stream = cuda::stream{cuda::devices[0]};
+   auto pool   = cuda::device_default_memory_pool(cuda::devices[0]);
+   auto env    = cuda::std::execution::env{cuda::stream_ref{stream}, pool};
+
+   cub::DeviceScan::ExclusiveSum(d_a, d_out_a, n, env);
+   cub::DeviceReduce::Sum(d_b, d_out_b, n, env);
+   cub::DeviceSelect::If(d_c, d_out_c, d_num_selected, n, my_predicate, env);
+
+   stream.sync();
+
+All three calls share the same stream and memory pool. Temporary storage is allocated and
+released from the pool independently for each call.
+
+.. note::
+
+   When a tuning policy selector is embedded in the environment, it applies to *every*
+   algorithm that can be tuned by this selector. For example, a policy selector returning
+   a ``cub::ReducePolicy`` will be used by all calls to ``cub::DeviceReduce::*`` that use
+   the same environment. If two algorithms using the same environment need different
+   tunings, build separate environments.
+
+
+.. _cub-environment-fallback:
+
+Default behavior
+----------------
+
+The environment argument is optional on every algorithm that supports it. CUB applies the
+following defaults when a property is absent from the environment (or when no environment
+is passed at all):
+
+.. list-table::
+   :header-rows: 1
+   :widths: 25 75
+
+   * - Property
+     - Default when absent
+   * - Stream
+     - The default CUDA stream (``cudaStream_t{}``).
+   * - Memory resource
+     - A stream-ordered allocator based on ``cudaMallocAsync``. Temporary storage is
+       allocated on the stream the algorithm runs on.
+   * - Determinism
+     - Algorithm-specific. See :ref:`cub-determinism`.
+   * - Policy selector
+     - CUB's built-in selector, returning architecture-tuned defaults for the current
+       device.
+
+Missing properties never cause an error. The algorithm simply falls back to its default for
+that property.
+
+..
+   TODO(gonidelis): link to the developer-facing environments page (queries, CPOs, prop,
+   building custom environment types) once it lands via
+   https://github.com/NVIDIA/cccl/pull/10013
+
+
+See also
+--------
+
+- :ref:`cub-determinism` - per-algorithm determinism support matrix
+- :ref:`cub-policy-selectors` - defining and composing custom policy selectors
+- :ref:`device-module` - overview of the device-wide API and the two-phase alternative
+- :ref:`cccl-determinism` - CCCL-level determinism concepts
diff --git a/cccl_upstream/docs/cub/index.rst b/cccl_upstream/docs/cub/index.rst
new file mode 100644
index 00000000..96245a52
--- /dev/null
+++ b/cccl_upstream/docs/cub/index.rst
@@ -0,0 +1,487 @@
+.. _cub-module:
+
+CUB
+==================================================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 3
+
+   Overview <self>
+   thread_level
+   warp_wide
+   block_wide
+   device_wide
+   environment
+   determinism
+   benchmarking
+   tuning
+   tuning_infra
+   API reference <api/index>
+   developer_overview
+
+What is CUB?
+==================================================
+
+CUB provides state-of-the-art, reusable software components for every layer
+of the CUDA programming model:
+
+* **Parallel primitives**
+
+  * :ref:`Thread <thread-module>` primitives
+
+    * Thread-level reduction, etc.
+    * Safely specialized for each underlying CUDA architecture
+
+  * :ref:`Warp-wide <warp-module>` "collective" primitives
+
+    * Cooperative warp-wide prefix scan, reduction, etc.
+    * Safely specialized for each underlying CUDA architecture
+
+  * :ref:`Block-wide <block-module>` "collective" primitives
+
+    * Cooperative I/O, sort, scan, reduction, histogram, etc.
+    * Compatible with arbitrary thread block sizes and types
+
+  * :ref:`Device-wide <device-module>` primitives
+
+    * Parallel sort, prefix scan, reduction, histogram, etc.
+    * Compatible with CUDA dynamic parallelism
+
+* **Utilities**
+
+  * **Fancy iterators**
+  * **Thread and thread block I/O**
+  * **PTX intrinsics**
+  * **Device, kernel, and storage management**
+
+.. _collective-primitives:
+
+CUB's collective primitives
+==================================================
+
+Collective software primitives are essential for constructing high-performance,
+maintainable CUDA kernel code.  Collectives allow complex parallel code to be
+re-used rather than re-implemented, and to be re-compiled rather than
+hand-ported.
+
+.. figure:: ../img/cub_overview.png
+    :align: center
+    :alt: Orientation of collective primitives within the CUDA software stack
+    :name: fig_cub_overview
+
+    Orientation of collective primitives within the CUDA software stack
+
+As a SIMT programming model, CUDA engenders both **scalar** and
+**collective** software interfaces. Traditional software
+interfaces are *scalar* : a single thread invokes a library routine to perform some
+operation (which may include spawning parallel subtasks). Alternatively, a *collective*
+interface is entered simultaneously by a group of parallel threads to perform
+some cooperative operation.
+
+CUB's collective primitives are not bound to any particular width of parallelism
+or data type. This flexibility makes them:
+
+* **Adaptable** to fit the needs of the enclosing kernel computation
+* **Trivially tunable** to different grain sizes (threads per block, items per thread, etc.)
+
+Thus CUB is *CUDA Unbound*.
+
+An example (block-wide sorting)
+==================================================
+
+The following code snippet presents a CUDA kernel in which each block of ``BLOCK_THREADS`` threads
+will collectively load, sort, and store its own segment of (``BLOCK_THREADS * ITEMS_PER_THREAD``)
+integer keys:
+
+.. code-block:: c++
+
+    #include <cub/block/block_load.cuh>
+    #include <cub/block/block_store.cuh>
+    #include <cub/block/block_radix_sort.cuh>
+
+    template <int BLOCK_THREADS, int ITEMS_PER_THREAD>
+    __global__ void BlockSortKernel(int *d_in, int *d_out)
+    {
+        // Specialize BlockLoad, BlockStore, and BlockRadixSort collective types
+        using BlockLoadT = cub::BlockLoad<
+          int, BLOCK_THREADS, ITEMS_PER_THREAD, cub::BLOCK_LOAD_TRANSPOSE>;
+        using BlockStoreT = cub::BlockStore<
+          int, BLOCK_THREADS, ITEMS_PER_THREAD, cub::BLOCK_STORE_TRANSPOSE>;
+        using BlockRadixSortT = cub::BlockRadixSort<
+          int, BLOCK_THREADS, ITEMS_PER_THREAD>;
+
+        // Allocate type-safe, repurposable shared memory for collectives
+        __shared__ union {
+            typename BlockLoadT::TempStorage       load;
+            typename BlockStoreT::TempStorage      store;
+            typename BlockRadixSortT::TempStorage  sort;
+        } temp_storage;
+
+        // Obtain this block's segment of consecutive keys (blocked across threads)
+        int thread_keys[ITEMS_PER_THREAD];
+        const int block_offset = blockIdx.x * (BLOCK_THREADS * ITEMS_PER_THREAD);
+        BlockLoadT(temp_storage.load).Load(d_in + block_offset, thread_keys);
+
+        __syncthreads();	// Barrier for smem reuse
+
+        // Collectively sort the keys
+        BlockRadixSortT(temp_storage.sort).Sort(thread_keys);
+
+        __syncthreads();	// Barrier for smem reuse
+
+        // Store the sorted segment
+        BlockStoreT(temp_storage.store).Store(d_out + block_offset, thread_keys);
+    }
+
+
+.. code-block:: c++
+
+    // Elsewhere in the host program: parameterize and launch a block-sorting
+    // kernel in which blocks of 128 threads each sort segments of 2048 keys
+    int *d_in = ...;
+    int *d_out = ...;
+    int num_blocks = ...;
+    BlockSortKernel<128, 16><<<num_blocks, 128>>>(d_in, d_out);
+
+
+In this example, threads use ``cub::BlockLoad``, ``cub::BlockRadixSort``, and ``cub::BlockStore``
+to collectively load, sort and store the block's segment of input items.  Because these operations
+are cooperative, each primitive requires an allocation of shared memory for threads to communicate
+through. The typical usage pattern for a CUB collective is:
+
+#. Statically specialize the primitive for the specific problem setting at hand, e.g.,
+   the data type being sorted, the number of threads per block, the number of keys per
+   thread, optional algorithmic alternatives, etc. (CUB primitives are also implicitly
+   specialized by the targeted compilation architecture.)
+#. Allocate (or alias) an instance of the specialized primitive's nested ``TempStorage``
+   type within a shared memory space.
+#. Specify communication details (e.g., the ``TempStorage`` allocation) to
+   construct an instance of the primitive.
+#. Invoke methods on the primitive instance.
+
+In particular, ``cub::BlockRadixSort`` is used to collectively sort the segment of data items
+that have been partitioned across the thread block. To provide coalesced accesses
+to device memory,  we configure the ``cub::BlockLoad`` and ``cub::BlockStore`` primitives
+to access memory using a striped access pattern (where consecutive threads
+simultaneously access consecutive items) and then *transpose* the keys into
+a :ref:`blocked arrangement <flexible-data-arrangement>` of elements across threads.
+To reuse shared memory across all three primitives, the thread block statically
+allocates a union of their ``TempStorage`` types.
+
+
+Why do you need CUB?
+==================================================
+
+Writing, tuning, and maintaining kernel code is perhaps the most challenging,
+time-consuming aspect of CUDA programming.  Kernel software is where
+the complexity of parallelism is expressed. Programmers must reason about
+deadlock, livelock, synchronization, race conditions, shared memory layout,
+plurality of state, granularity, throughput, latency, memory bottlenecks, etc.
+
+With the exception of CUB, however, there are few (if any) software libraries of
+*reusable* kernel primitives. In the CUDA ecosystem, CUB is unique in this regard.
+As a `SIMT <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#hardware-implementation>`_
+library and software abstraction layer, CUB provides:
+
+#. **Simplicity of composition**. CUB enhances programmer productivity by
+   allowing complex parallel operations to be easily sequenced and nested.
+   For example, ``cub::BlockRadixSort`` is constructed from ``cub::BlockExchange`` and
+   ``cub::BlockRadixRank``. The latter is composed of ``cub::BlockScan``
+   which incorporates ``cub::WarpScan``.
+
+   .. figure:: ../img/nested_composition.png
+    :align: center
+
+#. **High performance**. CUB simplifies high-performance program and kernel
+   development by taking care to implement the state-of-the-art in parallel algorithms.
+
+#. **Performance portability**.
+   CUB primitives are specialized to match the diversity of NVIDIA hardware, continuously
+   evolving to accommodate new architecture-specific features and instructions.  And
+   because CUB's device-wide primitives are implemented using flexible block-wide and
+   warp-wide collectives, we are able to performance-tune them to match the processor
+   resources provided by each CUDA processor architecture.
+
+#. **Simplicity of performance tuning**:
+
+   * **Resource utilization**.  CUB primitives allow developers to quickly
+     change grain sizes (threads per block, items per thread, etc.) to best match
+     the processor resources of their target architecture
+   * **Variant tuning**.  Most CUB primitives support alternative algorithmic
+     strategies. For example, ``cub::BlockHistogram`` is parameterized to implement either
+     an atomic-based approach or a sorting-based approach. (The latter provides uniform
+     performance regardless of input distribution.)
+   * **Co-optimization**.  When the enclosing kernel
+     is similarly parameterizable, a tuning configuration can be found that optimally
+     accommodates their combined register and shared memory pressure.
+
+#. **Robustness and durability**. CUB just works. CUB primitives
+   are designed to function properly for arbitrary data types and widths of
+   parallelism (not just for the built-in C++ types or for powers-of-two threads
+   per block).
+
+#. **Reduced maintenance burden**. CUB provides a SIMT software abstraction layer
+   over the diversity of CUDA hardware. With CUB, applications can enjoy
+   performance-portability without intensive and costly rewriting or porting efforts.
+
+#. **A path for language evolution**. CUB primitives are designed
+   to easily accommodate new features in the CUDA programming model, e.g., thread
+   subgroups and named barriers, dynamic shared memory allocators, etc.
+
+
+How do CUB collectives work?
+==================================================
+
+Four programming idioms are central to the design of CUB:
+
+#. :ref:`Generic programming <generic-programming>`. C++ templates provide the flexibility
+   and adaptive code generation needed for CUB primitives to be useful, reusable, and
+   fast in arbitrary kernel settings.
+#. :ref:`Reflective class interfaces <reflective-class-interfaces>`.
+   CUB collectives statically export their their resource requirements
+   (e.g., shared memory size and layout) for a given specialization, which allows compile-time
+   tuning decisions and resource allocation.
+#. :ref:`Flexible data arrangement across threads <flexible-data-arrangement>`.
+   CUB collectives operate on data that is logically partitioned across a group of threads.
+   For most collective operations, efficiency is increased with increased granularity
+   (i.e., items per thread).
+#. :ref:`Static tuning and co-tuning <static-tuning-and-co-tuning>`. Simple constants and static
+   types dictate the granularities and algorithmic alternatives to be employed by CUB collectives.
+   When the enclosing kernel is similarly parameterized, an optimal configuration can be determined
+   that best accommodates the combined behavior and resource consumption of all primitives within
+   the kernel.
+
+
+.. _generic-programming:
+
+Generic programming
+--------------------------------------------------
+
+We use template parameters to specialize CUB primitives for the particular
+problem setting at hand.  Until compile time, CUB primitives are not bound
+to any particular:
+
+* Data type (int, float, double, etc.)
+* Width of parallelism (threads per thread block)
+* Grain size (data items per thread)
+* Underlying processor (special instructions, warp size, rules for bank conflicts, etc.)
+* Tuning configuration (e.g., latency vs. throughput, algorithm selection, etc.)
+
+
+.. _reflective-class-interfaces:
+
+Reflective class interfaces
+--------------------------------------------------
+
+Unlike traditional function-oriented interfaces, CUB exposes its collective
+primitives as templated C++ classes. The resource requirements for a specific
+parameterization are reflectively advertised as members of the class. The
+resources can then be statically or dynamically allocated, aliased
+to global or shared memory, etc.  The following illustrates a CUDA kernel
+fragment performing a collective prefix sum across the threads of a thread block:
+
+.. code-block:: c++
+
+    #include <cub/cub.cuh>
+
+    __global__ void SomeKernelFoo(...)
+    {
+        // Specialize BlockScan for 128 threads on integer types
+      using BlockScan = cub::BlockScan<int, 128>;
+
+      // Allocate shared memory for BlockScan
+      __shared__ typename BlockScan::TempStorage scan_storage;
+
+      ...
+
+      // Obtain a segment of consecutive items that are blocked across threads
+      int thread_data_in[4];
+      int thread_data_out[4];
+      ...
+
+      // Perform an exclusive block-wide prefix sum
+      BlockScan(scan_storage).ExclusiveSum(thread_data_in, thread_data_out);
+
+Furthermore, the CUB interface is designed to separate parameter
+fields by concerns. CUB primitives have three distinct parameter fields:
+
+#. *Static template parameters*. These are constants that will
+   dictate the storage layout and the unrolling of algorithmic steps (e.g.,
+   the input data type and the number of block threads), and are used to specialize the class.
+#. *Constructor parameters*. These are optional parameters regarding
+   inter-thread communication (e.g., storage allocation, thread-identifier mapping,
+   named barriers, etc.), and are orthogonal to the functions exposed by the class.
+#. *Formal method parameters*. These are the operational inputs/outputs
+   for the various functions exposed by the class.
+
+This allows CUB types to easily accommodate new
+programming model features (e.g., named barriers, memory allocators, etc.)
+without incurring a combinatorial growth of interface methods.
+
+
+.. _flexible-data-arrangement:
+
+Flexible data arrangement across threads
+--------------------------------------------------
+
+CUDA kernels are often designed such that each thread block is assigned a
+segment of data items for processing.
+
+.. figure:: ../img/tile.png
+    :align: center
+    :alt: Segment of eight ordered data items
+    :name: fig_tile
+
+    Segment of eight ordered data items
+
+When the tile size equals the thread block size, the
+mapping of data onto threads is straightforward (one datum per thread).
+However, there are often performance advantages for processing more
+than one datum per thread. Increased granularity corresponds to
+decreased communication overhead. For these scenarios, CUB primitives
+will specify which of the following partitioning alternatives they
+accommodate:
+
+
+.. list-table::
+   :class: table-no-stripes
+   :widths: 70 30
+
+   * - **Blocked arrangement**.  The aggregate tile of items is partitioned
+       evenly across threads in "blocked" fashion with *thread*\ :sub:`i`
+       owning the *i*\ :sup:`th` segment of consecutive elements.
+       Blocked arrangements are often desirable for algorithmic benefits (where
+       long sequences of items can be processed sequentially within each thread).
+
+     - .. figure:: ../img/blocked.png
+          :align: center
+          :alt: *Blocked* arrangement across four threads
+          :name: fig_blocked
+
+          *Blocked* arrangement across four threads
+
+          (emphasis on items owned by *thread*\ :sub:`0`)
+
+   * - **Striped arrangement**. The aggregate tile of items is partitioned across threads in "striped"
+       fashion, i.e., the ``ITEMS_PER_THREAD`` items owned by each thread have logical stride
+       ``BLOCK_THREADS`` between them. Striped arrangements are often desirable for data movement through
+       global memory (where
+       `read/write coalescing <https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#coalesced-access-to-global-memory>`_
+       is an important performance consideration).
+     - .. figure:: ../img/striped.png
+          :align: center
+          :alt: *Striped* arrangement across four threads
+          :name: fig_striped
+
+          *Striped* arrangement across four threads
+
+          (emphasis on items owned by *thread*\ :sub:`0`)
+
+The benefits of processing multiple items per thread (a.k.a., *register blocking*,
+*granularity coarsening*, etc.) include:
+
+* Algorithmic efficiency.  Sequential work over multiple items in
+  thread-private registers is cheaper than synchronized, cooperative
+  work through shared memory spaces.
+* Data occupancy.  The number of items that can be resident on-chip in
+  thread-private register storage is often greater than the number of
+  schedulable threads.
+* Instruction-level parallelism.  Multiple items per thread also
+  facilitates greater ILP for improved throughput and utilization.
+
+Finally, ``cub::BlockExchange`` provides operations for converting between blocked
+and striped arrangements.
+
+
+.. _static-tuning-and-co-tuning:
+
+Static tuning and co-tuning
+--------------------------------------------------
+
+This style of flexible interface simplifies performance tuning. Most CUB
+primitives support alternative algorithmic strategies that can be
+statically targeted by a compiler-based or JIT-based autotuner. (For
+example, ``cub::BlockHistogram`` is parameterized to implement either an
+atomic-based approach or a sorting-based approach.) Algorithms are also
+tunable over parameters such as thread count and grain size as well.
+Taken together, each of the CUB algorithms provides a fairly rich tuning
+space.
+
+Whereas conventional libraries are optimized offline and in isolation, CUB
+provides interesting opportunities for whole-program optimization. For
+example, each CUB primitive is typically parameterized by threads-per-block
+and items-per-thread, both of which affect the underlying algorithm's
+efficiency and resource requirements. When the enclosing kernel is similarly
+parameterized, the coupled CUB primitives adjust accordingly. This enables
+autotuners to search for a single configuration that maximizes the performance
+of the entire kernel for a given set of hardware resources.
+
+
+How do I get started using CUB?
+==================================================
+
+CUB is a C++ header-only library, and part of the CUDA Core Compute Libraries (CCCL).
+It ships as part of the CUDA Toolkit and is thus readily available when using the ``nvcc`` compiler.
+Alternatively, consider fetching CCCL directly from GitHub to benefit from the latest improvements.
+
+
+How is CUB different than Thrust and Modern GPU?
+==================================================
+
+
+CUB and Thrust
+--------------------------------------------------
+
+CUB and :ref:`Thrust <thrust-module>` share some
+similarities in that they both provide similar device-wide primitives for CUDA.
+However, they target different abstraction layers for parallel computing.
+Thrust abstractions are agnostic of any particular parallel framework (e.g.,
+CUDA, TBB, OpenMP, sequential  CPU, etc.).  While Thrust has a "backend"
+for CUDA devices, Thrust interfaces themselves are not CUDA-specific and
+do not explicitly expose CUDA-specific details (e.g., ``cudaStream_t`` parameters).
+
+CUB, on the other hand, is slightly lower-level than Thrust. CUB is specific
+to CUDA C++ and its interfaces explicitly accommodate CUDA-specific features.
+Furthermore, CUB is also a library of SIMT collective primitives for block-wide
+and warp-wide kernel programming.
+
+CUB and Thrust are complementary and can be used together. In fact, the CUB
+project arose out of a maintenance need to achieve better performance-portability
+within Thrust by using reusable block-wide primitives to reduce maintenance and
+tuning effort.
+
+
+CUB and Modern GPU
+--------------------------------------------------
+
+CUB and `Modern GPU <https://github.com/moderngpu/moderngpu>`_ also
+share some similarities in that they both implement similar device-wide primitives for CUDA.
+However, they serve different purposes for the CUDA programming community.  MGPU
+is a pedagogical tool for high-performance GPU computing, providing clear and concise
+exemplary code and accompanying commentary.  It serves as an excellent source of
+educational, tutorial, CUDA-by-example material.  The MGPU source code is intended
+to be read and studied, and often favors simplicity at the expense of portability and
+flexibility.
+
+CUB, on the other hand, is a production-quality library whose sources are complicated
+by support for every version of CUDA architecture, and is validated by an extensive
+suite of regression tests.  Although well-documented, the CUB source text is verbose
+and relies heavily on C++ template metaprogramming for situational specialization.
+
+CUB and MGPU are complementary in that MGPU serves as an excellent descriptive source
+for many of the algorithmic techniques used by CUB.
+
+
+Contributors
+==================================================
+
+CUB is developed as open-source as part of the CUDA Core Compute Libraries (CCCL) by NVIDIA.
+
+
+Open Source License
+==================================================
+
+CUB is mostly licensed under the BSD 3-Clause "New" or "Revised" License.
+New files are created under the Apache-2.0 WITH LLVM-exception License.
+See also our `LICENSE <https://github.com/NVIDIA/cccl/blob/main/LICENSE>`_ file.
diff --git a/cccl_upstream/docs/cub/thread_level.rst b/cccl_upstream/docs/cub/thread_level.rst
new file mode 100644
index 00000000..e1ed8ff1
--- /dev/null
+++ b/cccl_upstream/docs/cub/thread_level.rst
@@ -0,0 +1,8 @@
+.. _thread-module:
+
+Thread-level Primitives
+==================================================
+
+CUB thread-level algorithms are specialized for execution by a single thread.
+
+* :cpp:func:`cub::ThreadReduce <cub::ThreadReduce>` computes reduction of a sequence of items
diff --git a/cccl_upstream/docs/cub/tuning.rst b/cccl_upstream/docs/cub/tuning.rst
new file mode 100644
index 00000000..c15f180f
--- /dev/null
+++ b/cccl_upstream/docs/cub/tuning.rst
@@ -0,0 +1,146 @@
+..
+    TODO(bgruber): rename the label below to _cub-tuning when all tuning API exposure PRs have landed
+
+.. _cub-policy-selectors:
+
+Tunings
+================================================================================
+
+Device-scope algorithms in CUB have many knobs that significantly impact performance (without affecting correctness).
+For instance, the number of threads per block and items per thread can be tuned to maximize performance for a given device and data type.
+But also algorithmic choices such as the load or store algorithm, a load vectorization size,
+the used block-level algorithm, or enabling the use of the tensor memory accelerator can be tweaked.
+Most device-scope algorithms in CUB accept a set of such tuning parameters,
+by passing a policy selector, wrapped into :code:`cuda::execution::tune(...)`,
+as part of the environment of a CUB device-scope API.
+In the following, we describe this process in more detail.
+
+
+Policy selectors
+--------------------------------------------------------------------------------
+
+Policy selectors are the mechanism by which CUB's device algorithms and kernels select
+tuning parameters for a given workload and GPU compute capability.
+A policy selector is a stateless callable that maps a :code:`cuda::compute_capability` to a policy struct
+containing the tuning values for that compute capability.
+Each set of CUB algorithms using a common underlying implementation defines a common policy struct,
+e.g. :code:`cub::ReducePolicy`, which must be returned by a policy selector passed to those algorithms.
+
+CUB employs internal default policy selectors providing tunings for known compute capabilities and workloads,
+which are not publicly accessible to users.
+Users can override CUB's policy selector for a given algorithm
+by passing a custom policy selector through the algorithm's environment parameter.
+
+For a description of how policy selectors are used internally in CUB's dispatch layer and kernels
+see the corresponding :ref:`developer documentation <cub-developer-guide-device-scope>`.
+
+Defining a policy selector
+--------------------------------------------------------------------------------
+
+A policy selector is any type with a :code:`__host__`,  :code:`__device__`, :code:`constexpr`, and :code:`const` call operator
+taking a ``cuda::compute_capability`` and returning the algorithm's policy struct:
+
+.. code:: c++
+
+    template <typename T>
+    struct my_reduce_tuning {
+      __host__ __device__ constexpr auto operator()(cuda::compute_capability cc) const -> cub::ReducePolicy {
+        // tuning for Hopper and later
+        if (cc >= cuda::compute_capability(9, 0)) {
+          const auto pass = cub::ReducePassPolicy{
+            .threads_per_block = 512,
+            .items_per_thread = std::max(64 / sizeof(T), 1), // 8 double, 16 float, 32 half_t, ...
+            .vec_size = 2,
+            .reduce_algorithm = cub::BLOCK_REDUCE_WARP_REDUCTIONS,
+            .load_modifier = cub::LOAD_DEFAULT
+          };
+          return { .multi_tile = pass, .single_tile = pass };
+        }
+        // fallback for older GPUs
+        const auto pass = cub::ReducePassPolicy{
+          .threads_per_block = 256,
+          .items_per_thread = 12,
+          .vec_size = 1,
+          .reduce_algorithm = cub::BLOCK_REDUCE_WARP_REDUCTIONS,
+          .load_modifier = cub::LOAD_DEFAULT
+        };
+        return { .multi_tile = pass, .single_tile = pass };
+      }
+    };
+
+.. warning::
+
+   The policy selector must be stateless (:code:`std::is_empty_v<T>` must be :code:`true`)
+   since only its type will be passed to a kernel — any captured state would be silently
+   lost. Policy selectors are also freely constructed and copied where needed, so they
+   must also be default constructible and copyable (:code:`std::semiregular<T>` must be
+   :code:`true`).
+
+It can be a class template, but then only a full specialization can be passed to CUB,
+e.g., :code:`my_reduce_tuning<float>`.
+The implementation can use branches and helper functions arbitrarily,
+as long as they can be evaluated at compile-time.
+The returned tuning policy can, and probably should, contain different values for different compute capabilities or workloads.
+
+Each CUB algorithm with an environment parameter searches the environment for a policy selector returning a matching policy struct.
+If one is found, the policy selector will be used to determine the tuning values for host-side dispatch and kernel compilation.
+Each CUB algorithm documents the policy struct to which it responds.
+If CUB does not find a matching policy selector in the environment, it falls back to its internal default policy selector.
+
+Multiple policy selectors returning different policy structs can be passed as part of the same environment,
+and each algorithm will pick the one with the matching policy struct.
+This is useful if the same environment is reused across several algorithm calls.
+
+The policy structs themselves are simple semiregular aggregates.
+They support C++20 designated initializers (i.e., the syntax :code:`{ .threads_per_block = 512, ... }`),
+comparison for (in-)equality, and serialization using :code:`operator<<`.
+They may occasionally contain member functions that compute derived values from the contained tuning values.
+All policy structs are public types and will evolve in a non-breaking way, at least during minor releases,
+by only adding new data members at the end of the struct.
+
+
+Passing a policy selector to CUB device-scope algorithms
+--------------------------------------------------------------------------------
+
+Custom policy selectors are passed to CUB algorithms via the environment argument.
+They first need to be wrapped by passing them to :code:`cuda::execution::tune(...)`:
+
+.. code:: c++
+
+    cub::DeviceReduce::Reduce(
+      d_in, d_out, num_items, op, init,
+      cuda::execution::tune(my_reduce_tuning<int>{}));
+
+Multiple tunings for different algorithms can be combined in a single environment:
+
+.. code:: c++
+
+    auto env = cuda::execution::tune(
+      my_reduce_tuning<int>{},
+      my_scan_tuning{}
+    );
+    cub::DeviceReduce::Reduce(d_in, d_out, num_items, op, init, env);
+    cub::DeviceScan::ExclusiveSum(d_in, d_out, num_items, env);
+
+An environment can carry further properties like a stream or a memory resource.
+Policy selectors can simply be added to those:
+
+.. code:: c++
+
+    auto env = cuda::std::execution::env{
+      stream_ref,
+      resource,
+      cuda::execution::tune(
+        my_reduce_tuning<int>{}, my_scan_tuning{})
+    };
+    cub::DeviceReduce::Reduce(d_in, d_out, num_items, op, init, env);
+
+    // alternatively, if we want to extend an env `other_env`
+    auto env = cuda::std::execution::env{
+      other_env,
+      cuda::execution::tune(my_scan_tuning{})
+    };
+    cub::DeviceReduce::Reduce(d_in, d_out, num_items, op, init, env);
+
+CUB's benchmarks also make heavy use of policy selectors for tuning.
+For more details on authoring benchmarks and their policy selectors for automatic tuning, see :ref:`cub-tuning-infra`.
diff --git a/cccl_upstream/docs/cub/tuning_infra.rst b/cccl_upstream/docs/cub/tuning_infra.rst
new file mode 100644
index 00000000..0c13ba12
--- /dev/null
+++ b/cccl_upstream/docs/cub/tuning_infra.rst
@@ -0,0 +1,629 @@
+.. _cub-tuning-infra:
+
+Automated Tuning Infrastructure
+================================================================================
+
+This page describes tuning infrastructure to automatically tune CUB device-scope algorithms for performance.
+It provides a set of tools facilitating the process of selecting optimal tuning parameters for a given device and data type.
+
+Terminology
+--------------------------------------------------------------------------------
+
+*We omit the word "tuning" but assume it in the definitions for all terms below,
+so those terms may mean something else in a more generic context.*
+
+The following three terms are fundamental to understanding the essence of CUB tuning:
+
+* **compile-time (ct) workload**: a workload that can be recognized only at compile time.
+
+*e.g. the combination of key type and offset type,* :code:`int16_t` *and* :code:`int32_t`
+
+* **runtime (rt) workload**: a workload that can be recognized only at runtime.
+
+*e.g. the number of input elements*
+
+* **tuning parameter (or parameter)**: a parameter that can be tuned to maximize performance for a given device and data type.
+
+*e.g. number of threads per block, items per thread*
+
+Algorithms are tuned for different workloads. These workloads are defined as subspaces by NVBench via the benchmarks' axis.
+For instance, radix sort can be tuned for different key types, different number of keys, and different distributions of keys. The tuning process is summarized in
+the following statement:
+
+.. raw:: html
+
+   <div style="display: flex; justify-content: center; align-items: center; height: 10; text-align: center; font-size: 1.5em; color: #76B900;">
+       "For each Compile-time Workload, we search for the best tuning parameters"
+   </div>
+
+
+More specifically, the tuning infrastructure optimizes algorithms for specific compile-time workloads,
+aggregating results across all runtime workloads.
+It searches through a space of parameters to find the combination for a given compile-time workload with the highest score.
+
+--------
+
+Following is supplemental terminology that will be used throughout the rest of this tuning guide:
+
+.. e.g. :math:`threads\_per\_block=128`
+
+* **Parameter Space**: the set of all possible values for a given Tuning Parameter.
+
+*It is specific to the algorithm*. For example the parameter space for the number of threads per block can be :math:`\{32, 64, 96, 128, \dots, 1024\}` for radix sort, but :math:`\{32, 64, 128, 256, 512\}` for merge sort.
+
+* **Search Space**: Cartesian product of all the Parameter Spaces of a single algorithm.
+
+For instance, the Search Space for an algorithm with tunable items per thread and threads per block might look like :math:`\{(ipt \times tpb) | ipt \in \{1, \dots, 25\} \text{and} tpb \in \{32, 64, 96, 128, \dots, 1024\}\}`.
+
+* **Variant** - a point in the corresponding Search Space.
+
+* **Base** - the variant that CUB uses by default.
+
+* **Score** - a single number representing the performance for a given compile-time workload across all runtime workloads. For instance, a weighted-sum of speedups of a given variant compared to its base for all runtime workloads is a score.
+
+.. * **Search** - a process consisting of covering all variants for all compile-time workloads to find a variant with maximal score.
+..  ^^^ @giannis: again we do not want to scare a first time user with too many terms. "search" is both evident and can also be explained with an introductory sentence in the "Search Process" chapter ^^^
+
+.. _cub-tuning-infra-authoring-benchmarks:
+
+Authoring Benchmarks
+--------------------------------------------------------------------------------
+
+CUB benchmarks are split into multiple files based on the algorithm they are testing
+and potentially further into compile-time flavors that are tuned for individually
+(e.g. sorting only keys vs. key-value pairs, or reducing using sum vs. using min).
+The name of the directory represents the name of the algorithm.
+The filename corresponds on the flavor.
+For instance, the benchmark :code:`benchmarks/bench/radix_sort/keys.cu` tests the radix sort implementation sorting only keys.
+The executable file name is going to be transformed into :code:`cub.bench.radix_sort.keys.*`,
+which is the benchmark name reported by the infrastructure.
+
++++++++++++++++
+Headers
++++++++++++++++
+
+**Benchmarks are based on NVBench.**
+You start writing a benchmark by including :code:`nvbench_helper.cuh`. This contains all
+necessary includes and definitions.
+
+.. code:: c++
+
+  #include <nvbench_helper.cuh>
+
+The next step is to define a search space. The search space is represented by a number of C++ comments.
+The format consists of the :code:`%RANGE%` keyword, the parameter macro, the parameter abbreviation, and its range of values.
+The range is represented by three numbers: :code:`start:end:step`.
+Start and end are included.
+For instance, the following code defines a search space for two parameters, the number of threads per block and items per thread.
+
+.. code:: c++
+
+  // %RANGE% TUNE_ITEMS_PER_THREAD ipt 7:24:1
+  // %RANGE% TUNE_THREADS_PER_BLOCK tpb 128:1024:32
+
+Next, you need to define a benchmark function. The function accepts :code:`nvbench::state &state` and
+a :code:`nvbench::type_list`. For more details on the benchmark signature, take a look at the
+`NVBench documentation <https://github.com/NVIDIA/nvbench>`_.
+
+.. code:: c++
+
+  template <typename T, typename OffsetT>
+  void algname(nvbench::state &state, nvbench::type_list<T, OffsetT>)
+  {...}
+
+Before proceeding further with the benchmark authoring
+it is imperative to understand policy selectors and how they provide tuning values.
+
++++++++++++++++
+Policy Selector
++++++++++++++++
+
+The tuning value of all CUB device algorithms can be customized
+by providing a custom policy selector to the environment argument of a CUB API call.
+See :ref:`cub-policy-selectors` for a full explanation.
+
+The tuning infrastructure will use the :code:`TUNE_BASE` macro to distinguish between compiling the base version (i.e. baseline) of a benchmark
+and compiling a variant for a given set of tuning parameters.
+When base is used, no custom policy selector is specified, so CUB's default tunings are used.
+If :code:`TUNE_BASE` is not defined, we define a custom policy selector
+that specifies the values for the current variant (i.e. the current set of tuning parameters),
+which are derived from the parameter macros defined in the :code:`%RANGE%` comments, which define the search space.
+This custom policy selector is passed to :code:`cuda::execution::tune`,
+and the returned value is included in the environment passed to the CUB API.
+
+The following code is included in the benchmark for the policy selector to be enabled
+and the parameters to have effect in execution:
+
+.. code:: c++
+
+  #if !TUNE_BASE
+  template <typename T>
+  struct policy_selector {
+    _CCCL_HOST_DEVICE_API constexpr auto operator()(cuda::compute_capability /*cc*/) const -> cub::ReducePolicy {
+      const auto [items, threads] = cub::detail::scale_mem_bound(
+                                     TUNE_THREADS_PER_BLOCK, TUNE_ITEMS_PER_THREAD, sizeof(T));
+      const auto pass = cub::ReducePassPolicy{
+        .threads_per_block = threads,
+        .items_per_thread = items,
+        .vec_size = 1 << TUNE_ITEMS_PER_VEC_LOAD_POW2,
+        .reduce_algorithm = cub::BLOCK_REDUCE_WARP_REDUCTIONS,
+        .load_modifier = cub::LOAD_DEFAULT
+      };
+      return { .multi_tile = pass, .single_tile = pass};
+    }
+  };
+  #endif
+
+The custom policy selector returns a fixed policy regardless of the compute capability,
+since tuning is usually done on a specific GPU and compiling only for that GPU's compute capability.
+The policy selector uses all tuning parameters from the search space to form the policy used by CUB.
+
+
++++++++++
+Main Body
++++++++++
+
+The :code:`state` passed into the benchmark function allows access to runtime workload axes,
+for example the number of elements to process.
+*When creating containers for the input avoid to initialize data yourself.
+Instead, use the* :code:`gen` *function,
+which will fill the input vector with random data on GPU with no compile-time overhead.*
+
+.. code:: c++
+
+    const auto elements = static_cast<std::size_t>(state.get_int64("Elements{io}"));
+    thrust::device_vector<T> in(elements);
+    thrust::device_vector<T> out(1);
+
+    gen(seed_t{}, in);
+
+In addition to the benchmark runtime, NVBench can also report information on the achieved memory bandwidth.
+For this, you can optionally provide information on the memory reads and writes of the algorithm to the :code:`state`:
+
+.. code:: c++
+
+    state.add_element_count(elements);
+    state.add_global_memory_reads<T>(elements, "Size");
+    state.add_global_memory_writes<T>(1);
+
+Finally, the actual benchmark region then calls the public CUB API with an appropriate environment.
+Temporary storage allocation is handled automatically by a caching allocator provided through the environment.
+The CUDA stream is provided by the :code:`nvbench::launch` parameter of the benchmark lambda.
+When tuning (:code:`TUNE_BASE` is not defined),
+an instance of the custom policy selector is wrapped by :code:`cuda::execution::tune` and passed to the environment as well.
+
+.. code:: c++
+
+    caching_allocator_t alloc;
+    state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch,
+               [&](nvbench::launch &launch) {
+      auto env = cub_bench_env(
+        alloc,
+        launch
+  #if !TUNE_BASE
+        , cuda::execution::tune(policy_selector<T>{})
+  #endif // !TUNE_BASE
+      );
+      cub::DeviceReduce::Reduce(d_in, d_out, elements, op_t{}, init, env);
+    });
+
+This concludes defining the benchmark function.
+Now we need to tell NVBench about it.
+
+++++++++++++++++++
+NVBench Attributes
+++++++++++++++++++
+
+.. code:: c++
+
+  NVBENCH_BENCH_TYPES(algname, NVBENCH_TYPE_AXES(all_types, offset_types))
+    .set_name("base")
+    .set_type_axes_names({"T{ct}", "OffsetT{ct}"})
+    .add_int64_power_of_two_axis("Elements{io}", nvbench::range(16, 28, 4));
+
+:code:`NVBENCH_BENCH_TYPES` registers the benchmark as one with multiple compile-time workloads,
+which are defined by the Cartesian product of the type lists in :code:`NVBENCH_TYPE_AXES`.
+:code:`set_name(...)` sets the name of the benchmark.
+Only alphabetical characters, numbers and underscores are allowed in the benchmark name.
+
+Furthermore, compile-time axes should be suffixed with :code:`{ct}`. The runtime axes might be optionally annotated
+as :code:`{io}` which stands for importance-ordered. *This will tell the tuning infrastructure that
+the later values on the axis are more important. If the axis is not annotated, each value will be
+treated as equally important.*
+
+When you define a type axis annotated with :code:`{ct}`, you should consider optimizing
+the build time. Many variants are going to be build, but the search is considering one compile-time
+use case at a time. This means that if you have many types to tune for, you'll end up having
+many template specializations that you don't need. To avoid this, for each compile time axis, the tuning framework will predefine
+a `TUNE_AxisName` macro with the type that's currently being tuned. For instance, if you
+have the type axes :code:`T{ct}` and :code:`OffsetT` (as shown above), you can use the following
+pattern to narrow down the types you compile for:
+
+.. code:: c++
+
+  #ifdef TUNE_T
+  using all_types = nvbench::type_list<TUNE_T>;
+  #else
+  using all_types = nvbench::type_list<char, short, int, long, ...>;
+  #endif
+
+  #ifdef TUNE_OffsetT
+  using offset_types = nvbench::type_list<TUNE_OffsetT>;
+  #else
+  using offset_types = nvbench::type_list<int32_t, int64_t>;
+  #endif
+
+
+This logic is already implemented if you use any of the following predefined type lists:
+
+.. list-table:: Predefined type lists
+   :header-rows: 1
+
+   * - Axis name
+     - C++ identifier
+     - Included types
+   * - :code:`T{ct}`
+     - :code:`integral_types`
+     - :code:`int8_t, int16_t, int32_t, int64_t`
+   * - :code:`T{ct}`
+     - :code:`fundamental_types`
+     - :code:`integral_types` and :code:`int128_t, float, double`
+   * - :code:`T{ct}`
+     - :code:`all_types`
+     - :code:`fundamental_types` and :code:`complex`
+   * - :code:`OffsetT{ct}`
+     - :code:`offset_types`
+     - :code:`int32_t, int64_t`
+
+
+You are free to define your own axis names and use the logic above for them (see the sort pairs example).
+
+A single benchmark file can define multiple benchmarks (multiple benchmark functions registered with :code:`NVBENCH_BENCH_TYPES`).
+All benchmarks in a single file must share the same compile-time axes.
+**The tuning infrastructure will run all benchmarks in a single file together for the same compile-time workload
+and compute a common score across all benchmarks and runtime workloads.
+Unless a benchmark axis is importance-ordered, each sample contributes equally to the score.**
+This is useful to tune an algorithm for multiple runtime use cases at once,
+that we don't intend to provide separate tuning policies for.
+Also, a large space of runtime workloads can be segmented this way,
+e.g. by splitting the benchmark entry point and supplying a few low and a few high values for a runtime axis:
+
+.. code:: c++
+
+  NVBENCH_BENCH_TYPES(algname, NVBENCH_TYPE_AXES(all_types, offset_types))
+    .set_name("small")
+    ...
+    .add_int64_power_of_two_axis("SegmentSize", nvbench::range(0, 3, 1)); // tests sizes 2^0, 2^1, 2^2, 2^3
+
+  NVBENCH_BENCH_TYPES(algname, NVBENCH_TYPE_AXES(all_types, offset_types))
+    .set_name("large")
+    ...
+    .add_int64_power_of_two_axis("SegmentSize", nvbench::range(12, 18, 2)); // tests sizes 2^12, 2^14, 2^16, 2^18
+
+
+Search Process
+--------------------------------------------------------------------------------
+
+During the Search Process we are covering all variants for all compile-time workloads to find a variant with a maximum (at least locally) score.
+
+To get started with tuning, you need to configure CMake.
+You can use the following command:
+
+.. code:: bash
+
+  $ mkdir build
+  $ cd build
+  $ cmake .. --preset=cub-tune
+
+You can then run the tuning search for a specific algorithm and compile-time workload. We use a CCCL internal script for that:
+
+.. code:: bash
+
+  $ ../benchmarks/scripts/search.py -R '.*merge_sort.*pairs' -a 'KeyT{ct}=I128' -a 'Elements{io}[pow2]=28'
+  cub.bench.merge_sort.pairs.trp_0.ld_1.ipt_13.tpb_6 0.6805093269929858
+  cub.bench.merge_sort.pairs.trp_0.ld_1.ipt_11.tpb_10 1.0774560502969677
+  ...
+
+This will search the space of merge sort for key-value pairs, for the key type :code:`int128_t` on :code:`2^28` elements.
+The :code:`-R` and :code:`-a` options are optional. **If not specified, all benchmarks are going to be tuned.**
+The :code:`-R` option can select multiple benchmarks using a regular expression.
+For the axis option :code:`-a`, you can also specify a range of values like :code:`-a 'KeyT{ct}=[I32,I64]'`.
+Any axis values not supported by a selected benchmark will be ignored.
+The first variant :code:`cub.bench.merge_sort.pairs.trp_0.ld_1.ipt_13.tpb_6` has a score <1 and is thus generally slower than the baseline,
+whereas the second variant :code:`cub.bench.merge_sort.pairs.trp_0.ld_1.ipt_11.tpb_10` has a score of >1 and is thus an improvement over the baseline.
+
+.. warning::
+  Notice there is currently a limitation in :code:`search.py`
+  which will only execute runs for the first axis value for each axis
+  (independently of whether the axis is specified on the command line or not).
+  Tuning for multiple axis values requires multiple runs of :code:`search.py`.
+  Please see `this issue <https://github.com/NVIDIA/cccl/issues/2267>`_ for more information.
+
+**Benchmarks do not need to be built a priori.** The tuning framework will handle building the benchmarks (base and variants) and running them by itself.
+It will keep track of the build time for base and variants.
+Sometimes, a tuning variant may lead the compiler to hang or take exceptionally long to compile.
+To keep the tuning process going, if the build time of a variant exceeds a threshold, the build is cancelled.
+The same applies to benchmarks running for too long.
+
+To get quick feedback on what benchmarks are selected and how big the search space is,
+you can add the :code:`-l` option:
+
+.. code:: bash
+
+  $ ../benchmarks/scripts/search.py -R '.*merge_sort.*pairs' -a 'KeyT{ct}=I128' -a 'Elements{io}[pow2]=28' -l
+  ctk:  12.6.85
+  cccl:  v2.7.0
+  ### Benchmarks
+    * `cub.bench.merge_sort.pairs`: 540 variants:
+      * `trp`: (0, 2, 1)
+      * `ld`: (0, 3, 1)
+      * `ipt`: (7, 25, 1)
+      * `tpb`: (6, 11, 1)
+
+It will list all selected benchmarks as well as the total number of variants (the magnitude of the search space)
+as a result of the Cartesian product of all its tuning parameter spaces.
+
+The tuning infrastructure stores the results in an SQLite database called :code:`cccl_meta_bench.db` in the build directory.
+This database persists across tuning runs.
+If you interrupt the benchmark script and then launch it again, only missing benchmark variants will be run.
+
+Tuning on multiple GPUs
+--------------------------------------------------------------------------------
+
+Because the search process computes scores by comparing the performance of a variant to the baseline,
+it has to store the baseline result in the tuning database.
+The baseline is specific to the physical GPU on which it was obtained.
+Therefore, a single tuning database should not be used to run the tuning search on two different GPUs, even of the same architecture.
+Similarly, you should also not interrupt the search and resume it on a different GPU.
+Be careful when sharing build directories over network file systems.
+Check whether a build directory already contains a :code:`cccl_meta_bench.db` from a previous run before starting a new search.
+
+..
+    TODO(bgruber): I don't yet understand whether we can tune a single variant on multiple GPUs.
+    I think this is possible, but would it then create a database per GPU (because 1 baseline per GPU)?
+    Does search.py do this automatically, or do I need to pass a flag? Or does this only work with our "internal extensions"?
+
+Because the search space can be separated based on different axis values,
+a tuning search can be run on multiple GPUs in parallel, even across multiple physical machines (e.g., on a cluster).
+To do this, :code:`search.py` is invoked in parallel, one invocation/process per GPU,
+with different axis values specified for each invocation.
+A dedicated tuning database will be created per physical GPU.
+If a shared filesystem is in use, make sure that :code:`search.py` is run from different directories,
+so the :code:`cccl_meta_bench.db` files are placed into distinct paths.
+
+It is recommended to drive a multi-GPU/multi-node search process from a script,
+iterating the axis values and invoking :code:`search.py` for each variant.
+This integrates nicely with workload managers on clusters, which allow submitting batch jobs.
+In such a scenario, it is recommended to submit a job per variant.
+
+After tuning on multiple GPUs, the results are available in multiple tuning databases, which can be analyzed together.
+
+
+Analyzing the results
+--------------------------------------------------------------------------------
+
+The result of the search is stored in one or more :code:`cccl_meta_bench.db` files. To analyze the
+result you can use the :code:`analyze.py` script.
+The :code:`--coverage` flag will show the amount of variants that were covered per compile-time workload:
+
+.. code:: bash
+
+  $ ../benchmarks/scripts/analyze.py --coverage
+    cub.bench.radix_sort.keys[T{ct}=I8, OffsetT{ct}=I32] coverage: 167 / 522 (31.9923%)
+    cub.bench.radix_sort.keys[T{ct}=I8, OffsetT{ct}=I64] coverage: 152 / 522 (29.1188%)
+
+The :code:`--top N` flag will list the best :code:`N` variants for each compile-time workload:
+
+.. code:: bash
+
+  $ ../benchmarks/scripts/analyze.py --top=5
+    cub.bench.radix_sort.keys[T{ct}=I8, OffsetT{ct}=I32]:
+              variant     score      mins     means      maxs
+    97  ipt_19.tpb_512  1.141015  1.039052  1.243448  1.679558
+    84  ipt_18.tpb_512  1.136463  1.030434  1.245825  1.668038
+    68  ipt_17.tpb_512  1.132696  1.020470  1.250665  1.688889
+    41  ipt_15.tpb_576  1.124077  1.011560  1.245011  1.722379
+    52  ipt_16.tpb_512  1.121044  0.995238  1.252378  1.717514
+    cub.bench.radix_sort.keys[T{ct}=I8, OffsetT{ct}=I64]:
+              variant     score      mins     means      maxs
+    71  ipt_19.tpb_512  1.250941  1.155738  1.321665  1.647868
+    86  ipt_20.tpb_512  1.250840  1.128940  1.308591  1.612382
+    55  ipt_17.tpb_512  1.244399  1.152033  1.327424  1.692091
+    98  ipt_21.tpb_448  1.231045  1.152798  1.298332  1.621110
+    85  ipt_20.tpb_480  1.229382  1.135447  1.294937  1.631225
+
+The name of the variant contains the short parameter names and values used for the variant.
+For each variant, a score is reported. The base has a score of 1.0, so each score higher than 1.0 is an improvement over the base.
+However, because a single variant contains multiple runtime workloads, also the minimum, mean, maximum score is reported.
+If all those three values are larger than 1.0, the variant is strictly better than the base.
+If only the mean or max are larger than 1.0, the variant may perform better in most runtime workloads, but regress in others.
+This information can be used to change the existing tuning policies in CUB. A detailed explanation of the output is presented
+in the following image:
+
+.. image:: ../images/top_results_expl.png
+
+By default, :code:`analyze.py` will look for a file named :code:`cccl_meta_bench.db` in the current directory.
+If the tuning results are available in multiple databases, e.g., after tuning on multiple GPUs,
+glob expressions matching multiple databases, or just multiple file paths, can be passed as arguments as well:
+
+.. code:: bash
+
+  $ ../benchmarks/scripts/analyze.py --top=5 <path-to-databases>/*.db
+
+In case the tuning database(s) store(s) results for several different benchmarks,
+the analysis can again be restricted using a regular expression via the :code:`-R` option:
+
+.. code:: bash
+
+  $ ../benchmarks/scripts/analyze.py -R=".*radix_sort.keys.*"  --top=5 <path-to-databases>/*.db
+
+
+Variant plots
+--------------------------------------------------------------------------------
+
+The reported score for a tuning aggregates the performance across all runtime workloads.
+Furthermore, NVBench collects and aggregates multiple samples for a single compile and runtime workload.
+So, even though the min, mean and max score are reported for a variant,
+it may be necessary to compare the distributions of raw speedups between the baseline and a variant across all runtime workloads and samples.
+This is achieved using variant plots.
+For more background information on this subject, we refer the reader to `this article <https://aakinshin.net/posts/shift-and-ratio-functions/>`_.
+
+A variant plot can be generated for one or more variants using the :code:`--variants-ratio=` option and specifying the specific variant to plot.
+For example:
+
+.. code:: bash
+
+  $ ../benchmarks/scripts/analyze.py -R=".*radix_sort.keys.*" --variants-ratio='ipt_18.tpb_288' <path-to-databases>/*.db
+
+May display a matrix of variant plots like:
+
+.. image:: ../images/variant_plot.png
+
+In the image above we see twelve diagrams for the Cartesian product of the :code:`Entropy` (horizontally) and :code:`Elements{io}` (vertically) runtime axes.
+The compile-time axes are fixed for one matrix of variant plots.
+Across each variant plot's x-axis, the speedup over the baseline (y-axis) is represented.
+The baseline is shown as a straight horizontal red line at 1.
+The found tuning thus results in a slowdown for :code:`Elements{io}` 2^16 and 2^20 (orange line below red baseline),
+but a speedup for 2^24 and 2^28 (orange line above red baseline).
+In general, bigger axis values for plots for importance-ordered axes, like :code:`Elements{io}`,
+should be prioritized in evaluating a given tuning, because GPUs are optimized for large problem sizes.
+However, while the almost 4% slowdown for 2^16 elements at entropy 0.544 may be bearable,
+a close to 7% slowdown for 2^20 elements at entropy 1 is probably too large to accept this tuning,
+despite the solid 3.5-8% speedup for larger element counts.
+
+The shown ratios are generated by fitting an equal amount of quantiles into the samples of the baseline and the variant,
+and then showing the quotient for each corresponding quantile from baseline and variant.
+For background information on the quantile-respectful density estimation,
+we refer the reader to this `article <https://aakinshin.net/posts/qrde-hd>`_.
+By default, a quantile corresponds to a percentile, and thus a ratio plot contains 100 data points
+expressing the speedup of the slowest 1% in the variant over the slowest 1% in the baseline (left),
+then the second slowest 1%, etc., until the speedup of the fastest 1% in the variant over the fastest 1% in the baseline (right).
+
+The detailed analysis via variant plots is needed,
+because a single aggregated score cannot represent the distribution of samples obtained from highly concurrent algorithms, such as those in CUB.
+Even though NVBench reruns a benchmark many times to gain statistical confidence in the result,
+the runtime of a CUB algorithm does not necessarily follow a normal distribution.
+For example, the concurrent nature of some algorithms may result in bimodal or even more complex distributions,
+as a consequence of how the hardware schedules and executes threads.
+Also, the kind of distribution may be different between baseline and variant.
+For all these reasons, comparing the distribution of samples is the only reliable way to determine,
+whether a tuning provides a consistent speedup for all runtime workloads.
+
+
+Creating and extending tuning policies
+--------------------------------------------------------------------------------
+
+Once a suitable tuning result has been selected, we have to translate it into C++ code that will be picked up by CUB.
+The tuning variant name shown by :code:`analyze.py` gives us all the information on the selected tuning values.
+Here is an example:
+
+.. code:: bash
+
+  $ ../benchmarks/scripts/analyze.py --top=1
+    cub.bench.radix_sort.keys[T{ct}=I8, OffsetT{ct}=I64]:
+              variant     score      mins     means      maxs
+    71  ipt_19.tpb_512  1.250941  1.155738  1.321665  1.647868
+
+Assume we have determined this tuning to be the best one for sorting I8 keys using radix_sort using I64 offsets.
+The ``variant`` can be decoded using the ``// %RANGE%`` comments in the C++ source code of the benchmark,
+since the names of the reported parameters in the variant are derived from these:
+
+.. code::  c++
+
+    // %RANGE% TUNE_ITEMS_PER_THREAD ipt 7:24:1
+    // %RANGE% TUNE_THREADS_PER_BLOCK tpb 128:1024:32
+
+The variant ``ipt_19.tpb_512``, which stands for 19 items per thread (``ipt``) and 512 threads per block (``tpb``),
+was thus compiled with ``-DTUNE_ITEMS_PER_THREAD=19 -DTUNE_THREADS_PER_BLOCK=512``.
+The meaning of these values is specific to the benchmark definition,
+and we have to check the benchmark's source code for how they are applied.
+Equally named tuning parameters may not translate to different benchmarks (please double check).
+
+As a user of CUB, such a new set of tuning parameters (i.e. a variant) can then be used to define a policy selector,
+which is passed to the public CUB API through the environment,
+as :ref:`sketched above <cub-tuning-infra-authoring-benchmarks>`:
+
+.. code:: c++
+
+  struct policy_selector {
+    _CCCL_HOST_DEVICE_API constexpr auto operator()(cuda::compute_capability /*cc*/) const -> cub::AlgorithmPolicy {
+      return {
+        .threads_per_block = 512,
+        .items_per_thread = 19,
+        ...
+      };
+    }
+  };
+
+The default tunings defined inside CUB's source use the same infrastructure
+but should only be changed and extended by the CCCL maintainers.
+All default tunings are found in the :code:`cub/device/dispatch/tuning/tuning_*.cuh` headers, organized by algorithm.
+CUB's policy selectors are highly parameterized on type information and traits of the input arguments to CUB algorithms
+(like accumulator type, offset size, and operation kind),
+which they turn into a policy for a given compute capability.
+
+The way tuning values are selected is different for each CUB algorithm and requires studying the corresponding code.
+The general principles of policy selectors and tunings are documented :ref:`here <cub-policy-selectors>`.
+For example, signed and unsigned integers of the same size are often represented by the same tuning.
+In general, variants for which the algorithmic behavior is expected to be the same
+(same arithmetic intensity, no special instructions for one of the data types, same amount of bytes to load/store, etc.)
+are covered by the same tuning.
+
+When a better variant has been found and CUB already has a tuning for this variant,
+the tuning parameter values can simply be updated in the corresponding CUB tuning header.
+This is usually the case when a CUB algorithm has been reengineered and shows different performance characteristics,
+or more tuning parameters are exposed (e.g., a new load algorithm is available).
+For example, an existing tuning selection function may contain code like:
+
+.. code:: c++
+
+    constexpr auto get_sm90_tuning(type_t accum_t, op_kind_t op, int offset_size, int accum_size) {
+      if (op == op_kind_t::plus && offset_size == 4 && accum_size == 4)
+        return { .threads_per_block = 256, .items_per_thread = 14 }; // tuning variant proposes: 512 and 19
+      ...
+    }
+
+Since we have found that 512 threads per block and 19 items per thread are better, we can update the value in place.
+
+A different case is when we tune beyond what's currently supported by CUB's existing tunings.
+This may be because we tune for a new GPU architecture,
+in which case a new branch based on the :code:`cuda::compute_capability` passed to the :code:`policy_selector::operator()`
+should be introduced, handling this new GPU's compute capability.
+Or we tune for new key, value or offset types, etc.,
+in which case the existing tuning functions may need additional branches.
+There is no general rule on how this extension is done, though.
+The implementation may be different for each CUB algorithm.
+
+In the seldom case, that no variant outperforms the baseline,
+it must be ensured that any newly added logic correctly falls back to the old tuning values.
+There is again no general rule on how this is implemented.
+
+
+Verification
+--------------------------------------------------------------------------------
+
+Once we have selected tunings and implemented them in CUB, we need to verify them.
+This process consists of two steps.
+
+Firstly, we need to ensure that adding new tunings and policies did not break existing tunings.
+This is most relevant when tunings for new compute capabilities have been added.
+To verify this, compile the corresponding benchmarks for the previous compute capabilities
+(excluding the new tunings) before and after modifying any tunings,
+and compare the generated SASS :code:(`cuobjdump -sass`).
+It should not have changed.
+
+Secondly, we must benchmark and compare the performance of the tuned algorithm before and after the tunings have been applied.
+This extra step is needed, because the score shown during the tuning analysis is just an aggregated result.
+Individual benchmarks may still have regressed for some compile-time workloads.
+Fortunately, this is no different than :ref:`running <cub-benchmarking-running>` the corresponding CUB benchmark with and without the changes,
+and :ref:`comparing <cub-benchmarking-comparing>` the resulting JSON files.
+Such a diff should be supplied to any request to change CUB tunings.
+
+If verification fails for some compile-time workloads (there are regressions), there are two options:
+
+1. Discard the tuning entirely and ensure the tuning selection falls back to the baseline tuning.
+2. Narrow down the tuning template specialization to only apply to the workloads where it improves performance,
+   and fallback where it regressed.
+
+The latter is more complex and may not be justified, if the improvements are small or the use case too narrow.
+Use your judgement. Good luck!
diff --git a/cccl_upstream/docs/cub/warp_wide.rst b/cccl_upstream/docs/cub/warp_wide.rst
new file mode 100644
index 00000000..bc09b9b3
--- /dev/null
+++ b/cccl_upstream/docs/cub/warp_wide.rst
@@ -0,0 +1,22 @@
+.. _warp-module:
+
+Warp-Wide "Collective" Primitives
+==================================================
+
+.. toctree::
+   :glob:
+   :hidden:
+   :maxdepth: 2
+
+   api/warp
+
+CUB warp-level algorithms are specialized for execution by threads in the same CUDA warp.
+These algorithms may only be invoked by ``1 <= n <= 32`` *consecutive* threads in the same warp:
+
+* :cpp:struct:`cub::WarpExchange` rearranges data partitioned across a CUDA warp
+* :cpp:class:`cub::WarpLoad` loads a linear segment of items from memory into a CUDA warp
+* :cpp:class:`cub::WarpMergeSort` sorts items partitioned across a CUDA warp
+* :cpp:struct:`cub::WarpReduce` computes reduction of items partitioned across a CUDA warp
+* :cpp:struct:`cub::WarpReduceBatched` computes reduction of multiple batches of items partitioned across a CUDA warp
+* :cpp:struct:`cub::WarpScan` computes a prefix scan of items partitioned across a CUDA warp
+* :cpp:class:`cub::WarpStore` stores items partitioned across a CUDA warp to a linear segment of memory
diff --git a/cccl_upstream/docs/cudax/Doxyfile b/cccl_upstream/docs/cudax/Doxyfile
new file mode 100644
index 00000000..301ae2dc
--- /dev/null
+++ b/cccl_upstream/docs/cudax/Doxyfile
@@ -0,0 +1,185 @@
+# Doxyfile for CUDAX
+
+PROJECT_NAME           = CUDAX
+OUTPUT_DIRECTORY       = ../_build/doxygen/cudax
+CREATE_SUBDIRS         = NO
+GENERATE_HTML          = NO
+GENERATE_LATEX         = NO
+GENERATE_XML           = YES
+XML_OUTPUT             = xml
+XML_PROGRAMLISTING     = YES
+
+# Input directory - specific paths for cudax
+INPUT                  = ../../cudax/include/cuda/experimental/__copy_bytes \
+                         ../../cudax/include/cuda/experimental/__fill_bytes \
+                         ../../cudax/include/cuda/experimental/__container \
+                         ../../cudax/include/cuda/experimental/__device \
+                         ../../cudax/include/cuda/experimental/graph.cuh \
+                         ../../cudax/include/cuda/experimental/__graph \
+                         ../../cudax/include/cuda/experimental/__group \
+                         ../../cudax/include/cuda/experimental/__launch \
+                         ../../cudax/include/cuda/experimental/__places \
+                         ../../cudax/include/cuda/experimental/__places/exec \
+                         ../../cudax/include/cuda/experimental/__stream \
+                         ../../cudax/include/cuda/experimental/stf.cuh \
+                         ../../cudax/include/cuda/experimental/__stf \
+                         ../../cudax/include/cuda/experimental/__stf/internal \
+                         ../../cudax/include/cuda/experimental/__stf/utility \
+                         ../../cudax/include/cuda/experimental/__stf/localization \
+                         ../../cudax/include/cuda/experimental/__stf/allocators \
+                         ../../cudax/include/cuda/experimental/__stf/graph \
+                         ../../cudax/include/cuda/experimental/__stf/graph/internal \
+                         ../../cudax/include/cuda/experimental/__stf/graph/interfaces \
+                         ../../cudax/include/cuda/experimental/__stf/places \
+                         ../../cudax/include/cuda/experimental/__stf/places/exec/host \
+                         ../../cudax/include/cuda/experimental/__stf/stream \
+                         ../../cudax/include/cuda/experimental/__stf/stream/internal \
+                         ../../cudax/include/cuda/experimental/__stf/stream/interfaces
+
+RECURSIVE              = YES
+# Exclude patterns from repo.toml
+EXCLUDE                = ../../cudax/include/cuda/experimental/__graph/fwd.cuh
+EXCLUDE_PATTERNS       = */detail/* */test/* */tests/* */examples/* */__detail/*
+# Exclude symbols from repo.toml
+EXCLUDE_SYMBOLS        = *detail* *RESERVED* *reserved* *__* _A* _B* _C* _D* _E* _F* _G* _H* _I* _J* _K* _L* _M* _N* _O* _P* _Q* _R* _S* _T* _U* _V* _W* _X* _Y* _Z* UNITTEST
+
+# Path for @snippet references to test/example files
+EXAMPLE_PATH           = ../../cudax/include/cuda/experimental/__stf/utility \
+                         ../../cudax/test/multi_gpu/algorithms \
+                         ../../cudax/test/coop/reduce
+
+FILE_PATTERNS          = *.h *.hpp *.cuh
+EXTENSION_MAPPING      = cuh=C++ cu=C++
+
+# Documentation extraction settings
+EXTRACT_ALL            = YES
+EXTRACT_PRIVATE        = NO
+EXTRACT_STATIC         = YES
+EXTRACT_LOCAL_CLASSES  = YES
+HIDE_UNDOC_MEMBERS     = NO
+HIDE_UNDOC_CLASSES     = NO
+SHOW_INCLUDE_FILES     = YES
+INLINE_INHERITED_MEMB  = YES
+FULL_PATH_NAMES        = YES
+STRIP_FROM_PATH        = ../../cudax
+SHORT_NAMES            = NO
+
+# Parsing settings
+JAVADOC_AUTOBRIEF      = YES
+QT_AUTOBRIEF           = NO
+MULTILINE_CPP_IS_BRIEF = NO
+INHERIT_DOCS           = YES
+SEPARATE_MEMBER_PAGES  = NO
+TAB_SIZE               = 4
+BUILTIN_STL_SUPPORT    = YES
+
+# Preprocessing
+ENABLE_PREPROCESSING   = YES
+MACRO_EXPANSION        = YES
+EXPAND_ONLY_PREDEF     = NO
+SEARCH_INCLUDES        = YES
+SKIP_FUNCTION_MACROS   = YES
+
+# Predefined macros for CUDAX
+PREDEFINED = \
+    "__device__=" \
+    "__host__=" \
+    "__global__=" \
+    "__forceinline__=" \
+    "__declspec(x)=" \
+    "__align__(x)=" \
+    "__cccl_lib_mdspan" \
+    "_CCCL_AND=&&" \
+    "_CCCL_CONCEPT=constexpr bool" \
+    "_CCCL_DOXYGEN_INVOKED" \
+    "_CCCL_HOST_DEVICE=" \
+    "_CCCL_DEVICE=" \
+    "_CCCL_HOST=" \
+    "_CCCL_FORCEINLINE=" \
+    "_CCCL_CONSTEXPR_CXX20=constexpr" \
+    "_CCCL_CONSTEXPR_CXX23=constexpr" \
+    "_CCCL_API=inline" \
+    "_CCCL_HOST_DEVICE_API=inline" \
+    "_CCCL_HOST_API=inline" \
+    "_CCCL_DEVICE_API=inline" \
+    "_CCCL_NODEBUG_API=inline" \
+    "_CCCL_NODEBUG_DEVICE_API=inline" \
+    "_CCCL_NODEBUG_HOST_API=inline" \
+    "_CCCL_TRIVIAL_API=inline" \
+    "_CCCL_TRIVIAL_HOST_API=inline" \
+    "_CCCL_TRIVIAL_DEVICE_API=inline" \
+    "_CCCL_PUBLIC_API=inline" \
+    "_CCCL_PUBLIC_DEVICE_API=inline" \
+    "_CCCL_PUBLIC_HOST_API=inline" \
+    "_CCCL_BEGIN_NAMESPACE_CUDA_STD=namespace cuda::std {" \
+    "_CCCL_END_NAMESPACE_CUDA_STD=}" \
+    "_CCCL_CATCH=catch" \
+    "_CCCL_CATCH_ALL=catch (...)" \
+    "_CCCL_CATCH_FALLTHROUGH=" \
+    "_CCCL_CONSTEXPR_FRIEND=friend " \
+    "_CCCL_CTK_AT_LEAST(x, y)=1" \
+    "_CCCL_CTK_BELOW(x, y)=0" \
+    "_CCCL_CUDACC_AT_LEAST(x, y)=1" \
+    "_CCCL_CUDACC_BELOW(x, y)=0" \
+    "_CCCL_DIAG_POP=" \
+    "_CCCL_DIAG_PUSH=" \
+    "_CCCL_DIAG_SUPPRESS_CLANG(x)=" \
+    "_CCCL_DIAG_SUPPRESS_GCC(x)=" \
+    "_CCCL_DIAG_SUPPRESS_MSVC(x)=" \
+    "_CCCL_DIAG_SUPPRESS_NVHPC(x)=" \
+    "_CCCL_EXEC_CHECK_DISABLE=" \
+    "_CCCL_GLOBAL_CONSTANT=inline constexpr" \
+    "_CCCL_HAS_CTK()=1" \
+    "_CCCL_HIDE_FROM_ABI=" \
+    "_CCCL_REQUIRES(x)= ::cuda::std::enable_if_t<x, int> = 0>" \
+    "_CCCL_STD_VER=2020" \
+    "_CCCL_SUPPRESS_DEPRECATED_POP=" \
+    "_CCCL_SUPPRESS_DEPRECATED_PUSH=" \
+    "_CCCL_TEMPLATE(x)=template<x, " \
+    "_CCCL_TRAILING_REQUIRES(x)=-> x requires " \
+    "_CCCL_TRY=try" \
+    "_CCCL_TYPE_VISIBILITY_DEFAULT=" \
+    "_CCCL_TYPE_VISIBILITY_HIDDEN=" \
+    "_CCCL_VISIBILITY_DEFAULT=" \
+    "_CCCL_VISIBILITY_HIDDEN=" \
+    "_CCCL_LIFETIMEBOUND=" \
+    "_CUDAX_CONSTEXPR_FRIEND=friend" \
+    "_LIBCUDACXX_HAS_SPACESHIP_OPERATOR()=1" \
+    "CCCL_DEPRECATED=" \
+    "CCCL_DEPRECATED_BECAUSE(x)=" \
+    "CCCL_IGNORE_DEPRECATED_CPP_DIALECT" \
+    "CUB_DISABLE_NAMESPACE_MAGIC" \
+    "CUB_IGNORE_NAMESPACE_MAGIC_ERROR" \
+    "CUB_NAMESPACE_BEGIN=namespace cub {" \
+    "CUB_NAMESPACE_END=}" \
+    "CUB_RUNTIME_FUNCTION=" \
+    "THRUST_FWD(x)=x" \
+    "THRUST_NAMESPACE_BEGIN=namespace thrust {" \
+    "THRUST_NAMESPACE_END=}" \
+    "THRUST_PREVENT_MACRO_SUBSTITUTION" \
+    "_CCCL_HOSTED()=1"
+
+# Additional settings from repo.toml
+DISTRIBUTE_GROUP_DOC   = YES
+DOXYFILE_ENCODING      = UTF-8
+INPUT_ENCODING         = UTF-8
+EXAMPLE_RECURSIVE      = YES
+EXAMPLE_PATTERNS       = *.cu
+AUTOLINK_SUPPORT       = YES
+
+# IMPORTANT: Aliases for custom commands
+# The rst alias enables embedding reStructuredText in doxygen comments
+# Using the same format as repo_docs for compatibility
+ALIASES = "rst=\verbatim embed:rst:leading-asterisk"
+ALIASES += "endrst=\endverbatim"
+ALIASES += "rststar=\verbatim embed:rst:leading-asterisk"
+ALIASES += "inlinerst=\verbatim embed:rst:inline"
+
+# Quiet mode
+QUIET                  = YES
+WARNINGS               = YES
+WARN_AS_ERROR          = FAIL_ON_WARNINGS
+WARN_IF_UNDOCUMENTED   = NO
+WARN_IF_DOC_ERROR      = YES
+WARN_IF_INCOMPLETE_DOC = NO
+WARN_NO_PARAMDOC       = NO
diff --git a/cccl_upstream/docs/cudax/container.rst b/cccl_upstream/docs/cudax/container.rst
new file mode 100644
index 00000000..afb589b9
--- /dev/null
+++ b/cccl_upstream/docs/cudax/container.rst
@@ -0,0 +1,27 @@
+.. _cudax-containers:
+
+Containers library
+===================
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   api/class*uninitialized__buffer*
+
+The headers of the container library provide facilities to store elements on the heap. They are heavily inspired by the
+C++ `containers library <https://en.cppreference.com/w/cpp/container>`__ but deviate from the standard provided ones due to different requirements from
+heterogeneous systems.
+
+They build upon :ref:`memory_resources <libcudacxx-extended-api-memory-resources>` to ensure that e.g. execution space
+annotations are checked by the type system.
+
+.. _libcudacxx-containers-uninitialized-async-buffer:
+
+.. list-table::
+   :widths: 25 45 30
+   :header-rows: 0
+
+   * - :ref:`<cuda/experimental/container.cuh> <cudax-containers-uninitialized-buffer>`
+     - Facilities providing uninitialized *heterogeneous* potentially stream ordered storage satisfying a set of properties
+     - cudax 2.7.0 / CCCL 2.7.0
diff --git a/cccl_upstream/docs/cudax/graph.rst b/cccl_upstream/docs/cudax/graph.rst
new file mode 100644
index 00000000..ead9e397
--- /dev/null
+++ b/cccl_upstream/docs/cudax/graph.rst
@@ -0,0 +1,52 @@
+.. _cudax-graph:
+
+Graphs library
+==============
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   api/struct*graph*
+
+The headers of the graph library provide facilities to create and manage CUDA graphs.
+
+This library is under construction and not yet ready for production use.
+
+.. list-table::
+   :widths: 25 45 30
+   :header-rows: 0
+
+   * - :ref:`<cuda/experimental/graph.cuh> <cudax-graph-graph-builder>`
+     - Class `cuda::experimental::graph_builder`: An owning wrapper for a `cudaGraph_t` object.
+     - cudax 2.9.0 / CCCL 2.9.0
+   * - :ref:`<cuda/experimental/graph.cuh> <cudax-graph-graph-node-ref>`
+     - Class `cuda::experimental::graph_node_ref`: A non-owning wrapper for a `cudaGraphNode_t` object.
+     - cudax 2.9.0 / CCCL 2.9.0
+   * - :ref:`<cuda/experimental/graph.cuh> <cudax-graph-graph>`
+     - Class `cuda::experimental::graph`: An owning wrapper for a `cudaGraphExec_t` object.
+     - cudax 2.9.0 / CCCL 2.9.0
+
+.. _cudax-graph-graph-builder:
+
+graph_builder
+-------------
+
+``cuda::experimental::graph_builder`` is an owning wrapper for a ``cudaGraph_t`` object.
+See the API reference entries in this section for details.
+
+.. _cudax-graph-graph-node-ref:
+
+graph_node_ref
+--------------
+
+``cuda::experimental::graph_node_ref`` is a non-owning wrapper for a ``cudaGraphNode_t`` object.
+See the API reference entries in this section for details.
+
+.. _cudax-graph-graph:
+
+graph
+-----
+
+``cuda::experimental::graph`` is an owning wrapper for a ``cudaGraphExec_t`` object.
+See the API reference entries in this section for details.
diff --git a/cccl_upstream/docs/cudax/index.rst b/cccl_upstream/docs/cudax/index.rst
new file mode 100644
index 00000000..1cfb3afa
--- /dev/null
+++ b/cccl_upstream/docs/cudax/index.rst
@@ -0,0 +1,39 @@
+.. _cudax-module:
+
+CUDA Experimental
+=================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   Overview <self>
+   container
+   graph
+   places
+   stf
+   API reference <api/index>
+
+``CUDA Experimental`` (``cudax``) provides experimental new features that are still in development and subject to change.
+However, any feature within this library has important use cases and we encourage users to experiment with them.
+
+Specifically, ``cudax`` provides:
+   - :ref:`asynchronous host from/to device byte-wise mdspan copy <cudax-copy-bytes>`
+   - :ref:`mdspan byte fill <cudax-fill-bytes>`
+   - :ref:`uninitialized storage <libcudacxx-containers-uninitialized-async-buffer>`
+   - :ref:`graph functionality <cudax-graph>`
+   - dimensions description functionality
+   - :ref:`places <cudax-places>` for managing execution and data affinity across devices
+   - :ref:`an implementation of the STF (Sequential Task Flow) programming model <stf>`
+
+Stability Guarantees
+---------------------
+
+There are no stability guarantees whatsoever. We reserve the right to change both the ABI and the API of any feature
+within ``cudax`` at any time without notice.
+
+Availability
+-------------
+
+Due to its experimental nature and the lack of stability guarantees, ``cudax`` is not shipped with the CUDA toolkit but
+is solely available through GitHub.
diff --git a/cccl_upstream/docs/cudax/places.rst b/cccl_upstream/docs/cudax/places.rst
new file mode 100644
index 00000000..7782cd3a
--- /dev/null
+++ b/cccl_upstream/docs/cudax/places.rst
@@ -0,0 +1,778 @@
+.. _cudax-places:
+
+Places
+======
+
+.. contents::
+   :depth: 2
+
+Places are abstractions that represent where code executes and where data
+resides across the non-uniform memory of a CUDA system. They provide a
+unified interface for managing execution affinity, stream pools, memory
+allocation, and device context switching -- independently of any task-based
+programming model.
+
+Places come in two flavors:
+
+- **Execution places** (``exec_place``) determine where code is executed.
+- **Data places** (``data_place``) specify where data is located in memory.
+
+The places API is part of the ``cuda::experimental::places`` C++ namespace
+and can be used standalone via the ``cuda/experimental/places.cuh`` header,
+without pulling in the full CUDASTF task-graph framework. For backward
+compatibility, all places types are also available in the
+``cuda::experimental::stf`` namespace.
+
+.. _places-execution-places:
+
+Execution places
+----------------
+
+An *execution place* describes a location where computation can occur.
+The following factory methods create the most common execution places:
+
+- ``exec_place::device(id)`` -- a specific CUDA device
+- ``exec_place::host()`` -- the host CPU
+- ``exec_place::current_device()`` -- the CUDA device that is currently active
+- ``exec_place::cuda_context(ctx, devid)`` -- an externally-owned CUDA driver
+  context; the device ordinal is derived from the context when ``devid`` is
+  omitted
+
+When an execution place is activated, it sets the appropriate CUDA context
+(e.g. calls ``cudaSetDevice``). Each execution place also has an *affine*
+data place: the memory location naturally associated with it. For a device
+execution place the affine data place is the device's global memory; for
+the host it is pinned host memory (RAM).
+
+A CUDA-context execution place is non-owning. The caller must keep the
+``CUcontext`` alive while the place and any streams obtained from it are in use.
+
+.. _places-data-places:
+
+Data places
+-----------
+
+A *data place* describes a memory location where data can reside. The
+following factory methods are available:
+
+- ``data_place::device(id)`` -- global memory of a specific CUDA device
+- ``data_place::host()`` -- pinned host memory
+- ``data_place::managed()`` -- CUDA managed (unified) memory
+- ``data_place::affine()`` -- the data place naturally associated with the
+  current execution place
+
+The *affine* data place is the default: when no data place is specified,
+data is placed in the memory that is local to the execution place. For
+example, a task running on device 0 will access data in device 0's global
+memory by default.
+
+Non-affine placement is also supported: data can be placed on a different
+device or in host memory regardless of where the computation runs. This is
+useful for sparse accesses (leveraging CUDA Unified Memory page faulting)
+or for addressing memory capacity constraints. Non-affine placement assumes
+the hardware and OS support such accesses (NVLINK, UVM, etc.).
+
+.. _places-container-keys:
+
+Places as container keys
+------------------------
+
+Both ``exec_place`` and ``data_place`` can be used as keys in standard
+associative containers. The library provides the required comparison and
+hash support:
+
+- **``std::map``** and **``std::set``** use ``operator<`` (strict weak
+  ordering) for keys. Both place types implement ``operator<``, so they
+  can be used as ordered map or set keys.
+
+- **``std::unordered_map``** and **``std::unordered_set``** require a
+  hash function and equality. The library specializes ``cuda::experimental::stf::hash``
+  for both place types (accessible from both the ``stf`` and ``places`` namespaces),
+  and both implement ``operator==``.
+
+This allows, for example, maintaining per-place handles (e.g. CUBLAS or
+CUSOLVER handles keyed by ``exec_place``) or per-place caches keyed by
+``data_place``, using either ordered or hash-based containers as needed.
+The following snippet shows lazy creation of a CUBLAS handle per execution
+place using an ``std::unordered_map`` keyed by ``exec_place``:
+
+.. code:: c++
+
+   #include <cuda/experimental/places.cuh>
+   #include <cublas_v2.h>
+
+   using namespace cuda::experimental::places;
+
+   cublasHandle_t& get_cublas_handle(const exec_place& ep = exec_place::current_device())
+   {
+     static std::unordered_map<exec_place, cublasHandle_t, hash<exec_place>> handles;
+     auto& h = handles[ep];
+     if (h == cublasHandle_t{})
+     {
+       exec_place_scope scope(ep);
+       cuda_safe_call(cublasCreate(&h));
+     }
+     return h;
+   }
+
+.. _places-activate:
+
+Setting the current device or context
+--------------------------------------
+
+The ``exec_place::activate()`` method provides a generic alternative to
+``cudaSetDevice()`` that works uniformly across different execution place types.
+This is useful when you want to set the current CUDA device or context without
+using tasks.
+
+The method returns an ``exec_place`` representing the previous state, which can
+be used to restore the original device or context.
+
+**Behavior by execution place type:**
+
+- **Device places** (``exec_place::device(id)``): Calls ``cudaSetDevice(id)``
+- **Green context places**: Sets the current CUDA driver context via ``cuCtxSetCurrent()``
+- **Host places**: No-op
+
+**Basic usage with devices:**
+
+.. code:: cpp
+
+    exec_place place = exec_place::device(1);
+    exec_place prev = place.activate();  // Switch to device 1
+
+    // ... perform operations on device 1 ...
+
+    place.deactivate(prev);  // Restore previous device
+
+**Alternative restoration pattern:**
+
+You can also restore by calling ``activate()`` on the returned place:
+
+.. code:: cpp
+
+    exec_place place = exec_place::device(1);
+    exec_place prev = place.activate();
+
+    // ... work on device 1 ...
+
+    prev.activate();  // Equivalent to place.deactivate(prev)
+
+**Usage with green contexts (CUDA 12.4+):**
+
+Green contexts provide SM-level partitioning of GPU resources. The
+``activate()``/``deactivate()`` methods handle the underlying driver context
+management:
+
+.. code:: cpp
+
+    // Create green contexts with 8 SMs each
+    green_context_helper gc(8, device_id);
+    auto view = gc.get_view(0);
+
+    exec_place gc_place = exec_place::green_ctx(view);
+    exec_place prev = gc_place.activate();  // Sets green context as current
+
+    // ... GPU work runs with SM affinity ...
+
+    gc_place.deactivate(prev);  // Restore original context
+
+**RAII scope for scoped activation:**
+
+For exception-safe code or when you want automatic restoration, use the
+``exec_place_scope`` RAII helper:
+
+.. code:: cpp
+
+    {
+        exec_place_scope scope(exec_place::device(1));
+        // Device 1 is now active
+        // ... perform operations on device 1 ...
+    }
+    // Previous device is automatically restored when scope goes out of scope
+
+The guard automatically restores the previous execution place when it goes out
+of scope, making it useful for exception-safe code.
+
+.. _places-stream-management:
+
+Stream management with execution places
+----------------------------------------
+
+Execution places can be used independently of any task system to manage CUDA
+streams in a structured way. This is useful when you want to use place
+abstractions (devices, green contexts) for stream management without the full
+task-based programming model.
+
+Stream pools for pooled places (``device(N)``, ``host()``) live in an
+``exec_place_resources`` registry that the caller owns. Pass the registry to
+``exec_place::pick_stream`` to get a CUDA stream; the per-place pool inside the
+registry is created lazily on first request and is destroyed when the registry
+is destroyed.
+
+The method accepts an optional ``for_computation`` hint (defaults to ``true``)
+that may select between computation and data transfer stream pools to improve
+overlapping. This is purely a performance hint, and it does not affect
+correctness. Not all execution places enforce it.
+
+.. code:: cpp
+
+    #include <cuda/experimental/places.cuh>
+    using namespace cuda::experimental::places;
+
+    // Standalone use: own the registry yourself.
+    exec_place_resources resources;
+
+    // Get a stream from the current device
+    exec_place place = exec_place::current_device();
+    cudaStream_t stream = place.pick_stream(resources);
+
+    // Use the stream for CUDA operations
+    myKernel<<<grid, block, 0, stream>>>(d_data);
+
+    // Get streams from specific devices (sharing the same registry)
+    cudaStream_t stream_dev0 = exec_place::device(0).pick_stream(resources);
+    cudaStream_t stream_dev1 = exec_place::device(1).pick_stream(resources);
+
+Inside a CUDASTF context, the context's ``async_resources_handle`` already
+holds an ``exec_place_resources`` registry. Convenience overloads accept the
+handle directly so call sites do not have to dereference it:
+
+.. code:: cpp
+
+    cudaStream_t stream = place.pick_stream(ctx.async_resources());
+
+Stream pools are populated lazily -- CUDA streams are only created when first
+requested via ``pick_stream(resources)`` (or ``pick_stream(ctx.async_resources())``
+inside CUDASTF). Self-contained places (``exec_place::cuda_stream(s)``,
+green-context places) ignore the registry and return their own embedded pool
+instead, so the user-provided ``cudaStream_t`` / ``CUgreenCtx`` must outlive
+any place that wraps it.
+
+.. _places-memory-allocation:
+
+Memory allocation with data places
+------------------------------------
+
+Data places provide a unified interface for memory allocation that works across
+different memory types (host, device, managed) and place extensions (green
+contexts, user-defined places). This allows you to allocate memory while
+benefiting from the place abstraction.
+
+The ``data_place::allocate()`` and ``data_place::deallocate()`` methods provide
+raw memory allocation. The stream parameter defaults to ``nullptr``, which is
+convenient for non-stream-ordered allocations (host, managed) where the stream
+is ignored:
+
+.. code:: cpp
+
+    #include <cuda/experimental/places.cuh>
+    using namespace cuda::experimental::places;
+
+    // Allocate on host (pinned memory) - stream defaults to nullptr
+    void* host_ptr = data_place::host().allocate(1024);
+    // ... use host_ptr ...
+    data_place::host().deallocate(host_ptr, 1024);
+
+    // Allocate on a specific device (stream-ordered)
+    cudaStream_t stream;
+    cudaStreamCreate(&stream);
+    void* dev_ptr = data_place::device(0).allocate(1024, stream);
+    // ... use dev_ptr with stream ...
+    data_place::device(0).deallocate(dev_ptr, 1024, stream);
+    cudaStreamDestroy(stream);
+
+    // Allocate managed memory - stream defaults to nullptr
+    void* managed_ptr = data_place::managed().allocate(1024);
+    // ... use managed_ptr from host or device ...
+    data_place::managed().deallocate(managed_ptr, 1024);
+
+**Stream-ordered vs immediate allocations:**
+
+Different data places have different allocation behaviors:
+
+- **Host** (``data_place::host()``): Uses ``cudaMallocHost()`` / ``cudaFreeHost()`` - immediate, stream parameter is ignored
+- **Managed** (``data_place::managed()``): Uses ``cudaMallocManaged()`` / ``cudaFree()`` - immediate, stream parameter is ignored (note: ``cudaFree`` may introduce implicit synchronization)
+- **Device** (``data_place::device(id)``): Uses ``cudaMallocAsync()`` / ``cudaFreeAsync()`` - stream-ordered
+- **Extensions** (green contexts, etc.): Behavior depends on the extension implementation
+
+You can query whether a place uses stream-ordered allocation with
+``allocation_is_stream_ordered()``:
+
+.. code:: cpp
+
+    data_place place = data_place::device(0);
+    if (place.allocation_is_stream_ordered()) {
+        // Allocation is stream-ordered - synchronize via the stream
+        void* ptr = place.allocate(size, stream);
+        myKernel<<<grid, block, 0, stream>>>(ptr);
+        place.deallocate(ptr, size, stream);
+        cudaStreamSynchronize(stream);
+    } else {
+        // Allocation is immediate - stream is ignored, safe to use right away
+        void* ptr = place.allocate(size);
+        // ... use ptr ...
+        place.deallocate(ptr, size);
+    }
+
+This abstraction is particularly useful when writing generic code that needs to
+work with different types of places, including custom place extensions.
+
+Geometry-aware allocation with allocate_nd
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Some places need to know the shape of the tensor being allocated, not just its
+size: a composite data place distributes the allocation according to a
+partitioner that maps *element coordinates* to places. ``allocate_nd()`` takes
+the tensor extents (dimension 0 varying fastest) and the element size:
+
+.. code:: cpp
+
+    // 2-D tensor of nx x ny doubles, distributed by the place's partitioner
+    void* ptr = place.allocate_nd(dim4(nx, ny), sizeof(double));
+    // ...
+    place.deallocate(ptr, nx * ny * sizeof(double));
+
+For most places this is equivalent to ``allocate(prod(dims) * elemsize)``. For
+composite places it is required: the byte-count ``allocate()`` throws there,
+since a byte count alone cannot carry the geometry the partitioner needs. A
+caller that genuinely has untyped bytes states that explicitly with
+``allocate_nd(dim4(nbytes), 1)``, which distributes the buffer with byte
+granularity. This raw-byte form applies to composite places built from
+scale-free partitioners only; a composite place backed by a structured
+partition (see :ref:`places-structured-partitions`) accepts exactly the
+extents of the tensor the partition was built for and rejects anything else,
+including a flat byte count.
+
+.. _places-vmm:
+
+VMM-based allocation with mem_create
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For advanced use cases involving CUDA's Virtual Memory Management (VMM) API,
+``data_place`` also provides the ``mem_create()`` method. This is a lower-level
+interface used internally by localized arrays (``composite_slice``) to create
+physical memory segments that are then mapped into a contiguous virtual address
+space.
+
+Unlike ``allocate()``, which returns a usable pointer directly, ``mem_create()``
+returns a ``CUmemGenericAllocationHandle`` that must be subsequently mapped with
+``cuMemMap()`` before use:
+
+.. code:: cpp
+
+    #include <cuda/experimental/places.cuh>
+    using namespace cuda::experimental::places;
+
+    // Create a physical memory handle for device 0
+    CUmemGenericAllocationHandle handle;
+    data_place::device(0).mem_create(&handle, size);
+
+    // The handle must be mapped to a virtual address before use
+    // (see CUDA VMM documentation for cuMemMap, cuMemSetAccess, etc.)
+
+**When to use each method:**
+
+- Use ``allocate()`` for most cases - it provides ready-to-use memory with
+  stream-ordered semantics where applicable.
+
+- Use ``mem_create()`` only when you need explicit control over virtual memory
+  mapping, such as creating localized arrays that span multiple devices with a
+  unified virtual address space.
+
+**Limitations of mem_create:**
+
+- Only supports device memory and host memory (pinned)
+- Managed memory is **not supported** by the VMM API
+- The returned handle requires additional VMM API calls to be usable
+
+Custom place extensions can override ``mem_create()`` to provide specialized
+VMM allocation behavior (e.g., memory localization for hardware partitions).
+
+.. _places-grid:
+
+Grid of places
+--------------
+
+It is possible to manipulate places which are a collection of multiple places.
+In particular, it is possible to define an execution place which corresponds
+to multiple device execution places.
+
+Creating grids of places
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+A grid of execution places is an ``exec_place`` that contains multiple
+underlying places. Grids are created with the ``make_grid`` free function:
+
+.. code:: c++
+
+   // Create a 1D grid from a vector of places
+   exec_place grid = make_grid(std::vector<exec_place>{
+       exec_place::device(0), exec_place::device(1)
+   });
+
+The ``exec_place::all_devices()`` helper creates a grid of all available
+CUDA devices:
+
+.. code:: c++
+
+   exec_place all = exec_place::all_devices();
+
+Similarly, ``exec_place::n_devices(n)`` creates a grid from the first ``n``
+devices:
+
+.. code:: c++
+
+   exec_place first_four = exec_place::n_devices(4);
+
+It is possible to retrieve the total number of elements in a grid using
+the ``size()`` method, and individual places with ``get_place(i)``:
+
+.. code:: c++
+
+   exec_place grid = exec_place::all_devices();
+   for (size_t i = 0; i < grid.size(); i++) {
+       exec_place dev = grid.get_place(i);
+       // ...
+   }
+
+Shaped grids
+^^^^^^^^^^^^
+
+Grids of places need not be 1D arrays. They can be structured as a
+multi-dimensional grid described with a ``dim4`` class by passing it to
+``make_grid`` or ``n_devices``:
+
+.. code:: c++
+
+   // Create a shaped grid: 8 devices arranged as a 2x2x2 cube
+   exec_place cube = exec_place::n_devices(8, dim4(2, 2, 2));
+
+   // Or from an explicit vector
+   exec_place shaped = make_grid(my_places, dim4(4, 2));
+
+Note that the total size of the ``dim4`` must match the number of places.
+
+It is possible to query the *shape* of the grid using ``get_dims()``,
+which returns a ``dim4`` object. Individual places can be accessed by
+multi-dimensional position using ``get_place(pos4)``.
+
+Reshaping and collapsing grid axes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+An existing grid can be viewed with different dimensions using
+``reshape()``. The new dimensions must contain exactly the same number of
+places:
+
+.. code:: c++
+
+   exec_place cube = make_grid(my_places, dim4(2, 3, 4));
+   exec_place flat = cube.reshape(dim4(24));
+
+Reshaping changes only the grid coordinate system. It preserves dimension-0-
+fastest linear order, so ``flat.get_place(i) == cube.get_place(i)`` for every
+linear index ``i``. It does not reorder, replicate, or remove places.
+
+``collapse_axes(first, last)`` is a convenience operation that combines a
+contiguous inclusive range of axes. The collapsed extent is the product of
+the selected extents; later axes shift left and trailing extents become one:
+
+.. code:: c++
+
+   exec_place grid = make_grid(my_places, dim4(2, 3, 4));
+
+   exec_place grid_6x4 = grid.collapse_axes(0, 1); // dim4(6, 4)
+   exec_place grid_2x12 = grid.collapse_axes(1, 2); // dim4(2, 12)
+   exec_place grid_24 = grid.collapse_axes(0, 3); // dim4(24)
+
+These operations are useful when a partition should consume several axes of
+a processor grid as one logical axis. They are coordinate transformations,
+not :ref:`places-partitioning`: the latter decomposes a place into constituent
+resources.
+
+.. _places-partitioning:
+
+Partitioning grids
+^^^^^^^^^^^^^^^^^^
+
+The ``place_partition`` class partitions an execution place at a given
+granularity. This is useful for splitting a multi-device grid into its
+constituent devices, or for partitioning a device into green contexts or
+CUDA streams.
+
+The partitioning granularity is specified by ``place_partition_scope``:
+
+- ``place_partition_scope::cuda_device`` -- partition into individual devices
+- ``place_partition_scope::green_context`` -- partition into green contexts (CUDA 12.4+)
+- ``place_partition_scope::cuda_stream`` -- partition into CUDA streams
+
+.. code:: c++
+
+   exec_place grid = exec_place::all_devices();
+
+   // Partition into individual devices
+   place_partition devices(grid, place_partition_scope::cuda_device);
+   for (auto& dev : devices) {
+       // dev is an exec_place for a single device
+   }
+
+   // Convert back to an exec_place grid
+   exec_place new_grid = devices.to_exec_place();
+
+The ``exec_place::partition_by_scope()`` method provides a shorthand that
+returns a new ``exec_place`` grid directly:
+
+.. code:: c++
+
+   exec_place grid = exec_place::all_devices();
+   exec_place by_device = grid.partition_by_scope(place_partition_scope::cuda_device);
+
+.. _places-data-partitioning:
+
+Data partitioning policies
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When using a grid of places with CUDASTF constructs such as ``parallel_for``,
+*data partitioning policies* express how data and index spaces are dispatched
+over the different places of a grid.
+
+.. code:: c++
+
+   class MyPartition : public partitioner_base {
+   public:
+       template <typename S_out, typename S_in>
+       static const S_out apply(const S_in& in, pos4 position, dim4 grid_dims);
+
+       void get_executor(pos4* result, pos4 data_coords, dim4 data_dims, dim4 grid_dims);
+   };
+
+A partitioning class must implement an ``apply`` method which takes:
+
+- a reference to a shape of type ``S_in``
+- a position within a grid of execution places, described using an object of
+  type ``pos4``
+- the dimension of this grid expressed as a ``dim4`` object
+
+``apply`` returns a shape which corresponds to the subset of the ``in``
+shape associated to this entry of the grid. Note that the output shape
+type ``S_out`` may be different from the ``S_in`` type of the input
+shape.
+
+To support different types of shapes, appropriate overloads of the
+``apply`` method should be implemented.
+
+This ``apply`` method is typically used by the ``parallel_for``
+construct in order to dispatch indices over the different places.
+
+A partitioning class must also implement the ``get_executor`` virtual
+method which allows localized data allocators. This
+method indicates, for each entry of a shape, on which place this entry
+should *preferably* be allocated.
+
+``get_executor`` writes a ``pos4`` coordinate in the execution place
+grid into ``*result``, and its input arguments are:
+
+- a coordinate within the shape described as a ``pos4`` object
+- the dimension of the shape expressed as a ``dim4`` object
+- the dimension of the execution place grid expressed as a ``dim4`` object
+
+Defining the ``get_executor`` makes it possible to map a piece of data
+over an execution place grid. The ``get_executor`` method of a partitioning
+policy in an execution place grid therefore defines the *affine data
+place* of a logical data accessed on that grid.
+
+Predefined partitioning policies
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are currently two policies readily available:
+
+- ``tiled_partition<TILE_SIZE>`` dispatches entries of a shape using a
+  *tiled* layout. For multi-dimensional shapes, the outermost dimension is
+  dispatched into contiguous tiles of size ``TILE_SIZE``.
+- ``blocked_partition`` dispatches entries of the shape using a *blocked*
+  layout, where each entry of the grid of places receives approximately
+  the same contiguous portion of the shape, dispatched along the outermost
+  dimension.
+
+This illustrates how a 2D shape is dispatched over 3 places using the
+blocked layout:
+
+.. code:: text
+
+    __________________________________
+   |           |           |         |
+   |           |           |         |
+   |           |           |         |
+   |    P 0    |    P 1    |   P 2   |
+   |           |           |         |
+   |           |           |         |
+   |___________|___________|_________|
+
+This illustrates how a 2D shape is dispatched over 3 places using a
+tiled layout, where the dimension of the tiles is indicated by the
+``TILE_SIZE`` parameter:
+
+.. code:: text
+
+    ________________________________
+   |     |     |     |     |     |  |
+   |     |     |     |     |     |  |
+   |     |     |     |     |     |  |
+   | P 0 | P 1 | P 2 | P 0 | P 1 |P2|
+   |     |     |     |     |     |  |
+   |     |     |     |     |     |  |
+   |_____|_____|_____|_____|_____|__|
+
+.. _places-structured-partitions:
+
+Structured partitions
+---------------------
+
+The classic partitioning policies above are *scale-free*: ``blocked_partition``
+splits whatever shape it is handed, knows nothing about the tensor it will be
+applied to, and always dispatches along the outermost dimension. A
+*structured partition* (``cute_partition``) is the complementary tool: it
+describes, dimension by dimension, how **one specific tensor** maps onto a
+grid of places.
+
+.. code:: c++
+
+   using namespace cuda::experimental::places;
+
+   // A 3-D tensor: dimension 1 blocked over the places of the grid,
+   // dimensions 0 and 2 not distributed
+   auto part = make_partition(
+       dim4(nx, ny, nz),
+       partition_spec{whole, blocked<0>, whole},
+       grid.get_dims());
+
+Each entry in ``partition_spec`` selects a policy for the corresponding
+tensor dimension: ``whole`` (not distributed), ``blocked<axis>``,
+``cyclic<axis>``, or ``block_cyclic<axis>(block_size)``. Rank, policy,
+mesh-axis, and leaf counts are preserved in the C++ type; tensor extents,
+strides, and block sizes remain runtime values. This is strictly more
+expressive than the classic policies -- splitting dimension 1 of a 3-D
+tensor, or mixing policies across dimensions, cannot be stated with
+``blocked_partition``.
+
+The reference shape, padding, and predication
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The first argument of ``make_partition`` is the tensor's extents: unlike a
+classic policy, a structured partition is **bound to one reference shape**,
+and remains the authority on it. This is a deliberate trade, and the source
+of most of the type's properties:
+
+- Split dimensions are *padded up to divisibility* (a 10-element dimension
+  blocked over 3 places is treated as 12, in chunks of 4). Padding makes the
+  underlying layout exact and bijective, which is what keeps every query
+  closed-form: validation is a linear pass over the layout, and the owner of
+  a coordinate is a chain of divisions and modulos.
+- Coordinates beyond the true extents (the *padding phantoms*) own no bytes
+  and do no work: consumers discard them by comparing coordinates against
+  the true extents. This is the *predication* idiom of CUTLASS/CuTe
+  ("partition the rounded-up shape, predicate the boundary") rather than
+  per-place clamping, which would break the layout's uniformity.
+
+Ownership can be queried directly, and -- more importantly -- a candidate
+mapping can be **scored before any memory is committed**:
+
+.. code:: c++
+
+   pos4 owner = part.owner(pos4(x, y, z));   // grid position owning (x,y,z)
+
+   // Dry run: same block-majority decision procedure as a real allocation
+   localized_stats stats = evaluate_localized_placement(grid, part, sizeof(double));
+   // stats.bytes_per_place, stats.accuracy() (estimated fraction of local bytes),
+   // stats.nallocs, ... -- tune the spec, then allocate
+
+Placement through a structured partition
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A structured partition can back a composite data place. Because the
+partition is bound to one tensor, such a place is *per-tensor* -- allocate
+with the partition's exact extents (compare with the classic composite
+place, which is a reusable shape-free policy):
+
+.. code:: c++
+
+   data_place dp = make_composite_data_place(grid, part);
+   void* ptr     = dp.allocate_nd(dim4(nx, ny, nz), sizeof(double));
+   // physical pages land on the place owning them, per the partition
+   dp.deallocate(ptr, nx * ny * nz * sizeof(double));
+
+Two structured composite places built from equal partitions compare equal,
+so they denote the same data placement wherever data places are compared.
+
+Conventions and limits
+^^^^^^^^^^^^^^^^^^^^^^
+
+- Extents follow the **dimension-0-fastest** linearization of
+  ``dim4::get_index()`` (the convention of STF slices). A row-major front-end
+  must present its *whole* description in this order -- the extents, the
+  per-dimension ``partition_spec``, and any coordinates passed to ``owner()``
+  reverse together, since reversing only the extents would silently re-target
+  each policy at the wrong axis.
+- At most 4 tensor dimensions (the ``pos4``/``dim4`` domain).
+- Typed partitions and their kernel-facing sub-shapes store exactly their
+  layout leaves. Runtime interfaces (including C/Python opaque handles) erase
+  them to a canonical descriptor only at the data-place boundary.
+- The partition object is trivially copyable and its queries are host/device
+  callable.
+
+The ``partitioned_axpy`` example shows the intended workflow end to end:
+express the partition once, evaluate it, run tasks over data placed by it,
+and perform a raw geometry-aware allocation.
+
+Computing over structured partitions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The same ``parallel_for`` entry point that accepts the classic policies
+accepts a structured partition instance, which then decides **both** the
+per-place kernel decomposition and (through the task's affine data place)
+the placement of the data those kernels touch -- one object, both sides:
+
+.. code:: c++
+
+   // Every place computes exactly the coordinates it owns
+   ctx.parallel_for(part, grid, lX.shape(), lX.write())
+       ->*[] __device__(size_t x, size_t y, size_t z, auto X) { ... };
+
+The shape argument may also be a ``box`` describing a *region within the
+tensor the partition was built for* (validated by containment) -- e.g. the
+interior of a stencil domain. Each place still enumerates its own
+coordinates; those outside the region (like the padding phantoms of uneven
+extents) are skipped by a per-coordinate predicate, so iteration stays
+aligned with data ownership rather than re-splitting the region:
+
+.. code:: c++
+
+   box interior({1ul, nx - 1}, {1ul, ny - 1}, {1ul, nz - 1});
+   ctx.parallel_for(part, grid, interior, lX.rw())->*...;
+
+Predication has a cost proportional to the *rejected* fraction of the
+enumerated coordinates, which makes it the right tool for regions that are
+dense in their bounds (interiors: the rejected boundary shell is a
+surface-to-volume fraction) and the wrong tool for thin regions. For
+boundary-style updates -- a face of the domain, say -- prefer one of:
+
+- **fuse** the boundary handling into the volumetric kernel's body when the
+  condition is cheap (application-dependent);
+- iterate the face with a **classic scale-free policy** (tight, no rejected
+  coordinates) while an explicit dependency keeps placement on the
+  partition's composite place:
+
+  .. code:: c++
+
+     auto dist = make_composite_data_place(grid, part);
+     box face({0ul, nx}, {0ul, ny}, {0ul, 1ul});
+     ctx.parallel_for(blocked_partition(), grid, face, lX.rw(dist))->*...;
+
+  The face's few remote writes (places computing parts of a face another
+  place owns) are typically negligible against the volumetric traffic.
+
+The ``fdtd_mgpu`` example demonstrates the full pattern: a single
+``make_partition`` call decides which dimension splits for every task --
+initialization over the full shape, updates over interior boxes, a point
+source -- and places the fields' data, so changing the distribution of the
+whole simulation is editing one ``partition_spec`` entry.
diff --git a/cccl_upstream/docs/cudax/stf.rst b/cccl_upstream/docs/cudax/stf.rst
new file mode 100644
index 00000000..3152f823
--- /dev/null
+++ b/cccl_upstream/docs/cudax/stf.rst
@@ -0,0 +1,2424 @@
+.. _stf:
+
+CUDASTF
+========
+
+.. contents::
+   :depth: 2
+
+CUDASTF is an implementation of the Sequential Task Flow model for CUDA.
+
+The availability of parallelism within modern hardware has dramatically
+increased, with large nodes now featuring multiple accelerators. As a
+result, maximizing concurrency at the application level in a scalable
+manner has become a crucial priority. To effectively hide latencies, it
+is essential to achieve the highest level of asynchrony possible.
+
+CUDASTF introduces a tasking model that automates data transfers while
+enforcing implicit data-driven dependencies.
+
+Implemented as a header-only C++ library, CUDASTF builds on top of CUDA
+APIs to simplify the development of multi-GPU applications.
+
+CUDASTF enables the creation of highly concurrent parallel applications,
+leveraging both the CUDA stream API and the CUDA graph API for efficient
+task orchestration and data management. The same client code can run
+optimally on single- and multi-GPU systems.
+
+The Sequential Task Flow (STF) programming model
+------------------------------------------------
+
+The CUDASTF programming model involves defining logical data and
+submitting tasks that operate on this data. CUDASTF automatically
+deduces the dependencies between different tasks and orchestrates both
+computation and data movement to ensure efficient execution with as much
+concurrency as possible.
+
+CUDASTF employs the `Sequential Task
+Flow <https://inria.hal.science/hal-01618526>`__ (STF) programming
+model, which enables the extraction of concurrency from a sequence of
+tasks annotated with appropriate data accesses and their respective
+modes (read-only, write-only, or read/write).
+
+For instance, two tasks modifying the same data will be serialized (in
+order to maintain read-after-write and write-after-write coherency),
+whereas two tasks reading the same data without modification can be
+executed concurrently (read-after-read is coherent in any order). A task
+must wait until all preceding modifications have been completed before
+reading a piece of data (read-after-write). Similarly, a task that needs
+to modify data can only do so once all preceding reads have finished
+(write-after-read).
+
+Applying these simple rules to a complex algorithm (initially expressed
+serially as a sequence of tasks) results in a directed acyclic graph
+(DAG) of tasks, which enables CUDASTF to devise concurrent execution for
+the given algorithm.
+
+By providing data use annotations to CUDASTF, programmers benefit from
+both automated parallelization and transparent data management. Through
+a specialized cache coherency protocol, CUDASTF automates data
+allocation and transfers. As a result, programmers can focus on
+developing efficient task-based algorithms instead of grappling with
+asynchrony and asynchronous data management.
+
+To illustrate how a sequence of tasks can be transformed into a parallel
+application using annotated data accesses, consider the following
+example involving three logical data pieces denoted as ``X``, ``Y``,
+and ``Z``:
+
+::
+
+   T1[X(rw)], T2[X(read), Y(rw)], T3[X(read), Z(rw)], T4[Y(read), Z(rw)]
+
+``T2`` and ``T3`` read ``X``, which is modified by ``T1``, creating a
+read-after-write dependency between ``T1`` and ``T2``, as well as
+between ``T1`` and ``T3``. Since ``T2`` and ``T3`` only perform
+concurrent read accesses, they can execute concurrently. ``T4`` reads
+``Y`` and ``Z``, which were modified by ``T2`` and ``T3``, respectively,
+resulting in write-after-read dependencies between ``T2`` and ``T4``,
+and between ``T3`` and ``T4``. The resulting dependency graph is shown
+below.
+
+.. image:: stf/images/graph_01.png
+
+Getting started with CUDASTF
+----------------------------
+
+Getting CUDASTF
+^^^^^^^^^^^^^^^
+
+CUDASTF is part of the CUDA Experimental library of the CCCL project. It is not distributed with the CUDA Toolkit like the rest of CCCL. It is only available on the `CCCL GitHub repository <https://github.com/NVIDIA/cccl>`_.
+
+Using CUDASTF
+^^^^^^^^^^^^^
+
+CUDASTF is a header-only C++ library which only require to include its
+main header. CUDASTF API is part of the ``cuda::experimental::stf`` C++
+namespace, and we will assume for brevity that we are using this
+workspace in the rest of this document.
+
+.. code:: cpp
+
+   #include <cuda/experimental/stf.cuh>
+
+   using cuda::experimental::stf;
+
+Compiling
+^^^^^^^^^
+
+CUDASTF requires a compiler conforming to the C++17 standard or later.
+Although there is no need to link against CUDASTF itself, the library
+internally utilizes the CUDA library.
+
+.. code:: bash
+
+   # Compilation flags
+   nvcc -std=c++17 --expt-relaxed-constexpr --extended-lambda -I$(cudastf_path)
+   # Linking flags
+   nvcc -lcuda
+
+It is also possible to use CUDASTF without ``nvcc``. This is for example
+useful when calling existing CUDA libraries such as CUBLAS which do not
+require authoring custom kernels. Note that CUDASTF APIs intended to
+automatically generate CUDA kernels such as ``parallel_for`` or
+``launch`` are disabled when compiling without nvcc.
+
+.. code:: bash
+
+   # Compilation flags
+   g++ -I$(cudastf_path)
+   # Linking flags
+   g++ -lcuda -lcudart
+
+Using CUDASTF within a CMake project
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+As part of the CCCL project, CUDASTF uses CMake for its build and installation
+infrastructure, and is the recommended way of building applications that use
+CUDASTF.
+
+This is facilitated by the CMake Package Manager as illustrated in this simple example which is available `here <https://github.com/NVIDIA/cccl/tree/main/examples/cudax_stf>`_, and which is described in the next paragraph.
+
+A simple example
+^^^^^^^^^^^^^^^^
+
+The following example illustrates the use of CUDASTF to implement the
+well-known AXPY kernel, which computes ``Y = Y + alpha * X`` where ``X``
+and ``Y`` are two vectors, and ``alpha`` is a scalar_view value.
+
+.. code:: cpp
+
+   #include <cuda/experimental/stf.cuh>
+
+   using namespace cuda::experimental::stf;
+
+   template <typename T>
+   __global__ void axpy(T a, slice<T> x, slice<T> y) {
+       int tid = blockIdx.x * blockDim.x + threadIdx.x;
+       int nthreads = gridDim.x * blockDim.x;
+
+       for (int ind = tid; ind < x.size(); ind += nthreads) {
+           y(ind) += a * x(ind);
+       }
+   }
+
+   int main(int argc, char** argv) {
+       context ctx;
+
+       const size_t N = 16;
+       double X[N], Y[N];
+
+       for (size_t ind = 0; ind < N; ind++) {
+           X[ind] = sin((double)ind);
+           Y[ind] = col((double)ind);
+       }
+
+       auto lX = ctx.logical_data(X);
+       auto lY = ctx.logical_data(Y);
+
+       double alpha = 3.14;
+
+       /* Compute Y = Y + alpha X */
+       ctx.task(lX.read(), lY.rw())->*[&](cudaStream_t s, auto sX, auto sY) {
+           axpy<<<16, 128, 0, s>>>(alpha, sX, sY);
+       };
+
+       ctx.finalize();
+   }
+
+The code is organized into several steps, which will be described in
+more detail in the following sections:
+
+1. include CUDASTF headers
+2. declare a CUDASTF context
+3. create logical data
+4. submit and wait for the completion of pending work
+
+More examples can be found in the ``examples`` directory in the sources.
+
+Compiling examples
+^^^^^^^^^^^^^^^^^^
+
+The following commands compile STF examples from the root of the CCCL sources.
+Please note the ``-j`` option, which specifies how many processes should be used to
+compile the examples. Not specifying it will launch as many processes as there
+are processors on the machine, which might lead to an excessive resource
+consumption and system instability.
+
+.. code:: bash
+
+    mkdir -p build
+    cd build
+    cmake .. --preset cudax
+    cd cudax
+    ninja cudax.examples.stf -j4
+
+To launch examples, simply run binaries under the `bin/`
+subdirectory in the current directory. For instance, to launch the `01-axpy`
+example:
+
+.. code:: bash
+
+    ./bin/cudax.cpp17.example.stf.01-axpy
+
+Backends and contexts
+-------------------------------
+
+The code snippet below includes the required CUDASTF header. It then
+creates a context object, which is an entry point for every API calls,
+and which stores the state of the CUDASTF library and to keep track of
+all resources and all dependencies. This context must eventually be
+destroyed using the ``finalize()`` method.
+
+.. code:: cpp
+
+   context ctx;
+
+There are currently three context backends available in CUDASTF, with a
+common API but possibly different implementations, and a few specific
+extensions. The ``context`` class, which is a generic context
+implementation should be preferred to write generic code. Using a
+specific context type might reduce compilation time, but provide less
+flexibility.
+
+The ``stream_ctx`` class defines a context type that relies on CUDA
+streams and CUDA events to implement synchronizations. Tasks are
+launched eagerly. This is the context type used by default in the
+generic ``context`` type.
+
+The ``graph_ctx`` class is a context type that implements task
+parallelism by the means of CUDA graphs. Tasks (and all related
+operations) are put into CUDA graphs. Note that the lambda function
+attached describing a task is captured immediately (during the
+``ctx.task`` API call) even if the execution is deferred. The underlying
+CUDA graph is launched when a synchronization with the host is needed,
+or when the context is finalized. Other circumstances such as task
+fences might flush all pending operations and result into a graph
+launch. Subsequent operations would be put in a new CUDA graph.
+Selecting this backend is an easy way to adopt CUDA graphs, and can be
+beneficial in terms of performance with a repeated task patterns. Unlike
+other context types, it is not allowed for a task to synchronize with
+the CUDA stream (e.g. with ``cudaStreamSynchronize``) within a task.
+
+Using either ``context``, ``stream_ctx``, or ``graph_ctx`` should result
+in the same behaviour, even if the underlying implementation differs.
+One may switch from a type to another one by adapting how we initialize
+the context object, or by selecting an appropriate type to decide
+statically:
+
+.. code:: cpp
+
+   // assigns a graph_ctx() to a generic context
+   context ctx = graph_ctx();
+
+   // statically select a context based on CUDA streams and CUDA events
+   stream_ctx ctx;
+
+   // statically select a context based on CUDA streams and CUDA events
+   graph_ctx ctx;
+
+For the most part, these types can be used interchangeably. The key
+difference is that ``stream_ctx`` and ``graph_ctx`` are statically bound to
+use either the CUDA stream or graph APIs, while ``context`` defers this
+decision to runtime, allowing dynamic selection of the appropriate backend.
+This flexibility does not introduce significant runtime overhead or
+compilation time differences, but it may be necessary when the user needs
+to select the context type dynamically (see Algorithms).
+
+Tasks in the Stream backend
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``stream_ctx`` backend utilizes CUDA streams and events to provide
+synchronization. Each ``stream_task`` in the ``stream_ctx`` backend
+represents a task that is associated with an input CUDA stream.
+Asynchronous work can be submitted in the body of the task using this
+input stream. Once the ``stream_task`` completes, all work submitted
+within the task's body is assumed to be synchronized with the associated
+stream.
+
+Users can query the stream associated to a ``stream_task`` using its
+``get_stream()`` method.
+
+Tasks in the Graph backend
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In the ``graph_ctx`` environment, a CUDA graph is either created
+internally or passed in by the user during construction. If the user
+supplies the CUDA graph, CUDASTF can automatically insert CUDA graph
+nodes to enable subsequent tasks to be submitted as child graphs of the
+user-supplied graph.
+
+Creating a ``graph_task`` object results in creating a child graph in the
+aforementioned graph associated to the ``graph_ctx`` object. The child
+graph implements the body of the task, and CUDASTF automatically inserts
+the appropriate dependencies to ensure this child graph is executed only
+after all of its dependencies are fulfilled. CUDASTF may also add other
+nodes in the supporting CUDA graph, such as those needed for data
+transfers or data allocations.
+
+Users can retrieve the graph associated to a ``graph_task`` by using its
+``get_graph()`` method.
+
+Logical data
+------------
+
+In traditional computing, "data", such as a matrix describing a neural
+network layer, typically refers to a location in memory with a defined
+address. However, in mixed CPU/GPU systems, the same conceptual data may
+exist simultaneously in multiple locations and have multiple addresses
+(typically the CPU-tied RAM plus one or more copies in the
+high-bandwidth memory used by GPUs). CUDASTF refers to such conceptual
+data as *logical data*, an abstract handle for data that may get
+transparently transferred to or replicated over the different places
+used by CUDASTF tasks. When user code creates a logical data object from
+a user-provided object (e.g. an array of ``double``), they transfer the
+ownership of the original data to CUDASTF. As a result, any access to
+the original data should be performed through the logical data
+interface, as CUDASTF may transfer the logical data to a CUDA device
+where it can be modified, rendering the original data invalid. By doing
+this, user code is relieved of all memory allocation chores and of
+keeping track of which physical location holds the correct data at
+different stages of computation.
+
+A logical data is created by calling the ``ctx.logical_data`` member
+function. The resulting object will be used to specify data accesses
+within tasks.
+
+In the following example, a stack array ``X`` is used to define a new
+logical data object ``lX``, which should be subsequently used instead of
+``X``:
+
+.. code:: cpp
+
+   double X[N];
+   auto lX = ctx.logical_data(X);
+
+Each logical data object internally maintains various *data instances*,
+which are replicas of the logical data at different *data places*. For
+instance, there could be an instance in host memory, as well as
+instances in the embedded memory of CUDA device 0 and CUDA device 1.
+CUDASTF ensures that tasks have access to *valid* data instances where
+they execute and may dynamically create new instances or destroy
+existing ones.
+
+In the example above, ``X`` is initially on the host (on the CPU stack).
+If a task is subsequently launched on device ``0`` that modifies data
+through ``lX``, a new data instance will be created in memory associated
+with device ``0``. In addition making that allocation, CUDASTF ensures
+that a data transfer is issued asynchronously from the host to the
+device, so that the task is given a *valid* replica of ``X``. Given that
+the task modifies data through ``lX``, the instance associated to the
+host will also be invalidated, so CUDASTF will later copy data back to
+the host if another task needs to access ``X`` from the CPU.
+
+Data interfaces
+^^^^^^^^^^^^^^^
+
+CUDASTF implements a generic interface to manipulate different types of
+data formats across the machine.
+
+Every type of data format is described using three separate types :
+
+- its shape, which stores parameters which will be common to all instance.  For
+  a fixed-sized vector, the shape would for example contain the length of the
+  vector.
+- a per-instance type that describes a specific data instance. For a
+  fixed-sized vector, this type would for example contain the address of the
+  vector.
+- a data interface class which implements operations such as allocating a data
+  instance based on its shape, or copying an instance into another instance.
+
+Defining custom data interfaces (advanced)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. toctree::
+   :maxdepth: 1
+
+   stf/custom_data_interface
+
+
+CUDASTF API is designed to be extensible, so that advanced users may
+define their own data interfaces. This can be useful when manipulating
+data formats which are not regular multidimensional arrays, or to
+provide a direct access to a domain-specific or an application-specific
+data format.
+
+A complete example is given :ref:`here <stf_custom_data_interface>` to
+illustrate how to implement a custom data interface.
+
+Write-back policy
+^^^^^^^^^^^^^^^^^
+
+When a logical data object is destroyed, the original data instance is
+updated (unless the logical data was created without a reference value,
+e.g. from a shape). The result is only guaranteed to be available on the
+corresponding data place when after the ``finalize()`` method was called
+on the context. Likewise, when calling ``finalize()`` a write-back
+mechanism is automatically issued on all logical data associated to the
+context if they were not already destroyed.
+
+Write back is enabled by default, but it is possible to disable it for a
+specific logical data by calling this method on a logical data :
+``set_write_back(bool flag)``. Enabling write-back on a logical data
+which was defined from a shape and has no reference data instance will
+result in an error.
+
+Slices
+^^^^^^
+
+To facilitate the use of potentially non-contiguous multi-dimensional
+arrays, we have introduced a C++ data structure class called ``slice``.
+A slice is a partial specialization of C++'s
+``std::mdspan`` (or ``std::experimental::mdspan`` depending on the C++ revision).
+
+.. code:: cpp
+
+   template <typename T, size_t dimensions = 1>
+   using slice = mdspan<T, dextents<size_t, dimensions>, layout_stride>;
+
+When creating a ``logical_data`` from a C++ array, CUDASTF automatically
+describes it as a slice instantiated with the scalar_view element type and
+the dimensionality of the array. Here is an example with an 1D array of
+``double``.
+
+.. code:: cpp
+
+   double A[128];
+   context ctx;
+   auto lA = ctx.logical_data(A);
+
+Internally, all instances of ``A`` are described as ``slice<double, 1>``
+where ``double`` is the scalar_view element type, and ``1`` is the
+dimensionality of the array. The default dimension corresponds to ``1``,
+so ``slice<double>`` is equivalent with ``slice<double, 1>``.
+
+The ``mdspan`` facility provides a `variety of
+methods <https://en.cppreference.com/w/cpp/container/mdspan>`__ also
+available to its alias ``slice``:
+
+-  ``T *data_handle()`` gives the address of the first element
+-  ``operator()`` so that ``A(i)`` is the ``i``-th element of a slice of
+   dimension ``1``, and ``A(i, j)`` is the element at coordinates
+   ``(i, j)`` in a 2D slice.
+-  ``size_t size()`` returns the total number of elements in the slice
+-  ``size_t extent(size_t dim)`` returns the size of a slice in a given
+   dimension (run-time version)
+-  ``size_t stride(size_t dim)`` returns the distance in memory between
+   two elements in a given dimension, expressed as a number of elements
+   (run-time version)
+
+Slices can be passed by value, copied, or moved. Copying a slice does
+not copy the underlying data. Slices can be passed as arguments to CUDA
+kernel. Example:
+
+.. code:: cpp
+
+   template <typename T>
+   __global__ void axpy(T a, slice<T> x, slice<T> y) {
+       int tid = blockIdx.x * blockDim.x + threadIdx.x;
+       int nthreads = gridDim.x * blockDim.x;
+
+       for (int ind = tid; ind < x.size(); ind += nthreads) {
+           y(ind) += a * x(ind);
+       }
+   }
+
+Defining multidimensional slices
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Slices can be used on data with multiple dimensions, and possibly
+non-contiguous data.
+
+For example, to define a 2D slice, we can use the ``make_slice`` method
+which takes a base pointer, a tuple with all dimensions, and then the
+*strides*. The number of stride values is equal to the number of
+dimensions minus one. The i-th stride defines the number of elements in
+memory between two successive elements along dimension i.
+
+.. code:: c++
+
+       double A[5 * 2];
+
+       // contiguous 2D slice
+       slice<double, 2> s = make_slice(A, std::tuple { 5, 2 }, 5);
+
+       // non-contiguous 2D slice
+       slice<double, 2> s2 = make_slice(A, std::tuple { 4, 2 }, 5);
+
+In the second example, ``s2`` defines a non-contiguous 2D slice because
+the stride is greater than the extent in the first dimension. We will
+here *skip* an element between between ``s2(3, 0)`` (which is ``A[3]``)
+and ``s2(0, 1)`` (which is ``A[5]``)
+
+Similarly with 3D data, we need to define 2 strides and 3 extent values
+:
+
+.. code:: c++
+
+       double A[5 * 3 * 40];
+
+       // contiguous 3D slice
+       slice<double, 3> s = make_slice(A, std::tuple { 5, 3, 40 }, 5, 5 * 3);
+
+       // non-contiguous 3D slice
+       slice<double, 3> s2 = make_slice(A, std::tuple { 4, 3, 40 }, 5, 5 * 3);
+
+       // non-contiguous 3D slice
+       slice<double, 3> s3 = make_slice(A, std::tuple { 5, 2, 40 }, 5, 5 * 3);
+
+Such slices can also be used to create logical data :
+
+.. code:: c++
+
+       double A[32 * 32];
+
+       // Contiguous 2D slice
+       auto lX = ctx.logical_data(make_slice(A, std::tuple { 32, 32 }, 32));
+
+       // Non-contiguous 2D slice
+       auto lX2 = ctx.logical_data(make_slice(A, std::tuple { 24, 32 }, 32));
+
+Defining logical data from a shape
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Data interfaces supports data which are only described as a shape. For
+example, a user may want to define a vector of 10 integers, and later
+fill it with a task. In this case, there is no need to have a *reference
+instance* associated to that logical data because CUDASTF will
+automatically allocate an instance on its first usage.
+
+.. code:: cpp
+
+   auto lX = ctx.logical_data(shape_of<slice<int>>(10));
+
+   ctx.task(lX.write())->*[](cudaStream_t stream, auto X) {
+       cudaMemsetAsync(X.data_handle(), 0, X.size()*sizeof(int), stream);
+   };
+
+In this example, ``lX`` is defined using a shape only, and there is no
+physical backing needed to create it. Note that since there exists no
+valid *data instance* of ``lX``, the first task needs to make a
+write-only access (using the ``write()`` member of ``lX``). A write-only
+access will indeed allocate ``lX`` at the appropriate location, but it
+will not try to load a valid copy of it prior to executing the task.
+
+Using other access modes such as ``read()``, ``relaxed()`` or ``rw()``
+that attempt to provide a valid instance will result in an error.  The
+``reduce()`` access mode can be used only if the reduction is not accumulating
+its result with an existing value, so we can for example use
+``reduce(reducer::sum<double>{})`` but not ``reduce(reducer::sum<double>{}, no_init{})``
+on a logical data which has valid data instance.
+
+Similarly, it is possible to define a logical data from a slice shapes
+with multiple dimensions.
+
+.. code:: cpp
+
+   auto lX_2D = ctx.logical_data(shape_of<slice<double, 2>>(16, 24));
+   auto lX_3D = ctx.logical_data(shape_of<slice<double, 3>>(16, 24, 10));
+
+Tasks
+-----
+
+A task is created by calling the ``ctx.task`` member function. It takes
+an optional argument that specifies the execution location of the task.
+If none is provided, the current CUDA device will be used, which is
+equivalent to passing ``exec_place::current_device()``. Data accesses
+are specified using a list of data dependencies. Each dependency is
+described by calling the ``read()``, ``rw()``, ``write()`` or ``reduce()``
+method of the logical data object.
+
+In the example below, ``X`` is accessed in read-only mode and ``Y``
+needs to be updated so it uses a read-write access mode.
+
+.. code:: cpp
+
+   __global__ void axpy(size_t n, double a, const double *x, double *y) {
+       int tid = blockIdx.x * blockDim.x + threadIdx.x;
+       int nthreads = gridDim.x * blockDim.x;
+
+       for (size_t ind = tid; ind < n; ind += nthreads) {
+           y[ind] += a * x[ind];
+       }
+   }
+   ...
+   ctx.task(lX.read(), lY.rw())->*[&](cudaStream_t s, slice<const double> sX, slice<double> sY) {
+       axpy<<<16, 128, 0, s>>>(sX.size(), alpha, sX.data_handle(), sY.data_handle());
+   };
+
+The object returned by the call ``ctx.task()`` overloads
+``operator->*()`` to accept a lambda function on the right-hand side.
+This makes it easy for user code to pass the task's body to the context
+with a syntax akin to a control flow statement. The first argument of
+the lambda function is a ``cudaStream_t`` that can be used to submit
+work asynchronously on the selected device within the body of the task.
+For each logical data, CUDASTF passes a *data instance* to the lambda
+function. These *data instances* provide access to a local copy of the
+logical data, which is coherent with respect to the CUDA stream passed
+to the task.
+
+For example, data instances associated to 1D arrays of ``double`` are
+typed as ``slice<double>`` if the data is in write or read-write mode,
+and ``slice<const double>`` if the data is in read-only mode. The
+``.data_handle()`` method of this type returns the base address of the
+underlying array, and the ``.size()`` method returns the total number of
+elements. For multi-dimensional arrays, ``.extent(d)`` returns the size
+along dimension ``d``. (For a 1D array, ``.size()`` is therefore
+equivalent to ``.extent(0)``.)
+
+Better yet, the CUDA kernel could manipulate slices directly instead of
+resorting to unsafe pointers as parameters:
+
+.. code:: cpp
+
+   __global__ void axpy(double a, slice<const double> x, slice<double> y) {
+       int tid = blockIdx.x * blockDim.x + threadIdx.x;
+       int nthreads = gridDim.x * blockDim.x;
+
+       for (size_t ind = tid; ind < x.size(); ind += nthreads) {
+           y(ind) += a * x(ind);
+       }
+   }
+   ...
+
+   ctx.task(lX.read(), lY.rw())->*[&](cudaStream_t s, slice<const double> sX, slice<double> sY) {
+       axpy<<<16, 128, 0, s>>>(alpha, sX, sY);
+   };
+
+Task submission can be further simplified to rely on type deduction with
+``auto``, which also makes code more generic:
+
+.. code:: cpp
+
+   ctx.task(lX.read(), lY.rw())->*[&](cudaStream_t s, auto sX, auto sY) {
+       axpy<<<16, 128, 0, s>>>(alpha, sX, sY);
+   };
+
+*It is important to note that the body of the task construct is executed
+directly at the submission of the task, and not when the task is
+actually ready for execution. As a result, the body of the task here
+submits a CUDA kernel in the stream, but it is not the CUDA kernel
+itself.* For example, attempting to use slices ``sX`` and ``sY`` in the
+example above immediately in the lambda function would be incorrect; the
+right way is to pass them to a kernel synchronized with the stream
+``s``. CUDA execution semantics will ensure that by the time the kernel
+runs, ``sX`` and ``sY`` will be valid.
+
+Example of creating and using multiple tasks
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Often, complex algorithms involve multiple processing stages, each with
+its own inputs and outputs. In CUDASTF it suffices to express computing
+stages in a sequential manner along with their data dependencies.
+CUDASTF will ensure optimal parallel execution without requiring the
+user code to explicitly define a dependency graph. Consider the
+following example consisting of four tasks, of which three run on GPUs:
+
+.. code:: cpp
+
+   auto lX = ctx.logical_data(X);
+   auto lY = ctx.logical_data(Y);
+
+   // Task 1
+   ctx.task(lX.read(), lY.read())->*[](cudaStream_t stream, auto sX, auto sY) {
+       K1<<<..., stream>>>(sX, sY);
+       K2<<<..., stream>>>(sX, sY);:
+   };
+
+   // Task 2
+   ctx.task(lX.rw())->*[](cudaStream_t stream, auto sX) {
+       K3<<<..., stream>>>(sX);
+   };
+
+   // Task 3
+   ctx.task(lY.rw())->*[](cudaStream_t stream, auto sY) {
+       K4<<<..., stream>>>(sY);
+   };
+
+   // Task 4
+   ctx.host_launch(lX.read(), lY.read())->*[](auto sX, auto sY) {
+       callback(sX, sY);
+   };
+
+Tasks ``T2`` and ``T3`` depend on ``T1`` because they respectively
+modify ``X`` and ``Y``, which were accessed in read-only mode by ``T1``.
+Task ``T4``, executed on the host, reads both ``X`` and ``Y``, and
+therefore needs to wait for the completion of ``T2`` and ``T3``. Note
+that Task ``T1`` submits multiple CUDA kernels in the same CUDA stream.
+This illustrates how a task in CUDASTF encapsulates a piece of work that
+is asynchronous with respect to CUDA stream semantics.
+
+The resulting task graph under the STF programming model is shown below.
+
+.. image:: stf/images/task-sequence-user.png
+
+In full detail, the resulting graph of asynchronous operations includes
+additional data allocations of ``X`` and ``Y`` on the current device, as
+well as copies to and from the device. These automated steps highlight
+how CUDASTF alleviates much of the complexity associated with using
+multiple processing units, allowing programmers to focus on algorithmic
+matters instead.
+
+.. image:: stf/images/task-sequence.png
+
+Lower-level task API
+^^^^^^^^^^^^^^^^^^^^
+
+.. toctree::
+   :maxdepth: 1
+
+   stf/lower_level_api
+
+A lower-level API that does not rely on lambda functions is also
+available, and is described `here <stf/lower_level_api.rst>`__.
+
+.. _stf-synchronization:
+
+Synchronization
+---------------
+
+It is important to note that each task body (passed to the context via
+``operator->*()``) is executed immediately and is used to *submit work
+asynchronously* with respect to the synchronization semantic of the CUDA
+stream. CUDASTF ensures that any operation enqueued in the stream
+attached to the task within task body may access the specified data in a
+coherently, with respect to the requested access modes.
+
+.. code:: cpp
+
+   ctx.submit();
+   // Unrelated CPU-based code might go here...
+   ctx.finalize();
+
+Due to the asynchronous nature of task parallelism, it is necessary to
+ensure that all operations are properly scheduled and executed. As
+CUDASTF transparently handles data management (allocations, transfers,
+…), there can be outstanding asynchronous operations that were not
+submitted explicitly by the user. Therefore it is not sufficient to use
+native CUDA synchronization operations because they are not aware of
+CUDASTF's state. Client code must call ``ctx.finalize()`` instead of
+``cudaStreamSynchronize()`` or ``cudaDeviceSynchronize()``.
+
+-  ``ctx.submit()`` initiates the submission of all asynchronous tasks
+   within the sequence
+-  ``ctx.finalize()`` awaits the conclusion of all outstanding
+   asynchronous operations in the context, automatically invoking
+   ``ctx.submit()`` if not previously called by user code
+
+Usually, creating the task and invoking ``ctx.finalize()`` is
+sufficient. However, manually calling ``ctx.submit()`` can be beneficial
+in at least two situations. First, it allows for executing additional
+unrelated work on the CPU (or another GPU) between submission and
+synchronization. Second, when it's necessary for two contexts to run
+concurrently, using the sequence
+``ctx1.submit(); ctx2.submit(); ctx1.finalize(); ctx2.finalize();``
+achieves this goal (whereas calling
+``ctx1.finalize(); ctx2.finalize();`` without the ``submit()`` calls
+would wait for the completion of the first task before starting the
+second).
+
+To wait for the completion of all pending operations (tasks, transfers, ...),
+an asynchronous fence mechanism is available :
+
+.. code:: cpp
+
+    cudaStream_t stream = ctx.fence();
+    cudaStreamSynchronize(stream);
+
+Another synchronization mechanism is the ``wait`` method of the
+context object. It is typically used in combination with the ``reduce()``
+access mode for dynamic control flow. ``auto val = ctx.wait(ld)`` is a
+blocking call that returns the content of the ``ld`` logical data. The type of
+the returned value is defined by the ``owning_container_of<interface>`` trait
+class where ``interface`` is the data interface of the logical data. For a
+logical data with a regular data interface, the ``wait`` method therefore
+cannot be called unless that interface overloads this trait class.
+
+This mechanism is illustrated in the dot product example of the
+:ref:`reduce_access_mode` section.
+
+``ctx.wait`` can also be called on a :ref:`token <stf-tokens>`. Since a token
+carries no content, ``ctx.wait(token)`` returns ``void`` and simply blocks the
+host until the work the token depends on has completed (see the token section
+for an example).
+
+Places
+------
+
+CUDASTF uses :ref:`places <cudax-places>` to manage data and execution
+affinity. Places can represent either *execution places*, which determine
+where code is executed, or *data places*, specifying the location of data
+across the machine's non-uniform memory. See :ref:`cudax-places` for the
+full places API reference, including stream management, memory allocation,
+grid of places, and partitioning policies.
+
+This section describes how places are used with CUDASTF tasks.
+
+Execution places in tasks
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A task's constructor allows choosing an execution place. The example
+below creates a logical data variable that describes an integer as a
+vector of one ``int``. The logical data variable is then updated on
+device ``0`` and on device ``1`` before being accessed again from the
+host.
+
+The first argument passed to ``ctx.task`` is called an *execution place*
+and tells CUDASTF where the task is expected to execute.
+``exec_place::device(id)`` means that the task will run on device
+``id``, and ``exec_place::host()`` specifies that the task will execute on
+the host.
+
+Regardless of the *execution place*, it is important to note that the
+task's body (i.e., the contents of the lambda function) corresponds to
+CPU code that is expected to launch computation asynchronously. When
+using ``exec_place::device(id)``, CUDASTF will automatically set the
+current CUDA device to ``id`` when the task is started, and restore the
+previous current device when the task ends. ``exec_place::host()`` does
+not affect the current CUDA device.
+
+.. code:: cpp
+
+   context ctx;
+
+   int X = 42;
+
+   auto lX = ctx.logical_data(slice<int>(&X, { 1 }));
+
+   ctx.task(exec_place::device(0), lX.rw())->*[](cudaStream_t stream, auto sX) {
+       inc_kernel<<<1, 1, 0, stream>>>(sX);
+   };
+
+   ctx.task(exec_place::device(1), lX.rw())->*[](cudaStream_t stream, auto sX) {
+       inc_kernel<<<1, 1, 0, stream>>>(sX);
+   };
+
+   ctx.task(exec_place::host(), lX.read())->*[](cudaStream_t stream, auto sX) {
+       cudaStreamSynchronize(stream);
+       assert(sX(0) == 44);
+   };
+
+   ctx.finalize();
+
+Tasks submitted on the host are also executed immediately upon task
+creation and not when dependencies are ready. Asynchronous semantics are
+observed in accordance to CUDA serialization on the ``cudaStream_t``
+lambda parameter. Therefore, the code shown synchronizes explicitly with
+the CUDA stream by calling ``cudaStreamSynchronize(stream)``. This
+ensures the value ``sX`` is read only after data is guaranteed to be
+valid, i.e., after the completion of prior operations in the stream.
+This is disallowed in the ``graph_ctx`` backend.
+
+An alternative solution which is compatible with all types of backend is
+to use ``ctx.host_launch``:
+
+.. code:: cpp
+
+   ctx.host_launch(lX.read())->*[](auto sX) {
+       assert(sX(0) == 44);
+   };
+
+The ``ctx.host_launch`` member function circumvents synchronization of
+the CPU thread with CUDA execution by invoking the lambda function as a
+CUDA callback, thereby maintaining optimal asynchronous semantics for
+the entire workload. Since no explicit synchronization with the
+underlying CUDA stream is needed, ``ctx.host_launch`` is thus compatible
+with the CUDA graph backend (i.e., a context of type ``graph_ctx``).
+
+Data places in tasks
+^^^^^^^^^^^^^^^^^^^^
+
+By default, logical data is associated with the device where it is
+currently processed. A task launched on a device should therefore have
+its data loaded into the global memory of that device, whereas a task
+executed on the host would access data in host memory (RAM). These are
+defined as the *affine* data places of an execution place (see
+:ref:`places-data-places`).
+
+In the example below, data places are not specified for the two tasks
+created. Consequently, the affine data places will be chosen for the two
+tasks: the memory of device ``0`` for the first task and the host RAM
+for the second task.
+
+.. code:: cpp
+
+   ctx.task(exec_place::device(0), lA.rw())->*[](cudaStream_t s, auto a) {
+       ...
+   };
+
+   ctx.task(exec_place::host(), lA.rw())->*[](cudaStream_t s, auto a) {
+       ...
+   };
+
+The code above is equivalent with:
+
+.. code:: cpp
+
+   ctx.task(exec_place::device(0), lA.rw(data_place::affine()))->*[](cudaStream_t s, auto a) {
+       ...
+   };
+
+   ctx.task(exec_place::device(0), lA.rw(data_place::affine()))->*[](cudaStream_t s, auto a) {
+       ...
+   };
+
+The affinity can also be made explicit:
+
+.. code:: cpp
+
+   ctx.task(exec_place::device(0), lA.rw(data_place::device(0)))->*[](cudaStream_t s, auto a) {
+       ...
+   };
+
+   ctx.task(exec_place::device(0), lA.rw(data_place::host()))->*[](cudaStream_t s, auto a) {
+       ...
+   };
+
+CUDASTF also allows to localize data and execution on different places.
+The example below ensures that an instance of logical data ``A`` located
+in host memory is passed to the task so that it can be accessed from
+device ``0``:
+
+.. code:: cpp
+
+   ctx.task(exec_place::device(0), lA.rw(data_place::host()))->*[](cudaStream_t s, auto a) {
+       ...
+   };
+
+Overriding affinity can be advantageous when a task is known to make
+only sparse accesses to a piece of logical data. By overriding affinity,
+transferring large amounts of data is avoided; the paging system of CUDA
+`Unified
+Memory <https://developer.nvidia.com/blog/unified-memory-cuda-beginners/>`__
+will automatically fault in the portions of the data actually used.
+Conversely, we can launch a task on the host that accesses data located
+on a device:
+
+.. code:: cpp
+
+   ctx.task(exec_place::host(), lA.rw(data_place::device(0)))->*[](cudaStream_t s, auto a) {
+       ...
+   };
+
+Alternatively, assuming there are at least two devices available, in
+unified memory it is possible to access the memory of one device from
+another:
+
+.. code:: cpp
+
+   ctx.task(exec_place::device(0), lA.rw(data_place::device(1)))->*[](cudaStream_t s, auto a) {
+       ...
+   };
+
+Non-affine data placement therefore provides flexibility and can be used
+to improve performance or to address memory capacity issues when
+accessing large data sets. They however assume that the system can
+perform such accesses, which may depend on the hardware (NVLINK, UVM, …)
+and the OS (WSL has limited support and lower performance when accessing
+host memory from CUDA kernels, for example).
+
+.. _parallel_for_construct:
+
+``parallel_for`` construct
+--------------------------
+
+CUDASTF provides a helper construct which creates CUDA kernels (or CPU
+kernels) which execute an operation over an index space described as a
+*shape*.
+
+Example with a 1-dimensional array
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The example below illustrates processing a 1D array using
+``parallel_for``:
+
+.. code:: cpp
+
+   int A[128];
+   auto lA = ctx.logical_data(A);
+
+   ctx.parallel_for(exec_place::device(1), lA.shape(), lA.write())->*[] __device__ (size_t i, auto sA) {
+       A(i) = 2*i + 1;
+   };
+
+The ``parallel_for`` construct consists of 4 main elements:
+
+-  an execution place that indicates where the code will be executed;
+-  a shape defining the index space of the generated kernel;
+-  a set of data dependencies;
+-  a body of code specified using the ``->*`` operator.
+
+In the example above, the kernel is launched on the CUDA device with
+index 1, which corresponds to the second installed GPU. Each logical
+data object has a corresponding *data shape*, which can be accessed
+through the ``shape()`` member function of the ``logical_data``
+parametrized type. (The shape of logical data can be thought of as full
+information about the layout, without the actual data.) In this example,
+``lA`` is the logical data associated with a 1D slice of size 128, which
+naturally entails iteration over indices in a 1D dimension ranging from
+0 to 127 (inclusive). The library associates the iteration strategy with
+the data shape. The statement modifies ``lA``, so the lambda function
+will be executed only when the data is ready. The lambda function has
+the ``__device__`` attribute because a device execution place was
+specified. The first parameter corresponds to the index within the shape
+(``size_t i`` for a 1D shape). Subsequent parameters are the data
+instances associated with the logical data arguments (e.g.,
+``slice<int> sA``).
+
+Example with multi-dimensional arrays
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For multidimensional data shapes, iteration (and consequently the lambda
+function) requires additional parameters. Consider an example that uses
+``parallel_for`` to iterate over 2D arrays:
+
+.. code:: cpp
+
+   const size_t N = 16;
+   double X[2 * N * 2 * N];
+   double Y[N * N];
+
+   auto lx = ctx.logical_data(make_slice(&X[0], std::tuple{ 2 * N, 2 * N }, 2 * N));
+   auto ly = ctx.logical_data(make_slice(&Y[0], std::tuple{ N, N }, N));
+
+   ctx.parallel_for(lx.shape(), lx.write())->*[=] __device__(size_t i, size_t j, auto sx) { sx(i, j) = 0.1; };
+
+   ctx.parallel_for(ly.shape(), lx.read(), ly.write())->*[=] __device__(size_t i, size_t j, auto sx, auto sy) {
+       sy(i, j) = y0(i, j);
+       for (size_t ii = 0; ii < 2; ii++)
+           for (size_t jj = 0; jj < 2; jj++) {
+               sy(i, j) += sx(2 * i + ii, 2 * j + jj);
+           }
+   };
+
+Variables ``lx`` and ``ly`` are logical data objects that describe 2D
+arrays, so their shapes are 2D index spaces as well. Consequently, a
+``parallel_for`` construct applied to ``lx.shape()`` is passed two
+indices, ``size_t i`` and ``size_t j``. In the second call of
+``parallel_for``, two logical data objects with different shapes are
+accessed within the same construct. Generally, ``parallel_for`` can
+iterate any number of objects in lockstep, regardless of their
+individual shapes.
+
+Passing a lambda with a signature that starts with a number of
+``size_t`` parameters that does not match the dimensionality of the
+shape will result in a compilation error.
+
+Box shape
+^^^^^^^^^
+
+There are situations where the desired index space does not correspond
+to the shape of a logical data object. For those cases, CUDASTF also
+provides the template class ``box<size_t dimensions = 1>`` (located in
+the header ``cudastf/utility/dimensions.h``) that allows user code to
+define multidimensional shapes with explicit bounds. The template
+parameter represents the dimension of the shape.
+
+Box shapes with extents
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Passing a shape object defined as ``box<2>({2, 3})`` to ``parallel_for``
+will correspond to a 2-dimensional iteration where the first index
+varies from 0 through 1 and the second from 0 through 2. Consider:
+
+.. code:: cpp
+
+   ctx.parallel_for(box<2>({2, 3}))->*[] __device__(size_t i, size_t j) {
+       printf("%ld, %ld\n", i, j);
+   };
+
+The code above will print (in an unspecified order):
+
+::
+
+   0, 0
+   1, 0
+   0, 1
+   1, 1
+   0, 2
+   1, 2
+
+Since the ``box`` default template parameter is 1, it is also possible
+to write code to iterate over all values of ``i`` from 0 through 3:
+
+.. code:: cpp
+
+   ctx.parallel_for(box({4}))->*[] __device__(size_t i) {
+       printf("%ld\n", i);
+   };
+
+Box shapes with lower and upper bounds
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Box shapes can be defined based on their lower and upper bounds. The
+lower bounds are inclusive, while the upper bounds are exclusive.
+Consider an example similar to the previous one:
+
+.. code:: cpp
+
+   ctx.parallel_for(box<2>({{5, 8}, {2, 4}}))->*[] __device__(size_t i, size_t j) {
+       printf("%ld, %ld\n", i, j);
+   };
+
+It will output (in an unspecified order):
+
+::
+
+   5, 2
+   6, 2
+   7, 2
+   5, 3
+   6, 3
+   7, 3
+
+Defining custom shapes (advanced)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Users typically map the ``parallel_for`` construct over the shape of a
+logical data, or over a box shape describing a regular multidimensional
+domain, but it is possible to define new types of shapes to describe an
+index space.
+
+To define a new type of shape ``S`` (where ``S`` typically has a form of
+``shape_of<I>``) that can be used as an indexable shape for
+``parallel_for``, ``shape_of<I>`` must define inner type ``coords_t``
+and member function ``index_to_coords`` as follows:
+
+.. code:: c++
+
+   template <typename I>
+   class shape_of<I> {
+       ...
+   public:
+       using coords_t = ...;
+
+       // This transforms a 1D index into a coordinate
+       __device__ __host__ coords_t index_to_coords(size_t index) const {
+           ...
+       }
+   };
+
+The dimensionality of this ``coord_t`` tuple type determines the number
+of arguments passed to the lambda function in ``parallel_for``.
+
+.. _reduce_access_mode:
+
+Reduce access mode
+^^^^^^^^^^^^^^^^^^
+
+The `parallel_for` construct supports the ``reduce()`` access mode. This mode
+implements reductions within the compute kernel generated by the `parallel_for`
+construct.
+
+``reduce()`` accepts different arguments which define its behavior:
+- The first argument must be the reduction operator which defines how multiple values are combined, and how to initialize a value (for example a sum   reduction will add two values, and initialize values to 0).
+- An optional ``no_init{}`` value indicate that the result of the reduction should be accumulated to the existing value stored in the logical data, similarly to a ``rw()`` access mode. By default, if this value is not passed to ``reduce()``, the content of the logical data would be overwritten as with a ``write()`` access mode. Using ``no_init{}`` with a logical data which has no valid instance is an error (for example when the logical data is just defined from its shape and that no previous write access was made).
+- Other arguments are the same passed to other access modes such as the data place.
+
+We can only apply the ``reduce()`` access mode on logical data which data
+interface have defined the ``owning_container_of`` trait class. This is the
+case of the ``scalar_view<T>`` data interface, which sets ``owning_container_of`` to
+be ``T``. The argument passed to the ``parallel_for`` construct is a reference
+to object of this type. The following piece of code for example computes the
+dot product of two vectors of double elements (``slice<double>``) using a
+reduction, and a ``scalar_view<double>``. Reductions are typically used in
+combination with the ``wait`` mechanism which synchronously returns
+the content of the logical data in a variable.
+
+.. code-block:: cpp
+  :caption: dot product using a reduction operation
+
+  auto lsum = ctx.logical_data(shape_of<scalar_view<double>>());
+
+  /* Compute sum(x_i * y_i)*/
+  ctx.parallel_for(lY.shape(), lX.read(), lY.read(), lsum.reduce(reducer::sum<double>{}))
+      ->*[] __device__(size_t i, auto dX, auto dY, double& sum) {
+            sum += dX(i) * dY(i);
+          };
+
+  double res = ctx.wait(lsum);
+
+Note that if we had put a ``no_init{}`` argument after
+``reducer::sum<double>{}`` we would have an error because ``lsum`` was not
+initialized.
+
+Multiple reductions can be used with different operators in the same
+`parallel_for` construct.
+
+.. list-table:: Predefined Reduction Operators and Neutral Elements
+   :header-rows: 1
+
+   * - Operator Name
+     - Purpose
+     - Neutral Element
+   * - ``sum``
+     - Computes the summation of elements.
+     - ``0``
+   * - ``product``
+     - Computes the product of elements.
+     - ``1``
+   * - ``maxval``
+     - Finds the maximum value.
+     - Smallest representable value (e.g., ``-inf`` for floats).
+   * - ``minval``
+     - Finds the minimum value.
+     - Largest representable value (e.g., ``+inf`` for floats).
+   * - ``logical_and``
+     - Performs logical AND reduction.
+     - ``true``
+   * - ``logical_or``
+     - Performs logical OR reduction.
+     - ``false``
+   * - ``bitwise_and``
+     - Performs bitwise AND reduction.
+     - All bits set (e.g., ``~0`` or ``-1`` for integers).
+   * - ``bitwise_or``
+     - Performs bitwise OR reduction.
+     - ``0``
+   * - ``bitwise_xor``
+     - Performs bitwise XOR reduction.
+     - ``0``
+
+A set of predefined reduction operators are defined, but users may easily
+define their own operators. These operators are defined in the
+``cuda::experimental::stf::reducer`` namespace. The following piece of code
+for example illustrates how to implement the ``sum`` reduction operator.
+
+.. code-block:: cpp
+   :caption: Defining the sum reduction operator
+
+   template <typename T>
+   class sum
+   {
+   public:
+     static __host__ __device__ void init_op(T& dst)
+     {
+       dst = static_cast<T>(0);
+     }
+
+     static __host__ __device__ void apply_op(T& dst, const T& src)
+     {
+       dst += src;
+     }
+   };
+
+Every reduction operator should therefore define both the ``ìnit_op`` method
+that sets the neutral element, and the ``apply_op`` method which combines two
+elements. Appropriate ``__host__`` and/or ``__device__`` annotations are
+required depending where the operation may occur. The type of the arguments are
+references to ``owning_container_of<interface>::type`` where ``interface`` is
+the data interface type of the logical data.
+
+
+
+.. _launch_construct:
+
+``launch`` construct
+--------------------
+
+The ``ctx.launch`` primitive in CUDASTF is a kernel-launch mechanism
+that handles the mapping and launching of a single kernel onto execution
+places implicitly.
+
+Syntax:
+
+.. code-block:: cpp
+
+    ctx.launch([thread hierarchy spec], [execution place], logicalData1.accessMode(), logicalData2.accessMode())
+        ->*[capture list] __device__ (auto th_spec, auto data1, auto data2 ...) {
+            // Kernel implementation
+        };
+
+.. _example-with-a-1-dimensional-array-1:
+
+Example with a 1-dimensional array
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The example below illustrates processing a 1D array using ``launch``:
+
+.. code:: cpp
+
+   ctx.launch(par(1024), all_devs, handle_X.read(cdp), handle_Y.rw(cdp))->*[=] __device__(thread_info t, slice<double> x, slice<double> y) {
+       size_t tid = t.thread_id();
+       size_t nthreads = t.get_num_threads();
+       for (size_t ind = tid; ind < N; ind += nthreads) {
+         y(ind) += alpha * x(ind);
+       }
+   };
+
+The ``launch`` construct consists of five main elements:
+
+- an optional ``execution_policy`` that explicitly specifies the launch
+  shape. Here we specify that a group of 1024 independent threads should
+  execute the loop described in the body of the launch construct.
+- an execution place that indicates where the code will be executed;
+- a set of data dependencies;
+- a body of code specified using the ``->*`` operator.
+- a parameter to the kernel ``thread_info t`` for thread properties.
+
+In the example above, the kernel is launched on all of the available
+CUDA devices. The lambda function has the ``__device__`` attribute
+because a device execution place was specified. The first parameter
+corresponds to the per thread information that the user can query. This
+includes a global thread id and the total number of threads that will be
+executing the kernel. Subsequent parameters are the data instances
+associated with the logical data arguments (e.g., ``slice<double> x``).
+
+Describing a thread hierarchy
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The thread hierarchy specification describes the structure of the parallelism of this kernel. Level sizes can be computed automatically, be a dynamic value or be specified at compile-time.
+Threads in a parallel group (`par`) are executed independently.
+Threads in a concurrent group (`con`) can be synchronized using the `sync()` API which issues a group-level barrier.
+
+.. code-block:: cpp
+
+    con() // A single level of threads which are allowed to synchronize
+    par(128) // A single level of 128 threads which cannot synchronize
+    par<128>() // A single level with a statically defined size
+
+Thread are described in a hierarchical manner : we can nest multiple groups with different characteristics.
+
+.. code-block:: cpp
+
+    par(128, con<256>()) // A two-level thread hierarchy with 128 independent groups of 256 synchronizable threads..
+
+Within each group, we can provide additional information such memory automatically shared among group members.
+
+.. code-block:: cpp
+
+    con(256, mem(64)) // A group of 256 threads sharing 64 bytes of memory
+
+We can also provide some affinity information if we want to ensure that  group of threads is mapped on a specific level of the machine hierarchy.
+
+The different scopes available are :
+
+- `hw_scope::thread` : CUDA threads
+- `hw_scope::block` : CUDA blocks
+- `hw_scope::device` : CUDA device
+- `hw_scope::all` : all machine
+
+This for example describes a thread hierarchy where the inner-most level is limited to CUDA threads (i.e. it cannot span multiple CUDA blocks). And the overall kernel can be mapped at most on a single device, but not on multiple devices).
+
+.. code-block:: cpp
+
+    par(hw_scope::device | hw_scope::block, par<128>(hw_scope::thread));
+
+Manipulating a thread hierarchy
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When the `ctx.launch` construct is executed, a thread hierarchy object is passed to the function which implements the kernel. This object is available on the device(s) (assuming a device execution place), and makes it possible to query the structure of the hierarchy (e.g. based on the type of the thread hierarchy object), and allows the threads in the hierarchy to interact by the means of the `sync()` primitive for groups marked as concurrent (`con`).
+
+As an example, let us consider that we have the `par(128, con<32>())` hierarchy.
+
+.. code-block:: cpp
+
+    th.rank(); // rank of the thread within the entire hierarchy (a number between 0 and 4096)
+    th.size(); // size of the group within the entire hierarchy (4096)
+    th.rank(i); // rank of the thread within the i-th level of the hierarchy (e.g. a number between 0 and 128 for th.rank(0))
+    th.size(i); // size of the group at the i-th level of the hierarchy (128)
+    th.static_with(i) // size of the group at the i-th level if known at compile time (constexpr value)
+
+We can query if the i-th level is synchronizable or not using the following API:
+
+.. code-block:: cpp
+
+   th.is_synchronizable(i); // if i is a dynamic value
+   th.template is_synchronizable<i>(); // if i is known at compile time
+
+If the level is synchonizable, we can call
+
+.. code-block:: cpp
+
+    th.sync(i) // synchronize all threads of the i-th level
+    th.sync() // synchronize all threads of the top-most level (level 0)
+
+We can query the affinity of the i-th level with
+
+.. code-block:: cpp
+
+    th.get_scope(i); // returns the scope of the i-th level
+
+It is possible to get the amount of memory available for each level :
+
+.. code-block:: cpp
+
+    th.get_mem(i); // returns the amount of memory available at i-th level
+
+And we can retrieve the corresponding per-level buffer as a slice (which
+CUDASTF will automatically allocate in the most appropriate level of the memory
+hierarchy among shared memory, device memory or managed memory) :
+
+.. code-block:: cpp
+
+    slice<char> smem = th.template storage<char>(1);
+
+The depth of the thread hierarchy corresponds to the number of nested levels in the hierarchy :
+
+.. code-block:: cpp
+
+    th.depth(); // (constexpr) return the number of levels in the hierarchy
+
+To simplify how we navigate within hierarchies, applying the `inner()` method
+returns a thread hierarchy where the top-most level was removed. The returned specification
+will differ between the different threads which call `inner()`.
+
+.. code-block:: cpp
+
+    auto spec = con<128>(con());
+    ...
+    auto ti = th.inner();
+    ti.size(); // size within the par() hierarchy (automatically computed value)
+    ti.rank(); // rank within the par() hierarchy
+    ...
+    th.inner().sync(); // synchronize threads in the same block of the second level of the hierarchy
+
+``cuda_kernel`` construct
+-------------------------
+
+CUDASTF provides the `cuda_kernel` construct to implement tasks executing a
+CUDA kernel. This construct is especially useful when we writing code that may
+be executed using a CUDA graph backend, because its `task` construct relies on
+a graph capture mechanism which has some overhead, while the `cuda_kernel`
+construct is directly translated to CUDA kernel launch APIs, thus avoiding this
+overhead.
+
+
+`cuda_kernel` accepts the same arguments as the task construct, including an
+execution place and a list of data dependencies.  It implements a `->*`
+operator that takes a lambda function as argument.  This lambda function must
+return an object of type `cuda_kernel_desc`, describing the CUDA kernel to
+execute.  The constructor of the `cuda_kernel_desc` class, shown below, takes
+the CUDA kernel function pointer (ie. the ``__global__`` method defining the
+kernel), a grid description, the amount of dynamically allocated shared memory,
+and finally all the arguments that must be passed to the CUDA kernel.
+
+.. code:: cpp
+
+  template <typename Fun, typename... Args>
+  cuda_kernel_desc(Fun func,           // Pointer to the CUDA kernel function (__global__)
+                   dim3 gridDim_,      // Dimensions of the grid (number of thread blocks)
+                   dim3 blockDim_,     // Dimensions of each thread block
+                   size_t sharedMem_,  // Amount of dynamically allocated shared memory
+                   Args... args)       // Arguments passed to the CUDA kernel
+
+For example, the following piece of code creates a task that launches a CUDA kernel that accesses two logical data.
+
+.. code:: cpp
+
+  ctx.cuda_kernel(lX.read(), lY.rw())->*[&](auto dX, auto dY) {
+    // calls __global__ void axpy(double a, slice<const double> x, slice<double> y);
+    // similarly to axpy<<<16, 128, 0, ...>>>(alpha, dX, dY)
+    return cuda_kernel_desc{axpy, 16, 128, 0, alpha, dX, dY};
+  };
+
+Similar to the `task` construct, the `cuda_kernel` construct also supports
+specifying dynamic dependencies using the `add_deps` method and retrieving data
+instances using `get`. The previous code can therefore be rewritten as:
+
+.. code:: cpp
+
+  auto t = ctx.cuda_kernel();
+  t.add_deps(lX.read());
+  t.add_deps(lY.rw());
+  t->*[&]() {
+    auto dX = t.template get<slice<double>>(0);
+    auto dY = t.template get<slice<double>>(1);
+    return cuda_kernel_desc{axpy, 16, 128, 0, alpha, dX, dY};
+  };
+
+``cuda_kernel_chain`` construct
+-------------------------------
+
+In addition to `cuda_kernel`, CUDASTF provides the `cuda_kernel_chain`
+construct to execute sequences of CUDA kernels within a single task. Unlike
+`cuda_kernel`, which expects a single kernel descriptor, the lambda passed to
+the `->*` operator of `cuda_kernel_chain` should return a
+`::std::vector<cuda_kernel_desc>` describing multiple kernel launches.
+Kernels specified within the vector are executed sequentially in the order they appear.
+
+The following two constructs are therefore equivalent, except that the
+`cuda_kernel_chain` implementation directly translate to efficient, direct CUDA
+kernel launch APIs, while the implementation of the `task` construct may rely
+on graph capture when using a CUDA graph backend.
+
+.. code:: cpp
+
+  /* Compute Y = Y + alpha X, Y = Y + beta X, then Y = Y + gamma X sequentially */
+  ctx.cuda_kernel_chain(lX.read(), lY.rw())->*[&](auto dX, auto dY) {
+     return ::std::vector<cuda_kernel_desc> {
+         { axpy, 16, 128, 0, alpha, dX, dY },
+         { axpy, 16, 128, 0, beta,  dX, dY },
+         { axpy, 16, 128, 0, gamma, dX, dY }
+     };
+  };
+
+  /* Equivalent to the previous construct, but possibly less efficient */
+  ctx.task(lX.read(), lY.rw())->*[&](cudaStream_t stream, auto dX, auto dY) {
+     axpy<<<16, 128, 0, stream>>>(alpha, dX, dY);
+     axpy<<<16, 128, 0, stream>>>(beta,  dX, dY);
+     axpy<<<16, 128, 0, stream>>>(gamma, dX, dY);
+  };
+
+Similarly to the `cuda_kernel` constructs, dependencies can be set dynamically:
+
+.. code:: cpp
+
+  /* Compute Y = Y + alpha X, Y = Y + beta X, then Y = Y + gamma X sequentially */
+  auto t = ctx.cuda_kernel_chain();
+  t.add_deps(lX.read());
+  t.add_deps(lY.rw());
+  t->*[&]() {
+    auto dX = t.template get<slice<double>>(0);
+    auto dY = t.template get<slice<double>>(1);
+    return ::std::vector<cuda_kernel_desc> {
+        { axpy, 16, 128, 0, alpha, dX, dY },
+        { axpy, 16, 128, 0, beta, dX, dY },
+        { axpy, 16, 128, 0, gamma, dX, dY }
+    };
+  };
+
+C++ Types of logical data and tasks
+-----------------------------------
+
+To prevent a common class of errors, CUDASTF strives to align its
+processing semantics with C++ types as closely as possible. As shown in
+the various examples, the use of the ``auto`` keyword is usually
+recommended to create readable code while type safety is still enforced.
+
+.. _logical-data-1:
+
+Logical data
+^^^^^^^^^^^^
+
+The result of calling ``ctx.logical_data()`` is an object whose type
+contains information about the underlying data interface used to
+manipulate the logical data object. For example, a contiguous array of
+``double`` is internally represented as a ``slice`` (which is an alias
+of ``std::experimental::mdspan``) so that we can use the following type:
+
+.. code:: cpp
+
+   double X[16];
+   logical_data<slice<double>> lX = ctx.logical_data(X);
+
+For simplicity and without losing any information, users can typically
+rely on the ``auto`` keyword:
+
+.. code:: cpp
+
+   double X[16];
+   auto lX = ctx.logical_data(X);
+
+One may for example store the logical data of a ``slice<int>`` in a C++
+class or structure in such as way:
+
+.. code:: cpp
+
+   class foo {
+      ...
+      mutable logical_data<slice<int>> ldata;
+   };
+
+Note the use of the ``mutable`` qualifier because a task accessing a
+``const foo`` object might want to read the ``ldata`` field. Submitting a
+task that use this logical data in read only mode would modify the
+internal data structures of the logical data, but should probably appear
+as a ``const`` operation from user's perspective. Without this ``mutable``
+qualifier, we could not have a ``const`` qualifier on the ``f`` variable
+in the following code :
+
+.. code:: cpp
+
+   void func(context &ctx, const foo &f) {
+       ctx.task(f.ldata.read())->*[](cudaStream_t stream, auto) {
+           ... do work ...
+       };
+   }
+
+Tasks
+^^^^^
+
+With a ``stream_ctx`` backend, ``ctx.task(lX.read(), lY.rw())`` returns
+an object of type ``stream_task<TX, TY>``, where the template arguments
+``TX`` and ``TY`` are the types associated to the data interfaces in
+logical data ``lX`` and ``lY``. Assuming two arrays of ``double``, which
+CUDASTF internally manages as ``slice<double>`` objects, the type of
+this task will be:
+
+.. code:: cpp
+
+   stream_task<slice<const double>, slice<double>>
+
+The type of the task contains information about the element type and its
+modifiability — read-only access is mapped to a slice of
+``const double`` as opposed to ``double``. The type information is
+propagated further from the task object to the lambda invoked by means
+of ``operator->*`` in such a way that type errors are detected during
+compilation.
+
+.. code:: cpp
+
+   double X[16], Y[16];
+   logical_data<slice<double>> lX = ctx.logical_data(X);
+   logical_data<slice<double>> lY = ctx.logical_data(Y);
+
+   // results in a compilation error due to the erroneous slice<int> type
+   ctx.task(lX.read(), lY.rw())->*[](cudaStream_t s, slice<int> x, slice<int> y) {
+       ...
+   };
+
+In most cases, it's recommended to use the ``auto`` C++ keyword to
+automatically obtain the correct data types:
+
+.. code:: cpp
+
+   double X[16], Y[16];
+   auto lX = ctx.logical_data(X);
+   auto lY = ctx.logical_data(Y);
+
+   ctx.task(lX.read(), lY.rw())->*[](cudaStream_t s, auto x, auto y) {
+       ...
+   };
+
+In the graph backend, the untyped task type equivalent to
+``stream_task<>`` is ``graph_task``, and the equivalent to
+``stream_task<T1, T2>`` would be, for example, ``graph_task<T1, T2>``.
+When using the generic context type, CUDASTF would create a task of type
+``unified_task<T1, T2>``.
+
+Dynamically-typed tasks
+^^^^^^^^^^^^^^^^^^^^^^^
+
+In certain circumstances, the exact data accessed by a task (and
+consequently the type of a task as discussed above) may not be available
+statically. For example, updating a part of the computation domain might
+require accessing the closest neighbors of that part. The neighbors are
+known only dynamically, meaning that it is not possible to directly pass
+task dependencies as arguments to the ``ctx.task()`` call.
+
+For such situations CUDASTF offers a dynamically-typed task, called
+``stream_task<>`` in the ``stream_ctx`` backend, whose member function
+``add_deps`` allows adding dependencies dynamically:
+
+.. code:: cpp
+
+   double X[16], Y[16];
+   auto lX = ctx.logical_data(X);
+   auto lY = ctx.logical_data(Y);
+
+   stream_task<> t = ctx.task();
+   t.add_deps(lX.read(), lY.rw());
+
+This dynamic approach entails a loss of expressiveness. The API based on
+the ``->*`` notation is only compatible with *statically-typed* tasks,
+as the user-provided lambda function needs to be passed data instances
+of the proper types (for example ``slice<double>``) by CUDASTF. As a
+consequence, the ``stream_task<>`` needs to be manipulated with the
+`low-level API <stf/lower_level_api.rst>`__.
+
+Combining typed and untyped tasks
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+It is possible to dynamically add dependencies to a typed task, but the
+type of the task will not reflect the dynamically added dependencies.
+This allows for combining the low-level API with the ``->*`` notation in
+the following way:
+
+.. code:: cpp
+
+   double X[16], Y[16];
+   auto lX = ctx.logical_data(X);
+   auto lY = ctx.logical_data(Y);
+
+   auto t = ctx.task(lX.read());
+   t.add_deps(lY.rw());
+   t->*[&](cudaStream_t s, auto x) {
+      slice<double> y = t.template get<slice<double>>(1);
+   };
+
+The program remains safe because accesses are checked dynamically.
+However, any errors will be caught at runtime instead of during
+compilation.
+
+Untyped tasks cannot be converted to typed tasks. On the other hand,
+typed tasks can be converted implicitly to untyped tasks (thus losing
+all the benefits of statically available types):
+
+.. code:: cpp
+
+   stream_task<> t = ctx.task(lX.read());
+
+Modular use of CUDASTF
+----------------------
+
+CUDASTF maintains data consistency throughout a context and infers
+concurrency opportunities based on data accesses. An existing application
+may however already manage coherency or enforce dependencies by other means.
+CUDASTF offers several facilities that facilitate incremental adoption
+in existing code.
+
+- The ``logical data freezing`` mechanism ensures data availability while letting
+  the application take care of synchronization.
+- ``Tokens`` make it possible to enforce concurrent execution while
+  letting the application manage data allocations and data transfers.
+- :ref:`Execution places <cudax-places>` can be used without tasks for example to
+  automate the management of CUDA streams, set the current execution context,
+  or allocate memory. See the :ref:`cudax-places` documentation for details on
+  standalone usage including :ref:`stream management <places-stream-management>`,
+  :ref:`device activation <places-activate>`, and
+  :ref:`memory allocation <places-memory-allocation>`.
+
+Freezing logical data
+^^^^^^^^^^^^^^^^^^^^^
+
+When a piece of data is used very often, it can be beneficial to avoid enforcing
+data dependencies every time it is accessed. A common example would be data that
+is written once and then read many times.
+
+CUDASTF provides a mechanism called ``logical data freeze`` that allows a
+logical data to be accessed outside of tasks—or within tasks—without
+enforcing data dependencies for every access, which minimizes overhead.
+
+By default, calling the ``freeze`` method returns a frozen logical data object
+that can be accessed in read-only mode without additional synchronization. The
+``get`` method of the frozen logical data returns a view of the underlying data
+on the specified data place. This view can be used asynchronously with respect
+to the stream passed to ``get`` until calling the non-blocking ``unfreeze``
+method on the frozen logical data. It is possible to call ``get`` multiple times.
+Modifying these frozen read-only views results in undefined behavior.
+If necessary, implicit data transfers or allocations are performed asynchronously
+when calling ``get``.
+
+.. code:: cpp
+
+    auto frozen_ld = ctx.freeze(ld);
+    auto dX = frozen_ld.get(data_place::current_device(), stream);
+    kernel<<<..., stream>>>(dX);
+
+    // Get a read-only copy of the frozen data on other data places
+    auto dX1 = frozen_ld.get(data_place::device(1), stream);
+    auto hX = frozen_ld.get(data_place::host(), stream);
+
+    frozen_ld.unfreeze(stream);
+
+While data are frozen, it is still possible to launch tasks which access
+them. CUDASTF will allow tasks with a read access modes to run
+concurrently before ``unfreeze`` is called, but it will defer write accesses
+until data is made is made modifiable again, after ``unfreeze``.
+
+.. code:: cpp
+
+    auto frozen_ld = ctx.freeze(ld, access_mode::rw, data_place::current_device());
+    auto dX = frozen_ld.get(data_place::current_device(), stream);
+    // kernel can modify dX
+    kernel<<<..., stream>>>(dX);
+    frozen_ld.unfreeze(stream);
+
+As shown above, it is also possible to create a modifiable frozen logical data,
+allowing an application to temporarily transfer ownership of the logical data
+to code that does not use tasks.  Because no further synchronization is
+performed to ensure the consistency of this logical data once it is frozen,
+users need to specify where the view of the data is needed.  Any tasks that
+access this modifiable frozen logical data will be deferred until ``unfreeze``
+is called.
+
+It is not possible to freeze the same logical data concurrently. Therefore, we
+need to call ``unfreeze`` before calling ``freeze`` again, and it is the
+programmer's responsibility to ensure that the stream passed to ``freeze``
+depends on the completions of all operations in the stream previously passed to
+``unfreeze``.
+
+It is possible to use different streams in the ``freeze``, ``get`` and
+``unfreeze`` methods. However it is also programmer's responsibility to ensure
+that the stream passed to ``get`` depends on the completion of the work in the
+stream passed to ``freeze`` (for example, by using a blocking call such as
+``cudaStreamSynchronize``). Similarly, the stream passed to ``unfreeze`` must
+depend on the completion of the work in the streams used for any preceding
+``freeze`` and ``get`` calls.
+
+It is possible to retrieve the access mode used to freeze a logical data with
+the ``get_access_mode()`` method of the ``frozen_logical_data`` object.
+
+.. _stf-tokens:
+
+Tokens
+^^^^^^
+
+A token is a specific type of logical data whose only purpose is to
+automate synchronization, while letting the application manage the actual data.
+This can, for example, be useful with user-provided buffers on a single device,
+where no allocations or transfers are required, but where concurrent accesses
+may occur.
+
+A token internally relies on the ``void_interface`` data interface,
+which is specifically optimized to skip unnecessary stages in the cache
+coherency protocol (e.g., data allocations or copying data). When appropriate,
+using a token rather than a logical data with a full-fledged data
+interface therefore minimizes runtime overhead.
+
+.. code:: cpp
+
+    auto token = ctx.token();
+
+    // A and B are assumed to be two other valid logical data
+    ctx.task(token.rw(), A.read(), B.rw())->*[](cudaStream_t stream, auto a, auto b)
+    {
+        ...
+    };
+
+The example above shows how to create a token and how to use it in a
+task.
+
+Since the token is only used for synchronization purposes, the
+corresponding argument may be omitted in the lambda function passed as the
+task's implementation. Thus, the above task is equivalent to this code:
+
+.. code:: cpp
+
+    ctx.task(token.rw(), A.read(), B.rw())->*[](cudaStream_t stream, void_interface dummy, auto a, auto b)
+
+To avoid ambiguities, you must either consistently ignore every
+``void_interface`` data instance or include them all, even if they remain
+unused. Eliding these token arguments is possible in the ``ctx.task`` and
+``ctx.host_launch`` constructs.
+
+Note that the token created by the ``token`` method of the context
+object is already valid, which means the first access can be either a ``read()``
+or an ``rw()`` access. There is no need to set any content in the token
+(unlike a logical data object created from a shape).
+
+A token corresponds to a ``logical_data<void_interface>`` object, so that the
+``token`` type serves as a short-hand for this type. ``ctx.token()`` thus
+returns an object with a ``token`` type.
+
+The blocking ``ctx.wait`` method (see :ref:`the synchronization section
+<stf-synchronization>`) can also be called on a token. Because a token has no
+content to materialize, ``ctx.wait(token)`` returns ``void`` and simply blocks
+the calling host thread until the work the token depends on has completed. This
+is convenient to synchronize the host with a phase of computation expressed with
+tokens before performing host-side operations on externally managed state (for
+example, operations that are not suitable for a ``host_launch`` callback):
+
+.. code:: cpp
+
+    auto token = ctx.token();
+
+    ctx.task(token.rw())->*[&](cudaStream_t stream) {
+        // launch some asynchronous work on user-provided buffers
+    };
+
+    // Block until the work above has completed.
+    ctx.wait(token);
+
+    // The host can now safely operate on the externally managed state.
+
+Debugging
+---------
+
+Enabling internal checks
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+CUDASTF includes internal assertions (``_CCCL_ASSERT``) that help detect
+programming errors and invalid usage patterns during development. These checks
+are disabled by default for performance but can be enabled to aid debugging.
+
+**With CMake:**
+
+When building in Debug mode, assertions are enabled automatically:
+
+.. code:: bash
+
+   cmake -DCMAKE_BUILD_TYPE=Debug ..
+
+To explicitly enable assertions for any build type, add the compile definition
+to your target:
+
+.. code:: cmake
+
+   target_compile_definitions(your_target PRIVATE CCCL_ENABLE_ASSERTIONS)
+
+**With Makefile or manual compilation:**
+
+Add the ``-DCCCL_ENABLE_ASSERTIONS`` flag to your compiler invocation:
+
+.. code:: bash
+
+   # For nvcc
+   nvcc -DCCCL_ENABLE_ASSERTIONS ...
+
+   # For host compiler
+   g++ -DCCCL_ENABLE_ASSERTIONS ...
+
+Note that this flag enables the assertion checks themselves. For full debugging
+support (setting breakpoints, inspecting variables), you may also want to add
+debug symbol flags (``-g`` for host code, ``-G`` for device code).
+
+Tools
+-----
+
+Visualizing task graphs
+^^^^^^^^^^^^^^^^^^^^^^^
+
+In order to visualize the task graph generated by CUDASTF, it is
+possible to generate a file in the Graphviz format. This visualization
+helps to better understand the application, and can be helpful to
+optimize the algorithms as it sometimes allow to identify inefficient
+patterns.
+
+Generating visualizations of task graphs
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Let us consider the ``examples/01-axpy.cu`` example which we compile as
+usual with ``make build/examples/01-axpy``.
+
+.. code:: bash
+
+   # Run the application with CUDASTF_DOT_FILE set to the filename
+   CUDASTF_DOT_FILE=axpy.dot build/examples/01-axpy
+
+   # Generate the visualization from this dot file
+   ## PDF format
+   dot -Tpdf axpy.dot -o axpy.pdf
+   ## PNG format
+   dot -Tpng axpy.dot -o axpy.png
+
+We obtain a visualization like this, where we only see a single task
+with little :
+
+.. image:: stf/images/dot-output-axpy.png
+
+To have more information, we can enhance the application with some extra
+debugging information. For example, we can specify what is the name of a
+logical data using the ``set_symbol`` method of the ``logical_data``
+class. As illustrated here :
+
+.. code:: c++
+
+   auto lX = ctx.logical_data(X).set_symbol("X");
+   auto lY = ctx.logical_data(Y);
+   lY.set_symbol("Y");
+
+We can also annotate tasks with symbols. Instead of writing this :
+
+.. code:: c++
+
+       ctx.task(lX.read(), lY.rw())->*[&](cudaStream_t s, auto dX, auto dY) { axpy<<<16, 128, 0, s>>>(alpha, dX, dY); };
+
+We can write code like this :
+
+.. code:: c++
+
+       // Inlined notation
+       ctx.task(lX.read(), lY.rw()).set_symbol("axpy")->*[&](cudaStream_t s, auto dX, auto dY) { axpy<<<16, 128, 0, s>>>(alpha, dX, dY); };
+
+       // Explicit manipulation of the task class
+       auto t = ctx.task(lX.read(), lY.rw());
+       t.set_symbol("axpy");
+       t->*[&](cudaStream_t s, auto dX, auto dY) { axpy<<<16, 128, 0, s>>>(alpha, dX, dY); };
+
+We then obtain the following output, which contains more interesting
+annotations :
+
+.. image:: stf/images/dot-output-axpy-annotated.png
+
+On a more elaborated application, such as the ``examples/heat_mgpu.cu``
+example, we can easily understand the overall workflow thanks to this
+visualization.
+
+.. code:: bash
+
+   CUDASTF_DOT_FILE=heat.dot build/examples/heat_mgpu 1000 8 4
+   dot -Tpng heat.dot -o heat.png
+
+.. image:: stf/images/dot-output-heat.png
+
+For advanced users, it is also possible to display internally generated
+asynchronous operations by setting the ``CUDASTF_DOT_IGNORE_PREREQS``
+environment variable to 0.
+
+.. code:: c++
+
+   CUDASTF_DOT_IGNORE_PREREQS=0 CUDASTF_DOT_FILE=axpy-with-events.dot build/examples/01-axpy
+   dot -Tpng axpy-with-events.dot -o axpy-with-events.png
+
+.. image:: stf/images/dot-output-axpy-events.png
+
+It is possible to color the different tasks accordingly to the device
+executing it by setting the ``CUDASTF_DOT_COLOR_BY_DEVICE`` environment
+variable.
+
+To reduce the amount of information displayed in the graph, we can
+remove the list of data associated to each task by setting the
+``CUDASTF_DOT_REMOVE_DATA_DEPS`` environment variable.
+
+It is also possible to include timing information in this graph by setting the
+``CUDASTF_DOT_TIMING`` environment variable to a non-null value. This will
+color the graph nodes according to their relative duration, and the measured
+duration will be included in task labels.
+
+Condensed and structured graphs visualization
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Realistic workloads are typically made of thousands or millions of tasks which
+cannot be easily visualized using graphviz (dot). To simplify the generated
+graphs we can further
+annotate the application using dot sections.
+Dot sections can also be nested to better structure the visualization.
+
+This is achieved by creating `dot_section` objects in the application. `ctx.dot_section` returns
+an object whose lifetime defines a dot section valid until it is destroyed, or
+when calling the `end()` method on this object. The following example
+illustrates how to add nested sections:
+
+.. code:: c++
+
+    context ctx;
+    auto lA = ctx.token().set_symbol("A");
+    auto lB = ctx.token().set_symbol("B");
+    auto lC = ctx.token().set_symbol("C");
+
+    // Begin a top-level section named "foo"
+    auto s_foo = ctx.dot_section("foo");
+    for (size_t i = 0; i < 2; i++)
+    {
+      // Section named "bar" using RAII
+      auto s_bar = ctx.dot_section("bar");
+      ctx.task(lA.read(), lB.rw()).set_symbol("t1")->*[](cudaStream_t, auto, auto) {};
+      for (size_t j = 0; j < 2; j++) {
+         // Section named "baz" using RAII
+         auto s_bar = ctx.dot_section("baz");
+         ctx.task(lA.read(), lC.rw()).set_symbol("t2")->*[](cudaStream_t, auto, auto) {};
+         ctx.task(lB.read(), lC.read(), lA.rw()).set_symbol("t3")->*[](cudaStream_t, auto, auto, auto) {};
+         // Implicit end of section "baz"
+      }
+      // Implicit end of section "bar"
+    }
+    s_foo.end(); // Explicit end of section "foo"
+    ctx.finalize();
+
+When running this with the `CUDASTF_DOT_FILE` environment variable for example
+set to `dag.dot`, we observe that the graph produced by `dot -Tpdf dag.dot -o
+dag.pdf` depicts these sections as dashed boxes.
+
+.. image:: stf/images/dag-sections.png
+
+Adding sections also makes it possible to define a maximum depth for the
+generated graphs by setting the `CUDASTF_DOT_MAX_DEPTH` environment variable.
+When it is undefined, CUDASTF will display all tasks. Otherwise, if
+`CUDASTF_DOT_MAX_DEPTH` is an integer value of `i` any sections and tasks which
+nesting level is deeper than `i` will be collapsed.
+
+When setting `CUDASTF_DOT_MAX_DEPTH=2`, the previous graph becomes:
+
+.. image:: stf/images/dag-sections-2.png
+
+When setting `CUDASTF_DOT_MAX_DEPTH=1`, one additional level is collapsed:
+
+.. image:: stf/images/dag-sections-1.png
+
+With `CUDASTF_DOT_MAX_DEPTH=0`, only the top-most tasks and sections are displayed:
+
+.. image:: stf/images/dag-sections-0.png
+
+Note that `CUDASTF_DOT_MAX_DEPTH` and `CUDASTF_DOT_TIMING` can be used in
+combination, and that the duration of a section corresponds to the duration of
+all tasks in this sections.
+
+Kernel tuning with ncu
+^^^^^^^^^^^^^^^^^^^^^^
+
+Users can analyze the performance of kernels generated using
+``ctx.parallel_for`` and ``ctx.launch`` using the ``ncu`` tool.
+
+Naming kernels
+~~~~~~~~~~~~~~
+
+However, displayed kernel names would be hardly exploitable as they
+would all have the same name. One possible work-around is to let ``ncu``
+rename kernels accordingly to ``NVTX`` annotations. To do so, a symbol
+must be associated to the ``ctx.parallel_for`` and ``ctx.launch``
+constructs using the ``set_symbol`` method. In the following example, we
+name the generated kernel "updateA" :
+
+.. code:: cpp
+
+   int A[128];
+   auto lA = ctx.logical_data(A);
+
+   ctx.parallel_for(lA.shape(), lA.write()).set_symbol("updateA")->*[] __device__ (size_t i, auto sA) {
+       A(i) = 2*i + 1;
+   };
+
+Example with miniWeather
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Kernel tuning should always be performed on optimized code :
+
+.. code:: bash
+
+   make build/examples/miniweather
+
+The following command will analyse the performance of kernels :
+
+.. code:: bash
+
+   ncu --section=ComputeWorkloadAnalysis --print-nvtx-rename=kernel --nvtx -o output build/examples/miniWeather
+
+Note that ``--print-nvtx-rename=kernel --nvtx`` is used to name kernels
+accordingly to ``NVTX`` traces (which are enabled by the ``set_symbol``
+API). Failing to do so would results in all kernel names being
+``thrust::cuda_cub::core::_kernel_agent``. To properly display renamed
+kernel names, users may have to set option
+``Options->Report UI->NVTX Rename Mode`` to a value equal to ``Kernel``
+or ``All``.
+
+Depending on machine configuration, users may also have to execute the
+``ncu`` command as root or to setup their machine accordingly. This is
+for example required when the following message is appears during
+``ncu`` trace collection :
+
+::
+
+   ==ERROR== ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device 0. For instructions on enabling permissions and to get more information see https://developer.nvidia.com/ERR_NVGPUCTRPERM
+
+The file generated by ``ncu`` can be opened using ``ncu-ui`` :
+
+.. code:: bash
+
+   ncu-ui output.ncu-rep
+
+In this case, we can see that the kernel are named accordingly to the
+symbols set in the tasks of the miniWeather examples : |image1|
+
+.. |image1| image:: stf/images/ncu-ui.png
+
+CUDASTF Reference Card
+----------------------
+
+This section gives a brief overview of the CUDASTF API.
+
+Using CUDASTF
+^^^^^^^^^^^^^
+
+STF is a C++ header-only library which API is defined in the `cuda::experimental::stf` namespace.
+
+.. code-block:: cpp
+
+    #include <cudastf/cudastf.h>
+
+    using cuda::experimental::stf;
+
+Creating a Context
+^^^^^^^^^^^^^^^^^^
+
+Contexts store the state of the CUDASTF library and are used as an entry point for all API calls. The `finalize()` method must be called upon completion.
+
+.. code-block:: cpp
+
+    context ctx;
+    ctx.finalize();
+
+By default, this relies on the ``stream_ctx`` back-end, but it is also possible to select at runtime a ``graph_ctx`` back-end :
+
+.. code-block:: cpp
+
+    context ctx = graph_ctx();
+
+Or one may statically select either backends :
+
+.. code-block:: cpp
+
+    stream_ctx ctx;
+    graph_ctx ctx;
+
+Logical Data
+^^^^^^^^^^^^
+
+Creating a Logical Data
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Purpose: Encapsulates data structures (e.g. arrays, slices) to be shared and accessed by tasks. Logical data represents the abstraction of data in the model.
+
+.. code-block:: cpp
+
+    // Create a logical data from an existing piece of data
+    auto ctx.logical_data(data view [data_place = data_place::current_device()]);
+
+Examples:
+
+- Describing an array
+
+.. code-block:: cpp
+
+    int X[1024];
+    auto lX = ctx.logical_data(X);
+
+- Describing a vector of n elements of type T located at address `addr`
+
+.. code-block:: cpp
+
+    auto data_handle = ctx.logical_data(slice<T>(addr {n}));
+
+- Describing a contiguous matrix of size (m, n)
+
+.. code-block:: cpp
+
+    auto data_handle = ctx.logical_data(slice<T, 2>(addr, {m, n}));
+
+- Describing a matrix of size (m, n) with a stride of ld elements
+
+.. code-block:: cpp
+
+    auto data_handle = ctx.logical_data(slice<T, 2>(addr, {m, n}, {ld}));
+
+- Create a logical data from a shape
+
+.. code-block:: cpp
+
+    auto ctx.logical_data(shape);
+
+Examples:
+
+.. code-block:: cpp
+
+    auto lX = ctx.logical_data(shape_of<slice<int>>(1024));
+    auto lY = ctx.logical_data(lX.shape());
+
+Tasks
+^^^^^
+
+Data Dependency
+~~~~~~~~~~~~~~~
+
+Purpose: Define how a logical data should be used in a task construct (and derivated constructs such as `parallel_for`, `launch`, `host_launch`).
+
+Syntax:
+
+.. code-block:: cpp
+
+    logicalData.accessMode([data place])
+
+- **Data Places** (see :ref:`cudax-places`): Specify where a logical data in the data dependencies should be located:
+
+  - `data_place::affine()` (default): Locate data on the data place affine to the execution place (e.g., device memory when running on a CUDA device).
+  - `data_place::managed()`: Use managed memory.
+  - `data_place::device(i)`: Put data in the memory of the i-th CUDA device (which may be different from the current device or the device of the execution place).
+
+- **Access Modes**:
+
+  - `.read()`: Read-only access.
+  - `.write()`: Write-only access.
+  - `.rw()`: Read and write access.
+
+Task Creation
+~~~~~~~~~~~~~
+
+Purpose: Define computational tasks that operate on logical data. Tasks can specify data dependencies and access modes.
+
+Syntax:
+
+.. code-block:: cpp
+
+    ctx.task([execution place] dependency1 dependency2 ...)
+        ->*[&](cudaStream_t stream, auto data1, auto data2 ...) {
+            // Task implementation using stream
+        };
+
+- **Execution Place** (see :ref:`cudax-places`): Specify where the task should be executed:
+
+  - `exec_place::current_device()` (default): Run on current CUDA device.
+  - `exec_place::device(ID)`: Run on CUDA device identified by its index.
+  - `exec_place::host()`: Run on the host (Note: this is providing a CUDA stream which should be used to submit CUDA callbacks. For example, users should typically use the `host_launch` API instead).
+
+Examples:
+
+.. code-block:: cpp
+
+    ctx.task(lX.read(), lY.rw())
+        ->*[&](cudaStream_t s, auto dX, auto dY) {
+            axpy<<<16, 128, 0, s>>>(alpha, dX, dY);
+        };
+
+Host-Side Task Execution with `host_launch`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Purpose: Execute tasks on the host (CPU) while still utilizing the task and data management system.
+
+.. code-block:: cpp
+
+    ctx.host_launch(logicalData1.accessMode(), logicalData2.accessMode() ...)
+        ->*[capture list](auto data1, auto data2 ...) {
+            // Host-based task implementation here
+        };
+
+Kernel authoring
+^^^^^^^^^^^^^^^^
+
+In addition to user-provided CUDA kernels or CUDA libraries, CUDASTF makes it possible to author compute kernels directly on top of logical data using the `parallel_for` and `launch` mechanisms.
+
+`parallel_for` construct
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Purpose: Apply a kernel on each coordinate of a shape.
+
+Syntax:
+
+.. code-block:: cpp
+
+    ctx.parallel_for([execution place], [partitioner], shape, logicalData1.accessMode(), logicalData2.accessMode(), ...)
+        ->*[capture list] __device__ (size_t index1, size_t index2, ... auto data1, auto data2 ...) {
+            // Kernel implementation
+        };
+
+Examples:
+
+- Applying a kernel on a 1D array:
+
+.. code-block:: cpp
+
+    double X[N];
+    double Y[N];
+
+    auto lX = ctx.logical_data(X);
+    auto lY = ctx.logical_data(Y);
+
+    ctx.parallel_for(lY.shape(), lX.read(), lY.rw())
+        ->*[alpha] __device__(size_t i, auto dX, auto dY) {
+            dY(i) += alpha * dX(i);
+        };
+
+- Applying a kernel on a 2D matrix:
+
+.. code-block:: cpp
+
+    double X[N*N];
+    double Y[N];
+
+    auto lX = ctx.logical_data(make_slice(&X[0], std::tuple{N, N}));
+    auto lY = ctx.logical_data(Y);
+
+    ctx.parallel_for(lX.shape(), lX.rw(), lY.read())
+        ->*[](size_t i, size_t j, auto dX, auto dY) {
+            dX(i, j) += dY(i) * dY(j);
+        };
+
+See :ref:`this section <parallel_for_construct>` for more details.
+
+`launch` construct
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The `launch` construct makes it possible to launch structured compute kernels. Contrary to `parallel_for` which applies the same operation on every member of a shape, `launch` executes a kernel over a thread hierarchy.
+
+Syntax:
+
+.. code-block:: cpp
+
+    template <typename thread_hierarchy_spec_t>
+    ctx.launch([thread hierarchy spec], [execution place], logicalData1.accessMode(), logicalData2.accessMode(), ...)
+        ->*[capture list] __device__ (thread_hierarchy_spec_t th, auto data1, auto data2 ...) {
+            // Kernel implementation
+        };
+
+
+The thread hierarchy passed to the construct is defined as a nested object of the form
+
+.. code-block:: cpp
+
+    {par|con}<[static_width]>([width], [scope], [mem(size)] [nested hierarchy specification])
+
+When passed to the user-provided kernel implementation, this object provides different mechanisms to let threads interact or to query the structure of the parallelism
+
+.. code-block:: cpp
+
+    th.rank(); // get thread rank within the entire hierarchy
+    th.rank(i); // get thread rank at i-th level
+    th.template rank<i>(); // (constexpr) get thread rank at the i-th level at compile time
+
+    th.size(); // get the total number of threads
+    th.size(i); // get the number of threads at the i-th level
+    th.template size<i>(); // (constexpr) get the number of threads at the i-th level at compile time
+
+    th.get_scope(i); // get the affinity of the i-th level
+    th.template storage<T>(i); // get the local storage associated to the i-th level as a slice<T>
+
+    th.sync(); // issue a barrier among all threads
+    th.sync(i); // issue a barrier among threads of the i-th level
+    th.is_synchronizable(i); // check if we can call sync(i)
+    th.template is_synchronizable<i>(); // (constexpr) check if we can call sync(i) at compile time
+
+    th.depth(); // (constexpr) get the depth of the thread hierarchy
+    th.inner(); // get the thread hierarchy subset obtained by removing the top-most level of the hierarchy
+
+Examples:
+
+.. code-block:: cpp
+
+    con(128);
+    par();
+    con<128>();
+    con(128, par(32));
+    con(128, mem(64), con(32));
+    par(hw_scope::device | hw_scope::block, par<128>(hw_scope::thread));
+
+See :ref:`this section <launch_construct>` for more details.
diff --git a/cccl_upstream/docs/cudax/stf/custom_data_interface.rst b/cccl_upstream/docs/cudax/stf/custom_data_interface.rst
new file mode 100644
index 00000000..fb873509
--- /dev/null
+++ b/cccl_upstream/docs/cudax/stf/custom_data_interface.rst
@@ -0,0 +1,330 @@
+.. _stf_custom_data_interface:
+
+CUDASTF offers an extensible API that allows users to implement their
+own data interface.
+
+Let us for example go through the different steps to implement a data
+interface for a very simple simple implementation of a matrix class.
+
+For the sake of simplicity, we here only consider the CUDA stream
+backend, but adding support for the CUDA graph backend simply require
+some extra steps which use the CUDA graph API.
+
+Implementation of the ``matrix`` class
+======================================
+
+For the sake of simplicity, we consider a very simple representation of
+matrix, only defined by the dimensions m and n, and by the base address
+of the matrix which we assume to be contiguous.
+
+.. code:: c++
+
+   template <typename T>
+   class matrix {
+   public:
+       matrix(size_t m, size_t n, T* base) : m(m), n(n), base(base) {}
+       __host__ __device__ T& operator()(size_t i, size_t j) { return base[i + j * m]; }
+       __host__ __device__ const T& operator()(size_t i, size_t j) const { return base[i + j * m]; }
+       size_t m, n;
+       T* base;
+   };
+
+Defining the shape of a matrix
+==============================
+
+The first step consists in defining what is the *shape* of a matrix. The
+shape of a matrix should be a class that defines all parameters which
+are the same for all data instances, ``m`` and ``n``. On the other hand,
+the base address should not be part of this shape class, because each
+data instance will have its own base address.
+
+To define what is the shape of a matrix, we need to specialize the
+``cudastf::shape_of`` trait class.
+
+.. code:: c++
+
+   template <typename T>
+   class cudastf::shape_of<matrix<T>> {
+   public:
+       /**
+        * @brief The default constructor.
+        *
+        * All `shape_of` specializations must define this constructor.
+        */
+       shape_of() = default;
+
+       explicit shape_of(size_t m, size_t n) : m(m), n(n) {}
+
+       /**
+        * @name Copies a shape.
+        *
+        * All `shape_of` specializations must define this constructor.
+        */
+       shape_of(const shape_of&) = default;
+
+       /**
+        * @brief Extracts the shape from a matrix
+        *
+        * @param M matrix to get the shape from
+        *
+        * All `shape_of` specializations must define this constructor.
+        */
+       shape_of(const matrix<T>& M) : shape_of<matrix<T>>(M.m, M.n) {}
+
+       /// Mandatory method : defined the total number of elements in the shape
+       size_t size() const { return m * n; }
+
+       size_t m;
+       size_t n;
+   };
+
+We here see that ``shape_of<matrix<T>>`` contains two ``size_t`` fields
+``m`` and ``n``.
+
+In addition, we need to define a default constructor and a copy
+constructors.
+
+To implement the ``.shape()`` member of the ``logical_data`` class, we
+need to define a constructor which takes a const reference to a matrix.
+
+Finally, if the ``ctx.parallel_for`` construct is needed, we must define
+a ``size_t size() const`` method which computes the total number of
+elements in a shape.
+
+Hash of a matrix
+================
+
+For internal needs, such as using (unordered) maps of data instances,
+CUDASTF need to have specialized forms of the ``std::hash`` trait class.
+
+The ``()`` operator of this class should compute a unique identifier
+associated to the description of the data instance. This typically means
+computing a hash of the matrix sizes, and of the base address. Note that
+this hash *does not* depend on the actual content of the matrix.
+
+In code snippet, we are using the ``cudastf::hash_combine`` helper which
+updates a hash value with another value. This function is available from
+the ``cudastf/utility/hash.h`` header.
+
+.. code:: c++
+
+   template <typename T>
+   struct std::hash<matrix<T>> {
+       std::size_t operator()(matrix<T> const& m) const noexcept {
+           // Combine hashes from the base address and sizes
+           return cudastf::hash_all(m.m, m.n, m.base);
+       }
+   };
+
+Defining a data interface
+=========================
+
+We can now implement the actual data interface for a matrix class, which
+defines the basic operations that CUDASTF need to perform on a matrix.
+
+The ``matrix_stream_interface`` class inherits from the
+``data_interface`` class, but to implement a data interface using APIs
+based on CUDA streams, ``matrix_stream_interface`` inherits from
+``stream_data_interface_simple<matrix<T>>`` which contains pure virtual
+functions that need to be implemented.
+
+.. code:: c++
+
+   template <typename T>
+   class matrix_stream_interface : public stream_data_interface_simple<matrix<T>> {
+   public:
+       using base = stream_data_interface_simple<matrix<T>>;
+       using base::shape_t;
+
+       /// Initialize from an existing matrix
+       matrix_stream_interface(matrix<T> m) : base(std::move(m)) {}
+
+       /// Initialize from a shape of matrix
+       matrix_stream_interface(shape_t s) : base(s) {}
+
+       /// Copy the content of an instance to another instance
+       ///
+       /// This implementation assumes that we have registered memory if one of the data place is the host
+       void stream_data_copy(const data_place& dst_memory_node, instance_id_t dst_instance_id,
+               const data_place& src_memory_node, instance_id_t src_instance_id, cudaStream_t stream) override {
+           assert(src_memory_node != dst_memory_node);
+
+           cudaMemcpyKind kind = cudaMemcpyDeviceToDevice;
+           if (src_memory_node == data_place::host) {
+               kind = cudaMemcpyHostToDevice;
+           }
+
+           if (dst_memory_node == data_place::host) {
+               kind = cudaMemcpyDeviceToHost;
+           }
+
+           const matrix<T>& src_instance = this->instance(src_instance_id);
+           const matrix<T>& dst_instance = this->instance(dst_instance_id);
+
+           size_t sz = src_instance.m * src_instance.n * sizeof(T);
+
+           cuda_safe_call(cudaMemcpyAsync((void*) dst_instance.base, (void*) src_instance.base, sz, kind, stream));
+       }
+
+       /// allocate an instance on a specific data place
+       ///
+       /// setting *s to a negative value informs CUDASTF that the allocation
+       /// failed, and that a memory reclaiming mechanism need to be performed.
+       void stream_data_allocate(backend_ctx_untyped& ctx, const data_place& memory_node, instance_id_t instance_id, ssize_t& s,
+               void** extra_args, cudaStream_t stream) override {
+           matrix<T>& instance = this->instance(instance_id);
+           size_t sz = instance.m * instance.n * sizeof(T);
+
+           T* base_ptr;
+
+           if (memory_node == data_place::host) {
+               // Fallback to a synchronous method as there is no asynchronous host allocation API
+               cuda_safe_call(cudaStreamSynchronize(stream));
+               cuda_safe_call(cudaHostAlloc(&base_ptr, sz, cudaHostAllocMapped));
+           } else {
+               cuda_safe_call(cudaMallocAsync(&base_ptr, sz, stream));
+           }
+
+           // By filling a positive number, we notify that the allocation was successful
+           *s = sz;
+
+           instance.base = base_ptr;
+       }
+
+       /// deallocate an instance
+       void stream_data_deallocate(backend_ctx_untyped& ctx, const data_place& memory_node, instance_id_t instance_id, void* extra_args,
+               cudaStream_t stream) override {
+           matrix<T>& instance = this->instance(instance_id);
+           if (memory_node == data_place::host) {
+               // Fallback to a synchronous method as there is no asynchronous host deallocation API
+               cuda_safe_call(cudaStreamSynchronize(stream));
+               cuda_safe_call(cudaFreeHost(instance.base));
+           } else {
+               cuda_safe_call(cudaFreeAsync(instance.base, stream));
+           }
+       }
+
+       /// Register the host memory associated to an instance of matrix
+       ///
+       /// Note that this pin_host_memory method is not mandatory, but then it is
+       /// the responsibility of the user to only passed memory that is already
+       /// registered, and the allocation method on the host must allocate
+       /// registered memory too. Otherwise, copy methods need to be synchronous.
+       bool pin_host_memory(instance_id_t instance_id) override {
+           matrix<T>& instance = this->instance(instance_id);
+           if (!instance.base) {
+               return false;
+           }
+
+           cuda_safe_call(pin_memory(instance.base, instance.m * instance.n * sizeof(T)));
+
+           return true;
+       }
+
+       /// Unregister memory pinned by pin_host_memory
+       void unpin_host_memory(instance_id_t instance_id) override {
+           matrix<T>& instance = this->instance(instance_id);
+           unpin_memory(instance.base);
+       }
+   };
+
+``matrix_stream_interface`` must meet the following requirements so that
+they can be used in the CUDA stream backend : - It must provide
+constructors which take either a matrix, or a shape of matrix as
+arguments. - It must implement the ``stream_data_copy``,
+``stream_data_allocate`` and ``stream_data_deallocate`` virtual methods,
+which respectively define how to copy an instance into another instance,
+how to allocate an instance, and how to deallocate an instance. - It may
+implement the ``pin_host_memory`` and ``unpin_host_memory`` virtual
+methods which respectively register and unregister the memory associated
+to an instance allocated on the host. These two methods are not
+mandatory, but it is the responsibility of the user to either only pass
+and allocate registered host buffers, or to ensure that the copy method
+does not require such memory pinning. Similarly, accessing an instance
+located in host memory from a device typically requires to access
+registered memory.
+
+Associating a data interface with the CUDA stream backend
+=========================================================
+
+To ensure that we can initialize a logical data from a matrix, or from
+the shape of a matrix with ``stream_ctx::logical_data``, we then need to
+specialize the ``cudastf::streamed_interface_of`` trait class.
+
+The resulting class must simply define a type named ``type`` which is
+the type of the data interface for the CUDA stream backend.
+
+.. code:: c++
+
+   template <typename T>
+   class cudastf::streamed_interface_of<matrix<T>> {
+   public:
+       using type = matrix_stream_interface<T>;
+   };
+
+Once we have defined this trait class, it is for example possible to
+initialize a logical data from a matrix, or from a matrix shape :
+
+.. code:: c++
+
+       std::vector<int> v(m * n, 0);
+       matrix M(m, n, &v[0]);
+
+       // Initialize from a matrix
+       auto lM = ctx.logical_data(M);
+
+       // Initialize from a shape
+       auto lM2 = ctx.logical_data(shape_of<matrix<int>>(m, n));
+
+Example of code using the ``matrix`` data interface
+===================================================
+
+We can now use the ``matrix`` class in CUDASTF, and access it from
+tasks. In this code, we first initialize a matrix on the host, we then
+apply a task which will update its content on the current device. We
+finally check that the content is correct, by the means of the
+write-back mechanism that automatically updates the reference data
+instance of a logical data when calling ``ctx.sync()``.
+
+.. code:: c++
+
+   template <typename T>
+   __global__ void kernel(matrix<T> M) {
+       int tid_x = blockIdx.x * blockDim.x + threadIdx.x;
+       int nthreads_x = gridDim.x * blockDim.x;
+
+       int tid_y = blockIdx.y * blockDim.y + threadIdx.y;
+       int nthreads_y = gridDim.y * blockDim.y;
+
+       for (int x = tid_x; x < M.m; x += nthreads_x)
+           for (int y = tid_y; y < M.n; y += nthreads_y) {
+               M(x, y) += -x + 7 * y;
+           }
+   }
+
+   int main() {
+       stream_ctx ctx;
+
+       const size_t m = 8;
+       const size_t n = 10;
+       std::vector<int> v(m * n);
+
+       for (size_t j = 0; j < n; j++)
+           for (size_t i = 0; i < m; i++) {
+               v[i + j * m] = 17 * i + 23 * j;
+           }
+
+       matrix<int> M(m, n, &v[0]);
+
+       auto lM = ctx.logical_data(M);
+
+       // M(i,j) +=  -i + 7*i
+       ctx.task(lM.rw())->*[](cudaStream_t s, auto dM) { kernel<<<dim3(8, 8), dim3(8, 8), 0, s>>>(dM); };
+
+       ctx.sync();
+
+       for (size_t j = 0; j < n; j++)
+           for (size_t i = 0; i < m; i++) {
+               assert(v[i + j * m] == (17 * i + 23 * j) + (-i + 7*i));
+           }
+   }
diff --git a/cccl_upstream/docs/cudax/stf/images/dag-sections-0.dot b/cccl_upstream/docs/cudax/stf/images/dag-sections-0.dot
new file mode 100644
index 00000000..c447276e
--- /dev/null
+++ b/cccl_upstream/docs/cudax/stf/images/dag-sections-0.dot
@@ -0,0 +1,5 @@
+digraph {
+"NODE_23" [style="filled" fillcolor="red" label="task fence"]
+"NODE_3" -> "NODE_23"
+"NODE_3" [style="filled" fillcolor="white" label="foo"]
+}
diff --git a/cccl_upstream/docs/cudax/stf/images/dag-sections-0.png b/cccl_upstream/docs/cudax/stf/images/dag-sections-0.png
new file mode 100644
index 00000000..bc15d535
Binary files /dev/null and b/cccl_upstream/docs/cudax/stf/images/dag-sections-0.png differ
diff --git a/cccl_upstream/docs/cudax/stf/images/dag-sections-1.dot b/cccl_upstream/docs/cudax/stf/images/dag-sections-1.dot
new file mode 100644
index 00000000..eddf1e29
--- /dev/null
+++ b/cccl_upstream/docs/cudax/stf/images/dag-sections-1.dot
@@ -0,0 +1,14 @@
+digraph {
+"NODE_23" [style="filled" fillcolor="red" label="task fence"]
+subgraph cluster_section_1 {
+     color=black;
+    style=dashed
+    label="foo"
+    "NODE_13"
+    "NODE_3"
+} // end subgraph cluster_section_1
+ "NODE_13" -> "NODE_23"
+"NODE_3" -> "NODE_13"
+"NODE_13" [style="filled" fillcolor="white" label="bar"]
+"NODE_3" [style="filled" fillcolor="white" label="bar"]
+}
diff --git a/cccl_upstream/docs/cudax/stf/images/dag-sections-1.png b/cccl_upstream/docs/cudax/stf/images/dag-sections-1.png
new file mode 100644
index 00000000..6f5ca311
Binary files /dev/null and b/cccl_upstream/docs/cudax/stf/images/dag-sections-1.png differ
diff --git a/cccl_upstream/docs/cudax/stf/images/dag-sections-2.dot b/cccl_upstream/docs/cudax/stf/images/dag-sections-2.dot
new file mode 100644
index 00000000..8c1da4d1
--- /dev/null
+++ b/cccl_upstream/docs/cudax/stf/images/dag-sections-2.dot
@@ -0,0 +1,36 @@
+digraph {
+"NODE_23" [style="filled" fillcolor="red" label="task fence"]
+subgraph cluster_section_1 {
+ subgraph cluster_section_2 {
+     color=black;
+    style=dashed
+    label="bar"
+    "NODE_9"
+    "NODE_5"
+    "NODE_3"
+} // end subgraph cluster_section_2
+ subgraph cluster_section_5 {
+     color=black;
+    style=dashed
+    label="bar"
+    "NODE_19"
+    "NODE_15"
+    "NODE_13"
+} // end subgraph cluster_section_5
+     color=black;
+    style=dashed
+    label="foo"
+} // end subgraph cluster_section_1
+ "NODE_19" -> "NODE_23"
+"NODE_3" -> "NODE_5"
+"NODE_5" -> "NODE_9"
+"NODE_15" -> "NODE_19"
+"NODE_9" -> "NODE_13"
+"NODE_13" -> "NODE_15"
+"NODE_19" [style="filled" fillcolor="white" label="baz"]
+"NODE_15" [style="filled" fillcolor="white" label="baz"]
+"NODE_13" [style="filled" fillcolor="white" label="t1\nA(read)(0) \nB(rw)(0) "]
+"NODE_9" [style="filled" fillcolor="white" label="baz"]
+"NODE_5" [style="filled" fillcolor="white" label="baz"]
+"NODE_3" [style="filled" fillcolor="white" label="t1\nA(read)(0) \nB(rw)(0) "]
+}
diff --git a/cccl_upstream/docs/cudax/stf/images/dag-sections-2.png b/cccl_upstream/docs/cudax/stf/images/dag-sections-2.png
new file mode 100644
index 00000000..aed7fb49
Binary files /dev/null and b/cccl_upstream/docs/cudax/stf/images/dag-sections-2.png differ
diff --git a/cccl_upstream/docs/cudax/stf/images/dag-sections.dot b/cccl_upstream/docs/cudax/stf/images/dag-sections.dot
new file mode 100644
index 00000000..bbf13630
--- /dev/null
+++ b/cccl_upstream/docs/cudax/stf/images/dag-sections.dot
@@ -0,0 +1,69 @@
+digraph {
+"NODE_23" [style="filled" fillcolor="red" label="task fence"]
+subgraph cluster_section_1 {
+ subgraph cluster_section_2 {
+ subgraph cluster_section_3 {
+     color=black;
+    style=dashed
+    label="baz"
+    "NODE_7"
+    "NODE_5"
+} // end subgraph cluster_section_3
+ subgraph cluster_section_4 {
+     color=black;
+    style=dashed
+    label="baz"
+    "NODE_11"
+    "NODE_9"
+} // end subgraph cluster_section_4
+     color=black;
+    style=dashed
+    label="bar"
+    "NODE_3"
+} // end subgraph cluster_section_2
+ subgraph cluster_section_5 {
+ subgraph cluster_section_6 {
+     color=black;
+    style=dashed
+    label="baz"
+    "NODE_17"
+    "NODE_15"
+} // end subgraph cluster_section_6
+ subgraph cluster_section_7 {
+     color=black;
+    style=dashed
+    label="baz"
+    "NODE_21"
+    "NODE_19"
+} // end subgraph cluster_section_7
+     color=black;
+    style=dashed
+    label="bar"
+    "NODE_13"
+} // end subgraph cluster_section_5
+     color=black;
+    style=dashed
+    label="foo"
+} // end subgraph cluster_section_1
+ "NODE_15" -> "NODE_17"
+"NODE_9" -> "NODE_11"
+"NODE_11" -> "NODE_15"
+"NODE_13" -> "NODE_17"
+"NODE_19" -> "NODE_21"
+"NODE_11" -> "NODE_13"
+"NODE_7" -> "NODE_9"
+"NODE_5" -> "NODE_7"
+"NODE_17" -> "NODE_19"
+"NODE_3" -> "NODE_7"
+"NODE_21" -> "NODE_23"
+"NODE_21" [style="filled" fillcolor="white" label="t3\nA(rw)(0) \nB(read)(0) \nC(read)(0) "]
+"NODE_19" [style="filled" fillcolor="white" label="t2\nA(read)(0) \nC(rw)(0) "]
+"NODE_17" [style="filled" fillcolor="white" label="t3\nA(rw)(0) \nB(read)(0) \nC(read)(0) "]
+"NODE_15" [style="filled" fillcolor="white" label="t2\nA(read)(0) \nC(rw)(0) "]
+"NODE_13" [style="filled" fillcolor="white" label="t1\nA(read)(0) \nB(rw)(0) "]
+"NODE_11" [style="filled" fillcolor="white" label="t3\nA(rw)(0) \nB(read)(0) \nC(read)(0) "]
+"NODE_9" [style="filled" fillcolor="white" label="t2\nA(read)(0) \nC(rw)(0) "]
+"NODE_7" [style="filled" fillcolor="white" label="t3\nA(rw)(0) \nB(read)(0) \nC(read)(0) "]
+"NODE_5" [style="filled" fillcolor="white" label="t2\nA(read)(0) \nC(rw)(0) "]
+"NODE_3" [style="filled" fillcolor="white" label="t1\nA(read)(0) \nB(rw)(0) "]
+}
diff --git a/cccl_upstream/docs/cudax/stf/images/dag-sections.png b/cccl_upstream/docs/cudax/stf/images/dag-sections.png
new file mode 100644
index 00000000..77a5ca20
Binary files /dev/null and b/cccl_upstream/docs/cudax/stf/images/dag-sections.png differ
diff --git a/cccl_upstream/docs/cudax/stf/images/dot-output-axpy-annotated.png b/cccl_upstream/docs/cudax/stf/images/dot-output-axpy-annotated.png
new file mode 100644
index 00000000..3101370b
Binary files /dev/null and b/cccl_upstream/docs/cudax/stf/images/dot-output-axpy-annotated.png differ
diff --git a/cccl_upstream/docs/cudax/stf/images/dot-output-axpy-events.png b/cccl_upstream/docs/cudax/stf/images/dot-output-axpy-events.png
new file mode 100644
index 00000000..d2c3e25a
Binary files /dev/null and b/cccl_upstream/docs/cudax/stf/images/dot-output-axpy-events.png differ
diff --git a/cccl_upstream/docs/cudax/stf/images/dot-output-axpy.png b/cccl_upstream/docs/cudax/stf/images/dot-output-axpy.png
new file mode 100644
index 00000000..c4baf5f9
Binary files /dev/null and b/cccl_upstream/docs/cudax/stf/images/dot-output-axpy.png differ
diff --git a/cccl_upstream/docs/cudax/stf/images/dot-output-heat.png b/cccl_upstream/docs/cudax/stf/images/dot-output-heat.png
new file mode 100644
index 00000000..bddd6d47
Binary files /dev/null and b/cccl_upstream/docs/cudax/stf/images/dot-output-heat.png differ
diff --git a/cccl_upstream/docs/cudax/stf/images/graph_01.dot b/cccl_upstream/docs/cudax/stf/images/graph_01.dot
new file mode 100644
index 00000000..387b4e0c
--- /dev/null
+++ b/cccl_upstream/docs/cudax/stf/images/graph_01.dot
@@ -0,0 +1,10 @@
+digraph {
+    T_1 [label="T1\nX(rw)"];
+    T_2 [label="T2\nX(read)\nY(rw)"];
+    T_3 [label="T3\nX(read)\nZ(rw)"];
+    T_4 [label="T4\nY(read)\nZ(rw)"];
+    T_1 -> T_2;
+    T_1 -> T_3;
+    T_2 -> T_4;
+    T_3 -> T_4;
+}
diff --git a/cccl_upstream/docs/cudax/stf/images/graph_01.png b/cccl_upstream/docs/cudax/stf/images/graph_01.png
new file mode 100644
index 00000000..b269fc60
Binary files /dev/null and b/cccl_upstream/docs/cudax/stf/images/graph_01.png differ
diff --git a/cccl_upstream/docs/cudax/stf/images/graph_02.dot b/cccl_upstream/docs/cudax/stf/images/graph_02.dot
new file mode 100644
index 00000000..37bf0f81
--- /dev/null
+++ b/cccl_upstream/docs/cudax/stf/images/graph_02.dot
@@ -0,0 +1,16 @@
+digraph {
+    subgraph cluster_0 {
+      label="device 0";
+      T_1 [label="T_1(A^W)"];
+      T_2 [label="T_2(A^R, B^W)"];
+    }
+    subgraph cluster_1 {
+      label="device 1";
+      T_3 [label="T_3(A^R, C^W)"];
+      T_4 [label="T_4(B^R, C^R, D^W)"];
+    }
+    T_1 -> T_2 [label="A"];
+    T_1 -> T_3 [label="A"];
+    T_2 -> T_4 [label="B"];
+    T_3 -> T_4 [label ="C"];
+}
diff --git a/cccl_upstream/docs/cudax/stf/images/graph_02.png b/cccl_upstream/docs/cudax/stf/images/graph_02.png
new file mode 100644
index 00000000..3fa4b870
Binary files /dev/null and b/cccl_upstream/docs/cudax/stf/images/graph_02.png differ
diff --git a/cccl_upstream/docs/cudax/stf/images/ncu-ui.png b/cccl_upstream/docs/cudax/stf/images/ncu-ui.png
new file mode 100644
index 00000000..60cce7df
Binary files /dev/null and b/cccl_upstream/docs/cudax/stf/images/ncu-ui.png differ
diff --git a/cccl_upstream/docs/cudax/stf/images/task-sequence-user.dot b/cccl_upstream/docs/cudax/stf/images/task-sequence-user.dot
new file mode 100644
index 00000000..af500308
--- /dev/null
+++ b/cccl_upstream/docs/cudax/stf/images/task-sequence-user.dot
@@ -0,0 +1,26 @@
+digraph {
+    compound=true;
+    subgraph cluster_0 {
+      label="T1";
+      K1 [label="K1"];
+      K2 [label="K2"];
+    }
+    subgraph cluster_1 {
+      label="T2";
+      K3 [label="K3"];
+    }
+    subgraph cluster_2 {
+      label="T3";
+      K4 [label="K4"];
+    }
+    subgraph cluster_3 {
+      label="T4";
+      cb [label="callback"];
+    }
+
+    K1 -> K2;
+    K2 -> K3 [ltail=cluster_0,lhead=cluster_1,minlen=2];
+    K2 -> K4 [ltail=cluster_0,lhead=cluster_2,minlen=2];
+    K3 -> cb [ltail=cluster_1,lhead=cluster_3,minlen=2];
+    K4 -> cb [ltail=cluster_2,lhead=cluster_3,minlen=2];
+}
diff --git a/cccl_upstream/docs/cudax/stf/images/task-sequence-user.png b/cccl_upstream/docs/cudax/stf/images/task-sequence-user.png
new file mode 100644
index 00000000..2ae56480
Binary files /dev/null and b/cccl_upstream/docs/cudax/stf/images/task-sequence-user.png differ
diff --git a/cccl_upstream/docs/cudax/stf/images/task-sequence.dot b/cccl_upstream/docs/cudax/stf/images/task-sequence.dot
new file mode 100644
index 00000000..b86a37ea
--- /dev/null
+++ b/cccl_upstream/docs/cudax/stf/images/task-sequence.dot
@@ -0,0 +1,68 @@
+digraph {
+    compound=true;
+    subgraph cluster_00 {
+      label="";
+      AA [label="Allocate A"];
+    }
+
+    subgraph cluster_01 {
+      label="";
+      CA [label="Copy A H->D"];
+    }
+
+    subgraph cluster_10 {
+      label="";
+      AB [label="Allocate B"];
+    }
+
+    subgraph cluster_11 {
+      label="";
+      CB [label="Copy B H->D"];
+    }
+
+    subgraph cluster_0 {
+      label="T1";
+      K1 [label="K1"];
+      K2 [label="K2"];
+    }
+
+    CA -> K1 [ltail=cluster_01,lhead=cluster_0,minlen=2];
+    AA -> CA [ltail=cluster_00,lhead=cluster_01,minlen=2];
+
+    CB -> K1 [ltail=cluster_11,lhead=cluster_0,minlen=2];
+    AB -> CB [ltail=cluster_10,lhead=cluster_11,minlen=2];
+
+    subgraph cluster_1 {
+      label="T2";
+      K3 [label="K3"];
+    }
+    subgraph cluster_2 {
+      label="T3";
+      K4 [label="K4"];
+    }
+
+    K1 -> K2;
+    K2 -> K3 [ltail=cluster_0,lhead=cluster_1,minlen=2];
+    K2 -> K4 [ltail=cluster_0,lhead=cluster_2,minlen=2];
+
+    subgraph cluster_02 {
+      label="";
+      CA2 [label="Copy A D->A"];
+    }
+
+    subgraph cluster_12 {
+      label="";
+      CB2 [label="Copy B D->A"];
+    }
+
+    subgraph cluster_3 {
+      label="T4";
+      cb [label="callback"];
+    }
+
+    K3 -> CA2 [ltail=cluster_1,lhead=cluster_02,minlen=2];
+    K4 -> CB2 [ltail=cluster_2,lhead=cluster_12,minlen=2];
+
+    CA2 -> cb [ltail=cluster_02,lhead=cluster_3,minlen=2]
+    CB2 -> cb [ltail=cluster_12,lhead=cluster_3,minlen=2]
+}
diff --git a/cccl_upstream/docs/cudax/stf/images/task-sequence.png b/cccl_upstream/docs/cudax/stf/images/task-sequence.png
new file mode 100644
index 00000000..333eb4c4
Binary files /dev/null and b/cccl_upstream/docs/cudax/stf/images/task-sequence.png differ
diff --git a/cccl_upstream/docs/cudax/stf/lower_level_api.rst b/cccl_upstream/docs/cudax/stf/lower_level_api.rst
new file mode 100644
index 00000000..77508e28
--- /dev/null
+++ b/cccl_upstream/docs/cudax/stf/lower_level_api.rst
@@ -0,0 +1,114 @@
+.. _stf_lower_level_api:
+
+Lower-level API
+===============
+
+In some situations, the use of ``operator->*()`` on the object returned
+by ``ctx.task()`` (where ``ctx`` is a stream or graph context) may not
+be suitable, for example when the number of parameters is not known
+statically. To address such situations, CUDASTF provides a lower-level
+interface for creating tasks, which is described below.
+
+.. code:: cpp
+
+   #include "cudastf/stf.h"
+   #include "cudastf/__stf/stream/stream_ctx.h"
+
+   using namespace cudastf;
+
+   template <typename T>
+   __global__ void axpy(int n, T a, T* x, T* y) {
+       int tid = blockIdx.x * blockDim.x + threadIdx.x;
+       int nthreads = gridDim.x * blockDim.x;
+
+       for (int ind = tid; ind < n; ind += nthreads) {
+           y[ind] += a * x[ind];
+       }
+   }
+
+   int main(int argc, char** argv) {
+       stream_ctx ctx;
+
+       const size_t N = 16;
+       double X[N], Y[N];
+
+       for (size_t ind = 0; ind < N; ind++) {
+           X[ind] = sin(double(ind));
+           Y[ind] = cos(double(ind));
+       }
+
+       auto lX = ctx.logical_data(X);
+       auto lY = ctx.logical_data(Y);
+
+       double alpha = 3.14;
+
+       /* Compute Y = Y + alpha X */
+       auto t = ctx.task(lX.read(), lY.rw());
+       t.start();
+       slice<double> sX = t.get<0>();
+       slice<double> sY = t.get<1>();
+       axpy<<<16, 128, 0, t.get_stream()>>>(sX.size(), alpha, sX.data_handle(), sY.data_handle());
+       t.end();
+
+       ctx.sync();
+   }
+
+The ``ctx.task()`` call returns a task object. This object provides
+access to the local description of the data associated with the task and
+a CUDA stream that can be used to submit work asynchronously. The
+beginning of the task body and its end are delimited by the ``.start()``
+and ``.end()`` calls. Failing to call either of these methods or calling
+them more than once or in the wrong order results in undefined behavior.
+
+Asynchrony is achieved by using the CUDA stream, which provides a
+mechanism to submit work on the execution place (here, implicitly the
+current CUDA device). CUDA ensures that all kernels synchronized with
+this CUDA stream will only be executed once all prerequisites have been
+fulfilled (e.g., preceding tasks, data transfers, etc.). In addition,
+CUDASTF performs all the necessary synchronization so that future tasks
+will be properly synchronized with the operations enqueued in the CUDA
+stream associated with this task after calling ``.start()`` and before
+calling ``.end()``.
+
+Compatibility with CUDA graphs
+==============================
+
+Similarly to the CUDA stream backend with a context of type
+``stream_ctx``, the CUDA graph backend ``graph_ctx`` also provides a
+low-level interface.
+
+.. code:: cpp
+
+       graph_ctx ctx;
+
+       double X[1024], Y[1024];
+       auto lX = ctx.logical_data(X);
+       auto lY = ctx.logical_data(Y);
+
+       for (int k = 0; k < 10; k++) {
+           graph_task t = ctx.task();
+           t.add_deps(handle_X.rw());
+           t.start();
+           cudaGraphNode_t n;
+           cuda_safe_call(cudaGraphAddEmptyNode(&n, t.get_graph(), nullptr, 0));
+           t.end();
+       }
+
+       graph_task t2 = ctx.task();
+       t2.add_deps(lX.read(), lY.rw());
+       t2.start();
+       cudaGraphNode_t n2;
+       cuda_safe_call(cudaGraphAddEmptyNode(&n2, t2.get_graph(), nullptr, 0));
+       t2.end();
+
+       ctx.sync();
+
+A task in the CUDA graph backend corresponds to a *child graph*
+automatically inserted into the CUDA graph associated to a ``graph_ctx``
+context. The example above creates 10 tasks that modify logical data
+``lX``, followed by a task that reads ``lX`` and modifies ``lY``. The
+code illustrates how one can add dependencies to a task by using the
+``add_deps`` method.
+
+Similarly to the CUDA stream backend, a task is outlined by a pair of
+calls to the ``start()``/``end()`` member functions.
diff --git a/cccl_upstream/docs/gen_docs.bash b/cccl_upstream/docs/gen_docs.bash
new file mode 100755
index 00000000..43284a11
--- /dev/null
+++ b/cccl_upstream/docs/gen_docs.bash
@@ -0,0 +1,312 @@
+#!/usr/bin/env bash
+
+# This script builds CCCL documentation using Sphinx directly
+#
+# Usage:
+#   ./gen_docs.bash                    - Build documentation
+#   ./gen_docs.bash --allow-dep-install - Build, auto-install missing system deps
+#   ./gen_docs.bash clean              - Clean build directory
+#   ./gen_docs.bash clean --all        - Clean build directory and Doxygen build
+#
+# The script will optionally build Doxygen 1.9.6 from source to ensure
+# consistent documentation generation. The built Doxygen will be stored
+# in _build/doxygen-build/ and reused for subsequent runs.
+
+set -euo pipefail
+
+ALLOW_DEP_INSTALL=false
+CLEAN=false
+CLEAN_ALL=false
+
+for arg in "$@"; do
+    case "$arg" in
+        --allow-dep-install) ALLOW_DEP_INSTALL=true ;;
+        clean)               CLEAN=true ;;
+        --all)               CLEAN_ALL=true ;;
+        *)                   echo "Unknown argument: $arg"; exit 1 ;;
+    esac
+done
+
+SCRIPT_PATH=$(cd "$(dirname "${0}")"; pwd -P)
+cd "$SCRIPT_PATH"
+
+BUILDDIR="_build"
+DOXYGEN_BUILD_DIR="${SCRIPT_PATH}/_build/doxygen-build"
+DOXYGEN_SRC_DIR="${SCRIPT_PATH}/_build/doxygen-src"
+
+# Handle clean command (before dep checks — clean doesn't need deps)
+if [[ "$CLEAN" = true ]]; then
+    echo "Cleaning build directory..."
+    rm -rf "${BUILDDIR:?}"/*
+    if [[ "$CLEAN_ALL" = true ]]; then
+        echo "Also removing Doxygen source and build directories..."
+        rm -rf "${DOXYGEN_SRC_DIR}" "${DOXYGEN_BUILD_DIR}"
+    fi
+    exit 0
+fi
+
+# Check and optionally install system dependencies
+check_system_deps() {
+    local missing=()
+    # Map of command -> package name
+    local -A cmd_to_pkg=(
+        [cmake]=cmake
+        [ninja]=ninja-build
+        [flex]=flex
+        [bison]=bison
+        [git]=git
+    )
+
+    # python3-venv is a package, not a command — check by trying to create a venv
+    if ! python3 -m venv --help &>/dev/null; then
+        missing+=(python3-venv)
+    fi
+
+    for cmd in "${!cmd_to_pkg[@]}"; do
+        if ! command -v "$cmd" &>/dev/null; then
+            missing+=("${cmd_to_pkg[$cmd]}")
+        fi
+    done
+
+    if [[ ${#missing[@]} -eq 0 ]]; then
+        return 0
+    fi
+
+    echo "Missing system dependencies: ${missing[*]}"
+
+    if [[ "$ALLOW_DEP_INSTALL" = true ]]; then
+        echo "Installing missing dependencies (--allow-dep-install)..."
+        sudo apt-get update -qq
+        sudo apt-get install -y -qq "${missing[@]}"
+    else
+        read -r -p "Install them now? [y/N] " response
+        if [[ "$response" =~ ^[Yy]$ ]]; then
+            sudo apt-get update -qq
+            sudo apt-get install -y -qq "${missing[@]}"
+        else
+            echo "Error: Missing dependencies. Install with:"
+            echo "  sudo apt-get install -y ${missing[*]}"
+            exit 1
+        fi
+    fi
+}
+
+check_system_deps
+
+# Configuration
+# Keep going to surface all warnings; -W makes warnings fail the build.
+declare -a SPHINXOPTS="(${SPHINXOPTS:---keep-going -W})"
+DOXYGEN_BIN="${DOXYGEN_BUILD_DIR}/bin/doxygen"
+
+# Use custom-built doxygen if available, otherwise fall back to system doxygen
+if [[ -f "${DOXYGEN_BIN}" ]]; then
+    DOXYGEN="${DOXYGEN_BIN}"
+else
+    DOXYGEN="${DOXYGEN:-doxygen}"
+fi
+
+## Clean image directory, without this any artifacts will prevent fetching
+rm -rf img
+mkdir -p img
+
+# Pull cub images
+if [[ ! -d cubimg ]]; then
+    git clone -b gh-pages https://github.com/NVlabs/cub.git cubimg
+fi
+
+if [[ -z "$(find cubimg -name 'example_range.png')" ]]; then
+    wget -q https://raw.githubusercontent.com/NVIDIA/NVTX/release-v3/docs/images/example_range.png -O cubimg/example_range.png
+fi
+
+if [[ -z "$(find img -name '*.png')" ]]; then
+    wget -q https://docs.nvidia.com/cuda/_static/Logo_and_CUDA.png -O img/logo.png
+
+    # Parse files and collects unique names ending with .png
+    imgs="$(grep -R -o -h '[[:alpha:][:digit:]_]*.png' ../cub/cub | uniq)"
+    declare -a imgs="($imgs)"
+    imgs+=( "cub_overview.png" "nested_composition.png" "tile.png" "blocked.png" "striped.png" )
+
+    for img in "${imgs[@]}"
+    do
+        echo "${img}"
+        cp cubimg/"${img}" img/"${img}"
+    done
+fi
+
+# Function to build Doxygen 1.9.6
+build_doxygen() {
+    echo "Building Doxygen 1.9.6..."
+
+    # Clone Doxygen if not already cloned
+    if [[ ! -d "${DOXYGEN_SRC_DIR}" ]]; then
+        echo "Cloning Doxygen repository..."
+        git clone https://github.com/doxygen/doxygen.git "${DOXYGEN_SRC_DIR}"
+    fi
+
+    # Checkout Release_1_9_6
+    cd "${DOXYGEN_SRC_DIR}"
+    git fetch
+    git checkout Release_1_9_6
+
+    # Create build directory
+    mkdir -p "${DOXYGEN_BUILD_DIR}"
+    cd "${DOXYGEN_BUILD_DIR}"
+
+    # Configure based on platform
+    echo "Configuring Doxygen build..."
+    if [[ "$OSTYPE" == "darwin"* ]]; then
+        # macOS
+        echo "Detected macOS, configuring with LLVM paths..."
+        if ! command -v brew &> /dev/null; then
+            echo "Warning: Homebrew not found, building without libclang support"
+            cmake -GNinja -DCMAKE_BUILD_TYPE=Release \
+                -DCMAKE_POLICY_VERSION_MINIMUM=3.5 \
+                "${DOXYGEN_SRC_DIR}"
+        else
+            cmake -GNinja -DCMAKE_BUILD_TYPE=Release \
+                -DCMAKE_POLICY_VERSION_MINIMUM=3.5 \
+                -Duse_libclang=NO \
+                -DBISON_EXECUTABLE="$(brew --prefix bison)/bin/bison" \
+                "${DOXYGEN_SRC_DIR}"
+        fi
+    else
+        # Linux/Ubuntu
+        echo "Configuring for Linux/Ubuntu..."
+        cmake -GNinja -DCMAKE_BUILD_TYPE=Release \
+            -DCMAKE_POLICY_VERSION_MINIMUM=3.5 \
+            -Duse_libclang=NO \
+            "${DOXYGEN_SRC_DIR}"
+    fi
+
+    # Build Doxygen
+    echo "Building Doxygen (this may take a few minutes)..."
+    ninja
+
+    echo "Doxygen 1.9.6 built successfully at ${DOXYGEN_BIN}"
+    cd "${SCRIPT_PATH}"
+}
+
+# Check if custom Doxygen needs to be built
+if [[ ! -f "${DOXYGEN_BIN}" ]]; then
+    echo "Custom Doxygen 1.9.6 not found, building it now..."
+    build_doxygen
+    DOXYGEN="${DOXYGEN_BIN}"
+else
+    echo "Using custom-built Doxygen 1.9.6 from ${DOXYGEN_BIN}"
+fi
+
+# Check if documentation dependencies are installed
+echo "Checking for documentation dependencies..."
+
+# Use virtual environment if it exists, otherwise create one
+if [[ -d "env" ]]; then
+    echo "Using existing virtual environment..."
+    # shellcheck disable=SC1091
+    source env/bin/activate
+else
+    echo "Creating virtual environment..."
+    python3 -m venv env
+    # shellcheck disable=SC1091
+    source env/bin/activate
+fi
+
+# Check if dependencies are installed in the virtual environment
+if ! python -c "import sphinx" 2>/dev/null; then
+    echo "Installing documentation dependencies..."
+    python3 -m pip install -r requirements.txt || {
+        echo "Error: Failed to install documentation dependencies"
+        echo "Please install manually: pip install -r requirements.txt"
+        exit 1
+    }
+fi
+
+# Generate Doxygen XML in parallel (if doxygen is available)
+if command -v "${DOXYGEN}" > /dev/null 2>&1; then
+    echo "Generating Doxygen XML..."
+    mkdir -p "${BUILDDIR}"/doxygen/cub "${BUILDDIR}"/doxygen/thrust "${BUILDDIR}"/doxygen/cudax "${BUILDDIR}"/doxygen/libcudacxx
+
+    # Copy all images to Doxygen XML output directories where they're expected
+    for project in cub thrust cudax libcudacxx; do
+        mkdir -p "${BUILDDIR}"/doxygen/"${project}"/xml
+        cp img/*.png "${BUILDDIR}"/doxygen/"${project}"/xml/ 2>/dev/null || true
+    done
+
+    # Run all Doxygen builds in parallel, fail if any produce warnings/errors
+    (cd cub && ${DOXYGEN} Doxyfile) &
+    pids+=($!)
+    (cd thrust && ${DOXYGEN} Doxyfile) &
+    pids+=($!)
+    (cd cudax && ${DOXYGEN} Doxyfile) &
+    pids+=($!)
+    (cd libcudacxx && ${DOXYGEN} Doxyfile) &
+    pids+=($!)
+
+    doxygen_failed=0
+    for pid in "${pids[@]}"; do
+        if ! wait "$pid"; then
+            doxygen_failed=1
+        fi
+    done
+    if [[ "$doxygen_failed" -ne 0 ]]; then
+        echo "Error: one or more Doxygen builds failed (see warnings above)"
+        exit 1
+    fi
+
+    echo "Doxygen complete"
+else
+    echo "Skipping Doxygen (not installed)"
+fi
+
+# Build Sphinx HTML documentation
+echo "Building documentation with Sphinx..."
+# Use the virtual environment's Python
+python -m sphinx.cmd.build -b html -d "${BUILDDIR}/doctrees" -j auto "." "${BUILDDIR}/html" "${SPHINXOPTS[@]}"
+
+# Reorganize output to include versioned directory and root assets
+VERSION="${SPHINX_CCCL_VER:-unstable}"
+BASE_URL="${CCCL_DOCS_BASE_URL:-https://nvidia.github.io/cccl/}"
+BASE_URL="${BASE_URL%/}/"
+IS_LATEST="${CCCL_DOCS_IS_LATEST:-true}"
+
+HTML_DIR="${BUILDDIR}/html"
+ORIG_DIR="${BUILDDIR}/html_orig"
+
+rm -rf "${ORIG_DIR}"
+mv "${HTML_DIR}" "${ORIG_DIR}"
+mkdir -p "${HTML_DIR}/${VERSION}"
+cp -a "${ORIG_DIR}/." "${HTML_DIR}/${VERSION}/"
+rm -rf "${ORIG_DIR}"
+
+# Copy objects.inv to the root to support intersphinx consumers
+if [[ -f "${HTML_DIR}/${VERSION}/objects.inv" ]]; then
+    cp "${HTML_DIR}/${VERSION}/objects.inv" "${HTML_DIR}/objects.inv"
+fi
+
+# Scrape docs to generate page list
+./scrape_docs.bash "${HTML_DIR}/${VERSION}"
+
+cp "./404.html" "${HTML_DIR}/404.html"
+cp "./index.html" "${HTML_DIR}/index.html"
+
+# Provide version metadata for the theme switcher
+cat > "${HTML_DIR}/nv-versions.json" <<EOF
+[
+  {
+    "name": "${VERSION}",
+    "version": "${VERSION}",
+    "url": "${BASE_URL}${VERSION}/",
+    "latest": ${IS_LATEST},
+    "preferred": ${IS_LATEST}
+  }
+]
+EOF
+
+cat > "${HTML_DIR}/versions.json" <<EOF
+{
+  "${VERSION}": "${VERSION}"
+}
+EOF
+
+touch "${HTML_DIR}/.nojekyll"
+
+echo "Documentation build complete! HTML output is in ${BUILDDIR}/html/"
diff --git a/cccl_upstream/docs/images/codespaces.png b/cccl_upstream/docs/images/codespaces.png
new file mode 100644
index 00000000..3f711d03
Binary files /dev/null and b/cccl_upstream/docs/images/codespaces.png differ
diff --git a/cccl_upstream/docs/images/pr-checks.png b/cccl_upstream/docs/images/pr-checks.png
new file mode 100644
index 00000000..6b86e1e4
Binary files /dev/null and b/cccl_upstream/docs/images/pr-checks.png differ
diff --git a/cccl_upstream/docs/images/repro_instructions.png b/cccl_upstream/docs/images/repro_instructions.png
new file mode 100644
index 00000000..f0579650
Binary files /dev/null and b/cccl_upstream/docs/images/repro_instructions.png differ
diff --git a/cccl_upstream/docs/images/top_results_expl.png b/cccl_upstream/docs/images/top_results_expl.png
new file mode 100644
index 00000000..ffff2b6a
Binary files /dev/null and b/cccl_upstream/docs/images/top_results_expl.png differ
diff --git a/cccl_upstream/docs/images/variant_plot.png b/cccl_upstream/docs/images/variant_plot.png
new file mode 100644
index 00000000..7a625300
Binary files /dev/null and b/cccl_upstream/docs/images/variant_plot.png differ
diff --git a/cccl_upstream/docs/index.html b/cccl_upstream/docs/index.html
new file mode 100644
index 00000000..e231155a
--- /dev/null
+++ b/cccl_upstream/docs/index.html
@@ -0,0 +1,12 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <meta http-equiv="refresh" content="0; url=unstable/">
+  <link rel="canonical" href="unstable/">
+  <title>CUDA Core Compute Libraries</title>
+</head>
+<body>
+  <a href="unstable/">CUDA Core Compute Libraries documentation</a>
+</body>
+</html>
diff --git a/cccl_upstream/docs/index.rst b/cccl_upstream/docs/index.rst
new file mode 100644
index 00000000..306672a1
--- /dev/null
+++ b/cccl_upstream/docs/index.rst
@@ -0,0 +1,25 @@
+CUDA Core Compute Libraries
+===========================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 3
+
+   cpp
+   python/index
+   contributors/index
+   maintainers/index
+   infrastructure/index
+
+Welcome to the CUDA Core Compute Libraries (CCCL) where our mission is to
+make CUDA C++ and Python more delightful.
+
+- :ref:`cccl-cpp-libraries`
+
+- :doc:`Python Libraries <python/index>`
+
+- :doc:`Contributor Docs <contributors/index>`
+
+- :doc:`Maintainer Docs <maintainers/index>`
+
+- :doc:`Infrastructure <infrastructure/index>`
diff --git a/cccl_upstream/docs/infrastructure/ci/references/change_detection.rst b/cccl_upstream/docs/infrastructure/ci/references/change_detection.rst
new file mode 100644
index 00000000..197f64b3
--- /dev/null
+++ b/cccl_upstream/docs/infrastructure/ci/references/change_detection.rst
@@ -0,0 +1,118 @@
+.. _infra-ci-change-detection:
+
+Change detection
+================
+
+CCCL's CI does not run every pull-request matrix job on every pull request.
+
+The PR matrix is split into two sections: ``pull_request`` and ``pull_request_lite``.
+The first contains a representative sampling of supported configurations for each CCCL project.
+The lite version is designed to 'smoke test' in situations where full coverage isn't warranted,
+but some coverage is prudent.
+
+This improves turnaround time, reduces hardware costs / energy use, and helps keep the
+runner queues short for CCCL and the other projects that we share the runners with.
+
+Two files drive the decision. ``ci/inspect_changes.py`` contains the logic, and
+``ci/project_files_and_dependencies.yaml`` defines the project graph and file mappings.
+
+Ultimately, two lists of projects are produced: ``FULL_BUILD`` and ``LITE_BUILD``.
+The workflow-build step reads these lists and assembles the final matrix by pulling the
+relevant jobs from the relevant matrix workflows.
+
+The project graph
+-----------------
+
+Each project in ``ci/project_files_and_dependencies.yaml`` declares how files
+map to it and how rebuilds propagate from it:
+
+``include_regexes``
+  Path patterns that mark this project dirty. Every pattern is a regex anchored
+  to the repository root.
+
+``exclude_regexes``
+  Patterns that remove files from the included set.
+
+``exclude_project_files``
+  When one project is nested inside another (common for public/internal splits),
+  this can be used to easily exclude the inner project's files from the outer project.
+
+``full_dependencies``
+  Projects whose direct changes force a full rebuild of this project.
+
+``lite_dependencies``
+  Projects whose changes trigger a reduced rebuild of this project.
+
+``matrix_project``
+  The name this project uses in ``ci/matrix.yaml``. A project without one is
+  internal to change detection and never appears in the output lists.
+
+The ``core`` project is a catch-all. It declares no ``include_regexes``. Any
+dirty file that no other project claims lands in ``core`` — a new top-level
+script, a ``CMakeLists.txt`` at the root, a ``ci/`` change. When ``core`` has
+any dirty file, every project goes to ``FULL_BUILD`` immediately. A CI
+infrastructure change rebuilds everything, because the extent of its impact
+cannot be easily evaluated.
+
+Files that should never trigger CI are filtered first, before project matching,
+via the top-level ``ignore_regexes`` list. Readmes, ``docs/``, config files,
+non-CI ``.github/workflows/``, and their unholy ilk should be catalogued here
+lest they anger the gods.
+
+The public/internal split
+-------------------------
+
+Large libraries are split into two projects: a ``_public`` project matching only
+the public headers, and an ``_internal`` project matching everything else.
+``libcudacxx_public`` matches ``libcudacxx/include/``. ``libcudacxx_internal``
+matches ``libcudacxx/`` and excludes ``libcudacxx_public``'s files via
+``exclude_project_files``. The same pattern holds for CUB, Thrust, cudax, and
+the C parallel library.
+
+The split exists to keep test and infrastructure churn from rebuilding the
+world. ``libcudacxx_internal`` lists ``libcudacxx_public`` as a full dependency,
+so a public-header change still rebuilds the library's own tests. But downstream
+libraries depend on ``libcudacxx_public``, not ``libcudacxx_internal``. A change
+to a libcudacxx unit test rebuilds libcudacxx and stops there. A change to a
+public header propagates outward to CUB, Thrust, and cudax — as a lite rebuild,
+not a full one.
+
+Propagation
+-----------
+
+Projects matched directly by a changed file always go to ``FULL_BUILD``,
+regardless of depth.
+
+If a dirty project has any explicit "full" dependencies, those projects are also
+added to ``FULL_BUILD``.
+
+From here, all downstream projects reachable via lite or full dependencies
+from any project in ``FULL_BUILD`` is added to ``LITE_BUILD``.
+
+Output
+------
+
+``inspect_changes.py`` emits two GitHub Actions outputs, each a space-separated
+list of ``matrix_project`` values:
+
+``FULL_BUILD``
+  Projects that build and test their full job set.
+
+``LITE_BUILD``
+  Projects that build and test a reduced set.
+
+These feed the workflow-build step. The build reads the ``pull_request`` section
+of ``ci/matrix.yaml`` (see :ref:`infra-ci-matrix-yaml`) and keeps only entries
+whose project is in ``FULL_BUILD``, then reads the ``pull_request_lite`` section
+and keeps only entries whose project is in ``LITE_BUILD``, then concatenates the
+two. A project absent from both lists contributes no jobs.
+
+To see the exact lists for any change, run ``inspect_changes.py`` locally:
+
+.. code-block:: bash
+
+    ci/inspect_changes.py --refs origin/main HEAD
+
+The script prints the dependency overview, the per-project dirty-file breakdown,
+and the final ``FULL_BUILD`` and ``LITE_BUILD`` values. Pass ``--file`` or
+``--stdin`` to supply a path list directly instead of diffing refs.
diff --git a/cccl_upstream/docs/infrastructure/ci/references/ci_artifacts.rst b/cccl_upstream/docs/infrastructure/ci/references/ci_artifacts.rst
new file mode 100644
index 00000000..f76eaef3
--- /dev/null
+++ b/cccl_upstream/docs/infrastructure/ci/references/ci_artifacts.rst
@@ -0,0 +1,103 @@
+.. _infra-ci-artifacts:
+
+Artifact system
+===============
+
+The artifact system carries data between CI jobs. A producer job uploads its build outputs;
+consumer jobs download them to run tests. It wraps GitHub Actions' native ``upload-artifact``
+and ``download-artifact`` and is GitHub Actions-specific.
+
+Artifacts are not mandatory. Projects that lean on the shared AWS sccache can let the cache
+serve build products to test jobs instead. They still pay off for larger projects: downloading
+one compressed archive of the test binaries is much faster than recompiling or fetching each
+target individually through the cache. Compare a test job's sccache build time against the cost
+of packing and downloading an archive to decide which a project should use.
+
+Scripts
+-------
+
+CI scripts call a small set of entry points under ``ci/util/artifacts/``. Each script's source
+is the authoritative reference for its full argument set.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 32 68
+
+   * - Script
+     - Purpose
+   * - ``stage.sh``
+     - Add files matching a regex to a named artifact's staged set.
+   * - ``unstage.sh``
+     - Remove files from a staged set — prune what no consumer needs.
+   * - ``upload_stage_packed.sh``
+     - Pack a staged set into a compressed ``tar.zst`` and upload it.
+   * - ``upload.sh``
+     - Upload a file, or regex-matched files, as a plain unpacked artifact.
+   * - ``download_packed.sh``
+     - Download a packed artifact and extract it to a path.
+   * - ``download.sh``
+     - Download a plain artifact file to a path.
+
+The other scripts under ``ci/util/artifacts/`` — ``register.sh``, ``pack.sh``,
+``upload_packed.sh``, and the ``upload/set_*.sh`` compression and retention knobs — are
+lower-level building blocks the entry points compose. ``upload_stage_packed.sh`` packs,
+registers, and sets compression on the archive itself, so producers rarely touch them directly.
+
+Producers and consumers find each other through ``ci/util/workflow/`` helpers:
+``get_producer_id.sh`` resolves a consumer's producer, and ``has_consumers.sh`` /
+``get_consumers.sh`` let a producer skip work when nothing downstream needs its output.
+
+Producer: stage and pack build outputs
+--------------------------------------
+
+A producer stages the files a consumer will need, prunes what it does not, then packs and
+uploads the set as one archive. It first checks whether any consumer exists and exits early if
+not:
+
+.. code-block:: bash
+
+    ci/util/workflow/has_consumers.sh "$JOB_ID" || exit 0
+
+    ci/util/artifacts/stage.sh   "<artifact_name>" '<regex>' ['<regex>' ...]
+    ci/util/artifacts/unstage.sh "<artifact_name>" '<regex>'
+    ci/util/artifacts/upload_stage_packed.sh "<artifact_name>"
+
+``stage.sh`` and ``unstage.sh`` build up the file set by inclusion and exclusion — regexes match
+against ``find`` within the stage path. ``upload_stage_packed.sh`` compresses the result and
+registers it for upload. CUB stages one packed artifact per launch-id variant (``no_lid``,
+``lid_0``–``lid_2``); ``ci/upload_cub_test_artifacts.sh`` is the authoritative example.
+
+Consumer: resolve and download
+------------------------------
+
+A consumer does not know its producer's job ID in advance. It resolves the producer from the
+run manifest, then fetches the packed outputs.
+
+Resolve the producer with ``get_producer_id.sh`` — it loads the run manifest (``workflow.json``)
+on demand and returns the producer's job ID. The packed artifact name is project-specific and
+embeds the producer ID; the consuming CI script constructs it, so there are no hardcoded names.
+
+.. code-block:: bash
+
+    producer_id=$(ci/util/workflow/get_producer_id.sh)
+    for tag in "${ARTIFACT_TAGS[@]}"; do
+      ci/util/artifacts/download_packed.sh \
+        "z_cub-test-artifacts-${DEVCONTAINER_NAME:?}-${producer_id}-${tag}" /home/coder/cccl
+    done
+
+See ``ci/test_cub.sh`` for the authoritative CUB form.
+
+Result record
+-------------
+
+Apart from build artifacts, every job records its own outcome. At exit it calls
+``ci/upload_job_result_artifacts.sh "$JOB_ID" $exit_code``, which uploads a ``zz_jobs-<job_id>``
+artifact containing a ``success`` file only when the exit code was zero. The ``ci:`` gate reads
+these records to compute the single pass/fail for the run — see :ref:`infra-ci-overview`.
+
+Python wheels
+-------------
+
+Python jobs exchange built wheels rather than packed test binaries. A producer uploads the wheel
+with ``upload.sh``; a consumer downloads it with ``download.sh``. Both resolve the filename
+through ``ci/util/workflow/get_wheel_artifact_name.sh``.
diff --git a/cccl_upstream/docs/infrastructure/ci/references/ci_overview.rst b/cccl_upstream/docs/infrastructure/ci/references/ci_overview.rst
new file mode 100644
index 00000000..2ffa3136
--- /dev/null
+++ b/cccl_upstream/docs/infrastructure/ci/references/ci_overview.rst
@@ -0,0 +1,160 @@
+.. _infra-ci-overview:
+
+CI overview
+===========
+
+CCCL runs its build and test matrix on NVIDIA's self-hosted GitHub Actions runners.
+The pipeline turns one push into hundreds of jobs across CUDA Toolkit versions,
+host compilers, GPU architectures, C++ standards, and operating systems.
+
+Triggering and the copy-pr-bot security model
+---------------------------------------------
+
+CCCL's runners have access to NVIDIA infrastructure, so arbitrary fork code must
+never run on them unreviewed. The ``copy-pr-bot`` GitHub App enforces this. CI does
+not trigger on ``pull_request`` events. It triggers on pushes to ``pull-request/<N>``
+branches in the main repository, where ``<N>`` is the PR number. ``copy-pr-bot``
+owns those branches and only writes vetted commits to them.
+
+For external contributors, CI does not begin until a maintainer leaves an
+``/ok to test [commit SHA]`` comment. ``copy-pr-bot`` verifies the SHA, copies the
+approved code to the ``pull-request/<N>`` branch, and the push to that branch starts
+CI. Every new commit on an external PR needs a fresh ``/ok to test``.
+
+For NVIDIA enterprise members with signed commits, the bot establishes identity from
+the commit signature and copies pushes automatically. CI begins immediately on each
+push, with no comment required. See the `copy-pr-bot documentation
+<https://docs.gha-runners.nvidia.com/apps/copy-pr-bot/>`_ for signing setup.
+Draft PRs do not auto-trigger — ``auto_sync_draft: false`` in
+``.github/copy-pr-bot.yaml`` holds the bot back until the PR is ready.
+
+SSH signing keys
+----------------
+
+`Signed commits <https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits>`_
+are required for any internal NVIDIA contributors who want the convenience of CI running
+automatically whenever a commit is pushed to a branch (i.e., doesn't require using ``/ok to test``).
+
+This is not required for external contributions, which will always require an explicit
+``/ok to test [commit SHA]`` comment from an approved account for each CI run.
+
+To enable commit signing using your existing ssh key, set the following git options:
+
+.. code-block:: bash
+
+    git config --global gpg.format ssh
+    git config --global user.signingKey ~/.ssh/YOUR_PUBLIC_KEY_FILE_HERE.pub
+
+    # These settings are optional. They tell git to automatically sign all new commits and tags.
+    # If these are set to false, use `git commit -S` to manually sign each commit.
+    git config --global commit.gpgsign true
+    git config --global tag.gpgsign true
+
+Git is now configured to sign commits with your ssh key.
+
+To complete the process, upload the public key to your `GitHub Signing Keys
+<https://github.com/settings/keys>`_ in your browser or using the ``gh`` CLI tool:
+
+.. code-block:: bash
+
+    gh ssh-key add ~/.ssh/YOUR_PUBLIC_KEY_FILE_HERE.pub --type signing
+
+Make sure that the key is uploaded to 'Signing Keys', not just 'Authentication Keys'.
+The same key may be used for both.
+
+Building the job matrix
+-----------------------
+
+``ci/matrix.yaml`` is the authoritative matrix definition, described in detail at
+:ref:`infra-ci-matrix-yaml`. The ``build-workflow`` action
+(``.github/actions/workflow-build/``) reads it and produces the concrete job list for
+the run. The action selects a workflow key by trigger: ``pull_request`` for PRs,
+``nightly`` and ``weekly`` for scheduled runs.
+
+Before expansion, ``ci/inspect_changes.py`` compares the PR against its merge base and
+reports which projects changed, the mechanism covered at
+:ref:`infra-ci-change-detection`. The action filters the ``pull_request`` matrix to the
+changed projects and pulls reduced ``pull_request_lite`` entries for projects only
+affected through a dependency. A change anywhere in CCCL infrastructure marks every
+project dirty and runs the full matrix. The expansion mechanics — dependency
+propagation, tag defaults, list explosion, ``job_map`` — are in
+``ci/matrix.yaml``'s ``tags``, ``jobs``, and ``projects`` sections and the
+``build-workflow.py`` action.
+
+Expansion produces ``workflow.json``: named groups (for example, "CUB GCC"), each
+carrying a ``standalone`` array and a ``two_stage`` array. A second step splits these
+into four buckets along two axes — operating system and structure:
+
+.. code-block:: text
+
+    linux_standalone     linux_two_stage
+    windows_standalone   windows_two_stage
+
+``ci-workflow-pull-request.yml`` consumes the four buckets as four parallel matrix
+dispatches, one per ``workflow-dispatch-<structure>-group-<os>.yml``. The split exists
+because Linux and Windows need different runner images and shell tooling, and because
+standalone and two-stage jobs have different dependency shapes. The workflow file
+mechanics are covered in full at :ref:`infra-ci-gha-workflows`.
+
+Standalone and two-stage jobs
+-----------------------------
+
+A standalone job builds and tests in one runner, sharing nothing with other jobs.
+
+A two-stage job splits build from test into a generic single-producer multiple-consumer model.
+A single producer job creates artifacts (usually on a cheap non-GPU runner), which are then
+consumed in one or more test jobs, which may uses GPUs if needed.
+
+Accelerating build times with sccache
+-------------------------------------
+
+CCCL's CI uses a `heavily modified fork of sccache <https://github.com/rapidsai/sccache>`_ with
+improved CUDA support to cache compiler artifacts for files that haven't changed and dramatically
+accelerate build times. Local builds inside `CCCL's
+Dev Containers <https://github.com/NVIDIA/cccl/blob/main/.devcontainer/README.md>`_ can share the
+same cache such that local builds and CI jobs mutually benefit from accelerated build times.
+Follow the `GitHub Authentication
+<https://github.com/NVIDIA/cccl/blob/main/.devcontainer/README.md#optional-authenticate-with-github-for-sccache>`_
+guide to enable this feature.
+
+PR Branch Protections
+---------------------
+
+The pull request workflow produces a single job that gates PR mergability, named "CI."
+This sentinel job depends on and checks for success of every required top-level job in the
+workflow.
+
+This test also checks for skip tags, override matrix modifications, open benchmark requests, etc.
+and will block merging until the tree is restored to a mergeable state.
+
+Override and skip enforcement
+-----------------------------
+
+A non-empty ``workflows.override`` in ``ci/matrix.yaml`` replaces the ``pull_request``
+matrix entirely, detailed at :ref:`infra-ci-override-matrix`. Use it to run a subset of jobs
+when a full run would be unnecessary / wasteful during iteration.
+The override deliberately blocks merge: the full suite must run before any PR lands,
+so the override must be emptied and re-pushed before acceptance testing.
+
+Skip tags in the last commit message on the branch, documented at :ref:`infra-ci-skip-tags`, drop job
+groups for the next run. They also block merge while present, forcing a clean final run:
+
+.. code-block:: bash
+
+    git commit -m "README tidy-up [skip-matrix][skip-vdc][skip-docs][skip-tpt]"
+    git commit -m "Run PR benchmarks [bench-only]"
+
+The recognized tags and their semantics are catalogued in :ref:`infra-ci-skip-tags`.
+``[bench-only]`` is shorthand for the common benchmark-request combination.
+
+Reproducing a failure locally
+-----------------------------
+
+CI jobs run the build and test scripts in ``ci/`` inside the devcontainers described in
+``.devcontainer/README.md``. A failing job's log names the exact container and script
+invocation. Pull the same container and run the same ``ci/build_<project>.sh`` or
+``ci/test_<project>.sh`` line to reproduce the CI environment, as walked through at
+:ref:`infra-ci-reproducing-locally`. For targeted single-test iteration,
+``ci/util/build_and_test_targets.sh`` builds and runs a named subset, covered at
+:ref:`infra-ci-targeted-builds`; for a regression hunt, see
+:doc:`/cccl/development/build_and_bisect_tools`.
diff --git a/cccl_upstream/docs/infrastructure/ci/references/ci_scripts.rst b/cccl_upstream/docs/infrastructure/ci/references/ci_scripts.rst
new file mode 100644
index 00000000..73e54c76
--- /dev/null
+++ b/cccl_upstream/docs/infrastructure/ci/references/ci_scripts.rst
@@ -0,0 +1,66 @@
+.. _infra-ci-scripts:
+
+CI scripts
+==========
+
+The ``ci/`` directory holds the build, test, benchmark, and utility scripts that CI jobs
+invoke. The same scripts reproduce a CI environment locally — a failing job's log prints the
+exact script and arguments it ran. Scripts are organized by role:
+
+- ``ci/`` root — per-project build and test entry points.
+- ``ci/util/`` — shared utilities, plus the artifact and workflow plumbing jobs use to pass data.
+- ``ci/bench/`` — benchmark drivers.
+- ``ci/windows/`` — PowerShell equivalents of the per-project scripts for Windows runners.
+
+Per-project build and test scripts
+----------------------------------
+
+Each project has a pair of scripts following a fixed naming convention:
+``ci/build_<project>.sh`` configures and builds it, ``ci/test_<project>.sh`` builds and runs
+its tests. Both source ``ci/build_common.sh`` for argument parsing and CMake setup, so every
+project shares one flag interface. Building tests needs no GPU; running them (usually) does.
+
+Specialized build scripts cover configurations that fall outside the per-project pattern —
+stdpar, clang-tidy, Python wheels, NVRTC, and codegen verification among them. They live
+beside the per-project scripts in ``ci/`` and source the same common setup.
+
+The flags come from ``ci/build_common.sh``: the host and CUDA compilers (``-cxx``, ``-cuda``),
+the C++ standard (``-std``), target architectures (``-arch``), forwarded CMake options
+(``-cmake-options``), and ``-configure`` to stop after configuration.
+``PARALLEL_LEVEL`` controls build parallelism. Run a script with ``-h`` for the authoritative
+flag list and current defaults.
+
+When run locally, the test scripts will invoke the build script to ensure that the targets are
+available. In CI, they may download GHA artifacts instead.
+
+::
+
+    ./ci/test_cub.sh  -cxx g++ -std 17 -arch "70;80;90"
+
+For fast local iteration on a single target rather than a whole project, see
+:doc:`/cccl/development/build_and_bisect_tools`.
+
+Utility scripts: ci/util/
+-------------------------
+
+``ci/util/`` collects tooling shared across jobs: a targeted build-and-test runner
+(``build_and_test_targets.sh``), automated ``git bisect`` over a build/test command, command
+retry, peak-memory monitoring, and a mock job environment (``create_mock_job_env.sh``) that
+lets the artifact and workflow scripts run outside GitHub Actions. Run any script with ``-h``
+for its options.
+
+Two subdirectories carry the producer/consumer plumbing for two-stage jobs:
+``ci/util/artifacts/`` uploads and downloads the files passed between jobs, and
+``ci/util/workflow/`` resolves producer/consumer relationships for the current run. Both are
+covered at :ref:`infra-ci-artifacts`.
+
+``ci/inspect_changes.py`` reports which projects are dirty between two refs and drives full
+versus lite matrix selection; see :ref:`infra-ci-change-detection`.
+
+Benchmark scripts: ci/bench/
+----------------------------
+
+``ci/bench/`` holds the benchmark drivers: ``bench.sh`` builds and runs the suite for a
+configuration, and the ``compare_*`` scripts build two refs or two paths and diff the results.
+The comparison workflow is ``.github/workflows/bench.yml``. PR request syntax lives in
+``ci/bench.yaml``; ``ci/bench/README.md`` documents local usage and artifact layout.
diff --git a/cccl_upstream/docs/infrastructure/ci/references/gha_workflows.rst b/cccl_upstream/docs/infrastructure/ci/references/gha_workflows.rst
new file mode 100644
index 00000000..6b1177ba
--- /dev/null
+++ b/cccl_upstream/docs/infrastructure/ci/references/gha_workflows.rst
@@ -0,0 +1,161 @@
+.. _infra-ci-gha-workflows:
+
+GitHub Actions workflows
+========================
+
+Every CCCL workflow lives in ``.github/workflows/``. They fall into five purposes:
+the CI matrix, manual developer tools, the release pipeline, documentation
+deployment, and repository automation. This page catalogs each workflow, then
+explains the workflow-file mechanics that turn one push into hundreds of jobs.
+For the high-level CI narrative — triggering, change detection, the ``ci:`` gate —
+see :ref:`infra-ci-overview`.
+
+Workflow reference
+------------------
+
+CI matrix
+~~~~~~~~~
+
+Three trigger workflows select a matrix key and run it. Six dispatcher workflows
+unfold the generated job list into GitHub matrix strategies. Contributors rarely
+invoke any of these directly; they fire on push or schedule.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 40 20 40
+
+   * - Workflow
+     - Trigger
+     - Purpose
+   * - ``.github/workflows/ci-workflow-pull-request.yml``
+     - Automatic
+     - Runs the ``pull_request`` matrix on every push to a ``pull-request/<N>`` branch.
+   * - ``.github/workflows/ci-workflow-nightly.yml``
+     - Both
+     - Runs the broader ``nightly`` matrix on a weekday schedule, with Slack notifications.
+   * - ``.github/workflows/ci-workflow-weekly.yml``
+     - Both
+     - Runs the exhaustive ``weekly`` matrix on a Sunday schedule, with Slack notifications.
+   * - ``.github/workflows/workflow-dispatch-standalone-group-linux.yml``
+     - Automatic
+     - Dispatches an array of standalone Linux jobs as a matrix.
+   * - ``.github/workflows/workflow-dispatch-standalone-group-windows.yml``
+     - Automatic
+     - Dispatches an array of standalone Windows jobs as a matrix.
+   * - ``.github/workflows/workflow-dispatch-two-stage-group-linux.yml``
+     - Automatic
+     - Dispatches an array of producer/consumer Linux chains as a matrix.
+   * - ``.github/workflows/workflow-dispatch-two-stage-group-windows.yml``
+     - Automatic
+     - Dispatches an array of producer/consumer Windows chains as a matrix.
+   * - ``.github/workflows/workflow-dispatch-two-stage-linux.yml``
+     - Automatic
+     - Executes one producer/consumer chain on Linux.
+   * - ``.github/workflows/workflow-dispatch-two-stage-windows.yml``
+     - Automatic
+     - Executes one producer/consumer chain on Windows.
+
+Manual tools
+~~~~~~~~~~~~
+
+Contributors invoke these through ``workflow_dispatch`` inputs on the Actions tab.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 40 20 40
+
+   * - Workflow
+     - Trigger
+     - Purpose
+   * - ``.github/workflows/bench.yml``
+     - Both
+     - Compares benchmark performance between two refs; also called by the PR workflow's bench dispatch.
+   * - ``.github/workflows/git-bisect.yml``
+     - Manual
+     - Runs an automated ``git bisect`` across a commit range to locate a regression.
+
+Documentation
+~~~~~~~~~~~~~
+
+.. list-table::
+   :header-rows: 1
+   :widths: 40 20 40
+
+   * - Workflow
+     - Trigger
+     - Purpose
+   * - ``.github/workflows/docs-deploy.yml``
+     - Both
+     - Builds the Sphinx docs and publishes them to GitHub Pages under a version-derived path.
+
+
+CI orchestration
+----------------
+
+The build-workflow action
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The "build workflow from matrix" job runs the ``build-workflow`` action at
+``.github/actions/workflow-build/``. Its script ``build-workflow.py`` reads
+``ci/matrix.yaml`` together with the change-detection output from
+``ci/inspect_changes.py`` and produces ``workflow/workflow.json``. That file holds
+named groups — "CUB GCC", for example — each carrying a ``standalone`` array and a
+``two_stage`` array of fully expanded job specs.
+
+A second script, ``prepare-workflow-dispatch.py``, splits ``workflow.json`` into
+``dispatch.json``. This is the structure the GHA workflow is actually dispatched over.
+
+The four-bucket split
+~~~~~~~~~~~~~~~~~~~~~
+
+``dispatch.json`` carries four buckets, keyed by operating system crossed with job
+structure:
+
+.. code-block:: text
+
+    linux_standalone     linux_two_stage
+    windows_standalone   windows_two_stage
+
+Each bucket holds an ordered ``keys`` list of group names and a ``jobs`` map from
+each key to its job specs:
+
+.. code-block:: text
+
+    {
+      "linux_two_stage": {
+        "keys": ["CUB GCC", "Thrust Clang", ...],
+        "jobs": {
+          "CUB GCC":      [{ "producers": [...], "consumers": [...] }],
+          "Thrust Clang": [{ "producers": [...], "consumers": [...] }]
+        }
+      },
+      "linux_standalone":   { "keys": [...], "jobs": {...} },
+      "windows_two_stage":  { "keys": [...], "jobs": {...} },
+      "windows_standalone": { "keys": [...], "jobs": {...} }
+    }
+
+``ci-workflow-pull-request.yml`` dispatches four parallel matrix jobs, one per
+bucket. Each fans out over its bucket's ``keys`` and calls the matching dispatcher
+workflow — ``workflow-dispatch-<structure>-group-<os>.yml``.
+
+Standalone jobs
+~~~~~~~~~~~~~~~
+
+A standalone job builds and tests on one runner, exchanging no artifacts with other
+jobs. The group dispatcher
+(``workflow-dispatch-standalone-group-<os>.yml``) fans out over the bucket's job
+array; each matrix entry runs one CI script — ``ci/build_<project>.sh`` or
+``ci/test_<project>.sh`` — start to finish. See :ref:`infra-ci-scripts` for details.
+
+Two-stage jobs
+~~~~~~~~~~~~~~
+
+A typical two-stage workflow splits a build job from one or more test jobs. Each stage
+has different workloads and hardware requirements, and this division allows us to schedule
+each on appropriate hardware.
+
+The producer runs on a GPU-less, CPU-heavy runner.
+It builds and uploads binaries through GHA artifacts (:ref:`infra-ci-artifacts`) or sccache.
+
+Each consumer runs on a CPU-light, GPU-attached runner.
+They fetch the pre-compiled test binaries and execute them.
diff --git a/cccl_upstream/docs/infrastructure/ci/references/index.rst b/cccl_upstream/docs/infrastructure/ci/references/index.rst
new file mode 100644
index 00000000..18522a2d
--- /dev/null
+++ b/cccl_upstream/docs/infrastructure/ci/references/index.rst
@@ -0,0 +1,18 @@
+.. _ci-references-index:
+
+CI Reference
+============
+
+Reference pages for CI configuration, scripts, and outputs.
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   ci_overview
+   change_detection
+   matrix_yaml
+   ci_scripts
+   ci_artifacts
+   gha_workflows
+   skip_tags
diff --git a/cccl_upstream/docs/infrastructure/ci/references/matrix_yaml.rst b/cccl_upstream/docs/infrastructure/ci/references/matrix_yaml.rst
new file mode 100644
index 00000000..fc4d6882
--- /dev/null
+++ b/cccl_upstream/docs/infrastructure/ci/references/matrix_yaml.rst
@@ -0,0 +1,183 @@
+.. _infra-ci-matrix-yaml:
+
+matrix.yaml reference
+=====================
+
+``ci/matrix.yaml`` is the authoritative definition of CCCL's CI job matrix. It declares
+the workflows, the toolchain and hardware vocabulary jobs draw from, and the per-workflow
+job entries that expand into individual GitHub Actions jobs.
+
+Top-level keys
+--------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 24 76
+
+   * - Key
+     - Purpose
+   * - ``workflows``
+     - Map of workflow name to a list of job entries. Holds ``override``, ``pull_request``,
+       ``pull_request_lite``, ``nightly``, ``weekly``, etc.
+   * - ``devcontainer_version``
+     - Image version tag for ``rapidsai/devcontainers``.
+   * - ``cuda99_gcc_version``, ``cuda99_clang_version``
+     - Compiler versions used for internal cuda99.X builds.
+   * - ``all_stds``
+     - Every C++ standard CCCL supports (used for ``std: 'all'`` et al).
+   * - ``ctk_versions``
+     - Map of supported CUDA Toolkit versions to supported standards and aliases.
+   * - ``device_compilers``
+     - Device compiler definitions (``nvcc``, ``clang``). Selected by the ``cudacxx`` tag.
+   * - ``host_compilers``
+     - Host compiler definitions (``gcc``, ``clang``, ``msvc``, ``nvhpc``) and their per-version
+       standards. Selected by the ``cxx`` tag.
+   * - ``jobs``
+     - Job type definitions: GPU requirement, dependencies, and script invocation.
+   * - ``projects``
+     - Project definitions: supported standards, display name, and ``job_map`` expansions.
+   * - ``gpus``
+     - GPU runner pools and their ``sm`` value.
+   * - ``tags``
+     - The fields a job entry accepts, with ``required`` flags and defaults.
+
+``exclude`` is nested under ``workflows``. Entries matching an ``exclude`` rule are removed
+from every workflow's generated matrix.
+
+Workflow types
+--------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 24 76
+
+   * - Workflow
+     - Purpose
+   * - ``override``
+     - Overrides the CI jobs run for the active PR when defined. Blocks merge while set.
+   * - ``pull_request``
+     - Default per-PR matrix.
+   * - ``pull_request_lite``
+     - Reduced matrix run when only an upstream dependency changed. (See :ref:`infra-ci-change-detection`.)
+   * - ``nightly``
+     - Extended scheduled matrix.
+   * - ``weekly``
+     - Broadest scheduled matrix, including ``sm: 'all-cccl'`` and compute-sanitizer coverage.
+   * - ``python-wheels``
+     - Python wheel build and test matrix.
+   * - ``devcontainers``
+     - Image-generation matrix. Entries catalog currently available devcontainer configs and map to no real jobs.
+
+Job entry format
+----------------
+
+A job entry is a YAML mapping. Array-valued fields expand to the cross-product of their
+elements; ``exclude`` rules are applied after expansion. The fields a user specifies are the
+tags in the ``tags`` section.
+
+Current defaults for all tags are defined in the ``tags`` section of ``ci/matrix.yaml``.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 18 82
+
+   * - Field
+     - Meaning
+   * - ``jobs``
+     - Job types to run (e.g. ``build``, ``test``, ``nvrtc``, ``verify_codegen``). Expanded by
+       the project's ``job_map``. Dependencies added automatically. Required.
+   * - ``project``
+     - Project key from the ``projects`` section.
+   * - ``ctk``
+     - CUDA Toolkit version or alias from ``ctk_versions``.
+   * - ``cxx``
+     - Host compiler. A bare name resolves to the latest version (see below).
+   * - ``cudacxx``
+     - Device compiler from ``device_compilers``; rarely needed, defaulting to ``nvcc``.
+   * - ``std``
+     - C++ standard. Accepts an integer or the ``all`` / ``min`` / ``max`` / ``minmax`` shortcuts.
+   * - ``cpu``
+     - CPU architecture (``amd64``, ``arm64``).
+   * - ``gpu``
+     - GPU runner type from ``gpus``.
+   * - ``sm``
+     - GPU architectures, ``CMAKE_CUDA_ARCHITECTURES`` syntax. ``gpu`` targets the ``gpu`` tag's SM.
+       Omitting ``sm`` defers architecture selection to defaults in build scripts and ``CMakePresets.json``.
+   * - ``py_version``
+     - Python version for Python jobs.
+   * - ``args``
+     - Arguments appended to the generated command. Forwards options to
+       ``ci/util/build_and_test_targets.sh`` for the ``target`` project, but works for any job.
+   * - ``cmake_options``
+     - Extra CMake defines, passed as ``-cmake_options "<value>"``.
+   * - ``environment``
+     - Environment variables injected into the job.
+
+Computed internally, not user-specified:
+
+- ``needs`` — defined in the ``jobs`` section. A ``test`` entry auto-generates its ``build`` producer.
+- ``gpu`` requirement, ``cuda_ext``, ``name``, and ``invoke`` script details - also taken from the ``jobs`` section.
+- ``force_producer_ctk`` — set in the ``jobs`` section to pin a producer build's CTK independent
+  of the consumer's ``ctk`` tag. Used mainly for python packaging special cases.
+
+Annotated example
+~~~~~~~~~~~~~~~~~
+
+::
+
+    - {jobs: ['test'],          # build (auto-added) then test
+       project: 'thrust',       # only thrust
+       std: 'max',              # highest std supported by thrust, ctk, and compiler
+       cxx: ['gcc', 'clang'],   # expands to two jobs, latest gcc and latest clang
+       gpu: 'h100',             # test job runs on an h100 runner
+       sm: 'gpu',               # build for the SM of the h100 runner (sm_90)
+       cmake_options: '-DCMAKE_CUDA_FLAGS="-lineinfo"'} # customize cmake config
+
+This expands to two ``build`` jobs — one per compiler. Thrust's ``job_map`` expands ``test``
+to ``test_cpu`` and ``test_gpu``, so the two compilers produce four combined test jobs. The test
+jobs request h100 runners; the auto-generated ``build`` producers run on CPU runners using
+``-lineinfo`` for CUDA targets and only produce device code for the h100's SM90 arch.
+
+In total this spawns six jobs:
+
+- build gcc
+- build clang
+- test_cpu gcc
+- test_cpu clang
+- test_gpu gcc
+- test_gpu clang
+
+C++ standard resolution
+-----------------------
+
+``std`` accepts integers or four keywords resolved against the intersection of the standards
+supported by the selected CTK, host compiler, device compiler, and project:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 18 82
+
+   * - Value
+     - Resolves to
+   * - ``all``
+     - One job per supported standard.
+   * - ``min``
+     - The lowest supported standard.
+   * - ``max``
+     - The highest supported standard.
+   * - ``minmax``
+     - The lowest and highest, two jobs.
+
+Compiler version resolution
+---------------------------
+
+A bare compiler name in ``cxx`` resolves to the latest version listed for that compiler in
+``host_compilers``. ``cxx: 'gcc'`` selects the highest ``gcc`` version under
+``host_compilers.gcc.versions``. Pin a version by naming it: ``cxx: 'gcc13'``.
+
+Version aliases resolve the same way. ``cxx: 'msvc2022'`` maps to the ``msvc`` version whose
+``alias`` is ``2022``. CTK aliases follow the ``ctk_versions`` map: ``ctk: '13.X'`` selects the
+newest CTK 13 entry, ``ctk: 'nvhpc'`` selects the CTK shipped in the current NVHPC.
+
+Each version entry declares its own supported standards. ``std`` resolution and the
+``exclude`` rules drop combinations a compiler version does not support.
diff --git a/cccl_upstream/docs/infrastructure/ci/references/skip_tags.rst b/cccl_upstream/docs/infrastructure/ci/references/skip_tags.rst
new file mode 100644
index 00000000..0470cae7
--- /dev/null
+++ b/cccl_upstream/docs/infrastructure/ci/references/skip_tags.rst
@@ -0,0 +1,69 @@
+.. _infra-ci-skip-tags:
+
+Skip tags
+=========
+
+Skip tags scope a pull-request CI run. Place a tag in the last commit message before pushing.
+The next PR run reads the last commit message and filters job groups accordingly. Combine tags
+with the override matrix in ``ci/matrix.yaml`` for finer control.
+
+All ``[skip-*]`` and ``[bench-only]`` tags block merge while present. Remove them from the last
+commit before merging.
+
+Tag reference
+-------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 30 50 20
+
+   * - Tag
+     - Skips
+     - Blocks merge
+   * - ``[skip-matrix]``
+     - All build and test jobs from ``ci/matrix.yaml``.
+     - Yes
+   * - ``[skip-vdc]``
+     - All "Validate Devcontainer" jobs.
+     - Yes
+   * - ``[skip-docs]``
+     - The documentation verification build.
+     - Yes
+   * - ``[skip-tpt]`` / ``[skip-third-party-testing]``
+     - All third-party canary builds (MatX, PyTorch, RAPIDS).
+     - Yes
+   * - ``[skip-rapids]``
+     - RAPIDS canary builds only.
+     - Yes
+   * - ``[skip-matx]``
+     - MatX canary builds only.
+     - Yes
+   * - ``[skip-pytorch]``
+     - PyTorch canary builds only.
+     - Yes
+   * - ``[bench-only]``
+     - Equivalent to ``[skip-matrix][skip-vdc][skip-docs][skip-tpt]``.
+     - Yes
+
+``[skip-tpt]`` and ``[skip-third-party-testing]`` are aliases for the same tag.
+
+``[bench-only]`` shorthand
+--------------------------
+
+``[bench-only]`` expands to ``[skip-matrix][skip-vdc][skip-docs][skip-tpt]``. It skips all
+non-benchmark job groups. Benchmarks are triggered separately by modifying ``ci/bench.yaml``
+relative to ``ci/bench.template.yaml``.
+
+Usage
+-----
+
+Place one or more tags in the last commit message before pushing::
+
+    git commit -m "Run PR benchmarks [bench-only]"
+    git commit -m "README tidy-up [skip-matrix][skip-vdc][skip-docs][skip-third-party-testing]"
+
+Tags act on the last commit message of the branch. To change which tags apply, amend the last
+commit and re-push.
+
+Remove all skip tags from the last commit before merging. The full CI suite must run on the final
+commit for the PR to land.
diff --git a/cccl_upstream/docs/infrastructure/cmake/references/architecture_flags.rst b/cccl_upstream/docs/infrastructure/cmake/references/architecture_flags.rst
new file mode 100644
index 00000000..94cfddcc
--- /dev/null
+++ b/cccl_upstream/docs/infrastructure/cmake/references/architecture_flags.rst
@@ -0,0 +1,23 @@
+.. _infra-cmake-architecture-flags:
+
+Architecture flags
+==================
+
+CCCL builds device code for the SM architectures named in ``CMAKE_CUDA_ARCHITECTURES``, a
+semicolon-separated list of SM numbers, each optionally tagged ``-real`` (embed SASS) or
+``-virtual`` (embed PTX for JIT on newer GPUs). CMake also accepts ``native``, ``all``, and
+``all-major``. The
+`CUDA_ARCHITECTURES property <https://cmake.org/cmake/help/latest/prop_tgt/CUDA_ARCHITECTURES.html>`_
+documents the standard syntax and values.
+
+CCCL adds two values, expanded at configure time by ``cccl_check_cuda_architectures()`` against
+the toolkit found at configure time:
+
+- ``all-cccl`` — every architecture the current nvcc supports at or above CCCL's minimum.
+- ``all-major-cccl`` — one entry per major architecture at or above the minimum, carrying
+  forward PTX on the highest.
+
+Both resolve against the installed toolkit rather than a fixed table. The minimum supported
+architecture and the full expansion logic live in ``cmake/CCCLCheckCudaArchitectures.cmake``.
+Expansion runs only for a top-level CCCL build; downstream consumers pass a concrete list or
+``native``.
diff --git a/cccl_upstream/docs/infrastructure/cmake/references/cmake_modules.rst b/cccl_upstream/docs/infrastructure/cmake/references/cmake_modules.rst
new file mode 100644
index 00000000..522385b2
--- /dev/null
+++ b/cccl_upstream/docs/infrastructure/cmake/references/cmake_modules.rst
@@ -0,0 +1,102 @@
+.. _infra-cmake-helper-modules:
+
+CMake helper modules
+====================
+
+CCCL's ``cmake/`` directory holds the helper modules, script templates, and the vendored
+CPM package manager that drive the build. Library ``CMakeLists.txt`` files include these
+modules to add executables, generate header tests, expand architecture lists, wire install
+rules, and fetch dependencies. Each entry covers the module's purpose and the functions or
+macros it provides.
+
+Helper modules
+--------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 26 40 34
+
+   * - File
+     - Purpose
+     - Key functions / macros
+   * - ``CCCLAddExecutable.cmake``
+     - Add an executable with standard CCCL configuration, optional CTest registration, metatargets, and clang-tidy.
+     - ``cccl_add_executable()``
+   * - ``CCCLAddSubdir.cmake``
+     - Pull the CCCL libraries into the build as in-tree subprojects via ``cccl_add_subdir_helper``, honoring ``CCCL_REQUIRED_COMPONENTS`` / ``CCCL_OPTIONAL_COMPONENTS``.
+     - (calls ``cccl_add_subdir_helper()``)
+   * - ``CCCLAddSubdirHelper.cmake``
+     - Standardizes project/subproject behavior when included with ``add_subdirectory()`` by a consumer/CPM.
+     - ``cccl_add_subdir_helper()``
+   * - ``CCCLAddTidyTarget.cmake``
+     - clang-tidy integration: global, per-subproject, and per-source analysis targets.
+     - ``cccl_tidy_init()``, ``cccl_tidy_make_subproject_target()``, ``cccl_tidy_add_target()``
+   * - ``CCCLBuildCompilerTargets.cmake``
+     - Build the ``cccl.compiler_interface`` target carrying warning, RTTI, exception, and ptxas flags.
+     - ``cccl_build_compiler_targets()``
+   * - ``CCCLCheckCudaArchitectures.cmake``
+     - Expand the special ``all-cccl`` and ``all-major-cccl`` values for ``CMAKE_CUDA_ARCHITECTURES`` against the current NVCC, filtered to the minimum CCCL-supported arch.
+     - ``cccl_check_cuda_architectures()``
+   * - ``CCCLClangdCompileInfo.cmake``
+     - Enable ``CMAKE_EXPORT_COMPILE_COMMANDS`` and symlink ``compile_commands.json`` into the source tree for clangd.
+     - (script; no public functions)
+   * - ``CCCLConfigureTarget.cmake``
+     - Apply common target properties: disable extensions, set and require the C++/CUDA standard, propagate dialect compile features, set output directories.
+     - ``cccl_configure_target()``
+   * - ``CCCLDevBuildChecks.cmake``
+     - Enforce supported developer-build configuration: require matching ``CMAKE_CXX_STANDARD`` and ``CMAKE_CUDA_STANDARD``, default both to 17.
+     - ``cccl_dev_build_checks()``
+   * - ``CCCLEnsureMetaTargets.cmake``
+     - Create the dot-path metatarget hierarchy so ``ninja cub.test`` builds all descendants of ``cub.test``.
+     - ``cccl_ensure_metatargets()``
+   * - ``CCCLGenerateHeaderTests.cmake``
+     - Generate per-header compilation tests from a template to verify headers are self-contained, plus a link-check executable that catches missing ``inline`` markup.
+     - ``cccl_generate_header_tests()``
+   * - ``CCCLGetDependencies.cmake``
+     - Fetch external and in-tree dependencies via ``find_package`` or CPM. NVBench SHA is pinned in ``CCCL_NVBENCH_SHA``.
+     - ``cccl_get_<dependency>()``
+   * - ``CCCLHideThirdPartyOptions.cmake``
+     - Mark Catch2, CPM, FetchContent, and LLVM cache variables advanced to keep them out of the default cache view.
+     - (script; ``mark_as_advanced`` only)
+   * - ``CCCLInstallRules.cmake``
+     - Generate header and CMake-config install rules per project, gated by a ``<project>_ENABLE_INSTALL_RULES`` cache option.
+     - ``cccl_generate_install_rules()``
+   * - ``CCCLTestParams.cmake``
+     - Parse ``%PARAM%`` comments in test sources into the cartesian product of variant labels and preprocessor definitions. See :doc:`/cccl/development/testing` for usage.
+     - ``cccl_parse_variant_params()``
+   * - ``CCCLUtilities.cmake``
+     - Shared utilities: non-fatal process execution, CPM-consumption compile tests, and expected-failure compile tests.
+     - ``cccl_execute_non_fatal_process()``, ``cccl_add_compile_test()``, ``cccl_add_xfail_compile_target_test()``
+   * - ``AppendOptionIfAvailable.cmake``
+     - Append a compiler flag to a list only if a ``check_cxx_compiler_flag`` probe accepts it.
+     - ``append_option_if_available()``
+   * - ``CPM.cmake``
+     - Vendored CPM.cmake package manager. Used by ``CCCLGetDependencies.cmake`` and by downstream consumers fetching CCCL.
+     - ``CPMAddPackage()`` (third-party)
+
+Install rule files
+------------------
+
+``cmake/install/`` holds one file per installable project. Each calls
+``cccl_generate_install_rules()`` with that project's header subdirectories and packaging
+options.
+
+Adding an executable
+--------------------
+
+``cccl_add_executable()`` is the entry point most test and example ``CMakeLists.txt`` files
+use. It calls ``cccl_configure_target()`` for standard properties, registers metatargets via
+``cccl_ensure_metatargets()``, and adds a clang-tidy target via ``cccl_tidy_add_target()``.
+
+::
+
+   cccl_add_executable(cub.test.device_reduce
+     SOURCES test_device_reduce.cu
+     ADD_CTEST
+   )
+
+``ADD_CTEST`` registers a CTest that runs the executable with no arguments. ``NO_METATARGETS``
+and ``NO_CLANG_TIDY`` opt out of those integrations. ``METATARGET_PATH`` overrides the dot-path
+(default: the target name). ``DIALECT`` forces a C++ standard for this target.
+
+Architecture-flag expansion is covered in :ref:`infra-cmake-architecture-flags`.
diff --git a/cccl_upstream/docs/infrastructure/cmake/references/index.rst b/cccl_upstream/docs/infrastructure/cmake/references/index.rst
new file mode 100644
index 00000000..c8ef08e6
--- /dev/null
+++ b/cccl_upstream/docs/infrastructure/cmake/references/index.rst
@@ -0,0 +1,14 @@
+.. _cmake-references:
+
+CMake Reference
+===============
+
+Reference pages for the CCCL CMake build system.
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   preset_reference
+   cmake_modules
+   architecture_flags
diff --git a/cccl_upstream/docs/infrastructure/cmake/references/preset_reference.rst b/cccl_upstream/docs/infrastructure/cmake/references/preset_reference.rst
new file mode 100644
index 00000000..921f0225
--- /dev/null
+++ b/cccl_upstream/docs/infrastructure/cmake/references/preset_reference.rst
@@ -0,0 +1,57 @@
+.. _infra-cmake-preset-reference:
+
+Preset reference
+================
+
+CCCL ships its CMake configurations as presets in ``CMakePresets.json``, using CMake's
+`preset format <https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html>`_. The file
+defines 40+ presets covering each library, multiple C++ standards, special build
+modes, and benchmarking.
+
+Preset structure
+----------------
+
+Each preset belongs to one of three sections:
+
+- ``configurePresets`` set the configuration: enabled libraries, build type, CUDA
+  architectures, C++ standard, and per-library options. Every configure preset
+  inherits from the hidden ``base`` preset, which selects the Ninja generator,
+  Release mode, ``all-major-cccl`` architectures, and disables all libraries by
+  default. A named preset enables the libraries and options it needs.
+- ``buildPresets`` reference a configure preset by name. Some pin an explicit
+  target list; most build everything the configuration enables.
+- ``testPresets`` reference a configure preset and add CTest filters. Filters
+  select a subset of tests by name regex. CUB launcher-mode presets and Thrust
+  GPU/CPU splits are examples of this pattern.
+
+A configure preset, its build preset, and its test preset share a name. Run all
+three with the same ``<name>``.
+
+Listing available presets
+-------------------------
+
+``cmake --list-presets`` prints all configure presets::
+
+   cmake --list-presets
+
+List build and test presets separately::
+
+   cmake --list-presets=build
+   cmake --build --list-presets
+   ctest --list-presets
+
+Using a preset
+--------------
+
+Configure, then build with the matching preset name::
+
+   cmake --preset cub-cpp17
+   cmake --build --preset cub-cpp17
+
+Run the test preset of the same name with ``ctest``::
+
+   ctest --preset cub-cpp17
+
+Build output lands under ``build/<infix>/<presetName>/``, where the infix comes from
+the ``CCCL_BUILD_INFIX`` environment variable used for devcontainer isolation.
+Distinct presets use distinct build directories and do not collide.
diff --git a/cccl_upstream/docs/infrastructure/devcontainer/references/devcontainer_overview.rst b/cccl_upstream/docs/infrastructure/devcontainer/references/devcontainer_overview.rst
new file mode 100644
index 00000000..34eaa9fa
--- /dev/null
+++ b/cccl_upstream/docs/infrastructure/devcontainer/references/devcontainer_overview.rst
@@ -0,0 +1,52 @@
+.. _infra-devcontainer-overview:
+
+Devcontainer overview
+=====================
+
+CCCL devcontainers package a CUDA toolkit and host compiler into a Docker image
+that matches a CI environment exactly. The same image that builds and tests a
+project in GitHub Actions runs on a developer's machine. Reproducing a CI
+result locally requires no guesswork about toolchain versions: pick the
+combination, launch the container, and the compiler, CTK, and supporting tools
+are identical to what CI used.
+
+Two uses: local development and CI
+----------------------------------
+
+**Local development.** Open the repository in VSCode and select a devcontainer
+from the picker, or launch one directly with ``.devcontainer/launch.sh``.
+:ref:`infra-devcontainer-launch-sh-reference` covers ``launch.sh`` flags and the available
+combinations. ``launch.sh --docker`` runs the container without VSCode and drops
+into a shell or runs a script. Without ``--docker``, it opens the container in
+VSCode.
+
+**CI.** GitHub Actions runs the identical images. A CI job for a given CTK and
+compiler combination builds and tests inside the same ``rapidsai/devcontainers``
+image a developer would launch locally.
+
+Image source: rapidsai/devcontainers
+------------------------------------
+
+The base images are built and published by the
+`rapidsai/devcontainers <https://github.com/rapidsai/devcontainers>`_ repository,
+not by CCCL. Each image bundles a CUDA toolkit, a host compiler, and
+development tooling. CCCL references these images by tag; it does not build them.
+
+Image tags follow the pattern
+``rapidsai/devcontainers:<version>-cpp-<compiler><version>-cuda<ctk>[ext]``.
+The ``-cuda<ctk>`` segment is present for every combination except nvhpc, which
+bundles its own CUDA toolkit; nvhpc images omit it.
+The ``<version>`` tag is defined in ``ci/matrix.yaml`` under
+``devcontainer_version``. A generated ``.devcontainer/<combo>/devcontainer.json``
+contains the resolved tag for any given combination.
+
+This upstream dependency gates toolchain changes. Adding a CUDA toolkit version
+or a host compiler requires the matching image to exist in rapidsai/devcontainers
+first. The sequence is:
+
+#. Update rapidsai/devcontainers to build and publish images for the new
+   combination.
+#. Update CCCL's ``ci/matrix.yaml`` to reference the new version.
+#. Regenerate CCCL's devcontainer configs to match all matrix workflow requirements.
+
+:ref:`infra-devcontainer-adding-toolchain` has additional details on this process.
diff --git a/cccl_upstream/docs/infrastructure/devcontainer/references/index.rst b/cccl_upstream/docs/infrastructure/devcontainer/references/index.rst
new file mode 100644
index 00000000..acd5575d
--- /dev/null
+++ b/cccl_upstream/docs/infrastructure/devcontainer/references/index.rst
@@ -0,0 +1,13 @@
+.. _infra-devcontainer-references-index:
+
+Devcontainer Reference
+======================
+
+Reference pages for the CCCL devcontainer environment.
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   devcontainer_overview
+   launch_sh_reference
diff --git a/cccl_upstream/docs/infrastructure/devcontainer/references/launch_sh_reference.rst b/cccl_upstream/docs/infrastructure/devcontainer/references/launch_sh_reference.rst
new file mode 100644
index 00000000..cba3ea48
--- /dev/null
+++ b/cccl_upstream/docs/infrastructure/devcontainer/references/launch_sh_reference.rst
@@ -0,0 +1,84 @@
+.. _infra-devcontainer-launch-sh-reference:
+
+launch.sh reference
+===================
+
+``.devcontainer/launch.sh`` launches a development container for a chosen CUDA
+toolkit and host compiler. It selects a generated ``devcontainer.json``, mounts
+the repository, and starts either a VSCode dev container or a raw Docker shell.
+Linux-only.
+
+Flags
+-----
+
+.. list-table::
+   :header-rows: 1
+   :widths: 22 12 12 54
+
+   * - Flag
+     - Type
+     - Default
+     - Description
+   * - ``-c``, ``--cuda <VER>``
+     - string
+     - unset
+     - CUDA toolkit version, e.g. ``12.9``. Combines with ``--host`` to select a
+       container.
+   * - ``-H``, ``--host <COMPILER>``
+     - string
+     - unset
+     - Host compiler, e.g. ``gcc12``. Combines with ``--cuda`` to select a
+       container.
+   * - ``--cuda-ext``
+     - flag
+     - false
+     - Select the extended-CTK-libraries image. Adds the ``ext`` suffix to the
+       container name.
+   * - ``-d``, ``--docker``
+     - flag
+     - false
+     - Launch directly in Docker, bypassing VSCode.
+   * - ``--gpus <REQUEST>``
+     - string
+     - inferred
+     - GPU devices to attach, e.g. ``all``. Overrides the
+       ``hostRequirements.gpu`` value from ``devcontainer.json``.
+   * - ``-e``, ``--env <LIST>``
+     - list
+     - none
+     - Set additional container environment variables. Repeatable.
+   * - ``-v``, ``--volume <LIST>``
+     - list
+     - none
+     - Bind-mount an additional volume. Repeatable.
+   * - ``-h``, ``--help``
+     - flag
+     - —
+     - Print usage and exit.
+
+Arguments after ``--`` pass through to the container as the command to run.
+
+Container selection
+-------------------
+
+With no ``--cuda`` and no ``--host``, ``launch.sh`` uses the top-level
+``.devcontainer/devcontainer.json`` (the default environment).
+
+Set both ``--cuda`` and ``--host`` to select a generated container.
+With both set, ``launch.sh`` builds a directory name and loads
+``.devcontainer/<name>/devcontainer.json``:
+
+.. code-block:: text
+
+   cuda<VER>[ext]-<COMPILER>
+
+``<VER>`` comes from ``--cuda``, ``<COMPILER>`` from ``--host``, and the
+``ext`` suffix is added when ``--cuda-ext`` is set. Examples:
+
+.. code-block:: text
+
+   cuda12.9-gcc12         # --cuda 12.9 --host gcc12
+   cuda13.3ext-clang20    # --cuda 13.3 --host clang20 --cuda-ext
+
+If the resolved ``devcontainer.json`` does not exist, ``launch.sh`` reports the
+unknown combination and exits non-zero.
diff --git a/cccl_upstream/docs/infrastructure/index.rst b/cccl_upstream/docs/infrastructure/index.rst
new file mode 100644
index 00000000..9bce82fd
--- /dev/null
+++ b/cccl_upstream/docs/infrastructure/index.rst
@@ -0,0 +1,20 @@
+.. _infra-index:
+
+Infrastructure
+==============
+
+.. toctree::
+   :hidden:
+   :maxdepth: 2
+
+   ci/references/index
+   cmake/references/index
+   devcontainer/references/index
+
+Build, test, and CI infrastructure for CCCL contributors and maintainers.
+
+- :doc:`CI System <ci/references/index>`
+
+- :doc:`CMake Build System <cmake/references/index>`
+
+- :doc:`Devcontainer Environment <devcontainer/references/index>`
diff --git a/cccl_upstream/docs/libcudacxx/Doxyfile b/cccl_upstream/docs/libcudacxx/Doxyfile
new file mode 100644
index 00000000..8804bb26
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/Doxyfile
@@ -0,0 +1,182 @@
+# Doxyfile for libcudacxx
+
+PROJECT_NAME           = libcudacxx
+OUTPUT_DIRECTORY       = ../_build/doxygen/libcudacxx
+CREATE_SUBDIRS         = NO
+GENERATE_HTML          = NO
+GENERATE_LATEX         = NO
+GENERATE_XML           = YES
+XML_OUTPUT             = xml
+XML_PROGRAMLISTING     = YES
+
+INPUT                  = ../../libcudacxx/include/cuda/__algorithm \
+                         ../../libcudacxx/include/cuda/__container \
+                         ../../libcudacxx/include/cuda/__device \
+                         ../../libcudacxx/include/cuda/__event \
+                         ../../libcudacxx/include/cuda/__hierarchy \
+                         ../../libcudacxx/include/cuda/__iterator \
+                         ../../libcudacxx/include/cuda/__launch \
+                         ../../libcudacxx/include/cuda/__memory_pool \
+                         ../../libcudacxx/include/cuda/__memory_resource \
+                         ../../libcudacxx/include/cuda/__stream \
+                         ../../libcudacxx/include/nv \
+                         ../../libcudacxx/include/cuda/std/__cccl/architecture.h \
+                         ../../libcudacxx/include/cuda/std/__cccl/os.h
+
+RECURSIVE              = YES
+EXCLUDE_PATTERNS       = */detail/* */test/* */tests/* */examples/* */__detail/* */std/__cccl/compiler.h
+EXCLUDE_SYMBOLS        = *detail* *__* _LIBCUDACXX_*
+
+FILE_PATTERNS          = *.h *.hpp *.cuh
+EXTENSION_MAPPING      = cuh=C++ cu=C++
+
+# Documentation extraction settings
+EXTRACT_ALL            = YES
+EXTRACT_PRIVATE        = NO
+EXTRACT_STATIC         = YES
+EXTRACT_LOCAL_CLASSES  = YES
+HIDE_UNDOC_MEMBERS     = NO
+HIDE_UNDOC_CLASSES     = NO
+SHOW_INCLUDE_FILES     = YES
+INLINE_INHERITED_MEMB  = YES
+FULL_PATH_NAMES        = YES
+STRIP_FROM_PATH        = ../../libcudacxx
+SHORT_NAMES            = NO
+
+# Parsing settings
+JAVADOC_AUTOBRIEF      = YES
+QT_AUTOBRIEF           = NO
+MULTILINE_CPP_IS_BRIEF = NO
+INHERIT_DOCS           = YES
+SEPARATE_MEMBER_PAGES  = NO
+TAB_SIZE               = 4
+BUILTIN_STL_SUPPORT    = YES
+
+# Preprocessing
+ENABLE_PREPROCESSING   = YES
+MACRO_EXPANSION        = YES
+EXPAND_ONLY_PREDEF     = NO
+SEARCH_INCLUDES        = YES
+SKIP_FUNCTION_MACROS   = YES
+
+# Predefined macros for libcudacxx
+PREDEFINED = \
+    "__device__=" \
+    "__host__=" \
+    "__global__=" \
+    "__forceinline__=" \
+    "__declspec(x)=" \
+    "__align__(x)=" \
+    "_CCCL_DOXYGEN_INVOKED" \
+    "_CCCL_HOST_DEVICE=" \
+    "_CCCL_DEVICE=" \
+    "_CCCL_HOST=" \
+    "_CCCL_FORCEINLINE=" \
+    "_CCCL_CONSTEXPR_CXX14=constexpr" \
+    "_CCCL_CONSTEXPR_CXX17=constexpr" \
+    "_CCCL_CONSTEXPR_CXX20=constexpr" \
+    "_CCCL_CONSTEXPR_CXX23=constexpr" \
+    "_LIBCUDACXX_HIDE_FROM_ABI=" \
+    "_LIBCUDACXX_INLINE_VISIBILITY=" \
+    "_LIBCUDACXX_TEMPLATE_VIS=" \
+    "__cccl_lib_mdspan" \
+    "_CCCL_AND=&&" \
+    "_CCCL_API=inline" \
+    "_CCCL_HOST_DEVICE_API=inline" \
+    "_CCCL_BEGIN_NAMESPACE_ARCH_DEPENDENT="\
+    "_CCCL_BEGIN_NAMESPACE_CUDA=namespace cuda {" \
+    "_CCCL_BEGIN_NAMESPACE_CUDA_DEVICE=namespace cuda::device {" \
+    "_CCCL_BEGIN_NAMESPACE_CUDA_DEVICE_EXPERIMENTAL=namespace cuda::device::experimental {" \
+    "_CCCL_BEGIN_NAMESPACE_CUDA_DRIVER=namespace cuda::__driver {" \
+    "_CCCL_BEGIN_NAMESPACE_CUDA_EXECUTION=namespace cuda::execution {" \
+    "_CCCL_BEGIN_NAMESPACE_CUDA_MR=namespace cuda::mr {" \
+    "_CCCL_BEGIN_NAMESPACE_CUDA_PTX =namespace cuda::ptx {" \
+    "_CCCL_BEGIN_NAMESPACE_CUDA_STD=namespace cuda::std {" \
+    "_CCCL_BEGIN_NAMESPACE_CUDA_STD_EXECUTION=namespace cuda::std::execution {" \
+    "_CCCL_BEGIN_NAMESPACE_CUDA_STD_RANGES=namespace cuda::std::ranges {" \
+    "_CCCL_BEGIN_NAMESPACE_CUDA_STD_VIEWS=namespace cuda::std::ranges::views {" \
+    "_CCCL_BEGIN_NAMESPACE_STD=namespace std {" \
+    "_CCCL_CATCH=catch" \
+    "_CCCL_CATCH_ALL=catch (...)" \
+    "_CCCL_CATCH_FALLTHROUGH=" \
+    "_CCCL_CONCEPT=constexpr bool " \
+    "_CCCL_CONSTEXPR_FRIEND=friend " \
+    "_CCCL_CTK_AT_LEAST(x, y)=1" \
+    "_CCCL_CTK_BELOW(x, y)=0" \
+    "_CCCL_CUDACC_AT_LEAST(x, y)=1" \
+    "_CCCL_CUDACC_BELOW(x, y)=0" \
+    "_CCCL_DEVICE_API=inline" \
+    "_CCCL_DIAG_POP=" \
+    "_CCCL_DIAG_PUSH=" \
+    "_CCCL_DIAG_SUPPRESS_CLANG(x)=" \
+    "_CCCL_DIAG_SUPPRESS_GCC(x)=" \
+    "_CCCL_DIAG_SUPPRESS_MSVC(x)=" \
+    "_CCCL_DIAG_SUPPRESS_NVHPC(x)=" \
+    "_CCCL_END_NAMESPACE_ARCH_DEPENDENT="\
+    "_CCCL_END_NAMESPACE_CUDA=}" \
+    "_CCCL_END_NAMESPACE_CUDA_DEVICE=}" \
+    "_CCCL_END_NAMESPACE_CUDA_DEVICE_EXPERIMENTAL=}" \
+    "_CCCL_END_NAMESPACE_CUDA_DRIVER=}" \
+    "_CCCL_END_NAMESPACE_CUDA_EXECUTION=}" \
+    "_CCCL_END_NAMESPACE_CUDA_MR=}" \
+    "_CCCL_END_NAMESPACE_CUDA_PTX=}" \
+    "_CCCL_END_NAMESPACE_CUDA_STD=}" \
+    "_CCCL_END_NAMESPACE_CUDA_STD_EXECUTION=}" \
+    "_CCCL_END_NAMESPACE_CUDA_STD_RANGES=}" \
+    "_CCCL_END_NAMESPACE_CUDA_STD_VIEWS=}" \
+    "_CCCL_END_NAMESPACE_STD=}" \
+    "_CCCL_EXEC_CHECK_DISABLE=" \
+    "_CCCL_GLOBAL_CONSTANT=inline constexpr" \
+    "_CCCL_HAS_CTK()=1" \
+    "_CCCL_HIDE_FROM_ABI=" \
+    "_CCCL_HOST_API=inline" \
+    "_CCCL_PUBLIC_API=inline" \
+    "_CCCL_PUBLIC_DEVICE_API=inline" \
+    "_CCCL_PUBLIC_HOST_API=inline" \
+    "_CCCL_REQUIRES(x)= ::cuda::std::enable_if_t<x, int> = 0>" \
+    "_CCCL_STD_VER=2020" \
+    "_CCCL_SUPPRESS_DEPRECATED_POP=" \
+    "_CCCL_SUPPRESS_DEPRECATED_PUSH=" \
+    "_CCCL_TEMPLATE(x)=template<x, " \
+    "_CCCL_TRAILING_REQUIRES(x)=-> x requires " \
+    "_CCCL_NODEBUG_API=inline" \
+    "_CCCL_NODEBUG_DEVICE_API=inline" \
+    "_CCCL_NODEBUG_HOST_API=inline" \
+    "_CCCL_TRY=try" \
+    "_CCCL_TYPE_VISIBILITY_DEFAULT=" \
+    "_CCCL_TYPE_VISIBILITY_HIDDEN=" \
+    "_CCCL_VISIBILITY_DEFAULT=" \
+    "_CCCL_VISIBILITY_HIDDEN=" \
+    "_CCCL_LIFETIMEBOUND=" \
+    "_CUDAX_CONSTEXPR_FRIEND=friend" \
+    "_LIBCUDACXX_HAS_SPACESHIP_OPERATOR()=0" \
+    "CCCL_DEPRECATED=" \
+    "CCCL_DEPRECATED_BECAUSE(x)=" \
+    "CCCL_IGNORE_DEPRECATED_CPP_DIALECT" \
+    "CUB_DISABLE_NAMESPACE_MAGIC" \
+    "CUB_IGNORE_NAMESPACE_MAGIC_ERROR" \
+    "CUB_NAMESPACE_BEGIN=namespace cub {" \
+    "CUB_NAMESPACE_END=}" \
+    "CUB_RUNTIME_FUNCTION=" \
+    "THRUST_FWD(x)=x" \
+    "THRUST_NAMESPACE_BEGIN=namespace thrust {" \
+    "THRUST_NAMESPACE_END=}" \
+    "THRUST_PREVENT_MACRO_SUBSTITUTION" \
+    "_CCCL_HOSTED()=1"
+
+# IMPORTANT: Aliases for custom commands
+# The rst alias enables embedding reStructuredText in doxygen comments
+# Using the same format as repo_docs for compatibility
+ALIASES = "rst=\verbatim embed:rst:leading-asterisk"
+ALIASES += "endrst=\endverbatim"
+ALIASES += "rststar=\verbatim embed:rst:leading-asterisk"
+ALIASES += "inlinerst=\verbatim embed:rst:inline"
+
+# Quiet mode
+QUIET                  = YES
+WARNINGS               = YES
+WARN_AS_ERROR          = FAIL_ON_WARNINGS
+WARN_IF_UNDOCUMENTED   = NO
+WARN_IF_DOC_ERROR      = YES
+WARN_IF_INCOMPLETE_DOC = NO
+WARN_NO_PARAMDOC       = NO
diff --git a/cccl_upstream/docs/libcudacxx/extended_api.rst b/cccl_upstream/docs/libcudacxx/extended_api.rst
new file mode 100644
index 00000000..a2416b09
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api.rst
@@ -0,0 +1,31 @@
+.. _libcudacxx-extended-api:
+
+Extended API
+============
+
+.. toctree::
+   :maxdepth: 2
+
+   extended_api/macros
+   extended_api/bit
+   extended_api/execution_model
+   extended_api/exceptions
+   extended_api/memory_model
+   extended_api/thread_groups
+   extended_api/synchronization_primitives
+   extended_api/asynchronous_operations
+   extended_api/memory_access_properties
+   extended_api/functional
+   extended_api/iterators
+   extended_api/type_traits
+   extended_api/vector_tuple_protocol
+   extended_api/numeric
+   extended_api/random
+   extended_api/memory
+   extended_api/memory_resource
+   extended_api/math
+   extended_api/mdspan
+   extended_api/tma
+   extended_api/warp
+   extended_api/utility
+   extended_api/work_stealing
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/asynchronous_operations.rst b/cccl_upstream/docs/libcudacxx/extended_api/asynchronous_operations.rst
new file mode 100644
index 00000000..ef9731d6
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/asynchronous_operations.rst
@@ -0,0 +1,35 @@
+.. _libcudacxx-extended-api-asynchronous-operations:
+
+Asynchronous Operations
+-----------------------
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   asynchronous_operations/memcpy_async_tx
+   asynchronous_operations/memcpy_async
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`cuda::memcpy_async <libcudacxx-extended-api-asynchronous-operations-memcpy-async>`
+     - Asynchronously copies one range to another
+     - libcu++ 1.1.0 / CCCL 2.0.0
+     - CUDA 11.0
+
+   * - :ref:`cuda::memcpy_async_tx <libcudacxx-extended-api-asynchronous-operations-memcpy-async-tx>`
+     - Asynchronously copies one range to another with manual transaction accounting
+     - libcu++ 1.2.0 / CCCL 2.0.0
+     - CUDA 11.1
+
+.. note::
+
+  **Asynchronous operations** like `memcpy_async <libcudacxx-extended-api-asynchronous-operations-memcpy-async>`
+  are non-blocking operations performed as-if by a new thread of execution.
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/asynchronous_operations/memcpy_async.rst b/cccl_upstream/docs/libcudacxx/extended_api/asynchronous_operations/memcpy_async.rst
new file mode 100644
index 00000000..565e501e
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/asynchronous_operations/memcpy_async.rst
@@ -0,0 +1,278 @@
+.. _libcudacxx-extended-api-asynchronous-operations-memcpy-async:
+
+``cuda::memcpy_async``
+======================
+
+Defined in header ``<cuda/barrier>``:
+
+.. code:: cuda
+
+   // (1)
+   template <typename Shape, cuda::thread_scope Scope, typename CompletionFunction>
+   __host__ __device__
+   void cuda::memcpy_async(void* destination, void const* source, Shape size,
+                           cuda::barrier<Scope, CompletionFunction>& barrier);
+
+   // (2)
+   template <typename Group,
+             typename Shape, cuda::thread_scope Scope, typename CompletionFunction>
+   __host__ __device__
+   void cuda::memcpy_async(Group const& group,
+                           void* destination, void const* source, Shape size,
+                           cuda::barrier<Scope, CompletionFunction>& barrier);
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   // (3)
+   template <typename Shape, cuda::thread_scope Scope>
+   __host__ __device__
+   void cuda::memcpy_async(void* destination, void const* source, Shape size,
+                           cuda::pipeline<Scope>& pipeline);
+
+   // (4)
+   template <typename Group, typename Shape, cuda::thread_scope Scope>
+   __host__ __device__
+   void cuda::memcpy_async(Group const& group,
+                           void* destination, void const* source, Shape size,
+                           cuda::pipeline<Scope>& pipeline);
+
+Defined in header ``<cuda/annotated_ptr>``:
+
+.. code:: cuda
+
+   // (5)
+   template <typename Dst, typename Src, typename SrcProperty, typename Shape, typename Sync>
+   __host__ __device__
+   void memcpy_async(Dst* dst, cuda::annotated_ptr<Src, SrcProperty> src, Shape size, Sync& sync);
+
+   // (6)
+   template<typename Dst, typename DstProperty, typename Src, typename SrcProperty, typename Shape, typename Sync>
+   __host__ __device__
+   void memcpy_async(cuda::annotated_ptr<Dst, DstProperty> dst, cuda::annotated_ptr<Src, SrcProperty> src, Shape size, Sync& sync);
+
+   // (7)
+   template<typename Group, typename Dst, typename Src, typename SrcProperty, typename Shape, typename Sync>
+   __host__ __device__
+   void memcpy_async(Group const& group, Dst* dst, cuda::annotated_ptr<Src, SrcProperty> src, Shape size, Sync& sync);
+
+   // (8)
+   template<typename Group, typename Dst, typename DstProperty, typename Src, typename SrcProperty, typename Shape, typename Sync>
+   __host__ __device__
+   void memcpy_async(Group const& group, cuda::annotated_ptr<Dst, DstProperty> dst, cuda::annotated_ptr<Src, SrcProperty> src, Shape size, Sync& sync);
+
+``cuda::memcpy_async`` asynchronously copies ``size`` bytes from the
+memory location pointed to by ``source`` to the memory location pointed
+to by ``destination``. Both objects are reinterpreted as arrays of
+``unsigned char``.
+
+1. Non-group version. Binds the asynchronous copy completion to ``cuda::barrier`` and
+   issues the copy in the current thread.
+2. Group version. Binds the asynchronous copy completion to ``cuda::barrier`` and
+   cooperatively issues the copy across all threads in ``group``.
+3. Non-group version. Binds the asynchronous copy completion to ``cuda::pipeline`` and
+   issues the copy in the current thread.
+4. Group version. Binds the asynchronous copy completion to ``cuda::pipeline`` and
+   cooperatively issues the copy across all threads in ``group``.
+5. 5-8: convenience wrappers using ``cuda::annotated_ptr`` where
+   ``Sync`` is either ``cuda::barrier`` or ``cuda::pipeline``.
+
+Notes
+-----
+
+``cuda::memcpy_async`` have similar constraints to `std::memcpy <https://en.cppreference.com/w/cpp/string/byte/memcpy>`_,
+namely:
+
+   - If the objects overlap, the behavior is undefined.
+   - If either ``destination`` or ``source`` is an invalid or null pointer, the behavior is undefined
+     (even if ``count`` is zero).
+   - If the objects are `potentially-overlapping <https://en.cppreference.com/w/cpp/language/object#Subobjects>`_
+     the behavior is undefined.
+   - If the objects are not of `TriviallyCopyable <https://en.cppreference.com/w/cpp/named_req/TriviallyCopyable>`_
+     type the program is ill-formed, no diagnostic required.
+
+Additionally:
+
+   - If *Shape* is :ref:`cuda::aligned_size_t <libcudacxx-extended-api-memory-aligned-size>`, ``source``
+     and ``destination`` are both required to be aligned on ``cuda::aligned_size_t::align``, else the behavior is
+     undefined.
+   - If ``cuda::pipeline`` is in a *quitted state*
+     (see :ref:`cuda::pipeline::quit <libcudacxx-extended-api-synchronization-pipeline-pipeline-quit>`),
+     the behavior is undefined.
+   - For cooperative overloads (with a group parameter),
+     if the parameters are not the same across all threads in ``group``,
+     or not all threads represented by ``group`` call the overload, the behavior is undefined.
+   - The group of a cooperative overload can also represent a partition of the active threads calling the overload,
+     in which case a copy is cooperatively issued per partition of the active threads described by ``group``.
+     For example, if ``group`` is a ``cooperative_groups::thread_block_tile<32, ...>``
+     and the overload is called with 128 threads active, 4 copies will be issued, one cooperatively per warp.
+   - If a non-group overload is called with multiple threads active,
+     each thread issues its own copy and thus must have different arguments and the copies must not overlap.
+
+
+Template Parameters
+-------------------
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``Group``
+     - A type satisfying the [*Group*] concept.
+   * - ``Shape``
+     - Either `cuda::std::size_t <https://en.cppreference.com/w/c/types/size_t>`_
+       or :ref:`cuda::aligned_size_t <libcudacxx-extended-api-memory-aligned-size>`.
+
+Parameters
+----------
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``group``
+     - The group of threads.
+   * - ``destination``
+     - Pointer to the memory location to copy to.
+   * - ``source``
+     - Pointer to the memory location to copy from.
+   * - ``size``
+     - The number of bytes to copy.
+   * - ``barrier``
+     - The barrier object used to wait on the copy completion.
+   * - ``pipeline``
+     - The pipeline object used to wait on the copy completion.
+
+Related traits
+--------------
+
+.. code:: cuda
+
+   template <typename Group>
+   constexpr inline bool is_thread_block_group_v;
+
+This trait is ``true`` if ``Group`` represents the full CUDA thread block.
+For example, ``cooperative_groups::thread_block`` satisfies this trait.
+Users are encouraged to specialize this trait for their own groups.
+
+.. code:: cuda
+
+   template <typename Group>
+   constexpr inline bool is_warp_group_v = false;
+
+This trait is ``true`` if ``Group`` represents a full CUDA warp.
+For example, ``cooperative_groups::thread_block_tile<32, ...>`` satisfies this trait.
+Users are encouraged to specialize this trait for their own groups.
+
+Implementation notes
+--------------------
+
+On Hopper+ GPUs, the overloads taking a barrier may use the Tensor Memory Accelerator (TMA)
+via the ``cp.async.bulk`` instruction to perform the copy if:
+- the barrier resides in shared memory,
+- the data is aligned to 16 bytes,
+- the source is global memory,
+- the destination is shared memory.
+Additionally, the cooperative overload (taking a group) can generate more efficient code
+if the group satisfies the trait ``cuda::is_thread_block_group_v`` or ``cuda::is_warp_group_v``.
+In those cases, a uniform data path is generated for the bulk copy and thread peeling is avoided.
+
+On Ampere+ GPUs, the ``cp.async`` instruction may be used to perform the copy if:
+- the data is aligned to at least 4 bytes,
+- the source is global memory,
+- the destination is shared memory.
+
+
+Examples
+--------
+
+.. rubric:: Example: Using a system-wide barrier to copy within global memory
+
+.. code:: cuda
+
+   #include <cuda/barrier>
+
+   __global__ void example_kernel(char* dst, char* src) {
+     cuda::barrier<cuda::thread_scope_system> bar;
+     init(&bar, 1);
+
+     cuda::memcpy_async(dst,     src,      1, bar);
+     cuda::memcpy_async(dst + 1, src + 8,  1, bar);
+     cuda::memcpy_async(dst + 2, src + 16, 1, bar);
+     cuda::memcpy_async(dst + 3, src + 24, 1, bar);
+
+     bar.arrive_and_wait();
+   }
+
+`See it on Godbolt <https://godbolt.org/z/od6q9s8fq>`_
+
+.. rubric:: Example: 1D load of two buffers from global to shared memory with a barrier
+
+.. code:: cuda
+
+   #include <cuda/barrier>
+
+   __global__ void example_kernel(int* gmem1, double* gmem2) {
+     constexpr int tile_size = 1024;
+     __shared__ alignas(16)    int smem1[tile_size];
+     __shared__ alignas(16) double smem2[tile_size];
+
+     #pragma nv_diag_suppress static_var_with_dynamic_init
+     __shared__  cuda::barrier<cuda::thread_scope_block> bar;
+
+     // setup the barrier where each thread in the block arrives at
+     if (threadIdx.x == 0) {
+       init(&bar, blockDim.x);
+     }
+     __syncthreads();
+
+     // issue two copy operations
+     auto group = ...;
+     cuda::memcpy_async(group, smem1, gmem1, cuda::aligned_size_t<16>(tile_size * sizeof(int)   ), bar);
+     cuda::memcpy_async(group, smem2, gmem2, cuda::aligned_size_t<16>(tile_size * sizeof(double)), bar);
+
+     // arrive and wait for copy operations to complete
+     bar.arrive_and_wait();
+
+     // process data in smem ...
+   }
+
+There are multiple possibilities to initialize the ``group`` variable.
+One option is to use the cooperative groups API:
+
+.. code:: cuda
+
+    #include <cooperative_groups.h>
+    auto group = cooperative_groups::this_thread_block();
+
+Another option, especially if the dimensionality of the thread block is known, e.g. 1D,
+a custom group can be defined like:
+
+.. code:: cuda
+
+   struct this_thread_block_1D {
+     static constexpr cuda::thread_scope thread_scope = cuda::thread_scope_block;
+
+     __device__ void sync() const {
+       __syncthreads();
+     }
+
+     __device__ auto size() const {
+       return blockDim.x;
+     }
+
+     __device__ auto thread_rank() const {
+       return threadIdx.x;
+     }
+   };
+
+   template <>
+   inline constexpr bool cuda::is_thread_block_group_v<this_thread_block_1D> = true;
+
+Such a group will emit the least amount of code when used with ``cuda::memcpy_async``,
+since the ``thread_rank()`` is easily computed (because the block is 1D)
+and we declared the group as representing the whole thread block,
+which allows emit a uniform data path on Hopper+ GPUs in certain conditions.
+
+`See it on Godbolt <https://godbolt.org/z/aM9cbabcW>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/asynchronous_operations/memcpy_async_tx.rst b/cccl_upstream/docs/libcudacxx/extended_api/asynchronous_operations/memcpy_async_tx.rst
new file mode 100644
index 00000000..5cbb4d96
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/asynchronous_operations/memcpy_async_tx.rst
@@ -0,0 +1,93 @@
+.. _libcudacxx-extended-api-asynchronous-operations-memcpy-async-tx:
+
+``cuda::device::memcpy_async_tx``
+=================================
+
+Defined in header ``<cuda/barrier>``:
+
+.. code:: cuda
+
+   template <typename T, size_t Alignment>
+   inline __device__
+   void cuda::device::memcpy_async_tx(
+     T* dest,
+     const T* src,
+     cuda::aligned_size_t<Alignment> size,
+     cuda::barrier<cuda::thread_scope_block>& bar);
+
+Copies ``size`` bytes from global memory ``src`` to shared memory ``dest`` and decrements the transaction count of ``bar`` by ``size`` bytes.
+
+Preconditions
+-------------
+
+-  ``src``, ``dest`` are 16-byte aligned and ``size`` is a multiple of 16, i.e., ``Alignment >= 16``.
+-  ``dest`` points to a shared memory allocation that is at least ``size`` bytes wide.
+-  ``src`` points to a global memory allocation that is at least ``size`` bytes wide.
+-  ``bar`` is located in shared memory
+-  If either ``destination`` or ``source`` is an invalid or null pointer, the behavior is undefined (even if ``count`` is zero).
+
+Requires
+--------
+
+-  ``is_trivially_copyable_v<T>`` is true.
+
+Notes
+-----
+
+This function can only be used under CUDA Compute Capability 9.0 (Hopper) or higher.
+
+There is no feature flag to check if ``cuda::device::memcpy_async_tx`` is available.
+
+**Comparison to cuda::memcpy_async**: ``memcpy_async_tx`` supports a subset of the operations of ``memcpy_async``.
+It gives more control over the synchronization with a barrier than ``memcpy_async``.
+Currently, ``memcpy_async_tx`` has no synchronous fallback mechanism., i.e., it currently does not work on older hardware
+(pre-CUDA Compute Capability 9.0, i.e., Hopper).
+
+.. _libcudacxx-extended-api-asynchronous-operations-memcpy-async-tx-example:
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/barrier>
+   #include <cuda/std/utility> // cuda::std::move
+
+   #if defined(__CUDA_MINIMUM_ARCH__) && __CUDA_MINIMUM_ARCH__ < 900
+   static_assert(false, "Insufficient CUDA Compute Capability: cuda::device::memcpy_async_tx is not available.");
+   #endif // __CUDA_MINIMUM_ARCH__
+
+   __device__ alignas(16) int gmem_x[2048];
+
+   __device__ inline bool elect_one() {
+     const unsigned int tid = threadIdx.x;
+     const unsigned int warp_id = tid / 32;
+     const unsigned int uniform_warp_id = __shfl_sync(0xFFFFFFFF, warp_id, 0); // broadcast from lane 0
+     return (uniform_warp_id == 0 && cuda::ptx::elect_sync(0xFFFFFFFF)); // elect a leader thread among warp 0
+   }
+
+   __global__ void example_kernel() {
+     alignas(16) __shared__ int smem_x[1024];
+     #pragma nv_diag_suppress static_var_with_dynamic_init
+     __shared__ cuda::barrier<cuda::thread_scope_block> bar;
+
+     // setup the mbarrier
+     if (threadIdx.x == 0) {
+       init(&bar, blockDim.x);
+     }
+     __syncthreads();
+
+     // issue the async copy from a single thread and wait for completion
+     const bool is_block_leader = elect_one();
+     const int tx_count = is_block_leader ? sizeof(smem_x) : 0;
+     if (is_block_leader) {
+       cuda::device::memcpy_async_tx(smem_x, gmem_x, cuda::aligned_size_t<16>(tx_count), bar);
+     }
+     auto token = cuda::device::barrier_arrive_tx(bar, 1, tx_count);
+     bar.wait(cuda::std::move(token));
+
+     // smem_x contains the contents of gmem_x[0], ..., gmem_x[1023]
+     smem_x[threadIdx.x] += 1;
+   }
+
+`See it on Godbolt <https://godbolt.org/z/M8zqnrz9b>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/bit.rst b/cccl_upstream/docs/libcudacxx/extended_api/bit.rst
new file mode 100644
index 00000000..521a663c
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/bit.rst
@@ -0,0 +1,42 @@
+.. _libcudacxx-extended-api-bit:
+
+Bit
+===
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   bit/bitmask
+   bit/bit_reverse
+   bit/bitfield_insert
+   bit/bitfield_extract
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`bitmask <libcudacxx-extended-api-bit-bitmask>`
+     - Generate a bitmask
+     - CCCL 3.0.0
+     - CUDA 13.0
+
+   * - :ref:`bit_reverse <libcudacxx-extended-api-bit-bit_reverse>`
+     - Reverse the order of bits
+     - CCCL 3.0.0
+     - CUDA 13.0
+
+   * - :ref:`bitfield_insert <libcudacxx-extended-api-bit-bitfield_insert>`
+     - Insert a bitfield
+     - CCCL 3.0.0
+     - CUDA 13.0
+
+   * - :ref:`bitfield_extract <libcudacxx-extended-api-bit-bitfield_extract>`
+     - Extract a bitfield
+     - CCCL 3.0.0
+     - CUDA 13.0
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/bit/bit_reverse.rst b/cccl_upstream/docs/libcudacxx/extended_api/bit/bit_reverse.rst
new file mode 100644
index 00000000..25ee7985
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/bit/bit_reverse.rst
@@ -0,0 +1,72 @@
+.. _libcudacxx-extended-api-bit-bit_reverse:
+
+``cuda::bit_reverse``
+=====================
+
+Defined in the ``<cuda/bit>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T>
+   [[nodiscard]] __host__ __device__ constexpr
+   T bit_reverse(T value) noexcept;
+
+   } // namespace cuda
+
+The function reverses the order of bits in a value.
+
+**Parameters**
+
+- ``value``: Input value
+
+**Return value**
+
+- Value with reversed bits
+
+**Constraints**
+
+- ``T`` is an unsigned integer type.
+
+**Performance considerations**
+
+The function performs the following operations:
+
+- Device:
+
+  - ``uint8_t`` ``uint16_t``: ``PRMT``, ``BREV``
+  - ``uint32_t``: ``BREV``
+  - ``uint64_t``: ``BREV`` x2, ``MOV`` x2
+  - ``uint128_t``: ``BREV`` x4, ``MOV`` x4
+
+- Host: ``__builtin_bitreverse<N>`` with clang
+
+.. note::
+
+    Using the function could not be optimal when the input is a run-time value that the compiler can resolve at compile-time, e.g. an index of a loop with a fixed number of iterations.
+
+.. note::
+
+    GCC <= 8 uses a slow path with more instructions even in device code.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/bit>
+    #include <cuda/std/cassert>
+
+    __global__ void bit_reverse_kernel() {
+        assert(cuda::bit_reverse(0x0000FFFFu) == 0xFFFF0000u);
+        assert(cuda::bit_reverse(uint8_t{0b00001011}) == uint8_t{0b11010000});
+    }
+
+    int main() {
+        bit_reverse_kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/nW6qe5fT4>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/bit/bitfield_extract.rst b/cccl_upstream/docs/libcudacxx/extended_api/bit/bitfield_extract.rst
new file mode 100644
index 00000000..f7e482b6
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/bit/bitfield_extract.rst
@@ -0,0 +1,75 @@
+.. _libcudacxx-extended-api-bit-bitfield_extract:
+
+``cuda::bitfield_extract``
+==========================
+
+Defined in the ``<cuda/bit>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T>
+   [[nodiscard]] __host__ __device__ constexpr
+   T bitfield_extract(T value, int start, int width) noexcept;
+
+   } // namespace cuda
+
+The function extracts a bitfield from a value and returns it in the lower bits.
+``bitfield_extract()`` computes ``(value >> start) & mask``, where ``mask`` is a sequence of bits of width ``width``.
+
+**Parameters**
+
+- ``value``: The value to apply the bitfield.
+- ``start``:  Initial position of the bitfield.
+- ``width``:  Width of the bitfield.
+
+**Return value**
+
+- ``(value >> start) & mask``, where ``mask`` is a bitmask of width ``width``.
+
+**Constraints**
+
+- ``T`` is an unsigned integer type.
+
+**Preconditions**
+
+- ``start >= 0 && start <= num_bits(T)``.
+- ``width >= 0 && width <= num_bits(T)``.
+- ``start + width <= num_bits(T)``.
+
+**Performance considerations**
+
+The function performs the following operations in CUDA for ``uint8_t``, ``uint16_t``, ``uint32_t``:
+
+- ``SM < 70``: ``BFE``.
+- ``SM >= 70``: ``BMSK``, bitwise operation x2.
+
+.. note::
+
+    Using the function could not be optimal when the input is a run-time value that the compiler can resolve at compile-time, e.g. an index of a loop with a fixed number of iterations.
+
+.. note::
+
+    GCC <= 8 uses a slow path with more instructions even in CUDA.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/bit>
+    #include <cuda/std/cassert>
+
+    __global__ void bitfield_extract_kernel() {
+        assert(cuda::bitfield_extract(~0u, 0, 4) == 0b1111);
+        assert(cuda::bitfield_extract(0b1011000u, 3, 4) == 0b1011);
+    }
+
+    int main() {
+        bitfield_extract_kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/ofGnbsxW5>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/bit/bitfield_insert.rst b/cccl_upstream/docs/libcudacxx/extended_api/bit/bitfield_insert.rst
new file mode 100644
index 00000000..eb0fea8b
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/bit/bitfield_insert.rst
@@ -0,0 +1,76 @@
+.. _libcudacxx-extended-api-bit-bitfield_insert:
+
+``cuda::bitfield_insert``
+=========================
+
+Defined in the ``<cuda/bit>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T>
+   [[nodiscard]] __host__ __device__ constexpr
+   T bitfield_insert(T dest, T source, int start, int width) noexcept;
+
+   } // namespace cuda
+
+The function extracts the lower bitfield of size ``width`` from ``source`` and inserts it into ``dest`` at position ``start``.
+
+**Parameters**
+
+- ``dest``:   The value to insert the bitfield into.
+- ``source``: The value from which extract the bitfield.
+- ``start``:  Initial position of the bitfield.
+- ``width``:  Width of the bitfield.
+
+**Return value**
+
+- ``((source << start) & mask) | (dest & ~mask)``, where ``mask`` is a bitmask of width ``width`` at position ``start``.
+
+**Constraints**
+
+- ``T`` is an unsigned integer type.
+
+**Preconditions**
+
+- ``start >= 0 && start <= num_bits(T)``.
+- ``width >= 0 && width <= num_bits(T)``.
+- ``start + width <= num_bits(T)``.
+
+**Performance considerations**
+
+The function performs the following operations in CUDA for ``uint8_t``, ``uint16_t``, ``uint32_t``:
+
+- ``SM < 70``: ``BFI``.
+- ``SM >= 70``: ``BMSK``, bitwise operation x5.
+
+.. note::
+
+    When the input values are run-time values that the compiler can resolve at compile-time, e.g. an index of a loop with a fixed number of iterations, using the function could not be optimal.
+
+.. note::
+
+    GCC <= 8 uses a slow path with more instructions even in CUDA.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/bit>
+    #include <cuda/std/cassert>
+
+    __global__ void bitfield_insert_kernel() {
+        assert(cuda::bitfield_insert(0u, 0xFFFFu, 0, 4) == 0b1111);
+        assert(cuda::bitfield_insert(0u, 0xFFFFu, 3, 4) == 0b1111000);
+        assert(cuda::bitfield_insert(1u, 0xFFFFu, 3, 4) == 0b1111001);
+    }
+
+    int main() {
+        bitfield_insert_kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/4Thzz516M>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/bit/bitmask.rst b/cccl_upstream/docs/libcudacxx/extended_api/bit/bitmask.rst
new file mode 100644
index 00000000..0d5da1bd
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/bit/bitmask.rst
@@ -0,0 +1,74 @@
+.. _libcudacxx-extended-api-bit-bitmask:
+
+``cuda::bitmask``
+=================
+
+Defined in the ``<cuda/bit>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T = uint32_t>
+   [[nodiscard]] __host__ __device__ constexpr
+   T bitmask(int start, int width) noexcept;
+
+   } // namespace cuda
+
+The function generates a bitmask of size ``width`` starting at position ``start``.
+
+**Parameters**
+
+- ``start``: starting position of the bitmask.
+- ``width``: width of the bitmask.
+
+**Return value**
+
+- Bitmask of size ``width`` starting at position``start``.
+
+**Constraints**
+
+- ``T`` is an unsigned integral type.
+
+**Preconditions**
+
+- ``start >= 0 && start <= num_bits(T)``.
+- ``width >= 0 && width <= num_bits(T)``.
+- ``start + width <= num_bits(T)``.
+
+**Performance considerations**
+
+The function performs the following operations in device code:
+
+- ``uint8_t``, ``uint16_t``, ``uint32_t``: ``BMSK``.
+- ``uint64_t``: ``SHL`` x4, ``ADD`` x2.
+
+.. note::
+
+    When the input values are run-time values that the compiler can resolve at compile-time, e.g. an index of a loop with a fixed number of iterations, using the function could not be optimal.
+
+.. note::
+
+    GCC <= 8 uses a slow path with more instructions even in CUDA.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/bit>
+    #include <cuda/std/cassert>
+    #include <cuda/std/cstdint>
+
+    __global__ void bitmask_kernel() {
+        assert(cuda::bitmask(2, 4) == 0b111100u);
+        assert(cuda::bitmask<uint64_t>(1, 3) == uint64_t{0b1110});
+    }
+
+    int main() {
+        bitmask_kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/habGohz7T>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/exceptions.rst b/cccl_upstream/docs/libcudacxx/extended_api/exceptions.rst
new file mode 100644
index 00000000..cb6f501e
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/exceptions.rst
@@ -0,0 +1,34 @@
+.. _libcudacxx-extended-api-exceptions:
+
+Exception Handling
+==================
+
+Standard C++ exception handling (``try``, ``catch``, ``throw``) is not supported in CUDA device code, while it is enabled by default in host code.
+
+**Device code**
+
+``libcu++`` maps exceptions to ``cuda::std::terminate()`` calls in device code, which translates to ``__trap()`` and terminates the kernel.
+
+**Host code**
+
+``libcu++`` allows users to manually disable exceptions in host code in two ways:
+
+- By defining ``CCCL_DISABLE_EXCEPTIONS`` before including any library headers.
+- By compiling with ``-fno-exceptions`` compiler flag with ``gcc`` or ``clang``, or ``/EH-`` compiler flag with ``msvc``.
+
+If exceptions are disabled, a ``throw`` exception is translated into a `cuda::std::terminate() <https://en.cppreference.com/w/cpp/error/terminate.html>`__ call, which terminates the program.
+
+``cuda::cuda_error``
+--------------------
+
+Exception class thrown when a CUDA error is encountered. It inherits from ``std::runtime_error``.
+
+.. code-block:: cpp
+
+    class cuda_error : public std::runtime_error
+    {
+    public:
+        cuda_error(cudaError_t status, const char* msg);
+
+        cudaError_t status() const noexcept;
+    };
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/execution_model.rst b/cccl_upstream/docs/libcudacxx/extended_api/execution_model.rst
new file mode 100644
index 00000000..4b576dfb
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/execution_model.rst
@@ -0,0 +1,296 @@
+.. _libcudacxx-extended-api-execution-model:
+
+Execution model
+===============
+
+CUDA C++ aims to provide `parallel forward progress [intro.progress.9] <https://eel.is/c++draft/intro.progress#9>`__
+for all device threads of execution, facilitating the parallelization of pre-existing C++ applications with CUDA C++.
+
+.. dropdown:: `[intro.progress] <https://eel.is/c++draft/intro.progress>`__
+
+    - `[intro.progress.7] <https://eel.is/c++draft/intro.progress#7>`__: For a thread of execution
+      providing `concurrent forward progress guarantees <https://eel.is/c++draft/intro.progress#def:concurrent_forward_progress_guarantees>`__,
+      the implementation ensures that the thread will eventually make progress for as long as it has not terminated.
+
+      [Note 5: This applies regardless of whether or not other threads of execution (if any) have been or are making progress.
+      To eventually fulfill this requirement means that this will happen in an unspecified but finite amount of time. — end note]
+
+    - `[intro.progress.9] <https://eel.is/c++draft/intro.progress>`__: For a thread of execution providing
+      `parallel forward progress guarantees <https://eel.is/c++draft/intro.progress#9>`__, the implementation is not required to ensure that
+      the thread will eventually make progress if it has not yet executed any execution step; once this thread has executed a step,
+      it provides concurrent forward progress guarantees.
+
+        [Note 6: This does not specify a requirement for when to start this thread of execution, which will typically be specified by the entity
+        that creates this thread of execution. For example, a thread of execution that provides concurrent forward progress guarantees and executes
+        tasks from a set of tasks in an arbitrary order, one after the other, satisfies the requirements of parallel forward progress for these
+        tasks. — end note]
+
+
+The CUDA C++ Programming Language is an extension of the C++ Programming Language.
+This section documents the modifications and extensions to the `[intro.progress] <https://eel.is/c++draft/intro.progress>`__ section of the current `ISO International Standard ISO/IEC 14882 – Programming Language C++ <https://eel.is/c++draft/>`__ draft.
+Modified sections are called out explicitly and their diff is shown in **bold**.
+All other sections are additions.
+
+.. _libcudacxx-extended-api-execution-model-host-threads:
+
+Host threads
+------------
+
+The forward progress provided by threads of execution created by the host implementation to
+execute `main <https://en.cppreference.com/w/cpp/language/main_function>`__, `std::thread <https://en.cppreference.com/w/cpp/thread/thread>`__,
+and `std::jthread <https://en.cppreference.com/w/cpp/thread/jthread>`__ is implementation-defined behavior of the host
+implementation `[intro.progress] <https://eel.is/c++draft/intro.progress>`__.
+General-purpose host implementations should provide concurrent forward progress.
+
+If the host implementation provides `concurrent forward progress [intro.progress.7] <https://eel.is/c++draft/intro.progress#7>`__,
+then CUDA C++ provides `parallel forward progress [intro.progress.9] <https://eel.is/c++draft/intro.progress#9>`__ for device threads.
+
+
+.. _libcudacxx-extended-api-execution-model-device-threads:
+
+Device threads
+--------------
+
+Once a device thread makes progress:
+
+- If it is part of a `Cooperative Grid <https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__EXECUTION.html#group__CUDART__EXECUTION_1g504b94170f83285c71031be6d5d15f73>`__,
+  all device threads in its grid shall eventually make progress.
+- Otherwise, all device threads in its `thread-block cluster <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#thread-block-clusters>`__
+  shall eventually make progress.
+
+    [Note: Threads in other thread-block clusters are not guaranteed to eventually make progress. - end note.]
+
+    [Note: This implies that all device threads within its thread block shall eventually make progress. - end note.]
+
+
+Modify `[intro.progress.1] <https://eel.is/c++draft/intro.progress>`__ as follows (modifications in **bold**):
+
+The implementation may assume that any **host** thread will eventually do one of the following:
+
+    1. terminate,
+    2. invoke the function `std::this_thread::yield <https://en.cppreference.com/w/cpp/thread/yield>`__ (`[thread.thread.this] <http://eel.is/c++draft/thread.thread.this>`__),
+    3. make a call to a library I/O function,
+    4. perform an access through a volatile glvalue,
+    5. perform a synchronization operation or an atomic operation, or
+    6. continue execution of a trivial infinite loop (`[stmt.iter.general] <http://eel.is/c++draft/stmt.iter.general>`__).
+
+**The implementation may assume that any device thread will eventually do one of the following:**
+
+    1. **terminate**,
+    2. **make a call to a library I/O function**,
+    3. **perform an access through a volatile glvalue except if the designated object has automatic storage duration, or**
+    4. **perform a synchronization operation or an atomic read operation except if the designated object has automatic storage duration.**
+
+    [Note: Some current limitations of device threads relative to host threads
+    are implementation defects known to us, that we may fix over time.
+    Examples include the undefined behavior that arises from device threads
+    that eventually only perform volatile or atomic operations
+    on automatic storage duration objects.
+    However, other limitations of device threads relative to host threads
+    are intentional choices.  They enable performance optimizations
+    that would not be possible if device threads followed the C++ Standard strictly.
+    For example, providing forward progress to programs
+    that eventually only perform atomic writes or fences
+    would degrade overall performance for little practical benefit. - end note.]
+
+.. dropdown:: Examples of forward progress guarantee differences between host and device threads due to modifications to `[intro.progress.1] <https://eel.is/c++draft/intro.progress#1>`__.
+
+    The following examples refer to the itemized sub-clauses of the implementation assumptions for host and device threads above
+    using "host.threads.<id>" and "device.threads.<id>", respectively.
+
+    .. code-block:: cuda
+        :linenos:
+
+        // Example: Execution.Model.Device.0
+        // Outcome: grid eventually terminates per device.threads.4 because the atomic object does not have automatic storage duration.
+        __global__ void ex0(cuda::atomic_ref<int, cuda::thread_scope_device> atom) {
+            if (threadIdx.x == 0) {
+                while(atom.load(cuda::memory_order_relaxed) == 0);
+            } else if (threadIdx.x == 1) {
+                atom.store(1, cuda::memory_order_relaxed);
+            }
+        }
+
+    .. code-block:: cuda
+        :linenos:
+
+        // Example: Execution.Model.Device.1
+        // Allowed outcome: No thread makes progress because device threads don't support host.threads.2.
+        __global__ void ex1() {
+            while(true) cuda::std::this_thread::yield();
+        }
+
+    .. code-block:: cuda
+        :linenos:
+
+        // Example: Execution.Model.Device.2
+        // Allowed outcome: No thread makes progress because device threads don't support host.threads.4
+        // for objects with automatic storage duration (see exception in device.threads.3).
+        __global__ void ex2() {
+            volatile bool True = true;
+            while(True);
+        }
+
+    .. code-block:: cuda
+        :linenos:
+
+        // Example: Execution.Model.Device.3
+        // Allowed outcome: No thread makes progress because device threads don't support host.threads.5
+        // for objects with automatic storage duration (see exception in device.threads.4).
+        __global__ void ex3() {
+            cuda::atomic<bool, cuda::thread_scope_thread> True = true;
+            while(True.load());
+        }
+
+    .. code-block:: cuda
+        :linenos:
+
+        // Example: Execution.Model.Device.4
+        // Allowed outcome: No thread makes progress because device threads don't support host.thread.6.
+        __global void ex4() {
+            while(true) { /* empty */ }
+        }
+
+.. _libcudacxx-extended-api-execution-model-cuda-apis:
+
+CUDA APIs
+---------
+
+A CUDA API call shall eventually either return or ensure at least one device thread makes progress.
+
+CUDA query functions (e.g. `cudaStreamQuery <https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__STREAM.html#group__CUDART__STREAM_1g2021adeb17905c7ec2a3c1bf125c5435>`__,
+`cudaEventQuery <https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__EVENT.html#group__CUDART__EVENT_1g2bf738909b4a059023537eaa29d8a5b7>`__, etc.) shall not consistently
+return ``cudaErrorNotReady`` without a device thread making progress.
+
+  [Note: The device thread need not be "related" to the API call, e.g., an API operating on one stream or process may ensure progress of a device thread on another stream or process. - end note.]
+
+  [Note: A simple but not sufficient method to test a program for CUDA API Forward Progress conformance is to run them with following environment variables set: ``CUDA_DEVICE_MAX_CONNECTIONS=1 CUDA_LAUNCH_BLOCKING=1``, and then check that the program still terminates.
+  If it does not, the program has a bug.
+  This method is not sufficient because it does not catch all Forward Progress bugs, but it does catch many such bugs. - end note.]
+
+.. dropdown:: Examples of CUDA API forward progress guarantees.
+
+    .. code-block:: cuda
+        :linenos:
+
+        // Example: Execution.Model.API.1
+        // Outcome: if no other device threads (e.g., from other processes) are making progress,
+        // this program terminates and returns cudaSuccess.
+        // Rationale: CUDA guarantees that if the device is empty:
+        // - `cudaDeviceSynchronize` eventually ensures that at least one device-thread makes progress, which implies that eventually `hello_world` grid and one of its device-threads start.
+        // - All thread-block threads eventually start (due to "if a device thread makes progress, all other threads in its thread-block cluster eventually make progress").
+        // - Once all threads in thread-block arrive at `__syncthreads` barrier, all waiting threads are unblocked.
+        // - Therefore all device threads eventually exit the `hello_world`` grid.
+        // - And `cudaDeviceSynchronize`` eventually unblocks.
+        __global__ void hello_world() { __syncthreads(); }
+        int main() {
+            hello_world<<<1,2>>>();
+            return (int)cudaDeviceSynchronize();
+        }
+
+    .. code-block:: cuda
+        :linenos:
+
+        // Example: Execution.Model.API.2
+        // Allowed outcome: eventually, no thread makes progress.
+        // Rationale: the `cudaDeviceSynchronize` API below is only called if a device thread eventually makes progress and sets the flag.
+        // However, CUDA only guarantees that `producer` device thread eventually starts if the synchronization API is called.
+        // Therefore, the host thread may never be unblocked from the flag spin-loop.
+        cuda::atomic<int, cuda::thread_scope_system> flag = 0;
+        __global__ void producer() { flag.store(1); }
+        int main() {
+            cudaHostRegister(&flag, sizeof(flag));
+            producer<<<1,1>>>();
+            while (flag.load() == 0);
+            return cudaDeviceSynchronize();
+        }
+
+    .. code-block:: cuda
+        :linenos:
+
+        // Example: Execution.Model.API.3
+        // Allowed outcome: eventually, no thread makes progress.
+        // Rationale: same as Example.Model.API.2, with the addition that a single CUDA query API call does not guarantee
+        // the device thread eventually starts, only repeated CUDA query API calls do (see Execution.Model.API.4).
+        cuda::atomic<int, cuda::thread_scope_system> flag = 0;
+        __global__ void producer() { flag.store(1); }
+        int main() {
+            cudaHostRegister(&flag, sizeof(flag));
+            producer<<<1,1>>>();
+            (void)cudaStreamQuery(0);
+            while (flag.load() == 0);
+            return cudaDeviceSynchronize();
+        }
+
+    .. code-block:: cuda
+        :linenos:
+
+        // Example: Execution.Model.API.4
+        // Outcome: terminates.
+        // Rationale: same as Execution.Model.API.3, but this example repeatedly calls
+        // a CUDA query API in within the flag spin-loop, which guarantees that the device thread
+        // eventually makes progress.
+        cuda::atomic<int, cuda::thread_scope_system> flag = 0;
+        __global__ void producer() { flag.store(1); }
+        int main() {
+            cudaHostRegister(&flag, sizeof(flag));
+            producer<<<1,1>>>();
+            while (flag.load() == 0) {
+                (void)cudaStreamQuery(0);
+            }
+            return cudaDeviceSynchronize();
+        }
+
+.. _libcudacxx-extended-api-execution-model-cuda-dependencies:
+
+Dependencies
+~~~~~~~~~~~~
+
+A device thread shall not start until all its dependencies have completed.
+
+  [Note: Dependencies that prevent device threads from starting to make progress can be created, for example, via `CUDA Stream Commands <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#streams>`__ .
+  These may include dependencies on the completion of, among others, `CUDA Events <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#events>`__ and `CUDA Kernels <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#kernels>`__ . - end note.]
+
+.. dropdown:: Examples of CUDA API forward progress guarantees due to dependencies
+
+    .. code-block:: cuda
+        :linenos:
+
+        // Example: Execution.Model.Stream.0
+        // Allowed outcome: eventually, no thread makes progress.
+        // Rationale: while CUDA guarantees that one device thread makes progress, since there
+        // is no dependency between `first` and `second`, it does not guarantee which thread,
+        // and therefore it could always pick the device thread from `second`, which then never
+        // unblocks from the spin-loop.
+        // That is, `second` may starve `first`.
+        cuda::atomic<int, cuda::thread_scope_system> flag = 0;
+        __global__ void first() { flag.store(1, cuda::memory_order_relaxed); }
+        __global__ void second() { while(flag.load(cuda::memory_order_relaxed) == 0) {} }
+        int main() {
+            cudaHostRegister(&flag, sizeof(flag));
+            cudaStream_t s0, s1;
+            cudaStreamCreate(&s0);
+            cudaStreamCreate(&s1);
+            first<<<1,1,0,s0>>>();
+            second<<<1,1,0,s1>>>();
+            return cudaDeviceSynchronize();
+        }
+
+    .. code-block:: cuda
+        :linenos:
+
+        // Example: Execution.Model.Stream.1
+        // Outcome: terminates.
+        // Rationale: same as Execution.Model.Stream.0, but this example has a stream dependency
+        // between first and second, which requires CUDA to run the grids in order.
+        cuda::atomic<int, cuda::thread_scope_system> flag = 0;
+        __global__ void first() { flag.store(1, cuda::memory_order_relaxed); }
+        __global__ void second() { while(flag.load(cuda::memory_order_relaxed) == 0) {} }
+        int main() {
+            cudaHostRegister(&flag, sizeof(flag));
+            cudaStream_t s0;
+            cudaStreamCreate(&s0);
+            first<<<1,1,0,s0>>>();
+            second<<<1,1,0,s0>>>();
+            return cudaDeviceSynchronize();
+        }
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/functional.rst b/cccl_upstream/docs/libcudacxx/extended_api/functional.rst
new file mode 100644
index 00000000..9a3726b8
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/functional.rst
@@ -0,0 +1,62 @@
+.. _libcudacxx-extended-api-functional:
+
+Functional
+----------
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   functional/always_true_false
+   functional/proclaim_return_type
+   functional/maximum_minimum
+   functional/operator_properties
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`cuda::always_true <libcudacxx-extended-api-functional-always-true-false>`
+     - Function object that always returns ``true``
+     - CCCL 3.4.0
+     - CUDA 13.4
+
+   * - :ref:`cuda::always_false <libcudacxx-extended-api-functional-always-true-false>`
+     - Function object that always returns ``false``
+     - CCCL 3.4.0
+     - CUDA 13.4
+
+   * - :ref:`cuda::maximum <libcudacxx-extended-api-functional-maximum-minimum>`
+     - Returns the maximum of two values
+     - CCCL 2.8.0
+     - CUDA 12.9
+
+   * - :ref:`cuda::minimum <libcudacxx-extended-api-functional-maximum-minimum>`
+     - Returns the minimum of two values
+     - CCCL 2.8.0
+     - CUDA 12.9
+
+   * - :ref:`cuda::proclaim_return_type <libcudacxx-extended-api-functional-proclaim-return-type>`
+     - Creates a forwarding call wrapper that proclaims return type
+     - libcu++ 1.9.0 / CCCL 2.0.0
+     - CUDA 11.8
+
+   * - ``cuda::proclaim_copyable_arguments``
+     - Creates a forwarding call wrapper that proclaims that arguments can be freely copied before an invocation of the wrapped callable
+     - CCCL 2.8.0
+     - CUDA 12.9
+
+   * - :ref:`cuda::get_device_address <libcudacxx-extended-api-memory-get-device-address>`
+     - Returns a valid address to a device object
+     - CCCL 2.8.0
+     - CUDA 12.9
+
+   * - :ref:`cuda::is_associative_v <libcudacxx-extended-api-functional-operator-properties>`, :ref:`cuda::is_commutative_v <libcudacxx-extended-api-functional-operator-properties>`, :ref:`cuda::identity_element() <libcudacxx-extended-api-functional-operator-properties>`, :ref:`cuda::absorbing_element() <libcudacxx-extended-api-functional-operator-properties>`
+     - Determines if an operator is associative for a type
+     - CCCL 3.3.0
+     - CUDA 13.3
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/functional/always_true_false.rst b/cccl_upstream/docs/libcudacxx/extended_api/functional/always_true_false.rst
new file mode 100644
index 00000000..cff10540
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/functional/always_true_false.rst
@@ -0,0 +1,44 @@
+.. _libcudacxx-extended-api-functional-always-true-false:
+
+``cuda::always_true`` and ``cuda::always_false``
+================================================
+
+Defined in the header ``<cuda/functional>``.
+
+.. code:: cuda
+
+    struct always_true {
+        template <typename... Ts>
+        [[nodiscard]] __host__ __device__ constexpr bool operator()(Ts&&...) const noexcept;
+    };
+
+    struct always_false {
+        template <typename... Ts>
+        [[nodiscard]] __host__ __device__ constexpr bool operator()(Ts&&...) const noexcept;
+    };
+
+``cuda::always_true`` is a function object that always returns ``true`` regardless of the number and type of arguments
+passed. ``cuda::always_false`` is a function object that always returns ``false`` regardless of the number and type of
+arguments passed.
+
+Both types are empty, trivially copyable, and their ``operator()`` is ``constexpr`` and ``noexcept``.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/functional>
+
+    __global__ void example_kernel() {
+        cuda::always_true  pred_true{};
+        cuda::always_false pred_false{};
+
+        // Returns true regardless of arguments
+        static_assert(pred_true());
+        static_assert(pred_true(1, 2, 3));
+
+        // Returns false regardless of arguments
+        static_assert(!pred_false());
+        static_assert(!pred_false(1, 2, 3));
+    }
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/functional/maximum_minimum.rst b/cccl_upstream/docs/libcudacxx/extended_api/functional/maximum_minimum.rst
new file mode 100644
index 00000000..5fcf9f6d
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/functional/maximum_minimum.rst
@@ -0,0 +1,74 @@
+.. _libcudacxx-extended-api-functional-maximum-minimum:
+
+``cuda::maximum`` and ``cuda::minimum``
+=======================================
+
+Defined in the header ``<cuda/functional>``.
+
+.. code:: cuda
+
+    template <typename T>
+    struct maximum {
+        [[nodiscard]] __host__ __device__ constexpr
+        T operator()(const T& a, const T& b) const noexcept(/* see below */);
+    };
+
+    template <>
+    struct maximum<void> {
+        template <typename T1, typename T2>
+        [[nodiscard]] __host__ __device__ constexpr
+        cuda::std::common_type_t<T1, T2> operator()(const T1& a, const T2& b) const noexcept(/* see below */);
+    };
+
+    template <typename T>
+    struct minimum {
+        [[nodiscard]] __host__ __device__ constexpr
+        T operator()(const T& a, const T& b) const noexcept(/* see below */);
+    };
+
+    template <>
+    struct minimum<void> {
+        template <typename T1, typename T2>
+        [[nodiscard]] __host__ __device__ constexpr
+        cuda::std::common_type_t<T1, T2> operator()(const T1& a, const T2& b) const noexcept(/* see below */);
+    };
+
+Function objects for performing maximum and minimum operations. The ``operator()`` is ``noexcept`` when the comparison between the values is also ``noexcept``.
+
+.. note::
+
+    Differently from ``std::plus`` and other functional operators, ``cuda::maximum`` and ``cuda::minimum`` specialized for ``void`` returns ``cuda::std::common_type_t`` and not the implicit promotion
+
+Floating-Point Behavior
+-----------------------
+
+For floating-point types (and extended floating-point types), ``cuda::maximum`` uses ``cuda::std::fmax`` and ``cuda::minimum`` uses ``cuda::std::fmin`` instead of the comparison operator, following the ``std::fmax``/``std::fmin`` specification for handling special values such as ``NaN``.
+
+This also makes ``cuda::maximum`` and ``cuda::minimum`` commutative for floating-point types, unlike a plain comparison-based approach.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/functional>
+    #include <cuda/std/cstdint>
+    #include <cstdio>
+    #include <numeric>
+
+    __global__ void maximum_minimum_kernel() {
+        uint16_t v1 = 7;
+        uint16_t v2 = 3;
+        printf("%d\n", cuda::maximum<uint16_t>{}(v1, v2)); // print "7" (uint16_t)
+        printf("%d\n", cuda::minimum{}(v1, v2));           // print "3" (int)
+    }
+
+    int main() {
+        maximum_minimum_kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+        int array[] = {3, 7, 5, 2};
+        printf("%d\n", std::accumulate(array, array + 4, 0, cuda::maximum{})); // 7
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/44fdTerre>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/functional/operator_properties.rst b/cccl_upstream/docs/libcudacxx/extended_api/functional/operator_properties.rst
new file mode 100644
index 00000000..7e831ee7
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/functional/operator_properties.rst
@@ -0,0 +1,340 @@
+.. _libcudacxx-extended-api-functional-operator-properties:
+
+Operator Properties
+===================
+
+Defined in the header ``<cuda/functional>``.
+
+The operator properties traits provide compile-time information about algebraic properties of binary operators.
+These traits are useful for generic algorithms that can apply optimizations based on operator properties,
+such as parallel reductions that can reorder operations for associative operators.
+
+Associativity
+-------------
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <class Op, class T, class Enable = void>
+   inline constexpr bool is_associative_v;
+
+   } // namespace cuda
+
+Determines whether a binary operator ``Op`` is associative for type ``T``, meaning ``op(op(a, b), c) == op(a, op(b, c))``
+for all values ``a``, ``b``, ``c`` of type ``T``. This allows the implementation to reorder operations.
+
+**Supported operators and types:**
+
+.. list-table::
+   :widths: 30 35 35
+   :header-rows: 1
+
+   * - Operator
+     - Integer Types
+     - Floating-Point Types
+   * - ``cuda::std::plus``
+     - ``true``
+     - ``false`` (due to rounding errors)
+   * - ``cuda::std::multiplies``
+     - ``true``
+     - ``false`` (due to rounding errors)
+   * - ``cuda::std::minus``
+     - ``false``
+     - ``false``
+   * - ``cuda::std::divides``
+     - ``false``
+     - ``false``
+   * - ``cuda::std::modulus``
+     - ``false``
+     - N/A
+   * - ``cuda::std::bit_and``
+     - ``true``
+     - N/A
+   * - ``cuda::std::bit_or``
+     - ``true``
+     - N/A
+   * - ``cuda::std::bit_xor``
+     - ``true``
+     - N/A
+   * - ``cuda::std::logical_and``
+     - ``true`` (``bool`` only)
+     - N/A
+   * - ``cuda::std::logical_or``
+     - ``true`` (``bool`` only)
+     - N/A
+   * - ``cuda::minimum``
+     - ``true``
+     - ``true``
+   * - ``cuda::maximum``
+     - ``true``
+     - ``true``
+
+.. note::
+
+   In the strictest sense of the term, the operations of plus and multiplication for integral values may result in undefined behaviour due to overflow. However, in the context of parallel algorithms, they are considered to be associative.
+
+Commutativity
+-------------
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <class Op, class T, class Enable = void>
+   inline constexpr bool is_commutative_v;
+
+   } // namespace cuda
+
+Determines whether a binary operator ``Op`` is commutative for type ``T``, meaning ``op(a, b) == op(b, a)``
+for all values ``a``, ``b`` of type ``T``.
+
+**Supported operators and types:**
+
+.. list-table::
+   :widths: 30 35 35
+   :header-rows: 1
+
+   * - Operator
+     - Integer Types
+     - Floating-Point Types
+   * - ``cuda::std::plus``
+     - ``true``
+     - ``true``
+   * - ``cuda::std::multiplies``
+     - ``true``
+     - ``true``
+   * - ``cuda::std::minus``
+     - ``false``
+     - ``false``
+   * - ``cuda::std::divides``
+     - ``false``
+     - ``false``
+   * - ``cuda::std::modulus``
+     - ``false``
+     - N/A
+   * - ``cuda::std::bit_and``
+     - ``true``
+     - N/A
+   * - ``cuda::std::bit_or``
+     - ``true``
+     - N/A
+   * - ``cuda::std::bit_xor``
+     - ``true``
+     - N/A
+   * - ``cuda::std::logical_and``
+     - ``true`` (``bool`` only)
+     - N/A
+   * - ``cuda::std::logical_or``
+     - ``true`` (``bool`` only)
+     - N/A
+   * - ``cuda::minimum``
+     - ``true``
+     - ``true``
+   * - ``cuda::maximum``
+     - ``true``
+     - ``true``
+
+Identity Element
+----------------
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <class Op, class T>
+   __host__ __device__
+   constexpr auto identity_element() noexcept;
+
+   template <class Op, class T, class Enable = void>
+   inline constexpr bool has_identity_element_v;
+
+   } // namespace cuda
+
+Provides the identity element for operator ``Op`` and type ``T``. The identity element ``e`` satisfies
+``op(e, x) == op(x, e) == x`` for all values ``x`` of type ``T``.
+
+The function ``identity_element<Op, T>()`` returns the identity element at compile time. If no identity element
+exists for the given operator and type combination, it returns an internal sentinel type.
+
+``has_identity_element_v`` evaluates to ``true`` if an identity element is defined for the given operator and type.
+
+**Identity elements by operator:**
+
+.. list-table::
+   :widths: 30 35 35
+   :header-rows: 1
+
+   * - Operator
+     - Integer Types
+     - Floating-Point Types
+   * - ``cuda::std::plus``
+     - ``T{0}``
+     - ``-0.0`` (negative zero)
+   * - ``cuda::std::multiplies``
+     - ``T{1}``
+     - ``1.0``
+   * - ``cuda::std::bit_and``
+     - ``~T{0}`` (all bits set)
+     - N/A
+   * - ``cuda::std::bit_or``
+     - ``T{0}``
+     - N/A
+   * - ``cuda::std::bit_xor``
+     - ``T{0}``
+     - N/A
+   * - ``cuda::std::logical_and``
+     - ``true`` (``bool`` only)
+     - N/A
+   * - ``cuda::std::logical_or``
+     - ``false`` (``bool`` only)
+     - N/A
+   * - ``cuda::minimum``
+     - ``numeric_limits<T>::max()``
+     - ``+infinity``
+   * - ``cuda::maximum``
+     - ``numeric_limits<T>::lowest()``
+     - ``-infinity``
+
+.. note::
+
+   For floating-point ``plus``, the identity element is negative zero (``-0.0``) rather than positive zero.
+   This preserves the sign when adding: ``-0.0 + (-0.0) == -0.0``.
+
+.. note::
+
+   ``cuda::std::minus``, ``cuda::std::divides``, and ``cuda::std::modulus`` do not have identity elements.
+
+Absorbing Element
+-----------------
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <class Op, class T>
+   __host__ __device__
+   constexpr auto absorbing_element() noexcept;
+
+   template <class Op, class T, class Enable = void>
+   inline constexpr bool has_absorbing_element_v;
+
+   } // namespace cuda
+
+Provides the absorbing (annihilating) element for operator ``Op`` and type ``T``. The absorbing element ``z`` satisfies
+``op(z, x) == op(x, z) == z`` for all values ``x`` of type ``T``.
+
+The function ``absorbing_element<Op, T>()`` returns the absorbing element at compile time. If no absorbing element
+exists for the given operator and type combination, it returns an internal sentinel type.
+
+``has_absorbing_element_v`` evaluates to ``true`` if an absorbing element is defined for the given operator and type.
+
+**Absorbing elements by operator:**
+
+.. list-table::
+   :widths: 30 35 35
+   :header-rows: 1
+
+   * - Operator
+     - Integer Types
+     - Floating-Point Types
+   * - ``cuda::std::multiplies``
+     - ``T{0}``
+     - N/A (see note)
+   * - ``cuda::std::bit_and``
+     - ``T{0}``
+     - N/A
+   * - ``cuda::std::bit_or``
+     - ``~T{0}`` (all bits set)
+     - N/A
+   * - ``cuda::std::logical_and``
+     - ``false`` (``bool`` only)
+     - N/A
+   * - ``cuda::std::logical_or``
+     - ``true`` (``bool`` only)
+     - N/A
+   * - ``cuda::minimum``
+     - ``numeric_limits<T>::lowest()``
+     - ``-infinity``
+   * - ``cuda::maximum``
+     - ``numeric_limits<T>::max()``
+     - ``+infinity``
+
+.. note::
+
+   Floating-point ``multiplies`` does not have an absorbing element because:
+
+   - ``0 * NaN = NaN`` (not ``0``)
+   - ``0 * infinity = NaN`` (not ``0``)
+   - ``(-1) * (+0) = -0`` (not ``+0``)
+
+.. note::
+
+   ``cuda::std::plus``, ``cuda::std::minus``, ``cuda::std::divides``, ``cuda::std::modulus``, and ``cuda::std::bit_xor``
+   do not have absorbing elements.
+
+Supported Types
+---------------
+
+The functionality supports all integer and floating-point types, including extended floating-point types.
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/functional>
+   #include <cuda/std/cstdio>
+
+   template <class Op, class T>
+   __host__ __device__ void print_properties() {
+        printf("Associative:   %s\n", cuda::is_associative_v<Op, T> ? "yes" : "no");
+        printf("Commutative:   %s\n", cuda::is_commutative_v<Op, T> ? "yes" : "no");
+        printf("Has identity:  %s\n", cuda::has_identity_element_v<Op, T> ? "yes" : "no");
+        printf("Has absorbing: %s\n", cuda::has_absorbing_element_v<Op, T> ? "yes" : "no");
+
+        if constexpr (cuda::has_identity_element_v<Op, T>) {
+            printf("Identity element exists\n");
+        }
+        if constexpr (cuda::has_absorbing_element_v<Op, T>) {
+            printf("Absorbing element exists\n");
+        }
+   }
+
+   __global__ void example_kernel() {
+        // Integer plus: associative, commutative, identity=0, no absorbing
+        print_properties<cuda::std::plus<int>, int>();
+
+        // Integer plus: NOT associative, commutative, identity=-0.0f, no absorbing
+        print_properties<cuda::std::plus<float>, float>();
+
+        // identity element
+        constexpr int sum_identity = cuda::identity_element<cuda::std::plus<int>, int>();       // 0
+        constexpr int mul_identity = cuda::identity_element<cuda::std::multiplies<int>, int>(); // 1
+   }
+
+Customization
+-------------
+
+Users can extend the operator properties traits to support custom operators or custom types by specializing
+the internal variable templates. The following specializations are available:
+
+.. code:: cuda
+
+   template <>
+   inline constexpr bool cuda::is_associative_v<MyOperator, MyType> = true;
+
+   template <>
+   inline constexpr bool cuda::is_commutative_v<MyOperator, MyType> = true;
+
+   template <>
+   __host__ __device__
+   constexpr auto cuda::identity_element<MyOperator, MyType>() noexcept {
+        return /* MyType identity element */;
+   }
+
+   template <>
+   __host__ __device__
+   constexpr auto cuda::absorbing_element<MyOperator, MyType>() noexcept {
+       return /* MyType absorbing element */;
+   }
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/functional/proclaim_return_type.rst b/cccl_upstream/docs/libcudacxx/extended_api/functional/proclaim_return_type.rst
new file mode 100644
index 00000000..e5947a35
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/functional/proclaim_return_type.rst
@@ -0,0 +1,64 @@
+.. _libcudacxx-extended-api-functional-proclaim-return-type:
+
+``cuda::proclaim_return_type``
+==============================
+
+Defined in the header ``<cuda/functional>``:
+
+.. code:: cuda
+
+   template <class Ret, class Fn>
+   __host__ __device__
+   unspecified<Ret, Fn> proclaim_return_type(Fn&& fn) {
+     return unspecified<Ret, Fn>{fn};
+   }
+
+``cuda::proclaim_return_type`` creates a forwarding call wrapper that uses ``Ret`` as a return type.
+The wrapper is useful in the case of extended device lambdas since an attempt to determine the return type of
+their ``operator()`` function may work incorrectly in host code.
+
+Template Parameters
+-------------------
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``Ret``
+     - Return type that's being proclaimed
+   * - ``Fn``
+     - Callable object type that's being wrapped
+
+Parameters
+----------
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``fn``
+     - Callable object that's being wrapped
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/functional>
+
+   template <class T, class Fn>
+   __global__ void example_kernel(T *out, Fn fn) {
+     *out = fn();
+   }
+
+   __host__ void example() {
+     auto fn = cuda::proclaim_return_type<char>([] __device__ () { return 'd'; });
+     using rt = cuda::std::invoke_result_t<decltype(fn)>;
+
+     rt* out {};
+     cudaMalloc(&out, sizeof(rt));
+
+     example_kernel<<<1, 1>>>(out, fn);
+
+     // ...
+   }
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/iterators.rst b/cccl_upstream/docs/libcudacxx/extended_api/iterators.rst
new file mode 100644
index 00000000..6279e249
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/iterators.rst
@@ -0,0 +1,15 @@
+.. _libcudacxx-extended-api-iterators:
+
+Fancy Iterators
+---------------
+
+CCCL provides a set of fancy iterators that originate from ``Thrust``.
+They allow the user to avoid storing data needlessly in memory and fuse multiple iterations into a single run.
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../api/class*__iterator
+   ../api/class*zip__function
+   ../api/group__iterators*
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/macros.rst b/cccl_upstream/docs/libcudacxx/extended_api/macros.rst
new file mode 100644
index 00000000..679478ea
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/macros.rst
@@ -0,0 +1,34 @@
+.. _libcudacxx-extended-api-macros:
+
+======
+Macros
+======
+
+CCCL provides a set of convenience macros for detecting various system and compile-time
+properties via the preprocessor. These macros are available when any CCCL header is
+included, and do not require including a specific header file.
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Macro**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - .. toctree::
+          :maxdepth: 1
+
+          ../api/macro_cccl_host_arch
+     - Detecting the current host architecture.
+     - CCCL 3.5.0
+     - CUDA 13.5
+
+   * - .. toctree::
+          :maxdepth: 1
+
+          ../api/macro_cccl_os
+     - Detecting the current operating system.
+     - CCCL 3.4.0
+     - CUDA 13.3
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/math.rst b/cccl_upstream/docs/libcudacxx/extended_api/math.rst
new file mode 100644
index 00000000..2b13c7c1
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/math.rst
@@ -0,0 +1,115 @@
+.. _libcudacxx-extended-api-math:
+
+Math
+====
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   math/ceil_div
+   math/round_up
+   math/round_down
+   math/ilog
+   math/ipow
+   math/pow2
+   math/isqrt
+   math/neg
+   math/uabs
+   math/fast_mod_div
+   math/mul_hi
+   math/sincos
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`ceil_div <libcudacxx-extended-api-math-ceil-div>`
+     - Ceiling division
+     - CCCL 2.7.0
+     - CUDA 12.8
+
+   * - :ref:`round_up <libcudacxx-extended-api-math-round-up>`
+     - Round up to the next multiple
+     - CCCL 3.0.0
+     - CUDA 13.0
+
+   * - :ref:`round_down <libcudacxx-extended-api-math-round-down>`
+     - Round down to the previous multiple
+     - CCCL 3.0.0
+     - CUDA 13.0
+
+   * - :ref:`ilog2 <libcudacxx-extended-api-math-ilog>`
+     - Integer logarithm to the base 2
+     - CCCL 3.0.0
+     - CUDA 13.0
+
+   * - :ref:`ceil_ilog2 <libcudacxx-extended-api-math-ilog>`
+     - Integer logarithm to the base 2, rounded up
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`ilog10 <libcudacxx-extended-api-math-ilog>`
+     - Integer logarithm to the base 10
+     - CCCL 3.0.0
+     - CUDA 13.0
+
+   * - :ref:`ceil_ilog10 <libcudacxx-extended-api-math-ilog>`
+     - Integer logarithm to the base 10, rounded up
+     - CCCL 3.5.0
+     - CUDA 13.5
+
+   * - :ref:`ipow <libcudacxx-extended-api-math-ipow>`
+     - Integer power
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`is_power_of_two <libcudacxx-extended-api-math-pow2>`
+     - If the value is a power of two
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`isqrt <libcudacxx-extended-api-math-isqrt>`
+     - Integer square root
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`neg <libcudacxx-extended-api-math-neg>`
+     - Integer negation
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`next_power_of_two <libcudacxx-extended-api-math-pow2>`
+     - Next power of two
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`prev_power_of_two <libcudacxx-extended-api-math-pow2>`
+     - Previous power of two
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`uabs <libcudacxx-extended-api-math-uabs>`
+     - Unsigned absolute value
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`fast_mod_div <libcudacxx-extended-api-math-fast-mod-div>`
+     - Fast Modulo/Division
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`mul_hi <libcudacxx-extended-api-math-mul-hi>`
+     - Most significant half of the product
+     - CCCL 3.2.0
+     - CUDA 13.2
+
+   * - :ref:`sincos <libcudacxx-extended-api-math-sincos>`
+     - Computes sine and cosine of a value at the same time.
+     - CCCL 3.3.0
+     - CUDA 13.3
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/math/ceil_div.rst b/cccl_upstream/docs/libcudacxx/extended_api/math/ceil_div.rst
new file mode 100644
index 00000000..aa6d3f20
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/math/ceil_div.rst
@@ -0,0 +1,77 @@
+.. _libcudacxx-extended-api-math-ceil-div:
+
+``cuda::ceil_div``
+==================
+
+Defined in the ``<cuda/cmath>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T, typename U>
+   [[nodiscard]] __host__ __device__ constexpr
+   cuda::std::common_type_t<T, U> ceil_div(T value, U divisor) noexcept;
+
+   } // namespace cuda
+
+The function computes the ceiling division between two integral or enumerator values :math:`ceil(\frac{value}{base\_multiple})`.
+
+**Parameters**
+
+- ``value``: The value to be divided.
+- ``divisor``: The divisor.
+
+**Return value**
+
+Divides ``value`` by ``divisor``. If ``value`` is not a multiple of ``divisor`` rounds the result up to the next integer value.
+
+**Constraints**
+
+- ``T`` and ``U`` are integer types or enumerators.
+
+**Preconditions**
+
+- ``value >= 0``
+- ``divisor > 0``
+
+**Performance considerations**
+
+- The function computes ``(value + divisor - 1) / divisor`` when the common type is a signed integer.
+- The function computes ``min(value, 1 + ((value - 1) / divisor)`` when the common type is an unsigned integer in CUDA, which generates less instructions than ``(value / divisor) + ((value / divisor) * divisor != value)``, especially for 64-bit types.
+
+Example
+-------
+
+This API is very useful for determining the *number of thread blocks* required to process a fixed amount of work, given a fixed number of threads per block:
+
+.. code:: cuda
+
+    #include <cuda/cmath>
+    #include <cuda/std/span>
+    #include <thrust/device_vector.h>
+
+    __global__ void vector_scale_kernel(cuda::std::span<float> span, float scale) {
+        int index = blockIdx.x * blockDim.x + threadIdx.x;
+        if (index < span.size())
+            span[index] *= scale;
+    }
+
+    int main() {
+        int   num_items = 100'000;
+        float scale = 2.f;
+        thrust::device_vector<float> d_vector(num_items, 1.f);
+        // Given a fixed number of threads per block...
+        constexpr int threads_per_block = 256;
+        // ...dividing some "n" by "threads_per_block" may lead to a remainder,
+        // requiring the kernel to be launched with an extra thread block to handle it.
+        auto num_thread_blocks = cuda::ceil_div(num_items, threads_per_block);
+        auto d_ptr             = thrust::raw_pointer_cast(d_vector.data());
+        cuda::std::span<float> d_span(d_ptr, num_items);
+
+        vector_scale_kernel<<<num_thread_blocks, threads_per_block>>>(d_span, scale);
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/dj9a5137z>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/math/fast_mod_div.rst b/cccl_upstream/docs/libcudacxx/extended_api/math/fast_mod_div.rst
new file mode 100644
index 00000000..2d76608d
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/math/fast_mod_div.rst
@@ -0,0 +1,92 @@
+.. _libcudacxx-extended-api-math-fast-mod-div:
+
+``cuda::fast_mod_div``
+======================
+
+Defined in the ``<cuda/cmath>`` header.
+
+.. code:: cuda
+
+    namespace cuda {
+
+    template <typename T, bool DivisorIsNeverOne = false>
+    class fast_mod_div {
+    public:
+        fast_mod_div() = delete;
+
+        __host__ __device__
+        explicit fast_mod_div(T divisor) noexcept;
+
+        template <typename U>
+        [[nodiscard]] __host__ __device__ friend
+        cuda:::std::common_type_t<T, U> operator/(U dividend, fast_mod_div<T> divisor) noexcept;
+
+        template <typename U>
+        [[nodiscard]] __host__ __device__ friend
+        cuda:::std::common_type_t<T, U> operator%(U dividend, fast_mod_div<T> divisor) noexcept;
+
+        [[nodiscard]] __host__ __device__
+        operator T() const noexcept;
+    };
+
+    } // namespace cuda
+
+.. code:: cuda
+
+    namespace cuda {
+
+    template <typename T, typename U>
+    [[nodiscard]] __host__ __device__
+    cuda::std::pair<T, U> div(T dividend, fast_mod_div<U> divisor) noexcept;
+
+    } // namespace cuda
+
+The class ``fast_mod_div`` is used to pre-compute the modulo and division of an integer value, to be used in a second stage for efficiency: :math:`floor\left(\frac{dividend}{divisor}\right)`.
+
+**Parameters**
+
+- ``divisor``:  The divisor.
+- ``dividend``: The dividend.
+- ``DivisorIsNeverOne``: Indicates that ``divisor != 1`` and skips one comparison in the second stage.
+
+**Constraints**
+
+- ``T`` and ``U`` are integer types.
+- ``max_value(dividend type) <= max_value(divisor type)``.
+
+**Preconditions**
+
+- ``divisor > 0``.
+- ``dividend >= 0``.
+- ``divisor > 1`` if ``DivisorIsNeverOne == true``.
+
+**Performance considerations**
+
+- ``fast_mod_div`` needs to be initialized on the host and executed on the device for optimal performance.
+- ``T`` signed type ensures the best performance. ``T == int`` translates to ``SEL``, ``IMAD``, and x2 ``SHF`` instructions.
+- Larger types (> 32-bits) are slower than smaller types.
+- ``DivisorIsNeverOne == true`` can be used to skip one comparison.
+- ``__builtin_assume(dividend != cuda::std::numeric_limits<U>::max())`` can be used to skip one comparison with unsigned values.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/cmath>
+    #include <cuda/std/cassert>
+
+    __global__ void div_kernel(cuda::fast_mod_div<int> divisor) {
+        assert(45 / divisor == 2);
+        assert(45 % divisor == 5);
+        assert((cuda::div(45, divisor) == cuda::std::pair{2, 5}));
+    }
+
+    int main() {
+        cuda::fast_mod_div<int> divisor(20);
+        div_kernel<<<1, 1>>>(divisor);
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/fM7E9v9aP>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/math/ilog.rst b/cccl_upstream/docs/libcudacxx/extended_api/math/ilog.rst
new file mode 100644
index 00000000..f356c614
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/math/ilog.rst
@@ -0,0 +1,83 @@
+.. _libcudacxx-extended-api-math-ilog:
+
+``cuda::ilog2``, ``cuda::ceil_ilog2``, ``cuda::ilog10``, and ``cuda::ceil_ilog10``
+=================================================================================
+
+Defined in the ``<cuda/cmath>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T>
+   [[nodiscard]] __host__ __device__ __tile__ constexpr
+   int ilog2(T value) noexcept;
+
+   template <typename T>
+   [[nodiscard]] __host__ __device__ __tile__ constexpr
+   int ceil_ilog2(T value) noexcept;
+
+   template <typename T>
+   [[nodiscard]] __host__ __device__ __tile__ constexpr
+   int ilog10(T value) noexcept;
+
+   template <typename T>
+   [[nodiscard]] __host__ __device__ __tile__ constexpr
+   int ceil_ilog10(T value) noexcept;
+
+   } // namespace cuda
+
+The functions compute the logarithm to the base 2 and 10 of an integer value.
+
+**Parameters**
+
+- ``value``: The input value.
+
+**Return value**
+
+- ``ilog2``, ``ceil_ilog2``: The logarithm to the base 2, rounded down and up to the nearest integer respectively.
+- ``ilog10``, ``ceil_ilog10``: The logarithm to the base 10, rounded down and up to the nearest integer respectively.
+
+**Constraints**
+
+- ``T`` is an integer type.
+
+**Preconditions**
+
+- ``value > 0``
+
+**Performance considerations**
+
+The functions perform the following operations in device code:
+
+- ``ilog2``: ``FLO``
+- ``ceil_ilog2``: ``FLO``, ``POPC``, ``ADD``, comparison
+- ``ilog10``: ``FLO``, ``FMUL``, ``F2I``, constant memory lookup, ``SEL`` + ``IADD`` only if ``T == uint32_t`` or ``T == __uint128_t``
+- ``ceil_ilog10``: ``ilog10`` with an additional comparison, subtraction, and ``IADD``
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/cmath>
+    #include <cuda/std/cassert>
+
+    __global__ void ilog_kernel() {
+        assert(cuda::ilog2(20) == 4);
+        assert(cuda::ceil_ilog2(20) == 5);
+        assert(cuda::ilog2(32) == 5);
+        assert(cuda::ceil_ilog2(32) == 5);
+        assert(cuda::ilog10(100) == 2);
+        assert(cuda::ilog10(2000) == 3);
+        assert(cuda::ceil_ilog10(100) == 2);
+        assert(cuda::ceil_ilog10(2000) == 4);
+    }
+
+    int main() {
+        ilog_kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/7W3WaGd3c>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/math/ipow.rst b/cccl_upstream/docs/libcudacxx/extended_api/math/ipow.rst
new file mode 100644
index 00000000..c77d373e
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/math/ipow.rst
@@ -0,0 +1,60 @@
+.. _libcudacxx-extended-api-math-ipow:
+
+``cuda::ipow``
+====================================
+
+Defined in the ``<cuda/cmath>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T, typename E>
+   [[nodiscard]] __host__ __device__ constexpr
+   T ipow(T base, E exp) noexcept;
+
+   } // namespace cuda
+
+The function computes the integer ``base`` raised to the power of ``exp``.
+
+**Parameters**
+
+- ``base``: The base value.
+- ``exp``: The exponent value.
+
+**Return value**
+
+- The result of raising ``base`` to the power of ``exp``. If ``exp`` is negative, the result is 0.
+
+**Constraints**
+
+- ``T`` is an integer type.
+- ``E`` is an integer type.
+
+**Preconditions**
+
+- if ``base`` is 0, then ``exp`` must be non-negative.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/cmath>
+    #include <cuda/std/cassert>
+
+    __global__ void ipow_kernel() {
+        assert(cuda::ipow(0, 0) == 1);
+        assert(cuda::ipow(2, 2) == 4);
+        assert(cuda::ipow(99, 1) == 99);
+        assert(cuda::ipow(4, 7) == 16384);
+        assert(cuda::ipow(-1, 3) == -1);
+        assert(cuda::ipow(23, -1) == 0);
+    }
+
+    int main() {
+        ipow_kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/TMacWvz8v>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/math/isqrt.rst b/cccl_upstream/docs/libcudacxx/extended_api/math/isqrt.rst
new file mode 100644
index 00000000..6ace1736
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/math/isqrt.rst
@@ -0,0 +1,57 @@
+.. _libcudacxx-extended-api-math-isqrt:
+
+``cuda::isqrt``
+====================================
+
+Defined in the ``<cuda/cmath>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T>
+   [[nodiscard]] __host__ __device__ constexpr
+   T isqrt(T value) noexcept;
+
+   } // namespace cuda
+
+The function computes the integer square root of the input value rounded down.
+
+**Parameters**
+
+- ``value``: The input value.
+
+**Return value**
+
+- The square root value of the input value rounded down.
+
+**Constraints**
+
+- ``T`` is an integer type.
+
+**Preconditions**
+
+- ``value`` is non-negative.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/cmath>
+    #include <cuda/std/cassert>
+
+    __global__ void isqrt_kernel() {
+        assert(cuda::isqrt(1) == 1);
+        assert(cuda::isqrt(4) == 2);
+        assert(cuda::isqrt(42) == 6);
+        assert(cuda::isqrt(99) == 9);
+        assert(cuda::isqrt(100) == 10);
+    }
+
+    int main() {
+        isqrt_kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/xPcj35dq6>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/math/mul_hi.rst b/cccl_upstream/docs/libcudacxx/extended_api/math/mul_hi.rst
new file mode 100644
index 00000000..c92a672a
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/math/mul_hi.rst
@@ -0,0 +1,64 @@
+.. _libcudacxx-extended-api-math-mul-hi:
+
+``cuda::mul_hi``
+================
+
+Defined in ``<cuda/cmath>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T>
+   [[nodiscard]] __host__ __device__ constexpr
+   T mul_hi(T lhs, T rhs) noexcept;
+
+   } // namespace cuda
+
+Computes the most significant half of the bits of the product of two non-negative integers ``lhs`` and ``rhs``.
+
+**Parameters**
+
+- ``lhs``: First multiplicand.
+- ``rhs``: Second multiplicand.
+
+**Return value**
+
+- The most significant half of ``lhs * rhs`` returned as ``T``.
+
+**Constraints**
+
+- ``T`` is an integer type.
+
+**Remarks**
+
+- Uses ``__mulhi``, ``__umulhi``, ``__mul64hi``, ``__umul64hi`` intrinsics on device when available.
+- Uses ``__mulh``, ``__umulh`` intrinsics on Windows host code when available.
+- Uses a double-width intermediate type when possible.
+- Relies on a manual decomposition fallback when 128-bit intermediates are unavailable for 64-bit integers.
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/cmath>
+   #include <cuda/std/cassert>
+   #include <cuda/std/cstdint>
+
+   __global__ void mul_hi_kernel()
+   {
+       uint32_t lhs       = 0xABCD1234;
+       uint32_t rhs       = 1 << 16; // 2^16
+       uint32_t high_half = cuda::mul_hi(lhs, rhs);
+       assert(high_half == 0xAB);
+   }
+
+   int main()
+   {
+       mul_hi_kernel<<<1, 1>>>();
+       cudaDeviceSynchronize();
+       return 0;
+   }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/64r6zT9Wq>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/math/neg.rst b/cccl_upstream/docs/libcudacxx/extended_api/math/neg.rst
new file mode 100644
index 00000000..75f9d866
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/math/neg.rst
@@ -0,0 +1,57 @@
+.. _libcudacxx-extended-api-math-neg:
+
+``cuda::neg``
+====================================
+
+Defined in the ``<cuda/cmath>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T>
+   [[nodiscard]] __host__ __device__ constexpr
+   T neg(T value) noexcept;
+
+   } // namespace cuda
+
+The function computes the negation of the input value accepting both signed and unsigned integer types. It doesn't emit any warnings for signed integer overflow and applying ``-`` to unsigned integer types.
+
+**Parameters**
+
+- ``value``: The input value.
+
+**Return value**
+
+- The negated value of the input value.
+
+**Constraints**
+
+- ``T`` is an integer type.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/cmath>
+    #include <cuda/std/cassert>
+    #include <cuda/std/limits>
+
+    __global__ void neg_kernel() {
+        using cuda::std::numeric_limits;
+
+        assert(cuda::neg(1) == -1);
+        assert(cuda::neg(20) == -20);
+        assert(cuda::neg(127u) == 4294967169u);
+        assert(cuda::neg(-127) == 127);
+        assert(cuda::neg(cuda::std::numeric_limits<int>::min()) == cuda::std::numeric_limits<int>::min());
+    }
+
+    int main() {
+        neg_kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/K3zcE9zqn>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/math/pow2.rst b/cccl_upstream/docs/libcudacxx/extended_api/math/pow2.rst
new file mode 100644
index 00000000..935e7326
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/math/pow2.rst
@@ -0,0 +1,70 @@
+.. _libcudacxx-extended-api-math-pow2:
+
+Power of Two Utilities
+======================
+
+Defined in the ``<cuda/cmath>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T>
+   [[nodiscard]] __host__ __device__ constexpr
+   bool is_power_of_two(T value) noexcept;
+
+   template <typename T>
+   [[nodiscard]] __host__ __device__ constexpr
+   T next_power_of_two(T value) noexcept;
+
+   template <typename T>
+   [[nodiscard]] __host__ __device__ constexpr
+   T prev_power_of_two(T value) noexcept;
+
+   } // namespace cuda
+
+The functions provide utilities to determine if an integer value is a power of two, and to compute the next and previous power of two.
+
+**Parameters**
+
+- ``value``: The input value.
+
+**Return value**
+
+- ``is_power_of_two``: Return ``true`` if ``value`` is a power of two, ``false`` otherwise.
+- ``next_power_of_two``: Return the smallest power of two greater than or equal to ``value``.
+- ``prev_power_of_two``: Return the largest power of two less than or equal to ``value``.
+
+**Constraints**
+
+- ``T`` is an integer types. Contrary to ``cuda::std::has_single_bit``, ``cuda::std::bit_floor``, and ``cuda::std::bit_ceil``, ``T`` can be both signed and unsigned.
+
+**Preconditions**
+
+- ``value > 0``
+
+**Performance considerations**
+
+See :ref:`\<cuda/std/bit\> performance considerations <libcudacxx-standard-api-numerics-bit>`
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/cmath>
+    #include <cuda/std/cassert>
+
+    __global__ void pow2_kernel() {
+        assert(!cuda::is_power_of_two(20));
+        assert(cuda::next_power_of_two(20) == 32);
+        assert(cuda::prev_power_of_two(20) == 16);
+    }
+
+    int main() {
+        pow2_kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/896Yx3vf8>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/math/round_down.rst b/cccl_upstream/docs/libcudacxx/extended_api/math/round_down.rst
new file mode 100644
index 00000000..56e46176
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/math/round_down.rst
@@ -0,0 +1,62 @@
+.. _libcudacxx-extended-api-math-round-down:
+
+``cuda::round_down``
+====================
+
+Defined in the ``<cuda/cmath>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T, typename U>
+   [[nodiscard]] __host__ __device__ constexpr
+   cuda::std::common_type_t<T, U> round_down(T value, U base_multiple) noexcept;
+
+   } // namespace cuda
+
+The function computes the round down to the largest multiple of an integral or enumerator value :math:`floor(\frac{value}{base\_multiple}) * base\_multiple`
+
+**Parameters**
+
+- ``value``: The value to be rounded down.
+- ``base_multiple``:  The base multiple to which the value rounds down.
+
+**Return value**
+
+``value`` rounded down to the largest multiple of ``base_multiple`` less than or equal to ``value``. If ``value`` is already a multiple of ``base_multiple``, returns ``value``.
+
+**Constraints**
+
+- ``T`` and ``U`` are integer types or enumerators.
+
+**Preconditions**
+
+- ``value >= 0``
+- ``base_multiple > 0``
+
+**Performance considerations**
+
+- The function performs a truncation division followed by a multiplication. It provides better performance than ``(value / base_multiple) * base_multiple`` when the common type is a signed integer
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/cmath>
+    #include <cstdio>
+
+    __global__ void round_up_kernel() {
+        int      value    = 7;
+        unsigned multiple = 3;
+        printf("%d\n", cuda::round_down(value, multiple)); // print "6"
+    }
+
+    int main() {
+        round_up_kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/cxGYfMGna>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/math/round_up.rst b/cccl_upstream/docs/libcudacxx/extended_api/math/round_up.rst
new file mode 100644
index 00000000..60703898
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/math/round_up.rst
@@ -0,0 +1,66 @@
+.. _libcudacxx-extended-api-math-round-up:
+
+``cuda::round_up``
+==================
+
+Defined in the ``<cuda/cmath>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T, typename U>
+   [[nodiscard]] __host__ __device__ constexpr
+   cuda::std::common_type_t<T, U> round_up(T value, U base_multiple) noexcept;
+
+   } // namespace cuda
+
+The function computes the round up to the smallest multiple of an integral or enumerator value :math:`ceil(\frac{value}{base\_multiple}) * base\_multiple`
+
+**Parameters**
+
+- ``value``: The value to be rounded up.
+- ``base_multiple``:  The base multiple to which the value rounds up.
+
+**Return value**
+
+``value`` rounded up to the smallest multiple of ``base_multiple`` greater than or equal to ``value``. If ``value`` is already a multiple of ``base_multiple``, return ``value``.
+
+.. note::
+
+    The result can overflow if ``ceil(value / base_multiple) * base_multiple`` exceeds the maximum value of the common type of ``value`` and ``base_multiple``. The condition is checked in debug mode.
+
+**Constraints**
+
+- ``T`` and ``U`` are integer types or enumerators.
+
+**Preconditions**
+
+- ``value >= 0``
+- ``base_multiple > 0``
+
+**Performance considerations**
+
+- The function performs a ceiling division (``cuda::ceil_div()``) followed by a multiplication
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/cmath>
+    #include <cstdio>
+
+    __global__ void round_up_kernel() {
+        int      value    = 7;
+        unsigned multiple = 3;
+        printf("%d\n", cuda::round_up(value, multiple)); // print "9"
+    }
+
+    int main() {
+        round_up_kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/WacYfxsTT>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/math/sincos.rst b/cccl_upstream/docs/libcudacxx/extended_api/math/sincos.rst
new file mode 100644
index 00000000..40d87ce9
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/math/sincos.rst
@@ -0,0 +1,68 @@
+.. _libcudacxx-extended-api-math-sincos:
+
+``cuda::sincos``
+====================================
+
+Defined in the ``<cuda/cmath>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <class T>
+   struct sincos_result
+   {
+     T sin;
+     T cos;
+   };
+
+   template </*floating-point-type*/ T>
+   [[nodiscard]] __host__ __device__
+   sincos_result<T> sincos(T value) noexcept; // (1)
+
+   template <class Integral>
+   [[nodiscard]] __host__ __device__
+   sincos_result<double> sincos(Integral value) noexcept; // (2)
+
+   } // namespace cuda
+
+Computes :math:`\sin value` and :math:`\cos value` at the same time using more efficient algorithms than if operations were computed separately.
+
+**Parameters**
+
+- ``value``: The input value.
+
+**Return value**
+
+- ``cuda::sincos_result`` object with both values set to ``NaN`` if the input value is :math:`\pm\infty` or ``NaN`` and to results of :math:`\sin value` and :math:`\cos value` otherwise. (1)
+- if ``T`` is an integral type, the input value is treated as ``double``. (2)
+
+**Constraints**
+
+- ``T`` is an arithmetic type.
+
+**Performance considerations**
+
+- If available, the functionality is implemented by compiler builtins, otherwise fallbacks to ``cuda::std::sin(value)`` and ``cuda::std::cos(value)``.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/cmath>
+    #include <cuda/std/cassert>
+
+    __global__ void sincos_kernel() {
+        auto [sin_pi, cos_pi] = cuda::sincos(0.f);
+        assert(sin_pi == 0.f);
+        assert(cos_pi == 1.f);
+    }
+
+    int main() {
+        sincos_kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/99PP9s1z6>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/math/uabs.rst b/cccl_upstream/docs/libcudacxx/extended_api/math/uabs.rst
new file mode 100644
index 00000000..a7c69bd4
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/math/uabs.rst
@@ -0,0 +1,56 @@
+.. _libcudacxx-extended-api-math-uabs:
+
+``cuda::uabs``
+====================================
+
+Defined in the ``<cuda/cmath>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T>
+   [[nodiscard]] __host__ __device__ constexpr
+   cuda::std::make_unsigned_t<T> uabs(T value) noexcept;
+
+   } // namespace cuda
+
+The function computes the absolute value of the input value. The result is returned as an unsigned integer type of the same size as the input value. In comparison to the standard ``abs`` function, the ``uabs`` eliminates the undefined behaviour when a signed ``T_MIN`` is passed as an input.
+
+**Parameters**
+
+- ``value``: The input value.
+
+**Return value**
+
+- The unsigned absolute value of the input value.
+
+**Constraints**
+
+- ``T`` is an integer type.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/cmath>
+    #include <cuda/std/cassert>
+    #include <cuda/std/limits>
+
+    __global__ void uabs_kernel() {
+        using cuda::std::numeric_limits;
+
+        assert(cuda::uabs(20) == 20u);
+        assert(cuda::uabs(-32) == 32u);
+        assert(cuda::uabs(numeric_limits<int>::max()) == static_cast<unsigned>(numeric_limits<int>::max()));
+        assert(cuda::uabs(numeric_limits<int>::min()) == static_cast<unsigned>(numeric_limits<int>::max()) + 1);
+    }
+
+    int main() {
+        uabs_kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/KEoYfq53G>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/mdspan.rst b/cccl_upstream/docs/libcudacxx/extended_api/mdspan.rst
new file mode 100644
index 00000000..59fb658b
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/mdspan.rst
@@ -0,0 +1,54 @@
+.. _libcudacxx-extended-api-mdspan:
+
+Mdspan
+======
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   mdspan/host_device_accessor
+   mdspan/layout_stride_relaxed
+   mdspan/restrict_accessor
+   mdspan/shared_memory_accessor
+   mdspan/mdspan_to_dlpack
+   mdspan/dlpack_to_mdspan
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`host/device/managed mdspan and accessor <libcudacxx-extended-api-mdspan-host-device-accessor>`
+     - CUDA memory space ``mdspan`` and accessors
+     - CCCL 3.0.0
+     - CUDA 13.0
+
+   * - :ref:`layout_stride_relaxed <libcudacxx-extended-api-mdspan-layout-stride-relaxed>`
+     - Layout mapping policy with negative/zero strides and offset support
+     - CCCL 3.0.0
+     - CUDA 13.0
+
+   * - :ref:`restrict mdspan and accessor <libcudacxx-extended-api-mdspan-restrict-accessor>`
+     - ``mdspan`` and accessor with the *restrict* aliasing policy
+     - CCCL 3.0.0
+     - CUDA 13.0
+
+   * - :ref:`shared_memory mdspan and accessor <libcudacxx-extended-api-mdspan-shared-memory-accessor>`
+     - ``mdspan`` and accessor for CUDA shared memory
+     - CCCL 3.3.0
+     - CUDA 13.3
+
+   * - :ref:`mdspan to dlpack <libcudacxx-extended-api-mdspan-mdspan-to-dlpack>`
+     - Convert a ``mdspan`` to a ``DLTensor``
+     - CCCL 3.2.0
+     - CUDA 13.2
+
+   * - :ref:`dlpack to mdspan <libcudacxx-extended-api-mdspan-dlpack-to-mdspan>`
+     - Convert a ``DLTensor`` to a ``mdspan``
+     - CCCL 3.2.0
+     - CUDA 13.2
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/mdspan/dlpack_to_mdspan.rst b/cccl_upstream/docs/libcudacxx/extended_api/mdspan/dlpack_to_mdspan.rst
new file mode 100644
index 00000000..4cadd7cf
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/mdspan/dlpack_to_mdspan.rst
@@ -0,0 +1,141 @@
+.. _libcudacxx-extended-api-mdspan-dlpack-to-mdspan:
+
+DLPack to ``mdspan``
+====================
+
+This functionality provides a conversion from `DLPack <https://dmlc.github.io/dlpack/latest/>`__ ``DLTensor`` to ``cuda::host_mdspan``, ``cuda::device_mdspan``, and ``cuda::managed_mdspan``.
+
+Defined in the ``<cuda/mdspan>`` header.
+
+Conversion functions
+--------------------
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename ElementType, size_t Rank, typename LayoutPolicy = cuda::layout_stride_relaxed>
+   [[nodiscard]] cuda::host_mdspan<ElementType, cuda::std::dims<Rank, int64_t>, LayoutPolicy>
+   to_host_mdspan(const DLTensor& tensor);
+
+   template <typename ElementType, size_t Rank, typename LayoutPolicy = cuda::layout_stride_relaxed>
+   [[nodiscard]] cuda::device_mdspan<ElementType, cuda::std::dims<Rank, int64_t>, LayoutPolicy>
+   to_device_mdspan(const DLTensor& tensor);
+
+   template <typename ElementType, size_t Rank, typename LayoutPolicy = cuda::layout_stride_relaxed>
+   [[nodiscard]] cuda::managed_mdspan<ElementType, cuda::std::dims<Rank, int64_t>, LayoutPolicy>
+   to_managed_mdspan(const DLTensor& tensor);
+
+   } // namespace cuda
+
+Template parameters
+-------------------
+
+- ``ElementType``: The element type of the resulting ``mdspan``. Must match the ``DLTensor::dtype``.
+- ``Rank``: The number of dimensions. Must match ``DLTensor::ndim``.
+- ``LayoutPolicy``: The layout policy for the resulting ``mdspan``. Defaults to ``cuda::layout_stride_relaxed``. Supported layouts are:
+
+  - ``cuda::std::layout_right`` (C-contiguous, row-major)
+  - ``cuda::std::layout_left`` (Fortran-contiguous, column-major)
+  - ``cuda::std::layout_stride`` (general strided layout)
+  - ``cuda::layout_stride_relaxed`` (general strided layout with negative/zero strides and offset support)
+
+Semantics
+---------
+
+The conversion produces a non-owning ``mdspan`` view of the ``DLTensor`` data:
+
+- For ``layout_right``, ``layout_left``, and ``layout_stride``, the data pointer is computed as ``static_cast<char*>(tensor.data) + tensor.byte_offset``.
+- For ``layout_stride_relaxed``, the data pointer is ``tensor.data`` directly (no ``byte_offset`` adjustment). Instead, ``tensor.byte_offset`` is converted to an element offset (``byte_offset / sizeof(ElementType)``) and stored in the mapping. This ensures that ``mapping(indices...) = offset + sum(index_i * stride_i)`` produces non-negative indices even with negative strides, and ``required_span_size()`` correctly reflects the actual memory span.
+- For ``rank > 0``, ``mdspan.extent(i)`` is ``tensor.shape[i]``.
+- For ``layout_stride`` and ``layout_stride_relaxed``, ``mdspan.stride(i)`` is ``tensor.strides[i]`` (or computed as row-major if ``strides`` is ``nullptr`` for DLPack < v1.2).
+- The device type is validated:
+
+  - ``kDLCPU`` for ``to_host_mdspan``
+  - ``kDLCUDA`` for ``to_device_mdspan``
+  - ``kDLCUDAManaged`` for ``to_managed_mdspan``
+
+Supported element types:
+
+- ``bool``.
+- Signed and unsigned integers.
+- IEEE-754 Floating-point and extended precision floating-point, including ``__half``, ``__nv_bfloat16``, ``__float128``, FP8, FP6, FP4 when available.
+- Complex: ``cuda::std::complex<__half>``, ``cuda::std::complex<float>``, and ``cuda::std::complex<double>``.
+- `CUDA built-in vector types <https://docs.nvidia.com/cuda/cuda-programming-guide/05-appendices/cpp-language-extensions.html#built-in-types>`__, such as ``int2``, ``float4``, etc.
+- Vector types for extended floating-point, such as ``__half2``, ``__nv_fp8x4_e4m3``, etc.
+
+Constraints
+-----------
+
+- ``LayoutPolicy`` must be one of ``cuda::std::layout_right``, ``cuda::std::layout_left``, ``cuda::std::layout_stride``, or ``cuda::layout_stride_relaxed``.
+- For ``layout_right`` and ``layout_left``, the ``DLTensor`` strides must be compatible with the layout.
+
+Runtime errors
+--------------
+
+The conversion throws ``std::invalid_argument`` in the following cases:
+
+- ``DLTensor::ndim`` does not match the specified ``Rank``.
+- ``DLTensor::dtype`` does not match ``ElementType``.
+- ``DLTensor::data`` is ``nullptr``.
+- ``DLTensor::shape`` is ``nullptr`` (for rank > 0).
+- Any ``DLTensor::shape[i]`` is negative.
+- ``DLTensor::strides`` is ``nullptr`` for DLPack v1.2 or later.
+- ``DLTensor::strides`` is ``nullptr`` for ``layout_left`` with rank > 1 (DLPack < v1.2).
+- ``DLTensor::strides[i]`` is not positive for ``layout_stride``.
+- ``DLTensor::strides`` are not compatible with the requested ``layout_right`` or ``layout_left``.
+- ``DLTensor::byte_offset`` is not a multiple of the element size for ``layout_stride_relaxed``.
+- ``DLTensor::device.device_type`` does not match the target mdspan type.
+- Data pointer is not properly aligned for the element type.
+
+Availability notes
+------------------
+
+- This API is available only when DLPack header is present, namely ``<dlpack/dlpack.h>`` is found in the include path.
+- This API can be disabled by defining ``CCCL_DISABLE_DLPACK`` before including any library headers. In this case, ``<dlpack/dlpack.h>`` will not be included.
+
+References
+----------
+
+- `DLPack C API <https://dmlc.github.io/dlpack/latest/c_api.html>`__ documentation.
+
+Example
+-------
+
+.. code:: cuda
+
+  #include <dlpack/dlpack.h>
+  #include <cuda/mdspan>
+  #include <cuda/std/cassert>
+  #include <cuda/std/cstdint>
+
+  int main() {
+    int data[6] = {0, 1, 2, 3, 4, 5};
+
+    // Create a DLTensor manually for demonstration
+    int64_t shape[2]   = {2, 3};
+    int64_t strides[2] = {3, 1};  // row-major strides
+
+    DLTensor tensor{};
+    tensor.data        = data;
+    tensor.device      = {kDLCPU, 0};
+    tensor.ndim        = 2;
+    tensor.dtype       = DLDataType{kDLInt, 32, 1};
+    tensor.shape       = shape;
+    tensor.strides     = strides;
+    tensor.byte_offset = 0;
+
+    // Convert to host_mdspan
+    auto md = cuda::to_host_mdspan<int, 2>(tensor);
+
+    assert(md.rank() == 2);
+    assert(md.extent(0) == 2 && md.extent(1) == 3);
+    assert(md.stride(0) == 3 && md.stride(1) == 1);
+    assert(md.data_handle() == data);
+    assert(md(0, 0) == 0 && md(1, 2) == 5);
+  }
+
+See also
+--------
+
+- :ref:`libcudacxx-extended-api-mdspan-mdspan-to-dlpack` for the reverse conversion.
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/mdspan/host_device_accessor.rst b/cccl_upstream/docs/libcudacxx/extended_api/mdspan/host_device_accessor.rst
new file mode 100644
index 00000000..09eae6ce
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/mdspan/host_device_accessor.rst
@@ -0,0 +1,205 @@
+.. _libcudacxx-extended-api-mdspan-host-device-accessor:
+
+``host/device/managed`` ``mdspan`` and ``accessors``
+====================================================
+
+*Host*, *device*, and *managed* ``mdspan`` allow to express multi-dimensional views of the respective CUDA memory spaces as *vocabulary types* and prevent potential errors.
+
+Types and Traits
+----------------
+
+.. code:: cpp
+
+  template <typename Accessor>
+  using host_accessor;
+
+  template <typename Accessor>
+  using device_accessor;
+
+  template <typename Accessor>
+  using managed_accessor;
+
+Alias types to create accessors tailored for the *host*, *device*, or *managed* memory spaces.
+
+----
+
+.. code:: cpp
+
+  template <typename ElementType,
+            typename Extents,
+            typename LayoutPolicy   = cuda::std::layout_right,
+            typename AccessorPolicy = cuda::std::default_accessor<_ElementType>>
+  using host_mdspan = cuda::std::mdspan<ElementType, Extents, LayoutPolicy, host_accessor<AccessorPolicy>>;
+
+  template <typename ElementType,
+            typename Extents,
+            typename LayoutPolicy   = cuda::std::layout_right,
+            typename AccessorPolicy = cuda::std::default_accessor<_ElementType>>
+  using device_mdspan = cuda::std::mdspan<ElementType, Extents, LayoutPolicy, device_accessor<AccessorPolicy>>;
+
+  template <typename ElementType,
+            typename Extents,
+            typename LayoutPolicy   = cuda::std::layout_right,
+            typename AccessorPolicy = cuda::std::default_accessor<_ElementType>>
+  using managed_mdspan = cuda::std::mdspan<ElementType, Extents, LayoutPolicy, managed_accessor<AccessorPolicy>>;
+
+Alias types to create ``mdspan`` with *host*, *device*, or *managed* accessors.
+
+----
+
+.. code:: cpp
+
+  template <typename T>
+  inline constexpr bool is_host_accessor_v = /* true if T is a host accessor, false otherwise */
+
+  template <typename T>
+  inline constexpr bool is_device_accessor_v = /* true if T is a device accessor, false otherwise */
+
+  template <typename T>
+  inline constexpr bool is_managed_accessor_v = /* true if T is a managed accessor, false otherwise */
+
+  template <typename T>
+  inline constexpr bool is_host_accessible_v = /* true if T is a mdspan/accessor accessible from the host, false otherwise */
+
+  template <typename T>
+  inline constexpr bool is_device_accessible_v = /* true if T is a mdspan/accessor accessible from the device, false otherwise */
+
+----
+
+Features
+--------
+
+**Memory spaces**
+
+*Host*, *device*, and *managed* ``mdspan`` can be created and "sliced" (``cuda::std::submdspan``) on any memory space. However, access to a specific memory space is restricted to the respective *accessor* type.
+
++----------------------------------+------------------+-------------------+
+| ``mdspan`` / memory space access | Host memory      | Device memory     |
++==================================+==================+===================+
+| ``host_mdspan``                  | Allowed          | *Compile error*   |
++----------------------------------+------------------+-------------------+
+| ``device_mdspan``                | *Compile error*  | Allowed           |
++----------------------------------+------------------+-------------------+
+| ``managed_mdspan``               | Allowed *****    | Allowed *****     |
++----------------------------------+------------------+-------------------+
+
+***** the validity of the *managed* memory space is checked at run-time in debug mode (host-side).
+
+**Conversions**
+
++-----------------------------+------------------+-------------------+---------------------+
+|                             | ``host_mdspan``  | ``device_mdspan`` | ``managed_mdspan``  |
++=============================+==================+===================+=====================+
+| ``host_mdspan``             | Allowed          | *Compile error*   | *Compile error*     |
++-----------------------------+------------------+-------------------+---------------------+
+| ``device_mdspan``           | *Compile error*  | Allowed           | *Compile error*     |
++-----------------------------+------------------+-------------------+---------------------+
+| ``managed_mdspan``          | Allowed          | Allowed           | Allowed             |
++-----------------------------+------------------+-------------------+---------------------+
+| Other mdspan                | Allowed          | Allowed           | Allowed             |
++-----------------------------+------------------+-------------------+---------------------+
+
+*Note:* the conversion is ``explicit`` if the base accessor is not directly convertible.
+
+Example 1
+---------
+
+``cuda::host_mdspan`` and ``cuda::device_mdspan`` usage:
+
+.. code:: cpp
+
+    #include <cuda/mdspan>
+
+    using dim = cuda::std::dims<1>;
+
+    __global__ void kernel_d(cuda::device_mdspan<int, dim> md) {
+        md[0] = 0;
+    }
+    __global__ void kernel_h(cuda::host_mdspan<int, dim> md) {
+        // md[0] = 0;  // compile error
+    }
+
+    __host__ void host_function_h(cuda::host_mdspan<int, dim> md) {
+        md[0] = 0;
+    }
+    __host__ void host_function_d(cuda::device_mdspan<int, dim> md) {
+        // md[0] = 0;  // compile error
+    }
+    __host__ void host_function_m(cuda::managed_mdspan<int, dim> md) {
+        md[0] = 0;
+    }
+
+    int main() {
+        int* d_ptr;
+        cudaMalloc(&d_ptr, 4 * sizeof(int));
+        int                 h_ptr[4];
+        cuda::host_mdspan   h_md{h_ptr};
+        cuda::device_mdspan d_md{d_ptr, 4};
+        kernel_d<<<1, 1>>>(d_md);    // ok
+        // kernel_d<<<1, 1>>>(h_md); // compile error
+        host_function_h(h_md);       // ok
+        host_function_d(h_md);       // compile error
+        // host_function_m(h_md);    // compile error
+        cudaFree(d_ptr);
+    }
+
+`See example 1 on Godbolt 🔗 <https://godbolt.org/z/fezxsbjaq>`_
+
+Example 2
+---------
+
+``cuda::managed_mdspan`` usage:
+
+.. code:: cpp
+
+    #include <cuda/mdspan>
+
+    using dim = cuda::std::dims<1>;
+
+    __global__ void kernel_d(cuda::device_mdspan<int, dim> md) {
+        md[0] = 0;
+    }
+
+    __host__ void host_function_h(cuda::host_mdspan<int, dim> md) {
+        md[0] = 0;
+    }
+
+    int main() {
+        int* m_ptr;
+        cudaMallocManaged(&m_ptr, 4 * sizeof(int));
+        cuda::managed_mdspan m_md{m_ptr, 4};
+        kernel_d<<<1, 1>>>(m_md); // ok
+        host_function_h(m_md);    // ok
+
+        cuda::managed_mdspan m_md2{d_ptr, 4};
+        m_md2[0]; // run-time error
+        cudaFree(d_ptr);
+    }
+
+`See example 2 on Godbolt 🔗 <https://godbolt.org/z/Kj39Pe4vP>`_
+
+
+Example 3
+---------
+
+Conversion from other accessors:
+
+.. code:: cpp
+
+    #include <cuda/mdspan>
+
+    using dim = cuda::std::dims<1>;
+
+    int main() {
+        using cuda::std::layout_right;
+        using cuda::std::aligned_accessor;
+        int               h_ptr[4];
+        cuda::std::mdspan md{h_ptr};
+        cuda::host_mdspan h_md = md; // ok
+
+        cuda::std::mdspan<int, dim, layout_right, aligned_accessor<int, 8>> md_a{h_ptr, 4};
+        // cuda::host_mdspan h_md = md_a; // compile-error
+        cuda::host_mdspan    h_md{md_a};  // ok
+    }
+
+`See example 3 on Godbolt 🔗 <https://godbolt.org/z/7dq7vcTWP>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/mdspan/layout_stride_relaxed.rst b/cccl_upstream/docs/libcudacxx/extended_api/mdspan/layout_stride_relaxed.rst
new file mode 100644
index 00000000..eda3df0a
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/mdspan/layout_stride_relaxed.rst
@@ -0,0 +1,440 @@
+.. _libcudacxx-extended-api-mdspan-layout-stride-relaxed:
+
+``layout_stride_relaxed``
+=========================
+
+Defined in the ``<cuda/mdspan>`` header.
+
+``layout_stride_relaxed`` is a *LayoutMappingPolicy* which provides a layout mapping where the strides are user-defined and can be negative or zero.
+
+Unlike ``cuda::std::layout_stride``, this layout allows:
+
+- **Negative strides** for reverse iteration.
+- **Zero strides** for broadcasting.
+- **A base offset** to accommodate negative strides.
+- **Compile-time stride values** for static arrays.
+
+.. note::
+
+    This layout is NOT always *unique*, *exhaustive*, or *strided* in the C++ standard sense.
+
+----
+
+Synopsis
+--------
+
+.. code:: cpp
+
+    namespace cuda {
+
+    // Tag value for dynamic stride (analogous to dynamic_extent)
+    inline constexpr ptrdiff_t dynamic_stride = /* implementation-defined */;
+
+    // Strides class template
+    template <class OffsetType, ptrdiff_t... Strides>
+    class strides;
+
+    // Alias for all-dynamic strides
+    template <class OffsetType, size_t Rank>
+    using dstrides = strides<OffsetType, /* Rank dynamic_stride values */>;
+
+    // Alias for steps (synonym for dstrides)
+    template <size_t Rank, class OffsetType = ptrdiff_t>
+    using steps = dstrides<OffsetType, Rank>;
+
+    // Layout policy
+    struct layout_stride_relaxed {
+        template <class Extents,
+                  class Stride = dstrides<make_signed_t<typename Extents::index_type>,
+                                          Extents::rank()>,
+                  class OffsetType = ptrdiff_t>
+        class mapping;
+    };
+
+    } // namespace cuda
+
+----
+
+``strides``
+-----------
+
+Class template to describe the strides of a multi-dimensional array layout. Similar to ``extents``, but for strides. Supports both *static* (compile-time known) and *dynamic* (runtime) stride values.
+
+.. code:: cpp
+
+    // Strides class template
+    template <class OffsetType, ptrdiff_t... Strides>
+    class strides;
+
+    // Alias for all-dynamic strides
+    template <class OffsetType, size_t Rank>
+    using dstrides = strides<OffsetType, /* Rank dynamic_stride values */>;
+
+    // Alias for steps (synonym for dstrides)
+    template <size_t Rank, class OffsetType = ptrdiff_t>
+    using steps = dstrides<OffsetType, Rank>;
+
+**Template Parameters**
+
+- ``OffsetType``: A signed integer type for stride values (supports negative strides) or an *integer-constant-like* type.
+- ``Strides...``: The stride values, where ``dynamic_stride`` indicates a runtime value.
+- ``Rank``: The number of dimensions.
+
+.. list-table::
+    :widths: 40 60
+    :header-rows: 1
+
+    * - **Member Types**
+      - Definition
+    * - ``offset_type``
+      - ``OffsetType``
+    * - ``size_type``
+      - ``cuda::std::make_unsigned_t<offset_type>``
+    * - ``rank_type``
+      - ``cuda::std::size_t``
+
+.. list-table::
+    :widths: 40 60
+    :header-rows: 1
+
+    * - **Static Member Functions**
+      - Description
+    * - ``rank()``
+      - Returns the number of dimensions.
+    * - ``rank_dynamic()``
+      - Returns the number of dynamic strides.
+    * - ``static_stride(rank_type r)``
+      - Returns the static stride at dimension ``r``, or ``dynamic_stride`` if dynamic.
+
+.. list-table::
+    :widths: 40 60
+    :header-rows: 1
+
+    * - **Member Functions**
+      - Description
+    * - ``stride(rank_type r)``
+      - Returns the stride at dimension ``r``.
+
+.. list-table::
+    :widths: 40 60
+    :header-rows: 1
+
+    * - **Non-member Functions**
+      - Description
+    * - ``operator==``
+      - ``true`` if ranks are equal and all extent and stride values compare equal; ``false`` otherwise.
+    * - ``operator!=``
+      - ``false`` if ranks are equal and all extent and stride values compare equal; ``true`` otherwise.
+
+**Constructors**
+
+.. code:: cpp
+
+    // (1) default constructor
+    constexpr strides() noexcept = default;
+
+    // (2) constructor from values
+    template <class... OtherIndexTypes>
+    constexpr explicit strides(OtherIndexTypes... values) noexcept;
+
+    // (3) constructor from span
+    template <class OtherIndexType, size_t Size>
+    constexpr explicit(/*see below*/) strides(cuda::std::span<OtherIndexType, Size> strs) noexcept;
+
+    // (4) constructor from array
+    template <class OtherIndexType, size_t Size>
+    constexpr explicit(/*see below*/) strides(const cuda::std::array<OtherIndexType, Size>& strs) noexcept;
+
+    // (5) constructor from strides
+    template <class OtherIndexType, ptrdiff_t... OtherStrides>
+    constexpr explicit(/*see below*/) strides(const strides<OtherIndexType, OtherStrides...>& other) noexcept;
+
+- **(1)** Default constructor. Value-initializes all dynamic strides to zero.
+- **(2)** Initializes the strides with the provided values.
+
+  - *Constraints*:
+
+    - ``sizeof...(OtherIndexTypes)`` equals ``rank()`` or ``rank_dynamic()``.
+    - ``OtherIndexTypes...`` is convertible and nothrow constructible to ``offset_type``.
+
+  - *Preconditions*:
+
+    - Each value is representable as ``offset_type``.
+
+- **(3)**, **(4)** Initializes the strides from ``span`` or ``array`` values.
+
+  - *Constraints*:
+
+    - ``Size`` equals ``rank_dynamic()`` (implicit) or ``Size`` equals ``rank()`` and ``Size != rank_dynamic()`` (explicit).
+    - ``OtherIndexType`` is convertible and nothrow constructible to ``offset_type``.
+
+  - *Preconditions*:
+
+    - Each value is representable as ``offset_type``.
+
+- **(5)** Initializes the strides from another ``strides`` object.
+
+  - The constructor is ``explicit`` if any static stride in ``Strides...`` corresponds to a ``dynamic_stride`` in ``OtherStrides...``.
+  - *Constraints*:
+
+    - ``sizeof...(OtherStrides)`` equals ``sizeof...(Strides)``.
+    - For each dimension, either stride is ``dynamic_stride``, or both strides are equal.
+
+  - *Preconditions*:
+
+    - Static strides must match their compile-time values.
+    - Each dynamic stride value is representable as ``offset_type``.
+
+``layout_stride_relaxed::mapping``
+----------------------------------
+
+The class template ``layout_stride_relaxed::mapping`` controls how multidimensional indices are mapped with user-defined strides (including negative and zero strides) and an optional offset to a one-dimensional value representing the offset.
+
+.. code:: cpp
+
+    template <class Extents,
+              class Stride = dstrides<make_signed_t<typename Extents::index_type>, Extents::rank()>>
+    class layout_stride_relaxed::mapping;
+
+**Template Parameters**
+
+- ``Extents``: Specifies number of dimensions, their sizes, and which are known at compile time. Must be a specialization of ``cuda::std::extents``.
+- ``Stride``: Specifies the strides for each dimension. Must be a specialization of ``cuda::strides``. Defaults to all-dynamic strides.
+
+**Constraints**
+
+- ``Extents`` must be a specialization of ``cuda::std::extents``.
+- ``Extents::rank()`` must equal ``Stride::rank()``.
+
+.. list-table::
+    :widths: 40 60
+    :header-rows: 1
+
+    * - **Member Types**
+      - Definition
+    * - ``extents_type``
+      - ``Extents``
+    * - ``strides_type``
+      - ``Stride``
+    * - ``index_type``
+      - ``extents_type::index_type``
+    * - ``size_type``
+      - ``extents_type::size_type``
+    * - ``rank_type``
+      - ``extents_type::rank_type``
+    * - ``offset_type``
+      - ``strides_type::offset_type``
+    * - ``layout_type``
+      - ``layout_stride_relaxed``
+
+.. list-table::
+    :widths: 40 60
+    :header-rows: 1
+
+    * - **Static Member Functions**
+      - Description
+    * - ``is_always_unique()``
+      - Returns ``false``. Uniqueness is not guaranteed due to zero/negative strides.
+    * - ``is_always_exhaustive()``
+      - Returns ``false``. Exhaustiveness is not guaranteed due to zero/negative strides.
+    * - ``is_always_strided()``
+      - Returns ``false`` if offset is non-zero (compile-time evaluation). Standard strided behavior is not guaranteed due to offset.
+
+.. list-table::
+    :widths: 40 60
+    :header-rows: 1
+
+    * - **Member Functions**
+      - Description
+    * - ``extents()``
+      - Returns the extents object.
+    * - ``strides()``
+      - Returns the strides object.
+    * - ``offset()``
+      - Returns the base offset (nonnegative).
+    * - ``required_span_size()``
+      - Returns the required span size to cover all valid indices.
+    * - ``operator()(Indices... indices)``
+      - Maps multidimensional indices to a linear index.
+    * - ``is_unique()``
+      - Returns ``false`` (conservative).
+    * - ``is_exhaustive()``
+      - Returns ``false`` (conservative).
+    * - ``is_strided()``
+      - Returns ``true`` if offset is zero, ``false`` otherwise.
+    * - ``stride(rank_type r)``
+      - Returns the stride along dimension ``r``.
+
+.. list-table::
+    :widths: 40 60
+    :header-rows: 1
+
+    * - **Non-member Functions**
+      - Description
+    * - ``operator==``
+      - ``true`` if extents, strides, and offsets are equal; ``false`` otherwise.
+    * - ``operator!=``
+      - ``false`` if extents, strides, and offsets are equal; ``true`` otherwise.
+
+**Constructors**
+
+.. code:: cpp
+
+    // (1) default constructor
+    constexpr mapping() noexcept;
+
+    // (2) copy constructor
+    constexpr mapping(const mapping&) noexcept = default;
+
+    // (3) constructor from strides
+    constexpr mapping(const extents_type& ext,
+                      const strides_type& strides,
+                      offset_type offset = 0) noexcept;
+
+    // (4) converting constructor from mapping
+    template <class OtherMapping>
+    constexpr explicit(/*see below*/) mapping(const OtherMapping& other) noexcept;
+
+- **(1)** Default constructs the mapping, delegating to a ``layout_right`` mapping with default extents.
+- **(2)** Copy constructor.
+- **(3)** Direct-non-list-initializes the extents, strides, and offset with the provided arguments.
+
+  - *Preconditions*:
+
+    - ``offset`` is nonnegative.
+    - ``required_span_size()`` is representable as ``index_type``.
+
+- **(4)** Constructs the mapping by copying extents and strides from ``other``. For ``layout_stride_relaxed`` sources, also copies the offset.
+
+  - The constructor is ``explicit`` if ``OtherMapping::extents_type`` is not convertible to ``extents_type``, or (for non-``layout_stride_relaxed`` mappings) the source is not a ``layout_left``, ``layout_right``, or ``layout_stride`` mapping.
+  - *Constraints*:
+
+    - ``OtherMapping`` is a layout mapping.
+    - ``extents_type`` is constructible from ``OtherMapping::extents_type``.
+
+  - *Preconditions*:
+
+    - ``offset`` is non-negative.
+    - ``required_span_size()`` is representable as ``index_type``.
+
+Examples
+--------
+
+**Compile-time strides (column-major layout)**
+
+.. code:: cpp
+
+    #include <cuda/mdspan>
+    #include <cassert>
+
+    int main() {
+        // 3x4 matrix in column-major order
+        int data[12] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12};
+
+        using extents_t = cuda::std::extents<int, 3, 4>;
+        // Compile-time strides: stride 1 for rows, stride 3 for columns
+        using strides_t = cuda::strides<int, 1, 3>;
+        using mapping_t = cuda::layout_stride_relaxed::mapping<extents_t, strides_t>;
+
+        mapping_t mapping(extents_t{}, strides_t{});
+
+        // Column-major: consecutive elements in same column are adjacent
+        assert(mapping(0, 0) == 0);
+        assert(mapping(1, 0) == 1);
+        assert(mapping(2, 0) == 2);
+        assert(mapping(0, 1) == 3);
+    }
+
+**Negative strides (reverse iteration)**
+
+.. code:: cpp
+
+    #include <cuda/mdspan>
+    #include <cassert>
+
+    int main() {
+        int data[] = {1, 2, 3, 4, 5};
+        // Create a reversed view using negative stride
+        // Offset points to the last element, stride is -1
+        using extents_t = cuda::std::extents<int, 5>;
+        using mapping_t = cuda::layout_stride_relaxed::mapping<extents_t>;
+        using stride_t  = mapping_t::strides_type;
+        mapping_t mapping(extents_t{}, stride_t(-1), 4); // offset=4, stride=-1
+        // Access pattern: mapping(i) = 4 + i * (-1) = 4 - i
+        assert(mapping(0) == 4);
+        assert(mapping(1) == 3);
+        assert(mapping(2) == 2);
+        assert(mapping(3) == 1);
+        assert(mapping(4) == 0);
+    }
+
+**Negative strides (reverse iteration) with constant offset**
+
+.. code:: cpp
+
+    #include <cuda/mdspan>
+    #include <cassert>
+
+    int main() {
+        int data[] = {1, 2, 3, 4, 5};
+        // Create a reversed view using negative stride
+        // Offset points to the last element, stride is -1
+        using extents_t = cuda::std::extents<int, 5>;
+        using offset_t = cuda::std::integral_constant<int, 4>;
+        using strides_t = steps<extents_t::rank()>;
+        using mapping_t = cuda::layout_stride_relaxed::mapping<extents_t, strides_t, offset_t>;
+        mapping_t mapping(extents_t{}, strides_t(-1));
+        assert(mapping(0) == 4);
+        assert(mapping(1) == 3);
+        assert(mapping(2) == 2);
+        assert(mapping(3) == 1);
+        assert(mapping(4) == 0);
+    }
+
+**Zero strides (broadcasting)**
+
+.. code:: cpp
+
+    #include <cuda/mdspan>
+    #include <cassert>
+
+    int main() {
+        int scalar = 42;
+        // Create a broadcast view: single value appears at all indices
+        using extents_t = cuda::std::extents<int, 4, 4>;
+        using mapping_t = cuda::layout_stride_relaxed::mapping<extents_t>;
+        using stride_t  = mapping_t::strides_type;
+
+        mapping_t mapping(extents_t{}, stride_t(0, 0)); // zero strides
+        // All indices map to offset 0
+        assert(mapping(0, 0) == 0);
+        assert(mapping(1, 2) == 0);
+        assert(mapping(3, 3) == 0);
+    }
+
+**Mixed strides**
+
+.. code:: cpp
+
+    #include <cuda/mdspan>
+    #include <cassert>
+
+    int main() {
+        // 2D array with column-major layout but reversed rows
+        // Data: row 0 at end, row 1 in middle, row 2 at start
+        int data[3][4] = {
+            {1, 2,  3,  4},
+            {5, 6,  7,  8},
+            {9, 10, 11, 12}
+        };
+        using extents_t = cuda::std::extents<int, 3, 4>;
+        using mapping_t = cuda::layout_stride_relaxed::mapping<extents_t>;
+        using stride_t  = mapping_t::strides_type;
+        // Reverse rows: stride of -4 in row dimension, +1 in column
+        // Offset to start at last row
+        mapping_t mapping(extents_t{}, stride_t(-4, 1), 8);
+        // Access gives reversed row order
+        assert(mapping(0, 0) == 8); // data[2][0]
+        assert(mapping(1, 0) == 4); // data[1][0]
+        assert(mapping(2, 0) == 0); // data[0][0]
+    }
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/mdspan/mdspan_to_dlpack.rst b/cccl_upstream/docs/libcudacxx/extended_api/mdspan/mdspan_to_dlpack.rst
new file mode 100644
index 00000000..3e4663f1
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/mdspan/mdspan_to_dlpack.rst
@@ -0,0 +1,167 @@
+.. _libcudacxx-extended-api-mdspan-mdspan-to-dlpack:
+
+``mdspan`` to DLPack
+====================
+
+This functionality provides a conversion from ``cuda::host_mdspan``, ``cuda::device_mdspan``, and ``cuda::managed_mdspan`` to `DLPack <https://dmlc.github.io/dlpack/latest/>`__ ``DLTensor`` view.
+
+Defined in the ``<cuda/mdspan>`` header.
+
+Conversion functions
+--------------------
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T, typename Extents, typename Layout, typename Accessor>
+   [[nodiscard]] /*dlpack_tensor*/<Extents::rank()>
+   to_dlpack_tensor(const host_mdspan<T, Extents, Layout, Accessor>& mdspan);
+
+   template <typename T, typename Extents, typename Layout, typename Accessor>
+   [[nodiscard]] /*dlpack_tensor*/<Extents::rank()>
+   to_dlpack_tensor(const device_mdspan<T, Extents, Layout, Accessor>& mdspan);
+
+   template <typename T, typename Extents, typename Layout, typename Accessor>
+   [[nodiscard]] /*dlpack_tensor*/<Extents::rank()>
+   to_dlpack_tensor(const managed_mdspan<T, Extents, Layout, Accessor>& mdspan);
+
+   } // namespace cuda
+
+Types
+-----
+
+``/*dlpack_tensor*/`` is a internal helper class that stores a ``DLTensor`` and owns the backing storage for its ``shape`` and ``strides`` pointers. The class does not use any heap allocation.
+
+.. code:: cuda
+
+  namespace cuda {
+
+  template <size_t Rank>
+  struct /*dlpack_tensor*/ {
+    // cuda::std::array<int64_t, Rank> shape;
+    // cuda::std::array<int64_t, Rank> strides;
+
+    DLTensor get() & const noexcept [[lifetimebound]];
+
+    DLTensor get() && = delete;
+  };
+
+  } // namespace cuda
+
+``/*dlpack_tensor*/`` stores a ``DLTensor`` and owns the backing storage for its ``shape`` and ``strides`` pointers. The class does not use any heap allocation.
+
+.. note:: **Lifetime**
+
+  The ``DLTensor`` associated with ``/*dlpack_tensor*/`` must not outlive the wrapper. If the wrapper is destroyed, the returned ``DLTensor::shape`` and ``DLTensor::strides`` pointers will dangle.
+
+.. note:: **Const-correctness**
+
+  ``DLTensor::data`` points at ``mdspan.data_handle()`` (or is ``nullptr`` if ``mdspan.size() == 0``). If ``T`` is ``const``, the pointer is ``const_cast``'d because ``DLTensor::data`` is unqualified.
+
+Semantics
+---------
+
+The conversion produces a non-owning DLPack view of the ``mdspan`` data and metadata:
+
+- ``DLTensor::ndim`` is ``mdspan.rank()``.
+- For rank > 0, ``DLTensor::shape[i]`` is ``mdspan.extent(i)``.
+- For rank > 0, ``DLTensor::strides[i]`` is ``mdspan.stride(i)``.
+- ``DLTensor::byte_offset`` is always ``0``.
+- ``DLTensor::device`` is:
+
+  - ``{kDLCPU, 0}`` for ``cuda::host_mdspan``
+  - ``{kDLCUDA, /*device_id*/}`` for ``cuda::device_mdspan``
+  - ``{kDLCUDAManaged, 0}`` for ``cuda::managed_mdspan``
+
+Element types are mapped to ``DLDataType`` according to the DLPack conventions, including:
+
+- ``bool``.
+- Signed and unsigned integers.
+- IEEE-754 Floating-point and extended precision floating-point, including ``__half``, ``__nv_bfloat16``, ``__float128``, FP8, FP6, FP4 when available.
+- Complex: ``cuda::std::complex<__half>``, ``cuda::std::complex<float>``, and ``cuda::std::complex<double>``.
+- `CUDA built-in vector types <https://docs.nvidia.com/cuda/cuda-programming-guide/05-appendices/cpp-language-extensions.html#built-in-types>`__, such as ``int2``, ``float4``, etc.
+- Vector types for extended floating-point, such as ``__half2``, ``__nv_fp8x4_e4m3``, etc.
+
+Constraints
+-----------
+
+- The accessor ``data_handle_type`` must be a pointer type.
+
+Runtime errors
+--------------
+
+- If any ``extent(i)`` or ``stride(i)`` cannot be represented in ``int64_t``, the conversion raises an ``std::invalid_argument`` exception.
+
+Availability notes
+------------------
+
+- This API is available only when DLPack header is present, namely ``<dlpack/dlpack.h>`` is found in the include path.
+- This API can be disabled by defining ``CCCL_DISABLE_DLPACK`` before including any library headers. In this case, ``<dlpack/dlpack.h>`` will not be included.
+
+References
+----------
+
+- `DLPack C API <https://dmlc.github.io/dlpack/latest/c_api.html>`__ documentation.
+
+Example
+-------
+
+.. code:: cuda
+
+  #include <dlpack/dlpack.h>
+  #include <cuda/mdspan>
+  #include <cuda/std/cassert>
+  #include <cuda/std/cstdint>
+
+  int main() {
+    using extents_t = cuda::std::extents<size_t, 2, 3>;
+
+    int data[6] = {0, 1, 2, 3, 4, 5};
+    cuda::host_mdspan<int, extents_t> md{data, extents_t{}};
+
+    auto dl       = cuda::to_dlpack_tensor(md);
+    auto dltensor = dl.get();
+
+    // `dl` owns the shape/stride storage; `dltensor.data` is a non-owning pointer to `data`.
+    assert(dltensor.device.device_type == kDLCPU);
+    assert(dltensor.ndim == 2);
+    assert(dltensor.shape[0] == 2 && dltensor.shape[1] == 3);
+    assert(dltensor.strides[0] == 3 && dltensor.strides[1] == 1);
+    assert(dltensor.data == data);
+  }
+
+Examples of invalid usage:
+
+.. code:: cuda
+
+  #include <dlpack/dlpack.h>
+  #include <cuda/mdspan>
+  #include <cuda/std/cstdint>
+
+  void show_invalid_usage1() {
+    using extents_t = cuda::std::extents<size_t, 2, 3>;
+
+    int data[6] = {0, 1, 2, 3, 4, 5};
+    cuda::host_mdspan<int, extents_t> md{data, extents_t{}};
+
+    // WRONG: calling get() on a temporary is deleted to prevent dangling references.
+    // const DLTensor& dltensor = cuda::to_dlpack_tensor(md).get(); // compile error
+  }
+
+.. code:: cuda
+
+  #include <dlpack/dlpack.h>
+  #include <cuda/mdspan>
+  #include <cuda/std/cstdint>
+
+  int64_t* show_invalid_usage2() {
+    using extents_t = cuda::std::extents<size_t, 2, 3>;
+
+    int data[6] = {0, 1, 2, 3, 4, 5};
+    cuda::host_mdspan<int, extents_t> md{data, extents_t{}};
+
+    auto dl       = cuda::to_dlpack_tensor(md);
+    auto dltensor = dl.get();
+    return dltensor.shape; // WRONG: returns a dangling pointer
+  }
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/mdspan/restrict_accessor.rst b/cccl_upstream/docs/libcudacxx/extended_api/mdspan/restrict_accessor.rst
new file mode 100644
index 00000000..fd723391
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/mdspan/restrict_accessor.rst
@@ -0,0 +1,80 @@
+.. _libcudacxx-extended-api-mdspan-restrict-accessor:
+
+``restrict`` ``mdspan`` and ``accessor``
+========================================
+
+.. code:: cpp
+
+  template <typename Accessor>
+  using restrict_accessor;
+
+An alias type to create an accessor with the *restrict aliasing policy* starting from an existing accessor.
+
+More information related to the *restrict aliasing policy* can be found in the CUDA programming guide: `__restrict__ keyword <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#restrict>`_.
+
+----
+
+.. code:: cpp
+
+  template <typename ElementType,
+            typename Extents,
+            typename LayoutPolicy   = cuda::std::layout_right,
+            typename AccessorPolicy = cuda::std::default_accessor<_ElementType>>
+  using restrict_mdspan = cuda::std::mdspan<ElementType, Extents, LayoutPolicy, restrict_accessor<AccessorPolicy>>;
+
+An alias type to create an ``mdspan`` with a *restrict aliasing policy* accessor.
+
+----
+
+Traits:
+
+.. code:: cpp
+
+    template <typename T>
+    inline constexpr bool is_restrict_accessor_v = /*true if T is a restrict accessor, false otherwise*/;
+
+    template <typename T>
+    inline constexpr bool is_restrict_mdspan_v = /*true if T is a restrict mdspan, false otherwise*/;
+
+----
+
+**Constraints**:
+
+- Accessor ``data_handle_type`` must be a pointer type.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/mdspan>
+
+    using restrict_mdspan = cuda::restrict_mdspan<int, cuda::std::dims<1>>;
+
+    __host__ __device__ void
+    compute(restrict_mdspan a, restrict_mdspan b, restrict_mdspan c) {
+        c[0] = a[0] * b[0];
+        c[1] = a[0] * b[0];
+        c[2] = a[0] * b[0] * a[1];
+        c[3] = a[0] * a[1];
+        c[4] = a[0] * b[0];
+        c[5] = b[0];
+    }
+
+    int main() {
+        using  dim      = cuda::std::dims<1>;
+        using  mdspan   = cuda::std::mdspan<int, dim>;
+        int    arrayA[] = {1, 2};
+        int    arrayB[] = {5};
+        int    arrayC[] = {9, 10, 11, 12, 13, 14};
+        mdspan mdA{arrayA, dim{1}};
+        mdspan mdB{arrayB, dim{5}};
+        mdspan mdC{arrayC, dim{6}};
+        compute(mdA, mdB, mdC);
+
+        using restrict_aligned_accesor = cuda::std::restrict_accessor<cuda::std::aligned_accessor<int, 8>>;
+        using restrict_aligned_mdspan  = cuda::std::mdspan<int, dim, layout_right, restrict_aligned_accesor>;
+        restrict_aligned_mdspan mdD{mdC};
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/Wjco996z8>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/mdspan/shared_memory_accessor.rst b/cccl_upstream/docs/libcudacxx/extended_api/mdspan/shared_memory_accessor.rst
new file mode 100644
index 00000000..6721d1e1
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/mdspan/shared_memory_accessor.rst
@@ -0,0 +1,90 @@
+.. _libcudacxx-extended-api-mdspan-shared-memory-accessor:
+
+``shared_memory`` ``mdspan`` and ``accessor``
+=============================================
+
+``shared_memory`` ``mdspan`` and ``accessor`` allow to express multi-dimensional views of the CUDA shared memory space and provide additional safety checks and performance optimizations.
+
+Defined in the ``<cuda/mdspan>`` header.
+
+Types and Traits
+----------------
+
+.. code:: cpp
+
+    namespace cuda {
+
+    template <typename AccessorPolicy>
+    using shared_memory_accessor;
+
+    template <typename ElementType,
+              typename Extents,
+              typename LayoutPolicy   = cuda::std::layout_right,
+              typename AccessorPolicy = cuda::shared_memory_accessor<ElementType>>
+    class shared_memory_mdspan;
+
+    } // namespace cuda
+
+``mdspan`` type and accessor tailored for the *shared* memory space.
+
+----
+
+.. code:: cpp
+
+    namespace cuda {
+
+    template <typename T>
+    inline constexpr bool is_shared_memory_accessor_v = /* true if T is a shared_memory_accessor, false otherwise */;
+
+    template <typename T>
+    inline constexpr bool is_shared_memory_mdspan_v = /* true if T is a shared_memory_mdspan, false otherwise */;
+
+    } // namespace cuda
+
+Features
+--------
+
+**Constraints**
+
+- Accessor ``data_handle_type`` must be a pointer type.
+
+**Preconditions**
+
+- Accessing elements through a ``shared_memory_accessor`` is only allowed in device code.
+- The underlying pointer must be in the *shared* memory space.
+- Access offset must be within the maximum possible shared memory allocation size.
+
+**Performance considerations**
+
+- The functionality guarantees that the accesses use shared memory instructions (``STS/LDS``) rather than generic memory instructions.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/mdspan>
+    #include <cstdio>
+
+    __global__ void kernel() {
+        extern __shared__ int shmem[];
+
+        // Create a shared_memory_mdspan over the dynamic shared memory
+        cuda::shared_memory_mdspan md(shmem, cuda:std::dims<2>{32, 32});
+
+        if (threadIdx.x < 32) {
+             md[threadIdx.x][threadIdx.x] = threadIdx.x; // write on the diagonal
+        }
+        __syncthreads();
+
+        if (threadIdx.x == 0) {
+            printf("md[5][5] = %d\n", md[5][5]); // read from the diagonal
+        }
+    }
+
+    int main() {
+        kernel<<<1, 32, 32 * 32 * sizeof(int)>>>();
+        cudaDeviceSynchronize();
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/sojGnKoY9>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory.rst
new file mode 100644
index 00000000..f4c21558
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory.rst
@@ -0,0 +1,84 @@
+.. _libcudacxx-extended-api-memory:
+
+Memory
+======
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   memory/align_down
+   memory/align_up
+   memory/aligned_size
+   memory/discard_memory
+   memory/get_device_address
+   memory/is_address_from
+   memory/is_aligned
+   memory/ptr_rebind
+   memory/ptr_in_range
+   memory/ranges_overlap
+   memory/is_pointer_accessible
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`aligned_size_t <libcudacxx-extended-api-memory-aligned-size>`
+     - Defines an extent of bytes with a statically defined alignment.
+     - libcu++ 1.2.0 / CCCL 2.0.0 (in ``<cuda/memory>`` since CCCL 3.1.0)
+     - CUDA 11.1
+
+   * - :ref:`discard_memory <libcudacxx-extended-api-memory-discard-memory>`
+     - Writes indeterminate values to memory
+     - libcu++ 1.6.0 / CCCL 2.0.0 (in ``<cuda/memory>`` since CCCL 3.1.0)
+     - CUDA 11.5
+
+   * - :ref:`get_device_address <libcudacxx-extended-api-memory-get-device-address>`
+     - Returns a valid address to a device object
+     - CCCL 2.8.0 (in ``<cuda/memory>`` since CCCL 3.1.0)
+     - CUDA 12.9
+
+   * - :ref:`is_address_from and is_object_from <libcudacxx-extended-api-memory-is_address_from>`
+     - Check if a pointer or object is from a specific address space
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`is_aligned <libcudacxx-extended-api-memory-is_aligned>`
+     - Check if a pointer is aligned
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`align_up <libcudacxx-extended-api-memory-align_up>`
+     - Align up a pointer to the specified alignment
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`align_down <libcudacxx-extended-api-memory-align_down>`
+     - Align down a pointer to the specified alignment
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`ptr_rebind <libcudacxx-extended-api-memory-ptr_rebind>`
+     - Rebind a pointer to a different type
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`ptr_in_range <libcudacxx-extended-api-memory-ptr_in_range>`
+     - Check if a pointer is in a range
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`ranges_overlap <libcudacxx-extended-api-memory-ranges_overlap>`
+     - Check if two ranges overlap
+     - CCCL 3.2.0
+     - CUDA 13.2
+
+   * - :ref:`is_host_accessible <libcudacxx-extended-api-memory-is_pointer_accessible>`, :ref:`is_device_accessible <libcudacxx-extended-api-memory-is_pointer_accessible>`, :ref:`is_managed <libcudacxx-extended-api-memory-is_pointer_accessible>`
+     - Check if a pointer is accessible from the host, device, or managed memory
+     - CCCL 3.2.0
+     - CUDA 13.2
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory/align_down.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory/align_down.rst
new file mode 100644
index 00000000..a8a44c57
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory/align_down.rst
@@ -0,0 +1,63 @@
+.. _libcudacxx-extended-api-memory-align_down:
+
+``cuda::align_down``
+====================
+
+Defined in the header ``<cuda/memory>``.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T>
+   [[nodiscard]] __host__ __device__ inline
+   T* align_down(T* ptr, size_t alignment) noexcept;
+
+   } // namespace cuda
+
+The function returns the original pointer or closest pointer smaller than ``ptr`` that is aligned to the specified alignment  :math:`floor\left(\frac{ptr}{alignment}\right) * alignment`.
+
+**Parameters**
+
+- ``ptr``: The pointer.
+- ``alignment``: The alignment.
+
+**Return value**
+
+- The original pointer or closest pointer smaller than ``ptr`` that is aligned to the specified alignment.
+
+**Constraints**
+
+- ``alignment`` must be a power of two.
+- ``alignment >= alignof(T)``.
+- ``ptr`` is aligned to ``alignof(T)``.
+
+**Performance considerations**
+
+- The function is optimized for compile-time values of ``alignment``.
+- The function does not perform any operations if ``alignment == alignof(T)``.
+- The function is translated to a single ``LOP3.LUT`` instruction for other values of ``alignment``.
+- The returned pointer is decorated with ``__builtin_assume_aligned`` to help the compiler generate better code.
+- The returned pointer maintains the same memory space, for example shared memory, as the input pointer.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/memory>
+
+    __global__ void kernel(int* ptr) {
+        auto ptr_align16 = cuda::align_down(ptr, 16);
+        reinterpret_cast<int4*>(ptr_align16)[0] = int4{1, 2, 3, 4};
+    }
+
+    int main() {
+        int* ptr;
+        cudaMalloc(&ptr, 100 * sizeof(int));
+        kernel<<<1, 1>>>(ptr + 20);
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/6acdx3KhY>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory/align_up.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory/align_up.rst
new file mode 100644
index 00000000..69b41881
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory/align_up.rst
@@ -0,0 +1,63 @@
+.. _libcudacxx-extended-api-memory-align_up:
+
+``cuda::align_up``
+==================
+
+Defined in the header ``<cuda/memory>``.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T>
+   [[nodiscard]] __host__ __device__ inline
+   T* align_up(T* ptr, size_t alignment) noexcept;
+
+   } // namespace cuda
+
+The function returns the original pointer or closest pointer larger than ``ptr`` that is aligned to the specified alignment :math:`ceil\left(\frac{ptr}{alignment}\right) * alignment`.
+
+**Parameters**
+
+- ``ptr``: The pointer.
+- ``alignment``: The alignment.
+
+**Return value**
+
+- The original pointer or closest pointer larger than ``ptr`` that is aligned to the specified alignment.
+
+**Constraints**
+
+- ``alignment`` must be a power of two.
+- ``alignment >= alignof(T)``.
+- ``ptr`` is aligned to ``alignof(T)``.
+
+**Performance considerations**
+
+- The function is optimized for compile-time values of ``alignment``.
+- The function does not perform any operations if ``alignment == alignof(T)``.
+- The function is translated to ``LOP3.LUT`` + ``IADD.64`` instructions for other values of ``alignment``.
+- The returned pointer is decorated with ``__builtin_assume_aligned`` to help the compiler generate better code.
+- The returned pointer maintains the same memory space, for example shared memory, as the input pointer.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/memory>
+
+    __global__ void kernel(int* ptr) {
+        auto ptr_align16 = cuda::align_up(ptr, 16);
+        reinterpret_cast<int4*>(ptr_align16)[0] = int4{1, 2, 3, 4};
+    }
+
+    int main() {
+        int* ptr;
+        cudaMalloc(&ptr, 100 * sizeof(int));
+        kernel<<<1, 1>>>(ptr);
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/d8e5KETeE>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory/aligned_size.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory/aligned_size.rst
new file mode 100644
index 00000000..1b8c65d6
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory/aligned_size.rst
@@ -0,0 +1,91 @@
+.. _libcudacxx-extended-api-memory-aligned-size:
+
+``cuda::aligned_size_t``
+========================
+
+Defined in headers ``<cuda/memory>``, ``<cuda/barrier>`` and ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::std::size_t Alignment>
+   struct cuda::aligned_size_t {
+     static constexpr cuda::std::size_t align = Align;
+     cuda::std::size_t value;
+     __host__ __device__ explicit constexpr aligned_size(cuda::std::size_t size);
+     __host__ __device__ constexpr operator cuda::std::size_t();
+   };
+
+The class template ``cuda::aligned_size_t`` is a *shape* representing an extent of bytes with a statically
+defined (address and size) alignment.
+
+*Preconditions*:
+
+-  The *address* of the extent of bytes must be aligned to an ``Alignment`` alignment boundary.
+-  The *size* of the extent of bytes must be a multiple of the ``Alignment``.
+
+Template Parameters
+-------------------
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``Alignment``
+     - The address and size alignment of the byte extent.
+
+Data Members
+------------
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``align``
+     - The alignment of the byte extent.
+   * - ``value``
+     - The size of the byte extent.
+
+Member Functions
+----------------
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``(constructor)``
+     - Constructs an *aligned size*. If the ``size`` is not a multiple of ``Alignment`` the behavior is undefined.
+   * - ``(destructor)``
+     - Trivial implicit destructor.
+   * - ``operator=``
+     - Trivial implicit copy/move.
+   * - ``operator cuda::std::size_t``
+     - Implicit conversion to `cuda::std::size_t <https://en.cppreference.com/w/cpp/types/size_t>`__.
+
+Notes
+-----
+
+If ``Alignment`` is not a `valid alignment <https://en.cppreference.com/w/c/language/object#Alignment>`_,
+the behavior is undefined.
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/memory>
+
+   __global__ void example_kernel(void* dst, void* src, size_t size) {
+     cuda::barrier<cuda::thread_scope_system> bar;
+     init(&bar, 1);
+
+     // Implementation cannot make assumptions about alignment.
+     cuda::memcpy_async(dst, src, size, bar);
+
+     // Implementation can assume that dst and src are 16-bytes aligned,
+     // and that size is a multiple of 16, and may optimize accordingly.
+     cuda::memcpy_async(dst, src, cuda::aligned_size_t<16>(size), bar);
+
+     bar.arrive_and_wait();
+   }
+
+`See it on Godbolt <https://godbolt.org/z/PWGdfTd7d>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory/discard_memory.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory/discard_memory.rst
new file mode 100644
index 00000000..92da43f3
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory/discard_memory.rst
@@ -0,0 +1,45 @@
+.. _libcudacxx-extended-api-memory-discard-memory:
+
+``cuda::discard_memory``
+========================
+
+Defined in header ``<cuda/memory>``, ``<cuda/discard_memory>`` (deprecated since CCCL 3.2).
+
+.. code:: cuda
+
+    __host__ __device__
+    void discard_memory(volatile void* ptr, size_t nbytes);
+
+Discard modified cache lines without writing back the cached data to memory. The functionality enables using global memory as temporary scratch space. Does **not** generate any HW store operations.
+
+Equivalent to ``memset(ptr, _indeterminate_, nbytes)``.
+
+**Preconditions**
+
+- ``ptr`` points to a valid allocation in *global memory* of size greater or equal to ``nbytes``.
+
+Example
+-------
+
+This kernel needs a scratch pad that does not fit in shared memory, so it uses an allocation in global memory instead:
+
+.. code:: cuda
+
+   #include <cuda/memory>
+
+    __device__ int compute(int* scratch, size_t N);
+
+    __global__ void kernel(const int* in, int* out, int* scratch, size_t N) {
+        // Each thread reads N elements into the scratch pad:
+        for (int i = 0; i < N; ++i) {
+            int idx      = threadIdx.x + i * blockDim.x;
+            scratch[idx] = in[idx];
+        }
+        __syncthreads();
+        // All threads compute on the scratch pad:
+        int result = compute(scratch, N);
+        // All threads discard the scratch pad memory to _hint_ that it does not need to be flushed from the cache:
+        cuda::discard_memory(scratch + threadIdx.x * N, N * sizeof(int));
+        __syncthreads();
+        out[threadIdx.x] = result;
+    }
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory/get_device_address.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory/get_device_address.rst
new file mode 100644
index 00000000..36ced283
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory/get_device_address.rst
@@ -0,0 +1,72 @@
+.. _libcudacxx-extended-api-memory-get-device-address:
+
+``cuda::get_device_address``
+============================
+
+Defined in the headers ``<cuda/memory>`` and ``<cuda/functional>``.
+
+.. code:: cuda
+
+  namespace cuda {
+
+  template <typename T>
+  [[nodiscard]] __host__ __device__ inline
+  T* get_device_address(T& device_object);                    // (1)
+
+  template <typename T>
+  [[nodiscard]] __host__ inline
+  T* get_device_address(T& device_object, device_ref device); // (2)
+
+  } // namespace cuda
+
+``cuda::get_device_address`` returns a valid pointer to a device object for the current (1) or ``device`` (2) device. It replaces uses of ``cudaGetSymbolAddress``, which requires an inout parameter.
+
+**Parameters**
+
+- ``device_object``: Reference to a device object. (1, 2)
+- ``device``: Device for which the object's address shall be retrieved. (2)
+
+**Constraints**
+
+- ``device_object`` must be a ``__device__`` or ``__constant__`` decorated variable.
+
+Example
+-------
+
+.. code:: cuda
+
+  #include <cuda/devices>
+  #include <cuda/memory>
+
+  __device__ int device_object[] = {42, 1337, -1, 0};
+
+  __global__ void example_kernel(int *data) { ... }
+
+  void example()
+  {
+    cuda::device_ref device{0};
+
+    {
+      T* host_address = cuda::std::addressof(device_object);
+
+      cudaPointerAttributes attributes;
+      cudaError_t status = cudaPointerGetAttributes(&attributes, host_address);
+      assert(status == cudaSuccess);
+      assert(attributes.devicePointer == nullptr);
+
+      // Calling a kernel with host_address would segfault
+      // example_kernel<<<1, 1>>>(host_address);
+    }
+
+    {
+      T* device_address = cuda::get_device_address(device_object, device);
+
+      cudaPointerAttributes attributes;
+      cudaError_t status = cudaPointerGetAttributes(&attributes, device_address);
+      assert(status == cudaSuccess);
+      assert(attributes.devicePointer == device_address);
+
+      // Safe to call a kernel
+      example_kernel<<<1, 1>>>(device_address);
+    }
+  }
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory/is_address_from.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory/is_address_from.rst
new file mode 100644
index 00000000..731f1043
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory/is_address_from.rst
@@ -0,0 +1,116 @@
+.. _libcudacxx-extended-api-memory-is_address_from:
+
+``cuda::device::is_address_from`` and ``cuda::device::is_object_from``
+======================================================================
+
+Defined in the ``<cuda/memory>`` header.
+
+.. code:: cuda
+
+   namespace cuda::device {
+
+   enum class address_space
+   {
+     global,         // Global state space
+     shared,         // Shared state space
+     constant,       // Constant state space
+     local,          // Local state space
+     grid_constant,  // Kernel function parameter in the parameter state space
+     cluster_shared, // Cluster shared window within the shared state space
+   };
+
+   } // namespace cuda::device
+
+Enumeration of device address spaces used with the ``is_address_from()`` and ``is_object_from()`` functions. See the `PTX ISA documentation for state spaces <https://docs.nvidia.com/cuda/parallel-thread-execution/#state-spaces>`_ for more details.
+
+----
+
+.. code:: cuda
+
+   namespace cuda::device {
+
+   [[nodiscard]] __device__ inline
+   bool is_address_from(const volatile void* ptr, address_space space) noexcept; // (1)
+
+   } // namespace cuda::device
+
+Checks whether a generic-address pointer ``ptr`` is from the specified address space.
+
+----
+
+.. code:: cuda
+
+   namespace cuda::device {
+
+   template <typename T>
+   [[nodiscard]] __device__ inline
+   bool is_object_from(T& obj, address_space space) noexcept; // (2)
+
+   } // namespace cuda::device
+
+Checks whether an object ``obj`` with a generic address is from the specified address space.
+
+----
+
+Unlike the corresponding CUDA intrinsic functions ``__isGlobal()``, ``__isShared()``, ``__isConstant()``, ``__isLocal()``, ``__isGridConstant()``, and ``__isClusterShared()``, ``is_address_from()`` and ``is_object_from()`` are portable across all compute capabilities and, in debug mode, also checks that the pointer is not null.
+
+**Parameters**
+
+- ``ptr``: The pointer. (1)
+- ``obj``: The object. (2)
+- ``space``: The address space. (1, 2)
+
+**Return value**
+
+- Returns ``true`` if the pointer (1) or object (2) is from the specified address space; ``false`` otherwise.
+
+.. note::
+
+  If the GPU architecture does not support the requested address space, the function always returns ``false``.
+
+**Preconditions**
+
+- ``ptr`` must not be null. (1)
+
+**Performance considerations**
+
+- When available, the built-in functions (``__isGlobal()``, ``__isShared()``, ``__isConstant()``, ``__isLocal()``, ``__isGridConstant()``, or ``__isClusterShared()``) are used to determine the address space.
+- If the memory space of the input pointer matches the requested address space,
+  the function marks the pointer as belonging to that address space. For example, a subsequent store to a generic address that maps to shared memory emits an ``STS`` SASS instruction rather than the generic ``ST`` instruction.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/memory>
+
+    __device__   int global_var;
+    __constant__ int constant_var;
+
+    __global__ void kernel(const __grid_constant__ int grid_constant_var)
+    {
+        using cuda::device::address_space;
+        __shared__ int shared_var;
+        int local_var{};
+
+        assert(cuda::device::is_address_from(&global_var, address_space::global));
+        assert(cuda::device::is_address_from(&shared_var, address_space::shared));
+        assert(cuda::device::is_address_from(&constant_var, address_space::constant));
+        assert(cuda::device::is_address_from(&local_var, address_space::local));
+        assert(cuda::device::is_address_from(&grid_constant_var, address_space::grid_constant));
+
+        assert(cuda::device::is_object_from(global_var, address_space::global));
+        assert(cuda::device::is_object_from(shared_var, address_space::shared));
+        assert(cuda::device::is_object_from(constant_var, address_space::constant));
+        assert(cuda::device::is_object_from(local_var, address_space::local));
+        assert(cuda::device::is_object_from(grid_constant_var, address_space::grid_constant));
+    }
+
+    int main(int, char**)
+    {
+        kernel<<<1, 1>>>(42);
+        cudaDeviceSynchronize();
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/5ajhe37df>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory/is_aligned.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory/is_aligned.rst
new file mode 100644
index 00000000..5e8a7a91
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory/is_aligned.rst
@@ -0,0 +1,59 @@
+.. _libcudacxx-extended-api-memory-is_aligned:
+
+``cuda::is_aligned``
+====================
+
+Defined in the header ``<cuda/memory>``.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   [[nodiscard]] __host__ __device__ inline
+   bool is_aligned(const void* ptr, size_t alignment) noexcept
+
+   } // namespace cuda
+
+The function determines if a pointer is aligned to a specific alignment.
+
+**Parameters**
+
+- ``ptr``: The pointer.
+- ``alignment``: The alignment.
+
+**Return value**
+
+- ``true`` if the pointer is aligned to the specified alignment, ``false`` otherwise.
+
+**Constraints**
+
+- ``alignment`` must be a power of two.
+
+.. note::
+
+  The function is similar to the C++ standard library function `cuda::std::is_sufficiently_aligned() <https://en.cppreference.com/w/cpp/memory/is_sufficiently_aligned.html>`__ from the ``<cuda/std/memory>`` header. The differences are the following:
+
+  - ``cuda::is_aligned()`` doesn't have a template parameter and might be less expensive to compile.
+  - ``cuda::is_aligned()`` supports run-time values of ``alignment``.
+  - ``cuda::std::is_sufficiently_aligned()`` additionally checks the compatibility between the alignment of the pointer type and the specified alignment.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/memory>
+
+    __global__ void kernel(const void* ptr) {
+        assert(cuda::is_aligned(ptr, 16));
+    }
+
+    int main() {
+        void* ptr;
+        cudaMalloc(&ptr, 100 * sizeof(int));
+        kernel<<<1, 1>>>(ptr);
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/Tr45EoKsT>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory/is_pointer_accessible.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory/is_pointer_accessible.rst
new file mode 100644
index 00000000..e68dcf73
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory/is_pointer_accessible.rst
@@ -0,0 +1,87 @@
+.. _libcudacxx-extended-api-memory-is_pointer_accessible:
+
+``cuda::is_host_accessible``, ``cuda::is_device_accessible``, ``cuda::is_managed``
+==================================================================================
+
+Defined in the ``<cuda/memory>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   [[nodiscard]] inline
+   bool is_host_accessible(const void* ptr); // (1)
+
+   [[nodiscard]] inline
+   bool is_device_accessible(const void* ptr, device_ref device); // (2)
+
+   [[nodiscard]] inline
+   bool is_managed(const void* ptr); // (3)
+
+   } // namespace cuda
+
+Determines whether the memory referenced by ``ptr`` is accessible from the host (1), from the specified ``device`` (2), or is backed by Unified Memory (managed memory) (3).
+
+- ``is_device_accessible()`` also checks whether the memory is peer-accessible or allocated from a memory pool accessible to the specified ``device``.
+- ``is_host_accessible()`` also checks whether the memory is allocated from a memory pool accessible to the host.
+
+----
+
+**Parameters**
+
+- ``ptr``: A pointer to the memory location to query.
+- ``device``: A ``device_ref`` that denotes the device to query. (2)
+
+**Return value**
+
+- ``true`` if the queried property (host access, device access, or managed allocation) holds; otherwise, ``false``.
+
+.. note::
+
+  A ``__device__`` global array or variable cannot be used directly from host code without first retrieving its address with ``cudaGetSymbolAddress()``.
+
+**Prerequisites**
+
+- The functions are available only when the CUDA Toolkit is available.
+
+**Exceptions**
+
+- The functions throw a ``cuda::cuda_error`` if the underlying driver API calls fail. Note that these functions may also fail with error codes from previously launched asynchronous operations.
+
+**Undefined Behavior**
+
+- The functions have undefined behavior if the pointer is not valid, for example, an already freed pointer.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cassert>
+    #include <cuda/memory>
+    #include <cuda_runtime_api.h>
+
+    int main() {
+        cuda::device_ref dev{0};
+        void* host_ptr    = nullptr;
+        void* device_ptr  = nullptr;
+        void* managed_ptr = nullptr;
+
+        cudaMallocHost(&host_ptr, 1024);
+        cudaMalloc(&device_ptr, 1024);
+        cudaMallocManaged(&managed_ptr, 1024);
+
+        assert(cuda::is_host_accessible(host_ptr));
+        assert(!cuda::is_device_accessible(host_ptr, dev));
+
+        assert(cuda::is_device_accessible(device_ptr, dev));
+        assert(!cuda::is_host_accessible(device_ptr));
+
+        assert(cuda::is_host_accessible(managed_ptr));
+        assert(cuda::is_device_accessible(managed_ptr, dev));
+        assert(cuda::is_managed(managed_ptr));
+
+        cudaFreeHost(host_ptr);
+        cudaFree(device_ptr);
+        cudaFree(managed_ptr);
+    }
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory/ptr_in_range.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory/ptr_in_range.rst
new file mode 100644
index 00000000..73cd7777
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory/ptr_in_range.rst
@@ -0,0 +1,67 @@
+.. _libcudacxx-extended-api-memory-ptr_in_range:
+
+``cuda::ptr_in_range``
+======================
+
+Defined in the header ``<cuda/memory>``.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T>
+   [[nodiscard]] __host__ __device__ constexpr
+   bool ptr_in_range(T* ptr, T* start, T* end) noexcept;
+
+   } // namespace cuda
+
+Checks whether ``ptr`` lies inside the half-open interval ``[start, end)``.
+
+**Template parameters**
+
+- ``T``: The type of the pointer.
+
+**Parameters**
+
+- ``ptr``: The pointer being tested.
+- ``start``: Pointer to the first element in the range.
+- ``end``: Pointer to one past the last element in the range.
+
+**Return value**
+
+- ``true`` when the pointer lies in ``[start, end)``, ``false`` otherwise.
+
+**Preconditions**
+
+- ``end`` must be greater than or equal to ``start``.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/memory>
+
+    __global__ void kernel(float* data, size_t count) {
+        float* first = data;
+        float* last  = data + count;
+
+        float* elem_ptr = data + threadIdx.x;
+        if (cuda::ptr_in_range(elem_ptr, first, last)) {
+            *elem_ptr = static_cast<float>(threadIdx.x);
+        }
+    }
+
+    int main() {
+        size_t N          = 32;
+        float* device_ptr = nullptr;
+        cudaMalloc(&device_ptr, N * sizeof(float));
+
+        kernel<<<1, N>>>(device_ptr, N);
+        cudaDeviceSynchronize();
+
+        cudaFree(device_ptr);
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/sMz76hGEc>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory/ptr_rebind.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory/ptr_rebind.rst
new file mode 100644
index 00000000..6f50561d
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory/ptr_rebind.rst
@@ -0,0 +1,70 @@
+.. _libcudacxx-extended-api-memory-ptr_rebind:
+
+``cuda::ptr_rebind``
+====================
+
+Defined in the header ``<cuda/memory>``.
+
+.. code:: cuda
+
+    namespace cuda {
+
+    template <typename U, typename T>
+    [[nodiscard]] __host__ __device__
+    U* ptr_rebind(T* ptr) noexcept;
+
+    template <typename U, typename T>
+    [[nodiscard]] __host__ __device__
+    const U* ptr_rebind(const T* ptr) noexcept;
+
+    template <typename U, typename T>
+    [[nodiscard]] __host__ __device__
+    volatile U* ptr_rebind(volatile T* ptr) noexcept;
+
+    template <typename U, typename T>
+    [[nodiscard]] __host__ __device__
+    const volatile U* ptr_rebind(const volatile T* ptr) noexcept;
+
+    } // namespace cuda
+
+The functions return the pointer ``ptr`` cast to type ``U*`` or ``const U*``. They are shorter and safer alternative to ``reinterpret_cast``.
+
+**Parameters**
+
+- ``ptr``: The pointer.
+
+**Return value**
+
+- The pointer cast to type ``U*`` or ``const U*``.
+
+**Constraints**
+
+- ``ptr`` must be aligned to ``alignof(U)`` and ``alignof(T)``.
+
+**Performance considerations**
+
+- The returned pointer is decorated with ``__builtin_assume_aligned`` with the ``alignof(U)`` value to help the compiler generate better code.
+- The returned pointer maintains the same memory space, for example shared memory, as the input pointer.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/memory>
+    #include <cuda/std/cstdint>
+
+    __global__ void kernel(const int* ptr, volatile int* ptr2) {
+        auto ptr_res1 = cuda::ptr_rebind<uint64_t>(ptr);  // ptr_res1: const uint64_t*
+        auto ptr_res2 = cuda::ptr_rebind<uint64_t>(ptr2); // ptr_res2: volatile uint64_t*
+    }
+
+    int main() {
+        int* ptr;
+        cudaMalloc(&ptr, 100 * sizeof(int));
+        kernel<<<1, 1>>>(ptr);
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/bavzabce9>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory/ranges_overlap.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory/ranges_overlap.rst
new file mode 100644
index 00000000..533b26be
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory/ranges_overlap.rst
@@ -0,0 +1,60 @@
+.. _libcudacxx-extended-api-memory-ranges_overlap:
+
+``cuda::ranges_overlap``
+========================
+
+Defined in the ``<cuda/memory>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T>
+   [[nodiscard]] __host__ __device__ constexpr
+   bool ranges_overlap(T lhs_start, T lhs_end, T rhs_start, T rhs_end) noexcept;
+
+   } // namespace cuda
+
+Returns ``true`` when the half-open byte ranges ``[lhs_start, lhs_end)`` and ``[rhs_start, rhs_end)`` intersect.
+
+**Constraints**
+
+- ``T`` must be a forward iterator.
+
+**Parameters**
+
+- ``lhs_start``: The beginning of the first range.
+- ``lhs_end``: The end of the first range.
+- ``rhs_start``: The beginning of the second range.
+- ``rhs_end``: The end of the second range.
+
+**Return value**
+
+- ``true`` when the two ranges overlap, ``false`` otherwise.
+
+**Performance considerations**
+
+- The function is optimized when the ranges are contiguous and random access iterators.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/memory>
+    #include <cuda/std/cassert>
+
+    __global__ void overlap_kernel() {
+        int arrayA[10];
+        int arrayB[10];
+        assert(cuda::ranges_overlap(arrayA + 2, arrayA + 7, arrayA, arrayA + 10)); // overlap
+        assert(!cuda::ranges_overlap(arrayA, arrayA + 10, arrayB, arrayB + 10));   // no overlap
+    }
+
+    int main() {
+        overlap_kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/nasnWz9Tv>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory_access_properties.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory_access_properties.rst
new file mode 100644
index 00000000..1cc14e49
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory_access_properties.rst
@@ -0,0 +1,41 @@
+.. _libcudacxx-extended-api-memory-access-properties:
+
+Memory access properties
+------------------------
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   memory_access_properties/access_property
+   memory_access_properties/annotated_ptr
+   memory_access_properties/apply_access_property
+   memory_access_properties/associate_access_property
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`cuda::access_property <libcudacxx-extended-api-memory-access-properties-access-property>`
+     - Represents a memory access property
+     - libcu++ 1.6.0 / CCCL 2.0.0 /
+     - CUDA 11.5
+
+   * - :ref:`cuda::annotated_ptr <libcudacxx-extended-api-memory-access-properties-annotated-ptr>`
+     - Binds an access property to a pointer
+     - libcu++ 1.6.0 / CCCL 2.0.0
+     - CUDA 11.5
+   * - :ref:`cuda::apply_access_property <libcudacxx-extended-api-memory-access-properties-apply-access-property>`
+     - Applies access property to memory
+     - libcu++ 1.6.0 / CCCL 2.0.0
+     - CUDA 11.5
+
+   * - :ref:`cuda::associate_access_property <libcudacxx-extended-api-memory-access-properties-associate-access-property>`
+     - Associates access property with raw pointer
+     - libcu++ 1.6.0 / CCCL 2.0.0
+     - CUDA 11.5
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory_access_properties/access_property.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory_access_properties/access_property.rst
new file mode 100644
index 00000000..552f2ab9
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory_access_properties/access_property.rst
@@ -0,0 +1,221 @@
+.. _libcudacxx-extended-api-memory-access-properties-access-property:
+
+``cuda::access_property``
+=========================
+
+Defined in header ``<cuda/annotated_ptr>``.
+
+The class ``cuda::access_property`` provides an opaque encoding for *L2 cache memory residence* control and *memory space* properties. It is used in combination with :ref:`cuda::annotated_ptr <libcudacxx-extended-api-memory-access-properties-annotated-ptr>`, :ref:`cuda::associate_access_property <libcudacxx-extended-api-memory-access-properties-associate-access-property>` and :ref:`cuda::apply_access_property <libcudacxx-extended-api-memory-access-properties-apply-access-property>` to *request* the application of properties to memory operations.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   class access_property {
+     public:
+       // Static memory space property:
+       struct shared {};
+       struct global {};
+
+       // Static global memory residence control property:
+       struct normal {
+           __host__ __device__ constexpr operator cudaAccessProperty() const noexcept;
+       };
+       struct persisting {
+           __host__ __device__ constexpr operator cudaAccessProperty() const noexcept;
+       };
+       struct streaming {
+           __host__ __device__ constexpr operator cudaAccessProperty() const noexcept;
+       };
+
+       access_property() noexcept = default;
+
+       // Constructors from static global memory residence control properties:
+       __host__ __device__ constexpr access_property(global)     noexcept;
+       __host__ __device__ constexpr access_property(normal)     noexcept;
+       __host__ __device__ constexpr access_property(streaming)  noexcept;
+       __host__ __device__ constexpr access_property(persisting) noexcept;
+
+       // Dynamic interleaved global memory residence control property constructors:
+       __host__ __device__ constexpr access_property(normal,     float probability) noexcept;
+       __host__ __device__ constexpr access_property(streaming,  float probability) noexcept;
+       __host__ __device__ constexpr access_property(persisting, float probability) noexcept;
+       __host__ __device__ constexpr access_property(normal,     float probability, streaming) noexcept;
+       __host__ __device__ constexpr access_property(persisting, float probability, streaming) noexcept;
+
+       // Dynamic range global memory residence control property constructors:
+       __host__ __device__ access_property(void* ptr, size_t primary_bytes, size_t total_bytes, normal) noexcept;
+       __host__ __device__ access_property(void* ptr, size_t primary_bytes, size_t total_bytes, streaming) noexcept;
+       __host__ __device__ access_property(void* ptr, size_t primary_bytes, size_t total_bytes, persisting) noexcept;
+       __host__ __device__ access_property(void* ptr, size_t primary_bytes, size_t total_bytes, global,     streaming) noexcept;
+       __host__ __device__ access_property(void* ptr, size_t primary_bytes, size_t total_bytes, normal,     streaming) noexcept;
+       __host__ __device__ access_property(void* ptr, size_t primary_bytes, size_t total_bytes, persisting, streaming) noexcept;
+       __host__ __device__ access_property(void* ptr, size_t primary_bytes, size_t total_bytes, streaming,  streaming) noexcept;
+   };
+
+   } // namespace cuda
+
+Kinds of Access Properties
+--------------------------
+
+Access properties are either *static* compile-time values or *dynamic* runtime values. The following properties
+of a memory access are provided:
+
+*Shared Memory property*:
+
+   .. _libcudacxx-extended-api-memory-access-properties-access-property-shared:
+
+- ``cuda::access_property::shared``: memory access to the shared memory space.
+
+*Global Memory properties*:
+
+   .. _libcudacxx-extended-api-memory-access-properties-access-property-global:
+
+- ``cuda::access_property::global``: memory access to the global memory space *without* indicating an expected frequency of access to that memory, namely the access behavior is not modified.
+
+   .. _libcudacxx-extended-api-memory-access-properties-access-property-normal:
+
+- ``cuda::access_property::normal``: memory access to the global memory space expecting the memory to be accessed as frequent as other memory.
+
+   .. _libcudacxx-extended-api-memory-access-properties-access-property-persisting:
+
+- ``cuda::access_property::persisting``: memory access to the global memory space expecting the memory to be accessed more frequently than other memory; this priority is suitable for data that should remain persistent in cache.
+
+   .. _libcudacxx-extended-api-memory-access-properties-access-property-streaming:
+
+- ``cuda::access_property::streaming``: memory access to the global memory space expecting the memory to be accessed infrequently; this priority is suitable for streaming data.
+
+**Note**: The difference between ``cuda::access_property::global`` and ``cuda::access_property::normal`` is subtle.
+The ``cuda::access_property::normal`` hints that the pointer points to the global address space *and* the memory will
+be accessed with "normal frequency", while ``cuda::access_property::global`` only hints that the pointer points to
+the global address-space, it does not hint about how frequent the accesses will be.
+
+.. warning::
+
+   The behavior of *requesting* the application of ``cuda::access_property`` to memory accesses, or their association
+   with memory addresses, outside of the corresponding address space is *undefined*
+   (note: even if that address is not used). The correctness of the input pointer and memory properties are verified in debug mode.
+
+----
+
+Global Memory Property Definition
+---------------------------------
+
+The L2 residence control can be specified in two ways:
+
+- **Interleaved**: A memory address is accessed with a property with a given ``probability``, while the remaining ``1 - probability`` accesses are performed with a second one.
+- **Range**: The first ``primary_bytes`` of a memory address is accessed with one property and the remaining ``total_bytes - primary_bytes`` addresses with a second one.
+
+Default constructor
+-------------------
+
+.. code:: cuda
+
+   access_property() noexcept = default;
+
+**Effects**: as if ``access_property(global)`` (unchanged).
+
+Static global memory residence control property constructors
+------------------------------------------------------------
+
+.. code:: cuda
+
+   __host__ __device__ constexpr access_property::access_property(global) noexcept;
+   __host__ __device__ constexpr access_property::access_property(normal) noexcept;
+   __host__ __device__ constexpr access_property::access_property(streaming) noexcept;
+   __host__ __device__ constexpr access_property::access_property(persisting) noexcept;
+
+**Effects**: as if ``access_property(PROPERTY, 1.0)`` where ``PROPERTY`` is one of ``global`` (unchanged), ``normal``, ``streaming``, or ``persisting``.
+
+Dynamic interleaved global memory residence control property constructors
+-------------------------------------------------------------------------
+
+.. code:: cuda
+
+   __host__ __device__ constexpr access_property::access_property(normal,     float probability) noexcept;
+   __host__ __device__ constexpr access_property::access_property(streaming,  float probability) noexcept;
+   __host__ __device__ constexpr access_property::access_property(persisting, float probability) noexcept;
+   __host__ __device__ constexpr access_property::access_property(normal,     float probability, streaming) noexcept;
+   __host__ __device__ constexpr access_property::access_property(persisting, float probability, streaming) noexcept;
+
+**Preconditions**: ``0 < probability <= 1.0``.
+
+**Effects**: constructs an *interleaved* access property that *requests* the first and third arguments - access properties - to be applied with ``probability`` and ``1 - probability`` to memory accesses. The overloads without a third argument request applying ``global`` (unchanged) with ``1 - probability``.
+
+Dynamic range global memory residence control property constructors
+-------------------------------------------------------------------
+
+.. code:: cuda
+
+   __host__ __device__ access_property::access_property(void* ptr, size_t leading_bytes, size_t total_bytes, normal) noexcept;
+   __host__ __device__ access_property::access_property(void* ptr, size_t leading_bytes, size_t total_bytes, streaming) noexcept;
+   __host__ __device__ access_property::access_property(void* ptr, size_t leading_bytes, size_t total_bytes, persisting) noexcept;
+   __host__ __device__ access_property::access_property(void* ptr, size_t leading_bytes, size_t total_bytes, normal,     streaming) noexcept;
+   __host__ __device__ access_property::access_property(void* ptr, size_t leading_bytes, size_t total_bytes, persisting, streaming) noexcept;
+
+..
+
+   note: pointer arithmetic below performed ``char* ptr`` instead of
+   ``void* ptr``
+
+**Preconditions**:
+
+   - ``ptr`` is a generic pointer that is *valid* to cast to a pointer to the global memory address space.
+   - ``0 < leading_bytes <= total_bytes <= 4GB``.
+
+**Postconditions**: memory accesses requesting the application of this property must be in range ``[ptr, ptr + total_bytes)``.
+
+**Effects**: the fourth and fifth arguments, access properties, are called *primary* and *secondary* properties. The overloads without a fifth argument use ``global`` as the *secondary* property. Constructs a *range* access property *requesting* the properties to be **approximately** applied to memory accesses as follows:
+
+-  *primary property* to accesses in address-range:   ``[ptr, ptr + leading_bytes)``
+-  *secondary property* to accesses in address-range: ``[ptr + leading_bytes, ptr + total_bytes)``
+
+**Note**: This property enables two main use cases:
+
+1. Unary range ``[ptr, ptr + total_bytes)`` with *primary property* by using ``leading_bytes == total_bytes``.
+
+2. Binary range ``[ptr, ptr + leading_bytes)`` and ``[ptr + leading_bytes, ptr + total_bytes)`` with *primary* and
+   *secondary properties* respectively.
+
+Conversion operators
+--------------------
+
+.. code:: cuda
+
+   __host__ __device__ constexpr access_property::normal::operator     cudaAccessProperty() const noexcept;
+   __host__ __device__ constexpr access_property::streaming::operator  cudaAccessProperty() const noexcept;
+   __host__ __device__ constexpr access_property::persisting::operator cudaAccessProperty() const noexcept;
+
+Allows ``constexpr cuda::access_property::normal{}``, ``cuda::access_property::streaming{}``, and ``cuda::access_property::persisting{}`` to be used in lieu of the corresponding CUDA Runtime `cudaAccessProperty <https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1g4991a8bc9c2356a8da28d093a1da6758>`_. See also `L2 Policy for Persisting Accesses <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#l2-policy-for-persisting-accesses>`_.
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/access_property>
+
+   __global__ void kernel(int* global_ptr, size_t num_bytes) {
+       __shared__ int smem;
+       cuda::access_property shared_prop{&smem, cuda::access_property::shared{}};
+       cuda::access_property streaming_prop{global_ptr, sizeof(int), sizeof(int), cuda::access_property::streaming{}};
+       cuda::access_property streaming_interleaved_prop{cuda::access_property::streaming{}, 1.0};
+       cuda::access_property persisting_prop{global_ptr, num_bytes, num_bytes, cuda::access_property::persisting{});
+   }
+
+   __global__ void undefined_behavior(int* global_ptr) { // verified in debug mode
+       __shared__ int smem;
+       // Associating pointers with mismatching address spaces is undefined:
+       cuda::access_property{global_ptr, cuda::access_property::shared{}}; // undefined behavior
+       cuda::access_property{&smem, cuda::access_property::normal{}};      // undefined behavior
+       cuda::access_property{&smem, cuda::access_property::streaming{}};   // undefined behavior
+       cuda::access_property{&smem, cuda::access_property::persisting{}};  // undefined behavior
+
+       // Using a zero probability or probability out-of-range (0, 1] is undefined:
+       cuda::access_property{cuda::access_property::streaming{}, 0.0f};    // undefined behavior
+       cuda::access_property{cuda::access_property::streaming{}, 2.0f};    // undefined behavior
+
+       // Providing size values out-of-range is undefined:
+       cuda::access_property{global_ptr, 0, 0, cuda::access_property::streaming{}, 0.0f}; // undefined behavior
+       cuda::access_property{global_ptr, 8, 4, cuda::access_property::streaming{}, 2.0f}; // undefined behavior
+   }
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory_access_properties/annotated_ptr.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory_access_properties/annotated_ptr.rst
new file mode 100644
index 00000000..e53e77d4
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory_access_properties/annotated_ptr.rst
@@ -0,0 +1,341 @@
+.. _libcudacxx-extended-api-memory-access-properties-annotated-ptr:
+
+``cuda::annotated_ptr``
+=======================
+
+Defined in header ``<cuda/annotated_ptr>``.
+
+``cuda::annotated_ptr`` is a pointer annotated with an access property that *may* be applied to its memory operations.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template<typename Type, typename Property>
+   class annotated_ptr {
+   public:
+     using value_type      = Type;
+     using size_type       = size_t;
+     using reference       = value_type&;
+     using pointer         = value_type*;
+     using const_pointer   = const value_type*;
+     using difference_type = ptrdiff_t;
+
+     annotated_ptr() noexcept = default;
+
+     __host__ __device__ explicit constexpr annotated_ptr(pointer) noexcept;
+
+     template <typename RuntimeProperty>
+     __host__ __device__ annotated_ptr(pointer, RuntimeProperty) noexcept;
+
+     template <typename T, typename P>
+     __host__ __device__ annotated_ptr(const annotated_ptr<T,P>&) noexcept;
+
+     __host__ __device__ constexpr explicit operator bool() const noexcept;
+
+     [[nodiscard]] __host__ __device__ pointer   get() const noexcept;
+     [[nodiscard]] __host__ __device__ reference operator*() const noexcept;
+     [[nodiscard]] __host__ __device__ pointer   operator->() const noexcept;
+     [[nodiscard]] __host__ __device__ reference operator[](ptrdiff_t) const noexcept;
+     [[nodiscard]] __host__ __device__ constexpr difference_type operator-(annotated_ptr) const noexcept;
+
+   private:
+     pointer  ptr;  // exposition only
+     Property prop; // exposition only
+   };
+
+   } // namespace cuda
+
+.. note::
+  If ``Property`` is :ref:`cuda::access_property <libcudacxx-extended-api-memory-access-properties-access-property>`,
+  namely a dynamic property with a runtime value,
+  then ``sizeof(cuda::annotated_ptr<Type, cuda::access_property>) == 2 * sizeof(Type*)``. Otherwise, its size is ``sizeof(Type*)``.
+
+In contrast to :ref:`cuda::associate_access_property <libcudacxx-extended-api-memory-access-properties-associate-access-property>`, ``cuda::annotated_ptr`` maintains the association between the pointer and the property when passed across translation units.
+
+**Constraints**
+
+``Property`` is one of:
+
+-  :ref:`cuda::access_property::shared <libcudacxx-extended-api-memory-access-properties-access-property-shared>`,
+-  :ref:`cuda::access_property::global <libcudacxx-extended-api-memory-access-properties-access-property-global>`,
+-  :ref:`cuda::access_property::persisting <libcudacxx-extended-api-memory-access-properties-access-property-persisting>`,
+-  :ref:`cuda::access_property::normal <libcudacxx-extended-api-memory-access-properties-access-property-normal>`,
+-  :ref:`cuda::access_property::streaming <libcudacxx-extended-api-memory-access-properties-access-property-streaming>` or,
+-  :ref:`cuda::access_property <libcudacxx-extended-api-memory-access-properties-access-property>`:
+   a type-erased specification that allows ``annotated_ptr`` to set the access property at runtime value.
+
+**Semantics**
+
+.. list-table::
+   :widths: 25 30 40
+   :header-rows: 1
+
+   * - Pointer Expression
+     - ``cuda::annotated_ptr<T, P>``
+     - Description
+
+   * - ``T* a``
+     - ``cuda::annotated_ptr<T, P> a``
+     - Non-``const`` pointer to non-``const`` memory
+
+   * - ``T const * a``
+     - ``cuda::annotated_ptr<T const, P> a``
+     - Non-``const`` pointer to ``const`` memory
+
+   * - ``T* const a``
+     - ``const cuda::annotated_ptr<T, P> a``
+     - ``const`` pointer to non-``const`` memory
+
+   * - ``T const* const a``
+     - ``const cuda::annotated_ptr<T const, P> a``
+     - ``const`` pointer to ``const`` memory
+
+   * - ``val = *a;``
+     - ``val = *a;``
+     - Dereference operator to load an element
+
+   * - ``*a = val;``
+     - ``*a = val;``
+     - Dereference operator to store an element
+
+   * - ``val = a[n];``
+     - ``val = a[n];``
+     - Subscript operator to load an element
+
+   * - ``a[n] = val;``
+     - ``a[n] = val;``
+     - Subscript operator to store an element
+
+   * - ``T* a = nullptr;``
+     - ``annotated_ptr<T, P> a = nullptr;``
+     - ``nullptr`` initialization
+
+   * - ``n = a - b;``
+     - ``n = a - b;``
+     - Difference operator
+
+   * - ``if (a) { ... }``
+     - ``if (a) { ... }``
+     - Bool conversion
+
+*Note*: It is not a drop-in replacement for pointers since, among others, it does not:
+
+-  model any `Iterator <https://en.cppreference.com/w/cpp/iterator>`_ concept,
+-  implement `cuda::std::pointer_traits <https://en.cppreference.com/w/cpp/memory/pointer_traits>`_,    `cuda::std::iterator_traits <https://en.cppreference.com/w/cpp/iterator/iterator_traits>`_, etc.
+-  have the same variance as pointer.
+
+----
+
+Constructors and Assignment
+---------------------------
+
+Default constructor
+~~~~~~~~~~~~~~~~~~~
+
+.. code:: cuda
+
+   annotated_ptr() noexcept = default;
+
+**Effects**:  as if constructed by ``annotated_ptr(nullptr)``;
+
+Constructor from pointer
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: cuda
+
+   constexpr explicit annotated_ptr(pointer ptr);
+
+Constructs an ``annotated_ptr`` requesting associating ``ptr`` with ``Property``.
+
+**Constraints**:
+
+- If ``Property`` is :ref:`cuda::access_property::shared <libcudacxx-extended-api-memory-access-properties-access-property-shared>`, :ref:`cuda::access_property::global <libcudacxx-extended-api-memory-access-properties-access-property-global>`,  :ref:`cuda::access_property::normal <libcudacxx-extended-api-memory-access-properties-access-property-normal>`, :ref:`cuda::access_property::streaming <libcudacxx-extended-api-memory-access-properties-access-property-streaming>`, :ref:`cuda::access_property::persisting <libcudacxx-extended-api-memory-access-properties-access-property-persisting>`, or `cuda::access_property` (dynamic).
+
+**Preconditions**:
+
+- If ``Property`` is :ref:`cuda::access_property::shared <libcudacxx-extended-api-memory-access-properties-access-property-shared>`, then ``ptr`` must be a generic pointer that is a valid pointer to the *shared memory* address space.
+- If ``Property`` is  not :ref:`cuda::access_property::shared <libcudacxx-extended-api-memory-access-properties-access-property-shared>`, then ``ptr`` must be a generic pointer    that is a valid pointer to the *global memory* address space.
+
+Constructor from pointer and access property
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code:: cuda
+
+   template <typename RuntimeProperty>
+   annotated_ptr(pointer ptr, RuntimeProperty prop);
+
+Constructs an ``annotated_ptr`` requesting the association of ``ptr`` with the property ``prop``.
+
+**Constraints**:
+
+-  ``RuntimeProperty`` is any of :ref:`cuda::access_property::global <libcudacxx-extended-api-memory-access-properties-access-property-global>`,
+   :ref:`cuda::access_property::normal <libcudacxx-extended-api-memory-access-properties-access-property-normal>`,
+   :ref:`cuda::access_property::streaming <libcudacxx-extended-api-memory-access-properties-access-property-streaming>`,
+   :ref:`cuda::access_property::persisting <libcudacxx-extended-api-memory-access-properties-access-property-persisting>`, or
+   :ref:`cuda::access_property <libcudacxx-extended-api-memory-access-properties-access-property>` (same as *global*).
+
+**Preconditions**:
+
+- ``ptr`` is a pointer to a valid allocation in the *global memory* address space.
+
+Copy Constructor from a different ``annotated_ptr``
+----------------------------------------------------
+
+.. code:: cuda
+
+   template <typename T, typename P>
+   constexpr annotated_ptr(const annotated_ptr<T, P>& a);
+
+Constructs an ``annotated_ptr`` for the same pointer as the input ``annotated_ptr``.
+
+**Constraints**
+
+-  ``annotated_ptr<Type, Property>::pointer`` is assignable from ``annotated_ptr<T, P>::pointer``.
+-  ``Property`` is either ``cuda::access_property`` (*dynamic*) or ``P``.
+-  ``Property`` and ``P`` specify the same memory space.
+
+**Preconditions**
+
+- ``pointer`` is compatible with ``Property``.
+
+Explicit conversion operator to ``bool``
+----------------------------------------
+
+.. code:: cuda
+
+   constexpr operator bool() const noexcept;
+
+**Returns**: ``false`` if the pointer is a ``nullptr``, ``true`` otherwise.
+
+Raw pointer access
+------------------
+
+.. code:: cuda
+
+   pointer get() const noexcept;
+
+**Returns**: A pointer derived from the ``annotated_ptr``.
+
+Operators
+---------
+
+Dereference
+~~~~~~~~~~~
+
+.. code:: cuda
+
+   reference operator*() const noexcept;
+
+**Returns**: value pointed by ``annotated_ptr``.
+
+**Preconditions**
+
+The underlying pointer is not null.
+
+Pointer-to-member
+~~~~~~~~~~~~~~~~~
+
+.. code:: cuda
+
+   pointer operator->() const noexcept;
+
+**Preconditions**
+
+- The underlying pointer is not null.
+
+**Returns**: underlying pointer.
+
+Subscript
+~~~~~~~~~
+
+.. code:: cuda
+
+   reference operator[](ptrdiff_t i) const noexcept;
+
+**Returns**: reference to element ``i``.
+
+**Preconditions**
+
+- The underlying pointer plus the offset ``i`` is not null.
+
+Pointer distance
+~~~~~~~~~~~~~~~~
+
+.. code:: cuda
+
+   constexpr difference_type operator-(annotated_ptr p) const;
+
+**Returns**: Difference of pointers, as-if ``get() - p.get()``.
+
+**Preconditions**
+
+- ``ptr >= p``.
+
+----
+
+Example
+-------
+
+Given three input and output vectors ``x``, ``y``, and ``z``, and two arrays of coefficients ``a`` and ``b``, all of length ``N``:
+
+.. code:: cuda
+
+    size_t N;
+    int* x, *y, *z;
+    int* a, *b;
+
+the grid-strided kernel:
+
+.. code:: cuda
+
+    __global__ void update(const int* x, const int* a, const int* b, size_t N) {
+        auto g = cooperative_groups::this_grid();
+        for (int i = g.thread_rank(); idx < N; idx += g.size()) {
+            x[i] = a[i] * x[i] + b[i];
+        }
+    }
+
+updates ``x``, ``y``, and ``z`` as follows:
+
+.. code:: cuda
+
+   update<<<grid, block>>>(x, a, b, N);
+   update<<<grid, block>>>(y, a, b, N);
+   update<<<grid, block>>>(z, a, b, N);
+
+The elements of ``a`` and ``b`` are used in all kernels. If ``N`` is large enough, elements of ``a`` and ``b`` might be evicted from the L2 cache, requiring these to be re-loaded from memory in the next ``update``.
+
+We can make the ``update`` kernel generic to allow the caller to pass ``cuda::annotated_ptr`` objects that hint at how memory will be accessed:
+
+.. code:: cuda
+
+    template <typename PointerX, typename PointerA, typename PointerB>
+    __global__ void update_template(PointerX x, PointerA a, PointerB b, size_t N) {
+        auto g = cooperative_groups::this_grid();
+        for (int idx = g.thread_rank(); idx < N; idx += g.size()) {
+            x[idx] = a[idx] * x[idx] + b[idx];
+        }
+    }
+
+With ``cuda::annotated_ptr``, the caller can then specify the temporal locality of the memory accesses:
+
+.. code:: cuda
+
+   // Frequent accesses to "a" and "b"; infrequent accesses to "x" and "y":
+   cuda::annotated_ptr<const int, cuda::access_property::persisting> a_persistent{a}, b_persistent{b};
+   cuda::annotated_ptr<int, cuda::access_property::streaming>        x_streaming{x}, y_streaming{y};
+   update_template<<<grid, block>>>(x_streaming, a_persistent, b_persistent, N);
+   update_template<<<grid, block>>>(y_streaming, a_persistent, b_persistent, N);
+
+   // Infrequent accesses to "a" and "b"; frequent accesses to "z":
+   cuda::annotated_ptr<const int, cuda::access_property::streaming> a_streaming{a}, b_streaming{b};
+   cuda::annotated_ptr<int, cuda::access_property::persisting>      z_persistent{z};
+   update_template<<<grid, block>>>(z_persistent, a_streaming, b_streaming, N);
+
+   // Different kernel, "update_z", uses "z" again one last time.
+   // Since "z" was accessed as "persisting" by the previous kernel,
+   // parts of it are more likely to have previously survived in the L2 cache.
+   update_z<<<grid, block>>>(z, ...);
+
+Notice how the raw pointers to ``a`` and ``b`` can be wrapped by both ``annotated_ptr<T, persistent>`` and ``annotated_ptr<T, streaming>``, and accesses through each pointer applies the corresponding access property.
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory_access_properties/apply_access_property.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory_access_properties/apply_access_property.rst
new file mode 100644
index 00000000..5957d739
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory_access_properties/apply_access_property.rst
@@ -0,0 +1,101 @@
+.. _libcudacxx-extended-api-memory-access-properties-apply-access-property:
+
+``cuda::apply_access_property``
+===============================
+
+Defined in header ``<cuda/annotated_ptr>``.
+
+.. code:: cuda
+
+   template <typename ShapeT>
+   [[nodiscard]] __host__ __device__
+   void apply_access_property(const volatile void* ptr, ShapeT shape, cuda::access_property::persisting) noexcept;
+
+   template <typename ShapeT>
+   [[nodiscard]] __host__ __device__
+   void apply_access_property(const volatile void* ptr, ShapeT shape, cuda::access_property::normal) noexcept;
+
+Prefetch memory in the L2 cache starting at ``ptr`` applying a residence control property.
+
+**Constraints**
+
+- ``ShapeT`` is either ``size_t`` or :ref:`cuda::aligned_size_t <libcudacxx-extended-api-memory-aligned-size>`.
+- Two properties are supported:
+
+    -  :ref:`cuda::access_property::persisting <libcudacxx-extended-api-memory-access-properties-access-property-persisting>`
+    -  :ref:`cuda::access_property::normal <libcudacxx-extended-api-memory-access-properties-access-property-normal>`
+
+**Preconditions**
+
+- ``ptr`` points to a valid allocation for ``shape`` in the *global memory* address space.
+
+    -  if ``ShapeT`` is ``aligned_size_t<N>(sz)``, then ``ptr`` is aligned to an ``N``-bytes alignment boundary, and
+    -  for all offsets ``i`` in the extent of ``shape``, namely ``i`` in ``[0, shape)``, then the expression ``*(ptr + i)`` does not exhibit undefined behavior.
+
+*Note*:  currently ``apply_access_property`` is ignored on the host.
+
+Example
+-------
+
+Given three input and output vectors ``x``, ``y``, and ``z``, and two arrays of coefficients ``a`` and ``b``, all of length ``N``:
+
+.. code:: cuda
+
+   size_t N;
+   int* x, *y, *z;
+   int* a, *b;
+
+the grid-strided kernel:
+
+.. code:: cuda
+
+    __global__ void update(const int* x, const int* a, const int* b, size_t N) {
+        auto g = cooperative_groups::this_grid();
+        for (int idx = g.thread_rank(); idx < N; idx += g.size()) {
+            x[idx] = a[idx] * x[idx] + b[idx];
+        }
+    }
+
+updates ``x``, ``y``, and ``z`` as follows:
+
+.. code:: cuda
+
+    update<<<grid, block>>>(x, a, b, N);
+    update<<<grid, block>>>(y, a, b, N);
+    update<<<grid, block>>>(z, a, b, N);
+
+The elements of ``a`` and ``b`` are used in all kernels. For certain values of ``N``, this may prevent parts of ``a`` and ``b`` from being evicted from the L2 cache, avoiding reloading these from memory in the subsequent ``update`` kernel.
+
+With :ref:`cuda::access_property <libcudacxx-extended-api-memory-access-properties-access-property>` and :ref:`cuda::apply_access_property <libcudacxx-extended-api-memory-access-properties-apply-access-property>`, we can write kernels that specify that ``a`` and ``b`` are accessed more often in the ``pin`` kernel and with normal access in the ``unpin`` kernel:
+
+.. code:: cuda
+
+    __global__ void pin(int* a, int* b, size_t N) {
+        auto g = cooperative_groups::this_grid();
+        for (int idx = g.thread_rank(); idx < N; idx += g.size()) {
+            cuda::apply_access_property(a + idx, sizeof(int), cuda::access_property::persisting{});
+            cuda::apply_access_property(b + idx, sizeof(int), cuda::access_property::persisting{});
+        }
+    }
+
+    __global__ void unpin(int* a, int* b, size_t N) {
+        auto g = cooperative_groups::this_grid();
+        for (int idx = g.thread_rank(); idx < N; idx += g.size()) {
+            cuda::apply_access_property(a + idx, sizeof(int), cuda::access_property::normal{});
+            cuda::apply_access_property(b + idx, sizeof(int), cuda::access_property::normal{});
+        }
+    }
+
+which we can launch before and after the ``update`` kernels:
+
+.. code:: cuda
+
+   pin<<<grid, block>>>(a, b, N);
+   update<<<grid, block>>>(x, a, b, N);
+   update<<<grid, block>>>(y, a, b, N);
+   update<<<grid, block>>>(z, a, b, N);
+   unpin<<<grid, block>>>(a, b, N);
+
+This does not require modifying the ``update`` kernel, and for certain values of ``N`` prevents ``a`` and ``b`` from having to be re-loaded from memory.
+
+The ``pin`` and ``unpin`` kernels can be fused into the kernels for the ``x`` and ``z`` updates by modifying these kernels.
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory_access_properties/associate_access_property.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory_access_properties/associate_access_property.rst
new file mode 100644
index 00000000..4889c6b5
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory_access_properties/associate_access_property.rst
@@ -0,0 +1,49 @@
+.. _libcudacxx-extended-api-memory-access-properties-associate-access-property:
+
+``cuda::associate_access_property``
+===================================
+
+Defined in header ``<cuda/annotated_ptr>``.
+
+.. code:: cuda
+
+   template <typename T, typename Property>
+   [[nodiscard]] __host__ __device__
+   T* associate_access_property(T* ptr, Property prop) noexcept;
+
+Associate an :ref:`cuda::access_property <libcudacxx-extended-api-memory-access-properties-access-property>` to the input pointer, such that subsequent memory operations with the returned pointer *or* pointers derived from it *may* apply the access property.
+
+-  The "association" is *not* part of the value representation of the pointer.
+-  The compiler is allowed to drop the association; it does not have a functional consequence.
+-  The association *may* hold through simple expressions, sequence of simple statements, or fully inlined function
+   calls where the pointer value or C++ reference is provably unchanged; this includes offset pointers used for
+   array access.
+-  The association is *not* expected to hold through the ABI of an unknown function call, e.g., when the pointer is
+   passed through a separately-compiled function interface, unless link-time optimizations are used.
+
+**Constraints**
+
+- ``Property`` is convertible to :ref:`cuda::access_property <libcudacxx-extended-api-memory-access-properties-access-property>`.
+
+**Preconditions**:
+
+- If ``Property`` is :ref:`cuda::access_property::shared <libcudacxx-extended-api-memory-access-properties-access-property-shared>`, then it must be valid to cast the generic pointer ``ptr`` to a pointer to the *shared memory* address space.
+
+- If ``Property`` is one of :ref:`cuda::access_property::global <libcudacxx-extended-api-memory-access-properties-access-property-global>`, :ref:`cuda::access_property::persisting <libcudacxx-extended-api-memory-access-properties-access-property-persisting>`, :ref:`cuda::access_property::normal <libcudacxx-extended-api-memory-access-properties-access-property-normal>`, or     :ref:`cuda::access_property::streaming <libcudacxx-extended-api-memory-access-properties-access-property-streaming>`, then it must be valid to cast the generic pointer ``ptr`` to a pointer to the *global memory* address space.
+
+*Note*: currently ``associate_access_property`` is ignored by nvcc and nvc++ on the host.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/cooperative_groups.h>
+
+    __global__ void memcpy_kernel(const int* in, int* out) {
+        __shared__ int smem[N];
+        auto in1          = cuda::associate_access_property(in, cuda::access_property::streaming{});
+        auto idx          = cooperative_groups::this_grid().thread_rank();
+        smem[threadIdx.x] = in1[idx]; // streaming access
+        // compute...
+    }
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory_model.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory_model.rst
new file mode 100644
index 00000000..261e935e
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory_model.rst
@@ -0,0 +1,222 @@
+.. _libcudacxx-extended-api-memory-model:
+
+Memory model
+============
+
+Standard C++ presents a view that the cost to synchronize threads is uniform and low.
+
+CUDA C++ is different: the cost to synchronize threads grows as threads are further apart. It is low across threads
+within a block, but high across arbitrary threads in the system running on multiple GPUs and CPUs.
+
+To account for non-uniform thread synchronization costs that are not always low, CUDA C++ extends the standard C++
+memory model and concurrency facilities in the ``cuda::`` namespace with **thread scopes**, retaining the syntax and
+semantics of standard C++ by default.
+
+.. _libcudacxx-extended-api-memory-model-thread-scopes:
+
+Thread Scopes
+-------------
+
+A **thread scope** specifies the kind of threads that can synchronize with each other using a synchronization primitive such
+as :ref:`atomic <libcudacxx-extended-api-synchronization-atomic>` or
+:ref:`barrier <libcudacxx-extended-api-synchronization-barrier>`.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   enum thread_scope {
+     thread_scope_system,
+     thread_scope_device,
+     thread_scope_block,
+     thread_scope_thread
+   };
+
+   }  // namespace cuda
+
+Scope Relationships
+~~~~~~~~~~~~~~~~~~~
+
+Each program thread is related to each other program thread by one or more thread scope relations:
+   - Each thread in the system is related to each other thread in the system by the *system* thread scope:
+     ``thread_scope_system``.
+   - Each GPU thread is related to each other GPU thread in the same CUDA device and within the same `memory
+     synchronization domain <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#memory-synchronization-domains>`__
+     by the *device* thread scope: ``thread_scope_device``.
+   - Each GPU thread is related to each other GPU thread in the same CUDA thread block by the *block* thread scope:
+     ``thread_scope_block``.
+   - Each thread is related to itself by the ``thread`` thread scope: ``thread_scope_thread``.
+
+Synchronization primitives
+--------------------------
+
+Types in namespaces ``std::`` and ``cuda::std::`` have the same behavior as corresponding types in namespace ``cuda::``
+when instantiated with a scope of ``cuda::thread_scope_system``.
+
+Atomicity
+---------
+
+An atomic operation is atomic at the scope it specifies if:
+
+   - it specifies a scope other than ``thread_scope_system``, **or**
+   - the scope is ``thread_scope_system`` and:
+
+      -  it affects an object in `system allocated memory <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#um-unified-memory-programming-hd>`__ and `pageableMemoryAccess <https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1gg49e2f8c2c0bd6fe264f2fc970912e5cddc80992427a92713e699953a6d249d6f>`__ is ``1`` [0],  **or**
+      -  it affects an object in `managed
+         memory <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#um-unified-memory-programming-hd>`__
+         and
+         `concurrentManagedAccess <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_116f9619ccc85e93bc456b8c69c80e78b>`__
+         is ``1``, **or**
+      -  it affects an object in `mapped
+         memory <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#mapped-memory>`__ and
+         `hostNativeAtomicSupported <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_1ef82fd7d1d0413c7d6f33287e5b6306f>`__
+         is ``1``, **or**
+      -  it is a load or store that affects a naturally-aligned object of
+         sizes ``1``, ``2``, ``4``, ``8``, or ``16`` bytes on `mapped
+         memory <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#mapped-memory>`__ [1],
+         **or**
+      -  it affects an object in GPU memory, only GPU threads access it, and
+          - `*val` returned from `cudaDeviceGetP2PAttribute(&val, cudaDevP2PAttrNativeAtomicSupported, srcDev, dstDev) <https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1g2f597e2acceab33f60bd61c41fea0c1b>`__ between each accessing `srcDev` and the GPU where the object resides, `dstDev`, is ``1``, or
+          - only GPU threads from a single GPU concurrently access it.
+
+.. note::
+   - [0] If `PageableMemoryAccessUsesHostPagetables <https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1gg49e2f8c2c0bd6fe264f2fc970912e5cdc228cf8983c97d0e035da72a71494eaa>`__ is ``0`` then atomic operations to memory mapped file or ``hugetlbfs`` allocations are not atomic.
+   - [1] If `hostNativeAtomicSupported <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_1ef82fd7d1d0413c7d6f33287e5b6306f>`__ is ``0``, atomic load or store operations at system scope that affect a
+     naturally-aligned 16-byte wide object in
+     `unified memory <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#um-unified-memory-programming-hd>`__ or
+     `mapped memory <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#mapped-memory>`__ require system
+     support. NVIDIA is not aware of any system that lacks this support and there is no CUDA API query available to
+     detect such systems.
+
+Refer to the `CUDA programming guide <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html>`__
+for more information on
+`system allocated memory <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#um-unified-memory-programming-hd>`__,
+`managed memory <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#um-unified-memory-programming-hd>`__,
+`mapped memory <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#mapped-memory>`__,
+CPU memory, and GPU memory.
+
+Data Races
+----------
+
+Modify `intro.races paragraph 21 <https://eel.is/c++draft/intro.races#21>`__ of ISO/IEC IS 14882 (the C++ Standard)
+as follows:
+
+   The execution of a program contains a data race if it contains two potentially concurrent conflicting actions, at
+   least one of which is not atomic **at a scope that includes the thread that performed the other operation**, and neither
+   happens before the other, except for the special case for signal handlers described below.
+   Any such data race results in undefined behavior. […]
+
+Modify `thread.barrier.class paragraph 4 <https://eel.is/c++draft/thread.barrier.class#4>`__ of ISO/IEC IS
+14882 (the C++ Standard) as follows
+
+   4. Concurrent invocations of the member functions of ``barrier``, other than its destructor, do not introduce data
+   races **as if they were atomic operations**. […]
+
+Modify `thread.latch.class paragraph 2 <https://eel.is/c++draft/thread.latch.class#2>`__ of ISO/IEC IS 14882
+(the C++ Standard) as follows:
+
+   2. Concurrent invocations of the member functions of ``latch``, other than its destructor, do not introduce data
+   races **as if they were atomic operations**. […]
+
+Modify `thread.sema.cnt paragraph 3 <https://eel.is/c++draft/thread.sema.cnt#3>`__ of ISO/IEC IS 14882
+(the C++ Standard) as follows:
+
+   3. Concurrent invocations of the member functions of ``counting_semaphore``, other than its destructor, do not
+   introduce data races **as if they were atomic operations**.
+
+Modify `thread.stoptoken.intro paragraph 5 <https://eel.is/c++draft/thread#stoptoken.intro-5>`__ of ISO/IEC IS
+14882 (the C++ Standard) as follows:
+
+   Calls to the functions ``request_stop``, ``stop_requested``, and ``stop_possible`` do not introduce data
+   races **as if they were atomic operations**. […]
+
+Modify `atomics.fences paragraph 2 through 4 <https://eel.is/c++draft/atomics.fences#2>`__ of ISO/IEC IS 14882 (the
+C++ Standard) as follows:
+
+   A release fence A synchronizes with an acquire fence B if there exist atomic operations X and Y, both operating on
+   some atomic object M, such that A is sequenced before X, X modifies M, Y is sequenced before B, and Y reads the value
+   written by X or a value written by any side effect in the hypothetical release sequence X would head if it were a
+   release operation, **and each operation (A, B, X, and Y) specifies a scope that includes the thread that performed
+   each other operation**.
+
+   A release fence A synchronizes with an atomic operation B that performs an acquire operation on an atomic object M if
+   there exists an atomic operation X such that A is sequenced before X, X modifies M, and B reads the value written by
+   X or a value written by any side effect in the hypothetical release sequence X would head if it were a release
+   operation, **and each operation (A, B, and X) specifies a scope that includes the thread that performed each other
+   operation**.
+
+   An atomic operation A that is a release operation on an atomic object M synchronizes with an acquire fence B if
+   there exists some atomic operation X on M such that X is sequenced before B and reads the value written by A or a
+   value written by any side effect in the release sequence headed by A, **and each operation (A, B, and X) specifies
+   a scope that includes the thread that performed each other operation**.
+
+.. _libcudacxx-extended-api-memory-model-message-passing:
+
+Example: Message Passing
+------------------------
+
+The following example passes a message stored to the ``x`` variable by a
+thread in block ``0`` to a thread in block ``1`` via the flag ``f``:
+
+.. code:: cpp
+
+   int x = 0;
+   int f = 0;
+
+.. list-table::
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Thread 0 Block 0
+     - Thread 0 Block 1
+   * -
+       .. code:: cpp
+
+          x = 42;
+          cuda::atomic_ref<int, cuda::thread_scope_device> flag(f);
+          flag.store(1, memory_order_release);
+     -
+       .. code:: cpp
+
+          cuda::atomic_ref<int, cuda::thread_scope_device> flag(f);
+          while(flag.load(memory_order_acquire) != 1);
+          assert(x == 42);
+
+In the following variation of the previous example, two threads
+concurrently access the ``f`` object without synchronization, which
+leads to a **data race**, and exhibits **undefined behavior**:
+
+.. code:: cpp
+
+   int x = 0;
+   int f = 0;
+
+.. list-table::
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Thread 0 Block 0
+     - Thread 0 Block 1
+   * -
+       .. code:: cpp
+
+          x = 42;
+          cuda::atomic_ref<int, cuda::thread_scope_block> flag(f);
+          flag.store(1, memory_order_release); // UB: data race
+     -
+       .. code:: cpp
+
+          cuda::atomic_ref<int, cuda::thread_scope_device> flag(f);
+          while(flag.load(memory_order_acquire) != 1); // UB: data race
+          assert(x == 42);
+
+While the memory operations on ``f`` - the store and the loads - are
+atomic, the scope of the store operation is "block scope". Since the
+store is performed by Thread 0 of Block 0, it only includes all other
+threads of Block 0. However, the thread doing the loads is in Block 1,
+i.e., it is not in a scope included by the store operation performed in
+Block 0, causing the store and the load to not be "atomic", and
+introducing a data-race.
+
+For more examples see the `PTX memory consistency model litmus
+tests <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#axioms>`__.
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory_resource.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory_resource.rst
new file mode 100644
index 00000000..99ea5322
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory_resource.rst
@@ -0,0 +1,69 @@
+.. _libcudacxx-extended-api-memory-resources:
+.. _libcudacxx-memory-resource-async:
+
+Memory Resources
+================
+
+.. toctree::
+   :maxdepth: 1
+
+   memory_resource/properties
+   Resources <memory_resource/resource>
+   Type-erased wrappers <memory_resource/wrappers>
+   Resource utilities <memory_resource/resource_utilities>
+
+The ``<cuda/memory_resource>`` header provides a standard C++ interface for *heterogeneous*, *stream-ordered* memory
+allocation tailored to the needs of CUDA C++ developers. This design builds off of the success of the `RAPIDS Memory Manager (RMM) <https://github.com/rapidsai/rmm>`__
+project and evolves the design based on lessons learned.
+
+``<cuda/memory_resource>`` is not intended to replace RMM, but instead moves the definition of the memory allocation
+interface to a more centralized home in CCCL. RMM will remain as a collection of implementations of the ``cuda::mr``
+interfaces.
+
+At a high level, the header provides:
+
+.. list-table::
+   :widths: 30 50 20
+   :header-rows: 0
+
+   * - :ref:`cuda::get_property <libcudacxx-extended-api-memory-resources-properties>`
+     - Infrastructure to tag a user defined type with a given property
+     - stable CCCL 3.1.0 / CUDA 13.1, experimental CCCL 2.2.0 / CUDA 12.3
+   * - :ref:`cuda::mr::{synchronous_}resource <libcudacxx-extended-api-memory-resources-resource>` and
+       :ref:`cuda::mr::{synchronous_}resource_with <libcudacxx-extended-api-memory-resources-resource>`
+     - Concepts that provide proper constraints for arbitrary memory resources.
+     - stable CCCL 3.1.0 / CUDA 13.1, experimental CCCL 2.2.0 / CUDA 12.3
+   * - :ref:`cuda::mr::{synchronous_}resource_ref <libcudacxx-extended-api-memory-resources-resource-ref>`
+     - A non-owning type-erased memory resource wrapper that enables consumers to specify properties of resources that they expect.
+     - stable CCCL 3.2.0 / CUDA 13.2, experimental CCCL 2.2.0 / CUDA 12.3
+   * - :ref:`cuda::mr::any_resource <libcudacxx-extended-api-memory-resources-any-resource>` and
+       :ref:`cuda::mr::any_synchronous_resource <libcudacxx-extended-api-memory-resources-any-synchronous-resource>`
+     - Owning type-erased wrappers for stream-ordered and synchronous resources.
+     - stable CCCL 3.2.0 / CUDA 13.2, experimental CCCL 2.2.0 / CUDA 12.3
+   * - :ref:`try_get_property <libcudacxx-extended-api-memory-resources-try-get-property>`
+     - Property query helper for type-erased resource wrappers.
+     - stable CCCL 3.2.0 / CUDA 13.2
+   * - :ref:`cuda::mr::resource_cast <libcudacxx-extended-api-memory-resources-resource-cast>` and
+       :ref:`cuda::mr::dynamic_resource_cast <libcudacxx-extended-api-memory-resources-dynamic-resource-cast>`
+     - Cast helpers for type-erased resource wrappers.
+     - stable CCCL 3.3.0 / CUDA 13.3
+   * - :ref:`cuda::mr::shared_resource <libcudacxx-extended-api-memory-resources-shared-resource>`
+     - Reference-counted wrapper to share a resource instance across objects.
+     - stable CCCL 3.2.0 / CUDA 13.2, experimental CCCL 2.2.0 / CUDA 12.3
+   * - :ref:`cuda::mr::synchronous_resource_adapter <libcudacxx-extended-api-memory-resources-synchronous-adapter>`
+     - Adapter that enables synchronous resources to work with streams.
+     - stable CCCL 3.2.0 / CUDA 13.2, experimental CCCL 2.2.0 / CUDA 12.3
+   * - :ref:`cuda::mr::get_memory_resource <libcudacxx-extended-api-memory-resources-get-memory-resource>`
+     - Customization point for retrieving a memory resource from an object or execution environment.
+     - stable CCCL 3.2.0 / CUDA 13.2
+
+These features are an evolution of `std::pmr::memory_resource <https://en.cppreference.com/w/cpp/header/memory_resource>`__
+that was introduced in C++17. While ``std::pmr::memory_resource`` provides a polymorphic memory resource that can be
+adopted through inheritance, it is not properly suited for heterogeneous systems.
+
+With the current design it ranges from cumbersome to impossible to verify whether a memory resource provides allocations
+that are e.g. accessible on device, or whether it can utilize other allocation mechanisms.
+
+To better support asynchronous CUDA `stream-ordered allocations <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#stream-ordered-memory-allocator>`__
+libcu++ provides :cpp:class:`cuda::stream_ref <cuda::stream_ref>` as a wrapper around
+``cudaStream_t``. The definition of ``cuda::stream_ref`` can be found in the ``<cuda/stream>`` header.
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory_resource/properties.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory_resource/properties.rst
new file mode 100644
index 00000000..06b62ae9
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory_resource/properties.rst
@@ -0,0 +1,121 @@
+.. _libcudacxx-extended-api-memory-resources-properties:
+
+Properties
+----------
+
+Modern C++ programs use the type system to verify statically known properties of the code. It is undesirable to use
+run-time assertions to verify whether a function that accesses device memory is passed a memory resource that provides
+device-accessible allocations. A property system is provided for this purpose.
+
+To tell the compiler that memory provided by ``my_memory_resource`` is device accessible, use the
+``cuda::mr::device_accessible`` tag type with a free function ``get_property`` to declare that the resource provides
+device-accessible memory.
+
+.. code:: cpp
+
+   struct my_memory_resource {
+       friend constexpr void get_property(const my_memory_resource&, cuda::mr::device_accessible) noexcept {}
+   };
+
+A library can constrain interfaces with ``cuda::has_property`` to require that a passed memory resource provides the
+right kind of memory
+
+`See it on Godbolt <https://godbolt.org/z/5hjoEnerb>`__
+
+.. code:: cpp
+
+   template<class MemoryResource>
+       requires cuda::has_property<MemoryResource, cuda::mr::device_accessible>
+   void function_that_dispatches_to_device(MemoryResource& resource);
+
+If C++20 is not available, the function can instead be constrained via SFINAE
+
+`See it on Godbolt  <https://godbolt.org/z/11sGbr333>`__
+
+.. code:: cpp
+
+   template<class MemoryResource, class = cuda::std::enable_if_t<cuda::has_property<MemoryResource, cuda::mr::device_accessible>>>
+   void function_that_dispatches_to_device(MemoryResource& resource);
+
+For now, libcu++ provides various commonly used properties:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 30 70
+
+   * - Property
+     - Meaning
+   * - ``cuda::mr::device_accessible``
+     - Memory allocated using the resource is accessible from device.
+   * - ``cuda::mr::host_accessible``
+     - Memory allocated using the resource is accessible from host.
+
+``cuda::mr::default_cuda_malloc_alignment`` and ``cuda::mr::default_cuda_malloc_host_alignment`` provide the default
+alignment values used by the CUDA allocation APIs.
+
+More properties may be added as the library and the hardware capabilities evolve. However, a user library is free to
+define custom properties.
+
+Resources may expose a ``default_queries`` member type to advertise a fixed set of properties for APIs that use default
+property queries, such as ``cuda::make_buffer``. Use ``cuda::mr::properties_list<Properties...>`` to define that list.
+
+.. code:: cpp
+
+   struct my_memory_resource {
+     using default_queries = cuda::mr::properties_list<cuda::mr::device_accessible>;
+
+     friend constexpr void get_property(const my_memory_resource&, cuda::mr::device_accessible) noexcept {}
+   };
+
+   static_assert(my_memory_resource::default_queries::has_property(cuda::mr::device_accessible{}));
+
+Note that currently the libcu++ provided properties are stateless. However, properties can also provide stateful
+information that is retrieved via the ``get_property`` free function. In order to communicate the desired type of the
+carried state, a stateful property must define the ``value_type`` alias. A library can constrain interfaces that
+require a stateful property with ``cuda::has_property_with`` as shown in the example below
+
+`See it on Godbolt  <https://godbolt.org/z/11sGbr333>`__
+
+.. code:: cpp
+
+   struct required_alignment{
+       using value_type = std::size_t;
+   };
+   struct my_memory_resource {
+       friend constexpr std::size_t get_property(const my_memory_resource& resource, required_alignment) noexcept { return resource.required_alignment; }
+
+       std::size_t required_alignment;
+   };
+   static_assert(cuda::has_property_with<my_memory_resource, required_alignment, std::size_t>);
+
+   template<class MemoryResource>
+   void* allocate_check_alignment(MemoryResource& resource, std::size_t size) {
+       if constexpr(cuda::has_property_with<MemoryResource, required_alignment, std::size_t>) {
+           return resource.allocate_sync(size, get_property(resource, required_alignment{}));
+       } else {
+           // Use default alignment
+           return resource.allocate_sync(size, 42);
+       }
+   }
+
+In generic code it is often desirable to propagate properties from a base type to a derived type, without knowing the
+base type at all. This common use case is covered by ``cuda::forward_property``, a simple CRTP template.
+
+.. code:: cpp
+
+   template<class MemoryResource>
+   class logging_resource : cuda::forward_property<logging_resource<MemoryResource>, MemoryResource> {
+       MemoryResource base;
+   public:
+       void* allocate_sync(std::size_t size, std::size_t alignment) {
+           std::cout << "allocating\n";
+           return base.allocate_sync(size, alignment);
+       }
+       void deallocate_sync(void* ptr, std::size_t size, std::size_t alignment) noexcept {
+           std::cout << "deallocating\n";
+           return base.deallocate_sync(ptr, size, alignment);
+       }
+
+       MemoryResource& upstream_resource() noexcept { return base; }
+       const MemoryResource& upstream_resource() const noexcept { return base; }
+   };
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory_resource/resource.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory_resource/resource.rst
new file mode 100644
index 00000000..478b8bb5
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory_resource/resource.rst
@@ -0,0 +1,154 @@
+.. _libcudacxx-extended-api-memory-resources-resource:
+.. _libcudacxx-extended-api-memory-resources-synchronous-resource:
+
+The ``cuda::mr::resource`` and ``cuda::mr::synchronous_resource`` concepts
+--------------------------------------------------------------------------
+
+The `std::pmr::memory_resource <https://en.cppreference.com/w/cpp/header/memory_resource>`__ feature provides only a
+single ``allocate`` interface, which is sufficient for homogeneous memory systems. However, CUDA provides both
+synchronous and `stream-ordered allocation <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#stream-ordered-memory-allocator>`__.
+
+With `std::pmr::memory_resource <https://en.cppreference.com/w/cpp/header/memory_resource>`__ there is no way to tell
+whether a memory resource can utilize stream-ordered allocations. Even if the application knows it can, there is no way
+to properly tell the memory resource to use stream-ordered allocation. Ideally, this should not be something discovered
+through an assert at run time, but should be checked by the compiler.
+
+Because asynchronous memory management is critical for performance, ``cuda::mr::resource`` includes the stream-ordered interface provided by ``allocate`` / ``deallocate``.
+For cases where stream-ordered allocation is not possible, ``cuda::mr::synchronous_resource`` is provided.
+
+The ``cuda::mr::synchronous_resource`` concept provides basic type checks to ensure that a given memory resource provides the
+expected ``allocate_sync`` / ``deallocate_sync`` interface and is also equality comparable, which covers the whole API surface of
+`std::pmr::memory_resource <https://en.cppreference.com/w/cpp/header/memory_resource>`__.
+See below for different memory resources and potential pitfalls.
+
+To demonstrate, the following example defines several resources, only some of which are valid implementations of the
+``cuda::mr::synchronous_resource`` concept. The ``static_assertions`` will result in compile-time errors for the invalid resources.
+
+.. code:: cpp
+
+   struct valid_resource {
+     void* allocate_sync(std::size_t, std::size_t) { return nullptr; }
+     void deallocate_sync(void*, std::size_t, std::size_t) noexcept {}
+     bool operator==(const valid_resource&) const { return true; }
+     // NOTE: C++20 thankfully added operator rewrite rules so defining operator!= is not required.
+     // However, if compiled with C++14 / C++17, operator != must also be defined.
+     bool operator!=(const valid_resource&) const { return false; }
+   };
+   static_assert(cuda::mr::synchronous_resource<valid_resource>);
+
+   struct invalid_argument {};
+   struct invalid_allocate_argument {
+     void* allocate_sync(invalid_argument, std::size_t) { return nullptr; }
+     void deallocate_sync(void*, std::size_t, std::size_t) noexcept {}
+     bool operator==(const invalid_allocate_argument&) { return true; }
+   };
+   static_assert(!cuda::mr::synchronous_resource<invalid_allocate_argument>);
+
+   struct invalid_allocate_return {
+     int allocate_sync(std::size_t, std::size_t) { return 42; }
+     void deallocate_sync(void*, std::size_t, std::size_t) noexcept {}
+     bool operator==(const invalid_allocate_return&) { return true; }
+   };
+   static_assert(!cuda::mr::synchronous_resource<invalid_allocate_return>);
+
+   struct invalid_deallocate_argument {
+     void* allocate_sync(std::size_t, std::size_t) { return nullptr; }
+     void deallocate_sync(void*, invalid_argument, std::size_t) noexcept {}
+     bool operator==(const invalid_deallocate_argument&) { return true; }
+   };
+   static_assert(!cuda::mr::synchronous_resource<invalid_deallocate_argument>);
+
+   struct non_comparable {
+     void* allocate_sync(std::size_t, std::size_t) { return nullptr; }
+     void deallocate_sync(void*, std::size_t, std::size_t) noexcept {}
+   };
+   static_assert(!cuda::mr::synchronous_resource<non_comparable>);
+
+   struct non_eq_comparable {
+     void* allocate_sync(std::size_t, std::size_t) { return nullptr; }
+     void deallocate_sync(void*, std::size_t, std::size_t) noexcept {}
+     bool operator!=(const non_eq_comparable&) { return false; }
+   };
+   static_assert(!cuda::mr::synchronous_resource<non_eq_comparable>);
+
+In addition to the `std::pmr::memory_resource <https://en.cppreference.com/w/cpp/header/memory_resource>`_ interface the
+``cuda::mr::resource`` concept verifies that a memory resource also satisfies the ``allocate`` /
+``deallocate`` interface. Requiring both the PMR interface and the async interface is a deliberate design decision.
+
+.. code:: cpp
+
+   struct valid_resource {
+     void* allocate_sync(std::size_t, std::size_t) { return nullptr; }
+     void deallocate_sync(void*, std::size_t, std::size_t) noexcept {}
+     void* allocate(cuda::stream_ref, std::size_t, std::size_t) { return nullptr; }
+     void deallocate(cuda::stream_ref, void*, std::size_t, std::size_t) {}
+     bool operator==(const valid_resource&) const { return true; }
+     bool operator!=(const valid_resource&) const { return false; }
+   };
+   static_assert(cuda::mr::resource<valid_resource>);
+
+A library can easily decide whether to use the async interface:
+
+.. code:: cpp
+
+   template<class MemoryResource>
+       requires cuda::mr::synchronous_resource<MemoryResource>
+   void* allocate_maybe_sync(cuda::stream_ref stream, MemoryResource& resource, std::size_t size, std::size_t align) {
+       if constexpr(cuda::mr::resource<MemoryResource>) {
+           return resource.allocate(stream, size, align);
+       } else {
+           return resource.allocate_sync(size, align);
+       }
+   }
+
+.. rubric:: Putting them together
+
+Applications and libraries may want to combine type checks for arbitrary properties with the ``{synchronous_}resource``
+concept. The ``{synchronous_}resource_with`` concept allows checking resources for arbitrary properties.
+
+.. code:: cpp
+
+   struct required_alignment{
+       using value_type = std::size_t;
+   };
+   struct my_memory_resource {
+       void* allocate_sync(std::size_t, std::size_t) { return nullptr; }
+       void deallocate_sync(void*, std::size_t, std::size_t) noexcept {}
+       bool operator==(const my_memory_resource&) const { return true; }
+       bool operator!=(const my_memory_resource&) const { return false; }
+
+       friend constexpr std::size_t get_property(const my_memory_resource& resource, required_alignment) noexcept { return resource.required_alignment; }
+
+       std::size_t required_alignment;
+   };
+
+   constexpr std::size_t my_default_alignment = 16;
+
+   template<class MemoryResource>
+       requires cuda::mr::synchronous_resource<MemoryResource>
+   void* allocate_maybe_sync_check_alignment(MemoryResource& resource, cuda::stream_ref stream, std::size_t size) {
+       if constexpr(cuda::mr::resource_with<MemoryResource, required_alignment>) {
+           return resource.allocate(stream, size, get_property(resource, required_alignment{}));
+       } else if constexpr (cuda::mr::resource<MemoryResource>) {
+           return resource.allocate(stream, size, my_default_alignment);
+       } else if constexpr (cuda::mr::synchronous_resource_with<MemoryResource, required_alignment>) {
+           return resource.allocate_sync(size, get_property(resource, required_alignment{}));
+       } else {
+           return resource.allocate_sync(size, my_default_alignment);
+       }
+   }
+
+   // Potentially more concise
+   template<class MemoryResource>
+       requires cuda::mr::synchronous_resource<MemoryResource>
+   void* allocate_maybe_sync_check_alignment2(MemoryResource& resource, cuda::stream_ref stream, std::size_t size) {
+       std::size_t align = my_default_alignment;
+       if constexpr(cuda::mr::synchronous_resource_with<MemoryResource, required_alignment>) {
+           align = get_property(resource, required_alignment{});
+       }
+       if constexpr(cuda::mr::resource<MemoryResource>) {
+           return resource.allocate(stream, size, align);
+       } else {
+           return resource.allocate_sync(size, align);
+       }
+   }
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory_resource/resource_utilities.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory_resource/resource_utilities.rst
new file mode 100644
index 00000000..68a8174f
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory_resource/resource_utilities.rst
@@ -0,0 +1,110 @@
+.. _libcudacxx-extended-api-memory-resources-utilities:
+
+Resource utilities
+------------------
+
+The ``cuda::mr`` memory resource system includes utilities that help manage resource lifetime and adapt synchronous
+resources for stream-ordered usage. These utilities complement the type-erased wrappers in :ref:`resource wrappers <libcudacxx-extended-api-memory-resources-resource-ref>`.
+
+shared_resource
+~~~~~~~~~~~~~~~
+.. _libcudacxx-extended-api-memory-resources-shared-resource:
+.. _cccl-runtime-memory-resource-shared-resource:
+
+``cuda::mr::shared_resource`` holds a reference-counted instance of a memory resource, allowing resources to be passed
+around with shared ownership semantics while avoiding lifetime issues. This is useful when multiple objects, like
+:ref:`cuda::buffer <cccl-runtime-buffer-buffer>`, must share the same resource instance.
+
+``cuda::mr::make_shared_resource<Resource>(args...)`` constructs a ``shared_resource<Resource>`` that owns a newly
+constructed ``Resource``. ``Resource`` must satisfy ``cuda::mr::synchronous_resource``. The resulting
+``shared_resource`` supports stream-ordered allocation when ``Resource`` also satisfies ``cuda::mr::resource``.
+
+.. code:: cpp
+
+   #include <cuda/devices>
+   #include <cuda/memory_resource>
+   #include <cuda/stream>
+
+   void use_shared_resource(cuda::stream_ref stream) {
+     // Create a shared resource
+     auto shared_mr = cuda::mr::shared_resource{
+       cuda::std::in_place_type<cuda::device_memory_pool>,
+       cuda::devices[0]
+     };
+
+     // Copy the shared resource (shares ownership)
+     auto shared_mr2 = shared_mr;
+
+     // Both can be used independently
+     void* ptr1 = shared_mr.allocate(stream, 1024, 16);
+     void* ptr2 = shared_mr2.allocate(stream, 2048, 16);
+
+     shared_mr.deallocate(stream, ptr1, 1024, 16);
+     shared_mr2.deallocate(stream, ptr2, 2048, 16);
+     // Resources are automatically cleaned up when last reference is destroyed
+   }
+
+get_memory_resource
+~~~~~~~~~~~~~~~~~~~
+.. _libcudacxx-extended-api-memory-resources-get-memory-resource:
+
+``cuda::mr::get_memory_resource`` is a customization point object that retrieves a memory resource from an object or
+execution environment.
+
+``cuda::mr::memory_resource_base<Derived>`` is a CRTP helper that makes a resource queryable by
+``cuda::mr::get_memory_resource`` when the resource is used as part of a composed execution environment.
+
+It supports three cases:
+
+- If the object is itself a synchronous resource, it returns that object.
+- If the object has a ``get_memory_resource()`` member, it calls that member.
+- Otherwise, if an execution environment supports ``query(cuda::mr::get_memory_resource)``, it calls that query.
+
+.. code:: cpp
+
+   #include <cuda/memory_resource>
+   #include <cuda/stream>
+   #include <cuda/std/type_traits>
+
+   template <class Env>
+   void* allocate_from_env(Env& env, cuda::stream_ref stream, std::size_t size, std::size_t align) {
+     auto&& resource = cuda::mr::get_memory_resource(env);
+     using resource_type = cuda::std::remove_reference_t<decltype(resource)>;
+     if constexpr (cuda::mr::resource<resource_type>) {
+       return resource.allocate(stream, size, align);
+     } else {
+       return resource.allocate_sync(size, align);
+     }
+   }
+
+synchronous_resource_adapter
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. _libcudacxx-extended-api-memory-resources-synchronous-adapter:
+.. _cccl-runtime-memory-resource-synchronous-adapter:
+
+``cuda::mr::synchronous_resource_adapter`` adapts a synchronous memory resource to work as a stream-ordered resource.
+If the underlying resource already supports stream-ordered allocation, it passes through the calls. Otherwise, it uses
+synchronous allocation/deallocation with proper stream synchronization.
+
+.. code:: cpp
+
+   #include <cuda/memory_resource>
+   #include <cuda/stream>
+
+   void adapt_sync_resource(cuda::stream_ref stream) {
+     // Create a synchronous resource
+     auto sync_mr = cuda::mr::legacy_pinned_memory_resource{};
+
+     // Adapt it to work with streams
+     auto adapted = cuda::mr::synchronous_resource_adapter{
+       sync_mr
+     };
+
+     // Now can use with stream (will synchronize internally)
+     void* ptr = adapted.allocate(stream, 1024, 16);
+
+     // Use memory...
+
+     // Deallocate (will synchronize stream before deallocation)
+     adapted.deallocate(stream, ptr, 1024, 16);
+   }
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/memory_resource/wrappers.rst b/cccl_upstream/docs/libcudacxx/extended_api/memory_resource/wrappers.rst
new file mode 100644
index 00000000..d4792644
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/memory_resource/wrappers.rst
@@ -0,0 +1,253 @@
+.. _libcudacxx-extended-api-memory-resources-wrappers:
+.. _libcudacxx-extended-api-memory-resources-resource-ref:
+.. _cccl-runtime-memory-resource-resource-ref:
+.. _libcudacxx-memory-resource-any-resource:
+.. _libcudacxx-memory-resource-any-async-resource:
+
+Type-erased resource wrappers
+-----------------------------
+
+With the property design depicted in :ref:`cuda::get_property <libcudacxx-extended-api-memory-resources-properties>`,
+a library has flexibility in checking constraints and querying custom properties. However, there is also a cost in
+providing function templates for a potentially wide range of inputs. Depending on the number of different memory
+resources, both compile time and binary size might increase considerably.
+
+The type-erased wrappers let you coalesce such APIs into a single function. Both ``resource_ref`` and ``any_resource``
+preserve property constraints while erasing the concrete resource type.
+
+.. list-table::
+   :widths: 30 20 50
+   :header-rows: 1
+
+   * - Wrapper
+     - Ownership
+     - Typical use
+   * - ``cuda::mr::resource_ref``
+     - Non-owning
+     - Accept references/pointers without extending lifetime
+   * - ``cuda::mr::any_resource``
+     - Owning
+     - Store a resource with the object that uses it
+
+Common usage
+~~~~~~~~~~~~
+
+Both wrappers provide a non-templated API surface for allocation and property queries. Choose which wrapper to use based
+on ownership, then the usage patterns are the same.
+
+``resource_ref`` is constructible from any non-const reference or pointer to a memory resource that satisfies
+``cuda::mr::resource``. ``any_resource`` is constructible from a resource object and takes ownership of it.
+
+Properties may be passed to both wrappers just as with ``cuda::mr::resource_with``.
+
+.. code:: cpp
+
+   void* do_allocate(cuda::mr::resource_ref<> resource, cuda::stream_ref stream, std::size_t size, std::size_t align) {
+       return resource.allocate(stream, size, align);
+   }
+
+   void* do_allocate_owned(cuda::mr::any_resource<> resource, cuda::stream_ref stream, std::size_t size, std::size_t align) {
+       return resource.allocate(stream, size, align);
+   }
+
+   my_memory_resource resource;
+   my_memory_resource* pointer_to_resource = &resource;
+
+   void* from_reference = do_allocate(resource, stream, 1337, 256);
+   void* from_ptr = do_allocate(pointer_to_resource, stream, 1337, 256);
+   void* from_owned = do_allocate_owned(cuda::mr::any_resource<>{resource}, stream, 1337, 256);
+
+resource_ref
+~~~~~~~~~~~~
+.. _libcudacxx-extended-api-memory-resources-resource-ref-wrapper:
+
+``cuda::mr::resource_ref`` is the non-owning, type-erased wrapper. Prefer it when the caller controls the resource
+lifetime.
+
+.. code:: cpp
+
+   struct required_alignment {
+       using value_type = std::size_t;
+   };
+   void* do_allocate_with_alignment(cuda::mr::resource_ref<required_alignment> resource, cuda::stream_ref stream, std::size_t size) {
+       return resource.allocate(stream, size, get_property(resource, required_alignment{}));
+   }
+
+However, the type erasure comes with the cost that arbitrary properties cannot be queried from either wrapper:
+
+.. code:: cpp
+
+   struct required_alignment {
+       using value_type = std::size_t;
+   };
+   void* buggy_allocate_with_alignment(cuda::mr::resource_ref<> resource, cuda::stream_ref stream, std::size_t size) {
+       if constexpr (cuda::has_property<decltype(resource), required_alignment>) { // BUG: This will always be false
+           return resource.allocate(stream, size, get_property(resource, required_alignment{}));
+       } else {
+           return resource.allocate(stream, size, my_default_alignment);
+       }
+   }
+
+So, choose wisely. If your library has a well-defined set of fixed properties that you expect to always be available,
+then ``cuda::mr::{synchronous_}resource_ref`` is an amazing tool to improve compile times and binary size. If you need a
+flexible interface then constraining a template argument through ``cuda::mr::{synchronous_}resource_with`` is the proper solution.
+
+any_resource
+~~~~~~~~~~~~
+.. _libcudacxx-extended-api-memory-resources-any-resource-wrapper:
+.. _libcudacxx-extended-api-memory-resources-any-resource:
+.. _cccl-runtime-memory-resource-any-resource:
+
+``cuda::mr::any_resource`` is the owning counterpart. It is especially suited for containers that must ensure the
+resource outlives the container.
+
+``cuda::mr::make_any_resource<Resource, Properties...>(args...)`` constructs an ``any_resource<Properties...>`` that
+owns a newly constructed ``Resource``. ``Resource`` must satisfy ``cuda::mr::resource_with<Resource, Properties...>``.
+
+.. code:: cpp
+
+   #include <cuda/devices>
+   #include <cuda/memory_resource>
+   #include <cuda/stream>
+
+   void use_any_resource(cuda::stream_ref stream) {
+     // Wrap a device memory resource
+     cuda::mr::any_resource<cuda::mr::device_accessible> resource{
+       cuda::device_default_memory_pool(cuda::devices[0])
+     };
+
+     // Allocate memory
+     void* ptr = resource.allocate(stream, 1024, 16);
+
+     // Use memory...
+
+     // Deallocate
+     resource.deallocate(stream, ptr, 1024, 16);
+   }
+
+Synchronous variants
+~~~~~~~~~~~~~~~~~~~~
+
+The synchronous wrappers mirror the same ownership split: ``synchronous_resource_ref`` is non-owning, while
+``any_synchronous_resource`` owns the resource instance.
+
+``cuda::mr::any_synchronous_resource``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. _libcudacxx-extended-api-memory-resources-any-synchronous-resource:
+.. _cccl-runtime-memory-resource-any-synchronous-resource:
+
+``cuda::mr::any_synchronous_resource`` is a type-erased wrapper for resources that satisfy
+``cuda::mr::synchronous_resource``. It can be constructed from an ``any_resource``, but only exposes synchronous
+allocation/deallocation.
+
+``cuda::mr::make_any_synchronous_resource<Resource, Properties...>(args...)`` constructs an
+``any_synchronous_resource<Properties...>`` that owns a newly constructed ``Resource``. ``Resource`` must satisfy
+``cuda::mr::synchronous_resource_with<Resource, Properties...>``.
+
+.. code:: cpp
+
+   #include <cuda/memory_resource>
+
+   void use_sync_resource() {
+     cuda::mr::any_synchronous_resource<cuda::mr::host_accessible> resource{
+       cuda::mr::legacy_pinned_memory_resource{}
+     };
+
+     // Synchronous allocation
+     void* ptr = resource.allocate_sync(1024, 16);
+
+     // Use memory...
+
+     // Synchronous deallocation
+     resource.deallocate_sync(ptr, 1024, 16);
+   }
+
+resource_cast
+~~~~~~~~~~~~~
+.. _libcudacxx-extended-api-memory-resources-resource-cast:
+
+``cuda::mr::resource_cast<T>(&resource)`` returns a pointer to the concrete resource stored in a type-erased wrapper
+when the stored resource has type ``T``. It returns ``nullptr`` when the stored type does not match.
+
+This is similar to pointer-form ``dynamic_cast<T*>`` in that callers must check for ``nullptr`` before using the
+result. Unlike ``dynamic_cast``, ``resource_cast`` matches the exact type stored in the wrapper.
+
+``resource_cast`` works with ``any_resource``, ``any_synchronous_resource``, ``resource_ref``, and
+``synchronous_resource_ref``.
+
+.. code:: cpp
+
+   #include <cuda/devices>
+   #include <cuda/memory_resource>
+   #include <cuda/stream>
+
+   void use_resource_cast(cuda::stream_ref stream) {
+     cuda::mr::any_resource<cuda::mr::device_accessible> resource{
+       cuda::device_default_memory_pool(cuda::devices[0])
+     };
+
+     if (auto* pool = cuda::mr::resource_cast<cuda::device_memory_pool_ref>(&resource)) {
+       pool->trim_to(1024 * 1024);
+
+       void* ptr = pool->allocate(stream, 1024, 16);
+       pool->deallocate(stream, ptr, 1024, 16);
+     }
+   }
+
+try_get_property
+~~~~~~~~~~~~~~~~
+.. _libcudacxx-extended-api-memory-resources-try-get-property:
+
+Type-erased wrapper conversions can discard properties from the wrapper's type. ``try_get_property(resource, property)``
+can query a property that was present before such a conversion, even when it is not listed in the resulting wrapper's
+property set. For stateless properties, it returns ``bool``. For stateful properties, it returns
+``cuda::std::optional<Property::value_type>``.
+
+.. code:: cpp
+
+   #include <cuda/memory_resource>
+
+   bool is_device_accessible(cuda::mr::any_synchronous_resource<> resource) {
+     return try_get_property(resource, cuda::mr::device_accessible{});
+   }
+
+dynamic_resource_cast
+~~~~~~~~~~~~~~~~~~~~~
+.. _libcudacxx-extended-api-memory-resources-dynamic-resource-cast:
+
+``cuda::mr::dynamic_resource_cast<Properties...>(resource)`` casts a type-erased resource wrapper to the same wrapper
+kind with a different property set. It checks the stored resource at run time and throws if the stored resource does
+not support the requested properties.
+
+.. code:: cpp
+
+   #include <cuda/memory_resource>
+
+   cuda::mr::any_resource<cuda::mr::device_accessible>
+   require_device_accessible(cuda::mr::any_resource<> resource) {
+     return cuda::mr::dynamic_resource_cast<cuda::mr::device_accessible>(cuda::std::move(resource));
+   }
+
+``cuda::mr::synchronous_resource_ref``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. _cccl-runtime-memory-resource-synchronous-resource-ref:
+
+``cuda::mr::synchronous_resource_ref`` provides the same type-erased reference behavior as ``resource_ref``, but only
+exposes synchronous allocation and deallocation.
+
+.. code:: cpp
+
+   #include <cuda/memory_resource>
+
+   // indicates that the resource must be host-accessible
+   void use_sync_ref(
+     cuda::mr::synchronous_resource_ref<cuda::mr::host_accessible> resource
+   ) {
+     // Synchronous allocation
+     void* ptr = resource.allocate_sync(1024, 16);
+
+     // Use memory...
+
+     // Synchronous deallocation
+     resource.deallocate_sync(ptr, 1024, 16);
+   }
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/numeric.rst b/cccl_upstream/docs/libcudacxx/extended_api/numeric.rst
new file mode 100644
index 00000000..7a6957a2
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/numeric.rst
@@ -0,0 +1,96 @@
+.. _libcudacxx-extended-api-numeric:
+
+Numeric
+========
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   numeric/add_overflow
+   numeric/saturating_add_overflow
+   numeric/div_overflow
+   numeric/saturating_div_overflow
+   numeric/isclose
+   numeric/mul_overflow
+   numeric/saturating_mul_overflow
+   numeric/narrow
+   numeric/overflow_cast
+   numeric/overflow_result
+   numeric/saturating_overflow_cast
+   numeric/sub_overflow
+   numeric/saturating_sub_overflow
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`cuda::narrow <libcudacxx-extended-api-numeric-narrow>`
+     - Casts a value and checks whether the value has changed
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`cuda::overflow_result <libcudacxx-extended-api-numeric-overflow_result>`
+     - Represents the result of arithmetic operations that may overflow
+     - CCCL 3.0.0
+     - CUDA 13.0
+
+   * - :ref:`cuda::overflow_cast <libcudacxx-extended-api-numeric-overflow_cast>`
+     - Casts a value with overflow checking
+     - CCCL 3.0.0
+     - CUDA 13.0
+
+   * - :ref:`cuda::add_overflow <libcudacxx-extended-api-numeric-add_overflow>`
+     - Performs addition with overflow checking
+     - CCCL 3.2.0
+     - CUDA 13.2
+
+   * - :ref:`cuda::sub_overflow <libcudacxx-extended-api-numeric-sub_overflow>`
+     - Performs subtraction with overflow checking
+     - CCCL 3.2.0
+     - CUDA 13.2
+
+   * - :ref:`cuda::div_overflow <libcudacxx-extended-api-numeric-div_overflow>`
+     - Performs division with overflow checking
+     - CCCL 3.2.0
+     - CUDA 13.2
+
+   * - :ref:`cuda::isclose <libcudacxx-extended-api-numeric-isclose>`
+     - Checks whether two values are approximately equal
+     - CCCL 3.5.0
+     - CUDA 13.5
+
+   * - :ref:`cuda::mul_overflow <libcudacxx-extended-api-numeric-mul_overflow>`
+     - Performs multiplication with overflow checking
+     - CCCL 3.4.0
+     - CUDA 13.4
+
+   * - :ref:`cuda::saturating_overflow_cast <libcudacxx-extended-api-numeric-saturating_overflow_cast>`
+     - Performs saturating cast of a value with overflow checking
+     - CCCL 3.4.0
+     - CUDA 13.4
+
+   * - :ref:`cuda::saturating_add_overflow <libcudacxx-extended-api-numeric-saturating_add_overflow>`
+     - Performs saturating addition with overflow checking
+     - CCCL 3.4.0
+     - CUDA 13.4
+
+   * - :ref:`cuda::saturating_sub_overflow <libcudacxx-extended-api-numeric-saturating_sub_overflow>`
+     - Performs saturating subtraction with overflow checking
+     - CCCL 3.4.0
+     - CUDA 13.4
+
+   * - :ref:`cuda::saturating_div_overflow <libcudacxx-extended-api-numeric-saturating_div_overflow>`
+     - Performs saturating division with overflow checking
+     - CCCL 3.4.0
+     - CUDA 13.4
+
+   * - :ref:`cuda::saturating_mul_overflow <libcudacxx-extended-api-numeric-saturating_mul_overflow>`
+     - Performs saturating multiplication with overflow checking
+     - CCCL 3.4.0
+     - CUDA 13.4
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/numeric/add_overflow.rst b/cccl_upstream/docs/libcudacxx/extended_api/numeric/add_overflow.rst
new file mode 100644
index 00000000..25a314e3
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/numeric/add_overflow.rst
@@ -0,0 +1,89 @@
+.. _libcudacxx-extended-api-numeric-add_overflow:
+
+``cuda::add_overflow``
+======================
+
+Defined in ``<cuda/numeric>`` header.
+
+.. code:: cpp
+
+   namespace cuda {
+
+   template <class T>
+   struct overflow_result;
+
+   template <class Result = /*unspecified*/, class Lhs, class Rhs>
+   [[nodiscard]] __host__ __device__ constexpr
+   overflow_result</*see-below*/> add_overflow(Lhs lhs, Rhs rhs) noexcept; // (1)
+
+   template <class Result, class Lhs, class Rhs>
+   [[nodiscard]] __host__ __device__ constexpr
+   bool add_overflow(Result& result, Lhs lhs, Rhs rhs) noexcept; // (2)
+
+   } // namespace cuda
+
+The function ``cuda::add_overflow`` performs addition of two values ``lhs`` and ``rhs`` with overflow checking. The result is the same as if the operands were first promoted to an infinite precision signed type, added together and the result truncated to the type of the return value.
+
+**Parameters**
+
+- ``result``: The result of the addition (2).
+- ``lhs``: The left-hand side operand (1, 2).
+- ``rhs``: The right-hand side operand (1, 2).
+
+**Return value**
+
+1. Returns an :ref:`overflow_result <libcudacxx-extended-api-numeric-overflow_result>` object  containing the result of the addition and a boolean indicating whether an overflow or underflow occurred. If the ``Result`` type is specified, it will be used as the type of the result, otherwise the common type of ``Lhs`` and ``Rhs`` is used.
+2. Returns ``true`` if an overflow or underflow occurred, ``false`` otherwise.
+
+**Constraints**
+
+- ``Result``, ``Lhs``, and ``Rhs`` must be `integer types <https://eel.is/c++draft/basic.fundamental#1>`_.
+
+**Performance considerations**
+
+- No overflow checking is required if ``Lhs +  Rhs`` is always representable with the ``Result`` type.
+- Computation is generally faster when ``Lhs``, ``Rhs``, and ``Result`` have the same signedness.
+- Unsigned computations are generally faster than signed computations.
+- The function uses PTX ``asm`` on device and compiler intrinsics on host whenever possible.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/numeric>
+    #include <cuda/std/cassert>
+    #include <cuda/std/limits>
+
+    __global__ void kernel()
+    {
+        constexpr auto int_max = cuda::std::numeric_limits<int>::max();
+        constexpr auto int_min = cuda::std::numeric_limits<int>::min();
+
+        // cuda::add_overflow(lhs, rhs) returning common type of lhs and rhs
+        // 'result' is evaluated to true if an overflow occurred, false otherwise
+        if (auto result = cuda::add_overflow(1, int_max))
+        {
+            assert(result.value == int_min);
+        }
+
+        // cuda::add_overflow<Result>(lhs, rhs) with explicit return type
+        auto result = cuda::add_overflow<long long>(-1, int_min)
+        assert(!result.overflow); // no overflow
+        assert(result.value == static_cast<long long>(int_min) + (-1ll));
+
+        unsigned result{};
+        // cuda::add_overflow(result, lhs, rhs) with bool return type
+        if (!cuda::add_overflow(result, 1, int_max))
+        {
+            assert(result.value == static_cast<unsigned>(int_max) + 1u);
+        }
+    }
+
+    int main()
+    {
+        kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/PPT17ozx6>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/numeric/div_overflow.rst b/cccl_upstream/docs/libcudacxx/extended_api/numeric/div_overflow.rst
new file mode 100644
index 00000000..1e6f8388
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/numeric/div_overflow.rst
@@ -0,0 +1,92 @@
+.. _libcudacxx-extended-api-numeric-div_overflow:
+
+``cuda::div_overflow``
+======================
+
+Defined in ``<cuda/numeric>`` header.
+
+.. code:: cpp
+
+   namespace cuda {
+
+   template <class T>
+   struct overflow_result;
+
+   template <class Result = /*unspecified*/, class Lhs, class Rhs>
+   [[nodiscard]] __host__ __device__ constexpr
+   overflow_result</*see-below*/> div_overflow(Lhs lhs, Rhs rhs) noexcept; // (1)
+
+   template <class Result, class Lhs, class Rhs>
+   [[nodiscard]] __host__ __device__ constexpr
+   bool div_overflow(Result& result, Lhs lhs, Rhs rhs) noexcept; // (2)
+
+   } // namespace cuda
+
+The function ``cuda::div_overflow`` performs integer division of ``lhs`` by ``rhs`` with overflow and error detection. The result is the same as if the operands were first promoted to an infinite precision signed type, divided, and the result truncated to the type of the return value.
+
+**Parameters**
+
+- ``result``: Receives the quotient when no overflow is detected (2).
+- ``lhs``: The dividend (1, 2).
+- ``rhs``: The divisor (1, 2).
+
+**Return value**
+
+1. Returns an :ref:`overflow_result <libcudacxx-extended-api-numeric-overflow_result>` object containing the computed quotient and a boolean flag indicating whether an overflow or underflow occurred. If the ``Result`` type is specified, it will be used as the type of the result, otherwise the common type of ``Lhs`` and ``Rhs`` is used.
+2. Returns ``true`` if an overflow or underflow occurred. When ``false`` is returned, the computed quotient is stored in ``result``.
+
+**Preconditions**
+
+- ``rhs != 0``
+
+**Constraints**
+
+- ``Result``, ``Lhs``, and ``Rhs`` must be `integer types <https://eel.is/c++draft/basic.fundamental#1>`_.
+
+**Notes**
+
+- For signed types, ``numeric_limits<Lhs>::min()`` divided by ``-1`` triggers overflow when ``Result`` matches the operand types.
+
+**Performance considerations**
+
+- No overflow checking if ``Lhs / Rhs`` is always  representable with the ``Result`` type.
+- The computation with unsigned types is faster than signed types.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/numeric>
+    #include <cuda/std/cassert>
+    #include <cuda/std/limits>
+
+    __global__ void kernel()
+    {
+        constexpr auto int_min = cuda::std::numeric_limits<int>::min();
+
+        // cuda::div_overflow(lhs, rhs) returning the common type of the operands
+        if (auto result = cuda::div_overflow(-1, int_min))
+        {
+            assert(result.overflow);
+        }
+
+        // cuda::div_overflow<Result>(lhs, rhs) with an explicitly wider result type
+        auto wide = cuda::div_overflow<long long>(-1, int_min);
+        assert(!wide.overflow);
+        assert(wide.value == 0);
+
+        unsigned quotient{};
+        // cuda::div_overflow(result, lhs, rhs) with bool return type
+        bool overflow = cuda::div_overflow(quotient, 10u, 2u);
+        assert(!overflow);
+        assert(quotient == 5u);
+    }
+
+    int main()
+    {
+        kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/dYG3dWss5>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/numeric/isclose.rst b/cccl_upstream/docs/libcudacxx/extended_api/numeric/isclose.rst
new file mode 100644
index 00000000..914c8d14
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/numeric/isclose.rst
@@ -0,0 +1,103 @@
+.. _libcudacxx-extended-api-numeric-isclose:
+
+``cuda::isclose``
+=================
+
+Defined in ``<cuda/numeric>`` header.
+
+.. code:: cpp
+
+   namespace cuda {
+
+   template <class T>
+   [[nodiscard]] __host__ __device__
+   bool isclose(T lhs, T rhs) noexcept;
+
+   template <class T>
+   [[nodiscard]] __host__ __device__
+   bool isclose(T lhs, T rhs, float relative_tol) noexcept;
+
+   template <class T>
+   [[nodiscard]] __host__ __device__
+   bool isclose(T lhs, T rhs, float relative_tol, T absolute_tol) noexcept;
+
+   template <class Complex>
+   [[nodiscard]] __host__ __device__
+   bool isclose(const Complex& lhs, const Complex& rhs) noexcept;
+
+   template <class Complex>
+   [[nodiscard]] __host__ __device__
+   bool isclose(const Complex& lhs, const Complex& rhs, float relative_tol) noexcept;
+
+   template <class Complex, class AbsTol>
+   [[nodiscard]] __host__ __device__
+   bool isclose(const Complex& lhs,
+                const Complex& rhs,
+                float  relative_tol,
+                AbsTol absolute_tol) noexcept;
+
+   } // namespace cuda
+
+``cuda::isclose`` checks whether two values are approximately equal using the weak symmetric comparison in a similar manner to `PEP 485 <https://peps.python.org/pep-0485/>`_:
+
+.. code:: cpp
+
+   abs(lhs - rhs) <= max(absolute_tol, relative_tol * max(abs(lhs), abs(rhs)))
+
+- For integral operands, ``relative_tol`` is interpreted as its exact binary floating-point value. Comparing it to the integral difference is equivalent to rounding the relative threshold down to the nearest integer.
+- The overloads without ``relative_tol`` use a default relative tolerance based on half of available digits of accuracy. The default relative tolerance for integer types is 0.
+- The overloads without ``absolute_tol`` use ``absolute_tol == 0``.
+
+**Parameters**
+
+- ``lhs``: The first value to compare.
+- ``rhs``: The second value to compare.
+- ``relative_tol``: The relative tolerance. Passing ``0`` performs a purely absolute tolerance check when ``absolute_tol`` is non-zero.
+- ``absolute_tol``: The absolute tolerance. This is useful for comparisons near zero.
+
+**Return value**
+
+- Returns ``true`` if ``lhs`` and ``rhs`` are close to each other, otherwise returns ``false``.
+
+**Preconditions**
+
+- ``relative_tol``: Must be in the range ``[0.0, 1.0]``.
+- ``absolute_tol``: Must be finite and non-negative.
+
+**Constraints**
+
+- Scalar overloads require ``lhs``, ``rhs``, ``absolute_tol`` to have the same arithmetic type (integer or floating point).
+- Complex overloads accept ``cuda::std::complex<T>`` and ``std::complex<T>`` operands.
+- ``AbsTol`` must be the same type as the complex value type.
+
+**Special values**
+
+- ``NaN`` is never close to any value, including another ``NaN``.
+- Infinity and negative infinity are only close to themselves.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/numeric>
+    #include <cuda/std/cassert>
+    #include <cuda/std/complex>
+
+    __global__ void kernel()
+    {
+        assert(cuda::isclose( 1.0f, 1.0f + 5e-6f));
+        assert(!cuda::isclose(1.0f, 1.0f + 2e-5f));
+
+        assert(cuda::isclose( 0.0f, 1e-12f, 0.0f, 1e-12f));
+
+        cuda::std::complex<float> z1{1.0f, 1.0f};
+        cuda::std::complex<float> z2{2.0f, 0.0f};
+        assert(cuda::isclose(z1, z2, 0.75f));
+    }
+
+    int main()
+    {
+        kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+    }
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/numeric/mul_overflow.rst b/cccl_upstream/docs/libcudacxx/extended_api/numeric/mul_overflow.rst
new file mode 100644
index 00000000..b91c2b95
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/numeric/mul_overflow.rst
@@ -0,0 +1,88 @@
+.. _libcudacxx-extended-api-numeric-mul_overflow:
+
+``cuda::mul_overflow``
+======================
+
+Defined in ``<cuda/numeric>`` header.
+
+.. code:: cpp
+
+   namespace cuda {
+
+   template <class T>
+   struct overflow_result;
+
+   template <class Result = /*unspecified*/, class Lhs, class Rhs>
+   [[nodiscard]] __host__ __device__ constexpr
+   overflow_result</*see-below*/> mul_overflow(Lhs lhs, Rhs rhs) noexcept; // (1)
+
+   template <class Result, class Lhs, class Rhs>
+   [[nodiscard]] __host__ __device__ constexpr
+   bool mul_overflow(Result& result, Lhs lhs, Rhs rhs) noexcept; // (2)
+
+   } // namespace cuda
+
+The function ``cuda::mul_overflow`` performs multiplication of two values ``lhs`` and ``rhs`` with overflow checking. The result is the same as if the operands were first promoted to an infinite precision signed type, multiplied together and the result truncated to the type of the return value.
+
+**Parameters**
+
+- ``result``: The result of the multiplication (2).
+- ``lhs``: The left-hand side operand (1, 2).
+- ``rhs``: The right-hand side operand (1, 2).
+
+**Return value**
+
+1. Returns an :ref:`overflow_result <libcudacxx-extended-api-numeric-overflow_result>` object  containing the result of the multiplication and a boolean indicating whether an overflow or underflow occurred. If the ``Result`` type is specified, it will be used as the type of the result, otherwise the common type of ``Lhs`` and ``Rhs`` is used.
+2. Returns ``true`` if an overflow or underflow occurred, ``false`` otherwise.
+
+**Constraints**
+
+- ``Result``, ``Lhs``, and ``Rhs`` must be `integer types <https://eel.is/c++draft/basic.fundamental#1>`_.
+
+**Performance considerations**
+
+- No overflow checking is required if ``Lhs +  Rhs`` is always representable with the ``Result`` type.
+- Computation is generally faster when ``Lhs``, ``Rhs``, and ``Result`` have the same signedness.
+- Unsigned computations are generally faster than signed computations.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/numeric>
+    #include <cuda/std/cassert>
+    #include <cuda/std/limits>
+
+    __global__ void kernel()
+    {
+        constexpr auto int_max = cuda::std::numeric_limits<int>::max();
+        constexpr auto int_min = cuda::std::numeric_limits<int>::min();
+
+        // cuda::mul_overflow(lhs, rhs) returning common type of lhs and rhs
+        // 'result' is evaluated to true if an overflow occurred, false otherwise
+        if (auto result1 = cuda::mul_overflow(-1, int_min))
+        {
+            assert(result1.value == int_min);
+        }
+
+        // cuda::mul_overflow<Result>(lhs, rhs) with explicit return type
+        auto [result2, overflow2] = cuda::mul_overflow<long long>(-1, int_min);
+        assert(!overflow2); // no overflow
+        assert(result2 == static_cast<long long>(int_min) * (-1ll));
+
+        unsigned result3{};
+        // cuda::mul_overflow(result, lhs, rhs) with bool return type
+        if (!cuda::mul_overflow(result3, 2, int_max))
+        {
+            assert(result3 == static_cast<unsigned>(int_max) * 2u);
+        }
+    }
+
+    int main()
+    {
+        kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/hTME8v6rK>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/numeric/narrow.rst b/cccl_upstream/docs/libcudacxx/extended_api/numeric/narrow.rst
new file mode 100644
index 00000000..0adf4ef5
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/numeric/narrow.rst
@@ -0,0 +1,55 @@
+.. _libcudacxx-extended-api-numeric-narrow:
+
+``cuda::narrow``
+=====================
+
+.. code:: cpp
+
+   struct narrowing_error;
+
+   template <typename To, typename From>
+   [[nodiscard]] constexpr
+   To narrow(From from);
+
+   template <typename To, typename From>
+   [[nodiscard]] constexpr
+   To narrow_cast(From&& __from) noexcept;
+
+
+Both functions use a ``static_cast`` to cast the value ``from`` to type ``To``.
+``From`` needs to be convertible to ``To``, and implement ``operator!=``.
+``cuda::narrow`` additionally checks whether the value has changed,
+and if so, throws ``cuda::narrowing_error`` in host code and traps in device code.
+In this case, ``To`` additionally needs to be convertible to ``From``.
+``cuda::narrow_cast`` does not perform such a check (it's a plain cast) and is just intended to show
+that narrowing and a potential change of the value is intended.
+The functions are modelled after ``gsl::narrow`` and  ``gsl::narrow_cast``.
+See also the C++ Core Guidelines
+`ES.46 <https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Res-narrowing>`_ and
+`ES.49 <https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Res-casts-named>`_.
+
+
+Example
+-------
+
+.. code:: cpp
+
+    #include <cuda/numeric>
+
+    __global__ void kernel(size_t n) {
+        unsigned int r1 = cuda::narrow<unsigned int>(n); // traps
+
+        unsigned int r2 = cuda::narrow_cast<unsigned int>(n); // truncation of value is intended
+    }
+
+    void host() {
+        unsigned char r1 = cuda::narrow<unsigned char>( 200); // ok
+        unsigned char r2 = cuda::narrow<unsigned char>( 300); // throws narrowing_error
+        unsigned int  r3 = cuda::narrow<unsigned int >(-100); // throws narrowing_error
+
+        unsigned char r4 = cuda::narrow_cast<unsigned char>(300); // truncation of value is intended
+
+        kernel<<<1, 1>>>(2LL << 35); // size larger than unsigned int
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/ahcqv6joY>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/numeric/overflow_cast.rst b/cccl_upstream/docs/libcudacxx/extended_api/numeric/overflow_cast.rst
new file mode 100644
index 00000000..fe429032
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/numeric/overflow_cast.rst
@@ -0,0 +1,68 @@
+.. _libcudacxx-extended-api-numeric-overflow_cast:
+
+``cuda::overflow_cast``
+==========================
+
+.. code:: cpp
+
+   template <class T>
+   struct overflow_result;
+
+   template <class To, class From>
+   [[nodiscard]] __host__ __device__ inline constexpr
+   overflow_result<To> overflow_cast(From from) noexcept;
+
+The function ``cuda::overflow_cast`` casts a value of type ``From`` to type ``To`` with overflow checking.
+
+**Parameters**
+
+- ``from``: The value to be casted.
+
+**Return value**
+
+- Returns an :ref:`overflow_result <libcudacxx-extended-api-numeric-overflow_result>` object that contains the result of the cast and a boolean indicating whether an overflow occurred.
+
+**Constraints**
+
+- ``To`` and ``From`` must be `integer types <https://eel.is/c++draft/basic.fundamental#1>`_.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/numeric>
+    #include <cuda/std/cassert>
+    #include <cuda/std/limits>
+
+    __global__ void kernel()
+    {
+        constexpr auto int_max = cuda::std::numeric_limits<int>::max();
+        constexpr auto int_min = cuda::std::numeric_limits<int>::min();
+
+        if (auto result = cuda::overflow_cast<unsigned>(int_max))
+        {
+            assert(false); // Should not be reached
+        }
+        else
+        {
+            assert(result.value == static_cast<unsigned>(int_max));
+        }
+
+        if (auto result = cuda::overflow_cast<unsigned>(int_min))
+        {
+            assert(result.value == static_cast<unsigned>(int_min));
+        }
+        else
+        {
+            assert(false); // Should not be reached
+        }
+    }
+
+    int main()
+    {
+        kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/vK3WMMs1n>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/numeric/overflow_result.rst b/cccl_upstream/docs/libcudacxx/extended_api/numeric/overflow_result.rst
new file mode 100644
index 00000000..c7823493
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/numeric/overflow_result.rst
@@ -0,0 +1,38 @@
+.. _libcudacxx-extended-api-numeric-overflow_result:
+
+``cuda::overflow_result``
+=========================
+
+.. code:: cpp
+
+   template <class T>
+   struct overflow_result
+   {
+        T    value;
+        bool overflow;
+
+        __host__ __device__
+        constexpr explicit operator bool() const noexcept;
+   };
+
+The ``overflow_result`` struct is used to represent the result of arithmetic operations that may overflow. It contains the following members:
+
+- ``value``: The result of the operation of type ``T``.
+- ``overflow``: A boolean indicating whether an overflow occurred during the operation.
+
+The ``operator bool()`` returns ``true`` if an overflow occurred, and ``false`` otherwise.
+It can be used in conditional expressions to check whether an overflow occurred.
+
+Example:
+
+.. code:: cpp
+
+    auto result = /* overflow operation */;
+    if (result)
+    {
+        // Overflow occurred
+    }
+
+**Constraints**
+
+- ``T`` must be an integer type.
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/numeric/saturating_add_overflow.rst b/cccl_upstream/docs/libcudacxx/extended_api/numeric/saturating_add_overflow.rst
new file mode 100644
index 00000000..948765b1
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/numeric/saturating_add_overflow.rst
@@ -0,0 +1,78 @@
+.. _libcudacxx-extended-api-numeric-saturating_add_overflow:
+
+``cuda::saturating_add_overflow``
+=================================
+
+Defined in ``<cuda/numeric>`` header.
+
+.. code:: cpp
+
+   namespace cuda {
+
+   template <class T>
+   struct overflow_result;
+
+   template <class T>
+   [[nodiscard]] __host__ __device__ constexpr
+   overflow_result<T> saturating_add_overflow(T lhs, T rhs) noexcept; // (1)
+
+   template <class T>
+   [[nodiscard]] __host__ __device__ constexpr
+   bool saturating_add_overflow(T& result, T lhs, T rhs) noexcept; // (2)
+
+   } // namespace cuda
+
+The function ``cuda::saturating_add_overflow`` performs saturating addition of two values ``lhs`` and ``rhs`` with overflow checking.
+
+**Parameters**
+
+- ``result``: The result of the saturating addition. (2)
+- ``lhs``: The left-hand side operand. (1, 2)
+- ``rhs``: The right-hand side operand. (1, 2)
+
+**Return value**
+
+1. Returns an :ref:`overflow_result <libcudacxx-extended-api-numeric-overflow_result>` object containing the result of the saturating addition and a boolean indicating whether an overflow or underflow occurred.
+2. Returns ``true`` if an overflow or underflow occurred, ``false`` otherwise.
+
+**Constraints**
+
+- ``T`` must be an `integer type <https://eel.is/c++draft/basic.fundamental#1>`_.
+
+**Performance considerations**
+
+- Functionality is implemented by correcting the ``cuda::add_overflow`` result in case of overflow/underflow.
+- Unsigned computations are generally faster than signed computations.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/numeric>
+    #include <cuda/std/cassert>
+    #include <cuda/std/limits>
+
+    __global__ void kernel()
+    {
+        constexpr auto int_max = cuda::std::numeric_limits<int>::max();
+
+        const auto result = cuda::saturating_add_overflow(1, int_max); // saturated
+        assert(result.value == int_max);
+        assert(result.overflow);
+
+        int value;
+        if (cuda::saturating_add_overflow(value, 42, 1024))
+        {
+            assert(false); // shouldn't be reached
+        }
+        assert(value == 1066);
+    }
+
+    int main()
+    {
+        kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/f4jrMb5cW>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/numeric/saturating_div_overflow.rst b/cccl_upstream/docs/libcudacxx/extended_api/numeric/saturating_div_overflow.rst
new file mode 100644
index 00000000..d809959c
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/numeric/saturating_div_overflow.rst
@@ -0,0 +1,83 @@
+.. _libcudacxx-extended-api-numeric-saturating_div_overflow:
+
+``cuda::saturating_div_overflow``
+=================================
+
+Defined in ``<cuda/numeric>`` header.
+
+.. code:: cpp
+
+   namespace cuda {
+
+   template <class T>
+   struct overflow_result;
+
+   template <class T>
+   [[nodiscard]] __host__ __device__ constexpr
+   overflow_result<T> saturating_div_overflow(T lhs, T rhs) noexcept; // (1)
+
+   template <class T>
+   [[nodiscard]] __host__ __device__ constexpr
+   bool saturating_div_overflow(T& result, T lhs, T rhs) noexcept; // (2)
+
+   } // namespace cuda
+
+The function ``cuda::saturating_div_overflow`` performs saturating integer division of ``lhs`` by ``rhs`` with overflow and error detection.
+
+**Parameters**
+
+- ``result``: Result of the saturating integer division. (2)
+- ``lhs``: The dividend. (1, 2)
+- ``rhs``: The divisor. (1, 2)
+
+**Return value**
+
+1. Returns an :ref:`overflow_result <libcudacxx-extended-api-numeric-overflow_result>` object containing the result of saturating integer division and a boolean flag indicating whether an overflow or underflow occurred.
+2. Returns ``true`` if an overflow or underflow occurred.
+
+**Preconditions**
+
+- ``rhs != 0``
+
+**Constraints**
+
+- ``T`` must be an `integer type <https://eel.is/c++draft/basic.fundamental#1>`_.
+
+**Performance considerations**
+
+- Functionality is implemented by correcting the ``cuda::div_overflow`` result in case of overflow/underflow.
+- Unsigned computations are generally faster than signed computations.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/numeric>
+    #include <cuda/std/cassert>
+    #include <cuda/std/limits>
+
+    __global__ void kernel()
+    {
+        constexpr auto int_max = cuda::std::numeric_limits<int>::max();
+        constexpr auto int_min = cuda::std::numeric_limits<int>::min();
+
+        const auto result = cuda::saturating_div_overflow(int_min, -1); // saturated
+        assert(result.value == int_max);
+        assert(result.overflow);
+
+        int value;
+        if (cuda::saturating_div_overflow(value, 256, 8))
+        {
+            assert(false); // shouldn't be reached
+        }
+        assert(value == 32);
+    }
+
+    int main()
+    {
+        kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/T91jMKEjY>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/numeric/saturating_mul_overflow.rst b/cccl_upstream/docs/libcudacxx/extended_api/numeric/saturating_mul_overflow.rst
new file mode 100644
index 00000000..0877d740
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/numeric/saturating_mul_overflow.rst
@@ -0,0 +1,79 @@
+.. _libcudacxx-extended-api-numeric-saturating_mul_overflow:
+
+``cuda::saturating_mul_overflow``
+=================================
+
+Defined in ``<cuda/numeric>`` header.
+
+.. code:: cpp
+
+   namespace cuda {
+
+   template <class T>
+   struct overflow_result;
+
+   template <class T>
+   [[nodiscard]] __host__ __device__ constexpr
+   overflow_result<T> saturating_mul_overflow(T lhs, T rhs) noexcept; // (1)
+
+   template <class T>
+   [[nodiscard]] __host__ __device__ constexpr
+   bool saturating_mul_overflow(T& result, T lhs, T rhs) noexcept; // (2)
+
+   } // namespace cuda
+
+The function ``cuda::saturating_mul_overflow`` performs saturating multiplication of two values ``lhs`` and ``rhs`` with overflow checking.
+
+**Parameters**
+
+- ``result``: The result of the saturating multiplication (2).
+- ``lhs``: The left-hand side operand (1, 2).
+- ``rhs``: The right-hand side operand (1, 2).
+
+**Return value**
+
+1. Returns an :ref:`overflow_result <libcudacxx-extended-api-numeric-overflow_result>` object  containing the result of the saturating multiplication and a boolean indicating whether an overflow or underflow occurred.
+2. Returns ``true`` if an overflow or underflow occurred, ``false`` otherwise.
+
+**Constraints**
+
+- ``T`` must be an `integer type <https://eel.is/c++draft/basic.fundamental#1>`_.
+
+**Performance considerations**
+
+- Functionality is implemented by correcting the ``cuda::mul_overflow`` result in case of overflow/underflow.
+- Unsigned computations are generally faster than signed computations.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/numeric>
+    #include <cuda/std/cassert>
+    #include <cuda/std/limits>
+
+    __global__ void kernel()
+    {
+        constexpr auto int_max = cuda::std::numeric_limits<int>::max();
+        constexpr auto int_min = cuda::std::numeric_limits<int>::min();
+
+        const auto result = cuda::saturating_mul_overflow(int_max, int_min); // saturated
+        assert(result.value == int_min);
+        assert(result.overflow);
+
+        int value;
+        if (cuda::saturating_mul_overflow(value, 4, 8))
+        {
+            assert(false); // shouldn't be reached
+        }
+        assert(value == 32);
+    }
+
+    int main()
+    {
+        kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/chrG67YWT>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/numeric/saturating_overflow_cast.rst b/cccl_upstream/docs/libcudacxx/extended_api/numeric/saturating_overflow_cast.rst
new file mode 100644
index 00000000..8d7fa3a7
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/numeric/saturating_overflow_cast.rst
@@ -0,0 +1,68 @@
+.. _libcudacxx-extended-api-numeric-saturating_overflow_cast:
+
+``cuda::overflow_cast``
+==========================
+
+.. code:: cpp
+
+   template <class T>
+   struct overflow_result;
+
+   template <class To, class From>
+   [[nodiscard]] __host__ __device__ inline constexpr
+   overflow_result<To> saturating_overflow_cast(From from) noexcept;
+
+The function ``cuda::saturating_overflow_cast`` does saturating cast of a value of type ``From`` to type ``To`` with overflow checking.
+
+**Parameters**
+
+- ``from``: The value to be casted.
+
+**Return value**
+
+- Returns an :ref:`overflow_result <libcudacxx-extended-api-numeric-overflow_result>` object that contains the result of the saturating cast and a boolean indicating whether an overflow occurred.
+
+**Constraints**
+
+- ``To`` and ``From`` must be `integer types <https://eel.is/c++draft/basic.fundamental#1>`_.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/numeric>
+    #include <cuda/std/cassert>
+    #include <cuda/std/limits>
+
+    __global__ void kernel()
+    {
+        constexpr auto int_max = cuda::std::numeric_limits<int>::max();
+        constexpr auto int_min = cuda::std::numeric_limits<int>::min();
+
+        if (auto result = cuda::saturating_overflow_cast<unsigned>(int_max))
+        {
+            assert(false); // Should not be reached
+        }
+        else
+        {
+            assert(result.value == static_cast<unsigned>(int_max));
+        }
+
+        if (auto result = cuda::saturating_overflow_cast<unsigned>(int_min)) // saturated
+        {
+            assert(result.value == 0);
+        }
+        else
+        {
+            assert(false); // Should not be reached
+        }
+    }
+
+    int main()
+    {
+        kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/jcv39ohso>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/numeric/saturating_sub_overflow.rst b/cccl_upstream/docs/libcudacxx/extended_api/numeric/saturating_sub_overflow.rst
new file mode 100644
index 00000000..b21edbee
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/numeric/saturating_sub_overflow.rst
@@ -0,0 +1,78 @@
+.. _libcudacxx-extended-api-numeric-saturating_sub_overflow:
+
+``cuda::saturating_sub_overflow``
+=================================
+
+Defined in ``<cuda/numeric>`` header.
+
+.. code:: cpp
+
+   namespace cuda {
+
+   template <class T>
+   struct overflow_result;
+
+   template <class T>
+   [[nodiscard]] __host__ __device__ constexpr
+   overflow_result<T> saturating_sub_overflow(T lhs, T rhs) noexcept; // (1)
+
+   template <class T>
+   [[nodiscard]] __host__ __device__ constexpr
+   bool saturating_sub_overflow(T& result, T lhs, T rhs) noexcept; // (2)
+
+   } // namespace cuda
+
+The function ``cuda::saturating_sub_overflow`` performs saturating subtraction of two values ``lhs`` and ``rhs`` with overflow detection.
+
+**Parameters**
+
+- ``result``: The result of the saturating subtraction. (2)
+- ``lhs``: The left-hand side operand. (1, 2)
+- ``rhs``: The right-hand side operand. (1, 2)
+
+**Return value**
+
+1. Returns an :ref:`overflow_result <libcudacxx-extended-api-numeric-overflow_result>` object containing the result of the saturating subtraction and a boolean flag indicating whether an overflow or underflow occurred.
+2. Returns ``true`` if an overflow or underflow occurred, ``false`` otherwise.
+
+**Constraints**
+
+- ``T`` must be an `integer type <https://eel.is/c++draft/basic.fundamental#1>`_.
+
+**Performance considerations**
+
+- Functionality is implemented by correcting the ``cuda::sub_overflow`` result in case of overflow/underflow.
+- Unsigned computations are generally faster than signed computations.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/numeric>
+    #include <cuda/std/cassert>
+    #include <cuda/std/limits>
+
+    __global__ void kernel()
+    {
+        constexpr auto uint_max = cuda::std::numeric_limits<unsigned>::max();
+
+        const auto result = cuda::saturating_sub_overflow(42u, uint_max); // saturated
+        assert(result.value == 0);
+        assert(result.overflow);
+
+        int value;
+        if (cuda::saturating_sub_overflow(value, 42, 1024))
+        {
+            assert(false); // shouldn't be reached
+        }
+        assert(value == -982);
+    }
+
+    int main()
+    {
+        kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/1M9j9oG9z>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/numeric/sub_overflow.rst b/cccl_upstream/docs/libcudacxx/extended_api/numeric/sub_overflow.rst
new file mode 100644
index 00000000..b7c0ff0c
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/numeric/sub_overflow.rst
@@ -0,0 +1,87 @@
+.. _libcudacxx-extended-api-numeric-sub_overflow:
+
+``cuda::sub_overflow``
+======================
+
+Defined in ``<cuda/numeric>`` header.
+
+.. code:: cpp
+
+   namespace cuda {
+
+   template <class T>
+   struct overflow_result;
+
+   template <class Result = /*unspecified*/, class Lhs, class Rhs>
+   [[nodiscard]] __host__ __device__ constexpr
+   overflow_result</*see-below*/> sub_overflow(Lhs lhs, Rhs rhs) noexcept; // (1)
+
+   template <class Result, class Lhs, class Rhs>
+   [[nodiscard]] __host__ __device__ constexpr
+   bool sub_overflow(Result& result, Lhs lhs, Rhs rhs) noexcept; // (2)
+
+   } // namespace cuda
+
+The function ``cuda::sub_overflow`` performs subtraction of two values ``lhs`` and ``rhs`` with overflow detection. The result is the same as if the operands were first promoted to an infinite precision signed type, subtracted, and then the result truncated to the type of the return value.
+
+**Parameters**
+
+- ``result``: The result of the subtraction (2).
+- ``lhs``: The left-hand side operand (1, 2).
+- ``rhs``: The right-hand side operand (1, 2).
+
+**Return value**
+
+1. Returns an :ref:`overflow_result <libcudacxx-extended-api-numeric-overflow_result>` object containing the computed  difference and a boolean flag indicating whether an overflow or underflow occurred. If the ``Result`` type is specified, it will be used as the type of the result, otherwise the common type of ``Lhs`` and ``Rhs`` is used.
+2. Returns ``true`` if an overflow or underflow occurred, ``false`` otherwise.
+
+**Constraints**
+
+- ``Result``, ``Lhs``, and ``Rhs`` must be `integer types <https://eel.is/c++draft/basic.fundamental#1>`_.
+
+**Performance considerations**
+
+- No overflow checking is required when ``lhs - rhs`` is always representable with the ``Result`` type.
+- Computation is generally faster when ``Lhs``, ``Rhs``, and ``Result`` have the same signedness.
+- Unsigned computations are generally faster than signed computations.
+- The function uses PTX ``asm`` on device and compiler intrinsics on host whenever possible.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/numeric>
+    #include <cuda/std/cassert>
+    #include <cuda/std/limits>
+
+    __global__ void kernel()
+    {
+        constexpr auto int_max = cuda::std::numeric_limits<int>::max();
+        constexpr auto int_min = cuda::std::numeric_limits<int>::min();
+
+        // cuda::sub_overflow(lhs, rhs) returning the common type of lhs and rhs
+        auto result = cuda::sub_overflow(3, 1))
+        assert(result.value == 2);
+
+
+        // cuda::sub_overflow<Result>(lhs, rhs) with an explicit result type
+        auto result2 = cuda::sub_overflow<long long>(int_min, 1);
+        assert(!result2.overflow);
+        assert(result2.value == static_cast<long long>(int_min) - 1ll);
+
+        unsigned result3{};
+        // cuda::sub_overflow(result, lhs, rhs) returning a bool flag
+        if (!cuda::sub_overflow(result, 5u, 3u))
+        {
+            assert(result == 2u);
+        }
+    }
+
+    int main()
+    {
+        kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/Pq8sc9s7a>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/random.rst b/cccl_upstream/docs/libcudacxx/extended_api/random.rst
new file mode 100644
index 00000000..32b7e022
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/random.rst
@@ -0,0 +1,24 @@
+.. _libcudacxx-extended-api-random:
+
+Random
+======
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   random/pcg64
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`cuda::pcg64 <libcudacxx-extended-api-random-pcg64>`
+     - 128-bit state PCG engine with 64-bit output
+     - CCCL 3.3.0
+     - CUDA 13.3
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/random/pcg64.rst b/cccl_upstream/docs/libcudacxx/extended_api/random/pcg64.rst
new file mode 100644
index 00000000..f602ba9e
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/random/pcg64.rst
@@ -0,0 +1,23 @@
+.. _libcudacxx-extended-api-random-pcg64:
+
+``pcg64``
+=========
+
+Defined in the ``<cuda/random>`` header.
+
+``cuda::pcg64`` is a 128-bit state PCG XSL RR 128/64 engine that produces 64-bit unsigned integer outputs. It has a
+period of ``2^128`` and supports logarithmic-time ``discard``. ``cuda::pcg64`` models the
+`UniformRandomBitGenerator <https://en.cppreference.com/w/cpp/named_req/UniformRandomBitGenerator>`_ named requirement.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/random>
+
+    __global__ void sample_kernel() {
+        cuda::pcg64 rng(42);
+        auto value = rng();
+        rng.discard(10);
+    }
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives.rst
new file mode 100644
index 00000000..9868e964
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives.rst
@@ -0,0 +1,137 @@
+.. _libcudacxx-extended-api-synchronization:
+
+Synchronization Primitives
+===========================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   synchronization_primitives/atomic
+   synchronization_primitives/atomic_ref
+   synchronization_primitives/latch
+   synchronization_primitives/barrier
+   synchronization_primitives/counting_semaphore
+   synchronization_primitives/binary_semaphore
+   synchronization_primitives/pipeline
+
+.. rubric:: Atomics
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`cuda::atomic <libcudacxx-extended-api-synchronization-atomic>`
+     - System-wide `std::atomic <https://en.cppreference.com/w/cpp/atomic/atomic>`_ objects and operations
+     - libcu++ 1.0.0 / CCCL 2.0.0
+     - CUDA 10.2
+
+   * - :ref:`cuda::atomic_ref <libcudacxx-extended-api-synchronization-atomic-ref>`
+     - System-wide `std::atomic_ref <https://en.cppreference.com/w/cpp/atomic/atomic_ref>`_ objects and operations
+     - libcu++ 1.7.0 / CCCL 2.0.0
+     - CUDA 11.6
+
+.. rubric:: Latches
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`cuda::latch <libcudacxx-extended-api-synchronization-latch>`
+     - System-wide `std::latch <https://en.cppreference.com/w/cpp/thread/latch>`_ single-phase asynchronous
+       thread coordination mechanism
+     - libcu++ 1.1.0 / CCCL 2.0.0
+     - CUDA 11.0
+
+.. rubric:: Barriers
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`cuda::barrier <libcudacxx-extended-api-synchronization-barrier>`
+     - System wide `std::barrier <https://en.cppreference.com/w/cpp/thread/barrier>`_ multi-phase asynchronous
+       thread coordination mechanism
+     - libcu++ 1.1.0 / CCCL 2.0.0
+     - CUDA 11.0
+
+.. rubric:: Semaphores
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`cuda::counting_semaphore <libcudacxx-extended-api-synchronization-counting-semaphore>`
+     - System wide `std::counting_semaphore <https://en.cppreference.com/w/cpp/thread/counting_semaphore>`_
+       primitive for constraining concurrent access
+     - libcu++ 1.1.0 / CCCL 2.0.0
+     - CUDA 11.0
+
+   * - :ref:`cuda::binary_semaphore <libcudacxx-extended-api-synchronization-counting-semaphore>`
+     - System wide `std::binary_semaphore <https://en.cppreference.com/w/cpp/thread/counting_semaphore>`_
+       primitive for mutual exclusion
+     - libcu++ 1.1.0 / CCCL 2.0.0
+     - CUDA 11.0
+
+.. rubric:: Pipelines
+
+The pipeline library is included in the CUDA Toolkit, but is not part of the open source libcu++ distribution.
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`cuda::pipeline <libcudacxx-extended-api-synchronization-pipeline>`
+     - Coordination mechanism for sequencing asynchronous operations
+     - libcu++ 1.2.0 / CCCL 2.0.0
+     - CUDA 11.1
+
+   * - :ref:`cuda::pipeline_shared_state <libcudacxx-extended-api-synchronization-pipeline-pipeline-shared-state>`
+     - :ref:`cuda::pipeline <libcudacxx-extended-api-synchronization-pipeline>` shared state object
+     - libcu++ 1.1.0 / CCCL 2.0.0
+     - CUDA 11.0
+
+   * - :ref:`cuda::pipeline_role <libcudacxx-extended-api-synchronization-pipeline-pipeline-role>`
+     - Defines producer/consumer role for a thread participating in a *pipeline*
+     - libcu++ 1.1.0 / CCCL 2.0.0
+     - CUDA 11.0
+
+   * - :ref:`cuda::make_pipeline <libcudacxx-extended-api-synchronization-pipeline-pipeline-role>`
+     - Creates a :ref:`cuda::pipeline <libcudacxx-extended-api-synchronization-pipeline>`
+     - libcu++ 1.1.0 / CCCL 2.0.0
+     - CUDA 11.0
+
+   * - :ref:`cuda::pipeline_consumer_wait_prior <libcudacxx-extended-api-synchronization-pipeline-pipeline-consumer-wait-prior>`
+     - Blocks the current thread until all operations committed up to a prior *pipeline stage* complete
+     - libcu++ 1.1.0 / CCCL 2.0.0
+     - CUDA 11.0
+
+   * - :ref:`cuda::pipeline_producer_commit <libcudacxx-extended-api-synchronization-pipeline-pipeline-producer-commit>`
+     - Binds operations previously issued by the current thread to a :ref:`cuda::barrier <libcudacxx-extended-api-synchronization-barrier>`
+     - libcu++ 1.1.0 / CCCL 2.0.0
+     - CUDA 11.0
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic.rst
new file mode 100644
index 00000000..71da26cd
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic.rst
@@ -0,0 +1,99 @@
+.. _libcudacxx-extended-api-synchronization-atomic:
+
+``cuda::atomic``
+================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   atomic/atomic_thread_fence
+   atomic/fetch_max
+   atomic/fetch_min
+
+Defined in header ``<cuda/atomic>``:
+
+.. code:: cuda
+
+   template <typename T, cuda::thread_scope Scope = cuda::thread_scope_system>
+   class cuda::atomic;
+
+The class template ``cuda::atomic`` is an extended form of `cuda::std::atomic <https://en.cppreference.com/w/cpp/atomic/atomic>`_
+that takes an additional :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` argument,
+defaulted to ``cuda::std::thread_scope_system``.
+
+It has the same interface and semantics as `cuda::std::atomic <https://en.cppreference.com/w/cpp/atomic/atomic>`_,
+with the following additional operations.
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - :ref:`cuda::atomic_thread_fence <libcudacxx-extended-api-synchronization-atomic-atomic-thread-fence>`
+     - Memory order and scope dependent fence synchronization primitive.
+   * - :ref:`cuda::atomic::fetch_min <libcudacxx-extended-api-synchronization-atomic-atomic-fetch-min>`
+     - Atomically find the minimum of the stored value and a provided value.
+   * - :ref:`cuda::atomic::fetch_max <libcudacxx-extended-api-synchronization-atomic-atomic-fetch-max>`
+     - Atomically find the maximum of the stored value and a provided value.
+
+Concurrency Restrictions
+------------------------
+
+An object of type ``cuda::atomic`` or `cuda::std::atomic <https://en.cppreference.com/w/cpp/atomic/atomic>`_
+shall not be accessed concurrently by CPU and GPU threads unless:
+
+  - it is in unified memory and the `concurrentManagedAccess property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_116f9619ccc85e93bc456b8c69c80e78b>`_
+    is 1, or
+  - it is in CPU memory and the `hostNativeAtomicSupported property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_1ef82fd7d1d0413c7d6f33287e5b6306f>`_
+    is 1.
+
+Note, for objects of scopes other than ``cuda::thread_scope_system`` this is a data-race, and therefore also
+prohibited regardless of memory characteristics.
+
+Under CUDA Compute Capability 6 (Pascal), an object of type ``atomic`` may not be used:
+
+  - with automatic storage duration, or
+  - if ``is_always_lock_free()`` is ``false``.
+
+Under CUDA Compute Capability prior to 6 (Pascal), objects of type ``cuda::atomic`` or
+`cuda::std::atomic <https://en.cppreference.com/w/cpp/atomic/atomic>`_ may not be used.
+
+Implementation-Defined Behavior
+-------------------------------
+
+For each type ``T`` and :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``,
+the value of ``cuda::atomic<T, S>::is_always_lock_free()`` is as follows:
+
+.. list-table::
+   :widths: 25 25 50
+   :header-rows: 0
+
+   * - Type ``T``
+     - :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``
+     - ``cuda::atomic<T, S>::is_always_lock_free()``
+   * - Any valid type
+     - Any thread scope
+     - ``sizeof(T) <= 8``
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/atomic>
+
+   __global__ void example_kernel() {
+     // This atomic is suitable for all threads in the system.
+     cuda::atomic<int, cuda::thread_scope_system> a;
+
+     // This atomic has the same type as the previous one (`a`).
+     cuda::atomic<int> b;
+
+     // This atomic is suitable for all threads on the current processor (e.g. GPU).
+     cuda::atomic<int, cuda::thread_scope_device> c;
+
+     // This atomic is suitable for threads in the same thread block.
+     cuda::atomic<int, cuda::thread_scope_block> d;
+   }
+
+`See it on Godbolt <https://godbolt.org/z/avo3Evbee>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic/atomic_thread_fence.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic/atomic_thread_fence.rst
new file mode 100644
index 00000000..3d098382
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic/atomic_thread_fence.rst
@@ -0,0 +1,51 @@
+.. _libcudacxx-extended-api-synchronization-atomic-atomic-thread-fence:
+
+cuda::atomic::atomic_thread_fence
+=====================================
+
+Defined in header ``<cuda/atomic>``:
+
+.. code:: cuda
+
+   __host__ __device__
+   void cuda::atomic_thread_fence(cuda::std::memory_order order,
+                                  cuda::thread_scope scope = cuda::thread_scope_system);
+
+Establishes memory synchronization ordering of non-atomic and relaxed atomic accesses, as instructed by ``order``,
+for all threads within ``scope`` without an associated atomic operation. It has the same semantics as
+`cuda::std::atomic_thread_fence <https://en.cppreference.com/w/cpp/atomic/atomic_thread_fence>`_.
+
+Example
+-------
+
+The following code is an example of the :ref:`MessagePassing <libcudacxx-extended-api-memory-model-message-passing>` pattern:
+
+.. code:: cuda
+
+   #include <cstdio>
+   #include <cuda/atomic>
+   #include <cooperative_groups.h>
+
+   namespace cg = cooperative_groups;
+
+   __global__ void example_kernel(int* data, cuda::std::atomic_flag* flag) {
+     assert(cg::grid_group::size() == 2);
+     assert(cg::thread_block::size() == 1);
+
+     if (blockIdx.x == 0) {
+       *data = 42;
+       cuda::atomic_thread_fence(cuda::memory_order_release,
+                                 cuda::thread_scope_device);
+       flag->test_and_set(cuda::std::memory_order_relaxed);
+       flag->notify_one();
+     }
+     else {
+       // an atomic operation is required to set up the synchronization
+       flag->wait(false, cuda::std::memory_order_relaxed);
+       cuda::atomic_thread_fence(cuda::memory_order_acquire,
+                                 cuda::thread_scope_device);
+       std::printf("%d\n", *data); // Prints 42
+     }
+   }
+
+`See it on Godbolt <https://godbolt.org/z/aG37o5qxx>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic/fetch_max.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic/fetch_max.rst
new file mode 100644
index 00000000..6a41cbc5
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic/fetch_max.rst
@@ -0,0 +1,34 @@
+.. _libcudacxx-extended-api-synchronization-atomic-atomic-fetch-max:
+
+cuda::atomic::fetch_max
+===========================
+
+Defined in header ``<cuda/atomic>``:
+
+.. code:: cuda
+
+   template <typename T, cuda::thread_scope Scope>
+   __host__ __device__
+   T cuda::atomic<T, Scope>::fetch_max(T const& val,
+                                       cuda::std::memory_order order
+                                         = cuda::std::memory_order_seq_cst);
+
+Atomically find the maximum of the value stored in the ``cuda::atomic``
+and ``val``. The maximum is found using
+`cuda::std::max <https://en.cppreference.com/w/cpp/algorithm/max>`_.
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/atomic>
+
+   __global__ void example_kernel() {
+     cuda::atomic<int> a(0);
+     auto x = a.fetch_max(1);
+     auto y = a.load();
+     assert(x == 0 && y == 1);
+   }
+
+`See it on Godbolt <https://godbolt.org/z/rexn5T78G>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic/fetch_min.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic/fetch_min.rst
new file mode 100644
index 00000000..293ce692
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic/fetch_min.rst
@@ -0,0 +1,34 @@
+.. _libcudacxx-extended-api-synchronization-atomic-atomic-fetch-min:
+
+cuda::atomic::fetch_min
+===========================
+
+Defined in header ``<cuda/atomic>``:
+
+.. code:: cuda
+
+   template <typename T, cuda::thread_scope Scope>
+   __host__ __device__
+   T cuda::atomic<T, Scope>::fetch_min(T const& val,
+                                       cuda::std::memory_order order
+                                         = cuda::std::memory_order_seq_cst);
+
+Atomically find the minimum of the value stored in the ``cuda::atomic``
+and ``val``. The minimum is found using
+`cuda::std::min <https://en.cppreference.com/w/cpp/algorithm/min>`_.
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/atomic>
+
+   __global__ void example_kernel() {
+     cuda::atomic<int> a(1);
+     auto x = a.fetch_min(0);
+     auto y = a.load();
+     assert(x == 1 && y == 0);
+   }
+
+`See it on Godbolt <https://godbolt.org/z/vMj9e5hdv>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic_ref.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic_ref.rst
new file mode 100644
index 00000000..dc28a9e7
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic_ref.rst
@@ -0,0 +1,108 @@
+.. _libcudacxx-extended-api-synchronization-atomic-ref:
+
+``cuda::atomic_ref``
+====================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+
+Defined in header ``<cuda/atomic>``:
+
+.. code:: cuda
+
+   template <typename T, cuda::thread_scope Scope = cuda::thread_scope_system>
+   class cuda::atomic_ref;
+
+The class template ``cuda::atomic_ref`` is an extended form of `cuda::std::atomic_ref <https://en.cppreference.com/w/cpp/atomic/atomic_ref>`_
+that takes an additional :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` argument, defaulted to
+``cuda::std::thread_scope_system``.
+
+It has the same interface and semantics as `cuda::std::atomic_ref <https://en.cppreference.com/w/cpp/atomic/atomic_ref>`_,
+with the following additional operations. This class additionally deviates from the standard by being backported to C++11.
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - :ref:`cuda::atomic_ref::fetch_min <libcudacxx-extended-api-synchronization-atomic-atomic-fetch-min>`
+     - Atomically find the minimum of the stored value and a provided value.
+   * - :ref:`cuda::atomic_ref::fetch_max <libcudacxx-extended-api-synchronization-atomic-atomic-fetch-max>`
+     - Atomically find the maximum of the stored value and a provided value.
+
+
+Limitations
+-----------
+
+``cuda::atomic_ref<T>`` and ``cuda::std::atomic_ref<T>`` may only be instantiated when ``T`` satisfies ``sizeof(T) <= 8`` or ``sizeof(T) <= 16`` when requirements are met.
+
+The operations available to ``T`` when ``sizeof(T) == 16`` depend on the architecture:
+  - On SM70 and later: ``load`` and ``store`` are supported.
+  - On SM90 and later: ``fetch_*`` and synchronization operations are supported, implemented via atomic compare-and-swap (CAS).
+
+No object or subobject of an object referenced by an ``atomic_­ref`` shall be concurrently referenced by any other
+``atomic_­ref`` that has a different ``Scope``.
+
+For ``cuda::atomic_ref<T>`` and ``cuda::std::atomic_ref<T>`` the type ``T`` must satisfy the following:
+  - ``sizeof(T) <= 16``.
+  - The referenced object must be aligned to its size: ``alignof(T) == sizeof(T)``.
+  - ``T`` must not have "padding bits", i.e., T's `object representation <https://en.cppreference.com/w/cpp/language/object#Object_representation_and_value_representation>`_
+    must not have bits that do not participate in it's value representation.
+
+Concurrency Restrictions
+------------------------
+
+See :ref:`memory model <libcudacxx-extended-api-memory-model>` documentation for general restrictions on atomicity.
+
+With CUDA Compute Capability 6 (Pascal), an object of type ``atomic_ref`` may not be used:
+  - with a reference to an object with a automatic storage duration in a GPU thread, or
+  - if ``is_always_lock_free()`` is ``false``.
+
+For CUDA Compute Capability prior to 6 (Pascal), objects of type ``cuda::atomic_ref`` or
+`cuda::std::atomic_ref <https://en.cppreference.com/w/cpp/atomic/atomic_ref>`_ may not be used.
+
+Implementation-Defined Behavior
+-------------------------------
+
+For each type ``T`` and :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``, the value of
+``cuda::atomic_ref<T, S>::is_always_lock_free()`` and ``cuda::std::atomic_ref<T>::is_always_lock_free()`` is as follows:
+
+.. list-table::
+   :widths: 25 25 50
+   :header-rows: 0
+
+   * - Type ``T``
+     - :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``
+     - ``cuda::atomic_ref<T, S>::is_always_lock_free()``
+   * - Any valid type
+     - Any thread scope
+     - ``sizeof(T) <= 8``
+
+Types of ``T``, where ``sizeof(T) < 4``, are not natively supported by the underlying hardware. For these types atomic
+operations are emulated and will be drastically slower. Contention with contiguous memory in the current 4 byte boundary
+will be exacerbated. In these situations it is advisable to perform a hierarchical reduction to non-adjacent memory first.
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/atomic>
+
+   __global__ void example_kernel(int *gmem, int *pinned_mem) {
+     // This atomic is suitable for all threads in the system.
+     cuda::atomic_ref<int, cuda::thread_scope_system> a(*pinned_mem);
+
+     // This atomic has the same type as the previous one (`a`).
+     cuda::atomic_ref<int> b(*pinned_mem);
+
+     // This atomic is suitable for all threads on the current processor (e.g. GPU).
+     cuda::atomic_ref<int, cuda::thread_scope_device> c(*gmem);
+
+     __shared__ int shared_v;
+     // This atomic is suitable for threads in the same thread block.
+     cuda::atomic_ref<int, cuda::thread_scope_block> d(shared_v);
+   }
+
+`See it on Godbolt <https://godbolt.org/z/fr4K7ErEh>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier.rst
new file mode 100644
index 00000000..01cec02e
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier.rst
@@ -0,0 +1,297 @@
+.. _libcudacxx-extended-api-synchronization-barrier:
+
+``cuda::barrier``
+=================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   barrier/init
+   barrier/barrier_native_handle
+   barrier/barrier_arrive_tx
+   barrier/barrier_expect_tx
+
+Defined in header ``<cuda/barrier>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope,
+             typename CompletionFunction = /* unspecified */>
+   class cuda::barrier;
+
+The class template ``cuda::barrier`` is an extended form of `cuda::std::barrier <https://en.cppreference.com/w/cpp/thread/barrier>`_
+that takes an additional :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` argument.
+
+If ``!(scope == cuda::thread_block_scope && cuda::device::is_address_from(this, cuda::device::address_space::shared))``, then the semantics are the same as
+`cuda::std::barrier <https://en.cppreference.com/w/cpp/thread/barrier>`_, otherwise, see below.
+
+The ``cuda::barrier`` class templates extends ``cuda::std::barrier`` with the following additional operations:
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - :ref:`cuda::barrier::init <libcudacxx-extended-api-synchronization-barrier-barrier-init>`
+     - Initialize a ``cuda::barrier``.
+   * - :ref:`cuda::device::barrier_native_handle <libcudacxx-extended-api-synchronization-barrier-barrier-native-handle>`
+     - Get the native handle to a ``cuda::barrier``.
+   * - :ref:`cuda::device::barrier_arrive_tx <libcudacxx-extended-api-synchronization-barrier-barrier-arrive-tx>`
+     - Arrive on a ``cuda::barrier<cuda::thread_scope_block>`` with transaction count update.
+   * - :ref:`cuda::device::barrier_expect_tx <libcudacxx-extended-api-synchronization-barrier-barrier-expect-tx>`
+     - Update transaction count of ``cuda::barrier<cuda::thread_scope_block>``.
+
+If ``scope == cuda::thread_scope_block && cuda::device::is_address_from(this, cuda::device::address_space::shared)``, then the semantics of `[thread.barrier.class] <http://eel.is/c++draft/thread.barrier.class>`_
+of ISO/IEC IS 14882 (the C++ Standard) are modified as follows:
+
+   A barrier is a thread coordination mechanism whose lifetime consists
+   of a sequence of barrier phases, where each phase allows at most an
+   expected number of threads to block until the expected number of
+   threads **and the expected number of transaction-based asynchronous
+   operations** arrive at the barrier. Each *barrier phase* consists of
+   the following steps:
+
+   1. The *expected count* is decremented by each call to ``arrive``,\ ``arrive_and_drop``\ **,
+      or cuda::device::barrier_arrive_tx**.
+   2. **The transaction count is incremented by each call to cuda::device::barrier_arrive_tx and decremented by the
+      completion of transaction-based asynchronous operations such as cuda::memcpy_async_tx.**
+   3. Exactly once after **both** the *expected count* **and the transaction count** reach zero, a thread executes the
+      *completion step* during its call to ``arrive``, ``arrive_and_drop``, ``cuda::device::barrier_arrive_tx``,
+      or ``wait``, except that it is implementation-defined whether the step executes if no thread calls ``wait``.
+   4. When the completion step finishes, the *expected count* is reset to what was specified by the ``expected``
+      argument to the constructor, possibly adjusted by calls to ``arrive_and_drop``, and the next phase starts.
+
+   Concurrent invocations of the member functions of barrier **and the non-member barrier APIs in cuda::device**,
+   other than its destructor, do not introduce data races. The member functions ``arrive`` and ``arrive_and_drop``,
+   **and the non-member function cuda::device::barrier_arrive_tx**, execute atomically.
+
+.. rubric:: NVCC ``__shared__`` Initialization Warnings
+
+When using libcu++ with NVCC, a ``__shared__`` ``cuda::barrier`` will lead to the following warning because
+``__shared__`` variables are not initialized:
+
+.. code:: bash
+
+   warning: dynamic initialization is not supported for a function-scope static
+   __shared__ variable within a __device__/__global__ function
+
+It can be silenced using ``#pragma nv_diag_suppress static_var_with_dynamic_init``.
+
+To properly initialize a ``__shared__`` ``cuda::barrier``, use the
+:ref:`cuda::barrier::init <libcudacxx-extended-api-synchronization-barrier-barrier-init>` friend function.
+
+.. rubric:: Concurrency Restrictions
+
+An object of type ``cuda::barrier`` or ``cuda::std::barrier`` shall not be accessed concurrently by CPU and GPU threads unless:
+
+   - it is in unified memory and the `concurrentManagedAccess property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_116f9619ccc85e93bc456b8c69c80e78b>`_
+     is 1, or
+   - it is in CPU memory and the `hostNativeAtomicSupported property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_1ef82fd7d1d0413c7d6f33287e5b6306f>`_
+     is 1.
+
+Note, for objects of scopes other than ``cuda::thread_scope_system`` this is a data-race, and therefore also prohibited
+regardless of memory characteristics.
+
+Under CUDA Compute Capability 8 (Ampere) or above, when an object of type ``cuda::barrier<thread_scope_block>`` is
+placed in ``__shared__`` memory, the member function ``arrive`` performs a reduction of the arrival count among
+`coalesced threads <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#coalesced-group-cg>`_ followed
+by the arrival operation in one thread. Programs shall ensure that this transformation would not introduce errors,
+for example relative to the requirements of `thread.barrier.class paragraph 12 <https://eel.is/c++draft/thread.barrier.class#12>`_
+of ISO/IEC IS 14882 (the C++ Standard).
+
+Under CUDA Compute Capability 6 (Pascal) or prior, an object of type ``cuda::barrier`` or ``cuda::std::barrier`` may
+not be used.
+
+.. rubric:: Shared memory barriers with transaction count
+
+In addition to the arrival count, a ``cuda::barrier<thread_scope_block>`` object located in shared memory supports a
+`tx-count <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-mbarrier-tracking-async-operations>`_,
+which is used for tracking the completion of some asynchronous memory operations or transactions.
+The tx-count tracks the number of asynchronous transactions, in units specified by the asynchronous memory operation
+(typically bytes), that are outstanding and yet to be complete. This capability is exposed, starting with the Hopper
+architecture (CUDA Compute Capability 9).
+
+The tx-count of ``cuda::barrier`` must be set to the total amount of asynchronous memory operations, in units as
+specified by the asynchronous operations, to be tracked by the current phase. This can be achieved with the
+:ref:`cuda::device::barrier_arrive_tx <libcudacxx-extended-api-synchronization-barrier-barrier-arrive-tx>` function call.
+
+Upon completion of each of the asynchronous operations, the tx-count of the ``cuda::barrier`` will be updated and thus
+progress the ``cuda::barrier`` towards the completion of the current phase. This may complete the current phase.
+
+.. rubric:: Implementation-Defined Behavior
+
+For each :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S`` and ``CompletionFunction``
+``F``, the value of ``cuda::barrier<S, F>::max()`` is as follows:
+
+.. list-table::
+   :widths: 25 25 50
+   :header-rows: 0
+
+   * - :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``
+     - ``CompletionFunction`` ``F``
+     - ``barrier<S, F>::max()``
+   * - ``cuda::thread_scope_block``
+     - Default or user-provided
+     - ``(1 << 20) - 1``
+   * - *Not* ``cuda::thread_scope_block``
+     - Default
+     - ``cuda::std::numeric_limits<cuda::std::int32_t>::max()``
+   * - *Not* ``cuda::thread_scope_block``
+     - User-provided
+     - ``cuda::std::numeric_limits<cuda::std::ptrdiff_t>::max()``
+
+.. rubric:: Example
+
+.. code:: cuda
+
+   #include <cuda/barrier>
+
+   __global__ void example_kernel() {
+     // This barrier is suitable for all threads in the system.
+     cuda::barrier<cuda::thread_scope_system> a(10);
+
+     // This barrier has the same type as the previous one (`a`).
+     cuda::std::barrier<> b(10);
+
+     // This barrier is suitable for all threads on the current processor (e.g. GPU).
+     cuda::barrier<cuda::thread_scope_device> c(10);
+
+     // This barrier is suitable for all threads in the same thread block.
+     cuda::barrier<cuda::thread_scope_block> d(10);
+   }
+
+`See it on Godbolt <https://godbolt.org/z/7Kbz5qqhh>`__
+
+.. rubric:: Example: 1D TMA load of two buffers with arrival token (sm90+)
+
+The following example shows how to use TMA to load two tiles of data from global memory into shared memory:
+
+.. code:: cuda
+
+   #include <cuda/barrier>
+   #include <cuda/ptx>
+
+   // selects a single leader thread from the block
+   __device__ bool elect_one() {
+     const unsigned int tid = threadIdx.x;
+     const unsigned int warp_id = tid / 32;
+     const unsigned int uniform_warp_id = __shfl_sync(0xFFFFFFFF, warp_id, 0); // broadcast from lane 0
+     return uniform_warp_id == 0 && cuda::ptx::elect_sync(0xFFFFFFFF); // elect a leader thread among warp 0
+   }
+
+   __global__ void example_kernel(int* gmem1, double* gmem2) {
+     constexpr int tile_size = 1024;
+     __shared__ alignas(16)    int smem1[tile_size];
+     __shared__ alignas(16) double smem2[tile_size];
+
+     #pragma nv_diag_suppress static_var_with_dynamic_init
+     __shared__  cuda::barrier<cuda::thread_scope_block> bar;
+
+     // setup the barrier where each thread in the block arrives at
+     if (threadIdx.x == 0) {
+       init(&bar, blockDim.x);
+     }
+     __syncthreads(); // need to sync so other threads can arrive
+
+     // select a single thread from the block and issue two TMA bulk copy operations
+     const auto elected = elect_one();
+     if (elected) {
+       cuda::device::memcpy_async_tx(smem1, gmem1, cuda::aligned_size_t<16>(tile_size * sizeof(int)   ), bar);
+       cuda::device::memcpy_async_tx(smem2, gmem2, cuda::aligned_size_t<16>(tile_size * sizeof(double)), bar);
+     }
+
+     // arrive at the barrier
+     // the elected thread also updates the barrier's expect_tx with the **total** number of loaded bytes
+     const int tx_count = elected ? tile_size * (sizeof(int) + sizeof(double)) : 0;
+     auto token = cuda::device::barrier_arrive_tx(bar, 1, tx_count);
+
+     // wait for TMA copies to complete
+     bar.wait(cuda::std::move(token));
+
+     // process data in smem ...
+   }
+
+`See it on Godbolt <https://godbolt.org/z/ddhzGeWPE>`__
+
+Data is loaded from the global memory pointers ``gmem1`` and ``gmem2``
+into the shared memory buffers ``smem1`` and ``smem2``.
+The shared memory buffers have to be aligned to 16 bytes.
+Additionally, a single barrier with block scope is setup by a single leader thread.
+Each thread will arrive at the barrier, so ``blockDim.x`` is passed as arrival count to ``init``.
+This barrier is used to synchronize the asynchronous TMA copies with the rest of the threads in the block.
+The leader initiates the TMA copies using ``ptx::cp_async_bulk``,
+and updates the barrier's tx-count with the total number of bytes transferred by the TMA copy operations
+while arriving at the barrier using ``cuda::device::barrier_arrive_tx``.
+All other threads just arrive normally at the barrier.
+All threads then wait on the barrier to complete the current phase by calling ``wait``.
+Once ``wait`` returns, all data is available in shared memory.
+
+.. rubric:: Example: 1D TMA load of two buffers with barrier phase parity check (sm90+)
+
+.. code:: cuda
+
+   #include <cuda/barrier>
+   #include <cuda/ptx>
+
+   // selects a single leader thread from the block
+   __device__ bool elect_one() {
+     const unsigned int tid = threadIdx.x;
+     const unsigned int warp_id = tid / 32;
+     const unsigned int uniform_warp_id = __shfl_sync(0xFFFFFFFF, warp_id, 0); // broadcast from lane 0
+     return uniform_warp_id == 0 && cuda::ptx::elect_sync(0xFFFFFFFF); // elect a leader thread among warp 0
+   }
+
+   __global__ void example_kernel(int* gmem1, double* gmem2) {
+     constexpr int tile_size = 1024;
+     __shared__ alignas(16)    int smem1[tile_size];
+     __shared__ alignas(16) double smem2[tile_size];
+
+     #pragma nv_diag_suppress static_var_with_dynamic_init
+     __shared__  cuda::barrier<cuda::thread_scope_block> bar;
+
+     // setup the barrier where only the leader thread arrives
+     if (elect_one()) {
+       init(&bar, 1);
+
+       // issue two TMA bulk copy operations
+       cuda::device::memcpy_async_tx(smem1, gmem1, cuda::aligned_size_t<16>(tile_size * sizeof(int)   ), bar);
+       cuda::device::memcpy_async_tx(smem2, gmem2, cuda::aligned_size_t<16>(tile_size * sizeof(double)), bar);
+
+       // arrive and update the barrier's expect_tx with the **total** number of loaded bytes
+       (void)cuda::device::barrier_arrive_tx(bar, 1, tile_size * (sizeof(int) + sizeof(double)));
+     }
+     __syncthreads(); // need to sync so the barrier is set up when the other threads arrive and wait
+
+     // wait for the current barrier phase to complete
+     bar.wait_parity(0);
+
+     // process data in smem ...
+   }
+
+`See it on Godbolt <https://godbolt.org/z/oq85PoKKj>`__
+
+This is similar to the above example, but this time only the leader thread arrives at the barrier.
+This has the advantage that only one leader election is necessary
+and a single uniform datapath section is generated.
+This generally generates less instructions.
+Because now we don't get an arrival token in each thread, we cannot use ``wait(token)`` with all threads.
+Instead, we just wait until the end of the barrier's current phase using ``wait_parity(0)``
+(0 is the parity of the current phase).
+
+Before CCCL 3.2, ``bar.wait_parity(0);`` contained additional logic which may have lead to unnecessary instructions.
+If you are using CCCL below 3.2, you may replace this line with:
+
+.. code:: cuda
+
+   while (!cuda::ptx::mbarrier_try_wait_parity(cuda::device::barrier_native_handle(bar), 0))
+     ;
+
+.. rubric:: Example: 1D TMA load and store using `cuda::device::memcpy_async_tx` (sm90+)
+
+This example can be found in :ref:`libcudacxx-extended-api-asynchronous-operations-memcpy-async-tx-example`.
+
+
+.. rubric:: Example: 1D TMA load and store using `cuda::memcpy_async` (sm90+)
+
+This example can be found in the
+`CUDA programming guide <https://docs.nvidia.com/cuda/cuda-c-programming-guide/#using-tma-to-transfer-one-dimensional-arrays>`__.
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier/barrier_arrive_tx.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier/barrier_arrive_tx.rst
new file mode 100644
index 00000000..462c5507
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier/barrier_arrive_tx.rst
@@ -0,0 +1,77 @@
+.. _libcudacxx-extended-api-synchronization-barrier-barrier-arrive-tx:
+
+cuda::device::barrier_arrive_tx
+===================================
+
+Defined in header ``<cuda/barrier>``:
+
+.. code:: cuda
+
+   __device__
+   cuda::barrier<cuda::thread_scope_block>::arrival_token
+   cuda::device::barrier_arrive_tx(
+     cuda::barrier<cuda::thread_scope_block>& bar,
+     ptrdiff_t arrive_count_update,
+     ptrdiff_t transaction_count_update);
+
+Arrives at a barrier in shared memory, decrementing the arrival count and incrementing the expected transaction count.
+
+Preconditions
+-------------
+
+-  ``cuda::device::is_object_from(bar, cuda::device::address_space::shared) == true``
+-  ``1 <= arrive_count_update && transaction_count_update <= (1 << 20) - 1``
+-  ``0 <= transaction_count_update && transaction_count_update <= (1 << 20) - 1``
+
+Effects
+-------
+
+-  This function constructs an arrival_token object associated with the
+   phase synchronization point for the current phase. Then, decrements
+   the arrival count by ``arrive_count_update`` and increments the
+   expected transaction count by ``transaction_count_update``.
+-  This function executes atomically. The call to this function strongly
+   happens-before the start of the phase completion step for the current
+   phase.
+
+Notes
+-----
+
+This function can only be used under CUDA Compute Capability 9.0 (Hopper) or higher.
+
+To check if ``cuda::device::barrier_arrive_tx`` is available, use the ``__cccl_lib_local_barrier_arrive_tx``
+feature flag, as shown in the example code below.
+
+Return Value
+------------
+
+``cuda::device::barrier_arrive_tx`` returns the constructed ``arrival_token`` object.
+
+Example
+-------
+
+Below example shows only ``cuda::device::barrier_arrive_tx``. A more extensive example can be found in the
+:ref:`cuda::device::memcpy_async_tx <libcudacxx-extended-api-asynchronous-operations-memcpy-async-tx>` documentation.
+
+.. code:: cuda
+
+   #include <cuda/barrier>
+   #include <cuda/std/utility> // cuda::std::move
+
+   #ifndef  __cccl_lib_local_barrier_arrive_tx
+   static_assert(false, "Insufficient libcu++ version: cuda::device::arrive_tx is not yet available.");
+   #endif // __cccl_lib_local_barrier_arrive_tx
+
+   __global__ void example_kernel() {
+     __shared__ cuda::barrier<cuda::thread_scope_block> bar;
+     if (threadIdx.x == 0) {
+       init(&bar, blockDim.x);
+     }
+     __syncthreads();
+
+     auto token = cuda::device::barrier_arrive_tx(bar, 1, 0);
+
+     bar.wait(cuda::std::move(token));
+   }
+
+`See it on Godbolt <https://godbolt.org/z/1vxcGrT8j>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier/barrier_expect_tx.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier/barrier_expect_tx.rst
new file mode 100644
index 00000000..7a8a57db
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier/barrier_expect_tx.rst
@@ -0,0 +1,70 @@
+.. _libcudacxx-extended-api-synchronization-barrier-barrier-expect-tx:
+
+cuda::device::barrier_expect_tx
+===================================
+
+Defined in header ``<cuda/barrier>``:
+
+.. code:: cuda
+
+   __device__
+   void cuda::device::barrier_expect_tx(
+     cuda::barrier<cuda::thread_scope_block>& bar,
+     ptrdiff_t transaction_count_update);
+
+Increments the expected transaction count of a barrier in shared memory.
+
+Preconditions
+-------------
+
+-  ``cuda::device::is_object_from(bar, cuda::device::address_space::shared) == true``
+-  ``0 <= transaction_count_update && transaction_count_update <= (1 << 20) - 1``
+
+Effects
+-------
+
+-  This function increments the expected transaction count by transaction_count_update``.
+-  This function executes atomically.
+
+Notes
+-----
+
+This function can only be used under CUDA Compute Capability 9.0 (Hopper) or higher.
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/barrier>
+   #include <cuda/std/utility> // cuda::std::move
+
+   #if defined(__CUDA_MINIMUM_ARCH__) && __CUDA_MINIMUM_ARCH__ < 900
+   static_assert(false, "Insufficient CUDA Compute Capability: cuda::device::memcpy_expect_tx is not available.");
+   #endif // __CUDA_MINIMUM_ARCH__
+
+   __device__ alignas(16) int gmem_x[2048];
+
+   __global__ void example_kernel() {
+       using barrier_t = cuda::barrier<cuda::thread_scope_block>;
+     alignas(16) __shared__ int smem_x[1024];
+     __shared__ barrier_t bar;
+
+     if (threadIdx.x == 0) {
+       init(&bar, blockDim.x);
+     }
+     __syncthreads();
+
+     if (threadIdx.x == 0) {
+       cuda::device::memcpy_async_tx(smem_x, gmem_x, cuda::aligned_size_t<16>(sizeof(smem_x)), bar);
+       cuda::device::barrier_expect_tx(bar, sizeof(smem_x));
+     }
+     auto token = bar.arrive(1);
+
+     bar.wait(cuda::std::move(token));
+
+     // smem_x contains the contents of gmem_x[0], ..., gmem_x[1023]
+     smem_x[threadIdx.x] += 1;
+   }
+
+`See it on Godbolt <https://godbolt.org/z/9Yj89P76z>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier/barrier_native_handle.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier/barrier_native_handle.rst
new file mode 100644
index 00000000..d36a63f2
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier/barrier_native_handle.rst
@@ -0,0 +1,46 @@
+.. _libcudacxx-extended-api-synchronization-barrier-barrier-native-handle:
+
+cuda::device::barrier_native_handle
+=======================================
+
+Defined in header ``<cuda/barrier>``:
+
+.. code:: cuda
+
+   __device__ cuda::std::uint64_t* cuda::device::barrier_native_handle(
+     cuda::barrier<cuda::thread_scope_block>& bar);
+
+Returns a pointer to the native handle of a :ref:`cuda::barrier <libcudacxx-extended-api-synchronization-barrier>`
+if its scope is ``cuda::thread_scope_block`` and it is allocated in shared memory.
+The pointer is suitable for use with PTX instructions.
+
+Notes
+-----
+
+If ``bar`` is not in ``__shared__`` memory, the behavior is undefined.
+
+Return Value
+------------
+
+A pointer to the PTX "mbarrier" subobject of the :ref:`cuda::barrier <libcudacxx-extended-api-synchronization-barrier>`
+object.
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/barrier>
+
+   __global__ void example_kernel(cuda::barrier<cuda::thread_scope_block>& bar) {
+     auto ptr = cuda::device::barrier_native_handle(bar);
+
+     asm volatile (
+         "mbarrier.arrive.b64 _, [%0];"
+         :
+         : "l" (ptr)
+         : "memory");
+     // Equivalent to: `(void)b.arrive()`.
+   }
+
+`See it on Godbolt <https://godbolt.org/z/dr4798Y76>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier/init.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier/init.rst
new file mode 100644
index 00000000..3502644f
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier/init.rst
@@ -0,0 +1,53 @@
+.. _libcudacxx-extended-api-synchronization-barrier-barrier-init:
+
+cuda::barrier::init
+=======================
+
+Defined in header ``<cuda/barrier>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope,
+             typename CompletionFunction = /* unspecified */>
+   class barrier {
+   public:
+     // ...
+
+     __host__ __device__
+     friend void init(cuda::std::barrier* bar,
+                      cuda::std::ptrdiff_t expected,
+                      CompletionFunction cf = CompletionFunction{});
+   };
+
+The friend function ``cuda::barrier::init`` may be used to initialize an
+:ref:`cuda::barrier <libcudacxx-extended-api-synchronization-barrier>` that has not been initialized.
+
+When using libcu++ with NVCC, ``__shared__`` ``cuda::barrier`` will not have its constructors run because ``__shared__``
+variables are not initialized. ``cuda::barrier::init`` should be used to properly initialize such a
+:ref:`cuda::barrier <libcudacxx-extended-api-synchronization-barrier>`.
+
+An NVCC diagnostic warning about the ignored constructor will be emitted:
+
+.. code:: bash
+
+   warning: dynamic initialization is not supported for a function-scope static
+   __shared__ variable within a __device__/__global__ function
+
+It can be silenced using ``#pragma nv_diag_suppress static_var_with_dynamic_init``.
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/barrier>
+
+   // Disables `cuda::barrier` initialization warning.
+   #pragma nv_diag_suppress static_var_with_dynamic_init
+
+   __global__ void example_kernel() {
+     __shared__ cuda::barrier<cuda::thread_scope_block> bar;
+     init(&bar, 1);
+   }
+
+`See it on Godbolt <https://godbolt.org/z/nK5q3xh34>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/binary_semaphore.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/binary_semaphore.rst
new file mode 100644
index 00000000..e6b47a0e
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/binary_semaphore.rst
@@ -0,0 +1,76 @@
+.. _libcudacxx-extended-api-synchronization-binary-semaphore:
+
+``cuda::binary_semaphore``
+==========================
+
+Defined in header ``<cuda/semaphore>``:
+
+.. code:: cpp
+
+   namespace cuda {
+
+   template <cuda::thread_scope Scope>
+   using binary_semaphore = cuda::std::counting_semaphore<Scope, 1>;
+
+   }
+
+The class template ``cuda::binary_semaphore`` is an extended form of `cuda::std::binary_semaphore <https://en.cppreference.com/w/cpp/thread/counting_semaphore>`_
+that takes an additional :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` argument.
+``cuda::binary_semaphore`` has the same interface and semantics as
+`cuda::std::binary_semaphore <https://en.cppreference.com/w/cpp/thread/counting_semaphore>`_, but
+``cuda::binary_semaphore`` is a class template.
+
+Concurrency Restrictions
+------------------------
+
+An object of type ``cuda::binary_semaphore`` or ``cuda::std::binary_semaphore``, shall not be accessed concurrently by
+CPU and GPU threads unless:
+
+   - it is in unified memory and the `concurrentManagedAccess property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_116f9619ccc85e93bc456b8c69c80e78b>`_
+     is 1, or
+   - it is in CPU memory and the `hostNativeAtomicSupported property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_1ef82fd7d1d0413c7d6f33287e5b6306f>`_
+     is 1.
+
+Note, for objects of scopes other than ``cuda::thread_scope_system`` this is a data-race, and therefore also prohibited
+regardless of memory characteristics.
+
+Under CUDA Compute Capability 6 (Pascal) or prior, an object of type ``cuda::binary_semaphore`` or
+``cuda::std::binary_semaphore`` may not be used.
+
+Implementation-Defined Behavior
+-------------------------------
+
+For each :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``,
+``cuda::binary_semaphore<S>::max()`` is as follows:
+
+.. list-table::
+   :widths: 50 50
+   :header-rows: 0
+
+   * - :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``
+     - ``cuda::binary_semaphore<S>::max()``
+   * - Any thread scope
+     - ``1``
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/semaphore>
+
+   __global__ void example_kernel() {
+     // This semaphore is suitable for all threads in the system.
+     cuda::binary_semaphore<cuda::thread_scope_system> a;
+
+     // This semaphore has the same type as the previous one (`a`).
+     cuda::std::binary_semaphore<> b;
+
+     // This semaphore is suitable for all threads on the current processor (e.g. GPU).
+     cuda::binary_semaphore<cuda::thread_scope_device> c;
+
+     // This semaphore is suitable for all threads in the same thread block.
+     cuda::binary_semaphore<cuda::thread_scope_block> d;
+   }
+
+`See it on Godbolt <https://godbolt.org/z/eKfjYYz58>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/counting_semaphore.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/counting_semaphore.rst
new file mode 100644
index 00000000..1fcd5576
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/counting_semaphore.rst
@@ -0,0 +1,72 @@
+.. _libcudacxx-extended-api-synchronization-counting-semaphore:
+
+``cuda::counting_semaphore``
+============================
+
+Defined in header ``<cuda/semaphore>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope,
+             cuda::std::ptrdiff_t LeastMaxValue = /* implementation-defined */>
+   class cuda::counting_semaphore;
+
+The class template ``cuda::counting_semaphore`` is an extended form of `cuda::std::counting_semaphore <https://en.cppreference.com/w/cpp/thread/counting_semaphore>`_
+that takes an additional :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` argument.
+``cuda::counting_semaphore`` has the same interface and semantics as
+`cuda::std::counting_semaphore <https://en.cppreference.com/w/cpp/thread/counting_semaphore>`_.
+
+Concurrency Restrictions
+------------------------
+
+An object of type ``cuda::counting_semaphore`` or ``cuda::std::counting_semaphore``, shall not be accessed concurrently
+by CPU and GPU threads unless:
+
+   - it is in unified memory and the `concurrentManagedAccess property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_116f9619ccc85e93bc456b8c69c80e78b>`_
+     is 1, or
+   - it is in CPU memory and the `hostNativeAtomicSupported property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_1ef82fd7d1d0413c7d6f33287e5b6306f>`_
+     is 1.
+
+Note, for objects of scopes other than ``cuda::thread_scope_system`` this is a data-race, and therefore also prohibited
+regardless of memory characteristics.
+
+Under CUDA Compute Capability 6 (Pascal) or prior, an object of type ``cuda::counting_semaphore`` or
+``cuda::std::counting_semaphore`` may not be used.
+
+Implementation-Defined Behavior
+-------------------------------
+
+For each :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S`` and least maximum value
+``V``, ``cuda::counting_semaphore<S, V>::max()`` is as follows:
+
+.. list-table::
+   :widths: 50 50
+   :header-rows: 0
+
+   * - :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``
+     - ``cuda::binary_semaphore<S>::max()``
+   * - Any thread scope
+     - ``cuda::std::numeric_limits<cuda::std::ptrdiff_t>::max()``
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/semaphore>
+
+   __global__ void example_kernel() {
+     // This semaphore is suitable for all threads in the system.
+     cuda::counting_semaphore<cuda::thread_scope_system> a;
+
+     // This semaphore has the same type as the previous one (`a`).
+     cuda::std::counting_semaphore<> b;
+
+     // This semaphore is suitable for all threads on the current processor (e.g. GPU).
+     cuda::counting_semaphore<cuda::thread_scope_device> c;
+
+     // This semaphore is suitable for all threads in the same thread block.
+     cuda::counting_semaphore<cuda::thread_scope_block> d;
+   }
+
+`See it on Godbolt <https://godbolt.org/z/3YrjjTvG6>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/latch.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/latch.rst
new file mode 100644
index 00000000..4fd6d59b
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/latch.rst
@@ -0,0 +1,70 @@
+.. _libcudacxx-extended-api-synchronization-latch:
+
+``cuda::latch``
+===============
+
+Defined in header ``<cuda/latch>``:
+
+.. code:: cpp
+
+   template <cuda::thread_scope Scope>
+   class cuda::latch;
+
+The class template ``cuda::latch`` is an extended form of `cuda::std::latch <https://en.cppreference.com/w/cpp/thread/latch>`_
+takes an additional :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` argument.
+It has the same interface and semantics as `cuda::std::latch <https://en.cppreference.com/w/cpp/thread/latch>`_.
+
+Concurrency Restrictions
+------------------------
+
+An object of type ``cuda::latch`` or `cuda::std::latch <https://en.cppreference.com/w/cpp/thread/latch>`_ shall not
+be accessed concurrently by CPU and GPU threads unless:
+
+   - it is in unified memory and the `concurrentManagedAccess property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_116f9619ccc85e93bc456b8c69c80e78b>`_
+     is 1, or
+   - it is in CPU memory and the `hostNativeAtomicSupported property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_1ef82fd7d1d0413c7d6f33287e5b6306f>`_
+     is 1.
+
+Note, for objects of scopes other than ``cuda::thread_scope_system`` this is a data-race, and therefore also prohibited
+regardless of memory characteristics.
+
+Under CUDA Compute Capability 6 (Pascal) or prior, an object of type ``cuda::latch`` or
+`cuda::std::latch <https://en.cppreference.com/w/cpp/thread/latch>`_ may not be used.
+
+Implementation-Defined Behavior
+-------------------------------
+
+For each :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``, the value of
+``cuda::latch<S>::max()`` is as follows:
+
+.. list-table::
+   :widths: 50 50
+   :header-rows: 0
+
+   * - :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``
+     - ``cuda::latch<S>::max()``
+   * - Any thread scope
+     - ``cuda::std::numeric_limits<cuda::std::ptrdiff_t>::max()``
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/latch>
+
+   __global__ void example_kernel() {
+     // This latch is suitable for all threads in the system.
+     cuda::latch<cuda::thread_scope_system> a(10);
+
+     // This latch has the same type as the previous one (`a`).
+     cuda::std::latch b(10);
+
+     // This latch is suitable for all threads on the current processor (e.g. GPU).
+     cuda::latch<cuda::thread_scope_device> c(10);
+
+     // This latch is suitable for all threads in the same thread block.
+     cuda::latch<cuda::thread_scope_block> d(10);
+   }
+
+`See it on Godbolt <https://godbolt.org/z/8v4dcK7fa>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline.rst
new file mode 100644
index 00000000..a83d8a68
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline.rst
@@ -0,0 +1,172 @@
+.. _libcudacxx-extended-api-synchronization-pipeline:
+
+``cuda::pipeline``
+==================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   pipeline/role
+   pipeline/shared_state
+   pipeline/destructor
+   pipeline/make_pipeline
+   pipeline/quit
+   pipeline/consumer_release
+   pipeline/consumer_wait
+   pipeline/consumer_wait_prior
+   pipeline/pipeline_producer_commit
+   pipeline/producer_acquire
+   pipeline/producer_commit
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope>
+   class cuda::pipeline {
+   public:
+     pipeline() = delete;
+
+     __host__ __device__ ~pipeline();
+
+     pipeline& operator=(pipeline const&) = delete;
+
+     __host__ __device__ void producer_acquire();
+
+     __host__ __device__ void producer_commit();
+
+     __host__ __device__ void consumer_wait();
+
+     template <typename Rep, typename Period>
+     __host__ __device__ bool consumer_wait_for(cuda::std::chrono::duration<Rep, Period> const& duration);
+
+     template <typename Clock, typename Duration>
+     __host__ __device__
+     bool consumer_wait_until(cuda::std::chrono::time_point<Clock, Duration> const& time_point);
+
+     __host__ __device__ void consumer_release();
+
+     __host__ __device__ bool quit();
+   };
+
+The class template ``cuda::pipeline`` provides a coordination mechanism which can sequence
+:ref:`asynchronous operations <libcudacxx-extended-api-asynchronous-operations>`, such as
+:ref:`cuda::memcpy_async <libcudacxx-extended-api-asynchronous-operations-memcpy-async>`, into stages.
+
+A thread interacts with a *pipeline stage* using the following pattern:
+
+  1. Acquire the pipeline stage.
+  2. Commit some operations to the stage.
+  3. Wait for the previously committed operations to complete.
+  4. Release the pipeline stage.
+
+For :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``s`` other than
+``cuda::thread_scope_thread``, a
+:ref:`cuda::pipeline_shared_state <libcudacxx-extended-api-synchronization-pipeline-pipeline-shared-state>` is
+required to coordinate the participating threads.
+
+*Pipelines* can be either *unified* or *partitioned*. In a *unified pipeline*, all the participating threads are both
+producers and consumers. In a *partitioned pipeline*, each participating thread is either a producer or a consumer.
+
+.. rubric:: Template Parameters
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``Scope``
+     - The scope of threads participating in the *pipeline*.
+
+.. rubric:: Member Functions
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 1
+
+   * - Member function
+     - Description
+   * - (constructor) [deleted]
+     - ``cuda::pipeline`` is not constructible.
+   * - ``operator=`` [deleted]
+     - ``cuda::pipeline`` is not assignable.
+   * - :ref:`(destructor) <libcudacxx-extended-api-synchronization-pipeline-pipeline-destructor>`
+     -  Destroys the ``cuda::pipeline``.
+   * - :ref:`producer_acquire <libcudacxx-extended-api-synchronization-pipeline-pipeline-producer-acquire>`
+     - Blocks the current thread until the next *pipeline stage* is available.
+   * - :ref:`producer_commit <libcudacxx-extended-api-synchronization-pipeline-pipeline-producer-commit>`
+     - Commits operations previously issued by the current thread to the current *pipeline stage*.
+   * - :ref:`consumer_wait <libcudacxx-extended-api-synchronization-pipeline-pipeline-consumer-wait>`
+     - Blocks the current thread until all operations committed to the current *pipeline stage* complete.
+   * - :ref:`consumer_wait_for <libcudacxx-extended-api-synchronization-pipeline-pipeline-consumer-wait>`
+     - Blocks the current thread until all operations committed to the current *pipeline stage* complete or after the
+       specified timeout duration.
+   * - :ref:`consumer_wait_until <libcudacxx-extended-api-synchronization-pipeline-pipeline-consumer-wait>`
+     - Blocks the current thread until all operations committed to the current *pipeline stage* complete or until
+       specified time point has been reached.
+   * - :ref:`consumer_release <libcudacxx-extended-api-synchronization-pipeline-pipeline-consumer-release>`
+     - Release the current *pipeline stage*.
+   * - :ref:`quit <libcudacxx-extended-api-synchronization-pipeline-pipeline-quit>`
+     - Quits current thread's participation in the *pipeline*.
+
+.. note::
+
+   - A thread role cannot change during the lifetime of the pipeline object.
+
+.. rubric:: Example
+
+.. code:: cuda
+
+   #include <cuda/pipeline>
+   #include <cooperative_groups.h>
+
+   // Disables `pipeline_shared_state` initialization warning.
+   #pragma nv_diag_suppress static_var_with_dynamic_init
+
+   template <typename T>
+   __device__ void compute(T* ptr);
+
+   template <typename T>
+   __global__ void example_kernel(T* global0, T* global1, cuda::std::size_t subset_count) {
+     extern __shared__ T s[];
+     auto group = cooperative_groups::this_thread_block();
+     T* shared[2] = { s, s + 2 * group.size() };
+
+     // Create a pipeline.
+     constexpr auto scope = cuda::thread_scope_block;
+     constexpr auto stages_count = 2;
+     __shared__ cuda::pipeline_shared_state<scope, stages_count> shared_state;
+     auto pipeline = cuda::make_pipeline(group, &shared_state);
+
+     // Prime the pipeline.
+     pipeline.producer_acquire();
+     cuda::memcpy_async(group, shared[0],
+                        &global0[0], sizeof(T) * group.size(), pipeline);
+     cuda::memcpy_async(group, shared[0] + group.size(),
+                        &global1[0], sizeof(T) * group.size(), pipeline);
+     pipeline.producer_commit();
+
+     // Pipelined copy/compute.
+     for (cuda::std::size_t subset = 1; subset < subset_count; ++subset) {
+       pipeline.producer_acquire();
+       cuda::memcpy_async(group, shared[subset % 2],
+                          &global0[subset * group.size()],
+                          sizeof(T) * group.size(), pipeline);
+       cuda::memcpy_async(group, shared[subset % 2] + group.size(),
+                          &global1[subset * group.size()],
+                          sizeof(T) * group.size(), pipeline);
+       pipeline.producer_commit();
+       pipeline.consumer_wait();
+       compute(shared[(subset - 1) % 2]);
+       pipeline.consumer_release();
+     }
+
+     // Drain the pipeline.
+     pipeline.consumer_wait();
+     compute(shared[(subset_count - 1) % 2]);
+     pipeline.consumer_release();
+   }
+
+   template void __global__ example_kernel<int>(int*, int*, cuda::std::size_t);
+
+`See it on Godbolt <https://godbolt.org/z/zc41bWvja>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/consumer_release.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/consumer_release.rst
new file mode 100644
index 00000000..ca5e897f
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/consumer_release.rst
@@ -0,0 +1,20 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-consumer-release:
+
+cuda::pipeline::consumer_release
+====================================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope>
+   __host__ __device__
+   void cuda::pipeline<Scope>::consumer_release();
+
+Releases the current *pipeline stage*.
+
+.. note::
+
+   - If the calling thread is a *producer thread*, the behavior is undefined.
+   - If the pipeline is in a :ref:`quitted state <libcudacxx-extended-api-synchronization-pipeline-pipeline-quit>`,
+     the behavior is undefined.
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/consumer_wait.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/consumer_wait.rst
new file mode 100644
index 00000000..2316ad40
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/consumer_wait.rst
@@ -0,0 +1,55 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-consumer-wait:
+
+cuda::pipeline::consumer_wait
+=================================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   // (1)
+   template <cuda::thread_scope Scope>
+   __host__ __device__
+   void cuda::pipeline<Scope>::consumer_wait();
+
+   // (2)
+   template <cuda::thread_scope Scope>
+   template <typename Rep, typename Period>
+   __host__ __device__
+   bool cuda::pipeline<Scope>::consumer_wait_for(
+     cuda::std::chrono::duration<Rep, Period> const& duration);
+
+   // (3)
+   template <cuda::thread_scope Scope>
+   template <typename Clock, typename Duration>
+   __host__ __device__
+   bool cuda::pipeline<Scope>::consumer_wait_until(
+     cuda::std::chrono::time_point<Clock, Duration> const& time_point);
+
+1. Blocks the current thread until all operations committed to the current *pipeline stage* complete.
+2. Blocks the current thread until all operations committed to the current *pipeline stage* complete or after the
+   specified timeout duration.
+3. Blocks the current thread until all operations committed to the current *pipeline stage* complete or until specified
+   time point has been reached.
+
+.. rubric:: Parameters
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``duration``
+     - An object of type ``cuda::std::chrono::duration`` representing the maximum time to spend waiting.
+   * - ``time_point``
+     - An object of type ``cuda::std::chrono::time_point`` representing the time when to stop waiting.
+
+
+.. rubric:: Return Value
+
+``false`` if the *wait* timed out, ``true`` otherwise.
+
+.. note::
+
+   - If the calling thread is a *producer thread*, the behavior is undefined.
+   - If the pipeline is in a :ref:`quitted state <libcudacxx-extended-api-synchronization-pipeline-pipeline-quit>`,
+     the behavior is undefined.
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/consumer_wait_prior.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/consumer_wait_prior.rst
new file mode 100644
index 00000000..1823275f
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/consumer_wait_prior.rst
@@ -0,0 +1,66 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-consumer-wait-prior:
+
+cuda::pipeline_consumer_wait_prior
+======================================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::std::uint8_t Prior>
+   __host__ __device__
+   void cuda::pipeline_consumer_wait_prior(cuda::pipeline<thread_scope_thread>& pipe);
+
+Let *Stage* be the pipeline stage ``Prior`` stages before the current one (counting the current one).
+Blocks the current thread until all operations committed to *pipeline stages* up to *Stage* complete.
+All stages up to *Stage* (exclusive) are implicitly released.
+
+.. rubric:: Template Parameters
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``Prior``
+     - The index of the pipeline stage *Stage* (see above) counting up from the current one. The index of the current stage is ``0``.
+
+.. rubric:: Parameters
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``pipe``
+     - The thread-scoped ``cuda::pipeline`` object to wait on.
+
+.. note::
+
+   - If the pipeline is in a :ref:`quitted state <libcudacxx-extended-api-synchronization-pipeline-pipeline-quit>`,
+     the behavior is undefined.
+
+.. rubric:: Example
+
+.. code:: cuda
+
+   #include <cuda/pipeline>
+
+   __global__ void example_kernel(uint64_t* global, cuda::std::size_t element_count) {
+     extern __shared__ uint64_t shared[];
+
+     cuda::pipeline<cuda::thread_scope_thread> pipe = cuda::make_pipeline();
+     for (cuda::std::size_t i = 0; i < element_count; ++i) {
+       pipe.producer_acquire();
+       cuda::memcpy_async(shared + i, global + i, sizeof(*global), pipe);
+       pipe.producer_commit();
+     }
+
+     // Wait for operations committed in all stages but the last one.
+     cuda::pipeline_consumer_wait_prior<1>(pipe);
+     pipe.consumer_release();
+
+     // Wait for operations committed in all stages.
+     cuda::pipeline_consumer_wait_prior<0>(pipe);
+     pipe.consumer_release();
+   }
+
+`See it on Godbolt <https://godbolt.org/z/aT5hb84PY>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/destructor.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/destructor.rst
new file mode 100644
index 00000000..1db81839
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/destructor.rst
@@ -0,0 +1,15 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-destructor:
+
+cuda::pipeline::~pipeline
+=============================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope>
+   __host__ __device__
+   cuda::pipeline<Scope>::~pipeline();
+
+Destructs the pipeline. Calls :ref:`cuda::pipeline::quit <libcudacxx-extended-api-synchronization-pipeline-pipeline-quit>`
+if it was not called by the current thread and destructs the pipeline.
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/make_pipeline.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/make_pipeline.rst
new file mode 100644
index 00000000..4e0b5781
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/make_pipeline.rst
@@ -0,0 +1,125 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-make-pipeline:
+
+cuda::make_pipeline
+=======================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   // (1)
+   __host__ __device__
+   cuda::pipeline<cuda::thread_scope_thread> cuda::make_pipeline();
+
+   // (2)
+   template <typename Group,
+             cuda::thread_scope Scope,
+             cuda::std::uint8_t StagesCount>
+   __host__ __device__
+   cuda::pipeline<Scope>
+   cuda::make_pipeline(Group const& group,
+                       cuda::pipeline_shared_state<Scope, StagesCount>* shared_state);
+
+   // (3)
+   template <typename Group,
+             cuda::thread_scope Scope,
+             cuda::std::uint8_t StagesCount>
+   __host__ __device__
+   cuda::pipeline<Scope>
+   cuda::make_pipeline(Group const& group,
+                       cuda::pipeline_shared_state<Scope, StagesCount>* shared_state,
+                       cuda::std::size_t producer_count);
+
+   // (4)
+   template <typename Group,
+             cuda::thread_scope Scope,
+             cuda::std::uint8_t StagesCount>
+   __host__ __device__
+   cuda::pipeline<Scope>
+   cuda::make_pipeline(Group const& group,
+                       cuda::pipeline_shared_state<Scope, StagesCount>* shared_state,
+                       cuda::pipeline_role role);
+
+1. Creates a *unified pipeline* such that the calling thread is the only participating thread and performs both
+   producer and consumer actions.
+2. Creates a *unified pipeline* such that all the threads in ``group`` are performing both producer and consumer actions.
+3. Creates a *partitioned pipeline* such that ``producer_threads`` number of threads in ``group`` are performing
+   producer actions while the others are performing consumer actions.
+4. Creates a *partitioned pipeline* where each thread's role is explicitly specified.
+
+.. rubric:: Template Parameters
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``Group``
+     - A type satisfying the :ref:`ThreadGroup <libcudacxx-extended-api-thread-groups>` concept.
+
+.. rubric:: Parameters
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``group``
+     - The group of threads.
+   * - ``shared_state``
+     - A pointer to an object of type :ref:`cuda::pipeline_shared_state\<Scope\> <libcudacxx-extended-api-synchronization-pipeline-pipeline-shared-state>`
+       with ``Scope`` including all the threads in ``group``.
+   * - ``producer_count``
+     - The number of *producer threads* in the pipeline.
+   * - ``role``
+     - The role of the current thread in the pipeline.
+
+.. rubric:: Return Value
+
+A ``cuda::pipeline`` object.
+
+.. note::
+
+  - All threads in ``group`` acquire collective ownership of the ``shared_state`` storage.
+  - ``make_pipeline`` must be invoked by every threads in ``group`` such that ``group::sync`` may be invoked.
+  - ``shared_state`` and ``producer_count`` must be the same across all threads in ``group``, else the behavior is undefined.
+  - ``producer_count`` must be strictly inferior to ``group::size``, otherwise the behavior is undefined.
+
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-make-pipeline-example:
+
+.. rubric:: Example
+
+.. code:: cuda
+
+   #include <cuda/pipeline>
+   #include <cooperative_groups.h>
+
+   // Disables `pipeline_shared_state` initialization warning.
+   #pragma nv_diag_suppress static_var_with_dynamic_init
+
+   __global__ void example_kernel() {
+     __shared__ cuda::pipeline_shared_state<cuda::thread_scope_block, 2> pss0;
+     __shared__ cuda::pipeline_shared_state<cuda::thread_scope_block, 2> pss1;
+     __shared__ cuda::pipeline_shared_state<cuda::thread_scope_block, 2> pss2;
+
+     auto group = cooperative_groups::this_thread_block();
+
+     // Create a single thread scoped pipeline.
+     cuda::pipeline<cuda::thread_scope_thread> p0 = cuda::make_pipeline();
+
+     // Create a unified block-scoped pipeline.
+     cuda::pipeline<cuda::thread_scope_block> p1 = cuda::make_pipeline(group, &pss0);
+
+     // Create a partitioned block-scoped pipeline where half the threads are producers.
+     cuda::std::size_t producer_count = group.size() / 2;
+     cuda::pipeline<cuda::thread_scope_block> p2
+       = cuda::make_pipeline(group, &pss1, producer_count);
+
+     // Create a partitioned block-scoped pipeline where all threads with an even
+     // `thread_rank` are producers.
+     auto thread_role = (group.thread_rank() % 2)
+                        ? cuda::pipeline_role::producer
+                       : cuda::pipeline_role::consumer;
+     cuda::pipeline<cuda::thread_scope_block> p3
+       = cuda::make_pipeline(group, &pss2, thread_role);
+   }
+
+`See it on Godbolt <https://godbolt.org/z/aPcGEr64j>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/pipeline_producer_commit.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/pipeline_producer_commit.rst
new file mode 100644
index 00000000..5e438e36
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/pipeline_producer_commit.rst
@@ -0,0 +1,60 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-pipeline-producer-commit:
+
+cuda::pipeline_producer_commit
+==================================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope>
+   __host__ __device__
+   void cuda::pipeline_producer_commit(cuda::pipeline<cuda::thread_scope_thread>& pipe,
+                                       cuda::barrier<Scope>& bar);
+
+Binds operations previously issued by the current thread to the named ``cuda::barrier`` such that a
+``cuda::barrier::arrive`` is performed on completion. The bind operation implicitly increments the barrier's
+current phase to account for the subsequent ``cuda::barrier::arrive``, resulting in a net change of 0.
+
+.. rubric:: Parameters
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``pipe``
+     - The thread-scoped ``cuda::pipeline`` object to wait on.
+   * - ``bar``
+     - The ``cuda::barrier`` to arrive on.
+
+.. note::
+
+   - If the pipeline is in a :ref:`quitted state <libcudacxx-extended-api-synchronization-pipeline-pipeline-quit>`,
+     the behavior is undefined.
+
+.. rubric:: Example
+
+.. code:: cuda
+
+   #include <cuda/pipeline>
+
+   // Disables `barrier` initialization warning.
+   #pragma nv_diag_suppress static_var_with_dynamic_init
+
+   __global__ void
+   example_kernel(cuda::std::uint64_t* global, cuda::std::size_t element_count) {
+     extern __shared__ cuda::std::uint64_t shared[];
+     __shared__ cuda::barrier<cuda::thread_scope_block> barrier;
+
+     init(&barrier, 1);
+     cuda::pipeline<cuda::thread_scope_thread> pipe = cuda::make_pipeline();
+
+     pipe.producer_acquire();
+     for (cuda::std::size_t i = 0; i < element_count; ++i)
+       cuda::memcpy_async(shared + i, global + i, sizeof(*global), pipe);
+     pipeline_producer_commit(pipe, barrier);
+     barrier.arrive_and_wait();
+     pipe.consumer_release();
+   }
+
+`See it on Godbolt <https://godbolt.org/z/sGzKe9obf>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/producer_acquire.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/producer_acquire.rst
new file mode 100644
index 00000000..e5eb9a11
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/producer_acquire.rst
@@ -0,0 +1,20 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-producer-acquire:
+
+cuda::pipeline::producer_acquire
+====================================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope>
+   __host__ __device__
+   void cuda::pipeline<Scope>::producer_acquire();
+
+Blocks the current thread until the next *pipeline stage* is available.
+
+.. note::
+
+   - If the calling thread is a *consumer thread*, the behavior is undefined.
+   - If the pipeline is in a :ref:`quitted state <libcudacxx-extended-api-synchronization-pipeline-pipeline-quit>`,
+     the behavior is undefined.
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/producer_commit.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/producer_commit.rst
new file mode 100644
index 00000000..260942dd
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/producer_commit.rst
@@ -0,0 +1,20 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-producer-commit:
+
+cuda::pipeline::producer_commit
+===================================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope>
+   __host__ __device__
+   void cuda::pipeline<Scope>::producer_commit();
+
+Commits operations previously issued by the current thread to the current *pipeline stage*.
+
+.. note::
+
+   - If the calling thread is a *consumer thread*, the behavior is undefined.
+   - If the pipeline is in a :ref:`quitted state <libcudacxx-extended-api-synchronization-pipeline-pipeline-quit>`,
+     the behavior is undefined.
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/quit.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/quit.rst
new file mode 100644
index 00000000..7c51c30e
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/quit.rst
@@ -0,0 +1,27 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-quit:
+
+cuda::pipeline::quit
+========================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope>
+   __host__ __device__
+   bool cuda::pipeline<Scope>::quit();
+
+Quits the current thread's participation in the collective ownership of the corresponding
+:ref:`cuda::pipeline_shared_state <libcudacxx-extended-api-synchronization-pipeline-pipeline-shared-state>`.
+Ownership of :ref:`cuda::pipeline_shared_state <libcudacxx-extended-api-synchronization-pipeline-pipeline-shared-state>`
+is released by the last invoking thread.
+
+.. rubric:: Return Value
+
+``true`` if ownership of the *shared state* was released, otherwise ``false``.
+
+.. note::
+
+   - After the completion of a call to ``cuda::pipeline::quit``, no other operations other than
+     :ref:`cuda::pipeline::~pipeline <libcudacxx-extended-api-synchronization-pipeline-pipeline-destructor>` may
+     called by the current thread.
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/role.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/role.rst
new file mode 100644
index 00000000..50e2b06c
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/role.rst
@@ -0,0 +1,32 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-role:
+
+cuda::pipeline_role
+=======================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   enum class pipeline_role : /* unspecified */ {
+     producer,
+     consumer
+   };
+
+``cuda::pipeline_role`` specifies the role of a particular thread in a partitioned producer/consumer pipeline.
+
+.. rubric:: Constants
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``cuda::pipeline_role::producer``
+     - A producer thread that generates data and issuing
+       :ref:`asynchronous operations <libcudacxx-extended-api-asynchronous-operations>`.
+   * - ``cuda::pipeline_role::consumer``
+     - A consumer thread that consumes data and waiting for previously
+       :ref:`asynchronous operations <libcudacxx-extended-api-asynchronous-operations>` to complete.
+
+.. rubric:: Example
+
+See the :ref:`cuda::make_pipeline example <libcudacxx-extended-api-synchronization-pipeline-pipeline-make-pipeline-example>`.
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/shared_state.rst b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/shared_state.rst
new file mode 100644
index 00000000..692bda91
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/shared_state.rst
@@ -0,0 +1,119 @@
+
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-shared-state:
+
+cuda::pipeline_shared_state
+===============================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope, cuda::std::uint8_t StagesCount>
+   class cuda::pipeline_shared_state {
+   public:
+     __host__ __device__
+     pipeline_shared_state();
+
+     ~pipeline_shared_state() = default;
+
+     pipeline_shared_state(pipeline_shared_state const&) = delete;
+
+     pipeline_shared_state(pipeline_shared_state&&) = delete;
+   };
+
+The class template ``cuda::pipeline_shared_state`` is a storage type used to coordinate the threads participating
+in a ``cuda::pipeline``.
+
+.. rubric:: Template Parameters
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``Scope``
+     - A :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` denoting a scope including all
+       the threads participating in the ``cuda::pipeline``. ``Scope`` cannot be ``cuda::thread_scope_thread``.
+   * - ``StagesCount``
+     - The number of stages for the *pipeline*.
+
+.. rubric:: Member Functions
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 1
+
+   * - Member function
+     - Description
+   * - ``(constructor)``
+     - Constructs a ``cuda::pipeline_shared_state``.
+   * - ``(destructor)`` [implicitly declared]
+     - Destroys the ``cuda::pipeline_shared_state``.
+   * - ``operator=`` [deleted]
+     -  ``cuda::pipeline_shared_state`` is not assignable.
+
+.. rubric:: Constructor
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope, cuda::std::uint8_t StagesCount>
+   __host__ __device__
+   cuda::pipeline_shared_state();
+
+   template <cuda::thread_scope Scope, cuda::std::uint8_t StagesCount>
+   cuda::pipeline_shared_state(cuda::pipeline_shared_state const&) = delete;
+
+   template <cuda::thread_scope Scope, cuda::std::uint8_t StagesCount>
+   cuda::pipeline_shared_state(cuda::pipeline_shared_state&&) = delete;
+
+Construct a ``cuda::pipeline`` *shared state* object.
+
+.. code:: cuda
+
+   #include <cuda/pipeline>
+
+   #pragma nv_diag_suppress static_var_with_dynamic_init
+
+   __global__ void example_kernel() {
+     __shared__ cuda::pipeline_shared_state<cuda::thread_scope_block, 2> shared_state;
+   }
+
+`See it on Godbolt <https://godbolt.org/z/K4vKq4vd3>`__
+
+.. rubric:: NVCC ``__shared__`` Initialization Warnings
+
+When using libcu++ with NVCC, a ``__shared__`` ``cuda::pipeline_shared_state`` will lead to the following warning
+because ``__shared__`` variables are not initialized:
+
+.. code:: bash
+
+   warning: dynamic initialization is not supported for a function-scope static
+   __shared__ variable within a __device__/__global__ function
+
+It can be silenced using ``#pragma nv_diag_suppress static_var_with_dynamic_init``.
+
+.. rubric:: Example
+
+.. code:: cuda
+
+   #include <cuda/pipeline>
+
+   // Disables `pipeline_shared_state` initialization warning.
+   #pragma nv_diag_suppress static_var_with_dynamic_init
+
+   __global__ void example_kernel(char* device_buffer, char* sysmem_buffer) {
+     // Allocate a 2 stage block scoped shared state in shared memory.
+     __shared__ cuda::pipeline_shared_state<cuda::thread_scope_block, 2> pss0;
+
+     // Allocate a 2 stage block scoped shared state in device memory.
+     auto* pss1 = new cuda::pipeline_shared_state<cuda::thread_scope_block, 2>;
+
+     // Construct a 2 stage device scoped shared state in device memory.
+     auto* pss2 =
+       new (device_buffer) cuda::pipeline_shared_state<cuda::thread_scope_device, 2>;
+
+     // Construct a 2 stage system scoped shared state in system memory.
+     auto* pss3 =
+       new (sysmem_buffer) cuda::pipeline_shared_state<cuda::thread_scope_system, 2>;
+   }
+
+`See it on Godbolt <https://godbolt.org/z/M9ah7r1Yx>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/thread_groups.rst b/cccl_upstream/docs/libcudacxx/extended_api/thread_groups.rst
new file mode 100644
index 00000000..1fc4c957
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/thread_groups.rst
@@ -0,0 +1,64 @@
+.. _libcudacxx-extended-api-thread-groups:
+
+Thread Groups
+=============
+
+.. code:: cuda
+
+   struct ThreadGroup {
+     static constexpr cuda::thread_scope thread_scope;
+     Integral size() const;
+     Integral thread_rank() const;
+     void sync() const;
+   };
+
+The *ThreadGroup concept* defines the requirements of a type that represents a group of cooperating threads.
+
+The `CUDA Cooperative Groups Library <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#group-collectives>`_
+provides a number of types that satisfy this concept.
+
+Data Members
+------------
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``thread_scope``
+     - The scope at which ``ThreadGroup::sync()`` synchronizes memory operations and thread execution.
+
+Member Functions
+----------------
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``size``
+     - Returns the number of participating threads.
+   * - ``thread_rank``
+     - Returns a unique value for each participating thread (``0 <= ThreadGroup::thread_rank() < ThreadGroup::size()``).
+   * - ``sync``
+     - Synchronizes the participating threads.
+
+Notes
+-----
+
+This concept is defined for documentation purposes but is not materialized in the library.
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/atomic>
+   #include <cuda/std/cstddef>
+
+   struct single_thread_group {
+     static constexpr cuda::thread_scope thread_scope = cuda::thread_scope::thread_scope_thread;
+     cuda::std::size_t size() const { return 1; }
+     cuda::std::size_t thread_rank() const { return 0; }
+     void sync() const {}
+   };
+
+`See it on Godbolt <https://godbolt.org/z/6c16KxqY7>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/tma.rst b/cccl_upstream/docs/libcudacxx/extended_api/tma.rst
new file mode 100644
index 00000000..e62d50d9
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/tma.rst
@@ -0,0 +1,24 @@
+.. _libcudacxx-extended-api-tma:
+
+Tensor Memory Accelerator (TMA)
+===============================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   cuda::make_tma_descriptor <tma/make_tma_descriptor>
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`make_tma_descriptor() <libcudacxx-extended-api-tma-make_tma_descriptor>`
+     - Construct a TMA descriptor from a DLPack tensor
+     - CCCL 3.2.0
+     - CUDA 13.2
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/tma/make_tma_descriptor.rst b/cccl_upstream/docs/libcudacxx/extended_api/tma/make_tma_descriptor.rst
new file mode 100644
index 00000000..8656adf9
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/tma/make_tma_descriptor.rst
@@ -0,0 +1,228 @@
+.. _libcudacxx-extended-api-tma-make_tma_descriptor:
+
+``cuda::make_tma_descriptor``
+=============================
+
+Defined in the ``<cuda/tma>`` header.
+
+**Function signatures**
+
+.. code:: cuda
+
+    namespace cuda {
+
+    [[nodiscard]] inline
+    CUtensorMap make_tma_descriptor(
+      const DLTensor&            tensor,
+      cuda::std::span<const int> box_sizes,
+      cuda::std::span<const int> elem_strides,
+      tma_interleave_layout      interleave_layout = tma_interleave_layout::none,
+      tma_swizzle                swizzle           = tma_swizzle::none,
+      tma_l2_fetch_size          l2_fetch_size     = tma_l2_fetch_size::none,
+      tma_oob_fill               oobfill           = tma_oob_fill::none);
+
+    [[nodiscard]] inline
+    CUtensorMap make_tma_descriptor(
+        const DLTensor&            tensor,
+        cuda::std::span<const int> box_sizes,
+        tma_interleave_layout      interleave_layout = tma_interleave_layout::none,
+        tma_swizzle                swizzle           = tma_swizzle::none,
+        tma_l2_fetch_size          l2_fetch_size     = tma_l2_fetch_size::none,
+        tma_oob_fill               oobfill           = tma_oob_fill::none);
+
+    } // namespace cuda
+
+**Enumerators**
+
+.. code:: cuda
+
+    namespace cuda {
+
+    enum class tma_oob_fill { none, nan };
+
+    enum class tma_l2_fetch_size { none, bytes64, bytes128, bytes256 };
+
+    enum class tma_interleave_layout { none, bytes16, bytes32 };
+
+    enum class tma_swizzle {
+        none,
+        bytes32,
+        bytes64,
+        bytes128,
+        bytes128_atom_32B,        // only CUDA Toolkit 12.8 and later, compute capability >= 10
+        bytes128_atom_32B_flip_8B,// only CUDA Toolkit 12.8 and later, compute capability >= 10
+        bytes128_atom_64B         // only CUDA Toolkit 12.8 and later, compute capability >= 10
+    };
+
+    } // namespace cuda
+
+The functions construct a `CUDA Tensor Memory Accelerator (TMA) descriptor <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#using-tma-to-transfer-multi-dimensional-arrays>`__ from a ``DLTensor``. The resulting ``CUtensorMap`` can be bound to TMA-based copy instructions to efficiently stage multi-dimensional tiles in shared memory on Compute Capability 9.0 and newer GPUs.
+
+
+.. note::
+
+  - **DLPack** assumes *row-major* convention for sizes and strides, where the fastest changing dimension is the last one (``rank - 1``).
+  - **cuTensorMap** assumes *column-major* convention for sizes and strides, where the fastest changing dimension is the first one (``0``).
+  - ``box_sizes`` and ``elem_strides`` are expected to be in the same order as the input tensor's dimensions provided by **DLPack**, namely *row-major*.
+
+Parameters
+----------
+
+- ``tensor``: The DLPack tensor describing the logical layout in device memory.
+- ``box_sizes``: Extent of the shared memory tile, one entry per tensor dimension.
+- ``elem_strides``: Stride, in elements, between consecutive accesses inside the shared memory tile. The second overload assumes a stride of ``1`` for every dimension with the special meaning of contiguous memory.
+
+*Optional parameters*:
+
+- ``interleave_layout``: Interleaving applied to the underlying memory.
+- ``swizzle``: Swizzle pattern matching the chosen interleave layout.
+- ``l2_fetch_size``: L2 cache promotion for TMA transfers.
+- ``oobfill``: Out-of-bounds fill policy for floating-point tensors.
+
+Return value
+------------
+
+- ``CUtensorMap`` encoding all metadata required to launch TMA transfers.
+
+Preconditions
+-------------
+
+See :ref:`libcudacxx-extended-api-exceptions` for more details on exception handling.
+
+**General preconditions**:
+
+* Compute Capability 9.0 or newer is required.
+* ``dlpack/dlpack.h`` (`DLPack v1 <https://github.com/dmlc/dlpack>`__) must be discoverable at compile time, namely available in the include path.
+
+**DLPack preconditions**:
+
+``tensor.device.device_type``:
+
+* Must be ``kDLCUDA`` or ``kDLCUDAManaged``.
+
+``tensor.device.device_id``:
+
+* Must be a valid GPU device ordinal
+* The selected device must have Compute Capability 9.0 or newer.
+
+``tensor.ndim`` (rank):
+
+* Must be greater than 0 and less than or equal to 5.
+* Must be greater than or equal to ``3`` when an interleaved layout is requested.
+
+``tensor.dtype``:
+
+* ``kDLUInt``:
+
+  - ``bits == 4``, ``lanes == 16``, namely ``U4 x 16``. Additionally, the innermost dimension must be a multiple of ``2`` when only 16-byte alignment is available. Requires CUDA Toolkit 12.8 and later, and compute capability >= 10.
+  - ``bits == 8``, ``lanes == 1``, namely ``uint8_t``.
+  - ``bits == 16``, ``lanes == 1``, namely ``uint16_t``.
+  - ``bits == 32``, ``lanes == 1``, namely ``uint32_t``.
+  - ``bits == 64``, ``lanes == 1``, namely ``uint64_t``.
+
+* ``kDLInt``
+
+  - ``bits == 32``, ``lanes == 1``, namely ``int32_t``.
+  - ``bits == 64``, ``lanes == 1``, namely ``int64_t``.
+
+* ``kDLFloat``
+
+  - ``bits == 16``, ``lanes == 1``, namely ``__half``.
+  - ``bits == 32``, ``lanes == 1``, namely ``float``.
+  - ``bits == 64``, ``lanes == 1``, namely ``double``.
+
+* ``kDLBfloat``
+
+  - ``bits == 16``, ``lanes == 1``, namely ``__nv_bfloat16``.
+
+* ``kDLFloat4_e2m1fn``
+
+  - ``bits == 4``, ``lanes == 16``, mapped to ``U4 x 16``.  See ``kDLUInt`` for additional requirements.
+
+* ``kDLBool``, ``kDLFloat8_e3m4``, ``kDLFloat8_e4m3``, ``kDLFloat8_e4m3b11fnuz``, ``kDLFloat8_e4m3fn``, ``kDLFloat8_e4m3fnuz``, ``kDLFloat8_e5m2``, ``kDLFloat8_e5m2fnuz``, ``kDLFloat8_e8m0fnu``: mapped to ``uint8_t``.
+
+``tensor.data`` (pointer):
+
+* Must be a valid GPU global address.
+* Must be aligned to at least 16 bytes. Must be aligned to 32 bytes when ``interleave_layout`` is ``bytes32``.
+
+``tensor.shape``:
+
+* Must be greater than 0 and not exceed ``2^32`` elements per dimension.
+* The innermost dimension must be a multiple of ``2`` when ``kDLFloat4_e2m1fn`` or ``U4 x 16`` are used.
+
+``tensor.strides``:
+
+* Each stride in bytes, namely ``tensor.strides[i] * element_size``, must be greater than 0 and not exceed ``2^40`` bytes per dimension.
+* The tensor mapping must be unique, namely ``tensor.strides[i]`` must be greater than or equal to ``tensor.shape[i - 1] * strides[i - 1]`` or equal to ``0``.
+* Each stride in bytes must be a multiple of the alignment 16 bytes when ``interleave_layout`` is ``none`` or ``bytes16``. It must be a multiple of 32 bytes when ``interleave_layout`` is ``bytes32``.
+* ``tensor.strides`` can be ``nullptr`` to indicate that the tensor is contiguous in memory.
+
+**User parameter preconditions**:
+
+``box_sizes``, ``elem_strides``, and ``tensor.ndim`` must have the same rank.
+
+``box_sizes``:
+
+* Must be positive and not exceed ``256`` elements per dimension.
+* ``box_sizes[i]`` must be less than or equal to ``tensor.shape[i]``.
+* The full size of ``box_sizes`` must fit in shared memory.
+* If the ``interleave_layout`` is ``tma_interleave_layout::none``, the inner dimension in bytes, computed as ``box_sizes[rank - 1] * element_size`` has the following additional requirements:
+
+  - It must be a multiple of 16 bytes.
+  - It must not exceed the byte-width of the selected ``swizzle`` pattern (``32``, ``64``, or ``128`` bytes).
+
+``elem_strides``:
+
+* Must be positive and not exceed ``8`` elements per dimension.
+* ``elem_strides[i]`` must be less than or equal to ``tensor.shape[i]``.
+*  If the ``interleave_layout`` is ``tma_interleave_layout::none``, the innner dimension (``elem_strides[0]``) is ignored.
+
+``oobfill``:
+
+* Must be ``tma_oob_fill::none`` for all integer data types.
+
+``interleave_layout``:
+
+* If ``interleave_layout`` is ``tma_interleave_layout::bytes32``, ``swizzle`` must be ``tma_swizzle::bytes32``.
+
+References
+----------
+
+- `DLPack C API <https://dmlc.github.io/dlpack/latest/c_api.html>`__ documentation.
+- `CUDA Tensor Memory Accelerator (TMA) <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#using-tma-to-transfer-multi-dimensional-arrays>`__ documentation.
+- ``cuTensorMapEncodeTiled()`` `CUDA driver API <https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__TENSOR__MEMORY.html#group__CUDA__TENSOR__MEMORY_1ga7c7d2aaac9e49294304e755e6f341d7>`__ documentation.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/tma>
+    #include <cuda/std/cstdint>
+    #include <dlpack/dlpack.h>
+
+    CUtensorMap create_2d_tile_descriptor(float* device_ptr) {
+        // Define DLPack tensor descriptor, commonly provided externally by the user, library, or framework.
+        constexpr int64_t shape_storage[2]   = {64, 64};
+        constexpr int64_t strides_storage[2] = {64, 1};
+
+        DLTensor tensor{};
+        tensor.data        = device_ptr;
+        tensor.device      = {kDLCUDA, 0};
+        tensor.ndim        = 2;
+        tensor.dtype.code  = static_cast<uint8_t>(kDLFloat);
+        tensor.dtype.bits  = 32;
+        tensor.dtype.lanes = 1;
+        tensor.shape       = const_cast<int64_t*>(shape_storage);
+        tensor.strides     = const_cast<int64_t*>(strides_storage);
+        tensor.byte_offset = 0;
+
+        // Define shared memory box sizes and element strides.
+        constexpr int BoxSizeX      = 8; // rows
+        constexpr int BoxSizeY      = 8; // columns
+        int box_sizes_storage[2]    = {BoxSizeX, BoxSizeY};
+        int elem_strides_storage[2] = {BoxSizeY, 1}; // {1, ..., 1} is also valid to specify contiguous memory
+
+        return cuda::make_tma_descriptor(tensor, box_sizes_storage, elem_strides_storage);
+    }
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/type_traits.rst b/cccl_upstream/docs/libcudacxx/extended_api/type_traits.rst
new file mode 100644
index 00000000..9f55314a
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/type_traits.rst
@@ -0,0 +1,42 @@
+.. _libcudacxx-extended-api-type_traits:
+
+Type traits
+-----------
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   type_traits/is_floating_point
+   type_traits/is_trivially_copyable
+   type_traits/is_bitwise_comparable
+   type_traits/vector_types
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`cuda::is_floating_point <libcudacxx-extended-api-type_traits-is_floating_point>`
+     - Tells whether a type is a floating point type
+     - CCCL 3.0.0
+     - CUDA 13.0
+
+   * - :ref:`Vector Type Traits <libcudacxx-extended-api-type_traits-vector_types>`
+     - Type traits for CUDA vector types
+     - CCCL 3.3.0
+     - CUDA 13.3
+
+   * - :ref:`cuda::is_trivially_copyable <libcudacxx-extended-api-type_traits-is_trivially_copyable>`
+     - Relaxed trivially copyable check including extended floating-point types
+     - CCCL 3.4.0
+     - CUDA 13.4
+
+   * - :ref:`cuda::is_bitwise_comparable <libcudacxx-extended-api-type_traits-is_bitwise_comparable>`
+     - User-specializable bitwise comparability check
+     - CCCL 3.4.0
+     - CUDA 13.4
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/type_traits/is_bitwise_comparable.rst b/cccl_upstream/docs/libcudacxx/extended_api/type_traits/is_bitwise_comparable.rst
new file mode 100644
index 00000000..83e0293b
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/type_traits/is_bitwise_comparable.rst
@@ -0,0 +1,82 @@
+.. _libcudacxx-extended-api-type_traits-is_bitwise_comparable:
+
+``cuda::is_bitwise_comparable``
+===============================
+
+Defined in the ``<cuda/type_traits>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T>
+   constexpr bool is_bitwise_comparable_v = /* see below */;
+
+   template <typename T>
+   using is_bitwise_comparable = cuda::std::bool_constant<is_bitwise_comparable_v<T>>;
+
+   } // namespace cuda
+
+``cuda::is_bitwise_comparable_v`` trait evaluates if a type is comparable at bit-level, meaning that two instances of the same type can be compared as a raw sequence of bytes, independently from their semantics.
+
+.. note::
+
+  The following conditions generally prevent a type from being bitwise comparable:
+
+  - The type is a floating-point type. ``NaN`` and ``+/-0`` values are not bitwise comparable.
+  - The type has internal padding, for example a structure with ``char`` and ``int`` members.
+  - The type has special comparison semantics, such as a user-defined ``operator==``.
+
+``cuda::is_bitwise_comparable_v<T>`` is a user-specializable variable template that relies on ``cuda::std::has_unique_object_representations`` but excludes extended floating-point scalar and vector types.
+
+The trait also propagates through composite types:
+
+- C-style arrays: ``T[N]`` and ``T[]`` are bitwise comparable when ``T`` is.
+- ``cuda::std::array<T, N>``: bitwise comparable when ``T`` is.
+- ``cuda::std::pair<T1, T2>``: bitwise comparable when both ``T1`` and ``T2`` are and the object has no padding.
+- ``cuda::std::tuple<Ts...>``: bitwise comparable when all ``Ts...`` are and the object has no padding.
+- ``cuda::complex<T>``, ``cuda::std::complex<T>``: not bitwise comparable.
+- Aggregates (types that can be initialized with a braced initializer list ``{}``): bitwise comparable when all their members are.
+
+  - On MSVC, recursive data-member inspection is not supported beyond the first level.
+
+``const``, ``volatile``, and ``const volatile`` qualifications are handled transparently.
+
+Custom Specialization
+---------------------
+
+Users may specialize ``cuda::is_bitwise_comparable_v`` for their own types to indicate that two object representations can be compared bitwise, even when the implementation cannot determine this automatically.
+The specialization must be provided for the unqualified type; cv-qualified forms are handled automatically.
+
+.. code:: cuda
+
+  struct MyType {
+    double value;
+  };
+
+  template <>
+  constexpr bool cuda::is_bitwise_comparable_v<MyType> = true;
+
+  static_assert(cuda::is_bitwise_comparable_v<MyType>);
+  static_assert(cuda::is_bitwise_comparable_v<const MyType>);
+
+.. warning::
+
+  Users are responsible for ensuring that the type is actually bitwise comparable when specializing this variable template. Otherwise, the behavior of functions that rely on this trait is undefined. For the reasons described above, bitwise comparison is especially problematic for floating-point types and types with internal padding.
+
+Examples
+--------
+
+.. code:: cuda
+
+   #include <cuda/type_traits>
+
+   // Integer types have unique object representations
+   static_assert(cuda::is_bitwise_comparable_v<int>);
+   static_assert(cuda::is_bitwise_comparable_v<unsigned>);
+   static_assert(cuda::is_bitwise_comparable_v<char>);
+   static_assert(cuda::is_bitwise_comparable_v<int[4]>);
+
+   // Floating-point types do not
+   static_assert(!cuda::is_bitwise_comparable_v<float>);
+   static_assert(!cuda::is_bitwise_comparable_v<double>);
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/type_traits/is_floating_point.rst b/cccl_upstream/docs/libcudacxx/extended_api/type_traits/is_floating_point.rst
new file mode 100644
index 00000000..45ac98ec
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/type_traits/is_floating_point.rst
@@ -0,0 +1,19 @@
+.. _libcudacxx-extended-api-type_traits-is_floating_point:
+
+``cuda::is_floating_point``
+===========================
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <class T>
+   inline constexpr bool is_floating_point_v = __ implementation defined __;
+
+   template <class T>
+   using is_floating_point = cuda::std::bool_constant<is_floating_point_v<T>>;
+
+   } // namespace cuda
+
+Tells whether a type is a floating point type, including implementation defined extended floating point types.
+Users are allowed to specialize the variable template for their own types, but CCCL does not provide support for any issues arising from that.
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/type_traits/is_trivially_copyable.rst b/cccl_upstream/docs/libcudacxx/extended_api/type_traits/is_trivially_copyable.rst
new file mode 100644
index 00000000..6617de3c
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/type_traits/is_trivially_copyable.rst
@@ -0,0 +1,73 @@
+.. _libcudacxx-extended-api-type_traits-is_trivially_copyable:
+
+``cuda::is_trivially_copyable``
+=======================================
+
+Defined in the ``<cuda/type_traits>`` header.
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <typename T>
+   constexpr bool is_trivially_copyable_v = /* see below */;
+
+   template <typename T>
+   using is_trivially_copyable = cuda::std::bool_constant<cuda::std::is_trivially_copyable_v<T>>;
+
+   } // namespace cuda
+
+``cuda::is_trivially_copyable_v`` trait evaluates if a type can be copied by copying its underlying bytes.
+It extends ``cuda::std::is_trivially_copyable`` to also recognize CUDA extended floating-point vector types as trivially copyable.
+
+The trait is true when ``T`` is any of the following:
+
+- A type for which ``cuda::std::is_trivially_copyable_v<T>`` is true.
+- An extended floating-point vector type, for example ``__half2``, ``__nv_bfloat162``.
+
+The trait also propagates through composite types:
+
+- C-style arrays: ``T[N]`` and ``T[]`` are trivially copyable when ``T`` is.
+- ``cuda::std::array<T, N>``: trivially copyable when ``T`` is.
+- ``cuda::std::pair<T1, T2>``: trivially copyable when both ``T1`` and ``T2`` are.
+- ``cuda::std::tuple<Ts...>``: trivially copyable when all ``Ts...`` are.
+- ``cuda::std::complex<T>``: trivially copyable when ``T`` is.
+- ``cuda::complex<T>``: trivially copyable when ``T`` is.
+- `Aggregates <https://en.cppreference.com/cpp/language/aggregate_initialization>`__: trivially copyable when all their members are.
+
+  - On MSVC, recursive data-member inspection is not supported beyond the first level.
+
+``const`` qualification is handled transparently, while ``volatile`` is compiler dependent.
+
+Examples
+--------
+
+.. code:: cuda
+
+   #include <cuda/type_traits>
+   #include <cuda/std/array>
+   #include <cuda/std/tuple>
+   #include <cuda/std/utility>
+
+   #include <cuda_fp16.h>
+
+   // Standard trivially copyable types
+   static_assert(cuda::is_trivially_copyable_v<int>);
+   static_assert(cuda::is_trivially_copyable_v<float>);
+
+   // Extended floating-point types
+   static_assert(cuda::is_trivially_copyable_v<__half>);
+   static_assert(cuda::is_trivially_copyable_v<__nv_bfloat16>);
+   static_assert(cuda::is_trivially_copyable_v<__half2>);
+   static_assert(cuda::is_trivially_copyable_v<cuda::std::complex<__half>>);
+   static_assert(cuda::is_trivially_copyable_v<cuda::complex<__half>>);
+
+   // Composite types containing extended floating-point types
+   static_assert(cuda::is_trivially_copyable_v<__half[4]>);
+   static_assert(cuda::is_trivially_copyable_v<cuda::std::array<__half2, 4>>);
+   static_assert(cuda::is_trivially_copyable_v<cuda::std::pair<__half2, __half>>);
+   static_assert(cuda::is_trivially_copyable_v<cuda::std::tuple<__half, __half2>>);
+   static_assert(cuda::is_trivially_copyable_v<cuda::std::pair<__half2, int>>);
+
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/PqccjfEv6>`__
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/type_traits/vector_types.rst b/cccl_upstream/docs/libcudacxx/extended_api/type_traits/vector_types.rst
new file mode 100644
index 00000000..9d85c438
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/type_traits/vector_types.rst
@@ -0,0 +1,135 @@
+.. _libcudacxx-extended-api-type_traits-vector_types:
+
+Vector Type Traits
+==================
+
+Defined in the ``<cuda/type_traits>`` header.
+
+``cuda::is_vector_type_v``
+--------------------------
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <class T>
+   inline constexpr bool is_vector_type_v = /* see below */;
+
+   template <class T>
+   using is_vector_type = cuda::std::bool_constant<is_vector_type_v<T>>;
+
+   } // namespace cuda
+
+``cuda::is_vector_type_v<T>`` is ``true`` if ``T`` is a cv-qualified CUDA vector type, such as ``int2``, ``float4``, or ``dim3``.
+
+**Supported vector types**
+
+- ``char1``, ``char2``, ``char3``, ``char4``.
+- ``uchar1``, ``uchar2``, ``uchar3``, ``uchar4``.
+- ``short1``, ``short2``, ``short3``, ``short4``.
+- ``ushort1``, ``ushort2``, ``ushort3``, ``ushort4``.
+- ``int1``, ``int2``, ``int3``, ``int4``.
+- ``uint1``, ``uint2``, ``uint3``, ``uint4``.
+- ``long1``, ``long2``, ``long3``, ``long4``, and alignment variants in CUDA Toolkit 13.0+.
+- ``ulong1``, ``ulong2``, ``ulong3``, ``ulong4``, and alignment variants in CUDA Toolkit 13.0+.
+- ``longlong1``, ``longlong2``, ``longlong3``, ``longlong4``, and alignment variants in CUDA Toolkit 13.0+.
+- ``ulonglong1``, ``ulonglong2``, ``ulonglong3``, ``ulonglong4``, and alignment variants in CUDA Toolkit 13.0+.
+- ``float1``, ``float2``, ``float3``, ``float4``.
+- ``double1``, ``double2``, ``double3``, ``double4``, and alignment variants in CUDA Toolkit 13.0+.
+- ``dim3``.
+
+``cuda::is_extended_fp_vector_type_v``
+--------------------------------------
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <class T>
+   inline constexpr bool is_extended_fp_vector_type_v = /* see below */;
+
+   template <class T>
+   using is_extended_fp_vector_type = cuda::std::bool_constant<is_extended_fp_vector_type_v<T>>;
+
+   } // namespace cuda
+
+``cuda::is_extended_fp_vector_type_v<T>`` is ``true`` if ``T`` is a cv-qualified extended floating-point vector type.
+
+**Supported extended floating-point vector types**
+
+- ``__half2``.
+- ``__nv_bfloat162``.
+- ``__nv_fp8x2_e4m3``, ``__nv_fp8x2_e5m2``, ``__nv_fp8x4_e4m3``, ``__nv_fp8x4_e5m2``.
+- ``__nv_fp8x2_e8m0``, ``__nv_fp8x4_e8m0``.
+- ``__nv_fp6x2_e2m3``, ``__nv_fp6x2_e3m2``, ``__nv_fp6x4_e2m3``, ``__nv_fp6x4_e3m2``.
+- ``__nv_fp4x2_e2m1``, ``__nv_fp4x4_e2m1``.
+
+``cuda::vector_type``
+-----------------------
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <class T, cuda::std::size_t Size>
+   struct vector_type
+   {
+     using type = /* see below */;
+   };
+
+   template <class T, cuda::std::size_t Size>
+   using vector_type_t = typename vector_type<T, Size>::type;
+
+   } // namespace cuda
+
+``cuda::vector_type<T, Size>::type`` is a type alias that maps a scalar type ``T`` and a vector size ``Size`` to the corresponding CUDA vector type. If no valid vector type exists for the given combination, the type is ``void``. It supports both integral and floating-point vector types.
+
+``cuda::scalar_type``
+-----------------------
+
+.. code:: cuda
+
+   namespace cuda {
+
+   template <class T>
+   struct scalar_type
+   {
+     using type = /* see below */;
+   };
+
+   template <class T>
+   using scalar_type_t = typename scalar_type<T>::type;
+
+   } // namespace cuda
+
+``cuda::scalar_type<T>::type`` extracts the scalar element type from a cv-qualified CUDA vector type. For example, ``cuda::scalar_type<int2>::type`` is ``int`` and ``cuda::scalar_type<float4>::type`` is ``float``. It supports both integral and floating-point vector types. If ``T`` is not a vector type, the ``type`` type alias is not defined.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/type_traits>
+
+    // Check if a type is a vector type
+    static_assert(cuda::is_vector_type_v<int2>);
+    static_assert(cuda::is_vector_type_v<float4>);
+    static_assert(cuda::is_vector_type_v<dim3>);
+    static_assert(!cuda::is_vector_type_v<int>);
+
+    // Get the vector type for a scalar type and size
+    static_assert(cuda::std::is_same_v<cuda::vector_type_t<int, 2>, int2>);
+    static_assert(cuda::std::is_same_v<cuda::vector_type_t<float, 4>, float4>);
+
+    // Extract the scalar type from a vector type
+    static_assert(cuda::std::is_same_v<cuda::scalar_type_t<int2>, int>);
+    static_assert(cuda::std::is_same_v<cuda::scalar_type_t<float4>, float>);
+
+    // Extended floating-point vector types
+    static_assert(cuda::is_extended_fp_vector_type_v<__half2>);
+    static_assert(cuda::std::is_same_v<cuda::vector_type_t<__half, 2>, __half2>);
+
+See Also
+--------
+
+- :ref:`Tuple Protocol for Vector Types <libcudacxx-extended-api-vector-tuple-protocol>` - Tuple protocol support for CUDA vector types.
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/utility.rst b/cccl_upstream/docs/libcudacxx/extended_api/utility.rst
new file mode 100644
index 00000000..09fd875a
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/utility.rst
@@ -0,0 +1,30 @@
+.. _libcudacxx-extended-api-utility:
+
+Utility
+========
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   cuda::in_range <utility/in_range>
+   cuda::static_for <utility/static_for>
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`in_range <libcudacxx-extended-api-utility-in-range>`
+     - Check if a value is within a range
+     - CCCL 3.2.0
+     - CUDA 13.2
+
+   * - :ref:`static_for <libcudacxx-extended-api-utility-static-for>`
+     - Compile-time ``for`` loop
+     - CCCL 3.1.0
+     - CUDA 13.1
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/utility/in_range.rst b/cccl_upstream/docs/libcudacxx/extended_api/utility/in_range.rst
new file mode 100644
index 00000000..8af9d152
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/utility/in_range.rst
@@ -0,0 +1,77 @@
+.. _libcudacxx-extended-api-utility-in-range:
+
+``in_range``
+============
+
+Defined in the ``<cuda/utility>`` header.
+
+.. code:: cuda
+
+    namespace cuda {
+
+    template <typename T>
+    [[nodiscard]] __host__ __device__ constexpr
+    bool in_range(T value, T start, T end) noexcept;
+
+    } // namespace cuda
+
+Checks whether a value ``value`` is within the range ``[start, end]``, inclusive.
+
+**Template Parameters**
+
+- ``T``: the type of the value and the range bounds.
+
+**Parameters**
+
+- ``value``: the value to check if it's within the range.
+- ``start``: the lower bound of the range.
+- ``end``: the upper bound of the range.
+
+**Return Value**
+
+- ``true`` if ``value`` is in the range ``[start, end]``, ``false`` otherwise.
+
+**Constraints**
+
+- ``T`` must be an integer or floating point type.
+
+**Preconditions**
+
+- ``end`` must be greater than or equal to ``start``.
+
+.. note::
+
+  For floating-point types, ``NaN`` is considered a valid value. The function returns ``false`` if any input is ``NaN``.
+
+**Performance considerations**
+
+- The function is optimized when ``value`` is an unsigned integer type. The optimization is useful when ``start`` and ``end`` are known at compile-time, or when ``in_range`` is used multiple times with the same range.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/utility>
+    #include <cuda/std/cassert>
+
+    __global__ void in_range_kernel() {
+        // unsigned integers
+        assert(cuda::in_range(5u, 1u, 10u));   // 5  is in the range     [1, 10]
+        assert(!cuda::in_range(15u, 1u, 10u)); // 15 is NOT in the range [1, 10]
+        assert(cuda::in_range(1u, 1u, 10u));   // 1  is in the range     [1, 10]
+        assert(cuda::in_range(10u, 1u, 10u));  // 10 is in the range     [1, 10]
+        assert(cuda::in_range(10u, 10u, 10u)); // 10 is in the range     [10, 10]
+
+        // signed integers
+        assert(cuda::in_range(-5, -10, 0));    // -5 is in the range    [-10, 0]
+        assert(!cuda::in_range(5, -10, 0));    // 5 is NOT in the range [-10, 0]
+    }
+
+    int main() {
+        in_range_kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See the example in Compiler Explorer 🔗 <https://godbolt.org/z/WjG4rWT3G>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/utility/static_for.rst b/cccl_upstream/docs/libcudacxx/extended_api/utility/static_for.rst
new file mode 100644
index 00000000..1a14fe0e
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/utility/static_for.rst
@@ -0,0 +1,92 @@
+.. _libcudacxx-extended-api-utility-static-for:
+
+``static_for``
+==============
+
+Defined in ``<cuda/utility>`` header.
+
+.. code:: cuda
+
+    namespace cuda {
+
+    template <auto Size, typename Operator, typename... TArgs>
+    __host__ __device__ constexpr
+    void static_for(Operator op, TArgs&&... args) noexcept(/*see-below*/); // (1)
+
+    template <auto Start, decltype(Start) End, decltype(Start) Step = 1, typename Operator, typename... TArgs>
+    __host__ __device__ constexpr
+    void static_for(Operator op, TArgs&&... args) noexcept(/*see-below*/); // (2)
+
+    template <typename T, T Size, typename Operator, typename... TArgs>
+    __host__ __device__ constexpr
+    void static_for(Operator op, TArgs&&... args) noexcept(/*see-below*/); // (3)
+
+    template <typename T, T Start, T End, T Step = 1, typename Operator, typename... TArgs>
+    __host__ __device__ constexpr
+    void static_for(Operator op, TArgs&&... args) noexcept(/*see-below*/); // (4)
+
+    } // namespace cuda
+
+| The functionality provides a ``for`` loop with compile-time indices.
+| ``static_for`` is available in two forms:
+
+- Executes ``op`` for each value in the range ``[0, Size)`` (1, 3).
+- Executes ``op`` for each value in the range ``[Start, End)`` with step ``Step`` (2, 4).
+
+| The function is ``noexcept`` if all invocations of ``op`` with ``integral_constant</*index-type*/, /*index-value*/>`` and the ``args...`` are *non-throwing*. Only visited indices participate in the ``noexcept`` evaluation.
+
+**Parameters**
+
+- ``Size``: the number of iterations (1, 3).
+- ``Start``, ``End``, ``Step``: the start, end, and step of the range. Note that ``End`` and ``Step`` are converted to the type of ``Start`` (2, 4).
+- ``T``: type of the loop index (3, 4).
+- ``op``: the function to execute.
+- ``args``: additional arguments to pass to ``op``.
+
+``op`` is a callable object that accepts an ``integral_constant`` of the same type of ``Size`` or ``Start``.
+
+**Performance considerations**
+
+- The functions are useful as metaprogramming utility and when a loop requires full unrolling, independently of the compiler's constrains, optimization level, and heuristics. In addition, the index is a compile-time constant, which can be used in a constant expression and further optimize the code.
+
+- Conversely, ``static_for`` is more expensive to compile compared to ``#pragma unroll``. Additionally, the preprocessor directive interacts with the compiler, which tunes the loop unrolling based on register usage, binary size, and instruction cache.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/utility>
+    #include <cstdio>
+
+    __global__ void kernel() {
+        cuda::static_for<5>([](auto i){ static_assert(i >= 0 && i < 5); });
+
+        cuda::static_for<5>([](auto i){ printf("%d, ", i()); }); // 0, 1, 2, 3, 4,
+        printf("\n");
+
+        cuda::static_for<short, 5>([](auto i){ printf("%d, ", i()); }); // 0, 1, 2, 3, 4,
+        printf("\n");
+
+        cuda::static_for<-5, 7, 3>([](auto i){ printf("%d, ", i()); }); // -5, -2, 1, 4,
+        printf("\n");
+
+        cuda::static_for<5>([](auto i){
+            if constexpr (i > 0) {
+                cuda::static_for<i()>([](auto j){ printf("%d, ", j()); });
+                printf("\n");
+            }
+        });
+        // 0,
+        // 0, 1,
+        // 0, 1, 2,
+        // 0, 1, 2, 3,
+        cuda::static_for<5>([](auto i, int a, int b, int c){}, 1, 2, 3); // 1, 2, 3 optional arguments
+    }
+
+    int main() {
+        kernel<<<1, 1>>>();
+        cudaDeviceSynchronize();
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/1GWc4dqKj>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/vector_tuple_protocol.rst b/cccl_upstream/docs/libcudacxx/extended_api/vector_tuple_protocol.rst
new file mode 100644
index 00000000..2671f50b
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/vector_tuple_protocol.rst
@@ -0,0 +1,111 @@
+.. _libcudacxx-extended-api-vector-tuple-protocol:
+
+Vector Tuple Protocol
+=====================
+
+Defined in the ``<cuda/std/tuple>`` header.
+
+``libcu++`` provides tuple protocol support for CUDA vector types, enabling structured bindings and tuple-like access to vector type elements.
+
+Please refer to the documentation of the C++ standard header `\<tuple\> <https://en.cppreference.com/w/cpp/header/tuple>`_ for more information.
+
+- ``cuda::std::tuple_size<VectorType>`` provides the number of elements in a CUDA vector type as a compile-time constant.
+- ``cuda::std::tuple_element<I, VectorType>`` provides the scalar element type of a CUDA vector type at index ``I``.
+- ``cuda::std::get<I>(v)`` returns a reference to the element at index ``I`` of the vector type ``v``.
+
+  - For ``I == 0``: reference to ``v.x``
+  - For ``I == 1``: reference to ``v.y``
+  - For ``I == 2``: reference to ``v.z``
+  - For ``I == 3``: reference to ``v.w``
+
+Structured Bindings
+-------------------
+
+CUDA vector types support C++17 structured bindings through the tuple protocol.
+
+.. code:: cuda
+
+   int2 vec{1, 2};
+   auto [x, y] = vec;  // x == 1, y == 2
+
+   float4 vec4{1.0f, 2.0f, 3.0f, 4.0f};
+   auto& [a, b, c, d] = vec4;  // references to vec4.x, vec4.y, vec4.z, vec4.w
+
+Supported Vector Types
+----------------------
+
+The tuple protocol is supported for all CUDA vector types:
+
+**Integral vector types**
+
+- ``char1``, ``char2``, ``char3``, ``char4``.
+- ``uchar1``, ``uchar2``, ``uchar3``, ``uchar4``.
+- ``short1``, ``short2``, ``short3``, ``short4``.
+- ``ushort1``, ``ushort2``, ``ushort3``, ``ushort4``.
+- ``int1``, ``int2``, ``int3``, ``int4``.
+- ``uint1``, ``uint2``, ``uint3``, ``uint4``.
+- ``long1``, ``long2``, ``long3``, ``long4``, and alignment variants in CUDA Toolkit 13.0+.
+- ``ulong1``, ``ulong2``, ``ulong3``, ``ulong4``, and alignment variants in CUDA Toolkit 13.0+.
+- ``longlong1``, ``longlong2``, ``longlong3``, ``longlong4``, and alignment variants in CUDA Toolkit 13.0+.
+- ``ulonglong1``, ``ulonglong2``, ``ulonglong3``, ``ulonglong4``, and alignment variants in CUDA Toolkit 13.0+.
+
+**Floating-point vector types**
+
+- ``float1``, ``float2``, ``float3``, ``float4``.
+- ``double1``, ``double2``, ``double3``, ``double4``, and alignment variants in CUDA Toolkit 13.0+.
+
+**Special types**
+
+- ``dim3``.
+
+**Extended floating-point vector types**
+
+- ``__half2``.
+- ``__nv_bfloat162``.
+- ``__nv_fp8x2_e4m3``, ``__nv_fp8x2_e5m2``, ``__nv_fp8x4_e4m3``, ``__nv_fp8x4_e5m2`` **\***.
+- ``__nv_fp8x2_e8m0``, ``__nv_fp8x4_e8m0``  **\***.
+- ``__nv_fp6x2_e2m3``, ``__nv_fp6x2_e3m2``, ``__nv_fp6x4_e2m3``, ``__nv_fp6x4_e3m2``  **\***.
+- ``__nv_fp4x2_e2m1``, ``__nv_fp4x4_e2m1``  **\***.
+
+**\*** Single-component vector types.
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/std/tuple>
+
+   __host__ __device__ void example() {
+       // tuple_size: get the number of elements
+       static_assert(cuda::std::tuple_size_v<int2>   == 2);
+       static_assert(cuda::std::tuple_size_v<float4> == 4);
+       static_assert(cuda::std::tuple_size_v<dim3>   == 3);
+
+       // tuple_element: get the element type
+       static_assert(cuda::std::is_same_v<cuda::std::tuple_element_t<0, int2>, int>);
+       static_assert(cuda::std::is_same_v<cuda::std::tuple_element_t<1, float4>, float>);
+       static_assert(cuda::std::is_same_v<cuda::std::tuple_element_t<2, dim3>, unsigned int>);
+
+       // get<>: access elements by index
+       int2 vec2{10, 20};
+       int x = cuda::std::get<0>(vec2);  // x == 10
+       int y = cuda::std::get<1>(vec2);  // y == 20
+
+       // Modify elements through get<>
+       cuda::std::get<0>(vec2) = 100;    // vec2.x == 100
+
+       // Structured bindings (C++17)
+       float3 color{0.5f, 0.8f, 1.0f};
+       auto [r, g, b] = color;           // r == 0.5f, g == 0.8f, b == 1.0f
+
+       // Reference bindings for modification
+       int4 data{1, 2, 3, 4};
+       auto& [a, b2, c, d] = data;
+       a = 10;                           // data.x == 10
+   }
+
+See Also
+--------
+
+- :ref:`Type Traits for CUDA Vector Types <libcudacxx-extended-api-type_traits-vector_types>` - Extract scalar type from vector types.
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/warp.rst b/cccl_upstream/docs/libcudacxx/extended_api/warp.rst
new file mode 100644
index 00000000..300aca82
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/warp.rst
@@ -0,0 +1,57 @@
+.. _libcudacxx-extended-api-warp:
+
+Warp
+====
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   warp/warp_shuffle
+   warp/warp_match_all
+   warp/warp_match_any
+   warp/lane_mask
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`warp_shuffle_idx <libcudacxx-extended-api-warp-warp-shuffle>`
+     - Warp shuffle from a specific lane
+     - CCCL 3.0.0
+     - CUDA 13.0
+
+   * - :ref:`warp_shuffle_up <libcudacxx-extended-api-warp-warp-shuffle>`
+     - Warp shuffle from original lane index - delta
+     - CCCL 3.0.0
+     - CUDA 13.0
+
+   * - :ref:`warp_shuffle_down <libcudacxx-extended-api-warp-warp-shuffle>`
+     - Warp shuffle from original lane index + delta
+     - CCCL 3.0.0
+     - CUDA 13.0
+
+   * - :ref:`warp_shuffle_xor <libcudacxx-extended-api-warp-warp-shuffle>`
+     - Warp shuffle from original lane index xor mask
+     - CCCL 3.0.0
+     - CUDA 13.0
+
+   * - :ref:`warp_match_all <libcudacxx-extended-api-warp-warp-match-all>`
+     - Check if all lanes have the same value
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`lane_mask <libcudacxx-extended-api-warp-lane-mask>`
+     - Class to represent a mask of lanes in a warp
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`warp_match_any <libcudacxx-extended-api-warp-warp-match-any>`
+     - Get the mask of lanes with the same value
+     - CCCL 3.5.0
+     - CUDA 13.5
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/warp/lane_mask.rst b/cccl_upstream/docs/libcudacxx/extended_api/warp/lane_mask.rst
new file mode 100644
index 00000000..afaa8420
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/warp/lane_mask.rst
@@ -0,0 +1,93 @@
+.. _libcudacxx-extended-api-warp-lane-mask:
+
+``cuda::device::lane_mask``
+===========================
+
+Defined in ``<cuda/warp>`` header.
+
+.. code:: cuda
+
+    #include <cuda/std/cstdint>
+
+    namespace cuda::device
+    {
+
+    class lane_mask
+    {
+        // constructors
+        __device__ explicit constexpr lane_mask(cuda::std::uint32_t v = 0) noexcept;
+
+        // member functions
+        [[nodiscard]] __device__ constexpr cuda::std::uint32_t value() const noexcept;
+
+        // conversion operators
+        __device__ explicit constexpr operator cuda::std::uint32_t() const noexcept;
+
+        // static member functions
+        [[nodiscard]] __device__ static constexpr lane_mask none() noexcept;
+        [[nodiscard]] __device__ static constexpr lane_mask all() noexcept;
+        [[nodiscard]] __device__ static lane_mask all_active() noexcept;
+        [[nodiscard]] __device__ static lane_mask this_lane() noexcept;
+        [[nodiscard]] __device__ static lane_mask all_less() noexcept;
+        [[nodiscard]] __device__ static lane_mask all_less_equal() noexcept;
+        [[nodiscard]] __device__ static lane_mask all_greater() noexcept;
+        [[nodiscard]] __device__ static lane_mask all_greater_equal() noexcept;
+        [[nodiscard]] __device__ static lane_mask all_not_equal() noexcept;
+
+        // bitwise assignment operators
+        __device__ constexpr lane_mask& operator&=(lane_mask mask) noexcept;
+        __device__ constexpr lane_mask& operator|=(lane_mask mask) noexcept;
+        __device__ constexpr lane_mask& operator^=(lane_mask mask) noexcept;
+        __device__ constexpr lane_mask& operator<<=(int shift) noexcept;
+        __device__ constexpr lane_mask& operator>>=(int shift) noexcept;
+
+        // bitwise operators
+        [[nodiscard]] __device__ friend constexpr lane_mask operator~(lane_mask mask) noexcept;
+        [[nodiscard]] __device__ friend constexpr lane_mask operator&(lane_mask lhs, lane_mask rhs) noexcept;
+        [[nodiscard]] __device__ friend constexpr lane_mask operator|(lane_mask lhs, lane_mask rhs) noexcept;
+        [[nodiscard]] __device__ friend constexpr lane_mask operator^(lane_mask lhs, lane_mask rhs) noexcept;
+        [[nodiscard]] __device__ friend constexpr lane_mask operator<<(lane_mask mask, int shift) noexcept;
+        [[nodiscard]] __device__ friend constexpr lane_mask operator>>(lane_mask mask, int shift) noexcept;
+
+        // comparison operators
+        [[nodiscard]] __device__ friend constexpr bool operator==(lane_mask lhs, lane_mask rhs) noexcept;
+        [[nodiscard]] __device__ friend constexpr bool operator!=(lane_mask lhs, lane_mask rhs) noexcept;
+    };
+
+    } // namespace cuda::device
+
+``cuda::device::lane_mask`` is a class that represents a mask of lanes in a warp. It is a fancy wrapper around a single 32-bit unsigned integer value that allows for bitwise operations and comparisons, making it easier and safer to work with lane masks in CUDA device code.
+
+The class provides several ``static`` member functions to create common lane masks:
+
+- ``none()`` and ``all()`` are equivalent to ``lane_mask{0x0}`` and ``lane_mask{0xFFFFFFFF}``, respectively
+- ``all_active()`` returns a mask with all currently active lanes in the warp, equivalent to the result ``__activemask()``, and finally
+- ``this_lane()`` and other functions like ``all_greater()`` or ``all_less_equal()`` return masks depending on the current lane index. They are implemented using the PTX special registers.
+
+**Preconditions**
+
+- ``shift`` is in the range ``[0, 32)``.
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/std/cassert>
+    #include <cuda/std/type_traits>
+    #include <cuda/warp>
+
+    __global__ void lane_mask_kernel() {
+        // import lane_mask symbol to current scope
+        using cuda::device::lane_mask;
+        // this_lane() is equivalent to ~(all_less() | all_greater())
+        assert(lane_mask::this_lane() == ~(lane_mask::all_less() | lane_mask::all_greater()));
+    }
+
+    int main() {
+        lane_mask_kernel<<<1, 32>>>();
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/W7hExs16v>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/warp/warp_match_all.rst b/cccl_upstream/docs/libcudacxx/extended_api/warp/warp_match_all.rst
new file mode 100644
index 00000000..79943957
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/warp/warp_match_all.rst
@@ -0,0 +1,87 @@
+.. _libcudacxx-extended-api-warp-warp-match-all:
+
+``cuda::device::warp_match_all``
+================================
+
+Defined in ``<cuda/warp>`` header.
+
+.. code:: cuda
+
+    namespace cuda::device {
+
+    template <typename T>
+    [[nodiscard]] __device__ bool
+    warp_match_all(const T& data, lane_mask = lane_mask::all());
+
+    } // namespace cuda::device
+
+The functionality provides a generalized and safe alternative to CUDA warp match all intrinsic ``__match_all_sync``.
+The function allows bitwise comparison of any data size, including raw arrays, pointers, and structs.
+
+.. note::
+
+  The underlying CUDA intrinsic does not provide memory ordering.
+
+**Parameters**
+
+- ``data``: data to compare.
+- ``lane_mask``: mask of the active lanes.
+
+**Return value**
+
+- ``true`` if all lanes in the ``lane_mask`` have the same value for ``data``. ``false`` otherwise.
+
+**Constraints**
+
+- ``T`` shall be trivially copyable, see :ref:`cuda::is_trivially_copyable <libcudacxx-extended-api-type_traits-is_trivially_copyable>`.
+- ``T`` shall be bitwise comparable, see :ref:`cuda::is_bitwise_comparable <libcudacxx-extended-api-type_traits-is_bitwise_comparable>`, except when ``__builtin_clear_padding`` is supported. In the latter case, ``T`` can have padding bits.
+
+**Preconditions**
+
+- The functionality is only supported on ``SM >= 70``.
+- ``lane_mask`` must be non-zero.
+
+**Undefined Behavior**
+
+- ``lane_mask`` must represent a subset of the active lanes.
+- All non-exited lanes specified by ``lane_mask`` must execute the function with the same ``lane_mask`` value.
+
+**Performance considerations**
+
+- The function calls the PTX instruction ``match.sync`` :math:`ceil\left(\frac{sizeof(data)}{4}\right)` times.
+- The function is faster when called with a mask representing all active lanes in a warp (default value of the second parameter ``lane_mask``).
+- The function uses ``__ballot_sync`` when ``T`` is ``bool``.
+
+**References**
+
+- `CUDA match_all Intrinsics <https://docs.nvidia.com/cuda/cuda-programming-guide/05-appendices/cpp-language-extensions.html#warp-match-functions>`_
+- `PTX match.sync instruction <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-match-sync>`_
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/std/array>
+    #include <cuda/std/cassert>
+    #include <cuda/warp>
+
+    struct MyStruct {
+        double x; // 8 bytes
+        int    y; // 4 bytes
+    };            // 4 bytes of padding
+
+    __global__ void warp_match_kernel() {
+        assert(cuda::device::warp_match_all(2));
+        assert(cuda::device::warp_match_all(2, cuda::device::lane_mask::all()));
+        assert(cuda::device::warp_match_all(MyStruct{1.0, 3})); // compile error, except when __builtin_clear_padding is supported
+        assert(!cuda::device::warp_match_all(threadIdx.x));
+    }
+
+    int main() {
+        warp_match_kernel<<<1, 32>>>();
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/x1sWbx14r>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/warp/warp_match_any.rst b/cccl_upstream/docs/libcudacxx/extended_api/warp/warp_match_any.rst
new file mode 100644
index 00000000..4b123921
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/warp/warp_match_any.rst
@@ -0,0 +1,99 @@
+.. _libcudacxx-extended-api-warp-warp-match-any:
+
+``cuda::device::warp_match_any``
+================================
+
+Defined in ``<cuda/warp>`` header.
+
+.. code:: cuda
+
+    namespace cuda::device {
+
+    template <typename T>
+    [[nodiscard]] __device__ lane_mask
+    warp_match_any(const T& data, lane_mask = lane_mask::all());
+
+    } // namespace cuda::device
+
+The functionality provides a generalized and safe alternative to CUDA warp match any intrinsic ``__match_any_sync``.
+The function allows bitwise comparison of any data size, including raw arrays, pointers, and structs.
+
+.. note::
+
+  The underlying CUDA intrinsic does not provide memory ordering.
+
+**Parameters**
+
+- ``data``: data to compare.
+- ``lane_mask``: mask of the active lanes.
+
+**Return value**
+
+- A ``lane_mask`` representing the non-exited lanes in ``lane_mask`` that have the same bitwise value for ``data`` as  the calling lane.
+
+**Constraints**
+
+- ``T`` shall be trivially copyable, see :ref:`cuda::is_trivially_copyable <libcudacxx-extended-api-type_traits-is_trivially_copyable>`.
+- When ``__builtin_clear_padding`` is not supported, ``T`` shall have no padding bits, that is, ``T``'s value representation shall be identical to its object representation.
+
+**Preconditions**
+
+- The functionality is only supported on ``SM >= 70``.
+- ``lane_mask`` must be non-zero.
+
+**Undefined Behavior**
+
+- ``lane_mask`` must represent a subset of the active lanes.
+- All non-exited lanes specified by ``lane_mask`` must execute the function with the same ``lane_mask`` value.
+
+**Performance considerations**
+
+- The function calls the PTX instruction ``match.sync`` :math:`ceil\left(\frac{sizeof(data)}{4}\right)` times.
+- The function is faster when called with a mask representing all active lanes in a warp (default value of the second parameter ``lane_mask``).
+- The function uses ``__ballot_sync`` when ``T`` is ``bool``.
+
+**References**
+
+- `CUDA match_any Intrinsics <https://docs.nvidia.com/cuda/cuda-programming-guide/05-appendices/cpp-language-extensions.html#warp-match-functions>`_
+- `PTX match.sync instruction <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-match-sync>`_
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/std/array>
+    #include <cuda/std/cassert>
+    #include <cuda/warp>
+
+    struct MyStruct {
+        double x; // 8 bytes
+        int    y; // 4 bytes
+    };            // 4 bytes of padding
+
+    __global__ void warp_match_kernel() {
+        {
+            auto mask     = cuda::device::warp_match_any(threadIdx.x / 4);
+            auto expected = cuda::device::lane_mask{0b1111 << ((threadIdx.x / 4) * 4)};
+            assert(mask == expected);
+        }
+        {
+            auto mask     = cuda::device::warp_match_any(2);
+            auto expected = cuda::device::lane_mask{0xFFFFFFFF};
+            assert(mask == expected);
+        }
+        {
+            // compile error, except when __builtin_clear_padding is supported
+            auto mask     = cuda::device::warp_match_any(MyStruct{1.0, 3});
+            auto expected = cuda::device::lane_mask{0xFFFFFFFF};
+            assert(mask == expected);
+        }
+    }
+
+    int main() {
+        warp_match_kernel<<<1, 32>>>();
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/Ys1McG8nv>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/warp/warp_shuffle.rst b/cccl_upstream/docs/libcudacxx/extended_api/warp/warp_shuffle.rst
new file mode 100644
index 00000000..cb5194f6
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/warp/warp_shuffle.rst
@@ -0,0 +1,192 @@
+.. _libcudacxx-extended-api-warp-warp-shuffle:
+
+``cuda::device::warp_shuffle_idx/up/down/xor``
+==============================================
+
+Defined in ``<cuda/warp>`` header.
+
+``warp_shuffle_idx``:
+
+.. code:: cuda
+
+    namespace cuda::device {
+
+    template <int Width = 32, typename T>
+    [[nodiscard]] __device__ warp_shuffle_result<T>
+    warp_shuffle_idx(const T& data,
+                     int      src_lane,
+                     uint32_t lane_mask = 0xFFFFFFFF,
+                     cuda::std::integral_constant<int, Width> = {})
+
+    template <int Width = 32, typename T>
+    [[nodiscard]] __device__ warp_shuffle_result<T>
+    warp_shuffle_idx(const T& data,
+                     int      src_lane,
+                     cuda::std::integral_constant<int, Width>) // lane_mask is 0xFFFFFFFF
+
+    } // namespace cuda::device
+
+``warp_shuffle_up``:
+
+.. code:: cuda
+
+    namespace cuda::device {
+
+    template <int Width = 32, typename T>
+    [[nodiscard]] __device__ warp_shuffle_result<T>
+    warp_shuffle_up(const T& data,
+                    int      delta,
+                    uint32_t lane_mask = 0xFFFFFFFF,
+                    cuda::std::integral_constant<int, Width> = {})
+
+    template <int Width = 32, typename T>
+    [[nodiscard]] __device__ warp_shuffle_result<T>
+    warp_shuffle_up(const T& data,
+                    int      delta,
+                    cuda::std::integral_constant<int, Width>) // lane_mask is 0xFFFFFFFF
+
+    } // namespace cuda::device
+
+``warp_shuffle_down``:
+
+.. code:: cuda
+
+    namespace cuda::device {
+
+    template <int Width = 32, typename T>
+    [[nodiscard]] __device__ warp_shuffle_result<T>
+    warp_shuffle_down(const T& data,
+                      int      delta,
+                      uint32_t lane_mask = 0xFFFFFFFF,
+                      cuda::std::integral_constant<int, Width> = {})
+
+    template <int Width = 32, typename T>
+    [[nodiscard]] __device__ warp_shuffle_result<T>
+    warp_shuffle_down(const T& data,
+                      int      delta,
+                      cuda::std::integral_constant<int, Width>) // lane_mask is 0xFFFFFFFF
+
+    } // namespace cuda::device
+
+``warp_shuffle_xor``:
+
+.. code:: cuda
+
+    namespace cuda::device {
+
+    template <int Width = 32, typename T>
+    [[nodiscard]] __device__ warp_shuffle_result<T>
+    warp_shuffle_xor(const T& data,
+                     int      xor_mask,
+                     uint32_t lane_mask = 0xFFFFFFFF,
+                     cuda::std::integral_constant<int, Width> = {})
+
+    template <int Width = 32, typename T>
+    [[nodiscard]] __device__ warp_shuffle_result<T>
+    warp_shuffle_xor(const T& data,
+                     int      xor_mask,
+                     cuda::std::integral_constant<int, Width>) // lane_mask is 0xFFFFFFFF
+
+    } // namespace cuda::device
+
+Result type:
+
+.. code:: cuda
+
+    namespace cuda::device {
+
+    template <typename T>
+    struct warp_shuffle_result {
+        T    data;
+        bool pred;
+
+        __device__ operator T() const { return data; }
+    };
+
+    } // namespace cuda::device
+
+The functionality provides a generalized and safe alternative to CUDA warp shuffle intrinsics.
+The functions allow to exchange data of any data size, including raw arrays, pointers, and structs.
+
+**Parameters**
+
+- ``data``: data to exchange.
+- ``src_lane``: source lane.
+- ``delta``: offset from the source lane.
+- ``xor_mask``: XOR mask to apply to the source lane.
+
+**Return value**
+
+``warp_shuffle_result``:
+
+- ``data``: data of the destination lane.
+- ``pred``: ``true`` if the destination lane is within the source lane window. ``false`` otherwise.
+
+**Constrains**
+
+- ``Width`` must be a power of two in the range [1, 32]
+- ``T``: all ``T`` are allowed except if ``T`` is a pointer, in which case it must be a ``void`` pointer to avoid bug-prone code
+
+**Preconditions**
+
+- The destination lane must be a member of the ``lane_mask``.
+- ``delta`` and ``xor_mask`` must be less than ``Width``. Modulo behavior is allowed for ``src_lane``.
+- ``lane_mask`` must be non-zero.
+
+**Undefined Behavior**
+
+- ``lane_mask`` must represent a subset of the active lanes, undefined behavior otherwise.
+- All lanes must have the same value for ``lane_mask``, ``delta`` and ``xor_mask``
+
+**Performance considerations**
+
+- The function calls the PTX instruction ``shfl.sync`` :math:`ceil\left(\frac{sizeof(data)}{4}\right)` times.
+
+**References**
+
+- `CUDA Warp Shuffle Intrinsics <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#warp-shuffle>`_
+- `PTX Shfl.sync instruction <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-shfl-sync>`_
+
+Example
+-------
+
+.. code:: cuda
+
+    #include <cuda/std/array>
+    #include <cuda/std/type_traits>
+    #include <cuda/warp>
+    #include <cstdio>
+
+    struct MyStruct {
+        double x;
+        int    y;
+    };
+
+    __global__ void warp_shuffle_kernel() {
+        cuda::std::integral_constant<int, 16> half_warp;
+        auto                     laneid      = cuda::ptx::get_sreg_laneid();
+        int                      raw_array[] = {threadIdx.x, threadIdx.x + 1, threadIdx.x + 2};
+        cuda::std::array<int, 3> array       = {threadIdx.x, threadIdx.x + 1, threadIdx.x + 2};
+        MyStruct                 my_structs{static_cast<double>(threadIdx.x), threadIdx.x + 1};
+        if (laneid < 16) {
+            // lanes [0, 15] get an array with values {5, 6, 7}
+            auto ret = cuda::device::warp_shuffle_idx(raw_array, 5, 0xFFFF, half_warp);
+            printf("lane %2d: [%d, %d, %d]\n", laneid, ret.data[0], ret.data[1], ret.data[2]);
+
+            // lanes [1, 15] get an array with values {threadIdx.x - 1, threadIdx.x, threadIdx.x + 1}
+            // lane 0 keeps the original values
+            auto array_ret = cuda::device::warp_shuffle_up(array, 1, half_warp).data;
+            printf("lane %2d: [%d, %d, %d]\n", laneid, array[0], array[1], array_ret[2]);
+        }
+        // lanes [0, 13] get my_structs with values {threadIdx.x + 2, threadIdx.x + 3} and pred=true
+        auto ret = cuda::device::warp_shuffle_down<16>(my_structs, 2);
+        printf("lane %2d: {%f, %d}, pred %d\n", laneid, ret.data.x, ret.data.y, ret.pred);
+    }
+
+    int main() {
+        warp_shuffle_kernel<<<1, 32>>>();
+        cudaDeviceSynchronize();
+        return 0;
+    }
+
+`See it on Godbolt 🔗 <https://godbolt.org/z/soWTaG6Eb>`_
diff --git a/cccl_upstream/docs/libcudacxx/extended_api/work_stealing.rst b/cccl_upstream/docs/libcudacxx/extended_api/work_stealing.rst
new file mode 100644
index 00000000..e6eb870f
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/extended_api/work_stealing.rst
@@ -0,0 +1,119 @@
+.. _libcudacxx-extended-api-work-stealing:
+
+Work stealing
+=============
+
+Defined in header ``<cuda/work_stealing>`` if the CUDA compiler supports at least PTX ISA 8.7:
+
+.. code:: cuda
+
+  namespace cuda::device {
+
+  template <int ThreadBlockRank = 3, typename UnaryFunction = /*unspecified*/>
+  __device__ void for_each_canceled_block(UnaryFunction uf);
+
+  } // namespace cuda::device
+
+**Note**: On devices with compute capability 10.0 or higher, this function may leverage hardware acceleration.
+
+This API is primarily intended for implementing work-stealing at the thread-block level.
+
+
+Compared to alternative work distribution techniques, such as  `grid-stride loops <https://developer.nvidia.com/blog/cuda-pro-tip-write-flexible-kernels-grid-stride-loops/>`__, which distribute work statically, or dynamic work distribution methods relying on global memory concurrency, this API offers several advantages:
+
+   - It enables dynamic work-stealing: thread blocks that complete their tasks sooner can take on additional work from slower thread blocks.
+   - It may cooperate with the GPU work scheduler to respect work priorities and improve load balancing.
+   - It may reduce work-stealing latency compared to global memory atomics.
+
+For better performance, extract the shared thread-block prologue and epilogue outside the lambda and reuse them across thread-block iterations:
+
+  - Prologue: Thread-block initialization code and data common to all thread blocks, such as ``__shared__`` memory allocation and initialization.
+  - Epilogue: Epilogue: Thread-block finalization code common to all thread blocks, such as writing shared memory back to global memory..
+
+**Mandates**:
+
+   - ``ThreadBlockRank`` equals the rank of the thread block: ``1``, ``2``, or ``3`` for one-dimensional, two-dimensional, and three-dimensional thread blocks, respectively.
+   - ``is_invocable_r_v<UnaryFunction, void, dim3>`` is true.
+
+**Preconditions**:
+
+   - All threads within a thread block shall call ``for_each_canceled_block`` **exactly once**.
+
+**Effects**:
+
+   - Invokes ``uf`` with ``blockIdx`` and then repeatedly attempts to cancel the launch of another thread block within the current grid:
+
+      - If successful: invokes ``uf`` with the canceled thread block's ``blockIdx`` and repeats.
+      - Otherwise, the function returns; it failed to cancel the launch of another thread block.
+
+Example
+-------
+
+This example demonstrates work-stealing at thread-block granularity using this API.
+
+.. code:: cuda
+
+  // Before:
+
+  #include <cuda/cmath>
+  #include <cuda/work_stealing>
+  #include <iostream>
+
+  __global__ void vec_add(int* a, int* b, int* c, int n) {
+    // Extract common prologue outside the lambda, e.g.,
+    // - __shared__ or global (malloc) memory allocation
+    // - common initialization code
+    // - etc.
+
+    cuda::device::for_each_canceled_block<1>([=](dim3 block_idx) {
+      // block_idx may be different than the built-in blockIdx variable, that is:
+      // assert(block_idx == blockIdx); // may fail!
+      // so we need to use "block_idx" consistently inside for_each_canceled:
+      int idx = threadIdx.x + block_idx.x * blockDim.x;
+      if (idx < n) {
+        c[idx] += a[idx] + b[idx];
+      }
+    });
+    // Note: Calling for_each_canceled_block<1> again from this
+    // thread block exhibits undefined behavior.
+
+    // Extract common epilogue outside the lambda, e.g.,
+    // - write back shared memory to global memory
+    // - external synchronization
+    // - global memory deallocation (free)
+    // - etc.
+  }
+
+  int main() {
+  int N = 10000;
+  int *a, *b, *c;
+  cudaMallocManaged(&a, N * sizeof(int));
+  cudaMallocManaged(&b, N * sizeof(int));
+  cudaMallocManaged(&c, N * sizeof(int));
+  for (int i = 0; i < N; ++i) {
+    a[i] = i;
+    b[i] = 1;
+    c[i] = 0;
+  }
+
+  const int threads_per_block = 256;
+  const int blocks_per_grid = cuda::ceil_div(N, threads_per_block);
+
+  vec_add<<<blocks_per_grid, threads_per_block>>>(a, b, c, N);
+  cudaDeviceSynchronize();
+
+  bool success = true;
+  for (int i = 0; i < N; ++i) {
+    if (c[i] != (1 + i)) {
+      std::cerr << "ERROR " << i << ", " << c[i] << '\n';
+      success = false;
+    }
+  }
+  cudaFree(a);
+  cudaFree(b);
+  cudaFree(c);
+
+  return success? 0 : 1;
+  }
+
+[Try it live on Godbolt!](https://godbolt.org/z/Ksj4vsfWf)
diff --git a/cccl_upstream/docs/libcudacxx/index.rst b/cccl_upstream/docs/libcudacxx/index.rst
new file mode 100644
index 00000000..846333d3
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/index.rst
@@ -0,0 +1,137 @@
+.. _libcudacxx-module:
+
+libcu++
+========
+
+.. toctree::
+   :hidden:
+   :maxdepth: 3
+
+   Overview <self>
+   setup
+   standard_api
+   extended_api
+   runtime
+   ptx_api
+   tile
+   API reference <api/index>
+
+``libcu++`` (``libcudacxx``) provides fundamental, idiomatic C++ abstractions that aim to make the lives of CUDA C++
+developers easier.
+
+Specifically, ``libcu++`` provides:
+   - C++ Standard Library features usable in both host and device code
+   - Extensions to C++ Standard Library features
+   - Fundamental, CUDA-specific programming model abstractions
+
+C++ Standard Library Features
+-----------------------------
+
+If you are a C++ developer, then you know the C++ Standard Library (`sometimes referred to as "The
+STL" <https://stackoverflow.com/questions/5205491/whats-the-difference-between-stl-and-c-standard-library>`_)
+as what comes along with your compiler and provides things like ``std::string`` or ``std::vector`` or ``std::atomic``.
+It provides the fundamental abstractions that C++ developers need to build high quality applications and libraries.
+
+By default, these abstractions aren't available when writing CUDA C++ device code because they don't have the necessary
+``__host__ __device__`` decorators, and their implementation may not be suitable for using in and across host and device
+code.
+
+libcu++ aims to solve this problem by providing an opt-in, incremental, heterogeneous implementation of C++
+Standard Library features:
+
+   1. **Opt-in**: It does not replace the Standard Library provided by your host compiler (aka anything in ``std::``)
+   2. **Incremental**: It does not provide a complete C++ Standard Library implementation
+   3. **Heterogeneous**: It works in both host and device code, as well as passing between host and device code.
+
+If you know how to use things like the ``<atomic>`` or ``<type_traits>`` headers from the C++ Standard Library, then
+you know how to use libcu++.
+
+All you have to do is add ``cuda/std/`` to the start of your includes and ``cuda::`` before any uses of ``std::``:
+
+.. code:: cuda
+
+   #include <cuda/std/atomic>
+   cuda::std::atomic<int> x;
+
+.. note::
+
+   libcu++ does not provide its own documentation for Standard Library features. Instead, libcu++
+   :ref:`documents which Standard Library headers <libcudacxx-standard-api>` are made available, and defers documentation of
+   individual features within those headers to other sources like `cppreference <https://en.cppreference.com/w/>`_.
+
+C++ Standard Library Extensions
+-------------------------------
+
+libcu++ provides CUDA C++ developers with familiar Standard Library utilities to improve productivity and flatten the
+learning curve of learning CUDA. However, there are many aspects of writing high-performance CUDA C++ code that cannot
+be expressed through purely Standard conforming APIs. For these cases, libcu++ also provides *extensions* of Standard
+Library utilities.
+
+For example, libcu++ extends ``atomic<T>`` and other synchronization primitives with the notion of a "thread scope"
+that controls the strength of the memory fence.
+
+To use utilities that are extensions to Standard Library features, drop the ``std``:
+
+.. code:: cuda
+
+   #include <cuda/atomic>
+   cuda::atomic<int, cuda::thread_scope_device> x;
+
+See the :ref:`Extended API <libcudacxx-extended-api>` section for more information.
+
+Fundamental CUDA-specific Abstractions
+--------------------------------------
+
+Some abstractions that libcu++ provides have no equivalent in the C++ Standard Library, but are fundamental to the CUDA
+C++ programming model. They include CUDA-specific vocabulary types, device-side APIs for certain hardware features, and
+abstractions for the CUDA Runtime APIs.
+
+For example, :ref:`cuda::memcpy_async <libcudacxx-extended-api-asynchronous-operations-memcpy-async>` is a vital abstraction
+for asynchronous data movement between global and shared memory. This abstracts hardware features such as ``LDGSTS`` on
+Ampere, and the Tensor Memory Accelerator (TMA) on Hopper.
+
+See the :ref:`Extended API <libcudacxx-extended-api>` and :ref:`Runtime API <libcudacxx-runtime-api>` sections for more information.
+
+Summary: ``std::``, ``cuda::`` and ``cuda::std::``
+--------------------------------------------------
+
+-  ``std::`` / ``<*>``: This is your host compiler's Standard Library that works in ``__host__`` code only, although you
+   can use the ``--expt-relaxed-constexpr`` flag to use any ``constexpr`` functions in ``__device__`` code.
+   libcu++ does not replace or interfere with host compiler's Standard Library.
+-  ``cuda::std::`` / ``<cuda/std/*>``: Conforming implementations of facilities from the Standard Library that work in
+   ``__host__`` and  ``__device__`` code.
+-  ``cuda::`` / ``<cuda/*>``: Conforming extensions to the Standard Library that work in ``__host__`` and ``__device__`` code.
+-  ``cuda::device`` / ``<cuda/device/*>``: Conforming extensions to the Standard Library that work only in
+   ``__device__`` code.
+-  ``cuda::ptx``: C++ convenience wrappers for inline PTX that work only in ``__device__`` code.
+
+Any libcu++ feature that works in ``__host__`` code also works when being compiled by a non-CUDA compiler (e.g., GCC).
+
+.. code:: cuda
+
+   // Standard C++, __host__ only.
+   #include <atomic>
+   std::atomic<int> x;
+
+   // CUDA C++, __host__ __device__.
+   // Strictly conforming to the C++ Standard.
+   #include <cuda/std/atomic>
+   cuda::std::atomic<int> x;
+
+   // CUDA C++, __host__ __device__.
+   // Conforming extensions to the C++ Standard.
+   #include <cuda/atomic>
+   cuda::atomic<int, cuda::thread_scope_block> x;
+
+Licensing
+---------
+
+libcu++ is an open source project developed on `GitHub <https://github.com/nvidia/cccl>`_. It is NVIDIA's variant of
+`LLVM's libc++ <https://libcxx.llvm.org>`_. libcu++ is distributed under the `Apache License v2.0 with LLVM
+Exceptions <https://llvm.org/LICENSE.txt>`_.
+
+Conformance
+-----------
+
+libcu++ aims to be a conforming implementation of the C++ Standard, `ISO/IEC IS 14882 <https://eel.is/c++draft>`_,
+Clause 16 through 32.
diff --git a/cccl_upstream/docs/libcudacxx/ptx/examples.rst b/cccl_upstream/docs/libcudacxx/ptx/examples.rst
new file mode 100644
index 00000000..aca241bb
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/examples.rst
@@ -0,0 +1,10 @@
+.. _libcudacxx-ptx-examples:
+
+Examples
+========
+
+.. toctree::
+   :maxdepth: 1
+
+   examples/st.async
+   examples/device_tensormap_initialization
diff --git a/cccl_upstream/docs/libcudacxx/ptx/examples/device_tensormap_initialization.rst b/cccl_upstream/docs/libcudacxx/ptx/examples/device_tensormap_initialization.rst
new file mode 100644
index 00000000..530969f8
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/examples/device_tensormap_initialization.rst
@@ -0,0 +1,8 @@
+.. _libcudacxx-ptx-examples-device-tensormap-initialization:
+
+How to initialize a tensor map on device
+========================================
+
+How to initialize a tensor map on device is explained in the `CUDA Programming
+Guide
+<https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#encoding-a-tensor-map-on-device>`__.
diff --git a/cccl_upstream/docs/libcudacxx/ptx/examples/st.async.rst b/cccl_upstream/docs/libcudacxx/ptx/examples/st.async.rst
new file mode 100644
index 00000000..ea60629f
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/examples/st.async.rst
@@ -0,0 +1,85 @@
+.. _libcudacxx-ptx-examples-st-async:
+
+How to use st.async
+===================
+
+.. code:: cuda
+
+   #include <cstdio>
+   #include <cuda/ptx>
+   #include <cuda/barrier>
+   #include <cooperative_groups.h>
+
+   __global__ void __cluster_dims__(8, 1, 1) kernel()
+   {
+     using cuda::ptx::sem_release;
+     using cuda::ptx::sem_acquire;
+     using cuda::ptx::space_cluster;
+     using cuda::ptx::space_shared;
+     using cuda::ptx::scope_cluster;
+
+     namespace cg = cooperative_groups;
+     cg::cluster_group cluster = cg::this_cluster();
+
+     using barrier_t = cuda::barrier<cuda::thread_scope_block>;
+
+   #pragma nv_diag_suppress static_var_with_dynamic_init
+     __shared__ int receive_buffer[4];
+     __shared__ barrier_t bar;
+     init(&bar, blockDim.x);
+
+     // Sync cluster to ensure remote barrier is initialized.
+     cluster.sync();
+
+     // Get address of remote cluster barrier:
+     unsigned int other_block_rank = cluster.block_rank() ^ 1;
+     uint64_t * remote_bar = cluster.map_shared_rank(cuda::device::barrier_native_handle(bar), other_block_rank);
+     // int * remote_buffer = cluster.map_shared_rank(&receive_buffer, other_block_rank);
+     int * remote_buffer = cluster.map_shared_rank(&receive_buffer[0], other_block_rank);
+
+     // Arrive on local barrier:
+     uint64_t arrival_token;
+     if (threadIdx.x == 0) {
+       // Thread 0 arrives and indicates it expects to receive a certain number of bytes as well
+       arrival_token = cuda::ptx::mbarrier_arrive_expect_tx(sem_release, scope_cluster, space_shared, cuda::device::barrier_native_handle(bar), sizeof(receive_buffer));
+     } else {
+       arrival_token = cuda::ptx::mbarrier_arrive(sem_release, scope_cluster, space_shared, cuda::device::barrier_native_handle(bar));
+     }
+
+     if (threadIdx.x == 0) {
+       printf("[block %d] arrived with expected tx count = %llu\n", cluster.block_rank(), sizeof(receive_buffer));
+     }
+
+     // Send bytes to remote buffer, arriving on remote barrier
+     if (threadIdx.x == 0) {
+       cuda::ptx::st_async(remote_buffer, {int(cluster.block_rank()), 2, 3, 4}, remote_bar);
+     }
+
+     if (threadIdx.x == 0) {
+       printf("[block %d] st_async to %p, %p\n",
+              cluster.block_rank(),
+              remote_buffer,
+              remote_bar
+       );
+     }
+
+     // Wait on local barrier:
+     while(!cuda::ptx::mbarrier_try_wait(sem_acquire, scope_cluster, cuda::device::barrier_native_handle(bar), arrival_token)) {}
+
+     // Print received values:
+     if (threadIdx.x == 0) {
+       printf(
+         "[block %d] receive_buffer = { %d, %d, %d, %d }\n",
+         cluster.block_rank(),
+         receive_buffer[0], receive_buffer[1], receive_buffer[2], receive_buffer[3]
+       );
+     }
+
+   }
+
+   int main() {
+     kernel<<<8, 128>>>();
+     cudaDeviceSynchronize();
+   }
+
+`See it on Godbolt <https://cuda.godbolt.org/z/36GdbGdbf>`_
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions.rst
new file mode 100644
index 00000000..19f1f7da
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions.rst
@@ -0,0 +1,722 @@
+.. _libcudacxx-ptx-instructions:
+
+PTX Instructions
+================
+
+.. toctree::
+   :maxdepth: 1
+
+   instructions/ld
+   instructions/st
+   instructions/shr
+   instructions/shl
+   instructions/bmsk
+   instructions/elect_sync
+   instructions/prmt
+   instructions/barrier_cluster
+   instructions/bfind
+   instructions/clusterlaunchcontrol
+   instructions/cp_async_bulk
+   instructions/cp_async_bulk_commit_group
+   instructions/cp_async_bulk_wait_group
+   instructions/cp_async_bulk_tensor
+   instructions/cp_async_mbarrier_arrive
+   instructions/cp_reduce_async_bulk
+   instructions/cp_reduce_async_bulk_tensor
+   instructions/exit
+   instructions/fence
+   instructions/getctarank
+   instructions/mapa
+   instructions/mbarrier_init
+   instructions/mbarrier_inval
+   instructions/mbarrier_arrive
+   instructions/mbarrier_expect_tx
+   instructions/mbarrier_test_wait
+   instructions/mbarrier_try_wait
+   instructions/multimem_ld_reduce
+   instructions/multimem_red
+   instructions/multimem_st
+   instructions/red_async
+   instructions/shfl_sync
+   instructions/st_async
+   instructions/st_bulk
+   instructions/tcgen05_alloc
+   instructions/tcgen05_commit
+   instructions/tcgen05_cp
+   instructions/tcgen05_fence
+   instructions/tcgen05_ld
+   instructions/tcgen05_mma
+   instructions/tcgen05_mma_ws
+   instructions/tcgen05_shift
+   instructions/tcgen05_st
+   instructions/tcgen05_wait
+   instructions/tensormap_replace
+   instructions/tensormap_cp_fenceproxy
+   instructions/trap
+   instructions/setmaxnreg
+   instructions/special_registers
+
+
+Instructions by section
+-----------------------
+
+.. list-table:: `Integer Arithmetic Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `sad <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-sad>`__
+     - No
+   * - `div <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-div>`__
+     - No
+   * - `rem <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-rem>`__
+     - No
+   * - `abs <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-abs>`__
+     - No
+   * - `neg <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-neg>`__
+     - No
+   * - `min <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-min>`__
+     - No
+   * - `max <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-max>`__
+     - No
+   * - `popc <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-popc>`__
+     - No
+   * - `clz <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-clz>`__
+     - No
+   * - `bfind <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-bfind>`__
+     - CCCL 3.0.0
+   * - `fns <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-fns>`__
+     - No
+   * - `brev <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-brev>`__
+     - No
+   * - `bfe <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-bfe>`__
+     - No
+   * - `bfi <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-bfi>`__
+     - No
+   * - `szext <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-szext>`__
+     - No
+   * - `bmsk <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-bmsk>`__
+     - Yes, CCCL 3.0.0 / CUDA 13.0
+   * - `dp4a <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-dp4a>`__
+     - No
+   * - `dp2a <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-dp2a>`__
+     - No
+
+.. list-table:: `Extended-Precision Integer Arithmetic Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#extended-precision-integer-arithmetic-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `add.cc <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#extended-precision-arithmetic-instructions-add-cc>`__
+     - No
+   * - `addc <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#extended-precision-arithmetic-instructions-addc>`__
+     - No
+   * - `sub.cc <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#extended-precision-arithmetic-instructions-sub-cc>`__
+     - No
+   * - `subc <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#extended-precision-arithmetic-instructions-subc>`__
+     - No
+   * - `mad.cc <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#extended-precision-arithmetic-instructions-mad-cc>`__
+     - No
+   * - `madc <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#extended-precision-arithmetic-instructions-madc>`__
+     - No
+
+.. list-table:: `Floating-Point Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `testp <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-testp>`__
+     - No
+   * - `copysign <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-copysign>`__
+     - No
+   * - `add <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-add>`__
+     - No
+   * - `sub <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-sub>`__
+     - No
+   * - `mul <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-mul>`__
+     - No
+   * - `fma <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-fma>`__
+     - No
+   * - `mad <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-mad>`__
+     - No
+   * - `div <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-div>`__
+     - No
+   * - `abs <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-abs>`__
+     - No
+   * - `neg <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-neg>`__
+     - No
+   * - `min <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-min>`__
+     - No
+   * - `max <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-max>`__
+     - No
+   * - `rcp <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-rcp>`__
+     - No
+   * - `rcp.approx.ftz.f64 <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-rcp-approx-ftz-f64>`__
+     - No
+   * - `sqrt <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-sqrt>`__
+     - No
+   * - `rsqrt <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-rsqrt>`__
+     - No
+   * - `rsqrt.approx.ftz.f64 <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-rsqrt-approx-ftz-f64>`__
+     - No
+   * - `sin <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-sin>`__
+     - No
+   * - `cos <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-cos>`__
+     - No
+   * - `lg2 <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-lg2>`__
+     - No
+   * - `ex2 <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-ex2>`__
+     - No
+   * - `tanh <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#floating-point-instructions-tanh>`__
+     - No
+
+.. list-table:: `Half Precision Floating-Point Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#half-precision-floating-point-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `add <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#half-precision-floating-point-instructions-add>`__
+     - No
+   * - `sub <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#half-precision-floating-point-instructions-sub>`__
+     - No
+   * - `mul <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#half-precision-floating-point-instructions-mul>`__
+     - No
+   * - `fma <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#half-precision-floating-point-instructions-fma>`__
+     - No
+   * - `neg <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#half-precision-floating-point-instructions-neg>`__
+     - No
+   * - `abs <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#half-precision-floating-point-instructions-abs>`__
+     - No
+   * - `min <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#half-precision-floating-point-instructions-min>`__
+     - No
+   * - `max <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#half-precision-floating-point-instructions-max>`__
+     - No
+   * - `tanh <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#half-precision-floating-point-instructions-tanh>`__
+     - No
+   * - `ex2 <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#half-precision-floating-point-instructions-ex2>`__
+     - No
+
+.. list-table:: `Comparison and Selection Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#comparison-and-selection-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `set <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#comparison-and-selection-instructions-set>`__
+     - No
+   * - `setp <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#comparison-and-selection-instructions-setp>`__
+     - No
+   * - `selp <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#comparison-and-selection-instructions-selp>`__
+     - No
+   * - `slct <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#comparison-and-selection-instructions-slct>`__
+     - No
+
+.. list-table:: `Half Precision Comparison Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#half-precision-comparison-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `set <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#half-precision-comparison-instructions-set>`__
+     - No
+   * - `setp <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#half-precision-comparison-instructions-setp>`__
+     - No
+
+.. list-table:: `Logic and Shift Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#logic-and-shift-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `and <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#logic-and-shift-instructions-and>`__
+     - No
+   * - `or <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#logic-and-shift-instructions-or>`__
+     - No
+   * - `xor <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#logic-and-shift-instructions-xor>`__
+     - No
+   * - `not <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#logic-and-shift-instructions-not>`__
+     - No
+   * - `cnot <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#logic-and-shift-instructions-cnot>`__
+     - No
+   * - `lop3 <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#logic-and-shift-instructions-lop3>`__
+     - No
+   * - `shf <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#logic-and-shift-instructions-shf>`__
+     - No
+   * - `shl <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#logic-and-shift-instructions-shl>`__
+     - Yes, CCCL 3.0.0 / CUDA 13.0
+   * - `shr <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#logic-and-shift-instructions-shr>`__
+     - Yes, CCCL 3.0.0 / CUDA 13.0
+
+.. list-table:: `Data Movement and Conversion Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `mov <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-mov-2>`__
+     - No
+   * - `shfl <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-shfl-deprecated>`__
+     - No
+   * - `shfl.sync <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-shfl-sync>`__
+     - Yes, CCCL 2.9.0 / CUDA 12.9
+   * - `prmt <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-prmt>`__
+     - Yes, CCCL 3.0.0 / CUDA 13.0
+   * - `ld <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-ld>`__
+     - Yes, CCCL 3.0.0 / CUDA 13.0
+   * - `ld.global.nc <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-ld-global-nc>`__
+     - Yes, CCCL 3.0.0 / CUDA 13.0
+   * - `ldu <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-ldu>`__
+     - No
+   * - `st <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-st>`__
+     - Yes, CCCL 3.0.0 / CUDA 13.0
+   * - :ref:`st.async <libcudacxx-ptx-instructions-st-async>`
+     - CCCL 2.3.0 / CUDA 12.4
+   * - :ref:`st.bulk <libcudacxx-ptx-instructions-st-bulk>`
+     - CCCL 2.8 / CUDA 12.9
+   * - `multimem.ld_reduce, multimem.st, multimem.red <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-multimem-ld-reduce-multimem-st-multimem-red>`__
+     - CCCL 2.8 / CUDA 12.9
+   * - `prefetch, prefetchu <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-prefetch-prefetchu>`__
+     - No
+   * - `applypriority <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-applypriority>`__
+     - No
+   * - `discard <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-discard>`__
+     - No
+   * - `createpolicy <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-createpolicy>`__
+     - No
+   * - `isspacep <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-isspacep>`__
+     - No
+   * - `cvta <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cvta>`__
+     - No
+   * - `cvt <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cvt>`__
+     - No
+   * - `cvt.pack <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cvt-pack>`__
+     - No
+   * - :ref:`mapa <libcudacxx-ptx-instructions-mapa>`
+     - No
+   * - :ref:`getctarank <libcudacxx-ptx-instructions-getctarank>`
+     - CCCL 2.4.0 / CUDA 12.5
+
+.. list-table:: `Data Movement and Conversion Instructions: Asynchronous copy <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-asynchronous-copy>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `cp.async <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async>`__
+     - No
+   * - `cp.async.commit_group <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-commit-group>`__
+     - No
+   * - `cp.async.wait_group <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-wait-group-cp-async-wait-all>`__
+     - No
+   * - :ref:`cp.async.bulk <libcudacxx-ptx-instructions-cp-async-bulk>`
+     - CCCL 2.4.0 / CUDA 12.5
+   * - :ref:`cp.reduce.async.bulk <libcudacxx-ptx-instructions-cp-reduce-async-bulk>`
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `cp.async.bulk.prefetch <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-prefetch>`__
+     - No
+   * - :ref:`cp.reduce.async.bulk <libcudacxx-ptx-instructions-cp-async-bulk-tensor>`
+     - CCCL 2.4.0 / CUDA 12.5
+   * - :ref:`cp.reduce.async.bulk.tensor <libcudacxx-ptx-instructions-cp-reduce-async-bulk-tensor>`
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `cp.async.bulk.prefetch.tensor <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-prefetch-tensor>`__
+     - No
+   * - :ref:`cp.async.bulk.commit_group <libcudacxx-ptx-instructions-cp-async-bulk-commit_group>`
+     - CCCL 2.4.0 / CUDA 12.5
+   * - :ref:`cp.async.bulk.wait_group <libcudacxx-ptx-instructions-cp-async-bulk-wait_group>`
+     - CCCL 2.4.0 / CUDA 12.5
+   * - :ref:`tensormap.replace <libcudacxx-ptx-instructions-tensormap-replace>`
+     - CCCL 2.4.0 / CUDA 12.5
+
+.. list-table:: `Texture Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#texture-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `tex <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#texture-instructions-tex>`__
+     - No
+   * - `tld4 <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#texture-instructions-tld4>`__
+     - No
+   * - `txq <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#texture-instructions-txq>`__
+     - No
+   * - `istypep <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#texture-instructions-istypep>`__
+     - No
+
+.. list-table:: `Surface Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#surface-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `suld <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#surface-instructions-suld>`__
+     - No
+   * - `sust <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#surface-instructions-sust>`__
+     - No
+   * - `sured <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#surface-instructions-sured>`__
+     - No
+   * - `suq <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#surface-instructions-suq>`__
+     - No
+
+.. list-table:: `Control Flow Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#control-flow-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `{} <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#control-flow-instructions-curly-braces>`__
+     - No
+   * - `@ <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#control-flow-instructions-at>`__
+     - No
+   * - `bra <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#control-flow-instructions-bra>`__
+     - No
+   * - `bra <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#control-flow-instructions-bra>`__
+     - No
+   * - `call <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#control-flow-instructions-call>`__
+     - No
+   * - `ret <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#control-flow-instructions-ret>`__
+     - No
+   * - `exit <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#control-flow-instructions-exit>`__
+     - CCCL 3.0.0
+
+.. list-table:: `Parallel Synchronization and Communication Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `bar, barrier <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-bar-barrier>`__
+     - No
+   * - `bar.warp.sync <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-bar-warp-sync>`__
+     - No
+   * - :ref:`barrier.cluster <libcudacxx-ptx-instructions-barrier-cluster>`
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `membar <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-membar-fence>`__
+     - No
+   * - :ref:`fence <libcudacxx-ptx-instructions-fence>`
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `atom <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-atom>`__
+     - No
+   * - `red <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-red>`__
+     - No
+   * - :ref:`red.async <libcudacxx-ptx-instructions-mbarrier-red-async>`
+     - CCCL 2.3.0 / CUDA 12.4
+   * - `vote <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-vote-deprecated>`__
+     - No
+   * - `vote.sync <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-vote-sync>`__
+     - No
+   * - `match.sync <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-match-sync>`__
+     - No
+   * - `activemask <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-activemask>`__
+     - No
+   * - `redux.sync <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-redux-sync>`__
+     - No
+   * - `griddepcontrol <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-griddepcontrol>`__
+     - No
+   * - `elect.sync <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-elect-sync>`__
+     - CCCL 3.1.0 / CUDA 13.1
+
+.. list-table:: `Parallel Synchronization and Communication Instructions: mbarrier <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-mbarrier>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - :ref:`mbarrier.init <libcudacxx-ptx-instructions-mbarrier-init>`
+     - CCCL 2.5.0 / CUDA Future
+   * - `mbarrier.inval <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-mbarrier-inval>`__
+     - CCCL 3.2.0 / CUDA 13.2
+   * - `mbarrier.complete_tx <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-mbarrier-complete-tx>`__
+     - No
+   * - :ref:`mbarrier.arrive <libcudacxx-ptx-instructions-mbarrier-arrive>`
+     - CCCL 2.3.0 / CUDA 12.4
+   * - `mbarrier.arrive_drop <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-mbarrier-arrive-drop>`__
+     - No
+   * - `cp.async.mbarrier.arrive <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-cp-async-mbarrier-arrive>`__
+     - CCCL 2.8 / CUDA 12.9
+   * - `mbarrier.expect_tx <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-mbarrier-expect-tx>`__
+     - CCCL 2.8 / CUDA 12.9
+   * - :ref:`mbarrier.test_wait <libcudacxx-ptx-instructions-mbarrier-test_wait>`
+     - CCCL 2.3.0 / CUDA 12.4
+   * - :ref:`mbarrier.try_wait <libcudacxx-ptx-instructions-mbarrier-try_wait>`
+     - CCCL 2.3.0 / CUDA 12.4
+   * - `mbarrier.pending_count <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-mbarrier-pending-count>`__
+     - No
+   * - :ref:`tensormap.cp_fenceproxy <libcudacxx-ptx-instructions-tensormap-cp_fenceproxy>`
+     - CCCL 2.4.0 / CUDA 12.5
+   * - :ref:`clusterlaunchcontrol.try_cancel <libcudacxx-ptx-instructions-clusterlaunchcontrol>`
+     - CCCL 2.8 / CUDA 12.9
+   * - :ref:`clusterlaunchcontrol.query_cancel <libcudacxx-ptx-instructions-clusterlaunchcontrol>`
+     - CCCL 2.8 / CUDA 12.9
+
+.. list-table:: `Warp Level Matrix Multiply-Accumulate Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-multiply-accumulate-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `wmma.load <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-load-instruction-wmma-load>`__
+     - No
+   * - `wmma.store <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-store-instruction-wmma-store>`__
+     - No
+   * - `wmma.mma <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-instructions-wmma-mma>`__
+     - No
+   * - `mma <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-instructions-mma>`__
+     - No
+   * - `ldmatrix <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-load-instruction-ldmatrix>`__
+     - No
+   * - `stmatrix <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-store-instruction-stmatrix>`__
+     - No
+   * - `movmatrix <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-transpose-instruction-movmatrix>`__
+     - No
+   * - `mma.sp <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-instructions-for-sparse-mma>`__
+     - No
+
+.. list-table:: `Asynchronous Warpgroup Level Matrix Multiply-Accumulate Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#asynchronous-warpgroup-level-matrix-multiply-accumulate-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `wgmma.mma_async <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#asynchronous-multiply-and-accumulate-instruction-wgmma-mma-async>`__
+     - No
+   * - `wgmma.mma_async.sp <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#asynchronous-multiply-and-accumulate-instruction-wgmma-mma-async-sp>`__
+     - No
+   * - `wgmma.fence <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#asynchronous-multiply-and-accumulate-instruction-wgmma-fence>`__
+     - No
+   * - `wgmma.commit_group <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#asynchronous-multiply-and-accumulate-instruction-wgmma-commit-group>`__
+     - No
+   * - `wgmma.wait_group <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#asynchronous-multiply-and-accumulate-instruction-wgmma-wait-group>`__
+     - No
+
+.. list-table:: `TensorCore 5th Generation Family Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-family-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `tcgen05.alloc <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-alloc-tcgen05-dealloc-tcgen05-relinquish-alloc-permit>`__
+     - CCCL 2.8 / CUDA 12.9
+   * - `tcgen05.commit <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-alloc-tcgen05-commit>`__
+     - CCCL 2.8 / CUDA 12.9
+   * - `tcgen05.cp <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-alloc-tcgen05-cp>`__
+     - CCCL 2.8 / CUDA 12.9
+   * - `tcgen05.fence <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-alloc-tcgen05-fence>`__
+     - CCCL 2.8 / CUDA 12.9
+   * - `tcgen05.ld <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-alloc-tcgen05-ld>`__
+     - CCCL 2.8 / CUDA 12.9
+   * - `tcgen05.mma <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-alloc-tcgen05-mma>`__
+     - CCCL 2.8 / CUDA 12.9
+   * - `tcgen05.mma.ws <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-alloc-tcgen05-mma-ws>`__
+     - CCCL 2.8 / CUDA 12.9
+   * - `tcgen05.shift <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-alloc-tcgen05-shift>`__
+     - CCCL 2.8 / CUDA 12.9
+   * - `tcgen05.st <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-alloc-tcgen05-st>`__
+     - CCCL 2.8 / CUDA 12.9
+   * - `tcgen05.wait <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-alloc-tcgen05-wait>`__
+     - CCCL 2.8 / CUDA 12.9
+
+
+.. list-table:: `Stack Manipulation Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#stack-manipulation-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `stacksave <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#stack-manipulation-instructions-stacksave>`__
+     - No
+   * - `stackrestore <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#stack-manipulation-instructions-stackrestore>`__
+     - No
+   * - `alloca <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#stack-manipulation-instructions-alloca>`__
+     - No
+
+.. list-table:: `Video Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#video-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `vadd, vsub, vabsdiff, vmin, vmax <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#scalar-video-instructions-vadd-vsub-vabsdiff-vmin-vmax>`__
+     - No
+   * - `vshl, vshr <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#scalar-video-instructions-vshl-vshr>`__
+     - No
+   * - `vmad <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#scalar-video-instructions-vmad>`__
+     - No
+   * - `vset <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#scalar-video-instructions-vset>`__
+     - No
+
+.. list-table:: `SIMD Video Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#simd-video-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `vadd2, vsub2, vavrg2, vabsdiff2, vmin2, vmax2 <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#simd-video-instructions-vadd2-vsub2-vavrg2-vabsdiff2-vmin2-vmax2>`__
+     - No
+   * - `vset2 <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#simd-video-instructions-vset2>`__
+     - No
+   * - `vadd4, vsub4, vavrg4, vabsdiff4, vmin4, vmax4 <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#simd-video-instructions-vadd4-vsub4-vavrg4-vabsdiff4-vmin4-vmax4>`__
+     - No
+   * - `vset4 <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#simd-video-instructions-vset4>`__
+     - No
+
+.. list-table:: `Miscellaneous Instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#miscellaneous-instructions>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Instruction
+     - Available in libcu++
+   * - `brkpt <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#miscellaneous-instructions-brkpt>`__
+     - No
+   * - `nanosleep <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#miscellaneous-instructions-nanosleep>`__
+     - No
+   * - `pmevent <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#miscellaneous-instructions-pmevent>`__
+     - No
+   * - `trap <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#miscellaneous-instructions-trap>`__
+     - CCCL 3.0.0
+   * - `setmaxnreg <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#miscellaneous-instructions-setmaxnreg>`__
+     - CCCL 3.2.0 / CUDA 13.2
+
+.. list-table:: `Special registers <libcudacxx-ptx-instructions-special-registers>`
+   :widths: 25 25 25 25
+   :header-rows: 1
+
+   * - Instruction
+     - PTX ISA
+     - SM Version
+     - Available in libcu++
+   * - `tid <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-tid>`__
+     - 20
+     - All
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `ntid <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-ntid>`__
+     - 20
+     - All
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `laneid <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-laneid>`__
+     - 13
+     - All
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `warpid <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-warpid>`__
+     - 13
+     - All
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `nwarpid <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-nwarpid>`__
+     - 20
+     - 20
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `ctaid <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-ctaid>`__
+     - 20
+     - All
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `nctaid <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-nctaid>`__
+     - 20
+     - All
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `smid <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-smid>`__
+     - 13
+     - All
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `nsmid <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-nsmid>`__
+     - 20
+     - 20
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `gridid <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-gridid>`__
+     - 30
+     - 30
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `is_explicit_cluster <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-is-explicit-cluster>`__
+     - 78
+     - 90
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `clusterid <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-clusterid>`__
+     - 78
+     - 90
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `nclusterid <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-nclusterid>`__
+     - 78
+     - 90
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `cluster_ctaid <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-cluster-ctaid>`__
+     - 78
+     - 90
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `cluster_nctaid <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-cluster-nctaid>`__
+     - 78
+     - 90
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `cluster_ctarank <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-cluster-ctarank>`__
+     - 78
+     - 90
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `cluster_nctarank <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-cluster-nctarank>`__
+     - 78
+     - 90
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `lanemask_eq <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-lanemask-eq>`__
+     - 20
+     - 20
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `lanemask_le <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-lanemask-le>`__
+     - 20
+     - 20
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `lanemask_lt <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-lanemask-lt>`__
+     - 20
+     - 20
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `lanemask_ge <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-lanemask-ge>`__
+     - 20
+     - 20
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `lanemask_gt <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-lanemask-gt>`__
+     - 20
+     - 20
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `clock, clock_hi <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-clock-clock-hi>`__
+     - 10
+     - All
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `clock64 <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-clock64>`__
+     - 20
+     - 20
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `pm0 <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-pm0-pm7>`__
+     -
+     -
+     - No
+   * - `pm0_64 <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-pm0-64-pm7-64>`__
+     -
+     -
+     - No
+   * - `envreg <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-envreg-32>`__
+     -
+     -
+     - No
+   * - `globaltimer, globaltimer_lo, globaltimer_hi <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-globaltimer-globaltimer-lo-globaltimer-hi>`__
+     - 31
+     - 31
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `reserved_smem_offset_begin, reserved_smem_offset_end, reserved_smem_offset_cap, reserved_smem_offset_2 <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-reserved-smem-offset-begin-reserved-smem-offset-end-reserved-smem-offset-cap-reserved-smem-offset-2>`__
+     -
+     -
+     - No
+   * - `total_smem_size <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-total-smem-size>`__
+     - 41
+     - 20
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `aggr_smem_size <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-aggr-smem-size>`__
+     - 81
+     - 90
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `dynamic_smem_size <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-dynamic-smem-size>`__
+     - 41
+     - 20
+     - CCCL 2.4.0 / CUDA 12.5
+   * - `current_graph_exec <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-current-graph-exec>`__
+     - 80
+     - 50
+     - CCCL 2.4.0 / CUDA 12.5
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/barrier_cluster.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/barrier_cluster.rst
new file mode 100644
index 00000000..bc8943bc
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/barrier_cluster.rst
@@ -0,0 +1,16 @@
+.. _libcudacxx-ptx-instructions-barrier-cluster:
+
+barrier.cluster
+===============
+
+-  PTX ISA:
+   `barrier.cluster <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-barrier-cluster>`__
+
+Similar functionality is provided through the builtins
+``__cluster_barrier_arrive(), __cluster_barrier_arrive_relaxed(), __cluster_barrier_wait()``,
+as well as the ``cooperative_groups::cluster_group``
+`API <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cluster-group>`__.
+
+The ``.aligned`` variants of the instructions are not exposed.
+
+.. include:: generated/barrier_cluster.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/bfind.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/bfind.rst
new file mode 100644
index 00000000..694cdef2
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/bfind.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-bfind:
+
+bfind
+=====
+
+-  PTX ISA:
+   `bfind <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-bfind>`__
+
+.. include:: generated/bfind.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/bmsk.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/bmsk.rst
new file mode 100644
index 00000000..167a3b24
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/bmsk.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-bmsk:
+
+bmsk
+====
+
+-  PTX ISA:
+   `bmsk <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#integer-arithmetic-instructions-bmsk>`__
+
+.. include:: generated/bmsk.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/clusterlaunchcontrol.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/clusterlaunchcontrol.rst
new file mode 100644
index 00000000..75fe44f6
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/clusterlaunchcontrol.rst
@@ -0,0 +1,11 @@
+.. _libcudacxx-ptx-instructions-clusterlaunchcontrol:
+
+clusterlaunchcontrol
+====================
+
+-  PTX ISA:
+   `clusterlaunchcontrol.try_cancel <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-clusterlaunchcontrol-try-cancel>`__
+-  PTX ISA:
+   `clusterlaunchcontrol.query_cancel <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-clusterlaunchcontrol-query-cancel>`__
+
+.. include:: generated/clusterlaunchcontrol.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_async_bulk.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_async_bulk.rst
new file mode 100644
index 00000000..32121ef8
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_async_bulk.rst
@@ -0,0 +1,30 @@
+.. _libcudacxx-ptx-instructions-cp-async-bulk:
+
+cp.async.bulk
+=============
+
+-  PTX ISA:
+   `cp.async.bulk <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk>`__
+
+Implementation notes
+--------------------
+
+**NOTE.** Both ``srcMem`` and ``dstMem`` must be 16-byte aligned, and
+``size`` must be a multiple of 16.
+
+Changelog
+---------
+
+-  In earlier versions, ``cp_async_bulk_multicast`` was enabled for
+   SM_90. This has been changed to SM_90a.
+
+
+Unicast
+-------
+
+.. include:: generated/cp_async_bulk.rst
+
+Multicast
+---------
+
+.. include:: generated/cp_async_bulk_multicast.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_async_bulk_commit_group.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_async_bulk_commit_group.rst
new file mode 100644
index 00000000..8efc5ac0
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_async_bulk_commit_group.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-cp-async-bulk-commit_group:
+
+cp.async.bulk.commit_group
+==========================
+
+-  PTX ISA:
+   `cp.async.bulk.commit_group <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-commit-group>`__
+
+.. include:: generated/cp_async_bulk_commit_group.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_async_bulk_tensor.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_async_bulk_tensor.rst
new file mode 100644
index 00000000..8dc9def9
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_async_bulk_tensor.rst
@@ -0,0 +1,28 @@
+.. _libcudacxx-ptx-instructions-cp-async-bulk-tensor:
+
+cp.async.bulk.tensor
+====================
+
+-  PTX ISA:
+   `cp.async.bulk.tensor <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor>`__
+
+Changelog
+---------
+
+-  In earlier versions, ``cp_async_bulk_tensor_multicast`` was enabled
+   for SM_90. This has been changed to SM_90a.
+
+Unicast
+-------
+
+.. include:: generated/cp_async_bulk_tensor.rst
+
+Multicast
+---------
+
+.. include:: generated/cp_async_bulk_tensor_multicast.rst
+
+Scatter / Gather
+----------------
+
+.. include:: generated/cp_async_bulk_tensor_gather_scatter.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_async_bulk_wait_group.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_async_bulk_wait_group.rst
new file mode 100644
index 00000000..e24bb0fc
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_async_bulk_wait_group.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-cp-async-bulk-wait_group:
+
+cp.async.bulk.wait_group
+========================
+
+-  PTX ISA:
+   `cp.async.bulk.wait_group <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-wait-group>`__
+
+.. include:: generated/cp_async_bulk_wait_group.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_async_mbarrier_arrive.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_async_mbarrier_arrive.rst
new file mode 100644
index 00000000..f2ff2ff5
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_async_mbarrier_arrive.rst
@@ -0,0 +1,10 @@
+.. _libcudacxx-ptx-instructions-cp-async-mbarrier-arrive:
+
+cp.async.mbarrier.arrive
+========================
+
+-  PTX ISA:
+   `cp.async.mbarrier.arrive <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-cp-async-mbarrier-arrive>`__
+
+.. include:: generated/cp_async_mbarrier_arrive.rst
+.. include:: generated/cp_async_mbarrier_arrive_noinc.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_reduce_async_bulk.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_reduce_async_bulk.rst
new file mode 100644
index 00000000..a4710b5c
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_reduce_async_bulk.rst
@@ -0,0 +1,61 @@
+.. _libcudacxx-ptx-instructions-cp-reduce-async-bulk:
+
+cp.reduce.async.bulk
+====================
+
+-  PTX ISA:
+   `cp.reduce.async.bulk <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-reduce-async-bulk>`__
+
+
+Integer and floating point instructions
+---------------------------------------
+
+.. include:: generated/cp_reduce_async_bulk.rst
+
+Emulation of ``.s64`` instruction
+---------------------------------
+
+PTX does not currently (CTK 12.3) expose
+``cp.reduce.async.bulk.add.s64``. This exposure is emulated in
+``cuda::ptx`` using:
+
+.. code:: cuda
+
+   // cp.reduce.async.bulk.dst.src.mbarrier::complete_tx::bytes.op.u64 [dstMem], [srcMem], size, [rdsmem_bar]; // 2. PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .shared::cta }
+   // .type      = { .s64 }
+   // .op        = { .add }
+   template <typename=void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_add_t,
+     int64_t* dstMem,
+     const int64_t* srcMem,
+     uint32_t size,
+     uint64_t* rdsmem_bar);
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.u64  [dstMem], [srcMem], size; // 6. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .s64 }
+   // .op        = { .add }
+   template <typename=void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_add_t,
+     int64_t* dstMem,
+     const int64_t* srcMem,
+     uint32_t size);
+
+FP16 instructions
+-----------------
+
+.. include:: generated/cp_reduce_async_bulk_f16.rst
+
+BF16 instructions
+-----------------
+
+.. include:: generated/cp_reduce_async_bulk_bf16.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_reduce_async_bulk_tensor.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_reduce_async_bulk_tensor.rst
new file mode 100644
index 00000000..598d9e1e
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/cp_reduce_async_bulk_tensor.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-cp-reduce-async-bulk-tensor:
+
+cp.reduce.async.bulk.tensor
+===========================
+
+-  PTX ISA:
+   `cp.reduce.async.bulk.tensor <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-reduce-async-bulk-tensor>`__
+
+.. include:: generated/cp_reduce_async_bulk_tensor.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/elect_sync.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/elect_sync.rst
new file mode 100644
index 00000000..dea88c19
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/elect_sync.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-elect_sync:
+
+elect.sync
+==========
+
+-  PTX ISA:
+   `elect.sync <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-elect-sync>`__
+
+.. include:: generated/elect_sync.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/exit.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/exit.rst
new file mode 100644
index 00000000..899735f1
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/exit.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-exit:
+
+exit
+====
+
+-  PTX ISA:
+   `exit <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#control-flow-instructions-exit>`__
+
+.. include:: generated/exit.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/fence.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/fence.rst
new file mode 100644
index 00000000..4d9126be
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/fence.rst
@@ -0,0 +1,45 @@
+.. _libcudacxx-ptx-instructions-fence:
+
+fence
+=====
+
+-  PTX ISA:
+   `fence <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-membar-fence>`__
+
+.. _fence-1:
+
+fence
+-----
+
+.. include:: generated/fence.rst
+
+fence.sync_restrict
+-------------------
+
+.. include:: generated/fence_sync_restrict.rst
+
+fence.mbarrier_init
+-------------------
+
+.. include:: generated/fence_mbarrier_init.rst
+
+fence.proxy.alias
+-----------------
+
+.. include:: generated/fence_proxy_alias.rst
+
+fence.proxy.async
+-----------------
+
+
+.. include:: generated/fence_proxy_async.rst
+
+fence.proxy.async.sync_restrict
+-------------------------------
+
+.. include:: generated/fence_proxy_async_generic_sync_restrict.rst
+
+fence.proxy.tensormap
+---------------------
+
+.. include:: generated/fence_proxy_tensormap_generic.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/barrier_cluster.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/barrier_cluster.rst
new file mode 100644
index 00000000..accccfa1
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/barrier_cluster.rst
@@ -0,0 +1,53 @@
+..
+   This file was automatically generated. Do not edit.
+
+barrier.cluster.arrive
+^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // barrier.cluster.arrive; // PTX ISA 78, SM_90
+   // Marked volatile and as clobbering memory
+   template <typename = void>
+   __device__ static inline void barrier_cluster_arrive();
+
+barrier.cluster.wait
+^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // barrier.cluster.wait; // PTX ISA 78, SM_90
+   // Marked volatile and as clobbering memory
+   template <typename = void>
+   __device__ static inline void barrier_cluster_wait();
+
+barrier.cluster.arrive.release
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // barrier.cluster.arrive.sem; // PTX ISA 80, SM_90
+   // .sem       = { .release }
+   // Marked volatile and as clobbering memory
+   template <typename = void>
+   __device__ static inline void barrier_cluster_arrive(
+     cuda::ptx::sem_release_t);
+
+barrier.cluster.arrive.relaxed
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // barrier.cluster.arrive.sem; // PTX ISA 80, SM_90
+   // .sem       = { .relaxed }
+   // Marked volatile
+   template <typename = void>
+   __device__ static inline void barrier_cluster_arrive(
+     cuda::ptx::sem_relaxed_t);
+
+barrier.cluster.wait.acquire
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // barrier.cluster.wait.sem; // PTX ISA 80, SM_90
+   // .sem       = { .acquire }
+   // Marked volatile and as clobbering memory
+   template <typename = void>
+   __device__ static inline void barrier_cluster_wait(
+     cuda::ptx::sem_acquire_t);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/bfind.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/bfind.rst
new file mode 100644
index 00000000..007f45c9
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/bfind.rst
@@ -0,0 +1,74 @@
+..
+   This file was automatically generated. Do not edit.
+
+bfind.u32
+^^^^^^^^^
+.. code-block:: cuda
+
+   // bfind.u32 dest, a_reg; // PTX ISA 20, SM_50
+   template <typename U32, enable_if_t<sizeof(U32) == 4 && is_integral_v<U32> && is_unsigned_v<U32>, bool> = true>
+   __device__ static inline uint32_t bfind(
+     U32 a_reg);
+
+bfind.shiftamt.u32
+^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // bfind.shiftamt.u32 dest, a_reg; // PTX ISA 20, SM_50
+   template <typename U32, enable_if_t<sizeof(U32) == 4 && is_integral_v<U32> && is_unsigned_v<U32>, bool> = true>
+   __device__ static inline uint32_t bfind_shiftamt(
+     U32 a_reg);
+
+bfind.u64
+^^^^^^^^^
+.. code-block:: cuda
+
+   // bfind.u64 dest, a_reg; // PTX ISA 20, SM_50
+   template <typename U64, enable_if_t<sizeof(U64) == 8 && is_integral_v<U64> && is_unsigned_v<U64>, bool> = true>
+   __device__ static inline uint32_t bfind(
+     U64 a_reg);
+
+bfind.shiftamt.u64
+^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // bfind.shiftamt.u64 dest, a_reg; // PTX ISA 20, SM_50
+   template <typename U64, enable_if_t<sizeof(U64) == 8 && is_integral_v<U64> && is_unsigned_v<U64>, bool> = true>
+   __device__ static inline uint32_t bfind_shiftamt(
+     U64 a_reg);
+
+bfind.s32
+^^^^^^^^^
+.. code-block:: cuda
+
+   // bfind.s32 dest, a_reg; // PTX ISA 20, SM_50
+   template <typename S32, enable_if_t<sizeof(S32) == 4 && is_integral_v<S32> && is_signed_v<S32>, bool> = true>
+   __device__ static inline uint32_t bfind(
+     S32 a_reg);
+
+bfind.shiftamt.s32
+^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // bfind.shiftamt.s32 dest, a_reg; // PTX ISA 20, SM_50
+   template <typename S32, enable_if_t<sizeof(S32) == 4 && is_integral_v<S32> && is_signed_v<S32>, bool> = true>
+   __device__ static inline uint32_t bfind_shiftamt(
+     S32 a_reg);
+
+bfind.s64
+^^^^^^^^^
+.. code-block:: cuda
+
+   // bfind.s64 dest, a_reg; // PTX ISA 20, SM_50
+   template <typename S64, enable_if_t<sizeof(S64) == 8 && is_integral_v<S64> && is_signed_v<S64>, bool> = true>
+   __device__ static inline uint32_t bfind(
+     S64 a_reg);
+
+bfind.shiftamt.s64
+^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // bfind.shiftamt.s64 dest, a_reg; // PTX ISA 20, SM_50
+   template <typename S64, enable_if_t<sizeof(S64) == 8 && is_integral_v<S64> && is_signed_v<S64>, bool> = true>
+   __device__ static inline uint32_t bfind_shiftamt(
+     S64 a_reg);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/bmsk.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/bmsk.rst
new file mode 100644
index 00000000..b71d0e55
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/bmsk.rst
@@ -0,0 +1,22 @@
+..
+   This file was automatically generated. Do not edit.
+
+bmsk.clamp.b32
+^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // bmsk.clamp.b32 dest, a_reg, b_reg; // PTX ISA 76, SM_70
+   template <typename = void>
+   __device__ static inline uint32_t bmsk_clamp(
+     uint32_t a_reg,
+     uint32_t b_reg);
+
+bmsk.wrap.b32
+^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // bmsk.wrap.b32 dest, a_reg, b_reg; // PTX ISA 76, SM_70
+   template <typename = void>
+   __device__ static inline uint32_t bmsk_wrap(
+     uint32_t a_reg,
+     uint32_t b_reg);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/clusterlaunchcontrol.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/clusterlaunchcontrol.rst
new file mode 100644
index 00000000..8467478d
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/clusterlaunchcontrol.rst
@@ -0,0 +1,68 @@
+..
+   This file was automatically generated. Do not edit.
+
+clusterlaunchcontrol.try_cancel.async.shared::cta.mbarrier::complete_tx::bytes.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // clusterlaunchcontrol.try_cancel.async.shared::cta.mbarrier::complete_tx::bytes.b128 [addr], [smem_bar]; // PTX ISA 86, SM_100
+   template <typename = void>
+   __device__ static inline void clusterlaunchcontrol_try_cancel(
+     void* addr,
+     uint64_t* smem_bar);
+
+clusterlaunchcontrol.try_cancel.async.shared::cta.mbarrier::complete_tx::bytes.multicast::cluster::all.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // clusterlaunchcontrol.try_cancel.async.shared::cta.mbarrier::complete_tx::bytes.multicast::cluster::all.b128 [addr], [smem_bar]; // PTX ISA 86, SM_100a, SM_110a
+   template <typename = void>
+   __device__ static inline void clusterlaunchcontrol_try_cancel_multicast(
+     void* addr,
+     uint64_t* smem_bar);
+
+clusterlaunchcontrol.query_cancel.is_canceled.pred.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // clusterlaunchcontrol.query_cancel.is_canceled.pred.b128 pred_is_canceled, try_cancel_response; // PTX ISA 86, SM_100
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline bool clusterlaunchcontrol_query_cancel_is_canceled(
+     B128 try_cancel_response);
+
+clusterlaunchcontrol.query_cancel.get_first_ctaid::x.b32.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // clusterlaunchcontrol.query_cancel.get_first_ctaid::x.b32.b128 ret_dim, try_cancel_response; // PTX ISA 86, SM_100
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B32 clusterlaunchcontrol_query_cancel_get_first_ctaid_x(
+     B128 try_cancel_response);
+
+clusterlaunchcontrol.query_cancel.get_first_ctaid::y.b32.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // clusterlaunchcontrol.query_cancel.get_first_ctaid::y.b32.b128 ret_dim, try_cancel_response; // PTX ISA 86, SM_100
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B32 clusterlaunchcontrol_query_cancel_get_first_ctaid_y(
+     B128 try_cancel_response);
+
+clusterlaunchcontrol.query_cancel.get_first_ctaid::z.b32.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // clusterlaunchcontrol.query_cancel.get_first_ctaid::z.b32.b128 ret_dim, try_cancel_response; // PTX ISA 86, SM_100
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B32 clusterlaunchcontrol_query_cancel_get_first_ctaid_z(
+     B128 try_cancel_response);
+
+clusterlaunchcontrol.query_cancel.get_first_ctaid.v4.b32.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // clusterlaunchcontrol.query_cancel.get_first_ctaid.v4.b32.b128 block_dim, try_cancel_response; // PTX ISA 86, SM_100
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline void clusterlaunchcontrol_query_cancel_get_first_ctaid(
+     B32 (&block_dim)[4],
+     B128 try_cancel_response);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk.rst
new file mode 100644
index 00000000..16a49b56
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk.rst
@@ -0,0 +1,99 @@
+..
+   This file was automatically generated. Do not edit.
+
+cp.async.bulk.shared::cluster.global.mbarrier::complete_tx::bytes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.dst.src.mbarrier::complete_tx::bytes [dstMem], [srcMem], size, [smem_bar]; // PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* srcMem,
+     const uint32_t& size,
+     uint64_t* smem_bar);
+
+cp.async.bulk.shared::cta.global.mbarrier::complete_tx::bytes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.dst.src.mbarrier::complete_tx::bytes [dstMem], [srcMem], size, [smem_bar]; // PTX ISA 86, SM_90
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* srcMem,
+     const uint32_t& size,
+     uint64_t* smem_bar);
+
+cp.async.bulk.shared::cta.global.mbarrier::complete_tx::bytes.ignore_oob
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.dst.src.mbarrier::complete_tx::bytes.ignore_oob [dstMem], [srcMem], size, ignoreBytesLeft, ignoreBytesRight, [smem_bar]; // PTX ISA 92, SM_90
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_ignore_oob(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* srcMem,
+     const uint32_t& size,
+     const uint32_t& ignoreBytesLeft,
+     const uint32_t& ignoreBytesRight,
+     uint64_t* smem_bar);
+
+cp.async.bulk.shared::cluster.shared::cta.mbarrier::complete_tx::bytes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.dst.src.mbarrier::complete_tx::bytes [dstMem], [srcMem], size, [rdsmem_bar]; // PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .shared::cta }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_shared_t,
+     void* dstMem,
+     const void* srcMem,
+     const uint32_t& size,
+     uint64_t* rdsmem_bar);
+
+cp.async.bulk.global.shared::cta.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.dst.src.bulk_group [dstMem], [srcMem], size; // PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     void* dstMem,
+     const void* srcMem,
+     const uint32_t& size);
+
+cp.async.bulk.global.shared::cta.bulk_group.cp_mask
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.dst.src.bulk_group.cp_mask [dstMem], [srcMem], size, byteMask; // PTX ISA 86, SM_100
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_cp_mask(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     void* dstMem,
+     const void* srcMem,
+     const uint32_t& size,
+     const uint16_t& byteMask);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk_commit_group.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk_commit_group.rst
new file mode 100644
index 00000000..ffb0ea4b
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk_commit_group.rst
@@ -0,0 +1,10 @@
+..
+   This file was automatically generated. Do not edit.
+
+cp.async.bulk.commit_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.commit_group; // PTX ISA 80, SM_90
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_commit_group();
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk_multicast.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk_multicast.rst
new file mode 100644
index 00000000..17425c58
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk_multicast.rst
@@ -0,0 +1,19 @@
+..
+   This file was automatically generated. Do not edit.
+
+cp.async.bulk.shared::cluster.global.mbarrier::complete_tx::bytes.multicast::cluster
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.dst.src.mbarrier::complete_tx::bytes.multicast::cluster [dstMem], [srcMem], size, [smem_bar], ctaMask; // PTX ISA 80, SM_90a, SM_100a, SM_110a
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* srcMem,
+     const uint32_t& size,
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk_tensor.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk_tensor.rst
new file mode 100644
index 00000000..213cbb31
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk_tensor.rst
@@ -0,0 +1,417 @@
+..
+   This file was automatically generated. Do not edit.
+
+cp.async.bulk.tensor.1d.shared::cluster.global.tile.mbarrier::complete_tx::bytes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.1d.dst.src.tile.mbarrier::complete_tx::bytes [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[1],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.1d.shared::cta.global.tile.mbarrier::complete_tx::bytes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.1d.dst.src.tile.mbarrier::complete_tx::bytes [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 86, SM_90
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[1],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.1d.shared::cta.global.tile.mbarrier::complete_tx::bytes.cta_group::1
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.1d.dst.src.tile.mbarrier::complete_tx::bytes.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[1],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.1d.shared::cta.global.tile.mbarrier::complete_tx::bytes.cta_group::2
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.1d.dst.src.tile.mbarrier::complete_tx::bytes.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[1],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.1d.global.shared::cta.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.1d.dst.src.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[1],
+     const void* srcMem);
+
+cp.async.bulk.tensor.2d.shared::cluster.global.tile.mbarrier::complete_tx::bytes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.2d.dst.src.tile.mbarrier::complete_tx::bytes [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[2],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.2d.shared::cta.global.tile.mbarrier::complete_tx::bytes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.2d.dst.src.tile.mbarrier::complete_tx::bytes [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 86, SM_90
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[2],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.2d.shared::cta.global.tile.mbarrier::complete_tx::bytes.cta_group::1
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.2d.dst.src.tile.mbarrier::complete_tx::bytes.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[2],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.2d.shared::cta.global.tile.mbarrier::complete_tx::bytes.cta_group::2
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.2d.dst.src.tile.mbarrier::complete_tx::bytes.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[2],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.2d.global.shared::cta.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.2d.dst.src.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[2],
+     const void* srcMem);
+
+cp.async.bulk.tensor.3d.shared::cluster.global.tile.mbarrier::complete_tx::bytes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.3d.dst.src.tile.mbarrier::complete_tx::bytes [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[3],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.3d.shared::cta.global.tile.mbarrier::complete_tx::bytes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.3d.dst.src.tile.mbarrier::complete_tx::bytes [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 86, SM_90
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[3],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.3d.shared::cta.global.tile.mbarrier::complete_tx::bytes.cta_group::1
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.3d.dst.src.tile.mbarrier::complete_tx::bytes.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[3],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.3d.shared::cta.global.tile.mbarrier::complete_tx::bytes.cta_group::2
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.3d.dst.src.tile.mbarrier::complete_tx::bytes.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[3],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.3d.global.shared::cta.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.3d.dst.src.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[3],
+     const void* srcMem);
+
+cp.async.bulk.tensor.4d.shared::cluster.global.tile.mbarrier::complete_tx::bytes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.4d.dst.src.tile.mbarrier::complete_tx::bytes [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[4],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.4d.shared::cta.global.tile.mbarrier::complete_tx::bytes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.4d.dst.src.tile.mbarrier::complete_tx::bytes [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 86, SM_90
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[4],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.4d.shared::cta.global.tile.mbarrier::complete_tx::bytes.cta_group::1
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.4d.dst.src.tile.mbarrier::complete_tx::bytes.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[4],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.4d.shared::cta.global.tile.mbarrier::complete_tx::bytes.cta_group::2
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.4d.dst.src.tile.mbarrier::complete_tx::bytes.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[4],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.4d.global.shared::cta.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.4d.dst.src.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[4],
+     const void* srcMem);
+
+cp.async.bulk.tensor.5d.shared::cluster.global.tile.mbarrier::complete_tx::bytes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.5d.dst.src.tile.mbarrier::complete_tx::bytes [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.5d.shared::cta.global.tile.mbarrier::complete_tx::bytes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.5d.dst.src.tile.mbarrier::complete_tx::bytes [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 86, SM_90
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.5d.shared::cta.global.tile.mbarrier::complete_tx::bytes.cta_group::1
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.5d.dst.src.tile.mbarrier::complete_tx::bytes.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.5d.shared::cta.global.tile.mbarrier::complete_tx::bytes.cta_group::2
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.5d.dst.src.tile.mbarrier::complete_tx::bytes.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.5d.global.shared::cta.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.5d.dst.src.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     const void* srcMem);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk_tensor_gather_scatter.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk_tensor_gather_scatter.rst
new file mode 100644
index 00000000..abe15e25
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk_tensor_gather_scatter.rst
@@ -0,0 +1,124 @@
+..
+   This file was automatically generated. Do not edit.
+
+cp.async.bulk.tensor.2d.shared::cta.global.tile::gather4.mbarrier::complete_tx::bytes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.2d.dst.src.tile::gather4.mbarrier::complete_tx::bytes [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 86, SM_100
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor_tile_gather4(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.2d.shared::cta.global.tile::gather4.mbarrier::complete_tx::bytes.cta_group::1
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.2d.dst.src.tile::gather4.mbarrier::complete_tx::bytes.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor_tile_gather4(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.2d.shared::cta.global.tile::gather4.mbarrier::complete_tx::bytes.cta_group::2
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.2d.dst.src.tile::gather4.mbarrier::complete_tx::bytes.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cta }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor_tile_gather4(
+     cuda::ptx::space_shared_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     uint64_t* smem_bar);
+
+cp.async.bulk.tensor.2d.shared::cluster.global.tile::gather4.mbarrier::complete_tx::bytes.multicast::cluster
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.2d.dst.src.tile::gather4.mbarrier::complete_tx::bytes.multicast::cluster [dstMem], [tensorMap, tensorCoords], [smem_bar], ctaMask; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor_tile_gather4(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
+
+cp.async.bulk.tensor.2d.shared::cluster.global.tile::gather4.mbarrier::complete_tx::bytes.multicast::cluster.cta_group::1
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.2d.dst.src.tile::gather4.mbarrier::complete_tx::bytes.multicast::cluster.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar], ctaMask; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor_tile_gather4(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
+
+cp.async.bulk.tensor.2d.shared::cluster.global.tile::gather4.mbarrier::complete_tx::bytes.multicast::cluster.cta_group::2
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.2d.dst.src.tile::gather4.mbarrier::complete_tx::bytes.multicast::cluster.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar], ctaMask; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor_tile_gather4(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
+
+cp.async.bulk.tensor.2d.global.shared::cta.tile::scatter4.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.2d.dst.src.tile::scatter4.bulk_group [tensorMap, tensorCoords], [srcMem]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor_tile_scatter4(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     const void* srcMem);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk_tensor_multicast.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk_tensor_multicast.rst
new file mode 100644
index 00000000..23a523d3
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk_tensor_multicast.rst
@@ -0,0 +1,277 @@
+..
+   This file was automatically generated. Do not edit.
+
+cp.async.bulk.tensor.1d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.multicast::cluster
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.1d.dst.src.tile.mbarrier::complete_tx::bytes.multicast::cluster [dstMem], [tensorMap, tensorCoords], [smem_bar], ctaMask; // PTX ISA 80, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[1],
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
+
+cp.async.bulk.tensor.1d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group::1
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.1d.dst.src.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar], ctaMask; // PTX ISA 80, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[1],
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
+
+cp.async.bulk.tensor.1d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group::2
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.1d.dst.src.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar], ctaMask; // PTX ISA 80, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[1],
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
+
+cp.async.bulk.tensor.2d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.multicast::cluster
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.2d.dst.src.tile.mbarrier::complete_tx::bytes.multicast::cluster [dstMem], [tensorMap, tensorCoords], [smem_bar], ctaMask; // PTX ISA 80, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[2],
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
+
+cp.async.bulk.tensor.2d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group::1
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.2d.dst.src.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar], ctaMask; // PTX ISA 80, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[2],
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
+
+cp.async.bulk.tensor.2d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group::2
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.2d.dst.src.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar], ctaMask; // PTX ISA 80, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[2],
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
+
+cp.async.bulk.tensor.3d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.multicast::cluster
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.3d.dst.src.tile.mbarrier::complete_tx::bytes.multicast::cluster [dstMem], [tensorMap, tensorCoords], [smem_bar], ctaMask; // PTX ISA 80, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[3],
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
+
+cp.async.bulk.tensor.3d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group::1
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.3d.dst.src.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar], ctaMask; // PTX ISA 80, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[3],
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
+
+cp.async.bulk.tensor.3d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group::2
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.3d.dst.src.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar], ctaMask; // PTX ISA 80, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[3],
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
+
+cp.async.bulk.tensor.4d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.multicast::cluster
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.4d.dst.src.tile.mbarrier::complete_tx::bytes.multicast::cluster [dstMem], [tensorMap, tensorCoords], [smem_bar], ctaMask; // PTX ISA 80, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[4],
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
+
+cp.async.bulk.tensor.4d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group::1
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.4d.dst.src.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar], ctaMask; // PTX ISA 80, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[4],
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
+
+cp.async.bulk.tensor.4d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group::2
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.4d.dst.src.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar], ctaMask; // PTX ISA 80, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[4],
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
+
+cp.async.bulk.tensor.5d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.multicast::cluster
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.5d.dst.src.tile.mbarrier::complete_tx::bytes.multicast::cluster [dstMem], [tensorMap, tensorCoords], [smem_bar], ctaMask; // PTX ISA 80, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   template <typename = void>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
+
+cp.async.bulk.tensor.5d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group::1
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.5d.dst.src.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar], ctaMask; // PTX ISA 80, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
+
+cp.async.bulk.tensor.5d.shared::cluster.global.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group::2
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.tensor.5d.dst.src.tile.mbarrier::complete_tx::bytes.multicast::cluster.cta_group [dstMem], [tensorMap, tensorCoords], [smem_bar], ctaMask; // PTX ISA 80, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .dst       = { .shared::cluster }
+   // .src       = { .global }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void cp_async_bulk_tensor(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_global_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     void* dstMem,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     uint64_t* smem_bar,
+     const uint16_t& ctaMask);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk_wait_group.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk_wait_group.rst
new file mode 100644
index 00000000..5010f2c0
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_bulk_wait_group.rst
@@ -0,0 +1,20 @@
+..
+   This file was automatically generated. Do not edit.
+
+cp.async.bulk.wait_group
+^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.wait_group N; // PTX ISA 80, SM_90
+   template <int N32>
+   __device__ static inline void cp_async_bulk_wait_group(
+     cuda::ptx::n32_t<N32> N);
+
+cp.async.bulk.wait_group.read
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.bulk.wait_group.read N; // PTX ISA 80, SM_90
+   template <int N32>
+   __device__ static inline void cp_async_bulk_wait_group_read(
+     cuda::ptx::n32_t<N32> N);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_mbarrier_arrive.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_mbarrier_arrive.rst
new file mode 100644
index 00000000..b0c56b3c
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_mbarrier_arrive.rst
@@ -0,0 +1,11 @@
+..
+   This file was automatically generated. Do not edit.
+
+cp.async.mbarrier.arrive.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.mbarrier.arrive.b64 [addr]; // PTX ISA 70, SM_80
+   template <typename = void>
+   __device__ static inline void cp_async_mbarrier_arrive(
+     uint64_t* addr);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_mbarrier_arrive_noinc.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_mbarrier_arrive_noinc.rst
new file mode 100644
index 00000000..9d17a092
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_async_mbarrier_arrive_noinc.rst
@@ -0,0 +1,11 @@
+..
+   This file was automatically generated. Do not edit.
+
+cp.async.mbarrier.arrive.noinc.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.async.mbarrier.arrive.noinc.b64 [addr]; // PTX ISA 70, SM_80
+   template <typename = void>
+   __device__ static inline void cp_async_mbarrier_arrive_noinc(
+     uint64_t* addr);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_reduce_async_bulk.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_reduce_async_bulk.rst
new file mode 100644
index 00000000..6b30c733
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_reduce_async_bulk.rst
@@ -0,0 +1,645 @@
+..
+   This file was automatically generated. Do not edit.
+
+cp.reduce.async.bulk.shared::cluster.shared::cta.mbarrier::complete_tx::bytes.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.mbarrier::complete_tx::bytes.op.type [dstMem], [srcMem], size, [rdsmem_bar]; // 1. PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .shared::cta }
+   // .type      = { .b32 }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_and_op_t,
+     B32* dstMem,
+     const B32* srcMem,
+     uint32_t size,
+     uint64_t* rdsmem_bar);
+
+cp.reduce.async.bulk.shared::cluster.shared::cta.mbarrier::complete_tx::bytes.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.mbarrier::complete_tx::bytes.op.type [dstMem], [srcMem], size, [rdsmem_bar]; // 1. PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .shared::cta }
+   // .type      = { .b32 }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_or_op_t,
+     B32* dstMem,
+     const B32* srcMem,
+     uint32_t size,
+     uint64_t* rdsmem_bar);
+
+cp.reduce.async.bulk.shared::cluster.shared::cta.mbarrier::complete_tx::bytes.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.mbarrier::complete_tx::bytes.op.type [dstMem], [srcMem], size, [rdsmem_bar]; // 1. PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .shared::cta }
+   // .type      = { .b32 }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_xor_op_t,
+     B32* dstMem,
+     const B32* srcMem,
+     uint32_t size,
+     uint64_t* rdsmem_bar);
+
+cp.reduce.async.bulk.shared::cluster.shared::cta.mbarrier::complete_tx::bytes.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.mbarrier::complete_tx::bytes.op.type [dstMem], [srcMem], size, [rdsmem_bar]; // 1. PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .shared::cta }
+   // .type      = { .u32 }
+   // .op        = { .min }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_min_t,
+     uint32_t* dstMem,
+     const uint32_t* srcMem,
+     uint32_t size,
+     uint64_t* rdsmem_bar);
+
+cp.reduce.async.bulk.shared::cluster.shared::cta.mbarrier::complete_tx::bytes.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.mbarrier::complete_tx::bytes.op.type [dstMem], [srcMem], size, [rdsmem_bar]; // 1. PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .shared::cta }
+   // .type      = { .u32 }
+   // .op        = { .max }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_max_t,
+     uint32_t* dstMem,
+     const uint32_t* srcMem,
+     uint32_t size,
+     uint64_t* rdsmem_bar);
+
+cp.reduce.async.bulk.shared::cluster.shared::cta.mbarrier::complete_tx::bytes.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.mbarrier::complete_tx::bytes.op.type [dstMem], [srcMem], size, [rdsmem_bar]; // 1. PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .shared::cta }
+   // .type      = { .u32 }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_add_t,
+     uint32_t* dstMem,
+     const uint32_t* srcMem,
+     uint32_t size,
+     uint64_t* rdsmem_bar);
+
+cp.reduce.async.bulk.shared::cluster.shared::cta.mbarrier::complete_tx::bytes.inc.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.mbarrier::complete_tx::bytes.op.type [dstMem], [srcMem], size, [rdsmem_bar]; // 1. PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .shared::cta }
+   // .type      = { .u32 }
+   // .op        = { .inc }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_inc_t,
+     uint32_t* dstMem,
+     const uint32_t* srcMem,
+     uint32_t size,
+     uint64_t* rdsmem_bar);
+
+cp.reduce.async.bulk.shared::cluster.shared::cta.mbarrier::complete_tx::bytes.dec.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.mbarrier::complete_tx::bytes.op.type [dstMem], [srcMem], size, [rdsmem_bar]; // 1. PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .shared::cta }
+   // .type      = { .u32 }
+   // .op        = { .dec }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_dec_t,
+     uint32_t* dstMem,
+     const uint32_t* srcMem,
+     uint32_t size,
+     uint64_t* rdsmem_bar);
+
+cp.reduce.async.bulk.shared::cluster.shared::cta.mbarrier::complete_tx::bytes.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.mbarrier::complete_tx::bytes.op.type [dstMem], [srcMem], size, [rdsmem_bar]; // 1. PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .shared::cta }
+   // .type      = { .s32 }
+   // .op        = { .min }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_min_t,
+     int32_t* dstMem,
+     const int32_t* srcMem,
+     uint32_t size,
+     uint64_t* rdsmem_bar);
+
+cp.reduce.async.bulk.shared::cluster.shared::cta.mbarrier::complete_tx::bytes.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.mbarrier::complete_tx::bytes.op.type [dstMem], [srcMem], size, [rdsmem_bar]; // 1. PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .shared::cta }
+   // .type      = { .s32 }
+   // .op        = { .max }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_max_t,
+     int32_t* dstMem,
+     const int32_t* srcMem,
+     uint32_t size,
+     uint64_t* rdsmem_bar);
+
+cp.reduce.async.bulk.shared::cluster.shared::cta.mbarrier::complete_tx::bytes.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.mbarrier::complete_tx::bytes.op.type [dstMem], [srcMem], size, [rdsmem_bar]; // 1. PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .shared::cta }
+   // .type      = { .s32 }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_add_t,
+     int32_t* dstMem,
+     const int32_t* srcMem,
+     uint32_t size,
+     uint64_t* rdsmem_bar);
+
+cp.reduce.async.bulk.shared::cluster.shared::cta.mbarrier::complete_tx::bytes.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.mbarrier::complete_tx::bytes.op.type [dstMem], [srcMem], size, [rdsmem_bar]; // 1. PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .shared::cta }
+   // .type      = { .u64 }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_add_t,
+     uint64_t* dstMem,
+     const uint64_t* srcMem,
+     uint32_t size,
+     uint64_t* rdsmem_bar);
+
+cp.reduce.async.bulk.shared::cluster.shared::cta.mbarrier::complete_tx::bytes.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.mbarrier::complete_tx::bytes.op.u64 [dstMem], [srcMem], size, [rdsmem_bar]; // 2. PTX ISA 80, SM_90
+   // .dst       = { .shared::cluster }
+   // .src       = { .shared::cta }
+   // .type      = { .s64 }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_add_t,
+     int64_t* dstMem,
+     const int64_t* srcMem,
+     uint32_t size,
+     uint64_t* rdsmem_bar);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 3. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .b32, .b64 }
+   // .op        = { .and }
+   template <typename Type>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_and_op_t,
+     Type* dstMem,
+     const Type* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.and.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 3. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .b32, .b64 }
+   // .op        = { .and }
+   template <typename Type>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_and_op_t,
+     Type* dstMem,
+     const Type* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 3. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .b32, .b64 }
+   // .op        = { .or }
+   template <typename Type>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_or_op_t,
+     Type* dstMem,
+     const Type* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.or.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 3. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .b32, .b64 }
+   // .op        = { .or }
+   template <typename Type>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_or_op_t,
+     Type* dstMem,
+     const Type* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 3. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .b32, .b64 }
+   // .op        = { .xor }
+   template <typename Type>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_xor_op_t,
+     Type* dstMem,
+     const Type* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.xor.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 3. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .b32, .b64 }
+   // .op        = { .xor }
+   template <typename Type>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_xor_op_t,
+     Type* dstMem,
+     const Type* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .u32 }
+   // .op        = { .min }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_min_t,
+     uint32_t* dstMem,
+     const uint32_t* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .u32 }
+   // .op        = { .max }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_max_t,
+     uint32_t* dstMem,
+     const uint32_t* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .u32 }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_add_t,
+     uint32_t* dstMem,
+     const uint32_t* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.inc.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .u32 }
+   // .op        = { .inc }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_inc_t,
+     uint32_t* dstMem,
+     const uint32_t* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.dec.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .u32 }
+   // .op        = { .dec }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_dec_t,
+     uint32_t* dstMem,
+     const uint32_t* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .s32 }
+   // .op        = { .min }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_min_t,
+     int32_t* dstMem,
+     const int32_t* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .s32 }
+   // .op        = { .max }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_max_t,
+     int32_t* dstMem,
+     const int32_t* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .s32 }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_add_t,
+     int32_t* dstMem,
+     const int32_t* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.min.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .u64 }
+   // .op        = { .min }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_min_t,
+     uint64_t* dstMem,
+     const uint64_t* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.max.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .u64 }
+   // .op        = { .max }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_max_t,
+     uint64_t* dstMem,
+     const uint64_t* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .u64 }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_add_t,
+     uint64_t* dstMem,
+     const uint64_t* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.min.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .s64 }
+   // .op        = { .min }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_min_t,
+     int64_t* dstMem,
+     const int64_t* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.max.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .s64 }
+   // .op        = { .max }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_max_t,
+     int64_t* dstMem,
+     const int64_t* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.add.f32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .f32 }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_add_t,
+     float* dstMem,
+     const float* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.add.f64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .f64 }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_add_t,
+     double* dstMem,
+     const double* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.u64  [dstMem], [srcMem], size; // 6. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .s64 }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_add_t,
+     int64_t* dstMem,
+     const int64_t* srcMem,
+     uint32_t size);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_reduce_async_bulk_bf16.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_reduce_async_bulk_bf16.rst
new file mode 100644
index 00000000..3f539f24
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_reduce_async_bulk_bf16.rst
@@ -0,0 +1,56 @@
+..
+   This file was automatically generated. Do not edit.
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.min.bf16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .bf16 }
+   // .op        = { .min }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_min_t,
+     __nv_bfloat16* dstMem,
+     const __nv_bfloat16* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.max.bf16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .bf16 }
+   // .op        = { .max }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_max_t,
+     __nv_bfloat16* dstMem,
+     const __nv_bfloat16* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.add.noftz.bf16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.noftz.type  [dstMem], [srcMem], size; // 5. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .bf16 }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_add_t,
+     __nv_bfloat16* dstMem,
+     const __nv_bfloat16* srcMem,
+     uint32_t size);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_reduce_async_bulk_f16.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_reduce_async_bulk_f16.rst
new file mode 100644
index 00000000..e3b27b3c
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_reduce_async_bulk_f16.rst
@@ -0,0 +1,56 @@
+..
+   This file was automatically generated. Do not edit.
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.min.f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .f16 }
+   // .op        = { .min }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_min_t,
+     __half* dstMem,
+     const __half* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.max.f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.type  [dstMem], [srcMem], size; // 4. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .f16 }
+   // .op        = { .max }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_max_t,
+     __half* dstMem,
+     const __half* srcMem,
+     uint32_t size);
+
+cp.reduce.async.bulk.global.shared::cta.bulk_group.add.noftz.f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.dst.src.bulk_group.op.noftz.type  [dstMem], [srcMem], size; // 5. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .type      = { .f16 }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline void cp_reduce_async_bulk(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_add_t,
+     __half* dstMem,
+     const __half* srcMem,
+     uint32_t size);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_reduce_async_bulk_tensor.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_reduce_async_bulk_tensor.rst
new file mode 100644
index 00000000..f48fb977
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/cp_reduce_async_bulk_tensor.rst
@@ -0,0 +1,682 @@
+..
+   This file was automatically generated. Do not edit.
+
+cp.reduce.async.bulk.tensor.1d.global.shared::cta.add.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.1d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1a. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[1],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.1d.global.shared::cta.min.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.1d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1a. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[1],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.1d.global.shared::cta.max.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.1d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1a. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[1],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.1d.global.shared::cta.inc.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.1d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1a. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[1],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.1d.global.shared::cta.dec.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.1d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1a. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[1],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.1d.global.shared::cta.and.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.1d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1a. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[1],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.1d.global.shared::cta.or.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.1d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1a. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[1],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.1d.global.shared::cta.xor.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.1d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1a. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[1],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.2d.global.shared::cta.add.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.2d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1b. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[2],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.2d.global.shared::cta.min.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.2d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1b. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[2],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.2d.global.shared::cta.max.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.2d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1b. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[2],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.2d.global.shared::cta.inc.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.2d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1b. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[2],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.2d.global.shared::cta.dec.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.2d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1b. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[2],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.2d.global.shared::cta.and.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.2d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1b. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[2],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.2d.global.shared::cta.or.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.2d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1b. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[2],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.2d.global.shared::cta.xor.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.2d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1b. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[2],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.3d.global.shared::cta.add.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.3d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1c. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[3],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.3d.global.shared::cta.min.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.3d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1c. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[3],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.3d.global.shared::cta.max.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.3d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1c. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[3],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.3d.global.shared::cta.inc.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.3d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1c. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[3],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.3d.global.shared::cta.dec.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.3d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1c. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[3],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.3d.global.shared::cta.and.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.3d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1c. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[3],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.3d.global.shared::cta.or.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.3d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1c. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[3],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.3d.global.shared::cta.xor.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.3d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1c. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[3],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.4d.global.shared::cta.add.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.4d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1d. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[4],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.4d.global.shared::cta.min.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.4d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1d. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[4],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.4d.global.shared::cta.max.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.4d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1d. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[4],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.4d.global.shared::cta.inc.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.4d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1d. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[4],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.4d.global.shared::cta.dec.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.4d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1d. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[4],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.4d.global.shared::cta.and.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.4d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1d. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[4],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.4d.global.shared::cta.or.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.4d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1d. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[4],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.4d.global.shared::cta.xor.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.4d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1d. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[4],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.5d.global.shared::cta.add.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.5d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1e. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.5d.global.shared::cta.min.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.5d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1e. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.5d.global.shared::cta.max.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.5d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1e. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.5d.global.shared::cta.inc.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.5d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1e. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.5d.global.shared::cta.dec.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.5d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1e. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.5d.global.shared::cta.and.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.5d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1e. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.5d.global.shared::cta.or.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.5d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1e. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     const void* srcMem);
+
+cp.reduce.async.bulk.tensor.5d.global.shared::cta.xor.tile.bulk_group
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // cp.reduce.async.bulk.tensor.5d.dst.src.op.tile.bulk_group [tensorMap, tensorCoords], [srcMem]; // 1e. PTX ISA 80, SM_90
+   // .dst       = { .global }
+   // .src       = { .shared::cta }
+   // .op        = { .add, .min, .max, .inc, .dec, .and, .or, .xor }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline void cp_reduce_async_bulk_tensor(
+     cuda::ptx::space_global_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::op_t<Op> op,
+     const void* tensorMap,
+     const int32_t (&tensorCoords)[5],
+     const void* srcMem);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/elect_sync.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/elect_sync.rst
new file mode 100644
index 00000000..2514ac4b
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/elect_sync.rst
@@ -0,0 +1,11 @@
+..
+   This file was automatically generated. Do not edit.
+
+elect.sync
+^^^^^^^^^^
+.. code-block:: cuda
+
+   // elect.sync _|is_elected, membermask; // PTX ISA 80, SM_90
+   template <typename = void>
+   __device__ static inline bool elect_sync(
+     const uint32_t& membermask);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/exit.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/exit.rst
new file mode 100644
index 00000000..f388a8b5
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/exit.rst
@@ -0,0 +1,10 @@
+..
+   This file was automatically generated. Do not edit.
+
+exit
+^^^^
+.. code-block:: cuda
+
+   // exit; // PTX ISA 10, SM_50
+   template <typename = void>
+   __device__ static inline void exit();
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence.rst
new file mode 100644
index 00000000..3cc2f365
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence.rst
@@ -0,0 +1,194 @@
+..
+   This file was automatically generated. Do not edit.
+
+fence.sc.cta
+^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.sem.scope; // 1. PTX ISA 60, SM_70
+   // .sem       = { .sc }
+   // .scope     = { .cta, .gpu, .sys }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence(
+     cuda::ptx::sem_sc_t,
+     cuda::ptx::scope_t<Scope> scope);
+
+fence.sc.gpu
+^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.sem.scope; // 1. PTX ISA 60, SM_70
+   // .sem       = { .sc }
+   // .scope     = { .cta, .gpu, .sys }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence(
+     cuda::ptx::sem_sc_t,
+     cuda::ptx::scope_t<Scope> scope);
+
+fence.sc.sys
+^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.sem.scope; // 1. PTX ISA 60, SM_70
+   // .sem       = { .sc }
+   // .scope     = { .cta, .gpu, .sys }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence(
+     cuda::ptx::sem_sc_t,
+     cuda::ptx::scope_t<Scope> scope);
+
+fence.sc.cluster
+^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.sem.scope; // 2. PTX ISA 78, SM_90
+   // .sem       = { .sc }
+   // .scope     = { .cluster }
+   template <typename = void>
+   __device__ static inline void fence(
+     cuda::ptx::sem_sc_t,
+     cuda::ptx::scope_cluster_t);
+
+fence.acq_rel.cta
+^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.sem.scope; // 1. PTX ISA 60, SM_70
+   // .sem       = { .acq_rel }
+   // .scope     = { .cta, .gpu, .sys }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence(
+     cuda::ptx::sem_acq_rel_t,
+     cuda::ptx::scope_t<Scope> scope);
+
+fence.acq_rel.gpu
+^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.sem.scope; // 1. PTX ISA 60, SM_70
+   // .sem       = { .acq_rel }
+   // .scope     = { .cta, .gpu, .sys }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence(
+     cuda::ptx::sem_acq_rel_t,
+     cuda::ptx::scope_t<Scope> scope);
+
+fence.acq_rel.sys
+^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.sem.scope; // 1. PTX ISA 60, SM_70
+   // .sem       = { .acq_rel }
+   // .scope     = { .cta, .gpu, .sys }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence(
+     cuda::ptx::sem_acq_rel_t,
+     cuda::ptx::scope_t<Scope> scope);
+
+fence.acq_rel.cluster
+^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.sem.scope; // 2. PTX ISA 78, SM_90
+   // .sem       = { .acq_rel }
+   // .scope     = { .cluster }
+   template <typename = void>
+   __device__ static inline void fence(
+     cuda::ptx::sem_acq_rel_t,
+     cuda::ptx::scope_cluster_t);
+
+fence.acquire.cta
+^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.sem.scope; // PTX ISA 86, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope);
+
+fence.acquire.cluster
+^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.sem.scope; // PTX ISA 86, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope);
+
+fence.acquire.gpu
+^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.sem.scope; // PTX ISA 86, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope);
+
+fence.acquire.sys
+^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.sem.scope; // PTX ISA 86, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope);
+
+fence.release.cta
+^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.sem.scope; // PTX ISA 86, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_t<Scope> scope);
+
+fence.release.cluster
+^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.sem.scope; // PTX ISA 86, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_t<Scope> scope);
+
+fence.release.gpu
+^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.sem.scope; // PTX ISA 86, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_t<Scope> scope);
+
+fence.release.sys
+^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.sem.scope; // PTX ISA 86, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_t<Scope> scope);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence_mbarrier_init.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence_mbarrier_init.rst
new file mode 100644
index 00000000..059d4e47
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence_mbarrier_init.rst
@@ -0,0 +1,14 @@
+..
+   This file was automatically generated. Do not edit.
+
+fence.mbarrier_init.release.cluster
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.mbarrier_init.sem.scope; // 3. PTX ISA 80, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cluster }
+   template <typename = void>
+   __device__ static inline void fence_mbarrier_init(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_cluster_t);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence_proxy_alias.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence_proxy_alias.rst
new file mode 100644
index 00000000..9c782c03
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence_proxy_alias.rst
@@ -0,0 +1,10 @@
+..
+   This file was automatically generated. Do not edit.
+
+fence.proxy.alias
+^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.proxy.alias; // 4. PTX ISA 75, SM_70
+   template <typename = void>
+   __device__ static inline void fence_proxy_alias();
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence_proxy_async.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence_proxy_async.rst
new file mode 100644
index 00000000..78884eb4
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence_proxy_async.rst
@@ -0,0 +1,40 @@
+..
+   This file was automatically generated. Do not edit.
+
+fence.proxy.async
+^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.proxy.async; // 5. PTX ISA 80, SM_90
+   template <typename = void>
+   __device__ static inline void fence_proxy_async();
+
+fence.proxy.async.global
+^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.proxy.async.space; // 6. PTX ISA 80, SM_90
+   // .space     = { .global, .shared::cluster, .shared::cta }
+   template <cuda::ptx::dot_space Space>
+   __device__ static inline void fence_proxy_async(
+     cuda::ptx::space_t<Space> space);
+
+fence.proxy.async.shared::cluster
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.proxy.async.space; // 6. PTX ISA 80, SM_90
+   // .space     = { .global, .shared::cluster, .shared::cta }
+   template <cuda::ptx::dot_space Space>
+   __device__ static inline void fence_proxy_async(
+     cuda::ptx::space_t<Space> space);
+
+fence.proxy.async.shared::cta
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.proxy.async.space; // 6. PTX ISA 80, SM_90
+   // .space     = { .global, .shared::cluster, .shared::cta }
+   template <cuda::ptx::dot_space Space>
+   __device__ static inline void fence_proxy_async(
+     cuda::ptx::space_t<Space> space);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence_proxy_async_generic_sync_restrict.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence_proxy_async_generic_sync_restrict.rst
new file mode 100644
index 00000000..06427b12
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence_proxy_async_generic_sync_restrict.rst
@@ -0,0 +1,30 @@
+..
+   This file was automatically generated. Do not edit.
+
+fence.proxy.async::generic.acquire.sync_restrict::shared::cluster.cluster
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.proxy.async::generic.sem.sync_restrict::space.scope; // PTX ISA 86, SM_90
+   // .sem       = { .acquire }
+   // .space     = { .shared::cluster }
+   // .scope     = { .cluster }
+   template <typename = void>
+   __device__ static inline void fence_proxy_async_generic_sync_restrict(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::scope_cluster_t);
+
+fence.proxy.async::generic.release.sync_restrict::shared::cta.cluster
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.proxy.async::generic.sem.sync_restrict::space.scope; // PTX ISA 86, SM_90
+   // .sem       = { .release }
+   // .space     = { .shared::cta }
+   // .scope     = { .cluster }
+   template <typename = void>
+   __device__ static inline void fence_proxy_async_generic_sync_restrict(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::scope_cluster_t);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence_proxy_tensormap_generic.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence_proxy_tensormap_generic.rst
new file mode 100644
index 00000000..ff7da587
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence_proxy_tensormap_generic.rst
@@ -0,0 +1,106 @@
+..
+   This file was automatically generated. Do not edit.
+
+fence.proxy.tensormap::generic.release.cta
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.proxy.tensormap::generic.release.scope; // 7. PTX ISA 83, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence_proxy_tensormap_generic(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_t<Scope> scope);
+
+fence.proxy.tensormap::generic.release.cluster
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.proxy.tensormap::generic.release.scope; // 7. PTX ISA 83, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence_proxy_tensormap_generic(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_t<Scope> scope);
+
+fence.proxy.tensormap::generic.release.gpu
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.proxy.tensormap::generic.release.scope; // 7. PTX ISA 83, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence_proxy_tensormap_generic(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_t<Scope> scope);
+
+fence.proxy.tensormap::generic.release.sys
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.proxy.tensormap::generic.release.scope; // 7. PTX ISA 83, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence_proxy_tensormap_generic(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_t<Scope> scope);
+
+fence.proxy.tensormap::generic.acquire.cta
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.proxy.tensormap::generic.sem.scope [addr], size; // 8. PTX ISA 83, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <int N32, cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence_proxy_tensormap_generic(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope,
+     const void* addr,
+     cuda::ptx::n32_t<N32> size);
+
+fence.proxy.tensormap::generic.acquire.cluster
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.proxy.tensormap::generic.sem.scope [addr], size; // 8. PTX ISA 83, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <int N32, cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence_proxy_tensormap_generic(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope,
+     const void* addr,
+     cuda::ptx::n32_t<N32> size);
+
+fence.proxy.tensormap::generic.acquire.gpu
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.proxy.tensormap::generic.sem.scope [addr], size; // 8. PTX ISA 83, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <int N32, cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence_proxy_tensormap_generic(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope,
+     const void* addr,
+     cuda::ptx::n32_t<N32> size);
+
+fence.proxy.tensormap::generic.acquire.sys
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.proxy.tensormap::generic.sem.scope [addr], size; // 8. PTX ISA 83, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <int N32, cuda::ptx::dot_scope Scope>
+   __device__ static inline void fence_proxy_tensormap_generic(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope,
+     const void* addr,
+     cuda::ptx::n32_t<N32> size);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence_sync_restrict.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence_sync_restrict.rst
new file mode 100644
index 00000000..4c62060f
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/fence_sync_restrict.rst
@@ -0,0 +1,30 @@
+..
+   This file was automatically generated. Do not edit.
+
+fence.acquire.sync_restrict::shared::cluster.cluster
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.sem.sync_restrict::space.scope; // PTX ISA 86, SM_90
+   // .sem       = { .acquire }
+   // .space     = { .shared::cluster }
+   // .scope     = { .cluster }
+   template <typename = void>
+   __device__ static inline void fence_sync_restrict(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::space_cluster_t,
+     cuda::ptx::scope_cluster_t);
+
+fence.release.sync_restrict::shared::cta.cluster
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // fence.sem.sync_restrict::space.scope; // PTX ISA 86, SM_90
+   // .sem       = { .release }
+   // .space     = { .shared::cta }
+   // .scope     = { .cluster }
+   template <typename = void>
+   __device__ static inline void fence_sync_restrict(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::space_shared_t,
+     cuda::ptx::scope_cluster_t);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/get_sreg.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/get_sreg.rst
new file mode 100644
index 00000000..cc3ad9ac
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/get_sreg.rst
@@ -0,0 +1,386 @@
+..
+   This file was automatically generated. Do not edit.
+
+tid.x
+^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%tid.x; // PTX ISA 20
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_tid_x();
+
+tid.y
+^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%tid.y; // PTX ISA 20
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_tid_y();
+
+tid.z
+^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%tid.z; // PTX ISA 20
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_tid_z();
+
+ntid.x
+^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%ntid.x; // PTX ISA 20
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_ntid_x();
+
+ntid.y
+^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%ntid.y; // PTX ISA 20
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_ntid_y();
+
+ntid.z
+^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%ntid.z; // PTX ISA 20
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_ntid_z();
+
+laneid
+^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%laneid; // PTX ISA 13
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_laneid();
+
+warpid
+^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%warpid; // PTX ISA 13
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_warpid();
+
+nwarpid
+^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%nwarpid; // PTX ISA 20, SM_35
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_nwarpid();
+
+ctaid.x
+^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%ctaid.x; // PTX ISA 20
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_ctaid_x();
+
+ctaid.y
+^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%ctaid.y; // PTX ISA 20
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_ctaid_y();
+
+ctaid.z
+^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%ctaid.z; // PTX ISA 20
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_ctaid_z();
+
+nctaid.x
+^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%nctaid.x; // PTX ISA 20
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_nctaid_x();
+
+nctaid.y
+^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%nctaid.y; // PTX ISA 20
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_nctaid_y();
+
+nctaid.z
+^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%nctaid.z; // PTX ISA 20
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_nctaid_z();
+
+smid
+^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%smid; // PTX ISA 13
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_smid();
+
+nsmid
+^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%nsmid; // PTX ISA 20, SM_35
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_nsmid();
+
+gridid
+^^^^^^
+.. code-block:: cuda
+
+   // mov.u64 sreg_value, %%gridid; // PTX ISA 30
+   template <typename = void>
+   __device__ static inline uint64_t get_sreg_gridid();
+
+is_explicit_cluster
+^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.pred sreg_value, %%is_explicit_cluster; // PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline bool get_sreg_is_explicit_cluster();
+
+clusterid.x
+^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%clusterid.x; // PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_clusterid_x();
+
+clusterid.y
+^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%clusterid.y; // PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_clusterid_y();
+
+clusterid.z
+^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%clusterid.z; // PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_clusterid_z();
+
+nclusterid.x
+^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%nclusterid.x; // PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_nclusterid_x();
+
+nclusterid.y
+^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%nclusterid.y; // PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_nclusterid_y();
+
+nclusterid.z
+^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%nclusterid.z; // PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_nclusterid_z();
+
+cluster_ctaid.x
+^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%cluster_ctaid.x; // PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_cluster_ctaid_x();
+
+cluster_ctaid.y
+^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%cluster_ctaid.y; // PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_cluster_ctaid_y();
+
+cluster_ctaid.z
+^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%cluster_ctaid.z; // PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_cluster_ctaid_z();
+
+cluster_nctaid.x
+^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%cluster_nctaid.x; // PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_cluster_nctaid_x();
+
+cluster_nctaid.y
+^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%cluster_nctaid.y; // PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_cluster_nctaid_y();
+
+cluster_nctaid.z
+^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%cluster_nctaid.z; // PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_cluster_nctaid_z();
+
+cluster_ctarank
+^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%cluster_ctarank; // PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_cluster_ctarank();
+
+cluster_nctarank
+^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%cluster_nctarank; // PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_cluster_nctarank();
+
+lanemask_eq
+^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%lanemask_eq; // PTX ISA 20, SM_35
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_lanemask_eq();
+
+lanemask_le
+^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%lanemask_le; // PTX ISA 20, SM_35
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_lanemask_le();
+
+lanemask_lt
+^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%lanemask_lt; // PTX ISA 20, SM_35
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_lanemask_lt();
+
+lanemask_ge
+^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%lanemask_ge; // PTX ISA 20, SM_35
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_lanemask_ge();
+
+lanemask_gt
+^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%lanemask_gt; // PTX ISA 20, SM_35
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_lanemask_gt();
+
+clock
+^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%clock; // PTX ISA 10
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_clock();
+
+clock_hi
+^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%clock_hi; // PTX ISA 50, SM_35
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_clock_hi();
+
+clock64
+^^^^^^^
+.. code-block:: cuda
+
+   // mov.u64 sreg_value, %%clock64; // PTX ISA 20, SM_35
+   template <typename = void>
+   __device__ static inline uint64_t get_sreg_clock64();
+
+globaltimer
+^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u64 sreg_value, %%globaltimer; // PTX ISA 31, SM_35
+   template <typename = void>
+   __device__ static inline uint64_t get_sreg_globaltimer();
+
+globaltimer_lo
+^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%globaltimer_lo; // PTX ISA 31, SM_35
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_globaltimer_lo();
+
+globaltimer_hi
+^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%globaltimer_hi; // PTX ISA 31, SM_35
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_globaltimer_hi();
+
+total_smem_size
+^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%total_smem_size; // PTX ISA 41, SM_35
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_total_smem_size();
+
+aggr_smem_size
+^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%aggr_smem_size; // PTX ISA 81, SM_90
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_aggr_smem_size();
+
+dynamic_smem_size
+^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u32 sreg_value, %%dynamic_smem_size; // PTX ISA 41, SM_35
+   template <typename = void>
+   __device__ static inline uint32_t get_sreg_dynamic_smem_size();
+
+current_graph_exec
+^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mov.u64 sreg_value, %%current_graph_exec; // PTX ISA 80, SM_50
+   template <typename = void>
+   __device__ static inline uint64_t get_sreg_current_graph_exec();
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/getctarank.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/getctarank.rst
new file mode 100644
index 00000000..b07b7d1e
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/getctarank.rst
@@ -0,0 +1,13 @@
+..
+   This file was automatically generated. Do not edit.
+
+getctarank.shared::cluster.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // getctarank.space.u32 dest, addr; // PTX ISA 78, SM_90
+   // .space     = { .shared::cluster }
+   template <typename = void>
+   __device__ static inline uint32_t getctarank(
+     cuda::ptx::space_cluster_t,
+     const void* addr);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/ld.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/ld.rst
new file mode 100644
index 00000000..f696855c
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/ld.rst
@@ -0,0 +1,2026 @@
+..
+   This file was automatically generated. Do not edit.
+
+ld.global.b8
+^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.b8 dest, [addr]; // PTX ISA 10, SM_50
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld(
+     cuda::ptx::space_global_t,
+     const B8* addr);
+
+ld.global.L2::256B.b8
+^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L2::256B.b8 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_L2_256B(
+     cuda::ptx::space_global_t,
+     const B8* addr);
+
+ld.global.b16
+^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.b16 dest, [addr]; // PTX ISA 10, SM_50
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld(
+     cuda::ptx::space_global_t,
+     const B16* addr);
+
+ld.global.L2::256B.b16
+^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L2::256B.b16 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_L2_256B(
+     cuda::ptx::space_global_t,
+     const B16* addr);
+
+ld.global.b32
+^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.b32 dest, [addr]; // PTX ISA 10, SM_50
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld(
+     cuda::ptx::space_global_t,
+     const B32* addr);
+
+ld.global.L2::256B.b32
+^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L2::256B.b32 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_L2_256B(
+     cuda::ptx::space_global_t,
+     const B32* addr);
+
+ld.global.b64
+^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.b64 dest, [addr]; // PTX ISA 10, SM_50
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld(
+     cuda::ptx::space_global_t,
+     const B64* addr);
+
+ld.global.L2::256B.b64
+^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L2::256B.b64 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_L2_256B(
+     cuda::ptx::space_global_t,
+     const B64* addr);
+
+ld.global.b128
+^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.b128 dest, [addr]; // PTX ISA 83, SM_70
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld(
+     cuda::ptx::space_global_t,
+     const B128* addr);
+
+ld.global.L2::256B.b128
+^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L2::256B.b128 dest, [addr]; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_L2_256B(
+     cuda::ptx::space_global_t,
+     const B128* addr);
+
+ld.global.v4.b64
+^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.v4.b64 dest, [addr]; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline B256 ld(
+     cuda::ptx::space_global_t,
+     const B256* addr);
+
+ld.global.L2::cache_hint.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L2::cache_hint.b8 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B8* addr,
+     uint64_t cache_policy);
+
+ld.global.L2::cache_hint.L2::256B.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L2::cache_hint.L2::256B.b8 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B8* addr,
+     uint64_t cache_policy);
+
+ld.global.L2::cache_hint.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L2::cache_hint.b16 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B16* addr,
+     uint64_t cache_policy);
+
+ld.global.L2::cache_hint.L2::256B.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L2::cache_hint.L2::256B.b16 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B16* addr,
+     uint64_t cache_policy);
+
+ld.global.L2::cache_hint.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L2::cache_hint.b32 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B32* addr,
+     uint64_t cache_policy);
+
+ld.global.L2::cache_hint.L2::256B.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L2::cache_hint.L2::256B.b32 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B32* addr,
+     uint64_t cache_policy);
+
+ld.global.L2::cache_hint.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L2::cache_hint.b64 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B64* addr,
+     uint64_t cache_policy);
+
+ld.global.L2::cache_hint.L2::256B.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L2::cache_hint.L2::256B.b64 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B64* addr,
+     uint64_t cache_policy);
+
+ld.global.L2::cache_hint.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L2::cache_hint.b128 dest, [addr], cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B128* addr,
+     uint64_t cache_policy);
+
+ld.global.L2::cache_hint.L2::256B.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L2::cache_hint.L2::256B.b128 dest, [addr], cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B128* addr,
+     uint64_t cache_policy);
+
+ld.global.L2::cache_hint.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L2::cache_hint.v4.b64 dest, [addr], cache_policy; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline B256 ld_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B256* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_first.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.b8 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_L1_evict_first(
+     cuda::ptx::space_global_t,
+     const B8* addr);
+
+ld.global.L1::evict_first.L2::256B.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.L2::256B.b8 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_L1_evict_first_L2_256B(
+     cuda::ptx::space_global_t,
+     const B8* addr);
+
+ld.global.L1::evict_first.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.b16 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_L1_evict_first(
+     cuda::ptx::space_global_t,
+     const B16* addr);
+
+ld.global.L1::evict_first.L2::256B.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.L2::256B.b16 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_L1_evict_first_L2_256B(
+     cuda::ptx::space_global_t,
+     const B16* addr);
+
+ld.global.L1::evict_first.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.b32 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_L1_evict_first(
+     cuda::ptx::space_global_t,
+     const B32* addr);
+
+ld.global.L1::evict_first.L2::256B.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.L2::256B.b32 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_L1_evict_first_L2_256B(
+     cuda::ptx::space_global_t,
+     const B32* addr);
+
+ld.global.L1::evict_first.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.b64 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_L1_evict_first(
+     cuda::ptx::space_global_t,
+     const B64* addr);
+
+ld.global.L1::evict_first.L2::256B.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.L2::256B.b64 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_L1_evict_first_L2_256B(
+     cuda::ptx::space_global_t,
+     const B64* addr);
+
+ld.global.L1::evict_first.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.b128 dest, [addr]; // PTX ISA 83, SM_70
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_L1_evict_first(
+     cuda::ptx::space_global_t,
+     const B128* addr);
+
+ld.global.L1::evict_first.L2::256B.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.L2::256B.b128 dest, [addr]; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_L1_evict_first_L2_256B(
+     cuda::ptx::space_global_t,
+     const B128* addr);
+
+ld.global.L1::evict_first.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.v4.b64 dest, [addr]; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline B256 ld_L1_evict_first(
+     cuda::ptx::space_global_t,
+     const B256* addr);
+
+ld.global.L1::evict_first.L2::cache_hint.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.L2::cache_hint.b8 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_L1_evict_first_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B8* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_first.L2::cache_hint.L2::256B.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.L2::cache_hint.L2::256B.b8 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_L1_evict_first_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B8* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_first.L2::cache_hint.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.L2::cache_hint.b16 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_L1_evict_first_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B16* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_first.L2::cache_hint.L2::256B.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.L2::cache_hint.L2::256B.b16 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_L1_evict_first_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B16* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_first.L2::cache_hint.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.L2::cache_hint.b32 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_L1_evict_first_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B32* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_first.L2::cache_hint.L2::256B.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.L2::cache_hint.L2::256B.b32 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_L1_evict_first_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B32* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_first.L2::cache_hint.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.L2::cache_hint.b64 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_L1_evict_first_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B64* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_first.L2::cache_hint.L2::256B.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.L2::cache_hint.L2::256B.b64 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_L1_evict_first_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B64* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_first.L2::cache_hint.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.L2::cache_hint.b128 dest, [addr], cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_L1_evict_first_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B128* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_first.L2::cache_hint.L2::256B.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.L2::cache_hint.L2::256B.b128 dest, [addr], cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_L1_evict_first_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B128* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_first.L2::cache_hint.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_first.L2::cache_hint.v4.b64 dest, [addr], cache_policy; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline B256 ld_L1_evict_first_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B256* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_last.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.b8 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_L1_evict_last(
+     cuda::ptx::space_global_t,
+     const B8* addr);
+
+ld.global.L1::evict_last.L2::256B.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.L2::256B.b8 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_L1_evict_last_L2_256B(
+     cuda::ptx::space_global_t,
+     const B8* addr);
+
+ld.global.L1::evict_last.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.b16 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_L1_evict_last(
+     cuda::ptx::space_global_t,
+     const B16* addr);
+
+ld.global.L1::evict_last.L2::256B.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.L2::256B.b16 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_L1_evict_last_L2_256B(
+     cuda::ptx::space_global_t,
+     const B16* addr);
+
+ld.global.L1::evict_last.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.b32 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_L1_evict_last(
+     cuda::ptx::space_global_t,
+     const B32* addr);
+
+ld.global.L1::evict_last.L2::256B.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.L2::256B.b32 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_L1_evict_last_L2_256B(
+     cuda::ptx::space_global_t,
+     const B32* addr);
+
+ld.global.L1::evict_last.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.b64 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_L1_evict_last(
+     cuda::ptx::space_global_t,
+     const B64* addr);
+
+ld.global.L1::evict_last.L2::256B.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.L2::256B.b64 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_L1_evict_last_L2_256B(
+     cuda::ptx::space_global_t,
+     const B64* addr);
+
+ld.global.L1::evict_last.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.b128 dest, [addr]; // PTX ISA 83, SM_70
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_L1_evict_last(
+     cuda::ptx::space_global_t,
+     const B128* addr);
+
+ld.global.L1::evict_last.L2::256B.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.L2::256B.b128 dest, [addr]; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_L1_evict_last_L2_256B(
+     cuda::ptx::space_global_t,
+     const B128* addr);
+
+ld.global.L1::evict_last.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.v4.b64 dest, [addr]; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline B256 ld_L1_evict_last(
+     cuda::ptx::space_global_t,
+     const B256* addr);
+
+ld.global.L1::evict_last.L2::cache_hint.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.L2::cache_hint.b8 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_L1_evict_last_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B8* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_last.L2::cache_hint.L2::256B.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.L2::cache_hint.L2::256B.b8 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_L1_evict_last_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B8* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_last.L2::cache_hint.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.L2::cache_hint.b16 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_L1_evict_last_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B16* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_last.L2::cache_hint.L2::256B.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.L2::cache_hint.L2::256B.b16 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_L1_evict_last_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B16* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_last.L2::cache_hint.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.L2::cache_hint.b32 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_L1_evict_last_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B32* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_last.L2::cache_hint.L2::256B.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.L2::cache_hint.L2::256B.b32 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_L1_evict_last_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B32* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_last.L2::cache_hint.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.L2::cache_hint.b64 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_L1_evict_last_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B64* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_last.L2::cache_hint.L2::256B.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.L2::cache_hint.L2::256B.b64 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_L1_evict_last_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B64* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_last.L2::cache_hint.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.L2::cache_hint.b128 dest, [addr], cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_L1_evict_last_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B128* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_last.L2::cache_hint.L2::256B.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.L2::cache_hint.L2::256B.b128 dest, [addr], cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_L1_evict_last_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B128* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::evict_last.L2::cache_hint.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::evict_last.L2::cache_hint.v4.b64 dest, [addr], cache_policy; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline B256 ld_L1_evict_last_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B256* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::no_allocate.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.b8 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_L1_no_allocate(
+     cuda::ptx::space_global_t,
+     const B8* addr);
+
+ld.global.L1::no_allocate.L2::256B.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.L2::256B.b8 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_L1_no_allocate_L2_256B(
+     cuda::ptx::space_global_t,
+     const B8* addr);
+
+ld.global.L1::no_allocate.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.b16 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_L1_no_allocate(
+     cuda::ptx::space_global_t,
+     const B16* addr);
+
+ld.global.L1::no_allocate.L2::256B.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.L2::256B.b16 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_L1_no_allocate_L2_256B(
+     cuda::ptx::space_global_t,
+     const B16* addr);
+
+ld.global.L1::no_allocate.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.b32 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_L1_no_allocate(
+     cuda::ptx::space_global_t,
+     const B32* addr);
+
+ld.global.L1::no_allocate.L2::256B.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.L2::256B.b32 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_L1_no_allocate_L2_256B(
+     cuda::ptx::space_global_t,
+     const B32* addr);
+
+ld.global.L1::no_allocate.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.b64 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_L1_no_allocate(
+     cuda::ptx::space_global_t,
+     const B64* addr);
+
+ld.global.L1::no_allocate.L2::256B.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.L2::256B.b64 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_L1_no_allocate_L2_256B(
+     cuda::ptx::space_global_t,
+     const B64* addr);
+
+ld.global.L1::no_allocate.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.b128 dest, [addr]; // PTX ISA 83, SM_70
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_L1_no_allocate(
+     cuda::ptx::space_global_t,
+     const B128* addr);
+
+ld.global.L1::no_allocate.L2::256B.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.L2::256B.b128 dest, [addr]; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_L1_no_allocate_L2_256B(
+     cuda::ptx::space_global_t,
+     const B128* addr);
+
+ld.global.L1::no_allocate.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.v4.b64 dest, [addr]; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline B256 ld_L1_no_allocate(
+     cuda::ptx::space_global_t,
+     const B256* addr);
+
+ld.global.L1::no_allocate.L2::cache_hint.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.L2::cache_hint.b8 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_L1_no_allocate_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B8* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::no_allocate.L2::cache_hint.L2::256B.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.L2::cache_hint.L2::256B.b8 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_L1_no_allocate_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B8* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::no_allocate.L2::cache_hint.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.L2::cache_hint.b16 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_L1_no_allocate_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B16* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::no_allocate.L2::cache_hint.L2::256B.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.L2::cache_hint.L2::256B.b16 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_L1_no_allocate_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B16* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::no_allocate.L2::cache_hint.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.L2::cache_hint.b32 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_L1_no_allocate_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B32* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::no_allocate.L2::cache_hint.L2::256B.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.L2::cache_hint.L2::256B.b32 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_L1_no_allocate_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B32* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::no_allocate.L2::cache_hint.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.L2::cache_hint.b64 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_L1_no_allocate_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B64* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::no_allocate.L2::cache_hint.L2::256B.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.L2::cache_hint.L2::256B.b64 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_L1_no_allocate_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B64* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::no_allocate.L2::cache_hint.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.L2::cache_hint.b128 dest, [addr], cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_L1_no_allocate_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B128* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::no_allocate.L2::cache_hint.L2::256B.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.L2::cache_hint.L2::256B.b128 dest, [addr], cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_L1_no_allocate_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B128* addr,
+     uint64_t cache_policy);
+
+ld.global.L1::no_allocate.L2::cache_hint.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.L1::no_allocate.L2::cache_hint.v4.b64 dest, [addr], cache_policy; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline B256 ld_L1_no_allocate_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B256* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.b8
+^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.b8 dest, [addr]; // PTX ISA 10, SM_50
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_nc(
+     cuda::ptx::space_global_t,
+     const B8* addr);
+
+ld.global.nc.L2::256B.b8
+^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L2::256B.b8 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_nc_L2_256B(
+     cuda::ptx::space_global_t,
+     const B8* addr);
+
+ld.global.nc.b16
+^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.b16 dest, [addr]; // PTX ISA 10, SM_50
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_nc(
+     cuda::ptx::space_global_t,
+     const B16* addr);
+
+ld.global.nc.L2::256B.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L2::256B.b16 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_nc_L2_256B(
+     cuda::ptx::space_global_t,
+     const B16* addr);
+
+ld.global.nc.b32
+^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.b32 dest, [addr]; // PTX ISA 10, SM_50
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_nc(
+     cuda::ptx::space_global_t,
+     const B32* addr);
+
+ld.global.nc.L2::256B.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L2::256B.b32 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_nc_L2_256B(
+     cuda::ptx::space_global_t,
+     const B32* addr);
+
+ld.global.nc.b64
+^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.b64 dest, [addr]; // PTX ISA 10, SM_50
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_nc(
+     cuda::ptx::space_global_t,
+     const B64* addr);
+
+ld.global.nc.L2::256B.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L2::256B.b64 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_nc_L2_256B(
+     cuda::ptx::space_global_t,
+     const B64* addr);
+
+ld.global.nc.b128
+^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.b128 dest, [addr]; // PTX ISA 83, SM_70
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_nc(
+     cuda::ptx::space_global_t,
+     const B128* addr);
+
+ld.global.nc.L2::256B.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L2::256B.b128 dest, [addr]; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_nc_L2_256B(
+     cuda::ptx::space_global_t,
+     const B128* addr);
+
+ld.global.nc.v4.b64
+^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.v4.b64 dest, [addr]; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline B256 ld_nc(
+     cuda::ptx::space_global_t,
+     const B256* addr);
+
+ld.global.nc.L2::cache_hint.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L2::cache_hint.b8 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_nc_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B8* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L2::cache_hint.L2::256B.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L2::cache_hint.L2::256B.b8 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_nc_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B8* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L2::cache_hint.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L2::cache_hint.b16 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_nc_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B16* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L2::cache_hint.L2::256B.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L2::cache_hint.L2::256B.b16 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_nc_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B16* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L2::cache_hint.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L2::cache_hint.b32 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_nc_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B32* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L2::cache_hint.L2::256B.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L2::cache_hint.L2::256B.b32 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_nc_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B32* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L2::cache_hint.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L2::cache_hint.b64 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_nc_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B64* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L2::cache_hint.L2::256B.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L2::cache_hint.L2::256B.b64 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_nc_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B64* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L2::cache_hint.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L2::cache_hint.b128 dest, [addr], cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_nc_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B128* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L2::cache_hint.L2::256B.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L2::cache_hint.L2::256B.b128 dest, [addr], cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_nc_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B128* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L2::cache_hint.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L2::cache_hint.v4.b64 dest, [addr], cache_policy; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline B256 ld_nc_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B256* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_first.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.b8 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_nc_L1_evict_first(
+     cuda::ptx::space_global_t,
+     const B8* addr);
+
+ld.global.nc.L1::evict_first.L2::256B.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.L2::256B.b8 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_nc_L1_evict_first_L2_256B(
+     cuda::ptx::space_global_t,
+     const B8* addr);
+
+ld.global.nc.L1::evict_first.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.b16 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_nc_L1_evict_first(
+     cuda::ptx::space_global_t,
+     const B16* addr);
+
+ld.global.nc.L1::evict_first.L2::256B.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.L2::256B.b16 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_nc_L1_evict_first_L2_256B(
+     cuda::ptx::space_global_t,
+     const B16* addr);
+
+ld.global.nc.L1::evict_first.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.b32 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_nc_L1_evict_first(
+     cuda::ptx::space_global_t,
+     const B32* addr);
+
+ld.global.nc.L1::evict_first.L2::256B.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.L2::256B.b32 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_nc_L1_evict_first_L2_256B(
+     cuda::ptx::space_global_t,
+     const B32* addr);
+
+ld.global.nc.L1::evict_first.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.b64 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_nc_L1_evict_first(
+     cuda::ptx::space_global_t,
+     const B64* addr);
+
+ld.global.nc.L1::evict_first.L2::256B.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.L2::256B.b64 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_nc_L1_evict_first_L2_256B(
+     cuda::ptx::space_global_t,
+     const B64* addr);
+
+ld.global.nc.L1::evict_first.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.b128 dest, [addr]; // PTX ISA 83, SM_70
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_nc_L1_evict_first(
+     cuda::ptx::space_global_t,
+     const B128* addr);
+
+ld.global.nc.L1::evict_first.L2::256B.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.L2::256B.b128 dest, [addr]; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_nc_L1_evict_first_L2_256B(
+     cuda::ptx::space_global_t,
+     const B128* addr);
+
+ld.global.nc.L1::evict_first.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.v4.b64 dest, [addr]; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline B256 ld_nc_L1_evict_first(
+     cuda::ptx::space_global_t,
+     const B256* addr);
+
+ld.global.nc.L1::evict_first.L2::cache_hint.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.L2::cache_hint.b8 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_nc_L1_evict_first_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B8* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_first.L2::cache_hint.L2::256B.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.L2::cache_hint.L2::256B.b8 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_nc_L1_evict_first_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B8* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_first.L2::cache_hint.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.L2::cache_hint.b16 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_nc_L1_evict_first_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B16* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_first.L2::cache_hint.L2::256B.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.L2::cache_hint.L2::256B.b16 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_nc_L1_evict_first_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B16* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_first.L2::cache_hint.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.L2::cache_hint.b32 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_nc_L1_evict_first_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B32* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_first.L2::cache_hint.L2::256B.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.L2::cache_hint.L2::256B.b32 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_nc_L1_evict_first_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B32* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_first.L2::cache_hint.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.L2::cache_hint.b64 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_nc_L1_evict_first_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B64* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_first.L2::cache_hint.L2::256B.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.L2::cache_hint.L2::256B.b64 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_nc_L1_evict_first_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B64* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_first.L2::cache_hint.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.L2::cache_hint.b128 dest, [addr], cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_nc_L1_evict_first_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B128* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_first.L2::cache_hint.L2::256B.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.L2::cache_hint.L2::256B.b128 dest, [addr], cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_nc_L1_evict_first_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B128* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_first.L2::cache_hint.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_first.L2::cache_hint.v4.b64 dest, [addr], cache_policy; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline B256 ld_nc_L1_evict_first_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B256* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_last.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.b8 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_nc_L1_evict_last(
+     cuda::ptx::space_global_t,
+     const B8* addr);
+
+ld.global.nc.L1::evict_last.L2::256B.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.L2::256B.b8 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_nc_L1_evict_last_L2_256B(
+     cuda::ptx::space_global_t,
+     const B8* addr);
+
+ld.global.nc.L1::evict_last.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.b16 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_nc_L1_evict_last(
+     cuda::ptx::space_global_t,
+     const B16* addr);
+
+ld.global.nc.L1::evict_last.L2::256B.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.L2::256B.b16 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_nc_L1_evict_last_L2_256B(
+     cuda::ptx::space_global_t,
+     const B16* addr);
+
+ld.global.nc.L1::evict_last.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.b32 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_nc_L1_evict_last(
+     cuda::ptx::space_global_t,
+     const B32* addr);
+
+ld.global.nc.L1::evict_last.L2::256B.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.L2::256B.b32 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_nc_L1_evict_last_L2_256B(
+     cuda::ptx::space_global_t,
+     const B32* addr);
+
+ld.global.nc.L1::evict_last.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.b64 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_nc_L1_evict_last(
+     cuda::ptx::space_global_t,
+     const B64* addr);
+
+ld.global.nc.L1::evict_last.L2::256B.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.L2::256B.b64 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_nc_L1_evict_last_L2_256B(
+     cuda::ptx::space_global_t,
+     const B64* addr);
+
+ld.global.nc.L1::evict_last.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.b128 dest, [addr]; // PTX ISA 83, SM_70
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_nc_L1_evict_last(
+     cuda::ptx::space_global_t,
+     const B128* addr);
+
+ld.global.nc.L1::evict_last.L2::256B.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.L2::256B.b128 dest, [addr]; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_nc_L1_evict_last_L2_256B(
+     cuda::ptx::space_global_t,
+     const B128* addr);
+
+ld.global.nc.L1::evict_last.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.v4.b64 dest, [addr]; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline B256 ld_nc_L1_evict_last(
+     cuda::ptx::space_global_t,
+     const B256* addr);
+
+ld.global.nc.L1::evict_last.L2::cache_hint.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.L2::cache_hint.b8 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_nc_L1_evict_last_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B8* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_last.L2::cache_hint.L2::256B.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.L2::cache_hint.L2::256B.b8 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_nc_L1_evict_last_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B8* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_last.L2::cache_hint.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.L2::cache_hint.b16 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_nc_L1_evict_last_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B16* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_last.L2::cache_hint.L2::256B.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.L2::cache_hint.L2::256B.b16 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_nc_L1_evict_last_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B16* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_last.L2::cache_hint.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.L2::cache_hint.b32 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_nc_L1_evict_last_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B32* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_last.L2::cache_hint.L2::256B.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.L2::cache_hint.L2::256B.b32 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_nc_L1_evict_last_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B32* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_last.L2::cache_hint.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.L2::cache_hint.b64 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_nc_L1_evict_last_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B64* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_last.L2::cache_hint.L2::256B.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.L2::cache_hint.L2::256B.b64 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_nc_L1_evict_last_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B64* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_last.L2::cache_hint.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.L2::cache_hint.b128 dest, [addr], cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_nc_L1_evict_last_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B128* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_last.L2::cache_hint.L2::256B.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.L2::cache_hint.L2::256B.b128 dest, [addr], cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_nc_L1_evict_last_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B128* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::evict_last.L2::cache_hint.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::evict_last.L2::cache_hint.v4.b64 dest, [addr], cache_policy; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline B256 ld_nc_L1_evict_last_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B256* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::no_allocate.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.b8 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_nc_L1_no_allocate(
+     cuda::ptx::space_global_t,
+     const B8* addr);
+
+ld.global.nc.L1::no_allocate.L2::256B.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.L2::256B.b8 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_nc_L1_no_allocate_L2_256B(
+     cuda::ptx::space_global_t,
+     const B8* addr);
+
+ld.global.nc.L1::no_allocate.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.b16 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_nc_L1_no_allocate(
+     cuda::ptx::space_global_t,
+     const B16* addr);
+
+ld.global.nc.L1::no_allocate.L2::256B.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.L2::256B.b16 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_nc_L1_no_allocate_L2_256B(
+     cuda::ptx::space_global_t,
+     const B16* addr);
+
+ld.global.nc.L1::no_allocate.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.b32 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_nc_L1_no_allocate(
+     cuda::ptx::space_global_t,
+     const B32* addr);
+
+ld.global.nc.L1::no_allocate.L2::256B.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.L2::256B.b32 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_nc_L1_no_allocate_L2_256B(
+     cuda::ptx::space_global_t,
+     const B32* addr);
+
+ld.global.nc.L1::no_allocate.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.b64 dest, [addr]; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_nc_L1_no_allocate(
+     cuda::ptx::space_global_t,
+     const B64* addr);
+
+ld.global.nc.L1::no_allocate.L2::256B.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.L2::256B.b64 dest, [addr]; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_nc_L1_no_allocate_L2_256B(
+     cuda::ptx::space_global_t,
+     const B64* addr);
+
+ld.global.nc.L1::no_allocate.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.b128 dest, [addr]; // PTX ISA 83, SM_70
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_nc_L1_no_allocate(
+     cuda::ptx::space_global_t,
+     const B128* addr);
+
+ld.global.nc.L1::no_allocate.L2::256B.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.L2::256B.b128 dest, [addr]; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_nc_L1_no_allocate_L2_256B(
+     cuda::ptx::space_global_t,
+     const B128* addr);
+
+ld.global.nc.L1::no_allocate.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.v4.b64 dest, [addr]; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline B256 ld_nc_L1_no_allocate(
+     cuda::ptx::space_global_t,
+     const B256* addr);
+
+ld.global.nc.L1::no_allocate.L2::cache_hint.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.L2::cache_hint.b8 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_nc_L1_no_allocate_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B8* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::no_allocate.L2::cache_hint.L2::256B.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.L2::cache_hint.L2::256B.b8 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline B8 ld_nc_L1_no_allocate_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B8* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::no_allocate.L2::cache_hint.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.L2::cache_hint.b16 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_nc_L1_no_allocate_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B16* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::no_allocate.L2::cache_hint.L2::256B.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.L2::cache_hint.L2::256B.b16 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 ld_nc_L1_no_allocate_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B16* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::no_allocate.L2::cache_hint.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.L2::cache_hint.b32 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_nc_L1_no_allocate_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B32* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::no_allocate.L2::cache_hint.L2::256B.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.L2::cache_hint.L2::256B.b32 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 ld_nc_L1_no_allocate_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B32* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::no_allocate.L2::cache_hint.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.L2::cache_hint.b64 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_nc_L1_no_allocate_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B64* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::no_allocate.L2::cache_hint.L2::256B.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.L2::cache_hint.L2::256B.b64 dest, [addr], cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 ld_nc_L1_no_allocate_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B64* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::no_allocate.L2::cache_hint.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.L2::cache_hint.b128 dest, [addr], cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_nc_L1_no_allocate_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B128* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::no_allocate.L2::cache_hint.L2::256B.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.L2::cache_hint.L2::256B.b128 dest, [addr], cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline B128 ld_nc_L1_no_allocate_L2_cache_hint_L2_256B(
+     cuda::ptx::space_global_t,
+     const B128* addr,
+     uint64_t cache_policy);
+
+ld.global.nc.L1::no_allocate.L2::cache_hint.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // ld.space.nc.L1::no_allocate.L2::cache_hint.v4.b64 dest, [addr], cache_policy; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline B256 ld_nc_L1_no_allocate_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     const B256* addr,
+     uint64_t cache_policy);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_arrive.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_arrive.rst
new file mode 100644
index 00000000..c8c4046d
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_arrive.rst
@@ -0,0 +1,207 @@
+..
+   This file was automatically generated. Do not edit.
+
+mbarrier.arrive.shared.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.shared.b64                                  state,  [addr];           // 1.  PTX ISA 70, SM_80
+   template <typename = void>
+   __device__ static inline uint64_t mbarrier_arrive(
+     uint64_t* addr);
+
+mbarrier.arrive.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.shared::cta.b64                             state,  [addr], count;    // 2.  PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline uint64_t mbarrier_arrive(
+     uint64_t* addr,
+     const uint32_t& count);
+
+mbarrier.arrive.release.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.sem.scope.space.b64                   state,  [addr];           // 3a.  PTX ISA 80, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cta, .cluster }
+   // .space     = { .shared::cta }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t mbarrier_arrive(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::space_shared_t,
+     uint64_t* addr);
+
+mbarrier.arrive.release.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.sem.scope.space.b64                   state,  [addr];           // 3a.  PTX ISA 80, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cta, .cluster }
+   // .space     = { .shared::cta }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t mbarrier_arrive(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::space_shared_t,
+     uint64_t* addr);
+
+mbarrier.arrive.release.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.sem.scope.space.b64                   state,  [addr], count;    // 3b.  PTX ISA 80, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cta, .cluster }
+   // .space     = { .shared::cta }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t mbarrier_arrive(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::space_shared_t,
+     uint64_t* addr,
+     const uint32_t& count);
+
+mbarrier.arrive.release.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.sem.scope.space.b64                   state,  [addr], count;    // 3b.  PTX ISA 80, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cta, .cluster }
+   // .space     = { .shared::cta }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t mbarrier_arrive(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::space_shared_t,
+     uint64_t* addr,
+     const uint32_t& count);
+
+mbarrier.arrive.release.cluster.shared::cluster.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.sem.scope.space.b64                   _, [addr];                // 4a.  PTX ISA 80, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cluster }
+   // .space     = { .shared::cluster }
+   template <typename = void>
+   __device__ static inline void mbarrier_arrive(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_cluster_t,
+     cuda::ptx::space_cluster_t,
+     uint64_t* addr);
+
+mbarrier.arrive.release.cluster.shared::cluster.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.sem.scope.space.b64                   _, [addr], count;         // 4b.  PTX ISA 80, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cluster }
+   // .space     = { .shared::cluster }
+   template <typename = void>
+   __device__ static inline void mbarrier_arrive(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_cluster_t,
+     cuda::ptx::space_cluster_t,
+     uint64_t* addr,
+     const uint32_t& count);
+
+mbarrier.arrive.relaxed.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.sem.scope.space.b64 state, [addr], count; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   // .space     = { .shared::cta }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t mbarrier_arrive(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::space_shared_t,
+     uint64_t* addr,
+     const uint32_t& count);
+
+mbarrier.arrive.relaxed.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.sem.scope.space.b64 state, [addr], count; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   // .space     = { .shared::cta }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t mbarrier_arrive(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::space_shared_t,
+     uint64_t* addr,
+     const uint32_t& count);
+
+mbarrier.arrive.relaxed.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.sem.scope.space.b64 state, [addr]; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   // .space     = { .shared::cta }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t mbarrier_arrive(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::space_shared_t,
+     uint64_t* addr);
+
+mbarrier.arrive.relaxed.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.sem.scope.space.b64 state, [addr]; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   // .space     = { .shared::cta }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t mbarrier_arrive(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::space_shared_t,
+     uint64_t* addr);
+
+mbarrier.arrive.relaxed.cluster.shared::cluster.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.sem.scope.space.b64 _, [addr], count; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cluster }
+   // .space     = { .shared::cluster }
+   template <typename = void>
+   __device__ static inline void mbarrier_arrive(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_cluster_t,
+     cuda::ptx::space_cluster_t,
+     uint64_t* addr,
+     const uint32_t& count);
+
+mbarrier.arrive.relaxed.cluster.shared::cluster.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.sem.scope.space.b64 _, [addr]; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cluster }
+   // .space     = { .shared::cluster }
+   template <typename = void>
+   __device__ static inline void mbarrier_arrive(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_cluster_t,
+     cuda::ptx::space_cluster_t,
+     uint64_t* addr);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_arrive_expect_tx.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_arrive_expect_tx.rst
new file mode 100644
index 00000000..093018f7
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_arrive_expect_tx.rst
@@ -0,0 +1,98 @@
+..
+   This file was automatically generated. Do not edit.
+
+mbarrier.arrive.expect_tx.release.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.expect_tx.sem.scope.space.b64 state, [addr], tx_count; // 8.  PTX ISA 80, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cta, .cluster }
+   // .space     = { .shared::cta }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t mbarrier_arrive_expect_tx(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::space_shared_t,
+     uint64_t* addr,
+     const uint32_t& tx_count);
+
+mbarrier.arrive.expect_tx.release.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.expect_tx.sem.scope.space.b64 state, [addr], tx_count; // 8.  PTX ISA 80, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cta, .cluster }
+   // .space     = { .shared::cta }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t mbarrier_arrive_expect_tx(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::space_shared_t,
+     uint64_t* addr,
+     const uint32_t& tx_count);
+
+mbarrier.arrive.expect_tx.release.cluster.shared::cluster.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.expect_tx.sem.scope.space.b64   _, [addr], tx_count; // 9.  PTX ISA 80, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cluster }
+   // .space     = { .shared::cluster }
+   template <typename = void>
+   __device__ static inline void mbarrier_arrive_expect_tx(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_cluster_t,
+     cuda::ptx::space_cluster_t,
+     uint64_t* addr,
+     const uint32_t& tx_count);
+
+mbarrier.arrive.expect_tx.relaxed.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.expect_tx.sem.scope.space.b64 state, [addr], txCount; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   // .space     = { .shared::cta }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t mbarrier_arrive_expect_tx(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::space_shared_t,
+     uint64_t* addr,
+     const uint32_t& txCount);
+
+mbarrier.arrive.expect_tx.relaxed.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.expect_tx.sem.scope.space.b64 state, [addr], txCount; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   // .space     = { .shared::cta }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t mbarrier_arrive_expect_tx(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::space_shared_t,
+     uint64_t* addr,
+     const uint32_t& txCount);
+
+mbarrier.arrive.expect_tx.relaxed.cluster.shared::cluster.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.expect_tx.sem.scope.space.b64 _, [addr], txCount; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cluster }
+   // .space     = { .shared::cluster }
+   template <typename = void>
+   __device__ static inline void mbarrier_arrive_expect_tx(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_cluster_t,
+     cuda::ptx::space_cluster_t,
+     uint64_t* addr,
+     const uint32_t& txCount);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_arrive_no_complete.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_arrive_no_complete.rst
new file mode 100644
index 00000000..9738efe3
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_arrive_no_complete.rst
@@ -0,0 +1,12 @@
+..
+   This file was automatically generated. Do not edit.
+
+mbarrier.arrive.noComplete.shared.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.arrive.noComplete.shared.b64                       state,  [addr], count;    // 5.  PTX ISA 70, SM_80
+   template <typename = void>
+   __device__ static inline uint64_t mbarrier_arrive_no_complete(
+     uint64_t* addr,
+     const uint32_t& count);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_expect_tx.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_expect_tx.rst
new file mode 100644
index 00000000..f17bc088
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_expect_tx.rst
@@ -0,0 +1,66 @@
+..
+   This file was automatically generated. Do not edit.
+
+mbarrier.expect_tx.relaxed.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.expect_tx.sem.scope.space.b64 [addr], txCount; // 1. PTX ISA 80, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   // .space     = { .shared::cta }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void mbarrier_expect_tx(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::space_shared_t,
+     uint64_t* addr,
+     uint32_t txCount);
+
+mbarrier.expect_tx.relaxed.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.expect_tx.sem.scope.space.b64 [addr], txCount; // 1. PTX ISA 80, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   // .space     = { .shared::cta }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void mbarrier_expect_tx(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::space_shared_t,
+     uint64_t* addr,
+     uint32_t txCount);
+
+mbarrier.expect_tx.relaxed.cta.shared::cluster.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.expect_tx.sem.scope.space.b64 [addr], txCount; // 2. PTX ISA 80, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   // .space     = { .shared::cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void mbarrier_expect_tx(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::space_cluster_t,
+     uint64_t* addr,
+     uint32_t txCount);
+
+mbarrier.expect_tx.relaxed.cluster.shared::cluster.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.expect_tx.sem.scope.space.b64 [addr], txCount; // 2. PTX ISA 80, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   // .space     = { .shared::cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline void mbarrier_expect_tx(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::space_cluster_t,
+     uint64_t* addr,
+     uint32_t txCount);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_init.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_init.rst
new file mode 100644
index 00000000..a210e5b9
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_init.rst
@@ -0,0 +1,12 @@
+..
+   This file was automatically generated. Do not edit.
+
+mbarrier.init.shared.b64
+^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.init.shared.b64 [addr], count; // PTX ISA 70, SM_80
+   template <typename = void>
+   __device__ static inline void mbarrier_init(
+     uint64_t* addr,
+     const uint32_t& count);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_inval.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_inval.rst
new file mode 100644
index 00000000..61fa9e3b
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_inval.rst
@@ -0,0 +1,11 @@
+..
+   This file was automatically generated. Do not edit.
+
+mbarrier.inval.shared.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.inval.shared.b64 [addr]; // PTX ISA 70, SM_80
+   template <typename = void>
+   __device__ static inline void mbarrier_inval(
+     uint64_t* addr);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_test_wait.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_test_wait.rst
new file mode 100644
index 00000000..14c06c04
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_test_wait.rst
@@ -0,0 +1,68 @@
+..
+   This file was automatically generated. Do not edit.
+
+mbarrier.test_wait.shared.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.test_wait.shared.b64 waitComplete, [addr], state;                                                  // 1.  PTX ISA 70, SM_80
+   template <typename = void>
+   __device__ static inline bool mbarrier_test_wait(
+     uint64_t* addr,
+     const uint64_t& state);
+
+mbarrier.test_wait.acquire.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.test_wait.sem.scope.shared::cta.b64        waitComplete, [addr], state;                        // 2.   PTX ISA 80, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_test_wait(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint64_t& state);
+
+mbarrier.test_wait.acquire.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.test_wait.sem.scope.shared::cta.b64        waitComplete, [addr], state;                        // 2.   PTX ISA 80, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_test_wait(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint64_t& state);
+
+mbarrier.test_wait.relaxed.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.test_wait.sem.scope.shared::cta.b64 waitComplete, [addr], state; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_test_wait(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint64_t& state);
+
+mbarrier.test_wait.relaxed.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.test_wait.sem.scope.shared::cta.b64 waitComplete, [addr], state; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_test_wait(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint64_t& state);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_test_wait_parity.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_test_wait_parity.rst
new file mode 100644
index 00000000..bf9e3eef
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_test_wait_parity.rst
@@ -0,0 +1,68 @@
+..
+   This file was automatically generated. Do not edit.
+
+mbarrier.test_wait.parity.shared.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.test_wait.parity.shared.b64 waitComplete, [addr], phaseParity;                                     // 3.  PTX ISA 71, SM_80
+   template <typename = void>
+   __device__ static inline bool mbarrier_test_wait_parity(
+     uint64_t* addr,
+     const uint32_t& phaseParity);
+
+mbarrier.test_wait.parity.acquire.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.test_wait.parity.sem.scope.shared::cta.b64 waitComplete, [addr], phaseParity;                  // 4.  PTX ISA 80, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_test_wait_parity(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint32_t& phaseParity);
+
+mbarrier.test_wait.parity.acquire.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.test_wait.parity.sem.scope.shared::cta.b64 waitComplete, [addr], phaseParity;                  // 4.  PTX ISA 80, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_test_wait_parity(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint32_t& phaseParity);
+
+mbarrier.test_wait.parity.relaxed.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.test_wait.parity.sem.scope.shared::cta.b64 waitComplete, [addr], phaseParity; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_test_wait_parity(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint32_t& phaseParity);
+
+mbarrier.test_wait.parity.relaxed.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.test_wait.parity.sem.scope.shared::cta.b64 waitComplete, [addr], phaseParity; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_test_wait_parity(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint32_t& phaseParity);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_try_wait.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_try_wait.rst
new file mode 100644
index 00000000..c37e6eff
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_try_wait.rst
@@ -0,0 +1,139 @@
+..
+   This file was automatically generated. Do not edit.
+
+mbarrier.try_wait.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.shared::cta.b64         waitComplete, [addr], state;                                      // 5a.  PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline bool mbarrier_try_wait(
+     uint64_t* addr,
+     const uint64_t& state);
+
+mbarrier.try_wait.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.shared::cta.b64         waitComplete, [addr], state, suspendTimeHint;                    // 5b.  PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline bool mbarrier_try_wait(
+     uint64_t* addr,
+     const uint64_t& state,
+     const uint32_t& suspendTimeHint);
+
+mbarrier.try_wait.acquire.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.sem.scope.shared::cta.b64         waitComplete, [addr], state;                        // 6a.  PTX ISA 80, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_try_wait(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint64_t& state);
+
+mbarrier.try_wait.acquire.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.sem.scope.shared::cta.b64         waitComplete, [addr], state;                        // 6a.  PTX ISA 80, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_try_wait(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint64_t& state);
+
+mbarrier.try_wait.acquire.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.sem.scope.shared::cta.b64         waitComplete, [addr], state , suspendTimeHint;      // 6b.  PTX ISA 80, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_try_wait(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint64_t& state,
+     const uint32_t& suspendTimeHint);
+
+mbarrier.try_wait.acquire.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.sem.scope.shared::cta.b64         waitComplete, [addr], state , suspendTimeHint;      // 6b.  PTX ISA 80, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_try_wait(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint64_t& state,
+     const uint32_t& suspendTimeHint);
+
+mbarrier.try_wait.relaxed.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.sem.scope.shared::cta.b64 waitComplete, [addr], state, suspendTimeHint; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_try_wait(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint64_t& state,
+     const uint32_t& suspendTimeHint);
+
+mbarrier.try_wait.relaxed.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.sem.scope.shared::cta.b64 waitComplete, [addr], state, suspendTimeHint; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_try_wait(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint64_t& state,
+     const uint32_t& suspendTimeHint);
+
+mbarrier.try_wait.relaxed.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.sem.scope.shared::cta.b64 waitComplete, [addr], state; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_try_wait(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint64_t& state);
+
+mbarrier.try_wait.relaxed.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.sem.scope.shared::cta.b64 waitComplete, [addr], state; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_try_wait(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint64_t& state);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_try_wait_parity.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_try_wait_parity.rst
new file mode 100644
index 00000000..6c8c9ba9
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/mbarrier_try_wait_parity.rst
@@ -0,0 +1,139 @@
+..
+   This file was automatically generated. Do not edit.
+
+mbarrier.try_wait.parity.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.parity.shared::cta.b64  waitComplete, [addr], phaseParity;                                // 7a.  PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline bool mbarrier_try_wait_parity(
+     uint64_t* addr,
+     const uint32_t& phaseParity);
+
+mbarrier.try_wait.parity.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.parity.shared::cta.b64  waitComplete, [addr], phaseParity, suspendTimeHint;               // 7b.  PTX ISA 78, SM_90
+   template <typename = void>
+   __device__ static inline bool mbarrier_try_wait_parity(
+     uint64_t* addr,
+     const uint32_t& phaseParity,
+     const uint32_t& suspendTimeHint);
+
+mbarrier.try_wait.parity.acquire.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.parity.sem.scope.shared::cta.b64  waitComplete, [addr], phaseParity;                  // 8a.  PTX ISA 80, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_try_wait_parity(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint32_t& phaseParity);
+
+mbarrier.try_wait.parity.acquire.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.parity.sem.scope.shared::cta.b64  waitComplete, [addr], phaseParity;                  // 8a.  PTX ISA 80, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_try_wait_parity(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint32_t& phaseParity);
+
+mbarrier.try_wait.parity.acquire.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.parity.sem.scope.shared::cta.b64  waitComplete, [addr], phaseParity, suspendTimeHint; // 8b.  PTX ISA 80, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_try_wait_parity(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint32_t& phaseParity,
+     const uint32_t& suspendTimeHint);
+
+mbarrier.try_wait.parity.acquire.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.parity.sem.scope.shared::cta.b64  waitComplete, [addr], phaseParity, suspendTimeHint; // 8b.  PTX ISA 80, SM_90
+   // .sem       = { .acquire }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_try_wait_parity(
+     cuda::ptx::sem_acquire_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint32_t& phaseParity,
+     const uint32_t& suspendTimeHint);
+
+mbarrier.try_wait.parity.relaxed.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.parity.sem.scope.shared::cta.b64 waitComplete, [addr], phaseParity, suspendTimeHint; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_try_wait_parity(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint32_t& phaseParity,
+     const uint32_t& suspendTimeHint);
+
+mbarrier.try_wait.parity.relaxed.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.parity.sem.scope.shared::cta.b64 waitComplete, [addr], phaseParity, suspendTimeHint; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_try_wait_parity(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint32_t& phaseParity,
+     const uint32_t& suspendTimeHint);
+
+mbarrier.try_wait.parity.relaxed.cta.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.parity.sem.scope.shared::cta.b64 waitComplete, [addr], phaseParity; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_try_wait_parity(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint32_t& phaseParity);
+
+mbarrier.try_wait.parity.relaxed.cluster.shared::cta.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // mbarrier.try_wait.parity.sem.scope.shared::cta.b64 waitComplete, [addr], phaseParity; // PTX ISA 86, SM_90
+   // .sem       = { .relaxed }
+   // .scope     = { .cta, .cluster }
+   template <cuda::ptx::dot_scope Scope>
+   __device__ static inline bool mbarrier_try_wait_parity(
+     cuda::ptx::sem_relaxed_t,
+     cuda::ptx::scope_t<Scope> scope,
+     uint64_t* addr,
+     const uint32_t& phaseParity);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/multimem_ld_reduce.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/multimem_ld_reduce.rst
new file mode 100644
index 00000000..1f4775a5
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/multimem_ld_reduce.rst
@@ -0,0 +1,2396 @@
+..
+   This file was automatically generated. Do not edit.
+
+multimem.ld_reduce.weak.global.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   // .op        = { .min }
+   template <typename = void>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_weak_t,
+     cuda::ptx::op_min_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.relaxed.cta.global.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.relaxed.cluster.global.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.relaxed.gpu.global.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.relaxed.sys.global.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.acquire.cta.global.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.acquire.cluster.global.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.acquire.gpu.global.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.acquire.sys.global.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.weak.global.min.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   // .op        = { .min }
+   template <typename = void>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_weak_t,
+     cuda::ptx::op_min_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.relaxed.cta.global.min.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.relaxed.cluster.global.min.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.relaxed.gpu.global.min.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.relaxed.sys.global.min.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.acquire.cta.global.min.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.acquire.cluster.global.min.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.acquire.gpu.global.min.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.acquire.sys.global.min.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.weak.global.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   // .op        = { .min }
+   template <typename = void>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_weak_t,
+     cuda::ptx::op_min_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.relaxed.cta.global.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.relaxed.cluster.global.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.relaxed.gpu.global.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.relaxed.sys.global.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.acquire.cta.global.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.acquire.cluster.global.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.acquire.gpu.global.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.acquire.sys.global.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.weak.global.min.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.global.op.s64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   // .op        = { .min }
+   template <typename = void>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_weak_t,
+     cuda::ptx::op_min_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.relaxed.cta.global.min.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.relaxed.cluster.global.min.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.relaxed.gpu.global.min.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.relaxed.sys.global.min.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.acquire.cta.global.min.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.acquire.cluster.global.min.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.acquire.gpu.global.min.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.acquire.sys.global.min.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.weak.global.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   // .op        = { .max }
+   template <typename = void>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_weak_t,
+     cuda::ptx::op_max_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.relaxed.cta.global.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.relaxed.cluster.global.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.relaxed.gpu.global.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.relaxed.sys.global.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.acquire.cta.global.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.acquire.cluster.global.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.acquire.gpu.global.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.acquire.sys.global.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.weak.global.max.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   // .op        = { .max }
+   template <typename = void>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_weak_t,
+     cuda::ptx::op_max_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.relaxed.cta.global.max.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.relaxed.cluster.global.max.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.relaxed.gpu.global.max.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.relaxed.sys.global.max.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.acquire.cta.global.max.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.acquire.cluster.global.max.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.acquire.gpu.global.max.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.acquire.sys.global.max.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.weak.global.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   // .op        = { .max }
+   template <typename = void>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_weak_t,
+     cuda::ptx::op_max_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.relaxed.cta.global.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.relaxed.cluster.global.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.relaxed.gpu.global.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.relaxed.sys.global.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.acquire.cta.global.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.acquire.cluster.global.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.acquire.gpu.global.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.acquire.sys.global.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.weak.global.max.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.global.op.s64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   // .op        = { .max }
+   template <typename = void>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_weak_t,
+     cuda::ptx::op_max_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.relaxed.cta.global.max.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.relaxed.cluster.global.max.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.relaxed.gpu.global.max.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.relaxed.sys.global.max.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.acquire.cta.global.max.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.acquire.cluster.global.max.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.acquire.gpu.global.max.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.acquire.sys.global.max.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.weak.global.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_weak_t,
+     cuda::ptx::op_add_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.relaxed.cta.global.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.relaxed.cluster.global.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.relaxed.gpu.global.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.relaxed.sys.global.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.acquire.cta.global.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.acquire.cluster.global.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.acquire.gpu.global.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.acquire.sys.global.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const uint32_t* addr);
+
+multimem.ld_reduce.weak.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_weak_t,
+     cuda::ptx::op_add_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.relaxed.cta.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.relaxed.cluster.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.relaxed.gpu.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.relaxed.sys.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.acquire.cta.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.acquire.cluster.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.acquire.gpu.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.acquire.sys.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline uint64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const uint64_t* addr);
+
+multimem.ld_reduce.weak.global.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_weak_t,
+     cuda::ptx::op_add_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.relaxed.cta.global.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.relaxed.cluster.global.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.relaxed.gpu.global.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.relaxed.sys.global.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.acquire.cta.global.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.acquire.cluster.global.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.acquire.gpu.global.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.acquire.sys.global.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.s32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int32_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const int32_t* addr);
+
+multimem.ld_reduce.weak.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_weak_t,
+     cuda::ptx::op_add_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.relaxed.cta.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.relaxed.cluster.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.relaxed.gpu.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.relaxed.sys.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.acquire.cta.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.acquire.cluster.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.acquire.gpu.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.acquire.sys.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.u64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline int64_t multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     const int64_t* addr);
+
+multimem.ld_reduce.weak.global.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_weak_t,
+     cuda::ptx::op_and_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.relaxed.cta.global.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.relaxed.cluster.global.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.relaxed.gpu.global.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.relaxed.sys.global.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.acquire.cta.global.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.acquire.cluster.global.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.acquire.gpu.global.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.acquire.sys.global.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.weak.global.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_weak_t,
+     cuda::ptx::op_or_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.relaxed.cta.global.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.relaxed.cluster.global.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.relaxed.gpu.global.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.relaxed.sys.global.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.acquire.cta.global.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.acquire.cluster.global.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.acquire.gpu.global.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.acquire.sys.global.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.weak.global.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_weak_t,
+     cuda::ptx::op_xor_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.relaxed.cta.global.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.relaxed.cluster.global.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.relaxed.gpu.global.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.relaxed.sys.global.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.acquire.cta.global.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.acquire.cluster.global.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.acquire.gpu.global.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.acquire.sys.global.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b32 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B32 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     const B32* addr);
+
+multimem.ld_reduce.weak.global.and.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   // .op        = { .and }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_weak_t,
+     cuda::ptx::op_and_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.relaxed.cta.global.and.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.relaxed.cluster.global.and.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.relaxed.gpu.global.and.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.relaxed.sys.global.and.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.acquire.cta.global.and.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.acquire.cluster.global.and.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.acquire.gpu.global.and.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.acquire.sys.global.and.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.weak.global.or.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   // .op        = { .or }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_weak_t,
+     cuda::ptx::op_or_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.relaxed.cta.global.or.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.relaxed.cluster.global.or.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.relaxed.gpu.global.or.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.relaxed.sys.global.or.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.acquire.cta.global.or.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.acquire.cluster.global.or.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.acquire.gpu.global.or.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.acquire.sys.global.or.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.weak.global.xor.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   // .op        = { .xor }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_weak_t,
+     cuda::ptx::op_xor_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.relaxed.cta.global.xor.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.relaxed.cluster.global.xor.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.relaxed.gpu.global.xor.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.relaxed.sys.global.xor.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.acquire.cta.global.xor.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.acquire.cluster.global.xor.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.acquire.gpu.global.xor.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     const B64* addr);
+
+multimem.ld_reduce.acquire.sys.global.xor.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.ld_reduce.sem.scope.global.op.b64 dest, [addr]; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .acquire }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline B64 multimem_ld_reduce(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     const B64* addr);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/multimem_red.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/multimem_red.rst
new file mode 100644
index 00000000..0036ea1d
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/multimem_red.rst
@@ -0,0 +1,2306 @@
+..
+   This file was automatically generated. Do not edit.
+
+multimem.red.relaxed.cta.global.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.relaxed.cluster.global.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.relaxed.gpu.global.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.relaxed.sys.global.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.release.cta.global.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.release.cluster.global.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.release.gpu.global.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.release.sys.global.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.relaxed.cta.global.min.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.relaxed.cluster.global.min.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.relaxed.gpu.global.min.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.relaxed.sys.global.min.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.release.cta.global.min.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.release.cluster.global.min.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.release.gpu.global.min.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.release.sys.global.min.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.relaxed.cta.global.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.relaxed.cluster.global.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.relaxed.gpu.global.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.relaxed.sys.global.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.release.cta.global.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.release.cluster.global.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.release.gpu.global.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.release.sys.global.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.relaxed.cta.global.min.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.relaxed.cluster.global.min.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.relaxed.gpu.global.min.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.relaxed.sys.global.min.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.release.cta.global.min.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.release.cluster.global.min.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.release.gpu.global.min.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.release.sys.global.min.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .min }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_min_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.relaxed.cta.global.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.relaxed.cluster.global.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.relaxed.gpu.global.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.relaxed.sys.global.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.release.cta.global.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.release.cluster.global.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.release.gpu.global.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.release.sys.global.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.relaxed.cta.global.max.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.relaxed.cluster.global.max.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.relaxed.gpu.global.max.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.relaxed.sys.global.max.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.release.cta.global.max.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.release.cluster.global.max.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.release.gpu.global.max.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.release.sys.global.max.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.relaxed.cta.global.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.relaxed.cluster.global.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.relaxed.gpu.global.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.relaxed.sys.global.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.release.cta.global.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.release.cluster.global.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.release.gpu.global.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.release.sys.global.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.relaxed.cta.global.max.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.relaxed.cluster.global.max.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.relaxed.gpu.global.max.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.relaxed.sys.global.max.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.release.cta.global.max.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.release.cluster.global.max.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.release.gpu.global.max.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.release.sys.global.max.s64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .max }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_max_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.relaxed.cta.global.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.relaxed.cluster.global.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.relaxed.gpu.global.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.relaxed.sys.global.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.release.cta.global.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.release.cluster.global.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.release.gpu.global.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.release.sys.global.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     uint32_t* addr,
+     uint32_t val);
+
+multimem.red.relaxed.cta.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.relaxed.cluster.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.relaxed.gpu.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.relaxed.sys.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.release.cta.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.release.cluster.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.release.gpu.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.release.sys.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     uint64_t* addr,
+     uint64_t val);
+
+multimem.red.relaxed.cta.global.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.relaxed.cluster.global.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.relaxed.gpu.global.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.relaxed.sys.global.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.release.cta.global.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.release.cluster.global.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.release.gpu.global.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.release.sys.global.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.s32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     int32_t* addr,
+     int32_t val);
+
+multimem.red.relaxed.cta.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.relaxed.cluster.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.relaxed.gpu.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.relaxed.sys.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.release.cta.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.release.cluster.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.release.gpu.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.release.sys.global.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.u64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .add }
+   template <cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_add_t,
+     int64_t* addr,
+     int64_t val);
+
+multimem.red.relaxed.cta.global.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.relaxed.cluster.global.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.relaxed.gpu.global.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.relaxed.sys.global.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.release.cta.global.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.release.cluster.global.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.release.gpu.global.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.release.sys.global.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.relaxed.cta.global.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.relaxed.cluster.global.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.relaxed.gpu.global.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.relaxed.sys.global.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.release.cta.global.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.release.cluster.global.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.release.gpu.global.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.release.sys.global.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.relaxed.cta.global.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.relaxed.cluster.global.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.relaxed.gpu.global.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.relaxed.sys.global.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.release.cta.global.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.release.cluster.global.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.release.gpu.global.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.release.sys.global.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     B32* addr,
+     B32 val);
+
+multimem.red.relaxed.cta.global.and.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.relaxed.cluster.global.and.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.relaxed.gpu.global.and.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.relaxed.sys.global.and.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.release.cta.global.and.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.release.cluster.global.and.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.release.gpu.global.and.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.release.sys.global.and.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .and }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_and_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.relaxed.cta.global.or.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.relaxed.cluster.global.or.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.relaxed.gpu.global.or.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.relaxed.sys.global.or.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.release.cta.global.or.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.release.cluster.global.or.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.release.gpu.global.or.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.release.sys.global.or.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .or }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_or_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.relaxed.cta.global.xor.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.relaxed.cluster.global.xor.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.relaxed.gpu.global.xor.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.relaxed.sys.global.xor.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.release.cta.global.xor.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.release.cluster.global.xor.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.release.gpu.global.xor.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     B64* addr,
+     B64 val);
+
+multimem.red.release.sys.global.xor.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.red.sem.scope.global.op.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   // .op        = { .xor }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_red(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     cuda::ptx::op_xor_op_t,
+     B64* addr,
+     B64 val);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/multimem_st.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/multimem_st.rst
new file mode 100644
index 00000000..f004c896
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/multimem_st.rst
@@ -0,0 +1,250 @@
+..
+   This file was automatically generated. Do not edit.
+
+multimem.st.weak.global.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.st.sem.global.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void multimem_st(
+     cuda::ptx::sem_weak_t,
+     B32* addr,
+     B32 val);
+
+multimem.st.relaxed.cta.global.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.st.sem.scope.global.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_st(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     B32* addr,
+     B32 val);
+
+multimem.st.relaxed.cluster.global.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.st.sem.scope.global.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_st(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     B32* addr,
+     B32 val);
+
+multimem.st.relaxed.gpu.global.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.st.sem.scope.global.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_st(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     B32* addr,
+     B32 val);
+
+multimem.st.relaxed.sys.global.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.st.sem.scope.global.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_st(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     B32* addr,
+     B32 val);
+
+multimem.st.release.cta.global.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.st.sem.scope.global.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_st(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     B32* addr,
+     B32 val);
+
+multimem.st.release.cluster.global.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.st.sem.scope.global.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_st(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     B32* addr,
+     B32 val);
+
+multimem.st.release.gpu.global.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.st.sem.scope.global.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_st(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     B32* addr,
+     B32 val);
+
+multimem.st.release.sys.global.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.st.sem.scope.global.b32 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_st(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     B32* addr,
+     B32 val);
+
+multimem.st.weak.global.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.st.sem.global.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .weak }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline void multimem_st(
+     cuda::ptx::sem_weak_t,
+     B64* addr,
+     B64 val);
+
+multimem.st.relaxed.cta.global.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.st.sem.scope.global.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_st(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     B64* addr,
+     B64 val);
+
+multimem.st.relaxed.cluster.global.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.st.sem.scope.global.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_st(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     B64* addr,
+     B64 val);
+
+multimem.st.relaxed.gpu.global.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.st.sem.scope.global.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_st(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     B64* addr,
+     B64 val);
+
+multimem.st.relaxed.sys.global.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.st.sem.scope.global.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_st(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     B64* addr,
+     B64 val);
+
+multimem.st.release.cta.global.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.st.sem.scope.global.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_st(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     B64* addr,
+     B64 val);
+
+multimem.st.release.cluster.global.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.st.sem.scope.global.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_st(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     B64* addr,
+     B64 val);
+
+multimem.st.release.gpu.global.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.st.sem.scope.global.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_st(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     B64* addr,
+     B64 val);
+
+multimem.st.release.sys.global.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // multimem.st.sem.scope.global.b64 [addr], val; // PTX ISA 81, SM_90
+   // .sem       = { .relaxed, .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true, cuda::ptx::dot_sem Sem, cuda::ptx::dot_scope Scope>
+   __device__ static inline void multimem_st(
+     cuda::ptx::sem_t<Sem> sem,
+     cuda::ptx::scope_t<Scope> scope,
+     B64* addr,
+     B64 val);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/prmt.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/prmt.rst
new file mode 100644
index 00000000..616eda56
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/prmt.rst
@@ -0,0 +1,79 @@
+..
+   This file was automatically generated. Do not edit.
+
+prmt.b32
+^^^^^^^^
+.. code-block:: cuda
+
+   // prmt.b32 dest, a_reg, b_reg, c_reg; // PTX ISA 20, SM_50
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline uint32_t prmt(
+     B32 a_reg,
+     B32 b_reg,
+     uint32_t c_reg);
+
+prmt.b32.f4e
+^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // prmt.b32.f4e dest, a_reg, b_reg, c_reg; // PTX ISA 20, SM_50
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline uint32_t prmt_f4e(
+     B32 a_reg,
+     B32 b_reg,
+     uint32_t c_reg);
+
+prmt.b32.b4e
+^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // prmt.b32.b4e dest, a_reg, b_reg, c_reg; // PTX ISA 20, SM_50
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline uint32_t prmt_b4e(
+     B32 a_reg,
+     B32 b_reg,
+     uint32_t c_reg);
+
+prmt.b32.rc8
+^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // prmt.b32.rc8 dest, a_reg, b_reg, c_reg; // PTX ISA 20, SM_50
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline uint32_t prmt_rc8(
+     B32 a_reg,
+     B32 b_reg,
+     uint32_t c_reg);
+
+prmt.b32.ecl
+^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // prmt.b32.ecl dest, a_reg, b_reg, c_reg; // PTX ISA 20, SM_50
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline uint32_t prmt_ecl(
+     B32 a_reg,
+     B32 b_reg,
+     uint32_t c_reg);
+
+prmt.b32.ecr
+^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // prmt.b32.ecr dest, a_reg, b_reg, c_reg; // PTX ISA 20, SM_50
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline uint32_t prmt_ecr(
+     B32 a_reg,
+     B32 b_reg,
+     uint32_t c_reg);
+
+prmt.b32.rc16
+^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // prmt.b32.rc16 dest, a_reg, b_reg, c_reg; // PTX ISA 20, SM_50
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline uint32_t prmt_rc16(
+     B32 a_reg,
+     B32 b_reg,
+     uint32_t c_reg);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/red_async.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/red_async.rst
new file mode 100644
index 00000000..b8546046
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/red_async.rst
@@ -0,0 +1,183 @@
+..
+   This file was automatically generated. Do not edit.
+
+red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.inc.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.op.type  [dest], value, [remote_bar];  // PTX ISA 81, SM_90
+   // .type      = { .u32 }
+   // .op        = { .inc }
+   template <typename = void>
+   __device__ static inline void red_async(
+     cuda::ptx::op_inc_t,
+     uint32_t* dest,
+     const uint32_t& value,
+     uint64_t* remote_bar);
+
+red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.dec.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.op.type  [dest], value, [remote_bar];  // PTX ISA 81, SM_90
+   // .type      = { .u32 }
+   // .op        = { .dec }
+   template <typename = void>
+   __device__ static inline void red_async(
+     cuda::ptx::op_dec_t,
+     uint32_t* dest,
+     const uint32_t& value,
+     uint64_t* remote_bar);
+
+red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.min.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.op.type  [dest], value, [remote_bar];  // PTX ISA 81, SM_90
+   // .type      = { .u32 }
+   // .op        = { .min }
+   template <typename = void>
+   __device__ static inline void red_async(
+     cuda::ptx::op_min_t,
+     uint32_t* dest,
+     const uint32_t& value,
+     uint64_t* remote_bar);
+
+red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.max.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.op.type  [dest], value, [remote_bar];  // PTX ISA 81, SM_90
+   // .type      = { .u32 }
+   // .op        = { .max }
+   template <typename = void>
+   __device__ static inline void red_async(
+     cuda::ptx::op_max_t,
+     uint32_t* dest,
+     const uint32_t& value,
+     uint64_t* remote_bar);
+
+red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.add.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.op.type  [dest], value, [remote_bar];  // PTX ISA 81, SM_90
+   // .type      = { .u32 }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline void red_async(
+     cuda::ptx::op_add_t,
+     uint32_t* dest,
+     const uint32_t& value,
+     uint64_t* remote_bar);
+
+red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.min.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.op.type  [dest], value, [remote_bar];  // PTX ISA 81, SM_90
+   // .type      = { .s32 }
+   // .op        = { .min }
+   template <typename = void>
+   __device__ static inline void red_async(
+     cuda::ptx::op_min_t,
+     int32_t* dest,
+     const int32_t& value,
+     uint64_t* remote_bar);
+
+red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.max.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.op.type  [dest], value, [remote_bar];  // PTX ISA 81, SM_90
+   // .type      = { .s32 }
+   // .op        = { .max }
+   template <typename = void>
+   __device__ static inline void red_async(
+     cuda::ptx::op_max_t,
+     int32_t* dest,
+     const int32_t& value,
+     uint64_t* remote_bar);
+
+red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.add.s32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.op.type  [dest], value, [remote_bar];  // PTX ISA 81, SM_90
+   // .type      = { .s32 }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline void red_async(
+     cuda::ptx::op_add_t,
+     int32_t* dest,
+     const int32_t& value,
+     uint64_t* remote_bar);
+
+red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.and.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.op.type  [dest], value, [remote_bar];  // PTX ISA 81, SM_90
+   // .type      = { .b32 }
+   // .op        = { .and }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void red_async(
+     cuda::ptx::op_and_op_t,
+     B32* dest,
+     const B32& value,
+     uint64_t* remote_bar);
+
+red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.or.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.op.type  [dest], value, [remote_bar];  // PTX ISA 81, SM_90
+   // .type      = { .b32 }
+   // .op        = { .or }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void red_async(
+     cuda::ptx::op_or_op_t,
+     B32* dest,
+     const B32& value,
+     uint64_t* remote_bar);
+
+red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.xor.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.op.type  [dest], value, [remote_bar];  // PTX ISA 81, SM_90
+   // .type      = { .b32 }
+   // .op        = { .xor }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void red_async(
+     cuda::ptx::op_xor_op_t,
+     B32* dest,
+     const B32& value,
+     uint64_t* remote_bar);
+
+red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.op.type  [dest], value, [remote_bar];  // PTX ISA 81, SM_90
+   // .type      = { .u64 }
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline void red_async(
+     cuda::ptx::op_add_t,
+     uint64_t* dest,
+     const uint64_t& value,
+     uint64_t* remote_bar);
+
+red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.add.u64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes.op.u64  [dest], value, [remote_bar]; // .u64 intentional PTX ISA 81, SM_90
+   // .op        = { .add }
+   template <typename = void>
+   __device__ static inline void red_async(
+     cuda::ptx::op_add_t,
+     int64_t* dest,
+     const int64_t& value,
+     int64_t* remote_bar);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/setmaxnreg.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/setmaxnreg.rst
new file mode 100644
index 00000000..5da76caf
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/setmaxnreg.rst
@@ -0,0 +1,20 @@
+..
+   This file was automatically generated. Do not edit.
+
+setmaxnreg.inc.sync.aligned.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // setmaxnreg.inc.sync.aligned.u32 imm_reg_count; // PTX ISA 80, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   template <int N32>
+   __device__ static inline void setmaxnreg_inc(
+     cuda::ptx::n32_t<N32> imm_reg_count);
+
+setmaxnreg.dec.sync.aligned.u32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // setmaxnreg.dec.sync.aligned.u32 imm_reg_count; // PTX ISA 80, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   template <int N32>
+   __device__ static inline void setmaxnreg_dec(
+     cuda::ptx::n32_t<N32> imm_reg_count);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/shl.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/shl.rst
new file mode 100644
index 00000000..c8a6faf5
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/shl.rst
@@ -0,0 +1,32 @@
+..
+   This file was automatically generated. Do not edit.
+
+shl.b16
+^^^^^^^
+.. code-block:: cuda
+
+   // shl.b16 dest, a_reg, b_reg; // PTX ISA 10, SM_50
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 shl(
+     B16 a_reg,
+     uint32_t b_reg);
+
+shl.b32
+^^^^^^^
+.. code-block:: cuda
+
+   // shl.b32 dest, a_reg, b_reg; // PTX ISA 10, SM_50
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline B32 shl(
+     B32 a_reg,
+     uint32_t b_reg);
+
+shl.b64
+^^^^^^^
+.. code-block:: cuda
+
+   // shl.b64 dest, a_reg, b_reg; // PTX ISA 10, SM_50
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline B64 shl(
+     B64 a_reg,
+     uint32_t b_reg);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/shr.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/shr.rst
new file mode 100644
index 00000000..6432cffa
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/shr.rst
@@ -0,0 +1,62 @@
+..
+   This file was automatically generated. Do not edit.
+
+shr.b16
+^^^^^^^
+.. code-block:: cuda
+
+   // shr.b16 dest, a_reg, b_reg; // PTX ISA 10, SM_50
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline B16 shr(
+     B16 a_reg,
+     uint32_t b_reg);
+
+shr.b32
+^^^^^^^
+.. code-block:: cuda
+
+   // shr.b32 dest, a_reg, b_reg; // PTX ISA 10, SM_50
+   template <typename B32, enable_if_t<sizeof(B32) == 4 && !(is_integral_v<B32> && is_signed_v<B32>), bool> = true>
+   __device__ static inline B32 shr(
+     B32 a_reg,
+     uint32_t b_reg);
+
+shr.b64
+^^^^^^^
+.. code-block:: cuda
+
+   // shr.b64 dest, a_reg, b_reg; // PTX ISA 10, SM_50
+   template <typename B64, enable_if_t<sizeof(B64) == 8 && !(is_integral_v<B64> && is_signed_v<B64>), bool> = true>
+   __device__ static inline B64 shr(
+     B64 a_reg,
+     uint32_t b_reg);
+
+shr.s16
+^^^^^^^
+.. code-block:: cuda
+
+   // shr.s16 dest, a_reg, b_reg; // PTX ISA 10, SM_50
+   template <typename = void>
+   __device__ static inline int16_t shr(
+     int16_t a_reg,
+     uint32_t b_reg);
+
+shr.s32
+^^^^^^^
+.. code-block:: cuda
+
+   // shr.s32 dest, a_reg, b_reg; // PTX ISA 10, SM_50
+   template <typename S32, enable_if_t<sizeof(S32) == 4 && is_integral_v<S32> && is_signed_v<S32>, bool> = true>
+   __device__ static inline S32 shr(
+     S32 a_reg,
+     uint32_t b_reg);
+
+shr.s64
+^^^^^^^
+.. code-block:: cuda
+
+   // shr.s64 dest, a_reg, b_reg; // PTX ISA 10, SM_50
+   template <typename S64, enable_if_t<sizeof(S64) == 8 && is_integral_v<S64> && is_signed_v<S64>, bool> = true>
+   __device__ static inline S64 shr(
+     S64 a_reg,
+     uint32_t b_reg);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/st.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/st.rst
new file mode 100644
index 00000000..431a57d9
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/st.rst
@@ -0,0 +1,602 @@
+..
+   This file was automatically generated. Do not edit.
+
+st.global.b8
+^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.b8 [addr], src; // PTX ISA 10, SM_50
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline void st(
+     cuda::ptx::space_global_t,
+     B8* addr,
+     B8 src);
+
+st.global.b16
+^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.b16 [addr], src; // PTX ISA 10, SM_50
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline void st(
+     cuda::ptx::space_global_t,
+     B16* addr,
+     B16 src);
+
+st.global.b32
+^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.b32 [addr], src; // PTX ISA 10, SM_50
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void st(
+     cuda::ptx::space_global_t,
+     B32* addr,
+     B32 src);
+
+st.global.b64
+^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.b64 [addr], src; // PTX ISA 10, SM_50
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline void st(
+     cuda::ptx::space_global_t,
+     B64* addr,
+     B64 src);
+
+st.global.b128
+^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.b128 [addr], src; // PTX ISA 83, SM_70
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline void st(
+     cuda::ptx::space_global_t,
+     B128* addr,
+     B128 src);
+
+st.global.v4.b64
+^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.v4.b64 [addr], src; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline void st(
+     cuda::ptx::space_global_t,
+     B256* addr,
+     B256 src);
+
+st.global.L2::cache_hint.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L2::cache_hint.b8 [addr], src, cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline void st_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B8* addr,
+     B8 src,
+     uint64_t cache_policy);
+
+st.global.L2::cache_hint.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L2::cache_hint.b16 [addr], src, cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline void st_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B16* addr,
+     B16 src,
+     uint64_t cache_policy);
+
+st.global.L2::cache_hint.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L2::cache_hint.b32 [addr], src, cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void st_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B32* addr,
+     B32 src,
+     uint64_t cache_policy);
+
+st.global.L2::cache_hint.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L2::cache_hint.b64 [addr], src, cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline void st_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B64* addr,
+     B64 src,
+     uint64_t cache_policy);
+
+st.global.L2::cache_hint.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L2::cache_hint.b128 [addr], src, cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline void st_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B128* addr,
+     B128 src,
+     uint64_t cache_policy);
+
+st.global.L2::cache_hint.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L2::cache_hint.v4.b64 [addr], src, cache_policy; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline void st_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B256* addr,
+     B256 src,
+     uint64_t cache_policy);
+
+st.global.L1::evict_first.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_first.b8 [addr], src; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline void st_L1_evict_first(
+     cuda::ptx::space_global_t,
+     B8* addr,
+     B8 src);
+
+st.global.L1::evict_first.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_first.b16 [addr], src; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline void st_L1_evict_first(
+     cuda::ptx::space_global_t,
+     B16* addr,
+     B16 src);
+
+st.global.L1::evict_first.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_first.b32 [addr], src; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void st_L1_evict_first(
+     cuda::ptx::space_global_t,
+     B32* addr,
+     B32 src);
+
+st.global.L1::evict_first.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_first.b64 [addr], src; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline void st_L1_evict_first(
+     cuda::ptx::space_global_t,
+     B64* addr,
+     B64 src);
+
+st.global.L1::evict_first.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_first.b128 [addr], src; // PTX ISA 83, SM_70
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline void st_L1_evict_first(
+     cuda::ptx::space_global_t,
+     B128* addr,
+     B128 src);
+
+st.global.L1::evict_first.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_first.v4.b64 [addr], src; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline void st_L1_evict_first(
+     cuda::ptx::space_global_t,
+     B256* addr,
+     B256 src);
+
+st.global.L1::evict_first.L2::cache_hint.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_first.L2::cache_hint.b8 [addr], src, cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline void st_L1_evict_first_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B8* addr,
+     B8 src,
+     uint64_t cache_policy);
+
+st.global.L1::evict_first.L2::cache_hint.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_first.L2::cache_hint.b16 [addr], src, cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline void st_L1_evict_first_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B16* addr,
+     B16 src,
+     uint64_t cache_policy);
+
+st.global.L1::evict_first.L2::cache_hint.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_first.L2::cache_hint.b32 [addr], src, cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void st_L1_evict_first_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B32* addr,
+     B32 src,
+     uint64_t cache_policy);
+
+st.global.L1::evict_first.L2::cache_hint.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_first.L2::cache_hint.b64 [addr], src, cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline void st_L1_evict_first_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B64* addr,
+     B64 src,
+     uint64_t cache_policy);
+
+st.global.L1::evict_first.L2::cache_hint.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_first.L2::cache_hint.b128 [addr], src, cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline void st_L1_evict_first_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B128* addr,
+     B128 src,
+     uint64_t cache_policy);
+
+st.global.L1::evict_first.L2::cache_hint.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_first.L2::cache_hint.v4.b64 [addr], src, cache_policy; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline void st_L1_evict_first_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B256* addr,
+     B256 src,
+     uint64_t cache_policy);
+
+st.global.L1::evict_last.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_last.b8 [addr], src; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline void st_L1_evict_last(
+     cuda::ptx::space_global_t,
+     B8* addr,
+     B8 src);
+
+st.global.L1::evict_last.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_last.b16 [addr], src; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline void st_L1_evict_last(
+     cuda::ptx::space_global_t,
+     B16* addr,
+     B16 src);
+
+st.global.L1::evict_last.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_last.b32 [addr], src; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void st_L1_evict_last(
+     cuda::ptx::space_global_t,
+     B32* addr,
+     B32 src);
+
+st.global.L1::evict_last.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_last.b64 [addr], src; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline void st_L1_evict_last(
+     cuda::ptx::space_global_t,
+     B64* addr,
+     B64 src);
+
+st.global.L1::evict_last.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_last.b128 [addr], src; // PTX ISA 83, SM_70
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline void st_L1_evict_last(
+     cuda::ptx::space_global_t,
+     B128* addr,
+     B128 src);
+
+st.global.L1::evict_last.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_last.v4.b64 [addr], src; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline void st_L1_evict_last(
+     cuda::ptx::space_global_t,
+     B256* addr,
+     B256 src);
+
+st.global.L1::evict_last.L2::cache_hint.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_last.L2::cache_hint.b8 [addr], src, cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline void st_L1_evict_last_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B8* addr,
+     B8 src,
+     uint64_t cache_policy);
+
+st.global.L1::evict_last.L2::cache_hint.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_last.L2::cache_hint.b16 [addr], src, cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline void st_L1_evict_last_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B16* addr,
+     B16 src,
+     uint64_t cache_policy);
+
+st.global.L1::evict_last.L2::cache_hint.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_last.L2::cache_hint.b32 [addr], src, cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void st_L1_evict_last_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B32* addr,
+     B32 src,
+     uint64_t cache_policy);
+
+st.global.L1::evict_last.L2::cache_hint.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_last.L2::cache_hint.b64 [addr], src, cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline void st_L1_evict_last_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B64* addr,
+     B64 src,
+     uint64_t cache_policy);
+
+st.global.L1::evict_last.L2::cache_hint.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_last.L2::cache_hint.b128 [addr], src, cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline void st_L1_evict_last_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B128* addr,
+     B128 src,
+     uint64_t cache_policy);
+
+st.global.L1::evict_last.L2::cache_hint.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::evict_last.L2::cache_hint.v4.b64 [addr], src, cache_policy; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline void st_L1_evict_last_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B256* addr,
+     B256 src,
+     uint64_t cache_policy);
+
+st.global.L1::no_allocate.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::no_allocate.b8 [addr], src; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline void st_L1_no_allocate(
+     cuda::ptx::space_global_t,
+     B8* addr,
+     B8 src);
+
+st.global.L1::no_allocate.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::no_allocate.b16 [addr], src; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline void st_L1_no_allocate(
+     cuda::ptx::space_global_t,
+     B16* addr,
+     B16 src);
+
+st.global.L1::no_allocate.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::no_allocate.b32 [addr], src; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void st_L1_no_allocate(
+     cuda::ptx::space_global_t,
+     B32* addr,
+     B32 src);
+
+st.global.L1::no_allocate.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::no_allocate.b64 [addr], src; // PTX ISA 74, SM_70
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline void st_L1_no_allocate(
+     cuda::ptx::space_global_t,
+     B64* addr,
+     B64 src);
+
+st.global.L1::no_allocate.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::no_allocate.b128 [addr], src; // PTX ISA 83, SM_70
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline void st_L1_no_allocate(
+     cuda::ptx::space_global_t,
+     B128* addr,
+     B128 src);
+
+st.global.L1::no_allocate.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::no_allocate.v4.b64 [addr], src; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline void st_L1_no_allocate(
+     cuda::ptx::space_global_t,
+     B256* addr,
+     B256 src);
+
+st.global.L1::no_allocate.L2::cache_hint.b8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::no_allocate.L2::cache_hint.b8 [addr], src, cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B8, enable_if_t<sizeof(B8) == 1, bool> = true>
+   __device__ static inline void st_L1_no_allocate_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B8* addr,
+     B8 src,
+     uint64_t cache_policy);
+
+st.global.L1::no_allocate.L2::cache_hint.b16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::no_allocate.L2::cache_hint.b16 [addr], src, cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B16, enable_if_t<sizeof(B16) == 2, bool> = true>
+   __device__ static inline void st_L1_no_allocate_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B16* addr,
+     B16 src,
+     uint64_t cache_policy);
+
+st.global.L1::no_allocate.L2::cache_hint.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::no_allocate.L2::cache_hint.b32 [addr], src, cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void st_L1_no_allocate_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B32* addr,
+     B32 src,
+     uint64_t cache_policy);
+
+st.global.L1::no_allocate.L2::cache_hint.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::no_allocate.L2::cache_hint.b64 [addr], src, cache_policy; // PTX ISA 74, SM_80
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline void st_L1_no_allocate_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B64* addr,
+     B64 src,
+     uint64_t cache_policy);
+
+st.global.L1::no_allocate.L2::cache_hint.b128
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::no_allocate.L2::cache_hint.b128 [addr], src, cache_policy; // PTX ISA 83, SM_80
+   // .space     = { .global }
+   template <typename B128, enable_if_t<sizeof(B128) == 16, bool> = true>
+   __device__ static inline void st_L1_no_allocate_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B128* addr,
+     B128 src,
+     uint64_t cache_policy);
+
+st.global.L1::no_allocate.L2::cache_hint.v4.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.space.L1::no_allocate.L2::cache_hint.v4.b64 [addr], src, cache_policy; // PTX ISA 88, SM_100
+   // .space     = { .global }
+   template <typename B256, enable_if_t<sizeof(B256) == 32, bool> = true>
+   __device__ static inline void st_L1_no_allocate_L2_cache_hint(
+     cuda::ptx::space_global_t,
+     B256* addr,
+     B256 src,
+     uint64_t cache_policy);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/st_async.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/st_async.rst
new file mode 100644
index 00000000..fefa14d7
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/st_async.rst
@@ -0,0 +1,61 @@
+..
+   This file was automatically generated. Do not edit.
+
+st.async.weak.shared::cluster.mbarrier::complete_tx::bytes.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.async.weak.shared::cluster.mbarrier::complete_tx::bytes.type [addr], value, [remote_bar];    // 1.  PTX ISA 81, SM_90
+   // .type      = { .b32, .b64 }
+   template <typename Type>
+   __device__ static inline void st_async(
+     Type* addr,
+     const Type& value,
+     uint64_t* remote_bar);
+
+st.async.weak.shared::cluster.mbarrier::complete_tx::bytes.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.async.weak.shared::cluster.mbarrier::complete_tx::bytes.type [addr], value, [remote_bar];    // 1.  PTX ISA 81, SM_90
+   // .type      = { .b32, .b64 }
+   template <typename Type>
+   __device__ static inline void st_async(
+     Type* addr,
+     const Type& value,
+     uint64_t* remote_bar);
+
+st.async.weak.shared::cluster.mbarrier::complete_tx::bytes.v2.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.async.weak.shared::cluster.mbarrier::complete_tx::bytes.v2.type [addr], value, [remote_bar]; // 2.  PTX ISA 81, SM_90
+   // .type      = { .b32, .b64 }
+   template <typename Type>
+   __device__ static inline void st_async(
+     Type* addr,
+     const Type (&value)[2],
+     uint64_t* remote_bar);
+
+st.async.weak.shared::cluster.mbarrier::complete_tx::bytes.v2.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.async.weak.shared::cluster.mbarrier::complete_tx::bytes.v2.type [addr], value, [remote_bar]; // 2.  PTX ISA 81, SM_90
+   // .type      = { .b32, .b64 }
+   template <typename Type>
+   __device__ static inline void st_async(
+     Type* addr,
+     const Type (&value)[2],
+     uint64_t* remote_bar);
+
+st.async.weak.shared::cluster.mbarrier::complete_tx::bytes.v4.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.async.weak.shared::cluster.mbarrier::complete_tx::bytes.v4.b32 [addr], value, [remote_bar];    // 3.  PTX ISA 81, SM_90
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void st_async(
+     B32* addr,
+     const B32 (&value)[4],
+     uint64_t* remote_bar);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/st_bulk.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/st_bulk.rst
new file mode 100644
index 00000000..82c78d39
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/st_bulk.rst
@@ -0,0 +1,13 @@
+..
+   This file was automatically generated. Do not edit.
+
+st.bulk.weak.shared::cta
+^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // st.bulk.weak.shared::cta [addr], size, initval; // PTX ISA 86, SM_100
+   template <int N32>
+   __device__ static inline void st_bulk(
+     void* addr,
+     uint64_t size,
+     cuda::ptx::n32_t<N32> initval);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_alloc.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_alloc.rst
new file mode 100644
index 00000000..1dfffdce
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_alloc.rst
@@ -0,0 +1,70 @@
+..
+   This file was automatically generated. Do not edit.
+
+tcgen05.alloc.cta_group::1.sync.aligned.shared::cta.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.alloc.cta_group.sync.aligned.shared::cta.b32 [dst], nCols; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_alloc(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t* dst,
+     const uint32_t& nCols);
+
+tcgen05.alloc.cta_group::2.sync.aligned.shared::cta.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.alloc.cta_group.sync.aligned.shared::cta.b32 [dst], nCols; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_alloc(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t* dst,
+     const uint32_t& nCols);
+
+tcgen05.dealloc.cta_group::1.sync.aligned.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.dealloc.cta_group.sync.aligned.b32 taddr, nCols; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_dealloc(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     const uint32_t& nCols);
+
+tcgen05.dealloc.cta_group::2.sync.aligned.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.dealloc.cta_group.sync.aligned.b32 taddr, nCols; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_dealloc(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     const uint32_t& nCols);
+
+tcgen05.relinquish_alloc_permit.cta_group::1.sync.aligned
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.relinquish_alloc_permit.cta_group.sync.aligned; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_relinquish_alloc_permit(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group);
+
+tcgen05.relinquish_alloc_permit.cta_group::2.sync.aligned
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.relinquish_alloc_permit.cta_group.sync.aligned; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_relinquish_alloc_permit(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_commit.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_commit.rst
new file mode 100644
index 00000000..fa197834
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_commit.rst
@@ -0,0 +1,48 @@
+..
+   This file was automatically generated. Do not edit.
+
+tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.commit.cta_group.mbarrier::arrive::one.shared::cluster.b64 [smem_bar]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_commit(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint64_t* smem_bar);
+
+tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.commit.cta_group.mbarrier::arrive::one.shared::cluster.b64 [smem_bar]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_commit(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint64_t* smem_bar);
+
+tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.multicast::cluster.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.commit.cta_group.mbarrier::arrive::one.shared::cluster.multicast::cluster.b64 [smem_bar], ctaMask; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_commit_multicast(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint64_t* smem_bar,
+     uint16_t ctaMask);
+
+tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.multicast::cluster.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.commit.cta_group.mbarrier::arrive::one.shared::cluster.multicast::cluster.b64 [smem_bar], ctaMask; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_commit_multicast(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint64_t* smem_bar,
+     uint16_t ctaMask);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_cp.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_cp.rst
new file mode 100644
index 00000000..2e112f2e
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_cp.rst
@@ -0,0 +1,434 @@
+..
+   This file was automatically generated. Do not edit.
+
+tcgen05.cp.cta_group::1.128x256b
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.128x256b [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_128x256b(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::2.128x256b
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.128x256b [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_128x256b(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::1.4x256b
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.4x256b [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_4x256b(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::2.4x256b
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.4x256b [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_4x256b(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::1.128x128b
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.128x128b [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_128x128b(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::2.128x128b
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.128x128b [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_128x128b(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::1.64x128b.warpx2::02_13
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.64x128b.warpx2::02_13 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_64x128b_warpx2_02_13(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::2.64x128b.warpx2::02_13
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.64x128b.warpx2::02_13 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_64x128b_warpx2_02_13(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::1.64x128b.warpx2::01_23
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.64x128b.warpx2::01_23 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_64x128b_warpx2_01_23(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::2.64x128b.warpx2::01_23
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.64x128b.warpx2::01_23 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_64x128b_warpx2_01_23(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::1.32x128b.warpx4
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.32x128b.warpx4 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_32x128b_warpx4(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::2.32x128b.warpx4
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.32x128b.warpx4 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_32x128b_warpx4(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::1.128x256b.b8x16.b6x16_p32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.128x256b.b8x16.b6x16_p32 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_128x256b_b8x16_b6x16_p32(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::2.128x256b.b8x16.b6x16_p32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.128x256b.b8x16.b6x16_p32 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_128x256b_b8x16_b6x16_p32(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::1.4x256b.b8x16.b6x16_p32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.4x256b.b8x16.b6x16_p32 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_4x256b_b8x16_b6x16_p32(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::2.4x256b.b8x16.b6x16_p32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.4x256b.b8x16.b6x16_p32 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_4x256b_b8x16_b6x16_p32(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::1.128x128b.b8x16.b6x16_p32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.128x128b.b8x16.b6x16_p32 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_128x128b_b8x16_b6x16_p32(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::2.128x128b.b8x16.b6x16_p32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.128x128b.b8x16.b6x16_p32 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_128x128b_b8x16_b6x16_p32(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::1.64x128b.warpx2::02_13.b8x16.b6x16_p32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.64x128b.warpx2::02_13.b8x16.b6x16_p32 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_64x128b_warpx2_02_13_b8x16_b6x16_p32(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::2.64x128b.warpx2::02_13.b8x16.b6x16_p32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.64x128b.warpx2::02_13.b8x16.b6x16_p32 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_64x128b_warpx2_02_13_b8x16_b6x16_p32(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::1.64x128b.warpx2::01_23.b8x16.b6x16_p32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.64x128b.warpx2::01_23.b8x16.b6x16_p32 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_64x128b_warpx2_01_23_b8x16_b6x16_p32(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::2.64x128b.warpx2::01_23.b8x16.b6x16_p32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.64x128b.warpx2::01_23.b8x16.b6x16_p32 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_64x128b_warpx2_01_23_b8x16_b6x16_p32(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::1.32x128b.warpx4.b8x16.b6x16_p32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.32x128b.warpx4.b8x16.b6x16_p32 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_32x128b_warpx4_b8x16_b6x16_p32(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::2.32x128b.warpx4.b8x16.b6x16_p32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.32x128b.warpx4.b8x16.b6x16_p32 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_32x128b_warpx4_b8x16_b6x16_p32(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::1.128x256b.b8x16.b4x16_p64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.128x256b.b8x16.b4x16_p64 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_128x256b_b8x16_b4x16_p64(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::2.128x256b.b8x16.b4x16_p64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.128x256b.b8x16.b4x16_p64 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_128x256b_b8x16_b4x16_p64(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::1.4x256b.b8x16.b4x16_p64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.4x256b.b8x16.b4x16_p64 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_4x256b_b8x16_b4x16_p64(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::2.4x256b.b8x16.b4x16_p64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.4x256b.b8x16.b4x16_p64 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_4x256b_b8x16_b4x16_p64(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::1.128x128b.b8x16.b4x16_p64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.128x128b.b8x16.b4x16_p64 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_128x128b_b8x16_b4x16_p64(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::2.128x128b.b8x16.b4x16_p64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.128x128b.b8x16.b4x16_p64 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_128x128b_b8x16_b4x16_p64(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::1.64x128b.warpx2::02_13.b8x16.b4x16_p64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.64x128b.warpx2::02_13.b8x16.b4x16_p64 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_64x128b_warpx2_02_13_b8x16_b4x16_p64(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::2.64x128b.warpx2::02_13.b8x16.b4x16_p64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.64x128b.warpx2::02_13.b8x16.b4x16_p64 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_64x128b_warpx2_02_13_b8x16_b4x16_p64(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::1.64x128b.warpx2::01_23.b8x16.b4x16_p64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.64x128b.warpx2::01_23.b8x16.b4x16_p64 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_64x128b_warpx2_01_23_b8x16_b4x16_p64(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::2.64x128b.warpx2::01_23.b8x16.b4x16_p64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.64x128b.warpx2::01_23.b8x16.b4x16_p64 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_64x128b_warpx2_01_23_b8x16_b4x16_p64(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::1.32x128b.warpx4.b8x16.b4x16_p64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.32x128b.warpx4.b8x16.b4x16_p64 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_32x128b_warpx4_b8x16_b4x16_p64(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
+
+tcgen05.cp.cta_group::2.32x128b.warpx4.b8x16.b4x16_p64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.cp.cta_group.32x128b.warpx4.b8x16.b4x16_p64 [taddr], s_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_cp_32x128b_warpx4_b8x16_b4x16_p64(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr,
+     uint64_t s_desc);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_fence.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_fence.rst
new file mode 100644
index 00000000..c5e8f298
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_fence.rst
@@ -0,0 +1,18 @@
+..
+   This file was automatically generated. Do not edit.
+
+tcgen05.fence::before_thread_sync
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.fence::before_thread_sync; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename = void>
+   __device__ static inline void tcgen05_fence_before_thread_sync();
+
+tcgen05.fence::after_thread_sync
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.fence::after_thread_sync; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename = void>
+   __device__ static inline void tcgen05_fence_after_thread_sync();
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_ld.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_ld.rst
new file mode 100644
index 00000000..5361f7f7
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_ld.rst
@@ -0,0 +1,2158 @@
+..
+   This file was automatically generated. Do not edit.
+
+tcgen05.ld.sync.aligned.16x64b.x1.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x64b.x1.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x64b(
+     B32 (&out)[1],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x64b.x1.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x64b.x1.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x64b_pack_16b(
+     B32 (&out)[1],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x64b.x2.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x64b.x2.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x64b(
+     B32 (&out)[2],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x64b.x2.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x64b.x2.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x64b_pack_16b(
+     B32 (&out)[2],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x64b.x4.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x64b.x4.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x64b(
+     B32 (&out)[4],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x64b.x4.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x64b.x4.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x64b_pack_16b(
+     B32 (&out)[4],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x64b.x8.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x64b.x8.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x64b(
+     B32 (&out)[8],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x64b.x8.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x64b.x8.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x64b_pack_16b(
+     B32 (&out)[8],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x64b.x16.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x64b.x16.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x64b(
+     B32 (&out)[16],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x64b.x16.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x64b.x16.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x64b_pack_16b(
+     B32 (&out)[16],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x64b.x32.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x64b.x32.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x64b(
+     B32 (&out)[32],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x64b.x32.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x64b.x32.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x64b_pack_16b(
+     B32 (&out)[32],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x64b.x64.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x64b.x64.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x64b(
+     B32 (&out)[64],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x64b.x64.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x64b.x64.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x64b_pack_16b(
+     B32 (&out)[64],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x64b.x128.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x64b.x128.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x64b(
+     B32 (&out)[128],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x64b.x128.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x64b.x128.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x64b_pack_16b(
+     B32 (&out)[128],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x128b.x1.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x128b.x1.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x128b(
+     B32 (&out)[2],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x128b.x1.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x128b.x1.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x128b_pack_16b(
+     B32 (&out)[2],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x128b.x2.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x128b.x2.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x128b(
+     B32 (&out)[4],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x128b.x2.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x128b.x2.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x128b_pack_16b(
+     B32 (&out)[4],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x128b.x4.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x128b.x4.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x128b(
+     B32 (&out)[8],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x128b.x4.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x128b.x4.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x128b_pack_16b(
+     B32 (&out)[8],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x128b.x8.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x128b.x8.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x128b(
+     B32 (&out)[16],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x128b.x8.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x128b.x8.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x128b_pack_16b(
+     B32 (&out)[16],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x128b.x16.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x128b.x16.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x128b(
+     B32 (&out)[32],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x128b.x16.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x128b.x16.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x128b_pack_16b(
+     B32 (&out)[32],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x128b.x32.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x128b.x32.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x128b(
+     B32 (&out)[64],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x128b.x32.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x128b.x32.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x128b_pack_16b(
+     B32 (&out)[64],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x128b.x64.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x128b.x64.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x128b(
+     B32 (&out)[128],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x128b.x64.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x128b.x64.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x128b_pack_16b(
+     B32 (&out)[128],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x256b.x1.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x256b.x1.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x256b(
+     B32 (&out)[4],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x256b.x1.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x256b.x1.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x256b_pack_16b(
+     B32 (&out)[4],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x256b.x2.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x256b.x2.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x256b(
+     B32 (&out)[8],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x256b.x2.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x256b.x2.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x256b_pack_16b(
+     B32 (&out)[8],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x256b.x4.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x256b.x4.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x256b(
+     B32 (&out)[16],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x256b.x4.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x256b.x4.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x256b_pack_16b(
+     B32 (&out)[16],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x256b.x8.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x256b.x8.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x256b(
+     B32 (&out)[32],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x256b.x8.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x256b.x8.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x256b_pack_16b(
+     B32 (&out)[32],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x256b.x16.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x256b.x16.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x256b(
+     B32 (&out)[64],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x256b.x16.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x256b.x16.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x256b_pack_16b(
+     B32 (&out)[64],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x256b.x32.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x256b.x32.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x256b(
+     B32 (&out)[128],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x256b.x32.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x256b.x32.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_16x256b_pack_16b(
+     B32 (&out)[128],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.32x32b.x1.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.32x32b.x1.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_32x32b(
+     B32 (&out)[1],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.32x32b.x1.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.32x32b.x1.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_32x32b_pack_16b(
+     B32 (&out)[1],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.32x32b.x2.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.32x32b.x2.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_32x32b(
+     B32 (&out)[2],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.32x32b.x2.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.32x32b.x2.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_32x32b_pack_16b(
+     B32 (&out)[2],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.32x32b.x4.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.32x32b.x4.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_32x32b(
+     B32 (&out)[4],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.32x32b.x4.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.32x32b.x4.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_32x32b_pack_16b(
+     B32 (&out)[4],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.32x32b.x8.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.32x32b.x8.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_32x32b(
+     B32 (&out)[8],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.32x32b.x8.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.32x32b.x8.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_32x32b_pack_16b(
+     B32 (&out)[8],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.32x32b.x16.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.32x32b.x16.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_32x32b(
+     B32 (&out)[16],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.32x32b.x16.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.32x32b.x16.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_32x32b_pack_16b(
+     B32 (&out)[16],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.32x32b.x32.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.32x32b.x32.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_32x32b(
+     B32 (&out)[32],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.32x32b.x32.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.32x32b.x32.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_32x32b_pack_16b(
+     B32 (&out)[32],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.32x32b.x64.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.32x32b.x64.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_32x32b(
+     B32 (&out)[64],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.32x32b.x64.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.32x32b.x64.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_32x32b_pack_16b(
+     B32 (&out)[64],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.32x32b.x128.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.32x32b.x128.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_32x32b(
+     B32 (&out)[128],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.32x32b.x128.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.32x32b.x128.pack::16b.b32 out, [taddr]; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_ld_32x32b_pack_16b(
+     B32 (&out)[128],
+     uint32_t taddr);
+
+tcgen05.ld.sync.aligned.16x32bx2.x1.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x32bx2.x1.b32 out, [taddr], immHalfSplitoff; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, int N32>
+   __device__ static inline void tcgen05_ld_16x32bx2(
+     B32 (&out)[1],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.sync.aligned.16x32bx2.x1.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x32bx2.x1.pack::16b.b32 out, [taddr], immHalfSplitoff; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, int N32>
+   __device__ static inline void tcgen05_ld_16x32bx2_pack_16b(
+     B32 (&out)[1],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.sync.aligned.16x32bx2.x2.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x32bx2.x2.b32 out, [taddr], immHalfSplitoff; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, int N32>
+   __device__ static inline void tcgen05_ld_16x32bx2(
+     B32 (&out)[2],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.sync.aligned.16x32bx2.x2.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x32bx2.x2.pack::16b.b32 out, [taddr], immHalfSplitoff; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, int N32>
+   __device__ static inline void tcgen05_ld_16x32bx2_pack_16b(
+     B32 (&out)[2],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.sync.aligned.16x32bx2.x4.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x32bx2.x4.b32 out, [taddr], immHalfSplitoff; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, int N32>
+   __device__ static inline void tcgen05_ld_16x32bx2(
+     B32 (&out)[4],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.sync.aligned.16x32bx2.x4.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x32bx2.x4.pack::16b.b32 out, [taddr], immHalfSplitoff; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, int N32>
+   __device__ static inline void tcgen05_ld_16x32bx2_pack_16b(
+     B32 (&out)[4],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.sync.aligned.16x32bx2.x8.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x32bx2.x8.b32 out, [taddr], immHalfSplitoff; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, int N32>
+   __device__ static inline void tcgen05_ld_16x32bx2(
+     B32 (&out)[8],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.sync.aligned.16x32bx2.x8.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x32bx2.x8.pack::16b.b32 out, [taddr], immHalfSplitoff; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, int N32>
+   __device__ static inline void tcgen05_ld_16x32bx2_pack_16b(
+     B32 (&out)[8],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.sync.aligned.16x32bx2.x16.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x32bx2.x16.b32 out, [taddr], immHalfSplitoff; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, int N32>
+   __device__ static inline void tcgen05_ld_16x32bx2(
+     B32 (&out)[16],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.sync.aligned.16x32bx2.x16.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x32bx2.x16.pack::16b.b32 out, [taddr], immHalfSplitoff; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, int N32>
+   __device__ static inline void tcgen05_ld_16x32bx2_pack_16b(
+     B32 (&out)[16],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.sync.aligned.16x32bx2.x32.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x32bx2.x32.b32 out, [taddr], immHalfSplitoff; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, int N32>
+   __device__ static inline void tcgen05_ld_16x32bx2(
+     B32 (&out)[32],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.sync.aligned.16x32bx2.x32.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x32bx2.x32.pack::16b.b32 out, [taddr], immHalfSplitoff; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, int N32>
+   __device__ static inline void tcgen05_ld_16x32bx2_pack_16b(
+     B32 (&out)[32],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.sync.aligned.16x32bx2.x64.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x32bx2.x64.b32 out, [taddr], immHalfSplitoff; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, int N32>
+   __device__ static inline void tcgen05_ld_16x32bx2(
+     B32 (&out)[64],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.sync.aligned.16x32bx2.x64.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x32bx2.x64.pack::16b.b32 out, [taddr], immHalfSplitoff; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, int N32>
+   __device__ static inline void tcgen05_ld_16x32bx2_pack_16b(
+     B32 (&out)[64],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.sync.aligned.16x32bx2.x128.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x32bx2.x128.b32 out, [taddr], immHalfSplitoff; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, int N32>
+   __device__ static inline void tcgen05_ld_16x32bx2(
+     B32 (&out)[128],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.sync.aligned.16x32bx2.x128.pack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.sync.aligned.16x32bx2.x128.pack::16b.b32 out, [taddr], immHalfSplitoff; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true, int N32>
+   __device__ static inline void tcgen05_ld_16x32bx2_pack_16b(
+     B32 (&out)[128],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.32x32b.x2.u32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x2.u32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[2],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x2.u32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x2.u32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[2],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x2.s32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x2.s32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[2],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x2.s32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x2.s32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[2],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x2.f32.min.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x2.f32.op.abs out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[2],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x2.f32.max.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x2.f32.op.abs out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[2],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x2.f32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x2.f32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[2],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x2.f32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x2.f32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[2],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x4.u32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x4.u32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[4],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x4.u32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x4.u32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[4],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x4.s32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x4.s32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[4],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x4.s32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x4.s32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[4],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x4.f32.min.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x4.f32.op.abs out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[4],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x4.f32.max.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x4.f32.op.abs out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[4],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x4.f32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x4.f32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[4],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x4.f32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x4.f32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[4],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x8.u32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x8.u32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[8],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x8.u32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x8.u32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[8],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x8.s32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x8.s32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[8],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x8.s32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x8.s32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[8],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x8.f32.min.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x8.f32.op.abs out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[8],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x8.f32.max.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x8.f32.op.abs out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[8],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x8.f32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x8.f32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[8],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x8.f32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x8.f32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[8],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x16.u32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x16.u32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[16],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x16.u32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x16.u32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[16],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x16.s32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x16.s32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[16],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x16.s32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x16.s32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[16],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x16.f32.min.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x16.f32.op.abs out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[16],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x16.f32.max.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x16.f32.op.abs out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[16],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x16.f32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x16.f32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[16],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x16.f32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x16.f32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[16],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x32.u32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x32.u32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[32],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x32.u32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x32.u32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[32],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x32.s32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x32.s32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[32],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x32.s32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x32.s32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[32],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x32.f32.min.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x32.f32.op.abs out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[32],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x32.f32.max.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x32.f32.op.abs out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[32],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x32.f32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x32.f32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[32],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x32.f32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x32.f32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[32],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x64.u32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x64.u32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[64],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x64.u32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x64.u32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[64],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x64.s32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x64.s32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[64],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x64.s32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x64.s32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[64],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x64.f32.min.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x64.f32.op.abs out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[64],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x64.f32.max.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x64.f32.op.abs out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[64],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x64.f32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x64.f32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[64],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x64.f32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x64.f32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[64],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x128.u32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x128.u32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[128],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x128.u32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x128.u32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[128],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x128.s32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x128.s32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[128],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x128.s32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x128.s32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[128],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x128.f32.min.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x128.f32.op.abs out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[128],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x128.f32.max.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x128.f32.op.abs out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[128],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x128.f32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x128.f32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[128],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.32x32b.x128.f32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.32x32b.x128.f32.op out, redval, [taddr]; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_32x32b(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[128],
+     uint32_t taddr);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x2.u32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x2.u32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[2],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x2.u32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x2.u32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[2],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x2.s32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x2.s32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[2],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x2.s32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x2.s32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[2],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x2.f32.min.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x2.f32.op.abs out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[2],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x2.f32.max.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x2.f32.op.abs out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[2],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x2.f32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x2.f32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[2],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x2.f32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x2.f32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[2],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x4.u32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x4.u32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[4],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x4.u32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x4.u32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[4],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x4.s32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x4.s32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[4],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x4.s32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x4.s32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[4],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x4.f32.min.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x4.f32.op.abs out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[4],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x4.f32.max.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x4.f32.op.abs out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[4],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x4.f32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x4.f32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[4],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x4.f32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x4.f32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[4],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x8.u32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x8.u32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[8],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x8.u32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x8.u32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[8],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x8.s32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x8.s32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[8],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x8.s32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x8.s32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[8],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x8.f32.min.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x8.f32.op.abs out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[8],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x8.f32.max.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x8.f32.op.abs out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[8],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x8.f32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x8.f32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[8],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x8.f32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x8.f32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[8],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x16.u32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x16.u32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[16],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x16.u32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x16.u32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[16],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x16.s32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x16.s32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[16],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x16.s32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x16.s32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[16],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x16.f32.min.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x16.f32.op.abs out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[16],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x16.f32.max.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x16.f32.op.abs out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[16],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x16.f32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x16.f32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[16],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x16.f32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x16.f32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[16],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x32.u32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x32.u32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[32],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x32.u32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x32.u32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[32],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x32.s32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x32.s32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[32],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x32.s32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x32.s32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[32],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x32.f32.min.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x32.f32.op.abs out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[32],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x32.f32.max.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x32.f32.op.abs out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[32],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x32.f32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x32.f32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[32],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x32.f32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x32.f32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[32],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x64.u32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x64.u32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[64],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x64.u32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x64.u32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[64],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x64.s32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x64.s32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[64],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x64.s32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x64.s32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[64],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x64.f32.min.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x64.f32.op.abs out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[64],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x64.f32.max.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x64.f32.op.abs out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[64],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x64.f32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x64.f32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[64],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x64.f32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x64.f32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[64],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x128.u32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x128.u32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[128],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x128.u32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x128.u32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline uint32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     uint32_t (&out)[128],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x128.s32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x128.s32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[128],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x128.s32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x128.s32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline int32_t tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     int32_t (&out)[128],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x128.f32.min.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x128.f32.op.abs out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[128],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x128.f32.max.abs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x128.f32.op.abs out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2_abs(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[128],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x128.f32.min
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x128.f32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[128],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
+
+tcgen05.ld.red.sync.aligned.16x32bx2.x128.f32.max
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.ld.red.sync.aligned.16x32bx2.x128.f32.op out, redval, [taddr], immHalfSplitoff; // PTX ISA 88, SM_103a, SM_103f, SM_110a, SM_110f
+   // .op        = { .min, .max }
+   template <int N32, cuda::ptx::dot_op Op>
+   __device__ static inline float tcgen05_ld_red_16x32bx2(
+     cuda::ptx::op_t<Op> op,
+     float (&out)[128],
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_mma.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_mma.rst
new file mode 100644
index 00000000..ba3f834d
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_mma.rst
@@ -0,0 +1,3898 @@
+..
+   This file was automatically generated. Do not edit.
+
+tcgen05.mma.cta_group::1.kind::f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, disable_output_lane, enable_input_d, scale_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f
+   // .kind      = { .kind::f16, .kind::tf32 }
+   // .cta_group = { .cta_group::1 }
+   template <int N32, cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_1_t,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[4],
+     bool enable_input_d,
+     cuda::ptx::n32_t<N32> scale_input_d);
+
+tcgen05.mma.cta_group::1.kind::tf32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, disable_output_lane, enable_input_d, scale_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f
+   // .kind      = { .kind::f16, .kind::tf32 }
+   // .cta_group = { .cta_group::1 }
+   template <int N32, cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_1_t,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[4],
+     bool enable_input_d,
+     cuda::ptx::n32_t<N32> scale_input_d);
+
+tcgen05.mma.cta_group::2.kind::f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, disable_output_lane, enable_input_d, scale_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f
+   // .kind      = { .kind::f16, .kind::tf32 }
+   // .cta_group = { .cta_group::2 }
+   template <int N32, cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_2_t,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[8],
+     bool enable_input_d,
+     cuda::ptx::n32_t<N32> scale_input_d);
+
+tcgen05.mma.cta_group::2.kind::tf32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, disable_output_lane, enable_input_d, scale_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f
+   // .kind      = { .kind::f16, .kind::tf32 }
+   // .cta_group = { .cta_group::2 }
+   template <int N32, cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_2_t,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[8],
+     bool enable_input_d,
+     cuda::ptx::n32_t<N32> scale_input_d);
+
+tcgen05.mma.cta_group::1.kind::f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, disable_output_lane, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_1_t,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[4],
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::tf32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, disable_output_lane, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_1_t,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[4],
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::f8f6f4
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, disable_output_lane, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_1_t,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[4],
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::i8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, disable_output_lane, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_1_t,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[4],
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, disable_output_lane, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_2_t,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[8],
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::tf32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, disable_output_lane, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_2_t,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[8],
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::f8f6f4
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, disable_output_lane, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_2_t,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[8],
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::i8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, disable_output_lane, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_2_t,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[8],
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, enable_input_d, scale_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f
+   // .kind      = { .kind::f16, .kind::tf32 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <int N32, cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     cuda::ptx::n32_t<N32> scale_input_d);
+
+tcgen05.mma.cta_group::2.kind::f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, enable_input_d, scale_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f
+   // .kind      = { .kind::f16, .kind::tf32 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <int N32, cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     cuda::ptx::n32_t<N32> scale_input_d);
+
+tcgen05.mma.cta_group::1.kind::tf32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, enable_input_d, scale_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f
+   // .kind      = { .kind::f16, .kind::tf32 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <int N32, cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     cuda::ptx::n32_t<N32> scale_input_d);
+
+tcgen05.mma.cta_group::2.kind::tf32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, enable_input_d, scale_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f
+   // .kind      = { .kind::f16, .kind::tf32 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <int N32, cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     cuda::ptx::n32_t<N32> scale_input_d);
+
+tcgen05.mma.cta_group::1.kind::f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::tf32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::tf32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::f8f6f4
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::f8f6f4
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::i8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::i8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, disable_output_lane, enable_input_d, scale_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f
+   // .kind      = { .kind::f16, .kind::tf32 }
+   // .cta_group = { .cta_group::1 }
+   template <int N32, cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_1_t,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[4],
+     bool enable_input_d,
+     cuda::ptx::n32_t<N32> scale_input_d);
+
+tcgen05.mma.cta_group::1.kind::tf32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, disable_output_lane, enable_input_d, scale_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f
+   // .kind      = { .kind::f16, .kind::tf32 }
+   // .cta_group = { .cta_group::1 }
+   template <int N32, cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_1_t,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[4],
+     bool enable_input_d,
+     cuda::ptx::n32_t<N32> scale_input_d);
+
+tcgen05.mma.cta_group::2.kind::f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, disable_output_lane, enable_input_d, scale_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f
+   // .kind      = { .kind::f16, .kind::tf32 }
+   // .cta_group = { .cta_group::2 }
+   template <int N32, cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_2_t,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[8],
+     bool enable_input_d,
+     cuda::ptx::n32_t<N32> scale_input_d);
+
+tcgen05.mma.cta_group::2.kind::tf32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, disable_output_lane, enable_input_d, scale_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f
+   // .kind      = { .kind::f16, .kind::tf32 }
+   // .cta_group = { .cta_group::2 }
+   template <int N32, cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_2_t,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[8],
+     bool enable_input_d,
+     cuda::ptx::n32_t<N32> scale_input_d);
+
+tcgen05.mma.cta_group::1.kind::f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, disable_output_lane, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_1_t,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[4],
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::tf32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, disable_output_lane, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_1_t,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[4],
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::f8f6f4
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, disable_output_lane, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_1_t,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[4],
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::i8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, disable_output_lane, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_1_t,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[4],
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, disable_output_lane, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_2_t,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[8],
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::tf32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, disable_output_lane, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_2_t,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[8],
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::f8f6f4
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, disable_output_lane, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_2_t,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[8],
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::i8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, disable_output_lane, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_2_t,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     const uint32_t (&disable_output_lane)[8],
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, scale_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f
+   // .kind      = { .kind::f16, .kind::tf32 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <int N32, cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     cuda::ptx::n32_t<N32> scale_input_d);
+
+tcgen05.mma.cta_group::2.kind::f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, scale_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f
+   // .kind      = { .kind::f16, .kind::tf32 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <int N32, cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     cuda::ptx::n32_t<N32> scale_input_d);
+
+tcgen05.mma.cta_group::1.kind::tf32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, scale_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f
+   // .kind      = { .kind::f16, .kind::tf32 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <int N32, cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     cuda::ptx::n32_t<N32> scale_input_d);
+
+tcgen05.mma.cta_group::2.kind::tf32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, scale_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f
+   // .kind      = { .kind::f16, .kind::tf32 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <int N32, cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     cuda::ptx::n32_t<N32> scale_input_d);
+
+tcgen05.mma.cta_group::1.kind::f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::f16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::tf32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::tf32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::f8f6f4
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::f8f6f4
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::i8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::i8
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.scale_vec::1X
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.scale_vec::1X
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.scale_vec::2X
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.scale_vec::2X
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::2X
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::2X
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::4X
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::4X
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16 [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16 [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.block32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32 [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.block32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32 [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.block32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32 [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.block32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32 [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32 [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32 [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.scale_vec::1X
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x_tmem_a(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.scale_vec::1X
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x_tmem_a(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.scale_vec::2X
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.scale_vec::2X
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::2X
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::2X
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::4X
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x_tmem_a(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::4X
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x_tmem_a(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16 [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16_tmem_a(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block16
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16 [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16_tmem_a(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.block32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32 [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.block32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32 [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.block32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32 [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.block32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32 [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32 [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32 [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.scale_vec::1X.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X.collector::a::fill [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x_collector_a_fill(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.scale_vec::1X.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X.collector::a::fill [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x_collector_a_fill(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.scale_vec::2X.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::fill [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.scale_vec::2X.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::fill [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::2X.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::fill [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::2X.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::fill [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::4X.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X.collector::a::fill [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x_collector_a_fill(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::4X.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X.collector::a::fill [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x_collector_a_fill(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block16.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16.collector::a::fill [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16_collector_a_fill(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block16.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16.collector::a::fill [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16_collector_a_fill(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.block32.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::fill [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.block32.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::fill [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.block32.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::fill [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.block32.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::fill [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block32.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::fill [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block32.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::fill [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.scale_vec::1X.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X.collector::a::fill [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x_tmem_a_collector_a_fill(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.scale_vec::1X.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X.collector::a::fill [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x_tmem_a_collector_a_fill(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.scale_vec::2X.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::fill [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.scale_vec::2X.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::fill [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::2X.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::fill [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::2X.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::fill [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::4X.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X.collector::a::fill [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x_tmem_a_collector_a_fill(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::4X.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X.collector::a::fill [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x_tmem_a_collector_a_fill(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block16.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16.collector::a::fill [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16_tmem_a_collector_a_fill(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block16.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16.collector::a::fill [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16_tmem_a_collector_a_fill(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.block32.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::fill [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.block32.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::fill [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.block32.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::fill [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.block32.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::fill [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block32.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::fill [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block32.collector::a::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::fill [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_fill(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.scale_vec::1X.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X.collector::a::use [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x_collector_a_use(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.scale_vec::1X.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X.collector::a::use [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x_collector_a_use(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.scale_vec::2X.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::use [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.scale_vec::2X.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::use [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::2X.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::use [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::2X.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::use [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::4X.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X.collector::a::use [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x_collector_a_use(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::4X.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X.collector::a::use [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x_collector_a_use(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block16.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16.collector::a::use [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16_collector_a_use(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block16.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16.collector::a::use [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16_collector_a_use(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.block32.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::use [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.block32.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::use [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.block32.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::use [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.block32.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::use [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block32.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::use [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block32.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::use [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.scale_vec::1X.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X.collector::a::use [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x_tmem_a_collector_a_use(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.scale_vec::1X.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X.collector::a::use [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x_tmem_a_collector_a_use(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.scale_vec::2X.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::use [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.scale_vec::2X.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::use [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::2X.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::use [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::2X.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::use [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::4X.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X.collector::a::use [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x_tmem_a_collector_a_use(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::4X.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X.collector::a::use [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x_tmem_a_collector_a_use(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block16.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16.collector::a::use [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16_tmem_a_collector_a_use(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block16.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16.collector::a::use [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16_tmem_a_collector_a_use(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.block32.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::use [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.block32.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::use [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.block32.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::use [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.block32.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::use [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block32.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::use [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block32.collector::a::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::use [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_use(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.scale_vec::1X.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X.collector::a::lastuse [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x_collector_a_lastuse(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.scale_vec::1X.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X.collector::a::lastuse [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x_collector_a_lastuse(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.scale_vec::2X.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::lastuse [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.scale_vec::2X.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::lastuse [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::2X.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::lastuse [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::2X.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::lastuse [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::4X.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X.collector::a::lastuse [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x_collector_a_lastuse(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::4X.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X.collector::a::lastuse [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x_collector_a_lastuse(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block16.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16.collector::a::lastuse [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16_collector_a_lastuse(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block16.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16.collector::a::lastuse [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16_collector_a_lastuse(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.block32.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::lastuse [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.block32.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::lastuse [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.block32.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::lastuse [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.block32.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::lastuse [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block32.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::lastuse [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block32.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::lastuse [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.scale_vec::1X.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X.collector::a::lastuse [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x_tmem_a_collector_a_lastuse(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.scale_vec::1X.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X.collector::a::lastuse [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x_tmem_a_collector_a_lastuse(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.scale_vec::2X.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::lastuse [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.scale_vec::2X.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::lastuse [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::2X.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::lastuse [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::2X.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::lastuse [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::4X.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X.collector::a::lastuse [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x_tmem_a_collector_a_lastuse(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::4X.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X.collector::a::lastuse [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x_tmem_a_collector_a_lastuse(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block16.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16.collector::a::lastuse [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16_tmem_a_collector_a_lastuse(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block16.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16.collector::a::lastuse [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16_tmem_a_collector_a_lastuse(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.block32.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::lastuse [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.block32.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::lastuse [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.block32.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::lastuse [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.block32.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::lastuse [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block32.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::lastuse [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block32.collector::a::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::lastuse [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_lastuse(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.scale_vec::1X.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X.collector::a::discard [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x_collector_a_discard(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.scale_vec::1X.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X.collector::a::discard [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x_collector_a_discard(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.scale_vec::2X.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::discard [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.scale_vec::2X.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::discard [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::2X.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::discard [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::2X.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::discard [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2x_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::4X.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X.collector::a::discard [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x_collector_a_discard(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::4X.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X.collector::a::discard [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x_collector_a_discard(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block16.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16.collector::a::discard [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16_collector_a_discard(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block16.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16.collector::a::discard [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16_collector_a_discard(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.block32.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::discard [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.block32.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::discard [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.block32.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::discard [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.block32.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::discard [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block32.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::discard [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block32.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::discard [d_tmem], a_desc, b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.scale_vec::1X.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X.collector::a::discard [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x_tmem_a_collector_a_discard(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.scale_vec::1X.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::1X.collector::a::discard [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf8f6f4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_1x_tmem_a_collector_a_discard(
+     cuda::ptx::kind_mxf8f6f4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.scale_vec::2X.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::discard [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.scale_vec::2X.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::discard [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::2X.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::discard [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::2X.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::2X.collector::a::discard [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_2_tmem_a_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.scale_vec::4X.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X.collector::a::discard [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x_tmem_a_collector_a_discard(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.scale_vec::4X.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.scale_vec::4X.collector::a::discard [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_vec_4x_tmem_a_collector_a_discard(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block16.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16.collector::a::discard [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16_tmem_a_collector_a_discard(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block16.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block16.collector::a::discard [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block16_tmem_a_collector_a_discard(
+     cuda::ptx::kind_mxf4nvf4_t,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf8f6f4.block_scale.block32.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::discard [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf8f6f4.block_scale.block32.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::discard [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4.block_scale.block32.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::discard [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4.block_scale.block32.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::discard [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block32.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::discard [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
+
+tcgen05.mma.cta_group::2.kind::mxf4nvf4.block_scale.block32.collector::a::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.cta_group.kind.block_scale.block32.collector::a::discard [d_tmem], [a_tmem], b_desc, idesc, [scale_A_tmem], [scale_B_tmem], enable_input_d; // PTX ISA 88, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .kind      = { .kind::mxf8f6f4, .kind::mxf4, .kind::mxf4nvf4 }
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_kind Kind, cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_mma_block_scale_block32_tmem_a_collector_a_discard(
+     cuda::ptx::kind_t<Kind> kind,
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     uint32_t scale_A_tmem,
+     uint32_t scale_B_tmem,
+     bool enable_input_d);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_mma_ws.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_mma_ws.rst
new file mode 100644
index 00000000..b3af8845
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_mma_ws.rst
@@ -0,0 +1,4482 @@
+..
+   This file was automatically generated. Do not edit.
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b0::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b0::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b0::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b0::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b0::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b0::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b0::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b0::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b0::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b0::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b0::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b0::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b0::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b0::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b0::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b0::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b0::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::use [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b0::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::use [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b0::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::use [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b0::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::use [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b0::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::use [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b0::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::use [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b0::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::use [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b0::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::use [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b0::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b0::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b0::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b0::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b0::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b0::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b0::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b0::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b0::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b0::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b0::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b0::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b0::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b0::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b0::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b0::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b0::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b0::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b0::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b0::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b0::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b0::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b0::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b0::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b0::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b0::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b0::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b0::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b0::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b0::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b0::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b0::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b0_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b0::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b0::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b0::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b0::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b0::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b0::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b0::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b0::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b0::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b0_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b1::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b1::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b1::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b1::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b1::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b1::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b1::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b1::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b1::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b1::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b1::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b1::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b1::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b1::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b1::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b1::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b1::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::use [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b1::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::use [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b1::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::use [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b1::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::use [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b1::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::use [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b1::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::use [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b1::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::use [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b1::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::use [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b1::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b1::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b1::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b1::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b1::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b1::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b1::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b1::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b1::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b1::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b1::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b1::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b1::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b1::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b1::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b1::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b1::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b1::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b1::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b1::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b1::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b1::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b1::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b1::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b1::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b1::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b1::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b1::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b1::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b1::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b1::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b1::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b1_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b1::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b1::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b1::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b1::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b1::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b1::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b1::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b1::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b1::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b1_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b2::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b2::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b2::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b2::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b2::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b2::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b2::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b2::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b2::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b2::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b2::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b2::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b2::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b2::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b2::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b2::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b2::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::use [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b2::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::use [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b2::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::use [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b2::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::use [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b2::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::use [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b2::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::use [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b2::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::use [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b2::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::use [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b2::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b2::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b2::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b2::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b2::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b2::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b2::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b2::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b2::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b2::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b2::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b2::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b2::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b2::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b2::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b2::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b2::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b2::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b2::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b2::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b2::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b2::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b2::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b2::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b2::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b2::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b2::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b2::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b2::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b2::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b2::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b2::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b2_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b2::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b2::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b2::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b2::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b2::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b2::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b2::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b2::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b2::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b2_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b3::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b3::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b3::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b3::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b3::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b3::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b3::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b3::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::fill [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b3::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b3::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b3::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b3::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b3::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b3::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b3::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b3::fill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::fill [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_fill(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b3::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::use [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b3::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::use [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b3::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::use [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b3::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::use [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b3::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::use [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b3::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::use [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b3::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::use [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b3::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::use [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b3::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b3::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b3::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b3::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b3::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b3::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b3::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b3::use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::use [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_use(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b3::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b3::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b3::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b3::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b3::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b3::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b3::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b3::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::lastuse [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b3::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b3::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b3::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b3::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b3::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b3::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b3::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b3::lastuse
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::lastuse [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_lastuse(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b3::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b3::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b3::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b3::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b3::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b3::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b3::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b3::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::discard [d_tmem], a_desc, b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_collector_b3_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint64_t a_desc,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b3::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b3::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b3::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b3::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d, zero_column_mask_desc; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d,
+     uint64_t zero_column_mask_desc);
+
+tcgen05.mma.ws.cta_group::1.kind::f16.collector::b3::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::tf32.collector::b3::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::f8f6f4.collector::b3::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
+
+tcgen05.mma.ws.cta_group::1.kind::i8.collector::b3::discard
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.mma.ws.cta_group.kind.collector::b3::discard [d_tmem], [a_tmem], b_desc, idesc, enable_input_d; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   // .cta_group = { .cta_group::1 }
+   // .kind      = { .kind::f16, .kind::tf32, .kind::f8f6f4, .kind::i8 }
+   template <cuda::ptx::dot_kind Kind>
+   __device__ static inline void tcgen05_mma_ws_tmem_a_collector_b3_discard(
+     cuda::ptx::cta_group_1_t,
+     cuda::ptx::kind_t<Kind> kind,
+     uint32_t d_tmem,
+     uint32_t a_tmem,
+     uint64_t b_desc,
+     uint32_t idesc,
+     bool enable_input_d);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_shift.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_shift.rst
new file mode 100644
index 00000000..15ea4839
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_shift.rst
@@ -0,0 +1,24 @@
+..
+   This file was automatically generated. Do not edit.
+
+tcgen05.shift.cta_group::1.down
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.shift.cta_group.down [taddr]; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_shift_down(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr);
+
+tcgen05.shift.cta_group::2.down
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.shift.cta_group.down [taddr]; // PTX ISA 86, SM_100a, SM_103a, SM_110a
+   // .cta_group = { .cta_group::1, .cta_group::2 }
+   template <cuda::ptx::dot_cta_group Cta_Group>
+   __device__ static inline void tcgen05_shift_down(
+     cuda::ptx::cta_group_t<Cta_Group> cta_group,
+     uint32_t taddr);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_st.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_st.rst
new file mode 100644
index 00000000..f23a2451
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_st.rst
@@ -0,0 +1,758 @@
+..
+   This file was automatically generated. Do not edit.
+
+tcgen05.st.sync.aligned.16x64b.x1.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x64b.x1.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x64b(
+     uint32_t taddr,
+     const B32 (&values)[1]);
+
+tcgen05.st.sync.aligned.16x64b.x1.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x64b.x1.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x64b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[1]);
+
+tcgen05.st.sync.aligned.16x64b.x2.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x64b.x2.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x64b(
+     uint32_t taddr,
+     const B32 (&values)[2]);
+
+tcgen05.st.sync.aligned.16x64b.x2.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x64b.x2.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x64b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[2]);
+
+tcgen05.st.sync.aligned.16x64b.x4.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x64b.x4.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x64b(
+     uint32_t taddr,
+     const B32 (&values)[4]);
+
+tcgen05.st.sync.aligned.16x64b.x4.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x64b.x4.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x64b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[4]);
+
+tcgen05.st.sync.aligned.16x64b.x8.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x64b.x8.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x64b(
+     uint32_t taddr,
+     const B32 (&values)[8]);
+
+tcgen05.st.sync.aligned.16x64b.x8.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x64b.x8.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x64b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[8]);
+
+tcgen05.st.sync.aligned.16x64b.x16.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x64b.x16.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x64b(
+     uint32_t taddr,
+     const B32 (&values)[16]);
+
+tcgen05.st.sync.aligned.16x64b.x16.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x64b.x16.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x64b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[16]);
+
+tcgen05.st.sync.aligned.16x64b.x32.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x64b.x32.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x64b(
+     uint32_t taddr,
+     const B32 (&values)[32]);
+
+tcgen05.st.sync.aligned.16x64b.x32.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x64b.x32.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x64b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[32]);
+
+tcgen05.st.sync.aligned.16x64b.x64.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x64b.x64.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x64b(
+     uint32_t taddr,
+     const B32 (&values)[64]);
+
+tcgen05.st.sync.aligned.16x64b.x64.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x64b.x64.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x64b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[64]);
+
+tcgen05.st.sync.aligned.16x64b.x128.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x64b.x128.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x64b(
+     uint32_t taddr,
+     const B32 (&values)[128]);
+
+tcgen05.st.sync.aligned.16x64b.x128.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x64b.x128.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x64b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[128]);
+
+tcgen05.st.sync.aligned.16x128b.x1.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x128b.x1.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x128b(
+     uint32_t taddr,
+     const B32 (&values)[2]);
+
+tcgen05.st.sync.aligned.16x128b.x1.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x128b.x1.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x128b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[2]);
+
+tcgen05.st.sync.aligned.16x128b.x2.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x128b.x2.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x128b(
+     uint32_t taddr,
+     const B32 (&values)[4]);
+
+tcgen05.st.sync.aligned.16x128b.x2.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x128b.x2.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x128b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[4]);
+
+tcgen05.st.sync.aligned.16x128b.x4.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x128b.x4.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x128b(
+     uint32_t taddr,
+     const B32 (&values)[8]);
+
+tcgen05.st.sync.aligned.16x128b.x4.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x128b.x4.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x128b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[8]);
+
+tcgen05.st.sync.aligned.16x128b.x8.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x128b.x8.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x128b(
+     uint32_t taddr,
+     const B32 (&values)[16]);
+
+tcgen05.st.sync.aligned.16x128b.x8.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x128b.x8.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x128b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[16]);
+
+tcgen05.st.sync.aligned.16x128b.x16.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x128b.x16.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x128b(
+     uint32_t taddr,
+     const B32 (&values)[32]);
+
+tcgen05.st.sync.aligned.16x128b.x16.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x128b.x16.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x128b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[32]);
+
+tcgen05.st.sync.aligned.16x128b.x32.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x128b.x32.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x128b(
+     uint32_t taddr,
+     const B32 (&values)[64]);
+
+tcgen05.st.sync.aligned.16x128b.x32.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x128b.x32.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x128b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[64]);
+
+tcgen05.st.sync.aligned.16x128b.x64.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x128b.x64.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x128b(
+     uint32_t taddr,
+     const B32 (&values)[128]);
+
+tcgen05.st.sync.aligned.16x128b.x64.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x128b.x64.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x128b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[128]);
+
+tcgen05.st.sync.aligned.16x256b.x1.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x256b.x1.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x256b(
+     uint32_t taddr,
+     const B32 (&values)[4]);
+
+tcgen05.st.sync.aligned.16x256b.x1.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x256b.x1.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x256b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[4]);
+
+tcgen05.st.sync.aligned.16x256b.x2.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x256b.x2.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x256b(
+     uint32_t taddr,
+     const B32 (&values)[8]);
+
+tcgen05.st.sync.aligned.16x256b.x2.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x256b.x2.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x256b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[8]);
+
+tcgen05.st.sync.aligned.16x256b.x4.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x256b.x4.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x256b(
+     uint32_t taddr,
+     const B32 (&values)[16]);
+
+tcgen05.st.sync.aligned.16x256b.x4.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x256b.x4.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x256b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[16]);
+
+tcgen05.st.sync.aligned.16x256b.x8.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x256b.x8.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x256b(
+     uint32_t taddr,
+     const B32 (&values)[32]);
+
+tcgen05.st.sync.aligned.16x256b.x8.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x256b.x8.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x256b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[32]);
+
+tcgen05.st.sync.aligned.16x256b.x16.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x256b.x16.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x256b(
+     uint32_t taddr,
+     const B32 (&values)[64]);
+
+tcgen05.st.sync.aligned.16x256b.x16.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x256b.x16.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x256b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[64]);
+
+tcgen05.st.sync.aligned.16x256b.x32.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x256b.x32.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x256b(
+     uint32_t taddr,
+     const B32 (&values)[128]);
+
+tcgen05.st.sync.aligned.16x256b.x32.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x256b.x32.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x256b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[128]);
+
+tcgen05.st.sync.aligned.32x32b.x1.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.32x32b.x1.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_32x32b(
+     uint32_t taddr,
+     const B32 (&values)[1]);
+
+tcgen05.st.sync.aligned.32x32b.x1.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.32x32b.x1.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_32x32b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[1]);
+
+tcgen05.st.sync.aligned.32x32b.x2.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.32x32b.x2.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_32x32b(
+     uint32_t taddr,
+     const B32 (&values)[2]);
+
+tcgen05.st.sync.aligned.32x32b.x2.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.32x32b.x2.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_32x32b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[2]);
+
+tcgen05.st.sync.aligned.32x32b.x4.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.32x32b.x4.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_32x32b(
+     uint32_t taddr,
+     const B32 (&values)[4]);
+
+tcgen05.st.sync.aligned.32x32b.x4.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.32x32b.x4.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_32x32b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[4]);
+
+tcgen05.st.sync.aligned.32x32b.x8.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.32x32b.x8.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_32x32b(
+     uint32_t taddr,
+     const B32 (&values)[8]);
+
+tcgen05.st.sync.aligned.32x32b.x8.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.32x32b.x8.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_32x32b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[8]);
+
+tcgen05.st.sync.aligned.32x32b.x16.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.32x32b.x16.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_32x32b(
+     uint32_t taddr,
+     const B32 (&values)[16]);
+
+tcgen05.st.sync.aligned.32x32b.x16.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.32x32b.x16.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_32x32b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[16]);
+
+tcgen05.st.sync.aligned.32x32b.x32.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.32x32b.x32.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_32x32b(
+     uint32_t taddr,
+     const B32 (&values)[32]);
+
+tcgen05.st.sync.aligned.32x32b.x32.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.32x32b.x32.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_32x32b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[32]);
+
+tcgen05.st.sync.aligned.32x32b.x64.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.32x32b.x64.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_32x32b(
+     uint32_t taddr,
+     const B32 (&values)[64]);
+
+tcgen05.st.sync.aligned.32x32b.x64.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.32x32b.x64.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_32x32b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[64]);
+
+tcgen05.st.sync.aligned.32x32b.x128.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.32x32b.x128.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_32x32b(
+     uint32_t taddr,
+     const B32 (&values)[128]);
+
+tcgen05.st.sync.aligned.32x32b.x128.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.32x32b.x128.unpack::16b.b32 [taddr], values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_32x32b_unpack_16b(
+     uint32_t taddr,
+     const B32 (&values)[128]);
+
+tcgen05.st.sync.aligned.16x32bx2.x1.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x32bx2.x1.b32 [taddr], immHalfSplitoff, values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x32bx2(
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff,
+     const B32 (&values)[1]);
+
+tcgen05.st.sync.aligned.16x32bx2.x1.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x32bx2.x1.unpack::16b.b32 [taddr], immHalfSplitoff, values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x32bx2_unpack_16b(
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff,
+     const B32 (&values)[1]);
+
+tcgen05.st.sync.aligned.16x32bx2.x2.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x32bx2.x2.b32 [taddr], immHalfSplitoff, values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x32bx2(
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff,
+     const B32 (&values)[2]);
+
+tcgen05.st.sync.aligned.16x32bx2.x2.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x32bx2.x2.unpack::16b.b32 [taddr], immHalfSplitoff, values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x32bx2_unpack_16b(
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff,
+     const B32 (&values)[2]);
+
+tcgen05.st.sync.aligned.16x32bx2.x4.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x32bx2.x4.b32 [taddr], immHalfSplitoff, values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x32bx2(
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff,
+     const B32 (&values)[4]);
+
+tcgen05.st.sync.aligned.16x32bx2.x4.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x32bx2.x4.unpack::16b.b32 [taddr], immHalfSplitoff, values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x32bx2_unpack_16b(
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff,
+     const B32 (&values)[4]);
+
+tcgen05.st.sync.aligned.16x32bx2.x8.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x32bx2.x8.b32 [taddr], immHalfSplitoff, values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x32bx2(
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff,
+     const B32 (&values)[8]);
+
+tcgen05.st.sync.aligned.16x32bx2.x8.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x32bx2.x8.unpack::16b.b32 [taddr], immHalfSplitoff, values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x32bx2_unpack_16b(
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff,
+     const B32 (&values)[8]);
+
+tcgen05.st.sync.aligned.16x32bx2.x16.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x32bx2.x16.b32 [taddr], immHalfSplitoff, values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x32bx2(
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff,
+     const B32 (&values)[16]);
+
+tcgen05.st.sync.aligned.16x32bx2.x16.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x32bx2.x16.unpack::16b.b32 [taddr], immHalfSplitoff, values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x32bx2_unpack_16b(
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff,
+     const B32 (&values)[16]);
+
+tcgen05.st.sync.aligned.16x32bx2.x32.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x32bx2.x32.b32 [taddr], immHalfSplitoff, values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x32bx2(
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff,
+     const B32 (&values)[32]);
+
+tcgen05.st.sync.aligned.16x32bx2.x32.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x32bx2.x32.unpack::16b.b32 [taddr], immHalfSplitoff, values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x32bx2_unpack_16b(
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff,
+     const B32 (&values)[32]);
+
+tcgen05.st.sync.aligned.16x32bx2.x64.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x32bx2.x64.b32 [taddr], immHalfSplitoff, values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x32bx2(
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff,
+     const B32 (&values)[64]);
+
+tcgen05.st.sync.aligned.16x32bx2.x64.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x32bx2.x64.unpack::16b.b32 [taddr], immHalfSplitoff, values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x32bx2_unpack_16b(
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff,
+     const B32 (&values)[64]);
+
+tcgen05.st.sync.aligned.16x32bx2.x128.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x32bx2.x128.b32 [taddr], immHalfSplitoff, values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x32bx2(
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff,
+     const B32 (&values)[128]);
+
+tcgen05.st.sync.aligned.16x32bx2.x128.unpack::16b.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.st.sync.aligned.16x32bx2.x128.unpack::16b.b32 [taddr], immHalfSplitoff, values; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tcgen05_st_16x32bx2_unpack_16b(
+     uint32_t taddr,
+     cuda::ptx::n32_t<N32> immHalfSplitoff,
+     const B32 (&values)[128]);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_wait.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_wait.rst
new file mode 100644
index 00000000..0a0efccc
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tcgen05_wait.rst
@@ -0,0 +1,18 @@
+..
+   This file was automatically generated. Do not edit.
+
+tcgen05.wait::ld.sync.aligned
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.wait::ld.sync.aligned; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename = void>
+   __device__ static inline void tcgen05_wait_ld();
+
+tcgen05.wait::st.sync.aligned
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tcgen05.wait::st.sync.aligned; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f
+   template <typename = void>
+   __device__ static inline void tcgen05_wait_st();
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tensormap_cp_fenceproxy.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tensormap_cp_fenceproxy.rst
new file mode 100644
index 00000000..27d3ce9d
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tensormap_cp_fenceproxy.rst
@@ -0,0 +1,62 @@
+..
+   This file was automatically generated. Do not edit.
+
+tensormap.cp_fenceproxy.global.shared::cta.tensormap::generic.release.cta.sync.aligned
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.cp_fenceproxy.global.shared::cta.tensormap::generic.sem.scope.sync.aligned  [dst], [src], size; // PTX ISA 83, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <int N32, cuda::ptx::dot_scope Scope>
+   __device__ static inline void tensormap_cp_fenceproxy(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_t<Scope> scope,
+     void* dst,
+     const void* src,
+     cuda::ptx::n32_t<N32> size);
+
+tensormap.cp_fenceproxy.global.shared::cta.tensormap::generic.release.cluster.sync.aligned
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.cp_fenceproxy.global.shared::cta.tensormap::generic.sem.scope.sync.aligned  [dst], [src], size; // PTX ISA 83, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <int N32, cuda::ptx::dot_scope Scope>
+   __device__ static inline void tensormap_cp_fenceproxy(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_t<Scope> scope,
+     void* dst,
+     const void* src,
+     cuda::ptx::n32_t<N32> size);
+
+tensormap.cp_fenceproxy.global.shared::cta.tensormap::generic.release.gpu.sync.aligned
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.cp_fenceproxy.global.shared::cta.tensormap::generic.sem.scope.sync.aligned  [dst], [src], size; // PTX ISA 83, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <int N32, cuda::ptx::dot_scope Scope>
+   __device__ static inline void tensormap_cp_fenceproxy(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_t<Scope> scope,
+     void* dst,
+     const void* src,
+     cuda::ptx::n32_t<N32> size);
+
+tensormap.cp_fenceproxy.global.shared::cta.tensormap::generic.release.sys.sync.aligned
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.cp_fenceproxy.global.shared::cta.tensormap::generic.sem.scope.sync.aligned  [dst], [src], size; // PTX ISA 83, SM_90
+   // .sem       = { .release }
+   // .scope     = { .cta, .cluster, .gpu, .sys }
+   template <int N32, cuda::ptx::dot_scope Scope>
+   __device__ static inline void tensormap_cp_fenceproxy(
+     cuda::ptx::sem_release_t,
+     cuda::ptx::scope_t<Scope> scope,
+     void* dst,
+     const void* src,
+     cuda::ptx::n32_t<N32> size);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tensormap_replace.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tensormap_replace.rst
new file mode 100644
index 00000000..056decee
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/tensormap_replace.rst
@@ -0,0 +1,300 @@
+..
+   This file was automatically generated. Do not edit.
+
+tensormap.replace.tile.global_address.global.b1024.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.global_address.space.b1024.b64 [tm_addr], new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .global }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline void tensormap_replace_global_address(
+     cuda::ptx::space_global_t,
+     void* tm_addr,
+     B64 new_val);
+
+tensormap.replace.tile.global_address.shared::cta.b1024.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.global_address.space.b1024.b64 [tm_addr], new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .shared::cta }
+   template <typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline void tensormap_replace_global_address(
+     cuda::ptx::space_shared_t,
+     void* tm_addr,
+     B64 new_val);
+
+tensormap.replace.tile.rank.global.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.rank.space.b1024.b32 [tm_addr], new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .global }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tensormap_replace_rank(
+     cuda::ptx::space_global_t,
+     void* tm_addr,
+     B32 new_val);
+
+tensormap.replace.tile.rank.shared::cta.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.rank.space.b1024.b32 [tm_addr], new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .shared::cta }
+   template <typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tensormap_replace_rank(
+     cuda::ptx::space_shared_t,
+     void* tm_addr,
+     B32 new_val);
+
+tensormap.replace.tile.box_dim.global.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.box_dim.space.b1024.b32 [tm_addr], ord, new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .global }
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tensormap_replace_box_dim(
+     cuda::ptx::space_global_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> ord,
+     B32 new_val);
+
+tensormap.replace.tile.box_dim.shared::cta.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.box_dim.space.b1024.b32 [tm_addr], ord, new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .shared::cta }
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tensormap_replace_box_dim(
+     cuda::ptx::space_shared_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> ord,
+     B32 new_val);
+
+tensormap.replace.tile.global_dim.global.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.global_dim.space.b1024.b32 [tm_addr], ord, new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .global }
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tensormap_replace_global_dim(
+     cuda::ptx::space_global_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> ord,
+     B32 new_val);
+
+tensormap.replace.tile.global_dim.shared::cta.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.global_dim.space.b1024.b32 [tm_addr], ord, new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .shared::cta }
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tensormap_replace_global_dim(
+     cuda::ptx::space_shared_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> ord,
+     B32 new_val);
+
+tensormap.replace.tile.global_stride.global.b1024.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.global_stride.space.b1024.b64 [tm_addr], ord, new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .global }
+   template <int N32, typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline void tensormap_replace_global_stride(
+     cuda::ptx::space_global_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> ord,
+     B64 new_val);
+
+tensormap.replace.tile.global_stride.shared::cta.b1024.b64
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.global_stride.space.b1024.b64 [tm_addr], ord, new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .shared::cta }
+   template <int N32, typename B64, enable_if_t<sizeof(B64) == 8, bool> = true>
+   __device__ static inline void tensormap_replace_global_stride(
+     cuda::ptx::space_shared_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> ord,
+     B64 new_val);
+
+tensormap.replace.tile.element_stride.global.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.element_stride.space.b1024.b32 [tm_addr], ord, new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .global }
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tensormap_replace_element_stride(
+     cuda::ptx::space_global_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> ord,
+     B32 new_val);
+
+tensormap.replace.tile.element_stride.shared::cta.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.element_stride.space.b1024.b32 [tm_addr], ord, new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .shared::cta }
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tensormap_replace_element_stride(
+     cuda::ptx::space_shared_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> ord,
+     B32 new_val);
+
+tensormap.replace.tile.element_stride.global.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.element_stride.space.b1024.b32 [tm_addr], ord, new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .global }
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tensormap_replace_element_size(
+     cuda::ptx::space_global_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> ord,
+     B32 new_val);
+
+tensormap.replace.tile.element_stride.shared::cta.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.element_stride.space.b1024.b32 [tm_addr], ord, new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .shared::cta }
+   template <int N32, typename B32, enable_if_t<sizeof(B32) == 4, bool> = true>
+   __device__ static inline void tensormap_replace_element_size(
+     cuda::ptx::space_shared_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> ord,
+     B32 new_val);
+
+tensormap.replace.tile.elemtype.global.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.elemtype.space.b1024.b32 [tm_addr], new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .global }
+   template <int N32>
+   __device__ static inline void tensormap_replace_elemtype(
+     cuda::ptx::space_global_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> new_val);
+
+tensormap.replace.tile.elemtype.shared::cta.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.elemtype.space.b1024.b32 [tm_addr], new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .shared::cta }
+   template <int N32>
+   __device__ static inline void tensormap_replace_elemtype(
+     cuda::ptx::space_shared_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> new_val);
+
+tensormap.replace.tile.interleave_layout.global.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.interleave_layout.space.b1024.b32 [tm_addr], new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .global }
+   template <int N32>
+   __device__ static inline void tensormap_replace_interleave_layout(
+     cuda::ptx::space_global_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> new_val);
+
+tensormap.replace.tile.interleave_layout.shared::cta.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.interleave_layout.space.b1024.b32 [tm_addr], new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .shared::cta }
+   template <int N32>
+   __device__ static inline void tensormap_replace_interleave_layout(
+     cuda::ptx::space_shared_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> new_val);
+
+tensormap.replace.tile.swizzle_mode.global.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.swizzle_mode.space.b1024.b32 [tm_addr], new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .global }
+   template <int N32>
+   __device__ static inline void tensormap_replace_swizzle_mode(
+     cuda::ptx::space_global_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> new_val);
+
+tensormap.replace.tile.swizzle_mode.shared::cta.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.swizzle_mode.space.b1024.b32 [tm_addr], new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .shared::cta }
+   template <int N32>
+   __device__ static inline void tensormap_replace_swizzle_mode(
+     cuda::ptx::space_shared_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> new_val);
+
+tensormap.replace.tile.fill_mode.global.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.fill_mode.space.b1024.b32 [tm_addr], new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .global }
+   template <int N32>
+   __device__ static inline void tensormap_replace_fill_mode(
+     cuda::ptx::space_global_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> new_val);
+
+tensormap.replace.tile.fill_mode.shared::cta.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.fill_mode.space.b1024.b32 [tm_addr], new_val; // PTX ISA 83, SM_90a, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .shared::cta }
+   template <int N32>
+   __device__ static inline void tensormap_replace_fill_mode(
+     cuda::ptx::space_shared_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> new_val);
+
+tensormap.replace.tile.swizzle_atomicity.global.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.swizzle_atomicity.space.b1024.b32 [tm_addr], new_val; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .global }
+   template <int N32>
+   __device__ static inline void tensormap_replace_swizzle_atomicity(
+     cuda::ptx::space_global_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> new_val);
+
+tensormap.replace.tile.swizzle_atomicity.shared::cta.b1024.b32
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. code-block:: cuda
+
+   // tensormap.replace.tile.swizzle_atomicity.space.b1024.b32 [tm_addr], new_val; // PTX ISA 86, SM_100a, SM_100f, SM_103a, SM_103f, SM_110a, SM_110f, SM_120a, SM_120f, SM_121a, SM_121f
+   // .space     = { .shared::cta }
+   template <int N32>
+   __device__ static inline void tensormap_replace_swizzle_atomicity(
+     cuda::ptx::space_shared_t,
+     void* tm_addr,
+     cuda::ptx::n32_t<N32> new_val);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/trap.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/trap.rst
new file mode 100644
index 00000000..89fe68d0
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/generated/trap.rst
@@ -0,0 +1,10 @@
+..
+   This file was automatically generated. Do not edit.
+
+trap
+^^^^
+.. code-block:: cuda
+
+   // trap; // PTX ISA 10, SM_50
+   template <typename = void>
+   __device__ static inline void trap();
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/getctarank.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/getctarank.rst
new file mode 100644
index 00000000..d355ed80
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/getctarank.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-getctarank:
+
+getctarank
+==========
+
+-  PTX ISA:
+   `getctarank <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-getctarank>`__
+
+.. include:: generated/getctarank.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/ld.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/ld.rst
new file mode 100644
index 00000000..16db4494
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/ld.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-ld:
+
+ld
+==
+
+-  PTX ISA:
+   `ld <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-ld>`__
+
+.. include:: generated/ld.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/manual/shfl_sync.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/manual/shfl_sync.rst
new file mode 100644
index 00000000..9a1c5731
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/manual/shfl_sync.rst
@@ -0,0 +1,80 @@
+
+shfl.sync
+^^^^^^^^^
+
+.. code:: cuda
+
+   // PTX ISA 6.0
+   // shfl.sync.mode.b32  d[|p], a, b, c, membermask;
+   //   .mode = { .up, .down, .bfly, .idx };
+
+   template<typename T>
+   [[nodiscard]] __device__ static inline
+   T shfl_sync_idx(T        data,
+                   uint32_t lane_idx_offset,
+                   uint32_t clamp_segmask,
+                   uint32_t lane_mask) noexcept;
+
+   template<typename T>
+   [[nodiscard]] __device__ static inline
+   T shfl_sync_idx(T        data,
+                   bool&    pred,
+                   uint32_t lane_idx_offset,
+                   uint32_t clamp_segmask,
+                   uint32_t lane_mask) noexcept;
+
+   template<typename T>
+   [[nodiscard]] __device__ static inline
+   T shfl_sync_up(T        data,
+                  uint32_t lane_idx_offset,
+                  uint32_t clamp_segmask,
+                  uint32_t lane_mask) noexcept;
+
+   template<typename T>
+   [[nodiscard]] __device__ static inline
+   T shfl_sync_up(T        data,
+                  bool&    pred,
+                  uint32_t lane_idx_offset,
+                  uint32_t clamp_segmask,
+                  uint32_t lane_mask) noexcept;
+
+   template<typename T>
+   [[nodiscard]] __device__ static inline
+   T shfl_sync_down(T        data,
+                    uint32_t lane_idx_offset,
+                    uint32_t clamp_segmask,
+                    uint32_t lane_mask) noexcept;
+
+   template<typename T>
+   [[nodiscard]] __device__ static inline
+   T shfl_sync_down(T        data,
+                    bool&    pred,
+                    uint32_t lane_idx_offset,
+                    uint32_t clamp_segmask,
+                    uint32_t lane_mask) noexcept;
+
+   template<typename T>
+   [[nodiscard]] __device__ static inline
+   T shfl_sync_bfly(T        data,
+                    uint32_t lane_idx_offset,
+                    uint32_t clamp_segmask,
+                    uint32_t lane_mask) noexcept;
+
+   template<typename T>
+   [[nodiscard]] __device__ static inline
+   T shfl_sync_bfly(T        data,
+                    bool&    pred,
+                    uint32_t lane_idx_offset,
+                    uint32_t clamp_segmask,
+                    uint32_t lane_mask) noexcept;
+
+**Constrains and checks**
+
+The following conditions are checked at *run-time* in debug mode:
+
+- ``T`` must have 32-bit size (compile-time)
+- ``lane_idx_offset`` must be less than the warp size
+- ``clamp_segmask`` must use the bit positions [0:4] and [8:12])
+- ``lane_mask`` must be a subset of the active mask
+- All lanes must have the same value for ``lane_mask``
+- The destination lane must be a member of the ``lane_mask``
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/mapa.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/mapa.rst
new file mode 100644
index 00000000..a73124b8
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/mapa.rst
@@ -0,0 +1,37 @@
+.. _libcudacxx-ptx-instructions-mapa:
+
+mapa
+====
+
+-  PTX ISA:
+   `mapa <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-mapa>`__
+
+This instruction can `currently not be
+implemented <https://github.com/NVIDIA/cccl/issues/1414>`__ by libcu++.
+The instruction can be accessed through the cooperative groups
+`cluster_group <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cluster-group>`__
+API:
+
+Usage:
+------
+
+.. code:: cuda
+
+   #include <cooperative_groups.h>
+
+   __cluster_dims__(2)
+   __global__ void kernel() {
+       __shared__ int x;
+       x = 1;
+       namespace cg = cooperative_groups;
+       cg::cluster_group cluster = cg::this_cluster();
+
+       cluster.sync();
+
+       // Get address of remote shared memory value:
+       unsigned int other_block_rank = cluster.block_rank() ^ 1;
+       int * remote_x = cluster.map_shared_rank(&bar, other_block_rank);
+
+       // Write to remote value:
+       *remote_x = 2;
+   }
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/mbarrier_arrive.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/mbarrier_arrive.rst
new file mode 100644
index 00000000..f01e7a95
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/mbarrier_arrive.rst
@@ -0,0 +1,68 @@
+.. _libcudacxx-ptx-instructions-mbarrier-arrive:
+
+mbarrier.arrive
+===============
+
+-  PTX ISA:
+   `mbarrier.arrive <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-mbarrier-arrive>`__
+
+.. _mbarrier.arrive-1:
+
+mbarrier.arrive
+---------------
+
+Some of the listed PTX instructions below are semantically equivalent.
+They differ in one important way: the shorter instructions are typically
+supported on older compilers.
+
+.. include:: generated/mbarrier_arrive.rst
+
+mbarrier.arrive.no_complete
+---------------------------
+
+.. include:: generated/mbarrier_arrive_no_complete.rst
+
+mbarrier.arrive.expect_tx
+-------------------------
+
+.. include:: generated/mbarrier_arrive_expect_tx.rst
+
+Usage
+-----
+
+.. code:: cuda
+
+   #include <cuda/ptx>
+   #include <cuda/barrier>
+   #include <cooperative_groups.h>
+
+   __global__ void kernel() {
+       using cuda::ptx::sem_release;
+       using cuda::ptx::space_cluster;
+       using cuda::ptx::space_shared;
+       using cuda::ptx::scope_cluster;
+       using cuda::ptx::scope_cta;
+
+       using barrier_t = cuda::barrier<cuda::thread_scope_block>;
+       __shared__ barrier_t bar;
+       init(&bar, blockDim.x);
+       __syncthreads();
+
+       NV_IF_TARGET(NV_PROVIDES_SM_90, (
+           // Arrive on local shared memory barrier:
+           uint64_t token;
+           token = cuda::ptx::mbarrier_arrive_expect_tx(sem_release, scope_cluster, space_shared, &bar, 1);
+
+           // Get address of remote cluster barrier:
+           namespace cg = cooperative_groups;
+           cg::cluster_group cluster = cg::this_cluster();
+           unsigned int other_block_rank = cluster.block_rank() ^ 1;
+           uint64_t * remote_bar = cluster.map_shared_rank(&bar, other_block_rank);
+
+           // Sync cluster to ensure remote barrier is initialized.
+           cluster.sync();
+
+           // Arrive on remote cluster barrier:
+           cuda::ptx::mbarrier_arrive_expect_tx(sem_release, scope_cluster, space_cluster, remote_bar, 1);
+       )
+   }
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/mbarrier_expect_tx.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/mbarrier_expect_tx.rst
new file mode 100644
index 00000000..6c348132
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/mbarrier_expect_tx.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-mbarrier-expect_tx:
+
+mbarrier.expect_tx
+==================
+
+-  PTX ISA:
+   `mbarrier.expect_tx <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-mbarrier-expect-tx>`__
+
+.. include:: generated/mbarrier_expect_tx.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/mbarrier_init.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/mbarrier_init.rst
new file mode 100644
index 00000000..a736f53b
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/mbarrier_init.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-mbarrier-init:
+
+mbarrier.init
+=============
+
+-  PTX ISA:
+   `mbarrier.arrive <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-mbarrier-init>`__
+
+.. include:: generated/mbarrier_init.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/mbarrier_inval.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/mbarrier_inval.rst
new file mode 100644
index 00000000..c69c1ebf
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/mbarrier_inval.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-mbarrier_inval:
+
+mbarrier.inval
+==============
+
+-  PTX ISA:
+   `mbarrier.inval <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-mbarrier-inval>`__
+
+.. include:: generated/mbarrier_inval.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/mbarrier_test_wait.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/mbarrier_test_wait.rst
new file mode 100644
index 00000000..d8a4e794
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/mbarrier_test_wait.rst
@@ -0,0 +1,19 @@
+.. _libcudacxx-ptx-instructions-mbarrier-test_wait:
+
+mbarrier.test_wait
+==================
+
+-  PTX ISA:
+   `mbarrier.test_wait <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-mbarrier-test-wait-mbarrier-try-wait>`__
+
+.. _mbarrier.test_wait-1:
+
+mbarrier.test_wait
+------------------
+
+.. include:: generated/mbarrier_test_wait.rst
+
+mbarrier.test_wait.parity
+-------------------------
+
+.. include:: generated/mbarrier_test_wait_parity.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/mbarrier_try_wait.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/mbarrier_try_wait.rst
new file mode 100644
index 00000000..1869695f
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/mbarrier_try_wait.rst
@@ -0,0 +1,20 @@
+.. _libcudacxx-ptx-instructions-mbarrier-try_wait:
+
+mbarrier.try_wait
+=================
+
+-  PTX ISA:
+   `mbarrier.try_wait <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-mbarrier-test-wait-mbarrier-try-wait>`__
+
+
+.. _mbarrier.try_wait-1:
+
+mbarrier.try_wait
+-----------------
+
+.. include:: generated/mbarrier_try_wait.rst
+
+mbarrier.try_wait.parity
+------------------------
+
+.. include:: generated/mbarrier_try_wait_parity.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/multimem_ld_reduce.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/multimem_ld_reduce.rst
new file mode 100644
index 00000000..e9f52121
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/multimem_ld_reduce.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-multimem-ld_reduce:
+
+multimem.ld_reduce
+==================
+
+-  PTX ISA:
+   `multimem.ld_reduce <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-multimem-ld-reduce-multimem-st-multimem-red>`__
+
+.. include:: generated/multimem_ld_reduce.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/multimem_red.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/multimem_red.rst
new file mode 100644
index 00000000..0a6511b7
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/multimem_red.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-multimem-red:
+
+multimem.red
+============
+
+-  PTX ISA:
+   `multimem.red <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-multimem-ld-reduce-multimem-st-multimem-red>`__
+
+.. include:: generated/multimem_red.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/multimem_st.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/multimem_st.rst
new file mode 100644
index 00000000..75197f44
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/multimem_st.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-multimem-st:
+
+multimem.st
+===========
+
+-  PTX ISA:
+   `multimem.st <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-multimem-ld-reduce-multimem-st-multimem-red>`__
+
+.. include:: generated/multimem_st.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/prmt.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/prmt.rst
new file mode 100644
index 00000000..3006a567
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/prmt.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-prmt:
+
+prmt
+====
+
+-  PTX ISA:
+   `prmt <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-prmt>`__
+
+.. include:: generated/prmt.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/red_async.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/red_async.rst
new file mode 100644
index 00000000..82ba07c3
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/red_async.rst
@@ -0,0 +1,31 @@
+.. _libcudacxx-ptx-instructions-mbarrier-red-async:
+
+red.async
+=========
+
+-  PTX ISA:
+   `red.async <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-red-async>`__
+
+.. _red.async-1:
+
+red.async
+---------
+
+.. include:: generated/red_async.rst
+
+red.async ``.s64`` emulation
+----------------------------
+
+PTX does not currently (CTK 12.3) expose ``red.async.add.s64``. This
+exposure is emulated in ``cuda::ptx`` using
+
+.. code:: cuda
+
+   // red.async.relaxed.cluster.shared::cluster.mbarrier::complete_tx::bytes{.op}.u64  [dest], value, [remote_bar]; // .u64 intentional PTX ISA 81, SM_90
+   // .op        = { .add }
+   template <typename=void>
+   __device__ static inline void red_async(
+     cuda::ptx::op_add_t,
+     int64_t* dest,
+     const int64_t& value,
+     int64_t* remote_bar);
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/setmaxnreg.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/setmaxnreg.rst
new file mode 100644
index 00000000..8e961e54
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/setmaxnreg.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-setmaxnreg:
+
+setmaxnreg
+==========
+
+-  PTX ISA:
+   `setmaxnreg <https://docs.nvidia.com/cuda/parallel-thread-execution/#miscellaneous-instructions-setmaxnreg>`__
+
+.. include:: generated/setmaxnreg.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/shfl_sync.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/shfl_sync.rst
new file mode 100644
index 00000000..6c20dff9
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/shfl_sync.rst
@@ -0,0 +1,10 @@
+
+.. _libcudacxx-ptx-instructions-shfl_sync:
+
+shfl.sync
+=========
+
+-  PTX ISA:
+   `shfl.sync <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-shfl-sync>`__
+
+.. include:: manual/shfl_sync.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/shl.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/shl.rst
new file mode 100644
index 00000000..da23c233
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/shl.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-shl:
+
+shl
+===
+
+-  PTX ISA:
+   `shl <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#logic-and-shift-instructions-shl>`__
+
+.. include:: generated/shl.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/shr.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/shr.rst
new file mode 100644
index 00000000..11922e48
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/shr.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-shr:
+
+shr
+===
+
+-  PTX ISA:
+   `shr <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#logic-and-shift-instructions-shr>`__
+
+.. include:: generated/shr.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/special_registers.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/special_registers.rst
new file mode 100644
index 00000000..1981f7fb
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/special_registers.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-special-registers:
+
+Special registers
+=================
+
+-  PTX ISA:
+   `Special Register <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers>`__
+
+.. include:: generated/get_sreg.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/st.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/st.rst
new file mode 100644
index 00000000..9cd17ef3
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/st.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-st:
+
+st
+==
+
+-  PTX ISA:
+   `st <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-st>`__
+
+.. include:: generated/st.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/st_async.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/st_async.rst
new file mode 100644
index 00000000..c71aebd7
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/st_async.rst
@@ -0,0 +1,14 @@
+.. _libcudacxx-ptx-instructions-st-async:
+
+st.async
+========
+
+-  PTX ISA:
+   `st.async <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-st-async>`__
+-  Used in: :ref:`How to use st.async <libcudacxx-ptx-examples-st-async>`
+
+**NOTE.** Alignment of ``addr`` must be a multiple of vector size. For
+instance, the ``addr`` supplied to the ``v2.b32`` variant must be
+aligned to ``2 x 4 = 8`` bytes.
+
+.. include:: generated/st_async.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/st_bulk.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/st_bulk.rst
new file mode 100644
index 00000000..fa5c4762
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/st_bulk.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-st-bulk:
+
+st.bulk
+========
+
+-  PTX ISA:
+   `st.bulk <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-st-bulk>`__
+
+.. include:: generated/st_bulk.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_alloc.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_alloc.rst
new file mode 100644
index 00000000..a30f2a25
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_alloc.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-tcgen05-alloc:
+
+tcgen05.alloc
+=============
+
+-  PTX ISA:
+   `tcgen05.alloc <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-alloc-tcgen05-dealloc-tcgen05-relinquish-alloc-permit>`__
+
+.. include:: generated/tcgen05_alloc.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_commit.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_commit.rst
new file mode 100644
index 00000000..a431350d
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_commit.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-tcgen05-commit:
+
+tcgen05.commit
+==============
+
+-  PTX ISA:
+   `tcgen05.commit <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-commit>`__
+
+.. include:: generated/tcgen05_commit.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_cp.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_cp.rst
new file mode 100644
index 00000000..5a220536
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_cp.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-tcgen05-cp:
+
+tcgen05.cp
+==========
+
+-  PTX ISA:
+   `tcgen05.cp <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-cp>`__
+
+.. include:: generated/tcgen05_cp.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_fence.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_fence.rst
new file mode 100644
index 00000000..6635131f
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_fence.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-tcgen05-fence:
+
+tcgen05.fence
+=============
+
+-  PTX ISA:
+   `tcgen05.fence <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-fence>`__
+
+.. include:: generated/tcgen05_fence.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_ld.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_ld.rst
new file mode 100644
index 00000000..165b8eb9
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_ld.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-tcgen05-ld:
+
+tcgen05.ld
+==========
+
+-  PTX ISA:
+   `tcgen05.ld <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-ld>`__
+
+.. include:: generated/tcgen05_ld.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_mma.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_mma.rst
new file mode 100644
index 00000000..9672ae0d
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_mma.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-tcgen05-mma:
+
+tcgen05.mma
+===========
+
+-  PTX ISA:
+   `tcgen05.mma <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-mma>`__
+
+.. include:: generated/tcgen05_mma.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_mma_ws.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_mma_ws.rst
new file mode 100644
index 00000000..e2206629
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_mma_ws.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-tcgen05-mma-ws:
+
+tcgen05.mma.ws
+==============
+
+-  PTX ISA:
+   `tcgen05.mma.ws <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-mma-ws>`__
+
+.. include:: generated/tcgen05_mma_ws.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_shift.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_shift.rst
new file mode 100644
index 00000000..eef04ae4
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_shift.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-tcgen05-shift:
+
+tcgen05.shift
+=============
+
+-  PTX ISA:
+   `tcgen05.shift <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-shift>`__
+
+.. include:: generated/tcgen05_shift.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_st.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_st.rst
new file mode 100644
index 00000000..f1011494
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_st.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-tcgen05-st:
+
+tcgen05.st
+==========
+
+-  PTX ISA:
+   `tcgen05.st <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-st>`__
+
+.. include:: generated/tcgen05_st.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_wait.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_wait.rst
new file mode 100644
index 00000000..cb149e5c
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/tcgen05_wait.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-tcgen05-wait:
+
+tcgen05.wait
+============
+
+-  PTX ISA:
+   `tcgen05.wait <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#tensorcore-5th-generation-instructions-tcgen05-wait>`__
+
+.. include:: generated/tcgen05_wait.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/tensormap_cp_fenceproxy.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/tensormap_cp_fenceproxy.rst
new file mode 100644
index 00000000..2f7622bb
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/tensormap_cp_fenceproxy.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-tensormap-cp_fenceproxy:
+
+tensormap.cp_fenceproxy
+=======================
+
+-  PTX ISA:
+   `tensormap.cp_fenceproxy <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-tensormap-cp-fenceproxy>`__
+
+.. include:: generated/tensormap_cp_fenceproxy.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/tensormap_replace.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/tensormap_replace.rst
new file mode 100644
index 00000000..331dcff3
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/tensormap_replace.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-tensormap-replace:
+
+tensormap.replace
+=================
+
+-  PTX ISA:
+   `tensormap.replace <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-tensormap-replace>`__
+
+.. include:: generated/tensormap_replace.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/instructions/trap.rst b/cccl_upstream/docs/libcudacxx/ptx/instructions/trap.rst
new file mode 100644
index 00000000..54b641f3
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/instructions/trap.rst
@@ -0,0 +1,9 @@
+.. _libcudacxx-ptx-instructions-trap:
+
+trap
+====
+
+-  PTX ISA:
+   `trap <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#miscellaneous-instructions-trap>`__
+
+.. include:: generated/trap.rst
diff --git a/cccl_upstream/docs/libcudacxx/ptx/pragmas.rst b/cccl_upstream/docs/libcudacxx/ptx/pragmas.rst
new file mode 100644
index 00000000..95d826f6
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/pragmas.rst
@@ -0,0 +1,24 @@
+.. _libcudacxx-ptx-pragmas:
+
+PTX Pragmas
+================
+
+.. toctree::
+   :maxdepth: 1
+
+   pragmas/enable_smem_spilling
+
+.. list-table:: `.pragma Strings <https://docs.nvidia.com/cuda/parallel-thread-execution/#descriptions-pragma-strings>`__
+   :widths: 50 50
+   :header-rows: 1
+
+   * - Pragma
+     - Available in libcu++
+   * - `"nounroll" <https://docs.nvidia.com/cuda/parallel-thread-execution/#pragma-strings-nounroll>`__
+     - No
+   * - `"used_bytes_mask" <https://docs.nvidia.com/cuda/parallel-thread-execution/#pragma-strings-used-bytes-mask>`__
+     - No
+   * - `"enable_smem_spilling" <https://docs.nvidia.com/cuda/parallel-thread-execution/#pragma-strings-enable-smem-spilling>`__
+     - Yes, CCCL 3.2.0 / CUDA 13.2
+   * - `"frequency" <https://docs.nvidia.com/cuda/parallel-thread-execution/#pragma-strings-frequency>`__
+     - No
diff --git a/cccl_upstream/docs/libcudacxx/ptx/pragmas/enable_smem_spilling.rst b/cccl_upstream/docs/libcudacxx/ptx/pragmas/enable_smem_spilling.rst
new file mode 100644
index 00000000..6e580f60
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx/pragmas/enable_smem_spilling.rst
@@ -0,0 +1,20 @@
+
+.. _libcudacxx-ptx-pragma-enable_smem_spilling:
+
+.pragma "enable_smem_spilling"
+==============================
+
+-  PTX ISA:
+   `.pragma "enable_smem_spilling" <https://docs.nvidia.com/cuda/parallel-thread-execution/#pragma-strings-enable-smem-spilling>`__
+
+.. code:: cuda
+
+  // PTX ISA 9.0
+  // .pragma "enable_smem_spilling";
+
+  namespace cuda::ptx {
+
+  __device__ static __forceinline__
+  void enable_smem_spilling() noexcept;
+
+  } // namespace cuda::ptx
diff --git a/cccl_upstream/docs/libcudacxx/ptx_api.rst b/cccl_upstream/docs/libcudacxx/ptx_api.rst
new file mode 100644
index 00000000..40992d5f
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/ptx_api.rst
@@ -0,0 +1,83 @@
+.. _libcudacxx-ptx:
+
+PTX API
+========
+
+
+The ``cuda::ptx`` namespace contains functions that map one-to-one to
+`PTX instructions <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html>`__.
+These can be used for maximal control of the generated code, or to
+experiment with new hardware features before a high-level C++ API is
+available.
+
+.. toctree::
+   :maxdepth: 1
+
+   ptx/examples
+   ptx/instructions
+   ptx/pragmas
+
+Versions and compatibility
+---------------------------
+
+The ``cuda/ptx`` header is intended to present a stable API within one major
+version of the CTK on a best effort basis. This means that:
+
+-  All functions are marked static inline.
+
+-  The type of a function parameter can be changed to be more generic if
+   that means that code that called the original version can still be
+   compiled.
+
+-  Good exposure of the PTX should be high priority. If, at a new major
+   version, we face a difficult choice between breaking
+   backward-compatibility and an improvement of the PTX exposure, we
+   will tend to the latter option more easily than in other parts of
+   libcu++.
+
+The API does not guarantee stability of template parameters. The order and
+number of template parameters may change. Use arguments to driver overload
+resolution as in the code below to ensure forward-compatibility:
+
+.. code:: cuda
+
+   // Use arguments to drive overload resolution:
+   cuda::ptx::mbarrier_arrive_expect_tx(cuda::ptx::sem_release, cuda::ptx::scope_cta, cuda::ptx::space_shared, &bar, 1);
+
+   // Specifying templates directly is not forward-compatible, as order and number
+   // of template parameters may change in a minor release:
+   cuda::ptx::mbarrier_arrive_expect_tx<cuda::ptx::sem_release_t>(
+     cuda::ptx::sem_release, cuda::ptx::scope_cta, cuda::ptx::space_shared, &bar, 1
+   );
+
+**PTX ISA version and compute capability.** Each binding notes under
+which PTX ISA version and SM version it may be used. Example:
+
+.. code:: cuda
+
+   // mbarrier.arrive.shared::cta.b64 state, [addr]; // 1.  PTX ISA 70, SM_80
+   __device__ inline uint64_t mbarrier_arrive(
+     cuda::ptx::sem_release_t sem,
+     cuda::ptx::scope_cta_t scope,
+     cuda::ptx::space_shared_t space,
+     uint64_t* addr);
+
+To check if the current compiler is recent enough, use:
+
+.. code:: cuda
+
+   #if __cccl_ptx_isa >= 700
+   cuda::ptx::mbarrier_arrive(cuda::ptx::sem_release, cuda::ptx::scope_cta, cuda::ptx::space_shared, &bar, 1);
+   #endif
+
+Ensure that you only call the function when compiling for a recent
+enough compute capability (SM version), like this:
+
+.. code:: cuda
+
+   NV_IF_TARGET(NV_PROVIDES_SM_80,(
+     cuda::ptx::mbarrier_arrive(cuda::ptx::sem_release, cuda::ptx::scope_cta, cuda::ptx::space_shared, &bar, 1);
+   ));
+
+For more information on which compilers correspond to which PTX ISA, see
+the `PTX ISA release notes <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#release-notes>`__.
diff --git a/cccl_upstream/docs/libcudacxx/runtime.rst b/cccl_upstream/docs/libcudacxx/runtime.rst
new file mode 100644
index 00000000..5e0a76cc
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/runtime.rst
@@ -0,0 +1,210 @@
+.. _libcudacxx-runtime-api:
+
+Runtime
+========
+
+The Runtime API provides higher-level building blocks for core CUDA functionality. It takes the existing CUDA Runtime API
+set and removes or replaces some problematic patterns, such as implicit state. It is designed to make common operations
+like resource management, work submission, and memory allocation easier to express and safer to compose. These APIs lower
+to the CUDA Driver API under the hood, but remain composable with the CUDA Runtime API by reusing runtime handle types
+(such as ``cudaStream_t``) in the interfaces. This results in an interface that applies RAII for lifetime management,
+while remaining composable with existing CUDA C++ code that manages resources explicitly.
+
+At a glance, the runtime layer includes:
+
+- Streams and events work submission and synchronization.
+- Buffers as a typed, stream-ordered storage with property-checked memory container.
+- Memory pools to allocate device, managed, and pinned memory, either directly or through buffers.
+- Launch API to configure and launch kernels.
+- Runtime algorithms like ``copy_bytes`` and ``fill_bytes`` for basic data movement.
+- Legacy memory resources as synchronous compatibility fallbacks for older toolkits.
+
+Error handling
+--------------
+
+CCCL Runtime APIs use C++ exceptions for error handling. Operations such as creating runtime objects, allocating memory,
+querying device properties, or synchronizing work report failures by throwing exceptions. Users can write ordinary
+control flow without checking a status value after every runtime call, and catch exceptions at the boundary where an
+operation can be retried, reported, or allowed to fail.
+
+See :ref:`Exception Handling <libcudacxx-extended-api-exceptions>` for details on ``cuda::cuda_error``, including how
+to access the stored ``cudaError_t`` status.
+
+This is part of the CCCL Runtime API model. It differs from the CUDA Runtime API, where operations generally return
+``cudaError_t`` values that callers check against ``cudaSuccess``.
+
+See :ref:`CUDA Runtime interactions <cccl-runtime-cudart-interactions>` if you are interested in CUDA Runtime interop.
+
+Example: vector add with buffers, pools, and launch
+---------------------------------------------------
+
+.. code:: cpp
+
+   #include <cuda/devices>
+   #include <cuda/stream>
+   #include <cuda/std/span>
+   #include <cuda/buffer>
+   #include <cuda/memory_pool>
+   #include <cuda/launch>
+
+   struct kernel {
+     template <typename Config>
+     __device__ void operator()(Config config,
+                                cuda::std::span<const float> A,
+                                cuda::std::span<const float> B,
+                                cuda::std::span<float> C) {
+       auto tid = cuda::gpu_thread.rank(cuda::grid, config);
+       if (tid < A.size())
+         C[tid] = A[tid] + B[tid];
+     }
+   };
+
+   int main() {
+     cuda::device_ref device = cuda::devices[0];
+     cuda::stream stream{device};
+     auto pool = cuda::device_default_memory_pool(device);
+
+     int num_elements = 1000;
+     auto A = cuda::make_buffer<float>(stream, pool, num_elements, 1.0);
+     auto B = cuda::make_buffer<float>(stream, pool, num_elements, 2.0);
+     auto C = cuda::make_buffer<float>(stream, pool, num_elements, cuda::no_init);
+
+     constexpr int threads_per_block = 256;
+     auto config = cuda::distribute<threads_per_block>(num_elements);
+
+     cuda::launch(stream, config, kernel{}, A, B, C);
+   }
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   runtime/cudart_interactions
+   runtime/stream
+   runtime/event
+   runtime/algorithm
+   runtime/device
+   runtime/hierarchy
+   runtime/launch
+   runtime/buffer
+   runtime/memory_pools
+   runtime/legacy_resources
+
+.. list-table::
+   :widths: 25 45 30 30
+   :header-rows: 1
+
+   * - **API**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+
+   * - :ref:`devices <cccl-runtime-device-devices>`
+     - A range of all available CUDA devices
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`device_ref <cccl-runtime-device-device-ref>`
+     - A non-owning representation of a CUDA device
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`arch_traits <cccl-runtime-device-arch-traits>`
+     - Per-architecture trait accessors
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`stream_ref <cccl-runtime-stream-stream-ref>`
+     - A non-owning wrapper around a ``cudaStream_t``
+     - CCCL 2.2.0
+     - CUDA 12.3
+
+   * - :ref:`stream <cccl-runtime-stream-stream>`
+     - An owning wrapper around a ``cudaStream_t``
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`event_ref <cccl-runtime-event-event-ref>`
+     - A non-owning wrapper around a ``cudaEvent_t``
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`event <cccl-runtime-event-event>`
+     - An owning wrapper around a ``cudaEvent_t`` (timing disabled)
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`timed_event <cccl-runtime-event-timed-event>`
+     - An owning wrapper around a ``cudaEvent_t`` with timing enabled and elapsed-time queries
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`copy_bytes <cccl-runtime-algorithm-copy_bytes>`
+     - Byte-wise copy into a ``cuda::stream_ref`` for ``cuda::std::span``/``cuda::std::mdspan`` sources and destinations
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`fill_bytes <cccl-runtime-algorithm-fill_bytes>`
+     - Byte-wise fill into a ``cuda::stream_ref`` for ``cuda::std::span``/``cuda::std::mdspan`` destinations
+     - CCCL 3.1.0
+     - CUDA 13.1
+
+   * - :ref:`hierarchy <cccl-runtime-hierarchy-hierarchy>`
+     - Representation of CUDA thread hierarchies (grid, cluster, block, warp, thread)
+     - CCCL 3.2.0
+     - CUDA 13.2
+
+   * - :ref:`launch <cccl-runtime-launch-launch>`
+     - Kernel launch with configuration and options
+     - CCCL 3.2.0
+     - CUDA 13.2
+
+   * - :ref:`kernel_config <cccl-runtime-launch-kernel-config>`
+     - Kernel launch configuration combining hierarchy dimensions and launch options
+     - CCCL 3.2.0
+     - CUDA 13.2
+
+   * - :ref:`make_config <cccl-runtime-launch-make-config>`
+     - Factory function to create kernel configurations from hierarchy dimensions and launch options
+     - CCCL 3.2.0
+     - CUDA 13.2
+
+   * - :ref:`device_memory_pool <cccl-runtime-memory-pools-device-memory-pool>`
+     - Stream-ordered device memory pool using CUDA memory pool API
+     - CCCL 3.2.0
+     - CUDA 13.2
+
+   * - :ref:`managed_memory_pool <cccl-runtime-memory-pools-managed-memory-pool>`
+     - Stream-ordered managed (unified) memory pool
+     - CCCL 3.2.0
+     - CUDA 13.2
+
+   * - :ref:`pinned_memory_pool <cccl-runtime-memory-pools-pinned-memory-pool>`
+     - Stream-ordered pinned (page-locked) host memory pool
+     - CCCL 3.2.0
+     - CUDA 13.2
+
+   * - :ref:`device_default_memory_pool <cccl-runtime-memory-pools-device-default>`
+     - Get the default device memory pool for a device
+     - CCCL 3.2.0
+     - CUDA 13.2
+
+   * - :ref:`managed_default_memory_pool <cccl-runtime-memory-pools-managed-default>`
+     - Get the default managed (unified) memory pool
+     - CCCL 3.2.0
+     - CUDA 13.2
+
+   * - :ref:`pinned_default_memory_pool <cccl-runtime-memory-pools-pinned-default>`
+     - Get the default pinned (page-locked) host memory pool
+     - CCCL 3.2.0
+     - CUDA 13.2
+
+   * - :ref:`buffer <cccl-runtime-buffer-buffer>`
+     - Typed data container allocated from memory resources. It handles stream-ordered allocation, initialization, and deallocation of memory.
+     - CCCL 3.2.0
+     - CUDA 13.2
+
+   * - :ref:`legacy resources <cccl-runtime-legacy-resources>`
+     - Synchronous compatibility resources backed by legacy CUDA allocation APIs.
+     - CCCL 3.2.0
+     - CUDA 13.2
diff --git a/cccl_upstream/docs/libcudacxx/runtime/algorithm.rst b/cccl_upstream/docs/libcudacxx/runtime/algorithm.rst
new file mode 100644
index 00000000..f0a447d8
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/runtime/algorithm.rst
@@ -0,0 +1,81 @@
+.. _cccl-runtime-algorithm:
+
+Algorithm
+==========
+
+The ``runtime`` part of the ``cuda/algorithm`` header provides stream-ordered, byte-wise primitives that operate on
+:cpp:class:`cuda::std::span` and :cpp:class:`cuda::std::mdspan`-compatible types. They require a
+:cpp:class:`cuda::stream_ref` to enqueue work.
+
+:cpp:func:`cuda::copy_bytes`
+-------------------------------
+.. _cccl-runtime-algorithm-copy_bytes:
+
+Launch a byte-wise copy from source to destination on the provided stream.
+
+- Signature: :cpp:func:`cuda::copy_bytes`
+- Overloads accept :cpp:class:`cuda::std::span`-convertible contiguous ranges or
+  :cpp:class:`cuda::std::mdspan`-convertible multi-dimensional views.
+- Elements must be trivially copyable
+- :cpp:class:`cuda::std::mdspan`-convertible types must convert to an mdspan that is exhaustive
+- The optional ``config`` argument is a :cpp:struct:`cuda::copy_configuration` that controls source access order and
+  managed-memory location hints
+
+Availability: CCCL 3.1.0 / CUDA 13.1
+
+.. code:: cpp
+
+   #include <cuda/algorithm>
+   #include <cuda/stream>
+   #include <cuda/std/algorithm>
+   #include <cuda/std/span>
+
+   void copy_example(cuda::stream_ref s, cuda::std::span<const int> src, cuda::std::span<int> dst) {
+     // copy_bytes copies up to src.size_bytes(); dst can be larger.
+     auto n = cuda::std::min(src.size(), dst.size());
+     auto src_prefix = src.first(n / 2);
+     auto dst_prefix = dst.first(n / 2);
+     auto src_suffix = src.subspan(n / 2);
+     auto dst_suffix = dst.subspan(n / 2);
+
+     // Default behavior: enqueue a stream-ordered byte-wise copy on stream s.
+     cuda::copy_bytes(s, src_prefix, dst_prefix);
+
+     // Advanced behavior: customize source access order for this copy.
+     auto config = cuda::copy_configuration{
+       .src_access_order = cuda::source_access_order::during_api_call,
+     };
+     cuda::copy_bytes(s, src_suffix, dst_suffix, config);
+   }
+
+
+:cpp:func:`cuda::fill_bytes`
+-------------------------------
+.. _cccl-runtime-algorithm-fill_bytes:
+
+Launch a byte-wise fill of the destination on the provided stream.
+
+- Overloads accept :cpp:class:`cuda::std::span`-convertible or :cpp:class:`cuda::std::mdspan`-convertible destinations.
+- Elements must be trivially copyable
+- :cpp:class:`cuda::std::mdspan`-convertible types must convert to an mdspan that is exhaustive
+
+Availability: CCCL 3.1.0 / CUDA 13.1
+
+.. code:: cpp
+
+   #include <cuda/algorithm>
+   #include <cuda/stream>
+   #include <cuda/std/algorithm>
+   #include <cuda/std/span>
+
+   void fill_example(cuda::stream_ref s, cuda::std::span<unsigned char> dst) {
+     // Reserve 16-byte red zones at both ends and clear the payload in between.
+     auto guard = cuda::std::min(static_cast<decltype(dst.size())>(16), dst.size() / 2);
+     auto head  = dst.first(guard);
+     auto body  = dst.subspan(guard, dst.size() - 2 * guard);
+     auto tail  = dst.last(guard);
+
+     cuda::fill_bytes(s, head, 0xCD); // debug guard pattern
+     cuda::fill_bytes(s, body, 0x00); // initialize payload
+     cuda::fill_bytes(s, tail, 0xCD); // debug guard pattern
+   }
diff --git a/cccl_upstream/docs/libcudacxx/runtime/buffer.rst b/cccl_upstream/docs/libcudacxx/runtime/buffer.rst
new file mode 100644
index 00000000..a12bcbe5
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/runtime/buffer.rst
@@ -0,0 +1,230 @@
+.. _cccl-runtime-buffer:
+
+.. |cuda_make_buffer| replace:: ``cuda::make_buffer``
+.. _cuda_make_buffer: ../api/namespacecuda_1a8d909070d4cf758e776659b91e473a6f.html
+
+Buffer
+======
+
+The buffer API provides a typed container allocated from memory resources. It handles stream-ordered allocation, initialization, and deallocation of memory.
+
+:cpp:class:`cuda::buffer`
+---------------------------
+.. _cccl-runtime-buffer-buffer:
+
+:cpp:class:`cuda::buffer` is a container that manages typed storage allocated from a given
+:ref:`memory resource <libcudacxx-extended-api-memory-resources-resource>` in stream order using a provided
+:ref:`stream_ref <cccl-runtime-stream-stream-ref>`. The elements are initialized during construction, which may require
+a kernel launch. The stream provided during construction is stored and later used for deallocation of the buffer,
+either explicitly or when the buffer destructor is called.
+
+Buffer owns a copy of the memory resource, which means it must be copy-constructible. If a resource is not copy-constructible, like memory pool objects, :ref:`shared_resource <libcudacxx-extended-api-memory-resources-shared-resource>` can be used to attach shared ownership to a resource type.
+
+In addition to being typed, :cpp:class:`cuda::buffer` also takes a set of
+:ref:`properties <libcudacxx-extended-api-memory-resources-properties>` to ensure that memory accessibility and other
+constraints are checked at compile time.
+
+While the buffer operates in stream order, it can also be constructed with a :ref:`synchronous_resource <libcudacxx-extended-api-memory-resources-synchronous-resource>`, in which case it will automatically use the :ref:`synchronous_resource_adapter <libcudacxx-extended-api-memory-resources-synchronous-adapter>` to wrap the provided resource.
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/buffer>
+   #include <cuda/devices>
+   #include <cuda/memory_pool>
+   #include <cuda/stream>
+
+   void use_buffer(cuda::stream_ref stream) {
+     // Create a device buffer
+     auto mr = cuda::device_default_memory_pool(cuda::devices[0]);
+     auto buf = cuda::make_buffer<float>(
+       stream,
+       mr,
+       1024,  // size
+       0.0f   // value
+     );
+
+     // Use buffer...
+
+     // Buffer is automatically deallocated when destroyed
+   }
+
+Type Aliases
+------------
+.. _cccl-runtime-buffer-type-aliases:
+
+Convenience type aliases are provided for common buffer types:
+
+- :cpp:any:`cuda::device_buffer` - Buffer with ``device_accessible`` property
+- :cpp:any:`cuda::host_buffer` - Buffer with ``host_accessible`` property
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/buffer>
+   #include <cuda/devices>
+   #include <cuda/memory_resource>
+   #include <cuda/stream>
+
+   void use_buffers(cuda::stream_ref stream) {
+     auto device_mr = cuda::device_default_memory_pool(cuda::devices[0]);
+     auto host_mr = cuda::pinned_default_memory_pool();
+
+     cuda::device_buffer<int> dev_buf{stream, device_mr, 1000};
+     cuda::host_buffer<int> host_buf{stream, host_mr, 1000};
+   }
+
+Construction
+------------
+.. _cccl-runtime-buffer-construction:
+
+Buffers can be constructed in several ways, depending on how you want to initialize the memory:
+
+- Empty buffer: ``buffer(stream, resource)``
+- With size (uninitialized): ``buffer(stream, resource, size, no_init)``
+- From iterator range: ``buffer(stream, resource, first, last)``
+- From initializer list: ``buffer(stream, resource, {val1, val2, ...})``
+- From range: ``buffer(stream, resource, range)``
+
+In each case the memory is allocated and initialized in stream order on the provided stream.
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/buffer>
+   #include <cuda/devices>
+   #include <cuda/memory_resource>
+   #include <vector>
+
+   void construct_buffers(cuda::stream_ref stream) {
+     auto mr = cuda::device_default_memory_pool(cuda::devices[0]);
+
+     // Empty buffer
+     cuda::device_buffer<int> buf1{stream, mr};
+
+     // Uninitialized buffer
+     cuda::device_buffer<int> buf2{stream, mr, 1000, cuda::no_init};
+
+     // Initialized with value
+     cuda::device_buffer<int> buf3{stream, mr, 1000, 42};
+
+     // From iterator range
+     std::vector<int> vec{1, 2, 3, 4, 5};
+     cuda::device_buffer<int> buf4{stream, mr, vec.begin(), vec.end()};
+
+     // From initializer list
+     cuda::device_buffer<int> buf5{stream, mr, {1, 2, 3, 4, 5}};
+   }
+
+Stored Stream Management and Deallocation
+-----------------------------------------
+.. _cccl-runtime-buffer-stream-management:
+
+Buffers store a reference to the stream they were constructed with, and can have that stream queried or changed:
+
+- ``stream()`` - Get the associated stream
+- ``set_stream(new_stream)`` - Change the associated stream (synchronizes with old stream)
+
+When the buffer is destroyed, the memory is deallocated using the stored stream. The behavior is undefined if the stream
+referenced by the buffer is destroyed before the buffer. Buffers can also be explicitly destroyed with ``destroy()`` or
+``destroy(stream_ref)``, which will deallocate the memory using the provided stream.
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/buffer>
+   #include <cuda/devices>
+   #include <cuda/memory_resource>
+   #include <cuda/stream>
+
+   void manage_stream_and_deallocate() {
+     cuda::stream stream1{cuda::devices[0]};
+     cuda::stream stream2{cuda::devices[0]};
+     auto mr = cuda::device_default_memory_pool(cuda::devices[0]);
+
+    // Allocate on stream1
+    cuda::device_buffer<int> buf{stream1, mr, 1024, cuda::no_init};
+
+    // Switch to stream2 (synchronizes with stream1)
+    buf.set_stream(stream2);
+
+    // Explicit deallocation on the stored stream (stream2)
+    buf.destroy();
+    // Alternative would be to call buf.destroy(stream2)
+   }
+
+|cuda_make_buffer|_
+------------------------------------------------------------------------------------------------
+.. _cccl-runtime-buffer-make-buffer:
+
+|cuda_make_buffer|_ is a factory function that
+creates buffers with automatic property deduction from the memory resource. It supports the same construction patterns
+as the buffer constructors, in addition to an overload that sets all elements of the buffer to the same value.
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/buffer>
+   #include <cuda/devices>
+   #include <cuda/memory_resource>
+
+   void make_buffers(cuda::stream_ref stream) {
+     auto mr = cuda::device_default_memory_pool(cuda::devices[0]);
+
+     // Properties are automatically deduced from the memory resource
+     // and all elements are set to 42.0f
+     auto buf = cuda::make_buffer<float>(stream, mr, 1024, 42.0f);
+   }
+
+Iterators and Access
+--------------------
+.. _cccl-runtime-buffer-iterators:
+
+Buffers provide standard container-like iterators and access methods:
+
+- ``begin()`` / ``end()`` - Iterator access
+- ``cbegin()`` / ``cend()`` - Const iterator access
+- ``rbegin()`` / ``rend()`` - Reverse iterator access
+- ``data()`` - Pointer to underlying data
+- ``size()`` - Number of elements
+- ``empty()`` - Check if buffer is empty
+- ``get_unsynchronized(n)`` - Access element without synchronization, instead of using ``operator[]``
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/buffer>
+   #include <cuda/devices>
+   #include <cuda/memory_resource>
+   #include <cuda/std/cstddef>
+   #include <algorithm>
+
+   void iterate_buffer(cuda::stream_ref stream) {
+     auto mr = cuda::pinned_default_memory_pool();
+     cuda::host_buffer<int> buf{stream, mr, {1, 2, 3, 4, 5}};
+
+     // Unsynchronized element access by index
+     for (cuda::std::size_t i = 0; i < buf.size(); ++i) {
+       buf.get_unsynchronized(i) += 1;
+     }
+
+     // Use with algorithms
+     auto it = std::find(buf.begin(), buf.end(), 3);
+   }
+
+
+Memory Resource Access
+----------------------
+.. _cccl-runtime-buffer-memory-resource:
+
+Buffers provide access to their underlying memory resource:
+
+- ``memory_resource()`` - Get a const reference to the memory resource
diff --git a/cccl_upstream/docs/libcudacxx/runtime/cudart_interactions.rst b/cccl_upstream/docs/libcudacxx/runtime/cudart_interactions.rst
new file mode 100644
index 00000000..e80f983d
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/runtime/cudart_interactions.rst
@@ -0,0 +1,131 @@
+.. _cccl-runtime-cudart-interactions:
+
+CUDA Runtime interactions
+=========================
+
+Some runtime objects have a non-owning ``_ref`` counterpart (for example, :cpp:struct:`cuda::stream` and
+:cpp:class:`cuda::stream_ref`). Prefer the
+owning type for lifetime management, and use the ``_ref`` type for code that would otherwise accept a C++ reference but
+needs to interoperate with existing CUDA Runtime code.
+
+CCCL runtime types that wrap CUDA Runtime handles support interoperating with CUDA Runtime handles via ``get()``,
+constructors that accept native handles, ``release()``, and ``from_native_handle`` helpers. This makes it straightforward
+to bridge between cccl-runtime APIs and existing CUDA Runtime code without losing ownership clarity.
+
+Use ``get()`` on both owning and non-owning types. Constructors from native handles are intended for ``_ref`` wrappers,
+while ``release()`` and ``from_native_handle`` are for owning objects that transfer or assume ownership.
+
+Example: handle interop patterns
+--------------------------------
+
+.. code:: cpp
+
+   #include <cuda/stream>
+
+   void use_handle_interop(cuda::device_ref device, cudaStream_t raw_stream) {
+     // _ref from native handle (non-owning).
+     cuda::stream_ref borrowed{raw_stream};
+
+     // Universal handle access.
+     assert(borrowed.get() == raw_stream);
+
+     // Owning from native handle (assumes ownership).
+     auto owned = cuda::stream::from_native_handle(raw_stream);
+
+     assert(owned.get() == raw_stream);
+
+     // Release ownership back to CUDA Runtime.
+     cudaStream_t released = owned.release();
+
+     assert(released == raw_stream);
+   }
+
+Error handling
+--------------
+
+CCCL Runtime APIs use C++ exceptions for error handling. Failures from runtime abstractions are reported by throwing an
+exception, so normal code can be written without manually checking and propagating a status code after each operation.
+
+This differs from the traditional CUDA Runtime API, where operations generally return ``cudaError_t`` values that the
+caller must check against ``cudaSuccess`` and propagate or handle. When using CUDA Runtime calls directly, continue to
+check their return values; when using CCCL Runtime wrappers, handle failures with normal C++ exception handling.
+
+At a CUDA Runtime-style boundary, catch ``cuda::cuda_error`` and return its stored status.
+
+.. code:: cpp
+
+   #include <cuda/stream>
+
+   cudaError_t use_stream(cuda::stream_ref stream) noexcept {
+     try {
+       // stream usage
+       stream.sync();
+       return cudaSuccess;
+     } catch (const cuda::cuda_error& err) {
+       return err.status();
+     }
+   }
+
+Device selection
+----------------
+
+The Runtime API emphasizes explicit device selection. Most entry points take a :cpp:class:`cuda::device_ref` or a
+device-bound resource (such as :cpp:struct:`cuda::stream`) rather than relying on implicit global state like
+``cudaSetDevice``. This
+makes device ownership and lifetime clearer, especially in multi-GPU code.
+
+The current device can still be set via the CUDA Runtime, but cccl-runtime APIs ignore that global state and require an
+explicit device argument. cccl-runtime also does not provide APIs that read or mutate the current device, by design.
+
+
+.. _cccl-runtime-cudart-default-stream:
+
+Default stream interop
+----------------------
+
+The CUDA default (NULL) stream is not exposed as a first-class runtime object because it is tied to implicit per-device
+state and encourages hidden dependencies. Instead, it can be wrapped into :cpp:class:`cuda::stream_ref` when needed for
+interop.
+
+.. note::
+
+   When wrapping the NULL stream, the current device must be set explicitly first. CUDA binds the NULL stream to the
+   active device, so the wrapper must be created after selecting the correct device.
+
+Example: wrapping the default stream
+------------------------------------
+
+.. code:: cpp
+
+   #include <cuda/stream>
+
+   void use_default_stream(int device_id) {
+     cudaSetDevice(device_id);
+
+     cuda::stream_ref default_stream{cudaStreamPerThread};
+     // Use default_stream with cccl-runtime APIs.
+   }
+
+The above applies to Driver API interop cases as well, where the current context must be managed by the user rather than
+the current device setting.
+
+
+.. _cccl-runtime-cudart-non-blocking-streams:
+
+Non-blocking stream creation
+----------------------------
+
+Constructing a new :cpp:struct:`cuda::stream` always creates a stream with CUDA Runtime non-blocking behavior. This is
+the behavior of CCCL Runtime-created streams; wrapping or taking ownership of an existing ``cudaStream_t`` preserves the
+behavior of that handle.
+
+In the CUDA Runtime API, the blocking/non-blocking stream creation flag controls synchronization with the CUDA default
+(NULL) stream. Because CCCL Runtime treats the default stream as an interop case rather than a first-class object,
+as described in :ref:`default stream interop <cccl-runtime-cudart-default-stream>`, :cpp:struct:`cuda::stream` does not
+expose a blocking/non-blocking construction option.
+
+New Runtime code should express ordering between explicit streams directly, for example by making one
+:cpp:class:`cuda::stream_ref` wait on another. Code that needs legacy CUDA Runtime implicit stream semantics should wrap
+the relevant CUDA Runtime stream handle in :cpp:class:`cuda::stream_ref` (or take ownership with
+``cuda::stream::from_native_handle``); operations submitted through the wrapper use the same native handle and preserve
+that handle's CUDA Runtime semantics, including any default-stream synchronization semantics.
diff --git a/cccl_upstream/docs/libcudacxx/runtime/device.rst b/cccl_upstream/docs/libcudacxx/runtime/device.rst
new file mode 100644
index 00000000..8726b110
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/runtime/device.rst
@@ -0,0 +1,106 @@
+.. _cccl-runtime-device:
+
+Devices
+========
+
+:cpp:class:`cuda::device_ref`
+-------------------------------
+.. _cccl-runtime-device-device-ref:
+
+:cpp:class:`cuda::device_ref` is a lightweight, non-owning handle to a CUDA device ordinal. It allows to query
+information about a device and serves as an argument to other runtime APIs which are tied to a specific device.
+It offers:
+
+- ``get()``: native device ordinal
+- ``name()``: device name
+- ``init()``: initialize the device context
+- ``peers()``: list peers for which peer access can be enabled
+- ``has_peer_access_to(cuda::device_ref)``: query if peer access can be enabled to the given device
+- ``attribute(attr)`` / ``attribute<::cudaDeviceAttr>()``: attribute queries
+
+Availability: CCCL 3.1.0 / CUDA 13.1
+
+:cpp:var:`cuda::devices`
+----------------------------
+.. _cccl-runtime-device-devices:
+
+:cpp:var:`cuda::devices` is a random-access view of all available CUDA devices in the form of
+:cpp:class:`cuda::device_ref` objects. It
+provides indexing, size, and iteration for use
+in range-based loops.
+
+Availability: CCCL 3.1.0 / CUDA 13.1
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/devices>
+   #include <iostream>
+
+   void print_devices() {
+     for (auto& dev : cuda::devices) {
+       std::cout << "Device " << dev.get() << ": " << dev.name() << '\n';
+     }
+   }
+
+Device attributes
+-----------------
+.. _cccl-runtime-device-attributes:
+
+``cuda::device_attributes`` provides strongly-typed attribute query objects usable with
+:cpp:func:`cuda::device_ref::attribute`. Selected examples:
+
+- ``compute_capability``
+- ``multiprocessor_count``
+- ``concurrent_managed_access``
+- ``clock_rate``
+- ``numa_id``
+
+Availability: CCCL 3.1.0 / CUDA 13.1
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/devices>
+
+   int get_max_blocks_on_device(cuda::device_ref dev) {
+     return cuda::device_attributes::multiprocessor_count(dev) * cuda::device_attributes::blocks_per_multiprocessor(dev);
+   }
+
+:cpp:any:`cuda::arch_traits`
+--------------------------------
+.. _cccl-runtime-device-arch-traits:
+
+Per-architecture trait accessors providing limits and capabilities common to all devices of an architecture.
+Compared to ``cuda::device_attributes``, :cpp:any:`cuda::arch_traits` provide a compile-time accessible
+structure that describes common characteristics of all devices of an architecture, while attributes are run-time
+queries of a single characteristic of a specific device.
+
+- :cpp:any:`cuda::arch_traits` and :cpp:any:`cuda::arch_traits_for` (compile-time and run-time forms).
+- Returns a :cpp:struct:`cuda::arch_traits_t` with fields like
+  ``max_threads_per_block``, ``max_shared_memory_per_block``, ``cluster_supported`` and other capability flags.
+- Traits for the current architecture can be accessed with :cpp:func:`cuda::device::current_arch_traits`
+
+Availability: CCCL 3.1.0 / CUDA 13.1
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/devices>
+
+   template <cuda::arch_id Arch>
+   __device__ void fn() {
+     auto traits = cuda::arch_traits<Arch>();
+     if constexpr (traits.cluster_supported) {
+       // cluster specific code
+     } else {
+       // non-cluster code
+     }
+   }
+
+   __global__ void kernel() {
+     fn<cuda::arch_id::sm_90>();
+   }
diff --git a/cccl_upstream/docs/libcudacxx/runtime/event.rst b/cccl_upstream/docs/libcudacxx/runtime/event.rst
new file mode 100644
index 00000000..0a7f23bb
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/runtime/event.rst
@@ -0,0 +1,81 @@
+.. _cccl-runtime-event:
+
+Events
+======
+
+Event is a snapshot of execution state of a stream. It can be used to synchronize work submitted to a stream up to a certain point, establish dependency between streams or measure time passed between two events.
+
+:cpp:class:`cuda::event_ref`
+--------------------------------------------------
+.. _cccl-runtime-event-event-ref:
+
+:cpp:class:`cuda::event_ref` is a non-owning wrapper around a ``cudaEvent_t``. It prevents unsafe implicit constructions from
+``nullptr`` or integer literals and provides convenient helpers:
+
+- ``record(cuda::stream_ref)``: record the event on a stream
+- ``sync()``: wait for the recorded work to complete
+- ``is_done()``: non-blocking completion query
+- comparison operators against other :cpp:class:`cuda::event_ref` or ``cudaEvent_t``
+
+Availability: CCCL 3.1.0 / CUDA 13.1
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/stream>
+
+   void record_on_stream(cuda::stream_ref stream, cudaEvent_t raw_handle) {
+     cuda::event_ref e{raw_handle};
+     e.record(stream);
+   }
+
+:cpp:class:`cuda::event`
+--------------------------------------------
+.. _cccl-runtime-event-event:
+
+:cpp:class:`cuda::event` is an owning wrapper around a ``cudaEvent_t`` (with timing disabled). It inherits from
+:cpp:class:`cuda::event_ref` and provides all of its functionality. It also creates and destroys the native event, can be moved (but
+not copied), and can release ownership via ``release()``. Construction can target a specific :cpp:class:`cuda::device_ref`
+or record immediately on a :cpp:class:`cuda::stream_ref`.
+
+Availability: CCCL 3.1.0 / CUDA 13.1
+
+.. code:: cpp
+
+   #include <cuda/stream>
+   #include <cuda/devices>
+   #include <cuda/std/optional>
+
+   cuda::std::optional<cuda::event> query_and_record_on_stream(cuda::stream_ref stream) {
+     if (stream.is_done()) {
+       return cuda::std::nullopt;
+     } else {
+       return cuda::event{stream};
+     }
+   }
+
+.. _cccl-runtime-event-timed-event:
+
+:cpp:class:`cuda::timed_event`
+-----------------------------------------------------
+
+:cpp:class:`cuda::timed_event` is an owning wrapper for a timed ``cudaEvent_t``. It inherits from :cpp:class:`cuda::event` and provides
+all of its functionality.
+It also supports elapsed-time queries between two events via ``operator-``, returning
+:cpp:class:`cuda::std::chrono::nanoseconds`.
+
+Availability: CCCL 3.1.0 / CUDA 13.1
+
+.. code:: cpp
+
+   #include <cuda/stream>
+   #include <cuda/std/chrono>
+
+   template <typename F>
+   cuda::std::chrono::nanoseconds measure_execution_time(cuda::stream_ref stream, F&& f) {
+     cuda::timed_event start{stream};
+     f(stream);
+     cuda::timed_event end{stream};
+     return end - start;
+   }
diff --git a/cccl_upstream/docs/libcudacxx/runtime/hierarchy.rst b/cccl_upstream/docs/libcudacxx/runtime/hierarchy.rst
new file mode 100644
index 00000000..0a6f31cc
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/runtime/hierarchy.rst
@@ -0,0 +1,208 @@
+.. _cccl-runtime-hierarchy:
+
+.. |cuda_hierarchy| replace:: ``cuda::hierarchy``
+.. _cuda_hierarchy: ../api/classcuda_1_1hierarchy.html
+
+.. |cuda_make_hierarchy| replace:: ``cuda::make_hierarchy``
+.. _cuda_make_hierarchy: ../api/namespacecuda_1a67bb05480718296ce6aff78859538637.html
+.. |cuda_make_config| replace:: ``cuda::make_config``
+.. _cuda_make_config: ../api/namespacecuda_1aa7b277627ddc60563f1818ae8e05ba2d.html
+.. |cuda_grid_dims| replace:: ``cuda::grid_dims``
+.. _cuda_grid_dims: ../api/namespacecuda_1a9b019989bfafbeec225ccfa07718216d.html
+.. |cuda_cluster_dims| replace:: ``cuda::cluster_dims``
+.. _cuda_cluster_dims: ../api/namespacecuda_1ad240665066f4a89a04af40e66e131ab7.html
+.. |cuda_block_dims| replace:: ``cuda::block_dims``
+.. _cuda_block_dims: ../api/namespacecuda_1a1649d0f7fed34582e19dba72f8c1b3d2.html
+.. |cuda_warp| replace:: ``cuda::warp``
+.. _cuda_warp: ../api/namespacecuda_1a25cebd54f74dcdc131654cb3977a1842.html
+.. |cuda_gpu_thread| replace:: ``cuda::gpu_thread``
+.. _cuda_gpu_thread: ../api/namespacecuda_1a1c4664dbad423f7bd37472020576c17c.html
+.. |cuda_hierarchy_add_level| replace:: ``cuda::hierarchy_add_level``
+.. _cuda_hierarchy_add_level: ../api/namespacecuda_1a2c197f19590504c7fccb5b0a9e8f361a.html
+.. |cuda_get_launch_dimensions| replace:: ``cuda::get_launch_dimensions``
+.. _cuda_get_launch_dimensions: ../api/namespacecuda_1a43e600724a8fbba0b8797014aa0246e9.html
+
+Hierarchy
+=========
+
+The hierarchy API provides abstractions for representing and querying levels in the CUDA thread hierarchy (grid, cluster,
+block, warp, and thread levels). It enables compile-time and runtime queries of thread dimensions and counts across
+different hierarchy levels.
+
+|cuda_hierarchy|_
+---------------------------------------------------------------------
+.. _cccl-runtime-hierarchy-hierarchy:
+
+|cuda_hierarchy|_ is a type representing a hierarchy of CUDA threads. It combines hierarchy level descriptors
+to represent dimensions of a (possibly partial) hierarchy. It supports accessing individual levels and queries
+combining dimensions of multiple levels.
+
+A hierarchy should be created using |cuda_make_hierarchy|_ rather than being constructed directly. The
+hierarchy type can be used by itself, but its main purpose is to be part of a kernel launch configuration described
+here: :ref:`Launch <cccl-runtime-launch>`. In that case, instead of calling |cuda_make_hierarchy|_, the same arguments
+can be passed to |cuda_make_config|_.
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/hierarchy>
+
+   auto h = cuda::make_hierarchy(
+     cuda::grid_dims(256),
+     cuda::block_dims<8, 8, 8>()
+   );
+
+   // Access level dimensions
+   assert(h.level(cuda::grid).dims.x == 256);
+
+   // Query counts across levels
+   static_assert(cuda::gpu_thread.count(cuda::block, h) == 8 * 8 * 8);
+
+|cuda_make_hierarchy|_
+----------------------------------------------------------------------------------------------------
+.. _cccl-runtime-hierarchy-make-hierarchy:
+
+|cuda_make_hierarchy|_ creates a hierarchy from passed hierarchy level descriptors. Levels can be passed in
+ascending or descending order, and the function will automatically order them correctly.
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/hierarchy>
+
+   // Levels can be passed in any order
+   auto h1 = cuda::make_hierarchy(
+     cuda::grid_dims(256),
+     cuda::cluster_dims<4>(),
+     cuda::block_dims<8, 8, 8>()
+   );
+
+   auto h2 = cuda::make_hierarchy(
+     cuda::block_dims<8, 8, 8>(),
+     cuda::cluster_dims<4>(),
+     cuda::grid_dims(256)
+   );
+
+   // Both create equivalent hierarchies
+   static_assert(cuda::std::is_same_v<decltype(h1), decltype(h2)>);
+
+Hierarchy Level Descriptors
+----------------------------
+.. _cccl-runtime-hierarchy-level-descriptors:
+
+The hierarchy API provides level descriptor functions for grid, cluster, and block levels.
+Each level supports both compile-time and runtime dimensions:
+
+- |cuda_grid_dims|_ (compile-time and runtime overload forms)
+- |cuda_cluster_dims|_ (compile-time and runtime overload forms)
+- |cuda_block_dims|_ (compile-time and runtime overload forms)
+
+Warp and thread levels are implicit and are queried via level objects (e.g., |cuda_warp|_,
+|cuda_gpu_thread|_).
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/hierarchy>
+
+   auto h = cuda::make_hierarchy(
+     cuda::grid_dims(256, 128),      // Runtime grid dimensions
+     cuda::cluster_dims<4>(),        // Compile-time cluster dimensions
+     cuda::block_dims<32, 16>()      // Compile-time block dimensions
+   );
+
+Hierarchy Queries
+-----------------
+.. _cccl-runtime-hierarchy-queries:
+
+Hierarchies support various query operations via level objects (``cuda::grid``, ``cuda::cluster``,
+``cuda::block``, |cuda_warp|_, |cuda_gpu_thread|_):
+
+- ``unit.count(level, hierarchy)`` - Count units within a level (e.g., threads per block)
+- ``unit.rank(level, hierarchy)`` - Get the rank (linear index) of a unit within a level (device only)
+- ``unit.dims(level, hierarchy)`` - Get dimensions of units within a level
+- ``hierarchy.level<Level>()`` - Get the level descriptor for a specific level
+- ``hierarchy.fragment<Unit, Level>()`` - Extract a fragment of the hierarchy
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/hierarchy>
+
+   auto h = cuda::make_hierarchy(
+     cuda::grid_dims(256),
+     cuda::block_dims<8, 8, 8>()
+   );
+
+   // Get block-level descriptor
+   auto block_desc = h.level(cuda::block);
+   assert(block_desc.dims.x == 8);
+
+   // Count threads per block
+   static_assert(cuda::gpu_thread.count(cuda::block, h) == 512);
+
+   // Get fragment (block to grid)
+   auto fragment = h.fragment(cuda::block, cuda::grid);
+
+|cuda_hierarchy_add_level|_
+---------------------------------------------------------------------------------------------------------
+.. _cccl-runtime-hierarchy-add-level:
+
+|cuda_hierarchy_add_level|_ returns a new hierarchy that is a copy of the supplied hierarchy with a new level
+added. The function automatically determines whether to add the level at the top or bottom based on the existing
+levels.
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/hierarchy>
+
+   auto partial = cuda::make_hierarchy<cuda::block_level>(
+     cuda::grid_dims(256),
+     cuda::cluster_dims<4>()
+   );
+
+   auto complete = cuda::hierarchy_add_level(
+     partial,
+     cuda::block_dims<8, 8, 8>()
+   );
+
+|cuda_get_launch_dimensions|_
+-----------------------------------------------------------------------------------------------------------
+.. _cccl-runtime-hierarchy-launch-dimensions:
+
+|cuda_get_launch_dimensions|_ returns a tuple of ``hierarchy_query_result`` objects containing dimensions from
+the hierarchy that can be used to launch kernels. The returned tuple has three elements if cluster_level is present
+(grid, cluster, block dimensions), or two elements otherwise (grid, block dimensions).
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/hierarchy>
+
+   auto h = cuda::make_hierarchy(
+     cuda::grid_dims(256),
+     cuda::cluster_dims<4>(),
+     cuda::block_dims<8, 8, 8>()
+   );
+
+   auto [grid_dims, cluster_dims, block_dims] = cuda::get_launch_dimensions(h);
+   // Can be used with cudaLaunchKernel or similar APIs
diff --git a/cccl_upstream/docs/libcudacxx/runtime/launch.rst b/cccl_upstream/docs/libcudacxx/runtime/launch.rst
new file mode 100644
index 00000000..14f27b6b
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/runtime/launch.rst
@@ -0,0 +1,290 @@
+.. _cccl-runtime-launch:
+
+.. |cuda_launch| replace:: ``cuda::launch``
+.. _cuda_launch: ../api/namespacecuda_1afd43c8d92fdb84879aed04f3e2ea25d2.html
+.. |cuda_kernel_config| replace:: ``cuda::kernel_config``
+.. _cuda_kernel_config: ../api/structcuda_1_1kernel__config.html
+.. |cuda_make_config| replace:: ``cuda::make_config``
+.. _cuda_make_config: ../api/namespacecuda_1aa7b277627ddc60563f1818ae8e05ba2d.html
+.. |cuda_cooperative_launch| replace:: ``cuda::cooperative_launch``
+.. _cuda_cooperative_launch: ../api/structcuda_1_1cooperative__launch.html
+.. |cuda_dynamic_shared_memory| replace:: ``cuda::dynamic_shared_memory``
+.. _cuda_dynamic_shared_memory: ../api/namespacecuda_1a737c80f87e6e727a865cd05b82ec2405.html
+.. |cuda_launch_priority| replace:: ``cuda::launch_priority``
+.. _cuda_launch_priority: ../api/structcuda_1_1launch__priority.html
+.. |cuda_host_launch| replace:: ``cuda::host_launch``
+.. _cuda_host_launch: ../api/namespacecuda_1a5af4f59c915edb056f346b904197ff3d.html
+
+Launch
+======
+
+The launch API provides abstractions for launching CUDA kernels with a given configuration. It supports kernel functions and device callable objects, cooperative launches, dynamic shared memory, and other launch options.
+
+|cuda_launch|_
+--------------------------------------------------------------------------------------------
+.. _cccl-runtime-launch-launch:
+
+|cuda_launch|_ launches a kernel function or a device callable object on the specified stream with a given
+configuration. The kernel can accept the configuration as its first argument to enable some device-side functionality,
+but it is not required. If the kernel does accept the configuration as its first argument, |cuda_launch|_
+will automatically pass it into the kernel without the need to pass the configuration as an argument twice.
+
+*Note:* Configuration won't be passed automatically into the kernel if it is an extended device lambda, it needs to be passed as the second launch function argument and as the first kernel argument.
+
+The benefit of using a callable object with a device call operator (later called a kernel functor) is that it can have its
+template arguments deduced from the arguments, while a kernel function needs to be explicitly instantiated. It also
+allows attaching a default configuration that is later combined with the configuration passed to the launch.
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example with kernel function:
+
+.. code:: cpp
+
+   #include <cuda/launch>
+   #include <cstdio>
+
+   template <typename Configuration>
+   __global__ void kernel(Configuration conf, unsigned int thread_to_print) {
+     if (cuda::gpu_thread.rank(cuda::grid, conf) == thread_to_print) {
+       printf("Hello from the GPU\n");
+     }
+   }
+
+   void launch_kernel(cuda::stream_ref stream) {
+     auto config = cuda::make_config(cuda::block_dims<128>(), cuda::grid_dims(4), cuda::cooperative_launch{});
+     // Here the template needs to be explicitly instantiated, unlike in the kernel functor example where it can be deduced
+     cuda::launch(stream, config, kernel<decltype(config)>, 42);
+   }
+
+Example with kernel functor:
+
+.. code:: cpp
+
+   #include <cuda/launch>
+   #include <cstdio>
+
+   struct kernel {
+     template <typename Configuration>
+     __device__ void operator()(Configuration conf, unsigned int thread_to_print) {
+       if (cuda::gpu_thread.rank(cuda::grid, conf) == thread_to_print) {
+         printf("Hello from the GPU\n");
+       }
+     }
+   };
+
+   void launch_kernel(cuda::stream_ref stream) {
+     auto config = cuda::make_config(cuda::block_dims<128>(), cuda::grid_dims(4), cuda::cooperative_launch{});
+     // It's enough to pass the configuration object once and launch will automatically pass it into the kernel
+     cuda::launch(stream, config, kernel{}, 42);
+   }
+
+Example with extended device lambda:
+
+.. code:: cpp
+
+   #include <cuda/launch>
+   #include <cstdio>
+
+   void launch_kernel(cuda::stream_ref stream) {
+     auto config = cuda::make_config(cuda::block_dims<128>(), cuda::grid_dims(4), cuda::cooperative_launch{});
+     auto lambda = [](cuda::config conf, unsigned int thread_to_print) {
+       if (cuda::gpu_thread.rank(cuda::grid, conf) == thread_to_print) {
+         printf("Hello from the GPU\n");
+       }
+     };
+     // Note that the configuration needs to be passed twice, unlike in other examples
+     cuda::launch(stream, config, lambda, config, 42);
+   }
+
+|cuda_kernel_config|_
+-------------------------------------------------------------------------------
+.. _cccl-runtime-launch-kernel-config:
+
+|cuda_kernel_config|_ represents a kernel launch configuration combining hierarchy dimensions and launch
+options. It should be created using |cuda_make_config|_ rather than being constructed directly.
+
+A |cuda_kernel_config|_ provides:
+
+- ``hierarchy()`` - Access to the hierarchy dimensions
+- ``options()`` - Access to launch options
+- ``combine(other_config)`` - Combine with another configuration
+- ``combine_with_default(kernel)`` - Combine with default options from a kernel of a kernel functor accessed via
+  ``kernel.default_config()``, equivalent to ``combine(kernel.default_config())``
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+|cuda_make_config|_
+-------------------------------------------------------------------------------------------------
+.. _cccl-runtime-launch-make-config:
+
+|cuda_make_config|_ creates a kernel configuration from `hierarchy dimensions <cccl-runtime-hierarchy>` and
+optional launch options. It can be called with:
+
+- A hierarchy and options: ``make_config(hierarchy, option1, option2, ...)``
+- Dimensions directly: ``make_config(grid_dims(...), block_dims<...>(), option1, option2, ...)``
+
+In the last case, the dimensions arguments must come first, followed by options.
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/launch>
+   #include <cooperative_groups.h>
+
+   // Create config with cooperative launch
+   auto config1 = cuda::make_config(cuda::grid_dims(256), cuda::cooperative_launch{});
+
+   // Create config with dynamic shared memory
+   auto config2 = cuda::make_config(
+     cuda::block_dims<128>(),
+     cuda::grid_dims(512),
+     cuda::dynamic_shared_memory<float>(1024)
+   );
+
+   // Combine configurations, configuration that combine was called on is prioritized.
+   auto config3 = config1.combine(config2);
+   assert(cuda::gpu_thread.count(cuda::grid, config3) == 256 * 128);
+
+   // Kernel functor can have a default configuration attached to it, that is later combined with the configuration passed to the launch.
+   struct kernel {
+     template <typename Configuration>
+     __global__ void operator()(Configuration conf, unsigned int thread_to_print) {
+       auto grid = cooperative_groups::this_grid();
+       grid.sync();
+       if (cuda::gpu_thread.rank(cuda::grid, conf) == thread_to_print) {
+         printf("Hello from the GPU\n");
+       }
+     }
+
+     auto default_config() const {
+       return cuda::make_config(cuda::block_dims<128>(), cuda::cooperative_launch{});
+     }
+   };
+
+   cuda::launch(stream, cuda::make_config(cuda::grid_dims(4)), kernel{});
+
+Launch Options
+--------------
+.. _cccl-runtime-launch-options:
+
+The launch API provides several launch options:
+
+|cuda_cooperative_launch|_
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Enables cooperative launch, restricting the grid to a number of blocks that can simultaneously execute on the device. This enables usage of ``cooperative_groups::grid_group::sync()`` in the kernel. This is a struct that can be default-constructed.
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/launch>
+   #include <cooperative_groups.h>
+
+   template <typename Configuration>
+   __global__ void kernel(Configuration conf) {
+     auto grid = cooperative_groups::this_grid();
+     grid.sync();
+   }
+
+   void launch(cuda::stream_ref stream) {
+     auto config = cuda::make_config(cuda::block_dims<128>(), cuda::grid_dims(4), cuda::cooperative_launch{});
+     cuda::launch(stream, config, kernel<decltype(config)>);
+   }
+
+|cuda_dynamic_shared_memory|_
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Specifies dynamic shared memory configuration. It provides a type-safe way to specify shared memory content and later access it through the configuration object passed to the kernel.
+
+- For non-array ``T`` (e.g., a struct), call |cuda_dynamic_shared_memory|_ with no size argument.
+- For bounded array ``T[n]`` (e.g., ``int[10]``), call |cuda_dynamic_shared_memory|_ with no size argument.
+- For unbounded array ``T[]`` (e.g., ``float[]``), pass the element count to |cuda_dynamic_shared_memory|_.
+- To opt in to non-portable dynamic shared memory sizes (greater than 48 KiB per block), pass
+  :cpp:any:`cuda::non_portable` to |cuda_dynamic_shared_memory|_.
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/launch>
+
+   template <typename Configuration>
+   __global__ void kernel(Configuration conf) {
+     auto smem = cuda::dynamic_shared_memory(conf);
+     // Use smem as span<T> in case of an array type or T& in case of a non-array type
+   }
+
+   void launch(cuda::stream_ref stream) {
+     auto config = cuda::make_config(
+     cuda::block_dims<128>(),
+     cuda::grid_dims(4),
+     cuda::dynamic_shared_memory<float[]>(1024)
+     );
+     cuda::launch(stream, config, kernel<decltype(config)>);
+   }
+
+   void launch_non_portable(cuda::stream_ref stream) {
+     auto config = cuda::make_config(
+     cuda::block_dims<128>(),
+     cuda::grid_dims(4),
+     cuda::dynamic_shared_memory<float[]>(32768, cuda::non_portable)
+     );
+     cuda::launch(stream, config, kernel<decltype(config)>);
+   }
+
+|cuda_launch_priority|_
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Specifies the priority launch option used when scheduling the kernel launch. Overrides the priority specified in the stream.
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/launch>
+
+   auto config = cuda::make_config(
+     cuda::block_dims<128>(),
+     cuda::grid_dims(4),
+     cuda::launch_priority{0}
+   );
+
+|cuda_host_launch|_
+-------------------------------------------------------------------------------------------------
+.. _cccl-runtime-launch-host-launch:
+
+|cuda_host_launch|_ launches a host callable for a stream-ordered execution. The callable can be a lambda
+function, a function pointer, or a callable object.
+The callable and arguments are taken by value and stored for later execution. This requires a dynamic allocation to store the callable and arguments. If the callable is a function pointer or cuda::std::reference_wrapper and there are no arguments, the dynamic allocation is avoided.
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/launch>
+   #include <iostream>
+
+   cuda::host_launch(stream, [](int arg) {
+     std::cout << "Callback executed" << '\n';
+     std::cout << "Argument: " << arg << '\n';
+   }, 42);
+
+   // Passing by reference requires using cuda::std::ref without arguments to avoid dynamic allocation,
+   // but the callable must live long enough for the callable to execute.
+   int arg = 42;
+   auto lambda = [&arg]() {
+     std::cout << "Callback executed" << '\n';
+     std::cout << "Argument: " << arg << '\n';
+   };
+   cuda::host_launch(stream, cuda::std::ref(lambda));
+   stream.sync();
diff --git a/cccl_upstream/docs/libcudacxx/runtime/legacy_resources.rst b/cccl_upstream/docs/libcudacxx/runtime/legacy_resources.rst
new file mode 100644
index 00000000..244a31b1
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/runtime/legacy_resources.rst
@@ -0,0 +1,50 @@
+.. _cccl-runtime-legacy-resources:
+.. _libcudacxx-extended-api-memory-resources-legacy-resources:
+
+Legacy resources
+================
+
+Legacy memory resources provide synchronous allocation interfaces backed by the CUDA Runtime's legacy allocation APIs.
+They are primarily intended for compatibility with older toolkits or platforms that do not support the newer memory
+pool-based resources. Prefer the modern memory resources where available.
+
+For the full memory resource model and property system, see
+:ref:`Memory Resources (Extended API) <libcudacxx-extended-api-memory-resources>`.
+
+:cpp:class:`cuda::mr::legacy_pinned_memory_resource`
+------------------------------------------------------
+.. _libcudacxx-memory-resource-legacy-pinned-memory-resource:
+
+Provides pinned (page-locked) host allocations using ``cudaMallocHost`` and ``cudaFreeHost``. This resource is
+*synchronous-only* and is intended as a compatibility fallback. For CUDA 12.9 and later, prefer
+:cpp:any:`cuda::pinned_memory_resource`.
+
+.. code:: cpp
+
+   #include <cuda/memory_resource>
+
+   void use_legacy_pinned() {
+     cuda::mr::legacy_pinned_memory_resource resource{};
+     void* ptr = resource.allocate_sync(1024, 64);
+     // Use memory...
+     resource.deallocate_sync(ptr, 1024, 64);
+   }
+
+:cpp:class:`cuda::mr::legacy_managed_memory_resource`
+-------------------------------------------------------
+.. _libcudacxx-memory-resource-legacy-managed-memory-resource:
+
+Provides managed (unified) allocations using ``cudaMallocManaged`` and ``cudaFree``. This resource is
+*synchronous-only* and accepts the CUDA attachment flags (``cudaMemAttachGlobal`` / ``cudaMemAttachHost``). Prefer
+:cpp:any:`cuda::managed_memory_resource` when available.
+
+.. code:: cpp
+
+   #include <cuda/memory_resource>
+
+   void use_legacy_managed() {
+     cuda::mr::legacy_managed_memory_resource resource{cudaMemAttachGlobal};
+     void* ptr = resource.allocate_sync(1024, 64);
+     // Use memory...
+     resource.deallocate_sync(ptr, 1024, 64);
+   }
diff --git a/cccl_upstream/docs/libcudacxx/runtime/memory_pools.rst b/cccl_upstream/docs/libcudacxx/runtime/memory_pools.rst
new file mode 100644
index 00000000..4208a17b
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/runtime/memory_pools.rst
@@ -0,0 +1,366 @@
+.. _cccl-runtime-memory-pools:
+
+.. |cuda_memory_pool_attributes| replace:: ``cuda::memory_pool_attributes``
+.. _cuda_memory_pool_attributes: ../api/memory_pool_attributes.html
+
+Memory Pools
+============
+
+Memory pools provide efficient, stream-ordered memory allocation using CUDA's memory pool API. They support both synchronous and stream-ordered allocation/deallocation and can be configured with various memory spaces, properties and attributes.
+
+Memory pool objects implement the :ref:`cuda::mr::resource <libcudacxx-extended-api-memory-resources-resource>`
+interface with ``allocate(stream, size, alignment)`` and ``deallocate(stream, ptr, size, alignment)`` member
+functions. They also provide synchronous variants with ``allocate_sync(size, alignment)`` and
+``deallocate_sync(ptr, size, alignment)`` member functions. For all of them, the alignment argument is optional.
+
+For the full memory resource model and property system, see :ref:`Memory Resources (Extended API) <libcudacxx-extended-api-memory-resources>`.
+
+Host memory pools are supported on CUDA 12.9 and later. Managed memory pools are supported on CUDA 13.0 and later and are not supported on Windows. For those cases use :ref:`cuda::mr::legacy_pinned_memory_resource <libcudacxx-memory-resource-legacy-pinned-memory-resource>` and :ref:`cuda::mr::legacy_managed_memory_resource <libcudacxx-memory-resource-legacy-managed-memory-resource>` instead.
+
+:cpp:struct:`cuda::device_memory_pool`
+---------------------------------------
+.. _cccl-runtime-memory-pools-device-memory-pool:
+
+:cpp:struct:`cuda::device_memory_pool` allocates device memory using CUDA's stream-ordered memory pool API
+(``cudaMallocFromPoolAsync`` / ``cudaFreeAsync``). When constructed, it creates and owns an underlying
+``cudaMemPool_t`` with location type set to ``cudaMemLocationTypeDevice``.
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/memory_resource>
+   #include <cuda/stream>
+   #include <cuda/devices>
+
+   void use_device_pool(cuda::stream_ref stream) {
+     // Create a device memory pool
+     cuda::device_memory_pool pool{cuda::devices[0]};
+
+     // Allocate memory in stream order
+     void* ptr = pool.allocate(stream, 1024, 16);
+
+     // Use memory...
+
+     // Deallocate in stream order
+     pool.deallocate(stream, ptr, 1024, 16);
+   }
+
+:cpp:class:`cuda::device_memory_pool_ref`
+-------------------------------------------
+.. _cccl-runtime-memory-pools-device-memory-pool-ref:
+
+:cpp:class:`cuda::device_memory_pool_ref` is a non-owning reference to a device memory pool. It does not own the
+underlying ``cudaMemPool_t``, so the user must ensure the pool's lifetime exceeds the reference's lifetime.
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/memory_resource>
+   #include <cuda/stream>
+
+   void use_pool_ref(cuda::stream_ref stream, cuda::device_memory_pool_ref pool_ref) {
+     void* ptr = pool_ref.allocate(stream, 1024);
+     // Use memory...
+     pool_ref.deallocate(stream, ptr, 1024);
+   }
+
+:cpp:struct:`cuda::managed_memory_pool`
+----------------------------------------
+.. _cccl-runtime-memory-pools-managed-memory-pool:
+
+:cpp:struct:`cuda::managed_memory_pool` allocates managed (unified) memory using CUDA's memory pool API. It creates and
+owns an underlying ``cudaMemPool_t`` with allocation type set to ``cudaMemAllocationTypeManaged``. Managed memory is
+accessible from both host and device.
+
+Availability: CCCL 3.2.0 / CUDA 13.2 (requires CTK 13.0+). Not supported on Windows
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/memory_resource>
+   #include <cuda/stream>
+
+   void use_managed_pool(cuda::stream_ref stream) {
+     cuda::managed_memory_pool pool{};
+
+     // Allocate managed memory
+     void* ptr = pool.allocate(stream, 1024);
+
+     // Accessible from both host and device
+     // Use memory...
+
+     pool.deallocate(stream, ptr, 1024);
+   }
+
+:cpp:class:`cuda::managed_memory_pool_ref`
+--------------------------------------------
+.. _cccl-runtime-memory-pools-managed-memory-pool-ref:
+
+:cpp:class:`cuda::managed_memory_pool_ref` is a non-owning reference to a managed memory pool.
+
+Availability: CCCL 3.2.0 / CUDA 13.2 (requires CTK 13.0+). Not supported on Windows
+
+:cpp:struct:`cuda::pinned_memory_pool`
+---------------------------------------
+.. _cccl-runtime-memory-pools-pinned-memory-pool:
+
+:cpp:struct:`cuda::pinned_memory_pool` allocates pinned (page-locked) host memory using CUDA's memory pool API. Pinned
+memory enables faster host-to-device transfers and can be accessed from all devices. The pool can be optionally
+created for a specific host NUMA node.
+
+Availability: CCCL 3.2.0 / CUDA 13.2 (requires CTK 12.9+)
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/memory_resource>
+   #include <cuda/stream>
+
+   void use_pinned_pool(cuda::stream_ref stream) {
+     // Create pinned memory pool
+     cuda::pinned_memory_pool pool{};
+
+     // Allocate pinned memory
+     void* ptr = pool.allocate(stream, 1024);
+
+     // Use for fast host-device transfers...
+
+     pool.deallocate(stream, ptr, 1024);
+   }
+
+   // With NUMA node
+   void use_pinned_pool_numa(cuda::stream_ref stream, int numa_id) {
+     cuda::pinned_memory_pool pool{numa_id};
+     void* ptr = pool.allocate(stream, 1024);
+     // Use memory...
+     pool.deallocate(stream, ptr, 1024);
+   }
+
+:cpp:class:`cuda::pinned_memory_pool_ref`
+-------------------------------------------
+.. _cccl-runtime-memory-pools-pinned-memory-pool-ref:
+
+:cpp:class:`cuda::pinned_memory_pool_ref` is a non-owning reference to a pinned memory pool.
+
+Availability: CCCL 3.2.0 / CUDA 13.2 (requires CTK 12.9+)
+
+Default Memory Pools
+--------------------
+.. _cccl-runtime-memory-pools-default-pools:
+
+CUDA provides default memory pools for each memory type. These pools are managed by the CUDA runtime and can be accessed through helper functions. Default pools are useful when you don't need custom pool configuration and want to use the system defaults.
+
+:cpp:func:`cuda::device_default_memory_pool`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. _cccl-runtime-memory-pools-device-default:
+
+:cpp:func:`cuda::device_default_memory_pool` returns a non-owning reference to the default device memory pool for the
+specified device. The default pool is created automatically by CUDA and is shared across all users of the device.
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/memory_resource>
+   #include <cuda/devices>
+   #include <cuda/stream>
+
+   void use_default_device_pool(cuda::stream_ref stream) {
+     // Get the default device memory pool
+     auto pool = cuda::device_default_memory_pool(cuda::devices[0]);
+
+     // Allocate from the default pool
+     void* ptr = pool.allocate(stream, 1024);
+
+     // Use memory...
+
+     // Deallocate back to the pool
+     pool.deallocate(stream, ptr, 1024);
+   }
+
+:cpp:func:`cuda::managed_default_memory_pool`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. _cccl-runtime-memory-pools-managed-default:
+
+:cpp:func:`cuda::managed_default_memory_pool` returns a non-owning reference to the default managed (unified) memory
+pool.
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/memory_resource>
+   #include <cuda/stream>
+
+   void use_default_managed_pool(cuda::stream_ref stream) {
+     // Get the default managed memory pool
+     auto pool = cuda::managed_default_memory_pool();
+
+     // Allocate managed memory
+     void* ptr = pool.allocate(stream, 1024);
+
+     // Accessible from both host and device
+     // Use memory...
+
+     pool.deallocate(stream, ptr, 1024);
+   }
+
+:cpp:func:`cuda::pinned_default_memory_pool`
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. _cccl-runtime-memory-pools-pinned-default:
+
+:cpp:func:`cuda::pinned_default_memory_pool` returns a non-owning reference to the default pinned (page-locked) host
+memory pool.
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/memory_resource>
+   #include <cuda/stream>
+
+   void use_default_pinned_pool(cuda::stream_ref stream) {
+     // Get the default pinned memory pool
+     auto pool = cuda::pinned_default_memory_pool();
+
+     // Allocate pinned memory
+     void* ptr = pool.allocate(stream, 1024);
+
+     // Use for fast host-device transfers...
+
+     pool.deallocate(stream, ptr, 1024);
+   }
+
+Notes on Default Pools
+~~~~~~~~~~~~~~~~~~~~~~
+
+- Default pools are created automatically by CUDA and shared across the application
+- The pools are returned as non-owning references (``*_pool_ref`` types)
+- Default pools use CUDA's default configuration and cannot be destroyed
+- Multiple calls to the same getter function return references to the same pool
+- Default pools are thread-safe and can be used concurrently from multiple threads
+- Underlying CUDA default memory pools have 0 release threshold by default. First access to a default pool through one of the getters above will set the release threshold to the maximum value, unless previously modified by the user.
+
+Memory Pool Properties
+----------------------
+.. _cccl-runtime-memory-pools-pool-properties:
+
+:cpp:struct:`cuda::memory_pool_properties` controls memory pool creation options:
+
+- ``initial_pool_size`` - Initial size of the pool (default: 0)
+- ``release_threshold`` - Threshold at which unused memory is released (default: no limit on the reserved memory)
+- ``allocation_handle_type`` - Handle type for inter-process sharing (default: none)
+- ``max_pool_size`` - Maximum size of the pool (default: no limit on the pool size)
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/memory_resource>
+   #include <cuda/devices>
+
+   void create_pool_with_properties() {
+     cuda::memory_pool_properties props{};
+     props.initial_pool_size = 1024 * 1024;  // 1 MB initial size
+     props.release_threshold = 10 * 1024 * 1024;  // Release if over 10 MB
+
+     cuda::device_memory_pool pool{cuda::devices[0], props};
+   }
+
+Memory Pool Attributes
+----------------------
+.. _cccl-runtime-memory-pools-pool-attributes:
+
+|cuda_memory_pool_attributes|_ provides access to pool attributes for
+querying and configuration:
+
+- ``release_threshold`` - Get/set the release threshold, which controls how much memory the pool can keep reserved, both used and unused
+- ``reuse_follow_event_dependencies`` - Enable/disable reuse across streams with event dependencies
+- ``reuse_allow_opportunistic`` - Enable/disable opportunistic reuse
+- ``reuse_allow_internal_dependencies`` - Enable/disable reuse with internal dependencies
+- ``reserved_mem_current`` - Query current reserved memory (read-only)
+- ``used_mem_current`` - Query current used memory (read-only)
+- ``reserved_mem_high`` - Get/set high watermark for reserved memory
+- ``used_mem_high`` - Get/set high watermark for used memory
+
+The following additional read-only creation attributes are available with CUDA 13.3:
+
+- ``allocation_type`` - Query the allocation type of the pool
+- ``export_handle_types`` - Query the export handle types available for the pool
+- ``location`` - Query the location of the pool as a ``cuda::memory_location``
+- ``location_id`` - Query only the location id of the pool
+- ``location_type`` - Query only the location type of the pool
+- ``max_pool_size`` - Query the maximum size of the pool
+- ``hw_decompress_enabled`` - Query whether hardware decompression is enabled for the pool
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/memory_resource>
+   #include <cuda/devices>
+
+   void configure_pool_attributes() {
+     cuda::device_memory_pool pool{cuda::devices[0]};
+
+     // Set release threshold
+     pool.set_attribute(cuda::memory_pool_attributes::release_threshold, 5 * 1024 * 1024);
+
+     // Enable opportunistic reuse
+     pool.set_attribute(cuda::memory_pool_attributes::reuse_allow_opportunistic, true);
+
+     // Query current usage
+     auto reserved = pool.attribute(cuda::memory_pool_attributes::reserved_mem_current);
+     auto used = pool.attribute(cuda::memory_pool_attributes::used_mem_current);
+   }
+
+Pool Management
+---------------
+.. _cccl-runtime-memory-pools-pool-management:
+
+Memory pools provide additional management functions:
+
+- ``trim_to(min_bytes)`` - Release memory down to a minimum size
+- ``enable/disable_access_from(devices)`` - Enable or disable access from specific devices (for peer access or access to host pinned memory)
+- ``get()`` - Get the underlying ``cudaMemPool_t`` handle
+- ``release()`` - Release ownership of the pool handle
+
+Availability: CCCL 3.2.0 / CUDA 13.2
+
+Example:
+
+.. code:: cpp
+
+   #include <cuda/memory_resource>
+   #include <cuda/devices>
+
+   void manage_pool() {
+     cuda::pinned_memory_pool pool{};
+
+     // Enable access from all devices
+     pool.enable_access_from(cuda::devices);
+
+     // Trim pool to 1 MB minimum
+     pool.trim_to(1024 * 1024);
+
+     // Get native handle
+     cudaMemPool_t handle = pool.get();
+   }
diff --git a/cccl_upstream/docs/libcudacxx/runtime/stream.rst b/cccl_upstream/docs/libcudacxx/runtime/stream.rst
new file mode 100644
index 00000000..6b8a1eaa
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/runtime/stream.rst
@@ -0,0 +1,70 @@
+.. _cccl-runtime-stream:
+
+Streams
+========
+
+Stream is conceptually a queue of operations for a specific device. It is passed as an argument to all asynchronous operations like kernel launch, memory copy and allocations.
+
+:cpp:class:`cuda::stream_ref`
+-------------------------------
+.. _cccl-runtime-stream-stream-ref:
+
+:cpp:class:`cuda::stream_ref` is a non-owning wrapper around a ``cudaStream_t``. It prevents unsafe implicit constructions from
+``nullptr`` or integer literals and provides convenient helpers for:
+
+- ``sync()``: wait for the recorded work to complete
+- ``is_done()``: non-blocking completion query
+- comparison operators against other :cpp:class:`cuda::stream_ref` or ``cudaStream_t``
+
+Availability: CCCL 2.2.0 / CUDA 12.3
+
+Example:
+
+.. code:: cpp
+
+    #include <cuda/stream>
+
+    cudaStream_t stream;
+    cudaStreamCreate(&stream);
+    cuda::stream_ref ref{stream};
+
+    ref.sync();            // synchronizes the stream via cudaStreamSynchronize
+    assert(ref.is_done()); // verifies that the stream has finished all operations via cudaStreamQuery
+
+    // compare against other stream_ref or cudaStream_t
+    assert(ref == stream);
+    assert(ref != cuda::invalid_stream);
+
+    cudaStreamDestroy(stream);
+
+:cpp:struct:`cuda::stream`
+---------------------------
+.. _cccl-runtime-stream-stream:
+
+:cpp:struct:`cuda::stream` is an owning wrapper around a ``cudaStream_t`` that manages the lifetime of the underlying CUDA
+stream.
+It derives from :cpp:class:`cuda::stream_ref`, provides all of its functionality, and can be used anywhere a
+:cpp:class:`cuda::stream_ref` is expected.
+It can be constructed for a specific :cpp:class:`cuda::device_ref`, moved (but not copied), and converted from or to a
+``cudaStream_t`` via ``from_native_handle``/``release()``.
+Constructing a new :cpp:struct:`cuda::stream` always creates a non-blocking stream; see
+:ref:`non-blocking stream creation <cccl-runtime-cudart-non-blocking-streams>` for CUDA Runtime interop details.
+
+Availability: CCCL 3.1.0 / CUDA 13.1
+
+.. code:: cpp
+
+   #include <cuda/stream>
+   #include <cuda/devices>
+
+   int main() {
+     {
+       // Create a stream on a specific device
+       cuda::stream s{cuda::devices[0]};
+
+      // Pass to a stream-ordered API
+
+       // Synchronize the stream
+       s.sync();
+     } // Stream is automatically destroyed here
+   }
diff --git a/cccl_upstream/docs/libcudacxx/setup.rst b/cccl_upstream/docs/libcudacxx/setup.rst
new file mode 100644
index 00000000..8de0a5da
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/setup.rst
@@ -0,0 +1,10 @@
+.. _libcudacxx-setup:
+
+Setup
+=====
+
+.. toctree::
+   :maxdepth: 1
+
+   setup/getting
+   setup/building_and_testing
diff --git a/cccl_upstream/docs/libcudacxx/setup/building_and_testing.rst b/cccl_upstream/docs/libcudacxx/setup/building_and_testing.rst
new file mode 100644
index 00000000..09045424
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/setup/building_and_testing.rst
@@ -0,0 +1,44 @@
+.. _libcudacxx-setup-building:
+
+Building & Testing libcu++
+==========================
+
+libcu++ can be build and tested as shown in our `contributor guidelines <https://github.com/NVIDIA/cccl/blob/main/CONTRIBUTING.md#building-and-testing>`_.
+
+However, often only a small subset of the full test suite needs to be run during development. For that we rely on ``lit``.
+After libcu++ has been configured either through the build scripts or directly via a cmake preset one can then run.
+
+.. code:: bash
+
+   cd build
+   lit libcudacxx/RELATIVE_PATH_TO_TEST_OR_SUBFOLDER -sv
+
+This will build and run all tests within ``RELATIVE_PATH_TO_TEST_OR_SUBFOLDER`` which must be a valid path within the CCCL.
+Note that the name of the top level folder is the same as the name of the preset. For the build script the default is
+``libcudacxx``. As an example this is how to run all tests for ``cuda::std::span``, which are located in
+``libcudacxx/test/libcudacxx/std/containers/views/views.span``
+
+.. code:: bash
+
+   cd build
+
+   # Builds all tests within libcudacxx/test/libcudacxx/std/containers/views/views.span
+   lit libcudacxx/libcudacxx/test/libcudacxx/std/containers/views/views.span -sv
+
+   # Builds the individual test array.pass.cpp
+   lit libcudacxx/libcudacxx/test/libcudacxx/std/containers/views/views.span/span.cons/array.pass.cpp -sv
+
+If only building the tests and not running them is desired one can pass ``-Dexecutor="NoopExecutor()"`` to the lit invocation.
+This is especially useful if the machine has no GPU or testing a different architecture
+
+.. code:: bash
+
+   cd build
+   lit libcudacxx/RELATIVE_PATH_TO_TEST_OR_SUBFOLDER -sv -Dexecutor="NoopExecutor()"
+
+Finally different standard modes can be tested by passing e.g ``--param=std=c++20``
+
+.. code:: bash
+
+   cd build
+   lit libcudacxx/RELATIVE_PATH_TO_TEST_OR_SUBFOLDER -sv --param=std=c++20
diff --git a/cccl_upstream/docs/libcudacxx/setup/getting.rst b/cccl_upstream/docs/libcudacxx/setup/getting.rst
new file mode 100644
index 00000000..b29d9b8f
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/setup/getting.rst
@@ -0,0 +1,18 @@
+.. _libcudacxx-setup-getting:
+
+Getting libcu++
+===============
+
+NVIDIA HPC SDK or CUDA Toolkit
+------------------------------
+
+libcu++ is included in the NVIDIA HPC SDK and the CUDA Toolkit. It is on
+the default include path. Today, there is no shared library component to
+libcu++; it's all header-only. No additional compiler flags are needed.
+
+GitHub
+------
+
+libcu++ is an open source project developed on GitHub, which is where
+you'll find the latest versions and the development branch. Our GitHub
+repository is `github.com/nvidia/cccl <https://github.com/nvidia/cccl>`_.
diff --git a/cccl_upstream/docs/libcudacxx/standard_api.rst b/cccl_upstream/docs/libcudacxx/standard_api.rst
new file mode 100644
index 00000000..2f353730
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api.rst
@@ -0,0 +1,152 @@
+.. _libcudacxx-standard-api:
+
+Standard API
+============
+
+.. toctree::
+   :hidden:
+   :maxdepth: 2
+
+   standard_api/algorithms_library
+   standard_api/c_library
+   standard_api/concepts_library
+   standard_api/container_library
+   standard_api/execution_library
+   standard_api/numerics_library
+   standard_api/ranges_library
+   standard_api/synchronization_library
+   standard_api/time_library
+   standard_api/type_support
+   standard_api/utility_library
+
+Standard Library Backports
+--------------------------
+
+C++ Standard versions include new language features and new library features. As the name implies, language features
+are new features of the language the require compiler support. Library features are simply new additions to the
+Standard Library that typically do not rely on new language features nor require compiler support and could conceivably
+be implemented in an older C++ Standard.
+
+Typically, library features are only available in the particular C++ Standard version (or newer) in which they were
+introduced, even if the library features do not depend on any particular language features.
+
+In effort to make library features available to a broader set of users, the NVIDIA C++ Standard Library relaxes this
+restriction. libcu++ makes a best-effort to provide access to C++ Standard Library features in older C++ Standard
+versions than they were introduced. For example, the calendar functionality added to ``<chrono>`` in C++20 is made
+available in C++14.
+
+----
+
+.. |V| unicode:: U+2705
+
+C++ Feature Availability Summary
+--------------------------------
+
+.. table::
+    :widths: 25 25 8 8 8 8 20
+
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | **Library**                                                                        | ``Libcu++``                                                                          |  **≤ C++17**   +   **C++20** |   **C++23** |   **C++26** | **Notes**                                                                                                      |
+    +====================================================================================+======================================================================================+================+=============+=============+=============+================================================================================================================+
+    | `<array> <https://en.cppreference.com/w/cpp/header/array>`_                        | :ref:`<cuda/std/array> <libcudacxx-standard-api-container-array>`                    |   |V|          |  |V|        |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<atomic> <https://en.cppreference.com/w/cpp/header/atomic>`_                      | ``<cuda/std/atomic>``                                                                |   |V|          |  |V|        |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<barrier> <https://en.cppreference.com/w/cpp/header/barrier>`_                    | ``<cuda/std/barrier>``                                                               |                |  |V|        |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<bit> <https://en.cppreference.com/w/cpp/header/bit>`_                            | :ref:`<cuda/std/bit> <libcudacxx-standard-api-numerics-bit>`                         |                |  |V|        |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<bitset> <https://en.cppreference.com/w/cpp/header/bitset>`_                      | :ref:`<cuda/std/bitset> <libcudacxx-standard-api-utility-bitset>`                    |   |V|          |  |V|        |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<charconv> <https://en.cppreference.com/w/cpp/header/charconv>`_                  | ``<cuda/std/charconv>``                                                              |   |V|          |             |  |V|        |  |V|        | Converting floating point types is currently unsupported                                                       |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<chrono> <https://en.cppreference.com/w/cpp/header/chrono>`_                      | :ref:`<cuda/std/chrono> <libcudacxx-standard-api-time>`                              |   |V|          |  |V|        |             |             | Timezone and clocks added in C++20 are not available                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<complex> <https://en.cppreference.com/w/cpp/header/complex>`_                    | :ref:`<cuda/std/complex> <libcudacxx-standard-api-numerics-complex>`                 |   |V|          |  |V|        |             |  |V|        | ``constexpr`` if ``is_constant_evaluated`` is supported                                                        |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<concepts> <https://en.cppreference.com/w/cpp/header/concepts>`_                  | :ref:`<cuda/std/concepts> <libcudacxx-standard-api-concepts>`                        |                |  |V|        |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<execution> <https://en.cppreference.com/w/cpp/header/execution>`_                | :ref:`<cuda/std/execution> <libcudacxx-standard-api-execution>`                      |                |             |             |  |V|        | Only ``cuda::std::execution::prop``, ``cuda::std::execution::env``, ``cuda::std::execution::get_env`` for now  |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<expected> <https://en.cppreference.com/w/cpp/header/expected>`_                  | :ref:`<cuda/std/expected> <libcudacxx-standard-api-utility-expected>`                |                |             |  |V|        |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<functional> <https://en.cppreference.com/w/cpp/header/functional>`_              | :ref:`<cuda/std/functional> <libcudacxx-standard-api-utility-functional>`            |   |V|          |             |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<initializer_list> <https://en.cppreference.com/w/cpp/header/initializer_list>`_  | ``<cuda/std/initializer_list>``                                                      |   |V|          |             |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<inplace_vector> <https://en.cppreference.com/w/cpp/header/inplace_vector>`_      | :ref:`<cuda/std/inplace_vector> <libcudacxx-standard-api-container-inplace-vector>`  |                |             |             |  |V|        |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<iterator> <https://en.cppreference.com/w/cpp/header/iterator>`_                  | ``<cuda/std/iterator>``                                                              |   |V|          |             |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<latch> <https://en.cppreference.com/w/cpp/header/latch>`_                        | ``<cuda/std/latch>``                                                                 |                |  |V|        |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<limits> <https://en.cppreference.com/w/cpp/header/limits>`_                      | ``<cuda/std/limits>``                                                                |   |V|          |             |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<linalg> <https://en.cppreference.com/w/cpp/header/linalg>`_                      | :ref:`<cuda/std/linalg> <libcudacxx-standard-api-numerics-linalg>`                   |                |             |             |  |V|        | Accessors, transposed layout, and related functions                                                            |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<mdspan> <https://en.cppreference.com/w/cpp/header/mdspan>`_                      | :ref:`<cuda/std/mdspan> <libcudacxx-standard-api-container-mdspan>`                  |                |             |  |V|        |  |V|        |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<memory> <https://en.cppreference.com/w/cpp/header/memory>`_                      | :ref:`<cuda/std/memory> <libcudacxx-standard-api-utility-memory>`                    |   |V|          |  |V|        |             |             | ``cuda::std::addressof``, ``cuda::std::align``, ``cuda::std::assume_aligned``, Uninitialized memory algorithms |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<numbers> <https://en.cppreference.com/w/cpp/header/numbers>`_                    | :ref:`<cuda/std/numbers> <libcudacxx-standard-api-numerics-numbers>`                 |                |  |V|        |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<numeric> <https://en.cppreference.com/w/cpp/header/numeric>`_                    | :ref:`<cuda/std/numeric> <libcudacxx-standard-api-numerics-numeric>`                 |   |V|          |             |             |  |V|        |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<optional> <https://en.cppreference.com/w/cpp/header/optional>`_                  | :ref:`<cuda/std/optional> <libcudacxx-standard-api-utility-optional>`                |   |V|          |             |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<ranges> <https://en.cppreference.com/w/cpp/header/ranges>`_                      | :ref:`<cuda/std/ranges> <libcudacxx-standard-api-ranges>`                            |   |V|          |  |V|        |             |             | Requires C++20, Range based algorithms and views are not provided                                              |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<ratio> <https://en.cppreference.com/w/cpp/header/ratio>`_                        | ``<cuda/std/ratio>``                                                                 |   |V|          |             |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<random> <https://en.cppreference.com/w/cpp/header/random>`_                      | :ref:`<cuda/std/random> <libcudacxx-standard-api-numerics-random>`                   |   |V|          |             |             |             | Includes some random engines and most random distributions                                                     |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<semaphore> <https://en.cppreference.com/w/cpp/header/semaphore>`_                | ``<cuda/std/semaphore>``                                                             |                |  |V|        |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<source_location> <https://en.cppreference.com/w/cpp/header/source_location>`_    | ``<cuda/std/source_location>``                                                       |                |  |V|        |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<span> <https://en.cppreference.com/w/cpp/header/span>`_                          | :ref:`<cuda/std/span> <libcudacxx-standard-api-container-span>`                      |                |  |V|        |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<tuple> <https://en.cppreference.com/w/cpp/header/tuple>`_                        | :ref:`<cuda/std/tuple> <libcudacxx-standard-api-utility-tuple>`                      |   |V|          |             |  |V|        |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<type_traits> <https://en.cppreference.com/w/cpp/header/type_traits>`_            | :ref:`<cuda/std/type_traits> <libcudacxx-standard-api-utility-type-traits>`          |   |V|          |  |V|        |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<utility> <https://en.cppreference.com/w/cpp/header/utility>`_                    | :ref:`<cuda/std/utility> <libcudacxx-standard-api-utility-utility>`                  |   |V|          |  |V|        |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<variant> <https://en.cppreference.com/w/cpp/header/variant>`_                    | :ref:`<cuda/std/variant> <libcudacxx-standard-api-utility-variant>`                  |   |V|          |             |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+    | `<version> <https://en.cppreference.com/w/cpp/header/version>`_                    | ``<cuda/std/version>``                                                               |   |V|          |             |             |             |                                                                                                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+----------------+-------------+-------------+-------------+----------------------------------------------------------------------------------------------------------------+
+
+----
+
+C Feature Availability Summary
+------------------------------
+
+.. table::
+    :widths: 25 25 10
+
+    +--------------------------------------------------------------------+--------------------------+---------------+
+    | **Library**                                                        |  ``Libcu++``             | **≤ C++17**   |
+    +====================================================================+==========================+===============+
+    | `<cassert> <https://en.cppreference.com/w/cpp/header/cassert>`_    | ``<cuda/std/cassert>``   |  |V|          |
+    +--------------------------------------------------------------------+--------------------------+---------------+
+    | `<ccomplex> <https://en.cppreference.com/w/cpp/header/ccomplex>`_  | ``<cuda/std/ccomplex>``  |  |V|          |
+    +--------------------------------------------------------------------+--------------------------+---------------+
+    | `<cfloat> <https://en.cppreference.com/w/cpp/header/cfloat>`_      | ``<cuda/std/cfloat>``    |  |V|          |
+    +--------------------------------------------------------------------+--------------------------+---------------+
+    | `<climits> <https://en.cppreference.com/w/cpp/header/climits>`_    | ``<cuda/std/climits>``   |  |V|          |
+    +--------------------------------------------------------------------+--------------------------+---------------+
+    | `<cmath> <https://en.cppreference.com/w/cpp/header/cmath>`_        |  ``<cuda/std/cmath>``    |  |V|          |
+    +--------------------------------------------------------------------+--------------------------+---------------+
+    | `<cstddef> <https://en.cppreference.com/w/cpp/header/cstddef>`_    | ``<cuda/std/cstddef>``   |  |V|          |
+    +--------------------------------------------------------------------+--------------------------+---------------+
+    | `<cstdint> <https://en.cppreference.com/w/cpp/header/cstdint>`_    | ``<cuda/std/cstdint>``   |  |V|          |
+    +--------------------------------------------------------------------+--------------------------+---------------+
+    | `<cstdlib> <https://en.cppreference.com/w/cpp/header/cstdlib>`_    | ``<cuda/std/cstdlib>``   |  |V|          |
+    +--------------------------------------------------------------------+--------------------------+---------------+
+    | `<cstring> <https://en.cppreference.com/w/cpp/header/cstring>`_    | ``<cuda/std/cstring>``   |  |V|          |
+    +--------------------------------------------------------------------+--------------------------+---------------+
+    | `<ctime> <https://en.cppreference.com/w/cpp/header/ctime>`_        | ``<cuda/std/ctime>``     |  |V|          |
+    +--------------------------------------------------------------------+--------------------------+---------------+
+
+see :ref:`C Libraries <libcudacxx-standard-api-c-compat>` for more details
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/algorithms_library.rst b/cccl_upstream/docs/libcudacxx/standard_api/algorithms_library.rst
new file mode 100644
index 00000000..d44491d6
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/algorithms_library.rst
@@ -0,0 +1,98 @@
+.. _libcudacxx-standard-api-algorithms:
+
+Algorithms Library
+===================
+
+.. list-table::
+   :widths: 25 45 30
+   :header-rows: 1
+
+   * - Header
+     - Content
+     - Availability
+   * - `\<cuda/std/algorithm\> <https://en.cppreference.com/w/cpp/header/algorithm>`_
+     - Fundamental library algorithms
+     - CCCL 3.2.0 / CUDA 13.2
+   * - `\<cuda/std/execution\> <https://en.cppreference.com/w/cpp/header/execution>`_
+     - Standard parallel algorithms
+     - CCCL 3.4.0 / CUDA 13.4
+
+Extensions
+----------
+
+  - All supported algorithms are available from C++17 onwards.
+  - All supported algorithms are constexpr, except the allocating ones.
+  - Because `<cuda/std/algorithm>` is a huge header with a considerable compile-time cost, we provide each algorithm
+    through a minimal subheader named e.g `<cuda/std/algorithm.find.h>`
+
+Restrictions
+------------
+
+  - Algorithms in namespace `ranges` are not yet supported.
+  - Some sorting algorithms are not yet supported:
+
+    * ``inplace_merge``
+    * ``nth_element``
+    * ``sort``
+    * ``stable_partition``
+    * ``stable_sort``
+
+Parallel standard algorithms
+----------------------------
+
+CCCL provides an implementation for the standard `parallel algorithms library <http://www.eel.is/c++draft/algorithms.parallel>`_
+
+Currently the CUDA backend is the only supported backend. It can be selected by passing the `cuda::execution::gpu`
+execution policy to one of the supported algorithms. The CUDA backend requires the passed in sequences to reside in
+device accessible memory and the iterators into those sequences to be at least random access iterators. The CUDA backend
+is enabled if the program is compiled with a CUDA compiler in CUDA mode.
+
+The use of any other execution policy is currently not supported and results in a compile time error.
+
+The following algorithms are supported:
+
+  * ``adjacent_find``
+  * ``all_of``
+  * ``any_of``
+  * ``copy``
+  * ``copy_if``
+  * ``copy_n``
+  * ``count``
+  * ``count_if``
+  * ``equal``
+  * ``fill``
+  * ``fill_n``
+  * ``find``
+  * ``find_if``
+  * ``find_if_not``
+  * ``for_each``
+  * ``for_each_n``
+  * ``generate``
+  * ``generate_n``
+  * ``is_partitioned``
+  * ``is_sorted``
+  * ``is_sorted_until``
+  * ``merge``
+  * ``mismatch``
+  * ``none_of``
+  * ``remove``
+  * ``remove_copy``
+  * ``remove_copy_if``
+  * ``remove_if``
+  * ``replace``
+  * ``replace_copy``
+  * ``replace_copy_if``
+  * ``replace_if``
+  * ``reverse``
+  * ``reverse_copy``
+  * ``rotate``
+  * ``rotate_copy``
+  * ``shift_left``
+  * ``shift_right``
+  * ``stable_partition``
+  * ``swap_ranges``
+  * ``transform``
+  * ``unique``
+  * ``unique_copy``
+
+The current implementation status is tracked in this `GitHub Issue <https://github.com/NVIDIA/cccl/issues/5592>`_
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/c_library.rst b/cccl_upstream/docs/libcudacxx/standard_api/c_library.rst
new file mode 100644
index 00000000..ca57f313
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/c_library.rst
@@ -0,0 +1,95 @@
+.. _libcudacxx-standard-api-c-compat:
+
+C Library
+=========
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   c_library/cstring
+
+
+Any Standard C++ header not listed below is omitted.
+
+.. list-table::
+   :widths: 25 45 30 30 20
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+     - **C++ Reference**
+
+   * - ``<cuda/std/cassert>``
+     - Lightweight assumption testing
+     - CCCL 2.0.0
+     - CUDA 10.2
+     - `\<cassert\> <https://en.cppreference.com/w/cpp/header/cassert>`_
+
+   * - ``<cuda/std/ccomplex>``
+     - C complex number arithmetic
+     - CCCL 2.0.0
+     - CUDA 11.4
+     - `\<ccomplex\> <https://en.cppreference.com/w/cpp/header/ccomplex>`_
+
+   * - ``<cuda/std/cfloat>``
+     - Type support library
+     - CCCL 2.2.0
+     - CUDA 12.3
+     - `\<cfloat\> <https://en.cppreference.com/w/cpp/header/cfloat>`_
+
+   * - ``<cuda/std/cfloat>``
+     - Limits of floating point types
+     - CCCL 2.0.0
+     - CUDA 10.2
+     - `\<cfloat\> <https://en.cppreference.com/w/cpp/header/cfloat>`_
+
+   * - ``<cuda/std/climits>``
+     - Limits of integral types
+     - CCCL 2.0.0
+     - CUDA 10.2
+     - `\<climits\> <https://en.cppreference.com/w/cpp/header/climits>`_
+
+   * - ``<cuda/std/cmath>``
+     - Common math functions
+     - CCCL 2.2.0
+     - CUDA 12.3
+     - `\<cmath\> <https://en.cppreference.com/w/cpp/header/cmath>`_
+
+   * - ``<cuda/std/cstddef>``
+     - Fundamental types
+     - CCCL 2.0.0
+     - CUDA 10.2
+     - `\<cstddef\> <https://en.cppreference.com/w/cpp/header/cstddef>`_
+
+   * - ``<cuda/std/cstdint>``
+     - Fundamental integer types
+     - CCCL 2.2.0
+     - CUDA 12.3
+     - `\<cstdint\> <https://en.cppreference.com/w/cpp/header/cstdint>`_
+
+   * - ``<cuda/std/cstdint>``
+     - Fixed-width integer types
+     - CCCL 2.0.0
+     - CUDA 10.2
+     - `\<cstdint\> <https://en.cppreference.com/w/cpp/header/cstdint>`_
+
+   * - ``<cuda/std/cstdlib>``
+     - Common utilities
+     - CCCL 2.2.0
+     - CUDA 12.3
+     - `\<cstdlib\> <https://en.cppreference.com/w/cpp/header/cstdlib>`_
+
+   * - :ref:`\<cuda/std/cstring\> <libcudacxx-standard-api-cstring>`
+     - Provides array manipulation functions such as ``memcpy``, ``memset`` and ``memcmp``
+     - CCCL 3.0.0
+     - CUDA 13.0
+     - `\<cstring\> <https://en.cppreference.com/w/cpp/header/cstring>`_
+
+   * - ``<cuda/std/ctime>``
+     - Provides ``clock``, ``difftime``, ``time`` and ``timespec_get`` functions
+     - CCCL 3.1.0
+     - CUDA 13.1
+     - `\<ctime\> <https://en.cppreference.com/w/cpp/header/ctime>`_
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/c_library/cstring.rst b/cccl_upstream/docs/libcudacxx/standard_api/c_library/cstring.rst
new file mode 100644
index 00000000..f3e46da1
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/c_library/cstring.rst
@@ -0,0 +1,47 @@
+.. _libcudacxx-standard-api-cstring:
+
+``<cuda/std/cstring>``
+======================
+
+``cuda::std::memset``
+---------------------
+
+.. code:: cuda
+
+   __host__ __device__
+   inline void* memset(void* dest, int ch, size_t count) noexcept;
+
+See `std::memset <https://en.cppreference.com/w/cpp/string/byte/memset.html>`_ for the full documentation.
+
+**Preconditions**
+
+The following preconditions are only enabled with CCCL 3.2 or later:
+
+    - ``dest`` is a valid pointer.
+    - ``dest + count`` is a valid pointer.
+
+A valid pointer is one that is not NULL and within the correct range if it belongs to the shared memory address space.
+
+----
+
+``cuda::std::memcpy``
+---------------------
+
+.. code:: cuda
+
+   __host__ __device__
+   inline void* memcpy(void* dest, const void* src, size_t count) noexcept;
+
+See `std::memcpy <https://en.cppreference.com/w/cpp/string/byte/memcpy.html>`_  for the full documentation.
+
+**Preconditions**
+
+The following preconditions are only enabled with CCCL 3.2 or later:
+
+    - ``src`` is a valid pointer.
+    - ``src + count`` is a valid pointer.
+    - ``dest`` is a valid pointer.
+    - ``dest + count`` is a valid pointer.
+    - ``src`` and ``dest`` don't overlap.
+
+A valid pointer is one that is not NULL and within the correct range if it belongs to the shared memory address space.
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/concepts_library.rst b/cccl_upstream/docs/libcudacxx/standard_api/concepts_library.rst
new file mode 100644
index 00000000..fd798d5f
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/concepts_library.rst
@@ -0,0 +1,49 @@
+.. _libcudacxx-standard-api-concepts:
+
+Concepts Library
+=======================
+
+.. list-table::
+   :widths: 25 45 30
+   :header-rows: 1
+
+   * - Header
+     - Content
+     - Availability
+   * - `\<cuda/std/concepts\> <https://en.cppreference.com/w/cpp/header/concepts>`_
+     - Fundamental library concepts
+     - CCCL 2.1.0 / CUDA 12.2
+
+Extensions
+----------
+
+-  All library features are available from C++14 onwards. The concepts
+   can be used like type traits prior to C++20.
+
+  .. code:: cpp
+
+    template<cuda::std::integral Integer>
+    void do_something_with_integers_in_cpp20(Integer&& i) {...}
+
+    template<class Integer, cuda::std::enable_if_t<cuda::std::integral<Integer>, int> = 0>
+    void do_something_with_integers_in_cpp17(Integer&& i) {...}
+
+    template<class Integer, cuda::std::enable_if_t<cuda::std::integral<Integer>, int> = 0>
+    void do_something_with_integers_in_cpp14(Integer&& i) {...}
+
+Restrictions
+------------
+
+-  Subsumption does not work prior to C++20
+
+  .. code:: cpp
+
+    template<class Integer, cuda::std::enable_if_t<subsuming_concept<Integer> && true, int> = 0>
+    void would_be_preferred_overload_in_cpp20(Integer&& i) {...}
+
+    template<class Integer, cuda::std::enable_if_t<cuda::std::integral<Integer>, int> = 0>
+    void is_always_ambiguous_in_cpp17(Integer&& i) {...}
+
+-  Subsumption is only partially implemented in the compiler until nvcc 12.4
+
+  nvcc has issues detecting subsumption of concepts that are composed of multiple concepts
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/container_library.rst b/cccl_upstream/docs/libcudacxx/standard_api/container_library.rst
new file mode 100644
index 00000000..05bfaad9
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/container_library.rst
@@ -0,0 +1,49 @@
+.. _libcudacxx-standard-api-container:
+
+Container Library
+=================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   container_library/array
+   container_library/inplace_vector
+   container_library/mdspan
+   container_library/span
+
+Any Standard C++ header not listed below is omitted.
+
+.. list-table::
+   :widths: 25 45 30 30 20
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+     - **C++ Reference**
+
+   * - :ref:`\<cuda/std/array\> <libcudacxx-standard-api-container-array>`
+     - Fixed size array
+     - CCCL 2.0.0
+     - CUDA 11.7
+     - `\<array\> <https://en.cppreference.com/w/cpp/header/array>`_
+
+   * - :ref:`\<cuda/std/inplace_vector\> <libcudacxx-standard-api-container-inplace-vector>`
+     - Flexible size container with fixed capacity
+     - CCCL 2.6.0
+     - CUDA 12.8
+     - `\<inplace_vector\> <https://en.cppreference.com/w/cpp/header/inplace_vector>`_
+
+   * - :ref:`\<cuda/std/mdspan\> <libcudacxx-standard-api-container-mdspan>`
+     - Non - owning view into a multidimensional contiguous sequence of objects
+     - CCCL 2.1.0
+     - CUDA 12.2
+     - `\<mdspan\> <https://en.cppreference.com/w/cpp/header/mdspan>`_
+
+   * - :ref:`\<cuda/std/span\> <libcudacxx-standard-api-container-span>`
+     - Non - owning view into a contiguous sequence of objects
+     - CCCL 2.1.0
+     - CUDA 12.2
+     - `\<span\> <https://en.cppreference.com/w/cpp/header/span>`_
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/container_library/array.rst b/cccl_upstream/docs/libcudacxx/standard_api/container_library/array.rst
new file mode 100644
index 00000000..327b2217
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/container_library/array.rst
@@ -0,0 +1,10 @@
+.. _libcudacxx-standard-api-container-array:
+
+``<cuda/std/array>``
+======================
+
+Extensions
+----------
+
+-  All features of ``<array>`` are made available in C++14 onwards
+-  All features of ``<array>`` are made constexpr in C++14 onwards
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/container_library/inplace_vector.rst b/cccl_upstream/docs/libcudacxx/standard_api/container_library/inplace_vector.rst
new file mode 100644
index 00000000..ad29594b
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/container_library/inplace_vector.rst
@@ -0,0 +1,14 @@
+.. _libcudacxx-standard-api-container-inplace-vector:
+
+``<cuda/std/inplace_vector>``
+==============================
+
+Extensions
+----------
+
+-  Most features of ``<inplace_vector>`` are made available in C++14 onwards
+
+Restrictions
+------------
+
+-  The range based interface is only available with ranges support in C++17
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/container_library/mdspan.rst b/cccl_upstream/docs/libcudacxx/standard_api/container_library/mdspan.rst
new file mode 100644
index 00000000..38976b73
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/container_library/mdspan.rst
@@ -0,0 +1,22 @@
+.. _libcudacxx-standard-api-container-mdspan:
+
+``<cuda/std/mdspan>``
+======================
+
+Provided functionalities
+------------------------
+
+-  All features of ``<mdspan>`` are made available in C++17 onwards
+-  C++26 ``std::dims`` is made available in C++17 onwards
+-  C++26 ``std::aligned_accessor`` is made available in C++17 onwards
+
+Extensions
+----------
+
+-  The C++23 multidimensional ``operator[]`` is replaced with ``operator()`` in previous C++ standards
+-  Detection of out-of-bounds accesses is available in debug mode
+
+Restrictions
+------------
+
+-  On device no exceptions are thrown in case of a bad access.
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/container_library/span.rst b/cccl_upstream/docs/libcudacxx/standard_api/container_library/span.rst
new file mode 100644
index 00000000..f6d5d667
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/container_library/span.rst
@@ -0,0 +1,16 @@
+.. _libcudacxx-standard-api-container-span:
+
+``<cuda/std/span>``
+======================
+
+Extensions
+----------
+
+-  All features of ``<span>`` are made available in C++14 onwards
+-  All features of ``<span>`` are made constexpr in C++14 onwards
+
+Restrictions
+------------
+
+-  On device no exceptions are thrown in case of a bad access.
+-  The range based constructors are emulated but not 100% equivalent.
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/execution_library.rst b/cccl_upstream/docs/libcudacxx/standard_api/execution_library.rst
new file mode 100644
index 00000000..1cd13ec0
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/execution_library.rst
@@ -0,0 +1,25 @@
+.. _libcudacxx-standard-api-execution:
+
+Execution Library
+=======================
+
+.. list-table::
+   :widths: 25 45 30
+   :header-rows: 1
+
+   * - Header
+     - Content
+     - Availability
+   * - `\<cuda/std/execution\> <https://en.cppreference.com/w/cpp/header/execution>`_
+     - Fundamental library concepts
+     - CCCL 3.0.0 / CUDA 13
+
+
+Omissions
+---------
+
+-  At present, only the following features are implemented:
+
+  -  `cuda::std::execution::prop <https://eel.is/c++draft/exec.prop>`_
+  -  `cuda::std::execution::env <https://eel.is/c++draft/exec.env>`_
+  -  `cuda::std::execution::get_env <https://eel.is/c++draft/exec.get.env>`_
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/numerics_library.rst b/cccl_upstream/docs/libcudacxx/standard_api/numerics_library.rst
new file mode 100644
index 00000000..f3ac6e27
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/numerics_library.rst
@@ -0,0 +1,69 @@
+.. _libcudacxx-standard-api-numerics:
+
+Numerics Library
+================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   numerics_library/bit
+   numerics_library/complex
+   numerics_library/linalg
+   numerics_library/numbers
+   numerics_library/numeric
+   numerics_library/random
+
+Any Standard C++ header not listed below is omitted.
+
+.. list-table::
+   :widths: 25 45 30 30 20
+   :header-rows: 1
+
+   * - **Header**
+     - **Content**
+     - **CCCL Availability**
+     - **CUDA Toolkit Availability**
+     - **C++ Reference**
+
+   * - ``<cuda/std/ratio>``
+     - Compile-time rational arithmetic
+     - CCCL 2.0.0
+     - CUDA 11.0
+     - `\<ratio\> <https://en.cppreference.com/w/cpp/header/ratio>`_
+
+   * - :ref:`\<cuda/std/bit\> <libcudacxx-standard-api-numerics-bit>`
+     - Access, manipulate, and process individual bits and bit sequences.
+     - CCCL 2.0.0
+     - CUDA 11.7
+     - `\<bit\> <https://en.cppreference.com/w/cpp/header/bit>`_
+
+   * - :ref:`\<cuda/std/complex\> <libcudacxx-standard-api-numerics-complex>`
+     - Complex number type
+     - CCCL 2.0.0
+     - CUDA 11.4
+     - `\<complex\> <https://en.cppreference.com/w/cpp/header/complex>`_
+
+   * - :ref:`\<cuda/std/linalg\> <libcudacxx-standard-api-numerics-linalg>`
+     - Linear algebra layouts and accessors
+     - CCCL 3.0.0
+     - CUDA 13.0
+     - `\<linalg\> <https://en.cppreference.com/w/cpp/header/linalg>`_
+
+   * - :ref:`\<cuda/std/numbers\> <libcudacxx-standard-api-numerics-numbers>`
+     - Numeric constants
+     - CCCL 3.0.0
+     - CUDA 13.0
+     - `\<numbers\> <https://en.cppreference.com/w/cpp/header/numbers>`_
+
+   * - :ref:`\<cuda/std/numeric\> <libcudacxx-standard-api-numerics-numeric>`
+     - Numeric algorithms
+     - CCCL 2.5.0
+     - CUDA 12.6
+     - `\<numeric\> <https://en.cppreference.com/w/cpp/header/numeric>`_
+
+   * - :ref:`\<cuda/std/random\> <libcudacxx-standard-api-numerics-random>`
+     - Random number generation
+     - CCCL 3.3.0
+     - CUDA 13.3
+     - `\<random\> <https://en.cppreference.com/w/cpp/header/random>`_
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/numerics_library/bit.rst b/cccl_upstream/docs/libcudacxx/standard_api/numerics_library/bit.rst
new file mode 100644
index 00000000..44079a4d
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/numerics_library/bit.rst
@@ -0,0 +1,47 @@
+.. _libcudacxx-standard-api-numerics-bit:
+
+``<cuda/std/bit>``
+==================
+
+``cuda::std::bit_cast``
+-----------------------
+
+``cuda::std::bit_cast`` extended the standard ``std::bit_cast`` to also recognize CUDA extended floating-point scalar and vector types as trivially copyable.
+
+**Limitations**
+
+- The function can be used in ``constexpr`` contexts only when the source and destination types are trivially copyable.
+- The function cannot be used in ``constexpr`` contexts with MSVC <= 19.25 and GCC <= 10.
+
+CUDA Performance Considerations
+-------------------------------
+
+Given an unsigned integer with ``N`` bits and ``N <= 32``, the ``<bit>`` functions translate into the following SASS instructions. For some functions, the results is decorated with a compile-time assumption to restrict its range and allowing further optimizations.
+
+- ``bit_width()`` translates into a single ``FLO`` SASS instruction. The result is assumed to be in the range ``[0, N]``.
+- ``bit_ceil()`` translates into ``ADD, FLO, SHL, IMINMAX`` SASS instructions. The result is assumed to be greater than or equal to the input.
+- ``bit_floor()`` translates into ``FLO, SHL`` SASS instructions. The result is assumed to be less than or equal to the input.
+- ``byteswap()`` translates into a single ``PRMT`` SASS instruction.
+- ``popcount()`` translates into a single ``POPC`` SASS instruction. The result is assumed to be in the range ``[0, N]``.
+- ``has_single_bit()`` translates into ``POPC + ISETP`` SASS instructions.
+- ``rotl()/rotr()`` translate into a single ``SHF`` (funned shift) SASS instruction.
+- ``countl_zero()`` translates into ``FLO, IMINMAX`` SASS instructions. The result is assumed to be in the range ``[0, N]``.
+- ``countl_one()`` translates into ``LOP3, FLO, IMINMAX`` SASS instructions. The result is assumed to be in the range ``[0, N]``.
+- ``countr_zero()`` translates into ``BREV, FLO, IMINMAX`` SASS instructions. The result is assumed to be in the range ``[0, N]``.
+- ``countr_one()`` translates into ``LOP3, BREV, FLO, IMINMAX`` SASS instructions. The result is assumed to be in the range ``[0, N]``.
+
+Additional Notes
+----------------
+
+- All functions are marked ``[[nodiscard]]`` and ``noexcept``
+- All functions support 128-bit integer types
+- ``bit_ceil()`` checks for overflow in debug mode
+- ``rotl()/rotr()`` checks for invalid count value (``INT_MIN``) in debug mode
+
+.. note::
+
+    When the input values are run-time values that the compiler can resolve at compile-time, e.g. an index of a loop with a fixed number of iterations, using the functions could not be optimal.
+
+.. note::
+
+    GCC <= 8 uses a slow path with more instructions even in CUDA
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/numerics_library/complex.rst b/cccl_upstream/docs/libcudacxx/standard_api/numerics_library/complex.rst
new file mode 100644
index 00000000..962b48f2
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/numerics_library/complex.rst
@@ -0,0 +1,41 @@
+.. _libcudacxx-standard-api-numerics-complex:
+
+``<cuda/std/complex>``
+======================
+
+Omissions
+---------
+
+  When using libcu++ with NVCC, ``complex`` does not support ``long double`` or ``complex`` literals (``_i``, ``_if``, and ``_il``).
+  NVCC warns on any usage of ``long double`` in device code, because ``long double`` will be demoted to ``double`` in device code.
+  This warning can be suppressed silenced with ``#pragma``\ s, but only globally, not just when using ``complex``.
+  User-defined floating-point literals must be specified in terms of ``long double``, so they lead to warnings
+  that are unable to be suppressed.
+
+Extensions
+--------------
+
+- Handling of infinities
+
+  Our implementation by default recovers infinite values during multiplication and division. This adds a significant runtime overhead,
+  so we allow disabling that canonicalization if it is not desired.
+
+  Definition of ``LIBCUDACXX_ENABLE_SIMPLIFIED_COMPLEX_OPERATIONS`` disables canonicalization for both multiplication *and* division.
+
+  Definition of ``LIBCUDACXX_ENABLE_SIMPLIFIED_COMPLEX_MULTIPLICATION`` or ``LIBCUDACXX_ENABLE_SIMPLIFIED_COMPLEX_DIVISION`` disables
+  canonicalization for multiplication or division individually.
+
+- Support for half and bfloat16 (since libcu++ 2.4.0)
+
+  Our implementation includes support for the ``__half`` type from ``<cuda_fp16.h>``, when the CUDA toolkit version is at
+  least 12.2, and when ``CCCL_DISABLE_FP16_SUPPORT`` is **not** defined.
+
+  This is detected automatically when compiling through NVCC. If you are compiling a host-only translation unit directly
+  with the host compiler, you must define the macro ``LIBCUDACXX_ENABLE_HOST_NVFP16`` prior to including any libcu++ headers,
+  and you must ensure that the ``<cuda_fp16.h>`` header that's found by the compiler comes from a CUDA toolkit version
+  12.2 or higher.
+
+  Our implementation includes support for the ``__nv_bfloat16`` type from ``<cuda_bf16.h>``, when the conditions for the
+  support of ``__half`` are fulfilled, and when ``CCCL_DISABLE_BF16_SUPPORT`` and ``CCCL_DISABLE_FP16_SUPPORT`` are **not** defined.
+
+- C++20 constexpr ``<complex>`` is available in C++14.
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/numerics_library/linalg.rst b/cccl_upstream/docs/libcudacxx/standard_api/numerics_library/linalg.rst
new file mode 100644
index 00000000..cc034eea
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/numerics_library/linalg.rst
@@ -0,0 +1,31 @@
+.. _libcudacxx-standard-api-numerics-linalg:
+
+``<cuda/std/linalg>``
+============================================
+
+Provided functionalities
+------------------------
+
+- ``scaled()`` `std::linalg::scaled <https://en.cppreference.com/w/cpp/numeric/linalg/scaled>`_
+- ``scaled_accessor`` `std::linalg::scaled_accessor <https://en.cppreference.com/w/cpp/numeric/linalg/scaled_accessor>`_
+- ``conjugated()`` `std::linalg::conjugated <https://en.cppreference.com/w/cpp/numeric/linalg/conjugated>`_
+- ``conjugated_accessor`` `std::linalg::conjugated_accessor <https://en.cppreference.com/w/cpp/numeric/linalg/conjugated_accessor>`_
+- ``transposed()`` `std::linalg::transposed <https://en.cppreference.com/w/cpp/numeric/linalg/transposed>`_
+- ``layout_transpose`` `std::linalg::layout_transpose <https://en.cppreference.com/w/cpp/numeric/linalg/layout_transpose>`_
+- ``conjugate_transposed()`` `std::linalg::conjugate_transposed <https://en.cppreference.com/w/cpp/numeric/linalg/conjugate_transposed>`_
+
+Extensions
+----------
+
+-  C++26 ``std::linalg`` accessors, transposed layout, and related functions are available in C++17
+
+Omissions
+---------
+
+-  Currently we do not expose any BLAS functions and layouts.
+
+Restrictions
+------------
+
+-  On device no exceptions are thrown in case of a bad access.
+-  MSVC is only supported with C++20
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/numerics_library/numbers.rst b/cccl_upstream/docs/libcudacxx/standard_api/numerics_library/numbers.rst
new file mode 100644
index 00000000..ae231d78
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/numerics_library/numbers.rst
@@ -0,0 +1,10 @@
+.. _libcudacxx-standard-api-numerics-numbers:
+
+``<cuda/std/numbers>``
+======================
+
+Extensions
+----------
+
+-  All features of ``<numbers>`` are made available in C++14 onwards
+-  Implementation is provided only on Linux systems due to NVCC's lack of template ``const __device__`` variables support
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/numerics_library/numeric.rst b/cccl_upstream/docs/libcudacxx/standard_api/numerics_library/numeric.rst
new file mode 100644
index 00000000..aa60075e
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/numerics_library/numeric.rst
@@ -0,0 +1,41 @@
+.. _libcudacxx-standard-api-numerics-numeric:
+
+``<cuda/std/numeric>``
+======================
+
+Omissions
+---------
+
+-  Currently we do not expose any parallel algorithms.
+
+Extensions
+----------
+
+-  All features of ``<numeric>`` are made available in C++11 onwards
+-  All features of ``<numeric>`` are made constexpr in C++14 onwards
+-  Algorithms that return a value and not an iterator have been marked ``[[nodiscard]]``
+
+
+Parallel standard algorithms
+----------------------------
+
+CCCL provides an implementation for the standard `parallel algorithms library <http://www.eel.is/c++draft/algorithms.parallel>`_
+
+Currently the CUDA backend is the only supported backend. It can be selected by passing the `cuda::execution::gpu`
+execution policy to one of the supported algorithms. The CUDA backend requires the passed in sequences to reside in
+device accessible memory and the iterators into those sequences to be at least random access iterators. The CUDA backend
+is enabled if the program is compiled with a CUDA compiler in CUDA mode.
+
+The use of any other execution policy is currently not supported and results in a compile time error.
+
+The following algorithms are supported:
+
+  * ``adjacent_difference``
+  * ``exclusive_scan``
+  * ``inclusive_scan``
+  * ``transform_exclusive_scan``
+  * ``transform_inclusive_scan``
+  * ``reduce``
+  * ``transform_reduce``
+
+The current implementation status is tracked in this `GitHub Issue <https://github.com/NVIDIA/cccl/issues/5592>`_
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/numerics_library/random.rst b/cccl_upstream/docs/libcudacxx/standard_api/numerics_library/random.rst
new file mode 100644
index 00000000..5a54b669
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/numerics_library/random.rst
@@ -0,0 +1,72 @@
+.. _libcudacxx-standard-api-numerics-random:
+
+``<cuda/std/random>``
+=====================
+
+Provided functionalities
+------------------------
+
+Random number engines:
+
+- `cuda::std::minstd_rand0 <https://en.cppreference.com/w/cpp/numeric/random/minstd_rand0>`_
+- `cuda::std::minstd_rand <https://en.cppreference.com/w/cpp/numeric/random/minstd_rand>`_
+- C++26 `cuda::std::philox4x32 <https://en.cppreference.com/w/cpp/numeric/random/philox_engine.html>`_ - available from C++17 onwards
+- C++26 `cuda::std::philox4x64 <https://en.cppreference.com/w/cpp/numeric/random/philox_engine.html>`_ - available from C++17 onwards
+
+.. note::
+
+    ``cuda::pcg64`` is provided in the non-standard ``<cuda/random>`` header. See
+    :ref:`cuda::pcg64 <libcudacxx-extended-api-random-pcg64>`.
+
+Random number distributions:
+
+- `cuda::std::bernoulli_distribution <https://en.cppreference.com/w/cpp/numeric/random/bernoulli_distribution>`_
+- `cuda::std::binomial_distribution <https://en.cppreference.com/w/cpp/numeric/random/binomial_distribution>`_
+- `cuda::std::cauchy_distribution <https://en.cppreference.com/w/cpp/numeric/random/cauchy_distribution>`_
+- `cuda::std::chi_squared_distribution <https://en.cppreference.com/w/cpp/numeric/random/chi_squared_distribution>`_
+- `cuda::std::exponential_distribution <https://en.cppreference.com/w/cpp/numeric/random/exponential_distribution>`_
+- `cuda::std::extreme_value_distribution <https://en.cppreference.com/w/cpp/numeric/random/extreme_value_distribution>`_
+- `cuda::std::fisher_f_distribution <https://en.cppreference.com/w/cpp/numeric/random/fisher_f_distribution>`_
+- `cuda::std::gamma_distribution <https://en.cppreference.com/w/cpp/numeric/random/gamma_distribution>`_
+- `cuda::std::geometric_distribution <https://en.cppreference.com/w/cpp/numeric/random/geometric_distribution>`_
+- `cuda::std::lognormal_distribution <https://en.cppreference.com/w/cpp/numeric/random/lognormal_distribution>`_
+- `cuda::std::negative_binomial_distribution <https://en.cppreference.com/w/cpp/numeric/random/negative_binomial_distribution>`_
+- `cuda::std::normal_distribution <https://en.cppreference.com/w/cpp/numeric/random/normal_distribution>`_
+- `cuda::std::poisson_distribution <https://en.cppreference.com/w/cpp/numeric/random/poisson_distribution>`_
+- `cuda::std::student_t_distribution <https://en.cppreference.com/w/cpp/numeric/random/student_t_distribution>`_
+- `cuda::std::uniform_int_distribution <https://en.cppreference.com/w/cpp/numeric/random/uniform_int_distribution>`_
+- `cuda::std::uniform_real_distribution <https://en.cppreference.com/w/cpp/numeric/random/uniform_real_distribution>`_
+- `cuda::std::weibull_distribution <https://en.cppreference.com/w/cpp/numeric/random/weibull_distribution>`_
+
+Utilities:
+
+- `cuda::std::seed_seq <https://en.cppreference.com/w/cpp/numeric/random/seed_seq>`_
+- `cuda::std::generate_canonical <https://en.cppreference.com/w/cpp/numeric/random/generate_canonical>`_
+
+
+.. note::
+
+    ``cuda::std::seed_seq`` should be used exclusively on host or exclusively on device. Do not share the same
+    ``seed_seq`` instance between host and device code.
+
+
+The following engines or distributions are not implemented as they are not convenient or practical to implement in CUDA device code, either due to dynamic memory allocations or large state sizes.
+
+Not supported
+-------------
+- `std::random_device <https://en.cppreference.com/w/cpp/numeric/random/random_device>`_
+- `std::mersenne_twister_engine <https://en.cppreference.com/w/cpp/numeric/random/mersenne_twister_engine>`_
+  (`std::mt19937 <https://en.cppreference.com/w/cpp/numeric/random/mersenne_twister_engine>`_,
+  `std::mt19937_64 <https://en.cppreference.com/w/cpp/numeric/random/mersenne_twister_engine>`_)
+- `std::subtract_with_carry_engine <https://en.cppreference.com/w/cpp/numeric/random/subtract_with_carry_engine.html>`_
+  (`std::ranlux24_base <https://en.cppreference.com/w/cpp/numeric/random/subtract_with_carry_engine.html>`_,
+  `std::ranlux48_base <https://en.cppreference.com/w/cpp/numeric/random/subtract_with_carry_engine.html>`_,
+  `std::ranlux24 <https://en.cppreference.com/w/cpp/numeric/random/subtract_with_carry_engine.html>`_,
+  `std::ranlux48 <https://en.cppreference.com/w/cpp/numeric/random/subtract_with_carry_engine.html>`_)
+- `std::discard_block_engine <https://en.cppreference.com/w/cpp/numeric/random/discard_block_engine.html>`_
+- `std::independent_bits_engine <https://en.cppreference.com/w/cpp/numeric/random/independent_bits_engine.html>`_
+- `std::shuffle_order_engine <https://en.cppreference.com/w/cpp/numeric/random/shuffle_order_engine>`_
+  (`std::knuth_b <https://en.cppreference.com/w/cpp/numeric/random/shuffle_order_engine>`_)
+- `std::discrete_distribution <https://en.cppreference.com/w/cpp/numeric/random/discrete_distribution.html>`_
+- `std::piecewise_constant_distribution <https://en.cppreference.com/w/cpp/numeric/random/piecewise_constant_distribution.html>`_
+- `std::piecewise_linear_distribution <https://en.cppreference.com/w/cpp/numeric/random/piecewise_linear_distribution.html>`_
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/ranges_library.rst b/cccl_upstream/docs/libcudacxx/standard_api/ranges_library.rst
new file mode 100644
index 00000000..cd0049b7
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/ranges_library.rst
@@ -0,0 +1,46 @@
+.. _libcudacxx-standard-api-ranges:
+
+Ranges Library
+=======================
+
+See the documentation of the standard headers `\<iterator\> <https://en.cppreference.com/w/cpp/header/iterator>`_ and
+`\<ranges\> <https://en.cppreference.com/w/cpp/header/ranges>`_
+
+.. list-table::
+   :widths: 25 45 30
+   :header-rows: 1
+
+   * - Header
+     - Content
+     - Availability
+   * - `\<cuda/std/iterator\> <https://en.cppreference.com/w/cpp/header/iterator>`_
+     - Iterator related concepts and machinery such as ``cuda::std::forward_iterator``
+     - CCCL 2.3.0 / CUDA 12.4
+   * - `\<cuda/std/ranges\> <https://en.cppreference.com/w/cpp/header/ranges>`_
+     - Range related concepts and machinery such as ``cuda::std::ranges::forward_range`` and ``cuda::std::ranges::subrange``
+     - CCCL 2.4.0 / CUDA 12.5
+
+Extensions
+----------
+
+-  All library features are available from C++17 onwards. The concepts can be used like type traits prior to C++20.
+
+  .. code:: cpp
+
+    template<cuda::std::contiguous_range Range>
+    void do_something_with_ranges_in_cpp20(Range&& range) {...}
+
+    template<class Range, cuda::std::enable_if_t<cuda::std::contiguous_range<Range>, int> = 0>
+    void do_something_with_ranges_in_cpp17(Range&& range) {...}
+
+Restrictions
+------------
+
+-  Subsumption does not work prior to C++20
+-  Subsumption is only partially implemented in the compiler until nvcc 12.4
+
+Omissions
+---------
+
+-  Range based algorithms have *not* been implemented
+-  Views have *not* been implemented
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/synchronization_library.rst b/cccl_upstream/docs/libcudacxx/standard_api/synchronization_library.rst
new file mode 100644
index 00000000..143e3854
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/synchronization_library.rst
@@ -0,0 +1,26 @@
+.. _libcudacxx-standard-api-synchronization:
+
+Synchronization Library
+=======================
+
+Any Standard C++ header not listed below is omitted.
+
+.. list-table::
+   :widths: 25 45 30
+   :header-rows: 1
+
+   * - Header
+     - Content
+     - Availability
+   * - `\<cuda/std/atomic\> <https://en.cppreference.com/w/cpp/header/atomic>`_
+     - Atomic objects and operations. See also :ref:`Extended API <libcudacxx-extended-api-synchronization-atomic>`
+     - libcu++ 1.0.0 / CCCL 2.0.0 / CUDA 10.2
+   * - `\<cuda/std/latch\> <https://en.cppreference.com/w/cpp/header/latch>`_
+     - Single-phase asynchronous thread-coordination mechanism. See also :ref:`Extended API <libcudacxx-extended-api-synchronization-latch>`
+     - libcu++ 1.1.0 / CCCL 2.0.0 / CUDA 11.0
+   * - `\<cuda/std/barrier\> <https://en.cppreference.com/w/cpp/header/barrier>`_
+     - Multi-phase asynchronous thread-coordination mechanism. See also :ref:`Extended API <libcudacxx-extended-api-synchronization-barrier>`
+     - libcu++ 1.1.0 / CCCL 2.0.0 / CUDA 11.0
+   * - `\<cuda/std/semaphore\> <https://en.cppreference.com/w/cpp/header/semaphore>`_
+     - Primitives for constraining concurrent access. See also :ref:`Extended API <libcudacxx-extended-api-synchronization-counting-semaphore>`
+     - libcu++ 1.1.0 / CCCL 2.0.0 / CUDA 11.0
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/time_library.rst b/cccl_upstream/docs/libcudacxx/standard_api/time_library.rst
new file mode 100644
index 00000000..e2868573
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/time_library.rst
@@ -0,0 +1,119 @@
+.. _libcudacxx-standard-api-time:
+
+Time Library
+=======================
+
+See the documentation of the standard header `\<chrono\> <https://en.cppreference.com/w/cpp/header/chrono>`_
+
+.. list-table::
+   :widths: 25 45 30
+   :header-rows: 1
+
+   * - Header
+     - Content
+     - Availability
+   * - ``<cuda/std/chrono>``
+     - Times, dates, and clocks
+     - libcu++ 1.1.0 / CCCL 2.0.0 / CUDA 12.3
+
+Implementation-Defined Behavior
+-------------------------------
+
+`std::chrono::system_clock <https://en.cppreference.com/w/cpp/chrono/system_clock>`_
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`std::chrono::system_clock <https://en.cppreference.com/w/cpp/chrono/system_clock>`_
+is a clock that track real-world time. In the C++ Standard, it is
+unspecified whether or not this clock is monotonically increasing. In
+our implementation, it is not.
+
+To implement
+`std::chrono::system_clock <https://en.cppreference.com/w/cpp/chrono/system_clock>`_,
+we use:
+
+-  `GetSystemTimePreciseAsFileTime <https://docs.microsoft.com/en-us/windows/win32/api/sysinfoapi/nf-sysinfoapi-getsystemtimepreciseasfiletime>`_ and
+   `GetSystemTimeAsFileTime <https://docs.microsoft.com/en-us/windows/win32/api/sysinfoapi/nf-sysinfoapi-getsystemtimeasfiletime>`_
+   for host code on Windows.
+-  `clock_gettime(CLOCK_REALTIME, ...) <https://man7.org/linux/man-pages/man3/clock_gettime.3.html>`_ and `gettimeofday <https://man7.org/linux/man-pages/man2/gettimeofday.2.html>`_
+   for host code on Linux, Android, and QNX.
+-  `PTX's %globaltimer <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-globaltimer>`_ for device code.
+
+`PTX's %globaltimer <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-globaltimer>`_
+is a system clock which also happens to be monotonically increasing on today's NVIDIA GPUs
+(e.g. it cannot be updated and is not changed when the host system clock changes).
+However, this is not necessarily the case with respect to host threads, where updates of the system clock may occur during the execution of the program.
+
+`PTX's %globaltimer <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-globaltimer>`_
+is initialized from the host system clock upon device attach; that may be at program start, but it could be earlier (for example, due to CUDA persistence mode).
+Since `PTX's %globaltimer <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-globaltimer>`_ is a system clock,
+it counts real-world time, and thus it has the same tick rate as the host system clock.
+
+There is potential for logical inconsistencies between the time that host threads and device threads observe from our
+`std::chrono::system_clock <https://en.cppreference.com/w/cpp/chrono/system_clock>`_.
+However, this is perfectly fine; it is an inherent property of system clocks. In fact, it is not even guaranteed that a system clock remain
+consistent between different host threads, or even within the same host thread. This can occur, for example, due to Daylights Savings Time or a
+time zone change.
+
+The requirements for `Clock <https://eel.is/c++draft/time.clock.req>`_ state:
+
+   ``C1`` denotes a clock type. ``t1`` and ``t2`` are values returned by
+   ``C1::now()`` where the call returning ``t1`` `happens before <http://eel.is/c++draft/intro.multithread#def:happens_before>`_
+   the call returning ``t2`` and both of these calls occur before
+   ``C1::time_point::max()``.
+
+   ``C1::is_steady`` is ``true`` if ``t1 <= t2`` is always true and the
+   time between clock ticks is constant, otherwise ``false``.
+
+The property is true for our `std::chrono::system_clock <https://en.cppreference.com/w/cpp/chrono/system_clock>`_
+within device code, but it is not true for all threads. Therefore, in the NVIDIA C++ Standard Library today,
+the value of the ``is_steady`` member of
+`std::chrono::system_clock <https://en.cppreference.com/w/cpp/chrono/system_clock>`_
+is ``false``.
+
+`std::chrono::high_resolution_clock <https://en.cppreference.com/w/cpp/chrono/high_resolution_clock>`_
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The `std::chrono::high_resolution_clock specification <http://eel.is/c++draft/time.clock.hires>`_ states:
+
+   Objects of ``class high_resolution_clock`` represent clocks with the shortest tick period.
+   ``high_resolution_clock`` may be a synonym for ``system_clock`` or ``steady_clock``.
+
+In the NVIDIA C++ Standard Library, `std::chrono::high_resolution_clock <https://en.cppreference.com/w/cpp/chrono/high_resolution_clock>`_
+is an alias for `std::chrono::system_clock <https://en.cppreference.com/w/cpp/chrono/system_clock>`_.
+This means that it counts real-world time and that ``is_steady`` is false for our
+`std::chrono::high_resolution_clock <https://en.cppreference.com/w/cpp/chrono/high_resolution_clock>`_.
+
+While our `std::chrono::high_resolution_clock <https://en.cppreference.com/w/cpp/chrono/high_resolution_clock>`_
+is not heterogeneously steady, it is steady within device code, so it is suitable for performance measurement within device code.
+
+Omissions
+---------
+
+The following facilities in section `time.syn <https://eel.is/c++draft/time.syn>`_ of ISO/IEC IS 14882 (the C++ Standard)
+are not available in the NVIDIA C++ Standard Library today:
+
+-  `std::chrono::steady_clock <https://en.cppreference.com/w/cpp/chrono/steady_clock>`_
+   - a monotonically increasing clock.
+-  `std::chrono::duration I/O operators <https://eel.is/c++draft/time.duration.io>`_.
+
+`std::chrono::steady_clock <https://en.cppreference.com/w/cpp/chrono/steady_clock>`_
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+`std::chrono::steady_clock <https://en.cppreference.com/w/cpp/chrono/steady_clock>`_
+is, by definition, a monotonically increasing clock (e.g. ``is_steady`` is ``true``). We do not currently have a heterogeneous steady clock.
+While we have a monotonically increasing clock in host code, and our system clock
+(`PTX's %globaltimer <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-globaltimer>`_)
+is monotonically increasing in device code, it is not guaranteed that the host clocks and the device clocks are monotonically increasing with
+respect to each other, due to how `PTX's %globaltimer <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#special-registers-globaltimer>`_
+is initialized. Additionally, ``%globaltime`` and the host steady clock may tick at different rates.
+
+It may be technically possible to synchronize the clocks and to compute and adjust for the difference in tick rates. However, it would be
+challenging to do so, and may introduce substantial overhead in the initialization and access of the heterogeneous clock.
+
+As such, today we do not provide `std::chrono::steady_clock <https://en.cppreference.com/w/cpp/chrono/steady_clock>`_,
+as we cannot easily provide an efficient implementation that is truly heterogeneous and conforms to the specification.
+
+`std::chrono::duration I/O Operators <https://eel.is/c++draft/time.duration.io>`_
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Implementing a heterogeneous C++ I/O streams library involves many challenges that we cannot overcome today.
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/type_support.rst b/cccl_upstream/docs/libcudacxx/standard_api/type_support.rst
new file mode 100644
index 00000000..52b64de6
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/type_support.rst
@@ -0,0 +1,24 @@
+.. _libcudacxx-standard-api-type-support:
+
+Type Support Library
+=======================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+Any Standard C++ header not listed below is omitted.
+
+.. list-table::
+   :widths: 25 45 30
+   :header-rows: 1
+
+   * - Header
+     - Content
+     - Availability
+   * - `\<cuda/std/climits\> <https://en.cppreference.com/w/cpp/header/climits>`_
+     - Limits of integral types
+     - libcu++ 1.0.0 / CCCL 2.0.0 / CUDA 10.2
+   * - `\<cuda/std/limits\> <https://en.cppreference.com/w/cpp/header/limits>`_
+     - Interface to query properties of all fundamental numeric types
+     - libcu++ 1.0.0 / CCCL 2.0.0 / CUDA 10.2
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/utility_library.rst b/cccl_upstream/docs/libcudacxx/standard_api/utility_library.rst
new file mode 100644
index 00000000..ba281514
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/utility_library.rst
@@ -0,0 +1,56 @@
+.. _libcudacxx-standard-api-utility:
+
+Utility Library
+=======================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   utility_library/bitset
+   utility_library/expected
+   utility_library/functional
+   utility_library/memory
+   utility_library/optional
+   utility_library/tuple
+   utility_library/type_traits
+   utility_library/utility
+   utility_library/variant
+
+Any Standard C++ header not listed below is omitted. Some of the Standard C++ facilities in this header are omitted, see
+the information about the individual features for details.
+
+.. list-table::
+   :widths: 25 45 30
+   :header-rows: 1
+
+   * - Header
+     - Content
+     - Availability
+   * - :ref:`libcudacxx-standard-api-utility-bitset`
+     - Fixed-size sequence of bits
+     - CCCL 2.8.0
+   * - :ref:`libcudacxx-standard-api-utility-expected`
+     - Optional value with error channel
+     - CCCL 2.3.0 / CUDA 12.4
+   * - :ref:`libcudacxx-standard-api-utility-functional`
+     - General-purpose polymorphic function wrapper
+     - CCCL 2.9.0 / CUDA 12.9
+   * - :ref:`libcudacxx-standard-api-utility-memory`
+     - Function objects and function wrappers
+     - libcu++ 1.1.0 / CCCL 2.0.0 / CUDA 11.2
+   * - :ref:`libcudacxx-standard-api-utility-optional`
+     - Optional value
+     - CCCL 2.3.0 / CUDA 12.4
+   * - :ref:`libcudacxx-standard-api-utility-tuple`
+     - Fixed-sized heterogeneous container
+     - libcu++ 1.3.0 / CCCL 2.0.0 / CUDA 11.2
+   * - :ref:`libcudacxx-standard-api-utility-type-traits`
+     - Compile-time type introspection
+     - libcu++ 1.0.0 / CCCL 2.0.0 / CUDA 10.2
+   * - :ref:`libcudacxx-standard-api-utility-utility`
+     - Various utility components
+     - libcu++ 1.3.0 / CCCL 2.0.0 / CUDA 11.2
+   * - :ref:`libcudacxx-standard-api-utility-variant`
+     - Type safe union type
+     - CCCL 2.4.0 / CUDA 12.5
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/utility_library/bitset.rst b/cccl_upstream/docs/libcudacxx/standard_api/utility_library/bitset.rst
new file mode 100644
index 00000000..a621cb01
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/utility_library/bitset.rst
@@ -0,0 +1,14 @@
+.. _libcudacxx-standard-api-utility-bitset:
+
+``<cuda/std/bitset>``
+======================
+
+Extensions
+----------
+
+-  All features of ``<bitset>`` are made constexpr in C++14 onwards
+
+Restrictions
+------------
+
+-  On device no exceptions are thrown in case of a bad access.
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/utility_library/expected.rst b/cccl_upstream/docs/libcudacxx/standard_api/utility_library/expected.rst
new file mode 100644
index 00000000..73a9f68e
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/utility_library/expected.rst
@@ -0,0 +1,16 @@
+.. _libcudacxx-standard-api-utility-expected:
+
+<cuda/std/expected>
+=======================
+
+See the documentation of the standard header `\<expected\> <https://en.cppreference.com/w/cpp/header/expected>`_
+
+Extensions
+----------
+
+-  All features are available from C++14 onwards.
+
+Restrictions
+------------
+
+-  On device no exceptions are thrown in case of a bad access.
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/utility_library/functional.rst b/cccl_upstream/docs/libcudacxx/standard_api/utility_library/functional.rst
new file mode 100644
index 00000000..598de1da
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/utility_library/functional.rst
@@ -0,0 +1,52 @@
+.. _libcudacxx-standard-api-utility-functional:
+
+<cuda/std/functional>
+=========================
+
+See the documentation of the standard header `\<functional\> <https://en.cppreference.com/w/cpp/header/functional>`_
+
+Omissions
+---------
+
+The following facilities in section
+`functional.syn <https://eel.is/c++draft/functional.syn>`_ of ISO/IEC
+IS 14882 (the C++ Standard) are not available in the NVIDIA C++ Standard
+Library today:
+
+-  `std::function <https://en.cppreference.com/w/cpp/utility/functional/function>`_
+   - Polymorphic function object wrapper.
+-  `std::bind <https://en.cppreference.com/w/cpp/utility/functional/bind>`_
+   - Generic function object binder / lambda facility.
+-  `std::hash <https://en.cppreference.com/w/cpp/utility/hash>`_
+   - Hash function object.
+
+std::function
+~~~~~~~~~~~~~~~~~
+
+`std::function <https://en.cppreference.com/w/cpp/utility/functional/function>`_
+is a polymorphic function object wrapper. Implementing it requires both
+polymorphism (either hand built dispatch tables or the use of C++
+virtual functions) and memory allocation. This means that it is
+non-trivial to implement a heterogeneous version of this facility today.
+As such, we have deferred it.
+
+std::bind
+~~~~~~~~~~~~~
+
+`std::bind <https://en.cppreference.com/w/cpp/utility/functional/bind>`_
+is a general-purpose function object binder / lambda facility. It relies
+on constexpr global variables for placeholders, which presents
+heterogeneous implementation challenges today due to how global
+variables work in NVCC. E.g. We cannot easily ensure the placeholders
+are the same object with the same address in host and device code.
+Therefore, we've decided to hold off on providing this feature for now.
+
+std::hash
+~~~~~~~~~~~~~
+
+`std::hash <https://en.cppreference.com/w/cpp/utility/hash>`_ is a
+function object which hashes entities. While this is an important
+feature, it is also important that we pick a hash implementation that
+makes sense for GPUs. That implementation might be different from the
+default that the upstream libc++ uses. Further research and
+investigation is required before we can provide this feature.
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/utility_library/memory.rst b/cccl_upstream/docs/libcudacxx/standard_api/utility_library/memory.rst
new file mode 100644
index 00000000..67625381
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/utility_library/memory.rst
@@ -0,0 +1,25 @@
+.. _libcudacxx-standard-api-utility-memory:
+
+<cuda/std/memory>
+===================
+
+Provided functionalities
+------------------------
+
+- ``cuda::std::addressof``. See the C++ documentation of `std::addressof <https://en.cppreference.com/w/cpp/memory/addressof>`_
+- ``cuda::std::align``. See the C++ documentation of `std::align <https://en.cppreference.com/w/cpp/memory/align>`_
+- ``cuda::std::assume_aligned``. See the C++ documentation of `std::assume_aligned <https://en.cppreference.com/w/cpp/memory/assume_aligned>`_
+- Uninitialized memory algorithms. See the C++ documentation `<https://en.cppreference.com/w/cpp/memory>`_
+
+Extensions
+----------
+
+-  Most features are available from C++11 onwards.
+-  ``cuda::std::addressof`` is constexpr from C++11 on if compiler support is available
+-  ``cuda::std::assume_aligned`` is constexpr from C++14 on
+
+Restrictions
+------------
+
+- `construct_at` and is only available in C++20 as that is explicitly mentioned in the standard
+- The specialized memory algorithms are not parallel
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/utility_library/optional.rst b/cccl_upstream/docs/libcudacxx/standard_api/utility_library/optional.rst
new file mode 100644
index 00000000..00f36018
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/utility_library/optional.rst
@@ -0,0 +1,17 @@
+.. _libcudacxx-standard-api-utility-optional:
+
+<cuda/std/optional>
+=======================
+
+See the documentation of the standard header `\<optional\> <https://en.cppreference.com/w/cpp/header/optional>`_
+
+Extensions
+----------
+
+- All features are available from C++14 onwards.
+- All features are available at compile time if the value type supports it.
+
+Restrictions
+------------
+
+-  On device no exceptions are thrown in case of a bad access.
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/utility_library/tuple.rst b/cccl_upstream/docs/libcudacxx/standard_api/utility_library/tuple.rst
new file mode 100644
index 00000000..d70322f9
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/utility_library/tuple.rst
@@ -0,0 +1,17 @@
+.. _libcudacxx-standard-api-utility-tuple:
+
+<cuda/std/tuple>
+====================
+
+See the documentation of the standard header `\<tuple\> <https://en.cppreference.com/w/cpp/header/tuple>`_
+
+Restrictions
+------------
+
+Before version 1.4.0, ``tuple`` is not available when using NVCC with
+MSVC as a host compiler, due to compiler bugs.
+
+Before version 2.3.0 internal compiler errors may be encountered when
+using ``tuple`` with older updates of MSVC 2017 and MSVC 2019. For MSVC
+2017, please use version 15.8 or later (``_MSC_VER >= 1915``). For MSVC
+2019, please use version 16.6 or later (``_MSC_VER >= 1926``).
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/utility_library/type_traits.rst b/cccl_upstream/docs/libcudacxx/standard_api/utility_library/type_traits.rst
new file mode 100644
index 00000000..120fa0c5
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/utility_library/type_traits.rst
@@ -0,0 +1,11 @@
+.. _libcudacxx-standard-api-utility-type-traits:
+
+<cuda/std/type_traits>
+=======================
+
+See the documentation of the standard header `\<type_traits\> <https://en.cppreference.com/w/cpp/header/type_traits>`_
+
+Extensions
+----------
+
+-  Most type traits are made available in C++11.
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/utility_library/utility.rst b/cccl_upstream/docs/libcudacxx/standard_api/utility_library/utility.rst
new file mode 100644
index 00000000..d1764a2e
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/utility_library/utility.rst
@@ -0,0 +1,21 @@
+.. _libcudacxx-standard-api-utility-utility:
+
+<cuda/std/utility>
+======================
+
+See the documentation of the standard header `\<utility\> <https://en.cppreference.com/w/cpp/header/utility>`_
+
+Extensions
+----------
+
+- ``pair`` has been made ``trivially_copyable`` in 2.3.0
+
+Omissions
+---------
+
+Prior to version 2.3.0 only ``pair`` is available.
+
+Since 2.3.0 we have implemented almost all functionality of
+``<utility>``. Notably support for operator spaceship is missing due to
+the specification relying on ``std`` types that are not accessible on
+device.
diff --git a/cccl_upstream/docs/libcudacxx/standard_api/utility_library/variant.rst b/cccl_upstream/docs/libcudacxx/standard_api/utility_library/variant.rst
new file mode 100644
index 00000000..db6e8897
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/standard_api/utility_library/variant.rst
@@ -0,0 +1,24 @@
+.. _libcudacxx-standard-api-utility-variant:
+
+<cuda/std/variant>
+======================
+
+See the documentation of the standard header `\<variant\> <https://en.cppreference.com/w/cpp/header/variant>`_
+
+Extensions
+----------
+
+-  All features are available from C++14 onwards.
+-  All features are available at compile time if the different value types support it.
+
+Restrictions
+------------
+
+-  On device no exceptions are thrown in case of a bad access.
+
+Cuda specific changes
+---------------------
+
+-  ``cuda::std::visit`` utilizes recursion instead of the usual function
+   pointer array. This greatly improves runtime behavior, but comes at
+   the cost of increased compile times.
diff --git a/cccl_upstream/docs/libcudacxx/tile.rst b/cccl_upstream/docs/libcudacxx/tile.rst
new file mode 100644
index 00000000..335e7886
--- /dev/null
+++ b/cccl_upstream/docs/libcudacxx/tile.rst
@@ -0,0 +1,215 @@
+.. _libcudacxx-tile-api:
+
+CUDA Tile support
+==================
+
+`CUDA Tile <https://developer.nvidia.com/cuda/tile#section-more-resources>`_ introduces a new way to program GPUs at a higher level than SIMT.
+
+Restrictions
+------------
+
+With the compiler taking more control over memory and threading there are a number of restrictions in a tile program:
+
+
+C++ Concurrency support
+~~~~~~~~~~~~~~~~~~~~~~
+
+Currently the use of inline ptx / assembly is not allowed in a tile program.
+All of our threading features rely on inline assembly in some capacity.
+Consequently, the following headers are not supported in tile mode:
+
+    - <cuda/atomic>
+    - <cuda/barrier>
+    - <cuda/latch>
+    - <cuda/pipeline>
+    - <cuda/semaphore>
+    - <cuda/std/atomic>
+    - <cuda/std/barrier>
+    - <cuda/std/execution>
+    - <cuda/std/latch>
+    - <cuda/std/semaphore>
+
+This also affects
+
+    - <cuda/cmath>
+    - <cuda/discard_memory>
+    - <cuda/ptx>
+
+
+C++ mathematical operations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+We rely heavily on compiler builtins or cuda runtime functions to implement C++ standard math functions such
+as ``cuda::std::exp``. Those compiler builtins are not currently supported in tile mode, so the following header
+is mostly unsupported in a tile kernel:
+
+    - <cuda/std/cmath>
+
+``cuda::std::complex`` is supported except for the various math function overloads that are specialized for complex.
+
+
+C++ customization point objects
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The standard library uses __ Customization Point Objects __ to enable user-customization of the behavior of many
+algorithms and ranges. We rely heavily on those for most of our iterator machinery such as e.g `cuda::std::begin`.
+
+Those CPOs are currently not accessible in a tile kernel. A potential workaround is to construct an instance of the
+empty type, e.g. ``decltype(cuda::std::begin){}(container);``
+
+C++ return statements in loops and switches
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Tile currently does not support return statements inside of a ``switch`` or a loop. We can work around most places but
+not all. This mainly affects algorithms and arch traits.
+
+In ``<cuda/std/algorithm>`` the following algorithms are not supported in a tile kernel
+
+    - ``cuda::std::equal_range``
+    - ``cuda::std::find_end``
+    - ``cuda::std::find_first_of``
+    - ``cuda::std::partition``
+    - ``cuda::std::search``
+    - ``cuda::std::search_n``
+
+Besides that the content of the following headers is unsupported in a tile program
+
+    - ``<cuda/std/devices>``
+
+
+CUDA device intrinsics
+~~~~~~~~~~~~~~~~~~~~~~~
+
+In tile mode the compiler handles threads, warps and blocks. Consequently, the access of CUDA device intrinsics such as
+``threadIdx`` is currently not allowed in a tile program. Therefore the following headers are not supported in tile mode:
+
+    - <cuda/access_property>
+    - <cuda/annotated_ptr>
+    - <cuda/discard_memory>
+    - <cuda/hierarchy>
+    - <cuda/ptx>
+
+
+CUDA extended floating point types
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Tile programs treat the CUDA extended floating point types as compiler builtin types.
+This disallows accessing their internals which we require internally.
+Support for extended floating point types such as ``__half``, ``__nv_bfloat16`` is only partial in libcu++.
+Support for extended floating point types of size 8, 6 and 4-bit is disabled.
+
+
+Taking the address of a function
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It is currently not supported to take the address of a function in a tile program.
+This affects our memory resource machinery, so the following headers are unsupported in tile mode:
+
+    - <cuda/memory>
+    - <cuda/memory_resource>
+
+----
+
+Standard C++ Tile Availability Summary
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. |V| unicode:: U+2705
+.. |X| unicode:: U+274C
+.. |P| unicode:: U+1F7E8
+
+.. table::
+    :widths: 25 25 10 40
+
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | **Library**                                                                        | ``Libcu++``                                                                          | Supported since | **Notes**                                                                                                 |
+    +====================================================================================+======================================================================================+=================+===========================================================================================================+
+    | `<algorithm> <https://en.cppreference.com/w/cpp/header/algorithm>`_                | :ref:`<cuda/std/algorithm> <libcudacxx-standard-api-algorithms>`                     |   |P| 3.4       | Partial support. Some algorithms rely on return statements inside of loops                                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<array> <https://en.cppreference.com/w/cpp/header/array>`_                        | :ref:`<cuda/std/array> <libcudacxx-standard-api-container-array>`                    |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<atomic> <https://en.cppreference.com/w/cpp/header/atomic>`_                      | ``<cuda/std/atomic>``                                                                |   |X|           | Requires ptx                                                                                              |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<barrier> <https://en.cppreference.com/w/cpp/header/barrier>`_                    | ``<cuda/std/barrier>``                                                               |   |X|           | Requires ptx                                                                                              |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<bit> <https://en.cppreference.com/w/cpp/header/bit>`_                            | :ref:`<cuda/std/bit> <libcudacxx-standard-api-numerics-bit>`                         |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<bitset> <https://en.cppreference.com/w/cpp/header/bitset>`_                      | :ref:`<cuda/std/bitset> <libcudacxx-standard-api-utility-bitset>`                    |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<cassert> <https://en.cppreference.com/w/cpp/header/cassert>`_                    | ``<cuda/std/cassert>``                                                               |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<ccomplex> <https://en.cppreference.com/w/cpp/header/ccomplex>`_                  | ``<cuda/std/ccomplex>``                                                              |   |P| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<cfloat> <https://en.cppreference.com/w/cpp/header/cfloat>`_                      | ``<cuda/std/cfloat>``                                                                |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<charconv> <https://en.cppreference.com/w/cpp/header/charconv>`_                  | ``<cuda/std/charconv>``                                                              |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<chrono> <https://en.cppreference.com/w/cpp/header/chrono>`_                      | :ref:`<cuda/std/chrono> <libcudacxx-standard-api-time>`                              |   |P| 3.4       | Partial support, requires ptx for clock timers                                                            |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<climits> <https://en.cppreference.com/w/cpp/header/climits>`_                    | ``<cuda/std/climits>``                                                               |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<cmath> <https://en.cppreference.com/w/cpp/header/cmath>`_                        |  ``<cuda/std/cmath>``                                                                |   |X|           | Requires unimplemented compiler builtins for math functions.                                              |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<complex> <https://en.cppreference.com/w/cpp/header/complex>`_                    | :ref:`<cuda/std/complex> <libcudacxx-standard-api-numerics-complex>`                 |   |P| 3.4       | Partial support, requires unimplemented compiler builtins for math functions                              |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<concepts> <https://en.cppreference.com/w/cpp/header/concepts>`_                  | :ref:`<cuda/std/concepts> <libcudacxx-standard-api-concepts>`                        |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<cstddef> <https://en.cppreference.com/w/cpp/header/cstddef>`_                    | ``<cuda/std/cstddef>``                                                               |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<cstdint> <https://en.cppreference.com/w/cpp/header/cstdint>`_                    | ``<cuda/std/cstdint>``                                                               |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<cstdlib> <https://en.cppreference.com/w/cpp/header/cstdlib>`_                    | ``<cuda/std/cstdlib>``                                                               |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<cstring> <https://en.cppreference.com/w/cpp/header/cstring>`_                    | ``<cuda/std/cstring>``                                                               |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<ctime> <https://en.cppreference.com/w/cpp/header/ctime>`_                        | ``<cuda/std/ctime>``                                                                 |   |X|           | Requires ptx for clock timers                                                                             |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<execution> <https://en.cppreference.com/w/cpp/header/execution>`_                | :ref:`<cuda/std/execution> <libcudacxx-standard-api-execution>`                      |   |X|           | Requires taking the address of member functions                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<expected> <https://en.cppreference.com/w/cpp/header/expected>`_                  | :ref:`<cuda/std/expected> <libcudacxx-standard-api-utility-expected>`                |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<functional> <https://en.cppreference.com/w/cpp/header/functional>`_              | :ref:`<cuda/std/functional> <libcudacxx-standard-api-utility-functional>`            |   |P| 3.4       | Partial support. Tile C++ disallows taking the address of functions, so ``mem_fn`` et al are not available|
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<initializer_list> <https://en.cppreference.com/w/cpp/header/initializer_list>`_  | ``<cuda/std/initializer_list>``                                                      |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<inplace_vector> <https://en.cppreference.com/w/cpp/header/inplace_vector>`_      | :ref:`<cuda/std/inplace_vector> <libcudacxx-standard-api-container-inplace-vector>`  |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<iterator> <https://en.cppreference.com/w/cpp/header/iterator>`_                  | ``<cuda/std/iterator>``                                                              |   |P| 3.4       | Partial support. We rely heavily on CPO's. This affects e.g ``cuda::std::begin``                          |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<latch> <https://en.cppreference.com/w/cpp/header/latch>`_                        | ``<cuda/std/latch>``                                                                 |   |X|           | Requires ptx                                                                                              |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<limits> <https://en.cppreference.com/w/cpp/header/limits>`_                      | ``<cuda/std/limits>``                                                                |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<linalg> <https://en.cppreference.com/w/cpp/header/linalg>`_                      | :ref:`<cuda/std/linalg> <libcudacxx-standard-api-numerics-linalg>`                   |   |V| 3.4       | Accessors, transposed layout, and related functions                                                       |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<mdspan> <https://en.cppreference.com/w/cpp/header/mdspan>`_                      | :ref:`<cuda/std/mdspan> <libcudacxx-standard-api-container-mdspan>`                  |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<memory> <https://en.cppreference.com/w/cpp/header/memory>`_                      | :ref:`<cuda/std/memory> <libcudacxx-standard-api-utility-memory>`                    |   |V| 3.4       | ``cuda::std::addressof``, ``cuda::std::align``, ``cuda::std::assume_aligned``                             |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<numbers> <https://en.cppreference.com/w/cpp/header/numbers>`_                    | :ref:`<cuda/std/numbers> <libcudacxx-standard-api-numerics-numbers>`                 |   |P| 3.4       | Partial support. ``double`` on windows is not supported. Extended floating point types are not supported. |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<numeric> <https://en.cppreference.com/w/cpp/header/numeric>`_                    | :ref:`<cuda/std/numeric> <libcudacxx-standard-api-numerics-numeric>`                 |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<optional> <https://en.cppreference.com/w/cpp/header/optional>`_                  | :ref:`<cuda/std/optional> <libcudacxx-standard-api-utility-optional>`                |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<ranges> <https://en.cppreference.com/w/cpp/header/ranges>`_                      | :ref:`<cuda/std/ranges> <libcudacxx-standard-api-ranges>`                            |   |P| 3.4       | Partial support. We rely heavily on CPO's. This affects e.g ``cuda::std::ranges::begin``                  |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<ratio> <https://en.cppreference.com/w/cpp/header/ratio>`_                        | ``<cuda/std/ratio>``                                                                 |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<random> <https://en.cppreference.com/w/cpp/header/random>`_                      | :ref:`<cuda/std/random> <libcudacxx-standard-api-numerics-random>`                   |   |P| 3.4       | Partial support. ``seed_seq`` relies on dynamic memory allocations, so it is not available                |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<semaphore> <https://en.cppreference.com/w/cpp/header/semaphore>`_                | ``<cuda/std/semaphore>``                                                             |   |X|           | Requires ptx                                                                                              |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<source_location> <https://en.cppreference.com/w/cpp/header/source_location>`_    | ``<cuda/std/source_location>``                                                       |   |X|           | Requires compiler support                                                                                 |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<span> <https://en.cppreference.com/w/cpp/header/span>`_                          | :ref:`<cuda/std/span> <libcudacxx-standard-api-container-span>`                      |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<tuple> <https://en.cppreference.com/w/cpp/header/tuple>`_                        | :ref:`<cuda/std/tuple> <libcudacxx-standard-api-utility-tuple>`                      |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<type_traits> <https://en.cppreference.com/w/cpp/header/type_traits>`_            | :ref:`<cuda/std/type_traits> <libcudacxx-standard-api-utility-type-traits>`          |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<utility> <https://en.cppreference.com/w/cpp/header/utility>`_                    | :ref:`<cuda/std/utility> <libcudacxx-standard-api-utility-utility>`                  |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<variant> <https://en.cppreference.com/w/cpp/header/variant>`_                    | :ref:`<cuda/std/variant> <libcudacxx-standard-api-utility-variant>`                  |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
+    | `<version> <https://en.cppreference.com/w/cpp/header/version>`_                    | ``<cuda/std/version>``                                                               |   |V| 3.4       |                                                                                                           |
+    +------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------+
diff --git a/cccl_upstream/docs/maintainers/backport_process.rst b/cccl_upstream/docs/maintainers/backport_process.rst
new file mode 100644
index 00000000..1d44e983
--- /dev/null
+++ b/cccl_upstream/docs/maintainers/backport_process.rst
@@ -0,0 +1,46 @@
+How to commit fixes to release branches (backport process)
+==========================================================
+
+This guide explains when and how to commit fixes to release branches.
+
+After a release branch is created and before a release tag is finalized,
+maintainers may need to apply fixes to that release branch.
+
+To keep ``main`` as the source of truth, each fix starts as a PR against
+``main``, then automation opens an equivalent *backport* PR against the
+relevant release branch.
+
+
+Backport Criteria
+-----------------
+
+Before starting a backport, use the questions below to decide if a change is worth backporting:
+
+- Does this fix a correctness bug (wrong result, UB, memory safety, data race, deadlock)?
+- Does this fix a crash?
+- Does this fix a regression?
+- Are users actively asking for this fix to be backported?
+- How likely are users to be affected by this? How many?
+- Is there a reasonable workaround?
+- How risky is this change?
+
+Examples *not* worth backporting:
+
+- Fixes to tests that do not impact the functionality of the library.
+- Fixes to infrastructure that does not impact the functionality of the library.
+
+Steps
+-----
+
+#. Create a PR with the fix against ``main`` via a PR following our :doc:`contributing guidelines </cccl/contributing>`.
+#. Add the label ``backport branch/X.Y.x`` to the PR.
+
+   - If the PR is already merged, you can still trigger a backport by commenting
+     ``/backport branch/X.Y.x`` on the merged PR.
+#. After merge to ``main``, confirm automation opens a backport PR targeting
+   ``branch/X.Y.x``.
+#. Review the generated backport PR for correctness and resolve any conflicts.
+#. Ensure all CI checks have passed and merge the backport PR into the target
+   release branch.
+
+   - Only members of the GitHub team `cccl-release-owners <https://github.com/orgs/NVIDIA/teams/cccl-release-owners>` can merge PRs to release branches.
diff --git a/cccl_upstream/docs/maintainers/branching_strategy.rst b/cccl_upstream/docs/maintainers/branching_strategy.rst
new file mode 100644
index 00000000..a2692017
--- /dev/null
+++ b/cccl_upstream/docs/maintainers/branching_strategy.rst
@@ -0,0 +1,32 @@
+Git Methodology
+=========================
+
+This page defines the canonical branch model used for CCCL development and
+release maintenance.
+
+Canonical branches
+------------------
+
+- ``main``
+
+  - The default development branch.
+  - Updates should be made via pull requests following our :doc:`contributing guidelines </cccl/contributing>`.
+
+- ``branch/X.Y.x``
+
+  - Branches from ``main`` meant for stabilizing and publishing tagged releases.
+  - Created via release automation.
+  - Changes should be made via the :doc:`backport process <backport_process>`.
+
+- ``ctk/X.Y.x``
+
+  - Branches from ``branch/X.Y.x`` meant for representing exact contents of CUDA Toolkit Releases
+  - Created by NVIDIA. May contain internally developed features that are additive with the release it was based on.
+  - This is a read-only branch and is created when the CTK or relevant documentation becomes available.
+
+Tagging conventions
+-------------------
+- ``vX.Y.Z``: finalized release tags on release branches
+- ``vX.Y.Z-ctkN.M.K``: finalized release tags exactly equivalent to the CCCL contents in NVIDIA CUDA Toolkit release `N.M.K`.
+- ``vX.Y.Z-rcN``: release-candidate tags for pre-release validation.
+- ``vX.Y.Z.dev``: the first commit of development for ``X.Y.Z`` (the commit that increments the library version)
diff --git a/cccl_upstream/docs/maintainers/coderabbit.rst b/cccl_upstream/docs/maintainers/coderabbit.rst
new file mode 100644
index 00000000..edab7ccf
--- /dev/null
+++ b/cccl_upstream/docs/maintainers/coderabbit.rst
@@ -0,0 +1,66 @@
+CodeRabbit
+==========
+
+This page explains how to configure and use CodeRabbit for CCCL pull request
+review. For the complete product documentation, see the
+`CodeRabbit documentation <https://docs.coderabbit.ai/>`__.
+
+Configuration
+-------------
+
+CCCL configures CodeRabbit through ``.coderabbit.yaml`` in the repository root.
+The configuration in the pull request branch is used for that review.
+
+When setting up or updating CodeRabbit:
+
+#. Keep repository-specific settings in ``.coderabbit.yaml``.
+#. Use ``@coderabbitai configuration`` on a pull request to inspect the
+   resolved configuration. This is useful when checking whether CodeRabbit is
+   using the expected repository settings for that pull request.
+#. Use ``@coderabbitai generate configuration`` to export the resolved
+   configuration if a new baseline is needed. This is useful when moving
+   settings into a reviewable repository configuration file.
+#. Keep configuration changes small and reviewable.
+#. Use ``reviews.path_instructions`` for path-specific review guidance.
+#. Use ``knowledge_base.code_guidelines.filePatterns`` for CCCL guidance files
+   that CodeRabbit should read as review context.
+
+The CCCL configuration should keep comments focused on correctness, API
+stability, performance, security, and other high-impact issues. Avoid enabling
+features that add noisy generated comments or code by default.
+
+Pull Request Reviews
+--------------------
+
+Automatic reviews may be disabled or restricted by the repository
+configuration. Maintainers can always request review explicitly from a pull
+request comment:
+
+.. code-block:: text
+
+   @coderabbitai review
+
+Use a full review when the pull request should be reviewed again from scratch:
+
+.. code-block:: text
+
+   @coderabbitai full review
+
+Other useful commands:
+
+- ``@coderabbitai help`` shows the current command reference.
+- ``@coderabbitai configuration`` shows the active configuration.
+- ``@coderabbitai summary`` can be placed in the pull request description as a
+  placeholder for the generated summary.
+- ``@coderabbitai pause`` and ``@coderabbitai resume`` pause or resume
+  automatic review behavior. This is useful when a pull request is still being
+  updated frequently and should not be reviewed again until it is ready.
+- ``@coderabbitai ignore`` can be placed in the pull request description to
+  disable automatic reviews.
+
+Review Guidance
+---------------
+
+Treat CodeRabbit feedback as review assistance, not as a merge requirement by
+itself. Maintainers remain responsible for deciding whether comments are
+actionable and whether a pull request has adequate tests and CI coverage.
diff --git a/cccl_upstream/docs/maintainers/how_tos/adding_ci_coverage.rst b/cccl_upstream/docs/maintainers/how_tos/adding_ci_coverage.rst
new file mode 100644
index 00000000..9806777f
--- /dev/null
+++ b/cccl_upstream/docs/maintainers/how_tos/adding_ci_coverage.rst
@@ -0,0 +1,163 @@
+.. _infra-ci-adding-coverage:
+
+Adding CI coverage
+==================
+
+CCCL's CI matrix is defined in ``ci/matrix.yaml``. You add coverage by writing new entries: each
+entry expands into one or more jobs through the cross-product of its array-valued fields. Place the
+entries under the right workflow sections, then validate them with the override matrix before merge.
+
+The field reference for every entry — ``jobs``, ``project``, ``ctk``, ``cxx``, ``std``, ``gpu``,
+``sm``, ``cmake_options``, ``args`` — lives in the ``tags:`` and ``jobs:`` maps in
+``ci/matrix.yaml``. Read those before authoring an entry.
+
+Choose the workflow sections to update
+--------------------------------------
+
+``ci/matrix.yaml`` defines separate matrices per trigger under ``workflows:``.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 80
+
+   * - Section
+     - Runs on
+   * - ``pull_request``
+     - Full coverage for projects modified in a PR. See :ref:`infra-ci-change-detection`.
+   * - ``pull_request_lite``
+     - Light coverage for project downstream of those modified in a PR. See :ref:`infra-ci-change-detection`.
+   * - ``nightly``
+     - Scheduled nightly. Broad compiler and CTK coverage.
+   * - ``weekly``
+     - Scheduled weekly. Widest coverage, including ``all-cccl`` architecture builds.
+
+
+Add the pull request matrix entry
+---------------------------------
+
+Each row targets the specific compiler, CTK, and GPU.
+Keep it narrow. Every array field multiplies the job count.
+
+This entry adds a Thrust test run pinned to one CTK and one GPU, across three host compilers:
+
+.. code-block:: yaml
+
+   - {jobs: ['test'], project: 'thrust', ctk: '<ctk-name>', std: 'max', cxx: ['<cxx-name-1>', '<cxx-name-2>', '<cxx-name-3>'], gpu: '<gpu-name>'}
+
+Use the CTK name from the ``ctk_versions:`` map in ``ci/matrix.yaml`` and the compiler name
+from the ``host_compilers:`` map. Use the GPU pool name from the ``gpus:`` map.
+
+Field by field:
+
+- ``jobs: ['test']`` — runs the ``test`` job. ``test`` requires a GPU and auto-generates its
+  ``build`` producer job (see the ``jobs:`` section). Other projects may have more specialized
+  options besides build+test, but these are the most common.
+- ``project: 'thrust'`` — restricts the entry to one project. Omit to use the default
+  ``['libcudacxx', 'cub', 'thrust']``.
+- ``ctk: '<ctk-name>'`` — the CTK name from the ``ctk_versions:`` map. See ``ci/matrix.yaml``
+  for the current names and what toolkit versions they resolve to. Prefer the convention of
+  ``<major>.X`` when requesting "the latest of this major version", eg. "13.X" instead of "13.2",
+  and only use exact versions when meaningfully required (packaging constraints, minimum versions, etc).
+- ``std: 'max'`` — the highest C++ standard the project supports. Use ``min``, ``minmax``, or
+  ``all`` for wider coverage, or the standard year if specifics are needed (e.g. ``[17, 23]``).
+- ``cxx: [...]`` — one or more host compiler names. The array expands to one job per element.
+- ``gpu: '<gpu-name>'`` — the GPU runner pool to use, see the ``gpus:`` map.
+- ``sm: [...]`` — Request specific CUDA SM architectures (eg. '75' for Turing). Use
+  ``sm: 'gpu'`` to build for only the arch needed by the requested ``gpu``.
+
+Add a build-only entry the same way with ``jobs: ['build']`` and no ``gpu``. Build jobs run on
+CPU-only runners.
+
+Add the pull request lite matrix entry
+--------------------------------------
+
+Rare, but if this is important coverage that is cheap, it may be worth adding a ``pull_request_lite`` entry.
+Jobs from this matrix are added to the PR run when an upstream internal dependency is modified.
+The goal is to keep this matrix as light as possible, extensions should be rare and well justified.
+
+Note that these jobs will **NOT** run as part of the PR that adds them, so they **MUST** be tested with
+the override matrix before merge.
+
+See :ref:`infra-ci-change-detection` for how CCCL's CI encodes these dependencies.
+
+Add the corresponding nightly / weekly entries
+----------------------------------------------
+
+The ``nightly`` and ``weekly`` entries run on a schedule, and carry the exhaustive, broad coverage that would
+be wasteful and excessive for PRs.
+It does not include the pull request matrix, so the PR jobs must be replicated, and possibly extended, here.
+Note that these jobs will **NOT** run as part of the PR that adds them, so they **MUST** be tested with
+the override matrix before merge.
+
+Group the new entries under the existing comment headers in each section. Keep CTK and project
+groupings together so the matrix stays readable.
+
+Update project_files_and_dependencies.yaml
+------------------------------------------
+
+The full details of this system are documented in :ref:`infra-ci-change-detection`.
+
+``ci/inspect_changes.py`` reads ``ci/project_files_and_dependencies.yaml`` to decide which projects
+a PR touched. Update this file when the new coverage involves a project or a source path the file
+does not already track. Skip this step when adding configurations to an existing project's existing
+paths.
+
+Add or extend a project entry so changed files map to the right matrix project:
+
+.. code-block:: yaml
+
+   my_project_public:
+     name: "My Project Public API" # public/API entries only include public headers.
+     lite_dependencies: [libcudacxx_public] # lite dependency on libcu++'s public headers (upstream only triggers lite PR coverage)
+     full_dependencies: []
+     include_regexes: ["my_project/include/"] # path to public headers
+
+   my_project_internal:
+     name: "My Project Tests/Infra" # Internal entries exclude public headers, include everything else.
+     matrix_project: "my_project" # Maps to the matrix.yaml project that will be triggered when project files change
+     lite_dependencies: []
+     full_dependencies: [my_project_public] # trigger the full PR coverage when public headers change.
+                                            # changes to transitive deps (eg. libcudacxx_public) will onlytrigger lite PR coverage.
+     include_regexes: ["my_project/"] # path to project root
+     exclude_project_files: [my_project_public] # ignore files matched by the public entry.
+
+- ``matrix_project`` — ties the change-detection key to the ``projects:`` key in ``ci/matrix.yaml``.
+  Without it the project never enters the build list.
+- ``include_regexes`` — paths that mark this project dirty, anchored to the repo root.
+- ``lite_dependencies`` / ``full_dependencies`` - See the file comments or :ref:`infra-ci-change-detection` for details.
+
+**Files matching no project fall into ``core`` and trigger a full build of everything.**
+
+If you're adding files that should not ever trigger CI, add them to the top-level ``ignore_regexes`` list to exclude them from
+change detection.
+
+Test the entry with the override matrix
+---------------------------------------
+
+Validate new entries with ``workflows.override`` before merge. A non-empty ``override`` replaces the
+entire ``pull_request`` matrix for the PR, so CI runs only the entries you are testing. The override
+blocks merge until removed, which guarantees the full suite runs before the change lands.
+
+#. **Copy the candidate entries into override.** Place the new ``pull_request`` and ``nightly``
+   entries under ``workflows.override`` in ``ci/matrix.yaml``:
+
+   .. code-block:: yaml
+
+      workflows:
+        override:
+          - {jobs: ['test'], project: 'thrust', ctk: '<ctk-name>', std: 'max', cxx: ['<cxx-name-1>', '<cxx-name-2>'], gpu: '<gpu-name>'}
+
+#. **Trim unrelated jobs.** Add ``[skip-tpt][skip-docs]`` to the **last commit message** to drop
+   third-party tests (eg RAPIDS, MatX) and doc builds while iterating.
+
+#. **Push and inspect.** The PR runs only the override entries. Confirm the jobs appear with the
+   expected compiler, CTK, and GPU, and that they pass.
+
+#. **Reset before merge.** Empty ``workflows.override`` and remove the ``[skip-*]`` tags from the
+   last commit message (or push a new commit). The merge gate fails until both are clean.
+
+For a tighter loop on a single test target, use ``project: 'target'`` with ``args`` forwarded to
+``ci/util/build_and_test_targets.sh``. The commented examples at the top of ``ci/matrix.yaml`` show
+the ``run_cpu`` and ``run_gpu`` invocation patterns. Reproduce any failing job locally with
+``.devcontainer/launch.sh`` and the matching ``ci/`` build or test script (see
+:ref:`infra-ci-reproducing-locally`).
diff --git a/cccl_upstream/docs/maintainers/how_tos/adding_new_devcontainer.rst b/cccl_upstream/docs/maintainers/how_tos/adding_new_devcontainer.rst
new file mode 100644
index 00000000..5d28bd32
--- /dev/null
+++ b/cccl_upstream/docs/maintainers/how_tos/adding_new_devcontainer.rst
@@ -0,0 +1,70 @@
+.. _infra-devcontainer-adding-toolchain:
+
+Adding a new devcontainer toolchain
+===================================
+
+A toolchain is one CTK version paired with one host compiler. CCCL generates a
+devcontainer config for every combination listed in the ``devcontainers:`` section of
+``ci/matrix.yaml``. The generated configs live under ``.devcontainer/<name>/devcontainer.json``,
+one directory per combination, all produced by ``.devcontainer/make_devcontainers.sh [--clean]``.
+
+Run the steps in order: edit the matrix, regenerate the configs, then verify before merge. The
+base image for the combination must already exist in the
+`rapidsai/devcontainers <https://github.com/rapidsai/devcontainers>`_ project before the matrix
+edit.
+
+Check that the base image exists
+--------------------------------
+
+Every CCCL devcontainer is built on a published ``rapidsai/devcontainers`` image. Image tags
+follow this pattern::
+
+    rapidsai/devcontainers:<devcontainer_version>-cpp-<compiler><version>-cuda<ctk>[ext]
+
+The ``-cuda<ctk>`` segment is present for every combination except nvhpc, which bundles its
+own CUDA toolkit and omits it.
+
+The ``<devcontainer_version>`` value is the ``devcontainer_version:`` field in ``ci/matrix.yaml``.
+
+The images are maintained in the https://github.com/rapidsai/devcontainers/ repo, in the top-level
+matrix file. If new images are required for the coverage, submit a PR against `main`.
+
+Add the combination to ci/matrix.yaml
+-------------------------------------
+
+The source-of-truth when generating devcontainer toolchains is the ``matrix.yaml`` file. All jobs
+from all workflows are parsed, the toolchains extracted, and the ``.devcontainer/...`` directories built.
+
+At the bottom of the workflows section of ``matrix.yaml`` is a ``devcontainers:`` section.
+This is intended to be a living mirror of the available images in the `rapidsai/devcontainers` repo,
+and is useful for quickly checking supported CTK / host compilers while editing the matrix.
+Occasionally we'll need a devcontainer that isn't referenced in any workflow, and this section is the place to add it.
+Make sure that your new toolchain is listed and documented here.
+
+Regenerate the devcontainer configs
+-----------------------------------
+
+From the repository root, regenerate every ``.devcontainer/<name>/devcontainer.json`` from the updated matrix:
+
+.. code-block:: bash
+
+    .devcontainer/make_devcontainers.sh --clean
+
+The script reads all matrix workflow entries, expands aliases, and writes one directory per combination
+using the naming pattern ``cuda<version>[ext]-<compiler><version>``.
+It also updates the root ``.devcontainer/devcontainer.json`` default to the newest GCC + newest
+CUDA combination.
+Pass ``--clean`` to remove directories for combinations no longer in the matrix (recommended).
+
+Never hand-edit a generated ``.devcontainer/<name>/devcontainer.json``. Edits are overwritten on
+the next run. To change settings that apply to every combination, edit the root
+``.devcontainer/devcontainer.json`` template, then rerun the generator to propagate the change.
+
+Verify before merge
+-------------------
+
+Locally test launching the devcontainer using the appropriate ``.devcontainer/launch.sh`` invocation.
+See :ref:`infra-devcontainer-launching` for details on launching and using the devcontainer.
+
+The ``verify-devcontainers`` CI workflow reruns ``make_devcontainers.sh --verbose --clean``
+and fails if the result differs from the committed files.
diff --git a/cccl_upstream/docs/maintainers/how_tos/compile_time.rst b/cccl_upstream/docs/maintainers/how_tos/compile_time.rst
new file mode 100644
index 00000000..07a37106
--- /dev/null
+++ b/cccl_upstream/docs/maintainers/how_tos/compile_time.rst
@@ -0,0 +1,115 @@
+Run compile-time benchmarks
+===========================
+
+Use the compile-time benchmark when you want to compare how a change affects
+CUDA TU compile time. The common workflow compares the current tree against
+``origin/main`` and reports the most important movements in generated public
+include-check TUs.
+
+For the full option, CSV, filter, and CI contract reference, see
+:doc:`../references/compile_time`.
+
+Run the common comparison
+-------------------------
+
+From the repository root, run:
+
+.. code-block:: bash
+
+  ci/build_compile_time_bench.sh \
+    -baseline-ref origin/main \
+    -- --slices /path/to/slices.json
+
+The CI slices live in ``ci/matrix.yaml`` under
+``compile_time.pull_request[].slices``. To reproduce the PR shape locally, copy
+those slice definitions to a JSON file shaped as:
+
+.. code-block:: json
+
+  {
+    "slices": [
+      {
+        "id": "file-processing",
+        "title": "Direct file processing",
+        "filter": "file-processing",
+        "timing": "exclusive",
+        "sort": "total",
+        "top": 15,
+        "threshold": 0.2
+      }
+    ]
+  }
+
+The wrapper builds both the current tree and the baseline commit with the same
+preset, target set, architecture, and other build arguments. It then writes
+baseline reports, current reports, comparison CSVs, a ``summary.json`` manifest,
+raw traces, and Perfetto-friendly traces under the preset build directory:
+
+.. code-block:: text
+
+  build/<infix>/<preset>/compile_time/
+
+Run a quick single-slice report
+-------------------------------
+
+For iteration after traces already exist, skip the build and regenerate only an
+event report:
+
+.. code-block:: bash
+
+  ci/build_compile_time_bench.sh -skip-build -- \
+    -f file-processing -e --sort total -n 25
+
+Other useful built-in filters include:
+
+- ``total-compilation``
+- ``file-processing``
+- ``scanning-function-body``
+- ``template-instantiation``
+- ``host-compiler``
+- ``code-generation``
+- ``all``
+
+Interpret the PR comment
+------------------------
+
+The PR comment contains one section per configured slice. Within each slice,
+regressions and improvements are intentionally separated. Rows are ranked by
+total impact across matched traces, not just by the largest single-TU movement.
+
+Important columns:
+
+- ``Regression impact`` / ``Improvement impact``: absolute total-impact change
+  across matched traces, in seconds.
+- ``Selected Δ``: signed movement in the selected metric for the event.
+- ``Baseline`` / ``Current``: selected metric values on each side.
+- ``Matched traces``: number of generated TUs where the event key was comparable.
+
+Small movements are filtered by per-slice thresholds from ``ci/matrix.yaml``.
+Those thresholds are intentionally non-zero to hide ordinary run-to-run noise.
+
+Inspect traces in Perfetto
+--------------------------
+
+The wrapper prepares trace copies whose event names include the useful file or
+symbol detail. Open files under:
+
+.. code-block:: text
+
+  build/<infix>/<preset>/compile_time/perfetto_traces/
+
+in Perfetto or another Chrome-trace-compatible viewer.
+
+Skip compile-time benchmark telemetry
+-------------------------------------
+
+Compile-time benchmark jobs are informational and do not gate the aggregate PR
+``CI`` job. For early iterations where the compile-time benchmark is unrelated,
+append this case-sensitive tag to the commit message:
+
+.. code-block:: text
+
+  [skip-compile-time-bench]
+
+Remove the tag before requesting final review if the compile-time benchmark
+scripts, workflow, or configuration changed.
diff --git a/cccl_upstream/docs/maintainers/how_tos/index.rst b/cccl_upstream/docs/maintainers/how_tos/index.rst
new file mode 100644
index 00000000..307d1f5b
--- /dev/null
+++ b/cccl_upstream/docs/maintainers/how_tos/index.rst
@@ -0,0 +1,13 @@
+How Tos
+=======
+
+How-to guides for maintainers.
+
+.. toctree::
+   :maxdepth: 1
+
+   ../backport_process
+   compile_time
+   plan_work_with_github_issues
+   adding_ci_coverage
+   adding_new_devcontainer
diff --git a/cccl_upstream/docs/maintainers/how_tos/plan_work_with_github_issues.rst b/cccl_upstream/docs/maintainers/how_tos/plan_work_with_github_issues.rst
new file mode 100644
index 00000000..c4fcb334
--- /dev/null
+++ b/cccl_upstream/docs/maintainers/how_tos/plan_work_with_github_issues.rst
@@ -0,0 +1,108 @@
+How To Plan Work with GitHub Issues
+===================================
+
+CCCL uses `GitHub issues <https://github.com/NVIDIA/cccl/issues>`__ and the
+`CCCL GitHub Project <https://github.com/orgs/NVIDIA/projects/6>`__ as the source of truth to make
+planned work visible, understandable, and coordinated.
+
+This guide explains how CCCL plans work, records that
+work in closable issues, and keeps priorities and ownership visible over time.
+
+CCCL plans work in monthly sprints, with each sprint identifying the issues the
+team intends to prioritize during that 4-week window.
+
+Understand current planned work
+-------------------------------
+
+The `Current Sprint <https://github.com/orgs/NVIDIA/projects/6/views/47>`__ view in the GitHub Project contains
+the planned work the team has agreed to prioritize now. It is not a complete
+list of all engineering activity.
+
+Planned work intentionally leaves capacity for reviews, support, debugging, and
+other interrupt-driven work.
+
+In-progress work is not automatically carried forward into the next planning
+cycle. Each cycle is planned from current priorities.
+
+Write a closable issue
+----------------------
+
+Every issue should answer:
+
+   "This issue can be closed when..."
+
+Examples:
+
+- a bug is fixed;
+- a refactoring is complete;
+- a benchmark is added;
+- a design decision is documented;
+- follow-on issues are created;
+- an investigation summary is written.
+
+Large efforts can start as tracking issues whose scope is still evolving. As
+concrete work becomes clear, create sub-issues for work with independent close
+conditions.
+
+Propose planned work
+--------------------
+
+Sprint planning happens monthly. Team leads review the roadmap, current priorities,
+and incoming requests to identify the highest-impact work for the upcoming
+four-week planning window.
+
+If you think work should be considered for the next planning cycle, raise it
+during planning or ask a team lead to add it to the
+`Sprint Planning <https://github.com/orgs/NVIDIA/projects/6/views/58>`__ view.
+
+During planning, the team reviews proposed issues, confirms ownership,
+identifies gaps or dependencies, and checks that issues are actionable and
+up-to-date.
+
+Confirmed planned work is moved to the
+`Current Sprint <https://github.com/orgs/NVIDIA/projects/6/views/47>`__ view.
+
+Keep assigned issues current
+----------------------------
+
+The assignee is responsible for driving the issue forward and keeping it
+accurate while the work evolves.
+
+Update the issue when:
+
+- the title no longer describes the work clearly;
+- the close condition changes;
+- the work has been split into sub-issues;
+- important context, decisions, or blockers appear;
+- the issue is no longer relevant and should be closed.
+
+Use issue comments for questions, decisions, blockers, and context that
+others may need to find later. If a comment changes the issue's scope or close
+condition, update the issue body as well.
+
+Complete assigned issues
+------------------------
+
+Use the issue's close condition to decide when the work is done. Often, this
+means opening a PR that completes the work described by the issue. Follow the
+:doc:`contributing guidelines </cccl/contributing>` for how to prepare and
+submit the PR.
+
+Link the PR to the issue so GitHub can close the issue when the PR merges. For
+example, include ``Fixes #123`` in the PR description.
+
+Linked PRs also keep the GitHub Project status current. A linked draft PR keeps
+the issue ``In Progress``. When the PR is ready for review, automation moves the
+issue to ``Review``.
+
+Plan release-critical work
+--------------------------
+
+Upcoming releases have corresponding
+`GitHub milestones <https://github.com/NVIDIA/cccl/milestones>`__.
+
+Team leads create release milestones and assign issues to them during planning.
+Add an issue to a release milestone when the work is
+important to complete and include in that release.
+
+Not all current planned work is tied to a particular release.
diff --git a/cccl_upstream/docs/maintainers/index.rst b/cccl_upstream/docs/maintainers/index.rst
new file mode 100644
index 00000000..657fb10b
--- /dev/null
+++ b/cccl_upstream/docs/maintainers/index.rst
@@ -0,0 +1,10 @@
+Maintainer Docs
+===============
+
+This section documents the policies and procedures to maintain and release artifacts from the CCCL repository.
+
+.. toctree::
+   :maxdepth: 1
+
+   how_tos/index
+   references/index
diff --git a/cccl_upstream/docs/maintainers/references/compile_time.rst b/cccl_upstream/docs/maintainers/references/compile_time.rst
new file mode 100644
index 00000000..d3c6d2d0
--- /dev/null
+++ b/cccl_upstream/docs/maintainers/references/compile_time.rst
@@ -0,0 +1,300 @@
+Compile-time benchmark reference
+================================
+
+The compile-time benchmark builds generated one-include CUDA translation units
+(TUs) and summarizes NVCC ``--fdevice-time-trace`` output. It is a TU
+compile-time benchmark, even when the current input set comes from public
+include-check targets.
+
+Entry point
+-----------
+
+The top-level entry point is:
+
+.. code-block:: bash
+
+  ci/build_compile_time_bench.sh
+
+The wrapper configures a caller-selected CMake preset with compile-time
+instrumentation, builds selected targets, prepares Perfetto-friendly trace
+copies, writes a generated-TU summary CSV, and emits one or more event summary
+CSVs.
+
+Generated outputs
+-----------------
+
+By default, outputs are written under:
+
+- ``build/<infix>/<preset>/compile_time/tu_summary.csv``
+- ``build/<infix>/<preset>/compile_time/event_reports/``
+- ``build/<infix>/<preset>/compile_time/perfetto_traces/``
+
+Raw NVCC traces are generated under:
+
+- ``build/<infix>/<preset>/compile_time/raw_traces/``
+
+When ``-baseline-ref`` is used, baseline raw traces are copied out of the
+temporary baseline worktree before cleanup and preserved under:
+
+- ``build/<infix>/<preset>/compile_time/baseline_raw_traces/``
+
+Build controls
+--------------
+
+The wrapper accepts build-shape parameters so it behaves like other
+``ci/build*.sh`` entry points:
+
+.. code-block:: bash
+
+  PARALLEL_LEVEL=16 ci/build_compile_time_bench.sh \
+    -preset all-dev \
+    -target libcudacxx.test.public_headers \
+    -cmake-options "-DCMAKE_CUDA_ARCHITECTURES=native"
+
+Useful build options include:
+
+- ``-preset <name>``
+- ``-cmake-options <args>``
+- ``-target <name>`` (repeatable; replaces the default public include-check target set)
+- ``-baseline-ref <commit-ish>`` (build a temporary baseline worktree for comparison)
+- ``-skip-configure``
+- ``-skip-build``
+
+The default target set is the current public include-check target set:
+
+- ``cub.headers.base``
+- ``thrust.cpp.cuda.headers.base``
+- ``libcudacxx.test.public_headers``
+
+The one-include source TUs are created by those CMake targets themselves. The
+benchmark wrapper only enables extra compile options on generated CUDA TUs:
+``CCCL_COMPILE_TIME_GENERATE_DEVICE_TIME_TRACES`` writes NVCC device-time trace
+JSON, and ``CCCL_COMPILE_TIME_SAVE_PREPROCESSED_TUS`` preserves compiler
+preprocessed/temporary TU artifacts for the TU summary.
+
+Event summaries
+---------------
+
+Arguments after ``--`` are forwarded to
+``ci/compile_time/summarize_events.py`` after the raw trace directory. If no
+event-summary arguments are provided, the wrapper runs:
+
+.. code-block:: bash
+
+  -f file-processing -e -n 15
+
+For CI-style multi-slice reports, pass a JSON slice file:
+
+.. code-block:: bash
+
+  ci/build_compile_time_bench.sh -baseline-ref origin/main -- \
+    --slices /path/to/slices.json
+
+The slice file contains a ``slices`` array. Each slice has a stable ``id``,
+display ``title``, ``filter``, ``timing`` (``inclusive`` or ``exclusive``),
+``sort``, ``top``, and ``threshold`` in seconds. Multi-slice mode writes each
+slice under ``event_reports/<slice-id>/`` and writes a normalized
+``event_reports/summary.json`` manifest for PR comment rendering.
+
+Empty slices are represented in the manifest and CSVs. Slices that match no
+events, have no matching trace files, or have no comparable event keys record
+warnings so they are visible in PR comments instead of looking like ordinary
+no-change results. Empty slices with no warnings are omitted recursively by the
+PR comment renderer.
+
+Single-slice examples:
+
+.. code-block:: bash
+
+  ci/build_compile_time_bench.sh -skip-build -- -f scanning-function-body -i -n 20
+  ci/build_compile_time_bench.sh -skip-build -- -f template-instantiation -e -n 15 --tag templates
+  ci/build_compile_time_bench.sh -skip-build -- -f 'Scanning|Instantiating' -i -n 25
+  ci/build_compile_time_bench.sh -skip-build -- -f code-generation -i --scope-filter ""
+
+Baseline comparisons
+--------------------
+
+Pass ``-baseline-ref <commit-ish>`` to compare the current tree state against a
+baseline commit. The wrapper creates a temporary detached worktree for the
+baseline, builds both the current tree and the baseline with the same preset,
+targets, and common build options, then runs the requested event slice as a
+baseline/current comparison:
+
+.. code-block:: bash
+
+  ci/build_compile_time_bench.sh \
+    -baseline-ref origin/main \
+    -- -f file-processing -e --sort total -n 25 --threshold 0.2
+
+Comparison mode writes three subdirectories under
+``<preset-build-dir>/compile_time/event_reports/``:
+
+- ``baseline/``: the normal report for the baseline traces
+- ``current/``: the normal report for the current traces
+- ``comparison/``: ``worse`` and ``better`` CSVs for the requested filter,
+  timing, exclusivity, sort, and top-N slice
+
+In multi-slice comparison mode, the same layout appears under each
+``event_reports/<slice-id>/`` directory.
+
+Trace files are matched by relative path. Delta CSVs only compare event keys
+that appear in both sides of the same matched trace file. Unmatched child events
+are still counted in their matched parent event's exclusive cost, so disappearing
+or newly appearing nested work remains visible as a parent cost change instead
+of being subtracted away. If there are no comparable event keys, the comparison
+CSVs are still written with headers and no rows.
+
+Pass ``--threshold <seconds>`` after the wrapper's ``--`` separator in
+comparison mode to omit ``worse`` / ``better`` rows whose total impact change is
+not greater than that threshold. Baseline/current reports use ``--sort`` for
+their own top-N ordering, but comparison reports are ranked by total impact
+across all matched traces so repeated small movements outrank a larger movement
+in only one trace.
+
+When ``-baseline-ref`` is used, the wrapper treats the invocation as an event
+comparison and skips the generated-TU CSV unless ``-tu-csv`` is provided
+explicitly. To compare arbitrary trace directories outside the wrapper layout,
+run ``ci/compile_time/summarize_events.py`` directly.
+
+Pull-request reporting
+----------------------
+
+Compile-time PR reporting is configured in ``ci/matrix.yaml`` under
+``compile_time.pull_request``. Each config selects the GPU runner, devcontainer
+launch arguments, baseline ref, preset, targets, wrapper arguments, and report
+slices. ``ci/compile_time/parse_matrix.py`` validates that section and emits the
+GitHub Actions matrix for the reusable compile-time benchmark workflow.
+
+The reusable workflow uploads:
+
+- event report CSVs and ``summary.json``
+- current raw traces
+- baseline raw traces
+- Perfetto-friendly traces
+- the rendered PR comment body
+
+``ci/compile_time/render_pr_comment.py`` renders the comment from
+``summary.json``. Regressions and improvements are rendered in separate
+``<details>`` sections and are never mixed in one table. Slice warnings are
+rendered separately. Empty sections with no warnings are omitted recursively.
+Sticky comments are keyed by ``compile-time-bench-<config-id>``; previous
+comments for the same config are archived as outdated when a new one is posted.
+
+This reporting is informational and is not part of the aggregate branch
+protection ``CI`` job. Commit messages containing ``[skip-compile-time-bench]``
+skip compile-time benchmark dispatch.
+
+Filters and scope filtering
+---------------------------
+
+Built-in filter names include:
+
+- ``file-processing``
+- ``scanning-function-body``
+- ``template-instantiation``
+- ``template-class-instantiation``
+- ``template-function-instantiation``
+- ``pending-instantiations``
+- ``frontend``
+- ``host-compiler``
+- ``code-generation``
+- ``optimizer``
+- ``total-compilation``
+- ``all``
+
+Unknown filters are interpreted as case-insensitive regular expressions over
+event names and event details.
+
+Symbol-like events, such as function parsing, template instantiation, function
+IR generation, and optimizer-function events, are scope-filtered by default to
+top-level CCCL-owned namespaces:
+
+- ``cuda::``
+- ``thrust::``
+- ``cub::``
+- ``cccl::``
+
+This keeps reports focused on CCCL symbols instead of system library symbols
+pulled into the same generated TU. The filter applies to demangled trace details
+and to decoded namespace prefixes from Itanium-mangled symbols in trace details.
+It does not filter path/phase events such as file processing, host compiler
+phases, or total compilation time.
+
+Pass ``--scope-filter <regex>`` after the wrapper's ``--`` separator to choose
+a different case-sensitive symbol-scope regex. Pass an empty string to disable
+symbol-scope filtering:
+
+.. code-block:: bash
+
+  ci/build_compile_time_bench.sh -skip-build -- \
+    -f template-instantiation -i --scope-filter ""
+
+``host-compiler`` matches the host compiler preprocessing / compiling events
+that appear in the device-time-trace output. ``total-compilation`` is a
+synthetic per-trace event whose inclusive time is the wall-clock span from the
+first timed trace event to the last timed trace event. This includes host
+compiler, cudafe, NVVM, fatbinary, and gaps visible inside the trace timeline,
+but it is not a separate external wall-clock measurement of untraced driver,
+Ninja, or process-launch overhead.
+
+Compared to Ninja log timings, ``total-compilation`` generally undercounts each
+TU by a small, consistent amount because the trace span starts at the first
+timed event and ends at the last timed event rather than at process launch/exit.
+That makes it a good relative-comparison and ranking metric, but not an exact
+replacement for external wall-clock command duration.
+
+Use ``--sort`` to choose the selected ranking metric:
+
+- ``total``
+- ``avg``
+- ``avg-root-tu``
+- ``max``
+
+Perfetto trace preparation
+--------------------------
+
+The wrapper prepares Perfetto-friendly trace copies by default. To prepare an
+existing trace directory manually:
+
+.. code-block:: bash
+
+  ci/compile_time/prepare_traces.py \
+    --input build/<infix>/<preset>/compile_time/raw_traces \
+    --output /tmp/compile_time_perfetto
+
+CSV output
+----------
+
+``summarize_tus.py`` writes:
+
+- ``tu_input``
+- ``transitive_loc``
+- ``tu_source``
+- ``preprocessed_tu``
+
+This CSV is a generated-TU input/LOC summary. Use the ``total-compilation``
+event filter when you need per-TU compile-time rankings from trace data.
+
+``summarize_events.py`` writes stable event keys and both inclusive and
+exclusive metrics, including:
+
+- ``event_name``
+- ``event_key`` (repo-root-relative path for project file-processing events)
+- ``selected_total_s``
+- ``selected_avg_per_event_s``
+- ``selected_avg_per_root_tu_s``
+- ``total_inclusive_s`` / ``total_exclusive_s``
+- ``event_count``
+- ``trace_count``
+- ``root_tu_count``
+
+Comparison CSVs additionally include total-impact columns
+(``baseline_impact_s``, ``current_impact_s``, ``impact_delta_s``,
+``impact_magnitude_s``) plus selected-metric columns
+(``baseline_selected_s``, ``current_selected_s``, ``selected_delta_s``,
+``selected_magnitude_s``).
+
+Notebook workflow
+-----------------
+
+For exploratory analysis, use ``ci/compile_time/analytics.ipynb``.
diff --git a/cccl_upstream/docs/maintainers/references/index.rst b/cccl_upstream/docs/maintainers/references/index.rst
new file mode 100644
index 00000000..82bbe3b2
--- /dev/null
+++ b/cccl_upstream/docs/maintainers/references/index.rst
@@ -0,0 +1,11 @@
+References
+==========
+
+Reference documentation for maintainers.
+
+.. toctree::
+   :maxdepth: 1
+
+   ../branching_strategy
+   ../coderabbit
+   compile_time
diff --git a/cccl_upstream/docs/python/api_reference.rst b/cccl_upstream/docs/python/api_reference.rst
new file mode 100644
index 00000000..497b219c
--- /dev/null
+++ b/cccl_upstream/docs/python/api_reference.rst
@@ -0,0 +1,7 @@
+API Reference
+=============
+
+.. toctree::
+   :maxdepth: 1
+
+   compute_api
diff --git a/cccl_upstream/docs/python/compute/developer_overview.rst b/cccl_upstream/docs/python/compute/developer_overview.rst
new file mode 100644
index 00000000..94a3c129
--- /dev/null
+++ b/cccl_upstream/docs/python/compute/developer_overview.rst
@@ -0,0 +1,751 @@
+``cuda.compute`` Developer Overview
+===================================
+
+This document provides an overview of the internal structure of
+``cuda.compute``. At a high level, ``cuda.compute`` exposes CUDA C++
+parallel algorithms through a Python API. Internally, it combines
+Python-side operator compilation, CUDA C++ source generation, and
+runtime just-in-time (JIT) compilation and linking.
+
+We start with a simplified prototype. As we encounter the limitations
+of that simplified model, we introduce the additional mechanisms
+needed by the full implementation, referring to the relevant source
+code where useful.
+
+We begin with a minimal example that invokes a CUDA C++ kernel from
+Python. In this simplified prototype, the kernel takes a single
+integer argument and prints it.
+
+.. code-block:: c++
+
+    #include <cstdio>
+
+    __global__ void kernel(int value) {
+        std::printf("thread %d: %d\n", threadIdx.x, value);
+    }
+
+    extern "C" void launcher(int value) {
+      kernel<<<1, 4>>>(value);
+      cudaDeviceSynchronize();
+    }
+
+We can compile this code using ``nvcc``:
+
+.. code-block:: bash
+
+    nvcc -Xcompiler=-fPIC -x cu kernel.cu -shared -o libkernel.so
+
+The resulting shared library exports the host function ``launcher``,
+which we can call from Python using ``ctypes``:
+
+.. code-block:: python
+
+    import ctypes
+
+    bindings = ctypes.CDLL('libkernel.so')
+    bindings.launcher.argtypes = [ctypes.c_int]
+    bindings.launcher(42)
+
+Running that Python code produces:
+
+.. code-block:: bash
+
+    thread 0: 42
+    thread 1: 42
+    thread 2: 42
+    thread 3: 42
+
+The example above works because all of the behavior is fixed ahead of
+time in the CUDA C++ source. The kernel and the operation it performs
+are both known in advance.
+
+A library primitive such as reduction is different. Its behavior
+depends not only on the input type, but also on the operator being
+applied. A practical Python API therefore cannot be limited to a
+single built-in case such as summing ``float`` values. It needs to
+support many data types and user-provided operators.
+
+That means the CUDA C++ side must be able to invoke device code that
+originates in Python. Reduction is a useful motivating example, but to
+keep the mechanics simple we will start with a much smaller building
+block: compiling a simple Python function and making it callable from
+CUDA C++. The same technique later applies to user-provided reduction
+operators.
+
+We can compile such a Python function to PTX using
+`Numba-CUDA <https://nvidia.github.io/numba-cuda/>`_ as follows:
+
+.. code-block:: python
+
+    import numba.cuda
+
+    def op(value):
+        return 2 * value
+
+    ptx, _ = numba.cuda.compile(op, sig=numba.int32(numba.int32))
+
+That'd give us the following PTX code:
+
+.. code-block:: bash
+
+    .visible .func  (.param .b32 func_retval0) op(.param .b32 op_param_0)
+    {
+            .reg .b32       %r<3>;
+
+
+            ld.param.u32    %r1, [op_param_0];
+           	shl.b32 	%r2, %r1, 1;
+            st.param.b32    [func_retval0+0], %r2;
+            ret;
+    }
+
+At this point, the Python function has been compiled to device code,
+but the CUDA C++ side still needs a way to refer to it.
+
+Conceptually, we would like to treat the operator as an externally
+defined device function and call it from CUDA C++:
+
+.. code-block:: c++
+
+    #include <cstdio>
+
+    extern "C" __device__ int op(int a); // defined in Python
+
+    extern "C" __global__ void kernel(int value) {
+        std::printf("thread %d: %d\n", threadIdx.x, op(value));
+    }
+
+    extern "C" void launcher(int value) {
+      kernel<<<1, 4>>>(value);
+      cudaDeviceSynchronize();
+    }
+
+This raises the next question: how do we combine device code produced
+from Python with CUDA C++ code that calls it?
+
+The difficulty is not just that the operator's implementation comes
+from Python. The CUDA C++ side must also declare and call that
+operator with the correct signature.
+
+In the code above, the operator has the fixed signature ``int
+op(int)``. A real API cannot assume that. The user might supply an
+operator on ``float``, ``complex``, or some user-defined type, and the
+generated CUDA C++ code has to match that interface exactly. In other
+words, the declaration of ``op`` and the CUDA C++ source that calls it
+depend on the user's types and operator signature.
+
+That means the CUDA C++ side must be generated and compiled at
+runtime. Using ``nvcc`` for that would make the API depend on an
+external compiler toolchain being available on every user machine.
+Instead, we use NVRTC, which is designed for runtime compilation of
+CUDA C++.
+
+Our Python code is now:
+
+.. code-block:: python
+
+    import ctypes
+    import numba.cuda
+
+    def op(value):
+        return 2 * value
+
+    ptx, _ = numba.cuda.compile(op, sig=numba.int32(numba.int32))
+
+    bindings = ctypes.CDLL('./build/libkernel.so')
+    bindings.launcher.argtypes = [ctypes.c_int, ctypes.c_char_p, ctypes.c_int]
+    bindings.launcher(42, ptx.encode('utf-8'), len(ptx))
+
+Correspondingly, the C++ launcher now accepts the operator PTX as an
+additional argument. Inside the launcher, the CUDA C++ kernel is now
+assembled as a source string and compiled with NVRTC:
+
+.. code-block:: c++
+
+    extern "C" void launcher(int value,
+                             const char* op_ptx, int op_ptx_size)
+    {
+      cudaSetDevice(0);
+
+      // Kernel is now a string!
+      std::string kernel_source = R"XXX(
+        extern "C" __device__ int op(int a);
+
+        extern "C" __global__ void kernel(int value) {
+            printf("thread %d prints value %d\n", threadIdx.x, op(value));
+        }
+      )XXX";
+
+Once that source string has been assembled, we compile it to PTX with
+NVRTC:
+
+.. code-block:: c++
+
+      nvrtcProgram prog;
+      const char *name = "test_kernel";
+      nvrtcCreateProgram(&prog, kernel_source.c_str(), name, 0, nullptr, nullptr);
+
+      cudaDeviceProp deviceProp;
+      cudaGetDeviceProperties(&deviceProp, 0);
+
+      const int cc_major = deviceProp.major;
+      const int cc_minor = deviceProp.minor;
+      const std::string arch = std::string("-arch=sm_") + std::to_string(cc_major) + std::to_string(cc_minor);
+
+      const char* args[] = { arch.c_str(), "-rdc=true" };
+      const int num_args = sizeof(args) / sizeof(args[0]);
+
+      // Compile the CUDA C++ kernel to PTX
+      std::size_t ptx_size{};
+      nvrtcResult compile_result = nvrtcCompileProgram(prog, num_args, args);
+      nvrtcGetPTXSize(prog, &ptx_size);
+      std::unique_ptr<char[]> ptx{new char[ptx_size]};
+      nvrtcGetPTX(prog, ptx.get());
+      nvrtcDestroyProgram(&prog);
+
+At this point, we have two PTX inputs: PTX for the generated CUDA C++
+kernel and PTX for the Python-defined operator. We can combine them
+using nvJitLink:
+
+.. code-block:: c++
+
+      const char* link_options[] = { arch.c_str() };
+
+      // Link PTX comping from kernel and PTX coming from Python operator
+      nvJitLinkHandle handle;
+      nvJitLinkCreate(&handle, 1, link_options);
+      nvJitLinkAddData(handle, NVJITLINK_INPUT_PTX, ptx.get(), ptx_size, name);
+      nvJitLinkAddData(handle, NVJITLINK_INPUT_PTX, op_ptx, op_ptx_size, name);
+      nvJitLinkComplete(handle);
+
+      // Get resulting cubin
+      std::size_t cubin_size{};
+      nvJitLinkGetLinkedCubinSize(handle, &cubin_size);
+      std::unique_ptr<char[]> cubin{new char[cubin_size]};
+      nvJitLinkGetLinkedCubin(handle, cubin.get());
+      nvJitLinkDestroy(&handle);
+
+The result of linking is a cubin containing the generated kernel and
+the Python-defined operator. We can load that cubin as a CUDA
+library, retrieve the kernel from it, and launch it:
+
+.. code-block:: c++
+
+      // Load cubin
+      CUlibrary library;
+      cuLibraryLoadData(&library, cubin.get(), nullptr, nullptr, 0, nullptr, nullptr, 0);
+
+      // Get kernel pointer out of the library
+      CUkernel kernel;
+      cuLibraryGetKernel(&kernel, library, "kernel");
+
+      // Launch the kernel
+      void *kernel_args[] = { &value };
+      cuLaunchKernel((CUfunction)kernel, 1, 1, 1, 4, 1, 1, 0, 0, kernel_args, nullptr);
+
+Now the output of the Python program would be:
+
+.. code-block:: bash
+
+    thread 0 prints value 84
+    thread 1 prints value 84
+    thread 2 prints value 84
+    thread 3 prints value 84
+
+This works, but it is still not optimal from a performance
+perspective. If the operator were compiled as part of the same CUDA
+C++ translation unit as the kernel, the compiler could inline it
+directly. In the PTX-linked version above, however, the generated
+cubin still contains a call to ``op`` instead of the operator body
+itself.
+
+To address this, we switch to a different intermediate representation.
+Instead of PTX, we use `LTO-IR
+<https://developer.nvidia.com/blog/cuda-12-0-compiler-support-for-runtime-lto-using-nvjitlink-library/>`_.
+LTO-IR preserves enough information for link-time optimization, which
+allows the operator to be inlined into the generated kernel.
+
+On the Python side, switching from PTX to LTO-IR requires only a small
+change:
+
+.. code-block:: python
+
+    ltoir, _ = numba.cuda.compile(op, sig=numba.int32(numba.int32), output="ltoir")
+
+On the C++ side, we make the same switch from PTX to LTO-IR:
+
+.. code-block:: c++
+
+    const char* args[] = { arch.c_str(), "-rdc=true", "-dlto" };
+    const int num_args = sizeof(args) / sizeof(args[0]);
+
+    nvrtcResult compile_result = nvrtcCompileProgram(prog, num_args, args);
+
+    std::size_t ltoir_size{};
+    nvrtcGetLTOIRSize(prog, &ltoir_size);
+    std::unique_ptr<char[]> ltoir{new char[ltoir_size]};
+    nvrtcGetLTOIR(prog, ltoir.get());
+    nvrtcDestroyProgram(&prog);
+
+    const char* link_options[] = { "-lto", arch.c_str() };
+
+    nvJitLinkHandle handle;
+    nvJitLinkCreate(&handle, 2, link_options);
+    nvJitLinkAddData(handle, NVJITLINK_INPUT_LTOIR, ltoir.get(), ltoir_size, name);
+    nvJitLinkAddData(handle, NVJITLINK_INPUT_LTOIR, op_ltoir, op_ltoir_size, name);
+
+If you inspect the generated cubin now, you will no longer see a call
+to ``op``. Instead, the operator has been inlined into the kernel,
+which improves performance. That is the key benefit of switching from
+PTX to LTO-IR.
+
+At this point, we have a working prototype that can pass
+Python-defined operators into CUDA C++ kernels without sacrificing
+performance. The next problem is user-defined data types. So far, the
+examples have used built-in scalar types, but a practical API also
+needs to support types whose layout is only known on the Python side.
+
+Fortunately, the kernel source is already being assembled as a string at
+runtime. That means we can also generate the type information needed by
+the CUDA C++ side.
+
+As a concrete example, suppose we want to pass a ``numba.complex128``
+value into the kernel. The C++ side does not see the original Python
+type definition, but that is not an issue. It only needs a storage
+type with matching size and alignment, and can type-erase everything
+else.
+
+.. code-block:: c++
+
+        extern "C" void launcher(void *value_ptr, int type_size, int type_alignment,
+                                 const char* op_ltoir, int op_ltoir_size)
+        {
+            std::string storage_t = "struct __align__(" + std::to_string(type_alignment) + ")"
+                                    + "storage_t { char data[" + std::to_string(type_size) + "]; };";
+
+            std::string kernel_source = storage_t + R"XXX(
+                extern "C" __device__ int op(char *state);
+
+                extern "C" __global__ void kernel(storage_t value) {
+                    printf("thread %d prints value %d\n", threadIdx.x, op(value.data));
+                }
+            )XXX";
+
+            // ...
+            void *kernel_args[] = { value_ptr };
+            cuLaunchKernel((CUfunction)kernel, 1, 1, 1, 4, 1, 1, 0, 0, kernel_args, nullptr);
+
+In this version, the operator takes a type-erased pointer. On the
+Python side, we therefore pass a pointer to the ``numba.complex128``
+value, together with the size and alignment needed to construct a
+matching storage type on the C++ side:
+
+.. code-block:: python
+
+        import ctypes
+        import numba
+        import numba.cuda
+        import numpy as np
+
+        def op(value):
+            return numba.int32(value[0].real + value[0].imag)
+
+        value_type = numba.complex128
+        context = numba.cuda.descriptor.cuda_target.target_context
+        size = context.get_value_type(value_type).get_abi_size(context.target_data)
+        alignment = context.get_value_type(value_type).get_abi_alignment(context.target_data)
+        ltoir, _ = numba.cuda.compile(op, sig=numba.int32(numba.types.CPointer(value_type)), output='ltoir')
+
+        value = np.array([1 + 2j], dtype=np.complex128)
+        type_erased_value_ptr = value.ctypes.data_as(ctypes.c_void_p)
+
+        bindings = ctypes.CDLL('./build/libkernel.so')
+        bindings.launcher.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.c_int, ctypes.c_char_p, ctypes.c_int]
+        bindings.launcher(type_erased_value_ptr, size, alignment, ltoir, len(ltoir))
+
+In this example, we obtain the size and alignment of
+``numba.complex128`` from Numba's type system. The remaining detail is
+how to pass the value to ``cuLaunchKernel``. Kernel arguments are
+described to ``cuLaunchKernel`` as pointers to host memory from which
+the launch parameters are copied. In Python, that host-memory pointer
+can be obtained in a few ways, for example with ``ctypes.byref`` or by
+placing the value in a ``numpy.array`` and retrieving the array's
+address with ``value.ctypes.data_as(ctypes.c_void_p)``.
+
+One more ingredient is needed to get closer to the full
+``cuda.compute`` implementation. The kernels in the CUDA C++ Core
+Compute Libraries are templates, so our generated kernel must be a
+template as well.
+
+.. code-block:: c++
+
+    std::string kernel_source = storage_t + R"XXX(
+        extern "C" __device__ int op(char *state);
+
+        template <class T>
+        __global__ void kernel(T value) {
+            printf("thread %d prints value %d\n", threadIdx.x, op(value.data));
+        }
+    )XXX";
+
+Defining the kernel as a template is still not enough. We also need to
+instantiate that template for the generated storage type. NVRTC
+provides the necessary API for that:
+
+.. code-block:: c++
+
+    nvrtcProgram prog;
+    const char *name = "test_kernel";
+    nvrtcCreateProgram(&prog, kernel_source.c_str(), name, 0, nullptr, nullptr);
+
+    // Get the name of the instantiated kernel
+    std::string kernel_name = "kernel<storage_t>";
+
+    // Instantiate kernel template
+    nvrtcAddNameExpression(prog, kernel_name.c_str());
+    // ...
+
+    // Get lowered name of the kernel
+    const char* kernel_lowered_name; // _Z6kernelI9storage_tEvT_
+    nvrtcGetLoweredName(prog, kernel_name.c_str(), &kernel_lowered_name);
+    // ...
+
+    // Use it to get kernel pointer
+    cuLibraryGetKernel(&kernel, library, kernel_lowered_name);
+
+With these pieces in place, we can connect the simplified prototype back
+to ``cuda.compute``.
+
+At a high level, the ``cuda.compute`` API follows the same overall
+structure, but packages it into three stages. Using parallel reduction
+as an example:
+
+#. In the first stage, ``cuda.compute.make_reduce_into(...)`` constructs
+   a reusable reduction object:
+
+   ``reducer = cuda.compute.make_reduce_into(d_in=d_in, d_out=d_out, op=op, h_init=h_init)``
+
+   Here ``op`` is a Python function that must be made available to the
+   CUDA kernel. As in the simplified prototype above, this stage
+   compiles ``op`` to LTO-IR, generates the corresponding CUDA C++
+   source, instantiates the necessary kernels, and compiles them with
+   NVRTC. The resulting build state is stored inside the returned
+   reduction object. At this stage, the concrete runtime values of the
+   provided arrays do not matter yet; later calls may use different
+   pointers or sizes, as long as the interface remains compatible.
+
+#. In the second stage, that reduction object is used to query the
+   amount of temporary storage required by the algorithm:
+
+   ``temp_storage_size = reducer(temp_storage=None, d_in=d_input, d_out=d_output, num_items=num_items, op=op, h_init=h_init)``
+
+   This returns the size of the temporary storage buffer, which must be
+   allocated in device-accessible memory. No kernels are launched at
+   this stage.
+
+#. In the third stage, the algorithm is executed using the allocated
+   temporary storage:
+
+   ``reducer(temp_storage=temp_storage, d_in=d_input, d_out=d_output, num_items=num_items, op=op, h_init=h_init)``
+
+   At this point, the kernels stored in the reduction object are
+   launched and the reduction is performed.
+
+Build results and device state
+------------------------------
+
+An algorithm is built in one of two ways. A **default build**
+(``compute_capability=None``, the common path) targets the current CUDA
+device — it queries that device's compute capability, then compiles and loads
+for it in one step. An **explicit ahead-of-time (AOT) build** (a
+``compute_capability=`` argument naming one or more compute capabilities) names
+its targets directly, compiles for each without loading, and needs no GPU, so it
+can run on a build machine with no device (see
+:ref:`cuda.compute.ahead_of_time_compilation`).
+
+Either way, building produces a native build result — a Cython build-result
+object wrapping the corresponding C runtime struct — that carries two kinds of
+state with different device affinity:
+
+* The **compiled payload** (the compiled device code and its launch policy)
+  depends only on the target compute capability. It is device-independent: the
+  same payload is valid on any device of that compute capability.
+* **Loaded state** is created when the build result is loaded for execution —
+  the registered ``CUlibrary`` and the kernel handles resolved from it. It
+  belongs to the device (and context) it was loaded on. CUB launch paths
+  resolve a ``CUkernel`` to the current-context ``CUfunction`` and may get or
+  set kernel attributes on it, and CUDA kernel-attribute behavior is
+  device-specific.
+
+Consequently, a *loaded* build result cannot be shared across two devices even
+when they have the same compute capability: its handles are device-specific.
+The compiled payload can be reused, but each device needs its own loaded result
+— built directly, or reconstructed from the shared payload. This is a property
+of CUDA and the build struct itself, independent of caching or free-threaded
+Python. The next section describes how ``cuda.compute`` caches build results to
+reuse the payload while giving each device its own loaded state.
+
+Caching and free-threaded Python
+--------------------------------
+
+The user-facing cache behavior is described in :ref:`cuda.compute.caching`. This
+section describes the implementation contracts that keep that behavior correct
+for free-threaded Python and multi-GPU use.
+
+Two cache layers
+++++++++++++++++
+
+Internally, ``cuda.compute`` separates two kinds of cached state:
+
+* **Wrapper objects** are the Python objects returned by ``make_*`` APIs, such as
+  ``make_reduce_into``. They own per-call descriptor state and are cached per
+  Python thread by ``cache_with_registered_key_functions`` in
+  ``cuda/compute/_caching.py``. Keeping wrapper caches thread-local avoids
+  sharing mutable wrapper state across concurrent calls from free-threaded
+  Python.
+* **Per-cc build results** (``_PerCCBuildResults``) hold one *canonical* Cython
+  build result per target compute capability — the single authoritative result
+  for that cc, carrying the device-independent compiled payload described in
+  `Build results and device state`_. They are cached by ``cache_build_results``
+  and may be shared by wrapper objects in different Python threads — and, for
+  default builds, across same-cc devices (except on the v2 HostJIT backend
+  today; see `Device keying`_). Each device's loaded result is tracked
+  separately within the entry, so sharing an entry never shares device-specific
+  state.
+
+The normal cache-hit path is intentionally cheap. A wrapper-cache hit is
+thread-local and does not consult the process-wide build-result cache. When a
+wrapper is constructed, a completed build-result hit requires one process-wide
+dictionary lookup and does not take an explicit cache lock. The two build
+kinds then diverge because they differ in whether the target device is known
+when the wrapper is constructed. A default build already knows its device — the
+wrapper cache queried it to build and keys the wrapper to it — so the wrapper
+resolves that device's loaded result once at construction and stores a direct
+reference; executing it then needs no current-device query and no shared-cache
+lookup. An explicit AOT or deserialized wrapper has no such binding — an AOT
+build targets compute capabilities with no GPU queried, and a deserialized
+wrapper is reconstructed without a device binding — so its device is known only
+at call time. Each call resolves the per-device loaded result from a dictionary
+inside the per-cc build results, where a completed lookup also takes no explicit
+lock.
+
+Design requirements
++++++++++++++++++++
+
+The free-threading design is constrained by the following requirements:
+
+* Importing ``cuda.compute`` in a free-threaded CPython interpreter must not
+  re-enable the GIL.
+* Free-threading support should not add global locking or shared-state
+  contention to the normal single-threaded execution path. Wrapper cache hits
+  should be thread-local, and normal algorithm execution should not take a
+  global cache lock.
+* Mutable wrapper state must not be shared across threads.
+* Expensive native build results should still be shared across threads when they
+  are safe to share.
+* Same-key concurrent cold builds should build once; waiters should receive the
+  same result or observe the same exception.
+
+The current free-threading support boundary is the ``minimal-cu12`` and
+``minimal-cu13`` extras. These extras omit Numba and Numba CUDA. Consequently,
+free-threaded support currently covers built-in ``OpKind`` operations and
+externally compiled ``RawOp`` operations, but not Python-callable operators.
+The full ``cu12`` and ``cu13`` extras remain
+outside the support claim until the Numba CUDA dependency is replaced by a
+free-threading-compatible implementation.
+
+CI runs ``test_free_threading_stress.py`` directly from the minimal test job.
+The v1 backend is covered across the supported CUDA 12 and 13 lanes, and a
+separate CTK 13.X minimal job runs the same suite against the v2 HostJIT
+backend. Pytest runs each suite in one process while the stress tests create
+and synchronize their own worker threads.
+
+Build and validation requirements
++++++++++++++++++++++++++++++++++
+
+The Cython extension that backs ``cuda.compute`` must opt in to free-threaded
+execution:
+
+.. code-block:: cython
+
+   # cython: freethreading_compatible=True
+
+Without this marker, importing the extension in a free-threaded CPython process
+can cause CPython to re-enable the GIL. The generated extension should advertise
+``Py_MOD_GIL_NOT_USED`` and importing ``cuda.compute`` should leave
+``sys._is_gil_enabled()`` false.
+
+The free-threaded wheel must also keep its free-threaded ABI tag after repair and
+merge steps. For CPython 3.14, the expected wheel tag contains
+``cp314-cp314t`` rather than the regular ``cp314-cp314`` tag. The acceptance
+criteria for a free-threaded build are:
+
+* the wheel has the expected ``cp314-cp314t`` ABI tag;
+* importing ``cuda.compute`` does not re-enable the GIL;
+* the free-threading stress suite passes without forcing ``PYTHON_GIL=0`` or
+  ``-X gil=0``.
+
+
+Device keying
++++++++++++++
+
+User-facing multi-GPU behavior and requirements are described in
+:ref:`cuda.compute.multi_gpu`; this section covers the keying mechanism.
+
+For the default build path, the wrapper cache includes
+the current CUDA runtime device ordinal and compute capability in its key:
+wrapper objects hold device-bound state, so each device (and thread) receives
+its own wrapper. The shared build-result cache is keyed by compute capability
+alone — the compiled payload depends only on the cc — so one shared entry
+serves every same-cc device ordinal.
+
+Per-device loaded state lives inside the shared entry. The device that built
+the entry loads the canonical result in place; each additional same-cc device
+loads its own clone of the compiled payload through serialization (serialize,
+deserialize without loading, then load on the new device) instead of running
+a full native compilation, and the clone re-validates the payload against the
+current device. When the backend cannot serialize build results (the v2 HostJIT
+backend today), sharing is not possible, so default builds are keyed per device
+ordinal instead and each device builds its own entry.
+
+Explicit AOT builds cannot include a device ordinal in their compilation key —
+they build with no GPU queried — so their canonical results are shared
+process-wide by specialization and target compute capabilities. Unlike a default
+build, an AOT build compiles without loading, so no device owns the canonical
+result until first execution: the first device to run claims and loads it, and
+other same-cc devices load their own clone, exactly as above.
+
+The first implementation intentionally keys shared build results by CUDA runtime
+device ordinal rather than by CUDA context handle. User-managed CUDA driver
+contexts are not a target use case for ``cuda.compute``. CUDA runtime,
+``cuda.core``, CuPy, and PyTorch-style applications are expected to use the
+primary-context model, and language frontends generally prefer that model.
+
+Concurrent build coordination
++++++++++++++++++++++++++++++
+
+When several threads miss the same cache key at once, only one should run the
+expensive build and the rest should wait for its result. A shared helper
+provides this coordination, a pattern called *single-flight*. The cache
+dictionary stores either a completed value or a temporary ``_InFlightBuild``
+entry. On a miss, each caller creates a candidate in-flight
+entry, and ``dict.setdefault`` elects one caller to run the builder. Other
+callers receive the winning entry and wait on its ``threading.Event``. If the
+operation succeeds, the in-flight entry is replaced by the completed result and
+all waiting threads receive that same object. If it fails, the exception is
+propagated to the waiting threads and the failed entry is removed so that a
+later call can retry. Completed-result hits do not allocate an in-flight entry
+or take an explicit cache lock.
+
+The same helper coordinates two kinds of misses: a compilation miss in the
+process-wide build cache, where ``cache_build_results`` runs the native build
+once per specialization, and a per-device load miss inside a per-cc build
+result, where ``resolve`` loads (or clones and loads) the result once per
+device.
+
+When adding a new algorithm, the factory that returns the reusable wrapper object
+should use ``cache_with_registered_key_functions``. The wrapper constructor
+should pass the expensive native build operation to ``cache_build_results``,
+which returns two values: the shared build results, and the loaded result bound
+to the constructing device (``None`` for an explicit AOT build, which has no
+constructing device). Store both; ``__call__`` passes them to
+``resolve_build_result`` (see any algorithm class for the pattern).
+Do not perform an expensive native build before entering
+``cache_build_results``; otherwise same-key cold factory calls can duplicate the
+build and bypass single-flight coordination.
+
+The specialization key must include every argument that can affect generated
+code, type layout, policy selection, or native build state. It should not include
+runtime-only values such as array pointers, array contents, item counts, streams,
+or temporary-storage pointers unless those values change the compiled interface.
+
+User-object and descriptor contracts
+++++++++++++++++++++++++++++++++++++
+
+Wrapper objects returned by ``make_*`` APIs are not safe for concurrent calls
+from multiple threads. If two threads need the same algorithm specialization,
+each thread should call the
+factory and receive its own wrapper object, or the caller must externally
+serialize access to a shared wrapper. The wrapper updates its Cython
+``Iterator``, ``Op``, ``Value``, and algorithm-specific descriptors before each
+native call, so concurrent calls through the same wrapper could overwrite the
+descriptor state another thread is about to use.
+
+The same contract applies to wrappers reconstructed by ``deserialize()`` —
+they are the same classes with the same mutable descriptors. Unlike the
+factories, ``deserialize()`` does not hand each calling thread its own object
+through the per-thread wrapper cache: every call constructs a fresh, uncached
+wrapper. The natural deserialize-once-and-share pattern therefore reintroduces
+exactly the descriptor races the per-thread factory cache prevents. Threads
+that need a deserialized algorithm concurrently should each deserialize the
+blob themselves; that performs no recompilation, at the cost of an independent
+native load per object.
+
+Read-only iterator and operator objects may be shared across threads. The
+iterator base class uses a per-iterator lock for first-time lazy construction of
+advance, input-dereference, and output-dereference ``Op`` objects; cached access
+after that remains lock-free. This lock does not make arbitrary mutation safe:
+concurrent mutation of iterator state, operator state, captured state, or child
+iterators remains unsupported unless the caller synchronizes externally.
+
+Mutable execution state belongs to one thread at a time unless the caller
+provides synchronization. This includes output arrays, temporary-storage buffers,
+streams, ``DoubleBuffer`` instances, and other objects whose state changes as
+part of a launch.
+
+Backend-specific notes
+++++++++++++++++++++++
+
+The v1 NVRTC/nvJitLink backend and the v2 HostJIT backend have different
+free-threading risk surfaces and must be audited independently. v1 stresses
+NVRTC, nvJitLink, CUDA library loading, and CUB host dispatch. v2 adds HostJIT
+compiler state, LLVM/Clang initialization, persistent PCH paths, generated
+source/cubin artifacts, and dynamic loader lifetime.
+
+Transform has one additional v1 native-cache rule. Each transform build result
+owns a native cache of launch configurations (``async_config`` /
+``prefetch_config``) in ``c/parallel/src/transform.cu``. Because one build
+result is shared by every thread using the same specialization, and the Cython
+bindings release the GIL around the native call, each configuration is filled
+exactly once through ``std::call_once``; later calls on any thread only pay the
+``once_flag`` fast-path check. This holds on every interpreter build — regular
+GIL builds also execute the native call concurrently once the GIL is released,
+so the cache must be thread-safe unconditionally.
+
+The v2 backend addresses the same transform concern differently, and only on
+Windows. HostJIT compiles generated code with ``-fno-threadsafe-statics``
+because the Windows CRT guard support that thread-safe function-local statics
+require is unavailable. Generated CUB code still initializes function-local
+statics lazily — transform's launch configuration among them — so a
+per-build-result ``first_call_gate``
+(``c/parallel.v2/src/util/first_call_gate.h``) serializes the first successful
+call into each generated function; after it completes, an atomic fast-path check
+lets later concurrent calls proceed without locking. Empty calls bypass the gate
+because they return before CUB initializes the static. This covers transform and
+binary search; other platforms keep thread-safe statics and need no gate.
+
+Clearing caches
++++++++++++++++
+
+``clear_all_caches()`` is process-local. It clears all known per-thread wrapper
+caches through a weak registry of live thread cache containers, and it clears the
+shared build-result cache. Separate Python processes build and cache
+independently.
+
+Calling ``clear_all_caches()`` concurrently with active factory calls or
+algorithm execution is not supported unless the caller synchronizes externally.
+
+
+Source map
+----------
+
+For readers who want to connect this overview back to the source tree:
+
+* The Python-facing API, operator compilation, and the logic for
+  constructing and invoking reusable algorithm objects live under
+  ``python/cuda_cccl/cuda/compute/``.
+* The lower-level C/C++ runtime compilation and kernel-building
+  machinery lives under ``c/parallel/`` (and ``c/parallel.v2/`` for the v2
+  HostJIT backend).
+* User-facing examples for ``cuda.compute`` live under
+  ``python/cuda_cccl/tests/compute/examples/``.
diff --git a/cccl_upstream/docs/python/compute/index.rst b/cccl_upstream/docs/python/compute/index.rst
new file mode 100644
index 00000000..24b272ab
--- /dev/null
+++ b/cccl_upstream/docs/python/compute/index.rst
@@ -0,0 +1,555 @@
+.. _cccl-python-compute:
+
+``cuda.compute``: Parallel Computing Primitives
+===============================================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 2
+
+   Overview <self>
+   developer_overview
+
+The ``cuda.compute`` library provides composable primitives for building custom
+parallel algorithms on the GPU—without writing CUDA kernels directly.
+
+Algorithms
+----------
+
+Algorithms are the core of ``cuda.compute``. They operate on arrays or
+:ref:`iterators <cuda.compute.iterators>` and can be composed to build specialized
+GPU operations—reductions, scans, sorts, transforms, and more.
+
+Typical usage of an algorithm looks like this:
+
+.. code-block:: python
+
+   cuda.compute.reduce_into(
+      d_in=...,       # input array or iterator
+      d_out=...,      # output array or iterator
+      op=...,         # binary operator (built-in or user-defined)
+      num_items=...,  # number of input elements
+      h_init=...,     # initial value for the reduction
+   )
+
+API conventions
++++++++++++++++
+
+* **Keyword-only parameters** — All algorithm parameters are **keyword-only**.
+  They must always be passed by name, not by position:
+
+  .. code-block:: python
+
+     # correct
+     cuda.compute.reduce_into(d_in=d_input, d_out=d_output, num_items=n, op=OpKind.PLUS, h_init=h_init)
+
+     # incorrect — positional arguments are not accepted
+     cuda.compute.reduce_into(d_input, d_output, n, OpKind.PLUS, h_init)  # TypeError
+
+* **Naming** — The ``d_`` prefix denotes *device* memory (e.g., CuPy arrays, PyTorch tensors);
+  ``h_`` denotes *host* memory (NumPy arrays). Some scalar values must be passed as
+  host arrays.
+
+* **Output semantics** — Algorithms write results into a user-provided array or iterator
+  rather than returning them. This keeps memory ownership explicit and lifetimes under
+  your control.
+
+* **Operators** — Many algorithms accept an ``op`` parameter. This can be a built-in
+  :class:`OpKind <cuda.compute.op.OpKind>` value or a
+  :ref:`user-defined operator <cuda.compute.user_defined_operators>`.
+  When possible, prefer built-in operators (e.g., ``OpKind.PLUS``) over the equivalent
+  user-defined operation (e.g., ``lambda a, b: a + b``) for better performance.
+
+* **Iterators** — Inputs and outputs can be :ref:`iterators <cuda.compute.iterators>`
+  instead of arrays, enabling lazy evaluation and operation fusion.
+
+Full Example
+++++++++++++
+
+The following example uses :func:`reduce_into <cuda.compute.algorithms.reduce_into>`
+to compute the sum of a sequence of integers:
+
+.. literalinclude:: ../../../python/cuda_cccl/tests/compute/examples/reduction/sum_reduction.py
+   :language: python
+   :start-after: # example-begin
+   :caption: Sum reduction example.
+
+Object-based API (expert mode)
+++++++++++++++++++++++++++++++
+
+Many algorithms allocate temporary device memory for intermediate results. For finer
+control over allocation—or to reuse buffers across calls—use the object-based API.
+For example, :func:`make_reduce_into <cuda.compute.algorithms.make_reduce_into>`
+returns a reusable reduction object that lets you manage memory explicitly.
+
+.. code-block:: python
+   :caption: Controlling temporary memory.
+
+   # create a reducer object:
+   reducer = cuda.compute.make_reduce_into(d_in=d_in, d_out=d_out, op=op, h_init=h_init)
+   # get the temporary storage size by passing None for the temp_storage argument:
+   temp_storage_bytes = reducer(temp_storage=None, d_in=d_in, d_out=d_out, num_items=num_items, op=op, h_init=h_init)
+   # allocate the temporary storage as any array-like object
+   # (e.g., CuPy array, Torch tensor):
+   temp_storage = cp.empty(temp_storage_bytes, dtype=np.uint8)
+   # perform the reduction:
+   reducer(temp_storage=temp_storage, d_in=d_in, d_out=d_out, num_items=num_items, op=op, h_init=h_init)
+
+The object-based API splits the algorithm invocation into three phases,
+
+1. Constructing an algorithm object
+2. Determining the amount of temporary memory needed by the computation
+3. Performing the computation
+
+It is important that the type of arguments passed during construction (step 1) match
+those passed during invocation (step 2 and 3). Otherwise you may see unexpected errors
+or silent bugs.
+
+- Data types of arrays/iterators must match. If you pass an array of `int32` data type
+  as the `d_in=` argument during construction of a reducer object, you must pass
+  an array of dtype `int32` when invoking it. The array can be of a different size.
+
+- Bytecode instructions of functions must match. If you pass a function/lambda for
+  the operator during construction, you must pass a function with the same bytecode
+  instructions during invocation. This means you _can_ pass a different function
+  referencing different global/closures, but the operations within the functions
+  must be the same.
+
+
+.. _cuda.compute.user_defined_operators:
+
+User-Defined Operators
+----------------------
+
+A powerful feature is the ability to use algorithms with user-defined operators.
+For example, to compute the sum of only the even values in a sequence,
+we can use :func:`reduce_into <cuda.compute.algorithms.reduce_into>` with a custom binary operation:
+
+.. literalinclude:: ../../../python/cuda_cccl/tests/compute/examples/reduction/sum_custom_reduction.py
+   :language: python
+   :start-after: # example-begin
+   :caption: Reduction with a custom binary operation.
+
+Features and Restrictions
++++++++++++++++++++++++++
+
+User-defined operations are just-in-time (JIT) compiled into device code using
+`Numba CUDA <https://nvidia.github.io/numba-cuda/>`_, so they inherit many
+of the same features and restrictions as Numba CUDA functions:
+
+* `Python features <https://nvidia.github.io/numba-cuda/user/cudapysupported.html>`_
+  and `atomic operations <https://nvidia.github.io/numba-cuda/user/intrinsics.html>`_
+  supported by Numba CUDA are also supported within user-defined operators.
+* Nested functions must be decorated with ``@numba.cuda.jit``.
+* Variables captured in closures or globals follow
+  `Numba CUDA semantics <https://nvidia.github.io/numba-cuda/user/globals.html>`_:
+  scalars and host arrays are captured by value (as constants),
+  while device arrays are captured by reference.
+
+
+.. _cuda.compute.iterators:
+
+Iterators
+---------
+
+Iterators represent sequences whose elements are computed **on the fly**. They can
+be used in place of arrays in most algorithms, enabling lazy evaluation, operation
+fusion, and custom data access patterns.
+
+A :func:`CountingIterator <cuda.compute.iterators.CountingIterator>`, for example,
+represents an integer sequence starting from a given value:
+
+.. code-block:: python
+
+   it = CountingIterator(np.int32(1))  # represents [1, 2, 3, 4, ...]
+
+To compute the sum of the first 100 integers, we can pass a
+:func:`CountingIterator <cuda.compute.iterators.CountingIterator>` directly to
+:func:`reduce_into <cuda.compute.algorithms.reduce_into>`. No memory is allocated
+to store the input sequence—the values are generated as needed.
+
+.. literalinclude:: ../../../python/cuda_cccl/tests/compute/examples/iterator/counting_iterator_basic.py
+   :language: python
+   :start-after: # example-begin
+   :caption: Counting iterator example.
+
+Iterators can also be used to *fuse* operations. In the example below, a
+:func:`TransformIterator <cuda.compute.iterators.TransformIterator>` lazily applies
+the square operation to each element of the input sequence. The resulting iterator
+is then passed to :func:`reduce_into <cuda.compute.algorithms.reduce_into>` to compute
+the sum of squares.
+
+Because the square is evaluated on demand during the reduction, there is no need
+to create or store an intermediate array of squared values. The transform and the
+reduction are fused into a single pass over the data.
+
+.. literalinclude:: ../../../python/cuda_cccl/tests/compute/examples/iterator/transform_iterator_basic.py
+   :language: python
+   :start-after: # example-begin
+   :caption: Transform iterator example.
+
+Some iterators can also be used as the output of an algorithm. In the example below,
+a :func:`TransformOutputIterator <cuda.compute.iterators.TransformOutputIterator>`
+applies the square-root operation to the result of a reduction before writing
+it into the underlying array.
+
+.. literalinclude:: ../../../python/cuda_cccl/tests/compute/examples/iterator/transform_output_iterator.py
+   :language: python
+   :start-after: # example-begin
+   :caption: Transform output iterator example.
+
+As another example, :func:`ZipIterator <cuda.compute.iterators.ZipIterator>` combines multiple
+arrays or iterators into a single logical sequence. In the example below, we combine
+a counting iterator and an array, creating an iterator that yields ``(index, value)``
+pairs. This combined iterator is then used as the input to
+:func:`reduce_into <cuda.compute.algorithms.reduce_into>` to compute the index of
+the maximum value in the array.
+
+.. literalinclude:: ../../../python/cuda_cccl/tests/compute/examples/iterator/zip_iterator_counting.py
+   :language: python
+   :start-after: # example-begin
+   :caption: Argmax using a zip iterator.
+
+These examples illustrate a few of the patterns enabled by iterators. See the
+:ref:`API reference <cuda_compute-module>` for the full set of available iterators.
+
+.. _cuda.compute.custom_types:
+
+Struct Types
+------------
+
+The :func:`gpu_struct <cuda.compute.struct.gpu_struct>` decorator defines
+GPU-compatible struct types. These are useful when you have data laid out
+as an "array of structures", similar to `NumPy structured arrays <https://numpy.org/doc/stable/user/basics.rec.html>`_.
+
+.. literalinclude:: ../../../python/cuda_cccl/tests/compute/examples/struct/struct_reduction.py
+   :language: python
+   :start-after: # example-begin
+   :caption: Custom struct type in a reduction.
+
+Array of Structures vs Structure of Arrays
+++++++++++++++++++++++++++++++++++++++++++
+
+When working with structured data, there are two common memory layouts:
+
+* **Array of Structures (AoS)** — each element is a complete struct, stored
+  contiguously. For example, an array of ``Point`` structs where each point's
+  ``x`` and ``y`` are adjacent in memory.
+
+* **Structure of Arrays (SoA)** — each field is stored in its own array.
+  For example, separate ``x_coords`` and ``y_coords`` arrays.
+
+``cuda.compute`` supports both layouts:
+
+* **``gpu_struct``** — defines a true AoS type with named fields
+* **``ZipIterator``** — combines separate arrays into tuples on the fly, letting
+  you work with SoA data as if it were AoS
+
+.. _cuda.compute.caching:
+
+Caching
+-------
+
+Algorithms in ``cuda.compute`` are compiled to GPU code at runtime. To
+avoid recompiling on every call, build results are cached in memory.
+When you invoke an algorithm with the same configuration—same dtypes,
+iterator kinds, operator, compute capability, and current device—the
+cached build is reused. On systems with multiple GPUs, GPUs with the same
+compute capability may share one compiled build, while each GPU keeps its
+own loaded state. Compiled build results may be reused by
+multiple threads in the same process on any interpreter build;
+free-threaded Python additionally runs such threads in parallel.
+
+What determines the cache key
++++++++++++++++++++++++++++++
+
+Each algorithm computes a cache key from:
+
+* **Array dtypes** — the data types of input and output arrays
+* **Iterator kinds** — for iterator inputs/outputs, a descriptor of the iterator type
+* **Operator identity** — for user-defined functions, the function's bytecode,
+  constants, and closure contents (see below)
+* **Compute capability** — the GPU architecture of the current device
+* **Current device** — determines the algorithm object and its loaded state;
+  the compiled code itself is shared across devices with the same compute
+  capability (see Multi-GPU behavior below). Not used when an explicit
+  ``compute_capability=`` is given, which keys on the requested compute
+  capabilities instead (see :ref:`cuda.compute.ahead_of_time_compilation`)
+* **Algorithm-specific parameters** — such as initial value dtype or determinism mode
+
+Note that array *contents* or *pointers* are not part of the cache key—only
+the array's dtype. This means you can reuse a cached algorithm across different
+arrays of the same type.
+
+.. _cuda.compute.multi_gpu:
+
+Multi-GPU behavior
+++++++++++++++++++
+
+Loaded builds are device-specific. With the default current-device build path,
+``cuda.compute`` compiles once per compute capability and reuses the compiled
+payload on other GPUs with the same compute capability; each GPU still receives
+its own loaded native state. An explicit ahead-of-time build behaves the same
+way across same-compute-capability GPUs. Set the intended current CUDA device
+before invoking an algorithm, and pass arrays — and a stream — that belong to
+that device; currently, the stream selects a queue on the current device and
+does not select the device itself.
+
+.. _cuda.compute.free_threading:
+
+Free-threaded Python
+++++++++++++++++++++
+
+.. important::
+
+   Free-threaded Python support is currently validated with the
+   ``minimal-cu12`` and ``minimal-cu13`` extras, which do not install Numba or
+   Numba CUDA:
+
+   .. code-block:: bash
+
+      pip install cuda-cccl[minimal-cu13]  # or minimal-cu12
+
+   The full ``cu12`` and ``cu13`` extras and Python-callable operators that
+   require Numba CUDA are not currently
+   supported in free-threaded Python. Use built-in
+   :class:`OpKind <cuda.compute.op.OpKind>` operations or externally compiled
+   :class:`RawOp <cuda.compute.op.RawOp>` operations with the minimal
+   installation.
+
+Independent calls from multiple Python threads reuse compiled build results
+within the same process on any interpreter build. A free-threaded interpreter
+additionally runs those calls in parallel instead of interleaving them under
+the GIL.
+
+The cache is local to the current Python process. Separate Python processes build
+and cache independently, even if they use the same GPU and algorithm
+configuration.
+
+This does not make user-provided memory or CUDA work automatically safe to share.
+Users are still responsible for avoiding data races, such as two threads writing
+to the same output array at the same time. Read-only iterator and operator
+objects may be shared across threads, but concurrent mutation of those objects,
+captured state, or underlying arrays requires external synchronization. For
+concurrent use, prefer the direct
+algorithm APIs, such as
+:func:`reduce_into <cuda.compute.algorithms.reduce_into>`, or create a separate
+reusable algorithm object in each thread (for example, the object returned by
+:func:`make_reduce_into <cuda.compute.algorithms.make_reduce_into>`). If multiple
+threads share one of these objects, serialize access to that object.
+
+The examples below additionally use CuPy for device arrays. CuPy is not part
+of the ``minimal`` extras, so install it separately (``pip install
+cupy-cuda13x`` or ``cupy-cuda12x``; free-threaded Linux wheels are available
+starting with CuPy 14.1).
+
+.. literalinclude:: ../../../python/cuda_cccl/tests/compute/examples/free_threading/direct_api.py
+   :language: python
+   :start-after: # example-begin
+   :caption: Concurrent reductions through the direct API from multiple threads.
+
+.. literalinclude:: ../../../python/cuda_cccl/tests/compute/examples/free_threading/object_api.py
+   :language: python
+   :start-after: # example-begin
+   :caption: Per-thread algorithm objects created with a ``make_*`` factory.
+
+How user-defined functions are cached
++++++++++++++++++++++++++++++++++++++
+
+User-defined operators and predicates are hashed based on their bytecode, constants,
+and closure contents. Two functions with identical bytecode and closures produce
+the same cache key, even if defined at different source locations.
+
+Closure contents are recursively hashed:
+
+* **Scalars and host arrays** — hashed by value
+* **Device arrays** — hashed by pointer, shape, and dtype (not contents)
+* **Nested functions** — hashed by their own bytecode and closures
+
+Because device arrays captured in closures are hashed by pointer, changing the
+array's contents does not invalidate the cache—only reassigning the variable to
+a different array does.
+
+Memory considerations
++++++++++++++++++++++
+
+The cache persists for the lifetime of the process and grows with the number of
+unique algorithm configurations. In long-running applications or exploratory
+notebooks, this can accumulate significant memory.
+
+To clear all caches and free memory:
+
+.. code-block:: python
+
+   import cuda.compute
+   cuda.compute.clear_all_caches()
+
+This forces recompilation on the next algorithm invocation—useful for benchmarking
+compilation time or reclaiming memory.
+
+In multi-threaded programs, make sure no other thread is building or running an
+algorithm while you call it: ``clear_all_caches()`` does not synchronize with
+concurrent use, and a build that is already in progress may finish afterwards
+and place its result back into the cache.
+
+.. _cuda.compute.serialization:
+
+Serialization
+-------------
+
+:ref:`Caching <cuda.compute.caching>` reuses build results *within* a single
+process. Serialization goes one step further: it lets you persist a built
+algorithm to a blob of bytes and reconstruct it later—in another process, or on
+another machine—**without recompiling**.
+
+Use :func:`serialize <cuda.compute.algorithms.serialize>` on any object returned by a
+``make_*`` factory to obtain a ``bytes`` blob, and
+:func:`deserialize <cuda.compute.algorithms.deserialize>` to reconstruct it. The blob stores
+the *compiled* build result, so :func:`deserialize <cuda.compute.algorithms.deserialize>`
+performs no JIT compilation—it neither invokes Numba nor recompiles device code.
+In practice you would write the blob to a file and load it in a later run or on
+another machine.
+
+.. literalinclude:: ../../../python/cuda_cccl/tests/compute/examples/serialization/serialize_roundtrip.py
+   :language: python
+   :start-after: # example-begin
+   :caption: Serializing a built algorithm and reconstructing it without recompiling.
+
+The same argument-matching rules described above for the object-based API apply
+to a deserialized algorithm: the dtypes, iterator kinds, and operator you pass
+when invoking it must match those used when it was originally built.
+
+Blobs are versioned and self-describing, but they are not a long-term storage
+format: compatibility across ``cuda-cccl`` versions is not guaranteed, and
+loading a blob produced by a different version may be rejected with a clear
+error. Persist the inputs needed to rebuild (or re-serialize) rather than
+relying on old blobs surviving an upgrade.
+
+The same threading rules also apply: like any other reusable algorithm object,
+a deserialized algorithm must be used by one thread at a time unless access is
+externally serialized (see :ref:`Free-threaded Python <cuda.compute.free_threading>`).
+For concurrent use, call :func:`deserialize <cuda.compute.algorithms.deserialize>`
+in each thread — reconstruction performs no recompilation, so per-thread
+deserialization from one shared blob is cheap. (Currently each deserialized
+object loads its native build state independently; a future release may share
+that state behind the scenes.)
+
+.. _cuda.compute.ahead_of_time_compilation:
+
+Ahead-of-Time Compilation
+-------------------------
+
+By default, an algorithm is compiled for the compute capability of the *current*
+device. To build for other GPUs—or to build on a machine with no GPU at all—pass
+``compute_capability=`` to any ``make_*`` factory. Combined with
+:ref:`serialization <cuda.compute.serialization>`, this lets you compile once on
+a build machine and ship a ready-to-run artifact to your deployment targets.
+
+Building for specific architectures
++++++++++++++++++++++++++++++++++++
+
+Pass a single compute capability, or a list of them, to build one artifact that
+runs on any of the listed architectures:
+
+.. code-block:: python
+
+   reducer = cuda.compute.make_reduce_into(
+       d_in=d_in, d_out=d_out, op=OpKind.PLUS, h_init=h_init,
+       compute_capability=[80, 90],   # build for sm_80 and sm_90
+   )
+
+A compute capability may be given as an integer (``90``, ``75``), a
+``(major, minor)`` pair (``(9, 0)``), or a string (``"9.0"``). When the argument
+is omitted, the current device's architecture is used.
+
+When a multi-architecture artifact is invoked, the build result matching the
+running GPU is selected and loaded on the first call. Invoking it on a GPU whose
+architecture was not built raises an error.
+
+The compiled artifact may be reused by multiple devices with the same compute
+capability. Loading remains device-specific — each device loads its own copy of
+the compiled payload — but no device recompiles.
+
+Building without a GPU
+++++++++++++++++++++++
+
+A ``make_*`` factory normally inspects its input arrays to determine dtypes,
+which requires real device allocations. To build on a machine that has no
+GPU—for example, in CI—pass :class:`ProxyArray <cuda.compute.ProxyArray>` and
+:class:`ProxyValue <cuda.compute.ProxyValue>` placeholders in place of real
+arrays and scalars. A proxy describes *only* the dtype (and, for arrays, shape
+and contiguity); it holds no GPU memory.
+
+.. literalinclude:: ../../../python/cuda_cccl/tests/compute/examples/serialization/ahead_of_time_compilation.py
+   :language: python
+   :start-after: # example-begin
+   :caption: Compiling for two architectures with no GPU present.
+
+When building without a GPU you must pass ``compute_capability=`` explicitly:
+with no device to query, there is no architecture to default to. A proxy is a
+build-time placeholder only—supply the real device arrays and scalars when you
+invoke the (possibly deserialized) algorithm. Passing a proxy to a compiled
+algorithm's ``__call__`` raises ``RuntimeError``.
+
+.. _cuda.compute.externally_compiled_operators:
+
+Externally Compiled Operators
+-----------------------------
+
+:class:`RawOp <cuda.compute.op.RawOp>` can be used to directly pass compiled device code
+(LTO-IR) implementing custom operators.
+
+This is useful for users who wish to use a different compilation pipeline than the default
+used by ``cuda.compute`` (JIT compilation of Python callables using Numba CUDA).
+
+The example below shows how to compile a C++ device function
+to LTO-IR using `cuda.core <https://nvidia.github.io/cuda-python/cuda-core/latest/>`_,
+
+:func:`reduce_into <cuda.compute.algorithms.reduce_into>`:
+
+.. literalinclude:: ../../../python/cuda_cccl/tests/compute/examples/raw_op/cpp_stateless.py
+   :language: python
+   :start-after: # example-begin
+
+.. important::
+
+   **Required calling convention**: Compiled functions must use untyped pointers for
+   all parameters, with manual type casting inside the function. In C++, this means
+   all arguments (and the return value) must be passed as ``void*`` pointers:
+
+   .. code-block:: cpp
+
+      extern "C" __device__ void my_binary_op(void* a, void* b, void* result) {
+          *static_cast<int*>(result) = *static_cast<int*>(a) + *static_cast<int*>(b);
+      }
+
+   You must ensure that:
+
+   * All parameters are untyped pointers with manual casting in the function body
+   * Type casts match the actual data types passed at runtime
+   * For stateful operators, state is the first parameter (also an untyped pointer)
+   * State bytes have the correct layout and alignment
+
+   Type mismatches can cause crashes, memory corruption, or silent incorrect results.
+
+If you wish to use ``cuda.compute`` solely with externally compiled operators
+(i.e., without native JIT support), you can install a
+minimal version of the `cuda-cccl` package that ships without Numba/Numba CUDA dependencies:
+
+.. code-block:: bash
+
+   pip install cuda-cccl[minimal-cu13]      # or minimal-cu12  (pip-installed cuda-toolkit)
+   pip install cuda-cccl[minimal-sysctk13]  # or minimal-sysctk12  (system CUDA toolkit)
+
+
+
+Examples
+--------
+
+For complete runnable examples and additional usage patterns, see the
+`examples directory <https://github.com/NVIDIA/CCCL/tree/main/python/cuda_cccl/tests/compute/examples>`_.
+
+API Reference
+-------------
+
+- :ref:`cuda_compute-module`
diff --git a/cccl_upstream/docs/python/compute_api.rst b/cccl_upstream/docs/python/compute_api.rst
new file mode 100644
index 00000000..0952cc42
--- /dev/null
+++ b/cccl_upstream/docs/python/compute_api.rst
@@ -0,0 +1,105 @@
+.. _cuda_compute-module:
+
+``cuda.compute`` API Reference
+==============================
+
+.. warning::
+  ``cuda.compute`` is in public beta.
+  The API is subject to change without notice.
+
+Algorithms
+----------
+
+.. automodule:: cuda.compute.algorithms
+  :members:
+  :undoc-members:
+  :imported-members:
+
+Iterators
+---------
+
+.. automodule:: cuda.compute.iterators
+  :members:
+  :undoc-members:
+  :imported-members:
+
+Operators
+---------
+
+.. py:currentmodule:: cuda.compute.op
+
+.. Unfortunately, we need to manually document the OpKind enum here because
+.. the `._bindings` module, where OpKind is defined, is mocked out when building
+.. docs. The mock out is needed to avoid the need for CUDA to be installed
+.. at docs build time.
+
+.. py:class:: OpKind
+
+   Enumeration of operator kinds for CUDA parallel algorithms.
+
+   This enum defines the types of operations that can be performed
+   in parallel algorithms, including arithmetic, logical, and bitwise operations.
+
+   .. py:attribute:: PLUS
+   .. py:attribute:: MINUS
+   .. py:attribute:: MULTIPLIES
+   .. py:attribute:: DIVIDES
+   .. py:attribute:: MODULUS
+   .. py:attribute:: EQUAL_TO
+   .. py:attribute:: NOT_EQUAL_TO
+   .. py:attribute:: GREATER
+   .. py:attribute:: LESS
+   .. py:attribute:: GREATER_EQUAL
+   .. py:attribute:: LESS_EQUAL
+   .. py:attribute:: LOGICAL_AND
+   .. py:attribute:: LOGICAL_OR
+   .. py:attribute:: LOGICAL_NOT
+   .. py:attribute:: BIT_AND
+   .. py:attribute:: BIT_OR
+   .. py:attribute:: BIT_XOR
+   .. py:attribute:: BIT_NOT
+   .. py:attribute:: IDENTITY
+   .. py:attribute:: NEGATE
+   .. py:attribute:: MINIMUM
+   .. py:attribute:: MAXIMUM
+
+.. autoclass:: cuda.compute.op.RawOp
+   :members:
+   :undoc-members:
+
+Ahead-of-Time Compilation
+-------------------------
+
+The :func:`serialize <cuda.compute.algorithms.serialize>` and
+:func:`deserialize <cuda.compute.algorithms.deserialize>` functions (listed under
+`Algorithms`_ above) persist and restore built algorithms. To build ahead of time
+for architectures other than the current device's—or with no GPU present—pass the
+following dtype-only placeholders to a ``make_*`` factory in place of real arrays
+and scalars:
+
+.. py:class:: cuda.compute.ProxyArray(dtype)
+
+   A dtype-only placeholder for a device array. Use in place of a real device
+   array when calling a ``make_*`` factory to compile an algorithm without
+   allocating GPU memory. Satisfies the ``DeviceArrayLike`` protocol; accessing
+   its data pointer raises ``RuntimeError``. See
+   :ref:`cuda.compute.ahead_of_time_compilation`.
+
+.. py:class:: cuda.compute.ProxyValue(dtype)
+
+   A dtype-only placeholder for a scalar or initial-value argument (such as
+   ``h_init``). Use in place of a real numpy scalar or array to compile an
+   algorithm without real data; accessing its data raises ``RuntimeError``. See
+   :ref:`cuda.compute.ahead_of_time_compilation`.
+
+Utilities
+---------
+
+.. automodule:: cuda.compute.struct
+   :members:
+
+Typing
+------
+
+.. automodule:: cuda.compute.typing
+   :members:
diff --git a/cccl_upstream/docs/python/index.rst b/cccl_upstream/docs/python/index.rst
new file mode 100644
index 00000000..cc964e40
--- /dev/null
+++ b/cccl_upstream/docs/python/index.rst
@@ -0,0 +1,34 @@
+CCCL Python Libraries
+======================
+
+Overview
+--------
+
+The CUDA Core Compute Libraries (CCCL) for Python are a collection of modules
+with the shared goal of providing **high-quality, high-performance, and easy-to-use**
+abstractions for CUDA Python developers.
+
+* :doc:`cuda.compute <compute/index>` — Composable device-level primitives for building
+  custom parallel algorithms, without writing CUDA kernels directly.
+
+These libraries expose the generic, highly-optimized algorithms from the
+`CCCL C++ libraries <https://nvidia.github.io/cccl/cpp.html>`_,
+which have been tuned to provide optimal performance across GPU architectures.
+
+Who is this for?
+----------------
+
+- **Library authors** building parallel algorithms that need portable performance
+  across GPU architectures—without dropping to CUDA C++.
+
+- **Application developers** using PyTorch, CuPy, or other GPU-accelerated frameworks
+  who need custom algorithms beyond what those libraries provide.
+
+.. toctree::
+   :maxdepth: 2
+   :caption: CCCL Python Libraries
+
+   setup
+   compute/index
+   resources
+   api_reference
diff --git a/cccl_upstream/docs/python/resources.rst b/cccl_upstream/docs/python/resources.rst
new file mode 100644
index 00000000..a73e74e5
--- /dev/null
+++ b/cccl_upstream/docs/python/resources.rst
@@ -0,0 +1,38 @@
+Resources
+=========
+
+Examples
+--------
+
+For recipes and patterns, see our examples:
+
+* ``cuda.compute`` `examples <https://github.com/NVIDIA/cccl/tree/main/python/cuda_cccl/tests/compute/examples>`_
+
+CUB and Thrust Documentation
+----------------------------
+
+The CCCL Python libraries are built on top of the CUB and Thrust libraries.
+See the `CUB documentation <https://nvlabs.github.io/cub/>`_ and `Thrust documentation <https://thrust.github.io/>`_
+for more information regarding the underlying libraries.
+
+
+Asking for Help
+---------------
+
+If you have a question, run into an issue, or have a feature request,
+please raise an issue or start a discussion on our `GitHub repository <https://github.com/NVIDIA/cccl/issues>`_.
+
+Contributing
+------------
+
+We welcome contributions! Please see the
+`contributing guide <https://github.com/NVIDIA/cccl/blob/main/CONTRIBUTING.md>`_
+for instructions on how to set up a development environment and submit a pull request.
+
+Once you have a development environment set up, see :doc:`setup` for instructions
+on how to install `cuda.cccl` in development mode.
+
+License
+-------
+
+The CCCL Python libraries are licensed under the `Apache License 2.0 <https://www.apache.org/licenses/LICENSE-2.0>`_.
diff --git a/cccl_upstream/docs/python/setup.rst b/cccl_upstream/docs/python/setup.rst
new file mode 100644
index 00000000..f6183947
--- /dev/null
+++ b/cccl_upstream/docs/python/setup.rst
@@ -0,0 +1,106 @@
+.. _cccl-python-setup:
+
+Setup and Installation
+======================
+
+This guide walks you through installing and setting up the CUDA Python Core Libraries (CCCL).
+
+Prerequisites
+-------------
+
+Before installing cuda-cccl, ensure you have:
+
+* **Python 3.10 or later**
+* **CUDA Toolkit 12.x or 13.x**
+* **Compatible NVIDIA GPU** with Compute Capability 7.5 or higher
+* **Operating Systems:** Linux (tested on Ubuntu 20.04+) or Windows 10/11 (with WSL2 support)
+
+Installation
+------------
+
+Install from PyPI
+~~~~~~~~~~~~~~~~~
+
+The easiest way to install ``cuda-cccl`` is using pip:
+
+.. code-block:: bash
+
+   pip install cuda-cccl[cu13]  # or cuda-cccl[cu12]
+
+This will install ``cuda-cccl`` along with all required dependencies, including
+the ``cuda-toolkit`` pip packages for the chosen CUDA major version.
+
+If you already have a CUDA toolkit installed on your system (e.g., via the
+NVIDIA runfile, package manager, or Conda) and do not want pip to install it,
+use the ``sysctk`` variants instead:
+
+.. code-block:: bash
+
+   pip install cuda-cccl[sysctk13]  # or cuda-cccl[sysctk12]
+
+These install the same dependencies except ``cuda-toolkit``; it is your
+responsibility to ensure a compatible CUDA toolkit is on ``PATH`` and
+``LD_LIBRARY_PATH``.
+
+For a minimal install without Numba (useful when you supply your own
+:ref:`pre-compiled operators <cuda.compute.externally_compiled_operators>`), use:
+
+.. code-block:: bash
+
+   pip install cuda-cccl[minimal-cu13]      # pip-installed CUDA toolkit
+   pip install cuda-cccl[minimal-sysctk13]  # system CUDA toolkit
+
+Free-threaded Python support is currently validated with the ``minimal-cu12``
+and ``minimal-cu13`` extras. The full ``cu12`` and ``cu13`` extras depend on
+Numba CUDA and are not currently supported in free-threaded Python.
+
+Install from conda-forge
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Alternatively, you can install ``cuda-cccl`` using conda:
+
+.. code-block:: bash
+
+   conda install -c conda-forge cccl-python
+
+This will install the CCCL Python libraries and their dependencies from the conda-forge channel.
+
+Install from Source
+~~~~~~~~~~~~~~~~~~~
+
+For development or to access the latest features:
+
+.. code-block:: bash
+
+   git clone https://github.com/NVIDIA/cccl.git
+   cd cccl/python/cuda_cccl
+   pip install -e .[test-cu13]  # or .[test-cu12], .[test-sysctk13], .[test-sysctk12]
+
+The test extras do not install CuPy. To also run the CuPy-based
+``cuda.compute`` examples, install CuPy separately, for example
+``pip install cupy-cuda13x``.
+
+
+Development Setup
+~~~~~~~~~~~~~~~~~~
+
+For contributing to cuda-cccl or advanced development:
+
+.. code-block:: bash
+
+   # Clone the repository
+   git clone https://github.com/NVIDIA/cccl.git
+   cd cccl/python/cuda_cccl
+
+   # Install in development mode with test dependencies
+   pip install -e .[test-cu13]  # or .[test-cu12], .[test-sysctk13], .[test-sysctk12]
+
+   # Run tests to verify everything works
+   pytest tests/
+
+Next Steps
+----------
+
+Now that you have ``cuda-cccl`` installed, check out:
+
+* :doc:`compute/index` - Parallel computing primitives for operations on arrays or data ranges
diff --git a/cccl_upstream/docs/requirements.txt b/cccl_upstream/docs/requirements.txt
new file mode 100644
index 00000000..05dee38b
--- /dev/null
+++ b/cccl_upstream/docs/requirements.txt
@@ -0,0 +1,11 @@
+# Documentation dependencies
+sphinx>=8.1.3
+breathe>=4.36.0
+exhale>=0.3.0
+nvidia-sphinx-theme>=0.0.8
+myst-parser>=4.0.1
+sphinx-copybutton>=0.5.2
+sphinx-design>=0.6.1
+nbsphinx>=0.9.0
+numpydoc>=1.5.0
+numpy>=1.24.0  # Required for type annotations in Python modules
diff --git a/cccl_upstream/docs/scrape_docs.bash b/cccl_upstream/docs/scrape_docs.bash
new file mode 100755
index 00000000..f54eedcc
--- /dev/null
+++ b/cccl_upstream/docs/scrape_docs.bash
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+
+## Usage:
+## ./scrape_docs.bash [path]
+## [path] is the starting point for searching for HTML. Ideally this is the siteroot
+## script will find all HTML files and record them into a CSV file that can be used for searching docs.
+
+set -euo pipefail
+
+SCRIPT_PATH=$(cd "$(dirname "$0")"; pwd -P)
+
+path_to_docs=$(realpath "$1")
+
+cd "$SCRIPT_PATH"
+
+pages=$(
+    cd "$path_to_docs";
+    find ./ -iname "*.html" -printf '/%P,'
+)
+
+echo "$pages" > "$path_to_docs/pagelist.txt"
diff --git a/cccl_upstream/docs/thrust/Doxyfile b/cccl_upstream/docs/thrust/Doxyfile
new file mode 100644
index 00000000..0468c283
--- /dev/null
+++ b/cccl_upstream/docs/thrust/Doxyfile
@@ -0,0 +1,169 @@
+# Doxyfile for Thrust
+
+PROJECT_NAME           = Thrust
+OUTPUT_DIRECTORY       = ../_build/doxygen/thrust
+CREATE_SUBDIRS         = NO
+GENERATE_HTML          = NO
+GENERATE_LATEX         = NO
+GENERATE_XML           = YES
+XML_OUTPUT             = xml
+XML_PROGRAMLISTING     = YES
+
+INPUT                  = ../../thrust/thrust \
+                         ../../thrust/thrust/iterator \
+                         ../../thrust/thrust/mr \
+                         ../../thrust/thrust/random \
+                         ../../thrust/thrust/system \
+                         ../../thrust/thrust/system/cpp \
+                         ../../thrust/thrust/system/omp \
+                         ../../thrust/thrust/system/tbb \
+                         ../../thrust/thrust/type_traits \
+                         ../../thrust/thrust/detail/pointer.h
+
+RECURSIVE              = YES
+EXCLUDE_PATTERNS       = */detail/* */test/* */examples/* */testing/*
+EXCLUDE_SYMBOLS        = *detail* *__* THRUST_NS_QUALIFIER optional
+
+FILE_PATTERNS          = *.h *.hpp
+EXTENSION_MAPPING      = cuh=C++ cu=C++
+
+# Documentation extraction settings
+EXTRACT_ALL            = YES
+EXTRACT_PRIVATE        = NO
+EXTRACT_STATIC         = YES
+EXTRACT_LOCAL_CLASSES  = YES
+HIDE_UNDOC_MEMBERS     = NO
+HIDE_UNDOC_CLASSES     = NO
+SHOW_INCLUDE_FILES     = YES
+INLINE_INHERITED_MEMB  = YES
+FULL_PATH_NAMES        = YES
+STRIP_FROM_PATH        = ../../thrust
+SHORT_NAMES            = NO
+DISTRIBUTE_GROUP_DOC   = YES
+
+# Parsing settings
+JAVADOC_AUTOBRIEF      = YES
+QT_AUTOBRIEF           = NO
+MULTILINE_CPP_IS_BRIEF = NO
+INHERIT_DOCS           = YES
+SEPARATE_MEMBER_PAGES  = NO
+TAB_SIZE               = 4
+BUILTIN_STL_SUPPORT    = YES
+
+# Preprocessing
+ENABLE_PREPROCESSING   = YES
+MACRO_EXPANSION        = YES
+EXPAND_ONLY_PREDEF     = NO
+SEARCH_INCLUDES        = YES
+SKIP_FUNCTION_MACROS   = YES
+
+# IMPORTANT: Aliases for custom commands
+# The rst alias enables embedding reStructuredText in doxygen comments
+ALIASES = "rst=\\verbatim embed:rst"
+ALIASES += "endrst=\\endverbatim"
+ALIASES += "rststar=\\verbatim embed:rst:leading-asterisk"
+ALIASES += "inlinerst=\\verbatim embed:rst:inline"
+
+# Predefined macros
+PREDEFINED             = __device__= \
+                        __host__= \
+                        __global__= \
+                        __forceinline__= \
+                        "__declspec(x)=" \
+                        "__align__(x)=" \
+                        __cccl_lib_mdspan \
+                        _CCCL_DOXYGEN_INVOKED \
+                        "_CCCL_AND=&&" \
+                        "_CCCL_CONCEPT=constexpr bool" \
+                        "_CCCL_CONSTEXPR_FRIEND=friend" \
+                        "_CCCL_CONSTEXPR_CXX20=constexpr" \
+                        "_CCCL_CONSTEXPR_CXX23=constexpr" \
+                        "_CCCL_CTK_AT_LEAST(x, y)=1" \
+                        "_CCCL_CTK_BELOW(x, y)=0" \
+                        "_CCCL_CUDACC_AT_LEAST(x, y)=1" \
+                        "_CCCL_CUDACC_BELOW(x, y)=0" \
+                        _CCCL_DEVICE= \
+                        _CCCL_DIAG_PUSH= \
+                        _CCCL_DIAG_POP= \
+                        "_CCCL_DIAG_SUPPRESS_CLANG(x)=" \
+                        "_CCCL_DIAG_SUPPRESS_GCC(x)=" \
+                        "_CCCL_DIAG_SUPPRESS_MSVC(x)=" \
+                        "_CCCL_DIAG_SUPPRESS_NVHPC(x)=" \
+                        _CCCL_EXEC_CHECK_DISABLE= \
+                        _CCCL_FORCEINLINE= \
+                        "_CCCL_GLOBAL_CONSTANT=inline constexpr" \
+                        "_CCCL_HAS_CTK()=1" \
+                        _CCCL_HIDE_FROM_ABI= \
+                        _CCCL_HOST= \
+                        _CCCL_HOST_DEVICE= \
+                        "_CCCL_IF_CONSTEVAL=if consteval" \
+                        "_CCCL_INLINE_VAR=inline" \
+                        "_CCCL_NODISCARD=[[nodiscard]]" \
+                        "_CCCL_NODISCARD_FRIEND=friend" \
+                        "_CCCL_REQUIRES(x)= ::cuda::std::enable_if_t<x, int> = 0>" \
+                        _CCCL_STD_VER=2020 \
+                        _CCCL_SUPPRESS_DEPRECATED_PUSH= \
+                        _CCCL_SUPPRESS_DEPRECATED_POP= \
+                        "_CCCL_TEMPLATE(x)=template<x, " \
+                        "_CCCL_TRAIT(x, y)=x<y>::value" \
+                        "_CCCL_TRAILING_REQUIRES(x)=-> x requires " \
+                        _CCCL_TYPE_VISIBILITY_DEFAULT= \
+                        _CCCL_TYPE_VISIBILITY_HIDDEN= \
+                        _CCCL_API=inline \
+                        _CCCL_HOST_DEVICE_API=inline \
+                        _CCCL_DEVICE_API=inline \
+                        _CCCL_HOST_API=inline \
+                        _CCCL_NODEBUG_API=inline \
+                        _CCCL_NODEBUG_DEVICE_API=inline \
+                        _CCCL_NODEBUG_HOST_API=inline \
+                        _CCCL_TRIVIAL_API=inline \
+                        _CCCL_TRIVIAL_DEVICE_API=inline \
+                        _CCCL_TRIVIAL_HOST_API=inline \
+                        _CCCL_PUBLIC_API=inline \
+                        _CCCL_PUBLIC_DEVICE_API=inline \
+                        _CCCL_PUBLIC_HOST_API=inline \
+                        _CCCL_VISIBILITY_DEFAULT= \
+                        _CCCL_VISIBILITY_HIDDEN= \
+                        _CCCL_LIFETIMEBOUND= \
+                        _CCCL_TRY=try \
+                        _CCCL_CATCH=catch \
+                        "_CCCL_CATCH_ALL=catch (...)" \
+                        "_CCCL_CATCH_FALLTHROUGH=" \
+                        "_LIBCUDACXX_NOEXCEPT_RETURN(...)=noexcept(noexcept(...)) { return ...; }" \
+                        "THRUST_NAMESPACE_BEGIN=namespace thrust {" \
+                        "THRUST_NAMESPACE_END=}" \
+                        THRUST_PREVENT_MACRO_SUBSTITUTION \
+                        "THRUST_FWD(x)=x" \
+                        "_CCCL_PUBLIC_API=inline" \
+                        "_CCCL_PUBLIC_DEVICE_API=inline" \
+                        "_CCCL_PUBLIC_HOST_API=inline" \
+                        "_CCCL_BEGIN_NAMESPACE_CUDA_STD=namespace cuda::std {" \
+                        "_CCCL_END_NAMESPACE_CUDA_STD=}" \
+                        "_CUDAX_CONSTEXPR_FRIEND=friend" \
+                        "_LIBCUDACXX_HAS_SPACESHIP_OPERATOR()=1" \
+                        "CCCL_DEPRECATED=" \
+                        "CCCL_DEPRECATED_BECAUSE(x)=" \
+                        "CCCL_IGNORE_DEPRECATED_CPP_DIALECT" \
+                        "CUB_DISABLE_NAMESPACE_MAGIC" \
+                        "CUB_IGNORE_NAMESPACE_MAGIC_ERROR" \
+                        "CUB_RDC_ENABLED" \
+                        "CUB_NAMESPACE_BEGIN=namespace cub {" \
+                        "CUB_NAMESPACE_END=}" \
+                        "CUB_RUNTIME_FUNCTION=" \
+                        "_CCCL_HOSTED()=1"
+
+# Images and examples
+IMAGE_PATH             = ../img
+EXAMPLE_PATH           = ../../thrust/examples
+EXAMPLE_RECURSIVE      = NO
+EXAMPLE_PATTERNS       = *.cu
+
+# Output settings
+AUTOLINK_SUPPORT       = YES
+QUIET                  = YES
+WARNINGS               = YES
+WARN_AS_ERROR          = FAIL_ON_WARNINGS
+WARN_IF_UNDOCUMENTED   = NO
+WARN_IF_DOC_ERROR      = YES
+WARN_IF_INCOMPLETE_DOC = NO
+WARN_NO_PARAMDOC       = NO
diff --git a/cccl_upstream/docs/thrust/algorithms.rst b/cccl_upstream/docs/thrust/algorithms.rst
new file mode 100644
index 00000000..2d74be22
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms.rst
@@ -0,0 +1,18 @@
+.. _thrust-module-api-algorithms:
+
+Algorithms
+===========
+
+.. toctree::
+   :glob:
+   :maxdepth: 2
+
+   algorithms/copying
+   algorithms/merging
+   algorithms/prefix_sum
+   algorithms/reductions
+   algorithms/reordering
+   algorithms/searching
+   algorithms/set_operations
+   algorithms/sorting
+   algorithms/transformations
diff --git a/cccl_upstream/docs/thrust/algorithms/copying.rst b/cccl_upstream/docs/thrust/algorithms/copying.rst
new file mode 100644
index 00000000..125c98e6
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/copying.rst
@@ -0,0 +1,12 @@
+.. _thrust-module-api-algorithms-copying:
+
+Copying
+-------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   copying/gather
+   copying/scatter
+   ../api/group__copying*
diff --git a/cccl_upstream/docs/thrust/algorithms/copying/gather.rst b/cccl_upstream/docs/thrust/algorithms/copying/gather.rst
new file mode 100644
index 00000000..7b4885b4
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/copying/gather.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-copying-gather:
+
+Gather
+^^^^^^^
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../../api/group__gather*
diff --git a/cccl_upstream/docs/thrust/algorithms/copying/scatter.rst b/cccl_upstream/docs/thrust/algorithms/copying/scatter.rst
new file mode 100644
index 00000000..1a9b4b55
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/copying/scatter.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-copying-scatter:
+
+Scatter
+^^^^^^^
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../../api/group__scatter*
diff --git a/cccl_upstream/docs/thrust/algorithms/merging.rst b/cccl_upstream/docs/thrust/algorithms/merging.rst
new file mode 100644
index 00000000..4d89d160
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/merging.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-merging:
+
+Merging
+-------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../api/group__merging*
diff --git a/cccl_upstream/docs/thrust/algorithms/prefix_sum.rst b/cccl_upstream/docs/thrust/algorithms/prefix_sum.rst
new file mode 100644
index 00000000..09404a08
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/prefix_sum.rst
@@ -0,0 +1,12 @@
+.. _thrust-module-api-algorithms-prefix_sums:
+
+Prefix sums
+-----------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   prefix_sums/segmented
+   prefix_sums/transformed
+   ../api/group__prefixsums*
diff --git a/cccl_upstream/docs/thrust/algorithms/prefix_sums/segmented.rst b/cccl_upstream/docs/thrust/algorithms/prefix_sums/segmented.rst
new file mode 100644
index 00000000..65d76b90
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/prefix_sums/segmented.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-prefix_sums-segmented:
+
+Segmented Prefix Sums
+----------------------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../../api/group__segmentedprefix*
diff --git a/cccl_upstream/docs/thrust/algorithms/prefix_sums/transformed.rst b/cccl_upstream/docs/thrust/algorithms/prefix_sums/transformed.rst
new file mode 100644
index 00000000..14538d64
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/prefix_sums/transformed.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-prefix_sums-transformed:
+
+Transformed Prefix Sums
+-------------------------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../../api/group__transformed__prefixsums*
diff --git a/cccl_upstream/docs/thrust/algorithms/reductions.rst b/cccl_upstream/docs/thrust/algorithms/reductions.rst
new file mode 100644
index 00000000..c74fc396
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/reductions.rst
@@ -0,0 +1,16 @@
+.. _thrust-module-api-algorithms-reductions:
+
+Reductions
+----------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   reductions/comparisons
+   reductions/counting
+   reductions/extrema
+   reductions/logical
+   reductions/predicates
+   reductions/transformed
+   ../api/group__reductions*
diff --git a/cccl_upstream/docs/thrust/algorithms/reductions/comparisons.rst b/cccl_upstream/docs/thrust/algorithms/reductions/comparisons.rst
new file mode 100644
index 00000000..16a45454
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/reductions/comparisons.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-reductions-comparisons:
+
+Comparisons
+-----------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../../api/group__comparisons*
diff --git a/cccl_upstream/docs/thrust/algorithms/reductions/counting.rst b/cccl_upstream/docs/thrust/algorithms/reductions/counting.rst
new file mode 100644
index 00000000..4caa9741
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/reductions/counting.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-reductions-counting:
+
+Counting
+-----------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../../api/group__counting*
diff --git a/cccl_upstream/docs/thrust/algorithms/reductions/extrema.rst b/cccl_upstream/docs/thrust/algorithms/reductions/extrema.rst
new file mode 100644
index 00000000..89bc944b
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/reductions/extrema.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-reductions-extrema:
+
+Extrema
+-----------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../../api/group__extrema*
diff --git a/cccl_upstream/docs/thrust/algorithms/reductions/logical.rst b/cccl_upstream/docs/thrust/algorithms/reductions/logical.rst
new file mode 100644
index 00000000..bfdde268
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/reductions/logical.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-reductions-logical:
+
+Logical
+-----------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../../api/group__logical*
diff --git a/cccl_upstream/docs/thrust/algorithms/reductions/predicates.rst b/cccl_upstream/docs/thrust/algorithms/reductions/predicates.rst
new file mode 100644
index 00000000..17d29eff
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/reductions/predicates.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-reductions-predicates:
+
+Predicates
+-----------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../../api/group__predicates*
diff --git a/cccl_upstream/docs/thrust/algorithms/reductions/transformed.rst b/cccl_upstream/docs/thrust/algorithms/reductions/transformed.rst
new file mode 100644
index 00000000..afa8163e
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/reductions/transformed.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-reductions-transformed:
+
+Transformed Reductions
+----------------------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../../api/group__transformed__reductions*
diff --git a/cccl_upstream/docs/thrust/algorithms/reordering.rst b/cccl_upstream/docs/thrust/algorithms/reordering.rst
new file mode 100644
index 00000000..40a778d7
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/reordering.rst
@@ -0,0 +1,12 @@
+.. _thrust-module-api-algorithms-reordering:
+
+Reordering
+----------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   reordering/partitioning
+   reordering/shuffling
+   reordering/stream_compaction
diff --git a/cccl_upstream/docs/thrust/algorithms/reordering/partitioning.rst b/cccl_upstream/docs/thrust/algorithms/reordering/partitioning.rst
new file mode 100644
index 00000000..3889c831
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/reordering/partitioning.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-reordering-partitioning:
+
+Partitioning
+-------------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../../api/group__partitioning*
diff --git a/cccl_upstream/docs/thrust/algorithms/reordering/shuffling.rst b/cccl_upstream/docs/thrust/algorithms/reordering/shuffling.rst
new file mode 100644
index 00000000..1bb707f1
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/reordering/shuffling.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-reordering-shuffling:
+
+Shuffling
+-------------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../../api/group__shuffling*
diff --git a/cccl_upstream/docs/thrust/algorithms/reordering/stream_compaction.rst b/cccl_upstream/docs/thrust/algorithms/reordering/stream_compaction.rst
new file mode 100644
index 00000000..5ab466a5
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/reordering/stream_compaction.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-reordering-stream-compactation:
+
+Stream Compactation
+-------------------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../../api/group__stream__compaction*
diff --git a/cccl_upstream/docs/thrust/algorithms/searching.rst b/cccl_upstream/docs/thrust/algorithms/searching.rst
new file mode 100644
index 00000000..24566bd6
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/searching.rst
@@ -0,0 +1,11 @@
+.. _thrust-module-api-algorithms-searching:
+
+Searching
+----------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   searching/binary_search
+   ../api/group__searching*
diff --git a/cccl_upstream/docs/thrust/algorithms/searching/binary_search.rst b/cccl_upstream/docs/thrust/algorithms/searching/binary_search.rst
new file mode 100644
index 00000000..e887289f
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/searching/binary_search.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-searching-binary-search:
+
+Binary Search
+--------------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../../api/group__binary__search*
diff --git a/cccl_upstream/docs/thrust/algorithms/set_operations.rst b/cccl_upstream/docs/thrust/algorithms/set_operations.rst
new file mode 100644
index 00000000..3432fe4b
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/set_operations.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-set-operations:
+
+Set Operations
+---------------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../api/group__set__operations*
diff --git a/cccl_upstream/docs/thrust/algorithms/sorting.rst b/cccl_upstream/docs/thrust/algorithms/sorting.rst
new file mode 100644
index 00000000..15cf64d0
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/sorting.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-sorting:
+
+Sorting
+--------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../api/group__sorting*
diff --git a/cccl_upstream/docs/thrust/algorithms/transformations.rst b/cccl_upstream/docs/thrust/algorithms/transformations.rst
new file mode 100644
index 00000000..fa7323ce
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/transformations.rst
@@ -0,0 +1,13 @@
+.. _thrust-module-api-algorithms-transformations:
+
+Transformations
+-----------------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   transformations/filling
+   transformations/modifying
+   transformations/replacing
+   ../api/group__transformations*
diff --git a/cccl_upstream/docs/thrust/algorithms/transformations/filling.rst b/cccl_upstream/docs/thrust/algorithms/transformations/filling.rst
new file mode 100644
index 00000000..a05fa848
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/transformations/filling.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-transformations-filling:
+
+Filling
+--------------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../../api/group__filling*
diff --git a/cccl_upstream/docs/thrust/algorithms/transformations/modifying.rst b/cccl_upstream/docs/thrust/algorithms/transformations/modifying.rst
new file mode 100644
index 00000000..3496c728
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/transformations/modifying.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-transformations-modifying:
+
+Modifying
+--------------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../../api/group__modifying*
diff --git a/cccl_upstream/docs/thrust/algorithms/transformations/replacing.rst b/cccl_upstream/docs/thrust/algorithms/transformations/replacing.rst
new file mode 100644
index 00000000..2e73f2eb
--- /dev/null
+++ b/cccl_upstream/docs/thrust/algorithms/transformations/replacing.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-algorithms-transformations-replacing:
+
+Replacing
+--------------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../../api/group__replacing*
diff --git a/cccl_upstream/docs/thrust/containers.rst b/cccl_upstream/docs/thrust/containers.rst
new file mode 100644
index 00000000..94698ee6
--- /dev/null
+++ b/cccl_upstream/docs/thrust/containers.rst
@@ -0,0 +1,12 @@
+.. _thrust-module-api-containers:
+
+Containers
+===========
+
+.. toctree::
+   :glob:
+   :maxdepth: 2
+
+   api/*host__vector*
+   api/*device__vector*
+   api/group__containers_*
diff --git a/cccl_upstream/docs/thrust/developer/cmake_options.rst b/cccl_upstream/docs/thrust/developer/cmake_options.rst
new file mode 100644
index 00000000..2a6ae959
--- /dev/null
+++ b/cccl_upstream/docs/thrust/developer/cmake_options.rst
@@ -0,0 +1,183 @@
+.. _cmake-options:
+
+Developer CMake Options
+=======================
+
+.. important::
+
+    This document details the CMake options used for **developer builds**
+    of the Thrust tests and examples included in the CCCL repository.
+    The options presented here are not available when using Thrust as an
+    end-user via our installed CMake packages, or when using
+    ``add_subdirectory`` / CPM to add CCCL/Thrust to your project's build system.
+    For details and examples of using Thrust as an end-user, please refer
+    to the `Thrust + CMake example <https://github.com/NVIDIA/cccl/tree/main/examples/thrust_flexible_device_system>`_
+
+
+A Thrust build is configured using CMake options. These may be passed to
+CMake using
+
+.. code:: bash
+
+   cmake -D<option_name>=<value> /path/to/thrust/sources
+
+or configured interactively with the ``ccmake`` or ``cmake-gui``
+interfaces.
+
+Thrust supports two build modes. By default, a single configuration is
+built that targets a specific host system, device system, and C++
+dialect. When ``THRUST_ENABLE_MULTICONFIG`` is ``ON``, multiple
+configurations targeting a variety of systems and dialects are
+generated.
+
+The CMake options are divided into these categories:
+
+1. `Generic CMake Options <#generic-cmake-options>`__: Options
+   applicable to all Thrust builds.
+2. `Single Config CMake Options <#single-config-cmake-options>`__
+   Options applicable only when ``THRUST_ENABLE_MULTICONFIG`` is
+   disabled.
+3. `Multi Config CMake Options <#multi-config-cmake-options>`__ Options
+   applicable only when ``THRUST_ENABLE_MULTICONFIG`` is enabled.
+4. `CUDA Specific CMake Options <#cuda-specific-cmake-options>`__
+   Options that control CUDA compilation. Only available when one or
+   more configurations targets the CUDA system.
+5. `TBB Specific CMake Options <#tbb-specific-cmake-options>`__
+   Options that control TBB compilation. Only available when one or
+   more configurations targets the TBB system.
+
+Generic CMake Options
+---------------------
+
+-  ``CMAKE_BUILD_TYPE={Release, Debug, RelWithDebInfo, MinSizeRel}``
+
+   -  Standard CMake build option. Default: ``RelWithDebInfo``
+
+-  ``THRUST_ENABLE_HEADER_TESTING={ON, OFF}``
+
+   -  Whether to test compile public headers. Default is ``ON``.
+
+-  ``THRUST_ENABLE_TESTING={ON, OFF}``
+
+   -  Whether to build unit tests. Default is ``ON``.
+
+-  ``THRUST_ENABLE_EXAMPLES={ON, OFF}``
+
+   -  Whether to build examples. Default is ``ON``.
+
+-  ``THRUST_ENABLE_MULTICONFIG={ON, OFF}``
+
+   -  Toggles single-config and multi-config modes. Default is ``OFF``
+      (single config).
+
+-  ``THRUST_ENABLE_EXAMPLE_FILECHECK={ON, OFF}``
+
+   -  Enable validation of example outputs using the LLVM FileCheck
+      utility. Default is ``OFF``.
+
+-  ``THRUST_ENABLE_INSTALL_RULES={ON, OFF}``
+
+   -  If true, installation rules will be generated for thrust. Default
+      is ``ON``.
+
+-  ``THRUST_DISPATCH_TYPE={Dynamic, Force32bit, Force64bit}``
+
+   -  Allows the user to force Thrust to use a specific size for the offset type. Default
+      is ``Dynamic``.
+
+      -  ``Dynamic`` lets Thrust choose the index type based on input size, allowing
+         large inputs and optimal performance at the cost of increased compile time and binary size,
+         as Thrust will compile each kernel twice, once for 32 bit and once for 64 bit.
+      -  ``Force32bit`` forces Thrust to use a 32 bit offset type. This improves compile time and
+         binary size but limits the input size.
+      -  ``Force64bit`` forces Thrust to use a 64 bit offset type. This improves compile time and
+         binary size and allows large input sizes. However, it might degrade runtime performance.
+
+Single Config CMake Options
+---------------------------
+
+-  ``THRUST_HOST_SYSTEM={CPP, TBB, OMP}``
+
+   -  Selects the host system. Default: ``CPP``
+
+-  ``THRUST_DEVICE_SYSTEM={CUDA, TBB, OMP, CPP}``
+
+   -  Selects the device system. Default: ``CUDA``
+
+-  ``THRUST_CPP_DIALECT={17, 20}``
+
+   -  Selects the C++ standard dialect to use. Default is ``14``
+      (C++14).
+
+.. _cmake-multi-config-options:
+
+Multi Config CMake Options
+--------------------------
+
+-  ``THRUST_MULTICONFIG_ENABLE_DIALECT_CPPXX={ON, OFF}``
+
+   -  Toggle whether a specific C++ dialect will be targeted.
+   -  Possible values of ``XX`` are ``{17, 20}``.
+   -  By default, only C++14 is enabled.
+
+-  ``THRUST_MULTICONFIG_ENABLE_SYSTEM_XXXX={ON, OFF}``
+
+   -  Toggle whether a specific system will be targeted.
+   -  Possible values of ``XXXX`` are ``{CPP, CUDA, TBB, OMP}``
+   -  By default, only ``CPP`` and ``CUDA`` are enabled.
+
+-  ``THRUST_MULTICONFIG_WORKLOAD={SMALL, MEDIUM, LARGE, FULL}``
+
+   -  Restricts the host/device combinations that will be targeted.
+   -  By default, the ``SMALL`` workload is used.
+   -  The full cross product of ``host x device`` systems results in 12
+      configurations, some of which are more important than others. This
+      option can be used to prune some of the less important ones.
+   -  ``SMALL``: (3 configs) Minimal coverage and validation of each
+      device system against the ``CPP`` host.
+   -  ``MEDIUM``: (6 configs) Cheap extended coverage.
+   -  ``LARGE``: (8 configs) Expensive extended coverage. Includes all
+      useful build configurations.
+   -  ``FULL``: (12 configs) The complete cross product of all possible
+      build configurations.
+
+======== =========== ========== ========= ============================
+Config   Workloads   Value      Expense   Note
+======== =========== ========== ========= ============================
+CPP/CUDA ``F L M S`` Essential  Expensive Validates CUDA against CPP
+CPP/OMP  ``F L M S`` Essential  Cheap     Validates OMP against CPP
+CPP/TBB  ``F L M S`` Essential  Cheap     Validates TBB against CPP
+CPP/CPP  ``F L M``   Important  Cheap     Tests CPP as device
+OMP/OMP  ``F L M``   Important  Cheap     Tests OMP as host
+TBB/TBB  ``F L M``   Important  Cheap     Tests TBB as host
+TBB/CUDA ``F L``     Important  Expensive Validates TBB/CUDA interop
+OMP/CUDA ``F L``     Important  Expensive Validates OMP/CUDA interop
+TBB/OMP  ``F``       Not useful Cheap     Mixes CPU-parallel systems
+OMP/TBB  ``F``       Not useful Cheap     Mixes CPU-parallel systems
+TBB/CPP  ``F``       Not Useful Cheap     Parallel host, serial device
+OMP/CPP  ``F``       Not Useful Cheap     Parallel host, serial device
+======== =========== ========== ========= ============================
+
+CUDA Specific CMake Options
+---------------------------
+
+-  ``THRUST_ENABLE_RDC_TESTS={ON, OFF}``
+
+   -  Enable tests that require separable compilation.
+   -  Default is ``ON``.
+
+-  ``THRUST_FORCE_RDC={ON, OFF}``
+
+   -  Enable separable compilation on all targets that are agnostic of
+      RDC.
+   -  Targets that explicitly require RDC to be enabled or disabled will
+      ignore this setting.
+   -  Default is ``OFF``.
+
+TBB Specific CMake Options
+--------------------------
+
+When using TBB as a host or device system, Thrust will automatically find and
+link against an installed Intel TBB. The TBB installation is discovered through
+CMake's standard ``find_package(TBB)`` mechanism. No additional Thrust-specific
+options are provided for TBB configuration.
diff --git a/cccl_upstream/docs/thrust/developer/systems.rst b/cccl_upstream/docs/thrust/developer/systems.rst
new file mode 100644
index 00000000..143411ac
--- /dev/null
+++ b/cccl_upstream/docs/thrust/developer/systems.rst
@@ -0,0 +1,271 @@
+.. _systems:
+
+Thrust systems
+==============
+
+Thrust offers a set of algorithms and APIs which can dispatch to various systems.
+A system is basically a backend and Thrust currently supports the following systems:
+
+ - cpp
+ - cuda
+ - omp
+ - tbb
+ - generic
+ - sequential
+
+The generic and sequential systems are implementation details.
+Users can define additional systems to add new backends.
+
+Each system lives in a directory under ``thrust/system/[detail/]``.
+
+
+Execution policy base classes
+*****************************
+
+Thrust defines common base classes for execution policies:
+
+.. code-block:: c++
+
+    namespace thrust::detail {
+      struct execution_policy_marker {};
+
+      template <typename DerivedPolicy>
+      struct execution_policy_base : execution_policy_marker {};
+    }
+    namespace thrust {
+      template <typename DerivedPolicy>
+      struct execution_policy : thrust::detail::execution_policy_base<DerivedPolicy> {};
+    }
+
+There is an execution policy marker, which sits at the top of the inheritance chain.
+Then, we have an execution policy base and the actual execution policy,
+both are templated on the derived policy type (CRTP).
+
+
+System execution policy base classes and tags
+*********************************************
+
+Inside each system directory is a header file ``execution_policy.h``
+which defines the execution policy and tag for that system,
+except for the generic system, which does not have a dedicated execution policy.
+The inheritance for a system, e.g. ``cpp``, looks like this:
+
+.. code-block:: c++
+
+    namespace thrust::system::cpp {
+      struct tag; // forward declaration
+
+      template <typename Derived>
+      struct execution_policy; // forward declaration
+
+      template <>
+      struct execution_policy<tag> : ... {};
+
+      struct tag : execution_policy<tag> {};
+
+      template <typename Derived>
+      struct execution_policy : ... {
+        using tag_type = tag;
+        _CCCL_HOST_DEVICE operator tag() const { return {}; }
+      };
+    }
+
+Each system has it's own execution policy, again templated on a further derived execution policy.
+The system's execution policy derives (directly or indirectly) from ``thrust::execution_policy``.
+System execution policies are templates and intended to be further derived from.
+Additionally, there is a tag for each system, without any template parameters,
+that derives from the system's execution policy.
+Tags are non-template class types.
+The system's execution policy is specialized for the tag type to have no members,
+otherwise it has an alias for the tag type and can convert to the tag type.
+Therefore, the execution policy can always be converted to a tag (either by downcasting or by a conversion).
+
+Various systems now further extend this hierarchy of execution policies, or play other tricks.
+The ``cpp::execution_policy<Derived>`` inherits from ``sequential::execution_policy<Derived>`` for example
+(which then inherits from ``thrust::execution_policy``),
+so any dispatch to an algorithm in the cpp system may fall back to the sequential system.
+The cuda tag additionally inherits from ``allocator_aware_execution_policy`` to provide further functionality.
+
+
+Parallel and sequential policy
+******************************
+
+Each system also defines an internal parallel policy ``thrust::system::*::detail::par_t``.
+The cpp system for example:
+
+.. code-block:: c++
+
+    namespace thrust::system::cpp {
+      namespace detail {
+        struct par_t : execution_policy<par_t>, ... {};
+      }
+      inline constexpr detail::par_t par;
+    }
+
+These policies can be used by a user directly, to pick an execution order with a specific backend.
+Some systems also provide additional parallel execution policies,
+or member functions which can further configure a policy.
+The CUDA system for example also provides ``par_nosync_t``
+or can customize ``par_t`` by calling ``par.on(stream)``.
+In any case, the type passed to a Thrust algorithm will always be
+a class derived from the system's ``execution_policy`` class template.
+
+Thrust further defines a single sequential policy, ``thrust::seq``:
+
+.. code-block:: c++
+
+    namespace thrust {
+      namespace detail {
+        struct seq_t : system::detail::sequential::execution_policy<seq_t>, ... { ... };
+      }
+      inline constexpr detail::seq_t seq;
+    }
+
+which is a global constant of the execution policy to the sequential system.
+
+
+Host and device system policy
+*****************************
+
+Thrust additionally defines an active host and device system,
+which are selected by the macros ``THRUST_HOST_SYSTEM`` and ``THRUST_DEVICE_SYSTEM``
+and alias to the corresponding system parallel policies:
+
+.. code-block:: c++
+
+    namespace thrust {
+      namespace detail {
+        using host_t   = thrust::__THRUST_HOST_SYSTEM_NAMESPACE::detail::par_t;
+        using device_t = thrust::__THRUST_DEVICE_SYSTEM_NAMESPACE::detail::par_t;
+      }
+      inline constexpr detail::host_t host;
+      inline constexpr detail::device_t device;
+    }
+
+Users most often use ``thrust::host`` and ``thrust::device`` to dispatch to the current host or device system.
+
+
+Algorithm dispatch
+******************
+
+Each Thrust algorithm overload requires an execution policy to determine the backend to use.
+The policy can either be specified as a first argument by the user,
+or determined from the other arguments.
+We will focus on the first case for now.
+Let's take the public API entry point ``thrust::sort`` as an example.
+
+.. code-block:: c++
+
+    namespace thrust {
+      template <typename DerivedPolicy, typename RandomAccessIterator>
+      void sort(const thrust::detail::execution_policy_base<DerivedPolicy>& exec,
+                RandomAccessIterator first, RandomAccessIterator last);
+    }
+
+We can see that the first argument is a reference to ``execution_policy_base``,
+the highest base class in the execution policy hierarchy
+that still carries compile-time information on the most derived type.
+This ensures that this overload is only selected, when the user passes a valid execution policy.
+For comparison, C++17 parallel algorithms use a plain template parameter for the execution policy
+and apply a constraint (SFINAE or requires).
+The reference is also ``const`` so users can pass a temporary execution policy object,
+which was just created at the call site.
+
+Let's have a look at the implementation of the public API entry point:
+
+.. code-block:: c++
+
+    namespace thrust {
+      template <typename DerivedPolicy, typename RandomAccessIterator>
+      void sort(const thrust::detail::execution_policy_base<DerivedPolicy>& exec,
+                RandomAccessIterator first, RandomAccessIterator last) {
+        using thrust::system::detail::generic::sort;
+        return sort(thrust::detail::derived_cast(thrust::detail::strip_const(exec)), first, last);
+      }
+    }
+
+We first bring the generic sort implementation from the generic system into scope.
+Then, we strip away ``const`` and cast the reference to the execution policy to the most derived type,
+and perform an unqualified call to ``sort`` with the same arguments apart from the execution policy.
+
+We have previously seen that execution policies form deeper inheritance chains,
+and some systems inherit the policy of other systems (e.g. the cpp system inherits the sequential system).
+The ``derived_cast`` makes sure we perform ADL (argument dependent lookup) using the most specialized execution policy
+when we try to find overloads of ``sort``.
+It's also necessary, because when an execution policy is passed to the public API,
+it binds to the reference of its base class ``execution_policy_base``,
+for which no backend system exists,
+so we have to bring the type down again the inheritance chain.
+
+ADL will find a set of overloads for ``sort`` depending on the type of the execution policy.
+This set will at least include ``sort`` from the generic system and the API entry point itself.
+In case of the cpp system, it will also find ``sort`` from the sequential and cpp system.
+The compiler then ranks the overloads and the best match is the overload from the most specialized execution policy.
+
+This is neat, because a system does not need to provide implementations of all algorithms.
+It can just fall back to a generic implementation (falling back to a different algorithm),
+or to an implementation from a different system.
+For example, ``thrust::count`` is not implemented in the cpp system,
+so it falls back to the generic implementation, which uses ``thrust::count_if``.
+That's also not implemented in the cpp system, so it falls back again to the generic system,
+which then implements it via ``thrust::transform_reduce``, and so on.
+As a different example, ``thrust::copy`` for the cpp system brings in the include of the sequential copy implementation,
+so ADL will find it and prefer it over the generic implementation.
+
+Any generic algorithm is always outranked by a system specific implementation
+due to the inheritance chain of execution policies.
+Let's look at the generic sort implementation's interface:
+
+.. code-block:: c++
+
+    namespace thrust::system::detail::generic {
+      template <typename DerivedPolicy, typename RandomAccessIterator>
+      void sort(thrust::execution_policy<DerivedPolicy>& exec,
+                RandomAccessIterator first, RandomAccessIterator last);
+    }
+
+Notice that it takes the execution policy argument as ``thrust::execution_policy``,
+which is derived from ``thrust::detail::execution_policy_base``, which appears in the public API.
+This is why any overload in the generic system will always outrank the public API entry point.
+
+
+System selection
+****************
+
+Thrust also provides overloads of most algorithms without an execution policy,
+in which case the execution policy is determined based on the remaining arguments, usually iterators.
+Let's look at ``thrust::adjacent_difference``:
+
+.. code-block:: c++
+
+    namespace thrust {
+      template <typename InputIterator, typename OutputIterator>
+      OutputIterator adjacent_difference(InputIterator first, InputIterator last, OutputIterator result) {
+        using system::detail::generic::select_system;
+        using System1 = iterator_system_t<InputIterator>;
+        using System2 = iterator_system_t<OutputIterator>;
+        System1 system1;
+        System2 system2;
+        return thrust::adjacent_difference(select_system(system1, system2), first, last, result);
+      }
+    }
+
+Such an API is implemented by first bringing ``select_system`` from the generic system into scope.
+Then, we determine the system types associated with all iterator types via ``thrust::iterator_system``,
+and instantiate these systems.
+We then select one of these systems and pass the it to the corresponding overload of ``adjacent_difference``,
+taking an execution policy as first argument.
+Notice that this call is qualified with ``thrust::``, so ADL is not used here.
+The dispatch to the correct system will be performed in the called overload of ``adjacent_difference``.
+
+``select_system`` is implemented in the generic system, but no other Thrust system provides a different version of it.
+However, since users can define their own systems,
+they could also provide a different algorithm for selecting between multiple systems.
+The generic implementation will select the system to which all other systems are convertible
+(this is called the minimum system).
+If we remember how execution policies and tags are defined,
+they form inheritance hierarchies and tags have conversion operators,
+so those play a role here.
+``select_system`` may not find a minimum system,
+in which case it returns ``thrust::detail::unrelated_systems<System1, System2, ...>``,
+which usually fails to find an overload via ADL and lead to a compilation error.
diff --git a/cccl_upstream/docs/thrust/developer_overview.rst b/cccl_upstream/docs/thrust/developer_overview.rst
new file mode 100644
index 00000000..84affd7f
--- /dev/null
+++ b/cccl_upstream/docs/thrust/developer_overview.rst
@@ -0,0 +1,11 @@
+Developer Overview
+##################
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   developer/cmake_options
+   developer/systems
+
+This living document serves as a guide to the design of the internal structure of Thrust.
diff --git a/cccl_upstream/docs/thrust/function_objects.rst b/cccl_upstream/docs/thrust/function_objects.rst
new file mode 100644
index 00000000..f1b94fe7
--- /dev/null
+++ b/cccl_upstream/docs/thrust/function_objects.rst
@@ -0,0 +1,61 @@
+.. _thrust-module-api-function-objects:
+
+Function Objects
+=================
+
+.. toctree::
+   :glob:
+   :maxdepth: 2
+
+   function_objects/adaptors
+   function_objects/placeholder
+   function_objects/predefined
+
+.. _address-stability:
+
+Copyable arguments
+------------------
+
+The C++ language allows to take the address of a parameter and depend on this value for the correctness of a code.
+Consider this example:
+
+.. code-block:: cpp
+
+    const int n = 10;
+    thrust::device_vector<int> a(n, 1);
+    thrust::device_vector<int> b(n);
+    int* a_ptr = thrust::raw_pointer_cast(a.data());
+    int* b_ptr = thrust::raw_pointer_cast(b.data());
+    thrust::transform(thrust::device, a.begin(), a.end(), a.begin(),
+        [a_ptr, b_ptr](const int& e) {
+            const auto i = &e - a_ptr; // &e expected to point into global memory
+            return e + b_ptr[i];
+        });
+
+Here, :code:`thrust::transform` is invoked on the range of elements in :code:`a`.
+The lambda function computes the index :code:`i` based on the start of the buffer held by :code:`a`
+and the address of the parameter :code:`e`,
+thus assuming that the reference :code:`e` points into the same memory block that :code:`a` holds,
+e.g., global memory for the CUDA system.
+
+While this example is contrived, such uses of Thrust exist and are currently valid.
+We strongly urge users though to not rely on parameter addresses,
+and we reserve the right to disallow this guarantee in the future.
+
+Relying on the address of a parameter constrains the internal implementation
+to serve the arguments to the callable directly from the input buffer,
+which inhibits optimizations, like bulk copies or vectorized loads.
+To permit the implementation to take advantage of such features,
+a function object can be marked using :code:`proclaim_copyable_arguments`:
+
+.. code-block:: cpp
+
+    thrust::transform(thrust::device, a.begin(), a.end(), a.begin(),
+        cuda::std::proclaim_copyable_arguments([](const int& a, const int& b) {
+            return a + b;
+        }));
+
+Wrapping a function object in :code:`proclaim_copyable_arguments` will attach a marker that the implementation can detect,
+and use for optimization.
+Many function objects in libcu++, CUB and Thrust are marked by default,
+but it does not hurt to mark them explicitly.
diff --git a/cccl_upstream/docs/thrust/function_objects/adaptors.rst b/cccl_upstream/docs/thrust/function_objects/adaptors.rst
new file mode 100644
index 00000000..5e5cec96
--- /dev/null
+++ b/cccl_upstream/docs/thrust/function_objects/adaptors.rst
@@ -0,0 +1,6 @@
+.. _thrust-module-api-function-objects-adaptors:
+
+Function Object Adaptors
+-------------------------
+
+  - :cpp:class:`thrust::zip_function <thrust::zip_function>`
diff --git a/cccl_upstream/docs/thrust/function_objects/placeholder.rst b/cccl_upstream/docs/thrust/function_objects/placeholder.rst
new file mode 100644
index 00000000..1c396543
--- /dev/null
+++ b/cccl_upstream/docs/thrust/function_objects/placeholder.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-function-objects-placeholder:
+
+Placeholder Objects
+--------------------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../api/*placeholder*
diff --git a/cccl_upstream/docs/thrust/function_objects/predefined.rst b/cccl_upstream/docs/thrust/function_objects/predefined.rst
new file mode 100644
index 00000000..290acc59
--- /dev/null
+++ b/cccl_upstream/docs/thrust/function_objects/predefined.rst
@@ -0,0 +1,11 @@
+.. _thrust-module-api-function-objects-predefined:
+
+Function Objects
+----------------
+
+Users are encouraged to use the standard C++ function objects available from `<cuda/std/functional>`.
+Thrust provides the following additional function objects:
+
+  - :cpp:struct:`thrust::square <thrust::square>`
+  - :cpp:struct:`thrust::project1st <thrust::project1st>`
+  - :cpp:struct:`thrust::project2nd <thrust::project2nd>`
diff --git a/cccl_upstream/docs/thrust/index.rst b/cccl_upstream/docs/thrust/index.rst
new file mode 100644
index 00000000..fd844504
--- /dev/null
+++ b/cccl_upstream/docs/thrust/index.rst
@@ -0,0 +1,114 @@
+.. _thrust-module:
+
+Thrust
+======
+
+.. toctree::
+   :hidden:
+   :maxdepth: 2
+
+   Overview <self>
+   algorithms
+   containers
+   function_objects
+   iterators
+   memory_management
+   numerics
+   parallel_execution_policies
+   random
+   system
+   utility
+   API reference <api/index>
+   developer_overview
+
+Thrust is the C++ parallel algorithms library which inspired the introduction of parallel algorithms to the
+C++ Standard Library. Thrust's **high-level** interface greatly enhances programmer **productivity** while
+enabling performance portability between GPUs and multicore CPUs.
+It builds on top of established parallel programming frameworks (such as CUDA, TBB, and OpenMP).
+It also provides a number of general-purpose facilities similar to those found in the C++ Standard Library.
+
+Thrust is an open source project; it is available on `GitHub <https://github.com/NVIDIA/cccl>`__
+and included in the NVIDIA HPC SDK and CUDA Toolkit.
+If you have one of those SDKs installed, no additional installation or compiler flags are needed to use Thrust.
+
+Examples
+--------
+
+Thrust is best learned through examples.
+
+-------------
+CMake Example
+-------------
+
+A complete, standalone example project showing how to write a CMake build system that uses Thrust with any supported
+device system is available in the CCCL repository `here <https://github.com/NVIDIA/cccl/tree/main/examples/thrust_flexible_device_system>`__.
+
+------------------
+Thrust API Example
+------------------
+
+The following example generates random numbers serially and then transfers them
+  to a parallel device where they are sorted.
+
+.. code:: cpp
+
+   #include <thrust/host_vector.h>
+   #include <thrust/device_vector.h>
+   #include <thrust/generate.h>
+   #include <thrust/sort.h>
+   #include <thrust/copy.h>
+   #include <thrust/random.h>
+
+   int main() {
+   // Generate 32M random numbers serially.
+   thrust::default_random_engine rng(1337);
+   thrust::uniform_int_distribution<int> dist;
+   thrust::host_vector<int> h_vec(32 << 20);
+   thrust::generate(h_vec.begin(), h_vec.end(), [&] { return dist(rng); });
+
+   // Transfer data to the device.
+   thrust::device_vector<int> d_vec = h_vec;
+
+   // Sort data on the device.
+   thrust::sort(d_vec.begin(), d_vec.end());
+
+   // Transfer data back to host.
+   thrust::copy(d_vec.begin(), d_vec.end(), h_vec.begin());
+   }
+
+
+`See it on Godbolt <https://godbolt.org/z/GeWEd8Er9>`__
+
+This example demonstrates computing the sum of some random numbers in parallel:
+
+.. code:: cpp
+
+   #include <thrust/host_vector.h>
+   #include <thrust/device_vector.h>
+   #include <thrust/generate.h>
+   #include <thrust/reduce.h>
+   #include <thrust/functional.h>
+   #include <thrust/random.h>
+
+   int main() {
+   // Generate random data serially.
+   thrust::default_random_engine rng(1337);
+   thrust::uniform_real_distribution<double> dist(-50.0, 50.0);
+   thrust::host_vector<double> h_vec(32 << 20);
+   thrust::generate(h_vec.begin(), h_vec.end(), [&] { return dist(rng); });
+
+   // Transfer to device and compute the sum.
+   thrust::device_vector<double> d_vec = h_vec;
+   double x = thrust::reduce(d_vec.begin(), d_vec.end(), 0, thrust::plus<int>());
+   }
+
+`See it on Godbolt <https://godbolt.org/z/cnsbWWME7>`__
+
+Getting The Thrust Source Code & Developing Thrust
+-----------------------------------------------------
+
+Thrust started as a stand-alone project, but as of March 2024 Thrust is a part of the
+CUDA Core Compute Libraries (CCCL). Please refer to the
+`CCCL Getting Started section <https://github.com/NVIDIA/cccl?tab=readme-ov-file#getting-started>`__ and the
+`Contributing Guide: <https://github.com/NVIDIA/cccl/blob/main/CONTRIBUTING.md>`__ for instructions on how to
+get started developing the CCCL sources.
diff --git a/cccl_upstream/docs/thrust/iterators.rst b/cccl_upstream/docs/thrust/iterators.rst
new file mode 100644
index 00000000..8178573c
--- /dev/null
+++ b/cccl_upstream/docs/thrust/iterators.rst
@@ -0,0 +1,88 @@
+.. _thrust-module-api-iterators:
+
+Iterators
+===========
+
+Thrust provides a rich collection of iterators that extend beyond the C++ standard library.
+
+.. important::
+
+    Many Thrust iterators have direct replacements in libcu++,
+    which are newer and thus fix some of the shortcomings of Thrust's iterators.
+    They should generally be preferred. See :ref:`libcudacxx-extended-api-iterators` for more.
+
+
+Fancy iterators
+-------------------------
+
+Thrust provides the following fancy iterators:
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   api/group__fancyiterator_*
+
+
+Iterator traits
+-------------------------
+
+Thrust offers the following traits to inspect iterators:
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   api/group__iterator__traits_*
+
+Iterator tags
+-------------------------
+
+Thrust provides the following iterator category tags:
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   api/group__iterator__tags_*
+
+Additionally, the following utilities for working with iterator category tags are provided:
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   api/group__iterator__tag__utilities_*
+
+Thrust provides the following iterator traversal tags:
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   api/structthrust*__traversal__tag
+
+
+Iterator model
+-------------------------
+
+Thrust's iterators extend the C++ standard library iterator model and were inspired by Boost.Iterator.
+In the C++ standard library, iterators are mostly distinguished by their iterator category (before C++20),
+and their iterator concept (since C++20).
+This is governed by the nested ``iterator_category`` and ``iterator_concept`` types, respectively,
+which in standard C++ are one of:
+
+- ``input_iterator_tag``
+- ``output_iterator_tag``
+- ``forward_iterator_tag``
+- ``bidirectional_iterator_tag``
+- ``random_access_iterator_tag``
+- ``contiguous_iterator_tag`` (since C++20)
+
+Thrust extends this model by introducing the notion of an iterator traversal and an iterator system.
+In order to fit into the existing schema of iterators,
+it encodes this additional information into the ``iterator_category``,
+which is no longer one of the standard tags listed above,
+but a templated type carrying the iterator tag, iterator traversal and iterator system.
+This information can be extracted using the :cpp:class:`thrust::iterator_traversal <thrust::iterator_traversal>`
+and :cpp:class:`thrust::iterator_system <thrust::iterator_system>` traits.
diff --git a/cccl_upstream/docs/thrust/memory_management.rst b/cccl_upstream/docs/thrust/memory_management.rst
new file mode 100644
index 00000000..8c671a65
--- /dev/null
+++ b/cccl_upstream/docs/thrust/memory_management.rst
@@ -0,0 +1,12 @@
+.. _thrust-module-api-memory-management:
+
+Memory Management
+==================
+
+.. toctree::
+   :glob:
+   :maxdepth: 2
+
+   memory_management/allocators
+   memory_management/memory_resources
+   api/group__memory__management*
diff --git a/cccl_upstream/docs/thrust/memory_management/allocators.rst b/cccl_upstream/docs/thrust/memory_management/allocators.rst
new file mode 100644
index 00000000..cf6945cc
--- /dev/null
+++ b/cccl_upstream/docs/thrust/memory_management/allocators.rst
@@ -0,0 +1,18 @@
+.. _thrust-module-api-memory-management-allocators:
+
+Allocators
+-----------
+
+  - :cpp:class:`thrust::device_allocator <thrust::device_allocator>`
+  - :cpp:class:`thrust::device_malloc_allocator <thrust::device_malloc_allocator>`
+  - :cpp:class:`thrust::device_new_allocator <thrust::device_new_allocator>`
+  - :cpp:class:`thrust::universal_allocator <thrust::universal_allocator>`
+  - :cpp:class:`thrust::universal_host_pinned_allocator <thrust::universal_host_pinned_allocator>`
+  - :cpp:class:`thrust::mr::allocator <thrust::mr::allocator>`
+  - :cpp:class:`thrust::mr::stateless_resource_allocator <thrust::mr::stateless_resource_allocator>`
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../api/group__allocators*
diff --git a/cccl_upstream/docs/thrust/memory_management/memory_resources.rst b/cccl_upstream/docs/thrust/memory_management/memory_resources.rst
new file mode 100644
index 00000000..5676144b
--- /dev/null
+++ b/cccl_upstream/docs/thrust/memory_management/memory_resources.rst
@@ -0,0 +1,18 @@
+.. _thrust-module-api-memory-management-memory-resources:
+
+Memory Resources
+-----------------
+
+  - :cpp:class:`thrust::mr::disjoint_unsynchronized_pool_resource <thrust::mr::disjoint_unsynchronized_pool_resource>`
+  - :cpp:struct:`thrust::mr::disjoint_synchronized_pool_resource <thrust::mr::disjoint_synchronized_pool_resource>`
+  - :cpp:class:`thrust::mr::memory_resource <thrust::mr::memory_resource>`
+  - :cpp:class:`thrust::mr::new_delete_resource <thrust::mr::new_delete_resource>`
+  - :cpp:class:`thrust::mr::unsynchronized_pool_resource <thrust::mr::unsynchronized_pool_resource>`
+  - :cpp:struct:`thrust::mr::pool_options <thrust::mr::pool_options>`
+  - :cpp:struct:`thrust::mr::synchronized_pool_resource <thrust::mr::synchronized_pool_resource>`
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../api/group__memory__resources*
diff --git a/cccl_upstream/docs/thrust/numerics.rst b/cccl_upstream/docs/thrust/numerics.rst
new file mode 100644
index 00000000..7855ebc5
--- /dev/null
+++ b/cccl_upstream/docs/thrust/numerics.rst
@@ -0,0 +1,12 @@
+.. _thrust-module-api-numerics:
+
+Numerics
+========
+
+  - :cpp:struct:`thrust::complex <thrust::complex>`
+
+.. toctree::
+   :glob:
+   :maxdepth: 2
+
+   api/group__complex*
diff --git a/cccl_upstream/docs/thrust/parallel_execution_policies.rst b/cccl_upstream/docs/thrust/parallel_execution_policies.rst
new file mode 100644
index 00000000..5988e21c
--- /dev/null
+++ b/cccl_upstream/docs/thrust/parallel_execution_policies.rst
@@ -0,0 +1,13 @@
+.. _thrust-module-api-parallel-execution-policies:
+
+Parallel Execution Policies
+===========================
+
+  - :cpp:struct:`thrust::host_execution_policy <thrust::host_execution_policy>`
+  - :cpp:struct:`thrust::device_execution_policy <thrust::device_execution_policy>`
+
+.. toctree::
+   :glob:
+   :maxdepth: 2
+
+   api/group__execution__policies*
diff --git a/cccl_upstream/docs/thrust/random.rst b/cccl_upstream/docs/thrust/random.rst
new file mode 100644
index 00000000..e21cb7c3
--- /dev/null
+++ b/cccl_upstream/docs/thrust/random.rst
@@ -0,0 +1,13 @@
+.. _thrust-module-api-random-number-generators:
+
+Random Number Generators
+========================
+
+.. toctree::
+   :glob:
+   :maxdepth: 2
+
+   random/predefined
+   random/distributions
+   random/engine
+   random/engine_adaptors
diff --git a/cccl_upstream/docs/thrust/random/distributions.rst b/cccl_upstream/docs/thrust/random/distributions.rst
new file mode 100644
index 00000000..c3c7a04c
--- /dev/null
+++ b/cccl_upstream/docs/thrust/random/distributions.rst
@@ -0,0 +1,14 @@
+.. _thrust-module-api-random-number-generators-distributions:
+
+Random Number Distributions
+----------------------------
+
+  - :cpp:class:`thrust::random::normal_distribution <thrust::random::normal_distribution>`
+  - :cpp:class:`thrust::random::uniform_int_distribution <thrust::random::uniform_int_distribution>`
+  - :cpp:class:`thrust::random::uniform_real_distribution <thrust::random::uniform_real_distribution>`
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../api/group__random__number__distributions*
diff --git a/cccl_upstream/docs/thrust/random/engine.rst b/cccl_upstream/docs/thrust/random/engine.rst
new file mode 100644
index 00000000..f9370a9e
--- /dev/null
+++ b/cccl_upstream/docs/thrust/random/engine.rst
@@ -0,0 +1,14 @@
+.. _thrust-module-api-random-number-generators-engines:
+
+Random Number Engines
+----------------------
+
+  - :cpp:class:`thrust::random::linear_congruential_engine <thrust::random::linear_congruential_engine>`
+  - :cpp:class:`thrust::random::linear_feedback_shift_engine <thrust::random::linear_feedback_shift_engine>`
+  - :cpp:class:`thrust::random::subtract_with_carry_engine <thrust::random::subtract_with_carry_engine>`
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../api/group__random__number__engine__templates*
diff --git a/cccl_upstream/docs/thrust/random/engine_adaptors.rst b/cccl_upstream/docs/thrust/random/engine_adaptors.rst
new file mode 100644
index 00000000..952fe101
--- /dev/null
+++ b/cccl_upstream/docs/thrust/random/engine_adaptors.rst
@@ -0,0 +1,13 @@
+.. _thrust-module-api-random-number-generators-engine-adaptors:
+
+Random Number Engine Adaptors
+------------------------------
+
+  - :cpp:class:`thrust::random::discard_block_engine <thrust::random::discard_block_engine>`
+  - :cpp:class:`thrust::random::xor_combine_engine <thrust::random::xor_combine_engine>`
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../api/group__random__number__engine__adaptors*
diff --git a/cccl_upstream/docs/thrust/random/predefined.rst b/cccl_upstream/docs/thrust/random/predefined.rst
new file mode 100644
index 00000000..d2053f3e
--- /dev/null
+++ b/cccl_upstream/docs/thrust/random/predefined.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-random-number-generators-predefined:
+
+Random Number Engines with Predefined Parameters
+-------------------------------------------------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../api/group__predefined__random*
diff --git a/cccl_upstream/docs/thrust/system.rst b/cccl_upstream/docs/thrust/system.rst
new file mode 100644
index 00000000..cd6f3459
--- /dev/null
+++ b/cccl_upstream/docs/thrust/system.rst
@@ -0,0 +1,11 @@
+.. _thrust-module-api-system:
+
+System
+========
+
+.. toctree::
+   :glob:
+   :maxdepth: 2
+
+   system/systems
+   system/diagnostics
diff --git a/cccl_upstream/docs/thrust/system/diagnostics.rst b/cccl_upstream/docs/thrust/system/diagnostics.rst
new file mode 100644
index 00000000..29f4727b
--- /dev/null
+++ b/cccl_upstream/docs/thrust/system/diagnostics.rst
@@ -0,0 +1,17 @@
+.. _thrust-module-api-system-diagnostics:
+
+System Diagnostics
+----------------------------
+
+  - :cpp:struct:`thrust::system::is_error_code_enum <thrust::system::is_error_code_enum>`
+  - :cpp:struct:`thrust::system::is_error_condition_enum <thrust::system::is_error_condition_enum>`
+  - :cpp:class:`thrust::system::error_category <thrust::system::error_category>`
+  - :cpp:class:`thrust::system::error_code <thrust::system::error_code>`
+  - :cpp:class:`thrust::system::error_condition <thrust::system::error_condition>`
+  - :cpp:class:`thrust::system::system_error <thrust::system::system_error>`
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../api/group__system__diagnostics*
diff --git a/cccl_upstream/docs/thrust/system/systems.rst b/cccl_upstream/docs/thrust/system/systems.rst
new file mode 100644
index 00000000..e57eaac1
--- /dev/null
+++ b/cccl_upstream/docs/thrust/system/systems.rst
@@ -0,0 +1,10 @@
+.. _thrust-module-api-system-systems:
+
+Systems
+----------------------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../api/group__system__backends*
diff --git a/cccl_upstream/docs/thrust/utility.rst b/cccl_upstream/docs/thrust/utility.rst
new file mode 100644
index 00000000..7ab78627
--- /dev/null
+++ b/cccl_upstream/docs/thrust/utility.rst
@@ -0,0 +1,12 @@
+.. _thrust-module-api-utility:
+
+Utility
+========
+
+.. toctree::
+   :glob:
+   :maxdepth: 2
+
+   utility/pair
+   utility/tuple
+   utility/type_traits
diff --git a/cccl_upstream/docs/thrust/utility/pair.rst b/cccl_upstream/docs/thrust/utility/pair.rst
new file mode 100644
index 00000000..5da75ae0
--- /dev/null
+++ b/cccl_upstream/docs/thrust/utility/pair.rst
@@ -0,0 +1,14 @@
+.. _thrust-module-api-utility-pair:
+
+Pair
+----
+
+Thrust provides an implementations of `std::pair`, that is pulled in from `libcu++`.
+
+It is recommended to replace `thrust::pair` and `thrust::make_pair` with `cuda::std::pair` and `cuda::std::make_pair`.
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../api/*pair*
diff --git a/cccl_upstream/docs/thrust/utility/tuple.rst b/cccl_upstream/docs/thrust/utility/tuple.rst
new file mode 100644
index 00000000..d0e07830
--- /dev/null
+++ b/cccl_upstream/docs/thrust/utility/tuple.rst
@@ -0,0 +1,14 @@
+.. _thrust-module-api-utility-tuple:
+
+Tuple
+------
+
+Thrust provides an implementation of `std::tuple`, that is pulled in from `libcu++`.
+
+It is recommended to replace `thrust::tuple` with `cuda::std::tuple`.
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../api/*tuple*
diff --git a/cccl_upstream/docs/thrust/utility/type_traits.rst b/cccl_upstream/docs/thrust/utility/type_traits.rst
new file mode 100644
index 00000000..ee30cd7b
--- /dev/null
+++ b/cccl_upstream/docs/thrust/utility/type_traits.rst
@@ -0,0 +1,11 @@
+.. _thrust-module-api-utility-type-traits:
+
+Type Traits
+-----------
+
+.. toctree::
+   :glob:
+   :maxdepth: 1
+
+   ../api/struct*proclaim__contiguous__iterator*
+   ../api/group__type__traits*
diff --git a/cccl_upstream/pyproject.toml b/cccl_upstream/pyproject.toml
new file mode 100644
index 00000000..cb69013d
--- /dev/null
+++ b/cccl_upstream/pyproject.toml
@@ -0,0 +1,27 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION.
+[tool.mypy]
+cache_dir = "./.cache/mypy"
+
+[tool.ruff]
+target-version = "py310"
+fix = true
+show-fixes = true
+exclude = ["docs/tools"]
+cache-dir = "./.cache/ruff"
+
+[tool.ruff.lint]
+extend-select = ["I"]
+
+[tool.codespell]
+# To run codespell interactively and fix errors that pre-commit reports, try
+# `codespell -i 3 -w -H`. This will run with interactive review (-i 3), writes
+# changes to disk (-w), and includes hidden files (-H).
+# Note: pre-commit passes explicit lists of files here, which this skip file
+# list doesn't override - the skip list is only to allow you to run codespell
+# interactively.
+skip = "./.git,./build,./CITATION.md"
+# ignore short words, and typename parameters like OffsetT
+ignore-regex = "\\b(.{1,4}|[A-Z]\\w*T)\\b"
+ignore-words-list = "inout,imovable,optionN,aCount,quitted,Invocable,countr,unexpect,numer,euclidian,couldn,OffsetT,FromM,Collet,NotIn"
+builtin = "clear"
+quiet-level = 3
diff --git a/cccl_upstream/python/cuda_cccl/.gitignore b/cccl_upstream/python/cuda_cccl/.gitignore
new file mode 100644
index 00000000..ddb73813
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/.gitignore
@@ -0,0 +1,42 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+build.log
+.python-version
+
+# CMake
+CMakeFiles/
+CMakeCache.txt
+cmake_install.cmake
+Makefile
+*.cmake
+!CMakeLists.txt
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# Generated by CMake at build time
+cuda/compute/_build_info.py
+cuda/compute/cu12/
+cuda/compute/cu13/
diff --git a/cccl_upstream/python/cuda_cccl/CMakeLists.txt b/cccl_upstream/python/cuda_cccl/CMakeLists.txt
new file mode 100644
index 00000000..c222af52
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/CMakeLists.txt
@@ -0,0 +1,187 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+cmake_minimum_required(VERSION 3.30)
+
+# Must be set before project() initializes the CUDA language; otherwise CMake
+# < 3.23 defaults to sm_52, which is below CCCL's minimum supported arch.
+include(../../cmake/CCCLCheckCudaArchitectures.cmake)
+set(
+  CMAKE_CUDA_ARCHITECTURES
+  "${minimum_cccl_arch}"
+  CACHE STRING
+  "CUDA architectures for CCCL"
+)
+
+project(cuda_cccl DESCRIPTION "Python package cuda_cccl" LANGUAGES CUDA CXX C)
+
+find_package(CUDAToolkit REQUIRED)
+
+set(CUDA_VERSION_MAJOR ${CUDAToolkit_VERSION_MAJOR})
+set(CUDA_VERSION_DIR "cu${CUDA_VERSION_MAJOR}")
+message(
+  STATUS
+  "Building for CUDA ${CUDA_VERSION_MAJOR}, output directory: ${CUDA_VERSION_DIR}"
+)
+
+# Build cuda_cccl against either cccl.c.parallel (v1, NVRTC) by default or
+# cccl.c.parallel.v2 (HostJIT) when CCCL_PYTHON_USE_V2=ON. v2 is opt-in until
+# it replaces v1 across the matrix.
+set(_cccl_root ../..)
+set(CCCL_TOPLEVEL_PROJECT ON) # Enable the developer builds
+option(
+  CCCL_PYTHON_USE_V2
+  "Build cuda_cccl against cccl.c.parallel.v2 (HostJIT)."
+  OFF
+)
+if (CCCL_PYTHON_USE_V2)
+  set(CCCL_ENABLE_C_PARALLEL_V2 ON)
+  set(CCCL_C_PARALLEL_V2_LIBRARY_OUTPUT_DIRECTORY ${SKBUILD_PROJECT_NAME})
+  set(_cccl_c_parallel_target cccl.c.parallel.v2)
+  set(_using_v2_py "True")
+else()
+  set(CCCL_ENABLE_C_PARALLEL ON)
+  set(CCCL_C_PARALLEL_LIBRARY_OUTPUT_DIRECTORY ${SKBUILD_PROJECT_NAME})
+  set(_cccl_c_parallel_target cccl.c.parallel)
+  set(_using_v2_py "False")
+endif()
+
+# Surface the v1/v2 choice to Python (tests use it to skip v2-only failures,
+# and __init__.py uses it to wire up wheel-bundled hostjit header paths).
+# Generated into the build dir and installed via CMake — writing into the
+# source tree would miss scikit-build-core's package-file snapshot.
+set(_build_info_py "${CMAKE_CURRENT_BINARY_DIR}/_build_info.py")
+file(
+  WRITE "${_build_info_py}"
+  "# Auto-generated by CMakeLists.txt; do not edit.\nUSING_V2 = ${_using_v2_py}\n"
+)
+install(FILES "${_build_info_py}" DESTINATION cuda/compute)
+# Just install the rest:
+set(libcudacxx_ENABLE_INSTALL_RULES ON)
+set(CUB_ENABLE_INSTALL_RULES ON)
+set(Thrust_ENABLE_INSTALL_RULES ON)
+# Install to our output location:
+include(GNUInstallDirs)
+set(old_libdir "${CMAKE_INSTALL_LIBDIR}") # push
+set(old_includedir "${CMAKE_INSTALL_INCLUDEDIR}") # push
+set(CMAKE_INSTALL_LIBDIR "cuda/cccl/headers/lib")
+set(CMAKE_INSTALL_INCLUDEDIR "cuda/cccl/headers/include")
+add_subdirectory(${_cccl_root} _parent_cccl)
+set(CMAKE_INSTALL_LIBDIR "${old_libdir}") # pop
+set(CMAKE_INSTALL_INCLUDEDIR "${old_includedir}") # pop
+
+# Install version-specific binaries
+set(_cccl_c_parallel_install_targets ${_cccl_c_parallel_target})
+if (CCCL_PYTHON_USE_V2)
+  list(APPEND _cccl_c_parallel_install_targets libnvcc)
+endif()
+install(
+  TARGETS ${_cccl_c_parallel_install_targets}
+  DESTINATION cuda/compute/${CUDA_VERSION_DIR}/cccl
+)
+
+# Build and install Cython extension
+find_package(Python3 COMPONENTS Interpreter Development.Module REQUIRED)
+
+set(CYTHON_version_command "${Python3_EXECUTABLE}" -m cython --version)
+execute_process(
+  COMMAND ${CYTHON_version_command}
+  OUTPUT_VARIABLE CYTHON_version_output
+  ERROR_VARIABLE CYTHON_version_output
+  OUTPUT_STRIP_TRAILING_WHITESPACE
+  ERROR_STRIP_TRAILING_WHITESPACE
+  COMMAND_ERROR_IS_FATAL ANY
+)
+
+if ("${CYTHON_version_output}" MATCHES "^[Cc]ython version ([^,]+)")
+  set(CYTHON_VERSION "${CMAKE_MATCH_1}")
+else()
+  message(
+    FATAL_ERROR
+    "Failed to parse Cython version from:\n${CYTHON_version_output}"
+  )
+endif()
+
+# -3 generates source for Python 3
+# -M generates depfile
+# -t cythonizes if PYX is newer than preexisting output
+# -w sets working directory
+set(
+  CYTHON_FLAGS
+  -3
+  -M
+  -t
+  -w
+  "${cuda_cccl_SOURCE_DIR}"
+)
+
+message(STATUS "Using Cython ${CYTHON_VERSION}")
+set(pyx_source_file "${cuda_cccl_SOURCE_DIR}/cuda/compute/_bindings_impl.pyx")
+
+set(_generated_extension_src "${cuda_cccl_BINARY_DIR}/_bindings_impl.c")
+set(_depfile "${cuda_cccl_BINARY_DIR}/_bindings_impl.c.dep")
+
+# Backend-conditional Cython .pxi files. Where v1 and v2 expose different
+# struct layouts or call signatures, the .pyx `include`s a generated .pxi
+# whose source is chosen here. The helpers inside present a uniform interface
+# so the rest of _bindings_impl.pyx stays backend-agnostic.
+if (CCCL_PYTHON_USE_V2)
+  set(_backend_suffix "v2")
+else()
+  set(_backend_suffix "v1")
+endif()
+foreach (
+  _pxi_stem
+  segmented_reduce_backend
+  binary_search_backend
+  op_code_type
+  serialization
+)
+  configure_file(
+    "${CMAKE_CURRENT_SOURCE_DIR}/cuda/compute/_bindings_${_pxi_stem}_${_backend_suffix}.pxi"
+    "${CMAKE_CURRENT_BINARY_DIR}/_bindings_${_pxi_stem}.pxi"
+    COPYONLY
+  )
+endforeach()
+
+# Custom Cython compilation command. `-I ${BINARY_DIR}` lets the .pyx's
+# `include "_bindings_..._backend.pxi"` resolve to the file we configured
+# above.
+add_custom_command(
+  OUTPUT "${_generated_extension_src}"
+  COMMAND
+    "${Python3_EXECUTABLE}" -m cython
+    # gersemi: off
+    ${CYTHON_FLAGS}
+    -I "${CMAKE_CURRENT_BINARY_DIR}"
+    "${pyx_source_file}"
+    --output-file "${_generated_extension_src}"
+  # gersemi: on
+  DEPENDS "${pyx_source_file}"
+  DEPFILE "${_depfile}"
+  COMMENT "Cythonizing ${pyx_source_file} for CUDA ${CUDA_VERSION_MAJOR}"
+)
+
+add_custom_target(
+  cythonize_bindings_impl
+  ALL
+  DEPENDS "${_generated_extension_src}"
+)
+
+python3_add_library(
+  _bindings_impl
+  MODULE
+  WITH_SOABI
+  "${_generated_extension_src}"
+)
+add_dependencies(_bindings_impl cythonize_bindings_impl)
+target_link_libraries(
+  _bindings_impl
+  PRIVATE #
+    ${_cccl_c_parallel_target}
+    CUDA::cuda_driver
+)
+set_target_properties(_bindings_impl PROPERTIES INSTALL_RPATH "$ORIGIN/cccl")
+
+install(TARGETS _bindings_impl DESTINATION cuda/compute/${CUDA_VERSION_DIR})
diff --git a/cccl_upstream/python/cuda_cccl/LICENSE b/cccl_upstream/python/cuda_cccl/LICENSE
new file mode 100644
index 00000000..fd50777c
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/LICENSE
@@ -0,0 +1 @@
+../../LICENSE
diff --git a/cccl_upstream/python/cuda_cccl/README.md b/cccl_upstream/python/cuda_cccl/README.md
new file mode 100644
index 00000000..f7a6a2bd
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/README.md
@@ -0,0 +1,49 @@
+# CUDA CCCL Python Package
+
+[`cuda.cccl`](https://nvidia.github.io/cccl/unstable/python)
+provides a Pythonic interface to the
+[CUDA Core Compute Libraries](https://nvidia.github.io/cccl/unstable/cpp.html#cccl-cpp-libraries).
+It provides the following modules:
+
+- **`cuda.compute`** - Device-level parallel algorithms (reduce, scan, sort, etc.) and iterators
+- **`cuda.cccl.headers`** - Programmatic access to CCCL headers
+
+## Installation
+
+Install from PyPI:
+
+```bash
+pip install cuda-cccl[cu13]  # For CUDA 13.x (pip-installed cuda-toolkit)
+pip install cuda-cccl[cu12]  # For CUDA 12.x (pip-installed cuda-toolkit)
+```
+
+If you already have a CUDA toolkit on your system and do not want pip to
+install it, use the `sysctk` variants:
+
+```bash
+pip install cuda-cccl[sysctk13]  # For CUDA 13.x (system CUDA toolkit)
+pip install cuda-cccl[sysctk12]  # For CUDA 12.x (system CUDA toolkit)
+```
+
+For a minimal install without Numba (useful when supplying pre-compiled operators):
+
+```bash
+pip install cuda-cccl[minimal-cu13]      # pip-installed cuda-toolkit
+pip install cuda-cccl[minimal-sysctk13]  # system CUDA toolkit
+```
+
+Install from conda-forge:
+
+```bash
+conda install -c conda-forge cccl-python
+```
+
+**Requirements:** Python 3.10+, CUDA Toolkit 12.x or 13.x, NVIDIA GPU with Compute Capability 7.5+
+
+## Documentation
+
+For complete documentation, examples, and API reference, visit:
+
+- **Full Documentation**: [nvidia.github.io/cccl/unstable/python](https://nvidia.github.io/cccl/unstable/python)
+- **Repository**: [github.com/NVIDIA/cccl](https://github.com/NVIDIA/cccl)
+- **Examples**: [github.com/NVIDIA/cccl/tree/main/python/cuda_cccl/tests/compute/examples](https://github.com/NVIDIA/cccl/tree/main/python/cuda_cccl/tests/compute/examples)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/.gitignore b/cccl_upstream/python/cuda_cccl/benchmarks/compute/.gitignore
new file mode 100644
index 00000000..64d654b1
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/.gitignore
@@ -0,0 +1,19 @@
+# Build artifacts
+build/
+bin/
+*.o
+*.so
+*.a
+compile_commands.json
+
+# Python cache
+__pycache__/
+*.pyc
+*.pyo
+
+# Benchmark results
+results*/
+
+# Pixi
+.pixi/
+pixi.lock
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/AGENTS.md b/cccl_upstream/python/cuda_cccl/benchmarks/compute/AGENTS.md
new file mode 100644
index 00000000..f3a9fd80
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/AGENTS.md
@@ -0,0 +1,12 @@
+# CCCL Python cuda.compute benchmarks
+
+This directory contains the code for the Python cuda.compute benchmarks.
+They are migrated from the original C++ benchmarks and they should match the C++ implementations
+as closely as possible.
+
+The original C++ benchmarks are available in this repository in: ../../../../cub/benchmarks/bench/
+We follow the same directory structure and naming conventions converting to Python were appropriate.
+
+The code for cuda.compute is in this repository under: `../../../../python/cuda_cccl/cuda/compute/`. Look into this directory when searching for existing APIs in Python.
+
+The benchmarks use nvbench to run the benchmarks and report the results.
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/README.md b/cccl_upstream/python/cuda_cccl/benchmarks/compute/README.md
new file mode 100644
index 00000000..c4377e51
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/README.md
@@ -0,0 +1,147 @@
+# cuda.compute Benchmarks
+
+Compare Python `cuda.compute` performance against C++ CUB implementations.
+
+## Setup
+
+This project uses [pixi](https://pixi.sh) to manage environments and dependencies.
+
+Two environments are available:
+
+- **`wheel`** - Uses the released `cuda-cccl` package
+- **`source`** - Builds `cuda-cccl` from the local repository
+
+### Build C++ Benchmarks
+
+Build CUB benchmarks using the CI script (one-time, ~13 minutes):
+
+```bash
+cd /path/to/cccl
+./ci/build_cub.sh -arch 89  # Use your GPU arch (89=RTX 4090, 80=A100, 90=H100)
+```
+
+Binaries are built to: `build/cub/bin/`
+
+## Run Benchmarks
+
+### Using pixi tasks
+
+```bash
+# Run Python benchmarks (released cuda-cccl)
+pixi run -e wheel bench
+
+# Run Python benchmarks (local source build)
+pixi run -e source bench
+
+# Run Python benchmarks with reduced parameter set
+pixi run -e wheel bench-quick
+
+# Run just one benchmark
+pixi run -e wheel bench -b transform/fill
+
+# Run C++ benchmarks
+pixi run -e wheel bench-cpp
+
+# Run both Python and C++ benchmarks
+pixi run -e wheel bench-all
+```
+
+### Using run_benchmarks.py directly
+
+```bash
+# Run both C++ and Python (default)
+pixi run -e wheel python run_benchmarks.py -b transform/fill -d 0
+
+# Run only C++
+pixi run -e wheel python run_benchmarks.py -b transform/fill --cpp
+
+# Run only Python
+pixi run -e wheel python run_benchmarks.py -b transform/fill --py
+
+# Show help
+pixi run -e wheel python run_benchmarks.py --help
+```
+
+To run the benchmarks using the "quick" configuration:
+
+```bash
+pixi run -e wheel python run_benchmarks.py --quick
+```
+
+## Compare Results
+
+```bash
+pixi run -e wheel python analysis/python_vs_cpp_summary.py -b transform/fill
+```
+
+## Web Report
+
+A simple page used to visualize a set of results.
+
+- Requires `results/` to be populated with benchmark results.
+
+First generate a manifest:
+
+```bash
+pixi run -e wheel python analysis/generate_web_report_manifest.py \
+  --results-dir results \
+  --output results/manifest.json
+```
+
+Build the web report single file app:
+
+```bash
+cd analysis/web-report
+npm install
+npm run build
+```
+
+This will output a single file app to `analysis/web-report/dist/` copy it to the `results/` directory and:
+
+```bash
+cd results/
+python3 -m http.server
+```
+
+Now its possible to share the results directory as a zip/tar file.
+
+## Manual Usage
+
+### List benchmark configurations
+
+```bash
+# Python
+pixi run -e wheel python transform/fill.py --list
+
+# C++
+/path/to/cccl/build/cub/bin/cub.bench.transform.fill.base --list
+```
+
+### Run with custom options
+
+```bash
+# Python - specific type and size
+pixi run -e wheel python transform/fill.py --axis "T=I32" --axis "Elements[pow2]=20" --devices 0
+
+# C++ - save JSON
+/path/to/cccl/build/cub/bin/cub.bench.transform.fill.base \
+  --json results/transform/fill_cpp.json \
+  --devices 0
+```
+
+### Compare manually
+
+```bash
+pixi run -e wheel python analysis/python_vs_cpp_summary.py \
+  results/transform/fill_py.json \
+  results/transform/fill_cpp.json \
+  --device 0
+```
+
+## AI commands
+
+These are using the .opencode folder but can be moved to other Agents.
+
+### /migration-status
+
+Generates a report of the migration status for each benchmark in CUB.
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/histogram/even.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/histogram/even.py
new file mode 100644
index 00000000..1d11b4af
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/histogram/even.py
@@ -0,0 +1,132 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for histogram_even using cuda.compute.
+
+C++ equivalent: cub/benchmarks/bench/histogram/even.cu
+
+Notes:
+- The C++ benchmark uses Entropy axis with nvbench_helper bit entropy generation
+- Migration: Python matches the bitwise-AND entropy approach and skips some I8/I16 large-bin cases due to CUDA errors.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+import numpy as np
+from utils import FUNDAMENTAL_TYPES as TYPE_MAP
+from utils import as_cupy_stream, generate_data_with_entropy
+
+import cuda.bench as bench
+from cuda.compute import make_histogram_even
+
+
+def get_upper_level(dtype, num_bins, num_elements):
+    """
+    Compute upper level for histogram bins.
+    Mirrors C++ get_upper_level() from histogram_common.cuh
+    """
+    if np.issubdtype(dtype, np.integer):
+        # For integer types, upper_level = min(num_bins, max_value_for_type)
+        max_val = np.iinfo(dtype).max
+        return dtype(min(num_bins, max_val))
+    else:
+        # For floating point types, upper_level = num_elements
+        return dtype(num_elements)
+
+
+def bench_histogram_even(state: bench.State):
+    type_str = state.get_string("SampleT{ct}")
+    dtype = TYPE_MAP[type_str]
+    num_elements = int(state.get_int64("Elements{io}"))
+    num_bins = int(state.get_int64("Bins"))
+    entropy_str = state.get_string("Entropy")
+
+    # Skip invalid configurations (like C++ does)
+    # For integer types, skip if num_bins > max value representable by SampleT
+    if np.issubdtype(dtype, np.integer):
+        max_val = np.iinfo(dtype).max
+        if num_bins > max_val:
+            state.skip("Number of bins exceeds what SampleT can represent")
+            return
+
+    num_levels = num_bins + 1
+    lower_level = dtype(0)
+    upper_level = get_upper_level(dtype, num_bins, num_elements)
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+
+    d_samples = generate_data_with_entropy(
+        num_elements,
+        dtype,
+        entropy_str,
+        alloc_stream,
+        min_val=lower_level,
+        max_val=upper_level,
+    )
+
+    # Output histogram (counter type is int32 in C++)
+    with alloc_stream:
+        d_histogram = cp.zeros(num_bins, dtype=np.int32)
+
+    alloc_stream.synchronize()
+
+    h_num_output_levels = np.array([num_levels], dtype=np.int32)
+    h_lower_level = np.array([lower_level], dtype=dtype)
+    h_upper_level = np.array([upper_level], dtype=dtype)
+
+    histogrammer = make_histogram_even(
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=num_elements,
+    )
+
+    temp_storage_bytes = histogrammer(
+        temp_storage=None,
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=num_elements,
+    )
+    with alloc_stream:
+        temp_storage = cp.empty(temp_storage_bytes, dtype=np.uint8)
+
+    state.add_element_count(num_elements)
+    state.add_global_memory_reads(num_elements * d_samples.dtype.itemsize)
+    state.add_global_memory_writes(num_bins * d_histogram.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        histogrammer(
+            temp_storage=temp_storage,
+            d_samples=d_samples,
+            d_histogram=d_histogram,
+            h_num_output_levels=h_num_output_levels,
+            h_lower_level=h_lower_level,
+            h_upper_level=h_upper_level,
+            num_samples=num_elements,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_histogram_even)
+    b.set_name("base")
+
+    b.add_string_axis("SampleT{ct}", list(TYPE_MAP.keys()))
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 29, 4))
+    b.add_int64_axis("Bins", [32, 128, 2048, 2097152])
+    b.add_string_axis("Entropy", ["0.201", "1.000"])
+
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/host/host_benchmark_cases.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/host/host_benchmark_cases.py
new file mode 100644
index 00000000..29fb1d92
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/host/host_benchmark_cases.py
@@ -0,0 +1,1562 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from types import SimpleNamespace
+from typing import Any, Callable, Literal
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute as cc
+from cuda.compute._cpp_compile import compile_cpp_op_code
+from cuda.compute.op import RawOp
+from cuda.core import Device
+
+NOOP_TEMP_STORAGE_BYTES = 1
+NUM_ITEMS = 128
+NUM_SEGMENTS = 4
+
+NoopReturnKind = Literal["none", "temp_storage_bytes", "temp_storage_and_selector"]
+
+
+@dataclass(frozen=True)
+class HostBenchmarkCase:
+    name: str
+    setup: Callable[[], SimpleNamespace]
+    make_wrapper: Callable[[SimpleNamespace], Any]
+    oneshot: Callable[[SimpleNamespace], None]
+    twoshot: Callable[[SimpleNamespace, Any], None]
+    noop_return_kind: NoopReturnKind
+    skip_reason: str | None = None
+
+
+class NoopBuildResult:
+    """Proxy over a loaded build result that skips the native kernel launch.
+
+    The host-overhead benchmarks want to measure everything ``__call__`` does on
+    the host — iterator/op state updates, stream validation, build-result
+    resolution — without paying for (or allocating correct temp storage for) the
+    GPU kernel. Wrapping the loaded build result intercepts every kernel-launch
+    entry point with a fabricated return, while delegating any other attribute
+    (e.g. ``determinism``, which reduce reads to pick its entry point) to the
+    real result.
+    """
+
+    # Every build-result method an algorithm's __call__ dispatches its kernel
+    # through. Keep in sync with cuda/compute/algorithms/ (grep for
+    # ``loaded_build_result.compute``). A new entry point missing here would run
+    # the real kernel on the benchmark's 1-byte temp storage and fail loudly —
+    # which the CI smoke test (rounds=iterations=1) exists to catch.
+    _COMPUTE_METHODS = frozenset(
+        {
+            "compute",
+            "compute_even",
+            "compute_nondeterministic",
+            "compute_inclusive",
+            "compute_inclusive_no_init",
+            "compute_inclusive_future_value",
+            "compute_exclusive",
+            "compute_exclusive_future_value",
+        }
+    )
+
+    def __init__(self, real_build_result: Any, return_kind: NoopReturnKind):
+        self._real_build_result = real_build_result
+        self._return_kind = return_kind
+
+    def __getattr__(self, name: str) -> Any:
+        if name in NoopBuildResult._COMPUTE_METHODS:
+
+            def noop_compute(*args, **kwargs):
+                return _noop_return(self._return_kind)
+
+            return noop_compute
+        return getattr(self._real_build_result, name)
+
+
+def _noop_return(return_kind: NoopReturnKind):
+    if return_kind == "none":
+        return None
+    if return_kind == "temp_storage_bytes":
+        return NOOP_TEMP_STORAGE_BYTES
+    if return_kind == "temp_storage_and_selector":
+        return NOOP_TEMP_STORAGE_BYTES, -1
+    raise ValueError(f"Unsupported no-op return kind: {return_kind}")
+
+
+def patch_wrapper_to_skip_native_compute(
+    wrapper: Any, return_kind: NoopReturnKind
+) -> None:
+    """Patch a cached wrapper so measured calls skip the native kernel launch.
+
+    Default-build wrappers resolve, on every ``__call__``, to the loaded build
+    result bound at construction (``_bound_build_result``; see
+    ``resolve_build_result``). Replacing it with a NoopBuildResult is therefore
+    enough: the per-call re-resolution and any re-binding of the compute fn both
+    read through it. ``_Select`` owns no build result of its own — it delegates
+    to a nested three-way-partition wrapper — so recurse into ``partitioner``.
+    """
+    if (bound := getattr(wrapper, "_bound_build_result", None)) is not None:
+        wrapper._bound_build_result = NoopBuildResult(bound, return_kind)
+
+    if (partitioner := getattr(wrapper, "partitioner", None)) is not None:
+        patch_wrapper_to_skip_native_compute(partitioner, return_kind)
+
+
+def make_tiny_temp_storage() -> cp.ndarray:
+    return cp.empty(NOOP_TEMP_STORAGE_BYTES, dtype=cp.uint8)
+
+
+def make_stream():
+    device = Device()
+    device.set_current()
+    return device.create_stream()
+
+
+def synchronize() -> None:
+    cp.cuda.Device().synchronize()
+
+
+def _numba_cuda_skip_reason() -> str | None:
+    try:
+        import numba.cuda  # noqa: F401
+    except Exception as exc:
+        return f"numba.cuda is not available: {exc}"
+    return None
+
+
+_NUMBA_CUDA_SKIP_REASON = _numba_cuda_skip_reason()
+
+
+def _raw_predicate_i32(name: str) -> RawOp:
+    source = f"""
+extern "C" __device__ void {name}(void* x, void* result) {{
+    int value = *static_cast<int*>(x);
+    *static_cast<bool*>(result) = value < {NUM_ITEMS // 2};
+}}
+"""
+    return RawOp(ltoir=compile_cpp_op_code(source), name=name)
+
+
+def _raw_plus_i32() -> RawOp:
+    source = """
+extern "C" __device__ void host_bench_plus_i32(
+    void* lhs,
+    void* rhs,
+    void* result
+) {
+    *static_cast<int*>(result) =
+        *static_cast<int*>(lhs) + *static_cast<int*>(rhs);
+}
+"""
+    return RawOp(ltoir=compile_cpp_op_code(source), name="host_bench_plus_i32")
+
+
+def _raw_identity_i32() -> RawOp:
+    source = """
+extern "C" __device__ void host_bench_identity_i32(void* x, void* result) {
+    *static_cast<int*>(result) = *static_cast<int*>(x);
+}
+"""
+    return RawOp(ltoir=compile_cpp_op_code(source), name="host_bench_identity_i32")
+
+
+def _raw_less_i32() -> RawOp:
+    source = """
+extern "C" __device__ void host_bench_less_i32(
+    void* lhs,
+    void* rhs,
+    void* result
+) {
+    *static_cast<bool*>(result) =
+        *static_cast<int*>(lhs) < *static_cast<int*>(rhs);
+}
+"""
+    return RawOp(ltoir=compile_cpp_op_code(source), name="host_bench_less_i32")
+
+
+def _raw_equal_i32() -> RawOp:
+    source = """
+extern "C" __device__ void host_bench_equal_i32(
+    void* lhs,
+    void* rhs,
+    void* result
+) {
+    *static_cast<bool*>(result) =
+        *static_cast<int*>(lhs) == *static_cast<int*>(rhs);
+}
+"""
+    return RawOp(ltoir=compile_cpp_op_code(source), name="host_bench_equal_i32")
+
+
+def _py_plus_i32(lhs, rhs):
+    return lhs + rhs
+
+
+def _py_identity_i32(x):
+    return x
+
+
+def _py_less_i32(lhs, rhs):
+    return lhs < rhs
+
+
+def _py_equal_i32(lhs, rhs):
+    return lhs == rhs
+
+
+def _py_predicate_i32(x):
+    return x < NUM_ITEMS // 2
+
+
+def _setup_unary_input_output() -> SimpleNamespace:
+    d_in = cp.arange(NUM_ITEMS, dtype=cp.int32)
+    d_out = cp.empty_like(d_in)
+    return SimpleNamespace(d_in=d_in, d_out=d_out, num_items=NUM_ITEMS)
+
+
+def _setup_binary_input_output() -> SimpleNamespace:
+    d_in1 = cp.arange(NUM_ITEMS, dtype=cp.int32)
+    d_in2 = cp.arange(NUM_ITEMS, dtype=cp.int32)
+    d_out = cp.empty_like(d_in1)
+    return SimpleNamespace(d_in1=d_in1, d_in2=d_in2, d_out=d_out, num_items=NUM_ITEMS)
+
+
+def _setup_reduce() -> SimpleNamespace:
+    state = _setup_unary_input_output()
+    state.h_init = np.array([0], dtype=np.int32)
+    state.op = cc.OpKind.PLUS
+    state.temp_storage = make_tiny_temp_storage()
+    return state
+
+
+def _make_reduce(state: SimpleNamespace):
+    return cc.make_reduce_into(
+        d_in=state.d_in,
+        d_out=state.d_out[:1],
+        op=state.op,
+        h_init=state.h_init,
+    )
+
+
+def _oneshot_reduce(state: SimpleNamespace) -> None:
+    cc.reduce_into(
+        d_in=state.d_in,
+        d_out=state.d_out[:1],
+        num_items=state.num_items,
+        op=state.op,
+        h_init=state.h_init,
+    )
+
+
+def _twoshot_reduce(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_in=state.d_in,
+        d_out=state.d_out[:1],
+        num_items=state.num_items,
+        op=state.op,
+        h_init=state.h_init,
+    )
+
+
+def _oneshot_reduce_stream(state: SimpleNamespace) -> None:
+    cc.reduce_into(
+        d_in=state.d_in,
+        d_out=state.d_out[:1],
+        num_items=state.num_items,
+        op=state.op,
+        h_init=state.h_init,
+        stream=state.stream,
+    )
+
+
+def _twoshot_reduce_stream(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_in=state.d_in,
+        d_out=state.d_out[:1],
+        num_items=state.num_items,
+        op=state.op,
+        h_init=state.h_init,
+        stream=state.stream,
+    )
+
+
+def _setup_scan() -> SimpleNamespace:
+    state = _setup_unary_input_output()
+    state.h_init = np.array([0], dtype=np.int32)
+    state.op = cc.OpKind.PLUS
+    state.temp_storage = make_tiny_temp_storage()
+    return state
+
+
+def _make_scan(state: SimpleNamespace):
+    return cc.make_exclusive_scan(
+        d_in=state.d_in,
+        d_out=state.d_out,
+        op=state.op,
+        init_value=state.h_init,
+    )
+
+
+def _oneshot_scan(state: SimpleNamespace) -> None:
+    cc.exclusive_scan(
+        d_in=state.d_in,
+        d_out=state.d_out,
+        op=state.op,
+        init_value=state.h_init,
+        num_items=state.num_items,
+    )
+
+
+def _twoshot_scan(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_in=state.d_in,
+        d_out=state.d_out,
+        op=state.op,
+        init_value=state.h_init,
+        num_items=state.num_items,
+    )
+
+
+def _oneshot_scan_stream(state: SimpleNamespace) -> None:
+    cc.exclusive_scan(
+        d_in=state.d_in,
+        d_out=state.d_out,
+        op=state.op,
+        init_value=state.h_init,
+        num_items=state.num_items,
+        stream=state.stream,
+    )
+
+
+def _twoshot_scan_stream(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_in=state.d_in,
+        d_out=state.d_out,
+        op=state.op,
+        init_value=state.h_init,
+        num_items=state.num_items,
+        stream=state.stream,
+    )
+
+
+def _setup_segmented_reduce() -> SimpleNamespace:
+    d_in = cp.arange(NUM_ITEMS, dtype=cp.int32)
+    d_out = cp.empty(NUM_SEGMENTS, dtype=cp.int32)
+    offsets = cp.asarray(np.linspace(0, NUM_ITEMS, NUM_SEGMENTS + 1, dtype=np.int64))
+    return SimpleNamespace(
+        d_in=d_in,
+        d_out=d_out,
+        start_offsets=offsets[:-1],
+        end_offsets=offsets[1:],
+        num_segments=NUM_SEGMENTS,
+        h_init=np.array([0], dtype=np.int32),
+        op=cc.OpKind.PLUS,
+        temp_storage=make_tiny_temp_storage(),
+    )
+
+
+def _make_segmented_reduce(state: SimpleNamespace):
+    return cc.make_segmented_reduce(
+        d_in=state.d_in,
+        d_out=state.d_out,
+        start_offsets_in=state.start_offsets,
+        end_offsets_in=state.end_offsets,
+        op=state.op,
+        h_init=state.h_init,
+    )
+
+
+def _oneshot_segmented_reduce(state: SimpleNamespace) -> None:
+    cc.segmented_reduce(
+        d_in=state.d_in,
+        d_out=state.d_out,
+        num_segments=state.num_segments,
+        start_offsets_in=state.start_offsets,
+        end_offsets_in=state.end_offsets,
+        op=state.op,
+        h_init=state.h_init,
+    )
+
+
+def _twoshot_segmented_reduce(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_in=state.d_in,
+        d_out=state.d_out,
+        num_segments=state.num_segments,
+        start_offsets_in=state.start_offsets,
+        end_offsets_in=state.end_offsets,
+        op=state.op,
+        h_init=state.h_init,
+    )
+
+
+def _oneshot_segmented_reduce_stream(state: SimpleNamespace) -> None:
+    cc.segmented_reduce(
+        d_in=state.d_in,
+        d_out=state.d_out,
+        num_segments=state.num_segments,
+        start_offsets_in=state.start_offsets,
+        end_offsets_in=state.end_offsets,
+        op=state.op,
+        h_init=state.h_init,
+        stream=state.stream,
+    )
+
+
+def _twoshot_segmented_reduce_stream(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_in=state.d_in,
+        d_out=state.d_out,
+        num_segments=state.num_segments,
+        start_offsets_in=state.start_offsets,
+        end_offsets_in=state.end_offsets,
+        op=state.op,
+        h_init=state.h_init,
+        stream=state.stream,
+    )
+
+
+def _make_unary_transform(state: SimpleNamespace):
+    return cc.make_unary_transform(
+        d_in=state.d_in,
+        d_out=state.d_out,
+        op=state.op,
+    )
+
+
+def _oneshot_unary_transform(state: SimpleNamespace) -> None:
+    cc.unary_transform(
+        d_in=state.d_in,
+        d_out=state.d_out,
+        op=state.op,
+        num_items=state.num_items,
+    )
+
+
+def _twoshot_unary_transform(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        d_in=state.d_in,
+        d_out=state.d_out,
+        op=state.op,
+        num_items=state.num_items,
+    )
+
+
+def _oneshot_unary_transform_stream(state: SimpleNamespace) -> None:
+    cc.unary_transform(
+        d_in=state.d_in,
+        d_out=state.d_out,
+        op=state.op,
+        num_items=state.num_items,
+        stream=state.stream,
+    )
+
+
+def _twoshot_unary_transform_stream(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        d_in=state.d_in,
+        d_out=state.d_out,
+        op=state.op,
+        num_items=state.num_items,
+        stream=state.stream,
+    )
+
+
+def _make_binary_transform(state: SimpleNamespace):
+    return cc.make_binary_transform(
+        d_in1=state.d_in1,
+        d_in2=state.d_in2,
+        d_out=state.d_out,
+        op=state.op,
+    )
+
+
+def _oneshot_binary_transform(state: SimpleNamespace) -> None:
+    cc.binary_transform(
+        d_in1=state.d_in1,
+        d_in2=state.d_in2,
+        d_out=state.d_out,
+        op=state.op,
+        num_items=state.num_items,
+    )
+
+
+def _twoshot_binary_transform(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        d_in1=state.d_in1,
+        d_in2=state.d_in2,
+        d_out=state.d_out,
+        op=state.op,
+        num_items=state.num_items,
+    )
+
+
+def _oneshot_binary_transform_stream(state: SimpleNamespace) -> None:
+    cc.binary_transform(
+        d_in1=state.d_in1,
+        d_in2=state.d_in2,
+        d_out=state.d_out,
+        op=state.op,
+        num_items=state.num_items,
+        stream=state.stream,
+    )
+
+
+def _twoshot_binary_transform_stream(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        d_in1=state.d_in1,
+        d_in2=state.d_in2,
+        d_out=state.d_out,
+        op=state.op,
+        num_items=state.num_items,
+        stream=state.stream,
+    )
+
+
+def _setup_histogram() -> SimpleNamespace:
+    d_samples = cp.arange(NUM_ITEMS, dtype=cp.int32)
+    num_output_levels = 17
+    d_histogram = cp.empty(num_output_levels - 1, dtype=cp.int32)
+    lower_level = np.int32(0)
+    upper_level = np.int32(NUM_ITEMS)
+    return SimpleNamespace(
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        num_output_levels=num_output_levels,
+        h_num_output_levels=np.array([num_output_levels], dtype=np.int32),
+        lower_level=lower_level,
+        upper_level=upper_level,
+        h_lower_level=np.array([lower_level], dtype=np.int32),
+        h_upper_level=np.array([upper_level], dtype=np.int32),
+        num_samples=NUM_ITEMS,
+        temp_storage=make_tiny_temp_storage(),
+    )
+
+
+def _make_histogram(state: SimpleNamespace):
+    return cc.make_histogram_even(
+        d_samples=state.d_samples,
+        d_histogram=state.d_histogram,
+        h_num_output_levels=state.h_num_output_levels,
+        h_lower_level=state.h_lower_level,
+        h_upper_level=state.h_upper_level,
+        num_samples=state.num_samples,
+    )
+
+
+def _oneshot_histogram(state: SimpleNamespace) -> None:
+    cc.histogram_even(
+        d_samples=state.d_samples,
+        d_histogram=state.d_histogram,
+        num_output_levels=state.num_output_levels,
+        lower_level=state.lower_level,
+        upper_level=state.upper_level,
+        num_samples=state.num_samples,
+    )
+
+
+def _twoshot_histogram(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_samples=state.d_samples,
+        d_histogram=state.d_histogram,
+        h_num_output_levels=state.h_num_output_levels,
+        h_lower_level=state.h_lower_level,
+        h_upper_level=state.h_upper_level,
+        num_samples=state.num_samples,
+    )
+
+
+def _oneshot_histogram_stream(state: SimpleNamespace) -> None:
+    cc.histogram_even(
+        d_samples=state.d_samples,
+        d_histogram=state.d_histogram,
+        num_output_levels=state.num_output_levels,
+        lower_level=state.lower_level,
+        upper_level=state.upper_level,
+        num_samples=state.num_samples,
+        stream=state.stream,
+    )
+
+
+def _twoshot_histogram_stream(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_samples=state.d_samples,
+        d_histogram=state.d_histogram,
+        h_num_output_levels=state.h_num_output_levels,
+        h_lower_level=state.h_lower_level,
+        h_upper_level=state.h_upper_level,
+        num_samples=state.num_samples,
+        stream=state.stream,
+    )
+
+
+def _setup_binary_search() -> SimpleNamespace:
+    d_data = cp.arange(NUM_ITEMS, dtype=cp.int32)
+    d_values = cp.arange(0, NUM_ITEMS, 2, dtype=cp.int32)
+    d_out = cp.empty(d_values.size, dtype=np.uintp)
+    return SimpleNamespace(
+        d_data=d_data,
+        d_values=d_values,
+        d_out=d_out,
+        num_items=NUM_ITEMS,
+        num_values=int(d_values.size),
+        comp=cc.OpKind.LESS,
+    )
+
+
+def _make_lower_bound(state: SimpleNamespace):
+    return cc.make_lower_bound(
+        d_data=state.d_data,
+        d_values=state.d_values,
+        d_out=state.d_out,
+        comp=state.comp,
+    )
+
+
+def _oneshot_lower_bound(state: SimpleNamespace) -> None:
+    cc.lower_bound(
+        d_data=state.d_data,
+        num_items=state.num_items,
+        d_values=state.d_values,
+        num_values=state.num_values,
+        d_out=state.d_out,
+        comp=state.comp,
+    )
+
+
+def _twoshot_lower_bound(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        d_data=state.d_data,
+        num_items=state.num_items,
+        d_values=state.d_values,
+        num_values=state.num_values,
+        d_out=state.d_out,
+        comp=state.comp,
+    )
+
+
+def _oneshot_lower_bound_stream(state: SimpleNamespace) -> None:
+    cc.lower_bound(
+        d_data=state.d_data,
+        num_items=state.num_items,
+        d_values=state.d_values,
+        num_values=state.num_values,
+        d_out=state.d_out,
+        comp=state.comp,
+        stream=state.stream,
+    )
+
+
+def _twoshot_lower_bound_stream(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        d_data=state.d_data,
+        num_items=state.num_items,
+        d_values=state.d_values,
+        num_values=state.num_values,
+        d_out=state.d_out,
+        comp=state.comp,
+        stream=state.stream,
+    )
+
+
+def _setup_select() -> SimpleNamespace:
+    state = _setup_unary_input_output()
+    state.d_num_selected = cp.empty(1, dtype=np.uint64)
+    state.cond = cc.OpKind.LOGICAL_NOT
+    state.temp_storage = make_tiny_temp_storage()
+    return state
+
+
+def _make_select(state: SimpleNamespace):
+    return cc.make_select(
+        d_in=state.d_in,
+        d_out=state.d_out,
+        d_num_selected_out=state.d_num_selected,
+        cond=state.cond,
+    )
+
+
+def _oneshot_select(state: SimpleNamespace) -> None:
+    cc.select(
+        d_in=state.d_in,
+        d_out=state.d_out,
+        d_num_selected_out=state.d_num_selected,
+        cond=state.cond,
+        num_items=state.num_items,
+    )
+
+
+def _twoshot_select(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_in=state.d_in,
+        d_out=state.d_out,
+        d_num_selected_out=state.d_num_selected,
+        cond=state.cond,
+        num_items=state.num_items,
+    )
+
+
+def _oneshot_select_stream(state: SimpleNamespace) -> None:
+    cc.select(
+        d_in=state.d_in,
+        d_out=state.d_out,
+        d_num_selected_out=state.d_num_selected,
+        cond=state.cond,
+        num_items=state.num_items,
+        stream=state.stream,
+    )
+
+
+def _twoshot_select_stream(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_in=state.d_in,
+        d_out=state.d_out,
+        d_num_selected_out=state.d_num_selected,
+        cond=state.cond,
+        num_items=state.num_items,
+        stream=state.stream,
+    )
+
+
+def _setup_three_way_partition() -> SimpleNamespace:
+    state = _setup_unary_input_output()
+    state.d_first = cp.empty_like(state.d_in)
+    state.d_second = cp.empty_like(state.d_in)
+    state.d_unselected = cp.empty_like(state.d_in)
+    state.d_num_selected = cp.empty(2, dtype=np.uint64)
+    state.first_op = cc.OpKind.LOGICAL_NOT
+    state.second_op = cc.OpKind.LOGICAL_NOT
+    state.temp_storage = make_tiny_temp_storage()
+    return state
+
+
+def _make_three_way_partition(state: SimpleNamespace):
+    return cc.make_three_way_partition(
+        d_in=state.d_in,
+        d_first_part_out=state.d_first,
+        d_second_part_out=state.d_second,
+        d_unselected_out=state.d_unselected,
+        d_num_selected_out=state.d_num_selected,
+        select_first_part_op=state.first_op,
+        select_second_part_op=state.second_op,
+    )
+
+
+def _oneshot_three_way_partition(state: SimpleNamespace) -> None:
+    cc.three_way_partition(
+        d_in=state.d_in,
+        d_first_part_out=state.d_first,
+        d_second_part_out=state.d_second,
+        d_unselected_out=state.d_unselected,
+        d_num_selected_out=state.d_num_selected,
+        select_first_part_op=state.first_op,
+        select_second_part_op=state.second_op,
+        num_items=state.num_items,
+    )
+
+
+def _twoshot_three_way_partition(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_in=state.d_in,
+        d_first_part_out=state.d_first,
+        d_second_part_out=state.d_second,
+        d_unselected_out=state.d_unselected,
+        d_num_selected_out=state.d_num_selected,
+        select_first_part_op=state.first_op,
+        select_second_part_op=state.second_op,
+        num_items=state.num_items,
+    )
+
+
+def _oneshot_three_way_partition_stream(state: SimpleNamespace) -> None:
+    cc.three_way_partition(
+        d_in=state.d_in,
+        d_first_part_out=state.d_first,
+        d_second_part_out=state.d_second,
+        d_unselected_out=state.d_unselected,
+        d_num_selected_out=state.d_num_selected,
+        select_first_part_op=state.first_op,
+        select_second_part_op=state.second_op,
+        num_items=state.num_items,
+        stream=state.stream,
+    )
+
+
+def _twoshot_three_way_partition_stream(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_in=state.d_in,
+        d_first_part_out=state.d_first,
+        d_second_part_out=state.d_second,
+        d_unselected_out=state.d_unselected,
+        d_num_selected_out=state.d_num_selected,
+        select_first_part_op=state.first_op,
+        select_second_part_op=state.second_op,
+        num_items=state.num_items,
+        stream=state.stream,
+    )
+
+
+def _setup_unique_by_key() -> SimpleNamespace:
+    d_keys = cp.arange(NUM_ITEMS, dtype=cp.int32)
+    d_items = cp.arange(NUM_ITEMS, dtype=cp.int32)
+    return SimpleNamespace(
+        d_in_keys=d_keys,
+        d_in_items=d_items,
+        d_out_keys=cp.empty_like(d_keys),
+        d_out_items=cp.empty_like(d_items),
+        d_num_selected=cp.empty(1, dtype=np.uint64),
+        op=cc.OpKind.EQUAL_TO,
+        num_items=NUM_ITEMS,
+        temp_storage=make_tiny_temp_storage(),
+    )
+
+
+def _make_unique_by_key(state: SimpleNamespace):
+    return cc.make_unique_by_key(
+        d_in_keys=state.d_in_keys,
+        d_in_items=state.d_in_items,
+        d_out_keys=state.d_out_keys,
+        d_out_items=state.d_out_items,
+        d_out_num_selected=state.d_num_selected,
+        op=state.op,
+    )
+
+
+def _oneshot_unique_by_key(state: SimpleNamespace) -> None:
+    cc.unique_by_key(
+        d_in_keys=state.d_in_keys,
+        d_in_items=state.d_in_items,
+        d_out_keys=state.d_out_keys,
+        d_out_items=state.d_out_items,
+        d_out_num_selected=state.d_num_selected,
+        op=state.op,
+        num_items=state.num_items,
+    )
+
+
+def _twoshot_unique_by_key(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_in_keys=state.d_in_keys,
+        d_in_items=state.d_in_items,
+        d_out_keys=state.d_out_keys,
+        d_out_items=state.d_out_items,
+        d_out_num_selected=state.d_num_selected,
+        op=state.op,
+        num_items=state.num_items,
+    )
+
+
+def _oneshot_unique_by_key_stream(state: SimpleNamespace) -> None:
+    cc.unique_by_key(
+        d_in_keys=state.d_in_keys,
+        d_in_items=state.d_in_items,
+        d_out_keys=state.d_out_keys,
+        d_out_items=state.d_out_items,
+        d_out_num_selected=state.d_num_selected,
+        op=state.op,
+        num_items=state.num_items,
+        stream=state.stream,
+    )
+
+
+def _twoshot_unique_by_key_stream(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_in_keys=state.d_in_keys,
+        d_in_items=state.d_in_items,
+        d_out_keys=state.d_out_keys,
+        d_out_items=state.d_out_items,
+        d_out_num_selected=state.d_num_selected,
+        op=state.op,
+        num_items=state.num_items,
+        stream=state.stream,
+    )
+
+
+def _setup_sort() -> SimpleNamespace:
+    d_in_keys = cp.arange(NUM_ITEMS, 0, -1, dtype=cp.int32)
+    d_out_keys = cp.empty_like(d_in_keys)
+    return SimpleNamespace(
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        op=cc.OpKind.LESS,
+        num_items=NUM_ITEMS,
+        temp_storage=make_tiny_temp_storage(),
+    )
+
+
+def _make_merge_sort(state: SimpleNamespace):
+    return cc.make_merge_sort(
+        d_in_keys=state.d_in_keys,
+        d_out_keys=state.d_out_keys,
+        op=state.op,
+    )
+
+
+def _oneshot_merge_sort(state: SimpleNamespace) -> None:
+    cc.merge_sort(
+        d_in_keys=state.d_in_keys,
+        d_out_keys=state.d_out_keys,
+        num_items=state.num_items,
+        op=state.op,
+    )
+
+
+def _twoshot_merge_sort(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_in_keys=state.d_in_keys,
+        d_in_values=None,
+        d_out_keys=state.d_out_keys,
+        d_out_values=None,
+        num_items=state.num_items,
+        op=state.op,
+    )
+
+
+def _oneshot_merge_sort_stream(state: SimpleNamespace) -> None:
+    cc.merge_sort(
+        d_in_keys=state.d_in_keys,
+        d_out_keys=state.d_out_keys,
+        num_items=state.num_items,
+        op=state.op,
+        stream=state.stream,
+    )
+
+
+def _twoshot_merge_sort_stream(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_in_keys=state.d_in_keys,
+        d_in_values=None,
+        d_out_keys=state.d_out_keys,
+        d_out_values=None,
+        num_items=state.num_items,
+        op=state.op,
+        stream=state.stream,
+    )
+
+
+def _make_radix_sort(state: SimpleNamespace):
+    return cc.make_radix_sort(
+        d_in_keys=state.d_in_keys,
+        d_out_keys=state.d_out_keys,
+        d_in_values=None,
+        d_out_values=None,
+        order=cc.SortOrder.ASCENDING,
+    )
+
+
+def _oneshot_radix_sort(state: SimpleNamespace) -> None:
+    cc.radix_sort(
+        d_in_keys=state.d_in_keys,
+        d_out_keys=state.d_out_keys,
+        num_items=state.num_items,
+        order=cc.SortOrder.ASCENDING,
+    )
+
+
+def _twoshot_radix_sort(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_in_keys=state.d_in_keys,
+        d_out_keys=state.d_out_keys,
+        d_in_values=None,
+        d_out_values=None,
+        num_items=state.num_items,
+    )
+
+
+def _oneshot_radix_sort_stream(state: SimpleNamespace) -> None:
+    cc.radix_sort(
+        d_in_keys=state.d_in_keys,
+        d_out_keys=state.d_out_keys,
+        num_items=state.num_items,
+        order=cc.SortOrder.ASCENDING,
+        stream=state.stream,
+    )
+
+
+def _twoshot_radix_sort_stream(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_in_keys=state.d_in_keys,
+        d_out_keys=state.d_out_keys,
+        d_in_values=None,
+        d_out_values=None,
+        num_items=state.num_items,
+        stream=state.stream,
+    )
+
+
+def _setup_segmented_sort() -> SimpleNamespace:
+    state = _setup_sort()
+    offsets = cp.asarray(np.linspace(0, NUM_ITEMS, NUM_SEGMENTS + 1, dtype=np.int64))
+    state.start_offsets = offsets[:-1]
+    state.end_offsets = offsets[1:]
+    state.num_segments = NUM_SEGMENTS
+    return state
+
+
+def _make_segmented_sort(state: SimpleNamespace):
+    return cc.make_segmented_sort(
+        d_in_keys=state.d_in_keys,
+        d_out_keys=state.d_out_keys,
+        d_in_values=None,
+        d_out_values=None,
+        start_offsets_in=state.start_offsets,
+        end_offsets_in=state.end_offsets,
+        order=cc.SortOrder.ASCENDING,
+    )
+
+
+def _oneshot_segmented_sort(state: SimpleNamespace) -> None:
+    cc.segmented_sort(
+        d_in_keys=state.d_in_keys,
+        d_out_keys=state.d_out_keys,
+        d_in_values=None,
+        d_out_values=None,
+        num_items=state.num_items,
+        num_segments=state.num_segments,
+        start_offsets_in=state.start_offsets,
+        end_offsets_in=state.end_offsets,
+        order=cc.SortOrder.ASCENDING,
+    )
+
+
+def _twoshot_segmented_sort(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_in_keys=state.d_in_keys,
+        d_out_keys=state.d_out_keys,
+        d_in_values=None,
+        d_out_values=None,
+        num_items=state.num_items,
+        num_segments=state.num_segments,
+        start_offsets_in=state.start_offsets,
+        end_offsets_in=state.end_offsets,
+    )
+
+
+def _oneshot_segmented_sort_stream(state: SimpleNamespace) -> None:
+    cc.segmented_sort(
+        d_in_keys=state.d_in_keys,
+        d_out_keys=state.d_out_keys,
+        d_in_values=None,
+        d_out_values=None,
+        num_items=state.num_items,
+        num_segments=state.num_segments,
+        start_offsets_in=state.start_offsets,
+        end_offsets_in=state.end_offsets,
+        order=cc.SortOrder.ASCENDING,
+        stream=state.stream,
+    )
+
+
+def _twoshot_segmented_sort_stream(state: SimpleNamespace, wrapper) -> None:
+    wrapper(
+        temp_storage=state.temp_storage,
+        d_in_keys=state.d_in_keys,
+        d_out_keys=state.d_out_keys,
+        d_in_values=None,
+        d_out_values=None,
+        num_items=state.num_items,
+        num_segments=state.num_segments,
+        start_offsets_in=state.start_offsets,
+        end_offsets_in=state.end_offsets,
+        stream=state.stream,
+    )
+
+
+def _setup_with_values(
+    setup_fn: Callable[[], SimpleNamespace], **values: Any
+) -> Callable[[], SimpleNamespace]:
+    def setup() -> SimpleNamespace:
+        state = setup_fn()
+        for name, value in values.items():
+            setattr(state, name, value)
+        return state
+
+    return setup
+
+
+def _setup_with_factories(
+    setup_fn: Callable[[], SimpleNamespace], **factories: Callable[[], Any]
+) -> Callable[[], SimpleNamespace]:
+    def setup() -> SimpleNamespace:
+        state = setup_fn()
+        for name, factory in factories.items():
+            setattr(state, name, factory())
+        return state
+
+    return setup
+
+
+def _setup_with_stream(
+    setup_fn: Callable[[], SimpleNamespace],
+) -> Callable[[], SimpleNamespace]:
+    def setup() -> SimpleNamespace:
+        state = setup_fn()
+        state.stream = make_stream()
+        return state
+
+    return setup
+
+
+def _make_case(
+    name: str,
+    setup: Callable[[], SimpleNamespace],
+    make_wrapper: Callable[[SimpleNamespace], Any],
+    oneshot: Callable[[SimpleNamespace], None],
+    twoshot: Callable[[SimpleNamespace, Any], None],
+    noop_return_kind: NoopReturnKind,
+    skip_reason: str | None = None,
+) -> HostBenchmarkCase:
+    return HostBenchmarkCase(
+        name,
+        setup,
+        make_wrapper,
+        oneshot,
+        twoshot,
+        noop_return_kind,
+        skip_reason,
+    )
+
+
+CASES = [
+    _make_case(
+        "reduce.plus",
+        _setup_with_values(_setup_reduce, op=cc.OpKind.PLUS),
+        _make_reduce,
+        _oneshot_reduce,
+        _twoshot_reduce,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "reduce.raw_cpp",
+        _setup_with_factories(_setup_reduce, op=_raw_plus_i32),
+        _make_reduce,
+        _oneshot_reduce,
+        _twoshot_reduce,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "reduce.python",
+        _setup_with_values(_setup_reduce, op=_py_plus_i32),
+        _make_reduce,
+        _oneshot_reduce,
+        _twoshot_reduce,
+        "temp_storage_bytes",
+        _NUMBA_CUDA_SKIP_REASON,
+    ),
+    _make_case(
+        "exclusive_scan.plus",
+        _setup_with_values(_setup_scan, op=cc.OpKind.PLUS),
+        _make_scan,
+        _oneshot_scan,
+        _twoshot_scan,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "exclusive_scan.raw_cpp",
+        _setup_with_factories(_setup_scan, op=_raw_plus_i32),
+        _make_scan,
+        _oneshot_scan,
+        _twoshot_scan,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "exclusive_scan.python",
+        _setup_with_values(_setup_scan, op=_py_plus_i32),
+        _make_scan,
+        _oneshot_scan,
+        _twoshot_scan,
+        "temp_storage_bytes",
+        _NUMBA_CUDA_SKIP_REASON,
+    ),
+    _make_case(
+        "segmented_reduce.plus",
+        _setup_with_values(_setup_segmented_reduce, op=cc.OpKind.PLUS),
+        _make_segmented_reduce,
+        _oneshot_segmented_reduce,
+        _twoshot_segmented_reduce,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "segmented_reduce.raw_cpp",
+        _setup_with_factories(_setup_segmented_reduce, op=_raw_plus_i32),
+        _make_segmented_reduce,
+        _oneshot_segmented_reduce,
+        _twoshot_segmented_reduce,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "segmented_reduce.python",
+        _setup_with_values(_setup_segmented_reduce, op=_py_plus_i32),
+        _make_segmented_reduce,
+        _oneshot_segmented_reduce,
+        _twoshot_segmented_reduce,
+        "temp_storage_bytes",
+        _NUMBA_CUDA_SKIP_REASON,
+    ),
+    _make_case(
+        "unary_transform.identity",
+        _setup_with_values(_setup_unary_input_output, op=cc.OpKind.IDENTITY),
+        _make_unary_transform,
+        _oneshot_unary_transform,
+        _twoshot_unary_transform,
+        "none",
+    ),
+    _make_case(
+        "unary_transform.raw_cpp",
+        _setup_with_factories(_setup_unary_input_output, op=_raw_identity_i32),
+        _make_unary_transform,
+        _oneshot_unary_transform,
+        _twoshot_unary_transform,
+        "none",
+    ),
+    _make_case(
+        "unary_transform.python",
+        _setup_with_values(_setup_unary_input_output, op=_py_identity_i32),
+        _make_unary_transform,
+        _oneshot_unary_transform,
+        _twoshot_unary_transform,
+        "none",
+        _NUMBA_CUDA_SKIP_REASON,
+    ),
+    _make_case(
+        "binary_transform.plus",
+        _setup_with_values(_setup_binary_input_output, op=cc.OpKind.PLUS),
+        _make_binary_transform,
+        _oneshot_binary_transform,
+        _twoshot_binary_transform,
+        "none",
+    ),
+    _make_case(
+        "binary_transform.raw_cpp",
+        _setup_with_factories(_setup_binary_input_output, op=_raw_plus_i32),
+        _make_binary_transform,
+        _oneshot_binary_transform,
+        _twoshot_binary_transform,
+        "none",
+    ),
+    _make_case(
+        "binary_transform.python",
+        _setup_with_values(_setup_binary_input_output, op=_py_plus_i32),
+        _make_binary_transform,
+        _oneshot_binary_transform,
+        _twoshot_binary_transform,
+        "none",
+        _NUMBA_CUDA_SKIP_REASON,
+    ),
+    _make_case(
+        "histogram_even",
+        _setup_histogram,
+        _make_histogram,
+        _oneshot_histogram,
+        _twoshot_histogram,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "lower_bound.less",
+        _setup_with_values(_setup_binary_search, comp=cc.OpKind.LESS),
+        _make_lower_bound,
+        _oneshot_lower_bound,
+        _twoshot_lower_bound,
+        "none",
+    ),
+    _make_case(
+        "lower_bound.raw_cpp",
+        _setup_with_factories(_setup_binary_search, comp=_raw_less_i32),
+        _make_lower_bound,
+        _oneshot_lower_bound,
+        _twoshot_lower_bound,
+        "none",
+    ),
+    _make_case(
+        "lower_bound.python",
+        _setup_with_values(_setup_binary_search, comp=_py_less_i32),
+        _make_lower_bound,
+        _oneshot_lower_bound,
+        _twoshot_lower_bound,
+        "none",
+        _NUMBA_CUDA_SKIP_REASON,
+    ),
+    _make_case(
+        "select.logical_not",
+        _setup_with_values(_setup_select, cond=cc.OpKind.LOGICAL_NOT),
+        _make_select,
+        _oneshot_select,
+        _twoshot_select,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "select.raw_cpp",
+        _setup_with_factories(
+            _setup_select,
+            cond=lambda: _raw_predicate_i32("host_bench_select_predicate_i32"),
+        ),
+        _make_select,
+        _oneshot_select,
+        _twoshot_select,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "select.python",
+        _setup_with_values(_setup_select, cond=_py_predicate_i32),
+        _make_select,
+        _oneshot_select,
+        _twoshot_select,
+        "temp_storage_bytes",
+        _NUMBA_CUDA_SKIP_REASON,
+    ),
+    _make_case(
+        "three_way_partition.logical_not",
+        _setup_with_values(
+            _setup_three_way_partition,
+            first_op=cc.OpKind.LOGICAL_NOT,
+            second_op=cc.OpKind.LOGICAL_NOT,
+        ),
+        _make_three_way_partition,
+        _oneshot_three_way_partition,
+        _twoshot_three_way_partition,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "three_way_partition.raw_cpp",
+        _setup_with_factories(
+            _setup_three_way_partition,
+            first_op=lambda: _raw_predicate_i32("host_bench_partition_first_i32"),
+            second_op=lambda: _raw_predicate_i32("host_bench_partition_second_i32"),
+        ),
+        _make_three_way_partition,
+        _oneshot_three_way_partition,
+        _twoshot_three_way_partition,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "three_way_partition.python",
+        _setup_with_values(
+            _setup_three_way_partition,
+            first_op=_py_predicate_i32,
+            second_op=_py_predicate_i32,
+        ),
+        _make_three_way_partition,
+        _oneshot_three_way_partition,
+        _twoshot_three_way_partition,
+        "temp_storage_bytes",
+        _NUMBA_CUDA_SKIP_REASON,
+    ),
+    _make_case(
+        "unique_by_key.equal",
+        _setup_with_values(_setup_unique_by_key, op=cc.OpKind.EQUAL_TO),
+        _make_unique_by_key,
+        _oneshot_unique_by_key,
+        _twoshot_unique_by_key,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "unique_by_key.raw_cpp",
+        _setup_with_factories(_setup_unique_by_key, op=_raw_equal_i32),
+        _make_unique_by_key,
+        _oneshot_unique_by_key,
+        _twoshot_unique_by_key,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "unique_by_key.python",
+        _setup_with_values(_setup_unique_by_key, op=_py_equal_i32),
+        _make_unique_by_key,
+        _oneshot_unique_by_key,
+        _twoshot_unique_by_key,
+        "temp_storage_bytes",
+        _NUMBA_CUDA_SKIP_REASON,
+    ),
+    _make_case(
+        "merge_sort.less",
+        _setup_with_values(_setup_sort, op=cc.OpKind.LESS),
+        _make_merge_sort,
+        _oneshot_merge_sort,
+        _twoshot_merge_sort,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "merge_sort.raw_cpp",
+        _setup_with_factories(_setup_sort, op=_raw_less_i32),
+        _make_merge_sort,
+        _oneshot_merge_sort,
+        _twoshot_merge_sort,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "merge_sort.python",
+        _setup_with_values(_setup_sort, op=_py_less_i32),
+        _make_merge_sort,
+        _oneshot_merge_sort,
+        _twoshot_merge_sort,
+        "temp_storage_bytes",
+        _NUMBA_CUDA_SKIP_REASON,
+    ),
+    _make_case(
+        "radix_sort",
+        _setup_sort,
+        _make_radix_sort,
+        _oneshot_radix_sort,
+        _twoshot_radix_sort,
+        "temp_storage_and_selector",
+    ),
+    _make_case(
+        "segmented_sort",
+        _setup_segmented_sort,
+        _make_segmented_sort,
+        _oneshot_segmented_sort,
+        _twoshot_segmented_sort,
+        "temp_storage_and_selector",
+    ),
+]
+
+
+STREAM_CASES = [
+    _make_case(
+        "reduce.plus.stream",
+        _setup_with_stream(_setup_with_values(_setup_reduce, op=cc.OpKind.PLUS)),
+        _make_reduce,
+        _oneshot_reduce_stream,
+        _twoshot_reduce_stream,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "exclusive_scan.plus.stream",
+        _setup_with_stream(_setup_with_values(_setup_scan, op=cc.OpKind.PLUS)),
+        _make_scan,
+        _oneshot_scan_stream,
+        _twoshot_scan_stream,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "segmented_reduce.plus.stream",
+        _setup_with_stream(
+            _setup_with_values(_setup_segmented_reduce, op=cc.OpKind.PLUS)
+        ),
+        _make_segmented_reduce,
+        _oneshot_segmented_reduce_stream,
+        _twoshot_segmented_reduce_stream,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "unary_transform.identity.stream",
+        _setup_with_stream(
+            _setup_with_values(_setup_unary_input_output, op=cc.OpKind.IDENTITY)
+        ),
+        _make_unary_transform,
+        _oneshot_unary_transform_stream,
+        _twoshot_unary_transform_stream,
+        "none",
+    ),
+    _make_case(
+        "binary_transform.plus.stream",
+        _setup_with_stream(
+            _setup_with_values(_setup_binary_input_output, op=cc.OpKind.PLUS)
+        ),
+        _make_binary_transform,
+        _oneshot_binary_transform_stream,
+        _twoshot_binary_transform_stream,
+        "none",
+    ),
+    _make_case(
+        "histogram_even.stream",
+        _setup_with_stream(_setup_histogram),
+        _make_histogram,
+        _oneshot_histogram_stream,
+        _twoshot_histogram_stream,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "lower_bound.less.stream",
+        _setup_with_stream(
+            _setup_with_values(_setup_binary_search, comp=cc.OpKind.LESS)
+        ),
+        _make_lower_bound,
+        _oneshot_lower_bound_stream,
+        _twoshot_lower_bound_stream,
+        "none",
+    ),
+    _make_case(
+        "select.logical_not.stream",
+        _setup_with_stream(
+            _setup_with_values(_setup_select, cond=cc.OpKind.LOGICAL_NOT)
+        ),
+        _make_select,
+        _oneshot_select_stream,
+        _twoshot_select_stream,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "three_way_partition.logical_not.stream",
+        _setup_with_stream(
+            _setup_with_values(
+                _setup_three_way_partition,
+                first_op=cc.OpKind.LOGICAL_NOT,
+                second_op=cc.OpKind.LOGICAL_NOT,
+            )
+        ),
+        _make_three_way_partition,
+        _oneshot_three_way_partition_stream,
+        _twoshot_three_way_partition_stream,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "unique_by_key.equal.stream",
+        _setup_with_stream(
+            _setup_with_values(_setup_unique_by_key, op=cc.OpKind.EQUAL_TO)
+        ),
+        _make_unique_by_key,
+        _oneshot_unique_by_key_stream,
+        _twoshot_unique_by_key_stream,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "merge_sort.less.stream",
+        _setup_with_stream(_setup_with_values(_setup_sort, op=cc.OpKind.LESS)),
+        _make_merge_sort,
+        _oneshot_merge_sort_stream,
+        _twoshot_merge_sort_stream,
+        "temp_storage_bytes",
+    ),
+    _make_case(
+        "radix_sort.stream",
+        _setup_with_stream(_setup_sort),
+        _make_radix_sort,
+        _oneshot_radix_sort_stream,
+        _twoshot_radix_sort_stream,
+        "temp_storage_and_selector",
+    ),
+    _make_case(
+        "segmented_sort.stream",
+        _setup_with_stream(_setup_segmented_sort),
+        _make_segmented_sort,
+        _oneshot_segmented_sort_stream,
+        _twoshot_segmented_sort_stream,
+        "temp_storage_and_selector",
+    ),
+]
+
+CALL_CASES = CASES + STREAM_CASES
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/host/test_host_pytest_benchmark.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/host/test_host_pytest_benchmark.py
new file mode 100644
index 00000000..1fb6b895
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/host/test_host_pytest_benchmark.py
@@ -0,0 +1,95 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+import pytest
+from host_benchmark_cases import (
+    CALL_CASES,
+    CASES,
+    HostBenchmarkCase,
+    patch_wrapper_to_skip_native_compute,
+    synchronize,
+)
+
+import cuda.compute as cc
+
+pytest.importorskip("pytest_benchmark")
+
+BUILD_TIME_ROUNDS = 10
+ONESHOT_ROUNDS = 20
+ONESHOT_ITERATIONS = 100
+TWOSHOT_ROUNDS = 20
+TWOSHOT_ITERATIONS = 1000
+
+
+def _case_params(cases: list[HostBenchmarkCase]) -> list[pytest.ParameterSet]:
+    params = []
+    for case in cases:
+        marks = []
+        if case.skip_reason is not None:
+            marks.append(pytest.mark.skip(reason=case.skip_reason))
+        params.append(pytest.param(case, id=case.name, marks=marks))
+    return params
+
+
+@pytest.mark.benchmark(group="cuda.compute.host.build_time")
+@pytest.mark.parametrize("case", _case_params(CASES))
+def test_build_time(benchmark, case: HostBenchmarkCase):
+    state = case.setup()
+    synchronize()
+
+    def setup() -> None:
+        cc.clear_all_caches()
+
+    def build():
+        return case.make_wrapper(state)
+
+    benchmark.pedantic(
+        build,
+        setup=setup,
+        rounds=BUILD_TIME_ROUNDS,
+        iterations=1,
+        warmup_rounds=0,
+    )
+
+
+@pytest.mark.benchmark(group="cuda.compute.host.oneshot_cached")
+@pytest.mark.parametrize("case", _case_params(CALL_CASES))
+def test_oneshot_cached_host_overhead(benchmark, case: HostBenchmarkCase):
+    cc.clear_all_caches()
+    state = case.setup()
+    wrapper = case.make_wrapper(state)
+    patch_wrapper_to_skip_native_compute(wrapper, case.noop_return_kind)
+    synchronize()
+
+    def call() -> None:
+        case.oneshot(state)
+
+    benchmark.pedantic(
+        call,
+        rounds=ONESHOT_ROUNDS,
+        iterations=ONESHOT_ITERATIONS,
+        warmup_rounds=0,
+    )
+
+
+@pytest.mark.benchmark(group="cuda.compute.host.twoshot_call")
+@pytest.mark.parametrize("case", _case_params(CALL_CASES))
+def test_twoshot_call_host_overhead(benchmark, case: HostBenchmarkCase):
+    cc.clear_all_caches()
+    state = case.setup()
+    wrapper = case.make_wrapper(state)
+    patch_wrapper_to_skip_native_compute(wrapper, case.noop_return_kind)
+    synchronize()
+
+    def call() -> None:
+        case.twoshot(state, wrapper)
+
+    benchmark.pedantic(
+        call,
+        rounds=TWOSHOT_ROUNDS,
+        iterations=TWOSHOT_ITERATIONS,
+        warmup_rounds=0,
+    )
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/merge_sort/keys.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/merge_sort/keys.py
new file mode 100644
index 00000000..3d24d98f
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/merge_sort/keys.py
@@ -0,0 +1,97 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for merge_sort keys using cuda.compute.
+
+C++ equivalent: cub/benchmarks/bench/merge_sort/keys.cu
+
+Notes:
+- The C++ benchmark uses Entropy axis to control data distribution
+- Uses less_t comparison operator (ascending sort)
+- Keys only (no values) - see pairs.cu for key-value sorting
+- Migration: Python fixes offsets and approximates entropy generation.
+- OffsetT axis is omitted because the Python API does not expose offset type.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+import numpy as np
+from utils import SIGNED_TYPES, as_cupy_stream, generate_data_with_entropy
+
+import cuda.bench as bench
+from cuda.compute import OpKind, make_merge_sort
+
+
+def bench_merge_sort_keys(state: bench.State):
+    type_str = state.get_string("T{ct}")
+    dtype = SIGNED_TYPES[type_str]
+    num_elements = int(state.get_int64("Elements{io}"))
+    entropy_str = state.get_string("Entropy")
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+
+    d_in_keys = generate_data_with_entropy(
+        num_elements, dtype, entropy_str, alloc_stream
+    )
+
+    # Output array for sorted keys (merge_sort requires separate output)
+    with alloc_stream:
+        d_out_keys = cp.empty(num_elements, dtype=dtype)
+
+    alloc_stream.synchronize()
+
+    sorter = make_merge_sort(
+        d_in_keys=d_in_keys,
+        d_in_values=None,
+        d_out_keys=d_out_keys,
+        d_out_values=None,
+        op=OpKind.LESS,
+    )
+
+    temp_storage_bytes = sorter(
+        temp_storage=None,
+        d_in_keys=d_in_keys,
+        d_in_values=None,
+        d_out_keys=d_out_keys,
+        d_out_values=None,
+        op=OpKind.LESS,
+        num_items=num_elements,
+    )
+    with alloc_stream:
+        temp_storage = cp.empty(temp_storage_bytes, dtype=np.uint8)
+
+    state.add_element_count(num_elements)
+    state.add_global_memory_reads(num_elements * d_in_keys.dtype.itemsize, "Size")
+    state.add_global_memory_writes(num_elements * d_out_keys.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        sorter(
+            temp_storage=temp_storage,
+            d_in_keys=d_in_keys,
+            d_in_values=None,
+            d_out_keys=d_out_keys,
+            d_out_values=None,
+            op=OpKind.LESS,
+            num_items=num_elements,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_merge_sort_keys)
+    b.set_name("base")
+
+    b.add_string_axis("T{ct}", list(SIGNED_TYPES.keys()))
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 29, 4))
+    b.add_string_axis("Entropy", ["1.000", "0.201"])
+    # Note: OffsetT axis from C++ is not exposed in Python API
+
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/merge_sort/pairs.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/merge_sort/pairs.py
new file mode 100644
index 00000000..efa47e85
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/merge_sort/pairs.py
@@ -0,0 +1,110 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for merge_sort pairs using cuda.compute.
+
+C++ equivalent: cub/benchmarks/bench/merge_sort/pairs.cu
+
+Notes:
+- Uses Entropy axis to control key distribution
+- Keys and values are sorted together
+- Migration: Python omits int128 values and OffsetT axis.
+- OffsetT axis is omitted because the Python API does not expose offset type.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+import numpy as np
+from utils import (
+    INTEGRAL_TYPES,
+    SIGNED_TYPES,
+    as_cupy_stream,
+    generate_data_with_entropy,
+)
+
+import cuda.bench as bench
+from cuda.compute import OpKind, make_merge_sort
+
+KEY_TYPE_MAP = SIGNED_TYPES
+VALUE_TYPE_MAP = INTEGRAL_TYPES
+
+
+def bench_merge_sort_pairs(state: bench.State):
+    key_type_str = state.get_string("KeyT{ct}")
+    value_type_str = state.get_string("ValueT{ct}")
+    key_dtype = KEY_TYPE_MAP[key_type_str]
+    value_dtype = VALUE_TYPE_MAP[value_type_str]
+    num_elements = int(state.get_int64("Elements{io}"))
+    entropy_str = state.get_string("Entropy")
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+
+    d_in_keys = generate_data_with_entropy(
+        num_elements, key_dtype, entropy_str, alloc_stream
+    )
+
+    with alloc_stream:
+        d_in_values = generate_data_with_entropy(
+            num_elements, value_dtype, "1.000", alloc_stream
+        )
+
+        d_out_keys = cp.empty(num_elements, dtype=key_dtype)
+        d_out_values = cp.empty(num_elements, dtype=value_dtype)
+
+    alloc_stream.synchronize()
+
+    sorter = make_merge_sort(
+        d_in_keys=d_in_keys,
+        d_in_values=d_in_values,
+        d_out_keys=d_out_keys,
+        d_out_values=d_out_values,
+        op=OpKind.LESS,
+    )
+
+    temp_storage_bytes = sorter(
+        temp_storage=None,
+        d_in_keys=d_in_keys,
+        d_in_values=d_in_values,
+        d_out_keys=d_out_keys,
+        d_out_values=d_out_values,
+        op=OpKind.LESS,
+        num_items=num_elements,
+    )
+    with alloc_stream:
+        temp_storage = cp.empty(temp_storage_bytes, dtype=np.uint8)
+
+    state.add_element_count(num_elements)
+    state.add_global_memory_reads(num_elements * d_in_keys.dtype.itemsize)
+    state.add_global_memory_reads(num_elements * d_in_values.dtype.itemsize)
+    state.add_global_memory_writes(num_elements * d_out_keys.dtype.itemsize)
+    state.add_global_memory_writes(num_elements * d_out_values.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        sorter(
+            temp_storage=temp_storage,
+            d_in_keys=d_in_keys,
+            d_in_values=d_in_values,
+            d_out_keys=d_out_keys,
+            d_out_values=d_out_values,
+            op=OpKind.LESS,
+            num_items=num_elements,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_merge_sort_pairs)
+    b.set_name("base")
+    b.add_string_axis("KeyT{ct}", list(KEY_TYPE_MAP.keys()))
+    b.add_string_axis("ValueT{ct}", list(VALUE_TYPE_MAP.keys()))
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 29, 4))
+    b.add_string_axis("Entropy", ["1.000", "0.201"])
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/partition/three_way.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/partition/three_way.py
new file mode 100644
index 00000000..d46423cb
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/partition/three_way.py
@@ -0,0 +1,138 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for three_way_partition using cuda.compute.
+
+C++ equivalent: cub/benchmarks/bench/partition/three_way.cu
+
+Notes:
+- The C++ benchmark uses Entropy axis to control data distribution
+- Uses less_then_t<T> predicate operators to divide data into three partitions:
+  - First partition: items < left_border (max/3)
+  - Second partition: items < right_border (max*2/3)
+  - Third partition (unselected): items >= right_border
+- T axis covers fundamental types (C++ fundamental_types minus int128)
+- Migration: Python uses FUNDAMENTAL_TYPES; omits OffsetT axis.
+- OffsetT axis is omitted because the Python API does not expose offset type.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+import numpy as np
+from utils import FUNDAMENTAL_TYPES, as_cupy_stream, generate_data_with_entropy
+
+import cuda.bench as bench
+from cuda.compute import make_three_way_partition
+
+
+def bench_three_way_partition(state: bench.State):
+    type_str = state.get_string("T{ct}")
+    dtype = FUNDAMENTAL_TYPES[type_str]
+    num_elements = int(state.get_int64("Elements{io}"))
+    entropy_str = state.get_string("Entropy")
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+
+    if np.issubdtype(dtype, np.integer):
+        info = np.iinfo(dtype)
+        min_val = 0
+        max_val = info.max
+    else:
+        info = np.finfo(dtype)
+        min_val = 0.0
+        max_val = info.max
+
+    left_border = max_val // 3 if np.issubdtype(dtype, np.integer) else max_val / 3
+    right_border = left_border * 2
+
+    d_in = generate_data_with_entropy(
+        num_elements,
+        dtype,
+        entropy_str,
+        alloc_stream,
+        min_val=min_val,
+        max_val=max_val,
+    )
+
+    with alloc_stream:
+        d_first_part_out = cp.empty(num_elements, dtype=dtype)
+        d_second_part_out = cp.empty(num_elements, dtype=dtype)
+        d_unselected_out = cp.empty(num_elements, dtype=dtype)
+        # d_num_selected_out stores [num_first_part, num_second_part]
+        d_num_selected_out = cp.empty(2, dtype=np.int32)
+
+    alloc_stream.synchronize()
+
+    # Convert borders to the correct type for closure capture
+    left_thresh = dtype(left_border)
+    right_thresh = dtype(right_border)
+
+    def select_first_part(x):
+        return x < left_thresh
+
+    def select_second_part(x):
+        return x < right_thresh
+
+    partitioner = make_three_way_partition(
+        d_in=d_in,
+        d_first_part_out=d_first_part_out,
+        d_second_part_out=d_second_part_out,
+        d_unselected_out=d_unselected_out,
+        d_num_selected_out=d_num_selected_out,
+        select_first_part_op=select_first_part,
+        select_second_part_op=select_second_part,
+    )
+
+    temp_storage_bytes = partitioner(
+        temp_storage=None,
+        d_in=d_in,
+        d_first_part_out=d_first_part_out,
+        d_second_part_out=d_second_part_out,
+        d_unselected_out=d_unselected_out,
+        d_num_selected_out=d_num_selected_out,
+        select_first_part_op=select_first_part,
+        select_second_part_op=select_second_part,
+        num_items=num_elements,
+    )
+    with alloc_stream:
+        temp_storage = cp.empty(temp_storage_bytes, dtype=np.uint8)
+
+    state.add_element_count(num_elements)
+    state.add_global_memory_reads(num_elements * d_in.dtype.itemsize)
+    state.add_global_memory_writes(num_elements * d_in.dtype.itemsize)
+    # C++ reports add_global_memory_writes<offset_t>(1) — 1 element of offset type.
+    state.add_global_memory_writes(d_num_selected_out.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        partitioner(
+            temp_storage=temp_storage,
+            d_in=d_in,
+            d_first_part_out=d_first_part_out,
+            d_second_part_out=d_second_part_out,
+            d_unselected_out=d_unselected_out,
+            d_num_selected_out=d_num_selected_out,
+            select_first_part_op=select_first_part,
+            select_second_part_op=select_second_part,
+            num_items=num_elements,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_three_way_partition)
+    b.set_name("base")
+
+    b.add_string_axis("T{ct}", list(FUNDAMENTAL_TYPES.keys()))
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 29, 4))
+    b.add_string_axis("Entropy", ["1.000", "0.544", "0.000"])
+    # Note: OffsetT axis from C++ is not exposed in Python API
+
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/pixi.toml b/cccl_upstream/python/cuda_cccl/benchmarks/compute/pixi.toml
new file mode 100644
index 00000000..b53fee44
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/pixi.toml
@@ -0,0 +1,89 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+[workspace]
+channels = ["conda-forge"]
+platforms = ["linux-64"]
+channel-priority = "disabled"
+
+# CUDA 13.1 system requirement
+[feature.cu13.system-requirements]
+cuda = "13"
+
+[feature.cu13.dependencies]
+cuda-version = "13.1.*"
+
+# Python benchmark dependencies
+[feature.bench.dependencies]
+python = "3.13.*"
+numpy = "*"
+cupy = "*"
+pytest-benchmark = "*"
+pyyaml = "*"
+pre-commit = "*"
+
+[feature.bench.pypi-dependencies]
+cuda-bench = ">=0.2.0"
+
+# C++ benchmark build dependencies
+[feature.cpp-bench.dependencies]
+cmake = "*"
+ninja = "*"
+cxx-compiler = "*"
+fmt = "*"
+cuda-cudart-dev = "*"
+
+[feature.cpp-bench.target.linux-64.dependencies]
+cuda-crt-dev_linux-64 = "*"
+cuda-driver-dev_linux-64 = "*"
+
+[feature.cpp-bench.target.linux-64.activation.env]
+CUDA_HOME = "$CONDA_PREFIX/targets/x86_64-linux"
+
+# Important: cuda-cccl installation variants
+
+# Released version from PyPI
+[feature.cccl-wheel.pypi-dependencies]
+cuda-cccl = { version = ">=0.1.0", extras = ["cu13"] }
+
+# Local source build (editable install from repo)
+# Needs nvcc, compilers, and CUDA dev libraries for scikit-build-core to build cuda-cccl
+[feature.cccl-source.dependencies]
+cuda-nvcc = "*"
+cuda-nvrtc-dev = "*"
+libnvjitlink-dev = "*"
+cuda-cudart-dev = "*"
+cuda-driver-dev = "*"
+c-compiler = "*"
+cxx-compiler = "*"
+
+[feature.cccl-source.pypi-dependencies]
+cuda-cccl = { path = "../..", editable = true, extras = ["cu13"] }
+
+# Environments
+[environments]
+wheel = { features = ["cu13", "bench", "cpp-bench", "cccl-wheel"] }
+source = { features = ["cu13", "bench", "cpp-bench", "cccl-source"] }
+
+# Tasks
+[tasks.bench]
+cmd = ["python", "run_benchmarks.py", "--py"]
+description = "Run Python cuda.compute benchmarks"
+
+[tasks.bench-quick]
+cmd = ["python", "run_benchmarks.py", "--py", "--quick"]
+description = "Run Python benchmarks with reduced parameter set"
+
+[tasks.bench-cpp]
+cmd = ["python", "run_benchmarks.py", "--cpp"]
+description = "Run C++ CUB benchmarks"
+
+[tasks.bench-all]
+cmd = ["python", "run_benchmarks.py"]
+description = "Run both Python and C++ benchmarks"
+
+[tasks.pre-commit]
+cmd = ["pre-commit", "run", "--all-files"]
+cwd = "../../../.."
+description = "Run pre-commit checks on the entire repo"
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/quick_configs.yaml b/cccl_upstream/python/cuda_cccl/benchmarks/compute/quick_configs.yaml
new file mode 100644
index 00000000..73976eb7
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/quick_configs.yaml
@@ -0,0 +1,130 @@
+# Quick mode configurations for fast benchmark testing
+# Used by run_benchmarks.py with --quick flag
+#
+# Each benchmark lists axis name -> value pairs.
+# Use C++ axis names (with {ct}/{io} suffixes where applicable).
+# The script strips these suffixes for Python benchmarks.
+#
+# For power-of-two axes (Elements{io}, MaxSegSize, MaxSegmentSize, SegmentSize, Segments{io}),
+# values are exponents (e.g., 16 means 2^16 = 65536). SegmentSize is converted to
+# actual values for Python benchmarks.
+#
+# NOTE: Only specify axes that exist in BOTH C++ and Python benchmarks.
+# Some C++ benchmarks have extra axes (like OffsetT{ct}) not in Python.
+
+transform/fill:
+  "T{ct}": "I32"
+  "Elements{io}": 16
+
+transform/babelstream:
+  "T{ct}": "F32"
+  "Elements{io}": 16
+
+transform/heavy:
+  "Heaviness{ct}": "64"
+  "Elements{io}": 16
+
+transform/fib:
+  "Elements{io}": 16
+
+transform/grayscale:
+  "T{ct}": "F32"
+  "Elements{io}": 16
+
+transform/complex_cmp:
+  "Elements{io}": 16
+
+transform_reduce/sum:
+  "T{ct}": "I32"
+  "Elements{io}": 16
+
+reduce/sum:
+  "T{ct}": "I32"
+  "Elements{io}": 16
+
+reduce/min:
+  "T{ct}": "I32"
+  "Elements{io}": 16
+
+reduce/custom:
+  "T{ct}": "I32"
+  "Elements{io}": 16
+
+reduce/nondeterministic:
+  "T{ct}": "I32"
+  "Elements{io}": 16
+
+scan/exclusive/sum:
+  "T{ct}": "I32"
+  "Elements{io}": 16
+
+scan/exclusive/custom:
+  "T{ct}": "I32"
+  "Elements{io}": 16
+
+histogram/even:
+  "SampleT{ct}": "I32"
+  "Elements{io}": 16
+  "Bins": 128
+  "Entropy": "1.000"
+
+select/if:
+  "T{ct}": "I32"
+  "Elements{io}": 16
+  "Entropy": "1.000"
+
+select/unique_by_key:
+  "KeyT{ct}": "I32"
+  "ValueT{ct}": "I32"
+  "Elements{io}": 16
+  "MaxSegSize": 4
+
+radix_sort/keys:
+  "T{ct}": "I32"
+  "Elements{io}": 16
+  "Entropy": "1.000"
+
+radix_sort/pairs:
+  "KeyT{ct}": "I32"
+  "ValueT{ct}": "I32"
+  "Elements{io}": 16
+  "Entropy": "1.000"
+
+merge_sort/keys:
+  "T{ct}": "I32"
+  "Elements{io}": 16
+  "Entropy": "1.000"
+
+merge_sort/pairs:
+  "KeyT{ct}": "I32"
+  "ValueT{ct}": "I32"
+  "Elements{io}": 16
+  "Entropy": "1.000"
+
+segmented_sort/keys:
+  benchmarks:
+    power:
+      "T{ct}": "I32"
+      "Elements{io}": 22
+      "Segments{io}": 12
+      "Entropy": "1.000"
+    small:
+      "T{ct}": "I32"
+      "Elements{io}": 22
+      "MaxSegmentSize": 1
+    large:
+      "T{ct}": "I32"
+      "Elements{io}": 22
+      "MaxSegmentSize": 10
+
+segmented_reduce/variable_sum:
+  benchmarks:
+    variable_default:
+      "T{ct}": "I32"
+      "Elements{io}": 16
+      "MaxSegmentSize": 4
+
+partition/three_way:
+  "T{ct}": "I32"
+  "Elements{io}": 16
+  "Entropy": "1.000"
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/radix_sort/keys.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/radix_sort/keys.py
new file mode 100644
index 00000000..dadd231e
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/radix_sort/keys.py
@@ -0,0 +1,97 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for radix_sort keys using cuda.compute.
+
+C++ equivalent: cub/benchmarks/bench/radix_sort/keys.cu
+
+Notes:
+- The C++ benchmark uses Entropy axis to control data distribution
+- Sort order is always ascending (C++ benchmark hardcodes this)
+- Keys only (no values) - see radix_sort/pairs.cu for key-value sorting
+- begin_bit=0, end_bit=sizeof(T)*8 (full key comparison)
+- Migration: Python fixes offsets, excludes int128, and approximates entropy generation.
+- OffsetT axis is omitted because the Python API does not expose offset type.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+import numpy as np
+from utils import FUNDAMENTAL_TYPES as TYPE_MAP
+from utils import as_cupy_stream, generate_data_with_entropy
+
+import cuda.bench as bench
+from cuda.compute import SortOrder, make_radix_sort
+
+
+def bench_radix_sort_keys(state: bench.State):
+    type_str = state.get_string("T{ct}")
+    dtype = TYPE_MAP[type_str]
+    num_elements = int(state.get_int64("Elements{io}"))
+    entropy_str = state.get_string("Entropy")
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+
+    d_in_keys = generate_data_with_entropy(
+        num_elements, dtype, entropy_str, alloc_stream
+    )
+
+    # Output array for sorted keys
+    with alloc_stream:
+        d_out_keys = cp.empty(num_elements, dtype=dtype)
+
+    alloc_stream.synchronize()
+
+    sorter = make_radix_sort(
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=None,
+        d_out_values=None,
+        order=SortOrder.ASCENDING,
+    )
+
+    temp_storage_bytes = sorter(
+        temp_storage=None,
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=None,
+        d_out_values=None,
+        num_items=num_elements,
+    )
+    with alloc_stream:
+        temp_storage = cp.empty(temp_storage_bytes, dtype=np.uint8)
+
+    state.add_element_count(num_elements)
+    state.add_global_memory_reads(num_elements * d_in_keys.dtype.itemsize, "Size")
+    state.add_global_memory_writes(num_elements * d_out_keys.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        sorter(
+            temp_storage=temp_storage,
+            d_in_keys=d_in_keys,
+            d_out_keys=d_out_keys,
+            d_in_values=None,
+            d_out_values=None,
+            num_items=num_elements,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_radix_sort_keys)
+    b.set_name("base")
+
+    b.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 29, 4))
+    b.add_string_axis("Entropy", ["1.000", "0.544", "0.201"])
+    # Note: OffsetT axis from C++ is not exposed in Python API
+
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/radix_sort/pairs.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/radix_sort/pairs.py
new file mode 100644
index 00000000..9b604dff
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/radix_sort/pairs.py
@@ -0,0 +1,109 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for radix_sort pairs (key-value) using cuda.compute.
+
+C++ equivalent: cub/benchmarks/bench/radix_sort/pairs.cu
+
+Notes:
+- The C++ benchmark uses Entropy axis to control key data distribution
+- Sort order is always ascending (C++ benchmark hardcodes this)
+- Keys and values are sorted together (values rearranged by key order)
+- begin_bit=0, end_bit=sizeof(KeyT)*8 (full key comparison)
+- C++ uses integral_types for keys and int8/16/32/64(+int128) for values
+- Migration: Python matches C++ integral_types for both keys and values; omits int128 and OffsetT axis.
+- OffsetT axis is omitted because the Python API does not expose offset type.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+import numpy as np
+from utils import INTEGRAL_TYPES, as_cupy_stream, generate_data_with_entropy
+
+import cuda.bench as bench
+from cuda.compute import SortOrder, make_radix_sort
+
+KEY_TYPE_MAP = INTEGRAL_TYPES
+VALUE_TYPE_MAP = INTEGRAL_TYPES
+
+
+def bench_radix_sort_pairs(state: bench.State):
+    key_type_str = state.get_string("KeyT{ct}")
+    value_type_str = state.get_string("ValueT{ct}")
+    key_dtype = KEY_TYPE_MAP[key_type_str]
+    value_dtype = VALUE_TYPE_MAP[value_type_str]
+    num_elements = int(state.get_int64("Elements{io}"))
+    entropy_str = state.get_string("Entropy")
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+
+    d_in_keys = generate_data_with_entropy(
+        num_elements, key_dtype, entropy_str, alloc_stream
+    )
+
+    d_in_values = generate_data_with_entropy(
+        num_elements, value_dtype, "1.000", alloc_stream
+    )
+
+    with alloc_stream:
+        d_out_keys = cp.empty(num_elements, dtype=key_dtype)
+        d_out_values = cp.empty(num_elements, dtype=value_dtype)
+
+    alloc_stream.synchronize()
+
+    sorter = make_radix_sort(
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_values,
+        d_out_values=d_out_values,
+        order=SortOrder.ASCENDING,
+    )
+
+    temp_storage_bytes = sorter(
+        temp_storage=None,
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_values,
+        d_out_values=d_out_values,
+        num_items=num_elements,
+    )
+    with alloc_stream:
+        temp_storage = cp.empty(temp_storage_bytes, dtype=np.uint8)
+
+    state.add_element_count(num_elements)
+    state.add_global_memory_reads(num_elements * d_in_keys.dtype.itemsize)
+    state.add_global_memory_reads(num_elements * d_in_values.dtype.itemsize)
+    state.add_global_memory_writes(num_elements * d_out_keys.dtype.itemsize)
+    state.add_global_memory_writes(num_elements * d_out_values.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        sorter(
+            temp_storage=temp_storage,
+            d_in_keys=d_in_keys,
+            d_out_keys=d_out_keys,
+            d_in_values=d_in_values,
+            d_out_values=d_out_values,
+            num_items=num_elements,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_radix_sort_pairs)
+    b.set_name("base")
+
+    b.add_string_axis("KeyT{ct}", list(KEY_TYPE_MAP.keys()))
+    b.add_string_axis("ValueT{ct}", list(VALUE_TYPE_MAP.keys()))
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 29, 4))
+    b.add_string_axis("Entropy", ["1.000", "0.201"])
+    # Note: OffsetT axis from C++ is not exposed in Python API
+
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/reduce/custom.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/reduce/custom.py
new file mode 100644
index 00000000..ee0ff65c
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/reduce/custom.py
@@ -0,0 +1,82 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for reduce custom operation using cuda.compute.reduce_into.
+
+C++ equivalent: cub/benchmarks/bench/reduce/custom.cu
+
+Notes:
+- Uses a custom max operator (not OpKind) to exercise generic path
+- int128 and complex32 are not supported by cupy
+- Migration: Python limits to basic numeric types; C++ includes int128/complex.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+import numpy as np
+from utils import SIGNED_TYPES as TYPE_MAP
+from utils import as_cupy_stream, generate_data_with_entropy
+
+import cuda.bench as bench
+from cuda.compute import make_reduce_into
+
+
+def max_op(a, b):
+    return a if a > b else b
+
+
+def bench_reduce_custom(state: bench.State):
+    type_str = state.get_string("T{ct}")
+    dtype = TYPE_MAP[type_str]
+    num_items = int(state.get_int64("Elements{io}"))
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+    with alloc_stream:
+        d_in = generate_data_with_entropy(num_items, dtype, "1.000", alloc_stream)
+        d_out = cp.empty(1, dtype=dtype)
+
+    h_init = np.zeros(1, dtype=dtype)
+
+    reducer = make_reduce_into(d_in=d_in, d_out=d_out, op=max_op, h_init=h_init)
+
+    temp_storage_bytes = reducer(
+        temp_storage=None,
+        d_in=d_in,
+        d_out=d_out,
+        num_items=num_items,
+        op=max_op,
+        h_init=h_init,
+    )
+    with alloc_stream:
+        temp_storage = cp.empty(temp_storage_bytes, dtype=np.uint8)
+
+    state.add_element_count(num_items)
+    state.add_global_memory_reads(num_items * d_in.dtype.itemsize, "Size")
+    state.add_global_memory_writes(d_out.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        reducer(
+            temp_storage=temp_storage,
+            d_in=d_in,
+            d_out=d_out,
+            num_items=num_items,
+            op=max_op,
+            h_init=h_init,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_reduce_custom)
+    b.set_name("base")
+    b.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 29, 4))
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/reduce/min.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/reduce/min.py
new file mode 100644
index 00000000..82c51520
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/reduce/min.py
@@ -0,0 +1,85 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for reduce min operation using cuda.compute.reduce_into.
+
+C++ equivalent: cub/benchmarks/bench/reduce/min.cu
+
+Notes:
+- Uses OpKind.MINIMUM for minimum reduction
+- C++ uses cuda::minimum<> which CUB recognizes for optimized code paths (DPX on Hopper+)
+- int128 and complex32 are not supported by cupy
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+import numpy as np
+from utils import FUNDAMENTAL_TYPES as TYPE_MAP
+from utils import as_cupy_stream, generate_data_with_entropy
+
+import cuda.bench as bench
+from cuda.compute import OpKind, make_reduce_into
+
+
+def bench_reduce_min(state: bench.State):
+    type_str = state.get_string("T{ct}")
+    dtype = TYPE_MAP[type_str]
+    num_items = int(state.get_int64("Elements{io}"))
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+    with alloc_stream:
+        d_in = generate_data_with_entropy(num_items, dtype, "1.000", alloc_stream)
+        d_out = cp.empty(1, dtype=dtype)
+
+    # Initial value for min reduction (max value of type)
+    if np.issubdtype(dtype, np.integer):
+        init_val = np.iinfo(dtype).max
+    else:
+        init_val = np.finfo(dtype).max
+    h_init = np.array([init_val], dtype=dtype)
+
+    reducer = make_reduce_into(d_in=d_in, d_out=d_out, op=OpKind.MINIMUM, h_init=h_init)
+
+    temp_storage_bytes = reducer(
+        temp_storage=None,
+        d_in=d_in,
+        d_out=d_out,
+        num_items=num_items,
+        op=OpKind.MINIMUM,
+        h_init=h_init,
+    )
+    with alloc_stream:
+        temp_storage = cp.empty(temp_storage_bytes, dtype=np.uint8)
+
+    state.add_element_count(num_items)
+    state.add_global_memory_reads(num_items * d_in.dtype.itemsize, "Size")
+    state.add_global_memory_writes(d_out.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        reducer(
+            temp_storage=temp_storage,
+            d_in=d_in,
+            d_out=d_out,
+            num_items=num_items,
+            op=OpKind.MINIMUM,
+            h_init=h_init,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_reduce_min)
+    b.set_name("base")
+
+    b.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 29, 4))
+
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/reduce/nondeterministic.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/reduce/nondeterministic.py
new file mode 100644
index 00000000..6e9d2e8f
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/reduce/nondeterministic.py
@@ -0,0 +1,87 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for nondeterministic reduce sum using cuda.compute.reduce_into.
+
+C++ equivalent: cub/benchmarks/bench/reduce/nondeterministic.cu
+
+Notes:
+- Uses Determinism.NOT_GUARANTEED
+- C++ tests int32, int64, float, double
+- Migration: Python fixes offsets; C++ exposes an OffsetT axis.
+- OffsetT axis is omitted because the Python API does not expose offset type.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+import numpy as np
+from utils import ALL_TYPES as _ALL_TYPES
+from utils import as_cupy_stream, generate_data_with_entropy
+
+import cuda.bench as bench
+from cuda.compute import Determinism, OpKind, make_reduce_into
+
+TYPE_MAP = {k: _ALL_TYPES[k] for k in ("I32", "I64", "F32", "F64")}
+
+
+def bench_reduce_nondeterministic(state: bench.State):
+    type_str = state.get_string("T{ct}")
+    dtype = TYPE_MAP[type_str]
+    num_items = int(state.get_int64("Elements{io}"))
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+    with alloc_stream:
+        d_in = generate_data_with_entropy(num_items, dtype, "1.000", alloc_stream)
+        d_out = cp.empty(1, dtype=dtype)
+
+    h_init = np.zeros(1, dtype=dtype)
+
+    reducer = make_reduce_into(
+        d_in=d_in,
+        d_out=d_out,
+        op=OpKind.PLUS,
+        h_init=h_init,
+        determinism=Determinism.NOT_GUARANTEED,
+    )
+
+    temp_storage_bytes = reducer(
+        temp_storage=None,
+        d_in=d_in,
+        d_out=d_out,
+        num_items=num_items,
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+    with alloc_stream:
+        temp_storage = cp.empty(temp_storage_bytes, dtype=np.uint8)
+
+    state.add_element_count(num_items)
+    state.add_global_memory_reads(num_items * d_in.dtype.itemsize, "Size")
+    state.add_global_memory_writes(1 * d_out.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        reducer(
+            temp_storage=temp_storage,
+            d_in=d_in,
+            d_out=d_out,
+            num_items=num_items,
+            op=OpKind.PLUS,
+            h_init=h_init,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_reduce_nondeterministic)
+    b.set_name("base")
+    b.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 29, 4))
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/reduce/sum.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/reduce/sum.py
new file mode 100644
index 00000000..765af40b
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/reduce/sum.py
@@ -0,0 +1,80 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for reduce sum operation using cuda.compute.reduce_into.
+
+C++ equivalent: cub/benchmarks/bench/reduce/sum.cu
+
+Notes:
+- int128 and complex32 are not supported by cupy
+- Migration: Python excludes int128/complex; C++ supports more types/tuning.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+import numpy as np
+from utils import SIGNED_TYPES as TYPE_MAP
+from utils import as_cupy_stream, generate_data_with_entropy
+
+import cuda.bench as bench
+from cuda.compute import OpKind, make_reduce_into
+
+
+def bench_reduce_sum(state: bench.State):
+    type_str = state.get_string("T{ct}")
+    dtype = TYPE_MAP[type_str]
+    num_items = int(state.get_int64("Elements{io}"))
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+    with alloc_stream:
+        d_in = generate_data_with_entropy(num_items, dtype, "1.000", alloc_stream)
+        d_out = cp.empty(1, dtype=dtype)
+
+    # Initial value for reduction
+    h_init = np.zeros(1, dtype=dtype)
+
+    reducer = make_reduce_into(d_in=d_in, d_out=d_out, op=OpKind.PLUS, h_init=h_init)
+
+    temp_storage_bytes = reducer(
+        temp_storage=None,
+        d_in=d_in,
+        d_out=d_out,
+        num_items=num_items,
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+    with alloc_stream:
+        temp_storage = cp.empty(temp_storage_bytes, dtype=np.uint8)
+
+    state.add_element_count(num_items)
+    state.add_global_memory_reads(num_items * d_in.dtype.itemsize, "Size")
+    state.add_global_memory_writes(d_out.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        reducer(
+            temp_storage=temp_storage,
+            d_in=d_in,
+            d_out=d_out,
+            num_items=num_items,
+            op=OpKind.PLUS,
+            h_init=h_init,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_reduce_sum)
+    b.set_name("base")
+
+    b.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 29, 4))
+
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/run_benchmarks.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/run_benchmarks.py
new file mode 100755
index 00000000..c32a3e90
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/run_benchmarks.py
@@ -0,0 +1,528 @@
+#!/usr/bin/env python3
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Run Python cuda.compute and C++ CUB benchmarks.
+
+Prerequisites: C++ benchmarks must be built first via:
+  cd /path/to/cccl && ./ci/build_cub.sh -arch 89
+
+Usage:
+  python run_benchmarks.py [options]
+
+Options:
+  -d, --device ID       GPU device ID [default: 0]
+  -b, --benchmark NAME  Run specific benchmark only [default: all]
+  --py                  Only run Python benchmarks
+  --cpp                 Only run C++ benchmarks
+  --quick, -q           Run with reduced parameter set for fast testing
+  --profile             Run each config once (nvbench profile mode), no sampling
+  -h, --help            Show this help message
+
+Benchmark names follow CUB structure:
+  e.g. transform/fill, transform/babelstream
+
+Examples:
+  python run_benchmarks.py                              # Run all benchmarks
+  python run_benchmarks.py -b transform/fill            # Only fill benchmark
+  python run_benchmarks.py -b reduce/sum -d 0           # Reduce sum on device 0
+  python run_benchmarks.py -b scan/exclusive/sum --py   # Only Python
+  python run_benchmarks.py --quick                      # Quick mode (reduced params)
+  python run_benchmarks.py -b merge_sort/keys -q        # Quick mode, single benchmark
+"""
+
+import argparse
+import os
+import shlex
+import subprocess
+import sys
+from pathlib import Path
+
+# ============================================================================
+# Configuration
+# ============================================================================
+
+SCRIPT_DIR = Path(__file__).parent
+CCCL_ROOT = SCRIPT_DIR.parents[3]
+RESULTS_DIR = SCRIPT_DIR / "results"
+CUB_BENCH_DIR = CCCL_ROOT / "build" / "cub" / "bin"
+QUICK_CONFIG_FILE = SCRIPT_DIR / "quick_configs.yaml"
+
+# Supported benchmarks (Python implementations available)
+SUPPORTED_BENCHMARKS = [
+    "transform/fill",
+    "transform/babelstream",
+    "transform/heavy",
+    "transform/fib",
+    "transform/grayscale",
+    "transform/complex_cmp",
+    "transform_reduce/sum",
+    "reduce/sum",
+    "reduce/min",
+    "reduce/custom",
+    "reduce/nondeterministic",
+    "scan/exclusive/sum",
+    "scan/exclusive/custom",
+    "histogram/even",
+    "select/if",
+    "select/unique_by_key",
+    "radix_sort/keys",
+    "radix_sort/pairs",
+    "merge_sort/keys",
+    "merge_sort/pairs",
+    "segmented_sort/keys",
+    "segmented_reduce/variable_sum",
+    "partition/three_way",
+]
+
+# Axes that use power-of-two values (need [pow2] suffix for nvbench)
+# These are the base names (without {ct}/{io} suffixes)
+POW2_AXES_CPP = {
+    "Elements",
+    "GuaranteedMaxSegSize",
+    "MaxSegSize",
+    "MaxSegmentSize",
+    "SegmentSize",
+    "Segments",
+}
+POW2_AXES_PY = {
+    "Elements",
+    "GuaranteedMaxSegSize",
+    "MaxSegSize",
+    "MaxSegmentSize",
+    "Segments",
+}
+
+# Axis name mappings from C++ to Python.
+# Keep type axes in their C++ form (`{ct}`) to match benchmark axis names.
+CPP_TO_PY_AXIS_MAP = {
+    "T{ct}": "T{ct}",
+    "KeyT{ct}": "KeyT{ct}",
+    "ValueT{ct}": "ValueT{ct}",
+    "SampleT{ct}": "SampleT{ct}",
+    "Heaviness{ct}": "Heaviness{ct}",
+}
+
+
+def strip_axis_suffix(axis_name: str) -> str:
+    """Strip {ct} or {io} suffix from axis name for Python benchmarks.
+
+    e.g., "T{ct}" -> "T{ct}", "Elements{io}" -> "Elements{io}".
+    """
+    if axis_name in CPP_TO_PY_AXIS_MAP:
+        return CPP_TO_PY_AXIS_MAP[axis_name]
+    # Generic suffix stripping for any other axes
+    if axis_name.endswith("{ct}"):
+        return axis_name.rsplit("{", 1)[0]
+    if axis_name.endswith("{io}"):
+        return axis_name
+    return axis_name
+
+
+def get_base_axis_name(axis_name: str) -> str:
+    """Get base axis name (without suffix) for POW2 check.
+
+    e.g., "Elements{io}" -> "Elements"
+    """
+    if axis_name.endswith("{ct}") or axis_name.endswith("{io}"):
+        return axis_name.rsplit("{", 1)[0]
+    return axis_name
+
+
+# ============================================================================
+# Helper Functions
+# ============================================================================
+
+
+def print_banner(msg: str) -> None:
+    """Print a banner message."""
+    print()
+    print("=" * 72)
+    print(msg)
+    print("=" * 72)
+    print()
+
+
+def print_section(msg: str) -> None:
+    """Print a section header."""
+    print()
+    print("-" * 72)
+    print(msg)
+    print("-" * 72)
+
+
+def load_quick_configs() -> dict:
+    """Load quick mode configurations from YAML file."""
+    import yaml  # only needed for --quick
+
+    with open(QUICK_CONFIG_FILE) as f:
+        return yaml.safe_load(f)
+
+
+def build_axis_args_from_config(axis_config: dict, for_python: bool) -> list:
+    """Build --axis arguments for nvbench CLI from an axis config dict.
+
+    Args:
+        axis_config: Dict of axis_name -> value
+        for_python: If True, strip C++ suffixes for Python benchmarks
+    """
+    args = []
+    for axis_name, value in axis_config.items():
+        # For Python, strip C++ suffixes from axis names
+        if for_python:
+            axis_name = strip_axis_suffix(axis_name)
+
+        # Check if this is a power-of-two axis (using base name)
+        base_name = get_base_axis_name(axis_name)
+        if for_python and base_name == "SegmentSize":
+            actual_value = 2 ** int(value)
+            args.extend(["--axis", f"{axis_name}={actual_value}"])
+            continue
+
+        pow2_axes = POW2_AXES_PY if for_python else POW2_AXES_CPP
+        if base_name in pow2_axes:
+            args.extend(["--axis", f"{axis_name}[pow2]={value}"])
+        else:
+            args.extend(["--axis", f"{axis_name}={value}"])
+    return args
+
+
+def get_quick_config_entry(benchmark: str, quick_configs: dict) -> dict:
+    """Get quick config entry for a benchmark, raising if missing."""
+    if benchmark not in quick_configs:
+        raise ValueError(
+            f"Benchmark '{benchmark}' not found in quick_configs.yaml.\n"
+            f"Cannot run in --quick mode. Add configuration for this benchmark."
+        )
+    return quick_configs[benchmark]
+
+
+def get_cpp_binary(benchmark: str) -> str:
+    """Get C++ binary name from benchmark path.
+
+    e.g., "transform/fill" -> "cub.bench.transform.fill.base"
+    """
+    return f"cub.bench.{benchmark.replace('/', '.')}.base"
+
+
+def get_py_script(benchmark: str) -> Path:
+    """Get Python script path from benchmark path.
+
+    e.g., "transform/fill" -> "transform/fill.py"
+    """
+    return SCRIPT_DIR / f"{benchmark}.py"
+
+
+def get_result_path(benchmark: str, suffix: str) -> Path:
+    """Get results file path.
+
+    e.g., "transform/fill", "cpp" -> "results/transform/fill_cpp.json"
+    """
+    bench_path = Path(benchmark)
+    return RESULTS_DIR / bench_path.parent / f"{bench_path.name}_{suffix}.json"
+
+
+def get_log_path(benchmark: str, suffix: str) -> Path:
+    """Get log file path under results/logs.
+
+    e.g., "transform/fill", "cpp" -> "results/logs/transform/fill_cpp.log"
+    """
+    bench_path = Path(benchmark)
+    return RESULTS_DIR / "logs" / bench_path.parent / f"{bench_path.name}_{suffix}.log"
+
+
+def run_and_log(cmd: list, log_path: Path, env: dict | None = None) -> dict:
+    """Run command and write stdout/stderr to log file."""
+    log_path.parent.mkdir(parents=True, exist_ok=True)
+    with open(log_path, "w", encoding="utf-8") as log_file:
+        log_file.write(f"Command: {shlex.join(cmd)}\n\n")
+        log_file.flush()
+        try:
+            result = subprocess.run(
+                cmd, check=False, stdout=log_file, stderr=log_file, env=env
+            )
+        except Exception as exc:  # noqa: BLE001
+            log_file.write("\nERROR: Runner failed to execute command.\n")
+            log_file.write(f"{exc}\n")
+            return {"status": "error", "returncode": None, "error": str(exc)}
+
+    status = "ok" if result.returncode == 0 else "failed"
+    return {"status": status, "returncode": result.returncode}
+
+
+# ============================================================================
+# Benchmark Runner
+# ============================================================================
+
+
+def run_benchmark(
+    benchmark: str,
+    device: str,
+    run_py: bool,
+    run_cpp: bool,
+    quick_mode: bool,
+    quick_configs: dict,
+    profile: bool,
+) -> dict:
+    """Run a single benchmark.
+
+    Returns dict with paths to generated result files.
+    """
+    cpp_binary = get_cpp_binary(benchmark)
+    py_script = get_py_script(benchmark)
+    cpp_result = get_result_path(benchmark, "cpp")
+    py_result = get_result_path(benchmark, "py")
+    cpp_log = get_log_path(benchmark, "cpp")
+    py_log = get_log_path(benchmark, "py")
+
+    # Ensure results subdirectory exists
+    cpp_result.parent.mkdir(parents=True, exist_ok=True)
+
+    # Build axis arguments for quick mode
+    cpp_axis_args = []
+    py_axis_args = []
+    if quick_mode:
+        config_entry = get_quick_config_entry(benchmark, quick_configs)
+        if "benchmarks" in config_entry:
+            for bench_name, axis_config in config_entry["benchmarks"].items():
+                cpp_axis_args.extend(["--benchmark", bench_name])
+                cpp_axis_args.extend(
+                    build_axis_args_from_config(axis_config, for_python=False)
+                )
+                py_axis_args.extend(["--benchmark", bench_name])
+                py_axis_args.extend(
+                    build_axis_args_from_config(axis_config, for_python=True)
+                )
+        else:
+            cpp_axis_args = build_axis_args_from_config(config_entry, for_python=False)
+            py_axis_args = build_axis_args_from_config(config_entry, for_python=True)
+
+    results = {}
+
+    # Run C++ benchmark
+    if run_cpp:
+        print(f"Running C++ benchmark: {cpp_binary}")
+
+        cpp_bin = CUB_BENCH_DIR / cpp_binary
+        if not cpp_bin.exists():
+            print(f"ERROR: C++ binary not found: {cpp_bin}")
+            print()
+            print("Please build C++ benchmarks first:")
+            print(f"  cd {CCCL_ROOT}")
+            print("  ./ci/build_cub.sh -arch <your_gpu_arch>  # e.g., 89 for RTX 4090")
+            print()
+            print("Available benchmarks in build directory:")
+            if CUB_BENCH_DIR.exists():
+                binaries = list(CUB_BENCH_DIR.glob("*.base"))[:10]
+                for b in binaries:
+                    print(f"  {b.name}")
+                if len(list(CUB_BENCH_DIR.glob("*.base"))) > 10:
+                    print("  ...")
+            else:
+                print(f"  (directory not found: {CUB_BENCH_DIR})")
+            sys.exit(1)
+
+        cmd = [str(cpp_bin), "--json", str(cpp_result), "--devices", device]
+        cmd.extend(cpp_axis_args)
+        if profile:
+            cmd.append("--profile")
+        # Ensure the CUB build lib dir (containing libnvbench.so) is on the
+        # dynamic linker search path for the child process.
+        cpp_lib_dir = str(CUB_BENCH_DIR.parent / "lib")
+        cpp_env = os.environ.copy()
+        existing_ld = cpp_env.get("LD_LIBRARY_PATH", "")
+        cpp_env["LD_LIBRARY_PATH"] = (
+            f"{cpp_lib_dir}:{existing_ld}" if existing_ld else cpp_lib_dir
+        )
+        cpp_status = run_and_log(cmd, cpp_log, env=cpp_env)
+        print(f"  Results: {cpp_result}")
+        print(f"  Log: {cpp_log}")
+        if cpp_status["status"] != "ok":
+            print(f"  WARNING: C++ benchmark failed (exit {cpp_status['returncode']}).")
+        results["cpp"] = cpp_result
+        results["cpp_status"] = cpp_status
+
+    # Run Python benchmark
+    if run_py:
+        print(f"Running Python benchmark: {py_script.relative_to(SCRIPT_DIR)}")
+
+        if not py_script.exists():
+            print(f"ERROR: Python script not found: {py_script}")
+            sys.exit(1)
+
+        cmd = [
+            sys.executable,
+            str(py_script),
+            "--json",
+            str(py_result),
+            "--devices",
+            device,
+        ]
+        cmd.extend(py_axis_args)
+        if profile:
+            cmd.append("--profile")
+        py_status = run_and_log(cmd, py_log)
+        print(f"  Results: {py_result}")
+        print(f"  Log: {py_log}")
+        if py_status["status"] != "ok":
+            print(
+                f"  WARNING: Python benchmark failed (exit {py_status['returncode']})."
+            )
+        results["py"] = py_result
+        results["py_status"] = py_status
+
+    return results
+
+
+# ============================================================================
+# Main
+# ============================================================================
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Run Python cuda.compute and C++ CUB benchmarks",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog=f"""
+Supported benchmarks:
+{chr(10).join(f"  {b}" for b in SUPPORTED_BENCHMARKS)}
+""",
+    )
+    parser.add_argument(
+        "-d", "--device", default="0", help="GPU device ID [default: 0]"
+    )
+    parser.add_argument(
+        "-b", "--benchmark", help="Run specific benchmark only [default: all]"
+    )
+    parser.add_argument("--py", action="store_true", help="Only run Python benchmarks")
+    parser.add_argument("--cpp", action="store_true", help="Only run C++ benchmarks")
+    parser.add_argument(
+        "-q",
+        "--quick",
+        action="store_true",
+        help="Run with reduced parameter set for fast testing",
+    )
+    parser.add_argument(
+        "--profile",
+        action="store_true",
+        help="Run each benchmark configuration once (nvbench profile mode) with "
+        "no sampling -- for smoke-testing that benchmarks execute without error, "
+        "not for measuring performance.",
+    )
+    args = parser.parse_args()
+
+    # Determine what to run
+    run_py = True
+    run_cpp = True
+    if args.py and not args.cpp:
+        run_cpp = False
+    elif args.cpp and not args.py:
+        run_py = False
+
+    # Load quick configs if needed
+    quick_configs = {}
+    if args.quick:
+        quick_configs = load_quick_configs()
+
+    # Validate and determine benchmarks to run
+    if args.benchmark:
+        if args.benchmark not in SUPPORTED_BENCHMARKS:
+            print(f"ERROR: Benchmark '{args.benchmark}' not supported.")
+            print()
+            print("Available benchmarks:")
+            for b in SUPPORTED_BENCHMARKS:
+                print(f"  {b}")
+            sys.exit(1)
+        benchmarks_to_run = [args.benchmark]
+    else:
+        benchmarks_to_run = SUPPORTED_BENCHMARKS
+
+    # Print configuration
+    print_banner("CCCL Benchmark Runner")
+
+    print("Configuration:")
+    print(f"  CCCL Root:      {CCCL_ROOT}")
+    print(f"  C++ Binaries:   {CUB_BENCH_DIR}")
+    print(f"  Results Dir:    {RESULTS_DIR}")
+    print(f"  Device:         {args.device}")
+    print(f"  Benchmarks:     {' '.join(benchmarks_to_run)}")
+    print(f"  Run C++:        {run_cpp}")
+    print(f"  Run Python:     {run_py}")
+    print(f"  Quick Mode:     {args.quick}")
+    print(f"  Profile Mode:   {args.profile}")
+    print()
+
+    # Check C++ binaries directory exists (if running C++)
+    if run_cpp and not CUB_BENCH_DIR.exists():
+        print(f"ERROR: C++ benchmark directory not found: {CUB_BENCH_DIR}")
+        print()
+        print("Please build C++ benchmarks first:")
+        print(f"  cd {CCCL_ROOT}")
+        print("  ./ci/build_cub.sh -arch <your_gpu_arch>  # e.g., 89 for RTX 4090")
+        sys.exit(1)
+
+    # Run benchmarks
+    all_results = {}
+    for bench in benchmarks_to_run:
+        print_section(f"Benchmark: {bench}")
+        results = run_benchmark(
+            bench, args.device, run_py, run_cpp, args.quick, quick_configs, args.profile
+        )
+        all_results[bench] = results
+        print()
+
+    # Print summary
+    print_banner("Summary")
+
+    print(f"Results directory: {RESULTS_DIR}")
+    print()
+    print("Generated files:")
+    for bench in benchmarks_to_run:
+        cpp_result = get_result_path(bench, "cpp")
+        py_result = get_result_path(bench, "py")
+        cpp_log = get_log_path(bench, "cpp")
+        py_log = get_log_path(bench, "py")
+        print(f"  {bench}:")
+        if cpp_result.exists():
+            print(f"    C++ results:    {cpp_result}")
+        if cpp_log.exists():
+            print(f"    C++ log:        {cpp_log}")
+        if py_result.exists():
+            print(f"    Python results: {py_result}")
+        if py_log.exists():
+            print(f"    Python log:     {py_log}")
+
+        status = all_results.get(bench, {})
+        cpp_status = status.get("cpp_status")
+        py_status = status.get("py_status")
+        if cpp_status and cpp_status.get("status") != "ok":
+            print(
+                "    C++ status:     "
+                f"{cpp_status.get('status')} (exit {cpp_status.get('returncode')})"
+            )
+        if py_status and py_status.get("status") != "ok":
+            print(
+                "    Python status:  "
+                f"{py_status.get('status')} (exit {py_status.get('returncode')})"
+            )
+
+    print()
+
+    # Exit non-zero if any benchmark failed so CI (e.g. the --profile smoke gate)
+    # catches benchmark rot instead of silently passing on printed warnings.
+    failed = []
+    for bench in benchmarks_to_run:
+        results = all_results.get(bench, {})
+        for key in ("cpp_status", "py_status"):
+            if results.get(key, {}).get("status") not in (None, "ok"):
+                failed.append(bench)
+                break
+    if failed:
+        print(f"ERROR: {len(failed)} benchmark(s) failed: {failed}")
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/scan/exclusive/custom.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/scan/exclusive/custom.py
new file mode 100644
index 00000000..703869a6
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/scan/exclusive/custom.py
@@ -0,0 +1,83 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for exclusive scan custom operation using cuda.compute.exclusive_scan.
+
+C++ equivalent: cub/benchmarks/bench/scan/exclusive/custom.cu
+
+Notes:
+- Uses a custom max operator (not OpKind)
+- int128 and complex32 are not supported by cupy
+- Migration: Python fixes offsets; C++ exposes an OffsetT axis.
+- OffsetT axis is omitted because the Python API does not expose offset type.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent.parent))
+
+import cupy as cp
+import numpy as np
+from utils import SIGNED_TYPES as TYPE_MAP
+from utils import as_cupy_stream, generate_data_with_entropy
+
+import cuda.bench as bench
+from cuda.compute import make_exclusive_scan
+
+
+def max_op(a, b):
+    return a if a > b else b
+
+
+def bench_scan_exclusive_custom(state: bench.State):
+    type_str = state.get_string("T{ct}")
+    dtype = TYPE_MAP[type_str]
+    num_items = int(state.get_int64("Elements{io}"))
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+    with alloc_stream:
+        d_in = generate_data_with_entropy(num_items, dtype, "1.000", alloc_stream)
+        d_out = cp.empty(num_items, dtype=dtype)
+
+    h_init = np.zeros(1, dtype=dtype)
+
+    scanner = make_exclusive_scan(d_in=d_in, d_out=d_out, op=max_op, init_value=h_init)
+
+    temp_storage_bytes = scanner(
+        temp_storage=None,
+        d_in=d_in,
+        d_out=d_out,
+        op=max_op,
+        init_value=h_init,
+        num_items=num_items,
+    )
+    with alloc_stream:
+        temp_storage = cp.empty(temp_storage_bytes, dtype=np.uint8)
+
+    state.add_element_count(num_items)
+    state.add_global_memory_reads(num_items * d_in.dtype.itemsize, "Size")
+    state.add_global_memory_writes(num_items * d_out.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        scanner(
+            temp_storage=temp_storage,
+            d_in=d_in,
+            d_out=d_out,
+            op=max_op,
+            init_value=h_init,
+            num_items=num_items,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_scan_exclusive_custom)
+    b.set_name("base")
+    b.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 33, 4))
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/scan/exclusive/sum.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/scan/exclusive/sum.py
new file mode 100644
index 00000000..7a1d5130
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/scan/exclusive/sum.py
@@ -0,0 +1,84 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for exclusive scan sum operation using cuda.compute.exclusive_scan.
+
+C++ equivalent: cub/benchmarks/bench/scan/exclusive/sum.cu
+
+Notes:
+- int128 and complex32 are not supported by cupy
+- Migration: Python fixes offsets; C++ exposes an OffsetT axis.
+- OffsetT axis is omitted because the Python API does not expose offset type.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent.parent))
+
+import cupy as cp
+import numpy as np
+from utils import SIGNED_TYPES as TYPE_MAP
+from utils import as_cupy_stream, generate_data_with_entropy
+
+import cuda.bench as bench
+from cuda.compute import OpKind, make_exclusive_scan
+
+
+def bench_scan_exclusive_sum(state: bench.State):
+    type_str = state.get_string("T{ct}")
+    dtype = TYPE_MAP[type_str]
+    num_items = int(state.get_int64("Elements{io}"))
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+    with alloc_stream:
+        d_in = generate_data_with_entropy(num_items, dtype, "1.000", alloc_stream)
+        # Output is same size as input for scan
+        d_out = cp.empty(num_items, dtype=dtype)
+
+    # Initial value for scan (identity for addition)
+    h_init = np.zeros(1, dtype=dtype)
+
+    scanner = make_exclusive_scan(
+        d_in=d_in, d_out=d_out, op=OpKind.PLUS, init_value=h_init
+    )
+
+    temp_storage_bytes = scanner(
+        temp_storage=None,
+        d_in=d_in,
+        d_out=d_out,
+        op=OpKind.PLUS,
+        init_value=h_init,
+        num_items=num_items,
+    )
+    with alloc_stream:
+        temp_storage = cp.empty(temp_storage_bytes, dtype=np.uint8)
+
+    state.add_element_count(num_items)
+    state.add_global_memory_reads(num_items * d_in.dtype.itemsize, "Size")
+    state.add_global_memory_writes(num_items * d_out.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        scanner(
+            temp_storage=temp_storage,
+            d_in=d_in,
+            d_out=d_out,
+            op=OpKind.PLUS,
+            init_value=h_init,
+            num_items=num_items,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_scan_exclusive_sum)
+    b.set_name("base")
+
+    b.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 33, 4))
+
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/segmented_reduce/variable_sum.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/segmented_reduce/variable_sum.py
new file mode 100644
index 00000000..9d27dfc7
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/segmented_reduce/variable_sum.py
@@ -0,0 +1,165 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for segmented_reduce (sum) with variable-size segments.
+
+C++ equivalent: cub/benchmarks/bench/segmented_reduce/variable_sum.cu (uses variable_base.cuh)
+
+Notes:
+- Implements four sub-benchmarks: variable_default, variable_small_dynamic,
+  variable_medium_dynamic, variable_large_dynamic.
+- OffsetT axis is omitted because the Python API does not expose offset type.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+import numpy as np
+from utils import (
+    ALL_TYPES,
+    as_cupy_stream,
+    generate_data_with_entropy,
+    generate_uniform_segment_offsets,
+)
+
+import cuda.bench as bench
+from cuda.compute import OpKind, make_segmented_reduce
+
+TYPE_MAP = {k: ALL_TYPES[k] for k in ("I32", "I64", "F32", "F64")}
+
+
+def run_segmented_reduce(
+    state: bench.State,
+    d_in,
+    d_out,
+    h_init,
+    start_offsets,
+    end_offsets,
+    num_segments,
+    guaranteed_max_seg_size,
+):
+    reducer = make_segmented_reduce(
+        d_in=d_in,
+        d_out=d_out,
+        start_offsets_in=start_offsets,
+        end_offsets_in=end_offsets,
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+
+    temp_storage_bytes = reducer(
+        temp_storage=None,
+        d_in=d_in,
+        d_out=d_out,
+        num_segments=num_segments,
+        start_offsets_in=start_offsets,
+        end_offsets_in=end_offsets,
+        op=OpKind.PLUS,
+        h_init=h_init,
+        max_segment_size=guaranteed_max_seg_size,
+    )
+    alloc_stream = as_cupy_stream(state.get_stream())
+    with alloc_stream:
+        temp_storage = cp.empty(temp_storage_bytes, dtype=np.uint8)
+
+    def launcher(launch: bench.Launch):
+        reducer(
+            temp_storage=temp_storage,
+            d_in=d_in,
+            d_out=d_out,
+            num_segments=num_segments,
+            start_offsets_in=start_offsets,
+            end_offsets_in=end_offsets,
+            op=OpKind.PLUS,
+            h_init=h_init,
+            max_segment_size=guaranteed_max_seg_size,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False, sync=True)
+
+
+def bench_variable_segmented_reduce(state: bench.State):
+    type_str = state.get_string("T{ct}")
+    dtype = TYPE_MAP[type_str]
+    num_elements = int(state.get_int64("Elements{io}"))
+    max_segment_size = int(state.get_int64("MaxSegmentSize"))
+    guaranteed_max_seg_size = int(state.get_int64("GuaranteedMaxSegSize"))
+
+    # Skip cases where hint would be incorrect (max > guaranteed)
+    if guaranteed_max_seg_size != 0 and max_segment_size > guaranteed_max_seg_size:
+        state.skip("max_segment_size > guaranteed_max_seg_size")
+        return
+
+    min_segment_size = 1
+    offsets = generate_uniform_segment_offsets(
+        num_elements, min_segment_size, max_segment_size
+    )
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+    h_init = np.zeros(1, dtype=dtype)
+    d_in = generate_data_with_entropy(num_elements, dtype, "1.000", alloc_stream)
+    with alloc_stream:
+        start_offsets = cp.asarray(offsets[:-1], dtype=np.int64)
+        end_offsets = cp.asarray(offsets[1:], dtype=np.int64)
+        d_out = cp.empty(int(start_offsets.size), dtype=dtype)
+
+    alloc_stream.synchronize()
+    num_segments = int(start_offsets.size)
+
+    state.add_element_count(num_elements)
+    state.add_global_memory_reads(num_elements * d_in.dtype.itemsize)
+    state.add_global_memory_writes(num_segments * d_out.dtype.itemsize)
+    state.add_global_memory_reads((num_segments + 1) * start_offsets.dtype.itemsize)
+
+    run_segmented_reduce(
+        state,
+        d_in,
+        d_out,
+        h_init,
+        start_offsets,
+        end_offsets,
+        num_segments,
+        guaranteed_max_seg_size,
+    )
+
+
+if __name__ == "__main__":
+    # Default: no size hint — uses generic large-reduce kernel regardless of segment size
+    b_default = bench.register(bench_variable_segmented_reduce)
+    b_default.set_name("variable_default")
+    b_default.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+    b_default.add_int64_power_of_two_axis("Elements{io}", range(16, 28, 4))
+    b_default.add_int64_power_of_two_axis("MaxSegmentSize", range(1, 17, 1))
+    b_default.add_int64_axis("GuaranteedMaxSegSize", [0])
+
+    # Small segments (1–16 items): hint enables warp-level reduction
+    b_small = bench.register(bench_variable_segmented_reduce)
+    b_small.set_name("variable_small_dynamic")
+    b_small.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+    b_small.add_int64_power_of_two_axis("Elements{io}", range(16, 28, 4))
+    b_small.add_int64_power_of_two_axis("MaxSegmentSize", range(1, 5, 1))
+    b_small.add_int64_power_of_two_axis("GuaranteedMaxSegSize", range(1, 5, 1))
+
+    # Medium segments (32–256 items): hint enables warp-level reduction
+    b_medium = bench.register(bench_variable_segmented_reduce)
+    b_medium.set_name("variable_medium_dynamic")
+    b_medium.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+    b_medium.add_int64_power_of_two_axis("Elements{io}", range(16, 28, 4))
+    b_medium.add_int64_power_of_two_axis("MaxSegmentSize", range(5, 9, 1))
+    b_medium.add_int64_power_of_two_axis("GuaranteedMaxSegSize", range(5, 9, 1))
+
+    # Large segments (512+ items): hint enables block-level reduction
+    b_large = bench.register(bench_variable_segmented_reduce)
+    b_large.set_name("variable_large_dynamic")
+    b_large.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+    b_large.add_int64_power_of_two_axis("Elements{io}", range(16, 28, 4))
+    b_large.add_int64_power_of_two_axis("MaxSegmentSize", range(9, 17, 1))
+    b_large.add_int64_power_of_two_axis("GuaranteedMaxSegSize", range(9, 17, 1))
+
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/segmented_sort/keys.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/segmented_sort/keys.py
new file mode 100644
index 00000000..b66c935f
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/segmented_sort/keys.py
@@ -0,0 +1,167 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for segmented_sort keys using cuda.compute.
+
+C++ equivalent: cub/benchmarks/bench/segmented_sort/keys.cu
+
+Notes:
+- Implements three sub-benchmarks: power, small, large
+- Power uses power-law segment sizes with Entropy axis
+- Small/large use uniform segment sizes with MaxSegmentSize axis
+- Migration: uniform offsets use min_segment_size ~ max/2.
+- OffsetT axis is omitted because the Python API does not expose offset type.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+import numpy as np
+from utils import (
+    FUNDAMENTAL_TYPES as TYPE_MAP,
+)
+from utils import (
+    as_cupy_stream,
+    generate_data_with_entropy,
+    generate_power_law_offsets,
+    generate_uniform_segment_offsets,
+)
+
+import cuda.bench as bench
+from cuda.compute import SortOrder, make_segmented_sort
+
+
+def run_segmented_sort(
+    state: bench.State,
+    d_in_keys,
+    d_out_keys,
+    start_offsets,
+    end_offsets,
+    num_items,
+    num_segments,
+):
+    sorter = make_segmented_sort(
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=None,
+        d_out_values=None,
+        start_offsets_in=start_offsets,
+        end_offsets_in=end_offsets,
+        order=SortOrder.ASCENDING,
+    )
+
+    temp_storage_bytes = sorter(
+        temp_storage=None,
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=None,
+        d_out_values=None,
+        num_items=num_items,
+        num_segments=num_segments,
+        start_offsets_in=start_offsets,
+        end_offsets_in=end_offsets,
+    )
+    alloc_stream = as_cupy_stream(state.get_stream())
+    with alloc_stream:
+        temp_storage = cp.empty(temp_storage_bytes, dtype=np.uint8)
+
+    def launcher(launch: bench.Launch):
+        sorter(
+            temp_storage=temp_storage,
+            d_in_keys=d_in_keys,
+            d_out_keys=d_out_keys,
+            d_in_values=None,
+            d_out_values=None,
+            num_items=num_items,
+            num_segments=num_segments,
+            start_offsets_in=start_offsets,
+            end_offsets_in=end_offsets,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False, sync=True)
+
+
+def bench_segmented_sort(state: bench.State, use_power_law: bool):
+    type_str = state.get_string("T{ct}")
+    dtype = TYPE_MAP[type_str]
+    num_elements = int(state.get_int64("Elements{io}"))
+    alloc_stream = as_cupy_stream(state.get_stream())
+
+    if use_power_law:
+        num_segments = int(state.get_int64("Segments{io}"))
+        entropy_str = state.get_string("Entropy")
+    else:
+        max_segment_size = int(state.get_int64("MaxSegmentSize"))
+        min_segment_size = max(1, max_segment_size // 2)
+        entropy_str = "1.000"
+
+    if use_power_law:
+        offsets = generate_power_law_offsets(num_elements, num_segments)
+    else:
+        offsets = generate_uniform_segment_offsets(
+            num_elements, min_segment_size, max_segment_size
+        )
+
+    d_in_keys = generate_data_with_entropy(
+        num_elements, dtype, entropy_str, alloc_stream
+    )
+    with alloc_stream:
+        d_out_keys = cp.empty(num_elements, dtype=dtype)
+
+        start_offsets = cp.asarray(offsets[:-1], dtype=np.int64)
+        end_offsets = cp.asarray(offsets[1:], dtype=np.int64)
+
+    alloc_stream.synchronize()
+    num_segments = int(start_offsets.size)
+
+    state.add_element_count(num_elements)
+    state.add_global_memory_reads(num_elements * d_in_keys.dtype.itemsize)
+    state.add_global_memory_writes(num_elements * d_out_keys.dtype.itemsize)
+    state.add_global_memory_reads((num_segments + 1) * start_offsets.dtype.itemsize)
+
+    run_segmented_sort(
+        state,
+        d_in_keys,
+        d_out_keys,
+        start_offsets,
+        end_offsets,
+        num_elements,
+        num_segments,
+    )
+
+
+def bench_segmented_sort_power(state: bench.State):
+    bench_segmented_sort(state, use_power_law=True)
+
+
+def bench_segmented_sort_uniform(state: bench.State):
+    bench_segmented_sort(state, use_power_law=False)
+
+
+if __name__ == "__main__":
+    b_power = bench.register(bench_segmented_sort_power)
+    b_power.set_name("power")
+    b_power.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+    b_power.add_int64_power_of_two_axis("Elements{io}", range(22, 31, 4))
+    b_power.add_int64_power_of_two_axis("Segments{io}", range(12, 21, 4))
+    b_power.add_string_axis("Entropy", ["1.000", "0.201"])
+
+    b_small = bench.register(bench_segmented_sort_uniform)
+    b_small.set_name("small")
+    b_small.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+    b_small.add_int64_power_of_two_axis("Elements{io}", range(22, 31, 4))
+    b_small.add_int64_power_of_two_axis("MaxSegmentSize", range(1, 9, 1))
+
+    b_large = bench.register(bench_segmented_sort_uniform)
+    b_large.set_name("large")
+    b_large.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+    b_large.add_int64_power_of_two_axis("Elements{io}", range(22, 31, 4))
+    b_large.add_int64_power_of_two_axis("MaxSegmentSize", range(10, 19, 2))
+
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/select/if.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/select/if.py
new file mode 100644
index 00000000..64fb9836
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/select/if.py
@@ -0,0 +1,119 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for select_if using cuda.compute.
+
+C++ equivalent: cub/benchmarks/bench/select/if.cu
+
+Notes:
+- The C++ benchmark uses a `less_then_t<T>` predicate with threshold based on entropy
+- Entropy controls what fraction of elements are selected:
+  - 1.000 → selects ~100% (threshold = max value)
+  - 0.544 → selects ~54.4% (threshold at 54.4% of range)
+  - 0.000 → selects ~0% (threshold = min value)
+- InPlace axis controls whether output can alias input (not exposed in Python API)
+- Migration: Python cannot expose InPlace axis; output is sized to num_elements but metrics use actual selected count.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+import numpy as np
+from utils import (
+    ENTROPY_TO_PROB,
+    as_cupy_stream,
+    generate_data_with_entropy,
+    lerp_min_max,
+)
+from utils import (
+    FUNDAMENTAL_TYPES as TYPE_MAP,
+)
+
+import cuda.bench as bench
+from cuda.compute import make_select
+
+# Entropy values from C++ benchmark
+# These control the selection threshold and thus how many elements are selected
+
+
+def bench_select_if(state: bench.State):
+    type_str = state.get_string("T{ct}")
+    dtype = TYPE_MAP[type_str]
+    num_elements = int(state.get_int64("Elements{io}"))
+    entropy_str = state.get_string("Entropy")
+
+    probability = ENTROPY_TO_PROB[entropy_str]
+    threshold = lerp_min_max(dtype, probability)
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+
+    # Match C++ benchmark: input data generation is independent of Entropy.
+    # Entropy only controls the selection threshold.
+    d_in = generate_data_with_entropy(num_elements, dtype, "1.000", alloc_stream)
+    with alloc_stream:
+        selected_elements = int(cp.count_nonzero(d_in < threshold).get())
+        d_out = cp.empty(selected_elements, dtype=dtype)
+
+        d_num_selected = cp.zeros(1, dtype=np.int64)
+
+    alloc_stream.synchronize()
+
+    # Create predicate: select elements less than threshold
+    # For numba device functions, we need to use the value directly in closure
+    thresh_val = threshold
+
+    def less_than_threshold(x):
+        return x < thresh_val
+
+    selector = make_select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=less_than_threshold,
+    )
+
+    temp_storage_bytes = selector(
+        temp_storage=None,
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=less_than_threshold,
+        num_items=num_elements,
+    )
+    with alloc_stream:
+        temp_storage = cp.empty(temp_storage_bytes, dtype=np.uint8)
+
+    state.add_element_count(num_elements)
+    state.add_global_memory_reads(num_elements * d_in.dtype.itemsize)
+    state.add_global_memory_writes(selected_elements * d_out.dtype.itemsize)
+    state.add_global_memory_writes(1 * d_num_selected.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        selector(
+            temp_storage=temp_storage,
+            d_in=d_in,
+            d_out=d_out,
+            d_num_selected_out=d_num_selected,
+            cond=less_than_threshold,
+            num_items=num_elements,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_select_if)
+    b.set_name("base")
+
+    b.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 29, 4))
+    b.add_string_axis("Entropy", ["1.000", "0.544", "0.000"])
+    # Note: InPlace axis is not exposed in Python API, so we skip it
+
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/select/unique_by_key.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/select/unique_by_key.py
new file mode 100644
index 00000000..8de29055
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/select/unique_by_key.py
@@ -0,0 +1,137 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for unique_by_key using cuda.compute.
+
+C++ equivalent: cub/benchmarks/bench/select/unique_by_key.cu
+
+Notes:
+- The C++ benchmark uses MaxSegSize axis to control segment sizes
+- Uses equal_to comparison operator for key equality
+- Generates key segments with sizes between 1 and MaxSegSize
+- Both keys and values are processed
+- Migration: Python fixes offsets and generates key segments on GPU to mirror C++.
+- OffsetT axis is omitted because the Python API does not expose offset type.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+import numpy as np
+from utils import INTEGRAL_TYPES, SIGNED_TYPES, as_cupy_stream, generate_key_segments
+
+import cuda.bench as bench
+from cuda.compute import OpKind, make_unique_by_key
+
+KEY_TYPE_MAP = INTEGRAL_TYPES
+VALUE_TYPE_MAP = {**SIGNED_TYPES, "C32": np.complex64}
+
+
+def bench_unique_by_key(state: bench.State):
+    key_type_str = state.get_string("KeyT{ct}")
+    value_type_str = state.get_string("ValueT{ct}")
+    key_dtype = KEY_TYPE_MAP[key_type_str]
+    value_dtype = VALUE_TYPE_MAP[value_type_str]
+    num_elements = int(state.get_int64("Elements{io}"))
+    max_seg_size = int(state.get_int64("MaxSegSize"))
+
+    if num_elements > np.iinfo(np.int32).max:
+        state.skip("Skipping: num_elements exceeds int32 limits")
+        return
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+
+    d_in_keys = generate_key_segments(
+        num_elements,
+        key_dtype,
+        min_segment_size=1,
+        max_segment_size=max_seg_size,
+        stream=alloc_stream,
+    )
+
+    with alloc_stream:
+        d_in_values = cp.zeros(num_elements, dtype=value_dtype)
+
+        d_out_keys = cp.empty(num_elements, dtype=key_dtype)
+        d_out_values = cp.empty(num_elements, dtype=value_dtype)
+        d_num_selected = cp.empty(1, dtype=np.int32)
+
+    alloc_stream.synchronize()
+
+    uniquer = make_unique_by_key(
+        d_in_keys=d_in_keys,
+        d_in_items=d_in_values,
+        d_out_keys=d_out_keys,
+        d_out_items=d_out_values,
+        d_out_num_selected=d_num_selected,
+        op=OpKind.EQUAL_TO,
+    )
+
+    temp_storage_bytes = uniquer(
+        temp_storage=None,
+        d_in_keys=d_in_keys,
+        d_in_items=d_in_values,
+        d_out_keys=d_out_keys,
+        d_out_items=d_out_values,
+        d_out_num_selected=d_num_selected,
+        op=OpKind.EQUAL_TO,
+        num_items=num_elements,
+    )
+    with alloc_stream:
+        temp_storage = cp.empty(temp_storage_bytes, dtype=np.uint8)
+
+    # Run once before timing to materialize the number of selected runs,
+    # matching the C++ metric accounting flow.
+    uniquer(
+        temp_storage=temp_storage,
+        d_in_keys=d_in_keys,
+        d_in_items=d_in_values,
+        d_out_keys=d_out_keys,
+        d_out_items=d_out_values,
+        d_out_num_selected=d_num_selected,
+        op=OpKind.EQUAL_TO,
+        num_items=num_elements,
+        stream=alloc_stream,
+    )
+    alloc_stream.synchronize()
+    num_runs = int(d_num_selected.get()[0])
+
+    state.add_element_count(num_elements)
+    state.add_global_memory_reads(int(num_elements * d_in_keys.dtype.itemsize))
+    state.add_global_memory_reads(int(num_elements * d_in_values.dtype.itemsize))
+    state.add_global_memory_writes(int(num_runs * d_out_keys.dtype.itemsize))
+    state.add_global_memory_writes(int(num_runs * d_out_values.dtype.itemsize))
+    state.add_global_memory_writes(int(d_num_selected.dtype.itemsize))
+
+    def launcher(launch: bench.Launch):
+        uniquer(
+            temp_storage=temp_storage,
+            d_in_keys=d_in_keys,
+            d_in_items=d_in_values,
+            d_out_keys=d_out_keys,
+            d_out_items=d_out_values,
+            d_out_num_selected=d_num_selected,
+            op=OpKind.EQUAL_TO,
+            num_items=num_elements,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_unique_by_key)
+    b.set_name("base")
+
+    b.add_string_axis("KeyT{ct}", list(KEY_TYPE_MAP.keys()))
+    b.add_string_axis("ValueT{ct}", list(VALUE_TYPE_MAP.keys()))
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 29, 4))
+    b.add_int64_power_of_two_axis("MaxSegSize", [1, 4, 8])
+    # Note: OffsetT axis from C++ is not exposed in Python API
+
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform/babelstream.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform/babelstream.py
new file mode 100644
index 00000000..5a256af8
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform/babelstream.py
@@ -0,0 +1,237 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for BabelStream operations using cuda.compute transforms.
+
+C++ equivalent: cub/benchmarks/bench/transform/babelstream.cu
+
+Notes:
+- Migration: Python omits OffsetT axis and int128 types.
+- OffsetT axis is omitted because the Python API does not expose offset type.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+from utils import ALL_TYPES as _ALL_TYPES
+from utils import as_cupy_stream
+
+import cuda.bench as bench
+import cuda.compute
+from cuda.compute import ZipIterator
+
+TYPE_MAP = {k: _ALL_TYPES[k] for k in ("I8", "I16", "F32", "F64")}
+
+START_A = 11
+START_B = 2
+START_C = 1
+START_SCALAR = -2
+
+assert START_A == START_A + START_B + START_SCALAR * START_C
+
+
+def _reset_pools():
+    cp.get_default_memory_pool().free_all_blocks()
+    cp.get_default_pinned_memory_pool().free_all_blocks()
+
+
+def bench_mul(state: bench.State):
+    """
+    Benchmark: b[i] = c[i] * scalar
+    Unary transform with scalar multiplication.
+    """
+    type_str = state.get_string("T{ct}")
+    dtype = TYPE_MAP[type_str]
+    num_items = int(state.get_int64("Elements{io}"))
+
+    _reset_pools()
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+    try:
+        with alloc_stream:
+            c = cp.full(num_items, START_C, dtype=dtype)
+            b = cp.full(num_items, START_B, dtype=dtype)
+    except (MemoryError, cp.cuda.memory.OutOfMemoryError):
+        state.skip("Skipping: out of memory.")
+        return
+
+    scalar = dtype(START_SCALAR)
+
+    def mul_op(ci):
+        return ci * scalar
+
+    transform = cuda.compute.make_unary_transform(d_in=c, d_out=b, op=mul_op)
+
+    state.add_element_count(num_items)
+    state.add_global_memory_reads(num_items * c.dtype.itemsize)
+    state.add_global_memory_writes(num_items * b.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        transform(
+            d_in=c, d_out=b, op=mul_op, num_items=num_items, stream=launch.get_stream()
+        )
+
+    state.exec(launcher, batched=False)
+
+
+def bench_add(state: bench.State):
+    """
+    Benchmark: c[i] = a[i] + b[i]
+    Binary transform with addition.
+    """
+    type_str = state.get_string("T{ct}")
+    dtype = TYPE_MAP[type_str]
+    num_items = int(state.get_int64("Elements{io}"))
+
+    _reset_pools()
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+    try:
+        with alloc_stream:
+            a = cp.full(num_items, START_A, dtype=dtype)
+            b = cp.full(num_items, START_B, dtype=dtype)
+            c = cp.full(num_items, START_C, dtype=dtype)
+    except (MemoryError, cp.cuda.memory.OutOfMemoryError):
+        state.skip("Skipping: out of memory.")
+        return
+
+    def add_op(ai, bi):
+        return ai + bi
+
+    transform = cuda.compute.make_binary_transform(d_in1=a, d_in2=b, d_out=c, op=add_op)
+
+    state.add_element_count(num_items)
+    state.add_global_memory_reads(2 * num_items * a.dtype.itemsize)
+    state.add_global_memory_writes(num_items * c.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        transform(
+            d_in1=a,
+            d_in2=b,
+            d_out=c,
+            op=add_op,
+            num_items=num_items,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+def bench_triad(state: bench.State):
+    """
+    Benchmark: a[i] = b[i] + scalar * c[i]
+    Binary transform with fused multiply-add.
+    """
+    type_str = state.get_string("T{ct}")
+    dtype = TYPE_MAP[type_str]
+    num_items = int(state.get_int64("Elements{io}"))
+
+    _reset_pools()
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+    try:
+        with alloc_stream:
+            a = cp.full(num_items, START_A, dtype=dtype)
+            b = cp.full(num_items, START_B, dtype=dtype)
+            c = cp.full(num_items, START_C, dtype=dtype)
+    except (MemoryError, cp.cuda.memory.OutOfMemoryError):
+        state.skip("Skipping: out of memory.")
+        return
+
+    scalar = dtype(START_SCALAR)
+
+    def triad_op(bi, ci):
+        return bi + scalar * ci
+
+    transform = cuda.compute.make_binary_transform(
+        d_in1=b, d_in2=c, d_out=a, op=triad_op
+    )
+
+    state.add_element_count(num_items)
+    state.add_global_memory_reads(2 * num_items * a.dtype.itemsize)
+    state.add_global_memory_writes(num_items * a.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        transform(
+            d_in1=b,
+            d_in2=c,
+            d_out=a,
+            op=triad_op,
+            num_items=num_items,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+def bench_nstream(state: bench.State):
+    """
+    Benchmark: a[i] = a[i] + b[i] + scalar * c[i]
+    Ternary transform using ZipIterator to combine (a, b, c) as input.
+    """
+    type_str = state.get_string("T{ct}")
+    dtype = TYPE_MAP[type_str]
+    num_items = int(state.get_int64("Elements{io}"))
+
+    _reset_pools()
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+    try:
+        with alloc_stream:
+            a = cp.full(num_items, START_A, dtype=dtype)
+            b = cp.full(num_items, START_B, dtype=dtype)
+            c = cp.full(num_items, START_C, dtype=dtype)
+    except (MemoryError, cp.cuda.memory.OutOfMemoryError):
+        state.skip("Skipping: out of memory.")
+        return
+
+    scalar = dtype(START_SCALAR)
+
+    # Use ZipIterator to combine 3 inputs into one for unary transform
+    zip_in = ZipIterator(a, b, c)
+
+    def nstream_op(abc):
+        return abc[0] + abc[1] + scalar * abc[2]
+
+    transform = cuda.compute.make_unary_transform(d_in=zip_in, d_out=a, op=nstream_op)
+
+    state.add_element_count(num_items)
+    state.add_global_memory_reads(3 * num_items * a.dtype.itemsize)
+    state.add_global_memory_writes(num_items * a.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        # Update ZipIterator state for each iteration
+        zip_in_iter = ZipIterator(a, b, c)
+        transform(
+            d_in=zip_in_iter,
+            d_out=a,
+            op=nstream_op,
+            num_items=num_items,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+# Registry of all BabelStream benchmarks
+BENCHMARKS = {
+    "mul": bench_mul,
+    "add": bench_add,
+    "triad": bench_triad,
+    "nstream": bench_nstream,
+}
+
+
+if __name__ == "__main__":
+    for name, bench_fn in BENCHMARKS.items():
+        b = bench.register(bench_fn)
+        b.set_name(name)
+        b.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+        b.add_int64_power_of_two_axis("Elements{io}", range(16, 33, 4))
+
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform/complex_cmp.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform/complex_cmp.py
new file mode 100644
index 00000000..5721e757
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform/complex_cmp.py
@@ -0,0 +1,105 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for complex comparison using cuda.compute.
+
+C++ equivalent: cub/benchmarks/bench/transform/complex_cmp.cu
+
+Notes:
+- Uses two overlapping input ranges (in[0:n-1], in[1:n])
+- Output is boolean array of size n-1
+- Benchmark name is "compare_complex" to match C++
+- Migration: Python uses explicit lexicographic compare for complex64.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import math
+
+import cupy as cp
+import numpy as np
+from utils import as_cupy_stream, generate_data_with_entropy
+
+import cuda.bench as bench
+from cuda.compute import make_binary_transform
+
+_COMPLEX_EPS = np.finfo(np.float32).eps
+_COMPLEX_THRESHOLD = _COMPLEX_EPS * 2.0
+
+
+def less_complex(a, b):
+    mag0 = math.sqrt(a.real * a.real + a.imag * a.imag)
+    mag1 = math.sqrt(b.real * b.real + b.imag * b.imag)
+
+    if math.isnan(mag0) or math.isnan(mag1):
+        return False
+
+    if math.isinf(mag0) or math.isinf(mag1):
+        scaler = 0.5
+        mag0 = math.sqrt(
+            (a.real * scaler) * (a.real * scaler)
+            + (a.imag * scaler) * (a.imag * scaler)
+        )
+        mag1 = math.sqrt(
+            (b.real * scaler) * (b.real * scaler)
+            + (b.imag * scaler) * (b.imag * scaler)
+        )
+
+    if abs(mag0 - mag1) < _COMPLEX_THRESHOLD:
+        phase0 = math.atan2(a.imag, a.real)
+        phase1 = math.atan2(b.imag, b.real)
+        return phase0 < phase1
+
+    return mag0 < mag1
+
+
+def bench_compare_complex(state: bench.State):
+    num_elements = int(state.get_int64("Elements{io}"))
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+    try:
+        with alloc_stream:
+            real = generate_data_with_entropy(
+                num_elements, np.float32, "1.000", alloc_stream
+            )
+            imag = generate_data_with_entropy(
+                num_elements, np.float32, "1.000", alloc_stream
+            )
+            d_in = (real + 1j * imag).astype(np.complex64)
+            d_out = cp.empty(num_elements - 1, dtype=np.bool_)
+    except (MemoryError, cp.cuda.memory.OutOfMemoryError):
+        state.skip("Skipping: out of memory.")
+        return
+
+    num_items = num_elements - 1
+    transformer = make_binary_transform(
+        d_in1=d_in[:-1], d_in2=d_in[1:], d_out=d_out, op=less_complex
+    )
+
+    state.add_element_count(num_elements)
+    state.add_global_memory_reads(num_elements * d_in.dtype.itemsize)
+    state.add_global_memory_writes(num_elements * d_out.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        transformer(
+            d_in1=d_in[:-1],
+            d_in2=d_in[1:],
+            d_out=d_out,
+            op=less_complex,
+            num_items=num_items,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_compare_complex)
+    b.set_name("compare_complex")
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 33, 4))
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform/fib.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform/fib.py
new file mode 100644
index 00000000..7ab77300
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform/fib.py
@@ -0,0 +1,94 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for transform fibonacci using cuda.compute.
+
+C++ equivalent: cub/benchmarks/bench/transform/fib.cu
+
+Notes:
+- Input values are int64 in [0, 42]
+- Output values are uint32
+- Benchmark name is "fibonacci" to match C++
+- Migration: Python fixes offsets to int64.
+- OffsetT axis is omitted because the Python API does not expose offset type.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+import numpy as np
+from utils import as_cupy_stream, generate_data_with_entropy
+
+import cuda.bench as bench
+from cuda.compute import make_unary_transform
+
+
+def fib_op(n):
+    t1 = 0
+    t2 = 1
+
+    if n < 1:
+        return t1
+    if n == 1:
+        return t1
+    if n == 2:
+        return t2
+
+    i = 3
+    while i <= n:
+        next_val = t1 + t2
+        t1 = t2
+        t2 = next_val
+        i += 1
+
+    return t2
+
+
+def bench_transform_fib(state: bench.State):
+    # Axes
+    num_elements = int(state.get_int64("Elements{io}"))
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+    try:
+        with alloc_stream:
+            d_in = generate_data_with_entropy(
+                num_elements,
+                np.int64,
+                "1.000",
+                alloc_stream,
+                min_val=np.int64(0),
+                max_val=np.int64(42),
+            )
+            d_out = cp.empty(num_elements, dtype=np.uint32)
+    except (MemoryError, cp.cuda.memory.OutOfMemoryError):
+        state.skip("Skipping: out of memory.")
+        return
+
+    transformer = make_unary_transform(d_in=d_in, d_out=d_out, op=fib_op)
+
+    state.add_element_count(num_elements)
+    state.add_global_memory_reads(num_elements * d_in.dtype.itemsize)
+    state.add_global_memory_writes(num_elements * d_out.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        transformer(
+            d_in=d_in,
+            d_out=d_out,
+            op=fib_op,
+            num_items=num_elements,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_transform_fib)
+    b.set_name("fibonacci")
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 33, 4))
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform/fill.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform/fill.py
new file mode 100644
index 00000000..db735027
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform/fill.py
@@ -0,0 +1,72 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for fill operation using cuda.compute.ConstantIterator.
+
+C++ equivalent: cub/benchmarks/bench/transform/fill.cu
+
+Notes:
+- Migration: Python matches C++ integral_types (I8-I64); no tune parameters.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+from utils import INTEGRAL_TYPES as TYPE_MAP
+from utils import as_cupy_stream
+
+import cuda.bench as bench
+import cuda.compute
+from cuda.compute import ConstantIterator, OpKind
+
+
+def bench_fill(state: bench.State):
+    type_str = state.get_string("T{ct}")
+    dtype = TYPE_MAP[type_str]
+    num_items = int(state.get_int64("Elements{io}"))
+
+    # Setup data
+    alloc_stream = as_cupy_stream(state.get_stream())
+    try:
+        with alloc_stream:
+            d_out = cp.empty(num_items, dtype=dtype)
+    except (MemoryError, cp.cuda.memory.OutOfMemoryError):
+        state.skip("Skipping: out of memory.")
+        return
+
+    # Python equivalent of C++ return_constant<T>{42}
+    constant_it = ConstantIterator(dtype(42))
+
+    transform = cuda.compute.make_unary_transform(
+        d_in=constant_it, d_out=d_out, op=OpKind.IDENTITY
+    )
+
+    state.add_element_count(num_items)
+    state.add_global_memory_reads(0)
+    state.add_global_memory_writes(num_items * d_out.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        transform(
+            d_in=constant_it,
+            d_out=d_out,
+            op=OpKind.IDENTITY,
+            num_items=num_items,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_fill)
+    b.set_name("fill")
+
+    b.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 33, 4))
+
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform/grayscale.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform/grayscale.py
new file mode 100644
index 00000000..e589e8c0
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform/grayscale.py
@@ -0,0 +1,89 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for transform grayscale using cuda.compute.
+
+C++ equivalent: cub/benchmarks/bench/transform/grayscale.cu
+
+Notes:
+- Input is an RGB struct with three channels
+- Output is grayscale value of the same type
+- Benchmark name is "grayscale" to match C++
+- Migration: Python uses AoS (`gpu_struct`) to mirror C++ `rgb_t<T>`.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+from utils import FLOAT_TYPES as TYPE_MAP
+from utils import as_cupy_stream, generate_data_with_entropy
+
+import cuda.bench as bench
+from cuda.compute import gpu_struct, make_unary_transform
+
+
+def bench_transform_grayscale(state: bench.State):
+    type_str = state.get_string("T{ct}")
+    dtype = TYPE_MAP[type_str]
+    num_elements = int(state.get_int64("Elements{io}"))
+
+    # Grayscale weights
+    w_r = dtype(0.2989)
+    w_g = dtype(0.587)
+    w_b = dtype(0.114)
+
+    RGB = gpu_struct({"r": dtype, "g": dtype, "b": dtype})
+
+    def to_grayscale(pixel: RGB):
+        return w_r * pixel.r + w_g * pixel.g + w_b * pixel.b
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+    try:
+        with alloc_stream:
+            r_data = generate_data_with_entropy(
+                num_elements, dtype, "1.000", alloc_stream
+            )
+            g_data = generate_data_with_entropy(
+                num_elements, dtype, "1.000", alloc_stream
+            )
+            b_data = generate_data_with_entropy(
+                num_elements, dtype, "1.000", alloc_stream
+            )
+            d_pixels = cp.empty(num_elements, dtype=RGB.dtype)
+            d_pixels["r"] = r_data
+            d_pixels["g"] = g_data
+            d_pixels["b"] = b_data
+            d_out = cp.empty(num_elements, dtype=dtype)
+    except (MemoryError, cp.cuda.memory.OutOfMemoryError):
+        state.skip("Skipping: out of memory.")
+        return
+
+    transformer = make_unary_transform(d_in=d_pixels, d_out=d_out, op=to_grayscale)
+
+    state.add_element_count(num_elements)
+    state.add_global_memory_reads(num_elements * d_pixels.dtype.itemsize)
+    state.add_global_memory_writes(num_elements * d_out.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        transformer(
+            d_in=d_pixels,
+            d_out=d_out,
+            op=to_grayscale,
+            num_items=num_elements,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_transform_grayscale)
+    b.set_name("grayscale")
+    b.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 33, 4))
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform/heavy.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform/heavy.py
new file mode 100644
index 00000000..27ba1781
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform/heavy.py
@@ -0,0 +1,138 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Python benchmark for heavy transform using cuda.compute.
+
+C++ equivalent: cub/benchmarks/bench/transform/heavy.cu
+
+Notes:
+- Migration: Python uses Numba local arrays to emulate register pressure.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+import numba
+import numpy as np
+from numba import cuda as lang
+from utils import as_cupy_stream, generate_data_with_entropy
+
+import cuda.bench as bench
+import cuda.compute
+
+
+def _heavy_op_32(data):
+    reg = lang.local.array(shape=32, dtype=numba.uint32)
+    reg[0] = data
+    for i in range(1, 32):
+        x = reg[i - 1]
+        reg[i] = x * x + 1
+    for i in range(32):
+        x = reg[i]
+        reg[i] = (x * x) % 19
+    for i in range(32):
+        reg[i] = reg[32 - i - 1] * reg[i]
+    out = data - data  # uint32(0)
+    for i in range(32):
+        out += reg[i]
+    return out
+
+
+def _heavy_op_64(data):
+    reg = lang.local.array(shape=64, dtype=numba.uint32)
+    reg[0] = data
+    for i in range(1, 64):
+        x = reg[i - 1]
+        reg[i] = x * x + 1
+    for i in range(64):
+        x = reg[i]
+        reg[i] = (x * x) % 19
+    for i in range(64):
+        reg[i] = reg[64 - i - 1] * reg[i]
+    out = data - data
+    for i in range(64):
+        out += reg[i]
+    return out
+
+
+def _heavy_op_128(data):
+    reg = lang.local.array(shape=128, dtype=numba.uint32)
+    reg[0] = data
+    for i in range(1, 128):
+        x = reg[i - 1]
+        reg[i] = x * x + 1
+    for i in range(128):
+        x = reg[i]
+        reg[i] = (x * x) % 19
+    for i in range(128):
+        reg[i] = reg[128 - i - 1] * reg[i]
+    out = data - data
+    for i in range(128):
+        out += reg[i]
+    return out
+
+
+def _heavy_op_256(data):
+    reg = lang.local.array(shape=256, dtype=numba.uint32)
+    reg[0] = data
+    for i in range(1, 256):
+        x = reg[i - 1]
+        reg[i] = x * x + 1
+    for i in range(256):
+        x = reg[i]
+        reg[i] = (x * x) % 19
+    for i in range(256):
+        reg[i] = reg[256 - i - 1] * reg[i]
+    out = data - data
+    for i in range(256):
+        out += reg[i]
+    return out
+
+
+_HEAVY_OPS = {
+    32: _heavy_op_32,
+    64: _heavy_op_64,
+    128: _heavy_op_128,
+    256: _heavy_op_256,
+}
+
+
+def bench_heavy(state: bench.State):
+    # Axes
+    n_regs = int(state.get_string("Heaviness{ct}"))
+    size = int(state.get_int64("Elements{io}"))
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+    try:
+        with alloc_stream:
+            d_in = generate_data_with_entropy(size, np.uint32, "1.000", alloc_stream)
+            d_out = cp.empty(size, dtype=np.uint32)
+    except (MemoryError, cp.cuda.memory.OutOfMemoryError):
+        state.skip("Skipping: out of memory.")
+        return
+
+    op = _HEAVY_OPS[n_regs]
+    transform = cuda.compute.make_unary_transform(d_in=d_in, d_out=d_out, op=op)
+
+    state.add_element_count(size)
+    state.add_global_memory_reads(size * d_in.dtype.itemsize)
+    state.add_global_memory_writes(size * d_out.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        transform(
+            d_in=d_in, d_out=d_out, op=op, num_items=size, stream=launch.get_stream()
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_heavy)
+    b.set_name("heavy")
+    b.add_string_axis("Heaviness{ct}", [str(v) for v in (32, 64, 128, 256)])
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 33, 4))
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform_reduce/sum.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform_reduce/sum.py
new file mode 100644
index 00000000..6e04d3eb
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/transform_reduce/sum.py
@@ -0,0 +1,86 @@
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Python benchmark for transform reduce sum using cuda.compute.
+
+C++ equivalent: cub/benchmarks/bench/transform_reduce/sum.cu
+
+Notes:
+- Uses TransformIterator with a square operation
+- OffsetT axis from C++ is fixed to Python default (int64)
+- Migration: Python fixes offsets and omits int128/complex types.
+- OffsetT axis is omitted because the Python API does not expose offset type.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import cupy as cp
+import numpy as np
+from utils import SIGNED_TYPES as TYPE_MAP
+from utils import as_cupy_stream, generate_data_with_entropy
+
+import cuda.bench as bench
+from cuda.compute import OpKind, TransformIterator, make_reduce_into
+
+
+def square_op(x):
+    return x * x
+
+
+def bench_transform_reduce_sum(state: bench.State):
+    type_str = state.get_string("T{ct}")
+    dtype = TYPE_MAP[type_str]
+    num_items = int(state.get_int64("Elements{io}"))
+
+    alloc_stream = as_cupy_stream(state.get_stream())
+    with alloc_stream:
+        d_in = generate_data_with_entropy(num_items, dtype, "1.000", alloc_stream)
+        d_out = cp.empty(1, dtype=dtype)
+
+    transform_it = TransformIterator(d_in, square_op)
+    h_init = np.zeros(1, dtype=dtype)
+
+    reducer = make_reduce_into(
+        d_in=transform_it, d_out=d_out, op=OpKind.PLUS, h_init=h_init
+    )
+
+    temp_storage_bytes = reducer(
+        temp_storage=None,
+        d_in=transform_it,
+        d_out=d_out,
+        num_items=num_items,
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+    with alloc_stream:
+        temp_storage = cp.empty(temp_storage_bytes, dtype=np.uint8)
+
+    state.add_element_count(num_items)
+    state.add_global_memory_reads(num_items * d_in.dtype.itemsize, "Size")
+    state.add_global_memory_writes(1 * d_out.dtype.itemsize)
+
+    def launcher(launch: bench.Launch):
+        reducer(
+            temp_storage=temp_storage,
+            d_in=transform_it,
+            d_out=d_out,
+            num_items=num_items,
+            op=OpKind.PLUS,
+            h_init=h_init,
+            stream=launch.get_stream(),
+        )
+
+    state.exec(launcher, batched=False)
+
+
+if __name__ == "__main__":
+    b = bench.register(bench_transform_reduce_sum)
+    b.set_name("base")
+    b.add_string_axis("T{ct}", list(TYPE_MAP.keys()))
+    b.add_int64_power_of_two_axis("Elements{io}", range(16, 29, 4))
+
+    bench.run_all_benchmarks(sys.argv)
diff --git a/cccl_upstream/python/cuda_cccl/benchmarks/compute/utils.py b/cccl_upstream/python/cuda_cccl/benchmarks/compute/utils.py
new file mode 100644
index 00000000..3b163db4
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/benchmarks/compute/utils.py
@@ -0,0 +1,252 @@
+import cupy as cp
+import numpy as np
+
+import cuda.bench as bench
+
+ALL_TYPES = {
+    "I8": np.int8,
+    "I16": np.int16,
+    "I32": np.int32,
+    "I64": np.int64,
+    "U8": np.uint8,
+    "U16": np.uint16,
+    "U32": np.uint32,
+    "U64": np.uint64,
+    "F32": np.float32,
+    "F64": np.float64,
+}
+
+SIGNED_TYPES = {k: ALL_TYPES[k] for k in ("I8", "I16", "I32", "I64", "F32", "F64")}
+FLOAT_TYPES = {k: ALL_TYPES[k] for k in ("F32", "F64")}
+
+# Matches C++ integral_types = {int8_t, int16_t, int32_t, int64_t}
+INTEGRAL_TYPES = {k: ALL_TYPES[k] for k in ("I8", "I16", "I32", "I64")}
+
+# Matches C++ fundamental_types = {int8..int64, [int128,] float, double}
+# int128 is excluded because it is not supported by numpy/cupy.
+FUNDAMENTAL_TYPES = {k: ALL_TYPES[k] for k in ("I8", "I16", "I32", "I64", "F32", "F64")}
+
+ENTROPY_TO_STEPS = {
+    "1.000": 0,
+    "0.811": 1,
+    "0.544": 2,
+    "0.337": 3,
+    "0.201": 4,
+    "0.000": 0,
+}
+
+ENTROPY_TO_PROB = {
+    "1.000": 1.0,
+    "0.811": 0.811,
+    "0.544": 0.544,
+    "0.337": 0.337,
+    "0.201": 0.201,
+    "0.000": 0.0,
+}
+
+
+def as_cupy_stream(cs: bench.CudaStream) -> cp.cuda.Stream:
+    """Convert nvbench CudaStream to CuPy Stream."""
+    return cp.cuda.ExternalStream(cs.addressof())
+
+
+def lerp_min_max(dtype, probability):
+    """Interpolate between min/max for dtype like nvbench_helper.cuh."""
+    if probability == 1.0:
+        if np.issubdtype(dtype, np.integer):
+            return np.iinfo(dtype).max
+        return np.finfo(dtype).max
+
+    if np.issubdtype(dtype, np.integer):
+        min_val = float(np.iinfo(dtype).min)
+        max_val = float(np.iinfo(dtype).max)
+    else:
+        min_val = float(np.finfo(dtype).min)
+        max_val = float(np.finfo(dtype).max)
+
+    return dtype(min_val + probability * (max_val - min_val))
+
+
+def _bitwise_and(a, b, dtype):
+    if np.issubdtype(dtype, np.floating):
+        view_dtype = cp.uint32 if dtype == np.float32 else cp.uint64
+        return (a.view(view_dtype) & b.view(view_dtype)).view(dtype)
+    return a & b
+
+
+def _uniform_random(num_elements, dtype, min_val, max_val):
+    rand = cp.random.random(num_elements)
+    if np.issubdtype(dtype, np.floating):
+        return ((float(max_val) - float(min_val)) * rand + float(min_val)).astype(dtype)
+    min_f = float(min_val)
+    max_f = float(max_val)
+    return cp.floor((max_f - min_f + 1) * rand + min_f).astype(dtype)
+
+
+def generate_data_with_entropy(
+    num_elements, dtype, entropy_str, stream, min_val=None, max_val=None
+):
+    """Generate data with nvbench_helper-style bit entropy."""
+    if min_val is None or max_val is None:
+        if np.issubdtype(dtype, np.integer):
+            info = np.iinfo(dtype)
+            default_min = info.min
+            default_max = info.max
+        else:
+            info = np.finfo(dtype)
+            default_min = info.tiny
+            default_max = info.max
+        min_val = default_min if min_val is None else min_val
+        max_val = default_max if max_val is None else max_val
+
+    steps = ENTROPY_TO_STEPS[entropy_str]
+
+    with stream:
+        if entropy_str == "0.000":
+            scalar = _uniform_random(1, dtype, min_val, max_val)[0]
+            data = cp.full(num_elements, scalar, dtype=dtype)
+        else:
+            data = _uniform_random(num_elements, dtype, min_val, max_val)
+            for _ in range(steps):
+                tmp = _uniform_random(num_elements, dtype, min_val, max_val)
+                data = _bitwise_and(data, tmp, dtype)
+
+    return data
+
+
+def generate_uniform_segment_offsets(num_elements, min_segment_size, max_segment_size):
+    num_elements = int(num_elements)
+    if min_segment_size <= 0:
+        raise ValueError("min_segment_size must be positive")
+    if max_segment_size < min_segment_size:
+        raise ValueError("max_segment_size must be >= min_segment_size")
+
+    num_segments_est = int(np.ceil(num_elements / min_segment_size))
+    if num_segments_est > np.iinfo(np.int32).max:
+        raise MemoryError("Too many segments for int32 offsets")
+
+    sizes = cp.random.randint(
+        min_segment_size,
+        max_segment_size + 1,
+        size=num_segments_est,
+        dtype=cp.int64,
+    )
+    cumsum = cp.cumsum(sizes)
+    cutoff = int(
+        cp.searchsorted(
+            cumsum, cp.asarray(num_elements, dtype=cp.int64), side="left"
+        ).item()
+    )
+    sizes = sizes[: cutoff + 1]
+    prev = 0 if cutoff == 0 else int(cumsum[cutoff - 1].item())
+    sizes[cutoff] = num_elements - prev
+
+    offsets = cp.empty(cutoff + 2, dtype=cp.int64)
+    offsets[0] = 0
+    offsets[1:] = cp.cumsum(sizes)
+    offsets[-1] = num_elements
+    return offsets
+
+
+def generate_power_law_offsets(num_elements, num_segments):
+    if num_segments <= 0:
+        return cp.asarray([0, num_elements], dtype=cp.int64)
+
+    # Mirror nvbench_helper power-law generation:
+    # draw log-normal samples, normalize to total elements,
+    # floor to integer segment sizes, then distribute remainder
+    # across the first `diff` segments.
+    samples = cp.random.lognormal(3.0, 1.2, size=num_segments)
+    if int(cp.count_nonzero(samples).item()) == 0:
+        samples = cp.ones(num_segments, dtype=cp.float64)
+
+    sample_sum = float(samples.sum().item())
+    sizes = cp.floor(samples * num_elements / sample_sum).astype(cp.int64)
+
+    diff = int(num_elements - sizes.sum().item())
+    if diff > 0:
+        sizes[:diff] += 1
+
+    offsets = cp.empty(num_segments + 1, dtype=cp.int64)
+    offsets[0] = 0
+    offsets[1:] = cp.cumsum(sizes)
+    return offsets
+
+
+def generate_fixed_segment_offsets(num_elements, segment_size, stream):
+    num_segments = max(1, num_elements // segment_size)
+    actual_elements = num_segments * segment_size
+
+    with stream:
+        start_offsets = cp.arange(0, actual_elements, segment_size, dtype=np.int64)
+        end_offsets = cp.arange(
+            segment_size, actual_elements + 1, segment_size, dtype=np.int64
+        )
+
+    return start_offsets, end_offsets, num_segments, actual_elements
+
+
+def generate_key_segments(
+    num_elements, key_dtype, min_segment_size, max_segment_size, stream
+):
+    """Generate GPU key segments (runs of equal keys) matching C++ generate.uniform.key_segments.
+
+    All computation stays on GPU via CuPy. We avoid ``cp.repeat`` (which
+    doesn't accept a device array for *repeats*) by building a segment-id
+    array through ``cp.searchsorted`` on cumulative offsets instead.
+    """
+    num_elements = int(num_elements)
+    if min_segment_size <= 0:
+        raise ValueError("min_segment_size must be positive")
+    if max_segment_size < min_segment_size:
+        raise ValueError("max_segment_size must be >= min_segment_size")
+
+    num_segments_est = int(np.ceil(num_elements / min_segment_size))
+    if num_segments_est > np.iinfo(np.int32).max:
+        raise MemoryError("Too many segments for int32 offsets")
+
+    with stream:
+        sizes = cp.random.randint(
+            min_segment_size,
+            max_segment_size + 1,
+            size=num_segments_est,
+            dtype=cp.int64,
+        )
+        cumsum = cp.cumsum(sizes)
+
+        # Find how many full segments fit within num_elements
+        cutoff = int(
+            cp.searchsorted(
+                cumsum, cp.asarray(num_elements, dtype=cp.int64), side="left"
+            ).item()
+        )
+        sizes = sizes[: cutoff + 1]
+        prev = 0 if cutoff == 0 else int(cumsum[cutoff - 1].item())
+        sizes[cutoff] = num_elements - prev
+
+        # Build cumulative offsets for the final segments
+        offsets = cp.empty(cutoff + 2, dtype=cp.int64)
+        offsets[0] = 0
+        offsets[1:] = cp.cumsum(sizes)
+
+        # Instead of cp.repeat (which doesn't support device repeats),
+        # use searchsorted to map each element index to its segment id.
+        indices = cp.arange(num_elements, dtype=cp.int64)
+        # searchsorted(offsets[1:], indices, side="right") gives the segment id
+        segment_ids = cp.searchsorted(offsets[1:], indices, side="right")
+
+        # Map segment ids to key values, wrapping within dtype range
+        if np.issubdtype(key_dtype, np.integer):
+            info = np.iinfo(key_dtype)
+            if np.dtype(key_dtype).itemsize < 8:
+                range_size = int(info.max) - int(info.min) + 1
+                keys = ((segment_ids % range_size) + int(info.min)).astype(
+                    key_dtype, copy=False
+                )
+            else:
+                # For int64, avoid Python overflow: just cast directly
+                keys = segment_ids.astype(key_dtype, copy=False)
+        else:
+            keys = segment_ids.astype(key_dtype, copy=False)
+
+    return keys
diff --git a/cccl_upstream/python/cuda_cccl/cuda/cccl/__init__.py b/cccl_upstream/python/cuda_cccl/cuda/cccl/__init__.py
new file mode 100644
index 00000000..96baccb1
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/cccl/__init__.py
@@ -0,0 +1,27 @@
+"""
+CUDA Core Library (CCCL) Python Package
+"""
+
+import importlib.metadata
+
+try:
+    __version__ = importlib.metadata.version("cuda-cccl")
+except Exception:
+    __version__ = "0.0.0"
+
+from .headers.include_paths import get_include_paths
+
+# cuda.bindings is required, but instead of being listed as a required dependency,
+# it is installed via an extra (e.g., [cu12] or [cu13]).
+#
+# One of the first things we should do is check that it is available, and raise
+# a helpful error message if it is not.
+try:
+    import cuda.bindings as _cuda_bindings  # type: ignore
+except ImportError:
+    raise ImportError(
+        "cuda.bindings is not installed. Please install the appropriate extra cuda-cccl[cu12] or cuda-cccl[cu13]."
+    ) from None
+del _cuda_bindings
+
+__all__ = ["get_include_paths", "__version__"]
diff --git a/cccl_upstream/python/cuda_cccl/cuda/cccl/_cuda_version_utils.py b/cccl_upstream/python/cuda_cccl/cuda/cccl/_cuda_version_utils.py
new file mode 100644
index 00000000..afcac5cc
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/cccl/_cuda_version_utils.py
@@ -0,0 +1,24 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+CUDA version detection utilities shared across the cccl package.
+"""
+
+from typing import Optional
+
+import cuda.bindings
+
+
+def detect_cuda_version() -> Optional[int]:
+    cuda_version = cuda.bindings.__version__
+    return int(cuda_version.split(".")[0])
+
+
+def get_recommended_extra(cuda_version: Optional[int]) -> str:
+    """Get the recommended pip extra for the detected CUDA version."""
+    if cuda_version == 13:
+        return "cu13"
+    else:
+        return "cu12"
diff --git a/cccl_upstream/python/cuda_cccl/cuda/cccl/headers/__init__.py b/cccl_upstream/python/cuda_cccl/cuda/cccl/headers/__init__.py
new file mode 100644
index 00000000..9765502b
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/cccl/headers/__init__.py
@@ -0,0 +1,7 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from .include_paths import get_include_paths
+
+__all__ = ["__version__", "get_include_paths"]
diff --git a/cccl_upstream/python/cuda_cccl/cuda/cccl/headers/include/__init__.py b/cccl_upstream/python/cuda_cccl/cuda/cccl/headers/include/__init__.py
new file mode 100644
index 00000000..bb7b160d
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/cccl/headers/include/__init__.py
@@ -0,0 +1 @@
+# Intentionally empty
diff --git a/cccl_upstream/python/cuda_cccl/cuda/cccl/headers/include_paths.py b/cccl_upstream/python/cuda_cccl/cuda/cccl/headers/include_paths.py
new file mode 100644
index 00000000..85bb6e6e
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/cccl/headers/include_paths.py
@@ -0,0 +1,51 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import sys
+from dataclasses import dataclass
+from functools import lru_cache
+from importlib.resources import as_file, files
+from pathlib import Path
+from typing import Optional
+
+# type: ignore[import-not-found]
+from cuda.pathfinder import find_nvidia_header_directory
+
+
+@dataclass
+class IncludePaths:
+    cuda: Optional[Path]
+    libcudacxx: Optional[Path]
+    cub: Optional[Path]
+    thrust: Optional[Path]
+
+    def as_tuple(self):
+        # Note: higher-level ... lower-level order:
+        return (self.thrust, self.cub, self.libcudacxx, self.cuda)
+
+
+@lru_cache()
+def get_include_paths(probe_file: str = "cub/version.cuh") -> IncludePaths:
+    cuda_incl = find_nvidia_header_directory("cudart")
+    if cuda_incl is None:
+        raise RuntimeError("Unable to locate CUDA include directory.")
+
+    with as_file(files("cuda.cccl.headers.include")) as f:
+        cccl_incl = Path(f)
+
+    probe_file_path = Path(probe_file)
+    if not (cccl_incl / probe_file_path).exists():
+        for sp in sys.path:
+            cccl_incl = Path(sp).resolve() / "cuda" / "cccl" / "headers" / "include"
+            if (cccl_incl / probe_file_path).exists():
+                break
+        else:
+            raise RuntimeError("Unable to locate CCCL include directory.")
+
+    return IncludePaths(
+        cuda=cuda_incl,
+        libcudacxx=cccl_incl,
+        cub=cccl_incl,
+        thrust=cccl_incl,
+    )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/cccl/parallel/__init__.py b/cccl_upstream/python/cuda_cccl/cuda/cccl/parallel/__init__.py
new file mode 100644
index 00000000..d4514af1
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/cccl/parallel/__init__.py
@@ -0,0 +1,9 @@
+# Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License  -Identifier: Apache-2.0 WITH LLVM-exception
+
+from . import experimental
+
+__all__ = [
+    "experimental",
+]
diff --git a/cccl_upstream/python/cuda_cccl/cuda/cccl/parallel/experimental/__init__.py b/cccl_upstream/python/cuda_cccl/cuda/cccl/parallel/experimental/__init__.py
new file mode 100644
index 00000000..037901e1
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/cccl/parallel/experimental/__init__.py
@@ -0,0 +1,24 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# alias for backwards compatibility
+
+from warnings import warn
+
+from cuda.compute import *  # noqa: F403
+
+warn(
+    "The module cuda.cccl.parallel.experimental is deprecated. Use cuda.compute instead.",
+    FutureWarning,
+)
diff --git a/cccl_upstream/python/cuda_cccl/cuda/cccl/py.typed b/cccl_upstream/python/cuda_cccl/cuda/cccl/py.typed
new file mode 100644
index 00000000..e69de29b
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/__init__.py b/cccl_upstream/python/cuda_cccl/cuda/compute/__init__.py
new file mode 100644
index 00000000..50becb77
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/__init__.py
@@ -0,0 +1,162 @@
+# Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+
+# When built against the v2 (HostJIT) backend, the JIT loads Clang's CUDA
+# headers and our cuda_minimal stubs from paths that don't exist on the
+# user's machine. The wheel bundles both under cuda/cccl/headers/{clang,…};
+# point hostjit at them via the env vars its detectDefaultConfig() reads.
+# Only sets vars that aren't already configured by the user, and skips
+# silently if the bundled directories are absent (e.g. v1 builds).
+def _configure_hostjit_paths() -> None:
+    import os
+    from pathlib import Path
+
+    try:
+        from ._build_info import USING_V2  # type: ignore[import-not-found]
+    except ImportError:
+        return
+    if not USING_V2:
+        return
+
+    # Probe for actual file presence, not just directory existence: editable
+    # (`pip install -e`) installs leave behind empty placeholder dirs in the
+    # source tree (with just `__pycache__`), so `is_dir()` succeeds but the
+    # bundled headers are absent. In that case, leave the env vars unset and
+    # let the C library use its build-time CLANG_HEADERS_DIR / HOSTJIT_INCLUDE_DIR
+    # macros (pointing at the LLVM source tree under the CMake build dir).
+    headers_dir = Path(__file__).resolve().parent.parent / "cccl" / "headers"
+    clang_dir = headers_dir / "clang"
+    if (
+        clang_dir / "__clang_cuda_math_forward_declares.h"
+    ).is_file() and not os.environ.get("HOSTJIT_CLANG_PATH"):
+        os.environ["HOSTJIT_CLANG_PATH"] = str(clang_dir)
+    if (
+        headers_dir / "hostjit" / "cuda_minimal" / "__clang_cuda_runtime_wrapper.h"
+    ).is_file() and not os.environ.get("HOSTJIT_INCLUDE_PATH"):
+        os.environ["HOSTJIT_INCLUDE_PATH"] = str(headers_dir)
+
+
+_configure_hostjit_paths()
+
+from ._bindings import _BINDINGS_AVAILABLE  # type: ignore[attr-defined]  # noqa: E402
+
+if not _BINDINGS_AVAILABLE:
+    __all__ = ["_BINDINGS_AVAILABLE"]
+
+    def __getattr__(name):
+        raise AttributeError(
+            f"Cannot access 'cuda.compute.{name}' because CUDA bindings are not available."
+            "This typically means you're running on a CPU-only machine without CUDA drivers installed."
+        )
+else:
+    from ._caching import clear_all_caches
+    from ._proxy import ProxyArray, ProxyValue
+    from .algorithms import (
+        DoubleBuffer,
+        SortOrder,
+        binary_transform,
+        deserialize,
+        exclusive_scan,
+        histogram_even,
+        inclusive_scan,
+        lower_bound,
+        make_binary_transform,
+        make_exclusive_scan,
+        make_histogram_even,
+        make_inclusive_scan,
+        make_lower_bound,
+        make_merge_sort,
+        make_radix_sort,
+        make_reduce_into,
+        make_segmented_reduce,
+        make_segmented_sort,
+        make_select,
+        make_three_way_partition,
+        make_unary_transform,
+        make_unique_by_key,
+        make_upper_bound,
+        merge_sort,
+        radix_sort,
+        reduce_into,
+        segmented_reduce,
+        segmented_sort,
+        select,
+        serialize,
+        three_way_partition,
+        unary_transform,
+        unique_by_key,
+        upper_bound,
+    )
+    from .determinism import Determinism
+    from .iterators import (
+        CacheModifiedInputIterator,
+        ConstantIterator,
+        CountingIterator,
+        DiscardIterator,
+        PermutationIterator,
+        ReverseIterator,
+        ShuffleIterator,
+        TransformIterator,
+        TransformOutputIterator,
+        ZipIterator,
+    )
+    from .op import OpKind
+    from .struct import gpu_struct
+
+    __all__ = [
+        "_BINDINGS_AVAILABLE",
+        "serialize",
+        "deserialize",
+        "ProxyArray",
+        "ProxyValue",
+        "binary_transform",
+        "clear_all_caches",
+        "CacheModifiedInputIterator",
+        "ConstantIterator",
+        "CountingIterator",
+        "DiscardIterator",
+        "DoubleBuffer",
+        "exclusive_scan",
+        "gpu_struct",
+        "histogram_even",
+        "inclusive_scan",
+        "lower_bound",
+        "make_binary_transform",
+        "make_exclusive_scan",
+        "make_select",
+        "make_histogram_even",
+        "make_inclusive_scan",
+        "make_lower_bound",
+        "make_merge_sort",
+        "make_radix_sort",
+        "make_reduce_into",
+        "make_segmented_reduce",
+        "make_segmented_sort",
+        "make_three_way_partition",
+        "make_unary_transform",
+        "make_unique_by_key",
+        "make_upper_bound",
+        "merge_sort",
+        "OpKind",
+        "Determinism",
+        "PermutationIterator",
+        "radix_sort",
+        "reduce_into",
+        "ReverseIterator",
+        "ShuffleIterator",
+        "segmented_reduce",
+        "segmented_sort",
+        "select",
+        "SortOrder",
+        "TransformIterator",
+        "TransformOutputIterator",
+        "three_way_partition",
+        "unary_transform",
+        "unique_by_key",
+        "upper_bound",
+        "ZipIterator",
+    ]
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings.py b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings.py
new file mode 100644
index 00000000..57d21c6c
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings.py
@@ -0,0 +1,87 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# _bindings.py is a shim module that imports symbols from a
+# _bindings_impl extension module. The shim serves the following purposes:
+#
+# 1. Import a CUDA-specific extension. The cuda.cccl wheel ships with multiple
+#    extensions, one for each CUDA version. At runtime, this shim chooses the
+#    appropriate extension based on the detected CUDA version, and imports all
+#    symbols from it.
+#
+# 2. Preload `nvrtc` and `nvJitLink` before importing the extension.
+#    These shared libraries are indirect dependencies, pulled in via the direct
+#    dependency `cccl.c.parallel`. To ensure reliable symbol resolution at
+#    runtime, we explicitly load them first using `cuda.pathfinder`.
+#    Without this step, importing the Cython extension directly may fail or behave
+#    inconsistently depending on environment setup and dynamic linker behavior.
+#    This indirection ensures the right loading order, regardless of how
+#    `_bindings` is first imported across the codebase.
+#
+# 3. On Windows, add the directory containing cccl.c.parallel's dependent DLL
+#    (e.g. cuda/cccl/parallel/experimental/cu13/_bindings_impl.cp312-win_amd64.pyd)
+#    to the current process's DLL search path using `os.add_dll_directory`.
+
+from __future__ import annotations
+
+import importlib
+import os
+
+from cuda.cccl._cuda_version_utils import detect_cuda_version, get_recommended_extra
+from cuda.pathfinder import (  # type: ignore[import-not-found]
+    load_nvidia_dynamic_lib,
+)
+
+
+def _load_cuda_libraries():
+    # Load appropriate libraries for the detected CUDA version
+    for libname in ("nvrtc", "nvJitLink"):
+        load_nvidia_dynamic_lib(libname)
+
+
+_load_cuda_libraries()
+
+
+# Import the appropriate bindings implementation depending on what
+# CUDA version is available:
+cuda_version = detect_cuda_version()
+if cuda_version not in [12, 13]:
+    raise RuntimeError(
+        f"Unsupported CUDA version: {cuda_version}. Only CUDA 12 and 13 are supported."
+    )
+
+# `extra_name` is one of "cu12", "cu13", etc.
+extra_name = get_recommended_extra(cuda_version)
+module_suffix = f".{extra_name}._bindings_impl"
+module_fullname = __package__ + module_suffix
+
+# On Windows, ensure the dependent DLLs next to the extension are discoverable.
+# The extension lives at .../experimental/<extra_name>/_bindings_impl.*.pyd
+# and its dependent DLLs are under .../experimental/<extra_name>/cccl/.
+if os.name == "nt":
+    spec = importlib.util.find_spec(module_fullname)
+    if spec and spec.origin:
+        dll_dir = os.path.join(os.path.dirname(spec.origin), "cccl")
+        if os.path.isdir(dll_dir):
+            # Assign the DLL directory handle to a global such that it stays
+            # alive for the lifetime of this module (and thus, keeps the DLL
+            # directory in the search path).
+            try:
+                _cccl_dll_dir_handle = os.add_dll_directory(dll_dir)  # noqa: F841
+            except Exception:
+                pass
+
+_BINDINGS_AVAILABLE = False
+
+try:
+    bindings_module = importlib.import_module(module_suffix, __package__)
+    # Import all symbols from the module
+    globals().update(bindings_module.__dict__)
+    _BINDINGS_AVAILABLE = True
+except ImportError as e:
+    import warnings
+
+    warnings.warn(
+        f"CUDA CCCL bindings for CUDA {cuda_version} not available: {e}",
+        RuntimeWarning,
+    )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings.pyi b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings.pyi
new file mode 100644
index 00000000..b59aac1d
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings.pyi
@@ -0,0 +1,658 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import ctypes
+from enum import IntEnum
+from typing import Any, Optional
+
+from typing_extensions import Buffer
+
+class OpKind(IntEnum):
+    _value_: int
+    STATELESS = ...
+    STATEFUL = ...
+    PLUS = ...
+    MINUS = ...
+    MULTIPLIES = ...
+    DIVIDES = ...
+    MODULUS = ...
+    EQUAL_TO = ...
+    NOT_EQUAL_TO = ...
+    GREATER = ...
+    LESS = ...
+    GREATER_EQUAL = ...
+    LESS_EQUAL = ...
+    LOGICAL_AND = ...
+    LOGICAL_OR = ...
+    LOGICAL_NOT = ...
+    BIT_AND = ...
+    BIT_OR = ...
+    BIT_XOR = ...
+    BIT_NOT = ...
+    IDENTITY = ...
+    NEGATE = ...
+    MINIMUM = ...
+    MAXIMUM = ...
+
+class TypeEnum(IntEnum):
+    _value_: int
+    INT8 = ...
+    INT16 = ...
+    INT32 = ...
+    INT64 = ...
+    UINT8 = ...
+    UINT16 = ...
+    UINT32 = ...
+    UINT64 = ...
+    FLOAT16 = ...
+    FLOAT32 = ...
+    FLOAT64 = ...
+    STORAGE = ...
+    BOOLEAN = ...
+
+class IteratorKind(IntEnum):
+    _value_: int
+    POINTER = ...
+    ITERATOR = ...
+
+class SortOrder(IntEnum):
+    _value_: int
+    ASCENDING = ...
+    DESCENDING = ...
+
+class InitKind(IntEnum):
+    _value_: int
+    NO_INIT = ...
+    FUTURE_VALUE_INIT = ...
+    VALUE_INIT = ...
+
+class Determinism(IntEnum):
+    _value_: int
+    NOT_GUARANTEED = ...
+    RUN_TO_RUN = ...
+    GPU_TO_GPU = ...
+
+class BinarySearchMode(IntEnum):
+    _value_: int
+    LOWER_BOUND = ...
+    UPPER_BOUND = ...
+
+class Op:
+    def __init__(
+        self,
+        name: Optional[str] = ...,
+        operator_type: OpKind = ...,
+        ltoir=None,
+        state=None,
+        state_alignment: int = 1,
+        extra_ltoirs=None,
+    ): ...
+    @property
+    def state(self) -> bytes: ...
+    @state.setter
+    def state(self, new_value: bytes) -> None: ...
+    @property
+    def name(self) -> str: ...
+    @property
+    def ltoir(self) -> bytes: ...
+    @property
+    def state_alignment(self) -> int: ...
+    @property
+    def operator_type(self) -> OpKind: ...
+    @property
+    def code(self) -> Any: ...
+    @property
+    def extra_code(self) -> list: ...
+
+class TypeInfo:
+    def __init__(self, size: int, alignment: int, type_enum: TypeEnum): ...
+    @property
+    def size(self) -> int: ...
+    @property
+    def alignment(self) -> int: ...
+    @property
+    def typenum(self) -> int: ...
+    def as_bytes(self) -> bytes: ...
+
+class Value:
+    def __init__(self, type: TypeInfo, state: memoryview): ...
+    @property
+    def type(self) -> TypeInfo: ...
+    @property
+    def state(self) -> memoryview: ...
+    @state.setter
+    def state(self, new_value: memoryview) -> None: ...
+    def as_bytes(self) -> bytes: ...
+
+class Pointer:
+    def __init__(self, arg): ...
+
+def make_pointer_object(ptr: int | ctypes.c_void_p, owner: Any) -> Pointer: ...
+
+class IteratorState(Buffer):
+    def __init__(self, arg): ...
+    @property
+    def size(self) -> int: ...
+
+class Iterator:
+    def __init__(
+        self,
+        alignment: int,
+        iterator_type: IteratorKind,
+        advance_fn: Op,
+        dereference_fn: Op,
+        value_type: TypeInfo,
+        state=None,
+        host_advance_fn=None,
+    ):
+        pass
+
+    @property
+    def advance_op(self) -> Op: ...
+    @property
+    def dereference_op(self) -> Op: ...
+    @property
+    def dereference_or_assign_op(self) -> Op: ...
+    @property
+    def state(self): ...
+    @state.setter
+    def state(self, value) -> None: ...
+    @property
+    def type(self) -> IteratorKind: ...
+    @property
+    def alignment(self) -> int: ...
+    @property
+    def value_type(self) -> TypeInfo: ...
+    def as_bytes(self) -> bytes: ...
+    def is_kind_pointer(self) -> bool: ...
+    def is_kind_iterator(self) -> bool: ...
+    @property
+    def host_advance_fn(self): ...
+    @host_advance_fn.setter
+    def host_advance_fn(self, value) -> None: ...
+
+class CommonData:
+    def __init__(
+        self,
+        cc_major: int,
+        cc_minor: int,
+        cub_path: str,
+        thrust_path: str,
+        libcudacxx_path: str,
+        ctk_path: str,
+    ): ...
+    @property
+    def compute_capability(self) -> tuple[int, int]: ...
+    @property
+    def cub_path(self) -> str: ...
+    @property
+    def thrust_path(self) -> str: ...
+    @property
+    def libcudacxx_path(self) -> str: ...
+    @property
+    def ctk_path(self) -> str: ...
+
+# ------------
+# DeviceReduce
+# ------------
+
+class DeviceReduceBuildResult:
+    def __init__(
+        self,
+        d_in: Iterator,
+        d_out: Iterator,
+        binary_op: Op,
+        h_init: Value,
+        determinism: Determinism,
+        info: CommonData,
+    ): ...
+    def compute(
+        self,
+        temp_storage_ptr: int | None,
+        temp_storage_nbytes: int,
+        d_in: Iterator,
+        d_out: Iterator,
+        num_items: int,
+        binary_op: Op,
+        h_init: Value,
+        stream,
+    ) -> int: ...
+    def compute_nondeterministic(
+        self,
+        temp_storage_ptr: int | None,
+        temp_storage_nbytes: int,
+        d_in: Iterator,
+        d_out: Iterator,
+        num_items: int,
+        binary_op: Op,
+        h_init: Value,
+        stream,
+    ) -> int: ...
+    def serialize(self) -> bytes: ...
+    @staticmethod
+    def deserialize(
+        blob: bytes, load: bool = ..., check_cc: bool = ...
+    ) -> DeviceReduceBuildResult: ...
+    @staticmethod
+    def compile(*args) -> DeviceReduceBuildResult: ...
+    def load(self) -> None: ...
+    @property
+    def determinism(self) -> int: ...
+
+# ----------
+# DeviceScan
+# ----------
+
+class DeviceScanBuildResult:
+    def __init__(
+        self,
+        d_in: Iterator,
+        d_out: Iterator,
+        binary_op: Op,
+        init_type: TypeInfo,
+        force_inclusive: bool,
+        init_kind: InitKind,
+        info: CommonData,
+    ): ...
+    def compute_inclusive(
+        self,
+        temp_storage_ptr: int | None,
+        temp_storage_nbytes: int,
+        d_in: Iterator,
+        d_out: Iterator,
+        num_items: int,
+        binary_op: Op,
+        h_init: Value,
+        stream,
+    ) -> int: ...
+    def compute_exclusive(
+        self,
+        temp_storage_ptr: int | None,
+        temp_storage_nbytes: int,
+        d_in: Iterator,
+        d_out: Iterator,
+        num_items: int,
+        binary_op: Op,
+        h_init: Value,
+        stream,
+    ) -> int: ...
+    def compute_inclusive_future_value(
+        self,
+        temp_storage_ptr: int | None,
+        temp_storage_nbytes: int,
+        d_in: Iterator,
+        d_out: Iterator,
+        num_items: int,
+        binary_op: Op,
+        h_init: Iterator,
+        stream,
+    ) -> int: ...
+    def compute_exclusive_future_value(
+        self,
+        temp_storage_ptr: int | None,
+        temp_storage_nbytes: int,
+        d_in: Iterator,
+        d_out: Iterator,
+        num_items: int,
+        binary_op: Op,
+        h_init: Iterator,
+        stream,
+    ) -> int: ...
+    def compute_inclusive_no_init(
+        self,
+        temp_storage_ptr: int | None,
+        temp_storage_nbytes: int,
+        d_in: Iterator,
+        d_out: Iterator,
+        num_items: int,
+        binary_op: Op,
+        h_init: None,
+        stream,
+    ) -> int: ...
+    def serialize(self) -> bytes: ...
+    @staticmethod
+    def deserialize(
+        blob: bytes, load: bool = ..., check_cc: bool = ...
+    ) -> DeviceScanBuildResult: ...
+    @staticmethod
+    def compile(*args) -> DeviceScanBuildResult: ...
+    def load(self) -> None: ...
+
+# ---------------------
+# DeviceSegmentedReduce
+# ---------------------
+
+class DeviceSegmentedReduceBuildResult:
+    def __init__(
+        self,
+        d_in: Iterator,
+        d_out: Iterator,
+        start_offsets: Iterator,
+        end_offsets: Iterator,
+        binary_op: Op,
+        h_init: Value,
+        info: CommonData,
+    ): ...
+    def compute(
+        self,
+        temp_storage_ptr: int | None,
+        temp_storage_nbytes: int,
+        d_in: Iterator,
+        d_out: Iterator,
+        num_items: int,
+        start_offsets: Iterator,
+        end_offsets: Iterator,
+        binary_op: Op,
+        h_init: Value,
+        max_segment_size: int | None = None,
+        stream=None,
+    ) -> int: ...
+    def serialize(self) -> bytes: ...
+    @staticmethod
+    def deserialize(
+        blob: bytes, load: bool = ..., check_cc: bool = ...
+    ) -> DeviceSegmentedReduceBuildResult: ...
+    @staticmethod
+    def compile(*args) -> DeviceSegmentedReduceBuildResult: ...
+    def load(self) -> None: ...
+
+# ---------------
+# DeviceMergeSort
+# ---------------
+
+class DeviceMergeSortBuildResult:
+    def __init__(
+        self,
+        d_in_keys: Iterator,
+        d_in_items: Iterator,
+        d_out_keys: Iterator,
+        d_out_items: Iterator,
+        binary_op: Op,
+        info: CommonData,
+    ): ...
+    def compute(
+        self,
+        temp_storage_ptr: int | None,
+        temp_storage_nbytes: int,
+        d_in_keys: Iterator,
+        d_in_items: Iterator,
+        d_out_keys: Iterator,
+        d_out_items: Iterator,
+        num_items: int,
+        binary_op: Op,
+        stream,
+    ) -> int: ...
+    def serialize(self) -> bytes: ...
+    @staticmethod
+    def deserialize(
+        blob: bytes, load: bool = ..., check_cc: bool = ...
+    ) -> DeviceMergeSortBuildResult: ...
+    @staticmethod
+    def compile(*args) -> DeviceMergeSortBuildResult: ...
+    def load(self) -> None: ...
+
+# -----------------
+# DeviceUniqueByKey
+# -----------------
+
+class DeviceUniqueByKeyBuildResult:
+    def __init__(
+        self,
+        d_keys_in: Iterator,
+        d_values_in: Iterator,
+        d_keys_out: Iterator,
+        d_values_out: Iterator,
+        d_num_selected_out: Iterator,
+        binary_op: Op,
+        info: CommonData,
+    ): ...
+    def compute(
+        self,
+        temp_storage_ptr: int | None,
+        temp_storage_nbytes: int,
+        d_keys_in: Iterator,
+        d_values_in: Iterator,
+        d_keys_out: Iterator,
+        d_values_out: Iterator,
+        d_num_selected_out: Iterator,
+        binary_op: Op,
+        num_items: int,
+        stream,
+    ) -> int: ...
+    def serialize(self) -> bytes: ...
+    @staticmethod
+    def deserialize(
+        blob: bytes, load: bool = ..., check_cc: bool = ...
+    ) -> DeviceUniqueByKeyBuildResult: ...
+    @staticmethod
+    def compile(*args) -> DeviceUniqueByKeyBuildResult: ...
+    def load(self) -> None: ...
+
+# -----------------
+# DeviceRadixSort
+# -----------------
+
+class DeviceRadixSortBuildResult:
+    def __init__(self): ...
+    def compute(
+        self,
+        temp_storage_ptr: int | None,
+        temp_storage_nbytes: int,
+        d_keys_in: Iterator,
+        d_keys_out: Iterator,
+        d_values_in: Iterator,
+        d_values_out: Iterator,
+        decomposer_op: Op,
+        num_items: int,
+        begin_bit: int,
+        end_bit: int,
+        is_overwrite_okay: bool,
+        selector: int,
+        stream,
+    ) -> tuple[int, int]: ...
+    def serialize(self) -> bytes: ...
+    @staticmethod
+    def deserialize(
+        blob: bytes, load: bool = ..., check_cc: bool = ...
+    ) -> DeviceRadixSortBuildResult: ...
+    @staticmethod
+    def compile(*args) -> DeviceRadixSortBuildResult: ...
+    def load(self) -> None: ...
+
+# --------------------
+# DeviceUnaryTransform
+# --------------------
+
+class DeviceUnaryTransform:
+    def __init__(
+        self,
+        d_in: Iterator,
+        d_out: Iterator,
+        op: Op,
+        info: CommonData,
+    ): ...
+    def compute(
+        self,
+        d_in: Iterator,
+        d_out: Iterator,
+        num_items: int,
+        stream,
+    ) -> None: ...
+    def serialize(self) -> bytes: ...
+    @staticmethod
+    def deserialize(
+        blob: bytes, load: bool = ..., check_cc: bool = ...
+    ) -> DeviceUnaryTransform: ...
+    @staticmethod
+    def compile(*args) -> DeviceUnaryTransform: ...
+    def load(self) -> None: ...
+
+# ---------------------
+# DeviceBinaryTransform
+# ---------------------
+
+class DeviceBinaryTransform:
+    def __init__(
+        self,
+        d_in1: Iterator,
+        d_in2: Iterator,
+        d_out: Iterator,
+        op: Op,
+        info: CommonData,
+    ): ...
+    def compute(
+        self,
+        d_in1: Iterator,
+        d_in2: Iterator,
+        d_out: Iterator,
+        num_items: int,
+        stream,
+    ) -> None: ...
+    def serialize(self) -> bytes: ...
+    @staticmethod
+    def deserialize(
+        blob: bytes, load: bool = ..., check_cc: bool = ...
+    ) -> DeviceBinaryTransform: ...
+    @staticmethod
+    def compile(*args) -> DeviceBinaryTransform: ...
+    def load(self) -> None: ...
+
+# ---------------
+# DeviceHistogram
+# ---------------
+
+class DeviceHistogramBuildResult:
+    def __init__(
+        self,
+        num_channels: int,
+        num_active_channels: int,
+        d_samples: Iterator,
+        num_levels: int,
+        d_histogram: Iterator,
+        level_type: TypeInfo,
+        num_rows: int,
+        row_stride_samples: int,
+        is_evenly_segmented: bool,
+        info: CommonData,
+    ): ...
+    def compute_even(
+        self,
+        d_samples: Iterator,
+        d_histogram: Iterator,
+        h_num_output_levels: Value,
+        h_lower_level: Value,
+        h_upper_level: Value,
+        num_row_pixels: int,
+        num_rows: int,
+        row_stride_samples: int,
+        stream,
+    ) -> None: ...
+    def serialize(self) -> bytes: ...
+    @staticmethod
+    def deserialize(
+        blob: bytes, load: bool = ..., check_cc: bool = ...
+    ) -> DeviceHistogramBuildResult: ...
+    @staticmethod
+    def compile(*args) -> DeviceHistogramBuildResult: ...
+    def load(self) -> None: ...
+
+# -------------------
+# DeviceBinarySearch
+# -------------------
+
+class DeviceBinarySearchBuildResult:
+    def __init__(
+        self,
+        mode: BinarySearchMode,
+        d_data: Iterator,
+        d_values: Iterator,
+        d_out: Iterator,
+        comparison_op: Op,
+        info: CommonData,
+    ): ...
+    def compute(
+        self,
+        d_data: Iterator,
+        num_items: int,
+        d_values: Iterator,
+        num_values: int,
+        d_out: Iterator,
+        comparison_op: Op,
+        stream,
+    ) -> None: ...
+    def serialize(self) -> bytes: ...
+    @staticmethod
+    def deserialize(
+        blob: bytes, load: bool = ..., check_cc: bool = ...
+    ) -> DeviceBinarySearchBuildResult: ...
+    @staticmethod
+    def compile(*args) -> DeviceBinarySearchBuildResult: ...
+    def load(self) -> None: ...
+
+# -----------------
+# DeviceSegmentedSort
+# -----------------
+
+class DeviceSegmentedSortBuildResult:
+    def __init__(self): ...
+    def compute(
+        self,
+        temp_storage_ptr: int | None,
+        temp_storage_nbytes: int,
+        d_in_keys: Iterator,
+        d_out_keys: Iterator,
+        d_in_values: Iterator,
+        d_out_values: Iterator,
+        num_items: int,
+        num_segments: int,
+        d_begin_offsets: Iterator,
+        d_end_offsets: Iterator,
+        is_overwrite_okay: bool,
+        selector: int,
+        stream,
+    ) -> tuple[int, int]: ...
+    def serialize(self) -> bytes: ...
+    @staticmethod
+    def deserialize(
+        blob: bytes, load: bool = ..., check_cc: bool = ...
+    ) -> DeviceSegmentedSortBuildResult: ...
+    @staticmethod
+    def compile(*args) -> DeviceSegmentedSortBuildResult: ...
+    def load(self) -> None: ...
+
+# ---------------------
+# DeviceThreeWayPartition
+# ---------------------
+
+class DeviceThreeWayPartitionBuildResult:
+    def __init__(
+        self,
+        d_in: Iterator,
+        d_first_part_out: Iterator,
+        d_second_part_out: Iterator,
+        d_unselected_out: Iterator,
+        d_num_selected_out: Iterator,
+        select_first_part_op: Op,
+        select_second_part_op: Op,
+        info: CommonData,
+    ): ...
+    def compute(
+        self,
+        d_in: Iterator,
+        d_first_part_out: Iterator,
+        d_second_part_out: Iterator,
+        d_unselected_out: Iterator,
+        d_num_selected_out: Iterator,
+        num_items: int,
+        stream,
+    ) -> int: ...
+    def serialize(self) -> bytes: ...
+    @staticmethod
+    def deserialize(
+        blob: bytes, load: bool = ..., check_cc: bool = ...
+    ) -> DeviceThreeWayPartitionBuildResult: ...
+    @staticmethod
+    def compile(*args) -> DeviceThreeWayPartitionBuildResult: ...
+    def load(self) -> None: ...
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_binary_search_backend_v1.pxi b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_binary_search_backend_v1.pxi
new file mode 100644
index 00000000..05069c6a
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_binary_search_backend_v1.pxi
@@ -0,0 +1,17 @@
+# v1 (cccl.c.parallel, NVRTC) — binary_search build_result_t struct +
+# uniform cubin-bytes helper. v1 nests a transform build_result and carries
+# op-state metadata; v2 (sibling file) flattens to top-level cubin fields.
+
+cdef extern from "cccl/c/binary_search.h":
+    cdef struct cccl_device_binary_search_build_result_t 'cccl_device_binary_search_build_result_t':
+        cccl_device_transform_build_result_t transform
+        size_t op_state_size
+        size_t op_state_alignment
+
+
+cdef inline bytes _binary_search_cubin_bytes(
+    cccl_device_binary_search_build_result_t* b,
+):
+    return PyBytes_FromStringAndSize(
+        <const char*>b.transform.payload, b.transform.payload_size
+    )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_binary_search_backend_v2.pxi b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_binary_search_backend_v2.pxi
new file mode 100644
index 00000000..4adac178
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_binary_search_backend_v2.pxi
@@ -0,0 +1,15 @@
+# v2 (cccl.c.parallel.v2, HostJIT) — binary_search build_result_t struct +
+# uniform cubin-bytes helper. v2 uses payload/payload_size matching v1.
+
+cdef extern from "cccl/c/binary_search.h":
+    cdef struct cccl_device_binary_search_build_result_t 'cccl_device_binary_search_build_result_t':
+        void* payload
+        size_t payload_size
+
+
+cdef inline bytes _binary_search_cubin_bytes(
+    cccl_device_binary_search_build_result_t* b,
+):
+    return PyBytes_FromStringAndSize(
+        <const char*>b.payload, b.payload_size
+    )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_impl.pyx b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_impl.pyx
new file mode 100644
index 00000000..6063ec75
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_impl.pyx
@@ -0,0 +1,2992 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# distutils: language = c++
+# cython: language_level=3
+# cython: linetrace=True
+# cython: freethreading_compatible=True
+
+# Python signatures are declared in the companion Python stub file _bindings.pyi
+# Make sure to update PYI with change to Python API to ensure that Python
+# static type checker tools like mypy green-lights cuda.compute
+
+from libc.string cimport memset, memcpy
+from libc.stdint cimport uint8_t, uint32_t, uint64_t, int64_t, uintptr_t
+from libc.stdlib cimport malloc, free
+from cpython.bytes cimport PyBytes_FromStringAndSize
+
+from cpython.buffer cimport (
+    Py_buffer, PyBUF_SIMPLE, PyBUF_ANY_CONTIGUOUS,
+    PyBuffer_Release, PyObject_CheckBuffer, PyObject_GetBuffer
+)
+from cpython.pycapsule cimport (
+    PyCapsule_CheckExact, PyCapsule_IsValid, PyCapsule_GetPointer
+)
+
+import ctypes
+import threading
+from enum import IntEnum
+
+# Serializes the one-time transition of a build result to "loaded". The backend
+# loader runs under `with nogil` (releasing the GIL), so without this lock two
+# concurrent first-time __call__s on the same build result could both pass the
+# `_loaded` guard and load the same handle twice.
+_LOAD_LOCK = threading.Lock()
+
+
+def _load_once(build_result, loader):
+    with _LOAD_LOCK:
+        if build_result._loaded:
+            return
+        loader(build_result)
+        build_result._loaded = True
+
+
+cdef extern from "<cuda.h>":
+    cdef struct OpaqueCUstream_st
+    cdef struct OpaqueCUkernel_st
+    cdef struct OpaqueCUlibrary_st
+
+    ctypedef int CUresult
+    ctypedef OpaqueCUstream_st *CUstream
+    ctypedef OpaqueCUkernel_st *CUkernel
+    ctypedef OpaqueCUlibrary_st *CUlibrary
+
+
+# Backend-conditional cccl_op_code_type enum + _parse_code_type helper.
+# CMake configures the v1 or v2 variant of `_bindings_op_code_type.pxi` into
+# the build dir. v2 declares the extra CCCL_OP_LLVM_IR value that v1's
+# types.h does not define.
+include "_bindings_op_code_type.pxi"
+
+
+cdef extern from "cccl/c/types.h":
+    cpdef enum cccl_type_enum:
+        INT8 "CCCL_INT8"
+        INT16 "CCCL_INT16"
+        INT32 "CCCL_INT32"
+        INT64 "CCCL_INT64"
+        UINT8 "CCCL_UINT8"
+        UINT16 "CCCL_UINT16"
+        UINT32 "CCCL_UINT32"
+        UINT64 "CCCL_UINT64"
+        FLOAT16 "CCCL_FLOAT16"
+        FLOAT32 "CCCL_FLOAT32"
+        FLOAT64 "CCCL_FLOAT64"
+        STORAGE "CCCL_STORAGE"
+        BOOLEAN "CCCL_BOOLEAN"
+
+    cpdef enum cccl_op_kind_t:
+       STATELESS "CCCL_STATELESS"
+       STATEFUL "CCCL_STATEFUL"
+       PLUS "CCCL_PLUS"
+       MINUS "CCCL_MINUS"
+       MULTIPLIES "CCCL_MULTIPLIES"
+       DIVIDES "CCCL_DIVIDES"
+       MODULUS "CCCL_MODULUS"
+       EQUAL_TO "CCCL_EQUAL_TO"
+       NOT_EQUAL_TO "CCCL_NOT_EQUAL_TO"
+       GREATER "CCCL_GREATER"
+       LESS "CCCL_LESS"
+       GREATER_EQUAL "CCCL_GREATER_EQUAL"
+       LESS_EQUAL "CCCL_LESS_EQUAL"
+       LOGICAL_AND "CCCL_LOGICAL_AND"
+       LOGICAL_OR "CCCL_LOGICAL_OR"
+       LOGICAL_NOT "CCCL_LOGICAL_NOT"
+       BIT_AND "CCCL_BIT_AND"
+       BIT_OR "CCCL_BIT_OR"
+       BIT_XOR "CCCL_BIT_XOR"
+       BIT_NOT "CCCL_BIT_NOT"
+       IDENTITY "CCCL_IDENTITY"
+       NEGATE "CCCL_NEGATE"
+       MINIMUM "CCCL_MINIMUM"
+       MAXIMUM "CCCL_MAXIMUM"
+
+    cpdef enum cccl_iterator_kind_t:
+       POINTER "CCCL_POINTER"
+       ITERATOR "CCCL_ITERATOR"
+
+    cdef struct cccl_type_info:
+        size_t size
+        size_t alignment
+        cccl_type_enum type
+
+    cdef struct cccl_op_t:
+        cccl_op_kind_t type
+        const char* name
+        const char* code
+        size_t code_size
+        cccl_op_code_type code_type
+        size_t size
+        size_t alignment
+        void *state
+        const char** extra_ltoirs
+        size_t* extra_ltoir_sizes
+        size_t num_extra_ltoirs
+        cccl_op_code_type* extra_code_types
+
+    cdef struct cccl_value_t:
+        cccl_type_info type
+        void *state
+
+    cdef union cccl_increment_t:
+        int64_t signed_offset
+        uint64_t unsigned_offset
+
+    ctypedef void (*cccl_host_op_fn_ptr_t)(void *, cccl_increment_t) nogil
+
+    cdef struct cccl_iterator_t:
+        size_t size
+        size_t alignment
+        cccl_iterator_kind_t type
+        cccl_op_t advance
+        cccl_op_t dereference
+        cccl_type_info value_type
+        void *state
+        cccl_host_op_fn_ptr_t host_advance
+
+    cpdef enum cccl_sort_order_t:
+        ASCENDING "CCCL_ASCENDING"
+        DESCENDING "CCCL_DESCENDING"
+
+    cpdef enum cccl_init_kind_t:
+        VALUE_INIT "CCCL_VALUE_INIT"
+        FUTURE_VALUE_INIT "CCCL_FUTURE_VALUE_INIT"
+        NO_INIT "CCCL_NO_INIT"
+
+    cpdef enum cccl_determinism_t:
+        NOT_GUARANTEED "CCCL_NOT_GUARANTEED"
+        RUN_TO_RUN "CCCL_RUN_TO_RUN"
+        GPU_TO_GPU "CCCL_GPU_TO_GPU"
+
+    cpdef enum cccl_binary_search_mode_t:
+        LOWER_BOUND "CCCL_BINARY_SEARCH_LOWER_BOUND"
+        UPPER_BOUND "CCCL_BINARY_SEARCH_UPPER_BOUND"
+
+cdef void arg_type_check(
+    str arg_name,
+    object expected_type,
+    object arg
+) except *:
+    if not isinstance(arg, expected_type):
+        raise TypeError(
+            f"Expected {arg_name} to have type '{expected_type}', "
+            f"got '{type(arg)}'"
+        )
+
+OpKind = cccl_op_kind_t
+TypeEnum = cccl_type_enum
+IteratorKind = cccl_iterator_kind_t
+SortOrder = cccl_sort_order_t
+InitKind = cccl_init_kind_t
+Determinism = cccl_determinism_t
+BinarySearchMode = cccl_binary_search_mode_t
+
+cdef void _validate_alignment(int alignment) except *:
+    """
+    Alignment must be positive integer and a power of two
+    that can be represented by uint32_t type.
+    """
+    cdef uint32_t val
+    if alignment < 1:
+        raise ValueError(
+            "Alignment must be non-negative, "
+            f"got {alignment}."
+        )
+    val = <uint32_t>alignment
+    if (val & (val - 1)) != 0:
+        raise ValueError(
+            "Alignment must be a power of two, "
+            f"got {alignment}"
+        )
+
+
+cdef class Op:
+    """
+    Represents CCCL Operation
+
+    Args:
+        name (str):
+            Name of the operation
+        operator_type (OpKind):
+            Whether operator is stateless or stateful
+        ltoir (bytes):
+            The LTOIR for the operation compiled for device
+        state (bytes, optional):
+            State for the stateful operation.
+        state_alignment (int, optional):
+            Alignment of the state struct. Default: `1`.
+        extra_ltoirs (list of bytes, optional):
+            Additional LTOIR modules to link with this operation.
+    """
+    # need Python owner of memory used for operator name
+    cdef bytes op_encoded_name
+    cdef bytes code_bytes
+    cdef bytes state_bytes
+    cdef list extra_ltoirs_list  # Python list to keep bytes alive
+    cdef list extra_code_types_list  # Python list of code_type strings; kept alive in parallel with extra_ltoirs_list
+    cdef const char** extra_ltoirs_ptrs
+    cdef size_t* extra_ltoir_sizes_arr
+    cdef cccl_op_code_type* extra_code_types_arr
+    cdef cccl_op_t op_data
+
+
+    cdef void _set_members(self, cccl_op_kind_t op_type, str name, bytes code_bytes, str code_kind, bytes state, int state_alignment, list extra_bytes, list extra_kinds):
+        memset(&self.op_data, 0, sizeof(cccl_op_t))
+        # Reference Python objects in the class to ensure lifetime.
+        self.op_encoded_name = name.encode("utf-8")
+        self.code_bytes = code_bytes
+        self.state_bytes = state
+        self.extra_ltoirs_list = extra_bytes
+        self.extra_code_types_list = extra_kinds
+        # set fields of op_data struct
+        self.op_data.type = op_type
+        self.op_data.name = <const char *>self.op_encoded_name
+        self.op_data.code = <const char *>code_bytes
+        self.op_data.code_size = len(code_bytes)
+        self.op_data.code_type = _parse_code_type(code_kind)
+        self.op_data.size = len(state)
+        self.op_data.alignment = state_alignment
+        self.op_data.state = <void *><const char *>state
+
+        # Handle extras (parallel arrays on the C side).
+        cdef size_t num_extra = len(extra_bytes)
+        if num_extra > 0:
+            self.extra_ltoirs_ptrs = <const char**>malloc(num_extra * sizeof(const char*))
+            self.extra_ltoir_sizes_arr = <size_t*>malloc(num_extra * sizeof(size_t))
+            self.extra_code_types_arr = <cccl_op_code_type*>malloc(num_extra * sizeof(cccl_op_code_type))
+            for i in range(num_extra):
+                eb = <bytes>extra_bytes[i]
+                self.extra_ltoirs_ptrs[i] = <const char*>eb
+                self.extra_ltoir_sizes_arr[i] = len(eb)
+                self.extra_code_types_arr[i] = _parse_code_type(extra_kinds[i])
+            self.op_data.extra_ltoirs = self.extra_ltoirs_ptrs
+            self.op_data.extra_ltoir_sizes = self.extra_ltoir_sizes_arr
+            self.op_data.extra_code_types = self.extra_code_types_arr
+            self.op_data.num_extra_ltoirs = num_extra
+        else:
+            self.extra_ltoirs_ptrs = NULL
+            self.extra_ltoir_sizes_arr = NULL
+            self.extra_code_types_arr = NULL
+            self.op_data.extra_ltoirs = NULL
+            self.op_data.extra_ltoir_sizes = NULL
+            self.op_data.extra_code_types = NULL
+            self.op_data.num_extra_ltoirs = 0
+
+
+    def __cinit__(self, /, *, name = None, operator_type = None, ltoir = None, state = None, state_alignment = 1, extra_ltoirs = None):
+        # ltoir accepts either raw `bytes` (legacy — treated as LTO-IR) or a
+        # DeviceCode instance (carries its own format tag via .kind / .op_bytes).
+        # extra_ltoirs accepts a list of either form, mixable.
+        if name is None and ltoir is None:
+            name = ""
+            ltoir = b""
+        if state is None:
+            state = b""
+        if operator_type is None:
+            operator_type = OpKind.STATELESS
+        if extra_ltoirs is None:
+            extra_ltoirs = []
+        arg_type_check(arg_name="name", expected_type=str, arg=name)
+        arg_type_check(arg_name="state", expected_type=bytes, arg=state)
+        arg_type_check(arg_name="state_alignment", expected_type=int, arg=state_alignment)
+        arg_type_check(arg_name="extra_ltoirs", expected_type=list, arg=extra_ltoirs)
+        if not isinstance(operator_type, OpKind):
+            raise TypeError(
+                f"The operator_type argument should be an enumerator of operator kinds"
+            )
+
+        # Unpack ltoir → (bytes, kind). DeviceCode duck-typed via attributes
+        # to avoid importing it here.
+        cdef bytes code_bytes
+        cdef str code_kind
+        if isinstance(ltoir, bytes):
+            code_bytes = <bytes>ltoir
+            code_kind = "ltoir"
+        elif hasattr(ltoir, "op_bytes") and hasattr(ltoir, "kind"):
+            if not isinstance(ltoir.op_bytes, bytes) or not isinstance(ltoir.kind, str):
+                raise TypeError(
+                    f"DeviceCode.op_bytes must be bytes and .kind must be str; "
+                    f"got {type(ltoir.op_bytes)} and {type(ltoir.kind)}"
+                )
+            code_bytes = <bytes>ltoir.op_bytes
+            code_kind = <str>ltoir.kind
+        else:
+            raise TypeError(
+                f"ltoir must be bytes or DeviceCode-like (with op_bytes and kind "
+                f"attributes); got {type(ltoir)}"
+            )
+
+        # Unpack each extra entry the same way.
+        extra_bytes = []
+        extra_kinds = []
+        for i, el in enumerate(extra_ltoirs):
+            if isinstance(el, bytes):
+                extra_bytes.append(el)
+                extra_kinds.append("ltoir")
+            elif hasattr(el, "op_bytes") and hasattr(el, "kind"):
+                if not isinstance(el.op_bytes, bytes) or not isinstance(el.kind, str):
+                    raise TypeError(
+                        f"extra_ltoirs[{i}].op_bytes must be bytes and .kind must be str; "
+                        f"got {type(el.op_bytes)} and {type(el.kind)}"
+                    )
+                extra_bytes.append(el.op_bytes)
+                extra_kinds.append(el.kind)
+            else:
+                raise TypeError(
+                    f"extra_ltoirs[{i}] must be bytes or DeviceCode-like; got {type(el)}"
+                )
+
+        _validate_alignment(state_alignment)
+        self._set_members(
+            <cccl_op_kind_t> operator_type.value,
+            <str> name,
+            code_bytes,
+            code_kind,
+            <bytes> state,
+            <int> state_alignment,
+            <list> extra_bytes,
+            <list> extra_kinds,
+        )
+
+    def __dealloc__(self):
+        if self.extra_ltoirs_ptrs != NULL:
+            free(self.extra_ltoirs_ptrs)
+            self.extra_ltoirs_ptrs = NULL
+        if self.extra_ltoir_sizes_arr != NULL:
+            free(self.extra_ltoir_sizes_arr)
+            self.extra_ltoir_sizes_arr = NULL
+        if self.extra_code_types_arr != NULL:
+            free(self.extra_code_types_arr)
+            self.extra_code_types_arr = NULL
+
+    cdef void set_state(self, bytes state):
+        self.state_bytes = state
+        self.op_data.state = <void *><const char *>state
+
+    @property
+    def state(self):
+       return self.state_bytes
+
+    @state.setter
+    def state(self, bytes new_value):
+        self.set_state(<bytes>new_value)
+
+    @property
+    def name(self):
+        return self.op_encoded_name.decode("utf-8")
+
+    @property
+    def ltoir(self):
+        # Backward compatibility property
+        return self.code_bytes
+
+    @property
+    def state_alignment(self):
+        return self.op_data.alignment
+
+    @property
+    def operator_type(self):
+        """Return the op kind (stateless/stateful) for serialization."""
+        return OpKind(<int>self.op_data.type)
+
+    @property
+    def code(self):
+        """Return a DeviceCode wrapping this op's main code blob + its kind."""
+        from cuda.compute._device_code import DeviceCode
+        return DeviceCode(op_bytes=self.code_bytes, kind=self._code_kind())
+
+    @property
+    def extra_code(self):
+        """Return DeviceCode instances for each extra blob."""
+        from cuda.compute._device_code import DeviceCode
+        return [
+            DeviceCode(op_bytes=b, kind=k)
+            for b, k in zip(self.extra_ltoirs_list, self.extra_code_types_list)
+        ]
+
+    cdef str _code_kind(self):
+        cdef cccl_op_code_type t = self.op_data.code_type
+        if t == CCCL_OP_LTOIR:
+            return "ltoir"
+        if t == CCCL_OP_CPP_SOURCE:
+            return "cpp_source"
+        # CCCL_OP_LLVM_IR is only defined on the v2 build; _parse_code_type
+        # maps "llvm_ir" to CCCL_OP_LTOIR on v1, so we must test CCCL_OP_LTOIR
+        # first (above) before reaching here.
+        if t == _parse_code_type("llvm_ir"):
+            return "llvm_ir"
+        return "ltoir"
+
+
+cdef class TypeInfo:
+    """
+    Represents CCCL type info structure
+
+    Args:
+        size (int):
+            Size of the type in bytes.
+        alignment (int):
+            Alignment of the type in bytes.
+        type_enum (TypeEnum):
+            Enumeration member identifying the type.
+    """
+    cdef cccl_type_info type_info
+
+    def __cinit__(self, int size, int alignment, cccl_type_enum type_enum):
+        if size < 1:
+            raise ValueError("Size argument must be positive")
+        _validate_alignment(alignment)
+        self.type_info.size = size
+        self.type_info.alignment = alignment
+        self.type_info.type = type_enum
+
+    @property
+    def size(self):
+        return self.type_info.size
+
+    @property
+    def alignment(self):
+        return self.type_info.alignment
+
+    @property
+    def typenum(self):
+        return self.type_info.type
+
+    def as_bytes(self):
+        "Debugging utility to view memory content of library struct"
+        cdef uint8_t[:] mem_view = bytearray(sizeof(self.type_info))
+        memcpy(&mem_view[0], &self.type_info, sizeof(self.type_info))
+        return bytes(mem_view)
+
+
+cdef class Value:
+    """
+    Represents CCCL value structure
+
+    Args:
+        value_type (TypeInfo):
+            type descriptor
+        state (object):
+            state of the value type. Object is expected to
+            implement Python buffer protocol and be able to provide
+            simple contiguous array of type `uint8_t`.
+    """
+    cdef uint8_t[::1] state_obj
+    cdef TypeInfo value_type
+    cdef cccl_value_t value_data;
+
+    def __cinit__(self, TypeInfo value_type, uint8_t[::1] state):
+        self.state_obj = state
+        self.value_type = value_type
+        self.value_data.type = value_type.type_info
+        self.value_data.state = <void *>&state[0]
+
+    @property
+    def type(self):
+        return self.value_type
+
+    @property
+    def state(self):
+        return self.state_obj
+
+    @state.setter
+    def state(self, uint8_t[::1] new_value):
+        if (len(self.state_obj) == len(new_value)):
+            self.state_obj = new_value
+            self.value_data.state = <void *>&self.state_obj[0]
+        else:
+            raise ValueError("Size mismatch")
+
+    def as_bytes(self):
+        "Debugging utility to view memory of native struct"
+        cdef uint8_t[:] mem_view = bytearray(sizeof(self.value_data))
+        memcpy(&mem_view[0], &self.value_data, sizeof(self.value_data))
+        return bytes(mem_view)
+
+
+cdef void ensure_buffer(object o) except *:
+    if not PyObject_CheckBuffer(o):
+        raise TypeError(
+            "Object with buffer protocol expected, "
+            f"got {type(o)}"
+        )
+
+
+cdef void * get_buffer_pointer(object o, size_t *size):
+    cdef int status = 0
+    cdef void *ptr = NULL
+    cdef Py_buffer view
+
+    status = PyObject_GetBuffer(o, &view, PyBUF_SIMPLE | PyBUF_ANY_CONTIGUOUS)
+    if status != 0:  # pragma: no cover
+        size[0] = 0
+        raise RuntimeError(
+            "Can not access simple contiguous buffer"
+        )
+
+    ptr = view.buf
+    if size is not NULL:
+        size[0] = <size_t>view.len
+    PyBuffer_Release(&view)
+
+    return ptr
+
+
+cdef void * ctypes_typed_pointer_payload_ptr(object ctypes_typed_ptr):
+    "Get pointer to the value buffer represented by ctypes.pointer(ctypes_val)"
+    cdef size_t size = 0
+    cdef size_t *ptr_ref = NULL
+    ensure_buffer(ctypes_typed_ptr)
+    ptr_ref = <size_t *>get_buffer_pointer(ctypes_typed_ptr, &size)
+    return <void *>(ptr_ref[0])
+
+
+cdef void * ctypes_value_ptr(object ctypes_cdata):
+    "Get pointer to the value buffer behind ctypes_val"
+    cdef size_t size = 0
+    ensure_buffer(ctypes_cdata)
+    return get_buffer_pointer(ctypes_cdata, &size)
+
+
+cdef inline void * int_as_ptr(size_t ptr_val):
+    return <void *>(ptr_val)
+
+
+cdef class StateBase:
+    cdef void *ptr
+    cdef object ref
+
+    def __cinit__(self):
+        self.ptr = NULL
+        self.ref = None
+
+    cdef inline void set_state(self, void *ptr, object ref):
+        self.ptr = ptr
+        self.ref = ref
+
+    @property
+    def pointer(self):
+        return <size_t>self.ptr
+
+    @property
+    def reference(self):
+        return self.ref
+
+
+cdef class Pointer(StateBase):
+    "Represents the pointer value"
+
+    def __cinit__(self, arg):
+        cdef void *ptr
+        cdef object ref
+
+        if isinstance(arg, int):
+            ptr = int_as_ptr(arg)
+            ref = None
+        elif isinstance(arg, ctypes._Pointer):
+            ptr = ctypes_typed_pointer_payload_ptr(arg)
+            ref = arg
+        elif isinstance(arg, ctypes.c_void_p):
+            ptr = int_as_ptr(arg.value)
+            ref = arg
+        else:
+            raise TypeError(
+                "Expect ctypes pointer, integers, or PointerProxy, "
+                f"got type {type(arg)}"
+            )
+        self.set_state(ptr, ref)
+
+
+def make_pointer_object(ptr, owner):
+    cdef Pointer res = Pointer(0)
+
+    if isinstance(ptr, int):
+        res.ptr = int_as_ptr(ptr)
+    elif isinstance(ptr, ctypes.c_void_p):
+        res.ptr = int_as_ptr(ptr.value)
+    else:
+        raise TypeError(
+            "First argument must be an integer, or ctypes.c_void_p, "
+            f"got {type(ptr)}"
+        )
+    res.ref = owner
+    return res
+
+
+cdef class IteratorState(StateBase):
+    "Represents blob referenced by pointer"
+    cdef size_t state_nbytes
+
+    def __cinit__(self, arg):
+        cdef size_t buffer_size = 0
+        cdef void *ptr = NULL
+        cdef object ref = None
+
+        super().__init__()
+        if isinstance(arg, ctypes._Pointer):
+            ptr = ctypes_typed_pointer_payload_ptr(arg)
+            ref = arg.contents
+            self.state_nbytes = ctypes.sizeof(ref)
+        elif PyObject_CheckBuffer(arg):
+            ptr = get_buffer_pointer(arg, &buffer_size)
+            ref = arg
+            self.state_nbytes = buffer_size
+        else:
+            raise TypeError(
+                "Expected a ctypes pointer with content, or object of type bytes or bytearray, "
+                f"got type {type(arg)}"
+            )
+        self.set_state(ptr, ref)
+
+    cdef inline size_t get_size(self):
+        return self.state_nbytes
+
+    @property
+    def size(self):
+        return self.state_nbytes
+
+    def __getbuffer__(self, Py_buffer *buffer, int flags):
+        cdef Py_ssize_t cast_size = <Py_ssize_t>self.state_nbytes
+        buffer.buf = <char *>self.ptr
+        buffer.obj = self
+        buffer.len = cast_size
+        buffer.readonly = 0
+        buffer.itemsize = 1
+        buffer.format = "B"  # unsigned char
+        buffer.ndim = 1
+        buffer.shape = <Py_ssize_t *>&self.state_nbytes
+        buffer.strides = &buffer.itemsize
+        buffer.suboffsets = NULL
+        buffer.internal = NULL
+
+    def __releasebuffer__(self, Py_buffer *buffer):
+        pass
+
+
+cdef const char *function_ptr_capsule_name = "void (void *, cccl_increment_t)";
+
+cdef bint is_function_pointer_capsule(object o) noexcept:
+    """
+    Returns non-zero if input is a valid capsule with
+    name 'void (void *, cccl_increment_t)'.
+    """
+    return (
+        PyCapsule_CheckExact(o) and
+        PyCapsule_IsValid(o, function_ptr_capsule_name)
+    )
+
+
+cdef inline void* get_function_pointer_from_capsule(object cap) except *:
+    return PyCapsule_GetPointer(cap, function_ptr_capsule_name)
+
+
+cdef cccl_host_op_fn_ptr_t unbox_host_advance_fn(object host_fn_obj) except *:
+    cdef void *fn_ptr = NULL
+    if isinstance(host_fn_obj, ctypes._CFuncPtr):
+        # the _CFuncPtr object encapsulates a pointer to the function pointer
+        fn_ptr = ctypes_typed_pointer_payload_ptr(host_fn_obj)
+        return <cccl_host_op_fn_ptr_t>fn_ptr
+
+    if isinstance(host_fn_obj, int):
+        fn_ptr = <void *><uintptr_t>host_fn_obj
+        return <cccl_host_op_fn_ptr_t>fn_ptr
+
+    if isinstance(host_fn_obj, ctypes.c_void_p):
+        fn_ptr = <void *><uintptr_t>host_fn_obj.value
+        return <cccl_host_op_fn_ptr_t>fn_ptr
+
+    if is_function_pointer_capsule(host_fn_obj):
+        fn_ptr = get_function_pointer_from_capsule(host_fn_obj)
+        return <cccl_host_op_fn_ptr_t>fn_ptr
+
+    raise TypeError(
+        "Expected ctypes function pointer, ctypes.c_void_p, integer or a named capsule, "
+        f"got {type(host_fn_obj)}"
+    )
+
+
+cdef class Iterator:
+    """
+    Represents CCCL iterator.
+
+    Args:
+        alignment (int):
+            Alignment of the iterator state
+        iterator_type (IteratorKind):
+            The type of iterator, `IteratorKind.POINTER` or
+            `IteratorKind.ITERATOR`
+        advance_fn (Op):
+            Descriptor for user-defined `advance` function
+            compiled for device
+        dereference_fn (Op):
+            Descriptor for user-defined `dereference` or `assign`
+            function compiled for device
+        value_type (TypeInfo):
+            Descriptor of the type addressed by the iterator
+        state (object, optional):
+            Python object for the state of the iterator. For iterators of
+            type `ITERATOR` the state object is expected to implement Python
+            buffer protocol for SIMPLE 1d buffer of type unsigned byte.
+            For iterators of type `POINTER` the state may be an integer convertible
+            to `uintptr_t`, or a `ctypes` pointer (typed or untyped).
+            Value `None` represents absence of iterator state.
+        host_advance_fn (object, optional):
+            Python object for host callable function to advance state by a given
+            increment. The argument may only be set for iterators of type
+            `IteratorKind.ITERATOR` and raise an exception otherwise. Supported
+            types are `int` or `ctypes.c_void_p` (raw pointer), ctypes function
+            pointer, or a Python capsule with name `"void *(void *, cccl_increment_t)"`.
+    """
+    cdef Op advance
+    cdef Op dereference
+    cdef object state_obj
+    cdef object host_advance_obj
+    cdef cccl_iterator_t iter_data
+
+    def __cinit__(self,
+        int alignment,
+        cccl_iterator_kind_t iterator_type,
+        Op advance_fn,
+        Op dereference_fn,
+        TypeInfo value_type,
+        state=None,
+        host_advance_fn=None
+    ):
+        cdef cccl_iterator_kind_t it_kind
+        _validate_alignment(alignment)
+        it_kind = iterator_type
+        if it_kind == cccl_iterator_kind_t.POINTER:
+            if state is None:
+                self.state_obj = None
+                self.iter_data.size = 0
+                self.iter_data.state = NULL
+            elif isinstance(state, int):
+                self.state_obj = None
+                self.iter_data.size = 0
+                self.iter_data.state = int_as_ptr(state)
+            elif isinstance(state, Pointer):
+                self.state_obj = state.reference
+                self.iter_data.size = 0
+                self.iter_data.state = (<Pointer>state).ptr
+            else:
+                raise TypeError(
+                    "Expect for Iterator of kind POINTER, state must have type Pointer or int, "
+                    f"got {type(state)}"
+                )
+            if host_advance_fn is not None:
+                raise ValueError(
+                    "host_advance_fn must be set to None for iterators of kind POINTER"
+                )
+            self.iter_data.host_advance = NULL
+            self.host_advance_obj = None
+        elif it_kind == cccl_iterator_kind_t.ITERATOR:
+            if state is None:
+                self.state_obj = None
+                self.iter_data.size = 0
+                self.iter_data.state = NULL
+            elif isinstance(state, IteratorState):
+                self.state_obj = state.reference
+                self.iter_data.size = (<IteratorState>state).size
+                self.iter_data.state = (<IteratorState>state).ptr
+            else:
+                raise TypeError(
+                    "For Iterator of kind ITERATOR, state must have type IteratorState, "
+                    f"got type {type(state)}"
+                )
+            if host_advance_fn is not None:
+                self.iter_data.host_advance = unbox_host_advance_fn(host_advance_fn)
+                self.host_advance_obj = host_advance_fn
+            else:
+                self.iter_data.host_advance = NULL
+                self.host_advance_obj = None
+        else:  # pragma: no cover
+            raise ValueError("Unrecognized iterator kind")
+        self.advance = advance_fn
+        self.dereference = dereference_fn
+        self.iter_data.alignment = alignment
+        self.iter_data.type = <cccl_iterator_kind_t> it_kind
+        self.iter_data.advance = self.advance.op_data
+        self.iter_data.dereference = self.dereference.op_data
+        self.iter_data.value_type = value_type.type_info
+
+    @property
+    def advance_op(self):
+        return self.advance
+
+    @property
+    def dereference_or_assign_op(self):
+        return self.dereference
+
+    @property
+    def state(self):
+        if self.iter_data.type == cccl_iterator_kind_t.POINTER:
+            return <size_t>self.iter_data.state
+        else:
+            return self.state_obj
+
+    @state.setter
+    def state(self, new_value):
+        cdef ssize_t state_sz = 0
+        cdef size_t ptr = 0
+        cdef cccl_iterator_kind_t it_kind = self.iter_data.type
+        if it_kind == cccl_iterator_kind_t.POINTER:
+            if isinstance(new_value, Pointer):
+                self.state_obj = (<Pointer>new_value).ref
+                self.iter_data.size = state_sz
+                self.iter_data.state = (<Pointer>new_value).ptr
+            elif isinstance(new_value, int):
+                self.state_obj = None
+                self.iter_data.size = state_sz
+                self.iter_data.state = int_as_ptr(new_value)
+            elif new_value is None:
+                self.state_obj = None
+                self.iter_data.size = 0
+                self.iter_data.state = NULL
+            else:
+                raise TypeError(
+                    "For iterator with type POINTER, state value must have type int or type Pointer, "
+                    f"got type {type(new_value)}"
+                )
+        elif it_kind == cccl_iterator_kind_t.ITERATOR:
+            if isinstance(new_value, IteratorState):
+                self.state_obj = new_value.reference
+                self.iter_data.size = (<IteratorState>new_value).size
+                self.iter_data.state = (<IteratorState>new_value).ptr
+            elif isinstance(new_value, Pointer):
+                self.state_obj = new_value.reference
+                if self.iter_data.size == 0:
+                    raise ValueError("Assigning incomplete state value to iterator without state size information")
+                self.iter_data.state = (<Pointer>new_value).ptr
+            elif PyObject_CheckBuffer(new_value):
+                self.iter_data.state = get_buffer_pointer(new_value, &self.iter_data.size)
+                self.state_obj = new_value
+            elif new_value is None:
+                self.state_obj = None
+                self.iter_data.size = 0
+                self.iter_data.state = NULL
+            else:
+                raise TypeError(
+                    "For iterator with type ITERATOR, state value must have type IteratorState or type bytes, "
+                    f"got type {type(new_value)}"
+                )
+        else:
+            raise TypeError("The new value should be an integer for iterators of POINTER kind, and bytes for ITERATOR kind")
+
+    @property
+    def type(self):
+        cdef cccl_iterator_kind_t it_kind = self.iter_data.type
+        if it_kind == cccl_iterator_kind_t.POINTER:
+            return IteratorKind.POINTER
+        else:
+            return IteratorKind.ITERATOR
+
+    @property
+    def value_type(self):
+        cdef cccl_type_info type_info = self.iter_data.value_type
+        return TypeInfo(type_info.size, type_info.alignment, type_info.type)
+
+    @property
+    def alignment(self):
+        """Return the iterator state alignment for serialization."""
+        return self.iter_data.alignment
+
+    def is_kind_pointer(self):
+        cdef cccl_iterator_kind_t it_kind = self.iter_data.type
+        return (it_kind == cccl_iterator_kind_t.POINTER)
+
+    def is_kind_iterator(self):
+        cdef cccl_iterator_kind_t it_kind = self.iter_data.type
+        return (it_kind == cccl_iterator_kind_t.ITERATOR)
+
+    def as_bytes(self):
+        "Debugging ulitity to get memory view into library struct"
+        cdef uint8_t[:] mem_view = bytearray(sizeof(self.iter_data))
+        memcpy(&mem_view[0], &self.iter_data, sizeof(self.iter_data))
+        return bytes(mem_view)
+
+    @property
+    def host_advance_fn(self):
+        return self.host_advance_obj
+
+    @host_advance_fn.setter
+    def host_advance_fn(self, func):
+        if (self.iter_data.type == cccl_iterator_kind_t.ITERATOR):
+            if func is not None:
+                self.iter_data.host_advance = unbox_host_advance_fn(func)
+                self.host_advance_obj = func
+            else:
+                self.iter_data.host_advance = NULL
+                self.host_advance_obj = None
+        else:
+            raise ValueError
+
+
+cdef class CommonData:
+    cdef int cc_major
+    cdef int cc_minor
+    cdef bytes encoded_cub_path
+    cdef bytes encoded_thrust_path
+    cdef bytes encoded_libcudacxx_path
+    cdef bytes encoded_ctk_path
+
+    def __cinit__(self, int cc_major, int cc_minor, str cub_path, str thrust_path, str libcudacxx_path, str ctk_path):
+        self.cc_major = cc_major
+        self.cc_minor = cc_minor
+        self.encoded_cub_path = cub_path.encode("utf-8")
+        self.encoded_thrust_path = thrust_path.encode("utf-8")
+        self.encoded_libcudacxx_path = libcudacxx_path.encode("utf-8")
+        self.encoded_ctk_path = ctk_path.encode("utf-8")
+
+    cdef inline int get_cc_major(self):
+        return self.cc_major
+
+    cdef inline int get_cc_minor(self):
+        return self.cc_minor
+
+    cdef inline const char * cub_path_get_c_str(self):
+        return <const char *>self.encoded_cub_path if self.encoded_cub_path else NULL
+
+    cdef inline const char * thrust_path_get_c_str(self):
+        return <const char *>self.encoded_thrust_path if self.encoded_thrust_path else NULL
+
+    cdef inline const char * libcudacxx_path_get_c_str(self):
+        return <const char *>self.encoded_libcudacxx_path if self.encoded_libcudacxx_path else NULL
+
+    cdef inline const char * ctk_path_get_c_str(self):
+        return <const char *>self.encoded_ctk_path if self.encoded_ctk_path else NULL
+
+    @property
+    def compute_capability(self):
+        return (self.cc_major, self.cc_minor)
+
+    @property
+    def cub_path(self):
+        return self.encoded_cub_path.decode("utf-8")
+
+    @property
+    def ctk_path(self):
+        return self.encoded_ctk_path.decode("utf-8")
+
+    @property
+    def thrust_path(self):
+        return self.encoded_thrust_path.decode("utf-8")
+
+    @property
+    def libcudacxx_path(self):
+        return self.encoded_libcudacxx_path.decode("utf-8")
+
+# --------------
+#   DeviceReduce
+# --------------
+
+cdef extern from "cccl/c/reduce.h":
+    cdef struct cccl_device_reduce_build_result_t 'cccl_device_reduce_build_result_t':
+        const char* payload
+        size_t payload_size
+        cccl_determinism_t determinism
+
+    cdef CUresult cccl_device_reduce_build(
+        cccl_device_reduce_build_result_t*,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_op_t,
+        cccl_value_t,
+        cccl_determinism_t,
+        int, int, const char*, const char*, const char*, const char*
+    ) nogil
+
+    cdef CUresult cccl_device_reduce(
+        cccl_device_reduce_build_result_t,
+        void *,
+        size_t *,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        uint64_t,
+        cccl_op_t,
+        cccl_value_t,
+        CUstream
+    ) nogil
+
+    cdef CUresult cccl_device_reduce_nondeterministic(
+        cccl_device_reduce_build_result_t,
+        void *,
+        size_t *,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        uint64_t,
+        cccl_op_t,
+        cccl_value_t,
+        CUstream
+    ) nogil
+
+    cdef CUresult cccl_device_reduce_cleanup(
+        cccl_device_reduce_build_result_t*
+    ) nogil
+
+
+cdef class DeviceReduceBuildResult:
+    cdef cccl_device_reduce_build_result_t build_data
+    # Whether cuLibraryLoadData + cuLibraryGetKernel have populated the
+    # library/kernel handles.
+    cdef public bint _loaded
+
+    def __cinit__(self, *args, **kwargs):
+        # Always zero the C struct so that __dealloc__ + cccl_device_reduce_cleanup
+        # is safe regardless of which initialization path is taken (regular build,
+        # alternate factory like deserialize, or a build that raises mid-way).
+        memset(&self.build_data, 0, sizeof(cccl_device_reduce_build_result_t))
+        self._loaded = False
+
+    def __init__(
+        DeviceReduceBuildResult self,
+        Iterator d_in,
+        Iterator d_out,
+        Op op,
+        Value h_init,
+        cccl_determinism_t determinism,
+        CommonData common_data
+    ):
+        cdef CUresult status = -1
+        cdef int cc_major = common_data.get_cc_major()
+        cdef int cc_minor = common_data.get_cc_minor()
+        cdef const char *cub_path = common_data.cub_path_get_c_str()
+        cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+        cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+        cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+
+        with nogil:
+            status = cccl_device_reduce_build(
+                &self.build_data,
+                d_in.iter_data,
+                d_out.iter_data,
+                op.op_data,
+                h_init.value_data,
+                determinism,
+                cc_major,
+                cc_minor,
+                cub_path,
+                thrust_path,
+                libcudacxx_path,
+                ctk_path,
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed building reduce, error code: {status}"
+            )
+
+    def __dealloc__(DeviceReduceBuildResult self):
+        cdef CUresult status = -1
+        with nogil:
+            status = cccl_device_reduce_cleanup(&self.build_data)
+        if (status != 0):
+            print(f"Return code {status} encountered during reduce result cleanup")
+
+    cpdef size_t compute(
+        DeviceReduceBuildResult self,
+        temp_storage_ptr,
+        temp_storage_bytes,
+        Iterator d_in,
+        Iterator d_out,
+        size_t num_items,
+        Op op,
+        Value h_init,
+        stream
+    ):
+        cdef CUresult status = -1
+        cdef void *storage_ptr = (<void *><uintptr_t>temp_storage_ptr) if temp_storage_ptr else NULL
+        cdef size_t storage_sz = <size_t>temp_storage_bytes
+        cdef CUstream c_stream = <CUstream><uintptr_t>(stream) if stream else NULL
+
+        with nogil:
+            status = cccl_device_reduce(
+                self.build_data,
+                storage_ptr,
+                &storage_sz,
+                d_in.iter_data,
+                d_out.iter_data,
+                <uint64_t>num_items,
+                op.op_data,
+                h_init.value_data,
+                c_stream
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed executing reduce, error code: {status}"
+            )
+        return storage_sz
+
+    cpdef size_t compute_nondeterministic(
+        DeviceReduceBuildResult self,
+        temp_storage_ptr,
+        temp_storage_bytes,
+        Iterator d_in,
+        Iterator d_out,
+        size_t num_items,
+        Op op,
+        Value h_init,
+        stream
+    ):
+        cdef CUresult status = -1
+        cdef void *storage_ptr = (<void *><uintptr_t>temp_storage_ptr) if temp_storage_ptr else NULL
+        cdef size_t storage_sz = <size_t>temp_storage_bytes
+        cdef CUstream c_stream = <CUstream><uintptr_t>(stream) if stream else NULL
+
+        with nogil:
+            status = cccl_device_reduce_nondeterministic(
+                self.build_data,
+                storage_ptr,
+                &storage_sz,
+                d_in.iter_data,
+                d_out.iter_data,
+                <uint64_t>num_items,
+                op.op_data,
+                h_init.value_data,
+                c_stream
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed executing reduce not guaranteed determinism, error code: {status}"
+            )
+        return storage_sz
+
+    def _get_cubin(self):
+        return PyBytes_FromStringAndSize(
+            <const char*>self.build_data.payload,
+            self.build_data.payload_size
+        )
+
+    @property
+    def determinism(DeviceReduceBuildResult self):
+        return <int>self.build_data.determinism
+
+    def serialize(DeviceReduceBuildResult self):
+        return _reduce_serialize(self)
+
+    @staticmethod
+    def deserialize(blob, load=True, check_cc=True):
+        return _reduce_deserialize(blob, load, check_cc)
+
+    @staticmethod
+    def compile(*args):
+        return _reduce_compile(*args)
+
+    def load(DeviceReduceBuildResult self):
+        # Idempotent: the fused build and a prior load() both set _loaded, so
+        # this is a cheap no-op after the first successful load. The guard also
+        # keeps the (unsupported) v2 backend's load stub from firing on an
+        # already-loaded result.
+        if self._loaded:
+            return
+        _load_once(self, _reduce_load)
+
+# ------------
+#   DeviceScan
+# ------------
+
+
+cdef extern from "cccl/c/scan.h":
+    ctypedef bint _Bool
+
+    cdef struct cccl_device_scan_build_result_t 'cccl_device_scan_build_result_t':
+        const char* payload
+        size_t payload_size
+
+    cdef CUresult cccl_device_scan_build(
+        cccl_device_scan_build_result_t*,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_op_t,
+        cccl_type_info,
+        _Bool,
+        cccl_init_kind_t,
+        int, int, const char*, const char*, const char*, const char*
+    ) nogil
+
+    cdef CUresult cccl_device_exclusive_scan(
+        cccl_device_scan_build_result_t,
+        void *,
+        size_t *,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        uint64_t,
+        cccl_op_t,
+        cccl_value_t,
+        CUstream
+    ) nogil
+
+    cdef CUresult cccl_device_inclusive_scan(
+        cccl_device_scan_build_result_t,
+        void *,
+        size_t *,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        uint64_t,
+        cccl_op_t,
+        cccl_value_t,
+        CUstream
+    ) nogil
+
+    cdef CUresult cccl_device_exclusive_scan_future_value(
+        cccl_device_scan_build_result_t,
+        void *,
+        size_t *,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        uint64_t,
+        cccl_op_t,
+        cccl_iterator_t,
+        CUstream
+    ) nogil
+
+    cdef CUresult cccl_device_inclusive_scan_future_value(
+        cccl_device_scan_build_result_t,
+        void *,
+        size_t *,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        uint64_t,
+        cccl_op_t,
+        cccl_iterator_t,
+        CUstream
+    ) nogil
+
+    cdef CUresult cccl_device_inclusive_scan_no_init(
+        cccl_device_scan_build_result_t,
+        void *,
+        size_t *,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        uint64_t,
+        cccl_op_t,
+        CUstream
+    ) nogil
+
+    cdef CUresult cccl_device_scan_cleanup(
+        cccl_device_scan_build_result_t*
+    ) nogil
+
+
+cdef class DeviceScanBuildResult:
+    cdef cccl_device_scan_build_result_t build_data
+    cdef public bint _loaded
+
+    def __cinit__(self, *args, **kwargs):
+        memset(&self.build_data, 0, sizeof(cccl_device_scan_build_result_t))
+        self._loaded = False
+
+    def __init__(
+        DeviceScanBuildResult self,
+        Iterator d_in,
+        Iterator d_out,
+        Op op,
+        TypeInfo init_type,
+        bint force_inclusive,
+        cccl_init_kind_t init_kind,
+        CommonData common_data
+    ):
+        cdef CUresult status = -1
+        cdef int cc_major = common_data.get_cc_major()
+        cdef int cc_minor = common_data.get_cc_minor()
+        cdef const char *cub_path = common_data.cub_path_get_c_str()
+        cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+        cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+        cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+
+        with nogil:
+            status = cccl_device_scan_build(
+                &self.build_data,
+                d_in.iter_data,
+                d_out.iter_data,
+                op.op_data,
+                init_type.type_info,
+                force_inclusive,
+                init_kind,
+                cc_major,
+                cc_minor,
+                cub_path,
+                thrust_path,
+                libcudacxx_path,
+                ctk_path,
+            )
+        if status != 0:
+            raise RuntimeError(f"Error {status} building scan")
+
+    def __dealloc__(DeviceScanBuildResult self):
+        cdef CUresult status = -1
+        with nogil:
+            status = cccl_device_scan_cleanup(&self.build_data)
+        if (status != 0):
+            print(f"Return code {status} encountered during scan result cleanup")
+
+    cpdef size_t compute_inclusive(
+        DeviceScanBuildResult self,
+        temp_storage_ptr,
+        temp_storage_bytes,
+        Iterator d_in,
+        Iterator d_out,
+        size_t num_items,
+        Op op,
+        Value init_value,
+        stream
+    ):
+        cdef CUresult status = -1
+        cdef void *storage_ptr = (<void *><uintptr_t>temp_storage_ptr) if temp_storage_ptr else NULL
+        cdef size_t storage_sz = <size_t>temp_storage_bytes
+        cdef CUstream c_stream = <CUstream><uintptr_t>(stream) if stream else NULL
+
+        with nogil:
+            status = cccl_device_inclusive_scan(
+                self.build_data,
+                storage_ptr,
+                &storage_sz,
+                d_in.iter_data,
+                d_out.iter_data,
+                <uint64_t>num_items,
+                op.op_data,
+                init_value.value_data,
+                c_stream
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed executing inclusive scan, error code: {status}"
+            )
+        return storage_sz
+
+    cpdef size_t compute_exclusive(
+        DeviceScanBuildResult self,
+        temp_storage_ptr,
+        temp_storage_bytes,
+        Iterator d_in,
+        Iterator d_out,
+        size_t num_items,
+        Op op,
+        Value init_value,
+        stream
+    ):
+        cdef CUresult status = -1
+        cdef void *storage_ptr = (<void *><uintptr_t>temp_storage_ptr) if temp_storage_ptr else NULL
+        cdef size_t storage_sz = <size_t>temp_storage_bytes
+        cdef CUstream c_stream = <CUstream><uintptr_t>(stream) if stream else NULL
+
+        with nogil:
+            status = cccl_device_exclusive_scan(
+                self.build_data,
+                storage_ptr,
+                &storage_sz,
+                d_in.iter_data,
+                d_out.iter_data,
+                <uint64_t>num_items,
+                op.op_data,
+                init_value.value_data,
+                c_stream
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed executing exclusive scan, error code: {status}"
+            )
+        return storage_sz
+
+    cpdef size_t compute_inclusive_future_value(
+        DeviceScanBuildResult self,
+        temp_storage_ptr,
+        temp_storage_bytes,
+        Iterator d_in,
+        Iterator d_out,
+        size_t num_items,
+        Op op,
+        Iterator init_value,
+        stream
+    ):
+        cdef CUresult status = -1
+        cdef void *storage_ptr = (<void *><uintptr_t>temp_storage_ptr) if temp_storage_ptr else NULL
+        cdef size_t storage_sz = <size_t>temp_storage_bytes
+        cdef CUstream c_stream = <CUstream><uintptr_t>(stream) if stream else NULL
+
+        with nogil:
+            status = cccl_device_inclusive_scan_future_value(
+                self.build_data,
+                storage_ptr,
+                &storage_sz,
+                d_in.iter_data,
+                d_out.iter_data,
+                <uint64_t>num_items,
+                op.op_data,
+                init_value.iter_data,
+                c_stream
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed executing inclusive scan, error code: {status}"
+            )
+        return storage_sz
+
+    cpdef size_t compute_exclusive_future_value(
+        DeviceScanBuildResult self,
+        temp_storage_ptr,
+        temp_storage_bytes,
+        Iterator d_in,
+        Iterator d_out,
+        size_t num_items,
+        Op op,
+        Iterator init_value,
+        stream
+    ):
+        cdef CUresult status = -1
+        cdef void *storage_ptr = (<void *><uintptr_t>temp_storage_ptr) if temp_storage_ptr else NULL
+        cdef size_t storage_sz = <size_t>temp_storage_bytes
+        cdef CUstream c_stream = <CUstream><uintptr_t>(stream) if stream else NULL
+
+        with nogil:
+            status = cccl_device_exclusive_scan_future_value(
+                self.build_data,
+                storage_ptr,
+                &storage_sz,
+                d_in.iter_data,
+                d_out.iter_data,
+                <uint64_t>num_items,
+                op.op_data,
+                init_value.iter_data,
+                c_stream
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed executing exclusive scan, error code: {status}"
+            )
+        return storage_sz
+
+    cpdef size_t compute_inclusive_no_init(
+        DeviceScanBuildResult self,
+        temp_storage_ptr,
+        temp_storage_bytes,
+        Iterator d_in,
+        Iterator d_out,
+        size_t num_items,
+        Op op,
+        object init_value,
+        stream
+    ):
+        cdef CUresult status = -1
+        cdef void *storage_ptr = (<void *><uintptr_t>temp_storage_ptr) if temp_storage_ptr else NULL
+        cdef size_t storage_sz = <size_t>temp_storage_bytes
+        cdef CUstream c_stream = <CUstream><uintptr_t>(stream) if stream else NULL
+
+        with nogil:
+            status = cccl_device_inclusive_scan_no_init(
+                self.build_data,
+                storage_ptr,
+                &storage_sz,
+                d_in.iter_data,
+                d_out.iter_data,
+                <uint64_t>num_items,
+                op.op_data,
+                c_stream
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed executing inclusive scan, error code: {status}"
+            )
+        return storage_sz
+
+    def _get_cubin(self):
+        return PyBytes_FromStringAndSize(
+            <const char*>self.build_data.payload,
+            self.build_data.payload_size
+        )
+
+    def serialize(DeviceScanBuildResult self):
+        return _scan_serialize(self)
+
+    @staticmethod
+    def deserialize(blob, load=True, check_cc=True):
+        return _scan_deserialize(blob, load, check_cc)
+
+    @staticmethod
+    def compile(*args):
+        return _scan_compile(*args)
+
+    def load(self):
+        if self._loaded:
+            return
+        _load_once(self, _scan_load)
+
+# -----------------------
+#   DeviceSegmentedReduce
+# -----------------------
+
+
+cdef extern from "cccl/c/segmented_reduce.h":
+    cdef struct cccl_device_segmented_reduce_build_result_t 'cccl_device_segmented_reduce_build_result_t':
+        const char* payload
+        size_t payload_size
+
+    cdef CUresult cccl_device_segmented_reduce_build(
+        cccl_device_segmented_reduce_build_result_t*,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_op_t,
+        cccl_value_t,
+        int, int, const char*, const char*, const char*, const char*
+    ) nogil
+
+    # `cccl_device_segmented_reduce` (the execute entry point) is declared in the
+    # generated _bindings_segmented_reduce_backend.pxi included below, because its
+    # signature differs between the v1 and v2 backends.
+
+
+    cdef CUresult cccl_device_segmented_reduce_cleanup(
+        cccl_device_segmented_reduce_build_result_t* bld_ptr
+    ) nogil
+
+
+# v1 and v2 disagree on whether `cccl_device_segmented_reduce` takes a
+# `size_t max_segment_size` argument. The .pxi pulled in here declares the
+# extern and a uniform `_call_segmented_reduce()` helper that hides the
+# difference; CMake configure_file picks the right backend variant.
+include "_bindings_segmented_reduce_backend.pxi"
+
+
+cdef class DeviceSegmentedReduceBuildResult:
+    cdef cccl_device_segmented_reduce_build_result_t build_data
+    cdef public bint _loaded
+
+    def __cinit__(self, *args, **kwargs):
+        memset(&self.build_data, 0, sizeof(cccl_device_segmented_reduce_build_result_t))
+        self._loaded = False
+
+    def __init__(
+        DeviceSegmentedReduceBuildResult self,
+        Iterator d_in,
+        Iterator d_out,
+        Iterator start_offsets,
+        Iterator end_offsets,
+        Op op,
+        Value h_init,
+        CommonData common_data
+    ):
+        cdef CUresult status = -1
+        cdef int cc_major = common_data.get_cc_major()
+        cdef int cc_minor = common_data.get_cc_minor()
+        cdef const char *cub_path = common_data.cub_path_get_c_str()
+        cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+        cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+        cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+
+        with nogil:
+            status = cccl_device_segmented_reduce_build(
+                &self.build_data,
+                d_in.iter_data,
+                d_out.iter_data,
+                start_offsets.iter_data,
+                end_offsets.iter_data,
+                op.op_data,
+                h_init.value_data,
+                cc_major,
+                cc_minor,
+                cub_path,
+                thrust_path,
+                libcudacxx_path,
+                ctk_path,
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed building segmented_reduce, error code: {status}"
+            )
+
+    def __dealloc__(DeviceSegmentedReduceBuildResult self):
+        cdef CUresult status = -1
+        with nogil:
+            status = cccl_device_segmented_reduce_cleanup(&self.build_data)
+        if (status != 0):
+            print(f"Return code {status} encountered during segmented_reduce result cleanup")
+
+    cpdef size_t compute(
+        DeviceSegmentedReduceBuildResult self,
+        temp_storage_ptr,
+        temp_storage_bytes,
+        Iterator d_in,
+        Iterator d_out,
+        size_t num_items,
+        Iterator start_offsets,
+        Iterator end_offsets,
+        Op op,
+        Value h_init,
+        size_t max_segment_size=0,  # accepted for v1 API compat; v2 ignores
+        stream=None
+    ):
+        cdef CUresult status = -1
+        cdef void *storage_ptr = (<void *><uintptr_t>temp_storage_ptr) if temp_storage_ptr else NULL
+        cdef size_t storage_sz = <size_t>temp_storage_bytes
+        cdef CUstream c_stream = <CUstream><uintptr_t>(stream) if stream else NULL
+
+        with nogil:
+            status = _call_segmented_reduce(
+                self.build_data,
+                storage_ptr,
+                &storage_sz,
+                d_in.iter_data,
+                d_out.iter_data,
+                <uint64_t>num_items,
+                start_offsets.iter_data,
+                end_offsets.iter_data,
+                op.op_data,
+                h_init.value_data,
+                max_segment_size,
+                c_stream,
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed executing segmented_reduce, error code: {status}"
+            )
+        return storage_sz
+
+    def _get_cubin(self):
+        return PyBytes_FromStringAndSize(
+            <const char*>self.build_data.payload,
+            self.build_data.payload_size
+        )
+
+    def serialize(DeviceSegmentedReduceBuildResult self):
+        return _segmented_reduce_serialize(self)
+
+    @staticmethod
+    def deserialize(blob, load=True, check_cc=True):
+        return _segmented_reduce_deserialize(blob, load, check_cc)
+
+    @staticmethod
+    def compile(*args):
+        return _segmented_reduce_compile(*args)
+
+    def load(self):
+        if self._loaded:
+            return
+        _load_once(self, _segmented_reduce_load)
+
+# -----------------
+#   DeviceMergeSort
+# -----------------
+
+
+cdef extern from "cccl/c/merge_sort.h":
+    cdef struct cccl_device_merge_sort_build_result_t 'cccl_device_merge_sort_build_result_t':
+        const char* payload
+        size_t payload_size
+
+    cdef CUresult cccl_device_merge_sort_build(
+        cccl_device_merge_sort_build_result_t *bld_ptr,
+        cccl_iterator_t d_in_keys,
+        cccl_iterator_t d_in_items,
+        cccl_iterator_t d_out_keys,
+        cccl_iterator_t d_out_items,
+        cccl_op_t,
+        int, int, const char*, const char*, const char*, const char*
+    ) nogil
+
+    cdef CUresult cccl_device_merge_sort(
+        cccl_device_merge_sort_build_result_t,
+        void *,
+        size_t *,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        uint64_t,
+        cccl_op_t,
+        CUstream
+    ) nogil
+
+    cdef CUresult cccl_device_merge_sort_cleanup(
+        cccl_device_merge_sort_build_result_t* bld_ptr
+    ) nogil
+
+
+cdef class DeviceMergeSortBuildResult:
+    cdef cccl_device_merge_sort_build_result_t build_data
+    cdef public bint _loaded
+
+    def __cinit__(self, *args, **kwargs):
+        memset(&self.build_data, 0, sizeof(cccl_device_merge_sort_build_result_t))
+        self._loaded = False
+
+    def __init__(
+        DeviceMergeSortBuildResult self,
+        Iterator d_in_keys,
+        Iterator d_in_items,
+        Iterator d_out_keys,
+        Iterator d_out_items,
+        Op op,
+        CommonData common_data
+    ):
+        cdef CUresult status = -1
+        cdef int cc_major = common_data.get_cc_major()
+        cdef int cc_minor = common_data.get_cc_minor()
+        cdef const char *cub_path = common_data.cub_path_get_c_str()
+        cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+        cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+        cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+
+        with nogil:
+            status = cccl_device_merge_sort_build(
+                &self.build_data,
+                d_in_keys.iter_data,
+                d_in_items.iter_data,
+                d_out_keys.iter_data,
+                d_out_items.iter_data,
+                op.op_data,
+                cc_major,
+                cc_minor,
+                cub_path,
+                thrust_path,
+                libcudacxx_path,
+                ctk_path,
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed building merge_sort, error code: {status}"
+            )
+
+    def __dealloc__(DeviceMergeSortBuildResult self):
+        cdef CUresult status = -1
+        with nogil:
+            status = cccl_device_merge_sort_cleanup(&self.build_data)
+        if (status != 0):
+            print(f"Return code {status} encountered during merge_sort result cleanup")
+
+    cpdef size_t compute(
+        DeviceMergeSortBuildResult self,
+        temp_storage_ptr,
+        temp_storage_bytes,
+        Iterator d_in_keys,
+        Iterator d_in_items,
+        Iterator d_out_keys,
+        Iterator d_out_items,
+        size_t num_items,
+        Op op,
+        stream
+    ):
+        cdef CUresult status = -1
+        cdef void *storage_ptr = (<void *><uintptr_t>temp_storage_ptr) if temp_storage_ptr else NULL
+        cdef size_t storage_sz = <size_t>temp_storage_bytes
+        cdef CUstream c_stream = <CUstream><uintptr_t>(stream) if stream else NULL
+        with nogil:
+            status = cccl_device_merge_sort(
+                self.build_data,
+                storage_ptr,
+                &storage_sz,
+                d_in_keys.iter_data,
+                d_in_items.iter_data,
+                d_out_keys.iter_data,
+                d_out_items.iter_data,
+                <uint64_t>num_items,
+                op.op_data,
+                c_stream
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed executing merge_sort, error code: {status}"
+            )
+        return storage_sz
+
+
+    def _get_cubin(self):
+        return PyBytes_FromStringAndSize(
+            <const char*>self.build_data.payload,
+            self.build_data.payload_size
+        )
+
+    def serialize(DeviceMergeSortBuildResult self):
+        return _merge_sort_serialize(self)
+
+    @staticmethod
+    def deserialize(blob, load=True, check_cc=True):
+        return _merge_sort_deserialize(blob, load, check_cc)
+
+    @staticmethod
+    def compile(*args):
+        return _merge_sort_compile(*args)
+
+    def load(self):
+        if self._loaded:
+            return
+        _load_once(self, _merge_sort_load)
+
+
+# -------------------
+#   DeviceUniqueByKey
+# -------------------
+
+cdef extern from "cccl/c/unique_by_key.h":
+    cdef struct cccl_device_unique_by_key_build_result_t 'cccl_device_unique_by_key_build_result_t':
+        const char* payload
+        size_t payload_size
+
+
+    cdef CUresult cccl_device_unique_by_key_build(
+        cccl_device_unique_by_key_build_result_t *build_ptr,
+        cccl_iterator_t d_keys_in,
+        cccl_iterator_t d_values_in,
+        cccl_iterator_t d_keys_out,
+        cccl_iterator_t d_values_out,
+        cccl_iterator_t d_num_selected_out,
+        cccl_op_t comparison_op,
+        int, int, const char *, const char *, const char *, const char *
+    ) nogil
+
+    cdef CUresult cccl_device_unique_by_key(
+        cccl_device_unique_by_key_build_result_t build,
+        void *d_storage_ptr,
+        size_t *d_storage_nbytes,
+        cccl_iterator_t d_keys_in,
+        cccl_iterator_t d_values_in,
+        cccl_iterator_t d_keys_out,
+        cccl_iterator_t d_values_out,
+        cccl_iterator_t d_num_selected_out,
+        cccl_op_t comparison_op,
+        size_t num_items,
+        CUstream stream
+    ) nogil
+
+    cdef CUresult cccl_device_unique_by_key_cleanup(
+        cccl_device_unique_by_key_build_result_t *build_ptr,
+    ) nogil
+
+
+cdef class DeviceUniqueByKeyBuildResult:
+    cdef cccl_device_unique_by_key_build_result_t build_data
+    cdef public bint _loaded
+
+    def __cinit__(self, *args, **kwargs):
+        memset(&self.build_data, 0, sizeof(cccl_device_unique_by_key_build_result_t))
+        self._loaded = False
+
+    def __init__(
+        DeviceUniqueByKeyBuildResult self,
+        Iterator d_keys_in,
+        Iterator d_values_in,
+        Iterator d_keys_out,
+        Iterator d_values_out,
+        Iterator d_num_selected_out,
+        Op comparison_op,
+        CommonData common_data
+    ):
+        cdef CUresult status = -1
+        cdef int cc_major = common_data.get_cc_major()
+        cdef int cc_minor = common_data.get_cc_minor()
+        cdef const char *cub_path = common_data.cub_path_get_c_str()
+        cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+        cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+        cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+
+        with nogil:
+            status = cccl_device_unique_by_key_build(
+                &self.build_data,
+                d_keys_in.iter_data,
+                d_values_in.iter_data,
+                d_keys_out.iter_data,
+                d_values_out.iter_data,
+                d_num_selected_out.iter_data,
+                comparison_op.op_data,
+                cc_major,
+                cc_minor,
+                cub_path,
+                thrust_path,
+                libcudacxx_path,
+                ctk_path,
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed building unique_by_key, error code: {status}"
+            )
+
+    def __dealloc__(DeviceUniqueByKeyBuildResult self):
+        cdef CUresult status = -1
+        with nogil:
+            status = cccl_device_unique_by_key_cleanup(&self.build_data)
+        if (status != 0):
+            print(f"Return code {status} encountered during unique_by_key result cleanup")
+
+    cpdef size_t compute(
+        DeviceUniqueByKeyBuildResult self,
+        temp_storage_ptr,
+        temp_storage_bytes,
+        Iterator d_keys_in,
+        Iterator d_values_in,
+        Iterator d_keys_out,
+        Iterator d_values_out,
+        Iterator d_num_selected_out,
+        Op comparison_op,
+        size_t num_items,
+        stream
+    ):
+        cdef CUresult status = -1
+        cdef void *storage_ptr = (<void *><uintptr_t>temp_storage_ptr) if temp_storage_ptr else NULL
+        cdef size_t storage_sz = <size_t>temp_storage_bytes
+        cdef CUstream c_stream = <CUstream><uintptr_t>(stream) if stream else NULL
+
+        with nogil:
+            status = cccl_device_unique_by_key(
+                self.build_data,
+                storage_ptr,
+                &storage_sz,
+                d_keys_in.iter_data,
+                d_values_in.iter_data,
+                d_keys_out.iter_data,
+                d_values_out.iter_data,
+                d_num_selected_out.iter_data,
+                comparison_op.op_data,
+                <uint64_t>num_items,
+                c_stream
+            )
+
+        if status != 0:
+            raise RuntimeError(
+                f"Failed executing unique_by_key, error code: {status}"
+            )
+        return storage_sz
+
+    def _get_cubin(self):
+        return PyBytes_FromStringAndSize(
+            <const char*>self.build_data.payload,
+            self.build_data.payload_size
+        )
+
+    def serialize(DeviceUniqueByKeyBuildResult self):
+        return _unique_by_key_serialize(self)
+
+    @staticmethod
+    def deserialize(blob, load=True, check_cc=True):
+        return _unique_by_key_deserialize(blob, load, check_cc)
+
+    @staticmethod
+    def compile(*args):
+        return _unique_by_key_compile(*args)
+
+    def load(self):
+        if self._loaded:
+            return
+        _load_once(self, _unique_by_key_load)
+
+# -----------------
+# DeviceRadixSort
+# -----------------
+
+cdef extern from "cccl/c/radix_sort.h":
+    cdef struct cccl_device_radix_sort_build_result_t 'cccl_device_radix_sort_build_result_t':
+        const char* payload
+        size_t payload_size
+
+    cdef CUresult cccl_device_radix_sort_build(
+        cccl_device_radix_sort_build_result_t *build_ptr,
+        cccl_sort_order_t sort_order,
+        cccl_iterator_t d_keys_in,
+        cccl_iterator_t d_values_in,
+        cccl_op_t decomposer,
+        const char* decomposer_return_type,
+        int, int, const char *, const char *, const char *, const char *
+    ) nogil
+
+    cdef CUresult cccl_device_radix_sort(
+        cccl_device_radix_sort_build_result_t build,
+        void *d_storage_ptr,
+        size_t *d_storage_nbytes,
+        cccl_iterator_t d_keys_in,
+        cccl_iterator_t d_keys_out,
+        cccl_iterator_t d_values_in,
+        cccl_iterator_t d_values_out,
+        cccl_op_t decomposer,
+        size_t num_items,
+        int begin_bit,
+        int end_bit,
+        bint is_overwrite_okay,
+        int* selector,
+        CUstream stream
+    ) nogil
+
+    cdef CUresult cccl_device_radix_sort_cleanup(
+        cccl_device_radix_sort_build_result_t *build_ptr,
+    ) nogil
+
+
+cdef class DeviceRadixSortBuildResult:
+    cdef cccl_device_radix_sort_build_result_t build_data
+    cdef public bint _loaded
+
+    def __cinit__(self, *args, **kwargs):
+        memset(&self.build_data, 0, sizeof(cccl_device_radix_sort_build_result_t))
+        self._loaded = False
+
+    def __dealloc__(DeviceRadixSortBuildResult self):
+        cdef CUresult status = -1
+        with nogil:
+            status = cccl_device_radix_sort_cleanup(&self.build_data)
+        if (status != 0):
+            print(f"Return code {status} encountered during radix_sort result cleanup")
+
+    def __init__(
+        DeviceRadixSortBuildResult self,
+        cccl_sort_order_t order,
+        Iterator d_keys_in,
+        Iterator d_values_in,
+        Op decomposer_op,
+        const char* decomposer_return_type,
+        CommonData common_data
+    ):
+        cdef CUresult status = -1
+        cdef int cc_major = common_data.get_cc_major()
+        cdef int cc_minor = common_data.get_cc_minor()
+        cdef const char *cub_path = common_data.cub_path_get_c_str()
+        cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+        cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+        cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+
+        with nogil:
+            status = cccl_device_radix_sort_build(
+                &self.build_data,
+                order,
+                d_keys_in.iter_data,
+                d_values_in.iter_data,
+                decomposer_op.op_data,
+                decomposer_return_type,
+                cc_major,
+                cc_minor,
+                cub_path,
+                thrust_path,
+                libcudacxx_path,
+                ctk_path,
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed building radix_sort, error code: {status}"
+            )
+
+    cpdef tuple compute(
+        DeviceRadixSortBuildResult self,
+        temp_storage_ptr,
+        temp_storage_bytes,
+        Iterator d_keys_in,
+        Iterator d_keys_out,
+        Iterator d_values_in,
+        Iterator d_values_out,
+        Op decomposer_op,
+        size_t num_items,
+        int begin_bit,
+        int end_bit,
+        bint is_overwrite_okay,
+        selector,
+        stream
+    ):
+        cdef CUresult status = -1
+        cdef void *storage_ptr = (<void *><size_t>temp_storage_ptr) if temp_storage_ptr else NULL
+        cdef size_t storage_sz = <size_t>temp_storage_bytes
+        cdef int selector_int = <int>selector
+        cdef CUstream c_stream = <CUstream><size_t>(stream) if stream else NULL
+
+        with nogil:
+            status = cccl_device_radix_sort(
+                self.build_data,
+                storage_ptr,
+                &storage_sz,
+                d_keys_in.iter_data,
+                d_keys_out.iter_data,
+                d_values_in.iter_data,
+                d_values_out.iter_data,
+                decomposer_op.op_data,
+                <uint64_t>num_items,
+                begin_bit,
+                end_bit,
+                is_overwrite_okay,
+                &selector_int,
+                c_stream
+            )
+
+        if status != 0:
+            raise RuntimeError(
+                f"Failed executing radix_sort, error code: {status}"
+            )
+        return <object>storage_sz, <object>selector_int
+
+
+    def _get_cubin(self):
+        return PyBytes_FromStringAndSize(
+            <const char*>self.build_data.payload,
+            self.build_data.payload_size
+        )
+
+    def serialize(DeviceRadixSortBuildResult self):
+        return _radix_sort_serialize(self)
+
+    @staticmethod
+    def deserialize(blob, load=True, check_cc=True):
+        return _radix_sort_deserialize(blob, load, check_cc)
+
+    @staticmethod
+    def compile(*args):
+        return _radix_sort_compile(*args)
+
+    def load(self):
+        if self._loaded:
+            return
+        _load_once(self, _radix_sort_load)
+
+# --------------------------------------------
+#   DeviceUnaryTransform/DeviceBinaryTransform
+# --------------------------------------------
+cdef extern from "cccl/c/transform.h":
+    cdef struct cccl_device_transform_build_result_t:
+        const char* payload
+        size_t payload_size
+
+    cdef CUresult cccl_device_unary_transform_build(
+        cccl_device_transform_build_result_t *build_ptr,
+        cccl_iterator_t d_in,
+        cccl_iterator_t d_out,
+        cccl_op_t op,
+        int, int, const char *, const char *, const char *, const char *
+    ) nogil
+
+    cdef CUresult cccl_device_unary_transform(
+      cccl_device_transform_build_result_t build,
+      cccl_iterator_t d_in,
+      cccl_iterator_t d_out,
+      uint64_t num_items,
+      cccl_op_t op,
+      CUstream stream) nogil
+
+    cdef CUresult cccl_device_binary_transform_build(
+      cccl_device_transform_build_result_t* build_ptr,
+      cccl_iterator_t d_in1,
+      cccl_iterator_t d_in2,
+      cccl_iterator_t d_out,
+      cccl_op_t op,
+      int, int, const char *, const char *, const char *, const char *
+    ) nogil
+
+    cdef CUresult cccl_device_binary_transform(
+      cccl_device_transform_build_result_t build,
+      cccl_iterator_t d_in1,
+      cccl_iterator_t d_in2,
+      cccl_iterator_t d_out,
+      uint64_t num_items,
+      cccl_op_t op,
+      CUstream stream) nogil
+
+    cdef CUresult cccl_device_transform_cleanup(
+        cccl_device_transform_build_result_t *build_ptr,
+    ) nogil
+
+
+cdef class DeviceUnaryTransform:
+    cdef cccl_device_transform_build_result_t build_data
+    cdef public bint _loaded
+
+    def __cinit__(self, *args, **kwargs):
+        memset(&self.build_data, 0, sizeof(cccl_device_transform_build_result_t))
+        self._loaded = False
+
+    def __init__(
+        self,
+        Iterator d_in,
+        Iterator d_out,
+        Op op,
+        CommonData common_data
+    ):
+        cdef CUresult status = -1
+        cdef int cc_major = common_data.get_cc_major()
+        cdef int cc_minor = common_data.get_cc_minor()
+        cdef const char *cub_path = common_data.cub_path_get_c_str()
+        cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+        cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+        cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+
+        with nogil:
+            status = cccl_device_unary_transform_build(
+                &self.build_data,
+                d_in.iter_data,
+                d_out.iter_data,
+                op.op_data,
+                cc_major,
+                cc_minor,
+                cub_path,
+                thrust_path,
+                libcudacxx_path,
+                ctk_path,
+            )
+        if status != 0:
+            raise RuntimeError("Failed to build unary transform")
+
+    def __dealloc__(DeviceUnaryTransform self):
+        cdef CUresult status = -1
+        with nogil:
+            status = cccl_device_transform_cleanup(&self.build_data)
+        if (status != 0):
+            print(f"Return code {status} encountered during unary transform result cleanup")
+
+    cpdef void compute(
+        DeviceUnaryTransform self,
+        Iterator d_in,
+        Iterator d_out,
+        size_t num_items,
+        Op op,
+        stream
+    ):
+        cdef CUresult status = -1
+        cdef CUstream c_stream = <CUstream><uintptr_t>(stream) if stream else NULL
+        with nogil:
+            status = cccl_device_unary_transform(
+                self.build_data,
+                d_in.iter_data,
+                d_out.iter_data,
+                <uint64_t>num_items,
+                op.op_data,
+                c_stream
+            )
+        if (status != 0):
+            raise RuntimeError("Failed to compute unary transform")
+
+
+    def _get_cubin(self):
+        return PyBytes_FromStringAndSize(
+            <const char*>self.build_data.payload,
+            self.build_data.payload_size
+        )
+
+    def serialize(DeviceUnaryTransform self):
+        return _unary_transform_serialize(self)
+
+    @staticmethod
+    def deserialize(blob, load=True, check_cc=True):
+        return _unary_transform_deserialize(blob, load, check_cc)
+
+    @staticmethod
+    def compile(*args):
+        return _unary_transform_compile(*args)
+
+    def load(self):
+        if self._loaded:
+            return
+        _load_once(self, _unary_transform_load)
+
+
+cdef class DeviceBinaryTransform:
+    cdef cccl_device_transform_build_result_t build_data
+    cdef public bint _loaded
+
+    def __cinit__(self, *args, **kwargs):
+        memset(&self.build_data, 0, sizeof(cccl_device_transform_build_result_t))
+        self._loaded = False
+
+    def __init__(
+        self,
+        Iterator d_in1,
+        Iterator d_in2,
+        Iterator d_out,
+        Op op,
+        CommonData common_data
+    ):
+        cdef CUresult status = -1
+        cdef int cc_major = common_data.get_cc_major()
+        cdef int cc_minor = common_data.get_cc_minor()
+        cdef const char *cub_path = common_data.cub_path_get_c_str()
+        cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+        cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+        cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+
+        with nogil:
+            status = cccl_device_binary_transform_build(
+                &self.build_data,
+                d_in1.iter_data,
+                d_in2.iter_data,
+                d_out.iter_data,
+                op.op_data,
+                cc_major,
+                cc_minor,
+                cub_path,
+                thrust_path,
+                libcudacxx_path,
+                ctk_path,
+            )
+        if status != 0:
+            raise RuntimeError("Failed to build binary transform")
+
+    def __dealloc__(DeviceBinaryTransform self):
+        cdef CUresult status = -1
+        with nogil:
+            status = cccl_device_transform_cleanup(&self.build_data)
+        if (status != 0):
+            print(f"Return code {status} encountered during binary transform result cleanup")
+
+    cpdef void compute(
+        DeviceBinaryTransform self,
+        Iterator d_in1,
+        Iterator d_in2,
+        Iterator d_out,
+        size_t num_items,
+        Op op,
+        stream
+    ):
+        cdef CUresult status = -1
+        cdef CUstream c_stream = <CUstream><uintptr_t>(stream) if stream else NULL
+        with nogil:
+            status = cccl_device_binary_transform(
+                self.build_data,
+                d_in1.iter_data,
+                d_in2.iter_data,
+                d_out.iter_data,
+                <uint64_t>num_items,
+                op.op_data,
+                c_stream
+            )
+        if (status != 0):
+            raise RuntimeError("Failed to compute binary transform")
+
+    def _get_cubin(self):
+        return PyBytes_FromStringAndSize(
+            <const char*>self.build_data.payload,
+            self.build_data.payload_size
+        )
+
+    def serialize(DeviceBinaryTransform self):
+        return _binary_transform_serialize(self)
+
+    @staticmethod
+    def deserialize(blob, load=True, check_cc=True):
+        return _binary_transform_deserialize(blob, load, check_cc)
+
+    @staticmethod
+    def compile(*args):
+        return _binary_transform_compile(*args)
+
+    def load(self):
+        if self._loaded:
+            return
+        _load_once(self, _binary_transform_load)
+
+
+# -----------------
+#   DeviceHistogram
+# -----------------
+cdef extern from "cccl/c/histogram.h":
+    cdef struct cccl_device_histogram_build_result_t 'cccl_device_histogram_build_result_t':
+        const char* payload
+        size_t payload_size
+
+    cdef CUresult cccl_device_histogram_build(
+        cccl_device_histogram_build_result_t *build_ptr,
+        int num_channels,
+        int num_active_channels,
+        cccl_iterator_t d_samples,
+        int num_output_levels_val,
+        cccl_iterator_t d_output_histograms,
+        cccl_type_info level_type,
+        int64_t num_rows,
+        int64_t row_stride_samples,
+        bint is_evenly_segmented,
+        int, int, const char *, const char *, const char *, const char *
+    ) nogil
+
+    cdef CUresult cccl_device_histogram_even(
+        cccl_device_histogram_build_result_t build,
+        void *d_storage_ptr,
+        size_t *d_storage_nbytes,
+        cccl_iterator_t d_samples,
+        cccl_iterator_t d_output_histograms,
+        cccl_value_t num_output_levels,
+        cccl_value_t lower_level,
+        cccl_value_t upper_level,
+        int64_t num_row_pixels,
+        int64_t num_rows,
+        int64_t row_stride_samples,
+        CUstream stream
+    ) nogil
+
+    cdef CUresult cccl_device_histogram_cleanup(
+        cccl_device_histogram_build_result_t *build_ptr,
+    ) nogil
+
+
+cdef class DeviceHistogramBuildResult:
+    cdef cccl_device_histogram_build_result_t build_data
+    cdef public bint _loaded
+
+    def __cinit__(self, *args, **kwargs):
+        memset(&self.build_data, 0, sizeof(cccl_device_histogram_build_result_t))
+        self._loaded = False
+
+    def __dealloc__(DeviceHistogramBuildResult self):
+        cdef CUresult status = -1
+        with nogil:
+            status = cccl_device_histogram_cleanup(&self.build_data)
+        if (status != 0):
+            print(f"Return code {status} encountered during histogram result cleanup")
+
+
+    def __init__(
+        DeviceHistogramBuildResult self,
+        int num_channels,
+        int num_active_channels,
+        Iterator d_samples,
+        int num_levels,
+        Iterator d_histogram,
+        TypeInfo level_type,
+        int num_rows,
+        int row_stride_samples,
+        bint is_evenly_segmented,
+        CommonData common_data
+    ):
+        cdef CUresult status = -1
+        cdef int cc_major = common_data.get_cc_major()
+        cdef int cc_minor = common_data.get_cc_minor()
+        cdef const char *cub_path = common_data.cub_path_get_c_str()
+        cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+        cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+        cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+
+        with nogil:
+            status = cccl_device_histogram_build(
+                &self.build_data,
+                num_channels,
+                num_active_channels,
+                d_samples.iter_data,
+                num_levels,
+                d_histogram.iter_data,
+                level_type.type_info,
+                num_rows,
+                row_stride_samples,
+                is_evenly_segmented,
+                cc_major,
+                cc_minor,
+                cub_path,
+                thrust_path,
+                libcudacxx_path,
+                ctk_path,
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed building histogram, error code: {status}"
+            )
+
+    cpdef size_t compute_even(
+        DeviceHistogramBuildResult self,
+        temp_storage_ptr,
+        temp_storage_bytes,
+        Iterator d_samples,
+        Iterator d_histogram,
+        Value h_num_output_levels,
+        Value h_lower_level,
+        Value h_upper_level,
+        int num_row_pixels,
+        int num_rows,
+        int row_stride_samples,
+        stream
+    ):
+        cdef CUresult status = -1
+        cdef void *storage_ptr = (<void *><size_t>temp_storage_ptr) if temp_storage_ptr else NULL
+        cdef size_t storage_sz = <size_t>temp_storage_bytes
+        cdef CUstream c_stream = <CUstream><size_t>(stream) if stream else NULL
+
+        with nogil:
+            status = cccl_device_histogram_even(
+                self.build_data,
+                storage_ptr,
+                &storage_sz,
+                d_samples.iter_data,
+                d_histogram.iter_data,
+                h_num_output_levels.value_data,
+                h_lower_level.value_data,
+                h_upper_level.value_data,
+                num_row_pixels,
+                num_rows,
+                row_stride_samples,
+                c_stream
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed executing histogram, error code: {status}"
+            )
+        return storage_sz
+
+
+    def _get_cubin(self):
+        return PyBytes_FromStringAndSize(
+            <const char*>self.build_data.payload,
+            self.build_data.payload_size
+        )
+
+    def serialize(DeviceHistogramBuildResult self):
+        return _histogram_serialize(self)
+
+    @staticmethod
+    def deserialize(blob, load=True, check_cc=True):
+        return _histogram_deserialize(blob, load, check_cc)
+
+    @staticmethod
+    def compile(*args):
+        return _histogram_compile(*args)
+
+    def load(self):
+        if self._loaded:
+            return
+        _load_once(self, _histogram_load)
+
+
+# -------------------
+#   DeviceBinarySearch
+# -------------------
+# Backend-specific struct decl + cubin-extract helper.
+include "_bindings_binary_search_backend.pxi"
+
+cdef extern from "cccl/c/binary_search.h":
+
+    cdef CUresult cccl_device_binary_search_build(
+        cccl_device_binary_search_build_result_t*,
+        cccl_binary_search_mode_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_op_t,
+        int, int, const char*, const char*, const char*, const char*
+    ) nogil
+
+    cdef CUresult cccl_device_binary_search(
+        cccl_device_binary_search_build_result_t,
+        cccl_iterator_t,
+        uint64_t,
+        cccl_iterator_t,
+        uint64_t,
+        cccl_iterator_t,
+        cccl_op_t,
+        CUstream
+    ) nogil
+
+    cdef CUresult cccl_device_binary_search_cleanup(
+        cccl_device_binary_search_build_result_t *build_ptr
+    ) nogil
+
+
+cdef class DeviceBinarySearchBuildResult:
+    cdef cccl_device_binary_search_build_result_t build_data
+    cdef public bint _loaded
+
+    def __cinit__(self, *args, **kwargs):
+        memset(&self.build_data, 0, sizeof(cccl_device_binary_search_build_result_t))
+        self._loaded = False
+
+    def __dealloc__(DeviceBinarySearchBuildResult self):
+        cdef CUresult status = -1
+        with nogil:
+            status = cccl_device_binary_search_cleanup(&self.build_data)
+        if (status != 0):
+            print(f"Return code {status} encountered during binary_search result cleanup")
+
+    def __init__(
+        DeviceBinarySearchBuildResult self,
+        cccl_binary_search_mode_t mode,
+        Iterator d_data,
+        Iterator d_values,
+        Iterator d_out,
+        Op op,
+        CommonData common_data
+    ):
+        cdef CUresult status = -1
+        cdef int cc_major = common_data.get_cc_major()
+        cdef int cc_minor = common_data.get_cc_minor()
+        cdef const char *cub_path = common_data.cub_path_get_c_str()
+        cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+        cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+        cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+
+        with nogil:
+            status = cccl_device_binary_search_build(
+                &self.build_data,
+                mode,
+                d_data.iter_data,
+                d_values.iter_data,
+                d_out.iter_data,
+                op.op_data,
+                cc_major,
+                cc_minor,
+                cub_path,
+                thrust_path,
+                libcudacxx_path,
+                ctk_path,
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed building binary_search, error code: {status}"
+            )
+
+    cpdef void compute(
+        DeviceBinarySearchBuildResult self,
+        Iterator d_data,
+        size_t num_items,
+        Iterator d_values,
+        size_t num_values,
+        Iterator d_out,
+        Op op,
+        stream
+    ):
+        cdef CUresult status = -1
+        cdef CUstream c_stream = <CUstream><uintptr_t>(stream) if stream else NULL
+
+        with nogil:
+            status = cccl_device_binary_search(
+                self.build_data,
+                d_data.iter_data,
+                <uint64_t>num_items,
+                d_values.iter_data,
+                <uint64_t>num_values,
+                d_out.iter_data,
+                op.op_data,
+                c_stream
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed executing binary_search, error code: {status}"
+            )
+
+    def _get_cubin(self):
+        return _binary_search_cubin_bytes(&self.build_data)
+
+    def serialize(DeviceBinarySearchBuildResult self):
+        return _binary_search_serialize(self)
+
+    @staticmethod
+    def deserialize(blob, load=True, check_cc=True):
+        return _binary_search_deserialize(blob, load, check_cc)
+
+    @staticmethod
+    def compile(*args):
+        return _binary_search_compile(*args)
+
+    def load(self):
+        if self._loaded:
+            return
+        _load_once(self, _binary_search_load)
+
+
+# ----------------------------------
+# DeviceThreeWayPartitionBuildResult
+# ----------------------------------
+cdef extern from "cccl/c/three_way_partition.h":
+    cdef struct cccl_device_three_way_partition_build_result_t 'cccl_device_three_way_partition_build_result_t':
+        const char* payload
+        size_t payload_size
+
+    cdef CUresult cccl_device_three_way_partition_build(
+        cccl_device_three_way_partition_build_result_t *build_ptr,
+        cccl_iterator_t d_in,
+        cccl_iterator_t d_first_part_out,
+        cccl_iterator_t d_second_part_out,
+        cccl_iterator_t d_unselected_out,
+        cccl_iterator_t d_num_selected_out,
+        cccl_op_t select_first_part_op,
+        cccl_op_t select_second_part_op,
+        int, int, const char *, const char *, const char *, const char *
+    ) nogil
+
+    CUresult cccl_device_three_way_partition(
+        cccl_device_three_way_partition_build_result_t build,
+        void* d_temp_storage,
+        size_t* temp_storage_bytes,
+        cccl_iterator_t d_in,
+        cccl_iterator_t d_first_part_out,
+        cccl_iterator_t d_second_part_out,
+        cccl_iterator_t d_unselected_out,
+        cccl_iterator_t d_num_selected_out,
+        cccl_op_t select_first_part_op,
+        cccl_op_t select_second_part_op,
+        int64_t num_items,
+        CUstream stream
+    ) nogil
+
+    cdef CUresult cccl_device_three_way_partition_cleanup(
+        cccl_device_three_way_partition_build_result_t *build_ptr
+    ) nogil
+
+
+cdef class DeviceThreeWayPartitionBuildResult:
+    cdef cccl_device_three_way_partition_build_result_t build_data
+    cdef public bint _loaded
+
+    def __cinit__(self, *args, **kwargs):
+        memset(&self.build_data, 0, sizeof(cccl_device_three_way_partition_build_result_t))
+        self._loaded = False
+
+    def __dealloc__(DeviceThreeWayPartitionBuildResult self):
+        cdef CUresult status = -1
+        with nogil:
+            status = cccl_device_three_way_partition_cleanup(&self.build_data)
+        if (status != 0):
+            print(f"Return code {status} encountered during three_way_partition result cleanup")
+
+
+    def __init__(
+        DeviceThreeWayPartitionBuildResult self,
+        Iterator d_in,
+        Iterator d_first_part_out,
+        Iterator d_second_part_out,
+        Iterator d_unselected_out,
+        Iterator d_num_selected_out,
+        Op select_first_part_op,
+        Op select_second_part_op,
+        CommonData common_data
+    ):
+        cdef CUresult status = -1
+        cdef int cc_major = common_data.get_cc_major()
+        cdef int cc_minor = common_data.get_cc_minor()
+        cdef const char *cub_path = common_data.cub_path_get_c_str()
+        cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+        cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+        cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+
+        with nogil:
+            status = cccl_device_three_way_partition_build(
+                &self.build_data,
+                d_in.iter_data,
+                d_first_part_out.iter_data,
+                d_second_part_out.iter_data,
+                d_unselected_out.iter_data,
+                d_num_selected_out.iter_data,
+                select_first_part_op.op_data,
+                select_second_part_op.op_data,
+                cc_major,
+                cc_minor,
+                cub_path,
+                thrust_path,
+                libcudacxx_path,
+                ctk_path,
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed building three_way_partition, error code: {status}"
+            )
+
+    cpdef size_t compute(
+        DeviceThreeWayPartitionBuildResult self,
+        temp_storage_ptr,
+        temp_storage_bytes,
+        Iterator d_in,
+        Iterator d_first_part_out,
+        Iterator d_second_part_out,
+        Iterator d_unselected_out,
+        Iterator d_num_selected_out,
+        Op select_first_part_op,
+        Op select_second_part_op,
+        size_t num_items,
+        stream
+    ):
+        cdef CUresult status = -1
+        cdef void *storage_ptr = (<void *><uintptr_t>temp_storage_ptr) if temp_storage_ptr else NULL
+        cdef size_t storage_sz = <size_t>temp_storage_bytes
+        cdef CUstream c_stream = <CUstream><uintptr_t>(stream) if stream else NULL
+
+        with nogil:
+            status = cccl_device_three_way_partition(
+                self.build_data,
+                storage_ptr,
+                &storage_sz,
+                d_in.iter_data,
+                d_first_part_out.iter_data,
+                d_second_part_out.iter_data,
+                d_unselected_out.iter_data,
+                d_num_selected_out.iter_data,
+                select_first_part_op.op_data,
+                select_second_part_op.op_data,
+                <uint64_t>num_items,
+                c_stream
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed executing three_way_partition, error code: {status}"
+            )
+        return storage_sz
+
+    def _get_cubin(self):
+        return PyBytes_FromStringAndSize(
+            <const char*>self.build_data.payload,
+            self.build_data.payload_size
+        )
+
+    def serialize(DeviceThreeWayPartitionBuildResult self):
+        return _three_way_partition_serialize(self)
+
+    @staticmethod
+    def deserialize(blob, load=True, check_cc=True):
+        return _three_way_partition_deserialize(blob, load, check_cc)
+
+    @staticmethod
+    def compile(*args):
+        return _three_way_partition_compile(*args)
+
+    def load(self):
+        if self._loaded:
+            return
+        _load_once(self, _three_way_partition_load)
+
+
+# -------------------
+# DeviceSegmentedSort
+# -------------------
+
+cdef extern from "cccl/c/segmented_sort.h":
+    cdef struct cccl_device_segmented_sort_build_result_t 'cccl_device_segmented_sort_build_result_t':
+        const char* payload
+        size_t payload_size
+
+    cdef CUresult cccl_device_segmented_sort_build(
+        cccl_device_segmented_sort_build_result_t *build_ptr,
+        cccl_sort_order_t sort_order,
+        cccl_iterator_t d_keys_in,
+        cccl_iterator_t d_keys_out,
+        cccl_iterator_t begin_offset_in,
+        cccl_iterator_t end_offset_in,
+        int, int, const char *, const char *, const char *, const char *
+    ) nogil
+
+    cdef CUresult cccl_device_segmented_sort(
+        cccl_device_segmented_sort_build_result_t build,
+        void* d_temp_storage,
+        size_t* temp_storage_bytes,
+        cccl_iterator_t d_keys_in,
+        cccl_iterator_t d_keys_out,
+        cccl_iterator_t d_values_in,
+        cccl_iterator_t d_values_out,
+        int64_t num_items,
+        int64_t num_segments,
+        cccl_iterator_t start_offset_in,
+        cccl_iterator_t end_offset_in,
+        bint is_overwrite_okay,
+        int* selector,
+        CUstream stream
+    ) nogil
+
+    cdef CUresult cccl_device_segmented_sort_cleanup(
+        cccl_device_segmented_sort_build_result_t* build_ptr
+    ) nogil
+
+cdef class DeviceSegmentedSortBuildResult:
+    cdef cccl_device_segmented_sort_build_result_t build_data
+    cdef public bint _loaded
+
+    def __cinit__(self, *args, **kwargs):
+        memset(&self.build_data, 0, sizeof(cccl_device_segmented_sort_build_result_t))
+        self._loaded = False
+
+    def __dealloc__(DeviceSegmentedSortBuildResult self):
+        cdef CUresult status = -1
+        with nogil:
+            status = cccl_device_segmented_sort_cleanup(&self.build_data)
+        if (status != 0):
+            print(f"Return code {status} encountered during segmented_sort result cleanup")
+
+    def __init__(
+        DeviceSegmentedSortBuildResult self,
+        cccl_sort_order_t order,
+        Iterator d_keys_in,
+        Iterator d_values_in,
+        Iterator begin_offset_in,
+        Iterator end_offset_in,
+        CommonData common_data,
+    ):
+        cdef CUresult status = -1
+        cdef int cc_major = common_data.get_cc_major()
+        cdef int cc_minor = common_data.get_cc_minor()
+        cdef const char *cub_path = common_data.cub_path_get_c_str()
+        cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+        cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+        cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+
+        with nogil:
+            status = cccl_device_segmented_sort_build(
+                &self.build_data,
+                order,
+                d_keys_in.iter_data,
+                d_values_in.iter_data,
+                begin_offset_in.iter_data,
+                end_offset_in.iter_data,
+                cc_major,
+                cc_minor,
+                cub_path,
+                thrust_path,
+                libcudacxx_path,
+                ctk_path,
+            )
+        if status != 0:
+            raise RuntimeError(
+                f"Failed building segmented_sort, error code: {status}"
+            )
+
+    cpdef tuple compute(
+        DeviceSegmentedSortBuildResult self,
+        temp_storage_ptr,
+        temp_storage_bytes,
+        Iterator d_keys_in,
+        Iterator d_keys_out,
+        Iterator d_values_in,
+        Iterator d_values_out,
+        size_t num_items,
+        size_t num_segments,
+        Iterator start_offset_in,
+        Iterator end_offset_in,
+        bint is_overwrite_okay,
+        selector,
+        stream
+    ):
+        cdef CUresult status = -1
+        cdef void *storage_ptr = (<void *><size_t>temp_storage_ptr) if temp_storage_ptr else NULL
+        cdef size_t storage_sz = <size_t>temp_storage_bytes
+        cdef int selector_int = <int>selector
+        cdef CUstream c_stream = <CUstream><size_t>(stream) if stream else NULL
+
+        with nogil:
+            status = cccl_device_segmented_sort(
+                self.build_data,
+                storage_ptr,
+                &storage_sz,
+                d_keys_in.iter_data,
+                d_keys_out.iter_data,
+                d_values_in.iter_data,
+                d_values_out.iter_data,
+                <uint64_t>num_items,
+                <uint64_t>num_segments,
+                start_offset_in.iter_data,
+                end_offset_in.iter_data,
+                is_overwrite_okay,
+                &selector_int,
+                c_stream
+            )
+
+        if status != 0:
+            raise RuntimeError(
+                f"Failed executing segmented_sort, error code: {status}"
+            )
+        return <object>storage_sz, <object>selector_int
+
+
+    def _get_cubin(self):
+        return PyBytes_FromStringAndSize(
+            <const char*>self.build_data.payload,
+            self.build_data.payload_size
+        )
+
+    def serialize(DeviceSegmentedSortBuildResult self):
+        return _segmented_sort_serialize(self)
+
+    @staticmethod
+    def deserialize(blob, load=True, check_cc=True):
+        return _segmented_sort_deserialize(blob, load, check_cc)
+
+    @staticmethod
+    def compile(*args):
+        return _segmented_sort_compile(*args)
+
+    def load(self):
+        if self._loaded:
+            return
+        _load_once(self, _segmented_sort_load)
+
+include "_bindings_serialization.pxi"
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_op_code_type_v1.pxi b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_op_code_type_v1.pxi
new file mode 100644
index 00000000..03a33935
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_op_code_type_v1.pxi
@@ -0,0 +1,14 @@
+# v1 (cccl.c.parallel, NVRTC) — cccl_op_code_type enum + string-to-enum helper.
+# Selected at CMake configure time and configure_file'd to the build dir as
+# `_bindings_op_code_type.pxi`. v1's types.h does not define CCCL_OP_LLVM_IR.
+
+cdef extern from "cccl/c/types.h":
+    cdef enum cccl_op_code_type:
+        CCCL_OP_LTOIR
+        CCCL_OP_CPP_SOURCE
+
+
+cdef inline cccl_op_code_type _parse_code_type(str s) noexcept:
+    if s == "cpp_source":
+        return CCCL_OP_CPP_SOURCE
+    return CCCL_OP_LTOIR
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_op_code_type_v2.pxi b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_op_code_type_v2.pxi
new file mode 100644
index 00000000..df6b3f14
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_op_code_type_v2.pxi
@@ -0,0 +1,17 @@
+# v2 (cccl.c.parallel.v2, HostJIT) — cccl_op_code_type enum + string-to-enum
+# helper. Selected at CMake configure time and configure_file'd to the build
+# dir as `_bindings_op_code_type.pxi`. v2's types.h adds CCCL_OP_LLVM_IR.
+
+cdef extern from "cccl/c/types.h":
+    cdef enum cccl_op_code_type:
+        CCCL_OP_LTOIR
+        CCCL_OP_CPP_SOURCE
+        CCCL_OP_LLVM_IR
+
+
+cdef inline cccl_op_code_type _parse_code_type(str s) noexcept:
+    if s == "llvm_ir":
+        return CCCL_OP_LLVM_IR
+    if s == "cpp_source":
+        return CCCL_OP_CPP_SOURCE
+    return CCCL_OP_LTOIR
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_segmented_reduce_backend_v1.pxi b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_segmented_reduce_backend_v1.pxi
new file mode 100644
index 00000000..bc499bd1
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_segmented_reduce_backend_v1.pxi
@@ -0,0 +1,40 @@
+# v1 (cccl.c.parallel, NVRTC) — segmented_reduce extern + uniform call helper.
+# Selected at CMake configure time and configure_file'd to the build dir as
+# `_bindings_segmented_reduce_backend.pxi`. v1's signature takes
+# `size_t max_segment_size` between `init` and `stream`.
+
+cdef extern from "cccl/c/segmented_reduce.h":
+    cdef CUresult cccl_device_segmented_reduce(
+        cccl_device_segmented_reduce_build_result_t,
+        void *,
+        size_t *,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        uint64_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_op_t,
+        cccl_value_t,
+        size_t,
+        CUstream
+    ) nogil
+
+
+cdef inline CUresult _call_segmented_reduce(
+    cccl_device_segmented_reduce_build_result_t bld,
+    void* storage_ptr,
+    size_t* storage_sz,
+    cccl_iterator_t d_in,
+    cccl_iterator_t d_out,
+    uint64_t num_items,
+    cccl_iterator_t start_offsets,
+    cccl_iterator_t end_offsets,
+    cccl_op_t op_data,
+    cccl_value_t init,
+    size_t max_segment_size,
+    CUstream stream,
+) nogil:
+    return cccl_device_segmented_reduce(
+        bld, storage_ptr, storage_sz, d_in, d_out, num_items,
+        start_offsets, end_offsets, op_data, init, max_segment_size, stream
+    )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_segmented_reduce_backend_v2.pxi b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_segmented_reduce_backend_v2.pxi
new file mode 100644
index 00000000..45de2bf2
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_segmented_reduce_backend_v2.pxi
@@ -0,0 +1,38 @@
+# v2 (cccl.c.parallel.v2, HostJIT) — segmented_reduce extern + uniform call
+# helper. v2 dropped `size_t max_segment_size`; the helper accepts it for
+# signature-compatibility with v1 and silently ignores it.
+
+cdef extern from "cccl/c/segmented_reduce.h":
+    cdef CUresult cccl_device_segmented_reduce(
+        cccl_device_segmented_reduce_build_result_t,
+        void *,
+        size_t *,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        uint64_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_op_t,
+        cccl_value_t,
+        CUstream
+    ) nogil
+
+
+cdef inline CUresult _call_segmented_reduce(
+    cccl_device_segmented_reduce_build_result_t bld,
+    void* storage_ptr,
+    size_t* storage_sz,
+    cccl_iterator_t d_in,
+    cccl_iterator_t d_out,
+    uint64_t num_items,
+    cccl_iterator_t start_offsets,
+    cccl_iterator_t end_offsets,
+    cccl_op_t op_data,
+    cccl_value_t init,
+    size_t max_segment_size,
+    CUstream stream,
+) nogil:
+    return cccl_device_segmented_reduce(
+        bld, storage_ptr, storage_sz, d_in, d_out, num_items,
+        start_offsets, end_offsets, op_data, init, stream
+    )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_serialization_v1.pxi b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_serialization_v1.pxi
new file mode 100644
index 00000000..9914af5d
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_serialization_v1.pxi
@@ -0,0 +1,1363 @@
+# Auto-generated serialization extern declarations and implementations for v1 backend.
+# Included at the end of _bindings_impl.pyx by CMake-selected include.
+
+cdef extern from "cccl/c/serialization.h":
+    cdef void cccl_serialization_buffer_free(void*) nogil
+
+cdef extern from "cccl/c/serialization_diagnostics.h":
+    cdef const char* cccl_serialization_last_error() nogil
+    cdef CUresult cccl_serialization_validate_blob(const void*, size_t) nogil
+
+
+# CUDA_ERROR_NO_BINARY_FOR_GPU: the one code validate_blob returns for a CUBIN
+# cc-major vs current-device mismatch (vs CUDA_ERROR_INVALID_VALUE for header
+# issues: bad magic / format / ABI / payload kind).
+cdef int _CUDA_ERROR_NO_BINARY_FOR_GPU = 209
+
+cdef _check_loadable(const void* buf, size_t size, bint check_cc):
+    # Always validate the blob header (magic / format / ABI version / payload
+    # kind) and raise the C layer's descriptive message on any problem. The
+    # compute-capability-major check is only enforced when check_cc is True
+    # (a single-target blob that must match this device); for a multi-arch
+    # artifact it is deferred — the matching build result is chosen by resolve_build_result
+    # and its cc is settled when it loads, so a non-matching-major build result here
+    # is expected and must not raise.
+    cdef CUresult st
+    with nogil:
+        st = cccl_serialization_validate_blob(buf, size)
+    if st == 0:
+        return
+    if st == _CUDA_ERROR_NO_BINARY_FOR_GPU and not check_cc:
+        return
+    raise RuntimeError((<bytes> cccl_serialization_last_error()).decode("utf-8", "replace"))
+
+cdef extern from "cccl/c/reduce.h":
+    cdef CUresult cccl_device_reduce_load(
+        cccl_device_reduce_build_result_t*
+    ) nogil
+    cdef CUresult cccl_device_reduce_serialize(
+        const cccl_device_reduce_build_result_t*,
+        void**,
+        size_t*
+    ) nogil
+    cdef CUresult cccl_device_reduce_deserialize(
+        cccl_device_reduce_build_result_t*,
+        const void*,
+        size_t
+    ) nogil
+
+cdef extern from "cccl/c/scan.h":
+    cdef CUresult cccl_device_scan_load(
+        cccl_device_scan_build_result_t*
+    ) nogil
+    cdef CUresult cccl_device_scan_serialize(
+        const cccl_device_scan_build_result_t*,
+        void**,
+        size_t*
+    ) nogil
+    cdef CUresult cccl_device_scan_deserialize(
+        cccl_device_scan_build_result_t*,
+        const void*,
+        size_t
+    ) nogil
+
+cdef extern from "cccl/c/segmented_reduce.h":
+    cdef CUresult cccl_device_segmented_reduce_load(
+        cccl_device_segmented_reduce_build_result_t*
+    ) nogil
+    cdef CUresult cccl_device_segmented_reduce_serialize(
+        const cccl_device_segmented_reduce_build_result_t*,
+        void**,
+        size_t*
+    ) nogil
+    cdef CUresult cccl_device_segmented_reduce_deserialize(
+        cccl_device_segmented_reduce_build_result_t*,
+        const void*,
+        size_t
+    ) nogil
+
+cdef extern from "cccl/c/merge_sort.h":
+    cdef CUresult cccl_device_merge_sort_load(
+        cccl_device_merge_sort_build_result_t*
+    ) nogil
+    cdef CUresult cccl_device_merge_sort_serialize(
+        const cccl_device_merge_sort_build_result_t*,
+        void**,
+        size_t*
+    ) nogil
+    cdef CUresult cccl_device_merge_sort_deserialize(
+        cccl_device_merge_sort_build_result_t*,
+        const void*,
+        size_t
+    ) nogil
+
+cdef extern from "cccl/c/unique_by_key.h":
+    cdef CUresult cccl_device_unique_by_key_load(
+        cccl_device_unique_by_key_build_result_t*
+    ) nogil
+    cdef CUresult cccl_device_unique_by_key_serialize(
+        const cccl_device_unique_by_key_build_result_t*,
+        void**,
+        size_t*
+    ) nogil
+    cdef CUresult cccl_device_unique_by_key_deserialize(
+        cccl_device_unique_by_key_build_result_t*,
+        const void*,
+        size_t
+    ) nogil
+
+cdef extern from "cccl/c/radix_sort.h":
+    cdef CUresult cccl_device_radix_sort_load(
+        cccl_device_radix_sort_build_result_t*
+    ) nogil
+    cdef CUresult cccl_device_radix_sort_serialize(
+        const cccl_device_radix_sort_build_result_t*,
+        void**,
+        size_t*
+    ) nogil
+    cdef CUresult cccl_device_radix_sort_deserialize(
+        cccl_device_radix_sort_build_result_t*,
+        const void*,
+        size_t
+    ) nogil
+
+cdef extern from "cccl/c/transform.h":
+    cdef CUresult cccl_device_transform_load(
+        cccl_device_transform_build_result_t*
+    ) nogil
+    cdef CUresult cccl_device_transform_serialize(
+        const cccl_device_transform_build_result_t*,
+        void**,
+        size_t*
+    ) nogil
+    cdef CUresult cccl_device_transform_deserialize(
+        cccl_device_transform_build_result_t*,
+        const void*,
+        size_t
+    ) nogil
+
+cdef extern from "cccl/c/histogram.h":
+    cdef CUresult cccl_device_histogram_load(
+        cccl_device_histogram_build_result_t*
+    ) nogil
+    cdef CUresult cccl_device_histogram_serialize(
+        const cccl_device_histogram_build_result_t*,
+        void**,
+        size_t*
+    ) nogil
+    cdef CUresult cccl_device_histogram_deserialize(
+        cccl_device_histogram_build_result_t*,
+        const void*,
+        size_t
+    ) nogil
+
+cdef extern from "cccl/c/binary_search.h":
+    cdef CUresult cccl_device_binary_search_load(
+        cccl_device_binary_search_build_result_t*
+    ) nogil
+    cdef CUresult cccl_device_binary_search_serialize(
+        const cccl_device_binary_search_build_result_t*,
+        void**,
+        size_t*
+    ) nogil
+    cdef CUresult cccl_device_binary_search_deserialize(
+        cccl_device_binary_search_build_result_t*,
+        const void*,
+        size_t
+    ) nogil
+
+cdef extern from "cccl/c/three_way_partition.h":
+    cdef CUresult cccl_device_three_way_partition_load(
+        cccl_device_three_way_partition_build_result_t*
+    ) nogil
+    cdef CUresult cccl_device_three_way_partition_serialize(
+        const cccl_device_three_way_partition_build_result_t*,
+        void**,
+        size_t*
+    ) nogil
+    cdef CUresult cccl_device_three_way_partition_deserialize(
+        cccl_device_three_way_partition_build_result_t*,
+        const void*,
+        size_t
+    ) nogil
+
+cdef extern from "cccl/c/segmented_sort.h":
+    cdef CUresult cccl_device_segmented_sort_load(
+        cccl_device_segmented_sort_build_result_t*
+    ) nogil
+    cdef CUresult cccl_device_segmented_sort_serialize(
+        const cccl_device_segmented_sort_build_result_t*,
+        void**,
+        size_t*
+    ) nogil
+    cdef CUresult cccl_device_segmented_sort_deserialize(
+        cccl_device_segmented_sort_build_result_t*,
+        const void*,
+        size_t
+    ) nogil
+
+
+# ---- ahead-of-time compile entry points (multi-cc / no-GPU) ----
+cdef extern from "cccl/c/reduce.h":
+    cdef CUresult cccl_device_reduce_compile(
+        cccl_device_reduce_build_result_t*,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_op_t,
+        cccl_value_t,
+        cccl_determinism_t,
+        int, int, const char*, const char*, const char*, const char*,
+        void*
+    ) nogil
+cdef extern from "cccl/c/scan.h":
+    cdef CUresult cccl_device_scan_compile(
+        cccl_device_scan_build_result_t*,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_op_t,
+        cccl_type_info,
+        bint,
+        cccl_init_kind_t,
+        int, int, const char*, const char*, const char*, const char*,
+        void*
+    ) nogil
+cdef extern from "cccl/c/segmented_reduce.h":
+    cdef CUresult cccl_device_segmented_reduce_compile(
+        cccl_device_segmented_reduce_build_result_t*,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_op_t,
+        cccl_value_t,
+        int, int, const char*, const char*, const char*, const char*,
+        void*
+    ) nogil
+cdef extern from "cccl/c/merge_sort.h":
+    cdef CUresult cccl_device_merge_sort_compile(
+        cccl_device_merge_sort_build_result_t*,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_op_t,
+        int, int, const char*, const char*, const char*, const char*,
+        void*
+    ) nogil
+cdef extern from "cccl/c/unique_by_key.h":
+    cdef CUresult cccl_device_unique_by_key_compile(
+        cccl_device_unique_by_key_build_result_t*,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_op_t,
+        int, int, const char*, const char*, const char*, const char*,
+        void*
+    ) nogil
+cdef extern from "cccl/c/radix_sort.h":
+    cdef CUresult cccl_device_radix_sort_compile(
+        cccl_device_radix_sort_build_result_t*,
+        cccl_sort_order_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_op_t,
+        const char*,
+        int, int, const char*, const char*, const char*, const char*,
+        void*
+    ) nogil
+cdef extern from "cccl/c/transform.h":
+    cdef CUresult cccl_device_unary_transform_compile(
+        cccl_device_transform_build_result_t*,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_op_t,
+        int, int, const char*, const char*, const char*, const char*,
+        void*
+    ) nogil
+cdef extern from "cccl/c/transform.h":
+    cdef CUresult cccl_device_binary_transform_compile(
+        cccl_device_transform_build_result_t*,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_op_t,
+        int, int, const char*, const char*, const char*, const char*,
+        void*
+    ) nogil
+cdef extern from "cccl/c/histogram.h":
+    cdef CUresult cccl_device_histogram_compile(
+        cccl_device_histogram_build_result_t*,
+        int,
+        int,
+        cccl_iterator_t,
+        int,
+        cccl_iterator_t,
+        cccl_type_info,
+        int64_t,
+        int64_t,
+        bint,
+        int, int, const char*, const char*, const char*, const char*,
+        void*
+    ) nogil
+cdef extern from "cccl/c/binary_search.h":
+    cdef CUresult cccl_device_binary_search_compile(
+        cccl_device_binary_search_build_result_t*,
+        cccl_binary_search_mode_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_op_t,
+        int, int, const char*, const char*, const char*, const char*,
+        void*
+    ) nogil
+cdef extern from "cccl/c/three_way_partition.h":
+    cdef CUresult cccl_device_three_way_partition_compile(
+        cccl_device_three_way_partition_build_result_t*,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_op_t,
+        cccl_op_t,
+        int, int, const char*, const char*, const char*, const char*,
+        void*
+    ) nogil
+cdef extern from "cccl/c/segmented_sort.h":
+    cdef CUresult cccl_device_segmented_sort_compile(
+        cccl_device_segmented_sort_build_result_t*,
+        cccl_sort_order_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        cccl_iterator_t,
+        int, int, const char*, const char*, const char*, const char*,
+        void*
+    ) nogil
+
+def _reduce_serialize(DeviceReduceBuildResult self):
+    cdef CUresult status = -1
+    cdef void* buf_ptr = NULL
+    cdef size_t buf_size = 0
+    with nogil:
+        status = cccl_device_reduce_serialize(&self.build_data, &buf_ptr, &buf_size)
+    if status != 0:
+        if buf_ptr != NULL:
+            cccl_serialization_buffer_free(buf_ptr)
+        raise RuntimeError(
+            f"Failed serializing reduce, error code: {status}"
+        )
+    try:
+        return PyBytes_FromStringAndSize(<const char*>buf_ptr, buf_size)
+    finally:
+        cccl_serialization_buffer_free(buf_ptr)
+
+def _reduce_deserialize(blob, load=True, check_cc=True):
+    # Copy to immutable bytes: the C deserialize reads the buffer under
+    # `nogil`, so a writable bytearray/memoryview could be mutated mid-read.
+    blob = bytes(blob)
+    cdef DeviceReduceBuildResult self = DeviceReduceBuildResult.__new__(
+        DeviceReduceBuildResult)
+    cdef CUresult status = -1
+    cdef const unsigned char[::1] view = blob
+    cdef size_t buf_size = view.shape[0]
+    if buf_size == 0:
+        raise RuntimeError("Cannot deserialize empty blob")
+    cdef const void* buf_ptr = <const void*>&view[0]
+    _check_loadable(buf_ptr, buf_size, check_cc)
+    with nogil:
+        status = cccl_device_reduce_deserialize(
+            &self.build_data, buf_ptr, buf_size)
+    if status != 0:
+        raise RuntimeError(f"Failed deserializing reduce, error code: {status}")
+    if load:
+        with nogil:
+            status = cccl_device_reduce_load(&self.build_data)
+        if status != 0:
+            raise RuntimeError(f"Failed loading reduce after deserialize, error code: {status}")
+        self._loaded = True
+    return self
+
+def _scan_serialize(DeviceScanBuildResult self):
+    cdef CUresult status = -1
+    cdef void* buf_ptr = NULL
+    cdef size_t buf_size = 0
+    with nogil:
+        status = cccl_device_scan_serialize(&self.build_data, &buf_ptr, &buf_size)
+    if status != 0:
+        if buf_ptr != NULL:
+            cccl_serialization_buffer_free(buf_ptr)
+        raise RuntimeError(f"Failed serializing scan, error code: {status}")
+    try:
+        return PyBytes_FromStringAndSize(<const char*>buf_ptr, buf_size)
+    finally:
+        cccl_serialization_buffer_free(buf_ptr)
+
+def _scan_deserialize(blob, load=True, check_cc=True):
+    # Copy to immutable bytes: the C deserialize reads the buffer under
+    # `nogil`, so a writable bytearray/memoryview could be mutated mid-read.
+    blob = bytes(blob)
+    cdef DeviceScanBuildResult self = DeviceScanBuildResult.__new__(
+        DeviceScanBuildResult)
+    cdef CUresult status = -1
+    cdef const unsigned char[::1] view = blob
+    cdef size_t buf_size = view.shape[0]
+    if buf_size == 0:
+        raise RuntimeError("Cannot deserialize empty blob")
+    cdef const void* buf_ptr = <const void*>&view[0]
+    _check_loadable(buf_ptr, buf_size, check_cc)
+    with nogil:
+        status = cccl_device_scan_deserialize(
+            &self.build_data, buf_ptr, buf_size)
+    if status != 0:
+        raise RuntimeError(f"Failed deserializing scan, error code: {status}")
+    if load:
+        with nogil:
+            status = cccl_device_scan_load(&self.build_data)
+        if status != 0:
+            raise RuntimeError(f"Failed loading scan after deserialize, error code: {status}")
+        self._loaded = True
+    return self
+
+def _segmented_reduce_serialize(DeviceSegmentedReduceBuildResult self):
+    cdef CUresult status = -1
+    cdef void* buf_ptr = NULL
+    cdef size_t buf_size = 0
+    with nogil:
+        status = cccl_device_segmented_reduce_serialize(&self.build_data, &buf_ptr, &buf_size)
+    if status != 0:
+        if buf_ptr != NULL:
+            cccl_serialization_buffer_free(buf_ptr)
+        raise RuntimeError(f"Failed serializing segmented_reduce, error code: {status}")
+    try:
+        return PyBytes_FromStringAndSize(<const char*>buf_ptr, buf_size)
+    finally:
+        cccl_serialization_buffer_free(buf_ptr)
+
+def _segmented_reduce_deserialize(blob, load=True, check_cc=True):
+    # Copy to immutable bytes: the C deserialize reads the buffer under
+    # `nogil`, so a writable bytearray/memoryview could be mutated mid-read.
+    blob = bytes(blob)
+    cdef DeviceSegmentedReduceBuildResult self = DeviceSegmentedReduceBuildResult.__new__(
+        DeviceSegmentedReduceBuildResult)
+    cdef CUresult status = -1
+    cdef const unsigned char[::1] view = blob
+    cdef size_t buf_size = view.shape[0]
+    if buf_size == 0:
+        raise RuntimeError("Cannot deserialize empty blob")
+    cdef const void* buf_ptr = <const void*>&view[0]
+    _check_loadable(buf_ptr, buf_size, check_cc)
+    with nogil:
+        status = cccl_device_segmented_reduce_deserialize(
+            &self.build_data, buf_ptr, buf_size)
+    if status != 0:
+        raise RuntimeError(f"Failed deserializing segmented_reduce, error code: {status}")
+    if load:
+        with nogil:
+            status = cccl_device_segmented_reduce_load(&self.build_data)
+        if status != 0:
+            raise RuntimeError(f"Failed loading segmented_reduce after deserialize, error code: {status}")
+        self._loaded = True
+    return self
+
+def _merge_sort_serialize(DeviceMergeSortBuildResult self):
+    cdef CUresult status = -1
+    cdef void* buf_ptr = NULL
+    cdef size_t buf_size = 0
+    with nogil:
+        status = cccl_device_merge_sort_serialize(&self.build_data, &buf_ptr, &buf_size)
+    if status != 0:
+        if buf_ptr != NULL:
+            cccl_serialization_buffer_free(buf_ptr)
+        raise RuntimeError(f"Failed serializing merge_sort, error code: {status}")
+    try:
+        return PyBytes_FromStringAndSize(<const char*>buf_ptr, buf_size)
+    finally:
+        cccl_serialization_buffer_free(buf_ptr)
+
+def _merge_sort_deserialize(blob, load=True, check_cc=True):
+    # Copy to immutable bytes: the C deserialize reads the buffer under
+    # `nogil`, so a writable bytearray/memoryview could be mutated mid-read.
+    blob = bytes(blob)
+    cdef DeviceMergeSortBuildResult self = DeviceMergeSortBuildResult.__new__(
+        DeviceMergeSortBuildResult)
+    cdef CUresult status = -1
+    cdef const unsigned char[::1] view = blob
+    cdef size_t buf_size = view.shape[0]
+    if buf_size == 0:
+        raise RuntimeError("Cannot deserialize empty blob")
+    cdef const void* buf_ptr = <const void*>&view[0]
+    _check_loadable(buf_ptr, buf_size, check_cc)
+    with nogil:
+        status = cccl_device_merge_sort_deserialize(
+            &self.build_data, buf_ptr, buf_size)
+    if status != 0:
+        raise RuntimeError(f"Failed deserializing merge_sort, error code: {status}")
+    if load:
+        with nogil:
+            status = cccl_device_merge_sort_load(&self.build_data)
+        if status != 0:
+            raise RuntimeError(f"Failed loading merge_sort after deserialize, error code: {status}")
+        self._loaded = True
+    return self
+
+def _unique_by_key_serialize(DeviceUniqueByKeyBuildResult self):
+    cdef CUresult status = -1
+    cdef void* buf_ptr = NULL
+    cdef size_t buf_size = 0
+    with nogil:
+        status = cccl_device_unique_by_key_serialize(&self.build_data, &buf_ptr, &buf_size)
+    if status != 0:
+        if buf_ptr != NULL:
+            cccl_serialization_buffer_free(buf_ptr)
+        raise RuntimeError(f"Failed serializing unique_by_key, error code: {status}")
+    try:
+        return PyBytes_FromStringAndSize(<const char*>buf_ptr, buf_size)
+    finally:
+        cccl_serialization_buffer_free(buf_ptr)
+
+def _unique_by_key_deserialize(blob, load=True, check_cc=True):
+    # Copy to immutable bytes: the C deserialize reads the buffer under
+    # `nogil`, so a writable bytearray/memoryview could be mutated mid-read.
+    blob = bytes(blob)
+    cdef DeviceUniqueByKeyBuildResult self = DeviceUniqueByKeyBuildResult.__new__(
+        DeviceUniqueByKeyBuildResult)
+    cdef CUresult status = -1
+    cdef const unsigned char[::1] view = blob
+    cdef size_t buf_size = view.shape[0]
+    if buf_size == 0:
+        raise RuntimeError("Cannot deserialize empty blob")
+    cdef const void* buf_ptr = <const void*>&view[0]
+    _check_loadable(buf_ptr, buf_size, check_cc)
+    with nogil:
+        status = cccl_device_unique_by_key_deserialize(
+            &self.build_data, buf_ptr, buf_size)
+    if status != 0:
+        raise RuntimeError(f"Failed deserializing unique_by_key, error code: {status}")
+    if load:
+        with nogil:
+            status = cccl_device_unique_by_key_load(&self.build_data)
+        if status != 0:
+            raise RuntimeError(f"Failed loading unique_by_key after deserialize, error code: {status}")
+        self._loaded = True
+    return self
+
+def _radix_sort_serialize(DeviceRadixSortBuildResult self):
+    cdef CUresult status = -1
+    cdef void* buf_ptr = NULL
+    cdef size_t buf_size = 0
+    with nogil:
+        status = cccl_device_radix_sort_serialize(&self.build_data, &buf_ptr, &buf_size)
+    if status != 0:
+        if buf_ptr != NULL:
+            cccl_serialization_buffer_free(buf_ptr)
+        raise RuntimeError(f"Failed serializing radix_sort, error code: {status}")
+    try:
+        return PyBytes_FromStringAndSize(<const char*>buf_ptr, buf_size)
+    finally:
+        cccl_serialization_buffer_free(buf_ptr)
+
+def _radix_sort_deserialize(blob, load=True, check_cc=True):
+    # Copy to immutable bytes: the C deserialize reads the buffer under
+    # `nogil`, so a writable bytearray/memoryview could be mutated mid-read.
+    blob = bytes(blob)
+    cdef DeviceRadixSortBuildResult self = DeviceRadixSortBuildResult.__new__(
+        DeviceRadixSortBuildResult)
+    cdef CUresult status = -1
+    cdef const unsigned char[::1] view = blob
+    cdef size_t buf_size = view.shape[0]
+    if buf_size == 0:
+        raise RuntimeError("Cannot deserialize empty blob")
+    cdef const void* buf_ptr = <const void*>&view[0]
+    _check_loadable(buf_ptr, buf_size, check_cc)
+    with nogil:
+        status = cccl_device_radix_sort_deserialize(
+            &self.build_data, buf_ptr, buf_size)
+    if status != 0:
+        raise RuntimeError(f"Failed deserializing radix_sort, error code: {status}")
+    if load:
+        with nogil:
+            status = cccl_device_radix_sort_load(&self.build_data)
+        if status != 0:
+            raise RuntimeError(f"Failed loading radix_sort after deserialize, error code: {status}")
+        self._loaded = True
+    return self
+
+def _unary_transform_serialize(DeviceUnaryTransform self):
+    cdef CUresult status = -1
+    cdef void* buf_ptr = NULL
+    cdef size_t buf_size = 0
+    with nogil:
+        status = cccl_device_transform_serialize(&self.build_data, &buf_ptr, &buf_size)
+    if status != 0:
+        if buf_ptr != NULL:
+            cccl_serialization_buffer_free(buf_ptr)
+        raise RuntimeError(f"Failed serializing unary_transform, error code: {status}")
+    try:
+        return PyBytes_FromStringAndSize(<const char*>buf_ptr, buf_size)
+    finally:
+        cccl_serialization_buffer_free(buf_ptr)
+
+def _unary_transform_deserialize(blob, load=True, check_cc=True):
+    # Copy to immutable bytes: the C deserialize reads the buffer under
+    # `nogil`, so a writable bytearray/memoryview could be mutated mid-read.
+    blob = bytes(blob)
+    cdef DeviceUnaryTransform self = DeviceUnaryTransform.__new__(
+        DeviceUnaryTransform)
+    cdef CUresult status = -1
+    cdef const unsigned char[::1] view = blob
+    cdef size_t buf_size = view.shape[0]
+    if buf_size == 0:
+        raise RuntimeError("Cannot deserialize empty blob")
+    cdef const void* buf_ptr = <const void*>&view[0]
+    _check_loadable(buf_ptr, buf_size, check_cc)
+    with nogil:
+        status = cccl_device_transform_deserialize(
+            &self.build_data, buf_ptr, buf_size)
+    if status != 0:
+        raise RuntimeError(f"Failed deserializing unary_transform, error code: {status}")
+    if load:
+        with nogil:
+            status = cccl_device_transform_load(&self.build_data)
+        if status != 0:
+            raise RuntimeError(f"Failed loading unary_transform after deserialize, error code: {status}")
+        self._loaded = True
+    return self
+
+def _binary_transform_serialize(DeviceBinaryTransform self):
+    cdef CUresult status = -1
+    cdef void* buf_ptr = NULL
+    cdef size_t buf_size = 0
+    with nogil:
+        status = cccl_device_transform_serialize(&self.build_data, &buf_ptr, &buf_size)
+    if status != 0:
+        if buf_ptr != NULL:
+            cccl_serialization_buffer_free(buf_ptr)
+        raise RuntimeError(f"Failed serializing binary_transform, error code: {status}")
+    try:
+        return PyBytes_FromStringAndSize(<const char*>buf_ptr, buf_size)
+    finally:
+        cccl_serialization_buffer_free(buf_ptr)
+
+def _binary_transform_deserialize(blob, load=True, check_cc=True):
+    # Copy to immutable bytes: the C deserialize reads the buffer under
+    # `nogil`, so a writable bytearray/memoryview could be mutated mid-read.
+    blob = bytes(blob)
+    cdef DeviceBinaryTransform self = DeviceBinaryTransform.__new__(
+        DeviceBinaryTransform)
+    cdef CUresult status = -1
+    cdef const unsigned char[::1] view = blob
+    cdef size_t buf_size = view.shape[0]
+    if buf_size == 0:
+        raise RuntimeError("Cannot deserialize empty blob")
+    cdef const void* buf_ptr = <const void*>&view[0]
+    _check_loadable(buf_ptr, buf_size, check_cc)
+    with nogil:
+        status = cccl_device_transform_deserialize(
+            &self.build_data, buf_ptr, buf_size)
+    if status != 0:
+        raise RuntimeError(f"Failed deserializing binary_transform, error code: {status}")
+    if load:
+        with nogil:
+            status = cccl_device_transform_load(&self.build_data)
+        if status != 0:
+            raise RuntimeError(f"Failed loading binary_transform after deserialize, error code: {status}")
+        self._loaded = True
+    return self
+
+def _histogram_serialize(DeviceHistogramBuildResult self):
+    cdef CUresult status = -1
+    cdef void* buf_ptr = NULL
+    cdef size_t buf_size = 0
+    with nogil:
+        status = cccl_device_histogram_serialize(&self.build_data, &buf_ptr, &buf_size)
+    if status != 0:
+        if buf_ptr != NULL:
+            cccl_serialization_buffer_free(buf_ptr)
+        raise RuntimeError(f"Failed serializing histogram, error code: {status}")
+    try:
+        return PyBytes_FromStringAndSize(<const char*>buf_ptr, buf_size)
+    finally:
+        cccl_serialization_buffer_free(buf_ptr)
+
+def _histogram_deserialize(blob, load=True, check_cc=True):
+    # Copy to immutable bytes: the C deserialize reads the buffer under
+    # `nogil`, so a writable bytearray/memoryview could be mutated mid-read.
+    blob = bytes(blob)
+    cdef DeviceHistogramBuildResult self = DeviceHistogramBuildResult.__new__(
+        DeviceHistogramBuildResult)
+    cdef CUresult status = -1
+    cdef const unsigned char[::1] view = blob
+    cdef size_t buf_size = view.shape[0]
+    if buf_size == 0:
+        raise RuntimeError("Cannot deserialize empty blob")
+    cdef const void* buf_ptr = <const void*>&view[0]
+    _check_loadable(buf_ptr, buf_size, check_cc)
+    with nogil:
+        status = cccl_device_histogram_deserialize(
+            &self.build_data, buf_ptr, buf_size)
+    if status != 0:
+        raise RuntimeError(f"Failed deserializing histogram, error code: {status}")
+    if load:
+        with nogil:
+            status = cccl_device_histogram_load(&self.build_data)
+        if status != 0:
+            raise RuntimeError(f"Failed loading histogram after deserialize, error code: {status}")
+        self._loaded = True
+    return self
+
+def _binary_search_serialize(DeviceBinarySearchBuildResult self):
+    cdef CUresult status = -1
+    cdef void* buf_ptr = NULL
+    cdef size_t buf_size = 0
+    with nogil:
+        status = cccl_device_binary_search_serialize(&self.build_data, &buf_ptr, &buf_size)
+    if status != 0:
+        if buf_ptr != NULL:
+            cccl_serialization_buffer_free(buf_ptr)
+        raise RuntimeError(f"Failed serializing binary_search, error code: {status}")
+    try:
+        return PyBytes_FromStringAndSize(<const char*>buf_ptr, buf_size)
+    finally:
+        cccl_serialization_buffer_free(buf_ptr)
+
+def _binary_search_deserialize(blob, load=True, check_cc=True):
+    # Copy to immutable bytes: the C deserialize reads the buffer under
+    # `nogil`, so a writable bytearray/memoryview could be mutated mid-read.
+    blob = bytes(blob)
+    cdef DeviceBinarySearchBuildResult self = DeviceBinarySearchBuildResult.__new__(
+        DeviceBinarySearchBuildResult)
+    cdef CUresult status = -1
+    cdef const unsigned char[::1] view = blob
+    cdef size_t buf_size = view.shape[0]
+    if buf_size == 0:
+        raise RuntimeError("Cannot deserialize empty blob")
+    cdef const void* buf_ptr = <const void*>&view[0]
+    _check_loadable(buf_ptr, buf_size, check_cc)
+    with nogil:
+        status = cccl_device_binary_search_deserialize(
+            &self.build_data, buf_ptr, buf_size)
+    if status != 0:
+        raise RuntimeError(f"Failed deserializing binary_search, error code: {status}")
+    if load:
+        with nogil:
+            status = cccl_device_binary_search_load(&self.build_data)
+        if status != 0:
+            raise RuntimeError(f"Failed loading binary_search after deserialize, error code: {status}")
+        self._loaded = True
+    return self
+
+def _three_way_partition_serialize(DeviceThreeWayPartitionBuildResult self):
+    cdef CUresult status = -1
+    cdef void* buf_ptr = NULL
+    cdef size_t buf_size = 0
+    with nogil:
+        status = cccl_device_three_way_partition_serialize(&self.build_data, &buf_ptr, &buf_size)
+    if status != 0:
+        if buf_ptr != NULL:
+            cccl_serialization_buffer_free(buf_ptr)
+        raise RuntimeError(f"Failed serializing three_way_partition, error code: {status}")
+    try:
+        return PyBytes_FromStringAndSize(<const char*>buf_ptr, buf_size)
+    finally:
+        cccl_serialization_buffer_free(buf_ptr)
+
+def _three_way_partition_deserialize(blob, load=True, check_cc=True):
+    # Copy to immutable bytes: the C deserialize reads the buffer under
+    # `nogil`, so a writable bytearray/memoryview could be mutated mid-read.
+    blob = bytes(blob)
+    cdef DeviceThreeWayPartitionBuildResult self = DeviceThreeWayPartitionBuildResult.__new__(
+        DeviceThreeWayPartitionBuildResult)
+    cdef CUresult status = -1
+    cdef const unsigned char[::1] view = blob
+    cdef size_t buf_size = view.shape[0]
+    if buf_size == 0:
+        raise RuntimeError("Cannot deserialize empty blob")
+    cdef const void* buf_ptr = <const void*>&view[0]
+    _check_loadable(buf_ptr, buf_size, check_cc)
+    with nogil:
+        status = cccl_device_three_way_partition_deserialize(
+            &self.build_data, buf_ptr, buf_size)
+    if status != 0:
+        raise RuntimeError(f"Failed deserializing three_way_partition, error code: {status}")
+    if load:
+        with nogil:
+            status = cccl_device_three_way_partition_load(&self.build_data)
+        if status != 0:
+            raise RuntimeError(f"Failed loading three_way_partition after deserialize, error code: {status}")
+        self._loaded = True
+    return self
+
+def _segmented_sort_serialize(DeviceSegmentedSortBuildResult self):
+    cdef CUresult status = -1
+    cdef void* buf_ptr = NULL
+    cdef size_t buf_size = 0
+    with nogil:
+        status = cccl_device_segmented_sort_serialize(&self.build_data, &buf_ptr, &buf_size)
+    if status != 0:
+        if buf_ptr != NULL:
+            cccl_serialization_buffer_free(buf_ptr)
+        raise RuntimeError(f"Failed serializing segmented_sort, error code: {status}")
+    try:
+        return PyBytes_FromStringAndSize(<const char*>buf_ptr, buf_size)
+    finally:
+        cccl_serialization_buffer_free(buf_ptr)
+
+def _segmented_sort_deserialize(blob, load=True, check_cc=True):
+    # Copy to immutable bytes: the C deserialize reads the buffer under
+    # `nogil`, so a writable bytearray/memoryview could be mutated mid-read.
+    blob = bytes(blob)
+    cdef DeviceSegmentedSortBuildResult self = DeviceSegmentedSortBuildResult.__new__(
+        DeviceSegmentedSortBuildResult)
+    cdef CUresult status = -1
+    cdef const unsigned char[::1] view = blob
+    cdef size_t buf_size = view.shape[0]
+    if buf_size == 0:
+        raise RuntimeError("Cannot deserialize empty blob")
+    cdef const void* buf_ptr = <const void*>&view[0]
+    _check_loadable(buf_ptr, buf_size, check_cc)
+    with nogil:
+        status = cccl_device_segmented_sort_deserialize(
+            &self.build_data, buf_ptr, buf_size)
+    if status != 0:
+        raise RuntimeError(f"Failed deserializing segmented_sort, error code: {status}")
+    if load:
+        with nogil:
+            status = cccl_device_segmented_sort_load(&self.build_data)
+        if status != 0:
+            raise RuntimeError(f"Failed loading segmented_sort after deserialize, error code: {status}")
+        self._loaded = True
+    return self
+
+
+# ---- compile()/load() implementations (multi-cc / no-GPU) ----
+
+def _reduce_compile(
+    Iterator d_in,
+    Iterator d_out,
+    Op op,
+    Value h_init,
+    cccl_determinism_t determinism,
+    CommonData common_data,
+):
+    cdef DeviceReduceBuildResult self = DeviceReduceBuildResult.__new__(DeviceReduceBuildResult)
+    cdef CUresult status = -1
+    cdef int cc_major = common_data.get_cc_major()
+    cdef int cc_minor = common_data.get_cc_minor()
+    cdef const char *cub_path = common_data.cub_path_get_c_str()
+    cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+    cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+    cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+    with nogil:
+        status = cccl_device_reduce_compile(
+            &self.build_data,
+            d_in.iter_data,
+            d_out.iter_data,
+            op.op_data,
+            h_init.value_data,
+            determinism,
+            cc_major,
+            cc_minor,
+            cub_path,
+            thrust_path,
+            libcudacxx_path,
+            ctk_path,
+            NULL,
+        )
+    if status != 0:
+        raise RuntimeError(f"Failed compiling reduce, error code: {status}")
+    return self
+
+def _reduce_load(DeviceReduceBuildResult self):
+    cdef CUresult status = -1
+    with nogil:
+        status = cccl_device_reduce_load(&self.build_data)
+    if status != 0:
+        raise RuntimeError(f"Failed loading reduce, error code: {status}")
+
+def _scan_compile(
+    Iterator d_in,
+    Iterator d_out,
+    Op op,
+    TypeInfo init_type,
+    bint force_inclusive,
+    cccl_init_kind_t init_kind,
+    CommonData common_data,
+):
+    cdef DeviceScanBuildResult self = DeviceScanBuildResult.__new__(DeviceScanBuildResult)
+    cdef CUresult status = -1
+    cdef int cc_major = common_data.get_cc_major()
+    cdef int cc_minor = common_data.get_cc_minor()
+    cdef const char *cub_path = common_data.cub_path_get_c_str()
+    cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+    cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+    cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+    with nogil:
+        status = cccl_device_scan_compile(
+            &self.build_data,
+            d_in.iter_data,
+            d_out.iter_data,
+            op.op_data,
+            init_type.type_info,
+            force_inclusive,
+            init_kind,
+            cc_major,
+            cc_minor,
+            cub_path,
+            thrust_path,
+            libcudacxx_path,
+            ctk_path,
+            NULL,
+        )
+    if status != 0:
+        raise RuntimeError(f"Failed compiling scan, error code: {status}")
+    return self
+
+def _scan_load(DeviceScanBuildResult self):
+    cdef CUresult status = -1
+    with nogil:
+        status = cccl_device_scan_load(&self.build_data)
+    if status != 0:
+        raise RuntimeError(f"Failed loading scan, error code: {status}")
+
+def _segmented_reduce_compile(
+    Iterator d_in,
+    Iterator d_out,
+    Iterator start_offsets,
+    Iterator end_offsets,
+    Op op,
+    Value h_init,
+    CommonData common_data,
+):
+    cdef DeviceSegmentedReduceBuildResult self = DeviceSegmentedReduceBuildResult.__new__(DeviceSegmentedReduceBuildResult)
+    cdef CUresult status = -1
+    cdef int cc_major = common_data.get_cc_major()
+    cdef int cc_minor = common_data.get_cc_minor()
+    cdef const char *cub_path = common_data.cub_path_get_c_str()
+    cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+    cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+    cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+    with nogil:
+        status = cccl_device_segmented_reduce_compile(
+            &self.build_data,
+            d_in.iter_data,
+            d_out.iter_data,
+            start_offsets.iter_data,
+            end_offsets.iter_data,
+            op.op_data,
+            h_init.value_data,
+            cc_major,
+            cc_minor,
+            cub_path,
+            thrust_path,
+            libcudacxx_path,
+            ctk_path,
+            NULL,
+        )
+    if status != 0:
+        raise RuntimeError(f"Failed compiling segmented_reduce, error code: {status}")
+    return self
+
+def _segmented_reduce_load(DeviceSegmentedReduceBuildResult self):
+    cdef CUresult status = -1
+    with nogil:
+        status = cccl_device_segmented_reduce_load(&self.build_data)
+    if status != 0:
+        raise RuntimeError(f"Failed loading segmented_reduce, error code: {status}")
+
+def _merge_sort_compile(
+    Iterator d_in_keys,
+    Iterator d_in_items,
+    Iterator d_out_keys,
+    Iterator d_out_items,
+    Op op,
+    CommonData common_data,
+):
+    cdef DeviceMergeSortBuildResult self = DeviceMergeSortBuildResult.__new__(DeviceMergeSortBuildResult)
+    cdef CUresult status = -1
+    cdef int cc_major = common_data.get_cc_major()
+    cdef int cc_minor = common_data.get_cc_minor()
+    cdef const char *cub_path = common_data.cub_path_get_c_str()
+    cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+    cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+    cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+    with nogil:
+        status = cccl_device_merge_sort_compile(
+            &self.build_data,
+            d_in_keys.iter_data,
+            d_in_items.iter_data,
+            d_out_keys.iter_data,
+            d_out_items.iter_data,
+            op.op_data,
+            cc_major,
+            cc_minor,
+            cub_path,
+            thrust_path,
+            libcudacxx_path,
+            ctk_path,
+            NULL,
+        )
+    if status != 0:
+        raise RuntimeError(f"Failed compiling merge_sort, error code: {status}")
+    return self
+
+def _merge_sort_load(DeviceMergeSortBuildResult self):
+    cdef CUresult status = -1
+    with nogil:
+        status = cccl_device_merge_sort_load(&self.build_data)
+    if status != 0:
+        raise RuntimeError(f"Failed loading merge_sort, error code: {status}")
+
+def _unique_by_key_compile(
+    Iterator d_keys_in,
+    Iterator d_values_in,
+    Iterator d_keys_out,
+    Iterator d_values_out,
+    Iterator d_num_selected_out,
+    Op comparison_op,
+    CommonData common_data,
+):
+    cdef DeviceUniqueByKeyBuildResult self = DeviceUniqueByKeyBuildResult.__new__(DeviceUniqueByKeyBuildResult)
+    cdef CUresult status = -1
+    cdef int cc_major = common_data.get_cc_major()
+    cdef int cc_minor = common_data.get_cc_minor()
+    cdef const char *cub_path = common_data.cub_path_get_c_str()
+    cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+    cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+    cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+    with nogil:
+        status = cccl_device_unique_by_key_compile(
+            &self.build_data,
+            d_keys_in.iter_data,
+            d_values_in.iter_data,
+            d_keys_out.iter_data,
+            d_values_out.iter_data,
+            d_num_selected_out.iter_data,
+            comparison_op.op_data,
+            cc_major,
+            cc_minor,
+            cub_path,
+            thrust_path,
+            libcudacxx_path,
+            ctk_path,
+            NULL,
+        )
+    if status != 0:
+        raise RuntimeError(f"Failed compiling unique_by_key, error code: {status}")
+    return self
+
+def _unique_by_key_load(DeviceUniqueByKeyBuildResult self):
+    cdef CUresult status = -1
+    with nogil:
+        status = cccl_device_unique_by_key_load(&self.build_data)
+    if status != 0:
+        raise RuntimeError(f"Failed loading unique_by_key, error code: {status}")
+
+def _radix_sort_compile(
+    cccl_sort_order_t order,
+    Iterator d_keys_in,
+    Iterator d_values_in,
+    Op decomposer_op,
+    const char* decomposer_return_type,
+    CommonData common_data,
+):
+    cdef DeviceRadixSortBuildResult self = DeviceRadixSortBuildResult.__new__(DeviceRadixSortBuildResult)
+    cdef CUresult status = -1
+    cdef int cc_major = common_data.get_cc_major()
+    cdef int cc_minor = common_data.get_cc_minor()
+    cdef const char *cub_path = common_data.cub_path_get_c_str()
+    cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+    cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+    cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+    with nogil:
+        status = cccl_device_radix_sort_compile(
+            &self.build_data,
+            order,
+            d_keys_in.iter_data,
+            d_values_in.iter_data,
+            decomposer_op.op_data,
+            decomposer_return_type,
+            cc_major,
+            cc_minor,
+            cub_path,
+            thrust_path,
+            libcudacxx_path,
+            ctk_path,
+            NULL,
+        )
+    if status != 0:
+        raise RuntimeError(f"Failed compiling radix_sort, error code: {status}")
+    return self
+
+def _radix_sort_load(DeviceRadixSortBuildResult self):
+    cdef CUresult status = -1
+    with nogil:
+        status = cccl_device_radix_sort_load(&self.build_data)
+    if status != 0:
+        raise RuntimeError(f"Failed loading radix_sort, error code: {status}")
+
+def _unary_transform_compile(
+    Iterator d_in,
+    Iterator d_out,
+    Op op,
+    CommonData common_data,
+):
+    cdef DeviceUnaryTransform self = DeviceUnaryTransform.__new__(DeviceUnaryTransform)
+    cdef CUresult status = -1
+    cdef int cc_major = common_data.get_cc_major()
+    cdef int cc_minor = common_data.get_cc_minor()
+    cdef const char *cub_path = common_data.cub_path_get_c_str()
+    cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+    cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+    cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+    with nogil:
+        status = cccl_device_unary_transform_compile(
+            &self.build_data,
+            d_in.iter_data,
+            d_out.iter_data,
+            op.op_data,
+            cc_major,
+            cc_minor,
+            cub_path,
+            thrust_path,
+            libcudacxx_path,
+            ctk_path,
+            NULL,
+        )
+    if status != 0:
+        raise RuntimeError(f"Failed compiling unary_transform, error code: {status}")
+    return self
+
+def _unary_transform_load(DeviceUnaryTransform self):
+    cdef CUresult status = -1
+    with nogil:
+        status = cccl_device_transform_load(&self.build_data)
+    if status != 0:
+        raise RuntimeError(f"Failed loading unary_transform, error code: {status}")
+
+def _binary_transform_compile(
+    Iterator d_in1,
+    Iterator d_in2,
+    Iterator d_out,
+    Op op,
+    CommonData common_data,
+):
+    cdef DeviceBinaryTransform self = DeviceBinaryTransform.__new__(DeviceBinaryTransform)
+    cdef CUresult status = -1
+    cdef int cc_major = common_data.get_cc_major()
+    cdef int cc_minor = common_data.get_cc_minor()
+    cdef const char *cub_path = common_data.cub_path_get_c_str()
+    cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+    cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+    cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+    with nogil:
+        status = cccl_device_binary_transform_compile(
+            &self.build_data,
+            d_in1.iter_data,
+            d_in2.iter_data,
+            d_out.iter_data,
+            op.op_data,
+            cc_major,
+            cc_minor,
+            cub_path,
+            thrust_path,
+            libcudacxx_path,
+            ctk_path,
+            NULL,
+        )
+    if status != 0:
+        raise RuntimeError(f"Failed compiling binary_transform, error code: {status}")
+    return self
+
+def _binary_transform_load(DeviceBinaryTransform self):
+    cdef CUresult status = -1
+    with nogil:
+        status = cccl_device_transform_load(&self.build_data)
+    if status != 0:
+        raise RuntimeError(f"Failed loading binary_transform, error code: {status}")
+
+def _histogram_compile(
+    int num_channels,
+    int num_active_channels,
+    Iterator d_samples,
+    int num_levels,
+    Iterator d_histogram,
+    TypeInfo level_type,
+    int num_rows,
+    int row_stride_samples,
+    bint is_evenly_segmented,
+    CommonData common_data,
+):
+    cdef DeviceHistogramBuildResult self = DeviceHistogramBuildResult.__new__(DeviceHistogramBuildResult)
+    cdef CUresult status = -1
+    cdef int cc_major = common_data.get_cc_major()
+    cdef int cc_minor = common_data.get_cc_minor()
+    cdef const char *cub_path = common_data.cub_path_get_c_str()
+    cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+    cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+    cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+    with nogil:
+        status = cccl_device_histogram_compile(
+            &self.build_data,
+            num_channels,
+            num_active_channels,
+            d_samples.iter_data,
+            num_levels,
+            d_histogram.iter_data,
+            level_type.type_info,
+            num_rows,
+            row_stride_samples,
+            is_evenly_segmented,
+            cc_major,
+            cc_minor,
+            cub_path,
+            thrust_path,
+            libcudacxx_path,
+            ctk_path,
+            NULL,
+        )
+    if status != 0:
+        raise RuntimeError(f"Failed compiling histogram, error code: {status}")
+    return self
+
+def _histogram_load(DeviceHistogramBuildResult self):
+    cdef CUresult status = -1
+    with nogil:
+        status = cccl_device_histogram_load(&self.build_data)
+    if status != 0:
+        raise RuntimeError(f"Failed loading histogram, error code: {status}")
+
+def _binary_search_compile(
+    cccl_binary_search_mode_t mode,
+    Iterator d_data,
+    Iterator d_values,
+    Iterator d_out,
+    Op op,
+    CommonData common_data,
+):
+    cdef DeviceBinarySearchBuildResult self = DeviceBinarySearchBuildResult.__new__(DeviceBinarySearchBuildResult)
+    cdef CUresult status = -1
+    cdef int cc_major = common_data.get_cc_major()
+    cdef int cc_minor = common_data.get_cc_minor()
+    cdef const char *cub_path = common_data.cub_path_get_c_str()
+    cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+    cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+    cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+    with nogil:
+        status = cccl_device_binary_search_compile(
+            &self.build_data,
+            mode,
+            d_data.iter_data,
+            d_values.iter_data,
+            d_out.iter_data,
+            op.op_data,
+            cc_major,
+            cc_minor,
+            cub_path,
+            thrust_path,
+            libcudacxx_path,
+            ctk_path,
+            NULL,
+        )
+    if status != 0:
+        raise RuntimeError(f"Failed compiling binary_search, error code: {status}")
+    return self
+
+def _binary_search_load(DeviceBinarySearchBuildResult self):
+    cdef CUresult status = -1
+    with nogil:
+        status = cccl_device_binary_search_load(&self.build_data)
+    if status != 0:
+        raise RuntimeError(f"Failed loading binary_search, error code: {status}")
+
+def _three_way_partition_compile(
+    Iterator d_in,
+    Iterator d_first_part_out,
+    Iterator d_second_part_out,
+    Iterator d_unselected_out,
+    Iterator d_num_selected_out,
+    Op select_first_part_op,
+    Op select_second_part_op,
+    CommonData common_data,
+):
+    cdef DeviceThreeWayPartitionBuildResult self = DeviceThreeWayPartitionBuildResult.__new__(DeviceThreeWayPartitionBuildResult)
+    cdef CUresult status = -1
+    cdef int cc_major = common_data.get_cc_major()
+    cdef int cc_minor = common_data.get_cc_minor()
+    cdef const char *cub_path = common_data.cub_path_get_c_str()
+    cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+    cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+    cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+    with nogil:
+        status = cccl_device_three_way_partition_compile(
+            &self.build_data,
+            d_in.iter_data,
+            d_first_part_out.iter_data,
+            d_second_part_out.iter_data,
+            d_unselected_out.iter_data,
+            d_num_selected_out.iter_data,
+            select_first_part_op.op_data,
+            select_second_part_op.op_data,
+            cc_major,
+            cc_minor,
+            cub_path,
+            thrust_path,
+            libcudacxx_path,
+            ctk_path,
+            NULL,
+        )
+    if status != 0:
+        raise RuntimeError(f"Failed compiling three_way_partition, error code: {status}")
+    return self
+
+def _three_way_partition_load(DeviceThreeWayPartitionBuildResult self):
+    cdef CUresult status = -1
+    with nogil:
+        status = cccl_device_three_way_partition_load(&self.build_data)
+    if status != 0:
+        raise RuntimeError(f"Failed loading three_way_partition, error code: {status}")
+
+def _segmented_sort_compile(
+    cccl_sort_order_t order,
+    Iterator d_keys_in,
+    Iterator d_values_in,
+    Iterator begin_offset_in,
+    Iterator end_offset_in,
+    CommonData common_data,
+):
+    cdef DeviceSegmentedSortBuildResult self = DeviceSegmentedSortBuildResult.__new__(DeviceSegmentedSortBuildResult)
+    cdef CUresult status = -1
+    cdef int cc_major = common_data.get_cc_major()
+    cdef int cc_minor = common_data.get_cc_minor()
+    cdef const char *cub_path = common_data.cub_path_get_c_str()
+    cdef const char *thrust_path = common_data.thrust_path_get_c_str()
+    cdef const char *libcudacxx_path = common_data.libcudacxx_path_get_c_str()
+    cdef const char *ctk_path = common_data.ctk_path_get_c_str()
+    with nogil:
+        status = cccl_device_segmented_sort_compile(
+            &self.build_data,
+            order,
+            d_keys_in.iter_data,
+            d_values_in.iter_data,
+            begin_offset_in.iter_data,
+            end_offset_in.iter_data,
+            cc_major,
+            cc_minor,
+            cub_path,
+            thrust_path,
+            libcudacxx_path,
+            ctk_path,
+            NULL,
+        )
+    if status != 0:
+        raise RuntimeError(f"Failed compiling segmented_sort, error code: {status}")
+    return self
+
+def _segmented_sort_load(DeviceSegmentedSortBuildResult self):
+    cdef CUresult status = -1
+    with nogil:
+        status = cccl_device_segmented_sort_load(&self.build_data)
+    if status != 0:
+        raise RuntimeError(f"Failed loading segmented_sort, error code: {status}")
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_serialization_v2.pxi b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_serialization_v2.pxi
new file mode 100644
index 00000000..4787dbd2
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_bindings_serialization_v2.pxi
@@ -0,0 +1,57 @@
+# v2 (HostJIT) backend — serialize/deserialize/compile/load not supported.
+# Included at the end of _bindings_impl.pyx; provides stub functions so that
+# the class methods exist but raise a clear error if called.
+
+_NOT_SUPPORTED_MSG = (
+    "serialize/deserialize (and ahead-of-time compile/load) is not supported "
+    "with the HostJIT (v2) backend."
+)
+
+def _reduce_serialize(self):                raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _reduce_deserialize(blob, load=True, check_cc=True):   raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _reduce_compile(*args):                 raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _reduce_load(self):                     raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _scan_serialize(self):                  raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _scan_deserialize(blob, load=True, check_cc=True):     raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _scan_compile(*args):                   raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _scan_load(self):                       raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _segmented_reduce_serialize(self):      raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _segmented_reduce_deserialize(blob, load=True, check_cc=True): raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _segmented_reduce_compile(*args):       raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _segmented_reduce_load(self):           raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _merge_sort_serialize(self):            raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _merge_sort_deserialize(blob, load=True, check_cc=True): raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _merge_sort_compile(*args):             raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _merge_sort_load(self):                 raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _unique_by_key_serialize(self):         raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _unique_by_key_deserialize(blob, load=True, check_cc=True): raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _unique_by_key_compile(*args):          raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _unique_by_key_load(self):              raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _radix_sort_serialize(self):            raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _radix_sort_deserialize(blob, load=True, check_cc=True): raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _radix_sort_compile(*args):             raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _radix_sort_load(self):                 raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _unary_transform_serialize(self):       raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _unary_transform_deserialize(blob, load=True, check_cc=True): raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _unary_transform_compile(*args):        raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _unary_transform_load(self):            raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _binary_transform_serialize(self):      raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _binary_transform_deserialize(blob, load=True, check_cc=True): raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _binary_transform_compile(*args):       raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _binary_transform_load(self):           raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _histogram_serialize(self):             raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _histogram_deserialize(blob, load=True, check_cc=True): raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _histogram_compile(*args):              raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _histogram_load(self):                  raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _binary_search_serialize(self):         raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _binary_search_deserialize(blob, load=True, check_cc=True): raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _binary_search_compile(*args):          raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _binary_search_load(self):              raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _three_way_partition_serialize(self):   raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _three_way_partition_deserialize(blob, load=True, check_cc=True): raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _three_way_partition_compile(*args):    raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _three_way_partition_load(self):        raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _segmented_sort_serialize(self):        raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _segmented_sort_deserialize(blob, load=True, check_cc=True): raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _segmented_sort_compile(*args):         raise NotImplementedError(_NOT_SUPPORTED_MSG)
+def _segmented_sort_load(self):             raise NotImplementedError(_NOT_SUPPORTED_MSG)
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_caching.py b/cccl_upstream/python/cuda_cccl/cuda/compute/_caching.py
new file mode 100644
index 00000000..052f8a27
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_caching.py
@@ -0,0 +1,740 @@
+# Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+import functools
+import threading
+import types
+import weakref
+from typing import Any, Callable, Hashable, NamedTuple, TypeVar
+
+import numpy as np
+
+from cuda.core import Device
+
+try:
+    from cuda.core._utils.cuda_utils import CUDAError
+except ImportError:
+    from cuda.core.experimental._utils.cuda_utils import CUDAError
+
+from ._utils.protocols import get_dtype, get_shape, is_device_array
+from .struct import _Struct
+
+try:
+    from ._build_info import USING_V2  # type: ignore[import-not-found]
+except ImportError:
+    USING_V2 = False
+
+# Whether the backend can serialize build results (the v2 HostJIT backend
+# cannot, today). Serialization is what lets same-cc devices share one default
+# build — a non-owner device loads by cloning the shared entry through
+# serialize -> deserialize -> load — so backends without it key default builds
+# per device ordinal and build independently per device instead.
+# TODO: delete this flag (and every branch on it) once v2 supports
+# build-result serialization.
+_BACKEND_SERIALIZES_BUILD_RESULTS = not USING_V2
+
+# Registry thet maps type -> key function for extracting cache key
+# from a value of that type.
+_KEY_FUNCTIONS: dict[type, Callable[[Any], Hashable]] = {}
+
+
+def _type_fqn(v):
+    # fully-qualified type name to distinguish np.ndarray from cp.ndarray from GpuStruct
+    return f"{type(v).__module__}.{type(v).__name__}"
+
+
+def _key_for(value: Any) -> Hashable:
+    """
+    Extract a cache key from a value using the registered KEY_FUNCTIONS.
+
+    This function checks the type of the value and delegates to the
+    appropriate registered keyer. Falls back to using the value
+    directly if no keyer is registered.
+
+    Args:
+        value: The value to extract a cache key from
+
+    Returns:
+        A hashable cache key
+    """
+    # Handle sequences (lists, tuples) by recursively converting to tuple
+    if isinstance(value, (list, tuple)):
+        return tuple(_key_for(item) for item in value)
+
+    # Check for exact type match first
+    value_type = type(value)
+    if value_type in _KEY_FUNCTIONS:
+        return _KEY_FUNCTIONS[value_type](value)
+
+    # DeviceArrayLike is not a runtime-checkable protocol, so
+    # we cannot isinstance() with it.
+    if is_device_array(value):
+        return (_type_fqn(value), get_dtype(value))
+
+    # Check for instance match (handles inheritance)
+    for registered_type, keyer in _KEY_FUNCTIONS.items():
+        if isinstance(value, registered_type):
+            return keyer(value)
+
+    # Fallback: use value directly (assumes it's hashable)
+    return value
+
+
+# The specialization part of every cache key: what _make_cache_key_from_args
+# extracts from the user-facing arguments (dtypes, op identity, iterator
+# kinds, ...).
+_SpecializationKey = tuple[Hashable, ...]
+
+
+def _make_cache_key_from_args(*args, **kwargs) -> _SpecializationKey:
+    """
+    Create a cache key from function arguments.
+
+    Args:
+        *args: Positional arguments
+        **kwargs: Keyword arguments
+
+    Returns:
+        A tuple containing the extracted cache keys
+    """
+
+    positional_keys = tuple(_key_for(arg) for arg in args)
+
+    # Sort kwargs by key name for consistent ordering
+    if kwargs:
+        sorted_kwargs = sorted(kwargs.items())
+        kwarg_keys = tuple((k, _key_for(v)) for k, v in sorted_kwargs)
+        return positional_keys + (kwarg_keys,)
+
+    return positional_keys
+
+
+# Process-wide registry of all algorithm caches.
+_process_wide_cache_registry: dict[str, object] = {}
+
+
+class _ThreadLocalCaches:
+    """
+    Container for wrapper caches owned by a single Python thread.
+
+    Each thread gets its own instance via ``threading.local()``. We use
+    ``__weakref__`` to enable the process-wide registry of caches to hold weak
+    references to the thread's caches. That way, if a thread exits, its caches
+    will be garbage collected and removed from the registry even if the
+    process-wide registry still references them.
+    """
+
+    __slots__ = ("wrapper_caches", "__weakref__")
+
+    def __init__(self) -> None:
+        # Outer key: decorated algorithm factory name, e.g.,
+        # make_reduce_into.__qualname__. Inner key: _WrapperCacheKey. No
+        # thread id is needed in either key because each thread holds a
+        # separate thread local object of this class.
+        self.wrapper_caches: dict[str, dict[_WrapperCacheKey, Any]] = {}
+
+
+class _InFlightBuild:
+    """
+    Coordination state for one shared build-result currently being built.
+
+    The first thread for a cache key runs the builder. Other threads wait on
+    ``event`` and receive either the completed build result or the builder's
+    exception.
+    """
+
+    def __init__(self) -> None:
+        self.event = threading.Event()
+        self.result: Any = None
+        self.exception: BaseException | None = None
+
+
+# A compute capability packed into one int as major * 10 + minor, e.g.
+# (9, 0) -> 90 and (12, 0) -> 120; cc_to_key / key_to_cc in _cccl_interop
+# convert to and from the (major, minor) pair form. Purely documentary:
+# mypy treats it as int.
+_PackedCCKey = int
+
+
+class _DeviceBuildTarget(NamedTuple):
+    """
+    Target identity of a per-device wrapper or build entry.
+
+    Two roles. In the wrapper cache it keys every default-build wrapper:
+    wrappers hold device-bound state (their construction-time binding), so
+    each device gets its own. In the build-results cache it is used only when
+    the backend cannot serialize build results (the v2 HostJIT backend
+    today): sharing one entry across same-cc devices requires cloning it
+    through serialization, so such backends key default builds per device
+    ordinal instead.
+    TODO: once v2 supports build-result serialization, delete the build-cache
+    role (the _BACKEND_SERIALIZES_BUILD_RESULTS branch in
+    cache_build_results); the wrapper-cache role remains.
+
+    NamedTuples compare as plain tuples, so all target kinds must keep
+    structurally disjoint layouts (arity or element types) to never compare
+    equal to each other.
+    """
+
+    device_id: int
+    cc: tuple[int, int]
+
+
+class _DefaultBuildTarget(NamedTuple):
+    """
+    Target identity of a default build shared across same-cc devices.
+
+    Holds the packed cc alone: the compiled payload depends only on the cc,
+    and _PerCCBuildResults.resolve() gives each device its own loaded state.
+    See _DeviceBuildTarget for the cross-kind equality constraint.
+    """
+
+    cc_key: _PackedCCKey
+
+
+class _AOTBuildTarget(NamedTuple):
+    """
+    Target identity of an explicit AOT build, with no device attached.
+
+    Holds the normalized, sorted, packed compute-capability keys. See
+    _DeviceBuildTarget for the cross-kind equality constraint.
+    """
+
+    cc_keys: tuple[_PackedCCKey, ...]
+
+
+# Inner key of a thread's per-factory wrapper cache; see _ThreadLocalCaches.
+# The target is None for explicit AOT builds: the decorator performs no device
+# query or cc normalization there, and the raw compute_capability kwarg is
+# already part of the specialization. Differently spelled ccs (80 vs (8, 0))
+# therefore yield distinct wrappers, which still share one compiled build
+# because cache_build_results normalizes its own key.
+_WrapperCacheKey = tuple[_DeviceBuildTarget | None, _SpecializationKey]
+# Composite key of the process-wide build-results cache:
+# (build-result type, build target (device or AOT), specialization). The
+# build-result type is the algorithm's Cython class from _bindings (e.g.
+# DeviceReduceBuildResult), namespacing entries per algorithm.
+# TODO: drop _DeviceBuildTarget from this union once v2 supports build-result
+# serialization; it then only keys the wrapper cache.
+_BuildResultsCacheKey = tuple[
+    type,
+    _DefaultBuildTarget | _DeviceBuildTarget | _AOTBuildTarget,
+    _SpecializationKey,
+]
+
+
+_thread_local = threading.local()
+# Process wide registry of per-thread caches. It enables a thread to call
+# clear_all_caches() to clear all caches across all threads.
+_process_wide_thread_cache_registry: weakref.WeakSet[_ThreadLocalCaches] = (
+    weakref.WeakSet()
+)
+_process_wide_thread_cache_registry_lock = threading.Lock()
+
+# _InFlightBuild entries are temporary: replaced by the completed build
+# results or removed on builder failure.
+_process_wide_build_results_cache: dict[
+    _BuildResultsCacheKey, _PerCCBuildResults | _InFlightBuild
+] = {}
+_CACHE_MISS = object()
+
+_KeyT = TypeVar("_KeyT", bound=Hashable)
+
+
+def _cache_single_flight(
+    cache: dict[_KeyT, Any], cache_key: _KeyT, builder: Callable[[], Any]
+) -> Any:
+    """Return a cached value, coalescing concurrent builds for the same key.
+
+    ``cache`` may be any single-flight dict — currently the process-wide
+    build-results cache and the per-device loaded results inside each
+    _PerCCBuildResults. Its entries are one of two things: a completed value,
+    which is terminal, or a temporary _InFlightBuild while the one elected
+    caller runs ``builder``; other callers wait on its event and receive the
+    same result or exception. A failed builder's entry is removed so a later
+    call retries.
+    """
+    cache_entry = cache.get(cache_key, _CACHE_MISS)
+    if cache_entry is _CACHE_MISS:
+        in_flight = _InFlightBuild()
+        # setdefault elects one builder without an explicit lock on cache hits.
+        cache_entry = cache.setdefault(cache_key, in_flight)
+        if cache_entry is in_flight:
+            try:
+                result = builder()
+                in_flight.result = result
+                cache[cache_key] = result
+            except BaseException as exc:
+                in_flight.exception = exc
+                cache.pop(cache_key, None)
+                raise
+            finally:
+                in_flight.event.set()
+            return result
+
+    if isinstance(cache_entry, _InFlightBuild):
+        cache_entry.event.wait()
+        if cache_entry.exception is not None:
+            raise cache_entry.exception
+        return cache_entry.result
+
+    return cache_entry
+
+
+class _PerCCBuildResults(dict[_PackedCCKey, Any]):
+    """One algorithm specialization's compiled build results, keyed by target cc.
+
+    Instances may be shared process-wide across threads: the factory build
+    cache hands all same-specialization wrappers one instance. The compiled
+    payload depends only on the cc, but a loaded build result holds
+    device-specific native state, so devices never share one. The first
+    device to execute claims and loads the canonical build result in place;
+    each additional device lazily loads its own clone of the compiled payload
+    (serialize -> deserialize -> load). This class tracks the loaded result —
+    canonical or clone — assigned to each device.
+    """
+
+    def __init__(
+        self,
+        build_results: dict[_PackedCCKey, Any],
+        *,
+        loaded_device_id: int | None = None,
+    ) -> None:
+        super().__init__(build_results)
+        # The single device that claimed each cc's canonical result and loads
+        # it in place; every other device clones instead. The atomic claim
+        # (resolve()'s setdefault) is what prevents double-loading the
+        # canonical object.
+        self._owner_devices: dict[_PackedCCKey, int] = {}
+        # The loaded result each (cc key, device ordinal) pair executes: the
+        # canonical result for its owner device; an independent clone — or,
+        # when cloning fails, an independently built result — for every other
+        # device. Also holds temporary _InFlightBuild entries while a first
+        # load is in flight.
+        self._loaded_results: dict[tuple[_PackedCCKey, int], Any] = {}
+        # Loading the canonical result mutates its native handle fields, while
+        # cloning serializes its payload. Serialize those source operations, but
+        # keep completed-result lookups lock-free.
+        self._source_locks = {cc: threading.Lock() for cc in self}
+
+        if loaded_device_id is not None:
+            # The caller already built and loaded the single entry on
+            # loaded_device_id: pre-record the post-conditions resolve()'s
+            # owner-load path would otherwise produce on first use.
+            if len(self) != 1:
+                raise ValueError("A device-bound _PerCCBuildResults must be singular")
+            for cc, build_result in self.items():
+                self._owner_devices[cc] = loaded_device_id
+                self._loaded_results[(cc, loaded_device_id)] = build_result
+
+    def resolve(self, cc: _PackedCCKey, device_id: int) -> Any:
+        """Return the build result loaded for ``device_id`` without recompiling."""
+        # Completed loads are terminal — never removed or replaced — so the
+        # warm path is one lock-free lookup with no dict mutation and no
+        # closure allocation. Misses and in-flight loads (which only exist
+        # around the first load per device) fall through to the single-flight
+        # machinery below.
+        loaded = self._loaded_results.get((cc, device_id))
+        if loaded is not None and not isinstance(loaded, _InFlightBuild):
+            return loaded
+
+        source = self[cc]
+        owner_device = self._owner_devices.setdefault(cc, device_id)
+
+        def load_for_device():
+            if owner_device == device_id:
+                with self._source_locks[cc]:
+                    source.load()
+                return source
+
+            with self._source_locks[cc]:
+                blob = source.serialize()
+            result = type(source).deserialize(blob, load=False, check_cc=True)
+            result.load()
+            return result
+
+        return _cache_single_flight(
+            self._loaded_results, (cc, device_id), load_for_device
+        )
+
+    def serialize_build_result(self, cc: _PackedCCKey) -> bytes:
+        """Serialize a canonical result without racing its first device load."""
+        with self._source_locks[cc]:
+            return self[cc].serialize()
+
+
+def _get_current_device_info() -> tuple[int, tuple[int, int]]:
+    device = Device()
+    cc_major, cc_minor = device.compute_capability
+    return device.device_id, (cc_major, cc_minor)
+
+
+def _get_thread_caches() -> _ThreadLocalCaches:
+    caches = getattr(_thread_local, "caches", None)
+    if caches is None:
+        caches = _ThreadLocalCaches()
+        _thread_local.caches = caches
+        with _process_wide_thread_cache_registry_lock:
+            _process_wide_thread_cache_registry.add(caches)
+    return caches
+
+
+def _clear_wrapper_caches(cache_name: str | None = None) -> None:
+    with _process_wide_thread_cache_registry_lock:
+        thread_caches = list(_process_wide_thread_cache_registry)
+
+    for caches in thread_caches:
+        if cache_name is None:
+            caches.wrapper_caches.clear()
+        else:
+            caches.wrapper_caches.pop(cache_name, None)
+
+
+def cache_build_results(
+    build_result_type: type,
+    *key_args,
+    compute_capability,
+    builder: Callable[[], Any],
+) -> Any:
+    """
+    Cache the shared Cython build results for one specialization.
+
+    Current-device builds are keyed by compute capability alone and shared
+    across same-cc device ordinals: the compiled payload only depends on the
+    cc, and _PerCCBuildResults.resolve() gives each device its own loaded
+    state. When the backend cannot serialize build results (the v2 HostJIT
+    backend today), a non-owner device has no way to load the shared entry —
+    loading it clones through serialization — so default builds are keyed per
+    device ordinal instead and each device builds its own entry. Explicit AOT
+    builds have no current device and are keyed by their normalized target
+    compute capabilities. The key intentionally excludes the current Python
+    thread so wrappers can share compiled results.
+
+    Args:
+        build_result_type: Cython build-result type. This separates entries
+            that may otherwise have identical specialization keys.
+        *key_args: Positional values used to form the specialization part of
+            the cache key.
+        compute_capability: Explicit AOT target or ``None`` for the current
+            device.
+        builder: Callable that creates the _PerCCBuildResults on a cache miss.
+            Exactly one thread runs this callable for a given key while other
+            threads wait for the result.
+
+    Returns:
+        ``(build_results, bound_result)``: the cached or newly built
+        _PerCCBuildResults and, for current-device builds, the loaded result
+        bound to the constructing device — resolved once here so ``__call__``
+        needs no device query. ``bound_result`` is ``None`` for explicit AOT
+        builds, which resolve per call.
+    """
+    from ._cccl_interop import cc_to_key, normalize_compute_capabilities
+
+    if compute_capability is None:
+        # The factory decorator already queried the device on the wrapper-cache
+        # miss path and hands the result through thread-local state; fall back
+        # to a fresh query for direct construction (e.g. deserialization).
+        device_info = getattr(_thread_local, "factory_device_info", None)
+        if device_info is None:
+            device_info = _get_current_device_info()
+        device_id, cc = device_info
+        packed_cc = cc_to_key(cc)
+        # TODO: reduce to _DefaultBuildTarget(packed_cc) once v2 supports
+        # build-result serialization.
+        target_key = (
+            _DefaultBuildTarget(packed_cc)
+            if _BACKEND_SERIALIZES_BUILD_RESULTS
+            else _DeviceBuildTarget(device_id, cc)
+        )
+        user_cache_key = _make_cache_key_from_args(*key_args)
+        cache_key = (build_result_type, target_key, user_cache_key)
+        build_results = _cache_single_flight(
+            _process_wide_build_results_cache, cache_key, builder
+        )
+        return build_results, _bind_default_build(
+            build_results, packed_cc, device_id, builder
+        )
+
+    target_ccs = normalize_compute_capabilities(compute_capability)
+    assert target_ccs is not None
+    aot_target_key = _AOTBuildTarget(tuple(cc_to_key(cc) for cc in target_ccs))
+    user_cache_key = _make_cache_key_from_args(*key_args)
+    aot_cache_key = (build_result_type, aot_target_key, user_cache_key)
+    return (
+        _cache_single_flight(_process_wide_build_results_cache, aot_cache_key, builder),
+        None,
+    )
+
+
+def _bind_default_build(
+    build_results, packed_cc: _PackedCCKey, device_id: int, builder
+):
+    """Resolve the loaded result the constructing device executes.
+
+    Default wrappers are bound to the device that was current at factory-call
+    time (their wrapper-cache key includes it), so the binding is resolved
+    once here and reused by every ``__call__`` with no device query. For the
+    device that built the shared entry this is a warm lookup; another same-cc
+    device loads its own clone of the compiled payload here (serialize ->
+    deserialize -> load, milliseconds where a build costs a second).
+
+    Cloning is only an optimization and its failures have no stable exception
+    type, so any resolve failure falls back to a full build for this device —
+    recorded in the shared per-device slot so same-device threads share it —
+    and genuine errors surface from the build path itself.
+    """
+    try:
+        return build_results.resolve(packed_cc, device_id)
+    except Exception:
+
+        def build_privately():
+            (result,) = builder().values()
+            return result
+
+        return _cache_single_flight(
+            build_results._loaded_results, (packed_cc, device_id), build_privately
+        )
+
+
+class _CacheWithRegisteredKeyFunctions:
+    """
+    Decorator to cache the result of the decorated function.
+
+    The cache key is automatically computed from the decorated function's
+    arguments using the registered key functions.
+    """
+
+    def __call__(self, func: Callable) -> Callable:
+        """
+        Decorator to cache the result of the decorated function.
+
+        Args:
+            func: The function whose result is to be cached.
+
+        Notes
+        -----
+        Default builds append the current CUDA device and compute capability to
+        the cache key. Explicit AOT builds include their normalized target
+        compute capabilities without querying a device.
+        """
+        cache_name = func.__qualname__
+
+        @functools.wraps(func)
+        def inner(*args, **kwargs):
+            user_cache_key = _make_cache_key_from_args(*args, **kwargs)
+            # When the caller targets explicit compute capabilities, that value
+            # is already part of user_cache_key (it arrives as a kwarg) and we
+            # must NOT query a device — the whole point is to build without a
+            # GPU. Otherwise, salt the key with the current device's cc so a
+            # build cached on one device isn't reused on another.
+            if kwargs.get("compute_capability") is None:
+                # Only device-availability failures should be reinterpreted as
+                # "pass compute_capability": no driver / no device raises
+                # CUDAError, and querying device 0 on a machine with zero
+                # devices raises ValueError. Anything else (a real bug) must
+                # propagate untouched. The original error is chained and echoed
+                # so a genuine driver/permission failure isn't hidden behind a
+                # misleading "no device" message.
+                try:
+                    device_id, cc = _get_current_device_info()
+                except (CUDAError, ValueError) as e:
+                    raise RuntimeError(
+                        "make_<algo> was called without compute_capability and the "
+                        f"current CUDA device could not be queried ({e}). Pass "
+                        "compute_capability=<cc or list of ccs> to compile without "
+                        "a GPU (e.g. with ProxyArray / ProxyValue)."
+                    ) from e
+                target = _DeviceBuildTarget(device_id, cc)
+                target_cc_arg = cc
+            else:
+                target = None
+                target_cc_arg = kwargs.get("compute_capability")
+            # No thread id in the key: the containing cache is threading.local,
+            # so each thread only ever sees its own entries.
+            cache_key = (target, user_cache_key)
+            thread_caches = _get_thread_caches()
+            cache = thread_caches.wrapper_caches.setdefault(cache_name, {})
+            if cache_key not in cache:
+                # Shared device code (operators, iterators) is compiled to LTO-IR
+                # once and linked into every per-arch build result, so it must target
+                # the lowest requested cc (nvJitLink requires final SM >= each
+                # linked input's arch). Set that target around the build.
+                from ._target_cc import target_cc
+
+                # Hand the device info queried above to cache_build_results
+                # (reached through the wrapper's __init__) so the miss path
+                # does not construct a second cuda.core Device. Saved/restored
+                # so nested factory calls fall back to their own query.
+                previous_device_info = getattr(
+                    _thread_local, "factory_device_info", None
+                )
+                _thread_local.factory_device_info = target
+                try:
+                    with target_cc(target_cc_arg):
+                        result = func(*args, **kwargs)
+                finally:
+                    _thread_local.factory_device_info = previous_device_info
+                cache[cache_key] = result
+            return cache[cache_key]
+
+        inner.cache_clear = lambda: _clear_wrapper_caches(cache_name)  # type: ignore[attr-defined]
+
+        # Register the cache in the central registry
+        _process_wide_cache_registry[func.__qualname__] = inner
+
+        return inner
+
+    def register(self, type_: type, key_function: Callable[[Any], Hashable]) -> None:
+        """
+        Register a key function for a specific type.
+
+        A key function extracts a hashable cache key from a value.
+
+        Args:
+            type_: The type to register
+            key_function: A callable that takes an instance of type_ and
+                returns a hashable cache key
+        """
+        _KEY_FUNCTIONS[type_] = key_function
+
+
+cache_with_registered_key_functions = _CacheWithRegisteredKeyFunctions()
+
+
+def _make_hashable(value):
+    # duck-type check for numba.cuda.CUDADispatcher:
+    if hasattr(value, "py_func") and callable(value.py_func):
+        return CachableFunction(value.py_func)
+    elif is_device_array(value):
+        # Ops with device arrays in globals/closures will be handled
+        # by stateful op machinery, which enables updating the state
+        # (pointers). Thus, we only cache on the dtype and shape of
+        # the referenced array, but not its pointer.
+        return (get_dtype(value), get_shape(value))
+    elif isinstance(value, (np.number, np.bool_)):
+        return ("numpy.scalar", value.dtype.str, value.tobytes())
+    elif isinstance(value, (bool, int, float)):
+        # Python scalars are immutable values; key them by type and value so
+        # equal-valued scalars share a cache entry. Without this they fall
+        # through to ``id(value)`` below, and a fresh (non-interned) ``int``/
+        # ``float`` with the same value misses the build cache on every call.
+        # ``_type_fqn`` keeps ``True`` distinct from ``1``/``1.0`` (and avoids
+        # collisions between like-named scalar subclasses from other modules).
+        return ("python.scalar", _type_fqn(value), value)
+    elif isinstance(value, (list, tuple)):
+        return tuple(_make_hashable(v) for v in value)
+    elif isinstance(value, dict):
+        return tuple(
+            sorted((_make_hashable(k), _make_hashable(v)) for k, v in value.items())
+        )
+    else:
+        return id(value)
+
+
+def clear_all_caches():
+    """
+    Clear all algorithm caches.
+
+    This function clears cached algorithm wrappers and completed build results
+    in the current process, forcing recompilation on the next invocation.
+    Useful for benchmarking compilation time.
+
+    This function is not synchronized with active factory calls or algorithm
+    execution. Callers that use it in a multi-threaded program must externally
+    synchronize with all threads that may create or use cuda.compute algorithm
+    objects. If a build is already in progress, that build may complete after
+    this function returns and repopulate the completed build-result cache.
+
+    Example
+    -------
+    >>> import cuda.compute
+    >>> cuda.compute.clear_all_caches()
+    """
+    _clear_wrapper_caches()
+    _process_wide_build_results_cache.clear()
+    # Auxiliary caches registered process-wide (e.g. _jit._infer_return_type)
+    # must be cleared too, so builds after a clear really are cold. Factory
+    # entries' cache_clear is idempotent with _clear_wrapper_caches above.
+    for cached_func in _process_wide_cache_registry.values():
+        cached_func.cache_clear()
+
+
+class CachableFunction:
+    """
+    A type that wraps a function and provides custom comparison
+    (__eq__) and hash (__hash__) implementations.
+
+    The purpose of this class is to enable caching and comparison of
+    functions based on their bytecode, constants, and closures, while
+    ignoring other attributes such as their names or docstrings.
+    """
+
+    # TODO: eventually, move this class to _jit.py as it only
+    # has to do with caching of Python callables that will be
+    # JIT compiled.
+    def __init__(self, func):
+        self._func = func
+
+        closure = func.__closure__ if func.__closure__ is not None else []
+        contents = []
+        # Make closure contents hashable
+        for cell in closure:
+            contents.append(_make_hashable(cell.cell_contents))
+        self._identity = (
+            func.__name__,
+            func.__code__.co_code,
+            func.__code__.co_consts,
+            tuple(contents),
+            tuple(
+                # if `name` is found in __globals__, try and hash
+                # the referenced object. If `name` is not found in
+                # __globals__, (e.g., `name` is part of a dotted
+                # name like `np.argmax`), for caching purposes we
+                # use the hash of the name itself. Assumes numba
+                # known how to interpret the dotted name at JIT
+                # time.
+                _make_hashable(func.__globals__.get(name, name))
+                for name in func.__code__.co_names
+            ),
+        )
+
+    def __eq__(self, other):
+        return self._identity == other._identity
+
+    def __hash__(self):
+        return hash(self._identity)
+
+    def __repr__(self):
+        return str(self._func)
+
+
+# Register keyers for built-in types
+
+cache_with_registered_key_functions.register(
+    np.ndarray, lambda arr: ("numpy.ndarray", arr.dtype)
+)
+cache_with_registered_key_functions.register(
+    types.FunctionType, lambda fn: CachableFunction(fn)
+)
+cache_with_registered_key_functions.register(_Struct, lambda v: (_type_fqn(v), v.dtype))
+
+
+def _register_proxy_types():
+    # Registered lazily to avoid importing _proxy (and numpy-dtype construction)
+    # at module import time; the keys are dtype-only so equal-dtype proxies share
+    # a cache entry.
+    from ._proxy import ProxyArray, ProxyValue
+
+    cache_with_registered_key_functions.register(
+        ProxyArray, lambda v: ("ProxyArray", v.dtype)
+    )
+    cache_with_registered_key_functions.register(
+        ProxyValue, lambda v: ("ProxyValue", v.dtype)
+    )
+
+
+_register_proxy_types()
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_cccl_interop.py b/cccl_upstream/python/cuda_cccl/cuda/compute/_cccl_interop.py
new file mode 100644
index 00000000..c1c15c09
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_cccl_interop.py
@@ -0,0 +1,471 @@
+# Copyright (c) 2024-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+from __future__ import annotations
+
+import enum
+import functools
+import os
+import subprocess
+import tempfile
+import warnings
+from typing import Callable, List
+
+try:
+    from cuda.core import Device as CudaDevice
+except ImportError:
+    from cuda.core.experimental import Device as CudaDevice
+
+
+import numpy as np
+
+# TODO: adding a type-ignore here because `cuda` being a
+# namespace package confuses mypy when `cuda.<something_else>`
+# is installed, but not `cuda.cccl`. For namespace packages,
+# it appears we need to actually install the sub-package
+# in order for mypy to find its py.typed file. However, CI
+# does type checking of `cuda.cccl` without actually installing
+# it.
+#
+# We need to find a better solution for this.
+from cuda.cccl import get_include_paths  # type: ignore
+
+from . import types
+from ._bindings import (
+    CommonData,
+    Iterator,
+    IteratorKind,
+    IteratorState,
+    Op,
+    OpKind,
+    Pointer,
+    TypeEnum,
+    TypeInfo,
+    Value,
+    make_pointer_object,
+)
+from ._caching import _PerCCBuildResults
+from ._utils.protocols import get_data_pointer, get_dtype, is_contiguous
+from .iterators._base import IteratorBase
+from .typing import DeviceArrayLike, GpuStruct
+
+# Mapping from numpy dtype to TypeEnum for creating TypeInfo
+_NUMPY_DTYPE_TO_ENUM = {
+    np.dtype("int8"): TypeEnum.INT8,
+    np.dtype("int16"): TypeEnum.INT16,
+    np.dtype("int32"): TypeEnum.INT32,
+    np.dtype("int64"): TypeEnum.INT64,
+    np.dtype("uint8"): TypeEnum.UINT8,
+    np.dtype("uint16"): TypeEnum.UINT16,
+    np.dtype("uint32"): TypeEnum.UINT32,
+    np.dtype("uint64"): TypeEnum.UINT64,
+    np.dtype("float16"): TypeEnum.FLOAT16,
+    np.dtype("float32"): TypeEnum.FLOAT32,
+    np.dtype("float64"): TypeEnum.FLOAT64,
+    np.dtype("bool"): TypeEnum.BOOLEAN,
+}
+
+
+@functools.lru_cache(maxsize=256)
+def _type_info_from_dtype(dtype: np.dtype) -> TypeInfo:
+    """
+    Create a TypeInfo from a numpy dtype.
+    Handles both primitive types and structured dtypes.
+    """
+    dtype = np.dtype(dtype)
+
+    # Handle structured dtypes
+    if dtype.type == np.void and dtype.fields is not None:
+        return TypeInfo(dtype.itemsize, dtype.alignment, TypeEnum.STORAGE)
+
+    if dtype.kind == "c":
+        return TypeInfo(dtype.itemsize, dtype.alignment, TypeEnum.STORAGE)
+
+    # Fallback for any other type
+    type_enum = _NUMPY_DTYPE_TO_ENUM.get(dtype, TypeEnum.STORAGE)
+    return TypeInfo(dtype.itemsize, dtype.alignment, type_enum)
+
+
+def _is_well_known_op(op: OpKind) -> bool:
+    return isinstance(op, OpKind) and op not in (OpKind.STATELESS, OpKind.STATEFUL)
+
+
+def _device_array_to_cccl_iter(array: DeviceArrayLike) -> Iterator:
+    from ._proxy import ProxyArray
+
+    if not is_contiguous(array):
+        raise ValueError("Non-contiguous arrays are not supported.")
+    dtype = get_dtype(array)
+
+    info = _type_info_from_dtype(dtype)
+    state_info = _type_info_from_dtype(np.intp)
+    # A ProxyArray has no GPU allocation: leave the pointer NULL for build-time
+    # (ahead-of-time) compilation. The real pointer is bound at __call__ via
+    # set_cccl_iterator_state().
+    state = None if isinstance(array, ProxyArray) else get_data_pointer(array)
+    return Iterator(
+        state_info.alignment,
+        IteratorKind.POINTER,
+        Op(),
+        Op(),
+        info,
+        # Note: this is slightly slower, but supports all ndarray-like objects
+        # as long as they support CAI
+        # TODO: switch to use gpumemoryview once it's ready
+        state=state,
+    )
+
+
+def _none_to_cccl_iter() -> Iterator:
+    # Any type could be used here, we just need to pass NULL.
+    info = _type_info_from_dtype(np.uint8)
+    return Iterator(info.alignment, IteratorKind.POINTER, Op(), Op(), info, state=None)
+
+
+class _IteratorIO(enum.Enum):
+    INPUT = 0
+    OUTPUT = 1
+
+
+def _to_cccl_iter(
+    it: DeviceArrayLike | IteratorBase | None, io_kind: _IteratorIO
+) -> Iterator:
+    if it is None:
+        return _none_to_cccl_iter()
+    if isinstance(it, IteratorBase):
+        return it.to_cccl_iter(io_kind == _IteratorIO.OUTPUT)
+    return _device_array_to_cccl_iter(it)
+
+
+def to_cccl_input_iter(array_or_iterator) -> Iterator:
+    return _to_cccl_iter(array_or_iterator, _IteratorIO.INPUT)
+
+
+def to_cccl_output_iter(array_or_iterator) -> Iterator:
+    return _to_cccl_iter(array_or_iterator, _IteratorIO.OUTPUT)
+
+
+def to_cccl_value_state(array_or_struct: np.ndarray | GpuStruct) -> memoryview:
+    from ._proxy import _PROXY_VALUE_DATA_ERROR, ProxyValue
+
+    if isinstance(array_or_struct, ProxyValue):
+        # Reached only if a proxy leaks into an execute call — proxies describe
+        # types for build, they carry no data to run with.
+        raise RuntimeError(_PROXY_VALUE_DATA_ERROR)
+    if isinstance(array_or_struct, np.ndarray):
+        assert array_or_struct.flags.contiguous
+        data = array_or_struct.data.cast("B")
+        return data
+    else:
+        # it's a GpuStruct, use the array underlying it
+        return to_cccl_value_state(array_or_struct._data)
+
+
+def to_cccl_value(array_or_struct: np.ndarray | GpuStruct) -> Value:
+    from ._proxy import ProxyValue
+
+    if isinstance(array_or_struct, ProxyValue):
+        # Build-time placeholder: describe the type with a correctly sized zero
+        # buffer. The real value bytes are bound at __call__ via
+        # set_cccl_value_state().
+        info = _type_info_from_dtype(array_or_struct.dtype)
+        zero_bytes = memoryview(bytearray(array_or_struct.dtype.itemsize))
+        return Value(info, zero_bytes)
+    if isinstance(array_or_struct, np.ndarray):
+        info = _type_info_from_dtype(array_or_struct.dtype)
+        return Value(info, array_or_struct.data.cast("B"))
+    else:
+        # it's a GpuStruct, use the array underlying it
+        return to_cccl_value(array_or_struct._data)
+
+
+def set_cccl_value_state(cccl_value: Value, array_or_struct: np.ndarray | GpuStruct):
+    """
+    Set the state of a CCCL Value object from a numpy array or GpuStruct.
+
+    Args:
+        cccl_value: The CCCL Value binding object
+        array_or_struct: The numpy array or GpuStruct to get the state from
+    """
+    cccl_value.state = to_cccl_value_state(array_or_struct)
+
+
+def get_value_type(
+    d_in: DeviceArrayLike | IteratorBase | GpuStruct | np.ndarray,
+):
+    from ._proxy import ProxyValue
+    from .struct import _Struct
+
+    if isinstance(d_in, IteratorBase):
+        return d_in.value_type
+
+    if isinstance(d_in, ProxyValue):
+        return types.from_numpy_dtype(d_in.dtype)
+
+    if isinstance(d_in, _Struct):
+        return type(d_in)._type_descriptor  # type: ignore[union-attr]
+
+    dtype = get_dtype(d_in)
+
+    if dtype.type == np.void:
+        return types.from_numpy_dtype(dtype)
+
+    return types.from_numpy_dtype(dtype)
+
+
+def set_cccl_iterator_state(cccl_it: Iterator, input_it):
+    if cccl_it.is_kind_pointer():
+        ptr = get_data_pointer(input_it)
+        ptr_obj = make_pointer_object(ptr, input_it)
+        cccl_it.state = ptr_obj
+    else:
+        state_ = input_it.state
+        if isinstance(state_, (IteratorState, Pointer)):
+            cccl_it.state = state_
+        else:
+            cccl_it.state = make_pointer_object(state_, input_it)
+
+
+@functools.lru_cache()
+def get_includes() -> List[str]:
+    def as_option(p):
+        if p is None:
+            return ""
+        return f"-I{p}"
+
+    paths = get_include_paths().as_tuple()
+    opts = [as_option(path) for path in paths]
+    return opts
+
+
+def _check_compile_result(cubin: bytes):
+    # check compiled code for LDL/STL instructions
+    temp_cubin_file = tempfile.NamedTemporaryFile(delete=False)
+    try:
+        temp_cubin_file.write(cubin)
+        out = subprocess.run(
+            ["nvdisasm", "-gi", temp_cubin_file.name], capture_output=True
+        )
+        if out.returncode != 0:
+            raise RuntimeError("nvdisasm failed")
+        sass = out.stdout.decode("utf-8")
+    except FileNotFoundError:
+        sass = "nvdiasm not found, skipping SASS validation"
+        warnings.warn(sass)
+
+    assert "LDL" not in sass, "LDL instruction found in SASS"
+    assert "STL" not in sass, "STL instruction found in SASS"
+    return temp_cubin_file.name
+
+
+# this global variable controls whether the compile result is checked
+# for LDL/STL instructions. Should be set to `True` for testing only.
+_check_sass: bool = False
+
+
+def _common_data_for_cc(cc):
+    """Build a ``CommonData`` for a given compute capability.
+
+    ``cc`` is a ``(major, minor)`` pair. When ``None``, the current device's
+    compute capability is queried (requires a live GPU).
+    """
+    if cc is None:
+        cc_major, cc_minor = CudaDevice().compute_capability
+    else:
+        cc_major, cc_minor = cc
+    cub_path, thrust_path, libcudacxx_path, cuda_include_path = get_includes()
+    return CommonData(
+        cc_major, cc_minor, cub_path, thrust_path, libcudacxx_path, cuda_include_path
+    )
+
+
+def call_build(build_impl_fn: Callable, *args, cc=None, **kwargs):
+    """Build (compile + load) via ``build_impl_fn``, supplying compute capability and paths.
+
+    ``cc`` is an optional ``(major, minor)`` pair; when ``None`` the current
+    device's compute capability is used (the default, load-bearing behavior).
+    Returns the loaded build result.
+    """
+    global _check_sass
+
+    common_data = _common_data_for_cc(cc)
+    result = build_impl_fn(
+        *args,
+        common_data,
+        **kwargs,
+    )
+
+    if _check_sass:
+        cubin = result._get_cubin()
+        temp_cubin_file_name = _check_compile_result(cubin)
+        os.unlink(temp_cubin_file_name)
+
+    return result
+
+
+def call_compile(build_impl_cls: Callable, *args, cc, **kwargs):
+    """Compile only (no load) for an explicit compute capability ``cc``.
+
+    ``build_impl_cls`` is a ``Device<Algo>BuildResult`` type exposing a
+    ``compile(...)`` staticmethod. Unlike :func:`call_build`, this never touches
+    the CUDA driver — it can run on a machine with no GPU. The returned build
+    result is *not* loaded; call ``.load()`` (once, on a matching device) before
+    executing. ``cc`` is a ``(major, minor)`` pair and is required.
+    """
+    common_data = _common_data_for_cc(cc)
+    # build_impl_cls is a Device<Algo>BuildResult class exposing a compile()
+    # staticmethod; it's typed Callable here, so silence the attr check.
+    return build_impl_cls.compile(*args, common_data, **kwargs)  # type: ignore[attr-defined]
+
+
+def build_for_ccs(build_impl_cls: Callable, *args, compute_capability=None, **kwargs):
+    """Build the ``{cc_key: build_result}`` map for an algorithm.
+
+    With ``compute_capability=None`` (the default), this performs a fused
+    build+load for the current device and returns a single-entry map whose
+    result is already loaded. Otherwise it compiles (without loading) for each
+    requested compute capability and returns ``{cc_key: build_result}``, with
+    each result loaded lazily on first use by ``resolve_build_result``.
+    """
+    ccs = normalize_compute_capabilities(compute_capability)
+    if ccs is None:
+        # Fused build+load for the current device. Query its cc once (clear error
+        # if no device) and pass it through, so call_build doesn't re-query.
+        device_id, cc_key = current_device_info()
+        build_result = call_build(build_impl_cls, *args, cc=key_to_cc(cc_key), **kwargs)
+        # The fused build already loaded the kernels; mark it so the lazy
+        # load() in resolve_build_result() is a no-op (a second C load would leak /
+        # re-register the library).
+        build_result._loaded = True
+        return _PerCCBuildResults({cc_key: build_result}, loaded_device_id=device_id)
+    return _PerCCBuildResults(
+        {
+            cc_to_key(cc): call_compile(build_impl_cls, *args, cc=cc, **kwargs)
+            for cc in ccs
+        }
+    )
+
+
+def cc_to_key(cc) -> int:
+    """Normalize a compute capability to the integer key ``major * 10 + minor``.
+
+    Accepts an int (``90``, ``75``), a ``(major, minor)`` pair, or a string
+    like ``"90"`` / ``"9.0"``.
+    """
+    if isinstance(cc, (tuple, list)):
+        major, minor = cc
+        return int(major) * 10 + int(minor)
+    if isinstance(cc, str):
+        cc = cc.replace(".", "")
+        return int(cc)
+    return int(cc)
+
+
+def key_to_cc(key: int):
+    """Inverse of :func:`cc_to_key`: integer key -> ``(major, minor)`` pair."""
+    return (key // 10, key % 10)
+
+
+def normalize_compute_capabilities(compute_capability):
+    """Normalize the ``compute_capability=`` argument of ``make_<algo>``.
+
+    Returns a sorted list of unique ``(major, minor)`` pairs, or ``None`` to
+    mean "use the current device" (the default build path). Accepts a single
+    cc (int / pair / str) or a list thereof.
+    """
+    if compute_capability is None:
+        return None
+    if isinstance(compute_capability, (int, str)):
+        ccs = [compute_capability]
+    elif (
+        isinstance(compute_capability, tuple)
+        and len(compute_capability) == 2
+        and all(isinstance(x, int) for x in compute_capability)
+    ):
+        # a single (major, minor) pair
+        ccs = [compute_capability]
+    else:
+        ccs = list(compute_capability)
+    keys = sorted({cc_to_key(cc) for cc in ccs})
+    if not keys:
+        raise ValueError("compute_capability list is empty")
+    return [key_to_cc(k) for k in keys]
+
+
+def current_device_info() -> tuple[int, int]:
+    """Return the current device ordinal and packed compute-capability key.
+
+    Raises a clear, actionable error if no CUDA device is available: building
+    without a GPU has no device to infer the target arch from, so the caller
+    must pass an explicit ``compute_capability=``.
+    """
+    try:
+        device = CudaDevice()
+        cc = device.compute_capability
+    except Exception as e:
+        raise RuntimeError(
+            "No compute_capability was given and no CUDA device is available to target."
+        ) from e
+    return device.device_id, cc_to_key(tuple(cc))
+
+
+def current_device_cc_key() -> int:
+    """The current device's compute capability as a ``major * 10 + minor`` key."""
+    return current_device_info()[1]
+
+
+def current_device_id() -> int:
+    """The current CUDA device ordinal, without a compute-capability query.
+
+    The compute-capability query roughly doubles the cost of
+    ``current_device_info()``, and callers that only key per-device state
+    (see ``resolve_build_result``) run on every algorithm invocation.
+    """
+    try:
+        return CudaDevice().device_id
+    except Exception as e:
+        raise RuntimeError("No CUDA device is available to execute on.") from e
+
+
+def resolve_build_result(build_results: dict, bound_result=None):
+    """Load the build result for the current device.
+
+    ``bound_result`` is a default-build wrapper's construction-time binding
+    (see cache_build_results): already the loaded result for the wrapper's
+    device, returned without any device query. Deserialized wrappers have no
+    binding and resolve per call.
+    """
+    if bound_result is not None:
+        return bound_result
+
+    # Wrappers always hold a _PerCCBuildResults (build_for_ccs and
+    # deserialization both produce one); the per-device ownership/clone
+    # protocol in resolve() relies on it, so fail loudly on anything else
+    # rather than fall back to an unprotected load.
+    assert isinstance(build_results, _PerCCBuildResults)
+
+    if len(build_results) == 1:
+        # A singular _PerCCBuildResults is used as-is whatever the current device's
+        # compute capability is (single-target blobs were already cc-checked at
+        # deserialization), so only the device ordinal is needed to key the
+        # per-device loaded state. This path runs on every call of AOT and
+        # deserialized wrappers; skip the costlier compute-capability query.
+        (build_result_cc,) = build_results
+        device_id = current_device_id()
+    else:
+        device_id, device_cc_key = current_device_info()
+        build_result_cc = device_cc_key
+        if build_result_cc not in build_results:
+            available = ", ".join(
+                f"{maj}.{minor}"
+                for maj, minor in (key_to_cc(k) for k in sorted(build_results))
+            )
+            major, minor = key_to_cc(build_result_cc)
+            raise RuntimeError(
+                f"This algorithm was compiled for compute capabilities [{available}], "
+                f"but the current device has compute capability {major}.{minor}. "
+                f"Rebuild with compute_capability including {major}{minor}."
+            )
+
+    return build_results.resolve(build_result_cc, device_id)
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_cpp_compile.py b/cccl_upstream/python/cuda_cccl/cuda/compute/_cpp_compile.py
new file mode 100644
index 00000000..87916ca7
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_cpp_compile.py
@@ -0,0 +1,172 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+C++ code generation and compilation infrastructure.
+"""
+
+from __future__ import annotations
+
+import functools
+
+from cuda.cccl import get_include_paths
+from cuda.core import Device, Program, ProgramOptions
+
+from ._bindings import TypeEnum
+from ._device_code import DeviceCode
+
+try:
+    from ._build_info import USING_V2  # type: ignore[import-not-found]
+except ImportError:
+    USING_V2 = False
+
+
+def _get_arch_string() -> str:
+    """Target arch string for iterator LTO-IR compilation.
+
+    Honors the build's target compute capability (set for multi-arch / no-GPU
+    builds) so iterator device code is compiled for the lowest target arch and
+    links into every build result; falls back to the current device otherwise.
+    """
+    from ._target_cc import get_target_cc
+
+    cc = get_target_cc()
+    if cc is None:
+        cc = Device().compute_capability
+    cc_major, cc_minor = cc
+    return f"sm_{cc_major}{cc_minor}"
+
+
+@functools.lru_cache(maxsize=1)
+def _get_include_paths() -> list[str]:
+    """Get include paths for CCCL headers."""
+    paths = get_include_paths().as_tuple()
+    return [p for p in paths if p is not None]
+
+
+def compile_cpp_to_ltoir(
+    source: str,
+    arch: str | None = None,
+) -> bytes:
+    """
+    Compile C++ source code to LTOIR.
+
+    Args:
+        source: C++ source code string
+        arch: Target architecture (e.g., "sm_80"). If None, uses current device.
+
+    Returns:
+        LTOIR bytes
+
+    Example:
+        source = '''
+        extern "C" __device__ void my_add(void* a, void* b, void* result) {
+            *static_cast<int*>(result) = *static_cast<int*>(a) + *static_cast<int*>(b);
+        }
+        '''
+        ltoir = compile_cpp_to_ltoir(source)
+    """
+    # Resolve the concrete arch before the cache lookup so the key reflects the
+    # compute capability compiled for. If arch stays None (the usual iterator/op
+    # call, resolved from target_cc), every target collapses to one key and
+    # LTO-IR built for one arch can be reused for another, which nvJitLink
+    # rejects.
+    if arch is None:
+        arch = _get_arch_string()
+    return _compile_cpp_to_ltoir_cached(source, arch)
+
+
+@functools.lru_cache(maxsize=256)
+def _compile_cpp_to_ltoir_cached(source: str, arch: str) -> bytes:
+    # Get include paths
+    include_paths = _get_include_paths()
+
+    # Configure compilation options for LTO
+    opts = ProgramOptions(
+        arch=arch,
+        relocatable_device_code=True,
+        link_time_optimization=True,
+        std="c++20",
+        define_macro="__NV_NO_VECTOR_DEPRECATION_DIAG",
+        include_path=include_paths,
+    )
+
+    # Compile to LTOIR
+    program = Program(source, "c++", options=opts)
+    result = program.compile("ltoir")
+
+    return result.code
+
+
+# Expose the cached-callable surface (cache_info/cache_clear) on the public
+# entry point, backed by the arch-aware inner cache.
+compile_cpp_to_ltoir.cache_clear = _compile_cpp_to_ltoir_cached.cache_clear  # type: ignore[attr-defined]
+compile_cpp_to_ltoir.cache_info = _compile_cpp_to_ltoir_cached.cache_info  # type: ignore[attr-defined]
+
+
+def compile_cpp_op_code(source: str, arch: str | None = None) -> DeviceCode:
+    """Compile C++ wrapper source to whatever form the active backend prefers.
+
+    Returns a :class:`DeviceCode` wrapping the bytes and the matching format tag.
+
+    Cached so identical iterator structures produce identical code bytes —
+    callers can inspect ``cache_info()`` to verify symbol determinism.
+    """
+    # v2 keeps the C++ source verbatim (arch-independent); v1 resolves the
+    # concrete arch before caching (see compile_cpp_to_ltoir).
+    if USING_V2:
+        return _compile_cpp_op_code_cached(source, None)
+    if arch is None:
+        arch = _get_arch_string()
+    return _compile_cpp_op_code_cached(source, arch)
+
+
+@functools.lru_cache(maxsize=256)
+def _compile_cpp_op_code_cached(source: str, arch: str | None) -> DeviceCode:
+    if USING_V2:
+        return DeviceCode(op_bytes=source.encode("utf-8"), kind="cpp_source")
+    return DeviceCode(op_bytes=compile_cpp_to_ltoir(source, arch=arch), kind="ltoir")
+
+
+compile_cpp_op_code.cache_clear = _compile_cpp_op_code_cached.cache_clear  # type: ignore[attr-defined]
+compile_cpp_op_code.cache_info = _compile_cpp_op_code_cached.cache_info  # type: ignore[attr-defined]
+
+
+def cpp_type_from_descriptor(type_desc) -> str | None:
+    """
+    Get the C++ type name from a TypeDescriptor.
+
+    Important: for efficiency, this function returns None
+    for non-primitive types. Callers must take care
+    to handle that case appropriately.
+    """
+    # Map TypeEnum to C++ types
+    type_map = {
+        TypeEnum.INT8: "int8_t",
+        TypeEnum.INT16: "int16_t",
+        TypeEnum.INT32: "int32_t",
+        TypeEnum.INT64: "int64_t",
+        TypeEnum.UINT8: "uint8_t",
+        TypeEnum.UINT16: "uint16_t",
+        TypeEnum.UINT32: "uint32_t",
+        TypeEnum.UINT64: "uint64_t",
+        TypeEnum.FLOAT16: "__half",
+        TypeEnum.FLOAT32: "float",
+        TypeEnum.FLOAT64: "double",
+        TypeEnum.BOOLEAN: "bool",
+        TypeEnum.STORAGE: None,
+    }
+    return type_map[type_desc.info.typenum]
+
+
+def make_variable_declaration(type_desc, name: str) -> str:
+    """
+    Generate a C++ variable declaration, like "int32_t temp;"
+    or "alignas(8) char temp[16];"
+    """
+    cpp_type = cpp_type_from_descriptor(type_desc)
+    if cpp_type is not None:
+        return f"{cpp_type} {name};"
+    # STORAGE type - use aligned char array
+    return f"alignas({type_desc.alignment}) char {name}[{type_desc.size}];"
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_device_code.py b/cccl_upstream/python/cuda_cccl/cuda/compute/_device_code.py
new file mode 100644
index 00000000..32b93693
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_device_code.py
@@ -0,0 +1,45 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Public type for passing compiled device-side operator code into ``Op`` /
+``RawOp``. Wraps the bytes together with their format tag so the two cannot
+get out of sync as they flow through the binding layer.
+
+Lives in its own module to keep ``op.py`` and the Cython ``_bindings_impl``
+free of import-cycle headaches: the Cython side duck-types on
+``(op_bytes, kind)`` attributes and never imports the class directly.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+# Tag values mirror the C-side ``cccl_op_code_type`` enum.
+_VALID_KINDS = ("ltoir", "llvm_ir", "cpp_source")
+
+
+@dataclass(frozen=True)
+class DeviceCode:
+    """A compiled-or-source device-code blob ready to hand to ``Op``.
+
+    Args:
+        op_bytes: the raw blob (LTO-IR, LLVM bitcode, or C++ source bytes).
+        kind: one of ``"ltoir"`` (default), ``"llvm_ir"``, ``"cpp_source"``;
+            tells the backend how to interpret ``op_bytes``.
+
+    For most uses you don't construct ``DeviceCode`` directly — the internal
+    JIT-compile helpers return one, and the iterator/algorithm machinery
+    forwards them. Construct explicitly when feeding a ``RawOp`` from outside
+    the default pipeline.
+    """
+
+    op_bytes: bytes
+    kind: str = "ltoir"
+
+    def __post_init__(self):
+        if self.kind not in _VALID_KINDS:
+            raise ValueError(
+                f"DeviceCode.kind must be one of {_VALID_KINDS!r}; got {self.kind!r}"
+            )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_jit.py b/cccl_upstream/python/cuda_cccl/cuda/compute/_jit.py
new file mode 100644
index 00000000..bf620e8d
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_jit.py
@@ -0,0 +1,1053 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+import ast
+import functools
+import inspect
+import operator
+import struct
+import textwrap
+from types import new_class
+from typing import TYPE_CHECKING, Callable, Hashable, List, Tuple
+
+import numba
+import numba.cuda
+import numba.np.numpy_support
+import numba.types
+import numpy as np
+from numba import types
+from numba.core import cgutils
+from numba.core.datamodel import models
+from numba.core.extending import (
+    as_numba_type,
+    make_attribute_wrapper,
+    overload,
+    register_model,
+    typeof_impl,
+)
+from numba.core.typeconv import Conversion
+from numba.core.typing import signature as nb_signature
+from numba.core.typing.templates import ConcreteTemplate
+from numba.cuda.cudadecl import registry as cuda_registry
+from numba.extending import lower_builtin, lower_cast
+
+from . import types as cccl_types
+from ._bindings import Op, OpKind
+from ._caching import (
+    CachableFunction,
+    _make_cache_key_from_args,
+    _process_wide_cache_registry,
+    cache_with_registered_key_functions,
+)
+
+try:
+    from ._build_info import USING_V2  # type: ignore[import-not-found]
+except ImportError:
+    USING_V2 = False
+
+from ._odr_helpers import create_stateful_op_void_ptr_wrapper
+from ._utils import sanitize_identifier
+from ._utils.protocols import (
+    get_data_pointer,
+    get_dtype,
+    is_contiguous,
+    is_device_array,
+)
+from .op import OpAdapter
+
+if TYPE_CHECKING:
+    from .typing import DeviceArrayLike
+
+
+def _compile_op_to_llvm_bitcode(wrapped_op, wrapper_sig) -> bytes:
+    """Compile a Numba device op to LLVM bitcode (.bc) bytes.
+
+    Used on the v2 (HostJIT) backend, which prefers LLVM bitcode over NVRTC
+    LTO-IR — the JIT linker routes "BC"-magic blobs through LLVM's native
+    bitcode linker instead of nvJitLink's LTO codegen.
+
+    Numba's public ``cuda.compile`` only emits PTX or LTO-IR. To get LLVM IR
+    with the C-ABI wrapper (the form CUB's PTX references by name), we go one
+    layer deeper to ``_compile_pyfunc_with_fixup`` with ``abi="c"`` and pull
+    the LLVM string off the code library before NVVM lowering to PTX.
+    """
+    import os
+    import re
+
+    import llvmlite.binding as llvm
+    from numba.cuda.compiler import _compile_pyfunc_with_fixup
+
+    target_name = wrapped_op.__name__
+    lib, _ = _compile_pyfunc_with_fixup(
+        wrapped_op,
+        wrapper_sig,
+        device=True,
+        abi="c",
+        abi_info={"abi_name": target_name},
+        lto=False,
+    )
+    text_ir = lib.get_llvm_str()
+
+    debug_dir = os.environ.get("CCCL_JIT_DEBUG")
+    if debug_dir:
+        os.makedirs(debug_dir, exist_ok=True)
+        with open(os.path.join(debug_dir, f"{target_name}.raw.ll"), "w") as f:
+            f.write(text_ir)
+
+    # get_llvm_str joins all modules in the library with "\n\n". Split on
+    # ModuleID markers so each chunk parses standalone, then link them.
+    parts = [p for p in re.split(r"(?m)^(?=; ModuleID = )", text_ir) if p.strip()]
+    if not parts:
+        parts = [text_ir]
+
+    # Strip Numba's `target datalayout = ...` line — llvmlite ships with an
+    # older NVVM layout (`e-p:64:64:64-...`) that doesn't match the modern
+    # CUDA layout (`e-p6:32:32-...`) emitted by hostjit's Clang. Linking
+    # modules with mismatched layouts triggers LLVM warnings and can lead to
+    # miscompiles. Removing the line lets LLVM default to the target triple's
+    # canonical layout, which agrees with Clang.
+    parts = [re.sub(r"(?m)^target datalayout =.*\n", "", p) for p in parts]
+
+    modules = []
+    for i, part in enumerate(parts):
+        try:
+            m = llvm.parse_assembly(part)
+            m.verify()
+        except Exception as exc:
+            raise RuntimeError(
+                f"Failed to parse LLVM IR module {i} for '{target_name}': {exc}"
+            ) from exc
+        modules.append(m)
+
+    main = modules[0]
+    for m in modules[1:]:
+        main.link_in(m, preserve=True)
+
+    if debug_dir:
+        with open(os.path.join(debug_dir, f"{target_name}.merged.ll"), "w") as f:
+            f.write(str(main))
+        with open(os.path.join(debug_dir, f"{target_name}.symbols.txt"), "w") as f:
+            f.write(f"target_name={target_name}\n")
+            for fn in main.functions:
+                f.write(
+                    f"  {fn.linkage} {'decl' if fn.is_declaration else 'def '} {fn.name}\n"
+                )
+
+    return bytes(main.as_bitcode())
+
+
+# -----------------------------------------------------------------------------
+# Struct registration and casting
+# -----------------------------------------------------------------------------
+
+
+# Base class for all struct types, used for struct-to-struct cast matching.
+class _StructBase(numba.types.Type):
+    """Base class for all CCCL GPU struct types."""
+
+    _field_spec: dict  # Mapping of field names to Numba types
+
+
+# The struct registration logic is isolated here to avoid polluting other
+# modules with Numba-specific type plumbing.
+@functools.lru_cache(maxsize=256)
+def _make_struct_type(struct_class_or_name, field_names, field_types):
+    """
+    Core factory function that uses the Numba extension machinery to
+    create a struct type with pass-by-value semantics.
+
+    This function is called lazily from _register_struct_with_numba.
+
+    Args:
+        struct_class_or_name: Either an existing struct class (from gpu_struct) or a string name
+        field_names: Tuple of field names
+        field_types: Tuple of TypeDescriptors
+    """
+    from .struct import _is_struct_type, _Struct
+
+    # If we're given an existing class, use it; otherwise create a new one
+    if isinstance(struct_class_or_name, type):
+        struct_class = struct_class_or_name
+    else:
+        struct_class = new_class(struct_class_or_name, bases=(_Struct,))
+
+    # Convert TypeDescriptors to Numba types
+    numba_field_types = [type_descriptor_to_numba(ft) for ft in field_types]
+
+    raw_field_spec = dict(zip(field_names, numba_field_types))
+    assert all(
+        _is_struct_type(tp) or isinstance(tp, types.Type)
+        for tp in raw_field_spec.values()
+    )
+
+    field_spec = {
+        name: as_numba_type(typ) if _is_struct_type(typ) else typ
+        for name, typ in raw_field_spec.items()
+    }
+
+    # Only set _field_spec if it doesn't already exist (for new classes created here)
+    # For classes from gpu_struct, they already have _field_spec with TypeDescriptors
+    if not hasattr(struct_class, "_field_spec"):
+        struct_class._field_spec = raw_field_spec
+
+    class StructType(_StructBase):
+        def __init__(self):
+            super().__init__(name=struct_class.__name__)
+            # Store field_spec for struct-to-struct conversion checks
+            self._field_spec = field_spec
+
+        def can_convert_from(self, typingctx, other):
+            if isinstance(other, types.UniTuple):
+                tuple_size = other.count
+                if tuple_size == len(field_types):
+                    return Conversion.safe
+
+            elif isinstance(other, types.Tuple):
+                tuple_size = len(other.types)
+                if tuple_size == len(field_types):
+                    all_compatible = all(
+                        typingctx.can_convert(src_type, tgt_type) is not None
+                        for src_type, tgt_type in zip(other.types, field_spec.values())
+                    )
+                    if all_compatible:
+                        return Conversion.safe
+
+            # Allow conversion from another StructType with identical field layout
+            elif hasattr(other, "_field_spec"):
+                other_field_spec = other._field_spec
+                # Check if field count matches
+                if len(other_field_spec) == len(field_spec):
+                    # Check if all field types are compatible (by position, not name)
+                    other_field_types = list(other_field_spec.values())
+                    self_field_types = list(field_spec.values())
+                    all_compatible = all(
+                        typingctx.can_convert(src_type, tgt_type) is not None
+                        for src_type, tgt_type in zip(
+                            other_field_types, self_field_types
+                        )
+                    )
+                    if all_compatible:
+                        return Conversion.safe
+
+            return None
+
+    numba_type = StructType()
+    numba_type.python_type = struct_class
+
+    as_numba_type.register(struct_class, numba_type)
+
+    @typeof_impl.register(struct_class)
+    def typeof_struct(val, c):
+        return numba_type  # Must return the SAME instance, not a new StructType()
+
+    @register_model(StructType)
+    class StructModel(models.StructModel):
+        def __init__(self, dmm, fe_type):
+            members = [(name, typ) for name, typ in field_spec.items()]
+            super().__init__(dmm, fe_type, members)
+
+    for field_name in field_spec:
+        make_attribute_wrapper(StructType, field_name, field_name)
+
+    field_names_list = list(field_spec.keys())
+
+    # Validate that all field names are valid Python identifiers before
+    # we exec any generated code that accesses them:
+    for name in field_names_list:
+        if not name.isidentifier():
+            raise ValueError(
+                f"Struct field name {name!r} is not a valid Python identifier"
+            )
+
+    @overload(operator.getitem)
+    def struct_getitem(struct_val, idx):
+        if not isinstance(struct_val, StructType):
+            return
+
+        if isinstance(idx, (types.IntegerLiteral)):
+            idx_val = getattr(idx, "literal_value", getattr(idx, "value", None))
+
+            if idx_val is None or not (0 <= idx_val < len(field_names_list)):
+
+                def error_impl(struct_val, idx):
+                    raise IndexError(
+                        f"Index out of range for struct with {len(field_names_list)} fields"
+                    )
+
+                return error_impl
+
+            field_name = field_names_list[idx_val]
+            exec(
+                f"def impl(struct_val, idx): return struct_val.{field_name}",
+                namespace := {},
+            )
+            return namespace["impl"]
+
+        conditions = "\n".join(
+            f"    {'if' if i == 0 else 'elif'} idx == {i}: return struct_val.{name}"
+            for i, name in enumerate(field_names_list)
+        )
+        exec(
+            f"def impl(struct_val, idx):\n{conditions}\n    else: raise IndexError('Index out of range')",
+            namespace := {},
+        )
+        return namespace["impl"]
+
+    @cuda_registry.register
+    class StructConstructor(ConcreteTemplate):
+        key = struct_class
+        cases = [nb_signature(numba_type, *list(field_spec.values()))]
+
+    cuda_registry.register_global(struct_class, numba.types.Function(StructConstructor))
+
+    def struct_constructor(context, builder, sig, args):
+        ty = sig.return_type
+        retval = cgutils.create_struct_proxy(ty)(context, builder)
+        for field_name, val in zip(field_spec.keys(), args):
+            setattr(retval, field_name, val)
+        return retval._getvalue()
+
+    lower_builtin(struct_class, *list(field_spec.values()))(struct_constructor)
+
+    @lower_cast(types.BaseTuple, StructType)
+    def tuple_to_struct_cast(context, builder, fromty, toty, val):
+        if isinstance(fromty, types.UniTuple):
+            tuple_size = fromty.count
+            element_types = [fromty.dtype] * tuple_size
+        elif isinstance(fromty, types.Tuple):
+            tuple_size = len(fromty.types)
+            element_types = list(fromty.types)
+        else:
+            tuple_size = len(field_spec)
+            element_types = list(field_spec.values())
+
+        if tuple_size != len(field_spec):
+            raise ValueError(
+                f"Cannot cast tuple of size {tuple_size} to {struct_class.__name__} "
+                f"with {len(field_types)} fields"
+            )
+
+        retval = cgutils.create_struct_proxy(toty)(context, builder)
+
+        for i, (field_name, target_type) in enumerate(field_spec.items()):
+            element = builder.extract_value(val, i)
+
+            source_type = element_types[i]
+            if source_type != target_type:
+                element = context.cast(builder, element, source_type, target_type)
+
+            setattr(retval, field_name, element)
+
+        return retval._getvalue()
+
+    @lower_cast(types.Tuple, StructType)
+    @lower_cast(types.UniTuple, StructType)
+    def cast_tuple_to_struct(context, builder, fromty, toty, val):
+        if isinstance(fromty, types.UniTuple):
+            if fromty.count != len(field_spec):
+                return None
+            tuple_types = [fromty.dtype] * fromty.count
+        else:
+            if len(fromty.types) != len(field_spec):
+                return None
+            tuple_types = list(fromty.types)
+
+        struct_val = cgutils.create_struct_proxy(toty)(context, builder)
+        for i, (field_name, field_type) in enumerate(field_spec.items()):
+            elem = builder.extract_value(val, i)
+            elem = context.cast(builder, elem, tuple_types[i], field_type)
+            setattr(struct_val, field_name, elem)
+
+        return struct_val._getvalue()
+
+    @lower_cast(_StructBase, StructType)
+    def cast_struct_to_struct(context, builder, fromty, toty, val):
+        """Cast from one CCCL struct type to another with identical layout."""
+        # Get field specs from both types
+        from_field_spec = fromty._field_spec
+        to_field_spec = toty._field_spec
+
+        if len(from_field_spec) != len(to_field_spec):
+            return None
+
+        from_field_types = list(from_field_spec.values())
+        from_field_names = list(from_field_spec.keys())
+        to_field_types = list(to_field_spec.values())
+        to_field_names = list(to_field_spec.keys())
+
+        # Create struct proxy for source value
+        from_struct = cgutils.create_struct_proxy(fromty)(context, builder, value=val)
+
+        # Create struct proxy for target value
+        to_struct = cgutils.create_struct_proxy(toty)(context, builder)
+
+        # Copy and cast each field by position
+        for i, (to_name, to_type) in enumerate(zip(to_field_names, to_field_types)):
+            from_name = from_field_names[i]
+            from_type = from_field_types[i]
+
+            # Get the field value from source struct
+            elem = getattr(from_struct, from_name)
+
+            # Cast if types differ
+            if from_type != to_type:
+                elem = context.cast(builder, elem, from_type, to_type)
+
+            # Set the field in target struct
+            setattr(to_struct, to_name, elem)
+
+        return to_struct._getvalue()
+
+    return struct_class
+
+
+@functools.lru_cache(maxsize=256)
+def _register_struct_with_numba(struct_class):
+    field_spec = struct_class._type_descriptor.fields
+
+    registered_class = _make_struct_type(
+        struct_class,
+        tuple(field_spec.keys()),
+        tuple(field_spec.values()),
+    )
+
+    return as_numba_type(registered_class)
+
+
+# -----------------------------------------------------------------------------
+# TypeDescriptor <-> Numba conversions
+# -----------------------------------------------------------------------------
+
+
+@functools.cache
+def type_descriptor_to_numba(td):
+    """
+    Convert a TypeDescriptor to a Numba type.
+
+    Handles:
+    - PointerTypeDescriptor: creates CPointer to the pointee's numba type
+    - StructTypeDescriptor: registers a struct class for the layout
+    - POD TypeDescriptor: uses numba.from_dtype
+    - Numba types: pass through
+    """
+
+    # Pass through if already a Numba type
+    if isinstance(td, numba.types.Type):
+        return td
+
+    # Handle PointerTypeDescriptor (must check before TypeDescriptor since it's a subclass)
+    if isinstance(td, cccl_types.PointerTypeDescriptor):
+        return types.CPointer(type_descriptor_to_numba(td.pointee))
+
+    # Handle TypeDescriptor (includes StructTypeDescriptor)
+    if isinstance(td, cccl_types.TypeDescriptor):
+        return _convert_type_descriptor_to_numba(td)
+
+    raise TypeError(f"Expected TypeDescriptor or numba type, got {type(td)}")
+
+
+def _convert_type_descriptor_to_numba(td):
+    """Internal helper to convert TypeDescriptor to Numba type."""
+
+    # For struct types
+    if isinstance(td, cccl_types.StructTypeDescriptor):
+        from .struct import (
+            _get_struct_record_dtype,
+            _get_struct_type_descriptor,
+            _Struct,
+        )
+
+        layout_key = tuple(td.layout_key())
+        struct_class = new_class(td.name, bases=(_Struct,))
+        struct_class._field_spec = dict(layout_key)
+        struct_class._type_descriptor = _get_struct_type_descriptor(struct_class)
+        struct_class.dtype = _get_struct_record_dtype(struct_class)
+        try:
+            return as_numba_type(struct_class)
+        except numba.core.errors.NumbaError:
+            return _register_struct_with_numba(struct_class)
+
+    # For POD types
+    return numba.from_dtype(td.dtype)
+
+
+def _is_gpu_struct_class(obj):
+    """Check if an object is a gpu_struct class (not instance)."""
+    return (
+        isinstance(obj, type)
+        and hasattr(obj, "_type_descriptor")
+        and hasattr(obj, "_field_spec")
+    )
+
+
+def _iter_function_objects(py_func):
+    if hasattr(py_func, "__globals__"):
+        yield from py_func.__globals__.values()
+
+    if py_func.__closure__ is not None:
+        for cell in py_func.__closure__:
+            try:
+                yield cell.cell_contents
+            except ValueError:
+                pass
+
+
+def _ensure_function_structs_registered(py_func):
+    """
+    Scan a function's globals and closure for gpu_struct classes and ensure
+    they're registered with Numba before compilation.
+    """
+
+    def _register_if_needed(struct_class):
+        try:
+            return as_numba_type(struct_class)
+        except numba.core.errors.NumbaError:
+            return _register_struct_with_numba(struct_class)
+
+    for value in _iter_function_objects(py_func):
+        if _is_gpu_struct_class(value):
+            _register_if_needed(value)
+
+
+def _numba_type_to_type_descriptor(numba_type):
+    """Convert a Numba type to a TypeDescriptor (internal helper)."""
+    from .struct import _is_struct_type
+
+    # Already a TypeDescriptor
+    if isinstance(numba_type, cccl_types.TypeDescriptor):
+        return numba_type
+
+    # Custom StructType with an associated gpu_struct Python class
+    if hasattr(numba_type, "python_type") and _is_struct_type(numba_type.python_type):
+        return numba_type.python_type._type_descriptor
+
+    # POD type - convert via numpy dtype
+    dtype = numba.np.numpy_support.as_dtype(numba_type)
+    return cccl_types.from_numpy_dtype(dtype)
+
+
+# Return-type inference is architecture-independent — it depends only on the
+# function and its input types — so it must NOT go through
+# ``cache_with_registered_key_functions``, whose key is salted with
+# ``Device().compute_capability`` (that query requires a visible GPU). Otherwise
+# merely *constructing* an unannotated ``TransformIterator`` on a GPU-free build
+# machine would fail, before any ``compute_capability=<cc>`` build argument can
+# take effect. Use a plain cache keyed the same way (minus the device salt).
+_infer_return_type_cache: dict = {}
+
+
+def _infer_return_type(py_func, input_types):
+    key = _make_cache_key_from_args(py_func, input_types)
+    if key not in _infer_return_type_cache:
+        _infer_return_type_cache[key] = _infer_return_type_impl(py_func, input_types)
+    return _infer_return_type_cache[key]
+
+
+_infer_return_type.cache_clear = _infer_return_type_cache.clear  # type: ignore[attr-defined]
+# Keep clear_all_caches() covering this cache too.
+_process_wide_cache_registry["_jit._infer_return_type"] = _infer_return_type
+
+
+def _infer_return_type_impl(py_func, input_types):
+    # Ensure any gpu_struct classes referenced in the function are registered
+    _ensure_function_structs_registered(py_func)
+
+    # Compile to infer return type
+    from ._utils import sanitize_identifier
+
+    sanitized_name = sanitize_identifier(py_func.__name__)
+    unique_suffix = hex(id(py_func))[2:]
+    abi_name = f"{sanitized_name}_{unique_suffix}"
+    input_numba_types = tuple(type_descriptor_to_numba(t) for t in input_types)
+    _, return_type = numba.cuda.compile(
+        py_func, input_numba_types, abi_info={"abi_name": abi_name}
+    )
+    return _numba_type_to_type_descriptor(return_type)
+
+
+# -----------------------------------------------------------------------------
+# Stateless ops
+# -----------------------------------------------------------------------------
+
+
+@functools.lru_cache(maxsize=256)
+def _compile_op_impl(cachable_op, input_types_tuple: tuple, output_type, cc=None):
+    """Cached implementation of op compilation.
+
+    Args:
+        cachable_op: CachableFunction wrapper around the operator
+        input_types_tuple: Tuple of input TypeDescriptors
+        output_type: Output TypeDescriptor
+        cc: Target compute capability ``(major, minor)`` for the LTO-IR, or None
+            to use Numba's configured default PTX compute capability
+            (``config.CUDA_DEFAULT_PTX_CC``). Part of the cache key so the same
+            operator compiled for different arches does not collide.
+    """
+    from ._bindings import Op, OpKind
+    from ._odr_helpers import create_op_void_ptr_wrapper
+
+    # Extract the actual function from CachableFunction
+    op = cachable_op._func
+
+    # Ensure any gpu_struct classes referenced in the op are registered
+    _ensure_function_structs_registered(op)
+
+    numba_input_types = tuple(
+        type_descriptor_to_numba(t) if isinstance(t, cccl_types.TypeDescriptor) else t
+        for t in input_types_tuple
+    )
+
+    if isinstance(output_type, cccl_types.TypeDescriptor):
+        numba_output_type = type_descriptor_to_numba(output_type)
+    else:
+        numba_output_type = output_type
+
+    sig = numba_output_type(*numba_input_types)
+    wrapped_op, wrapper_sig = create_op_void_ptr_wrapper(op, sig)
+
+    from ._device_code import DeviceCode
+
+    if USING_V2:
+        code = DeviceCode(
+            op_bytes=_compile_op_to_llvm_bitcode(wrapped_op, wrapper_sig),
+            kind="llvm_ir",
+        )
+    else:
+        ltoir, _ = numba.cuda.compile(
+            wrapped_op, sig=wrapper_sig, output="ltoir", cc=cc
+        )
+        code = DeviceCode(op_bytes=ltoir, kind="ltoir")
+
+    return Op(
+        operator_type=OpKind.STATELESS,
+        name=wrapped_op.__name__,
+        ltoir=code,
+        state_alignment=1,
+        state=None,
+    )
+
+
+def compile_op(op, input_types, output_type=None):
+    """Compile a user-provided binary operator for use with CCCL algorithms.
+
+    This function is cached to ensure that identical operators with identical
+    types produce the same compiled result (same symbol names and LTOIR),
+    which allows proper deduplication during linking.
+    """
+    from ._caching import CachableFunction
+
+    cachable_op = CachableFunction(op)
+    from ._target_cc import get_target_cc
+
+    return _compile_op_impl(
+        cachable_op, tuple(input_types), output_type, get_target_cc()
+    )
+
+
+class _StatelessOp(OpAdapter):
+    """Adapter for stateless callables."""
+
+    __slots__ = ()
+
+    def __init__(self, func):
+        self._func = func
+        self._cachable = CachableFunction(func)
+
+    def compile(self, input_types, output_type=None) -> Op:
+        return compile_op(self._func, input_types, output_type)
+
+    @property
+    def func(self) -> Callable:
+        """Access the wrapped callable."""
+        return self._func
+
+    def __eq__(self, other):
+        return self._cachable == other._cachable
+
+    def __hash__(self):
+        return hash(
+            self._cachable,
+        )
+
+    def get_return_type(self, input_types):
+        return _infer_return_type(self._func, input_types)
+
+
+# -----------------------------------------------------------------------------
+# Stateful ops
+# -----------------------------------------------------------------------------
+#
+# Stateful ops are callables that capture device arrays (state) as globals or
+# closure variables.
+#
+# When numba-cuda encounters a device function that references device
+# arrays in its globals/closures, it captures pointers to the
+# referenced arrays as constants.  If an object being referenced
+# changes (e.g., common in a loop) the device function will be
+# recompiled because it sees a new pointer.
+#
+# We avoid this by relying on support for stateful ops in the CCCL C
+# library itself. Here's how this works
+#
+# 1. Detect any global/closure device arrays referenced by a given callable.
+#
+# 2. Transform the callable's AST to add parameters for each detected device array.
+#
+# 3. Compilation: the compiled device function whose LTOIR we
+#    eventually pass to the CCCL C library must take a single `void*`
+#    argument for is state.  The corresponding declaration of stateful
+#    operators in CCCL C looks like:
+#
+#       extern "C" __device__ void foo(void* state, )`.
+#
+#    Thus, we have Thus, at compilation time, we
+#    define some intrinsics to unpack a `void*` into typed arrays and
+#    invoke the transformed callable from step 2.  Much of this is
+#    implemented in _odr_helpers.py
+
+
+def _detect_device_array_globals(func: Callable) -> List[Tuple[str, object]]:
+    """
+    Detect device arrays referenced as globals in a function.
+
+    Args:
+        func: The function to inspect
+
+    Returns:
+        List of (name, array) tuples for detected device arrays
+    """
+    state_arrays = []
+    code = func.__code__
+
+    for name in code.co_names:
+        val = func.__globals__.get(name)
+        if val is not None and is_device_array(val):
+            state_arrays.append((name, val))
+
+    return state_arrays
+
+
+def _detect_device_array_closures(func: Callable) -> List[Tuple[str, object]]:
+    """
+    Detect device arrays captured in function closures.
+
+    Args:
+        func: The function to inspect
+
+    Returns:
+        List of (name, array) tuples for detected device arrays
+    """
+    state_arrays: List[Tuple[str, object]] = []
+    code = func.__code__
+    closure = func.__closure__
+
+    if closure is None:
+        return state_arrays
+
+    # co_freevars contains the names of closure variables
+    for name, cell in zip(code.co_freevars, closure):
+        try:
+            val = cell.cell_contents
+            if is_device_array(val):
+                state_arrays.append((name, val))
+        except ValueError:
+            # Cell is empty
+            pass
+
+    return state_arrays
+
+
+def _detect_all_device_arrays(func: Callable) -> List[Tuple[str, object]]:
+    """
+    Detect all device arrays referenced by a function (globals + closures).
+
+    Args:
+        func: The function to inspect
+
+    Returns:
+        List of (name, array) tuples for detected device arrays
+    """
+    globals_arrays = _detect_device_array_globals(func)
+    closure_arrays = _detect_device_array_closures(func)
+    return globals_arrays + closure_arrays
+
+
+class _AddStateParameters(ast.NodeTransformer):
+    """AST transformer that adds state parameters to a function definition."""
+
+    def __init__(self, state_names: List[str]):
+        self.state_names = state_names
+
+    def visit_FunctionDef(self, node: ast.FunctionDef) -> ast.FunctionDef:
+        # Prepend state parameters to the function arguments
+        # Inner function signature: (state_arrays..., regular_args...)
+        new_args = [ast.arg(arg=name, annotation=None) for name in self.state_names]
+        node.args.args = new_args + node.args.args
+        return node
+
+    def visit_AsyncFunctionDef(
+        self, node: ast.AsyncFunctionDef
+    ) -> ast.AsyncFunctionDef:
+        # Handle async functions the same way
+        new_args = [ast.arg(arg=name, annotation=None) for name in self.state_names]
+        node.args.args = new_args + node.args.args
+        return node
+
+
+def _transform_function_ast(func: Callable, state_names: List[str]) -> Callable:
+    """
+    Transform a function to add state arrays captured as globals or closures
+    as explicit parameters.
+
+    For example, if the function is:
+
+        def func(x): return x + state[0]  # state is a global array
+
+    Then the transformed function will be:
+
+        def func(state, x): return x + state[0]
+
+    Args:
+        func: The original function
+        state_names: Names of device arrays to add as parameters
+
+    Returns:
+        A new function with state arrays as explicit parameters,
+        appearing after the regular parameters.
+
+    Raises:
+        ValueError: If the function source cannot be obtained
+    """
+    # Get source code
+    try:
+        source = inspect.getsource(func)
+    except (OSError, TypeError) as e:
+        raise ValueError(
+            f"Cannot get source code for function '{func.__name__}'. "
+        ) from e
+
+    # Dedent source (in case function is defined inside a class/function)
+    source = textwrap.dedent(source)
+
+    # Parse to AST
+    tree = ast.parse(source)
+
+    # Transform: add state parameters
+    transformer = _AddStateParameters(state_names)
+    tree = transformer.visit(tree)
+    ast.fix_missing_locations(tree)
+
+    # Compile and execute to create new function
+    # We need to provide the original function's globals for imports
+    # AND inject closure variables so they're accessible in the new function
+    globals_dict = func.__globals__.copy()
+
+    # Inject closure variables (except state arrays which become parameters)
+    if func.__closure__:
+        for name, cell in zip(func.__code__.co_freevars, func.__closure__):
+            if name not in state_names:  # Don't inject state arrays
+                try:
+                    globals_dict[name] = cell.cell_contents
+                except ValueError:
+                    pass  # Cell is empty
+
+    local_ns: dict[str, Callable] = {}
+    exec(
+        compile(tree, filename=f"<auto_stateful:{func.__name__}>", mode="exec"),
+        globals_dict,
+        local_ns,
+    )
+
+    # Get the transformed function
+    transformed_func = local_ns[func.__name__]
+
+    return transformed_func
+
+
+def _extract_state(func: Callable):
+    # Detect device arrays
+    state_info = _detect_all_device_arrays(func)
+
+    # Extract names and arrays
+    state_names = [name for name, _ in state_info]
+    state_arrays: List[DeviceArrayLike] = [arr for _, arr in state_info]  # type: ignore[misc]
+
+    return state_names, state_arrays
+
+
+def _compile_stateful_op(op, input_types, state_arrays, output_type=None):
+    """
+    Compile a stateful operator for use with CCCL algorithms.
+
+    Args:
+        op: The operator function (already transformed to take state as explicit parameters)
+        input_types: Tuple of TypeDescriptors for regular input arguments
+        state_arrays: Tuple of device arrays containing state
+        output_type: Optional TypeDescriptor for return value (inferred if None)
+
+    Returns:
+        Compiled Op object for C++ interop
+    """
+
+    # Ensure any gpu_struct classes referenced in the op are registered
+    _ensure_function_structs_registered(op)
+
+    # Validate all state arrays are contiguous
+    for i, state_array in enumerate(state_arrays):
+        if not is_contiguous(state_array):
+            raise ValueError(f"state array {i} must be contiguous")
+
+    # Convert input types to Numba types
+    numba_input_types = tuple(type_descriptor_to_numba(t) for t in input_types)
+
+    # Create Numba array types for state arrays
+    state_array_types = [
+        numba.types.Array(numba.from_dtype(get_dtype(s)), 1, "A") for s in state_arrays
+    ]
+
+    # Infer output type if needed
+    if output_type is None:
+        # Compile with Numba to infer return type
+        # The transformed function expects (state_arrays..., regular_args...)
+        all_numba_input_types = tuple(state_array_types) + numba_input_types
+        sanitized_name = sanitize_identifier(op.__name__)
+        unique_suffix = hex(id(op))[2:]
+        abi_name = f"{sanitized_name}_{unique_suffix}"
+        _, return_type = numba.cuda.compile(
+            op, all_numba_input_types, abi_info={"abi_name": abi_name}
+        )
+        # Convert return type to TypeDescriptor
+        output_type = cccl_types.from_numpy_dtype(
+            numba.np.numpy_support.as_dtype(return_type)
+        )
+
+    # Convert output type to Numba type
+    numba_output_type = type_descriptor_to_numba(output_type)
+
+    # Build full signature: output_type(state_arrays..., regular_args...)
+    sig = numba_output_type(*state_array_types, *numba_input_types)
+
+    # Get state pointers - pointers to the device array data
+    state_ptrs = [get_data_pointer(arr) for arr in state_arrays]
+
+    # Get shape and itemsize from each state array
+    state_info = []
+    for state_array in state_arrays:
+        state_info.append(
+            {
+                "shape": len(state_array),
+                "itemsize": get_dtype(state_array).itemsize,
+                "strides": get_dtype(state_array).itemsize,
+            }
+        )
+
+    # All pointers have the same alignment, use pointer-sized int alignment
+    state_alignment = np.dtype(np.intp).alignment
+
+    # Create the stateful wrapper (constructs arrays from pointers)
+    wrapped_op, wrapper_sig = create_stateful_op_void_ptr_wrapper(
+        op, sig, state_array_types, state_info
+    )
+
+    # Compile the wrapper — LLVM bitcode for v2 (HostJIT), LTO-IR for v1 (NVRTC).
+    from ._device_code import DeviceCode
+
+    if USING_V2:
+        code = DeviceCode(
+            op_bytes=_compile_op_to_llvm_bitcode(wrapped_op, wrapper_sig),
+            kind="llvm_ir",
+        )
+    else:
+        from ._target_cc import get_target_cc
+
+        ltoir, _ = numba.cuda.compile(
+            wrapped_op, sig=wrapper_sig, output="ltoir", cc=get_target_cc()
+        )
+        code = DeviceCode(op_bytes=ltoir, kind="ltoir")
+
+    # Pack all data pointers as bytes (sequentially)
+    state_bytes = struct.pack(f"{len(state_ptrs)}P", *state_ptrs)
+
+    # Return Op with STATEFUL kind and packed pointers
+    return Op(
+        operator_type=OpKind.STATEFUL,
+        name=wrapped_op.__name__,
+        ltoir=code,
+        state_alignment=state_alignment,
+        state=state_bytes,
+    )
+
+
+class _JitOpState:
+    def __init__(self, names: List[str], arrays: List[DeviceArrayLike]):
+        self.names = names
+        self.arrays = arrays
+
+    def get_cache_key(self) -> Hashable:
+        return (tuple(self.names), tuple(get_dtype(s) for s in self.arrays))
+
+    def to_bytes(self):
+        state_ptrs = [get_data_pointer(arr) for arr in self.arrays]
+        return struct.pack(f"{len(state_ptrs)}P", *state_ptrs)
+
+
+class _StatefulOp(OpAdapter):
+    __slots__ = "_state"
+
+    def __init__(self, func, state):
+        self._func = func
+        self._cachable = CachableFunction(func)
+        self._state = state
+
+    def get_state(self):
+        return self._state.to_bytes()
+
+    def compile(self, input_types, output_type=None) -> Op:
+        transformed_func = _transform_function_ast(self._func, self._state.names)
+        return _compile_stateful_op(
+            transformed_func, input_types, self._state.arrays, output_type
+        )
+
+    def get_cache_key(self):
+        return (self._func.__name__, self._cachable, self._state.get_cache_key())
+
+    @property
+    def is_stateful(self) -> bool:
+        return True
+
+    @property
+    def func(self) -> Callable:
+        """Access the wrapped callable."""
+        return self._func
+
+    def __hash__(self) -> int:
+        return hash(self.get_cache_key())
+
+    def __eq__(self, other):
+        return (self._cachable == other._cachable) and (self._state == other._state)
+
+
+def to_jit_op_adapter(op: Callable) -> OpAdapter:
+    """
+    Convert the Python callable into the appropriate stateful or stateless
+    OpAdapter, depending on whether or not it references any global state.
+    """
+    state_names, state_arrays = _extract_state(op)
+    if state_names:
+        return _StatefulOp(op, _JitOpState(state_names, state_arrays))
+    else:
+        return _StatelessOp(op)
+
+
+cache_with_registered_key_functions.register(_StatelessOp, lambda op: op._cachable)
+
+cache_with_registered_key_functions.register(_StatefulOp, lambda op: op.get_cache_key())
+
+
+__all__ = [
+    "to_jit_op_adapter",
+]
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_nvtx.py b/cccl_upstream/python/cuda_cccl/cuda/compute/_nvtx.py
new file mode 100644
index 00000000..e69de29b
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_odr_helpers.py b/cccl_upstream/python/cuda_cccl/cuda/compute/_odr_helpers.py
new file mode 100644
index 00000000..8be67036
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_odr_helpers.py
@@ -0,0 +1,370 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+"""
+ODR (One Definition Rule) Helpers for CCCL Python Interop.
+
+This module provides utilities to create wrapper functions for
+device functions that are defined in Python and JIT compiled by Numba.
+
+On the C++ side, these functions are declared as `extern "C"` functions with
+void* parameters - the arguments types can not be known at C++ compile time.
+
+Thus, the helpers in this module generate wrapper device functions that accept
+void* arguments (matching C++ declarations), cast them to the correct
+typed arguments, load/store values as needed, and call the original
+function with properly typed arguments.
+
+Example flow:
+    User provides: def add(x: int32, y: int32) -> int32
+    Wrapper signature: void(void*, void*, void*)  # x_ptr, y_ptr, result_ptr
+    C++ sees: extern "C" void wrapped_add(void*, void*, void*);
+"""
+
+from __future__ import annotations
+
+import enum
+import itertools
+import textwrap
+import threading
+from typing import TYPE_CHECKING
+
+from numba import types
+from numba.core.extending import intrinsic
+
+from ._utils import sanitize_identifier
+
+if TYPE_CHECKING:
+    from numba.core.typing import Signature
+
+# Global counter to generate unique symbol names even when the same function
+# is used multiple times (e.g., as both selectors in `three_way_partition`).
+_wrapper_name_counter = itertools.count()
+_wrapper_name_lock = threading.Lock()
+
+__all__ = [
+    "create_op_void_ptr_wrapper",
+    "create_stateful_op_void_ptr_wrapper",
+    "create_advance_void_ptr_wrapper",
+    "create_input_dereference_void_ptr_wrapper",
+    "create_output_dereference_void_ptr_wrapper",
+]
+
+
+class _ArgMode(enum.Enum):
+    """How a void* argument should be handled in wrapper codegen."""
+
+    LOAD = "load"  # Cast to typed pointer, load value
+    PTR = "ptr"  # Cast to typed pointer, pass pointer directly
+    STORE = "store"  # Cast to typed pointer, store return value here
+    # Unpack packed data pointers into array structs
+    STATE = "state"
+
+
+class _ArgSpec:
+    """Specification for a wrapper argument."""
+
+    __slots__ = ("numba_type", "mode")
+
+    def __init__(self, numba_type, mode: _ArgMode):
+        self.numba_type = numba_type
+        self.mode = mode
+
+
+def _build_numba_array_struct(context, builder, array_type, data_ptr, info):
+    """Build a numba array struct from a data pointer and array info.
+
+    Args:
+        context: Numba codegen context
+        builder: LLVM IR builder
+        array_type: Numba Array type for the array
+        data_ptr: LLVM value for the data pointer
+        info: Dict with 'shape', 'itemsize', 'strides' for the array
+
+    Returns:
+        LLVM value representing the array struct
+    """
+    import llvmlite.ir as ir
+    from numba.cuda.np.arrayobj import make_array, populate_array
+
+    out_ary = make_array(array_type)(context, builder)
+
+    populate_array(
+        out_ary,
+        data=data_ptr,
+        shape=[ir.Constant(ir.IntType(64), info["shape"])],
+        strides=[ir.Constant(ir.IntType(64), info["strides"])],
+        itemsize=info["itemsize"],
+        meminfo=None,
+    )
+
+    return out_ary._getvalue()
+
+
+def _unpack_state_arrays(context, builder, packed_ptr, type_info_pairs):
+    """Unpack packed data pointers into numba array structs.
+
+    Args:
+        context: Numba codegen context
+        builder: LLVM IR builder
+        packed_ptr: void* pointing to an array of data pointers
+        type_info_pairs: List of (array_type, info) tuples
+
+    Returns:
+        List of LLVM values representing the unpacked array structs
+    """
+    import llvmlite.ir as ir
+
+    # Cast void* to pointer-to-pointer (array of pointers)
+    ptr_type = ir.IntType(64).as_pointer()
+    base_ptr = builder.bitcast(packed_ptr, ptr_type.as_pointer())
+
+    result = []
+    for j, (array_type, info) in enumerate(type_info_pairs):
+        # Load j-th pointer from the array and cast to correct type
+        elem_ptr = builder.gep(base_ptr, [ir.Constant(ir.IntType(32), j)])
+        dtype_llvm = context.get_value_type(array_type.dtype)
+        typed_ptr_ptr = builder.bitcast(elem_ptr, dtype_llvm.as_pointer().as_pointer())
+        data_ptr = builder.load(typed_ptr_ptr)
+
+        # Build array struct from pointer
+        array_val = _build_numba_array_struct(
+            context, builder, array_type, data_ptr, info
+        )
+        result.append(array_val)
+
+    return result
+
+
+def _codegen_void_ptr_wrapper(
+    context, builder, args, arg_specs, func_device, inner_sig
+):
+    """Generate LLVM IR for a void* wrapper function.
+
+    This is the codegen implementation shared by all void* wrappers.
+    It processes each argument according to its _ArgSpec mode, calls
+    the inner function, and stores the result if needed.
+
+    Args:
+        context: Numba codegen context
+        builder: LLVM IR builder
+        args: LLVM values for the void* arguments
+        arg_specs: List of _ArgSpec describing each argument
+        func_device: The device function to call
+        inner_sig: Numba signature for the inner function
+
+    Returns:
+        LLVM dummy value (for void return)
+    """
+
+    input_vals = []
+    state_array_vals = []
+    ret_ptr = None
+
+    for i, (arg, spec) in enumerate(zip(args, arg_specs)):
+        match spec.mode:
+            case _ArgMode.LOAD:
+                # Cast void* to typed pointer and load value
+                llvm_type = context.get_value_type(spec.numba_type)
+                typed_ptr = builder.bitcast(arg, llvm_type.as_pointer())
+                val = builder.load(typed_ptr)
+                input_vals.append(val)
+            case _ArgMode.PTR:
+                # Cast void* to typed pointer, pass pointer directly
+                llvm_type = context.get_value_type(spec.numba_type.dtype)
+                typed_ptr = builder.bitcast(arg, llvm_type.as_pointer())
+                input_vals.append(typed_ptr)
+            case _ArgMode.STORE:
+                # Cast void* to typed pointer for storing result
+                llvm_type = context.get_value_type(spec.numba_type)
+                ret_ptr = builder.bitcast(arg, llvm_type.as_pointer())
+            case _ArgMode.STATE:
+                # Cast void* to a packed array of pointers and unpack them
+                array_vals = _unpack_state_arrays(
+                    context, builder, arg, spec.numba_type
+                )
+                state_array_vals.extend(array_vals)
+            case _:
+                raise ValueError(f"Invalid arg mode: {spec.mode}")
+
+    # Prepend state arrays at the beginning (inner_sig expects state args first)
+    input_vals = state_array_vals + input_vals
+
+    # Call the inner function
+    cres = context.compile_subroutine(builder, func_device, inner_sig, caching=False)
+    result = context.call_internal(builder, cres.fndesc, inner_sig, input_vals)
+
+    # Store result if needed
+    if ret_ptr is not None:
+        builder.store(result, ret_ptr)
+
+    return context.get_dummy_value()
+
+
+def _create_void_ptr_wrapper(
+    func, name: str, arg_specs: list[_ArgSpec], inner_sig: "Signature"
+):
+    """
+    Given a function and a list of _ArgSpec, create a wrapper function
+    that takes all void* arguments, bitcasts them to the
+    appropriate typed pointers, and calls the inner function with
+    the typed arguments. Each void* argument is handled according
+    to its _ArgSpec.
+
+    Args:
+        func: The function to wrap (will be compiled as device function)
+        name: Base name for the wrapper function
+        arg_specs: List of _ArgSpec describing each void* argument
+        inner_sig: Numba signature for the inner function call
+
+    Returns:
+        Tuple of (wrapper_func, wrapper_sig)
+    """
+    from numba.cuda import jit as cuda_jit
+
+    # Wrap function as device function
+    func_device = cuda_jit(device=True)(func)
+
+    # Generate argument names and signature
+    arg_names = [f"arg_{i}" for i in range(len(arg_specs))]
+    arg_str = ", ".join(arg_names)
+    void_sig = types.void(*(types.voidptr for _ in arg_specs))
+
+    # Create unique wrapper name using global counter
+    sanitized_name = sanitize_identifier(name)
+    if not sanitized_name.isidentifier():
+        raise ValueError(
+            f"Function name '{name}' cannot be sanitized into a valid identifier"
+        )
+
+    for arg_name in arg_names:
+        if not arg_name.isidentifier():
+            raise ValueError(
+                f"Invalid argument name '{arg_name}' - must be a valid identifier"
+            )
+    with _wrapper_name_lock:
+        unique_suffix = next(_wrapper_name_counter)
+    wrapper_name = f"wrapped_{sanitized_name}_{unique_suffix}"
+
+    # We need exec() here because Numba's @intrinsic decorator requires:
+    # 1. A function with a specific signature visible at parse time
+    # 2. The number of arguments must match the wrapper signature
+    # The actual codegen logic is in _codegen_void_ptr_wrapper - this just
+    # creates the minimal intrinsic shell that delegates to it.
+    wrapper_src = textwrap.dedent(f"""
+    @intrinsic
+    def impl(typingctx, {arg_str}):
+        def codegen(context, builder, impl_sig, args):
+            return codegen_helper(context, builder, args, arg_specs, func_device, inner_sig)
+        return void_sig, codegen
+
+    def {wrapper_name}({arg_str}):
+        return impl({arg_str})
+    """)
+
+    local_dict = {
+        "intrinsic": intrinsic,
+        "void_sig": void_sig,
+        "arg_specs": arg_specs,
+        "func_device": func_device,
+        "inner_sig": inner_sig,
+        "codegen_helper": _codegen_void_ptr_wrapper,
+    }
+    exec(wrapper_src, {}, local_dict)
+
+    wrapper_func = local_dict[wrapper_name]
+    wrapper_func.__globals__.update(local_dict)
+
+    return wrapper_func, void_sig
+
+
+def create_op_void_ptr_wrapper(op, sig: "Signature"):
+    """Creates a wrapper function for user-defined operators like unary or binary operators.
+
+    The wrapper takes N+1 arguments where N is the number of input arguments to `op`, the last
+    argument is a pointer to the result.
+    """
+    arg_specs = [_ArgSpec(t, _ArgMode.LOAD) for t in sig.args]
+    arg_specs.append(_ArgSpec(sig.return_type, _ArgMode.STORE))
+    return _create_void_ptr_wrapper(op, op.__name__, arg_specs, sig)
+
+
+def create_stateful_op_void_ptr_wrapper(
+    op, sig: "Signature", state_array_types, state_info
+):
+    """Creates a wrapper function for a stateful operator with void* arguments.
+
+    The wrapper takes N+2 void* arguments:
+    - states_ptr: pointer to packed array of data pointers for state arrays
+    - N input args: one for each regular input argument
+    - result: pointer where result is stored
+
+    Args:
+        op: The user's callable operator
+        sig: The signature of the operator (state_array1, state_array2, ..., regular_arg1, regular_arg2, ...) -> return_type
+        state_array_types: List/tuple of numba Array types for the state parameters
+        state_info: List/tuple of dicts with 'shape', 'itemsize', 'strides' for each state array
+
+    Returns:
+        Tuple of (wrapper_func, wrapper_sig)
+    """
+    num_states = len(state_array_types)
+
+    # Build arg_specs: states_ptr + regular inputs + result
+    # The packed state arrays spec goes first, then regular LOAD args, then STORE for result
+    # numba_type is a list of (array_type, info) tuples
+    type_info_pairs = list(zip(state_array_types, state_info))
+    arg_specs = [_ArgSpec(type_info_pairs, _ArgMode.STATE)]
+    for i in range(num_states, len(sig.args)):
+        arg_specs.append(_ArgSpec(sig.args[i], _ArgMode.LOAD))
+    arg_specs.append(_ArgSpec(sig.return_type, _ArgMode.STORE))
+
+    return _create_void_ptr_wrapper(op, op.__name__, arg_specs, sig)
+
+
+def create_advance_void_ptr_wrapper(advance_fn, state_ptr_type):
+    """Creates a wrapper function for iterator advance method.
+
+    The wrapper takes 2 void* arguments:
+    - state pointer
+    - offset pointer (points to uint64 value)
+    """
+    arg_specs = [
+        _ArgSpec(state_ptr_type, _ArgMode.PTR),
+        _ArgSpec(types.uint64, _ArgMode.LOAD),  # uint64 is the offset type
+    ]
+    inner_sig = types.void(state_ptr_type, types.uint64)
+    return _create_void_ptr_wrapper(
+        advance_fn, advance_fn.__name__, arg_specs, inner_sig
+    )
+
+
+def create_input_dereference_void_ptr_wrapper(deref_fn, state_ptr_type, value_type):
+    """Creates a wrapper function for input iterator dereference method.
+
+    The wrapper takes 2 void* arguments:
+    - state pointer
+    - result pointer (function writes result here)
+    """
+    arg_specs = [
+        _ArgSpec(state_ptr_type, _ArgMode.PTR),
+        _ArgSpec(types.CPointer(value_type), _ArgMode.PTR),
+    ]
+    inner_sig = types.void(state_ptr_type, types.CPointer(value_type))
+    return _create_void_ptr_wrapper(deref_fn, deref_fn.__name__, arg_specs, inner_sig)
+
+
+def create_output_dereference_void_ptr_wrapper(deref_fn, state_ptr_type, value_type):
+    """Creates a wrapper function for output iterator dereference method.
+
+    The wrapper takes 2 void* arguments:
+    - state pointer
+    - value pointer (value to write)
+    """
+    arg_specs = [
+        _ArgSpec(state_ptr_type, _ArgMode.PTR),
+        _ArgSpec(value_type, _ArgMode.LOAD),
+    ]
+    inner_sig = types.void(state_ptr_type, value_type)
+    return _create_void_ptr_wrapper(deref_fn, deref_fn.__name__, arg_specs, inner_sig)
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_proxy.py b/cccl_upstream/python/cuda_cccl/cuda/compute/_proxy.py
new file mode 100644
index 00000000..4f498dde
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_proxy.py
@@ -0,0 +1,132 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Device-less placeholders for ahead-of-time (no-GPU) compilation.
+
+``ProxyArray`` / ``ProxyValue`` describe *only* the dtype (and, for arrays,
+shape/contiguity) of an argument, with no backing GPU allocation. Pass them to
+``make_<algo>(...)`` together with ``compute_capability=`` to compile an
+algorithm on a machine that has no GPU (or no live data), then ``serialize()``
+the result. The real device arrays / scalars are supplied later at ``__call__``.
+
+Accessing a proxy's data pointer raises ``RuntimeError`` — a proxy can be used
+to *build* an algorithm but never to *run* one.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+
+_PROXY_DATA_ERROR = (
+    "ProxyArray has no GPU data — it is a build-time placeholder only. "
+    "Pass a real device array when calling the compiled algorithm."
+)
+
+_PROXY_VALUE_DATA_ERROR = (
+    "ProxyValue has no data — it is a build-time placeholder only. "
+    "Pass a real scalar or numpy array when calling the compiled algorithm."
+)
+
+
+class _ProxyCAI(dict):
+    """CAI dict whose 'data' key raises on access."""
+
+    def __missing__(self, key):
+        if key == "data":
+            raise RuntimeError(_PROXY_DATA_ERROR)
+        raise KeyError(key)
+
+    def get(self, key, default=None):
+        # dict.get() bypasses __missing__, so guard it too: a consumer that
+        # defensively does cai.get("data") must still hit the loud failure
+        # rather than silently receiving a null pointer.
+        if key == "data":
+            raise RuntimeError(_PROXY_DATA_ERROR)
+        return super().get(key, default)
+
+
+class ProxyArray:
+    """Dtype-only placeholder for a device array.
+
+    Use in place of a real device array when calling ``make_<algo>()`` to
+    trigger ahead-of-time compilation without allocating GPU memory — for
+    example, on a build machine that has no GPU or no live data.
+
+    Satisfies the ``DeviceArrayLike`` protocol:
+
+    * ``is_device_array(proxy)`` -> ``True``
+    * ``get_dtype(proxy)``       -> the dtype supplied at construction
+    * ``get_data_pointer(proxy)``-> raises ``RuntimeError``
+    * ``is_contiguous(proxy)``   -> ``True``
+
+    Accessing the data pointer raises ``RuntimeError``; passing a
+    ``ProxyArray`` to a compiled algorithm's ``__call__`` is not supported.
+
+    Example::
+
+        from cuda.compute import ProxyArray, make_reduce_into, OpKind
+        import numpy as np
+
+        reducer = make_reduce_into(
+            d_in=ProxyArray(np.float32),
+            d_out=ProxyArray(np.float32),
+            op=OpKind.PLUS,
+            h_init=np.zeros(1, dtype=np.float32),
+            compute_capability=[80, 90],
+        )
+        reducer.serialize()
+    """
+
+    __slots__ = ("_dtype",)
+
+    def __init__(self, dtype):
+        self._dtype = np.dtype(dtype)
+
+    @property
+    def dtype(self) -> np.dtype:
+        return self._dtype
+
+    @property
+    def __cuda_array_interface__(self) -> dict:
+        return _ProxyCAI(
+            {
+                "shape": (1,),
+                "typestr": self._dtype.str,
+                "version": 3,
+                "strides": None,  # C-contiguous
+                # "data" is intentionally absent — accessing it raises RuntimeError
+            }
+        )
+
+    def __repr__(self) -> str:
+        return f"ProxyArray(dtype={self._dtype})"
+
+
+class ProxyValue:
+    """Dtype-only placeholder for a scalar / initial-value argument.
+
+    Use in place of a real numpy scalar or array when calling ``make_<algo>()``
+    to trigger ahead-of-time compilation without real data — for example, for
+    the ``h_init`` argument of :func:`~cuda.compute.make_reduce_into`.
+
+    Accessing the data of a ``ProxyValue`` raises ``RuntimeError``; passing
+    one to a compiled algorithm's ``__call__`` is not supported.
+    """
+
+    __slots__ = ("_dtype",)
+
+    def __init__(self, dtype):
+        self._dtype = np.dtype(dtype)
+
+    @property
+    def dtype(self) -> np.dtype:
+        return self._dtype
+
+    def __repr__(self) -> str:
+        return f"ProxyValue(dtype={self._dtype})"
+
+
+def is_proxy(obj) -> bool:
+    """True if *obj* is a build-time placeholder (ProxyArray/ProxyValue)."""
+    return isinstance(obj, (ProxyArray, ProxyValue))
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_serialization/__init__.py b/cccl_upstream/python/cuda_cccl/cuda/compute/_serialization/__init__.py
new file mode 100644
index 00000000..cd8b98b3
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_serialization/__init__.py
@@ -0,0 +1,41 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Serialization for cuda.compute algorithms."""
+
+from __future__ import annotations
+
+from .dispatch import deserialize as deserialize
+from .dispatch import serialize as serialize
+from .serializable import BOOL as BOOL
+from .serializable import BUILD_RESULT as BUILD_RESULT
+from .serializable import BUILD_RESULTS as BUILD_RESULTS
+from .serializable import CONDITIONAL as CONDITIONAL
+from .serializable import ENUM as ENUM
+from .serializable import ITER as ITER
+from .serializable import NESTED as NESTED
+from .serializable import OP as OP
+from .serializable import U8 as U8
+from .serializable import U32 as U32
+from .serializable import U64 as U64
+from .serializable import VALUE as VALUE
+from .serializable import Serializable as Serializable
+
+__all__ = [
+    "serialize",
+    "deserialize",
+    "Serializable",
+    "ITER",
+    "OP",
+    "VALUE",
+    "U8",
+    "U32",
+    "U64",
+    "BOOL",
+    "ENUM",
+    "CONDITIONAL",
+    "BUILD_RESULT",
+    "BUILD_RESULTS",
+    "NESTED",
+]
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_serialization/codec.py b/cccl_upstream/python/cuda_cccl/cuda/compute/_serialization/codec.py
new file mode 100644
index 00000000..3bb35973
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_serialization/codec.py
@@ -0,0 +1,217 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Byte serialization of CCCL descriptors (``cccl_op_t``, ``cccl_iterator_t``, etc.)."""
+
+from __future__ import annotations
+
+import struct
+
+import numpy as np
+
+from cuda.cccl import __version__ as _PKG_VERSION
+
+from .._bindings import Iterator, IteratorKind, Op, OpKind, TypeEnum, TypeInfo, Value
+from .._device_code import DeviceCode
+
+# An opaque 8-byte marker identifying a cuda.compute serialization blob.
+_MAGIC = b"CCCLPYS1"
+
+
+class Writer:
+    """Append-only little-endian byte buffer."""
+
+    def __init__(self) -> None:
+        self.buf = bytearray()
+
+    def u8(self, v: int) -> None:
+        self.buf += struct.pack("<B", v)
+
+    def u32(self, v: int) -> None:
+        self.buf += struct.pack("<I", v)
+
+    def u64(self, v: int) -> None:
+        self.buf += struct.pack("<Q", v)
+
+    def blob(self, b: bytes) -> None:
+        self.u64(len(b))
+        self.buf += b
+
+    def text(self, s: str) -> None:
+        self.blob(s.encode("utf-8"))
+
+    def getvalue(self) -> bytes:
+        return bytes(self.buf)
+
+
+class Reader:
+    """Bounds-checked little-endian reader over a bytes blob."""
+
+    def __init__(self, data: bytes) -> None:
+        self._data = memoryview(data)
+        self.pos = 0
+
+    def _take(self, n: int) -> memoryview:
+        end = self.pos + n
+        if end > len(self._data):
+            raise ValueError("serialization descriptor blob truncated")
+        out = self._data[self.pos : end]
+        self.pos = end
+        return out
+
+    def u8(self) -> int:
+        return struct.unpack("<B", self._take(1))[0]
+
+    def u32(self) -> int:
+        return struct.unpack("<I", self._take(4))[0]
+
+    def u64(self) -> int:
+        return struct.unpack("<Q", self._take(8))[0]
+
+    def blob(self) -> bytes:
+        return bytes(self._take(self.u64()))
+
+    def text(self) -> str:
+        return self.blob().decode("utf-8")
+
+    def remaining(self) -> bytes:
+        """Bytes after the descriptor region: the C build_result blob."""
+        return bytes(self._data[self.pos :])
+
+
+# --- framing -----------------------------------------------------------------
+
+
+def _check_header(r: Reader) -> None:
+    """Validate the magic and package-version stamp at the start of a blob."""
+    if bytes(r._take(len(_MAGIC))) != _MAGIC:
+        raise ValueError(
+            "serialization blob: bad magic (not a cuda.compute serialization blob)"
+        )
+    version = r.text()
+    if version != _PKG_VERSION:
+        raise ValueError(
+            "serialization blob: cuda-cccl version mismatch "
+            f"(blob={version!r}, current={_PKG_VERSION!r}); "
+            "re-serialize with this version of cuda-cccl"
+        )
+
+
+def begin(algo_tag: str) -> Writer:
+    """Start a descriptor sidecar with the magic/version/algo header.
+
+    ``algo_tag`` is the algorithm class's ``__qualname__`` (e.g. ``"_Reduce"``).
+    """
+    w = Writer()
+    w.buf += _MAGIC
+    w.text(_PKG_VERSION)
+    w.text(algo_tag)
+    return w
+
+
+def open(blob: bytes, expected_algo: str) -> Reader:
+    """Validate the header and return a reader positioned at the first field."""
+    r = Reader(blob)
+    _check_header(r)
+    algo = r.text()
+    if algo != expected_algo:
+        raise ValueError(
+            f"serialization blob: wrong algorithm (blob tag={algo!r}, expected={expected_algo!r})"
+        )
+    return r
+
+
+def peek_algo(blob: bytes) -> str:
+    """Return the algorithm tag (class ``__qualname__``) from a blob header
+    without consuming the blob.
+
+    Validates magic + version. Used by the generic ``deserialize`` dispatcher to
+    pick the right algorithm reconstructor.
+    """
+    r = Reader(blob)
+    _check_header(r)
+    return r.text()
+
+
+# --- descriptor (de)serialization --------------------------------------------
+
+
+def write_type_info(w: Writer, ti: TypeInfo) -> None:
+    w.u64(ti.size)
+    w.u64(ti.alignment)
+    w.u32(int(ti.typenum))
+
+
+def read_type_info(r: Reader) -> TypeInfo:
+    size = r.u64()
+    alignment = r.u64()
+    type_enum = r.u32()
+    return TypeInfo(size, alignment, TypeEnum(type_enum))
+
+
+def write_op(w: Writer, op: Op) -> None:
+    # Serialize the operator's device code in full so reconstruction needs no
+    # JIT; only per-call op state is omitted.
+    w.u32(int(op.operator_type))
+    w.text(op.name)
+    w.blob(op.ltoir)
+    w.text(op.code.kind)
+    w.u32(op.state_alignment)
+    # State size is structural: it fixes op_data.size at construction. The state
+    # bytes themselves are bound per-call.
+    w.u64(len(op.state))
+    extras = op.extra_code
+    w.u32(len(extras))
+    for dc in extras:
+        w.blob(dc.op_bytes)
+        w.text(dc.kind)
+
+
+def read_op(r: Reader) -> Op:
+    operator_type = OpKind(r.u32())
+    name = r.text()
+    code = r.blob()
+    code_kind = r.text()
+    state_alignment = r.u32()
+    state_size = r.u64()
+    n_extra = r.u32()
+    extras = [DeviceCode(op_bytes=r.blob(), kind=r.text()) for _ in range(n_extra)]
+    return Op(
+        name=name,
+        operator_type=operator_type,
+        ltoir=DeviceCode(op_bytes=code, kind=code_kind),
+        state=bytes(state_size),  # zero placeholder; real bytes bound per-call
+        state_alignment=state_alignment,
+        extra_ltoirs=extras,
+    )
+
+
+def write_iterator(w: Writer, it: Iterator) -> None:
+    w.u8(1 if it.is_kind_pointer() else 0)
+    w.u32(it.alignment)
+    write_type_info(w, it.value_type)
+    write_op(w, it.advance_op)
+    write_op(w, it.dereference_or_assign_op)
+
+
+def read_iterator(r: Reader) -> Iterator:
+    kind = IteratorKind.POINTER if r.u8() else IteratorKind.ITERATOR
+    alignment = r.u32()
+    value_type = read_type_info(r)
+    advance = read_op(r)
+    deref = read_op(r)
+    # state is bound per-call (set_cccl_iterator_state); start with none.
+    return Iterator(alignment, kind, advance, deref, value_type, state=None)
+
+
+def write_value(w: Writer, val: Value) -> None:
+    # Only the type is static; the value bytes are bound per-call.
+    write_type_info(w, val.type)
+
+
+def read_value(r: Reader) -> Value:
+    value_type = read_type_info(r)
+    # Placeholder state sized to the value type; __call__ rebinds the real bytes.
+    placeholder = np.zeros(value_type.size, dtype=np.uint8)
+    return Value(value_type, placeholder)
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_serialization/dispatch.py b/cccl_upstream/python/cuda_cccl/cuda/compute/_serialization/dispatch.py
new file mode 100644
index 00000000..c1b93811
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_serialization/dispatch.py
@@ -0,0 +1,60 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Public, free-standing serialize/deserialize entry points."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from . import codec
+from .serializable import Serializable
+
+
+def serialize(algorithm: Any) -> bytes:
+    """Serialize a built algorithm into a blob of bytes.
+
+    Args:
+        algorithm: An object returned by a ``make_*`` factory (e.g.
+            :func:`make_reduce_into`, :func:`make_exclusive_scan`).
+
+    Returns:
+        A versioned, self-describing byte blob. Reconstruct it with
+        :func:`deserialize`; no objects required at load time.
+    """
+    if not callable(getattr(type(algorithm), "serialize", None)):
+        raise TypeError(
+            f"{type(algorithm).__name__} is not a serializable algorithm "
+            "(expected an object from a make_* factory)."
+        )
+    return algorithm.serialize()
+
+
+def deserialize(blob: bytes):
+    """Reconstruct a built algorithm from a blob produced by :func:`serialize`.
+
+    Warning:
+        The returned object is **not safe to use from multiple threads
+        concurrently**. Do not deserialize once and share the object across
+        threads: every call writes its arguments (array pointers, sizes,
+        operator and initial-value state) into the object before launching, so
+        overlapping calls can launch kernels with another thread's arguments —
+        silently wrong results or CUDA errors, with no exception raised at the
+        point of misuse. Unlike the ``make_*`` factories, which hand each
+        calling thread its own cached object, ``deserialize`` returns a fresh
+        uncached object with no per-thread protection. For concurrent use,
+        call :func:`deserialize` in each thread — reconstruction performs no
+        recompilation, so per-thread deserialization from one shared blob is
+        cheap. One thread at a time (for example, handing the object between
+        threads with proper ordering) is fine.
+
+    Raises:
+        ValueError: if the blob is malformed or its algorithm tag is unknown.
+    """
+    tag = codec.peek_algo(blob)
+    try:
+        cls = Serializable._registry[tag]
+    except KeyError:
+        raise ValueError(f"serialization blob: unknown algorithm tag {tag!r}") from None
+    return cls.deserialize(blob)
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_serialization/serializable.py b/cccl_upstream/python/cuda_cccl/cuda/compute/_serialization/serializable.py
new file mode 100644
index 00000000..e2bd7cfd
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_serialization/serializable.py
@@ -0,0 +1,285 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Schema-driven serialize/deserialize base for cuda.compute algorithms.
+
+A built-algorithm class declares a ``__serialization_schema__`` listing its
+serialized members as ``(attr_name, kind)`` pairs, including its ``build_result``
+as a ``BUILD_RESULT(<type>)`` member. ``Serializable`` provides generic
+``serialize``/``deserialize`` that walk the schema, so subclasses need no
+hand-written codec and both directions share one field order. Subclasses
+auto-register by their ``__qualname__`` for the free-function
+``deserialize`` dispatcher.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Callable, TypeVar
+
+from . import codec
+
+
+class _Kind:
+    """A serialized member kind: writes/reads one value.
+
+    ``obj`` is the instance being (de)serialized; most kinds ignore it, but
+    ``CONDITIONAL`` uses it to read a selector member deserialized earlier.
+    """
+
+    __slots__ = ()
+
+    def write(self, w: codec.Writer, value: Any, obj: Any) -> None:
+        raise NotImplementedError
+
+    def read(self, r: codec.Reader, obj: Any) -> Any:
+        raise NotImplementedError
+
+
+class _Descriptor(_Kind):
+    """Iterator / Op / Value descriptor, delegating to the codec codec."""
+
+    __slots__ = ("_write", "_read")
+
+    def __init__(self, writer: Callable, reader: Callable) -> None:
+        self._write = writer
+        self._read = reader
+
+    def write(self, w: codec.Writer, value: Any, obj: Any) -> None:
+        self._write(w, value)
+
+    def read(self, r: codec.Reader, obj: Any) -> Any:
+        return self._read(r)
+
+
+ITER = _Descriptor(codec.write_iterator, codec.read_iterator)
+OP = _Descriptor(codec.write_op, codec.read_op)
+VALUE = _Descriptor(codec.write_value, codec.read_value)
+
+
+class _Scalar(_Kind):
+    """A fixed-width little-endian unsigned integer (u8/u32/u64)."""
+
+    __slots__ = ("width",)
+
+    def __init__(self, width: int) -> None:
+        self.width = width
+
+    def write(self, w: codec.Writer, value: Any, obj: Any) -> None:
+        {1: w.u8, 4: w.u32, 8: w.u64}[self.width](int(value))
+
+    def read(self, r: codec.Reader, obj: Any) -> int:
+        return {1: r.u8, 4: r.u32, 8: r.u64}[self.width]()
+
+
+U8, U32, U64 = _Scalar(1), _Scalar(4), _Scalar(8)
+
+
+class _Bool(_Kind):
+    """A boolean, stored as a u8 (0/1)."""
+
+    __slots__ = ()
+
+    def write(self, w: codec.Writer, value: Any, obj: Any) -> None:
+        w.u8(1 if value else 0)
+
+    def read(self, r: codec.Reader, obj: Any) -> bool:
+        return bool(r.u8())
+
+
+BOOL = _Bool()
+
+
+class _Enum(_Kind):
+    """An IntEnum member, stored as a u8 and reconstructed as the enum type."""
+
+    __slots__ = ("enum_cls",)
+
+    def __init__(self, enum_cls: Any) -> None:
+        self.enum_cls = enum_cls
+
+    def write(self, w: codec.Writer, value: Any, obj: Any) -> None:
+        w.u8(int(value))
+
+    def read(self, r: codec.Reader, obj: Any) -> Any:
+        return self.enum_cls(r.u8())
+
+
+def ENUM(enum_cls: Any) -> _Enum:
+    """Schema kind for a u8-backed IntEnum member."""
+    return _Enum(enum_cls)
+
+
+class _SubObject(_Kind):
+    """A sub-object with its own ``serialize()``/``deserialize()``, carried as a
+    length-prefixed blob. Used for the C ``build_result`` and for a nested
+    ``Serializable`` (e.g. select wrapping three_way_partition)."""
+
+    __slots__ = ("cls",)
+
+    def __init__(self, cls: Any) -> None:
+        self.cls = cls
+
+    def write(self, w: codec.Writer, value: Any, obj: Any) -> None:
+        w.blob(value.serialize())
+
+    def read(self, r: codec.Reader, obj: Any) -> Any:
+        return self.cls.deserialize(r.blob())
+
+
+def BUILD_RESULT(cls: type) -> _SubObject:
+    """Schema kind for an algorithm's ``Device<Algo>BuildResult`` member."""
+    return _SubObject(cls)
+
+
+class _BuildResults(_Kind):
+    """A ``{cc: Device<Algo>BuildResult}`` mapping — one compiled build result per
+    target compute capability.
+
+    Wire form: ``u32`` count, then for each entry a ``u32`` cc key
+    (``cc_major * 10 + cc_minor``) followed by the length-prefixed build_result
+    blob. Entries are written in sorted-key order so the encoding is
+    deterministic. On read, each build_result is deserialized *without* loading
+    (``load=False``); the matching build result is loaded lazily on first call, so a
+    multi-arch artifact stays portable across GPUs and needs no live device to
+    deserialize.
+    """
+
+    __slots__ = ("cls",)
+
+    def __init__(self, cls: Any) -> None:
+        self.cls = cls
+
+    def write(self, w: codec.Writer, value: Any, obj: Any) -> None:
+        from .._caching import _PerCCBuildResults
+
+        # Wrappers always hold a _PerCCBuildResults (build_for_ccs and
+        # read() below both produce one). serialize_build_result takes the
+        # per-cc source lock, so serialization cannot observe a source whose
+        # first device load is still in progress; a plain dict here would
+        # silently bypass that lock.
+        assert isinstance(value, _PerCCBuildResults)
+        ccs = sorted(value)
+        w.u32(len(ccs))
+        for cc in ccs:
+            w.u32(int(cc))
+            w.blob(value.serialize_build_result(cc))
+
+    def read(self, r: codec.Reader, obj: Any) -> Any:
+        count = r.u32()
+        # A single-target blob must match this device, so validate its cc-major
+        # eagerly (clear error at deserialize). A multi-arch blob legitimately
+        # carries build results for other archs, so defer the cc check — resolve_build_result
+        # picks the matching one at call time. Kernel load stays lazy either way.
+        check_cc = count == 1
+        entries = [(r.u32(), r.blob()) for _ in range(count)]
+        result: dict[int, Any] = {}
+        for cc, blob in entries:
+            if cc in result:
+                raise ValueError(
+                    f"duplicate compute-capability key {cc} in build_results blob"
+                )
+            result[cc] = self.cls.deserialize(blob, load=False, check_cc=check_cc)
+
+        from .._caching import _PerCCBuildResults
+
+        return _PerCCBuildResults(result)
+
+
+def BUILD_RESULTS(cls: type) -> _BuildResults:
+    """Schema kind for a ``{cc: Device<Algo>BuildResult}`` build result mapping."""
+    return _BuildResults(cls)
+
+
+def NESTED(cls: type) -> _SubObject:
+    """Schema kind for a nested ``Serializable`` member (its blob is embedded)."""
+    return _SubObject(cls)
+
+
+class _Conditional(_Kind):
+    """A member whose kind depends on an earlier member's value.
+
+    ``selector`` names a member deserialized *before* this one; ``branches``
+    maps each possible selector value to the kind to use (or ``None`` for an
+    absent member that (de)serializes to ``None``).
+    """
+
+    __slots__ = ("selector", "branches")
+
+    def __init__(self, selector: str, branches: dict) -> None:
+        self.selector = selector
+        self.branches = branches
+
+    def _kind(self, obj: Any) -> "_Kind | None":
+        return self.branches[getattr(obj, self.selector)]
+
+    def write(self, w: codec.Writer, value: Any, obj: Any) -> None:
+        kind = self._kind(obj)
+        if kind is not None:
+            kind.write(w, value, obj)
+
+    def read(self, r: codec.Reader, obj: Any) -> Any:
+        kind = self._kind(obj)
+        return None if kind is None else kind.read(r, obj)
+
+
+def CONDITIONAL(selector: str, branches: dict) -> _Conditional:
+    """Schema kind for a member whose kind is chosen by ``selector``'s value."""
+    return _Conditional(selector, branches)
+
+
+_S = TypeVar("_S", bound="Serializable")
+
+
+class Serializable:
+    """Mixin providing schema-driven serialize/deserialize + registration."""
+
+    __slots__ = ()
+
+    # __qualname__ -> subclass, populated as algorithm modules are imported.
+    _registry: dict[str, type[Serializable]] = {}
+
+    # Subclasses declare their serialized members here.
+    __serialization_schema__: tuple = ()
+
+    # Construction-time binding of a default-build wrapper's loaded result
+    # (see cache_build_results). Annotation only: storage comes from each
+    # subclass's __slots__; __init__ or deserialize() below assigns it.
+    _bound_build_result: Any
+
+    def __init_subclass__(cls, **kwargs: Any) -> None:
+        super().__init_subclass__(**kwargs)
+        Serializable._registry[cls.__qualname__] = cls
+
+    def _after_deserialize(self) -> None:
+        """Hook to bind derived, non-serialized state after schema members are read.
+
+        Called once at the end of ``deserialize``. Subclasses that keep a cached
+        attribute derived from serialized members (and set it in ``__init__``)
+        override this to rebind it; the default is a no-op.
+        """
+
+    def serialize(self) -> bytes:
+        """Serialize this built algorithm to a self-contained serialization blob."""
+        w = codec.begin(type(self).__qualname__)
+        for attr, kind in self.__serialization_schema__:
+            kind.write(w, getattr(self, attr), self)
+        return w.getvalue()
+
+    @classmethod
+    def deserialize(cls: type[_S], blob: bytes) -> _S:
+        """Reconstruct a built algorithm from a blob; no objects required.
+
+        Members are read in schema order and set on the instance as they are
+        read, so a ``CONDITIONAL`` member can consult a selector read earlier.
+        """
+        r = codec.open(blob, cls.__qualname__)
+        obj = cls.__new__(cls)
+        # deserialize() bypasses __init__, which is where default-build
+        # wrappers bind their loaded result (see cache_build_results); an
+        # unbound wrapper resolves per call instead.
+        obj._bound_build_result = None
+        for attr, kind in cls.__serialization_schema__:
+            setattr(obj, attr, kind.read(r, obj))
+        obj._after_deserialize()
+        return obj
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_target_cc.py b/cccl_upstream/python/cuda_cccl/cuda/compute/_target_cc.py
new file mode 100644
index 00000000..a20157ae
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_target_cc.py
@@ -0,0 +1,58 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Target compute-capability context for device-code (LTO-IR) compilation.
+
+Operators (numba) and iterators (NVRTC C++) are compiled to LTO-IR once and
+linked into every per-arch build result of an algorithm. nvJitLink requires the final
+target SM to be **at least as new** as every linked LTO/PTX input's arch. So for
+a multi-arch build, this shared device code must be compiled for the **lowest**
+target arch; otherwise linking a (say) sm_89 operator into an sm_80 cubin fails
+with ``nvJitLink error``.
+
+This module holds a context-local "target cc" that the leaf compilers
+(``_jit`` for operators, ``_cpp_compile`` for iterators) consult. It is set
+around a build by the caching decorator that wraps every ``make_<algo>``.
+``None`` means "use the current device" — the default single-target behavior,
+unchanged.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import contextvars
+
+# (major, minor) tuple, or None to mean "current device default".
+_target_cc: contextvars.ContextVar = contextvars.ContextVar(
+    "cccl_target_cc", default=None
+)
+
+
+def get_target_cc():
+    """The current build's target cc as ``(major, minor)``, or ``None``.
+
+    ``None`` means device code should target the current device (the default).
+    """
+    return _target_cc.get()
+
+
+@contextlib.contextmanager
+def target_cc(compute_capability):
+    """Set the shared-device-code target cc for the duration of a build.
+
+    ``compute_capability`` is the ``make_<algo>`` argument (``None`` / int /
+    ``(major, minor)`` / list). For a multi-arch build the shared operator /
+    iterator LTO-IR is compiled for the **lowest** requested arch so it links
+    into every build result. ``None`` leaves the current-device default in place.
+    """
+    from ._cccl_interop import normalize_compute_capabilities
+
+    ccs = normalize_compute_capabilities(compute_capability)
+    # normalized list is sorted ascending, so ccs[0] is the minimum target.
+    cc = ccs[0] if ccs else None
+    token = _target_cc.set(cc)
+    try:
+        yield
+    finally:
+        _target_cc.reset(token)
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_utils/__init__.py b/cccl_upstream/python/cuda_cccl/cuda/compute/_utils/__init__.py
new file mode 100644
index 00000000..51a9aba3
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_utils/__init__.py
@@ -0,0 +1,22 @@
+from __future__ import annotations
+
+import re
+
+__all__ = ["sanitize_identifier"]
+
+
+def sanitize_identifier(name: str) -> str:
+    """Sanitize a name to be a valid Python/LLVM identifier.
+
+    This replaces any character that isn't alphanumeric or underscore with
+    an underscore. This is needed because:
+    - Lambda functions have __name__ = "<lambda>" which contains angle brackets
+    - Python identifiers and LLVM/NVVM global names don't allow special characters
+
+    Args:
+        name: The name to sanitize (e.g., function __name__)
+
+    Returns:
+        A sanitized name safe for use as a Python identifier or LLVM symbol
+    """
+    return re.sub(r"[^a-zA-Z0-9_]", "_", name)
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_utils/protocols.py b/cccl_upstream/python/cuda_cccl/cuda/compute/_utils/protocols.py
new file mode 100644
index 00000000..eb992e28
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_utils/protocols.py
@@ -0,0 +1,168 @@
+# Copyright (c) 2024-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, List, Optional, Tuple
+
+import numpy as np
+
+"""
+Utilities for extracting information from protocols such as `__cuda_array_interface__` and `__cuda_stream__`.
+"""
+
+if TYPE_CHECKING:
+    from ..typing import DeviceArrayLike, GpuStruct
+
+
+def is_device_array(obj: object) -> bool:
+    """Check if an object implements the `__cuda_array_interface__` protocol."""
+    return hasattr(obj, "__cuda_array_interface__")
+
+
+def get_data_pointer(arr: DeviceArrayLike) -> int:
+    # TODO: these are fast paths for CuPy and PyTorch until
+    # we have a more general solution.
+
+    # Fast path for PyTorch (arr.data_ptr())
+    try:
+        return arr.data_ptr()  # type: ignore
+    except AttributeError:
+        pass
+
+    # Fast path for CuPy (arr.data.ptr)
+    try:
+        return arr.data.ptr  # type: ignore
+    except AttributeError:
+        pass
+
+    # Fall back to __cuda_array_interface__
+    return arr.__cuda_array_interface__["data"][0]
+
+
+def get_dtype(arr: DeviceArrayLike | GpuStruct | np.ndarray) -> np.dtype:
+    # Try the fast path via .dtype attribute (works for np.ndarray, GpuStruct, and most device arrays)
+    try:
+        return np.dtype(arr.dtype)  # type: ignore
+    except (AttributeError, TypeError):
+        pass
+
+    # Fall back to __cuda_array_interface__ for DeviceArrayLike
+    cai = arr.__cuda_array_interface__  # type: ignore
+    typestr = cai["typestr"]
+
+    if typestr.startswith("|V"):
+        # it's a structured dtype, use the descr field:
+        return np.dtype(cai["descr"])
+    else:
+        # a simple dtype, use the typestr field:
+        return np.dtype(typestr)
+
+
+def get_shape(arr: DeviceArrayLike) -> Tuple[int]:
+    try:
+        # TODO: this is a fast path for CuPy until
+        # we have a more general solution.
+        return arr.shape  # type: ignore
+    except AttributeError:
+        return arr.__cuda_array_interface__["shape"]
+
+
+def get_size(arr: DeviceArrayLike) -> int:
+    """Get the total number of elements in an array."""
+    # Try fast path via .size attribute
+    try:
+        return int(arr.size)  # type: ignore
+    except AttributeError:
+        pass
+
+    # Fall back to computing from shape
+    shape = get_shape(arr)
+    import math
+
+    return math.prod(shape)
+
+
+def is_contiguous(arr: DeviceArrayLike) -> bool:
+    cai = arr.__cuda_array_interface__
+
+    strides = cai["strides"]
+
+    if strides is None:
+        return True
+
+    shape = cai["shape"]
+
+    if any(dim == 0 for dim in shape):
+        # array has no elements
+        return True
+
+    if all(dim == 1 for dim in shape):
+        # there is a single element:
+        return True
+
+    itemsize = get_dtype(arr).itemsize
+
+    if strides[-1] == itemsize:
+        # assume C-contiguity
+        expected_stride = itemsize
+        for dim, stride in zip(reversed(shape), reversed(strides)):
+            if stride != expected_stride:
+                return False
+            expected_stride *= dim
+        return True
+    elif strides[0] == itemsize:
+        # assume F-contiguity
+        expected_stride = itemsize
+        for dim, stride in zip(shape, strides):
+            if stride != expected_stride:
+                return False
+            expected_stride *= dim
+        return True
+    else:
+        # not contiguous
+        return False
+
+
+def compute_c_contiguous_strides_in_bytes(
+    shape: Tuple[int], itemsize: int
+) -> Tuple[int, ...]:
+    """Return C-contiguous strides in bytes for a given shape and itemsize (compatible with NumPy .strides)."""
+
+    strides: List[int] = []
+    acc = itemsize
+
+    for dim in reversed(shape):
+        strides.insert(0, acc)
+        acc *= dim
+
+    return tuple(strides)
+
+
+def validate_and_get_stream(stream) -> Optional[int]:
+    # null stream is allowed
+    if stream is None:
+        return None
+
+    try:
+        stream_property = stream.__cuda_stream__()
+    except AttributeError as e:
+        raise TypeError(
+            f"stream argument {stream} does not implement the '__cuda_stream__' protocol"
+        ) from e
+
+    try:
+        version, handle, *_ = stream_property
+    except (TypeError, ValueError) as e:
+        raise TypeError(
+            f"could not obtain __cuda_stream__ protocol version and handle from {stream_property}"
+        ) from e
+
+    if version == 0:
+        if not isinstance(handle, int):
+            raise TypeError(f"invalid stream handle {handle}")
+        return handle
+
+    raise TypeError(f"unsupported __cuda_stream__ version {version}")
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/_utils/temp_storage_buffer.py b/cccl_upstream/python/cuda_cccl/cuda/compute/_utils/temp_storage_buffer.py
new file mode 100644
index 00000000..e56cc8c9
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/_utils/temp_storage_buffer.py
@@ -0,0 +1,94 @@
+from __future__ import annotations
+
+import functools
+import weakref
+from types import SimpleNamespace
+from typing import Optional
+
+from cuda.bindings import driver, runtime
+
+try:
+    from cuda.core import Device
+    from cuda.core._utils.cuda_utils import handle_return
+except ImportError:
+    from cuda.core.experimental import Device
+    from cuda.core.experimental._utils.cuda_utils import handle_return
+
+from ..typing import StreamLike
+
+
+@functools.cache
+def _set_default_mempool_threshold(device_id: int):
+    """
+    Set the release threshold for the default memory pool on this device,
+    if we haven't already done so. This prevents the driver from attempting
+    to shrink the pool after every sync, which can be slow.
+    """
+    default_pool = handle_return(driver.cuDeviceGetDefaultMemPool(device_id))
+    threshold = handle_return(
+        driver.cuMemPoolGetAttribute(
+            default_pool, driver.CUmemPool_attribute.CU_MEMPOOL_ATTR_RELEASE_THRESHOLD
+        )
+    )
+    if int(threshold) == 0:
+        handle_return(
+            driver.cuMemPoolSetAttribute(
+                default_pool,
+                driver.CUmemPool_attribute.CU_MEMPOOL_ATTR_RELEASE_THRESHOLD,
+                driver.cuuint64_t(0xFFFFFFFFFFFFFFFF),
+            )
+        )
+
+
+def _finalize_buffer(ptr: int, stream_handle: Optional[int] = None):
+    """Cleanup function for weakref finalizer."""
+    if ptr != 0:
+        try:
+            handle_return(runtime.cudaFreeAsync(ptr, stream_handle))
+        except Exception as e:
+            # Don't raise in finalizer, just print warning
+            print(f"Warning: Failed to free CUDA memory: {e}")
+
+
+class TempStorageBuffer:
+    """
+    Simple wrapper type around the memory allocation used for temporary storage,
+    exposing __cuda_array_interface__ and some other attributes for fast access.
+
+    This implementation uses cuda.bindings.runtime.cudaMallocAsync and
+    cudaFreeAsync for allocation and deallocation.
+    """
+
+    def __init__(self, size: int, stream: Optional[StreamLike] = None):
+        # Get the current device
+        dev = Device()
+
+        stream_handle = stream.__cuda_stream__()[1] if stream is not None else None
+
+        # Set the release threshold for the default memory pool on this device
+        _set_default_mempool_threshold(dev.device_id)
+
+        # Allocate memory using cudaMallocAsync
+        device_ptr_int = handle_return(runtime.cudaMallocAsync(size, stream_handle))
+        self._ptr = int(device_ptr_int)
+        self._stream_handle = stream_handle
+        self._size = size
+
+        # attributes for fast path access in protocols.py
+        self.nbytes = size
+        self.data = SimpleNamespace(ptr=self._ptr)
+
+        # Set up weakref finalizer for cleanup
+        self._finalizer = weakref.finalize(
+            self, _finalize_buffer, self._ptr, self._stream_handle
+        )
+
+    @property
+    def __cuda_array_interface__(self):
+        return {
+            "data": (self._ptr, False),
+            "shape": (self._size,),
+            "strides": (1,),
+            "typestr": "|u1",
+            "version": 3,
+        }
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/__init__.py b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/__init__.py
new file mode 100644
index 00000000..c8e3ec7b
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/__init__.py
@@ -0,0 +1,76 @@
+# Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+from .._serialization import deserialize as deserialize
+from .._serialization import serialize as serialize
+from ._binary_search import lower_bound as lower_bound
+from ._binary_search import make_lower_bound as make_lower_bound
+from ._binary_search import make_upper_bound as make_upper_bound
+from ._binary_search import upper_bound as upper_bound
+from ._histogram import histogram_even as histogram_even
+from ._histogram import make_histogram_even as make_histogram_even
+from ._reduce import make_reduce_into as make_reduce_into
+from ._reduce import reduce_into as reduce_into
+from ._scan import exclusive_scan as exclusive_scan
+from ._scan import inclusive_scan as inclusive_scan
+from ._scan import make_exclusive_scan as make_exclusive_scan
+from ._scan import make_inclusive_scan as make_inclusive_scan
+from ._segmented_reduce import make_segmented_reduce as make_segmented_reduce
+from ._segmented_reduce import segmented_reduce
+from ._select import make_select as make_select
+from ._select import select as select
+from ._sort import DoubleBuffer, SortOrder
+from ._sort import make_merge_sort as make_merge_sort
+from ._sort import make_radix_sort as make_radix_sort
+from ._sort import make_segmented_sort as make_segmented_sort
+from ._sort import merge_sort as merge_sort
+from ._sort import radix_sort as radix_sort
+from ._sort import segmented_sort as segmented_sort
+from ._three_way_partition import make_three_way_partition as make_three_way_partition
+from ._three_way_partition import three_way_partition as three_way_partition
+from ._transform import binary_transform, unary_transform
+from ._transform import make_binary_transform as make_binary_transform
+from ._transform import make_unary_transform as make_unary_transform
+from ._unique_by_key import make_unique_by_key as make_unique_by_key
+from ._unique_by_key import unique_by_key as unique_by_key
+
+__all__ = [
+    "serialize",
+    "deserialize",
+    "reduce_into",
+    "make_reduce_into",
+    "lower_bound",
+    "make_lower_bound",
+    "upper_bound",
+    "make_upper_bound",
+    "inclusive_scan",
+    "make_inclusive_scan",
+    "exclusive_scan",
+    "make_exclusive_scan",
+    "unary_transform",
+    "make_unary_transform",
+    "binary_transform",
+    "make_binary_transform",
+    "histogram_even",
+    "make_histogram_even",
+    "merge_sort",
+    "make_merge_sort",
+    "radix_sort",
+    "make_radix_sort",
+    "segmented_reduce",
+    "make_segmented_reduce",
+    "unique_by_key",
+    "make_unique_by_key",
+    "segmented_sort",
+    "make_segmented_sort",
+    "three_way_partition",
+    "make_three_way_partition",
+    "select",
+    "make_select",
+    "DoubleBuffer",
+    "SortOrder",
+]
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_binary_search.py b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_binary_search.py
new file mode 100644
index 00000000..4cedcb2e
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_binary_search.py
@@ -0,0 +1,345 @@
+# Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+from typing import ClassVar
+
+import numpy as np
+
+from .. import _bindings, types
+from .. import _cccl_interop as cccl
+from .._caching import cache_build_results, cache_with_registered_key_functions
+from .._cccl_interop import set_cccl_iterator_state
+from .._serialization import (
+    BUILD_RESULTS,
+    ITER,
+    OP,
+    Serializable,
+)
+from .._utils import protocols
+from ..op import OpAdapter, OpKind, make_op_adapter
+from ..typing import DeviceArrayLike, IteratorT, Operator
+
+
+def _data_pointer_or_none(array) -> int | None:
+    # A ProxyArray is a build-time placeholder with no GPU allocation and thus no
+    # data pointer; return None for it (these pointers are only cache-key
+    # discriminators) so binary_search can be built without a GPU.
+    from .._proxy import is_proxy
+
+    return None if is_proxy(array) else protocols.get_data_pointer(array)
+
+
+class _BinarySearch:
+    # Shared implementation for the lower/upper bound searchers.
+    _MODE: ClassVar[_bindings.BinarySearchMode]
+
+    __slots__ = [
+        "_bound_build_result",
+        "build_results",
+        "loaded_build_result",
+        "d_data_cccl",
+        "d_values_cccl",
+        "d_out_cccl",
+        "op_cccl",
+        "data_ptr",
+        "out_ptr",
+    ]
+
+    __serialization_schema__ = (
+        ("d_data_cccl", ITER),
+        ("d_values_cccl", ITER),
+        ("d_out_cccl", ITER),
+        ("op_cccl", OP),
+        ("build_results", BUILD_RESULTS(_bindings.DeviceBinarySearchBuildResult)),
+    )
+
+    def __init__(
+        self,
+        d_data: DeviceArrayLike,
+        d_values: DeviceArrayLike | IteratorT,
+        d_out: DeviceArrayLike,
+        comp: OpAdapter,
+        compute_capability=None,
+    ):
+        if not protocols.is_device_array(d_data):
+            raise ValueError("d_data must be a device array for index outputs.")
+        if not protocols.is_device_array(d_out):
+            raise ValueError("d_out must be a device array for index outputs.")
+
+        out_dtype = protocols.get_dtype(d_out)
+        if out_dtype.kind != "u":
+            raise TypeError("d_out must use an unsigned integer dtype for indices.")
+        if out_dtype.itemsize != np.dtype(np.uintp).itemsize:
+            raise ValueError(
+                "d_out must use a pointer-sized unsigned integer dtype (np.uintp)."
+            )
+
+        self.data_ptr = _data_pointer_or_none(d_data)
+        self.out_ptr = _data_pointer_or_none(d_out)
+
+        self.d_data_cccl = cccl.to_cccl_input_iter(d_data)
+        self.d_values_cccl = cccl.to_cccl_input_iter(d_values)
+        data_value_type = cccl.get_value_type(d_data)
+        self.d_out_cccl = cccl.to_cccl_output_iter(d_out)
+
+        self.op_cccl = comp.compile((data_value_type, data_value_type), types.uint8)
+
+        self.build_results, self._bound_build_result = cache_build_results(
+            _bindings.DeviceBinarySearchBuildResult,
+            d_data,
+            d_values,
+            d_out,
+            comp,
+            self._MODE,
+            compute_capability=compute_capability,
+            builder=lambda: cccl.build_for_ccs(
+                _bindings.DeviceBinarySearchBuildResult,
+                self._MODE,
+                self.d_data_cccl,
+                self.d_values_cccl,
+                self.d_out_cccl,
+                self.op_cccl,
+                compute_capability=compute_capability,
+            ),
+        )
+
+    def __call__(
+        self,
+        *,
+        d_data,
+        num_items: int,
+        d_values,
+        num_values: int,
+        d_out,
+        comp: Operator | None,
+        stream=None,
+    ):
+        # Select (and lazily load) the build result for the current device.
+        self.loaded_build_result = cccl.resolve_build_result(
+            self.build_results, self._bound_build_result
+        )
+
+        set_cccl_iterator_state(self.d_data_cccl, d_data)
+        set_cccl_iterator_state(self.d_values_cccl, d_values)
+        set_cccl_iterator_state(self.d_out_cccl, d_out)
+
+        # Update op state for stateful ops
+        comp_adapter = make_op_adapter(OpKind.LESS if comp is None else comp)
+        self.op_cccl.state = comp_adapter.get_state()
+
+        stream_handle = protocols.validate_and_get_stream(stream)
+        self.loaded_build_result.compute(
+            self.d_data_cccl,
+            num_items,
+            self.d_values_cccl,
+            num_values,
+            self.d_out_cccl,
+            self.op_cccl,
+            stream_handle,
+        )
+
+
+class _LowerBound(_BinarySearch, Serializable):
+    __slots__ = ()
+    _MODE = _bindings.BinarySearchMode.LOWER_BOUND
+
+
+class _UpperBound(_BinarySearch, Serializable):
+    __slots__ = ()
+    _MODE = _bindings.BinarySearchMode.UPPER_BOUND
+
+
+@cache_with_registered_key_functions
+def _make_binary_search(
+    d_data: DeviceArrayLike,
+    d_values: DeviceArrayLike | IteratorT,
+    d_out: DeviceArrayLike,
+    comp: OpAdapter,
+    mode: _bindings.BinarySearchMode,
+    data_ptr: int,
+    out_ptr: int,
+    compute_capability=None,
+):
+    """Cached factory for the binary_search searchers."""
+    cls = _LowerBound if mode == _bindings.BinarySearchMode.LOWER_BOUND else _UpperBound
+    return cls(d_data, d_values, d_out, comp, compute_capability=compute_capability)
+
+
+def make_lower_bound(
+    *,
+    d_data: DeviceArrayLike,
+    d_values: DeviceArrayLike | IteratorT,
+    d_out: DeviceArrayLike,
+    comp: Operator | None = None,
+    compute_capability=None,
+):
+    """
+    Create a lower_bound object that can be called to find insertion positions.
+
+    Example:
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/binary_search/lower_bound_object.py
+            :language: python
+            :start-after: # example-begin
+
+    Args:
+        d_data: Device array containing the sorted input range.
+        d_values: Device array or iterator containing the search values.
+        d_out: Device array to store the index results.
+        comp: Optional comparison operator (default: ``OpKind.LESS``).
+        compute_capability: Compute capability, or list of capabilities, to
+            build for ahead of time. Accepts a packed int (e.g. ``90``), a
+            ``(major, minor)`` pair, a string (e.g. ``"9.0"``), or a list
+            thereof. When ``None`` (the default), the current device's
+            architecture is used.
+
+    Returns:
+        A callable object that performs lower_bound.
+
+    See Also:
+        :func:`lower_bound`
+    """
+    comp_adapter = make_op_adapter(OpKind.LESS if comp is None else comp)
+    return _make_binary_search(
+        d_data,
+        d_values,
+        d_out,
+        comp_adapter,
+        _bindings.BinarySearchMode.LOWER_BOUND,
+        _data_pointer_or_none(d_data),
+        _data_pointer_or_none(d_out),
+        compute_capability=compute_capability,
+    )
+
+
+def make_upper_bound(
+    *,
+    d_data: DeviceArrayLike,
+    d_values: DeviceArrayLike | IteratorT,
+    d_out: DeviceArrayLike,
+    comp: Operator | None = None,
+    compute_capability=None,
+):
+    """
+    Create an upper_bound object that can be called to find insertion positions.
+
+    Example:
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/binary_search/upper_bound_object.py
+            :language: python
+            :start-after: # example-begin
+
+    Args:
+        d_data: Device array containing the sorted input range.
+        d_values: Device array or iterator containing the search values.
+        d_out: Device array to store the index results.
+        comp: Optional comparison operator (default: ``OpKind.LESS``).
+        compute_capability: Compute capability, or list of capabilities, to
+            build for ahead of time. Accepts a packed int (e.g. ``90``), a
+            ``(major, minor)`` pair, a string (e.g. ``"9.0"``), or a list
+            thereof. When ``None`` (the default), the current device's
+            architecture is used.
+
+    Returns:
+        A callable object that performs upper_bound.
+
+    See Also:
+        :func:`upper_bound`
+    """
+    comp_adapter = make_op_adapter(OpKind.LESS if comp is None else comp)
+    return _make_binary_search(
+        d_data,
+        d_values,
+        d_out,
+        comp_adapter,
+        _bindings.BinarySearchMode.UPPER_BOUND,
+        _data_pointer_or_none(d_data),
+        _data_pointer_or_none(d_out),
+        compute_capability=compute_capability,
+    )
+
+
+def lower_bound(
+    *,
+    d_data: DeviceArrayLike,
+    num_items: int,
+    d_values: DeviceArrayLike | IteratorT,
+    num_values: int,
+    d_out: DeviceArrayLike,
+    comp: Operator | None = None,
+    stream=None,
+):
+    """
+    Find the *first* position that each value in ``d_values`` would be inserted into
+    ``d_data`` to maintain sorted order.
+
+    Example:
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/binary_search/lower_bound_basic.py
+            :language: python
+            :start-after: # example-begin
+
+    Args:
+        d_data: Device array containing the sorted input range.
+        num_items: Number of items in ``d_data``.
+        d_values: Device array or iterator containing the search values.
+        num_values: Number of items in ``d_values``.
+        d_out: Device array to store the index results.
+        comp: Optional comparison operator (default: ``OpKind.LESS``).
+        stream: CUDA stream for the operation (optional).
+    """
+    searcher = make_lower_bound(
+        d_data=d_data, d_values=d_values, d_out=d_out, comp=comp
+    )
+    searcher(
+        d_data=d_data,
+        num_items=num_items,
+        d_values=d_values,
+        num_values=num_values,
+        d_out=d_out,
+        comp=comp,
+        stream=stream,
+    )
+
+
+def upper_bound(
+    *,
+    d_data: DeviceArrayLike,
+    num_items: int,
+    d_values: DeviceArrayLike | IteratorT,
+    num_values: int,
+    d_out: DeviceArrayLike,
+    comp: Operator | None = None,
+    stream=None,
+):
+    """
+    Find the *last* position that each value in ``d_values`` would be inserted into
+    ``d_data`` to maintain sorted order.
+
+    Example:
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/binary_search/upper_bound_basic.py
+            :language: python
+            :start-after: # example-begin
+
+    Args:
+        d_data: Device array containing the sorted input range.
+        num_items: Number of items in ``d_data``.
+        d_values: Device array or iterator containing the search values.
+        num_values: Number of items in ``d_values``.
+        d_out: Device array to store the index results.
+        comp: Optional comparison operator (default: ``OpKind.LESS``).
+        stream: CUDA stream for the operation (optional).
+    """
+    searcher = make_upper_bound(
+        d_data=d_data, d_values=d_values, d_out=d_out, comp=comp
+    )
+    searcher(
+        d_data=d_data,
+        num_items=num_items,
+        d_values=d_values,
+        num_values=num_values,
+        d_out=d_out,
+        comp=comp,
+        stream=stream,
+    )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_histogram.py b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_histogram.py
new file mode 100644
index 00000000..612133f7
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_histogram.py
@@ -0,0 +1,330 @@
+# Copyright (c) 2024-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+import math
+from typing import Union
+
+import numpy as np
+
+from .. import _bindings
+from .. import _cccl_interop as cccl
+from .._caching import cache_build_results, cache_with_registered_key_functions
+from .._cccl_interop import set_cccl_iterator_state, to_cccl_value_state
+from .._serialization import BUILD_RESULTS, ITER, U64, VALUE, Serializable
+from .._utils.protocols import get_data_pointer, validate_and_get_stream
+from .._utils.temp_storage_buffer import TempStorageBuffer
+from ..typing import DeviceArrayLike, IteratorT
+
+
+class _Histogram(Serializable):
+    __slots__ = [
+        "_bound_build_result",
+        "num_rows",
+        "d_samples_cccl",
+        "d_histogram_cccl",
+        "h_num_output_levels_cccl",
+        "h_lower_level_cccl",
+        "h_upper_level_cccl",
+        "build_results",
+        "loaded_build_result",
+    ]
+
+    __serialization_schema__ = (
+        ("num_rows", U64),
+        ("d_samples_cccl", ITER),
+        ("d_histogram_cccl", ITER),
+        ("h_num_output_levels_cccl", VALUE),
+        ("h_lower_level_cccl", VALUE),
+        ("h_upper_level_cccl", VALUE),
+        ("build_results", BUILD_RESULTS(_bindings.DeviceHistogramBuildResult)),
+    )
+
+    def __init__(
+        self,
+        d_samples: DeviceArrayLike | IteratorT,
+        d_histogram: DeviceArrayLike,
+        h_num_output_levels: np.ndarray,
+        h_lower_level: np.ndarray,
+        h_upper_level: np.ndarray,
+        num_samples: int,
+        compute_capability=None,
+    ):
+        num_channels = 1
+        num_active_channels = 1
+        is_evenly_segmented = True
+        self.num_rows = 1
+        num_levels = h_num_output_levels[0]
+        row_stride_samples = num_samples
+
+        self.d_samples_cccl = cccl.to_cccl_input_iter(d_samples)
+        self.d_histogram_cccl = cccl.to_cccl_output_iter(d_histogram)
+        self.h_num_output_levels_cccl = cccl.to_cccl_value(h_num_output_levels)
+        self.h_lower_level_cccl = cccl.to_cccl_value(h_lower_level)
+        self.h_upper_level_cccl = cccl.to_cccl_value(h_upper_level)
+
+        self.build_results, self._bound_build_result = cache_build_results(
+            _bindings.DeviceHistogramBuildResult,
+            d_samples,
+            d_histogram,
+            int(num_levels),
+            h_lower_level.dtype,
+            num_samples,
+            is_evenly_segmented,
+            compute_capability=compute_capability,
+            builder=lambda: cccl.build_for_ccs(
+                _bindings.DeviceHistogramBuildResult,
+                num_channels,
+                num_active_channels,
+                self.d_samples_cccl,
+                num_levels,
+                self.d_histogram_cccl,
+                self.h_lower_level_cccl.type,
+                self.num_rows,
+                row_stride_samples,
+                is_evenly_segmented,
+                compute_capability=compute_capability,
+            ),
+        )
+
+    def __call__(
+        self,
+        *,
+        temp_storage,
+        d_samples: DeviceArrayLike | IteratorT,
+        d_histogram: DeviceArrayLike,
+        h_num_output_levels: np.ndarray,
+        h_lower_level: np.ndarray,
+        h_upper_level: np.ndarray,
+        num_samples: int,
+        stream=None,
+    ):
+        # Select (and lazily load) the build result for the current device.
+        self.loaded_build_result = cccl.resolve_build_result(
+            self.build_results, self._bound_build_result
+        )
+
+        set_cccl_iterator_state(self.d_samples_cccl, d_samples)
+        set_cccl_iterator_state(self.d_histogram_cccl, d_histogram)
+        self.h_num_output_levels_cccl.state = to_cccl_value_state(h_num_output_levels)
+        self.h_lower_level_cccl.state = to_cccl_value_state(h_lower_level)
+        self.h_upper_level_cccl.state = to_cccl_value_state(h_upper_level)
+
+        stream_handle = validate_and_get_stream(stream)
+        if temp_storage is None:
+            temp_storage_bytes = 0
+            d_temp_storage = 0
+        else:
+            temp_storage_bytes = temp_storage.nbytes
+            # Note: this is slightly slower, but supports all ndarray-like objects as long as they support CAI
+            # TODO: switch to use gpumemoryview once it's ready
+            d_temp_storage = get_data_pointer(temp_storage)
+
+        temp_storage_bytes = self.loaded_build_result.compute_even(
+            d_temp_storage,
+            temp_storage_bytes,
+            self.d_samples_cccl,
+            self.d_histogram_cccl,
+            self.h_num_output_levels_cccl,
+            self.h_lower_level_cccl,
+            self.h_upper_level_cccl,
+            num_samples,
+            self.num_rows,
+            num_samples,
+            stream_handle,
+        )
+
+        return temp_storage_bytes
+
+
+@cache_with_registered_key_functions
+def _make_histogram_even_impl(
+    d_samples: DeviceArrayLike | IteratorT,
+    d_histogram: DeviceArrayLike,
+    num_output_levels_val: int,
+    level_dtype,
+    uses_64bit_offset: bool,
+    uses_privatized_smem: bool,
+    compute_capability=None,
+):
+    """Internal cached implementation of make_histogram_even.
+
+    The uses_64bit_offset and uses_privatized_smem parameters ensure
+    kernels compiled for different offset and bin count regimes aren't reused.
+    """
+    # Reconstruct the numpy arrays expected by _Histogram
+    h_num_output_levels = np.array([num_output_levels_val], dtype=np.int32)
+
+    # Bounds are runtime values. These placeholders only provide storage for
+    # cccl_value_t wrappers; build receives only the level type.
+    h_lower_level = np.zeros(1, dtype=level_dtype)
+    h_upper_level = np.ones(1, dtype=level_dtype)
+
+    # v1 only needs num_samples to select the generated offset type, so use a
+    # representative value for the requested offset-width regime.
+    if uses_64bit_offset:
+        sample_size = cccl.get_value_type(d_samples).size
+        int_max = np.iinfo(np.int32).max
+        # Smallest representative sample count that still selects long long
+        # offsets in v1's build-time offset type check.
+        build_num_samples = math.ceil(int_max / sample_size)
+    else:
+        build_num_samples = 1
+
+    return _Histogram(
+        d_samples,
+        d_histogram,
+        h_num_output_levels,
+        h_lower_level,
+        h_upper_level,
+        build_num_samples,
+        compute_capability=compute_capability,
+    )
+
+
+def make_histogram_even(
+    *,
+    d_samples: DeviceArrayLike | IteratorT,
+    d_histogram: DeviceArrayLike,
+    h_num_output_levels: np.ndarray,
+    h_lower_level: np.ndarray,
+    h_upper_level: np.ndarray,
+    num_samples: int,
+    compute_capability=None,
+):
+    """Implements a device-wide histogram that places ``d_samples`` into evenly-spaced bins.
+
+    Example:
+        Below, ``make_histogram_even`` is used to create a histogram object that can be reused.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/histogram/histogram_object.py
+          :language: python
+          :start-after: # example-begin
+
+    Args:
+        d_samples: Device array or iterator containing the input samples to be histogrammed
+        d_histogram: Device array to store the histogram
+        h_num_output_levels: Host array containing the number of output levels
+        h_lower_level: Host array containing the lower level
+        h_upper_level: Host array containing the upper level
+        num_samples: Number of samples to be histogrammed
+        compute_capability: Compute capability, or list of capabilities, to
+            build for ahead of time. Accepts a packed int (e.g. ``90``), a
+            ``(major, minor)`` pair, a string (e.g. ``"9.0"``), or a list
+            thereof. When ``None`` (the default), the current device's
+            architecture is used.
+
+    Returns:
+        A callable object that can be used to perform the histogram
+    """
+    # Extract compile-relevant cache inputs from arrays.
+    num_output_levels_val = int(h_num_output_levels[0])
+    if h_lower_level.dtype != h_upper_level.dtype:
+        raise TypeError(
+            "h_lower_level and h_upper_level must have the same dtype; "
+            f"got {h_lower_level.dtype} and {h_upper_level.dtype}"
+        )
+    level_dtype = h_lower_level.dtype
+
+    # Mirrors v1 c/parallel/src/histogram.cu offset_cpp selection:
+    # (num_rows * row_stride_samples * sample_size) < INT_MAX selects int,
+    # otherwise long long. cuda.compute currently builds one-row histograms,
+    # so row_stride_samples is num_samples.
+    sample_size = cccl.get_value_type(d_samples).size
+    int_max = np.iinfo(np.int32).max
+    uses_64bit_offset = num_samples * sample_size >= int_max
+
+    # Mirrors CUB's even-histogram dispatch:
+    # detail::histogram::max_privatized_smem_bins is 256, and
+    # dispatch_histogram.cuh uses PRIVATIZED_SMEM_BINS=256 for <=256 bins
+    # and 0 for >256 bins.
+    num_bins = num_output_levels_val - 1
+    uses_privatized_smem = num_bins <= 256
+
+    # TODO: Once v2 is the default, remove uses_64bit_offset,
+    # num_output_levels_val, and uses_privatized_smem from this cache key;
+    # v2 passes row sizing and num_output_levels at runtime.
+
+    return _make_histogram_even_impl(
+        d_samples,
+        d_histogram,
+        num_output_levels_val,
+        level_dtype,
+        uses_64bit_offset,
+        uses_privatized_smem,
+        compute_capability=compute_capability,
+    )
+
+
+def histogram_even(
+    *,
+    d_samples: DeviceArrayLike | IteratorT,
+    d_histogram: DeviceArrayLike,
+    num_output_levels: int,
+    lower_level: Union[np.floating, np.integer],
+    upper_level: Union[np.floating, np.integer],
+    num_samples: int,
+    stream=None,
+):
+    """
+    Performs device-wide histogram computation with evenly-spaced bins.
+
+    This function automatically handles temporary storage allocation and execution.
+
+    Example:
+        Below, ``histogram_even`` is used to compute a histogram with evenly-spaced bins.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/histogram/histogram_even_basic.py
+            :language: python
+            :start-after: # example-begin
+            :caption: Basic histogram example.
+
+    Args:
+        d_samples: Device array or iterator containing the input sequence of data samples
+        d_histogram: Device array to store the computed histogram
+        num_output_levels: Number of histogram bin levels (num_bins = num_output_levels - 1)
+        lower_level: Lower sample value bound (inclusive)
+        upper_level: Upper sample value bound (exclusive)
+        num_samples: Number of input samples
+        stream: CUDA stream for the operation (optional)
+    """
+    # Histogram can accept multiple channels, with one value per channel for
+    # each of these parameters. The API only supports one channel for now but we
+    # pass arrays to make_histogram_even to support multiple channels in the
+    # future.
+    h_num_output_levels = np.array([num_output_levels], dtype=np.int32)
+    h_lower_level = np.array([lower_level], dtype=type(lower_level))
+    h_upper_level = np.array([upper_level], dtype=type(upper_level))
+
+    histogram = make_histogram_even(
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=num_samples,
+    )
+    temp_storage_bytes = histogram(
+        temp_storage=None,
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=num_samples,
+        stream=stream,
+    )
+    temp_storage = TempStorageBuffer(temp_storage_bytes, stream)
+    histogram(
+        temp_storage=temp_storage,
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=num_samples,
+        stream=stream,
+    )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_reduce.py b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_reduce.py
new file mode 100644
index 00000000..3a3976e5
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_reduce.py
@@ -0,0 +1,277 @@
+# Copyright (c) 2024-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+from typing import Callable
+
+import numpy as np
+
+from .. import _bindings
+from .. import _cccl_interop as cccl
+from .._caching import cache_build_results, cache_with_registered_key_functions
+from .._cccl_interop import (
+    get_value_type,
+    set_cccl_iterator_state,
+    to_cccl_value_state,
+)
+from .._serialization import BUILD_RESULTS, ITER, OP, VALUE, Serializable
+from .._utils.protocols import get_data_pointer, get_dtype, validate_and_get_stream
+from .._utils.temp_storage_buffer import TempStorageBuffer
+from ..determinism import Determinism
+from ..op import OpAdapter, make_op_adapter
+from ..typing import (
+    DeviceArrayLike,
+    GpuStruct,
+    IteratorBase,
+    IteratorT,
+    Operator,
+    _Struct,
+)
+
+
+class _Reduce(Serializable):
+    __slots__ = [
+        "_bound_build_result",
+        "d_in_cccl",
+        "d_out_cccl",
+        "h_init_cccl",
+        "op_cccl",
+        "build_results",
+        "loaded_build_result",
+        "device_reduce_fn",
+    ]
+
+    __serialization_schema__ = (
+        ("d_in_cccl", ITER),
+        ("d_out_cccl", ITER),
+        ("op_cccl", OP),
+        ("h_init_cccl", VALUE),
+        ("build_results", BUILD_RESULTS(_bindings.DeviceReduceBuildResult)),
+    )
+
+    # TODO: constructor shouldn't require concrete `d_in`, `d_out`:
+    def __init__(
+        self,
+        d_in: DeviceArrayLike | IteratorT,
+        d_out: DeviceArrayLike | IteratorT,
+        op: OpAdapter,
+        h_init: np.ndarray | GpuStruct,
+        determinism: Determinism,
+        compute_capability=None,
+    ):
+        self.d_in_cccl = cccl.to_cccl_input_iter(d_in)
+        self.d_out_cccl = cccl.to_cccl_output_iter(d_out)
+        self.h_init_cccl = cccl.to_cccl_value(h_init)
+
+        # Compile the op with value types
+        value_type = get_value_type(h_init)
+        self.op_cccl = op.compile((value_type, value_type), value_type)
+
+        # loaded_build_result / device_reduce_fn are bound lazily on the first
+        # __call__ (see _bind_device_reduce_fn).
+        self.build_results, self._bound_build_result = cache_build_results(
+            _bindings.DeviceReduceBuildResult,
+            d_in,
+            d_out,
+            op,
+            h_init,
+            determinism,
+            compute_capability=compute_capability,
+            builder=lambda: cccl.build_for_ccs(
+                _bindings.DeviceReduceBuildResult,
+                self.d_in_cccl,
+                self.d_out_cccl,
+                self.op_cccl,
+                self.h_init_cccl,
+                determinism,
+                compute_capability=compute_capability,
+            ),
+        )
+
+    def _bind_device_reduce_fn(self) -> None:
+        # Derived from the loaded build result (not serialized); bound at __call__
+        # once resolve_build_result picks + loads the current device's build result.
+        if (
+            Determinism(self.loaded_build_result.determinism)
+            is Determinism.NOT_GUARANTEED
+        ):
+            self.device_reduce_fn = self.loaded_build_result.compute_nondeterministic
+        else:
+            self.device_reduce_fn = self.loaded_build_result.compute
+
+    def __call__(
+        self,
+        *,
+        temp_storage,
+        d_in,
+        d_out,
+        num_items: int,
+        op: Callable | OpAdapter,
+        h_init: np.ndarray | GpuStruct,
+        stream=None,
+    ):
+        # Select (and lazily load) the current device's build result, then bind the
+        # derived compute fn from it.
+        self.loaded_build_result = cccl.resolve_build_result(
+            self.build_results, self._bound_build_result
+        )
+        self._bind_device_reduce_fn()
+
+        set_cccl_iterator_state(self.d_in_cccl, d_in)
+        set_cccl_iterator_state(self.d_out_cccl, d_out)
+
+        # Update op state for stateful ops
+        op_adapter = make_op_adapter(op)
+        self.op_cccl.state = op_adapter.get_state()
+
+        self.h_init_cccl.state = to_cccl_value_state(h_init)
+
+        stream_handle = validate_and_get_stream(stream)
+
+        if temp_storage is None:
+            temp_storage_bytes = 0
+            d_temp_storage = 0
+        else:
+            temp_storage_bytes = temp_storage.nbytes
+            d_temp_storage = get_data_pointer(temp_storage)
+
+        temp_storage_bytes = self.device_reduce_fn(
+            d_temp_storage,
+            temp_storage_bytes,
+            self.d_in_cccl,
+            self.d_out_cccl,
+            num_items,
+            self.op_cccl,
+            self.h_init_cccl,
+            stream_handle,
+        )
+        return temp_storage_bytes
+
+
+@cache_with_registered_key_functions
+def make_reduce_into(
+    *,
+    d_in: DeviceArrayLike | IteratorT,
+    d_out: DeviceArrayLike | IteratorT,
+    op: Operator,
+    h_init: np.ndarray | GpuStruct,
+    **kwargs,
+):
+    """Computes a device-wide reduction using the specified binary ``op`` and initial value ``init``.
+
+    Example:
+        Below, ``make_reduce_into`` is used to create a reduction object that can be reused.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/reduction/reduce_object.py
+            :language: python
+            :start-after: # example-begin
+
+
+    Args:
+        d_in: Device array or iterator containing the input sequence of data items
+        d_out: Device array (of size 1) or iterator that will store the result of the reduction
+        op: Binary operator to apply.
+            The signature is ``(T, T) -> T``, where ``T`` is
+            the data type of the initial value ``h_init``.
+        init: Numpy array storing initial value of the reduction
+        compute_capability: Compute capability, or list of capabilities, to
+            build for ahead of time. Accepts a packed int (e.g. ``90``), a
+            ``(major, minor)`` pair, a string (e.g. ``"9.0"``), or a list
+            thereof. When ``None`` (the default), the current device's
+            architecture is used.
+
+    Returns:
+        A callable object that can be used to perform the reduction
+    """
+    try:
+        accum_dtype = get_dtype(h_init)
+    except (AttributeError, TypeError) as e:
+        raise TypeError(
+            "Could not determine accumulator dtype from h_init; "
+            "expected numpy array or object with .dtype"
+        ) from e
+
+    # Validate d_in and d_out if they are device arrays (iterators may not expose
+    # dtype reliably here). Additionally, only require equality of dtypes for
+    # struct objects; mixed scalar dtypes (e.g. int8 input with int64 output)
+    # is acceptable
+    if isinstance(h_init, _Struct):
+        for arr, name in ((d_in, "input"), (d_out, "output")):
+            if isinstance(arr, IteratorBase):
+                continue
+
+            dtype = get_dtype(arr)
+            if dtype != accum_dtype:
+                raise TypeError(
+                    f"reduce_into dtype mismatch: {name} dtype {dtype} != "
+                    f"accumulator dtype {accum_dtype}. "
+                    f"Ensure {name} elements and h_init have identical dtype to "
+                    "avoid truncation or misinterpretation."
+                )
+
+    op_adapter = make_op_adapter(op)
+    return _Reduce(
+        d_in,
+        d_out,
+        op_adapter,
+        h_init,
+        kwargs.get("determinism", Determinism.RUN_TO_RUN),
+        compute_capability=kwargs.get("compute_capability"),
+    )
+
+
+def reduce_into(
+    *,
+    d_in: DeviceArrayLike | IteratorT,
+    d_out: DeviceArrayLike | IteratorT,
+    num_items: int,
+    op: Operator,
+    h_init: np.ndarray | GpuStruct,
+    stream=None,
+    **kwargs,
+):
+    """
+    Performs device-wide reduction.
+
+    This function automatically handles temporary storage allocation and execution.
+
+    Example:
+        Below, ``reduce_into`` is used to compute the sum of a sequence of integers.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/reduction/sum_reduction.py
+            :language: python
+            :start-after: # example-begin
+
+    Args:
+        d_in: Device array or iterator containing the input sequence of data items
+        d_out: Device array or iterator to store the result of the reduction
+        num_items: Number of items to reduce
+        op: Binary operator to apply.
+            The signature is ``(T, T) -> T``, where ``T`` is
+            the data type of the initial value ``h_init``.
+        h_init: Initial value for the reduction
+        stream: CUDA stream for the operation (optional)
+    """
+    reducer = make_reduce_into(d_in=d_in, d_out=d_out, op=op, h_init=h_init, **kwargs)
+    tmp_storage_bytes = reducer(
+        temp_storage=None,
+        d_in=d_in,
+        d_out=d_out,
+        num_items=num_items,
+        op=op,
+        h_init=h_init,
+        stream=stream,
+    )
+    tmp_storage = TempStorageBuffer(tmp_storage_bytes, stream)
+    reducer(
+        temp_storage=tmp_storage,
+        d_in=d_in,
+        d_out=d_out,
+        num_items=num_items,
+        op=op,
+        h_init=h_init,
+        stream=stream,
+    )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_scan.py b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_scan.py
new file mode 100644
index 00000000..bfb91194
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_scan.py
@@ -0,0 +1,436 @@
+# Copyright (c) 2024-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+from typing import Callable, cast
+
+import numpy as np
+
+from .. import _bindings
+from .. import _cccl_interop as cccl
+from .._caching import cache_build_results, cache_with_registered_key_functions
+from .._cccl_interop import (
+    get_value_type,
+    set_cccl_iterator_state,
+    to_cccl_value_state,
+)
+from .._serialization import (
+    BOOL,
+    BUILD_RESULTS,
+    CONDITIONAL,
+    ENUM,
+    ITER,
+    OP,
+    VALUE,
+    Serializable,
+)
+from .._utils.protocols import (
+    get_data_pointer,
+    is_device_array,
+    validate_and_get_stream,
+)
+from .._utils.temp_storage_buffer import TempStorageBuffer
+from ..op import OpAdapter, make_op_adapter
+from ..typing import DeviceArrayLike, GpuStruct, IteratorT, Operator
+
+
+def get_init_kind(
+    init_value: np.ndarray | DeviceArrayLike | GpuStruct | None,
+) -> _bindings.InitKind:
+    match init_value:
+        case None:
+            return _bindings.InitKind.NO_INIT
+        case _ if is_device_array(init_value):
+            return _bindings.InitKind.FUTURE_VALUE_INIT
+        case _:
+            return _bindings.InitKind.VALUE_INIT
+
+
+class _Scan(Serializable):
+    __slots__ = [
+        "_bound_build_result",
+        "build_results",
+        "loaded_build_result",
+        "d_in_cccl",
+        "d_out_cccl",
+        "init_value_cccl",
+        "op_cccl",
+        "init_kind",
+        "force_inclusive",
+        "device_scan_fn",
+    ]
+
+    __serialization_schema__ = (
+        ("init_kind", ENUM(_bindings.InitKind)),
+        ("force_inclusive", BOOL),
+        ("d_in_cccl", ITER),
+        ("d_out_cccl", ITER),
+        ("op_cccl", OP),
+        (
+            "init_value_cccl",
+            CONDITIONAL(
+                "init_kind",
+                {
+                    _bindings.InitKind.NO_INIT: None,
+                    _bindings.InitKind.FUTURE_VALUE_INIT: ITER,
+                    _bindings.InitKind.VALUE_INIT: VALUE,
+                },
+            ),
+        ),
+        ("build_results", BUILD_RESULTS(_bindings.DeviceScanBuildResult)),
+    )
+
+    # TODO: constructor shouldn't require concrete `d_in`, `d_out`:
+    def __init__(
+        self,
+        d_in: DeviceArrayLike | IteratorT,
+        d_out: DeviceArrayLike | IteratorT,
+        op: OpAdapter,
+        init_value: np.ndarray | DeviceArrayLike | GpuStruct | None,
+        force_inclusive: bool,
+        compute_capability=None,
+    ):
+        self.d_in_cccl = cccl.to_cccl_input_iter(d_in)
+        self.d_out_cccl = cccl.to_cccl_output_iter(d_out)
+
+        self.init_kind = get_init_kind(init_value)
+
+        self.init_value_cccl: _bindings.Iterator | _bindings.Value | None
+
+        match self.init_kind:
+            case _bindings.InitKind.NO_INIT:
+                self.init_value_cccl = None
+                value_type = get_value_type(d_in)
+                init_value_type_info = self.d_in_cccl.value_type
+
+            case _bindings.InitKind.FUTURE_VALUE_INIT:
+                self.init_value_cccl = cccl.to_cccl_input_iter(
+                    cast(DeviceArrayLike, init_value)
+                )
+                value_type = get_value_type(cast(DeviceArrayLike, init_value))
+                init_value_type_info = self.init_value_cccl.value_type
+
+            case _bindings.InitKind.VALUE_INIT:
+                init_value_typed = cast(np.ndarray | GpuStruct, init_value)
+                self.init_value_cccl = cccl.to_cccl_value(init_value_typed)
+                value_type = get_value_type(init_value_typed)
+                init_value_type_info = self.init_value_cccl.type
+
+        self.force_inclusive = force_inclusive
+
+        # Compile the op with value types
+        self.op_cccl = op.compile((value_type, value_type), value_type)
+
+        # loaded_build_result / device_scan_fn are bound lazily on the first
+        # __call__ (see _bind_device_scan_fn).
+        self.build_results, self._bound_build_result = cache_build_results(
+            _bindings.DeviceScanBuildResult,
+            d_in,
+            d_out,
+            op,
+            init_value,
+            force_inclusive,
+            self.init_kind,
+            compute_capability=compute_capability,
+            builder=lambda: cccl.build_for_ccs(
+                _bindings.DeviceScanBuildResult,
+                self.d_in_cccl,
+                self.d_out_cccl,
+                self.op_cccl,
+                init_value_type_info,
+                force_inclusive,
+                self.init_kind,
+                compute_capability=compute_capability,
+            ),
+        )
+
+    def _bind_device_scan_fn(self) -> None:
+        # Derived from force_inclusive + init_kind, from the loaded build result (not
+        # serialized as a function); bound at __call__ once resolve_build_result picks
+        # + loads the current device's build result.
+        match (self.force_inclusive, self.init_kind):
+            case (True, _bindings.InitKind.FUTURE_VALUE_INIT):
+                self.device_scan_fn = (
+                    self.loaded_build_result.compute_inclusive_future_value
+                )
+            case (True, _bindings.InitKind.VALUE_INIT):
+                self.device_scan_fn = self.loaded_build_result.compute_inclusive
+            case (True, _bindings.InitKind.NO_INIT):
+                self.device_scan_fn = self.loaded_build_result.compute_inclusive_no_init
+            case (False, _bindings.InitKind.FUTURE_VALUE_INIT):
+                self.device_scan_fn = (
+                    self.loaded_build_result.compute_exclusive_future_value
+                )
+            case (False, _bindings.InitKind.VALUE_INIT):
+                self.device_scan_fn = self.loaded_build_result.compute_exclusive
+            case (False, _bindings.InitKind.NO_INIT):
+                raise ValueError("Exclusive scan with No init value is not supported")
+
+    def __call__(
+        self,
+        *,
+        temp_storage,
+        d_in,
+        d_out,
+        op: Callable | OpAdapter,
+        init_value: np.ndarray | DeviceArrayLike | GpuStruct | None,
+        num_items: int,
+        stream=None,
+    ):
+        # Select (and lazily load) the current device's build result, then bind the
+        # derived compute fn from it.
+        self.loaded_build_result = cccl.resolve_build_result(
+            self.build_results, self._bound_build_result
+        )
+        self._bind_device_scan_fn()
+
+        set_cccl_iterator_state(self.d_in_cccl, d_in)
+        set_cccl_iterator_state(self.d_out_cccl, d_out)
+
+        # Update op state for stateful ops
+        op_adapter = make_op_adapter(op)
+        self.op_cccl.state = op_adapter.get_state()
+
+        match self.init_kind:
+            case _bindings.InitKind.FUTURE_VALUE_INIT:
+                # We know that the init_value_cccl is an Iterator, so this cast
+                # tells MyPy what the actual type is. cast() is a no-op at runtime,
+                # which makes it better than isinstance() since this is a hot path
+                # and we have to minimize the work we do prior to calling the
+                # kernel.
+                self.init_value_cccl = cast(_bindings.Iterator, self.init_value_cccl)
+                set_cccl_iterator_state(self.init_value_cccl, init_value)
+
+            case _bindings.InitKind.VALUE_INIT:
+                self.init_value_cccl = cast(_bindings.Value, self.init_value_cccl)
+                self.init_value_cccl.state = to_cccl_value_state(
+                    cast(np.ndarray | GpuStruct, init_value)
+                )
+
+        stream_handle = validate_and_get_stream(stream)
+
+        if temp_storage is None:
+            temp_storage_bytes = 0
+            d_temp_storage = 0
+        else:
+            temp_storage_bytes = temp_storage.nbytes
+            d_temp_storage = get_data_pointer(temp_storage)
+
+        temp_storage_bytes = self.device_scan_fn(
+            d_temp_storage,
+            temp_storage_bytes,
+            self.d_in_cccl,
+            self.d_out_cccl,
+            num_items,
+            self.op_cccl,
+            self.init_value_cccl,
+            stream_handle,
+        )
+        return temp_storage_bytes
+
+
+# TODO Figure out `sum` without operator and initial value
+# TODO Accept stream
+@cache_with_registered_key_functions
+def make_exclusive_scan(
+    *,
+    d_in: DeviceArrayLike | IteratorT,
+    d_out: DeviceArrayLike | IteratorT,
+    op: Operator,
+    init_value: np.ndarray | DeviceArrayLike | GpuStruct,
+    compute_capability=None,
+):
+    """Computes a device-wide scan using the specified binary ``op`` and initial value ``init``.
+
+    Example:
+        Below, ``make_exclusive_scan`` is used to create an exclusive scan object that can be reused.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/scan/exclusive_scan_object.py
+          :language: python
+          :start-after: # example-begin
+
+
+    Args:
+        d_in: Device array or iterator containing the input sequence of data items
+        d_out: Device array that will store the result of the scan
+        op: Binary scan operator.
+            The signature is ``(T, T) -> T``, where ``T`` is the data type of
+            the initial value ``init_value``.
+        init_value: Numpy array, device array, or GPU struct storing initial value of the scan
+        compute_capability: Compute capability, or list of capabilities, to
+            build for ahead of time. Accepts a packed int (e.g. ``90``), a
+            ``(major, minor)`` pair, a string (e.g. ``"9.0"``), or a list
+            thereof. When ``None`` (the default), the current device's
+            architecture is used.
+
+    Returns:
+        A callable object that can be used to perform the scan
+    """
+    op_adapter = make_op_adapter(op)
+    return _Scan(
+        d_in,
+        d_out,
+        op_adapter,
+        init_value,
+        False,
+        compute_capability=compute_capability,
+    )
+
+
+def exclusive_scan(
+    *,
+    d_in: DeviceArrayLike | IteratorT,
+    d_out: DeviceArrayLike | IteratorT,
+    op: Operator,
+    init_value: np.ndarray | DeviceArrayLike | GpuStruct,
+    num_items: int,
+    stream=None,
+):
+    """
+    Performs device-wide exclusive scan.
+
+    This function automatically handles temporary storage allocation and execution.
+
+    Example:
+        Below, ``exclusive_scan`` is used to compute an exclusive scan with max operation.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/scan/exclusive_scan_max.py
+            :language: python
+            :start-after: # example-begin
+
+
+    Args:
+        d_in: Device array or iterator containing the input sequence of data items
+        d_out: Device array or iterator to store the result of the scan
+        op: Binary scan operator.
+            The signature is ``(T, T) -> T``, where ``T`` is the data type of
+            the initial value ``init_value``.
+        init_value: Initial value for the scan
+        num_items: Number of items to scan
+        stream: CUDA stream for the operation (optional)
+    """
+    scanner = make_exclusive_scan(d_in=d_in, d_out=d_out, op=op, init_value=init_value)
+    tmp_storage_bytes = scanner(
+        temp_storage=None,
+        d_in=d_in,
+        d_out=d_out,
+        op=op,
+        init_value=init_value,
+        num_items=num_items,
+        stream=stream,
+    )
+    tmp_storage = TempStorageBuffer(tmp_storage_bytes, stream)
+    scanner(
+        temp_storage=tmp_storage,
+        d_in=d_in,
+        d_out=d_out,
+        op=op,
+        init_value=init_value,
+        num_items=num_items,
+        stream=stream,
+    )
+
+
+# TODO Figure out `sum` without operator and initial value
+# TODO Accept stream
+@cache_with_registered_key_functions
+def make_inclusive_scan(
+    *,
+    d_in: DeviceArrayLike | IteratorT,
+    d_out: DeviceArrayLike | IteratorT,
+    op: Operator,
+    init_value: np.ndarray | DeviceArrayLike | GpuStruct | None = None,
+    compute_capability=None,
+):
+    """Computes a device-wide scan using the specified binary ``op`` and initial value ``init``.
+
+    Example:
+        Below, ``make_inclusive_scan`` is used to create an inclusive scan object that can be reused.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/scan/inclusive_scan_object.py
+          :language: python
+          :start-after: # example-begin
+
+
+    Args:
+        d_in: Device array or iterator containing the input sequence of data items
+        d_out: Device array that will store the result of the scan
+        op: Binary scan operator.
+            The signature is ``(T, T) -> T``, where ``T`` is the data type of
+            the initial value ``init_value``.
+        init_value: Numpy array, device array, or GPU struct storing initial value of the scan, or None for no initial value
+        compute_capability: Compute capability, or list of capabilities, to
+            build for ahead of time. Accepts a packed int (e.g. ``90``), a
+            ``(major, minor)`` pair, a string (e.g. ``"9.0"``), or a list
+            thereof. When ``None`` (the default), the current device's
+            architecture is used.
+
+    Returns:
+        A callable object that can be used to perform the scan
+    """
+    op_adapter = make_op_adapter(op)
+    return _Scan(
+        d_in,
+        d_out,
+        op_adapter,
+        init_value,
+        True,
+        compute_capability=compute_capability,
+    )
+
+
+def inclusive_scan(
+    *,
+    d_in: DeviceArrayLike | IteratorT,
+    d_out: DeviceArrayLike | IteratorT,
+    op: Operator,
+    init_value: np.ndarray | DeviceArrayLike | GpuStruct | None = None,
+    num_items: int,
+    stream=None,
+):
+    """
+    Performs device-wide inclusive scan.
+
+    This function automatically handles temporary storage allocation and execution.
+
+    Example:
+        Below, ``inclusive_scan`` is used to compute an inclusive scan (prefix sum).
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/scan/inclusive_scan_custom.py
+            :language: python
+            :start-after: # example-begin
+
+
+    Args:
+        d_in: Device array or iterator containing the input sequence of data items
+        d_out: Device array or iterator to store the result of the scan
+        op: Binary scan operator.
+            The signature is ``(T, T) -> T``, where ``T`` is the data type of
+            the initial value ``init_value``.
+        init_value: Initial value for the scan
+        num_items: Number of items to scan
+        stream: CUDA stream for the operation (optional)
+    """
+    scanner = make_inclusive_scan(d_in=d_in, d_out=d_out, op=op, init_value=init_value)
+    tmp_storage_bytes = scanner(
+        temp_storage=None,
+        d_in=d_in,
+        d_out=d_out,
+        op=op,
+        init_value=init_value,
+        num_items=num_items,
+        stream=stream,
+    )
+    tmp_storage = TempStorageBuffer(tmp_storage_bytes, stream)
+    scanner(
+        temp_storage=tmp_storage,
+        d_in=d_in,
+        d_out=d_out,
+        op=op,
+        init_value=init_value,
+        num_items=num_items,
+        stream=stream,
+    )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_segmented_reduce.py b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_segmented_reduce.py
new file mode 100644
index 00000000..305f123d
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_segmented_reduce.py
@@ -0,0 +1,293 @@
+# Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+from typing import Callable
+
+import numpy as np
+
+from .. import _bindings
+from .. import _cccl_interop as cccl
+from .._caching import cache_build_results, cache_with_registered_key_functions
+from .._cccl_interop import (
+    get_value_type,
+    set_cccl_iterator_state,
+    to_cccl_value_state,
+)
+from .._serialization import BUILD_RESULTS, ITER, OP, VALUE, Serializable
+from .._utils.protocols import (
+    get_data_pointer,
+    validate_and_get_stream,
+)
+from .._utils.temp_storage_buffer import TempStorageBuffer
+from ..op import OpAdapter, make_op_adapter
+from ..typing import DeviceArrayLike, GpuStruct, IteratorT, Operator
+
+
+class _SegmentedReduce(Serializable):
+    __slots__ = [
+        "_bound_build_result",
+        "build_results",
+        "loaded_build_result",
+        "d_in_cccl",
+        "d_out_cccl",
+        "start_offsets_in_cccl",
+        "end_offsets_in_cccl",
+        "h_init_cccl",
+        "op_cccl",
+    ]
+
+    __serialization_schema__ = (
+        ("d_in_cccl", ITER),
+        ("d_out_cccl", ITER),
+        ("start_offsets_in_cccl", ITER),
+        ("end_offsets_in_cccl", ITER),
+        ("h_init_cccl", VALUE),
+        ("op_cccl", OP),
+        ("build_results", BUILD_RESULTS(_bindings.DeviceSegmentedReduceBuildResult)),
+    )
+
+    def __init__(
+        self,
+        d_in: DeviceArrayLike | IteratorT,
+        d_out: DeviceArrayLike | IteratorT,
+        start_offsets_in: DeviceArrayLike | IteratorT,
+        end_offsets_in: DeviceArrayLike | IteratorT,
+        op: OpAdapter,
+        h_init: np.ndarray | GpuStruct,
+        compute_capability=None,
+    ):
+        self.d_in_cccl = cccl.to_cccl_input_iter(d_in)
+        self.d_out_cccl = cccl.to_cccl_output_iter(d_out)
+        self.start_offsets_in_cccl = cccl.to_cccl_input_iter(start_offsets_in)
+        self.end_offsets_in_cccl = cccl.to_cccl_input_iter(end_offsets_in)
+        self.h_init_cccl = cccl.to_cccl_value(h_init)
+
+        # Compile the op with value types
+        value_type = get_value_type(h_init)
+
+        self.op_cccl = op.compile((value_type, value_type), value_type)
+
+        self.build_results, self._bound_build_result = cache_build_results(
+            _bindings.DeviceSegmentedReduceBuildResult,
+            d_in,
+            d_out,
+            start_offsets_in,
+            end_offsets_in,
+            op,
+            h_init,
+            compute_capability=compute_capability,
+            builder=lambda: cccl.build_for_ccs(
+                _bindings.DeviceSegmentedReduceBuildResult,
+                self.d_in_cccl,
+                self.d_out_cccl,
+                self.start_offsets_in_cccl,
+                self.end_offsets_in_cccl,
+                self.op_cccl,
+                self.h_init_cccl,
+                compute_capability=compute_capability,
+            ),
+        )
+
+    def __call__(
+        self,
+        *,
+        temp_storage,
+        d_in,
+        d_out,
+        num_segments: int,
+        start_offsets_in,
+        end_offsets_in,
+        op: Callable | OpAdapter,
+        h_init,
+        max_segment_size: int | None = None,
+        stream=None,
+    ):
+        # Select (and lazily load) the build result for the current device.
+        self.loaded_build_result = cccl.resolve_build_result(
+            self.build_results, self._bound_build_result
+        )
+
+        if num_segments > np.iinfo(np.int32).max:
+            raise RuntimeError(
+                "Segmented sort does not currently support more than 2^31-1 segments."
+            )
+
+        if max_segment_size is None:
+            max_segment_size = 0  # CCCL.c treats 0 as "not specified"
+
+        if max_segment_size > 0:
+            try:
+                from .._build_info import USING_V2  # type: ignore[import-not-found]
+            except ImportError:
+                USING_V2 = False
+            if USING_V2:
+                import warnings
+
+                warnings.warn(
+                    "max_segment_size is not used by the v2 backend and will be ignored",
+                    stacklevel=4,
+                )
+
+        set_cccl_iterator_state(self.d_in_cccl, d_in)
+        set_cccl_iterator_state(self.d_out_cccl, d_out)
+        set_cccl_iterator_state(self.start_offsets_in_cccl, start_offsets_in)
+        set_cccl_iterator_state(self.end_offsets_in_cccl, end_offsets_in)
+
+        op_adapter = make_op_adapter(op)
+        self.op_cccl.state = op_adapter.get_state()
+
+        self.h_init_cccl.state = to_cccl_value_state(h_init)
+
+        stream_handle = validate_and_get_stream(stream)
+
+        if temp_storage is None:
+            temp_storage_bytes = 0
+            d_temp_storage = 0
+        else:
+            temp_storage_bytes = temp_storage.nbytes
+            d_temp_storage = get_data_pointer(temp_storage)
+
+        temp_storage_bytes = self.loaded_build_result.compute(
+            d_temp_storage,
+            temp_storage_bytes,
+            self.d_in_cccl,
+            self.d_out_cccl,
+            num_segments,
+            self.start_offsets_in_cccl,
+            self.end_offsets_in_cccl,
+            self.op_cccl,
+            self.h_init_cccl,
+            max_segment_size,
+            stream_handle,
+        )
+        return temp_storage_bytes
+
+
+@cache_with_registered_key_functions
+def make_segmented_reduce(
+    *,
+    d_in: DeviceArrayLike | IteratorT,
+    d_out: DeviceArrayLike | IteratorT,
+    start_offsets_in: DeviceArrayLike | IteratorT,
+    end_offsets_in: DeviceArrayLike | IteratorT,
+    op: Operator,
+    h_init: np.ndarray | GpuStruct,
+    compute_capability=None,
+):
+    """Computes a device-wide segmented reduction using the specified binary ``op`` and initial value ``init``.
+
+    Example:
+        Below, ``make_segmented_reduce`` is used to create a segmented reduction object that can be reused.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/segmented/segmented_reduce_object.py
+            :language: python
+            :start-after: # example-begin
+
+
+    Args:
+        d_in: Device array or iterator containing the input sequence of data items
+        d_out: Device array that will store the result of the reduction
+        start_offsets_in: Device array or iterator containing offsets to start of segments
+        end_offsets_in: Device array or iterator containing offsets to end of segments
+        op: Binary operator to apply.
+            The signature is ``(T, T) -> T``, where ``T`` is
+            the data type of the initial value ``h_init``.
+        init: Numpy array storing initial value of the reduction
+        compute_capability: Compute capability, or list of capabilities, to
+            build for ahead of time. Accepts a packed int (e.g. ``90``), a
+            ``(major, minor)`` pair, a string (e.g. ``"9.0"``), or a list
+            thereof. When ``None`` (the default), the current device's
+            architecture is used.
+
+    Returns:
+        A callable object that can be used to perform the reduction
+    """
+    op_adapter = make_op_adapter(op)
+    return _SegmentedReduce(
+        d_in,
+        d_out,
+        start_offsets_in,
+        end_offsets_in,
+        op_adapter,
+        h_init,
+        compute_capability=compute_capability,
+    )
+
+
+def segmented_reduce(
+    *,
+    d_in: DeviceArrayLike | IteratorT,
+    d_out: DeviceArrayLike | IteratorT,
+    num_segments: int,
+    start_offsets_in: DeviceArrayLike | IteratorT,
+    end_offsets_in: DeviceArrayLike | IteratorT,
+    op: Operator,
+    h_init: np.ndarray | GpuStruct,
+    max_segment_size: int | None = None,
+    stream=None,
+):
+    """
+    Performs device-wide segmented reduction.
+
+    This function automatically handles temporary storage allocation and execution.
+
+    Example:
+        Below, ``segmented_reduce`` is used to compute the minimum value of segments in a sequence of integers.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/segmented/segmented_reduce_basic.py
+            :language: python
+            :start-after: # example-begin
+
+
+    Args:
+        d_in: Device array or iterator containing the input sequence of data items
+        d_out: Device array to store the result of the reduction for each segment
+        num_segments: Number of segments to reduce
+        start_offsets_in: Device array or iterator containing the sequence of beginning offsets
+        end_offsets_in: Device array or iterator containing the sequence of ending offsets
+        op: Binary operator to apply.
+            The signature is ``(T, T) -> T``, where ``T`` is
+            the data type of the initial value ``h_init``.
+        h_init: Initial value for the reduction
+        max_segment_size: The number of elements in the largest segment (optional)
+            If provided, this information is used to dispatch to the
+            optimal kernel for best performance.
+        stream: CUDA stream for the operation (optional)
+    """
+    reducer = make_segmented_reduce(
+        d_in=d_in,
+        d_out=d_out,
+        start_offsets_in=start_offsets_in,
+        end_offsets_in=end_offsets_in,
+        op=op,
+        h_init=h_init,
+    )
+    tmp_storage_bytes = reducer(
+        temp_storage=None,
+        d_in=d_in,
+        d_out=d_out,
+        num_segments=num_segments,
+        start_offsets_in=start_offsets_in,
+        end_offsets_in=end_offsets_in,
+        op=op,
+        h_init=h_init,
+        max_segment_size=max_segment_size,
+        stream=stream,
+    )
+    tmp_storage = TempStorageBuffer(tmp_storage_bytes, stream)
+    reducer(
+        temp_storage=tmp_storage,
+        d_in=d_in,
+        d_out=d_out,
+        num_segments=num_segments,
+        start_offsets_in=start_offsets_in,
+        end_offsets_in=end_offsets_in,
+        op=op,
+        h_init=h_init,
+        max_segment_size=max_segment_size,
+        stream=stream,
+    )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_select.py b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_select.py
new file mode 100644
index 00000000..8beeed6c
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_select.py
@@ -0,0 +1,239 @@
+# Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+from functools import cache
+
+from .._caching import cache_with_registered_key_functions
+from .._cpp_compile import compile_cpp_op_code
+from .._serialization import NESTED, Serializable
+from .._utils.temp_storage_buffer import TempStorageBuffer
+from ..iterators import DiscardIterator
+from ..op import OpAdapter, RawOp, make_op_adapter
+from ..typing import DeviceArrayLike, IteratorT, Operator
+from ._three_way_partition import _ThreeWayPartition, make_three_way_partition
+
+
+@cache
+def _always_false_op(_target_cc):
+    # ``_target_cc`` (the build's get_target_cc()) is part of the cache key so the
+    # predicate's LTO-IR is recompiled per target arch: this RawOp is linked into
+    # the three-way-partition build, and nvJitLink rejects a newer-arch input in
+    # an older-arch result. Without the key, the first build's arch would leak
+    # into every later build (module-global cache). compile_cpp_op_code() reads
+    # the same target internally; the arg only distinguishes cache entries.
+    source = """
+extern "C" __device__ void always_false(void*, void* result) {{
+    *static_cast<bool*>(result) = false;
+}}
+"""
+    code = compile_cpp_op_code(source)
+    return RawOp(ltoir=code, name="always_false")
+
+
+def _get_always_false_op():
+    """The always-false predicate compiled for the current build's target cc."""
+    from .._target_cc import get_target_cc
+
+    return _always_false_op(get_target_cc())
+
+
+class _Select(Serializable):
+    __slots__ = ["_bound_build_result", "partitioner", "always_false_op", "_discards"]
+
+    __serialization_schema__ = (("partitioner", NESTED(_ThreeWayPartition)),)
+
+    def __init__(
+        self,
+        d_in: DeviceArrayLike | IteratorT,
+        d_out: DeviceArrayLike | IteratorT,
+        d_num_selected_out: DeviceArrayLike,
+        cond: OpAdapter,
+        compute_capability=None,
+    ):
+        self.always_false_op = _get_always_false_op()
+        d_second, d_unselected = self._discard_iterators(d_out)
+        self.partitioner = make_three_way_partition(
+            d_in=d_in,
+            d_first_part_out=d_out,
+            d_second_part_out=d_second,
+            d_unselected_out=d_unselected,
+            d_num_selected_out=d_num_selected_out,
+            select_first_part_op=cond,
+            select_second_part_op=self.always_false_op,
+            compute_capability=compute_capability,
+        )
+
+    def _discard_iterators(self, d_out):
+        # The second/unselected outputs are discarded; their iterators depend
+        # only on d_out's type, so build the pair once and cache it. Bound
+        # lazily (on first construction or first call) so a deserialized
+        # _Select, which has no construction d_out, builds them on first use.
+        try:
+            return self._discards
+        except AttributeError:
+            self._discards = (DiscardIterator(d_out), DiscardIterator(d_out))
+            return self._discards
+
+    def _after_deserialize(self) -> None:
+        # always_false_op (the always-false second predicate) is not serialized.
+        # Its compiled LTO-IR is already baked into the (serialized) three-way
+        # partition build result, and __call__ reads only this op's runtime state
+        # (which is empty — the predicate is stateless). So reconstruct an
+        # empty-state stand-in WITHOUT compiling: deserialize() must neither
+        # recompile nor require a GPU, and calling _get_always_false_op() here
+        # would do both (cold cache -> compile_cpp_op_code -> Device() fallback).
+        self.always_false_op = RawOp(ltoir=b"", name="always_false")
+
+    def __call__(
+        self,
+        *,
+        temp_storage,
+        d_in,
+        d_out,
+        d_num_selected_out,
+        cond,
+        num_items: int,
+        stream=None,
+    ):
+        d_second, d_unselected = self._discard_iterators(d_out)
+        return self.partitioner(
+            temp_storage=temp_storage,
+            d_in=d_in,
+            d_first_part_out=d_out,
+            d_second_part_out=d_second,
+            d_unselected_out=d_unselected,
+            d_num_selected_out=d_num_selected_out,
+            select_first_part_op=make_op_adapter(cond),
+            select_second_part_op=self.always_false_op,
+            num_items=num_items,
+            stream=stream,
+        )
+
+
+@cache_with_registered_key_functions
+def make_select(
+    *,
+    d_in: DeviceArrayLike | IteratorT,
+    d_out: DeviceArrayLike | IteratorT,
+    d_num_selected_out: DeviceArrayLike,
+    cond: Operator,
+    compute_capability=None,
+):
+    """
+    Create a select object that can be called to select elements matching a condition.
+
+    This is the object-oriented API that allows explicit control over temporary
+    storage allocation. For simpler usage, consider using :func:`select`.
+
+    Example:
+        Below, ``make_select`` is used to create a select object that can be reused.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/select/select_object.py
+            :language: python
+            :start-after: # example-begin
+
+    Args:
+        d_in: Device array or iterator containing the input sequence of data items.
+        d_out: Device array or iterator to store the selected output items.
+        d_num_selected_out: Device array to store the number of items that passed the selection.
+            The count is stored in ``d_num_selected_out[0]``.
+        cond: Selection condition (predicate).
+            The signature is ``(T) -> uint8``, where ``T`` is the input data type.
+            Returns 1 (selected) or 0 (not selected).
+        compute_capability: Compute capability, or list of capabilities, to
+            build for ahead of time. Accepts a packed int (e.g. ``90``), a
+            ``(major, minor)`` pair, a string (e.g. ``"9.0"``), or a list
+            thereof. When ``None`` (the default), the current device's
+            architecture is used.
+
+    Returns:
+        A callable object that performs the selection operation.
+    """
+    cond_adapter = make_op_adapter(cond)
+    # Note: _Select internally calls make_three_way_partition which will
+    # normalize the cond. But we've already normalized it, so the Op
+    # will be passed through make_op unchanged.
+    return _Select(
+        d_in,
+        d_out,
+        d_num_selected_out,
+        cond_adapter,
+        compute_capability=compute_capability,
+    )
+
+
+def select(
+    *,
+    d_in: DeviceArrayLike | IteratorT,
+    d_out: DeviceArrayLike | IteratorT,
+    d_num_selected_out: DeviceArrayLike,
+    cond: Operator,
+    num_items: int,
+    stream=None,
+):
+    """
+    Performs device-wide selection of elements based on a condition.
+
+    Given an input sequence, this function selects all elements for which the condition
+    function ``cond`` returns true (non-zero) and writes them to the output in a
+    compacted form. The number of selected elements is written to ``d_num_selected_out[0]``.
+
+    This function automatically handles temporary storage allocation and execution.
+
+    The ``cond`` function can reference device arrays as globals or closures - they will
+    be automatically captured as state arrays, enabling stateful operations like counting.
+
+    Example:
+        Below, ``select`` is used to select even numbers from an input array:
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/select/select_basic.py
+            :language: python
+            :start-after: # example-begin
+
+        You can also use iterators for more complex selection patterns:
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/select/select_with_iterator.py
+            :language: python
+            :start-after: # example-begin
+
+    Args:
+        d_in: Device array or iterator containing the input sequence of data items.
+        d_out: Device array or iterator to store the selected output items.
+        d_num_selected_out: Device array to store the number of items that passed the selection.
+            The count is stored in ``d_num_selected_out[0]``.
+        cond: Selection condition (predicate).
+            The signature is ``(T) -> uint8``, where ``T`` is the input data type.
+            Returns 1 (selected) or 0 (not selected).
+            Can reference device arrays as globals/closures - they will be automatically captured.
+        num_items: Number of items in the input sequence.
+        stream: CUDA stream to use for the operation (optional).
+    """
+    # Create adapter to support stateful ops
+    cond_adapter = make_op_adapter(cond)
+    selector = make_select(
+        d_in=d_in, d_out=d_out, d_num_selected_out=d_num_selected_out, cond=cond_adapter
+    )
+
+    tmp_storage_bytes = selector(
+        temp_storage=None,
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected_out,
+        cond=cond_adapter,
+        num_items=num_items,
+        stream=stream,
+    )
+    tmp_storage = TempStorageBuffer(tmp_storage_bytes, stream)
+    selector(
+        temp_storage=tmp_storage,
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected_out,
+        cond=cond_adapter,
+        num_items=num_items,
+        stream=stream,
+    )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_sort/__init__.py b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_sort/__init__.py
new file mode 100644
index 00000000..7fca073c
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_sort/__init__.py
@@ -0,0 +1,25 @@
+# Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+from ._merge_sort import make_merge_sort as make_merge_sort
+from ._merge_sort import merge_sort as merge_sort
+from ._radix_sort import make_radix_sort as make_radix_sort
+from ._radix_sort import radix_sort as radix_sort
+from ._segmented_sort import make_segmented_sort as make_segmented_sort
+from ._segmented_sort import segmented_sort as segmented_sort
+from ._sort_common import DoubleBuffer, SortOrder
+
+__all__ = [
+    "make_merge_sort",
+    "merge_sort",
+    "make_radix_sort",
+    "radix_sort",
+    "make_segmented_sort",
+    "segmented_sort",
+    "DoubleBuffer",
+    "SortOrder",
+]
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_sort/_merge_sort.py b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_sort/_merge_sort.py
new file mode 100644
index 00000000..67e88ac6
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_sort/_merge_sort.py
@@ -0,0 +1,261 @@
+# Copyright (c) 2024-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+
+from __future__ import annotations
+
+from ... import _bindings, types
+from ... import _cccl_interop as cccl
+from ..._caching import cache_build_results, cache_with_registered_key_functions
+from ..._cccl_interop import set_cccl_iterator_state
+from ..._serialization import BUILD_RESULTS, ITER, OP, Serializable
+from ..._utils.protocols import (
+    get_data_pointer,
+    validate_and_get_stream,
+)
+from ..._utils.temp_storage_buffer import TempStorageBuffer
+from ...op import OpAdapter, make_op_adapter
+from ...typing import DeviceArrayLike, IteratorT, Operator
+
+
+class _MergeSort(Serializable):
+    __slots__ = [
+        "_bound_build_result",
+        "d_in_keys_cccl",
+        "d_in_values_cccl",
+        "d_out_keys_cccl",
+        "d_out_values_cccl",
+        "op_adapter",
+        "op_cccl",
+        "build_results",
+        "loaded_build_result",
+    ]
+
+    __serialization_schema__ = (
+        ("d_in_keys_cccl", ITER),
+        ("d_in_values_cccl", ITER),
+        ("d_out_keys_cccl", ITER),
+        ("d_out_values_cccl", ITER),
+        ("op_cccl", OP),
+        ("build_results", BUILD_RESULTS(_bindings.DeviceMergeSortBuildResult)),
+    )
+
+    def __init__(
+        self,
+        d_in_keys: DeviceArrayLike | IteratorT,
+        d_in_values: DeviceArrayLike | IteratorT | None,
+        d_out_keys: DeviceArrayLike,
+        d_out_values: DeviceArrayLike | None,
+        op: OpAdapter,
+        compute_capability=None,
+    ):
+        present_in_values = d_in_values is not None
+        present_out_values = d_out_values is not None
+        assert present_in_values == present_out_values
+
+        self.d_in_keys_cccl = cccl.to_cccl_input_iter(d_in_keys)
+        self.d_in_values_cccl = cccl.to_cccl_input_iter(d_in_values)
+        self.d_out_keys_cccl = cccl.to_cccl_output_iter(d_out_keys)
+        self.d_out_values_cccl = cccl.to_cccl_output_iter(d_out_values)
+        self.op_adapter = op
+
+        # Compile the op - merge_sort expects int8 return (comparison)
+        value_type = cccl.get_value_type(d_in_keys)
+        self.op_cccl = op.compile((value_type, value_type), types.int8)
+
+        self.build_results, self._bound_build_result = cache_build_results(
+            _bindings.DeviceMergeSortBuildResult,
+            d_in_keys,
+            d_in_values,
+            d_out_keys,
+            d_out_values,
+            op,
+            compute_capability=compute_capability,
+            builder=lambda: cccl.build_for_ccs(
+                _bindings.DeviceMergeSortBuildResult,
+                self.d_in_keys_cccl,
+                self.d_in_values_cccl,
+                self.d_out_keys_cccl,
+                self.d_out_values_cccl,
+                self.op_cccl,
+                compute_capability=compute_capability,
+            ),
+        )
+
+    def __call__(
+        self,
+        *,
+        temp_storage,
+        d_in_keys: DeviceArrayLike | IteratorT,
+        d_in_values: DeviceArrayLike | IteratorT | None,
+        d_out_keys: DeviceArrayLike,
+        d_out_values: DeviceArrayLike | None,
+        num_items: int,
+        op: Operator,
+        stream=None,
+    ):
+        # Select (and lazily load) the build result for the current device.
+        self.loaded_build_result = cccl.resolve_build_result(
+            self.build_results, self._bound_build_result
+        )
+
+        present_in_values = d_in_values is not None
+        present_out_values = d_out_values is not None
+        assert present_in_values == present_out_values
+
+        set_cccl_iterator_state(self.d_in_keys_cccl, d_in_keys)
+        if present_in_values:
+            set_cccl_iterator_state(self.d_in_values_cccl, d_in_values)
+        set_cccl_iterator_state(self.d_out_keys_cccl, d_out_keys)
+        if present_out_values:
+            set_cccl_iterator_state(self.d_out_values_cccl, d_out_values)
+
+        op_adapter = make_op_adapter(op)
+        self.op_cccl.state = op_adapter.get_state()
+
+        stream_handle = validate_and_get_stream(stream)
+        if temp_storage is None:
+            temp_storage_bytes = 0
+            d_temp_storage = 0
+        else:
+            temp_storage_bytes = temp_storage.nbytes
+            # Note: this is slightly slower, but supports all ndarray-like objects as long as they support CAI
+            # TODO: switch to use gpumemoryview once it's ready
+            d_temp_storage = get_data_pointer(temp_storage)
+
+        temp_storage_bytes = self.loaded_build_result.compute(
+            d_temp_storage,
+            temp_storage_bytes,
+            self.d_in_keys_cccl,
+            self.d_in_values_cccl,
+            self.d_out_keys_cccl,
+            self.d_out_values_cccl,
+            num_items,
+            self.op_cccl,
+            stream_handle,
+        )
+
+        return temp_storage_bytes
+
+
+@cache_with_registered_key_functions
+def make_merge_sort(
+    *,
+    d_in_keys: DeviceArrayLike | IteratorT,
+    d_in_values: DeviceArrayLike | IteratorT | None = None,
+    d_out_keys: DeviceArrayLike,
+    d_out_values: DeviceArrayLike | None = None,
+    op: Operator,
+    compute_capability=None,
+):
+    """Implements a device-wide merge sort using ``d_in_keys`` and the comparison operator ``op``.
+
+    Example:
+        Below, ``make_merge_sort`` is used to create a merge sort object that can be reused.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/sort/merge_sort_object.py
+          :language: python
+          :start-after: # example-begin
+
+
+    Args:
+        d_in_keys: Device array or iterator containing the input keys to be sorted
+        d_in_values: Optional device array or iterator that contains each key's corresponding value
+        d_out_keys: Device array to store the sorted keys
+        d_out_values: Device array to store the sorted values
+        op: The comparison operator for sorting. The signature is  ``(T, T) -> int8``, where ``T`` is the input data type. See notes below.
+        compute_capability: Compute capability, or list of capabilities, to
+            build for ahead of time. Accepts a packed int (e.g. ``90``), a
+            ``(major, minor)`` pair, a string (e.g. ``"9.0"``), or a list
+            thereof. When ``None`` (the default), the current device's
+            architecture is used.
+
+    Returns:
+        A callable object that can be used to perform the merge sort
+
+    .. important::
+
+      The provided comparison operator must follow `strict weak ordering <https://en.cppreference.com/w/cpp/concepts/strict_weak_order.html>`_
+      semantics. For example, the comparator ``lambda lhs, rhs: lhs < rhs``  follows strict weak ordering,  but the comparator
+      ``lambda lhs, rhs: rhs >= lhs`` does not, because it is reflexive: ``r(x, x) == True``. Providing a comparator that does not
+      follow the required semantics can lead to incorrect results, silent memory corruption, or crashes.
+    """
+    op_adapter = make_op_adapter(op)
+    return _MergeSort(
+        d_in_keys,
+        d_in_values,
+        d_out_keys,
+        d_out_values,
+        op_adapter,
+        compute_capability=compute_capability,
+    )
+
+
+def merge_sort(
+    *,
+    d_in_keys: DeviceArrayLike | IteratorT,
+    d_in_values: DeviceArrayLike | IteratorT | None = None,
+    d_out_keys: DeviceArrayLike,
+    d_out_values: DeviceArrayLike | None = None,
+    num_items: int,
+    op: Operator,
+    stream=None,
+):
+    """
+    Performs device-wide merge sort.
+
+    This function automatically handles temporary storage allocation and execution.
+
+    Example:
+        Below, ``merge_sort`` is used to sort a sequence of keys inplace. It also rearranges the values according to the keys' order.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/sort/merge_sort_basic.py
+            :language: python
+            :start-after: # example-begin
+
+    Args:
+        d_in_keys: Device array or iterator containing the input sequence of keys
+        d_in_values: Device array or iterator containing the input sequence of values (optional)
+        d_out_keys: Device array to store the sorted keys
+        d_out_values: Device array to store the sorted values (optional)
+        num_items: Number of items to sort
+        op: The comparison operator for sorting. The signature is  ``(T, T) -> int8``, where ``T`` is the input data type. See notes below.
+        stream: CUDA stream for the operation (optional)
+
+    .. important::
+
+      The provided comparison operator must follow `strict weak ordering <https://en.cppreference.com/w/cpp/concepts/strict_weak_order.html>`_
+      semantics. For example, the comparator ``lambda lhs, rhs: lhs < rhs``  follows strict weak ordering,  but the comparator
+      ``lambda lhs, rhs: rhs >= lhs`` does not, because it is reflexive: ``r(x, x) == True``. Providing a comparator that does not
+      follow the required semantics can lead to incorrect results, silent memory corruption, or crashes.
+    """
+    sorter = make_merge_sort(
+        d_in_keys=d_in_keys,
+        d_in_values=d_in_values,
+        d_out_keys=d_out_keys,
+        d_out_values=d_out_values,
+        op=op,
+    )
+    tmp_storage_bytes = sorter(
+        temp_storage=None,
+        d_in_keys=d_in_keys,
+        d_in_values=d_in_values,
+        d_out_keys=d_out_keys,
+        d_out_values=d_out_values,
+        num_items=num_items,
+        op=op,
+        stream=stream,
+    )
+    tmp_storage = TempStorageBuffer(tmp_storage_bytes, stream)
+    sorter(
+        temp_storage=tmp_storage,
+        d_in_keys=d_in_keys,
+        d_in_values=d_in_values,
+        d_out_keys=d_out_keys,
+        d_out_values=d_out_values,
+        num_items=num_items,
+        op=op,
+        stream=stream,
+    )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_sort/_radix_sort.py b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_sort/_radix_sort.py
new file mode 100644
index 00000000..b2f80712
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_sort/_radix_sort.py
@@ -0,0 +1,289 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+from ... import _bindings
+from ... import _cccl_interop as cccl
+from ..._caching import cache_build_results, cache_with_registered_key_functions
+from ..._cccl_interop import set_cccl_iterator_state
+from ..._serialization import BUILD_RESULTS, ITER, OP, Serializable
+from ..._utils.protocols import (
+    get_data_pointer,
+    get_dtype,
+    validate_and_get_stream,
+)
+from ..._utils.temp_storage_buffer import TempStorageBuffer
+from ...typing import DeviceArrayLike
+from ._sort_common import DoubleBuffer, SortOrder, _get_arrays
+
+
+class _RadixSort(Serializable):
+    __slots__ = [
+        "_bound_build_result",
+        "d_in_keys_cccl",
+        "d_out_keys_cccl",
+        "d_in_values_cccl",
+        "d_out_values_cccl",
+        "decomposer_op",
+        "build_results",
+        "loaded_build_result",
+    ]
+
+    __serialization_schema__ = (
+        ("d_in_keys_cccl", ITER),
+        ("d_out_keys_cccl", ITER),
+        ("d_in_values_cccl", ITER),
+        ("d_out_values_cccl", ITER),
+        ("decomposer_op", OP),
+        ("build_results", BUILD_RESULTS(_bindings.DeviceRadixSortBuildResult)),
+    )
+
+    def __init__(
+        self,
+        d_in_keys: DeviceArrayLike | DoubleBuffer,
+        d_out_keys: DeviceArrayLike | None,
+        d_in_values: DeviceArrayLike | DoubleBuffer | None,
+        d_out_values: DeviceArrayLike | None,
+        order: SortOrder,
+        compute_capability=None,
+    ):
+        d_in_keys_array, d_out_keys_array, d_in_values_array, d_out_values_array = (
+            _get_arrays(d_in_keys, d_out_keys, d_in_values, d_out_values)
+        )
+
+        self.d_in_keys_cccl = cccl.to_cccl_input_iter(d_in_keys_array)
+        self.d_out_keys_cccl = cccl.to_cccl_output_iter(d_out_keys_array)
+        self.d_in_values_cccl = cccl.to_cccl_input_iter(d_in_values_array)
+        self.d_out_values_cccl = cccl.to_cccl_output_iter(d_out_values_array)
+
+        # TODO: decomposer op is not supported for now
+        self.decomposer_op = cccl.Op(
+            name="",
+            operator_type=cccl.OpKind.STATELESS,
+            ltoir=b"",
+            state_alignment=1,
+            state=b"",  # explicit empty bytes so the serialize path is byte-safe
+        )
+        decomposer_return_type = "".encode("utf-8")
+
+        build_order = (
+            _bindings.SortOrder.ASCENDING
+            if order is SortOrder.ASCENDING
+            else _bindings.SortOrder.DESCENDING
+        )
+        self.build_results, self._bound_build_result = cache_build_results(
+            _bindings.DeviceRadixSortBuildResult,
+            d_in_keys,
+            d_out_keys,
+            d_in_values,
+            d_out_values,
+            order,
+            compute_capability=compute_capability,
+            builder=lambda: cccl.build_for_ccs(
+                _bindings.DeviceRadixSortBuildResult,
+                build_order,
+                self.d_in_keys_cccl,
+                self.d_in_values_cccl,
+                self.decomposer_op,
+                decomposer_return_type,
+                compute_capability=compute_capability,
+            ),
+        )
+
+    def __call__(
+        self,
+        *,
+        temp_storage,
+        d_in_keys: DeviceArrayLike | DoubleBuffer,
+        d_out_keys: DeviceArrayLike | None,
+        d_in_values: DeviceArrayLike | DoubleBuffer | None,
+        d_out_values: DeviceArrayLike | None,
+        num_items: int,
+        begin_bit: int | None = None,
+        end_bit: int | None = None,
+        stream=None,
+    ):
+        # Select (and lazily load) the build result for the current device.
+        self.loaded_build_result = cccl.resolve_build_result(
+            self.build_results, self._bound_build_result
+        )
+
+        d_in_keys_array, d_out_keys_array, d_in_values_array, d_out_values_array = (
+            _get_arrays(d_in_keys, d_out_keys, d_in_values, d_out_values)
+        )
+
+        set_cccl_iterator_state(self.d_in_keys_cccl, d_in_keys_array)
+        if d_in_values_array is not None:
+            set_cccl_iterator_state(self.d_in_values_cccl, d_in_values_array)
+        set_cccl_iterator_state(self.d_out_keys_cccl, d_out_keys_array)
+        if d_out_values_array is not None:
+            set_cccl_iterator_state(self.d_out_values_cccl, d_out_values_array)
+
+        is_overwrite_okay = isinstance(d_in_keys, DoubleBuffer)
+
+        stream_handle = validate_and_get_stream(stream)
+        if temp_storage is None:
+            temp_storage_bytes = 0
+            d_temp_storage = 0
+        else:
+            temp_storage_bytes = temp_storage.nbytes
+            # Note: this is slightly slower, but supports all ndarray-like objects as long as they support CAI
+            # TODO: switch to use gpumemoryview once it's ready
+            d_temp_storage = get_data_pointer(temp_storage)
+
+        if begin_bit is None:
+            begin_bit = 0
+        if end_bit is None:
+            key_type = get_dtype(d_in_keys_array)
+            end_bit = key_type.itemsize * 8
+
+        selector = -1
+
+        temp_storage_bytes, selector = self.loaded_build_result.compute(
+            d_temp_storage,
+            temp_storage_bytes,
+            self.d_in_keys_cccl,
+            self.d_out_keys_cccl,
+            self.d_in_values_cccl,
+            self.d_out_values_cccl,
+            self.decomposer_op,
+            num_items,
+            begin_bit,
+            end_bit,
+            is_overwrite_okay,
+            selector,
+            stream_handle,
+        )
+
+        if is_overwrite_okay and temp_storage is not None:
+            assert selector in (0, 1)
+            assert isinstance(d_in_keys, DoubleBuffer)
+            d_in_keys.selector = selector
+            if d_in_values is not None:
+                assert isinstance(d_in_values, DoubleBuffer)
+                d_in_values.selector = selector
+
+        return temp_storage_bytes
+
+
+@cache_with_registered_key_functions
+def make_radix_sort(
+    *,
+    d_in_keys: DeviceArrayLike | DoubleBuffer,
+    d_out_keys: DeviceArrayLike | None,
+    d_in_values: DeviceArrayLike | DoubleBuffer | None,
+    d_out_values: DeviceArrayLike | None,
+    order: SortOrder,
+    compute_capability=None,
+):
+    """Implements a device-wide radix sort using ``d_in_keys`` in the requested order.
+
+    Example:
+        Below, ``make_radix_sort`` is used to create a radix sort object that can be reused.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/sort/radix_sort_object.py
+          :language: python
+          :start-after: # example-begin
+
+
+    Args:
+        d_in_keys: Device array or DoubleBuffer containing the input keys to be sorted
+        d_out_keys: Device array to store the sorted keys
+        d_in_values: Optional Device array or DoubleBuffer containing the input keys to be sorted
+        d_out_values: Device array to store the sorted values
+        op: Callable representing the comparison operator
+        compute_capability: Compute capability, or list of capabilities, to
+            build for ahead of time. Accepts a packed int (e.g. ``90``), a
+            ``(major, minor)`` pair, a string (e.g. ``"9.0"``), or a list
+            thereof. When ``None`` (the default), the current device's
+            architecture is used.
+
+    Returns:
+        A callable object that can be used to perform the radix sort
+    """
+    return _RadixSort(
+        d_in_keys,
+        d_out_keys,
+        d_in_values,
+        d_out_values,
+        order,
+        compute_capability=compute_capability,
+    )
+
+
+def radix_sort(
+    *,
+    d_in_keys: DeviceArrayLike | DoubleBuffer,
+    d_out_keys: DeviceArrayLike | None,
+    d_in_values: DeviceArrayLike | DoubleBuffer | None = None,
+    d_out_values: DeviceArrayLike | None = None,
+    num_items: int,
+    order: SortOrder,
+    begin_bit: int | None = None,
+    end_bit: int | None = None,
+    stream=None,
+):
+    """
+    Performs device-wide radix sort.
+
+    This function automatically handles temporary storage allocation and execution.
+
+    Example:
+        Below, ``radix_sort`` is used to sort a sequence of keys. It also rearranges the values according to the keys' order.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/sort/radix_sort_basic.py
+            :language: python
+            :start-after: # example-begin
+
+
+        In the following example, ``radix_sort`` is used to sort a sequence of keys with a ``DoubleBuffer`` for reduced temporary storage.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/sort/radix_sort_buffer.py
+            :language: python
+            :start-after: # example-begin
+
+
+    Args:
+        d_in_keys: Device array or DoubleBuffer containing the input sequence of keys
+        d_out_keys: Device array to store the sorted keys (optional)
+        d_in_values: Device array or DoubleBuffer containing the input sequence of values (optional)
+        d_out_values: Device array to store the sorted values (optional)
+        num_items: Number of items to sort
+        order: Sort order (ascending or descending)
+        begin_bit: Beginning bit position for comparison (optional)
+        end_bit: Ending bit position for comparison (optional)
+        stream: CUDA stream for the operation (optional)
+    """
+    sorter = make_radix_sort(
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_values,
+        d_out_values=d_out_values,
+        order=order,
+    )
+    tmp_storage_bytes = sorter(
+        temp_storage=None,
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_values,
+        d_out_values=d_out_values,
+        num_items=num_items,
+        begin_bit=begin_bit,
+        end_bit=end_bit,
+        stream=stream,
+    )
+    tmp_storage = TempStorageBuffer(tmp_storage_bytes, stream)
+    sorter(
+        temp_storage=tmp_storage,
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_values,
+        d_out_values=d_out_values,
+        num_items=num_items,
+        begin_bit=begin_bit,
+        end_bit=end_bit,
+        stream=stream,
+    )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_sort/_segmented_sort.py b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_sort/_segmented_sort.py
new file mode 100644
index 00000000..a89ccfb3
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_sort/_segmented_sort.py
@@ -0,0 +1,298 @@
+# Copyright (c) 2024-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+import numpy as np
+
+from ... import _bindings
+from ... import _cccl_interop as cccl
+from ..._caching import cache_build_results, cache_with_registered_key_functions
+from ..._cccl_interop import set_cccl_iterator_state
+from ..._serialization import BUILD_RESULTS, ITER, Serializable
+from ..._utils.protocols import (
+    get_data_pointer,
+    validate_and_get_stream,
+)
+from ..._utils.temp_storage_buffer import TempStorageBuffer
+from ...typing import DeviceArrayLike
+from ._sort_common import DoubleBuffer, SortOrder, _get_arrays
+
+
+class _SegmentedSort(Serializable):
+    __slots__ = [
+        "_bound_build_result",
+        "build_results",
+        "loaded_build_result",
+        "d_in_keys_cccl",
+        "d_out_keys_cccl",
+        "d_in_values_cccl",
+        "d_out_values_cccl",
+        "start_offsets_in_cccl",
+        "end_offsets_in_cccl",
+    ]
+
+    __serialization_schema__ = (
+        ("d_in_keys_cccl", ITER),
+        ("d_out_keys_cccl", ITER),
+        ("d_in_values_cccl", ITER),
+        ("d_out_values_cccl", ITER),
+        ("start_offsets_in_cccl", ITER),
+        ("end_offsets_in_cccl", ITER),
+        ("build_results", BUILD_RESULTS(_bindings.DeviceSegmentedSortBuildResult)),
+    )
+
+    def __init__(
+        self,
+        d_in_keys: DeviceArrayLike | DoubleBuffer,
+        d_out_keys: DeviceArrayLike | None,
+        d_in_values: DeviceArrayLike | DoubleBuffer | None,
+        d_out_values: DeviceArrayLike | None,
+        start_offsets_in: DeviceArrayLike,
+        end_offsets_in: DeviceArrayLike,
+        order: SortOrder,
+        compute_capability=None,
+    ):
+        d_in_keys_array, d_out_keys_array, d_in_values_array, d_out_values_array = (
+            _get_arrays(d_in_keys, d_out_keys, d_in_values, d_out_values)
+        )
+
+        self.d_in_keys_cccl = cccl.to_cccl_input_iter(d_in_keys_array)
+        self.d_out_keys_cccl = cccl.to_cccl_output_iter(d_out_keys_array)
+        self.d_in_values_cccl = cccl.to_cccl_input_iter(d_in_values_array)
+        self.d_out_values_cccl = cccl.to_cccl_output_iter(d_out_values_array)
+        self.start_offsets_in_cccl = cccl.to_cccl_input_iter(start_offsets_in)
+        self.end_offsets_in_cccl = cccl.to_cccl_input_iter(end_offsets_in)
+
+        build_order = (
+            _bindings.SortOrder.ASCENDING
+            if order is SortOrder.ASCENDING
+            else _bindings.SortOrder.DESCENDING
+        )
+        self.build_results, self._bound_build_result = cache_build_results(
+            _bindings.DeviceSegmentedSortBuildResult,
+            d_in_keys,
+            d_out_keys,
+            d_in_values,
+            d_out_values,
+            start_offsets_in,
+            end_offsets_in,
+            order,
+            compute_capability=compute_capability,
+            builder=lambda: cccl.build_for_ccs(
+                _bindings.DeviceSegmentedSortBuildResult,
+                build_order,
+                self.d_in_keys_cccl,
+                self.d_in_values_cccl,
+                self.start_offsets_in_cccl,
+                self.end_offsets_in_cccl,
+                compute_capability=compute_capability,
+            ),
+        )
+
+    def __call__(
+        self,
+        *,
+        temp_storage,
+        d_in_keys,
+        d_out_keys,
+        d_in_values,
+        d_out_values,
+        num_items,
+        num_segments,
+        start_offsets_in,
+        end_offsets_in,
+        stream=None,
+    ):
+        # Select (and lazily load) the build result for the current device.
+        self.loaded_build_result = cccl.resolve_build_result(
+            self.build_results, self._bound_build_result
+        )
+
+        if num_segments > np.iinfo(np.int32).max:
+            raise RuntimeError(
+                "Segmented sort does not currently support more than 2^31-1 segments."
+            )
+        d_in_keys_array, d_out_keys_array, d_in_values_array, d_out_values_array = (
+            _get_arrays(d_in_keys, d_out_keys, d_in_values, d_out_values)
+        )
+
+        set_cccl_iterator_state(self.d_in_keys_cccl, d_in_keys_array)
+        set_cccl_iterator_state(self.d_out_keys_cccl, d_out_keys_array)
+        if d_in_values_array is not None:
+            set_cccl_iterator_state(self.d_in_values_cccl, d_in_values_array)
+        if d_out_values_array is not None:
+            set_cccl_iterator_state(self.d_out_values_cccl, d_out_values_array)
+        set_cccl_iterator_state(self.start_offsets_in_cccl, start_offsets_in)
+        set_cccl_iterator_state(self.end_offsets_in_cccl, end_offsets_in)
+
+        stream_handle = validate_and_get_stream(stream)
+        if temp_storage is None:
+            temp_storage_bytes = 0
+            d_temp_storage = 0
+        else:
+            temp_storage_bytes = temp_storage.nbytes
+            d_temp_storage = get_data_pointer(temp_storage)
+
+        # Detect overwrite mode and selector, similar to radix sort
+        is_overwrite_okay = isinstance(d_in_keys, DoubleBuffer)
+        selector = -1
+
+        temp_storage_bytes, selector = self.loaded_build_result.compute(
+            d_temp_storage,
+            temp_storage_bytes,
+            self.d_in_keys_cccl,
+            self.d_out_keys_cccl,
+            self.d_in_values_cccl,
+            self.d_out_values_cccl,
+            num_items,
+            num_segments,
+            self.start_offsets_in_cccl,
+            self.end_offsets_in_cccl,
+            is_overwrite_okay,
+            selector,
+            stream_handle,
+        )
+
+        if is_overwrite_okay and temp_storage is not None:
+            assert selector in (0, 1)
+            assert isinstance(d_in_keys, DoubleBuffer)
+            d_in_keys.selector = selector
+            if d_in_values is not None:
+                assert isinstance(d_in_values, DoubleBuffer)
+                d_in_values.selector = selector
+
+        return temp_storage_bytes
+
+
+@cache_with_registered_key_functions
+def make_segmented_sort(
+    *,
+    d_in_keys: DeviceArrayLike | DoubleBuffer,
+    d_out_keys: DeviceArrayLike | None = None,
+    d_in_values: DeviceArrayLike | DoubleBuffer | None = None,
+    d_out_values: DeviceArrayLike | None = None,
+    start_offsets_in: DeviceArrayLike,
+    end_offsets_in: DeviceArrayLike,
+    order: SortOrder,
+    compute_capability=None,
+):
+    """
+    Performs a device-wide segmented sort using the specified keys and values.
+
+    Example:
+        Below, ``make_segmented_sort`` is used to create a segmented sort object that can be reused.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/sort/segmented_sort_object.py
+            :language: python
+            :start-after: # example-begin
+
+    Args:
+        d_in_keys: Device array or DoubleBuffer containing the input keys to be sorted
+        d_out_keys: Device array to store the sorted keys
+        d_in_values: Optional Device array or DoubleBuffer containing the input values to be sorted
+        d_out_values: Device array to store the sorted values
+        start_offsets_in: Device array or iterator containing the sequence of beginning offsets
+        end_offsets_in: Device array or iterator containing the sequence of ending offsets
+        order: SortOrder specifying the order of the sort
+        compute_capability: Compute capability, or list of capabilities, to
+            build for ahead of time. Accepts a packed int (e.g. ``90``), a
+            ``(major, minor)`` pair, a string (e.g. ``"9.0"``), or a list
+            thereof. When ``None`` (the default), the current device's
+            architecture is used.
+
+    Returns:
+        A callable object that can be used to perform the segmented sort
+    """
+    return _SegmentedSort(
+        d_in_keys,
+        d_out_keys,
+        d_in_values,
+        d_out_values,
+        start_offsets_in,
+        end_offsets_in,
+        order,
+        compute_capability=compute_capability,
+    )
+
+
+def segmented_sort(
+    *,
+    d_in_keys: DeviceArrayLike | DoubleBuffer,
+    d_out_keys: DeviceArrayLike | None = None,
+    d_in_values: DeviceArrayLike | DoubleBuffer | None = None,
+    d_out_values: DeviceArrayLike | None = None,
+    num_items: int,
+    num_segments: int,
+    start_offsets_in: DeviceArrayLike,
+    end_offsets_in: DeviceArrayLike,
+    order: SortOrder,
+    stream=None,
+):
+    """
+    Performs device-wide segmented sort.
+
+    This function automatically handles temporary storage allocation and execution.
+
+    Example:
+        Below, ``segmented_sort`` is used to perform a segmented sort. It also rearranges the values according to the keys' order.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/sort/segmented_sort_basic.py
+            :language: python
+            :start-after: # example-begin
+
+
+        In the following example, ``segmented_sort`` is used to perform a segmented sort with a ``DoubleBuffer`` for reduced temporary storage.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/sort/segmented_sort_buffer.py
+            :language: python
+            :start-after: # example-begin
+
+    Args:
+        d_in_keys: Device array or DoubleBuffer containing the input keys to be sorted
+        d_out_keys: Device array to store the sorted keys (optional)
+        d_in_values: Device array or DoubleBuffer containing the input values to be sorted (optional)
+        d_out_values: Device array to store the sorted values (optional)
+        num_items: Total number of items to sort
+        num_segments: Number of segments to sort
+        start_offsets_in: Device array or iterator containing the sequence of beginning offsets
+        end_offsets_in: Device array or iterator containing the sequence of ending offsets
+        order: Sort order (ascending or descending)
+        stream: CUDA stream for the operation (optional)
+    """
+    sorter = make_segmented_sort(
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_values,
+        d_out_values=d_out_values,
+        start_offsets_in=start_offsets_in,
+        end_offsets_in=end_offsets_in,
+        order=order,
+    )
+    tmp_storage_bytes = sorter(
+        temp_storage=None,
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_values,
+        d_out_values=d_out_values,
+        num_items=num_items,
+        num_segments=num_segments,
+        start_offsets_in=start_offsets_in,
+        end_offsets_in=end_offsets_in,
+        stream=stream,
+    )
+    tmp_storage = TempStorageBuffer(tmp_storage_bytes, stream)
+    sorter(
+        temp_storage=tmp_storage,
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_values,
+        d_out_values=d_out_values,
+        num_items=num_items,
+        num_segments=num_segments,
+        start_offsets_in=start_offsets_in,
+        end_offsets_in=end_offsets_in,
+        stream=stream,
+    )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_sort/_sort_common.py b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_sort/_sort_common.py
new file mode 100644
index 00000000..be447e50
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_sort/_sort_common.py
@@ -0,0 +1,62 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+from enum import Enum
+from typing import Tuple
+
+from ..._caching import cache_with_registered_key_functions
+from ..._utils.protocols import get_dtype
+from ...typing import DeviceArrayLike
+
+
+class SortOrder(Enum):
+    ASCENDING = 0
+    DESCENDING = 1
+
+
+class DoubleBuffer:
+    def __init__(self, d_current: DeviceArrayLike, d_alternate: DeviceArrayLike):
+        self.d_buffers = [d_current, d_alternate]
+        self.selector = 0
+
+    def current(self):
+        return self.d_buffers[self.selector]
+
+    def alternate(self):
+        return self.d_buffers[1 - self.selector]
+
+
+def _get_arrays(
+    d_in_keys: DeviceArrayLike | DoubleBuffer,
+    d_out_keys: DeviceArrayLike | None,
+    d_in_values: DeviceArrayLike | DoubleBuffer | None,
+    d_out_values: DeviceArrayLike | None,
+) -> Tuple[DeviceArrayLike, DeviceArrayLike, DeviceArrayLike, DeviceArrayLike]:
+    if isinstance(d_in_keys, DoubleBuffer):
+        d_in_keys_array = d_in_keys.current()
+        d_out_keys_array = d_in_keys.alternate()
+
+        if d_in_values is not None:
+            assert isinstance(d_in_values, DoubleBuffer)
+            d_in_values_array = d_in_values.current()
+            d_out_values_array = d_in_values.alternate()
+        else:
+            d_in_values_array = None
+            d_out_values_array = None
+    else:
+        d_in_keys_array = d_in_keys
+        d_in_values_array = d_in_values
+        d_out_keys_array = d_out_keys
+        d_out_values_array = d_out_values
+
+    return d_in_keys_array, d_out_keys_array, d_in_values_array, d_out_values_array
+
+
+# DoubleBuffer: extract dtype from current buffer
+cache_with_registered_key_functions.register(
+    DoubleBuffer, lambda buf: get_dtype(buf.current())
+)
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_three_way_partition.py b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_three_way_partition.py
new file mode 100644
index 00000000..6099b662
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_three_way_partition.py
@@ -0,0 +1,289 @@
+# Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+from typing import Callable
+
+from .. import _bindings, types
+from .. import _cccl_interop as cccl
+from .._caching import cache_build_results, cache_with_registered_key_functions
+from .._cccl_interop import set_cccl_iterator_state
+from .._serialization import BUILD_RESULTS, ITER, OP, Serializable
+from .._utils import protocols
+from .._utils.temp_storage_buffer import TempStorageBuffer
+from ..op import OpAdapter, make_op_adapter
+from ..typing import DeviceArrayLike, IteratorT, Operator
+
+
+class _ThreeWayPartition(Serializable):
+    __slots__ = [
+        "_bound_build_result",
+        "build_results",
+        "loaded_build_result",
+        "d_in_cccl",
+        "d_first_part_out_cccl",
+        "d_second_part_out_cccl",
+        "d_unselected_out_cccl",
+        "d_num_selected_out_cccl",
+        "select_first_part_op_cccl",
+        "select_second_part_op_cccl",
+    ]
+
+    __serialization_schema__ = (
+        ("d_in_cccl", ITER),
+        ("d_first_part_out_cccl", ITER),
+        ("d_second_part_out_cccl", ITER),
+        ("d_unselected_out_cccl", ITER),
+        ("d_num_selected_out_cccl", ITER),
+        ("select_first_part_op_cccl", OP),
+        ("select_second_part_op_cccl", OP),
+        ("build_results", BUILD_RESULTS(_bindings.DeviceThreeWayPartitionBuildResult)),
+    )
+
+    def __init__(
+        self,
+        d_in: DeviceArrayLike | IteratorT,
+        d_first_part_out: DeviceArrayLike | IteratorT,
+        d_second_part_out: DeviceArrayLike | IteratorT,
+        d_unselected_out: DeviceArrayLike | IteratorT,
+        d_num_selected_out: DeviceArrayLike | IteratorT,
+        select_first_part_op: OpAdapter,
+        select_second_part_op: OpAdapter,
+        compute_capability=None,
+    ):
+        self.d_in_cccl = cccl.to_cccl_input_iter(d_in)
+        self.d_first_part_out_cccl = cccl.to_cccl_output_iter(d_first_part_out)
+        self.d_second_part_out_cccl = cccl.to_cccl_output_iter(d_second_part_out)
+        self.d_unselected_out_cccl = cccl.to_cccl_output_iter(d_unselected_out)
+        self.d_num_selected_out_cccl = cccl.to_cccl_output_iter(d_num_selected_out)
+
+        # Compile ops - partition predicates return uint8 (boolean)
+        value_type = cccl.get_value_type(d_in)
+        self.select_first_part_op_cccl = select_first_part_op.compile(
+            (value_type,), types.uint8
+        )
+        self.select_second_part_op_cccl = select_second_part_op.compile(
+            (value_type,), types.uint8
+        )
+
+        self.build_results, self._bound_build_result = cache_build_results(
+            _bindings.DeviceThreeWayPartitionBuildResult,
+            d_in,
+            d_first_part_out,
+            d_second_part_out,
+            d_unselected_out,
+            d_num_selected_out,
+            select_first_part_op,
+            select_second_part_op,
+            compute_capability=compute_capability,
+            builder=lambda: cccl.build_for_ccs(
+                _bindings.DeviceThreeWayPartitionBuildResult,
+                self.d_in_cccl,
+                self.d_first_part_out_cccl,
+                self.d_second_part_out_cccl,
+                self.d_unselected_out_cccl,
+                self.d_num_selected_out_cccl,
+                self.select_first_part_op_cccl,
+                self.select_second_part_op_cccl,
+                compute_capability=compute_capability,
+            ),
+        )
+
+    def __call__(
+        self,
+        *,
+        temp_storage,
+        d_in,
+        d_first_part_out,
+        d_second_part_out,
+        d_unselected_out,
+        d_num_selected_out,
+        select_first_part_op: Callable | OpAdapter,
+        select_second_part_op: Callable | OpAdapter,
+        num_items: int,
+        stream=None,
+    ):
+        # Select (and lazily load) the build result for the current device.
+        self.loaded_build_result = cccl.resolve_build_result(
+            self.build_results, self._bound_build_result
+        )
+
+        set_cccl_iterator_state(self.d_in_cccl, d_in)
+        set_cccl_iterator_state(self.d_first_part_out_cccl, d_first_part_out)
+        set_cccl_iterator_state(self.d_second_part_out_cccl, d_second_part_out)
+        set_cccl_iterator_state(self.d_unselected_out_cccl, d_unselected_out)
+        set_cccl_iterator_state(self.d_num_selected_out_cccl, d_num_selected_out)
+
+        first_op_adapter = make_op_adapter(select_first_part_op)
+        second_op_adapter = make_op_adapter(select_second_part_op)
+        self.select_first_part_op_cccl.state = first_op_adapter.get_state()
+        self.select_second_part_op_cccl.state = second_op_adapter.get_state()
+
+        stream_handle = protocols.validate_and_get_stream(stream)
+
+        if temp_storage is None:
+            temp_storage_bytes = 0
+            d_temp_storage = 0
+        else:
+            temp_storage_bytes = temp_storage.nbytes
+            d_temp_storage = protocols.get_data_pointer(temp_storage)
+
+        temp_storage_bytes = self.loaded_build_result.compute(
+            d_temp_storage,
+            temp_storage_bytes,
+            self.d_in_cccl,
+            self.d_first_part_out_cccl,
+            self.d_second_part_out_cccl,
+            self.d_unselected_out_cccl,
+            self.d_num_selected_out_cccl,
+            self.select_first_part_op_cccl,
+            self.select_second_part_op_cccl,
+            num_items,
+            stream_handle,
+        )
+        return temp_storage_bytes
+
+
+@cache_with_registered_key_functions
+def make_three_way_partition(
+    *,
+    d_in: DeviceArrayLike | IteratorT,
+    d_first_part_out: DeviceArrayLike | IteratorT,
+    d_second_part_out: DeviceArrayLike | IteratorT,
+    d_unselected_out: DeviceArrayLike | IteratorT,
+    d_num_selected_out: DeviceArrayLike | IteratorT,
+    select_first_part_op: Operator,
+    select_second_part_op: Operator,
+    compute_capability=None,
+):
+    """
+    Computes a device-wide three-way partition using the specified unary ``select_first_part_op`` and ``select_second_part_op`` operators.
+
+    Example:
+        Below, ``make_three_way_partition`` is used to create a three-way partition object that can be reused.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/partition/three_way_partition_object.py
+            :language: python
+            :start-after: # example-begin
+
+    Args:
+        d_in: Device array or iterator containing the input sequence of data items
+        d_first_part_out: Device array or iterator to store the first part of the output
+        d_second_part_out: Device array or iterator to store the second part of the output
+        d_unselected_out: Device array or iterator to store the unselected items
+        d_num_selected_out: Device array to store the number of items selected. The total number of items selected by ``select_first_part_op`` and ``select_second_part_op`` is stored in ``d_num_selected_out[0]`` and ``d_num_selected_out[1]``, respectively.
+        select_first_part_op: Unary operator to select the first part.
+            The signature is ``(T) -> uint8``, where ``T`` is the input data type.
+            Returns 1 (selected) or 0 (not selected).
+            Can reference device arrays as globals/closures - they will be automatically captured.
+        select_second_part_op: Unary operator to select the second part.
+            The signature is ``(T) -> uint8``, where ``T`` is the input data type.
+            Returns 1 (selected) or 0 (not selected).
+            Can reference device arrays as globals/closures - they will be automatically captured.
+        compute_capability: Compute capability, or list of capabilities, to
+            build for ahead of time. Accepts a packed int (e.g. ``90``), a
+            ``(major, minor)`` pair, a string (e.g. ``"9.0"``), or a list
+            thereof. When ``None`` (the default), the current device's
+            architecture is used.
+
+    Returns:
+        A callable object that can be used to perform the three-way partition
+    """
+    first_op_adapter = make_op_adapter(select_first_part_op)
+    second_op_adapter = make_op_adapter(select_second_part_op)
+
+    return _ThreeWayPartition(
+        d_in,
+        d_first_part_out,
+        d_second_part_out,
+        d_unselected_out,
+        d_num_selected_out,
+        first_op_adapter,
+        second_op_adapter,
+        compute_capability=compute_capability,
+    )
+
+
+def three_way_partition(
+    *,
+    d_in: DeviceArrayLike | IteratorT,
+    d_first_part_out: DeviceArrayLike | IteratorT,
+    d_second_part_out: DeviceArrayLike | IteratorT,
+    d_unselected_out: DeviceArrayLike | IteratorT,
+    d_num_selected_out: DeviceArrayLike | IteratorT,
+    select_first_part_op: Operator,
+    select_second_part_op: Operator,
+    num_items: int,
+    stream=None,
+):
+    """
+    Performs device-wide three-way partition. Given an input sequence of data items, it partitions the items into three parts:
+    - The first part is selected by the ``select_first_part_op`` operator.
+    - The second part is selected by the ``select_second_part_op`` operator.
+    - The unselected items are not selected by either operator.
+
+    This function automatically handles temporary storage allocation and execution.
+
+    Example:
+        Below, ``three_way_partition`` is used to partition a sequence of integers into three parts.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/partition/three_way_partition_basic.py
+            :language: python
+            :start-after: # example-begin
+
+    Args:
+        d_in: Device array or iterator containing the input sequence of data items
+        d_first_part_out: Device array or iterator to store the first part of the output
+        d_second_part_out: Device array or iterator to store the second part of the output
+        d_unselected_out: Device array or iterator to store the unselected items
+        d_num_selected_out: Device array to store the number of items selected. The total number of items selected by ``select_first_part_op`` and ``select_second_part_op`` is stored in ``d_num_selected_out[0]`` and ``d_num_selected_out[1]``, respectively.
+        select_first_part_op: Unary operator to select the first part.
+            The signature is ``(T) -> uint8``, where ``T`` is the input data type.
+            Returns 1 (selected) or 0 (not selected).
+        select_second_part_op: Unary operator to select the second part.
+            The signature is ``(T) -> uint8``, where ``T`` is the input data type.
+            Returns 1 (selected) or 0 (not selected).
+        num_items: Number of items to partition
+        stream: CUDA stream for the operation (optional)
+    """
+    # Create adapters to support stateful ops
+    first_op_adapter = make_op_adapter(select_first_part_op)
+    second_op_adapter = make_op_adapter(select_second_part_op)
+
+    partitioner = make_three_way_partition(
+        d_in=d_in,
+        d_first_part_out=d_first_part_out,
+        d_second_part_out=d_second_part_out,
+        d_unselected_out=d_unselected_out,
+        d_num_selected_out=d_num_selected_out,
+        select_first_part_op=first_op_adapter,
+        select_second_part_op=second_op_adapter,
+    )
+    tmp_storage_bytes = partitioner(
+        temp_storage=None,
+        d_in=d_in,
+        d_first_part_out=d_first_part_out,
+        d_second_part_out=d_second_part_out,
+        d_unselected_out=d_unselected_out,
+        d_num_selected_out=d_num_selected_out,
+        select_first_part_op=first_op_adapter,
+        select_second_part_op=second_op_adapter,
+        num_items=num_items,
+        stream=stream,
+    )
+    tmp_storage = TempStorageBuffer(tmp_storage_bytes, stream)
+    partitioner(
+        temp_storage=tmp_storage,
+        d_in=d_in,
+        d_first_part_out=d_first_part_out,
+        d_second_part_out=d_second_part_out,
+        d_unselected_out=d_unselected_out,
+        d_num_selected_out=d_num_selected_out,
+        select_first_part_op=first_op_adapter,
+        select_second_part_op=second_op_adapter,
+        num_items=num_items,
+        stream=stream,
+    )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_transform.py b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_transform.py
new file mode 100644
index 00000000..c55ca5d7
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_transform.py
@@ -0,0 +1,371 @@
+# Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+from typing import Callable
+
+from .. import _bindings
+from .. import _cccl_interop as cccl
+from .._caching import cache_build_results, cache_with_registered_key_functions
+from .._cccl_interop import set_cccl_iterator_state
+from .._serialization import BUILD_RESULTS, ITER, OP, Serializable
+from .._utils import protocols
+from ..op import OpAdapter, make_op_adapter
+from ..typing import DeviceArrayLike, IteratorT, Operator
+
+
+class _UnaryTransform(Serializable):
+    __slots__ = [
+        "_bound_build_result",
+        "d_in_cccl",
+        "d_out_cccl",
+        "op_cccl",
+        "build_results",
+        "loaded_build_result",
+    ]
+
+    __serialization_schema__ = (
+        ("d_in_cccl", ITER),
+        ("d_out_cccl", ITER),
+        ("op_cccl", OP),
+        ("build_results", BUILD_RESULTS(_bindings.DeviceUnaryTransform)),
+    )
+
+    def __init__(
+        self,
+        d_in: DeviceArrayLike | IteratorT,
+        d_out: DeviceArrayLike | IteratorT,
+        op: OpAdapter,
+        compute_capability=None,
+    ):
+        self.d_in_cccl = cccl.to_cccl_input_iter(d_in)
+        self.d_out_cccl = cccl.to_cccl_output_iter(d_out)
+
+        # Compile the op with input/output types
+        in_type = cccl.get_value_type(d_in)
+        out_type = cccl.get_value_type(d_out)
+        self.op_cccl = op.compile((in_type,), out_type)
+
+        self.build_results, self._bound_build_result = cache_build_results(
+            _bindings.DeviceUnaryTransform,
+            d_in,
+            d_out,
+            op,
+            compute_capability=compute_capability,
+            builder=lambda: cccl.build_for_ccs(
+                _bindings.DeviceUnaryTransform,
+                self.d_in_cccl,
+                self.d_out_cccl,
+                self.op_cccl,
+                compute_capability=compute_capability,
+            ),
+        )
+
+    def __call__(
+        self,
+        *,
+        d_in,
+        d_out,
+        op: Callable | OpAdapter,
+        num_items: int,
+        stream=None,
+    ):
+        # Select (and lazily load) the build result for the current device.
+        self.loaded_build_result = cccl.resolve_build_result(
+            self.build_results, self._bound_build_result
+        )
+
+        op_adapter = make_op_adapter(op)
+
+        set_cccl_iterator_state(self.d_in_cccl, d_in)
+        set_cccl_iterator_state(self.d_out_cccl, d_out)
+        self.op_cccl.state = op_adapter.get_state()
+
+        stream_handle = protocols.validate_and_get_stream(stream)
+        self.loaded_build_result.compute(
+            self.d_in_cccl,
+            self.d_out_cccl,
+            num_items,
+            self.op_cccl,
+            stream_handle,
+        )
+        return None
+
+
+class _BinaryTransform(Serializable):
+    __slots__ = [
+        "_bound_build_result",
+        "d_in1_cccl",
+        "d_in2_cccl",
+        "d_out_cccl",
+        "op_cccl",
+        "build_results",
+        "loaded_build_result",
+    ]
+
+    __serialization_schema__ = (
+        ("d_in1_cccl", ITER),
+        ("d_in2_cccl", ITER),
+        ("d_out_cccl", ITER),
+        ("op_cccl", OP),
+        ("build_results", BUILD_RESULTS(_bindings.DeviceBinaryTransform)),
+    )
+
+    def __init__(
+        self,
+        d_in1: DeviceArrayLike | IteratorT,
+        d_in2: DeviceArrayLike | IteratorT,
+        d_out: DeviceArrayLike | IteratorT,
+        op: OpAdapter,
+        compute_capability=None,
+    ):
+        self.d_in1_cccl = cccl.to_cccl_input_iter(d_in1)
+        self.d_in2_cccl = cccl.to_cccl_input_iter(d_in2)
+        self.d_out_cccl = cccl.to_cccl_output_iter(d_out)
+
+        # Compile the op with input/output types
+        in1_type = cccl.get_value_type(d_in1)
+        in2_type = cccl.get_value_type(d_in2)
+        out_type = cccl.get_value_type(d_out)
+        self.op_cccl = op.compile((in1_type, in2_type), out_type)
+
+        self.build_results, self._bound_build_result = cache_build_results(
+            _bindings.DeviceBinaryTransform,
+            d_in1,
+            d_in2,
+            d_out,
+            op,
+            compute_capability=compute_capability,
+            builder=lambda: cccl.build_for_ccs(
+                _bindings.DeviceBinaryTransform,
+                self.d_in1_cccl,
+                self.d_in2_cccl,
+                self.d_out_cccl,
+                self.op_cccl,
+                compute_capability=compute_capability,
+            ),
+        )
+
+    def __call__(
+        self,
+        *,
+        d_in1,
+        d_in2,
+        d_out,
+        op: Callable | OpAdapter,
+        num_items: int,
+        stream=None,
+    ):
+        # Select (and lazily load) the build result for the current device.
+        self.loaded_build_result = cccl.resolve_build_result(
+            self.build_results, self._bound_build_result
+        )
+
+        set_cccl_iterator_state(self.d_in1_cccl, d_in1)
+        set_cccl_iterator_state(self.d_in2_cccl, d_in2)
+        set_cccl_iterator_state(self.d_out_cccl, d_out)
+
+        op_adapter = make_op_adapter(op)
+        self.op_cccl.state = op_adapter.get_state()
+
+        stream_handle = protocols.validate_and_get_stream(stream)
+        self.loaded_build_result.compute(
+            self.d_in1_cccl,
+            self.d_in2_cccl,
+            self.d_out_cccl,
+            num_items,
+            self.op_cccl,
+            stream_handle,
+        )
+        return None
+
+
+@cache_with_registered_key_functions
+def make_unary_transform(
+    *,
+    d_in: DeviceArrayLike | IteratorT,
+    d_out: DeviceArrayLike | IteratorT,
+    op: Operator,
+    compute_capability=None,
+):
+    """
+    Create a unary transform object that can be called to apply a transformation
+    to each element of the input according to the unary operation ``op``.
+
+    This is the object-oriented API that allows explicit control over temporary
+    storage allocation. For simpler usage, consider using :func:`unary_transform`.
+
+    Example:
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/transform/unary_transform_object.py
+           :language: python
+           :start-after: # example-begin
+
+
+    Args:
+        d_in: Device array or iterator containing the input sequence of data items.
+        d_out: Device array or iterator to store the result of the transformation.
+        op: Unary operation to apply to each element.
+            The signature is ``(T) -> U``, where ``T`` is
+            the input data type and ``U`` is the output data type.
+        compute_capability: Compute capability, or list of capabilities, to
+            build for ahead of time. Accepts a packed int (e.g. ``90``), a
+            ``(major, minor)`` pair, a string (e.g. ``"9.0"``), or a list
+            thereof. When ``None`` (the default), the current device's
+            architecture is used.
+
+    Returns:
+        A callable object that performs the transformation.
+    """
+    op_adapter = make_op_adapter(op)
+    return _UnaryTransform(
+        d_in, d_out, op_adapter, compute_capability=compute_capability
+    )
+
+
+@cache_with_registered_key_functions
+def make_binary_transform(
+    *,
+    d_in1: DeviceArrayLike | IteratorT,
+    d_in2: DeviceArrayLike | IteratorT,
+    d_out: DeviceArrayLike | IteratorT,
+    op: Operator,
+    compute_capability=None,
+):
+    """
+    Create a binary transform object that can be called to apply a transformation
+    to the given pair of input sequences according to the binary operation ``op``.
+
+    This is the object-oriented API that allows explicit control over temporary
+    storage allocation. For simpler usage, consider using :func:`binary_transform`.
+
+    Example:
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/transform/binary_transform_object.py
+           :language: python
+           :start-after: # example-begin
+
+
+    Args:
+        d_in1: Device array or iterator containing the first input sequence of data items.
+        d_in2: Device array or iterator containing the second input sequence of data items.
+        d_out: Device array or iterator to store the result of the transformation.
+        op: Binary operation.
+            The signature is ``(T1, T2) -> U``, where ``T1`` and ``T2`` are the input data types and
+            ``U`` is the output data type.
+        compute_capability: Compute capability, or list of capabilities, to
+            build for ahead of time. Accepts a packed int (e.g. ``90``), a
+            ``(major, minor)`` pair, a string (e.g. ``"9.0"``), or a list
+            thereof. When ``None`` (the default), the current device's
+            architecture is used.
+
+    Returns:
+        A callable object that performs the transformation.
+    """
+    op_adapter = make_op_adapter(op)
+    return _BinaryTransform(
+        d_in1, d_in2, d_out, op_adapter, compute_capability=compute_capability
+    )
+
+
+def unary_transform(
+    *,
+    d_in: DeviceArrayLike | IteratorT,
+    d_out: DeviceArrayLike | IteratorT,
+    op: Operator,
+    num_items: int,
+    stream=None,
+):
+    """
+    Performs device-wide unary transform.
+
+    This function automatically handles temporary storage allocation and execution.
+
+    The ``op`` function can reference device arrays as globals or closures - they will
+    be automatically captured as state arrays, enabling stateful operations like counting.
+
+    Example:
+        Below, ``unary_transform`` is used to apply a transformation to each element of the input.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/transform/unary_transform_basic.py
+           :language: python
+           :start-after: # example-begin
+
+        When working with custom struct types, you need to provide type annotations
+        to help with type inference. See the binary transform struct example for reference:
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/struct/struct_transform.py
+           :language: python
+           :start-after: # example-begin
+
+
+    Args:
+        d_in: Device array or iterator containing the input sequence of data items.
+        d_out: Device array or iterator to store the result of the transformation.
+        op: Unary operation to apply to each element.
+            The signature is ``(T) -> U``, where ``T`` is
+            the input data type and ``U`` is the output data type.
+            Can reference device arrays as globals/closures - they will be automatically captured.
+        num_items: Number of items to transform.
+        stream: CUDA stream to use for the operation.
+    """
+    op_adapter = make_op_adapter(op)
+    transformer = make_unary_transform(d_in=d_in, d_out=d_out, op=op_adapter)
+    transformer(
+        d_in=d_in, d_out=d_out, op=op_adapter, num_items=num_items, stream=stream
+    )
+
+
+def binary_transform(
+    *,
+    d_in1: DeviceArrayLike | IteratorT,
+    d_in2: DeviceArrayLike | IteratorT,
+    d_out: DeviceArrayLike | IteratorT,
+    op: Operator,
+    num_items: int,
+    stream=None,
+):
+    """
+    Performs device-wide binary transform.
+
+    This function automatically handles temporary storage allocation and execution.
+
+    Example:
+        Below, ``binary_transform`` is used to apply a transformation to pairs of elements from two input sequences.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/transform/binary_transform_basic.py
+           :language: python
+           :start-after: # example-begin
+
+        When working with custom struct types, you need to provide type annotations
+        to help with type inference. See the following example:
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/struct/struct_transform.py
+           :language: python
+           :start-after: # example-begin
+
+
+    Args:
+        d_in1: Device array or iterator containing the first input sequence of data items.
+        d_in2: Device array or iterator containing the second input sequence of data items.
+        d_out: Device array or iterator to store the result of the transformation.
+        op: Binary operation.
+            The signature is ``(T1, T2) -> U``, where ``T1`` and ``T2`` are the input data types and
+            ``U`` is the output data type.
+            Can reference device arrays as globals/closures - they will be automatically captured.
+        num_items: Number of items to transform.
+        stream: CUDA stream to use for the operation.
+    """
+    op_adapter = make_op_adapter(op)
+    transformer = make_binary_transform(
+        d_in1=d_in1, d_in2=d_in2, d_out=d_out, op=op_adapter
+    )
+    transformer(
+        d_in1=d_in1,
+        d_in2=d_in2,
+        d_out=d_out,
+        op=op_adapter,
+        num_items=num_items,
+        stream=stream,
+    )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_unique_by_key.py b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_unique_by_key.py
new file mode 100644
index 00000000..0bd0772a
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/algorithms/_unique_by_key.py
@@ -0,0 +1,253 @@
+# Copyright (c) 2024-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+
+from __future__ import annotations
+
+from .. import _bindings, types
+from .. import _cccl_interop as cccl
+from .._caching import cache_build_results, cache_with_registered_key_functions
+from .._cccl_interop import set_cccl_iterator_state
+from .._serialization import BUILD_RESULTS, ITER, OP, Serializable
+from .._utils.protocols import (
+    get_data_pointer,
+    validate_and_get_stream,
+)
+from .._utils.temp_storage_buffer import TempStorageBuffer
+from ..op import OpAdapter, make_op_adapter
+from ..typing import DeviceArrayLike, IteratorT, Operator
+
+
+class _UniqueByKey(Serializable):
+    __slots__ = [
+        "_bound_build_result",
+        "build_results",
+        "loaded_build_result",
+        "d_in_keys_cccl",
+        "d_in_items_cccl",
+        "d_out_keys_cccl",
+        "d_out_items_cccl",
+        "d_out_num_selected_cccl",
+        "op_cccl",
+    ]
+
+    __serialization_schema__ = (
+        ("d_in_keys_cccl", ITER),
+        ("d_in_items_cccl", ITER),
+        ("d_out_keys_cccl", ITER),
+        ("d_out_items_cccl", ITER),
+        ("d_out_num_selected_cccl", ITER),
+        ("op_cccl", OP),
+        ("build_results", BUILD_RESULTS(_bindings.DeviceUniqueByKeyBuildResult)),
+    )
+
+    def __init__(
+        self,
+        d_in_keys: DeviceArrayLike | IteratorT,
+        d_in_items: DeviceArrayLike | IteratorT,
+        d_out_keys: DeviceArrayLike | IteratorT,
+        d_out_items: DeviceArrayLike | IteratorT,
+        d_out_num_selected: DeviceArrayLike,
+        op: OpAdapter,
+        compute_capability=None,
+    ):
+        self.d_in_keys_cccl = cccl.to_cccl_input_iter(d_in_keys)
+        self.d_in_items_cccl = cccl.to_cccl_input_iter(d_in_items)
+        self.d_out_keys_cccl = cccl.to_cccl_output_iter(d_out_keys)
+        self.d_out_items_cccl = cccl.to_cccl_output_iter(d_out_items)
+        self.d_out_num_selected_cccl = cccl.to_cccl_output_iter(d_out_num_selected)
+
+        # Compile the op - unique_by_key expects bool return (comparison)
+        value_type = cccl.get_value_type(d_in_keys)
+        self.op_cccl = op.compile((value_type, value_type), types.uint8)
+
+        self.build_results, self._bound_build_result = cache_build_results(
+            _bindings.DeviceUniqueByKeyBuildResult,
+            d_in_keys,
+            d_in_items,
+            d_out_keys,
+            d_out_items,
+            d_out_num_selected,
+            op,
+            compute_capability=compute_capability,
+            builder=lambda: cccl.build_for_ccs(
+                _bindings.DeviceUniqueByKeyBuildResult,
+                self.d_in_keys_cccl,
+                self.d_in_items_cccl,
+                self.d_out_keys_cccl,
+                self.d_out_items_cccl,
+                self.d_out_num_selected_cccl,
+                self.op_cccl,
+                compute_capability=compute_capability,
+            ),
+        )
+
+    def __call__(
+        self,
+        *,
+        temp_storage,
+        d_in_keys: DeviceArrayLike | IteratorT,
+        d_in_items: DeviceArrayLike | IteratorT,
+        d_out_keys: DeviceArrayLike | IteratorT,
+        d_out_items: DeviceArrayLike | IteratorT,
+        d_out_num_selected: DeviceArrayLike,
+        op: Operator,
+        num_items: int,
+        stream=None,
+    ):
+        # Select (and lazily load) the build result for the current device.
+        self.loaded_build_result = cccl.resolve_build_result(
+            self.build_results, self._bound_build_result
+        )
+
+        set_cccl_iterator_state(self.d_in_keys_cccl, d_in_keys)
+        set_cccl_iterator_state(self.d_in_items_cccl, d_in_items)
+        set_cccl_iterator_state(self.d_out_keys_cccl, d_out_keys)
+        set_cccl_iterator_state(self.d_out_items_cccl, d_out_items)
+        set_cccl_iterator_state(self.d_out_num_selected_cccl, d_out_num_selected)
+
+        # Update op state for stateful ops
+        op_adapter = make_op_adapter(op)
+        self.op_cccl.state = op_adapter.get_state()
+
+        stream_handle = validate_and_get_stream(stream)
+        if temp_storage is None:
+            temp_storage_bytes = 0
+            d_temp_storage = 0
+        else:
+            temp_storage_bytes = temp_storage.nbytes
+            # Note: this is slightly slower, but supports all ndarray-like objects as long as they support CAI
+            # TODO: switch to use gpumemoryview once it's ready
+            d_temp_storage = get_data_pointer(temp_storage)
+
+        temp_storage_bytes = self.loaded_build_result.compute(
+            d_temp_storage,
+            temp_storage_bytes,
+            self.d_in_keys_cccl,
+            self.d_in_items_cccl,
+            self.d_out_keys_cccl,
+            self.d_out_items_cccl,
+            self.d_out_num_selected_cccl,
+            self.op_cccl,
+            num_items,
+            stream_handle,
+        )
+        return temp_storage_bytes
+
+
+@cache_with_registered_key_functions
+def make_unique_by_key(
+    *,
+    d_in_keys: DeviceArrayLike | IteratorT,
+    d_in_items: DeviceArrayLike | IteratorT,
+    d_out_keys: DeviceArrayLike | IteratorT,
+    d_out_items: DeviceArrayLike | IteratorT,
+    d_out_num_selected: DeviceArrayLike,
+    op: Operator,
+    compute_capability=None,
+):
+    """Implements a device-wide unique by key operation using ``d_in_keys`` and the comparison operator ``op``. Only the first key and its value from each run is selected and the total number of items selected is also reported.
+
+    Example:
+        Below, ``make_unique_by_key`` is used to create a unique by key object that can be reused.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/unique/unique_by_key_object.py
+          :language: python
+          :start-after: # example-begin
+
+
+    Args:
+        d_in_keys: Device array or iterator containing the input sequence of keys
+        d_in_items: Device array or iterator that contains each key's corresponding item
+        d_out_keys: Device array or iterator to store the outputted keys
+        d_out_items: Device array or iterator to store each outputted key's item
+        d_out_num_selected: Device array to store how many items were selected
+        op: Callable or OpKind representing the equality operator
+        compute_capability: Compute capability, or list of capabilities, to
+            build for ahead of time. Accepts a packed int (e.g. ``90``), a
+            ``(major, minor)`` pair, a string (e.g. ``"9.0"``), or a list
+            thereof. When ``None`` (the default), the current device's
+            architecture is used.
+
+    Returns:
+        A callable object that can be used to perform unique by key
+    """
+    op_adapter = make_op_adapter(op)
+    return _UniqueByKey(
+        d_in_keys,
+        d_in_items,
+        d_out_keys,
+        d_out_items,
+        d_out_num_selected,
+        op_adapter,
+        compute_capability=compute_capability,
+    )
+
+
+def unique_by_key(
+    *,
+    d_in_keys: DeviceArrayLike | IteratorT,
+    d_in_items: DeviceArrayLike | IteratorT,
+    d_out_keys: DeviceArrayLike | IteratorT,
+    d_out_items: DeviceArrayLike | IteratorT,
+    d_out_num_selected: DeviceArrayLike,
+    op: Operator,
+    num_items: int,
+    stream=None,
+):
+    """
+    Performs device-wide unique by key operation using the single-phase API.
+
+    This function automatically handles temporary storage allocation and execution.
+
+    Example:
+        Below, ``unique_by_key`` is used to populate the arrays of output keys and items with the first key and its corresponding item from each sequence of equal keys. It also outputs the number of items selected.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/unique/unique_by_key_basic.py
+            :language: python
+            :start-after: # example-begin
+
+
+    Args:
+        d_in_keys: Device array or iterator containing the input sequence of keys
+        d_in_items: Device array or iterator that contains each key's corresponding item
+        d_out_keys: Device array or iterator to store the outputted keys
+        d_out_items: Device array or iterator to store each outputted key's item
+        d_out_num_selected: Device array to store how many items were selected
+        op: Callable or OpKind representing the equality operator
+        num_items: Number of items to process
+        stream: CUDA stream for the operation (optional)
+    """
+    uniquer = make_unique_by_key(
+        d_in_keys=d_in_keys,
+        d_in_items=d_in_items,
+        d_out_keys=d_out_keys,
+        d_out_items=d_out_items,
+        d_out_num_selected=d_out_num_selected,
+        op=op,
+    )
+    tmp_storage_bytes = uniquer(
+        temp_storage=None,
+        d_in_keys=d_in_keys,
+        d_in_items=d_in_items,
+        d_out_keys=d_out_keys,
+        d_out_items=d_out_items,
+        d_out_num_selected=d_out_num_selected,
+        op=op,
+        num_items=num_items,
+        stream=stream,
+    )
+    tmp_storage = TempStorageBuffer(tmp_storage_bytes, stream)
+    uniquer(
+        temp_storage=tmp_storage,
+        d_in_keys=d_in_keys,
+        d_in_items=d_in_items,
+        d_out_keys=d_out_keys,
+        d_out_items=d_out_items,
+        d_out_num_selected=d_out_num_selected,
+        op=op,
+        num_items=num_items,
+        stream=stream,
+    )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/cccl/.gitkeep b/cccl_upstream/python/cuda_cccl/cuda/compute/cccl/.gitkeep
new file mode 100644
index 00000000..e69de29b
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/determinism.py b/cccl_upstream/python/cuda_cccl/cuda/compute/determinism.py
new file mode 100644
index 00000000..10eb099b
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/determinism.py
@@ -0,0 +1,5 @@
+from __future__ import annotations
+
+from ._bindings import Determinism
+
+__all__ = ["Determinism"]
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/__init__.py b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/__init__.py
new file mode 100644
index 00000000..81599bb1
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/__init__.py
@@ -0,0 +1,31 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+from ._base import IteratorBase
+from ._cache_modified import CacheModifiedInputIterator
+from ._constant import ConstantIterator
+from ._counting import CountingIterator
+from ._discard import DiscardIterator
+from ._permutation import PermutationIterator
+from ._reverse import ReverseIterator
+from ._shuffle import ShuffleIterator
+from ._transform import TransformIterator, TransformOutputIterator
+from ._zip import ZipIterator
+
+__all__ = [
+    "CacheModifiedInputIterator",
+    "ConstantIterator",
+    "CountingIterator",
+    "DiscardIterator",
+    "IteratorBase",
+    "PermutationIterator",
+    "ReverseIterator",
+    "ShuffleIterator",
+    "TransformIterator",
+    "TransformOutputIterator",
+    "ZipIterator",
+]
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_base.py b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_base.py
new file mode 100644
index 00000000..aafdd18a
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_base.py
@@ -0,0 +1,293 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Base classes for iterators.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import threading
+from typing import Hashable
+
+from .._bindings import Iterator, IteratorKind, IteratorState, Op
+from .._caching import cache_with_registered_key_functions
+from ..types import TypeDescriptor
+
+
+class IteratorBase:
+    """
+    Iterators represent streams of data computed on the fly.
+
+    See :py:mod:`cuda.compute.iterators` for available iterators.
+    """
+
+    # Subclassing
+    # -----------
+    #
+    # Subclasses must implement the following methods that return
+    # Op objects.
+    #
+    # - _make_advance_op() -> Op
+    # - _make_input_deref_op() -> Op | None
+    # - _make_output_deref_op() -> Op | None
+    #
+    # Iterators composed of other iterators must also implement:
+    #
+    # - children property to return tuple of child iterators for dependency tracking
+    #
+    # Examples of such "compound" iterators include TransformIterator,
+    # PermutationIterator, ReverseIterator and ZipIterator.
+    #
+    # The base class provides public cached accessors:
+    #
+    # - get_advance_op() -> Op (cached)
+    # - get_input_deref_op() -> Op | None (cached)
+    # - get_output_deref_op() -> Op | None (cached)
+
+    __slots__ = [
+        "_state_bytes",
+        "_state_alignment",
+        "_value_type",
+        "_advance_op",
+        "_input_deref_op",
+        "_output_deref_op",
+        "_uid_cached",
+        "_op_lock",
+    ]
+
+    def __init__(
+        self,
+        state_bytes: bytes,
+        state_alignment: int,
+        value_type: TypeDescriptor,
+    ):
+        """
+        Args:
+            state_bytes: bytes object representing iterator's state
+            state_alignment: Alignment of the state
+            value_type: Type of dereferenced values
+        """
+        self._state_bytes = state_bytes
+        self._state_alignment = state_alignment
+        self._value_type = value_type
+        # Per-cc caches: the compiled Op is arch-specific (its LTO-IR is
+        # built for the current build's target compute capability), so the memo
+        # must be keyed on that cc. Reusing one iterator instance across builds
+        # targeting different arches otherwise leaks the first arch's LTO-IR into
+        # the others, which nvJitLink rejects. Keyed on get_target_cc() (None ==
+        # current device); see get_advance_op() below.
+        self._advance_op: dict[Hashable, Op] = {}
+        self._input_deref_op: dict[Hashable, Op | None] = {}
+        self._output_deref_op: dict[Hashable, Op | None] = {}
+        self._uid_cached: str | None = None
+        # Free-threaded Python can let multiple threads share a read-only
+        # iterator object and race during the first lazy Op construction.
+        # The lock only protects that cache miss path; cached access stays
+        # lock-free and iterator mutation remains the caller's responsibility.
+        self._op_lock = threading.Lock()
+
+    @property
+    def state(self) -> IteratorState:
+        """Return the iterator state for CCCL interop."""
+        return IteratorState(self._state_bytes)
+
+    @property
+    def state_alignment(self) -> int:
+        """Return the alignment of the iterator state."""
+        return self._state_alignment
+
+    @property
+    def value_type(self) -> TypeDescriptor:
+        """Return the TypeDescriptor for dereferenced values."""
+        return self._value_type
+
+    @property
+    def children(self) -> tuple["IteratorBase", ...]:
+        """Return child iterators for automatic dependency tracking. Override in subclasses."""
+        return ()
+
+    def _get_uid(self) -> str:
+        """Return a deterministic unique identifier for this iterator type."""
+        if self._uid_cached is None:
+            self._uid_cached = _deterministic_suffix(self.kind)
+        return self._uid_cached
+
+    def _make_advance_symbol(self) -> str:
+        """Generate symbol name for advance operation."""
+        return f"{self.__class__.__name__}_advance_{self._get_uid()}"
+
+    def _make_input_deref_symbol(self) -> str:
+        """Generate symbol name for input dereference operation."""
+        return f"{self.__class__.__name__}_input_deref_{self._get_uid()}"
+
+    def _make_output_deref_symbol(self) -> str:
+        """Generate symbol name for output dereference operation."""
+        return f"{self.__class__.__name__}_output_deref_{self._get_uid()}"
+
+    def get_advance_op(self) -> Op:
+        """Get the cached Op for the advance operation."""
+        from .._target_cc import get_target_cc
+
+        key = get_target_cc()
+        if key not in self._advance_op:
+            with self._op_lock:
+                if key not in self._advance_op:
+                    self._advance_op[key] = self._make_advance_op()
+        return self._advance_op[key]
+
+    def get_input_deref_op(self) -> Op | None:
+        """Get the cached Op for input dereference operation, or None if not supported."""
+        from .._target_cc import get_target_cc
+
+        key = get_target_cc()
+        if key not in self._input_deref_op:
+            with self._op_lock:
+                if key not in self._input_deref_op:
+                    self._input_deref_op[key] = self._make_input_deref_op()
+        return self._input_deref_op[key]
+
+    def get_output_deref_op(self) -> Op | None:
+        """Get the cached Op for output dereference operation, or None if not supported."""
+        from .._target_cc import get_target_cc
+
+        key = get_target_cc()
+        if key not in self._output_deref_op:
+            with self._op_lock:
+                if key not in self._output_deref_op:
+                    self._output_deref_op[key] = self._make_output_deref_op()
+        return self._output_deref_op[key]
+
+    @property
+    def is_input_iterator(self) -> bool:
+        """Return True if this iterator supports input dereference."""
+        return self.get_input_deref_op() is not None
+
+    @property
+    def is_output_iterator(self) -> bool:
+        """Return True if this iterator supports output dereference."""
+        return self.get_output_deref_op() is not None
+
+    def to_cccl_iter(self, is_output: bool = False) -> Iterator:
+        """
+        Convert this iterator to a CCCL Iterator for algorithm interop.
+
+        Args:
+            is_output: If True, use output_dereference; otherwise use input_dereference
+
+        Returns:
+            CCCL Iterator object
+        """
+        # Get advance op
+        advance_op = self.get_advance_op()
+
+        # Get dereference op based on direction
+        if is_output:
+            deref_op = self.get_output_deref_op()
+            if deref_op is None:
+                raise ValueError("This iterator does not support output operations")
+        else:
+            deref_op = self.get_input_deref_op()
+            if deref_op is None:
+                raise ValueError("This iterator does not support input operations")
+
+        # Create the CCCL Iterator
+        return Iterator(
+            self._state_alignment,
+            IteratorKind.ITERATOR,
+            advance_op,
+            deref_op,
+            self._value_type.info,
+            state=self.state,
+        )
+
+    @property
+    def kind(self) -> Hashable:
+        """Return a hashable kind for caching purposes.
+
+        Note: state_bytes is intentionally excluded - iterators with the same
+        type structure but different runtime state should share cached reducers.
+        """
+        return (type(self).__name__, self._value_type)
+
+    # Abstract methods for subclasses
+    def _make_advance_op(self) -> Op:
+        """
+        Create Op object for advance operation.
+
+        Returns:
+            Op object with compiled LTOIR
+        """
+        raise NotImplementedError
+
+    def _make_input_deref_op(self) -> Op | None:
+        """
+        Create Op object for input dereference operation.
+
+        Returns:
+            Op object with compiled LTOIR, or None if not supported
+        """
+        raise NotImplementedError
+
+    def _make_output_deref_op(self) -> Op | None:
+        """
+        Create Op object for output dereference operation.
+
+        Returns:
+            Op object with compiled LTOIR, or None if not supported
+        """
+        raise NotImplementedError
+
+
+def _deterministic_suffix(kind: Hashable) -> str:
+    kind_str = str(kind)
+    return hashlib.sha256(kind_str.encode()).hexdigest()[:16]
+
+
+def compose_iterator_states(
+    iterators: list[IteratorBase],
+) -> tuple[bytes, int, list[int]]:
+    """
+    Concatenate multiple iterator states with proper alignment.
+
+    This is used by composite iterators (like ZipIterator and PermutationIterator)
+    that need to store multiple child iterator states in their own state.
+
+    Args:
+        iterators: List of child iterators whose states should be composed
+
+    Returns:
+        Tuple of:
+        - combined_state_bytes: Concatenated state bytes with padding
+        - combined_alignment: Maximum alignment requirement
+        - offsets: List of byte offsets for each iterator's state
+    """
+    if not iterators:
+        return (b"", 1, [])
+
+    states = [bytes(memoryview(it.state)) for it in iterators]
+    alignments = [it.state_alignment for it in iterators]
+
+    offsets = []
+    current_offset = 0
+    combined = b""
+
+    for state, align in zip(states, alignments):
+        # Add padding to meet alignment requirement
+        padding = (align - (current_offset % align)) % align
+        combined += b"\x00" * padding
+        current_offset += padding
+
+        offsets.append(current_offset)
+        combined += state
+        current_offset += len(state)
+
+    max_alignment = max(alignments)
+    return (combined, max_alignment, offsets)
+
+
+cache_with_registered_key_functions.register(IteratorBase, lambda it: it.kind)
+
+__all__ = ["IteratorBase"]
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_cache_modified.py b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_cache_modified.py
new file mode 100644
index 00000000..b58a42b5
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_cache_modified.py
@@ -0,0 +1,156 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""CacheModifiedInputIterator implementation."""
+
+from __future__ import annotations
+
+import struct
+from textwrap import dedent
+from typing import Literal
+
+from .._bindings import Op, OpKind
+from .._cpp_compile import compile_cpp_op_code, cpp_type_from_descriptor
+from .._utils.protocols import get_data_pointer, get_dtype
+from ..types import from_numpy_dtype
+from ._base import IteratorBase
+from ._common import CUDA_PREAMBLE
+
+# Map modifier names to PTX cache operators and C++ intrinsics
+_CACHE_MODIFIERS = {
+    "stream": ("cs", "__ldcs"),  # Cache streaming (evict first)
+    "global": ("cg", "__ldcg"),  # Cache at L2 only
+    "volatile": ("cv", "__ldcv"),  # Don't cache, always fetch
+}
+
+
+class CacheModifiedInputIterator(IteratorBase):
+    """
+    Iterator that wraps a device pointer with cache-modified loads.
+
+    This iterator uses PTX cache modifiers to control how data is loaded:
+    - "stream": Uses streaming loads (ld.global.cs) - hints that data will not be reused
+    - "global": Uses global cache loads (ld.global.cg) - caches only at L2
+    - "volatile": Uses volatile loads (ld.global.cv) - always fetches from memory
+
+    Supports element types of size 1, 2, 4, 8, or 16 bytes.
+    """
+
+    __slots__ = [
+        "_modifier",
+        "_array",
+        "_ptr",
+    ]
+
+    def __init__(
+        self,
+        array,
+        modifier: Literal["stream", "global", "volatile"] = "stream",
+    ):
+        """
+        Create a cache-modified input iterator.
+
+        Args:
+            array: Device array to wrap (must support __cuda_array_interface__)
+            modifier: Cache modifier - "stream", "global", or "volatile"
+        """
+        if modifier not in _CACHE_MODIFIERS:
+            raise ValueError(
+                f"Unknown modifier: {modifier}. Must be one of {list(_CACHE_MODIFIERS.keys())}"
+            )
+
+        self._modifier = modifier
+        self._array = array  # Keep reference to prevent GC
+        ptr = get_data_pointer(array)
+        dtype = get_dtype(array)
+
+        self._ptr = ptr
+        value_type = from_numpy_dtype(dtype)
+
+        # Cache-modified loads only supported for power-of-two sizes up to 16 bytes
+        # These correspond to PTX instructions: ld.global.{modifier}.b{8,16,32,64,128}
+        if value_type.size not in (1, 2, 4, 8, 16):
+            raise ValueError(
+                f"CacheModifiedInputIterator only supports types of size 1, 2, 4, 8, or 16 bytes. "
+                f"Got type with size {value_type.size} bytes. "
+                f"This matches PTX cache-modified load instruction limitations."
+            )
+
+        # State is just the pointer (8 bytes on 64-bit)
+        state_bytes = struct.pack("Q", ptr)
+
+        super().__init__(
+            state_bytes=state_bytes,
+            state_alignment=8,  # Pointer alignment
+            value_type=value_type,
+        )
+
+    def _make_advance_op(self) -> Op:
+        symbol = self._make_advance_symbol()
+        cpp_type = cpp_type_from_descriptor(self._value_type)
+
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            extern "C" __device__ void {symbol}(void* state, void* offset) {{
+                auto* s = static_cast<{cpp_type}**>(state);
+                auto dist = *static_cast<uint64_t*>(offset);
+                *s += dist;
+            }}
+        """).strip()
+
+        code = compile_cpp_op_code(source)
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[],
+        )
+
+    def _make_input_deref_op(self) -> Op | None:
+        symbol = self._make_input_deref_symbol()
+        cpp_type = cpp_type_from_descriptor(self._value_type)
+        _, intrinsic = _CACHE_MODIFIERS[self._modifier]
+
+        # Use cache-modified intrinsic for all supported sizes (1, 2, 4, 8, 16 bytes)
+        # These correspond to PTX instructions: ld.global.{modifier}.b{8,16,32,64,128}
+        # Note: __ldcs, __ldcg, __ldcv intrinsics work for all these sizes
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            extern "C" __device__ void {symbol}(void* state, void* result) {{
+                auto* ptr = *static_cast<{cpp_type}**>(state);
+                *static_cast<{cpp_type}*>(result) = {intrinsic}(ptr);
+            }}
+        """).strip()
+
+        code = compile_cpp_op_code(source)
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[],
+        )
+
+    def _make_output_deref_op(self) -> Op | None:
+        # Cache-modified iterator is input-only
+        return None
+
+    def __add__(self, offset: int) -> "CacheModifiedInputIterator":
+        """Advance the iterator by offset elements."""
+        out = CacheModifiedInputIterator(self._array, self._modifier)
+        offset_ptr = self._ptr + offset * get_dtype(out._array).itemsize
+        out._ptr = offset_ptr
+        out._state_bytes = struct.pack("Q", offset_ptr)
+        out._uid_cached = None
+        return out
+
+    @property
+    def kind(self):
+        """Return a hashable kind for caching purposes."""
+        return (
+            "CacheModifiedInputIterator",
+            self._modifier,
+            self._value_type,
+        )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_common.py b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_common.py
new file mode 100644
index 00000000..76ae0b54
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_common.py
@@ -0,0 +1,21 @@
+from __future__ import annotations
+
+from .._utils.protocols import is_device_array
+from ._base import IteratorBase
+
+CUDA_PREAMBLE = """#include <cuda/std/cstdint>
+#include <cuda_fp16.h>
+#include <cuda/std/cstring>
+using namespace cuda::std;
+"""
+
+
+def ensure_iterator(obj):
+    """Wrap array in PointerIterator if needed."""
+    from ._pointer import PointerIterator
+
+    if isinstance(obj, IteratorBase):
+        return obj
+    if is_device_array(obj):
+        return PointerIterator(obj)
+    raise TypeError("Expected an iterator or a device array")
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_constant.py b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_constant.py
new file mode 100644
index 00000000..32e4f204
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_constant.py
@@ -0,0 +1,103 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""ConstantIterator implementation."""
+
+from __future__ import annotations
+
+from textwrap import dedent
+
+import numpy as np
+
+from .._bindings import Op, OpKind
+from .._cpp_compile import compile_cpp_op_code, cpp_type_from_descriptor
+from ..types import from_numpy_dtype
+from ._base import IteratorBase
+from ._common import CUDA_PREAMBLE
+
+
+class ConstantIterator(IteratorBase):
+    """
+    Iterator representing a sequence of constant values.
+
+    Similar to `thrust::constant_iterator <https://nvidia.github.io/cccl/thrust/api/classthrust_1_1constant__iterator.html>`_.
+
+    Every dereference returns the same constant value.
+
+    Example:
+        The code snippet below demonstrates the usage of a ``ConstantIterator``
+        representing a sequence of constant values:
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/iterator/constant_iterator_basic.py
+            :language: python
+            :start-after: # example-begin
+
+    Args:
+        value: The value of every item in the sequence
+    """
+
+    def __init__(self, value: np.number):
+        """
+        Create a constant iterator with the given value.
+
+        Args:
+            value: The constant value (must be a numpy scalar)
+        """
+        if not isinstance(value, np.generic):
+            value = np.array(value).flatten()[0]
+
+        self._constant_value = value
+        value_type = from_numpy_dtype(value.dtype)
+        state_bytes = value.tobytes()
+
+        super().__init__(
+            state_bytes=state_bytes,
+            state_alignment=value_type.alignment,
+            value_type=value_type,
+        )
+
+    def _make_advance_op(self) -> Op:
+        symbol = self._make_advance_symbol()
+
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            extern "C" __device__ void {symbol}(void*, void*) {{
+            }}
+        """).strip()
+
+        code = compile_cpp_op_code(source)
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[],
+        )
+
+    def _make_input_deref_op(self) -> Op | None:
+        symbol = self._make_input_deref_symbol()
+        cpp_type = cpp_type_from_descriptor(self._value_type)
+
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            extern "C" __device__ void {symbol}(void* state, void* result) {{
+                *static_cast<{cpp_type}*>(result) = *static_cast<{cpp_type}*>(state);
+            }}
+        """).strip()
+
+        code = compile_cpp_op_code(source)
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[],
+        )
+
+    def _make_output_deref_op(self) -> Op | None:
+        return None
+
+    def __add__(self, offset: int) -> "ConstantIterator":
+        """Return a new ConstantIterator (value doesn't change with position)."""
+        return ConstantIterator(self._constant_value)
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_counting.py b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_counting.py
new file mode 100644
index 00000000..97f359d8
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_counting.py
@@ -0,0 +1,108 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""CountingIterator implementation."""
+
+from __future__ import annotations
+
+from textwrap import dedent
+
+import numpy as np
+
+from .._bindings import Op, OpKind
+from .._cpp_compile import compile_cpp_op_code, cpp_type_from_descriptor
+from ..types import from_numpy_dtype
+from ._base import IteratorBase
+from ._common import CUDA_PREAMBLE
+
+
+class CountingIterator(IteratorBase):
+    """
+    Iterator representing a sequence of incrementing values.
+
+    Similar to `thrust::counting_iterator <https://nvidia.github.io/cccl/thrust/api/classthrust_1_1counting__iterator.html>`_.
+
+    The iterator starts at `start` and increments by 1 for each advance.
+
+    Example:
+        The code snippet below demonstrates the usage of a ``CountingIterator``
+        representing the sequence ``[10, 11, 12]``:
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/iterator/counting_iterator_basic.py
+            :language: python
+            :start-after: # example-begin
+
+    Args:
+        start: The initial value of the sequence
+    """
+
+    def __init__(self, start: np.number):
+        """
+        Create a counting iterator starting at `start`.
+
+        Args:
+            start: The initial value (must be a numpy scalar)
+        """
+        if not isinstance(start, np.generic):
+            start = np.array(start).flatten()[0]
+
+        self._start_value = start
+        value_type = from_numpy_dtype(start.dtype)
+        state_bytes = start.tobytes()
+
+        super().__init__(
+            state_bytes=state_bytes,
+            state_alignment=value_type.alignment,
+            value_type=value_type,
+        )
+
+    def _make_advance_op(self) -> Op:
+        symbol = self._make_advance_symbol()
+        cpp_type = cpp_type_from_descriptor(self._value_type)
+
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            extern "C" __device__ void {symbol}(void* state, void* offset) {{
+                auto* s = static_cast<{cpp_type}*>(state);
+                auto dist = *static_cast<uint64_t*>(offset);
+                *s += static_cast<{cpp_type}>(dist);
+            }}
+        """).strip()
+
+        code = compile_cpp_op_code(source)
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[],
+        )
+
+    def _make_input_deref_op(self) -> Op | None:
+        symbol = self._make_input_deref_symbol()
+        cpp_type = cpp_type_from_descriptor(self._value_type)
+
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            extern "C" __device__ void {symbol}(void* state, void* result) {{
+                *static_cast<{cpp_type}*>(result) = *static_cast<{cpp_type}*>(state);
+            }}
+        """).strip()
+
+        code = compile_cpp_op_code(source)
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[],
+        )
+
+    def _make_output_deref_op(self) -> Op | None:
+        return None
+
+    def __add__(self, offset: int) -> "CountingIterator":
+        """Return a new CountingIterator advanced by offset elements."""
+        new_start = self._start_value + offset
+        return CountingIterator(new_start)
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_discard.py b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_discard.py
new file mode 100644
index 00000000..ea1da2d1
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_discard.py
@@ -0,0 +1,103 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""DiscardIterator implementation."""
+
+from __future__ import annotations
+
+from .._bindings import Op, OpKind
+from .._cpp_compile import compile_cpp_op_code
+from .._utils.protocols import get_dtype
+from .._utils.temp_storage_buffer import TempStorageBuffer
+from ..types import TypeDescriptor, from_numpy_dtype
+from ._base import IteratorBase
+from ._common import CUDA_PREAMBLE
+
+
+class DiscardIterator(IteratorBase):
+    """
+    Iterator that discards all reads and writes.
+    """
+
+    def __init__(self, reference_iterator=None):
+        """
+        Create a discard iterator.
+
+        Args:
+            reference_iterator: Optional iterator or device array used to infer
+                value_type/state_type. Defaults to a temporary byte buffer.
+        """
+        if reference_iterator is None:
+            reference_iterator = TempStorageBuffer(1)
+
+        self._reference_iterator = reference_iterator
+
+        if hasattr(reference_iterator, "__cuda_array_interface__"):
+            value_type = from_numpy_dtype(get_dtype(reference_iterator))
+            state_bytes = bytes(value_type.dtype.itemsize)
+        elif isinstance(reference_iterator, IteratorBase):
+            value_type = reference_iterator.value_type
+            if isinstance(value_type, TypeDescriptor):
+                state_bytes = bytes(value_type.dtype.itemsize)
+            else:
+                state_bytes = bytes(value_type.info.size)
+        else:
+            raise TypeError("reference_iterator must be a device array or iterator")
+
+        super().__init__(
+            state_bytes=state_bytes,
+            state_alignment=value_type.alignment,
+            value_type=value_type,
+        )
+
+    def _make_advance_op(self) -> Op:
+        symbol = self._make_advance_symbol()
+
+        source = f"""{CUDA_PREAMBLE}
+
+extern "C" __device__ void {symbol}(void*, void*) {{
+}}
+"""
+        code = compile_cpp_op_code(source)
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[],
+        )
+
+    def _make_input_deref_op(self) -> Op | None:
+        symbol = self._make_input_deref_symbol()
+
+        source = f"""{CUDA_PREAMBLE}
+
+extern "C" __device__ void {symbol}(void*, void*) {{
+}}
+"""
+        code = compile_cpp_op_code(source)
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[],
+        )
+
+    def _make_output_deref_op(self) -> Op | None:
+        symbol = self._make_output_deref_symbol()
+
+        source = f"""{CUDA_PREAMBLE}
+
+extern "C" __device__ void {symbol}(void*, void*) {{
+}}
+"""
+        code = compile_cpp_op_code(source)
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[],
+        )
+
+    def __add__(self, offset: int) -> "DiscardIterator":
+        return DiscardIterator(self._reference_iterator)
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_permutation.py b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_permutation.py
new file mode 100644
index 00000000..2a973f8f
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_permutation.py
@@ -0,0 +1,220 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""PermutationIterator implementation."""
+
+from __future__ import annotations
+
+from textwrap import dedent
+
+from .._bindings import Op, OpKind
+from .._cpp_compile import compile_cpp_op_code, make_variable_declaration
+from ._base import IteratorBase, compose_iterator_states
+from ._common import CUDA_PREAMBLE, ensure_iterator
+
+
+class PermutationIterator(IteratorBase):
+    """
+    Iterator that accesses values through an index mapping.
+
+    At position i, yields values[indices[i]].
+
+    Similar to `thrust::permutation_iterator <https://nvidia.github.io/cccl/thrust/api/classthrust_1_1permutation__iterator.html>`_.
+
+    Example:
+        The code snippet below demonstrates accessing values through an index mapping.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/iterator/permutation_iterator_basic.py
+            :language: python
+            :start-after: # example-begin
+    """
+
+    __slots__ = [
+        "_values",
+        "_indices",
+        "_values_offset",
+        "_indices_offset",
+    ]
+
+    def __init__(
+        self,
+        values,
+        indices,
+    ):
+        """
+        Create a permutation iterator.
+
+        Args:
+            values: Iterator or array providing the values to be permuted
+            indices: Iterator or array providing the indices for permutation
+        """
+        # Wrap arrays in PointerIterator
+        self._values = ensure_iterator(values)
+        self._indices = ensure_iterator(indices)
+
+        # Compose states from both iterators
+        state_bytes, state_alignment, offsets = compose_iterator_states(
+            [self._values, self._indices]
+        )
+        self._values_offset = offsets[0]
+        self._indices_offset = offsets[1]
+
+        super().__init__(
+            state_bytes=state_bytes,
+            state_alignment=state_alignment,
+            value_type=self._values.value_type,
+        )
+
+    def _make_advance_op(self) -> Op:
+        """Provide Op for advance that only advances indices iterator."""
+        child_op = self._indices.get_advance_op()
+        symbol = self._make_advance_symbol()
+
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            extern "C" __device__ void {child_op.name}(void* state, void* offset);
+
+            extern "C" __device__ void {symbol}(void* state, void* offset) {{
+                char* indices_state = static_cast<char*>(state) + {self._indices_offset};
+                {child_op.name}(indices_state, offset);
+            }}
+        """).strip()
+
+        code = compile_cpp_op_code(source)
+
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[child_op.code, *child_op.extra_code],
+        )
+
+    def _make_input_deref_op(self) -> Op | None:
+        """Provide Op for input deref that reads index then accesses values."""
+        indices_deref_op = self._indices.get_input_deref_op()
+        if indices_deref_op is None:
+            raise ValueError("Indices iterator must support input dereference")
+
+        values_deref_op = self._values.get_input_deref_op()
+        if values_deref_op is None:
+            return None
+
+        # Also need values advance for random access
+        values_advance_op = self._values.get_advance_op()
+
+        symbol = self._make_input_deref_symbol()
+        idx_decl = make_variable_declaration(self._indices.value_type, "idx")
+        values_state_size = len(bytes(memoryview(self._values.state)))
+
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            extern "C" __device__ void {indices_deref_op.name}(void* state, void* result);
+            extern "C" __device__ void {values_advance_op.name}(void* state, void* offset);
+            extern "C" __device__ void {values_deref_op.name}(void* state, void* result);
+
+            extern "C" __device__ void {symbol}(void* state, void* result) {{
+                char* values_state = static_cast<char*>(state) + {self._values_offset};
+                char* indices_state = static_cast<char*>(state) + {self._indices_offset};
+
+                {idx_decl}
+                {indices_deref_op.name}(indices_state, &idx);
+
+                alignas({self._values.state_alignment}) char temp_values[{values_state_size}];
+                memcpy(temp_values, values_state, {values_state_size});
+
+                uint64_t offset = static_cast<uint64_t>(idx);
+                {values_advance_op.name}(temp_values, &offset);
+                {values_deref_op.name}(temp_values, result);
+            }}
+        """).strip()
+
+        code = compile_cpp_op_code(source)
+
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[
+                values_advance_op.code,
+                *values_advance_op.extra_code,
+                indices_deref_op.code,
+                *indices_deref_op.extra_code,
+                values_deref_op.code,
+                *values_deref_op.extra_code,
+            ],
+        )
+
+    def _make_output_deref_op(self) -> Op | None:
+        """Provide Op for output deref that reads index then writes to values."""
+        indices_deref_op = self._indices.get_input_deref_op()
+        if indices_deref_op is None:
+            raise ValueError("Indices iterator must support input dereference")
+
+        values_deref_op = self._values.get_output_deref_op()
+        if values_deref_op is None:
+            return None
+
+        # Also need values advance for random access
+        values_advance_op = self._values.get_advance_op()
+
+        symbol = self._make_output_deref_symbol()
+        idx_decl = make_variable_declaration(self._indices.value_type, "idx")
+        values_state_size = len(bytes(memoryview(self._values.state)))
+
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            extern "C" __device__ void {indices_deref_op.name}(void* state, void* result);
+            extern "C" __device__ void {values_advance_op.name}(void* state, void* offset);
+            extern "C" __device__ void {values_deref_op.name}(void* state, void* value);
+
+            extern "C" __device__ void {symbol}(void* state, void* value) {{
+                char* values_state = static_cast<char*>(state) + {self._values_offset};
+                char* indices_state = static_cast<char*>(state) + {self._indices_offset};
+
+                {idx_decl}
+                {indices_deref_op.name}(indices_state, &idx);
+
+                alignas({self._values.state_alignment}) char temp_values[{values_state_size}];
+                memcpy(temp_values, values_state, {values_state_size});
+
+                uint64_t offset = static_cast<uint64_t>(idx);
+                {values_advance_op.name}(temp_values, &offset);
+                {values_deref_op.name}(temp_values, value);
+            }}
+        """).strip()
+
+        code = compile_cpp_op_code(source)
+
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[
+                values_advance_op.code,
+                *values_advance_op.extra_code,
+                indices_deref_op.code,
+                *indices_deref_op.extra_code,
+                values_deref_op.code,
+                *values_deref_op.extra_code,
+            ],
+        )
+
+    @property
+    def children(self):
+        return (self._values, self._indices)
+
+    def __add__(self, offset: int) -> "PermutationIterator":
+        """Advance the indices iterator by offset, keeping values at base."""
+        return PermutationIterator(
+            self._values,  # values stays at base for random access
+            self._indices + offset,  # only indices advances  # type: ignore[operator]
+        )
+
+    @property
+    def kind(self):
+        """Return a hashable kind for caching purposes."""
+        return ("PermutationIterator", self._values.kind, self._indices.kind)
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_pointer.py b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_pointer.py
new file mode 100644
index 00000000..855e2d02
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_pointer.py
@@ -0,0 +1,191 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""PointerIterator implementation - simple bidirectional iterator for device arrays."""
+
+from __future__ import annotations
+
+import ctypes
+import sys
+from textwrap import dedent
+
+from .._bindings import Op, OpKind
+from .._cpp_compile import compile_cpp_op_code, cpp_type_from_descriptor
+from .._utils.protocols import get_data_pointer, get_dtype
+from ..types import from_numpy_dtype
+from ._base import IteratorBase
+from ._common import CUDA_PREAMBLE
+
+
+class PointerIterator(IteratorBase):
+    """
+    Simple iterator wrapping a device array pointer.
+
+    Supports both input (reading) and output (writing) operations.
+    Handles both scalar types (using typed C++ code) and struct types
+    (using byte-level memcpy).
+    """
+
+    def __init__(self, array):
+        """
+        Create a pointer iterator from a device array.
+
+        Args:
+            array: Device array with __cuda_array_interface__
+        """
+        # Get pointer and dtype from array
+        ptr = get_data_pointer(array)
+        dtype = get_dtype(array)
+        value_type = from_numpy_dtype(dtype)
+
+        # State is just the pointer
+        state_bytes = ctypes.c_void_p(ptr)
+        state_bytes_buffer = (ctypes.c_char * 8)()
+        ctypes.memmove(state_bytes_buffer, ctypes.byref(state_bytes), 8)
+        state_bytes = bytes(state_bytes_buffer)
+
+        self._cpp_type = cpp_type_from_descriptor(value_type)  # None for struct types
+        self._element_size = value_type.info.size
+        self._array = array  # Keep reference to prevent GC
+
+        super().__init__(
+            state_bytes=state_bytes,
+            state_alignment=8,  # pointer alignment
+            value_type=value_type,
+        )
+
+    @property
+    def array(self):
+        return self._array
+
+    def _make_advance_op(self) -> Op:
+        symbol = self._make_advance_symbol()
+
+        if self._cpp_type:
+            # Scalar type - use typed pointer arithmetic
+            source = dedent(f"""
+                {CUDA_PREAMBLE}
+
+                extern "C" __device__ void {symbol}(void* state, void* offset) {{
+                    auto* ptr_state = static_cast<{self._cpp_type}**>(state);
+                    auto dist = *static_cast<int64_t*>(offset);
+                    *ptr_state += dist;
+                }}
+            """).strip()
+        else:
+            # Struct type - use byte-level pointer arithmetic
+            source = dedent(f"""
+                {CUDA_PREAMBLE}
+
+                extern "C" __device__ void {symbol}(void* state, void* offset) {{
+                    auto* ptr_state = static_cast<char**>(state);
+                    auto dist = *static_cast<int64_t*>(offset);
+                    *ptr_state += dist * {self._element_size};
+                }}
+            """).strip()
+
+        code = compile_cpp_op_code(source)
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[],
+        )
+
+    def _make_input_deref_op(self) -> Op | None:
+        symbol = self._make_input_deref_symbol()
+
+        if self._cpp_type:
+            # Scalar type - use typed dereference
+            source = dedent(f"""
+                {CUDA_PREAMBLE}
+
+                extern "C" __device__ void {symbol}(void* state, void* result) {{
+                    auto* ptr_state = static_cast<{self._cpp_type}**>(state);
+                    *static_cast<{self._cpp_type}*>(result) = **ptr_state;
+                }}
+            """).strip()
+        else:
+            # Struct type - use memcpy
+            source = dedent(f"""
+                {CUDA_PREAMBLE}
+
+                extern "C" __device__ void {symbol}(void* state, void* result) {{
+                    auto* ptr_state = static_cast<char**>(state);
+                    memcpy(result, *ptr_state, {self._element_size});
+                }}
+            """).strip()
+
+        code = compile_cpp_op_code(source)
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[],
+        )
+
+    def _make_output_deref_op(self) -> Op | None:
+        symbol = self._make_output_deref_symbol()
+
+        if self._cpp_type:
+            # Scalar type - use typed dereference
+            source = dedent(f"""
+                {CUDA_PREAMBLE}
+
+                extern "C" __device__ void {symbol}(void* state, void* value) {{
+                    auto* ptr_state = static_cast<{self._cpp_type}**>(state);
+                    **ptr_state = *static_cast<{self._cpp_type}*>(value);
+                }}
+            """).strip()
+        else:
+            # Struct type - use memcpy
+            source = dedent(f"""
+                {CUDA_PREAMBLE}
+
+                extern "C" __device__ void {symbol}(void* state, void* value) {{
+                    auto* ptr_state = static_cast<char**>(state);
+                    memcpy(*ptr_state, value, {self._element_size});
+                }}
+            """).strip()
+
+        code = compile_cpp_op_code(source)
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[],
+        )
+
+    def __add__(self, offset: int):
+        dtype = get_dtype(self._array)
+        offset_ptr = self._current_pointer() + offset * dtype.itemsize
+        return self._clone_with_pointer(offset_ptr)
+
+    def _current_pointer(self) -> int:
+        return int.from_bytes(self._state_bytes, sys.byteorder, signed=False)
+
+    def _clone_with_pointer(self, pointer_value: int):
+        """Clone this iterator with a different pointer value."""
+        clone = PointerIterator(self._array)
+        state_bytes_buffer = (ctypes.c_char * 8)()
+        ptr_obj = ctypes.c_void_p(pointer_value)
+        ctypes.memmove(state_bytes_buffer, ctypes.byref(ptr_obj), 8)
+        clone._state_bytes = bytes(state_bytes_buffer)
+        clone._uid_cached = None
+        return clone
+
+    @property
+    def kind(self):
+        """
+        Return a hashable kind for caching purposes.
+
+        Include _cpp_type and _element_size since they affect generated code.
+        Different code paths are taken for scalar vs struct types.
+        """
+        return (
+            type(self).__name__,
+            self._value_type,
+            self._cpp_type,
+            self._element_size,
+        )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_reverse.py b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_reverse.py
new file mode 100644
index 00000000..0eb90b38
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_reverse.py
@@ -0,0 +1,140 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""ReverseIterator implementation."""
+
+from __future__ import annotations
+
+from textwrap import dedent
+
+from .._bindings import Op, OpKind
+from .._cpp_compile import compile_cpp_op_code
+from .._utils.protocols import get_size, is_device_array
+from ._base import IteratorBase
+from ._common import CUDA_PREAMBLE, ensure_iterator
+
+
+class ReverseIterator(IteratorBase):
+    """
+    Iterator that reverses the direction of an underlying iterator.
+
+    Advance with positive offset moves backward in the underlying iterator.
+    """
+
+    __slots__ = [
+        "_underlying",
+    ]
+
+    def __init__(self, underlying):
+        """
+        Create a reverse iterator.
+
+        Args:
+            underlying: The underlying iterator or array to reverse
+        """
+
+        if is_device_array(underlying):
+            # TODO: this is probably incorrect behaviour. In C++, initializing
+            # with a pointer to the end of the array is left to be done explicitly
+            # by the user.
+            self._underlying = ensure_iterator(underlying) + (get_size(underlying) - 1)
+        else:
+            self._underlying = ensure_iterator(underlying)
+
+        super().__init__(
+            state_bytes=bytes(self._underlying.state),
+            state_alignment=self._underlying.state_alignment,
+            value_type=self._underlying.value_type,
+        )
+
+    def _make_advance_op(self) -> Op:
+        """Provide Op for advance that negates offset direction."""
+        child_op = self._underlying.get_advance_op()
+        symbol = self._make_advance_symbol()
+
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            extern "C" __device__ void {child_op.name}(void* state, void* offset);
+
+            extern "C" __device__ void {symbol}(void* state, void* offset) {{
+                int64_t neg_offset = -static_cast<int64_t>(*static_cast<uint64_t*>(offset));
+                {child_op.name}(state, &neg_offset);
+            }}
+        """).strip()
+
+        code = compile_cpp_op_code(source)
+
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[child_op.code, *child_op.extra_code],
+        )
+
+    def _make_input_deref_op(self) -> Op | None:
+        """Provide Op for input dereference that delegates to underlying."""
+        child_op = self._underlying.get_input_deref_op()
+        if child_op is None:
+            return None
+
+        symbol = self._make_input_deref_symbol()
+
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            extern "C" __device__ void {child_op.name}(void* state, void* result);
+
+            extern "C" __device__ void {symbol}(void* state, void* result) {{
+                {child_op.name}(state, result);
+            }}
+        """).strip()
+
+        code = compile_cpp_op_code(source)
+
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[child_op.code, *child_op.extra_code],
+        )
+
+    def _make_output_deref_op(self) -> Op | None:
+        """Provide Op for output dereference that delegates to underlying."""
+        child_op = self._underlying.get_output_deref_op()
+        if child_op is None:
+            return None
+
+        symbol = self._make_output_deref_symbol()
+
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            extern "C" __device__ void {child_op.name}(void* state, void* value);
+
+            extern "C" __device__ void {symbol}(void* state, void* value) {{
+                {child_op.name}(state, value);
+            }}
+        """).strip()
+
+        code = compile_cpp_op_code(source)
+
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[child_op.code, *child_op.extra_code],
+        )
+
+    @property
+    def children(self):
+        return (self._underlying,)
+
+    @property
+    def kind(self):
+        """Return a hashable kind for caching purposes."""
+        return ("ReverseIterator", self._underlying.kind)
+
+    def __add__(self, offset: int):
+        return ReverseIterator(self._underlying + offset)
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_shuffle.py b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_shuffle.py
new file mode 100644
index 00000000..0b490bf1
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_shuffle.py
@@ -0,0 +1,148 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""ShuffleIterator implementation."""
+
+from __future__ import annotations
+
+import struct
+from textwrap import dedent
+
+import numpy as np
+
+from .._bindings import Op, OpKind
+from .._cpp_compile import compile_cpp_op_code
+from ..types import from_numpy_dtype
+from ._base import IteratorBase
+from ._common import CUDA_PREAMBLE
+
+_SHUFFLE_STATE_STRUCT = """\
+struct ShuffleState {
+    int64_t current_index;
+    uint64_t num_items;
+    uint64_t seed;
+};"""
+
+
+class ShuffleIterator(IteratorBase):
+    """
+    Iterator that produces a deterministic random permutation of indices.
+
+    At position ``i``, yields ``bijection(i)`` where the bijection is a random
+    permutation of ``[0, num_items)`` parameterized by ``seed``.
+
+    Example:
+        The code snippet below demonstrates the usage of a ``ShuffleIterator``
+        to randomly permute indices:
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/iterator/shuffle_iterator_basic.py
+            :language: python
+            :start-after: # example-begin
+
+    Args:
+        num_items: Number of elements in the domain to permute. Must be > 0.
+        seed: Seed for the random permutation. Different seeds produce
+            different (deterministic) permutations. Defaults to 0.
+    """
+
+    __slots__ = ["_num_items", "_seed", "_current_index"]
+
+    def __init__(self, num_items: int, seed: int = 0, *, _current_index: int = 0):
+        if num_items <= 0:
+            raise ValueError("num_items must be > 0")
+
+        self._num_items = int(num_items)
+        self._seed = int(seed)
+        self._current_index = int(_current_index)
+
+        # State layout matches C++ ShuffleState:
+        #   int64_t  current_index  (offset 0,  size 8)
+        #   uint64_t num_items      (offset 8,  size 8)
+        #   uint64_t seed           (offset 16, size 8)
+        state_bytes = struct.pack(
+            "<qQQ", self._current_index, self._num_items, self._seed
+        )
+
+        super().__init__(
+            state_bytes=state_bytes,
+            state_alignment=8,
+            value_type=from_numpy_dtype(np.dtype("int64")),
+        )
+
+    def _make_advance_op(self) -> Op:
+        symbol = self._make_advance_symbol()
+
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            {_SHUFFLE_STATE_STRUCT}
+
+            extern "C" __device__ void {symbol}(void* state, void* offset) {{
+                auto s = static_cast<ShuffleState*>(state);
+                auto dist = *static_cast<int64_t*>(offset);
+                s->current_index += dist;
+            }}
+        """)
+
+        code = compile_cpp_op_code(source)
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[],
+        )
+
+    def _make_input_deref_op(self) -> Op | None:
+        symbol = self._make_input_deref_symbol()
+
+        # Note: a potential optimization is to avoid constructing
+        # `cuda::random_bijection` objects upon every dereference,
+        # instead constructing it once and using it as the state
+        # object. The tradeoff is that it would require a C++
+        # extension providing a constructor for
+        # `cuda::random_bijection` objects, since we would now be
+        # doing it on the host.  See discussion in #7721.
+        source = dedent(f"""
+            #include <cuda/__random/random_bijection.h>
+            #include <cuda/__random/pcg_engine.h>
+            {CUDA_PREAMBLE}
+
+            {_SHUFFLE_STATE_STRUCT}
+
+            // __noinline__ is required to prevent the compiler from merging
+            // this function's register usage into the calling kernel during LTO
+            // inlining. feistel_bijection constructs 24 round keys with
+            // UNROLL_FULL, which exhausts the kernel's register budget and
+            // causes spilling to local memory (LDL/STL instructions).
+            // Keeping it non-inlined gives it an isolated register frame.
+            __device__ __noinline__ int64_t __shuffle_apply(uint64_t num_items, uint64_t seed, uint64_t idx) {{
+                cuda::pcg64 rng(seed);
+                cuda::random_bijection<uint64_t> bijection(num_items, rng);
+                return static_cast<int64_t>(bijection(idx));
+            }}
+
+            extern "C" __device__ void {symbol}(void* state, void* result) {{
+                const auto* s = static_cast<const ShuffleState*>(state);
+                *static_cast<int64_t*>(result) = __shuffle_apply(
+                    s->num_items, s->seed, static_cast<uint64_t>(s->current_index));
+            }}
+        """)
+
+        code = compile_cpp_op_code(source)
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[],
+        )
+
+    def _make_output_deref_op(self) -> Op | None:
+        return None
+
+    def __add__(self, offset: int) -> "ShuffleIterator":
+        return ShuffleIterator(
+            self._num_items,
+            self._seed,
+            _current_index=self._current_index + offset,
+        )
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_transform.py b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_transform.py
new file mode 100644
index 00000000..e43ed8cd
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_transform.py
@@ -0,0 +1,292 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""TransformIterator implementation."""
+
+from __future__ import annotations
+
+from textwrap import dedent
+
+from .._bindings import Op, OpKind
+from .._cpp_compile import compile_cpp_op_code, make_variable_declaration
+from ..op import make_op_adapter
+from ..types import TypeDescriptor, signature_from_annotations
+from ._base import IteratorBase
+from ._common import CUDA_PREAMBLE, ensure_iterator
+
+
+class TransformIterator(IteratorBase):
+    """
+    An iterator that applies a unary function to elements as they are read from an underlying iterator.
+
+    Similar to `thrust::transform_iterator <https://nvidia.github.io/cccl/thrust/api/classthrust_1_1transform__iterator.html>`_.
+
+    For input iteration (default): reads from underlying, applies transform, returns result.
+    For output iteration: applies transform to input values, writes to underlying.
+
+    Example:
+        The code snippet below demonstrates the usage of a ``TransformIterator`` composed with a ``CountingIterator``
+        to transform the input before performing a reduction:
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/iterator/transform_iterator_basic.py
+            :language: python
+            :start-after: # example-begin
+
+    Args:
+        underlying: The underlying iterator or device array
+        transform_op: The unary operation to apply
+        output_value_type: TypeDescriptor for the output type (optional, will be inferred if not provided)
+        is_input: True for input iterator (default), False for output iterator
+    """
+
+    __slots__ = [
+        "_underlying",
+        "_transform_op",
+        "_value_type",
+        "_is_input",
+        "_compiled_op",
+    ]
+
+    def __init__(
+        self,
+        underlying,
+        transform_op,
+        value_type: TypeDescriptor | None = None,
+        is_input: bool = True,
+    ):
+        """
+        Create a transform iterator.
+
+        Args:
+            underlying: The underlying iterator or device array to transform
+            transform_op: The unary transform operation (callable or OpKind)
+            value_type: TypeDescriptor for the transformed value type.
+                            For input iterators: inferred if None.
+                            For output iterators: must be provided or have annotations.
+            is_input: True for input iterator, False for output iterator
+        """
+        underlying = ensure_iterator(underlying)
+
+        self._underlying = underlying
+        self._transform_op = make_op_adapter(transform_op)
+        self._is_input = is_input
+        # Lazily compiled transform Op, keyed on the build's target compute
+        # capability: its LTO-IR is arch-specific, so reusing one iterator
+        # instance across builds targeting different arches must not reuse the
+        # first arch's op (nvJitLink rejects a newer-arch input linked into an
+        # older-arch result). Mirrors the per-cc op caches in IteratorBase.
+        self._compiled_op: dict = {}
+
+        # Determine value type
+        if value_type is None:
+            if is_input:
+                # value_type is the return type
+                _, value_type = signature_from_annotations(transform_op)
+                if value_type is None:
+                    value_type = self._transform_op.get_return_type(
+                        (underlying.value_type,)
+                    )
+            else:
+                # value_type is the input type
+                input_types, _ = signature_from_annotations(transform_op)
+                if len(input_types) != 1:
+                    raise ValueError(
+                        "TransformOutputIterator transform function must take exactly one argument with type annotation"
+                    )
+                value_type = input_types[0]
+        assert value_type is not None
+        self._value_type = value_type
+
+        super().__init__(
+            state_bytes=bytes(self._underlying.state),
+            state_alignment=self._underlying.state_alignment,
+            value_type=value_type,
+        )
+
+    def _get_compiled_op(self):
+        """Get the compiled Op for the current target cc, compiling lazily if needed."""
+        from .._target_cc import get_target_cc
+
+        key = get_target_cc()
+        if key not in self._compiled_op:
+            if self._is_input:
+                input_type = self._underlying.value_type
+                output_type = self._value_type
+            else:
+                input_type = self._value_type
+                output_type = self._underlying.value_type
+
+            self._compiled_op[key] = self._transform_op.compile(
+                (input_type,),
+                output_type,
+            )
+        return self._compiled_op[key]
+
+    def _make_advance_op(self) -> Op:
+        """Provide Op for advance that delegates to underlying iterator."""
+        child_op = self._underlying.get_advance_op()
+        symbol = self._make_advance_symbol()
+
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            extern "C" __device__ void {child_op.name}(void* state, void* offset);
+
+            extern "C" __device__ void {symbol}(void* state, void* offset) {{
+                {child_op.name}(state, offset);
+            }}
+        """).strip()
+
+        code = compile_cpp_op_code(source)
+
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[child_op.code, *child_op.extra_code],
+        )
+
+    def _make_input_deref_op(self) -> Op | None:
+        """Provide Op for input dereference that reads from underlying then transforms."""
+        if not self._is_input:
+            return None
+
+        child_op = self._underlying.get_input_deref_op()
+        if child_op is None:
+            raise ValueError("Underlying iterator must support input dereference")
+
+        compiled_op = self._get_compiled_op()
+        symbol = self._make_input_deref_symbol()
+        temp_decl = make_variable_declaration(self._underlying.value_type, "temp")
+
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            extern "C" __device__ void {child_op.name}(void* state, void* result);
+            extern "C" __device__ void {compiled_op.name}(void* input, void* output);
+
+            extern "C" __device__ void {symbol}(void* state, void* result) {{
+                {temp_decl}
+                {child_op.name}(state, &temp);
+                {compiled_op.name}(&temp, result);
+            }}
+        """).strip()
+
+        code = compile_cpp_op_code(source)
+
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[
+                compiled_op.code,
+                *compiled_op.extra_code,
+                child_op.code,
+                *child_op.extra_code,
+            ],
+        )
+
+    def _make_output_deref_op(self) -> Op | None:
+        """Provide Op for output dereference that transforms then writes to underlying."""
+        if self._is_input:
+            return None
+
+        child_op = self._underlying.get_output_deref_op()
+        if child_op is None:
+            raise ValueError("Underlying iterator must support output dereference")
+
+        compiled_op = self._get_compiled_op()
+        symbol = self._make_output_deref_symbol()
+        temp_decl = make_variable_declaration(self._underlying.value_type, "temp")
+
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            extern "C" __device__ void {child_op.name}(void* state, void* value);
+            extern "C" __device__ void {compiled_op.name}(void* input, void* output);
+
+            extern "C" __device__ void {symbol}(void* state, void* value) {{
+                {temp_decl}
+                {compiled_op.name}(value, &temp);
+                {child_op.name}(state, &temp);
+            }}
+        """).strip()
+
+        code = compile_cpp_op_code(source)
+
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[
+                compiled_op.code,
+                *compiled_op.extra_code,
+                child_op.code,
+                *child_op.extra_code,
+            ],
+        )
+
+    def advance(self, offset: int) -> "TransformIterator":
+        """Return a new iterator advanced by offset elements."""
+        if not hasattr(self._underlying, "__add__"):
+            raise AttributeError("Underlying iterator does not support advance")
+        return TransformIterator(
+            self._underlying + offset,  # type: ignore[operator, arg-type]
+            self._transform_op,
+            self._value_type,
+            is_input=self._is_input,
+        )
+
+    def __add__(self, offset: int) -> "TransformIterator":
+        return self.advance(offset)
+
+    def __radd__(self, offset: int) -> "TransformIterator":
+        return self.advance(offset)
+
+    @property
+    def children(self):
+        return (self._underlying,)
+
+    @property
+    def kind(self):
+        """Return a hashable kind for caching purposes."""
+        # Convert _value_type to tuple if it's a list (for output iterators)
+        value_type = (
+            tuple(self._value_type)
+            if isinstance(self._value_type, list)
+            else self._value_type
+        )
+        return (
+            "TransformIterator",
+            self._is_input,
+            self._transform_op,
+            self._underlying.kind,
+            value_type,
+        )
+
+
+class TransformOutputIterator(TransformIterator):
+    """
+    An iterator that applies a unary function to values before writing them to an underlying iterator.
+
+    Similar to `thrust::transform_output_iterator <https://nvidia.github.io/cccl/thrust/api/classthrust_1_1transform__output__iterator.html>`_.
+
+    This is a convenience subclass of TransformIterator configured for output mode.
+
+    Example:
+        The code snippet below demonstrates the usage of a ``TransformOutputIterator`` to transform the output
+        of a reduction before writing to an output array:
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/iterator/transform_output_iterator.py
+            :language: python
+            :start-after: # example-begin
+
+    Args:
+        underlying: The underlying iterator or device array
+        transform_op: The operation to be applied to values before they are written
+        output_value_type: TypeDescriptor for the input value type (optional, will be extracted from annotations if not provided)
+    """
+
+    def __init__(self, underlying, transform_op, output_value_type=None):
+        super().__init__(underlying, transform_op, output_value_type, is_input=False)
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_utils.py b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_utils.py
new file mode 100644
index 00000000..9d48db4f
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_utils.py
@@ -0,0 +1 @@
+from __future__ import annotations
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_zip.py b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_zip.py
new file mode 100644
index 00000000..33c74dce
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/iterators/_zip.py
@@ -0,0 +1,216 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""ZipIterator implementation."""
+
+from __future__ import annotations
+
+from textwrap import dedent
+
+from .._bindings import Op, OpKind
+from .._cpp_compile import compile_cpp_op_code
+from ..types import struct
+from ._base import IteratorBase, compose_iterator_states
+from ._common import CUDA_PREAMBLE, ensure_iterator
+
+
+class ZipIterator(IteratorBase):
+    """
+    Iterator that zips multiple iterators together.
+
+    At each position, yields a tuple of values from all underlying iterators.
+
+    Similar to `thrust::zip_iterator <https://nvidia.github.io/cccl/thrust/api/classthrust_1_1zip__iterator.html>`_.
+
+    Example:
+        The code snippet below demonstrates how to zip together an array and a
+        :class:`CountingIterator <cuda.compute.iterators.CountingIterator>` to
+        find the index of the maximum value of the array.
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/iterator/zip_iterator_counting.py
+            :language: python
+            :start-after: # example-begin
+    """
+
+    __slots__ = [
+        "_iterators",
+        "_field_names",
+        "_value_offsets",
+        "_state_offsets",
+        "_advance_result",
+        "_input_deref_result",
+        "_output_deref_result",
+    ]
+
+    def __init__(self, *args):
+        """
+        Create a zip iterator.
+
+        Args:
+            *args: Iterators or arrays to zip together. Can be:
+                   - Multiple iterators/arrays: ZipIterator(it1, it2, it3)
+                   - A single sequence of iterators: ZipIterator([it1, it2, it3])
+        """
+        # Handle both ZipIterator(it1, it2) and ZipIterator([it1, it2])
+        if len(args) == 1 and isinstance(args[0], (list, tuple)):
+            iterators = args[0]
+        else:
+            iterators = args
+
+        if len(iterators) < 1:
+            raise ValueError("ZipIterator requires at least one iterator")
+
+        # Wrap arrays in PointerIterator
+        iterators = [ensure_iterator(it) for it in iterators]
+
+        self._iterators = list(iterators)
+
+        # Compose states from all iterators
+        self._state_bytes, self._state_alignment, self._state_offsets = (
+            compose_iterator_states(self._iterators)
+        )
+
+        # Build combined value type (struct layout)
+        self._field_names = [f"field_{i}" for i in range(len(self._iterators))]
+        fields = {
+            name: it.value_type for name, it in zip(self._field_names, self._iterators)
+        }
+        self._value_type = struct(fields, name=f"Zip{len(iterators)}")
+        self._value_offsets = [
+            self._value_type.dtype.fields[name][1] for name in self._field_names
+        ]
+
+        super().__init__(
+            state_bytes=self._state_bytes,
+            state_alignment=self._state_alignment,
+            value_type=self._value_type,
+        )
+
+    def _make_advance_op(self) -> Op:
+        """Provide Op for advance that calls all child iterator advances."""
+        child_ops = [it.get_advance_op() for it in self._iterators]
+        symbol = self._make_advance_symbol()
+
+        externs = "\n".join(
+            f'extern "C" __device__ void {op.name}(void* state, void* offset);'
+            for op in child_ops
+        )
+
+        calls = "\n        ".join(
+            f"{op.name}(static_cast<char*>(state) + {offset}, offset);"
+            for op, offset in zip(child_ops, self._state_offsets)
+        )
+
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            {externs}
+
+            extern "C" __device__ void {symbol}(void* state, void* offset) {{
+                {calls}
+            }}
+        """).strip()
+
+        code = compile_cpp_op_code(source)
+
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[c for op in child_ops for c in [op.code, *op.extra_code]],
+        )
+
+    def _make_input_deref_op(self) -> Op | None:
+        """Provide Op for input deref that calls all child iterator input derefs."""
+        child_ops = [it.get_input_deref_op() for it in self._iterators]
+        if not all(op is not None for op in child_ops):
+            return None
+
+        symbol = self._make_input_deref_symbol()
+
+        externs = "\n".join(
+            f'extern "C" __device__ void {op.name}(void* state, void* result);'
+            for op in child_ops
+        )
+
+        calls = "\n        ".join(
+            f"{op.name}(static_cast<char*>(state) + {state_off}, "
+            f"static_cast<char*>(result) + {val_off});"
+            for op, state_off, val_off in zip(
+                child_ops, self._state_offsets, self._value_offsets
+            )
+        )
+
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            {externs}
+
+            extern "C" __device__ void {symbol}(void* state, void* result) {{
+                {calls}
+            }}
+        """).strip()
+
+        code = compile_cpp_op_code(source)
+
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[c for op in child_ops for c in [op.code, *op.extra_code]],
+        )
+
+    def _make_output_deref_op(self) -> Op | None:
+        """Provide Op for output deref that calls all child iterator output derefs."""
+        child_ops = [it.get_output_deref_op() for it in self._iterators]
+        if not all(op is not None for op in child_ops):
+            return None
+
+        symbol = self._make_output_deref_symbol()
+
+        externs = "\n".join(
+            f'extern "C" __device__ void {op.name}(void* state, void* value);'
+            for op in child_ops
+        )
+
+        calls = "\n        ".join(
+            f"{op.name}(static_cast<char*>(state) + {state_off}, "
+            f"static_cast<char*>(value) + {val_off});"
+            for op, state_off, val_off in zip(
+                child_ops, self._state_offsets, self._value_offsets
+            )
+        )
+
+        source = dedent(f"""
+            {CUDA_PREAMBLE}
+
+            {externs}
+
+            extern "C" __device__ void {symbol}(void* state, void* value) {{
+                {calls}
+            }}
+        """).strip()
+
+        code = compile_cpp_op_code(source)
+
+        return Op(
+            operator_type=OpKind.STATELESS,
+            name=symbol,
+            ltoir=code,
+            extra_ltoirs=[c for op in child_ops for c in [op.code, *op.extra_code]],
+        )
+
+    @property
+    def children(self):
+        return tuple(self._iterators)
+
+    def __add__(self, offset: int) -> "ZipIterator":
+        """Advance all child iterators by offset."""
+        advanced_iterators = [it + offset for it in self._iterators]  # type: ignore[operator]
+        return ZipIterator(*advanced_iterators)
+
+    @property
+    def kind(self):
+        """Return a hashable kind for caching purposes."""
+        return ("ZipIterator", tuple(it.kind for it in self._iterators))
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/op.py b/cccl_upstream/python/cuda_cccl/cuda/compute/op.py
new file mode 100644
index 00000000..02eeaff3
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/op.py
@@ -0,0 +1,253 @@
+# Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+
+from __future__ import annotations
+
+from ._bindings import Op, OpKind
+from ._caching import CachableFunction, cache_with_registered_key_functions
+from ._device_code import DeviceCode
+
+
+def _is_well_known_op(op: OpKind) -> bool:
+    return isinstance(op, OpKind) and op not in (OpKind.STATELESS, OpKind.STATEFUL)
+
+
+class _OpAdapter:
+    """
+    Provides a unified interface for operators, whether they are:
+    - Well-known operations (OpKind.PLUS, OpKind.MAXIMUM, etc.)
+    - Stateless user-provided callables
+    - Stateful user-provided callables
+    """
+
+    def compile(self, input_types, output_type=None) -> Op:
+        """
+        Compile this operator to an Op for CCCL interop.
+
+        Args:
+            input_types: Tuple of TypeDescriptors for input arguments
+            output_type: Optional TypeDescriptor for return value (inferred if None)
+
+        Returns:
+            Compiled Op object for C++ interop
+        """
+        raise NotImplementedError("Subclasses must implement this method")
+
+    @property
+    def is_stateful(self) -> bool:
+        """Return True if this op has runtime state."""
+        return False
+
+    def get_state(self) -> bytes:
+        """
+        Return the op's state bytes.
+        """
+        return b""
+
+    def get_return_type(self, input_types):
+        """Get the return type for this op given input types."""
+        raise NotImplementedError(
+            f"get_return_type not implemented for {self.__class__.__name__}"
+        )
+
+
+class _WellKnownOp(_OpAdapter):
+    """Internal wrapper for well-known OpKind values."""
+
+    __slots__ = ["_kind"]
+
+    def __init__(self, kind: OpKind):
+        if not _is_well_known_op(kind):
+            raise ValueError(
+                f"OpKind.{kind.name} is not a well-known operation. "
+                "Use OpKind.PLUS, OpKind.MAXIMUM, etc."
+            )
+        self._kind = kind
+
+    def compile(self, input_types, output_type=None) -> Op:
+        return Op(
+            operator_type=self._kind,
+            name="",
+            ltoir=b"",
+            state_alignment=1,
+            state=b"",
+        )
+
+    @property
+    def kind(self) -> OpKind:
+        """The underlying OpKind."""
+        return self._kind
+
+    def __eq__(self, other):
+        if not isinstance(other, _WellKnownOp):
+            return False
+        return self._kind == other._kind
+
+    def __hash__(self):
+        return hash(self._kind)
+
+
+class RawOp(_OpAdapter):
+    """
+    ``RawOp`` lets you supply pre-compiled device code (LTO-IR) implementing a
+    custom operator, bypassing the default Numba-based JIT pipeline.
+
+    Example:
+        Supplying C++ device code compiled to LTO-IR via NVRTC:
+
+        .. literalinclude:: ../../python/cuda_cccl/tests/compute/examples/raw_op/cpp_stateless.py
+            :language: python
+            :start-after: # example-begin
+
+    Args:
+        name: The ABI name of the operator.
+        ltoir: Raw ``bytes`` of pre-compiled LTO-IR implementing the operator
+            (for example, produced by ``nvcc -dlto`` or NVRTC).
+        state: Optional bytes representing the operator's state.
+        state_alignment: Alignment requirement for the state bytes (default: 1).
+        extra_ltoirs: Optional list of additional LTO-IR ``bytes`` to link.
+
+    Notes:
+        - The provided code must define a function with the specified name and the correct signature.
+        - The function must use untyped pointers for all parameters and return type. The function body
+          is responsible for correctly interpreting the pointer arguments based on the expected input and output types.
+          For stateless operators, the signature is
+
+             void func(void* arg1, void* arg2, ..., void* result)`
+
+          For stateful operators, the first parameter must be a pointer to the state:
+
+             void func(void* state, void* arg1, void* arg2, ...)
+    """
+
+    __slots__ = [
+        "_ltoir",
+        "_name",
+        "_state",
+        "_state_alignment",
+        "_extra_ltoirs",
+    ]
+
+    def __init__(
+        self,
+        *,
+        ltoir: bytes | DeviceCode,
+        name: str,
+        state: bytes = b"",
+        state_alignment: int = 1,
+        extra_ltoirs: list[bytes | DeviceCode] | None = None,
+    ):
+        self._ltoir = ltoir
+        self._name = name
+        self._state = state
+        self._state_alignment = state_alignment
+        self._extra_ltoirs = extra_ltoirs or []
+
+    def compile(self, input_types, output_type=None) -> Op:
+        # Determine if stateful based on whether state is provided
+        op_kind = OpKind.STATEFUL if self._state else OpKind.STATELESS
+
+        return Op(
+            operator_type=op_kind,
+            name=self._name,
+            ltoir=self._ltoir,
+            state=self._state,
+            state_alignment=self._state_alignment,
+            extra_ltoirs=self._extra_ltoirs,
+        )
+
+    def get_state(self) -> bytes:
+        """Return the op's state bytes."""
+        return self._state
+
+    @property
+    def _identity(self):
+        return (
+            self._ltoir,
+            self._name,
+            self._state,
+            self._state_alignment,
+            tuple(self._extra_ltoirs),
+        )
+
+    def __eq__(self, other):
+        if not isinstance(other, RawOp):
+            return False
+        return self._identity == other._identity
+
+    def __hash__(self):
+        return hash(self._identity)
+
+
+# Public aliases
+OpAdapter = _OpAdapter
+
+
+def _jit_op_adapter_factory():
+    # helper that tries to import `_jit.py`. If it fails,
+    # returns a function that raises an appropriate error when called.
+    try:
+        from ._jit import to_jit_op_adapter
+
+        return to_jit_op_adapter
+    except ModuleNotFoundError as e:
+        if "numba" in str(e):
+
+            def _missing_jit_adapter(op):
+                raise ImportError(
+                    "numba-cuda is required to JIT compile Python callables"
+                )
+
+            return _missing_jit_adapter
+        raise
+
+
+to_jit_op_adapter = _jit_op_adapter_factory()
+
+
+def make_op_adapter(op) -> OpAdapter:
+    """
+    Create an Op from a callable or well-known OpKind.
+
+    Args:
+        op: Callable or OpKind
+
+    Returns:
+        A value with appropriate subtype of _BaseOp
+    """
+    # Already an _OpAdapter instance:
+    if isinstance(op, _OpAdapter):
+        return op
+
+    # Well-known operation
+    if isinstance(op, OpKind):
+        return _WellKnownOp(op)
+
+    # It's a Python callable
+    return to_jit_op_adapter(op)
+
+
+cache_with_registered_key_functions.register(
+    _WellKnownOp, lambda op: (op._kind.name, op._kind.value)
+)
+
+cache_with_registered_key_functions.register(
+    OpKind, lambda kind: (kind.name, kind.value)
+)
+
+cache_with_registered_key_functions.register(
+    type(lambda: None), lambda func: CachableFunction(func)
+)
+
+cache_with_registered_key_functions.register(RawOp, lambda op: op._identity)
+
+
+__all__ = [
+    "OpAdapter",
+    "OpKind",
+    "make_op_adapter",
+    "RawOp",
+]
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/struct.py b/cccl_upstream/python/cuda_cccl/cuda/compute/struct.py
new file mode 100644
index 00000000..63290f72
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/struct.py
@@ -0,0 +1,194 @@
+# Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+import functools
+from types import new_class
+from typing import Any, ClassVar, TypeGuard, Union, cast, get_type_hints
+
+import numpy as np
+
+from . import types
+
+"""
+This module provides `gpu_struct`, a factory for producing struct types.
+
+"""
+
+
+def gpu_struct(
+    field_dict: Union[dict, np.dtype, type],
+    name: str = "AnonymousStruct",
+):
+    """
+    A factory for creating struct types.
+
+    Args:
+        field_dict
+                A dictionary, numpy dtype, or annotated class providing the
+            mapping of field names to data types.
+
+        name
+            The name of the struct type that will be returned.
+
+    Returns:
+        A struct class helpful for writing operations on struct values.
+    """
+
+    # Handle numpy dtype input
+    if isinstance(field_dict, np.dtype):
+        if field_dict.type != np.void or field_dict.fields is None:
+            field_dict = {}
+        else:
+            field_dict = {
+                name: field_info[0] for name, field_info in field_dict.fields.items()
+            }
+
+    # Handle annotated class (decorator usage)
+    if isinstance(field_dict, type) and hasattr(field_dict, "__annotations__"):
+        name = field_dict.__name__
+        field_dict = get_type_hints(field_dict)
+
+    # At this point, field_dict must be a dict
+    assert isinstance(field_dict, dict)
+
+    # Validate field names are valid Python identifiers
+    for key in field_dict:
+        if not isinstance(key, str) or not key.isidentifier():
+            raise ValueError(
+                f"gpu_struct field name {key!r} is not a valid Python identifier"
+            )
+
+    # Normalize fields for storage on the struct class
+    field_spec = {}
+    for key, val in field_dict.items():
+        if _is_struct_type(val):
+            field_spec[key] = val
+        elif isinstance(val, dict):
+            # Nested struct definition - recursively create inner struct
+            field_spec[key] = gpu_struct(val, name=key)
+        else:
+            field_spec[key] = val
+
+    # Create a simple Python class for user-facing struct values
+    struct_class = cast(type[_Struct], new_class(name, bases=(_Struct,)))
+    struct_class._field_spec = field_spec
+    struct_class._type_descriptor = _get_struct_type_descriptor(struct_class)  # type: ignore[arg-type]
+    struct_class.dtype = _get_struct_record_dtype(struct_class)  # type: ignore[arg-type]
+
+    return struct_class
+
+
+class _Struct:
+    """Internal base class for all gpu_structs."""
+
+    _field_spec: ClassVar[dict[str, Any]]
+    _type_descriptor: ClassVar[types.StructTypeDescriptor]
+    dtype: ClassVar[np.dtype]
+    _fields: dict[str, Any]
+
+    @classmethod
+    def _fields_from_args(cls, *args, **kwargs):
+        field_spec = cls._field_spec
+
+        if args and isinstance(args[0], dict):
+            fields = args[0]
+        elif args:
+            assert len(args) == len(field_spec), (
+                f"Expected {len(field_spec)} arguments, got {len(args)}"
+            )
+            fields = dict(zip(field_spec.keys(), args))
+        else:
+            fields = kwargs
+
+        assert fields.keys() == field_spec.keys()
+
+        return {
+            name: _coerce_value(field_spec[name], fields[name]) for name in field_spec
+        }
+
+    def __init__(self, *args, **kwargs):
+        """Supporting construction from positional, keyword, and dict arguments."""
+
+        self._fields = self._fields_from_args(*args, **kwargs)
+
+        for name, value in self._fields.items():
+            setattr(self, name, value)
+
+        # NumPy array representation:
+        self._data = np.asarray(_as_numpy_record_value(self))
+        self.__array_interface__ = self._data.__array_interface__
+
+
+def _as_numpy_record_value(val) -> np.void:
+    """Convert a gpu_struct *value* to a numpy record."""
+
+    def _fields_to_tuples(fields_dict: dict[str, Any]) -> tuple[Any, ...]:
+        return tuple(
+            _fields_to_tuples(v._fields) if isinstance(v, _Struct) else v
+            for v in fields_dict.values()
+        )
+
+    return np.void(
+        _fields_to_tuples(val._fields),
+        dtype=_get_struct_record_dtype(type(val)),  # type: ignore[arg-type]
+    )
+
+
+@functools.cache
+def _get_struct_record_dtype(struct_class: type) -> np.dtype:
+    return _get_struct_type_descriptor(struct_class).dtype
+
+
+def _coerce_value(field_type, value: Any) -> Any:
+    if isinstance(value, _Struct):
+        return value
+
+    if isinstance(field_type, np.dtype):
+        return field_type.type(value)
+
+    if isinstance(field_type, type) and issubclass(field_type, np.generic):
+        return field_type(value)  # type: ignore[call-arg]
+
+    if isinstance(value, tuple):
+        return field_type(*value)
+
+    if isinstance(value, dict):
+        return field_type(**value)
+
+    # field_type is a class (e.g., another gpu_struct)
+    raise TypeError(f"Cannot coerce {type(value).__name__} into {field_type.__name__}")
+
+
+def _is_struct_type(typ: Any) -> TypeGuard[type[_Struct]]:
+    """Check if a type is a GPU struct class."""
+    return isinstance(typ, type) and issubclass(typ, _Struct)
+
+
+@functools.cache
+def _get_struct_type_descriptor(
+    struct_class: type,
+) -> types.StructTypeDescriptor:
+    type_descriptors = _field_spec_to_type_descriptors(
+        struct_class._field_spec  # type: ignore[attr-defined]
+    )
+    return types.struct(type_descriptors, name=struct_class.__name__)
+
+
+def _field_spec_to_type_descriptors(
+    field_spec: dict[str, Any],
+) -> dict[str, types.TypeDescriptor]:
+    type_descriptors = {}
+    for key, val in field_spec.items():
+        if isinstance(val, types.TypeDescriptor):
+            type_descriptors[key] = val
+        elif _is_struct_type(val):
+            type_descriptors[key] = val._type_descriptor
+        elif isinstance(val, np.dtype):
+            type_descriptors[key] = types.from_numpy_dtype(val)
+        else:
+            type_descriptors[key] = types.from_numpy_dtype(np.dtype(val))
+    return type_descriptors
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/types.py b/cccl_upstream/python/cuda_cccl/cuda/compute/types.py
new file mode 100644
index 00000000..4fdf7d1e
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/types.py
@@ -0,0 +1,292 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+from __future__ import annotations
+
+import inspect
+from typing import get_type_hints
+
+import numpy as np
+
+from ._bindings import TypeEnum, TypeInfo
+
+_ENUM_TO_DTYPE: dict[TypeEnum, np.dtype] = {
+    TypeEnum.INT8: np.dtype("int8"),
+    TypeEnum.INT16: np.dtype("int16"),
+    TypeEnum.INT32: np.dtype("int32"),
+    TypeEnum.INT64: np.dtype("int64"),
+    TypeEnum.UINT8: np.dtype("uint8"),
+    TypeEnum.UINT16: np.dtype("uint16"),
+    TypeEnum.UINT32: np.dtype("uint32"),
+    TypeEnum.UINT64: np.dtype("uint64"),
+    TypeEnum.FLOAT16: np.dtype("float16"),
+    TypeEnum.FLOAT32: np.dtype("float32"),
+    TypeEnum.FLOAT64: np.dtype("float64"),
+    TypeEnum.BOOLEAN: np.dtype("bool"),
+}
+
+
+class TypeDescriptor:
+    def __init__(self, size: int, alignment: int, type_enum: TypeEnum):
+        self._type_info = TypeInfo(size, alignment, type_enum)
+        self._dtype = _ENUM_TO_DTYPE.get(type_enum, np.dtype(f"V{size}"))
+
+    @property
+    def info(self) -> TypeInfo:
+        """Return the TypeInfo for this type."""
+        return self._type_info
+
+    @property
+    def size(self):
+        return self._type_info.size
+
+    @property
+    def alignment(self):
+        return self._type_info.alignment
+
+    @property
+    def dtype(self) -> np.dtype:
+        """Return the numpy dtype for this type."""
+        return self._dtype
+
+    def pointer(self) -> "PointerTypeDescriptor":
+        """Create a pointer type to this type."""
+        return PointerTypeDescriptor(self)
+
+    def __repr__(self) -> str:
+        return f"TypeDescriptor({self._dtype})"
+
+    def __eq__(self, other):
+        if not isinstance(other, TypeDescriptor):
+            return False
+        return self._dtype == other._dtype
+
+    def __hash__(self):
+        return hash(self._dtype)
+
+
+class StructTypeDescriptor(TypeDescriptor):
+    def __init__(
+        self,
+        fields: dict[str, "TypeDescriptor"],
+        name: str = "AnonStruct",
+    ):
+        dtype = _build_struct_dtype(fields)
+        if not dtype.isalignedstruct:
+            raise ValueError(f"dtype {dtype} must be aligned")
+        # Structs use STORAGE type enum
+        super().__init__(dtype.itemsize, dtype.alignment, TypeEnum.STORAGE)
+        self._dtype = dtype
+        self._fields = fields
+        self._name = name
+
+    def __repr__(self) -> str:
+        return f"StructTypeDescriptor({self._name}, dtype={self._dtype})"
+
+    @property
+    def name(self) -> str:
+        return self._name
+
+    @property
+    def fields(self) -> dict[str, "TypeDescriptor"]:
+        return self._fields
+
+    def layout_key(self) -> tuple[tuple[str, "TypeDescriptor"], ...]:
+        """Return a stable, hashable key for this struct layout."""
+        return tuple(self._fields.items())
+
+    def __eq__(self, other):
+        if not isinstance(other, StructTypeDescriptor):
+            return False
+        # Compare by fields (TypeDescriptors) to correctly distinguish structs
+        # with different pointer pointee types, which have identical numpy dtypes
+        # (both uint64) but different semantics.
+        # Must compare as ordered items because field order matters for struct layout.
+        return list(self._fields.items()) == list(other._fields.items())
+
+    def __hash__(self):
+        return hash(tuple(self._fields.items()))
+
+
+class PointerTypeDescriptor(TypeDescriptor):
+    def __init__(self, pointee: TypeDescriptor):
+        # Pointer is 8 bytes, 8-byte aligned, represented as uint64
+        super().__init__(8, 8, TypeEnum.UINT64)
+        self._pointee = pointee
+
+    @property
+    def pointee(self) -> TypeDescriptor:
+        """Return the type this pointer points to."""
+        return self._pointee
+
+    def __repr__(self) -> str:
+        return f"PointerTypeDescriptor({self._pointee})"
+
+    def __eq__(self, other):
+        if not isinstance(other, PointerTypeDescriptor):
+            return False
+        return self._pointee == other._pointee
+
+    def __hash__(self):
+        return hash(("PointerTypeDescriptor", self._pointee))
+
+
+def _build_struct_dtype(
+    fields: dict[str, TypeDescriptor],
+) -> np.dtype:
+    dtype_list = []
+    for field_name, field_type in fields.items():
+        dtype_list.append((field_name, field_type.dtype))
+    return np.dtype(dtype_list, align=True)
+
+
+def struct(
+    fields: dict[str, TypeDescriptor],
+    name: str = "AnonStruct",
+) -> StructTypeDescriptor:
+    """Create a type descriptor for a struct."""
+    return StructTypeDescriptor(fields, name=name)
+
+
+def pointer(pointee: TypeDescriptor) -> PointerTypeDescriptor:
+    """
+    Create a pointer to the given type.
+    """
+    return PointerTypeDescriptor(pointee)
+
+
+def from_numpy_dtype(dtype: np.dtype | type) -> TypeDescriptor:
+    """
+    Convert a numpy dtype (or numpy type) to a TypeDescriptor.
+
+    Handles POD types and structured dtypes (recursively for nested structs).
+    """
+    dtype = np.dtype(dtype)
+
+    # Check if it's a known POD type
+    td = _DTYPE_TO_TD.get(dtype)
+    if td is not None:
+        return td
+
+    # Handle structured dtypes (structs)
+    if dtype.names is not None:
+        fields: dict[str, TypeDescriptor] = {}
+        assert dtype.fields is not None
+        for name in dtype.names:
+            field_info = dtype.fields[name]
+            field_dtype = field_info[0]
+            fields[name] = from_numpy_dtype(field_dtype)
+        return struct(fields)  # type: ignore[arg-type]
+
+    # Some other NumPy type (e.g., complex64) for which we don't
+    # have a specific TypeDescriptor. Use STORAGE and preserve the original dtype.
+    td = TypeDescriptor(dtype.itemsize, dtype.alignment, TypeEnum.STORAGE)
+    td._dtype = dtype
+    return td
+
+
+# Signed integer types
+int8 = TypeDescriptor(1, 1, TypeEnum.INT8)
+int16 = TypeDescriptor(2, 2, TypeEnum.INT16)
+int32 = TypeDescriptor(4, 4, TypeEnum.INT32)
+int64 = TypeDescriptor(8, 8, TypeEnum.INT64)
+
+# Unsigned integer types
+uint8 = TypeDescriptor(1, 1, TypeEnum.UINT8)
+uint16 = TypeDescriptor(2, 2, TypeEnum.UINT16)
+uint32 = TypeDescriptor(4, 4, TypeEnum.UINT32)
+uint64 = TypeDescriptor(8, 8, TypeEnum.UINT64)
+
+# Floating point types
+float16 = TypeDescriptor(2, 2, TypeEnum.FLOAT16)
+float32 = TypeDescriptor(4, 4, TypeEnum.FLOAT32)
+float64 = TypeDescriptor(8, 8, TypeEnum.FLOAT64)
+
+# Boolean
+boolean = TypeDescriptor(1, 1, TypeEnum.BOOLEAN)
+
+
+# Mapping from numpy dtype to TypeDescriptor for POD types
+_DTYPE_TO_TD: dict[np.dtype, TypeDescriptor] = {
+    np.dtype("int8"): int8,
+    np.dtype("int16"): int16,
+    np.dtype("int32"): int32,
+    np.dtype("int64"): int64,
+    np.dtype("uint8"): uint8,
+    np.dtype("uint16"): uint16,
+    np.dtype("uint32"): uint32,
+    np.dtype("uint64"): uint64,
+    np.dtype("float16"): float16,
+    np.dtype("float32"): float32,
+    np.dtype("float64"): float64,
+    np.dtype("bool"): boolean,
+}
+
+
+def to_ctypes_type(td: TypeDescriptor):
+    """Convert a TypeDescriptor to a ctypes type."""
+    return np.ctypeslib.as_ctypes_type(td.dtype)
+
+
+def _annotation_to_type_descriptor(annotation):
+    """
+    Convert a type annotation to a TypeDescriptor.
+
+    Handles:
+    - TypeDescriptor: returns as-is
+    - gpu_struct classes: returns their _type_descriptor
+    - numpy dtypes/types: converts via from_numpy_dtype
+    """
+    from .struct import _is_struct_type
+
+    if isinstance(annotation, TypeDescriptor):
+        return annotation
+
+    if _is_struct_type(annotation):
+        return annotation._type_descriptor  # type: ignore[union-attr]
+
+    # numpy dtype or type
+    return from_numpy_dtype(np.dtype(annotation))
+
+
+def signature_from_annotations(py_func):
+    try:
+        annotations = get_type_hints(py_func)
+    except Exception:
+        annotations = py_func.__annotations__
+    spec = inspect.getfullargspec(py_func)
+    arg_names = list(spec.args)
+
+    input_tds = []
+    # Try to get input types from annotations
+    for name in arg_names:
+        if name in annotations:
+            input_tds.append(_annotation_to_type_descriptor(annotations[name]))
+            break
+
+    if "return" in annotations:
+        output_td = _annotation_to_type_descriptor(annotations["return"])
+    else:
+        output_td = None
+
+    return input_tds, output_td
+
+
+__all__ = [
+    "int8",
+    "int16",
+    "int32",
+    "int64",
+    "uint8",
+    "uint16",
+    "uint32",
+    "uint64",
+    "float16",
+    "float32",
+    "float64",
+    "boolean",
+    "struct",
+    "pointer",
+    "from_numpy_dtype",
+    "to_ctypes_type",
+]
diff --git a/cccl_upstream/python/cuda_cccl/cuda/compute/typing.py b/cccl_upstream/python/cuda_cccl/cuda/compute/typing.py
new file mode 100644
index 00000000..8d88f1ab
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/cuda/compute/typing.py
@@ -0,0 +1,64 @@
+# Copyright (c) 2024-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from typing import Callable, Protocol, TypeVar
+
+from .iterators import IteratorBase
+from .op import OpAdapter, OpKind, RawOp
+from .struct import _Struct
+
+
+class DeviceArrayLike(Protocol):
+    """Protocol for array-like objects that expose device memory via CUDA Array Interface.
+
+    Any object implementing the ``__cuda_array_interface__`` attribute can be used
+    where a :class:`DeviceArrayLike` is expected. This includes CuPy arrays, Numba
+    device arrays, PyTorch CUDA tensors, and other GPU array types.
+
+    See `CUDA Array Interface specification <https://nvidia.github.io/numba-cuda/user/cuda_array_interface.html>`_
+    for details.
+    """
+
+    __cuda_array_interface__: dict
+
+
+class StreamLike(Protocol):
+    """Protocol for CUDA stream objects.
+
+    Any object implementing the ``__cuda_stream__()`` method can be used where a
+    :class:`StreamLike` is expected. This includes stream objects from CuPy,
+    Numba CUDA, PyTorch, and other CUDA libraries.
+    """
+
+    def __cuda_stream__(self) -> tuple[int, int]: ...
+
+
+GpuStruct = TypeVar("GpuStruct", bound=_Struct)
+"""
+Instance of types created with :class:`cuda.compute.struct.gpu_struct`.
+"""
+
+IteratorT = TypeVar("IteratorT", bound=IteratorBase)
+"""Type variable for iterator objects.
+
+Represents any subclass of :class:`IteratorBase <cuda.compute.iterators.IteratorBase>`.
+See :py:mod:`cuda.compute.iterators` for all available iterators.
+"""
+
+Operator = Callable | OpKind | RawOp | OpAdapter
+"""Type alias for operator objects passed to algorithm functions.
+
+Algorithms accept the following objects as operators:
+
+* Python functions or lambdas implementing the operator. This function will be JIT
+  compiled into device code using `numba.cuda <https://nvidia.github.io/numba-cuda/>`_.
+
+* :class:`OpKind <cuda.compute.op.OpKind>` enumerators which are pre-defined constants
+  for common operations.
+
+* :class:`RawOp <cuda.compute.op.RawOp` objects containing pre-compiled device code.
+"""
+
+__all__ = ["DeviceArrayLike", "GpuStruct", "IteratorT", "Operator"]
diff --git a/cccl_upstream/python/cuda_cccl/merge_cuda_wheels.py b/cccl_upstream/python/cuda_cccl/merge_cuda_wheels.py
new file mode 100755
index 00000000..33555236
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/merge_cuda_wheels.py
@@ -0,0 +1,196 @@
+#!/usr/bin/env python3
+"""
+Script to merge CUDA-specific wheels into a single multi-CUDA wheel.
+
+This script takes wheels built for different CUDA versions (cu12, cu13) and merges them
+into a single wheel that supports both CUDA versions.
+
+In particular, each wheel contains a CUDA-specific build of the `cccl.c.parallel` library
+and the associated bindings. These are present in the directory `compute/cu<version>`.
+For example, for a wheel built with CUDA 12, the directory is `compute/cu12`,
+and for a wheel built with CUDA 13, the directory is `compute/cu13`.
+This script merges these directories into a single wheel that supports both CUDA versions, i.e.,
+containing both `compute/cu12` and `compute/cu13`.
+At runtime, a shim module `compute/_bindings.py` is used to import the appropriate
+CUDA-specific bindings. See `compute/_bindings.py` for more details.
+"""
+
+import argparse
+import shutil
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+from typing import List
+
+
+def run_command(
+    cmd: List[str], cwd: Path = None, env: dict = None
+) -> subprocess.CompletedProcess:
+    """Run a command with error handling."""
+    print(f"Running: {' '.join(cmd)}")
+    if cwd:
+        print(f"  Working directory: {cwd}")
+
+    result = subprocess.run(cmd, cwd=cwd, env=env, capture_output=True, text=True)
+
+    if result.returncode != 0:
+        print(f"Command failed with return code {result.returncode}")
+        print("STDOUT:", result.stdout)
+        print("STDERR:", result.stderr)
+        sys.exit(1)
+
+    return result
+
+
+def merge_wheels(wheels: List[Path], output_dir: Path) -> Path:
+    """Merge multiple wheels into a single wheel with version-specific binaries."""
+    print("\n=== Merging wheels ===")
+    print(f"Input wheels: {[w.name for w in wheels]}")
+
+    if len(wheels) == 1:
+        # Single wheel, just copy it and remove CUDA version suffix
+        output_dir.mkdir(parents=True, exist_ok=True)
+        final_wheel = output_dir / wheels[0].name.replace(
+            f".cu{wheels[0].name.split('.cu')[1].split('.')[0]}.whl", ".whl"
+        )
+        shutil.copy2(wheels[0], final_wheel)
+        print(f"Single wheel copied to: {final_wheel}")
+        return final_wheel
+
+    # Extract all wheels to temporary directories
+    with tempfile.TemporaryDirectory() as temp_dir:
+        temp_path = Path(temp_dir)
+        extracted_wheels = []
+
+        for i, wheel in enumerate(wheels):
+            print(f"Extracting wheel {i + 1}/{len(wheels)}: {wheel.name}")
+            # Extract wheel - wheel unpack creates the directory itself
+            run_command(
+                [
+                    sys.executable,
+                    "-m",
+                    "wheel",
+                    "unpack",
+                    str(wheel),
+                    "--dest",
+                    str(temp_path),
+                ]
+            )
+
+            # Find the extracted directory (wheel unpack creates a subdirectory)
+            extract_dir = None
+            for item in temp_path.iterdir():
+                if item.is_dir() and item.name.startswith("cuda_cccl"):
+                    extract_dir = item
+                    break
+
+            if not extract_dir:
+                raise RuntimeError(
+                    f"Could not find extracted wheel directory for {wheel.name}"
+                )
+
+            # Rename to our expected name
+            expected_name = temp_path / f"wheel_{i}"
+            extract_dir.rename(expected_name)
+            extract_dir = expected_name
+
+            extracted_wheels.append(extract_dir)
+
+        # Use the first wheel as the base and merge binaries from others
+        base_wheel = extracted_wheels[0]
+
+        # now copy the version-specific directory from other wheels
+        # into the appropriate place in the base wheel
+        for i, wheel_dir in enumerate(extracted_wheels):
+            cuda_version = wheels[i].name.split(".cu")[1].split(".")[0]
+            if i == 0:
+                # For base wheel, do nothing
+                continue
+            else:
+                version_dir = Path("cuda") / "compute" / f"cu{cuda_version}"
+                # Copy from other wheels
+                print(f"  Copying {version_dir} to {base_wheel}")
+                shutil.copytree(wheel_dir / version_dir, base_wheel / version_dir)
+
+        # Repack the merged wheel
+        output_dir.mkdir(parents=True, exist_ok=True)
+
+        # Create a clean wheel name without CUDA version suffixes
+        base_wheel_name = wheels[0].name
+        # Remove any .cu* suffix from the wheel name
+        if ".cu" in base_wheel_name:
+            base_wheel_name = base_wheel_name.split(".cu")[0] + ".whl"
+
+        print(f"Repacking merged wheel as: {base_wheel_name}")
+        run_command(
+            [
+                sys.executable,
+                "-m",
+                "wheel",
+                "pack",
+                str(base_wheel),
+                "--dest-dir",
+                str(output_dir),
+            ]
+        )
+
+        # Find the output wheel
+        output_wheels = list(output_dir.glob("*.whl"))
+        if not output_wheels:
+            raise RuntimeError("Failed to create merged wheel")
+
+        merged_wheel = output_wheels[0]
+        print(f"Successfully merged wheel: {merged_wheel}")
+        return merged_wheel
+
+
+def main():
+    """Main merge script."""
+    parser = argparse.ArgumentParser(
+        description="Merge CUDA-specific wheels into a single multi-CUDA wheel"
+    )
+    parser.add_argument(
+        "wheels", nargs="+", help="Paths to the CUDA-specific wheels to merge"
+    )
+    parser.add_argument(
+        "--output-dir", "-o", default="dist", help="Output directory for merged wheel"
+    )
+
+    args = parser.parse_args()
+
+    print("CUDA CCCL Wheel Merger")
+    print("======================")
+
+    # Convert wheel paths to Path objects and validate
+    wheels = []
+    for wheel_path in args.wheels:
+        wheel = Path(wheel_path)
+        if not wheel.exists():
+            print(f"Error: Wheel not found: {wheel}")
+            sys.exit(1)
+        if not wheel.name.endswith(".whl"):
+            print(f"Error: Not a wheel file: {wheel}")
+            sys.exit(1)
+        wheels.append(wheel)
+
+    if not wheels:
+        print("Error: No wheels provided")
+        sys.exit(1)
+
+    output_dir = Path(args.output_dir)
+
+    # Check that we have wheel tool available
+    try:
+        run_command([sys.executable, "-m", "wheel", "--help"])
+    except Exception:
+        print("Error: wheel package not available. Install with: pip install wheel")
+        sys.exit(1)
+
+    # Merge the wheels
+    merged_wheel = merge_wheels(wheels, output_dir)
+    print(f"\nMerge complete! Output: {merged_wheel}")
+
+
+if __name__ == "__main__":
+    main()
diff --git a/cccl_upstream/python/cuda_cccl/pyproject.toml b/cccl_upstream/python/cuda_cccl/pyproject.toml
new file mode 100644
index 00000000..b37553ec
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/pyproject.toml
@@ -0,0 +1,204 @@
+# Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+[build-system]
+# cmake>=3.30 is needed for FindCUDAToolkit's CUDA::nvfatbin /
+# CUDA::nvfatbin_static imported targets, which the v2 (HostJIT) backend links
+# against. Listed here so isolated builds (pip's default) pick it up.
+requires = ["scikit-build-core>=0.10", "setuptools_scm", "cython>=3.1.0", "cmake>=3.30"]
+build-backend = "scikit_build_core.build"
+
+[project]
+name = "cuda-cccl"
+description = "CUDA Core Compute Libraries for Python"
+authors = [{ name = "NVIDIA Corporation" }]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Developers",
+  "Topic :: Software Development :: Libraries :: Python Modules",
+  "Programming Language :: Python :: 3 :: Only",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Programming Language :: Python :: 3.14",
+  "Environment :: GPU :: NVIDIA CUDA",
+  "License :: OSI Approved :: Apache Software License",
+  "Operating System :: POSIX :: Linux",
+  "Operating System :: Microsoft :: Windows",
+]
+requires-python = ">=3.10"
+
+dependencies = [
+  "numpy",
+  "cuda-pathfinder>=1.2.3",
+  "cuda-core",
+  "typing_extensions",
+]
+
+dynamic = ["version"]
+readme = { file = "README.md", content-type = "text/markdown" }
+
+[project.optional-dependencies]
+minimal-cu12 = [
+  "cuda-bindings>=12.9.1,<13.0.0",
+  "cuda-toolkit[nvrtc,nvjitlink,cudart,nvcc]==12.*",
+]
+minimal-cu13 = [
+  "cuda-bindings>=13.0.0,<14.0.0",
+  "cuda-toolkit[nvrtc,nvjitlink,cudart,nvcc,nvvm]==13.*",
+]
+# sysctk variants: like cu12/cu13 but without cuda-toolkit — the user is
+# responsible for providing a compatible CUDA toolkit
+minimal-sysctk12 = [
+  "cuda-bindings>=12.9.1,<13.0.0",
+]
+minimal-sysctk13 = [
+  "cuda-bindings>=13.0.0,<14.0.0",
+]
+cu12 = [
+  "cuda-cccl[minimal-cu12]",
+  # numba-cuda currently expects numba.cuda.types.NPDatetime, removed in numba 0.66.
+  "numba>=0.60.0,<0.66",
+  "numba-cuda[cu12]>=0.23.0,!=0.27.*,!=0.28.*,!=0.29.*,!=0.30.0",
+]
+cu13 = [
+  "cuda-cccl[minimal-cu13]",
+  # numba-cuda currently expects numba.cuda.types.NPDatetime, removed in numba 0.66.
+  "numba>=0.60.0,<0.66",
+  "numba-cuda[cu13]>=0.23.0,!=0.27.*,!=0.28.*,!=0.29.*,!=0.30.0"
+]
+sysctk12 = [
+  "cuda-cccl[minimal-sysctk12]",
+  # numba-cuda currently expects numba.cuda.types.NPDatetime, removed in numba 0.66.
+  "numba>=0.60.0,<0.66",
+  # plain numba-cuda (no [cu12]) so it relies on the system-provided CUDA toolkit
+  # instead of pulling the toolkit from pip -- the whole point of the sysctk extra.
+  "numba-cuda>=0.23.0,!=0.27.*,!=0.28.*,!=0.29.*,!=0.30.0"
+]
+sysctk13 = [
+  "cuda-cccl[minimal-sysctk13]",
+  # numba-cuda currently expects numba.cuda.types.NPDatetime, removed in numba 0.66.
+  "numba>=0.60.0,<0.66",
+  # plain numba-cuda (no [cu13]) so it relies on the system-provided CUDA toolkit
+  # instead of pulling the toolkit from pip -- the whole point of the sysctk extra.
+  "numba-cuda>=0.23.0,!=0.27.*,!=0.28.*,!=0.29.*,!=0.30.0",
+]
+# The test extras do not include CuPy. The cuda.compute examples (run via
+# tests/test_examples.py) require it; install it separately (e.g.
+# `pip install cupy-cuda13x`).
+test-cu12 = [
+  # an undocumented way to inherit the dependencies of the cu12 extra.
+  # GH pypa #11296
+  "cuda-cccl[cu12]",
+  "pytest",
+  "pytest-xdist",
+]
+test-cu13 = ["cuda-cccl[cu13]", "pytest", "pytest-xdist"]
+test-sysctk12 = [
+  "cuda-cccl[sysctk12]",
+  "pytest",
+  "pytest-xdist",
+]
+test-sysctk13 = [
+  "cuda-cccl[sysctk13]",
+  "pytest",
+  "pytest-xdist",
+]
+bench-cu12 = [
+  "cuda-cccl[cu12]",
+  "cuda-bench[cu12]",
+  "cupy-cuda12x",
+  "pytest-benchmark",
+]
+bench-cu13 = [
+  "cuda-cccl[cu13]",
+  "cuda-bench[cu13]",
+  "cupy-cuda13x",
+  "pytest-benchmark",
+]
+bench-sysctk12 = [
+  "cuda-cccl[sysctk12]",
+  "cuda-bench[cu12]",
+  "cupy-cuda12x",
+  "pytest-benchmark",
+]
+bench-sysctk13 = [
+  "cuda-cccl[sysctk13]",
+  "cuda-bench[cu13]",
+  "cupy-cuda13x",
+  "pytest-benchmark",
+]
+
+[project.urls]
+Homepage = "https://github.com/NVIDIA/cccl"
+Repository = "https://github.com/NVIDIA/cccl"
+Documentation = "https://nvidia.github.io/cccl"
+Issues = "https://github.com/NVIDIA/cccl/issues"
+
+[tool.scikit-build]
+minimum-version = "build-system.requires"
+build-dir = "build/{wheel_tag}"
+
+[tool.scikit-build.cmake]
+version = ">=3.30"
+args = []
+build-type = "Release"
+source-dir = "."
+
+[tool.scikit-build.ninja]
+version = ">=1.11"
+make-fallback = true
+
+[tool.scikit-build.metadata.version]
+provider = "scikit_build_core.metadata.setuptools_scm"
+
+[tool.setuptools_scm]
+root = "../.."
+git_describe_command = ["git", "describe", "--tags", "--match", "v[0-9]*"]
+
+
+[tool.scikit-build.wheel.packages]
+"cuda" = "cuda"
+
+[tool.mypy]
+cache_dir = "../../.cache/mypy"
+python_version = "3.10"
+
+[[tool.mypy.overrides]]
+module = [
+  # numpy>=2.3 stubs use PEP 695 syntax that mypy can't parse under
+  # python_version=3.10; skip following its imports to keep the 3.10 target.
+  "numpy",
+  "numpy.*",
+  "numba.*",
+  "llvmlite.*",
+  "cuda.cccl.*",
+  "cuda.bindings.*",
+  "cuda.core.*",
+  "cuda.pathfinder.*",
+]
+ignore_missing_imports = true
+follow_imports = "skip"
+
+[tool.ruff]
+extend = "../../pyproject.toml"
+
+[tool.ruff.lint.isort]
+known-first-party = [
+  "cuda.cccl",
+  "cuda.cccl.headers",
+  "cuda.compute",
+]
+
+[tool.pytest.ini_options]
+# Make shared helpers in tests/_utils importable when pytest runs outside tests/.
+pythonpath = ["tests"]
+markers = [
+  "no_verify_sass: skip SASS verification check",
+  "large: tests requiring large device memory allocations",
+  "no_numba: tests that should not import numba or numba.cuda",
+  "free_threading: tests requiring free-threaded CPython with the GIL disabled",
+  "serialization: serialize/deserialize tests (skipped on the v2 HostJIT backend)",
+]
diff --git a/cccl_upstream/python/cuda_cccl/tests/_utils/__init__.py b/cccl_upstream/python/cuda_cccl/tests/_utils/__init__.py
new file mode 100644
index 00000000..b221129e
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/_utils/__init__.py
@@ -0,0 +1,5 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Shared test utilities for cuda-cccl."""
diff --git a/cccl_upstream/python/cuda_cccl/tests/_utils/device_array.py b/cccl_upstream/python/cuda_cccl/tests/_utils/device_array.py
new file mode 100644
index 00000000..b2496173
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/_utils/device_array.py
@@ -0,0 +1,252 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+import math
+import operator
+from collections.abc import Iterable
+
+import numpy as np
+from numpy.typing import DTypeLike
+
+from cuda.core import Buffer, Device, Stream
+
+
+def get_compute_capability() -> tuple[int, int]:
+    return Device().compute_capability
+
+
+def _normalize_shape(shape: int | Iterable[int]) -> tuple[int, ...]:
+    try:
+        dimensions = (operator.index(shape),)  # type: ignore[arg-type]
+    except TypeError:
+        dimensions = tuple(operator.index(dimension) for dimension in shape)  # type: ignore[union-attr]
+
+    if any(dimension < 0 for dimension in dimensions):
+        raise ValueError("negative dimensions are not allowed")
+
+    return dimensions
+
+
+def _contiguous_strides(
+    shape: tuple[int, ...], itemsize: int, order: str
+) -> tuple[int, ...]:
+    if any(dimension == 0 for dimension in shape):
+        return (0,) * len(shape)
+
+    strides = [0] * len(shape)
+    stride = itemsize
+
+    if order == "C":
+        for index in range(len(shape) - 1, -1, -1):
+            strides[index] = stride
+            stride *= shape[index]
+    else:
+        for index, dimension in enumerate(shape):
+            strides[index] = stride
+            stride *= dimension
+
+    return tuple(strides)
+
+
+def _resolve_device_and_stream(
+    device: Device | None, stream: Stream | None
+) -> tuple[Device, Stream]:
+    if device is None:
+        device = stream.device if stream is not None else Device()
+
+    if stream is not None and stream.device.device_id != device.device_id:
+        raise ValueError("device and stream must refer to the same device")
+
+    device.set_current()
+    return device, device.default_stream if stream is None else stream
+
+
+class DeviceArray:
+    """A small, Buffer-backed device array for cuda-cccl tests.
+
+    The class intentionally provides only allocation, NumPy transfers, array
+    metadata, and the CUDA Array Interface. Array operations and initialization
+    belong on the NumPy host arrays used by the tests.
+    """
+
+    def __init__(
+        self,
+        buffer: Buffer,
+        device: Device,
+        stream: Stream,
+        shape: tuple[int, ...],
+        dtype: np.dtype,
+        strides: tuple[int, ...],
+        order: str,
+    ) -> None:
+        self._buffer = buffer
+        self._device = device
+        self._stream = stream
+        self._order = order
+        self._shape = shape
+        self._dtype = dtype
+        self._strides = strides
+
+    @classmethod
+    def empty(
+        cls,
+        shape: int | Iterable[int],
+        dtype: DTypeLike,
+        *,
+        order: str = "C",
+        device: Device | None = None,
+        stream: Stream | None = None,
+    ) -> DeviceArray:
+        """Allocate an uninitialized device array."""
+        shape = _normalize_shape(shape)
+        dtype = np.dtype(dtype)
+        order = order.upper()
+        if order not in ("C", "F"):
+            raise ValueError("order must be either 'C' or 'F'")
+        if dtype.itemsize == 0:
+            raise ValueError("zero-sized dtypes are not supported")
+
+        device, stream = _resolve_device_and_stream(device, stream)
+        buffer = device.allocate(math.prod(shape) * dtype.itemsize, stream=stream)
+        result = cls(
+            buffer,
+            device,
+            stream,
+            shape,
+            dtype,
+            _contiguous_strides(shape, dtype.itemsize, order),
+            order,
+        )
+
+        # Device allocation is stream ordered. Synchronizing makes an empty array
+        # safe to hand to a test that subsequently uses a different stream.
+        stream.sync()
+        return result
+
+    @classmethod
+    def from_numpy(
+        cls,
+        array: np.ndarray,
+        *,
+        device: Device | None = None,
+        stream: Stream | None = None,
+    ) -> DeviceArray:
+        """Allocate a device array and initialize it from a NumPy array."""
+        host_array = np.asarray(array)
+        if host_array.dtype.itemsize == 0:
+            raise ValueError("zero-sized dtypes are not supported")
+
+        if host_array.flags.c_contiguous:
+            order = "C"
+        elif host_array.flags.f_contiguous:
+            order = "F"
+        else:
+            host_array = np.ascontiguousarray(host_array)
+            order = "C"
+
+        device, stream = _resolve_device_and_stream(device, stream)
+        buffer = device.allocate(host_array.nbytes, stream=stream)
+        result = cls(
+            buffer,
+            device,
+            stream,
+            host_array.shape,
+            host_array.dtype,
+            host_array.strides,
+            order,
+        )
+        result._copy_from_host_array(host_array, stream)
+        stream.sync()
+        return result
+
+    @property
+    def nbytes(self) -> int:
+        return self._buffer.size
+
+    @property
+    def dtype(self) -> np.dtype:
+        return self._dtype
+
+    def __len__(self) -> int:
+        if not self._shape:
+            raise TypeError("len() of unsized object")
+        return self._shape[0]
+
+    @property
+    def __cuda_array_interface__(self) -> dict[str, object]:
+        interface: dict[str, object] = {
+            "data": (0 if self.nbytes == 0 else int(self._buffer.handle), False),
+            "shape": self._shape,
+            "strides": None if self._is_c_contiguous() else self._strides,
+            "typestr": self._dtype.str,
+            "version": 3,
+        }
+        if self._dtype.fields is not None:
+            interface["descr"] = self._dtype.descr
+        return interface
+
+    def _is_c_contiguous(self) -> bool:
+        return (
+            self._order == "C"
+            or self.nbytes == 0
+            or sum(dimension > 1 for dimension in self._shape) <= 1
+        )
+
+    @staticmethod
+    def _host_buffer(array: np.ndarray) -> Buffer:
+        # Buffer.from_handle does not own the host memory. `owner` ties the NumPy
+        # allocation to this temporary Buffer; the caller also retains the array
+        # and synchronizes the copy stream before returning.
+        return Buffer.from_handle(
+            ptr=int(array.ctypes.data), size=array.nbytes, owner=array
+        )
+
+    def _copy_stream(self, stream: Stream | None) -> Stream:
+        if stream is None:
+            # The allocation stream is not necessarily the last stream to have
+            # used the array. Synchronize the device when that stream is unknown.
+            self._device.sync()
+            return self._stream
+        if stream.device.device_id != self._device.device_id:
+            raise ValueError("copy stream must belong to the array's device")
+        return stream
+
+    def _copy_from_host_array(self, array: np.ndarray, stream: Stream) -> None:
+        self._buffer.copy_from(self._host_buffer(array), stream=stream)
+
+    def copy_from_host(
+        self, array: np.ndarray, *, stream: Stream | None = None
+    ) -> None:
+        """Replace the array's contents from a shape- and dtype-matched NumPy array."""
+        host_array = np.asarray(array)
+        if host_array.shape != self._shape:
+            raise ValueError(
+                f"source shape {host_array.shape} does not match {self._shape}"
+            )
+        if host_array.dtype != self._dtype:
+            raise TypeError(
+                f"source dtype {host_array.dtype} does not match {self._dtype}"
+            )
+
+        if self._order == "F":
+            host_array = np.asfortranarray(host_array)
+        else:
+            host_array = np.ascontiguousarray(host_array)
+
+        self._device.set_current()
+        stream = self._copy_stream(stream)
+        self._copy_from_host_array(host_array, stream)
+        stream.sync()
+
+    def copy_to_host(self, *, stream: Stream | None = None) -> np.ndarray:
+        """Return an owning NumPy copy of the array."""
+        self._device.set_current()
+        stream = self._copy_stream(stream)
+
+        result = np.empty(self._shape, dtype=self._dtype, order=self._order)
+        self._buffer.copy_to(self._host_buffer(result), stream=stream)
+        stream.sync()
+        return result
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/conftest.py b/cccl_upstream/python/cuda_cccl/tests/compute/conftest.py
new file mode 100644
index 00000000..1971448d
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/conftest.py
@@ -0,0 +1,169 @@
+import builtins
+from collections.abc import Generator
+
+import numpy as np
+import pytest
+
+from cuda.core import Device, Stream
+
+try:
+    from cuda.compute._build_info import USING_V2
+except ImportError:
+    USING_V2 = False
+
+check_ldl_stl_in_sass = False
+
+
+# Define a pytest fixture that returns random arrays with different dtypes
+@pytest.fixture(
+    params=[
+        np.int8,
+        np.int16,
+        np.int32,
+        np.int64,
+        np.uint8,
+        np.uint16,
+        np.uint32,
+        np.uint64,
+        np.float16,
+        np.float32,
+        np.float64,
+        np.complex64,
+        np.complex128,
+    ]
+)
+def input_array(request):
+    dtype = np.dtype(request.param)
+    sample_size = 1000
+
+    # Generate random values based on the dtype
+    if np.issubdtype(dtype, np.integer):
+        is_unsigned = dtype.kind == "u"
+        # For integer types, use np.random.randint for random integers
+        if is_unsigned:
+            low_inclusive, high_exclusive = 0, 8
+        else:
+            low_inclusive, high_exclusive = -5, 6
+        array = np.random.randint(
+            low=low_inclusive, high=high_exclusive, size=sample_size, dtype=dtype
+        )
+    elif np.issubdtype(dtype, np.floating):
+        # For floating-point types, use np.random.random and cast to the required dtype
+        array = np.random.random(sample_size).astype(dtype)
+    elif np.issubdtype(dtype, np.complexfloating):
+        # For complex types, generate random real and imaginary parts
+        packed = np.random.random(2 * sample_size)
+        real_part = packed[:sample_size]
+        imag_part = packed[sample_size:]
+        array = (real_part + 1j * imag_part).astype(dtype)
+
+    return array
+
+
+# Define a pytest fixture that returns random floating-point arrays only
+@pytest.fixture(
+    params=[
+        np.float32,
+        np.float64,
+    ]
+)
+def floating_array(request):
+    dtype = np.dtype(request.param)
+    sample_size = 1000
+
+    # Generate random floating-point values
+    array = np.random.random(sample_size).astype(dtype)
+    return array
+
+
+@pytest.fixture(scope="function")
+def cuda_stream() -> Generator[Stream, None, None]:
+    device = Device()
+    device.set_current()
+    stream = device.create_stream()
+    try:
+        yield stream
+    finally:
+        stream.close()
+
+
+@pytest.fixture(scope="function", autouse=True)
+def verify_sass(request):
+    if request.node.get_closest_marker("no_verify_sass"):
+        return
+
+    if not check_ldl_stl_in_sass:
+        return
+
+    # Pull monkeypatch dynamically rather than as a fixture parameter so this
+    # autouse fixture does not add monkeypatch to every test's static fixture
+    # closure. pytest-run-parallel treats monkeypatch as thread-unsafe based on
+    # that closure, so a parameter here would serialize the entire free-threaded
+    # parallel sweep -- even though this fixture only patches on the opt-in
+    # SASS-check path (check_ldl_stl_in_sass, off by default and in CI).
+    monkeypatch = request.getfixturevalue("monkeypatch")
+
+    import cuda.compute._cccl_interop
+
+    monkeypatch.setattr(
+        cuda.compute._cccl_interop,
+        "_check_sass",
+        True,
+    )
+
+
+@pytest.fixture
+def raise_on_numba_import(monkeypatch):
+    """This fixture will raise if a test attempts to import numba"""
+    real_import = builtins.__import__
+
+    def guarded_import(name, *args, **kwargs):
+        if name == "numba" or name.startswith("numba."):
+            raise ModuleNotFoundError(
+                "This test is marked 'no_numba' but attempted to import it"
+            )
+        return real_import(name, *args, **kwargs)
+
+    monkeypatch.setattr(builtins, "__import__", guarded_import)
+
+
+def pytest_collection_modifyitems(config, items):
+    """Runs after pytest collects the tests. Makes a test marked no_numba fail
+    if it imports numba, and skips a test marked serialization when running on
+    the v2 (HostJIT) backend."""
+    serialization_skip = pytest.mark.skip(
+        reason="serialization not supported on v2 (HostJIT) backend"
+    )
+
+    # Tests marked no_numba must not import numba. We enforce that by attaching
+    # the raise_on_numba_import fixture defined above to each one; it raises if
+    # numba is imported.
+    #
+    # We skip attaching it during a real pytest-run-parallel sweep of more than
+    # one thread: the fixture uses monkeypatch, which pytest-run-parallel
+    # serializes as thread-unsafe, so attaching it to every no_numba test would
+    # make the whole sweep run serially and defeat its purpose. A single-threaded
+    # run has no sweep to protect, so we attach it and keep the check.
+    #
+    # config.getoption gives the --parallel-threads value as an int for the
+    # default but a str when passed on the command line; normalize to str and
+    # count the run as parallel only for an explicit number > 1:
+    #
+    #   pytest ...                          -> 1       single-threaded, attach
+    #   pytest --parallel-threads=1 ...     -> "1"     single-threaded, attach
+    #   pytest --parallel-threads=8 ...     -> "8"     parallel, skip (CI sweep)
+    #   pytest --parallel-threads=auto ...  -> "auto"  single-threaded, attach
+    #
+    # "auto" counts as single-threaded on purpose: it can resolve to one CPU, so
+    # keeping the check is safer than dropping it on a non-parallel run. isdigit
+    # also stops int() from raising on the non-numeric "auto".
+    parallel_threads = str(config.getoption("parallel_threads", 1))
+    running_parallel = parallel_threads.isdigit() and int(parallel_threads) > 1
+    for item in items:
+        # no_numba: add raise_on_numba_import unless we skip it for the sweep
+        if item.get_closest_marker("no_numba") and not running_parallel:
+            if "raise_on_numba_import" not in item.fixturenames:
+                item.fixturenames.append("raise_on_numba_import")
+        # serialization is unsupported on v2 (HostJIT); skip those tests there
+        if USING_V2 and item.get_closest_marker("serialization"):
+            item.add_marker(serialization_skip)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/binary_search/__init__.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/binary_search/__init__.py
new file mode 100644
index 00000000..852281b2
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/binary_search/__init__.py
@@ -0,0 +1,3 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/binary_search/lower_bound_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/binary_search/lower_bound_basic.py
new file mode 100644
index 00000000..03ce684e
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/binary_search/lower_bound_basic.py
@@ -0,0 +1,28 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+# example-begin
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+h_data = np.array([1, 3, 3, 5, 7, 9], dtype=np.int32)
+h_values = np.array([0, 3, 4, 10], dtype=np.int32)
+
+d_data = cp.asarray(h_data)
+d_values = cp.asarray(h_values)
+d_out = cp.empty(len(h_values), dtype=np.uintp)
+
+cuda.compute.lower_bound(
+    d_data=d_data,
+    num_items=len(d_data),
+    d_values=d_values,
+    num_values=len(d_values),
+    d_out=d_out,
+)
+
+expected = np.searchsorted(h_data, h_values, side="left").astype(np.uintp)
+got = cp.asnumpy(d_out)
+
+assert np.array_equal(got, expected)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/binary_search/lower_bound_object.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/binary_search/lower_bound_object.py
new file mode 100644
index 00000000..6f09782a
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/binary_search/lower_bound_object.py
@@ -0,0 +1,30 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+# example-begin
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+h_data = np.array([1, 3, 3, 5, 7, 9], dtype=np.int32)
+h_values = np.array([0, 3, 4, 10], dtype=np.int32)
+
+d_data = cp.asarray(h_data)
+d_values = cp.asarray(h_values)
+d_out = cp.empty(len(h_values), dtype=np.uintp)
+
+searcher = cuda.compute.make_lower_bound(d_data=d_data, d_values=d_values, d_out=d_out)
+searcher(
+    d_data=d_data,
+    num_items=len(d_data),
+    d_values=d_values,
+    num_values=len(d_values),
+    d_out=d_out,
+    comp=None,
+)
+
+expected = np.searchsorted(h_data, h_values, side="left").astype(np.uintp)
+got = cp.asnumpy(d_out)
+
+assert np.array_equal(got, expected)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/binary_search/upper_bound_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/binary_search/upper_bound_basic.py
new file mode 100644
index 00000000..4ec13881
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/binary_search/upper_bound_basic.py
@@ -0,0 +1,28 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+# example-begin
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+h_data = np.array([1, 3, 3, 5, 7, 9], dtype=np.int32)
+h_values = np.array([0, 3, 4, 10], dtype=np.int32)
+
+d_data = cp.asarray(h_data)
+d_values = cp.asarray(h_values)
+d_out = cp.empty(len(h_values), dtype=np.uintp)
+
+cuda.compute.upper_bound(
+    d_data=d_data,
+    num_items=len(d_data),
+    d_values=d_values,
+    num_values=len(d_values),
+    d_out=d_out,
+)
+
+expected = np.searchsorted(h_data, h_values, side="right").astype(np.uintp)
+got = cp.asnumpy(d_out)
+
+assert np.array_equal(got, expected)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/binary_search/upper_bound_object.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/binary_search/upper_bound_object.py
new file mode 100644
index 00000000..208e3714
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/binary_search/upper_bound_object.py
@@ -0,0 +1,30 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+# example-begin
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+h_data = np.array([1, 3, 3, 5, 7, 9], dtype=np.int32)
+h_values = np.array([0, 3, 4, 10], dtype=np.int32)
+
+d_data = cp.asarray(h_data)
+d_values = cp.asarray(h_values)
+d_out = cp.empty(len(h_values), dtype=np.uintp)
+
+searcher = cuda.compute.make_upper_bound(d_data=d_data, d_values=d_values, d_out=d_out)
+searcher(
+    d_data=d_data,
+    num_items=len(d_data),
+    d_values=d_values,
+    num_values=len(d_values),
+    d_out=d_out,
+    comp=None,
+)
+
+expected = np.searchsorted(h_data, h_values, side="right").astype(np.uintp)
+got = cp.asnumpy(d_out)
+
+assert np.array_equal(got, expected)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/free_threading/__init__.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/free_threading/__init__.py
new file mode 100644
index 00000000..8bbe3ce1
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/free_threading/__init__.py
@@ -0,0 +1,3 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/free_threading/direct_api.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/free_threading/direct_api.py
new file mode 100644
index 00000000..33618080
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/free_threading/direct_api.py
@@ -0,0 +1,51 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Run independent direct API calls from multiple Python threads.
+
+The direct algorithm APIs (``cuda.compute.reduce_into`` and friends) are safe
+to call concurrently from any thread: each calling thread transparently
+receives its own per-thread algorithm object, while the expensive compiled
+build result is shared across threads.
+"""
+
+from concurrent.futures import ThreadPoolExecutor
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import OpKind
+
+
+def reduce_values(h_input):
+    dtype = np.int32
+    h_init = np.array([0], dtype=dtype)
+    d_input = cp.asarray(h_input, dtype=dtype)
+    d_output = cp.empty(1, dtype=dtype)
+
+    cuda.compute.reduce_into(
+        d_in=d_input,
+        d_out=d_output,
+        num_items=len(h_input),
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+
+    return int(d_output.get()[0])
+
+
+inputs = [
+    np.array([1, 2, 3, 4], dtype=np.int32),
+    np.array([5, 6, 7, 8], dtype=np.int32),
+]
+
+with ThreadPoolExecutor(max_workers=len(inputs)) as executor:
+    results = list(executor.map(reduce_values, inputs))
+
+expected = [int(np.sum(h_input)) for h_input in inputs]
+assert results == expected
+print(f"Free-threaded direct API results: {results}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/free_threading/object_api.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/free_threading/object_api.py
new file mode 100644
index 00000000..6a25671f
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/free_threading/object_api.py
@@ -0,0 +1,69 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Run independent object-based API calls from multiple Python threads.
+
+This demonstrates the supported object-API pattern for concurrency: each
+thread calls the ``make_*`` factory itself and uses the returned object only
+on that thread. Algorithm objects must be used by one thread at a time, so a
+per-thread factory call (cheap: objects are cached per thread and the compiled
+build result is shared process-wide) is the way to use them concurrently.
+"""
+
+from concurrent.futures import ThreadPoolExecutor
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import OpKind
+
+
+def reduce_values(h_input):
+    dtype = np.int32
+    h_init = np.array([0], dtype=dtype)
+    d_input = cp.asarray(h_input, dtype=dtype)
+    d_output = cp.empty(1, dtype=dtype)
+
+    reducer = cuda.compute.make_reduce_into(
+        d_in=d_input,
+        d_out=d_output,
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+    temp_storage_size = reducer(
+        temp_storage=None,
+        d_in=d_input,
+        d_out=d_output,
+        num_items=len(h_input),
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+    d_temp_storage = cp.empty(temp_storage_size, dtype=np.uint8)
+
+    reducer(
+        temp_storage=d_temp_storage,
+        d_in=d_input,
+        d_out=d_output,
+        num_items=len(h_input),
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+
+    return int(d_output.get()[0])
+
+
+inputs = [
+    np.array([1, 2, 3, 4], dtype=np.int32),
+    np.array([5, 6, 7, 8], dtype=np.int32),
+]
+
+with ThreadPoolExecutor(max_workers=len(inputs)) as executor:
+    results = list(executor.map(reduce_values, inputs))
+
+expected = [int(np.sum(h_input)) for h_input in inputs]
+assert results == expected
+print(f"Free-threaded object API results: {results}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/histogram/__init__.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/histogram/__init__.py
new file mode 100644
index 00000000..2ad64316
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/histogram/__init__.py
@@ -0,0 +1,5 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Parallel histogram algorithms examples package."""
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/histogram/histogram_even_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/histogram/histogram_even_basic.py
new file mode 100644
index 00000000..f3fbcdf0
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/histogram/histogram_even_basic.py
@@ -0,0 +1,44 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use histogram_even to bin a sequence of samples.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+# Prepare the input and output arrays.
+num_samples = 10
+h_samples = np.array(
+    [2.2, 6.1, 7.1, 2.9, 3.5, 0.3, 2.9, 2.1, 6.1, 999.5], dtype="float32"
+)
+d_samples = cp.asarray(h_samples)
+num_levels = 7
+d_histogram = cp.empty(num_levels - 1, dtype="int32")
+lower_level = np.float32(0)
+upper_level = np.float32(12)
+
+# Perform the histogram even.
+cuda.compute.histogram_even(
+    d_samples=d_samples,
+    d_histogram=d_histogram,
+    num_output_levels=num_levels,
+    lower_level=lower_level,
+    upper_level=upper_level,
+    num_samples=num_samples,
+)
+
+# Verify the result.
+h_actual_histogram = cp.asnumpy(d_histogram)
+h_expected_histogram, _ = np.histogram(
+    h_samples, bins=num_levels - 1, range=(lower_level, upper_level)
+)
+h_expected_histogram = h_expected_histogram.astype("int32")
+
+np.testing.assert_array_equal(h_actual_histogram, h_expected_histogram)
+print(f"Histogram even basic result: {h_actual_histogram}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/histogram/histogram_object.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/histogram/histogram_object.py
new file mode 100644
index 00000000..d67425b4
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/histogram/histogram_object.py
@@ -0,0 +1,71 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use histogram object API to bin a sequence of samples.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+# Prepare the input and output arrays.
+h_samples = np.array(
+    [1.5, 2.3, 4.7, 6.2, 7.8, 3.1, 5.5, 8.9, 2.7, 6.4], dtype="float32"
+)
+d_samples = cp.asarray(h_samples)
+
+num_levels = 6
+
+# note that the object API requires passing numpy arrays
+# rather than scalars:
+h_num_output_levels = np.array([num_levels], dtype=np.int32)
+h_lower_level = np.array([0.0], dtype=np.float32)
+h_upper_level = np.array([10.0], dtype=np.float32)
+
+d_histogram = cp.zeros(num_levels - 1, dtype="int32")
+
+# Create the histogram object.
+histogrammer = cuda.compute.make_histogram_even(
+    d_samples=d_samples,
+    d_histogram=d_histogram,
+    h_num_output_levels=h_num_output_levels,
+    h_lower_level=h_lower_level,
+    h_upper_level=h_upper_level,
+    num_samples=len(h_samples),
+)
+
+# Get the temporary storage size.
+temp_storage_size = histogrammer(
+    temp_storage=None,
+    d_samples=d_samples,
+    d_histogram=d_histogram,
+    h_num_output_levels=h_num_output_levels,
+    h_lower_level=h_lower_level,
+    h_upper_level=h_upper_level,
+    num_samples=len(h_samples),
+)
+
+# Allocate the temporary storage.
+d_temp_storage = cp.empty(temp_storage_size, dtype=np.uint8)
+
+# Perform the histogram.
+histogrammer(
+    temp_storage=d_temp_storage,
+    d_samples=d_samples,
+    d_histogram=d_histogram,
+    h_num_output_levels=h_num_output_levels,
+    h_lower_level=h_lower_level,
+    h_upper_level=h_upper_level,
+    num_samples=len(h_samples),
+)
+
+# Verify the result.
+h_result = cp.asnumpy(d_histogram)
+expected_histogram = np.array([1, 3, 2, 3, 1], dtype="int32")
+
+np.testing.assert_array_equal(h_result, expected_histogram)
+print("Histogram object example completed successfully")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/__init__.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/__init__.py
new file mode 100644
index 00000000..b3d87885
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/__init__.py
@@ -0,0 +1,5 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Iterator examples for reduction operations."""
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/cache_modified_iterator_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/cache_modified_iterator_basic.py
new file mode 100644
index 00000000..2da3edc1
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/cache_modified_iterator_basic.py
@@ -0,0 +1,40 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use cache_modified_iterator.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    CacheModifiedInputIterator,
+    OpKind,
+)
+
+# Prepare the input array.
+h_input = np.array([1, 2, 3, 4, 5], dtype=np.int32)
+d_input = cp.asarray(h_input)
+
+# Create the cache modified iterator.
+cache_it = CacheModifiedInputIterator(d_input, "stream")
+
+# Prepare the initial value for the reduction.
+h_init = np.array([0], dtype=np.int32)
+
+# Prepare the output array.
+d_output = cp.empty(1, dtype=np.int32)
+
+# Perform the reduction.
+cuda.compute.reduce_into(
+    d_in=cache_it, d_out=d_output, num_items=len(d_input), op=OpKind.PLUS, h_init=h_init
+)
+
+# Verify the result.
+expected_output = sum(h_input)  # 1 + 2 + 3 + 4 + 5 = 15
+assert (d_output == expected_output).all()
+print(f"Cache modified iterator result: {d_output[0]} (expected: {expected_output})")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/constant_iterator_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/constant_iterator_basic.py
new file mode 100644
index 00000000..a5ea1c77
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/constant_iterator_basic.py
@@ -0,0 +1,40 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use constant_iterator.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    ConstantIterator,
+    OpKind,
+)
+
+# Prepare the input and output arrays.
+constant_value = 42
+num_items = 5
+
+# Create the constant iterator.
+constant_it = ConstantIterator(np.int32(constant_value))
+
+# Prepare the initial value for the reduction.
+h_init = np.array([0], dtype=np.int32)
+
+# Prepare the output array.
+d_output = cp.empty(1, dtype=np.int32)
+
+# Perform the reduction.
+cuda.compute.reduce_into(
+    d_in=constant_it, d_out=d_output, num_items=num_items, op=OpKind.PLUS, h_init=h_init
+)
+
+# Verify the result.
+expected_output = constant_value * num_items
+assert (d_output == expected_output).all()
+print(f"Constant iterator result: {d_output[0]} (expected: {expected_output})")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/counting_iterator_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/counting_iterator_basic.py
new file mode 100644
index 00000000..0646e5a7
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/counting_iterator_basic.py
@@ -0,0 +1,44 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use counting_iterator.
+"""
+
+import functools
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    CountingIterator,
+    OpKind,
+)
+
+# Prepare the input and output arrays.
+first_item = 1
+num_items = 100
+
+# Create the counting iterator.
+first_it = CountingIterator(np.int32(first_item))
+
+# Prepare the initial value for the reduction.
+h_init = np.array([0], dtype=np.int32)
+
+# Prepare the output array.
+d_output = cp.empty(1, dtype=np.int32)
+
+# Perform the reduction.
+cuda.compute.reduce_into(
+    d_in=first_it, d_out=d_output, num_items=num_items, op=OpKind.PLUS, h_init=h_init
+)
+
+# Verify the result.
+expected_output = functools.reduce(
+    lambda a, b: a + b, range(first_item, first_item + num_items)
+)
+assert (d_output == expected_output).all()
+print(f"Counting iterator result: {d_output[0]} (expected: {expected_output})")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/discard_iterator_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/discard_iterator_basic.py
new file mode 100644
index 00000000..3a546cac
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/discard_iterator_basic.py
@@ -0,0 +1,47 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use DiscardIterator.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    DiscardIterator,
+    OpKind,
+)
+
+# Prepare the input and output arrays.
+h_in_keys = np.array([1, 1, 2, 3, 3, 7, 8, 8], dtype="int32")
+d_in_keys = cp.asarray(h_in_keys)
+d_out_keys = cp.empty_like(d_in_keys)
+d_out_num_selected = cp.empty(1, np.int32)
+
+# Prepare the discard iterator for values.
+d_in_values = DiscardIterator()
+d_out_values = DiscardIterator()
+
+# Perform the unique by key operation.
+cuda.compute.unique_by_key(
+    d_in_keys=d_in_keys,
+    d_in_items=d_in_values,
+    d_out_keys=d_out_keys,
+    d_out_items=d_out_values,
+    d_out_num_selected=d_out_num_selected,
+    op=OpKind.EQUAL_TO,
+    num_items=d_in_keys.size,
+)
+
+# Verify the result.
+num_selected = cp.asnumpy(d_out_num_selected)[0]
+h_out_keys = cp.asnumpy(d_out_keys)[:num_selected]
+
+expected_keys = np.array([1, 2, 3, 7, 8])
+
+assert np.array_equal(h_out_keys, expected_keys)
+print(f"Discard iterator result - keys: {h_out_keys}, count: {num_selected}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/permutation_iterator_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/permutation_iterator_basic.py
new file mode 100644
index 00000000..6cd0bbc9
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/permutation_iterator_basic.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Demonstrate reduction with permutation iterator as input.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    OpKind,
+    PermutationIterator,
+)
+
+# Create a permutation iterator which selects values at the given indices:
+d_values = cp.asarray([10, 20, 30, 40, 50], dtype=np.int32)
+d_indices = cp.asarray([2, 0, 4, 1], dtype=np.int32)  # permutation indices
+perm_it = PermutationIterator(d_values, d_indices)
+
+# Prepare the initial value and output for the reduction.
+h_init = np.array([0], dtype=np.int32)
+d_output = cp.empty(1, dtype=np.int32)
+
+# Perform the reduction on the permuted values.
+num_items = len(d_indices)
+cuda.compute.reduce_into(
+    d_in=perm_it, d_out=d_output, num_items=num_items, op=OpKind.PLUS, h_init=h_init
+)
+
+# Verify the result:
+expected_output = d_values[d_indices].sum()
+assert d_output[0] == expected_output
+print(f"Permutation iterator result: {d_output[0]} (expected: {expected_output})")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/permutation_iterator_composed.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/permutation_iterator_composed.py
new file mode 100644
index 00000000..9da92afd
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/permutation_iterator_composed.py
@@ -0,0 +1,54 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Demonstrate composed permutation iterator with transform iterator.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    CountingIterator,
+    OpKind,
+    PermutationIterator,
+    TransformIterator,
+)
+
+
+def square_op(x):
+    return x * x
+
+
+# Create a CountingIterator that generates: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
+counting_it = CountingIterator(np.int32(0))
+
+# Wrap it in a TransformIterator to square the values: 0, 1, 4, 9, 16, 25, 36, 49, 64, 81
+transform_it = TransformIterator(counting_it, square_op)
+
+# Create indices to permute the squared values
+d_indices = cp.asarray([3, 1, 5, 2], dtype=np.int32)
+
+# Create permutation iterator that accesses the squared counting iterator
+# This will access: squares[3]=9, squares[1]=1, squares[5]=25, squares[2]=4
+perm_it = PermutationIterator(transform_it, d_indices)
+
+# Prepare the initial value and output for the reduction
+h_init = np.array([0], dtype=np.int32)
+d_output = cp.empty(1, dtype=np.int32)
+
+# Perform the reduction on the composed iterator
+num_items = len(d_indices)
+cuda.compute.reduce_into(
+    d_in=perm_it, d_out=d_output, num_items=num_items, op=OpKind.PLUS, h_init=h_init
+)
+
+# Verify the result: 9 + 1 + 25 + 4 = 39
+expected_output = 9 + 1 + 25 + 4
+assert d_output[0] == expected_output
+print(
+    f"Composed permutation iterator result: {d_output[0]} (expected: {expected_output})"
+)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/permutation_iterator_output.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/permutation_iterator_output.py
new file mode 100644
index 00000000..65b555a3
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/permutation_iterator_output.py
@@ -0,0 +1,49 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Demonstrate transform with permutation iterator as output (scatter operation).
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    CountingIterator,
+    PermutationIterator,
+)
+
+
+def square_op(x):
+    return x * x
+
+
+# Prepare the output values array and indices.
+d_values = cp.zeros(10, dtype=np.int32)  # Output array
+d_indices = cp.asarray([9, 3, 7, 1, 5], dtype=np.int32)  # Scatter indices
+
+# Create input iterator that generates: 0, 1, 2, 3, 4
+input_it = CountingIterator(np.int32(0))
+
+# Create permutation iterator for output (scatter).
+# This will write to: values[9], values[3], values[7], values[1], values[5]
+perm_it = PermutationIterator(d_values, d_indices)
+
+# Perform the transform, scattering squared values to permuted locations.
+num_items = len(d_indices)
+cuda.compute.unary_transform(
+    d_in=input_it, d_out=perm_it, op=square_op, num_items=num_items
+)
+
+# Verify the result: values[9]=0, values[3]=1, values[7]=4, values[1]=9, values[5]=16
+# Other positions should remain 0
+expected = np.zeros(10, dtype=np.int32)
+for i, idx in enumerate(d_indices.get()):
+    expected[idx] = i * i
+
+assert np.array_equal(d_values.get(), expected)
+print(f"Permutation output iterator result: {d_values.get()}")
+print(f"Expected: {expected}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/reverse_input_iterator.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/reverse_input_iterator.py
new file mode 100644
index 00000000..5bb54718
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/reverse_input_iterator.py
@@ -0,0 +1,46 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use reverse_input_iterator.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    OpKind,
+    ReverseIterator,
+)
+
+# Prepare the input and output arrays.
+h_input = np.array([1, 2, 3, 4, 5], dtype=np.int32)
+d_input = cp.asarray(h_input)
+
+# Create the reverse input iterator.
+reverse_it = ReverseIterator(d_input)
+d_output = cp.empty(len(d_input), dtype=np.int32)
+
+# Prepare the initial value for the reduction.
+h_init = np.array(0, dtype=np.int32)
+
+# Perform the reduction.
+cuda.compute.inclusive_scan(
+    d_in=reverse_it,
+    d_out=d_output,
+    op=OpKind.PLUS,
+    init_value=h_init,
+    num_items=len(d_input),
+)
+
+# Verify the result.
+expected_output = np.array([5, 9, 12, 14, 15], dtype=np.int32)
+result = d_output.get()
+
+np.testing.assert_array_equal(result, expected_output)
+print(f"Original input: {h_input}")
+print(f"Reverse scan result: {result}")
+print(f"Expected result: {expected_output}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/reverse_output_iterator.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/reverse_output_iterator.py
new file mode 100644
index 00000000..3fb30599
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/reverse_output_iterator.py
@@ -0,0 +1,46 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use reverse_output_iterator.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    OpKind,
+    ReverseIterator,
+)
+
+# Prepare the input and output arrays.
+h_input = np.array([1, 2, 3, 4, 5], dtype=np.int32)
+d_input = cp.asarray(h_input)
+
+# Prepare the output array.
+d_output = cp.empty(len(d_input), dtype=np.int32)
+h_init = np.array(0, dtype=np.int32)
+
+# Create the reverse output iterator.
+reverse_out_it = ReverseIterator(d_output)
+
+# Perform the reduction.
+cuda.compute.inclusive_scan(
+    d_in=d_input,
+    d_out=reverse_out_it,
+    op=OpKind.PLUS,
+    init_value=h_init,
+    num_items=len(d_input),
+)
+
+# Verify the result.
+expected_output = np.array([15, 10, 6, 3, 1], dtype=np.int32)
+result = d_output.get()
+
+np.testing.assert_array_equal(result, expected_output)
+print(f"Original input: {h_input}")
+print(f"Reverse output result: {result}")
+print(f"Expected result: {expected_output}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/shuffle_iterator_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/shuffle_iterator_basic.py
new file mode 100644
index 00000000..e0bb6a0b
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/shuffle_iterator_basic.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Using ShuffleIterator to obtain a random permutation of an array
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import PermutationIterator, ShuffleIterator
+
+# Input data and output array:
+d_input = cp.asarray([10, 20, 30, 40, 50, 60, 70, 80, 100], dtype=np.int32)
+d_output = cp.empty_like(d_input)
+num_items = len(d_input)
+
+# Create a shuffle iterator that produces a random permutation of [0, num_items)
+shuffle_it = ShuffleIterator(num_items, seed=42)
+
+# Use PermutationIterator to permute the data according to the random indices
+perm_it = PermutationIterator(d_input, shuffle_it)
+
+# Use unary_transform to write values into d_output
+cuda.compute.unary_transform(
+    d_in=perm_it, d_out=d_output, op=lambda x: x, num_items=num_items
+)
+
+# Verify it is a valid permutation of the input data:
+cp.testing.assert_array_equal(cp.sort(d_output), d_input)
+
+# Print the values
+print(f"Input data: {d_input}")
+print(f"Shuffled data: {d_output}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/transform_iterator_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/transform_iterator_basic.py
new file mode 100644
index 00000000..5787f00b
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/transform_iterator_basic.py
@@ -0,0 +1,36 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Using ``reduce_into`` with a ``TransformIterator`` to compute the
+sum of squares of a sequence of numbers.
+"""
+
+import cupy as cp
+import numpy as np
+
+from cuda.compute import (
+    OpKind,
+    TransformIterator,
+    reduce_into,
+)
+
+# Prepare the input and output arrays.
+d_input = cp.arange(10, dtype=np.int32)
+d_output = cp.empty(1, dtype=np.int32)
+h_init = np.array([0], dtype=np.int32)  # Initial value for the reduction
+
+# Create a TransformIterator to (lazily) apply the square
+it_input = TransformIterator(d_input, lambda a: a**2)
+
+# Use `reduce_into` to compute the sum of the squares of the input.
+reduce_into(
+    d_in=it_input, d_out=d_output, num_items=len(d_input), op=OpKind.PLUS, h_init=h_init
+)
+
+# Verify the result.
+expected_output = cp.sum(d_input**2).get()
+assert d_output[0] == expected_output
+print(f"Transform iterator result: {d_output[0]} (expected: {expected_output})")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/transform_iterator_lambda.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/transform_iterator_lambda.py
new file mode 100644
index 00000000..178fda79
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/transform_iterator_lambda.py
@@ -0,0 +1,51 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Demonstrate TransformIterator with a lambda function.
+
+This example shows how to use a lambda function with TransformIterator
+to apply a transformation on-the-fly during reduction, without needing
+to define a named function.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    CountingIterator,
+    OpKind,
+    TransformIterator,
+)
+
+# Prepare the parameters.
+first_item = 1
+num_items = 10
+
+# Create a TransformIterator that squares each value from a CountingIterator
+# using a lambda function.
+transform_it = TransformIterator(
+    CountingIterator(np.int32(first_item)), lambda x: x * x
+)
+
+h_init = np.array([0], dtype=np.int32)
+d_output = cp.empty(1, dtype=np.int32)
+
+# Perform the reduction: sum of squares from 1 to 10.
+cuda.compute.reduce_into(
+    d_in=transform_it,
+    d_out=d_output,
+    num_items=num_items,
+    op=OpKind.PLUS,
+    h_init=h_init,
+)
+
+# Verify the result: 1^2 + 2^2 + ... + 10^2 = 385
+expected_output = sum(x * x for x in range(first_item, first_item + num_items))
+assert d_output[0] == expected_output
+print(
+    f"Sum of squares with lambda TransformIterator: {d_output[0]} (expected: {expected_output})"
+)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/transform_output_iterator.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/transform_output_iterator.py
new file mode 100644
index 00000000..ec91f8cb
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/transform_output_iterator.py
@@ -0,0 +1,44 @@
+# Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+TransformOutputIterator example demonstrating reduction with transform output iterator.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    OpKind,
+    TransformOutputIterator,
+)
+
+# Create input and output arrays
+d_input = cp.array([1, 2, 3, 4, 5.0], dtype=np.float32)
+d_output = cp.empty(shape=1, dtype=np.float32)
+
+
+# Define the transform operation to be applied
+# to the result of the sum reduction.
+# TransformOutputIterator requires type annotations:
+def sqrt(x: np.float32) -> np.float32:
+    return x**0.5
+
+
+# Create transform output iterator
+d_out_it = TransformOutputIterator(d_output, sqrt)
+
+
+# Apply a sum reduction into the transform output iterator
+cuda.compute.reduce_into(
+    d_in=d_input,
+    d_out=d_out_it,
+    num_items=len(d_input),
+    op=OpKind.PLUS,
+    h_init=np.asarray([0], dtype=np.float32),
+)
+
+assert cp.allclose(d_output, cp.sqrt(cp.sum(d_input)), atol=1e-6)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/zip_iterator_counting.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/zip_iterator_counting.py
new file mode 100644
index 00000000..7f726b7a
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/zip_iterator_counting.py
@@ -0,0 +1,58 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use zip_iterator with counting iterator to
+find the index with maximum value in an array.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    CountingIterator,
+    ZipIterator,
+)
+
+
+def max_by_value(p1, p2):
+    """Reduction operation that returns the pair with the larger value."""
+    return p1 if p1[1] > p2[1] else p2
+
+
+# Create the counting iterator.
+counting_it = CountingIterator(np.int32(0))
+
+# Prepare the input array.
+arr = cp.asarray([0, 1, 2, 4, 7, 3, 5, 6], dtype=np.int32)
+
+# Create the zip iterator.
+zip_it = ZipIterator(counting_it, arr)
+
+num_items = 8
+
+# Note: initial value passed as a numpy struct
+dtype = np.dtype([("index", np.int32), ("value", np.int32)], align=True)
+h_init = np.asarray([(-1, -1)], dtype=dtype)
+
+d_output = cp.empty(1, dtype=dtype)
+
+# Perform the reduction.
+cuda.compute.reduce_into(
+    d_in=zip_it, d_out=d_output, num_items=num_items, op=max_by_value, h_init=h_init
+)
+
+result = d_output.get()[0]
+expected_index = 4
+expected_value = 7
+
+assert result["index"] == expected_index
+assert result["value"] == expected_value
+
+print(
+    f"Zip iterator with counting result: index={result['index']} "
+    f"(expected: {expected_index}), value={result['value']} (expected: {expected_value})"
+)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/zip_iterator_elementwise.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/zip_iterator_elementwise.py
new file mode 100644
index 00000000..d21b7530
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/zip_iterator_elementwise.py
@@ -0,0 +1,50 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use zip_iterator to perform elementwise sum of two arrays.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    ZipIterator,
+)
+
+# Prepare the input arrays.
+d_input1 = cp.array([1, 2, 3, 4, 5], dtype=np.int32)
+d_input2 = cp.array([10, 20, 30, 40, 50], dtype=np.int32)
+
+# Create the zip iterator.
+zip_it = ZipIterator(d_input1, d_input2)
+
+# Prepare the output array.
+num_items = len(d_input1)
+d_output = cp.empty(num_items, dtype=np.int32)
+
+
+def sum_paired_values(pair):
+    """Extract values from the zip iterator pair and sum them."""
+    return pair[0] + pair[1]
+
+
+# Perform the unary transform.
+cuda.compute.unary_transform(
+    d_in=zip_it, d_out=d_output, op=sum_paired_values, num_items=num_items
+)
+
+# Calculate the expected results.
+expected = d_input1.get() + d_input2.get()
+result = d_output.get()
+
+# Verify the result.
+np.testing.assert_allclose(result, expected)
+
+print(f"Input array 1: {d_input1.get()}")
+print(f"Input array 2: {d_input2.get()}")
+print(f"Elementwise sum result: {result}")
+print(f"Expected result: {expected}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/zip_iterator_reduction.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/zip_iterator_reduction.py
new file mode 100644
index 00000000..33aaae93
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/iterator/zip_iterator_reduction.py
@@ -0,0 +1,61 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use zip_iterator to simultaneously perform a reduction
+operation on two arrays.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    ZipIterator,
+    gpu_struct,
+)
+
+
+@gpu_struct
+class Pair:
+    first: np.int32
+    second: np.float32
+
+
+def sum_pairs(p1, p2):
+    """Reduction operation that adds corresponding elements of pairs."""
+    return Pair(p1[0] + p2[0], p1[1] + p2[1])
+
+
+# Prepare the input arrays.
+d_input1 = cp.array([1, 2, 3, 4, 5], dtype=np.int32)
+d_input2 = cp.array([1.0, 2.0, 3.0, 4.0, 5.0], dtype=np.float32)
+
+# Create the zip iterator.
+zip_it = ZipIterator(d_input1, d_input2)
+
+# Prepare the initial value for the reduction.
+h_init = Pair(0, 0.0)
+
+# Prepare the output array.
+d_output = cp.empty(1, dtype=Pair.dtype)
+
+# Perform the reduction.
+cuda.compute.reduce_into(
+    d_in=zip_it, d_out=d_output, num_items=len(d_input1), op=sum_pairs, h_init=h_init
+)
+
+# Calculate the expected results.
+expected_first = sum(d_input1.get())
+expected_second = sum(d_input2.get())
+
+result = d_output.get()[0]
+assert result["first"] == expected_first
+assert result["second"] == expected_second
+
+print(
+    f"Zip iterator result: first={result['first']} (expected: {expected_first}), "
+    f"second={result['second']} (expected: {expected_second})"
+)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/partition/__init__.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/partition/__init__.py
new file mode 100644
index 00000000..5bcd82b7
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/partition/__init__.py
@@ -0,0 +1,5 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Parallel partitioning algorithms examples package."""
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/partition/three_way_partition_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/partition/three_way_partition_basic.py
new file mode 100644
index 00000000..7099815f
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/partition/three_way_partition_basic.py
@@ -0,0 +1,65 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use three_way_partition to partition a sequence of integers into three parts.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+# Prepare the input and output arrays.
+dtype = np.int32
+h_input = np.array([0, 2, 9, 1, 5, 6, 7, -3, 17, 10], dtype=dtype)
+d_input = cp.asarray(h_input)
+d_first_part = cp.empty_like(d_input)
+d_second_part = cp.empty_like(d_input)
+d_unselected = cp.empty_like(d_input)
+d_num_selected = cp.empty(2, dtype=np.int64)
+
+
+def less_than_op(x):
+    return x < 8 and x >= 0
+
+
+def greater_than_equal_op(x):
+    return x >= 8
+
+
+# Perform the three_way_partition.
+cuda.compute.three_way_partition(
+    d_in=d_input,
+    d_first_part_out=d_first_part,
+    d_second_part_out=d_second_part,
+    d_unselected_out=d_unselected,
+    d_num_selected_out=d_num_selected,
+    select_first_part_op=less_than_op,
+    select_second_part_op=greater_than_equal_op,
+    num_items=len(h_input),
+)
+
+# Verify the result.
+expected_first_part = np.array([0, 2, 1, 5, 6, 7], dtype=dtype)
+expected_second_part = np.array([9, 17, 10], dtype=dtype)
+expected_unselected = np.array([-3], dtype=dtype)
+expected_num_selected = np.array([6, 3], dtype=np.int64)
+
+actual_num_selected = d_num_selected.get()
+num_selected_first_part = int(actual_num_selected[0])
+num_selected_second_part = int(actual_num_selected[1])
+actual_first_part = d_first_part.get()[:num_selected_first_part]
+actual_second_part = d_second_part.get()[:num_selected_second_part]
+actual_unselected = d_unselected.get()[
+    : d_input.size - num_selected_first_part - num_selected_second_part
+]
+
+np.testing.assert_array_equal(actual_first_part, expected_first_part)
+np.testing.assert_array_equal(actual_second_part, expected_second_part)
+np.testing.assert_array_equal(actual_unselected, expected_unselected)
+np.testing.assert_array_equal(actual_num_selected, expected_num_selected)
+
+print("Three way partition basic example completed successfully")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/partition/three_way_partition_object.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/partition/three_way_partition_object.py
new file mode 100644
index 00000000..9b9191ad
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/partition/three_way_partition_object.py
@@ -0,0 +1,91 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use three_way_partition with the object API.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+# Prepare the input and output arrays.
+dtype = np.int32
+h_input = np.array([0, 2, 9, 1, 5, 6, 7, -3, 17, 10], dtype=dtype)
+d_input = cp.asarray(h_input)
+d_first_part = cp.empty_like(d_input)
+d_second_part = cp.empty_like(d_input)
+d_unselected = cp.empty_like(d_input)
+d_num_selected = cp.empty(2, dtype=np.int64)
+
+
+def less_than_op(x):
+    return x < 8 and x >= 0
+
+
+def greater_than_equal_op(x):
+    return x >= 8
+
+
+# Create the three_way_partition object.
+partitioner = cuda.compute.make_three_way_partition(
+    d_in=d_input,
+    d_first_part_out=d_first_part,
+    d_second_part_out=d_second_part,
+    d_unselected_out=d_unselected,
+    d_num_selected_out=d_num_selected,
+    select_first_part_op=less_than_op,
+    select_second_part_op=greater_than_equal_op,
+)
+
+# Get the temporary storage size.
+temp_storage_size = partitioner(
+    temp_storage=None,
+    d_in=d_input,
+    d_first_part_out=d_first_part,
+    d_second_part_out=d_second_part,
+    d_unselected_out=d_unselected,
+    d_num_selected_out=d_num_selected,
+    select_first_part_op=less_than_op,
+    select_second_part_op=greater_than_equal_op,
+    num_items=len(h_input),
+)
+d_temp_storage = cp.empty(temp_storage_size, dtype=np.uint8)
+
+# Perform the three_way_partition.
+partitioner(
+    temp_storage=d_temp_storage,
+    d_in=d_input,
+    d_first_part_out=d_first_part,
+    d_second_part_out=d_second_part,
+    d_unselected_out=d_unselected,
+    d_num_selected_out=d_num_selected,
+    select_first_part_op=less_than_op,
+    select_second_part_op=greater_than_equal_op,
+    num_items=len(h_input),
+)
+
+# Verify the result.
+expected_first_part = np.array([0, 2, 1, 5, 6, 7], dtype=dtype)
+expected_second_part = np.array([9, 17, 10], dtype=dtype)
+expected_unselected = np.array([-3], dtype=dtype)
+expected_num_selected = np.array([6, 3], dtype=np.int64)
+
+actual_num_selected = d_num_selected.get()
+num_selected_first_part = int(actual_num_selected[0])
+num_selected_second_part = int(actual_num_selected[1])
+actual_first_part = d_first_part.get()[:num_selected_first_part]
+actual_second_part = d_second_part.get()[:num_selected_second_part]
+actual_unselected = d_unselected.get()[
+    : d_input.size - num_selected_first_part - num_selected_second_part
+]
+
+np.testing.assert_array_equal(actual_first_part, expected_first_part)
+np.testing.assert_array_equal(actual_second_part, expected_second_part)
+np.testing.assert_array_equal(actual_unselected, expected_unselected)
+np.testing.assert_array_equal(actual_num_selected, expected_num_selected)
+
+print("Three way partition object example completed successfully")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/raw_op/__init__.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/raw_op/__init__.py
new file mode 100644
index 00000000..8bbe3ce1
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/raw_op/__init__.py
@@ -0,0 +1,3 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/raw_op/cpp_stateful.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/raw_op/cpp_stateful.py
new file mode 100644
index 00000000..970498e3
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/raw_op/cpp_stateful.py
@@ -0,0 +1,125 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Create a stateful custom operator using RawOp.
+
+This example demonstrates how to create a stateful operator that maintains
+runtime state (in this case, a counter on the device). The operator selects
+even numbers and atomically increments a counter for each selected item.
+"""
+
+import struct
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute.op import RawOp
+from cuda.core import Device, Program, ProgramOptions
+
+
+def get_arch():
+    """Get the SM architecture string for the current device."""
+    device = Device()
+    device.set_current()
+    cc_major, cc_minor = device.compute_capability
+    return f"sm_{cc_major}{cc_minor}"
+
+
+def compile_cpp_to_ltoir(source: str, arch: str) -> bytes:
+    """Compile C++ source to LTOIR using cuda.core."""
+    opts = ProgramOptions(
+        arch=arch,
+        relocatable_device_code=True,
+        link_time_optimization=True,
+    )
+    prog = Program(source, "c++", options=opts)
+    return prog.compile("ltoir").code
+
+
+# Create a device counter initialized to 0
+d_counter = cp.zeros(1, dtype=np.int32)
+
+# Pack the counter pointer as state bytes
+counter_ptr = d_counter.__cuda_array_interface__["data"][0]
+state_bytes = struct.pack("P", counter_ptr)
+state_alignment = np.dtype(np.intp).alignment
+
+# Define a C++ stateful select operator
+# The operator selects even numbers and counts them using atomic operations
+cpp_source = """
+extern "C" __device__ void select_even_with_count(void* state, void* input, void* result) {
+    // Extract counter pointer from state
+    int* counter = *reinterpret_cast<int**>(state);
+
+    // Get input value
+    int value = *static_cast<int*>(input);
+
+    // Check if even
+    bool is_even = (value % 2 == 0);
+
+    // If selected, atomically increment the counter
+    if (is_even) {
+        atomicAdd(counter, 1);
+    }
+
+    // Store result as bool (uint8)
+    *static_cast<unsigned char*>(result) = is_even ? 1 : 0;
+}
+"""
+
+# Compile C++ to LTOIR
+arch = get_arch()
+ltoir_bytes = compile_cpp_to_ltoir(cpp_source, arch)
+
+# Create a stateful RawOp with the state bytes
+select_op = RawOp(
+    ltoir=ltoir_bytes,
+    name="select_even_with_count",
+    state=state_bytes,
+    state_alignment=state_alignment,
+)
+
+# Prepare test data: numbers 0 to 19
+num_items = 20
+h_input = np.arange(num_items, dtype=np.int32)
+d_input = cp.array(h_input)
+
+# Allocate output arrays
+d_output = cp.empty(num_items, dtype=np.int32)
+d_num_selected = cp.empty(1, dtype=np.int32)
+
+# Run select with the stateful operator
+cuda.compute.select(
+    d_in=d_input,
+    d_out=d_output,
+    d_num_selected_out=d_num_selected,
+    cond=select_op,
+    num_items=num_items,
+)
+
+# Get results
+num_selected = d_num_selected.get()[0]
+counter_value = d_counter.get()[0]
+
+# Verify: should have selected 10 even numbers (0, 2, 4, ..., 18)
+expected_count = 10
+assert num_selected == expected_count, (
+    f"Expected {expected_count} selected, got {num_selected}"
+)
+assert counter_value == expected_count, (
+    f"Expected counter={expected_count}, got {counter_value}"
+)
+
+# Verify the selected values are correct
+selected_values = d_output.get()[:num_selected]
+expected_selected = np.arange(0, 20, 2, dtype=np.int32)
+assert np.array_equal(selected_values, expected_selected), "Selected values don't match"
+
+print(f"Selected {num_selected} even numbers")
+print(f"Counter value: {counter_value}")
+print(f"Selected values: {selected_values}")
+print("RawOp stateful example completed successfully!")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/raw_op/cpp_stateless.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/raw_op/cpp_stateless.py
new file mode 100644
index 00000000..2db4e09e
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/raw_op/cpp_stateless.py
@@ -0,0 +1,71 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Create a custom C++ operator from LTOIR bytecode using RawOp.
+
+This example demonstrates how to compile C++ device code to LTOIR and use it
+as a custom operator.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute.op import RawOp
+from cuda.core import Device, Program, ProgramOptions
+
+
+def get_arch():
+    """Get the SM architecture string for the current device."""
+    device = Device()
+    device.set_current()
+    cc_major, cc_minor = device.compute_capability
+    return f"sm_{cc_major}{cc_minor}"
+
+
+def compile_cpp_to_ltoir(source: str, arch: str) -> bytes:
+    """Compile C++ source to LTOIR using cuda.core."""
+    opts = ProgramOptions(
+        arch=arch,
+        relocatable_device_code=True,
+        link_time_optimization=True,
+    )
+    prog = Program(source, "c++", options=opts)
+    return prog.compile("ltoir").code
+
+
+# Define a C++ custom multiply operator
+cpp_source = """
+extern "C" __device__ void multiply_op(void* a, void* b, void* result) {
+    *static_cast<int*>(result) = *static_cast<int*>(a) * *static_cast<int*>(b);
+}
+"""
+
+# Compile C++ to LTOIR
+arch = get_arch()
+ltoir_bytes = compile_cpp_to_ltoir(cpp_source, arch)
+
+# Create a RawOp from the LTOIR bytecode
+multiply_op = RawOp(ltoir=ltoir_bytes, name="multiply_op")
+
+# Prepare test data
+h_input = np.array([1, 2, 3, 4, 5], dtype=np.int32)
+d_input = cp.array(h_input)
+d_output = cp.empty(1, dtype=np.int32)
+h_init = np.array(1, dtype=np.int32)
+
+# Use the custom operator with reduce_into
+cuda.compute.reduce_into(
+    d_in=d_input, d_out=d_output, num_items=len(d_input), op=multiply_op, h_init=h_init
+)
+
+# Verify the result
+result = d_output.get()[0]
+expected = np.prod(h_input)  # 1 * 2 * 3 * 4 * 5 = 120
+assert result == expected, f"Expected {expected}, got {result}"
+
+print(f"Custom multiply reduction result: {result}")
+print("RawOp stateless example completed successfully!")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/raw_op/llvm_stateless.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/raw_op/llvm_stateless.py
new file mode 100644
index 00000000..2df59f56
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/raw_op/llvm_stateless.py
@@ -0,0 +1,85 @@
+# Copyright (c) 2026 NVIDIA CORPORATION.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ruff: noqa: E402 — the v1-skip block below intentionally precedes the
+# example's imports so the imports stay grouped at the start of the example
+# body (after `# example-begin`).
+
+# Skip cleanly on v1 — the LLVM-bitcode code-path (DeviceCode(kind="llvm_ir"))
+# requires cccl.c.parallel built with CCCL_PYTHON_USE_V2=ON. On a v1 wheel the
+# binding silently treats it as LTO-IR and nvJitLink rejects the bitcode bytes
+# as malformed LTO-IR.
+import sys
+
+try:
+    from cuda.compute._build_info import USING_V2
+except ImportError:
+    USING_V2 = False
+
+if not USING_V2:
+    print("llvm_stateless requires cccl.c.parallel v2; skipping.")
+    sys.exit(0)
+
+# example-begin
+"""
+Create a custom operator from LLVM bitcode using RawOp.
+
+This example demonstrates how to supply pre-compiled LLVM bitcode to RawOp,
+which is the preferred path for cccl.parallel v2 because the bitcode is
+linked into the CUB module at the LLVM IR level and the optimizer inlines
+the operator through kernel inner loops.
+"""
+
+import cupy as cp
+import llvmlite.binding as llvm
+import numpy as np
+
+import cuda.compute
+from cuda.compute._device_code import DeviceCode
+from cuda.compute.op import RawOp
+
+# Hand-written LLVM IR for an extern "C" multiply operator with the
+# void(void*, void*, void*) ABI RawOp expects. llvmlite parses the text and
+# serializes it to LLVM bitcode (the binary form, starting with magic "BC")
+# that the v2 backend's LLVM linker accepts.
+llvm_ir = """
+target triple = "nvptx64-nvidia-cuda"
+
+define void @multiply_op(ptr %a, ptr %b, ptr %result) {
+entry:
+  %x = load i32, ptr %a, align 4
+  %y = load i32, ptr %b, align 4
+  %r = mul i32 %x, %y
+  store i32 %r, ptr %result, align 4
+  ret void
+}
+"""
+
+mod = llvm.parse_assembly(llvm_ir)
+mod.verify()
+bitcode = bytes(mod.as_bitcode())
+
+# Wrap the bitcode in DeviceCode so RawOp knows to treat it as LLVM bitcode
+# rather than the default LTO-IR. (Raw `bytes` is accepted too and treated
+# as LTO-IR — the legacy form.)
+multiply_op = RawOp(
+    ltoir=DeviceCode(op_bytes=bitcode, kind="llvm_ir"),
+    name="multiply_op",
+)
+
+h_input = np.array([1, 2, 3, 4, 5], dtype=np.int32)
+d_input = cp.array(h_input)
+d_output = cp.empty(1, dtype=np.int32)
+h_init = np.array(1, dtype=np.int32)
+
+cuda.compute.reduce_into(
+    d_in=d_input, d_out=d_output, num_items=len(d_input), op=multiply_op, h_init=h_init
+)
+
+result = d_output.get()[0]
+expected = np.prod(h_input)  # 1 * 2 * 3 * 4 * 5 = 120
+assert result == expected, f"Expected {expected}, got {result}"
+
+print(f"Custom multiply reduction result: {result}")
+print("RawOp LLVM-IR example completed successfully!")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/__init__.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/__init__.py
new file mode 100644
index 00000000..54937b4e
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/__init__.py
@@ -0,0 +1,5 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Parallel reduction algorithms examples package."""
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/min_reduction.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/min_reduction.py
new file mode 100644
index 00000000..b8023306
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/min_reduction.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Computing the minimum value of a sequence using `reduce_into`.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+
+def min_op(a, b):
+    # the binary operation for the reduction
+    return a if a < b else b
+
+
+# Prepare the input and output arrays.
+dtype = np.int32
+h_init = np.array([42], dtype=dtype)
+d_input = cp.array([8, 6, 7, 5, 3, 0, 9], dtype=dtype)
+d_output = cp.empty(1, dtype=dtype)
+
+# Perform the reduction.
+cuda.compute.reduce_into(
+    d_in=d_input, d_out=d_output, num_items=len(d_input), op=min_op, h_init=h_init
+)
+
+# Verify the result.
+expected_output = 0
+result = d_output.get()[0]
+
+assert result == expected_output
+print(f"Min reduction result: {result}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/minmax_reduction.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/minmax_reduction.py
new file mode 100644
index 00000000..d99fb372
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/minmax_reduction.py
@@ -0,0 +1,61 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Simultaneously computing the minimum and maximum values of a sequence using `reduce_into`
+with a custom data type.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    TransformIterator,
+    gpu_struct,
+)
+
+
+# Define a custom data type for the accumulator.
+@gpu_struct
+class MinMax:
+    min_val: np.float64
+    max_val: np.float64
+
+
+# Define the binary operation for the reduction.
+def minmax_op(v1: MinMax, v2: MinMax):
+    c_min = min(v1.min_val, v2.min_val)
+    c_max = max(v1.max_val, v2.max_val)
+    return MinMax(c_min, c_max)
+
+
+# Define a transform operation to convert a value `x` to MinMax(abs(x), abs(x)).
+def transform_op(v):
+    av = abs(v)
+    return MinMax(av, av)
+
+
+# Prepare the input and output data.
+nelems = 4096
+d_in = cp.random.randn(nelems)
+tr_it = TransformIterator(d_in, transform_op)
+
+d_out = cp.empty(tuple(), dtype=MinMax.dtype)
+
+h_init = MinMax(np.inf, -np.inf)
+
+# Perform the reduction.
+cuda.compute.reduce_into(
+    d_in=tr_it, d_out=d_out, num_items=nelems, op=minmax_op, h_init=h_init
+)
+
+# Verify the result.
+actual = d_out.get()
+h = np.abs(d_in.get())
+expected = np.asarray([(h.min(), h.max())], dtype=MinMax.dtype)
+
+assert actual == expected
+print(f"MinMax reduction result: {actual}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/reduce_object.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/reduce_object.py
new file mode 100644
index 00000000..5edbc6d1
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/reduce_object.py
@@ -0,0 +1,58 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Reduction example using the object API.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    OpKind,
+)
+
+# Prepare the input and output arrays.
+dtype = np.int32
+init_value = 5
+h_init = np.array([init_value], dtype=dtype)
+h_input = np.array([1, 2, 3, 4], dtype=dtype)
+d_input = cp.asarray(h_input)
+d_output = cp.empty(1, dtype=dtype)
+
+# Create a reducer object.
+reducer = cuda.compute.make_reduce_into(
+    d_in=d_input, d_out=d_output, op=OpKind.PLUS, h_init=h_init
+)
+
+# Get the temporary storage size.
+temp_storage_size = reducer(
+    temp_storage=None,
+    d_in=d_input,
+    d_out=d_output,
+    num_items=len(h_input),
+    op=OpKind.PLUS,
+    h_init=h_init,
+)
+
+# Allocate temporary storage using any user-defined allocator.
+# The result must be an object exposing `__cuda_array_interface__`.
+d_temp_storage = cp.empty(temp_storage_size, dtype=np.uint8)
+
+# Perform the reduction.
+reducer(
+    temp_storage=d_temp_storage,
+    d_in=d_input,
+    d_out=d_output,
+    num_items=len(h_input),
+    op=OpKind.PLUS,
+    h_init=h_init,
+)
+
+expected_result = np.sum(h_input) + init_value
+actual_result = d_output.get()[0]
+assert actual_result == expected_result
+print("Reduce object example completed successfully")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/sum_custom_reduction.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/sum_custom_reduction.py
new file mode 100644
index 00000000..64985048
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/sum_custom_reduction.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Sum only even values in an array using reduction with custom operation.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+# Prepare the input and output arrays.
+dtype = np.int32
+h_init = np.array([0], dtype=dtype)
+d_input = cp.array([1, 2, 3, 4, 5], dtype=dtype)
+d_output = cp.empty(1, dtype=dtype)
+
+# Define the binary operation for the reduction.
+
+
+def add_op(a, b):
+    return (a if a % 2 == 0 else 0) + (b if b % 2 == 0 else 0)
+
+
+# Perform the reduction.
+cuda.compute.reduce_into(
+    d_in=d_input, d_out=d_output, num_items=len(d_input), op=add_op, h_init=h_init
+)
+
+# Verify the result.
+expected_output = 6
+assert (d_output == expected_output).all()
+result = d_output[0]
+print(f"Custom sum reduction result: {result}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/sum_reduction.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/sum_reduction.py
new file mode 100644
index 00000000..096b3649
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/sum_reduction.py
@@ -0,0 +1,31 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Sum all values in an array using reduction with PLUS operation.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import OpKind
+
+# Prepare the input and output arrays.
+dtype = np.int32
+h_init = np.array([0], dtype=dtype)
+d_input = cp.array([1, 2, 3, 4, 5], dtype=dtype)
+d_output = cp.empty(1, dtype=dtype)
+
+# Perform the reduction.
+cuda.compute.reduce_into(
+    d_in=d_input, d_out=d_output, num_items=len(d_input), op=OpKind.PLUS, h_init=h_init
+)
+
+# Verify the result.
+expected_output = 15
+assert (d_output == expected_output).all()
+result = d_output[0]
+print(f"Sum reduction result: {result}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/sum_reduction_lambda.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/sum_reduction_lambda.py
new file mode 100644
index 00000000..2018fa8a
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/reduction/sum_reduction_lambda.py
@@ -0,0 +1,38 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Sum all values in an array using reduction with a lambda function.
+
+This example demonstrates that lambda functions can be used directly
+as reduction operators, providing a concise alternative to defining
+named functions.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+# Prepare the input and output arrays.
+dtype = np.int32
+h_init = np.array([0], dtype=dtype)
+d_input = cp.array([1, 2, 3, 4, 5], dtype=dtype)
+d_output = cp.empty(1, dtype=dtype)
+
+# Perform the reduction using a lambda function.
+cuda.compute.reduce_into(
+    d_in=d_input,
+    d_out=d_output,
+    num_items=len(d_input),
+    op=lambda a, b: a + b,
+    h_init=h_init,
+)
+
+# Verify the result.
+expected_output = 15
+assert (d_output == expected_output).all()
+result = d_output[0]
+print(f"Sum reduction with lambda result: {result}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/__init__.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/__init__.py
new file mode 100644
index 00000000..e4c4d770
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/__init__.py
@@ -0,0 +1,5 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Parallel scan algorithms examples package."""
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/ema_example.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/ema_example.py
new file mode 100644
index 00000000..43f9e71b
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/ema_example.py
@@ -0,0 +1,101 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    CountingIterator,
+    TransformIterator,
+    ZipIterator,
+    gpu_struct,
+)
+
+# Given input vector u and smoothing parameter 0<alpha<1
+# the exponential moving average sequence ma is defined
+# by ma[t] = (1 - alpha) * u[t] + alpha * ma[t-1] with
+# initial condition ma[0] = u[0].
+#
+# This recurrence equation admits a closed form solution:
+#
+# ma[t] = alpha ** (t + 1) * u[0] + (1 - alpha) * (alpha ** t) *
+#     sum(u[s] * alpha ** (-s) for s in range(t))
+#
+# Closed form solution could be computed using inclusive_scan,
+# except naive implementation suffers from underflow/overflow
+# problem for long sequences.
+#
+# This implementation solves this problem by representing number
+# using (fp_value, int_exponent) to extend representable range.
+u = 3.0 + cp.sin(cp.linspace(-3.0, 3.0, num=1024, dtype=cp.double))
+u += cp.random.normal(0.0, 0.1, size=u.size)
+alpha = 0.05
+
+assert 0.0 < alpha < 1.0
+
+
+@gpu_struct
+class ValueScale:
+    value: cp.float64
+    scale: cp.int64
+
+
+def add_op(v1: ValueScale, v2: ValueScale) -> ValueScale:
+    if v1.scale > v2.scale:
+        s = v2.scale
+        v = v2.value + v1.value * (alpha ** (v1.scale - v2.scale))
+    else:
+        s = v1.scale
+        v = v1.value + v2.value * (alpha ** (v2.scale - v1.scale))
+    return ValueScale(v, s)
+
+
+def negative_op(i: cp.int64) -> cp.int64:
+    return -i
+
+
+seq_it = CountingIterator(cp.int64(0))
+negative_exponents_it = TransformIterator(seq_it, negative_op)
+d_inp = ZipIterator(u, negative_exponents_it)
+
+d_cumsum = cp.empty(u.shape, dtype=ValueScale.dtype)
+h_init = ValueScale(0.0, 0)
+
+cuda.compute.inclusive_scan(
+    d_in=d_inp, d_out=d_cumsum, op=add_op, init_value=h_init, num_items=u.size
+)
+
+it_seq = CountingIterator(cp.int64(0))
+d_ema = cp.empty_like(u)
+
+
+def combine_op(v: ValueScale, t: cp.int64) -> cp.float64:
+    return (1 - alpha) * v.value * alpha ** (t + v.scale)
+
+
+cuda.compute.binary_transform(
+    d_in1=d_cumsum, d_in2=it_seq, d_out=d_ema, op=combine_op, num_items=u.size
+)
+
+d_ema += (alpha ** cp.arange(1, u.size + 1)) * u[0]
+
+
+def ema_ref(u: np.ndarray, alpha: float):
+    "Sequential reference implementation of EMA"
+    a = np.empty_like(u)
+    a[0] = u[0]
+    for t in range(1, u.size):
+        a[t] = (1 - alpha) * u[t] + alpha * a[t - 1]
+
+    return a
+
+
+h_u = u.get()
+h_ema = ema_ref(h_u, alpha)
+
+assert np.allclose(h_ema, d_ema.get())
+print("Exponential moving average example completed successfully")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/exclusive_scan_max.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/exclusive_scan_max.py
new file mode 100644
index 00000000..59d453ec
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/exclusive_scan_max.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Exclusive scan using custom maximum operation.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+# Define the binary operation for the scan.
+
+
+def max_op(a, b):
+    return max(a, b)
+
+
+# Prepare the input and output arrays.
+h_init = np.array([1], dtype="int32")
+d_input = cp.array([-5, 0, 2, -3, 2, 4, 0, -1, 2, 8], dtype="int32")
+d_output = cp.empty_like(d_input, dtype="int32")
+
+# Perform the exclusive scan.
+cuda.compute.exclusive_scan(
+    d_in=d_input, d_out=d_output, op=max_op, init_value=h_init, num_items=d_input.size
+)
+
+# Verify the result.
+expected = np.asarray([1, 1, 1, 2, 2, 2, 4, 4, 4, 4])
+result = d_output.get()
+
+np.testing.assert_equal(result, expected)
+print(f"Exclusive scan max result: {result}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/exclusive_scan_object.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/exclusive_scan_object.py
new file mode 100644
index 00000000..7e6200ca
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/exclusive_scan_object.py
@@ -0,0 +1,53 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Exclusive scan example demonstrating the object API.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    OpKind,
+)
+
+# Prepare the input and output arrays.
+dtype = np.int32
+h_init = np.array([0], dtype=dtype)
+h_input = np.array([1, 2, 3, 4], dtype=dtype)
+d_input = cp.asarray(h_input)
+d_output = cp.empty(len(h_input), dtype=dtype)
+
+# Create the scanner object and allocate temporary storage.
+scanner = cuda.compute.make_exclusive_scan(
+    d_in=d_input, d_out=d_output, op=OpKind.PLUS, init_value=h_init
+)
+temp_storage_size = scanner(
+    temp_storage=None,
+    d_in=d_input,
+    d_out=d_output,
+    op=OpKind.PLUS,
+    num_items=len(h_input),
+    init_value=h_init,
+)
+d_temp_storage = cp.empty(temp_storage_size, dtype=np.uint8)
+
+# Perform the exclusive scan.
+scanner(
+    temp_storage=d_temp_storage,
+    d_in=d_input,
+    d_out=d_output,
+    op=OpKind.PLUS,
+    num_items=len(h_input),
+    init_value=h_init,
+)
+
+# Verify the result.
+expected_result = np.array([0, 1, 3, 6], dtype=dtype)
+actual_result = d_output.get()
+np.testing.assert_array_equal(actual_result, expected_result)
+print("Exclusive scan object example completed successfully")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/inclusive_scan_custom.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/inclusive_scan_custom.py
new file mode 100644
index 00000000..9a62bca2
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/inclusive_scan_custom.py
@@ -0,0 +1,36 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Inclusive scan with custom operation (prefix sum of even values).
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+# Prepare the input and output arrays.
+h_init = np.array([0], dtype="int32")
+d_input = cp.array([1, 2, 3, 4, 5], dtype="int32")
+d_output = cp.empty_like(d_input, dtype="int32")
+
+# Define the binary operation for the scan.
+
+
+def add_op(a, b):
+    return (a if a % 2 == 0 else 0) + (b if b % 2 == 0 else 0)
+
+
+# Perform the inclusive scan.
+cuda.compute.inclusive_scan(
+    d_in=d_input, d_out=d_output, op=add_op, init_value=h_init, num_items=d_input.size
+)
+
+# Verify the result.
+expected = np.asarray([0, 2, 2, 6, 6])
+assert np.array_equal(d_output.get(), expected)
+result = d_output.get()
+print(f"Inclusive scan custom result: {result}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/inclusive_scan_object.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/inclusive_scan_object.py
new file mode 100644
index 00000000..17e56a69
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/inclusive_scan_object.py
@@ -0,0 +1,53 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Inclusive scan example demonstrating the object API.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    OpKind,
+)
+
+# Prepare the input and output arrays.
+dtype = np.int32
+h_init = np.array([0], dtype=dtype)
+h_input = np.array([1, 2, 3, 4], dtype=dtype)
+d_input = cp.asarray(h_input)
+d_output = cp.empty(len(h_input), dtype=dtype)
+
+# Create the scanner object and allocate temporary storage.
+scanner = cuda.compute.make_inclusive_scan(
+    d_in=d_input, d_out=d_output, op=OpKind.PLUS, init_value=h_init
+)
+temp_storage_size = scanner(
+    temp_storage=None,
+    d_in=d_input,
+    d_out=d_output,
+    op=OpKind.PLUS,
+    num_items=len(h_input),
+    init_value=h_init,
+)
+d_temp_storage = cp.empty(temp_storage_size, dtype=np.uint8)
+
+# Perform the inclusive scan.
+scanner(
+    temp_storage=d_temp_storage,
+    d_in=d_input,
+    d_out=d_output,
+    op=OpKind.PLUS,
+    num_items=len(h_input),
+    init_value=h_init,
+)
+
+# Verify the result.
+expected_result = np.array([1, 3, 6, 10], dtype=dtype)
+actual_result = d_output.get()
+np.testing.assert_array_equal(actual_result, expected_result)
+print("Inclusive scan object example completed successfully")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/logcdf_example.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/logcdf_example.py
new file mode 100644
index 00000000..c711d98b
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/logcdf_example.py
@@ -0,0 +1,79 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Given a vector of log-probabilities, compute a vector of logarithms of cumulative density function.
+"""
+
+import math
+
+import cupy as cp
+import cupyx.scipy.special as cp_special
+import numpy as np
+
+import cuda.compute
+
+# Prepare the input data and compute log-probabilities.
+# Use log-add-exp binary operation to sidestep flush-to-zero
+# and numerical exceptions computing logarithms of that.
+n = 500
+p = 0.31
+m = cp.arange(n + 1, dtype=cp.float64)
+nm = n - m
+lognorm = (
+    cp_special.loggamma(1 + n)
+    - cp_special.loggamma(1 + m)
+    - cp_special.loggamma(1 + nm)
+)
+logpdf = lognorm + m * cp.log(p) + nm * cp.log1p(-p)
+
+assert n + 1 == logpdf.size
+
+# Define the binary operations for the scans.
+
+
+def logaddexp(logp1: cp.float64, logp2: cp.float64):
+    m_max = max(logp1, logp2)
+    m_min = min(logp1, logp2)
+    return m_max + math.log(1.0 + math.exp(m_min - m_max))
+
+
+def maximum(v1: cp.float64, v2: cp.float64):
+    return max(v1, v2)
+
+
+# Prepare the output arrays and initial value.
+logcdf = cp.empty_like(logpdf)
+h_init = np.array(-np.inf, dtype=np.float64)
+
+logcdf2 = cp.empty_like(logpdf)
+
+# Perform the first inclusive scan (log-add-exp).
+cuda.compute.inclusive_scan(
+    d_in=logpdf, d_out=logcdf, op=logaddexp, init_value=h_init, num_items=logpdf.size
+)
+
+# Perform the second inclusive scan (maximum).
+cuda.compute.inclusive_scan(
+    d_in=logcdf, d_out=logcdf2, op=maximum, init_value=h_init, num_items=logpdf.size
+)
+
+# Verify the results and compute quantiles.
+assert cp.all(logcdf2[:-1] <= logcdf2[1:])
+
+assert float(cp.max(logcdf2)) <= 0.0
+
+q25, q75 = cp.searchsorted(logcdf2, cp.asarray(np.log([0.25, 0.75])))
+
+try:
+    from scipy.stats.distributions import binom
+
+    q25_ref, q75_ref = binom(n, p).isf([0.75, 0.25])
+    assert q25 == q25_ref
+    assert q75 == q75_ref
+except ImportError:
+    print("scipy not found, skipping assertions")
+
+print(f"Log CDF example completed. q25: {q25}, q75: {q75}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/running_average.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/running_average.py
new file mode 100644
index 00000000..6f2acd8e
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/running_average.py
@@ -0,0 +1,50 @@
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    ConstantIterator,
+    TransformOutputIterator,
+    ZipIterator,
+    gpu_struct,
+)
+
+# example-begin
+"""
+Inclusive scan using zip iterator and output transform iterator to compute running average.
+"""
+
+
+@gpu_struct
+class SumAndCount:  # data type to store the running sum and the count
+    sum: np.float32
+    count: np.int32
+
+
+# binary operation for the scan computes the running sum and running count
+def add_op(x1: SumAndCount, x2: SumAndCount) -> SumAndCount:
+    return SumAndCount(x1.sum + x2.sum, x1.count + x2.count)
+
+
+# output transform operation divides the sum by the count to get the running average
+def write_op(x: SumAndCount) -> np.float32:
+    return x.sum / x.count
+
+
+# construct a zip iterator to pair the input with the sequence [1, 1, ..., 1]
+d_input = cp.array([1.0, 2.0, 3.0, 4.0, 5.0], dtype=np.float32)
+it_input = ZipIterator(d_input, ConstantIterator(np.int32(1)))
+
+# output transform iterator divides the sum by the count to get the running average
+d_output = cp.empty_like(d_input)
+it_output = TransformOutputIterator(d_output, write_op)
+
+h_init = SumAndCount(0.0, 0)
+
+cuda.compute.inclusive_scan(
+    d_in=it_input, d_out=it_output, op=add_op, init_value=h_init, num_items=len(d_input)
+)
+
+expected = np.array([1.0, 1.5, 2.0, 2.5, 3.0], dtype=np.float32)
+np.testing.assert_allclose(d_output.get(), expected)
+# example-end
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/segmented_sum.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/segmented_sum.py
new file mode 100644
index 00000000..6e50aa73
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/scan/segmented_sum.py
@@ -0,0 +1,66 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Implement segmented scan using zip iterator and ordinary scan.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    ZipIterator,
+    gpu_struct,
+)
+
+# Prepare the input data and head flags.
+# Segmented inclusive sum on array of values and head-flags
+# array demarkating locations of start of segments can be implemented
+# using ordinary inclusive scan using Schwarz operator acting
+# of value-flag pairs. `ZipIterator` can be used to efficiently
+# load data from pair of input arrays, instead of copying them
+# to array of structs.
+#
+# For example, for data = [1, 1, 1, 1, 1, 1, 1, 1] with
+# 3 segments encoded by head_flags = [0, 0, 1, 0, 0, 1, 1, 0]
+# corresponding to segmented data [[1, 1], [1, 1, 1], [1], [1, 1]],
+# the expected prefix-sum values are [1, 2, 1, 2, 3, 1, 1, 2]
+data = cp.asarray([1, 1, 1, 1, 1, 1, 1, 1], dtype=cp.int64)
+hflg = cp.asarray([0, 0, 1, 0, 0, 1, 1, 0], dtype=cp.int32)
+
+# Define the custom data type and binary operation.
+
+
+@gpu_struct
+class ValueFlag:
+    value: cp.int64
+    flag: cp.int32
+
+
+def schwartz_sum(op1: ValueFlag, op2: ValueFlag) -> ValueFlag:
+    f1: cp.int32 = 1 if op1.flag else 0
+    f2: cp.int32 = 1 if op2.flag else 0
+    f: cp.int32 = f1 | f2
+    v: cp.int64 = op2.value if f2 else op1.value + op2.value
+    return ValueFlag(v, f)
+
+
+# Prepare the output array and initial value.
+zip_it = ZipIterator(data, hflg)
+d_output = cp.empty(data.shape, dtype=ValueFlag.dtype)
+h_init = ValueFlag(0, 0)
+
+# Perform the segmented scan.
+cuda.compute.inclusive_scan(
+    d_in=zip_it, d_out=d_output, op=schwartz_sum, init_value=h_init, num_items=data.size
+)
+
+# Verify the result.
+expected_prefix = np.asarray([1, 2, 1, 2, 3, 1, 1, 2], dtype=np.int64)
+result = d_output.get()
+
+assert np.array_equal(result["value"], expected_prefix)
+print(f"Segmented sum result: {result}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/segmented/__init__.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/segmented/__init__.py
new file mode 100644
index 00000000..5b2a3750
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/segmented/__init__.py
@@ -0,0 +1,5 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Parallel segmented algorithms examples package."""
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/segmented/segmented_reduce_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/segmented/segmented_reduce_basic.py
new file mode 100644
index 00000000..25d6cfbc
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/segmented/segmented_reduce_basic.py
@@ -0,0 +1,58 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use segmented_reduce to find the minimum in each segment.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+
+def min_op(a, b):
+    return a if a < b else b
+
+
+dtype = np.dtype(np.int32)
+max_val = np.iinfo(dtype).max
+h_init = np.asarray(max_val, dtype=dtype)
+
+# Prepare the offsets.
+offsets = cp.array([0, 7, 11, 16], dtype=np.int64)
+first_segment = (8, 6, 7, 5, 3, 0, 9)
+second_segment = (-4, 3, 0, 1)
+third_segment = (3, 1, 11, 25, 8)
+
+# Prepare the input array.
+d_input = cp.array(
+    [*first_segment, *second_segment, *third_segment],
+    dtype=dtype,
+)
+
+# Prepare the start and end offsets.
+start_o = offsets[:-1]
+end_o = offsets[1:]
+
+# Prepare the output array.
+n_segments = start_o.size
+d_output = cp.empty(n_segments, dtype=dtype)
+
+# Perform the segmented reduce.
+cuda.compute.segmented_reduce(
+    d_in=d_input,
+    d_out=d_output,
+    num_segments=n_segments,
+    start_offsets_in=start_o,
+    end_offsets_in=end_o,
+    op=min_op,
+    h_init=h_init,
+)
+
+# Verify the result.
+expected_output = cp.asarray([0, -4, 1], dtype=d_output.dtype)
+assert (d_output == expected_output).all()
+print(f"Segmented reduce basic result: {d_output.get()}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/segmented/segmented_reduce_object.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/segmented/segmented_reduce_object.py
new file mode 100644
index 00000000..026a3a55
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/segmented/segmented_reduce_object.py
@@ -0,0 +1,69 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Segmented reduction using the object API.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    OpKind,
+)
+
+# Prepare the input and output arrays.
+dtype = np.int32
+h_init = np.array([0], dtype=dtype)
+h_input = np.array([1, 2, 3, 4, 5, 6], dtype=dtype)
+d_input = cp.asarray(h_input)
+d_output = cp.empty(2, dtype=dtype)
+
+start_offsets = cp.array([0, 3], dtype=np.int64)
+end_offsets = cp.array([3, 6], dtype=np.int64)
+
+# Create the segmented reduce object.
+reducer = cuda.compute.make_segmented_reduce(
+    d_in=d_input,
+    d_out=d_output,
+    start_offsets_in=start_offsets,
+    end_offsets_in=end_offsets,
+    op=OpKind.PLUS,
+    h_init=h_init,
+)
+
+# Get the temporary storage size.
+temp_storage_size = reducer(
+    temp_storage=None,
+    d_in=d_input,
+    d_out=d_output,
+    num_segments=2,
+    start_offsets_in=start_offsets,
+    end_offsets_in=end_offsets,
+    op=OpKind.PLUS,
+    h_init=h_init,
+)
+
+# Allocate the temporary storage.
+d_temp_storage = cp.empty(temp_storage_size, dtype=np.uint8)
+
+# Perform the segmented reduce.
+reducer(
+    temp_storage=d_temp_storage,
+    d_in=d_input,
+    d_out=d_output,
+    num_segments=2,
+    start_offsets_in=start_offsets,
+    end_offsets_in=end_offsets,
+    op=OpKind.PLUS,
+    h_init=h_init,
+)
+
+# Verify the result.
+expected_result = np.array([6, 15], dtype=dtype)
+actual_result = d_output.get()
+np.testing.assert_array_equal(actual_result, expected_result)
+print("Segmented reduce object example completed successfully")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/select/__init__.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/select/__init__.py
new file mode 100644
index 00000000..a3dd39c1
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/select/__init__.py
@@ -0,0 +1,4 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/select/select_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/select/select_basic.py
new file mode 100644
index 00000000..2cd3fae3
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/select/select_basic.py
@@ -0,0 +1,39 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+import cupy as cp
+
+from cuda.compute.algorithms import select
+
+# Create input data
+d_in = cp.array([1, 2, 3, 4, 5, 6, 7, 8], dtype=cp.int32)
+d_out = cp.empty_like(d_in)
+d_num_selected = cp.zeros(2, dtype=cp.uint64)
+
+
+# Define select condition (keep even numbers)
+def is_even(x):
+    return x % 2 == 0
+
+
+# Execute select
+select(
+    d_in=d_in,
+    d_out=d_out,
+    d_num_selected_out=d_num_selected,
+    cond=is_even,
+    num_items=len(d_in),
+)
+
+# Get results
+num_selected = int(d_num_selected[0])
+result = d_out[:num_selected].get()
+print(f"Selected {num_selected} items: {result}")
+# Output: Selected 4 items: [2 4 6 8]
+# example-end
+
+assert num_selected == 4
+assert (result == [2, 4, 6, 8]).all()
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/select/select_object.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/select/select_object.py
new file mode 100644
index 00000000..39fc68de
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/select/select_object.py
@@ -0,0 +1,78 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+import cupy as cp
+
+from cuda.compute.algorithms import make_select
+
+# Create input data
+d_in = cp.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], dtype=cp.int32)
+d_out = cp.empty_like(d_in)
+d_num_selected = cp.zeros(2, dtype=cp.uint64)
+
+
+# Define select condition (keep values > 5)
+def greater_than_5(x):
+    return x > 5
+
+
+# Create select object (can be reused)
+selector = make_select(
+    d_in=d_in,
+    d_out=d_out,
+    d_num_selected_out=d_num_selected,
+    cond=greater_than_5,
+)
+
+# Get required temp storage
+temp_storage_bytes = selector(
+    temp_storage=None,
+    d_in=d_in,
+    d_out=d_out,
+    d_num_selected_out=d_num_selected,
+    cond=greater_than_5,
+    num_items=len(d_in),
+)
+d_temp_storage = cp.empty(temp_storage_bytes, dtype=cp.uint8)
+
+# Execute select
+selector(
+    temp_storage=d_temp_storage,
+    d_in=d_in,
+    d_out=d_out,
+    d_num_selected_out=d_num_selected,
+    cond=greater_than_5,
+    num_items=len(d_in),
+)
+
+# Get results
+num_selected = int(d_num_selected[0])
+result = d_out[:num_selected].get()
+print(f"Selected {num_selected} items: {result}")
+# Output: Selected 5 items: [ 6  7  8  9 10]
+
+# Reuse the same select object with different input
+d_in2 = cp.array([10, 20, 3, 15, 2, 8, 30], dtype=cp.int32)
+d_out2 = cp.empty_like(d_in2)
+d_num_selected2 = cp.zeros(2, dtype=cp.uint64)
+
+selector(
+    temp_storage=d_temp_storage,
+    d_in=d_in2,
+    d_out=d_out2,
+    d_num_selected_out=d_num_selected2,
+    cond=greater_than_5,
+    num_items=len(d_in2),
+)
+
+num_selected2 = int(d_num_selected2[0])
+result2 = d_out2[:num_selected2].get()
+print(f"Second select: {num_selected2} items: {result2}")
+# Output: Second select: 5 items: [10 20 15  8 30]
+# example-end
+
+assert num_selected == 5
+assert (result == [6, 7, 8, 9, 10]).all()
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/select/select_with_iterator.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/select/select_with_iterator.py
new file mode 100644
index 00000000..d8739cb2
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/select/select_with_iterator.py
@@ -0,0 +1,48 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+import cupy as cp
+
+from cuda.compute.algorithms import select
+from cuda.compute.iterators import TransformIterator
+
+# Create input data
+d_in = cp.array([1, 2, 3, 4, 5, 6, 7, 8], dtype=cp.int32)
+d_out = cp.empty_like(d_in)
+d_num_selected = cp.zeros(2, dtype=cp.uint64)
+
+
+# Create iterator that squares each value
+def square(x):
+    return x * x
+
+
+squared_iter = TransformIterator(d_in, square)
+
+
+# Select squared values that are greater than 20
+def greater_than_20(x):
+    return x > 20
+
+
+select(
+    d_in=squared_iter,
+    d_out=d_out,
+    d_num_selected_out=d_num_selected,
+    cond=greater_than_20,
+    num_items=len(d_in),
+)
+
+# Get results
+num_selected = int(d_num_selected[0])
+result = d_out[:num_selected].get()
+print(f"Selected {num_selected} items: {result}")
+# Output: Selected 4 items: [25 36 49 64]
+# (5^2=25, 6^2=36, 7^2=49, 8^2=64, all > 20)
+# example-end
+
+assert num_selected == 4
+assert (result == [25, 36, 49, 64]).all()
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/select/select_with_side_effect.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/select/select_with_side_effect.py
new file mode 100644
index 00000000..cb1fca62
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/select/select_with_side_effect.py
@@ -0,0 +1,54 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+import cupy as cp
+from numba import cuda as numba_cuda
+
+from cuda.compute.algorithms import select
+
+# Create input data: values 0 to 99
+d_in = cp.arange(100, dtype=cp.int32)
+d_out = cp.empty_like(d_in)
+d_num_selected = cp.empty(1, dtype=cp.uint64)
+
+# Counter for rejected items (side effect state)
+reject_count = cp.zeros(1, dtype=cp.int32)
+
+
+# Define condition that counts rejected items as a side effect
+def count_rejects(x):
+    if x % 2 == 0:
+        return True
+    else:
+        numba_cuda.atomic.add(reject_count, 0, 1)
+        return False
+
+
+# Execute select - selects even numbers, counts rejections
+select(
+    d_in=d_in,
+    d_out=d_out,
+    d_num_selected_out=d_num_selected,
+    cond=count_rejects,
+    num_items=len(d_in),
+)
+
+# Get results
+num_selected = int(d_num_selected.get()[0])
+num_rejected = int(reject_count.get()[0])
+result = d_out[:num_selected].get()
+
+print(f"Selected {num_selected} items (values % 2 == 0)")
+print(f"Rejected {num_rejected} items (values % 2 != 0)")
+print(f"First 5 selected: {result[:5]}")
+# Output:
+# Selected 50 items (even numbers)
+# Rejected 50 items (odd numbers)
+# First 5 selected: [0 2 4 6 8]
+# example-end
+
+assert num_selected == 50  # Even numbers
+assert num_rejected == 50  # Odd numbers
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/serialization/__init__.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/serialization/__init__.py
new file mode 100644
index 00000000..62b6c059
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/serialization/__init__.py
@@ -0,0 +1,5 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Serialization and ahead-of-time compilation examples package."""
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/serialization/ahead_of_time_compilation.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/serialization/ahead_of_time_compilation.py
new file mode 100644
index 00000000..3012c33e
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/serialization/ahead_of_time_compilation.py
@@ -0,0 +1,50 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ruff: noqa: E402 — the v2-skip block below intentionally precedes the
+# example's imports so the imports stay grouped at the start of the example
+# body (after `# example-begin`).
+
+# Ahead-of-time compilation is only supported on the default (v1) backend; the
+# HostJIT (v2) backend raises NotImplementedError. Skip cleanly there so the
+# example runner treats it as a pass.
+import sys
+
+try:
+    from cuda.compute._build_info import USING_V2
+except ImportError:
+    USING_V2 = False
+
+if USING_V2:
+    print("ahead-of-time build is unsupported on the HostJIT (v2) backend; skipping.")
+    sys.exit(0)
+
+# example-begin
+"""
+Compile a reduction ahead of time for multiple GPU architectures without a GPU
+present, using dtype-only placeholders, and serialize the result for deployment.
+"""
+
+import numpy as np
+
+from cuda.compute import OpKind, ProxyArray, ProxyValue, make_reduce_into, serialize
+
+# ProxyArray / ProxyValue describe only the dtype of each argument; they hold no
+# GPU memory, so no device is required to build. compute_capability must be given
+# explicitly, since there is no device whose architecture we could default to.
+reducer = make_reduce_into(
+    d_in=ProxyArray(np.int32),
+    d_out=ProxyArray(np.int32),
+    op=OpKind.PLUS,
+    h_init=ProxyValue(np.int32),
+    compute_capability=[80, 90],  # build for sm_80 and sm_90
+)
+
+# Serialize the multi-architecture build for shipping to deployment targets. On
+# a target GPU, cuda.compute.deserialize(blob) reconstructs the object, and the
+# build result matching the running architecture is loaded on the first call.
+blob = serialize(reducer)
+
+assert len(blob) > 0
+print(f"Compiled ahead of time for sm_80 and sm_90; serialized {len(blob)} bytes")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/serialization/serialize_roundtrip.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/serialization/serialize_roundtrip.py
new file mode 100644
index 00000000..f5914e6e
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/serialization/serialize_roundtrip.py
@@ -0,0 +1,74 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ruff: noqa: E402 — the v2-skip block below intentionally precedes the
+# example's imports so the imports stay grouped at the start of the example
+# body (after `# example-begin`).
+
+# Serialization is only supported on the default (v1) backend; the HostJIT (v2)
+# backend raises NotImplementedError. Skip cleanly there so the example runner
+# treats it as a pass.
+import sys
+
+try:
+    from cuda.compute._build_info import USING_V2
+except ImportError:
+    USING_V2 = False
+
+if USING_V2:
+    print("serialization is unsupported on the HostJIT (v2) backend; skipping.")
+    sys.exit(0)
+
+# example-begin
+"""
+Serialize a built reduction to bytes, then reconstruct and run it without
+recompiling.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import OpKind
+
+# Build a reduction object for the current device, as usual.
+dtype = np.int32
+h_init = np.array([0], dtype=dtype)
+d_input = cp.array([1, 2, 3, 4, 5], dtype=dtype)
+d_output = cp.empty(1, dtype=dtype)
+
+reducer = cuda.compute.make_reduce_into(
+    d_in=d_input, d_out=d_output, op=OpKind.PLUS, h_init=h_init
+)
+
+# Serialize the compiled build result to a blob of bytes. In practice you would
+# write this to a file and load it in a later run or on another machine; here we
+# keep it in memory to stay self-contained.
+blob = cuda.compute.serialize(reducer)
+
+# Reconstruct the reduction from the blob. This performs no JIT compilation.
+restored = cuda.compute.deserialize(blob)
+
+# Invoke the restored object exactly as if it had just been built.
+temp_storage_size = restored(
+    temp_storage=None,
+    d_in=d_input,
+    d_out=d_output,
+    num_items=len(d_input),
+    op=OpKind.PLUS,
+    h_init=h_init,
+)
+d_temp_storage = cp.empty(temp_storage_size, dtype=np.uint8)
+restored(
+    temp_storage=d_temp_storage,
+    d_in=d_input,
+    d_out=d_output,
+    num_items=len(d_input),
+    op=OpKind.PLUS,
+    h_init=h_init,
+)
+
+# The reconstructed object produces the same result as the original.
+assert d_output.get()[0] == 15
+print("Serialize/deserialize round-trip result:", d_output.get()[0])
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/__init__.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/__init__.py
new file mode 100644
index 00000000..6b7fa22a
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/__init__.py
@@ -0,0 +1,5 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Parallel sorting algorithms examples package."""
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/merge_sort_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/merge_sort_basic.py
new file mode 100644
index 00000000..7558f314
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/merge_sort_basic.py
@@ -0,0 +1,47 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Demonstrate basic merge sort with keys and values.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    OpKind,
+)
+
+# Prepare the input and output arrays.
+h_in_keys = np.array([-5, 0, 2, -3, 2, 4, 0, -1, 2, 8], dtype="int32")
+h_in_values = np.array(
+    [-3.2, 2.2, 1.9, 4.0, -3.9, 2.7, 0, 8.3 - 1, 2.9, 5.4], dtype="float32"
+)
+
+d_in_keys = cp.asarray(h_in_keys)
+d_in_values = cp.asarray(h_in_values)
+
+# Perform the merge sort.
+cuda.compute.merge_sort(
+    d_in_keys=d_in_keys,
+    d_in_values=d_in_values,
+    d_out_keys=d_in_keys,
+    d_out_values=d_in_values,
+    op=OpKind.LESS,
+    num_items=d_in_keys.size,
+)
+
+# Verify the result.
+h_out_keys = cp.asnumpy(d_in_keys)
+h_out_values = cp.asnumpy(d_in_values)
+
+argsort = np.argsort(h_in_keys, stable=True)
+expected_keys = np.array(h_in_keys)[argsort]
+expected_values = np.array(h_in_values)[argsort]
+
+assert np.array_equal(h_out_keys, expected_keys)
+assert np.array_equal(h_out_values, expected_values)
+print(f"Merge sort basic result - keys: {h_out_keys}, values: {h_out_values}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/merge_sort_object.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/merge_sort_object.py
new file mode 100644
index 00000000..c737c9cd
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/merge_sort_object.py
@@ -0,0 +1,68 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Merge sort example demonstrating the object API.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    OpKind,
+)
+
+# Prepare the input and output arrays.
+dtype = np.int32
+h_input_keys = np.array([4, 2, 3, 1], dtype=dtype)
+h_input_values = np.array([40, 20, 30, 10], dtype=dtype)
+d_input_keys = cp.asarray(h_input_keys)
+d_input_values = cp.asarray(h_input_values)
+d_output_keys = cp.empty_like(d_input_keys)
+d_output_values = cp.empty_like(d_input_values)
+
+# Create the merge sort object.
+sorter = cuda.compute.make_merge_sort(
+    d_in_keys=d_input_keys,
+    d_in_values=d_input_values,
+    d_out_keys=d_output_keys,
+    d_out_values=d_output_values,
+    op=OpKind.LESS,
+)
+
+# Get the temporary storage size.
+temp_storage_size = sorter(
+    temp_storage=None,
+    d_in_keys=d_input_keys,
+    d_in_values=d_input_values,
+    d_out_keys=d_output_keys,
+    d_out_values=d_output_values,
+    op=OpKind.LESS,
+    num_items=len(h_input_keys),
+)
+
+# Allocate the temporary storage.
+d_temp_storage = cp.empty(temp_storage_size, dtype=np.uint8)
+
+# Perform the merge sort.
+sorter(
+    temp_storage=d_temp_storage,
+    d_in_keys=d_input_keys,
+    d_in_values=d_input_values,
+    d_out_keys=d_output_keys,
+    d_out_values=d_output_values,
+    op=OpKind.LESS,
+    num_items=len(h_input_keys),
+)
+
+# Verify the result.
+expected_keys = np.array([1, 2, 3, 4], dtype=dtype)
+expected_values = np.array([10, 20, 30, 40], dtype=dtype)
+actual_keys = d_output_keys.get()
+actual_values = d_output_values.get()
+np.testing.assert_array_equal(actual_keys, expected_keys)
+np.testing.assert_array_equal(actual_values, expected_values)
+print("Merge sort object example completed successfully")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/radix_sort_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/radix_sort_basic.py
new file mode 100644
index 00000000..d4bdfbb9
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/radix_sort_basic.py
@@ -0,0 +1,51 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use radix_sort to sort keys and values.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    SortOrder,
+)
+
+# Prepare the input and output arrays.
+h_in_keys = np.array([-5, 0, 2, -3, 2, 4, 0, -1, 2, 8], dtype="int32")
+h_in_values = np.array(
+    [-3.2, 2.2, 1.9, 4.0, -3.9, 2.7, 0, 8.3 - 1, 2.9, 5.4], dtype="float32"
+)
+
+d_in_keys = cp.asarray(h_in_keys)
+d_in_values = cp.asarray(h_in_values)
+
+# Prepare the output arrays.
+d_out_keys = cp.empty_like(d_in_keys)
+d_out_values = cp.empty_like(d_in_values)
+
+# Perform the radix sort.
+cuda.compute.radix_sort(
+    d_in_keys=d_in_keys,
+    d_out_keys=d_out_keys,
+    d_in_values=d_in_values,
+    d_out_values=d_out_values,
+    order=SortOrder.ASCENDING,
+    num_items=d_in_keys.size,
+)
+
+# Verify the result.
+h_out_keys = cp.asnumpy(d_out_keys)
+h_out_values = cp.asnumpy(d_out_values)
+
+argsort = np.argsort(h_in_keys, stable=True)
+expected_keys = np.array(h_in_keys)[argsort]
+expected_values = np.array(h_in_values)[argsort]
+
+assert np.array_equal(h_out_keys, expected_keys)
+assert np.array_equal(h_out_values, expected_values)
+print(f"Radix sort basic result - keys: {h_out_keys}, values: {h_out_values}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/radix_sort_buffer.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/radix_sort_buffer.py
new file mode 100644
index 00000000..ea3c93b5
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/radix_sort_buffer.py
@@ -0,0 +1,55 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use radix_sort with DoubleBuffer for reduced temporary storage.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    DoubleBuffer,
+    SortOrder,
+)
+
+# Prepare the input and output arrays.
+h_in_keys = np.array([-5, 0, 2, -3, 2, 4, 0, -1, 2, 8], dtype="int32")
+h_in_values = np.array(
+    [-3.2, 2.2, 1.9, 4.0, -3.9, 2.7, 0, 8.3 - 1, 2.9, 5.4], dtype="float32"
+)
+
+d_in_keys = cp.asarray(h_in_keys)
+d_in_values = cp.asarray(h_in_values)
+
+d_out_keys = cp.empty_like(d_in_keys)
+d_out_values = cp.empty_like(d_in_values)
+
+# Create the double buffer.
+keys_double_buffer = DoubleBuffer(d_in_keys, d_out_keys)
+values_double_buffer = DoubleBuffer(d_in_values, d_out_values)
+
+# Perform the radix sort.
+cuda.compute.radix_sort(
+    d_in_keys=keys_double_buffer,
+    d_out_keys=None,
+    d_in_values=values_double_buffer,
+    d_out_values=None,
+    order=SortOrder.ASCENDING,
+    num_items=d_in_keys.size,
+)
+
+# Verify the result.
+h_out_keys = cp.asnumpy(keys_double_buffer.current())
+h_out_values = cp.asnumpy(values_double_buffer.current())
+
+argsort = np.argsort(h_in_keys, stable=True)
+h_expected_keys = np.array(h_in_keys)[argsort]
+h_expected_values = np.array(h_in_values)[argsort]
+
+assert np.array_equal(h_out_keys, h_expected_keys)
+assert np.array_equal(h_out_values, h_expected_values)
+print(f"Radix sort buffer result - keys: {h_out_keys}, values: {h_out_values}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/radix_sort_object.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/radix_sort_object.py
new file mode 100644
index 00000000..7522520c
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/radix_sort_object.py
@@ -0,0 +1,64 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use radix_sort with the object API.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    SortOrder,
+)
+
+# Prepare the input and output arrays.
+dtype = np.int32
+h_input_keys = np.array([4, 2, 3, 1], dtype=dtype)
+h_input_values = np.array([40, 20, 30, 10], dtype=dtype)
+d_input_keys = cp.asarray(h_input_keys)
+d_input_values = cp.asarray(h_input_values)
+d_output_keys = cp.empty_like(d_input_keys)
+d_output_values = cp.empty_like(d_input_values)
+
+# Create the radix sort object.
+sorter = cuda.compute.make_radix_sort(
+    d_in_keys=d_input_keys,
+    d_out_keys=d_output_keys,
+    d_in_values=d_input_values,
+    d_out_values=d_output_values,
+    order=SortOrder.ASCENDING,
+)
+
+# Get the temporary storage size.
+temp_storage_size = sorter(
+    temp_storage=None,
+    d_in_keys=d_input_keys,
+    d_out_keys=d_output_keys,
+    d_in_values=d_input_values,
+    d_out_values=d_output_values,
+    num_items=len(h_input_keys),
+)
+d_temp_storage = cp.empty(temp_storage_size, dtype=np.uint8)
+
+# Perform the radix sort.
+sorter(
+    temp_storage=d_temp_storage,
+    d_in_keys=d_input_keys,
+    d_out_keys=d_output_keys,
+    d_in_values=d_input_values,
+    d_out_values=d_output_values,
+    num_items=len(h_input_keys),
+)
+
+# Verify the result.
+expected_keys = np.array([1, 2, 3, 4], dtype=dtype)
+expected_values = np.array([10, 20, 30, 40], dtype=dtype)
+actual_keys = d_output_keys.get()
+actual_values = d_output_values.get()
+np.testing.assert_array_equal(actual_keys, expected_keys)
+np.testing.assert_array_equal(actual_values, expected_values)
+print("Radix sort object example completed successfully")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/segmented_sort_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/segmented_sort_basic.py
new file mode 100644
index 00000000..82a1b779
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/segmented_sort_basic.py
@@ -0,0 +1,55 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use segmented_sort to sort keys and values within segments.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+# Prepare input keys and values, and segment offsets.
+h_in_keys = np.array([9, 1, 5, 4, 2, 8, 7, 3, 6], dtype="int32")
+h_in_vals = np.array([90, 10, 50, 40, 20, 80, 70, 30, 60], dtype="int32")
+
+# 3 segments: [0,3), [3,5), [5,9)
+start_offsets = np.array([0, 3, 5], dtype=np.int64)
+end_offsets = np.array([3, 5, 9], dtype=np.int64)
+
+d_in_keys = cp.asarray(h_in_keys)
+d_in_vals = cp.asarray(h_in_vals)
+d_out_keys = cp.empty_like(d_in_keys)
+d_out_vals = cp.empty_like(d_in_vals)
+
+# Perform the segmented sort (ascending within each segment).
+cuda.compute.segmented_sort(
+    d_in_keys=d_in_keys,
+    d_out_keys=d_out_keys,
+    d_in_values=d_in_vals,
+    d_out_values=d_out_vals,
+    num_items=d_in_keys.size,
+    num_segments=start_offsets.size,
+    start_offsets_in=cp.asarray(start_offsets),
+    end_offsets_in=cp.asarray(end_offsets),
+    order=cuda.compute.SortOrder.ASCENDING,
+)
+
+# Verify the result.
+h_out_keys = cp.asnumpy(d_out_keys)
+h_out_vals = cp.asnumpy(d_out_vals)
+
+expected_pairs = []
+for s, e in zip(start_offsets, end_offsets):
+    seg_pairs = sorted(zip(h_in_keys[s:e], h_in_vals[s:e]), key=lambda kv: kv[0])
+    expected_pairs.extend(seg_pairs)
+
+expected_keys = np.array([k for k, _ in expected_pairs], dtype=h_in_keys.dtype)
+expected_vals = np.array([v for _, v in expected_pairs], dtype=h_in_vals.dtype)
+
+assert np.array_equal(h_out_keys, expected_keys)
+assert np.array_equal(h_out_vals, expected_vals)
+print(f"Segmented sort basic result - keys: {h_out_keys}, values: {h_out_vals}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/segmented_sort_buffer.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/segmented_sort_buffer.py
new file mode 100644
index 00000000..c7522211
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/segmented_sort_buffer.py
@@ -0,0 +1,61 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use segmented_sort with DoubleBuffer for reduced temporary storage.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+# Prepare input keys and values, and segment offsets.
+h_in_keys = np.array([9, 1, 5, 4, 2, 8, 7, 3, 6], dtype="int32")
+h_in_vals = np.array([90, 10, 50, 40, 20, 80, 70, 30, 60], dtype="int32")
+
+# 3 segments: [0,3), [3,5), [5,9)
+start_offsets = np.array([0, 3, 5], dtype=np.int64)
+end_offsets = np.array([3, 5, 9], dtype=np.int64)
+
+d_in_keys = cp.asarray(h_in_keys)
+d_in_vals = cp.asarray(h_in_vals)
+d_tmp_keys = cp.empty_like(d_in_keys)
+d_tmp_vals = cp.empty_like(d_in_vals)
+
+# Create double buffers for keys and values.
+keys_db = cuda.compute.DoubleBuffer(d_in_keys, d_tmp_keys)
+vals_db = cuda.compute.DoubleBuffer(d_in_vals, d_tmp_vals)
+
+# Perform the segmented sort (descending within each segment).
+cuda.compute.segmented_sort(
+    d_in_keys=keys_db,
+    d_out_keys=None,
+    d_in_values=vals_db,
+    d_out_values=None,
+    num_items=d_in_keys.size,
+    num_segments=start_offsets.size,
+    start_offsets_in=cp.asarray(start_offsets),
+    end_offsets_in=cp.asarray(end_offsets),
+    order=cuda.compute.SortOrder.DESCENDING,
+)
+
+# Verify the result.
+h_out_keys = cp.asnumpy(keys_db.current())
+h_out_vals = cp.asnumpy(vals_db.current())
+
+expected_pairs = []
+for s, e in zip(start_offsets, end_offsets):
+    seg_pairs = sorted(
+        zip(h_in_keys[s:e], h_in_vals[s:e]), key=lambda kv: kv[0], reverse=True
+    )
+    expected_pairs.extend(seg_pairs)
+
+expected_keys = np.array([k for k, _ in expected_pairs], dtype=h_in_keys.dtype)
+expected_vals = np.array([v for _, v in expected_pairs], dtype=h_in_vals.dtype)
+
+assert np.array_equal(h_out_keys, expected_keys)
+assert np.array_equal(h_out_vals, expected_vals)
+print(f"Segmented sort buffer result - keys: {h_out_keys}, values: {h_out_vals}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/segmented_sort_object.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/segmented_sort_object.py
new file mode 100644
index 00000000..f515a17d
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/sort/segmented_sort_object.py
@@ -0,0 +1,78 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use segmented_sort with the object API.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+# Prepare the input and segment offsets.
+dtype = np.int32
+h_input_keys = np.array([9, 1, 5, 4, 2, 8, 7, 3, 6], dtype=dtype)
+h_input_vals = np.array([90, 10, 50, 40, 20, 80, 70, 30, 60], dtype=dtype)
+start_offsets = np.array([0, 3, 5], dtype=np.int64)
+end_offsets = np.array([3, 5, 9], dtype=np.int64)
+
+d_input_keys = cp.asarray(h_input_keys)
+d_input_vals = cp.asarray(h_input_vals)
+d_output_keys = cp.empty_like(d_input_keys)
+d_output_vals = cp.empty_like(d_input_vals)
+
+# Create the segmented sort object.
+sorter = cuda.compute.make_segmented_sort(
+    d_in_keys=d_input_keys,
+    d_out_keys=d_output_keys,
+    d_in_values=d_input_vals,
+    d_out_values=d_output_vals,
+    start_offsets_in=cp.asarray(start_offsets),
+    end_offsets_in=cp.asarray(end_offsets),
+    order=cuda.compute.SortOrder.ASCENDING,
+)
+
+# Get the temporary storage size.
+temp_storage_size = sorter(
+    temp_storage=None,
+    d_in_keys=d_input_keys,
+    d_out_keys=d_output_keys,
+    d_in_values=d_input_vals,
+    d_out_values=d_output_vals,
+    num_items=h_input_keys.size,
+    num_segments=start_offsets.size,
+    start_offsets_in=cp.asarray(start_offsets),
+    end_offsets_in=cp.asarray(end_offsets),
+)
+d_temp_storage = cp.empty(temp_storage_size, dtype=np.uint8)
+
+# Perform the segmented sort.
+sorter(
+    temp_storage=d_temp_storage,
+    d_in_keys=d_input_keys,
+    d_out_keys=d_output_keys,
+    d_in_values=d_input_vals,
+    d_out_values=d_output_vals,
+    num_items=h_input_keys.size,
+    num_segments=start_offsets.size,
+    start_offsets_in=cp.asarray(start_offsets),
+    end_offsets_in=cp.asarray(end_offsets),
+)
+
+# Verify the result.
+expected_pairs = []
+for s, e in zip(start_offsets, end_offsets):
+    seg_pairs = sorted(zip(h_input_keys[s:e], h_input_vals[s:e]), key=lambda kv: kv[0])
+    expected_pairs.extend(seg_pairs)
+
+expected_keys = np.array([k for k, _ in expected_pairs], dtype=dtype)
+expected_values = np.array([v for _, v in expected_pairs], dtype=dtype)
+
+actual_keys = d_output_keys.get()
+actual_values = d_output_vals.get()
+np.testing.assert_array_equal(actual_keys, expected_keys)
+np.testing.assert_array_equal(actual_values, expected_values)
+print("Segmented sort object example completed successfully")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/struct/__init__.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/struct/__init__.py
new file mode 100644
index 00000000..852281b2
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/struct/__init__.py
@@ -0,0 +1,3 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/struct/nested_struct_reduction.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/struct/nested_struct_reduction.py
new file mode 100644
index 00000000..ec42b03f
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/struct/nested_struct_reduction.py
@@ -0,0 +1,77 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example demonstrating reductions with nested gpu_struct types.
+
+This example shows how to define nested structs and use them in reduction
+operations. The reduction combines values from both the outer and inner
+struct fields.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import gpu_struct
+
+
+# Define an inner struct to hold coordinate data
+@gpu_struct
+class Point:
+    x: np.int32
+    y: np.int32
+
+
+# Define an outer struct that contains the inner struct
+@gpu_struct
+class Particle:
+    id: np.int64
+    position: Point
+
+
+def sum_particles(p1, p2):
+    """Reduction operation that sums all fields of two particles."""
+    return Particle(
+        p1.id + p2.id,
+        Point(p1.position.x + p2.position.x, p1.position.y + p2.position.y),
+    )
+
+
+# Prepare the input data
+num_items = 10
+h_data = np.zeros(num_items, dtype=Particle.dtype)
+for i in range(num_items):
+    h_data[i]["id"] = i * 10
+    h_data[i]["position"]["x"] = i
+    h_data[i]["position"]["y"] = i * 2
+
+# Copy to device
+d_input = cp.empty(num_items, dtype=Particle.dtype)
+d_input.set(h_data)
+
+# Prepare output and initial value
+d_output = cp.empty(1, dtype=Particle.dtype)
+h_init = Particle(0, Point(0, 0))
+
+# Perform the reduction
+cuda.compute.reduce_into(
+    d_in=d_input, d_out=d_output, num_items=num_items, op=sum_particles, h_init=h_init
+)
+
+# Verify the result
+result = d_output.get()[0]
+expected_id = sum(i * 10 for i in range(num_items))
+expected_x = sum(range(num_items))
+expected_y = sum(i * 2 for i in range(num_items))
+
+assert result["id"] == expected_id
+assert result["position"]["x"] == expected_x
+assert result["position"]["y"] == expected_y
+
+print("Nested struct reduction result:")
+print(f"  id: {result['id']} (expected: {expected_id})")
+print(f"  position.x: {result['position']['x']} (expected: {expected_x})")
+print(f"  position.y: {result['position']['y']} (expected: {expected_y})")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/struct/nested_struct_tuple_construction.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/struct/nested_struct_tuple_construction.py
new file mode 100644
index 00000000..623004ef
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/struct/nested_struct_tuple_construction.py
@@ -0,0 +1,82 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing tuple syntax for constructing nested gpu_struct types.
+
+When working with nested structs in device functions, you can use tuple syntax
+as a convenient shorthand for constructing the nested struct values. This can
+make code more concise while maintaining the same functionality.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import gpu_struct
+
+
+# Define nested structs
+@gpu_struct
+class Stats:
+    count: np.int32
+    sum: np.float32
+
+
+@gpu_struct
+class DataPoint:
+    value: np.int64
+    stats: Stats
+
+
+def sum_with_tuples(d1, d2):
+    """
+    Reduction operation using tuple syntax for nested struct construction.
+
+    Instead of writing: Stats(d1.stats.count + d2.stats.count, ...)
+    We can use tuple syntax: (d1.stats.count + d2.stats.count, ...)
+    """
+    return DataPoint(
+        d1.value + d2.value,
+        # Tuple syntax for constructing the nested Stats struct
+        (d1.stats.count + d2.stats.count, d1.stats.sum + d2.stats.sum),
+    )
+
+
+# Prepare the input data
+num_items = 10
+h_data = np.zeros(num_items, dtype=DataPoint.dtype)
+for i in range(num_items):
+    h_data[i]["value"] = i * 10
+    h_data[i]["stats"]["count"] = 1
+    h_data[i]["stats"]["sum"] = float(i)
+
+# Copy to device
+d_input = cp.empty(num_items, dtype=DataPoint.dtype)
+d_input.set(h_data)
+
+# Prepare output and initial value
+d_output = cp.empty(1, dtype=DataPoint.dtype)
+h_init = DataPoint(0, Stats(0, 0.0))
+
+# Perform the reduction
+cuda.compute.reduce_into(
+    d_in=d_input, d_out=d_output, num_items=num_items, op=sum_with_tuples, h_init=h_init
+)
+
+# Verify the result
+result = d_output.get()[0]
+expected_value = sum(i * 10 for i in range(num_items))
+expected_count = num_items
+expected_sum = sum(float(i) for i in range(num_items))
+
+assert result["value"] == expected_value
+assert result["stats"]["count"] == expected_count
+assert np.isclose(result["stats"]["sum"], expected_sum)
+
+print("Nested struct with tuple construction result:")
+print(f"  value: {result['value']} (expected: {expected_value})")
+print(f"  stats.count: {result['stats']['count']} (expected: {expected_count})")
+print(f"  stats.sum: {result['stats']['sum']:.2f} (expected: {expected_sum:.2f})")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/struct/nested_struct_zip_iterator.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/struct/nested_struct_zip_iterator.py
new file mode 100644
index 00000000..92234d3c
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/struct/nested_struct_zip_iterator.py
@@ -0,0 +1,98 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing ZipIterator with nested gpu_struct types.
+
+This example demonstrates combining separate arrays of nested structs using
+ZipIterator, then performing a reduction that operates on the combined data.
+This is useful when you have related data stored in separate arrays that need
+to be processed together.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import ZipIterator, gpu_struct
+
+
+# Define nested structs for geometric and color data
+@gpu_struct
+class Point:
+    x: np.int32
+    y: np.int32
+
+
+@gpu_struct
+class Color:
+    r: np.uint8
+    g: np.uint8
+    b: np.uint8
+
+
+@gpu_struct
+class Pixel:
+    position: Point
+    color: Color
+
+
+def sum_pixels(p1, p2):
+    """Reduction operation that sums all fields of two pixels."""
+    return Pixel(
+        Point(p1.position.x + p2.position.x, p1.position.y + p2.position.y),
+        Color(
+            p1.color.r + p2.color.r, p1.color.g + p2.color.g, p1.color.b + p2.color.b
+        ),
+    )
+
+
+# Prepare separate arrays for points and colors
+num_items = 100
+
+h_points = np.array([(i, i * 2) for i in range(num_items)], dtype=Point.dtype)
+h_colors = np.array(
+    [(i % 256, (i * 2) % 256, (i * 3) % 256) for i in range(num_items)],
+    dtype=Color.dtype,
+)
+
+d_points = cp.empty(num_items, dtype=Point.dtype)
+d_points.set(h_points)
+
+d_colors = cp.empty(num_items, dtype=Color.dtype)
+d_colors.set(h_colors)
+
+# Create a zip iterator to combine the points and colors
+zip_it = ZipIterator(d_points, d_colors)
+
+# Prepare output and initial value
+d_output = cp.empty(1, dtype=Pixel.dtype)
+h_init = Pixel(Point(0, 0), Color(0, 0, 0))
+
+# Perform the reduction on the zipped data
+cuda.compute.reduce_into(
+    d_in=zip_it, d_out=d_output, num_items=num_items, op=sum_pixels, h_init=h_init
+)
+
+# Verify the result
+result = d_output.get()[0]
+expected_x = sum(range(num_items))
+expected_y = sum(i * 2 for i in range(num_items))
+expected_r = sum(i % 256 for i in range(num_items)) % 256
+expected_g = sum((i * 2) % 256 for i in range(num_items)) % 256
+expected_b = sum((i * 3) % 256 for i in range(num_items)) % 256
+
+assert result["position"]["x"] == expected_x
+assert result["position"]["y"] == expected_y
+assert result["color"]["r"] == expected_r
+assert result["color"]["g"] == expected_g
+assert result["color"]["b"] == expected_b
+
+print("Nested struct with ZipIterator result:")
+print(f"  position.x: {result['position']['x']} (expected: {expected_x})")
+print(f"  position.y: {result['position']['y']} (expected: {expected_y})")
+print(f"  color.r: {result['color']['r']} (expected: {expected_r})")
+print(f"  color.g: {result['color']['g']} (expected: {expected_g})")
+print(f"  color.b: {result['color']['b']} (expected: {expected_b})")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/struct/struct_reduction.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/struct/struct_reduction.py
new file mode 100644
index 00000000..a15285fc
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/struct/struct_reduction.py
@@ -0,0 +1,50 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Finding the maximum green value in a sequence of pixels using `reduce_into`
+with a custom data type.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import gpu_struct
+
+
+# Define a custom data type to store the pixel values.
+@gpu_struct
+class Pixel:
+    r: np.int32
+    g: np.int32
+    b: np.int32
+
+
+# Define a reduction operation that returns the pixel with the maximum green value.
+def max_g_value(x, y):
+    return x if x.g > y.g else y
+
+
+# Prepare the input and output arrays.
+d_rgb = cp.random.randint(0, 256, (10, 3), dtype=np.int32).view(Pixel.dtype)
+d_out = cp.empty(1, Pixel.dtype)
+
+# Prepare the initial value for the reduction.
+h_init = Pixel(0, 0, 0)
+
+# Perform the reduction.
+cuda.compute.reduce_into(
+    d_in=d_rgb, d_out=d_out, num_items=d_rgb.size, op=max_g_value, h_init=h_init
+)
+
+# Calculate the expected result.
+h_rgb = d_rgb.get()
+expected = h_rgb[h_rgb.view("int32")[:, 1].argmax()]
+
+# Verify the result.
+assert expected["g"] == d_out.get()["g"]
+result = d_out.get()
+print(f"Pixel reduction result: {result}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/struct/struct_transform.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/struct/struct_transform.py
new file mode 100644
index 00000000..218a10fe
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/struct/struct_transform.py
@@ -0,0 +1,60 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example demonstrating binary_transform with custom struct types.
+
+When working with struct inputs in transform operations, you need to provide
+type annotations to help Numba infer the correct types. Unlike reduce_into
+which can infer types from h_init, transform operations require explicit
+annotations when using struct inputs.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import gpu_struct
+
+
+@gpu_struct
+class Point2D:
+    x: np.float32
+    y: np.float32
+
+
+def add_points(p1: Point2D, p2: Point2D) -> Point2D:
+    return Point2D(p1.x + p2.x, p1.y + p2.y)
+
+
+num_items = 1000
+
+h_in1 = np.empty(num_items, dtype=Point2D.dtype)
+h_in1["x"] = np.random.rand(num_items).astype(np.float32)
+h_in1["y"] = np.random.rand(num_items).astype(np.float32)
+
+h_in2 = np.empty(num_items, dtype=Point2D.dtype)
+h_in2["x"] = np.random.rand(num_items).astype(np.float32)
+h_in2["y"] = np.random.rand(num_items).astype(np.float32)
+
+d_in1 = cp.empty_like(h_in1)
+d_in1.set(h_in1)
+
+d_in2 = cp.empty_like(h_in2)
+d_in2.set(h_in2)
+
+d_out = cp.empty_like(d_in1)
+
+cuda.compute.binary_transform(
+    d_in1=d_in1, d_in2=d_in2, d_out=d_out, op=add_points, num_items=num_items
+)
+
+result = d_out.get()
+
+np.testing.assert_allclose(result["x"], h_in1["x"] + h_in2["x"], rtol=1e-5)
+np.testing.assert_allclose(result["y"], h_in1["y"] + h_in2["y"], rtol=1e-5)
+
+print("Binary transform with structs completed successfully")
+print(f"First result point: x={result[0]['x']:.4f}, y={result[0]['y']:.4f}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/transform/binary_transform_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/transform/binary_transform_basic.py
new file mode 100644
index 00000000..371ae55c
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/transform/binary_transform_basic.py
@@ -0,0 +1,35 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use binary_transform to perform elementwise addition.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    OpKind,
+)
+
+# Prepare the input and output arrays.
+input1_data = np.array([1, 2, 3, 4], dtype=np.int32)
+input2_data = np.array([10, 20, 30, 40], dtype=np.int32)
+d_in1 = cp.asarray(input1_data)
+d_in2 = cp.asarray(input2_data)
+d_out = cp.empty_like(d_in1)
+
+# Perform the binary transform.
+cuda.compute.binary_transform(
+    d_in1=d_in1, d_in2=d_in2, d_out=d_out, op=OpKind.PLUS, num_items=len(d_in1)
+)
+
+# Verify the result.
+result = d_out.get()
+expected = input1_data + input2_data
+
+np.testing.assert_array_equal(result, expected)
+print(f"Binary transform result: {result}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/transform/binary_transform_object.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/transform/binary_transform_object.py
new file mode 100644
index 00000000..cf86dbf8
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/transform/binary_transform_object.py
@@ -0,0 +1,44 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Binary transform examples demonstrating the transform object API.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    OpKind,
+)
+
+# Prepare the input and output arrays.
+dtype = np.int32
+h_input1 = np.array([1, 2, 3, 4], dtype=dtype)
+h_input2 = np.array([10, 20, 30, 40], dtype=dtype)
+d_input1 = cp.asarray(h_input1)
+d_input2 = cp.asarray(h_input2)
+d_output = cp.empty_like(d_input1)
+
+# Create the binary transform object.
+transformer = cuda.compute.make_binary_transform(
+    d_in1=d_input1, d_in2=d_input2, d_out=d_output, op=OpKind.PLUS
+)
+
+# Perform the binary transform.
+transformer(
+    d_in1=d_input1,
+    d_in2=d_input2,
+    d_out=d_output,
+    op=OpKind.PLUS,
+    num_items=len(h_input1),
+)
+
+# Verify the result.
+expected_result = np.array([11, 22, 33, 44], dtype=dtype)
+actual_result = d_output.get()
+np.testing.assert_array_equal(actual_result, expected_result)
+print("Binary transform object example completed successfully")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/transform/unary_transform_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/transform/unary_transform_basic.py
new file mode 100644
index 00000000..d405a2b9
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/transform/unary_transform_basic.py
@@ -0,0 +1,34 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use unary_transform to apply a unary operation to each element.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+# Prepare the input and output arrays.
+input_data = np.array([1, 2, 3, 4, 5], dtype=np.int32)
+d_in = cp.asarray(input_data)
+d_out = cp.empty_like(d_in)
+
+
+# Define the unary operation.
+def op(a):
+    return a + 1
+
+
+# Perform the unary transform.
+cuda.compute.unary_transform(d_in=d_in, d_out=d_out, op=op, num_items=len(d_in))
+
+# Verify the result.
+result = d_out.get()
+expected = input_data + 1
+
+np.testing.assert_array_equal(result, expected)
+print(f"Unary transform result: {result}")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/transform/unary_transform_object.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/transform/unary_transform_object.py
new file mode 100644
index 00000000..c1605266
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/transform/unary_transform_object.py
@@ -0,0 +1,39 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Unary transform examples demonstrating the object API and well-known operations.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+
+# Prepare the input and output arrays.
+dtype = np.int32
+h_input = np.array([1, 2, 3, 4], dtype=dtype)
+d_input = cp.asarray(h_input)
+d_output = cp.empty_like(d_input)
+
+
+# Define the unary operation.
+def add_one_op(a):
+    return a + 1
+
+
+# Create the unary transform object.
+transformer = cuda.compute.make_unary_transform(
+    d_in=d_input, d_out=d_output, op=add_one_op
+)
+
+# Perform the unary transform.
+transformer(d_in=d_input, d_out=d_output, op=add_one_op, num_items=len(h_input))
+
+# Verify the result.
+expected_result = np.array([2, 3, 4, 5], dtype=dtype)
+actual_result = d_output.get()
+np.testing.assert_array_equal(actual_result, expected_result)
+print("Unary transform object example completed successfully")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/unique/__init__.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/unique/__init__.py
new file mode 100644
index 00000000..ca2aa8df
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/unique/__init__.py
@@ -0,0 +1,5 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Parallel unique algorithms examples package."""
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/unique/unique_by_key_basic.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/unique/unique_by_key_basic.py
new file mode 100644
index 00000000..1ffc4c30
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/unique/unique_by_key_basic.py
@@ -0,0 +1,52 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use unique_by_key to remove all
+but the first value for each group of consecutive keys.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    OpKind,
+)
+
+# Prepare the input and output arrays.
+h_in_keys = np.array([0, 2, 2, 9, 5, 5, 5, 8], dtype="int32")
+h_in_values = np.array([1, 2, 3, 4, 5, 6, 7, 8], dtype="float32")
+
+d_in_keys = cp.asarray(h_in_keys)
+d_in_values = cp.asarray(h_in_values)
+d_out_keys = cp.empty_like(d_in_keys)
+d_out_values = cp.empty_like(d_in_values)
+d_out_num_selected = cp.empty(1, np.int32)
+
+# Perform the unique by key operation.
+cuda.compute.unique_by_key(
+    d_in_keys=d_in_keys,
+    d_in_items=d_in_values,
+    d_out_keys=d_out_keys,
+    d_out_items=d_out_values,
+    d_out_num_selected=d_out_num_selected,
+    op=OpKind.EQUAL_TO,
+    num_items=d_in_keys.size,
+)
+
+# Verify the result.
+num_selected = cp.asnumpy(d_out_num_selected)[0]
+h_out_keys = cp.asnumpy(d_out_keys)[:num_selected]
+h_out_values = cp.asnumpy(d_out_values)[:num_selected]
+
+expected_keys = np.array([0, 2, 9, 5, 8])
+expected_values = np.array([1, 2, 4, 5, 8])
+
+assert np.array_equal(h_out_keys, expected_keys)
+assert np.array_equal(h_out_values, expected_values)
+print(
+    f"Unique by key basic result - keys: {h_out_keys}, values: {h_out_values}, count: {num_selected}"
+)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/examples/unique/unique_by_key_object.py b/cccl_upstream/python/cuda_cccl/tests/compute/examples/unique/unique_by_key_object.py
new file mode 100644
index 00000000..a087738d
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/examples/unique/unique_by_key_object.py
@@ -0,0 +1,73 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# example-begin
+"""
+Example showing how to use unique_by_key with the object API.
+"""
+
+import cupy as cp
+import numpy as np
+
+import cuda.compute
+from cuda.compute import (
+    OpKind,
+)
+
+# Unique by key example demonstrating the object API
+dtype = np.int32
+h_input_keys = np.array([1, 1, 2, 3, 3], dtype=dtype)
+h_input_values = np.array([10, 20, 30, 40, 50], dtype=dtype)
+d_input_keys = cp.asarray(h_input_keys)
+d_input_values = cp.asarray(h_input_values)
+d_output_keys = cp.empty_like(d_input_keys)
+d_output_values = cp.empty_like(d_input_values)
+d_num_selected = cp.empty(1, dtype=np.int32)
+
+# Create the unique by key object.
+uniquer = cuda.compute.make_unique_by_key(
+    d_in_keys=d_input_keys,
+    d_in_items=d_input_values,
+    d_out_keys=d_output_keys,
+    d_out_items=d_output_values,
+    d_out_num_selected=d_num_selected,
+    op=OpKind.EQUAL_TO,
+)
+
+# Get the temporary storage size.
+temp_storage_size = uniquer(
+    temp_storage=None,
+    d_in_keys=d_input_keys,
+    d_in_items=d_input_values,
+    d_out_keys=d_output_keys,
+    d_out_items=d_output_values,
+    d_out_num_selected=d_num_selected,
+    op=OpKind.EQUAL_TO,
+    num_items=len(h_input_keys),
+)
+
+# Allocate the temporary storage.
+d_temp_storage = cp.empty(temp_storage_size, dtype=np.uint8)
+
+# Perform the unique by key operation.
+uniquer(
+    temp_storage=d_temp_storage,
+    d_in_keys=d_input_keys,
+    d_in_items=d_input_values,
+    d_out_keys=d_output_keys,
+    d_out_items=d_output_values,
+    d_out_num_selected=d_num_selected,
+    op=OpKind.EQUAL_TO,
+    num_items=len(h_input_keys),
+)
+
+# Verify the result.
+num_selected = d_num_selected.get()[0]
+expected_keys = np.array([1, 2, 3], dtype=dtype)
+expected_values = np.array([10, 30, 40], dtype=dtype)
+actual_keys = d_output_keys.get()[:num_selected]
+actual_values = d_output_values.get()[:num_selected]
+np.testing.assert_array_equal(actual_keys, expected_keys)
+np.testing.assert_array_equal(actual_values, expected_values)
+print("Unique by key object example completed successfully")
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_binary_search.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_binary_search.py
new file mode 100644
index 00000000..9d136025
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_binary_search.py
@@ -0,0 +1,287 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray
+
+import cuda.compute
+from cuda.compute import (
+    OpKind,
+    deserialize,
+    make_lower_bound,
+    make_upper_bound,
+    serialize,
+)
+
+DTYPE_LIST = [
+    np.int32,
+    np.int64,
+    np.uint32,
+    np.uint64,
+    np.float32,
+    np.float64,
+]
+
+
+def random_sorted_array(size, dtype, max_value=1000):
+    rng = np.random.default_rng()
+    if np.isdtype(dtype, "integral"):
+        data = rng.integers(max_value, size=size, dtype=dtype)
+    else:
+        if dtype == np.float16:  # pragma: no cover - float16 not used here
+            data = rng.random(size=size, dtype=np.float32).astype(dtype)
+        else:
+            data = rng.random(size=size, dtype=dtype)
+    data.sort()
+    return data
+
+
+@pytest.fixture(scope="function", autouse=True)
+def disable_sass_check(monkeypatch):
+    monkeypatch.setattr(
+        cuda.compute._cccl_interop,
+        "_check_sass",
+        False,
+    )
+
+
+@pytest.mark.parametrize(
+    "search, side",
+    [
+        (cuda.compute.lower_bound, "left"),
+        (cuda.compute.upper_bound, "right"),
+    ],
+)
+def test_binary_search_explicit_opkind_less(search, side):
+    h_data = np.array([1, 3, 3, 7, 9], dtype=np.int32)
+    h_values = np.array([0, 3, 4, 10], dtype=np.int32)
+
+    d_data = DeviceArray.from_numpy(h_data)
+    d_values = DeviceArray.from_numpy(h_values)
+    d_out = DeviceArray.empty(len(h_values), np.uintp)
+
+    search(
+        d_data=d_data,
+        num_items=len(h_data),
+        d_values=d_values,
+        num_values=len(h_values),
+        d_out=d_out,
+        comp=OpKind.LESS,
+    )
+
+    expected = np.searchsorted(h_data, h_values, side=side).astype(np.uintp)
+    np.testing.assert_array_equal(d_out.copy_to_host(), expected)
+
+
+@pytest.mark.parametrize(
+    "search, side",
+    [
+        (cuda.compute.lower_bound, "left"),
+        (cuda.compute.upper_bound, "right"),
+    ],
+)
+def test_binary_search_custom_comparator(search, side):
+    h_data = np.array([9, 7, 3, 3, 1], dtype=np.int32)
+    h_values = np.array([10, 4, 3, 0], dtype=np.int32)
+
+    def greater(lhs, rhs):
+        return lhs > rhs
+
+    d_data = DeviceArray.from_numpy(h_data)
+    d_values = DeviceArray.from_numpy(h_values)
+    d_out = DeviceArray.empty(len(h_values), np.uintp)
+
+    search(
+        d_data=d_data,
+        num_items=len(h_data),
+        d_values=d_values,
+        num_values=len(h_values),
+        d_out=d_out,
+        comp=greater,
+    )
+
+    expected = np.searchsorted(-h_data, -h_values, side=side).astype(np.uintp)
+    np.testing.assert_array_equal(d_out.copy_to_host(), expected)
+
+
+@pytest.mark.parametrize("dtype", DTYPE_LIST)
+@pytest.mark.parametrize(
+    "num_items,num_values", [(0, 0), (0, 128), (128, 0), (512, 128)]
+)
+def test_lower_bound_basic(dtype, num_items, num_values):
+    h_data = random_sorted_array(num_items, dtype)
+    h_values = random_sorted_array(num_values, dtype)
+
+    d_data = DeviceArray.from_numpy(h_data)
+    d_values = DeviceArray.from_numpy(h_values)
+    d_out = DeviceArray.empty(num_values, np.uintp)
+
+    cuda.compute.lower_bound(
+        d_data=d_data,
+        num_items=num_items,
+        d_values=d_values,
+        num_values=num_values,
+        d_out=d_out,
+    )
+
+    expected = np.searchsorted(h_data, h_values, side="left").astype(np.uintp)
+    got = d_out.copy_to_host()
+    assert np.array_equal(got, expected)
+
+
+@pytest.mark.parametrize("dtype", DTYPE_LIST)
+@pytest.mark.parametrize(
+    "num_items,num_values", [(0, 0), (0, 128), (128, 0), (512, 128)]
+)
+def test_upper_bound_basic(dtype, num_items, num_values):
+    h_data = random_sorted_array(num_items, dtype)
+    h_values = random_sorted_array(num_values, dtype)
+
+    d_data = DeviceArray.from_numpy(h_data)
+    d_values = DeviceArray.from_numpy(h_values)
+    d_out = DeviceArray.empty(num_values, np.uintp)
+
+    cuda.compute.upper_bound(
+        d_data=d_data,
+        num_items=num_items,
+        d_values=d_values,
+        num_values=num_values,
+        d_out=d_out,
+    )
+
+    expected = np.searchsorted(h_data, h_values, side="right").astype(np.uintp)
+    got = d_out.copy_to_host()
+    assert np.array_equal(got, expected)
+
+
+@pytest.mark.parametrize("dtype", DTYPE_LIST)
+def test_binary_search_with_duplicates(dtype):
+    rng = np.random.default_rng()
+    h_data = (
+        rng.integers(10, size=1024, dtype=dtype)
+        if np.isdtype(dtype, "integral")
+        else rng.random(1024, dtype=dtype)
+    )
+    h_data.sort()
+    h_values = (
+        rng.integers(10, size=128, dtype=dtype)
+        if np.isdtype(dtype, "integral")
+        else rng.random(128, dtype=dtype)
+    )
+
+    d_data = DeviceArray.from_numpy(h_data)
+    d_values = DeviceArray.from_numpy(h_values)
+    d_out = DeviceArray.empty(len(h_values), np.uintp)
+
+    cuda.compute.lower_bound(
+        d_data=d_data,
+        num_items=len(h_data),
+        d_values=d_values,
+        num_values=len(h_values),
+        d_out=d_out,
+    )
+    expected = np.searchsorted(h_data, h_values, side="left").astype(np.uintp)
+    got = d_out.copy_to_host()
+    assert np.array_equal(got, expected)
+
+    cuda.compute.upper_bound(
+        d_data=d_data,
+        num_items=len(h_data),
+        d_values=d_values,
+        num_values=len(h_values),
+        d_out=d_out,
+    )
+    expected = np.searchsorted(h_data, h_values, side="right").astype(np.uintp)
+    got = d_out.copy_to_host()
+    assert np.array_equal(got, expected)
+
+
+def test_binary_search_requires_unsigned_output():
+    """Output must be unsigned integer dtype for indices."""
+    h_data = np.array([1, 2, 3, 4], dtype=np.int32)
+    h_values = np.array([2, 3], dtype=np.int32)
+    d_data = DeviceArray.from_numpy(h_data)
+    d_values = DeviceArray.from_numpy(h_values)
+    d_out = DeviceArray.empty(len(h_values), np.int32)  # signed, should fail
+
+    with pytest.raises(TypeError, match="unsigned integer"):
+        cuda.compute.lower_bound(
+            d_data=d_data,
+            num_items=len(h_data),
+            d_values=d_values,
+            num_values=len(h_values),
+            d_out=d_out,
+        )
+
+
+def test_binary_search_requires_pointer_sized_output():
+    """Output must be pointer-sized (np.uintp) to hold any valid index."""
+    h_data = np.array([1, 2, 3, 4], dtype=np.int32)
+    h_values = np.array([2, 3], dtype=np.int32)
+    d_data = DeviceArray.from_numpy(h_data)
+    d_values = DeviceArray.from_numpy(h_values)
+    d_out = DeviceArray.empty(
+        len(h_values), np.uint32
+    )  # unsigned but not pointer-sized (on 64-bit)
+
+    with pytest.raises(ValueError, match="pointer-sized"):
+        cuda.compute.lower_bound(
+            d_data=d_data,
+            num_items=len(h_data),
+            d_values=d_values,
+            num_values=len(h_values),
+            d_out=d_out,
+        )
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_lower_bound_round_trip():
+    h_data = np.array([1, 3, 3, 5, 7, 9], dtype=np.int32)
+    h_values = np.array([0, 3, 4, 10], dtype=np.int32)
+    d_data = DeviceArray.from_numpy(h_data)
+    d_values = DeviceArray.from_numpy(h_values)
+    d_out = DeviceArray.empty(len(h_values), np.uintp)
+
+    builder = make_lower_bound(d_data=d_data, d_values=d_values, d_out=d_out)
+    blob = serialize(builder)
+    assert len(blob) > 0
+
+    loaded = deserialize(blob)
+    loaded(
+        d_data=d_data,
+        num_items=len(d_data),
+        d_values=d_values,
+        num_values=len(d_values),
+        d_out=d_out,
+        comp=None,
+    )
+
+    expected = np.searchsorted(h_data, h_values, side="left").astype(np.uintp)
+    np.testing.assert_array_equal(d_out.copy_to_host(), expected)
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_upper_bound_round_trip():
+    h_data = np.array([1, 3, 3, 5, 7, 9], dtype=np.int32)
+    h_values = np.array([0, 3, 4, 10], dtype=np.int32)
+    d_data = DeviceArray.from_numpy(h_data)
+    d_values = DeviceArray.from_numpy(h_values)
+    d_out = DeviceArray.empty(len(h_values), np.uintp)
+
+    builder = make_upper_bound(d_data=d_data, d_values=d_values, d_out=d_out)
+    blob = serialize(builder)
+    assert len(blob) > 0
+
+    loaded = deserialize(blob)
+    loaded(
+        d_data=d_data,
+        num_items=len(d_data),
+        d_values=d_values,
+        num_values=len(d_values),
+        d_out=d_out,
+        comp=None,
+    )
+
+    expected = np.searchsorted(h_data, h_values, side="right").astype(np.uintp)
+    np.testing.assert_array_equal(d_out.copy_to_host(), expected)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_bindings.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_bindings.py
new file mode 100644
index 00000000..2f16da62
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_bindings.py
@@ -0,0 +1,165 @@
+import ctypes
+
+import pytest
+
+import cuda.compute._bindings as bindings
+
+
+@pytest.fixture(
+    params=[
+        "INT8",
+        "INT16",
+        "INT32",
+        "INT64",
+        "UINT8",
+        "UINT16",
+        "UINT32",
+        "UINT64",
+        "FLOAT32",
+        "FLOAT64",
+        "STORAGE",
+    ]
+)
+def cccl_type_enum(request):
+    return getattr(bindings.TypeEnum, request.param)
+
+
+@pytest.fixture(params=["STATEFUL", "STATELESS"])
+def cccl_op_kind(request):
+    return getattr(bindings.OpKind, request.param)
+
+
+@pytest.fixture(params=["POINTER", "ITERATOR"])
+def cccl_iterator_kind(request):
+    return getattr(bindings.IteratorKind, request.param)
+
+
+def test_TypeEnum_positive(cccl_type_enum):
+    assert isinstance(cccl_type_enum, bindings.TypeEnum)
+    assert isinstance(cccl_type_enum.value, int)
+
+
+def test_TypeEnum_negative(cccl_iterator_kind):
+    assert not isinstance(cccl_iterator_kind, bindings.TypeEnum)
+
+
+def test_OpKind_positive(cccl_op_kind):
+    assert isinstance(cccl_op_kind, bindings.OpKind)
+    assert isinstance(cccl_op_kind.value, int)
+
+
+def test_OpKind_negative(cccl_iterator_kind):
+    assert not isinstance(cccl_iterator_kind, bindings.OpKind)
+
+
+def test_IteratorKind_positive(cccl_iterator_kind):
+    assert isinstance(cccl_iterator_kind, bindings.IteratorKind)
+    assert isinstance(cccl_iterator_kind.value, int)
+
+
+def test_IteratorKind_negative(cccl_type_enum):
+    assert not isinstance(cccl_type_enum, bindings.IteratorKind)
+
+
+def test_Op_default():
+    res = bindings.Op()
+    assert isinstance(res, bindings.Op)
+
+
+def test_Op_state_setter():
+    res = bindings.Op()
+    bytes = b"\x00" * 20
+    res.state = bytes
+
+
+def test_Op_state_getter():
+    res = bindings.Op()
+    assert isinstance(res.state, bytes)
+
+
+def test_Op_params_stateless():
+    fake_ltoir = b"\x00" * 127
+    res = bindings.Op(
+        name="fn", operator_type=bindings.OpKind.STATELESS, ltoir=fake_ltoir
+    )
+
+    assert isinstance(res, bindings.Op)
+
+
+def test_Op_params_stateful():
+    fake_ltoir = b"\x42" * 127
+    fake_state = b"\x01" * 16
+    res = bindings.Op(
+        name="fn",
+        operator_type=bindings.OpKind.STATEFUL,
+        ltoir=fake_ltoir,
+        state=fake_state,
+        state_alignment=16,
+    )
+
+    assert isinstance(res, bindings.Op)
+    assert res.state == fake_state
+    assert res.ltoir == fake_ltoir
+
+
+def test_TypeInfo_ctor(cccl_type_enum):
+    ti = bindings.TypeInfo(4, 1, cccl_type_enum)
+    assert isinstance(ti, bindings.TypeInfo)
+
+
+def test_TypeInfo_validate(cccl_type_enum):
+    # size must positive
+    with pytest.raises(ValueError):
+        bindings.TypeInfo(0, 1, bindings.TypeEnum.FLOAT32)
+
+    # alignment must be positive, power of two
+    with pytest.raises(ValueError):
+        bindings.TypeInfo(8, 3, bindings.TypeEnum.FLOAT32)
+
+
+def test_Value_ctor():
+    ti = bindings.TypeInfo(64, 64, bindings.TypeEnum.UINT64)
+    state = bytearray(ctypes.c_uint64(2**63 + 17))
+    v = bindings.Value(ti, state)
+    assert isinstance(v, bindings.Value)
+
+
+def test_Iterator_ctor1():
+    fake_ptr = 42
+    type_info = bindings.TypeInfo(32, 32, bindings.TypeEnum.INT32)
+    cccl_it = bindings.Iterator(
+        1,  # state alignment
+        bindings.IteratorKind.POINTER,
+        bindings.Op(),
+        bindings.Op(),
+        type_info,
+        bindings.Pointer(fake_ptr),
+    )
+    assert isinstance(cccl_it, bindings.Iterator)
+
+
+def test_Iterator_ctor2():
+    fake_ptr = 42
+    type_info = bindings.TypeInfo(32, 32, bindings.TypeEnum.INT32)
+    cccl_it = bindings.Iterator(
+        1,  # state alignment
+        bindings.IteratorKind.ITERATOR,
+        bindings.Op(),
+        bindings.Op(),
+        type_info,
+        bindings.IteratorState(
+            ctypes.c_void_p(fake_ptr),
+        ),
+    )
+    assert isinstance(cccl_it, bindings.Iterator)
+
+
+def test_CommonData():
+    cub_path = "/example/path/to/cub/includes"
+    thrust_path = "/example/path/to/thrust/includes"
+    libcudacxx_path = "/example/path/to/libcudacxx/includes"
+    gtk_path = "/usr/local/cuda/include"
+    common_data = bindings.CommonData(
+        8, 6, cub_path, thrust_path, libcudacxx_path, gtk_path
+    )
+    assert isinstance(common_data, bindings.CommonData)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_deferred_annotations.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_deferred_annotations.py
new file mode 100644
index 00000000..757c5d6e
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_deferred_annotations.py
@@ -0,0 +1,43 @@
+# Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+import numpy as np
+from _utils.device_array import DeviceArray
+
+from cuda.compute import OpKind, TransformIterator, gpu_struct, reduce_into
+
+
+def test_deferred_annotations():
+    # test that we can use @gpu_struct with deferred annotations
+    # GH: #6421
+
+    @gpu_struct
+    class MyStruct:
+        x: np.int32
+        y: np.int32
+
+
+def test_transform_iterator_future_annotations():
+    def add_one(x: "np.int32") -> "np.int32":
+        return x + np.int32(1)
+
+    h_in = np.arange(8, dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(1, np.int32)
+    h_init = np.array([0], dtype=np.int32)
+
+    transform_it = TransformIterator(d_in, add_one)
+    reduce_into(
+        d_in=transform_it,
+        d_out=d_out,
+        num_items=h_in.size,
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+
+    expected = int(np.sum(h_in + 1))
+    assert int(d_out.copy_to_host()[0]) == expected
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_free_threading_stress.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_free_threading_stress.py
new file mode 100644
index 00000000..56298d0f
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_free_threading_stress.py
@@ -0,0 +1,1671 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from __future__ import annotations
+
+import concurrent.futures
+import sys
+import sysconfig
+import threading
+from dataclasses import dataclass
+from typing import Callable
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray
+
+from cuda.core import Device
+
+pytestmark = [
+    pytest.mark.free_threading,
+    pytest.mark.no_numba,
+    pytest.mark.no_verify_sass(
+        reason="Free-threading stress tests intentionally run concurrent workers."
+    ),
+]
+
+STRESS_ITERATIONS = 10
+# Four workers give the single-flight build path multiple simultaneous waiters
+# and richer interleavings than a single winner/waiter pair.
+STRESS_THREADS = 4
+TRANSFORM_NATIVE_CACHE_THREADS = 4
+# Each iteration compiles one distinct specialization per worker, so iterations
+# are capped well below STRESS_ITERATIONS to bound native-compile time.
+DISTINCT_KEY_STORM_ITERATIONS = 3
+
+
+def _is_free_threaded_build() -> bool:
+    return sysconfig.get_config_var("Py_GIL_DISABLED") in (1, "1")
+
+
+def _assert_gil_disabled(where: str) -> None:
+    is_gil_enabled = getattr(sys, "_is_gil_enabled", None)
+    if is_gil_enabled is not None and is_gil_enabled():
+        pytest.fail(f"the GIL is enabled {where}")
+
+
+def _require_free_threaded_python() -> None:
+    if not _is_free_threaded_build():
+        pytest.skip("requires a free-threaded CPython build")
+    _assert_gil_disabled("at free-threaded test start")
+
+
+def _require_serialization_backend() -> None:
+    from cuda.compute._build_info import USING_V2
+
+    if USING_V2:
+        pytest.skip("serialization is not supported by the C Parallel v2 backend")
+
+
+@pytest.fixture
+def compute_module():
+    _require_free_threaded_python()
+
+    import cuda.compute as cc
+
+    _assert_gil_disabled("after importing cuda.compute")
+    cc.clear_all_caches()
+    try:
+        yield cc
+    finally:
+        cc.clear_all_caches()
+
+
+def _make_stream():
+    # cuda.core streams implement __cuda_stream__, so one object serves both
+    # as the allocation/synchronization handle and the per-call stream arg.
+    device = Device()
+    device.set_current()
+    return device.create_stream()
+
+
+def _run_threaded(workers: list[Callable[[threading.Barrier], None]]) -> None:
+    # The default timeout turns a worker that dies before reaching the barrier
+    # into a BrokenBarrierError in its peers instead of hanging the CI job.
+    barrier = threading.Barrier(len(workers), timeout=60)
+    with concurrent.futures.ThreadPoolExecutor(max_workers=len(workers)) as executor:
+        futures = [executor.submit(worker, barrier) for worker in workers]
+        errors = []
+        for future in futures:
+            try:
+                future.result()
+            except BaseException as exc:  # noqa: BLE001
+                errors.append(exc)
+        if errors:
+            # Surface the root-cause worker failure, not the barrier breakage
+            # it caused in the other workers.
+            raise next(
+                (
+                    error
+                    for error in errors
+                    if not isinstance(error, threading.BrokenBarrierError)
+                ),
+                errors[0],
+            )
+    _assert_gil_disabled("after concurrent cuda.compute operations")
+
+
+def _call_with_temp(algorithm, **kwargs):
+    temp_storage_bytes = algorithm(temp_storage=None, **kwargs)
+    temp_storage = DeviceArray.empty(temp_storage_bytes, dtype=np.uint8)
+    return algorithm(temp_storage=temp_storage, **kwargs)
+
+
+def _get_build_result(algorithm):
+    if hasattr(algorithm, "build_results"):
+        assert len(algorithm.build_results) == 1
+        return next(iter(algorithm.build_results.values()))
+    if hasattr(algorithm, "build_result"):
+        return algorithm.build_result
+    if hasattr(algorithm, "partitioner"):
+        return _get_build_result(algorithm.partitioner)
+    raise AssertionError(f"{type(algorithm).__name__} does not expose a build result")
+
+
+@dataclass(frozen=True)
+class _AlgorithmCase:
+    name: str
+    make_shared: Callable
+    make_worker: Callable
+    run: Callable
+    check: Callable
+
+    def __str__(self):
+        return self.name
+
+
+def _run_thread_local_algorithm_case(cc, case: _AlgorithmCase) -> None:
+    warm_algorithm = case.make_shared(cc)
+
+    warm_worker = case.make_worker(cc, worker_id=0, iteration=-1)
+    case.run(cc, warm_algorithm, warm_worker)
+    case.check(cc, warm_worker)
+
+    for iteration in range(STRESS_ITERATIONS):
+        worker_state = [
+            case.make_worker(cc, worker_id=worker_id, iteration=iteration)
+            for worker_id in range(STRESS_THREADS)
+        ]
+        returned_algorithms = [None] * STRESS_THREADS
+
+        def make_thread(worker_id, worker):
+            def thread(barrier):
+                barrier.wait()
+                algorithm = case.make_shared(cc)
+                returned_algorithms[worker_id] = algorithm
+                case.run(cc, algorithm, worker)
+                case.check(cc, worker)
+
+            return thread
+
+        _run_threaded(
+            [
+                make_thread(worker_id, worker)
+                for worker_id, worker in enumerate(worker_state)
+            ]
+        )
+
+        assert len({id(algorithm) for algorithm in returned_algorithms}) == len(
+            returned_algorithms
+        )
+        assert (
+            len({id(_get_build_result(algorithm)) for algorithm in returned_algorithms})
+            == 1
+        )
+
+
+def _make_reduce_worker(cc, worker_id, iteration):
+    stream = _make_stream()
+    h_in = np.arange(64, dtype=np.int32) + worker_id * 101 + iteration
+    h_init = np.array([7 + worker_id], dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(1, dtype=np.int32)
+    return {
+        "stream": stream,
+        "h_in": h_in,
+        "d_in": d_in,
+        "d_out": d_out,
+        "h_init": h_init,
+    }
+
+
+def _make_reduce_shared(cc):
+    worker = _make_reduce_worker(cc, 0, -1)
+    return cc.make_reduce_into(
+        d_in=worker["d_in"],
+        d_out=worker["d_out"],
+        op=cc.OpKind.PLUS,
+        h_init=worker["h_init"],
+    )
+
+
+def _run_reduce(cc, reducer, worker):
+    _call_with_temp(
+        reducer,
+        d_in=worker["d_in"],
+        d_out=worker["d_out"],
+        op=cc.OpKind.PLUS,
+        h_init=worker["h_init"],
+        num_items=worker["h_in"].size,
+        stream=worker["stream"],
+    )
+
+
+def _check_reduce(cc, worker):
+    worker["stream"].sync()
+    expected = worker["h_in"].sum(dtype=np.int64) + int(worker["h_init"][0])
+    assert int(worker["d_out"].copy_to_host()[0]) == int(expected)
+
+
+def _make_unary_worker(cc, worker_id, iteration):
+    stream = _make_stream()
+    h_in = np.arange(32, dtype=np.int32) + worker_id * 17 + iteration
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    return {
+        "stream": stream,
+        "h_in": h_in,
+        "d_in": d_in,
+        "d_out": d_out,
+    }
+
+
+def _make_unary_shared(cc):
+    worker = _make_unary_worker(cc, 0, -1)
+    return cc.make_unary_transform(
+        d_in=worker["d_in"], d_out=worker["d_out"], op=cc.OpKind.NEGATE
+    )
+
+
+def _make_unary_for_worker(cc, worker):
+    return cc.make_unary_transform(
+        d_in=worker["d_in"], d_out=worker["d_out"], op=cc.OpKind.NEGATE
+    )
+
+
+def _run_unary(cc, transformer, worker):
+    transformer(
+        d_in=worker["d_in"],
+        d_out=worker["d_out"],
+        op=cc.OpKind.NEGATE,
+        num_items=worker["h_in"].size,
+        stream=worker["stream"],
+    )
+
+
+def _run_unary_empty(cc, transformer, worker):
+    transformer(
+        d_in=worker["d_in"],
+        d_out=worker["d_out"],
+        op=cc.OpKind.NEGATE,
+        num_items=0,
+        stream=worker["stream"],
+    )
+
+
+def _check_unary(cc, worker):
+    worker["stream"].sync()
+    np.testing.assert_array_equal(worker["d_out"].copy_to_host(), -worker["h_in"])
+
+
+def _make_binary_worker(cc, worker_id, iteration):
+    stream = _make_stream()
+    h_in1 = np.arange(32, dtype=np.int32) + worker_id * 13
+    h_in2 = np.arange(32, dtype=np.int32) + iteration * 7
+    d_in1 = DeviceArray.from_numpy(h_in1)
+    d_in2 = DeviceArray.from_numpy(h_in2)
+    d_out = DeviceArray.empty(h_in1.shape, h_in1.dtype)
+    return {
+        "stream": stream,
+        "h_in1": h_in1,
+        "h_in2": h_in2,
+        "d_in1": d_in1,
+        "d_in2": d_in2,
+        "d_out": d_out,
+    }
+
+
+def _make_binary_for_worker(cc, worker):
+    return cc.make_binary_transform(
+        d_in1=worker["d_in1"],
+        d_in2=worker["d_in2"],
+        d_out=worker["d_out"],
+        op=cc.OpKind.PLUS,
+    )
+
+
+def _run_binary(cc, transformer, worker):
+    transformer(
+        d_in1=worker["d_in1"],
+        d_in2=worker["d_in2"],
+        d_out=worker["d_out"],
+        op=cc.OpKind.PLUS,
+        num_items=worker["h_in1"].size,
+        stream=worker["stream"],
+    )
+
+
+def _run_binary_empty(cc, transformer, worker):
+    transformer(
+        d_in1=worker["d_in1"],
+        d_in2=worker["d_in2"],
+        d_out=worker["d_out"],
+        op=cc.OpKind.PLUS,
+        num_items=0,
+        stream=worker["stream"],
+    )
+
+
+def _check_binary(cc, worker):
+    worker["stream"].sync()
+    np.testing.assert_array_equal(
+        worker["d_out"].copy_to_host(), worker["h_in1"] + worker["h_in2"]
+    )
+
+
+def _make_scan_worker(cc, worker_id, iteration):
+    stream = _make_stream()
+    h_in = np.arange(1, 33, dtype=np.int32) + worker_id + iteration
+    h_init = np.array([3 + worker_id], dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    return {
+        "stream": stream,
+        "h_in": h_in,
+        "h_init": h_init,
+        "d_in": d_in,
+        "d_out": d_out,
+    }
+
+
+def _make_exclusive_scan_shared(cc):
+    worker = _make_scan_worker(cc, 0, -1)
+    return cc.make_exclusive_scan(
+        d_in=worker["d_in"],
+        d_out=worker["d_out"],
+        op=cc.OpKind.PLUS,
+        init_value=worker["h_init"],
+    )
+
+
+def _run_scan(cc, scanner, worker):
+    _call_with_temp(
+        scanner,
+        d_in=worker["d_in"],
+        d_out=worker["d_out"],
+        op=cc.OpKind.PLUS,
+        init_value=worker["h_init"],
+        num_items=worker["h_in"].size,
+        stream=worker["stream"],
+    )
+
+
+def _check_exclusive_scan(cc, worker):
+    worker["stream"].sync()
+    expected = np.empty_like(worker["h_in"])
+    expected[0] = worker["h_init"][0]
+    expected[1:] = worker["h_init"][0] + np.cumsum(worker["h_in"][:-1])
+    np.testing.assert_array_equal(worker["d_out"].copy_to_host(), expected)
+
+
+def _make_segmented_reduce_worker(cc, worker_id, iteration):
+    stream = _make_stream()
+    h_in = np.arange(1, 17, dtype=np.int32) + worker_id * 3 + iteration
+    h_start_offsets = np.array([0, 3, 8, 12], dtype=np.int32)
+    h_end_offsets = np.array([3, 8, 12, 16], dtype=np.int32)
+    h_init = np.array([worker_id], dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(len(h_start_offsets), dtype=np.int32)
+    d_start_offsets = DeviceArray.from_numpy(h_start_offsets)
+    d_end_offsets = DeviceArray.from_numpy(h_end_offsets)
+    return {
+        "stream": stream,
+        "h_in": h_in,
+        "h_start_offsets": h_start_offsets,
+        "h_end_offsets": h_end_offsets,
+        "h_init": h_init,
+        "d_in": d_in,
+        "d_out": d_out,
+        "d_start_offsets": d_start_offsets,
+        "d_end_offsets": d_end_offsets,
+    }
+
+
+def _make_segmented_reduce_shared(cc):
+    worker = _make_segmented_reduce_worker(cc, 0, -1)
+    return cc.make_segmented_reduce(
+        d_in=worker["d_in"],
+        d_out=worker["d_out"],
+        start_offsets_in=worker["d_start_offsets"],
+        end_offsets_in=worker["d_end_offsets"],
+        op=cc.OpKind.PLUS,
+        h_init=worker["h_init"],
+    )
+
+
+def _run_segmented_reduce(cc, reducer, worker):
+    _call_with_temp(
+        reducer,
+        d_in=worker["d_in"],
+        d_out=worker["d_out"],
+        num_segments=len(worker["h_start_offsets"]),
+        start_offsets_in=worker["d_start_offsets"],
+        end_offsets_in=worker["d_end_offsets"],
+        op=cc.OpKind.PLUS,
+        h_init=worker["h_init"],
+        stream=worker["stream"],
+    )
+
+
+def _check_segmented_reduce(cc, worker):
+    worker["stream"].sync()
+    expected = np.array(
+        [
+            worker["h_in"][start:end].sum() + worker["h_init"][0]
+            for start, end in zip(worker["h_start_offsets"], worker["h_end_offsets"])
+        ],
+        dtype=np.int32,
+    )
+    np.testing.assert_array_equal(worker["d_out"].copy_to_host(), expected)
+
+
+def _make_binary_search_worker(cc, worker_id, iteration):
+    stream = _make_stream()
+    h_data = np.array([90, 70, 50, 30, 10], dtype=np.int32) - worker_id
+    h_values = np.array([95, 70, 45, 10, 5], dtype=np.int32) - worker_id
+    d_data = DeviceArray.from_numpy(h_data)
+    d_values = DeviceArray.from_numpy(h_values)
+    d_out = DeviceArray.empty(h_values.size, dtype=np.uintp)
+    return {
+        "stream": stream,
+        "h_data": h_data,
+        "h_values": h_values,
+        "d_data": d_data,
+        "d_values": d_values,
+        "d_out": d_out,
+    }
+
+
+def _make_lower_bound_for_worker(cc, worker):
+    return cc.make_lower_bound(
+        d_data=worker["d_data"],
+        d_values=worker["d_values"],
+        d_out=worker["d_out"],
+        comp=cc.OpKind.GREATER,
+    )
+
+
+def _make_upper_bound_for_worker(cc, worker):
+    return cc.make_upper_bound(
+        d_data=worker["d_data"],
+        d_values=worker["d_values"],
+        d_out=worker["d_out"],
+        comp=cc.OpKind.GREATER,
+    )
+
+
+def _run_binary_search(cc, searcher, worker):
+    searcher(
+        d_data=worker["d_data"],
+        num_items=worker["h_data"].size,
+        d_values=worker["d_values"],
+        num_values=worker["h_values"].size,
+        d_out=worker["d_out"],
+        comp=cc.OpKind.GREATER,
+        stream=worker["stream"],
+    )
+
+
+def _run_binary_search_empty(cc, searcher, worker):
+    searcher(
+        d_data=worker["d_data"],
+        num_items=worker["h_data"].size,
+        d_values=worker["d_values"],
+        num_values=0,
+        d_out=worker["d_out"],
+        comp=cc.OpKind.GREATER,
+        stream=worker["stream"],
+    )
+
+
+def _check_lower_bound(cc, worker):
+    worker["stream"].sync()
+    expected = np.searchsorted(-worker["h_data"], -worker["h_values"], side="left")
+    np.testing.assert_array_equal(
+        worker["d_out"].copy_to_host(), expected.astype(np.uintp)
+    )
+
+
+def _check_upper_bound(cc, worker):
+    worker["stream"].sync()
+    expected = np.searchsorted(-worker["h_data"], -worker["h_values"], side="right")
+    np.testing.assert_array_equal(
+        worker["d_out"].copy_to_host(), expected.astype(np.uintp)
+    )
+
+
+def _make_select_worker(cc, worker_id, iteration):
+    stream = _make_stream()
+    h_in = np.array(
+        [True, False, worker_id % 2 == 0, True, False, iteration % 2 == 0],
+        dtype=np.bool_,
+    )
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_count = DeviceArray.empty(2, dtype=np.uint64)
+    return {
+        "stream": stream,
+        "h_in": h_in,
+        "d_in": d_in,
+        "d_out": d_out,
+        "d_count": d_count,
+    }
+
+
+def _make_three_way_shared(cc):
+    worker = _make_select_worker(cc, 0, -1)
+    d_unselected = DeviceArray.empty(worker["h_in"].shape, worker["h_in"].dtype)
+    return cc.make_three_way_partition(
+        d_in=worker["d_in"],
+        d_first_part_out=worker["d_out"],
+        d_second_part_out=d_unselected,
+        d_unselected_out=DeviceArray.empty(worker["h_in"].shape, worker["h_in"].dtype),
+        d_num_selected_out=worker["d_count"],
+        select_first_part_op=cc.OpKind.IDENTITY,
+        select_second_part_op=cc.OpKind.LOGICAL_NOT,
+    )
+
+
+def _make_three_way_worker(cc, worker_id, iteration):
+    worker = _make_select_worker(cc, worker_id, iteration)
+    worker["d_second_out"] = DeviceArray.empty(
+        worker["h_in"].shape, worker["h_in"].dtype
+    )
+    worker["d_unselected"] = DeviceArray.empty(
+        worker["h_in"].shape, worker["h_in"].dtype
+    )
+    return worker
+
+
+def _run_three_way(cc, partitioner, worker):
+    _call_with_temp(
+        partitioner,
+        d_in=worker["d_in"],
+        d_first_part_out=worker["d_out"],
+        d_second_part_out=worker["d_second_out"],
+        d_unselected_out=worker["d_unselected"],
+        d_num_selected_out=worker["d_count"],
+        select_first_part_op=cc.OpKind.IDENTITY,
+        select_second_part_op=cc.OpKind.LOGICAL_NOT,
+        num_items=worker["h_in"].size,
+        stream=worker["stream"],
+    )
+
+
+def _check_three_way(cc, worker):
+    worker["stream"].sync()
+    counts = worker["d_count"].copy_to_host()
+    true_count = int(np.count_nonzero(worker["h_in"]))
+    false_count = int(worker["h_in"].size - true_count)
+    assert int(counts[0]) == true_count
+    assert int(counts[1]) == false_count
+    np.testing.assert_array_equal(
+        worker["d_out"].copy_to_host()[:true_count], np.ones(true_count, dtype=np.bool_)
+    )
+    np.testing.assert_array_equal(
+        worker["d_second_out"].copy_to_host()[:false_count],
+        np.zeros(false_count, dtype=np.bool_),
+    )
+
+
+def _make_radix_sort_worker(cc, worker_id, iteration):
+    stream = _make_stream()
+    h_keys = np.array([7, 3, 5, 3, 1, 9], dtype=np.uint32) + np.uint32(worker_id * 11)
+    h_values = np.arange(h_keys.size, dtype=np.int32) + iteration * 10
+    d_in_keys = DeviceArray.from_numpy(h_keys)
+    d_tmp_keys = DeviceArray.empty(h_keys.shape, h_keys.dtype)
+    d_in_values = DeviceArray.from_numpy(h_values)
+    d_tmp_values = DeviceArray.empty(h_values.shape, h_values.dtype)
+    return {
+        "stream": stream,
+        "h_keys": h_keys,
+        "h_values": h_values,
+        "keys": cc.DoubleBuffer(d_in_keys, d_tmp_keys),
+        "values": cc.DoubleBuffer(d_in_values, d_tmp_values),
+    }
+
+
+def _make_radix_sort_shared(cc):
+    worker = _make_radix_sort_worker(cc, 0, -1)
+    return cc.make_radix_sort(
+        d_in_keys=worker["keys"],
+        d_out_keys=None,
+        d_in_values=worker["values"],
+        d_out_values=None,
+        order=cc.SortOrder.ASCENDING,
+    )
+
+
+def _run_radix_sort(cc, sorter, worker):
+    _call_with_temp(
+        sorter,
+        d_in_keys=worker["keys"],
+        d_out_keys=None,
+        d_in_values=worker["values"],
+        d_out_values=None,
+        num_items=worker["h_keys"].size,
+        stream=worker["stream"],
+    )
+
+
+def _check_radix_sort(cc, worker):
+    worker["stream"].sync()
+    order = np.argsort(worker["h_keys"], kind="stable")
+    np.testing.assert_array_equal(
+        worker["keys"].current().copy_to_host(), worker["h_keys"][order]
+    )
+    np.testing.assert_array_equal(
+        worker["values"].current().copy_to_host(), worker["h_values"][order]
+    )
+    assert worker["keys"].selector == worker["values"].selector
+
+
+# A representative subset rather than every algorithm: these tests exercise the
+# algorithm-agnostic caching machinery (per-thread wrappers, one shared build,
+# single-flight coalescing, concurrent deserialize), so the cases are chosen
+# for build-result/descriptor variety rather than coverage of every algorithm —
+# basic reduce, a transform, a scan (init_value), a segmented op (offset
+# iterators), a sort (DoubleBuffer + SortOrder), and a multi-output partition.
+# Algorithm-specific native concurrency (transform's launch-config cache, the
+# v2 first-call gate) is covered by its own dedicated tests below, and
+# per-algorithm correctness lives in the per-algorithm test files.
+SHARED_ALGORITHM_CASES = [
+    _AlgorithmCase(
+        "reduce", _make_reduce_shared, _make_reduce_worker, _run_reduce, _check_reduce
+    ),
+    _AlgorithmCase(
+        "unary_transform",
+        _make_unary_shared,
+        _make_unary_worker,
+        _run_unary,
+        _check_unary,
+    ),
+    _AlgorithmCase(
+        "exclusive_scan",
+        _make_exclusive_scan_shared,
+        _make_scan_worker,
+        _run_scan,
+        _check_exclusive_scan,
+    ),
+    _AlgorithmCase(
+        "segmented_reduce",
+        _make_segmented_reduce_shared,
+        _make_segmented_reduce_worker,
+        _run_segmented_reduce,
+        _check_segmented_reduce,
+    ),
+    _AlgorithmCase(
+        "radix_sort",
+        _make_radix_sort_shared,
+        _make_radix_sort_worker,
+        _run_radix_sort,
+        _check_radix_sort,
+    ),
+    _AlgorithmCase(
+        "three_way_partition",
+        _make_three_way_shared,
+        _make_three_way_worker,
+        _run_three_way,
+        _check_three_way,
+    ),
+]
+
+
+def test_free_threaded_import_keeps_gil_disabled(compute_module):
+    """True first-import smoke for the PR's headline claim.
+
+    Asserting around imports in this process cannot guarantee ordering:
+    sibling test modules import cuda.compute and cuda.core at collection time.
+    A fresh interpreter gives the assert -> import -> assert bracket genuine
+    first-import semantics.
+    """
+    import os
+    import subprocess
+
+    del compute_module  # only used to gate on a free-threaded build
+
+    tests_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    code = f"""
+import sys
+
+assert not sys._is_gil_enabled(), "the GIL is enabled at interpreter start"
+sys.path.insert(0, {tests_dir!r})
+from _utils.device_array import DeviceArray
+
+assert not sys._is_gil_enabled(), "the GIL is enabled after importing the device-array utility"
+import numpy as np
+
+import cuda.compute as cc
+
+assert not sys._is_gil_enabled(), "the GIL is enabled after importing cuda.compute"
+h_in = np.arange(8, dtype=np.int32)
+d_in = DeviceArray.from_numpy(h_in)
+d_out = DeviceArray.empty(1, dtype=np.int32)
+h_init = np.array([0], dtype=np.int32)
+cc.reduce_into(d_in=d_in, d_out=d_out, num_items=h_in.size, op=cc.OpKind.PLUS, h_init=h_init)
+assert int(d_out.copy_to_host()[0]) == int(h_in.sum())
+assert not sys._is_gil_enabled(), "the GIL is enabled after running a cuda.compute operation"
+"""
+    result = subprocess.run(
+        [sys.executable, "-c", code],
+        capture_output=True,
+        text=True,
+        timeout=600,
+    )
+    assert result.returncode == 0, (
+        f"first-import GIL smoke subprocess failed (exit {result.returncode})\n"
+        f"stdout:\n{result.stdout}\nstderr:\n{result.stderr}"
+    )
+    _assert_gil_disabled("after the first-import smoke subprocess")
+
+
+@pytest.mark.parametrize("case", SHARED_ALGORITHM_CASES, ids=str)
+def test_thread_local_algorithm_objects_share_build_result(compute_module, case):
+    cc = compute_module
+
+    _run_thread_local_algorithm_case(cc, case)
+
+
+@pytest.mark.parametrize("case", SHARED_ALGORITHM_CASES, ids=str)
+def test_concurrent_deserialize_and_execute(compute_module, case):
+    cc = compute_module
+    _require_serialization_backend()
+
+    source_algorithm = case.make_shared(cc)
+    blob = cc.serialize(source_algorithm)
+    device_id = Device().device_id
+    _assert_gil_disabled("after serializing a cuda.compute algorithm")
+
+    for iteration in range(STRESS_ITERATIONS):
+        workers = [
+            case.make_worker(cc, worker_id=worker_id, iteration=iteration)
+            for worker_id in range(STRESS_THREADS)
+        ]
+        algorithms = [None] * STRESS_THREADS
+        execute_barrier = threading.Barrier(STRESS_THREADS)
+
+        def make_thread(worker_id, worker):
+            def thread(barrier):
+                Device(device_id).set_current()
+                barrier.wait()
+                try:
+                    algorithm = cc.deserialize(blob)
+                    algorithms[worker_id] = algorithm
+                    # Force the independently deserialized build results to
+                    # perform their first native loads concurrently.
+                    execute_barrier.wait(timeout=60)
+                    case.run(cc, algorithm, worker)
+                    case.check(cc, worker)
+                except BaseException:
+                    execute_barrier.abort()
+                    raise
+
+            return thread
+
+        _run_threaded(
+            [make_thread(worker_id, worker) for worker_id, worker in enumerate(workers)]
+        )
+
+        # deserialize() intentionally returns independent mutable wrappers and
+        # independently owned native build results. Sharing one wrapper between
+        # concurrent callers remains unsupported.
+        assert len({id(algorithm) for algorithm in algorithms}) == STRESS_THREADS
+        assert (
+            len({id(_get_build_result(algorithm)) for algorithm in algorithms})
+            == STRESS_THREADS
+        )
+
+
+def test_free_threaded_aot_factory_shares_compile_and_first_load(compute_module):
+    cc = compute_module
+    _require_serialization_backend()
+
+    device = Device()
+    target_cc = tuple(device.compute_capability)
+    proxy_in = cc.ProxyArray(np.int32)
+    proxy_out = cc.ProxyArray(np.int32)
+    workers = [
+        _make_unary_worker(cc, worker_id=worker_id, iteration=0)
+        for worker_id in range(STRESS_THREADS)
+    ]
+    algorithms = [None] * STRESS_THREADS
+    execute_barrier = threading.Barrier(STRESS_THREADS)
+
+    def make_thread(worker_id, worker):
+        def thread(barrier):
+            Device(device.device_id).set_current()
+            barrier.wait()
+            try:
+                algorithm = cc.make_unary_transform(
+                    d_in=proxy_in,
+                    d_out=proxy_out,
+                    op=cc.OpKind.NEGATE,
+                    compute_capability=target_cc,
+                )
+                algorithms[worker_id] = algorithm
+                execute_barrier.wait(timeout=60)
+                _run_unary(cc, algorithm, worker)
+                _check_unary(cc, worker)
+            except BaseException:
+                execute_barrier.abort()
+                raise
+
+        return thread
+
+    _run_threaded(
+        [make_thread(worker_id, worker) for worker_id, worker in enumerate(workers)]
+    )
+
+    assert len({id(algorithm) for algorithm in algorithms}) == STRESS_THREADS
+    assert len({id(algorithm.build_results) for algorithm in algorithms}) == 1
+    assert len({id(_get_build_result(algorithm)) for algorithm in algorithms}) == 1
+    assert _get_build_result(algorithms[0])._loaded
+
+
+def test_free_threaded_aot_blob_concurrent_deserialize_and_load(compute_module):
+    cc = compute_module
+    _require_serialization_backend()
+
+    device = Device()
+    target_cc = tuple(device.compute_capability)
+    source_algorithm = cc.make_unary_transform(
+        d_in=cc.ProxyArray(np.int32),
+        d_out=cc.ProxyArray(np.int32),
+        op=cc.OpKind.NEGATE,
+        compute_capability=target_cc,
+    )
+    source_build_result = _get_build_result(source_algorithm)
+    assert not source_build_result._loaded
+    blob = cc.serialize(source_algorithm)
+
+    workers = [
+        _make_unary_worker(cc, worker_id=worker_id, iteration=0)
+        for worker_id in range(STRESS_THREADS)
+    ]
+    algorithms = [None] * STRESS_THREADS
+    execute_barrier = threading.Barrier(STRESS_THREADS)
+
+    def make_thread(worker_id, worker):
+        def thread(barrier):
+            Device(device.device_id).set_current()
+            barrier.wait()
+            try:
+                algorithm = cc.deserialize(blob)
+                algorithms[worker_id] = algorithm
+                assert not _get_build_result(algorithm)._loaded
+                execute_barrier.wait(timeout=60)
+                _run_unary(cc, algorithm, worker)
+                _check_unary(cc, worker)
+            except BaseException:
+                execute_barrier.abort()
+                raise
+
+        return thread
+
+    _run_threaded(
+        [make_thread(worker_id, worker) for worker_id, worker in enumerate(workers)]
+    )
+
+    build_results = [_get_build_result(algorithm) for algorithm in algorithms]
+    assert len({id(algorithm) for algorithm in algorithms}) == STRESS_THREADS
+    assert len({id(build_result) for build_result in build_results}) == STRESS_THREADS
+    assert all(build_result._loaded for build_result in build_results)
+    assert not source_build_result._loaded
+
+
+def test_free_threaded_multi_cc_blob_concurrent_deserialize_and_execute(
+    compute_module,
+):
+    """Concurrent per-thread use of a genuine multi-arch blob.
+
+    Multi-cc blobs take a different path from the single-cc blobs above: the
+    cc check is deferred from deserialize time to call time, and each call
+    resolves the current device's entry out of a multi-entry collection. The
+    entry for the other arch must stay lazy in every thread.
+    """
+    cc = compute_module
+    _require_serialization_backend()
+
+    device = Device()
+    cc_major, cc_minor = device.compute_capability
+    current_key = cc_major * 10 + cc_minor
+
+    # Find one more arch this toolchain can compile for.
+    other_key = None
+    for candidate in (90, 100, 89, 120, 80, 86, 75):
+        if candidate == current_key:
+            continue
+        try:
+            cc.make_unary_transform(
+                d_in=cc.ProxyArray(np.int32),
+                d_out=cc.ProxyArray(np.int32),
+                op=cc.OpKind.NEGATE,
+                compute_capability=candidate,
+            )
+        except Exception:
+            continue
+        other_key = candidate
+        break
+    if other_key is None:
+        pytest.skip("toolchain compiles for <2 target arches")
+
+    source_algorithm = cc.make_unary_transform(
+        d_in=cc.ProxyArray(np.int32),
+        d_out=cc.ProxyArray(np.int32),
+        op=cc.OpKind.NEGATE,
+        compute_capability=[current_key, other_key],
+    )
+    blob = cc.serialize(source_algorithm)
+
+    workers = [
+        _make_unary_worker(cc, worker_id=worker_id, iteration=0)
+        for worker_id in range(STRESS_THREADS)
+    ]
+    algorithms = [None] * STRESS_THREADS
+    execute_barrier = threading.Barrier(STRESS_THREADS)
+
+    def make_thread(worker_id, worker):
+        def thread(barrier):
+            Device(device.device_id).set_current()
+            barrier.wait()
+            try:
+                algorithm = cc.deserialize(blob)
+                algorithms[worker_id] = algorithm
+                execute_barrier.wait(timeout=60)
+                _run_unary(cc, algorithm, worker)
+                _check_unary(cc, worker)
+            except BaseException:
+                execute_barrier.abort()
+                raise
+
+        return thread
+
+    _run_threaded(
+        [make_thread(worker_id, worker) for worker_id, worker in enumerate(workers)]
+    )
+
+    for algorithm in algorithms:
+        assert set(algorithm.build_results.keys()) == {current_key, other_key}
+        # Only the current device's arch loads; the other entry stays lazy.
+        assert algorithm.build_results[current_key]._loaded
+        assert not algorithm.build_results[other_key]._loaded
+
+
+def _cache_miss_reduce(cc, worker_id, iteration):
+    worker = _make_reduce_worker(cc, worker_id, iteration)
+    reducer = cc.make_reduce_into(
+        d_in=worker["d_in"],
+        d_out=worker["d_out"],
+        op=cc.OpKind.PLUS,
+        h_init=worker["h_init"],
+    )
+    _run_reduce(cc, reducer, worker)
+    _check_reduce(cc, worker)
+    return reducer
+
+
+def _cache_miss_unary_transform(cc, worker_id, iteration):
+    worker = _make_unary_worker(cc, worker_id, iteration)
+    transformer = cc.make_unary_transform(
+        d_in=worker["d_in"], d_out=worker["d_out"], op=cc.OpKind.NEGATE
+    )
+    _run_unary(cc, transformer, worker)
+    _check_unary(cc, worker)
+    return transformer
+
+
+def _cache_miss_binary_transform(cc, worker_id, iteration):
+    worker = _make_binary_worker(cc, worker_id, iteration)
+    transformer = cc.make_binary_transform(
+        d_in1=worker["d_in1"],
+        d_in2=worker["d_in2"],
+        d_out=worker["d_out"],
+        op=cc.OpKind.PLUS,
+    )
+    _run_binary(cc, transformer, worker)
+    _check_binary(cc, worker)
+    return transformer
+
+
+@pytest.mark.parametrize(
+    "factory",
+    [_cache_miss_reduce, _cache_miss_unary_transform, _cache_miss_binary_transform],
+    ids=["reduce", "unary_transform", "binary_transform"],
+)
+def test_same_key_factory_cache_miss_storm(compute_module, factory):
+    cc = compute_module
+
+    for iteration in range(STRESS_ITERATIONS):
+        cc.clear_all_caches()
+        returned_objects = [None] * STRESS_THREADS
+
+        def make_thread(worker_id):
+            def thread(barrier):
+                barrier.wait()
+                returned_objects[worker_id] = factory(cc, worker_id, iteration)
+
+            return thread
+
+        _run_threaded([make_thread(worker_id) for worker_id in range(STRESS_THREADS)])
+
+        assert len({id(obj) for obj in returned_objects}) == len(returned_objects)
+        assert len({id(_get_build_result(obj)) for obj in returned_objects}) == 1
+
+
+def _storm_reduce_int32(cc, worker_id, iteration):
+    stream = _make_stream()
+    h_in = np.arange(64, dtype=np.int32) + worker_id + iteration
+    h_init = np.array([worker_id], dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(1, dtype=np.int32)
+    reducer = cc.make_reduce_into(
+        d_in=d_in, d_out=d_out, op=cc.OpKind.PLUS, h_init=h_init
+    )
+    _call_with_temp(
+        reducer,
+        d_in=d_in,
+        d_out=d_out,
+        op=cc.OpKind.PLUS,
+        h_init=h_init,
+        num_items=h_in.size,
+        stream=stream,
+    )
+    stream.sync()
+    assert int(d_out.copy_to_host()[0]) == int(h_in.sum()) + int(h_init[0])
+    return reducer
+
+
+def _storm_reduce_float64(cc, worker_id, iteration):
+    stream = _make_stream()
+    h_in = np.arange(64, dtype=np.float64) + worker_id + iteration
+    h_init = np.array([worker_id], dtype=np.float64)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(1, dtype=np.float64)
+    reducer = cc.make_reduce_into(
+        d_in=d_in, d_out=d_out, op=cc.OpKind.PLUS, h_init=h_init
+    )
+    _call_with_temp(
+        reducer,
+        d_in=d_in,
+        d_out=d_out,
+        op=cc.OpKind.PLUS,
+        h_init=h_init,
+        num_items=h_in.size,
+        stream=stream,
+    )
+    stream.sync()
+    assert float(d_out.copy_to_host()[0]) == float(h_in.sum()) + float(h_init[0])
+    return reducer
+
+
+def _storm_unary_transform_int64(cc, worker_id, iteration):
+    stream = _make_stream()
+    h_in = np.arange(64, dtype=np.int64) + worker_id + iteration
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    transformer = cc.make_unary_transform(d_in=d_in, d_out=d_out, op=cc.OpKind.NEGATE)
+    transformer(
+        d_in=d_in,
+        d_out=d_out,
+        op=cc.OpKind.NEGATE,
+        num_items=h_in.size,
+        stream=stream,
+    )
+    stream.sync()
+    np.testing.assert_array_equal(d_out.copy_to_host(), -h_in)
+    return transformer
+
+
+def _storm_binary_transform_float32(cc, worker_id, iteration):
+    stream = _make_stream()
+    h_in1 = np.arange(64, dtype=np.float32) + worker_id
+    h_in2 = np.arange(64, dtype=np.float32) + iteration
+    d_in1 = DeviceArray.from_numpy(h_in1)
+    d_in2 = DeviceArray.from_numpy(h_in2)
+    d_out = DeviceArray.empty(h_in1.shape, h_in1.dtype)
+    transformer = cc.make_binary_transform(
+        d_in1=d_in1, d_in2=d_in2, d_out=d_out, op=cc.OpKind.PLUS
+    )
+    transformer(
+        d_in1=d_in1,
+        d_in2=d_in2,
+        d_out=d_out,
+        op=cc.OpKind.PLUS,
+        num_items=h_in1.size,
+        stream=stream,
+    )
+    stream.sync()
+    np.testing.assert_array_equal(d_out.copy_to_host(), h_in1 + h_in2)
+    return transformer
+
+
+_DISTINCT_KEY_STORM_FACTORIES = [
+    _storm_reduce_int32,
+    _storm_reduce_float64,
+    _storm_unary_transform_int64,
+    _storm_binary_transform_float32,
+]
+
+
+def test_distinct_key_cold_build_storm(compute_module):
+    """Force truly concurrent native compilations.
+
+    Same-key storms coalesce through _cache_single_flight to a single builder
+    thread, so they never overlap the native compile pipeline. Distinct keys
+    (different algorithms and dtypes) elect one builder per worker, running
+    NVRTC/nvJitLink (v1) or HostJIT (v2) compilations genuinely in parallel.
+    """
+    cc = compute_module
+
+    for iteration in range(DISTINCT_KEY_STORM_ITERATIONS):
+        cc.clear_all_caches()
+        returned_objects = [None] * len(_DISTINCT_KEY_STORM_FACTORIES)
+
+        def make_thread(worker_id, factory):
+            def thread(barrier):
+                barrier.wait()
+                returned_objects[worker_id] = factory(cc, worker_id, iteration)
+
+            return thread
+
+        _run_threaded(
+            [
+                make_thread(worker_id, factory)
+                for worker_id, factory in enumerate(_DISTINCT_KEY_STORM_FACTORIES)
+            ]
+        )
+
+        # Distinct keys must not share builds: one build result per worker.
+        build_ids = {id(_get_build_result(obj)) for obj in returned_objects}
+        assert len(build_ids) == len(_DISTINCT_KEY_STORM_FACTORIES)
+
+
+def test_shared_raw_op_object_direct_algorithm_stress(compute_module):
+    cc = compute_module
+
+    from cuda.compute._cpp_compile import compile_cpp_op_code
+    from cuda.compute.op import RawOp
+
+    source = """
+    extern "C" __device__ void raw_add_i32(void* a, void* b, void* result) {
+        *static_cast<int*>(result) = *static_cast<int*>(a) + *static_cast<int*>(b);
+    }
+    """
+    shared_op = RawOp(ltoir=compile_cpp_op_code(source), name="raw_add_i32")
+
+    for iteration in range(STRESS_ITERATIONS):
+        cc.clear_all_caches()
+        returned_reducers = [None] * STRESS_THREADS
+
+        def make_thread(worker_id):
+            stream = _make_stream()
+            h_in = np.arange(32, dtype=np.int32) + worker_id * 31 + iteration
+            h_init = np.array([worker_id + 5], dtype=np.int32)
+            d_in = DeviceArray.from_numpy(h_in)
+            d_out = DeviceArray.empty(1, dtype=np.int32)
+
+            def thread(barrier):
+                barrier.wait()
+                reducer = cc.make_reduce_into(
+                    d_in=d_in,
+                    d_out=d_out,
+                    op=shared_op,
+                    h_init=h_init,
+                )
+                returned_reducers[worker_id] = reducer
+                _call_with_temp(
+                    reducer,
+                    d_in=d_in,
+                    d_out=d_out,
+                    op=shared_op,
+                    h_init=h_init,
+                    num_items=h_in.size,
+                    stream=stream,
+                )
+                stream.sync()
+                expected = int(h_in.sum(dtype=np.int64) + h_init[0])
+                assert int(d_out.copy_to_host()[0]) == expected
+
+            return thread
+
+        _run_threaded([make_thread(worker_id) for worker_id in range(STRESS_THREADS)])
+
+        assert len({id(reducer) for reducer in returned_reducers}) == len(
+            returned_reducers
+        )
+        assert (
+            len({id(_get_build_result(reducer)) for reducer in returned_reducers}) == 1
+        )
+
+
+@dataclass(frozen=True)
+class _IteratorCase:
+    name: str
+    make_iterator: Callable
+    dtype: np.dtype
+    num_items: int
+    expected_sum: int
+
+    def __str__(self):
+        return self.name
+
+
+@dataclass(frozen=True)
+class _ColdTransformCase:
+    name: str
+    make_worker: Callable
+    make_transformer: Callable
+    run: Callable
+    check: Callable
+
+    def __str__(self):
+        return self.name
+
+
+def _run_cold_transform_native_cache_case(cc, case: _ColdTransformCase) -> None:
+    for iteration in range(STRESS_ITERATIONS):
+        cc.clear_all_caches()
+        workers = [
+            case.make_worker(cc, worker_id=worker_id, iteration=iteration)
+            for worker_id in range(TRANSFORM_NATIVE_CACHE_THREADS)
+        ]
+        returned_algorithms = [None] * TRANSFORM_NATIVE_CACHE_THREADS
+        # Transform's native launch config cache is filled on first execution,
+        # so build wrappers first and synchronize the first call separately.
+        execute_barrier = threading.Barrier(TRANSFORM_NATIVE_CACHE_THREADS)
+
+        def make_thread(worker_id, worker):
+            def thread(barrier):
+                barrier.wait()
+                try:
+                    algorithm = case.make_transformer(cc, worker)
+                    returned_algorithms[worker_id] = algorithm
+                except BaseException:
+                    execute_barrier.abort()
+                    raise
+
+                execute_barrier.wait(timeout=60)
+                case.run(cc, algorithm, worker)
+                case.check(cc, worker)
+
+            return thread
+
+        _run_threaded(
+            [make_thread(worker_id, worker) for worker_id, worker in enumerate(workers)]
+        )
+
+        assert len({id(algorithm) for algorithm in returned_algorithms}) == len(
+            returned_algorithms
+        )
+        assert (
+            len({id(_get_build_result(algorithm)) for algorithm in returned_algorithms})
+            == 1
+        )
+
+
+@pytest.mark.parametrize(
+    "case",
+    [
+        _ColdTransformCase(
+            "unary_transform",
+            _make_unary_worker,
+            _make_unary_for_worker,
+            _run_unary,
+            _check_unary,
+        ),
+        _ColdTransformCase(
+            "binary_transform",
+            _make_binary_worker,
+            _make_binary_for_worker,
+            _run_binary,
+            _check_binary,
+        ),
+    ],
+    ids=str,
+)
+def test_cold_transform_native_cache_initialization_stress(compute_module, case):
+    cc = compute_module
+
+    _run_cold_transform_native_cache_case(cc, case)
+
+
+@dataclass(frozen=True)
+class _V2FirstCallCase:
+    name: str
+    make_worker: Callable
+    make_algorithm: Callable
+    run_empty: Callable
+    run: Callable
+    check: Callable
+
+    def __str__(self):
+        return self.name
+
+
+def _run_v2_first_call_gate_case(cc, case: _V2FirstCallCase) -> None:
+    for iteration in range(STRESS_ITERATIONS):
+        cc.clear_all_caches()
+        workers = [
+            case.make_worker(cc, worker_id=worker_id, iteration=iteration)
+            for worker_id in range(TRANSFORM_NATIVE_CACHE_THREADS)
+        ]
+        returned_algorithms = [None] * TRANSFORM_NATIVE_CACHE_THREADS
+        algorithms_built_barrier = threading.Barrier(TRANSFORM_NATIVE_CACHE_THREADS)
+        nonempty_call_barrier = threading.Barrier(TRANSFORM_NATIVE_CACHE_THREADS)
+
+        def make_thread(worker_id, worker):
+            def thread(barrier):
+                barrier.wait()
+                try:
+                    algorithm = case.make_algorithm(cc, worker)
+                    returned_algorithms[worker_id] = algorithm
+                    algorithms_built_barrier.wait(timeout=60)
+
+                    # Empty calls return before CUB initializes its static launch
+                    # configuration and therefore must not complete the gate.
+                    if worker_id == 0:
+                        case.run_empty(cc, algorithm, worker)
+
+                    nonempty_call_barrier.wait(timeout=60)
+                    case.run(cc, algorithm, worker)
+                    case.check(cc, worker)
+                except BaseException:
+                    algorithms_built_barrier.abort()
+                    nonempty_call_barrier.abort()
+                    raise
+
+            return thread
+
+        _run_threaded(
+            [make_thread(worker_id, worker) for worker_id, worker in enumerate(workers)]
+        )
+
+        assert len({id(algorithm) for algorithm in returned_algorithms}) == len(
+            returned_algorithms
+        )
+        assert (
+            len({id(_get_build_result(algorithm)) for algorithm in returned_algorithms})
+            == 1
+        )
+
+
+_V2_FIRST_CALL_CASES = [
+    _V2FirstCallCase(
+        "unary_transform",
+        _make_unary_worker,
+        _make_unary_for_worker,
+        _run_unary_empty,
+        _run_unary,
+        _check_unary,
+    ),
+    _V2FirstCallCase(
+        "binary_transform",
+        _make_binary_worker,
+        _make_binary_for_worker,
+        _run_binary_empty,
+        _run_binary,
+        _check_binary,
+    ),
+    _V2FirstCallCase(
+        "lower_bound",
+        _make_binary_search_worker,
+        _make_lower_bound_for_worker,
+        _run_binary_search_empty,
+        _run_binary_search,
+        _check_lower_bound,
+    ),
+    _V2FirstCallCase(
+        "upper_bound",
+        _make_binary_search_worker,
+        _make_upper_bound_for_worker,
+        _run_binary_search_empty,
+        _run_binary_search,
+        _check_upper_bound,
+    ),
+]
+
+
+def _run_concurrent_cold_llvm_initialization():
+    """Body of test_v2_concurrent_cold_llvm_initialization, run in a fresh process."""
+    import cuda.compute as cc
+
+    cc.clear_all_caches()
+    workers = [
+        case.make_worker(cc, worker_id=worker_id, iteration=0)
+        for worker_id, case in enumerate(_V2_FIRST_CALL_CASES)
+    ]
+    returned_algorithms = [None] * len(_V2_FIRST_CALL_CASES)
+    build_barrier = threading.Barrier(len(_V2_FIRST_CALL_CASES))
+
+    def make_thread(worker_id, case, worker):
+        def thread():
+            build_barrier.wait(timeout=60)
+            algorithm = case.make_algorithm(cc, worker)
+            returned_algorithms[worker_id] = algorithm
+            case.run(cc, algorithm, worker)
+            case.check(cc, worker)
+
+        return thread
+
+    try:
+        # Distinct algorithm keys prevent the Python build cache from
+        # coalescing these concurrent HostJIT compiler initializations.
+        with concurrent.futures.ThreadPoolExecutor(
+            max_workers=len(_V2_FIRST_CALL_CASES)
+        ) as executor:
+            futures = [
+                executor.submit(make_thread(worker_id, case, worker))
+                for worker_id, (case, worker) in enumerate(
+                    zip(_V2_FIRST_CALL_CASES, workers)
+                )
+            ]
+            for future in futures:
+                future.result()
+    finally:
+        cc.clear_all_caches()
+
+    assert len(
+        {id(_get_build_result(algorithm)) for algorithm in returned_algorithms}
+    ) == len(returned_algorithms)
+
+
+def test_v2_concurrent_cold_llvm_initialization():
+    from cuda.compute._build_info import USING_V2
+
+    if not USING_V2:
+        pytest.skip("requires the C Parallel v2 backend")
+
+    import os
+    import subprocess
+
+    # LLVM target registration is process-wide and only cold once, so earlier
+    # tests in this pytest process would leave it warm and this test would no
+    # longer exercise concurrent *cold* initialization. Run the storm in a
+    # fresh interpreter regardless of what already ran here. The tests root
+    # must be on sys.path for the _utils.device_array import (pytest's
+    # pythonpath setting does not apply to a raw subprocess).
+    compute_dir = os.path.dirname(os.path.abspath(__file__))
+    tests_dir = os.path.dirname(compute_dir)
+    code = (
+        "import sys; "
+        f"sys.path.insert(0, {tests_dir!r}); "
+        f"sys.path.insert(0, {compute_dir!r}); "
+        "import test_free_threading_stress as m; "
+        "m._run_concurrent_cold_llvm_initialization()"
+    )
+    result = subprocess.run(
+        [sys.executable, "-c", code],
+        capture_output=True,
+        text=True,
+        timeout=600,
+    )
+    assert result.returncode == 0, (
+        f"cold LLVM initialization subprocess failed "
+        f"(exit {result.returncode})\nstdout:\n{result.stdout}\n"
+        f"stderr:\n{result.stderr}"
+    )
+
+
+@pytest.mark.parametrize("case", _V2_FIRST_CALL_CASES, ids=str)
+def test_v2_first_call_gate_stress(compute_module, case):
+    cc = compute_module
+
+    from cuda.compute._build_info import USING_V2
+
+    if not USING_V2:
+        pytest.skip("requires the C Parallel v2 backend")
+
+    _run_v2_first_call_gate_case(cc, case)
+
+
+def _iterator_counting(cc):
+    return cc.CountingIterator(np.int32(0)), np.dtype(np.int32), 32, sum(range(32))
+
+
+def _iterator_permutation(cc):
+    h_values = np.arange(32, dtype=np.int32)
+    h_indices = np.arange(31, -1, -1, dtype=np.int32)
+    d_values = DeviceArray.from_numpy(h_values)
+    d_indices = DeviceArray.from_numpy(h_indices)
+    return (
+        cc.PermutationIterator(d_values, d_indices),
+        h_values.dtype,
+        h_indices.size,
+        int(h_values[h_indices].sum()),
+    )
+
+
+def _iterator_transform(cc):
+    from cuda.compute import types
+    from cuda.compute._cpp_compile import compile_cpp_op_code
+    from cuda.compute.op import RawOp
+
+    num_items = 32
+    source = """
+    extern "C" __device__ void negate_i32(void* input, void* result) {
+        *static_cast<int*>(result) = -*static_cast<int*>(input);
+    }
+    """
+    op = RawOp(ltoir=compile_cpp_op_code(source), name="negate_i32")
+    return (
+        cc.TransformIterator(
+            cc.CountingIterator(np.int32(0)), op, value_type=types.int32
+        ),
+        np.dtype(np.int32),
+        num_items,
+        -sum(range(num_items)),
+    )
+
+
+# A representative subset: the shared-iterator race is IteratorBase's per-instance
+# lazy Op construction (guarded by _op_lock), which is identical for every
+# iterator type, so these cover the structural variety rather than every iterator
+# — a leaf iterator, a compound iterator that wraps a child and memoizes a
+# compiled op (transform), and a compound iterator with two children
+# (permutation). Per-iterator correctness lives in the dedicated iterator tests.
+ITERATOR_FACTORIES = [
+    _iterator_counting,
+    _iterator_permutation,
+    _iterator_transform,
+]
+
+
+@pytest.mark.parametrize(
+    "make_iterator",
+    ITERATOR_FACTORIES,
+    ids=lambda fn: fn.__name__.removeprefix("_iterator_"),
+)
+def test_shared_iterator_object_stress(compute_module, make_iterator):
+    cc = compute_module
+
+    device = Device()
+    device.set_current()
+    device.sync()
+
+    for iteration in range(STRESS_ITERATIONS):
+        cc.clear_all_caches()
+        # Recreate the shared iterator every iteration: its lazy advance/deref
+        # Op construction (guarded by IteratorBase._op_lock) is only racy while
+        # the per-instance caches are cold, so a fresh instance re-arms that
+        # first-construction race on each pass instead of only once per test.
+        shared_iterator, dtype, num_items, expected_sum = make_iterator(cc)
+
+        def make_thread(worker_id):
+            stream = _make_stream()
+            h_init = np.array([worker_id], dtype=dtype)
+            d_out = DeviceArray.empty(1, dtype=dtype)
+
+            def thread(barrier):
+                barrier.wait()
+                reducer = cc.make_reduce_into(
+                    d_in=shared_iterator,
+                    d_out=d_out,
+                    op=cc.OpKind.PLUS,
+                    h_init=h_init,
+                )
+                _call_with_temp(
+                    reducer,
+                    d_in=shared_iterator,
+                    d_out=d_out,
+                    op=cc.OpKind.PLUS,
+                    h_init=h_init,
+                    num_items=num_items,
+                    stream=stream,
+                )
+                stream.sync()
+                assert int(d_out.copy_to_host()[0]) == int(expected_sum + h_init[0])
+
+            return thread
+
+        _run_threaded([make_thread(worker_id) for worker_id in range(STRESS_THREADS)])
+
+
+def test_runtime_ownership_isolation(compute_module):
+    cc = compute_module
+
+    def make_thread(worker_id):
+        def thread(barrier):
+            barrier.wait()
+            stream = _make_stream()
+            h_in = np.arange(16, dtype=np.int32) + worker_id * 10
+            h_init = np.array([worker_id], dtype=np.int32)
+
+            d_in = DeviceArray.from_numpy(h_in)
+            d_reduce_out = DeviceArray.empty(1, dtype=np.int32)
+            d_scan_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+            d_transform_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+            d_hist = DeviceArray.from_numpy(np.zeros(4, dtype=np.int32))
+            h_keys = np.array([3, 1, 2, 1], dtype=np.uint32) + worker_id
+            d_keys_in = DeviceArray.from_numpy(h_keys)
+            d_keys_tmp = DeviceArray.empty(h_keys.shape, h_keys.dtype)
+
+            cc.reduce_into(
+                d_in=d_in,
+                d_out=d_reduce_out,
+                num_items=h_in.size,
+                op=cc.OpKind.PLUS,
+                h_init=h_init,
+                stream=stream,
+            )
+            cc.exclusive_scan(
+                d_in=d_in,
+                d_out=d_scan_out,
+                op=cc.OpKind.PLUS,
+                init_value=h_init,
+                num_items=h_in.size,
+                stream=stream,
+            )
+            cc.unary_transform(
+                d_in=d_in,
+                d_out=d_transform_out,
+                op=cc.OpKind.NEGATE,
+                num_items=h_in.size,
+                stream=stream,
+            )
+            cc.histogram_even(
+                d_samples=d_in,
+                d_histogram=d_hist,
+                num_output_levels=5,
+                lower_level=np.int32(worker_id * 10),
+                upper_level=np.int32(worker_id * 10 + 16),
+                num_samples=h_in.size,
+                stream=stream,
+            )
+            keys = cc.DoubleBuffer(d_keys_in, d_keys_tmp)
+            cc.radix_sort(
+                d_in_keys=keys,
+                d_out_keys=None,
+                d_in_values=None,
+                d_out_values=None,
+                num_items=h_keys.size,
+                order=cc.SortOrder.ASCENDING,
+                stream=stream,
+            )
+
+            stream.sync()
+            assert int(d_reduce_out.copy_to_host()[0]) == int(h_in.sum() + worker_id)
+            expected_scan = np.empty_like(h_in)
+            expected_scan[0] = worker_id
+            expected_scan[1:] = worker_id + np.cumsum(h_in[:-1])
+            np.testing.assert_array_equal(d_scan_out.copy_to_host(), expected_scan)
+            np.testing.assert_array_equal(d_transform_out.copy_to_host(), -h_in)
+            assert int(d_hist.copy_to_host().sum()) == h_in.size
+            np.testing.assert_array_equal(
+                keys.current().copy_to_host(), np.sort(h_keys)
+            )
+
+        return thread
+
+    for _ in range(STRESS_ITERATIONS):
+        _run_threaded([make_thread(worker_id) for worker_id in range(STRESS_THREADS)])
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_func_caching.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_func_caching.py
new file mode 100644
index 00000000..e08807c2
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_func_caching.py
@@ -0,0 +1,209 @@
+import numpy as np
+
+from cuda.compute._caching import CachableFunction, _make_hashable
+
+global_x = 1
+
+
+def test_func_caching_basic():
+    def func(x):
+        return x
+
+    f1 = CachableFunction(func)
+
+    def func(x):
+        return x
+
+    f2 = CachableFunction(func)
+
+    assert f1 == f2
+
+
+def test_func_caching_different_names():
+    def func(x):
+        return x
+
+    f1 = CachableFunction(func)
+
+    def func2(x):
+        return x
+
+    f2 = CachableFunction(func2)
+
+    assert f1 != f2
+
+
+def test_func_caching_different_code():
+    def func(x):
+        return x
+
+    f1 = CachableFunction(func)
+
+    def func(x):
+        return x + 1
+
+    f2 = CachableFunction(func)
+    assert f1 != f2
+
+
+def test_func_caching_with_closure():
+    def factory(x):
+        def func(y):
+            return x + y
+
+        return func
+
+    f1 = CachableFunction(factory(1))
+    f2 = CachableFunction(factory(1))
+    assert f1 == f2
+
+    f3 = CachableFunction(factory(2))
+    assert f1 != f3
+
+
+def test_func_caching_with_numpy_numeric_scalar_closure():
+    def factory(indexlength, regularsize):
+        index_dtype = np.int64
+        idx_len = index_dtype(indexlength)
+        reg_size = index_dtype(regularsize)
+
+        def func(counter):
+            return counter % idx_len + reg_size
+
+        return func
+
+    f1 = CachableFunction(factory(100_000, 16))
+    f2 = CachableFunction(factory(100_000, 16))
+    assert f1 == f2
+
+    f3 = CachableFunction(factory(100_000, 32))
+    assert f1 != f3
+
+
+def test_make_hashable_python_scalars_keyed_by_value():
+    # Regression test for gh-9626: plain Python int/float/bool scalars used to
+    # fall through to ``id(value)``, so two equal-valued but distinct (non-
+    # interned) objects produced different cache keys and missed the cache.
+    # int(str(...)) forces fresh, non-interned objects.
+    a = int(str(10**6))
+    b = int(str(10**6))
+    assert a is not b
+    assert _make_hashable(a) == _make_hashable(b)
+
+    x = float(str(3.5))
+    y = float(str(3.5))
+    assert _make_hashable(x) == _make_hashable(y)
+
+    # Distinct values, types, and bool-vs-int must not collide.
+    assert _make_hashable(a) != _make_hashable(int(str(10**6 + 1)))
+    assert _make_hashable(1) != _make_hashable(1.0)
+    assert _make_hashable(True) != _make_hashable(1)
+
+
+def test_func_caching_with_python_scalar_closure():
+    # gh-9626: closures capturing equal-valued Python scalars must compare
+    # equal so the algorithm build cache hits instead of rebuilding every call.
+    def factory(indexlength, regularsize):
+        # int(str(...)) forces fresh, non-interned int objects.
+        idx_len = int(str(indexlength))
+        reg_size = int(str(regularsize))
+
+        def func(counter):
+            return counter % idx_len + reg_size
+
+        return func
+
+    f1 = CachableFunction(factory(100_000, 16))
+    f2 = CachableFunction(factory(100_000, 16))
+    assert f1 == f2
+
+    f3 = CachableFunction(factory(100_000, 32))
+    assert f1 != f3
+
+
+def test_func_caching_with_global_variable():
+    global global_x
+
+    def func(y):
+        return global_x + y
+
+    f1 = CachableFunction(func)
+    f2 = CachableFunction(func)
+    assert f1 == f2
+
+    global_x = 2
+    f3 = CachableFunction(func)
+    assert f1 != f3
+
+
+def test_func_caching_wrapped_cuda_jit_function():
+    import numba.cuda
+
+    def make_func():
+        @numba.cuda.jit
+        def inner(x):
+            return x
+
+        def func(x):
+            return inner(x) + 1
+
+        return func
+
+    def make_func2():
+        @numba.cuda.jit
+        def inner(x):
+            return 2 * x
+
+        def func(x):
+            return inner(x) + 1
+
+        return func
+
+    func1 = make_func()
+    func2 = make_func()
+    func3 = make_func2()
+
+    assert CachableFunction(func1) == CachableFunction(func2)
+    assert CachableFunction(func1) != CachableFunction(func3)
+
+
+def test_func_caching_with_global_np_ufunc():
+    def make_func():
+        def func(x):
+            return np.argmin(x) + 1
+
+        return func
+
+    def make_func2():
+        def func(x):
+            return np.argmax(x) + 1
+
+        return func
+
+    func1 = make_func()
+    func2 = make_func2()
+
+    assert CachableFunction(func1) != CachableFunction(func2)
+
+
+def test_func_caching_with_aliased_np_ufunc():
+    def make_func1():
+        amin = np.argmin
+
+        def func(x):
+            return amin(x) + 1
+
+        return func
+
+    def make_func2():
+        amax = np.argmax
+
+        def func(x):
+            return amax(x) + 1
+
+        return func
+
+    func1 = make_func1()
+    func2 = make_func2()
+
+    assert CachableFunction(func1) != CachableFunction(func2)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_histogram.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_histogram.py
new file mode 100644
index 00000000..537f102e
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_histogram.py
@@ -0,0 +1,614 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import math
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray
+
+import cuda.compute
+from cuda.compute import (
+    ConstantIterator,
+    CountingIterator,
+    deserialize,
+    make_histogram_even,
+    serialize,
+)
+from cuda.compute._utils.temp_storage_buffer import TempStorageBuffer
+
+DTYPE_LIST = [
+    np.uint8,
+    np.uint16,
+    np.uint32,
+    np.uint64,
+    np.int8,
+    np.int16,
+    np.int32,
+    np.int64,
+    np.float16,
+    np.float32,
+    np.float64,
+]
+
+
+def get_mark(dt, log_size):
+    if log_size + np.log2(np.dtype(dt).itemsize) < 21:
+        return tuple()
+    return pytest.mark.large
+
+
+def type_to_problem_sizes(dtype):
+    if dtype in [np.uint8, np.int8]:
+        return [8, 10, 12, 14]
+    elif dtype in [np.float16, np.uint16, np.int16]:
+        return [10, 12, 14, 16]
+    elif dtype in [np.uint32, np.int32, np.float32]:
+        return [12, 14, 16, 18]
+    elif dtype in [np.uint64, np.int64, np.float64]:
+        return [12, 14, 16, 18]
+    else:
+        raise ValueError("Unsupported dtype")
+
+
+dtype_size_pairs = [
+    pytest.param(dt, 2**log_size, marks=get_mark(dt, log_size))
+    for dt in DTYPE_LIST
+    for log_size in type_to_problem_sizes(dt)
+]
+
+
+def random_int_array(size, dtype):
+    if np.issubdtype(dtype, np.integer):
+        if dtype in [np.uint8, np.int8]:
+            max_val = 126
+        else:
+            max_val = 1024
+        return np.random.randint(0, max_val, size=size).astype(dtype)
+    else:
+        # For floating point, generate values in similar range
+        return (np.random.random(size) * 1024).astype(dtype)
+
+
+def compute_reference_histogram(h_samples, num_levels, lower_level, upper_level):
+    # Filter samples within range [lower_level, upper_level)
+    valid_mask = (h_samples >= lower_level) & (h_samples < upper_level)
+    valid_samples = h_samples[valid_mask]
+
+    if len(valid_samples) == 0:
+        return np.zeros(num_levels - 1, dtype=np.int32)
+
+    # Compute bin indices for valid samples using float arithmetic
+    bin_indices = (
+        (valid_samples.astype(np.float64) - lower_level)
+        * (num_levels - 1)
+        / (upper_level - lower_level)
+    ).astype(int)
+
+    # Ensure indices are within valid range [0, num_levels-2]
+    bin_indices = np.clip(bin_indices, 0, num_levels - 2)
+
+    # Use bincount to get histogram
+    histogram = np.bincount(bin_indices, minlength=num_levels - 1)
+
+    return histogram.astype(np.int32)
+
+
+@pytest.mark.no_verify_sass(reason="LDL/STL instructions emitted for this test.")
+@pytest.mark.parametrize("dtype,num_samples", dtype_size_pairs)
+def test_device_histogram_basic_use(dtype, num_samples):
+    if dtype in [np.uint8, np.int8]:
+        max_level = 126.0
+        max_level_count = 127
+    else:
+        max_level = 1024.0
+        max_level_count = 1025
+
+    num_levels = max_level_count
+    lower_level = dtype(0.0)
+    upper_level = dtype(max_level)
+
+    h_samples = random_int_array(num_samples, dtype)
+    d_samples = DeviceArray.from_numpy(h_samples)
+
+    d_histogram = DeviceArray.from_numpy(np.zeros(num_levels - 1, dtype=np.int32))
+
+    cuda.compute.histogram_even(
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        num_output_levels=num_levels,
+        lower_level=lower_level,
+        upper_level=upper_level,
+        num_samples=num_samples,
+    )
+
+    h_expected = compute_reference_histogram(
+        h_samples, num_levels, lower_level, upper_level
+    )
+    h_result = d_histogram.copy_to_host()
+
+    np.testing.assert_array_equal(h_result, h_expected)
+
+
+@pytest.mark.no_verify_sass(reason="LDL/STL instructions emitted for this test.")
+def test_device_histogram_sample_iterator():
+    max_level_count = 1025
+    num_levels = max_level_count
+    num_bins = num_levels - 1
+
+    samples_per_bin = 10
+    adjusted_total_samples = num_bins * samples_per_bin
+
+    counting_it = CountingIterator(np.int32(0))
+
+    d_histogram = DeviceArray.from_numpy(np.zeros(num_levels - 1, dtype=np.int32))
+
+    # Set up levels so that values 0 to adjusted_total_samples-1 are evenly distributed
+    lower_level = np.int32(0.0)
+    upper_level = np.int32(adjusted_total_samples)
+
+    cuda.compute.histogram_even(
+        d_samples=counting_it,
+        d_histogram=d_histogram,
+        num_output_levels=num_levels,
+        lower_level=lower_level,
+        upper_level=upper_level,
+        num_samples=adjusted_total_samples,
+    )
+
+    # Each bin should have exactly samples_per_bin elements
+    h_expected = np.full(num_bins, samples_per_bin, dtype=np.int32)
+    h_result = d_histogram.copy_to_host()
+
+    np.testing.assert_array_equal(h_result, h_expected)
+
+
+def test_device_histogram_single_sample():
+    h_samples = np.array([5.0], dtype=np.float32)
+    d_samples = DeviceArray.from_numpy(h_samples)
+
+    num_levels = 5
+    lower_level = np.float32(0.0)
+    upper_level = np.float32(10.0)
+
+    d_histogram = DeviceArray.from_numpy(np.zeros(num_levels - 1, dtype=np.int32))
+
+    cuda.compute.histogram_even(
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        num_output_levels=num_levels,
+        lower_level=lower_level,
+        upper_level=upper_level,
+        num_samples=1,
+    )
+
+    # Sample 5.0 should go into bin 2 (bins: [0,2.5), [2.5,5), [5,7.5), [7.5,10))
+    h_expected = np.array([0, 0, 1, 0], dtype=np.int32)
+    h_result = d_histogram.copy_to_host()
+
+    np.testing.assert_array_equal(h_result, h_expected)
+
+
+def test_device_histogram_out_of_range():
+    h_samples = np.array([-1.0, 0.5, 5.5, 10.5, 15.0], dtype=np.float32)
+    d_samples = DeviceArray.from_numpy(h_samples)
+
+    num_levels = 3  # 2 bins: [0,5), [5,10)
+    lower_level = np.float32(0.0)
+    upper_level = np.float32(10.0)
+
+    d_histogram = DeviceArray.from_numpy(np.zeros(num_levels - 1, dtype=np.int32))
+
+    cuda.compute.histogram_even(
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        num_output_levels=num_levels,
+        lower_level=lower_level,
+        upper_level=upper_level,
+        num_samples=len(h_samples),
+    )
+
+    # Only 0.5 (bin 0) and 5.5 (bin 1) should be counted
+    # -1.0, 10.5, and 15.0 are out of range
+    h_expected = np.array([1, 1], dtype=np.int32)
+    h_result = d_histogram.copy_to_host()
+
+    np.testing.assert_array_equal(h_result, h_expected)
+
+
+def test_device_histogram_with_stream(cuda_stream):
+    h_samples = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0], dtype=np.float32)
+    d_samples = DeviceArray.from_numpy(h_samples, stream=cuda_stream)
+
+    num_levels = 5  # 4 bins: [0,2), [2,4), [4,6), [6,8)
+    lower_level = np.float32(0.0)
+    upper_level = np.float32(8.0)
+
+    d_histogram = DeviceArray.from_numpy(
+        np.zeros(num_levels - 1, dtype=np.int32), stream=cuda_stream
+    )
+
+    cuda.compute.histogram_even(
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        num_output_levels=num_levels,
+        lower_level=lower_level,
+        upper_level=upper_level,
+        num_samples=len(h_samples),
+        stream=cuda_stream,
+    )
+
+    h_result = d_histogram.copy_to_host(stream=cuda_stream)
+
+    # Expected: bin 0: [1.0, 2.0), bin 1: [2.0, 4.0), bin 2: [4.0, 6.0), bin 3: [6.0, 8.0)
+    # Values: 1.0->bin0, 2.0->bin1, 3.0->bin1, 4.0->bin2, 5.0->bin2, 6.0->bin3, 7.0->bin3, 8.0->out_of_range
+    h_expected = np.array([1, 2, 2, 2], dtype=np.int32)
+
+    np.testing.assert_array_equal(h_result, h_expected)
+
+
+@pytest.mark.no_verify_sass(reason="LDL/STL instructions emitted for this test.")
+def test_device_histogram_with_constant_iterator():
+    constant_it = ConstantIterator(np.float32(3.0))
+
+    num_samples = 10
+    num_levels = 5  # 4 bins: [0,2), [2,4), [4,6), [6,8)
+    lower_level = np.float32(0.0)
+    upper_level = np.float32(8.0)
+
+    d_histogram = DeviceArray.from_numpy(np.zeros(num_levels - 1, dtype=np.int32))
+
+    cuda.compute.histogram_even(
+        d_samples=constant_it,
+        d_histogram=d_histogram,
+        num_output_levels=num_levels,
+        lower_level=lower_level,
+        upper_level=upper_level,
+        num_samples=num_samples,
+    )
+
+    h_result = d_histogram.copy_to_host()
+
+    # Expected: All 10 samples have value 3.0, which falls in bin 1 [2,4)
+    h_expected = np.array([0, 10, 0, 0], dtype=np.int32)
+
+    np.testing.assert_array_equal(h_result, h_expected)
+
+
+def test_histogram_even():
+    num_samples = 10
+    h_samples = np.array(
+        [2.2, 6.1, 7.1, 2.9, 3.5, 0.3, 2.9, 2.1, 6.1, 999.5], dtype="float32"
+    )
+    d_samples = DeviceArray.from_numpy(h_samples)
+    num_levels = 7
+    d_histogram = DeviceArray.empty(num_levels - 1, np.int32)
+    lower_level = np.float32(0)
+    upper_level = np.float32(12)
+
+    # Run histogram with automatic temp storage allocation
+    cuda.compute.histogram_even(
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        num_output_levels=num_levels,
+        lower_level=lower_level,
+        upper_level=upper_level,
+        num_samples=num_samples,
+    )
+
+    # Check the result is correct
+    h_actual_histogram = d_histogram.copy_to_host()
+    # Calculate expected histogram using numpy
+    h_expected_histogram, _ = np.histogram(
+        h_samples, bins=num_levels - 1, range=(lower_level, upper_level)
+    )
+    h_expected_histogram = h_expected_histogram.astype("int32")
+
+    np.testing.assert_array_equal(h_actual_histogram, h_expected_histogram)
+
+
+def test_histogram_cache_bug_crosses_256_bin_threshold():
+    # GH:#7622
+    # Regression test for a bug where the histogram build artifact for
+    # num_bins <= 256 would be reused for larger bin counts, resulting
+    # in invalid shared memory accesses, because a different shared
+    # memory strategy is used for num_bins > 256.
+    num_samples = 128
+    h_num_output_levels = np.array([0], dtype=np.int32)
+    h_lower_level = np.array([0], dtype=np.int32)
+    h_upper_level = np.array([0], dtype=np.int32)
+
+    # First: 128 bins (uses shared memory, privatized_smem_bins=256)
+    num_bins_1 = 128
+    h_num_output_levels[0] = num_bins_1 + 1
+    h_lower_level[0] = 0
+    h_upper_level[0] = num_bins_1
+
+    h_samples = np.random.randint(0, num_bins_1, size=num_samples, dtype=np.int32)
+    d_samples = DeviceArray.from_numpy(h_samples)
+    d_histogram = DeviceArray.from_numpy(np.zeros(num_bins_1, dtype=np.int32))
+
+    hist = cuda.compute.make_histogram_even(
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=num_samples,
+    )
+    temp_bytes = hist(
+        temp_storage=None,
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=num_samples,
+    )
+    temp_storage = DeviceArray.empty(temp_bytes, np.uint8)
+    hist(
+        temp_storage=temp_storage,
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=num_samples,
+    )
+    assert int(d_histogram.copy_to_host().sum()) == num_samples
+
+    num_bins_2 = 2048
+    h_num_output_levels[0] = num_bins_2 + 1
+    h_lower_level[0] = 0
+    h_upper_level[0] = num_bins_2
+
+    h_samples = np.random.randint(0, num_bins_2, size=num_samples, dtype=np.int32)
+    d_samples = DeviceArray.from_numpy(h_samples)
+    d_histogram = DeviceArray.from_numpy(np.zeros(num_bins_2, dtype=np.int32))
+
+    hist2 = cuda.compute.make_histogram_even(
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=num_samples,
+    )
+
+    temp_bytes2 = hist2(
+        temp_storage=None,
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=num_samples,
+    )
+    temp_storage2 = DeviceArray.empty(temp_bytes2, np.uint8)
+    hist2(
+        temp_storage=temp_storage2,
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=num_samples,
+    )
+    assert int(d_histogram.copy_to_host().sum()) == num_samples
+
+
+def test_histogram_cache_reuses_artifact_when_bounds_change():
+    cuda.compute.clear_all_caches()
+
+    num_samples = 8
+    num_levels = 5
+    d_samples = DeviceArray.from_numpy(np.arange(num_samples, dtype=np.float32))
+    d_histogram = DeviceArray.empty(num_levels - 1, np.int32)
+    h_num_output_levels = np.array([num_levels], dtype=np.int32)
+
+    h_lower_level_1 = np.array([0], dtype=np.float32)
+    h_upper_level_1 = np.array([8], dtype=np.float32)
+    h_lower_level_2 = np.array([10], dtype=np.float32)
+    h_upper_level_2 = np.array([18], dtype=np.float32)
+
+    hist1 = cuda.compute.make_histogram_even(
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level_1,
+        h_upper_level=h_upper_level_1,
+        num_samples=num_samples,
+    )
+    hist2 = cuda.compute.make_histogram_even(
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level_2,
+        h_upper_level=h_upper_level_2,
+        num_samples=num_samples,
+    )
+    assert hist1 is hist2
+
+    d_samples = DeviceArray.from_numpy(np.arange(10, 18, dtype=np.float32))
+    d_histogram.copy_from_host(np.zeros(num_levels - 1, dtype=np.int32))
+    temp_bytes = hist2(
+        temp_storage=None,
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level_2,
+        h_upper_level=h_upper_level_2,
+        num_samples=num_samples,
+    )
+    temp_storage = DeviceArray.empty(temp_bytes, np.uint8)
+    hist2(
+        temp_storage=temp_storage,
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level_2,
+        h_upper_level=h_upper_level_2,
+        num_samples=num_samples,
+    )
+    np.testing.assert_array_equal(
+        d_histogram.copy_to_host(), np.array([2, 2, 2, 2], dtype=np.int32)
+    )
+
+
+def test_histogram_cache_reuses_artifact_for_same_offset_width():
+    cuda.compute.clear_all_caches()
+
+    num_levels = 5
+    d_histogram = DeviceArray.empty(num_levels - 1, np.int32)
+    h_num_output_levels = np.array([num_levels], dtype=np.int32)
+    h_lower_level = np.array([0], dtype=np.float32)
+    h_upper_level = np.array([12], dtype=np.float32)
+
+    hist1 = cuda.compute.make_histogram_even(
+        d_samples=DeviceArray.from_numpy(np.arange(8, dtype=np.float32)),
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=8,
+    )
+    hist2 = cuda.compute.make_histogram_even(
+        d_samples=DeviceArray.from_numpy(np.arange(12, dtype=np.float32)),
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=12,
+    )
+    assert hist1 is hist2
+
+    # Build-only call that crosses v1's offset-width threshold without
+    # allocating or executing that many samples.
+    large_num_samples = math.ceil(
+        np.iinfo(np.int32).max / np.dtype(np.float32).itemsize
+    )
+    hist3 = cuda.compute.make_histogram_even(
+        d_samples=DeviceArray.from_numpy(np.arange(12, dtype=np.float32)),
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=large_num_samples,
+    )
+    assert hist3 is not hist1
+
+    d_samples = DeviceArray.from_numpy(np.arange(12, dtype=np.float32))
+    d_histogram.copy_from_host(np.zeros(num_levels - 1, dtype=np.int32))
+    temp_bytes = hist2(
+        temp_storage=None,
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=12,
+    )
+    temp_storage = DeviceArray.empty(temp_bytes, np.uint8)
+    hist2(
+        temp_storage=temp_storage,
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=12,
+    )
+    np.testing.assert_array_equal(
+        d_histogram.copy_to_host(), np.array([3, 3, 3, 3], dtype=np.int32)
+    )
+
+
+def test_make_histogram_even_rejects_mismatched_bound_dtypes():
+    num_samples = 8
+    d_samples = DeviceArray.from_numpy(np.arange(num_samples, dtype=np.int32))
+    d_histogram = DeviceArray.empty(4, np.int32)
+
+    with pytest.raises(TypeError, match="must have the same dtype"):
+        cuda.compute.make_histogram_even(
+            d_samples=d_samples,
+            d_histogram=d_histogram,
+            h_num_output_levels=np.array([5], dtype=np.int32),
+            h_lower_level=np.array([0], dtype=np.int32),
+            h_upper_level=np.array([8], dtype=np.float32),
+            num_samples=num_samples,
+        )
+
+
+def _run(
+    histogram,
+    *,
+    d_samples,
+    d_histogram,
+    h_num_output_levels,
+    h_lower_level,
+    h_upper_level,
+    num_samples,
+):
+    bytes_needed = histogram(
+        temp_storage=None,
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=num_samples,
+    )
+    tmp = TempStorageBuffer(bytes_needed, None)
+    histogram(
+        temp_storage=tmp,
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=num_samples,
+    )
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_histogram_even_round_trip():
+    num_samples = 10
+    h_samples = np.array(
+        [2.2, 6.1, 7.1, 2.9, 3.5, 0.3, 2.9, 2.1, 6.1, 999.5], dtype="float32"
+    )
+    d_samples = DeviceArray.from_numpy(h_samples)
+    num_levels = 7
+    d_histogram = DeviceArray.empty(num_levels - 1, "int32")
+    h_num_output_levels = np.array([num_levels], dtype=np.int32)
+    h_lower_level = np.array([0.0], dtype=np.float32)
+    h_upper_level = np.array([12.0], dtype=np.float32)
+
+    builder = make_histogram_even(
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=num_samples,
+    )
+    blob = serialize(builder)
+    assert len(blob) > 0
+
+    loaded = deserialize(blob)
+    _run(
+        loaded,
+        d_samples=d_samples,
+        d_histogram=d_histogram,
+        h_num_output_levels=h_num_output_levels,
+        h_lower_level=h_lower_level,
+        h_upper_level=h_upper_level,
+        num_samples=num_samples,
+    )
+
+    expected, _ = np.histogram(
+        h_samples,
+        bins=num_levels - 1,
+        range=(float(h_lower_level[0]), float(h_upper_level[0])),
+    )
+    np.testing.assert_array_equal(d_histogram.copy_to_host(), expected.astype(np.int32))
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_iterators.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_iterators.py
new file mode 100644
index 00000000..c5b57e05
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_iterators.py
@@ -0,0 +1,223 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray
+
+import cuda.compute
+from cuda.compute import OpKind
+from cuda.compute._utils.protocols import (
+    compute_c_contiguous_strides_in_bytes,
+)
+from cuda.compute.iterators import (
+    CacheModifiedInputIterator,
+    ConstantIterator,
+    CountingIterator,
+    ReverseIterator,
+    TransformIterator,
+)
+
+
+def test_constant_iterator_equality():
+    it1 = ConstantIterator(np.int32(0))
+    it2 = ConstantIterator(np.int32(0))
+    it3 = ConstantIterator(np.int32(1))
+    it4 = ConstantIterator(np.int64(0))
+
+    assert it1.kind == it2.kind == it3.kind
+    assert it1.kind != it4.kind
+
+
+def test_counting_iterator_equality():
+    it1 = CountingIterator(np.int32(0))
+    it2 = CountingIterator(np.int32(0))
+    it3 = CountingIterator(np.int32(1))
+    it4 = CountingIterator(np.int64(0))
+
+    assert it1.kind == it2.kind == it3.kind
+    assert it1.kind != it4.kind
+
+
+def test_cache_modified_input_iterator_equality():
+    ary1 = DeviceArray.from_numpy(np.asarray([0, 1, 2], dtype="int32"))
+    ary2 = DeviceArray.from_numpy(np.asarray([3, 4, 5], dtype="int32"))
+    ary3 = DeviceArray.from_numpy(np.asarray([0, 1, 2], dtype="int64"))
+
+    it1 = CacheModifiedInputIterator(ary1, "stream")
+    it2 = CacheModifiedInputIterator(ary1, "stream")
+    it3 = CacheModifiedInputIterator(ary2, "stream")
+    it4 = CacheModifiedInputIterator(ary3, "stream")
+
+    assert it1.kind == it2.kind == it3.kind
+    assert it1.kind != it4.kind
+
+
+def test_equality_transform_iterator():
+    def op1(x):
+        return x
+
+    def op2(x):
+        return 2 * x
+
+    def op3(x):
+        return x
+
+    it = CountingIterator(np.int32(0))
+    it = CountingIterator(np.int32(1))
+    it1 = TransformIterator(it, op1)
+    it2 = TransformIterator(it, op1)
+    it3 = TransformIterator(it, op3)
+
+    assert it1.kind == it2.kind
+    # op3 has a different name than op1, so should have a different kind
+    assert it1.kind != it3.kind
+
+    ary1 = DeviceArray.from_numpy(np.asarray([0, 1, 2]))
+    ary2 = DeviceArray.from_numpy(np.asarray([3, 4, 5]))
+    it4 = TransformIterator(ary1, op1)
+    it5 = TransformIterator(ary1, op1)
+    it6 = TransformIterator(ary1, op2)
+    it7 = TransformIterator(ary1, op3)
+    it8 = TransformIterator(ary2, op1)
+
+    assert it4.kind == it5.kind == it8.kind
+    # op2 has different bytecode, so should have a different kind
+    assert it4.kind != it6.kind
+    # op3 has a different name than op1, so should have a different kind
+    assert it4.kind != it7.kind
+
+
+def test_reverse_input_iterator_equality():
+    ary1 = DeviceArray.from_numpy(np.asarray([0, 1, 2], dtype="int32"))
+    ary2 = DeviceArray.from_numpy(np.asarray([3, 4, 5], dtype="int32"))
+    ary3 = DeviceArray.from_numpy(np.asarray([0, 1, 2], dtype="int64"))
+
+    it1 = ReverseIterator(ary1)
+    it2 = ReverseIterator(ary1)
+    it3 = ReverseIterator(ary2)
+    it4 = ReverseIterator(ary3)
+
+    assert it1.kind == it2.kind == it3.kind
+    assert it1.kind != it4.kind
+
+
+def test_reverse_output_iterator_equality():
+    ary1 = DeviceArray.from_numpy(np.asarray([0, 1, 2], dtype="int32"))
+    ary2 = DeviceArray.from_numpy(np.asarray([3, 4, 5], dtype="int32"))
+    ary3 = DeviceArray.from_numpy(np.asarray([0, 1, 2], dtype="int64"))
+
+    it1 = ReverseIterator(ary1)
+    it2 = ReverseIterator(ary1)
+    it3 = ReverseIterator(ary2)
+    it4 = ReverseIterator(ary3)
+
+    assert it1.kind == it2.kind == it3.kind
+    assert it1.kind != it4.kind
+
+
+@pytest.mark.parametrize(
+    "shape, itemsize, expected",
+    [
+        # Basic 1D
+        ((5,), 4, (4,)),
+        ((10,), 1, (1,)),
+        # Basic 2D
+        ((2, 3), 4, (12, 4)),
+        ((3, 2), 8, (16, 8)),
+        # Basic 3D
+        ((2, 3, 4), 1, (12, 4, 1)),
+        ((2, 3, 4), 2, (24, 8, 2)),
+        # Scalars (0D array)
+        ((), 4, ()),
+        # Shape with a zero-length dimension
+        ((0, 3), 4, (12, 4)),
+        ((3, 0), 4, (0, 4)),
+    ],
+)
+def test_compute_c_contiguous_strides_in_bytes(shape, itemsize, expected):
+    result = compute_c_contiguous_strides_in_bytes(shape, itemsize)
+    assert result == expected
+
+
+@pytest.mark.parametrize(
+    "shape, dtype",
+    [
+        ((2, 3), np.int32),
+        ((4, 5, 6), np.float64),
+        ((10,), np.uint8),
+        ((1,), np.float16),
+    ],
+)
+def test_matches_numpy_strides_for_c_contiguous_arrays(shape, dtype):
+    arr = np.zeros(shape, dtype=dtype, order="C")
+    expected = arr.strides
+    result = compute_c_contiguous_strides_in_bytes(shape, dtype().itemsize)
+    assert result == expected
+
+
+def test_transform_iterator_with_lambda():
+    """Test TransformIterator with a lambda function."""
+    first_item = 10
+    num_items = 100
+
+    # Use a lambda function directly with TransformIterator
+    transform_it = TransformIterator(
+        CountingIterator(np.int32(first_item)), lambda x: x * 2
+    )
+    h_init = np.array([0], dtype=np.int32)
+    d_output = DeviceArray.empty(1, np.int32)
+
+    # Perform reduction on the transformed iterator
+    cuda.compute.reduce_into(
+        d_in=transform_it,
+        d_out=d_output,
+        num_items=num_items,
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+
+    # Expected: sum of (10*2, 11*2, ..., 109*2) = 2 * sum(10..109)
+    expected = 2 * sum(range(first_item, first_item + num_items))
+    assert d_output.copy_to_host()[0] == expected
+
+
+def test_transform_iterator_with_zip_iterator():
+    """Test TransformIterator wrapping ZipIterator (struct types)."""
+    from cuda.compute.iterators import ZipIterator
+
+    # Create a ZipIterator with two int32 arrays
+    h_a = np.arange(10, dtype=np.int32)
+    h_b = np.arange(100, 110, dtype=np.int32)
+    d_a = DeviceArray.from_numpy(h_a)
+    d_b = DeviceArray.from_numpy(h_b)
+
+    zip_it = ZipIterator(d_a, d_b)
+
+    # Create a transform that sums the two fields
+    # Input is a struct with two int32 fields, output is a single int32
+    def sum_fields(pair):
+        return pair[0] + pair[1]
+
+    # Create TransformIterator wrapping ZipIterator
+    # This tests that cpp_type_from_descriptor handles struct types correctly
+    transform_it = TransformIterator(zip_it, sum_fields)
+
+    # Use it in a reduction
+    h_init = np.array([0], dtype=np.int32)
+    d_output = DeviceArray.empty(1, np.int32)
+
+    cuda.compute.reduce_into(
+        d_in=transform_it,
+        d_out=d_output,
+        num_items=len(h_a),
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+
+    result = d_output.copy_to_host()[0]
+    expected = (h_a + h_b).sum()
+
+    assert result == expected, f"Expected {expected}, got {result}"
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_merge_sort.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_merge_sort.py
new file mode 100644
index 00000000..667c64f1
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_merge_sort.py
@@ -0,0 +1,496 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from typing import List
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray
+
+import cuda.compute
+from cuda.compute import (
+    CacheModifiedInputIterator,
+    OpKind,
+    deserialize,
+    gpu_struct,
+    make_merge_sort,
+    serialize,
+)
+from cuda.compute._utils.temp_storage_buffer import TempStorageBuffer
+
+DTYPE_LIST = [
+    np.uint8,
+    np.uint16,
+    np.uint32,
+    np.uint64,
+    np.int8,
+    np.int16,
+    np.int32,
+    np.int64,
+    np.float16,
+    np.float32,
+    np.float64,
+]
+
+
+def random_array(size, dtype, max_value=None) -> np.typing.NDArray:
+    rng = np.random.default_rng()
+    if np.isdtype(dtype, "integral"):
+        if max_value is None:
+            max_value = np.iinfo(dtype).max
+        return rng.integers(max_value, size=size, dtype=dtype)
+    elif np.isdtype(dtype, "real floating"):
+        if dtype == np.float16:  # Cannot generate float16 directly
+            return rng.random(size=size, dtype=np.float32).astype(dtype)
+        else:
+            return rng.random(size=size, dtype=dtype)
+    else:
+        raise ValueError(f"Unsupported dtype {dtype}")
+
+
+def type_to_problem_sizes(dtype) -> List[int]:
+    if dtype in DTYPE_LIST:
+        return [2, 4, 6, 8, 10, 16, 20]
+    else:
+        raise ValueError("Unsupported dtype")
+
+
+def merge_sort_device(
+    d_in_keys, d_in_items, d_out_keys, d_out_items, op, num_items, stream=None
+):
+    cuda.compute.merge_sort(
+        d_in_keys=d_in_keys,
+        d_in_values=d_in_items,
+        d_out_keys=d_out_keys,
+        d_out_values=d_out_items,
+        num_items=num_items,
+        op=op,
+        stream=stream,
+    )
+
+
+def compare_op(lhs, rhs):
+    return np.uint8(lhs < rhs)
+
+
+merge_sort_params = [
+    (dt, 2**log_size, OpKind.LESS if dt == np.float16 else compare_op)
+    for dt in DTYPE_LIST
+    for log_size in type_to_problem_sizes(dt)
+]
+
+
+@pytest.mark.parametrize("dtype,num_items,op", merge_sort_params)
+def test_merge_sort_keys(dtype, num_items, op):
+    h_in_keys = random_array(num_items, dtype)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+
+    merge_sort_device(d_in_keys, None, d_in_keys, None, op, num_items)
+
+    h_out_keys = d_in_keys.copy_to_host()
+    h_in_keys.sort()
+
+    np.testing.assert_array_equal(h_out_keys, h_in_keys)
+
+
+@pytest.mark.parametrize("dtype,num_items,op", merge_sort_params)
+def test_merge_sort_pairs(dtype, num_items, op, monkeypatch):
+    if dtype == np.float16:
+        import cuda.compute._cccl_interop
+
+        monkeypatch.setattr(cuda.compute._cccl_interop, "_check_sass", False)
+
+    h_in_keys = random_array(num_items, dtype)
+    h_in_items = random_array(num_items, np.float32)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_items = DeviceArray.from_numpy(h_in_items)
+
+    merge_sort_device(d_in_keys, d_in_items, d_in_keys, d_in_items, op, num_items)
+
+    h_out_keys = d_in_keys.copy_to_host()
+    h_out_items = d_in_items.copy_to_host()
+
+    argsort = np.argsort(h_in_keys, stable=True)
+    h_in_keys = np.array(h_in_keys)[argsort]
+    h_in_items = np.array(h_in_items)[argsort]
+
+    np.testing.assert_array_equal(h_out_keys, h_in_keys)
+    np.testing.assert_array_equal(h_out_items, h_in_items)
+
+
+@pytest.mark.parametrize("dtype,num_items,op", merge_sort_params)
+def test_merge_sort_keys_copy(dtype, num_items, op):
+    h_in_keys = random_array(num_items, dtype)
+    h_out_keys = np.empty(num_items, dtype=dtype)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_out_keys = DeviceArray.empty(h_out_keys.shape, h_out_keys.dtype)
+
+    merge_sort_device(d_in_keys, None, d_out_keys, None, op, num_items)
+
+    h_out_keys = d_out_keys.copy_to_host()
+    h_in_keys.sort()
+
+    np.testing.assert_array_equal(h_out_keys, h_in_keys)
+
+
+@pytest.mark.parametrize("dtype,num_items,op", merge_sort_params)
+def test_merge_sort_pairs_copy(dtype, num_items, op, monkeypatch):
+    if dtype == np.float16:
+        import cuda.compute._cccl_interop
+
+        monkeypatch.setattr(cuda.compute._cccl_interop, "_check_sass", False)
+
+    h_in_keys = random_array(num_items, dtype)
+    h_in_items = random_array(num_items, np.float32)
+    h_out_keys = np.empty(num_items, dtype=dtype)
+    h_out_items = np.empty(num_items, dtype=np.float32)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_items = DeviceArray.from_numpy(h_in_items)
+    d_out_keys = DeviceArray.empty(h_out_keys.shape, h_out_keys.dtype)
+    d_out_items = DeviceArray.empty(h_out_items.shape, h_out_items.dtype)
+
+    merge_sort_device(d_in_keys, d_in_items, d_out_keys, d_out_items, op, num_items)
+
+    h_out_keys = d_out_keys.copy_to_host()
+    h_out_items = d_out_items.copy_to_host()
+
+    argsort = np.argsort(h_in_keys, stable=True)
+    h_in_keys = np.array(h_in_keys)[argsort]
+    h_in_items = np.array(h_in_items)[argsort]
+
+    np.testing.assert_array_equal(h_out_keys, h_in_keys)
+    np.testing.assert_array_equal(h_out_items, h_in_items)
+
+
+def test_merge_sort_pairs_struct_type():
+    @gpu_struct
+    class key_pair:
+        a: np.int16
+        b: np.uint64
+
+    @gpu_struct
+    class item_pair:
+        a: np.int32
+        b: np.float32
+
+    def struct_compare_op(lhs, rhs):
+        return np.uint8(lhs.b < rhs.b) if lhs.a == rhs.a else np.uint8(lhs.a < rhs.a)
+
+    num_items = 1000
+
+    a_keys = np.random.randint(0, 100, num_items, dtype=np.int16)
+    b_keys = np.random.randint(0, 100, num_items, dtype=np.uint64)
+
+    a_items = np.random.randint(0, 100, num_items, dtype=np.int32)
+    b_items = np.random.rand(num_items).astype(np.float32)
+
+    h_in_keys = np.empty(num_items, dtype=key_pair.dtype)
+    h_in_items = np.empty(num_items, dtype=item_pair.dtype)
+
+    h_in_keys["a"] = a_keys
+    h_in_keys["b"] = b_keys
+
+    h_in_items["a"] = a_items
+    h_in_items["b"] = b_items
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_items = DeviceArray.from_numpy(h_in_items)
+
+    merge_sort_device(
+        d_in_keys, d_in_items, d_in_keys, d_in_items, struct_compare_op, num_items
+    )
+
+    h_out_keys = d_in_keys.copy_to_host()
+    h_out_items = d_in_items.copy_to_host()
+
+    argsort = np.argsort(h_in_keys, stable=True)
+    h_in_keys = np.array(h_in_keys)[argsort]
+    h_in_items = np.array(h_in_items)[argsort]
+
+    np.testing.assert_array_equal(h_out_keys, h_in_keys)
+    np.testing.assert_array_equal(h_out_items, h_in_items)
+
+
+def test_merge_sort_keys_complex():
+    def compare_complex(lhs, rhs):
+        return np.uint8(lhs.real < rhs.real)
+
+    num_items = 10000
+    max_value = 20  # To ensure that the stability property is being tested
+    real = random_array(num_items, np.int64, max_value)
+    imaginary = random_array(num_items, np.int64, max_value)
+
+    h_in_keys = real + 1j * imaginary
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+
+    merge_sort_device(d_in_keys, None, d_in_keys, None, compare_complex, num_items)
+
+    h_out_keys = d_in_keys.copy_to_host()
+    h_in_keys = h_in_keys[np.argsort(h_in_keys.real, stable=True)]
+
+    np.testing.assert_array_equal(h_out_keys, h_in_keys)
+
+
+@pytest.mark.parametrize("dtype,num_items,op", merge_sort_params)
+def test_merge_sort_keys_copy_iterator_input(dtype, num_items, op):
+    h_in_keys = random_array(num_items, dtype)
+    h_out_keys = np.empty(num_items, dtype=dtype)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_out_keys = DeviceArray.empty(h_out_keys.shape, h_out_keys.dtype)
+
+    i_input = CacheModifiedInputIterator(d_in_keys, modifier="stream")
+
+    merge_sort_device(i_input, None, d_out_keys, None, op, num_items)
+
+    h_in_keys.sort()
+    h_out_keys = d_out_keys.copy_to_host()
+
+    np.testing.assert_array_equal(h_out_keys, h_in_keys)
+
+
+@pytest.mark.parametrize("dtype,num_items,op", merge_sort_params)
+def test_merge_sort_pairs_copy_iterator_input(dtype, num_items, op, monkeypatch):
+    if dtype == np.float16:
+        import cuda.compute._cccl_interop
+
+        monkeypatch.setattr(cuda.compute._cccl_interop, "_check_sass", False)
+
+    h_in_keys = random_array(num_items, dtype)
+    h_in_items = random_array(num_items, np.float32)
+    h_out_keys = np.empty(num_items, dtype=dtype)
+    h_out_items = np.empty(num_items, dtype=np.float32)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_items = DeviceArray.from_numpy(h_in_items)
+    d_out_keys = DeviceArray.empty(h_out_keys.shape, h_out_keys.dtype)
+    d_out_items = DeviceArray.empty(h_out_items.shape, h_out_items.dtype)
+
+    i_input_keys = CacheModifiedInputIterator(d_in_keys, modifier="stream")
+    i_input_items = CacheModifiedInputIterator(d_in_items, modifier="stream")
+
+    merge_sort_device(
+        i_input_keys, i_input_items, d_out_keys, d_out_items, op, num_items
+    )
+
+    h_out_keys = d_out_keys.copy_to_host()
+    h_out_items = d_out_items.copy_to_host()
+
+    argsort = np.argsort(h_in_keys, stable=True)
+    h_in_keys = np.array(h_in_keys)[argsort]
+    h_in_items = np.array(h_in_items)[argsort]
+
+    np.testing.assert_array_equal(h_out_keys, h_in_keys)
+    np.testing.assert_array_equal(h_out_items, h_in_items)
+
+
+def test_merge_sort_with_stream(cuda_stream):
+    num_items = 10000
+
+    h_in_keys = random_array(num_items, np.int32)
+    d_in_keys = DeviceArray.from_numpy(h_in_keys, stream=cuda_stream)
+    d_out_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype, stream=cuda_stream)
+
+    merge_sort_device(
+        d_in_keys, None, d_out_keys, None, compare_op, num_items, stream=cuda_stream
+    )
+
+    got = d_out_keys.copy_to_host(stream=cuda_stream)
+    h_in_keys.sort()
+
+    np.testing.assert_array_equal(got, h_in_keys)
+
+
+def test_merge_sort_well_known_less():
+    dtype = np.int32
+
+    h_in_keys = np.array([5, 2, 8, 1, 9, 3], dtype=dtype)
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_out_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype)
+
+    cuda.compute.merge_sort(
+        d_in_keys=d_in_keys,
+        d_in_values=None,
+        d_out_keys=d_out_keys,
+        d_out_values=None,
+        num_items=len(h_in_keys),
+        op=OpKind.LESS,
+    )
+
+    expected = np.array([1, 2, 3, 5, 8, 9])
+    np.testing.assert_equal(d_out_keys.copy_to_host(), expected)
+
+
+def test_merge_sort_well_known_greater():
+    dtype = np.int32
+
+    h_in_keys = np.array([5, 2, 8, 1, 9, 3], dtype=dtype)
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_out_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype)
+
+    cuda.compute.merge_sort(
+        d_in_keys=d_in_keys,
+        d_in_values=None,
+        d_out_keys=d_out_keys,
+        d_out_values=None,
+        num_items=len(h_in_keys),
+        op=OpKind.GREATER,
+    )
+
+    expected = np.array([9, 8, 5, 3, 2, 1])
+    np.testing.assert_equal(d_out_keys.copy_to_host(), expected)
+
+
+def test_merge_sort_large_temp_storage_not_negative():
+    """Regression test for https://github.com/NVIDIA/cccl/issues/7911.
+
+    temp_storage_bytes was returned as a signed 32-bit int, overflowing
+    to a negative value for large inputs requiring >2GB temp storage.
+    """
+    num_items = 2**28
+    dtype = np.int64
+    d_in_keys = DeviceArray.empty(num_items, dtype)
+    d_out_keys = DeviceArray.empty(num_items, dtype)
+
+    sorter = cuda.compute.make_merge_sort(
+        d_in_keys=d_in_keys,
+        d_in_values=None,
+        d_out_keys=d_out_keys,
+        d_out_values=None,
+        op=OpKind.LESS,
+    )
+
+    temp_storage_bytes = sorter(
+        temp_storage=None,
+        d_in_keys=d_in_keys,
+        d_in_values=None,
+        d_out_keys=d_out_keys,
+        d_out_values=None,
+        op=OpKind.LESS,
+        num_items=num_items,
+    )
+
+    assert temp_storage_bytes > 0
+
+
+def test_merge_sort_with_values_well_known():
+    dtype = np.int32
+
+    h_in_keys = np.array([3, 1, 4, 2], dtype=dtype)
+    h_in_values = np.array([30, 10, 40, 20], dtype=dtype)
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_values = DeviceArray.from_numpy(h_in_values)
+    d_out_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype)
+    d_out_values = DeviceArray.empty(h_in_values.shape, h_in_values.dtype)
+
+    cuda.compute.merge_sort(
+        d_in_keys=d_in_keys,
+        d_in_values=d_in_values,
+        d_out_keys=d_out_keys,
+        d_out_values=d_out_values,
+        num_items=len(h_in_keys),
+        op=OpKind.LESS,
+    )
+
+    expected_keys = np.array([1, 2, 3, 4])
+    expected_values = np.array([10, 20, 30, 40])
+    np.testing.assert_equal(d_out_keys.copy_to_host(), expected_keys)
+    np.testing.assert_equal(d_out_values.copy_to_host(), expected_values)
+
+
+def _run(sorter, *, d_in_keys, d_in_values, d_out_keys, d_out_values, num_items, op):
+    bytes_needed = sorter(
+        temp_storage=None,
+        d_in_keys=d_in_keys,
+        d_in_values=d_in_values,
+        d_out_keys=d_out_keys,
+        d_out_values=d_out_values,
+        num_items=num_items,
+        op=op,
+    )
+    tmp = TempStorageBuffer(bytes_needed, None)
+    sorter(
+        temp_storage=tmp,
+        d_in_keys=d_in_keys,
+        d_in_values=d_in_values,
+        d_out_keys=d_out_keys,
+        d_out_values=d_out_values,
+        num_items=num_items,
+        op=op,
+    )
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_merge_sort_keys_values():
+    h_in_keys = np.array([-5, 0, 2, -3, 2, 4, 0, -1, 2, 8], dtype="int32")
+    h_in_values = np.array(
+        [-3.2, 2.2, 1.9, 4.0, -3.9, 2.7, 0, 8.3 - 1, 2.9, 5.4], dtype="float32"
+    )
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_values = DeviceArray.from_numpy(h_in_values)
+    d_out_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype)
+    d_out_values = DeviceArray.empty(h_in_values.shape, h_in_values.dtype)
+
+    builder = make_merge_sort(
+        d_in_keys=d_in_keys,
+        d_in_values=d_in_values,
+        d_out_keys=d_out_keys,
+        d_out_values=d_out_values,
+        op=OpKind.LESS,
+    )
+    blob = serialize(builder)
+    assert len(blob) > 0
+
+    loaded = deserialize(blob)
+    _run(
+        loaded,
+        d_in_keys=d_in_keys,
+        d_in_values=d_in_values,
+        d_out_keys=d_out_keys,
+        d_out_values=d_out_values,
+        num_items=h_in_keys.size,
+        op=OpKind.LESS,
+    )
+
+    # kind="stable" works on all supported NumPy versions; the stable= keyword
+    # was only added in NumPy 2.0 and cuda-cccl pins no numpy floor.
+    argsort = np.argsort(h_in_keys, kind="stable")
+    np.testing.assert_array_equal(d_out_keys.copy_to_host(), h_in_keys[argsort])
+    np.testing.assert_array_equal(d_out_values.copy_to_host(), h_in_values[argsort])
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_merge_sort_keys_only():
+    # Keys-only: d_in_values / d_out_values are None, which become "none"
+    # iterators — the plain ITER schema members round-trip them fine.
+    h_in_keys = np.array([5, 2, 8, 1, 9, 3, 7, 0, 6, 4], dtype="int32")
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_out_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype)
+
+    builder = make_merge_sort(
+        d_in_keys=d_in_keys,
+        d_in_values=None,
+        d_out_keys=d_out_keys,
+        d_out_values=None,
+        op=OpKind.LESS,
+    )
+    blob = serialize(builder)
+    assert len(blob) > 0
+
+    loaded = deserialize(blob)
+    _run(
+        loaded,
+        d_in_keys=d_in_keys,
+        d_in_values=None,
+        d_out_keys=d_out_keys,
+        d_out_values=None,
+        num_items=h_in_keys.size,
+        op=OpKind.LESS,
+    )
+
+    np.testing.assert_array_equal(d_out_keys.copy_to_host(), np.sort(h_in_keys))
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_multi_cc_serialization.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_multi_cc_serialization.py
new file mode 100644
index 00000000..8e8cc819
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_multi_cc_serialization.py
@@ -0,0 +1,1051 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Tests for multi-compute-capability / no-GPU ahead-of-time compilation.
+
+``make_<algo>(..., compute_capability=...)`` compiles one build result per target
+arch without touching the CUDA driver (so it works on a build machine with no
+GPU, using ``ProxyArray`` / ``ProxyValue`` placeholders). The resulting object
+holds a ``{cc_key: build_result}`` map; the matching build result is loaded lazily on
+the first call on a real device, and the whole map round-trips through
+serialize/deserialize.
+
+The executable tests use ``unary_transform`` as a representative algorithm and
+discover which target arches this toolchain can actually compile for, so they
+adapt to the installed CTK rather than hard-coding architectures.
+"""
+
+import concurrent.futures
+import threading
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray
+
+import cuda.compute._cccl_interop as cccl_interop
+from cuda.compute import (
+    ProxyArray,
+    deserialize,
+    make_select,
+    make_unary_transform,
+    serialize,
+)
+from cuda.compute._caching import (
+    _PerCCBuildResults,
+    cache_build_results,
+    cache_with_registered_key_functions,
+)
+from cuda.compute._cccl_interop import (
+    cc_to_key,
+    current_device_cc_key,
+    key_to_cc,
+    normalize_compute_capabilities,
+)
+from cuda.compute._target_cc import target_cc
+from cuda.compute.iterators import CountingIterator, TransformIterator
+from cuda.compute.types import from_numpy_dtype
+from cuda.core import Device
+from cuda.core.system import get_num_devices
+
+try:
+    from cuda.compute._build_info import USING_V2
+except ImportError:
+    USING_V2 = False
+
+# Marker for tests that exercise real v1 serialization or NVRTC/LTO-IR
+# compilation. The fake-backed protocol tests (``_FakeBuildResult`` +
+# ``_PerCCBuildResults``) and pure-logic tests below are backend-agnostic and
+# deliberately run on both backends, so the skip is applied per-test here
+# rather than to the whole module.
+requires_serialization = pytest.mark.skipif(
+    USING_V2, reason="serialization not supported on v2 (HostJIT) backend"
+)
+
+
+def _add_one(a):
+    return a + 1
+
+
+# ----------------------------------------------------------------------------
+# Pure-logic tests (no GPU, no build result needed)
+# ----------------------------------------------------------------------------
+
+
+def test_normalize_compute_capabilities():
+    assert normalize_compute_capabilities(None) is None
+    assert normalize_compute_capabilities(90) == [(9, 0)]
+    assert normalize_compute_capabilities((8, 0)) == [(8, 0)]
+    assert normalize_compute_capabilities("75") == [(7, 5)]
+    # list of packed ints, deduplicated + sorted
+    assert normalize_compute_capabilities([90, 80, 90]) == [(8, 0), (9, 0)]
+    # list of pairs
+    assert normalize_compute_capabilities([(9, 0), (8, 0)]) == [(8, 0), (9, 0)]
+
+
+def test_cc_key_roundtrip():
+    for cc in (70, 75, 80, 86, 90, 120):
+        assert cc_to_key(key_to_cc(cc)) == cc
+
+
+def test_proxy_array_data_pointer_raises():
+    p = ProxyArray(np.float32)
+    with pytest.raises(RuntimeError, match="build-time placeholder"):
+        _ = p.__cuda_array_interface__["data"]
+
+
+class _FakeBuildResult:
+    def __init__(self, payload: bytes, load_gate=None, load_failures=0):
+        self.payload = payload
+        self.load_gate = load_gate
+        self.load_failures = load_failures
+        self.load_count = 0
+        self.serialize_count = 0
+        self._loaded = False
+
+    def load(self):
+        self.load_count += 1
+        if self.load_gate is not None:
+            started, proceed = self.load_gate
+            started.set()
+            assert proceed.wait(timeout=5)
+        if self.load_count <= self.load_failures:
+            raise RuntimeError("synthetic build-result load failure")
+        self._loaded = True
+
+    def serialize(self):
+        self.serialize_count += 1
+        return self.payload
+
+    @classmethod
+    def deserialize(cls, blob, load=True, check_cc=True):
+        del check_cc
+        result = cls(blob)
+        if load:
+            result.load()
+        return result
+
+
+class _CoordinatedGetCache(dict):
+    """Make concurrent callers all observe the initial cache miss."""
+
+    def __init__(self, participants):
+        super().__init__()
+        self._get_barrier = threading.Barrier(participants)
+        self._setdefault_barrier = threading.Barrier(participants)
+
+    def get(self, key, default=None):
+        value = super().get(key, default)
+        self._get_barrier.wait(timeout=5)
+        return value
+
+    def setdefault(self, key, default=None):
+        value = super().setdefault(key, default)
+        self._setdefault_barrier.wait(timeout=5)
+        return value
+
+
+class _TrackingLock:
+    """Signal when a second caller tries to acquire an already-held lock."""
+
+    def __init__(self):
+        self.second_attempt = threading.Event()
+        self._attempt_count = 0
+        self._attempt_count_lock = threading.Lock()
+        self._lock = threading.Lock()
+
+    def __enter__(self):
+        with self._attempt_count_lock:
+            self._attempt_count += 1
+            if self._attempt_count == 2:
+                self.second_attempt.set()
+        self._lock.acquire()
+        return self
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self._lock.release()
+
+
+def test_aot_build_results_load_once_per_device_without_recompiling():
+    """Owner device loads the canonical result in place; a second same-cc device
+    loads an independent serialize -> deserialize -> load clone. Each is cached
+    per device, so repeat resolves reuse it without re-loading or recompiling.
+    """
+    source = _FakeBuildResult(b"sm80-cubin")
+    build_results = _PerCCBuildResults({80: source})
+
+    device_0_result = build_results.resolve(80, 0)
+    device_1_result = build_results.resolve(80, 1)
+
+    assert device_0_result is source
+    assert device_1_result is not source
+    assert build_results.resolve(80, 0) is device_0_result
+    assert build_results.resolve(80, 1) is device_1_result
+    assert source.load_count == 1
+    assert source.serialize_count == 1
+    assert device_1_result.load_count == 1
+
+
+def test_aot_build_result_concurrent_load_is_coalesced_per_device():
+    """Concurrent first-load requests for one device coalesce through
+    single-flight to a single load, and every caller receives that same
+    loaded result.
+    """
+    load_started = threading.Event()
+    allow_load = threading.Event()
+    source = _FakeBuildResult(b"sm80-cubin", (load_started, allow_load))
+    build_results = _PerCCBuildResults({80: source})
+    barrier = threading.Barrier(4)
+
+    def resolve():
+        barrier.wait()
+        return build_results.resolve(80, 0)
+
+    with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
+        futures = [executor.submit(resolve) for _ in range(4)]
+        assert load_started.wait(timeout=5)
+        allow_load.set()
+        results = [future.result() for future in futures]
+
+    assert all(result is source for result in results)
+    assert source.load_count == 1
+
+
+def test_aot_build_result_load_failure_is_shared_and_retryable():
+    """A failing first load hands every concurrent waiter the same exception and
+    removes the failed entry, so a later call retries and can succeed.
+    """
+    thread_count = 4
+    source = _FakeBuildResult(b"sm80-cubin", load_failures=1)
+    build_results = _PerCCBuildResults({80: source})
+    build_results._loaded_results = _CoordinatedGetCache(thread_count)
+    barrier = threading.Barrier(thread_count)
+
+    def resolve():
+        barrier.wait()
+        try:
+            build_results.resolve(80, 0)
+        except RuntimeError as error:
+            return error
+        raise AssertionError("the first load must fail")
+
+    with concurrent.futures.ThreadPoolExecutor(max_workers=thread_count) as executor:
+        errors = [
+            future.result()
+            for future in [executor.submit(resolve) for _ in range(thread_count)]
+        ]
+
+    assert len({id(error) for error in errors}) == 1
+    assert str(errors[0]) == "synthetic build-result load failure"
+    assert source.load_count == 1
+
+    # A failed in-flight entry is removed. A later caller retries the load and
+    # can cache the successful result.
+    build_results._loaded_results = {}
+    assert build_results.resolve(80, 0) is source
+    assert source.load_count == 2
+    assert source._loaded
+
+
+def test_aot_serialization_waits_for_canonical_first_load():
+    """serialize_build_result() takes the same per-cc source lock as the first
+    in-place load, so a serialize started mid-load blocks until the load
+    finishes instead of reading half-loaded state.
+    """
+    load_started = threading.Event()
+    allow_load = threading.Event()
+    source = _FakeBuildResult(b"sm80-cubin", (load_started, allow_load))
+    build_results = _PerCCBuildResults({80: source})
+    source_lock = _TrackingLock()
+    build_results._source_locks[80] = source_lock
+
+    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
+        load_future = executor.submit(build_results.resolve, 80, 0)
+        assert load_started.wait(timeout=5)
+        serialize_future = executor.submit(build_results.serialize_build_result, 80)
+        try:
+            assert source_lock.second_attempt.wait(timeout=5)
+            assert source.serialize_count == 0
+        finally:
+            allow_load.set()
+
+        assert load_future.result() is source
+        assert serialize_future.result() == b"sm80-cubin"
+
+    assert source.serialize_count == 1
+
+
+def _patch_default_build_env(monkeypatch, device, *, backend_serializes):
+    import cuda.compute._caching as _caching_mod
+
+    monkeypatch.setattr(
+        _caching_mod, "_get_current_device_info", lambda: (device["id"], (8, 0))
+    )
+    monkeypatch.setattr(_caching_mod, "_process_wide_build_results_cache", {})
+    # Pin the backend capability so these fake-backed tests exercise a fixed
+    # branch regardless of which backend the suite runs against.
+    monkeypatch.setattr(
+        _caching_mod, "_BACKEND_SERIALIZES_BUILD_RESULTS", backend_serializes
+    )
+
+
+def _make_fake_default_builder(device, built_sources):
+    def build():
+        source = _FakeBuildResult(b"sm80-cubin")
+        source._loaded = True
+        built_sources.append(source)
+        return _PerCCBuildResults({80: source}, loaded_device_id=device["id"])
+
+    return build
+
+
+def test_default_build_second_same_cc_device_shares_the_entry(monkeypatch):
+    """A default build on a second same-cc device reuses the shared entry.
+
+    Default builds are keyed by compute capability alone, so a second same-cc
+    device hits the same _PerCCBuildResults; its construction-time binding
+    loads a clone of the compiled payload (serialize -> deserialize -> load)
+    instead of running the builder.
+    """
+    device = {"id": 0}
+    _patch_default_build_env(monkeypatch, device, backend_serializes=True)
+    built_sources = []
+    build = _make_fake_default_builder(device, built_sources)
+
+    first, first_bound = cache_build_results(
+        _FakeBuildResult, "spec", compute_capability=None, builder=build
+    )
+    assert len(built_sources) == 1
+    assert first_bound is first[80]
+
+    # Same device: plain cache hit, no builder, warm binding.
+    again, again_bound = cache_build_results(
+        _FakeBuildResult, "spec", compute_capability=None, builder=build
+    )
+    assert again is first
+    assert again_bound is first_bound
+    assert len(built_sources) == 1
+    assert first[80].serialize_count == 0
+
+    # Second same-cc device: same shared entry; the binding clones the
+    # compiled payload, and the builder never runs.
+    device["id"] = 1
+    second, second_bound = cache_build_results(
+        _FakeBuildResult, "spec", compute_capability=None, builder=build
+    )
+    assert len(built_sources) == 1
+    assert second is first
+    assert first[80].serialize_count == 1
+    assert second_bound is not first_bound
+    assert second_bound._loaded
+
+    # A different specialization has no cached entry and builds from scratch.
+    device["id"] = 0
+    cache_build_results(
+        _FakeBuildResult, "other-spec", compute_capability=None, builder=build
+    )
+    assert len(built_sources) == 2
+
+
+def test_default_build_keys_per_device_without_backend_serialization(monkeypatch):
+    """Without build-result serialization, each device builds its own entry.
+
+    A non-owner device loads a shared entry by cloning it through
+    serialization, so a backend that cannot serialize (v2 HostJIT today) keys
+    default builds per device ordinal instead of sharing them.
+    TODO: delete this test once v2 supports build-result serialization and
+    the per-ordinal branch is removed.
+    """
+    device = {"id": 0}
+    _patch_default_build_env(monkeypatch, device, backend_serializes=False)
+    built_sources = []
+    build = _make_fake_default_builder(device, built_sources)
+
+    first, first_bound = cache_build_results(
+        _FakeBuildResult, "spec", compute_capability=None, builder=build
+    )
+    assert len(built_sources) == 1
+    assert first_bound is first[80]
+
+    # Second same-cc device: its own key, its own build; nothing serialized.
+    device["id"] = 1
+    second, second_bound = cache_build_results(
+        _FakeBuildResult, "spec", compute_capability=None, builder=build
+    )
+    assert len(built_sources) == 2
+    assert second is not first
+    assert second_bound is second[80]
+    assert first[80].serialize_count == 0
+
+
+def test_default_build_binding_falls_back_to_full_build_when_cloning_fails(
+    monkeypatch,
+):
+    """A failed clone degrades to a full build for that device.
+
+    Cloning is only an optimization: if resolving the second device's binding
+    fails (here: the canonical result cannot serialize), _bind_default_build
+    runs the builder for that device and records the result in the shared
+    per-device slot.
+    """
+    device = {"id": 0}
+    _patch_default_build_env(monkeypatch, device, backend_serializes=True)
+    built_sources = []
+    build = _make_fake_default_builder(device, built_sources)
+
+    first, first_bound = cache_build_results(
+        _FakeBuildResult, "spec", compute_capability=None, builder=build
+    )
+
+    def broken_serialize():
+        raise RuntimeError("synthetic serialization failure")
+
+    monkeypatch.setattr(first[80], "serialize", broken_serialize)
+
+    device["id"] = 1
+    second, second_bound = cache_build_results(
+        _FakeBuildResult, "spec", compute_capability=None, builder=build
+    )
+    assert second is first
+    assert len(built_sources) == 2
+    assert second_bound is built_sources[1]
+    assert second_bound._loaded
+    # The fallback result is recorded so same-device callers share it.
+    third, third_bound = cache_build_results(
+        _FakeBuildResult, "spec", compute_capability=None, builder=build
+    )
+    assert third is first
+    assert third_bound is second_bound
+    assert len(built_sources) == 2
+
+
+# ----------------------------------------------------------------------------
+# Helpers to build with proxies (no GPU data) and discover compilable arches
+# ----------------------------------------------------------------------------
+
+
+def _make_transform(cc):
+    """Compile a unary transform for compute capability/ies ``cc`` using proxies."""
+    return make_unary_transform(
+        d_in=ProxyArray(np.int32),
+        d_out=ProxyArray(np.int32),
+        op=_add_one,
+        compute_capability=cc,
+    )
+
+
+def _compilable_ccs(want=2):
+    """Return up to ``want`` cc keys this toolchain can compile for.
+
+    Device-free: compilation does not require a GPU, so this discovers arches
+    without querying the current device (used by the no-GPU proxy tests).
+    """
+    candidates = [90, 100, 89, 120, 80, 86, 75]
+    found = []
+    for cc in dict.fromkeys(candidates):  # de-dupe, preserve order
+        try:
+            _make_transform(cc)
+        except Exception:
+            continue
+        found.append(cc)
+        if len(found) == want:
+            break
+    return found
+
+
+# ----------------------------------------------------------------------------
+# No-GPU compile via proxies (needs the built extension, but no device data)
+# ----------------------------------------------------------------------------
+
+
+@requires_serialization
+def test_proxy_single_cc_compile_is_not_loaded():
+    ccs = _compilable_ccs(1)
+    if not ccs:
+        pytest.skip("toolchain compiles for no target arch")
+    cc = ccs[0]
+    builder = _make_transform(cc)
+    assert set(builder.build_results.keys()) == {cc}
+    # compile() must not touch the driver — no kernels loaded yet.
+    assert not builder.build_results[cc]._loaded
+
+
+@requires_serialization
+def test_proxy_multi_cc_compile_produces_one_build_result_per_cc():
+    ccs = _compilable_ccs(2)
+    if len(ccs) < 2:
+        pytest.skip("toolchain compiles for <2 target arches")
+    builder = _make_transform(ccs)
+    assert set(builder.build_results.keys()) == set(ccs)
+    assert all(not br._loaded for br in builder.build_results.values())
+
+
+@requires_serialization
+def test_multi_cc_serialize_deserialize_roundtrip_defers_load():
+    ccs = _compilable_ccs(2)
+    if len(ccs) < 2:
+        pytest.skip("toolchain compiles for <2 target arches")
+    builder = _make_transform(ccs)
+    blob = serialize(builder)
+    assert len(blob) > 0
+
+    loaded = deserialize(blob)
+    assert set(loaded.build_results.keys()) == set(ccs)
+    # Deserialized build results are not loaded (load is deferred to first call) — so
+    # a multi-arch artifact stays portable and needs no live device to load.
+    assert all(not br._loaded for br in loaded.build_results.values())
+
+
+# ----------------------------------------------------------------------------
+# Iterator device code must be recompiled per target arch, even when the same
+# iterator instance is reused across builds (no GPU needed)
+# ----------------------------------------------------------------------------
+
+
+def _counting_case():
+    """Leaf iterator: the arch-specific bytes are its own advance/deref ops,
+    memoized by IteratorBase's per-op caches."""
+
+    def make():
+        return CountingIterator(np.int32(0))
+
+    def arch_bytes(it):
+        # CountingIterator op symbol names derive only from (type, value_type),
+        # so these LTO-IR blobs are byte-stable across instances at a given arch.
+        return (
+            bytes(it.get_advance_op().code.op_bytes),
+            bytes(it.get_input_deref_op().code.op_bytes),
+        )
+
+    return make, arch_bytes
+
+
+def _transform_case():
+    """Compound iterator: the arch-specific leak surface is the compiled transform
+    op, memoized by TransformIterator._compiled_op and embedded as an extra_ltoir.
+
+    (The top-level wrapper LTO-IR is deliberately excluded: its symbol name is
+    instance-dependent, so it is not byte-stable across instances. It is always
+    recompiled per arch anyway; the compiled op below is the byte we must guard.)
+    """
+
+    def make():
+        return TransformIterator(CountingIterator(np.int32(0)), _add_one)
+
+    def arch_bytes(it):
+        it.get_input_deref_op()  # drive the real path that compiles the op
+        # wrapped_<fn> symbol names are content-based, so byte-stable per arch.
+        return (bytes(it._get_compiled_op().code.op_bytes),)
+
+    return make, arch_bytes
+
+
+@requires_serialization
+@pytest.mark.parametrize(
+    "case", [_counting_case(), _transform_case()], ids=["counting", "transform"]
+)
+def test_iterator_op_ltoir_tracks_target_cc_across_instance_reuse(case):
+    """Reusing one iterator instance across builds targeting different arches
+    must recompile its device code for each arch.
+
+    Iterators memoize their compiled ops on the instance (``IteratorBase``'s
+    per-op caches and ``TransformIterator._compiled_op``). If that memo is not
+    keyed on the target compute capability, passing the same iterator object to
+    two ``make_<algo>`` calls whose targets *decrease* (e.g. 90 then 80) leaves
+    the first build's newer-arch LTO-IR cached on the instance; it then gets
+    linked into the older-arch build result, which nvJitLink rejects (final SM
+    must be >= every linked input's arch). Guard against that by requiring the
+    reused instance to yield, for each target, the same LTO-IR a fresh instance
+    produces.
+    """
+    make, arch_bytes = case
+    ccs = _compilable_ccs(2)
+    if len(ccs) < 2:
+        pytest.skip("toolchain compiles for <2 target arches")
+    hi, lo = sorted(ccs, reverse=True)  # build the higher arch first
+
+    # Ground truth: fresh instances compiled directly for each target arch.
+    ref = {}
+    for cc in (hi, lo):
+        with target_cc(cc):
+            ref[cc] = arch_bytes(make())
+    # The two arches must produce different device code, else the test below
+    # could pass vacuously.
+    assert ref[hi] != ref[lo]
+
+    # Reuse a single instance across both targets, higher arch first.
+    it = make()
+    with target_cc(hi):
+        assert arch_bytes(it) == ref[hi]
+    with target_cc(lo):
+        # Regression: the memoized hi-arch LTO-IR must not leak into the lo build.
+        assert arch_bytes(it) == ref[lo]
+
+
+@requires_serialization
+def test_default_build_keys_device_code_on_resolved_device_cc(monkeypatch):
+    ccs = _compilable_ccs(2)
+    if len(ccs) < 2:
+        pytest.skip("toolchain compiles for <2 target arches")
+    lo_cc, hi_cc = key_to_cc(sorted(ccs)[0]), key_to_cc(sorted(ccs)[-1])
+
+    import cuda.compute._caching as _caching_mod
+
+    class _FakeDevice:
+        cc = hi_cc  # flipped between builds to emulate two different devices
+        device_id = 0
+
+        def __init__(self, *a, **k):
+            pass
+
+        @property
+        def compute_capability(self):
+            return _FakeDevice.cc
+
+    monkeypatch.setattr(_caching_mod, "Device", _FakeDevice)
+
+    # A cc-cached build that exercises the iterator's deref op under the default
+    # (compute_capability=None) path, returning the build's target cc and LTO-IR.
+    @cache_with_registered_key_functions
+    def _build(*, it, compute_capability=None):
+        from cuda.compute._target_cc import get_target_cc
+
+        return get_target_cc(), bytes(it.get_input_deref_op().code.op_bytes)
+
+    it = CountingIterator(np.int32(0))
+    _FakeDevice.cc = hi_cc
+    seen_hi, bytes_hi = _build(it=it)
+    _FakeDevice.cc = lo_cc
+    seen_lo, bytes_lo = _build(it=it)
+
+    # The build's target cc is the resolved device cc, never None.
+    assert seen_hi == hi_cc and seen_lo == lo_cc
+    # Regression: the second device must not reuse the first device's arch.
+    assert bytes_hi != bytes_lo
+    assert set(it._input_deref_op.keys()) == {hi_cc, lo_cc}
+
+
+@requires_serialization
+def test_select_always_false_op_recompiles_per_target_cc():
+    """``_always_false_op`` is a module-global cache whose LTO-IR is linked into
+    every ``make_select`` build (as the three-way-partition's second predicate).
+
+    It must be keyed on the target compute capability. A cc-unkeyed cache would
+    compile once for whatever arch built first and hand that same op to every
+    later build, linking newer-arch code into an older-arch result — which
+    nvJitLink rejects. Unlike the iterator memos this leaks process-globally,
+    across unrelated select builds.
+    """
+    from cuda.compute.algorithms._select import _get_always_false_op
+
+    ccs = _compilable_ccs(2)
+    if len(ccs) < 2:
+        pytest.skip("toolchain compiles for <2 target arches")
+    hi, lo = sorted(ccs, reverse=True)  # populate the cache for the higher arch first
+
+    def code(cc):
+        with target_cc(cc):
+            return bytes(_get_always_false_op()._ltoir.op_bytes)
+
+    b_hi = code(hi)
+    b_lo = code(lo)
+    # Regression: lo must not return the cached hi-arch LTO-IR.
+    assert b_hi != b_lo
+
+
+# ----------------------------------------------------------------------------
+# GPU-free construction / deserialization (no device query, no recompile)
+# ----------------------------------------------------------------------------
+
+
+@requires_serialization
+def test_unannotated_transform_iterator_construction_needs_no_gpu(monkeypatch):
+    """Constructing an unannotated TransformIterator infers the lambda's return
+    type; that inference is architecture-independent and must not query the
+    current device.
+
+    Otherwise a GPU-free build machine cannot even construct the iterator to hand
+    to ``make_<algo>(compute_capability=...)`` — the device query happens at
+    construction, before the explicit cc can take effect.
+    """
+    import cuda.compute._caching as _caching_mod
+
+    class _NoDevice:
+        def __init__(self, *a, **k):
+            raise AssertionError("iterator construction must not query a CUDA device")
+
+    # Any current-device query (e.g. the build cache's cc salt) now fails loudly.
+    monkeypatch.setattr(_caching_mod, "Device", _NoDevice)
+
+    it = TransformIterator(CountingIterator(np.int32(0)), lambda x: x + 1)
+    # int32 + Python int promotes to int64 in Numba; the point is that the type
+    # resolved with no device query.
+    assert it.value_type == from_numpy_dtype(np.dtype(np.int64))
+
+
+@requires_serialization
+def test_select_deserialize_needs_no_gpu_and_no_recompile(monkeypatch):
+    """``deserialize()`` must neither recompile device code nor require a GPU.
+
+    ``_Select``'s always-false predicate is not serialized: its LTO-IR is already
+    baked into the serialized three-way-partition build result, and ``__call__``
+    needs only its (empty) runtime state. Regression: ``_after_deserialize`` must
+    not call ``_always_false_op()``, which on a cold cache compiles via NVRTC and
+    falls back to ``Device().compute_capability``.
+    """
+    # A multi-arch blob defers the single-target cc check at deserialize (see
+    # _check_loadable), so deserialization is device-independent — the real
+    # GPU-free AoT scenario (ship one blob, load on the target).
+    ccs = _compilable_ccs(2)
+    if len(ccs) < 2:
+        pytest.skip("toolchain compiles for <2 target arches")
+
+    def _cond(x):
+        return np.uint8(x < np.int32(5))
+
+    # Device-free AoT build + serialize (this may legitimately compile).
+    sel = make_select(
+        d_in=ProxyArray(np.int32),
+        d_out=ProxyArray(np.int32),
+        d_num_selected_out=ProxyArray(np.int32),
+        cond=_cond,
+        compute_capability=ccs,
+    )
+    blob = serialize(sel)
+
+    # Now simulate a GPU-free machine on which any recompile or device query is
+    # fatal, and require deserialize to succeed regardless.
+    import cuda.compute._caching as _caching_mod
+    import cuda.compute._cpp_compile as _cpp_mod
+    import cuda.compute.algorithms._select as _select_mod
+
+    def _boom(*a, **k):
+        raise AssertionError("deserialize must not recompile the always-false op")
+
+    class _NoDevice:
+        def __init__(self, *a, **k):
+            raise AssertionError("deserialize must not query a CUDA device")
+
+    monkeypatch.setattr(_select_mod, "_always_false_op", _boom)
+    monkeypatch.setattr(_select_mod, "_get_always_false_op", _boom)
+    monkeypatch.setattr(_cpp_mod, "Device", _NoDevice)
+    monkeypatch.setattr(_caching_mod, "Device", _NoDevice)
+
+    loaded = deserialize(blob)
+    # The always-false predicate is stateless, so its reconstructed runtime state
+    # is empty — exactly what __call__ reads.
+    assert loaded.always_false_op.get_state() == b""
+    assert set(loaded.partitioner.build_results.keys()) == set(ccs)
+
+
+# ----------------------------------------------------------------------------
+# GPU execution: lazily-loaded build result for the current device runs correctly
+# ----------------------------------------------------------------------------
+
+
+@requires_serialization
+def test_compile_only_then_lazy_load_and_execute():
+    cc = current_device_cc_key()
+
+    # Single-cc compile-only build (no fused load) for the current device.
+    reducer = _make_transform(cc)
+    assert not reducer.build_results[cc]._loaded
+
+    h = np.arange(1024, dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h)
+    d_out = DeviceArray.empty(h.shape, h.dtype)
+    reducer(d_in=d_in, d_out=d_out, op=_add_one, num_items=h.size)
+    Device().sync()
+
+    # First call lazily loaded the current-device build result and ran correctly.
+    assert reducer.build_results[cc]._loaded
+    np.testing.assert_array_equal(d_out.copy_to_host(), h + 1)
+
+
+@requires_serialization
+def test_multi_cc_loads_only_the_current_device_build_result():
+    # This test executes kernels, so it legitimately needs a GPU; ensure the
+    # current device's cc is one of the built arches.
+    cc = current_device_cc_key()
+    other = next((c for c in _compilable_ccs(3) if c != cc), None)
+    if other is None:
+        pytest.skip("toolchain compiles for <2 target arches")
+    ccs = [cc, other]
+
+    builder = _make_transform(ccs)
+    blob = serialize(builder)
+    loaded = deserialize(blob)
+
+    h = np.arange(256, dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h)
+    d_out = DeviceArray.empty(h.shape, h.dtype)
+    loaded(d_in=d_in, d_out=d_out, op=_add_one, num_items=h.size)
+    Device().sync()
+
+    np.testing.assert_array_equal(d_out.copy_to_host(), h + 1)
+    # Only the current device's build result is loaded; the other stays lazy.
+    assert loaded.build_results[cc]._loaded
+    assert not loaded.build_results[other]._loaded
+
+
+@requires_serialization
+def test_fused_fast_path_unchanged_when_no_cc_given():
+    h = np.arange(512, dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h)
+    d_out = DeviceArray.empty(h.shape, h.dtype)
+
+    # compute_capability omitted -> single, already-loaded build result (fused build).
+    builder = make_unary_transform(d_in=d_in, d_out=d_out, op=_add_one)
+    (only,) = builder.build_results.values()
+    assert only._loaded  # fused build already loaded the kernels
+
+    builder(d_in=d_in, d_out=d_out, op=_add_one, num_items=h.size)
+    Device().sync()
+    np.testing.assert_array_equal(d_out.copy_to_host(), h + 1)
+
+
+# ----------------------------------------------------------------------------
+# Multi-GPU: the second device loads via serialize/deserialize clone, with real
+# native build results (single-GPU CI skips these; see the fake-backed tests
+# above for the single-device protocol coverage)
+# ----------------------------------------------------------------------------
+
+
+def _same_cc_device_pair():
+    """Two device ordinals sharing a compute capability, or None.
+
+    The clone branch of _PerCCBuildResults.resolve() is reachable only
+    when a second device resolves the same cc entry that another device
+    already owns, which requires two devices of equal compute capability.
+    """
+    by_cc = {}
+    for device_id in range(get_num_devices()):
+        major, minor = Device(device_id).compute_capability
+        by_cc.setdefault(cc_to_key((int(major), int(minor))), []).append(device_id)
+    for cc_key, device_ids in by_cc.items():
+        if len(device_ids) >= 2:
+            return cc_key, device_ids[0], device_ids[1]
+    return None
+
+
+@requires_serialization
+def test_second_device_loads_via_clone_without_recompiling():
+    """On real same-cc GPUs, the first device loads the canonical build result
+    in place and the second loads an independent serialize -> deserialize ->
+    load clone, reusing the compiled payload without recompiling.
+    """
+    pair = _same_cc_device_pair()
+    if pair is None:
+        pytest.skip("requires two devices with the same compute capability")
+    cc_key, device_a, device_b = pair
+
+    transformer = _make_transform(cc_key)
+    h = np.arange(512, dtype=np.int32)
+
+    first_device = Device(device_a)
+    first_device.set_current()
+    d_in = DeviceArray.from_numpy(h, device=first_device)
+    d_out = DeviceArray.empty(h.shape, h.dtype, device=first_device)
+    transformer(d_in=d_in, d_out=d_out, op=_add_one, num_items=h.size)
+    first_device.sync()
+    np.testing.assert_array_equal(d_out.copy_to_host(), h + 1)
+
+    second_device = Device(device_b)
+    second_device.set_current()
+    d_in_b = DeviceArray.from_numpy(h, device=second_device)
+    d_out_b = DeviceArray.empty(h.shape, h.dtype, device=second_device)
+    second_device.set_current()
+    transformer(d_in=d_in_b, d_out=d_out_b, op=_add_one, num_items=h.size)
+    second_device.sync()
+    np.testing.assert_array_equal(d_out_b.copy_to_host(), h + 1)
+
+    collection = transformer.build_results
+    loaded_a = collection._loaded_results[(cc_key, device_a)]
+    loaded_b = collection._loaded_results[(cc_key, device_b)]
+    # The first device claimed and loaded the canonical result; the second
+    # device received an independent serialize -> deserialize -> load clone.
+    assert loaded_a is collection[cc_key]
+    assert loaded_b is not loaded_a
+    assert loaded_b._loaded
+
+
+@requires_serialization
+def test_deserialized_wrapper_runs_on_both_devices():
+    """One deserialized wrapper runs correctly on two same-cc GPUs: the first
+    device to execute claims and loads the canonical result, the second loads
+    its own clone.
+    """
+    pair = _same_cc_device_pair()
+    if pair is None:
+        pytest.skip("requires two devices with the same compute capability")
+    cc_key, device_a, device_b = pair
+
+    blob = serialize(_make_transform(cc_key))
+    restored = deserialize(blob)
+    h = np.arange(512, dtype=np.int32)
+
+    # Run on device_b FIRST so a non-zero ordinal claims canonical ownership,
+    # then device_a exercises the clone path from a non-zero owner.
+    for device_id in (device_b, device_a):
+        device = Device(device_id)
+        device.set_current()
+        d_in = DeviceArray.from_numpy(h, device=device)
+        d_out = DeviceArray.empty(h.shape, h.dtype, device=device)
+        device.set_current()
+        restored(d_in=d_in, d_out=d_out, op=_add_one, num_items=h.size)
+        device.sync()
+        np.testing.assert_array_equal(d_out.copy_to_host(), h + 1)
+
+    collection = restored.build_results
+    loaded_b = collection._loaded_results[(cc_key, device_b)]
+    loaded_a = collection._loaded_results[(cc_key, device_a)]
+    assert loaded_b is collection[cc_key]
+    assert loaded_a is not loaded_b
+
+
+# ----------------------------------------------------------------------------
+# Serialization under concurrent use, and round-trip stability
+# ----------------------------------------------------------------------------
+
+
+@requires_serialization
+def test_serialize_while_other_threads_execute():
+    """serialize() must be safe while other threads run the same build.
+
+    Worker threads each obtain their own wrapper from the factory (sharing one
+    _PerCCBuildResults through the process-wide build cache) and execute
+    repeatedly — including the first device load — while the main thread
+    serializes its wrapper concurrently. serialize_build_result and the first
+    load take the same per-cc source lock; compute never mutates the fields
+    serialize reads. Every produced blob must be identical and usable.
+    """
+    cc = current_device_cc_key()
+
+    # int16 keeps this specialization's cache key distinct from every other
+    # test in this module, so the first device load happens inside this test.
+    def make_wrapper():
+        return make_unary_transform(
+            d_in=ProxyArray(np.int16),
+            d_out=ProxyArray(np.int16),
+            op=_add_one,
+            compute_capability=cc,
+        )
+
+    main_wrapper = make_wrapper()
+    h = np.arange(1024, dtype=np.int16)
+    n_workers = 2
+    iterations = 20
+    barrier = threading.Barrier(n_workers + 1, timeout=120)
+    shared_collections = []
+
+    def worker():
+        wrapper = make_wrapper()
+        shared_collections.append(wrapper.build_results)
+        d_in = DeviceArray.from_numpy(h)
+        d_out = DeviceArray.empty(h.shape, h.dtype)
+        barrier.wait()
+        for _ in range(iterations):
+            wrapper(d_in=d_in, d_out=d_out, op=_add_one, num_items=h.size)
+            Device().sync()
+            np.testing.assert_array_equal(d_out.copy_to_host(), h + 1)
+
+    with concurrent.futures.ThreadPoolExecutor(max_workers=n_workers) as executor:
+        futures = [executor.submit(worker) for _ in range(n_workers)]
+        barrier.wait()
+        blobs = [serialize(main_wrapper) for _ in range(iterations)]
+        for future in futures:
+            future.result()
+
+    # All threads shared one collection, and serialization was stable
+    # throughout the concurrent first-load and execution window.
+    assert all(coll is main_wrapper.build_results for coll in shared_collections)
+    assert len(set(blobs)) == 1
+
+    restored = deserialize(blobs[0])
+    d_in = DeviceArray.from_numpy(h)
+    d_out = DeviceArray.empty(h.shape, h.dtype)
+    restored(d_in=d_in, d_out=d_out, op=_add_one, num_items=h.size)
+    Device().sync()
+    np.testing.assert_array_equal(d_out.copy_to_host(), h + 1)
+
+
+@requires_serialization
+def test_double_serialize_roundtrip_is_stable_and_executes():
+    """serialize -> deserialize -> serialize -> deserialize is stable.
+
+    Iterator/value/op runtime state is not serialized and loading only
+    populates native handles, so a reconstructed wrapper must serialize back
+    to the identical bytes both before and after it has been loaded and
+    executed, and the second reconstruction must still run correctly.
+    """
+    cc = current_device_cc_key()
+
+    blob1 = serialize(_make_transform(cc))
+
+    # Byte-stable before any load.
+    assert serialize(deserialize(blob1)) == blob1
+
+    # Load + execute the reconstruction, then round-trip again: loading must
+    # leak nothing into the blob.
+    loaded = deserialize(blob1)
+    h = np.arange(256, dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h)
+    d_out = DeviceArray.empty(h.shape, h.dtype)
+    loaded(d_in=d_in, d_out=d_out, op=_add_one, num_items=h.size)
+    Device().sync()
+    np.testing.assert_array_equal(d_out.copy_to_host(), h + 1)
+
+    blob2 = serialize(loaded)
+    assert blob2 == blob1
+
+    second = deserialize(blob2)
+    d_out.copy_from_host(np.zeros_like(h))
+    second(d_in=d_in, d_out=d_out, op=_add_one, num_items=h.size)
+    Device().sync()
+    np.testing.assert_array_equal(d_out.copy_to_host(), h + 1)
+
+
+@requires_serialization
+def test_default_build_on_second_same_cc_device_clones_payload(monkeypatch):
+    """Real-GPU check of the shared default entry (see the fake-based
+    test_default_build_second_same_cc_device_shares_the_entry for the
+    protocol): the second same-cc device must never run a native build."""
+    pair = _same_cc_device_pair()
+    if pair is None:
+        pytest.skip("requires two devices with the same compute capability")
+    _, device_a, device_b = pair
+
+    # uint8 keeps this specialization's cache key distinct from other tests.
+    h = np.arange(256, dtype=np.uint8)
+
+    first_device = Device(device_a)
+    first_device.set_current()
+    d_in = DeviceArray.from_numpy(h, device=first_device)
+    d_out = DeviceArray.empty(h.shape, h.dtype, device=first_device)
+    w_a = make_unary_transform(d_in=d_in, d_out=d_out, op=_add_one)
+    w_a(d_in=d_in, d_out=d_out, op=_add_one, num_items=h.size)
+    first_device.sync()
+    np.testing.assert_array_equal(d_out.copy_to_host(), h + 1)
+
+    def _no_native_build(*args, **kwargs):
+        raise AssertionError("second same-cc device must clone, not rebuild")
+
+    monkeypatch.setattr(cccl_interop, "call_build", _no_native_build)
+    second_device = Device(device_b)
+    second_device.set_current()
+    d_in_b = DeviceArray.from_numpy(h, device=second_device)
+    d_out_b = DeviceArray.empty(h.shape, h.dtype, device=second_device)
+    second_device.set_current()
+    w_b = make_unary_transform(d_in=d_in_b, d_out=d_out_b, op=_add_one)
+    w_b(d_in=d_in_b, d_out=d_out_b, op=_add_one, num_items=h.size)
+    second_device.sync()
+    np.testing.assert_array_equal(d_out_b.copy_to_host(), h + 1)
+
+    # One shared per-cc entry; distinct per-device loaded results (the
+    # wrappers' construction-time bindings).
+    assert w_b.build_results is w_a.build_results
+    result_a = w_a._bound_build_result
+    result_b = w_b._bound_build_result
+    assert result_a is not None and result_b is not None
+    assert result_b is not result_a
+    assert result_b._loaded
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_nested_struct.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_nested_struct.py
new file mode 100644
index 00000000..d9aa986b
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_nested_struct.py
@@ -0,0 +1,499 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+import numpy as np
+from _utils.device_array import DeviceArray
+
+import cuda.compute
+from cuda.compute import ZipIterator, gpu_struct
+
+
+def test_reduce_nested_struct_direct():
+    Inner = gpu_struct({"a": np.int32, "b": np.float32})
+    Outer = gpu_struct({"x": np.int64, "inner": Inner})
+
+    def sum_nested(s1, s2):
+        return Outer(
+            s1.x + s2.x, Inner(s1.inner.a + s2.inner.a, s1.inner.b + s2.inner.b)
+        )
+
+    num_items = 10
+
+    h_data = np.zeros(num_items, dtype=Outer.dtype)
+    for i in range(num_items):
+        h_data[i]["x"] = i
+        h_data[i]["inner"]["a"] = i * 2
+        h_data[i]["inner"]["b"] = float(i * 3)
+
+    d_input = DeviceArray.from_numpy(h_data)
+    d_output = DeviceArray.empty(1, Outer.dtype)
+
+    h_init = Outer(0, Inner(0, 0.0))
+
+    cuda.compute.reduce_into(
+        d_in=d_input, d_out=d_output, num_items=num_items, op=sum_nested, h_init=h_init
+    )
+
+    result = d_output.copy_to_host()[0]
+
+    expected_x = sum(range(num_items))
+    expected_a = sum(i * 2 for i in range(num_items))
+    expected_b = sum(float(i * 3) for i in range(num_items))
+
+    assert result["x"] == expected_x
+    assert result["inner"]["a"] == expected_a
+    assert np.isclose(result["inner"]["b"], expected_b)
+
+
+def test_nested_struct_inline():
+    """Test creating nested structs using inline dictionary syntax."""
+    # Create a struct with an inline nested struct definition
+    Outer = gpu_struct({"x": np.int64, "inner": {"a": np.int32, "b": np.float32}})
+
+    # Get the nested struct type from the outer struct for construction
+    Inner = type(Outer(0, (0, 0.0)).inner)
+
+    def sum_nested(s1, s2):
+        return Outer(
+            s1.x + s2.x, Inner(s1.inner.a + s2.inner.a, s1.inner.b + s2.inner.b)
+        )
+
+    num_items = 10
+
+    h_data = np.zeros(num_items, dtype=Outer.dtype)
+    for i in range(num_items):
+        h_data[i]["x"] = i
+        h_data[i]["inner"]["a"] = i * 2
+        h_data[i]["inner"]["b"] = float(i * 3)
+
+    d_input = DeviceArray.from_numpy(h_data)
+    d_output = DeviceArray.empty(1, Outer.dtype)
+
+    h_init = Outer(0, Inner(0, 0.0))
+
+    cuda.compute.reduce_into(
+        d_in=d_input, d_out=d_output, num_items=num_items, op=sum_nested, h_init=h_init
+    )
+
+    result = d_output.copy_to_host()[0]
+
+    expected_x = sum(range(num_items))
+    expected_a = sum(i * 2 for i in range(num_items))
+    expected_b = sum(float(i * 3) for i in range(num_items))
+
+    assert result["x"] == expected_x
+    assert result["inner"]["a"] == expected_a
+    assert np.isclose(result["inner"]["b"], expected_b)
+
+
+def test_nested_struct_in_zip_iterator():
+    Point = gpu_struct({"x": np.int32, "y": np.int32})
+    Color = gpu_struct({"r": np.uint8, "g": np.uint8, "b": np.uint8})
+    Pixel = gpu_struct({"position": Point, "color": Color})
+
+    def sum_pixels(p1, p2):
+        return Pixel(
+            Point(p1.position.x + p2.position.x, p1.position.y + p2.position.y),
+            Color(
+                p1.color.r + p2.color.r,
+                p1.color.g + p2.color.g,
+                p1.color.b + p2.color.b,
+            ),
+        )
+
+    num_items = 100
+
+    h_points = np.array([(i, i * 2) for i in range(num_items)], dtype=Point.dtype)
+    h_colors = np.array(
+        [(i % 256, (i * 2) % 256, (i * 3) % 256) for i in range(num_items)],
+        dtype=Color.dtype,
+    )
+
+    d_points = DeviceArray.from_numpy(h_points)
+    d_colors = DeviceArray.from_numpy(h_colors)
+
+    zip_it = ZipIterator(d_points, d_colors)
+
+    d_output = DeviceArray.empty(1, Pixel.dtype)
+    h_init = Pixel(Point(0, 0), Color(0, 0, 0))
+
+    cuda.compute.reduce_into(
+        d_in=zip_it, d_out=d_output, num_items=num_items, op=sum_pixels, h_init=h_init
+    )
+
+    result = d_output.copy_to_host()[0]
+
+    expected_x = sum(i for i in range(num_items))
+    expected_y = sum(i * 2 for i in range(num_items))
+    expected_r = sum(i % 256 for i in range(num_items)) % 256
+    expected_g = sum((i * 2) % 256 for i in range(num_items)) % 256
+    expected_b = sum((i * 3) % 256 for i in range(num_items)) % 256
+
+    assert result["position"]["x"] == expected_x
+    assert result["position"]["y"] == expected_y
+    assert result["color"]["r"] == expected_r
+    assert result["color"]["g"] == expected_g
+    assert result["color"]["b"] == expected_b
+
+
+def test_dict_init_nested_struct():
+    """Test initializing a nested struct with a dictionary."""
+    Inner = gpu_struct({"a": np.int32, "b": np.float32})
+    Outer = gpu_struct({"x": np.int64, "inner": Inner})
+
+    # Initialize with nested dictionary
+    obj = Outer({"x": 42, "inner": {"a": 10, "b": 3.14}})
+
+    assert obj.x == 42
+    assert obj.inner.a == 10
+    assert np.isclose(obj.inner.b, 3.14)
+
+
+def test_dict_init_per_field():
+    """Test initializing a struct with a dictionary for a nested field."""
+    Inner = gpu_struct({"a": np.int32, "b": np.float32})
+    Outer = gpu_struct({"x": np.int64, "inner": Inner})
+
+    # Mix positional value with dictionary for nested field
+    obj = Outer(42, {"a": 10, "b": 3.14})
+
+    assert obj.x == 42
+    assert obj.inner.a == 10
+    assert np.isclose(obj.inner.b, 3.14)
+
+
+def test_dict_init_deeply_nested():
+    """Test initializing deeply nested structs (3+ levels) with dictionaries."""
+    Level1 = gpu_struct({"value": np.int32})
+    Level2 = gpu_struct({"data": np.float32, "nested": Level1})
+    Level3 = gpu_struct({"id": np.int64, "middle": Level2})
+
+    # Initialize with deeply nested dictionary
+    obj = Level3({"id": 100, "middle": {"data": 2.5, "nested": {"value": 42}}})
+
+    assert obj.id == 100
+    assert np.isclose(obj.middle.data, 2.5)
+    assert obj.middle.nested.value == 42
+
+
+def test_dict_init_mixed():
+    """Test mixed initialization with some dicts and some direct values."""
+    Inner1 = gpu_struct({"a": np.int32, "b": np.int32})
+    Inner2 = gpu_struct({"c": np.float32, "d": np.float32})
+    Outer = gpu_struct({"x": np.int64, "inner1": Inner1, "inner2": Inner2})
+
+    # Mix different initialization styles
+    inner1_obj = Inner1(1, 2)  # Direct instantiation
+    # Mix direct object and dict
+    obj = Outer(100, inner1_obj, {"c": 3.0, "d": 4.0})
+
+    assert obj.x == 100
+    assert obj.inner1.a == 1
+    assert obj.inner1.b == 2
+    assert np.isclose(obj.inner2.c, 3.0)
+    assert np.isclose(obj.inner2.d, 4.0)
+
+
+def test_dict_init_with_reduction():
+    """Test that dict-initialized structs work correctly in reductions."""
+    Inner = gpu_struct({"a": np.int32, "b": np.float32})
+    Outer = gpu_struct({"x": np.int64, "inner": Inner})
+
+    def sum_nested(s1, s2):
+        return Outer(
+            s1.x + s2.x, Inner(s1.inner.a + s2.inner.a, s1.inner.b + s2.inner.b)
+        )
+
+    num_items = 10
+    h_data = np.zeros(num_items, dtype=Outer.dtype)
+    for i in range(num_items):
+        h_data[i]["x"] = i
+        h_data[i]["inner"]["a"] = i * 2
+        h_data[i]["inner"]["b"] = float(i * 3)
+
+    d_input = DeviceArray.from_numpy(h_data)
+    d_output = DeviceArray.empty(1, Outer.dtype)
+
+    # Use dictionary initialization for the init value
+    h_init = Outer({"x": 0, "inner": {"a": 0, "b": 0.0}})
+
+    cuda.compute.reduce_into(
+        d_in=d_input, d_out=d_output, num_items=num_items, op=sum_nested, h_init=h_init
+    )
+
+    result = d_output.copy_to_host()[0]
+
+    expected_x = sum(range(num_items))
+    expected_a = sum(i * 2 for i in range(num_items))
+    expected_b = sum(float(i * 3) for i in range(num_items))
+
+    assert result["x"] == expected_x
+    assert result["inner"]["a"] == expected_a
+    assert np.isclose(result["inner"]["b"], expected_b)
+
+
+def test_nested_struct_tuple_construction():
+    """Test constructing nested structs using tuple syntax in device functions."""
+    Inner = gpu_struct({"a": np.int32, "b": np.float32})
+    Outer = gpu_struct({"x": np.int64, "inner": Inner})
+
+    def sum_nested_with_tuples(s1, s2):
+        # Use tuple syntax instead of Inner(...)
+        return Outer(s1.x + s2.x, (s1.inner.a + s2.inner.a, s1.inner.b + s2.inner.b))
+
+    num_items = 10
+
+    h_data = np.zeros(num_items, dtype=Outer.dtype)
+    for i in range(num_items):
+        h_data[i]["x"] = i
+        h_data[i]["inner"]["a"] = i * 2
+        h_data[i]["inner"]["b"] = float(i * 3)
+
+    d_input = DeviceArray.from_numpy(h_data)
+    d_output = DeviceArray.empty(1, Outer.dtype)
+
+    h_init = Outer(0, Inner(0, 0.0))
+
+    cuda.compute.reduce_into(
+        d_in=d_input,
+        d_out=d_output,
+        num_items=num_items,
+        op=sum_nested_with_tuples,
+        h_init=h_init,
+    )
+
+    result = d_output.copy_to_host()[0]
+
+    expected_x = sum(range(num_items))
+    expected_a = sum(i * 2 for i in range(num_items))
+    expected_b = sum(float(i * 3) for i in range(num_items))
+
+    assert result["x"] == expected_x
+    assert result["inner"]["a"] == expected_a
+    assert np.isclose(result["inner"]["b"], expected_b)
+
+
+def test_deeply_nested_tuple_construction():
+    """Test constructing deeply nested structs (3 levels) using tuple syntax."""
+    Level1 = gpu_struct({"value": np.int32})
+    Level2 = gpu_struct({"data": np.float32, "nested": Level1})
+    Level3 = gpu_struct({"id": np.int64, "middle": Level2})
+
+    def sum_deeply_nested(v1, v2):
+        # Use nested tuple syntax: (float, (int,))
+        return Level3(
+            v1.id + v2.id,
+            (
+                v1.middle.data + v2.middle.data,
+                (v1.middle.nested.value + v2.middle.nested.value,),
+            ),
+        )
+
+    num_items = 10
+
+    h_data = np.zeros(num_items, dtype=Level3.dtype)
+    for i in range(num_items):
+        h_data[i]["id"] = i * 10
+        h_data[i]["middle"]["data"] = float(i * 2.5)
+        h_data[i]["middle"]["nested"]["value"] = i * 3
+
+    d_input = DeviceArray.from_numpy(h_data)
+    d_output = DeviceArray.empty(1, Level3.dtype)
+
+    h_init = Level3(0, Level2(0.0, Level1(0)))
+
+    cuda.compute.reduce_into(
+        d_in=d_input,
+        d_out=d_output,
+        num_items=num_items,
+        op=sum_deeply_nested,
+        h_init=h_init,
+    )
+
+    result = d_output.copy_to_host()[0]
+
+    expected_id = sum(i * 10 for i in range(num_items))
+    expected_data = sum(float(i * 2.5) for i in range(num_items))
+    expected_value = sum(i * 3 for i in range(num_items))
+
+    assert result["id"] == expected_id
+    assert np.isclose(result["middle"]["data"], expected_data)
+    assert result["middle"]["nested"]["value"] == expected_value
+
+
+def test_mixed_tuple_and_direct_construction():
+    """Test mixing tuple construction with direct struct construction."""
+    Inner1 = gpu_struct({"a": np.int32, "b": np.int32})
+    Inner2 = gpu_struct({"c": np.float32, "d": np.float32})
+    Outer = gpu_struct({"x": np.int64, "inner1": Inner1, "inner2": Inner2})
+
+    def sum_mixed(s1, s2):
+        # Mix direct struct construction with tuple construction
+        return Outer(
+            s1.x + s2.x,
+            Inner1(s1.inner1.a + s2.inner1.a, s1.inner1.b + s2.inner1.b),
+            (s1.inner2.c + s2.inner2.c, s1.inner2.d + s2.inner2.d),
+        )
+
+    num_items = 10
+
+    h_data = np.zeros(num_items, dtype=Outer.dtype)
+    for i in range(num_items):
+        h_data[i]["x"] = i
+        h_data[i]["inner1"]["a"] = i * 2
+        h_data[i]["inner1"]["b"] = i * 3
+        h_data[i]["inner2"]["c"] = float(i * 4)
+        h_data[i]["inner2"]["d"] = float(i * 5)
+
+    d_input = DeviceArray.from_numpy(h_data)
+    d_output = DeviceArray.empty(1, Outer.dtype)
+
+    h_init = Outer(0, Inner1(0, 0), Inner2(0.0, 0.0))
+
+    cuda.compute.reduce_into(
+        d_in=d_input, d_out=d_output, num_items=num_items, op=sum_mixed, h_init=h_init
+    )
+
+    result = d_output.copy_to_host()[0]
+
+    expected_x = sum(range(num_items))
+    expected_a = sum(i * 2 for i in range(num_items))
+    expected_b = sum(i * 3 for i in range(num_items))
+    expected_c = sum(float(i * 4) for i in range(num_items))
+    expected_d = sum(float(i * 5) for i in range(num_items))
+
+    assert result["x"] == expected_x
+    assert result["inner1"]["a"] == expected_a
+    assert result["inner1"]["b"] == expected_b
+    assert np.isclose(result["inner2"]["c"], expected_c)
+    assert np.isclose(result["inner2"]["d"], expected_d)
+
+
+def test_tuple_construction_in_zip_iterator():
+    """Test tuple construction with ZipIterator combining nested structs."""
+    Point = gpu_struct({"x": np.int32, "y": np.int32})
+    Color = gpu_struct({"r": np.uint8, "g": np.uint8, "b": np.uint8})
+    Pixel = gpu_struct({"position": Point, "color": Color})
+
+    def sum_pixels_with_tuples(p1, p2):
+        # Use tuple syntax for both nested structs
+        return Pixel(
+            (p1.position.x + p2.position.x, p1.position.y + p2.position.y),
+            (
+                p1.color.r + p2.color.r,
+                p1.color.g + p2.color.g,
+                p1.color.b + p2.color.b,
+            ),
+        )
+
+    num_items = 100
+
+    h_points = np.array([(i, i * 2) for i in range(num_items)], dtype=Point.dtype)
+    h_colors = np.array(
+        [(i % 256, (i * 2) % 256, (i * 3) % 256) for i in range(num_items)],
+        dtype=Color.dtype,
+    )
+
+    d_points = DeviceArray.from_numpy(h_points)
+    d_colors = DeviceArray.from_numpy(h_colors)
+
+    zip_it = ZipIterator(d_points, d_colors)
+
+    d_output = DeviceArray.empty(1, Pixel.dtype)
+    h_init = Pixel(Point(0, 0), Color(0, 0, 0))
+
+    cuda.compute.reduce_into(
+        d_in=zip_it,
+        d_out=d_output,
+        num_items=num_items,
+        op=sum_pixels_with_tuples,
+        h_init=h_init,
+    )
+
+    result = d_output.copy_to_host()[0]
+
+    expected_x = sum(i for i in range(num_items))
+    expected_y = sum(i * 2 for i in range(num_items))
+    expected_r = sum(i % 256 for i in range(num_items)) % 256
+    expected_g = sum((i * 2) % 256 for i in range(num_items)) % 256
+    expected_b = sum((i * 3) % 256 for i in range(num_items)) % 256
+
+    assert result["position"]["x"] == expected_x
+    assert result["position"]["y"] == expected_y
+    assert result["color"]["r"] == expected_r
+    assert result["color"]["g"] == expected_g
+    assert result["color"]["b"] == expected_b
+
+
+def test_all_tuple_construction():
+    """Test constructing a struct where all fields use tuple syntax."""
+    Inner1 = gpu_struct({"a": np.int32})
+    Inner2 = gpu_struct({"b": np.float32})
+    Outer = gpu_struct({"field1": Inner1, "field2": Inner2})
+
+    def sum_all_tuples(s1, s2):
+        # All fields use tuple syntax
+        return Outer((s1.field1.a + s2.field1.a,), (s1.field2.b + s2.field2.b,))
+
+    num_items = 5
+
+    h_data = np.zeros(num_items, dtype=Outer.dtype)
+    for i in range(num_items):
+        h_data[i]["field1"]["a"] = i
+        h_data[i]["field2"]["b"] = float(i * 2)
+
+    d_input = DeviceArray.from_numpy(h_data)
+    d_output = DeviceArray.empty(1, Outer.dtype)
+
+    h_init = Outer(Inner1(0), Inner2(0.0))
+
+    cuda.compute.reduce_into(
+        d_in=d_input,
+        d_out=d_output,
+        num_items=num_items,
+        op=sum_all_tuples,
+        h_init=h_init,
+    )
+
+    result = d_output.copy_to_host()[0]
+
+    expected_a = sum(range(num_items))
+    expected_b = sum(float(i * 2) for i in range(num_items))
+
+    assert result["field1"]["a"] == expected_a
+    assert np.isclose(result["field2"]["b"], expected_b)
+
+
+def test_struct_field_order_matters():
+    """Test that struct types with same fields in different order are not equal.
+
+    This is a regression test for a bug where StructTypeDescriptor.__hash__()
+    sorted fields before hashing, causing structs with the same fields but
+    different order to hash to the same value and be considered equal.
+    """
+
+    # Create two structs with identical field types but different order
+    @gpu_struct
+    class SumAndCount:
+        sum: np.float32
+        count: np.int32
+
+    @gpu_struct
+    class CountAndSum:
+        count: np.int32
+        sum: np.float32
+
+    # These should NOT be equal (field order matters for struct layout)
+    assert SumAndCount._type_descriptor != CountAndSum._type_descriptor
+
+    # They should have different hashes
+    assert hash(SumAndCount._type_descriptor) != hash(CountAndSum._type_descriptor)
+
+    # Verify they can be used as distinct dict keys
+    cache = {
+        SumAndCount._type_descriptor: "sum_first",
+        CountAndSum._type_descriptor: "count_first",
+    }
+    assert len(cache) == 2
+    assert cache[SumAndCount._type_descriptor] == "sum_first"
+    assert cache[CountAndSum._type_descriptor] == "count_first"
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_no_numba.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_no_numba.py
new file mode 100644
index 00000000..da605d0d
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_no_numba.py
@@ -0,0 +1,606 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray
+
+import cuda.compute
+from cuda.compute import (
+    CacheModifiedInputIterator,
+    ConstantIterator,
+    CountingIterator,
+    DiscardIterator,
+    OpKind,
+    PermutationIterator,
+    ReverseIterator,
+    ShuffleIterator,
+    SortOrder,
+    TransformIterator,
+    TransformOutputIterator,
+    ZipIterator,
+)
+from cuda.compute._cpp_compile import compile_cpp_op_code
+from cuda.compute.op import RawOp
+from cuda.compute.types import int16 as cccl_int16
+from cuda.compute.types import int32 as cccl_int32
+
+# These tests define the minimal-extra integration contract. They intentionally
+# use small fixed inputs and avoid the Python-callable operator path.
+pytestmark = pytest.mark.no_numba
+
+
+def _raw_op(source: str, name: str) -> RawOp:
+    return RawOp(ltoir=compile_cpp_op_code(source), name=name)
+
+
+def _raw_even_i32_op() -> RawOp:
+    source = """
+extern "C" __device__ void no_numba_even_i32(void* x, void* result) {
+    int value = *static_cast<int*>(x);
+    *static_cast<bool*>(result) = (value % 2) == 0;
+}
+"""
+    return _raw_op(source, "no_numba_even_i32")
+
+
+def _raw_less_than_i32_op(name: str, threshold: int) -> RawOp:
+    source = f"""
+extern "C" __device__ void {name}(void* x, void* result) {{
+    int value = *static_cast<int*>(x);
+    *static_cast<unsigned char*>(result) = value < {threshold} ? 1 : 0;
+}}
+"""
+    return _raw_op(source, name)
+
+
+def _raw_plus_i64_op() -> RawOp:
+    source = """
+extern "C" __device__ void no_numba_plus_i64(
+    void* lhs,
+    void* rhs,
+    void* result
+) {
+    *static_cast<long long*>(result) =
+        *static_cast<long long*>(lhs) + *static_cast<long long*>(rhs);
+}
+"""
+    return _raw_op(source, "no_numba_plus_i64")
+
+
+def _raw_square_i32_op() -> RawOp:
+    source = """
+extern "C" __device__ void no_numba_square_i32(void* x, void* result) {
+    int value = *static_cast<int*>(x);
+    *static_cast<int*>(result) = value * value;
+}
+"""
+    return _raw_op(source, "no_numba_square_i32")
+
+
+def _raw_zip_sum_i32_op() -> RawOp:
+    source = """
+struct Zip2I32 {
+    int field_0;
+    int field_1;
+};
+
+extern "C" __device__ void no_numba_zip_sum_i32(void* x, void* result) {
+    auto values = static_cast<Zip2I32*>(x);
+    *static_cast<int*>(result) = values->field_0 + values->field_1;
+}
+"""
+    return _raw_op(source, "no_numba_zip_sum_i32")
+
+
+def _raw_negate_i16_op() -> RawOp:
+    source = """
+extern "C" __device__ void no_numba_negate_i16(void* x, void* result) {
+    *static_cast<short*>(result) = -*static_cast<short*>(x);
+}
+"""
+    return _raw_op(source, "no_numba_negate_i16")
+
+
+def test_import_numba_raises(raise_on_numba_import):
+    # Request the guard explicitly rather than relying on the no_numba
+    # auto-injection (skipped under the parallel sweep, see conftest
+    # pytest_collection_modifyitems): this test exercises the guard directly, and
+    # the monkeypatch it depends on keeps it single-threaded under the sweep.
+    with pytest.raises(
+        ImportError, match="This test is marked 'no_numba' but attempted to import it"
+    ):
+        import numba.cuda  # noqa: F401
+
+
+def test_reduce_well_known_plus():
+    h_input = np.arange(1, 14, dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(1, np.int32)
+    h_init = np.array([5], dtype=np.int32)
+
+    cuda.compute.reduce_into(
+        d_in=d_input,
+        d_out=d_output,
+        num_items=h_input.size,
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+
+    assert d_output.copy_to_host()[0] == np.sum(h_input, initial=h_init[0])
+
+
+def test_exclusive_scan_well_known_plus():
+    h_input = np.asarray([2, 4, 6, 8, 10, 12], dtype=np.uint16)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, h_input.dtype)
+    h_init = np.array([1], dtype=np.uint16)
+
+    cuda.compute.exclusive_scan(
+        d_in=d_input,
+        d_out=d_output,
+        op=OpKind.PLUS,
+        init_value=h_init,
+        num_items=h_input.size,
+    )
+
+    expected = np.asarray([1, 3, 7, 13, 21, 31], dtype=np.uint16)
+    np.testing.assert_array_equal(d_output.copy_to_host(), expected)
+
+
+def test_binary_transform_well_known_plus():
+    h_lhs = np.asarray([1.5, 2.5, 3.5, 4.5], dtype=np.float32)
+    h_rhs = np.asarray([10.0, 20.0, 30.0, 40.0], dtype=np.float32)
+    d_lhs = DeviceArray.from_numpy(h_lhs)
+    d_rhs = DeviceArray.from_numpy(h_rhs)
+    d_output = DeviceArray.empty(h_lhs.shape, h_lhs.dtype)
+
+    cuda.compute.binary_transform(
+        d_in1=d_lhs,
+        d_in2=d_rhs,
+        d_out=d_output,
+        op=OpKind.PLUS,
+        num_items=h_lhs.size,
+    )
+
+    np.testing.assert_allclose(d_output.copy_to_host(), h_lhs + h_rhs)
+
+
+def test_unary_transform_well_known_negate():
+    h_input = np.asarray([-4, -2, 0, 2, 4], dtype=np.int8)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, h_input.dtype)
+
+    cuda.compute.unary_transform(
+        d_in=d_input,
+        d_out=d_output,
+        op=OpKind.NEGATE,
+        num_items=h_input.size,
+    )
+
+    np.testing.assert_array_equal(
+        d_output.copy_to_host(), np.asarray([4, 2, 0, -2, -4])
+    )
+
+
+@pytest.mark.parametrize(
+    "search, side",
+    [
+        (cuda.compute.lower_bound, "left"),
+        (cuda.compute.upper_bound, "right"),
+    ],
+)
+def test_binary_search_explicit_opkind_less(search, side):
+    h_data = np.asarray([1, 3, 3, 7, 9, 11], dtype=np.int64)
+    h_values = np.asarray([0, 3, 4, 10, 12], dtype=np.int64)
+    d_out = DeviceArray.empty(h_values.shape, np.uintp)
+
+    search(
+        d_data=DeviceArray.from_numpy(h_data),
+        num_items=h_data.size,
+        d_values=DeviceArray.from_numpy(h_values),
+        num_values=h_values.size,
+        d_out=d_out,
+        comp=OpKind.LESS,
+    )
+
+    expected = np.searchsorted(h_data, h_values, side=side).astype(np.uintp)
+    np.testing.assert_array_equal(d_out.copy_to_host(), expected)
+
+
+@pytest.mark.no_verify_sass
+def test_segmented_reduce_well_known_plus():
+    h_input = np.asarray([1, 2, 3, 4, 5, 6, 7, 8], dtype=np.uint32)
+    h_starts = np.asarray([0, 3, 5], dtype=np.int32)
+    h_ends = np.asarray([3, 5, 8], dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_starts = DeviceArray.from_numpy(h_starts)
+    d_ends = DeviceArray.from_numpy(h_ends)
+    d_output = DeviceArray.empty(3, np.uint32)
+    h_init = np.array([0], dtype=np.uint32)
+
+    cuda.compute.segmented_reduce(
+        d_in=d_input,
+        d_out=d_output,
+        num_segments=3,
+        start_offsets_in=d_starts,
+        end_offsets_in=d_ends,
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+
+    np.testing.assert_array_equal(d_output.copy_to_host(), np.asarray([6, 9, 21]))
+
+
+def test_merge_sort_well_known_less():
+    h_input = np.asarray([3.5, -1.0, 2.25, 2.0, 7.0], dtype=np.float64)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, h_input.dtype)
+
+    cuda.compute.merge_sort(
+        d_in_keys=d_input,
+        d_in_values=None,
+        d_out_keys=d_output,
+        d_out_values=None,
+        num_items=h_input.size,
+        op=OpKind.LESS,
+    )
+
+    np.testing.assert_array_equal(d_output.copy_to_host(), np.sort(h_input))
+
+
+def test_radix_sort_key_value_pairs():
+    h_keys = np.asarray([4, -2, 7, 1, -2, 0], dtype=np.int16)
+    h_values = np.asarray([40, 20, 70, 10, 21, 0], dtype=np.uint8)
+    d_out_keys = DeviceArray.empty(h_keys.shape, h_keys.dtype)
+    d_out_values = DeviceArray.empty(h_values.shape, h_values.dtype)
+
+    cuda.compute.radix_sort(
+        d_in_keys=DeviceArray.from_numpy(h_keys),
+        d_out_keys=d_out_keys,
+        d_in_values=DeviceArray.from_numpy(h_values),
+        d_out_values=d_out_values,
+        num_items=h_keys.size,
+        order=SortOrder.ASCENDING,
+    )
+
+    order = np.argsort(h_keys, stable=True)
+    np.testing.assert_array_equal(d_out_keys.copy_to_host(), h_keys[order])
+    np.testing.assert_array_equal(d_out_values.copy_to_host(), h_values[order])
+
+
+def test_segmented_sort_keys():
+    h_keys = np.asarray([3, 1, 2, 9, 7, 8, 6, 5], dtype=np.uint64)
+    h_offsets = np.asarray([0, 3, 6, 8], dtype=np.int64)
+    d_output = DeviceArray.empty(h_keys.shape, h_keys.dtype)
+
+    cuda.compute.segmented_sort(
+        d_in_keys=DeviceArray.from_numpy(h_keys),
+        d_out_keys=d_output,
+        d_in_values=None,
+        d_out_values=None,
+        num_items=h_keys.size,
+        num_segments=h_offsets.size - 1,
+        start_offsets_in=DeviceArray.from_numpy(h_offsets[:-1]),
+        end_offsets_in=DeviceArray.from_numpy(h_offsets[1:]),
+        order=SortOrder.ASCENDING,
+    )
+
+    expected = np.asarray([1, 2, 3, 7, 8, 9, 5, 6], dtype=np.uint64)
+    np.testing.assert_array_equal(d_output.copy_to_host(), expected)
+
+
+@pytest.mark.no_verify_sass
+def test_unique_by_key_well_known_equal_to():
+    h_keys = np.asarray([1, 1, 2, 2, 2, 3, 4, 4], dtype=np.int16)
+    h_values = np.asarray([10, 11, 20, 21, 22, 30, 40, 41], dtype=np.int8)
+    d_keys = DeviceArray.from_numpy(h_keys)
+    d_values = DeviceArray.from_numpy(h_values)
+    d_out_keys = DeviceArray.empty(h_keys.shape, h_keys.dtype)
+    d_out_values = DeviceArray.empty(h_values.shape, h_values.dtype)
+    d_num_selected = DeviceArray.empty(1, np.int64)
+
+    cuda.compute.unique_by_key(
+        d_in_keys=d_keys,
+        d_in_items=d_values,
+        d_out_keys=d_out_keys,
+        d_out_items=d_out_values,
+        d_out_num_selected=d_num_selected,
+        op=OpKind.EQUAL_TO,
+        num_items=h_keys.size,
+    )
+
+    num_selected = int(d_num_selected.copy_to_host()[0])
+    np.testing.assert_array_equal(
+        d_out_keys.copy_to_host()[:num_selected], [1, 2, 3, 4]
+    )
+    np.testing.assert_array_equal(
+        d_out_values.copy_to_host()[:num_selected], [10, 20, 30, 40]
+    )
+
+
+def test_histogram_even_small_range():
+    h_samples = np.asarray([0.5, 1.5, 2.5, 2.75, 3.0, 3.5], dtype=np.float32)
+    d_histogram = DeviceArray.empty(4, np.int32)
+
+    cuda.compute.histogram_even(
+        d_samples=DeviceArray.from_numpy(h_samples),
+        d_histogram=d_histogram,
+        num_output_levels=5,
+        lower_level=np.float32(0.0),
+        upper_level=np.float32(4.0),
+        num_samples=h_samples.size,
+    )
+
+    expected, _ = np.histogram(h_samples, bins=4, range=(0.0, 4.0))
+    np.testing.assert_array_equal(d_histogram.copy_to_host(), expected.astype(np.int32))
+
+
+def test_select_raw_op():
+    h_input = np.arange(12, dtype=np.int32)
+    d_output = DeviceArray.empty(h_input.shape, h_input.dtype)
+    d_num_selected = DeviceArray.empty(1, np.uint64)
+
+    cuda.compute.select(
+        d_in=DeviceArray.from_numpy(h_input),
+        d_out=d_output,
+        d_num_selected_out=d_num_selected,
+        cond=_raw_even_i32_op(),
+        num_items=h_input.size,
+    )
+
+    num_selected = int(d_num_selected.copy_to_host()[0])
+    np.testing.assert_array_equal(d_output.copy_to_host()[:num_selected], h_input[::2])
+
+
+def test_three_way_partition_raw_op():
+    h_input = np.arange(12, dtype=np.int32)
+    d_first = DeviceArray.empty(h_input.shape, h_input.dtype)
+    d_second = DeviceArray.empty(h_input.shape, h_input.dtype)
+    d_unselected = DeviceArray.empty(h_input.shape, h_input.dtype)
+    d_num_selected = DeviceArray.empty(2, np.uint64)
+
+    cuda.compute.three_way_partition(
+        d_in=DeviceArray.from_numpy(h_input),
+        d_first_part_out=d_first,
+        d_second_part_out=d_second,
+        d_unselected_out=d_unselected,
+        d_num_selected_out=d_num_selected,
+        select_first_part_op=_raw_less_than_i32_op("no_numba_less_than_4_i32", 4),
+        select_second_part_op=_raw_less_than_i32_op("no_numba_less_than_8_i32", 8),
+        num_items=h_input.size,
+    )
+
+    selected = d_num_selected.copy_to_host()
+    first_count = int(selected[0])
+    second_count = int(selected[1])
+    unselected_count = h_input.size - first_count - second_count
+
+    np.testing.assert_array_equal(d_first.copy_to_host()[:first_count], h_input[:4])
+    np.testing.assert_array_equal(d_second.copy_to_host()[:second_count], h_input[4:8])
+    np.testing.assert_array_equal(
+        d_unselected.copy_to_host()[:unselected_count], h_input[8:]
+    )
+
+
+def test_raw_op_reduce():
+    h_input = np.asarray([10, 20, 30, 40], dtype=np.int64)
+    d_output = DeviceArray.empty(1, np.int64)
+
+    cuda.compute.reduce_into(
+        d_in=DeviceArray.from_numpy(h_input),
+        d_out=d_output,
+        num_items=h_input.size,
+        op=_raw_plus_i64_op(),
+        h_init=np.array([5], dtype=np.int64),
+    )
+
+    assert d_output.copy_to_host()[0] == 105
+
+
+def test_stream_argument(cuda_stream):
+    h_lhs = np.asarray([2, 4, 6, 8, 10], dtype=np.int32)
+    h_rhs = np.asarray([1, 3, 5, 7, 9], dtype=np.int32)
+    d_lhs = DeviceArray.from_numpy(h_lhs, stream=cuda_stream)
+    d_rhs = DeviceArray.from_numpy(h_rhs, stream=cuda_stream)
+    d_output = DeviceArray.empty(h_lhs.shape, h_lhs.dtype, stream=cuda_stream)
+
+    cuda.compute.binary_transform(
+        d_in1=d_lhs,
+        d_in2=d_rhs,
+        d_out=d_output,
+        op=OpKind.PLUS,
+        num_items=h_lhs.size,
+        stream=cuda_stream,
+    )
+
+    np.testing.assert_array_equal(
+        d_output.copy_to_host(stream=cuda_stream),
+        np.asarray([3, 7, 11, 15, 19]),
+    )
+
+
+def test_counting_iterator_reduce():
+    d_output = DeviceArray.empty(1, np.int32)
+
+    cuda.compute.reduce_into(
+        d_in=CountingIterator(np.int32(3)),
+        d_out=d_output,
+        num_items=8,
+        op=OpKind.PLUS,
+        h_init=np.array([0], dtype=np.int32),
+    )
+
+    assert d_output.copy_to_host()[0] == 52
+
+
+def test_constant_iterator_reduce():
+    d_output = DeviceArray.empty(1, np.float32)
+
+    cuda.compute.reduce_into(
+        d_in=ConstantIterator(np.float32(1.5)),
+        d_out=d_output,
+        num_items=8,
+        op=OpKind.PLUS,
+        h_init=np.array([0], dtype=np.float32),
+    )
+
+    np.testing.assert_allclose(d_output.copy_to_host()[0], np.float32(12.0))
+
+
+def test_cache_modified_input_iterator_reduce():
+    h_input = np.asarray([2, 4, 6, 8, 10], dtype=np.uint16)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(1, np.uint16)
+    iterator = CacheModifiedInputIterator(d_input, modifier="stream")
+
+    cuda.compute.reduce_into(
+        d_in=iterator,
+        d_out=d_output,
+        num_items=h_input.size,
+        op=OpKind.PLUS,
+        h_init=np.array([0], dtype=np.uint16),
+    )
+
+    assert d_output.copy_to_host()[0] == 30
+
+
+def test_reverse_input_iterator_scan():
+    h_input = np.asarray([1, 2, 3, 4, 5], dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, h_input.dtype)
+
+    cuda.compute.inclusive_scan(
+        d_in=ReverseIterator(d_input),
+        d_out=d_output,
+        op=OpKind.PLUS,
+        init_value=np.array([0], dtype=np.int32),
+        num_items=h_input.size,
+    )
+
+    np.testing.assert_array_equal(
+        d_output.copy_to_host(), np.asarray([5, 9, 12, 14, 15])
+    )
+
+
+def test_reverse_output_iterator_scan():
+    h_input = np.asarray([1, 2, 3, 4, 5], dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, h_input.dtype)
+
+    cuda.compute.inclusive_scan(
+        d_in=d_input,
+        d_out=ReverseIterator(d_output),
+        op=OpKind.PLUS,
+        init_value=np.array([0], dtype=np.int32),
+        num_items=h_input.size,
+    )
+
+    np.testing.assert_array_equal(
+        d_output.copy_to_host(), np.asarray([15, 10, 6, 3, 1])
+    )
+
+
+def test_permutation_iterator_reduce():
+    h_values = np.asarray([10, 20, 30, 40, 50, 60], dtype=np.int64)
+    h_indices = np.asarray([4, 2, 5, 1], dtype=np.int32)
+    d_values = DeviceArray.from_numpy(h_values)
+    d_indices = DeviceArray.from_numpy(h_indices)
+    d_output = DeviceArray.empty(1, np.int64)
+
+    cuda.compute.reduce_into(
+        d_in=PermutationIterator(d_values, d_indices),
+        d_out=d_output,
+        num_items=h_indices.size,
+        op=OpKind.PLUS,
+        h_init=np.array([0], dtype=np.int64),
+    )
+
+    assert d_output.copy_to_host()[0] == 160
+
+
+def test_transform_iterator_reduce():
+    d_output = DeviceArray.empty(1, np.int32)
+    iterator = TransformIterator(
+        CountingIterator(np.int32(1)), _raw_square_i32_op(), value_type=cccl_int32
+    )
+
+    cuda.compute.reduce_into(
+        d_in=iterator,
+        d_out=d_output,
+        num_items=6,
+        op=OpKind.PLUS,
+        h_init=np.array([0], dtype=np.int32),
+    )
+
+    assert d_output.copy_to_host()[0] == 91
+
+
+def test_transform_output_iterator_reduce():
+    h_input = np.asarray([1, 2, 3, 4], dtype=np.int16)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(1, np.int16)
+    output_iterator = TransformOutputIterator(
+        d_output, _raw_negate_i16_op(), output_value_type=cccl_int16
+    )
+
+    cuda.compute.reduce_into(
+        d_in=d_input,
+        d_out=output_iterator,
+        num_items=h_input.size,
+        op=OpKind.PLUS,
+        h_init=np.array([0], dtype=np.int16),
+    )
+
+    assert d_output.copy_to_host()[0] == -10
+
+
+def test_zip_iterator_transform():
+    h_lhs = np.asarray([1, 2, 3, 4, 5], dtype=np.int32)
+    h_rhs = np.asarray([10, 20, 30, 40, 50], dtype=np.int32)
+    d_lhs = DeviceArray.from_numpy(h_lhs)
+    d_rhs = DeviceArray.from_numpy(h_rhs)
+    d_output = DeviceArray.empty(h_lhs.shape, h_lhs.dtype)
+
+    cuda.compute.unary_transform(
+        d_in=ZipIterator(d_lhs, d_rhs),
+        d_out=d_output,
+        op=_raw_zip_sum_i32_op(),
+        num_items=h_lhs.size,
+    )
+
+    np.testing.assert_array_equal(d_output.copy_to_host(), h_lhs + h_rhs)
+
+
+def test_shuffle_iterator_transform():
+    num_items = 17
+    d_output = DeviceArray.empty(num_items, np.int64)
+
+    cuda.compute.unary_transform(
+        d_in=ShuffleIterator(num_items, seed=123),
+        d_out=d_output,
+        op=OpKind.IDENTITY,
+        num_items=num_items,
+    )
+
+    result = d_output.copy_to_host()
+    assert sorted(result.tolist()) == list(range(num_items))
+
+
+def test_discard_iterator_transform():
+    h_input = np.asarray([1, 2, 3, 4, 5], dtype=np.int32)
+    h_reference = np.full_like(h_input, -1)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_reference = DeviceArray.from_numpy(h_reference)
+
+    cuda.compute.unary_transform(
+        d_in=d_input,
+        d_out=DiscardIterator(d_reference),
+        op=OpKind.IDENTITY,
+        num_items=h_input.size,
+    )
+
+    np.testing.assert_array_equal(
+        d_reference.copy_to_host(), np.full(5, -1, dtype=np.int32)
+    )
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_numba_path_thread_safety.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_numba_path_thread_safety.py
new file mode 100644
index 00000000..ef9f3400
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_numba_path_thread_safety.py
@@ -0,0 +1,387 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Thread-safety of the shared build/cache machinery for the numba-dependent
+op paths, on regular (GIL) interpreters.
+
+cuda.compute releases the GIL around native builds and
+launches, so concurrent calls overlap in the C layer even under the GIL.
+So the shared caching/native machinery must be thread-safe on every interpreter
+build, and these tests validate that for the numba op paths.
+
+They cover the paths the free-threading stress suite
+(test_free_threading_stress.py) cannot: that suite runs on the minimal extra,
+which omits numba, so Python-callable ops, stateful closure ops, gpu_struct
+types, and return-type inference are untested there. This file exercises them
+from multiple threads on a regular GIL interpreter. The C-layer build/launch
+machinery itself is op-source-agnostic and already covered by the stress suite;
+the unique surface here is the numba frontend (CachableFunction hashing,
+inference, closure/struct handling).
+
+Only the native build/launch phase genuinely overlaps (the GIL is released
+around it); numba's own compilation serializes on its compiler lock and the
+Python-side key hashing serializes under the GIL. So the goal is not to prove
+numba runs in parallel, but that cuda.compute's caching, key hashing, and
+descriptor machinery stay correct when these paths are entered concurrently.
+"""
+
+import concurrent.futures
+import threading
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray
+
+import cuda.compute
+from cuda.compute import (
+    CountingIterator,
+    OpKind,
+    TransformIterator,
+    gpu_struct,
+    make_reduce_into,
+    make_unary_transform,
+)
+from cuda.core import Device
+
+pytestmark = pytest.mark.no_verify_sass(
+    reason="Concurrency tests intentionally run concurrent workers."
+)
+
+# Every input in this file is tiny (32-128 elements), so raising the thread count
+# adds no meaningful GPU-memory pressure -- it just exercises the concurrent
+# build/cache machinery harder.
+THREADS = 8
+ITERATIONS = 3
+
+# The distinct-op build storm is the canonical stress for concurrent NATIVE
+# builds. On the v2 (HostJIT) backend clang is embedded in-process and is not
+# thread-safe, so a regression in the compile-serialization lock corrupts the
+# heap and crashes here -- but only at high concurrency (at 4 threads it fires
+# only under external load). Use many threads so this stays a reliable gate.
+# One barrier-synced round of that many simultaneous builds already corrupts the
+# heap pre-fix (it crashed on the first round every time), and extra rounds are
+# expensive on v2 (builds serialize), so a single iteration is enough here.
+BUILD_STORM_THREADS = 24
+BUILD_STORM_ITERATIONS = 1
+
+
+def _run_threaded(workers):
+    # The default timeout turns a worker that dies before reaching the barrier
+    # into a BrokenBarrierError in its peers instead of hanging the CI job.
+    barrier = threading.Barrier(len(workers), timeout=60)
+    with concurrent.futures.ThreadPoolExecutor(max_workers=len(workers)) as executor:
+        futures = [executor.submit(worker, barrier) for worker in workers]
+        errors = []
+        for future in futures:
+            try:
+                future.result()
+            except BaseException as exc:  # noqa: BLE001
+                errors.append(exc)
+        if errors:
+            # Surface the root-cause worker failure, not the barrier breakage
+            # it caused in the other workers.
+            raise next(
+                (
+                    error
+                    for error in errors
+                    if not isinstance(error, threading.BrokenBarrierError)
+                ),
+                errors[0],
+            )
+
+
+def _reduce_with_temp(reducer, **kwargs):
+    temp_storage_bytes = reducer(temp_storage=None, **kwargs)
+    temp_storage = DeviceArray.empty(temp_storage_bytes, np.uint8)
+    return reducer(temp_storage=temp_storage, **kwargs)
+
+
+def _single_build_result(algorithm):
+    assert len(algorithm.build_results) == 1
+    return next(iter(algorithm.build_results.values()))
+
+
+def _make_clamped_max_op(k):
+    def clamped_max(a, b):
+        m = a if a > b else b
+        return m if m > k else k
+
+    return clamped_max
+
+
+def test_concurrent_distinct_python_ops_build_storm():
+    """Distinct Python callables force a separate numba+native build per worker.
+
+    Distinct closure constants give each worker its own cache key, so
+    _cache_single_flight elects a separate builder per thread. The native
+    builds overlap (the GIL is released around them) while the CachableFunction
+    hashing and numba compilation serialize (under the GIL and numba's own
+    compiler lock); the target is that this concurrent entry keeps each
+    worker's op and build uncontaminated. The op computes max(a, b, k) with a
+    per-worker k that dominates every input, so the expected result is exactly
+    k however CUB shapes the reduction tree — and a wrong k directly exposes
+    any cross-thread op/build contamination.
+    """
+    num_items = 64
+    for iteration in range(BUILD_STORM_ITERATIONS):
+        cuda.compute.clear_all_caches()
+        returned_reducers = [None] * BUILD_STORM_THREADS
+
+        def make_thread(worker_id):
+            k = 10_000 + worker_id * 7 + iteration
+            op = _make_clamped_max_op(k)
+            h_in = np.arange(num_items, dtype=np.int64) + worker_id
+            h_init = np.array([0], dtype=np.int64)
+            d_in = DeviceArray.from_numpy(h_in)
+            d_out = DeviceArray.empty(1, np.int64)
+
+            def thread(barrier):
+                # cuda.core device state is per-thread; initialize explicitly
+                # rather than relying on DeviceArray-construction side effects.
+                Device().set_current()
+                barrier.wait()
+                reducer = make_reduce_into(d_in=d_in, d_out=d_out, op=op, h_init=h_init)
+                returned_reducers[worker_id] = reducer
+                _reduce_with_temp(
+                    reducer,
+                    d_in=d_in,
+                    d_out=d_out,
+                    op=op,
+                    h_init=h_init,
+                    num_items=num_items,
+                )
+                Device().sync()
+                assert int(d_out.copy_to_host()[0]) == k
+
+            return thread
+
+        _run_threaded(
+            [make_thread(worker_id) for worker_id in range(BUILD_STORM_THREADS)]
+        )
+
+        build_ids = {id(_single_build_result(r)) for r in returned_reducers}
+        assert len(build_ids) == BUILD_STORM_THREADS
+
+
+def test_concurrent_shared_python_op_coalesces():
+    """One shared Python callable from all threads coalesces to a single build.
+
+    All workers hash the same function concurrently (CachableFunction walks
+    bytecode, constants, and closures on every factory call) and race the
+    same cache key; exactly one build must result, with one wrapper per thread.
+    """
+
+    def add(a, b):
+        return a + b
+
+    num_items = 64
+    for iteration in range(ITERATIONS):
+        cuda.compute.clear_all_caches()
+        returned_reducers = [None] * THREADS
+
+        def make_thread(worker_id):
+            h_in = np.arange(num_items, dtype=np.int32) + worker_id + iteration
+            h_init = np.array([worker_id], dtype=np.int32)
+            d_in = DeviceArray.from_numpy(h_in)
+            d_out = DeviceArray.empty(1, np.int32)
+
+            def thread(barrier):
+                # cuda.core device state is per-thread; initialize explicitly
+                # rather than relying on DeviceArray-construction side effects.
+                Device().set_current()
+                barrier.wait()
+                reducer = make_reduce_into(
+                    d_in=d_in, d_out=d_out, op=add, h_init=h_init
+                )
+                returned_reducers[worker_id] = reducer
+                _reduce_with_temp(
+                    reducer,
+                    d_in=d_in,
+                    d_out=d_out,
+                    op=add,
+                    h_init=h_init,
+                    num_items=num_items,
+                )
+                Device().sync()
+                assert int(d_out.copy_to_host()[0]) == int(h_in.sum()) + worker_id
+
+            return thread
+
+        _run_threaded([make_thread(worker_id) for worker_id in range(THREADS)])
+
+        # Per-thread wrappers, one shared native build.
+        assert len({id(r) for r in returned_reducers}) == THREADS
+        assert len({id(_single_build_result(r)) for r in returned_reducers}) == 1
+
+
+def _make_scale_op(k):
+    def scale(x):
+        return x * k
+
+    return scale
+
+
+def test_concurrent_transform_iterator_return_type_inference():
+    """Unannotated TransformIterator ops infer return types concurrently.
+
+    TransformIterator without value_type routes through _infer_return_type
+    (numba typing). Ops are distinct per worker and per iteration so the
+    inference cache stays cold and each thread runs the typing path itself.
+    """
+    num_items = 32
+    for iteration in range(ITERATIONS):
+        cuda.compute.clear_all_caches()
+
+        def make_thread(worker_id):
+            k = np.int32(worker_id + 1 + iteration * THREADS)
+            op = _make_scale_op(k)
+            h_init = np.array([0], dtype=np.int32)
+            d_out = DeviceArray.empty(1, np.int32)
+
+            def thread(barrier):
+                # cuda.core device state is per-thread; initialize explicitly
+                # rather than relying on DeviceArray-construction side effects.
+                Device().set_current()
+                barrier.wait()
+                # Inference widens int32 * int32-closure to an int64 value type;
+                # the int64-iterator/int32-accumulator mix is a supported,
+                # CI-exercised configuration and all expected values fit int32.
+                d_in = TransformIterator(CountingIterator(np.int32(0)), op)
+                reducer = make_reduce_into(
+                    d_in=d_in, d_out=d_out, op=OpKind.PLUS, h_init=h_init
+                )
+                _reduce_with_temp(
+                    reducer,
+                    d_in=d_in,
+                    d_out=d_out,
+                    op=OpKind.PLUS,
+                    h_init=h_init,
+                    num_items=num_items,
+                )
+                Device().sync()
+                expected = sum(i * int(k) for i in range(num_items))
+                assert int(d_out.copy_to_host()[0]) == expected
+
+            return thread
+
+        _run_threaded([make_thread(worker_id) for worker_id in range(THREADS)])
+
+
+def test_concurrent_stateful_closure_ops_isolate_state():
+    """Same op code with different captured device arrays across threads.
+
+    The captured arrays do not change the cache key (stateful op machinery
+    updates pointers per call), so all threads share one native build while
+    each thread's wrapper must carry its own captured-state pointer. If state
+    isolation broke, one thread's output would use another thread's offset.
+    """
+
+    def make_adder(arr):
+        def add_offset(x):
+            return x + arr[0]
+
+        return add_offset
+
+    num_items = 64
+    for iteration in range(ITERATIONS):
+        cuda.compute.clear_all_caches()
+        returned_transformers = [None] * THREADS
+
+        def make_thread(worker_id):
+            offset = worker_id * 10 + iteration
+            d_offset = DeviceArray.from_numpy(np.array([offset], dtype=np.int32))
+            op = make_adder(d_offset)
+            h_in = np.arange(num_items, dtype=np.int32)
+            d_in = DeviceArray.from_numpy(h_in)
+            d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+
+            def thread(barrier):
+                # cuda.core device state is per-thread; initialize explicitly
+                # rather than relying on DeviceArray-construction side effects.
+                Device().set_current()
+                barrier.wait()
+                transformer = make_unary_transform(d_in=d_in, d_out=d_out, op=op)
+                returned_transformers[worker_id] = transformer
+                transformer(d_in=d_in, d_out=d_out, op=op, num_items=num_items)
+                Device().sync()
+                np.testing.assert_array_equal(d_out.copy_to_host(), h_in + offset)
+
+            return thread
+
+        _run_threaded([make_thread(worker_id) for worker_id in range(THREADS)])
+
+        assert len({id(t) for t in returned_transformers}) == THREADS
+        assert len({id(_single_build_result(t)) for t in returned_transformers}) == 1
+
+
+def test_concurrent_gpu_struct_reduce():
+    """gpu_struct types and struct-typed Python ops used from multiple threads."""
+
+    @gpu_struct
+    class MinMax:
+        min_val: np.int32
+        max_val: np.int32
+
+    def minmax_op(a, b):
+        c_min = min(a.min_val, b.min_val)
+        c_max = max(a.max_val, b.max_val)
+        return MinMax(c_min, c_max)
+
+    num_items = 128
+    info = np.iinfo(np.int32)
+
+    # Warm the numba struct registration once on the main thread (mirroring the
+    # stress suite's warm pass). First-time registration goes through
+    # non-coalescing lru_caches in _jit.py that mutate numba's global
+    # registries, so racing it cold from four threads is a separate library
+    # hardening question, not this test's target.
+    warm_in = DeviceArray.from_numpy(
+        np.zeros((1, 2), dtype=np.int32).view(MinMax.dtype)
+    )
+    warm_out = DeviceArray.empty(1, MinMax.dtype)
+    cuda.compute.reduce_into(
+        d_in=warm_in,
+        d_out=warm_out,
+        op=minmax_op,
+        h_init=MinMax(info.max, info.min),
+        num_items=1,
+    )
+    Device().sync()
+
+    for iteration in range(ITERATIONS):
+        cuda.compute.clear_all_caches()
+
+        def make_thread(worker_id):
+            base = worker_id * 1000 + iteration
+            h_pairs = np.stack(
+                [
+                    np.arange(num_items, dtype=np.int32) + base,
+                    np.arange(num_items, dtype=np.int32) * 2 + base,
+                ],
+                axis=1,
+            )
+            d_in = DeviceArray.from_numpy(h_pairs.view(MinMax.dtype))
+            d_out = DeviceArray.empty(1, MinMax.dtype)
+            h_init = MinMax(info.max, info.min)
+
+            def thread(barrier):
+                # cuda.core device state is per-thread; initialize explicitly
+                # rather than relying on DeviceArray-construction side effects.
+                Device().set_current()
+                barrier.wait()
+                cuda.compute.reduce_into(
+                    d_in=d_in,
+                    d_out=d_out,
+                    op=minmax_op,
+                    h_init=h_init,
+                    num_items=num_items,
+                )
+                Device().sync()
+                result = d_out.copy_to_host()
+                assert int(result["min_val"][0]) == int(h_pairs[:, 0].min())
+                assert int(result["max_val"][0]) == int(h_pairs[:, 1].max())
+
+            return thread
+
+        _run_threaded([make_thread(worker_id) for worker_id in range(THREADS)])
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_permutation_iterator.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_permutation_iterator.py
new file mode 100644
index 00000000..862f058f
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_permutation_iterator.py
@@ -0,0 +1,263 @@
+# Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+import numpy as np
+from _utils.device_array import DeviceArray
+
+import cuda.compute
+from cuda.compute.iterators import (
+    CountingIterator,
+    PermutationIterator,
+    ZipIterator,
+)
+
+
+def test_permutation_iterator_equality():
+    d_values1 = DeviceArray.from_numpy(np.asarray([10, 20, 30, 40, 50], dtype="int32"))
+    d_values2 = DeviceArray.from_numpy(np.asarray([100, 200, 300], dtype="int32"))
+    d_values3 = DeviceArray.from_numpy(np.asarray([10, 20, 30, 40, 50], dtype="int64"))
+
+    d_indices1 = DeviceArray.from_numpy(np.asarray([0, 2, 1], dtype="int32"))
+    d_indices2 = DeviceArray.from_numpy(np.asarray([1, 0, 2], dtype="int32"))
+    d_indices3 = DeviceArray.from_numpy(np.asarray([0, 2, 1], dtype="int64"))
+
+    # Same value and index types should have same kind
+    it1 = PermutationIterator(d_values1, d_indices1)
+    it2 = PermutationIterator(d_values1, d_indices2)
+    it3 = PermutationIterator(d_values2, d_indices1)
+
+    assert it1.kind == it2.kind == it3.kind
+
+    # Different value type should have different kind
+    it4 = PermutationIterator(d_values3, d_indices1)
+    assert it1.kind != it4.kind
+
+    # Different index type should have different kind
+    it5 = PermutationIterator(d_values1, d_indices3)
+    assert it1.kind != it5.kind
+
+
+def test_permutation_iterator_with_array_values():
+    h_values = np.asarray([10, 20, 30, 40, 50], dtype="int32")
+    h_indices = np.asarray([2, 0, 4, 1], dtype="int32")
+    d_values = DeviceArray.from_numpy(h_values)
+    d_indices = DeviceArray.from_numpy(h_indices)
+    perm_it = PermutationIterator(d_values, d_indices)
+
+    h_init = np.array([0], dtype="int32")
+    d_output = DeviceArray.empty(1, np.int32)
+    cuda.compute.reduce_into(
+        d_in=perm_it,
+        d_out=d_output,
+        num_items=len(h_indices),
+        op=cuda.compute.OpKind.PLUS,
+        h_init=h_init,
+    )
+    assert d_output.copy_to_host()[0] == h_values[h_indices].sum()
+
+
+def test_permutation_iterator_with_iterator_values():
+    values_it = CountingIterator(np.int32(10))
+    h_indices = np.asarray([2, 0, 4, 1], dtype="int32")
+    d_indices = DeviceArray.from_numpy(h_indices)
+    perm_it = PermutationIterator(values_it, d_indices)
+
+    h_init = np.array([0], dtype="int32")
+    d_output = DeviceArray.empty(1, np.int32)
+
+    cuda.compute.reduce_into(
+        d_in=perm_it,
+        d_out=d_output,
+        num_items=len(h_indices),
+        op=cuda.compute.OpKind.PLUS,
+        h_init=h_init,
+    )
+
+    expected = np.arange(10, 20)[h_indices].sum()
+    assert d_output.copy_to_host()[0] == expected
+
+
+def test_permutation_iterator_of_zip_iterator():
+    @cuda.compute.gpu_struct
+    class Pair:
+        value_0: np.int32
+        value_1: np.int32
+
+    h_values1 = np.asarray([10, 20, 30, 40, 50], dtype="int32")
+    h_values2 = np.asarray([1, 2, 3, 4, 5], dtype="int32")
+    d_values1 = DeviceArray.from_numpy(h_values1)
+    d_values2 = DeviceArray.from_numpy(h_values2)
+    zip_it = ZipIterator(d_values1, d_values2)
+    h_indices = np.asarray([2, 0, 4], dtype="int32")
+    d_indices = DeviceArray.from_numpy(h_indices)
+    perm_it = PermutationIterator(zip_it, d_indices)
+
+    def sum_both_fields(a, b):
+        return Pair(a.value_0 + b.value_0, a.value_1 + b.value_1)
+
+    h_init = Pair(0, 0)
+    d_output = DeviceArray.empty(1, Pair.dtype)
+
+    cuda.compute.reduce_into(
+        d_in=perm_it,
+        d_out=d_output,
+        num_items=len(h_indices),
+        op=sum_both_fields,
+        h_init=h_init,
+    )
+
+    result = d_output.copy_to_host()[0]
+    assert result["value_0"] == h_values1[h_indices].sum()
+    assert result["value_1"] == h_values2[h_indices].sum()
+
+
+def test_zip_iterator_of_permutation_iterators():
+    @cuda.compute.gpu_struct
+    class Pair:
+        value_0: np.int32
+        value_1: np.int32
+
+    h_values1 = np.asarray([10, 20, 30, 40, 50], dtype="int32")
+    h_values2 = np.asarray([100, 200, 300, 400, 500], dtype="int32")
+    h_indices1 = np.asarray([4, 1, 3, 0], dtype="int32")
+    h_indices2 = np.asarray([2, 4, 0, 1], dtype="int32")
+    d_values1 = DeviceArray.from_numpy(h_values1)
+    d_values2 = DeviceArray.from_numpy(h_values2)
+    d_indices1 = DeviceArray.from_numpy(h_indices1)
+    d_indices2 = DeviceArray.from_numpy(h_indices2)
+    perm_it1 = PermutationIterator(d_values1, d_indices1)
+    perm_it2 = PermutationIterator(d_values2, d_indices2)
+
+    zip_it = ZipIterator(perm_it1, perm_it2)
+
+    def sum_both_fields(a, b):
+        return Pair(a.value_0 + b.value_0, a.value_1 + b.value_1)
+
+    h_init = Pair(0, 0)
+    d_output = DeviceArray.empty(1, Pair.dtype)
+
+    num_items = len(h_indices1)
+    cuda.compute.reduce_into(
+        d_in=zip_it,
+        d_out=d_output,
+        num_items=num_items,
+        op=sum_both_fields,
+        h_init=h_init,
+    )
+
+    result = d_output.copy_to_host()[0]
+    assert result["value_0"] == h_values1[h_indices1].sum()
+    assert result["value_1"] == h_values2[h_indices2].sum()
+
+
+def test_unary_transform_of_permutation_iterator():
+    h_values = np.asarray([10, 20, 30, 40, 50], dtype="int32")
+    h_indices = np.asarray([2, 0, 4, 1], dtype="int32")
+    d_values = DeviceArray.from_numpy(h_values)
+    d_indices = DeviceArray.from_numpy(h_indices)
+    perm_it = PermutationIterator(d_values, d_indices)
+
+    def op(a):
+        return a + 1
+
+    d_out = DeviceArray.empty(len(h_indices), h_values.dtype)
+    cuda.compute.unary_transform(
+        d_in=perm_it, d_out=d_out, op=op, num_items=len(h_indices)
+    )
+
+    expected = h_values[h_indices] + 1
+    np.testing.assert_array_equal(d_out.copy_to_host(), expected)
+
+
+def test_caching_permutation_iterator():
+    """Test that iterator compilation is cached across instances with the same structure."""
+    from cuda.compute._cpp_compile import compile_cpp_op_code
+
+    # Test 1: Same structure → same kind
+    it1 = PermutationIterator(
+        DeviceArray.from_numpy(np.arange(10, dtype=np.int32)),
+        DeviceArray.from_numpy(np.arange(10, dtype=np.int32)),
+    )
+    it2 = PermutationIterator(
+        DeviceArray.from_numpy(np.arange(20, dtype=np.int32)),
+        DeviceArray.from_numpy(np.arange(5, dtype=np.int32)),
+    )
+    assert it1.kind == it2.kind, "Same structure should have same kind"
+
+    # Test 2: Different index type → different kind
+    it3 = PermutationIterator(
+        DeviceArray.from_numpy(np.arange(10, dtype=np.int32)),
+        DeviceArray.from_numpy(np.arange(10, dtype=np.int64)),
+    )
+    assert it1.kind != it3.kind, "Different index type should have different kind"
+
+    # Test 3: Different value type → different kind
+    it4 = PermutationIterator(
+        DeviceArray.from_numpy(np.arange(10, dtype=np.int64)),
+        DeviceArray.from_numpy(np.arange(10, dtype=np.int32)),
+    )
+    assert it1.kind != it4.kind, "Different value type should have different kind"
+
+    # Test 4: Verify compilation caching with cache statistics
+    compile_cpp_op_code.cache_clear()
+
+    # Create multiple instances with same structure
+    iterators = []
+    for i in range(3):
+        it = PermutationIterator(
+            DeviceArray.from_numpy(np.arange(i * 10, (i + 1) * 10, dtype=np.float32)),
+            DeviceArray.from_numpy(np.arange(5, dtype=np.int32)),
+        )
+        # Trigger compilation by accessing Op objects
+        it.get_advance_op()
+        it.get_input_deref_op()
+        iterators.append(it)
+
+    cache_info = compile_cpp_op_code.cache_info()
+    assert cache_info.hits >= 2, (
+        f"Expected cache hits for same structure, got {cache_info.hits} hits, "
+        f"{cache_info.misses} misses"
+    )
+
+
+def test_permutation_iterator_advance():
+    """Test PermutationIterator.__add__ only advances indices, not values."""
+    # Create values array [10, 20, 30, 40, 50, 60, 70]
+    h_values = np.asarray([10, 20, 30, 40, 50, 60, 70], dtype="int32")
+    d_values = DeviceArray.from_numpy(h_values)
+
+    # Create indices array [2, 0, 4, 1, 3, 5]
+    # indices[0] = 2 -> values[2] = 30
+    # indices[1] = 0 -> values[0] = 10
+    # indices[2] = 4 -> values[4] = 50
+    # indices[3] = 1 -> values[1] = 20
+    # indices[4] = 3 -> values[3] = 40
+    # indices[5] = 5 -> values[5] = 60
+    h_indices = np.asarray([2, 0, 4, 1, 3, 5], dtype="int32")
+    d_indices = DeviceArray.from_numpy(h_indices)
+
+    perm_it = PermutationIterator(d_values, d_indices)
+
+    # Advance by 2 positions (should skip first 2 indices)
+    offset = 2
+    advanced_perm_it = perm_it + offset
+
+    # Reduce from the advanced position
+    # Should process indices[2:] = [4, 1, 3, 5]
+    # Which accesses values[4, 1, 3, 5] = [50, 20, 40, 60]
+    h_init = np.array([0], dtype="int32")
+    d_output = DeviceArray.empty(1, np.int32)
+
+    remaining_items = len(h_indices) - offset
+    cuda.compute.reduce_into(
+        d_in=advanced_perm_it,
+        d_out=d_output,
+        num_items=remaining_items,
+        op=cuda.compute.OpKind.PLUS,
+        h_init=h_init,
+    )
+
+    # Expected: values[indices[2:]] = values[[4, 1, 3, 5]] = [50, 20, 40, 60]
+    expected = h_values[h_indices[offset:]].sum()
+    result = d_output.copy_to_host()[0]
+    assert result == expected, f"Expected {expected}, got {result}"
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_radix_sort.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_radix_sort.py
new file mode 100644
index 00000000..68410528
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_radix_sort.py
@@ -0,0 +1,629 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import itertools
+from typing import Tuple
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray, get_compute_capability
+
+import cuda.compute
+from cuda.compute import (
+    DoubleBuffer,
+    SortOrder,
+    deserialize,
+    make_radix_sort,
+    serialize,
+)
+from cuda.compute._utils.temp_storage_buffer import TempStorageBuffer
+
+
+def get_mark(dt, log_size):
+    if log_size < 20:
+        return tuple()
+    return pytest.mark.large
+
+
+DTYPE_LIST = [
+    np.uint8,
+    np.uint16,
+    np.uint32,
+    np.uint64,
+    np.int8,
+    np.int16,
+    np.int32,
+    np.int64,
+    np.float16,
+    np.float32,
+    np.float64,
+]
+
+PROBLEM_SIZES = [2, 10, 20]
+
+DTYPE_SIZE = [
+    pytest.param(dt, 2**log_size, marks=get_mark(dt, log_size))
+    for dt in DTYPE_LIST
+    for log_size in PROBLEM_SIZES
+]
+
+
+def random_array(size, dtype, max_value=None) -> np.typing.NDArray:
+    rng = np.random.default_rng()
+    if np.isdtype(dtype, "integral"):
+        if max_value is None:
+            max_value = np.iinfo(dtype).max
+        return rng.integers(max_value, size=size, dtype=dtype)
+    elif np.isdtype(dtype, "real floating"):
+        return np.random.uniform(low=-10.0, high=10.0, size=size).astype(dtype)
+    else:
+        raise ValueError(f"Unsupported dtype {dtype}")
+
+
+def radix_sort_device(
+    d_in_keys,
+    d_out_keys,
+    d_in_values,
+    d_out_values,
+    order,
+    num_items,
+    begin_bit=None,
+    end_bit=None,
+    stream=None,
+):
+    # Use the new single-phase API with automatic temp storage allocation
+    cuda.compute.radix_sort(
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_values,
+        d_out_values=d_out_values,
+        num_items=num_items,
+        order=order,
+        begin_bit=begin_bit,
+        end_bit=end_bit,
+        stream=stream,
+    )
+
+
+def get_floating_point_keys(array):
+    """
+    This function computes the keys for floating point types.
+    From the cub docs, this is the required behavior:
+
+    For positive floating point values, the sign bit is inverted.
+    For negative floating point values, the full key is inverted.
+    """
+    if array.dtype == np.float32:
+        uint_type = np.uint32
+        sign_mask = np.uint32(0x80000000)
+    elif array.dtype == np.float64:
+        uint_type = np.uint64
+        sign_mask = np.uint64(0x8000000000000000)
+
+    # Get the binary representation as unsigned integers
+    binary = array.copy().view(uint_type)
+
+    # Create masks for positive and negative numbers
+    is_positive = array >= 0
+    is_negative = ~is_positive
+
+    # For positive numbers: flip the sign bit (leftmost bit)
+    binary[is_positive] ^= sign_mask
+
+    # For negative numbers: invert all bits
+    binary[is_negative] = ~binary[is_negative]
+
+    return binary
+
+
+def host_sort(h_in_keys, h_in_values, order, begin_bit=None, end_bit=None) -> Tuple:
+    if begin_bit is not None and end_bit is not None:
+        num_bits = end_bit - begin_bit
+        mask = np.array(((1 << (num_bits)) - 1) << begin_bit, dtype=np.uint64)
+
+        if np.issubdtype(h_in_keys.dtype, np.floating):
+            h_in_keys_copy = get_floating_point_keys(h_in_keys)
+        else:
+            h_in_keys_copy = h_in_keys
+
+        h_in_keys_copy = h_in_keys_copy.astype(np.uint64)
+        h_in_keys_copy = (h_in_keys_copy & mask) >> begin_bit
+    else:
+        h_in_keys_copy = h_in_keys
+
+    if order is SortOrder.DESCENDING:
+        # We do this for stability. We need to cast to a signed integer to properly negate the keys.
+        signed_dtype = np.dtype(h_in_keys_copy.dtype.name.replace("uint", "int"))
+        argsort = np.argsort(-h_in_keys_copy.astype(signed_dtype), stable=True)
+    else:
+        argsort = np.argsort(h_in_keys_copy, stable=True)
+
+    h_in_keys = h_in_keys[argsort]
+    if h_in_values is not None:
+        h_in_values = h_in_values[argsort]
+
+    return h_in_keys, h_in_values
+
+
+@pytest.mark.parametrize(
+    "dtype, num_items",
+    DTYPE_SIZE,
+)
+def test_radix_sort_keys(dtype, num_items, monkeypatch):
+    cc_major, _ = get_compute_capability()
+    # Skip sass verification for CC 9.0+ due to a bug in NVRTC.
+    # TODO: add NVRTC version check, ref nvbug 5243118
+    if cc_major >= 9:
+        import cuda.compute._cccl_interop
+
+        monkeypatch.setattr(
+            cuda.compute._cccl_interop,
+            "_check_sass",
+            False,
+        )
+
+    order = SortOrder.ASCENDING
+    h_in_keys = random_array(num_items, dtype, max_value=20)
+    h_out_keys = np.empty(num_items, dtype=dtype)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_out_keys = DeviceArray.empty(h_out_keys.shape, h_out_keys.dtype)
+
+    radix_sort_device(d_in_keys, d_out_keys, None, None, order, num_items)
+
+    h_out_keys = d_out_keys.copy_to_host()
+
+    h_in_keys, _ = host_sort(h_in_keys, None, order)
+
+    np.testing.assert_array_equal(h_out_keys, h_in_keys)
+
+
+@pytest.mark.parametrize(
+    "dtype, num_items",
+    DTYPE_SIZE,
+)
+def test_radix_sort_pairs(dtype, num_items, monkeypatch):
+    import cuda.compute._cccl_interop
+
+    monkeypatch.setattr(
+        cuda.compute._cccl_interop,
+        "_check_sass",
+        False,
+    )
+
+    order = SortOrder.DESCENDING
+    h_in_keys = random_array(num_items, dtype, max_value=20)
+    h_in_values = random_array(num_items, np.float32)
+    h_out_keys = np.empty(num_items, dtype=dtype)
+    h_out_values = np.empty(num_items, dtype=np.float32)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_values = DeviceArray.from_numpy(h_in_values)
+    d_out_keys = DeviceArray.empty(h_out_keys.shape, h_out_keys.dtype)
+    d_out_values = DeviceArray.empty(h_out_values.shape, h_out_values.dtype)
+
+    radix_sort_device(
+        d_in_keys, d_out_keys, d_in_values, d_out_values, order, num_items
+    )
+
+    h_out_keys = d_out_keys.copy_to_host()
+    h_out_values = d_out_values.copy_to_host()
+
+    h_in_keys, h_in_values = host_sort(h_in_keys, h_in_values, order)
+
+    np.testing.assert_array_equal(h_out_keys, h_in_keys)
+    np.testing.assert_array_equal(h_out_values, h_in_values)
+
+
+@pytest.mark.parametrize(
+    "dtype, num_items",
+    DTYPE_SIZE,
+)
+def test_radix_sort_keys_double_buffer(dtype, num_items, monkeypatch):
+    cc_major, _ = get_compute_capability()
+    # Skip sass verification for CC 9.0+ due to a bug in NVRTC.
+    # TODO: add NVRTC version check, ref nvbug 5243118
+    if cc_major >= 9:
+        import cuda.compute._cccl_interop
+
+        monkeypatch.setattr(
+            cuda.compute._cccl_interop,
+            "_check_sass",
+            False,
+        )
+
+    order = SortOrder.DESCENDING
+    h_in_keys = random_array(num_items, dtype, max_value=20)
+    h_out_keys = np.empty(num_items, dtype=dtype)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_out_keys = DeviceArray.empty(h_out_keys.shape, h_out_keys.dtype)
+
+    keys_double_buffer = DoubleBuffer(d_in_keys, d_out_keys)
+
+    radix_sort_device(keys_double_buffer, None, None, None, order, num_items)
+
+    h_out_keys = keys_double_buffer.current().copy_to_host()
+
+    h_in_keys, _ = host_sort(h_in_keys, None, order)
+
+    np.testing.assert_array_equal(h_out_keys, h_in_keys)
+
+
+@pytest.mark.parametrize(
+    "dtype, num_items",
+    DTYPE_SIZE,
+)
+def test_radix_sort_pairs_double_buffer(dtype, num_items, monkeypatch):
+    cc_major, _ = get_compute_capability()
+    # NOTE: int16 failures seen only with NVRTC 13.1:
+    if cc_major >= 9 or np.isdtype(dtype, (np.int16, np.uint32)):
+        import cuda.compute._cccl_interop
+
+        monkeypatch.setattr(
+            cuda.compute._cccl_interop,
+            "_check_sass",
+            False,
+        )
+
+    order = SortOrder.ASCENDING
+    h_in_keys = random_array(num_items, dtype, max_value=20)
+    h_in_values = random_array(num_items, np.float32)
+    h_out_keys = np.empty(num_items, dtype=dtype)
+    h_out_values = np.empty(num_items, dtype=np.float32)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_values = DeviceArray.from_numpy(h_in_values)
+    d_out_keys = DeviceArray.empty(h_out_keys.shape, h_out_keys.dtype)
+    d_out_values = DeviceArray.empty(h_out_values.shape, h_out_values.dtype)
+
+    keys_double_buffer = DoubleBuffer(d_in_keys, d_out_keys)
+    values_double_buffer = DoubleBuffer(d_in_values, d_out_values)
+
+    radix_sort_device(
+        keys_double_buffer, None, values_double_buffer, None, order, num_items
+    )
+
+    h_out_keys = keys_double_buffer.current().copy_to_host()
+    h_out_values = values_double_buffer.current().copy_to_host()
+
+    h_in_keys, h_in_values = host_sort(h_in_keys, h_in_values, order)
+
+    np.testing.assert_array_equal(h_out_keys, h_in_keys)
+    np.testing.assert_array_equal(h_out_values, h_in_values)
+
+
+# These tests take longer to execute so we reduce the number of test cases
+DTYPE_SIZE_BIT_WINDOW = [
+    pytest.param(dt, 2**log_size, marks=get_mark(dt, log_size))
+    for dt in [np.uint8, np.int16, np.uint32, np.int64, np.float64]
+    for log_size in [2, 24]
+]
+
+
+@pytest.mark.parametrize(
+    "dtype, num_items",
+    DTYPE_SIZE_BIT_WINDOW,
+)
+def test_radix_sort_pairs_bit_window(dtype, num_items, monkeypatch):
+    cc_major, _ = get_compute_capability()
+    # NOTE: int16 failures seen only with NVRTC 13.1:
+    if cc_major >= 9 or np.isdtype(dtype, (np.int16, np.uint32)):
+        import cuda.compute._cccl_interop
+
+        monkeypatch.setattr(
+            cuda.compute._cccl_interop,
+            "_check_sass",
+            False,
+        )
+
+    order = SortOrder.ASCENDING
+    num_bits = dtype().itemsize
+    begin_bits = [0, num_bits // 3, 3 * num_bits // 4, num_bits]
+    end_bits = [0, num_bits // 3, 3 * num_bits // 4, num_bits]
+
+    for begin_bit, end_bit in itertools.product(begin_bits, end_bits):
+        if end_bit < begin_bit:
+            continue
+
+        h_in_keys = random_array(num_items, dtype)
+        h_in_values = random_array(num_items, np.float32)
+        h_out_keys = np.empty(num_items, dtype=dtype)
+        h_out_values = np.empty(num_items, dtype=np.float32)
+
+        d_in_keys = DeviceArray.from_numpy(h_in_keys)
+        d_in_values = DeviceArray.from_numpy(h_in_values)
+        d_out_keys = DeviceArray.empty(h_out_keys.shape, h_out_keys.dtype)
+        d_out_values = DeviceArray.empty(h_out_values.shape, h_out_values.dtype)
+
+        radix_sort_device(
+            d_in_keys,
+            d_out_keys,
+            d_in_values,
+            d_out_values,
+            order,
+            num_items,
+            begin_bit,
+            end_bit,
+        )
+
+        h_out_keys = d_out_keys.copy_to_host()
+        h_out_values = d_out_values.copy_to_host()
+
+        h_in_keys, h_in_values = host_sort(
+            h_in_keys, h_in_values, order, begin_bit, end_bit
+        )
+
+        np.testing.assert_array_equal(h_out_keys, h_in_keys)
+        np.testing.assert_array_equal(h_out_values, h_in_values)
+
+
+@pytest.mark.parametrize(
+    "dtype, num_items",
+    DTYPE_SIZE_BIT_WINDOW,
+)
+def test_radix_sort_pairs_double_buffer_bit_window(dtype, num_items, monkeypatch):
+    if np.isdtype(dtype, (np.uint8, np.int16, np.uint32)):
+        import cuda.compute._cccl_interop
+
+        monkeypatch.setattr(
+            cuda.compute._cccl_interop,
+            "_check_sass",
+            False,
+        )
+
+    order = SortOrder.DESCENDING
+    num_bits = dtype().itemsize
+    begin_bits = [0, num_bits // 3, 3 * num_bits // 4, num_bits]
+    end_bits = [0, num_bits // 3, 3 * num_bits // 4, num_bits]
+
+    for begin_bit, end_bit in itertools.product(begin_bits, end_bits):
+        if end_bit < begin_bit:
+            continue
+
+        h_in_keys = random_array(num_items, dtype)
+        h_in_values = random_array(num_items, np.float32)
+        h_out_keys = np.empty(num_items, dtype=dtype)
+        h_out_values = np.empty(num_items, dtype=np.float32)
+
+        d_in_keys = DeviceArray.from_numpy(h_in_keys)
+        d_in_values = DeviceArray.from_numpy(h_in_values)
+        d_out_keys = DeviceArray.empty(h_out_keys.shape, h_out_keys.dtype)
+        d_out_values = DeviceArray.empty(h_out_values.shape, h_out_values.dtype)
+
+        keys_double_buffer = DoubleBuffer(d_in_keys, d_out_keys)
+        values_double_buffer = DoubleBuffer(d_in_values, d_out_values)
+
+        radix_sort_device(
+            keys_double_buffer,
+            None,
+            values_double_buffer,
+            None,
+            order,
+            num_items,
+            begin_bit,
+            end_bit,
+        )
+
+        h_out_keys = keys_double_buffer.current().copy_to_host()
+        h_out_values = values_double_buffer.current().copy_to_host()
+
+        h_in_keys, h_in_values = host_sort(
+            h_in_keys, h_in_values, order, begin_bit, end_bit
+        )
+
+        np.testing.assert_array_equal(h_out_keys, h_in_keys)
+        np.testing.assert_array_equal(h_out_values, h_in_values)
+
+
+@pytest.mark.large
+@pytest.mark.parametrize("dtype", [np.int32, np.float32])
+def test_radix_sort_large_num_items(dtype, monkeypatch):
+    """Regression test for https://github.com/NVIDIA/cccl/issues/7938.
+
+    Radix sort produces incorrect output for large inputs that require
+    multiple "portions" internally (roughly >= 2**28 elements, depending
+    on the tuning policy chosen for the current GPU).
+    """
+    import cuda.compute._cccl_interop
+
+    monkeypatch.setattr(
+        cuda.compute._cccl_interop,
+        "_check_sass",
+        False,
+    )
+    num_items = 2**28
+
+    h_in_keys = np.arange(num_items - 1, -1, -1, dtype=dtype)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_out_keys = DeviceArray.empty(num_items, dtype)
+
+    cuda.compute.radix_sort(
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=None,
+        d_out_values=None,
+        num_items=num_items,
+        order=SortOrder.ASCENDING,
+    )
+
+    h_out_keys = d_out_keys.copy_to_host()
+    h_expected, _ = host_sort(h_in_keys, None, SortOrder.ASCENDING)
+
+    np.testing.assert_array_equal(h_out_keys, h_expected)
+
+
+def test_radix_sort_with_stream(cuda_stream):
+    num_items = 10000
+
+    h_in_keys = random_array(num_items, np.int32)
+    d_in_keys = DeviceArray.from_numpy(h_in_keys, stream=cuda_stream)
+    d_out_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype, stream=cuda_stream)
+
+    radix_sort_device(
+        d_in_keys,
+        d_out_keys,
+        None,
+        None,
+        SortOrder.ASCENDING,
+        num_items,
+        stream=cuda_stream,
+    )
+
+    got = d_out_keys.copy_to_host(stream=cuda_stream)
+    h_in_keys.sort()
+
+    np.testing.assert_array_equal(got, h_in_keys)
+
+
+def test_radix_sort(monkeypatch):
+    cc_major, _ = get_compute_capability()
+    # Skip sass verification for CC 9.0+ due to a bug in NVRTC.
+    # TODO: add NVRTC version check, ref nvbug 5243118
+    if cc_major >= 9:
+        import cuda.compute._cccl_interop as cccl_interop
+
+        monkeypatch.setattr(
+            cccl_interop,
+            "_check_sass",
+            False,
+        )
+
+    h_in_keys = np.array([-5, 0, 2, -3, 2, 4, 0, -1, 2, 8], dtype="int32")
+    h_in_values = np.array(
+        [-3.2, 2.2, 1.9, 4.0, -3.9, 2.7, 0, 8.3 - 1, 2.9, 5.4], dtype="float32"
+    )
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_values = DeviceArray.from_numpy(h_in_values)
+
+    d_out_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype)
+    d_out_values = DeviceArray.empty(h_in_values.shape, h_in_values.dtype)
+
+    # Call single-phase API directly with num_items parameter
+    cuda.compute.radix_sort(
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_values,
+        d_out_values=d_out_values,
+        num_items=h_in_keys.size,
+        order=SortOrder.ASCENDING,
+    )
+
+    # Check the result is correct
+    h_out_keys = d_out_keys.copy_to_host()
+    h_out_items = d_out_values.copy_to_host()
+
+    argsort = np.argsort(h_in_keys, stable=True)
+    h_in_keys = np.array(h_in_keys)[argsort]
+    h_in_values = np.array(h_in_values)[argsort]
+
+    np.testing.assert_array_equal(h_out_keys, h_in_keys)
+    np.testing.assert_array_equal(h_out_items, h_in_values)
+
+
+def test_radix_sort_double_buffer(monkeypatch):
+    cc_major, _ = get_compute_capability()
+    # Skip sass verification for CC 9.0+ due to a bug in NVRTC.
+    # TODO: add NVRTC version check, ref nvbug 5243118
+    if cc_major >= 9:
+        import cuda.compute._cccl_interop as cccl_interop
+
+        monkeypatch.setattr(
+            cccl_interop,
+            "_check_sass",
+            False,
+        )
+
+    h_in_keys = np.array([-5, 0, 2, -3, 2, 4, 0, -1, 2, 8], dtype="int32")
+    h_in_values = np.array(
+        [-3.2, 2.2, 1.9, 4.0, -3.9, 2.7, 0, 8.3 - 1, 2.9, 5.4], dtype="float32"
+    )
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_values = DeviceArray.from_numpy(h_in_values)
+
+    d_out_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype)
+    d_out_values = DeviceArray.empty(h_in_values.shape, h_in_values.dtype)
+
+    keys_double_buffer = DoubleBuffer(d_in_keys, d_out_keys)
+    values_double_buffer = DoubleBuffer(d_in_values, d_out_values)
+
+    # Call single-phase API directly with num_items parameter
+    cuda.compute.radix_sort(
+        d_in_keys=keys_double_buffer,
+        d_out_keys=None,
+        d_in_values=values_double_buffer,
+        d_out_values=None,
+        num_items=h_in_keys.size,
+        order=SortOrder.ASCENDING,
+    )
+
+    # Check the result is correct
+    h_out_keys = keys_double_buffer.current().copy_to_host()
+    h_out_values = values_double_buffer.current().copy_to_host()
+
+    argsort = np.argsort(h_in_keys, stable=True)
+    h_in_keys = np.array(h_in_keys)[argsort]
+    h_in_values = np.array(h_in_values)[argsort]
+
+    np.testing.assert_array_equal(h_out_keys, h_in_keys)
+    np.testing.assert_array_equal(h_out_values, h_in_values)
+
+
+def _run(sorter, *, d_in_keys, d_out_keys, d_in_values, d_out_values, num_items):
+    bytes_needed = sorter(
+        temp_storage=None,
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_values,
+        d_out_values=d_out_values,
+        num_items=num_items,
+    )
+    tmp = TempStorageBuffer(bytes_needed, None)
+    sorter(
+        temp_storage=tmp,
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_values,
+        d_out_values=d_out_values,
+        num_items=num_items,
+    )
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_radix_sort_keys_values():
+    h_in_keys = np.array([-5, 0, 2, -3, 2, 4, 0, -1, 2, 8], dtype="int32")
+    h_in_values = np.array(
+        [-3.2, 2.2, 1.9, 4.0, -3.9, 2.7, 0, 8.3 - 1, 2.9, 5.4], dtype="float32"
+    )
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_values = DeviceArray.from_numpy(h_in_values)
+    d_out_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype)
+    d_out_values = DeviceArray.empty(h_in_values.shape, h_in_values.dtype)
+
+    builder = make_radix_sort(
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_values,
+        d_out_values=d_out_values,
+        order=SortOrder.ASCENDING,
+    )
+    blob = serialize(builder)
+    assert len(blob) > 0
+
+    loaded = deserialize(blob)
+    _run(
+        loaded,
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_values,
+        d_out_values=d_out_values,
+        num_items=h_in_keys.size,
+    )
+
+    argsort = np.argsort(h_in_keys, stable=True)
+    np.testing.assert_array_equal(d_out_keys.copy_to_host(), h_in_keys[argsort])
+    np.testing.assert_array_equal(d_out_values.copy_to_host(), h_in_values[argsort])
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_raw_op.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_raw_op.py
new file mode 100644
index 00000000..b8544658
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_raw_op.py
@@ -0,0 +1,525 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import re
+import struct
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray, get_compute_capability
+
+import cuda.compute
+from cuda.compute import types
+from cuda.compute._cpp_compile import _get_include_paths
+from cuda.compute.op import RawOp
+from cuda.core import Program, ProgramOptions
+
+# Mark all tests in this module as no_numba
+pytestmark = pytest.mark.no_numba
+
+
+def get_arch():
+    """Get the SM architecture string for the current device."""
+    cc_major, cc_minor = get_compute_capability()
+    return f"sm_{cc_major}{cc_minor}"
+
+
+def compile_to_ltoir(
+    source: str, arch: str, include_paths: list | None = None
+) -> bytes:
+    """Compile C++ source to LTOIR using cuda.core."""
+    opts = ProgramOptions(
+        arch=arch,
+        relocatable_device_code=True,
+        link_time_optimization=True,
+        include_path=include_paths,
+    )
+    prog = Program(source, "c++", options=opts)
+    return prog.compile("ltoir").code
+
+
+def extract_function_name(source: str) -> str:
+    """Extract function name from C++ source."""
+    match = re.search(r'extern\s+"C"\s+__device__\s+\w+\s+(\w+)\s*\(', source)
+    if match:
+        return match.group(1)
+    raise ValueError("Could not extract function name from C++ source.")
+
+
+def make_cpp_op(source: str, name: str = None, include_paths=None) -> RawOp:
+    """
+    Compile C++ source to LTO-IR and create a stateless RawOp. For v2 users
+    the LLVM-bitcode path is preferred (see ``examples/raw_op/llvm_stateless.py``);
+    this helper exercises the LTO-IR escape hatch.
+
+    Args:
+        source: C++ source code containing the operator function
+        name: Optional function name. If not provided, will be extracted from source
+
+    Returns:
+        RawOp instance ready to use with CCCL algorithms
+    """
+    if name is None:
+        name = extract_function_name(source)
+
+    arch = get_arch()
+    ltoir = compile_to_ltoir(source, arch, include_paths=include_paths)
+
+    return RawOp(ltoir=ltoir, name=name)
+
+
+def make_cpp_stateful_op(
+    source: str, state: bytes, name: str = None, state_alignment: int = 8
+) -> RawOp:
+    """
+    Compile C++ source to LTO-IR and create a stateful RawOp. See the note on
+    :func:`make_cpp_op` regarding the v2-preferred LLVM-bitcode path.
+
+    Args:
+        source: C++ source code containing the operator function
+        state: State data as bytes
+        name: Optional function name. If not provided, will be extracted from source
+        state_alignment: Memory alignment for state (default: 8)
+
+    Returns:
+        RawOp instance ready to use with CCCL algorithms
+    """
+    if name is None:
+        name = extract_function_name(source)
+
+    arch = get_arch()
+    ltoir = compile_to_ltoir(source, arch)
+
+    return RawOp(
+        ltoir=ltoir,
+        name=name,
+        state=state,
+        state_alignment=state_alignment,
+    )
+
+
+def test_cpp_op_basic_add():
+    """Test a basic C++ addition operator with reduce_into."""
+    cpp_source = """
+    extern "C" __device__ void add_op(void* a, void* b, void* result) {
+        *static_cast<int*>(result) = *static_cast<int*>(a) + *static_cast<int*>(b);
+    }
+    """
+
+    op = make_cpp_op(cpp_source, "add_op")
+
+    # Create test data
+    num_items = 100
+    h_input = np.arange(num_items, dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(1, np.int32)
+
+    # Use the custom op with reduce_into
+    h_init = np.array(0, dtype=np.int32)
+    cuda.compute.reduce_into(
+        d_in=d_input, d_out=d_output, num_items=num_items, op=op, h_init=h_init
+    )
+
+    # Verify result
+    result = d_output.copy_to_host()[0]
+    expected = np.sum(h_input)
+    assert result == expected, f"Expected {expected}, got {result}"
+
+
+def test_cpp_op_max():
+    """Test a C++ max operator with reduce_into."""
+    cpp_source = """
+    #include <cuda/std/algorithm>
+
+    extern "C" __device__ void max_op(void* a, void* b, void* result) {
+        float va = *static_cast<float*>(a);
+        float vb = *static_cast<float*>(b);
+        *static_cast<float*>(result) = cuda::std::max(va, vb);
+    }
+    """
+    # _get_include_paths() returns list of CCCL include paths;
+    # needed for libcudacxx headers like cuda/std/algorithm
+    op = make_cpp_op(cpp_source, include_paths=_get_include_paths())
+
+    # Create test data
+    num_items = 100
+    h_input = np.random.randn(num_items).astype(np.float32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(1, np.float32)
+
+    # Use the custom op with reduce_into
+    h_init = np.array(-np.inf, dtype=np.float32)
+    cuda.compute.reduce_into(
+        d_in=d_input, d_out=d_output, num_items=num_items, op=op, h_init=h_init
+    )
+
+    # Verify result
+    result = d_output.copy_to_host()[0]
+    expected = np.max(h_input)
+    assert np.isclose(result, expected), f"Expected {expected}, got {result}"
+
+
+def test_cpp_op_multiply():
+    """Test a C++ multiply operator."""
+    cpp_source = """
+    extern "C" __device__ void multiply(void* a, void* b, void* result) {
+        *static_cast<int*>(result) = *static_cast<int*>(a) * *static_cast<int*>(b);
+    }
+    """
+
+    op = make_cpp_op(cpp_source, "multiply")
+
+    # Create test data - use small numbers to avoid overflow
+    num_items = 5
+    h_input = np.array([1, 2, 3, 4, 5], dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(1, np.int32)
+
+    # Use the custom op with reduce_into
+    h_init = np.array(1, dtype=np.int32)
+    cuda.compute.reduce_into(
+        d_in=d_input, d_out=d_output, num_items=num_items, op=op, h_init=h_init
+    )
+
+    # Verify result
+    result = d_output.copy_to_host()[0]
+    expected = np.prod(h_input)
+    assert result == expected, f"Expected {expected}, got {result}"
+
+
+def test_cpp_op_complex_logic():
+    """Test a C++ operator with more complex logic - bitwise OR (associative)."""
+    cpp_source = """
+    extern "C" __device__ void bitwise_or(void* a, void* b, void* result) {
+        // Bitwise OR is associative: (a | b) | c = a | (b | c)
+        int va = *static_cast<int*>(a);
+        int vb = *static_cast<int*>(b);
+        *static_cast<int*>(result) = va | vb;
+    }
+    """
+
+    op = make_cpp_op(cpp_source, "bitwise_or")
+
+    # Create test data with specific bit patterns
+    num_items = 5
+    h_input = np.array([1, 2, 4, 8, 16], dtype=np.int32)  # Powers of 2
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(1, np.int32)
+
+    # Use the custom op with reduce_into
+    h_init = np.array(0, dtype=np.int32)
+    cuda.compute.reduce_into(
+        d_in=d_input, d_out=d_output, num_items=num_items, op=op, h_init=h_init
+    )
+
+    # Expected: 1 | 2 | 4 | 8 | 16 = 31 (all bits set)
+    result = d_output.copy_to_host()[0]
+    expected = 31
+    assert result == expected, f"Expected {expected}, got {result}"
+
+
+def test_cpp_op_different_types():
+    """Test C++ operator with different numeric types."""
+    cpp_source = """
+    extern "C" __device__ void add_doubles(void* a, void* b, void* result) {
+        *static_cast<double*>(result) = *static_cast<double*>(a) + *static_cast<double*>(b);
+    }
+    """
+
+    op = make_cpp_op(cpp_source, "add_doubles")
+
+    # Create test data
+    num_items = 50
+    h_input = np.random.randn(num_items).astype(np.float64)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(1, np.float64)
+
+    # Use the custom op with reduce_into
+    h_init = np.array(0.0, dtype=np.float64)
+    cuda.compute.reduce_into(
+        d_in=d_input, d_out=d_output, num_items=num_items, op=op, h_init=h_init
+    )
+
+    # Verify result
+    result = d_output.copy_to_host()[0]
+    expected = np.sum(h_input)
+    assert np.isclose(result, expected), f"Expected {expected}, got {result}"
+
+
+def test_cpp_op_name_extraction():
+    """Test that function name is correctly extracted from C++ source."""
+    cpp_source = """
+    extern "C" __device__ void my_function_name(void* a, void* b, void* result) {
+        *static_cast<int*>(result) = *static_cast<int*>(a) + *static_cast<int*>(b);
+    }
+    """
+
+    # Don't provide name - it should be extracted
+    op = make_cpp_op(cpp_source)
+
+    # Create test data
+    num_items = 10
+    h_input = np.arange(num_items, dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(1, np.int32)
+
+    # Use the custom op with reduce_into
+    h_init = np.array(0, dtype=np.int32)
+    cuda.compute.reduce_into(
+        d_in=d_input, d_out=d_output, num_items=num_items, op=op, h_init=h_init
+    )
+
+    # Verify result
+    result = d_output.copy_to_host()[0]
+    expected = np.sum(h_input)
+    assert result == expected, f"Expected {expected}, got {result}"
+
+
+def test_cpp_op_min():
+    """Test a C++ min operator."""
+    cpp_source = """
+    extern "C" __device__ void min_op(void* a, void* b, void* result) {
+        int va = *static_cast<int*>(a);
+        int vb = *static_cast<int*>(b);
+        *static_cast<int*>(result) = va < vb ? va : vb;
+    }
+    """
+
+    op = make_cpp_op(cpp_source, "min_op")
+
+    # Create test data
+    num_items = 100
+    h_input = np.random.randint(-1000, 1000, num_items, dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(1, np.int32)
+
+    # Use the custom op with reduce_into
+    h_init = np.array(np.iinfo(np.int32).max, dtype=np.int32)
+    cuda.compute.reduce_into(
+        d_in=d_input, d_out=d_output, num_items=num_items, op=op, h_init=h_init
+    )
+
+    # Verify result
+    result = d_output.copy_to_host()[0]
+    expected = np.min(h_input)
+    assert result == expected, f"Expected {expected}, got {result}"
+
+
+def test_cpp_op_with_struct():
+    """Test a C++ operator that works with struct types."""
+    from cuda.compute import gpu_struct
+
+    # Define a simple 2D point struct
+    Point = gpu_struct({"x": np.int32, "y": np.int32})
+
+    # C++ operator to add two points (field by field)
+    cpp_source = """
+    struct Point {
+        int x;
+        int y;
+    };
+
+    extern "C" __device__ void add_points(void* a, void* b, void* result) {
+        Point* pa = static_cast<Point*>(a);
+        Point* pb = static_cast<Point*>(b);
+        Point* pr = static_cast<Point*>(result);
+        pr->x = pa->x + pb->x;
+        pr->y = pa->y + pb->y;
+    }
+    """
+
+    op = make_cpp_op(cpp_source, "add_points")
+
+    # Create test data
+    num_items = 10
+    h_data = np.zeros(num_items, dtype=Point.dtype)
+    for i in range(num_items):
+        h_data[i]["x"] = i
+        h_data[i]["y"] = i * 2
+
+    d_input = DeviceArray.from_numpy(h_data)
+    d_output = DeviceArray.empty(1, Point.dtype)
+
+    # Initial point (0, 0)
+    h_init = Point(0, 0)
+
+    # Use the custom op with reduce_into
+    cuda.compute.reduce_into(
+        d_in=d_input, d_out=d_output, num_items=num_items, op=op, h_init=h_init
+    )
+
+    # Verify result
+    result = d_output.copy_to_host()[0]
+    expected_x = sum(range(num_items))  # 0+1+2+...+9 = 45
+    expected_y = sum(i * 2 for i in range(num_items))  # 0+2+4+...+18 = 90
+
+    assert result["x"] == expected_x, f"Expected x={expected_x}, got {result['x']}"
+    assert result["y"] == expected_y, f"Expected y={expected_y}, got {result['y']}"
+
+
+def test_cpp_op_with_transform_iterator():
+    """Test that RawOp works with TransformIterator."""
+    from cuda.compute import OpKind, TransformIterator
+
+    # C++ unary operator that doubles a value
+    cpp_source = """
+    extern "C" __device__ void double_op(void* input, void* result) {
+        *static_cast<int*>(result) = *static_cast<int*>(input) * 2;
+    }
+    """
+
+    op = make_cpp_op(cpp_source, "double_op")
+
+    # Create input data
+    num_items = 10
+    h_input = np.arange(num_items, dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+
+    # Create transform iterator with RawOp
+    transform_iter = TransformIterator(d_input, op, value_type=types.int32)
+
+    # Use the transform iterator with reduce
+    d_output = DeviceArray.empty(1, np.int32)
+    h_init = np.array(0, dtype=np.int32)
+
+    # Sum the doubled values using built-in PLUS operator
+    cuda.compute.reduce_into(
+        d_in=transform_iter,
+        d_out=d_output,
+        num_items=num_items,
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+
+    # Verify result: sum of (0*2, 1*2, 2*2, ..., 9*2) = 2 * sum(0..9) = 2 * 45 = 90
+    result = d_output.copy_to_host()[0]
+    expected = 2 * np.sum(h_input)
+    assert result == expected, f"Expected {expected}, got {result}"
+
+
+def test_cpp_stateful_op_reduce_with_constant():
+    """Test stateful RawOp with a simple stateful reduce."""
+    # State: a single int32 constant value (10) on device
+    d_constant = DeviceArray.from_numpy(np.array([10], dtype=np.int32))
+    constant_ptr = d_constant.__cuda_array_interface__["data"][0]
+    state_data = struct.pack("P", constant_ptr)
+    state_alignment = np.dtype(np.intp).alignment
+
+    # C++ operator that adds inputs plus reads constant from state
+    cpp_source = """
+    extern "C" __device__ void add_with_state_constant(void* state, void* a, void* b, void* result) {
+        // Extract constant pointer from state
+        int* constant_ptr = *reinterpret_cast<int**>(state);
+        int constant = *constant_ptr;
+
+        int va = *static_cast<int*>(a);
+        int vb = *static_cast<int*>(b);
+        *static_cast<int*>(result) = va + vb + constant;
+    }
+    """
+
+    op = make_cpp_stateful_op(
+        cpp_source, state_data, "add_with_state_constant", state_alignment
+    )
+
+    # Create test data
+    num_items = 5
+    h_input = np.array([1, 2, 3, 4, 5], dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(1, np.int32)
+
+    # Use the stateful op with reduce_into
+    h_init = np.array(0, dtype=np.int32)
+    cuda.compute.reduce_into(
+        d_in=d_input, d_out=d_output, num_items=num_items, op=op, h_init=h_init
+    )
+
+    # Get result
+    result = d_output.copy_to_host()[0]
+    # Each reduction adds 10, so we expect input sum + some multiple of 10
+    # The exact value depends on tree structure, but should be > sum(inputs)
+    sum_inputs = np.sum(h_input)
+    assert result > sum_inputs, f"Expected result > {sum_inputs}, got {result}"
+
+
+def test_cpp_stateful_op_select_with_counter():
+    """Test stateful RawOp with select_if that atomically updates a counter."""
+    # Create a device counter initialized to 0
+    d_counter = DeviceArray.from_numpy(np.zeros(1, dtype=np.int32))
+
+    # State: pointer to the counter
+    counter_ptr = d_counter.__cuda_array_interface__["data"][0]
+    state_data = struct.pack("P", counter_ptr)  # Pack pointer as bytes
+
+    # Use proper pointer alignment for the platform
+    state_alignment = np.dtype(np.intp).alignment
+
+    # C++ select operator that counts selected items
+    # Selects even numbers and atomically increments counter for each selection
+    cpp_source = """
+    extern "C" __device__ void select_even_with_count(void* state, void* input, void* result) {
+        // Extract counter pointer from state
+        int* counter = *reinterpret_cast<int**>(state);
+
+        // Get input value
+        int value = *static_cast<int*>(input);
+
+        // Check if even
+        bool is_even = (value % 2 == 0);
+
+        // If selected, atomically increment the counter
+        if (is_even) {
+            atomicAdd(counter, 1);
+        }
+
+        // Store result as bool (uint8)
+        *static_cast<unsigned char*>(result) = is_even ? 1 : 0;
+    }
+    """
+
+    op = make_cpp_stateful_op(
+        cpp_source,
+        state_data,
+        "select_even_with_count",
+        state_alignment,
+    )
+
+    # Create test data: 0 to 19
+    num_items = 20
+    h_input = np.arange(num_items, dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+
+    # Allocate output arrays
+    d_output = DeviceArray.empty(num_items, np.int32)
+    d_num_selected = DeviceArray.empty(1, np.int32)
+
+    # Run select
+    cuda.compute.select(
+        d_in=d_input,
+        d_out=d_output,
+        d_num_selected_out=d_num_selected,
+        cond=op,
+        num_items=num_items,
+    )
+
+    # Get results
+    num_selected = d_num_selected.copy_to_host()[0]
+    counter_value = d_counter.copy_to_host()[0]
+
+    # Verify: should have selected 10 even numbers (0, 2, 4, ..., 18)
+    expected_count = 10
+    assert num_selected == expected_count, (
+        f"Expected {expected_count} selected, got {num_selected}"
+    )
+    assert counter_value == expected_count, (
+        f"Expected counter={expected_count}, got {counter_value}"
+    )
+
+    # Verify the selected values are correct
+    selected_values = d_output.copy_to_host()[:num_selected]
+    expected_selected = np.arange(0, 20, 2, dtype=np.int32)
+    assert np.array_equal(selected_values, expected_selected), (
+        "Selected values don't match"
+    )
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_reduce.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_reduce.py
new file mode 100644
index 00000000..8eb63c2c
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_reduce.py
@@ -0,0 +1,1169 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import functools
+import random
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray
+
+import cuda.compute
+from cuda.compute import (
+    CacheModifiedInputIterator,
+    ConstantIterator,
+    CountingIterator,
+    Determinism,
+    OpKind,
+    TransformIterator,
+    TransformOutputIterator,
+    deserialize,
+    gpu_struct,
+    make_reduce_into,
+    serialize,
+)
+from cuda.compute._utils.temp_storage_buffer import TempStorageBuffer
+
+
+def random_int(shape, dtype):
+    return np.random.randint(0, 5, size=shape).astype(dtype)
+
+
+def type_to_problem_sizes(dtype):
+    if dtype in [np.uint8, np.int8]:
+        return [2, 4, 5, 6]
+    elif dtype in [np.float16]:
+        return [4, 8, 10]
+    elif dtype in [np.uint16, np.int16]:
+        return [4, 8, 12, 14]
+    elif dtype in [np.uint32, np.int32]:
+        return [16, 20, 24, 26]
+    elif dtype in [np.uint64, np.int64]:
+        return [16, 20, 24, 25]
+    else:
+        raise ValueError("Unsupported dtype")
+
+
+def get_mark(dt, log_size):
+    if log_size + np.log2(np.dtype(dt).itemsize) < 21:
+        return tuple()
+    return pytest.mark.large
+
+
+def add_op(a, b):
+    return a + b
+
+
+# Lambda function for testing lambda support as reducers
+add_op_lambda = lambda a, b: a + b  # noqa: E731
+
+
+reduce_params = [
+    pytest.param(
+        dt,
+        2**log_size,
+        OpKind.PLUS if dt == np.float16 else add_op,
+        marks=get_mark(dt, log_size),
+    )
+    for dt in [np.uint8, np.uint16, np.uint32, np.uint64, np.float16]
+    for log_size in type_to_problem_sizes(dt)
+]
+
+
+@pytest.mark.parametrize("dtype,num_items,op", reduce_params)
+def test_device_reduce(dtype, num_items, op):
+    init_value = 42
+    h_init = np.array([init_value], dtype=dtype)
+    d_output = DeviceArray.empty(1, dtype)
+
+    h_input = random_int(num_items, dtype)
+    d_input = DeviceArray.from_numpy(h_input)
+
+    cuda.compute.reduce_into(
+        d_in=d_input, d_out=d_output, num_items=h_input.size, op=op, h_init=h_init
+    )
+    h_output = d_output.copy_to_host()
+    assert h_output[0] == pytest.approx(
+        sum(h_input) + init_value, rel=0.08 if dtype == np.float16 else 0
+    )  # obtained relative error value from c2h/include/c2h/check_results.cuh
+
+
+def test_device_reduce_with_lambda():
+    """Test that lambda functions can be used as reducers."""
+    dtype = np.int32
+    init_value = 42
+    num_items = 1024
+
+    h_init = np.array([init_value], dtype=dtype)
+    d_output = DeviceArray.empty(1, dtype)
+
+    h_input = random_int(num_items, dtype)
+    d_input = DeviceArray.from_numpy(h_input)
+
+    # Use a lambda function directly as the reducer
+    cuda.compute.reduce_into(
+        d_in=d_input,
+        d_out=d_output,
+        num_items=h_input.size,
+        op=lambda a, b: a + b,
+        h_init=h_init,
+    )
+    h_output = d_output.copy_to_host()
+    assert h_output[0] == sum(h_input) + init_value
+
+
+def test_device_reduce_with_lambda_variable():
+    """Test that lambda functions assigned to variables can be used as reducers."""
+    dtype = np.int32
+    init_value = 42
+    num_items = 1024
+
+    h_init = np.array([init_value], dtype=dtype)
+    d_output = DeviceArray.empty(1, dtype)
+
+    h_input = random_int(num_items, dtype)
+    d_input = DeviceArray.from_numpy(h_input)
+
+    # Use a lambda function assigned to a variable as the reducer
+    cuda.compute.reduce_into(
+        d_in=d_input,
+        d_out=d_output,
+        num_items=h_input.size,
+        op=add_op_lambda,
+        h_init=h_init,
+    )
+    h_output = d_output.copy_to_host()
+    assert h_output[0] == sum(h_input) + init_value
+
+
+def test_complex_device_reduce():
+    h_init = np.array([40.0 + 2.0j], dtype=complex)
+    d_output = DeviceArray.empty(1, complex)
+
+    for num_items in [42, 420000]:
+        real_imag = np.random.random((2, num_items))
+        h_input = real_imag[0] + 1j * real_imag[1]
+        d_input = DeviceArray.from_numpy(h_input)
+        assert h_input.size == num_items
+        cuda.compute.reduce_into(
+            d_in=d_input, d_out=d_output, num_items=num_items, op=add_op, h_init=h_init
+        )
+
+        result = d_output.copy_to_host()[0]
+        expected = np.sum(h_input, initial=h_init[0])
+        assert result == pytest.approx(expected)
+
+
+def _test_device_sum_with_iterator(
+    l_varr, start_sum_with, i_input, dtype_inp, dtype_out, use_numpy_array
+):
+    expected_result = start_sum_with
+    for v in l_varr:
+        expected_result = add_op(expected_result, v)
+
+    if use_numpy_array:
+        h_input = np.array(l_varr, dtype_inp)
+        d_input = DeviceArray.from_numpy(h_input)
+    else:
+        d_input = i_input
+
+    d_output = DeviceArray.empty(1, dtype_out)  # to store device sum
+
+    h_init = np.array([start_sum_with], dtype_out)
+
+    cuda.compute.reduce_into(
+        d_in=d_input, d_out=d_output, num_items=len(l_varr), op=add_op, h_init=h_init
+    )
+
+    h_output = d_output.copy_to_host()
+    assert h_output[0] == expected_result
+
+
+def mul2(val):
+    return 2 * val
+
+
+def mul3(val):
+    return 3 * val
+
+
+SUPPORTED_VALUE_TYPE_NAMES = (
+    "int16",
+    "uint16",
+    "int32",
+    "uint32",
+    "int64",
+    "uint64",
+    # "float16", # TODO: this doesn't work with iterators right now due to a numba ctypes error
+    "float32",
+    "float64",
+)
+
+
+@pytest.fixture(params=SUPPORTED_VALUE_TYPE_NAMES)
+def supported_value_type(request):
+    return request.param
+
+
+@pytest.fixture(params=[True, False])
+def use_numpy_array(request):
+    return request.param
+
+
+def test_device_sum_cache_modified_input_it(
+    use_numpy_array, supported_value_type, num_items=3, start_sum_with=10
+):
+    rng = random.Random(0)
+    l_varr = [rng.randrange(100) for _ in range(num_items)]
+    dtype_inp = np.dtype(supported_value_type)
+    dtype_out = dtype_inp
+    input_devarr = DeviceArray.from_numpy(np.array(l_varr, dtype=dtype_inp))
+    i_input = CacheModifiedInputIterator(input_devarr, modifier="stream")
+    _test_device_sum_with_iterator(
+        l_varr, start_sum_with, i_input, dtype_inp, dtype_out, use_numpy_array
+    )
+
+
+def test_device_sum_constant_it(
+    use_numpy_array, supported_value_type, num_items=3, start_sum_with=10
+):
+    l_varr = [42 for distance in range(num_items)]
+    dtype_inp = np.dtype(supported_value_type)
+    dtype_out = dtype_inp
+    i_input = ConstantIterator(dtype_inp.type(42))
+    _test_device_sum_with_iterator(
+        l_varr, start_sum_with, i_input, dtype_inp, dtype_out, use_numpy_array
+    )
+
+
+def test_device_sum_counting_it(
+    use_numpy_array, supported_value_type, num_items=3, start_sum_with=10
+):
+    l_varr = [start_sum_with + distance for distance in range(num_items)]
+    dtype_inp = np.dtype(supported_value_type)
+    dtype_out = dtype_inp
+    i_input = CountingIterator(dtype_inp.type(start_sum_with))
+    _test_device_sum_with_iterator(
+        l_varr, start_sum_with, i_input, dtype_inp, dtype_out, use_numpy_array
+    )
+
+
+@pytest.mark.parametrize(
+    "value_type_name_pair",
+    list(zip(SUPPORTED_VALUE_TYPE_NAMES, SUPPORTED_VALUE_TYPE_NAMES))
+    + [
+        # ("float16", "int16"), # TODO: this doesn't work with numba right now due to an unresolved extern operator definition
+        ("float32", "int16"),
+        ("float32", "int32"),
+        ("float64", "int32"),
+        ("float64", "int64"),
+        ("int64", "float32"),
+    ],
+)
+def test_device_sum_map_mul2_count_it(
+    use_numpy_array, value_type_name_pair, num_items=3, start_sum_with=10
+):
+    l_varr = [2 * (start_sum_with + distance) for distance in range(num_items)]
+    vtn_out, vtn_inp = value_type_name_pair
+    dtype_inp = np.dtype(vtn_inp)
+    dtype_out = np.dtype(vtn_out)
+    i_input = TransformIterator(CountingIterator(dtype_inp.type(start_sum_with)), mul2)
+    _test_device_sum_with_iterator(
+        l_varr, start_sum_with, i_input, dtype_inp, dtype_out, use_numpy_array
+    )
+
+
+@pytest.mark.parametrize(
+    ("fac_out", "fac_mid", "vtn_out", "vtn_mid", "vtn_inp"),
+    [
+        (3, 2, "int32", "int32", "int32"),
+        (2, 2, "float64", "float32", "int16"),
+    ],
+)
+def test_device_sum_map_mul_map_mul_count_it(
+    use_numpy_array,
+    fac_out,
+    fac_mid,
+    vtn_out,
+    vtn_mid,
+    vtn_inp,
+    num_items=3,
+    start_sum_with=10,
+):
+    l_varr = [
+        fac_out * (fac_mid * (start_sum_with + distance))
+        for distance in range(num_items)
+    ]
+    dtype_inp = np.dtype(vtn_inp)
+    dtype_out = np.dtype(vtn_out)
+    mul_funcs = {2: mul2, 3: mul3}
+    i_input = TransformIterator(
+        TransformIterator(
+            CountingIterator(dtype_inp.type(start_sum_with)),
+            mul_funcs[fac_mid],
+        ),
+        mul_funcs[fac_out],
+    )
+    _test_device_sum_with_iterator(
+        l_varr, start_sum_with, i_input, dtype_inp, dtype_out, use_numpy_array
+    )
+
+
+@pytest.mark.parametrize(
+    "value_type_name_pair",
+    [
+        ("int32", "int32"),
+        ("int64", "int32"),
+        ("int32", "int64"),
+    ],
+)
+def test_device_sum_map_mul2_device_array_it(
+    use_numpy_array, value_type_name_pair, num_items=3, start_sum_with=10
+):
+    vtn_out, vtn_inp = value_type_name_pair
+    dtype_inp = np.dtype(vtn_inp)
+    dtype_out = np.dtype(vtn_out)
+    rng = random.Random(0)
+    l_d_in = [rng.randrange(100) for _ in range(num_items)]
+    a_d_in = DeviceArray.from_numpy(np.asarray(l_d_in, dtype=dtype_inp))
+    i_input = TransformIterator(a_d_in, mul2)
+    l_varr = [mul2(v) for v in l_d_in]
+    _test_device_sum_with_iterator(
+        l_varr, start_sum_with, i_input, dtype_inp, dtype_out, use_numpy_array
+    )
+
+
+def test_reducer_caching():
+    def sum_op(x, y):
+        return x + y
+
+    # inputs are device arrays
+    reducer_1 = cuda.compute.make_reduce_into(
+        d_in=DeviceArray.empty(3, dtype="int64"),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    reducer_2 = cuda.compute.make_reduce_into(
+        d_in=DeviceArray.empty(3, dtype="int64"),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    assert reducer_1 is reducer_2
+
+    # inputs are device arrays of different dtype:
+    reducer_1 = cuda.compute.make_reduce_into(
+        d_in=DeviceArray.empty(3, dtype="int64"),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    reducer_2 = cuda.compute.make_reduce_into(
+        d_in=DeviceArray.empty(3, dtype="int32"),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    assert reducer_1 is not reducer_2
+
+    # outputs are of different dtype:
+    reducer_1 = cuda.compute.make_reduce_into(
+        d_in=DeviceArray.empty(3, dtype="int64"),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    reducer_2 = cuda.compute.make_reduce_into(
+        d_in=DeviceArray.empty(3, dtype="int64"),
+        d_out=DeviceArray.empty(1, dtype="int32"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    assert reducer_1 is not reducer_2
+
+    # inputs are of same dtype but different size
+    # (should still use cached reducer):
+    reducer_1 = cuda.compute.make_reduce_into(
+        d_in=DeviceArray.empty(3, dtype="int64"),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    reducer_2 = cuda.compute.make_reduce_into(
+        d_in=DeviceArray.empty(5, dtype="int64"),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    assert reducer_1 is reducer_2
+
+    # inputs are counting iterators of the
+    # same value type:
+    reducer_1 = cuda.compute.make_reduce_into(
+        d_in=CountingIterator(np.int32(0)),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    reducer_2 = cuda.compute.make_reduce_into(
+        d_in=CountingIterator(np.int32(0)),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    assert reducer_1 is reducer_2
+
+    # inputs are counting iterators of different value type:
+    reducer_1 = cuda.compute.make_reduce_into(
+        d_in=CountingIterator(np.int32(0)),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    reducer_2 = cuda.compute.make_reduce_into(
+        d_in=CountingIterator(np.int64(0)),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    assert reducer_1 is not reducer_2
+
+    def op1(x):
+        return x
+
+    def op2(x):
+        return 2 * x
+
+    def op3(x):
+        return x
+
+    # inputs are TransformIterators
+    reducer_1 = cuda.compute.make_reduce_into(
+        d_in=TransformIterator(CountingIterator(np.int32(0)), op1),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    reducer_2 = cuda.compute.make_reduce_into(
+        d_in=TransformIterator(CountingIterator(np.int32(0)), op1),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    assert reducer_1 is reducer_2
+
+    # inputs are TransformIterators with different
+    # op:
+    reducer_1 = cuda.compute.make_reduce_into(
+        d_in=TransformIterator(CountingIterator(np.int32(0)), op1),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    reducer_2 = cuda.compute.make_reduce_into(
+        d_in=TransformIterator(CountingIterator(np.int32(0)), op2),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    assert reducer_1 is not reducer_2
+
+    # inputs are TransformIterators with same op
+    # but different name:
+    reducer_1 = cuda.compute.make_reduce_into(
+        d_in=TransformIterator(CountingIterator(np.int32(0)), op1),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    reducer_2 = cuda.compute.make_reduce_into(
+        d_in=TransformIterator(CountingIterator(np.int32(0)), op3),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+
+    # inputs are CountingIterators of same kind
+    # but different state:
+    reducer_1 = cuda.compute.make_reduce_into(
+        d_in=CountingIterator(np.int32(0)),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    reducer_2 = cuda.compute.make_reduce_into(
+        d_in=CountingIterator(np.int32(1)),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+
+    assert reducer_1 is reducer_2
+
+    # inputs are TransformIterators of same kind
+    # but different state:
+    ary1 = DeviceArray.from_numpy(np.asarray([0, 1, 2], dtype="int64"))
+    ary2 = DeviceArray.from_numpy(np.asarray([0, 1], dtype="int64"))
+    reducer_1 = cuda.compute.make_reduce_into(
+        d_in=TransformIterator(ary1, op1),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    reducer_2 = cuda.compute.make_reduce_into(
+        d_in=TransformIterator(ary2, op1),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    assert reducer_1 is reducer_2
+
+    # inputs are TransformIterators of same kind
+    # but different state:
+    reducer_1 = cuda.compute.make_reduce_into(
+        d_in=TransformIterator(CountingIterator(np.int32(0)), op1),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    reducer_2 = cuda.compute.make_reduce_into(
+        d_in=TransformIterator(CountingIterator(np.int32(1)), op1),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    assert reducer_1 is reducer_2
+
+    # inputs are TransformIterators with different kind:
+    reducer_1 = cuda.compute.make_reduce_into(
+        d_in=TransformIterator(CountingIterator(np.int32(0)), op1),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    reducer_2 = cuda.compute.make_reduce_into(
+        d_in=TransformIterator(CountingIterator(np.int64(0)), op1),
+        d_out=DeviceArray.empty(1, dtype="int64"),
+        op=sum_op,
+        h_init=np.zeros(1, dtype="int64"),
+    )
+    assert reducer_1 is not reducer_2
+
+
+@pytest.fixture(params=[True, False])
+def array_2d(request):
+    f_contiguous = request.param
+    array = np.random.rand(5, 10)
+    return np.asfortranarray(array) if f_contiguous else array
+
+
+def test_reduce_2d_array(array_2d):
+    def binary_op(x, y):
+        return x + y
+
+    d_in = DeviceArray.from_numpy(array_2d)
+    d_out = DeviceArray.empty(1, dtype=array_2d.dtype)
+    h_init = np.asarray([0], dtype=array_2d.dtype)
+    cuda.compute.reduce_into(
+        d_in=d_in,
+        d_out=d_out,
+        num_items=array_2d.size,
+        op=binary_op,
+        h_init=h_init,
+    )
+    np.testing.assert_allclose(array_2d.sum(), d_out.copy_to_host())
+
+
+def test_reduce_non_contiguous():
+    def binary_op(x, y):
+        return x + y
+
+    size = 10
+
+    class DeviceArrayView:
+        def __init__(self, base, host_view):
+            self._base = base
+            self.__cuda_array_interface__ = {
+                **base.__cuda_array_interface__,
+                "shape": host_view.shape,
+                "strides": host_view.strides,
+            }
+
+    d_out = DeviceArray.empty(1, dtype="int64")
+    h_init = np.asarray([0], dtype="int64")
+
+    h_base = np.zeros((size, 2))
+    d_in = DeviceArrayView(DeviceArray.from_numpy(h_base), h_base[:, 0])
+    with pytest.raises(ValueError, match="Non-contiguous arrays are not supported."):
+        _ = cuda.compute.make_reduce_into(
+            d_in=d_in, d_out=d_out, op=binary_op, h_init=h_init
+        )
+
+    h_base = np.zeros(size)
+    d_in = DeviceArrayView(DeviceArray.from_numpy(h_base), h_base[::2])
+    with pytest.raises(ValueError, match="Non-contiguous arrays are not supported."):
+        _ = cuda.compute.make_reduce_into(
+            d_in=d_in, d_out=d_out, op=binary_op, h_init=h_init
+        )
+
+
+def test_reduce_with_stream(cuda_stream):
+    def add_op(x, y):
+        return x + y
+
+    h_init = np.asarray([0], dtype=np.int32)
+    h_in = random_int(5, np.int32)
+
+    d_in = DeviceArray.from_numpy(h_in, stream=cuda_stream)
+    d_out = DeviceArray.empty(1, np.int32, stream=cuda_stream)
+
+    cuda.compute.reduce_into(
+        d_in=d_in,
+        d_out=d_out,
+        num_items=h_in.size,
+        op=add_op,
+        h_init=h_init,
+        stream=cuda_stream,
+    )
+    np.testing.assert_allclose(h_in.sum(), d_out.copy_to_host(stream=cuda_stream))
+
+
+def test_reduce_invalid_stream():
+    # Invalid stream that doesn't implement __cuda_stream__
+    class Stream1:
+        def __init__(self):
+            pass
+
+    # Invalid stream that implements __cuda_stream__ but returns the wrong type
+    class Stream2:
+        def __init__(self):
+            pass
+
+        def __cuda_stream__(self):
+            return None
+
+    # Invalid stream that returns an invalid handle
+    class Stream3:
+        def __init__(self):
+            pass
+
+        def __cuda_stream__(self):
+            return (0, None)
+
+    def add_op(x, y):
+        return x + y
+
+    d_out = DeviceArray.empty(1, np.float64)
+    h_init = np.empty(1)
+    d_in = DeviceArray.empty(1, np.float64)
+    reduce_into = cuda.compute.make_reduce_into(
+        d_in=d_in, d_out=d_out, op=add_op, h_init=h_init
+    )
+
+    with pytest.raises(
+        TypeError, match="does not implement the '__cuda_stream__' protocol"
+    ):
+        _ = reduce_into(
+            temp_storage=None,
+            d_in=d_in,
+            d_out=d_out,
+            op=add_op,
+            num_items=1,
+            h_init=h_init,
+            stream=Stream1(),
+        )
+
+    with pytest.raises(
+        TypeError, match="could not obtain __cuda_stream__ protocol version and handle"
+    ):
+        _ = reduce_into(
+            temp_storage=None,
+            d_in=d_in,
+            d_out=d_out,
+            op=add_op,
+            num_items=1,
+            h_init=h_init,
+            stream=Stream2(),
+        )
+
+    with pytest.raises(TypeError, match="invalid stream handle"):
+        _ = reduce_into(
+            temp_storage=None,
+            d_in=d_in,
+            d_out=d_out,
+            op=add_op,
+            num_items=1,
+            h_init=h_init,
+            stream=Stream3(),
+        )
+
+
+def test_device_reduce_well_known_plus():
+    dtype = np.int32
+    h_init = np.array([0], dtype=dtype)
+    h_input = np.array([1, 2, 3, 4, 5], dtype=dtype)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(1, dtype=dtype)
+
+    cuda.compute.reduce_into(
+        d_in=d_input,
+        d_out=d_output,
+        num_items=len(h_input),
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+
+    expected_output = 15
+    assert d_output.copy_to_host()[0] == expected_output
+
+
+def test_device_reduce_well_known_minimum():
+    dtype = np.int32
+    h_init = np.array([100], dtype=dtype)
+    h_input = np.array([8, 6, 7, 5, 3, 0, 9], dtype=dtype)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(1, dtype=dtype)
+
+    cuda.compute.reduce_into(
+        d_in=d_input,
+        d_out=d_output,
+        num_items=len(h_input),
+        op=OpKind.MINIMUM,
+        h_init=h_init,
+    )
+
+    expected_output = 0
+    assert d_output.copy_to_host()[0] == expected_output
+
+
+def test_device_reduce_well_known_maximum():
+    dtype = np.int32
+    h_init = np.array([-100], dtype=dtype)
+    h_input = np.array([8, 6, 7, 5, 3, 0, 9], dtype=dtype)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(1, dtype=dtype)
+
+    cuda.compute.reduce_into(
+        d_in=d_input,
+        d_out=d_output,
+        num_items=len(h_input),
+        op=OpKind.MAXIMUM,
+        h_init=h_init,
+    )
+
+    expected_output = 9
+    assert d_output.copy_to_host()[0] == expected_output
+
+
+def test_cache_modified_input_iterator():
+    def add_op(a, b):
+        return a + b
+
+    values = [8, 6, 7, 5, 3, 0, 9]
+    d_input = DeviceArray.from_numpy(np.asarray(values, dtype=np.int32))
+
+    iterator = CacheModifiedInputIterator(d_input, modifier="stream")
+    h_init = np.array([0], dtype=np.int32)
+    d_output = DeviceArray.empty(1, dtype=np.int32)
+
+    cuda.compute.reduce_into(
+        d_in=iterator, d_out=d_output, num_items=len(values), op=add_op, h_init=h_init
+    )
+
+    expected_output = functools.reduce(lambda a, b: a + b, values)
+    assert d_output.copy_to_host()[0] == expected_output
+
+
+def test_constant_iterator():
+    def add_op(a, b):
+        return a + b
+
+    value = 10
+    num_items = 3
+
+    constant_it = ConstantIterator(np.int32(value))
+    h_init = np.array([0], dtype=np.int32)
+    d_output = DeviceArray.empty(1, dtype=np.int32)
+
+    cuda.compute.reduce_into(
+        d_in=constant_it, d_out=d_output, num_items=num_items, op=add_op, h_init=h_init
+    )
+
+    expected_output = functools.reduce(lambda a, b: a + b, [value] * num_items)
+    assert d_output.copy_to_host()[0] == expected_output
+
+
+def test_counting_iterator():
+    def add_op(a, b):
+        return a + b
+
+    first_item = 10
+    num_items = 3
+
+    first_it = CountingIterator(np.int32(first_item))  # Input sequence
+    h_init = np.array([0], dtype=np.int32)  # Initial value for the reduction
+    d_output = DeviceArray.empty(1, dtype=np.int32)  # Storage for output
+
+    cuda.compute.reduce_into(
+        d_in=first_it, d_out=d_output, num_items=num_items, op=add_op, h_init=h_init
+    )
+
+    expected_output = functools.reduce(
+        lambda a, b: a + b, range(first_item, first_item + num_items)
+    )
+    assert d_output.copy_to_host()[0] == expected_output
+
+
+def test_transform_iterator():
+    def add_op(a, b):
+        return a + b
+
+    def square_op(a):
+        return a**2
+
+    first_item = 10
+    num_items = 3
+
+    transform_it = TransformIterator(CountingIterator(np.int32(first_item)), square_op)
+    h_init = np.array([0], dtype=np.int32)
+    d_output = DeviceArray.empty(1, dtype=np.int32)
+
+    cuda.compute.reduce_into(
+        d_in=transform_it, d_out=d_output, num_items=num_items, op=add_op, h_init=h_init
+    )
+
+    expected_output = functools.reduce(
+        lambda a, b: a + b, [a**2 for a in range(first_item, first_item + num_items)]
+    )
+    assert d_output.copy_to_host()[0] == expected_output
+
+
+def test_reduce_struct_type():
+    @gpu_struct
+    class Pixel:
+        r: np.int32
+        g: np.int32
+        b: np.int32
+
+    def max_g_value(x, y):
+        return x if x.g > y.g else y
+
+    h_rgb = np.random.randint(0, 256, (10, 3), dtype=np.int32).view(Pixel.dtype)
+    d_rgb = DeviceArray.from_numpy(h_rgb)
+    d_out = DeviceArray.empty(1, Pixel.dtype)
+
+    h_init = Pixel(0, 0, 0)
+
+    cuda.compute.reduce_into(
+        d_in=d_rgb, d_out=d_out, num_items=h_rgb.size, op=max_g_value, h_init=h_init
+    )
+
+    expected = h_rgb[h_rgb.view("int32")[:, 1].argmax()]
+
+    np.testing.assert_equal(expected["g"], d_out.copy_to_host()["g"])
+
+
+@pytest.mark.no_verify_sass(reason="LDL/STL instructions emitted for this test.")
+def test_reduce_struct_type_minmax():
+    @gpu_struct
+    class MinMax:
+        min_val: np.float64
+        max_val: np.float64
+
+    def minmax_op(v1: MinMax, v2: MinMax):
+        c_min = min(v1.min_val, v2.min_val)
+        c_max = max(v1.max_val, v2.max_val)
+        return MinMax(c_min, c_max)
+
+    def transform_op(v):
+        av = abs(v)
+        return MinMax(av, av)
+
+    nelems = 4096
+
+    h_in = np.random.randn(nelems)
+    d_in = DeviceArray.from_numpy(h_in)
+    # input values must be transformed to MinMax structures
+    # in-place to map computation to data-parallel reduction
+    # algorithm that requires commutative binary operation
+    # with both operands having the same type.
+    tr_it = TransformIterator(d_in, transform_op)
+
+    d_out = DeviceArray.empty(tuple(), dtype=MinMax.dtype)
+
+    # initial value set with identity elements of
+    # minimum and maximum operators
+    h_init = MinMax(np.inf, -np.inf)
+
+    # run the reduction algorithm
+    cuda.compute.reduce_into(
+        d_in=tr_it, d_out=d_out, num_items=nelems, op=minmax_op, h_init=h_init
+    )
+
+    # display values computed on the device
+    actual = d_out.copy_to_host()
+
+    h = np.abs(h_in)
+    expected = np.asarray([(h.min(), h.max())], dtype=MinMax.dtype)
+
+    assert actual == expected
+
+
+def test_reduce_transform_output_iterator(floating_array):
+    """Test reduce with TransformOutputIterator."""
+    dtype = floating_array.dtype
+    h_init = np.array([0], dtype=dtype)
+
+    # Use the floating_array fixture which provides random floating-point data of size 1000
+    d_input = DeviceArray.from_numpy(floating_array)
+    d_output = DeviceArray.empty(1, dtype=dtype)
+
+    def sqrt(x: dtype) -> dtype:
+        return x**0.5
+
+    d_out_it = TransformOutputIterator(d_output, sqrt)
+
+    cuda.compute.reduce_into(
+        d_in=d_input,
+        d_out=d_out_it,
+        num_items=len(floating_array),
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+
+    expected = np.sqrt(np.sum(floating_array))
+    np.testing.assert_allclose(d_output.copy_to_host(), expected, atol=1e-6)
+
+
+def test_reduce_with_not_guaranteed_determinism(floating_array):
+    dtype = floating_array.dtype
+    h_init = np.array([0], dtype=dtype)
+
+    d_input = DeviceArray.from_numpy(floating_array)
+    d_output = DeviceArray.empty(1, dtype=dtype)
+
+    cuda.compute.reduce_into(
+        d_in=d_input,
+        d_out=d_output,
+        num_items=len(floating_array),
+        op=OpKind.PLUS,
+        h_init=h_init,
+        determinism=Determinism.NOT_GUARANTEED,
+    )
+
+
+def test_reduce_bool():
+    h_init = np.array([False])
+    h_input = np.array([True, False, True])
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(1, h_input.dtype)
+
+    # Perform the reduction.
+    cuda.compute.reduce_into(
+        d_in=d_input,
+        d_out=d_output,
+        num_items=len(h_input),
+        op=OpKind.MAXIMUM,
+        h_init=h_init,
+    )
+
+    expected = True
+    assert d_output.copy_to_host()[0] == expected
+
+
+def test_reduce_input_and_accumulator_type_mismatch():
+    @gpu_struct
+    class AccumulatorType:
+        x: np.int32
+        y: np.int32
+
+    def op(foo1: AccumulatorType, foo2: AccumulatorType):
+        return AccumulatorType(foo1.x + foo2.x, foo1.y + foo2.y)
+
+    # input data is {int32, int64}
+    dtype = np.dtype([("x", np.int32), ("y", np.int64)], align=True)
+    h_data = np.asarray([(1, 2), (3, 4), (5, 6)], dtype=dtype)
+    d_data = DeviceArray.from_numpy(h_data)
+
+    # output and h_init, both are AccumulatorType
+    d_out = DeviceArray.empty(1, AccumulatorType.dtype)
+    h_init = AccumulatorType(0, 0)  # Init is AccumulatorType
+
+    with pytest.raises(TypeError, match="reduce_into dtype mismatch: input dtype"):
+        cuda.compute.reduce_into(
+            d_in=d_data, d_out=d_out, op=op, num_items=h_data.size, h_init=h_init
+        )
+
+
+def _serialization_add(a, b):
+    return a + b
+
+
+def _serialization_plus_one(x):
+    return x + 1
+
+
+def _run_loaded_reducer(loaded, *, d_in, d_out, num_items, op, h_init):
+    """Drive a loaded reducer through the (size query, execute) two-step."""
+    bytes_needed = loaded(
+        temp_storage=None,
+        d_in=d_in,
+        d_out=d_out,
+        num_items=num_items,
+        op=op,
+        h_init=h_init,
+    )
+    tmp = TempStorageBuffer(bytes_needed, None)
+    loaded(
+        temp_storage=tmp,
+        d_in=d_in,
+        d_out=d_out,
+        num_items=num_items,
+        op=op,
+        h_init=h_init,
+    )
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_well_known_op_round_trip():
+    h_in = np.arange(1024, dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.from_numpy(np.zeros(1, dtype=np.int32))
+    h_init = np.zeros(1, dtype=np.int32)
+
+    reducer = make_reduce_into(d_in=d_in, d_out=d_out, op=OpKind.PLUS, h_init=h_init)
+    blob = serialize(reducer)
+    assert len(blob) > 0
+
+    # Loaded reducer is fully usable without any JIT and without supplying objects.
+    loaded = deserialize(blob)
+    _run_loaded_reducer(
+        loaded,
+        d_in=d_in,
+        d_out=d_out,
+        num_items=h_in.size,
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+
+    assert int(d_out.copy_to_host()[0]) == int(h_in.sum())
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_jit_op_round_trip():
+    h_in = np.arange(1024, dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.from_numpy(np.zeros(1, dtype=np.int32))
+    h_init = np.zeros(1, dtype=np.int32)
+
+    reducer = make_reduce_into(
+        d_in=d_in, d_out=d_out, op=_serialization_add, h_init=h_init
+    )
+    blob = serialize(reducer)
+
+    loaded = deserialize(blob)
+    # The user op's device code is rebuilt from the blob, not from a re-supplied
+    # / recompiled operator: assert it is present before any object reaches the
+    # call below (deserialize took only the blob).
+    assert len(loaded.op_cccl.ltoir) > 0
+    _run_loaded_reducer(
+        loaded,
+        d_in=d_in,
+        d_out=d_out,
+        num_items=h_in.size,
+        op=_serialization_add,
+        h_init=h_init,
+    )
+
+    assert int(d_out.copy_to_host()[0]) == int(h_in.sum())
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_counting_iterator_input():
+    # Custom (ITERATOR-kind) input: its device advance/dereference code must be
+    # captured in the descriptor sidecar and rebuilt with no object supplied.
+    n = 1024
+    d_out = DeviceArray.from_numpy(np.zeros(1, dtype=np.int64))
+    h_init = np.zeros(1, dtype=np.int64)
+
+    reducer = make_reduce_into(
+        d_in=CountingIterator(np.int32(0)), d_out=d_out, op=OpKind.PLUS, h_init=h_init
+    )
+    blob = serialize(reducer)
+
+    loaded = deserialize(blob)
+    # The ITERATOR-kind descriptor (with its advance/dereference device code) was
+    # rebuilt purely from the blob — no iterator object was passed to deserialize,
+    # so a regression to caller-supplied descriptors would fail here, not silently
+    # pass via the object reconstructed for the call.
+    assert loaded.d_in_cccl.is_kind_iterator()
+    _run_loaded_reducer(
+        loaded,
+        d_in=CountingIterator(np.int32(0)),
+        d_out=d_out,
+        num_items=n,
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+
+    assert int(d_out.copy_to_host()[0]) == n * (n - 1) // 2
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_transform_iterator_input():
+    # TransformIterator carries a user op (device code) inside the iterator's
+    # dereference op — exercises iterator-embedded LTOIR round-tripping.
+    n = 512
+    d_out = DeviceArray.from_numpy(np.zeros(1, dtype=np.int64))
+    h_init = np.zeros(1, dtype=np.int64)
+
+    def make_it():
+        return TransformIterator(CountingIterator(np.int32(0)), _serialization_plus_one)
+
+    reducer = make_reduce_into(
+        d_in=make_it(), d_out=d_out, op=OpKind.PLUS, h_init=h_init
+    )
+    blob = serialize(reducer)
+
+    loaded = deserialize(blob)
+    # Iterator descriptor (incl. the transform op's embedded LTOIR) rebuilt from
+    # the blob alone — deserialize took no objects.
+    assert loaded.d_in_cccl.is_kind_iterator()
+    _run_loaded_reducer(
+        loaded, d_in=make_it(), d_out=d_out, num_items=n, op=OpKind.PLUS, h_init=h_init
+    )
+
+    # sum of (i + 1) for i in 0..n-1
+    assert int(d_out.copy_to_host()[0]) == n * (n - 1) // 2 + n
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_preserves_determinism():
+    d_in = DeviceArray.from_numpy(np.arange(64, dtype=np.int32))
+    d_out = DeviceArray.from_numpy(np.zeros(1, dtype=np.int32))
+    h_init = np.zeros(1, dtype=np.int32)
+
+    reducer = make_reduce_into(
+        d_in=d_in,
+        d_out=d_out,
+        op=OpKind.PLUS,
+        h_init=h_init,
+        determinism=Determinism.NOT_GUARANTEED,
+    )
+    blob = serialize(reducer)
+
+    loaded = deserialize(blob)
+    # determinism is a build-time property, preserved on each compiled per-arch
+    # build result (loaded_build_result binds lazily on first call, so inspect build_results).
+    assert all(
+        v.determinism == int(Determinism.NOT_GUARANTEED)
+        for v in loaded.build_results.values()
+    )
+
+
+@pytest.mark.serialization
+def test_deserialize_garbage_raises():
+    with pytest.raises((ValueError, RuntimeError)):
+        deserialize(b"not a real serialization blob" + b"\0" * 64)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_scan.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_scan.py
new file mode 100644
index 00000000..85569ec9
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_scan.py
@@ -0,0 +1,649 @@
+# Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray, get_compute_capability
+
+import cuda.compute
+from cuda.compute import (
+    CountingIterator,
+    OpKind,
+    ReverseIterator,
+    TransformOutputIterator,
+    clear_all_caches,
+    deserialize,
+    exclusive_scan,
+    gpu_struct,
+    make_exclusive_scan,
+    make_inclusive_scan,
+    serialize,
+)
+from cuda.compute._utils.temp_storage_buffer import TempStorageBuffer
+
+
+def scan_host(h_input: np.ndarray, op, h_init, force_inclusive):
+    result = h_input.copy()
+    if force_inclusive:
+        result[0] = op(h_init[0], result[0])
+    else:
+        result[0] = h_init[0]
+
+    for i in range(1, len(result)):
+        if force_inclusive:
+            result[i] = op(result[i - 1], h_input[i])
+        else:
+            result[i] = op(result[i - 1], h_input[i - 1])
+    return result
+
+
+def scan_device(d_input, d_output, num_items, op, h_init, force_inclusive, stream=None):
+    scan_algorithm = (
+        cuda.compute.inclusive_scan if force_inclusive else cuda.compute.exclusive_scan
+    )
+    scan_algorithm(
+        d_in=d_input,
+        d_out=d_output,
+        op=op,
+        init_value=h_init,
+        num_items=num_items,
+        stream=stream,
+    )
+
+
+@pytest.mark.parametrize(
+    "force_inclusive",
+    [True, False],
+)
+def test_scan_array_input(force_inclusive, input_array, monkeypatch):
+    cc_major, _ = get_compute_capability()
+    # Skip sass verification if input is complex
+    # as LDL/STL instructions are emitted for complex types.
+    # Also skip for:
+    # * uint8-True
+    # * int8-True
+    # * float64-False
+    # Also skip for CC 9.0+, due to a bug in NVRTC.
+    # TODO: add NVRTC version check, ref nvbug 5243118
+    if (
+        np.issubdtype(input_array.dtype, np.complexfloating)
+        or (force_inclusive and np.isdtype(input_array.dtype, (np.uint8, np.int8)))
+        or (not force_inclusive and input_array.dtype == np.float64)
+        or cc_major >= 9
+    ):
+        import cuda.compute._cccl_interop
+
+        monkeypatch.setattr(
+            cuda.compute._cccl_interop,
+            "_check_sass",
+            False,
+        )
+
+    def op(a, b):
+        return a + b
+
+    dtype = input_array.dtype
+
+    if dtype == np.float16:
+        reduce_op = OpKind.PLUS
+    else:
+        reduce_op = op
+
+    is_short_dtype = dtype.itemsize < 16
+    # for small range data types make input small to assure that
+    # accumulation does not overflow
+    h_input = input_array[:31] if is_short_dtype else input_array
+    d_input = DeviceArray.from_numpy(h_input)
+
+    h_init = np.array([42], dtype=dtype)
+    d_output = DeviceArray.empty(h_input.shape, h_input.dtype)
+
+    scan_device(d_input, d_output, h_input.size, reduce_op, h_init, force_inclusive)
+
+    got = d_output.copy_to_host()
+    expected = scan_host(h_input, op, h_init, force_inclusive)
+
+    if np.isdtype(dtype, ("real floating", "complex floating")):
+        real_dt = np.finfo(dtype).dtype
+        eps = np.finfo(real_dt).eps
+        rtol = 82 * eps
+        np.testing.assert_allclose(expected, got, rtol=rtol)
+    else:
+        np.testing.assert_array_equal(expected, got)
+
+
+@pytest.mark.parametrize(
+    "force_inclusive",
+    [True, False],
+)
+def test_scan_iterator_input(force_inclusive):
+    def op(a, b):
+        return a + b
+
+    d_input = CountingIterator(np.int32(1))
+    num_items = 1024
+    dtype = np.dtype("int32")
+    h_init = np.array([42], dtype=dtype)
+    d_output = DeviceArray.empty(num_items, dtype)
+
+    scan_device(d_input, d_output, num_items, op, h_init, force_inclusive)
+
+    got = d_output.copy_to_host()
+    expected = scan_host(
+        np.arange(1, num_items + 1, dtype=dtype), op, h_init, force_inclusive
+    )
+
+    np.testing.assert_allclose(expected, got, rtol=1e-5)
+
+
+@pytest.mark.parametrize(
+    "force_inclusive",
+    [True, False],
+)
+def test_scan_reverse_counting_iterator_input(force_inclusive):
+    def op(a, b):
+        return a + b
+
+    num_items = 1024
+    d_input = ReverseIterator(CountingIterator(np.int32(num_items)))
+    dtype = np.dtype("int32")
+    h_init = np.array([0], dtype=dtype)
+    d_output = DeviceArray.empty(num_items, dtype)
+
+    scan_device(d_input, d_output, num_items, op, h_init, force_inclusive)
+
+    got = d_output.copy_to_host()
+    expected = scan_host(
+        np.arange(num_items, 0, -1, dtype=dtype), op, h_init, force_inclusive
+    )
+
+    np.testing.assert_allclose(expected, got, rtol=1e-5)
+
+
+@pytest.mark.parametrize(
+    "force_inclusive",
+    [True, False],
+)
+@pytest.mark.no_verify_sass(reason="LDL/STL instructions emitted for this test.")
+def test_scan_struct_type(force_inclusive):
+    @gpu_struct
+    class XY:
+        x: np.int32
+        y: np.int32
+
+    def op(a, b):
+        return XY(a.x + b.x, a.y + b.y)
+
+    h_input = np.random.randint(0, 256, (10, 2), dtype=np.int32).view(XY.dtype)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, h_input.dtype)
+
+    h_init = XY(0, 0)
+
+    scan_device(d_input, d_output, len(h_input), op, h_init, force_inclusive)
+
+    got = d_output.copy_to_host()
+    expected_x = scan_host(
+        h_input["x"], lambda a, b: a + b, np.asarray([h_init.x]), force_inclusive
+    )
+    expected_y = scan_host(
+        h_input["y"], lambda a, b: a + b, np.asarray([h_init.y]), force_inclusive
+    )
+
+    np.testing.assert_allclose(expected_x, got["x"], rtol=1e-5)
+    np.testing.assert_allclose(expected_y, got["y"], rtol=1e-5)
+
+
+@pytest.mark.parametrize(
+    "force_inclusive",
+    [True, False],
+)
+def test_scan_with_stream(force_inclusive, cuda_stream):
+    def op(a, b):
+        return a + b
+
+    h_input = np.random.randint(0, 256, 1024, dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input, stream=cuda_stream)
+    d_output = DeviceArray.empty(h_input.shape, h_input.dtype, stream=cuda_stream)
+
+    h_init = np.array([42], dtype=np.int32)
+
+    scan_device(
+        d_input, d_output, h_input.size, op, h_init, force_inclusive, stream=cuda_stream
+    )
+
+    got = d_output.copy_to_host(stream=cuda_stream)
+    expected = scan_host(h_input, op, h_init, force_inclusive)
+
+    np.testing.assert_allclose(expected, got, rtol=1e-5)
+
+
+def test_exclusive_scan_well_known_plus():
+    dtype = np.int32
+    h_init = np.array([0], dtype=dtype)
+    h_input = np.array([1, 2, 3, 4, 5], dtype=dtype)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, dtype)
+
+    cuda.compute.exclusive_scan(
+        d_in=d_input,
+        d_out=d_output,
+        op=OpKind.PLUS,
+        init_value=h_init,
+        num_items=h_input.size,
+    )
+
+    expected = np.array([0, 1, 3, 6, 10])
+    np.testing.assert_equal(d_output.copy_to_host(), expected)
+
+
+def test_inclusive_scan_well_known_plus(monkeypatch):
+    cc_major, _ = get_compute_capability()
+    # Skip SASS check for CC 9.0+, due to a bug in NVRTC.
+    # TODO: add NVRTC version check, ref nvbug 5243118
+    if cc_major >= 9:
+        import cuda.compute._cccl_interop as cccl_interop
+
+        monkeypatch.setattr(
+            cccl_interop,
+            "_check_sass",
+            False,
+        )
+
+    dtype = np.int32
+    h_init = np.array([0], dtype=dtype)
+    h_input = np.array([1, 2, 3, 4, 5], dtype=dtype)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, dtype)
+
+    cuda.compute.inclusive_scan(
+        d_in=d_input,
+        d_out=d_output,
+        op=OpKind.PLUS,
+        init_value=h_init,
+        num_items=h_input.size,
+    )
+
+    expected = np.array([1, 3, 6, 10, 15])
+    np.testing.assert_equal(d_output.copy_to_host(), expected)
+
+
+@pytest.mark.xfail(
+    reason="CCCL_MAXIMUM well-known operation fails with NVRTC compilation error in C++ library"
+)
+def test_exclusive_scan_well_known_maximum():
+    dtype = np.int32
+    h_init = np.array([1], dtype=dtype)
+    h_input = np.array([-5, 0, 2, -3, 2, 4, 0, -1, 2, 8], dtype=dtype)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, dtype)
+
+    cuda.compute.exclusive_scan(
+        d_in=d_input,
+        d_out=d_output,
+        op=OpKind.MAXIMUM,
+        init_value=h_init,
+        num_items=h_input.size,
+    )
+
+    expected = np.array([1, 1, 1, 2, 2, 2, 4, 4, 4, 4])
+    np.testing.assert_equal(d_output.copy_to_host(), expected)
+
+
+def test_scan_transform_output_iterator(floating_array):
+    """Test scan with TransformOutputIterator."""
+    dtype = floating_array.dtype
+    h_init = np.array([0], dtype=dtype)
+
+    # Use the floating_array fixture which provides random floating-point data of size 1000
+    h_input = floating_array
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, dtype)
+
+    def square(x: dtype) -> dtype:
+        return x * x
+
+    d_out_it = TransformOutputIterator(d_output, square)
+
+    cuda.compute.inclusive_scan(
+        d_in=d_input,
+        d_out=d_out_it,
+        op=OpKind.PLUS,
+        init_value=h_init,
+        num_items=h_input.size,
+    )
+
+    expected = np.cumsum(h_input) ** 2
+    # Use more lenient tolerance for float32 due to precision differences
+    if dtype == np.float32:
+        np.testing.assert_allclose(
+            d_output.copy_to_host(), expected, atol=1e-4, rtol=1e-4
+        )
+    else:
+        np.testing.assert_allclose(d_output.copy_to_host(), expected, atol=1e-6)
+
+
+def test_exclusive_scan_max():
+    def max_op(a, b):
+        return max(a, b)
+
+    h_init = np.array([1], dtype="int32")
+    h_input = np.array([-5, 0, 2, -3, 2, 4, 0, -1, 2, 8], dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, h_input.dtype)
+
+    cuda.compute.exclusive_scan(
+        d_in=d_input,
+        d_out=d_output,
+        op=max_op,
+        init_value=h_init,
+        num_items=h_input.size,
+    )
+
+    expected = np.asarray([1, 1, 1, 2, 2, 2, 4, 4, 4, 4])
+    np.testing.assert_equal(d_output.copy_to_host(), expected)
+
+
+def test_inclusive_scan_add():
+    def add_op(a, b):
+        return a + b
+
+    h_init = np.array([0], dtype="int32")
+    h_input = np.array([-5, 0, 2, -3, 2, 4, 0, -1, 2, 8], dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, h_input.dtype)
+
+    cuda.compute.inclusive_scan(
+        d_in=d_input,
+        d_out=d_output,
+        op=add_op,
+        init_value=h_init,
+        num_items=h_input.size,
+    )
+
+    expected = np.asarray([-5, -5, -3, -6, -4, 0, 0, -1, 1, 9])
+    np.testing.assert_equal(d_output.copy_to_host(), expected)
+
+
+def test_reverse_input_iterator(monkeypatch):
+    cc_major, _ = get_compute_capability()
+    # Skip SASS check for CC 9.0+, due to a bug in NVRTC.
+    # TODO: add NVRTC version check, ref nvbug 5243118
+    if cc_major >= 9:
+        import cuda.compute._cccl_interop as cccl_interop
+
+        monkeypatch.setattr(
+            cccl_interop,
+            "_check_sass",
+            False,
+        )
+
+    def add_op(a, b):
+        return a + b
+
+    h_init = np.array([0], dtype="int32")
+    h_input = np.array([-5, 0, 2, -3, 2, 4, 0, -1, 2, 8], dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, h_input.dtype)
+    reverse_it = ReverseIterator(d_input)
+
+    cuda.compute.inclusive_scan(
+        d_in=reverse_it,
+        d_out=d_output,
+        op=add_op,
+        init_value=h_init,
+        num_items=h_input.size,
+    )
+
+    # Check the result is correct
+    expected = np.asarray([8, 10, 9, 9, 13, 15, 12, 14, 14, 9])
+    np.testing.assert_equal(d_output.copy_to_host(), expected)
+
+
+@pytest.mark.no_verify_sass(reason="LDL/STL instructions emitted for this test.")
+def test_reverse_output_iterator():
+    def add_op(a, b):
+        return a + b
+
+    h_init = np.array([0], dtype="int32")
+    h_input = np.array([-5, 0, 2, -3, 2, 4, 0, -1, 2, 8], dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, h_input.dtype)
+    reverse_it = ReverseIterator(d_output)
+
+    cuda.compute.inclusive_scan(
+        d_in=d_input,
+        d_out=reverse_it,
+        op=add_op,
+        init_value=h_init,
+        num_items=h_input.size,
+    )
+
+    expected = np.asarray([9, 1, -1, 0, 0, -4, -6, -3, -5, -5])
+    np.testing.assert_equal(d_output.copy_to_host(), expected)
+
+
+@pytest.mark.parametrize(
+    "force_inclusive",
+    [True, False],
+)
+def test_future_init_value(force_inclusive):
+    num_items = 1024
+    dtype = np.dtype("int32")
+
+    h_input = np.random.randint(0, 256, num_items, dtype=dtype)
+    h_init = np.array([42], dtype=dtype)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, h_input.dtype)
+    init_value = DeviceArray.from_numpy(h_init)
+
+    scan_device(d_input, d_output, num_items, OpKind.PLUS, init_value, force_inclusive)
+
+    got = d_output.copy_to_host()
+    expected = scan_host(h_input, lambda a, b: a + b, h_init, force_inclusive)
+    np.testing.assert_array_equal(expected, got)
+
+
+def test_no_init_value(monkeypatch):
+    force_inclusive = True
+    num_items = 1024
+    dtype = np.dtype("int32")
+
+    # Skip SASS check for CC 9.0 due to LDL/STL CI failure.
+    cc_major, _ = get_compute_capability()
+    if cc_major >= 9:
+        import cuda.compute._cccl_interop
+
+        monkeypatch.setattr(
+            cuda.compute._cccl_interop,
+            "_check_sass",
+            False,
+        )
+
+    h_input = np.random.randint(0, 256, num_items, dtype=dtype)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, h_input.dtype)
+
+    scan_device(d_input, d_output, num_items, OpKind.PLUS, None, force_inclusive)
+
+    got = d_output.copy_to_host()
+    expected = scan_host(h_input, lambda a, b: a + b, [0], force_inclusive)
+    np.testing.assert_array_equal(expected, got)
+
+
+def test_no_init_value_iterator():
+    force_inclusive = True
+    num_items = 1024
+    dtype = np.dtype("float64")
+
+    d_input = CountingIterator(np.float64(0))
+    d_output = DeviceArray.empty(num_items, dtype)
+
+    scan_device(d_input, d_output, num_items, OpKind.PLUS, None, force_inclusive)
+
+    got = d_output.copy_to_host()
+    expected = scan_host(
+        np.arange(0, num_items, dtype=dtype), lambda a, b: a + b, [0], force_inclusive
+    )
+
+    np.testing.assert_array_equal(expected, got)
+
+
+def test_inclusive_scan_with_lambda():
+    """Test inclusive_scan with a lambda function as the scan operator."""
+    h_init = np.array([0], dtype=np.int32)
+    h_input = np.array([1, 2, 3, 4, 5], dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, h_input.dtype)
+
+    # Use a lambda function directly as the scan operator
+    cuda.compute.inclusive_scan(
+        d_in=d_input,
+        d_out=d_output,
+        op=lambda a, b: a + b,
+        init_value=h_init,
+        num_items=h_input.size,
+    )
+
+    expected = np.array([1, 3, 6, 10, 15], dtype=np.int32)
+    np.testing.assert_array_equal(d_output.copy_to_host(), expected)
+
+
+@pytest.mark.parametrize("force_inclusive", [True, False])
+def test_scan_bool_maximum(force_inclusive):
+    h_init = np.array([False], dtype=np.bool_)
+    h_input = np.array([False, True, False, True], dtype=np.bool_)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, h_input.dtype)
+
+    scan_device(
+        d_input, d_output, h_input.size, OpKind.MAXIMUM, h_init, force_inclusive
+    )
+
+    if force_inclusive:
+        expected = np.array([False, True, True, True], dtype=np.bool_)
+    else:
+        expected = np.array([False, False, True, True], dtype=np.bool_)
+
+    np.testing.assert_array_equal(d_output.copy_to_host(), expected)
+
+
+def _run(scanner, *, d_in, d_out, op, init_value, num_items):
+    bytes_needed = scanner(
+        temp_storage=None,
+        d_in=d_in,
+        d_out=d_out,
+        op=op,
+        init_value=init_value,
+        num_items=num_items,
+    )
+    tmp = TempStorageBuffer(bytes_needed, None)
+    scanner(
+        temp_storage=tmp,
+        d_in=d_in,
+        d_out=d_out,
+        op=op,
+        init_value=init_value,
+        num_items=num_items,
+    )
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_exclusive_scan_round_trip():
+    h_in = np.arange(1, 33, dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    init_value = np.array([0], dtype=np.int32)
+
+    builder = make_exclusive_scan(
+        d_in=d_in, d_out=d_out, op=OpKind.PLUS, init_value=init_value
+    )
+    blob = serialize(builder)
+    assert len(blob) > 0
+
+    loaded = deserialize(blob)
+    _run(
+        loaded,
+        d_in=d_in,
+        d_out=d_out,
+        op=OpKind.PLUS,
+        init_value=init_value,
+        num_items=h_in.size,
+    )
+
+    expected = np.zeros_like(h_in)
+    np.cumsum(h_in[:-1], out=expected[1:])
+    np.testing.assert_array_equal(d_out.copy_to_host(), expected)
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_inclusive_scan_round_trip():
+    h_in = np.arange(1, 33, dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    init_value = np.array([0], dtype=np.int32)
+
+    builder = make_inclusive_scan(
+        d_in=d_in, d_out=d_out, op=OpKind.PLUS, init_value=init_value
+    )
+    blob = serialize(builder)
+
+    loaded = deserialize(blob)
+    _run(
+        loaded,
+        d_in=d_in,
+        d_out=d_out,
+        op=OpKind.PLUS,
+        init_value=init_value,
+        num_items=h_in.size,
+    )
+
+    np.testing.assert_array_equal(d_out.copy_to_host(), np.cumsum(h_in))
+
+
+@pytest.mark.serialization
+def test_deserialize_after_jit_matches_jit_result():
+    """Serialize a JITed scan, deserialize, and confirm output matches a fresh JIT."""
+    h_in = np.array([-5, 0, 2, -3, 2, 4, 0, -1, 2, 8], dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out_jit = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_out_serialization = DeviceArray.empty(h_in.shape, h_in.dtype)
+    init_value = np.array([1], dtype=np.int32)
+
+    def max_op(a, b):
+        return a if a > b else b
+
+    # Build + serialize (this JITs), then clear every in-process cache so the serialization
+    # leg below runs cold: the deserialized scan must stand on its own and cannot
+    # free-ride on a callable warmed by the build or by the JIT reference.
+    blob = serialize(
+        make_exclusive_scan(
+            d_in=d_in, d_out=d_out_serialization, op=max_op, init_value=init_value
+        )
+    )
+    clear_all_caches()
+
+    loaded = deserialize(blob)
+    _run(
+        loaded,
+        d_in=d_in,
+        d_out=d_out_serialization,
+        op=max_op,
+        init_value=init_value,
+        num_items=h_in.size,
+    )
+
+    # Compute the JIT reference only after the serialization path has already run.
+    exclusive_scan(
+        d_in=d_in,
+        d_out=d_out_jit,
+        op=max_op,
+        init_value=init_value,
+        num_items=h_in.size,
+    )
+
+    np.testing.assert_array_equal(
+        d_out_serialization.copy_to_host(), d_out_jit.copy_to_host()
+    )
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_segmented_reduce.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_segmented_reduce.py
new file mode 100644
index 00000000..3b7b2286
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_segmented_reduce.py
@@ -0,0 +1,647 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray
+
+import cuda.compute
+from cuda.compute import (
+    ConstantIterator,
+    CountingIterator,
+    OpKind,
+    TransformIterator,
+    TransformOutputIterator,
+    deserialize,
+    gpu_struct,
+    make_segmented_reduce,
+    serialize,
+)
+from cuda.compute._utils.temp_storage_buffer import TempStorageBuffer
+
+
+def is_out_of_memory_error(error):
+    # cuda-core exception types vary by memory resource, so classify by message.
+    message = str(error).lower()
+    return any(
+        marker in message
+        for marker in (
+            "out of memory",
+            "out_of_memory",
+            "failed to allocate memory from pool",
+        )
+    )
+
+
+@pytest.fixture(params=["i4", "u4", "i8", "u8"])
+def offset_dtype(request):
+    return np.dtype(request.param)
+
+
+def test_segmented_reduce(input_array, offset_dtype, monkeypatch):
+    "Test for all supported input types and for some offset types"
+    # Disable SASS verification for this test (LDL instruction in SASS).
+    monkeypatch.setattr(
+        cuda.compute._cccl_interop,
+        "_check_sass",
+        False,
+    )
+
+    def binary_op(a, b):
+        return a + b
+
+    assert input_array.ndim == 1
+    sz = input_array.size
+    rng = np.random.default_rng()
+    n_segments = 16
+    h_offsets = np.zeros(n_segments + 1, dtype="int64")
+    h_offsets[1:] = rng.multinomial(sz, [1 / n_segments] * n_segments)
+
+    offsets = np.cumsum(np.asarray(h_offsets, dtype=offset_dtype), dtype=offset_dtype)
+
+    h_start_offsets = offsets[:-1]
+    h_end_offsets = offsets[1:]
+
+    assert offsets.dtype == np.dtype(offset_dtype)
+    assert np.all(h_start_offsets <= h_end_offsets)
+    assert h_end_offsets[-1] == sz
+
+    d_in = DeviceArray.from_numpy(input_array)
+    d_out = DeviceArray.empty(n_segments, input_array.dtype)
+    start_offsets = DeviceArray.from_numpy(h_start_offsets)
+    end_offsets = DeviceArray.from_numpy(h_end_offsets)
+
+    h_init = np.zeros(tuple(), dtype=input_array.dtype)
+
+    if input_array.dtype == np.float16:
+        reduce_op = OpKind.PLUS
+    else:
+        reduce_op = binary_op
+
+    # Call single-phase API directly with num_segments parameter
+    cuda.compute.segmented_reduce(
+        d_in=d_in,
+        d_out=d_out,
+        num_segments=n_segments,
+        start_offsets_in=start_offsets,
+        end_offsets_in=end_offsets,
+        op=reduce_op,
+        h_init=h_init,
+    )
+
+    expected = np.empty(n_segments, dtype=input_array.dtype)
+    for i in range(n_segments):
+        expected[i] = np.sum(input_array[h_start_offsets[i] : h_end_offsets[i]])
+
+    result = d_out.copy_to_host()
+    if np.issubdtype(input_array.dtype, np.inexact):
+        tolerance = 4 * np.finfo(input_array.dtype).eps
+        np.testing.assert_allclose(result, expected, rtol=tolerance, atol=tolerance)
+    else:
+        np.testing.assert_array_equal(result, expected)
+
+
+def test_segmented_reduce_struct_type(monkeypatch):
+    # Disable SASS verification for this test (LDL instruction in SASS).
+    monkeypatch.setattr(
+        cuda.compute._cccl_interop,
+        "_check_sass",
+        False,
+    )
+
+    @gpu_struct
+    class Pixel:
+        r: np.int32
+        g: np.int32
+        b: np.int32
+
+    def max_g_value(x, y):
+        return x if x.g > y.g else y
+
+    def align_up(n, m):
+        return ((n + m - 1) // m) * m
+
+    segment_size = 64
+    n_pixels = align_up(4000, 64)
+    offsets = np.arange(n_pixels + segment_size - 1, step=segment_size, dtype=np.int64)
+    h_start_offsets = offsets[:-1]
+    h_end_offsets = offsets[1:]
+    n_segments = h_start_offsets.size
+
+    rng = np.random.default_rng()
+    h_rgb = rng.integers(0, 256, (n_pixels, 3), dtype=np.int32)
+    h_rgb = h_rgb.view(Pixel.dtype).reshape(n_pixels)
+    d_rgb = DeviceArray.from_numpy(h_rgb)
+    d_out = DeviceArray.empty(n_segments, Pixel.dtype)
+    start_offsets = DeviceArray.from_numpy(h_start_offsets)
+    end_offsets = DeviceArray.from_numpy(h_end_offsets)
+
+    h_init = Pixel(0, 0, 0)
+
+    # Call single-phase API directly with n_segments parameter
+    cuda.compute.segmented_reduce(
+        d_in=d_rgb,
+        d_out=d_out,
+        num_segments=n_segments,
+        start_offsets_in=start_offsets,
+        end_offsets_in=end_offsets,
+        op=max_g_value,
+        h_init=h_init,
+    )
+
+    h_rgb = np.reshape(h_rgb, (n_segments, -1))
+    expected = h_rgb[np.arange(h_rgb.shape[0]), h_rgb["g"].argmax(axis=-1)]
+
+    np.testing.assert_equal(expected["g"], d_out.copy_to_host()["g"])
+
+
+@pytest.mark.large
+def test_large_num_segments_uniform_segment_sizes_nonuniform_input(monkeypatch):
+    """
+    This test verifies that segmented_reduce raises an error when
+    num_segments exceeds 2^31-1.
+    """
+    # Disable SASS verification for this test (LDL instruction in SASS).
+    monkeypatch.setattr(
+        cuda.compute._cccl_interop,
+        "_check_sass",
+        False,
+    )
+
+    def make_difference(idx: np.int64) -> np.uint8:
+        p = np.uint8(7)
+
+        # Annotations on this nested function are intentionally omitted: on
+        # Python 3.14 they cause numba_cuda to abort with `AssertionError:
+        # unreachable` in op_SET_FUNCTION_ATTRIBUTE, because the new 0x10
+        # (`__annotate__`, PEP 649) flag isn't handled.
+        # Original signature: def Fu(idx: np.int64) -> np.uint8:
+        def Fu(idx):
+            i8 = np.uint8(idx % 5) + np.uint8(idx % 3)
+            f = (i8 * (i8 + 1)) % p
+            return f
+
+        return (Fu(idx + 1) - Fu(idx)) % p
+
+    input_it = TransformIterator(CountingIterator(np.int64(0)), make_difference)
+
+    def make_scaler(step):
+        def scale(row_id):
+            return row_id * step
+
+        return scale
+
+    segment_size = 116
+    offset0 = np.int64(0)
+    row_offset = make_scaler(np.int64(segment_size))
+    start_offsets = TransformIterator(CountingIterator(offset0), row_offset)
+    end_offsets = start_offsets + 1
+
+    num_segments = (2**15 + 2**3) * 2**16
+    try:
+        res = DeviceArray.empty(num_segments, np.uint8)
+    except Exception as error:
+        if not is_out_of_memory_error(error):
+            raise
+        pytest.skip("Insufficient memory to run the large number of segments test")
+    assert res.nbytes == num_segments * np.dtype(np.uint8).itemsize
+
+    def my_add(a: np.uint8, b: np.uint8) -> np.uint8:
+        return (a + b) % np.uint8(7)
+
+    h_init = np.zeros(tuple(), dtype=np.uint8)
+
+    # Verify that the appropriate error is raised
+    with pytest.raises(
+        RuntimeError,
+        match="Segmented sort does not currently support more than 2\\^31-1 segments\\.",
+    ):
+        cuda.compute.segmented_reduce(
+            d_in=input_it,
+            d_out=res,
+            num_segments=num_segments,
+            start_offsets_in=start_offsets,
+            end_offsets_in=end_offsets,
+            op=my_add,
+            h_init=h_init,
+        )
+
+
+@pytest.mark.large
+def test_large_num_segments_nonuniform_segment_sizes_uniform_input(monkeypatch):
+    """
+    This test verifies that segmented_reduce raises an error when
+    num_segments exceeds 2^31-1.
+    """
+    # Disable SASS verification for this test (LDL instruction in SASS).
+    monkeypatch.setattr(
+        cuda.compute._cccl_interop,
+        "_check_sass",
+        False,
+    )
+    input_it = ConstantIterator(np.int16(1))
+
+    def offset_functor(m0: np.int64, p: np.int64):
+        def offset_value(n: np.int64) -> np.int64:
+            """
+            Offset value computes closed form for
+            :math:`sum(1 + (k % p), k=0..n)`.
+
+            So segment lengths are periodic linearly
+            increasing sequences, e.g,
+            [min , min + 1, ..., min + p - 2,
+                min + p - 1, min, min +1 , ....]
+            """
+            q = n // p
+            r = n - q * p
+            p2 = (p * (p - 1)) // 2
+            r2 = (r * (r + 1)) // 2
+
+            offset_val = (n + 1) * m0 + q * p2 + r2
+            return offset_val
+
+        return offset_value
+
+    m0, p = np.int64(265), np.int64(163)
+    offsets_it = TransformIterator(
+        CountingIterator(np.int64(-1)), offset_functor(m0, p)
+    )
+    start_offsets = offsets_it
+    end_offsets = offsets_it + 1
+
+    def _plus(a, b):
+        return a + b
+
+    num_segments = (2**15 + 2**3) * 2**16
+    try:
+        res = DeviceArray.empty(num_segments, np.int16)
+    except Exception as error:
+        if not is_out_of_memory_error(error):
+            raise
+        pytest.skip("Insufficient memory to run the large number of segments test")
+    assert res.nbytes == num_segments * np.dtype(np.int16).itemsize
+
+    h_init = np.zeros(tuple(), dtype=np.int16)
+
+    # Verify that the appropriate error is raised
+    with pytest.raises(
+        RuntimeError,
+        match="Segmented sort does not currently support more than 2\\^31-1 segments\\.",
+    ):
+        cuda.compute.segmented_reduce(
+            d_in=input_it,
+            d_out=res,
+            num_segments=num_segments,
+            start_offsets_in=start_offsets,
+            end_offsets_in=end_offsets,
+            op=_plus,
+            h_init=h_init,
+        )
+
+
+def test_segmented_reduce_well_known_plus(monkeypatch):
+    # Disable SASS verification for this test (LDL instruction in SASS).
+    monkeypatch.setattr(
+        cuda.compute._cccl_interop,
+        "_check_sass",
+        False,
+    )
+    dtype = np.int32
+    h_init = np.array([0], dtype=dtype)
+
+    # Create segmented data: [1, 2, 3] | [4, 5] | [6, 7, 8, 9]
+    h_input = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=dtype)
+    h_starts = np.array([0, 3, 5], dtype=np.int32)
+    h_ends = np.array([3, 5, 9], dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_starts = DeviceArray.from_numpy(h_starts)
+    d_ends = DeviceArray.from_numpy(h_ends)
+    d_output = DeviceArray.empty(3, dtype)
+
+    cuda.compute.segmented_reduce(
+        d_in=d_input,
+        d_out=d_output,
+        num_segments=3,
+        start_offsets_in=d_starts,
+        end_offsets_in=d_ends,
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+
+    expected = np.array([6, 9, 30])
+    np.testing.assert_equal(d_output.copy_to_host(), expected)
+
+
+def test_segmented_reduce_well_known_maximum(monkeypatch):
+    # Disable SASS verification for this test (LDL instruction in SASS).
+    monkeypatch.setattr(
+        cuda.compute._cccl_interop,
+        "_check_sass",
+        False,
+    )
+    dtype = np.int32
+    h_init = np.array([-100], dtype=dtype)
+
+    # Create segmented data: [1, 9, 3] | [4, 2] | [6, 7, 1, 8]
+    h_input = np.array([1, 9, 3, 4, 2, 6, 7, 1, 8], dtype=dtype)
+    h_starts = np.array([0, 3, 5], dtype=np.int32)
+    h_ends = np.array([3, 5, 9], dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_starts = DeviceArray.from_numpy(h_starts)
+    d_ends = DeviceArray.from_numpy(h_ends)
+    d_output = DeviceArray.empty(3, dtype)
+
+    cuda.compute.segmented_reduce(
+        d_in=d_input,
+        d_out=d_output,
+        num_segments=3,
+        start_offsets_in=d_starts,
+        end_offsets_in=d_ends,
+        op=OpKind.MAXIMUM,
+        h_init=h_init,
+    )
+
+    expected = np.array([9, 4, 8])  # max of each segment
+    np.testing.assert_equal(d_output.copy_to_host(), expected)
+
+
+def test_segmented_reduce_bool_maximum(monkeypatch):
+    # Disable SASS verification for this test (LDL instruction in SASS).
+    monkeypatch.setattr(
+        cuda.compute._cccl_interop,
+        "_check_sass",
+        False,
+    )
+    h_init = np.array([False], dtype=np.bool_)
+
+    # Create segmented data: [False, True] | [False, False] | [True]
+    h_input = np.array([False, True, False, False, True], dtype=np.bool_)
+    h_starts = np.array([0, 2, 4], dtype=np.int32)
+    h_ends = np.array([2, 4, 5], dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_starts = DeviceArray.from_numpy(h_starts)
+    d_ends = DeviceArray.from_numpy(h_ends)
+    d_output = DeviceArray.empty(3, np.bool_)
+
+    cuda.compute.segmented_reduce(
+        d_in=d_input,
+        d_out=d_output,
+        num_segments=3,
+        start_offsets_in=d_starts,
+        end_offsets_in=d_ends,
+        op=OpKind.MAXIMUM,
+        h_init=h_init,
+    )
+
+    expected = np.array([True, False, True], dtype=np.bool_)
+    np.testing.assert_equal(d_output.copy_to_host(), expected)
+
+
+def test_segmented_reduce_transform_output_iterator(floating_array, monkeypatch):
+    """Test segmented reduce with TransformOutputIterator."""
+    # Disable SASS verification for this test (LDL instruction in SASS).
+    monkeypatch.setattr(
+        cuda.compute._cccl_interop,
+        "_check_sass",
+        False,
+    )
+    dtype = floating_array.dtype
+    h_init = np.array([0], dtype=dtype)
+
+    # Use the floating_array fixture which provides random floating-point data of size 1000
+    d_input = DeviceArray.from_numpy(floating_array)
+
+    # Create 2 segments of roughly equal size
+    segment_size = floating_array.size // 2
+    d_output = DeviceArray.empty(2, dtype)
+    start_offsets = DeviceArray.from_numpy(np.array([0, segment_size], dtype=np.int32))
+    end_offsets = DeviceArray.from_numpy(
+        np.array([segment_size, floating_array.size], dtype=np.int32)
+    )
+
+    def sqrt(x: dtype) -> dtype:
+        return x**0.5
+
+    d_out_it = TransformOutputIterator(d_output, sqrt)
+
+    cuda.compute.segmented_reduce(
+        d_in=d_input,
+        d_out=d_out_it,
+        num_segments=2,
+        start_offsets_in=start_offsets,
+        end_offsets_in=end_offsets,
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+
+    expected = np.sqrt(
+        np.array(
+            [
+                np.sum(floating_array[:segment_size]),
+                np.sum(floating_array[segment_size:]),
+            ]
+        )
+    )
+    np.testing.assert_allclose(d_output.copy_to_host(), expected, atol=1e-6)
+
+
+def test_device_segmented_reduce_for_rowwise_sum(monkeypatch):
+    # Disable SASS verification for this test (LDL instruction in SASS).
+    monkeypatch.setattr(
+        cuda.compute._cccl_interop,
+        "_check_sass",
+        False,
+    )
+
+    def add_op(a, b):
+        return a + b
+
+    n_rows, n_cols = 67, 12345
+    rng = np.random.default_rng()
+    mat = rng.integers(low=-31, high=32, dtype=np.int32, size=(n_rows, n_cols))
+
+    def make_scaler(step):
+        def scale(row_id):
+            return row_id * step
+
+        return scale
+
+    zero = np.int32(0)
+    row_offset = make_scaler(np.int32(n_cols))
+    start_offsets = TransformIterator(CountingIterator(zero), row_offset)
+
+    end_offsets = start_offsets + 1
+
+    d_input = DeviceArray.from_numpy(mat)
+    h_init = np.zeros(tuple(), dtype=np.int32)
+    d_output = DeviceArray.empty(n_rows, mat.dtype)
+
+    cuda.compute.segmented_reduce(
+        d_in=d_input,
+        d_out=d_output,
+        num_segments=n_rows,
+        start_offsets_in=start_offsets,
+        end_offsets_in=end_offsets,
+        op=add_op,
+        h_init=h_init,
+    )
+
+    expected = np.sum(mat, axis=-1)
+    np.testing.assert_array_equal(d_output.copy_to_host(), expected)
+
+
+def test_segmented_reduce_with_lambda(monkeypatch):
+    """Test segmented_reduce with a lambda function as the reducer."""
+    # Disable SASS verification for this test (LDL instruction in SASS).
+    monkeypatch.setattr(
+        cuda.compute._cccl_interop,
+        "_check_sass",
+        False,
+    )
+    dtype = np.int32
+    h_init = np.array([0], dtype=dtype)
+
+    # Create segmented data: [1, 2, 3] | [4, 5] | [6, 7, 8, 9]
+    h_input = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=dtype)
+    h_starts = np.array([0, 3, 5], dtype=np.int32)
+    h_ends = np.array([3, 5, 9], dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_starts = DeviceArray.from_numpy(h_starts)
+    d_ends = DeviceArray.from_numpy(h_ends)
+    d_output = DeviceArray.empty(3, dtype)
+
+    # Use a lambda function directly as the reducer
+    cuda.compute.segmented_reduce(
+        d_in=d_input,
+        d_out=d_output,
+        num_segments=3,
+        start_offsets_in=d_starts,
+        end_offsets_in=d_ends,
+        op=lambda a, b: a + b,
+        h_init=h_init,
+    )
+
+    expected = np.array([6, 9, 30])  # sum of each segment
+    np.testing.assert_equal(d_output.copy_to_host(), expected)
+
+
+@pytest.mark.parametrize(
+    "max_seg_size",
+    [
+        4,  # small: warp-level reduction path
+        64,  # medium: warp-level reduction path
+        512,  # large: block-level reduction path
+    ],
+)
+def test_segmented_reduce_max_segment_size(max_seg_size, monkeypatch):
+    """Test that max_segment_size hint produces correct results with non-uniform segments.
+
+    max_segment_size is a performance hint that selects an optimized kernel
+    dispatch path. Segments vary in size from 1 to max_seg_size elements.
+    """
+    monkeypatch.setattr(
+        cuda.compute._cccl_interop,
+        "_check_sass",
+        False,
+    )
+    dtype = np.int32
+    rng = np.random.default_rng()
+    num_segments = 1024
+    h_init = np.zeros(1, dtype=dtype)
+
+    # Non-uniform segment sizes in [1, max_seg_size]
+    sizes = rng.integers(1, max_seg_size + 1, size=num_segments, dtype=np.int64)
+    offsets = np.zeros(num_segments + 1, dtype=np.int64)
+    offsets[1:] = np.cumsum(sizes)
+
+    total = int(offsets[-1])
+    h_input = rng.integers(0, 100, size=total, dtype=dtype)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(num_segments, dtype)
+
+    h_starts = offsets[:-1]
+    h_ends = offsets[1:]
+    d_starts = DeviceArray.from_numpy(h_starts)
+    d_ends = DeviceArray.from_numpy(h_ends)
+
+    cuda.compute.segmented_reduce(
+        d_in=d_input,
+        d_out=d_output,
+        num_segments=num_segments,
+        start_offsets_in=d_starts,
+        end_offsets_in=d_ends,
+        op=OpKind.PLUS,
+        h_init=h_init,
+        max_segment_size=max_seg_size,
+    )
+
+    expected = np.empty(num_segments, dtype=dtype)
+    for i in range(num_segments):
+        expected[i] = np.sum(h_input[h_starts[i] : h_ends[i]])
+
+    np.testing.assert_array_equal(d_output.copy_to_host(), expected)
+
+
+def _run(reducer, *, d_in, d_out, num_segments, start, end, op, h_init):
+    bytes_needed = reducer(
+        temp_storage=None,
+        d_in=d_in,
+        d_out=d_out,
+        num_segments=num_segments,
+        start_offsets_in=start,
+        end_offsets_in=end,
+        op=op,
+        h_init=h_init,
+    )
+    tmp = TempStorageBuffer(bytes_needed, None)
+    reducer(
+        temp_storage=tmp,
+        d_in=d_in,
+        d_out=d_out,
+        num_segments=num_segments,
+        start_offsets_in=start,
+        end_offsets_in=end,
+        op=op,
+        h_init=h_init,
+    )
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_segmented_reduce_round_trip():
+    h_in = np.array([8, 6, 7, 5, 3, 0, 9, -4, 3, 0, 1, 3, 1, 11, 25, 8], dtype=np.int32)
+    offsets = np.array([0, 7, 11, 16], dtype=np.int64)
+    d_in = DeviceArray.from_numpy(h_in)
+    start = DeviceArray.from_numpy(offsets[:-1])
+    end = DeviceArray.from_numpy(offsets[1:])
+    n_segments = offsets.size - 1
+    d_out = DeviceArray.empty(n_segments, np.int32)
+    h_init = np.array([0], dtype=np.int32)
+
+    builder = make_segmented_reduce(
+        d_in=d_in,
+        d_out=d_out,
+        start_offsets_in=start,
+        end_offsets_in=end,
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+    blob = serialize(builder)
+    assert len(blob) > 0
+
+    loaded = deserialize(blob)
+    _run(
+        loaded,
+        d_in=d_in,
+        d_out=d_out,
+        num_segments=n_segments,
+        start=start,
+        end=end,
+        op=OpKind.PLUS,
+        h_init=h_init,
+    )
+
+    expected = np.array(
+        [h_in[s:e].sum() for s, e in zip(offsets[:-1], offsets[1:])], dtype=np.int32
+    )
+    np.testing.assert_array_equal(d_out.copy_to_host(), expected)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_segmented_sort.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_segmented_sort.py
new file mode 100644
index 00000000..a011b05c
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_segmented_sort.py
@@ -0,0 +1,422 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from typing import Tuple
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray
+
+import cuda.compute
+from cuda.compute import SortOrder, deserialize, make_segmented_sort, serialize
+from cuda.compute._utils.temp_storage_buffer import TempStorageBuffer
+
+DTYPE_LIST = [
+    np.uint8,
+    np.int16,
+    np.uint32,
+    np.int64,
+    np.float16,
+    np.float32,
+    np.float64,
+]
+
+
+def get_mark_by_size(num_segments: int, segment_size: int):
+    num_items = num_segments * segment_size
+    return pytest.mark.large if num_items >= (1 << 20) else tuple()
+
+
+NUM_SEGMENTS_AND_SEGMENT_SIZES = [
+    (1, 1),
+    (1, 2048),
+    (13, 12),
+    (1024, 1024),
+    (2048, 1),
+    (2048, 13),
+]
+
+DTYPE_SEGMENT_PARAMS = [
+    pytest.param(dt, ns, ss, marks=get_mark_by_size(ns, ss))
+    for dt in DTYPE_LIST
+    for (ns, ss) in NUM_SEGMENTS_AND_SEGMENT_SIZES
+]
+
+
+def random_array(size, dtype, max_value=None) -> np.typing.NDArray:
+    rng = np.random.default_rng()
+    if np.isdtype(dtype, "integral"):
+        if max_value is None:
+            max_value = np.iinfo(dtype).max
+        return rng.integers(max_value, size=size, dtype=dtype)
+    elif np.isdtype(dtype, "real floating"):
+        return np.random.uniform(low=-10.0, high=10.0, size=size).astype(dtype)
+    else:
+        raise ValueError(f"Unsupported dtype {dtype}")
+
+
+def make_uniform_segments(
+    num_segments: int, segment_size: int
+) -> Tuple[np.ndarray, np.ndarray]:
+    start_offsets = np.arange(num_segments, dtype=np.int64) * segment_size
+    end_offsets = start_offsets + segment_size
+    return start_offsets, end_offsets
+
+
+def host_segmented_sort(
+    h_keys: np.ndarray,
+    h_vals: np.ndarray | None,
+    start_offsets: np.ndarray,
+    end_offsets: np.ndarray,
+    order: cuda.compute.SortOrder,
+) -> Tuple[np.ndarray, np.ndarray | None]:
+    assert start_offsets.shape == end_offsets.shape
+    keys = h_keys.copy()
+    vals = None if h_vals is None else h_vals.copy()
+
+    for s, e in zip(start_offsets, end_offsets):
+        if e <= s:
+            continue
+        if vals is None:
+            if order is cuda.compute.SortOrder.DESCENDING:
+                # stable descending
+                signed_dtype = (
+                    np.dtype(keys.dtype.name.replace("uint", "int"))
+                    if np.issubdtype(keys.dtype, np.unsignedinteger)
+                    else keys.dtype
+                )
+                idx = np.argsort(-keys[s:e].astype(signed_dtype), stable=True)
+            else:
+                idx = np.argsort(keys[s:e], stable=True)
+            keys[s:e] = keys[s:e][idx]
+        else:
+            # build pairs for stable sort
+            pairs = list(zip(keys[s:e], vals[s:e]))
+            if order is cuda.compute.SortOrder.DESCENDING:
+                pairs.sort(key=lambda kv: kv[0], reverse=True)
+            else:
+                pairs.sort(key=lambda kv: kv[0])
+            ks, vs = zip(*pairs) if pairs else ([], [])
+            keys[s:e] = np.array(ks, dtype=keys.dtype)
+            vals[s:e] = np.array(vs, dtype=vals.dtype)
+
+    return keys, vals
+
+
+@pytest.mark.parametrize("dtype, num_segments, segment_size", DTYPE_SEGMENT_PARAMS)
+def test_segmented_sort_keys(dtype, num_segments, segment_size, monkeypatch):
+    # Disable SASS verification only for this test when dtype is int64
+    if np.dtype(dtype) == np.dtype(np.int64):
+        monkeypatch.setattr(
+            cuda.compute._cccl_interop,
+            "_check_sass",
+            False,
+        )
+    order = cuda.compute.SortOrder.ASCENDING
+    num_items = num_segments * segment_size
+
+    h_in_keys = random_array(num_items, dtype, max_value=50)
+    start_offsets, end_offsets = make_uniform_segments(num_segments, segment_size)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_out_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype)
+    d_start_offsets = DeviceArray.from_numpy(start_offsets)
+    d_end_offsets = DeviceArray.from_numpy(end_offsets)
+
+    cuda.compute.segmented_sort(
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=None,
+        d_out_values=None,
+        num_items=num_items,
+        num_segments=num_segments,
+        start_offsets_in=d_start_offsets,
+        end_offsets_in=d_end_offsets,
+        order=order,
+    )
+
+    h_out_keys = d_out_keys.copy_to_host()
+    expected_keys, _ = host_segmented_sort(
+        h_in_keys, None, start_offsets, end_offsets, order
+    )
+
+    np.testing.assert_array_equal(h_out_keys, expected_keys)
+
+
+@pytest.mark.parametrize("dtype, num_segments, segment_size", DTYPE_SEGMENT_PARAMS)
+def test_segmented_sort_pairs(dtype, num_segments, segment_size):
+    order = cuda.compute.SortOrder.DESCENDING
+    num_items = num_segments * segment_size
+
+    h_in_keys = random_array(
+        num_items, dtype, max_value=50 if np.isdtype(dtype, "integral") else None
+    )
+    h_in_vals = random_array(num_items, np.float32)
+
+    start_offsets, end_offsets = make_uniform_segments(num_segments, segment_size)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_vals = DeviceArray.from_numpy(h_in_vals)
+    d_out_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype)
+    d_out_vals = DeviceArray.empty(h_in_vals.shape, h_in_vals.dtype)
+    d_start_offsets = DeviceArray.from_numpy(start_offsets)
+    d_end_offsets = DeviceArray.from_numpy(end_offsets)
+
+    cuda.compute.segmented_sort(
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_vals,
+        d_out_values=d_out_vals,
+        num_items=num_items,
+        num_segments=num_segments,
+        start_offsets_in=d_start_offsets,
+        end_offsets_in=d_end_offsets,
+        order=order,
+    )
+
+    h_out_keys = d_out_keys.copy_to_host()
+    h_out_vals = d_out_vals.copy_to_host()
+
+    expected_keys, expected_vals = host_segmented_sort(
+        h_in_keys, h_in_vals, start_offsets, end_offsets, order
+    )
+
+    np.testing.assert_array_equal(h_out_keys, expected_keys)
+    np.testing.assert_array_equal(h_out_vals, expected_vals)
+
+
+@pytest.mark.parametrize("dtype, num_segments, segment_size", DTYPE_SEGMENT_PARAMS)
+def test_segmented_sort_keys_double_buffer(dtype, num_segments, segment_size):
+    order = cuda.compute.SortOrder.ASCENDING
+    num_items = num_segments * segment_size
+
+    h_in_keys = random_array(num_items, dtype, max_value=20)
+    start_offsets, end_offsets = make_uniform_segments(num_segments, segment_size)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_tmp_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype)
+    d_start_offsets = DeviceArray.from_numpy(start_offsets)
+    d_end_offsets = DeviceArray.from_numpy(end_offsets)
+    keys_db = cuda.compute.DoubleBuffer(d_in_keys, d_tmp_keys)
+
+    cuda.compute.segmented_sort(
+        d_in_keys=keys_db,
+        d_out_keys=None,
+        d_in_values=None,
+        d_out_values=None,
+        num_items=num_items,
+        num_segments=num_segments,
+        start_offsets_in=d_start_offsets,
+        end_offsets_in=d_end_offsets,
+        order=order,
+    )
+
+    h_out_keys = keys_db.current().copy_to_host()
+    expected_keys, _ = host_segmented_sort(
+        h_in_keys, None, start_offsets, end_offsets, order
+    )
+    np.testing.assert_array_equal(h_out_keys, expected_keys)
+
+
+@pytest.mark.parametrize("dtype, num_segments, segment_size", DTYPE_SEGMENT_PARAMS)
+def test_segmented_sort_pairs_double_buffer(dtype, num_segments, segment_size):
+    order = cuda.compute.SortOrder.DESCENDING
+    num_items = num_segments * segment_size
+
+    h_in_keys = random_array(
+        num_items, dtype, max_value=50 if np.isdtype(dtype, "integral") else None
+    )
+    h_in_vals = random_array(num_items, np.float32)
+
+    start_offsets, end_offsets = make_uniform_segments(num_segments, segment_size)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_vals = DeviceArray.from_numpy(h_in_vals)
+    d_tmp_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype)
+    d_tmp_vals = DeviceArray.empty(h_in_vals.shape, h_in_vals.dtype)
+    d_start_offsets = DeviceArray.from_numpy(start_offsets)
+    d_end_offsets = DeviceArray.from_numpy(end_offsets)
+
+    keys_db = cuda.compute.DoubleBuffer(d_in_keys, d_tmp_keys)
+    vals_db = cuda.compute.DoubleBuffer(d_in_vals, d_tmp_vals)
+
+    cuda.compute.segmented_sort(
+        d_in_keys=keys_db,
+        d_out_keys=None,
+        d_in_values=vals_db,
+        d_out_values=None,
+        num_items=num_items,
+        num_segments=num_segments,
+        start_offsets_in=d_start_offsets,
+        end_offsets_in=d_end_offsets,
+        order=order,
+    )
+
+    h_out_keys = keys_db.current().copy_to_host()
+    h_out_vals = vals_db.current().copy_to_host()
+
+    expected_keys, expected_vals = host_segmented_sort(
+        h_in_keys, h_in_vals, start_offsets, end_offsets, order
+    )
+    np.testing.assert_array_equal(h_out_keys, expected_keys)
+    np.testing.assert_array_equal(h_out_vals, expected_vals)
+
+
+@pytest.mark.parametrize("num_segments", [20, 600])
+def test_segmented_sort_variable_segment_sizes(num_segments):
+    order = cuda.compute.SortOrder.ASCENDING
+    base_pattern = [
+        1,
+        5,
+        10,
+        20,
+        30,
+        50,
+        100,
+        3,
+        25,
+        600,
+        7,
+        18,
+        300,
+        4,
+        35,
+        9,
+        14,
+        700,
+        28,
+        11,
+    ]
+    segment_sizes = []
+    while len(segment_sizes) < num_segments:
+        remaining = num_segments - len(segment_sizes)
+        copy_count = min(remaining, len(base_pattern))
+        segment_sizes.extend(base_pattern[:copy_count])
+
+    start_offsets = np.zeros(num_segments, dtype=np.int64)
+    end_offsets = np.zeros(num_segments, dtype=np.int64)
+    current = 0
+    for i, sz in enumerate(segment_sizes):
+        start_offsets[i] = current
+        current += sz
+        end_offsets[i] = current
+    num_items = current
+
+    h_in_keys = random_array(num_items, np.int32, max_value=100)
+    h_in_vals = random_array(num_items, np.float32)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_vals = DeviceArray.from_numpy(h_in_vals)
+    d_out_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype)
+    d_out_vals = DeviceArray.empty(h_in_vals.shape, h_in_vals.dtype)
+    d_start_offsets = DeviceArray.from_numpy(start_offsets)
+    d_end_offsets = DeviceArray.from_numpy(end_offsets)
+
+    cuda.compute.segmented_sort(
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_vals,
+        d_out_values=d_out_vals,
+        num_items=num_items,
+        num_segments=num_segments,
+        start_offsets_in=d_start_offsets,
+        end_offsets_in=d_end_offsets,
+        order=order,
+    )
+
+    h_out_keys = d_out_keys.copy_to_host()
+    h_out_vals = d_out_vals.copy_to_host()
+    expected_keys, expected_vals = host_segmented_sort(
+        h_in_keys, h_in_vals, start_offsets, end_offsets, order
+    )
+
+    np.testing.assert_array_equal(h_out_keys, expected_keys)
+    np.testing.assert_array_equal(h_out_vals, expected_vals)
+
+
+def _run(
+    sorter,
+    *,
+    d_in_keys,
+    d_out_keys,
+    d_in_values,
+    d_out_values,
+    num_items,
+    num_segments,
+    start,
+    end,
+):
+    bytes_needed = sorter(
+        temp_storage=None,
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_values,
+        d_out_values=d_out_values,
+        num_items=num_items,
+        num_segments=num_segments,
+        start_offsets_in=start,
+        end_offsets_in=end,
+    )
+    tmp = TempStorageBuffer(bytes_needed, None)
+    sorter(
+        temp_storage=tmp,
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_values,
+        d_out_values=d_out_values,
+        num_items=num_items,
+        num_segments=num_segments,
+        start_offsets_in=start,
+        end_offsets_in=end,
+    )
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_segmented_sort_round_trip():
+    h_in_keys = np.array([9, 1, 5, 4, 2, 8, 7, 3, 6], dtype="int32")
+    h_in_vals = np.array([90, 10, 50, 40, 20, 80, 70, 30, 60], dtype="int32")
+    start_offsets = np.array([0, 3, 5], dtype=np.int64)
+    end_offsets = np.array([3, 5, 9], dtype=np.int64)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_vals = DeviceArray.from_numpy(h_in_vals)
+    d_out_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype)
+    d_out_vals = DeviceArray.empty(h_in_vals.shape, h_in_vals.dtype)
+    start = DeviceArray.from_numpy(start_offsets)
+    end = DeviceArray.from_numpy(end_offsets)
+
+    builder = make_segmented_sort(
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_vals,
+        d_out_values=d_out_vals,
+        start_offsets_in=start,
+        end_offsets_in=end,
+        order=SortOrder.ASCENDING,
+    )
+    blob = serialize(builder)
+    assert len(blob) > 0
+
+    loaded = deserialize(blob)
+    _run(
+        loaded,
+        d_in_keys=d_in_keys,
+        d_out_keys=d_out_keys,
+        d_in_values=d_in_vals,
+        d_out_values=d_out_vals,
+        num_items=h_in_keys.size,
+        num_segments=start_offsets.size,
+        start=start,
+        end=end,
+    )
+
+    expected_pairs = []
+    for s, e in zip(start_offsets, end_offsets):
+        expected_pairs.extend(
+            sorted(zip(h_in_keys[s:e], h_in_vals[s:e]), key=lambda kv: kv[0])
+        )
+    expected_keys = np.array([k for k, _ in expected_pairs], dtype=h_in_keys.dtype)
+    expected_vals = np.array([v for _, v in expected_pairs], dtype=h_in_vals.dtype)
+    np.testing.assert_array_equal(d_out_keys.copy_to_host(), expected_keys)
+    np.testing.assert_array_equal(d_out_vals.copy_to_host(), expected_vals)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_select.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_select.py
new file mode 100644
index 00000000..f9eeec57
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_select.py
@@ -0,0 +1,818 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray
+
+import cuda.compute
+from cuda.compute import (
+    CacheModifiedInputIterator,
+    ZipIterator,
+    deserialize,
+    gpu_struct,
+    make_select,
+    serialize,
+)
+from cuda.compute._utils.temp_storage_buffer import TempStorageBuffer
+
+DTYPE_LIST = [
+    np.uint8,
+    np.uint16,
+    np.uint32,
+    np.uint64,
+    np.int8,
+    np.int16,
+    np.int32,
+    np.int64,
+    np.float16,
+    np.float32,
+    np.float64,
+]
+
+
+def random_array(size, dtype, max_value=None) -> np.typing.NDArray:
+    rng = np.random.default_rng()
+    if np.isdtype(dtype, "integral"):
+        if max_value is None:
+            max_value = np.iinfo(dtype).max
+        return rng.integers(max_value, size=size, dtype=dtype)
+    elif np.isdtype(dtype, "real floating"):
+        if dtype == np.float16:  # Cannot generate float16 directly
+            return rng.random(size=size, dtype=np.float32).astype(dtype)
+        return rng.random(size=size, dtype=dtype)
+    else:
+        raise ValueError(f"Unsupported dtype {dtype}")
+
+
+select_params = [
+    (dt, 2**log_size) for dt in DTYPE_LIST for log_size in [2, 4, 6, 8, 10, 16, 20]
+]
+
+
+@pytest.fixture(scope="function", autouse=True)
+def disable_sass_check(monkeypatch):
+    monkeypatch.setattr(
+        cuda.compute._cccl_interop,
+        "_check_sass",
+        False,
+    )
+
+
+def _host_select(h_in: np.ndarray, cond):
+    # Vectorize condition to produce boolean mask
+    mask = np.vectorize(cond, otypes=[np.uint8])(h_in).astype(bool)
+    selected = h_in[mask]
+    return selected, np.int64(selected.size)
+
+
+def _read_count(array: DeviceArray) -> int:
+    return int(array.copy_to_host()[0])
+
+
+@pytest.mark.parametrize("dtype,num_items", select_params)
+def test_select_basic(dtype, num_items):
+    h_in = random_array(num_items, dtype, max_value=100)
+
+    def even_op(x):
+        return x % 2 == 0
+
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.uint64)
+
+    cuda.compute.select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=even_op,
+        num_items=num_items,
+    )
+
+    num_selected = _read_count(d_num_selected)
+    got = d_out.copy_to_host()[:num_selected]
+
+    expected, expected_count = _host_select(h_in, even_op)
+
+    assert num_selected == expected_count
+    assert np.array_equal(got, expected)
+
+
+@pytest.mark.parametrize("dtype,num_items", select_params)
+def test_select_greater_than(dtype, num_items):
+    h_in = random_array(num_items, dtype, max_value=100)
+
+    def greater_than_42(x):
+        return x > 42
+
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.uint64)
+
+    cuda.compute.select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=greater_than_42,
+        num_items=num_items,
+    )
+
+    num_selected = _read_count(d_num_selected)
+    got = d_out.copy_to_host()[:num_selected]
+
+    expected, expected_count = _host_select(h_in, greater_than_42)
+
+    assert num_selected == expected_count
+    assert np.array_equal(got, expected)
+
+
+@pytest.mark.parametrize("dtype", DTYPE_LIST)
+def test_select_all_pass(dtype):
+    num_items = 1000
+    h_in = random_array(num_items, dtype, max_value=100)
+
+    def always_true(x):
+        return True
+
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.uint64)
+
+    cuda.compute.select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=always_true,
+        num_items=num_items,
+    )
+
+    num_selected = _read_count(d_num_selected)
+    got = d_out.copy_to_host()[:num_selected]
+
+    assert num_selected == num_items
+    assert np.array_equal(got, h_in)
+
+
+@pytest.mark.parametrize("dtype", DTYPE_LIST)
+def test_select_none_pass(monkeypatch, dtype):
+    num_items = 1000
+    h_in = random_array(num_items, dtype, max_value=100)
+
+    def always_false(x):
+        return False
+
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.int32)
+
+    cuda.compute.select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=always_false,
+        num_items=num_items,
+    )
+
+    num_selected = _read_count(d_num_selected)
+
+    assert num_selected == 0
+
+
+def test_select_empty():
+    dtype = np.int32
+    num_items = 0
+    h_in = np.array([], dtype=dtype)
+
+    def even_op(x):
+        return x % 2 == 0
+
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.uint64)
+
+    cuda.compute.select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=even_op,
+        num_items=num_items,
+    )
+
+    num_selected = _read_count(d_num_selected)
+
+    assert num_selected == 0
+
+
+@pytest.mark.parametrize("dtype", DTYPE_LIST)
+def test_select_with_iterator(dtype):
+    num_items = 10000
+    h_in = random_array(num_items, dtype, max_value=100)
+
+    def less_than_50(x):
+        return x < 50
+
+    d_in = DeviceArray.from_numpy(h_in)
+    d_in_iter = CacheModifiedInputIterator(d_in, modifier="stream")
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.uint64)
+
+    cuda.compute.select(
+        d_in=d_in_iter,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=less_than_50,
+        num_items=num_items,
+    )
+
+    num_selected = _read_count(d_num_selected)
+    got = d_out.copy_to_host()[:num_selected]
+
+    expected, expected_count = _host_select(h_in, less_than_50)
+
+    assert num_selected == expected_count
+    assert np.array_equal(got, expected)
+
+
+@pytest.mark.parametrize("dtype", DTYPE_LIST)
+def test_select_object_api(dtype):
+    num_items = 10000
+    h_in = random_array(num_items, dtype, max_value=100)
+
+    def divisible_by_3(x):
+        return x % 3 == 0
+
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.uint64)
+
+    # Create select object
+    selector = cuda.compute.make_select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=divisible_by_3,
+    )
+
+    # Get temp storage size
+    temp_storage_bytes = selector(
+        temp_storage=None,
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=divisible_by_3,
+        num_items=num_items,
+    )
+
+    # Allocate temp storage
+    d_temp_storage = DeviceArray.empty(temp_storage_bytes, np.uint8)
+
+    # Execute select
+    selector(
+        temp_storage=d_temp_storage,
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=divisible_by_3,
+        num_items=num_items,
+    )
+
+    num_selected = _read_count(d_num_selected)
+    got = d_out.copy_to_host()[:num_selected]
+
+    expected, expected_count = _host_select(h_in, divisible_by_3)
+
+    assert num_selected == expected_count
+    assert np.array_equal(got, expected)
+
+
+@pytest.mark.parametrize("dtype", DTYPE_LIST)
+def test_select_reuse_object(dtype):
+    """Test that the select object can be reused multiple times with different inputs"""
+    num_items = 1000
+
+    def positive_op(x):
+        return x > 0
+
+    d_out = DeviceArray.empty(num_items, dtype)
+    d_num_selected = DeviceArray.empty(2, np.uint64)
+
+    # Create select object with initial input
+    h_in1 = random_array(num_items, dtype, max_value=100) - 50
+    d_in1 = DeviceArray.from_numpy(h_in1)
+    selector = cuda.compute.make_select(
+        d_in=d_in1,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=positive_op,
+    )
+
+    # First execution
+    temp_storage_bytes = selector(
+        temp_storage=None,
+        d_in=d_in1,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=positive_op,
+        num_items=num_items,
+    )
+    d_temp_storage = DeviceArray.empty(temp_storage_bytes, np.uint8)
+    selector(
+        temp_storage=d_temp_storage,
+        d_in=d_in1,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=positive_op,
+        num_items=num_items,
+    )
+
+    num_selected1 = _read_count(d_num_selected)
+    got1 = d_out.copy_to_host()[:num_selected1]
+    expected1, expected_count1 = _host_select(h_in1, positive_op)
+
+    assert num_selected1 == expected_count1
+    assert np.array_equal(got1, expected1)
+
+    # Reuse with different input
+    h_in2 = random_array(num_items, dtype, max_value=100) - 50
+    d_in2 = DeviceArray.from_numpy(h_in2)
+
+    selector(
+        temp_storage=d_temp_storage,
+        d_in=d_in2,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=positive_op,
+        num_items=num_items,
+    )
+
+    num_selected2 = _read_count(d_num_selected)
+    got2 = d_out.copy_to_host()[:num_selected2]
+    expected2, expected_count2 = _host_select(h_in2, positive_op)
+
+    assert num_selected2 == expected_count2
+    assert np.array_equal(got2, expected2)
+
+
+@pytest.mark.parametrize("dtype", [np.int32, np.float32])
+def test_select_with_struct(dtype):
+    """Test selection with custom struct types"""
+
+    @gpu_struct
+    class Point:
+        x: dtype
+        y: dtype
+
+    num_items = 1000
+    h_x = random_array(num_items, dtype, max_value=100)
+    h_y = random_array(num_items, dtype, max_value=100)
+
+    h_in = np.empty(num_items, dtype=Point.dtype)
+    h_in["x"] = h_x
+    h_in["y"] = h_y
+
+    def in_first_quadrant(p: Point) -> np.uint8:
+        return (p.x > 50) and (p.y > 50)
+
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.uint64)
+
+    cuda.compute.select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=in_first_quadrant,
+        num_items=num_items,
+    )
+
+    num_selected = _read_count(d_num_selected)
+    got = d_out.copy_to_host()[:num_selected]
+
+    # Host reference
+    def host_in_first_quadrant(p):
+        return (p[0] > 50) and (p[1] > 50)
+
+    expected, expected_count = _host_select(h_in, host_in_first_quadrant)
+
+    assert num_selected == expected_count
+    assert np.array_equal(got["x"], expected["x"])
+    assert np.array_equal(got["y"], expected["y"])
+
+
+def test_select_with_zip_iterator(monkeypatch):
+    """Test select with ZipIterator input and output"""
+
+    dtype = np.int32
+    num_items = 10_000
+
+    # Create two arrays
+    h_in1 = random_array(num_items, dtype, max_value=100)
+    h_in2 = random_array(num_items, dtype, max_value=100)
+
+    # Select condition: sum of elements < 70
+    def condition(pair):
+        return (pair[0] + pair[1]) < 70
+
+    # Device arrays
+    d_in1 = DeviceArray.from_numpy(h_in1)
+    d_in2 = DeviceArray.from_numpy(h_in2)
+
+    # Create zip iterator for input
+    zip_in = ZipIterator(d_in1, d_in2)
+
+    # Allocate output arrays
+    d_out1 = DeviceArray.empty(h_in1.shape, h_in1.dtype)
+    d_out2 = DeviceArray.empty(h_in2.shape, h_in2.dtype)
+
+    # Create zip iterator for output
+    zip_out = ZipIterator(d_out1, d_out2)
+    d_num_selected = DeviceArray.empty(1, np.int32)
+
+    cuda.compute.select(
+        d_in=zip_in,
+        d_out=zip_out,
+        d_num_selected_out=d_num_selected,
+        cond=condition,
+        num_items=num_items,
+    )
+
+    num_selected = _read_count(d_num_selected)
+
+    # Get results
+    got1 = d_out1.copy_to_host()[:num_selected]
+    got2 = d_out2.copy_to_host()[:num_selected]
+
+    # Verify results: all elements should satisfy the condition
+    for i in range(num_selected):
+        assert (got1[i] + got2[i]) < 70
+
+    # Verify count
+    h_sums = h_in1 + h_in2
+    expected_count = np.sum(h_sums < 70)
+
+    assert num_selected == expected_count
+
+
+def test_select_stateful_threshold():
+    """Test stateful select that uses state for threshold"""
+    num_items = 1000
+    h_in = random_array(num_items, np.int32, max_value=100)
+
+    # Create device state containing threshold value
+    threshold_value = 50
+    threshold_state = DeviceArray.from_numpy(
+        np.array([threshold_value], dtype=np.int32)
+    )
+
+    # Define condition that references state as closure
+    def threshold_select(x):
+        return x > threshold_state[0]
+
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.uint64)
+
+    cuda.compute.select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=threshold_select,
+        num_items=num_items,
+    )
+
+    # Check selected output
+    num_selected = _read_count(d_num_selected)
+    got = d_out.copy_to_host()[:num_selected]
+
+    # Verify all output values are > threshold
+    assert np.all(got > threshold_value)
+
+    # Verify we got the expected number of items
+    expected_selected = h_in[h_in > threshold_value]
+    expected_count = len(expected_selected)
+
+    assert num_selected == expected_count
+
+    # Verify exact results
+    assert np.array_equal(got, expected_selected)
+
+
+def test_select_stateful_atomic():
+    """Test stateful select with atomic operations to count rejected items"""
+    from numba import cuda as numba_cuda
+
+    num_items = 1000
+    h_in = random_array(num_items, np.int32, max_value=100)
+
+    # Create device state for counting rejected items
+    reject_counter = DeviceArray.from_numpy(np.zeros(1, dtype=np.int32))
+
+    # Define condition that references state as closure
+    def count_rejects(x):
+        if x > 50:
+            return True
+        else:
+            numba_cuda.atomic.add(reject_counter, 0, 1)
+            return False
+
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.uint64)
+
+    cuda.compute.select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=count_rejects,
+        num_items=num_items,
+    )
+
+    # Check selected output
+    num_selected = _read_count(d_num_selected)
+    got = d_out.copy_to_host()[:num_selected]
+
+    # Verify all output values are > 50
+    assert np.all(got > 50)
+
+    # Verify we got the expected number of items
+    expected_selected = h_in[h_in > 50]
+    expected_count = len(expected_selected)
+
+    assert num_selected == expected_count
+
+    # Verify exact results
+    assert np.array_equal(got, expected_selected)
+
+    # Verify state contains count of rejected items
+    rejected_count = _read_count(reject_counter)
+    expected_rejected = len(h_in[h_in <= 50])
+    assert rejected_count == expected_rejected, (
+        f"Expected {expected_rejected} rejections, got {rejected_count}"
+    )
+
+
+def test_select_with_side_effect_counting_rejects():
+    """Select with side effect that counts rejected items"""
+    from numba import cuda as numba_cuda
+
+    h_in = np.arange(100, dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(1, np.uint64)
+
+    reject_count = DeviceArray.from_numpy(np.zeros(1, dtype=np.int32))
+
+    # Define condition that references state as closure
+    def count_rejects(x):
+        if x >= 50:
+            return True
+        else:
+            numba_cuda.atomic.add(reject_count, 0, 1)
+            return False
+
+    cuda.compute.select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=count_rejects,
+        num_items=h_in.size,
+    )
+
+    num_selected = _read_count(d_num_selected)
+    num_rejected = _read_count(reject_count)
+
+    assert num_selected == 50  # Values 50-99
+    assert num_rejected == 50  # Values 0-49
+
+
+def test_select_with_lambda():
+    """Test select with a lambda function as predicate."""
+    num_items = 100
+    h_in = np.arange(num_items, dtype=np.int32)
+
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.uint64)
+
+    # Use a lambda function directly as the predicate
+    cuda.compute.select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_num_selected,
+        cond=lambda x: x % 2 == 0,
+        num_items=num_items,
+    )
+
+    num_selected = _read_count(d_num_selected)
+    expected_selected = [x for x in h_in if x % 2 == 0]
+
+    assert num_selected == len(expected_selected)
+    np.testing.assert_array_equal(
+        d_out.copy_to_host()[:num_selected], expected_selected
+    )
+
+
+def test_select_stateful_state_updates():
+    """Test that select correctly updates state between calls with different thresholds."""
+    num_items = 20
+    h_in = np.arange(num_items, dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_count = DeviceArray.from_numpy(np.zeros(2, dtype=np.uint64))
+
+    # Create two different thresholds
+    threshold_5 = DeviceArray.from_numpy(np.array([5], dtype=np.int32))
+    threshold_15 = DeviceArray.from_numpy(np.array([15], dtype=np.int32))
+
+    # Call 1: Select items > 5 (should get 14 items: 6-19)
+    def select_gt_5(x):
+        return x > threshold_5[0]
+
+    cuda.compute.select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_count,
+        cond=select_gt_5,
+        num_items=num_items,
+    )
+    count1 = _read_count(d_count)
+    assert count1 == 14
+    expected_1 = list(range(6, 20))
+    np.testing.assert_array_equal(d_out.copy_to_host()[:count1], expected_1)
+
+    # Call 2: Select items > 15 (should get 4 items: 16-19)
+    def select_gt_15(x):
+        return x > threshold_15[0]
+
+    d_count.copy_from_host(np.zeros(2, dtype=np.uint64))
+    cuda.compute.select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_count,
+        cond=select_gt_15,
+        num_items=num_items,
+    )
+    count2 = _read_count(d_count)
+    assert count2 == 4
+    expected_2 = list(range(16, 20))
+    np.testing.assert_array_equal(d_out.copy_to_host()[:count2], expected_2)
+
+    # Call 3: Back to first threshold (test cache reuse with updated state)
+    d_count.copy_from_host(np.zeros(2, dtype=np.uint64))
+    cuda.compute.select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_count,
+        cond=select_gt_5,
+        num_items=num_items,
+    )
+    count3 = _read_count(d_count)
+    assert count3 == 14
+    np.testing.assert_array_equal(d_out.copy_to_host()[:count3], expected_1)
+
+
+def test_select_stateful_same_bytecode_different_state():
+    """
+    Test that select works correctly when using factory functions that produce
+    identical bytecode but capture different state arrays.
+
+    This is a regression test for the cache collision bug where functions with
+    the same bytecode but different captured arrays would reuse stale state.
+    """
+    num_items = 20
+    h_in = np.arange(num_items, dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_count = DeviceArray.from_numpy(np.zeros(2, dtype=np.uint64))
+
+    # Factory that creates functions with identical bytecode
+    def make_selector(threshold_array):
+        def selector(x):
+            return x > threshold_array[0]
+
+        return selector
+
+    threshold_5 = DeviceArray.from_numpy(np.array([5], dtype=np.int32))
+    threshold_15 = DeviceArray.from_numpy(np.array([15], dtype=np.int32))
+
+    select_5 = make_selector(threshold_5)
+    select_15 = make_selector(threshold_15)
+
+    # Call 1: threshold > 5
+    cuda.compute.select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_count,
+        cond=select_5,
+        num_items=num_items,
+    )
+    count1 = _read_count(d_count)
+    assert count1 == 14
+
+    # Call 2: threshold > 15 (different state, same bytecode)
+    d_count.copy_from_host(np.zeros(2, dtype=np.uint64))
+    cuda.compute.select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_count,
+        cond=select_15,
+        num_items=num_items,
+    )
+    count2 = _read_count(d_count)
+    assert count2 == 4  # If this fails, cache collision bug is present
+
+
+def test_stateful_caching_same_dtype_different_values():
+    """
+    Test that stateful ops with same dtype but different values work correctly.
+    After transformation, values are runtime parameters, so they should use the
+    same compiled code.
+    """
+    num_items = 100
+    h_in = np.arange(num_items, dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_count = DeviceArray.from_numpy(np.zeros(2, dtype=np.uint64))
+
+    # Two thresholds with SAME dtype, SAME size, DIFFERENT values
+    threshold_30 = DeviceArray.from_numpy(np.array([30], dtype=np.int32))
+    threshold_70 = DeviceArray.from_numpy(np.array([70], dtype=np.int32))
+
+    # Test with threshold_30
+    def select_gt_30(x):
+        return x > threshold_30[0]
+
+    cuda.compute.select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_count,
+        cond=select_gt_30,
+        num_items=num_items,
+    )
+    count_30 = _read_count(d_count)
+
+    # Test with threshold_70
+    def select_gt_70(x):
+        return x > threshold_70[0]
+
+    d_out.copy_from_host(np.zeros_like(h_in))
+    d_count.copy_from_host(np.zeros(2, dtype=np.uint64))
+    cuda.compute.select(
+        d_in=d_in,
+        d_out=d_out,
+        d_num_selected_out=d_count,
+        cond=select_gt_70,
+        num_items=num_items,
+    )
+    count_70 = _read_count(d_count)
+
+    # Verify correct results (not cache collision)
+    assert count_30 == 69  # Values 31-99
+    assert count_70 == 29  # Values 71-99
+
+
+def _even(x):
+    return x % 2 == 0
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_select_round_trip():
+    n = 1024
+    h_in = np.arange(n, dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.uint64)
+
+    builder = make_select(
+        d_in=d_in, d_out=d_out, d_num_selected_out=d_num_selected, cond=_even
+    )
+    blob = serialize(builder)
+    assert len(blob) > 0
+
+    loaded = deserialize(blob)
+
+    def _run():
+        nbytes = loaded(
+            temp_storage=None,
+            d_in=d_in,
+            d_out=d_out,
+            d_num_selected_out=d_num_selected,
+            cond=_even,
+            num_items=n,
+        )
+        loaded(
+            temp_storage=TempStorageBuffer(nbytes, None),
+            d_in=d_in,
+            d_out=d_out,
+            d_num_selected_out=d_num_selected,
+            cond=_even,
+            num_items=n,
+        )
+
+    _run()
+
+    k = _read_count(d_num_selected)
+    expected = h_in[h_in % 2 == 0]
+    assert k == expected.size
+    np.testing.assert_array_equal(d_out.copy_to_host()[:k], expected)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_serialization_diagnostics.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_serialization_diagnostics.py
new file mode 100644
index 00000000..f489957f
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_serialization_diagnostics.py
@@ -0,0 +1,94 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""Tests for serialization deserialize diagnostics.
+
+A deserialize/load failure should carry an actionable message, not an opaque
+``error code: <n>``. The C layer validates the blob header magic and, for CUBIN
+payloads, the target compute-capability major against the current device before
+the opaque ``cuLibraryLoadData`` failure, and propagates a descriptive message
+via ``cccl_serialization_last_error()``.
+
+The compute-capability case is exercised single-GPU by patching the ``cc`` field
+of a real blob (a true cross-GPU load is the same code path but needs a second
+architecture).
+"""
+
+import struct
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray, get_compute_capability
+
+from cuda.compute import OpKind, deserialize, make_reduce_into
+
+try:
+    from cuda.compute._build_info import USING_V2
+except ImportError:
+    USING_V2 = False
+
+pytestmark = pytest.mark.skipif(
+    USING_V2, reason="serialization not supported on v2 (HostJIT) backend"
+)
+
+_C_MAGIC = b"CCCLSER1"
+# Field offsets within the C build_result header, past the 8-byte magic:
+#   algo_tag u32 | payload_kind u32 | cc u32
+_OFF_CC = 4 + 4
+
+
+def _reduce_blob():
+    d = DeviceArray.from_numpy(np.arange(1024, dtype=np.int32))
+    o = DeviceArray.from_numpy(np.zeros(1, dtype=np.int32))
+    r = make_reduce_into(d_in=d, d_out=o, op=OpKind.PLUS, h_init=np.zeros(1, np.int32))
+    return r.serialize()
+
+
+def _patch_u32(blob, field_off, value):
+    b = bytearray(blob)
+    i = b.find(_C_MAGIC)
+    struct.pack_into("<I", b, i + len(_C_MAGIC) + field_off, value)
+    return bytes(b)
+
+
+def test_wrong_cc_major_reports_clear_message():
+    maj = int(get_compute_capability()[0])
+    other_major = 7 if maj != 7 else 8
+    bad = _patch_u32(_reduce_blob(), _OFF_CC, other_major * 10 + 5)
+    with pytest.raises(RuntimeError, match="compute-capability major"):
+        deserialize(bad)
+
+
+def test_wrong_cc_minor_reports_clear_message():
+    # A CUBIN built for a higher minor than the device is not backward-compatible
+    # (SASS is only forward-compatible across minors within a major), so a blob
+    # targeting the same major but a higher minor must be rejected.
+    maj, minr = (int(part) for part in get_compute_capability())
+    if minr >= 9:
+        pytest.skip("device minor is already the maximum within its major")
+    bad = _patch_u32(_reduce_blob(), _OFF_CC, maj * 10 + (minr + 1))
+    with pytest.raises(RuntimeError, match="device minor"):
+        deserialize(bad)
+
+
+def test_bad_magic_reports_clear_message():
+    b = bytearray(_reduce_blob())
+    i = b.find(_C_MAGIC)
+    b[i : i + len(_C_MAGIC)] = b"XXXXXXXX"  # corrupt only the C header magic
+    with pytest.raises(RuntimeError, match="bad magic"):
+        deserialize(bytes(b))
+
+
+def test_valid_blob_still_roundtrips():
+    from cuda.compute._utils.temp_storage_buffer import TempStorageBuffer
+
+    reducer = deserialize(_reduce_blob())
+    d = DeviceArray.from_numpy(np.arange(1024, dtype=np.int32))
+    o = DeviceArray.from_numpy(np.zeros(1, dtype=np.int32))
+    kw = dict(
+        d_in=d, d_out=o, num_items=1024, op=OpKind.PLUS, h_init=np.zeros(1, np.int32)
+    )
+    nbytes = reducer(temp_storage=None, **kw)
+    reducer(temp_storage=TempStorageBuffer(nbytes, None), **kw)
+    assert int(o.copy_to_host()[0]) == 1024 * 1023 // 2
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_shuffle_iterator.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_shuffle_iterator.py
new file mode 100644
index 00000000..2d442abc
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_shuffle_iterator.py
@@ -0,0 +1,96 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray
+
+import cuda.compute
+from cuda.compute.iterators import (
+    PermutationIterator,
+    ShuffleIterator,
+)
+
+
+def test_shuffle_iterator_bijectivity():
+    num_items = 100
+    seed = 42
+
+    shuffle_it = ShuffleIterator(num_items, seed)
+
+    d_output = DeviceArray.empty(num_items, np.int64)
+    cuda.compute.unary_transform(
+        d_in=shuffle_it, d_out=d_output, op=lambda x: x, num_items=num_items
+    )
+
+    result = d_output.copy_to_host()
+
+    assert len(set(result)) == num_items
+    assert set(result) == set(range(num_items))
+
+
+def test_shuffle_iterator_determinism():
+    num_items = 50
+    seed = 12345
+
+    shuffle_it1 = ShuffleIterator(num_items, seed)
+    shuffle_it2 = ShuffleIterator(num_items, seed)
+
+    d_output1 = DeviceArray.empty(num_items, np.int64)
+    d_output2 = DeviceArray.empty(num_items, np.int64)
+
+    cuda.compute.unary_transform(
+        d_in=shuffle_it1, d_out=d_output1, op=lambda x: x, num_items=num_items
+    )
+    cuda.compute.unary_transform(
+        d_in=shuffle_it2, d_out=d_output2, op=lambda x: x, num_items=num_items
+    )
+
+    np.testing.assert_array_equal(d_output1.copy_to_host(), d_output2.copy_to_host())
+
+
+@pytest.mark.parametrize("num_items", [1, 2, 7, 16, 17, 100, 1000, 1023, 1024, 1025])
+def test_shuffle_iterator_various_sizes(num_items):
+    seed = 42
+
+    shuffle_it = ShuffleIterator(num_items, seed)
+
+    d_output = DeviceArray.empty(num_items, np.int64)
+    cuda.compute.unary_transform(
+        d_in=shuffle_it, d_out=d_output, op=lambda x: x, num_items=num_items
+    )
+
+    result = d_output.copy_to_host()
+
+    assert len(set(result)) == num_items
+    assert set(result) == set(range(num_items))
+
+
+def test_shuffle_iterator_with_permutation_iterator():
+    num_items = 10
+    seed = 42
+
+    h_values = np.asarray([10, 20, 30, 40, 50, 60, 70, 80, 90, 100], dtype=np.int32)
+    d_values = DeviceArray.from_numpy(h_values)
+
+    shuffle_it = ShuffleIterator(num_items, seed)
+    perm_it = PermutationIterator(d_values, shuffle_it)
+
+    d_output = DeviceArray.empty(num_items, np.int32)
+    cuda.compute.unary_transform(
+        d_in=perm_it, d_out=d_output, op=lambda x: x, num_items=num_items
+    )
+
+    result = d_output.copy_to_host()
+
+    assert result.sum() == h_values.sum()
+    assert sorted(result) == sorted(h_values)
+
+
+def test_shuffle_iterator_invalid_num_items():
+    with pytest.raises(ValueError, match="num_items must be > 0"):
+        ShuffleIterator(0, seed=42)
+
+    with pytest.raises(ValueError, match="num_items must be > 0"):
+        ShuffleIterator(-1, seed=42)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_struct_field_validation.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_struct_field_validation.py
new file mode 100644
index 00000000..3ddaee54
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_struct_field_validation.py
@@ -0,0 +1,66 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Tests for gpu_struct field-name validation.
+
+These tests verify that invalid field names are rejected before
+they can reach the exec() calls in _jit.py.
+"""
+
+import numpy as np
+import pytest
+
+from cuda.compute import gpu_struct
+
+
+def test_newline_in_field_name_is_rejected():
+    """Field names with newlines must be rejected — they are the exec() injection vector."""
+    payload = "x\n    else: pass\nprint('injected')\ndef _absorb():\n    if True:\n        pass\n"
+    with pytest.raises(ValueError, match="not a valid Python identifier"):
+        gpu_struct({payload: np.float32, "y": np.float32})
+
+
+def test_space_in_field_name_is_rejected():
+    with pytest.raises(ValueError, match="not a valid Python identifier"):
+        gpu_struct({"field name": np.float32})
+
+
+def test_hyphen_in_field_name_is_rejected():
+    with pytest.raises(ValueError, match="not a valid Python identifier"):
+        gpu_struct({"field-name": np.float32})
+
+
+def test_semicolon_in_field_name_is_rejected():
+    with pytest.raises(ValueError, match="not a valid Python identifier"):
+        gpu_struct({"a; import os": np.float32})
+
+
+def test_non_string_key_is_rejected():
+    with pytest.raises(ValueError, match="not a valid Python identifier"):
+        gpu_struct({0: np.float32})
+
+
+def test_empty_string_field_name_is_rejected():
+    with pytest.raises(ValueError, match="not a valid Python identifier"):
+        gpu_struct({"": np.float32})
+
+
+def test_valid_field_names_are_accepted():
+    """Sanity check: ordinary identifiers must still work."""
+    S = gpu_struct(
+        {"x": np.float32, "y": np.float32, "_z": np.int32, "field1": np.int64}
+    )
+    obj = S(1.0, 2.0, 3, 4)
+    assert obj.x == np.float32(1.0)
+    assert obj.y == np.float32(2.0)
+    assert obj._z == np.int32(3)
+    assert obj.field1 == np.int64(4)
+
+
+def test_unicode_identifier_is_accepted():
+    """Python allows unicode identifiers; they should be valid field names."""
+    S = gpu_struct({"α": np.float32, "β": np.float32})
+    obj = S(1.0, 2.0)
+    assert obj.α == np.float32(1.0)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_three_way_partition.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_three_way_partition.py
new file mode 100644
index 00000000..91974b19
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_three_way_partition.py
@@ -0,0 +1,548 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray
+
+import cuda.compute
+from cuda.compute import (
+    CacheModifiedInputIterator,
+    deserialize,
+    gpu_struct,
+    make_three_way_partition,
+    serialize,
+)
+from cuda.compute._utils.temp_storage_buffer import TempStorageBuffer
+
+DTYPE_LIST = [
+    np.uint8,
+    np.uint16,
+    np.uint32,
+    np.uint64,
+    np.int8,
+    np.int16,
+    np.int32,
+    np.int64,
+    np.float16,
+    np.float32,
+    np.float64,
+]
+
+
+def random_array(size, dtype, max_value=None) -> np.typing.NDArray:
+    rng = np.random.default_rng()
+    if np.isdtype(dtype, "integral"):
+        if max_value is None:
+            max_value = np.iinfo(dtype).max
+        return rng.integers(max_value, size=size, dtype=dtype)
+    elif np.isdtype(dtype, "real floating"):
+        if dtype == np.float16:  # Cannot generate float16 directly
+            return rng.random(size=size, dtype=np.float32).astype(dtype)
+        return rng.random(size=size, dtype=dtype)
+    else:
+        raise ValueError(f"Unsupported dtype {dtype}")
+
+
+three_way_partition_params = [
+    (dt, 2**log_size) for dt in DTYPE_LIST for log_size in [2, 4, 6, 8, 10, 16, 20]
+]
+
+
+def _host_three_way_partition(h_in: np.ndarray, less_than_op, greater_equal_op):
+    # Vectorize ops to produce boolean masks
+    first_mask = np.vectorize(less_than_op, otypes=[np.uint8])(h_in).astype(bool)
+    remaining = h_in[~first_mask]
+    second_mask = np.vectorize(greater_equal_op, otypes=[np.uint8])(remaining).astype(
+        bool
+    )
+
+    first_part = h_in[first_mask]
+    second_part = remaining[second_mask]
+    unselected = remaining[~second_mask]
+
+    return (
+        first_part,
+        second_part,
+        unselected,
+        np.int64(first_part.size),
+        np.int64(second_part.size),
+        np.int64(unselected.size),
+    )
+
+
+@pytest.mark.parametrize("dtype,num_items", three_way_partition_params)
+def test_three_way_partition_basic(dtype, num_items, monkeypatch):
+    # NOTE: the SASS check failure is seen only with NVRTC 13.1:
+    if np.isdtype(dtype, np.float16):
+        import cuda.compute._cccl_interop as cccl_interop
+
+        monkeypatch.setattr(
+            cccl_interop,
+            "_check_sass",
+            False,
+        )
+
+    h_in = random_array(num_items, dtype, max_value=100)
+
+    def less_than_op(x):
+        return x < 42
+
+    def greater_equal_op(x):
+        return x >= 42
+
+    d_in = DeviceArray.from_numpy(h_in)
+    d_first = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_second = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_unselected = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.int32)
+    cuda.compute.three_way_partition(
+        d_in=d_in,
+        d_first_part_out=d_first,
+        d_second_part_out=d_second,
+        d_unselected_out=d_unselected,
+        d_num_selected_out=d_num_selected,
+        select_first_part_op=less_than_op,
+        select_second_part_op=greater_equal_op,
+        num_items=num_items,
+    )
+
+    num_selected = d_num_selected.copy_to_host()
+    got_first = d_first.copy_to_host()[: int(num_selected[0])]
+    got_second = d_second.copy_to_host()[: int(num_selected[1])]
+    got_unselected = d_unselected.copy_to_host()[
+        : int(num_items) - int(num_selected[0]) - int(num_selected[1])
+    ]
+
+    expected_first, expected_second, expected_unselected, n1, n2, n3 = (
+        _host_three_way_partition(h_in, less_than_op, greater_equal_op)
+    )
+
+    np.testing.assert_array_equal(got_first, expected_first)
+    np.testing.assert_array_equal(got_second, expected_second)
+    np.testing.assert_array_equal(got_unselected, expected_unselected)
+    assert num_selected[0] == n1 and num_selected[1] == n2
+
+
+def test_three_way_partition_well_known_logical_not():
+    h_in = np.array([True, False, False, True, True, False], dtype=np.bool_)
+    num_items = h_in.size
+
+    d_in = DeviceArray.from_numpy(h_in)
+    d_first = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_second = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_unselected = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.int64)
+
+    cuda.compute.three_way_partition(
+        d_in=d_in,
+        d_first_part_out=d_first,
+        d_second_part_out=d_second,
+        d_unselected_out=d_unselected,
+        d_num_selected_out=d_num_selected,
+        select_first_part_op=cuda.compute.OpKind.IDENTITY,
+        select_second_part_op=cuda.compute.OpKind.LOGICAL_NOT,
+        num_items=num_items,
+    )
+
+    num_selected = d_num_selected.copy_to_host()
+    num_unselected = num_items - int(num_selected.sum())
+
+    np.testing.assert_array_equal(num_selected, np.array([3, 3], dtype=np.int64))
+    np.testing.assert_array_equal(d_first.copy_to_host()[: num_selected[0]], h_in[h_in])
+    np.testing.assert_array_equal(
+        d_second.copy_to_host()[: num_selected[1]], h_in[~h_in]
+    )
+    np.testing.assert_array_equal(
+        d_unselected.copy_to_host()[:num_unselected], np.empty(0, dtype=np.bool_)
+    )
+    assert num_unselected == 0
+
+
+def test_three_way_partition_empty():
+    dtype = np.int32
+    d_in = DeviceArray.empty(0, dtype)
+    d_first = DeviceArray.empty(0, dtype)
+    d_second = DeviceArray.empty(0, dtype)
+    d_unselected = DeviceArray.empty(0, dtype)
+    d_num_selected = DeviceArray.from_numpy(np.zeros(2, dtype=np.int64))
+
+    def less_than_op(x):
+        return x < 42
+
+    def greater_equal_op(x):
+        return x >= 42
+
+    cuda.compute.three_way_partition(
+        d_in=d_in,
+        d_first_part_out=d_first,
+        d_second_part_out=d_second,
+        d_unselected_out=d_unselected,
+        d_num_selected_out=d_num_selected,
+        select_first_part_op=less_than_op,
+        select_second_part_op=greater_equal_op,
+        num_items=0,
+    )
+
+    np.testing.assert_array_equal(d_num_selected.copy_to_host(), np.array([0, 0]))
+
+
+def test_three_way_partition_with_iterators():
+    dtype = np.int32
+    num_items = 10_000
+    h_in = random_array(num_items, dtype, max_value=100)
+
+    def less_than_op(x):
+        return np.uint8(x < 42)
+
+    def greater_equal_op(x):
+        return np.uint8(x >= 42)
+
+    expected_first, expected_second, expected_unselected, _, _, _ = (
+        _host_three_way_partition(h_in, less_than_op, greater_equal_op)
+    )
+
+    d_in = DeviceArray.from_numpy(h_in)
+    in_it = CacheModifiedInputIterator(d_in, modifier="stream")
+
+    d_first = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_second = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_unselected = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.uint32)
+
+    cuda.compute.three_way_partition(
+        d_in=in_it,
+        d_first_part_out=d_first,
+        d_second_part_out=d_second,
+        d_unselected_out=d_unselected,
+        d_num_selected_out=d_num_selected,
+        select_first_part_op=less_than_op,
+        select_second_part_op=greater_equal_op,
+        num_items=num_items,
+    )
+
+    num_selected = d_num_selected.copy_to_host()
+    got_first = d_first.copy_to_host()[: int(num_selected[0])]
+    got_second = d_second.copy_to_host()[: int(num_selected[1])]
+    got_unselected = d_unselected.copy_to_host()[
+        : int(num_items) - int(num_selected[0]) - int(num_selected[1])
+    ]
+
+    np.testing.assert_array_equal(got_first, expected_first)
+    np.testing.assert_array_equal(got_second, expected_second)
+    np.testing.assert_array_equal(got_unselected, expected_unselected)
+
+
+def test_three_way_partition_struct_type():
+    @gpu_struct
+    class pair_type:
+        a: np.int32
+        b: np.uint64
+
+    comparison_value = np.int32(42)
+
+    def less_than_op(x: pair_type):
+        return (x.a < 42) & (x.b < 42)
+
+    def greater_equal_op(x: pair_type):
+        return (x.a >= 42) & (x.b >= 42)
+
+    num_items = 20_000
+    a_vals = random_array(num_items, np.int32, max_value=100)
+    b_vals = a_vals.astype(np.uint64)
+
+    h_in = np.empty(num_items, dtype=pair_type.dtype)
+    h_in["a"] = a_vals
+    h_in["b"] = b_vals
+
+    expected_first_mask = (a_vals < comparison_value) & (b_vals < comparison_value)
+    remaining_mask = ~expected_first_mask
+    expected_second_mask = (a_vals[remaining_mask] >= comparison_value) & (
+        b_vals[remaining_mask] >= comparison_value
+    )
+
+    expected_first = h_in[expected_first_mask]
+    expected_second = h_in[remaining_mask][expected_second_mask]
+    expected_unselected = h_in[remaining_mask][~expected_second_mask]
+
+    d_in = DeviceArray.from_numpy(h_in)
+    d_first = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_second = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_unselected = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.uint64)
+
+    cuda.compute.three_way_partition(
+        d_in=d_in,
+        d_first_part_out=d_first,
+        d_second_part_out=d_second,
+        d_unselected_out=d_unselected,
+        d_num_selected_out=d_num_selected,
+        select_first_part_op=less_than_op,
+        select_second_part_op=greater_equal_op,
+        num_items=num_items,
+    )
+
+    num_selected = d_num_selected.copy_to_host()
+    got_first = d_first.copy_to_host()[: int(num_selected[0])]
+    got_second = d_second.copy_to_host()[: int(num_selected[1])]
+    got_unselected = d_unselected.copy_to_host()[
+        : int(num_items) - int(num_selected[0]) - int(num_selected[1])
+    ]
+
+    np.testing.assert_array_equal(got_first, expected_first)
+    np.testing.assert_array_equal(got_second, expected_second)
+    np.testing.assert_array_equal(got_unselected, expected_unselected)
+
+
+def test_three_way_partition_with_stream(cuda_stream):
+    dtype = np.int32
+    num_items = 50_000
+    h_in = random_array(num_items, dtype, max_value=100)
+
+    def less_than_op(x):
+        return x < 42
+
+    def greater_equal_op(x):
+        return x >= 42
+
+    expected_first, expected_second, expected_unselected, _, _, _ = (
+        _host_three_way_partition(h_in, less_than_op, greater_equal_op)
+    )
+
+    d_in = DeviceArray.from_numpy(h_in, stream=cuda_stream)
+    d_first = DeviceArray.empty(h_in.shape, h_in.dtype, stream=cuda_stream)
+    d_second = DeviceArray.empty(h_in.shape, h_in.dtype, stream=cuda_stream)
+    d_unselected = DeviceArray.empty(h_in.shape, h_in.dtype, stream=cuda_stream)
+    d_num_selected = DeviceArray.empty(2, np.int64, stream=cuda_stream)
+
+    cuda.compute.three_way_partition(
+        d_in=d_in,
+        d_first_part_out=d_first,
+        d_second_part_out=d_second,
+        d_unselected_out=d_unselected,
+        d_num_selected_out=d_num_selected,
+        select_first_part_op=less_than_op,
+        select_second_part_op=greater_equal_op,
+        num_items=num_items,
+        stream=cuda_stream,
+    )
+
+    num_selected = d_num_selected.copy_to_host(stream=cuda_stream)
+    got_first = d_first.copy_to_host(stream=cuda_stream)[: int(num_selected[0])]
+    got_second = d_second.copy_to_host(stream=cuda_stream)[: int(num_selected[1])]
+    got_unselected = d_unselected.copy_to_host(stream=cuda_stream)[
+        : int(num_items) - int(num_selected[0]) - int(num_selected[1])
+    ]
+
+    np.testing.assert_array_equal(got_first, expected_first)
+    np.testing.assert_array_equal(got_second, expected_second)
+    np.testing.assert_array_equal(got_unselected, expected_unselected)
+
+
+def test_three_way_partition_no_selection():
+    dtype = np.int32
+    num_items = 10_000
+    h_in = random_array(num_items, dtype, max_value=100)
+
+    def less_than_op(x):
+        return x == 101
+
+    def greater_equal_op(x):
+        return x == 102
+
+    d_in = DeviceArray.from_numpy(h_in)
+    d_first = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_second = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_unselected = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.int64)
+
+    cuda.compute.three_way_partition(
+        d_in=d_in,
+        d_first_part_out=d_first,
+        d_second_part_out=d_second,
+        d_unselected_out=d_unselected,
+        d_num_selected_out=d_num_selected,
+        select_first_part_op=less_than_op,
+        select_second_part_op=greater_equal_op,
+        num_items=num_items,
+    )
+
+    num_selected = d_num_selected.copy_to_host()
+    assert int(num_selected[0]) == 0 and int(num_selected[1]) == 0
+
+    got_first = d_first.copy_to_host()[: int(num_selected[0])]
+    got_second = d_second.copy_to_host()[: int(num_selected[1])]
+    got_unselected = d_unselected.copy_to_host()[:num_items]
+
+    np.testing.assert_array_equal(got_first, np.empty(0, dtype=dtype))
+    np.testing.assert_array_equal(got_second, np.empty(0, dtype=dtype))
+    np.testing.assert_array_equal(got_unselected, h_in)
+
+
+def test_three_way_partition_same_predicate():
+    dtype = np.int32
+    num_items = 100
+    h_in = random_array(num_items, dtype, max_value=100)
+
+    def always_true(x):
+        return True
+
+    d_in = DeviceArray.from_numpy(h_in)
+    d_first = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_second = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_unselected = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.int64)
+
+    cuda.compute.three_way_partition(
+        d_in=d_in,
+        d_first_part_out=d_first,
+        d_second_part_out=d_second,
+        d_unselected_out=d_unselected,
+        d_num_selected_out=d_num_selected,
+        select_first_part_op=always_true,
+        select_second_part_op=always_true,
+        num_items=num_items,
+    )
+
+    num_selected = d_num_selected.copy_to_host()
+    assert int(num_selected[0]) == num_items
+    assert int(num_selected[1]) == 0
+
+
+def test_three_way_partition_all_selected_first():
+    dtype = np.int32
+    num_items = 20_000
+    h_in = np.full(num_items, 37, dtype=dtype)
+
+    def less_than_op(x):
+        return x == 37
+
+    def greater_equal_op(x):
+        return x == 42
+
+    d_in = DeviceArray.from_numpy(h_in)
+    d_first = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_second = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_unselected = DeviceArray.empty(h_in.shape, h_in.dtype)
+    d_num_selected = DeviceArray.empty(2, np.int64)
+
+    cuda.compute.three_way_partition(
+        d_in=d_in,
+        d_first_part_out=d_first,
+        d_second_part_out=d_second,
+        d_unselected_out=d_unselected,
+        d_num_selected_out=d_num_selected,
+        select_first_part_op=less_than_op,
+        select_second_part_op=greater_equal_op,
+        num_items=num_items,
+    )
+
+    num_selected = d_num_selected.copy_to_host()
+    assert int(num_selected[0]) == num_items and int(num_selected[1]) == 0
+
+    got_first = d_first.copy_to_host()[: int(num_selected[0])]
+    got_second = d_second.copy_to_host()[: int(num_selected[1])]
+    got_unselected = d_unselected.copy_to_host()[
+        : int(num_items) - int(num_selected[0]) - int(num_selected[1])
+    ]
+
+    np.testing.assert_array_equal(got_first, h_in)
+    np.testing.assert_array_equal(got_second, np.empty(0, dtype=dtype))
+    np.testing.assert_array_equal(got_unselected, np.empty(0, dtype=dtype))
+
+
+def _less_than_8(x):
+    return x < 8 and x >= 0
+
+
+def _greater_eq_8(x):
+    return x >= 8
+
+
+def _run(
+    partitioner,
+    *,
+    d_in,
+    d_first,
+    d_second,
+    d_unselected,
+    d_num_selected,
+    num_items,
+    op1,
+    op2,
+):
+    bytes_needed = partitioner(
+        temp_storage=None,
+        d_in=d_in,
+        d_first_part_out=d_first,
+        d_second_part_out=d_second,
+        d_unselected_out=d_unselected,
+        d_num_selected_out=d_num_selected,
+        select_first_part_op=op1,
+        select_second_part_op=op2,
+        num_items=num_items,
+    )
+    tmp = TempStorageBuffer(bytes_needed, None)
+    partitioner(
+        temp_storage=tmp,
+        d_in=d_in,
+        d_first_part_out=d_first,
+        d_second_part_out=d_second,
+        d_unselected_out=d_unselected,
+        d_num_selected_out=d_num_selected,
+        select_first_part_op=op1,
+        select_second_part_op=op2,
+        num_items=num_items,
+    )
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_three_way_partition_round_trip():
+    dtype = np.int32
+    h_input = np.array([0, 2, 9, 1, 5, 6, 7, -3, 17, 10], dtype=dtype)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_first = DeviceArray.empty(h_input.shape, h_input.dtype)
+    d_second = DeviceArray.empty(h_input.shape, h_input.dtype)
+    d_unselected = DeviceArray.empty(h_input.shape, h_input.dtype)
+    d_num_selected = DeviceArray.empty(2, np.int64)
+
+    builder = make_three_way_partition(
+        d_in=d_input,
+        d_first_part_out=d_first,
+        d_second_part_out=d_second,
+        d_unselected_out=d_unselected,
+        d_num_selected_out=d_num_selected,
+        select_first_part_op=_less_than_8,
+        select_second_part_op=_greater_eq_8,
+    )
+    blob = serialize(builder)
+    assert len(blob) > 0
+
+    loaded = deserialize(blob)
+    _run(
+        loaded,
+        d_in=d_input,
+        d_first=d_first,
+        d_second=d_second,
+        d_unselected=d_unselected,
+        d_num_selected=d_num_selected,
+        num_items=h_input.size,
+        op1=_less_than_8,
+        op2=_greater_eq_8,
+    )
+
+    actual_num_selected = d_num_selected.copy_to_host()
+    n_first = int(actual_num_selected[0])
+    n_second = int(actual_num_selected[1])
+    n_unselected = h_input.size - n_first - n_second
+
+    np.testing.assert_array_equal(
+        d_first.copy_to_host()[:n_first], np.array([0, 2, 1, 5, 6, 7], dtype=dtype)
+    )
+    np.testing.assert_array_equal(
+        d_second.copy_to_host()[:n_second], np.array([9, 17, 10], dtype=dtype)
+    )
+    np.testing.assert_array_equal(
+        d_unselected.copy_to_host()[:n_unselected], np.array([-3], dtype=dtype)
+    )
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_transform.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_transform.py
new file mode 100644
index 00000000..2c61b2d2
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_transform.py
@@ -0,0 +1,739 @@
+# Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray
+
+import cuda.compute
+from cuda.compute import (
+    CountingIterator,
+    OpKind,
+    deserialize,
+    gpu_struct,
+    make_binary_transform,
+    make_unary_transform,
+    serialize,
+)
+
+
+def unary_transform_host(h_input: np.ndarray, op):
+    return np.vectorize(op)(h_input)
+
+
+def unary_transform_device(d_input, d_output, num_items, op, stream=None):
+    cuda.compute.unary_transform(
+        d_in=d_input, d_out=d_output, op=op, num_items=num_items, stream=stream
+    )
+
+
+def binary_transform_host(h_input1: np.ndarray, h_input2: np.ndarray, op):
+    return np.vectorize(op)(h_input1, h_input2)
+
+
+def binary_transform_device(d_input1, d_input2, d_output, num_items, op, stream=None):
+    cuda.compute.binary_transform(
+        d_in1=d_input1,
+        d_in2=d_input2,
+        d_out=d_output,
+        op=op,
+        num_items=num_items,
+        stream=stream,
+    )
+
+
+def test_unary_transform(input_array):
+    if input_array.dtype == np.float16:
+        pytest.skip("float16 is not supported with custom operators")
+
+    def op(a):
+        return a + 1
+
+    h_in = input_array
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+
+    unary_transform_device(d_in, d_out, h_in.size, op)
+
+    got = d_out.copy_to_host()
+    expected = unary_transform_host(h_in, op)
+
+    np.testing.assert_allclose(expected, got, rtol=1e-5)
+
+
+def test_binary_transform(input_array):
+    if input_array.dtype == np.float16:
+        pytest.skip("float16 is not supported with custom operators")
+
+    def op(a, b):
+        return a + b
+
+    h_in1 = input_array
+    h_in2 = input_array
+    d_in1 = DeviceArray.from_numpy(h_in1)
+    d_in2 = DeviceArray.from_numpy(h_in2)
+    d_out = DeviceArray.empty(h_in1.shape, h_in1.dtype)
+
+    binary_transform_device(d_in1, d_in2, d_out, h_in1.size, op)
+
+    got = d_out.copy_to_host()
+    expected = binary_transform_host(h_in1, h_in2, op)
+
+    np.testing.assert_allclose(expected, got, rtol=1e-5)
+
+
+def test_unary_transform_struct_type():
+    import numpy as np
+
+    @gpu_struct
+    class MyStruct:
+        x: np.int16
+        y: np.uint64
+
+    def op(a):
+        return MyStruct(a.x * 2, a.y + 10)
+
+    num_values = 10_000
+
+    h_in = np.empty(num_values, dtype=MyStruct.dtype)
+    h_in["x"] = np.arange(num_values)
+    h_in["y"] = 1
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+
+    cuda.compute.unary_transform(d_in=d_in, d_out=d_out, op=op, num_items=h_in.size)
+
+    got = d_out.copy_to_host()
+
+    np.testing.assert_allclose(got["x"], np.arange(num_values) * 2)
+    np.testing.assert_allclose(got["y"], np.ones(num_values) + 10)
+
+
+def test_binary_transform_struct_type():
+    import numpy as np
+
+    @gpu_struct
+    class MyStruct:
+        x: np.int16
+        y: np.uint64
+
+    def op(a, b):
+        return MyStruct(a.x + b.x, a.y + b.y)
+
+    num_values = 10_000
+
+    h_in1 = np.empty(num_values, dtype=MyStruct.dtype)
+    h_in1["x"] = np.random.randint(0, num_values, num_values, dtype="int16")
+    h_in1["y"] = np.random.randint(0, num_values, num_values, dtype="uint64")
+
+    h_in2 = np.empty(num_values, dtype=MyStruct.dtype)
+    h_in2["x"] = np.random.randint(0, num_values, num_values, dtype="int16")
+    h_in2["y"] = np.random.randint(0, num_values, num_values, dtype="uint64")
+
+    d_in1 = DeviceArray.from_numpy(h_in1)
+    d_in2 = DeviceArray.from_numpy(h_in2)
+    d_out = DeviceArray.empty(h_in1.shape, h_in1.dtype)
+
+    cuda.compute.binary_transform(
+        d_in1=d_in1, d_in2=d_in2, d_out=d_out, op=op, num_items=h_in1.size
+    )
+
+    got = d_out.copy_to_host()
+
+    np.testing.assert_allclose(got["x"], h_in1["x"] + h_in2["x"])
+    np.testing.assert_allclose(got["y"], h_in1["y"] + h_in2["y"])
+
+
+def test_unary_transform_iterator_input():
+    def op(a):
+        return a + 1
+
+    d_in = CountingIterator(np.int32(0))
+
+    num_items = 1024
+    d_out = DeviceArray.empty(num_items, np.int32)
+
+    unary_transform_device(d_in, d_out, num_items, op)
+
+    got = d_out.copy_to_host()
+    expected = np.arange(1, num_items + 1, dtype=np.int32)
+
+    np.testing.assert_allclose(expected, got)
+
+
+def test_binary_transform_iterator_input():
+    def op(a, b):
+        return a + b
+
+    d_in1 = CountingIterator(np.int32(0))
+    d_in2 = CountingIterator(np.int32(1))
+
+    num_items = 1024
+    d_out = DeviceArray.empty(num_items, np.int32)
+
+    binary_transform_device(d_in1, d_in2, d_out, num_items, op)
+
+    got = d_out.copy_to_host()
+    expected = np.arange(1, 2 * num_items + 1, step=2, dtype=np.int32)
+
+    np.testing.assert_allclose(expected, got)
+
+
+def test_unary_transform_with_stream(cuda_stream):
+    def op(a):
+        return a + 1
+
+    n = 10
+    h_in = np.arange(n, dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in, stream=cuda_stream)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype, stream=cuda_stream)
+
+    unary_transform_device(d_in, d_out, n, op, stream=cuda_stream)
+
+    got = d_out.copy_to_host(stream=cuda_stream)
+    expected = unary_transform_host(h_in, op)
+
+    np.testing.assert_allclose(expected, got, rtol=1e-5)
+
+
+def test_binary_transform_with_stream(cuda_stream):
+    def op(a, b):
+        return a + b
+
+    n = 10
+    h_in1 = np.arange(n, dtype=np.int32)
+    h_in2 = np.arange(n, dtype=np.int32)
+    d_in1 = DeviceArray.from_numpy(h_in1, stream=cuda_stream)
+    d_in2 = DeviceArray.from_numpy(h_in2, stream=cuda_stream)
+    d_out = DeviceArray.empty(h_in1.shape, h_in1.dtype, stream=cuda_stream)
+
+    binary_transform_device(d_in1, d_in2, d_out, n, op, stream=cuda_stream)
+
+    got = d_out.copy_to_host(stream=cuda_stream)
+    expected = binary_transform_host(h_in1, h_in2, op)
+
+    np.testing.assert_allclose(expected, got, rtol=1e-5)
+
+
+def test_transform_reuse_input_iterator():
+    def op(a, b):
+        return a + b
+
+    d_in1 = CountingIterator(np.int32(0))
+    d_in2 = CountingIterator(np.int32(1))
+
+    num_items = 1024
+    d_out = DeviceArray.empty(num_items, np.int32)
+
+    binary_transform_device(d_in1, d_in2, d_out, num_items, op)
+
+    got = d_out.copy_to_host()
+    expected = np.arange(1, 2 * num_items + 1, step=2, dtype=np.int32)
+
+    np.testing.assert_allclose(expected, got)
+
+    # Reusing the second input iterator should work.
+    # This is to test that the iterator is not modified by LTOIR scrubbing,
+    # which is correctly done on a copy of the iterator.
+    def op2(a):
+        return a + 1
+
+    unary_transform_device(d_in2, d_out, num_items, op2)
+    got = d_out.copy_to_host()
+    expected = np.arange(1, num_items + 1, dtype=np.int32) + 1
+
+    np.testing.assert_allclose(expected, got)
+
+
+def test_unary_transform_well_known_negate():
+    """Test unary transform with well-known NEGATE operation."""
+    dtype = np.int32
+    h_input = np.array([1, -2, 3, -4, 5], dtype=dtype)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, dtype)
+
+    # Run unary transform with well-known NEGATE operation
+    cuda.compute.unary_transform(
+        d_in=d_input, d_out=d_output, op=OpKind.NEGATE, num_items=h_input.size
+    )
+
+    # Check the result is correct
+    expected = np.array([-1, 2, -3, 4, -5])
+    np.testing.assert_equal(d_output.copy_to_host(), expected)
+
+
+def test_unary_transform_well_known_identity():
+    """Test unary transform with well-known IDENTITY operation."""
+    dtype = np.int32
+    h_input = np.array([1, 2, 3, 4, 5], dtype=dtype)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, dtype)
+
+    # Run unary transform with well-known IDENTITY operation
+    cuda.compute.unary_transform(
+        d_in=d_input, d_out=d_output, op=OpKind.IDENTITY, num_items=h_input.size
+    )
+
+    # Check the result is correct
+    expected = np.array([1, 2, 3, 4, 5])
+    np.testing.assert_equal(d_output.copy_to_host(), expected)
+
+
+def test_unary_transform_well_known_bit_not():
+    h_input = np.array([0, 1, -2, 42, -100], dtype=np.int32)
+    d_input = DeviceArray.from_numpy(h_input)
+    d_output = DeviceArray.empty(h_input.shape, h_input.dtype)
+
+    cuda.compute.unary_transform(
+        d_in=d_input,
+        d_out=d_output,
+        op=OpKind.BIT_NOT,
+        num_items=len(d_input),
+    )
+
+    expected = np.array([-1, -2, 1, -43, 99], dtype=np.int32)
+    np.testing.assert_array_equal(d_output.copy_to_host(), expected)
+
+
+@pytest.mark.parametrize("dtype", [np.int32, np.float16])
+def test_binary_transform_well_known_plus(dtype):
+    """Test binary transform with well-known PLUS operation."""
+    h_input1 = np.array([1, 2, 3, 4, 5], dtype=dtype)
+    h_input2 = np.array([10, 20, 30, 40, 50], dtype=dtype)
+    d_input1 = DeviceArray.from_numpy(h_input1)
+    d_input2 = DeviceArray.from_numpy(h_input2)
+    d_output = DeviceArray.empty(h_input1.shape, dtype)
+
+    # Run binary transform with well-known PLUS operation
+    cuda.compute.binary_transform(
+        d_in1=d_input1,
+        d_in2=d_input2,
+        d_out=d_output,
+        op=OpKind.PLUS,
+        num_items=h_input1.size,
+    )
+
+    # Check the result is correct
+    expected = np.array([11, 22, 33, 44, 55])
+    np.testing.assert_equal(d_output.copy_to_host(), expected)
+
+
+def test_binary_transform_well_known_multiplies():
+    """Test binary transform with well-known MULTIPLIES operation."""
+    dtype = np.int32
+    h_input1 = np.array([1, 2, 3, 4, 5], dtype=dtype)
+    h_input2 = np.array([2, 3, 4, 5, 6], dtype=dtype)
+    d_input1 = DeviceArray.from_numpy(h_input1)
+    d_input2 = DeviceArray.from_numpy(h_input2)
+    d_output = DeviceArray.empty(h_input1.shape, dtype)
+
+    # Run binary transform with well-known MULTIPLIES operation
+    cuda.compute.binary_transform(
+        d_in1=d_input1,
+        d_in2=d_input2,
+        d_out=d_output,
+        op=OpKind.MULTIPLIES,
+        num_items=h_input1.size,
+    )
+
+    # Check the result is correct
+    expected = np.array([2, 6, 12, 20, 30])
+    np.testing.assert_equal(d_output.copy_to_host(), expected)
+
+
+@pytest.mark.parametrize(
+    "op,host_op",
+    [
+        pytest.param(OpKind.LOGICAL_AND, np.logical_and, id="logical_and"),
+        pytest.param(OpKind.LOGICAL_OR, np.logical_or, id="logical_or"),
+    ],
+)
+def test_binary_transform_well_known_logical(op, host_op):
+    h_input1 = np.array([True, True, False, False], dtype=np.bool_)
+    h_input2 = np.array([True, False, True, False], dtype=np.bool_)
+    d_input1 = DeviceArray.from_numpy(h_input1)
+    d_input2 = DeviceArray.from_numpy(h_input2)
+    d_output = DeviceArray.empty(h_input1.shape, h_input1.dtype)
+
+    cuda.compute.binary_transform(
+        d_in1=d_input1,
+        d_in2=d_input2,
+        d_out=d_output,
+        op=op,
+        num_items=len(d_input1),
+    )
+
+    np.testing.assert_array_equal(d_output.copy_to_host(), host_op(h_input1, h_input2))
+
+
+def test_unary_transform_struct_type_with_annotations():
+    @gpu_struct
+    class Point:
+        x: np.float32
+        y: np.float32
+
+    def scale_op(p: Point) -> Point:
+        return Point(p.x * 2.0, p.y * 3.0)
+
+    num_items = 100
+
+    h_in = np.empty(num_items, dtype=Point.dtype)
+    h_in["x"] = np.random.rand(num_items).astype(np.float32)
+    h_in["y"] = np.random.rand(num_items).astype(np.float32)
+
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+
+    cuda.compute.unary_transform(
+        d_in=d_in, d_out=d_out, op=scale_op, num_items=num_items
+    )
+
+    result = d_out.copy_to_host()
+
+    np.testing.assert_allclose(result["x"], h_in["x"] * 2.0, rtol=1e-5)
+    np.testing.assert_allclose(result["y"], h_in["y"] * 3.0, rtol=1e-5)
+
+
+def test_binary_transform_struct_type_with_annotations():
+    @gpu_struct
+    class Vec2D:
+        x: np.int32
+        y: np.int32
+
+    def add_vectors(v1: Vec2D, v2: Vec2D) -> Vec2D:
+        return Vec2D(v1.x + v2.x, v1.y + v2.y)
+
+    num_items = 100
+
+    h_in1 = np.empty(num_items, dtype=Vec2D.dtype)
+    h_in1["x"] = np.random.randint(-100, 100, num_items, dtype=np.int32)
+    h_in1["y"] = np.random.randint(-100, 100, num_items, dtype=np.int32)
+
+    h_in2 = np.empty(num_items, dtype=Vec2D.dtype)
+    h_in2["x"] = np.random.randint(-100, 100, num_items, dtype=np.int32)
+    h_in2["y"] = np.random.randint(-100, 100, num_items, dtype=np.int32)
+
+    d_in1 = DeviceArray.from_numpy(h_in1)
+    d_in2 = DeviceArray.from_numpy(h_in2)
+    d_out = DeviceArray.empty(h_in1.shape, h_in1.dtype)
+
+    cuda.compute.binary_transform(
+        d_in1=d_in1, d_in2=d_in2, d_out=d_out, op=add_vectors, num_items=num_items
+    )
+
+    result = d_out.copy_to_host()
+
+    np.testing.assert_equal(result["x"], h_in1["x"] + h_in2["x"])
+    np.testing.assert_equal(result["y"], h_in1["y"] + h_in2["y"])
+
+
+def test_unary_transform_stateful_counting():
+    """Test unary_transform with state that counts even numbers."""
+    from numba import cuda as numba_cuda
+
+    h_in = np.arange(100, dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+
+    even_count = DeviceArray.from_numpy(np.zeros(1, dtype=np.int32))
+
+    # Define op that references state as closure
+    def count_evens(x):
+        if x % 2 == 0:
+            numba_cuda.atomic.add(even_count, 0, 1)
+        return x * 2
+
+    cuda.compute.unary_transform(
+        d_in=d_in, d_out=d_out, op=count_evens, num_items=h_in.size
+    )
+
+    expected_output = h_in * 2
+    np.testing.assert_array_equal(d_out.copy_to_host(), expected_output)
+
+    num_evens = int(even_count.copy_to_host()[0])
+    assert num_evens == 50  # 0, 2, 4, ..., 98
+
+
+def test_unary_transform_stateful_state_updates():
+    """Test that stateful transform correctly updates state between calls."""
+    num_items = 20
+    h_in = np.arange(num_items, dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+
+    # Create two different thresholds
+    threshold_10 = DeviceArray.from_numpy(np.array([10], dtype=np.int32))
+    threshold_15 = DeviceArray.from_numpy(np.array([15], dtype=np.int32))
+
+    # Call 1: x + 10
+    def add_threshold_10(x):
+        return x + threshold_10[0]
+
+    cuda.compute.unary_transform(
+        d_in=d_in, d_out=d_out, op=add_threshold_10, num_items=num_items
+    )
+    result_1 = d_out.copy_to_host()
+    expected_1 = h_in + 10
+    np.testing.assert_array_equal(result_1, expected_1)
+
+    # Call 2: x + 15 (different state)
+    def add_threshold_15(x):
+        return x + threshold_15[0]
+
+    d_out.copy_from_host(np.zeros_like(h_in))
+    cuda.compute.unary_transform(
+        d_in=d_in, d_out=d_out, op=add_threshold_15, num_items=num_items
+    )
+    result_2 = d_out.copy_to_host()
+    expected_2 = h_in + 15
+    np.testing.assert_array_equal(result_2, expected_2)
+
+    # Call 3: Back to first threshold (test cache reuse with updated state)
+    d_out.copy_from_host(np.zeros_like(h_in))
+    cuda.compute.unary_transform(
+        d_in=d_in, d_out=d_out, op=add_threshold_10, num_items=num_items
+    )
+    result_3 = d_out.copy_to_host()
+    expected_3 = h_in + 10
+    np.testing.assert_array_equal(result_3, expected_3)
+
+
+def test_unary_transform_stateful_multiple_arrays():
+    """Test stateful transform with multiple captured arrays."""
+    num_items = 10
+    h_in = np.arange(num_items, dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+
+    # Multiple state arrays
+    offset = DeviceArray.from_numpy(np.array([5], dtype=np.int32))
+    multiplier = DeviceArray.from_numpy(np.array([2], dtype=np.int32))
+
+    def transform_with_multiple_state(x):
+        return (x + offset[0]) * multiplier[0]
+
+    cuda.compute.unary_transform(
+        d_in=d_in, d_out=d_out, op=transform_with_multiple_state, num_items=num_items
+    )
+    result = d_out.copy_to_host()
+    expected = (h_in + 5) * 2
+    np.testing.assert_array_equal(result, expected)
+
+    # Update state and verify it works with new values
+    offset = DeviceArray.from_numpy(np.array([10], dtype=np.int32))
+    multiplier = DeviceArray.from_numpy(np.array([3], dtype=np.int32))
+
+    def transform_with_updated_state(x):
+        return (x + offset[0]) * multiplier[0]
+
+    d_out.copy_from_host(np.zeros_like(h_in))
+    cuda.compute.unary_transform(
+        d_in=d_in, d_out=d_out, op=transform_with_updated_state, num_items=num_items
+    )
+    result = d_out.copy_to_host()
+    expected = (h_in + 10) * 3
+    np.testing.assert_array_equal(result, expected)
+
+
+def test_unary_transform_stateful_closure_factory():
+    """Test stateful transform with dynamically created closures.
+
+    This test verifies that state arrays captured in closures created by
+    a factory function are properly detected and updated on each call.
+    """
+
+    def make_adder(arr):
+        """Factory that creates functions with different closure-captured arrays."""
+
+        def func(x):
+            return x + arr[0]
+
+        return func
+
+    h_in = np.array([0, 1, 2], dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+
+    # First call with offset 10
+    cuda.compute.unary_transform(
+        d_in=d_in,
+        d_out=d_out,
+        op=make_adder(DeviceArray.from_numpy(np.array([10], dtype=np.int64))),
+        num_items=h_in.size,
+    )
+    np.testing.assert_array_equal(d_out.copy_to_host(), np.array([10, 11, 12]))
+
+    # Multiple calls with different offsets to test state re-detection
+    for i in range(5):
+        offset = i * 10
+        cuda.compute.unary_transform(
+            d_in=d_in,
+            d_out=d_out,
+            op=make_adder(DeviceArray.from_numpy(np.array([offset], dtype=np.int64))),
+            num_items=h_in.size,
+        )
+        expected = np.array([offset, offset + 1, offset + 2])
+        np.testing.assert_array_equal(
+            d_out.copy_to_host(),
+            expected,
+            err_msg=f"Failed at iteration {i} with offset {offset}",
+        )
+
+
+def test_unary_transform_with_lambda():
+    """Test unary_transform with a lambda function."""
+    h_in = np.array([1, 2, 3, 4, 5], dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+
+    # Use a lambda function directly
+    cuda.compute.unary_transform(
+        d_in=d_in, d_out=d_out, op=lambda x: x * 2, num_items=h_in.size
+    )
+
+    expected = np.array([2, 4, 6, 8, 10], dtype=np.int32)
+    np.testing.assert_array_equal(d_out.copy_to_host(), expected)
+
+
+def test_binary_transform_with_lambda():
+    """Test binary_transform with a lambda function."""
+    h_in1 = np.array([1, 2, 3, 4, 5], dtype=np.int32)
+    h_in2 = np.array([10, 20, 30, 40, 50], dtype=np.int32)
+    d_in1 = DeviceArray.from_numpy(h_in1)
+    d_in2 = DeviceArray.from_numpy(h_in2)
+    d_out = DeviceArray.empty(h_in1.shape, h_in1.dtype)
+
+    # Use a lambda function directly
+    cuda.compute.binary_transform(
+        d_in1=d_in1,
+        d_in2=d_in2,
+        d_out=d_out,
+        op=lambda a, b: a + b,
+        num_items=h_in1.size,
+    )
+
+    expected = np.array([11, 22, 33, 44, 55], dtype=np.int32)
+    np.testing.assert_array_equal(d_out.copy_to_host(), expected)
+
+
+def test_binary_transform_bool_equal_to():
+    h_input1 = np.array([True, False, True, False], dtype=np.bool_)
+    h_input2 = np.array([True, True, False, False], dtype=np.bool_)
+    d_input1 = DeviceArray.from_numpy(h_input1)
+    d_input2 = DeviceArray.from_numpy(h_input2)
+    d_output = DeviceArray.empty(h_input1.shape, h_input1.dtype)
+
+    cuda.compute.binary_transform(
+        d_in1=d_input1,
+        d_in2=d_input2,
+        d_out=d_output,
+        op=OpKind.EQUAL_TO,
+        num_items=h_input1.size,
+    )
+
+    expected = np.array([True, False, False, True], dtype=np.bool_)
+    np.testing.assert_array_equal(d_output.copy_to_host(), expected)
+
+
+def test_stateful_transform_same_bytecode_different_sizes():
+    """
+    Test that stateful op with same bytecode, but referencing arrays
+    of different sizes produce the correct result
+    """
+
+    def make_op(arr):
+        def op(x):
+            return x > len(arr)
+
+        return op
+
+    h_in = np.asarray([1, 2, 3])
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, bool)
+    op1 = make_op(DeviceArray.empty(1, np.float64))  # len(arr) == 1
+    op2 = make_op(DeviceArray.empty(2, np.float64))  # len(arr) == 2
+
+    cuda.compute.unary_transform(d_in=d_in, d_out=d_out, op=op1, num_items=h_in.size)
+    np.testing.assert_array_equal(np.asarray([False, True, True]), d_out.copy_to_host())
+
+    cuda.compute.unary_transform(d_in=d_in, d_out=d_out, op=op2, num_items=h_in.size)
+    np.testing.assert_array_equal(
+        np.asarray([False, False, True]), d_out.copy_to_host()
+    )
+
+
+def test_transform_caching_with_global_np_ufunc():
+    # regression test for a case where if multiple, identically named,
+    # ops referenced dotted globals like `np.<func>` those
+    # ops would all hash to the same value.
+
+    h_in = np.asarray([1.0, 2.0, 3.0])
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+
+    def make_op():
+        sin = np.sin
+
+        def op(x):
+            return sin(x)
+
+        return op
+
+    cuda.compute.unary_transform(
+        d_in=d_in, d_out=d_out, op=make_op(), num_items=h_in.size
+    )
+    np.testing.assert_allclose(d_out.copy_to_host(), np.sin(h_in))
+
+    def make_op():
+        cos = np.cos
+
+        def op(x):
+            return cos(x)
+
+        return op
+
+    cuda.compute.unary_transform(
+        d_in=d_in, d_out=d_out, op=make_op(), num_items=h_in.size
+    )
+    np.testing.assert_allclose(d_out.copy_to_host(), np.cos(h_in))
+
+
+def _add_one(a):
+    return a + 1
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_unary_transform_round_trip():
+    h_in = np.array([1, 2, 3, 4, 5], dtype=np.int32)
+    d_in = DeviceArray.from_numpy(h_in)
+    d_out = DeviceArray.empty(h_in.shape, h_in.dtype)
+
+    builder = make_unary_transform(d_in=d_in, d_out=d_out, op=_add_one)
+    blob = serialize(builder)
+    assert len(blob) > 0
+
+    loaded = deserialize(blob)
+    loaded(d_in=d_in, d_out=d_out, op=_add_one, num_items=h_in.size)
+
+    np.testing.assert_array_equal(d_out.copy_to_host(), h_in + 1)
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_binary_transform_round_trip():
+    h_in1 = np.array([1, 2, 3, 4], dtype=np.int32)
+    h_in2 = np.array([10, 20, 30, 40], dtype=np.int32)
+    d_in1 = DeviceArray.from_numpy(h_in1)
+    d_in2 = DeviceArray.from_numpy(h_in2)
+    d_out = DeviceArray.empty(h_in1.shape, h_in1.dtype)
+
+    builder = make_binary_transform(
+        d_in1=d_in1, d_in2=d_in2, d_out=d_out, op=OpKind.PLUS
+    )
+    blob = serialize(builder)
+    assert len(blob) > 0
+
+    loaded = deserialize(blob)
+    loaded(d_in1=d_in1, d_in2=d_in2, d_out=d_out, op=OpKind.PLUS, num_items=h_in1.size)
+
+    np.testing.assert_array_equal(d_out.copy_to_host(), h_in1 + h_in2)
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_unique_by_key.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_unique_by_key.py
new file mode 100644
index 00000000..0da164d6
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_unique_by_key.py
@@ -0,0 +1,553 @@
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray, get_compute_capability
+
+import cuda.compute
+from cuda.compute import (
+    CacheModifiedInputIterator,
+    DiscardIterator,
+    OpKind,
+    deserialize,
+    gpu_struct,
+    make_unique_by_key,
+    serialize,
+)
+from cuda.compute._utils.temp_storage_buffer import TempStorageBuffer
+
+DTYPE_LIST = [
+    np.uint8,
+    np.uint16,
+    np.uint32,
+    np.uint64,
+    np.int8,
+    np.int16,
+    np.int32,
+    np.int64,
+    np.float16,
+    np.float32,
+    np.float64,
+]
+
+
+def get_mark(dt, log_size):
+    if log_size < 20:
+        return tuple()
+    return pytest.mark.large
+
+
+PROBLEM_SIZES = [2, 8, 16, 22]
+
+
+def random_array(size, dtype, max_value=None) -> np.typing.NDArray:
+    rng = np.random.default_rng()
+    if np.isdtype(dtype, "integral"):
+        if max_value is None:
+            max_value = np.iinfo(dtype).max
+        return rng.integers(max_value, size=size, dtype=dtype)
+    elif np.isdtype(dtype, "real floating"):
+        if dtype == np.float16:  # Cannot generate float16 directly
+            return rng.random(size=size, dtype=np.float32).astype(dtype)
+        return rng.random(size=size, dtype=dtype)
+    else:
+        raise ValueError(f"Unsupported dtype {dtype}")
+
+
+def unique_by_key_device(
+    d_in_keys,
+    d_in_items,
+    d_out_keys,
+    d_out_items,
+    d_out_num_selected,
+    op,
+    num_items,
+    stream=None,
+):
+    # Call single-phase API directly with all parameters including num_items
+    cuda.compute.unique_by_key(
+        d_in_keys=d_in_keys,
+        d_in_items=d_in_items,
+        d_out_keys=d_out_keys,
+        d_out_items=d_out_items,
+        d_out_num_selected=d_out_num_selected,
+        op=op,
+        num_items=num_items,
+        stream=stream,
+    )
+
+
+def is_equal_func(lhs, rhs):
+    return lhs == rhs
+
+
+def unique_by_key_host(keys, items, is_equal=is_equal_func):
+    # Must implement our own version of unique_by_key since np.unique() returns
+    # unique elements across the entire array, while cub::UniqueByKey
+    # de-duplicates consecutive keys that are equal.
+    if len(keys) == 0:
+        return np.empty(0, dtype=keys.dtype), np.empty(0, dtype=items.dtype)
+
+    prev_key = keys[0]
+    keys_out = [prev_key]
+
+    if len(items) > 0:
+        items_out = [items[0]]
+    else:
+        items_out = []
+
+    for idx, (previous, next) in enumerate(zip(keys, keys[1:])):
+        if not is_equal(previous, next):
+            keys_out.append(next)
+
+            if len(items) > 0:
+                # add 1 since we are enumerating over pairs
+                items_out.append(items[idx + 1])
+
+    return np.array(keys_out), np.array(items_out)
+
+
+def compare_op(lhs, rhs):
+    return np.uint8(lhs == rhs)
+
+
+unique_by_key_params = [
+    pytest.param(
+        dt,
+        2**log_size,
+        OpKind.EQUAL_TO if dt == np.float16 else compare_op,
+        marks=get_mark(dt, log_size),
+    )
+    for dt in DTYPE_LIST
+    for log_size in PROBLEM_SIZES
+]
+
+
+@pytest.mark.parametrize("dtype, num_items, op", unique_by_key_params)
+def test_unique_by_key(dtype, num_items, op, monkeypatch):
+    cc_major, _ = get_compute_capability()
+    # Skip sass verification for CC 9.0+ due to a bug in NVRTC.
+    # TODO: add NVRTC version check, ref nvbug 5243118
+    if cc_major >= 9:
+        import cuda.compute._cccl_interop
+
+        monkeypatch.setattr(
+            cuda.compute._cccl_interop,
+            "_check_sass",
+            False,
+        )
+
+    h_in_keys = random_array(num_items, dtype, max_value=20)
+    h_in_items = random_array(num_items, np.float32)
+    h_out_keys = np.empty(num_items, dtype=dtype)
+    h_out_items = np.empty(num_items, dtype=np.float32)
+    h_out_num_selected = np.empty(1, np.int32)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_items = DeviceArray.from_numpy(h_in_items)
+    d_out_keys = DeviceArray.empty(h_out_keys.shape, h_out_keys.dtype)
+    d_out_items = DeviceArray.empty(h_out_items.shape, h_out_items.dtype)
+    d_out_num_selected = DeviceArray.empty(
+        h_out_num_selected.shape, h_out_num_selected.dtype
+    )
+
+    unique_by_key_device(
+        d_in_keys,
+        d_in_items,
+        d_out_keys,
+        d_out_items,
+        d_out_num_selected,
+        op,
+        num_items,
+    )
+
+    h_out_num_selected = d_out_num_selected.copy_to_host()
+    num_selected = h_out_num_selected[0]
+    h_out_keys = d_out_keys.copy_to_host()[:num_selected]
+    h_out_items = d_out_items.copy_to_host()[:num_selected]
+
+    expected_keys, expected_items = unique_by_key_host(h_in_keys, h_in_items)
+
+    np.testing.assert_array_equal(h_out_keys, expected_keys)
+    np.testing.assert_array_equal(h_out_items, expected_items)
+
+
+@pytest.mark.parametrize("dtype, num_items, op", unique_by_key_params)
+def test_unique_by_key_iterators(dtype, num_items, op, monkeypatch):
+    cc_major, _ = get_compute_capability()
+    # Skip sass verification for CC 9.0+, due to a bug in NVRTC.
+    # TODO: add NVRTC version check, ref nvbug 5243118
+    if cc_major >= 9:
+        import cuda.compute._cccl_interop
+
+        monkeypatch.setattr(
+            cuda.compute._cccl_interop,
+            "_check_sass",
+            False,
+        )
+
+    h_in_keys = random_array(num_items, dtype, max_value=20)
+    h_in_items = random_array(num_items, np.float32)
+    h_out_keys = np.empty(num_items, dtype=dtype)
+    h_out_items = np.empty(num_items, dtype=np.float32)
+    h_out_num_selected = np.empty(1, np.int64)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_items = DeviceArray.from_numpy(h_in_items)
+    d_out_keys = DeviceArray.empty(h_out_keys.shape, h_out_keys.dtype)
+    d_out_items = DeviceArray.empty(h_out_items.shape, h_out_items.dtype)
+    d_out_num_selected = DeviceArray.empty(
+        h_out_num_selected.shape, h_out_num_selected.dtype
+    )
+
+    i_in_keys = CacheModifiedInputIterator(d_in_keys, modifier="stream")
+    i_in_items = CacheModifiedInputIterator(d_in_items, modifier="stream")
+
+    unique_by_key_device(
+        i_in_keys,
+        i_in_items,
+        d_out_keys,
+        d_out_items,
+        d_out_num_selected,
+        op,
+        num_items,
+    )
+
+    h_out_num_selected = d_out_num_selected.copy_to_host()
+    num_selected = h_out_num_selected[0]
+    h_out_keys = d_out_keys.copy_to_host()[:num_selected]
+    h_out_items = d_out_items.copy_to_host()[:num_selected]
+
+    expected_keys, expected_items = unique_by_key_host(h_in_keys, h_in_items)
+
+    np.testing.assert_array_equal(h_out_keys, expected_keys)
+    np.testing.assert_array_equal(h_out_items, expected_items)
+
+
+def test_unique_by_key_keys_only():
+    num_items = 100000
+    h_in_keys = random_array(num_items, np.int32, max_value=20)
+    h_out_keys = np.empty(num_items, dtype=np.int32)
+    h_out_num_selected = np.empty(1, np.int32)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_out_keys = DeviceArray.empty(h_out_keys.shape, h_out_keys.dtype)
+    d_out_num_selected = DeviceArray.empty(
+        h_out_num_selected.shape, h_out_num_selected.dtype
+    )
+
+    unique_by_key_device(
+        d_in_keys,
+        DiscardIterator(),
+        d_out_keys,
+        DiscardIterator(),
+        d_out_num_selected,
+        OpKind.EQUAL_TO,
+        num_items,
+    )
+
+    h_out_num_selected = d_out_num_selected.copy_to_host()
+    num_selected = h_out_num_selected[0]
+    h_out_keys = d_out_keys.copy_to_host()[:num_selected]
+
+    expected_keys, _ = unique_by_key_host(
+        h_in_keys,
+        np.empty(0, dtype=np.float32),
+    )
+
+    np.testing.assert_array_equal(h_out_keys, expected_keys)
+
+
+def test_unique_by_key_complex():
+    def compare_complex(lhs, rhs):
+        return np.uint8(lhs.real == rhs.real)
+
+    num_items = 100000
+    max_value = 20
+    real = random_array(num_items, np.int64, max_value)
+    imaginary = random_array(num_items, np.int64, max_value)
+
+    h_in_keys = real + 1j * imaginary
+    h_in_items = random_array(num_items, np.float32)
+    h_out_keys = np.empty(num_items, dtype=h_in_keys.dtype)
+    h_out_items = np.empty(num_items, dtype=np.float32)
+    h_out_num_selected = np.empty(1, np.int32)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_items = DeviceArray.from_numpy(h_in_items)
+    d_out_keys = DeviceArray.empty(h_out_keys.shape, h_out_keys.dtype)
+    d_out_items = DeviceArray.empty(h_out_items.shape, h_out_items.dtype)
+    d_out_num_selected = DeviceArray.empty(
+        h_out_num_selected.shape, h_out_num_selected.dtype
+    )
+
+    unique_by_key_device(
+        d_in_keys,
+        d_in_items,
+        d_out_keys,
+        d_out_items,
+        d_out_num_selected,
+        compare_complex,
+        num_items,
+    )
+
+    h_out_num_selected = d_out_num_selected.copy_to_host()
+    num_selected = h_out_num_selected[0]
+    h_out_keys = d_out_keys.copy_to_host()[:num_selected]
+    h_out_items = d_out_items.copy_to_host()[:num_selected]
+
+    expected_keys, expected_items = unique_by_key_host(
+        h_in_keys, h_in_items, compare_complex
+    )
+
+    np.testing.assert_array_equal(h_out_keys, expected_keys)
+    np.testing.assert_array_equal(h_out_items, expected_items)
+
+
+def test_unique_by_key_struct_types():
+    @gpu_struct
+    class key_pair:
+        a: np.int16
+        b: np.uint64
+
+    @gpu_struct
+    class item_pair:
+        a: np.int32
+        b: np.float32
+
+    def struct_compare_op(lhs, rhs):
+        return np.uint8((lhs.a == rhs.a) and (lhs.b == rhs.b))
+
+    num_items = 10000
+
+    a_keys = np.random.randint(0, 20, num_items, dtype=np.int16)
+    b_keys = np.random.randint(0, 20, num_items, dtype=np.uint64)
+
+    a_items = np.random.randint(0, 20, num_items, dtype=np.int32)
+    b_items = np.random.rand(num_items).astype(np.float32)
+
+    h_in_keys = np.empty(num_items, dtype=key_pair.dtype)
+    h_in_items = np.empty(num_items, dtype=item_pair.dtype)
+    h_out_num_selected = np.empty(1, np.int64)
+
+    h_in_keys["a"] = a_keys
+    h_in_keys["b"] = b_keys
+
+    h_in_items["a"] = a_items
+    h_in_items["b"] = b_items
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_items = DeviceArray.from_numpy(h_in_items)
+    d_out_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype)
+    d_out_items = DeviceArray.empty(h_in_items.shape, h_in_items.dtype)
+    d_out_num_selected = DeviceArray.empty(
+        h_out_num_selected.shape, h_out_num_selected.dtype
+    )
+
+    unique_by_key_device(
+        d_in_keys,
+        d_in_items,
+        d_out_keys,
+        d_out_items,
+        d_out_num_selected,
+        struct_compare_op,
+        num_items,
+    )
+
+    h_out_num_selected = d_out_num_selected.copy_to_host()
+    num_selected = h_out_num_selected[0]
+    h_out_keys = d_out_keys.copy_to_host()[:num_selected]
+    h_out_items = d_out_items.copy_to_host()[:num_selected]
+
+    expected_keys, expected_items = unique_by_key_host(
+        h_in_keys,
+        h_in_items,
+        lambda lhs, rhs: (lhs["a"] == rhs["a"]) and (lhs["b"] == rhs["b"]),
+    )
+
+    np.testing.assert_array_equal(h_out_keys, expected_keys)
+    np.testing.assert_array_equal(h_out_items, expected_items)
+
+
+def test_unique_by_key_with_stream(cuda_stream, monkeypatch):
+    cc_major, _ = get_compute_capability()
+    # Skip sass verification for CC 9.0+ due to a bug in NVRTC.
+    # TODO: add NVRTC version check, ref nvbug 5243118
+    if cc_major >= 9:
+        import cuda.compute._cccl_interop
+
+        monkeypatch.setattr(
+            cuda.compute._cccl_interop,
+            "_check_sass",
+            False,
+        )
+
+    num_items = 10000
+
+    h_in_keys = random_array(num_items, np.int32, max_value=20)
+    h_in_items = random_array(num_items, np.float32)
+    h_out_keys = np.empty(num_items, dtype=np.int32)
+    h_out_items = np.empty(num_items, dtype=np.float32)
+    h_out_num_selected = np.empty(1, np.int32)
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys, stream=cuda_stream)
+    d_in_items = DeviceArray.from_numpy(h_in_items, stream=cuda_stream)
+    d_out_keys = DeviceArray.empty(
+        h_out_keys.shape, h_out_keys.dtype, stream=cuda_stream
+    )
+    d_out_items = DeviceArray.empty(
+        h_out_items.shape, h_out_items.dtype, stream=cuda_stream
+    )
+    d_out_num_selected = DeviceArray.empty(
+        h_out_num_selected.shape,
+        h_out_num_selected.dtype,
+        stream=cuda_stream,
+    )
+
+    unique_by_key_device(
+        d_in_keys,
+        d_in_items,
+        d_out_keys,
+        d_out_items,
+        d_out_num_selected,
+        compare_op,
+        num_items,
+        stream=cuda_stream,
+    )
+
+    h_out_keys = d_out_keys.copy_to_host(stream=cuda_stream)
+    h_out_items = d_out_items.copy_to_host(stream=cuda_stream)
+    h_out_num_selected = d_out_num_selected.copy_to_host(stream=cuda_stream)
+
+    num_selected = h_out_num_selected[0]
+    h_out_keys = h_out_keys[:num_selected]
+    h_out_items = h_out_items[:num_selected]
+
+    expected_keys, expected_items = unique_by_key_host(h_in_keys, h_in_items)
+
+    np.testing.assert_array_equal(h_out_keys, expected_keys)
+    np.testing.assert_array_equal(h_out_items, expected_items)
+
+
+def test_unique_by_key_well_known_equal_to(monkeypatch):
+    cc_major, _ = get_compute_capability()
+    # Skip sass verification for CC 9.0+ due to a bug in NVRTC.
+    # TODO: add NVRTC version check, ref nvbug 5243118
+    if cc_major >= 9:
+        import cuda.compute._cccl_interop as cccl_interop
+
+        monkeypatch.setattr(
+            cccl_interop,
+            "_check_sass",
+            False,
+        )
+
+    dtype = np.int32
+
+    # Create input keys and values: keys=[1,1,1,2,2,3] values=[10,20,30,40,50,60]
+    h_in_keys = np.array([1, 1, 1, 2, 2, 3], dtype=dtype)
+    h_in_values = np.array([10, 20, 30, 40, 50, 60], dtype=dtype)
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_values = DeviceArray.from_numpy(h_in_values)
+    d_out_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype)
+    d_out_values = DeviceArray.empty(h_in_values.shape, h_in_values.dtype)
+    d_num_selected = DeviceArray.empty(1, dtype)
+
+    # Run unique by key with well-known EQUAL_TO operation
+    cuda.compute.unique_by_key(
+        d_in_keys=d_in_keys,
+        d_in_items=d_in_values,
+        d_out_keys=d_out_keys,
+        d_out_items=d_out_values,
+        d_out_num_selected=d_num_selected,
+        op=OpKind.EQUAL_TO,
+        num_items=h_in_keys.size,
+    )
+
+    # Check the result is correct
+    assert d_num_selected.copy_to_host()[0] == 3  # three unique keys
+    expected_keys = [1, 2, 3]
+    expected_values = [10, 40, 60]  # first occurrence of each key
+
+    np.testing.assert_equal(d_out_keys.copy_to_host()[:3], expected_keys)
+    np.testing.assert_equal(d_out_values.copy_to_host()[:3], expected_values)
+
+
+def _run(
+    uniquer,
+    *,
+    d_in_keys,
+    d_in_values,
+    d_out_keys,
+    d_out_values,
+    d_out_num_selected,
+    num_items,
+    op,
+):
+    bytes_needed = uniquer(
+        temp_storage=None,
+        d_in_keys=d_in_keys,
+        d_in_items=d_in_values,
+        d_out_keys=d_out_keys,
+        d_out_items=d_out_values,
+        d_out_num_selected=d_out_num_selected,
+        op=op,
+        num_items=num_items,
+    )
+    tmp = TempStorageBuffer(bytes_needed, None)
+    uniquer(
+        temp_storage=tmp,
+        d_in_keys=d_in_keys,
+        d_in_items=d_in_values,
+        d_out_keys=d_out_keys,
+        d_out_items=d_out_values,
+        d_out_num_selected=d_out_num_selected,
+        op=op,
+        num_items=num_items,
+    )
+
+
+@pytest.mark.serialization
+def test_serialize_deserialize_unique_by_key_round_trip():
+    h_in_keys = np.array([0, 2, 2, 9, 5, 5, 5, 8], dtype="int32")
+    h_in_values = np.array([1, 2, 3, 4, 5, 6, 7, 8], dtype="float32")
+
+    d_in_keys = DeviceArray.from_numpy(h_in_keys)
+    d_in_values = DeviceArray.from_numpy(h_in_values)
+    d_out_keys = DeviceArray.empty(h_in_keys.shape, h_in_keys.dtype)
+    d_out_values = DeviceArray.empty(h_in_values.shape, h_in_values.dtype)
+    d_out_num_selected = DeviceArray.empty(1, np.int32)
+
+    builder = make_unique_by_key(
+        d_in_keys=d_in_keys,
+        d_in_items=d_in_values,
+        d_out_keys=d_out_keys,
+        d_out_items=d_out_values,
+        d_out_num_selected=d_out_num_selected,
+        op=OpKind.EQUAL_TO,
+    )
+    blob = serialize(builder)
+    assert len(blob) > 0
+
+    loaded = deserialize(blob)
+    _run(
+        loaded,
+        d_in_keys=d_in_keys,
+        d_in_values=d_in_values,
+        d_out_keys=d_out_keys,
+        d_out_values=d_out_values,
+        d_out_num_selected=d_out_num_selected,
+        num_items=h_in_keys.size,
+        op=OpKind.EQUAL_TO,
+    )
+
+    n = int(d_out_num_selected.copy_to_host()[0])
+    np.testing.assert_array_equal(
+        d_out_keys.copy_to_host()[:n], np.array([0, 2, 9, 5, 8])
+    )
+    np.testing.assert_array_equal(
+        d_out_values.copy_to_host()[:n], np.array([1, 2, 4, 5, 8], dtype=np.float32)
+    )
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_void_ptr_wrapper_validation.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_void_ptr_wrapper_validation.py
new file mode 100644
index 00000000..e08709b6
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_void_ptr_wrapper_validation.py
@@ -0,0 +1,106 @@
+# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Tests for _create_void_ptr_wrapper name handling and sanitize_identifier.
+
+sanitize_identifier replaces non-alphanumeric/underscore characters with
+underscores before names reach exec(). _create_void_ptr_wrapper then validates
+that the *sanitized* name is a valid identifier (e.g. not empty, not
+leading-digit-only).
+"""
+
+import pytest
+from numba import types
+
+from cuda.compute._odr_helpers import _ArgMode, _ArgSpec, _create_void_ptr_wrapper
+from cuda.compute._utils import sanitize_identifier
+
+
+def _make_arg_specs():
+    """One float32 input, one float32 output."""
+    return [
+        _ArgSpec(types.float32, _ArgMode.LOAD),
+        _ArgSpec(types.float32, _ArgMode.STORE),
+    ]
+
+
+def _make_inner_sig():
+    return types.float32(types.float32)
+
+
+def _passthrough(x):
+    return x
+
+
+# ---------------------------------------------------------------------------
+# sanitize_identifier — the exec() injection boundary
+# ---------------------------------------------------------------------------
+
+
+def test_sanitize_lambda_name():
+    assert sanitize_identifier("<lambda>") == "_lambda_"
+
+
+def test_sanitize_newline():
+    assert sanitize_identifier("foo\nbar") == "foo_bar"
+
+
+def test_sanitize_space():
+    assert sanitize_identifier("my func") == "my_func"
+
+
+def test_sanitize_hyphen():
+    assert sanitize_identifier("my-func") == "my_func"
+
+
+def test_sanitize_semicolon_injection():
+    assert sanitize_identifier("f; import os") == "f__import_os"
+
+
+def test_sanitize_plain_name_unchanged():
+    assert sanitize_identifier("my_op") == "my_op"
+
+
+# ---------------------------------------------------------------------------
+# _create_void_ptr_wrapper — names that sanitize to a valid identifier are OK
+# ---------------------------------------------------------------------------
+
+
+def test_lambda_name_is_accepted():
+    """Lambdas have __name__ == '<lambda>'; sanitizes to '_lambda_'."""
+    op = lambda x: x  # noqa: E731
+    _create_void_ptr_wrapper(op, op.__name__, _make_arg_specs(), _make_inner_sig())
+
+
+def test_newline_in_name_is_accepted():
+    """Newlines sanitize to underscores — must not raise."""
+    _create_void_ptr_wrapper(
+        _passthrough, "foo\nbar", _make_arg_specs(), _make_inner_sig()
+    )
+
+
+def test_plain_name_is_accepted():
+    _create_void_ptr_wrapper(
+        _passthrough, "my_op", _make_arg_specs(), _make_inner_sig()
+    )
+
+
+# ---------------------------------------------------------------------------
+# _create_void_ptr_wrapper — names that sanitize to an invalid identifier
+# ---------------------------------------------------------------------------
+
+
+def test_empty_name_is_rejected():
+    """Empty string sanitizes to empty string, which is not a valid identifier."""
+    with pytest.raises(ValueError, match="cannot be sanitized into a valid identifier"):
+        _create_void_ptr_wrapper(_passthrough, "", _make_arg_specs(), _make_inner_sig())
+
+
+def test_digits_only_name_is_rejected():
+    """'123' sanitizes to '123', which is not a valid identifier (leading digit)."""
+    with pytest.raises(ValueError, match="cannot be sanitized into a valid identifier"):
+        _create_void_ptr_wrapper(
+            _passthrough, "123", _make_arg_specs(), _make_inner_sig()
+        )
diff --git a/cccl_upstream/python/cuda_cccl/tests/compute/test_zip_iterator.py b/cccl_upstream/python/cuda_cccl/tests/compute/test_zip_iterator.py
new file mode 100644
index 00000000..973e5a3c
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/compute/test_zip_iterator.py
@@ -0,0 +1,755 @@
+# Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+import numpy as np
+import pytest
+from _utils.device_array import DeviceArray, get_compute_capability
+
+import cuda.compute
+from cuda.compute import (
+    CountingIterator,
+    TransformIterator,
+    ZipIterator,
+    gpu_struct,
+)
+
+
+@pytest.mark.parametrize("num_items", [10, 1_000, 100_000])
+def test_zip_iterator_basic(num_items):
+    @gpu_struct
+    class Pair:
+        first: np.int64
+        second: np.float32
+
+    def sum_pairs(p1, p2):
+        return Pair(p1[0] + p2[0], p1[1] + p2[1])
+
+    h_input1 = np.arange(num_items, dtype=np.int64)
+    h_input2 = np.arange(num_items, dtype=np.float32)
+    d_input1 = DeviceArray.from_numpy(h_input1)
+    d_input2 = DeviceArray.from_numpy(h_input2)
+
+    zip_it = ZipIterator(d_input1, d_input2)
+
+    d_output = DeviceArray.empty(1, Pair.dtype)
+    h_init = Pair(0, 0.0)
+
+    cuda.compute.reduce_into(
+        d_in=zip_it, d_out=d_output, num_items=num_items, op=sum_pairs, h_init=h_init
+    )
+
+    expected_first = h_input1.sum()
+    expected_second = h_input2.sum()
+
+    result = d_output.copy_to_host()[0]
+    np.testing.assert_array_equal(result["first"], expected_first)
+    np.testing.assert_allclose(result["second"], expected_second, rtol=1e-6)
+
+
+@pytest.mark.parametrize("num_items", [10, 1_000, 100_000])
+def test_zip_iterator_with_counting_iterator(num_items):
+    """Test ZipIterator with counting iterator and numpy struct dtype as initial value."""
+
+    def max_by_value(p1, p2):
+        # Return the pair with the larger value
+        return p1 if p1[1] > p2[1] else p2
+
+    counting_it = CountingIterator(np.int32(0))
+    h_arr = np.arange(num_items, dtype=np.int32)
+    d_arr = DeviceArray.from_numpy(h_arr)
+
+    zip_it = ZipIterator(counting_it, d_arr)
+
+    dtype = np.dtype([("index", np.int32), ("value", np.int32)], align=True)
+    h_init = np.asarray([(-1, -1)], dtype=dtype)
+
+    d_output = DeviceArray.empty(1, dtype)
+
+    cuda.compute.reduce_into(
+        d_in=zip_it, d_out=d_output, num_items=num_items, op=max_by_value, h_init=h_init
+    )
+
+    result = d_output.copy_to_host()[0]
+
+    expected_index = np.argmax(h_arr)
+    expected_value = h_arr[expected_index]
+
+    assert result["index"] == expected_index
+    assert result["value"] == expected_value
+
+
+@pytest.mark.parametrize("num_items", [10, 1_000, 100_000])
+def test_zip_iterator_with_counting_iterator_and_transform(num_items):
+    @gpu_struct
+    class IndexValuePair:
+        index: np.int32
+        value: np.int64
+
+    def max_by_value(p1, p2):
+        return p1 if p1[1] > p2[1] else p2
+
+    counting_it = CountingIterator(np.int32(0))
+    h_arr = np.arange(num_items, dtype=np.int32)
+    d_arr = DeviceArray.from_numpy(h_arr)
+
+    def double_op(x):
+        return x * 2
+
+    transform_it = TransformIterator(d_arr, double_op)
+
+    zip_it = ZipIterator(counting_it, transform_it)
+
+    d_output = DeviceArray.empty(1, IndexValuePair.dtype)
+    h_init = IndexValuePair(-1, -1)
+
+    cuda.compute.reduce_into(
+        d_in=zip_it, d_out=d_output, num_items=num_items, op=max_by_value, h_init=h_init
+    )
+
+    result = d_output.copy_to_host()[0]
+
+    expected_index = np.argmax(h_arr)
+    expected_value = h_arr[expected_index] * 2
+
+    assert result["index"] == expected_index
+    assert result["value"] == expected_value
+
+
+@pytest.mark.parametrize("num_items", [10, 1_000, 100_000])
+def test_zip_iterator_n_iterators(num_items):
+    """Test generalized ZipIterator with N iterators (3 in this case)."""
+
+    @gpu_struct
+    class Triple:
+        first: np.int64
+        second: np.float32
+        third: np.int64
+
+    def sum_triples(t1, t2):
+        return Triple(t1[0] + t2[0], t1[1] + t2[1], t1[2] + t2[2])
+
+    h_input1 = np.arange(num_items, dtype=np.int64)
+    h_input2 = np.arange(num_items, dtype=np.float32)
+    d_input1 = DeviceArray.from_numpy(h_input1)
+    d_input2 = DeviceArray.from_numpy(h_input2)
+    counting_it = CountingIterator(np.int64(10))
+
+    zip_it = ZipIterator(d_input1, d_input2, counting_it)
+
+    d_output = DeviceArray.empty(1, Triple.dtype)
+    h_init = Triple(0, 0.0, 0)
+
+    cuda.compute.reduce_into(
+        d_in=zip_it, d_out=d_output, num_items=num_items, op=sum_triples, h_init=h_init
+    )
+
+    result = d_output.copy_to_host()[0]
+
+    expected_first = h_input1.sum()
+    expected_second = h_input2.sum()
+    expected_third = np.arange(10, 10 + num_items).sum()
+
+    np.testing.assert_array_equal(result["first"], expected_first)
+    np.testing.assert_allclose(result["second"], expected_second, rtol=1e-6)
+    np.testing.assert_array_equal(result["third"], expected_third)
+
+
+@pytest.mark.parametrize("num_items", [10, 1_000, 100_000])
+def test_zip_iterator_single_iterator(num_items):
+    """Test ZipIterator with a single iterator."""
+
+    @gpu_struct
+    class Single:
+        value: np.int64
+
+    def sum_singles(s1, s2):
+        return Single(s1[0] + s2[0])
+
+    h_input = np.arange(num_items, dtype=np.int64)
+    d_input = DeviceArray.from_numpy(h_input)
+
+    zip_it = ZipIterator(d_input)
+
+    d_output = DeviceArray.empty(1, Single.dtype)
+    h_init = Single(0)
+
+    cuda.compute.reduce_into(
+        d_in=zip_it, d_out=d_output, num_items=num_items, op=sum_singles, h_init=h_init
+    )
+
+    result = d_output.copy_to_host()[0]
+
+    expected_value = h_input.sum()
+    assert result["value"] == expected_value
+
+
+@pytest.mark.parametrize("num_items", [10, 1_000])
+def test_zip_iterator_with_transform(num_items):
+    @gpu_struct
+    class TransformedPair:
+        sum_indices: np.int32
+        product_values: np.int64
+
+    def binary_transform(pair1, pair2):
+        return TransformedPair(pair1[0] + pair2[0], pair1[1] * pair2[1])
+
+    counting_it1 = CountingIterator(np.int32(0))
+    h_arr1 = np.arange(num_items, dtype=np.int32)
+    d_arr1 = DeviceArray.from_numpy(h_arr1)
+    zip_it1 = ZipIterator(counting_it1, d_arr1)
+
+    counting_it2 = CountingIterator(np.int32(0))
+    h_arr2 = np.arange(num_items, dtype=np.int32)
+    d_arr2 = DeviceArray.from_numpy(h_arr2)
+    zip_it2 = ZipIterator(counting_it2, d_arr2)
+
+    d_output = DeviceArray.empty(num_items, TransformedPair.dtype)
+
+    cuda.compute.binary_transform(
+        d_in1=zip_it1,
+        d_in2=zip_it2,
+        d_out=d_output,
+        op=binary_transform,
+        num_items=num_items,
+    )
+
+    result = d_output.copy_to_host()
+
+    expected_sum_indices = h_arr1 + h_arr2
+    expected_product_values = h_arr1 * h_arr2
+
+    for i, result_item in enumerate(result):
+        assert result_item["sum_indices"] == expected_sum_indices[i]
+        assert result[i]["product_values"] == expected_product_values[i]
+
+
+@pytest.mark.parametrize("num_items", [10, 1_000])
+def test_zip_iterator_with_scan(num_items):
+    """Test ZipIterator with scan operations."""
+
+    @gpu_struct
+    class Pair:
+        first_min: np.int64
+        second_min: np.int64
+
+    def min_pairs(p1, p2):
+        # p1 is the accumulated result, p2 is the current input
+        # Compute running minimums for both arrays
+        return Pair(min(p1[0], p2[0]), min(p1[1], p2[1]))
+
+    # Create two randomized arrays to make min operations interesting
+    h_arr1 = np.random.randint(0, 1000, num_items, dtype=np.int64)
+    h_arr2 = np.random.randint(0, 1000, num_items, dtype=np.int64)
+    d_arr1 = DeviceArray.from_numpy(h_arr1)
+    d_arr2 = DeviceArray.from_numpy(h_arr2)
+
+    zip_it = ZipIterator(d_arr1, d_arr2)
+
+    d_output = DeviceArray.empty(num_items, Pair.dtype)
+    h_init = Pair(np.iinfo(np.int64).max, np.iinfo(np.int64).max)
+
+    cuda.compute.inclusive_scan(
+        d_in=zip_it,
+        d_out=d_output,
+        op=min_pairs,
+        init_value=h_init,
+        num_items=num_items,
+    )
+
+    result = d_output.copy_to_host()
+
+    # Verify the scan operation produces running minimums for both arrays
+    expected_first_running_mins = np.minimum.accumulate(h_arr1)
+    expected_second_running_mins = np.minimum.accumulate(h_arr2)
+
+    for i, result_item in enumerate(result):
+        assert result_item["first_min"] == expected_first_running_mins[i]
+        assert result_item["second_min"] == expected_second_running_mins[i]
+
+
+@pytest.mark.parametrize("num_items", [10, 1000])
+def test_output_zip_iterator_with_scan(monkeypatch, num_items):
+    """Test ZipIterator as output iterator with scan operations."""
+    # Skip SASS check for CC 8.0+ due to LDL/STL CI failure.
+    cc_major, _ = get_compute_capability()
+    if cc_major >= 8:
+        monkeypatch.setattr(
+            cuda.compute._cccl_interop,
+            "_check_sass",
+            False,
+        )
+
+    h_in1 = np.random.randint(0, 1000, num_items, dtype=np.int64)
+    h_in2 = np.random.randint(0, 1000, num_items, dtype=np.int64)
+    d_in1 = DeviceArray.from_numpy(h_in1)
+    d_in2 = DeviceArray.from_numpy(h_in2)
+
+    zip_it = ZipIterator(d_in1, d_in2)
+
+    d_out1 = DeviceArray.empty(h_in1.shape, h_in1.dtype)
+    d_out2 = DeviceArray.empty(h_in2.shape, h_in2.dtype)
+
+    zip_out_it = ZipIterator(d_out1, d_out2)
+
+    def add_pairs(p1, p2):
+        return p1[0] + p2[0], p1[1] + p2[1]
+
+    cuda.compute.inclusive_scan(
+        d_in=zip_it,
+        d_out=zip_out_it,
+        op=add_pairs,
+        init_value=None,
+        num_items=num_items,
+    )
+
+    expected_out1 = np.empty_like(h_in1)
+    expected_out2 = np.empty_like(h_in2)
+
+    # First element is just the input
+    expected_out1[0] = h_in1[0]
+    expected_out2[0] = h_in2[0]
+    for i in range(1, num_items):
+        expected_out1[i] = expected_out1[i - 1] + h_in1[i]
+        expected_out2[i] = expected_out2[i - 1] + h_in2[i]
+
+    np.testing.assert_array_equal(d_out1.copy_to_host(), expected_out1)
+    np.testing.assert_array_equal(d_out2.copy_to_host(), expected_out2)
+
+
+def test_nested_zip_iterators():
+    """Test that ZipIterator can be nested inside another ZipIterator.
+
+    This creates a structure like: ZipIterator(ZipIterator(a, b), c)
+    which should produce values with a nested structure.
+    """
+
+    InnerPair = gpu_struct({"first": np.int32, "second": np.int64})
+    OuterTriple = gpu_struct({"inner": InnerPair, "third": np.float32})
+
+    def sum_nested_zips(v1, v2):
+        return OuterTriple(
+            InnerPair(
+                v1.inner.first + v2.inner.first, v1.inner.second + v2.inner.second
+            ),
+            v1.third + v2.third,
+        )
+
+    num_items = 100
+
+    # Create three input arrays
+    h_input_a = np.arange(num_items, dtype=np.int32)
+    h_input_b = np.arange(num_items, dtype=np.int64) * 2
+    h_input_c = np.arange(num_items, dtype=np.float32) * 3.0
+    d_input_a = DeviceArray.from_numpy(h_input_a)
+    d_input_b = DeviceArray.from_numpy(h_input_b)
+    d_input_c = DeviceArray.from_numpy(h_input_c)
+
+    # Create an inner zip iterator combining a and b
+    inner_zip = ZipIterator(d_input_a, d_input_b)
+
+    # Create an outer zip iterator combining inner_zip and c
+    outer_zip = ZipIterator(inner_zip, d_input_c)
+
+    # Perform reduction
+    d_output = DeviceArray.empty(1, OuterTriple.dtype)
+    h_init = OuterTriple(InnerPair(0, 0), 0.0)
+
+    cuda.compute.reduce_into(
+        d_in=outer_zip,
+        d_out=d_output,
+        num_items=num_items,
+        op=sum_nested_zips,
+        h_init=h_init,
+    )
+
+    result = d_output.copy_to_host()[0]
+
+    # Calculate expected values
+    expected_first = h_input_a.sum()
+    expected_second = h_input_b.sum()
+    expected_third = h_input_c.sum()
+
+    assert result["inner"]["first"] == expected_first, (
+        f"Expected inner.first={expected_first}, got {result['inner']['first']}"
+    )
+    assert result["inner"]["second"] == expected_second, (
+        f"Expected inner.second={expected_second}, got {result['inner']['second']}"
+    )
+    assert np.isclose(result["third"], expected_third, rtol=1e-5), (
+        f"Expected third={expected_third}, got {result['third']}"
+    )
+
+
+# Test deeply nested zip iterators
+def test_deeply_nested_zip_iterators():
+    """Test 3 levels of nested zip iterators."""
+    # outer_zip produces a struct like: {value_0: {value_0: int32, value_1: float32}, value_1: int64}
+    # Define matching struct types with our own names
+    InnerPair = gpu_struct({"a": np.int32, "b": np.float32})
+    OuterPair = gpu_struct({"inner": InnerPair, "c": np.int64})
+
+    def sum_nested_zips(v1, v2):
+        return OuterPair(
+            InnerPair(v1.inner.a + v2.inner.a, v1.inner.b + v2.inner.b),
+            v1.c + v2.c,
+        )
+
+    num_items = 100
+
+    h_input_a = np.arange(num_items, dtype=np.int32)
+    h_input_b = np.arange(num_items, dtype=np.float32)
+    h_input_c = np.arange(num_items, dtype=np.int64)
+    d_input_a = DeviceArray.from_numpy(h_input_a)
+    d_input_b = DeviceArray.from_numpy(h_input_b)
+    d_input_c = DeviceArray.from_numpy(h_input_c)
+
+    inner_zip = ZipIterator(d_input_a, d_input_b)
+    outer_zip = ZipIterator(inner_zip, d_input_c)
+
+    d_output = DeviceArray.empty(1, OuterPair.dtype)
+    h_init = OuterPair(InnerPair(0, 0.0), 0)
+
+    cuda.compute.reduce_into(
+        d_in=outer_zip,
+        d_out=d_output,
+        num_items=num_items,
+        op=sum_nested_zips,
+        h_init=h_init,
+    )
+
+    result = d_output.copy_to_host()[0]
+
+    # outer_zip produces: {value_0: {value_0: int32, value_1: float32}, value_1: int64}
+    # which maps to our OuterPair: {inner: {a: int32, b: float32}, c: int64}
+    expected_a = h_input_a.sum()  # int32
+    expected_b = h_input_b.sum()  # float32
+    expected_c = h_input_c.sum()  # int64
+
+    assert result["inner"]["a"] == expected_a
+    assert np.isclose(result["inner"]["b"], expected_b)
+    assert result["c"] == expected_c
+
+
+@pytest.mark.parametrize("num_items", [10, 1000])
+@pytest.mark.parametrize(
+    "dtype_map",
+    [
+        {"x": np.float32, "y": np.float32},
+        {"x": np.float64, "y": np.float32},
+    ],
+)
+def test_nested_output_zip_iterator_with_scan(monkeypatch, num_items, dtype_map):
+    cc_major, _ = get_compute_capability()
+    if cc_major >= 8:
+        monkeypatch.setattr(
+            cuda.compute._cccl_interop,
+            "_check_sass",
+            False,
+        )
+
+    Vec2 = gpu_struct(dtype_map)
+
+    h_in1 = np.zeros(num_items, dtype=Vec2.dtype)
+    h_in2 = np.zeros(num_items, dtype=Vec2.dtype)
+    for i in range(num_items):
+        h_in1[i]["x"] = float(i)
+        h_in1[i]["y"] = float(i * 2)
+        h_in2[i]["x"] = float(i * 10)
+        h_in2[i]["y"] = float(i * 20)
+
+    d_in1 = DeviceArray.from_numpy(h_in1)
+    d_in2 = DeviceArray.from_numpy(h_in2)
+
+    zip_it = ZipIterator(d_in1, d_in2)
+
+    d_out1 = DeviceArray.empty(h_in1.shape, h_in1.dtype)
+    d_out2 = DeviceArray.empty(h_in2.shape, h_in2.dtype)
+
+    zip_out_it = ZipIterator(d_out1, d_out2)
+
+    def add_vec2_pairs(v1, v2):
+        result1 = (v1[0].x + v2[0].x, v1[0].y + v2[0].y)
+        result2 = (v1[1].x + v2[1].x, v1[1].y + v2[1].y)
+        return Vec2(result1[0], result1[1]), Vec2(result2[0], result2[1])
+
+    cuda.compute.inclusive_scan(
+        d_in=zip_it,
+        d_out=zip_out_it,
+        op=add_vec2_pairs,
+        init_value=None,
+        num_items=num_items,
+    )
+
+    expected_out1 = np.empty_like(h_in1)
+    expected_out2 = np.empty_like(h_in2)
+
+    expected_out1[0] = h_in1[0]
+    expected_out2[0] = h_in2[0]
+    for i in range(1, num_items):
+        expected_out1[i]["x"] = expected_out1[i - 1]["x"] + h_in1[i]["x"]
+        expected_out1[i]["y"] = expected_out1[i - 1]["y"] + h_in1[i]["y"]
+        expected_out2[i]["x"] = expected_out2[i - 1]["x"] + h_in2[i]["x"]
+        expected_out2[i]["y"] = expected_out2[i - 1]["y"] + h_in2[i]["y"]
+
+    np.testing.assert_array_equal(d_out1.copy_to_host(), expected_out1)
+    np.testing.assert_array_equal(d_out2.copy_to_host(), expected_out2)
+
+
+def test_zip_iterator_of_transform_iterator_kind():
+    d_arr = DeviceArray.from_numpy(np.arange(10, dtype=np.int64))
+
+    def f(x):
+        return x
+
+    def g(x):
+        return x + 1
+
+    it1 = ZipIterator(TransformIterator(d_arr, f))
+    it2 = ZipIterator(TransformIterator(d_arr, g))
+    assert it1.kind != it2.kind
+
+
+def test_caching_zip_iterator():
+    """Test that iterator compilation is cached across instances with the same structure."""
+    from cuda.compute._cpp_compile import compile_cpp_op_code
+
+    # Test 1: Iterators with same structure should have same kind
+    z1 = ZipIterator(CountingIterator(np.int32(0)))
+    z2 = ZipIterator(CountingIterator(np.int32(100)))  # Different value, same type
+    assert z1.kind == z2.kind, "Same structure should have same kind"
+
+    # Test 2: Different types should have different kind
+    z3 = ZipIterator(CountingIterator(np.int64(0)))
+    assert z1.kind != z3.kind, "Different types should have different kind"
+
+    # Test 3: Verify compilation caching with cache statistics
+    # Clear cache to get clean measurements
+    compile_cpp_op_code.cache_clear()
+
+    # Create multiple instances with same structure
+    iterators = []
+    for i in range(5):
+        d_arr = DeviceArray.from_numpy(
+            np.arange(i * 10, (i + 1) * 10, dtype=np.float32)
+        )
+        z = ZipIterator(d_arr)
+        # Trigger compilation by accessing LTOIR
+        z.get_advance_op()
+        z.get_input_deref_op()
+        iterators.append(z)
+
+    # Check cache statistics
+    cache_info = compile_cpp_op_code.cache_info()
+
+    # With deterministic symbols: only first instance misses, rest hit the cache
+    # With random UUIDs: all instances would miss
+    assert cache_info.hits >= 3, (
+        f"Expected multiple cache hits for same structure, got {cache_info.hits} hits, "
+        f"{cache_info.misses} misses"
+    )
+
+    # Test 4: Arrays with different dtypes should not share cache
+    compile_cpp_op_code.cache_clear()
+
+    z_int32 = ZipIterator(DeviceArray.from_numpy(np.arange(10, dtype=np.int32)))
+    z_int32.get_advance_op()
+    z_int32.get_input_deref_op()
+    misses_after_first = compile_cpp_op_code.cache_info().misses
+
+    z_int64 = ZipIterator(DeviceArray.from_numpy(np.arange(10, dtype=np.int64)))
+    z_int64.get_advance_op()
+    z_int64.get_input_deref_op()
+    misses_after_second = compile_cpp_op_code.cache_info().misses
+
+    # Different dtypes should not share cache
+    assert misses_after_second > misses_after_first, (
+        "Different dtypes should cause cache miss"
+    )
+    assert z_int32.kind != z_int64.kind
+
+    # Test 5: Verify basic iterator types share compilation cache
+    compile_cpp_op_code.cache_clear()
+
+    # CountingIterators with same type
+    count_iters = [ZipIterator(CountingIterator(np.int32(i * 10))) for i in range(3)]
+    for z in count_iters:
+        z.get_advance_op()
+        z.get_input_deref_op()
+
+    cache_info = compile_cpp_op_code.cache_info()
+    assert cache_info.hits >= 2, (
+        f"CountingIterators with same type should share cache, got {cache_info}"
+    )
+
+    # All should have same kind
+    kinds = [z.kind for z in count_iters]
+    assert len(set(kinds)) == 1, "Same CountingIterator types should have same kind"
+
+
+def test_compilation_caching_across_iterator_types():
+    """Test that compilation caching works across different iterator types."""
+    from cuda.compute import ConstantIterator
+    from cuda.compute._cpp_compile import compile_cpp_op_code
+
+    # Test ConstantIterator caching
+    compile_cpp_op_code.cache_clear()
+
+    const_iterators = [ConstantIterator(np.int32(i)) for i in range(5)]
+    for it in const_iterators:
+        it.get_advance_op()
+        it.get_input_deref_op()
+
+    cache_info = compile_cpp_op_code.cache_info()
+    assert cache_info.hits >= 3, (
+        f"ConstantIterator: Expected cache hits across instances, "
+        f"got {cache_info.hits} hits, {cache_info.misses} misses"
+    )
+
+    # All should have same kind
+    kinds = [it.kind for it in const_iterators]
+    assert len(set(kinds)) == 1, (
+        "All ConstantIterators with same type should have same kind"
+    )
+
+    # Test CountingIterator caching
+    compile_cpp_op_code.cache_clear()
+
+    counting_iterators = [CountingIterator(np.int64(i * 100)) for i in range(5)]
+    for it in counting_iterators:
+        it.get_advance_op()
+        it.get_input_deref_op()
+
+    cache_info = compile_cpp_op_code.cache_info()
+    assert cache_info.hits >= 3, (
+        f"CountingIterator: Expected cache hits across instances, "
+        f"got {cache_info.hits} hits, {cache_info.misses} misses"
+    )
+
+    # All should have same kind
+    kinds = [it.kind for it in counting_iterators]
+    assert len(set(kinds)) == 1, (
+        "All CountingIterators with same type should have same kind"
+    )
+
+    # Test that different types don't incorrectly share cache
+    const_int32 = ConstantIterator(np.int32(5))
+    const_float32 = ConstantIterator(np.float32(5.0))
+    assert const_int32.kind != const_float32.kind, (
+        "Different types should have different kind"
+    )
+
+    count_int32 = CountingIterator(np.int32(0))
+    count_int64 = CountingIterator(np.int64(0))
+    assert count_int32.kind != count_int64.kind, (
+        "Different types should have different kind"
+    )
+
+
+def test_zip_iterator_advance():
+    """Test ZipIterator.__add__ advances all child iterators."""
+
+    @gpu_struct
+    class Pair:
+        first: np.int32
+        second: np.int32
+
+    num_items = 100
+    offset = 10
+
+    h_input1 = np.arange(num_items, dtype=np.int32)
+    h_input2 = np.arange(num_items, dtype=np.int32) * 2
+    d_input1 = DeviceArray.from_numpy(h_input1)
+    d_input2 = DeviceArray.from_numpy(h_input2)
+
+    # Create base zip iterator
+    zip_it = ZipIterator(d_input1, d_input2)
+
+    # Advance by offset
+    advanced_zip_it = zip_it + offset
+
+    # Reduce starting from the advanced position
+    def sum_pairs(p1, p2):
+        return Pair(p1[0] + p2[0], p1[1] + p2[1])
+
+    h_init = Pair(0, 0)
+    d_output = DeviceArray.empty(1, Pair.dtype)
+
+    remaining_items = num_items - offset
+    cuda.compute.reduce_into(
+        d_in=advanced_zip_it,
+        d_out=d_output,
+        num_items=remaining_items,
+        op=sum_pairs,
+        h_init=h_init,
+    )
+
+    result = d_output.copy_to_host()[0]
+
+    # Expected values should be sum from offset onwards
+    expected_first = h_input1[offset:].sum()
+    expected_second = h_input2[offset:].sum()
+
+    assert result["first"] == expected_first
+    assert result["second"] == expected_second
+
+
+def test_nested_zip_iterator_advance():
+    """Test that advancing a nested ZipIterator advances all child iterators."""
+    InnerPair = gpu_struct({"first": np.int32, "second": np.int64})
+    OuterTriple = gpu_struct({"inner": InnerPair, "third": np.float32})
+
+    def sum_nested_zips(v1, v2):
+        return OuterTriple(
+            InnerPair(
+                v1.inner.first + v2.inner.first, v1.inner.second + v2.inner.second
+            ),
+            v1.third + v2.third,
+        )
+
+    num_items = 100
+    offset = 15
+
+    # Create three input arrays
+    h_input_a = np.arange(num_items, dtype=np.int32)
+    h_input_b = np.arange(num_items, dtype=np.int64) * 2
+    h_input_c = np.arange(num_items, dtype=np.float32) * 3.0
+    d_input_a = DeviceArray.from_numpy(h_input_a)
+    d_input_b = DeviceArray.from_numpy(h_input_b)
+    d_input_c = DeviceArray.from_numpy(h_input_c)
+
+    # Create nested zip: ZipIterator(ZipIterator(a, b), c)
+    inner_zip = ZipIterator(d_input_a, d_input_b)
+    outer_zip = ZipIterator(inner_zip, d_input_c)
+
+    # Advance the nested zip by offset
+    advanced_outer_zip = outer_zip + offset
+
+    # Perform reduction from the advanced position
+    d_output = DeviceArray.empty(1, OuterTriple.dtype)
+    h_init = OuterTriple(InnerPair(0, 0), 0.0)
+
+    remaining_items = num_items - offset
+    cuda.compute.reduce_into(
+        d_in=advanced_outer_zip,
+        d_out=d_output,
+        num_items=remaining_items,
+        op=sum_nested_zips,
+        h_init=h_init,
+    )
+
+    result = d_output.copy_to_host()[0]
+
+    # Calculate expected values from offset onwards
+    expected_first = h_input_a[offset:].sum()
+    expected_second = h_input_b[offset:].sum()
+    expected_third = h_input_c[offset:].sum()
+
+    assert result["inner"]["first"] == expected_first, (
+        f"Expected inner.first={expected_first}, got {result['inner']['first']}"
+    )
+    assert result["inner"]["second"] == expected_second, (
+        f"Expected inner.second={expected_second}, got {result['inner']['second']}"
+    )
+    assert np.isclose(result["third"], expected_third, rtol=1e-5), (
+        f"Expected third={expected_third}, got {result['third']}"
+    )
diff --git a/cccl_upstream/python/cuda_cccl/tests/headers/test_cuda_cccl.py b/cccl_upstream/python/cuda_cccl/tests/headers/test_cuda_cccl.py
new file mode 100644
index 00000000..abf057de
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/headers/test_cuda_cccl.py
@@ -0,0 +1,64 @@
+import pytest
+
+import cuda.cccl as cccl
+
+
+def test_version():
+    v = cccl.__version__
+    assert isinstance(v, str)
+
+
+@pytest.fixture
+def inc_paths():
+    return cccl.get_include_paths()
+
+
+def test_headers_has_cuda(inc_paths):
+    assert hasattr(inc_paths, "cuda")
+
+
+def test_headers_has_cub(inc_paths):
+    assert hasattr(inc_paths, "cub")
+
+
+def test_headers_has_cudacxx(inc_paths):
+    assert hasattr(inc_paths, "libcudacxx")
+
+
+def test_headers_has_thrust(inc_paths):
+    assert hasattr(inc_paths, "thrust")
+
+
+def test_headers_as_tuple(inc_paths):
+    tpl = inc_paths.as_tuple()
+    assert len(tpl) == 4
+
+    thrust_, cub_, cudacxx_, cuda_ = tpl
+    assert cuda_ == inc_paths.cuda
+    assert cub_ == inc_paths.cub
+    assert cudacxx_ == inc_paths.libcudacxx
+    assert thrust_ == inc_paths.thrust
+
+
+def test_cub_version(inc_paths):
+    cub_dir = inc_paths.cub / "cub"
+    cub_version = cub_dir / "version.cuh"
+    assert cub_version.exists()
+
+
+def test_thrust_version(inc_paths):
+    thrust_dir = inc_paths.thrust / "thrust"
+    thrust_version = thrust_dir / "version.h"
+    assert thrust_version.exists()
+
+
+def test_cudacxx_version(inc_paths):
+    cudacxx_dir = inc_paths.libcudacxx / "cuda"
+    cudacxx_version = cudacxx_dir / "version"
+    assert cudacxx_version.exists()
+
+
+def test_nv_target(inc_paths):
+    nv_dir = inc_paths.libcudacxx / "nv"
+    nv_target = nv_dir / "target"
+    assert nv_target.exists()
diff --git a/cccl_upstream/python/cuda_cccl/tests/test_examples.py b/cccl_upstream/python/cuda_cccl/tests/test_examples.py
new file mode 100644
index 00000000..c52d2e7c
--- /dev/null
+++ b/cccl_upstream/python/cuda_cccl/tests/test_examples.py
@@ -0,0 +1,166 @@
+#!/usr/bin/env python3
+# Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+#
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+"""
+Test runner for CCCL examples.
+
+This module automatically discovers and runs all example scripts from the
+compute directory to ensure they execute without errors.
+"""
+
+import importlib
+import importlib.util
+import inspect
+import sys
+import traceback
+from pathlib import Path
+
+import pytest
+
+
+def discover_examples():
+    """Automatically discover all example files and their functions."""
+    tests_dir = Path(__file__).parent
+    examples = []
+
+    example_directories = [
+        ("Compute", "compute/examples"),
+    ]
+
+    for framework, example_dir in example_directories:
+        example_path = tests_dir / example_dir
+        if not example_path.exists():
+            continue
+
+        # Find all Python files in subdirectories
+        for python_file in example_path.rglob("*.py"):
+            if (
+                python_file.name == "__init__.py"
+                or python_file.name == "test_examples.py"
+            ):
+                continue
+
+            # Calculate the relative path from the tests directory
+            rel_path = python_file.relative_to(tests_dir)
+
+            # Convert path to module name (OS-agnostic)
+            # Example: compute/examples/reduce/reduce_basic.py
+            #          -> compute.examples.reduce.reduce_basic
+            module_name = ".".join(rel_path.with_suffix("").parts)
+
+            # Extract category info for display
+            parts = python_file.relative_to(example_path).parts
+            if len(parts) >= 2:
+                category = parts[0].title()  # Block, Warp, Reduction, etc.
+                filename = parts[1].replace(".py", "").replace("_", " ").title()
+                display_name = f"{framework} - {category} - {filename}"
+            elif len(parts) == 1:
+                filename = parts[-1].replace(".py", "").replace("_", " ").title()
+                display_name = f"{framework} - {filename}"
+            else:
+                display_name = rel_path.stem.replace("_", " ").title()
+
+            examples.append((display_name, module_name))
+
+    return sorted(examples)
+
+
+def run_example_module(module_name, display_name):
+    """Run all example functions from a module."""
+    try:
+        print(f"Testing {display_name}...")
+
+        # Import the module. Examples may sys.exit(0) at module load to skip
+        # when their preconditions aren't met on this build (e.g. v2-only
+        # RawOp examples loaded against a v1 wheel). Treat that as a pass.
+        try:
+            module = importlib.import_module(module_name)
+        except SystemExit as exit_exc:
+            if exit_exc.code in (None, 0):
+                print(f"  {display_name} skipped (sys.exit({exit_exc.code}))")
+                return True
+            raise
+
+        # Check if module has a main function - if so, run it
+        if hasattr(module, "__main__") or hasattr(module, "main"):
+            # Call main if it exists
+            if hasattr(module, "main"):
+                module.main()
+            else:
+                # Try to run the module as if it were called directly
+                exec(f"import {module_name}; {module_name}.__main__()")
+        else:
+            # Find and run all example functions (those ending with _example)
+            example_functions = []
+            for name, obj in inspect.getmembers(module):
+                if (
+                    inspect.isfunction(obj)
+                    and name.endswith("_example")
+                    and not name.startswith("_")
+                ):
+                    example_functions.append((name, obj))
+
+            if example_functions:
+                for func_name, func in sorted(example_functions):
+                    print(f"  Running {func_name}...")
+                    func()
+            else:
+                # If no example functions found, try to run the module directly
+                # by checking if it has a __name__ == "__main__" block
+                print(f"  Running {module_name} as script...")
+                import os
+                import subprocess
+
+                module_file = module.__file__
+                if module_file:
+                    # Run the module as a script
+                    result = subprocess.run(
+                        [sys.executable, module_file],
+                        capture_output=True,
+                        text=True,
+                        cwd=os.path.dirname(module_file),
+                    )
+                    if result.returncode != 0:
+                        raise Exception(f"Module execution failed: {result.stderr}")
+                    print(f"  Output: {result.stdout.strip()}")
+
+        print(f"✓ {display_name} examples passed")
+        return True
+
+    except Exception as e:
+        print(f"✗ {display_name} examples failed: {e}")
+        traceback.print_exc()
+        return False
+
+
+# Create pytest-compatible test functions dynamically
+def create_test_functions():
+    """Create pytest-compatible test functions for each discovered example."""
+    examples = discover_examples()
+
+    for display_name, module_name in examples:
+        # Create a test function name from the module name
+        test_name = f"test_{module_name.replace('.', '_')}"
+
+        # Create the test function
+        def make_test_func(mod_name, disp_name):
+            def test_func():
+                assert run_example_module(mod_name, disp_name)
+
+            return test_func
+
+        # Add the test function to the global namespace
+        globals()[test_name] = make_test_func(module_name, display_name)
+        globals()[test_name].__name__ = test_name
+        globals()[test_name].__doc__ = f"Test {display_name} examples"
+        if module_name.startswith("compute.examples."):
+            globals()[test_name] = pytest.mark.skipif(
+                importlib.util.find_spec("cupy") is None,
+                reason="cuda.compute examples require the optional CuPy dependency",
+            )(globals()[test_name])
+
+
+# Create test functions for pytest
+create_test_functions()
diff --git a/cccl_upstream/test/CMakeLists.txt b/cccl_upstream/test/CMakeLists.txt
new file mode 100644
index 00000000..24c8f171
--- /dev/null
+++ b/cccl_upstream/test/CMakeLists.txt
@@ -0,0 +1 @@
+add_subdirectory(cmake)
diff --git a/cccl_upstream/test/cmake/CMakeLists.txt b/cccl_upstream/test/cmake/CMakeLists.txt
new file mode 100644
index 00000000..55d682ad
--- /dev/null
+++ b/cccl_upstream/test/cmake/CMakeLists.txt
@@ -0,0 +1,78 @@
+set(
+  cmake_opts
+  "-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}"
+  "-DCMAKE_MAKE_PROGRAM=${CMAKE_MAKE_PROGRAM}"
+  "-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}"
+)
+
+# Temporary installation prefix for tests against installed project:
+set(tmp_install_prefix "${CMAKE_CURRENT_BINARY_DIR}/test_install")
+
+foreach (root_type IN ITEMS SOURCE INSTALL)
+  if (root_type STREQUAL "INSTALL")
+    set(cccl_root "${tmp_install_prefix}")
+  else()
+    set(cccl_root "${CCCL_SOURCE_DIR}")
+  endif()
+
+  set(comps DEFAULT Thrust CUB libcudacxx)
+  if (CCCL_ENABLE_UNSTABLE)
+    list(APPEND comps cudax)
+  endif()
+
+  foreach (components IN LISTS comps)
+    set(package_types CCCL)
+    if (NOT components STREQUAL "DEFAULT")
+      list(APPEND package_types NATIVE)
+    endif()
+    if (root_type STREQUAL "SOURCE")
+      list(APPEND package_types SUBDIR)
+    endif()
+    foreach (package_type IN LISTS package_types)
+      string(TOLOWER "${root_type}.${package_type}.${components}" suffix)
+      cccl_add_compile_test(
+        test_name
+        cccl.test.cmake
+        test_export
+        "${suffix}"
+        ${cmake_opts}
+        "-DCCCL_ROOT=${cccl_root}"
+        "-DCOMPONENTS=${components}"
+        "-DPACKAGE_TYPE=${package_type}"
+        "-DCCCL_ENABLE_UNSTABLE=${CCCL_ENABLE_UNSTABLE}"
+      )
+
+      if (root_type STREQUAL "INSTALL")
+        set_tests_properties(
+          ${test_name}
+          PROPERTIES FIXTURES_REQUIRED install_tree
+        )
+      endif()
+    endforeach() # package_type
+  endforeach() # components
+endforeach() # root_type
+
+################################################################################
+# Install tree fixtures
+add_test(
+  NAME cccl.test.cmake.install_tree.install
+  # gersemi: off
+  COMMAND
+    "${CMAKE_COMMAND}"
+      --install "${CCCL_BINARY_DIR}"
+      --prefix "${tmp_install_prefix}"
+  # gersemi: on
+)
+set_tests_properties(
+  cccl.test.cmake.install_tree.install
+  PROPERTIES FIXTURES_SETUP install_tree
+)
+
+add_test(
+  NAME cccl.test.cmake.install_tree.cleanup
+  COMMAND "${CMAKE_COMMAND}" -E rm -rf "${tmp_install_prefix}"
+)
+set_tests_properties(
+  cccl.test.cmake.install_tree.cleanup
+  PROPERTIES FIXTURES_CLEANUP install_tree
+)
diff --git a/cccl_upstream/test/cmake/test_export/CMakeLists.txt b/cccl_upstream/test/cmake/test_export/CMakeLists.txt
new file mode 100644
index 00000000..c58c7ae1
--- /dev/null
+++ b/cccl_upstream/test/cmake/test_export/CMakeLists.txt
@@ -0,0 +1,166 @@
+# Test the CMake packages for CCCL and all subprojects.
+#
+# Parameters:
+# - CCCL_ROOT [Path] Root of the CCCL repo, or an installation root.
+# - COMPONENTS [StringList] {Thrust CUB libcudacxx} Which CCCL subprojects
+#   should be found.
+# - PACKAGE_TYPE [String] {CCCL | NATIVE | SUBDIR}:
+#   - CCCL -> `find_package(CCCL COMPONENTS <subproject>)`
+#   - NATIVE -> `find_package(<subproject>)`
+#   - SUBDIR -> `set(CCCL_REQUIRED_COMPONENTS <subproject>)`
+#               `add_subdirectory(${cccl_root})`
+
+cmake_minimum_required(VERSION 3.21)
+project(CCCLTestExport LANGUAGES CXX)
+
+include(CTest)
+enable_testing()
+
+set(CCCL_ROOT "" CACHE PATH "Root of the CCCL repo, or an installation root.")
+set(
+  COMPONENTS
+  ""
+  CACHE STRING
+  "DEFAULT for no components, or semi-colon delimited list of Thrust, CUB, and/or libcudacxx."
+)
+set(
+  PACKAGE_TYPE
+  ""
+  CACHE STRING
+  "CCCL: Find CCCL with subpackages as components; NATIVE: Find subpackages directly; SUBDIR: add_subdirectory(${CCCL_ROOT}"
+)
+set_property(CACHE PACKAGE_TYPE PROPERTY STRINGS CCCL NATIVE SUBDIR)
+
+message(STATUS "CCCL_ROOT=${CCCL_ROOT}")
+message(STATUS "COMPONENTS=${COMPONENTS}")
+message(STATUS "PACKAGE_TYPE=${PACKAGE_TYPE}")
+
+function(do_find_package pkg_name pkg_prefix)
+  list(
+    APPEND arg_list
+    REQUIRED
+    ${ARGN}
+    NO_DEFAULT_PATH
+    HINTS
+    "${pkg_prefix}"
+  )
+  list(JOIN arg_list " " arg_str)
+  message(STATUS "Executing: find_package(${pkg_name} ${arg_str})")
+  find_package(${pkg_name} ${arg_list})
+  if (NOT ${pkg_name}_FOUND)
+    message(FATAL_ERROR "Failed: find_package(${pkg_name} ${arg_str})")
+  endif()
+  # Re-execute find_package to ensure that repeated calls don't break:
+  find_package(${pkg_name} ${arg_list})
+endfunction()
+
+# Run find package with the requested configuration:
+if (PACKAGE_TYPE STREQUAL "CCCL")
+  if (COMPONENTS STREQUAL "DEFAULT")
+    do_find_package(CCCL "${CCCL_ROOT}")
+  else()
+    do_find_package(CCCL "${CCCL_ROOT}" COMPONENTS ${COMPONENTS})
+  endif()
+elseif (PACKAGE_TYPE STREQUAL "NATIVE")
+  if (COMPONENTS STREQUAL "DEFAULT")
+    message(
+      FATAL_ERROR
+      "COMPONENTS=DEFAULT incompatible with PACKAGE_TYPE=NATIVE"
+    )
+  endif()
+  foreach (component IN LISTS COMPONENTS)
+    do_find_package(${component} "${CCCL_ROOT}")
+  endforeach()
+elseif (PACKAGE_TYPE STREQUAL "SUBDIR")
+  if (COMPONENTS STREQUAL "DEFAULT")
+    set(CCCL_REQUIRED_COMPONENTS)
+  else()
+    set(CCCL_REQUIRED_COMPONENTS ${COMPONENTS})
+  endif()
+  add_subdirectory("${CCCL_ROOT}" "${CMAKE_CURRENT_BINARY_DIR}/subdir")
+else()
+  message(FATAL_ERROR "Invalid PACKAGE_TYPE: ${PACKAGE_TYPE}")
+endif()
+
+if (COMPONENTS STREQUAL "DEFAULT")
+  set(COMPONENTS libcudacxx CUB Thrust)
+  if (CCCL_ENABLE_UNSTABLE)
+    list(APPEND COMPONENTS cudax)
+  endif()
+endif()
+
+foreach (component IN LISTS COMPONENTS)
+  set(test_target version_check.${component})
+  set(component_target "${component}::${component}")
+  add_executable(${test_target} version_check.cxx)
+  target_compile_features(${test_target} PUBLIC cxx_std_17)
+  target_link_libraries(${test_target} PRIVATE ${component_target})
+  add_test(NAME ${test_target} COMMAND ${test_target})
+
+  if (component STREQUAL "libcudacxx")
+    math(
+      EXPR
+      component_cmake_version
+      "(${LIBCUDACXX_VERSION_MAJOR} * 1000000) +
+        ${LIBCUDACXX_VERSION_MINOR} * 1000 +
+        ${LIBCUDACXX_VERSION_PATCH}"
+    )
+    target_compile_definitions(
+      ${test_target}
+      PRIVATE
+        "VERSION_HEADER=cuda/std/version"
+        "VERSION_MACRO=_LIBCUDACXX_CUDA_API_VERSION"
+        "EXPECTED_VERSION=${component_cmake_version}"
+    )
+  elseif (component STREQUAL "CUB")
+    math(
+      EXPR
+      component_cmake_version
+      "(${CUB_VERSION_MAJOR} * 100000) +
+        ${CUB_VERSION_MINOR} * 100 +
+        ${CUB_VERSION_PATCH}"
+    )
+    target_compile_definitions(
+      ${test_target}
+      PRIVATE
+        "VERSION_HEADER=cub/version.cuh"
+        "VERSION_MACRO=CUB_VERSION"
+        "EXPECTED_VERSION=${component_cmake_version}"
+    )
+  elseif (component STREQUAL "Thrust")
+    math(
+      EXPR
+      component_cmake_version
+      "(${THRUST_VERSION_MAJOR} * 100000) +
+        ${THRUST_VERSION_MINOR} * 100 +
+        ${THRUST_VERSION_PATCH}"
+    )
+    target_compile_definitions(
+      ${test_target}
+      PRIVATE
+        "VERSION_HEADER=thrust/version.h"
+        "VERSION_MACRO=THRUST_VERSION"
+        "EXPECTED_VERSION=${component_cmake_version}"
+    )
+  elseif (component STREQUAL "cudax")
+    math(
+      EXPR
+      component_cmake_version
+      "(${CUDAX_VERSION_MAJOR} * 1000000) +
+        ${CUDAX_VERSION_MINOR} * 1000 +
+        ${CUDAX_VERSION_PATCH}"
+    )
+    target_compile_definitions(
+      ${test_target}
+      PRIVATE
+        "VERSION_HEADER=cuda/experimental/version.cuh"
+        "VERSION_MACRO=CUDAX_VERSION"
+        "EXPECTED_VERSION=${component_cmake_version}"
+    )
+  else()
+    message(
+      FATAL_ERROR
+      "Valid COMPONENTS are (case-sensitive): Thrust;CUB;libcudacxx;cudax"
+    )
+  endif()
+endforeach()
diff --git a/cccl_upstream/test/cmake/test_export/version_check.cxx b/cccl_upstream/test/cmake/test_export/version_check.cxx
new file mode 100644
index 00000000..c09e8d9c
--- /dev/null
+++ b/cccl_upstream/test/cmake/test_export/version_check.cxx
@@ -0,0 +1,20 @@
+// Compile with:
+//   -DVERSION_HEADER=include/path/for/version.h
+//   -DEXPECTED_VERSION=XXYYZZ
+//   -DVERSION_MACRO=PROJECT_VERSION
+
+#define HEADER <VERSION_HEADER>
+#include HEADER
+
+#include <cstdio>
+
+#define DETECTED_VERSION VERSION_MACRO
+
+int main()
+{
+  printf("Expected version: %d\n"
+         "Detected version: %d\n",
+         EXPECTED_VERSION,
+         VERSION_MACRO);
+  return EXPECTED_VERSION == DETECTED_VERSION ? 0 : 1;
+}
diff --git a/cccl_upstream/test/cuda_smoke/CMakeLists.txt b/cccl_upstream/test/cuda_smoke/CMakeLists.txt
new file mode 100644
index 00000000..272fe686
--- /dev/null
+++ b/cccl_upstream/test/cuda_smoke/CMakeLists.txt
@@ -0,0 +1,14 @@
+# Simple smoke test verifying the CUDA runtime is functional (device visible,
+# kernel launch + managed-memory round-trip work). Invoked explicitly from CI
+# via ci/build_common.sh before GPU test presets run. Not registered in CTest
+# (GPU-only; CPU test presets such as thrust-cpu would pick it up otherwise).
+#
+# Declare CUDA inside this subproject only so the root project() can stay
+# CXX-only (external consumers via add_subdirectory(cccl) see no change).
+project(CCCL_CUDA_SMOKE LANGUAGES CUDA)
+
+cccl_get_catch2()
+
+cccl_add_executable(${CCCL_CUDA_SMOKE_TARGET} SOURCES cuda_runtime_smoke.cu)
+
+target_link_libraries(${CCCL_CUDA_SMOKE_TARGET} PRIVATE Catch2::Catch2WithMain)
diff --git a/cccl_upstream/test/cuda_smoke/cuda_runtime_smoke.cu b/cccl_upstream/test/cuda_smoke/cuda_runtime_smoke.cu
new file mode 100644
index 00000000..c2feb3b4
--- /dev/null
+++ b/cccl_upstream/test/cuda_smoke/cuda_runtime_smoke.cu
@@ -0,0 +1,184 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDA C++ Core Libraries, under the Apache License v2.0 with
+// LLVM Exceptions. See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+#include <cuda_runtime.h>
+
+#include <catch2/catch_test_macros.hpp>
+
+#define CUDART_REQUIRE(call) REQUIRE((call) == cudaSuccess)
+
+__global__ void increment_kernel(int* p, int n)
+{
+  int idx = static_cast<int>(blockIdx.x * blockDim.x + threadIdx.x);
+  if (idx < n)
+  {
+    p[idx] += 1;
+  }
+}
+
+TEST_CASE("CUDA device is available", "[cuda_smoke]")
+{
+  int device_count = 0;
+  CUDART_REQUIRE(cudaGetDeviceCount(&device_count));
+  REQUIRE(device_count > 0);
+
+  CUDART_REQUIRE(cudaSetDevice(0));
+
+  cudaDeviceProp props{};
+  CUDART_REQUIRE(cudaGetDeviceProperties(&props, 0));
+  REQUIRE(props.name[0] != '\0');
+
+  REQUIRE(cudaGetLastError() == cudaSuccess);
+}
+
+TEST_CASE("cudaMallocManaged round-trip works", "[cuda_smoke][managed_memory]")
+{
+  (void) cudaGetLastError(); // clear any pre-existing error state
+
+  int managed_supported = 0;
+  CUDART_REQUIRE(cudaDeviceGetAttribute(&managed_supported, cudaDevAttrManagedMemory, 0));
+  if (!managed_supported)
+  {
+    SKIP("Device does not support managed memory (cudaDevAttrManagedMemory == 0).");
+  }
+
+  constexpr int n = 256;
+  int* p          = nullptr;
+  CUDART_REQUIRE(cudaMallocManaged(&p, n * sizeof(int)));
+
+  for (int i = 0; i < n; ++i) // host write
+  {
+    p[i] = i;
+  }
+  CUDART_REQUIRE(cudaDeviceSynchronize());
+
+  increment_kernel<<<4, 64>>>(p, n); // device transform
+  CUDART_REQUIRE(cudaGetLastError());
+  CUDART_REQUIRE(cudaDeviceSynchronize());
+
+  for (int i = 0; i < n; ++i) // host read-back
+  {
+    REQUIRE(p[i] == i + 1);
+  }
+
+  CUDART_REQUIRE(cudaFree(p));
+  REQUIRE(cudaGetLastError() == cudaSuccess);
+}
+
+// smoke test for GPU memory allocation/deallocation
+
+TEST_CASE("cudaMalloc/cudaFree round-trip works", "[cuda_smoke][device_memory]")
+{
+  (void) cudaGetLastError();
+
+  constexpr int n = 256;
+
+  int* d_ptr = nullptr;
+  CUDART_REQUIRE(cudaMalloc(&d_ptr, n * sizeof(int)));
+  REQUIRE(d_ptr != nullptr);
+
+  int h_ins[n];
+  for (int i = 0; i < n; ++i)
+  {
+    h_ins[i] = i;
+  }
+  CUDART_REQUIRE(cudaMemcpy(d_ptr, h_ins, n * sizeof(int), cudaMemcpyHostToDevice));
+
+  increment_kernel<<<4, 64>>>(d_ptr, n);
+  CUDART_REQUIRE(cudaGetLastError());
+  CUDART_REQUIRE(cudaDeviceSynchronize());
+
+  int h_outs[n];
+  CUDART_REQUIRE(cudaMemcpy(h_outs, d_ptr, n * sizeof(int), cudaMemcpyDeviceToHost));
+  for (int i = 0; i < n; ++i)
+  {
+    REQUIRE(h_outs[i] == i + 1);
+  }
+
+  CUDART_REQUIRE(cudaFree(d_ptr));
+  REQUIRE(cudaGetLastError() == cudaSuccess);
+}
+
+// smoke test for pinned host memory
+
+TEST_CASE("cudaMallocHost round-trip works", "[cuda_smoke][pinned_memory]")
+{
+  (void) cudaGetLastError();
+
+  constexpr int n = 256;
+
+  int* h_pinned = nullptr;
+  CUDART_REQUIRE(cudaMallocHost(&h_pinned, n * sizeof(int)));
+  REQUIRE(h_pinned != nullptr);
+
+  int* d_ptr = nullptr;
+  CUDART_REQUIRE(cudaMalloc(&d_ptr, n * sizeof(int)));
+  REQUIRE(d_ptr != nullptr);
+
+  for (int i = 0; i < n; ++i)
+  {
+    h_pinned[i] = i;
+  }
+  CUDART_REQUIRE(cudaMemcpy(d_ptr, h_pinned, n * sizeof(int), cudaMemcpyHostToDevice));
+
+  increment_kernel<<<4, 64>>>(d_ptr, n);
+  CUDART_REQUIRE(cudaGetLastError());
+  CUDART_REQUIRE(cudaDeviceSynchronize());
+
+  CUDART_REQUIRE(cudaMemcpy(h_pinned, d_ptr, n * sizeof(int), cudaMemcpyDeviceToHost));
+  for (int i = 0; i < n; ++i)
+  {
+    REQUIRE(h_pinned[i] == i + 1);
+  }
+
+  CUDART_REQUIRE(cudaFree(d_ptr));
+  CUDART_REQUIRE(cudaFreeHost(h_pinned));
+  REQUIRE(cudaGetLastError() == cudaSuccess);
+}
+
+// smoke test for mapped pinned host memory
+
+TEST_CASE("cudaHostAlloc mapped (zero-copy) works", "[cuda_smoke][pinned_memory][mapped]")
+{
+  (void) cudaGetLastError();
+
+  int can_map = 0;
+  CUDART_REQUIRE(cudaDeviceGetAttribute(&can_map, cudaDevAttrCanMapHostMemory, 0));
+  if (!can_map)
+  {
+    SKIP("Device cannot map host memory (cudaDevAttrCanMapHostMemory == 0).");
+  }
+
+  constexpr int n = 256;
+
+  int* h_mapped = nullptr;
+  CUDART_REQUIRE(cudaHostAlloc(&h_mapped, n * sizeof(int), cudaHostAllocMapped));
+  REQUIRE(h_mapped != nullptr);
+
+  for (int i = 0; i < n; ++i)
+  {
+    h_mapped[i] = i;
+  }
+
+  int* d_view = nullptr;
+  CUDART_REQUIRE(cudaHostGetDevicePointer(&d_view, h_mapped, 0));
+  REQUIRE(d_view != nullptr);
+
+  increment_kernel<<<4, 64>>>(d_view, n);
+  CUDART_REQUIRE(cudaGetLastError());
+  CUDART_REQUIRE(cudaDeviceSynchronize());
+
+  for (int i = 0; i < n; ++i)
+  {
+    REQUIRE(h_mapped[i] == i + 1);
+  }
+
+  CUDART_REQUIRE(cudaFreeHost(h_mapped));
+  REQUIRE(cudaGetLastError() == cudaSuccess);
+}
diff --git a/cccl_upstream/test/stdpar/CMakeLists.txt b/cccl_upstream/test/stdpar/CMakeLists.txt
new file mode 100644
index 00000000..13a92427
--- /dev/null
+++ b/cccl_upstream/test/stdpar/CMakeLists.txt
@@ -0,0 +1,54 @@
+cmake_minimum_required(VERSION 3.21)
+
+# NOTE: this is build outside of the libcu++ test harness
+project(CCCL_STDPAR_TESTS LANGUAGES CXX)
+
+if (NOT CMAKE_CXX_COMPILER_ID STREQUAL NVHPC)
+  message(FATAL_ERROR "The stdpar tests require nvc++ for CMAKE_CXX_COMPILER.")
+endif()
+
+# Enable testing for the project
+enable_testing()
+
+find_package(
+  CCCL
+  CONFIG
+  REQUIRED
+  NO_DEFAULT_PATH # Only check the explicit HINTS below:
+  HINTS "${CMAKE_CURRENT_LIST_DIR}/../../lib/cmake/cccl/"
+)
+
+file(
+  GLOB test_files
+  LIST_DIRECTORIES false
+  RELATIVE "${CMAKE_CURRENT_LIST_DIR}"
+  CONFIGURE_DEPENDS
+  "tests/*.cpp"
+)
+
+function(cccl_add_stdpar_test test_file)
+  get_filename_component(test_name ${test_file} NAME_WE)
+
+  add_executable(stdpar_test_${test_name} ${test_file})
+  target_link_libraries(stdpar_test_${test_name} PUBLIC CCCL::CCCL)
+
+  # Ensure that we are testing with GPU support
+  target_compile_options(stdpar_test_${test_name} PUBLIC -stdpar=gpu)
+  target_link_options(stdpar_test_${test_name} PUBLIC -stdpar=gpu)
+
+  # Ensure that we are indeed testing the same CCCL version
+  target_compile_definitions(
+    stdpar_test_${test_name}
+    PUBLIC
+      CMAKE_CCCL_VERSION_MAJOR=${CCCL_VERSION_MAJOR}
+      CMAKE_CCCL_VERSION_MINOR=${CCCL_VERSION_MINOR}
+      CMAKE_CCCL_VERSION_PATCH=${CCCL_VERSION_PATCH}
+  )
+
+  # Register with ctest
+  add_test(NAME stdpar_test_${test_name} COMMAND stdpar_test_${test_name})
+endfunction()
+
+foreach (test IN LISTS test_files)
+  cccl_add_stdpar_test(${test})
+endforeach()
diff --git a/cccl_upstream/test/stdpar/tests/adjacent_difference.cpp b/cccl_upstream/test/stdpar/tests/adjacent_difference.cpp
new file mode 100644
index 00000000..9c286c79
--- /dev/null
+++ b/cccl_upstream/test/stdpar/tests/adjacent_difference.cpp
@@ -0,0 +1,52 @@
+#include <algorithm>
+#include <cstddef>
+#include <execution>
+#include <numeric>
+#include <vector>
+
+int main()
+{
+  constexpr std::size_t N = 1 << 16;
+
+  auto all_one = [](const int val) {
+    return val == 1;
+  };
+
+  std::vector<int> in(N);
+  std::vector<int> out(N);
+  std::iota(in.begin(), in.end(), 0);
+
+  // Default op (difference)
+  std::adjacent_difference(std::execution::par, in.begin(), in.end(), out.begin());
+
+  if (out[0] != in[0])
+  {
+    return 1;
+  }
+  if (!std::all_of(out.begin() + 1, out.end(), all_one))
+  {
+    return 1;
+  }
+
+  // Custom binary op: sum of neighbors: out[0] = in[0]
+  std::fill(out.begin(), out.end(), 0);
+  std::adjacent_difference(std::execution::par, in.begin(), in.end(), out.begin(), [](int x, int y) {
+    return x + y;
+  });
+
+  if (out[0] != in[0])
+  {
+    return 1;
+  }
+
+  for (std::size_t i = 1; i < N; ++i)
+  {
+    const int expected = in[i] + in[i - 1];
+    if (out[i] != expected)
+    {
+      return 1;
+    }
+  }
+
+  return 0;
+}
diff --git a/cccl_upstream/test/stdpar/tests/all_of.cpp b/cccl_upstream/test/stdpar/tests/all_of.cpp
new file mode 100644
index 00000000..822f5c32
--- /dev/null
+++ b/cccl_upstream/test/stdpar/tests/all_of.cpp
@@ -0,0 +1,38 @@
+#include <algorithm>
+#include <cstddef>
+#include <execution>
+#include <vector>
+
+// Ensure that we are indeed using the correct CCCL version
+static_assert(CCCL_MAJOR_VERSION == CMAKE_CCCL_VERSION_MAJOR);
+static_assert(CCCL_MINOR_VERSION == CMAKE_CCCL_VERSION_MINOR);
+static_assert(CCCL_PATCH_VERSION == CMAKE_CCCL_VERSION_PATCH);
+
+int main()
+{
+  constexpr std::size_t N = 1 << 16;
+  std::vector<int> v(N, 1);
+
+  // All elements are 1, so this should be true
+  const bool all_ones = std::all_of(std::execution::par, v.begin(), v.end(), [](int x) {
+    return x == 1;
+  });
+
+  if (!all_ones)
+  {
+    return 1;
+  }
+
+  // Flip a single element in the middle, now all elements are not 1
+  v[N / 2]                  = 2;
+  const bool still_all_ones = std::all_of(std::execution::par, v.begin(), v.end(), [](int x) {
+    return x == 1;
+  });
+
+  if (still_all_ones)
+  {
+    return 1;
+  }
+
+  return 0;
+}
diff --git a/cccl_upstream/test/stdpar/tests/any_of.cpp b/cccl_upstream/test/stdpar/tests/any_of.cpp
new file mode 100644
index 00000000..195e91b3
--- /dev/null
+++ b/cccl_upstream/test/stdpar/tests/any_of.cpp
@@ -0,0 +1,47 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDA C++ Core Libraries, under the Apache License v2.0 with
+// LLVM Exceptions. See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+#include <cuda/version>
+
+#include <algorithm>
+#include <cstddef>
+#include <execution>
+#include <vector>
+
+// Ensure that we are indeed using the correct CCCL version
+static_assert(CCCL_MAJOR_VERSION == CMAKE_CCCL_VERSION_MAJOR);
+static_assert(CCCL_MINOR_VERSION == CMAKE_CCCL_VERSION_MINOR);
+static_assert(CCCL_PATCH_VERSION == CMAKE_CCCL_VERSION_PATCH);
+
+int main()
+{
+  constexpr std::size_t num_items = 1 << 16;
+  std::vector<int> values(num_items, 0);
+
+  constexpr auto is_one = [](const int value) {
+    return value == 1;
+  };
+
+  const bool initially_contains_one = std::any_of(std::execution::par, values.begin(), values.end(), is_one);
+  if (initially_contains_one)
+  {
+    return 1;
+  }
+
+  values[num_items / 2]      = 1;
+  const bool contains_one    = std::any_of(std::execution::par, values.begin(), values.end(), is_one);
+  const bool empty_has_a_one = std::any_of(std::execution::par, values.begin(), values.begin(), is_one);
+
+  if (!contains_one || empty_has_a_one)
+  {
+    return 1;
+  }
+
+  return 0;
+}
diff --git a/cccl_upstream/test/stdpar/tests/reduce.cpp b/cccl_upstream/test/stdpar/tests/reduce.cpp
new file mode 100644
index 00000000..cb799dba
--- /dev/null
+++ b/cccl_upstream/test/stdpar/tests/reduce.cpp
@@ -0,0 +1,26 @@
+#include <algorithm>
+#include <cassert>
+#include <execution>
+#include <numeric>
+#include <vector>
+
+// Ensure that we are indeed using the correct CCCL version
+static_assert(CCCL_MAJOR_VERSION == CMAKE_CCCL_VERSION_MAJOR);
+static_assert(CCCL_MINOR_VERSION == CMAKE_CCCL_VERSION_MINOR);
+static_assert(CCCL_PATCH_VERSION == CMAKE_CCCL_VERSION_PATCH);
+
+constexpr int N = 1000;
+
+int main()
+{
+  std::vector<int> v(N);
+  std::fill(std::execution::par_unseq, v.begin(), v.end(), 42);
+
+  int sum = std::reduce(std::execution::par_unseq, v.begin(), v.end(), 100, [](int a, int b) {
+    return a + b;
+  });
+  assert(sum == (42 * N) + 100);
+
+  sum = std::reduce(std::execution::par_unseq, v.begin(), v.end(), 100);
+  assert(sum == (42 * N) + 100);
+}
diff --git a/cccl_upstream/test/stdpar/tests/transform.cpp b/cccl_upstream/test/stdpar/tests/transform.cpp
new file mode 100644
index 00000000..940005fe
--- /dev/null
+++ b/cccl_upstream/test/stdpar/tests/transform.cpp
@@ -0,0 +1,80 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDA C++ Core Libraries, under the Apache License v2.0 with
+// LLVM Exceptions. See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+#include <cuda/version>
+
+#include <algorithm>
+#include <cstddef>
+#include <execution>
+#include <numeric>
+#include <vector>
+
+// Ensure that we are indeed using the correct CCCL version
+static_assert(CCCL_MAJOR_VERSION == CMAKE_CCCL_VERSION_MAJOR);
+static_assert(CCCL_MINOR_VERSION == CMAKE_CCCL_VERSION_MINOR);
+static_assert(CCCL_PATCH_VERSION == CMAKE_CCCL_VERSION_PATCH);
+
+int main()
+{
+  constexpr std::size_t num_items = 1 << 16;
+  std::vector<int> values(num_items);
+  const std::vector<int> offsets(num_items, 5);
+  std::vector<int> output(num_items, 0);
+  std::iota(values.begin(), values.end(), 0);
+
+  constexpr auto double_value = [](const int value) {
+    return value * 2;
+  };
+
+  constexpr auto add_values = [](const int lhs, const int rhs) {
+    return lhs + rhs;
+  };
+
+  const auto unary_end =
+    std::transform(std::execution::par, values.begin(), values.end(), output.begin(), double_value);
+
+  if (unary_end != output.end())
+  {
+    return 1;
+  }
+
+  for (std::size_t i = 0; i < num_items; ++i)
+  {
+    if (output[i] != values[i] * 2)
+    {
+      return 1;
+    }
+  }
+
+  const auto binary_end =
+    std::transform(std::execution::par, values.begin(), values.end(), offsets.begin(), output.begin(), add_values);
+
+  if (binary_end != output.end())
+  {
+    return 1;
+  }
+
+  for (std::size_t i = 0; i < num_items; ++i)
+  {
+    if (output[i] != values[i] + offsets[i])
+    {
+      return 1;
+    }
+  }
+
+  const auto empty_end =
+    std::transform(std::execution::par, values.begin(), values.begin(), output.begin(), double_value);
+
+  if (empty_end != output.begin())
+  {
+    return 1;
+  }
+
+  return 0;
+}