[CCCL] Add missing CCCL components: c2h, nvbench_helper, cmake, cudax, AGENTS.md

Added 863 files from NVIDIA/cccl sparse checkout:
- c2h/ (27 files): Catch2 test helpers — generators, validators, runner
- nvbench_helper/ (10 files): Benchmark harness utilities
- cmake/ (29 files): CMake presets and build helpers
- cudax/ (794 files): Experimental CUDA extensions
- AGENTS.md: NVIDIA's official AI agent instructions for CCCL
- CMakePresets.json: Standardized build configurations
- cccl-version.json: Version tracking

Also added CCCL_ASSET_MAP.md mapping all 4295 CCCL files to
competition value and PRD items.

cccl_upstream now covers 100% of competition-critical assets:
- 27 tuning headers (SM80/90/100 benchmark data)
- 32 dispatch headers (algorithm implementations)
- 60 Thrust examples (correctness verification)
- 217 CUB Catch2 tests (regression matrix)
- 153 CUB benchmarks (parameter space search)
- 18 CUB examples (API verification)
- 27 test helpers + benchmark harness
- 794 cudax experimental extensions

cccl_upstream/AGENTS.md

@@ -0,0 +1,404 @@
+# Agent Instructions
+
+This document provides guidelines for building, testing, and contributing to the CCCL repository. It is primarily written for agentic AIs, but the information is also useful for CCCL developers.
+
+---
+
+## Overview
+
+CCCL is a collection of CUDA C++ libraries and Python packages:
+
+* **libcudacxx** — CUDA C++ Standard Library
+* **CUB** — Block-level primitives
+* **Thrust** — High-level parallel algorithms
+* **cudax** — Experimental features
+* **C Parallel Library** — C bindings for CCCL algorithms
+* **Python CCCL packages** (`cuda-cccl`) — Python APIs for parallel primitives and programmatic access to CCCL headers
+
+The repository uses **CMake** with the **Ninja** generator and provides standardized presets for consistent builds.
+
+---
+
+## Iteration Cycles
+
+For a given task, you should:
+
+1. Research. Search the web, read existing code, look up system/dependency headers / implementations of related functionality. Figure out best practices and common pitfalls. Look for existing tests of the functionality; if none exist, plan a new test that integrates with the relevant existing testing frameworks.
+2. Plan. Create a high-level plan to implement the requested feature.
+3. Review and Refine plan. Look for pitfalls, find ways to smooth out rough edges. Verify any assumptions, edgecases, or identified pitfalls. Repeat until the plan is solid.
+4. Gather consistency context. Look at similar code (sibling classes if possible, otherwise just related source files) to learn the style and patterns used in the project. Consistency is important -- similar features should be organized and implemented similarly. Naming conventions should be followed.
+5. If requested: Present the plan. Only do this if the user asks for a plan to do something -- if they just ask you implement something without requesting a plan, skip this step.
+6. Draft. Implement the requested task to the best of your ability.
+7. Review and Refine. Read through your changes. Verify that API calls are correct. Assess clarity, performance, and readability. Iterate as needed.
+8. Style check. Ensure that your changes follow style and naming conventions.
+9. Build and test. Once you're confident that your changes are functionally and stylistically correct start build, test, and iterate cycles. If you don't have permissions to do these, ask the user to run specific build/test commands for you.
+
+---
+
+## Known Agent Limitations
+
+### OpenAI Codex
+
+Codex cloud instances cannot:
+
+* Run Docker containers with devcontainer scripts
+* Access GPUs or run GPU-dependent tests
+
+---
+
+## Build and Test Tools
+
+All CCCL subprojects are computationally expensive to build and test. Use the provided helper scripts to minimize work and target only what you need.
+
+### CMake Presets
+
+Presets are defined in `CMakePresets.json`. Names follow a `project` or `<project>-cpp<std>` format, such as `cub-cpp20`, `thrust-cpp17`, or `libcudacxx`. Use `cmake --list-presets` to view available options. Build trees are placed under `build/${CCCL_BUILD_INFIX}/${PRESET}`.
+
+### `.devcontainer/launch.sh`
+
+Launches a container configured with a CUDA Toolkit and host compiler. First startup may take time, but cached environments are faster. In agent environments, container launches may not be supported. To check if you are already inside a container, verify if `CCCL_BUILD_INFIX` is set.
+
+Common options:
+
+* `-d, --docker` — Run without VSCode (required for agents)
+* `--cuda <version>` — Select CUDA Toolkit (optional)
+* `--cuda-ext` — Use a docker image with extended CTK libraries
+* `--host <compiler>` — Select host compiler (optional)
+* `--gpus <request>` — GPU devices to add to the container (use `all` to pass all GPUs)
+* `-e/--env`, `-v/--volume` — Environment variables / volume mounts
+* `-- <script>` — Run script inside container after setup
+
+Example:
+
+```bash
+.devcontainer/launch.sh -d --cuda 13.3 --host gcc14 -- <script> [args...]
+```
+
+### `ci/util/build_and_test_targets.sh`
+
+Configures, builds, and tests selected Ninja, CTest, or lit targets. Many tests require GPUs. Options that generally work without GPUs include `--preset`, `--cmake-options`, `--configure-override`, `--build-targets`, `--lit-precompile-tests`, and `--custom-test-cmd`.
+
+Key options:
+
+* `--preset <name>` — Use a CMake preset
+* `--cmake-options <str>` — Extra CMake arguments
+* `--configure-override <cmd>` — Custom configuration command
+* `--build-targets "<targets>"` — Space-separated Ninja targets
+* `--ctest-targets "<regex>"` — Regex for CTest targets (may fail without GPUs)
+* `--lit-precompile-tests "<paths>"` — Precompile specified libcudacxx lit tests (paths are relative to `libcudacxx/test/libcudacxx/`)
+* `--lit-tests "<paths>"` — Run specified libcudacxx lit tests (also relative to `libcudacxx/test/libcudacxx/`)
+* `--custom-test-cmd "<cmd>"` — Run arbitrary command after tests
+
+### `ci/util/git_bisect.sh`
+
+Wraps `git bisect` with the build/test helper. Useful for identifying regression commits. Can take a very long time—minimize scope by restricting build/test targets.
+
+Extra options:
+
+* `--good-ref <rev>` — Known good commit/tag, or `-Nd` for origin/main N days ago (default: latest release)
+* `--bad-ref <rev>` — Known bad commit/tag, or `-Nd` (default: origin/main)
+
+See `docs/cccl/development/build_and_bisect_tools.rst` for details.
+
+---
+
+## Building and Testing
+
+Always prefer targeted builds and tests, as full builds are time-consuming. If required tools or hardware are unavailable, note this in the PR but run as many relevant tests as possible.
+
+### Targeted Build and Test Examples
+
+* **CUB** (`cub/`):
+
+```bash
+ci/util/build_and_test_targets.sh \
+  --preset cub-cpp20 \
+  --build-targets "cub.cpp20.test.iterator" \
+  --ctest-targets "cub.cpp20.test.iterator"
+```
+
+* **Thrust** (`thrust/`):
+
+```bash
+ci/util/build_and_test_targets.sh \
+  --preset thrust-cpp20 \
+  --build-targets "thrust.cpp20.test.reduce" \
+  --ctest-targets "thrust.cpp20.test.reduce"
+```
+
+* **libcudacxx** (`libcudacxx/`):
+  Avoid the expensive `libcudacxx.cpp20.precompile.lit`. Instead, precompile and run a small set of lit tests:
+
+```bash
+ci/util/build_and_test_targets.sh \
+  --preset libcudacxx \
+  --lit-precompile-tests "std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp" \
+  --lit-tests "std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp"
+```
+
+* **CUDA Experimental** (`cudax/`):
+
+```bash
+ci/util/build_and_test_targets.sh \
+  --preset cudax \
+  --build-targets "cudax.cpp20.test.async_buffer" \
+  --ctest-targets "cudax.cpp20.test.async_buffer"
+```
+
+* **C Parallel API** (`c/parallel/`):
+
+```bash
+ci/util/build_and_test_targets.sh \
+  --preset cccl-c-parallel \
+  --build-targets "cccl.c.test.reduce" \
+  --ctest-targets "cccl.c.test.reduce"
+```
+
+### Full Builds
+
+> ⚠️ **Important:** Full builds are costly. Always allow 60+ minutes for builds and 30+ minutes for tests. Do not cancel once started.
+
+Use scripts like:
+
+```bash
+./ci/build_cub.sh [-cxx g++] [-std 17] [-arch "75;80;90;120"]
+./ci/build_thrust.sh [-cxx clang++] [-std 17] [-arch "75;80;90;120"]
+./ci/build_libcudacxx.sh [-cxx g++] [-std 17] [-arch "75;80;90;120"]
+./ci/build_cudax.sh [-cxx g++] [-std 20] [-arch "75;80;90;120"]
+./ci/build_cccl_c_parallel.sh [-cxx g++] [-std 17] [-arch "75;80;90;120"]
+./ci/build_cuda_cccl_python.sh -py-version 3.10
+```
+
+### Architectures
+
+* `<XX>` — Generate PTX and SASS
+* `<XX-real>` — Generate only SASS
+* `<XX-virtual>` — Generate only PTX
+* `native` — Detect host GPU
+* `all-major-cccl` — Default for PR builds
+
+### Testing
+
+> ⚠️ Requires an NVIDIA GPU. Tests take 15+ minutes. Use targeted testing whenever possible.
+
+Examples:
+
+```bash
+./ci/test_cub.sh -cxx g++ -std 17 -arch "75;80;90;120"
+./ci/test_thrust.sh -cxx g++ -std 17 -arch "75;80;90;120"
+./ci/test_libcudacxx.sh -cxx g++ -std 17 -arch "75;80;90;120"
+./ci/test_cudax.sh -cxx g++ -std 20 -arch "75;80;90;120"
+ctest --preset=cub-cpp17
+```
+
+Options:
+
+* `-compute-sanitizer-memcheck` — Run with memory checking or other compute-sanitizer tools (not all projects support this)
+
+---
+
+## Python CCCL Packages
+
+Python components require different parameters than C++ builds. Use `-py-version` instead of compiler flags.
+
+Supported versions: `3.10`, `3.11`, `3.12`, `3.13`
+
+### Modules
+
+* **cuda.compute** — Device-level algorithms, iterators, custom GPU types
+* **cuda.cccl.headers** — Programmatic access to headers
+
+### Installation
+
+From PyPI:
+
+```bash
+pip install cuda-cccl[cu13] # or [cu12] for CTK 12.X
+```
+
+From conda-forge:
+
+```bash
+conda install -c conda-forge cccl-python
+```
+
+From source:
+
+```bash
+git clone https://github.com/NVIDIA/cccl.git
+cd cccl/python/cuda_cccl
+pip install -e .[test-cu13] # or [test-cu12] for CTK 12.X
+```
+
+Requirements:
+
+* Python 3.10+
+* CUDA Toolkit 12.x or 13.x
+* NVIDIA GPU (CC 7.5+)
+* Base dependencies: `numba>=0.60.0`, `numpy`, `cuda-pathfinder>=1.2.3`, `cuda-core`, `typing_extensions`
+* CUDA extras: `cuda-bindings` + `cuda-toolkit` + `numba-cuda` via `cuda-cccl[cu12]` or `cuda-cccl[cu13]`
+
+### Usage Examples
+
+```python
+import cuda.compute
+result = cuda.compute.reduce_into(input_array, output_scalar, init_val, binary_op)
+
+import cuda.cccl.headers as headers
+include_paths = headers.get_include_paths()
+```
+
+### Build and Test
+
+```bash
+./ci/build_cuda_cccl_python.sh -py-version 3.10
+./ci/test_cuda_compute_python.sh -py-version 3.10
+./ci/test_cuda_cccl_headers_python.sh -py-version 3.10
+./ci/test_cuda_cccl_examples_python.sh -py-version 3.10
+```
+
+Test organization:
+
+* `tests/compute` — Algorithms and iterators
+* `tests/headers` — Header integration
+* `test_examples.py` — Runs compute examples
+
+---
+
+## Continuous Integration (CI)
+
+See `docs/infrastructure/ci/references/ci_overview.rst` for detailed examples and troubleshooting guidance.
+
+CCCL's CI is built on GitHub Actions and relies on a dynamically generated job matrix plus several helper scripts.
+
+### Key Components
+
+* **`ci/matrix.yaml`**
+
+  * Declares build and test jobs for `pull_request`, `nightly`, and `weekly` workflows.
+  * Pull request (PR) runs typically spawn ~250 jobs.
+  * To reduce overhead, you can add an override matrix in `workflows.override`. This limits the PR CI run to a targeted subset of jobs. Overrides are recommended when:
+    * Changes touch high-dependency areas (e.g. top-level CI/devcontainers, libcudacxx, thrust, CUB). See `ci/inspect_changes.py` for dependency information.
+    * A smaller subset of jobs is enough to validate the change (e.g. infra changes, targeted fixes).
+  * Important rules:
+    * PR merges are blocked while an override matrix is active.
+    * The override must be reset to empty (not removed) before merging.
+    * Only add overrides when starting a new draft that qualifies; never remove one without being asked.
+
+* **`.github/actions/workflow-build/`**
+
+  * Runs `build-workflow.py`.
+  * Reads `ci/matrix.yaml` and prunes jobs using `ci/inspect_changes.py`.
+  * Calls `prepare-workflow-dispatch.py` to produce a formatted job matrix for dispatch.
+
+* **`.github/actions/workflow-run-job-{linux,windows}/`**
+
+  * Runs a single matrix job inside a devcontainer.
+
+* **`.github/actions/workflow-results/`**
+
+  * Aggregates artifacts and results.
+  * Marks workflow as failed if any job fails or an override matrix is present.
+
+* **`.github/workflows/ci-workflow-{pull-request,nightly,weekly}.yml`**
+
+  * Top-level GitHub Actions workflows invoking CI.
+
+* **`ci/inspect_changes.py`**
+
+  * Detects which subprojects changed between commits.
+  * Defines internal dependencies between CCCL projects. If a project is marked dirty, all dependent projects are also marked dirty and tested.
+  * Allows `build-workflow.py` to skip unaffected jobs.
+
+---
+
+### Commit Message Controls
+
+Tags appended to the commit summary (case-sensitive) control CI behavior:
+
+* `[bench-only]`: Skip all non-benchmark CI jobs. Equivalent to `[skip-matrix][skip-vdc][skip-docs][skip-tpt]`.
+* `[skip-matrix]`: Skip CCCL project build/test jobs. (Docs, devcontainers, and third-party builds still run.)
+* `[skip-vdc]`: Skip "Verify Devcontainer" jobs. Safe unless CI or devcontainer infra is modified.
+* `[skip-docs]`: Skip doc tests/previews. Safe if docs are unaffected.
+* `[skip-compile-time-bench]`: Skip informational compile-time benchmark telemetry. Safe if compile-time benchmark scripts/configuration are unaffected.
+* `[skip-third-party-testing]` / `[skip-tpt]`: Skip third-party smoke tests (MatX, PyTorch, RAPIDS).
+* `[skip-matx]`: Skip building the MatX third-party smoke test.
+* `[skip-pytorch]`: Skip building the PyTorch third-party smoke test.
+* `[skip-rapids]`: Skip building the RAPIDS third-party smoke test.
+
+> ⚠️ All of these tags block merging until removed and a full CI run (with no overrides) succeeds.
+
+Use these tags for early iterations to save resources. Remove them before review/merge.
+
+---
+
+## Code Formatting and Linting
+
+> ⚠️ Always run before committing. CI will fail otherwise.
+
+```bash
+pip install pre-commit
+pre-commit install
+pre-commit run --all-files
+pre-commit run --files <file1> <file2>
+```
+
+### Style Guidance
+
+When editing or reviewing CCCL code for style, read `.agent/skills/cccl-style/SKILL.md`. It routes to the common CCCL style reference and any path-specific style reference that applies to the files being changed.
+
+### Test Guidance
+
+When writing, updating, reviewing, or validating CCCL tests, read `.agent/skills/cccl-test/SKILL.md`. It routes to the common CCCL test reference and any path-specific test reference that applies to the files being changed.
+
+---
+
+## General Guidelines
+
+* Validate changes with builds/tests; report results.
+* Run `pre-commit` before committing.
+* Review `CONTRIBUTING.md` and `docs/infrastructure/ci/references/ci_overview.rst` before starting work.
+
+### Performance Tips
+
+* Use development containers with `sccache` (CCCL team only).
+* Limit architectures to reduce compile time (e.g. `-arch "native"` or `"80"` if no GPU).
+* Build with Ninja for fast, parallel builds.
+
+
+---
+
+## Repository Structure
+
+```
+cccl/
+├── .github/            # Workflows
+├── .devcontainer/      # Dev containers
+├── libcudacxx/         # CUDA C++ Standard Library
+├── cub/                # CUB primitives
+├── thrust/             # Thrust algorithms
+├── cudax/              # Experimental features
+├── c/                  # C Parallel library
+├── python/cuda_cccl/   # Python bindings
+├── ci/                 # Build/test scripts
+├── examples/           # Usage examples
+└── CMakePresets.json   # Preset configurations
+```
+
+Python package layout:
+
+```
+python/cuda_cccl/
+├── cuda/
+│   ├── compute/
+│   └── cccl/
+│       ├── parallel/
+│       └── headers/
+├── tests/
+├── benchmarks/
+└── pyproject.toml
+```
+
+---
+
+⚠️ **Reminder:** Long-running builds/tests are normal. Never cancel them; allow to complete.

cccl_upstream/CMakePresets.json

@@ -0,0 +1,972 @@
+{
+  "version": 3,
+  "cmakeMinimumRequired": {
+    "major": 3,
+    "minor": 21,
+    "patch": 0
+  },
+  "configurePresets": [
+    {
+      "name": "base",
+      "hidden": true,
+      "generator": "Ninja",
+      "binaryDir": "${sourceDir}/build/$env{CCCL_BUILD_INFIX}/${presetName}",
+      "cacheVariables": {
+        "CMAKE_BUILD_TYPE": "Release",
+        "CMAKE_CUDA_ARCHITECTURES": "all-major-cccl",
+        "CCCL_ENABLE_UNSTABLE": true,
+        "CCCL_ENABLE_LIBCUDACXX": false,
+        "CCCL_ENABLE_CUB": false,
+        "CCCL_ENABLE_THRUST": false,
+        "CCCL_ENABLE_CUDAX": false,
+        "CCCL_ENABLE_TESTING": false,
+        "CCCL_ENABLE_EXAMPLES": false,
+        "CCCL_ENABLE_C_PARALLEL": false,
+        "CCCL_ENABLE_C_EXPERIMENTAL_STF": false,
+        "CCCL_ENABLE_CUDA_SMOKE_TESTS": true,
+        "CCCL_SKIP_BUILD_CHECKS": false,
+        "libcudacxx_ENABLE_INSTALL_RULES": true,
+        "CUB_ENABLE_INSTALL_RULES": true,
+        "Thrust_ENABLE_INSTALL_RULES": true,
+        "cudax_ENABLE_INSTALL_RULES": true
+      }
+    },
+    {
+      "name": "install",
+      "displayName": "Installation / Packaging (only stable libraries)",
+      "inherits": "base",
+      "cacheVariables": {
+        "CCCL_ENABLE_CUDA_SMOKE_TESTS": false,
+        "cudax_ENABLE_INSTALL_RULES": false,
+        "CCCL_SKIP_BUILD_CHECKS": true
+      }
+    },
+    {
+      "name": "install-unstable",
+      "displayName": "Installation / Packaging (includes experimental libraries)",
+      "inherits": "install"
+    },
+    {
+      "name": "install-unstable-only",
+      "displayName": "Installation / Packaging (*only* experimental libraries)",
+      "inherits": "install",
+      "cacheVariables": {
+        "libcudacxx_ENABLE_INSTALL_RULES": false,
+        "CUB_ENABLE_INSTALL_RULES": false,
+        "Thrust_ENABLE_INSTALL_RULES": false
+      }
+    },
+    {
+      "name": "all-dev",
+      "inherits": "base",
+      "cacheVariables": {
+        "CMAKE_CUDA_ARCHITECTURES": "native",
+        "CCCL_ENABLE_LIBCUDACXX": true,
+        "CCCL_ENABLE_CUB": true,
+        "CCCL_ENABLE_THRUST": true,
+        "CCCL_ENABLE_CUDAX": true,
+        "CCCL_ENABLE_TESTING": true,
+        "CCCL_ENABLE_EXAMPLES": true,
+        "CCCL_ENABLE_BENCHMARKS": true,
+        "CCCL_ENABLE_C_PARALLEL": true,
+        "CCCL_ENABLE_C_EXPERIMENTAL_STF": true,
+        "CCCL_IGNORE_DEPRECATED_CPP_DIALECT": true,
+        "LIBCUDACXX_ENABLE_LIBCUDACXX_TESTS": true,
+        "CUB_ENABLE_HEADER_TESTING": true,
+        "CUB_ENABLE_TESTING": true,
+        "CUB_ENABLE_EXAMPLES": true,
+        "THRUST_ENABLE_MULTICONFIG": true,
+        "THRUST_MULTICONFIG_WORKLOAD": "LARGE",
+        "THRUST_MULTICONFIG_ENABLE_SYSTEM_CPP": true,
+        "THRUST_MULTICONFIG_ENABLE_SYSTEM_CUDA": true,
+        "THRUST_MULTICONFIG_ENABLE_SYSTEM_OMP": true,
+        "THRUST_MULTICONFIG_ENABLE_SYSTEM_TBB": true,
+        "cudax_ENABLE_HEADER_TESTING": true,
+        "cudax_ENABLE_TESTING": true,
+        "cudax_ENABLE_EXAMPLES": true,
+        "cudax_ENABLE_PLACES": true,
+        "cudax_ENABLE_CUDASTF": true,
+        "cudax_ENABLE_CUDASTF_BOUNDSCHECK": false,
+        "cudax_ENABLE_CUDASTF_CODE_GENERATION": true,
+        "cudax_ENABLE_CUDASTF_MATHLIBS": false,
+        "cudax_ENABLE_CUFILE": false,
+        "cudax_ENABLE_NCCL": true,
+        "HACK_cudax_ALLOW_MISSING_NCCL": true,
+        "CCCL_C_Parallel_ENABLE_TESTING": true,
+        "CCCL_C_Parallel_ENABLE_HEADER_TESTING": true,
+        "CCCL_C_EXPERIMENTAL_STF_ENABLE_TESTING": true
+      }
+    },
+    {
+      "name": "all-dev-debug",
+      "displayName": "all-dev debug",
+      "inherits": "all-dev",
+      "cacheVariables": {
+        "CMAKE_CUDA_ARCHITECTURES": "native",
+        "CMAKE_BUILD_TYPE": "Debug",
+        "CMAKE_CUDA_FLAGS": "-G",
+        "CCCL_ENABLE_BENCHMARKS": false,
+        "cudax_ENABLE_CUDASTF_BOUNDSCHECK": true
+      }
+    },
+    {
+      "name": "all-tidy",
+      "displayName": "clang-tidy",
+      "inherits": "all-dev-debug",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "17",
+        "CMAKE_CUDA_STANDARD": "17",
+        "THRUST_MULTICONFIG_WORKLOAD": "MEDIUM",
+        "CCCL_ENABLE_BENCHMARKS": true,
+        "CCCL_ENABLE_CLANG_TIDY": true,
+        "CMAKE_C_COMPILER": "clang",
+        "CMAKE_CXX_COMPILER": "clang++",
+        "CMAKE_CUDA_FLAGS": "",
+        "CMAKE_CUDA_COMPILER": "clang++"
+      }
+    },
+    {
+      "name": "libcudacxx-codegen",
+      "displayName": "libcu++: codegen",
+      "inherits": "base",
+      "cacheVariables": {
+        "CCCL_ENABLE_LIBCUDACXX": true,
+        "CCCL_IGNORE_DEPRECATED_CPP_DIALECT": true,
+        "LIBCUDACXX_ENABLE_LIBCUDACXX_TESTS": false,
+        "libcudacxx_ENABLE_CODEGEN": true,
+        "LIBCUDACXX_ENABLE_CUDA": false
+      }
+    },
+    {
+      "name": "libcudacxx",
+      "inherits": "base",
+      "cacheVariables": {
+        "CCCL_ENABLE_LIBCUDACXX": true,
+        "LIBCUDACXX_ENABLE_LIBCUDACXX_TESTS": true
+      }
+    },
+    {
+      "name": "libcudacxx-cpp17",
+      "displayName": "libcu++: C++17",
+      "inherits": "libcudacxx",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "17",
+        "CMAKE_CUDA_STANDARD": "17"
+      }
+    },
+    {
+      "name": "libcudacxx-cpp20",
+      "displayName": "libcu++: C++20",
+      "inherits": "libcudacxx",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "20",
+        "CMAKE_CUDA_STANDARD": "20"
+      }
+    },
+    {
+      "name": "libcudacxx-cpp23",
+      "displayName": "libcu++: C++23",
+      "inherits": "libcudacxx",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "23",
+        "CMAKE_CUDA_STANDARD": "23"
+      }
+    },
+    {
+      "name": "libcudacxx-nvrtc",
+      "inherits": "libcudacxx",
+      "cacheVariables": {
+        "LIBCUDACXX_TEST_WITH_NVRTC": true,
+        "CMAKE_CUDA_ARCHITECTURES": "70"
+      }
+    },
+    {
+      "name": "libcudacxx-nvrtc-cpp17",
+      "displayName": "libcu++: NVRTC C++17",
+      "inherits": "libcudacxx-nvrtc",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "17",
+        "CMAKE_CUDA_STANDARD": "17"
+      }
+    },
+    {
+      "name": "libcudacxx-nvrtc-cpp20",
+      "displayName": "libcu++: NVRTC C++20",
+      "inherits": "libcudacxx-nvrtc",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "20",
+        "CMAKE_CUDA_STANDARD": "20"
+      }
+    },
+    {
+      "name": "cub",
+      "displayName": "CUB",
+      "inherits": "base",
+      "cacheVariables": {
+        "CCCL_ENABLE_CUB": true,
+        "CUB_ENABLE_HEADER_TESTING": true,
+        "CUB_ENABLE_TESTING": true,
+        "CUB_ENABLE_EXAMPLES": true
+      }
+    },
+    {
+      "name": "cub-cpp17",
+      "displayName": "CUB: C++17",
+      "inherits": "cub",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "17",
+        "CMAKE_CUDA_STANDARD": "17"
+      }
+    },
+    {
+      "name": "cub-cpp20",
+      "displayName": "CUB: C++20",
+      "inherits": "cub",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "20",
+        "CMAKE_CUDA_STANDARD": "20"
+      }
+    },
+    {
+      "name": "cub-nolid",
+      "displayName": "CUB: No Launcher",
+      "inherits": "cub",
+      "cacheVariables": {
+        "CUB_ENABLE_LAUNCH_NO_LAUNCHER": true,
+        "CUB_ENABLE_LAUNCH_HOST_LAUNCHER": false,
+        "CUB_ENABLE_LAUNCH_DEVICE_LAUNCHER": false,
+        "CUB_ENABLE_LAUNCH_GRAPH_LAUNCHER": false
+      }
+    },
+    {
+      "name": "cub-nolid-cpp17",
+      "displayName": "CUB: No Launcher C++17",
+      "inherits": "cub-nolid",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "17",
+        "CMAKE_CUDA_STANDARD": "17"
+      }
+    },
+    {
+      "name": "cub-nolid-cpp20",
+      "displayName": "CUB: No Launcher C++20",
+      "inherits": "cub-nolid",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "20",
+        "CMAKE_CUDA_STANDARD": "20"
+      }
+    },
+    {
+      "name": "cub-lid0",
+      "displayName": "CUB: Host Launch",
+      "inherits": "cub",
+      "cacheVariables": {
+        "CUB_ENABLE_LAUNCH_NO_LAUNCHER": false,
+        "CUB_ENABLE_LAUNCH_HOST_LAUNCHER": true,
+        "CUB_ENABLE_LAUNCH_DEVICE_LAUNCHER": false,
+        "CUB_ENABLE_LAUNCH_GRAPH_LAUNCHER": false
+      }
+    },
+    {
+      "name": "cub-lid0-cpp17",
+      "displayName": "CUB: Host Launch C++17",
+      "inherits": "cub-lid0",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "17",
+        "CMAKE_CUDA_STANDARD": "17"
+      }
+    },
+    {
+      "name": "cub-lid0-cpp20",
+      "displayName": "CUB: Host Launch C++20",
+      "inherits": "cub-lid0",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "20",
+        "CMAKE_CUDA_STANDARD": "20"
+      }
+    },
+    {
+      "name": "cub-lid1",
+      "displayName": "CUB: Device Launch",
+      "inherits": "cub",
+      "cacheVariables": {
+        "CUB_ENABLE_LAUNCH_NO_LAUNCHER": false,
+        "CUB_ENABLE_LAUNCH_HOST_LAUNCHER": false,
+        "CUB_ENABLE_LAUNCH_DEVICE_LAUNCHER": true,
+        "CUB_ENABLE_LAUNCH_GRAPH_LAUNCHER": false
+      }
+    },
+    {
+      "name": "cub-lid1-cpp17",
+      "displayName": "CUB: Device Launch C++17",
+      "inherits": "cub-lid1",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "17",
+        "CMAKE_CUDA_STANDARD": "17"
+      }
+    },
+    {
+      "name": "cub-lid1-cpp20",
+      "displayName": "CUB: Device Launch C++20",
+      "inherits": "cub-lid1",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "20",
+        "CMAKE_CUDA_STANDARD": "20"
+      }
+    },
+    {
+      "name": "cub-lid2",
+      "displayName": "CUB: Graph Launch",
+      "inherits": "cub",
+      "cacheVariables": {
+        "CUB_ENABLE_LAUNCH_NO_LAUNCHER": false,
+        "CUB_ENABLE_LAUNCH_HOST_LAUNCHER": false,
+        "CUB_ENABLE_LAUNCH_DEVICE_LAUNCHER": false,
+        "CUB_ENABLE_LAUNCH_GRAPH_LAUNCHER": true
+      }
+    },
+    {
+      "name": "cub-lid2-cpp17",
+      "displayName": "CUB: Graph Launch C++17",
+      "inherits": "cub-lid2",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "17",
+        "CMAKE_CUDA_STANDARD": "17"
+      }
+    },
+    {
+      "name": "cub-lid2-cpp20",
+      "displayName": "CUB: Graph Launch C++20",
+      "inherits": "cub-lid2",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "20",
+        "CMAKE_CUDA_STANDARD": "20"
+      }
+    },
+    {
+      "name": "thrust",
+      "displayName": "Thrust",
+      "inherits": "base",
+      "cacheVariables": {
+        "CCCL_ENABLE_THRUST": true,
+        "THRUST_ENABLE_MULTICONFIG": true,
+        "THRUST_MULTICONFIG_ENABLE_SYSTEM_CPP": true,
+        "THRUST_MULTICONFIG_ENABLE_SYSTEM_CUDA": true,
+        "THRUST_MULTICONFIG_ENABLE_SYSTEM_OMP": true,
+        "THRUST_MULTICONFIG_ENABLE_SYSTEM_TBB": true
+      }
+    },
+    {
+      "name": "thrust-cpp17",
+      "displayName": "Thrust: C++17",
+      "inherits": "thrust",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "17",
+        "CMAKE_CUDA_STANDARD": "17"
+      }
+    },
+    {
+      "name": "thrust-cpp20",
+      "displayName": "Thrust: C++20",
+      "inherits": "thrust",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "20",
+        "CMAKE_CUDA_STANDARD": "20"
+      }
+    },
+    {
+      "name": "cudax",
+      "displayName": "cudax",
+      "inherits": "base",
+      "cacheVariables": {
+        "CCCL_ENABLE_CUDAX": true,
+        "cudax_ENABLE_HEADER_TESTING": true,
+        "cudax_ENABLE_TESTING": true,
+        "cudax_ENABLE_EXAMPLES": true,
+        "cudax_ENABLE_PLACES": true,
+        "cudax_ENABLE_CUDASTF": true,
+        "cudax_ENABLE_CUDASTF_BOUNDSCHECK": false,
+        "cudax_ENABLE_CUDASTF_CODE_GENERATION": true,
+        "cudax_ENABLE_CUDASTF_MATHLIBS": false,
+        "cudax_ENABLE_CUFILE": false,
+        "cudax_ENABLE_NCCL": true,
+        "HACK_cudax_ALLOW_MISSING_NCCL": true
+      }
+    },
+    {
+      "name": "cudax-cpp17",
+      "displayName": "cudax: C++17",
+      "inherits": "cudax",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "17",
+        "CMAKE_CUDA_STANDARD": "17"
+      }
+    },
+    {
+      "name": "cudax-cpp20",
+      "displayName": "cudax: C++20",
+      "inherits": "cudax",
+      "cacheVariables": {
+        "CMAKE_CXX_STANDARD": "20",
+        "CMAKE_CUDA_STANDARD": "20"
+      }
+    },
+    {
+      "name": "cccl-c-parallel",
+      "displayName": "CCCL C Parallel Library",
+      "inherits": "base",
+      "cacheVariables": {
+        "CCCL_ENABLE_C_PARALLEL": true,
+        "CCCL_C_Parallel_ENABLE_TESTING": true,
+        "CCCL_C_Parallel_ENABLE_HEADER_TESTING": true
+      }
+    },
+    {
+      "name": "cccl-c-parallel-v2",
+      "displayName": "CCCL C Parallel Library v2 (HostJIT)",
+      "inherits": "base",
+      "cacheVariables": {
+        "CCCL_ENABLE_C_PARALLEL_V2": true,
+        "CCCL_C_Parallel_V2_ENABLE_TESTING": true
+      }
+    },
+    {
+      "name": "cccl-c-stf",
+      "displayName": "CCCL C CUDASTF Library",
+      "inherits": "base",
+      "cacheVariables": {
+        "CCCL_ENABLE_C_EXPERIMENTAL_STF": true,
+        "CCCL_C_EXPERIMENTAL_STF_ENABLE_TESTING": true
+      }
+    },
+    {
+      "name": "packaging",
+      "displayName": "CCCL Packaging Tests/Examples",
+      "inherits": "base",
+      "cacheVariables": {
+        "CCCL_ENABLE_EXAMPLES": true,
+        "CCCL_ENABLE_TESTING": true
+      }
+    },
+    {
+      "name": "nvbench-helper",
+      "displayName": "NVBench Helper",
+      "inherits": "base",
+      "cacheVariables": {
+        "CCCL_ENABLE_NVBENCH_HELPER": true,
+        "nvbench_helper_ENABLE_TESTING": true
+      }
+    },
+    {
+      "name": "cub-benchmark",
+      "displayName": "CUB benchmarking",
+      "generator": "Ninja",
+      "cacheVariables": {
+        "CMAKE_BUILD_TYPE": "Release",
+        "CMAKE_CUDA_ARCHITECTURES": "native",
+        "CCCL_ENABLE_BENCHMARKS": true,
+        "CCCL_ENABLE_CUB": true,
+        "CCCL_ENABLE_THRUST": false,
+        "CCCL_ENABLE_LIBCUDACXX": false,
+        "CCCL_ENABLE_CUDAX": false,
+        "CCCL_ENABLE_C_PARALLEL": false,
+        "CCCL_ENABLE_C_EXPERIMENTAL_STF": false,
+        "CCCL_ENABLE_TESTING": false,
+        "CCCL_ENABLE_EXAMPLES": false,
+        "CUB_ENABLE_EXAMPLES": false,
+        "CUB_ENABLE_TESTING": false,
+        "CUB_ENABLE_HEADER_TESTING": false
+      }
+    },
+    {
+      "name": "cub-tune",
+      "displayName": "CUB tuning",
+      "inherits": "cub-benchmark",
+      "cacheVariables": {
+        "CUB_ENABLE_TUNING": true
+      }
+    },
+    {
+      "name": "benchmark",
+      "displayName": "CCCL benchmarking",
+      "inherits": "cub-benchmark",
+      "cacheVariables": {
+        "CCCL_ENABLE_THRUST": true,
+        "CCCL_ENABLE_LIBCUDACXX": true,
+        "THRUST_ENABLE_EXAMPLES": false,
+        "THRUST_ENABLE_TESTING": false,
+        "THRUST_ENABLE_HEADER_TESTING": false,
+        "LIBCUDACXX_ENABLE_LIBCUDACXX_TESTS": false
+      }
+    }
+  ],
+  "buildPresets": [
+    {
+      "name": "all-dev",
+      "configurePreset": "all-dev"
+    },
+    {
+      "name": "all-dev-debug",
+      "configurePreset": "all-dev-debug"
+    },
+    {
+      "name": "all-tidy",
+      "configurePreset": "all-tidy",
+      "targets": [
+        "cccl.tidy"
+      ]
+    },
+    {
+      "name": "install",
+      "configurePreset": "install"
+    },
+    {
+      "name": "libcudacxx",
+      "configurePreset": "libcudacxx",
+      "targets": [
+        "cccl.test.cuda_runtime_smoke",
+        "libcudacxx.test.internal_headers",
+        "libcudacxx.test.public_headers",
+        "libcudacxx.test.public_headers_host_only",
+        "libcudacxx.test.lit.precompile",
+        "libcudacxx.test.nvtarget",
+        "libcudacxx.test.atomics.ptx",
+        "libcudacxx.test.simd.ptx",
+        "libcudacxx.test.c2h_all",
+        "libcudacxx.test.debugging"
+      ]
+    },
+    {
+      "name": "libcudacxx-cpp17",
+      "configurePreset": "libcudacxx-cpp17",
+      "inherits": "libcudacxx"
+    },
+    {
+      "name": "libcudacxx-cpp20",
+      "configurePreset": "libcudacxx-cpp20",
+      "inherits": "libcudacxx"
+    },
+    {
+      "name": "libcudacxx-codegen",
+      "configurePreset": "libcudacxx-codegen",
+      "targets": [
+        "libcudacxx.atomics.codegen"
+      ]
+    },
+    {
+      "name": "libcudacxx-codegen-install",
+      "configurePreset": "libcudacxx-codegen",
+      "targets": [
+        "libcudacxx.atomics.codegen.install"
+      ]
+    },
+    {
+      "name": "libcudacxx-nvrtc",
+      "configurePreset": "libcudacxx-nvrtc",
+      "targets": [
+        "libcudacxx.nvrtcc"
+      ]
+    },
+    {
+      "name": "libcudacxx-nvrtc-cpp17",
+      "configurePreset": "libcudacxx-nvrtc-cpp17",
+      "inherits": "libcudacxx-nvrtc"
+    },
+    {
+      "name": "libcudacxx-nvrtc-cpp20",
+      "configurePreset": "libcudacxx-nvrtc-cpp20",
+      "inherits": "libcudacxx-nvrtc"
+    },
+    {
+      "name": "cub",
+      "configurePreset": "cub"
+    },
+    {
+      "name": "cub-cpp17",
+      "configurePreset": "cub-cpp17"
+    },
+    {
+      "name": "cub-cpp20",
+      "configurePreset": "cub-cpp20"
+    },
+    {
+      "name": "cub-nolid",
+      "configurePreset": "cub-nolid"
+    },
+    {
+      "name": "cub-nolid-cpp17",
+      "configurePreset": "cub-nolid-cpp17"
+    },
+    {
+      "name": "cub-nolid-cpp20",
+      "configurePreset": "cub-nolid-cpp20"
+    },
+    {
+      "name": "cub-lid0",
+      "configurePreset": "cub-lid0"
+    },
+    {
+      "name": "cub-lid0-cpp17",
+      "configurePreset": "cub-lid0-cpp17"
+    },
+    {
+      "name": "cub-lid0-cpp20",
+      "configurePreset": "cub-lid0-cpp20"
+    },
+    {
+      "name": "cub-lid1",
+      "configurePreset": "cub-lid1"
+    },
+    {
+      "name": "cub-lid1-cpp17",
+      "configurePreset": "cub-lid1-cpp17"
+    },
+    {
+      "name": "cub-lid1-cpp20",
+      "configurePreset": "cub-lid1-cpp20"
+    },
+    {
+      "name": "cub-lid2",
+      "configurePreset": "cub-lid2"
+    },
+    {
+      "name": "cub-lid2-cpp17",
+      "configurePreset": "cub-lid2-cpp17"
+    },
+    {
+      "name": "cub-lid2-cpp20",
+      "configurePreset": "cub-lid2-cpp20"
+    },
+    {
+      "name": "thrust",
+      "configurePreset": "thrust"
+    },
+    {
+      "name": "thrust-cpp17",
+      "configurePreset": "thrust-cpp17"
+    },
+    {
+      "name": "thrust-cpp20",
+      "configurePreset": "thrust-cpp20"
+    },
+    {
+      "name": "cudax",
+      "configurePreset": "cudax"
+    },
+    {
+      "name": "cudax-cpp17",
+      "configurePreset": "cudax-cpp17"
+    },
+    {
+      "name": "cudax-cpp20",
+      "configurePreset": "cudax-cpp20"
+    },
+    {
+      "name": "cccl-c-parallel",
+      "configurePreset": "cccl-c-parallel"
+    },
+    {
+      "name": "cccl-c-parallel-v2",
+      "configurePreset": "cccl-c-parallel-v2"
+    },
+    {
+      "name": "cccl-c-stf",
+      "configurePreset": "cccl-c-stf"
+    },
+    {
+      "name": "packaging",
+      "configurePreset": "packaging"
+    },
+    {
+      "name": "nvbench-helper",
+      "configurePreset": "nvbench-helper"
+    }
+  ],
+  "testPresets": [
+    {
+      "name": "base",
+      "hidden": true,
+      "output": {
+        "outputOnFailure": true
+      },
+      "execution": {
+        "noTestsAction": "error",
+        "stopOnFailure": false
+      }
+    },
+    {
+      "name": "all-dev",
+      "configurePreset": "all-dev",
+      "inherits": "base"
+    },
+    {
+      "name": "all-dev-debug",
+      "configurePreset": "all-dev-debug",
+      "inherits": "all-dev"
+    },
+    {
+      "name": "libcudacxx-codegen",
+      "configurePreset": "libcudacxx-codegen",
+      "inherits": "base",
+      "filter": {
+        "include": {
+          "name": "^libcudacxx\\.test\\.atomics\\.codegen.*$"
+        }
+      }
+    },
+    {
+      "name": "libcudacxx-ctest",
+      "configurePreset": "libcudacxx",
+      "inherits": "base",
+      "filter": {
+        "exclude": {
+          "name": "^libcudacxx\\.test\\.lit$"
+        }
+      }
+    },
+    {
+      "name": "libcudacxx-ctest-cpp17",
+      "configurePreset": "libcudacxx-cpp17",
+      "inherits": "libcudacxx-ctest"
+    },
+    {
+      "name": "libcudacxx-ctest-cpp20",
+      "configurePreset": "libcudacxx-cpp20",
+      "inherits": "libcudacxx-ctest"
+    },
+    {
+      "name": "libcudacxx-lit",
+      "configurePreset": "libcudacxx",
+      "inherits": "base",
+      "filter": {
+        "include": {
+          "name": "^libcudacxx\\.test\\.lit$"
+        }
+      },
+      "output": {
+        "verbosity": "extra",
+        "outputOnFailure": false
+      }
+    },
+    {
+      "name": "libcudacxx-lit-cpp17",
+      "configurePreset": "libcudacxx-cpp17",
+      "inherits": "libcudacxx-lit"
+    },
+    {
+      "name": "libcudacxx-lit-cpp20",
+      "configurePreset": "libcudacxx-cpp20",
+      "inherits": "libcudacxx-lit"
+    },
+    {
+      "name": "libcudacxx-nvrtc",
+      "configurePreset": "libcudacxx-nvrtc",
+      "inherits": "libcudacxx-lit"
+    },
+    {
+      "name": "libcudacxx-nvrtc-cpp17",
+      "configurePreset": "libcudacxx-nvrtc-cpp17",
+      "inherits": "libcudacxx-nvrtc"
+    },
+    {
+      "name": "libcudacxx-nvrtc-cpp20",
+      "configurePreset": "libcudacxx-nvrtc-cpp20",
+      "inherits": "libcudacxx-nvrtc"
+    },
+    {
+      "name": "cub",
+      "configurePreset": "cub",
+      "inherits": "base"
+    },
+    {
+      "name": "cub-cpp17",
+      "configurePreset": "cub-cpp17",
+      "inherits": "cub"
+    },
+    {
+      "name": "cub-cpp20",
+      "configurePreset": "cub-cpp20",
+      "inherits": "cub"
+    },
+    {
+      "name": "cub-nolid",
+      "configurePreset": "cub-nolid",
+      "inherits": "cub",
+      "filter": {
+        "exclude": {
+          "name": "^cub.*\\.lid_[0-2].*$"
+        }
+      }
+    },
+    {
+      "name": "cub-nolid-cpp17",
+      "configurePreset": "cub-nolid-cpp17",
+      "inherits": "cub-nolid"
+    },
+    {
+      "name": "cub-nolid-cpp20",
+      "configurePreset": "cub-nolid-cpp20",
+      "inherits": "cub-nolid"
+    },
+    {
+      "name": "cub-lid0",
+      "configurePreset": "cub-lid0",
+      "inherits": "cub",
+      "filter": {
+        "include": {
+          "name": "^cub.*\\.lid_0.*$"
+        }
+      }
+    },
+    {
+      "name": "cub-lid0-cpp17",
+      "configurePreset": "cub-lid0-cpp17",
+      "inherits": "cub-lid0"
+    },
+    {
+      "name": "cub-lid0-cpp20",
+      "configurePreset": "cub-lid0-cpp20",
+      "inherits": "cub-lid0"
+    },
+    {
+      "name": "cub-lid1",
+      "configurePreset": "cub-lid1",
+      "inherits": "cub",
+      "filter": {
+        "include": {
+          "name": "^cub.*\\.lid_1.*$"
+        }
+      }
+    },
+    {
+      "name": "cub-lid1-cpp17",
+      "configurePreset": "cub-lid1-cpp17",
+      "inherits": "cub-lid1"
+    },
+    {
+      "name": "cub-lid1-cpp20",
+      "configurePreset": "cub-lid1-cpp20",
+      "inherits": "cub-lid1"
+    },
+    {
+      "name": "cub-lid2",
+      "configurePreset": "cub-lid2",
+      "inherits": "cub",
+      "filter": {
+        "include": {
+          "name": "^cub.*\\.lid_2.*$"
+        }
+      }
+    },
+    {
+      "name": "cub-lid2-cpp17",
+      "configurePreset": "cub-lid2-cpp17",
+      "inherits": "cub-lid2"
+    },
+    {
+      "name": "cub-lid2-cpp20",
+      "configurePreset": "cub-lid2-cpp20",
+      "inherits": "cub-lid2"
+    },
+    {
+      "name": "thrust",
+      "configurePreset": "thrust",
+      "inherits": "base"
+    },
+    {
+      "name": "thrust-cpp17",
+      "configurePreset": "thrust-cpp17",
+      "inherits": "thrust"
+    },
+    {
+      "name": "thrust-cpp20",
+      "configurePreset": "thrust-cpp20",
+      "inherits": "thrust"
+    },
+    {
+      "name": "thrust-gpu",
+      "configurePreset": "thrust",
+      "inherits": "thrust",
+      "filter": {
+        "include": {
+          "name": "^thrust.*\\.cuda\\..*$"
+        }
+      }
+    },
+    {
+      "name": "thrust-gpu-cpp17",
+      "configurePreset": "thrust-cpp17",
+      "inherits": "thrust-gpu"
+    },
+    {
+      "name": "thrust-gpu-cpp20",
+      "configurePreset": "thrust-cpp20",
+      "inherits": "thrust-gpu"
+    },
+    {
+      "name": "thrust-cpu",
+      "configurePreset": "thrust",
+      "inherits": "thrust",
+      "filter": {
+        "exclude": {
+          "name": "^thrust.*\\.cuda\\..*$"
+        }
+      }
+    },
+    {
+      "name": "thrust-cpu-cpp17",
+      "configurePreset": "thrust-cpp17",
+      "inherits": "thrust-cpu"
+    },
+    {
+      "name": "thrust-cpu-cpp20",
+      "configurePreset": "thrust-cpp20",
+      "inherits": "thrust-cpu"
+    },
+    {
+      "name": "cudax",
+      "configurePreset": "cudax",
+      "inherits": "base",
+      "filter": {
+        "exclude": {
+          "name": "^cudax\\.test\\.(cufile|stf\\.stress).*$"
+        }
+      }
+    },
+    {
+      "name": "cudax-cpp17",
+      "configurePreset": "cudax-cpp17",
+      "inherits": "cudax"
+    },
+    {
+      "name": "cudax-cpp20",
+      "configurePreset": "cudax-cpp20",
+      "inherits": "cudax"
+    },
+    {
+      "name": "cccl-c-parallel",
+      "configurePreset": "cccl-c-parallel",
+      "inherits": "base"
+    },
+    {
+      "name": "cccl-c-parallel-v2",
+      "configurePreset": "cccl-c-parallel-v2",
+      "inherits": "base"
+    },
+    {
+      "name": "cccl-c-stf",
+      "configurePreset": "cccl-c-stf",
+      "inherits": "base"
+    },
+    {
+      "name": "packaging",
+      "configurePreset": "packaging",
+      "inherits": "base"
+    },
+    {
+      "name": "nvbench-helper",
+      "configurePreset": "nvbench-helper",
+      "inherits": "base"
+    }
+  ]
+}

cccl_upstream/c2h/CMakeLists.txt

@@ -0,0 +1,65 @@
+cmake_minimum_required(VERSION 3.21)
+
+project(C2H LANGUAGES CXX CUDA)
+
+# when CCCL is used from the CTK, C2H does not need to be rebuilt when making local changes to CCCL (faster iteration)
+option(C2H_USE_CCCL_FROM_CTK "Use CCCL from the CTK in c2h." OFF)
+
+# Get dependencies.
+cccl_get_catch2()
+if (NOT C2H_USE_CCCL_FROM_CTK)
+  cccl_get_cccl()
+endif()
+cccl_get_cudatoolkit()
+
+set(has_curand OFF)
+if (TARGET CUDA::curand)
+  set(has_curand ON)
+endif()
+
+option(C2H_ENABLE_CURAND "Use CUDA CURAND library in c2h." ${has_curand})
+
+# Disable C2H_ENABLE_CURAND if cuRAND is unavailable.
+if (C2H_ENABLE_CURAND AND NOT has_curand)
+  message(
+    WARNING
+    "C2H_ENABLE_CURAND is requested, but CUDA::curand is unavailable. Disabling."
+  )
+  set(C2H_ENABLE_CURAND OFF)
+endif()
+
+add_library(
+  cccl.c2h
+  STATIC
+  generators.cu
+  generators_gen_values.cu
+  generators_uniform_offsets.cu
+  generators_vector.cu
+)
+target_compile_definitions(cccl.c2h PUBLIC CATCH_CONFIG_PREFIX_ALL)
+target_include_directories(cccl.c2h PUBLIC "${C2H_SOURCE_DIR}/include")
+target_link_libraries(
+  cccl.c2h
+  PUBLIC #
+    cccl.compiler_interface
+    Catch2::Catch2
+)
+if (NOT C2H_USE_CCCL_FROM_CTK)
+  target_link_libraries(cccl.c2h PUBLIC CCCL::CCCL)
+endif()
+cccl_configure_target(cccl.c2h DIALECT 17)
+
+if (C2H_ENABLE_CURAND)
+  target_link_libraries(cccl.c2h PRIVATE CUDA::curand)
+  target_compile_definitions(cccl.c2h PRIVATE C2H_HAS_CURAND=1)
+else()
+  target_compile_definitions(cccl.c2h PRIVATE C2H_HAS_CURAND=0)
+endif()
+
+set_target_properties(
+  cccl.c2h
+  PROPERTIES CXX_VISIBILITY_PRESET "default" CUDA_VISIBILITY_PRESET "default"
+)
+
+add_library(cccl.c2h.main OBJECT catch2_runner.cu catch2_runner_helper.cu)
+target_link_libraries(cccl.c2h.main PUBLIC cccl.c2h)

cccl_upstream/c2h/catch2_runner.cu

@@ -0,0 +1,10 @@
+// SPDX-FileCopyrightText: Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3
+
+//! @file
+//! This file includes a custom Catch2 main function when CMake is configured to build all tests into a single
+//! executable.
+
+#define C2H_CONFIG_MAIN
+#define C2H_EXCLUDE_CATCH2_HELPER_IMPL
+#include <c2h/catch2_main.h>

cccl_upstream/c2h/catch2_runner_helper.cu

@@ -0,0 +1,9 @@
+// SPDX-FileCopyrightText: Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3
+
+//! @file
+//! This file includes CUDA-specific utilities for custom Catch2 main function when CMake is configured to build all
+//! tests into a single executable. In this case, we have to have a CUDA target in the final Catch2 executable,
+//! otherwise CMake confuses linker options and MSVC/RDC build fails.
+
+#include "catch2_runner_helper.inl"

cccl_upstream/c2h/catch2_runner_helper.inl

@@ -0,0 +1,38 @@
+// SPDX-FileCopyrightText: Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3
+
+#pragma once
+
+//! @file
+//! This file includes implementation of CUDA-specific utilities for custom Catch2 main. When CMake is configured to
+//! include all the tests into a single executable, this file is only included into catch2_runner_helper.cu. When CMake
+//! is configured to compile each test as a separate binary, this file is included into each test.
+
+#include <iostream>
+
+int device_guard(int device_id)
+{
+  int device_count{};
+  if (cudaGetDeviceCount(&device_count) != cudaSuccess)
+  {
+    std::cerr << "Failed getting number of devices" << '\n';
+    std::exit(-1);
+  }
+
+  if (device_id >= device_count || device_id < 0)
+  {
+    std::cerr << "Invalid device ID: " << device_id << '\n';
+    std::exit(-1);
+  }
+
+  return device_id;
+}
+
+void set_device(int device_id)
+{
+  if (cudaSetDevice(device_guard(device_id)) != cudaSuccess)
+  {
+    std::cerr << "Failed to set device ID: " << device_id << '\n';
+    std::exit(-1);
+  }
+}

cccl_upstream/c2h/generators.cu

@@ -0,0 +1,291 @@
+// SPDX-FileCopyrightText: Copyright (c) 2011-2025, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3-Clause
+
+#include <cub/device/device_copy.cuh>
+
+#include <thrust/for_each.h>
+#include <thrust/iterator/counting_iterator.h>
+#include <thrust/iterator/transform_iterator.h>
+#include <thrust/tabulate.h>
+
+#include <cuda/iterator>
+#include <cuda/std/optional>
+
+#include <c2h/bfloat16.cuh>
+#include <c2h/custom_type.h>
+#include <c2h/detail/generators.cuh>
+#include <c2h/device_policy.h>
+#include <c2h/extended_types.h>
+#include <c2h/generators.h>
+#include <c2h/half.cuh>
+#include <c2h/vector.h>
+
+#if C2H_HAS_CURAND
+#  include <curand.h>
+#else
+#  include <thrust/random.h>
+#endif
+
+namespace c2h::detail
+{
+#if !C2H_HAS_CURAND
+struct i_to_rnd_t
+{
+  __host__ __device__ i_to_rnd_t(thrust::default_random_engine engine)
+      : m_engine(engine)
+  {}
+
+  thrust::default_random_engine m_engine{};
+
+  template <typename IndexType>
+  __host__ __device__ float operator()(IndexType n)
+  {
+    m_engine.discard(n);
+    return thrust::uniform_real_distribution<float>{0.0f, 1.0f}(m_engine);
+  }
+};
+#endif // !C2H_HAS_CURAND
+
+class generator_t
+{
+public:
+  generator_t()
+  {
+#if C2H_HAS_CURAND
+    curandCreateGenerator(&m_gen, CURAND_RNG_PSEUDO_DEFAULT);
+#endif
+  }
+
+  ~generator_t()
+  {
+#if C2H_HAS_CURAND
+    curandDestroyGenerator(m_gen);
+#endif
+  }
+
+  float* prepare_random_generator(seed_t seed, std::size_t num_items)
+  {
+    m_distribution.resize(num_items);
+
+#if C2H_HAS_CURAND
+    curandSetPseudoRandomGeneratorSeed(m_gen, seed.get());
+#else
+    m_gen.seed(seed.get());
+#endif
+
+    generate();
+
+    return thrust::raw_pointer_cast(m_distribution.data());
+  }
+
+  // re-fills the currently held distribution vector with new random values
+  void generate()
+  {
+#if C2H_HAS_CURAND
+    curandGenerateUniform(m_gen, thrust::raw_pointer_cast(m_distribution.data()), m_distribution.size());
+#else
+    thrust::tabulate(device_policy, m_distribution.begin(), m_distribution.end(), i_to_rnd_t{m_gen});
+    m_gen.discard(m_distribution.size());
+#endif
+  }
+
+private:
+#if C2H_HAS_CURAND
+  curandGenerator_t
+#else
+  thrust::default_random_engine
+#endif
+    m_gen;
+  c2h::device_vector<float> m_distribution;
+};
+
+// global generator state
+cuda::std::optional<generator_t> generator;
+
+void init_generator()
+{
+  _CCCL_VERIFY(!generator.has_value(), "");
+  generator.emplace();
+}
+
+float* prepare_random_data(seed_t seed, std::size_t num_items)
+{
+  return generator.value().prepare_random_generator(seed, num_items);
+}
+
+void cleanup_generator()
+{
+  _CCCL_VERIFY(generator.has_value(), "");
+  generator.reset();
+}
+
+struct random_to_custom_t
+{
+  static constexpr std::size_t m_max_key = std::numeric_limits<std::size_t>::max();
+
+  __device__ void operator()(std::size_t idx) const
+  {
+    auto out = reinterpret_cast<custom_type_state_t*>(m_out + idx * m_element_size);
+    out->key = static_cast<std::size_t>(static_cast<float>(m_max_key) * m_in[idx * 2 + 0]);
+    out->val = static_cast<std::size_t>(static_cast<float>(m_max_key) * m_in[idx * 2 + 1]);
+  }
+
+  float* m_in{};
+  char* m_out{};
+  std::size_t m_element_size{};
+};
+
+void gen_custom_type_state(
+  seed_t seed,
+  char* d_out,
+  custom_type_state_t /* min */,
+  custom_type_state_t /* max */,
+  std::size_t elements,
+  std::size_t element_size)
+{
+  // FIXME(bgruber): implement min/max handling for custom_type_state_t
+  float* d_in = prepare_random_data(seed, elements * 2);
+  thrust::for_each(device_policy,
+                   thrust::counting_iterator<std::size_t>{0},
+                   thrust::counting_iterator<std::size_t>{elements},
+                   random_to_custom_t{d_in, d_out, element_size});
+}
+
+template <typename T>
+struct spaced_out_it_op
+{
+  char* base_it;
+  std::size_t element_size;
+
+  __host__ __device__ __forceinline__ T& operator()(std::size_t offset) const
+  {
+    return *reinterpret_cast<T*>(base_it + (element_size * offset));
+  }
+};
+
+template <typename T>
+struct offset_to_iterator_t
+{
+  char* base_it;
+  std::size_t element_size;
+
+  __host__
+    __device__ __forceinline__ thrust::transform_iterator<spaced_out_it_op<T>, thrust::counting_iterator<std::size_t>>
+    operator()(std::size_t offset) const
+  {
+    // The pointer to the beginning of this "buffer" (aka a series of same "keys")
+    auto base_ptr = base_it + (element_size * offset);
+
+    // We need to make sure that the i-th element within this "buffer" is spaced out by
+    // `element_size`
+    auto counting_it = thrust::make_counting_iterator(std::size_t{0});
+    spaced_out_it_op<T> space_out_op{base_ptr, element_size};
+    return thrust::make_transform_iterator(counting_it, space_out_op);
+  }
+};
+
+template <class T>
+struct repeat_index_t
+{
+  __host__ __device__ __forceinline__ cuda::constant_iterator<T> operator()(std::size_t i)
+  {
+    return cuda::constant_iterator<T>(static_cast<T>(i));
+  }
+};
+
+template <>
+struct repeat_index_t<custom_type_state_t>
+{
+  __host__ __device__ __forceinline__ cuda::constant_iterator<custom_type_state_t> operator()(std::size_t i)
+  {
+    custom_type_state_t item{};
+    item.key = i;
+    item.val = i;
+    return cuda::constant_iterator<custom_type_state_t>(item);
+  }
+};
+
+template <typename OffsetT>
+struct offset_to_size_t
+{
+  const OffsetT* offsets;
+
+  __host__ __device__ __forceinline__ std::size_t operator()(std::size_t i)
+  {
+    return offsets[i + 1] - offsets[i];
+  }
+};
+
+/**
+ * @brief Initializes key-segment ranges from an offsets-array like the one given by
+ * `gen_uniform_offset`.
+ */
+template <typename OffsetT, typename KeyT>
+void init_key_segments(::cuda::std::span<const OffsetT> segment_offsets, KeyT* d_out, std::size_t element_size)
+{
+  OffsetT total_segments   = static_cast<OffsetT>(segment_offsets.size() - 1);
+  const OffsetT* d_offsets = segment_offsets.data();
+
+  thrust::counting_iterator<int> iota(0);
+  offset_to_iterator_t<KeyT> dst_transform_op{reinterpret_cast<char*>(d_out), element_size};
+
+  auto d_range_srcs  = thrust::make_transform_iterator(iota, repeat_index_t<KeyT>{});
+  auto d_range_dsts  = thrust::make_transform_iterator(d_offsets, dst_transform_op);
+  auto d_range_sizes = thrust::make_transform_iterator(iota, offset_to_size_t<OffsetT>{d_offsets});
+
+#if THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+  std::uint8_t* d_temp_storage   = nullptr;
+  std::size_t temp_storage_bytes = 0;
+  // TODO(bgruber): replace by a non-CUB implementation
+  cub::DeviceCopy::Batched(
+    d_temp_storage, temp_storage_bytes, d_range_srcs, d_range_dsts, d_range_sizes, total_segments);
+
+#  if THRUST_VERSION >= 300100
+  device_vector<std::uint8_t> temp_storage(temp_storage_bytes, thrust::no_init);
+#  else
+  device_vector<std::uint8_t> temp_storage(temp_storage_bytes);
+#  endif // THRUST_VERSION >= 300100
+
+  d_temp_storage = thrust::raw_pointer_cast(temp_storage.data());
+
+  // TODO(bgruber): replace by a non-CUB implementation
+  cub::DeviceCopy::Batched(
+    d_temp_storage, temp_storage_bytes, d_range_srcs, d_range_dsts, d_range_sizes, total_segments);
+  cudaDeviceSynchronize();
+#else // THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+  static_assert(sizeof(OffsetT) == 0, "Need to implement a non-CUB version of cub::DeviceCopy::Batched");
+  // TODO(bgruber): implement and *test* a non-CUB version, here is a sketch:
+  // thrust::for_each(
+  //   thrust::device,
+  //   thrust::counting_iterator<OffsetT>{0},
+  //   thrust::counting_iterator<OffsetT>{total_segments},
+  //   [&](OffsetT i) {
+  //     const auto value = d_range_srcs[i];
+  //     const auto start = d_range_sizes[i];
+  //     const auto end   = d_range_sizes[i + 1];
+  //     for (auto j = start; j < end; ++j)
+  //     {
+  //       d_range_dsts[j] = value;
+  //     }
+  //   });
+#endif // THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+}
+
+template void
+init_key_segments(::cuda::std::span<const std::uint32_t> segment_offsets, std::int32_t* out, std::size_t element_size);
+template void
+init_key_segments(::cuda::std::span<const std::uint32_t> segment_offsets, std::uint8_t* out, std::size_t element_size);
+template void
+init_key_segments(::cuda::std::span<const std::uint32_t> segment_offsets, float* out, std::size_t element_size);
+template void init_key_segments(
+  ::cuda::std::span<const std::uint32_t> segment_offsets, custom_type_state_t* out, std::size_t element_size);
+#if TEST_HALF_T()
+template void
+init_key_segments(::cuda::std::span<const std::uint32_t> segment_offsets, half_t* out, std::size_t element_size);
+#endif // TEST_HALF_T()
+
+#if TEST_BF_T()
+template void
+init_key_segments(::cuda::std::span<const std::uint32_t> segment_offsets, bfloat16_t* out, std::size_t element_size);
+#endif // TEST_BF_T()
+} // namespace c2h::detail

cccl_upstream/c2h/generators_gen_values.cu

@@ -0,0 +1,96 @@
+// SPDX-FileCopyrightText: Copyright (c) 2011-2025, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3-Clause
+
+#include <thrust/tabulate.h>
+#include <thrust/transform.h>
+
+#include <c2h/bfloat16.cuh>
+#include <c2h/detail/generators.cuh>
+#include <c2h/device_policy.h>
+#include <c2h/extended_types.h>
+#include <c2h/generators.h>
+#include <c2h/half.cuh>
+
+namespace c2h::detail
+{
+template <typename T>
+void gen_values_between(seed_t seed, ::cuda::std::span<T> data, T min, T max)
+{
+  const auto* dist = prepare_random_data(seed, data.size());
+  thrust::transform(device_policy, dist, dist + data.size(), data.begin(), random_to_item_t<T>(min, max));
+}
+
+template <typename T>
+struct counter_to_cyclic_item_t
+{
+  std::size_t n;
+
+  template <typename CounterT>
+  __device__ T operator()(CounterT id)
+  {
+    // This has to be a type for which extended floating point types like __nv_fp8_e5m2 provide an overload
+    return static_cast<T>(static_cast<float>(static_cast<uint64_t>(id) % n));
+  }
+};
+
+template <typename T>
+void gen_values_cyclic(modulo_t mod, ::cuda::std::span<T> data)
+{
+  thrust::tabulate(device_policy, data.begin(), data.end(), counter_to_cyclic_item_t<T>{mod.get()});
+}
+
+#define INSTANTIATE_RND(TYPE) \
+  template void gen_values_between<TYPE>(seed_t, ::cuda::std::span<TYPE> data, TYPE min, TYPE max)
+#define INSTANTIATE_MOD(TYPE) template void gen_values_cyclic<TYPE>(modulo_t, ::cuda::std::span<TYPE> data)
+
+#define INSTANTIATE(TYPE) \
+  INSTANTIATE_RND(TYPE);  \
+  INSTANTIATE_MOD(TYPE)
+
+INSTANTIATE(std::uint8_t);
+INSTANTIATE(std::uint16_t);
+INSTANTIATE(std::uint32_t);
+INSTANTIATE(std::uint64_t);
+
+INSTANTIATE(std::int8_t);
+INSTANTIATE(std::int16_t);
+INSTANTIATE(std::int32_t);
+INSTANTIATE(std::int64_t);
+
+#if _CCCL_HAS_NVFP8()
+INSTANTIATE(__nv_fp8_e5m2);
+INSTANTIATE(__nv_fp8_e4m3);
+#endif // _CCCL_HAS_NVFP8()
+INSTANTIATE(float);
+INSTANTIATE(double);
+INSTANTIATE(cuda::std::complex<float>);
+INSTANTIATE(cuda::std::complex<double>);
+
+INSTANTIATE(bool);
+INSTANTIATE(char);
+
+#if TEST_HALF_T()
+INSTANTIATE(half_t);
+INSTANTIATE(__half);
+#  if _CCCL_CTK_AT_LEAST(12, 2)
+INSTANTIATE(cuda::std::complex<__half>);
+#  endif // _CCCL_CTK_AT_LEAST(12, 2)
+#endif // TEST_HALF_T()
+
+#if TEST_BF_T()
+INSTANTIATE(bfloat16_t);
+INSTANTIATE(__nv_bfloat16);
+#  if _CCCL_CTK_AT_LEAST(12, 2)
+INSTANTIATE(cuda::std::complex<__nv_bfloat16>);
+#  endif // _CCCL_CTK_AT_LEAST(12, 2)
+#endif // TEST_BF_T()
+
+#if TEST_INT128()
+INSTANTIATE(__int128_t);
+INSTANTIATE(__uint128_t);
+#endif // TEST_INT128()
+
+#undef INSTANTIATE_RND
+#undef INSTANTIATE_MOD
+#undef INSTANTIATE
+} // namespace c2h::detail

cccl_upstream/c2h/generators_uniform_offsets.cu

@@ -0,0 +1,66 @@
+// SPDX-FileCopyrightText: Copyright (c) 2011-2025, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3-Clause
+
+#include <c2h/generators.h>
+// #include <c2h/detail/generators.cuh>
+
+#include <thrust/device_ptr.h>
+#include <thrust/find.h>
+#include <thrust/scan.h>
+
+#include <cuda/std/cstdint>
+#include <cuda/std/span>
+
+#include <c2h/device_policy.h>
+
+namespace c2h::detail
+{
+template <class T>
+struct greater_equal_op
+{
+  T val;
+
+  __device__ bool operator()(T x)
+  {
+    return x >= val;
+  }
+};
+
+template <typename T>
+std::size_t gen_uniform_offsets(
+  seed_t seed, cuda::std::span<T> segment_offsets, T total_elements, T min_segment_size, T max_segment_size)
+{
+  gen_values_between(seed, segment_offsets, min_segment_size, max_segment_size);
+  *thrust::device_ptr<T>(&segment_offsets[total_elements]) = total_elements + 1;
+  thrust::exclusive_scan(device_policy, segment_offsets.begin(), segment_offsets.end(), segment_offsets.begin());
+  const auto iter =
+    thrust::find_if(device_policy, segment_offsets.begin(), segment_offsets.end(), greater_equal_op<T>{total_elements});
+  *thrust::device_ptr<T>(&*iter) = total_elements;
+  return iter - segment_offsets.begin() + 1;
+}
+
+template std::size_t gen_uniform_offsets(
+  seed_t seed,
+  cuda::std::span<int32_t> segment_offsets,
+  int32_t total_elements,
+  int32_t min_segment_size,
+  int32_t max_segment_size);
+template std::size_t gen_uniform_offsets(
+  seed_t seed,
+  cuda::std::span<uint32_t> segment_offsets,
+  uint32_t total_elements,
+  uint32_t min_segment_size,
+  uint32_t max_segment_size);
+template std::size_t gen_uniform_offsets(
+  seed_t seed,
+  cuda::std::span<int64_t> segment_offsets,
+  int64_t total_elements,
+  int64_t min_segment_size,
+  int64_t max_segment_size);
+template std::size_t gen_uniform_offsets(
+  seed_t seed,
+  cuda::std::span<uint64_t> segment_offsets,
+  uint64_t total_elements,
+  uint64_t min_segment_size,
+  uint64_t max_segment_size);
+} // namespace c2h::detail

cccl_upstream/c2h/generators_vector.cu

@@ -0,0 +1,167 @@
+// SPDX-FileCopyrightText: Copyright (c) 2011-2022, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3-Clause
+
+#include <thrust/tabulate.h>
+
+#include <cuda/std/cstddef>
+#include <cuda/std/cstdint>
+
+#include <c2h/detail/generators.cuh>
+#include <c2h/device_policy.h>
+#include <c2h/extended_types.h>
+#include <c2h/fill_striped.h>
+#include <c2h/generators.h>
+#include <c2h/vector.h>
+
+namespace c2h::detail
+{
+template <typename T, int VecSize>
+struct random_to_vec_item_t
+{
+  __device__ void operator()(std::size_t idx)
+  {
+#define SET_FIELD(VEC_FIELD) \
+  m_out[idx].VEC_FIELD = random_to_item_t<decltype(m_min.VEC_FIELD)>(m_min.VEC_FIELD, m_max.VEC_FIELD)(m_in[idx]);
+
+    if constexpr (VecSize >= 4)
+    {
+      SET_FIELD(w);
+    }
+    if constexpr (VecSize >= 3)
+    {
+      SET_FIELD(z);
+    }
+    if constexpr (VecSize >= 2)
+    {
+      SET_FIELD(y);
+    }
+    if constexpr (VecSize >= 1)
+    {
+      SET_FIELD(x);
+    }
+#undef SET_FIELD
+  }
+
+  T m_min;
+  T m_max;
+  const float* m_in{};
+  T* m_out{};
+};
+
+#if THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+#  define VEC_SPECIALIZATION(T)                                                                                   \
+    template <>                                                                                                   \
+    void gen_values_between(seed_t seed, ::cuda::std::span<T> data, T min, T max)                                 \
+    {                                                                                                             \
+      const auto* dist = prepare_random_data(seed, data.size());                                                  \
+      auto op          = random_to_vec_item_t<T, ::cuda::std::tuple_size_v<T>>{min, max, dist, data.data()};      \
+      thrust::for_each(                                                                                           \
+        device_policy, thrust::counting_iterator<size_t>{0}, thrust::counting_iterator<size_t>{data.size()}, op); \
+    }
+
+VEC_SPECIALIZATION(char2);
+VEC_SPECIALIZATION(char3);
+VEC_SPECIALIZATION(char4);
+
+// VEC_SPECIALIZATION(uchar2);
+VEC_SPECIALIZATION(uchar3);
+// VEC_SPECIALIZATION(uchar4);
+
+VEC_SPECIALIZATION(short2);
+VEC_SPECIALIZATION(short3);
+VEC_SPECIALIZATION(short4);
+
+VEC_SPECIALIZATION(ushort2);
+
+VEC_SPECIALIZATION(int2);
+VEC_SPECIALIZATION(int3);
+VEC_SPECIALIZATION(int4);
+
+// VEC_SPECIALIZATION(uint2);
+// VEC_SPECIALIZATION(uint3);
+// VEC_SPECIALIZATION(uint4);
+
+VEC_SPECIALIZATION(long2);
+VEC_SPECIALIZATION(long3);
+#  if _CCCL_CTK_AT_LEAST(13, 0)
+VEC_SPECIALIZATION(long4_16a);
+VEC_SPECIALIZATION(long4_32a);
+#  else
+VEC_SPECIALIZATION(long4);
+#  endif // _CCCL_CTK_AT_LEAST(13, 0)
+
+// VEC_SPECIALIZATION(ulong2);
+// VEC_SPECIALIZATION(ulong3);
+// VEC_SPECIALIZATION(ulong4);
+
+VEC_SPECIALIZATION(longlong2);
+VEC_SPECIALIZATION(longlong3);
+#  if _CCCL_CTK_AT_LEAST(13, 0)
+VEC_SPECIALIZATION(longlong4_16a);
+VEC_SPECIALIZATION(longlong4_32a);
+#  else
+VEC_SPECIALIZATION(longlong4);
+#  endif // _CCCL_CTK_AT_LEAST(13, 0)
+
+VEC_SPECIALIZATION(ulonglong2);
+// VEC_SPECIALIZATION(ulonglong3);
+#  if _CCCL_CTK_AT_LEAST(13, 0)
+VEC_SPECIALIZATION(ulonglong4_16a);
+VEC_SPECIALIZATION(ulonglong4_32a);
+#  else
+VEC_SPECIALIZATION(ulonglong4);
+#  endif // _CCCL_CTK_AT_LEAST(13, 0)
+
+VEC_SPECIALIZATION(float2);
+VEC_SPECIALIZATION(float3);
+VEC_SPECIALIZATION(float4);
+
+VEC_SPECIALIZATION(double2);
+VEC_SPECIALIZATION(double3);
+#  if _CCCL_CTK_AT_LEAST(13, 0)
+VEC_SPECIALIZATION(double4_16a);
+VEC_SPECIALIZATION(double4_32a);
+#  else
+VEC_SPECIALIZATION(double4);
+#  endif // _CCCL_CTK_AT_LEAST(13, 0)
+
+#  if CCCL_VERSION > 3001000
+#    if TEST_HALF_T()
+VEC_SPECIALIZATION(__half2);
+#    endif // TEST_HALF_T()
+#    if TEST_BF_T()
+VEC_SPECIALIZATION(__nv_bfloat162);
+#    endif // TEST_BF_T()
+#  endif // CCCL_VERSION > 3001000
+
+template <typename VecType, typename Type>
+struct counter_to_cyclic_vector_t
+{
+  std::size_t n;
+
+  template <typename CounterT>
+  __device__ VecType operator()(CounterT id) const
+  {
+    return scalar_to_vec_t<VecType>{}(static_cast<Type>(id) % n);
+  }
+};
+
+#  define VEC_GEN_MOD_SPECIALIZATION(VEC_TYPE, SCALAR_TYPE)                                                     \
+    template <>                                                                                                 \
+    void gen_values_cyclic<VEC_TYPE>(modulo_t mod, ::cuda::std::span<VEC_TYPE> data)                            \
+    {                                                                                                           \
+      thrust::tabulate(                                                                                         \
+        device_policy, data.begin(), data.end(), counter_to_cyclic_vector_t<VEC_TYPE, SCALAR_TYPE>{mod.get()}); \
+    }
+
+VEC_GEN_MOD_SPECIALIZATION(short2, short);
+VEC_GEN_MOD_SPECIALIZATION(uchar3, unsigned char);
+#  if _CCCL_CTK_AT_LEAST(13, 0)
+VEC_GEN_MOD_SPECIALIZATION(ulonglong4_16a, unsigned long long);
+VEC_GEN_MOD_SPECIALIZATION(ulonglong4_32a, unsigned long long);
+#  else
+VEC_GEN_MOD_SPECIALIZATION(ulonglong4, unsigned long long);
+#  endif // _CCCL_CTK_AT_LEAST(13, 0)
+VEC_GEN_MOD_SPECIALIZATION(ushort4, unsigned short);
+#endif // THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+} // namespace c2h::detail

cccl_upstream/c2h/include/c2h/bfloat16.cuh

@@ -0,0 +1,256 @@
+// SPDX-FileCopyrightText: Copyright (c) 2011-2021, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3-Clause
+
+#pragma once
+
+/**
+ * \file
+ * Utilities for interacting with the opaque CUDA __nv_bfloat16 type
+ */
+
+#include <cuda_bf16.h>
+
+#include <cub/util_type.cuh>
+
+#include <cuda/std/limits>
+#include <cuda/std/type_traits>
+
+#include <cstdint>
+#include <iosfwd>
+
+#ifdef __GNUC__
+// There's a ton of type-punning going on in this file.
+#  pragma GCC diagnostic push
+#  pragma GCC diagnostic ignored "-Wstrict-aliasing"
+#endif
+
+/******************************************************************************
+ * bfloat16_t
+ ******************************************************************************/
+
+/**
+ * Host-based fp16 data type compatible and convertible with __nv_bfloat16
+ */
+struct bfloat16_t
+{
+  uint16_t __x;
+
+  /// Constructor from __nv_bfloat16
+  __host__ __device__ __forceinline__ explicit bfloat16_t(const __nv_bfloat16& other)
+  {
+    __x = reinterpret_cast<const uint16_t&>(other);
+  }
+
+  /// Constructor from integer
+  __host__ __device__ __forceinline__ explicit bfloat16_t(int a)
+  {
+    *this = bfloat16_t(float(a));
+  }
+
+  /// Constructor from std::size_t
+  __host__ __device__ __forceinline__ explicit bfloat16_t(std::size_t a)
+  {
+    *this = bfloat16_t(float(a));
+  }
+
+  /// Constructor from double
+  __host__ __device__ __forceinline__ explicit bfloat16_t(double a)
+  {
+    *this = bfloat16_t(float(a));
+  }
+
+  /// Constructor from unsigned long long int
+  template <typename T,
+            typename = typename ::cuda::std::enable_if<
+              ::cuda::std::is_same<T, unsigned long long int>::value
+              && (!::cuda::std::is_same<std::size_t, unsigned long long int>::value)>::type>
+  __host__ __device__ __forceinline__ explicit bfloat16_t(T a)
+  {
+    *this = bfloat16_t(float(a));
+  }
+
+  /// Default constructor
+  bfloat16_t() = default;
+
+  /// Constructor from float
+  __host__ __device__ __forceinline__ explicit bfloat16_t(float a)
+  {
+    // Reference:
+    // https://github.com/pytorch/pytorch/blob/44cc873fba5e5ffc4d4d4eef3bd370b653ce1ce1/c10/util/BFloat16.h#L51
+    uint16_t ir;
+    if (a != a)
+    {
+      ir = UINT16_C(0x7FFF);
+    }
+    else
+    {
+      union
+      {
+        uint32_t U32;
+        float F32;
+      };
+
+      F32                    = a;
+      uint32_t rounding_bias = ((U32 >> 16) & 1) + UINT32_C(0x7FFF);
+      ir                     = static_cast<uint16_t>((U32 + rounding_bias) >> 16);
+    }
+    this->__x = ir;
+  }
+
+  /// Cast to __nv_bfloat16
+  __host__ __device__ __forceinline__ operator __nv_bfloat16() const
+  {
+    return reinterpret_cast<const __nv_bfloat16&>(__x);
+  }
+
+  /// Cast to float
+  __host__ __device__ __forceinline__ operator float() const
+  {
+    float f     = 0;
+    uint32_t* p = reinterpret_cast<uint32_t*>(&f);
+    *p          = uint32_t(__x) << 16;
+    return f;
+  }
+
+  /// Get raw storage
+  __host__ __device__ __forceinline__ uint16_t raw() const
+  {
+    return this->__x;
+  }
+
+  /// Equality
+  __host__ __device__ __forceinline__ friend bool operator==(const bfloat16_t& a, const bfloat16_t& b)
+  {
+    return (a.__x == b.__x);
+  }
+
+  /// Inequality
+  __host__ __device__ __forceinline__ friend bool operator!=(const bfloat16_t& a, const bfloat16_t& b)
+  {
+    return (a.__x != b.__x);
+  }
+
+  /// Assignment by sum
+  __host__ __device__ __forceinline__ bfloat16_t& operator+=(const bfloat16_t& rhs)
+  {
+    *this = bfloat16_t(float(*this) + float(rhs));
+    return *this;
+  }
+
+  /// Multiply
+  __host__ __device__ __forceinline__ bfloat16_t operator*(const bfloat16_t& other) const
+  {
+    return bfloat16_t(float(*this) * float(other));
+  }
+
+  /// Add
+  __host__ __device__ __forceinline__ bfloat16_t operator+(const bfloat16_t& other) const
+  {
+    return bfloat16_t(float(*this) + float(other));
+  }
+
+  /// Sub
+  __host__ __device__ __forceinline__ bfloat16_t operator-(const bfloat16_t& other) const
+  {
+    return bfloat16_t(float(*this) - float(other));
+  }
+
+  /// Less-than
+  __host__ __device__ __forceinline__ bool operator<(const bfloat16_t& other) const
+  {
+    return float(*this) < float(other);
+  }
+
+  /// Less-than-equal
+  __host__ __device__ __forceinline__ bool operator<=(const bfloat16_t& other) const
+  {
+    return float(*this) <= float(other);
+  }
+
+  /// Greater-than
+  __host__ __device__ __forceinline__ bool operator>(const bfloat16_t& other) const
+  {
+    return float(*this) > float(other);
+  }
+
+  /// Greater-than-equal
+  __host__ __device__ __forceinline__ bool operator>=(const bfloat16_t& other) const
+  {
+    return float(*this) >= float(other);
+  }
+
+  /// numeric_traits<bfloat16_t>::max
+  __host__ __device__ __forceinline__ static bfloat16_t(max)()
+  {
+    uint16_t max_word = 0x7F7F;
+    return reinterpret_cast<bfloat16_t&>(max_word);
+  }
+
+  /// numeric_traits<bfloat16_t>::lowest
+  __host__ __device__ __forceinline__ static bfloat16_t lowest()
+  {
+    uint16_t lowest_word = 0xFF7F;
+    return reinterpret_cast<bfloat16_t&>(lowest_word);
+  }
+};
+
+/******************************************************************************
+ * I/O stream overloads
+ ******************************************************************************/
+
+/// Insert formatted \p bfloat16_t into the output stream
+inline std::ostream& operator<<(std::ostream& out, const bfloat16_t& x)
+{
+  out << (float) x;
+  return out;
+}
+
+/// Insert formatted \p __nv_bfloat16 into the output stream
+inline std::ostream& operator<<(std::ostream& out, const __nv_bfloat16& x)
+{
+  return out << bfloat16_t(x);
+}
+
+/******************************************************************************
+ * Traits overloads
+ ******************************************************************************/
+
+namespace cuda
+{
+template <>
+inline constexpr bool is_floating_point_v<bfloat16_t> = true;
+}
+
+template <>
+class cuda::std::numeric_limits<bfloat16_t>
+{
+public:
+  static constexpr bool is_specialized = true;
+
+  static _CCCL_HOST_DEVICE _CCCL_FORCEINLINE bfloat16_t max()
+  {
+    return bfloat16_t(numeric_limits<__nv_bfloat16>::max());
+  }
+
+  static _CCCL_HOST_DEVICE _CCCL_FORCEINLINE bfloat16_t min()
+  {
+    return bfloat16_t(numeric_limits<__nv_bfloat16>::min());
+  }
+
+  static _CCCL_HOST_DEVICE _CCCL_FORCEINLINE bfloat16_t lowest()
+  {
+    return bfloat16_t(numeric_limits<__nv_bfloat16>::lowest());
+  }
+};
+
+CUB_NAMESPACE_BEGIN
+
+template <>
+struct NumericTraits<bfloat16_t> : BaseTraits<FLOATING_POINT, true, uint16_t, bfloat16_t>
+{};
+
+CUB_NAMESPACE_END
+
+#ifdef __GNUC__
+#  pragma GCC diagnostic pop
+#endif

cccl_upstream/c2h/include/c2h/catch2_main.h

@@ -0,0 +1,51 @@
+// SPDX-FileCopyrightText: Copyright (c) 2011-2022, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3
+
+#pragma once
+
+#include <thrust/detail/config/device_system.h>
+
+#include <c2h/detail/generators.cuh>
+
+//! @file
+//! This file includes a custom Catch2 main function. When CMake is configured to build each test as a separate
+//! executable, this header is included into each test. On the other hand, when all the tests are compiled into a single
+//! executable, this header is excluded from the tests and included into catch2_runner.cpp
+
+#include <catch2/catch_session.hpp>
+
+#ifdef C2H_CONFIG_MAIN
+#  if THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+#    include <c2h/catch2_runner_helper.h>
+
+#    ifndef C2H_EXCLUDE_CATCH2_HELPER_IMPL
+#      include "catch2_runner_helper.inl"
+#    endif // !C2H_EXCLUDE_CATCH2_HELPER_IMPL
+#  endif // THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+
+int main(int argc, char* argv[])
+{
+  Catch::Session session;
+
+#  if THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+  int device_id{};
+
+  // Build a new parser on top of Catch's
+  using namespace Catch::Clara;
+  auto cli = session.cli() | Opt(device_id, "device")["-d"]["--device"]("device id to use");
+  session.cli(cli);
+
+  int returnCode = session.applyCommandLine(argc, argv);
+  if (returnCode != 0)
+  {
+    return returnCode;
+  }
+
+  set_device(device_id);
+#  endif // THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+  c2h::detail::init_generator();
+  const auto ret = session.run();
+  c2h::detail::cleanup_generator();
+  return ret;
+}
+#endif // C2H_CONFIG_MAIN

cccl_upstream/c2h/include/c2h/catch2_runner_helper.h

@@ -0,0 +1,7 @@
+// SPDX-FileCopyrightText: Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3
+
+#pragma once
+
+int device_guard(int device_id);
+void set_device(int device_id);

cccl_upstream/c2h/include/c2h/catch2_test_helper.h

@@ -0,0 +1,670 @@
+// SPDX-FileCopyrightText: Copyright (c) 2011-2022, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3-Clause
+
+#pragma once
+
+#include <cuda/std/detail/__config>
+
+#include <cuda/__memory_resource/legacy_pinned_memory_resource.h>
+#include <cuda/__nvtx/nvtx.h>
+#include <cuda/buffer>
+#include <cuda/std/bit>
+#include <cuda/std/cmath>
+#include <cuda/std/limits>
+#include <cuda/std/type_traits>
+#include <cuda/std/utility>
+
+#include <cstdint>
+#include <cstdlib>
+#include <iomanip>
+#include <tuple>
+#include <type_traits>
+
+#include <c2h/catch2_main.h>
+#include <c2h/catch2_test_macros.h>
+#include <c2h/checked_allocator.cuh>
+#include <c2h/device_policy.h>
+#include <c2h/extended_types.h>
+#include <c2h/test_util_vec.h>
+#include <c2h/utility.h>
+#include <c2h/vector.h>
+#include <catch2/catch_template_test_macros.hpp>
+#include <catch2/catch_test_macros.hpp>
+#include <catch2/generators/catch_generators_all.hpp>
+#include <catch2/matchers/catch_matchers.hpp>
+#include <catch2/matchers/catch_matchers_templated.hpp>
+#include <catch2/matchers/catch_matchers_vector.hpp>
+
+#ifndef VAR_IDX
+#  define VAR_IDX 0
+#endif
+
+namespace c2h
+{
+template <typename... Ts>
+using type_list = ::cuda::std::__type_list<Ts...>;
+
+template <typename TypeList>
+using size = ::cuda::std::__type_list_size<TypeList>;
+
+template <std::size_t Index, typename TypeList>
+using get = ::cuda::std::__type_at_c<Index, TypeList>;
+
+template <class... TypeLists>
+using cartesian_product = ::cuda::std::__type_cartesian_product<TypeLists...>;
+
+template <typename T, T... Ts>
+using enum_type_list = ::cuda::std::__type_value_list<T, Ts...>;
+
+template <typename T0, typename T1>
+using pair = ::cuda::std::__type_pair<T0, T1>;
+
+template <typename P>
+using first = ::cuda::std::__type_pair_first<P>;
+
+template <typename P>
+using second = ::cuda::std::__type_pair_second<P>;
+
+template <std::size_t Start, std::size_t Size, std::size_t Stride = 1>
+using iota = ::cuda::std::__type_iota<std::size_t, Start, Size, Stride>;
+
+template <typename TypeList, typename T>
+using remove = ::cuda::std::__type_remove<TypeList, T>;
+
+/**
+ * Return a value of type `T` with the same bitwise representation of `in`.
+ * Types `T` and `U` must be the same size.
+ */
+template <typename T, typename U>
+__host__ __device__ constexpr T SafeBitCast(const U& in) noexcept
+{
+  static_assert(sizeof(T) == sizeof(U), "Types must be same size.");
+  T out;
+  memcpy(&out, &in, sizeof(T));
+  return out;
+}
+
+template <typename T>
+[[nodiscard]] constexpr bool isnan(T value) noexcept
+{
+  return cuda::std::isnan(value);
+}
+
+[[nodiscard]] constexpr bool isnan(float1 val) noexcept
+{
+  return (cuda::std::isnan(val.x));
+}
+
+[[nodiscard]] constexpr bool isnan(float2 val) noexcept
+{
+  return (cuda::std::isnan(val.y) || cuda::std::isnan(val.x));
+}
+
+[[nodiscard]] constexpr bool isnan(float3 val) noexcept
+{
+  return (cuda::std::isnan(val.z) || cuda::std::isnan(val.y) || cuda::std::isnan(val.x));
+}
+
+[[nodiscard]] constexpr bool isnan(float4 val) noexcept
+{
+  return (cuda::std::isnan(val.y) || cuda::std::isnan(val.x) || cuda::std::isnan(val.w) || cuda::std::isnan(val.z));
+}
+
+[[nodiscard]] constexpr bool isnan(double1 val) noexcept
+{
+  return (cuda::std::isnan(val.x));
+}
+
+[[nodiscard]] constexpr bool isnan(double2 val) noexcept
+{
+  return (cuda::std::isnan(val.y) || cuda::std::isnan(val.x));
+}
+
+[[nodiscard]] constexpr bool isnan(double3 val) noexcept
+{
+  return (cuda::std::isnan(val.z) || cuda::std::isnan(val.y) || cuda::std::isnan(val.x));
+}
+
+_CCCL_SUPPRESS_DEPRECATED_PUSH
+[[nodiscard]] constexpr bool isnan(double4 val) noexcept
+{
+  return (cuda::std::isnan(val.y) || cuda::std::isnan(val.x) || cuda::std::isnan(val.w) || cuda::std::isnan(val.z));
+}
+_CCCL_SUPPRESS_DEPRECATED_POP
+
+// TODO: move to libcu++
+#if TEST_HALF_T()
+
+[[nodiscard]] constexpr bool isnan(__half2 value) noexcept
+{
+  return cuda::std::isnan(value.x) || cuda::std::isnan(value.y);
+}
+
+[[nodiscard]] constexpr bool isnan(half_t val) noexcept
+{
+  const auto bits = SafeBitCast<uint16_t>(val);
+  // commented bit is always true, leaving for documentation:
+  return (((bits >= 0x7C01) && (bits <= 0x7FFF)) || ((bits >= 0xFC01) /*&& (bits <= 0xFFFFFFFF)*/));
+}
+
+#endif // TEST_HALF_T()
+
+#if TEST_BF_T()
+
+[[nodiscard]] constexpr bool isnan(__nv_bfloat162 value) noexcept
+{
+  return cuda::std::isnan(value.x) || cuda::std::isnan(value.y);
+}
+
+[[nodiscard]] constexpr bool isnan(bfloat16_t val) noexcept
+{
+  const auto bits = SafeBitCast<uint16_t>(val);
+  // commented bit is always true, leaving for documentation:
+  return (((bits >= 0x7F81) && (bits <= 0x7FFF)) || ((bits >= 0xFF81) /*&& (bits <= 0xFFFFFFFF)*/));
+}
+
+#endif // TEST_BF_T()
+} // namespace c2h
+
+namespace detail
+{
+template <class T>
+std::vector<T> to_vec(c2h::device_vector<T> const& vec)
+{
+  c2h::host_vector<T> temp = vec;
+  return std::vector<T>{temp.begin(), temp.end()};
+}
+
+template <class T>
+std::vector<T> to_vec(c2h::host_vector<T> const& vec)
+{
+  return std::vector<T>{vec.begin(), vec.end()};
+}
+
+template <class T>
+std::vector<T> to_vec(std::vector<T> const& vec)
+{
+  return vec;
+}
+} // namespace detail
+
+#define REQUIRE_APPROX_EQ(ref, out)                          \
+  {                                                          \
+    auto vec_ref = detail::to_vec(ref);                      \
+    auto vec_out = detail::to_vec(out);                      \
+    REQUIRE_THAT(vec_ref, Catch::Matchers::Approx(vec_out)); \
+  }
+
+#define REQUIRE_APPROX_EQ_EPSILON(ref, out, eps)                          \
+  {                                                                       \
+    auto vec_ref = detail::to_vec(ref);                                   \
+    auto vec_out = detail::to_vec(out);                                   \
+    REQUIRE_THAT(vec_ref, Catch::Matchers::Approx(vec_out).epsilon(eps)); \
+  }
+
+#define REQUIRE_APPROX_EQ_ABS(ref, out, abs)                             \
+  {                                                                      \
+    auto vec_ref = detail::to_vec(ref);                                  \
+    auto vec_out = detail::to_vec(out);                                  \
+    REQUIRE_THAT(vec_ref, Catch::Matchers::Approx(vec_out).margin(abs)); \
+  }
+
+namespace c2h::detail
+{
+// Copy of Catch2::MatchExpr, but streamReconstructedExpression does not print arg
+template <typename ArgT, typename MatcherT>
+class QuietMatchExpr : public Catch::ITransientExpression
+{
+  ArgT&& m_arg;
+  MatcherT const& m_matcher;
+
+public:
+  constexpr QuietMatchExpr(ArgT&& arg, MatcherT const& matcher)
+      : ITransientExpression{true, matcher.match(arg)}
+      , m_arg(CATCH_FORWARD(arg))
+      , m_matcher(matcher)
+  {}
+
+  void streamReconstructedExpression(std::ostream& os) const override
+  {
+    os << m_matcher.toString();
+  }
+};
+
+template <typename ArgT, typename MatcherT>
+QuietMatchExpr(ArgT&&, MatcherT) -> QuietMatchExpr<ArgT, MatcherT>;
+} // namespace c2h::detail
+
+// Copy of Catch2's INTERNAL_CHECK_THAT macro, but using QuietMatchExpr to suppress printing arg
+#define INTERNAL_CHECK_THAT_QUIET(macroName, matcher, resultDisposition, arg)        \
+  do                                                                                 \
+  {                                                                                  \
+    Catch::AssertionHandler catchAssertionHandler(                                   \
+      macroName##_catch_sr,                                                          \
+      CATCH_INTERNAL_LINEINFO,                                                       \
+      CATCH_INTERNAL_STRINGIFY(arg) ", " CATCH_INTERNAL_STRINGIFY(matcher),          \
+      resultDisposition);                                                            \
+    INTERNAL_CATCH_TRY                                                               \
+    {                                                                                \
+      catchAssertionHandler.handleExpr(::c2h::detail::QuietMatchExpr(arg, matcher)); \
+    }                                                                                \
+    INTERNAL_CATCH_CATCH(catchAssertionHandler)                                      \
+    catchAssertionHandler.complete();                                                \
+  } while (false)
+
+// Copy of Catch2's CHECK_THAT macro, but suppressing printing arg
+#define CHECK_THAT_QUIET(arg, matcher) \
+  INTERNAL_CHECK_THAT_QUIET("CHECK_THAT", matcher, Catch::ResultDisposition::ContinueOnFailure, arg)
+
+// Copy of Catch2's REQUIRE_THAT macro, but suppressing printing arg
+#define REQUIRE_THAT_QUIET(arg, matcher) \
+  INTERNAL_CHECK_THAT_QUIET("REQUIRE_THAT", matcher, Catch::ResultDisposition::Normal, arg)
+
+namespace detail
+{
+// Returns true if values are equal, or both NaN:
+struct equal_or_nans
+{
+  template <typename T>
+  bool operator()(const T& a, const T& b) const
+  {
+    return (c2h::isnan(a) && c2h::isnan(b)) || a == b;
+  }
+};
+
+struct bitwise_equal
+{
+  template <typename T>
+  bool operator()(const T& a, const T& b) const
+  {
+    return ::cuda::std::memcmp(&a, &b, sizeof(T)) == 0;
+  }
+};
+
+// Catch2 Matcher that calls `std::equal` with a default-constructable custom predicate
+template <typename Range, typename Pred>
+struct CustomEqualsRangeMatcher : Catch::Matchers::MatcherBase<Range>
+{
+  CustomEqualsRangeMatcher(Range const& range)
+      : range{range}
+  {}
+
+  bool match(Range const& other) const override
+  {
+    using std::begin;
+    using std::end;
+
+    return std::equal(begin(range), end(range), begin(other), Pred{});
+  }
+
+  std::string describe() const override
+  {
+    return "Equals: " + Catch::rangeToString(range);
+  }
+
+private:
+  Range const& range;
+};
+
+template <typename Range>
+auto NaNEqualsRange(const Range& range) -> CustomEqualsRangeMatcher<Range, equal_or_nans>
+{
+  return CustomEqualsRangeMatcher<Range, equal_or_nans>(range);
+}
+
+template <typename Range>
+auto BitwiseEqualsRange(const Range& range) -> CustomEqualsRangeMatcher<Range, bitwise_equal>
+{
+  return CustomEqualsRangeMatcher<Range, bitwise_equal>(range);
+}
+} // namespace detail
+
+#define REQUIRE_EQ_WITH_NAN_MATCHING(ref, out)              \
+  {                                                         \
+    auto vec_ref = detail::to_vec(ref);                     \
+    auto vec_out = detail::to_vec(out);                     \
+    REQUIRE_THAT(vec_ref, detail::NaNEqualsRange(vec_out)); \
+  }
+
+#define REQUIRE_BITWISE_EQ(ref, out)                        \
+  {                                                         \
+    auto vec_ref = detail::to_vec(ref);                     \
+    auto vec_out = detail::to_vec(out);                     \
+    REQUIRE_THAT(vec_ref, detail::NaNEqualsRange(vec_out)); \
+  }
+
+namespace c2h::detail
+{
+template <typename T>
+struct indexed_value_t
+{
+  size_t index;
+  T value;
+};
+
+template <typename T>
+struct element_compare_result_t
+{
+  size_t index;
+  T actual;
+  T expected;
+};
+
+template <typename T>
+struct vector_compare_result_t
+{
+  size_t actual_size;
+  size_t expected_size;
+  size_t total_mismatches;
+  std::vector<indexed_value_t<T>> good_values;
+  std::vector<element_compare_result_t<T>> first_mismatches;
+  std::optional<std::vector<element_compare_result_t<T>>> last_mismatches;
+};
+
+template <typename LhsRange, typename RhsRange, typename T = typename LhsRange::value_type>
+auto compare_host_ranges(const LhsRange& actual, const RhsRange& expected) -> vector_compare_result_t<T>
+{
+  constexpr size_t good_values_before_mismatch = 3;
+  constexpr size_t first_mismatches_count      = 5;
+  constexpr size_t last_mismatches_count       = 5;
+
+  vector_compare_result_t<T> result{};
+  result.actual_size   = actual.size();
+  result.expected_size = expected.size();
+  if (result.actual_size != result.expected_size)
+  {
+    result.total_mismatches = actual.size();
+    return result;
+  }
+
+  std::vector<element_compare_result_t<T>> mismatches;
+  mismatches.reserve(actual.size()); // TODO(bgruber): this seems excessive
+  for (size_t i = 0; i < actual.size(); ++i)
+  {
+    if (actual[i] != expected[i])
+    {
+      if (mismatches.empty()) // at the first mismatch
+      {
+        // store up to 3 good values before the first mismatch
+        const size_t count = ::cuda::std::min(good_values_before_mismatch, i);
+        for (size_t j = i - count; j < i; j++)
+        {
+          result.good_values.emplace_back(indexed_value_t<T>{j, actual[j]});
+        }
+      }
+      mismatches.emplace_back(element_compare_result_t<T>{i, actual[i], expected[i]});
+    }
+  }
+  result.total_mismatches = mismatches.size();
+
+  // Handle first mismatches
+  size_t first_count = cuda::std::min<size_t>(mismatches.size(), first_mismatches_count);
+  result.first_mismatches.assign(mismatches.begin(), mismatches.begin() + first_count);
+
+  // Handle last mismatches
+  if (mismatches.size() > first_mismatches_count)
+  {
+    const auto start =
+      mismatches.end() - cuda::std::min<size_t>(mismatches.size() - first_mismatches_count, last_mismatches_count);
+    result.last_mismatches.emplace(start, mismatches.end());
+  }
+
+  return result;
+}
+
+template <typename T>
+auto compare_vectors(const host_vector<T>& actual, const host_vector<T>& expected) -> vector_compare_result_t<T>
+{
+  return compare_host_ranges(actual, expected);
+}
+
+template <typename T>
+auto compare_vectors(const device_vector<T>& actual, const device_vector<T>& expected) -> vector_compare_result_t<T>
+{
+  return compare_vectors<T>(host_vector<T>(actual), host_vector<T>(expected));
+}
+
+template <typename T, typename... LhsProps, typename... RhsProps>
+auto compare_vectors(const cuda::buffer<T, LhsProps...>& actual, const cuda::buffer<T, RhsProps...>& expected)
+  -> vector_compare_result_t<T>
+{
+  const auto actual_host   = cuda::make_buffer(actual.stream(), cuda::mr::legacy_pinned_memory_resource{}, actual);
+  const auto expected_host = cuda::make_buffer(expected.stream(), cuda::mr::legacy_pinned_memory_resource{}, expected);
+
+  actual.stream().sync();
+  expected.stream().sync();
+  return compare_host_ranges(actual_host, expected_host);
+}
+
+template <typename LhsVec, typename RhsVec, typename T = typename LhsVec::value_type>
+auto compare_vectors(const LhsVec& actual, const RhsVec& expected) -> vector_compare_result_t<T>
+{
+  return compare_vectors<T>(host_vector<T>(actual), host_vector<T>(expected));
+}
+
+template <typename T>
+void print_comparison(const vector_compare_result_t<T>& res, std::ostream& os)
+{
+  if (res.actual_size != res.expected_size)
+  {
+    os << "Actual size (" << res.actual_size << ") != expected size (" << res.expected_size << ")\n";
+    return;
+  }
+
+  const auto mismatch_percent = (static_cast<double>(res.total_mismatches) / res.actual_size) * 100.0;
+  os << res.total_mismatches << " mismatch" << (res.total_mismatches > 1 ? "es" : "") << " (" << std::fixed
+     << std::setprecision(2) << mismatch_percent << "% of " << res.expected_size << " elements)\n";
+
+  // print good values
+  for (const auto& [idx, v] : res.good_values)
+  {
+    os << "good [" << idx << "]: " << CoutCast(v) << " == " << CoutCast(v) << '\n';
+  }
+
+  // insert dots between mismatches that are not consecutive
+  size_t last_printed_idx = res.good_values.empty() ? 0 : res.good_values.back().index;
+  auto print_dots         = [&](size_t idx) {
+    if (last_printed_idx + 1 != idx)
+    {
+      os << "...\n";
+    }
+    last_printed_idx = idx;
+  };
+
+  // print first mismatches
+  for (const auto& [idx, a, b] : res.first_mismatches)
+  {
+    print_dots(idx);
+    os << "BAD  [" << idx << "]: " << CoutCast(a) << " != " << CoutCast(b) << '\n';
+  }
+
+  // print last mismatches if we have any
+  if (res.last_mismatches)
+  {
+    for (const auto& [idx, a, b] : *res.last_mismatches)
+    {
+      print_dots(idx);
+      os << "BAD  [" << idx << "]: " << CoutCast(a) << " != " << CoutCast(b) << '\n';
+    }
+  }
+}
+
+template <typename Vec>
+struct vector_matcher : Catch::Matchers::MatcherGenericBase
+{
+  vector_matcher(Vec const& expected)
+      : expected_vec{expected}
+  {}
+
+  template <typename OtherVec>
+  bool match(OtherVec const& actual_vec) const // TODO(Bgruber): remove const?
+  {
+    comparison_result = compare_vectors(actual_vec, expected_vec);
+    return comparison_result.total_mismatches == 0;
+  }
+
+  std::string describe() const override
+  {
+    std::stringstream ss;
+    print_comparison(comparison_result, ss);
+    return ss.str();
+  }
+
+private:
+  mutable vector_compare_result_t<typename Vec::value_type> comparison_result;
+  Vec const& expected_vec;
+};
+} // namespace c2h::detail
+
+//! Compare thrust vectors in a match expression. Example: CHECK_THAT_QUIET(vec_a, Equals(vec_v))
+template <typename T, typename Alloc>
+auto Equals(const THRUST_NS_QUALIFIER::detail::vector_base<T, Alloc>& expected)
+  -> c2h::detail::vector_matcher<THRUST_NS_QUALIFIER::detail::vector_base<T, Alloc>>
+{
+  return {expected};
+}
+
+template <typename T, typename... Props>
+auto Equals(const cuda::buffer<T, Props...>& expected) -> c2h::detail::vector_matcher<cuda::buffer<T, Props...>>
+{
+  return {expected};
+}
+
+#include <cuda/std/tuple>
+#include <cuda/std/utility>
+
+_CCCL_BEGIN_NAMESPACE_CUDA_STD
+template <typename T1,
+          typename T2,
+          // provide this operator only when the pair's content is also streamable
+          ::cuda::std::void_t<decltype(::cuda::std::declval<::std::ostream>()
+                                       << ::cuda::std::declval<T1>() << ::cuda::std::declval<T2>())>* = nullptr>
+::std::ostream& operator<<(::std::ostream& os, const pair<T1, T2>& pair)
+{
+  return os << "[" << pair.first << ", " << pair.second << "]";
+}
+
+template <size_t N, typename... T>
+enable_if_t<(N == sizeof...(T))> print_elem(::std::ostream&, const tuple<T...>&)
+{}
+
+template <size_t N, typename... T>
+enable_if_t<(N < sizeof...(T))> print_elem(::std::ostream& os, const tuple<T...>& tup)
+{
+  if constexpr (N != 0)
+  {
+    os << ", ";
+  }
+  os << ::cuda::std::get<N>(tup);
+  ::cuda::std::print_elem<N + 1>(os, tup);
+}
+
+template <typename... T>
+::std::ostream& operator<<(::std::ostream& os, const tuple<T...>& tup)
+{
+  os << "[";
+  ::cuda::std::print_elem<0>(os, tup);
+  return os << "]";
+}
+_CCCL_END_NAMESPACE_CUDA_STD
+
+_CCCL_BEGIN_NAMESPACE_CUDA
+template <typename T, typename... Props>
+::std::ostream& operator<<(::std::ostream& os, const cuda::buffer<T, Props...>& buffer)
+{
+  const auto host_buf = cuda::make_buffer(buffer.stream(), cuda::mr::legacy_pinned_memory_resource{}, buffer);
+
+  buffer.stream().sync();
+  os << ::Catch::Detail::stringify(::std::vector<T>{host_buf.begin(), host_buf.end()});
+  return os;
+}
+_CCCL_END_NAMESPACE_CUDA
+
+template <>
+struct Catch::StringMaker<cudaError>
+{
+  static auto convert(cudaError e) -> std::string
+  {
+    return std::to_string(cuda::std::to_underlying(e)) + " (" + cudaGetErrorString(e) + ")";
+  }
+};
+
+#include <c2h/custom_type.h>
+#include <c2h/generators.h>
+
+namespace detail
+{
+struct nvtx_c2h_domain
+{
+  static constexpr const char* name = "C2H";
+};
+
+template <typename T>
+class nvtx_fixture
+{
+#if _CCCL_HAS_NVTX3()
+  ::nvtx3::v1::scoped_range_in<nvtx_c2h_domain> nvtx_range{Catch::getResultCapture().getCurrentTestName()};
+#endif // _CCCL_HAS_NVTX3()
+};
+} // namespace detail
+
+#define C2H_TEST_NAME_IMPL(NAME, PARAM) C2H_TEST_STR(NAME) "(" C2H_TEST_STR(PARAM) ")"
+
+#define C2H_TEST_NAME(NAME) C2H_TEST_NAME_IMPL(NAME, VAR_IDX)
+
+#define C2H_TEST_CONCAT(A, B)       C2H_TEST_CONCAT_INNER(A, B)
+#define C2H_TEST_CONCAT_INNER(A, B) A##B
+
+#define C2H_TEST_IMPL(ID, NAME, TAG, ...)                                  \
+  using C2H_TEST_CONCAT(types_, ID) = c2h::cartesian_product<__VA_ARGS__>; \
+  CATCH_TEMPLATE_LIST_TEST_CASE_METHOD(::detail::nvtx_fixture, C2H_TEST_NAME(NAME), TAG, C2H_TEST_CONCAT(types_, ID))
+
+#define C2H_TEST(NAME, TAG, ...) C2H_TEST_IMPL(__LINE__, NAME, TAG, __VA_ARGS__)
+
+#define C2H_TEST_WITH_FIXTURE_IMPL(ID, FIXTURE, NAME, TAG, ...)            \
+  using C2H_TEST_CONCAT(types_, ID) = c2h::cartesian_product<__VA_ARGS__>; \
+  CATCH_TEMPLATE_LIST_TEST_CASE_METHOD(FIXTURE, C2H_TEST_NAME(NAME), TAG, C2H_TEST_CONCAT(types_, ID))
+
+#define C2H_TEST_WITH_FIXTURE(FIXTURE, NAME, TAG, ...) \
+  C2H_TEST_WITH_FIXTURE_IMPL(__LINE__, FIXTURE, NAME, TAG, __VA_ARGS__)
+
+#define C2H_TEST_LIST_IMPL(ID, NAME, TAG, ...)                     \
+  using C2H_TEST_CONCAT(types_, ID) = c2h::type_list<__VA_ARGS__>; \
+  CATCH_TEMPLATE_LIST_TEST_CASE_METHOD(::detail::nvtx_fixture, C2H_TEST_NAME(NAME), TAG, C2H_TEST_CONCAT(types_, ID))
+
+#define C2H_TEST_LIST(NAME, TAG, ...) C2H_TEST_LIST_IMPL(__LINE__, NAME, TAG, __VA_ARGS__)
+
+#define C2H_TEST_LIST_WITH_FIXTURE_IMPL(ID, FIXTURE, NAME, TAG, ...) \
+  using C2H_TEST_CONCAT(types_, ID) = c2h::type_list<__VA_ARGS__>;   \
+  CATCH_TEMPLATE_LIST_TEST_CASE_METHOD(FIXTURE, C2H_TEST_NAME(NAME), TAG, C2H_TEST_CONCAT(types_, ID))
+
+#define C2H_TEST_LIST_WITH_FIXTURE(FIXTURE, NAME, TAG, ...) \
+  C2H_TEST_LIST_WITH_FIXTURE_IMPL(__LINE__, FIXTURE, NAME, TAG, __VA_ARGS__)
+
+#define C2H_TEST_STR(a) #a
+
+namespace c2h
+{
+inline std::size_t get_override_seed_count()
+{
+  // Setting this environment variable forces a fixed number of seeds to be generated, regardless of the requested
+  // count. Set to 1 to reduce redundant, expensive testing when using sanitizers, etc.
+  static std::optional<std::string> override_str = c2h::detail::get_env("C2H_SEED_COUNT_OVERRIDE");
+  static const int override_seeds                = override_str ? std::atoi(override_str->c_str()) : 0;
+  return override_seeds;
+}
+
+inline std::size_t adjust_seed_count(std::size_t requested)
+{
+  static std::size_t override_seeds = get_override_seed_count();
+  return override_seeds != 0 ? override_seeds : requested;
+}
+} // namespace c2h
+
+#define C2H_SEED(N)                                                                         \
+  c2h::seed_t                                                                               \
+  {                                                                                         \
+    GENERATE_COPY(take(c2h::adjust_seed_count(N),                                           \
+                       random(::cuda::std::numeric_limits<unsigned long long int>::min(),   \
+                              ::cuda::std::numeric_limits<unsigned long long int>::max()))) \
+  }

cccl_upstream/c2h/include/c2h/catch2_test_macros.h

@@ -0,0 +1,200 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+#pragma once
+
+#include <cuda/std/detail/__config>
+
+#include <nv/target>
+
+#include <catch2/catch_message.hpp>
+#include <catch2/catch_template_test_macros.hpp>
+#include <catch2/catch_test_macros.hpp>
+#include <catch2/matchers/catch_matchers.hpp>
+
+// This file implements Catch2's test macros that work both in host and device code. We globally define the
+// CATCH_CONFIG_PREFIX_ALL macro to force Catch2 to prepend it's macros with CATCH_ prefix. That allows us to implement
+// the non-prefixed versions ourselves.
+//
+// In host code, we just use the CATCH_-prefixed variant, in device code we implement the functionality, so it
+// corresponds the desired functionality.
+//
+// Only a subset of the Catch2's macro are provided. If needed, feel free to extend the support. Host-only macros can
+// be determined by missing NV_IF_ELSE_TARGET wrapper and immediate dispatch to CATCH_-prefixed variant.
+
+// workaround for error #3185-D: no '#pragma diagnostic push' was found to match this 'diagnostic pop'
+#if _CCCL_COMPILER(NVHPC)
+#  undef CATCH_INTERNAL_START_WARNINGS_SUPPRESSION
+#  undef CATCH_INTERNAL_STOP_WARNINGS_SUPPRESSION
+#  define CATCH_INTERNAL_START_WARNINGS_SUPPRESSION _Pragma("diag push")
+#  define CATCH_INTERNAL_STOP_WARNINGS_SUPPRESSION  _Pragma("diag pop")
+#endif
+// The nv_diagnostic pragmas in Catch2 macros cause cicc to hang indefinitely in CTK 13.0.
+// See NVBugs 5475335.
+#if _CCCL_VERSION_COMPARE(_CCCL_CTK_, _CCCL_CTK, ==, 13, 0)
+#  undef CATCH_INTERNAL_START_WARNINGS_SUPPRESSION
+#  undef CATCH_INTERNAL_STOP_WARNINGS_SUPPRESSION
+#  define CATCH_INTERNAL_START_WARNINGS_SUPPRESSION
+#  define CATCH_INTERNAL_STOP_WARNINGS_SUPPRESSION
+#endif
+// workaround for error
+// * MSVC14.39: #3185-D: no '#pragma diagnostic push' was found to match this 'diagnostic pop'
+// * MSVC14.29: internal error: assertion failed: alloc_copy_of_pending_pragma: copied pragma has source sequence entry
+//              (pragma.c, line 526 in alloc_copy_of_pending_pragma)
+// see also upstream Catch2 issue: https://github.com/catchorg/Catch2/issues/2636
+#if _CCCL_COMPILER(MSVC)
+#  undef CATCH_INTERNAL_START_WARNINGS_SUPPRESSION
+#  undef CATCH_INTERNAL_STOP_WARNINGS_SUPPRESSION
+#  undef CATCH_INTERNAL_SUPPRESS_UNUSED_VARIABLE_WARNINGS
+#  define CATCH_INTERNAL_START_WARNINGS_SUPPRESSION
+#  define CATCH_INTERNAL_STOP_WARNINGS_SUPPRESSION
+#  define CATCH_INTERNAL_SUPPRESS_UNUSED_VARIABLE_WARNINGS
+#endif
+
+// We must pass the COND as a cstring parameter, because it might contain the '%' character that would break the printf
+// formatting.
+#define C2H_INTERNAL_DEVICE_TEST_PRINT(KIND, COND)                                                               \
+  ::printf(                                                                                                      \
+    __FILE__                                                                                                     \
+    ":" _CCCL_TO_STRING(__LINE__) ":\n    " KIND "(%s) failed\n    block [%u, %u, %u], thread [%u, %u, %u]\n\n", \
+    COND,                                                                                                        \
+    blockIdx.x,                                                                                                  \
+    blockIdx.y,                                                                                                  \
+    blockIdx.z,                                                                                                  \
+    threadIdx.x,                                                                                                 \
+    threadIdx.y,                                                                                                 \
+    threadIdx.z)
+
+// <catch2/catch2_test_macros.hpp>
+
+#define REQUIRE(...)                                                             \
+  NV_IF_ELSE_TARGET(NV_IS_HOST, (CATCH_REQUIRE(__VA_ARGS__);), ({                \
+                      if (!(__VA_ARGS__))                                        \
+                      {                                                          \
+                        C2H_INTERNAL_DEVICE_TEST_PRINT("REQUIRE", #__VA_ARGS__); \
+                        ::__trap();                                              \
+                      }                                                          \
+                    }))
+#define REQUIRE_FALSE(...)                                                             \
+  NV_IF_ELSE_TARGET(NV_IS_HOST, (CATCH_REQUIRE_FALSE(__VA_ARGS__);), ({                \
+                      if (__VA_ARGS__)                                                 \
+                      {                                                                \
+                        C2H_INTERNAL_DEVICE_TEST_PRINT("REQUIRE_FALSE", #__VA_ARGS__); \
+                        ::__trap();                                                    \
+                      }                                                                \
+                    }))
+
+#define REQUIRE_THROWS(...)    CATCH_REQUIRE_THROWS(__VA_ARGS__)
+#define REQUIRE_THROWS_AS(...) CATCH_REQUIRE_THROWS_AS(__VA_ARGS__)
+#define REQUIRE_NOTHROW(...)   NV_IF_ELSE_TARGET(NV_IS_HOST, (CATCH_REQUIRE_NOTHROW(__VA_ARGS__);), (__VA_ARGS__;))
+
+#define CHECK(...)                                                             \
+  NV_IF_ELSE_TARGET(NV_IS_HOST, (CATCH_CHECK(__VA_ARGS__);), ({                \
+                      if (!(__VA_ARGS__))                                      \
+                      {                                                        \
+                        C2H_INTERNAL_DEVICE_TEST_PRINT("CHECK", #__VA_ARGS__); \
+                        ::__trap();                                            \
+                      }                                                        \
+                    }))
+#define CHECK_FALSE(...)                                                             \
+  NV_IF_ELSE_TARGET(NV_IS_HOST, (CATCH_CHECK_FALSE(__VA_ARGS__);), ({                \
+                      if (__VA_ARGS__)                                               \
+                      {                                                              \
+                        C2H_INTERNAL_DEVICE_TEST_PRINT("CHECK_FALSE", #__VA_ARGS__); \
+                        ::__trap();                                                  \
+                      }                                                              \
+                    }))
+#define CHECKED_IF(...)   CATCH_CHECKED_IF(__VA_ARGS__)
+#define CHECKED_ELSE(...) CATCH_CHECKED_ELSE(__VA_ARGS__)
+#define CHECK_NOFAIL(...) CATCH_CHECK_NOFAIL(__VA_ARGS__)
+
+#define CHECK_THROWS(...)    CATCH_CHECK_THROWS(__VA_ARGS__)
+#define CHECK_THROWS_AS(...) CATCH_CHECK_THROWS_AS(__VA_ARGS__)
+#define CHECK_NOTHROW(...)   NV_IF_ELSE_TARGET(NV_IS_HOST, (CATCH_CHECK_NOTHROW(__VA_ARGS__);), (__VA_ARGS__;))
+
+#define TEST_CASE(...)           CATCH_TEST_CASE(__VA_ARGS__)
+#define TEST_CASE_METHOD(...)    CATCH_TEST_CASE_METHOD(__VA_ARGS__)
+#define METHOD_AS_TEST_CASE(...) CATCH_METHOD_AS_TEST_CASE(__VA_ARGS__)
+#define REGISTER_TEST_CASE(...)  CATCH_REGISTER_TEST_CASE(__VA_ARGS__)
+#define SECTION(...)             CATCH_SECTION(__VA_ARGS__)
+#define DYNAMIC_SECTION(...)     CATCH_DYNAMIC_SECTION(__VA_ARGS__)
+#define FAIL(...)                                                           \
+  NV_IF_ELSE_TARGET(NV_IS_HOST, (CATCH_FAIL(__VA_ARGS__);), ({              \
+                      C2H_INTERNAL_DEVICE_TEST_PRINT("FAIL", #__VA_ARGS__); \
+                      ::__trap();                                           \
+                    }))
+#define FAIL_CHECK(...) CATCH_FAIL_CHECK(__VA_ARGS__)
+#define SUCCEED(...)    CATCH_SUCCEED(__VA_ARGS__)
+#define SKIP(...)       CATCH_SKIP(__VA_ARGS__)
+
+#define STATIC_REQUIRE(...) \
+  NV_IF_ELSE_TARGET(NV_IS_HOST, (CATCH_STATIC_REQUIRE(__VA_ARGS__);), (static_assert(__VA_ARGS__, #__VA_ARGS__);))
+#define STATIC_REQUIRE_FALSE(...) \
+  NV_IF_ELSE_TARGET(              \
+    NV_IS_HOST, (CATCH_STATIC_REQUIRE_FALSE(__VA_ARGS__);), (static_assert(!(__VA_ARGS__), "!(" #__VA_ARGS__ ")");))
+#define STATIC_CHECK(...) \
+  NV_IF_ELSE_TARGET(NV_IS_HOST, (CATCH_STATIC_CHECK(__VA_ARGS__);), (static_assert(__VA_ARGS__, #__VA_ARGS__);))
+#define STATIC_CHECK_FALSE(...) \
+  NV_IF_ELSE_TARGET(            \
+    NV_IS_HOST, (CATCH_STATIC_CHECK_FALSE(__VA_ARGS__);), (static_assert(!(__VA_ARGS__), "!(" #__VA_ARGS__ ")");))
+
+#define SCENARIO(...)        CATCH_SCENARIO(__VA_ARGS__)
+#define SCENARIO_METHOD(...) CATCH_SCENARIO_METHOD(__VA_ARGS__)
+#define GIVEN(...)           CATCH_GIVEN(__VA_ARGS__)
+#define AND_GIVEN(...)       CATCH_AND_GIVEN(__VA_ARGS__)
+#define WHEN(...)            CATCH_WHEN(__VA_ARGS__)
+#define AND_WHEN(...)        CATCH_AND_WHEN(__VA_ARGS__)
+#define THEN(...)            CATCH_THEN(__VA_ARGS__)
+#define AND_THEN(...)        CATCH_AND_THEN(__VA_ARGS__)
+
+// <catch2/catch_message.hpp>
+
+#define INFO(...)          CATCH_INFO(__VA_ARGS__)
+#define UNSCOPED_INFO(...) CATCH_UNSCOPED_INFO(__VA_ARGS__)
+#define WARN(...)          CATCH_WARN(__VA_ARGS__)
+#define CAPTURE(...)       CATCH_CAPTURE(__VA_ARGS__)
+
+// <catch2/catch_template_test_macros.hpp>
+
+#define TEMPLATE_TEST_CASE(...)                    CATCH_TEMPLATE_TEST_CASE(__VA_ARGS__)
+#define TEMPLATE_TEST_CASE_SIG(...)                CATCH_TEMPLATE_TEST_CASE_SIG(__VA_ARGS__)
+#define TEMPLATE_TEST_CASE_METHOD(...)             CATCH_TEMPLATE_TEST_CASE_METHOD(__VA_ARGS__)
+#define TEMPLATE_TEST_CASE_METHOD_SIG(...)         CATCH_TEMPLATE_TEST_CASE_METHOD_SIG(__VA_ARGS__)
+#define TEMPLATE_PRODUCT_TEST_CASE(...)            CATCH_TEMPLATE_PRODUCT_TEST_CASE(__VA_ARGS__)
+#define TEMPLATE_PRODUCT_TEST_CASE_SIG(...)        CATCH_TEMPLATE_PRODUCT_TEST_CASE_SIG(__VA_ARGS__)
+#define TEMPLATE_PRODUCT_TEST_CASE_METHOD(...)     CATCH_TEMPLATE_PRODUCT_TEST_CASE_METHOD(__VA_ARGS__)
+#define TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG(...) CATCH_TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG(__VA_ARGS__)
+#define TEMPLATE_LIST_TEST_CASE(...)               CATCH_TEMPLATE_LIST_TEST_CASE(__VA_ARGS__)
+#define TEMPLATE_LIST_TEST_CASE_METHOD(...)        CATCH_TEMPLATE_LIST_TEST_CASE_METHOD(__VA_ARGS__)
+
+// <catch2/matchers/catch_matchers.hpp>
+
+#define REQUIRE_THROWS_WITH(...)    CATCH_REQUIRE_THROWS_WITH(__VA_ARGS__)
+#define REQUIRE_THROWS_MATCHES(...) CATCH_REQUIRE_THROWS_MATCHES(__VA_ARGS__)
+#define CHECK_THROWS_WITH(...)      CATCH_CHECK_THROWS_WITH(__VA_ARGS__)
+#define CHECK_THROWS_MATCHES(...)   CATCH_CHECK_THROWS_MATCHES(__VA_ARGS__)
+#define CHECK_THAT(...)             CATCH_CHECK_THAT(__VA_ARGS__)
+#define REQUIRE_THAT(...)           CATCH_REQUIRE_THAT(__VA_ARGS__)
+
+// extensions
+
+// Sometimes clang-cuda has problems with REQUIRE(...) when used in __device__ function - it tries to instantiate the
+// host path. This is related to clang-cuda's compilation trajectory. For these cases, we provide REQUIRE_DEVICE(...) as
+// a fallback.
+#define REQUIRE_DEVICE(...)                                    \
+  do                                                           \
+  {                                                            \
+    if (!(__VA_ARGS__))                                        \
+    {                                                          \
+      C2H_INTERNAL_DEVICE_TEST_PRINT("REQUIRE", #__VA_ARGS__); \
+      ::__trap();                                              \
+    }                                                          \
+  } while (false)
+
+// Macros to require/check success of a CUDA Driver call.
+#define REQUIRE_CUDA(...) REQUIRE((__VA_ARGS__) == CUDA_SUCCESS)
+#define CHECK_CUDA(...)   CHECK((__VA_ARGS__) == CUDA_SUCCESS)
+
+// Macros to require/check success of a CUDA Runtime call.
+#define REQUIRE_CUDART(...) REQUIRE((__VA_ARGS__) == cudaSuccess)
+#define CHECK_CUDART(...)   CHECK((__VA_ARGS__) == cudaSuccess)

cccl_upstream/c2h/include/c2h/check_results.cuh

@@ -0,0 +1,154 @@
+// SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3-Clause
+
+#pragma once
+
+#include <cub/detail/type_traits.cuh>
+#include <cub/util_device.cuh>
+
+#include <cuda/std/complex>
+#include <cuda/std/type_traits>
+
+#include <test_util.h>
+
+#include <catch2/matchers/catch_matchers_floating_point.hpp>
+
+template <typename T>
+void verify_results(const c2h::host_vector<T>& expected_data, const c2h::host_vector<T>& test_results)
+{
+  using namespace cub::detail;
+  int device_id = 0;
+  CubDebugExit(cudaGetDevice(&device_id));
+  int ptx_version = 0;
+  CubDebugExit(CUB_NS_QUALIFIER::PtxVersion(ptx_version, device_id));
+  if (ptx_version < 80 && is_any_bfloat16_v<T>)
+  {
+    return;
+  }
+  if (ptx_version < 53 && is_any_half_v<T>)
+  {
+    return;
+  }
+  if constexpr (cuda::std::is_floating_point_v<T>)
+  {
+    REQUIRE_APPROX_EQ(expected_data, test_results);
+  }
+  else if constexpr (cuda::std::is_same_v<T, __nv_bfloat16> || cuda::std::is_same_v<T, __half>)
+  {
+    constexpr auto rel_err = cuda::std::is_same_v<T, __half> ? 0.08f : 0.2f;
+    REQUIRE_APPROX_EQ_EPSILON(expected_data, test_results, rel_err);
+  }
+  else if constexpr (cuda::std::is_same_v<T, float2>)
+  {
+    for (size_t i = 0; i < test_results.size(); ++i)
+    {
+      REQUIRE_THAT(expected_data[i].x, Catch::Matchers::WithinRel(test_results[i].x, 0.01f));
+      REQUIRE_THAT(expected_data[i].y, Catch::Matchers::WithinRel(test_results[i].y, 0.01f));
+    }
+  }
+  else if constexpr (cuda::std::is_same_v<T, __nv_bfloat162> || cuda::std::is_same_v<T, __half2>)
+  {
+    constexpr auto rel_err = cuda::std::is_same_v<T, __half2> ? 0.08f : 0.2f;
+    for (size_t i = 0; i < test_results.size(); ++i)
+    {
+      REQUIRE_THAT(expected_data[i].x, Catch::Matchers::WithinRel(test_results[i].x, rel_err));
+      REQUIRE_THAT(expected_data[i].y, Catch::Matchers::WithinRel(test_results[i].y, rel_err));
+    }
+  }
+  else if constexpr (cuda::std::is_same_v<T, cuda::std::complex<__nv_bfloat16>>
+                     || cuda::std::is_same_v<T, cuda::std::complex<__half>>)
+  {
+    constexpr auto rel_err = cuda::std::is_same_v<T, cuda::std::complex<__half>> ? 0.08f : 0.2f;
+    for (size_t i = 0; i < test_results.size(); ++i)
+    {
+      auto expected_real = static_cast<float>(expected_data[i].real());
+      auto test_real     = test_results[i].real();
+      auto expected_imag = static_cast<float>(expected_data[i].imag());
+      auto test_imag     = test_results[i].imag();
+      REQUIRE_THAT(expected_real, Catch::Matchers::WithinRel(test_real, rel_err));
+      REQUIRE_THAT(expected_imag, Catch::Matchers::WithinRel(test_imag, rel_err));
+    }
+  }
+  else if constexpr (cuda::std::__is_cuda_std_complex_v<T>)
+  {
+    for (size_t i = 0; i < test_results.size(); ++i)
+    {
+      auto expected_real = expected_data[i].real();
+      auto test_real     = test_results[i].real();
+      auto expected_imag = expected_data[i].imag();
+      auto test_imag     = test_results[i].imag();
+      REQUIRE_THAT(expected_real, Catch::Matchers::WithinRel(test_real));
+      REQUIRE_THAT(expected_imag, Catch::Matchers::WithinRel(test_imag));
+    }
+  }
+  else
+  {
+    REQUIRE(expected_data == test_results);
+  }
+}
+
+template <typename T>
+void verify_results(const c2h::host_vector<T>& expected_data, const c2h::device_vector<T>& test_results)
+{
+  c2h::host_vector<T> test_results_host = test_results;
+  verify_results(expected_data, test_results_host);
+}
+
+//----------------------------------------------------------------------------------------------------------------------
+// Min/Max comparison requires bitwise identical results (excluding NaN). Vector Types require only the first element to
+// match due to how it defined the operator<
+
+template <typename T>
+void verify_results_exact(const c2h::host_vector<T>& expected_data, const c2h::host_vector<T>& test_results)
+{
+  using namespace cub::detail;
+  int device_id                = 0;
+  int compute_capability_major = 0;
+  int compute_capability_minor = 0;
+  CubDebugExit(cudaGetDevice(&device_id));
+  CubDebugExit(cudaDeviceGetAttribute(&compute_capability_major, cudaDevAttrComputeCapabilityMajor, device_id));
+  CubDebugExit(cudaDeviceGetAttribute(&compute_capability_minor, cudaDevAttrComputeCapabilityMinor, device_id));
+  int compute_capability = 10 * compute_capability_major + compute_capability_minor;
+  if (compute_capability < 80 && is_any_bfloat16_v<T>)
+  {
+    return;
+  }
+  if (compute_capability < 53 && is_any_half_v<T>)
+  {
+    return;
+  }
+  if constexpr (is_vector2_fp_type_v<T>)
+  {
+    for (size_t i = 0; i < test_results.size(); ++i)
+    {
+      auto expected    = static_cast<float>(expected_data[i].x);
+      auto test_result = static_cast<float>(test_results[i].x);
+      REQUIRE(expected == test_result);
+    }
+  }
+  if constexpr (is_vector2_type_v<T>)
+  {
+    for (size_t i = 0; i < test_results.size(); ++i)
+    {
+      REQUIRE(expected_data[i].x == test_results[i].x);
+    }
+  }
+  else
+  {
+    REQUIRE_BITWISE_EQ(expected_data, test_results);
+  }
+}
+
+template <typename T>
+void verify_results_exact(const c2h::host_vector<T>& expected_data, const c2h::device_vector<T>& test_results)
+{
+  c2h::host_vector<T> test_results_host = test_results;
+  if constexpr (is_vector2_type_v<T> || cuda::is_floating_point_v<T>)
+  {
+    verify_results_exact(expected_data, test_results_host);
+  }
+  else
+  {
+    verify_results(expected_data, test_results_host);
+  }
+}

cccl_upstream/c2h/include/c2h/checked_allocator.cuh

@@ -0,0 +1,205 @@
+// SPDX-FileCopyrightText: Copyright (c) 2011-2024, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3
+
+#pragma once
+
+#include <thrust/device_allocator.h>
+#include <thrust/mr/new.h>
+#include <thrust/system/cuda/memory.h>
+#include <thrust/system/cuda/memory_resource.h>
+#include <thrust/system/cuda/pointer.h>
+
+#include <cstdlib>
+#include <iostream>
+#include <new>
+#include <optional>
+#include <string>
+
+#include <cuda_runtime_api.h>
+
+namespace c2h
+{
+namespace detail
+{
+inline std::optional<std::string> get_env(const char* name)
+{
+#ifdef _WIN32
+  char* buf       = nullptr;
+  std::size_t len = 0;
+  if (_dupenv_s(&buf, &len, name) || !buf)
+  {
+    return std::nullopt;
+  }
+  std::string val(buf);
+  free(buf);
+  return val;
+#else
+  if (const char* v = std::getenv(name))
+  {
+    return std::string(v);
+  }
+  return std::nullopt;
+#endif
+}
+
+struct memory_info
+{
+  std::size_t free{};
+  std::size_t total{};
+  bool override{false};
+};
+
+// If the environment variable C2H_DEVICE_MEMORY_LIMIT is set, the total device memory
+// will be limited to this number of bytes.
+inline std::size_t get_device_memory_limit()
+{
+  static std::optional<std::string> override_str = get_env("C2H_DEVICE_MEMORY_LIMIT");
+  static std::size_t result = override_str ? static_cast<std::size_t>(std::atoll(override_str->c_str())) : 0;
+  return result;
+}
+
+inline bool get_debug_checked_allocs()
+{
+  static std::optional<std::string> debug_checked_allocs = get_env("C2H_DEBUG_CHECKED_ALLOC_FAILURES");
+  static bool result = debug_checked_allocs && (std::atoi(debug_checked_allocs->c_str()) != 0);
+  return result;
+}
+
+inline cudaError_t get_device_memory(memory_info& info)
+{
+  static std::size_t device_memory_limit = get_device_memory_limit();
+
+  cudaError_t status = cudaMemGetInfo(&info.free, &info.total);
+  if (status != cudaSuccess)
+  {
+    return status;
+  }
+
+  if (device_memory_limit > 0)
+  {
+    info.free  = (std::max) (std::size_t{0}, static_cast<std::size_t>(info.free - (info.total - device_memory_limit)));
+    info.total = device_memory_limit;
+    info.override = true;
+  }
+
+  return cudaSuccess;
+}
+
+inline cudaError_t check_free_device_memory(std::size_t bytes)
+{
+  memory_info info;
+  cudaError_t status = get_device_memory(info);
+  if (status != cudaSuccess)
+  {
+    return status;
+  }
+
+  // Avoid allocating all available memory:
+  constexpr std::size_t padding = 16 * 1024 * 1024; // 16 MiB
+  if (info.free < (bytes + padding))
+  {
+    if (get_debug_checked_allocs())
+    {
+      const double total_GiB     = static_cast<double>(info.total) / (1024 * 1024 * 1024);
+      const double free_GiB      = static_cast<double>(info.free) / (1024 * 1024 * 1024);
+      const double requested_GiB = static_cast<double>(bytes) / (1024 * 1024 * 1024);
+      const double padded_GiB    = static_cast<double>(bytes + padding) / (1024 * 1024 * 1024);
+
+      std::cerr << "Device memory allocation failed due to insufficient free device memory.\n";
+
+      if (info.override)
+      {
+        std::cerr
+          << "Available device memory has been limited (env var C2H_DEVICE_MEMORY_LIMIT=" << get_device_memory_limit()
+          << ").\n";
+      }
+
+      std::cerr
+        << "Total device mem:     " << total_GiB << " GiB\n" //
+        << "Free device mem:      " << free_GiB << " GiB\n" //
+        << "Requested device mem: " << requested_GiB << " GiB\n" //
+        << "Padded device mem:    " << padded_GiB << " GiB\n";
+    }
+
+    return cudaErrorMemoryAllocation;
+  }
+
+  return cudaSuccess;
+}
+
+// Check available memory prior to calling cudaMalloc.
+// This avoids hangups and slowdowns from allocating swap / non-device memory
+// on some platforms, namely tegra.
+inline cudaError_t checked_cuda_malloc(void** ptr, std::size_t bytes)
+{
+  auto status = check_free_device_memory(bytes);
+  if (status != cudaSuccess)
+  {
+    return status;
+  }
+
+  return cudaMalloc(ptr, bytes);
+}
+} // namespace detail
+
+using checked_cuda_memory_resource = THRUST_NS_QUALIFIER::system::cuda::detail::
+  cuda_memory_resource<detail::checked_cuda_malloc, cudaFree, THRUST_NS_QUALIFIER::cuda::pointer<void>>;
+
+template <typename T>
+class checked_cuda_allocator
+    : public THRUST_NS_QUALIFIER::mr::
+        stateless_resource_allocator<T, THRUST_NS_QUALIFIER::device_ptr_memory_resource<checked_cuda_memory_resource>>
+{
+  using base = THRUST_NS_QUALIFIER::mr::
+    stateless_resource_allocator<T, THRUST_NS_QUALIFIER::device_ptr_memory_resource<checked_cuda_memory_resource>>;
+
+public:
+  template <typename U>
+  struct rebind
+  {
+    using other = checked_cuda_allocator<U>;
+  };
+
+  checked_cuda_allocator() = default;
+
+  _CCCL_HOST_DEVICE checked_cuda_allocator(const checked_cuda_allocator& other)
+      : base(other)
+  {}
+
+  template <typename U>
+  _CCCL_HOST_DEVICE checked_cuda_allocator(const checked_cuda_allocator<U>& other)
+      : base(other)
+  {}
+
+  checked_cuda_allocator& operator=(const checked_cuda_allocator&) = default;
+
+  ~checked_cuda_allocator() = default;
+};
+
+struct checked_host_memory_resource final : public THRUST_NS_QUALIFIER::mr::new_delete_resource_base
+{
+  void* do_allocate(std::size_t bytes, std::size_t alignment = THRUST_MR_DEFAULT_ALIGNMENT) final
+  {
+    // Some systems with integrated host/device memory have issues with allocating more memory
+    // than is available. Check the amount of free memory before attempting to allocate on
+    // integrated systems.
+    int device = 0;
+    CubDebugExit(cudaGetDevice(&device));
+    cudaDeviceProp prop;
+    CubDebugExit(cudaGetDeviceProperties(&prop, device));
+    if (prop.integrated)
+    {
+      auto status = detail::check_free_device_memory(bytes + alignment + sizeof(std::size_t));
+      if (status != cudaSuccess)
+      {
+        throw std::bad_alloc{};
+      }
+    }
+
+    return this->new_delete_resource_base::do_allocate(bytes, alignment);
+  }
+};
+
+template <typename T>
+using checked_host_allocator = THRUST_NS_QUALIFIER::mr::stateless_resource_allocator<T, checked_host_memory_resource>;
+} // namespace c2h

cccl_upstream/c2h/include/c2h/cpu_timer.h

@@ -0,0 +1,83 @@
+// SPDX-FileCopyrightText: Copyright (c) 2011-2024, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3
+
+#pragma once
+
+#include <cuda/std/tuple>
+
+#include <chrono>
+#include <iostream>
+#include <string>
+
+// #define C2H_DEBUG_TIMING
+
+#ifdef C2H_DEBUG_TIMING
+#  define C2H_TIME_SECTION_INIT()  [[maybe_unused]] c2h::cpu_timer _c2h_timer_
+#  define C2H_TIME_SECTION_RESET() _c2h_timer_.reset()
+#  define C2H_TIME_SECTION(label)  _c2h_timer_.print_elapsed_seconds_and_reset(label)
+#  define C2H_TIME_SCOPE(label)    [[maybe_unused]] c2h::scoped_cpu_timer _c2h_scoped_cpu_timer_(label)
+#else
+#  define C2H_TIME_SECTION_INIT()  /* no-op */ []() {}()
+#  define C2H_TIME_SECTION_RESET() /* no-op */ []() {}()
+#  define C2H_TIME_SECTION(label)  /* no-op */ []() {}()
+#  define C2H_TIME_SCOPE(label)    /* no-op */ []() {}()
+#endif
+
+namespace c2h
+{
+class cpu_timer
+{
+  std::chrono::high_resolution_clock::time_point m_start;
+
+public:
+  cpu_timer()
+      : m_start(std::chrono::high_resolution_clock::now())
+  {}
+
+  void reset()
+  {
+    m_start = std::chrono::high_resolution_clock::now();
+  }
+
+  int elapsed_ms() const
+  {
+    auto duration = std::chrono::high_resolution_clock::now() - m_start;
+    auto ms       = std::chrono::duration_cast<std::chrono::milliseconds>(duration);
+    return static_cast<int>(ms.count());
+  }
+
+  std::uint64_t elapsed_us() const
+  {
+    auto duration = std::chrono::high_resolution_clock::now() - m_start;
+    auto us       = std::chrono::duration_cast<std::chrono::microseconds>(duration);
+    return static_cast<std::uint64_t>(us.count());
+  }
+
+  void print_elapsed_seconds(const std::string& label)
+  {
+    printf("%0.6f s: %s\n", static_cast<float>(this->elapsed_us()) / 1000000.f, label.c_str());
+  }
+
+  void print_elapsed_seconds_and_reset(const std::string& label)
+  {
+    this->print_elapsed_seconds(label);
+    this->reset();
+  }
+};
+
+class scoped_cpu_timer
+{
+  cpu_timer m_timer;
+  std::string m_label;
+
+public:
+  explicit scoped_cpu_timer(std::string label)
+      : m_label(std::move(label))
+  {}
+
+  ~scoped_cpu_timer()
+  {
+    m_timer.print_elapsed_seconds(m_label);
+  }
+};
+} // namespace c2h

cccl_upstream/c2h/include/c2h/custom_type.h

@@ -0,0 +1,191 @@
+// SPDX-FileCopyrightText: Copyright (c) 2011-2022, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3
+
+#pragma once
+
+#include <cuda/std/limits>
+
+#include <memory>
+#include <ostream>
+
+namespace c2h
+{
+struct custom_type_state_t
+{
+  std::size_t key{};
+  std::size_t val{};
+};
+
+template <template <typename> class... Policies>
+class custom_type_t
+    : public custom_type_state_t
+    , public Policies<custom_type_t<Policies...>>...
+{
+public:
+  friend __host__ std::ostream& operator<<(std::ostream& os, const custom_type_t& self)
+  {
+    return os << "{ " << self.key << ", " << self.val << " }";
+  }
+};
+
+template <std::size_t TotalSize>
+struct huge_data
+{
+  template <class CustomType>
+  class type
+  {
+    static constexpr auto extra_member_bytes = (TotalSize - sizeof(custom_type_state_t));
+    std::uint8_t data[extra_member_bytes];
+  };
+};
+
+template <class CustomType>
+class less_comparable_t
+{
+  // The CUDA compiler follows the IA64 ABI for class layout, while the
+  // Microsoft host compiler does not.
+  char workaround_msvc{};
+
+public:
+  friend __host__ __device__ bool operator<(const CustomType& lhs, const CustomType& rhs)
+  {
+    return lhs.key < rhs.key;
+  }
+};
+
+template <class CustomType>
+class greater_comparable_t
+{
+  // The CUDA compiler follows the IA64 ABI for class layout, while the
+  // Microsoft host compiler does not.
+  char workaround_msvc{};
+
+public:
+  friend __host__ __device__ bool operator>(const CustomType& lhs, const CustomType& rhs)
+  {
+    return lhs.key > rhs.key;
+  }
+};
+
+template <class CustomType>
+class lexicographical_less_comparable_t
+{
+  // The CUDA compiler follows the IA64 ABI for class layout, while the
+  // Microsoft host compiler does not.
+  char workaround_msvc{};
+
+public:
+  friend __host__ __device__ bool operator<(const CustomType& lhs, const CustomType& rhs)
+  {
+    return lhs.key == rhs.key ? lhs.val < rhs.val : lhs.key < rhs.key;
+  }
+};
+
+template <class CustomType>
+class lexicographical_greater_comparable_t
+{
+  // The CUDA compiler follows the IA64 ABI for class layout, while the
+  // Microsoft host compiler does not.
+  char workaround_msvc{};
+
+public:
+  friend __host__ __device__ bool operator>(const CustomType& lhs, const CustomType& rhs)
+  {
+    return lhs.key == rhs.key ? lhs.val > rhs.val : lhs.key > rhs.key;
+  }
+};
+
+template <class CustomType>
+class equal_comparable_t
+{
+  // The CUDA compiler follows the IA64 ABI for class layout, while the
+  // Microsoft host compiler does not.
+  char workaround_msvc{};
+
+public:
+  friend __host__ __device__ bool operator==(const CustomType& lhs, const CustomType& rhs)
+  {
+    return lhs.key == rhs.key && lhs.val == rhs.val;
+  }
+
+  friend __host__ __device__ bool operator!=(const CustomType& lhs, const CustomType& rhs)
+  {
+    return !(lhs == rhs);
+  }
+};
+
+template <class CustomType>
+class subtractable_t
+{
+  // The CUDA compiler follows the IA64 ABI for class layout, while the
+  // Microsoft host compiler does not.
+  char workaround_msvc{};
+
+public:
+  friend __host__ __device__ CustomType operator-(const CustomType& lhs, const CustomType& rhs)
+  {
+    CustomType result{};
+
+    result.key = lhs.key - rhs.key;
+    result.val = lhs.val - rhs.val;
+
+    return result;
+  }
+};
+
+template <class CustomType>
+class accumulateable_t
+{
+  // The CUDA compiler follows the IA64 ABI for class layout, while the
+  // Microsoft host compiler does not.
+  char workaround_msvc{};
+
+public:
+  friend __host__ __device__ CustomType operator+(const CustomType& lhs, const CustomType& rhs)
+  {
+    CustomType result{};
+
+    result.key = lhs.key + rhs.key;
+    result.val = lhs.val + rhs.val;
+
+    return result;
+  }
+};
+} // namespace c2h
+
+template <template <typename> class... Policies>
+class cuda::std::numeric_limits<c2h::custom_type_t<Policies...>>
+{
+public:
+  static constexpr bool is_specialized = true;
+
+  // template <class SizeT = size_t> is a workaround for cudafe++ < 13.1 + gcc < 13 replacing `numeric_limits<size_t>`
+  // with `numeric_limits<conditional<is_void_v<void>, __common_type2_imp<uint64_t, uint64_t>::type, void>::type>`
+
+  template <class SizeT = std::size_t>
+  static __host__ __device__ c2h::custom_type_t<Policies...> max()
+  {
+    c2h::custom_type_t<Policies...> val;
+    val.key = numeric_limits<SizeT>::max();
+    val.val = numeric_limits<SizeT>::max();
+    return val;
+  }
+
+  template <class SizeT = std::size_t>
+  static __host__ __device__ c2h::custom_type_t<Policies...> min()
+  {
+    c2h::custom_type_t<Policies...> val;
+    val.key = numeric_limits<SizeT>::min();
+    val.val = numeric_limits<SizeT>::min();
+    return val;
+  }
+
+  template <class SizeT = std::size_t>
+  static __host__ __device__ c2h::custom_type_t<Policies...> lowest()
+  {
+    c2h::custom_type_t<Policies...> val;
+    val.key = numeric_limits<SizeT>::lowest();
+    val.val = numeric_limits<SizeT>::lowest();
+    return val;
+  }
+};

cccl_upstream/c2h/include/c2h/detail/generators.cuh

@@ -0,0 +1,70 @@
+// SPDX-FileCopyrightText: Copyright (c) 2011-2022, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3-Clause
+
+#include <cuda/std/complex>
+
+#include <c2h/generators.h>
+
+namespace c2h::detail
+{
+// called once from main to set up the generator state
+void init_generator();
+
+// sets the seed and resizes the distribution vector, fills it, and returns a pointer the start of the data
+float* prepare_random_data(seed_t seed, std::size_t num_items);
+
+// called once before main returns to clean up the generator state
+void cleanup_generator();
+
+template <typename T, bool = ::cuda::is_floating_point_v<T>>
+struct random_to_item_t
+{
+  float m_min;
+  float m_max;
+
+  __host__ __device__ random_to_item_t(T min, T max)
+      : m_min(static_cast<float>(min))
+      , m_max(static_cast<float>(max))
+  {}
+
+  __device__ T operator()(float random_value)
+  {
+    return static_cast<T>((m_max - m_min) * random_value + m_min);
+  }
+};
+
+template <typename T>
+struct random_to_item_t<T, true>
+{
+  using storage_t = ::cuda::std::_If<(sizeof(T) > 4), double, float>;
+  storage_t m_min;
+  storage_t m_max;
+
+  __host__ __device__ random_to_item_t(T min, T max)
+      : m_min(static_cast<storage_t>(min))
+      , m_max(static_cast<storage_t>(max))
+  {}
+
+  __device__ T operator()(float random_value)
+  {
+    return static_cast<T>(m_max * random_value + m_min * (1.0f - random_value));
+  }
+};
+
+template <typename T>
+struct random_to_item_t<cuda::std::complex<T>, false>
+{
+  cuda::std::complex<T> m_min;
+  cuda::std::complex<T> m_max;
+
+  __host__ __device__ random_to_item_t(cuda::std::complex<T> min, cuda::std::complex<T> max)
+      : m_min(min)
+      , m_max(max)
+  {}
+
+  __device__ cuda::std::complex<T> operator()(float random_value) const
+  {
+    return (m_max - m_min) * cuda::std::complex<T>(random_value) + m_min;
+  }
+};
+} // namespace c2h::detail

cccl_upstream/c2h/include/c2h/device_policy.h

@@ -0,0 +1,19 @@
+// SPDX-FileCopyrightText: Copyright (c) 2011-2024, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3
+
+#pragma once
+
+#include <thrust/execution_policy.h>
+
+#include <c2h/checked_allocator.cuh>
+
+namespace c2h
+{
+#if THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+static const auto device_policy        = THRUST_NS_QUALIFIER::cuda::par(checked_cuda_allocator<char>{});
+static const auto nosync_device_policy = THRUST_NS_QUALIFIER::cuda::par_nosync(checked_cuda_allocator<char>{});
+#else // THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+static const auto device_policy        = THRUST_NS_QUALIFIER::device;
+static const auto nosync_device_policy = THRUST_NS_QUALIFIER::device;
+#endif // THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+} // namespace c2h

cccl_upstream/c2h/include/c2h/extended_types.h

@@ -0,0 +1,41 @@
+// SPDX-FileCopyrightText: Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3-Clause
+#pragma once
+
+#include <cuda/__cccl_config>
+
+#ifndef TEST_HALF_T
+#  if _CCCL_HAS_NVFP16()
+#    define TEST_HALF_T() 1
+#  else
+#    define TEST_HALF_T() 0
+#  endif
+#endif // TEST_HALF_T
+
+#ifndef TEST_BF_T
+#  if _CCCL_HAS_NVBF16()
+#    define TEST_BF_T() 1
+#  else
+#    define TEST_BF_T() 0
+#  endif
+#endif // TEST_BF_T
+
+#ifndef TEST_INT128
+#  if _CCCL_HAS_INT128() && !_CCCL_CUDA_COMPILER(CLANG) // clang-cuda crashes with int128 in generator.cu
+#    define TEST_INT128() 1
+#  else
+#    define TEST_INT128() 0
+#  endif
+#endif // TEST_INT128
+
+#if TEST_HALF_T()
+#  include <cuda_fp16.h>
+
+#  include <c2h/half.cuh>
+#endif // TEST_HALF_T()
+
+#if TEST_BF_T()
+#  include <cuda_bf16.h>
+
+#  include <c2h/bfloat16.cuh>
+#endif // TEST_BF_T()

cccl_upstream/c2h/include/c2h/fill_striped.h

@@ -0,0 +1,70 @@
+// SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3
+
+#pragma once
+
+#include <cuda/std/type_traits>
+
+template <typename VectorT, typename = void>
+struct scalar_to_vec_t
+{
+  template <typename T>
+  __host__ __device__ __forceinline__ auto operator()(T scalar) const -> VectorT
+  {
+    return static_cast<VectorT>(scalar);
+  }
+};
+
+template <typename VectorT>
+struct scalar_to_vec_t<VectorT, ::cuda::std::void_t<decltype(VectorT::x)>>
+{
+  template <typename T>
+  __host__ __device__ __forceinline__ auto operator()(T scalar) const -> VectorT
+  {
+    const auto c = static_cast<decltype(VectorT::x)>(scalar);
+    VectorT r;
+    constexpr auto components = ::cuda::std::tuple_size_v<VectorT>;
+    if constexpr (components >= 1)
+    {
+      r.x = c;
+    }
+    if constexpr (components >= 2)
+    {
+      r.y = c;
+    }
+    if constexpr (components >= 3)
+    {
+      r.z = c;
+    }
+    if constexpr (components >= 4)
+    {
+      r.w = c;
+    }
+    return r;
+  }
+};
+
+template <int LogicalWarpThreads, int ItemsPerThread, int ThreadsPerBlock, typename IteratorT>
+void fill_striped(IteratorT it)
+{
+  using T = cub::detail::it_value_t<IteratorT>;
+
+  constexpr int warps_in_block = ThreadsPerBlock / LogicalWarpThreads;
+  constexpr int items_per_warp = LogicalWarpThreads * ItemsPerThread;
+  scalar_to_vec_t<T> convert;
+
+  for (int warp_id = 0; warp_id < warps_in_block; warp_id++)
+  {
+    const int warp_offset_val = items_per_warp * warp_id;
+
+    for (int lane_id = 0; lane_id < LogicalWarpThreads; lane_id++)
+    {
+      const int lane_offset = warp_offset_val + lane_id;
+
+      for (int item = 0; item < ItemsPerThread; item++)
+      {
+        *(it++) = convert(lane_offset + item * LogicalWarpThreads);
+      }
+    }
+  }
+}

cccl_upstream/c2h/include/c2h/generators.h

@@ -0,0 +1,166 @@
+// SPDX-FileCopyrightText: Copyright (c) 2011-2022, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3
+
+#pragma once
+
+#include <thrust/detail/config/device_system.h>
+
+#include <cuda/std/limits>
+
+#include <c2h/custom_type.h>
+#include <c2h/vector.h>
+
+#if THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+#  if _CCCL_HAS_NVFP16()
+#    include <cuda_fp16.h>
+#  endif // _CCCL_HAS_NVFP16()
+
+#  if _CCCL_HAS_NVBF16()
+_CCCL_DIAG_PUSH
+_CCCL_DIAG_SUPPRESS_CLANG("-Wunused-function")
+#    include <cuda_bf16.h>
+_CCCL_DIAG_POP
+#  endif // _CCCL_HAS_NVBF16
+
+#  if _CCCL_HAS_NVFP8()
+// cuda_fp8.h resets default for C4127, so we have to guard the inclusion
+_CCCL_DIAG_PUSH
+#    include <cuda_fp8.h>
+_CCCL_DIAG_POP
+#  endif // _CCCL_HAS_NVFP8()
+#endif // THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+
+namespace c2h
+{
+namespace detail
+{
+template <class T>
+class value_wrapper_t
+{
+  T m_val{};
+
+public:
+  using value_type = T;
+
+  explicit value_wrapper_t(T val)
+      : m_val(val)
+  {}
+  explicit value_wrapper_t(int val)
+      : m_val(static_cast<T>(val))
+  {}
+  T get() const
+  {
+    return m_val;
+  }
+};
+} // namespace detail
+
+struct seed_t : detail::value_wrapper_t<unsigned long long int>
+{
+  using value_wrapper_t::value_wrapper_t;
+};
+
+struct modulo_t : detail::value_wrapper_t<std::size_t>
+{
+  using value_wrapper_t::value_wrapper_t;
+};
+
+namespace detail
+{
+void gen_custom_type_state(
+  seed_t seed,
+  char* data,
+  custom_type_state_t min,
+  custom_type_state_t max,
+  std::size_t elements,
+  std::size_t element_size);
+
+template <typename OffsetT, typename KeyT>
+void init_key_segments(::cuda::std::span<const OffsetT> segment_offsets, KeyT* d_out, std::size_t element_size);
+
+template <typename T>
+void gen_values_between(seed_t seed, ::cuda::std::span<T> data, T min, T max);
+
+template <typename T>
+void gen_values_cyclic(modulo_t mod, ::cuda::std::span<T> data);
+
+template <typename T>
+std::size_t gen_uniform_offsets(
+  seed_t seed, cuda::std::span<T> segment_offsets, T total_elements, T min_segment_size, T max_segment_size);
+} // namespace detail
+
+template <template <typename> class... Ps>
+void gen(seed_t seed,
+         device_vector<custom_type_t<Ps...>>& data,
+         custom_type_t<Ps...> min = ::cuda::std::numeric_limits<custom_type_t<Ps...>>::lowest(),
+         custom_type_t<Ps...> max = ::cuda::std::numeric_limits<custom_type_t<Ps...>>::max())
+{
+  detail::gen_custom_type_state(
+    seed,
+    reinterpret_cast<char*>(THRUST_NS_QUALIFIER::raw_pointer_cast(data.data())),
+    min,
+    max,
+    data.size(),
+    sizeof(custom_type_t<Ps...>));
+}
+
+template <typename T>
+void gen(seed_t seed,
+         device_vector<T>& data,
+         T min = ::cuda::std::numeric_limits<T>::lowest(),
+         T max = ::cuda::std::numeric_limits<T>::max())
+{
+  detail::gen_values_between(seed, {THRUST_NS_QUALIFIER::raw_pointer_cast(data.data()), data.size()}, min, max);
+}
+
+template <typename T>
+void gen(modulo_t mod, device_vector<T>& data)
+{
+  detail::gen_values_cyclic(mod, ::cuda::std::span<T>{THRUST_NS_QUALIFIER::raw_pointer_cast(data.data()), data.size()});
+}
+
+/**
+ * @brief Generates an array of offsets with uniformly distributed segment sizes in the range
+ * between [min_segment_size, max_segment_size]. The last offset in the array corresponds to
+ * `total_element`. At most `total_element+2` offsets (or `total_elements+1` segments) and, because
+ * the very last offset must corresponds to `total_element`, the last segment may comprise more than
+ * `max_segment_size` items.
+ */
+template <typename T>
+device_vector<T> gen_uniform_offsets(seed_t seed, T total_elements, T min_segment_size, T max_segment_size)
+{
+  device_vector<T> segment_offsets(total_elements + 2);
+  const auto new_size = detail::gen_uniform_offsets(
+    seed,
+    {THRUST_NS_QUALIFIER::raw_pointer_cast(segment_offsets.data()), segment_offsets.size()},
+    total_elements,
+    min_segment_size,
+    max_segment_size);
+  segment_offsets.resize(new_size);
+  return segment_offsets;
+}
+
+/**
+ * @brief Generates key-segment ranges from an offsets-array like the one given by
+ * `gen_uniform_offset`.
+ */
+template <typename OffsetT, typename KeyT>
+void init_key_segments(const device_vector<OffsetT>& segment_offsets, device_vector<KeyT>& keys_out)
+{
+  detail::init_key_segments(
+    ::cuda::std::span<const OffsetT>{
+      THRUST_NS_QUALIFIER::raw_pointer_cast(segment_offsets.data()), segment_offsets.size()},
+    THRUST_NS_QUALIFIER::raw_pointer_cast(keys_out.data()),
+    sizeof(KeyT));
+}
+
+template <typename OffsetT, template <typename> class... Ps>
+void init_key_segments(const device_vector<OffsetT>& segment_offsets, device_vector<custom_type_t<Ps...>>& keys_out)
+{
+  detail::init_key_segments(
+    ::cuda::std::span<const OffsetT>{
+      THRUST_NS_QUALIFIER::raw_pointer_cast(segment_offsets.data()), segment_offsets.size()},
+    static_cast<custom_type_state_t*>(THRUST_NS_QUALIFIER::raw_pointer_cast(keys_out.data())),
+    sizeof(custom_type_t<Ps...>));
+}
+} // namespace c2h

cccl_upstream/c2h/include/c2h/half.cuh

@@ -0,0 +1,345 @@
+// SPDX-FileCopyrightText: Copyright (c) 2011, Duane Merrill. All rights reserved.
+// SPDX-FileCopyrightText: Copyright (c) 2011-2019, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3
+
+#pragma once
+
+/**
+ * \file
+ * Utilities for interacting with the opaque CUDA __half type
+ */
+
+#include <cuda_fp16.h>
+
+#include <cub/util_type.cuh>
+
+#include <cuda/std/limits>
+#include <cuda/std/type_traits>
+
+#include <cstdint>
+#include <cstring>
+#include <iosfwd>
+
+#ifdef __GNUC__
+// There's a ton of type-punning going on in this file.
+#  pragma GCC diagnostic push
+#  pragma GCC diagnostic ignored "-Wstrict-aliasing"
+#endif
+
+/******************************************************************************
+ * half_t
+ ******************************************************************************/
+
+/**
+ * Host-based fp16 data type compatible and convertible with __half
+ */
+// TODO(bgruber): drop this when CTK 12.2 is the minimum, since it provides __host__ __device__ operators of __half
+struct half_t
+{
+  uint16_t __x;
+
+  /// Constructor from __half
+  __host__ __device__ __forceinline__ explicit half_t(const __half& other)
+  {
+    __x = reinterpret_cast<const uint16_t&>(other);
+  }
+
+  /// Constructor from integer
+  __host__ __device__ __forceinline__ explicit half_t(int a)
+  {
+    *this = half_t(float(a));
+  }
+
+  /// Constructor from std::size_t
+  __host__ __device__ __forceinline__ explicit half_t(std::size_t a)
+  {
+    *this = half_t(float(a));
+  }
+
+  /// Constructor from double
+  __host__ __device__ __forceinline__ explicit half_t(double a)
+  {
+    *this = half_t(float(a));
+  }
+
+  /// Constructor from unsigned long long int
+  template <typename T,
+            typename = typename ::cuda::std::enable_if<
+              ::cuda::std::is_same<T, unsigned long long int>::value
+              && (!::cuda::std::is_same<std::size_t, unsigned long long int>::value)>::type>
+  __host__ __device__ __forceinline__ explicit half_t(T a)
+  {
+    *this = half_t(float(a));
+  }
+
+  /// Default constructor
+  half_t() = default;
+
+  /// Constructor from float
+  __host__ __device__ __forceinline__ explicit half_t(float a)
+  {
+    // Stolen from Norbert Juffa
+    uint32_t ia = *reinterpret_cast<uint32_t*>(&a);
+    uint16_t ir;
+
+    ir = (ia >> 16) & 0x8000;
+
+    if ((ia & 0x7f800000) == 0x7f800000)
+    {
+      if ((ia & 0x7fffffff) == 0x7f800000)
+      {
+        ir |= 0x7c00; /* infinity */
+      }
+      else
+      {
+        ir = 0x7fff; /* canonical NaN */
+      }
+    }
+    else if ((ia & 0x7f800000) >= 0x33000000)
+    {
+      int32_t shift = (int32_t) ((ia >> 23) & 0xff) - 127;
+      if (shift > 15)
+      {
+        ir |= 0x7c00; /* infinity */
+      }
+      else
+      {
+        ia = (ia & 0x007fffff) | 0x00800000; /* extract mantissa */
+        if (shift < -14)
+        { /* denormal */
+          ir |= ia >> (-1 - shift);
+          ia = ia << (32 - (-1 - shift));
+        }
+        else
+        { /* normal */
+          ir |= ia >> (24 - 11);
+          ia = ia << (32 - (24 - 11));
+          ir = static_cast<uint16_t>(ir + ((14 + shift) << 10));
+        }
+        /* IEEE-754 round to nearest of even */
+        if ((ia > 0x80000000) || ((ia == 0x80000000) && (ir & 1)))
+        {
+          ir++;
+        }
+      }
+    }
+
+    this->__x = ir;
+  }
+
+  /// Cast to __half
+  __host__ __device__ __forceinline__ operator __half() const
+  {
+    return reinterpret_cast<const __half&>(__x);
+  }
+
+  /// Cast to float
+  __host__ __device__ __forceinline__ operator float() const
+  {
+    // Stolen from Andrew Kerr
+
+    int sign        = ((this->__x >> 15) & 1);
+    int exp         = ((this->__x >> 10) & 0x1f);
+    int mantissa    = (this->__x & 0x3ff);
+    std::uint32_t f = 0;
+
+    if (exp > 0 && exp < 31)
+    {
+      // normal
+      exp += 112;
+      f = (sign << 31) | (exp << 23) | (mantissa << 13);
+    }
+    else if (exp == 0)
+    {
+      if (mantissa)
+      {
+        // subnormal
+        exp += 113;
+        while ((mantissa & (1 << 10)) == 0)
+        {
+          mantissa <<= 1;
+          exp--;
+        }
+        mantissa &= 0x3ff;
+        f = (sign << 31) | (exp << 23) | (mantissa << 13);
+      }
+      else if (sign)
+      {
+        f = 0x80000000; // negative zero
+      }
+      else
+      {
+        f = 0x0; // zero
+      }
+    }
+    else if (exp == 31)
+    {
+      if (mantissa)
+      {
+        f = 0x7fffffff; // not a number
+      }
+      else
+      {
+        f = (0xff << 23) | (sign << 31); //  inf
+      }
+    }
+
+    static_assert(sizeof(float) == sizeof(std::uint32_t), "4-byte size check");
+    float ret{};
+    std::memcpy(&ret, &f, sizeof(float));
+    return ret;
+  }
+
+  /// Get raw storage
+  __host__ __device__ __forceinline__ uint16_t raw() const
+  {
+    return this->__x;
+  }
+
+  /// Equality
+  __host__ __device__ __forceinline__ friend bool operator==(const half_t& a, const half_t& b)
+  {
+    return (a.__x == b.__x);
+  }
+
+  /// Inequality
+  __host__ __device__ __forceinline__ friend bool operator!=(const half_t& a, const half_t& b)
+  {
+    return (a.__x != b.__x);
+  }
+
+  /// Assignment by sum
+  __host__ __device__ __forceinline__ half_t& operator+=(const half_t& rhs)
+  {
+    *this = half_t(float(*this) + float(rhs));
+    return *this;
+  }
+
+  /// Multiply
+  __host__ __device__ __forceinline__ half_t operator*(const half_t& other) const
+  {
+    return half_t(float(*this) * float(other));
+  }
+
+  /// Divide
+  __host__ __device__ __forceinline__ half_t& operator/=(const half_t& other)
+  {
+    return *this = half_t(float(*this) / float(other));
+  }
+
+  friend __host__ __device__ __forceinline__ half_t operator/(half_t self, const half_t& other)
+  {
+    return self /= other;
+  }
+
+  /// Add
+  __host__ __device__ __forceinline__ half_t operator+(const half_t& other) const
+  {
+    return half_t(float(*this) + float(other));
+  }
+
+  /// Sub
+  __host__ __device__ __forceinline__ half_t operator-(const half_t& other) const
+  {
+    return half_t(float(*this) - float(other));
+  }
+
+  /// Less-than
+  __host__ __device__ __forceinline__ bool operator<(const half_t& other) const
+  {
+    return float(*this) < float(other);
+  }
+
+  /// Less-than-equal
+  __host__ __device__ __forceinline__ bool operator<=(const half_t& other) const
+  {
+    return float(*this) <= float(other);
+  }
+
+  /// Greater-than
+  __host__ __device__ __forceinline__ bool operator>(const half_t& other) const
+  {
+    return float(*this) > float(other);
+  }
+
+  /// Greater-than-equal
+  __host__ __device__ __forceinline__ bool operator>=(const half_t& other) const
+  {
+    return float(*this) >= float(other);
+  }
+
+  /// numeric_traits<half_t>::max
+  __host__ __device__ __forceinline__ static half_t(max)()
+  {
+    uint16_t max_word = 0x7BFF;
+    return reinterpret_cast<half_t&>(max_word);
+  }
+
+  /// numeric_traits<half_t>::lowest
+  __host__ __device__ __forceinline__ static half_t lowest()
+  {
+    uint16_t lowest_word = 0xFBFF;
+    return reinterpret_cast<half_t&>(lowest_word);
+  }
+};
+
+/******************************************************************************
+ * I/O stream overloads
+ ******************************************************************************/
+
+/// Insert formatted \p half_t into the output stream
+inline std::ostream& operator<<(std::ostream& out, const half_t& x)
+{
+  out << (float) x;
+  return out;
+}
+
+/// Insert formatted \p __half into the output stream
+inline std::ostream& operator<<(std::ostream& out, const __half& x)
+{
+  return out << half_t(x);
+}
+
+/******************************************************************************
+ * Traits overloads
+ ******************************************************************************/
+
+namespace cuda
+{
+template <>
+inline constexpr bool is_floating_point_v<half_t> = true;
+}
+
+template <>
+class cuda::std::numeric_limits<half_t>
+{
+public:
+  static constexpr bool is_specialized = true;
+
+  static _CCCL_HOST_DEVICE _CCCL_FORCEINLINE half_t max()
+  {
+    return half_t(numeric_limits<__half>::max());
+  }
+
+  static _CCCL_HOST_DEVICE _CCCL_FORCEINLINE half_t min()
+  {
+    return half_t(numeric_limits<__half>::min());
+  }
+
+  static _CCCL_HOST_DEVICE _CCCL_FORCEINLINE half_t lowest()
+  {
+    return half_t(numeric_limits<__half>::lowest());
+  }
+};
+
+CUB_NAMESPACE_BEGIN
+
+template <>
+struct NumericTraits<half_t> : BaseTraits<FLOATING_POINT, true, uint16_t, half_t>
+{};
+
+CUB_NAMESPACE_END
+
+#ifdef __GNUC__
+#  pragma GCC diagnostic pop
+#endif

cccl_upstream/c2h/include/c2h/operator.cuh

@@ -0,0 +1,104 @@
+// SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3-Clause
+#pragma once
+
+#include <cuda/functional>
+#include <cuda/std/functional>
+#include <cuda/std/limits>
+#include <cuda/type_traits>
+
+#include <c2h/custom_type.h>
+#include <c2h/extended_types.h>
+#include <c2h/test_util_vec.h>
+
+/***********************************************************************************************************************
+ * CUB operator to identity
+ **********************************************************************************************************************/
+
+template <typename Operator, typename T, typename = void>
+inline constexpr T identity_v = cuda::identity_element<Operator, T>();
+
+template <typename T>
+inline const T identity_v<cuda::std::plus<>, T> = T{}; // e.g. short2, float2, complex<__half> etc.
+
+/***********************************************************************************************************************
+ * half_t specializations
+ **********************************************************************************************************************/
+
+template <>
+inline const half_t identity_v<cuda::std::plus<>, half_t> = half_t{0.0f};
+
+template <>
+inline const half_t identity_v<cuda::std::multiplies<>, half_t> = half_t{1.0f};
+
+template <>
+inline const half_t identity_v<cuda::minimum<>, half_t> = cuda::std::numeric_limits<half_t>::max();
+
+template <>
+inline const half_t identity_v<cuda::maximum<>, half_t> = cuda::std::numeric_limits<half_t>::lowest();
+
+/***********************************************************************************************************************
+ * bfloat16_t specializations
+ **********************************************************************************************************************/
+
+template <>
+inline const bfloat16_t identity_v<cuda::std::plus<>, bfloat16_t> = bfloat16_t{0.0f};
+
+template <>
+inline const bfloat16_t identity_v<cuda::std::multiplies<>, bfloat16_t> = bfloat16_t{1.0f};
+
+template <>
+inline const bfloat16_t identity_v<cuda::minimum<>, bfloat16_t> = cuda::std::numeric_limits<bfloat16_t>::max();
+
+template <>
+inline const bfloat16_t identity_v<cuda::maximum<>, bfloat16_t> = cuda::std::numeric_limits<bfloat16_t>::lowest();
+
+/***********************************************************************************************************************
+ * short2, ushort2, float2 specializations
+ **********************************************************************************************************************/
+
+template <>
+inline constexpr short2 identity_v<cuda::maximum<>, short2> =
+  short2{cuda::std::numeric_limits<int16_t>::lowest(), cuda::std::numeric_limits<int16_t>::lowest()};
+
+template <>
+inline constexpr ushort2 identity_v<cuda::maximum<>, ushort2> = ushort2{0, 0};
+
+template <>
+inline constexpr float2 identity_v<cuda::maximum<>, float2> =
+  float2{cuda::std::numeric_limits<float>::lowest(), cuda::std::numeric_limits<float>::lowest()};
+
+template <>
+inline const __half2 identity_v<cuda::maximum<>, __half2> =
+  __half2{cuda::std::numeric_limits<__half>::lowest(), cuda::std::numeric_limits<__half>::lowest()};
+
+template <>
+inline const __nv_bfloat162 identity_v<cuda::maximum<>, __nv_bfloat162> = __nv_bfloat162{
+  cuda::std::numeric_limits<__nv_bfloat16>::lowest(), cuda::std::numeric_limits<__nv_bfloat16>::lowest()};
+
+template <>
+inline constexpr short2 identity_v<cuda::minimum<>, short2> =
+  short2{cuda::std::numeric_limits<int16_t>::max(), cuda::std::numeric_limits<int16_t>::max()};
+
+template <>
+inline constexpr ushort2 identity_v<cuda::minimum<>, ushort2> =
+  ushort2{cuda::std::numeric_limits<uint16_t>::max(), cuda::std::numeric_limits<uint16_t>::max()};
+
+template <>
+inline const __half2 identity_v<cuda::minimum<>, __half2> =
+  __half2{cuda::std::numeric_limits<__half>::max(), cuda::std::numeric_limits<__half>::max()};
+
+template <>
+inline const __nv_bfloat162 identity_v<cuda::minimum<>, __nv_bfloat162> =
+  __nv_bfloat162{cuda::std::numeric_limits<__nv_bfloat16>::max(), cuda::std::numeric_limits<__nv_bfloat16>::max()};
+
+template <template <typename> class... Policies>
+inline const c2h::custom_type_t<Policies...> identity_v<cuda::maximum<>, c2h::custom_type_t<Policies...>> =
+  cuda::std::numeric_limits<c2h::custom_type_t<Policies...>>::lowest();
+
+template <template <typename> class... Policies>
+inline const c2h::custom_type_t<Policies...> identity_v<cuda::minimum<>, c2h::custom_type_t<Policies...>> =
+  cuda::std::numeric_limits<c2h::custom_type_t<Policies...>>::max();
+
+struct custom_plus : cuda::std::plus<>
+{};

cccl_upstream/c2h/include/c2h/test_util_vec.h

@@ -0,0 +1,416 @@
+// SPDX-FileCopyrightText: Copyright (c) 2011-2022, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3-Clause
+
+#pragma once
+
+#include <thrust/detail/config/device_system.h>
+
+#include <cuda/std/limits>
+#include <cuda/std/type_traits>
+
+#include <iostream>
+
+#include <c2h/extended_types.h>
+
+/******************************************************************************
+ * Console printing utilities
+ ******************************************************************************/
+
+/**
+ * Helper for casting character types to integers for cout printing
+ */
+template <typename T>
+T CoutCast(T val)
+{
+  return val;
+}
+
+inline int CoutCast(char val)
+{
+  return val;
+}
+
+inline int CoutCast(unsigned char val)
+{
+  return val;
+}
+
+inline int CoutCast(signed char val)
+{
+  return val;
+}
+
+/******************************************************************************
+ * Comparison and ostream operators for CUDA vector types
+ ******************************************************************************/
+
+#if THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+
+/**
+ * Vector1 overloads
+ */
+#  define C2H_VEC_OVERLOAD_1(T)                                                  \
+    /* Ostream output */                                                         \
+    inline std::ostream& operator<<(std::ostream& os, const T& val)              \
+    {                                                                            \
+      os << '(' << CoutCast(val.x) << ')';                                       \
+      return os;                                                                 \
+    }                                                                            \
+    /* Inequality */                                                             \
+    inline __host__ __device__ constexpr bool operator!=(const T& a, const T& b) \
+    {                                                                            \
+      return (a.x != b.x);                                                       \
+    }                                                                            \
+    /* Equality */                                                               \
+    inline __host__ __device__ constexpr bool operator==(const T& a, const T& b) \
+    {                                                                            \
+      return (a.x == b.x);                                                       \
+    }                                                                            \
+    /* Max */                                                                    \
+    inline __host__ __device__ constexpr bool operator>(const T& a, const T& b)  \
+    {                                                                            \
+      return (a.x > b.x);                                                        \
+    }                                                                            \
+    /* Min */                                                                    \
+    inline __host__ __device__ constexpr bool operator<(const T& a, const T& b)  \
+    {                                                                            \
+      return (a.x < b.x);                                                        \
+    }                                                                            \
+    /* Summation (non-reference addends for VS2003 -O3 warpscan workaround */    \
+    inline __host__ __device__ constexpr T operator+(T a, T b)                   \
+    {                                                                            \
+      using V = decltype(T::x);                                                  \
+      return T{static_cast<V>(a.x + b.x)};                                       \
+    }
+
+/**
+ * Vector2 overloads
+ */
+#  define C2H_VEC_OVERLOAD_2(T)                                                  \
+    /* Ostream output */                                                         \
+    inline std::ostream& operator<<(std::ostream& os, const T& val)              \
+    {                                                                            \
+      os << '(' << CoutCast(val.x) << ',' << CoutCast(val.y) << ')';             \
+      return os;                                                                 \
+    }                                                                            \
+    /* Inequality */                                                             \
+    inline __host__ __device__ constexpr bool operator!=(const T& a, const T& b) \
+    {                                                                            \
+      return (a.x != b.x) || (a.y != b.y);                                       \
+    }                                                                            \
+    /* Equality */                                                               \
+    inline __host__ __device__ constexpr bool operator==(const T& a, const T& b) \
+    {                                                                            \
+      return (a.x == b.x) && (a.y == b.y);                                       \
+    }                                                                            \
+    /* Max */                                                                    \
+    inline __host__ __device__ constexpr bool operator>(const T& a, const T& b)  \
+    {                                                                            \
+      if (a.x > b.x)                                                             \
+        return true;                                                             \
+      else if (b.x > a.x)                                                        \
+        return false;                                                            \
+      return a.y > b.y;                                                          \
+    }                                                                            \
+    /* Min */                                                                    \
+    inline __host__ __device__ constexpr bool operator<(const T& a, const T& b)  \
+    {                                                                            \
+      if (a.x < b.x)                                                             \
+        return true;                                                             \
+      else if (b.x < a.x)                                                        \
+        return false;                                                            \
+      return a.y < b.y;                                                          \
+    }                                                                            \
+    /* Summation (non-reference addends for VS2003 -O3 warpscan workaround */    \
+    inline __host__ __device__ constexpr T operator+(T a, T b)                   \
+    {                                                                            \
+      using V = decltype(T::x);                                                  \
+      return T{static_cast<V>(a.x + b.x), static_cast<V>(a.y + b.y)};            \
+    }
+
+/**
+ * Vector3 overloads
+ */
+#  define C2H_VEC_OVERLOAD_3(T)                                                                  \
+    /* Ostream output */                                                                         \
+    inline std::ostream& operator<<(std::ostream& os, const T& val)                              \
+    {                                                                                            \
+      os << '(' << CoutCast(val.x) << ',' << CoutCast(val.y) << ',' << CoutCast(val.z) << ')';   \
+      return os;                                                                                 \
+    }                                                                                            \
+    /* Inequality */                                                                             \
+    inline __host__ __device__ constexpr bool operator!=(const T& a, const T& b)                 \
+    {                                                                                            \
+      return (a.x != b.x) || (a.y != b.y) || (a.z != b.z);                                       \
+    }                                                                                            \
+    /* Equality */                                                                               \
+    inline __host__ __device__ constexpr bool operator==(const T& a, const T& b)                 \
+    {                                                                                            \
+      return (a.x == b.x) && (a.y == b.y) && (a.z == b.z);                                       \
+    }                                                                                            \
+    /* Max */                                                                                    \
+    inline __host__ __device__ constexpr bool operator>(const T& a, const T& b)                  \
+    {                                                                                            \
+      if (a.x > b.x)                                                                             \
+        return true;                                                                             \
+      else if (b.x > a.x)                                                                        \
+        return false;                                                                            \
+      if (a.y > b.y)                                                                             \
+        return true;                                                                             \
+      else if (b.y > a.y)                                                                        \
+        return false;                                                                            \
+      return a.z > b.z;                                                                          \
+    }                                                                                            \
+    /* Min */                                                                                    \
+    inline __host__ __device__ constexpr bool operator<(const T& a, const T& b)                  \
+    {                                                                                            \
+      if (a.x < b.x)                                                                             \
+        return true;                                                                             \
+      else if (b.x < a.x)                                                                        \
+        return false;                                                                            \
+      if (a.y < b.y)                                                                             \
+        return true;                                                                             \
+      else if (b.y < a.y)                                                                        \
+        return false;                                                                            \
+      return a.z < b.z;                                                                          \
+    }                                                                                            \
+    /* Summation (non-reference addends for VS2003 -O3 warpscan workaround */                    \
+    inline __host__ __device__ constexpr T operator+(T a, T b)                                   \
+    {                                                                                            \
+      using V = decltype(T::x);                                                                  \
+      return T{static_cast<V>(a.x + b.x), static_cast<V>(a.y + b.y), static_cast<V>(a.z + b.z)}; \
+    }
+
+/**
+ * Vector4 overloads
+ */
+#  define C2H_VEC_OVERLOAD_4(T)                                                                                      \
+    /* Ostream output */                                                                                             \
+    inline std::ostream& operator<<(std::ostream& os, const T& val)                                                  \
+    {                                                                                                                \
+      os << '(' << CoutCast(val.x) << ',' << CoutCast(val.y) << ',' << CoutCast(val.z) << ',' << CoutCast(val.w)     \
+         << ')';                                                                                                     \
+      return os;                                                                                                     \
+    }                                                                                                                \
+    /* Inequality */                                                                                                 \
+    inline __host__ __device__ constexpr bool operator!=(const T& a, const T& b)                                     \
+    {                                                                                                                \
+      return (a.x != b.x) || (a.y != b.y) || (a.z != b.z) || (a.w != b.w);                                           \
+    }                                                                                                                \
+    /* Equality */                                                                                                   \
+    inline __host__ __device__ constexpr bool operator==(const T& a, const T& b)                                     \
+    {                                                                                                                \
+      return (a.x == b.x) && (a.y == b.y) && (a.z == b.z) && (a.w == b.w);                                           \
+    }                                                                                                                \
+    /* Max */                                                                                                        \
+    inline __host__ __device__ constexpr bool operator>(const T& a, const T& b)                                      \
+    {                                                                                                                \
+      if (a.x > b.x)                                                                                                 \
+        return true;                                                                                                 \
+      else if (b.x > a.x)                                                                                            \
+        return false;                                                                                                \
+      if (a.y > b.y)                                                                                                 \
+        return true;                                                                                                 \
+      else if (b.y > a.y)                                                                                            \
+        return false;                                                                                                \
+      if (a.z > b.z)                                                                                                 \
+        return true;                                                                                                 \
+      else if (b.z > a.z)                                                                                            \
+        return false;                                                                                                \
+      return a.w > b.w;                                                                                              \
+    }                                                                                                                \
+    /* Min */                                                                                                        \
+    inline __host__ __device__ constexpr bool operator<(const T& a, const T& b)                                      \
+    {                                                                                                                \
+      if (a.x < b.x)                                                                                                 \
+        return true;                                                                                                 \
+      else if (b.x < a.x)                                                                                            \
+        return false;                                                                                                \
+      if (a.y < b.y)                                                                                                 \
+        return true;                                                                                                 \
+      else if (b.y < a.y)                                                                                            \
+        return false;                                                                                                \
+      if (a.z < b.z)                                                                                                 \
+        return true;                                                                                                 \
+      else if (b.z < a.z)                                                                                            \
+        return false;                                                                                                \
+      return a.w < b.w;                                                                                              \
+    }                                                                                                                \
+    /* Summation (non-reference addends for VS2003 -O3 warpscan workaround */                                        \
+    inline __host__ __device__ constexpr T operator+(T a, T b)                                                       \
+    {                                                                                                                \
+      using V = decltype(T::x);                                                                                      \
+      return T{                                                                                                      \
+        static_cast<V>(a.x + b.x), static_cast<V>(a.y + b.y), static_cast<V>(a.z + b.z), static_cast<V>(a.w + b.w)}; \
+    }
+
+/**
+ * All vector overloads
+ */
+#  define C2H_VEC_OVERLOAD(VecName) \
+    C2H_VEC_OVERLOAD_1(VecName##1)  \
+    C2H_VEC_OVERLOAD_2(VecName##2)  \
+    C2H_VEC_OVERLOAD_3(VecName##3)  \
+    C2H_VEC_OVERLOAD_4(VecName##4)
+
+/**
+ * Define for types
+ */
+C2H_VEC_OVERLOAD(char)
+C2H_VEC_OVERLOAD(short)
+C2H_VEC_OVERLOAD(int)
+_CCCL_SUPPRESS_DEPRECATED_PUSH
+C2H_VEC_OVERLOAD(long)
+C2H_VEC_OVERLOAD(longlong)
+_CCCL_SUPPRESS_DEPRECATED_POP
+#  if _CCCL_CTK_AT_LEAST(13, 0)
+C2H_VEC_OVERLOAD_4(long4_16a)
+C2H_VEC_OVERLOAD_4(long4_32a)
+C2H_VEC_OVERLOAD_4(longlong4_16a)
+C2H_VEC_OVERLOAD_4(longlong4_32a)
+#  endif // _CCCL_CTK_AT_LEAST(13, 0)
+C2H_VEC_OVERLOAD(uchar)
+C2H_VEC_OVERLOAD(ushort)
+C2H_VEC_OVERLOAD(uint)
+_CCCL_SUPPRESS_DEPRECATED_PUSH
+C2H_VEC_OVERLOAD(ulong)
+C2H_VEC_OVERLOAD(ulonglong)
+_CCCL_SUPPRESS_DEPRECATED_POP
+#  if _CCCL_CTK_AT_LEAST(13, 0)
+C2H_VEC_OVERLOAD_4(ulong4_16a)
+C2H_VEC_OVERLOAD_4(ulong4_32a)
+C2H_VEC_OVERLOAD_4(ulonglong4_16a)
+C2H_VEC_OVERLOAD_4(ulonglong4_32a)
+#  endif // _CCCL_CTK_AT_LEAST(13, 0)
+C2H_VEC_OVERLOAD(float)
+_CCCL_SUPPRESS_DEPRECATED_PUSH
+C2H_VEC_OVERLOAD(double)
+_CCCL_SUPPRESS_DEPRECATED_POP
+#  if _CCCL_CTK_AT_LEAST(13, 0)
+C2H_VEC_OVERLOAD_4(double4_16a)
+C2H_VEC_OVERLOAD_4(double4_32a)
+#  endif // _CCCL_CTK_AT_LEAST(13, 0)
+
+// Specialize cuda::std::numeric_limits for vector types.
+
+#  define REPEAT_TO_LIST_1(a)  a
+#  define REPEAT_TO_LIST_2(a)  a, a
+#  define REPEAT_TO_LIST_3(a)  a, a, a
+#  define REPEAT_TO_LIST_4(a)  a, a, a, a
+#  define REPEAT_TO_LIST(N, a) _CCCL_PP_CAT(REPEAT_TO_LIST_, N)(a)
+
+#  define C2H_VEC_TRAITS_OVERLOAD_IMPL(T, BaseT, N)                               \
+    _CCCL_BEGIN_NAMESPACE_CUDA_STD                                                \
+    template <>                                                                   \
+    class numeric_limits<T>                                                       \
+    {                                                                             \
+    public:                                                                       \
+      static constexpr bool is_specialized = true;                                \
+      static __host__ __device__ T max()                                          \
+      {                                                                           \
+        return {REPEAT_TO_LIST(N, ::cuda::std::numeric_limits<BaseT>::max())};    \
+      }                                                                           \
+      static __host__ __device__ T min()                                          \
+      {                                                                           \
+        return {REPEAT_TO_LIST(N, ::cuda::std::numeric_limits<BaseT>::min())};    \
+      }                                                                           \
+      static __host__ __device__ T lowest()                                       \
+      {                                                                           \
+        return {REPEAT_TO_LIST(N, ::cuda::std::numeric_limits<BaseT>::lowest())}; \
+      }                                                                           \
+    };                                                                            \
+    _CCCL_END_NAMESPACE_CUDA_STD
+
+#  define C2H_VEC_TRAITS_OVERLOAD(COMPONENT_T, BaseT)      \
+    C2H_VEC_TRAITS_OVERLOAD_IMPL(COMPONENT_T##1, BaseT, 1) \
+    C2H_VEC_TRAITS_OVERLOAD_IMPL(COMPONENT_T##2, BaseT, 2) \
+    C2H_VEC_TRAITS_OVERLOAD_IMPL(COMPONENT_T##3, BaseT, 3) \
+    C2H_VEC_TRAITS_OVERLOAD_IMPL(COMPONENT_T##4, BaseT, 4)
+
+C2H_VEC_TRAITS_OVERLOAD(char, signed char)
+C2H_VEC_TRAITS_OVERLOAD(short, short)
+C2H_VEC_TRAITS_OVERLOAD(int, int)
+_CCCL_SUPPRESS_DEPRECATED_PUSH
+C2H_VEC_TRAITS_OVERLOAD(long, long)
+C2H_VEC_TRAITS_OVERLOAD(longlong, long long)
+_CCCL_SUPPRESS_DEPRECATED_POP
+C2H_VEC_TRAITS_OVERLOAD(uchar, unsigned char)
+C2H_VEC_TRAITS_OVERLOAD(ushort, unsigned short)
+C2H_VEC_TRAITS_OVERLOAD(uint, unsigned int)
+_CCCL_SUPPRESS_DEPRECATED_PUSH
+C2H_VEC_TRAITS_OVERLOAD(ulong, unsigned long)
+C2H_VEC_TRAITS_OVERLOAD(ulonglong, unsigned long long)
+_CCCL_SUPPRESS_DEPRECATED_POP
+C2H_VEC_TRAITS_OVERLOAD(float, float)
+_CCCL_SUPPRESS_DEPRECATED_PUSH
+C2H_VEC_TRAITS_OVERLOAD(double, double)
+_CCCL_SUPPRESS_DEPRECATED_POP
+
+#  if _CCCL_CTK_AT_LEAST(13, 0)
+C2H_VEC_TRAITS_OVERLOAD_IMPL(long4_16a, long, 4)
+C2H_VEC_TRAITS_OVERLOAD_IMPL(long4_32a, long, 4)
+C2H_VEC_TRAITS_OVERLOAD_IMPL(ulong4_16a, unsigned long, 4)
+C2H_VEC_TRAITS_OVERLOAD_IMPL(ulong4_32a, unsigned long, 4)
+C2H_VEC_TRAITS_OVERLOAD_IMPL(longlong4_16a, long long, 4)
+C2H_VEC_TRAITS_OVERLOAD_IMPL(longlong4_32a, long long, 4)
+C2H_VEC_TRAITS_OVERLOAD_IMPL(ulonglong4_16a, unsigned long long, 4)
+C2H_VEC_TRAITS_OVERLOAD_IMPL(ulonglong4_32a, unsigned long long, 4)
+#  endif // _CCCL_CTK_AT_LEAST(13, 0)
+
+#  undef C2H_VEC_TRAITS_OVERLOAD
+#  undef C2H_VEC_TRAITS_OVERLOAD_IMPL
+#  undef REPEAT_TO_LIST_1
+#  undef REPEAT_TO_LIST_2
+#  undef REPEAT_TO_LIST_3
+#  undef REPEAT_TO_LIST_4
+#  undef REPEAT_TO_LIST
+
+//----------------------------------------------------------------------------------------------------------------------
+// vector2 type traits
+
+template <typename T>
+inline constexpr bool is_vector2_type_v = cuda::std::__is_one_of_v<
+  cuda::std::remove_cv_t<T>,
+  char2,
+  short2,
+  int2,
+  long2,
+  longlong2,
+  uchar2,
+  ushort2,
+  uint2,
+  ulong2,
+  ulonglong2,
+  float2,
+  double2
+#  if TEST_HALF_T()
+  ,
+  __half2
+#  endif // TEST_HALF_T()
+#  if TEST_BF_T()
+  ,
+  __nv_bfloat162
+#  endif // TEST_BF_T()
+  >;
+
+//----------------------------------------------------------------------------------------------------------------------
+// vector2 floating point type traits
+
+template <typename T>
+inline constexpr bool is_vector2_fp_type_v = cuda::std::__is_one_of_v<cuda::std::remove_cv_t<T>, float2, double2>;
+
+#  if TEST_HALF_T()
+
+template <>
+inline constexpr bool is_vector2_fp_type_v<__half2> = true;
+
+#  endif // TEST_HALF_T()
+
+#  if TEST_BF_T()
+
+template <>
+inline constexpr bool is_vector2_fp_type_v<__nv_bfloat162> = true;
+
+#  endif // TEST_BF_T()
+
+#endif // THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA

cccl_upstream/c2h/include/c2h/utility.h

@@ -0,0 +1,36 @@
+// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3
+
+#pragma once
+
+#include <cstdlib>
+#include <cstring>
+#include <string>
+#include <typeinfo>
+#ifdef __GNUC__
+#  include <cxxabi.h>
+#endif // __GNUC__
+
+namespace c2h
+{
+// TODO(bgruber): duplicated version of thrust/testing/unittest/system.h
+inline std::string demangle(const char* name)
+{
+#if __GNUC__ && !_NVHPC_CUDA
+  int status     = 0;
+  char* realname = abi::__cxa_demangle(name, nullptr, nullptr, &status);
+  std::string result(realname);
+  std::free(realname);
+  return result;
+#else // __GNUC__ && !_NVHPC_CUDA
+  return name;
+#endif // __GNUC__ && !_NVHPC_CUDA
+}
+
+// TODO(bgruber): duplicated version of thrust/testing/unittest/util.h
+template <typename T>
+std::string type_name()
+{
+  return demangle(typeid(T).name());
+}
+} // namespace c2h

cccl_upstream/c2h/include/c2h/vector.h

@@ -0,0 +1,68 @@
+// SPDX-FileCopyrightText: Copyright (c) 2011-2024, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: BSD-3
+
+#pragma once
+
+#include <thrust/detail/vector_base.h>
+
+#include <cuda/std/type_traits>
+
+#include <memory>
+
+#include <catch2/catch_tostring.hpp>
+
+#if THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+#  include <c2h/checked_allocator.cuh>
+#else
+#  include <thrust/device_vector.h>
+#  include <thrust/host_vector.h>
+#endif
+
+namespace c2h
+{
+#if THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+template <typename T>
+using host_vector = THRUST_NS_QUALIFIER::detail::vector_base<T, c2h::checked_host_allocator<T>>;
+
+template <typename T>
+using device_vector = THRUST_NS_QUALIFIER::detail::vector_base<T, c2h::checked_cuda_allocator<T>>;
+#else // THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+using THRUST_NS_QUALIFIER::device_vector;
+using THRUST_NS_QUALIFIER::host_vector;
+#endif // THRUST_DEVICE_SYSTEM == THRUST_DEVICE_SYSTEM_CUDA
+} // namespace c2h
+
+// We specialize how Catch2 prints ([signed|unsigned]) char vectors for better readability. Let's print them as numbers
+// instead of characters.
+template <typename T, typename A>
+struct Catch::StringMaker<THRUST_NS_QUALIFIER::detail::vector_base<T, A>,
+                          ::cuda::std::enable_if_t<sizeof(T) == 1 && ::cuda::std::is_fundamental_v<T>>>
+{
+  // Copied from `rangeToString` in catch_tostring.hpp
+  static auto convert(const THRUST_NS_QUALIFIER::detail::vector_base<T, A>& v) -> std::string
+  {
+    auto first = v.begin();
+    auto last  = v.end();
+
+    ReusableStringStream rss;
+    rss << "{ ";
+    if (first != last)
+    {
+      rss << Detail::stringify(static_cast<unsigned>(static_cast<T>(*first)));
+      for (++first; first != last; ++first)
+      {
+        rss << ", " << Detail::stringify(static_cast<unsigned>(static_cast<T>(*first)));
+      }
+    }
+    rss << " }";
+    return rss.str();
+  }
+};
+
+// due to an nvcc bug, the above specialization of StringMaker is ambiguous with one inside Catch2, so let's disable
+// Catch2 range formatting for vector_base with sizeof(T) == 1 entirely
+template <typename T, typename A>
+struct Catch::is_range<THRUST_NS_QUALIFIER::detail::vector_base<T, A>>
+{
+  static constexpr bool value = !(sizeof(T) == 1 && ::cuda::std::is_fundamental_v<T>);
+};

cccl_upstream/cccl-version.json

@@ -0,0 +1,6 @@
+{
+  "full": "3.6.0",
+  "major": 3,
+  "minor": 6,
+  "patch": 0
+}

cccl_upstream/cmake/AppendOptionIfAvailable.cmake

@@ -0,0 +1,11 @@
+include_guard(GLOBAL)
+include(CheckCXXCompilerFlag)
+
+macro(append_option_if_available _FLAG _LIST)
+  string(MAKE_C_IDENTIFIER "CXX_FLAG_${_FLAG}" _VAR)
+  check_cxx_compiler_flag(${_FLAG} ${_VAR})
+
+  if (${${_VAR}})
+    list(APPEND ${_LIST} ${_FLAG})
+  endif()
+endmacro()

cccl_upstream/cmake/CCCLAddExecutable.cmake

@@ -0,0 +1,53 @@
+# Adds an executable target from SOURCES with standard CCCL configuration.
+#
+# By default, metatargets are created (e.g. target name foo.bar.baz will built by metatargets
+# `foo` and `foo.bar`). This can be disabled with NO_METATARGETS. By default, the metatarget
+# path is the same as the target name, but can be overridden with METATARGET_PATH
+#
+# If ADD_CTEST is specified, a CTest test is added with the same name as the target,
+# which runs the executable with no arguments.
+function(cccl_add_executable target_name)
+  set(options ADD_CTEST NO_METATARGETS NO_CLANG_TIDY)
+  set(oneValueArgs METATARGET_PATH DIALECT)
+  set(multiValueArgs SOURCES)
+  cmake_parse_arguments(
+    _cccl
+    "${options}"
+    "${oneValueArgs}"
+    "${multiValueArgs}"
+    ${ARGN}
+  )
+
+  if (_cccl_UNPARSED_ARGUMENTS)
+    message(FATAL_ERROR "Unrecognized arguments: ${_cccl_UNPARSED_ARGUMENTS}")
+  endif()
+
+  if (NOT DEFINED _cccl_SOURCES)
+    message(FATAL_ERROR "cccl_add_executable requires SOURCES argument")
+  endif()
+
+  add_executable(${target_name} ${_cccl_SOURCES})
+
+  if (_cccl_DIALECT)
+    set(configure_args DIALECT "${_cccl_DIALECT}")
+  else()
+    set(configure_args)
+  endif()
+  cccl_configure_target(${target_name} ${configure_args})
+
+  if (_cccl_ADD_CTEST)
+    add_test(NAME ${target_name} COMMAND "$<TARGET_FILE:${target_name}>")
+  endif()
+
+  if (NOT _cccl_NO_METATARGETS)
+    set(metatarget_path ${target_name})
+    if (DEFINED _cccl_METATARGET_PATH)
+      set(metatarget_path ${_cccl_METATARGET_PATH})
+    endif()
+    cccl_ensure_metatargets(${target_name} METATARGET_PATH ${metatarget_path})
+  endif()
+
+  if (NOT _cccl_NO_CLANG_TIDY)
+    cccl_tidy_add_target(SOURCES ${_cccl_SOURCES})
+  endif()
+endfunction()

cccl_upstream/cmake/CCCLAddSubdir.cmake

@@ -0,0 +1,6 @@
+cccl_add_subdir_helper(
+  CCCL
+  # These component lists may be set by users to explicitly request subprojects:
+  REQUIRED_COMPONENTS "${CCCL_REQUIRED_COMPONENTS}"
+  OPTIONAL_COMPONENTS "${CCCL_OPTIONAL_COMPONENTS}"
+)

cccl_upstream/cmake/CCCLAddSubdirHelper.cmake

@@ -0,0 +1,75 @@
+# project_name: The name of the project when calling `find_package`. Case sensitive.
+# `PACKAGE_FILEBASE` the name of the project in the config files, ie. ${PACKAGE_FILEBASE}-config.cmake.
+# `PACKAGE_PATH` the absolute path to the project's CMake package config files.
+function(cccl_add_subdir_helper project_name)
+  set(options)
+  set(
+    oneValueArgs
+    PACKAGE_PATH
+    PACKAGE_FILEBASE
+    REQUIRED_COMPONENTS
+    OPTIONAL_COMPONENTS
+  )
+  set(multiValueArgs)
+  cmake_parse_arguments(
+    CCCL_SUBDIR
+    "${options}"
+    "${oneValueArgs}"
+    "${multiValueArgs}"
+    ${ARGN}
+  )
+
+  if (NOT DEFINED CCCL_SUBDIR_PACKAGE_FILEBASE)
+    string(TOLOWER "${project_name}" CCCL_SUBDIR_PACKAGE_FILEBASE)
+  endif()
+
+  if (NOT DEFINED CCCL_SUBDIR_PACKAGE_PATH)
+    set(
+      CCCL_SUBDIR_PACKAGE_PATH
+      "${CCCL_SOURCE_DIR}/lib/cmake/${CCCL_SUBDIR_PACKAGE_FILEBASE}"
+    )
+  endif()
+
+  set(
+    package_prefix
+    "${CCCL_SUBDIR_PACKAGE_PATH}/${CCCL_SUBDIR_PACKAGE_FILEBASE}"
+  )
+
+  set(CMAKE_FIND_PACKAGE_NAME ${project_name})
+  set(${CMAKE_FIND_PACKAGE_NAME}_FIND_COMPONENTS)
+  if (DEFINED CCCL_SUBDIR_REQUIRED_COMPONENTS)
+    list(
+      APPEND ${CMAKE_FIND_PACKAGE_NAME}_FIND_COMPONENTS
+      ${CCCL_SUBDIR_REQUIRED_COMPONENTS}
+    )
+    foreach (component IN LISTS CCCL_SUBDIR_REQUIRED_COMPONENTS)
+      set(${CMAKE_FIND_PACKAGE_NAME}_FIND_REQUIRED_${component} TRUE)
+    endforeach()
+  endif()
+
+  if (DEFINED CCCL_SUBDIR_OPTIONAL_COMPONENTS)
+    list(
+      APPEND ${CMAKE_FIND_PACKAGE_NAME}_FIND_COMPONENTS
+      ${CCCL_SUBDIR_OPTIONAL_COMPONENTS}
+    )
+  endif()
+
+  # This effectively does a `find_package` actually going through the find_package
+  # machinery. Using `find_package` works for the first configure, but creates
+  # inconsistencies during subsequent configurations when using CPM..
+  #
+  # More details are in the discussion at
+  # https://github.com/NVIDIA/libcudacxx/pull/242#discussion_r794003857
+  include("${package_prefix}-config-version.cmake")
+  include("${package_prefix}-config.cmake")
+
+  if (${project_name}_FOUND)
+    # Set the dir var so that later `find_package` calls work as expected.
+    set(
+      ${project_name}_DIR
+      "${CCCL_SUBDIR_PACKAGE_PATH}"
+      CACHE PATH
+      "Path to ${project_name} package"
+    )
+  endif()
+endfunction()

cccl_upstream/cmake/CCCLAddTidyTarget.cmake

@@ -0,0 +1,186 @@
+include_guard(GLOBAL)
+
+#[=======================================================================[.rst:
+cccl_tidy_init
+--------------
+
+Initialize ``clang-tidy`` support and define the global ``cccl.tidy`` target. It must be
+called before adding any CCCL ``clang-tidy`` targets.
+
+Subsequent calls to this functions are no-ops.
+
+Result Variables
+^^^^^^^^^^^^^^^^
+
+  ``CCCL_TIDY_INITIALIZED`` set to true in the parent scope.
+
+#]=======================================================================]
+function(cccl_tidy_init)
+  list(APPEND CMAKE_MESSAGE_CONTEXT "tidy_init")
+
+  if (CCCL_TIDY_INITIALIZED)
+    return()
+  endif()
+
+  find_program(CCCL_CLANG_TIDY clang-tidy REQUIRED)
+
+  execute_process(
+    COMMAND ${CCCL_CLANG_TIDY} --version
+    OUTPUT_VARIABLE version
+    ERROR_VARIABLE version
+    OUTPUT_STRIP_TRAILING_WHITESPACE
+    COMMAND_ERROR_IS_FATAL ANY
+  )
+
+  message(STATUS "Found clang-tidy: ${CCCL_CLANG_TIDY} (${version})")
+
+  add_custom_target(cccl.tidy COMMENT "clang-tidy CCCL")
+
+  set(
+    CCCL_RUN_CLANG_TIDY_SCRIPT
+    "${CMAKE_CURRENT_BINARY_DIR}/run_clang_tidy.sh"
+  )
+  set(CCCL_RUN_CLANG_TIDY_SCRIPT "${CCCL_RUN_CLANG_TIDY_SCRIPT}" PARENT_SCOPE)
+
+  configure_file(
+    "${CMAKE_CURRENT_FUNCTION_LIST_DIR}/run_clang_tidy.sh.in"
+    "${CCCL_RUN_CLANG_TIDY_SCRIPT}"
+    @ONLY
+  )
+  # Do not set to cache; multiple separate instances of CCCL in a build should not
+  # conflict.
+  set(CCCL_TIDY_INITIALIZED TRUE)
+  set(CCCL_TIDY_INITIALIZED TRUE PARENT_SCOPE)
+endfunction()
+
+#[=======================================================================[.rst:
+cccl_tidy_make_subproject_target
+--------------------------------
+
+Create a meta target per sub-project that depends on all the targets for that
+subproject. It itself will depend on the ``cccl.tidy target``. For example, this will
+create:
+
+- cub.tidy
+- libcudacxx.tidy
+- thrust.tidy
+
+etc. This allows running clang-tidy over just a subset of the repository.
+
+The generated target name depends on the current value of ``PROJECT_NAME``.
+
+Arguments
+^^^^^^^^^
+
+``result_var``
+  The variable in which to store the created target name.
+
+#]=======================================================================]
+function(cccl_tidy_make_subproject_target result_var)
+  list(APPEND CMAKE_MESSAGE_CONTEXT "tidy_make_subproject_target")
+
+  if (NOT CCCL_TIDY_INITIALIZED)
+    # For the cccl.tidy target
+    message(FATAL_ERROR "Must call cccl_tidy_init() first")
+  endif()
+
+  string(TOLOWER "${PROJECT_NAME}.tidy" target_name)
+
+  if (NOT TARGET "${target_name}")
+    add_custom_target("${target_name}" COMMENT "clang-tidy ${PROJECT_NAME}")
+    add_dependencies(cccl.tidy "${target_name}")
+  endif()
+  set(${result_var} "${target_name}" PARENT_SCOPE)
+endfunction()
+
+#[=======================================================================[.rst:
+cccl_tidy_add_target
+--------------------
+
+Create per-source ``clang-tidy`` targets and attach them to both the global ``cccl.tidy``
+target and per sub-project target (e.g. ``cub.tidy``)
+
+.. note::
+
+  :command:`cccl_tidy_init` must be called before using this function to establish the
+  global ``cccl.tidy`` target.
+
+If ``CCCL_ENABLE_CLANG_TIDY`` is false, this does nothing (except error-check the function
+call signature).
+
+Passing the same source file multiple times is allowed. A target is created for it only
+once.
+
+If ``SOURCES`` is empty, this function does nothing.
+
+Arguments
+^^^^^^^^^
+
+``SOURCES``
+  List of source files to analyze. Paths may be absolute or relative. Relative paths are
+  resolved against ``CMAKE_CURRENT_SOURCE_DIR``.
+
+#]=======================================================================]
+function(cccl_tidy_add_target)
+  list(APPEND CMAKE_MESSAGE_CONTEXT "tidy_add_target")
+
+  set(options)
+  set(one_value_args)
+  set(multi_value_args SOURCES)
+
+  cmake_parse_arguments(
+    _cccl
+    "${options}"
+    "${one_value_args}"
+    "${multi_value_args}"
+    ${ARGN}
+  )
+
+  if (_cccl_UNPARSED_ARGUMENTS)
+    message(FATAL_ERROR "Unrecognized arguments: ${_cccl_UNPARSED_ARGUMENTS}")
+  endif()
+
+  # It is still possible to call this function even if clang-tidy has not been
+  # disabled. We handle this gracefully to avoid complicating the callsite.
+  #
+  # This must come before the CCCL_TIDY_INITIALIZED check because that is only called when
+  # CCCL_ENABLE_CLANG_TIDY is true.
+  if (NOT CCCL_ENABLE_CLANG_TIDY)
+    return()
+  endif()
+
+  if (NOT CCCL_TIDY_INITIALIZED)
+    message(FATAL_ERROR "Must call cccl_tidy_init() first")
+  endif()
+
+  cccl_tidy_make_subproject_target(subproject_target)
+
+  foreach (src IN LISTS _cccl_SOURCES)
+    cmake_path(SET src NORMALIZE "${src}")
+    if (NOT IS_ABSOLUTE "${src}")
+      cmake_path(SET src NORMALIZE "${CMAKE_CURRENT_SOURCE_DIR}/${src}")
+    endif()
+
+    cmake_path(
+      RELATIVE_PATH src
+      BASE_DIRECTORY "${CCCL_SOURCE_DIR}"
+      OUTPUT_VARIABLE rel_src
+    )
+    string(MAKE_C_IDENTIFIER "${rel_src}" tidy_target)
+    set(tidy_target "${tidy_target}.tidy")
+
+    if (TARGET "${tidy_target}")
+      # We have seen this file before
+      continue()
+    endif()
+
+    add_custom_target(
+      "${tidy_target}"
+      DEPENDS "${src}" "${CCCL_RUN_CLANG_TIDY_SCRIPT}"
+      COMMAND ${CCCL_RUN_CLANG_TIDY_SCRIPT} "${src}"
+      COMMENT "clang-tidy ${rel_src}"
+    )
+
+    add_dependencies("${subproject_target}" "${tidy_target}")
+  endforeach()
+endfunction()

cccl_upstream/cmake/CCCLBuildCompilerTargets.cmake

@@ -0,0 +1,262 @@
+# This file defines the `cccl_build_compiler_targets()` function, which
+# creates the following interface targets:
+#
+# cccl.compiler_interface
+# - Interface target providing compiler-specific options needed to build
+#   CCCL's tests, examples, etc. for the current CMAKE_CUDA_STANDARD.
+#   This includes warning flags and the like.
+
+# sccache cannot handle the -Fd option generating pdb files
+set(CMAKE_MSVC_DEBUG_INFORMATION_FORMAT Embedded)
+
+option(CCCL_ENABLE_EXCEPTIONS "Enable exceptions within CCCL libraries." ON)
+option(CCCL_ENABLE_RTTI "Enable RTTI within CCCL libraries." ON)
+option(CCCL_ENABLE_WERROR "Treat warnings as errors for CCCL targets." ON)
+option(
+  CCCL_ENABLE_PRAGMA_SYSTEM_HEADER
+  "When OFF, disables the system header pragma in CCCL headers so that their warnings are visible."
+  OFF
+)
+option(CCCL_ENABLE_PTXAS_WARNINGS "Enable ptxas warnings" OFF) # currently used only in CUB
+
+function(
+  cccl_build_compiler_interface
+  interface_target
+  cuda_compile_options
+  cxx_compile_options
+  compile_defs
+)
+  # We test to see if C++ compiler options exist using try-compiles in the CXX lang, and then reuse those flags as
+  # -Xcompiler flags for CUDA targets. This requires that the CXX compiler and CUDA_HOST compilers are the same when
+  # using nvcc.
+  if (CCCL_TOPLEVEL_PROJECT AND CMAKE_CUDA_COMPILER_ID STREQUAL "NVIDIA")
+    set(cuda_host_matches_cxx_compiler FALSE)
+    if (CMAKE_VERSION VERSION_GREATER_EQUAL 3.31)
+      set(
+        host_info
+        "${CMAKE_CUDA_HOST_COMPILER} (${CMAKE_CUDA_HOST_COMPILER_ID} ${CMAKE_CUDA_HOST_COMPILER_VERSION})"
+      )
+      set(
+        cxx_info
+        "${CMAKE_CXX_COMPILER} (${CMAKE_CXX_COMPILER_ID} ${CMAKE_CXX_COMPILER_VERSION})"
+      )
+      if (
+        CMAKE_CUDA_HOST_COMPILER_ID STREQUAL CMAKE_CXX_COMPILER_ID
+        AND
+          CMAKE_CUDA_HOST_COMPILER_VERSION
+            VERSION_EQUAL
+            CMAKE_CXX_COMPILER_VERSION
+      )
+        set(cuda_host_matches_cxx_compiler TRUE)
+      endif()
+    else() # CMake < 3.31 doesn't have the CMAKE_CUDA_HOST_COMPILER_ID/VERSION variables
+      set(host_info "${CMAKE_CUDA_HOST_COMPILER}")
+      set(cxx_info "${CMAKE_CXX_COMPILER}")
+      if (CMAKE_CUDA_HOST_COMPILER STREQUAL CMAKE_CXX_COMPILER)
+        set(cuda_host_matches_cxx_compiler TRUE)
+      endif()
+    endif()
+
+    if (NOT cuda_host_matches_cxx_compiler)
+      message(
+        FATAL_ERROR
+        "CCCL developer builds require that CMAKE_CUDA_HOST_COMPILER matches "
+        "CMAKE_CXX_COMPILER when using nvcc:\n"
+        "CMAKE_CUDA_COMPILER: ${CMAKE_CUDA_COMPILER}\n"
+        "CMAKE_CUDA_HOST_COMPILER: ${host_info}\n"
+        "CMAKE_CXX_COMPILER: ${cxx_info}\n"
+        "Rerun cmake with \"-DCMAKE_CUDA_HOST_COMPILER=${CMAKE_CXX_COMPILER}\".\n"
+        "Alternatively, configure the CUDAHOSTCXX and CXX environment variables to match.\n"
+      )
+    endif()
+  endif()
+
+  add_library(${interface_target} INTERFACE)
+
+  foreach (cuda_option IN LISTS cuda_compile_options)
+    target_compile_options(
+      ${interface_target}
+      INTERFACE $<$<COMPILE_LANG_AND_ID:CUDA,NVIDIA>:${cuda_option}>
+    )
+  endforeach()
+
+  foreach (cxx_option IN LISTS cxx_compile_options)
+    target_compile_options(
+      ${interface_target}
+      INTERFACE
+        $<$<COMPILE_LANGUAGE:CXX>:${cxx_option}>
+        $<$<COMPILE_LANG_AND_ID:CUDA,NVIDIA>:-Xcompiler=${cxx_option}>
+    )
+  endforeach()
+
+  target_compile_definitions(${interface_target} INTERFACE ${compile_defs})
+endfunction()
+
+function(cccl_build_compiler_targets)
+  set(cuda_compile_options)
+  set(cxx_compile_options)
+  set(cxx_compile_definitions)
+
+  list(APPEND cuda_compile_options "-Xcudafe=--display_error_number")
+  list(APPEND cuda_compile_options "-Wno-deprecated-gpu-targets")
+  if (CCCL_ENABLE_WERROR)
+    list(APPEND cuda_compile_options "-Xcudafe=--promote_warnings")
+  endif()
+  if (CCCL_ENABLE_TILE)
+    list(APPEND cuda_compile_options "--enable-tile")
+  endif()
+
+  if (NOT CCCL_ENABLE_PRAGMA_SYSTEM_HEADER)
+    # Ensure that we build our tests without treating ourself as system header
+    list(APPEND cxx_compile_definitions "_CCCL_NO_SYSTEM_HEADER")
+  endif()
+
+  if (NOT CCCL_ENABLE_EXCEPTIONS)
+    list(APPEND cxx_compile_definitions "CCCL_DISABLE_EXCEPTIONS")
+  endif()
+
+  if (NOT CCCL_ENABLE_RTTI)
+    list(APPEND cxx_compile_definitions "CCCL_DISABLE_RTTI")
+  endif()
+
+  #  if (CCCL_USE_LIBCXX)
+  #    list(APPEND cxx_compile_options "-stdlib=libc++")
+  #    list(APPEND cxx_compile_definitions "_ALLOW_UNSUPPORTED_LIBCPP=1")
+  #  endif()
+
+  if ("MSVC" STREQUAL "${CMAKE_CXX_COMPILER_ID}")
+    list(APPEND cuda_compile_options "--use-local-env")
+    list(APPEND cxx_compile_options "/bigobj")
+    list(APPEND cxx_compile_definitions "_ENABLE_EXTENDED_ALIGNED_STORAGE")
+    list(APPEND cxx_compile_definitions "NOMINMAX")
+
+    append_option_if_available("/W4" cxx_compile_options)
+    # Treat all warnings as errors. This is only supported on Release builds,
+    # as `nv_exec_check_disable` doesn't seem to work with MSVC debug iterators
+    # and spurious warnings are emitted.
+    # See NVIDIA/thrust#1273, NVBug 3129879.
+    if (CCCL_ENABLE_WERROR)
+      if (CMAKE_BUILD_TYPE STREQUAL "Release")
+        append_option_if_available("/WX" cxx_compile_options)
+      endif()
+    endif()
+
+    # Suppress overly-pedantic/unavoidable warnings brought in with /W4:
+    # C4324: structure was padded due to alignment specifier
+    append_option_if_available("/wd4324" cxx_compile_options)
+    # C4505: unreferenced local function has been removed
+    # The CUDA `host_runtime.h` header emits this for
+    # `__cudaUnregisterBinaryUtil`.
+    append_option_if_available("/wd4505" cxx_compile_options)
+    # C4706: assignment within conditional expression
+    # MSVC doesn't provide an opt-out for this warning when the assignment is
+    # intentional. Clang will warn for these, but suppresses the warning when
+    # double-parentheses are used around the assignment. We'll let Clang catch
+    # unintentional assignments and suppress all such warnings on MSVC.
+    append_option_if_available("/wd4706" cxx_compile_options)
+
+    # MSVC STL assumes that `allocator_traits`'s allocator will use raw pointers,
+    # and the `__DECLSPEC_ALLOCATOR` macro causes issues with thrust's universal
+    # allocators:
+    #   warning C4494: 'std::allocator_traits<_Alloc>::allocate' :
+    #      Ignoring __declspec(allocator) because the function return type is not
+    #      a pointer or reference
+    # See https://github.com/microsoft/STL/issues/696
+    append_option_if_available("/wd4494" cxx_compile_options)
+
+    # Get error messages with a little arrow indicating the error location more exactly
+    append_option_if_available("/diagnostics:caret" cxx_compile_options)
+
+    if (MSVC_TOOLSET_VERSION LESS 143)
+      # winbase.h(9572): warning C5105: macro expansion producing 'defined' has undefined behavior
+      append_option_if_available("/wd5105" cxx_compile_options)
+    endif()
+  else()
+    list(APPEND cuda_compile_options "-Wreorder")
+
+    if (CCCL_ENABLE_WERROR)
+      append_option_if_available("-Werror" cxx_compile_options)
+    endif()
+
+    append_option_if_available("-Wall" cxx_compile_options)
+    append_option_if_available("-Wextra" cxx_compile_options)
+    append_option_if_available("-Wreorder" cxx_compile_options)
+    append_option_if_available("-Winit-self" cxx_compile_options)
+    append_option_if_available("-Woverloaded-virtual" cxx_compile_options)
+    append_option_if_available("-Wcast-qual" cxx_compile_options)
+    append_option_if_available("-Wpointer-arith" cxx_compile_options)
+    append_option_if_available("-Wunused-local-typedefs" cxx_compile_options)
+    append_option_if_available("-Wvla" cxx_compile_options)
+
+    # Clang-only
+    append_option_if_available("-Wnvcc-compat" cxx_compile_options)
+    append_option_if_available("-Wimplicit-fallthrough" cxx_compile_options)
+    append_option_if_available(
+      "-fdiagnostics-show-template-tree"
+      cxx_compile_options
+    )
+    append_option_if_available("-Wignored-qualifiers" cxx_compile_options)
+    append_option_if_available(
+      "-Wmissing-field-initializers"
+      cxx_compile_options
+    )
+    # Inundated with error: ISO C++11 requires at least one argument for the "..." in a
+    # variadic macro for _CCCL_REQUIRES_EXPR(), so cannot enable this.
+    #
+    # append_option_if_available("-pedantic" cxx_compile_options)
+    append_option_if_available("-Wsign-compare" cxx_compile_options)
+    append_option_if_available(
+      "-Warray-bounds-pointer-arithmetic"
+      cxx_compile_options
+    )
+    append_option_if_available("-Wassign-enum" cxx_compile_options)
+    append_option_if_available("-Wformat-pedantic" cxx_compile_options)
+    append_option_if_available("-Walloc-size" cxx_compile_options)
+    append_option_if_available("-Walloc-zero" cxx_compile_options)
+    append_option_if_available("-Wtsan" cxx_compile_options)
+    append_option_if_available("-Wenum-conversion" cxx_compile_options)
+    append_option_if_available("-Wpacked" cxx_compile_options)
+    # Clang and GCC
+    append_option_if_available(
+      "-ftemplate-backtrace-limit=0"
+      cxx_compile_options
+    )
+    append_option_if_available("-fmacro-backtrace-limit=0" cxx_compile_options)
+    # Disable GNU extensions (flag is clang only)
+    append_option_if_available("-Wgnu" cxx_compile_options)
+    append_option_if_available("-Wno-gnu-line-marker" cxx_compile_options) # WAR 3916341
+    # Calling a variadic macro with zero args is a GNU extension until C++20,
+    # but the THRUST_PP_ARITY macro is used with zero args. Need to see if this
+    # is a real problem worth fixing.
+    append_option_if_available(
+      "-Wno-gnu-zero-variadic-macro-arguments"
+      cxx_compile_options
+    )
+
+    # This complains about functions in CUDA system headers when used with nvcc.
+    append_option_if_available("-Wno-unused-function" cxx_compile_options)
+  endif()
+
+  if ("GNU" STREQUAL "${CMAKE_CXX_COMPILER_ID}")
+    if (CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL 7.3)
+      # GCC 7.3 complains about name mangling changes due to `noexcept`
+      # becoming part of the type system; we don't care.
+      append_option_if_available("-Wno-noexcept-type" cxx_compile_options)
+    endif()
+  endif()
+
+  cccl_build_compiler_interface(
+    cccl.compiler_interface
+    "${cuda_compile_options}"
+    "${cxx_compile_options}"
+    "${cxx_compile_definitions}"
+  )
+
+  # Clang-cuda only:
+  target_compile_options(
+    cccl.compiler_interface
+    INTERFACE
+      $<$<COMPILE_LANG_AND_ID:CUDA,Clang>:-Xclang=-fcuda-allow-variadic-functions>
+      $<$<COMPILE_LANG_AND_ID:CUDA,Clang>:-Wno-unknown-cuda-version>
+  )
+endfunction()

cccl_upstream/cmake/CCCLCheckCudaArchitectures.cmake

@@ -0,0 +1,122 @@
+# This file provides utilities to handle special CMAKE_CUDA_ARCHITECTURES lists for CCCL.
+#
+# If CMAKE_CUDA_ARCHITECTURES is set to one of the following values, it will be replaced
+# as described:
+#
+# 'all-cccl': All architectures known to the current NVCC above minimum_cccl_arch.
+#
+# 'all-major-cccl': All major architectures known to the current NVCC above minimum_cccl_arch,
+# plus 'minimum_cccl_arch'.
+#
+# For example on 12.9:
+#   all: 50-real;52-real;53-real;60-real;61-real;62-real;70-real;72-real;75-real;80-real;86-real;87-real;89-real;90-real;100-real;101-real;103-real;120-real;121-real;121-virtual
+#   all-cccl: 75-real;80-real;86-real;87-real;89-real;90-real;100-real;101-real;103-real;120-real;121-real;121-virtual
+#   all-major: 50-real;60-real;70-real;80-real;90-real;100-real;120-real;120-virtual
+#   all-major-cccl: 75-real;80-real;90-real;100-real;120-real;120-virtual
+
+# We don't support arches below what the latest CTK release supports:
+set(minimum_cccl_arch 75) # 13.x dropped below Turing
+
+# Check CMAKE_CUDA_ARCHITECTURES for special CCCL values and update as described above.
+function(cccl_check_cuda_architectures)
+  if (CMAKE_CUDA_ARCHITECTURES MATCHES "-cccl$")
+    message(
+      STATUS
+      "Detected special CCCL arch request: CMAKE_CUDA_ARCHITECTURES=${CMAKE_CUDA_ARCHITECTURES}"
+    )
+
+    _cccl_detect_nvcc_arch_support(arches)
+    _cccl_filter_to_supported_arches(arches)
+
+    if (CMAKE_CUDA_ARCHITECTURES STREQUAL "all-major-cccl")
+      _cccl_filter_to_all_major_cccl(arches)
+    elseif (CMAKE_CUDA_ARCHITECTURES STREQUAL "all-cccl")
+      # No further filtering needed, just use the arches as is.
+    else()
+      message(
+        FATAL_ERROR
+        "Invalid CMAKE_CUDA_ARCHITECTURES value: ${CMAKE_CUDA_ARCHITECTURES}"
+      )
+    endif()
+
+    _cccl_add_real_virtual_arch_tags(arches)
+    message(STATUS "Replacing with CMAKE_CUDA_ARCHITECTURES=${arches}")
+    set(
+      CMAKE_CUDA_ARCHITECTURES
+      "${arches}"
+      CACHE STRING
+      "CUDA architectures for CCCL"
+      FORCE
+    )
+  endif()
+endfunction()
+
+# Query nvcc --help to determine which architectures are supported.
+function(_cccl_detect_nvcc_arch_support arches_var)
+  # cccl_get_cudatoolkit() is intentionally not used here.
+  find_package(CUDAToolkit)
+  if (NOT CUDAToolkit_FOUND)
+    message(
+      FATAL_ERROR
+      "CUDAToolkit not found, '${CMAKE_CUDA_ARCHITECTURES}' arch detection failed."
+    )
+  endif()
+
+  execute_process(
+    COMMAND "${CUDAToolkit_NVCC_EXECUTABLE}" --help
+    OUTPUT_VARIABLE nvcc_help_output
+    COMMAND_ERROR_IS_FATAL ANY
+    OUTPUT_STRIP_TRAILING_WHITESPACE
+  )
+
+  string(REGEX MATCHALL "compute_[0-9]+" supported_arches "${nvcc_help_output}")
+  string(REPLACE "compute_" "" supported_arches "${supported_arches}")
+  list(SORT supported_arches COMPARE NATURAL)
+  list(REMOVE_DUPLICATES supported_arches)
+  message(VERBOSE "NVCC supports: ${supported_arches}")
+  set(${arches_var} ${supported_arches} PARENT_SCOPE)
+endfunction()
+
+# Remove all arches < minimum_cccl_arch
+function(_cccl_filter_to_supported_arches arches_var)
+  set(cccl_arches "")
+  foreach (arch IN LISTS ${arches_var})
+    if (arch GREATER_EQUAL minimum_cccl_arch)
+      list(APPEND cccl_arches ${arch})
+    endif()
+  endforeach()
+  message(VERBOSE "CCCL supported arches: ${cccl_arches}")
+  set(${arches_var} ${cccl_arches} PARENT_SCOPE)
+endfunction()
+
+# Convert all-cccl to all-major-cccl.
+function(_cccl_filter_to_all_major_cccl arches_var)
+  set(major_arches "")
+  foreach (arch IN LISTS ${arches_var})
+    math(EXPR major "(${arch} / 10) * 10")
+    if (major LESS minimum_cccl_arch)
+      set(major "${minimum_cccl_arch}")
+    endif()
+    if (NOT major IN_LIST major_arches)
+      list(APPEND major_arches ${major})
+    endif()
+  endforeach()
+  message(VERBOSE "CCCL all-major arches: ${major_arches}")
+  set(${arches_var} ${major_arches} PARENT_SCOPE)
+endfunction()
+
+function(_cccl_add_real_virtual_arch_tags arches_var)
+  set(tagged_arches "")
+
+  list(POP_BACK ${arches_var} last_arch)
+
+  foreach (arch IN LISTS ${arches_var})
+    list(APPEND tagged_arches "${arch}-real")
+  endforeach()
+
+  list(APPEND tagged_arches "${last_arch}-real")
+  list(APPEND tagged_arches "${last_arch}-virtual")
+
+  message(VERBOSE "CCCL tagged arches: ${tagged_arches}")
+  set(${arches_var} ${tagged_arches} PARENT_SCOPE)
+endfunction()

cccl_upstream/cmake/CCCLClangdCompileInfo.cmake

@@ -0,0 +1,39 @@
+# SPDX-FileCopyrightText: Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Tell cmake to generate a json file of compile commands for clangd:
+set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
+
+# Symlink the compile command output to the source dir, where clangd will find it.
+set(compile_commands_file "${CMAKE_BINARY_DIR}/compile_commands.json")
+set(compile_commands_link "${CMAKE_SOURCE_DIR}/compile_commands.json")
+message(
+  STATUS
+  "Creating symlink from ${compile_commands_link} to ${compile_commands_file}..."
+)
+cccl_execute_non_fatal_process(
+  COMMAND "${CMAKE_COMMAND}" -E rm -f "${compile_commands_link}"
+)
+cccl_execute_non_fatal_process(
+  COMMAND "${CMAKE_COMMAND}" -E touch "${compile_commands_file}"
+)
+# gersemi: off
+cccl_execute_non_fatal_process(
+  COMMAND
+    "${CMAKE_COMMAND}" -E create_symlink
+      "${compile_commands_file}"
+      "${compile_commands_link}"
+)
+# gersemi: on

cccl_upstream/cmake/CCCLConfigureTarget.cmake

@@ -0,0 +1,66 @@
+set(CCCL_EXECUTABLE_OUTPUT_DIR "${CCCL_BINARY_DIR}/bin")
+set(CCCL_LIBRARY_OUTPUT_DIR "${CCCL_BINARY_DIR}/lib")
+
+# Setup common properties for all test/example/etc targets.
+function(cccl_configure_target target_name)
+  set(options)
+  set(oneValueArgs DIALECT)
+  set(multiValueArgs)
+  cmake_parse_arguments(
+    CCT
+    "${options}"
+    "${oneValueArgs}"
+    "${multiValueArgs}"
+    ${ARGN}
+  )
+
+  get_target_property(type ${target_name} TYPE)
+
+  set_target_properties(
+    ${target_name}
+    PROPERTIES
+      # Disable compiler extensions:
+      CXX_EXTENSIONS OFF
+      CUDA_EXTENSIONS OFF
+  )
+
+  if (DEFINED CCT_DIALECT)
+    set(CMAKE_CXX_STANDARD ${CCT_DIALECT})
+    set(CMAKE_CUDA_STANDARD ${CCT_DIALECT})
+  endif()
+
+  set_target_properties(
+    ${target_name}
+    PROPERTIES
+      CXX_STANDARD ${CMAKE_CXX_STANDARD}
+      CUDA_STANDARD ${CMAKE_CUDA_STANDARD}
+      CXX_STANDARD_REQUIRED ON
+      CUDA_STANDARD_REQUIRED ON
+  )
+
+  get_property(langs GLOBAL PROPERTY ENABLED_LANGUAGES)
+  set(dialect_features)
+  if (CUDA IN_LIST langs)
+    list(APPEND dialect_features cuda_std_${CMAKE_CUDA_STANDARD})
+  endif()
+  if (CXX IN_LIST langs)
+    list(APPEND dialect_features cxx_std_${CMAKE_CXX_STANDARD})
+  endif()
+
+  get_target_property(type ${target_name} TYPE)
+  if (${type} STREQUAL "INTERFACE_LIBRARY")
+    target_compile_features(${target_name} INTERFACE ${dialect_features})
+  else()
+    target_compile_features(${target_name} PUBLIC ${dialect_features})
+  endif()
+
+  if (NOT ${type} STREQUAL "INTERFACE_LIBRARY")
+    set_target_properties(
+      ${target_name}
+      PROPERTIES
+        ARCHIVE_OUTPUT_DIRECTORY "${CCCL_LIBRARY_OUTPUT_DIR}"
+        LIBRARY_OUTPUT_DIRECTORY "${CCCL_LIBRARY_OUTPUT_DIR}"
+        RUNTIME_OUTPUT_DIRECTORY "${CCCL_EXECUTABLE_OUTPUT_DIR}"
+    )
+  endif()
+endfunction()

cccl_upstream/cmake/CCCLDevBuildChecks.cmake

@@ -0,0 +1,36 @@
+# This file contains checks that ensure a supported build configuration is provided for CCCL.
+# These checks are only enforced when building CCCL tests, examples, etc. and are not required
+# for users of CCCL.
+
+# The default CXX/CUDA standard to use if none is specified:
+set(_cccl_default_dialect 17)
+
+function(cccl_dev_build_checks)
+  # Similarly, we expect the CXX and CUDA standards to match, if either is set:
+  if (CMAKE_CXX_STANDARD OR CMAKE_CUDA_STANDARD)
+    if (NOT CMAKE_CXX_STANDARD EQUAL CMAKE_CUDA_STANDARD)
+      message(
+        FATAL_ERROR
+        "CCCL developer builds require that CMAKE_CXX_STANDARD matches "
+        "CMAKE_CUDA_STANDARD when either is set:\n"
+        "CMAKE_CXX_STANDARD: ${CMAKE_CXX_STANDARD}\n"
+        "CMAKE_CUDA_STANDARD: ${CMAKE_CUDA_STANDARD}\n"
+        "Rerun cmake with:\n"
+        "\t\"-DCMAKE_CUDA_STANDARD=<std> -DCMAKE_CXX_STANDARD=<std>\"."
+      )
+    endif()
+  else()
+    # Neither is set; initialize to a default of 20.
+    message(
+      VERBOSE
+      "Setting CMAKE_CXX_STANDARD and CMAKE_CUDA_STANDARD to CCCL default of ${_cccl_default_dialect}."
+    )
+    set(CMAKE_CXX_STANDARD ${_cccl_default_dialect})
+    set(CMAKE_CUDA_STANDARD ${_cccl_default_dialect})
+    set(CMAKE_CXX_STANDARD ${CMAKE_CXX_STANDARD} PARENT_SCOPE)
+    set(CMAKE_CUDA_STANDARD ${CMAKE_CUDA_STANDARD} PARENT_SCOPE)
+  endif()
+
+  message(STATUS "CMAKE_CXX_STANDARD: ${CMAKE_CXX_STANDARD}")
+  message(STATUS "CMAKE_CUDA_STANDARD: ${CMAKE_CUDA_STANDARD}")
+endfunction()

cccl_upstream/cmake/CCCLEnsureMetaTargets.cmake

@@ -0,0 +1,52 @@
+# Adds "metatargets" using the target_name or METATARGET_PATH.
+#
+# A metatarget is a custom target that depends on its children targets. For example,
+# a target named foo.bar.baz would create metatargets foo and foo.bar, where
+# foo depends on foo.bar, and foo.bar depends on foo.bar.baz.
+# This allows, for instance, `ninja cudax` to build all cudax.* targets, and `ninja cudax.test`
+# to build all cudax.test.* targets.
+function(cccl_ensure_metatargets target_name)
+  set(options)
+  set(oneValueArgs METATARGET_PATH)
+  set(multiValueArgs)
+  cmake_parse_arguments(
+    _cccl
+    "${options}"
+    "${oneValueArgs}"
+    "${multiValueArgs}"
+    ${ARGN}
+  )
+
+  if (_cccl_UNPARSED_ARGUMENTS)
+    message(FATAL_ERROR "Unrecognized arguments: ${_cccl_UNPARSED_ARGUMENTS}")
+  endif()
+
+  if (NOT DEFINED _cccl_METATARGET_PATH)
+    set(_cccl_METATARGET_PATH ${target_name})
+  endif()
+
+  set(parent_path "")
+  set(current_path "")
+  string(REPLACE "." ";" path_parts "${_cccl_METATARGET_PATH}")
+  foreach (part IN LISTS path_parts)
+    if (current_path STREQUAL "")
+      set(current_path "${part}")
+    else()
+      set(current_path "${current_path}.${part}")
+    endif()
+
+    if (NOT TARGET ${current_path})
+      add_custom_target(${current_path})
+    endif()
+
+    if (NOT parent_path STREQUAL "")
+      add_dependencies(${parent_path} ${current_path})
+    endif()
+
+    set(parent_path ${current_path})
+  endforeach()
+
+  if (NOT target_name STREQUAL current_path)
+    add_dependencies(${current_path} ${target_name})
+  endif()
+endfunction()

cccl_upstream/cmake/CCCLGenerateHeaderTests.cmake

@@ -0,0 +1,242 @@
+# Usage:
+# cccl_generate_header_tests(<target_name> <project_include_path>
+#                            [cccl_configure_target options]
+#                            [LANGUAGE <CXX|CUDA>]
+#                            [HEADER_TEMPLATE <template>]
+#                            [GLOBS <glob1> [glob2 ...]]
+#                            [EXCLUDES <glob1> [glob2 ...]]
+#                            [HEADERS <header1> [header2 ...]]
+#                            [PER_HEADER_DEFINES
+#                               DEFINE <definition> <regex> [<regex> ...]
+#                              [DEFINE <definition> <regex> [<regex> ...]] ...]
+# )
+#
+# Options:
+# target_name: The name of the meta-target that will build this set of header tests.
+# project_include_path: The path to the project's include directory, relative to <CCCL_SOURCE_DIR>.
+# cccl_configure_target options: Options to pass to cccl_configure_target. Must appear before any other named arguments.
+# LANGUAGE: The language to use for the header tests. Defaults to CUDA.
+# HEADER_TEMPLATE: A file that will be used as a template for each header test. The template will be configured for each header.
+# GLOBS: All files that match these globbing patterns will be included in the header tests, unless they also match EXCLUDES.
+# EXCLUDES: Files that match these globbing patterns will be excluded from the header tests.
+# HEADERS: An explicit list of headers to include in the header tests.
+# PER_HEADER_DEFINES: A list of definitions to add to specific headers. Each definition is followed by one or more regexes that match the headers it should be applied to.
+# NO_METATARGETS: If specified, metatargets will not be created for the header test targets.
+#
+# Notes:
+# - The header globs are applied relative to <project_include_path>.
+# - If no HEADER_TEMPLATE is provided, a default template will be used.
+# - The HEADER_TEMPLATE will be configured for each header, with the following variables:
+#   - @header@: The path to the target header, relative to <project_include_path>.
+option(
+  CCCL_COMPILE_TIME_SAVE_PREPROCESSED_TUS
+  "Save preprocessed generated one-include CUDA TUs for compile-time benchmarks."
+  OFF
+)
+option(
+  CCCL_COMPILE_TIME_GENERATE_DEVICE_TIME_TRACES
+  "Emit NVCC device time traces for compile-time benchmarks."
+  OFF
+)
+mark_as_advanced(
+  CCCL_COMPILE_TIME_SAVE_PREPROCESSED_TUS
+  CCCL_COMPILE_TIME_GENERATE_DEVICE_TIME_TRACES
+)
+
+function(cccl_generate_header_tests target_name project_include_path)
+  set(options NO_METATARGETS)
+  set(oneValueArgs LANGUAGE HEADER_TEMPLATE)
+  set(multiValueArgs GLOBS EXCLUDES HEADERS PER_HEADER_DEFINES)
+  cmake_parse_arguments(
+    CGHT
+    "${options}"
+    "${oneValueArgs}"
+    "${multiValueArgs}"
+    ${ARGN}
+  )
+
+  if (CGHT_UNPARSED_ARGUMENTS)
+    message(FATAL_ERROR "Unrecognized arguments: ${CGHT_UNPARSED_ARGUMENTS}")
+  endif()
+
+  # Setup defaults
+  if (NOT DEFINED CGHT_LANGUAGE)
+    set(CGHT_LANGUAGE CUDA)
+  endif()
+
+  if (NOT DEFINED CGHT_HEADER_TEMPLATE)
+    set(CGHT_HEADER_TEMPLATE "${CCCL_SOURCE_DIR}/cmake/header_test.cu.in")
+  endif()
+
+  # Derived vars:
+  if (${CGHT_LANGUAGE} STREQUAL "C")
+    set(extension "c")
+  elseif (${CGHT_LANGUAGE} STREQUAL "CXX")
+    set(extension "cpp")
+  elseif (${CGHT_LANGUAGE} STREQUAL "CUDA")
+    set(extension "cu")
+  else()
+    message(FATAL_ERROR "Unsupported language: ${CGHT_LANGUAGE}")
+  endif()
+
+  set(cccl_configure_target_options ${CGHT_UNPARSED_ARGUMENTS})
+  set(base_path "${CCCL_SOURCE_DIR}/${project_include_path}")
+
+  # Prepend the basepath to all globbing expressions:
+  if (DEFINED CGHT_GLOBS)
+    set(globs)
+    foreach (glob IN LISTS CGHT_GLOBS)
+      list(APPEND globs "${base_path}/${glob}")
+    endforeach()
+    set(CGHT_GLOBS ${globs})
+  endif()
+  if (DEFINED CGHT_EXCLUDES)
+    set(excludes)
+    foreach (exclude IN LISTS CGHT_EXCLUDES)
+      list(APPEND excludes "${base_path}/${exclude}")
+    endforeach()
+    set(CGHT_EXCLUDES ${excludes})
+  endif()
+
+  # Determine header list
+  set(headers)
+
+  # Add globs:
+  if (DEFINED CGHT_GLOBS)
+    file(
+      GLOB_RECURSE headers
+      RELATIVE "${base_path}"
+      CONFIGURE_DEPENDS
+      ${CGHT_GLOBS}
+    )
+  endif()
+
+  # Remove excludes:
+  if (DEFINED CGHT_EXCLUDES)
+    file(
+      GLOB_RECURSE header_excludes
+      RELATIVE "${base_path}"
+      CONFIGURE_DEPENDS
+      ${CGHT_EXCLUDES}
+    )
+    list(REMOVE_ITEM headers ${header_excludes})
+  endif()
+
+  # Add explicit headers:
+  if (DEFINED CGHT_HEADERS)
+    list(APPEND headers ${CGHT_HEADERS})
+  endif()
+
+  # Cleanup:
+  list(REMOVE_DUPLICATES headers)
+
+  # Helper function for applying per-header defines:
+  # header: The original header filepath
+  # src: The generated source file for the header test
+  function(cght_apply_per_header_defines header src)
+    if (NOT DEFINED CGHT_PER_HEADER_DEFINES)
+      return()
+    endif()
+    set(current_definition)
+    foreach (item IN LISTS CGHT_PER_HEADER_DEFINES)
+      if (item STREQUAL "DEFINE")
+        # New definition
+        set(current_definition)
+      elseif (NOT current_definition)
+        # First item after DEFINE is the definition
+        set(current_definition "${item}")
+      else()
+        # Subsequent items are regexes to match against the header
+        if (header MATCHES ${item})
+          set_property(
+            SOURCE "${src}"
+            APPEND
+            PROPERTY COMPILE_DEFINITIONS "${current_definition}"
+          )
+        endif()
+      endif()
+    endforeach()
+  endfunction()
+
+  # Configure header templates:
+  set(header_srcs)
+  foreach (header IN LISTS headers)
+    set(
+      header_src
+      "${CMAKE_CURRENT_BINARY_DIR}/headers/${target_name}/${header}.${extension}"
+    )
+    configure_file("${CGHT_HEADER_TEMPLATE}" "${header_src}" @ONLY)
+    cght_apply_per_header_defines("${header}" "${header_src}")
+
+    # Compile-time benchmark workflows can ask generated one-include CUDA TUs to
+    # preserve preprocessed artifacts and/or emit NVCC device time traces.
+    if (
+      (
+        CCCL_COMPILE_TIME_SAVE_PREPROCESSED_TUS
+        OR CCCL_COMPILE_TIME_GENERATE_DEVICE_TIME_TRACES
+      )
+      AND CGHT_LANGUAGE STREQUAL "CUDA"
+    )
+      get_filename_component(header_src_dir "${header_src}" DIRECTORY)
+      if ("${CMAKE_CUDA_COMPILER_ID}" STREQUAL "NVIDIA")
+        if (CCCL_COMPILE_TIME_SAVE_PREPROCESSED_TUS)
+          set_property(
+            SOURCE "${header_src}"
+            APPEND
+            PROPERTY COMPILE_OPTIONS "--keep" "--keep-dir=${header_src_dir}"
+          )
+        endif()
+        if (CCCL_COMPILE_TIME_GENERATE_DEVICE_TIME_TRACES)
+          set(trace_id "${header}")
+          string(REPLACE "/" "__" trace_id "${trace_id}")
+          string(REPLACE "." "_" trace_id "${trace_id}")
+          set(
+            trace_dir
+            "${CMAKE_BINARY_DIR}/compile_time/raw_traces/${target_name}"
+          )
+          file(MAKE_DIRECTORY "${trace_dir}")
+          set_property(
+            SOURCE "${header_src}"
+            APPEND
+            PROPERTY
+              COMPILE_OPTIONS "--fdevice-time-trace=${trace_dir}/${trace_id}"
+          )
+        endif()
+      elseif (
+        CCCL_COMPILE_TIME_SAVE_PREPROCESSED_TUS
+        AND "${CMAKE_CUDA_COMPILER_ID}" STREQUAL "Clang"
+      )
+        set_property(
+          SOURCE "${header_src}"
+          APPEND
+          PROPERTY COMPILE_OPTIONS "-save-temps=obj"
+        )
+      endif()
+    endif()
+    list(APPEND header_srcs "${header_src}")
+  endforeach()
+
+  # Object library that compiles each header:
+  add_library(${target_name} OBJECT ${header_srcs})
+  cccl_configure_target(${target_name} ${cccl_configure_target_options})
+  if (NOT CGHT_NO_METATARGETS)
+    cccl_ensure_metatargets(${target_name})
+  endif()
+
+  # Check that all functions in headers are either template functions or inline:
+  set(link_target ${target_name}.link_check)
+  cccl_add_executable(
+    ${link_target}
+    SOURCES "${CCCL_SOURCE_DIR}/cmake/link_check_main.cpp"
+    NO_METATARGETS
+  )
+  # Linking both ${target_name} and $<TARGET_OBJECTS:${target_name}> forces CMake to
+  # link the same objects twice. The compiler will complain about duplicate symbols if
+  # any functions are missing inline markup.
+  target_link_libraries(
+    ${link_target}
+    PRIVATE #
+      ${target_name}
+      $<TARGET_OBJECTS:${target_name}>
+  )
+endfunction()

cccl_upstream/cmake/CCCLGetDependencies.cmake

@@ -0,0 +1,127 @@
+set(_cccl_cpm_file "${CMAKE_CURRENT_LIST_DIR}/CPM.cmake")
+set(_cccl_find_module_dir "${CMAKE_CURRENT_LIST_DIR}/find_modules")
+
+macro(cccl_get_boost)
+  include("${_cccl_cpm_file}")
+  CPMAddPackage(
+    NAME Boost
+    GITHUB_REPOSITORY boostorg/boost
+    GIT_TAG "boost-1.83.0"
+    EXCLUDE_FROM_ALL TRUE
+    SYSTEM TRUE
+    GIT_SHALLOW TRUE
+    # Boost requests compatibility with obsolete CMake versions. Disable warning:
+    OPTIONS "CMAKE_POLICY_VERSION_MINIMUM 3.5"
+  )
+endmacro()
+
+# The CCCL Catch2Helper library:
+macro(cccl_get_c2h)
+  if (NOT TARGET cccl.c2h)
+    add_subdirectory("${CCCL_SOURCE_DIR}/c2h" "${CCCL_BINARY_DIR}/c2h")
+  endif()
+endmacro()
+
+macro(cccl_get_catch2)
+  include("${_cccl_cpm_file}")
+  CPMAddPackage("gh:catchorg/Catch2@3.12.0")
+endmacro()
+
+macro(cccl_get_cccl)
+  find_package(
+    CCCL
+    CONFIG
+    REQUIRED
+    NO_DEFAULT_PATH # Only check the explicit HINTS below:
+    HINTS "${CCCL_SOURCE_DIR}/lib/cmake/cccl/"
+  )
+endmacro()
+
+macro(cccl_get_cub)
+  find_package(
+    CUB
+    CONFIG
+    REQUIRED
+    NO_DEFAULT_PATH # Only check the explicit HINTS below:
+    HINTS "${CCCL_SOURCE_DIR}/lib/cmake/cub/"
+  )
+endmacro()
+
+macro(cccl_get_cudatoolkit)
+  find_package(CUDAToolkit REQUIRED)
+endmacro()
+
+macro(cccl_get_cudax)
+  find_package(
+    cudax
+    CONFIG
+    REQUIRED
+    NO_DEFAULT_PATH # Only check the explicit HINTS below:
+    HINTS "${CCCL_SOURCE_DIR}/lib/cmake/cudax/"
+  )
+endmacro()
+
+macro(cccl_get_dlpack)
+  include("${_cccl_cpm_file}")
+  CPMAddPackage("gh:dmlc/dlpack#v1.2")
+endmacro()
+
+macro(cccl_get_libcudacxx)
+  find_package(
+    libcudacxx
+    CONFIG
+    REQUIRED
+    NO_DEFAULT_PATH # Only check the explicit HINTS below:
+    HINTS "${CCCL_SOURCE_DIR}/lib/cmake/libcudacxx/"
+  )
+endmacro()
+
+set(
+  CCCL_NVBENCH_SHA
+  "56d552687e6a462a812d6f046f5a85a07f13c9f3"
+  CACHE STRING
+  "SHA/tag to use for CCCL's NVBench."
+)
+mark_as_advanced(CCCL_NVBENCH_SHA)
+macro(cccl_get_nvbench)
+  include("${_cccl_cpm_file}")
+  CPMAddPackage("gh:NVIDIA/nvbench#${CCCL_NVBENCH_SHA}")
+endmacro()
+
+# CCCL-specific NVBench utilities
+macro(cccl_get_nvbench_helper)
+  if (NOT TARGET cccl.nvbench_helper)
+    add_subdirectory(
+      "${CCCL_SOURCE_DIR}/nvbench_helper"
+      "${CCCL_BINARY_DIR}/nvbench_helper"
+    )
+  endif()
+endmacro()
+
+macro(cccl_get_nvtx)
+  include("${_cccl_cpm_file}")
+  CPMAddPackage(
+    NAME NVTX
+    GITHUB_REPOSITORY NVIDIA/NVTX
+    GIT_TAG release-v3
+    DOWNLOAD_ONLY ON
+    SYSTEM ON
+  )
+  include("${NVTX_SOURCE_DIR}/c/nvtxImportedTargets.cmake")
+endmacro()
+
+macro(cccl_get_thrust)
+  find_package(
+    Thrust
+    CONFIG
+    REQUIRED
+    NO_DEFAULT_PATH # Only check the explicit HINTS below:
+    HINTS "${CCCL_SOURCE_DIR}/lib/cmake/thrust/"
+  )
+endmacro()
+
+macro(cccl_get_nccl)
+  list(APPEND CMAKE_MODULE_PATH "${_cccl_find_module_dir}")
+  find_package(NCCL ${ARGN})
+  list(POP_BACK CMAKE_MODULE_PATH)
+endmacro()

cccl_upstream/cmake/CCCLHideThirdPartyOptions.cmake

@@ -0,0 +1,39 @@
+mark_as_advanced(
+  BUILD_TESTING
+  CATCH_BUILD_EXAMPLES
+  CATCH_BUILD_EXTRA_TESTS
+  CATCH_BUILD_STATIC_LIBRARY
+  CATCH_BUILD_TESTING
+  CATCH_ENABLE_COVERAGE
+  CATCH_ENABLE_WERROR
+  CATCH_INSTALL_DOCS
+  CATCH_INSTALL_HELPERS
+  CATCH_USE_VALGRIND
+  CLANG_FORMAT
+  CLANG_TIDY
+  CPM_DONT_CREATE_PACKAGE_LOCK
+  CPM_DONT_UPDATE_MODULE_PATH
+  CPM_DOWNLOAD_ALL
+  CPM_INCLUDE_ALL_IN_PACKAGE_LOCK
+  CPM_LOCAL_PACKAGES_ONLY
+  CPM_SOURCE_CACHE
+  CPM_USE_LOCAL_PACKAGES
+  CPM_USE_NAMED_CACHE_DIRECTORIES
+  CPPCHECK
+  CUB_DIR
+  FETCHCONTENT_BASE_DIR
+  FETCHCONTENT_FULLY_DISCONNECTED
+  FETCHCONTENT_QUIET
+  FETCHCONTENT_SOURCE_DIR_CATCH2
+  FETCHCONTENT_UPDATES_DISCONNECTED
+  FETCHCONTENT_UPDATES_DISCONNECTED_CATCH2
+  LIBCXX_CXX_ABI
+  LIT_EXTRA_ARGS
+  LLVM_DEFAULT_EXTERNAL_LIT
+  LLVM_DEFAULT_TARGET_TRIPLE
+  LLVM_EXTERNAL_LIT
+  LLVM_HOST_TRIPLE
+  LLVM_PATH
+  Thrust_DIR
+  libcudacxx_DIR
+)

cccl_upstream/cmake/CCCLInstallRules.cmake

@@ -0,0 +1,146 @@
+# Bring in CMAKE_INSTALL_* vars
+include(GNUInstallDirs)
+
+# CCCL has no installable binaries, no need to build before installing:
+set(CMAKE_SKIP_INSTALL_ALL_DEPENDENCY TRUE)
+
+# Usage:
+# cccl_generate_install_rules(PROJECT_NAME DEFAULT_ENABLE
+#                             [NO_HEADERS]
+#                             [HEADER_SUBDIR <subdir1> [subdir2 ...]]
+#                             [HEADERS_INCLUDE <pattern1> [pattern2 ...]]
+#                             [HEADERS_EXCLUDE <pattern1> [pattern2 ...]]
+#                             [PACKAGE]
+# )
+#
+# Options:
+# PROJECT_NAME: The case-sensitive name of the project. Used to generate the option flag.
+# DEFAULT_ENABLE: Whether the install rules should be enabled by default.
+# NO_HEADERS: If set, no install rules will be generated for headers.
+# HEADERS_SUBDIRS: If set, a separate install rule will be generated for each subdirectory relative to the project dir.
+#                  If not set, <CCCL_SOURCE_DIR>/<PROJECT_NAME_LOWER>/<PROJECT_NAME_LOWER> will be used.
+# HEADERS_INCLUDE: A list of globbing patterns that match installable header files.
+# HEADERS_EXCLUDE: A list of globbing patterns that match header files to exclude from installation.
+# PACKAGE: If set, install the project's CMake package.
+#
+# Notes:
+# - The generated cache option will be named <PROJECT_NAME>_ENABLE_INSTALL_RULES.
+# - The header globs are applied relative to <CCCL_SOURCE_DIR>/<PROJECT_NAME_LOWER>/<SUBDIR>.
+# - The cmake package is assumed to be located at <CCCL_SOURCE_DIR>/lib/cmake/<PROJECT_NAME_LOWER>.
+# - If a <PROJECT_NAME_LOWER>-header-search.cmake.in file exists in the CMake package directory,
+#   it will be configured and installed.
+#
+function(cccl_generate_install_rules project_name enable_rules_by_default)
+  set(options PACKAGE NO_HEADERS)
+  set(oneValueArgs)
+  set(multiValueArgs HEADERS_SUBDIRS HEADERS_INCLUDE HEADERS_EXCLUDE)
+  cmake_parse_arguments(
+    CGIR
+    "${options}"
+    "${oneValueArgs}"
+    "${multiValueArgs}"
+    ${ARGN}
+  )
+
+  string(TOLOWER ${project_name} project_name_lower)
+  set(project_source_dir "${CCCL_SOURCE_DIR}/${project_name_lower}")
+  set(header_dest_dir "${CMAKE_INSTALL_INCLUDEDIR}")
+  set(package_source_dir "${CCCL_SOURCE_DIR}/lib/cmake/${project_name_lower}")
+  set(package_dest_dir "${CMAKE_INSTALL_LIBDIR}/cmake/")
+  set(
+    header_search_template
+    "${package_source_dir}/${project_name_lower}-header-search.cmake.in"
+  )
+  set(
+    header_search_temporary
+    "${CCCL_BINARY_DIR}/${project_name_lower}-header-search.cmake"
+  )
+
+  if (NOT DEFINED CGIR_HEADERS_SUBDIRS)
+    set(CGIR_HEADERS_SUBDIRS "${project_name_lower}")
+  endif()
+
+  set(flag_name ${project_name}_ENABLE_INSTALL_RULES)
+  option(
+    ${flag_name}
+    "Enable installation of ${project_name} files."
+    ${enable_rules_by_default}
+  )
+  if (${flag_name})
+    # Headers:
+    if (NOT CGIR_NO_HEADERS)
+      foreach (subdir IN LISTS CGIR_HEADERS_SUBDIRS)
+        set(header_globs)
+        if (DEFINED CGIR_HEADERS_INCLUDE OR DEFINED CGIR_HEADERS_EXCLUDE)
+          set(header_globs "FILES_MATCHING")
+
+          foreach (header_glob IN LISTS CGIR_HEADERS_INCLUDE)
+            list(APPEND header_globs "PATTERN" "${header_glob}")
+          endforeach()
+
+          foreach (header_glob IN LISTS CGIR_HEADERS_EXCLUDE)
+            list(APPEND header_globs "PATTERN" "${header_glob}" "EXCLUDE")
+          endforeach()
+        endif()
+
+        install(
+          DIRECTORY "${project_source_dir}/${subdir}"
+          DESTINATION "${header_dest_dir}"
+          ${header_globs}
+        )
+      endforeach()
+    endif()
+
+    # CMake package:
+    install(
+      DIRECTORY "${package_source_dir}"
+      DESTINATION "${package_dest_dir}"
+      REGEX .*header-search.cmake.* EXCLUDE
+    )
+
+    # Header search infra:
+    if (EXISTS "${header_search_template}")
+      # Need to configure a file to store the infix specified in
+      # CMAKE_INSTALL_INCLUDEDIR since it can be defined by the user
+      set(_CCCL_RELATIVE_LIBDIR "${CMAKE_INSTALL_LIBDIR}")
+      if (_CCCL_RELATIVE_LIBDIR MATCHES "^${CMAKE_INSTALL_PREFIX}")
+        # libdir is an abs string that starts with prefix
+        string(LENGTH "${CMAKE_INSTALL_PREFIX}" to_remove)
+        string(SUBSTRING "${_CCCL_RELATIVE_LIBDIR}" ${to_remove} -1 relative)
+        # remove any leading "/""
+        string(REGEX REPLACE "^/(.)" "\\1" _CCCL_RELATIVE_LIBDIR "${relative}")
+      elseif (_CCCL_RELATIVE_LIBDIR MATCHES "^/")
+        message(
+          FATAL_ERROR
+          "CMAKE_INSTALL_LIBDIR ('${CMAKE_INSTALL_LIBDIR}') must be a relative path or an absolute path under CMAKE_INSTALL_PREFIX ('${CMAKE_INSTALL_PREFIX}')"
+        )
+      endif()
+      set(
+        install_location
+        "${_CCCL_RELATIVE_LIBDIR}/cmake/${project_name_lower}"
+      )
+
+      # Transform to a list of directories, replace each directory with "../"
+      # and convert back to a string
+      string(REGEX REPLACE "/" ";" from_install_prefix "${install_location}")
+      list(TRANSFORM from_install_prefix REPLACE ".+" "../")
+      list(JOIN from_install_prefix "" from_install_prefix)
+
+      configure_file(
+        "${header_search_template}"
+        "${header_search_temporary}"
+        @ONLY
+      )
+      install(
+        FILES "${header_search_temporary}"
+        DESTINATION "${install_location}"
+      )
+    endif()
+  endif()
+endfunction()
+
+include("${CMAKE_CURRENT_LIST_DIR}/install/cccl.cmake")
+include("${CMAKE_CURRENT_LIST_DIR}/install/cub.cmake")
+include("${CMAKE_CURRENT_LIST_DIR}/install/cudax.cmake")
+include("${CMAKE_CURRENT_LIST_DIR}/install/libcudacxx.cmake")
+include("${CMAKE_CURRENT_LIST_DIR}/install/thrust.cmake")

cccl_upstream/cmake/CCCLTestParams.cmake

@@ -0,0 +1,110 @@
+# Further documentation and examples are provided in docs/cccl/development/testing.rst.
+
+# The function below reads the filepath `src`, extracts the %PARAM% comments,
+# and fills `all_variant_labels_var` with a list of `label1_value1.label2_value2...`
+# strings, and puts the corresponding `DEFINITION=value1:DEFINITION=value2`
+# entries into `all_variant_defs_var`.
+function(
+  cccl_parse_variant_params
+  src
+  num_variants_var
+  all_variant_labels_var
+  all_variant_defs_var
+)
+  file(READ "${src}" file_data)
+  set(param_regex "//[ ]+%PARAM%[ ]+([^ ]+)[ ]+([^ ]+)[ ]+([^\n]*)")
+
+  string(REGEX MATCHALL "${param_regex}" matches "${file_data}")
+
+  set(variant_labels)
+  set(variant_defs)
+
+  foreach (match IN LISTS matches)
+    string(REGEX MATCH "${param_regex}" unused "${match}")
+
+    set(def ${CMAKE_MATCH_1})
+    set(label ${CMAKE_MATCH_2})
+    set(values "${CMAKE_MATCH_3}")
+    string(REPLACE ":" ";" values "${values}")
+
+    # Build lists of test name suffixes (labels) and preprocessor definitions
+    # (defs) containing the cartesian product of all param values:
+    if (NOT variant_labels)
+      foreach (value IN LISTS values)
+        list(APPEND variant_labels ${label}_${value})
+      endforeach()
+    else()
+      set(tmp_labels)
+      foreach (old_label IN LISTS variant_labels)
+        foreach (value IN LISTS values)
+          list(APPEND tmp_labels ${old_label}.${label}_${value})
+        endforeach()
+      endforeach()
+      set(variant_labels "${tmp_labels}")
+    endif()
+
+    if (NOT variant_defs)
+      foreach (value IN LISTS values)
+        list(APPEND variant_defs ${def}=${value})
+      endforeach()
+    else()
+      set(tmp_defs)
+      foreach (old_def IN LISTS variant_defs)
+        foreach (value IN LISTS values)
+          list(APPEND tmp_defs ${old_def}:${def}=${value})
+        endforeach()
+      endforeach()
+      set(variant_defs "${tmp_defs}")
+    endif()
+  endforeach()
+
+  list(LENGTH variant_labels num_variants)
+
+  set(${num_variants_var} "${num_variants}" PARENT_SCOPE)
+  set(${all_variant_labels_var} "${variant_labels}" PARENT_SCOPE)
+  set(${all_variant_defs_var} "${variant_defs}" PARENT_SCOPE)
+endfunction()
+
+# Extracts the variant label and definitions for the given variant index and prepares them for use.
+function(
+  cccl_get_variant_data
+  all_variant_labels_var
+  all_variant_defs_var
+  var_idx
+  label_var
+  defs_var
+)
+  list(GET ${all_variant_labels_var} ${var_idx} label)
+  list(GET ${all_variant_defs_var} ${var_idx} defs)
+  string(REPLACE ":" ";" defs "${defs}")
+  list(APPEND defs "VAR_IDX=${var_idx}")
+  set(${label_var} "${label}" PARENT_SCOPE)
+  set(${defs_var} "${defs}" PARENT_SCOPE)
+endfunction()
+
+# Logs the detected variant info to CMake's VERBOSE output stream.
+function(
+  cccl_log_variant_params
+  name_base
+  num_variants
+  all_variant_labels_var
+  all_variant_defs_var
+)
+  # Verbose output:
+  if (num_variants GREATER 0)
+    message(VERBOSE "Detected ${num_variants} variants of '${name_base}':")
+
+    # Subtract 1 to support the inclusive endpoint of foreach(...RANGE...):
+    math(EXPR range_end "${num_variants} - 1")
+    foreach (var_idx RANGE ${range_end})
+      cccl_get_variant_data(
+        ${all_variant_labels_var}
+        ${all_variant_defs_var}
+        ${var_idx}
+        label
+        defs
+      )
+      message(VERBOSE "  ${var_idx}: ${label} ${defs}")
+    endforeach()
+  endif()
+endfunction()

cccl_upstream/cmake/CCCLUtilities.cmake

@@ -0,0 +1,291 @@
+# SPDX-FileCopyrightText: Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Passes all args directly to execute_process while setting up the following
+# results variables and propagating them to the caller's scope:
+#
+# - cccl_process_exit_code
+# - cccl_process_stdout
+# - cccl_process_stderr
+#
+# If the command
+# is not successful (e.g. the last command does not return zero), a non-fatal
+# warning is printed.
+function(cccl_execute_non_fatal_process)
+  # Skip parsing this function's signature -- it is handled by .gersemi/ext/cccl.py.
+  # gersemi: ignore
+
+  execute_process(
+    ${ARGN}
+    RESULT_VARIABLE cccl_process_exit_code
+    OUTPUT_VARIABLE cccl_process_stdout
+    ERROR_VARIABLE cccl_process_stderr
+  )
+
+  if (NOT cccl_process_exit_code EQUAL 0)
+    message(
+      WARNING
+      "execute_process failed with non-zero exit code: ${cccl_process_exit_code}\n"
+      "${ARGN}\n"
+      "stdout:\n${cccl_process_stdout}\n"
+      "stderr:\n${cccl_process_stderr}\n"
+    )
+  endif()
+
+  set(cccl_process_exit_code "${cccl_process_exit_code}" PARENT_SCOPE)
+  set(cccl_process_stdout "${cccl_process_stdout}" PARENT_SCOPE)
+  set(cccl_process_stderr "${cccl_process_stderr}" PARENT_SCOPE)
+endfunction()
+
+# Add a build-and-test CTest.
+# - full_test_name_var will be set to the full name of the test.
+# - name_prefix is the prefix of the test's name (e.g. `cccl.test.cmake`)
+# - subdir is the relative path to the test project directory.
+# - test_id is used to generate a unique name for this test, allowing the
+#   subdir to be reused.
+# - CTEST_COMMAND is the command to use for running CTest [optional]
+# - Any additional args will be passed to the project configure step.
+function(cccl_add_compile_test full_test_name_var name_prefix subdir test_id)
+  set(options)
+  set(oneValueArgs CTEST_COMMAND)
+  set(multiValueArgs)
+  cmake_parse_arguments(
+    cccl_compile_test
+    "${options}"
+    "${oneValueArgs}"
+    "${multiValueArgs}"
+    ${ARGN}
+  )
+
+  if (NOT DEFINED cccl_compile_test_CTEST_COMMAND)
+    set(cccl_compile_test_CTEST_COMMAND "${CMAKE_CTEST_COMMAND}")
+  endif()
+
+  set(test_name ${name_prefix}.${subdir}.${test_id})
+  set(src_dir "${CMAKE_CURRENT_SOURCE_DIR}/${subdir}")
+  set(build_dir "${CMAKE_CURRENT_BINARY_DIR}/${subdir}/${test_id}")
+  add_test(
+    NAME ${test_name}
+    # gersemi: off
+    COMMAND
+      "${cccl_compile_test_CTEST_COMMAND}"
+        --build-and-test "${src_dir}" "${build_dir}"
+        --build-generator "${CMAKE_GENERATOR}"
+        --build-options ${cccl_compile_test_UNPARSED_ARGUMENTS}
+        --test-command "${cccl_compile_test_CTEST_COMMAND}" --output-on-failure
+    # gersemi: on
+  )
+  set(${full_test_name_var} ${test_name} PARENT_SCOPE)
+endfunction()
+
+# cccl_add_xfail_compile_target_test(
+#   <target_name>
+#   [TEST_NAME <test_name>]
+#   [ERROR_REGEX <regex>]
+#   [SOURCE_FILE <source_file>]
+#   [ERROR_REGEX_LABEL <error_string>]
+#   [ERROR_NUMBER <error_number>]
+#   [ERROR_NUMBER_TARGET_NAME_REGEX <regex>]
+# )
+#
+# Given a configured build target that is expected to fail to compile:
+# - Mark the target as excluded from the `all` target.
+# - Create a CTest test that compiles the target. If TEST_NAME is provided, it is used.
+#   Otherwise, the target_name is used as the test name.
+# - When the test runs, it passes if exactly one of the following conditions is met:
+#   - A provided / detected error regex matches the compilation output, ignoring exit code.
+#   - No error regex is provided / detected, and the compilation fails.
+#
+# An error regex may be explicitly provided via ERROR_REGEX, or it may be
+# detected by scanning the SOURCE_FILE for a specially formatted comment.
+#
+# If ERROR_REGEX_LABEL is provided, the SOURCE_FILE will read, looking for a comment of the form:
+#
+# // <ERROR_REGEX_LABEL> {{"error_regex"}}
+#
+# An error number may be appended to the ERROR_REGEX_LABEL in the comment:
+#
+# // <ERROR_REGEX_LABEL>-<error_number> {{"error_regex"}}
+#
+# If ERROR_NUMBER_TARGET_NAME_REGEX is specified, the regex is used to capture
+# the error_number from the target name. If target_name is
+# "cccl.test.my_test.err_5.foo_3" and ERROR_NUMBER_TARGET_NAME_REGEX is
+# "\\.err_([0-9]+)", the captured error number "5."
+#
+# // <ERROR_REGEX_LABEL>-<captured_error_number> {{"error_regex"}}
+#
+# If ERROR_NUMBER is provided, ERROR_NUMBER_TARGET_NAME_REGEX is ignored.
+# If ERROR_NUMBER_TARGET_NAME_REGEX is provided but does not match, a plain ERROR_REGEX_LABEL is used.
+#
+# If both SOURCE_FILE and ERROR_REGEX_LABEL are provided, the source file will be added to the
+# current directory's CMAKE_CONFIGURE_DEPENDS to ensure that changes to the file will re-trigger CMake.
+function(cccl_add_xfail_compile_target_test target_name)
+  set(options)
+  set(
+    oneValueArgs
+    TEST_NAME
+    ERROR_REGEX
+    SOURCE_FILE
+    ERROR_REGEX_LABEL
+    ERROR_NUMBER
+    ERROR_NUMBER_TARGET_NAME_REGEX
+  )
+  set(multiValueArgs)
+  cmake_parse_arguments(
+    cccl_xfail
+    "${options}"
+    "${oneValueArgs}"
+    "${multiValueArgs}"
+    ${ARGN}
+  )
+
+  if (cccl_xfail_UNPARSED_ARGUMENTS)
+    message(FATAL_ERROR "Unparsed arguments: ${cccl_xfail_UNPARSED_ARGUMENTS}")
+  endif()
+
+  set(test_name "${target_name}")
+  if (DEFINED cccl_xfail_TEST_NAME)
+    set(test_name "${cccl_xfail_TEST_NAME}")
+  endif()
+
+  set(regex)
+  if (DEFINED cccl_xfail_ERROR_REGEX)
+    set(regex "${cccl_xfail_ERROR_REGEX}")
+  elseif (
+    DEFINED cccl_xfail_SOURCE_FILE
+    AND DEFINED cccl_xfail_ERROR_REGEX_LABEL
+  )
+    get_filename_component(src_absolute "${cccl_xfail_SOURCE_FILE}" ABSOLUTE)
+    set(error_label_regex "${cccl_xfail_ERROR_REGEX_LABEL}")
+
+    # Cache all error label matches (with and without error numbers) as global properties.
+    # This avoids re-reading and re-parsing the source file multiple times if multiple
+    # tests are added for the same source file. Properties are used instead of cache variables
+    # to ensure that the source is not cached in between CMake executions.
+    string(MD5 source_filename_md5 "${src_absolute}")
+    set(error_cache_property "_cccl_xfail_error_cache_${source_filename_md5}")
+    get_property(error_cache_set GLOBAL PROPERTY "${error_cache_property}" SET)
+    if (error_cache_set)
+      get_property(error_cache GLOBAL PROPERTY "${error_cache_property}")
+    else()
+      file(READ "${src_absolute}" source_contents)
+      string(
+        REGEX MATCHALL
+        "//[ \t]*${error_label_regex}(-[0-9]+)?[ \t]*{{\"([^\"]+)\"}}"
+        error_cache
+        "${source_contents}"
+      )
+      set_property(GLOBAL PROPERTY "${error_cache_property}" "${error_cache}")
+    endif()
+
+    # Changes to the source file should re-run CMake to pick-up new error specs:
+    set_property(
+      DIRECTORY
+      APPEND
+      PROPERTY CMAKE_CONFIGURE_DEPENDS "${src_absolute}"
+    )
+
+    set(error_number)
+    if (DEFINED cccl_xfail_ERROR_NUMBER)
+      set(error_number "${cccl_xfail_ERROR_NUMBER}")
+    elseif (DEFINED cccl_xfail_ERROR_NUMBER_TARGET_NAME_REGEX)
+      string(
+        REGEX MATCH
+        "${cccl_xfail_ERROR_NUMBER_TARGET_NAME_REGEX}"
+        matched
+        ${target_name}
+      )
+      if (matched)
+        set(error_number "${CMAKE_MATCH_1}")
+      endif()
+    endif()
+
+    # Look for a labeled error with the specific error number.
+    if (NOT "${error_number}" STREQUAL "") # Check strings to allow "0"
+      string(
+        REGEX MATCH
+        "//[ \t]*${error_label_regex}-${error_number}[ \t]*{{\"([^\"]+)\"}}"
+        matched
+        "${error_cache}"
+      )
+      if (matched)
+        set(regex "${CMAKE_MATCH_1}")
+      endif()
+    endif()
+
+    if (NOT regex)
+      # Look for a labeled error without an error number.
+      string(
+        REGEX MATCH
+        "//[ \t]*${error_label_regex}[ \t]*{{\"([^\"]+)\"}}"
+        matched
+        "${error_cache}"
+      )
+      if (matched)
+        set(regex "${CMAKE_MATCH_1}")
+      endif()
+    endif()
+  endif()
+
+  message(VERBOSE "CCCL: Adding XFAIL test: ${test_name}")
+  if (regex)
+    message(VERBOSE "CCCL:   with expected regex: '${regex}'")
+  endif()
+
+  set_target_properties(${test_target} PROPERTIES EXCLUDE_FROM_ALL true)
+
+  # The same target may be reused for multiple tests, and the output file
+  # may exist if using a regex to check for warnings. Add a setup fixture to
+  # delete the output file before each test run.
+  if (NOT TEST ${target_name}.clean)
+    add_test(
+      NAME ${target_name}.clean
+      # gersemi: off
+      COMMAND
+        "${CMAKE_COMMAND}" -E rm -f
+          "$<TARGET_FILE:${target_name}>"
+          "$<TARGET_OBJECTS:${target_name}>"
+      # gersemi: on
+    )
+    set_tests_properties(
+      ${test_name}.clean
+      PROPERTIES FIXTURES_SETUP ${target_name}.clean
+    )
+  endif()
+
+  add_test(
+    NAME ${test_name}
+    # gersemi: off
+    COMMAND
+      "${CMAKE_COMMAND}"
+        --build "${CMAKE_BINARY_DIR}"
+        --target ${test_target}
+        --config $<CONFIGURATION>
+    # gersemi: on
+  )
+  set_tests_properties(
+    ${test_name}
+    PROPERTIES FIXTURES_CLEANUP ${target_name}.clean
+  )
+
+  if (regex)
+    set_tests_properties(
+      ${test_name}
+      PROPERTIES PASS_REGULAR_EXPRESSION "${regex}"
+    )
+  else()
+    set_tests_properties(${test_name} PROPERTIES WILL_FAIL true)
+  endif()
+endfunction()

cccl_upstream/cmake/PrintCTestRunTimes.cmake

@@ -0,0 +1,135 @@
+## This CMake script parses the output of ctest and prints a formatted list
+## of individual test runtimes, sorted longest first.
+##
+## ctest > ctest_log
+## cmake -DLOGFILE=ctest_log \
+##       -DMINSEC=10 \
+##       -P PrintCTestRunTimes.cmake
+##
+################################################################################
+
+cmake_minimum_required(VERSION 3.15)
+
+# Prepend the string with "0" until the string length equals the specified width
+function(pad_string_with_zeros string_var width)
+  # gersemi: ignore
+  set(local_string "${${string_var}}")
+  string(LENGTH "${local_string}" size)
+  while(size LESS width)
+    string(PREPEND local_string "0")
+    string(LENGTH "${local_string}" size)
+  endwhile()
+  set(${string_var} "${local_string}" PARENT_SCOPE)
+endfunction()
+
+################################################################################
+
+if (NOT LOGFILE)
+  message(FATAL_ERROR "Missing -DLOGFILE=<ctest output> argument.")
+endif()
+
+if (NOT DEFINED MINSEC)
+  set(MINSEC 10)
+endif()
+
+set(num_below_thresh 0)
+
+# Check if logfile exists
+if (NOT EXISTS "${LOGFILE}")
+  message(FATAL_ERROR "LOGFILE does not exist ('${LOGFILE}').")
+endif()
+
+# gersemi: off
+string(JOIN "" regex
+  "[0-9]+/[0-9]+[ ]+Test[ ]+#"
+  "([0-9]+)"                        # Test ID
+  ":[ ]+"
+  "([^ ]+)"                         # Test Name
+  "[ ]*\\.+[ ]*\\**[ ]*"
+  "([^ ]+)"                         # Result
+  "[ ]+"
+  "([0-9]+)"                        # Seconds
+  "\\.[0-9]+[ ]+sec"
+)
+# gersemi: on
+
+message(DEBUG "LOGFILE: ${LOGFILE}")
+message(DEBUG "MINSEC: ${MINSEC}")
+message(DEBUG "regex: ${regex}")
+
+# Read the logfile and generate a map / keylist
+set(keys)
+file(STRINGS "${LOGFILE}" lines)
+foreach (line ${lines})
+  # Parse each build time
+  string(REGEX MATCH "${regex}" _DUMMY "${line}")
+
+  if (CMAKE_MATCH_COUNT EQUAL 4)
+    # gersemi: off
+    set(test_id      "${CMAKE_MATCH_1}")
+    set(test_name    "${CMAKE_MATCH_2}")
+    set(test_result  "${CMAKE_MATCH_3}")
+    set(tmp          "${CMAKE_MATCH_4}") # floor(runtime_seconds)
+    # gersemi: on
+
+    if (tmp LESS MINSEC)
+      math(EXPR num_below_thresh "${num_below_thresh} + 1")
+      continue()
+    endif()
+
+    # Compute human readable time
+    # gersemi: off
+    math(EXPR days         "${tmp} / (60 * 60 * 24)")
+    math(EXPR tmp          "${tmp} - (${days} * 60 * 60 * 24)")
+    math(EXPR hours        "${tmp} / (60 * 60)")
+    math(EXPR tmp          "${tmp} - (${hours} * 60 * 60)")
+    math(EXPR minutes      "${tmp} / (60)")
+    math(EXPR tmp          "${tmp} - (${minutes} * 60)")
+    math(EXPR seconds      "${tmp}")
+    # gersemi: on
+
+    # Format time components
+    pad_string_with_zeros(days 3)
+    pad_string_with_zeros(hours 2)
+    pad_string_with_zeros(minutes 2)
+    pad_string_with_zeros(seconds 2)
+
+    # Construct table entry
+    # Later values in the file for the same command overwrite earlier entries
+    string(MAKE_C_IDENTIFIER "${test_id}" key)
+    string(
+      JOIN " | "
+      ENTRY_${key}
+      "${days}d ${hours}h ${minutes}m ${seconds}s"
+      "${test_result}"
+      "${test_id}: ${test_name}"
+    )
+
+    # Record the key:
+    list(APPEND keys "${key}")
+  endif()
+endforeach()
+
+list(REMOVE_DUPLICATES keys)
+
+# Build the entry list:
+set(entries)
+foreach (key ${keys})
+  list(APPEND entries "${ENTRY_${key}}")
+endforeach()
+
+if (NOT entries)
+  message(STATUS "LOGFILE contained no test times ('${LOGFILE}').")
+endif()
+
+# Sort in descending order:
+list(SORT entries ORDER DESCENDING)
+
+# Dump table:
+foreach (entry ${entries})
+  message(STATUS ${entry})
+endforeach()
+
+if (num_below_thresh GREATER 0)
+  message(STATUS "${num_below_thresh} additional tests took < ${MINSEC}s each.")
+endif()

cccl_upstream/cmake/PrintNinjaBuildTimes.cmake

@@ -0,0 +1,108 @@
+## This CMake script parses a .ninja_log file (LOGFILE) and prints a list of
+## build/link times, sorted longest first.
+##
+## cmake -DLOGFILE=<.ninja_log file> \
+##       -P PrintNinjaBuildTimes.cmake
+##
+## If LOGFILE is omitted, the current directory's .ninja_log file is used.
+################################################################################
+
+cmake_minimum_required(VERSION 3.15)
+
+# Prepend the string with "0" until the string length equals the specified width
+function(pad_string_with_zeros string_var width)
+  # gersemi: ignore
+  set(local_string "${${string_var}}")
+  string(LENGTH "${local_string}" size)
+  while(size LESS width)
+    string(PREPEND local_string "0")
+    string(LENGTH "${local_string}" size)
+  endwhile()
+  set(${string_var} "${local_string}" PARENT_SCOPE)
+endfunction()
+
+################################################################################
+
+if (NOT LOGFILE)
+  set(LOGFILE ".ninja_log")
+endif()
+
+# Check if logfile exists
+if (NOT EXISTS "${LOGFILE}")
+  message(FATAL_ERROR "LOGFILE does not exist ('${LOGFILE}').")
+endif()
+
+# Read the logfile and generate a map / keylist
+set(keys)
+file(STRINGS "${LOGFILE}" lines)
+foreach (line ${lines})
+  # Parse each build time
+  string(
+    REGEX MATCH
+    "^([0-9]+)\t([0-9]+)\t[0-9]+\t([^\t]+)+\t[0-9a-fA-F]+$"
+    _DUMMY
+    "${line}"
+  )
+
+  if (CMAKE_MATCH_COUNT EQUAL 3)
+    set(start_ms ${CMAKE_MATCH_1})
+    set(end_ms ${CMAKE_MATCH_2})
+    set(command "${CMAKE_MATCH_3}")
+    math(EXPR runtime_ms "${end_ms} - ${start_ms}")
+
+    # Compute human readable time
+    # gersemi: off
+    math(EXPR days         "${runtime_ms} / (1000 * 60 * 60 * 24)")
+    math(EXPR runtime_ms   "${runtime_ms} - (${days} * 1000 * 60 * 60 * 24)")
+    math(EXPR hours        "${runtime_ms} / (1000 * 60 * 60)")
+    math(EXPR runtime_ms   "${runtime_ms} - (${hours} * 1000 * 60 * 60)")
+    math(EXPR minutes      "${runtime_ms} / (1000 * 60)")
+    math(EXPR runtime_ms   "${runtime_ms} - (${minutes} * 1000 * 60)")
+    math(EXPR seconds      "${runtime_ms} / 1000")
+    math(EXPR milliseconds "${runtime_ms} - (${seconds} * 1000)")
+    # gersemi: on
+
+    # Format time components
+    pad_string_with_zeros(days 3)
+    pad_string_with_zeros(hours 2)
+    pad_string_with_zeros(minutes 2)
+    pad_string_with_zeros(seconds 2)
+    pad_string_with_zeros(milliseconds 3)
+
+    # Construct table entry
+    # Later values in the file for the same command overwrite earlier entries
+    string(MAKE_C_IDENTIFIER "${command}" key)
+    set(
+      ENTRY_${key}
+      "${days}d ${hours}h ${minutes}m ${seconds}s ${milliseconds}ms | ${command}"
+    )
+
+    # Record the key:
+    list(APPEND keys "${key}")
+  endif()
+endforeach()
+
+list(REMOVE_DUPLICATES keys)
+
+# Build the entry list:
+set(entries)
+foreach (key ${keys})
+  list(APPEND entries "${ENTRY_${key}}")
+endforeach()
+
+if (NOT entries)
+  message(FATAL_ERROR "LOGFILE contained no build entries ('${LOGFILE}').")
+endif()
+
+# Sort in descending order:
+list(SORT entries)
+list(REVERSE entries)
+
+# Dump table:
+message(STATUS "-----------------------+----------------------------")
+message(STATUS "Time                   | Command                    ")
+message(STATUS "-----------------------+----------------------------")
+
+foreach (entry ${entries})
+  message(STATUS ${entry})
+endforeach()

cccl_upstream/cmake/find_modules/FindNCCL.cmake

@@ -0,0 +1,120 @@
+#===----------------------------------------------------------------------===##
+#
+# Part of CUDA Experimental in CUDA C++ Core Libraries,
+# under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES.
+#
+#===----------------------------------------------------------------------===##
+
+#[=======================================================================[.rst:
+FindNCCL
+--------
+
+Find NCCL
+
+Imported targets
+^^^^^^^^^^^^^^^^
+
+This module defines the following :prop_tgt:`IMPORTED` target(s):
+
+``NCCL::nccl``
+  The NCCL library, if found.
+
+Result variables
+^^^^^^^^^^^^^^^^
+
+This module will set the following variables in your project:
+
+``NCCL_FOUND``
+  True if NCCL is found.
+``NCCL_INCLUDE_DIRS``
+  The include directories needed to use NCCL.
+``NCCL_LIBRARIES``
+  The libraries needed to useNCCL.
+``NCCL_VERSION_STRING``
+  The version of the NCCL library found. [OPTIONAL]
+
+#]=======================================================================]
+
+# Prefer using a Config module if it exists for this project
+
+include(${CMAKE_ROOT}/Modules/FindPackageHandleStandardArgs.cmake)
+
+# Also search CUDA paths for good measure
+if (CUDAToolkit_ROOT)
+  list(APPEND CMAKE_PREFIX_PATH ${CUDAToolkit_ROOT})
+endif()
+
+find_package(NCCL CONFIG QUIET)
+if (NCCL_FOUND)
+  find_package_handle_standard_args(NCCL DEFAULT_MSG NCCL_CONFIG)
+  return()
+endif()
+
+find_path(NCCL_INCLUDE_DIR NAMES nccl.h)
+
+if (NOT NCCL_LIBRARY)
+  find_library(NCCL_LIBRARY_RELEASE NAMES nccl)
+  find_library(NCCL_LIBRARY_DEBUG NAMES nccld)
+
+  include(${CMAKE_ROOT}/Modules/SelectLibraryConfigurations.cmake)
+  select_library_configurations(NCCL)
+  unset(NCCL_FOUND) # incorrectly set by select_library_configurations
+endif()
+
+find_package_handle_standard_args(
+  NCCL
+  FOUND_VAR NCCL_FOUND
+  REQUIRED_VARS NCCL_LIBRARY NCCL_INCLUDE_DIR
+  VERSION_VAR NCCL_VERSION
+)
+
+if (NCCL_FOUND)
+  set(NCCL_INCLUDE_DIRS ${NCCL_INCLUDE_DIR})
+
+  if (NOT NCCL_LIBRARIES)
+    set(NCCL_LIBRARIES ${NCCL_LIBRARY})
+  endif()
+
+  if (NOT TARGET NCCL::nccl)
+    add_library(NCCL::nccl UNKNOWN IMPORTED GLOBAL)
+    set_target_properties(
+      NCCL::nccl
+      PROPERTIES INTERFACE_INCLUDE_DIRECTORIES "${NCCL_INCLUDE_DIRS}"
+    )
+
+    if (NCCL_LIBRARY_RELEASE)
+      set_property(
+        TARGET NCCL::nccl
+        APPEND
+        PROPERTY IMPORTED_CONFIGURATIONS RELEASE
+      )
+      set_target_properties(
+        NCCL::nccl
+        PROPERTIES IMPORTED_LOCATION_RELEASE "${NCCL_LIBRARY_RELEASE}"
+      )
+    endif()
+
+    if (NCCL_LIBRARY_DEBUG)
+      set_property(
+        TARGET NCCL::nccl
+        APPEND
+        PROPERTY IMPORTED_CONFIGURATIONS DEBUG
+      )
+      set_target_properties(
+        NCCL::nccl
+        PROPERTIES IMPORTED_LOCATION_DEBUG "${NCCL_LIBRARY_DEBUG}"
+      )
+    endif()
+
+    if (NOT NCCL_LIBRARY_RELEASE AND NOT NCCL_LIBRARY_DEBUG)
+      set_property(
+        TARGET NCCL::nccl
+        APPEND
+        PROPERTY IMPORTED_LOCATION "${NCCL_LIBRARY}"
+      )
+    endif()
+  endif()
+endif()

cccl_upstream/cmake/header_test.cu.in

@@ -0,0 +1,89 @@
+// This source file checks that:
+// 1) Header <@header@> compiles without error.
+// 2) Common macro collisions with platform/system headers are avoided.
+// 3) half/bf16 aren't included when these are explicitly disabled.
+
+// Define CCCL_HEADER_MACRO_CHECK(macro, header), which emits a diagnostic indicating
+// a potential macro collision and halts.
+//
+// Hacky way to build a string, but it works on all tested platforms.
+#define CCCL_HEADER_MACRO_CHECK(MACRO, HEADER) \
+  CCCL_HEADER_MACRO_CHECK_IMPL(                \
+    Identifier MACRO should not be used from Thrust headers due to conflicts with HEADER macros.)
+
+// Use raw platform macros instead of the CCCL macros since we
+// don't want to #include any headers other than the one being tested.
+//
+// This is only implemented for MSVC/GCC/Clang.
+#if defined(_MSC_VER) // MSVC
+
+// Fake up an error for MSVC
+#  define CCCL_HEADER_MACRO_CHECK_IMPL(msg)                                                  \
+    /* Print message that looks like an error: */                                            \
+    __pragma(message(__FILE__ ":" CCCL_HEADER_MACRO_CHECK_IMPL0(__LINE__) ": error: " #msg)) \
+                                                                                             \
+      static_assert(false, #msg); /* abort compilation due to static_assert or syntax error */
+#  define CCCL_HEADER_MACRO_CHECK_IMPL0(x) CCCL_HEADER_MACRO_CHECK_IMPL1(x)
+#  define CCCL_HEADER_MACRO_CHECK_IMPL1(x) #x
+
+#elif defined(__clang__) || defined(__GNUC__)
+
+// GCC/clang are easy:
+#  define CCCL_HEADER_MACRO_CHECK_IMPL(msg)   CCCL_HEADER_MACRO_CHECK_IMPL0(GCC error #msg)
+#  define CCCL_HEADER_MACRO_CHECK_IMPL0(expr) _Pragma(#expr)
+
+#endif // msvc vs. the world
+
+// May be defined to skip macro check for certain configurations.
+#ifndef CCCL_IGNORE_HEADER_MACRO_CHECKS
+
+// complex.h conflicts
+#  define I CCCL_HEADER_MACRO_CHECK('I', complex.h)
+
+// windows.h conflicts
+#  define small CCCL_HEADER_MACRO_CHECK('small', windows.h)
+// We can't enable these checks without breaking some builds -- some standard
+// library implementations unconditionally `#undef` these macros, which then
+// causes random failures later.
+// Leaving these commented out as a warning: Here be dragons.
+// #define min(...) CCCL_HEADER_MACRO_CHECK('min', windows.h)
+// #define max(...) CCCL_HEADER_MACRO_CHECK('max', windows.h)
+
+#  ifdef _WIN32
+// On Windows, make sure any include of Windows.h (e.g. via NVTX) does not define the checked macros
+#    define WIN32_LEAN_AND_MEAN
+#  endif // _WIN32
+
+// termios.h conflicts (NVIDIA/thrust#1547)
+#  define B0 CCCL_HEADER_MACRO_CHECK("B0", termios.h)
+
+#endif // CCCL_IGNORE_HEADER_MACRO_CHECKS
+
+#include <@header@>
+
+#if defined(CCCL_DISABLE_NVFP8_SUPPORT)
+#  if defined(__CUDA_FP8_TYPES_EXIST__)
+#    error We should not include cuda_fp8.h when FP8 support is disabled
+#  endif // __CUDA_FP16_TYPES_EXIST__
+#endif // CCCL_DISABLE_BF16_SUPPORT
+
+#if defined(CCCL_DISABLE_BF16_SUPPORT)
+#  if defined(__CUDA_BF16_TYPES_EXIST__)
+#    error We should not include cuda_bf16.h when BF16 support is disabled
+#  endif // __CUDA_BF16_TYPES_EXIST__
+#  if defined(__CUDA_FP8_TYPES_EXIST__)
+#    error We should not include cuda_fp8.h when BF16 support is disabled
+#  endif // __CUDA_FP16_TYPES_EXIST__
+#endif // CCCL_DISABLE_BF16_SUPPORT
+
+#if defined(CCCL_DISABLE_FP16_SUPPORT)
+#  if defined(__CUDA_FP8_TYPES_EXIST__)
+#    error We should not include cuda_fp8.h when half support is disabled
+#  endif // __CUDA_FP16_TYPES_EXIST__
+#  if defined(__CUDA_FP16_TYPES_EXIST__)
+#    error We should not include cuda_fp16.h when half support is disabled
+#  endif // __CUDA_FP16_TYPES_EXIST__
+#  if defined(__CUDA_BF16_TYPES_EXIST__)
+#    error We should not include cuda_bf16.h when half support is disabled
+#  endif // __CUDA_BF16_TYPES_EXIST__
+#endif // CCCL_DISABLE_FP16_SUPPORT

cccl_upstream/cmake/install/cccl.cmake

@@ -0,0 +1 @@
+cccl_generate_install_rules(CCCL ${CCCL_TOPLEVEL_PROJECT} NO_HEADERS PACKAGE)

cccl_upstream/cmake/install/cub.cmake

@@ -0,0 +1,6 @@
+cccl_generate_install_rules(
+  CUB
+  ${CCCL_TOPLEVEL_PROJECT}
+  HEADERS_INCLUDE "*.cuh"
+  PACKAGE
+)

cccl_upstream/cmake/install/cudax.cmake

@@ -0,0 +1,7 @@
+cccl_generate_install_rules(
+  cudax
+  ${CCCL_ENABLE_CUDAX}
+  HEADERS_SUBDIRS "include/cuda"
+  HEADERS_INCLUDE "*.cuh"
+  PACKAGE
+)

cccl_upstream/cmake/install/libcudacxx.cmake

@@ -0,0 +1,8 @@
+cccl_generate_install_rules(
+  libcudacxx
+  ${CCCL_TOPLEVEL_PROJECT}
+  HEADERS_SUBDIRS "include/cuda" "include/nv"
+  HEADERS_INCLUDE "*"
+  HEADERS_EXCLUDE "CMakeLists.txt"
+  PACKAGE
+)

cccl_upstream/cmake/install/thrust.cmake

@@ -0,0 +1,6 @@
+cccl_generate_install_rules(
+  Thrust
+  ${CCCL_TOPLEVEL_PROJECT}
+  HEADERS_INCLUDE "*.h" "*.inl"
+  PACKAGE
+)

cccl_upstream/cmake/link_check_main.cpp

@@ -0,0 +1,4 @@
+int main()
+{
+  return 0;
+}

cccl_upstream/cmake/run_clang_tidy.sh.in

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+set -eou pipefail
+
+clang_tidy_args=()
+if test -n "${CCCL_CLANG_TIDY_ARGS:+x}"; then
+    mapfile -t clang_tidy_args < <(echo "${CCCL_CLANG_TIDY_ARGS}" | tr ' ' '\n' | sort -u)
+fi
+
+set -x
+
+"@CCCL_CLANG_TIDY@" \
+  --use-color \
+  --quiet \
+  --extra-arg='-Wno-error=unused-command-line-argument' \
+  --extra-arg='-D_CCCL_CLANG_TIDY_INVOKED=1' \
+  -p '@CMAKE_BINARY_DIR@' \
+  "${clang_tidy_args[@]}" \
+  "$@"

cccl_upstream/cudax/CMakeLists.txt

@@ -0,0 +1,94 @@
+if (NOT CCCL_ENABLE_CUDAX)
+  include(cmake/cudaxAddSubdir.cmake)
+  return()
+endif()
+
+cmake_minimum_required(VERSION 3.21)
+project(cudax LANGUAGES CXX CUDA)
+
+option(
+  cudax_ENABLE_HEADER_TESTING
+  "Test that CUDA Experimental's public headers compile."
+  ON
+)
+option(cudax_ENABLE_TESTING "Build CUDA Experimental's tests." ON)
+option(cudax_ENABLE_EXAMPLES "Build CUDA Experimental's examples." ON)
+option(cudax_ENABLE_PLACES "Enable standalone Places subproject" ON)
+option(cudax_ENABLE_CUDASTF "Enable CUDASTF subproject" ON)
+option(
+  cudax_ENABLE_CUDASTF_CODE_GENERATION
+  "Enable code generation using STF's parallel_for or launch with CUDA compiler."
+  ON
+)
+option(
+  cudax_ENABLE_CUDASTF_BOUNDSCHECK
+  "Enable bounds checks for STF targets. Requires debug build."
+  OFF
+)
+option(
+  cudax_ENABLE_CUDASTF_MATHLIBS
+  "Enable STF tests/examples that use cublas/cusolver."
+  OFF
+)
+option(cudax_ENABLE_CUFILE "Enable cuFile in CUDA Experimental" ON)
+
+if (cudax_ENABLE_CUFILE)
+  if (WIN32)
+    message(FATAL_ERROR "cuFile is not available on Windows.")
+  endif()
+
+  if (CMAKE_VERSION VERSION_LESS "3.25.0")
+    message(
+      FATAL_ERROR
+      "cuFile is not available before cmake 3.25.0, please, use newer cmake."
+    )
+  endif()
+
+  cccl_get_cudatoolkit()
+
+  if (CUDAToolkit_VERSION VERSION_LESS "12.9.0")
+    message(FATAL_ERROR "cuFile support requires at least CUDA 12.9.")
+  endif()
+
+  if (NOT TARGET CUDA::cuFile)
+    message(
+      FATAL_ERROR
+      "CUDA::cuFile target required for requested cuFile support was not found."
+    )
+  endif()
+endif()
+
+if (
+  cudax_ENABLE_CUDASTF_BOUNDSCHECK
+  AND NOT CMAKE_BUILD_TYPE MATCHES "Debug"
+  AND NOT CMAKE_BUILD_TYPE MATCHES "RelWithDebInfo"
+)
+  message(
+    FATAL_ERROR
+    "cudax_ENABLE_CUDASTF_BOUNDSCHECK requires a Debug build."
+  )
+endif()
+
+include(cmake/cudaxBuildCompilerTargets.cmake)
+if (cudax_ENABLE_PLACES)
+  include(cmake/cudaxPlacesConfigureTarget.cmake)
+endif()
+if (cudax_ENABLE_CUDASTF)
+  include(cmake/cudaxSTFConfigureTarget.cmake)
+endif()
+
+if (cudax_ENABLE_HEADER_TESTING)
+  include(cmake/cudaxHeaderTesting.cmake)
+endif()
+
+if (cudax_ENABLE_TESTING)
+  add_subdirectory(test)
+endif()
+
+if (cudax_ENABLE_EXAMPLES)
+  add_subdirectory(examples)
+endif()
+
+if (CCCL_ENABLE_BENCHMARKS)
+  add_subdirectory(benchmarks)
+endif()

cccl_upstream/cudax/LICENSE.TXT

@@ -0,0 +1,222 @@
+====================================================================================
+The CUDA Experimental library is under the Apache License v2.0 with LLVM Exceptions:
+====================================================================================
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+    TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+    1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+    2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+    3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+    4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+    5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+    6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+    7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+    8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+    9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+    END OF TERMS AND CONDITIONS
+
+    APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+    Copyright [yyyy] [name of copyright owner]
+
+    Licensed under the Apache License, Version 2.0 (the "License");
+    you may not use this file except in compliance with the License.
+    You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+
+
+---- LLVM Exceptions to the Apache 2.0 License ----
+
+As an exception, if, as a result of your compiling your source code, portions
+of this Software are embedded into an Object form of such source code, you
+may redistribute such embedded portions in such Object form without complying
+with the conditions of Sections 4(a), 4(b) and 4(d) of the License.
+
+In addition, if you combine or link compiled forms of this Software with
+software that is licensed under the GPLv2 ("Combined Software") and if a
+court of competent jurisdiction determines that the patent provision (Section
+3), the indemnity provision (Section 9) or other Section of the License
+conflicts with the conditions of the GPLv2, you may retroactively and
+prospectively choose to deem waived or otherwise exclude such Section(s) of
+the License, but only in their entirety and only with respect to the Combined
+Software.

cccl_upstream/cudax/README.md

@@ -0,0 +1,32 @@
+## CUDA Experimental: Library for experimental features in CUDA Core Compute Libraries.
+CUDA Experimental serves as a distribution channel for features that are considered experimental in the CUDA Core Compute Libraries.
+Some of them are still actively designed or developed and their API is evolving.
+Some of them are specific to one hardware architecture and are still looking for a generic and forward compatible exposure.
+Finally, some of them need to prove useful enough to deserve long term support.
+
+**All APIs available in CUDA Experimental are not considered stable and can change without a notice.** They can also be deprecated or removed on a much faster cadence than in other CCCL libraries.
+
+Features are exposed here for the CUDA C++ community to experiment with and provide feedback on how to shape it to best fit their use cases.
+Once we become confident a feature is ready and would be a great permanent addition in CCCL, it will become a part of some other CCCL library with a stable API.
+
+## Installation
+CUDA Experimental library is **not** distributed with the CUDA Toolkit like the rest of CCCL. It is only available on the [CCCL GitHub repository](https://github.com/NVIDIA/cccl).
+
+CUDA Experimental compilation requires C++17 standard or newer. Supported compilers are:
+
+CUDA Compilers:
+- NVCC 12.3+
+
+NVCC host compilers:
+- GCC 7+
+- Clang 9+
+- MSVC 2019+
+
+Everything in CUDA Experimental is header-only, so cloning and including it in a simple project is as easy as the following:
+```bash
+git clone https://github.com/NVIDIA/cccl.git
+# Note:
+nvcc -Icccl/cudax/include main.cu -o main
+```
+
+A CMake target `cudax::cudax` is available as part of the CCCL package when `CCCL_ENABLE_UNSTABLE` is set to a truthy value before calling `find_package` or `add_subdirectory`.

cccl_upstream/cudax/benchmarks/CMakeLists.txt

@@ -0,0 +1,69 @@
+include(${CMAKE_SOURCE_DIR}/benchmarks/cmake/CCCLBenchmarkRegistry.cmake)
+
+cccl_get_nvbench()
+cccl_get_nvbench_helper()
+
+set(benches_root "${CMAKE_CURRENT_LIST_DIR}")
+
+function(get_recursive_subdirs subdirs)
+  set(dirs)
+  file(
+    GLOB_RECURSE contents
+    CONFIGURE_DEPENDS
+    LIST_DIRECTORIES ON
+    "${CMAKE_CURRENT_LIST_DIR}/bench/*"
+  )
+
+  foreach (bench_dir IN LISTS contents)
+    if (IS_DIRECTORY "${bench_dir}")
+      list(APPEND dirs "${bench_dir}")
+    endif()
+  endforeach()
+
+  set(${subdirs} "${dirs}" PARENT_SCOPE)
+endfunction()
+
+function(add_bench target_name bench_name bench_src)
+  set(bench_target ${bench_name})
+  set(${target_name} ${bench_target} PARENT_SCOPE)
+
+  cccl_add_executable(${bench_target} SOURCES "${bench_src}")
+  target_link_libraries(
+    ${bench_target}
+    PRIVATE #
+      cccl.nvbench_helper
+      nvbench::main
+  )
+endfunction()
+
+function(add_bench_dir bench_dir)
+  file(GLOB bench_srcs CONFIGURE_DEPENDS "${bench_dir}/*.cu")
+  file(RELATIVE_PATH bench_prefix "${benches_root}" "${bench_dir}")
+  file(TO_CMAKE_PATH "${bench_prefix}" bench_prefix)
+  string(REPLACE "/" "." bench_prefix "${bench_prefix}")
+
+  foreach (bench_src IN LISTS bench_srcs)
+    get_filename_component(bench_name "${bench_src}" NAME_WLE)
+    string(PREPEND bench_name "cudax.${bench_prefix}.")
+    register_cccl_benchmark("${bench_name}" "")
+
+    string(APPEND bench_name ".base")
+
+    add_bench(base_bench_target ${bench_name} "${bench_src}")
+    target_link_libraries(${bench_name} PRIVATE cudax.compiler_interface)
+    target_compile_options(
+      ${bench_name}
+      PRIVATE
+        "$<$<COMPILE_LANG_AND_ID:CUDA,NVIDIA>:--extended-lambda>"
+        # cudax.compiler_interface enables assertions for tests/examples; benchmarks should measure release behavior.
+        "$<$<COMPILE_LANGUAGE:CUDA>:-UCCCL_ENABLE_ASSERTIONS>"
+        "$<$<COMPILE_LANGUAGE:CXX>:-UCCCL_ENABLE_ASSERTIONS>"
+    )
+  endforeach()
+endfunction()
+
+get_recursive_subdirs(subdirs)
+
+foreach (subdir IN LISTS subdirs)
+  add_bench_dir("${subdir}")
+endforeach()

cccl_upstream/cudax/benchmarks/bench/copy/copy_bench.cu

@@ -0,0 +1,331 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+#include <thrust/device_vector.h>
+
+#include <cuda/mdspan>
+#include <cuda/std/array>
+#include <cuda/stream>
+
+#include <cuda/experimental/copy.cuh>
+
+#include <cstddef>
+#include <cstdint>
+#include <cstdlib>
+
+#include <nvbench/nvbench.cuh>
+
+// GCC -Warray-bounds false positive for high-rank (20+) __raw_tensor instantiations
+_CCCL_DIAG_SUPPRESS_GCC("-Warray-bounds")
+
+template <size_t Rank, typename idx_t>
+size_t
+compute_alloc(size_t offset, const cuda::std::array<idx_t, Rank>& shape, const cuda::std::array<idx_t, Rank>& strides)
+{
+  int64_t max_pos = static_cast<int64_t>(offset);
+  for (size_t i = 0; i < Rank; ++i)
+  {
+    auto delta = static_cast<ptrdiff_t>(shape[i] - 1) * strides[i];
+    if (delta > 0)
+    {
+      max_pos += delta;
+    }
+  }
+  return max_pos + 1;
+}
+
+template <typename data_t = int, typename idx_t = int, size_t Rank>
+void bench_copy(nvbench::state& state,
+                size_t src_offset,
+                const cuda::std::array<idx_t, Rank>& shape,
+                const cuda::std::array<idx_t, Rank>& src_strides,
+                size_t dst_offset,
+                const cuda::std::array<idx_t, Rank>& dst_strides)
+{
+  const auto src_alloc = compute_alloc(src_offset, shape, src_strides);
+  const auto dst_alloc = compute_alloc(dst_offset, shape, dst_strides);
+
+  thrust::device_vector<data_t> d_src(src_alloc);
+  thrust::device_vector<data_t> d_dst(dst_alloc);
+
+  size_t num_items = 1;
+  for (size_t i = 0; i < Rank; ++i)
+  {
+    num_items *= shape[i];
+  }
+  state.add_element_count(num_items);
+  state.add_global_memory_reads<data_t>(num_items);
+  state.add_global_memory_writes<data_t>(num_items);
+
+  using extents_t = cuda::std::dextents<idx_t, Rank>;
+  using strides_t = cuda::dstrides<idx_t, Rank>;
+  using mapping_t = cuda::layout_stride_relaxed::mapping<extents_t>;
+
+  extents_t ext(shape);
+  auto src_ptr = thrust::raw_pointer_cast(d_src.data()) + src_offset;
+  auto dst_ptr = thrust::raw_pointer_cast(d_dst.data()) + dst_offset;
+  mapping_t src_map(ext, strides_t(src_strides));
+  mapping_t dst_map(ext, strides_t(dst_strides));
+
+  cuda::device_mdspan<data_t, extents_t, cuda::layout_stride_relaxed> src(src_ptr, src_map);
+  cuda::device_mdspan<data_t, extents_t, cuda::layout_stride_relaxed> dst(dst_ptr, dst_map);
+
+  state.exec([&](nvbench::launch& launch) {
+    cuda::stream_ref stream{launch.get_stream()};
+    cuda::experimental::copy(src, dst, stream);
+  });
+}
+
+template <typename data_t = int, typename idx_t = int, size_t Rank>
+void bench_copy(nvbench::state& state,
+                size_t offset,
+                const cuda::std::array<idx_t, Rank>& shape,
+                const cuda::std::array<idx_t, Rank>& strides)
+{
+  bench_copy<data_t>(state, offset, shape, strides, offset, strides);
+}
+
+/***********************************************************************************************************************
+ * Memcpy benchmarks
+ **********************************************************************************************************************/
+
+// src: (25, 70, 90, 80, 80):(40320000, 576000, 6400, 80, 1)
+// dst: (25, 70, 90, 80, 80):(40320000, 576000, 6400, 80, 1)
+void memcpy_layout_0(nvbench::state& state)
+{
+  cuda::std::array<int, 5> shape{25, 70, 90, 80, 80};
+  cuda::std::array<int, 5> strides{40320000, 576000, 6400, 80, 1};
+  bench_copy(state, 0, shape, strides);
+}
+NVBENCH_BENCH(memcpy_layout_0).set_name("contiguous (5D, int, 4GB)");
+
+// src: (25, 80, 70, 80, 90):(40320000, 1, 576000, 80, 6400)
+// dst: (25, 80, 70, 80, 90):(40320000, 1, 576000, 80, 6400)
+void memcpy_layout_1(nvbench::state& state)
+{
+  cuda::std::array<int, 5> shape{25, 80, 70, 80, 90};
+  cuda::std::array<int, 5> strides{40320000, 1, 576000, 80, 6400};
+  bench_copy(state, 0, shape, strides);
+}
+NVBENCH_BENCH(memcpy_layout_1).set_name("contiguous-perm (5D, int, 4GB)");
+
+// src: (1, 25, 1, 80, 1, 70, 1, 80, 1, 90):(1, 40320000, 1, 1, 1, 576000, 1, 80, 1, 6400)
+// dst: (1, 25, 1, 80, 1, 70, 1, 80, 1, 90):(1, 40320000, 1, 1, 1, 576000, 1, 80, 1, 6400)
+void memcpy_layout_1b(nvbench::state& state)
+{
+  cuda::std::array<int, 10> shape{1, 25, 1, 80, 1, 70, 1, 80, 1, 90};
+  cuda::std::array<int, 10> strides{1, 40320000, 1, 1, 1, 576000, 1, 80, 1, 6400};
+  bench_copy(state, 0, shape, strides);
+}
+NVBENCH_BENCH(memcpy_layout_1b).set_name("contiguous-1-sized (10D, int, 4GB)");
+
+// src: (25, 70, 90, 80, 80):(40320000, 576000, 6400, 80, 1)
+// dst: (25, 70, 90, 80, 80):(40320000, 576000, 6400, 80, 1)
+void memcpy_layout_2(nvbench::state& state)
+{
+  cuda::std::array<int, 5> shape{25, 70, 90, 80, 80};
+  cuda::std::array<int, 5> strides{40320000, 576000, 6400, 80, 1};
+  bench_copy(state, 1, shape, strides);
+}
+NVBENCH_BENCH(memcpy_layout_2).set_name("contiguous-not-aligned (5D, int, 4GB)");
+
+// src: (100, 70, 90, 80, 80):(40320000, 576000, 6400, 80, 1)
+// dst: (100, 70, 90, 80, 80):(40320000, 576000, 6400, 80, 1)
+void memcpy_layout_3(nvbench::state& state)
+{
+  cuda::std::array<int64_t, 5> shape{100, 70, 90, 80, 80};
+  cuda::std::array<int64_t, 5> strides{40320000, 576000, 6400, 80, 1};
+  bench_copy<char, int64_t>(state, 0, shape, strides);
+}
+NVBENCH_BENCH(memcpy_layout_3).set_name("contiguous-small (5D, char, 4GB)");
+
+// src: (100, 70, 90, 80, 80):(40320000, 576000, 6400, 80, 1)
+// dst: (100, 70, 90, 80, 80):(40320000, 576000, 6400, 80, 1)
+void memcpy_layout_4(nvbench::state& state)
+{
+  cuda::std::array<int64_t, 5> shape{100, 70, 90, 80, 80};
+  cuda::std::array<int64_t, 5> strides{40320000, 576000, 6400, 80, 1};
+  bench_copy<char, int64_t>(state, 1, shape, strides);
+}
+NVBENCH_BENCH(memcpy_layout_4).set_name("contiguous-small-not-aligned (5D, char, 4GB)");
+
+// src: (25, 70, 90, 80, 80):(40320000, 576000, 6400, 80, -1), offset=80
+// dst: (25, 70, 90, 80, 80):(40320000, 576000, 6400, 80, -1), offset=80
+void memcpy_neg(nvbench::state& state)
+{
+  cuda::std::array<int, 5> shape{25, 70, 90, 80, 80};
+  cuda::std::array<int, 5> strides{40320000, 576000, 6400, 80, -1};
+  bench_copy(state, 80, shape, strides);
+}
+NVBENCH_BENCH(memcpy_neg).set_name("contiguous-negative-stride (5D, int, 4GB)");
+
+// src: (134217600, 32):(128, 1), offset=32
+// dst: (134217600, 32):(128, 1), offset=32
+// Copies 4GB while allocating 16GB per tensor because of the padded outer stride.
+void vectorization(nvbench::state& state)
+{
+  cuda::std::array<int64_t, 2> shape{134217600, 32};
+  cuda::std::array<int64_t, 2> strides{128, 1};
+  bench_copy<char, int64_t>(state, 32, shape, strides);
+}
+NVBENCH_BENCH(vectorization).set_name("vectorization (2D, char, 4GB copy, 16GB alloc)");
+
+// src: (32767, (128 * 1024) / sizeof(int)):(128 * 1024, 1)
+// dst: (32767, (128 * 1024) / sizeof(int)):(128 * 1024, 1)
+// Copies 4GB while allocating 16GB per tensor because each row is padded to 128K elements.
+void block_contiguous(nvbench::state& state)
+{
+  cuda::std::array<int, 2> shape{32767, (128 * 1024) / sizeof(int)};
+  cuda::std::array<int, 2> strides{128 * 1024, 1};
+  bench_copy(state, 0, shape, strides);
+}
+NVBENCH_BENCH(block_contiguous).set_name("block-contiguous (2D, int, 4GB copy, 16GB alloc)");
+// (non-vectorizable)
+
+void several_dimensions(nvbench::state& state)
+{
+  cuda::std::array<int, 5> shape{64, 64, 64, 64, 64};
+  cuda::std::array<int, 5> strides{17043520 + 1, 266304 + 1, 4160 + 1, 64 + 1, 1};
+  bench_copy(state, 0, shape, strides);
+}
+NVBENCH_BENCH(several_dimensions).set_name("several_dimensions (5D, int, 4GB)");
+
+void several_dimensions_non_square(nvbench::state& state)
+{
+  cuda::std::array<int, 5> shape{63, 65, 67, 69, 57};
+  cuda::std::array<int, 5> strides{17433131, 268202, 4003, 58, 1};
+  bench_copy(state, 0, shape, strides);
+}
+NVBENCH_BENCH(several_dimensions_non_square).set_name("several_dimensions_non_square (5D, int, 4GB)");
+
+/***********************************************************************************************************************
+ * Transpose benchmark
+ **********************************************************************************************************************/
+
+// src: (32768,32768):(1,32768)
+// dst: (32768,32768):(32768,1)
+void transpose_2D_col_row(nvbench::state& state)
+{
+  cuda::std::array<int, 2> shape{32768, 32768};
+  cuda::std::array<int, 2> src_strides{1, 32768};
+  cuda::std::array<int, 2> dst_strides{32768, 1};
+  bench_copy(state, 0, shape, src_strides, 0, dst_strides);
+}
+NVBENCH_BENCH(transpose_2D_col_row).set_name("transpose_2D_col_row (2D, int, 4GB)");
+
+void transpose_2D_row_col(nvbench::state& state)
+{
+  cuda::std::array<int, 2> shape{32768, 32768};
+  cuda::std::array<int, 2> src_strides{32768, 1};
+  cuda::std::array<int, 2> dst_strides{1, 32768};
+  bench_copy(state, 0, shape, src_strides, 0, dst_strides);
+}
+NVBENCH_BENCH(transpose_2D_row_col).set_name("transpose_2D_row_col (2D, int, 4GB)");
+
+void transpose_2D_char(nvbench::state& state)
+{
+  cuda::std::array<int64_t, 2> shape{65536, 65536};
+  cuda::std::array<int64_t, 2> src_strides{1, 65536};
+  cuda::std::array<int64_t, 2> dst_strides{65536, 1};
+  bench_copy<char, int64_t>(state, 0, shape, src_strides, 0, dst_strides);
+}
+NVBENCH_BENCH(transpose_2D_char).set_name("transpose_2D_char (2D, char, 4GB)");
+
+void transpose_2D_short(nvbench::state& state)
+{
+  cuda::std::array<int, 2> shape{32760, 32768 * 2};
+  cuda::std::array<int, 2> src_strides{1, 32760};
+  cuda::std::array<int, 2> dst_strides{32768 * 2, 1};
+  bench_copy<short>(state, 0, shape, src_strides, 0, dst_strides);
+}
+NVBENCH_BENCH(transpose_2D_short).set_name("transpose_2D_short (2D, short, 4GB)");
+
+void transpose_2D_double(nvbench::state& state)
+{
+  cuda::std::array<int64_t, 2> shape{32768, 16384};
+  cuda::std::array<int64_t, 2> src_strides{1, 32768};
+  cuda::std::array<int64_t, 2> dst_strides{16384, 1};
+  bench_copy<double, int64_t>(state, 0, shape, src_strides, 0, dst_strides);
+}
+NVBENCH_BENCH(transpose_2D_double).set_name("transpose_2D_double (2D, double, 4GB)");
+
+void transpose_2D_odd_both(nvbench::state& state)
+{
+  cuda::std::array<int, 2> shape{32767, 32769};
+  cuda::std::array<int, 2> src_strides{1, 32767};
+  cuda::std::array<int, 2> dst_strides{32769, 1};
+  bench_copy(state, 0, shape, src_strides, 0, dst_strides);
+}
+NVBENCH_BENCH(transpose_2D_odd_both).set_name("transpose_2D_odd_both (2D, int, 4GB)");
+
+void transpose_3D(nvbench::state& state)
+{
+  cuda::std::array<int, 3> shape{1024, 1024, 1024};
+  cuda::std::array<int, 3> src_strides{1, 1024, 1024 * 1024};
+  cuda::std::array<int, 3> dst_strides{1024 * 1024, 1024, 1};
+  bench_copy(state, 0, shape, src_strides, 0, dst_strides);
+}
+NVBENCH_BENCH(transpose_3D).set_name("transpose_3D (3D, int, 4GB)");
+
+void transpose_3D_odd_edges(nvbench::state& state)
+{
+  cuda::std::array<int, 3> shape{1023, 1025, 1024};
+  cuda::std::array<int, 3> src_strides{1, 1023, 1023 * 1025};
+  cuda::std::array<int, 3> dst_strides{1025 * 1024, 1024, 1};
+  bench_copy(state, 0, shape, src_strides, 0, dst_strides);
+}
+NVBENCH_BENCH(transpose_3D_odd_edges).set_name("transpose_3D_odd_edges (3D, int, 4GB)");
+
+void transpose_src_small_15(nvbench::state& state)
+{
+  cuda::std::array<int, 3> shape{15, 2236962, 32};
+  cuda::std::array<int, 3> src_strides{1, 15 * 32, 15};
+  cuda::std::array<int, 3> dst_strides{2236962 * 32, 32, 1};
+  bench_copy(state, 0, shape, src_strides, 0, dst_strides);
+}
+NVBENCH_BENCH(transpose_src_small_15).set_name("transpose_src_small_15 (3D, int, 4GB)");
+
+void transpose_src_small_16(nvbench::state& state)
+{
+  cuda::std::array<int, 3> shape{16, 2097152, 32};
+  cuda::std::array<int, 3> src_strides{1, 16 * 32, 16};
+  cuda::std::array<int, 3> dst_strides{2097152 * 32, 32, 1};
+  bench_copy(state, 0, shape, src_strides, 0, dst_strides);
+}
+NVBENCH_BENCH(transpose_src_small_16).set_name("transpose_src_small_16 (3D, int, 4GB)");
+
+void transpose_src_small_17(nvbench::state& state)
+{
+  cuda::std::array<int, 3> shape{17, 1973790, 32};
+  cuda::std::array<int, 3> src_strides{1, 17 * 32, 17};
+  cuda::std::array<int, 3> dst_strides{1973790 * 32, 32, 1};
+  bench_copy(state, 0, shape, src_strides, 0, dst_strides);
+}
+NVBENCH_BENCH(transpose_src_small_17).set_name("transpose_src_small_17 (3D, int, 4GB)");
+
+void transpose_dst_small_8_padded(nvbench::state& state)
+{
+  cuda::std::array<int64_t, 3> shape{32, 4194304, 8};
+  cuda::std::array<int64_t, 3> src_strides{1, 32 * 8, 32};
+  cuda::std::array<int64_t, 3> dst_strides{4194304 * 16, 16, 1};
+  bench_copy<int, int64_t>(state, 0, shape, src_strides, 0, dst_strides);
+}
+NVBENCH_BENCH(transpose_dst_small_8_padded).set_name("transpose_dst_small_8_padded (3D, int, 4GB)");
+
+void transpose_dst_small_16_padded(nvbench::state& state)
+{
+  cuda::std::array<int64_t, 3> shape{32, 2097152, 16};
+  cuda::std::array<int64_t, 3> src_strides{1, 32 * 16, 32};
+  cuda::std::array<int64_t, 3> dst_strides{2097152 * 32, 32, 1};
+  bench_copy<int, int64_t>(state, 0, shape, src_strides, 0, dst_strides);
+}
+NVBENCH_BENCH(transpose_dst_small_16_padded).set_name("transpose_dst_small_16_padded (3D, int, 4GB)");
+
+void transpose_src_small_16_4D(nvbench::state& state)
+{
+  cuda::std::array<int, 4> shape{16, 1024, 2048, 32};
+  cuda::std::array<int, 4> src_strides{1, 16 * 32, 16 * 32 * 1024, 16};
+  cuda::std::array<int, 4> dst_strides{1024 * 2048 * 32, 32, 1024 * 32, 1};
+  bench_copy(state, 0, shape, src_strides, 0, dst_strides);
+}
+NVBENCH_BENCH(transpose_src_small_16_4D).set_name("transpose_src_small_16_4D (4D, int, 4GB)");

cccl_upstream/cudax/benchmarks/bench/cuco/common/defaults.cuh

@@ -0,0 +1,44 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDA Experimental in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+#pragma once
+
+#include <cuda/std/cstdint>
+
+#include <vector>
+
+#include <nvbench/nvbench.cuh>
+#include <nvbench/range.cuh>
+
+namespace cuda::experimental::cuco::benchmark::defaults
+{
+//! Key types covered by the default CUCO benchmark type axes.
+using key_type_range = ::nvbench::type_list<::nvbench::int32_t, ::nvbench::int64_t>;
+//! Value types covered by the default CUCO benchmark type axes.
+using value_type_range = ::nvbench::type_list<::nvbench::int32_t, ::nvbench::int64_t>;
+
+//! Default number of inputs used when sweeping another benchmark axis.
+inline constexpr auto n = ::nvbench::int64_t{100'000'000};
+//! Default fixed-capacity map target occupancy.
+inline constexpr auto occupancy = 0.5;
+//! Default lookup matching rate for contains-style benchmarks.
+inline constexpr auto matching_rate = 1.0;
+//! Default deterministic seed used by benchmark data generators.
+inline constexpr auto seed = ::cuda::std::uint32_t{42};
+
+//! Input-size sweep that remains cacheable for direct comparisons with CUCO benchmarks.
+inline const auto n_range_cache = ::std::vector<::nvbench::int64_t>{8'000, 80'000, 800'000, 8'000'000, 80'000'000};
+//! Occupancy sweep used by fixed-capacity container benchmarks.
+inline const auto occupancy_range = ::nvbench::range(0.1, 0.9, 0.1);
+//! Average multiplicity sweep for duplicate-key distributions.
+inline const auto multiplicity_range = ::std::vector<double>{1.0, 2.0, 4.0, 8.0, 16.0};
+//! Matching-rate sweep used by contains-style benchmarks.
+inline const auto matching_rate_range = ::nvbench::range(0.1, 1.0, 0.1);
+} // namespace cuda::experimental::cuco::benchmark::defaults

cccl_upstream/cudax/benchmarks/bench/cuco/common/key_generator.cuh

@@ -0,0 +1,285 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDA Experimental in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+#pragma once
+
+#include <thrust/execution_policy.h>
+#include <thrust/random.h>
+#include <thrust/sequence.h>
+#include <thrust/shuffle.h>
+#include <thrust/transform.h>
+
+#include <cuda/iterator>
+#include <cuda/std/cmath>
+#include <cuda/std/cstddef>
+#include <cuda/std/cstdint>
+#include <cuda/std/iterator>
+#include <cuda/std/limits>
+#include <cuda/std/type_traits>
+
+#include <stdexcept>
+
+#include "defaults.cuh"
+#include <nvbench/nvbench.cuh>
+
+namespace cuda::experimental::cuco::benchmark
+{
+namespace distribution
+{
+//! Distribution tag for unique keys generated by shuffling the sequence `[0, N)`.
+struct unique
+{};
+
+//! Distribution tag for uniformly sampled keys with controlled average multiplicity.
+struct uniform
+{
+  //! Constructs a uniform distribution tag with the requested average key multiplicity.
+  //!
+  //! @param multiplicity Average number of generated keys that map to each unique key value.
+  //! @throws std::invalid_argument if `multiplicity` is not finite or is less than 1.0.
+  explicit uniform(double multiplicity)
+      : multiplicity{multiplicity}
+  {
+    if (!cuda::std::isfinite(multiplicity) || multiplicity < 1.0)
+    {
+      throw ::std::invalid_argument{"Multiplicity must be finite and at least 1"};
+    }
+  }
+
+  //! Average number of generated keys that map to each unique key value.
+  double multiplicity;
+};
+} // namespace distribution
+
+namespace detail
+{
+template <typename Key, typename Distribution, typename Rng>
+struct generate_uniform_fn
+{
+  __host__ __device__ constexpr generate_uniform_fn(cuda::std::size_t num, Distribution dist, cuda::std::size_t seed)
+      : num{num}
+      , dist{dist}
+      , seed{seed}
+  {}
+
+  __host__ __device__ constexpr Key operator()(cuda::std::size_t idx) const noexcept
+  {
+    Rng rng;
+    rng.seed(seed + idx * 1664525ull + 1013904223ull);
+
+    const auto num_unique_keys_unclamped =
+      static_cast<cuda::std::size_t>(cuda::std::ceil(static_cast<double>(num) / dist.multiplicity));
+    const auto num_unique_keys =
+      num_unique_keys_unclamped < cuda::std::size_t{1} ? cuda::std::size_t{1} : num_unique_keys_unclamped;
+    thrust::uniform_int_distribution<Key> key_dist{Key{0}, static_cast<Key>(num_unique_keys - 1)};
+    return key_dist(rng);
+  }
+
+  cuda::std::size_t num;
+  Distribution dist;
+  cuda::std::size_t seed;
+};
+
+template <typename Key, typename Rng>
+struct dropout_fn
+{
+  __host__ __device__ constexpr explicit dropout_fn(cuda::std::size_t num)
+      : num{num}
+  {}
+
+  __host__ __device__ Key operator()(cuda::std::size_t seed) const noexcept
+  {
+    Rng rng;
+    thrust::uniform_int_distribution<Key> dist{static_cast<Key>(num), cuda::std::numeric_limits<Key>::max()};
+    rng.seed(seed);
+    return dist(rng);
+  }
+
+  cuda::std::size_t num;
+};
+
+template <typename Rng>
+struct dropout_pred
+{
+  __host__ __device__ constexpr explicit dropout_pred(double keep_prob)
+      : keep_prob{keep_prob}
+  {}
+
+  __host__ __device__ bool operator()(cuda::std::size_t seed) const noexcept
+  {
+    Rng rng;
+    thrust::uniform_real_distribution<double> dist{0.0, 1.0};
+    rng.seed(seed);
+    return dist(rng) > keep_prob;
+  }
+
+  double keep_prob;
+};
+} // namespace detail
+
+//! Random key generator used by CUCO benchmarks.
+//!
+//! The generator defaults to `defaults::seed` to keep benchmark data reproducible across runs.
+//!
+//! @tparam Rng Pseudo-random number generator type compatible with Thrust random distributions.
+template <typename Rng = thrust::default_random_engine>
+class key_generator
+{
+public:
+  //! Constructs a key generator with the given seed.
+  //!
+  //! @param seed Seed used to initialize the generator state.
+  explicit key_generator(cuda::std::uint32_t seed = defaults::seed)
+      : rng{seed}
+  {}
+
+  //! Generates keys according to the given distribution using the default device execution policy.
+  //!
+  //! @tparam Distribution Distribution tag type.
+  //! @tparam OutputIt Output iterator type whose value type is the generated key type.
+  //! @param dist Distribution tag controlling how keys are generated.
+  //! @param out_begin Beginning of the output key range.
+  //! @param out_end End of the output key range.
+  //! @throws std::invalid_argument if `Distribution` is not a supported distribution tag.
+  template <typename Distribution, typename OutputIt>
+  void generate(Distribution dist, OutputIt out_begin, OutputIt out_end)
+  {
+    generate(dist, out_begin, out_end, thrust::device);
+  }
+
+  //! Generates keys according to the given distribution using the provided execution policy.
+  //!
+  //! @tparam Distribution Distribution tag type.
+  //! @tparam OutputIt Output iterator type whose value type is the generated key type.
+  //! @tparam ExecPolicy Thrust execution policy type.
+  //! @param dist Distribution tag controlling how keys are generated.
+  //! @param out_begin Beginning of the output key range.
+  //! @param out_end End of the output key range.
+  //! @param exec_policy Execution policy used for the underlying Thrust algorithms.
+  //! @throws std::invalid_argument if `Distribution` is not a supported distribution tag.
+  template <typename Distribution, typename OutputIt, typename ExecPolicy>
+  void generate(Distribution dist, OutputIt out_begin, OutputIt out_end, ExecPolicy exec_policy)
+  {
+    using value_type = typename cuda::std::iterator_traits<OutputIt>::value_type;
+
+    if constexpr (cuda::std::is_same_v<Distribution, distribution::unique>)
+    {
+      thrust::sequence(exec_policy, out_begin, out_end, value_type{0});
+      thrust::shuffle(exec_policy, out_begin, out_end, rng);
+    }
+    else if constexpr (cuda::std::is_same_v<Distribution, distribution::uniform>)
+    {
+      const auto num_keys = static_cast<cuda::std::size_t>(cuda::std::distance(out_begin, out_end));
+      const auto seed     = static_cast<cuda::std::size_t>(rng());
+
+      thrust::transform(
+        exec_policy,
+        cuda::counting_iterator<cuda::std::size_t>{0},
+        cuda::counting_iterator<cuda::std::size_t>{num_keys},
+        out_begin,
+        detail::generate_uniform_fn<value_type, Distribution, Rng>{num_keys, dist, seed});
+    }
+    else
+    {
+      throw ::std::invalid_argument{"Unexpected distribution type"};
+    }
+  }
+
+  //! Drops keys with probability `1 - keep_prob` using the default device execution policy.
+  //!
+  //! Replaced keys are sampled from `[N, max_key]`, where `N` is the number of keys in the range.
+  //!
+  //! The full range is shuffled afterward, even when all keys are kept.
+  //!
+  //! @tparam InOutIt Mutable iterator type whose value type is the key type.
+  //! @param begin Beginning of the key range to update in place.
+  //! @param end End of the key range to update in place.
+  //! @param keep_prob Probability of keeping each original key. Must be in `[0, 1]`.
+  //! @throws std::invalid_argument if `keep_prob` is outside `[0, 1]`.
+  template <typename InOutIt>
+  void dropout(InOutIt begin, InOutIt end, double keep_prob)
+  {
+    dropout(begin, end, keep_prob, thrust::device);
+  }
+
+  //! Drops keys with probability `1 - keep_prob` using the provided execution policy.
+  //!
+  //! Replaced keys are sampled from `[N, max_key]`, where `N` is the number of keys in the range.
+  //!
+  //! The full range is shuffled afterward, even when all keys are kept.
+  //!
+  //! @tparam InOutIt Mutable iterator type whose value type is the key type.
+  //! @tparam ExecPolicy Thrust execution policy type.
+  //! @param begin Beginning of the key range to update in place.
+  //! @param end End of the key range to update in place.
+  //! @param keep_prob Probability of keeping each original key. Must be in `[0, 1]`.
+  //! @param exec_policy Execution policy used for the underlying Thrust algorithms.
+  //! @throws std::invalid_argument if `keep_prob` is outside `[0, 1]`.
+  template <typename InOutIt, typename ExecPolicy>
+  void dropout(InOutIt begin, InOutIt end, double keep_prob, ExecPolicy exec_policy)
+  {
+    using value_type = typename cuda::std::iterator_traits<InOutIt>::value_type;
+
+    if (keep_prob < 0.0 || keep_prob > 1.0)
+    {
+      throw ::std::invalid_argument{"Probability needs to be between 0 and 1"};
+    }
+
+    if (keep_prob < 1.0)
+    {
+      const auto num_keys = static_cast<cuda::std::size_t>(cuda::std::distance(begin, end));
+      cuda::counting_iterator<cuda::std::size_t> seeds{static_cast<cuda::std::size_t>(rng())};
+
+      thrust::transform_if(
+        exec_policy,
+        seeds,
+        seeds + num_keys,
+        begin,
+        detail::dropout_fn<value_type, Rng>{num_keys},
+        detail::dropout_pred<Rng>{keep_prob});
+    }
+
+    thrust::shuffle(exec_policy, begin, end, rng);
+  }
+
+private:
+  Rng rng;
+};
+
+//! Constructs the requested distribution tag from NVBench axis values.
+//!
+//! `distribution::uniform` reads the `Multiplicity` axis from `state`.
+//!
+//! @tparam Distribution Distribution tag type to construct.
+//! @param state NVBench state containing distribution-specific axis values.
+//! @return Distribution tag initialized from the benchmark state.
+//! @throws std::invalid_argument if `Distribution` is not a supported distribution tag.
+template <typename Distribution>
+Distribution dist_from_state(nvbench::state const& state)
+{
+  if constexpr (cuda::std::is_same_v<Distribution, distribution::unique>)
+  {
+    return Distribution{};
+  }
+  else if constexpr (cuda::std::is_same_v<Distribution, distribution::uniform>)
+  {
+    return Distribution{state.get_float64("Multiplicity")};
+  }
+  else
+  {
+    throw ::std::invalid_argument{"Unexpected distribution type"};
+  }
+}
+} // namespace cuda::experimental::cuco::benchmark
+
+NVBENCH_DECLARE_TYPE_STRINGS(
+  cuda::experimental::cuco::benchmark::distribution::unique, "UNIQUE", "distribution::unique");
+NVBENCH_DECLARE_TYPE_STRINGS(
+  cuda::experimental::cuco::benchmark::distribution::uniform, "UNIFORM", "distribution::uniform");

cccl_upstream/cudax/benchmarks/bench/cuco/fixed_capacity_map/contains.cu

@@ -0,0 +1,110 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDA Experimental in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+#include <thrust/execution_policy.h>
+#include <thrust/transform.h>
+
+#include <cuda/buffer>
+#include <cuda/memory_resource>
+#include <cuda/std/cstddef>
+#include <cuda/std/functional>
+#include <cuda/std/utility>
+#include <cuda/stream>
+
+#include <cuda/experimental/__cuco/fixed_capacity_map.cuh>
+#include <cuda/experimental/__cuco/types.cuh>
+
+#include "../common/defaults.cuh"
+#include "../common/key_generator.cuh"
+#include <nvbench/nvbench.cuh>
+
+namespace cudax = cuda::experimental;
+namespace bench = cudax::cuco::benchmark;
+
+/**
+ * @brief A benchmark evaluating `cudax::cuco::fixed_capacity_map::contains_async` performance.
+ */
+template <typename Key, typename Value, typename Dist>
+void fixed_capacity_map_contains(nvbench::state& state, nvbench::type_list<Key, Value, Dist>)
+{
+  if constexpr (sizeof(Key) != sizeof(Value))
+  {
+    state.skip("Key and Value must have the same size.");
+  }
+  else
+  {
+    using pair_type = cuda::std::pair<Key, Value>;
+    using map_type  = cudax::cuco::fixed_capacity_map<Key, Value>;
+
+    const auto num_keys      = state.get_int64("NumInputs");
+    const auto occupancy     = state.get_float64("Occupancy");
+    const auto matching_rate = state.get_float64("MatchingRate");
+
+    const auto size = static_cast<cuda::std::size_t>(static_cast<double>(num_keys) / occupancy);
+
+    const auto device = cuda::device_ref{0};
+    cuda::stream stream{device};
+    const cuda::device_memory_pool_ref mr = cuda::device_default_memory_pool(device);
+    const auto exec_policy                = thrust::cuda::par_nosync.on(stream.get());
+
+    auto keys = cuda::make_device_buffer<Key>(stream, device, num_keys, cuda::no_init);
+
+    bench::key_generator gen{};
+    gen.generate(bench::dist_from_state<Dist>(state), keys.begin(), keys.end(), exec_policy);
+
+    auto pairs = cuda::make_device_buffer<pair_type>(stream, device, num_keys, cuda::no_init);
+    thrust::transform(exec_policy, keys.begin(), keys.end(), pairs.begin(), [] __device__(Key const& key) {
+      return pair_type{key, Value{}};
+    });
+
+    map_type map{stream, mr, size, cudax::cuco::empty_key(Key{-1}), cudax::cuco::empty_value(Value{-1})};
+    map.insert(stream, pairs.begin(), pairs.end());
+
+    gen.dropout(keys.begin(), keys.end(), matching_rate, exec_policy);
+
+    auto result = cuda::make_device_buffer<bool>(stream, device, num_keys, cuda::no_init);
+    stream.sync();
+
+    state.add_element_count(num_keys);
+    state.exec([&](nvbench::launch& launch) {
+      map.contains_async({launch.get_stream()}, keys.begin(), keys.end(), result.begin());
+    });
+  }
+}
+
+NVBENCH_BENCH_TYPES(fixed_capacity_map_contains,
+                    NVBENCH_TYPE_AXES(bench::defaults::key_type_range,
+                                      bench::defaults::value_type_range,
+                                      nvbench::type_list<bench::distribution::unique>))
+  .set_name("fixed_capacity_map_contains_unique_capacity")
+  .set_type_axes_names({"Key", "Value", "Distribution"})
+  .add_int64_axis("NumInputs", bench::defaults::n_range_cache)
+  .add_float64_axis("Occupancy", {bench::defaults::occupancy})
+  .add_float64_axis("MatchingRate", {bench::defaults::matching_rate});
+
+NVBENCH_BENCH_TYPES(fixed_capacity_map_contains,
+                    NVBENCH_TYPE_AXES(bench::defaults::key_type_range,
+                                      bench::defaults::value_type_range,
+                                      nvbench::type_list<bench::distribution::unique>))
+  .set_name("fixed_capacity_map_contains_unique_occupancy")
+  .set_type_axes_names({"Key", "Value", "Distribution"})
+  .add_int64_axis("NumInputs", {bench::defaults::n})
+  .add_float64_axis("Occupancy", bench::defaults::occupancy_range)
+  .add_float64_axis("MatchingRate", {bench::defaults::matching_rate});
+
+NVBENCH_BENCH_TYPES(fixed_capacity_map_contains,
+                    NVBENCH_TYPE_AXES(bench::defaults::key_type_range,
+                                      bench::defaults::value_type_range,
+                                      nvbench::type_list<bench::distribution::unique>))
+  .set_name("fixed_capacity_map_contains_unique_matching_rate")
+  .set_type_axes_names({"Key", "Value", "Distribution"})
+  .add_int64_axis("NumInputs", {bench::defaults::n})
+  .add_float64_axis("Occupancy", {bench::defaults::occupancy})
+  .add_float64_axis("MatchingRate", bench::defaults::matching_rate_range);

cccl_upstream/cudax/benchmarks/bench/cuco/fixed_capacity_map/find.cu

@@ -0,0 +1,108 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDA Experimental in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+#include <thrust/execution_policy.h>
+#include <thrust/transform.h>
+
+#include <cuda/buffer>
+#include <cuda/memory_resource>
+#include <cuda/std/cstddef>
+#include <cuda/std/functional>
+#include <cuda/std/utility>
+#include <cuda/stream>
+
+#include <cuda/experimental/__cuco/fixed_capacity_map.cuh>
+#include <cuda/experimental/__cuco/types.cuh>
+
+#include "../common/defaults.cuh"
+#include "../common/key_generator.cuh"
+#include <nvbench/nvbench.cuh>
+
+namespace cudax = cuda::experimental;
+namespace bench = cudax::cuco::benchmark;
+
+/**
+ * @brief A benchmark evaluating `cudax::cuco::fixed_capacity_map::find_async` performance.
+ */
+template <typename Key, typename Value, typename Dist>
+void fixed_capacity_map_find(nvbench::state& state, nvbench::type_list<Key, Value, Dist>)
+{
+  if constexpr (sizeof(Key) != sizeof(Value))
+  {
+    state.skip("Key and Value must have the same size.");
+  }
+  else
+  {
+    using pair_type = cuda::std::pair<Key, Value>;
+    using map_type  = cudax::cuco::fixed_capacity_map<Key, Value>;
+
+    const auto num_keys      = static_cast<::cuda::std::size_t>(state.get_int64("NumInputs"));
+    const auto occupancy     = state.get_float64("Occupancy");
+    const auto matching_rate = state.get_float64("MatchingRate");
+
+    const auto device = cuda::device_ref{0};
+    cuda::stream stream{device};
+    const cuda::device_memory_pool_ref mr = cuda::device_default_memory_pool(device);
+    const auto exec_policy                = thrust::cuda::par_nosync.on(stream.get());
+
+    auto keys = cuda::make_device_buffer<Key>(stream, device, num_keys, cuda::no_init);
+
+    bench::key_generator gen{};
+    gen.generate(bench::dist_from_state<Dist>(state), keys.begin(), keys.end(), exec_policy);
+
+    auto pairs = cuda::make_device_buffer<pair_type>(stream, device, num_keys, cuda::no_init);
+    thrust::transform(exec_policy, keys.begin(), keys.end(), pairs.begin(), [] __device__(Key const& key) {
+      return pair_type{key, Value{}};
+    });
+
+    map_type map{stream, mr, num_keys, occupancy, cudax::cuco::empty_key(Key{-1}), cudax::cuco::empty_value(Value{-1})};
+    map.insert(stream, pairs.begin(), pairs.end());
+
+    gen.dropout(keys.begin(), keys.end(), matching_rate, exec_policy);
+
+    auto result = cuda::make_device_buffer<Value>(stream, device, num_keys, cuda::no_init);
+    stream.sync();
+
+    state.add_element_count(num_keys);
+    state.exec([&](nvbench::launch& launch) {
+      map.find_async({launch.get_stream()}, keys.begin(), keys.end(), result.begin());
+    });
+  }
+}
+
+NVBENCH_BENCH_TYPES(fixed_capacity_map_find,
+                    NVBENCH_TYPE_AXES(bench::defaults::key_type_range,
+                                      bench::defaults::value_type_range,
+                                      nvbench::type_list<bench::distribution::unique>))
+  .set_name("fixed_capacity_map_find_unique_capacity")
+  .set_type_axes_names({"Key", "Value", "Distribution"})
+  .add_int64_axis("NumInputs", bench::defaults::n_range_cache)
+  .add_float64_axis("Occupancy", {bench::defaults::occupancy})
+  .add_float64_axis("MatchingRate", {bench::defaults::matching_rate});
+
+NVBENCH_BENCH_TYPES(fixed_capacity_map_find,
+                    NVBENCH_TYPE_AXES(bench::defaults::key_type_range,
+                                      bench::defaults::value_type_range,
+                                      nvbench::type_list<bench::distribution::unique>))
+  .set_name("fixed_capacity_map_find_unique_occupancy")
+  .set_type_axes_names({"Key", "Value", "Distribution"})
+  .add_int64_axis("NumInputs", {bench::defaults::n})
+  .add_float64_axis("Occupancy", bench::defaults::occupancy_range)
+  .add_float64_axis("MatchingRate", {bench::defaults::matching_rate});
+
+NVBENCH_BENCH_TYPES(fixed_capacity_map_find,
+                    NVBENCH_TYPE_AXES(bench::defaults::key_type_range,
+                                      bench::defaults::value_type_range,
+                                      nvbench::type_list<bench::distribution::unique>))
+  .set_name("fixed_capacity_map_find_unique_matching_rate")
+  .set_type_axes_names({"Key", "Value", "Distribution"})
+  .add_int64_axis("NumInputs", {bench::defaults::n})
+  .add_float64_axis("Occupancy", {bench::defaults::occupancy})
+  .add_float64_axis("MatchingRate", bench::defaults::matching_rate_range);

cccl_upstream/cudax/benchmarks/bench/cuco/fixed_capacity_map/insert.cu

@@ -0,0 +1,105 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDA Experimental in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+#include <thrust/execution_policy.h>
+#include <thrust/transform.h>
+
+#include <cuda/buffer>
+#include <cuda/memory_resource>
+#include <cuda/std/cstddef>
+#include <cuda/std/functional>
+#include <cuda/std/utility>
+#include <cuda/stream>
+
+#include <cuda/experimental/__cuco/fixed_capacity_map.cuh>
+#include <cuda/experimental/__cuco/types.cuh>
+
+#include "../common/defaults.cuh"
+#include "../common/key_generator.cuh"
+#include <nvbench/nvbench.cuh>
+
+namespace cudax = cuda::experimental;
+namespace bench = cudax::cuco::benchmark;
+
+/**
+ * @brief A benchmark evaluating `cudax::cuco::fixed_capacity_map::insert_async` performance.
+ */
+template <typename Key, typename Value, typename Dist>
+void fixed_capacity_map_insert(nvbench::state& state, nvbench::type_list<Key, Value, Dist>)
+{
+  if constexpr (sizeof(Key) != sizeof(Value))
+  {
+    state.skip("Key and Value must have the same size.");
+  }
+  else
+  {
+    using pair_type = cuda::std::pair<Key, Value>;
+    using map_type  = cudax::cuco::fixed_capacity_map<Key, Value>;
+
+    const auto num_keys  = state.get_int64("NumInputs");
+    const auto occupancy = state.get_float64("Occupancy");
+
+    const auto size = static_cast<cuda::std::size_t>(static_cast<double>(num_keys) / occupancy);
+
+    const auto device = cuda::device_ref{0};
+    cuda::stream stream{device};
+    const cuda::device_memory_pool_ref mr = cuda::device_default_memory_pool(device);
+    const auto exec_policy                = thrust::cuda::par_nosync.on(stream.get());
+
+    auto keys = cuda::make_device_buffer<Key>(stream, device, num_keys, cuda::no_init);
+
+    bench::key_generator gen{};
+    gen.generate(bench::dist_from_state<Dist>(state), keys.begin(), keys.end(), exec_policy);
+
+    auto pairs = cuda::make_device_buffer<pair_type>(stream, device, num_keys, cuda::no_init);
+    thrust::transform(exec_policy, keys.begin(), keys.end(), pairs.begin(), [] __device__(Key const& key) {
+      return pair_type{key, Value{}};
+    });
+
+    map_type map{stream, mr, size, cudax::cuco::empty_key(Key{-1}), cudax::cuco::empty_value(Value{-1})};
+    stream.sync();
+
+    state.add_element_count(num_keys);
+    state.exec(nvbench::exec_tag::timer, [&](nvbench::launch& launch, auto& timer) {
+      timer.start();
+      map.insert_async({launch.get_stream()}, pairs.begin(), pairs.end());
+      timer.stop();
+      map.clear_async({launch.get_stream()});
+    });
+  }
+}
+
+NVBENCH_BENCH_TYPES(fixed_capacity_map_insert,
+                    NVBENCH_TYPE_AXES(bench::defaults::key_type_range,
+                                      bench::defaults::value_type_range,
+                                      nvbench::type_list<bench::distribution::unique>))
+  .set_name("fixed_capacity_map_insert_unique_capacity")
+  .set_type_axes_names({"Key", "Value", "Distribution"})
+  .add_int64_axis("NumInputs", bench::defaults::n_range_cache)
+  .add_float64_axis("Occupancy", {bench::defaults::occupancy});
+
+NVBENCH_BENCH_TYPES(fixed_capacity_map_insert,
+                    NVBENCH_TYPE_AXES(bench::defaults::key_type_range,
+                                      bench::defaults::value_type_range,
+                                      nvbench::type_list<bench::distribution::unique>))
+  .set_name("fixed_capacity_map_insert_unique_occupancy")
+  .set_type_axes_names({"Key", "Value", "Distribution"})
+  .add_int64_axis("NumInputs", {bench::defaults::n})
+  .add_float64_axis("Occupancy", bench::defaults::occupancy_range);
+
+NVBENCH_BENCH_TYPES(fixed_capacity_map_insert,
+                    NVBENCH_TYPE_AXES(bench::defaults::key_type_range,
+                                      bench::defaults::value_type_range,
+                                      nvbench::type_list<bench::distribution::uniform>))
+  .set_name("fixed_capacity_map_insert_uniform_multiplicity")
+  .set_type_axes_names({"Key", "Value", "Distribution"})
+  .add_int64_axis("NumInputs", {bench::defaults::n})
+  .add_float64_axis("Occupancy", {bench::defaults::occupancy})
+  .add_float64_axis("Multiplicity", bench::defaults::multiplicity_range);

cccl_upstream/cudax/benchmarks/bench/cuco/hashers.cu

@@ -0,0 +1,131 @@
+// SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+// SPDX-License-Identifier: SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+#include <thrust/device_vector.h>
+
+#include <cuda/std/cstddef>
+#include <cuda/std/cstdint>
+
+#include <cuda/experimental/__cuco/hash_functions.cuh>
+
+#include <nvbench/nvbench.cuh>
+#include <nvbench/range.cuh>
+
+namespace cudax = cuda::experimental;
+
+// repeat hash computation n times
+static constexpr auto n_repeats = 100;
+
+template <cuda::std::int32_t Words>
+struct large_key
+{
+  constexpr __host__ __device__ large_key(cuda::std::int32_t seed) noexcept
+  {
+    for (cuda::std::int32_t i = 0; i < Words; ++i)
+    {
+      data_[i] = seed;
+    }
+  }
+
+private:
+  cuda::std::int32_t data_[Words];
+};
+
+template <cuda::std::int32_t BlockSize, typename Key, typename Hasher, typename OutputIt>
+__global__ void hash_bench_kernel(Hasher hash, size_t n, OutputIt out, bool materialize_result)
+{
+  size_t const gid         = static_cast<size_t>(BlockSize) * blockIdx.x + threadIdx.x;
+  size_t const loop_stride = static_cast<size_t>(gridDim.x) * BlockSize;
+  size_t idx               = gid;
+  using result_t           = decltype(hash(0));
+
+  result_t agg{};
+
+  while (idx < n)
+  {
+    Key key(idx);
+    for (cuda::std::int32_t i = 0; i < n_repeats; ++i)
+    { // execute hash func n times
+      agg += hash(key);
+    }
+    idx += loop_stride;
+  }
+
+  if (materialize_result)
+  {
+    out[gid] = agg;
+  }
+}
+
+// benchmark evaluating performance of various hash functions
+template <typename HasherTag, typename Key>
+void hash_eval(nvbench::state& state, nvbench::type_list<HasherTag, Key>)
+{
+  using Hash = typename HasherTag::template fn<Key>;
+
+  bool const materialize_result = false;
+  constexpr auto block_size     = 128;
+  auto const num_keys           = state.get_int64("NumInputs");
+  auto const grid_size          = (num_keys + block_size * 16 - 1) / block_size * 16;
+  using result_t                = decltype(std::declval<Hash>()(std::declval<cuda::std::int32_t>()));
+
+  thrust::device_vector<result_t> hash_values((materialize_result) ? num_keys : 1);
+
+  state.add_element_count(num_keys);
+
+  state.exec([&](nvbench::launch& launch) {
+    hash_bench_kernel<block_size, Key>
+      <<<grid_size, block_size, 0, launch.get_stream()>>>(Hash{}, num_keys, hash_values.begin(), materialize_result);
+  });
+}
+
+struct xxhash_32_tag
+{
+  template <typename Key>
+  using fn = cudax::cuco::hash<Key, cudax::cuco::hash_algorithm::xxhash_32>;
+};
+
+struct xxhash_64_tag
+{
+  template <typename Key>
+  using fn = cudax::cuco::hash<Key, cudax::cuco::hash_algorithm::xxhash_64>;
+};
+
+struct murmurhash3_32_tag
+{
+  template <typename Key>
+  using fn = cudax::cuco::hash<Key, cudax::cuco::hash_algorithm::murmurhash3_32>;
+};
+
+#if _CCCL_HAS_INT128()
+
+struct murmurhash3_x86_128_tag
+{
+  template <typename Key>
+  using fn = cudax::cuco::hash<Key, cudax::cuco::hash_algorithm::murmurhash3_x86_128>;
+};
+
+struct murmurhash3_x64_128_tag
+{
+  template <typename Key>
+  using fn = cudax::cuco::hash<Key, cudax::cuco::hash_algorithm::murmurhash3_x64_128>;
+};
+
+#endif // _CCCL_HAS_INT128()
+
+NVBENCH_BENCH_TYPES(
+  hash_eval,
+  NVBENCH_TYPE_AXES(
+    nvbench::type_list<xxhash_32_tag,
+                       xxhash_64_tag,
+                       murmurhash3_32_tag
+#if _CCCL_HAS_INT128()
+                       ,
+                       murmurhash3_x86_128_tag,
+                       murmurhash3_x64_128_tag
+#endif // _CCCL_HAS_INT128()
+                       >,
+    nvbench::type_list<cuda::std::int32_t, large_key<4>, large_key<8>, large_key<16>, large_key<32>>))
+  .set_name("hash_function_eval")
+  .set_type_axes_names({"Hash", "Key"})
+  .add_int64_power_of_two_axis("NumInputs", nvbench::range(18, 26, 4));

cccl_upstream/cudax/benchmarks/bench/cuco/hyperloglog.cu

@@ -0,0 +1,133 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDA Experimental in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+#include <thrust/execution_policy.h>
+#include <thrust/sequence.h>
+
+#include <cuda/buffer>
+#include <cuda/memory_resource>
+#include <cuda/std/cmath>
+#include <cuda/std/cstddef>
+#include <cuda/stream>
+
+#include <cuda/experimental/__cuco/hyperloglog.cuh>
+
+#include "common/defaults.cuh"
+#include <nvbench/nvbench.cuh>
+
+namespace cudax = cuda::experimental;
+namespace bench = cudax::cuco::benchmark;
+
+namespace
+{
+template <typename Key>
+void add_relative_error_summary(
+  nvbench::state& state,
+  cudax::cuco::hyperloglog<Key>& estimator,
+  cuda::stream_ref stream,
+  Key* first,
+  cuda::std::size_t num_items)
+{
+  estimator.add(stream, first, first + num_items);
+  const auto estimated_cardinality = estimator.estimate(stream);
+  const auto relative_error =
+    cuda::std::abs(static_cast<double>(estimated_cardinality) / static_cast<double>(num_items) - 1.0);
+  estimator.clear(stream);
+
+  auto& summary = state.add_summary("RelativeError");
+  summary.set_string("hint", "RelErr");
+  summary.set_string("short_name", "RelativeError");
+  summary.set_string("description", "Relative approximation error.");
+  summary.set_float64("value", relative_error);
+}
+} // namespace
+
+/**
+ * @brief A benchmark evaluating `cudax::cuco::hyperloglog` end-to-end performance.
+ */
+template <typename Key>
+void hyperloglog_e2e(nvbench::state& state, nvbench::type_list<Key>)
+{
+  using estimator_type      = cudax::cuco::hyperloglog<Key>;
+  using sketch_size_kb_type = typename estimator_type::sketch_size_kb;
+
+  const auto num_items      = static_cast<cuda::std::size_t>(state.get_int64("NumInputs"));
+  const auto sketch_size_kb = sketch_size_kb_type{static_cast<double>(state.get_int64("SketchSizeKB"))};
+
+  const auto device = cuda::device_ref{0};
+  cuda::stream stream{device};
+  const cuda::device_memory_pool_ref mr = cuda::device_default_memory_pool(device);
+
+  auto items = cuda::make_device_buffer<Key>(stream, device, num_items, cuda::no_init);
+  thrust::sequence(thrust::cuda::par_nosync.on(stream.get()), items.begin(), items.end(), Key{0});
+
+  estimator_type estimator{stream, mr, sketch_size_kb};
+  stream.sync();
+
+  state.add_element_count(num_items);
+  state.add_global_memory_reads<Key>(num_items, "InputSize");
+
+  add_relative_error_summary(state, estimator, stream, items.data(), num_items);
+
+  state.exec(nvbench::exec_tag::sync | nvbench::exec_tag::timer, [&](nvbench::launch& launch, auto& timer) {
+    timer.start();
+    estimator.add_async({launch.get_stream()}, items.begin(), items.end());
+    [[maybe_unused]] const auto estimated_cardinality = estimator.estimate({launch.get_stream()});
+    timer.stop();
+
+    estimator.clear_async({launch.get_stream()});
+  });
+}
+
+/**
+ * @brief A benchmark evaluating `cudax::cuco::hyperloglog::add_async` performance.
+ */
+template <typename Key>
+void hyperloglog_add(nvbench::state& state, nvbench::type_list<Key>)
+{
+  using estimator_type      = cudax::cuco::hyperloglog<Key>;
+  using sketch_size_kb_type = typename estimator_type::sketch_size_kb;
+
+  const auto num_items      = static_cast<cuda::std::size_t>(state.get_int64("NumInputs"));
+  const auto sketch_size_kb = sketch_size_kb_type{static_cast<double>(state.get_int64("SketchSizeKB"))};
+
+  const auto device = cuda::device_ref{0};
+  cuda::stream stream{device};
+  const cuda::device_memory_pool_ref mr = cuda::device_default_memory_pool(device);
+
+  auto items = cuda::make_device_buffer<Key>(stream, device, num_items, cuda::no_init);
+  thrust::sequence(thrust::cuda::par_nosync.on(stream.get()), items.begin(), items.end(), Key{0});
+
+  estimator_type estimator{stream, mr, sketch_size_kb};
+  stream.sync();
+
+  state.add_element_count(num_items);
+  state.add_global_memory_reads<Key>(num_items, "InputSize");
+
+  state.exec(nvbench::exec_tag::timer, [&](nvbench::launch& launch, auto& timer) {
+    timer.start();
+    estimator.add_async({launch.get_stream()}, items.begin(), items.end());
+    timer.stop();
+
+    estimator.clear_async({launch.get_stream()});
+  });
+}
+
+NVBENCH_BENCH_TYPES(hyperloglog_e2e, NVBENCH_TYPE_AXES(bench::defaults::key_type_range))
+  .set_name("hyperloglog_e2e")
+  .set_type_axes_names({"Key"})
+  .add_int64_power_of_two_axis("NumInputs", {30})
+  .add_int64_axis("SketchSizeKB", {8, 16, 32, 64, 128, 256});
+
+NVBENCH_BENCH_TYPES(hyperloglog_add, NVBENCH_TYPE_AXES(bench::defaults::key_type_range))
+  .set_name("hyperloglog_add")
+  .set_type_axes_names({"Key"})
+  .add_int64_power_of_two_axis("NumInputs", {30})
+  .add_int64_axis("SketchSizeKB", {8, 16, 32, 64, 128, 256});

cccl_upstream/cudax/cmake/cudaxAddSubdir.cmake

@@ -0,0 +1 @@
+cccl_add_subdir_helper(cudax)

cccl_upstream/cudax/cmake/cudaxBuildCompilerTargets.cmake

@@ -0,0 +1,65 @@
+# Including this file defines the following targets:
+#
+# cudax.compiler_interface
+# - Interface target that includes all compiler settings for cudax tests, etc.
+
+cccl_get_cub()
+cccl_get_cudax()
+cccl_get_libcudacxx()
+cccl_get_thrust()
+
+set(cuda_compile_options)
+set(cxx_compile_options)
+set(cxx_compile_definitions)
+
+if ("MSVC" STREQUAL "${CMAKE_CXX_COMPILER_ID}")
+  # C4848: support for attribute 'msvc::no_unique_address' in C++17 and earlier is a vendor extension
+  append_option_if_available("/wd4848" cxx_compile_options)
+
+  # XXX Temporary hack for STF !
+  # C4267: conversion from 'meow' to 'purr', possible loss of data
+  append_option_if_available("/wd4267" cxx_compile_options)
+
+  # C4459 : declaration of 'identifier' hides global declaration
+  # We work around std::chrono::last which hides some internal "last" variable
+  append_option_if_available("/wd4459" cxx_compile_options)
+
+  # stf used getenv which is potentially unsafe but not in our context
+  list(APPEND cxx_compile_definitions "_CRT_SECURE_NO_WARNINGS")
+endif()
+
+if ("Clang" STREQUAL "${CMAKE_CXX_COMPILER_ID}")
+  # stf heavily uses host device lambdas which break on clang due to a warning about the implicitly
+  # deleted copy constructor
+  # TODO(bgruber): remove this when NVBug 4980157 is resolved
+  append_option_if_available("-Wno-deprecated-copy" cxx_compile_options)
+endif()
+
+list(APPEND cxx_compile_definitions CCCL_ENABLE_ASSERTIONS)
+
+# Some groups related experimental code is located directly in libcu++ and is guarded by
+# _CUDAX_ENABLE_GROUP_FEATURES_IN_LIBCUDACXX macro, otherwise it would lead to a lot of code duplication. We define this
+# macro for cudax code globally, to get access to get access to the code.
+#
+# Can be removed once groups are no longer experimental.
+list(APPEND cxx_compile_definitions _CUDAX_ENABLE_GROUP_FEATURES_IN_LIBCUDACXX)
+
+cccl_build_compiler_interface(
+  cudax.compiler_flags
+  "${cuda_compile_options}"
+  "${cxx_compile_options}"
+  "${cxx_compile_definitions}"
+)
+
+add_library(cudax.compiler_interface INTERFACE)
+target_link_libraries(
+  cudax.compiler_interface
+  INTERFACE
+    # order matters here, we need the cudax options to override the cccl options.
+    cccl.compiler_interface
+    cudax.compiler_flags
+    libcudacxx::libcudacxx
+    CUB::CUB
+    Thrust::Thrust
+    cudax::cudax
+)

cccl_upstream/cudax/cmake/cudaxHeaderTesting.cmake

@@ -0,0 +1,109 @@
+# For every public header, build a translation unit containing `#include <header>`
+# to let the compiler try to figure out warnings in that header if it is not otherwise
+# included in tests, and also to verify if the headers are modular enough.
+# .inl files are not globbed for, because they are not supposed to be used as public
+# entrypoints.
+
+cccl_get_cudatoolkit()
+
+# Meta target for all configs' header builds:
+add_custom_target(cudax.all.headers)
+
+function(cudax_add_header_test label definitions)
+  ###################
+  # Non-STF headers #
+  set(headertest_target cudax.headers.${label}.no_stf)
+  cccl_generate_header_tests(
+    ${headertest_target}
+    cudax/include
+    # The cudax header template removes the check for the `small` macro.
+    HEADER_TEMPLATE "${cudax_SOURCE_DIR}/cmake/header_test.in.cu"
+    GLOBS "cuda/experimental/*.cuh"
+    EXCLUDES
+      # The following internal headers are not required to compile independently:
+      "cuda/experimental/__execution/prologue.cuh"
+      "cuda/experimental/__execution/epilogue.cuh"
+      # cuFile headers are compiled separately:
+      "cuda/experimental/cufile.cuh"
+      "cuda/experimental/__cufile/*"
+      # Places headers are compiled separately:
+      "cuda/experimental/places.cuh"
+      "cuda/experimental/__places/*"
+      # STF headers are compiled separately:
+      "cuda/experimental/stf.cuh"
+      "cuda/experimental/__stf/*"
+  )
+  target_link_libraries(${headertest_target} PUBLIC cudax.compiler_interface)
+
+  if (cudax_ENABLE_CUFILE)
+    ###############
+    # cuFile headers #
+    set(headertest_target cudax.headers.${label}.cufile)
+    cccl_generate_header_tests(
+      ${headertest_target}
+      cudax/include
+      HEADER_TEMPLATE "${cudax_SOURCE_DIR}/cmake/header_test.in.cu"
+      GLOBS #
+        "cuda/experimental/cufile.cuh"
+        "cuda/experimental/__cufile/*.cuh"
+    )
+    target_link_libraries(${headertest_target} PUBLIC cudax.compiler_interface)
+  endif()
+
+  # FIXME: Enable MSVC
+  if (cudax_ENABLE_PLACES AND NOT "MSVC" STREQUAL "${CMAKE_CXX_COMPILER_ID}")
+    ##################
+    # Places headers #
+    set(headertest_target cudax.headers.${label}.places)
+    cccl_generate_header_tests(
+      ${headertest_target}
+      cudax/include
+      GLOBS #
+        "cuda/experimental/places.cuh"
+        "cuda/experimental/__places/*.cuh"
+      HEADER_TEMPLATE "${cudax_SOURCE_DIR}/cmake/header_test.in.cu"
+    )
+    target_link_libraries(${headertest_target} PUBLIC cudax.compiler_interface)
+    target_compile_options(
+      ${headertest_target}
+      PRIVATE
+        $<$<COMPILE_LANG_AND_ID:CUDA,NVIDIA>:--extended-lambda>
+        $<$<COMPILE_LANG_AND_ID:CUDA,NVIDIA>:--expt-relaxed-constexpr>
+    )
+  endif()
+
+  # FIXME: Enable MSVC
+  if (cudax_ENABLE_CUDASTF AND NOT "MSVC" STREQUAL "${CMAKE_CXX_COMPILER_ID}")
+    ###############
+    # STF headers #
+    set(headertest_target cudax.headers.${label}.stf)
+    cccl_generate_header_tests(
+      ${headertest_target}
+      cudax/include
+      GLOBS #
+        "cuda/experimental/stf.cuh"
+        "cuda/experimental/__stf/*.cuh"
+      # FIXME: The cudax header template removes the check for the `small` macro.
+      # cuda/experimental/__stf/utility/memory.cuh defines functions named `small`.
+      # These should be renamed to avoid conflicts with windows system headers, and
+      # the following line removed:
+      HEADER_TEMPLATE "${cudax_SOURCE_DIR}/cmake/header_test.in.cu"
+    )
+    target_link_libraries(
+      ${headertest_target}
+      PUBLIC cudax.compiler_interface CUDA::cuda_driver
+    )
+    target_compile_options(
+      ${headertest_target}
+      PRIVATE
+        # Required by stf headers:
+        $<$<COMPILE_LANG_AND_ID:CUDA,NVIDIA>:--extended-lambda>
+        # FIXME: We should be able to refactor away from needing this by
+        # using _CCCL_HOST_DEVICE and friends + `::cuda::std` utilities where
+        # necessary.
+        $<$<COMPILE_LANG_AND_ID:CUDA,NVIDIA>:--expt-relaxed-constexpr>
+    )
+  endif()
+endfunction()
+
+cudax_add_header_test(basic "")

cccl_upstream/cudax/cmake/cudaxPlacesConfigureTarget.cmake

@@ -0,0 +1,23 @@
+# Configures a target for the Places framework.
+function(cudax_places_configure_target target_name)
+  target_link_libraries(
+    ${target_name}
+    PRIVATE #
+      CUDA::cudart_static
+      CUDA::cuda_driver
+  )
+
+  target_compile_options(
+    ${target_name}
+    PRIVATE
+      $<$<COMPILE_LANG_AND_ID:CUDA,NVIDIA>:--extended-lambda>
+      $<$<COMPILE_LANG_AND_ID:CUDA,NVIDIA>:--expt-relaxed-constexpr>
+  )
+
+  set_target_properties(
+    ${target_name}
+    PROPERTIES #
+      CUDA_RUNTIME_LIBRARY Static
+      CUDA_SEPARABLE_COMPILATION ON
+  )
+endfunction()

cccl_upstream/cudax/cmake/cudaxSTFConfigureTarget.cmake

@@ -0,0 +1,58 @@
+# Configures a target for the STF framework.
+function(cudax_stf_configure_target target_name)
+  set(options LINK_MATHLIBS)
+  set(oneValueArgs)
+  set(multiValueArgs)
+  cmake_parse_arguments(
+    CSCT
+    "${options}"
+    "${oneValueArgs}"
+    "${multiValueArgs}"
+    ${ARGN}
+  )
+
+  target_link_libraries(
+    ${target_name}
+    PRIVATE #
+      CUDA::cudart_static
+      CUDA::curand
+      CUDA::cuda_driver
+  )
+
+  if (cudax_ENABLE_CUDASTF_CODE_GENERATION)
+    target_compile_options(
+      ${target_name}
+      PRIVATE $<$<COMPILE_LANG_AND_ID:CUDA,NVIDIA>:--extended-lambda>
+    )
+  else()
+    target_compile_definitions(
+      ${target_name}
+      PRIVATE "CUDASTF_DISABLE_CODE_GENERATION"
+    )
+  endif()
+
+  target_compile_options(
+    ${target_name}
+    PRIVATE $<$<COMPILE_LANG_AND_ID:CUDA,NVIDIA>:--expt-relaxed-constexpr>
+  )
+
+  set_target_properties(
+    ${target_name}
+    PROPERTIES #
+      CUDA_RUNTIME_LIBRARY Static
+      CUDA_SEPARABLE_COMPILATION ON
+  )
+
+  if (CSCT_LINK_MATHLIBS)
+    target_link_libraries(
+      ${target_name}
+      PRIVATE #
+        CUDA::cublas
+        CUDA::cusolver
+    )
+  endif()
+
+  if (cudax_ENABLE_CUDASTF_BOUNDSCHECK)
+    target_compile_definitions(${target_name} PRIVATE "CUDASTF_BOUNDSCHECK")
+  endif()
+endfunction()

cccl_upstream/cudax/cmake/header_test.in.cu

@@ -0,0 +1,66 @@
+// This source file checks that:
+// 1) Header <@header@> compiles without error.
+// 2) Common macro collisions with platform/system headers are avoided.
+// 3) half/bf16 aren't included when these are explicitly disabled.
+
+// Define CUDAX_MACRO_CHECK(macro, header), which emits a diagnostic indicating
+// a potential macro collision and halts.
+//
+// Use raw platform checks instead of the CCCL macros since we
+// don't want to #include any headers other than the one being tested.
+//
+// This is only implemented for MSVC/GCC/Clang.
+#if defined(_MSC_VER) // MSVC
+
+// Fake up an error for MSVC
+#  define CUDAX_MACRO_CHECK_IMPL(msg)             \
+    /* Print message that looks like an error: */ \
+    __pragma(message(__FILE__ ":" CUDAX_MACRO_CHECK_IMPL0(__LINE__) ": error: " #msg)) static_assert(false, #msg);
+#  define CUDAX_MACRO_CHECK_IMPL0(x) CUDAX_MACRO_CHECK_IMPL1(x)
+#  define CUDAX_MACRO_CHECK_IMPL1(x) #x
+
+#elif defined(__clang__) || defined(__GNUC__)
+
+// GCC/clang are easy:
+#  define CUDAX_MACRO_CHECK_IMPL(msg)   CUDAX_MACRO_CHECK_IMPL0(GCC error #msg)
+#  define CUDAX_MACRO_CHECK_IMPL0(expr) _Pragma(#expr)
+
+#endif
+
+// Hacky way to build a string, but it works on all tested platforms.
+#define CUDAX_MACRO_CHECK(MACRO, HEADER) \
+  CUDAX_MACRO_CHECK_IMPL(Identifier MACRO should not be used from CCCL headers due to conflicts with HEADER macros.)
+
+// complex.h conflicts
+#define I CUDAX_MACRO_CHECK('I', complex.h)
+
+// windows.h conflicts
+// @eniebler 2024-08-30: This test is disabled because it causes build
+// failures in some configurations.
+// #define small CUDAX_MACRO_CHECK('small', windows.h)
+// We can't enable these checks without breaking some builds -- some standard
+// library implementations unconditionally `#undef` these macros, which then
+// causes random failures later.
+// Leaving these commented out as a warning: Here be dragons.
+// #define min(...) CUDAX_MACRO_CHECK('min', windows.h)
+// #define max(...) CUDAX_MACRO_CHECK('max', windows.h)
+
+// termios.h conflicts (NVIDIA/thrust#1547)
+#define B0 CUDAX_MACRO_CHECK("B0", termios.h)
+
+#include <@header@>
+
+#if defined(CCCL_DISABLE_BF16_SUPPORT)
+#  if defined(__CUDA_BF16_TYPES_EXIST__)
+#    error We should not include cuda_bf16.h when BF16 support is disabled
+#  endif // __CUDA_BF16_TYPES_EXIST__
+#endif // CCCL_DISABLE_BF16_SUPPORT
+
+#if defined(CCCL_DISABLE_FP16_SUPPORT)
+#  if defined(__CUDA_FP16_TYPES_EXIST__)
+#    error We should not include cuda_fp16.h when half support is disabled
+#  endif // __CUDA_FP16_TYPES_EXIST__
+#  if defined(__CUDA_BF16_TYPES_EXIST__)
+#    error We should not include cuda_bf16.h when half support is disabled
+#  endif // __CUDA_BF16_TYPES_EXIST__
+#endif // CCCL_DISABLE_FP16_SUPPORT

cccl_upstream/cudax/cmake/places_header_unittest.in.cu

@@ -0,0 +1,9 @@
+// This file is autogenerated by configuring places_header_unittest.in.cu.
+
+// clang-format off
+#define UNITTESTED_FILE "@source@"
+
+#include <cuda/experimental/__stf/utility/unittest.cuh>
+
+#include <@source@>
+//clang-format on

cccl_upstream/cudax/cmake/stf_header_unittest.in.cu

@@ -0,0 +1,9 @@
+// This file is autogenerated by configuring stf_header_unittest.in.cu.
+
+// clang-format off
+#define UNITTESTED_FILE "@source@"
+
+#include <cuda/experimental/__stf/utility/unittest.cuh>
+
+#include <@source@>
+//clang-format on

cccl_upstream/cudax/examples/CMakeLists.txt

@@ -0,0 +1,56 @@
+thrust_create_target(cudax.examples.thrust)
+
+function(cudax_add_example target_name_var example_src)
+  get_filename_component(example_name ${example_src} NAME_WE)
+
+  # The actual name of the test's target:
+  set(example_target cudax.example.${example_name})
+  set(${target_name_var} ${example_target} PARENT_SCOPE)
+
+  cccl_add_executable(${example_target} SOURCES "${example_src}" ADD_CTEST)
+  target_link_libraries(
+    ${example_target}
+    PRIVATE #
+      cudax.compiler_interface
+      cudax.examples.thrust
+  )
+  target_compile_options(
+    ${example_target}
+    PRIVATE
+      $<$<COMPILE_LANG_AND_ID:CUDA,NVIDIA>:--expt-relaxed-constexpr>
+      $<$<COMPILE_LANG_AND_ID:CUDA,NVIDIA>:--extended-lambda>
+  )
+  target_include_directories(
+    ${example_target}
+    PRIVATE "${CUB_SOURCE_DIR}/examples"
+  )
+endfunction()
+
+file(
+  GLOB example_srcs
+  RELATIVE "${cudax_SOURCE_DIR}/examples"
+  CONFIGURE_DEPENDS
+  *.cu
+  *.cpp
+)
+
+cccl_get_cudatoolkit()
+
+# Example requires pinned_memory_resource.
+if (CUDAToolkit_VERSION VERSION_LESS 12.9)
+  list(REMOVE_ITEM example_srcs async_buffer_add.cu cub_reduce.cu)
+endif()
+
+foreach (example_src IN LISTS example_srcs)
+  cudax_add_example(example_target "${example_src}")
+endforeach()
+
+# FIXME: Enable MSVC
+if (cudax_ENABLE_CUDASTF AND NOT "MSVC" STREQUAL "${CMAKE_CXX_COMPILER_ID}")
+  # STF examples are handled separately:
+  add_subdirectory(stf)
+endif()
+
+if (cudax_ENABLE_PLACES AND NOT "MSVC" STREQUAL "${CMAKE_CXX_COMPILER_ID}")
+  add_subdirectory(places)
+endif()

cccl_upstream/cudax/examples/async_buffer_add.cu

@@ -0,0 +1,92 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDA Experimental in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+/**
+ * Vector addition: C = A + B.
+ *
+ * This sample is a very basic sample that implements element by element
+ * vector addition. It is the same as the sample illustrating Chapter 2
+ * of the programming guide with some additions like error checking.
+ */
+
+#include <thrust/execution_policy.h>
+#include <thrust/random.h>
+#include <thrust/tabulate.h>
+#include <thrust/transform.h>
+
+#include <cuda/experimental/container.cuh>
+#include <cuda/experimental/memory_resource.cuh>
+#include <cuda/experimental/stream.cuh>
+
+#include <iostream>
+
+namespace cudax = cuda::experimental;
+
+constexpr int numElements = 50000;
+
+struct generator
+{
+  thrust::default_random_engine gen{};
+  thrust::uniform_real_distribution<float> dist{-10.0f, 10.0f};
+
+  __host__ __device__ generator(const unsigned seed)
+      : gen{seed}
+  {}
+
+  __host__ __device__ float operator()(cuda::std::size_t idx) noexcept
+  {
+    gen.discard(idx);
+    return dist(gen);
+  }
+};
+
+int main()
+{
+  // A CUDA stream on which to execute the vector addition kernel
+  cudax::stream stream{cuda::device_ref{0}};
+
+  // The execution policy we want to use to run all work on the same stream
+  auto policy = thrust::cuda::par_nosync.on(stream.get());
+
+  cuda::device_memory_pool_ref device_resource = cuda::device_default_memory_pool(cuda::device_ref{0});
+
+  // Allocate the two inputs and output, but do not zero initialize via `cuda::no_init`
+  cuda::device_buffer<float> A{stream, device_resource, numElements, cuda::no_init};
+  cuda::device_buffer<float> B{stream, device_resource, numElements, cuda::no_init};
+  cuda::device_buffer<float> C{stream, device_resource, numElements, cuda::no_init};
+
+  // Fill both vectors on stream using a random number generator
+  thrust::tabulate(policy, A.begin(), A.end(), generator{42});
+  thrust::tabulate(policy, B.begin(), B.end(), generator{1337});
+
+  // Add the vectors together
+  thrust::transform(policy, A.begin(), A.end(), B.begin(), C.begin(), cuda::std::plus<>{});
+
+  cuda::pinned_memory_pool_ref pinned_resource = cuda::pinned_default_memory_pool();
+
+  // Verify that the result vector is correct, by copying it to host
+  cuda::host_buffer<float> h_A{stream, pinned_resource, A};
+  cuda::host_buffer<float> h_B{stream, pinned_resource, B};
+  cuda::host_buffer<float> h_C{stream, pinned_resource, C};
+
+  // Do not forget to sync afterwards
+  stream.sync();
+
+  for (int i = 0; i < numElements; ++i)
+  {
+    if (cuda::std::abs(h_A.get_unsynchronized(i) + h_B.get_unsynchronized(i) - h_C.get_unsynchronized(i)) > 1e-5)
+    {
+      std::cerr << "Result verification failed at element " << i << "\n";
+      exit(EXIT_FAILURE);
+    }
+  }
+
+  return 0;
+}

cccl_upstream/cudax/examples/cub_reduce.cu

@@ -0,0 +1,53 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDA Experimental in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+// Example of using `cub::DeviceReduce::Reduce` with cudax environment.
+
+#include <cub/device/device_reduce.cuh>
+
+#include <cuda/experimental/container.cuh>
+#include <cuda/experimental/memory_resource.cuh>
+#include <cuda/experimental/stream.cuh>
+
+#include <iostream>
+
+namespace cudax = cuda::experimental;
+
+int main()
+{
+  constexpr int num_items = 50000;
+
+  // A CUDA stream on which to execute the reduction
+  cuda::stream stream{cuda::devices[0]};
+  cuda::device_memory_pool_ref mr = cuda::device_default_memory_pool(cuda::devices[0]);
+
+  // Allocate input and output, but do not zero initialize output (`cuda::no_init`)
+  auto d_in  = cuda::make_buffer<int>(stream, mr, num_items, 1);
+  auto d_out = cuda::make_buffer<float>(stream, mr, 1, cuda::no_init);
+
+  // An environment we use to pass all necessary information to CUB
+  cudax::env_t<cuda::mr::device_accessible> env{mr, stream};
+  auto error = cub::DeviceReduce::Reduce(d_in.begin(), d_out.begin(), num_items, cuda::std::plus{}, 0, env);
+  if (error != cudaSuccess)
+  {
+    std::cerr << "cub::DeviceReduce::Reduce failed: " << cudaGetErrorString(error) << "\n";
+    exit(EXIT_FAILURE);
+  }
+
+  auto h_out = cuda::make_buffer<float>(stream, cuda::pinned_default_memory_pool(), d_out);
+
+  stream.sync();
+
+  if (h_out.get_unsynchronized(0) != num_items)
+  {
+    std::cerr << "Result verification failed: " << h_out.get_unsynchronized(0) << " != " << num_items << "\n";
+    exit(EXIT_FAILURE);
+  }
+}

cccl_upstream/cudax/examples/places/CMakeLists.txt

@@ -0,0 +1,30 @@
+set(places_example_sources thrust_device_data_place_allocator.cu)
+
+## cudax_add_places_example
+#
+# Add a places example executable and register it with ctest.
+#
+# target_name_var: Variable name to overwrite with the name of the example
+#   target. Useful for modifying the example/target after creation.
+# source: The source file for the example.
+#
+function(cudax_add_places_example target_name_var source)
+  get_filename_component(filename ${source} NAME_WE)
+
+  set(example_target cudax.example.places.${filename})
+
+  cccl_add_executable(${example_target} SOURCES ${source} ADD_CTEST)
+  cudax_places_configure_target(${example_target})
+  target_link_libraries(
+    ${example_target}
+    PRIVATE #
+      cudax.compiler_interface
+      cudax.examples.thrust
+  )
+
+  set(${target_name_var} ${example_target} PARENT_SCOPE)
+endfunction()
+
+foreach (source IN LISTS places_example_sources)
+  cudax_add_places_example(example_target "${source}")
+endforeach()

cccl_upstream/cudax/examples/places/thrust_device_data_place_allocator.cu

@@ -0,0 +1,117 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDASTF in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+/**
+ * @file
+ *
+ * @brief Thrust device_vector with an allocator backed by a data_place.
+ *
+ * Wraps data_place::allocate/deallocate as a thrust::mr::memory_resource,
+ * then uses thrust::mr::allocator to create a compatible allocator.
+ * Storage is allocated via data_place (device, composite/VMM, or other
+ * place types). The same Thrust code works unchanged for single-device,
+ * multi-device (VMM), or green-context placement.
+ */
+
+#include <thrust/copy.h>
+#include <thrust/device_ptr.h>
+#include <thrust/device_vector.h>
+#include <thrust/execution_policy.h>
+#include <thrust/host_vector.h>
+#include <thrust/iterator/counting_iterator.h>
+#include <thrust/mr/allocator.h>
+#include <thrust/mr/memory_resource.h>
+#include <thrust/transform.h>
+
+#include <cuda/experimental/__places/partitions/blocked_partition.cuh>
+
+#include <cstdio>
+
+using namespace cuda::experimental::places;
+
+// Minimal adapter: data_place is STF's abstraction; Thrust expects a
+// memory_resource. This class bridges the two. The resource must outlive
+// any vectors/allocators that use it.
+class data_place_memory_resource final : public thrust::mr::memory_resource<thrust::device_ptr<void>>
+{
+public:
+  explicit data_place_memory_resource(const data_place& place)
+      : place_(place)
+  {}
+
+  pointer do_allocate(std::size_t bytes, std::size_t /*alignment*/) override
+  {
+    // A memory resource hands out untyped bytes, so declare the geometry
+    // explicitly as a flat byte array: composite places distribute it with
+    // byte granularity (equivalent for every other place type).
+    void* raw = place_.allocate_nd(dim4(bytes), 1);
+    return thrust::device_ptr<void>(raw);
+  }
+
+  void do_deallocate(pointer p, std::size_t bytes, std::size_t /*alignment*/) override
+  {
+    place_.deallocate(p.get(), bytes);
+  }
+
+  __host__ __device__ bool do_is_equal(const memory_resource& other) const noexcept override
+  {
+#if defined(__CUDA_ARCH__)
+    (void) other;
+    return false;
+#else
+    auto* o = dynamic_cast<const data_place_memory_resource*>(&other);
+    return o && place_ == o->place_;
+#endif
+  }
+
+private:
+  data_place place_;
+};
+
+template <typename T>
+using data_place_allocator = thrust::mr::allocator<T, data_place_memory_resource>;
+
+bool run_with_place(const data_place& place, const char* label)
+{
+  const size_t n = 1024 * 1024;
+
+  data_place_memory_resource memres(place);
+  data_place_allocator<double> alloc(&memres);
+  thrust::device_vector<double, data_place_allocator<double>> d_vec(n, 0.0, alloc);
+
+  thrust::transform(
+    thrust::device,
+    thrust::counting_iterator<size_t>(0),
+    thrust::counting_iterator<size_t>(n),
+    d_vec.begin(),
+    [] __device__(size_t i) {
+      return 2.0 * static_cast<double>(i);
+    });
+
+  thrust::host_vector<double> h_sample(4);
+  thrust::copy(d_vec.begin(), d_vec.begin() + 4, h_sample.begin());
+
+  bool ok = (h_sample[0] == 0.0 && h_sample[1] == 2.0 && h_sample[2] == 4.0 && h_sample[3] == 6.0);
+  printf(
+    "thrust_device_data_place_allocator: %s (%s): %s\n", label, place.to_string().c_str(), ok ? "PASSED" : "FAILED");
+  return ok;
+}
+
+int main()
+{
+  bool all_ok = true;
+
+  all_ok &= run_with_place(data_place::device(0), "device(0)");
+
+  all_ok &= run_with_place(data_place::composite(blocked_partition(), exec_place::all_devices()),
+                           "composite(blocked_partition, all_devices)");
+
+  return all_ok ? 0 : 1;
+}

cccl_upstream/cudax/examples/simple_p2p.cu

@@ -0,0 +1,262 @@
+/* Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *  * Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *  * Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *  * Neither the name of NVIDIA CORPORATION nor the names of its
+ *    contributors may be used to endorse or promote products derived
+ *    from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
+ * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+ * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+ * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+ * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
+ * OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+/*
+ * This sample demonstrates a combination of Peer-to-Peer (P2P) and
+ * Unified Virtual Address Space (UVA) features.
+ */
+
+#include <cuda/algorithm>
+#include <cuda/devices>
+#include <cuda/memory_pool>
+#include <cuda/memory_resource>
+
+#include <cuda/experimental/container.cuh>
+#include <cuda/experimental/launch.cuh>
+#include <cuda/experimental/memory_resource.cuh>
+
+#include <algorithm>
+#include <cstdio>
+#include <cstdlib>
+
+namespace cudax = cuda::experimental;
+
+struct simple_kernel
+{
+  template <typename Configuration>
+  __device__ void operator()(Configuration config, ::cuda::std::span<const float> src, ::cuda::std::span<float> dst)
+  {
+    // Just a dummy kernel, doing enough for us to verify that everything worked
+    const auto idx = cuda::gpu_thread.rank(cuda::grid, config);
+    dst[idx]       = src[idx] * 2.0f;
+  }
+};
+
+void print_peer_accessibility()
+{
+  // Check possibility for peer access
+  printf("\nChecking GPU(s) for support of peer to peer memory access...\n");
+
+  for (auto& dev_i : cuda::devices)
+  {
+    for (auto& dev_j : cuda::devices)
+    {
+      if (dev_i != dev_j)
+      {
+        bool can_access_peer  = dev_i.has_peer_access_to(dev_j);
+        const auto dev_i_name = dev_i.name();
+        const auto dev_j_name = dev_j.name();
+        printf("> Peer access from %.*s (GPU%d) -> %.*s (GPU%d) : %s\n",
+               static_cast<int>(dev_i_name.size()),
+               dev_i_name.data(),
+               dev_i.get(),
+               static_cast<int>(dev_j_name.size()),
+               dev_j_name.data(),
+               dev_j.get(),
+               can_access_peer ? "Yes" : "No");
+      }
+    }
+  }
+}
+
+template <typename BufferType>
+void benchmark_cross_device_ping_pong_copy(
+  cudax::stream_ref dev0_stream, cudax::stream_ref dev1_stream, BufferType& dev0_buffer, BufferType& dev1_buffer)
+{
+  // Use dev1 stream due to some surprising performance issue
+  constexpr int cpy_count = 100;
+  auto start_event        = dev1_stream.record_timed_event();
+  for (int i = 0; i < cpy_count; i++)
+  {
+    // Ping-pong copy between GPUs
+    if (i % 2 == 0)
+    {
+      cuda::copy_bytes(dev1_stream, dev0_buffer, dev1_buffer);
+    }
+    else
+    {
+      cuda::copy_bytes(dev1_stream, dev1_buffer, dev0_buffer);
+    }
+  }
+
+  auto end_event = dev1_stream.record_timed_event();
+  dev1_stream.sync();
+  cuda::std::chrono::duration<double> duration(end_event - start_event);
+  printf("Peer copy between GPU%d and GPU%d: %.2fGB/s\n",
+         dev0_stream.device().get(),
+         dev1_stream.device().get(),
+         (static_cast<float>(cpy_count * dev0_buffer.size_bytes()) / static_cast<float>(1024 * 1024 * 1024)
+          / duration.count()));
+}
+
+template <typename BufferType>
+void test_cross_device_access_from_kernel(
+  cudax::stream_ref dev0_stream, cudax::stream_ref dev1_stream, BufferType& dev0_buffer, BufferType& dev1_buffer)
+{
+  cuda::device_ref dev0 = dev0_stream.device();
+  cuda::device_ref dev1 = dev1_stream.device();
+
+  // Prepare host buffer and copy to GPU 0
+  printf("Preparing host buffer and copy to GPU%d...\n", dev0.get());
+
+  // This will be a pinned memory vector once available
+  cudax::uninitialized_buffer<float, cuda::mr::host_accessible> host_buffer(
+    cuda::mr::legacy_pinned_memory_resource(), dev0_buffer.size());
+  std::generate(host_buffer.begin(), host_buffer.end(), []() {
+    static int i = 0;
+    return static_cast<float>((i++) % 4096);
+  });
+
+  cuda::copy_bytes(dev0_stream, host_buffer, dev0_buffer);
+  dev1_stream.wait(dev0_stream);
+
+  // Kernel launch configuration
+  auto config = cuda::distribute<512>(dev0_buffer.size());
+
+  // Run kernel on GPU 1, reading input from the GPU 0 buffer, writing output to the GPU 1 buffer
+  printf("Run kernel on GPU%d, taking source data from GPU%d and writing to "
+         "GPU%d...\n",
+         dev1.get(),
+         dev0.get(),
+         dev1.get());
+  cudax::launch(dev1_stream, config, simple_kernel{}, dev0_buffer, dev1_buffer);
+  dev0_stream.wait(dev1_stream);
+
+  // Run kernel on GPU 0, reading input from the GPU 1 buffer, writing output to the GPU 0 buffer
+  printf("Run kernel on GPU%d, taking source data from GPU%d and writing to "
+         "GPU%d...\n",
+         dev0.get(),
+         dev1.get(),
+         dev0.get());
+  cudax::launch(dev0_stream, config, simple_kernel{}, dev1_buffer, dev0_buffer);
+
+  // Copy data back to host and verify
+  printf("Copy data back to host from GPU%d and verify results...\n", dev0.get());
+  cuda::copy_bytes(dev0_stream, dev0_buffer, host_buffer);
+  dev0_stream.sync();
+
+  int error_count = 0;
+  for (size_t i = 0; i < host_buffer.size(); i++)
+  {
+    cuda::std::span<float> host_span(host_buffer);
+    // Re-generate input data and apply 2x '* 2.0f' computation of both kernel runs
+    float expected = float(i % 4096) * 2.0f * 2.0f;
+    if (host_span[i] != expected)
+    {
+      printf("Verification error @ element %zu: val = %f, ref = %f\n", i, host_span[i], expected);
+
+      if (error_count++ > 10)
+      {
+        break;
+      }
+    }
+  }
+  if (error_count != 0)
+  {
+    printf("Test failed!\n");
+    exit(EXIT_FAILURE);
+  }
+}
+
+int main([[maybe_unused]] int argc, char** argv)
+try
+{
+  printf("[%s] - Starting...\n", argv[0]);
+
+  // Number of GPUs
+  printf("Checking for multiple GPUs...\n");
+  printf("CUDA-capable device count: %zu\n", cuda::devices.size());
+
+  if (cuda::devices.size() < 2)
+  {
+    printf("Two or more GPUs with Peer-to-Peer access capability are required for %s.\n", argv[0]);
+    printf("Waiving test.\n");
+    return 0;
+  }
+
+  // Print full peer access matrix
+  print_peer_accessibility();
+
+  // But use a shorthand to find all peers of a device
+  std::vector<cuda::device_ref> peers;
+  for (auto& dev : cuda::devices)
+  {
+    const auto dev_peers = dev.peers();
+    if (dev_peers.size() != 0)
+    {
+      peers.assign(dev_peers.begin(), dev_peers.end());
+      peers.insert(peers.begin(), dev);
+      break;
+    }
+  }
+
+  if (peers.size() == 0)
+  {
+    printf("Two or more GPUs with Peer-to-Peer access capability are required, waving the test.\n");
+    return 0;
+  }
+
+  cuda::stream dev0_stream(peers[0]);
+  cuda::stream dev1_stream(peers[1]);
+
+  printf("Enabling peer access between GPU%d and GPU%d...\n", peers[0].get(), peers[1].get());
+  cuda::device_memory_pool_ref dev0_resource = cuda::device_default_memory_pool(peers[0]);
+  dev0_resource.enable_access_from(peers[1]);
+  cuda::device_memory_pool_ref dev1_resource = cuda::device_default_memory_pool(peers[1]);
+  dev1_resource.enable_access_from(peers[0]);
+
+  // Allocate buffers
+  constexpr size_t buf_cnt = 1024 * 1024 * 16;
+  printf("Allocating buffers (%iMB on GPU%d, GPU%d and CPU Host)...\n",
+         int(buf_cnt / 1024 / 1024 * sizeof(float)),
+         peers[0].get(),
+         peers[1].get());
+
+  cudax::uninitialized_buffer<float, cuda::mr::device_accessible> dev0_buffer(dev0_resource, buf_cnt);
+  cudax::uninitialized_buffer<float, cuda::mr::device_accessible> dev1_buffer(dev1_resource, buf_cnt);
+
+  benchmark_cross_device_ping_pong_copy(dev0_stream, dev1_stream, dev0_buffer, dev1_buffer);
+
+  test_cross_device_access_from_kernel(dev0_stream, dev1_stream, dev0_buffer, dev1_buffer);
+
+  // Disable peer access
+  printf("Disabling peer access...\n");
+  dev0_resource.disable_access_from(peers[1]);
+  dev1_resource.disable_access_from(peers[0]);
+
+  // No cleanup needed
+  printf("Test passed\n");
+  return 0;
+}
+catch (const std::exception& e)
+{
+  printf("caught an exception: \"%s\"\n", e.what());
+}
+catch (...)
+{
+  printf("caught an unknown exception\n");
+}

cccl_upstream/cudax/examples/stdexec_stream.cu

@@ -0,0 +1,82 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDA Experimental in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+#include <cuda/experimental/execution.cuh>
+
+#include <nv/target>
+
+#include <cstdio>
+
+#include <cuda_runtime_api.h>
+
+namespace cudax = cuda::experimental;
+namespace ex    = cudax::execution;
+
+// This example demonstrates how to use the experimental CUDA implementation of
+// C++26's std::execution async tasking framework.
+
+int main()
+{
+  try
+  {
+    auto tctx = ex::thread_context{};
+    auto sctx = ex::stream_context{cuda::device_ref{0}};
+    auto gpu  = sctx.get_scheduler();
+
+    const auto bulk_shape = 10;
+    const auto bulk_fn    = [] __device__(const int index, int i) noexcept {
+      const int tid = static_cast<int>(blockIdx.x * blockDim.x + threadIdx.x);
+      if (tid < bulk_shape)
+      {
+        printf("Hello from bulk task on device! index = %d, i = %d\n", index, i);
+      }
+    };
+
+    auto start =
+      // begin work on the GPU:
+      ex::schedule(gpu)
+
+      // execute a device lambda on the GPU:
+      | ex::then([] __device__() noexcept -> int {
+          printf("Hello from lambda on device!\n");
+          return 42;
+        })
+
+      // do some parallel work on the GPU:
+      | ex::bulk(ex::par, bulk_shape, bulk_fn) //
+
+      // transfer execution back to the CPU:
+      | ex::continues_on(tctx.get_scheduler())
+
+      // execute a host/device lambda on the CPU:
+      | ex::then([] __host__ __device__(int i) noexcept -> int {
+          NV_IF_ELSE_TARGET(NV_IS_HOST,
+                            (printf("Hello from lambda on host! i = %d\n", i);),
+                            (printf("OOPS! still on the device! i = %d\n", i);))
+          return i + 1;
+        });
+
+    // run the task, wait for it to finish, and get the result
+    auto [i] = ex::sync_wait(std::move(start)).value();
+    printf("All done on the host! result = %d\n", i);
+  }
+  catch (cuda::cuda_error const& e)
+  {
+    std::printf("CUDA error: %s\n", e.what());
+  }
+  catch (std::exception const& e)
+  {
+    std::printf("Exception: %s\n", e.what());
+  }
+  catch (...)
+  {
+    std::printf("Unknown exception\n");
+  }
+}

cccl_upstream/cudax/examples/stf/01-axpy-cuda_kernel.cu

@@ -0,0 +1,73 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDASTF in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+/**
+ * @file
+ *
+ * @brief An AXPY kernel described using a cuda_kernel construct
+ *
+ */
+
+#include <cuda/experimental/stf.cuh>
+
+using namespace cuda::experimental::stf;
+
+__global__ void axpy(double a, slice<const double> x, slice<double> y)
+{
+  int tid      = blockIdx.x * blockDim.x + threadIdx.x;
+  int nthreads = gridDim.x * blockDim.x;
+
+  for (int i = tid; i < x.size(); i += nthreads)
+  {
+    y(i) += a * x(i);
+  }
+}
+
+double X0(int i)
+{
+  return sin((double) i);
+}
+
+double Y0(int i)
+{
+  return cos((double) i);
+}
+
+int main()
+{
+  context ctx    = graph_ctx();
+  const size_t N = 16;
+  double X[N], Y[N];
+
+  for (size_t i = 0; i < N; i++)
+  {
+    X[i] = X0(i);
+    Y[i] = Y0(i);
+  }
+
+  double alpha = 3.14;
+
+  auto lX = ctx.logical_data(X);
+  auto lY = ctx.logical_data(Y);
+
+  /* Compute Y = Y + alpha X */
+  ctx.cuda_kernel(lX.read(), lY.rw())->*[&](auto dX, auto dY) {
+    // axpy<<<16, 128, 0, ...>>>(alpha, dX, dY)
+    return cuda_kernel_desc{axpy, 16, 128, 0, alpha, dX, dY};
+  };
+
+  ctx.finalize();
+
+  for (size_t i = 0; i < N; i++)
+  {
+    assert(fabs(Y[i] - (Y0(i) + alpha * X0(i))) < 0.0001);
+    assert(fabs(X[i] - X0(i)) < 0.0001);
+  }
+}

cccl_upstream/cudax/examples/stf/01-axpy-cuda_kernel_chain.cu

@@ -0,0 +1,80 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDASTF in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+/**
+ * @file
+ *
+ * @brief Example of task implementing a chain of CUDA kernels
+ *
+ */
+
+#include <cuda/experimental/stf.cuh>
+
+using namespace cuda::experimental::stf;
+
+__global__ void axpy(double a, slice<const double> x, slice<double> y)
+{
+  int tid      = blockIdx.x * blockDim.x + threadIdx.x;
+  int nthreads = gridDim.x * blockDim.x;
+
+  for (int i = tid; i < x.size(); i += nthreads)
+  {
+    y(i) += a * x(i);
+  }
+}
+
+double X0(int i)
+{
+  return sin((double) i);
+}
+
+double Y0(int i)
+{
+  return cos((double) i);
+}
+
+int main()
+{
+  context ctx    = graph_ctx();
+  const size_t N = 16;
+  double X[N], Y[N];
+
+  for (size_t i = 0; i < N; i++)
+  {
+    X[i] = X0(i);
+    Y[i] = Y0(i);
+  }
+
+  double alpha = 3.14;
+  double beta  = 4.5;
+  double gamma = -4.1;
+
+  auto lX = ctx.logical_data(X);
+  auto lY = ctx.logical_data(Y);
+
+  /* Compute Y = Y + alpha X, Y = Y + beta X and then  Y = Y + gamma X */
+  ctx.cuda_kernel_chain(lX.read(), lY.rw())->*[&](auto dX, auto dY) {
+    // clang-format off
+        return std::vector<cuda_kernel_desc> {
+            { axpy, 16, 128, 0, alpha, dX, dY },
+            { axpy, 16, 128, 0, beta, dX, dY },
+            { axpy, 16, 128, 0, gamma, dX, dY }
+        };
+    // clang-format on
+  };
+
+  ctx.finalize();
+
+  for (size_t i = 0; i < N; i++)
+  {
+    assert(fabs(Y[i] - (Y0(i) + (alpha + beta + gamma) * X0(i))) < 0.0001);
+    assert(fabs(X[i] - X0(i)) < 0.0001);
+  }
+}

cccl_upstream/cudax/examples/stf/01-axpy-launch.cu

@@ -0,0 +1,62 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDASTF in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+/**
+ * @file
+ * @brief Example of AXPY kernel implemented with the launch API
+ */
+
+#include <cuda/experimental/stf.cuh>
+
+using namespace cuda::experimental::stf;
+
+double X0(int i)
+{
+  return sin((double) i);
+}
+
+double Y0(int i)
+{
+  return cos((double) i);
+}
+
+int main()
+{
+  context ctx;
+  const size_t N = 16;
+  double X[N], Y[N];
+
+  for (size_t i = 0; i < N; i++)
+  {
+    X[i] = X0(i);
+    Y[i] = Y0(i);
+  }
+
+  double alpha = 3.14;
+
+  auto lX = ctx.logical_data(X);
+  auto lY = ctx.logical_data(Y);
+
+  /* Compute Y = Y + alpha X */
+  ctx.launch(lX.read(), lY.rw())->*[=] _CCCL_DEVICE(auto t, auto dX, auto dY) {
+    for (auto ind : t.apply_partition(shape(dX)))
+    {
+      dY(ind) += alpha * dX(ind);
+    }
+  };
+
+  ctx.finalize();
+
+  for (size_t i = 0; i < N; i++)
+  {
+    assert(fabs(Y[i] - (Y0(i) + alpha * X0(i))) < 0.0001);
+    assert(fabs(X[i] - X0(i)) < 0.0001);
+  }
+}

cccl_upstream/cudax/examples/stf/01-axpy-parallel_for.cu

@@ -0,0 +1,61 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDASTF in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+/**
+ * @file
+ *
+ * @brief An AXPY kernel implemented using the parallel_for construct
+ *
+ */
+
+#include <cuda/experimental/stf.cuh>
+
+using namespace cuda::experimental::stf;
+
+double X0(int i)
+{
+  return sin((double) i);
+}
+
+double Y0(int i)
+{
+  return cos((double) i);
+}
+
+int main()
+{
+  context ctx;
+  const size_t N = 16;
+  double X[N], Y[N];
+
+  for (size_t i = 0; i < N; i++)
+  {
+    X[i] = X0(i);
+    Y[i] = Y0(i);
+  }
+
+  double alpha = 3.14;
+
+  auto lX = ctx.logical_data(X);
+  auto lY = ctx.logical_data(Y);
+
+  /* Compute Y = Y + alpha X */
+  ctx.parallel_for(lY.shape(), lX.read(), lY.rw())->*[alpha] __device__(size_t i, auto dX, auto dY) {
+    dY(i) += alpha * dX(i);
+  };
+
+  ctx.finalize();
+
+  for (size_t i = 0; i < N; i++)
+  {
+    assert(fabs(Y[i] - (Y0(i) + alpha * X0(i))) < 0.0001);
+    assert(fabs(X[i] - X0(i)) < 0.0001);
+  }
+}

cccl_upstream/cudax/examples/stf/01-axpy.cu

@@ -0,0 +1,72 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDASTF in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+/**
+ * @file
+ *
+ * @brief An AXPY kernel implemented with CUDA kernel in a task
+ *
+ */
+
+#include <cuda/experimental/stf.cuh>
+
+using namespace cuda::experimental::stf;
+
+__global__ void axpy(double a, slice<const double> x, slice<double> y)
+{
+  int tid      = blockIdx.x * blockDim.x + threadIdx.x;
+  int nthreads = gridDim.x * blockDim.x;
+
+  for (int i = tid; i < x.size(); i += nthreads)
+  {
+    y(i) += a * x(i);
+  }
+}
+
+double X0(int i)
+{
+  return sin((double) i);
+}
+
+double Y0(int i)
+{
+  return cos((double) i);
+}
+
+int main()
+{
+  context ctx;
+  const size_t N = 16;
+  double X[N], Y[N];
+
+  for (size_t i = 0; i < N; i++)
+  {
+    X[i] = X0(i);
+    Y[i] = Y0(i);
+  }
+
+  double alpha = 3.14;
+
+  auto lX = ctx.logical_data(X);
+  auto lY = ctx.logical_data(Y);
+
+  /* Compute Y = Y + alpha X */
+  ctx.task(lX.read(), lY.rw())->*[&](cudaStream_t s, auto dX, auto dY) {
+    axpy<<<16, 128, 0, s>>>(alpha, dX, dY);
+  };
+
+  ctx.finalize();
+
+  for (size_t i = 0; i < N; i++)
+  {
+    assert(fabs(Y[i] - (Y0(i) + alpha * X0(i))) < 0.0001);
+    assert(fabs(X[i] - X0(i)) < 0.0001);
+  }
+}

cccl_upstream/cudax/examples/stf/02-axpy-host_launch.cu

@@ -0,0 +1,81 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDASTF in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+/**
+ * @file
+ *
+ * @brief An AXPY kernel implemented with a task of the CUDA graph backend and
+ *        a host callback
+ *
+ * The host_launch mechanism is also illustrated
+ *
+ */
+
+#include <cuda/experimental/stf.cuh>
+
+using namespace cuda::experimental::stf;
+
+__global__ void axpy(double a, slice<const double> x, slice<double> y)
+{
+  int tid      = blockIdx.x * blockDim.x + threadIdx.x;
+  int nthreads = gridDim.x * blockDim.x;
+
+  for (int i = tid; i < x.size(); i += nthreads)
+  {
+    y(i) += a * x(i);
+  }
+}
+
+double X0(int i)
+{
+  return sin((double) i);
+}
+
+double Y0(int i)
+{
+  return cos((double) i);
+}
+
+int main()
+{
+  graph_ctx ctx;
+  const size_t N = 16;
+  double X[N], Y[N];
+
+  for (size_t i = 0; i < N; i++)
+  {
+    X[i] = X0(i);
+    Y[i] = Y0(i);
+  }
+
+  double alpha = 3.14;
+
+  auto lX = ctx.logical_data(X);
+  auto lY = ctx.logical_data(Y);
+
+  /* Compute Y = Y + alpha X */
+  ctx.task(lX.read(), lY.rw())->*[&](cudaStream_t s, auto dX, auto dY) {
+    axpy<<<16, 128, 0, s>>>(alpha, dX, dY);
+  };
+
+  /* Asynchronously check the result on the host */
+  ctx.host_launch(lX.read(), lY.read())->*[&](auto hX, auto hY) {
+    for (size_t ind = 0; ind < hX.extent(0); ind++)
+    {
+      // Y should be Y0 + alpha X0
+      EXPECT(fabs(hY(ind) - (Y0(ind) + alpha * X0(ind))) < 0.0001);
+
+      // X should be X0
+      EXPECT(fabs(hX(ind) - X0(ind)) < 0.0001);
+    }
+  };
+
+  ctx.finalize();
+}

cccl_upstream/cudax/examples/stf/03-temporary-data.cu

@@ -0,0 +1,69 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDASTF in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+/**
+ * @file
+ *
+ * @brief This example illustrates how we can create temporary data from shapes, and use them in tasks
+ */
+
+#include <cuda/experimental/stf.cuh>
+
+using namespace cuda::experimental::stf;
+
+int main()
+{
+  const int n = 4096;
+  int X[n];
+  int Y[n];
+
+  for (size_t i = 0; i < n; i++)
+  {
+    X[i] = 3 * i;
+    Y[i] = 2 * i - 3;
+  }
+
+  context ctx;
+
+  auto lX = ctx.logical_data(X);
+  auto lY = ctx.logical_data(Y);
+
+  // Select an odd number
+  int niter = 19;
+  assert(niter % 2 == 1);
+
+  for (int iter = 0; iter < niter; iter++)
+  {
+    // We here define a temporary vector with the same shape as X, for which there is no existing copy
+    // This data handle has a limited scope, so that it is automatically destroyed at each iteration of the loop
+    auto tmp = ctx.logical_data(lX.shape());
+
+    ctx.task(lY.rw(), lX.rw(), tmp.write())->*[](cudaStream_t s, auto sY, auto sX, auto sTMP) {
+      // We swap X and Y using TMP as temporary buffer
+      // TMP = X
+      cuda_safe_call(
+        cudaMemcpyAsync(sTMP.data_handle(), sX.data_handle(), n * sizeof(int), cudaMemcpyDeviceToDevice, s));
+      // X = Y
+      cuda_safe_call(cudaMemcpyAsync(sX.data_handle(), sY.data_handle(), n * sizeof(int), cudaMemcpyDeviceToDevice, s));
+      // Y = TMP
+      cuda_safe_call(
+        cudaMemcpyAsync(sY.data_handle(), sTMP.data_handle(), n * sizeof(int), cudaMemcpyDeviceToDevice, s));
+    };
+  }
+
+  ctx.finalize();
+
+  // We have exchanged an odd number of times, so they must be inverted
+  for (size_t i = 0; i < n; i++)
+  {
+    assert(X[i] == 2 * i - 3);
+    assert(Y[i] == 3 * i);
+  }
+}

cccl_upstream/cudax/examples/stf/04-fibonacci-run_once.cu

@@ -0,0 +1,79 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDASTF in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+/**
+ * @file
+ *
+ * @brief An example of Fibonacci sequence illustrating how we can use
+ *        dynamically created logical data and the run_once utility
+ */
+
+#include <cuda/experimental/stf.cuh>
+
+using namespace cuda::experimental::stf;
+
+int fibo_ref(int n)
+{
+  if (n < 2)
+  {
+    return n;
+  }
+  else
+  {
+    return fibo_ref(n - 1) + fibo_ref(n - 2);
+  }
+}
+
+__global__ void add(slice<int> out, const slice<const int> in1, const slice<const int> in2)
+{
+  out(0) = in1(0) + in2(0);
+}
+
+__global__ void set(slice<int> out, int val)
+{
+  out(0) = val;
+}
+
+logical_data<slice<int>> compute_fibo(context& ctx, int n)
+{
+  // The result for a given value n is memoized in a logical_data that will be reused every time we compute the same
+  // value
+  return run_once(n)->*[&](int n) {
+    auto result = ctx.logical_data(shape_of<slice<int>>(1)).set_symbol(std::to_string(n));
+    if (n < 2)
+    {
+      ctx.task(result.write()).set_symbol("fibo" + std::to_string(n))->*[=](cudaStream_t s, auto sresult) {
+        set<<<1, 1, 0, s>>>(sresult, n);
+      };
+    }
+    else
+    {
+      auto fib2 = compute_fibo(ctx, n - 2);
+      auto fib1 = compute_fibo(ctx, n - 1);
+      ctx.task(fib1.read(), fib2.read(), result.write()).set_symbol("fibo" + std::to_string(n))
+          ->*[=](cudaStream_t s, auto s1, auto s2, auto sresult) {
+                add<<<1, 1, 0, s>>>(sresult, s1, s2);
+              };
+    }
+    return result;
+  };
+}
+
+int main(int argc, char** argv)
+{
+  int n = (argc > 1) ? atoi(argv[1]) : 4;
+
+  context ctx;
+  auto result = compute_fibo(ctx, n);
+  ctx.host_launch(result.read())->*[&](auto res) {
+    EXPECT(res(0) == fibo_ref(n));
+  };
+  ctx.finalize();
+}

cccl_upstream/cudax/examples/stf/04-fibonacci.cu

@@ -0,0 +1,75 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDASTF in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+/**
+ * @file
+ *
+ * @brief An example of Fibonacci sequence illustrating how we can use
+ *        dynamically created logical data
+ */
+
+#include <cuda/experimental/stf.cuh>
+
+using namespace cuda::experimental::stf;
+
+int fibo_ref(int n)
+{
+  if (n < 2)
+  {
+    return n;
+  }
+  else
+  {
+    return fibo_ref(n - 1) + fibo_ref(n - 2);
+  }
+}
+
+__global__ void add(slice<int> out, const slice<const int> in1, const slice<const int> in2)
+{
+  out(0) = in1(0) + in2(0);
+}
+
+__global__ void set(slice<int> out, int val)
+{
+  out(0) = val;
+}
+
+logical_data<slice<int>> compute_fibo(context& ctx, int n)
+{
+  auto out = ctx.logical_data(shape_of<slice<int>>(1));
+  if (n < 2)
+  {
+    ctx.task(out.write())->*[=](cudaStream_t s, auto sout) {
+      set<<<1, 1, 0, s>>>(sout, n);
+    };
+  }
+  else
+  {
+    auto fib1 = compute_fibo(ctx, n - 1);
+    auto fib2 = compute_fibo(ctx, n - 2);
+    ctx.task(fib1.read(), fib2.read(), out.write())->*[=](cudaStream_t s, auto s1, auto s2, auto sout) {
+      add<<<1, 1, 0, s>>>(sout, s1, s2);
+    };
+  }
+
+  return out;
+}
+
+int main(int argc, char** argv)
+{
+  int n = (argc > 1) ? atoi(argv[1]) : 4;
+
+  context ctx; // = graph_ctx();
+  auto result = compute_fibo(ctx, n);
+  ctx.host_launch(result.read())->*[&](auto res) {
+    EXPECT(res(0) == fibo_ref(n));
+  };
+  ctx.finalize();
+}

cccl_upstream/cudax/examples/stf/08-cub-reduce.cu

@@ -0,0 +1,95 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDASTF in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+/**
+ * @file
+ * @brief Example of reduction implementing using CUB kernels
+ */
+
+#include <thrust/device_vector.h>
+
+#include <cuda/experimental/stf.cuh>
+
+using namespace cuda::experimental::stf;
+
+template <int BLOCK_THREADS, typename T>
+__global__ void reduce(slice<const T> values, slice<T> partials, size_t nelems)
+{
+  using namespace cub;
+  typedef BlockReduce<T, BLOCK_THREADS> BlockReduceT;
+
+  auto thread_id = BLOCK_THREADS * blockIdx.x + threadIdx.x;
+
+  // Local reduction
+  T local_sum = 0;
+  for (size_t ind = thread_id; ind < nelems; ind += blockDim.x * gridDim.x)
+  {
+    local_sum += values(ind);
+  }
+
+  __shared__ typename BlockReduceT::TempStorage temp_storage;
+
+  // Per-thread tile data
+  T result = BlockReduceT(temp_storage).Sum(local_sum);
+
+  if (threadIdx.x == 0)
+  {
+    partials(blockIdx.x) = result;
+  }
+}
+
+template <typename Ctx>
+void run()
+{
+  Ctx ctx;
+
+  const size_t N          = 1024 * 16;
+  const size_t BLOCK_SIZE = 128;
+  const size_t num_blocks = 32;
+
+  int *X, ref_tot;
+
+  X       = new int[N];
+  ref_tot = 0;
+
+  for (size_t ind = 0; ind < N; ind++)
+  {
+    X[ind] = rand() % N;
+    ref_tot += X[ind];
+  }
+
+  auto values   = ctx.logical_data(X, {N});
+  auto partials = ctx.logical_data(shape_of<slice<int>>(num_blocks));
+  auto result   = ctx.logical_data(shape_of<slice<int>>(1));
+
+  ctx.task(values.read(), partials.write(), result.write())->*[&](auto stream, auto values, auto partials, auto result) {
+    // reduce values into partials
+    reduce<BLOCK_SIZE, int><<<num_blocks, BLOCK_SIZE, 0, stream>>>(values, partials, N);
+
+    // reduce partials on a single block into result
+    reduce<BLOCK_SIZE, int><<<1, BLOCK_SIZE, 0, stream>>>(partials, result, num_blocks);
+  };
+
+  ctx.host_launch(result.read())->*[&](auto p) {
+    if (p(0) != ref_tot)
+    {
+      fprintf(stderr, "INCORRECT RESULT: p sum = %d, ref tot = %d\n", p(0), ref_tot);
+      abort();
+    }
+  };
+
+  ctx.finalize();
+}
+
+int main()
+{
+  run<stream_ctx>();
+  run<graph_ctx>();
+}

cccl_upstream/cudax/examples/stf/09-dot-reduce.cu

@@ -0,0 +1,55 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDASTF in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+/**
+ * @file
+ *
+ * @brief Implementation of the DOT kernel using a reduce access mode
+ *
+ */
+
+#include <cuda/experimental/stf.cuh>
+
+using namespace cuda::experimental::stf;
+
+int main()
+{
+  const size_t N = 16;
+  double X[N], Y[N];
+
+  double ref_res = 0.0;
+
+  for (size_t i = 0; i < N; i++)
+  {
+    X[i] = cos(double(i));
+    Y[i] = sin(double(i));
+
+    // Compute the reference result of the DOT product of X and Y
+    ref_res += X[i] * Y[i];
+  }
+
+  context ctx;
+  auto lX = ctx.logical_data(X);
+  auto lY = ctx.logical_data(Y);
+
+  auto lsum = ctx.logical_data(shape_of<scalar_view<double>>());
+
+  /* Compute sum(x_i * y_i)*/
+  ctx.parallel_for(lY.shape(), lX.read(), lY.read(), lsum.reduce(reducer::sum<double>{}))
+      ->*[] __device__(size_t i, auto dX, auto dY, double& sum) {
+            sum += dX(i) * dY(i);
+          };
+
+  double res = ctx.wait(lsum);
+
+  ctx.finalize();
+
+  _CCCL_ASSERT(fabs(res - ref_res) < 0.0001, "Invalid result");
+}

cccl_upstream/cudax/examples/stf/1f1b.cu

@@ -0,0 +1,118 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDASTF in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+/**
+ * @file
+ * @brief Toy example to reproduce the asynchrony of a 1F1B pipeline
+ */
+
+#include <cuda/experimental/stf.cuh>
+
+using namespace cuda::experimental::stf;
+
+__global__ void forward(slice<int>, long long int clock_cnt)
+{
+  long long int start_clock  = clock64();
+  long long int clock_offset = 0;
+  while (clock_offset < clock_cnt)
+  {
+    clock_offset = clock64() - start_clock;
+  }
+}
+
+__global__ void backward(slice<int>, long long int clock_cnt)
+{
+  long long int start_clock  = clock64();
+  long long int clock_offset = 0;
+  while (clock_offset < clock_cnt)
+  {
+    clock_offset = clock64() - start_clock;
+  }
+}
+
+int main(int argc, char** argv)
+{
+  context ctx;
+  // Use a graph context if the second argument is set and not null
+  if (argc > 2 && atoi(argv[2]))
+  {
+    ctx = graph_ctx();
+  }
+
+  int device;
+  cudaGetDevice(&device);
+
+  // cudaDevAttrClockRate: Peak clock frequency in kilohertz;
+  int clock_rate;
+  cudaDeviceGetAttribute(&clock_rate, cudaDevAttrClockRate, device);
+
+  auto occ_f = reserved::compute_occupancy(forward);
+  auto occ_b = reserved::compute_occupancy(backward);
+
+  int factor = 1;
+  if (argc > 1)
+  {
+    factor = atoi(argv[1]);
+  }
+
+  size_t num_batches = 8 * factor;
+  int num_devs       = 8;
+  int real_devs;
+  cuda_safe_call(cudaGetDeviceCount(&real_devs));
+
+  std::vector<logical_data<slice<int>>> data;
+
+  for (size_t b = 0; b < num_batches; b++)
+  {
+    auto batch_data = ctx.logical_data(shape_of<slice<int>>(1024));
+    data.push_back(batch_data);
+
+    ctx.task(exec_place::device(0), data[b].write())->*[](cudaStream_t, auto) {
+      // Init ...
+    };
+  }
+
+  cuda_safe_call(cudaStreamSynchronize(ctx.fence()));
+
+  size_t niter = 10;
+
+  for (size_t iter = 0; iter < niter; iter++)
+  {
+    for (size_t b = 0; b < num_batches; b++)
+    {
+      for (int d = 0; d < num_devs; d++)
+      {
+        ctx.task(exec_place::device(d % real_devs), data[b].rw())->*[=](cudaStream_t s, auto bd) {
+          int ms                  = 10;
+          long long int clock_cnt = (long long int) (ms * clock_rate / factor);
+          forward<<<occ_f.min_grid_size, occ_f.block_size, 0, s>>>(bd, clock_cnt);
+        };
+      }
+      //        }
+      //
+      //        for (size_t b = 0; b < num_batches; b++) {
+      for (int d = num_devs; d-- > 0;)
+      {
+        ctx.task(exec_place::device(d % real_devs), data[b].rw())->*[=](cudaStream_t s, auto bd) {
+          int ms                  = 20;
+          long long int clock_cnt = (long long int) (ms * clock_rate / factor);
+          backward<<<occ_b.min_grid_size, occ_b.block_size, 0, s>>>(bd, clock_cnt);
+        };
+      }
+    }
+
+    /* We introduce a fence because the actual pipeline would introduce
+     * some all to all communication to update coefficients */
+    cuda_safe_call(cudaStreamSynchronize(ctx.fence()));
+  }
+
+  ctx.finalize();
+  return 0;
+}

cccl_upstream/cudax/examples/stf/CMakeLists.txt

@@ -0,0 +1,124 @@
+set(
+  stf_example_sources
+  01-axpy.cu
+  01-axpy-cuda_kernel.cu
+  01-axpy-cuda_kernel_chain.cu
+  02-axpy-host_launch.cu
+  03-temporary-data.cu
+  04-fibonacci.cu
+  04-fibonacci-run_once.cu
+  08-cub-reduce.cu
+  axpy-annotated.cu
+  void_data_interface.cu
+  explicit_data_places.cu
+  partitioned_axpy.cu
+  thrust_zip_iterator.cu
+  1f1b.cu
+)
+
+# Examples which rely on code generation (parallel_for or launch)
+set(
+  stf_example_codegen_sources
+  01-axpy-launch.cu
+  01-axpy-parallel_for.cu
+  binary_fhe.cu
+  binary_fhe_stackable.cu
+  09-dot-reduce.cu
+  cfd.cu
+  custom_data_interface.cu
+  fdtd_mgpu.cu
+  fdtd_while.cu
+  fdtd_repeat_n.cu
+  frozen_data_init.cu
+  graph_algorithms/degree_centrality.cu
+  graph_algorithms/jaccard.cu
+  graph_algorithms/pagerank.cu
+  graph_algorithms/pagerank_batched.cu
+  graph_algorithms/pagerank_while.cu
+  graph_algorithms/tricount.cu
+  graph_scope.cu
+  heat.cu
+  heat_mgpu.cu
+  jacobi.cu
+  jacobi_pfor.cu
+  jacobi_stackable.cu
+  jacobi_stackable_raii.cu
+  jacobi_update_cond.cu
+  launch_histogram.cu
+  launch_scan.cu
+  launch_sum.cu
+  launch_sum_cub.cu
+  linear_algebra/burger.cu
+  linear_algebra/burger_sensitivity.cu
+  linear_algebra/cg_csr.cu
+  linear_algebra/cg_csr_stackable.cu
+  logical_gates_composition.cu
+  mandelbrot.cu
+  parallel_for_2D.cu
+  pi.cu
+  scan.cu
+  sqrt_newton_stackable.cu
+  standalone-launches.cu
+  word_count.cu
+  word_count_reduce.cu
+)
+
+# Examples using CUBLAS, CUSOLVER...
+set(
+  stf_example_mathlib_sources
+  linear_algebra/06-pdgemm.cu
+  linear_algebra/06-pdgemm-stackable.cu
+  linear_algebra/07-cholesky.cu
+  linear_algebra/07-potri.cu
+  linear_algebra/cg_dense_2D.cu
+  linear_algebra/strassen.cu
+)
+
+cccl_get_cudatoolkit()
+
+## cudax_add_stf_example
+#
+# Add an stf example executable and register it with ctest.
+#
+# target_name_var: Variable name to overwrite with the name of the example
+#   target. Useful for modifying the example/target after creation.
+# source: The source file for the example.
+#
+# Additional args are passed to cudax_stf_configure_target.
+function(cudax_add_stf_example target_name_var source)
+  get_filename_component(dir ${source} DIRECTORY)
+  get_filename_component(filename ${source} NAME_WE)
+  if (dir)
+    set(filename "${dir}/${filename}")
+  endif()
+  string(REPLACE "/" "." example_name "stf/${filename}")
+
+  set(example_target cudax.example.${example_name})
+
+  cccl_add_executable(${example_target} SOURCES ${source} ADD_CTEST)
+  cudax_stf_configure_target(${example_target} ${ARGN})
+  target_link_libraries(
+    ${example_target}
+    PRIVATE #
+      cudax.compiler_interface
+      cudax.examples.thrust
+  )
+
+  set(${target_name_var} ${example_target} PARENT_SCOPE)
+endfunction()
+
+foreach (source IN LISTS stf_example_sources)
+  cudax_add_stf_example(example_target "${source}")
+endforeach()
+
+if (cudax_ENABLE_CUDASTF_CODE_GENERATION)
+  foreach (source IN LISTS stf_example_codegen_sources)
+    cudax_add_stf_example(example_target "${source}")
+  endforeach()
+endif()
+
+if (cudax_ENABLE_CUDASTF_MATHLIBS)
+  foreach (source IN LISTS stf_example_mathlib_sources)
+    cudax_add_stf_example(example_target "${source}" LINK_MATHLIBS)
+  endforeach()
+endif()

cccl_upstream/cudax/examples/stf/axpy-annotated.cu

@@ -0,0 +1,82 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of CUDASTF in CUDA C++ Core Libraries,
+// under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+// SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES.
+//
+//===----------------------------------------------------------------------===//
+
+/**
+ * @file
+ *
+ * @brief This example illustrates how we can annotate tasks and logical data with debugging symbol
+ *
+ * CUDASTF_DOT_FILE=axpy.dot build/examples/axpy-annotated
+ *
+ * # Generate the visualization from this dot file in PDF or PNG format
+ * dot -Tpdf axpy.dot -o axpy.pdf
+ * dot -Tpng axpy.dot -o axpy.png
+ *
+ * # Generate visualization with events (for advanced users)
+ * CUDASTF_DOT_IGNORE_PREREQS=0 CUDASTF_DOT_FILE=axpy-with-events.dot build/examples/axpy-annotated
+ * dot -Tpng axpy-with-events.dot -o axpy-with-events.png
+ *
+ */
+
+#include <cuda/experimental/stf.cuh>
+
+using namespace cuda::experimental::stf;
+
+__global__ void axpy(double a, slice<const double> x, slice<double> y)
+{
+  int tid      = blockIdx.x * blockDim.x + threadIdx.x;
+  int nthreads = gridDim.x * blockDim.x;
+
+  for (int i = tid; i < x.size(); i += nthreads)
+  {
+    y(i) += a * x(i);
+  }
+}
+
+double X0(int i)
+{
+  return sin((double) i);
+}
+
+double Y0(int i)
+{
+  return cos((double) i);
+}
+
+int main()
+{
+  context ctx;
+  const size_t N = 16;
+  double X[N], Y[N];
+
+  for (size_t i = 0; i < N; i++)
+  {
+    X[i] = X0(i);
+    Y[i] = Y0(i);
+  }
+
+  double alpha = 3.14;
+
+  auto lX = ctx.logical_data(X).set_symbol("X");
+  auto lY = ctx.logical_data(Y).set_symbol("Y");
+
+  /* Compute Y = Y + alpha X */
+  ctx.task(lX.read(), lY.rw()).set_symbol("axpy")->*[&](cudaStream_t s, auto dX, auto dY) {
+    axpy<<<16, 128, 0, s>>>(alpha, dX, dY);
+  };
+
+  ctx.finalize();
+
+  for (size_t i = 0; i < N; i++)
+  {
+    assert(fabs(Y[i] - (Y0(i) + alpha * X0(i))) < 0.0001);
+    assert(fabs(X[i] - X0(i)) < 0.0001);
+  }
+}