feat(cccl): integrate missing CCCL directories — python/, ci/, .agent/, docs/, test/

Sparse-checkout from NVIDIA/cccl main branch to complete cccl_upstream:

Added:
- python/cuda_cccl/ (226 files) — Python bindings for device-level algorithms
  Critical for muh toolchain: cuda.compute.reduce_into, scan, radix_sort, etc.
  Includes 204 .py files with full test coverage for all 27 algorithms
- ci/ (163 files) — Build/test infrastructure
  build_cub.sh, test_cub.sh, build_and_test_targets.sh, matrix.yaml
  Directly maps to our [INFRA-CI] and [INFRA-BUILD] items
- .agent/skills/ (7 files) — NVIDIA's own agent skills for CCCL
  cccl-style/SKILL.md, cccl-test/SKILL.md, sass-diff/SKILL.md
- docs/ (491 files) — Official CCCL documentation
  CI references, CMake guides, Python compute docs, libcudacxx PTX docs
- test/ (12 files) — Top-level integration tests (cuda_smoke, stdpar)
- Root configs: .clang-format, .clang-tidy, CONTRIBUTING.md, pyproject.toml
- CLAUDE.md symlink → AGENTS.md (NVIDIA's standard)

cccl_upstream now mirrors full NVIDIA/cccl structure:
  Before: 42M (cub + thrust + libcudacxx + cudax + c + examples + benchmarks)
  After:  53M (+python +ci +docs +.agent +test +configs)

This completes the CCCL base needed for:
- [muh-bench] items: ci/util/build_and_test_targets.sh for targeted builds
- [CCCL-verify] items: python/cuda_cccl/tests/ as reference implementations
- [CCCL-test] items: ci/test_cub.sh, ci/test_thrust.sh
- Agent workflow: .agent/skills/ for consistent style and test patterns

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic.rst

@@ -0,0 +1,99 @@
+.. _libcudacxx-extended-api-synchronization-atomic:
+
+``cuda::atomic``
+================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   atomic/atomic_thread_fence
+   atomic/fetch_max
+   atomic/fetch_min
+
+Defined in header ``<cuda/atomic>``:
+
+.. code:: cuda
+
+   template <typename T, cuda::thread_scope Scope = cuda::thread_scope_system>
+   class cuda::atomic;
+
+The class template ``cuda::atomic`` is an extended form of `cuda::std::atomic <https://en.cppreference.com/w/cpp/atomic/atomic>`_
+that takes an additional :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` argument,
+defaulted to ``cuda::std::thread_scope_system``.
+
+It has the same interface and semantics as `cuda::std::atomic <https://en.cppreference.com/w/cpp/atomic/atomic>`_,
+with the following additional operations.
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - :ref:`cuda::atomic_thread_fence <libcudacxx-extended-api-synchronization-atomic-atomic-thread-fence>`
+     - Memory order and scope dependent fence synchronization primitive.
+   * - :ref:`cuda::atomic::fetch_min <libcudacxx-extended-api-synchronization-atomic-atomic-fetch-min>`
+     - Atomically find the minimum of the stored value and a provided value.
+   * - :ref:`cuda::atomic::fetch_max <libcudacxx-extended-api-synchronization-atomic-atomic-fetch-max>`
+     - Atomically find the maximum of the stored value and a provided value.
+
+Concurrency Restrictions
+------------------------
+
+An object of type ``cuda::atomic`` or `cuda::std::atomic <https://en.cppreference.com/w/cpp/atomic/atomic>`_
+shall not be accessed concurrently by CPU and GPU threads unless:
+
+  - it is in unified memory and the `concurrentManagedAccess property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_116f9619ccc85e93bc456b8c69c80e78b>`_
+    is 1, or
+  - it is in CPU memory and the `hostNativeAtomicSupported property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_1ef82fd7d1d0413c7d6f33287e5b6306f>`_
+    is 1.
+
+Note, for objects of scopes other than ``cuda::thread_scope_system`` this is a data-race, and therefore also
+prohibited regardless of memory characteristics.
+
+Under CUDA Compute Capability 6 (Pascal), an object of type ``atomic`` may not be used:
+
+  - with automatic storage duration, or
+  - if ``is_always_lock_free()`` is ``false``.
+
+Under CUDA Compute Capability prior to 6 (Pascal), objects of type ``cuda::atomic`` or
+`cuda::std::atomic <https://en.cppreference.com/w/cpp/atomic/atomic>`_ may not be used.
+
+Implementation-Defined Behavior
+-------------------------------
+
+For each type ``T`` and :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``,
+the value of ``cuda::atomic<T, S>::is_always_lock_free()`` is as follows:
+
+.. list-table::
+   :widths: 25 25 50
+   :header-rows: 0
+
+   * - Type ``T``
+     - :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``
+     - ``cuda::atomic<T, S>::is_always_lock_free()``
+   * - Any valid type
+     - Any thread scope
+     - ``sizeof(T) <= 8``
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/atomic>
+
+   __global__ void example_kernel() {
+     // This atomic is suitable for all threads in the system.
+     cuda::atomic<int, cuda::thread_scope_system> a;
+
+     // This atomic has the same type as the previous one (`a`).
+     cuda::atomic<int> b;
+
+     // This atomic is suitable for all threads on the current processor (e.g. GPU).
+     cuda::atomic<int, cuda::thread_scope_device> c;
+
+     // This atomic is suitable for threads in the same thread block.
+     cuda::atomic<int, cuda::thread_scope_block> d;
+   }
+
+`See it on Godbolt <https://godbolt.org/z/avo3Evbee>`_

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic/atomic_thread_fence.rst

@@ -0,0 +1,51 @@
+.. _libcudacxx-extended-api-synchronization-atomic-atomic-thread-fence:
+
+cuda::atomic::atomic_thread_fence
+=====================================
+
+Defined in header ``<cuda/atomic>``:
+
+.. code:: cuda
+
+   __host__ __device__
+   void cuda::atomic_thread_fence(cuda::std::memory_order order,
+                                  cuda::thread_scope scope = cuda::thread_scope_system);
+
+Establishes memory synchronization ordering of non-atomic and relaxed atomic accesses, as instructed by ``order``,
+for all threads within ``scope`` without an associated atomic operation. It has the same semantics as
+`cuda::std::atomic_thread_fence <https://en.cppreference.com/w/cpp/atomic/atomic_thread_fence>`_.
+
+Example
+-------
+
+The following code is an example of the :ref:`MessagePassing <libcudacxx-extended-api-memory-model-message-passing>` pattern:
+
+.. code:: cuda
+
+   #include <cstdio>
+   #include <cuda/atomic>
+   #include <cooperative_groups.h>
+
+   namespace cg = cooperative_groups;
+
+   __global__ void example_kernel(int* data, cuda::std::atomic_flag* flag) {
+     assert(cg::grid_group::size() == 2);
+     assert(cg::thread_block::size() == 1);
+
+     if (blockIdx.x == 0) {
+       *data = 42;
+       cuda::atomic_thread_fence(cuda::memory_order_release,
+                                 cuda::thread_scope_device);
+       flag->test_and_set(cuda::std::memory_order_relaxed);
+       flag->notify_one();
+     }
+     else {
+       // an atomic operation is required to set up the synchronization
+       flag->wait(false, cuda::std::memory_order_relaxed);
+       cuda::atomic_thread_fence(cuda::memory_order_acquire,
+                                 cuda::thread_scope_device);
+       std::printf("%d\n", *data); // Prints 42
+     }
+   }
+
+`See it on Godbolt <https://godbolt.org/z/aG37o5qxx>`_

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic/fetch_max.rst

@@ -0,0 +1,34 @@
+.. _libcudacxx-extended-api-synchronization-atomic-atomic-fetch-max:
+
+cuda::atomic::fetch_max
+===========================
+
+Defined in header ``<cuda/atomic>``:
+
+.. code:: cuda
+
+   template <typename T, cuda::thread_scope Scope>
+   __host__ __device__
+   T cuda::atomic<T, Scope>::fetch_max(T const& val,
+                                       cuda::std::memory_order order
+                                         = cuda::std::memory_order_seq_cst);
+
+Atomically find the maximum of the value stored in the ``cuda::atomic``
+and ``val``. The maximum is found using
+`cuda::std::max <https://en.cppreference.com/w/cpp/algorithm/max>`_.
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/atomic>
+
+   __global__ void example_kernel() {
+     cuda::atomic<int> a(0);
+     auto x = a.fetch_max(1);
+     auto y = a.load();
+     assert(x == 0 && y == 1);
+   }
+
+`See it on Godbolt <https://godbolt.org/z/rexn5T78G>`_

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic/fetch_min.rst

@@ -0,0 +1,34 @@
+.. _libcudacxx-extended-api-synchronization-atomic-atomic-fetch-min:
+
+cuda::atomic::fetch_min
+===========================
+
+Defined in header ``<cuda/atomic>``:
+
+.. code:: cuda
+
+   template <typename T, cuda::thread_scope Scope>
+   __host__ __device__
+   T cuda::atomic<T, Scope>::fetch_min(T const& val,
+                                       cuda::std::memory_order order
+                                         = cuda::std::memory_order_seq_cst);
+
+Atomically find the minimum of the value stored in the ``cuda::atomic``
+and ``val``. The minimum is found using
+`cuda::std::min <https://en.cppreference.com/w/cpp/algorithm/min>`_.
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/atomic>
+
+   __global__ void example_kernel() {
+     cuda::atomic<int> a(1);
+     auto x = a.fetch_min(0);
+     auto y = a.load();
+     assert(x == 1 && y == 0);
+   }
+
+`See it on Godbolt <https://godbolt.org/z/vMj9e5hdv>`_

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/atomic_ref.rst

@@ -0,0 +1,108 @@
+.. _libcudacxx-extended-api-synchronization-atomic-ref:
+
+``cuda::atomic_ref``
+====================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+
+Defined in header ``<cuda/atomic>``:
+
+.. code:: cuda
+
+   template <typename T, cuda::thread_scope Scope = cuda::thread_scope_system>
+   class cuda::atomic_ref;
+
+The class template ``cuda::atomic_ref`` is an extended form of `cuda::std::atomic_ref <https://en.cppreference.com/w/cpp/atomic/atomic_ref>`_
+that takes an additional :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` argument, defaulted to
+``cuda::std::thread_scope_system``.
+
+It has the same interface and semantics as `cuda::std::atomic_ref <https://en.cppreference.com/w/cpp/atomic/atomic_ref>`_,
+with the following additional operations. This class additionally deviates from the standard by being backported to C++11.
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - :ref:`cuda::atomic_ref::fetch_min <libcudacxx-extended-api-synchronization-atomic-atomic-fetch-min>`
+     - Atomically find the minimum of the stored value and a provided value.
+   * - :ref:`cuda::atomic_ref::fetch_max <libcudacxx-extended-api-synchronization-atomic-atomic-fetch-max>`
+     - Atomically find the maximum of the stored value and a provided value.
+
+
+Limitations
+-----------
+
+``cuda::atomic_ref<T>`` and ``cuda::std::atomic_ref<T>`` may only be instantiated when ``T`` satisfies ``sizeof(T) <= 8`` or ``sizeof(T) <= 16`` when requirements are met.
+
+The operations available to ``T`` when ``sizeof(T) == 16`` depend on the architecture:
+  - On SM70 and later: ``load`` and ``store`` are supported.
+  - On SM90 and later: ``fetch_*`` and synchronization operations are supported, implemented via atomic compare-and-swap (CAS).
+
+No object or subobject of an object referenced by an ``atomic_­ref`` shall be concurrently referenced by any other
+``atomic_­ref`` that has a different ``Scope``.
+
+For ``cuda::atomic_ref<T>`` and ``cuda::std::atomic_ref<T>`` the type ``T`` must satisfy the following:
+  - ``sizeof(T) <= 16``.
+  - The referenced object must be aligned to its size: ``alignof(T) == sizeof(T)``.
+  - ``T`` must not have "padding bits", i.e., T's `object representation <https://en.cppreference.com/w/cpp/language/object#Object_representation_and_value_representation>`_
+    must not have bits that do not participate in it's value representation.
+
+Concurrency Restrictions
+------------------------
+
+See :ref:`memory model <libcudacxx-extended-api-memory-model>` documentation for general restrictions on atomicity.
+
+With CUDA Compute Capability 6 (Pascal), an object of type ``atomic_ref`` may not be used:
+  - with a reference to an object with a automatic storage duration in a GPU thread, or
+  - if ``is_always_lock_free()`` is ``false``.
+
+For CUDA Compute Capability prior to 6 (Pascal), objects of type ``cuda::atomic_ref`` or
+`cuda::std::atomic_ref <https://en.cppreference.com/w/cpp/atomic/atomic_ref>`_ may not be used.
+
+Implementation-Defined Behavior
+-------------------------------
+
+For each type ``T`` and :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``, the value of
+``cuda::atomic_ref<T, S>::is_always_lock_free()`` and ``cuda::std::atomic_ref<T>::is_always_lock_free()`` is as follows:
+
+.. list-table::
+   :widths: 25 25 50
+   :header-rows: 0
+
+   * - Type ``T``
+     - :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``
+     - ``cuda::atomic_ref<T, S>::is_always_lock_free()``
+   * - Any valid type
+     - Any thread scope
+     - ``sizeof(T) <= 8``
+
+Types of ``T``, where ``sizeof(T) < 4``, are not natively supported by the underlying hardware. For these types atomic
+operations are emulated and will be drastically slower. Contention with contiguous memory in the current 4 byte boundary
+will be exacerbated. In these situations it is advisable to perform a hierarchical reduction to non-adjacent memory first.
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/atomic>
+
+   __global__ void example_kernel(int *gmem, int *pinned_mem) {
+     // This atomic is suitable for all threads in the system.
+     cuda::atomic_ref<int, cuda::thread_scope_system> a(*pinned_mem);
+
+     // This atomic has the same type as the previous one (`a`).
+     cuda::atomic_ref<int> b(*pinned_mem);
+
+     // This atomic is suitable for all threads on the current processor (e.g. GPU).
+     cuda::atomic_ref<int, cuda::thread_scope_device> c(*gmem);
+
+     __shared__ int shared_v;
+     // This atomic is suitable for threads in the same thread block.
+     cuda::atomic_ref<int, cuda::thread_scope_block> d(shared_v);
+   }
+
+`See it on Godbolt <https://godbolt.org/z/fr4K7ErEh>`_

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier.rst

@@ -0,0 +1,297 @@
+.. _libcudacxx-extended-api-synchronization-barrier:
+
+``cuda::barrier``
+=================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   barrier/init
+   barrier/barrier_native_handle
+   barrier/barrier_arrive_tx
+   barrier/barrier_expect_tx
+
+Defined in header ``<cuda/barrier>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope,
+             typename CompletionFunction = /* unspecified */>
+   class cuda::barrier;
+
+The class template ``cuda::barrier`` is an extended form of `cuda::std::barrier <https://en.cppreference.com/w/cpp/thread/barrier>`_
+that takes an additional :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` argument.
+
+If ``!(scope == cuda::thread_block_scope && cuda::device::is_address_from(this, cuda::device::address_space::shared))``, then the semantics are the same as
+`cuda::std::barrier <https://en.cppreference.com/w/cpp/thread/barrier>`_, otherwise, see below.
+
+The ``cuda::barrier`` class templates extends ``cuda::std::barrier`` with the following additional operations:
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - :ref:`cuda::barrier::init <libcudacxx-extended-api-synchronization-barrier-barrier-init>`
+     - Initialize a ``cuda::barrier``.
+   * - :ref:`cuda::device::barrier_native_handle <libcudacxx-extended-api-synchronization-barrier-barrier-native-handle>`
+     - Get the native handle to a ``cuda::barrier``.
+   * - :ref:`cuda::device::barrier_arrive_tx <libcudacxx-extended-api-synchronization-barrier-barrier-arrive-tx>`
+     - Arrive on a ``cuda::barrier<cuda::thread_scope_block>`` with transaction count update.
+   * - :ref:`cuda::device::barrier_expect_tx <libcudacxx-extended-api-synchronization-barrier-barrier-expect-tx>`
+     - Update transaction count of ``cuda::barrier<cuda::thread_scope_block>``.
+
+If ``scope == cuda::thread_scope_block && cuda::device::is_address_from(this, cuda::device::address_space::shared)``, then the semantics of `[thread.barrier.class] <http://eel.is/c++draft/thread.barrier.class>`_
+of ISO/IEC IS 14882 (the C++ Standard) are modified as follows:
+
+   A barrier is a thread coordination mechanism whose lifetime consists
+   of a sequence of barrier phases, where each phase allows at most an
+   expected number of threads to block until the expected number of
+   threads **and the expected number of transaction-based asynchronous
+   operations** arrive at the barrier. Each *barrier phase* consists of
+   the following steps:
+
+   1. The *expected count* is decremented by each call to ``arrive``,\ ``arrive_and_drop``\ **,
+      or cuda::device::barrier_arrive_tx**.
+   2. **The transaction count is incremented by each call to cuda::device::barrier_arrive_tx and decremented by the
+      completion of transaction-based asynchronous operations such as cuda::memcpy_async_tx.**
+   3. Exactly once after **both** the *expected count* **and the transaction count** reach zero, a thread executes the
+      *completion step* during its call to ``arrive``, ``arrive_and_drop``, ``cuda::device::barrier_arrive_tx``,
+      or ``wait``, except that it is implementation-defined whether the step executes if no thread calls ``wait``.
+   4. When the completion step finishes, the *expected count* is reset to what was specified by the ``expected``
+      argument to the constructor, possibly adjusted by calls to ``arrive_and_drop``, and the next phase starts.
+
+   Concurrent invocations of the member functions of barrier **and the non-member barrier APIs in cuda::device**,
+   other than its destructor, do not introduce data races. The member functions ``arrive`` and ``arrive_and_drop``,
+   **and the non-member function cuda::device::barrier_arrive_tx**, execute atomically.
+
+.. rubric:: NVCC ``__shared__`` Initialization Warnings
+
+When using libcu++ with NVCC, a ``__shared__`` ``cuda::barrier`` will lead to the following warning because
+``__shared__`` variables are not initialized:
+
+.. code:: bash
+
+   warning: dynamic initialization is not supported for a function-scope static
+   __shared__ variable within a __device__/__global__ function
+
+It can be silenced using ``#pragma nv_diag_suppress static_var_with_dynamic_init``.
+
+To properly initialize a ``__shared__`` ``cuda::barrier``, use the
+:ref:`cuda::barrier::init <libcudacxx-extended-api-synchronization-barrier-barrier-init>` friend function.
+
+.. rubric:: Concurrency Restrictions
+
+An object of type ``cuda::barrier`` or ``cuda::std::barrier`` shall not be accessed concurrently by CPU and GPU threads unless:
+
+   - it is in unified memory and the `concurrentManagedAccess property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_116f9619ccc85e93bc456b8c69c80e78b>`_
+     is 1, or
+   - it is in CPU memory and the `hostNativeAtomicSupported property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_1ef82fd7d1d0413c7d6f33287e5b6306f>`_
+     is 1.
+
+Note, for objects of scopes other than ``cuda::thread_scope_system`` this is a data-race, and therefore also prohibited
+regardless of memory characteristics.
+
+Under CUDA Compute Capability 8 (Ampere) or above, when an object of type ``cuda::barrier<thread_scope_block>`` is
+placed in ``__shared__`` memory, the member function ``arrive`` performs a reduction of the arrival count among
+`coalesced threads <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#coalesced-group-cg>`_ followed
+by the arrival operation in one thread. Programs shall ensure that this transformation would not introduce errors,
+for example relative to the requirements of `thread.barrier.class paragraph 12 <https://eel.is/c++draft/thread.barrier.class#12>`_
+of ISO/IEC IS 14882 (the C++ Standard).
+
+Under CUDA Compute Capability 6 (Pascal) or prior, an object of type ``cuda::barrier`` or ``cuda::std::barrier`` may
+not be used.
+
+.. rubric:: Shared memory barriers with transaction count
+
+In addition to the arrival count, a ``cuda::barrier<thread_scope_block>`` object located in shared memory supports a
+`tx-count <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-mbarrier-tracking-async-operations>`_,
+which is used for tracking the completion of some asynchronous memory operations or transactions.
+The tx-count tracks the number of asynchronous transactions, in units specified by the asynchronous memory operation
+(typically bytes), that are outstanding and yet to be complete. This capability is exposed, starting with the Hopper
+architecture (CUDA Compute Capability 9).
+
+The tx-count of ``cuda::barrier`` must be set to the total amount of asynchronous memory operations, in units as
+specified by the asynchronous operations, to be tracked by the current phase. This can be achieved with the
+:ref:`cuda::device::barrier_arrive_tx <libcudacxx-extended-api-synchronization-barrier-barrier-arrive-tx>` function call.
+
+Upon completion of each of the asynchronous operations, the tx-count of the ``cuda::barrier`` will be updated and thus
+progress the ``cuda::barrier`` towards the completion of the current phase. This may complete the current phase.
+
+.. rubric:: Implementation-Defined Behavior
+
+For each :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S`` and ``CompletionFunction``
+``F``, the value of ``cuda::barrier<S, F>::max()`` is as follows:
+
+.. list-table::
+   :widths: 25 25 50
+   :header-rows: 0
+
+   * - :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``
+     - ``CompletionFunction`` ``F``
+     - ``barrier<S, F>::max()``
+   * - ``cuda::thread_scope_block``
+     - Default or user-provided
+     - ``(1 << 20) - 1``
+   * - *Not* ``cuda::thread_scope_block``
+     - Default
+     - ``cuda::std::numeric_limits<cuda::std::int32_t>::max()``
+   * - *Not* ``cuda::thread_scope_block``
+     - User-provided
+     - ``cuda::std::numeric_limits<cuda::std::ptrdiff_t>::max()``
+
+.. rubric:: Example
+
+.. code:: cuda
+
+   #include <cuda/barrier>
+
+   __global__ void example_kernel() {
+     // This barrier is suitable for all threads in the system.
+     cuda::barrier<cuda::thread_scope_system> a(10);
+
+     // This barrier has the same type as the previous one (`a`).
+     cuda::std::barrier<> b(10);
+
+     // This barrier is suitable for all threads on the current processor (e.g. GPU).
+     cuda::barrier<cuda::thread_scope_device> c(10);
+
+     // This barrier is suitable for all threads in the same thread block.
+     cuda::barrier<cuda::thread_scope_block> d(10);
+   }
+
+`See it on Godbolt <https://godbolt.org/z/7Kbz5qqhh>`__
+
+.. rubric:: Example: 1D TMA load of two buffers with arrival token (sm90+)
+
+The following example shows how to use TMA to load two tiles of data from global memory into shared memory:
+
+.. code:: cuda
+
+   #include <cuda/barrier>
+   #include <cuda/ptx>
+
+   // selects a single leader thread from the block
+   __device__ bool elect_one() {
+     const unsigned int tid = threadIdx.x;
+     const unsigned int warp_id = tid / 32;
+     const unsigned int uniform_warp_id = __shfl_sync(0xFFFFFFFF, warp_id, 0); // broadcast from lane 0
+     return uniform_warp_id == 0 && cuda::ptx::elect_sync(0xFFFFFFFF); // elect a leader thread among warp 0
+   }
+
+   __global__ void example_kernel(int* gmem1, double* gmem2) {
+     constexpr int tile_size = 1024;
+     __shared__ alignas(16)    int smem1[tile_size];
+     __shared__ alignas(16) double smem2[tile_size];
+
+     #pragma nv_diag_suppress static_var_with_dynamic_init
+     __shared__  cuda::barrier<cuda::thread_scope_block> bar;
+
+     // setup the barrier where each thread in the block arrives at
+     if (threadIdx.x == 0) {
+       init(&bar, blockDim.x);
+     }
+     __syncthreads(); // need to sync so other threads can arrive
+
+     // select a single thread from the block and issue two TMA bulk copy operations
+     const auto elected = elect_one();
+     if (elected) {
+       cuda::device::memcpy_async_tx(smem1, gmem1, cuda::aligned_size_t<16>(tile_size * sizeof(int)   ), bar);
+       cuda::device::memcpy_async_tx(smem2, gmem2, cuda::aligned_size_t<16>(tile_size * sizeof(double)), bar);
+     }
+
+     // arrive at the barrier
+     // the elected thread also updates the barrier's expect_tx with the **total** number of loaded bytes
+     const int tx_count = elected ? tile_size * (sizeof(int) + sizeof(double)) : 0;
+     auto token = cuda::device::barrier_arrive_tx(bar, 1, tx_count);
+
+     // wait for TMA copies to complete
+     bar.wait(cuda::std::move(token));
+
+     // process data in smem ...
+   }
+
+`See it on Godbolt <https://godbolt.org/z/ddhzGeWPE>`__
+
+Data is loaded from the global memory pointers ``gmem1`` and ``gmem2``
+into the shared memory buffers ``smem1`` and ``smem2``.
+The shared memory buffers have to be aligned to 16 bytes.
+Additionally, a single barrier with block scope is setup by a single leader thread.
+Each thread will arrive at the barrier, so ``blockDim.x`` is passed as arrival count to ``init``.
+This barrier is used to synchronize the asynchronous TMA copies with the rest of the threads in the block.
+The leader initiates the TMA copies using ``ptx::cp_async_bulk``,
+and updates the barrier's tx-count with the total number of bytes transferred by the TMA copy operations
+while arriving at the barrier using ``cuda::device::barrier_arrive_tx``.
+All other threads just arrive normally at the barrier.
+All threads then wait on the barrier to complete the current phase by calling ``wait``.
+Once ``wait`` returns, all data is available in shared memory.
+
+.. rubric:: Example: 1D TMA load of two buffers with barrier phase parity check (sm90+)
+
+.. code:: cuda
+
+   #include <cuda/barrier>
+   #include <cuda/ptx>
+
+   // selects a single leader thread from the block
+   __device__ bool elect_one() {
+     const unsigned int tid = threadIdx.x;
+     const unsigned int warp_id = tid / 32;
+     const unsigned int uniform_warp_id = __shfl_sync(0xFFFFFFFF, warp_id, 0); // broadcast from lane 0
+     return uniform_warp_id == 0 && cuda::ptx::elect_sync(0xFFFFFFFF); // elect a leader thread among warp 0
+   }
+
+   __global__ void example_kernel(int* gmem1, double* gmem2) {
+     constexpr int tile_size = 1024;
+     __shared__ alignas(16)    int smem1[tile_size];
+     __shared__ alignas(16) double smem2[tile_size];
+
+     #pragma nv_diag_suppress static_var_with_dynamic_init
+     __shared__  cuda::barrier<cuda::thread_scope_block> bar;
+
+     // setup the barrier where only the leader thread arrives
+     if (elect_one()) {
+       init(&bar, 1);
+
+       // issue two TMA bulk copy operations
+       cuda::device::memcpy_async_tx(smem1, gmem1, cuda::aligned_size_t<16>(tile_size * sizeof(int)   ), bar);
+       cuda::device::memcpy_async_tx(smem2, gmem2, cuda::aligned_size_t<16>(tile_size * sizeof(double)), bar);
+
+       // arrive and update the barrier's expect_tx with the **total** number of loaded bytes
+       (void)cuda::device::barrier_arrive_tx(bar, 1, tile_size * (sizeof(int) + sizeof(double)));
+     }
+     __syncthreads(); // need to sync so the barrier is set up when the other threads arrive and wait
+
+     // wait for the current barrier phase to complete
+     bar.wait_parity(0);
+
+     // process data in smem ...
+   }
+
+`See it on Godbolt <https://godbolt.org/z/oq85PoKKj>`__
+
+This is similar to the above example, but this time only the leader thread arrives at the barrier.
+This has the advantage that only one leader election is necessary
+and a single uniform datapath section is generated.
+This generally generates less instructions.
+Because now we don't get an arrival token in each thread, we cannot use ``wait(token)`` with all threads.
+Instead, we just wait until the end of the barrier's current phase using ``wait_parity(0)``
+(0 is the parity of the current phase).
+
+Before CCCL 3.2, ``bar.wait_parity(0);`` contained additional logic which may have lead to unnecessary instructions.
+If you are using CCCL below 3.2, you may replace this line with:
+
+.. code:: cuda
+
+   while (!cuda::ptx::mbarrier_try_wait_parity(cuda::device::barrier_native_handle(bar), 0))
+     ;
+
+.. rubric:: Example: 1D TMA load and store using `cuda::device::memcpy_async_tx` (sm90+)
+
+This example can be found in :ref:`libcudacxx-extended-api-asynchronous-operations-memcpy-async-tx-example`.
+
+
+.. rubric:: Example: 1D TMA load and store using `cuda::memcpy_async` (sm90+)
+
+This example can be found in the
+`CUDA programming guide <https://docs.nvidia.com/cuda/cuda-c-programming-guide/#using-tma-to-transfer-one-dimensional-arrays>`__.

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier/barrier_arrive_tx.rst

@@ -0,0 +1,77 @@
+.. _libcudacxx-extended-api-synchronization-barrier-barrier-arrive-tx:
+
+cuda::device::barrier_arrive_tx
+===================================
+
+Defined in header ``<cuda/barrier>``:
+
+.. code:: cuda
+
+   __device__
+   cuda::barrier<cuda::thread_scope_block>::arrival_token
+   cuda::device::barrier_arrive_tx(
+     cuda::barrier<cuda::thread_scope_block>& bar,
+     ptrdiff_t arrive_count_update,
+     ptrdiff_t transaction_count_update);
+
+Arrives at a barrier in shared memory, decrementing the arrival count and incrementing the expected transaction count.
+
+Preconditions
+-------------
+
+-  ``cuda::device::is_object_from(bar, cuda::device::address_space::shared) == true``
+-  ``1 <= arrive_count_update && transaction_count_update <= (1 << 20) - 1``
+-  ``0 <= transaction_count_update && transaction_count_update <= (1 << 20) - 1``
+
+Effects
+-------
+
+-  This function constructs an arrival_token object associated with the
+   phase synchronization point for the current phase. Then, decrements
+   the arrival count by ``arrive_count_update`` and increments the
+   expected transaction count by ``transaction_count_update``.
+-  This function executes atomically. The call to this function strongly
+   happens-before the start of the phase completion step for the current
+   phase.
+
+Notes
+-----
+
+This function can only be used under CUDA Compute Capability 9.0 (Hopper) or higher.
+
+To check if ``cuda::device::barrier_arrive_tx`` is available, use the ``__cccl_lib_local_barrier_arrive_tx``
+feature flag, as shown in the example code below.
+
+Return Value
+------------
+
+``cuda::device::barrier_arrive_tx`` returns the constructed ``arrival_token`` object.
+
+Example
+-------
+
+Below example shows only ``cuda::device::barrier_arrive_tx``. A more extensive example can be found in the
+:ref:`cuda::device::memcpy_async_tx <libcudacxx-extended-api-asynchronous-operations-memcpy-async-tx>` documentation.
+
+.. code:: cuda
+
+   #include <cuda/barrier>
+   #include <cuda/std/utility> // cuda::std::move
+
+   #ifndef  __cccl_lib_local_barrier_arrive_tx
+   static_assert(false, "Insufficient libcu++ version: cuda::device::arrive_tx is not yet available.");
+   #endif // __cccl_lib_local_barrier_arrive_tx
+
+   __global__ void example_kernel() {
+     __shared__ cuda::barrier<cuda::thread_scope_block> bar;
+     if (threadIdx.x == 0) {
+       init(&bar, blockDim.x);
+     }
+     __syncthreads();
+
+     auto token = cuda::device::barrier_arrive_tx(bar, 1, 0);
+
+     bar.wait(cuda::std::move(token));
+   }
+
+`See it on Godbolt <https://godbolt.org/z/1vxcGrT8j>`_

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier/barrier_expect_tx.rst

@@ -0,0 +1,70 @@
+.. _libcudacxx-extended-api-synchronization-barrier-barrier-expect-tx:
+
+cuda::device::barrier_expect_tx
+===================================
+
+Defined in header ``<cuda/barrier>``:
+
+.. code:: cuda
+
+   __device__
+   void cuda::device::barrier_expect_tx(
+     cuda::barrier<cuda::thread_scope_block>& bar,
+     ptrdiff_t transaction_count_update);
+
+Increments the expected transaction count of a barrier in shared memory.
+
+Preconditions
+-------------
+
+-  ``cuda::device::is_object_from(bar, cuda::device::address_space::shared) == true``
+-  ``0 <= transaction_count_update && transaction_count_update <= (1 << 20) - 1``
+
+Effects
+-------
+
+-  This function increments the expected transaction count by transaction_count_update``.
+-  This function executes atomically.
+
+Notes
+-----
+
+This function can only be used under CUDA Compute Capability 9.0 (Hopper) or higher.
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/barrier>
+   #include <cuda/std/utility> // cuda::std::move
+
+   #if defined(__CUDA_MINIMUM_ARCH__) && __CUDA_MINIMUM_ARCH__ < 900
+   static_assert(false, "Insufficient CUDA Compute Capability: cuda::device::memcpy_expect_tx is not available.");
+   #endif // __CUDA_MINIMUM_ARCH__
+
+   __device__ alignas(16) int gmem_x[2048];
+
+   __global__ void example_kernel() {
+       using barrier_t = cuda::barrier<cuda::thread_scope_block>;
+     alignas(16) __shared__ int smem_x[1024];
+     __shared__ barrier_t bar;
+
+     if (threadIdx.x == 0) {
+       init(&bar, blockDim.x);
+     }
+     __syncthreads();
+
+     if (threadIdx.x == 0) {
+       cuda::device::memcpy_async_tx(smem_x, gmem_x, cuda::aligned_size_t<16>(sizeof(smem_x)), bar);
+       cuda::device::barrier_expect_tx(bar, sizeof(smem_x));
+     }
+     auto token = bar.arrive(1);
+
+     bar.wait(cuda::std::move(token));
+
+     // smem_x contains the contents of gmem_x[0], ..., gmem_x[1023]
+     smem_x[threadIdx.x] += 1;
+   }
+
+`See it on Godbolt <https://godbolt.org/z/9Yj89P76z>`_

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier/barrier_native_handle.rst

@@ -0,0 +1,46 @@
+.. _libcudacxx-extended-api-synchronization-barrier-barrier-native-handle:
+
+cuda::device::barrier_native_handle
+=======================================
+
+Defined in header ``<cuda/barrier>``:
+
+.. code:: cuda
+
+   __device__ cuda::std::uint64_t* cuda::device::barrier_native_handle(
+     cuda::barrier<cuda::thread_scope_block>& bar);
+
+Returns a pointer to the native handle of a :ref:`cuda::barrier <libcudacxx-extended-api-synchronization-barrier>`
+if its scope is ``cuda::thread_scope_block`` and it is allocated in shared memory.
+The pointer is suitable for use with PTX instructions.
+
+Notes
+-----
+
+If ``bar`` is not in ``__shared__`` memory, the behavior is undefined.
+
+Return Value
+------------
+
+A pointer to the PTX "mbarrier" subobject of the :ref:`cuda::barrier <libcudacxx-extended-api-synchronization-barrier>`
+object.
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/barrier>
+
+   __global__ void example_kernel(cuda::barrier<cuda::thread_scope_block>& bar) {
+     auto ptr = cuda::device::barrier_native_handle(bar);
+
+     asm volatile (
+         "mbarrier.arrive.b64 _, [%0];"
+         :
+         : "l" (ptr)
+         : "memory");
+     // Equivalent to: `(void)b.arrive()`.
+   }
+
+`See it on Godbolt <https://godbolt.org/z/dr4798Y76>`_

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/barrier/init.rst

@@ -0,0 +1,53 @@
+.. _libcudacxx-extended-api-synchronization-barrier-barrier-init:
+
+cuda::barrier::init
+=======================
+
+Defined in header ``<cuda/barrier>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope,
+             typename CompletionFunction = /* unspecified */>
+   class barrier {
+   public:
+     // ...
+
+     __host__ __device__
+     friend void init(cuda::std::barrier* bar,
+                      cuda::std::ptrdiff_t expected,
+                      CompletionFunction cf = CompletionFunction{});
+   };
+
+The friend function ``cuda::barrier::init`` may be used to initialize an
+:ref:`cuda::barrier <libcudacxx-extended-api-synchronization-barrier>` that has not been initialized.
+
+When using libcu++ with NVCC, ``__shared__`` ``cuda::barrier`` will not have its constructors run because ``__shared__``
+variables are not initialized. ``cuda::barrier::init`` should be used to properly initialize such a
+:ref:`cuda::barrier <libcudacxx-extended-api-synchronization-barrier>`.
+
+An NVCC diagnostic warning about the ignored constructor will be emitted:
+
+.. code:: bash
+
+   warning: dynamic initialization is not supported for a function-scope static
+   __shared__ variable within a __device__/__global__ function
+
+It can be silenced using ``#pragma nv_diag_suppress static_var_with_dynamic_init``.
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/barrier>
+
+   // Disables `cuda::barrier` initialization warning.
+   #pragma nv_diag_suppress static_var_with_dynamic_init
+
+   __global__ void example_kernel() {
+     __shared__ cuda::barrier<cuda::thread_scope_block> bar;
+     init(&bar, 1);
+   }
+
+`See it on Godbolt <https://godbolt.org/z/nK5q3xh34>`_

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/binary_semaphore.rst

@@ -0,0 +1,76 @@
+.. _libcudacxx-extended-api-synchronization-binary-semaphore:
+
+``cuda::binary_semaphore``
+==========================
+
+Defined in header ``<cuda/semaphore>``:
+
+.. code:: cpp
+
+   namespace cuda {
+
+   template <cuda::thread_scope Scope>
+   using binary_semaphore = cuda::std::counting_semaphore<Scope, 1>;
+
+   }
+
+The class template ``cuda::binary_semaphore`` is an extended form of `cuda::std::binary_semaphore <https://en.cppreference.com/w/cpp/thread/counting_semaphore>`_
+that takes an additional :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` argument.
+``cuda::binary_semaphore`` has the same interface and semantics as
+`cuda::std::binary_semaphore <https://en.cppreference.com/w/cpp/thread/counting_semaphore>`_, but
+``cuda::binary_semaphore`` is a class template.
+
+Concurrency Restrictions
+------------------------
+
+An object of type ``cuda::binary_semaphore`` or ``cuda::std::binary_semaphore``, shall not be accessed concurrently by
+CPU and GPU threads unless:
+
+   - it is in unified memory and the `concurrentManagedAccess property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_116f9619ccc85e93bc456b8c69c80e78b>`_
+     is 1, or
+   - it is in CPU memory and the `hostNativeAtomicSupported property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_1ef82fd7d1d0413c7d6f33287e5b6306f>`_
+     is 1.
+
+Note, for objects of scopes other than ``cuda::thread_scope_system`` this is a data-race, and therefore also prohibited
+regardless of memory characteristics.
+
+Under CUDA Compute Capability 6 (Pascal) or prior, an object of type ``cuda::binary_semaphore`` or
+``cuda::std::binary_semaphore`` may not be used.
+
+Implementation-Defined Behavior
+-------------------------------
+
+For each :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``,
+``cuda::binary_semaphore<S>::max()`` is as follows:
+
+.. list-table::
+   :widths: 50 50
+   :header-rows: 0
+
+   * - :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``
+     - ``cuda::binary_semaphore<S>::max()``
+   * - Any thread scope
+     - ``1``
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/semaphore>
+
+   __global__ void example_kernel() {
+     // This semaphore is suitable for all threads in the system.
+     cuda::binary_semaphore<cuda::thread_scope_system> a;
+
+     // This semaphore has the same type as the previous one (`a`).
+     cuda::std::binary_semaphore<> b;
+
+     // This semaphore is suitable for all threads on the current processor (e.g. GPU).
+     cuda::binary_semaphore<cuda::thread_scope_device> c;
+
+     // This semaphore is suitable for all threads in the same thread block.
+     cuda::binary_semaphore<cuda::thread_scope_block> d;
+   }
+
+`See it on Godbolt <https://godbolt.org/z/eKfjYYz58>`_

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/counting_semaphore.rst

@@ -0,0 +1,72 @@
+.. _libcudacxx-extended-api-synchronization-counting-semaphore:
+
+``cuda::counting_semaphore``
+============================
+
+Defined in header ``<cuda/semaphore>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope,
+             cuda::std::ptrdiff_t LeastMaxValue = /* implementation-defined */>
+   class cuda::counting_semaphore;
+
+The class template ``cuda::counting_semaphore`` is an extended form of `cuda::std::counting_semaphore <https://en.cppreference.com/w/cpp/thread/counting_semaphore>`_
+that takes an additional :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` argument.
+``cuda::counting_semaphore`` has the same interface and semantics as
+`cuda::std::counting_semaphore <https://en.cppreference.com/w/cpp/thread/counting_semaphore>`_.
+
+Concurrency Restrictions
+------------------------
+
+An object of type ``cuda::counting_semaphore`` or ``cuda::std::counting_semaphore``, shall not be accessed concurrently
+by CPU and GPU threads unless:
+
+   - it is in unified memory and the `concurrentManagedAccess property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_116f9619ccc85e93bc456b8c69c80e78b>`_
+     is 1, or
+   - it is in CPU memory and the `hostNativeAtomicSupported property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_1ef82fd7d1d0413c7d6f33287e5b6306f>`_
+     is 1.
+
+Note, for objects of scopes other than ``cuda::thread_scope_system`` this is a data-race, and therefore also prohibited
+regardless of memory characteristics.
+
+Under CUDA Compute Capability 6 (Pascal) or prior, an object of type ``cuda::counting_semaphore`` or
+``cuda::std::counting_semaphore`` may not be used.
+
+Implementation-Defined Behavior
+-------------------------------
+
+For each :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S`` and least maximum value
+``V``, ``cuda::counting_semaphore<S, V>::max()`` is as follows:
+
+.. list-table::
+   :widths: 50 50
+   :header-rows: 0
+
+   * - :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``
+     - ``cuda::binary_semaphore<S>::max()``
+   * - Any thread scope
+     - ``cuda::std::numeric_limits<cuda::std::ptrdiff_t>::max()``
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/semaphore>
+
+   __global__ void example_kernel() {
+     // This semaphore is suitable for all threads in the system.
+     cuda::counting_semaphore<cuda::thread_scope_system> a;
+
+     // This semaphore has the same type as the previous one (`a`).
+     cuda::std::counting_semaphore<> b;
+
+     // This semaphore is suitable for all threads on the current processor (e.g. GPU).
+     cuda::counting_semaphore<cuda::thread_scope_device> c;
+
+     // This semaphore is suitable for all threads in the same thread block.
+     cuda::counting_semaphore<cuda::thread_scope_block> d;
+   }
+
+`See it on Godbolt <https://godbolt.org/z/3YrjjTvG6>`_

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/latch.rst

@@ -0,0 +1,70 @@
+.. _libcudacxx-extended-api-synchronization-latch:
+
+``cuda::latch``
+===============
+
+Defined in header ``<cuda/latch>``:
+
+.. code:: cpp
+
+   template <cuda::thread_scope Scope>
+   class cuda::latch;
+
+The class template ``cuda::latch`` is an extended form of `cuda::std::latch <https://en.cppreference.com/w/cpp/thread/latch>`_
+takes an additional :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` argument.
+It has the same interface and semantics as `cuda::std::latch <https://en.cppreference.com/w/cpp/thread/latch>`_.
+
+Concurrency Restrictions
+------------------------
+
+An object of type ``cuda::latch`` or `cuda::std::latch <https://en.cppreference.com/w/cpp/thread/latch>`_ shall not
+be accessed concurrently by CPU and GPU threads unless:
+
+   - it is in unified memory and the `concurrentManagedAccess property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_116f9619ccc85e93bc456b8c69c80e78b>`_
+     is 1, or
+   - it is in CPU memory and the `hostNativeAtomicSupported property <https://docs.nvidia.com/cuda/cuda-runtime-api/structcudaDeviceProp.html#structcudaDeviceProp_1ef82fd7d1d0413c7d6f33287e5b6306f>`_
+     is 1.
+
+Note, for objects of scopes other than ``cuda::thread_scope_system`` this is a data-race, and therefore also prohibited
+regardless of memory characteristics.
+
+Under CUDA Compute Capability 6 (Pascal) or prior, an object of type ``cuda::latch`` or
+`cuda::std::latch <https://en.cppreference.com/w/cpp/thread/latch>`_ may not be used.
+
+Implementation-Defined Behavior
+-------------------------------
+
+For each :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``, the value of
+``cuda::latch<S>::max()`` is as follows:
+
+.. list-table::
+   :widths: 50 50
+   :header-rows: 0
+
+   * - :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``S``
+     - ``cuda::latch<S>::max()``
+   * - Any thread scope
+     - ``cuda::std::numeric_limits<cuda::std::ptrdiff_t>::max()``
+
+Example
+-------
+
+.. code:: cuda
+
+   #include <cuda/latch>
+
+   __global__ void example_kernel() {
+     // This latch is suitable for all threads in the system.
+     cuda::latch<cuda::thread_scope_system> a(10);
+
+     // This latch has the same type as the previous one (`a`).
+     cuda::std::latch b(10);
+
+     // This latch is suitable for all threads on the current processor (e.g. GPU).
+     cuda::latch<cuda::thread_scope_device> c(10);
+
+     // This latch is suitable for all threads in the same thread block.
+     cuda::latch<cuda::thread_scope_block> d(10);
+   }
+
+`See it on Godbolt <https://godbolt.org/z/8v4dcK7fa>`_

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline.rst

@@ -0,0 +1,172 @@
+.. _libcudacxx-extended-api-synchronization-pipeline:
+
+``cuda::pipeline``
+==================
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+
+   pipeline/role
+   pipeline/shared_state
+   pipeline/destructor
+   pipeline/make_pipeline
+   pipeline/quit
+   pipeline/consumer_release
+   pipeline/consumer_wait
+   pipeline/consumer_wait_prior
+   pipeline/pipeline_producer_commit
+   pipeline/producer_acquire
+   pipeline/producer_commit
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope>
+   class cuda::pipeline {
+   public:
+     pipeline() = delete;
+
+     __host__ __device__ ~pipeline();
+
+     pipeline& operator=(pipeline const&) = delete;
+
+     __host__ __device__ void producer_acquire();
+
+     __host__ __device__ void producer_commit();
+
+     __host__ __device__ void consumer_wait();
+
+     template <typename Rep, typename Period>
+     __host__ __device__ bool consumer_wait_for(cuda::std::chrono::duration<Rep, Period> const& duration);
+
+     template <typename Clock, typename Duration>
+     __host__ __device__
+     bool consumer_wait_until(cuda::std::chrono::time_point<Clock, Duration> const& time_point);
+
+     __host__ __device__ void consumer_release();
+
+     __host__ __device__ bool quit();
+   };
+
+The class template ``cuda::pipeline`` provides a coordination mechanism which can sequence
+:ref:`asynchronous operations <libcudacxx-extended-api-asynchronous-operations>`, such as
+:ref:`cuda::memcpy_async <libcudacxx-extended-api-asynchronous-operations-memcpy-async>`, into stages.
+
+A thread interacts with a *pipeline stage* using the following pattern:
+
+  1. Acquire the pipeline stage.
+  2. Commit some operations to the stage.
+  3. Wait for the previously committed operations to complete.
+  4. Release the pipeline stage.
+
+For :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` ``s`` other than
+``cuda::thread_scope_thread``, a
+:ref:`cuda::pipeline_shared_state <libcudacxx-extended-api-synchronization-pipeline-pipeline-shared-state>` is
+required to coordinate the participating threads.
+
+*Pipelines* can be either *unified* or *partitioned*. In a *unified pipeline*, all the participating threads are both
+producers and consumers. In a *partitioned pipeline*, each participating thread is either a producer or a consumer.
+
+.. rubric:: Template Parameters
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``Scope``
+     - The scope of threads participating in the *pipeline*.
+
+.. rubric:: Member Functions
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 1
+
+   * - Member function
+     - Description
+   * - (constructor) [deleted]
+     - ``cuda::pipeline`` is not constructible.
+   * - ``operator=`` [deleted]
+     - ``cuda::pipeline`` is not assignable.
+   * - :ref:`(destructor) <libcudacxx-extended-api-synchronization-pipeline-pipeline-destructor>`
+     -  Destroys the ``cuda::pipeline``.
+   * - :ref:`producer_acquire <libcudacxx-extended-api-synchronization-pipeline-pipeline-producer-acquire>`
+     - Blocks the current thread until the next *pipeline stage* is available.
+   * - :ref:`producer_commit <libcudacxx-extended-api-synchronization-pipeline-pipeline-producer-commit>`
+     - Commits operations previously issued by the current thread to the current *pipeline stage*.
+   * - :ref:`consumer_wait <libcudacxx-extended-api-synchronization-pipeline-pipeline-consumer-wait>`
+     - Blocks the current thread until all operations committed to the current *pipeline stage* complete.
+   * - :ref:`consumer_wait_for <libcudacxx-extended-api-synchronization-pipeline-pipeline-consumer-wait>`
+     - Blocks the current thread until all operations committed to the current *pipeline stage* complete or after the
+       specified timeout duration.
+   * - :ref:`consumer_wait_until <libcudacxx-extended-api-synchronization-pipeline-pipeline-consumer-wait>`
+     - Blocks the current thread until all operations committed to the current *pipeline stage* complete or until
+       specified time point has been reached.
+   * - :ref:`consumer_release <libcudacxx-extended-api-synchronization-pipeline-pipeline-consumer-release>`
+     - Release the current *pipeline stage*.
+   * - :ref:`quit <libcudacxx-extended-api-synchronization-pipeline-pipeline-quit>`
+     - Quits current thread's participation in the *pipeline*.
+
+.. note::
+
+   - A thread role cannot change during the lifetime of the pipeline object.
+
+.. rubric:: Example
+
+.. code:: cuda
+
+   #include <cuda/pipeline>
+   #include <cooperative_groups.h>
+
+   // Disables `pipeline_shared_state` initialization warning.
+   #pragma nv_diag_suppress static_var_with_dynamic_init
+
+   template <typename T>
+   __device__ void compute(T* ptr);
+
+   template <typename T>
+   __global__ void example_kernel(T* global0, T* global1, cuda::std::size_t subset_count) {
+     extern __shared__ T s[];
+     auto group = cooperative_groups::this_thread_block();
+     T* shared[2] = { s, s + 2 * group.size() };
+
+     // Create a pipeline.
+     constexpr auto scope = cuda::thread_scope_block;
+     constexpr auto stages_count = 2;
+     __shared__ cuda::pipeline_shared_state<scope, stages_count> shared_state;
+     auto pipeline = cuda::make_pipeline(group, &shared_state);
+
+     // Prime the pipeline.
+     pipeline.producer_acquire();
+     cuda::memcpy_async(group, shared[0],
+                        &global0[0], sizeof(T) * group.size(), pipeline);
+     cuda::memcpy_async(group, shared[0] + group.size(),
+                        &global1[0], sizeof(T) * group.size(), pipeline);
+     pipeline.producer_commit();
+
+     // Pipelined copy/compute.
+     for (cuda::std::size_t subset = 1; subset < subset_count; ++subset) {
+       pipeline.producer_acquire();
+       cuda::memcpy_async(group, shared[subset % 2],
+                          &global0[subset * group.size()],
+                          sizeof(T) * group.size(), pipeline);
+       cuda::memcpy_async(group, shared[subset % 2] + group.size(),
+                          &global1[subset * group.size()],
+                          sizeof(T) * group.size(), pipeline);
+       pipeline.producer_commit();
+       pipeline.consumer_wait();
+       compute(shared[(subset - 1) % 2]);
+       pipeline.consumer_release();
+     }
+
+     // Drain the pipeline.
+     pipeline.consumer_wait();
+     compute(shared[(subset_count - 1) % 2]);
+     pipeline.consumer_release();
+   }
+
+   template void __global__ example_kernel<int>(int*, int*, cuda::std::size_t);
+
+`See it on Godbolt <https://godbolt.org/z/zc41bWvja>`_

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/consumer_release.rst

@@ -0,0 +1,20 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-consumer-release:
+
+cuda::pipeline::consumer_release
+====================================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope>
+   __host__ __device__
+   void cuda::pipeline<Scope>::consumer_release();
+
+Releases the current *pipeline stage*.
+
+.. note::
+
+   - If the calling thread is a *producer thread*, the behavior is undefined.
+   - If the pipeline is in a :ref:`quitted state <libcudacxx-extended-api-synchronization-pipeline-pipeline-quit>`,
+     the behavior is undefined.

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/consumer_wait.rst

@@ -0,0 +1,55 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-consumer-wait:
+
+cuda::pipeline::consumer_wait
+=================================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   // (1)
+   template <cuda::thread_scope Scope>
+   __host__ __device__
+   void cuda::pipeline<Scope>::consumer_wait();
+
+   // (2)
+   template <cuda::thread_scope Scope>
+   template <typename Rep, typename Period>
+   __host__ __device__
+   bool cuda::pipeline<Scope>::consumer_wait_for(
+     cuda::std::chrono::duration<Rep, Period> const& duration);
+
+   // (3)
+   template <cuda::thread_scope Scope>
+   template <typename Clock, typename Duration>
+   __host__ __device__
+   bool cuda::pipeline<Scope>::consumer_wait_until(
+     cuda::std::chrono::time_point<Clock, Duration> const& time_point);
+
+1. Blocks the current thread until all operations committed to the current *pipeline stage* complete.
+2. Blocks the current thread until all operations committed to the current *pipeline stage* complete or after the
+   specified timeout duration.
+3. Blocks the current thread until all operations committed to the current *pipeline stage* complete or until specified
+   time point has been reached.
+
+.. rubric:: Parameters
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``duration``
+     - An object of type ``cuda::std::chrono::duration`` representing the maximum time to spend waiting.
+   * - ``time_point``
+     - An object of type ``cuda::std::chrono::time_point`` representing the time when to stop waiting.
+
+
+.. rubric:: Return Value
+
+``false`` if the *wait* timed out, ``true`` otherwise.
+
+.. note::
+
+   - If the calling thread is a *producer thread*, the behavior is undefined.
+   - If the pipeline is in a :ref:`quitted state <libcudacxx-extended-api-synchronization-pipeline-pipeline-quit>`,
+     the behavior is undefined.

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/consumer_wait_prior.rst

@@ -0,0 +1,66 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-consumer-wait-prior:
+
+cuda::pipeline_consumer_wait_prior
+======================================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::std::uint8_t Prior>
+   __host__ __device__
+   void cuda::pipeline_consumer_wait_prior(cuda::pipeline<thread_scope_thread>& pipe);
+
+Let *Stage* be the pipeline stage ``Prior`` stages before the current one (counting the current one).
+Blocks the current thread until all operations committed to *pipeline stages* up to *Stage* complete.
+All stages up to *Stage* (exclusive) are implicitly released.
+
+.. rubric:: Template Parameters
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``Prior``
+     - The index of the pipeline stage *Stage* (see above) counting up from the current one. The index of the current stage is ``0``.
+
+.. rubric:: Parameters
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``pipe``
+     - The thread-scoped ``cuda::pipeline`` object to wait on.
+
+.. note::
+
+   - If the pipeline is in a :ref:`quitted state <libcudacxx-extended-api-synchronization-pipeline-pipeline-quit>`,
+     the behavior is undefined.
+
+.. rubric:: Example
+
+.. code:: cuda
+
+   #include <cuda/pipeline>
+
+   __global__ void example_kernel(uint64_t* global, cuda::std::size_t element_count) {
+     extern __shared__ uint64_t shared[];
+
+     cuda::pipeline<cuda::thread_scope_thread> pipe = cuda::make_pipeline();
+     for (cuda::std::size_t i = 0; i < element_count; ++i) {
+       pipe.producer_acquire();
+       cuda::memcpy_async(shared + i, global + i, sizeof(*global), pipe);
+       pipe.producer_commit();
+     }
+
+     // Wait for operations committed in all stages but the last one.
+     cuda::pipeline_consumer_wait_prior<1>(pipe);
+     pipe.consumer_release();
+
+     // Wait for operations committed in all stages.
+     cuda::pipeline_consumer_wait_prior<0>(pipe);
+     pipe.consumer_release();
+   }
+
+`See it on Godbolt <https://godbolt.org/z/aT5hb84PY>`_

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/destructor.rst

@@ -0,0 +1,15 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-destructor:
+
+cuda::pipeline::~pipeline
+=============================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope>
+   __host__ __device__
+   cuda::pipeline<Scope>::~pipeline();
+
+Destructs the pipeline. Calls :ref:`cuda::pipeline::quit <libcudacxx-extended-api-synchronization-pipeline-pipeline-quit>`
+if it was not called by the current thread and destructs the pipeline.

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/make_pipeline.rst

@@ -0,0 +1,125 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-make-pipeline:
+
+cuda::make_pipeline
+=======================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   // (1)
+   __host__ __device__
+   cuda::pipeline<cuda::thread_scope_thread> cuda::make_pipeline();
+
+   // (2)
+   template <typename Group,
+             cuda::thread_scope Scope,
+             cuda::std::uint8_t StagesCount>
+   __host__ __device__
+   cuda::pipeline<Scope>
+   cuda::make_pipeline(Group const& group,
+                       cuda::pipeline_shared_state<Scope, StagesCount>* shared_state);
+
+   // (3)
+   template <typename Group,
+             cuda::thread_scope Scope,
+             cuda::std::uint8_t StagesCount>
+   __host__ __device__
+   cuda::pipeline<Scope>
+   cuda::make_pipeline(Group const& group,
+                       cuda::pipeline_shared_state<Scope, StagesCount>* shared_state,
+                       cuda::std::size_t producer_count);
+
+   // (4)
+   template <typename Group,
+             cuda::thread_scope Scope,
+             cuda::std::uint8_t StagesCount>
+   __host__ __device__
+   cuda::pipeline<Scope>
+   cuda::make_pipeline(Group const& group,
+                       cuda::pipeline_shared_state<Scope, StagesCount>* shared_state,
+                       cuda::pipeline_role role);
+
+1. Creates a *unified pipeline* such that the calling thread is the only participating thread and performs both
+   producer and consumer actions.
+2. Creates a *unified pipeline* such that all the threads in ``group`` are performing both producer and consumer actions.
+3. Creates a *partitioned pipeline* such that ``producer_threads`` number of threads in ``group`` are performing
+   producer actions while the others are performing consumer actions.
+4. Creates a *partitioned pipeline* where each thread's role is explicitly specified.
+
+.. rubric:: Template Parameters
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``Group``
+     - A type satisfying the :ref:`ThreadGroup <libcudacxx-extended-api-thread-groups>` concept.
+
+.. rubric:: Parameters
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``group``
+     - The group of threads.
+   * - ``shared_state``
+     - A pointer to an object of type :ref:`cuda::pipeline_shared_state\<Scope\> <libcudacxx-extended-api-synchronization-pipeline-pipeline-shared-state>`
+       with ``Scope`` including all the threads in ``group``.
+   * - ``producer_count``
+     - The number of *producer threads* in the pipeline.
+   * - ``role``
+     - The role of the current thread in the pipeline.
+
+.. rubric:: Return Value
+
+A ``cuda::pipeline`` object.
+
+.. note::
+
+  - All threads in ``group`` acquire collective ownership of the ``shared_state`` storage.
+  - ``make_pipeline`` must be invoked by every threads in ``group`` such that ``group::sync`` may be invoked.
+  - ``shared_state`` and ``producer_count`` must be the same across all threads in ``group``, else the behavior is undefined.
+  - ``producer_count`` must be strictly inferior to ``group::size``, otherwise the behavior is undefined.
+
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-make-pipeline-example:
+
+.. rubric:: Example
+
+.. code:: cuda
+
+   #include <cuda/pipeline>
+   #include <cooperative_groups.h>
+
+   // Disables `pipeline_shared_state` initialization warning.
+   #pragma nv_diag_suppress static_var_with_dynamic_init
+
+   __global__ void example_kernel() {
+     __shared__ cuda::pipeline_shared_state<cuda::thread_scope_block, 2> pss0;
+     __shared__ cuda::pipeline_shared_state<cuda::thread_scope_block, 2> pss1;
+     __shared__ cuda::pipeline_shared_state<cuda::thread_scope_block, 2> pss2;
+
+     auto group = cooperative_groups::this_thread_block();
+
+     // Create a single thread scoped pipeline.
+     cuda::pipeline<cuda::thread_scope_thread> p0 = cuda::make_pipeline();
+
+     // Create a unified block-scoped pipeline.
+     cuda::pipeline<cuda::thread_scope_block> p1 = cuda::make_pipeline(group, &pss0);
+
+     // Create a partitioned block-scoped pipeline where half the threads are producers.
+     cuda::std::size_t producer_count = group.size() / 2;
+     cuda::pipeline<cuda::thread_scope_block> p2
+       = cuda::make_pipeline(group, &pss1, producer_count);
+
+     // Create a partitioned block-scoped pipeline where all threads with an even
+     // `thread_rank` are producers.
+     auto thread_role = (group.thread_rank() % 2)
+                        ? cuda::pipeline_role::producer
+                       : cuda::pipeline_role::consumer;
+     cuda::pipeline<cuda::thread_scope_block> p3
+       = cuda::make_pipeline(group, &pss2, thread_role);
+   }
+
+`See it on Godbolt <https://godbolt.org/z/aPcGEr64j>`_

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/pipeline_producer_commit.rst

@@ -0,0 +1,60 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-pipeline-producer-commit:
+
+cuda::pipeline_producer_commit
+==================================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope>
+   __host__ __device__
+   void cuda::pipeline_producer_commit(cuda::pipeline<cuda::thread_scope_thread>& pipe,
+                                       cuda::barrier<Scope>& bar);
+
+Binds operations previously issued by the current thread to the named ``cuda::barrier`` such that a
+``cuda::barrier::arrive`` is performed on completion. The bind operation implicitly increments the barrier's
+current phase to account for the subsequent ``cuda::barrier::arrive``, resulting in a net change of 0.
+
+.. rubric:: Parameters
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``pipe``
+     - The thread-scoped ``cuda::pipeline`` object to wait on.
+   * - ``bar``
+     - The ``cuda::barrier`` to arrive on.
+
+.. note::
+
+   - If the pipeline is in a :ref:`quitted state <libcudacxx-extended-api-synchronization-pipeline-pipeline-quit>`,
+     the behavior is undefined.
+
+.. rubric:: Example
+
+.. code:: cuda
+
+   #include <cuda/pipeline>
+
+   // Disables `barrier` initialization warning.
+   #pragma nv_diag_suppress static_var_with_dynamic_init
+
+   __global__ void
+   example_kernel(cuda::std::uint64_t* global, cuda::std::size_t element_count) {
+     extern __shared__ cuda::std::uint64_t shared[];
+     __shared__ cuda::barrier<cuda::thread_scope_block> barrier;
+
+     init(&barrier, 1);
+     cuda::pipeline<cuda::thread_scope_thread> pipe = cuda::make_pipeline();
+
+     pipe.producer_acquire();
+     for (cuda::std::size_t i = 0; i < element_count; ++i)
+       cuda::memcpy_async(shared + i, global + i, sizeof(*global), pipe);
+     pipeline_producer_commit(pipe, barrier);
+     barrier.arrive_and_wait();
+     pipe.consumer_release();
+   }
+
+`See it on Godbolt <https://godbolt.org/z/sGzKe9obf>`_

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/producer_acquire.rst

@@ -0,0 +1,20 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-producer-acquire:
+
+cuda::pipeline::producer_acquire
+====================================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope>
+   __host__ __device__
+   void cuda::pipeline<Scope>::producer_acquire();
+
+Blocks the current thread until the next *pipeline stage* is available.
+
+.. note::
+
+   - If the calling thread is a *consumer thread*, the behavior is undefined.
+   - If the pipeline is in a :ref:`quitted state <libcudacxx-extended-api-synchronization-pipeline-pipeline-quit>`,
+     the behavior is undefined.

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/producer_commit.rst

@@ -0,0 +1,20 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-producer-commit:
+
+cuda::pipeline::producer_commit
+===================================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope>
+   __host__ __device__
+   void cuda::pipeline<Scope>::producer_commit();
+
+Commits operations previously issued by the current thread to the current *pipeline stage*.
+
+.. note::
+
+   - If the calling thread is a *consumer thread*, the behavior is undefined.
+   - If the pipeline is in a :ref:`quitted state <libcudacxx-extended-api-synchronization-pipeline-pipeline-quit>`,
+     the behavior is undefined.

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/quit.rst

@@ -0,0 +1,27 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-quit:
+
+cuda::pipeline::quit
+========================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope>
+   __host__ __device__
+   bool cuda::pipeline<Scope>::quit();
+
+Quits the current thread's participation in the collective ownership of the corresponding
+:ref:`cuda::pipeline_shared_state <libcudacxx-extended-api-synchronization-pipeline-pipeline-shared-state>`.
+Ownership of :ref:`cuda::pipeline_shared_state <libcudacxx-extended-api-synchronization-pipeline-pipeline-shared-state>`
+is released by the last invoking thread.
+
+.. rubric:: Return Value
+
+``true`` if ownership of the *shared state* was released, otherwise ``false``.
+
+.. note::
+
+   - After the completion of a call to ``cuda::pipeline::quit``, no other operations other than
+     :ref:`cuda::pipeline::~pipeline <libcudacxx-extended-api-synchronization-pipeline-pipeline-destructor>` may
+     called by the current thread.

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/role.rst

@@ -0,0 +1,32 @@
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-role:
+
+cuda::pipeline_role
+=======================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   enum class pipeline_role : /* unspecified */ {
+     producer,
+     consumer
+   };
+
+``cuda::pipeline_role`` specifies the role of a particular thread in a partitioned producer/consumer pipeline.
+
+.. rubric:: Constants
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``cuda::pipeline_role::producer``
+     - A producer thread that generates data and issuing
+       :ref:`asynchronous operations <libcudacxx-extended-api-asynchronous-operations>`.
+   * - ``cuda::pipeline_role::consumer``
+     - A consumer thread that consumes data and waiting for previously
+       :ref:`asynchronous operations <libcudacxx-extended-api-asynchronous-operations>` to complete.
+
+.. rubric:: Example
+
+See the :ref:`cuda::make_pipeline example <libcudacxx-extended-api-synchronization-pipeline-pipeline-make-pipeline-example>`.

cccl_upstream/docs/libcudacxx/extended_api/synchronization_primitives/pipeline/shared_state.rst

@@ -0,0 +1,119 @@
+
+.. _libcudacxx-extended-api-synchronization-pipeline-pipeline-shared-state:
+
+cuda::pipeline_shared_state
+===============================
+
+Defined in header ``<cuda/pipeline>``:
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope, cuda::std::uint8_t StagesCount>
+   class cuda::pipeline_shared_state {
+   public:
+     __host__ __device__
+     pipeline_shared_state();
+
+     ~pipeline_shared_state() = default;
+
+     pipeline_shared_state(pipeline_shared_state const&) = delete;
+
+     pipeline_shared_state(pipeline_shared_state&&) = delete;
+   };
+
+The class template ``cuda::pipeline_shared_state`` is a storage type used to coordinate the threads participating
+in a ``cuda::pipeline``.
+
+.. rubric:: Template Parameters
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 0
+
+   * - ``Scope``
+     - A :ref:`cuda::thread_scope <libcudacxx-extended-api-memory-model-thread-scopes>` denoting a scope including all
+       the threads participating in the ``cuda::pipeline``. ``Scope`` cannot be ``cuda::thread_scope_thread``.
+   * - ``StagesCount``
+     - The number of stages for the *pipeline*.
+
+.. rubric:: Member Functions
+
+.. list-table::
+   :widths: 25 75
+   :header-rows: 1
+
+   * - Member function
+     - Description
+   * - ``(constructor)``
+     - Constructs a ``cuda::pipeline_shared_state``.
+   * - ``(destructor)`` [implicitly declared]
+     - Destroys the ``cuda::pipeline_shared_state``.
+   * - ``operator=`` [deleted]
+     -  ``cuda::pipeline_shared_state`` is not assignable.
+
+.. rubric:: Constructor
+
+.. code:: cuda
+
+   template <cuda::thread_scope Scope, cuda::std::uint8_t StagesCount>
+   __host__ __device__
+   cuda::pipeline_shared_state();
+
+   template <cuda::thread_scope Scope, cuda::std::uint8_t StagesCount>
+   cuda::pipeline_shared_state(cuda::pipeline_shared_state const&) = delete;
+
+   template <cuda::thread_scope Scope, cuda::std::uint8_t StagesCount>
+   cuda::pipeline_shared_state(cuda::pipeline_shared_state&&) = delete;
+
+Construct a ``cuda::pipeline`` *shared state* object.
+
+.. code:: cuda
+
+   #include <cuda/pipeline>
+
+   #pragma nv_diag_suppress static_var_with_dynamic_init
+
+   __global__ void example_kernel() {
+     __shared__ cuda::pipeline_shared_state<cuda::thread_scope_block, 2> shared_state;
+   }
+
+`See it on Godbolt <https://godbolt.org/z/K4vKq4vd3>`__
+
+.. rubric:: NVCC ``__shared__`` Initialization Warnings
+
+When using libcu++ with NVCC, a ``__shared__`` ``cuda::pipeline_shared_state`` will lead to the following warning
+because ``__shared__`` variables are not initialized:
+
+.. code:: bash
+
+   warning: dynamic initialization is not supported for a function-scope static
+   __shared__ variable within a __device__/__global__ function
+
+It can be silenced using ``#pragma nv_diag_suppress static_var_with_dynamic_init``.
+
+.. rubric:: Example
+
+.. code:: cuda
+
+   #include <cuda/pipeline>
+
+   // Disables `pipeline_shared_state` initialization warning.
+   #pragma nv_diag_suppress static_var_with_dynamic_init
+
+   __global__ void example_kernel(char* device_buffer, char* sysmem_buffer) {
+     // Allocate a 2 stage block scoped shared state in shared memory.
+     __shared__ cuda::pipeline_shared_state<cuda::thread_scope_block, 2> pss0;
+
+     // Allocate a 2 stage block scoped shared state in device memory.
+     auto* pss1 = new cuda::pipeline_shared_state<cuda::thread_scope_block, 2>;
+
+     // Construct a 2 stage device scoped shared state in device memory.
+     auto* pss2 =
+       new (device_buffer) cuda::pipeline_shared_state<cuda::thread_scope_device, 2>;
+
+     // Construct a 2 stage system scoped shared state in system memory.
+     auto* pss3 =
+       new (sysmem_buffer) cuda::pipeline_shared_state<cuda::thread_scope_system, 2>;
+   }
+
+`See it on Godbolt <https://godbolt.org/z/M9ah7r1Yx>`__