.. _cccl-runtime-memory-pools: .. |cuda_memory_pool_attributes| replace:: ``cuda::memory_pool_attributes`` .. _cuda_memory_pool_attributes: ../api/memory_pool_attributes.html Memory Pools ============ Memory pools provide efficient, stream-ordered memory allocation using CUDA's memory pool API. They support both synchronous and stream-ordered allocation/deallocation and can be configured with various memory spaces, properties and attributes. Memory pool objects implement the :ref:`cuda::mr::resource ` interface with ``allocate(stream, size, alignment)`` and ``deallocate(stream, ptr, size, alignment)`` member functions. They also provide synchronous variants with ``allocate_sync(size, alignment)`` and ``deallocate_sync(ptr, size, alignment)`` member functions. For all of them, the alignment argument is optional. For the full memory resource model and property system, see :ref:`Memory Resources (Extended API) `. Host memory pools are supported on CUDA 12.9 and later. Managed memory pools are supported on CUDA 13.0 and later and are not supported on Windows. For those cases use :ref:`cuda::mr::legacy_pinned_memory_resource ` and :ref:`cuda::mr::legacy_managed_memory_resource ` instead. :cpp:struct:`cuda::device_memory_pool` --------------------------------------- .. _cccl-runtime-memory-pools-device-memory-pool: :cpp:struct:`cuda::device_memory_pool` allocates device memory using CUDA's stream-ordered memory pool API (``cudaMallocFromPoolAsync`` / ``cudaFreeAsync``). When constructed, it creates and owns an underlying ``cudaMemPool_t`` with location type set to ``cudaMemLocationTypeDevice``. Availability: CCCL 3.2.0 / CUDA 13.2 Example: .. code:: cpp #include #include #include void use_device_pool(cuda::stream_ref stream) { // Create a device memory pool cuda::device_memory_pool pool{cuda::devices[0]}; // Allocate memory in stream order void* ptr = pool.allocate(stream, 1024, 16); // Use memory... // Deallocate in stream order pool.deallocate(stream, ptr, 1024, 16); } :cpp:class:`cuda::device_memory_pool_ref` ------------------------------------------- .. _cccl-runtime-memory-pools-device-memory-pool-ref: :cpp:class:`cuda::device_memory_pool_ref` is a non-owning reference to a device memory pool. It does not own the underlying ``cudaMemPool_t``, so the user must ensure the pool's lifetime exceeds the reference's lifetime. Availability: CCCL 3.2.0 / CUDA 13.2 Example: .. code:: cpp #include #include void use_pool_ref(cuda::stream_ref stream, cuda::device_memory_pool_ref pool_ref) { void* ptr = pool_ref.allocate(stream, 1024); // Use memory... pool_ref.deallocate(stream, ptr, 1024); } :cpp:struct:`cuda::managed_memory_pool` ---------------------------------------- .. _cccl-runtime-memory-pools-managed-memory-pool: :cpp:struct:`cuda::managed_memory_pool` allocates managed (unified) memory using CUDA's memory pool API. It creates and owns an underlying ``cudaMemPool_t`` with allocation type set to ``cudaMemAllocationTypeManaged``. Managed memory is accessible from both host and device. Availability: CCCL 3.2.0 / CUDA 13.2 (requires CTK 13.0+). Not supported on Windows Example: .. code:: cpp #include #include void use_managed_pool(cuda::stream_ref stream) { cuda::managed_memory_pool pool{}; // Allocate managed memory void* ptr = pool.allocate(stream, 1024); // Accessible from both host and device // Use memory... pool.deallocate(stream, ptr, 1024); } :cpp:class:`cuda::managed_memory_pool_ref` -------------------------------------------- .. _cccl-runtime-memory-pools-managed-memory-pool-ref: :cpp:class:`cuda::managed_memory_pool_ref` is a non-owning reference to a managed memory pool. Availability: CCCL 3.2.0 / CUDA 13.2 (requires CTK 13.0+). Not supported on Windows :cpp:struct:`cuda::pinned_memory_pool` --------------------------------------- .. _cccl-runtime-memory-pools-pinned-memory-pool: :cpp:struct:`cuda::pinned_memory_pool` allocates pinned (page-locked) host memory using CUDA's memory pool API. Pinned memory enables faster host-to-device transfers and can be accessed from all devices. The pool can be optionally created for a specific host NUMA node. Availability: CCCL 3.2.0 / CUDA 13.2 (requires CTK 12.9+) Example: .. code:: cpp #include #include void use_pinned_pool(cuda::stream_ref stream) { // Create pinned memory pool cuda::pinned_memory_pool pool{}; // Allocate pinned memory void* ptr = pool.allocate(stream, 1024); // Use for fast host-device transfers... pool.deallocate(stream, ptr, 1024); } // With NUMA node void use_pinned_pool_numa(cuda::stream_ref stream, int numa_id) { cuda::pinned_memory_pool pool{numa_id}; void* ptr = pool.allocate(stream, 1024); // Use memory... pool.deallocate(stream, ptr, 1024); } :cpp:class:`cuda::pinned_memory_pool_ref` ------------------------------------------- .. _cccl-runtime-memory-pools-pinned-memory-pool-ref: :cpp:class:`cuda::pinned_memory_pool_ref` is a non-owning reference to a pinned memory pool. Availability: CCCL 3.2.0 / CUDA 13.2 (requires CTK 12.9+) Default Memory Pools -------------------- .. _cccl-runtime-memory-pools-default-pools: CUDA provides default memory pools for each memory type. These pools are managed by the CUDA runtime and can be accessed through helper functions. Default pools are useful when you don't need custom pool configuration and want to use the system defaults. :cpp:func:`cuda::device_default_memory_pool` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. _cccl-runtime-memory-pools-device-default: :cpp:func:`cuda::device_default_memory_pool` returns a non-owning reference to the default device memory pool for the specified device. The default pool is created automatically by CUDA and is shared across all users of the device. Availability: CCCL 3.2.0 / CUDA 13.2 Example: .. code:: cpp #include #include #include void use_default_device_pool(cuda::stream_ref stream) { // Get the default device memory pool auto pool = cuda::device_default_memory_pool(cuda::devices[0]); // Allocate from the default pool void* ptr = pool.allocate(stream, 1024); // Use memory... // Deallocate back to the pool pool.deallocate(stream, ptr, 1024); } :cpp:func:`cuda::managed_default_memory_pool` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. _cccl-runtime-memory-pools-managed-default: :cpp:func:`cuda::managed_default_memory_pool` returns a non-owning reference to the default managed (unified) memory pool. Availability: CCCL 3.2.0 / CUDA 13.2 Example: .. code:: cpp #include #include void use_default_managed_pool(cuda::stream_ref stream) { // Get the default managed memory pool auto pool = cuda::managed_default_memory_pool(); // Allocate managed memory void* ptr = pool.allocate(stream, 1024); // Accessible from both host and device // Use memory... pool.deallocate(stream, ptr, 1024); } :cpp:func:`cuda::pinned_default_memory_pool` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. _cccl-runtime-memory-pools-pinned-default: :cpp:func:`cuda::pinned_default_memory_pool` returns a non-owning reference to the default pinned (page-locked) host memory pool. Availability: CCCL 3.2.0 / CUDA 13.2 Example: .. code:: cpp #include #include void use_default_pinned_pool(cuda::stream_ref stream) { // Get the default pinned memory pool auto pool = cuda::pinned_default_memory_pool(); // Allocate pinned memory void* ptr = pool.allocate(stream, 1024); // Use for fast host-device transfers... pool.deallocate(stream, ptr, 1024); } Notes on Default Pools ~~~~~~~~~~~~~~~~~~~~~~ - Default pools are created automatically by CUDA and shared across the application - The pools are returned as non-owning references (``*_pool_ref`` types) - Default pools use CUDA's default configuration and cannot be destroyed - Multiple calls to the same getter function return references to the same pool - Default pools are thread-safe and can be used concurrently from multiple threads - Underlying CUDA default memory pools have 0 release threshold by default. First access to a default pool through one of the getters above will set the release threshold to the maximum value, unless previously modified by the user. Memory Pool Properties ---------------------- .. _cccl-runtime-memory-pools-pool-properties: :cpp:struct:`cuda::memory_pool_properties` controls memory pool creation options: - ``initial_pool_size`` - Initial size of the pool (default: 0) - ``release_threshold`` - Threshold at which unused memory is released (default: no limit on the reserved memory) - ``allocation_handle_type`` - Handle type for inter-process sharing (default: none) - ``max_pool_size`` - Maximum size of the pool (default: no limit on the pool size) Availability: CCCL 3.2.0 / CUDA 13.2 Example: .. code:: cpp #include #include void create_pool_with_properties() { cuda::memory_pool_properties props{}; props.initial_pool_size = 1024 * 1024; // 1 MB initial size props.release_threshold = 10 * 1024 * 1024; // Release if over 10 MB cuda::device_memory_pool pool{cuda::devices[0], props}; } Memory Pool Attributes ---------------------- .. _cccl-runtime-memory-pools-pool-attributes: |cuda_memory_pool_attributes|_ provides access to pool attributes for querying and configuration: - ``release_threshold`` - Get/set the release threshold, which controls how much memory the pool can keep reserved, both used and unused - ``reuse_follow_event_dependencies`` - Enable/disable reuse across streams with event dependencies - ``reuse_allow_opportunistic`` - Enable/disable opportunistic reuse - ``reuse_allow_internal_dependencies`` - Enable/disable reuse with internal dependencies - ``reserved_mem_current`` - Query current reserved memory (read-only) - ``used_mem_current`` - Query current used memory (read-only) - ``reserved_mem_high`` - Get/set high watermark for reserved memory - ``used_mem_high`` - Get/set high watermark for used memory The following additional read-only creation attributes are available with CUDA 13.3: - ``allocation_type`` - Query the allocation type of the pool - ``export_handle_types`` - Query the export handle types available for the pool - ``location`` - Query the location of the pool as a ``cuda::memory_location`` - ``location_id`` - Query only the location id of the pool - ``location_type`` - Query only the location type of the pool - ``max_pool_size`` - Query the maximum size of the pool - ``hw_decompress_enabled`` - Query whether hardware decompression is enabled for the pool Availability: CCCL 3.2.0 / CUDA 13.2 Example: .. code:: cpp #include #include void configure_pool_attributes() { cuda::device_memory_pool pool{cuda::devices[0]}; // Set release threshold pool.set_attribute(cuda::memory_pool_attributes::release_threshold, 5 * 1024 * 1024); // Enable opportunistic reuse pool.set_attribute(cuda::memory_pool_attributes::reuse_allow_opportunistic, true); // Query current usage auto reserved = pool.attribute(cuda::memory_pool_attributes::reserved_mem_current); auto used = pool.attribute(cuda::memory_pool_attributes::used_mem_current); } Pool Management --------------- .. _cccl-runtime-memory-pools-pool-management: Memory pools provide additional management functions: - ``trim_to(min_bytes)`` - Release memory down to a minimum size - ``enable/disable_access_from(devices)`` - Enable or disable access from specific devices (for peer access or access to host pinned memory) - ``get()`` - Get the underlying ``cudaMemPool_t`` handle - ``release()`` - Release ownership of the pool handle Availability: CCCL 3.2.0 / CUDA 13.2 Example: .. code:: cpp #include #include void manage_pool() { cuda::pinned_memory_pool pool{}; // Enable access from all devices pool.enable_access_from(cuda::devices); // Trim pool to 1 MB minimum pool.trim_to(1024 * 1024); // Get native handle cudaMemPool_t handle = pool.get(); }