dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

[INFRA] Import NVIDIA/CCCL upstream as optimization reference library

Browse Source 
CCCL (CUDA C++ Core Libraries) provides:
- CUB: device/block/warp-level GPU primitives (reduce, scan, sort, topk)
- Thrust: high-level parallel algorithms (transform_reduce, sort, scan)
- libcudacxx: CUDA C++ standard library (atomics, barriers, memory)
- cudax: experimental features (memory resources, allocators)
- Tuning policies: per-SM hardware-specific algorithm parameters

Competition optimization vectors mapped to CCCL:
- Output TPS (83% weight): warp_reduce, block_reduce, device_topk
- Input TPS (14% weight): device_scan, block_load, prefetch
- Cache TPS (3% weight): prefix caching strategy patterns
- Memory (0.9 util): pooled/cached/buddy allocators

Source: https://github.com/NVIDIA/cccl (shallow clone, HEAD only)
License: Apache-2.0
This commit is contained in:
EngineX CI
2026-07-30 09:35:51 +00:00
parent b4d01f481e
commit 56fd68e7dd
8871 changed files with 1454674 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

65
cccl_upstream/thrust/CMakeLists.txt Normal file
View File

@@ -0,0 +1,65 @@
if (NOT CCCL_ENABLE_THRUST)
include(cmake/ThrustAddSubdir.cmake)
return()
endif()
cmake_minimum_required(VERSION 3.21)
project(Thrust LANGUAGES CXX) # CUDA enabled later only if needed
option(
THRUST_ENABLE_HEADER_TESTING
"Test that all public headers compile."
"ON"
)
option(THRUST_ENABLE_TESTING "Build Thrust testing suite." "ON")
option(THRUST_ENABLE_EXAMPLES "Build Thrust examples." "ON")
# Allow the user to optionally select offset type dispatch to fixed 32 or 64 bit types
set(
THRUST_DISPATCH_TYPE
"Dynamic"
CACHE STRING
"Select Thrust offset type dispatch."
)
set_property(
CACHE THRUST_DISPATCH_TYPE
PROPERTY STRINGS "Dynamic" "Force32bit" "Force64bit"
)
#include first:
include(cmake/ThrustUtilities.cmake)
include(cmake/ThrustBuildCompilerTargets.cmake)
include(cmake/ThrustBuildTargetList.cmake)
include(cmake/ThrustFindThrust.cmake)
include(cmake/ThrustMultiConfig.cmake)
thrust_configure_multiconfig()
thrust_find_thrust()
thrust_build_compiler_targets()
thrust_update_system_found_flags()
if (THRUST_CUDA_FOUND)
include(cmake/ThrustCudaConfig.cmake)
endif()
thrust_build_target_list()
message(STATUS "CPP system found? ${THRUST_CPP_FOUND}")
message(STATUS "CUDA system found? ${THRUST_CUDA_FOUND}")
message(STATUS "TBB system found? ${THRUST_TBB_FOUND}")
message(STATUS "OMP system found? ${THRUST_OMP_FOUND}")
if (THRUST_ENABLE_HEADER_TESTING)
include(cmake/ThrustHeaderTesting.cmake)
endif()
if (THRUST_ENABLE_TESTING)
add_subdirectory(testing)
endif()
if (THRUST_ENABLE_EXAMPLES)
add_subdirectory(examples)
endif()
if (CCCL_ENABLE_BENCHMARKS)
add_subdirectory(benchmarks)
endif()

249
cccl_upstream/thrust/LICENSE Normal file
View File

@@ -0,0 +1,249 @@
Unless otherwise noted, Thrust's source code is released under the Apache
License, Version 2.0:
================================================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
================================================================================
Some portions of Thrust may be licensed under other compatible open-source
licenses. Any divergence from the Apache 2 license will be noted in the source
code where applicable.
Portions under other terms include, but are not limited to:
================================================================================
Various C++ utility classes in Thrust are based on the Boost Iterator, Tuple,
System, and Random Number libraries, which are provided under the Boost Software
License:
Boost Software License - Version 1.0 - August 17th, 2003
Permission is hereby granted, free of charge, to any person or organization
obtaining a copy of the software and accompanying documentation covered by
this license (the "Software") to use, reproduce, display, distribute,
execute, and transmit the Software, and to prepare derivative works of the
Software, and to permit third-parties to whom the Software is furnished to
do so, all subject to the following:
The copyright notices in the Software and this entire statement, including
the above license grant, this restriction and the following disclaimer,
must be included in all copies of the Software, in whole or in part, and
all derivative works of the Software, unless such copies or derivative
works are solely in the form of machine-executable object code generated by
a source language processor.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT
SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE
FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.
================================================================================
Portions of the thrust::complex implementation are derived from FreeBSD with the
following terms:
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice[1] unmodified, this list of conditions, and the following
disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
[1] Individual copyright notices from the original authors are included in
the relevant source files.
================================================================================

91
cccl_upstream/thrust/benchmarks/CMakeLists.txt Normal file
View File

@@ -0,0 +1,91 @@
include(${CMAKE_SOURCE_DIR}/benchmarks/cmake/CCCLBenchmarkRegistry.cmake)
cccl_get_nvbench()
cccl_get_nvbench_helper()
set(benches_root "${CMAKE_CURRENT_LIST_DIR}")
function(get_recursive_subdirs subdirs)
set(dirs)
file(
GLOB_RECURSE contents
CONFIGURE_DEPENDS
LIST_DIRECTORIES ON
"${CMAKE_CURRENT_LIST_DIR}/bench/*"
)
foreach (test_dir IN LISTS contents)
if (IS_DIRECTORY "${test_dir}")
list(APPEND dirs "${test_dir}")
endif()
endforeach()
set(${subdirs} "${dirs}" PARENT_SCOPE)
endfunction()
function(add_bench target_name bench_name bench_src)
set(bench_target ${bench_name})
set(${target_name} ${bench_target} PARENT_SCOPE)
cccl_add_executable(${bench_target} SOURCES "${bench_src}")
target_link_libraries(
${bench_target}
PRIVATE #
cccl.nvbench_helper
nvbench::main
)
endfunction()
function(thrust_wrap_bench_in_cpp cpp_file_var cu_file thrust_target)
thrust_get_target_property(prefix ${thrust_target} PREFIX)
set(wrapped_source_file "${cu_file}")
set(cpp_file "${CMAKE_CURRENT_BINARY_DIR}/${prefix}/${cu_file}.cpp")
configure_file(
"${Thrust_SOURCE_DIR}/cmake/wrap_source_file.cpp.in"
"${cpp_file}"
)
set(${cpp_file_var} "${cpp_file}" PARENT_SCOPE)
endfunction()
function(add_bench_dir bench_dir)
file(GLOB bench_srcs CONFIGURE_DEPENDS "${bench_dir}/*.cu")
file(RELATIVE_PATH bench_prefix "${benches_root}" "${bench_dir}")
file(TO_CMAKE_PATH "${bench_prefix}" bench_prefix)
string(REPLACE "/" "." bench_prefix "${bench_prefix}")
foreach (bench_src IN LISTS bench_srcs)
foreach (thrust_target IN LISTS THRUST_TARGETS)
thrust_get_target_property(config_prefix ${thrust_target} PREFIX)
thrust_get_target_property(config_device ${thrust_target} DEVICE)
# Wrap the .cu file in .cpp for non-CUDA backends
if ("CUDA" STREQUAL "${config_device}")
set(real_bench_src "${bench_src}")
else()
thrust_wrap_bench_in_cpp(real_bench_src "${bench_src}" ${thrust_target})
endif()
get_filename_component(bench_name "${bench_src}" NAME_WLE)
string(PREPEND bench_name "${config_prefix}.${bench_prefix}.")
register_cccl_benchmark("${bench_name}" "")
string(APPEND bench_name ".base")
add_bench(base_bench_target ${bench_name} "${real_bench_src}")
cccl_configure_target(${bench_name})
target_link_libraries(${bench_name} PRIVATE ${thrust_target})
if ("CUDA" STREQUAL "${config_device}")
target_compile_options(
${bench_name}
PRIVATE "$<$<COMPILE_LANG_AND_ID:CUDA,NVIDIA>:--extended-lambda>"
)
endif()
endforeach()
endforeach()
endfunction()
get_recursive_subdirs(subdirs)
foreach (subdir IN LISTS subdirs)
add_bench_dir("${subdir}")
endforeach()

33
cccl_upstream/thrust/benchmarks/bench/adjacent_difference/basic.cu Normal file
View File

@@ -0,0 +1,33 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/adjacent_difference.h>
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> input = generate(elements);
thrust::device_vector<T> output(elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
thrust::adjacent_difference(policy(alloc, launch), input.cbegin(), input.cend(), output.begin());
});
}
using types = nvbench::type_list<int8_t, int16_t, int32_t, int64_t, float, double>;
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

52
cccl_upstream/thrust/benchmarks/bench/adjacent_difference/custom.cu Normal file
View File

@@ -0,0 +1,52 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/adjacent_difference.h>
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include "nvbench_helper.cuh"
template <typename T>
struct custom_op
{
T val;
custom_op() = delete;
explicit custom_op(T val)
: val(val)
{}
__device__ T operator()(const T& lhs, const T& rhs)
{
return lhs * rhs + val; // Hope to gen mad
}
};
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> input = generate(elements);
thrust::device_vector<T> output(elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::adjacent_difference(
policy(alloc, launch), input.cbegin(), input.cend(), output.begin(), custom_op<T>{42});
});
}
using types = nvbench::type_list<int8_t, int16_t, int32_t, int64_t, float, double>;
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

33
cccl_upstream/thrust/benchmarks/bench/adjacent_difference/in_place.cu Normal file
View File

@@ -0,0 +1,33 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/adjacent_difference.h>
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> vec(elements, 0);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::adjacent_difference(policy(alloc, launch), vec.begin(), vec.end(), vec.begin());
});
}
using types = nvbench::type_list<int8_t, int16_t, int32_t, int64_t, float, double>;
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

41
cccl_upstream/thrust/benchmarks/bench/all_of/basic.cu Normal file
View File

@@ -0,0 +1,41 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/fill.h>
#include <thrust/logical.h>
#include <cuda/functional>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
T val = 1;
// set up input
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const auto common_prefix = state.get_float64("MismatchAt");
const auto mismatch_point = static_cast<std::size_t>(static_cast<double>(elements) * common_prefix);
thrust::device_vector<T> dinput(elements, thrust::no_init);
thrust::fill(dinput.begin(), dinput.begin() + mismatch_point, T{0});
thrust::fill(dinput.begin() + mismatch_point, dinput.end(), val);
state.add_global_memory_reads<T>(mismatch_point + 1);
state.add_global_memory_writes<size_t>(1);
caching_allocator_t alloc{};
state.exec(
nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
do_not_optimize(thrust::all_of(policy(alloc, launch), dinput.begin(), dinput.end(), cuda::equal_to_value{val}));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_float64_axis("MismatchAt", std::vector{1.0, 0.5, 0.01});

41
cccl_upstream/thrust/benchmarks/bench/any_of/basic.cu Normal file
View File

@@ -0,0 +1,41 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/fill.h>
#include <thrust/logical.h>
#include <cuda/functional>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
T val = 1;
// set up input
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const auto common_prefix = state.get_float64("MismatchAt");
const auto mismatch_point = static_cast<std::size_t>(static_cast<double>(elements) * common_prefix);
thrust::device_vector<T> dinput(elements, thrust::no_init);
thrust::fill(dinput.begin(), dinput.begin() + mismatch_point, T{0});
thrust::fill(dinput.begin() + mismatch_point, dinput.end(), val);
state.add_global_memory_reads<T>(mismatch_point + 1);
state.add_global_memory_writes<size_t>(1);
caching_allocator_t alloc{};
state.exec(
nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
do_not_optimize(thrust::any_of(policy(alloc, launch), dinput.begin(), dinput.end(), cuda::equal_to_value{val}));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_float64_axis("MismatchAt", std::vector{1.0, 0.5, 0.01});

68
cccl_upstream/thrust/benchmarks/bench/copy/basic.cu Normal file
View File

@@ -0,0 +1,68 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/copy.h>
#include <thrust/count.h>
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <nvbench_helper.cuh>
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> input(elements, T{1});
thrust::device_vector<T> output(elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::copy(policy(alloc, launch), input.cbegin(), input.cend(), output.begin());
});
}
// Non-trivially-copyable/relocatable type which is not allowed to be copied using std::memcpy or cudaMemcpy
struct non_trivial
{
int a;
int b;
non_trivial() = default;
_CCCL_HOST_DEVICE explicit non_trivial(int i)
: a(i)
, b(i)
{}
// the user-defined copy constructor prevents the type from being trivially copyable
// NOLINTNEXTLINE(modernize-use-equals-default)
_CCCL_HOST_DEVICE non_trivial(const non_trivial& nt)
: a(nt.a)
, b(nt.b)
{}
// NOLINTNEXTLINE(modernize-use-equals-default)
_CCCL_HOST_DEVICE non_trivial& operator=(const non_trivial& nt)
{
a = nt.a;
b = nt.b;
return *this;
}
};
static_assert(!::cuda::std::is_trivially_copyable<non_trivial>::value); // as required by the C++ standard
static_assert(!thrust::is_trivially_relocatable<non_trivial>::value); // thrust uses this check internally
using types =
nvbench::type_list<nvbench::uint8_t, nvbench::uint16_t, nvbench::uint32_t, nvbench::uint64_t, non_trivial>;
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

43
cccl_upstream/thrust/benchmarks/bench/copy/if.cu Normal file
View File

@@ -0,0 +1,43 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/copy.h>
#include <thrust/count.h>
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <nvbench_helper.cuh>
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
using select_op_t = less_then_t<T>;
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const bit_entropy entropy = str_to_entropy(state.get_string("Entropy"));
const T val = lerp_min_max<T>(entropy_to_probability(entropy));
select_op_t select_op{val};
thrust::device_vector<T> input = generate(elements);
const auto selected_elements = thrust::count_if(input.cbegin(), input.cend(), select_op);
thrust::device_vector<T> output(selected_elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(selected_elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::copy_if(policy(alloc, launch), input.cbegin(), input.cend(), output.begin(), select_op);
});
}
using types = nvbench::type_list<int8_t, int16_t, int32_t, int64_t>;
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_string_axis("Entropy", {"1.000", "0.544", "0.000"});

44
cccl_upstream/thrust/benchmarks/bench/copy_if/basic.cu Normal file
View File

@@ -0,0 +1,44 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/copy.h>
#include <thrust/device_vector.h>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
struct is_even
{
template <class T>
__device__ constexpr bool operator()(const T& val) const noexcept
{
return static_cast<int>(val) % 2 == 0;
}
};
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in = generate(elements, bit_entropy::_1_000, T{0}, T{42});
thrust::device_vector<T> out(elements, thrust::no_init);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
do_not_optimize(thrust::copy_if(policy(alloc, launch), in.begin(), in.end(), out.begin(), is_even{}));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

33
cccl_upstream/thrust/benchmarks/bench/count/basic.cu Normal file
View File

@@ -0,0 +1,33 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/count.h>
#include <thrust/device_vector.h>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in = generate(elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(1);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
do_not_optimize(thrust::count(policy(alloc, launch), in.begin(), in.end(), T{42}));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

42
cccl_upstream/thrust/benchmarks/bench/count_if/basic.cu Normal file
View File

@@ -0,0 +1,42 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/count.h>
#include <thrust/device_vector.h>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
struct equal_to_42
{
template <class T>
__device__ constexpr bool operator()(const T& val) const noexcept
{
return val == 42;
}
};
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in = generate(elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(1);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
do_not_optimize(thrust::count_if(policy(alloc, launch), in.begin(), in.end(), equal_to_42{}));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

38
cccl_upstream/thrust/benchmarks/bench/equal/basic.cu Normal file
View File

@@ -0,0 +1,38 @@
// SPDX-FileCopyrightText: Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/equal.h>
#include <thrust/execution_policy.h>
#include "nvbench_helper.cuh"
template <typename T>
static void benchmark(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> a(elements, T{1});
thrust::device_vector<T> b(elements, T{1});
const auto common_prefix = state.get_float64("CommonPrefixRatio");
const auto same_elements =
std::min(static_cast<std::size_t>(static_cast<double>(elements) * common_prefix), elements);
caching_allocator_t alloc;
thrust::fill(policy(alloc), b.begin() + same_elements, b.end(), T{2});
state.add_element_count(elements);
state.add_global_memory_reads<T>(2 * std::max(same_elements, std::size_t(1))); // using `same_elements` instead
// of `elements` corresponds to the
// actual elements read in an early
// exit
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
do_not_optimize(thrust::equal(policy(alloc, launch), a.begin(), a.end(), b.begin()));
});
}
NVBENCH_BENCH_TYPES(benchmark, NVBENCH_TYPE_AXES(integral_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_float64_axis("CommonPrefixRatio", std::vector{1.0, 0.5, 0.0});

54
cccl_upstream/thrust/benchmarks/bench/extrema/basic.cu Normal file
View File

@@ -0,0 +1,54 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/extrema.h>
#include "nvbench_helper.cuh"
template <typename T, typename Func>
static void bench_extremum(nvbench::state& state, nvbench::type_list<T>, Func func)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in = generate(elements);
using offset_t = typename decltype(in.cbegin())::difference_type;
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<offset_t>(1);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
do_not_optimize(func(policy(alloc, launch), in.cbegin(), in.cend()));
});
}
template <typename T>
static void min_element(nvbench::state& state, nvbench::type_list<T> list)
{
bench_extremum(state, list, [](auto&&... args) {
return thrust::min_element(args...);
});
}
NVBENCH_BENCH_TYPES(min_element, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("min_element")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));
template <typename T>
static void max_element(nvbench::state& state, nvbench::type_list<T> list)
{
bench_extremum(state, list, [](auto&&... args) {
return thrust::max_element(args...);
});
}
NVBENCH_BENCH_TYPES(max_element, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("max_element")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

30
cccl_upstream/thrust/benchmarks/bench/fill/basic.cu Normal file
View File

@@ -0,0 +1,30 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/fill.h>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> output(elements);
state.add_element_count(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::fill(policy(alloc, launch), output.begin(), output.end(), T{42});
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

33
cccl_upstream/thrust/benchmarks/bench/fill_n/basic.cu Normal file
View File

@@ -0,0 +1,33 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/fill.h>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> output(elements);
state.add_element_count(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
do_not_optimize(thrust::fill_n(policy(alloc, launch), output.begin(), elements, T{42}));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

39
cccl_upstream/thrust/benchmarks/bench/find/basic.cu Normal file
View File

@@ -0,0 +1,39 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/find.h>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
T val = 1;
// set up input
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const auto common_prefix = state.get_float64("MismatchAt");
const auto mismatch_point = static_cast<std::size_t>(static_cast<double>(elements) * common_prefix);
thrust::device_vector<T> dinput(elements, thrust::no_init);
thrust::fill(dinput.begin(), dinput.begin() + mismatch_point, T{0});
thrust::fill(dinput.begin() + mismatch_point, dinput.end(), val);
state.add_global_memory_reads<T>(mismatch_point + 1);
state.add_global_memory_writes<size_t>(1);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
do_not_optimize(thrust::find(policy(alloc, launch), dinput.begin(), dinput.end(), val));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_float64_axis("MismatchAt", std::vector{1.0, 0.5, 0.01});

41
cccl_upstream/thrust/benchmarks/bench/find_if/basic.cu Normal file
View File

@@ -0,0 +1,41 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/find.h>
#include <cuda/functional>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
T val = 1;
// set up input
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const auto common_prefix = state.get_float64("MismatchAt");
const auto mismatch_point = static_cast<std::size_t>(static_cast<double>(elements) * common_prefix);
thrust::device_vector<T> dinput(elements, thrust::no_init);
thrust::fill(dinput.begin(), dinput.begin() + mismatch_point, T{0});
thrust::fill(dinput.begin() + mismatch_point, dinput.end(), val);
state.add_global_memory_reads<T>(mismatch_point + 1);
state.add_global_memory_writes<size_t>(1);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
do_not_optimize(
thrust::find_if(policy(alloc, launch), dinput.begin(), dinput.end(), cuda::equal_to_value<T>{val}));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_float64_axis("MismatchAt", std::vector{1.0, 0.5, 0.01});

41
cccl_upstream/thrust/benchmarks/bench/find_if_not/basic.cu Normal file
View File

@@ -0,0 +1,41 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/find.h>
#include <cuda/functional>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
T val = 1;
// set up input
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const auto common_prefix = state.get_float64("MismatchAt");
const auto mismatch_point = static_cast<std::size_t>(static_cast<double>(elements) * common_prefix);
thrust::device_vector<T> dinput(elements, thrust::no_init);
thrust::fill(dinput.begin(), dinput.begin() + mismatch_point, T{0});
thrust::fill(dinput.begin() + mismatch_point, dinput.end(), val);
state.add_global_memory_reads<T>(mismatch_point + 1);
state.add_global_memory_writes<size_t>(1);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
do_not_optimize(thrust::find_if_not(
policy(alloc, launch), dinput.begin(), dinput.end(), cuda::std::not_fn(cuda::equal_to_value{val})));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_float64_axis("MismatchAt", std::vector{1.0, 0.5, 0.01});

41
cccl_upstream/thrust/benchmarks/bench/for_each/basic.cu Normal file
View File

@@ -0,0 +1,41 @@
// SPDX-FileCopyrightText: Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/for_each.h>
#include "nvbench_helper.cuh"
template <class T>
struct square_t
{
__device__ void operator()(T& x) const
{
x = x * x;
}
};
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in(elements, T{1});
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
square_t<T> op{};
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::for_each(policy(alloc, launch), in.begin(), in.end(), op);
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

45
cccl_upstream/thrust/benchmarks/bench/for_each_n/basic.cu Normal file
View File

@@ -0,0 +1,45 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/for_each.h>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <class T>
struct square_t
{
__device__ void operator()(T& x) const
{
x = x * x;
}
};
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in(elements, T{1});
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
square_t<T> op{};
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::for_each_n(policy(alloc, launch), in.begin(), elements, op);
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

39
cccl_upstream/thrust/benchmarks/bench/generate/basic.cu Normal file
View File

@@ -0,0 +1,39 @@
// SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/generate.h>
#include "nvbench_helper.cuh"
template <typename T>
struct generator
{
_CCCL_DEVICE_API _CCCL_FORCEINLINE auto operator()() const -> T
{
return 42;
}
};
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> output(elements);
state.add_element_count(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::generate(policy(alloc, launch), output.begin(), output.end(), generator<T>{});
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

42
cccl_upstream/thrust/benchmarks/bench/generate_n/basic.cu Normal file
View File

@@ -0,0 +1,42 @@
// SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/generate.h>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <typename T>
struct generator
{
_CCCL_DEVICE_API _CCCL_FORCEINLINE auto operator()() const -> T
{
return 42;
}
};
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> output(elements, thrust::no_init);
state.add_element_count(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::generate_n(policy(alloc, launch), output.begin(), elements, generator<T>{});
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

33
cccl_upstream/thrust/benchmarks/bench/inner_product/basic.cu Normal file
View File

@@ -0,0 +1,33 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/inner_product.h>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
auto generator = generate(elements);
thrust::device_vector<T> lhs = generator;
thrust::device_vector<T> rhs = generator;
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements * 2);
state.add_global_memory_writes<T>(1);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::inner_product(policy(alloc, launch), lhs.begin(), lhs.end(), rhs.begin(), T{0});
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(all_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

42
cccl_upstream/thrust/benchmarks/bench/is_partitioned/basic.cu Normal file
View File

@@ -0,0 +1,42 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/partition.h>
#include <thrust/sequence.h>
#include <cuda/functional>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
using select_op_t = less_then_t<T>;
// set up input
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const auto common_prefix = state.get_float64("MismatchAt");
const auto mismatch_point = static_cast<std::size_t>(
::cuda::std::clamp(static_cast<double>(elements) * common_prefix, 0.0, static_cast<double>(elements - 1)));
thrust::device_vector<T> dinput(elements, thrust::no_init);
thrust::sequence(dinput.begin(), dinput.end(), T{0});
state.add_global_memory_reads<T>(2 * elements);
state.add_global_memory_writes<size_t>(1);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
do_not_optimize(thrust::is_partitioned(
policy(alloc, launch), dinput.begin(), dinput.end(), select_op_t{static_cast<T>(mismatch_point)}));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_float64_axis("MismatchAt", std::vector{1.0, 0.5, 0.01});

69
cccl_upstream/thrust/benchmarks/bench/is_sorted/basic.cu Normal file
View File

@@ -0,0 +1,69 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/sequence.h>
#include <thrust/sort.h>
#include <cuda/functional>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
// set up input
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const auto common_prefix = state.get_float64("MismatchAt");
const auto mismatch_point = static_cast<std::size_t>(
::cuda::std::clamp(static_cast<double>(elements) * common_prefix, 0.0, static_cast<double>(elements - 1)));
thrust::device_vector<T> dinput(elements, thrust::no_init);
thrust::sequence(dinput.begin(), dinput.end(), T{0});
dinput[mismatch_point] = T{-1};
state.add_global_memory_reads<T>(mismatch_point + 1);
state.add_global_memory_writes<size_t>(1);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
do_not_optimize(thrust::is_sorted(policy(alloc, launch), dinput.begin(), dinput.end()));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_float64_axis("MismatchAt", std::vector{1.0, 0.5, 0.01});
template <typename T>
static void with_predicate(nvbench::state& state, nvbench::type_list<T>)
{
// set up input
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const auto common_prefix = state.get_float64("MismatchAt");
const auto mismatch_point = static_cast<std::size_t>(
::cuda::std::clamp(static_cast<double>(elements) * common_prefix, 0.0, static_cast<double>(elements - 1)));
thrust::device_vector<T> dinput(elements, thrust::no_init);
thrust::sequence(dinput.begin(), dinput.end(), T{0});
dinput[mismatch_point] = T{-1};
state.add_global_memory_reads<T>(mismatch_point + 1);
state.add_global_memory_writes<size_t>(1);
caching_allocator_t alloc{};
state.exec(
nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
do_not_optimize(thrust::is_sorted(policy(alloc, launch), dinput.begin(), dinput.end(), cuda::std::less<>{}));
});
}
NVBENCH_BENCH_TYPES(with_predicate, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("with_predicate")
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_float64_axis("MismatchAt", std::vector{1.0, 0.5, 0.01});

70
cccl_upstream/thrust/benchmarks/bench/is_sorted_until/basic.cu Normal file
View File

@@ -0,0 +1,70 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/sequence.h>
#include <thrust/sort.h>
#include <cuda/functional>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
// set up input
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const auto common_prefix = state.get_float64("MismatchAt");
const auto mismatch_point = static_cast<std::size_t>(
::cuda::std::clamp(static_cast<double>(elements) * common_prefix, 0.0, static_cast<double>(elements - 1)));
thrust::device_vector<T> dinput(elements, thrust::no_init);
thrust::sequence(dinput.begin(), dinput.end(), T{0});
dinput[mismatch_point] = T{-1};
state.add_global_memory_reads<T>(mismatch_point + 1);
state.add_global_memory_writes<size_t>(1);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
do_not_optimize(thrust::is_sorted_until(policy(alloc, launch), dinput.begin(), dinput.end()));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_float64_axis("MismatchAt", std::vector{1.0, 0.5, 0.01});
template <typename T>
static void with_predicate(nvbench::state& state, nvbench::type_list<T>)
{
// set up input
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const auto common_prefix = state.get_float64("MismatchAt");
const auto mismatch_point = static_cast<std::size_t>(
::cuda::std::clamp(static_cast<double>(elements) * common_prefix, 0.0, static_cast<double>(elements - 1)));
thrust::device_vector<T> dinput(elements, thrust::no_init);
thrust::sequence(dinput.begin(), dinput.end(), T{0});
dinput[mismatch_point] = T{-1};
state.add_global_memory_reads<T>(mismatch_point + 1);
state.add_global_memory_writes<size_t>(1);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
do_not_optimize(
thrust::is_sorted_until(policy(alloc, launch), dinput.begin(), dinput.end(), cuda::std::less<>{}));
});
}
NVBENCH_BENCH_TYPES(with_predicate, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("with_predicate")
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_float64_axis("MismatchAt", std::vector{1.0, 0.5, 0.01});

46
cccl_upstream/thrust/benchmarks/bench/merge/basic.cu Normal file
View File

@@ -0,0 +1,46 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/merge.h>
#include <thrust/sort.h>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const auto size_ratio = static_cast<std::size_t>(state.get_int64("InputSizeRatio"));
const auto entropy = str_to_entropy(state.get_string("Entropy"));
const auto elements_in_lhs = static_cast<std::size_t>(static_cast<double>(size_ratio * elements) / 100.0);
thrust::device_vector<T> out(elements);
thrust::device_vector<T> in = generate(elements, entropy);
thrust::sort(in.begin(), in.begin() + elements_in_lhs);
thrust::sort(in.begin() + elements_in_lhs, in.end());
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::merge(
policy(alloc, launch),
in.cbegin(),
in.cbegin() + elements_in_lhs,
in.cbegin() + elements_in_lhs,
in.cend(),
out.begin());
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_string_axis("Entropy", {"1.000", "0.201"})
.add_int64_axis("InputSizeRatio", {25, 50, 75});

42
cccl_upstream/thrust/benchmarks/bench/mismatch/basic.cu Normal file
View File

@@ -0,0 +1,42 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/fill.h>
#include <thrust/mismatch.h>
#include <cuda/iterator>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <typename T>
static void range_iter(nvbench::state& state, nvbench::type_list<T>)
{
T val = 1;
// set up input
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const auto common_prefix = state.get_float64("MismatchAt");
const auto mismatch_point = static_cast<std::size_t>(static_cast<double>(elements) * common_prefix);
thrust::device_vector<T> dinput(elements, thrust::no_init);
thrust::fill(dinput.begin(), dinput.begin() + mismatch_point, T{0});
thrust::fill(dinput.begin() + mismatch_point, dinput.end(), val);
state.add_global_memory_reads<T>(mismatch_point + 1);
state.add_global_memory_writes<size_t>(1);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
do_not_optimize(
thrust::mismatch(policy(alloc, launch), dinput.begin(), dinput.end(), cuda::constant_iterator<T>{0}));
});
}
NVBENCH_BENCH_TYPES(range_iter, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base_range_iter")
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_float64_axis("MismatchAt", std::vector{1.0, 0.5, 0.01});

41
cccl_upstream/thrust/benchmarks/bench/none_of/basic.cu Normal file
View File

@@ -0,0 +1,41 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/fill.h>
#include <thrust/logical.h>
#include <cuda/functional>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
T val = 1;
// set up input
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const auto common_prefix = state.get_float64("MismatchAt");
const auto mismatch_point = static_cast<std::size_t>(static_cast<double>(elements) * common_prefix);
thrust::device_vector<T> dinput(elements, thrust::no_init);
thrust::fill(dinput.begin(), dinput.begin() + mismatch_point, T{0});
thrust::fill(dinput.begin() + mismatch_point, dinput.end(), val);
state.add_global_memory_reads<T>(mismatch_point + 1);
state.add_global_memory_writes<size_t>(1);
caching_allocator_t alloc{};
state.exec(
nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
do_not_optimize(thrust::none_of(policy(alloc, launch), dinput.begin(), dinput.end(), cuda::equal_to_value{val}));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_float64_axis("MismatchAt", std::vector{1.0, 0.5, 0.01});

45
cccl_upstream/thrust/benchmarks/bench/partition/basic.cu Normal file
View File

@@ -0,0 +1,45 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/partition.h>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
using select_op_t = less_then_t<T>;
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const bit_entropy entropy = str_to_entropy(state.get_string("Entropy"));
const T val = lerp_min_max<T>(entropy_to_probability(entropy));
select_op_t select_op{val};
thrust::device_vector<T> input = generate(elements);
thrust::device_vector<T> output(elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::partition_copy(
policy(alloc, launch),
input.cbegin(),
input.cend(),
output.begin(),
cuda::std::make_reverse_iterator(output.begin() + elements),
select_op);
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_string_axis("Entropy", {"1.000", "0.544", "0.000"});

30
cccl_upstream/thrust/benchmarks/bench/reduce/basic.cu Normal file
View File

@@ -0,0 +1,30 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/reduce.h>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in = generate(elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(1);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
do_not_optimize(thrust::reduce(policy(alloc, launch), in.begin(), in.end()));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

60
cccl_upstream/thrust/benchmarks/bench/reduce/by_key.cu Normal file
View File

@@ -0,0 +1,60 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/reduce.h>
#include <thrust/unique.h>
#include "nvbench_helper.cuh"
template <class KeyT, class ValueT>
static void basic(nvbench::state& state, nvbench::type_list<KeyT, ValueT>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
constexpr std::size_t min_segment_size = 1;
const std::size_t max_segment_size = static_cast<std::size_t>(state.get_int64("MaxSegSize"));
thrust::device_vector<KeyT> in_keys = generate.uniform.key_segments(elements, min_segment_size, max_segment_size);
thrust::device_vector<KeyT> out_keys = in_keys;
thrust::device_vector<ValueT> in_vals(elements);
const std::size_t unique_keys =
::cuda::std::distance(out_keys.begin(), thrust::unique(out_keys.begin(), out_keys.end()));
thrust::device_vector<ValueT> out_vals(unique_keys);
state.add_element_count(elements);
state.add_global_memory_reads<KeyT>(elements);
state.add_global_memory_reads<ValueT>(elements);
state.add_global_memory_writes<KeyT>(unique_keys);
state.add_global_memory_writes<ValueT>(unique_keys);
caching_allocator_t alloc;
state.exec(
nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
thrust::reduce_by_key(
policy(alloc, launch), in_keys.begin(), in_keys.end(), in_vals.begin(), out_keys.begin(), out_vals.begin());
});
}
using key_types =
nvbench::type_list<int8_t,
int16_t,
int32_t,
int64_t
#if _CCCL_HAS_INT128()
,
int128_t
#endif
>;
using value_types = all_types;
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(key_types, value_types))
.set_name("base")
.set_type_axes_names({"KeyT{ct}", "ValueT{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_int64_power_of_two_axis("MaxSegSize", {1, 4, 8});

37
cccl_upstream/thrust/benchmarks/bench/remove/basic.cu Normal file
View File

@@ -0,0 +1,37 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/count.h>
#include <thrust/device_vector.h>
#include <thrust/remove.h>
#include <cuda/memory_pool>
#include <cuda/std/complex>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in = generate(elements, bit_entropy::_1_000, T{0}, T{42});
const auto count = thrust::count(thrust::device, in.begin(), in.end(), T{42});
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements - count);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::remove(policy(alloc, launch), in.begin(), in.end(), T{42});
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

38
cccl_upstream/thrust/benchmarks/bench/remove_copy/basic.cu Normal file
View File

@@ -0,0 +1,38 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/count.h>
#include <thrust/device_vector.h>
#include <thrust/remove.h>
#include <cuda/memory_pool>
#include <cuda/std/complex>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in = generate(elements, bit_entropy::_1_000, T{0}, T{42});
thrust::device_vector<T> out(elements, thrust::no_init);
const auto count = thrust::count(thrust::device, in.begin(), in.end(), T{42});
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements - count);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
do_not_optimize(thrust::remove_copy(policy(alloc, launch), in.begin(), in.end(), out.begin(), T{42}));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

44
cccl_upstream/thrust/benchmarks/bench/remove_copy_if/basic.cu Normal file
View File

@@ -0,0 +1,44 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/remove.h>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
struct is_even
{
template <class T>
__device__ constexpr bool operator()(const T& val) const noexcept
{
return static_cast<int>(val) % 2 == 0;
}
};
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in = generate(elements, bit_entropy::_1_000, T{0}, T{42});
thrust::device_vector<T> out(elements, thrust::no_init);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements / 2);
caching_allocator_t alloc{};
state.exec(
nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
do_not_optimize(thrust::remove_copy_if(policy(alloc, launch), in.begin(), in.end(), out.begin(), is_even{}));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

43
cccl_upstream/thrust/benchmarks/bench/remove_if/basic.cu Normal file
View File

@@ -0,0 +1,43 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/remove.h>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
struct is_even
{
template <class T>
__device__ constexpr bool operator()(const T& val) const noexcept
{
return static_cast<int>(val) % 2 == 0;
}
};
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in = generate(elements, bit_entropy::_1_000, T{0}, T{42});
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements / 2);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::remove_if(policy(alloc, launch), in.begin(), in.end(), is_even{});
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

34
cccl_upstream/thrust/benchmarks/bench/replace/basic.cu Normal file
View File

@@ -0,0 +1,34 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/replace.h>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in = generate(elements, bit_entropy::_1_000, T{0}, T{42});
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::replace(policy(alloc, launch), in.begin(), in.end(), 42, 1337);
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

35
cccl_upstream/thrust/benchmarks/bench/replace_copy/basic.cu Normal file
View File

@@ -0,0 +1,35 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/replace.h>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in = generate(elements, bit_entropy::_1_000, T{0}, T{42});
thrust::device_vector<T> out(elements, thrust::no_init);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc{};
state.exec(
nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
do_not_optimize(thrust::replace_copy(policy(alloc, launch), in.begin(), in.end(), out.begin(), 42, 1337));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

45
cccl_upstream/thrust/benchmarks/bench/replace_copy_if/basic.cu Normal file
View File

@@ -0,0 +1,45 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/replace.h>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
struct equal_to_42
{
template <class T>
__device__ constexpr bool operator()(const T& val) const noexcept
{
return val == static_cast<T>(42);
}
};
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in = generate(elements, bit_entropy::_1_000, T{0}, T{42});
thrust::device_vector<T> out(elements, thrust::no_init);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc{};
state.exec(
nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
do_not_optimize(
thrust::replace_copy_if(policy(alloc, launch), in.begin(), in.end(), out.begin(), equal_to_42{}, 1337));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

43
cccl_upstream/thrust/benchmarks/bench/replace_if/basic.cu Normal file
View File

@@ -0,0 +1,43 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/replace.h>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
struct equal_to_42
{
template <class T>
__device__ constexpr bool operator()(const T& val) const noexcept
{
return val == static_cast<T>(42);
}
};
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in = generate(elements, bit_entropy::_1_000, T{0}, T{42});
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::replace_if(policy(alloc, launch), in.begin(), in.end(), equal_to_42{}, 1337);
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

47
cccl_upstream/thrust/benchmarks/bench/scan/exclusive/by_key.cu Normal file
View File

@@ -0,0 +1,47 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/scan.h>
#include "nvbench_helper.cuh"
template <class KeyT, class ValueT>
static void scan(nvbench::state& state, nvbench::type_list<KeyT, ValueT>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<ValueT> in_vals(elements);
thrust::device_vector<ValueT> out_vals(elements);
thrust::device_vector<KeyT> keys = generate.uniform.key_segments(elements, 0, 5200);
state.add_element_count(elements);
state.add_global_memory_reads<KeyT>(elements);
state.add_global_memory_reads<ValueT>(elements);
state.add_global_memory_writes<ValueT>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::exclusive_scan_by_key(
policy(alloc, launch), keys.cbegin(), keys.cend(), in_vals.cbegin(), out_vals.begin());
});
}
using key_types = all_types;
using value_types =
nvbench::type_list<int8_t,
int16_t,
int32_t,
int64_t
#if _CCCL_HAS_INT128()
,
int128_t
#endif
>;
NVBENCH_BENCH_TYPES(scan, NVBENCH_TYPE_AXES(key_types, value_types))
.set_name("base")
.set_type_axes_names({"KeyT{ct}", "ValueT{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

32
cccl_upstream/thrust/benchmarks/bench/scan/exclusive/max.cu Normal file
View File

@@ -0,0 +1,32 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/scan.h>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> input = generate(elements);
thrust::device_vector<T> output(elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc;
state.exec(
nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
thrust::exclusive_scan(policy(alloc, launch), input.cbegin(), input.cend(), output.begin(), T{}, max_t{});
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(all_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

32
cccl_upstream/thrust/benchmarks/bench/scan/exclusive/sum.cu Normal file
View File

@@ -0,0 +1,32 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/scan.h>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> input = generate(elements);
thrust::device_vector<T> output(elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::exclusive_scan(policy(alloc, launch), input.cbegin(), input.cend(), output.begin());
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(all_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

47
cccl_upstream/thrust/benchmarks/bench/scan/inclusive/by_key.cu Normal file
View File

@@ -0,0 +1,47 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/scan.h>
#include "nvbench_helper.cuh"
template <class KeyT, class ValueT>
static void scan(nvbench::state& state, nvbench::type_list<KeyT, ValueT>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<ValueT> in_vals(elements);
thrust::device_vector<ValueT> out_vals(elements);
thrust::device_vector<KeyT> keys = generate.uniform.key_segments(elements, 0, 5200);
state.add_element_count(elements);
state.add_global_memory_reads<KeyT>(elements);
state.add_global_memory_reads<ValueT>(elements);
state.add_global_memory_writes<ValueT>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::inclusive_scan_by_key(
policy(alloc, launch), keys.cbegin(), keys.cend(), in_vals.cbegin(), out_vals.begin());
});
}
using key_types = all_types;
using value_types =
nvbench::type_list<int8_t,
int16_t,
int32_t,
int64_t
#if _CCCL_HAS_INT128()
,
int128_t
#endif
>;
NVBENCH_BENCH_TYPES(scan, NVBENCH_TYPE_AXES(key_types, value_types))
.set_name("base")
.set_type_axes_names({"KeyT{ct}", "ValueT{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

32
cccl_upstream/thrust/benchmarks/bench/scan/inclusive/max.cu Normal file
View File

@@ -0,0 +1,32 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/scan.h>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> input = generate(elements);
thrust::device_vector<T> output(elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::inclusive_scan(policy(alloc, launch), input.cbegin(), input.cend(), output.begin(), max_t{});
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(all_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

32
cccl_upstream/thrust/benchmarks/bench/scan/inclusive/sum.cu Normal file
View File

@@ -0,0 +1,32 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/scan.h>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> input = generate(elements);
thrust::device_vector<T> output(elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::inclusive_scan(policy(alloc, launch), input.cbegin(), input.cend(), output.begin());
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(all_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

54
cccl_upstream/thrust/benchmarks/bench/set_operations/base.cuh Normal file
View File

@@ -0,0 +1,54 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#pragma once
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/set_operations.h>
#include <thrust/sort.h>
#include "nvbench_helper.cuh"
template <typename T, typename OpT>
static void basic(nvbench::state& state, nvbench::type_list<T>, OpT op)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const auto size_ratio = static_cast<std::size_t>(state.get_int64("SizeRatio"));
const bit_entropy entropy = str_to_entropy(state.get_string("Entropy"));
const auto elements_in_A = static_cast<std::size_t>(static_cast<double>(size_ratio * elements) / 100.0f);
thrust::device_vector<T> input = generate(elements, entropy);
thrust::device_vector<T> output(elements);
thrust::sort(input.begin(), input.begin() + elements_in_A);
thrust::sort(input.begin() + elements_in_A, input.end());
caching_allocator_t alloc;
// not a warm-up run, we need to run once to determine the size of the output
const auto result_ends =
op(policy(alloc),
input.cbegin(),
input.cbegin() + elements_in_A,
input.cbegin() + elements_in_A,
input.cend(),
output.begin());
const std::size_t elements_in_AB = ::cuda::std::distance(output.begin(), result_ends);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements_in_AB);
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
op(policy(alloc, launch),
input.cbegin(),
input.cbegin() + elements_in_A,
input.cbegin() + elements_in_A,
input.cend(),
output.begin());
});
}
using types = nvbench::type_list<int8_t, int16_t, int32_t, int64_t>;

66
cccl_upstream/thrust/benchmarks/bench/set_operations/by_key.cuh Normal file
View File

@@ -0,0 +1,66 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#pragma once
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/set_operations.h>
#include <thrust/sort.h>
#include "nvbench_helper.cuh"
template <class KeyT, class ValueT, class OpT>
static void basic(nvbench::state& state, nvbench::type_list<KeyT, ValueT>, OpT op)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const auto size_ratio = static_cast<std::size_t>(state.get_int64("SizeRatio"));
const bit_entropy entropy = str_to_entropy(state.get_string("Entropy"));
const auto elements_in_A = static_cast<std::size_t>(static_cast<double>(size_ratio * elements) / 100.0f);
thrust::device_vector<KeyT> in_keys = generate(elements, entropy);
thrust::device_vector<KeyT> out_keys(elements);
thrust::device_vector<ValueT> in_vals(elements);
thrust::device_vector<ValueT> out_vals(elements);
thrust::sort(in_keys.begin(), in_keys.begin() + elements_in_A);
thrust::sort(in_keys.begin() + elements_in_A, in_keys.end());
caching_allocator_t alloc;
// not a warm-up run, we need to run once to determine the size of the output
auto result_ends = op(
policy(alloc),
in_keys.cbegin(),
in_keys.cbegin() + elements_in_A,
in_keys.cbegin() + elements_in_A,
in_keys.cend(),
in_vals.cbegin(),
in_vals.cbegin() + elements_in_A,
out_keys.begin(),
out_vals.begin());
const std::size_t elements_in_AB = ::cuda::std::distance(out_keys.begin(), result_ends.first);
state.add_element_count(elements);
state.add_global_memory_reads<KeyT>(elements);
state.add_global_memory_writes<KeyT>(elements_in_AB);
state.add_global_memory_reads<ValueT>(OpT::read_all_values ? elements : elements_in_A);
state.add_global_memory_writes<ValueT>(elements_in_AB);
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
op(policy(alloc, launch),
in_keys.cbegin(),
in_keys.cbegin() + elements_in_A,
in_keys.cbegin() + elements_in_A,
in_keys.cend(),
in_vals.cbegin(),
in_vals.cbegin() + elements_in_A,
out_keys.begin(),
out_vals.begin());
});
}
using key_types = nvbench::type_list<int8_t, int16_t, int32_t, int64_t>;
using value_types = nvbench::type_list<int8_t, int64_t>;

32
cccl_upstream/thrust/benchmarks/bench/set_operations/difference.cu Normal file
View File

@@ -0,0 +1,32 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include "base.cuh"
struct op_t
{
template <class PolicyT, class InputIterator1, class InputIterator2, class OutputIterator>
__host__ OutputIterator operator()(
const PolicyT& policy,
InputIterator1 first1,
InputIterator1 last1,
InputIterator2 first2,
InputIterator2 last2,
OutputIterator result) const
{
return thrust::set_difference(policy, first1, last1, first2, last2, result);
}
};
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T> tl)
{
basic(state, tl, op_t{});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_string_axis("Entropy", {"1.000", "0.201"})
.add_int64_axis("SizeRatio", {25, 50, 75});

44
cccl_upstream/thrust/benchmarks/bench/set_operations/difference_by_key.cu Normal file
View File

@@ -0,0 +1,44 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include "by_key.cuh"
struct op_t
{
static constexpr bool read_all_values = true;
template <class PolicyT,
class InputIterator1,
class InputIterator2,
class InputIterator3,
class InputIterator4,
class OutputIterator1,
class OutputIterator2>
__host__ cuda::std::pair<OutputIterator1, OutputIterator2> operator()(
const PolicyT& policy,
InputIterator1 keys_first1,
InputIterator1 keys_last1,
InputIterator2 keys_first2,
InputIterator2 keys_last2,
InputIterator3 values_first1,
InputIterator4 values_first2,
OutputIterator1 keys_result,
OutputIterator2 values_result) const
{
return thrust::set_difference_by_key(
policy, keys_first1, keys_last1, keys_first2, keys_last2, values_first1, values_first2, keys_result, values_result);
}
};
template <class KeyT, class ValueT>
static void basic(nvbench::state& state, nvbench::type_list<KeyT, ValueT> tl)
{
basic(state, tl, op_t{});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(key_types, value_types))
.set_name("base")
.set_type_axes_names({"KeyT{ct}", "ValueT{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_string_axis("Entropy", {"1.000", "0.201"})
.add_int64_axis("SizeRatio", {25, 50, 75});

32
cccl_upstream/thrust/benchmarks/bench/set_operations/intersection.cu Normal file
View File

@@ -0,0 +1,32 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include "base.cuh"
struct op_t
{
template <class PolicyT, class InputIterator1, class InputIterator2, class OutputIterator>
__host__ OutputIterator operator()(
const PolicyT& policy,
InputIterator1 first1,
InputIterator1 last1,
InputIterator2 first2,
InputIterator2 last2,
OutputIterator result) const
{
return thrust::set_intersection(policy, first1, last1, first2, last2, result);
}
};
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T> tl)
{
basic(state, tl, op_t{});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_string_axis("Entropy", {"1.000", "0.201"})
.add_int64_axis("SizeRatio", {25, 50, 75});

44
cccl_upstream/thrust/benchmarks/bench/set_operations/intersection_by_key.cu Normal file
View File

@@ -0,0 +1,44 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include "by_key.cuh"
struct op_t
{
static constexpr bool read_all_values = false;
template <class PolicyT,
class InputIterator1,
class InputIterator2,
class InputIterator3,
class InputIterator4,
class OutputIterator1,
class OutputIterator2>
__host__ cuda::std::pair<OutputIterator1, OutputIterator2> operator()(
const PolicyT& policy,
InputIterator1 keys_first1,
InputIterator1 keys_last1,
InputIterator2 keys_first2,
InputIterator2 keys_last2,
InputIterator3 values_first1,
InputIterator4 /* values_first2 */,
OutputIterator1 keys_result,
OutputIterator2 values_result) const
{
return thrust::set_intersection_by_key(
policy, keys_first1, keys_last1, keys_first2, keys_last2, values_first1, keys_result, values_result);
}
};
template <class KeyT, class ValueT>
static void basic(nvbench::state& state, nvbench::type_list<KeyT, ValueT> tl)
{
basic(state, tl, op_t{});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(key_types, value_types))
.set_name("base")
.set_type_axes_names({"KeyT{ct}", "ValueT{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_string_axis("Entropy", {"1.000", "0.201"})
.add_int64_axis("SizeRatio", {25, 50, 75});

32
cccl_upstream/thrust/benchmarks/bench/set_operations/symmetric_difference.cu Normal file
View File

@@ -0,0 +1,32 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include "base.cuh"
struct op_t
{
template <class PolicyT, class InputIterator1, class InputIterator2, class OutputIterator>
__host__ OutputIterator operator()(
const PolicyT& policy,
InputIterator1 first1,
InputIterator1 last1,
InputIterator2 first2,
InputIterator2 last2,
OutputIterator result) const
{
return thrust::set_symmetric_difference(policy, first1, last1, first2, last2, result);
}
};
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T> tl)
{
basic(state, tl, op_t{});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_string_axis("Entropy", {"1.000", "0.201"})
.add_int64_axis("SizeRatio", {25, 50, 75});

44
cccl_upstream/thrust/benchmarks/bench/set_operations/symmetric_difference_by_key.cu Normal file
View File

@@ -0,0 +1,44 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include "by_key.cuh"
struct op_t
{
static constexpr bool read_all_values = true;
template <class PolicyT,
class InputIterator1,
class InputIterator2,
class InputIterator3,
class InputIterator4,
class OutputIterator1,
class OutputIterator2>
__host__ cuda::std::pair<OutputIterator1, OutputIterator2> operator()(
const PolicyT& policy,
InputIterator1 keys_first1,
InputIterator1 keys_last1,
InputIterator2 keys_first2,
InputIterator2 keys_last2,
InputIterator3 values_first1,
InputIterator4 values_first2,
OutputIterator1 keys_result,
OutputIterator2 values_result) const
{
return thrust::set_symmetric_difference_by_key(
policy, keys_first1, keys_last1, keys_first2, keys_last2, values_first1, values_first2, keys_result, values_result);
}
};
template <class KeyT, class ValueT>
static void basic(nvbench::state& state, nvbench::type_list<KeyT, ValueT> tl)
{
basic(state, tl, op_t{});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(key_types, value_types))
.set_name("base")
.set_type_axes_names({"KeyT{ct}", "ValueT{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_string_axis("Entropy", {"1.000", "0.201"})
.add_int64_axis("SizeRatio", {25, 50, 75});

32
cccl_upstream/thrust/benchmarks/bench/set_operations/union.cu Normal file
View File

@@ -0,0 +1,32 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include "base.cuh"
struct op_t
{
template <class PolicyT, class InputIterator1, class InputIterator2, class OutputIterator>
__host__ OutputIterator operator()(
const PolicyT& policy,
InputIterator1 first1,
InputIterator1 last1,
InputIterator2 first2,
InputIterator2 last2,
OutputIterator result) const
{
return thrust::set_union(policy, first1, last1, first2, last2, result);
}
};
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T> tl)
{
basic(state, tl, op_t{});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_string_axis("Entropy", {"1.000", "0.201"})
.add_int64_axis("SizeRatio", {25, 50, 75});

44
cccl_upstream/thrust/benchmarks/bench/set_operations/union_by_key.cu Normal file
View File

@@ -0,0 +1,44 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include "by_key.cuh"
struct op_t
{
static constexpr bool read_all_values = true;
template <class PolicyT,
class InputIterator1,
class InputIterator2,
class InputIterator3,
class InputIterator4,
class OutputIterator1,
class OutputIterator2>
__host__ cuda::std::pair<OutputIterator1, OutputIterator2> operator()(
const PolicyT& policy,
InputIterator1 keys_first1,
InputIterator1 keys_last1,
InputIterator2 keys_first2,
InputIterator2 keys_last2,
InputIterator3 values_first1,
InputIterator4 values_first2,
OutputIterator1 keys_result,
OutputIterator2 values_result) const
{
return thrust::set_union_by_key(
policy, keys_first1, keys_last1, keys_first2, keys_last2, values_first1, values_first2, keys_result, values_result);
}
};
template <class KeyT, class ValueT>
static void basic(nvbench::state& state, nvbench::type_list<KeyT, ValueT> tl)
{
basic(state, tl, op_t{});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(key_types, value_types))
.set_name("base")
.set_type_axes_names({"KeyT{ct}", "ValueT{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_string_axis("Entropy", {"1.000", "0.201"})
.add_int64_axis("SizeRatio", {25, 50, 75});

72
cccl_upstream/thrust/benchmarks/bench/shuffle/basic.cu Normal file
View File

@@ -0,0 +1,72 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/random.h>
#include <thrust/shuffle.h>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> data(elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
auto do_engine = [&](auto&& engine_constructor) {
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::shuffle(policy(alloc, launch), data.begin(), data.end(), engine_constructor());
});
};
const auto rng_engine = state.get_string("Engine");
if (rng_engine == "minstd")
{
do_engine([] {
return thrust::random::minstd_rand{};
});
}
else if (rng_engine == "ranlux24")
{
do_engine([] {
return thrust::random::ranlux24{};
});
}
else if (rng_engine == "ranlux48")
{
do_engine([] {
return thrust::random::ranlux48{};
});
}
else if (rng_engine == "taus88")
{
do_engine([] {
return thrust::random::taus88{};
});
}
}
using types =
nvbench::type_list<int8_t,
int16_t,
int32_t,
int64_t
#if _CCCL_HAS_INT128()
,
int128_t
#endif
>;
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_string_axis("Engine", {"minstd", "ranlux24", "ranlux48", "taus88"});

38
cccl_upstream/thrust/benchmarks/bench/sort/keys.cu Normal file
View File

@@ -0,0 +1,38 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/sort.h>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const bit_entropy entropy = str_to_entropy(state.get_string("Entropy"));
thrust::device_vector<T> input = generate(elements, entropy);
thrust::device_vector<T> vec(elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::timer | nvbench::exec_tag::sync,
[&](nvbench::launch& launch, auto& timer) {
vec = input;
timer.start();
thrust::sort(policy(alloc, launch), vec.begin(), vec.end());
timer.stop();
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_string_axis("Entropy", {"1.000", "0.201"});

38
cccl_upstream/thrust/benchmarks/bench/sort/keys_custom.cu Normal file
View File

@@ -0,0 +1,38 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/sort.h>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const bit_entropy entropy = str_to_entropy(state.get_string("Entropy"));
thrust::device_vector<T> input = generate(elements, entropy);
thrust::device_vector<T> vec(elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::timer | nvbench::exec_tag::sync,
[&](nvbench::launch& launch, auto& timer) {
vec = input;
timer.start();
thrust::sort(policy(alloc, launch), vec.begin(), vec.end(), less_t{});
timer.stop();
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_string_axis("Entropy", {"1.000", "0.201"});

45
cccl_upstream/thrust/benchmarks/bench/sort/pairs.cu Normal file
View File

@@ -0,0 +1,45 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/sort.h>
#include "nvbench_helper.cuh"
template <class KeyT, class ValueT>
static void basic(nvbench::state& state, nvbench::type_list<KeyT, ValueT>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const bit_entropy entropy = str_to_entropy(state.get_string("Entropy"));
thrust::device_vector<KeyT> in_keys = generate(elements, entropy);
thrust::device_vector<KeyT> keys(elements);
thrust::device_vector<ValueT> in_vals = generate(elements);
thrust::device_vector<ValueT> vals(elements);
state.add_element_count(elements);
state.add_global_memory_reads<KeyT>(elements);
state.add_global_memory_reads<ValueT>(elements);
state.add_global_memory_writes<KeyT>(elements);
state.add_global_memory_writes<ValueT>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::timer | nvbench::exec_tag::sync,
[&](nvbench::launch& launch, auto& timer) {
keys = in_keys;
vals = in_vals;
timer.start();
thrust::sort_by_key(policy(alloc, launch), keys.begin(), keys.end(), vals.begin());
timer.stop();
});
}
using key_types = integral_types;
using value_types = integral_types;
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(key_types, value_types))
.set_name("base")
.set_type_axes_names({"KeyT{ct}", "ValueT{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_string_axis("Entropy", {"1.000", "0.201"});

46
cccl_upstream/thrust/benchmarks/bench/sort/pairs_custom.cu Normal file
View File

@@ -0,0 +1,46 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/sort.h>
#include "nvbench_helper.cuh"
template <class KeyT, class ValueT>
static void basic(nvbench::state& state, nvbench::type_list<KeyT, ValueT>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const bit_entropy entropy = str_to_entropy(state.get_string("Entropy"));
thrust::device_vector<KeyT> in_keys = generate(elements, entropy);
thrust::device_vector<KeyT> keys(elements);
thrust::device_vector<ValueT> in_vals = generate(elements);
thrust::device_vector<ValueT> vals(elements);
state.add_element_count(elements);
state.add_global_memory_reads<KeyT>(elements);
state.add_global_memory_reads<ValueT>(elements);
state.add_global_memory_writes<KeyT>(elements);
state.add_global_memory_writes<ValueT>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::timer | nvbench::exec_tag::sync,
[&](nvbench::launch& launch, auto& timer) {
keys = in_keys;
vals = in_vals;
timer.start();
thrust::sort_by_key(policy(alloc, launch), keys.begin(), keys.end(), vals.begin(), less_t{});
timer.stop();
});
}
using key_types = integral_types;
using value_types = integral_types;
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(key_types, value_types))
.set_name("base")
.set_type_axes_names({"KeyT{ct}", "ValueT{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_string_axis("Entropy", {"1.000", "0.201"});

30
cccl_upstream/thrust/benchmarks/bench/swap_ranges/basic.cu Normal file
View File

@@ -0,0 +1,30 @@
// SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: BSD-3-Clause
#include <thrust/device_vector.h>
#include <thrust/swap.h>
#include <nvbench_helper.cuh>
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> a = generate(elements);
thrust::device_vector<T> b = generate(elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(2 * elements);
state.add_global_memory_writes<T>(2 * elements);
caching_allocator_t alloc; // swap_ranges shouldn't allocate, but let's be consistent
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::swap_ranges(policy(alloc, launch), a.begin(), a.end(), b.begin());
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(integral_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

71
cccl_upstream/thrust/benchmarks/bench/tabulate/basic.cu Normal file
View File

@@ -0,0 +1,71 @@
// SPDX-FileCopyrightText: Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/sequence.h>
#include <thrust/tabulate.h>
#include <nvbench_helper.cuh>
#include "thrust/detail/raw_pointer_cast.h"
template <typename T>
static void sequence(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> output(elements, thrust::no_init);
state.add_element_count(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
// sequence is implemented via thrust::tabulate
thrust::sequence(policy(alloc, launch), output.begin(), output.end());
});
}
NVBENCH_BENCH_TYPES(sequence, NVBENCH_TYPE_AXES(integral_types))
.set_name("sequence")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));
template <class T>
struct seg_size_t
{
T* d_offsets{};
template <class OffsetT>
__device__ T operator()(OffsetT i)
{
return static_cast<T>(d_offsets[i + 1] - d_offsets[i]);
}
};
template <typename T>
static void seg_size(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> input(elements + 1);
thrust::device_vector<T> output(elements, thrust::no_init);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements + 1);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc;
seg_size_t<T> op{thrust::raw_pointer_cast(input.data())};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::tabulate(policy(alloc, launch), output.begin(), output.end(), op);
});
}
NVBENCH_BENCH_TYPES(seg_size, NVBENCH_TYPE_AXES(integral_types))
.set_name("seg_size")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

173
cccl_upstream/thrust/benchmarks/bench/transform/babelstream.cu Normal file
View File

@@ -0,0 +1,173 @@
// SPDX-FileCopyrightText: Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/iterator/zip_iterator.h>
#include <thrust/transform.h>
#include <thrust/zip_function.h>
#include <cuda/__functional/address_stability.h>
#include <nvbench_helper.cuh>
// The benchmarks are inspired by the BabelStream thrust version:
// https://github.com/UoB-HPC/BabelStream/blob/main/src/thrust/ThrustStream.cu
// Modified from BabelStream to also work for integers
constexpr auto startA = 1; // BabelStream: 0.1
constexpr auto startB = 2; // BabelStream: 0.2
constexpr auto startC = 3; // BabelStream: 0.1
constexpr auto startScalar = 4; // BabelStream: 0.4
using element_types = nvbench::type_list<std::int8_t, std::int16_t, float, double, __int128>;
// Different benchmarks use a different number of buffers. H200/B200 can fit 2^31 elements for all benchmarks and types.
// Upstream BabelStream uses 2^25. Allocation failure just skips the benchmark
auto array_size_powers = std::vector<std::int64_t>{25, 31};
template <typename... Args>
void bench_transform(nvbench::state& state, Args&&... args)
{
caching_allocator_t alloc; // transform shouldn't allocate, but let's be consistent
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::transform(policy(alloc, launch), ::cuda::std::forward<Args>(args)...);
});
}
template <typename T>
static void mul(nvbench::state& state, nvbench::type_list<T>)
{
const auto n = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> b(n, startB);
thrust::device_vector<T> c(n, startC);
state.add_element_count(n);
state.add_global_memory_reads<T>(n);
state.add_global_memory_writes<T>(n);
const T scalar = startScalar;
bench_transform(
state, c.begin(), c.end(), b.begin(), cuda::proclaim_copyable_arguments([=] _CCCL_DEVICE(const T& ci) {
return ci * scalar;
}));
}
NVBENCH_BENCH_TYPES(mul, NVBENCH_TYPE_AXES(element_types))
.set_name("mul")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", array_size_powers);
template <typename T>
static void add(nvbench::state& state, nvbench::type_list<T>)
{
const auto n = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> a(n, startA);
thrust::device_vector<T> b(n, startB);
thrust::device_vector<T> c(n, startC);
state.add_element_count(n);
state.add_global_memory_reads<T>(2 * n);
state.add_global_memory_writes<T>(n);
bench_transform(
state,
a.begin(),
a.end(),
b.begin(),
c.begin(),
cuda::proclaim_copyable_arguments([] _CCCL_DEVICE(const T& ai, const T& bi) -> T {
return ai + bi;
}));
}
NVBENCH_BENCH_TYPES(add, NVBENCH_TYPE_AXES(element_types))
.set_name("add")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", array_size_powers);
template <typename T>
static void triad(nvbench::state& state, nvbench::type_list<T>)
{
const auto n = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> a(n, startA);
thrust::device_vector<T> b(n, startB);
thrust::device_vector<T> c(n, startC);
state.add_element_count(n);
state.add_global_memory_reads<T>(2 * n);
state.add_global_memory_writes<T>(n);
const T scalar = startScalar;
bench_transform(
state,
b.begin(),
b.end(),
c.begin(),
a.begin(),
cuda::proclaim_copyable_arguments([=] _CCCL_DEVICE(const T& bi, const T& ci) {
return bi + scalar * ci;
}));
}
NVBENCH_BENCH_TYPES(triad, NVBENCH_TYPE_AXES(element_types))
.set_name("triad")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", array_size_powers);
template <typename T>
static void nstream(nvbench::state& state, nvbench::type_list<T>)
{
const auto n = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> a(n, startA);
thrust::device_vector<T> b(n, startB);
thrust::device_vector<T> c(n, startC);
state.add_element_count(n);
state.add_global_memory_reads<T>(3 * n);
state.add_global_memory_writes<T>(n);
const T scalar = startScalar;
bench_transform(
state,
thrust::make_zip_iterator(a.begin(), b.begin(), c.begin()),
thrust::make_zip_iterator(a.end(), b.end(), c.end()),
a.begin(),
thrust::make_zip_function(cuda::proclaim_copyable_arguments([=] _CCCL_DEVICE(const T& ai, const T& bi, const T& ci) {
return ai + bi + scalar * ci;
})));
}
NVBENCH_BENCH_TYPES(nstream, NVBENCH_TYPE_AXES(element_types))
.set_name("nstream")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", array_size_powers);
// variation of nstream requiring a stable parameter address because it recovers the element index
template <typename T>
static void nstream_stable(nvbench::state& state, nvbench::type_list<T>)
{
const auto n = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> a(n, startA);
thrust::device_vector<T> b(n, startB);
thrust::device_vector<T> c(n, startC);
const T* a_start = thrust::raw_pointer_cast(a.data());
const T* b_start = thrust::raw_pointer_cast(b.data());
const T* c_start = thrust::raw_pointer_cast(c.data());
state.add_element_count(n);
state.add_global_memory_reads<T>(3 * n);
state.add_global_memory_writes<T>(n);
const T scalar = startScalar;
bench_transform(state, a.begin(), a.end(), a.begin(), [=] _CCCL_DEVICE(const T& ai) {
const auto i = &ai - a_start;
return ai + b_start[i] + scalar * c_start[i];
});
}
NVBENCH_BENCH_TYPES(nstream_stable, NVBENCH_TYPE_AXES(element_types))
.set_name("nstream_stable")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", array_size_powers);

62
cccl_upstream/thrust/benchmarks/bench/transform/fib.cu Normal file
View File

@@ -0,0 +1,62 @@
// SPDX-FileCopyrightText: Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/transform.h>
#include <nvbench_helper.cuh>
template <class InT, class OutT>
struct fib_t
{
__device__ OutT operator()(InT n)
{
OutT t1 = 0;
OutT t2 = 1;
if (n <= 1)
{
return t1;
}
else if (n == 2)
{
return t2;
}
for (InT i = 3; i <= n; ++i)
{
const auto next = t1 + t2;
t1 = t2;
t2 = next;
}
return t2;
}
};
template <typename T>
static void fib(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> input = generate(elements, bit_entropy::_1_000, T{0}, T{42});
thrust::device_vector<T> output(elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<nvbench::uint32_t>(elements);
fib_t<T, nvbench::uint32_t> op{};
caching_allocator_t alloc; // transform shouldn't allocate, but let's be consistent
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::transform(policy(alloc, launch), input.cbegin(), input.cend(), output.begin(), op);
});
}
using types = nvbench::type_list<nvbench::uint32_t, nvbench::uint64_t>;
NVBENCH_BENCH_TYPES(fib, NVBENCH_TYPE_AXES(types))
.set_name("fib")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

38
cccl_upstream/thrust/benchmarks/bench/transform/if.cu Normal file
View File

@@ -0,0 +1,38 @@
// SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: BSD-3-Clause
#include <thrust/count.h>
#include <thrust/device_vector.h>
#include <thrust/transform.h>
#include <nvbench_helper.cuh>
template <typename T>
static void negate_if(nvbench::state& state, nvbench::type_list<T>)
{
const auto n = static_cast<std::size_t>(state.get_int64("Elements"));
const auto entropy = str_to_entropy(state.get_string("Entropy"));
const auto val = lerp_min_max<T>(entropy_to_probability(entropy));
auto transform_op = ::cuda::std::negate<T>{};
auto select_op = less_then_t<T>{val};
thrust::device_vector<T> input = generate(n);
thrust::device_vector<T> output(n, thrust::no_init);
const auto selected_elements = thrust::count_if(input.cbegin(), input.cend(), select_op);
state.add_element_count(n);
state.add_global_memory_reads<T>(n);
state.add_global_memory_writes<T>(selected_elements);
caching_allocator_t alloc; // transform_if shouldn't allocate, but let's be consistent
state.exec(
nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
thrust::transform_if(policy(alloc, launch), input.begin(), input.end(), output.begin(), transform_op, select_op);
});
}
NVBENCH_BENCH_TYPES(negate_if, NVBENCH_TYPE_AXES(integral_types))
// .set_name("negate_if")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_string_axis("Entropy", {"1.000", "0.544", "0.000"});

39
cccl_upstream/thrust/benchmarks/bench/transform/if_stencil.cu Normal file
View File

@@ -0,0 +1,39 @@
// SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/count.h>
#include <thrust/device_vector.h>
#include <thrust/transform.h>
#include <nvbench_helper.cuh>
template <typename T>
static void negate_if_stencil(nvbench::state& state, nvbench::type_list<T>)
{
const auto n = static_cast<std::size_t>(state.get_int64("Elements"));
const auto entropy = str_to_entropy(state.get_string("Entropy"));
const auto val = lerp_min_max<T>(entropy_to_probability(entropy));
auto transform_op = ::cuda::std::negate<T>{};
auto select_op = less_then_t<T>{val};
thrust::device_vector<T> input = generate(n);
thrust::device_vector<T> stencil = generate(n);
thrust::device_vector<T> output(n, thrust::no_init);
const auto selected_elements = thrust::count_if(input.cbegin(), input.cend(), select_op);
state.add_element_count(n);
state.add_global_memory_reads<T>(n + selected_elements);
state.add_global_memory_writes<T>(selected_elements);
caching_allocator_t alloc; // transform_if shouldn't allocate, but let's be consistent
state.exec(
nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
thrust::transform_if(
policy(alloc, launch), input.begin(), input.end(), stencil.begin(), output.begin(), transform_op, select_op);
});
}
NVBENCH_BENCH_TYPES(negate_if_stencil, NVBENCH_TYPE_AXES(integral_types))
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_string_axis("Entropy", {"1.000", "0.544", "0.000"});

55
cccl_upstream/thrust/benchmarks/bench/transform/zip_transform.cu Normal file
View File

@@ -0,0 +1,55 @@
// SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <cuda/functional>
#include <cuda/iterator>
#include <nvbench_helper.cuh>
// This benchmark is intended to be compared to the nstream benchmark from babelstream.cu, so we can:
// * detect regressions in the unpacking of a zip_transform_iterator
// same variables as in basic.cu so we can compare results
constexpr auto startA = 1; // BabelStream: 0.1
constexpr auto startB = 2; // BabelStream: 0.2
constexpr auto startC = 3; // BabelStream: 0.1
constexpr auto startScalar = 4; // BabelStream: 0.4
using element_types = nvbench::type_list<std::int8_t, std::int16_t, float, double, __int128>;
auto array_size_powers = std::vector<std::int64_t>{25, 31};
template <typename T>
static void nstream_zip_transform(nvbench::state& state, nvbench::type_list<T>)
{
const auto n = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> a(n, startA);
thrust::device_vector<T> b(n, startB);
thrust::device_vector<T> c(n, startC);
state.add_element_count(n);
state.add_global_memory_reads<T>(3 * n);
state.add_global_memory_writes<T>(n);
const T scalar = startScalar;
auto lambda = cuda::proclaim_copyable_arguments([scalar] _CCCL_DEVICE(const T& ai, const T& bi, const T& ci) -> T {
// Needed to silence clangs -Wunused-lambda-capture. We cannot just remove it because other
// implementations (e.g. MSVC) will emit errors if we don't capture it. See discussion in
// https://reviews.llvm.org/D28467.
static_cast<void>(scalar);
return ai + bi + scalar * ci;
});
cuda::zip_transform_iterator begin{lambda, a.begin(), b.begin(), c.begin()};
cuda::zip_transform_iterator end{lambda, a.end(), b.end(), c.end()};
caching_allocator_t alloc; // transform shouldn't allocate, but let's be consistent
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::copy(policy(alloc, launch), begin, end, a.begin());
});
}
NVBENCH_BENCH_TYPES(nstream_zip_transform, NVBENCH_TYPE_AXES(element_types))
.set_name("nstream")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", array_size_powers);

52
cccl_upstream/thrust/benchmarks/bench/transform_exclusive_scan/basic.cu Normal file
View File

@@ -0,0 +1,52 @@
//===----------------------------------------------------------------------===//
//
// Part of CUDA Experimental in CUDA C++ Core Libraries,
// under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES.
//
//===----------------------------------------------------------------------===//
#include <thrust/device_vector.h>
#include <thrust/transform_scan.h>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <class T>
struct times_two
{
_CCCL_DEVICE constexpr T operator()(const T val) const noexcept
{
return 2 * val;
}
};
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in = generate(elements);
thrust::device_vector<T> out(elements, thrust::no_init);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc{};
state.exec(
nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
do_not_optimize(thrust::transform_exclusive_scan(
policy(alloc, launch), in.begin(), in.end(), out.begin(), times_two<T>{}, T{42}, cuda::std::plus<T>{}));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

78
cccl_upstream/thrust/benchmarks/bench/transform_inclusive_scan/basic.cu Normal file
View File

@@ -0,0 +1,78 @@
//===----------------------------------------------------------------------===//
//
// Part of CUDA Experimental in CUDA C++ Core Libraries,
// under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES.
//
//===----------------------------------------------------------------------===//
#include <thrust/device_vector.h>
#include <thrust/transform_scan.h>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <class T>
struct times_two
{
_CCCL_DEVICE constexpr T operator()(const T val) const noexcept
{
return 2 * val;
}
};
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in = generate(elements);
thrust::device_vector<T> out(elements, thrust::no_init);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
do_not_optimize(thrust::transform_inclusive_scan(
policy(alloc, launch), in.begin(), in.end(), out.begin(), times_two<T>{}, cuda::std::plus<T>{}));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("basic")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));
template <typename T>
static void with_init(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in = generate(elements);
thrust::device_vector<T> out(elements, thrust::no_init);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc{};
state.exec(
nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
do_not_optimize(thrust::transform_inclusive_scan(
policy(alloc, launch), in.begin(), in.end(), out.begin(), times_two<T>{}, T{42}, cuda::std::plus<T>{}));
});
}
NVBENCH_BENCH_TYPES(with_init, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("with_init")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

41
cccl_upstream/thrust/benchmarks/bench/transform_reduce/sum.cu Normal file
View File

@@ -0,0 +1,41 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/transform_reduce.h>
#include "nvbench_helper.cuh"
template <class T>
struct square_t
{
__host__ __device__ T operator()(const T& x) const
{
return x * x;
}
};
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in = generate(elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(1);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
do_not_optimize(thrust::transform_reduce(
policy(alloc, launch), in.begin(), in.end(), square_t<T>{}, T{}, ::cuda::std::plus<T>{}));
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

44
cccl_upstream/thrust/benchmarks/bench/transform_reduce/unary.cu Normal file
View File

@@ -0,0 +1,44 @@
// SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include <thrust/device_vector.h>
#include <thrust/transform_reduce.h>
#include <cuda/memory_pool>
#include <cuda/stream>
#include "nvbench_helper.cuh"
template <class T>
struct plus_one
{
template <class U>
[[nodiscard]] __device__ constexpr T operator()(const U val) const noexcept
{
return static_cast<T>(val + 1);
}
};
template <typename T>
static void unary(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> in = generate(elements);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(1);
caching_allocator_t alloc{};
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
do_not_optimize(
thrust::transform_reduce(policy(alloc, launch), in.begin(), in.end(), plus_one<T>{}, 42, cuda::std::plus<T>{}));
});
}
NVBENCH_BENCH_TYPES(unary, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

66
cccl_upstream/thrust/benchmarks/bench/uninitialized_copy/basic.cu Normal file
View File

@@ -0,0 +1,66 @@
// SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: BSD-3-Clause
#include <thrust/device_vector.h>
#include <thrust/uninitialized_copy.h>
#include <nvbench_helper.cuh>
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
thrust::device_vector<T> input(elements, T{0xAA});
thrust::device_vector<T> output(elements, thrust::default_init);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(elements);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::uninitialized_copy(policy(alloc, launch), input.cbegin(), input.cend(), output.begin());
});
}
// Not allowed to be copied using std::memcpy or cudaMemcpy. Cannot use TMA copies.
struct no_copy
{
nvbench::uint32_t a;
no_copy() = default;
_CCCL_HOST_DEVICE no_copy(nvbench::uint32_t i)
: a(i)
{}
// the user-defined copy constructor prevents the type from being trivially copyable
// NOLINTNEXTLINE(modernize-use-equals-default)
_CCCL_HOST_DEVICE no_copy(const no_copy& nt)
: a(nt.a)
{}
};
static_assert(::cuda::std::is_trivially_default_constructible_v<no_copy>);
static_assert(!::cuda::std::is_trivially_copyable_v<no_copy>); // as required by the C++ standard
static_assert(!thrust::is_trivially_relocatable_v<no_copy>); // thrust uses this check internally
// Requires use of placement new
struct no_construct
{
nvbench::uint32_t a = 1337;
};
static_assert(!::cuda::std::is_trivially_default_constructible_v<no_construct>);
static_assert(::cuda::std::is_trivially_copyable_v<no_construct>); // as required by the C++ standard
static_assert(thrust::is_trivially_relocatable_v<no_construct>); // thrust uses this check internally
using types =
nvbench::type_list<nvbench::uint8_t, nvbench::uint16_t, nvbench::uint32_t, nvbench::uint64_t, no_copy, no_construct>;
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4));

40
cccl_upstream/thrust/benchmarks/bench/unique/basic.cu Normal file
View File

@@ -0,0 +1,40 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/unique.h>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const std::size_t min_segment_size = 1;
const std::size_t max_segment_size = static_cast<std::size_t>(state.get_int64("MaxSegSize"));
thrust::device_vector<T> input = generate.uniform.key_segments(elements, min_segment_size, max_segment_size);
thrust::device_vector<T> output(elements);
caching_allocator_t alloc;
// not a warm-up run, we need to run once to determine the size of the output
const auto new_end = thrust::unique_copy(policy(alloc), input.cbegin(), input.cend(), output.begin());
const std::size_t unique_items = ::cuda::std::distance(output.begin(), new_end);
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements);
state.add_global_memory_writes<T>(unique_items);
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::unique_copy(policy(alloc, launch), input.cbegin(), input.cend(), output.begin());
});
}
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(fundamental_types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_int64_power_of_two_axis("MaxSegSize", {1, 4, 8});

58
cccl_upstream/thrust/benchmarks/bench/unique/by_key.cu Normal file
View File

@@ -0,0 +1,58 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/unique.h>
#include "nvbench_helper.cuh"
template <class KeyT, class ValueT>
static void basic(nvbench::state& state, nvbench::type_list<KeyT, ValueT>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const std::size_t min_segment_size = 1;
const std::size_t max_segment_size = static_cast<std::size_t>(state.get_int64("MaxSegSize"));
thrust::device_vector<KeyT> in_keys = generate.uniform.key_segments(elements, min_segment_size, max_segment_size);
thrust::device_vector<KeyT> out_keys(elements);
thrust::device_vector<ValueT> in_vals(elements);
thrust::device_vector<ValueT> out_vals(elements);
caching_allocator_t alloc;
// not a warm-up run, we need to run once to determine the size of the output
const auto [new_key_end, new_val_end] = thrust::unique_by_key_copy(
policy(alloc), in_keys.cbegin(), in_keys.cend(), in_vals.cbegin(), out_keys.begin(), out_vals.begin());
const std::size_t unique_elements = ::cuda::std::distance(out_keys.begin(), new_key_end);
state.add_element_count(elements);
state.add_global_memory_reads<KeyT>(elements);
state.add_global_memory_writes<KeyT>(unique_elements);
state.add_global_memory_reads<ValueT>(elements);
state.add_global_memory_writes<ValueT>(unique_elements);
state.exec(
nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync, [&](nvbench::launch& launch) {
thrust::unique_by_key_copy(
policy(alloc, launch), in_keys.cbegin(), in_keys.cend(), in_vals.cbegin(), out_keys.begin(), out_vals.begin());
});
}
using key_types =
nvbench::type_list<int8_t,
int16_t,
int32_t,
int64_t
#if _CCCL_HAS_INT128()
,
int128_t
#endif
>;
using value_types = all_types;
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(key_types, value_types))
.set_name("base")
.set_type_axes_names({"KeyT{ct}", "ValueT{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_int64_power_of_two_axis("MaxSegSize", {1, 8});

43
cccl_upstream/thrust/benchmarks/bench/vectorized_search/base.cu Normal file
View File

@@ -0,0 +1,43 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/binary_search.h>
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/sort.h>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const auto needles_ratio = static_cast<std::size_t>(state.get_int64("NeedlesRatio"));
const auto needles = needles_ratio * static_cast<std::size_t>(static_cast<double>(elements) / 100.0);
thrust::device_vector<T> data = generate(elements + needles);
thrust::device_vector<bool> result(needles);
thrust::sort(data.begin(), data.begin() + elements);
state.add_element_count(needles);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::binary_search(
policy(alloc, launch),
data.begin(),
data.begin() + elements,
data.begin() + elements,
data.end(),
result.begin());
});
}
using types = nvbench::type_list<int8_t, int16_t, int32_t, int64_t>;
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_int64_axis("NeedlesRatio", {1, 25, 50});

43
cccl_upstream/thrust/benchmarks/bench/vectorized_search/lower_bound.cu Normal file
View File

@@ -0,0 +1,43 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/binary_search.h>
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/sort.h>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const auto needles_ratio = static_cast<std::size_t>(state.get_int64("NeedlesRatio"));
const auto needles = needles_ratio * static_cast<std::size_t>(static_cast<double>(elements) / 100.0);
thrust::device_vector<T> data = generate(elements + needles);
thrust::device_vector<T> result(needles);
thrust::sort(data.begin(), data.begin() + elements);
state.add_element_count(needles);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::lower_bound(
policy(alloc, launch),
data.begin(),
data.begin() + elements,
data.begin() + elements,
data.end(),
result.begin());
});
}
using types = nvbench::type_list<int8_t, int16_t, int32_t, int64_t>;
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_int64_axis("NeedlesRatio", {1, 25, 50});

43
cccl_upstream/thrust/benchmarks/bench/vectorized_search/upper_bound.cu Normal file
View File

@@ -0,0 +1,43 @@
// SPDX-FileCopyrightText: Copyright (c) 2011-2023, NVIDIA CORPORATION. All rights reserved.
// SPDX-License-Identifier: BSD-3
#include <thrust/binary_search.h>
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>
#include <thrust/sort.h>
#include "nvbench_helper.cuh"
template <typename T>
static void basic(nvbench::state& state, nvbench::type_list<T>)
{
const auto elements = static_cast<std::size_t>(state.get_int64("Elements"));
const auto needles_ratio = static_cast<std::size_t>(state.get_int64("NeedlesRatio"));
const auto needles = needles_ratio * static_cast<std::size_t>(static_cast<double>(elements) / 100.0);
thrust::device_vector<T> data = generate(elements + needles);
thrust::device_vector<T> result(needles);
thrust::sort(data.begin(), data.begin() + elements);
state.add_element_count(needles);
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch | nvbench::exec_tag::sync,
[&](nvbench::launch& launch) {
thrust::upper_bound(
policy(alloc, launch),
data.begin(),
data.begin() + elements,
data.begin() + elements,
data.end(),
result.begin());
});
}
using types = nvbench::type_list<int8_t, int16_t, int32_t, int64_t>;
NVBENCH_BENCH_TYPES(basic, NVBENCH_TYPE_AXES(types))
.set_name("base")
.set_type_axes_names({"T{ct}"})
.add_int64_power_of_two_axis("Elements", nvbench::range(16, 28, 4))
.add_int64_axis("NeedlesRatio", {1, 25, 50});

6
cccl_upstream/thrust/cmake/ThrustAddSubdir.cmake Normal file
View File

@@ -0,0 +1,6 @@
cccl_add_subdir_helper(
Thrust
# These component lists may be set by users to explicitly request subprojects:
REQUIRED_COMPONENTS ${THRUST_REQUIRED_SYSTEMS}
OPTIONAL_COMPONENTS ${THRUST_OPTIONAL_SYSTEMS}
)

38
cccl_upstream/thrust/cmake/ThrustBuildCompilerTargets.cmake Normal file
View File

@@ -0,0 +1,38 @@
# This file defines the `thrust_build_compiler_targets()` function, which
# creates the following interface targets:
#
# thrust.compiler_interface
# Provides compiler settings for all thrust tests, examples, etc. This should not be used
# directly, as it is linked to by all thrust configuration targets in THRUST_TARGETS.
function(thrust_build_compiler_targets)
set(cuda_compile_options)
set(cxx_compile_options)
set(cxx_compile_definitions)
if ("MSVC" STREQUAL "${CMAKE_CXX_COMPILER_ID}")
# Disabled loss-of-data conversion warnings.
# TODO Re-enable.
append_option_if_available("/wd4244" cxx_compile_options)
# Disable warning about applying unary operator- to unsigned type.
# TODO Re-enable.
append_option_if_available("/wd4146" cxx_compile_options)
endif()
cccl_build_compiler_interface(
thrust.compiler_flags
"${cuda_compile_options}"
"${cxx_compile_options}"
"${cxx_compile_definitions}"
)
add_library(thrust.compiler_interface INTERFACE)
target_link_libraries(
thrust.compiler_interface
INTERFACE
# order matters here, we need the project options to override the cccl options.
cccl.compiler_interface
thrust.compiler_flags
)
endfunction()

164
cccl_upstream/thrust/cmake/ThrustBuildTargetList.cmake Normal file
View File

@@ -0,0 +1,164 @@
# This file provides utilities for building and working with thrust
# configuration targets.
#
# THRUST_TARGETS
# - Built by the calling the `thrust_build_target_list()` function.
# - Each item is the name of a thrust interface target that is configured for a
# certain combination of host/device.
#
# thrust_build_target_list()
# - Creates the THRUST_TARGETS list.
#
# The following functions can be used to test/set metadata on a thrust target:
#
# thrust_get_target_property(<prop_var> <target_name> <prop>)
# - Checks the ${prop} target property on thrust target ${target_name}
# and sets the ${prop_var} variable in the caller's scope.
# - <prop_var> is any valid cmake identifier.
# - <target_name> is the name of a thrust target.
# - <prop> is one of the following:
# - HOST: The host system. Valid values: CPP, OMP, TBB.
# - DEVICE: The device system. Valid values: CUDA, CPP, OMP, TBB.
# - PREFIX: A unique "thrust.<host>.<device>" prefix that should be used to name all
# targets/tests/examples that use this configuration
# (e.g. ${config_label}.test.foo).
define_property(
TARGET
PROPERTY _THRUST_HOST
BRIEF_DOCS "A target's host system: CPP, TBB, or OMP."
FULL_DOCS "A target's host system: CPP, TBB, or OMP."
)
define_property(
TARGET
PROPERTY _THRUST_DEVICE
BRIEF_DOCS "A target's device system: CUDA, CPP, TBB, or OMP."
FULL_DOCS "A target's device system: CUDA, CPP, TBB, or OMP."
)
define_property(
TARGET
PROPERTY _THRUST_PREFIX
BRIEF_DOCS
"A prefix describing the host.device config, eg. 'thrust.cpp.cuda'."
FULL_DOCS "A prefix describing the host.device config, eg. 'thrust.cpp.cuda'."
)
function(thrust_set_target_properties target_name host device prefix)
cccl_configure_target(${target_name})
set_target_properties(
${target_name}
PROPERTIES
_THRUST_HOST ${host}
_THRUST_DEVICE ${device}
_THRUST_PREFIX ${prefix}
)
endfunction()
# Get a thrust property from a target and store it in var_name
# thrust_get_target_property(<var_name> <target_name> [HOST|DEVICE|PREFIX]
macro(thrust_get_target_property prop_var target_name prop)
get_property(${prop_var} TARGET ${target_name} PROPERTY _THRUST_${prop})
endmacro()
# Set ${var_name} to TRUE or FALSE in the caller's scope
function(_thrust_is_config_valid var_name host device)
# gersemi: off
if (THRUST_MULTICONFIG_ENABLE_SYSTEM_${host} AND
THRUST_MULTICONFIG_ENABLE_SYSTEM_${device} AND
"${host}_${device}" IN_LIST
THRUST_MULTICONFIG_WORKLOAD_${THRUST_MULTICONFIG_WORKLOAD}_CONFIGS)
# gersemi: on
set(${var_name} TRUE PARENT_SCOPE)
else()
set(${var_name} FALSE PARENT_SCOPE)
endif()
endfunction()
function(_thrust_init_target_list)
set(THRUST_TARGETS "" CACHE INTERNAL "" FORCE)
endfunction()
function(_thrust_add_target_to_target_list target_name host device prefix)
thrust_set_target_properties(${target_name} ${host} ${device} ${prefix})
target_link_libraries(${target_name} INTERFACE thrust.compiler_interface)
set(THRUST_TARGETS ${THRUST_TARGETS} ${target_name} CACHE INTERNAL "" FORCE)
message(STATUS "Enabling Thrust configuration: ${host}.${device}")
endfunction()
function(_thrust_build_target_list_multiconfig)
# Build THRUST_TARGETS
foreach (host IN LISTS THRUST_HOST_SYSTEM_OPTIONS)
foreach (device IN LISTS THRUST_DEVICE_SYSTEM_OPTIONS)
_thrust_is_config_valid(config_valid ${host} ${device})
if (config_valid)
set(prefix "thrust.${host}.${device}")
string(TOLOWER "${prefix}" prefix)
# Configure a thrust interface target for this host/device
set(target_name "${prefix}.config")
thrust_create_target(
${target_name}
HOST ${host}
DEVICE ${device}
DISPATCH ${THRUST_DISPATCH_TYPE}
${THRUST_TARGET_FLAGS}
)
# Set configuration metadata for this thrust interface target:
_thrust_add_target_to_target_list(${target_name} ${host} ${device} ${prefix})
endif()
endforeach() # devices
endforeach() # hosts
list(LENGTH THRUST_TARGETS count)
message(STATUS "${count} unique Thrust host.device configurations generated")
endfunction()
function(_thrust_build_target_list_singleconfig)
set(host ${THRUST_HOST_SYSTEM})
set(device ${THRUST_DEVICE_SYSTEM})
set(dialect ${THRUST_CPP_DIALECT})
set(prefix "thrust") # single config
# Target is created in ThrustFindThrust.cmake:
_thrust_add_target_to_target_list(thrust.config ${host} ${device} ${dialect} ${prefix})
endfunction()
# Build a ${THRUST_TARGETS} list containing target names for all
# requested configurations
function(thrust_build_target_list)
# Clear the list of targets:
_thrust_init_target_list()
# Generic config flags:
set(THRUST_TARGET_FLAGS)
macro(add_flag_option prefix flag docstring default)
set(opt "${prefix}_${flag}")
option(${opt} "${docstring}" "${default}")
mark_as_advanced(${opt})
if ("${prefix}" STREQUAL "CCCL" AND DEFINED THRUST_${flag})
message(
WARNING
"The THRUST_${flag} cmake option is deprecated. Use CCCL_${flag} instead."
)
set(CCCL_${flag} ${THRUST_${flag}})
endif()
if (${${opt}})
list(APPEND THRUST_TARGET_FLAGS ${flag})
endif()
endmacro()
# FIXME these should be moved out of the Thrust build if we care about them...
add_flag_option(CCCL IGNORE_DEPRECATED_CPP_DIALECT "Don't warn about any deprecated C++ standards and compilers." OFF)
add_flag_option(CCCL IGNORE_DEPRECATED_CPP_11 "Don't warn about deprecated C++11." OFF)
add_flag_option(CCCL IGNORE_DEPRECATED_CPP_14 "Don't warn about deprecated C++14." OFF)
add_flag_option(CCCL IGNORE_DEPRECATED_COMPILER "Don't warn about deprecated compilers." OFF)
add_flag_option(THRUST IGNORE_CUB_VERSION_CHECK "Don't warn about mismatched CUB versions." OFF)
add_flag_option(CCCL IGNORE_DEPRECATED_API "Don't warn about deprecated Thrust or CUB APIs." OFF)
if (THRUST_ENABLE_MULTICONFIG)
_thrust_build_target_list_multiconfig()
else()
_thrust_build_target_list_singleconfig()
endif()
endfunction()

21
cccl_upstream/thrust/cmake/ThrustCudaConfig.cmake Normal file
View File

@@ -0,0 +1,21 @@
enable_language(CUDA)
if (TARGET libcudacxx::libcudacxx)
# CUDA may not have been enabled when libcudacxx was found:
libcudacxx_update_language_compat_flags()
endif()
#
# Architecture options:
#
option(
THRUST_ENABLE_RDC_TESTS
"Enable tests that require separable compilation."
ON
)
option(
THRUST_FORCE_RDC
"Enable separable compilation on all targets that support it."
OFF
)

42
cccl_upstream/thrust/cmake/ThrustFindThrust.cmake Normal file
View File

@@ -0,0 +1,42 @@
function(_thrust_find_thrust_multiconfig)
# Check which systems are enabled by multiconfig:
set(req_systems)
if (THRUST_MULTICONFIG_ENABLE_SYSTEM_CUDA)
list(APPEND req_systems CUDA)
endif()
if (THRUST_MULTICONFIG_ENABLE_SYSTEM_CPP)
list(APPEND req_systems CPP)
endif()
if (THRUST_MULTICONFIG_ENABLE_SYSTEM_TBB)
list(APPEND req_systems TBB)
endif()
if (THRUST_MULTICONFIG_ENABLE_SYSTEM_OMP)
list(APPEND req_systems OMP)
endif()
find_package(
Thrust
REQUIRED
CONFIG
NO_DEFAULT_PATH # Only check the explicit path in HINTS:
HINTS "${CCCL_SOURCE_DIR}/lib/cmake/thrust/"
COMPONENTS ${req_systems}
)
endfunction()
function(_thrust_find_thrust_singleconfig)
cccl_get_thrust()
# Create target now to prepare system found flags:
thrust_create_target(thrust.config FROM_OPTIONS ${THRUST_TARGET_FLAGS})
thrust_debug_target(thrust.config "${THRUST_VERSION}")
endfunction()
# Find thrust along with all requested backends.
function(thrust_find_thrust)
if (THRUST_ENABLE_MULTICONFIG)
_thrust_find_thrust_multiconfig()
else()
_thrust_find_thrust_singleconfig()
endif()
endfunction()

170
cccl_upstream/thrust/cmake/ThrustHeaderTesting.cmake Normal file
View File

@@ -0,0 +1,170 @@
# For every public header, build a translation unit containing `#include <header>`
# to let the compiler try to figure out warnings in that header if it is not otherwise
# included in tests, and also to verify if the headers are modular enough.
# .inl files are not globbed for, because they are not supposed to be used as public
# entrypoints.
# Add regexes matching deprecated headers here to disable warnings for them:
set(
deprecated_headers_regexes
"thrust/iterator/tabulate_output_iterator\\.h"
"thrust/iterator/strided_iterator\\.h"
"thrust/iterator/constant_iterator\\.h"
)
cccl_get_cudatoolkit()
function(thrust_add_header_test thrust_target label definitions)
thrust_get_target_property(config_host ${thrust_target} HOST)
thrust_get_target_property(config_device ${thrust_target} DEVICE)
thrust_get_target_property(config_prefix ${thrust_target} PREFIX)
set(config_systems ${config_host} ${config_device})
string(TOLOWER "${config_host}" host_lower)
string(TOLOWER "${config_device}" device_lower)
if (config_device STREQUAL "CUDA")
set(langs CUDA)
else()
# Compile headers with both host and cuda compilers for CPU backends.
set(langs CXX CUDA)
endif()
# GLOB ALL THE THINGS
set(headers_globs thrust/*.h)
set(headers_exclude_systems_globs thrust/system/*/*)
set(
headers_systems_globs
thrust/system/${host_lower}/*
thrust/system/${device_lower}/*
)
set(
headers_exclude_details_globs
thrust/detail/*
thrust/*/detail/*
thrust/*/*/detail/*
)
# Get all .h files...
file(
GLOB_RECURSE headers
RELATIVE "${Thrust_SOURCE_DIR}"
CONFIGURE_DEPENDS
${headers_globs}
)
# ...then remove all system specific headers...
file(
GLOB_RECURSE headers_exclude_systems
RELATIVE "${Thrust_SOURCE_DIR}"
CONFIGURE_DEPENDS
${headers_exclude_systems_globs}
)
list(REMOVE_ITEM headers ${headers_exclude_systems})
# ...then add all headers specific to the selected host and device systems back again...
file(
GLOB_RECURSE headers_systems
RELATIVE "${Thrust_SOURCE_DIR}"
CONFIGURE_DEPENDS
${headers_systems_globs}
)
list(APPEND headers ${headers_systems})
# ...and remove all the detail headers (also removing the detail headers from the selected systems).
file(
GLOB_RECURSE headers_exclude_details
RELATIVE "${Thrust_SOURCE_DIR}"
CONFIGURE_DEPENDS
${headers_exclude_details_globs}
)
list(REMOVE_ITEM headers ${headers_exclude_details})
foreach (lang IN LISTS langs)
set(headertest_target ${config_prefix}.headers.${label})
if (lang STREQUAL "CUDA" AND (NOT config_device STREQUAL "CUDA"))
# Append .cuda to the header test target name when compiling
# CPU backends with cuda compilers.
set(headertest_target ${headertest_target}.cuda)
endif()
cccl_generate_header_tests(
${headertest_target}
thrust
LANGUAGE ${lang}
HEADERS ${headers}
PER_HEADER_DEFINES
DEFINE
CCCL_IGNORE_DEPRECATED_API
${deprecated_headers_regexes}
)
target_link_libraries(${headertest_target} PUBLIC ${thrust_target})
if (definitions)
target_compile_definitions(${headertest_target} PRIVATE ${definitions})
endif()
if (lang STREQUAL "CUDA")
thrust_configure_cuda_target(${headertest_target} RDC ${THRUST_FORCE_RDC})
endif()
if ("TBB" IN_LIST config_systems)
# In some cases cudafe++ doesn't implement certain builtins that are used in <immintrin.h> which causes the
# compilation to fail. In tbb_nvcc_preinclude.h, we forward declare those functions, so cudafe++ has no problems.
if (
"${lang}" STREQUAL "CUDA"
AND "${CMAKE_CUDA_COMPILER_ID}" STREQUAL "NVIDIA"
AND NOT MSVC
)
target_compile_options(
${headertest_target}
PUBLIC "-include" "${Thrust_SOURCE_DIR}/testing/tbb_nvcc_preinclude.h"
)
endif()
# Disable macro checks on TBB; the TBB atomic implementation uses `I` and
# our checks will issue false errors.
target_compile_definitions(
${headertest_target}
PRIVATE CCCL_IGNORE_HEADER_MACRO_CHECKS
)
# error #550-D: variable "alloc" was set but never used (in TBB headers)
# Only very specific configs are emitting this:
# gersemi: off
if (lang STREQUAL "CUDA" AND
"${CUDAToolkit_VERSION_MAJOR}.${CUDAToolkit_VERSION_MINOR}" VERSION_EQUAL "12.0" AND
MSVC_VERSION EQUAL 1939 AND
CMAKE_CUDA_STANDARD EQUAL 20
)
target_compile_options(
${headertest_target}
PRIVATE $<$<COMPILE_LANGUAGE:CUDA>:-diag-suppress 550>
)
endif()
endif()
endforeach() # lang
endfunction()
foreach (thrust_target IN LISTS THRUST_TARGETS)
thrust_get_target_property(config_device ${thrust_target} DEVICE)
thrust_add_header_test(${thrust_target} base "")
# Wrap Thrust/CUB in a custom namespace to check proper use of ns macros:
set(
header_definitions
"THRUST_WRAPPED_NAMESPACE=wrapped_thrust"
"CUB_WRAPPED_NAMESPACE=wrapped_cub"
)
thrust_add_header_test(${thrust_target} wrap "${header_definitions}")
if ("CUDA" STREQUAL "${config_device}")
# Check that BF16 support can be disabled
set(header_definitions "CCCL_DISABLE_BF16_SUPPORT")
thrust_add_header_test(${thrust_target} no_bf16 "${header_definitions}")
# Check that half support can be disabled
set(header_definitions "CCCL_DISABLE_FP16_SUPPORT")
thrust_add_header_test(${thrust_target} no_half "${header_definitions}")
endif()
endforeach()

143
cccl_upstream/thrust/cmake/ThrustMultiConfig.cmake Normal file
View File

@@ -0,0 +1,143 @@
# This file defines thrust_configure_multiconfig(), which sets up and handles
# the MultiConfig options that allow multiple host/device configurations
# to be generated from a single thrust build.
function(thrust_configure_multiconfig)
option(
THRUST_ENABLE_MULTICONFIG
"Enable multiconfig options for coverage testing."
OFF
)
if (THRUST_ENABLE_MULTICONFIG)
# Systems:
option(
THRUST_MULTICONFIG_ENABLE_SYSTEM_CPP
"Generate build configurations that use CPP."
ON
)
option(
THRUST_MULTICONFIG_ENABLE_SYSTEM_CUDA
"Generate build configurations that use CUDA."
ON
)
option(
THRUST_MULTICONFIG_ENABLE_SYSTEM_OMP
"Generate build configurations that use OpenMP."
OFF
)
option(
THRUST_MULTICONFIG_ENABLE_SYSTEM_TBB
"Generate build configurations that use TBB."
OFF
)
# Workload:
# - `SMALL`: [3 configs] Minimal coverage and validation of each device system against the `CPP` host.
# - `MEDIUM`: [6 configs] Cheap extended coverage.
# - `LARGE`: [8 configs] Expensive extended coverage. Include all useful build configurations.
# - `FULL`: [12 configs] The complete cross product of all possible build configurations.
#
# Config | Workloads | Value | Expense | Note
# ---------|-----------|------------|-----------|-----------------------------
# CPP/CUDA | F L M S | Essential | Expensive | Validates CUDA against CPP
# CPP/OMP | F L M S | Essential | Cheap | Validates OMP against CPP
# CPP/TBB | F L M S | Essential | Cheap | Validates TBB against CPP
# CPP/CPP | F L M | Important | Cheap | Tests CPP as device
# OMP/OMP | F L M | Important | Cheap | Tests OMP as host
# TBB/TBB | F L M | Important | Cheap | Tests TBB as host
# TBB/CUDA | F L | Important | Expensive | Validates TBB/CUDA interop
# OMP/CUDA | F L | Important | Expensive | Validates OMP/CUDA interop
# TBB/OMP | F | Not useful | Cheap | Mixes CPU-parallel systems
# OMP/TBB | F | Not useful | Cheap | Mixes CPU-parallel systems
# TBB/CPP | F | Not Useful | Cheap | Parallel host, serial device
# OMP/CPP | F | Not Useful | Cheap | Parallel host, serial device
set(
THRUST_MULTICONFIG_WORKLOAD
SMALL
CACHE STRING
"Limit host/device configs: SMALL (up to 3 h/d combos per dialect), MEDIUM(6), LARGE(8), FULL(12)"
)
set_property(
CACHE THRUST_MULTICONFIG_WORKLOAD
PROPERTY STRINGS SMALL MEDIUM LARGE FULL
)
set(
THRUST_MULTICONFIG_WORKLOAD_SMALL_CONFIGS
CPP_OMP
CPP_TBB
CPP_CUDA
CACHE INTERNAL
"Host/device combos enabled for SMALL workloads."
FORCE
)
set(
THRUST_MULTICONFIG_WORKLOAD_MEDIUM_CONFIGS
${THRUST_MULTICONFIG_WORKLOAD_SMALL_CONFIGS}
CPP_CPP
TBB_TBB
OMP_OMP
CACHE INTERNAL
"Host/device combos enabled for MEDIUM workloads."
FORCE
)
set(
THRUST_MULTICONFIG_WORKLOAD_LARGE_CONFIGS
${THRUST_MULTICONFIG_WORKLOAD_MEDIUM_CONFIGS}
OMP_CUDA
TBB_CUDA
CACHE INTERNAL
"Host/device combos enabled for LARGE workloads."
FORCE
)
set(
THRUST_MULTICONFIG_WORKLOAD_FULL_CONFIGS
${THRUST_MULTICONFIG_WORKLOAD_LARGE_CONFIGS}
OMP_CPP
TBB_CPP
OMP_TBB
TBB_OMP
CACHE INTERNAL
"Host/device combos enabled for FULL workloads."
FORCE
)
# Hide the single config options if they exist from a previous run:
if (DEFINED THRUST_HOST_SYSTEM)
set_property(CACHE THRUST_HOST_SYSTEM PROPERTY TYPE INTERNAL)
set_property(CACHE THRUST_DEVICE_SYSTEM PROPERTY TYPE INTERNAL)
endif()
else() # Single config:
# Restore system option visibility if these cache options already exist
# from a previous run.
if (DEFINED THRUST_HOST_SYSTEM)
set_property(CACHE THRUST_HOST_SYSTEM PROPERTY TYPE STRING)
else()
set(
THRUST_HOST_SYSTEM
"CPP"
CACHE STRING
"The targeted host system: ${THRUST_HOST_SYSTEM_OPTIONS}"
)
set_property(
CACHE THRUST_HOST_SYSTEM
PROPERTY STRINGS ${THRUST_HOST_SYSTEM_OPTIONS}
)
endif()
if (DEFINED THRUST_DEVICE_SYSTEM)
set_property(CACHE THRUST_DEVICE_SYSTEM PROPERTY TYPE STRING)
else()
set(
THRUST_DEVICE_SYSTEM
"CUDA"
CACHE STRING
"The targeted device system: ${THRUST_DEVICE_SYSTEM_OPTIONS}"
)
set_property(
CACHE THRUST_DEVICE_SYSTEM
PROPERTY STRINGS ${THRUST_DEVICE_SYSTEM_OPTIONS}
)
endif()
endif()
endfunction()

52
cccl_upstream/thrust/cmake/ThrustRunExample.cmake Normal file
View File

@@ -0,0 +1,52 @@
# Inputs:
#
# Variable | Type | Doc
# ---------------------|----------|--------------------------------------
# EXAMPLE_EXECUTABLE | FilePath | Path to example executable
# FILECHECK_ENABLED | Boolean | Run FileCheck comparison test
# FILECHECK_EXECUTABLE | FilePath | Path to the LLVM FileCheck utility
# REFERENCE_FILE | FilePath | Path to the FileCheck reference file
if (FILECHECK_ENABLED)
if (NOT EXISTS "${REFERENCE_FILE}")
message(
FATAL_ERROR
"FileCheck requested for '${EXAMPLE_EXECUTABLE}', but reference file "
"does not exist at '${REFERENCE_FILE}`."
)
endif()
# If the reference file is empty, validate that the example doesn't
# produce any output.
file(SIZE "${REFERENCE_FILE}" file_size)
message("${REFERENCE_FILE}: ${file_size} bytes")
if (file_size EQUAL 0)
set(check_empty_output TRUE)
set(filecheck_command)
else()
set(check_empty_output FALSE)
set(filecheck_command COMMAND "${FILECHECK_EXECUTABLE}" "${REFERENCE_FILE}")
endif()
endif()
execute_process(
COMMAND "${EXAMPLE_EXECUTABLE}" ${filecheck_command}
RESULT_VARIABLE exit_code
OUTPUT_VARIABLE stdout
ERROR_VARIABLE stderr
)
if (NOT 0 EQUAL exit_code)
message(FATAL_ERROR "${EXAMPLE_EXECUTABLE} failed (${exit_code}):\n${stderr}")
endif()
if (check_empty_output)
string(LENGTH "${stdout}" stdout_size)
if (NOT stdout_size EQUAL 0)
message(
FATAL_ERROR
"${EXAMPLE_EXECUTABLE}: output received, but not expected:\n${stdout}"
)
endif()
endif()

60
cccl_upstream/thrust/cmake/ThrustUtilities.cmake Normal file
View File

@@ -0,0 +1,60 @@
# Given a cu_file (e.g. foo/bar.cu) relative to CMAKE_CURRENT_SOURCE_DIR
# and a thrust_target, create a cpp file that includes the .cu file, and set
# ${cpp_file_var} in the parent scope to the full path of the new file. The new
# file will be generated in:
# ${CMAKE_CURRENT_BINARY_DIR}/<thrust_target_prefix>/${cu_file}.cpp
function(thrust_wrap_cu_in_cpp cpp_file_var cu_file thrust_target)
thrust_get_target_property(prefix ${thrust_target} PREFIX)
set(wrapped_source_file "${CMAKE_CURRENT_SOURCE_DIR}/${cu_file}")
set(cpp_file "${CMAKE_CURRENT_BINARY_DIR}/${prefix}/${cu_file}.cpp")
configure_file(
"${Thrust_SOURCE_DIR}/cmake/wrap_source_file.cpp.in"
"${cpp_file}"
)
set(${cpp_file_var} "${cpp_file}" PARENT_SCOPE)
endfunction()
# thrust_configure_cuda_target(<target_name> RDC <ON|OFF>)
#
# Configures `target_name` with the appropriate CUDA architectures and RDC state.
function(thrust_configure_cuda_target target_name)
set(options)
set(one_value_args RDC)
set(multi_value_args)
cmake_parse_arguments(
thrust_cuda
"${options}"
"${one_value_args}"
"${multi_value_args}"
${ARGN}
)
if (thrust_cuda_UNPARSED_ARGUMENTS)
message(
AUTHOR_WARNING
"Unrecognized arguments passed to thrust_configure_cuda_target: "
${thrust_cuda_UNPARSED_ARGUMENTS}
)
endif()
if (NOT DEFINED thrust_cuda_RDC)
message(
AUTHOR_WARNING
"RDC option required for thrust_configure_cuda_target."
)
endif()
if (thrust_cuda_RDC)
set_target_properties(
${target_name}
PROPERTIES #
CUDA_SEPARABLE_COMPILATION ON
POSITION_INDEPENDENT_CODE ON
)
else()
set_target_properties(
${target_name}
PROPERTIES CUDA_SEPARABLE_COMPILATION OFF
)
endif()
endfunction()

1
cccl_upstream/thrust/cmake/filecheck_smoke_test Normal file
View File

@@ -0,0 +1 @@
SMOKE

1
cccl_upstream/thrust/cmake/wrap_source_file.cpp.in Normal file
View File

@@ -0,0 +1 @@
#include <${wrapped_source_file}> // NOLINT(bugprone-suspicious-include)

179
cccl_upstream/thrust/examples/CMakeLists.txt Normal file
View File

@@ -0,0 +1,179 @@
# Setup FileCheck if requested and available:
option(
THRUST_ENABLE_EXAMPLE_FILECHECK
"Check example output with the LLVM FileCheck utility."
OFF
)
set(filecheck_data_path "${Thrust_SOURCE_DIR}/internal/test")
if (THRUST_ENABLE_EXAMPLE_FILECHECK)
# TODO this should go into a find module
find_program(
THRUST_FILECHECK_EXECUTABLE
DOC "Path to the LLVM FileCheck utility."
NAMES
FileCheck
FileCheck-3.9
FileCheck-4.0
FileCheck-5.0
FileCheck-6.0
FileCheck-7
FileCheck-8
FileCheck-9
)
if (NOT THRUST_FILECHECK_EXECUTABLE)
message(
FATAL_ERROR
"Could not find the LLVM FileCheck utility. Set THRUST_FILECHECK_EXECUTABLE manually, "
"or disable THRUST_ENABLE_EXAMPLE_FILECHECK."
)
endif()
execute_process(
COMMAND
"${THRUST_FILECHECK_EXECUTABLE}"
"${filecheck_data_path}/thrust.smoke.filecheck"
INPUT_FILE "${Thrust_SOURCE_DIR}/cmake/filecheck_smoke_test"
RESULT_VARIABLE exit_code
)
if (0 EQUAL exit_code)
message(STATUS "FileCheck enabled: ${THRUST_FILECHECK_EXECUTABLE}")
else()
message(
FATAL_ERROR
"The current THRUST_FILECHECK_EXECUTABLE ('${THRUST_FILECHECK_EXECUTABLE}') "
"does not seem to be a valid FileCheck executable."
)
endif()
endif()
## thrust_add_example
#
# Add an example executable and register it with ctest.
#
# target_name_var: Variable name to overwrite with the name of the example
# target. Useful for post-processing target information per-backend.
# example_name: The name of the example minus "<config_prefix>.example." For
# instance, examples/vector.cu will be "vector", and examples/cuda/copy.cu
# would be "cuda.copy".
# example_src: The source file that implements the example.
# thrust_target: The reference thrust target with configuration information.
#
function(
thrust_add_example
target_name_var
example_name
example_src
thrust_target
)
thrust_get_target_property(config_host ${thrust_target} HOST)
thrust_get_target_property(config_device ${thrust_target} DEVICE)
thrust_get_target_property(config_prefix ${thrust_target} PREFIX)
# Wrap the .cu file in .cpp for non-CUDA backends
if ("CUDA" STREQUAL "${config_device}")
set(real_example_src "${example_src}")
else()
thrust_wrap_cu_in_cpp(real_example_src "${example_src}" ${thrust_target})
endif()
# The actual name of the test's target:
set(example_target ${config_prefix}.example.${example_name})
set(${target_name_var} ${example_target} PARENT_SCOPE)
cccl_add_executable(${example_target} SOURCES "${real_example_src}")
target_link_libraries(${example_target} PRIVATE ${thrust_target})
target_include_directories(
${example_target}
PRIVATE "${Thrust_SOURCE_DIR}/examples"
)
if ("CUDA" STREQUAL "${config_device}")
thrust_configure_cuda_target(${example_target} RDC ${THRUST_FORCE_RDC})
endif()
# We do not want to explicitly include `host_device.h` if not needed,
# so force include the file for non CUDA targets.
set(gx_msvc "$<CXX_COMPILER_ID:MSVC>")
set(gx_cxx "$<COMPILE_LANGUAGE:CXX>")
set(gx_cxx_msvc "$<AND:${gx_cxx},${gx_msvc}>")
set(gx_cxx_not_msvc "$<AND:${gx_cxx},$<NOT:${gx_msvc}>>")
target_compile_options(
${example_target}
PRIVATE
"$<${gx_cxx_msvc}:SHELL:/FI include/host_device.h>"
"$<${gx_cxx_not_msvc}:SHELL:-include include/host_device.h>"
)
target_compile_definitions(
${example_target}
PRIVATE $<$<COMPILE_LANG_AND_ID:CUDA,NVIDIA>:THRUST_EXAMPLE_DEVICE_SIDE>
)
# Get the name of FileCheck input by stripping out the config name.
# (e.g. "thrust.cpp.cuda.cpp14.example.xxx" -> "thrust.example.xxx.filecheck")
string(
REPLACE
"${config_prefix}"
"thrust"
filecheck_reference_file
"${example_target}.filecheck"
)
add_test(
NAME ${example_target}
# gersemi: off
COMMAND
"${CMAKE_COMMAND}"
"-DEXAMPLE_EXECUTABLE=$<TARGET_FILE:${example_target}>"
"-DFILECHECK_ENABLED=${THRUST_ENABLE_EXAMPLE_FILECHECK}"
"-DFILECHECK_EXECUTABLE=${THRUST_FILECHECK_EXECUTABLE}"
"-DREFERENCE_FILE=${filecheck_data_path}/${filecheck_reference_file}"
-P "${Thrust_SOURCE_DIR}/cmake/ThrustRunExample.cmake"
# gersemi: on
)
# Run OMP/TBB tests in serial. Multiple OMP processes will massively
# oversubscribe the machine with GCC's OMP, and we want to test these with
# the full CPU available to each unit test.
set(config_systems ${config_host} ${config_device})
if (("OMP" IN_LIST config_systems) OR ("TBB" IN_LIST config_systems))
set_tests_properties(${example_target} PROPERTIES RUN_SERIAL ON)
endif()
# Check for per-example script. Script will be included in the current scope
# to allow custom property modifications.
get_filename_component(example_cmake_script "${example_src}" NAME_WLE)
set(
example_cmake_script
"${CMAKE_CURRENT_LIST_DIR}/${example_cmake_script}.cmake"
)
# Use a glob so we can detect if this changes:
file(
GLOB example_cmake_script
RELATIVE "${CMAKE_CURRENT_LIST_DIR}"
CONFIGURE_DEPENDS
"${example_cmake_script}"
)
if (example_cmake_script) # Will be non-empty only if the script exists
include("${example_cmake_script}")
endif()
endfunction()
file(
GLOB example_srcs
RELATIVE "${CMAKE_CURRENT_LIST_DIR}"
CONFIGURE_DEPENDS
*.cu
*.cpp
)
foreach (thrust_target IN LISTS THRUST_TARGETS)
foreach (example_src IN LISTS example_srcs)
get_filename_component(example_name "${example_src}" NAME_WLE)
thrust_add_example(example_target ${example_name} "${example_src}" ${thrust_target})
endforeach()
endforeach()
add_subdirectory(cuda)

13
cccl_upstream/thrust/examples/README.md Normal file
View File

@@ -0,0 +1,13 @@
Once Thrust has been installed, these example programs can be compiled
directly with nvcc. For example, the following command will compile the
`norm` example.
```bash
$ nvcc norm.cu -o norm
```
These examples are also available online:
https://github.com/NVIDIA/cccl/tree/main/thrust/examples
For any serious experimentation, we recommend using CMake and [CCCL from GitHub](https://github.com/NVIDIA/cccl).
We also provide consistent and convenient development environments as [devcontainers](../../.devcontainers/README.md).

93
cccl_upstream/thrust/examples/arbitrary_transformation.cu Normal file
View File

@@ -0,0 +1,93 @@
#include <thrust/detail/config.h>
#include <thrust/device_vector.h>
#include <thrust/for_each.h>
#include <thrust/iterator/zip_iterator.h>
#include <thrust/zip_function.h>
#include <iostream>
// This example shows how to implement an arbitrary transformation of
// the form output[i] = F(first[i], second[i], third[i], ... ).
// In this example, we use a function with 3 inputs and 1 output.
//
// Iterators for all four vectors (3 inputs + 1 output) are "zipped"
// into a single sequence of tuples with the zip_iterator.
//
// The arbitrary_functor receives a tuple that contains four elements,
// which are references to values in each of the four sequences. When we
// access the tuple 't' with the get() function,
// get<0>(t) returns a reference to A[i],
// get<1>(t) returns a reference to B[i],
// get<2>(t) returns a reference to C[i],
// get<3>(t) returns a reference to D[i].
//
// In this example, we can implement the transformation,
// D[i] = A[i] + B[i] * C[i];
// by invoking arbitrary_functor() on each of the tuples using for_each.
//
// If we are using a functor that is not designed for zip iterators by taking a
// tuple instead of individual arguments we can adapt this function using the
// zip_function adaptor (C++11 only).
//
// Note that we could extend this example to implement functions with an
// arbitrary number of input arguments by zipping more sequence together.
// With the same approach we can have multiple *output* sequences, if we
// wanted to implement something like
// D[i] = A[i] + B[i] * C[i];
// E[i] = A[i] + B[i] + C[i];
//
// The possibilities are endless! :)
struct arbitrary_functor1
{
template <typename Tuple>
__host__ __device__ void operator()(Tuple t)
{
// D[i] = A[i] + B[i] * C[i];
cuda::std::get<3>(t) = cuda::std::get<0>(t) + cuda::std::get<1>(t) * cuda::std::get<2>(t);
}
};
struct arbitrary_functor2
{
__host__ __device__ void operator()(const float& a, const float& b, const float& c, float& d)
{
// D[i] = A[i] + B[i] * C[i];
d = a + b * c;
}
};
int main()
{
// allocate and initialize
thrust::device_vector<float> A{3, 4, 0, 8, 2};
thrust::device_vector<float> B{6, 7, 2, 1, 8};
thrust::device_vector<float> C{2, 5, 7, 4, 3};
thrust::device_vector<float> D1(5);
// apply the transformation
thrust::for_each(thrust::make_zip_iterator(A.begin(), B.begin(), C.begin(), D1.begin()),
thrust::make_zip_iterator(A.end(), B.end(), C.end(), D1.end()),
arbitrary_functor1());
// print the output
std::cout << "Tuple functor" << '\n';
for (size_t i = 0; i < A.size(); i++)
{
std::cout << A[i] << " + " << B[i] << " * " << C[i] << " = " << D1[i] << '\n';
}
// apply the transformation using zip_function
thrust::device_vector<float> D2(5);
thrust::for_each(thrust::make_zip_iterator(A.begin(), B.begin(), C.begin(), D2.begin()),
thrust::make_zip_iterator(A.end(), B.end(), C.end(), D2.end()),
thrust::make_zip_function(arbitrary_functor2()));
// print the output
std::cout << "N-ary functor" << '\n';
for (size_t i = 0; i < A.size(); i++)
{
std::cout << A[i] << " + " << B[i] << " * " << C[i] << " = " << D2[i] << '\n';
}
}

40
cccl_upstream/thrust/examples/basic_vector.cu Normal file
View File

@@ -0,0 +1,40 @@
#include <thrust/device_vector.h>
#include <thrust/host_vector.h>
#include <iostream>
int main()
{
// H holds 4 integers
thrust::host_vector<int> H{14, 20, 38, 46};
// H.size() returns the size of vector H
std::cout << "H has size " << H.size() << '\n';
// print contents of H
for (size_t i = 0; i < H.size(); i++)
{
std::cout << "H[" << i << "] = " << H[i] << '\n';
}
// resize H
H.resize(2);
std::cout << "H now has size " << H.size() << '\n';
// Copy host_vector H to device_vector D
thrust::device_vector<int> D = H;
// elements of D can be modified
D[0] = 99;
D[1] = 88;
// print contents of D
for (size_t i = 0; i < D.size(); i++)
{
std::cout << "D[" << i << "] = " << D[i] << '\n';
}
// H and D are automatically deleted when the function returns
return 0;
}

101
cccl_upstream/thrust/examples/bounding_box.cu Normal file
View File

@@ -0,0 +1,101 @@
#include <thrust/device_vector.h>
#include <thrust/extrema.h>
#include <thrust/random.h>
#include <thrust/transform_reduce.h>
#include <cuda/std/utility>
#include <iostream>
// This example shows how to compute a bounding box
// for a set of points in two dimensions.
struct point2d
{
float x, y;
__host__ __device__ point2d()
: x(0)
, y(0)
{}
__host__ __device__ point2d(float _x, float _y)
: x(_x)
, y(_y)
{}
};
// bounding box type
struct bbox
{
// construct an empty box
bbox() = default;
// construct a box from a single point
__host__ __device__ bbox(const point2d& point)
: lower_left(point)
, upper_right(point)
{}
// construct a box from a single point
__host__ __device__ bbox& operator=(const point2d& point)
{
lower_left = point;
upper_right = point;
return *this;
}
// construct a box from a pair of points
__host__ __device__ bbox(const point2d& ll, const point2d& ur)
: lower_left(ll)
, upper_right(ur)
{}
point2d lower_left, upper_right;
};
// reduce a pair of bounding boxes (a,b) to a bounding box containing a and b
struct bbox_union
{
__host__ __device__ bbox operator()(bbox a, bbox b)
{
// lower left corner
point2d ll(thrust::min(a.lower_left.x, b.lower_left.x), thrust::min(a.lower_left.y, b.lower_left.y));
// upper right corner
point2d ur(thrust::max(a.upper_right.x, b.upper_right.x), thrust::max(a.upper_right.y, b.upper_right.y));
return bbox(ll, ur);
}
};
int main()
{
const size_t N = 40;
// allocate storage for points
thrust::device_vector<point2d> points(N);
// generate some random points in the unit square
thrust::default_random_engine rng;
thrust::uniform_real_distribution<float> u01(0.0f, 1.0f);
for (size_t i = 0; i < N; i++)
{
float x = u01(rng);
float y = u01(rng);
points[i] = point2d(x, y);
}
// initial bounding box contains first point
bbox init(points[0], points[0]);
// compute the bounding box for the point set
bbox result = thrust::reduce(points.begin(), points.end(), init, bbox_union{});
// print output
std::cout << "bounding box " << std::fixed;
std::cout << "(" << result.lower_left.x << "," << result.lower_left.y << ") ";
std::cout << "(" << result.upper_right.x << "," << result.upper_right.y << ")" << '\n';
return 0;
}

106
cccl_upstream/thrust/examples/bucket_sort2d.cu Normal file
View File

@@ -0,0 +1,106 @@
#include <thrust/binary_search.h>
#include <thrust/device_vector.h>
#include <thrust/generate.h>
#include <thrust/host_vector.h>
#include <thrust/iterator/counting_iterator.h>
#include <thrust/random.h>
#include <thrust/sort.h>
#include <iomanip>
#include <iostream>
// define a 2d float vector
using vec2 = cuda::std::tuple<float, float>;
// return a random vec2 in [0,1)^2
vec2 make_random_vec2()
{
static thrust::default_random_engine rng;
static thrust::uniform_real_distribution<float> u01(0.0f, 1.0f);
float x = u01(rng);
float y = u01(rng);
return vec2(x, y);
}
// hash a point in the unit square to the index of
// the grid bucket that contains it
struct point_to_bucket_index
{
unsigned int width; // buckets in the x dimension (grid spacing = 1/width)
unsigned int height; // buckets in the y dimension (grid spacing = 1/height)
__host__ __device__ point_to_bucket_index(unsigned int width, unsigned int height)
: width(width)
, height(height)
{}
__host__ __device__ unsigned int operator()(const vec2& v) const
{
// find the raster indices of p's bucket
unsigned int x = static_cast<unsigned int>(cuda::std::get<0>(v) * static_cast<float>(width));
unsigned int y = static_cast<unsigned int>(cuda::std::get<1>(v) * static_cast<float>(height));
// return the bucket's linear index
return y * width + x;
}
};
int main()
{
const size_t N = 1000000;
// allocate some random points in the unit square on the host
thrust::host_vector<vec2> h_points(N);
thrust::generate(h_points.begin(), h_points.end(), make_random_vec2);
// transfer to device
thrust::device_vector<vec2> points = h_points;
// allocate storage for a 2D grid
// of dimensions w x h
unsigned int w = 200, h = 100;
// the grid data structure keeps a range per grid bucket:
// each bucket_begin[i] indexes the first element of bucket i's list of points
// each bucket_end[i] indexes one past the last element of bucket i's list of points
thrust::device_vector<unsigned int> bucket_begin(w * h);
thrust::device_vector<unsigned int> bucket_end(w * h);
// allocate storage for each point's bucket index
thrust::device_vector<unsigned int> bucket_indices(N);
// transform the points to their bucket indices
thrust::transform(points.begin(), points.end(), bucket_indices.begin(), point_to_bucket_index(w, h));
// sort the points by their bucket index
thrust::sort_by_key(bucket_indices.begin(), bucket_indices.end(), points.begin());
// find the beginning of each bucket's list of points
thrust::counting_iterator<unsigned int> search_begin(0);
thrust::lower_bound(
bucket_indices.begin(),
bucket_indices.end(),
search_begin,
search_begin + static_cast<decltype(search_begin)::difference_type>(w) * h,
bucket_begin.begin());
// find the end of each bucket's list of points
thrust::upper_bound(
bucket_indices.begin(),
bucket_indices.end(),
search_begin,
search_begin + static_cast<decltype(search_begin)::difference_type>(w) * h,
bucket_end.begin());
// write out bucket (150, 50)'s list of points
unsigned int bucket_idx = 50 * w + 150;
std::cout << "bucket (150, 50)'s list of points:" << '\n';
std::cout << std::fixed << std::setprecision(6);
for (unsigned int point_idx = bucket_begin[bucket_idx]; point_idx != bucket_end[bucket_idx]; ++point_idx)
{
vec2 p = points[point_idx];
std::cout << "(" << cuda::std::get<0>(p) << "," << cuda::std::get<1>(p) << ")" << '\n';
}
return 0;
}

26
cccl_upstream/thrust/examples/constant_iterator.cu Normal file
View File

@@ -0,0 +1,26 @@
#define CCCL_IGNORE_DEPRECATED_API
#include <thrust/copy.h>
#include <thrust/device_vector.h>
#include <thrust/functional.h>
#include <thrust/transform.h>
#include <cuda/iterator>
#include <iostream>
#include <iterator>
int main()
{
thrust::device_vector<int> data{3, 7, 2, 5};
// add 10 to all values in data
thrust::transform(data.begin(), data.end(), cuda::constant_iterator<int>(10), data.begin(), cuda::std::plus<int>());
// data is now [13, 17, 12, 15]
// print result
thrust::copy(data.begin(), data.end(), std::ostream_iterator<int>(std::cout, "\n"));
return 0;
}

34
cccl_upstream/thrust/examples/counting_iterator.cu Normal file
View File

@@ -0,0 +1,34 @@
#include <thrust/copy.h>
#include <thrust/device_vector.h>
#include <thrust/functional.h>
#include <thrust/iterator/counting_iterator.h>
#include <iostream>
#include <iterator>
int main()
{
// this example computes indices for all the nonzero values in a sequence
// sequence of zero and nonzero values
thrust::device_vector<int> stencil{0, 1, 1, 0, 0, 1, 0, 1};
// storage for the nonzero indices
thrust::device_vector<int> indices(8);
// counting iterators define a sequence [0, 8)
thrust::counting_iterator<int> first(0);
thrust::counting_iterator<int> last = first + 8;
// compute indices of nonzero elements
using IndexIterator = thrust::device_vector<int>::iterator;
IndexIterator indices_end = thrust::copy_if(first, last, stencil.begin(), indices.begin(), cuda::std::identity{});
// indices now contains [1,2,5,7]
// print result
std::cout << "found " << cuda::std::distance(indices.begin(), indices_end) << " nonzero values at indices:\n";
thrust::copy(indices.begin(), indices_end, std::ostream_iterator<int>(std::cout, "\n"));
return 0;
}

20
cccl_upstream/thrust/examples/cpp_integration/README Normal file
View File

@@ -0,0 +1,20 @@
This example shows how to link a Thrust program contained in
a .cu file with a C++ program contained in a .cpp file. Note
that device_vector only appears in the .cu file while host_vector
appears in both. This relects the fact that algorithms on device
vectors are only available when the contents of the program are
located in a .cu file and compiled with the nvcc compiler.
On a Linux system where Thrust is installed in the default location
we can use the following procedure to compile the two parts of the
program and link them together.
$ nvcc -O2 -c device.cu
$ g++ -O2 -c host.cpp -I/usr/local/cuda/include/
$ nvcc -o tester device.o host.o
Alternatively, we can use g++ to perform final linking step.
$ nvcc -O2 -c device.cu
$ g++ -O2 -c host.cpp -I/usr/local/cuda/include/
$ g++ -o tester device.o host.o -L/usr/local/cuda/lib64 -lcudart

Some files were not shown because too many files have changed in this diff Show More